s3cmd是一款 Amazon S3 命令行工具。
它不仅能上传、下载、同步,还能设置权限,下面是完整的安装使用指南。
本文将指导Linux下安装s3cmd程序,以及对象存储服务桶和对象数据管理,包括创建桶、上传、检索、删除及本地与对象存储服务间数据同步等。
// 从 yum仓库下载
# yum -y install s3cmd
# rpm -qa | grep s3cmd
s3cmd-2.3.0-4.el7.noarch
# rpm -ql s3cmd
/usr/bin/s3cmd
/usr/lib/python2.7/site-packages/S3
/usr/lib/python2.7/site-packages/S3/ACL.py
/usr/lib/python2.7/site-packages/S3/ACL.pyc
/usr/lib/python2.7/site-packages/S3/ACL.pyo
/usr/lib/python2.7/site-packages/S3/AccessLog.py
/usr/lib/python2.7/site-packages/S3/AccessLog.pyc
/usr/lib/python2.7/site-packages/S3/AccessLog.pyo
/usr/lib/python2.7/site-packages/S3/BaseUtils.py
/usr/lib/python2.7/site-packages/S3/BaseUtils.pyc
/usr/lib/python2.7/site-packages/S3/BaseUtils.pyo
/usr/lib/python2.7/site-packages/S3/BidirMap.py
/usr/lib/python2.7/site-packages/S3/BidirMap.pyc
/usr/lib/python2.7/site-packages/S3/BidirMap.pyo
/usr/lib/python2.7/site-packages/S3/CloudFront.py
/usr/lib/python2.7/site-packages/S3/CloudFront.pyc
/usr/lib/python2.7/site-packages/S3/CloudFront.pyo
/usr/lib/python2.7/site-packages/S3/Config.py
/usr/lib/python2.7/site-packages/S3/Config.pyc
/usr/lib/python2.7/site-packages/S3/Config.pyo
/usr/lib/python2.7/site-packages/S3/ConnMan.py
/usr/lib/python2.7/site-packages/S3/ConnMan.pyc
/usr/lib/python2.7/site-packages/S3/ConnMan.pyo
/usr/lib/python2.7/site-packages/S3/Crypto.py
/usr/lib/python2.7/site-packages/S3/Crypto.pyc
/usr/lib/python2.7/site-packages/S3/Crypto.pyo
/usr/lib/python2.7/site-packages/S3/Custom_httplib27.py
/usr/lib/python2.7/site-packages/S3/Custom_httplib27.pyc
/usr/lib/python2.7/site-packages/S3/Custom_httplib27.pyo
/usr/lib/python2.7/site-packages/S3/Exceptions.py
/usr/lib/python2.7/site-packages/S3/Exceptions.pyc
/usr/lib/python2.7/site-packages/S3/Exceptions.pyo
/usr/lib/python2.7/site-packages/S3/ExitCodes.py
/usr/lib/python2.7/site-packages/S3/ExitCodes.pyc
/usr/lib/python2.7/site-packages/S3/ExitCodes.pyo
/usr/lib/python2.7/site-packages/S3/FileDict.py
/usr/lib/python2.7/site-packages/S3/FileDict.pyc
/usr/lib/python2.7/site-packages/S3/FileDict.pyo
/usr/lib/python2.7/site-packages/S3/FileLists.py
/usr/lib/python2.7/site-packages/S3/FileLists.pyc
/usr/lib/python2.7/site-packages/S3/FileLists.pyo
/usr/lib/python2.7/site-packages/S3/HashCache.py
/usr/lib/python2.7/site-packages/S3/HashCache.pyc
/usr/lib/python2.7/site-packages/S3/HashCache.pyo
/usr/lib/python2.7/site-packages/S3/MultiPart.py
/usr/lib/python2.7/site-packages/S3/MultiPart.pyc
/usr/lib/python2.7/site-packages/S3/MultiPart.pyo
/usr/lib/python2.7/site-packages/S3/PkgInfo.py
/usr/lib/python2.7/site-packages/S3/PkgInfo.pyc
/usr/lib/python2.7/site-packages/S3/PkgInfo.pyo
/usr/lib/python2.7/site-packages/S3/Progress.py
/usr/lib/python2.7/site-packages/S3/Progress.pyc
/usr/lib/python2.7/site-packages/S3/Progress.pyo
/usr/lib/python2.7/site-packages/S3/S3.py
/usr/lib/python2.7/site-packages/S3/S3.pyc
/usr/lib/python2.7/site-packages/S3/S3.pyo
/usr/lib/python2.7/site-packages/S3/S3Uri.py
/usr/lib/python2.7/site-packages/S3/S3Uri.pyc
/usr/lib/python2.7/site-packages/S3/S3Uri.pyo
/usr/lib/python2.7/site-packages/S3/SortedDict.py
/usr/lib/python2.7/site-packages/S3/SortedDict.pyc
/usr/lib/python2.7/site-packages/S3/SortedDict.pyo
/usr/lib/python2.7/site-packages/S3/Utils.py
/usr/lib/python2.7/site-packages/S3/Utils.pyc
/usr/lib/python2.7/site-packages/S3/Utils.pyo
/usr/lib/python2.7/site-packages/S3/__init__.py
/usr/lib/python2.7/site-packages/S3/__init__.pyc
/usr/lib/python2.7/site-packages/S3/__init__.pyo
/usr/lib/python2.7/site-packages/s3cmd-2.3.0-py2.7.egg-info
/usr/lib/python2.7/site-packages/s3cmd-2.3.0-py2.7.egg-info/PKG-INFO
/usr/lib/python2.7/site-packages/s3cmd-2.3.0-py2.7.egg-info/SOURCES.txt
/usr/lib/python2.7/site-packages/s3cmd-2.3.0-py2.7.egg-info/dependency_links.txt
/usr/lib/python2.7/site-packages/s3cmd-2.3.0-py2.7.egg-info/requires.txt
/usr/lib/python2.7/site-packages/s3cmd-2.3.0-py2.7.egg-info/top_level.txt
/usr/share/doc/s3cmd-2.3.0
/usr/share/doc/s3cmd-2.3.0/NEWS
/usr/share/doc/s3cmd-2.3.0/README.md
/usr/share/licenses/s3cmd-2.3.0
/usr/share/licenses/s3cmd-2.3.0/LICENSE
/usr/share/man/man1/s3cmd.1.gz
# which /usr/bin/s3cmd
/usr/bin/s3cmd
# file /usr/bin/s3cmd
/usr/bin/s3cmd: Python script, ASCII text executable, with very long lines
# head /usr/bin/s3cmd
# -*- coding: utf-8 -*-
from __future__ import absolute_import, print_function, division
import sys
if sys.version_info < (2, 6):
sys.stderr.write(u"ERROR: Python 2.6 or higher required, sorry.\n")
# 72 == EX_OSFILE
sys.exit(72)
PY3 = (sys.version_info >= (3, 0))
这段代码是典型的Python脚本环境配置样板,主要实现了三个关键功能:
- 环境适配
- 强制要求Python 2.6+环境,通过版本检查阻止低版本运行
- 通过
__future__
导入预置Python 3特性,提升代码的跨版本兼容性 - 使用
PY3
标志位明确区分Python 2/3运行时环境
- 执行保障
- 规范shebang声明确保*nix系统正确识别解释器
- UTF-8编码声明规避字符集解析问题
- 通过标准错误流输出错误提示,符合Unix工具规范
- 工程化设计
- 使用系统标准错误码72(EX_OSFILE)规范退出状态
- 模块级字符串说明工具用途
- 结构化异常处理前置,提前阻断不兼容的执行环境
这段代码展现了Python脚本开发的最佳实践:通过严格的版本控制和环境预检,确保程序在兼容环境中稳定运行;利用语言特性实现跨版本支持,体现了防御性编程思想。其设计模式特别适合于需要长期维护的命令行工具开发场景。
// 从官方github仓库下载
# wget https://github.com/s3tools/s3cmd/releases/download/v2.4.0/s3cmd-2.4.0.tar.gz
# tar -zxf s3cmd-2.4.0.tar.gz -C /usr/local/
# ln -s /usr/local/s3cmd-2.4.0/s3cmd /usr/bin/
# s3cmd -help
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ImportError trying to import dateutil.parser.
Please install the python dateutil module:
$ sudo apt-get install python-dateutil
or
$ sudo yum install python-dateutil
or
$ pip install python-dateutil
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# yum install python-dateutil
# s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 123456
Secret Key: 123456
Default Region [US]: CN
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: 192.168.1.101:1443
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:
这个 DNS-style bucket+hostname:port template for accessing a bucket 是什么,应该如何填写?
在配置s3cmd
时,当遇到DNS-style bucket+hostname:port template选项时,填写方式取决于你的S3服务是否支持虚拟主机(DNS)风格的访问。
- 什么是DNS-style模板?
该选项用于生成桶的访问URL,格式为:
<bucket-name>.<hostname>:<port>
例如,默认值%(bucket)s.s3.amazonaws.com
会生成URL:mybucket.s3.amazonaws.com
。
这种风格要求服务支持DNS解析(即通过子域名访问桶)。
- 如何填写?
根据你的S3服务类型选择以下方案:
场景1:使用亚马逊S3或支持DNS风格的服务
- 保持默认值:
%(bucket)s.s3.amazonaws.com
。
场景2:自建S3服务(如MinIO、Ceph)或第三方兼容服务(使用IP/域名+端口)
如果服务支持虚拟主机风格(如MinIO默认支持):
填写%(bucket)s.192.168.1.101:1443
。此时生成的URL为:mybucket.192.168.1.101:1443
。如果服务仅支持路径风格(常见于本地测试或旧版本服务):
留空或填写%(bucket)s
。此时生成的URL为:192.168.1.101:1443/mybucket
。
- 你的情况分析
你配置的端点为192.168.1.101:1443
(可能是MinIO或本地S3服务),通常这类服务默认支持路径风格。
推荐做法:
直接按Enter
留空(或输入%(bucket)s
),强制使用路径风格访问,避免DNS解析问题。
- 验证配置
完成配置后,通过以下命令测试:
s3cmd ls
若出现错误(如Bucket not found
),可能是DNS风格不兼容,需返回配置并将模板留空。
- 总结
- 虚拟主机风格需要DNS支持,适合公有云服务(如AWS S3)。
-
路径风格更通用,适合自建服务或IP直连场景。
根据你的网络环境和服务类型选择即可!
基本上你无法直接通过 s3cmd --configure 交互式配置对象存储信息,可以根据配置文件语法填写你的信息。
# vim .s3cfg
#----------------------------------------------------#
[default]
access_key = xxxxxx
secret_key = xxxxx
host_base = 192.168.1.101:1443
host_bucket = 192.168.1.101:1443/your-bucket-name
use_https = False
#----------------------------------------------------#
// 查看桶文件
# s3cmd -c ${配置文件} ls s3://${桶名}/${桶里面路径}
参考
s3cmd s3命令行工具
https://www.cnblogs.com/hiyang/p/12631922.html
对象存储测试工具s3cmd介绍及使用说明
https://www.ctyun.cn/developer/article/418540363796549
s3tools/s3cmd
https://github.com/s3tools/s3cmd
https://github.com/s3tools/s3cmd/tags
Amazon S3 Tools: Command Line S3 Client Software and S3 Backup
https://s3tools.org/download
AWS S3工具最佳实践 S3 cmd
https://docs.ksyun.com/documents/42570