Table of Contents
This can help you on following:
- How to use S3cmd to manage your Object Storage?
- How to Sync Files and Directories to AWS S3 using s3cmd Tool?
- Getting started with AWS s3cmd command by examples.
- How to use Object Storage with s3cmd?
- How to upload multiple files from directory to S3?
s3cmd is a command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol, such as Google Cloud Storage or DreamHost DreamObjects.
We can manage S3 buckets and objects without logging into AWS console. It is also ideal for batch scripts and automated backup to S3, triggered from cron, etc.
For example: If you need to configure any backups (say db backups or anything) to s3 we can use s3cmd to manage it.
Installation steps
We can install s3cmd using “yum” and “apt-get” based on the Linux flavour that you use. The versions that you get from the yum or apt-get may not be the latest versions. If you need to use the latest, you need to install it from the source code.
On CentOS systems, we can use:
# yum install s3cmd
On Ubuntu/Debian based use:
# apt-get install s3cmd
If you’re a non-root user, use with sudo instead.
# sudo apt-get install s3cmd
If you need to install s3cmd from source code:
# wget https://github.com/s3tools/s3cmd/archive/v2.1.0.tar.gz
# tar -xvzf v2.1.0.tar.gz
# cd s3cmd-2.1.0/
# sudo python setup.py install (You need to have python 2.5 or 3 available on the machine)
Now, we need to configure s3cmd. We need to give privilege for the machine to access our s3 service. Before configuring s3cmd, we need to have an IAM user that have access to s3 service. Now, it’s time to create an IAM user if you don’t have any.
- Login into your Amazon console.
- Create an IAM user (Identity Access and Management) by navigating to Security Identity and Compliance.
- Attach a policy directly to the user, so that the user can access s3 service. You can go with Attach existing policies directly.
You will be provided with an Access key and secret key. Take a note of that.
It’s time to run s3cmd configure on the machine.
# s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: **********************
Secret Key: ***********************
Default Region [US]: ap-south-1
Use “s3.amazonaws.com” for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: s3.amazonaws.com
Use “%(bucket)s.s3.amazonaws.com” to the target Amazon S3. “%(bucket)s” and “%(location)s” vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: **********
Path to GPG program [/usr/bin/gpg]: /usr/bin/gpg
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: yes
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can’t connect to S3 directly
HTTP Proxy server name:
New settings:
Access Key: **********************
Secret Key: ************************
Default Region: ap-south-1
S3 Endpoint: s3.amazonaws.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
Encryption password: **********
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine 🙂
Now verifying that encryption works…
Success. Encryption and decryption worked fine 🙂
Save settings? [y/N] y
Configuration saved to '/home/heba/.s3cfg'
Yeah, now the configuration is completed!
Configuration file of s3cmd:
We will get all the configuration settings for s3cmd from /home/user/.s3cfg. We will get the accesskey and secretkey as well.
Are you ready to get your hands dirty with s3cmd…?? Yeah, let’s move on.
It’s time for us to create a new bucket using s3cmd.
Create a new bucket
# s3cmd mb s3://bucketname
# s3cmd mb s3://tx.tripmentor.in
Bucket 's3://tx.tripmentor.in/' created
List the buckets
# s3cmd ls
$ s3cmd ls
2021-02-14 00:59 s3://austin.tripmentor.in
2021-02-15 00:33 s3://tx.tripmentor.in
Listing contents in the folder in S3 bucket
# s3cmd ls s3://bucketname/directory/*
$ s3cmd ls s3://austin.tripmentor.in/testdir/*
2021-02-15 08:05 13 s3://austin.tripmentor.in/testdir/hi.txt
2021-02-15 08:05 78 s3://austin.tripmentor.in/testdir/mytest.txt
2021-02-15 08:05 25 s3://austin.tripmentor.in/testdir/new.txt
Uploading files to S3 bucket
# s3cmd put file_name s3://bucketname
Uploading multiple files
#s3cmd put file1.txt hi.txt s3://bucketname
$ s3cmd put file1.txt hi.txt s3://austin.tripmentor.in
upload: 'file1.txt' -> 's3://austin.tripmentor.in/file1.txt' [1 of 2]
27 of 27 100% in 4s 5.61 B/s done
upload: 'hi.txt' -> 's3://austin.tripmentor.in/hi.txt' [2 of 2]
13 of 13 100% in 1s 10.80 B/s done
Uploading directory in s3
# s3cmd put -r directory_name s3://bucketname
$ s3cmd put -r testdir s3://tx.tripmentor.in
upload: 'testdir/hi.txt' -> 's3://tx.tripmentor.in/testdir/hi.txt' [1 of 3]
13 of 13 100% in 1s 8.79 B/s done
upload: 'testdir/hi.txt' -> 's3://tx.tripmentor.in/testdir/hi.txt' [1 of 3]
13 of 13 100% in 0s 35.02 B/s done
upload: 'testdir/mytest.txt' -> 's3://tx.tripmentor.in/testdir/mytest.txt' [2 of 3]
78 of 78 100% in 0s 583.98 B/s done
upload: 'testdir/new.txt' -> 's3://tx.tripmentor.in/testdir/new.txt' [3 of 3]
25 of 25 100% in 0s 138.74 B/s done
Downloading file/directory
# s3cmd get s3://bucketname/file1
$ s3cmd get s3://tx.tripmentor.in/testdir/hi.txt
download: 's3://tx.tripmentor.in/testdir/hi.txt' -> './hi.txt' [1 of 1]
download: 's3://tx.tripmentor.in/testdir/hi.txt' -> './hi.txt' [1 of 1]
13 of 13 100% in 0s 57.85 B/s done
For a directory to download, use -r option so that all the contents will be downloaded recursively.
# s3cmd get -r s3://bucketname/directory
# s3cmd get -r s3://austin.tripmentor.in/testdir
download: 's3://austin.tripmentor.in/testdir/hi.txt' -> './testdir/hi.txt' [1 of 3]
13 of 13 100% in 1s 7.03 B/s done
download: 's3://austin.tripmentor.in/testdir/mytest.txt' -> './testdir/mytest.txt' [2 of 3]
78 of 78 100% in 2s 32.83 B/s done
download: 's3://austin.tripmentor.in/testdir/new.txt' -> './testdir/new.txt' [3 of 3]
25 of 25 100% in 0s 34.96 B/s done
Emptying a bucket
# s3cmd del s3://bucketname/file1
# s3cmd del s3://bucketname/directory/*
$ s3cmd del s3://austin.tripmentor.in/error.html
delete: 's3://austin.tripmentor.in/error.html'
$ s3cmd del s3://austin.tripementor.in/testdir/*
delete: 's3://austin.tripmentor.in/testdir/hi.txt'
delete: 's3://austin.tripmentor.in/testdir/mytest.txt'
delete: 's3://austin.tripmentor.in/testdir/new.txt'
Deleting a bucket
# s3cmd rb s3://bucketname
$ s3cmd rb s3://austin.tripmentor.in
ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty
Here, the bucket was not empty. So, first we need to empty the bucket and then proceed with removing it.
# s3cmd del s3://austin.tripmentor.in/* delete: 's3://austin.tripmentor.in/file1.txt' delete: 's3://austin.tripmentor.in/hello.html' delete: 's3://austin.tripmentor.in/hi.txt' $ s3cmd rb s3://austin.tripmentor.in Bucket 's3://austin.tripmentor.in/' removed
That’s how we manage s3 using s3cmd.
S3 service is not only used for storage and retrieval operations. It is used to host a static website too. Static webpage means hosting a website using simple HTML page. You may be wondering about the use case of this. Say, if you need to host a maintenance page if your website is down or in case if you’re just building your website and you may need to have a landing page, in that case you can use it. If interested to know more, please refer to my blog post https://www.crybit.com/configure-s3-to-host-a-static-website/
