All about s3cmd

s3cmd is a command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol, such as Google Cloud Storage or DreamHost DreamObjects.

We can manage S3 buckets and objects without logging into AWS console. It is also ideal for batch scripts and automated backup to S3, triggered from cron, etc.

For example: If you need to configure any backups (say db backups or anything) to s3 we can use s3cmd to manage it. 

Installation steps

We can install s3cmd using “yum” and “apt-get” based on the Linux flavour that you use. The versions that you get from the yum or apt-get may not be the latest versions. If you need to use the latest, you need to install it from the source code.

On CentOS systems, we can use:

# yum install s3cmd

On Ubuntu/Debian based use:

# apt-get install s3cmd

If you’re a non-root user, use with sudo instead.

# sudo apt-get install s3cmd

If you need to install s3cmd from source code:

# wget https://github.com/s3tools/s3cmd/archive/v2.1.0.tar.gz
# tar -xvzf v2.1.0.tar.gz
# cd s3cmd-2.1.0/
# sudo python setup.py install (You need to have python 2.5 or 3 available on the machine)

Now, we need to configure s3cmd. We need to give privilege for the machine to access our s3 service. Before configuring s3cmd, we need to have an IAM user that have access to s3 service. Now, it’s time to create an IAM user if you don’t have any.

  • Login into your Amazon console.
  • Create an IAM user (Identity Access and Management) by navigating to Security Identity and Compliance.
  • Attach a policy directly to the user, so that the user can access s3 service. You can go with Attach existing policies directly.

You will be provided with an Access key and secret key. Take a note of that.

It’s time to run s3cmd configure on the machine.

# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: **********************
Secret Key: ***********************
Default Region [US]: ap-south-1

Use “s3.amazonaws.com” for S3 Endpoint and not modify it to the target Amazon S3.

S3 Endpoint [s3.amazonaws.com]: s3.amazonaws.com

Use “%(bucket)s.s3.amazonaws.com” to the target Amazon S3. “%(bucket)s” and “%(location)s” vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3

Encryption password: **********
Path to GPG program [/usr/bin/gpg]: /usr/bin/gpg

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer

Use HTTPS protocol [Yes]: yes

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can’t connect to S3 directly

HTTP Proxy server name:

New settings:
  Access Key: **********************
  Secret Key: ************************
  Default Region: ap-south-1
  S3 Endpoint: s3.amazonaws.com
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
  Encryption password: **********
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...

Success. Your access key and secret key worked fine 🙂

Now verifying that encryption works…
Success. Encryption and decryption worked fine 🙂

Save settings? [y/N] y
Configuration saved to '/home/heba/.s3cfg'

Yeah, now the configuration is completed!

Configuration  file of s3cmd:

We will get all the configuration settings for s3cmd from /home/user/.s3cfg. We will get the accesskey and secretkey as well.

Are you ready to get your hands dirty with s3cmd…?? Yeah, let’s move on.

It’s time for us to create a new bucket using s3cmd.

Create a new bucket

# s3cmd mb s3://bucketname
# s3cmd mb s3://tx.tripmentor.in
Bucket 's3://tx.tripmentor.in/' created

List the buckets

# s3cmd ls
$ s3cmd ls
2021-02-14 00:59  s3://austin.tripmentor.in
2021-02-15 00:33  s3://tx.tripmentor.in

Listing contents in the folder in S3 bucket

# s3cmd ls s3://bucketname/directory/*
$ s3cmd ls s3://austin.tripmentor.in/testdir/*
2021-02-15 08:05           13  s3://austin.tripmentor.in/testdir/hi.txt
2021-02-15 08:05           78  s3://austin.tripmentor.in/testdir/mytest.txt
2021-02-15 08:05           25  s3://austin.tripmentor.in/testdir/new.txt

Uploading files to S3 bucket

# s3cmd put file_name s3://bucketname

          Uploading multiple files
         #s3cmd put file1.txt hi.txt s3://bucketname
  $ s3cmd put file1.txt hi.txt s3://austin.tripmentor.in
upload: 'file1.txt' -> 's3://austin.tripmentor.in/file1.txt'  [1 of 2]
 27 of 27   100% in    4s     5.61 B/s  done
upload: 'hi.txt' -> 's3://austin.tripmentor.in/hi.txt'  [2 of 2]
 13 of 13   100% in    1s    10.80 B/s  done

Uploading directory in s3

# s3cmd put -r directory_name s3://bucketname
$ s3cmd put -r testdir s3://tx.tripmentor.in
upload: 'testdir/hi.txt' -> 's3://tx.tripmentor.in/testdir/hi.txt'  [1 of 3]
 13 of 13   100% in    1s     8.79 B/s  done
upload: 'testdir/hi.txt' -> 's3://tx.tripmentor.in/testdir/hi.txt'  [1 of 3]
 13 of 13   100% in    0s    35.02 B/s  done
upload: 'testdir/mytest.txt' -> 's3://tx.tripmentor.in/testdir/mytest.txt'  [2 of 3]
 78 of 78   100% in    0s   583.98 B/s  done
upload: 'testdir/new.txt' -> 's3://tx.tripmentor.in/testdir/new.txt'  [3 of 3]
 25 of 25   100% in    0s   138.74 B/s  done

Downloading file/directory

# s3cmd get s3://bucketname/file1
$ s3cmd get s3://tx.tripmentor.in/testdir/hi.txt
download: 's3://tx.tripmentor.in/testdir/hi.txt' -> './hi.txt'  [1 of 1]
download: 's3://tx.tripmentor.in/testdir/hi.txt' -> './hi.txt'  [1 of 1]
 13 of 13   100% in    0s    57.85 B/s  done

For a directory to download, use -r option so that all the contents will be downloaded recursively.

# s3cmd get -r s3://bucketname/directory
# s3cmd get -r s3://austin.tripmentor.in/testdir
download: 's3://austin.tripmentor.in/testdir/hi.txt' -> './testdir/hi.txt'  [1 of 3]
 13 of 13   100% in    1s     7.03 B/s  done
download: 's3://austin.tripmentor.in/testdir/mytest.txt' -> './testdir/mytest.txt'  [2 of 3]
 78 of 78   100% in    2s    32.83 B/s  done
download: 's3://austin.tripmentor.in/testdir/new.txt' -> './testdir/new.txt'  [3 of 3]
 25 of 25   100% in    0s    34.96 B/s  done

Emptying a bucket

# s3cmd del s3://bucketname/file1
# s3cmd del s3://bucketname/directory/*
$ s3cmd del s3://austin.tripmentor.in/error.html
delete: 's3://austin.tripmentor.in/error.html'
$ s3cmd del s3://austin.tripementor.in/testdir/*
delete: 's3://austin.tripmentor.in/testdir/hi.txt'
delete: 's3://austin.tripmentor.in/testdir/mytest.txt'
delete: 's3://austin.tripmentor.in/testdir/new.txt'

Deleting a bucket

# s3cmd rb s3://bucketname
$ s3cmd rb s3://austin.tripmentor.in
ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty

Here, the bucket was not empty. So, first we need to empty the bucket and then proceed with removing it.

# s3cmd del s3://austin.tripmentor.in/*
delete: 's3://austin.tripmentor.in/file1.txt'
delete: 's3://austin.tripmentor.in/hello.html'
delete: 's3://austin.tripmentor.in/hi.txt'

$ s3cmd rb s3://austin.tripmentor.in
Bucket 's3://austin.tripmentor.in/' removed

That’s how we manage s3 using s3cmd.

S3 service is not only used for storage and retrieval operations. It is used to host a static website too. Static webpage means hosting a website using simple HTML page. You may be wondering about the use case of this. Say, if you need to host a maintenance page if your website is down or in case if you’re just building your website and you may need to have a landing page, in that case you can use it. If interested to know more, please refer to my blog post https://www.crybit.com/configure-s3-to-host-a-static-website/

Post navigation

Heba Habeeb

Working as a Linux Server Admin, Infopark, Cochin, Kerala.

Leave a Reply

Your email address will not be published. Required fields are marked *