S3cmd

S3cmd is a command line tool to upload, retrieve and manage data in Amazon S3. It is best suited for power users who don't fear command line. The project is also ideal for scripts, automated backups triggered from cron, etc.

Amazon S3 is a reasonably priced data storage service. Ideal for off-site backups, archiving and other data storage needs. Check out our pages about Amazon S3 to find out more.

Simple S3cmd HowTo

Register for Amazon AWS / S3

Go to Amazon S3 homepage, click on the "Sign up for web service" button in the right column and work through the registration. You will have to supply your Credit Card details in order to allow Amazon charge you for S3 usage. At the end you should posses your Access and Secret Keys

Run s3cmd --configure

You will be asked for the two keys - copy and paste them from your confirmation email or from your Amazon account page. Be careful when copying them! They are case sensitive and must be entered accurately or you'll keep getting errors about invalid signatures or similar.

Run s3cmd ls to list all your buckets.

As you just started using S3 there are no buckets owned by you as of now. So the output will be empty.

Make a bucket with s3cmd mb s3://my-new-bucket-name

As mentioned above bucket names must be unique amongst _all_ users of S3. That means the simple names like "test" or "asdf" are already taken and you must make up something more original. I sometimes prefix my bucket names with my e-mail domain name (logix.cz) leading to a bucket name, for instance, 'logix.cz-test':

~$ s3cmd mb s3://logix.cz-test
Bucket 'logix.cz-test' created

List your buckets again with s3cmd ls

Now you should see your freshly created bucket

~$ s3cmd ls
2007-01-19 01:41 s3://logix.cz-test

List the contents of the bucket

~$ s3cmd ls s3://logix.cz-test
Bucket 'logix.cz-test':
~$

It's empty, indeed.

Upload a file into the bucket

~$ s3cmd put addressbook.xml s3://logix.cz-test/addrbook.xml
File 'addressbook.xml' stored as s3://logix.cz-test/addrbook.xml (123456 bytes)

Now we can list the bucket contents again

~$ s3cmd ls s3://logix.cz-test
Bucket 'logix.cz-test':
2007-01-19 01:46 120k s3://logix.cz-test/addrbook.xml

Retrieve the file back and verify that its hasn't been corrupted

~$ s3cmd get s3://logix.cz-test/addrbook.xml addressbook-2.xml
Object s3://logix.cz-test/addrbook.xml saved as 'addressbook-2.xml' (123456 bytes)

~$ md5sum addressbook.xml addressbook-2.xml
39bcb6992e461b269b95b3bda303addf addressbook.xml
39bcb6992e461b269b95b3bda303addf addressbook-2.xml

Checksums of the original file matches the one of the retrieved one. Looks like it worked.

Clean up: delete the object and remove the bucket

~$ s3cmd rb s3://logix.cz-test
ERROR: S3 error: 409 (Conflict): BucketNotEmpty

Ouch, we can only remove empty buckets!

~$ s3cmd del s3://logix.cz-test/addrbook.xml
Object s3://logix.cz-test/addrbook.xml deleted

~$ s3cmd rb s3://logix.cz-test
Bucket 'logix.cz-test' removed

What is new in this release:

• This release added commands for copying and moving remote files, CloudFront support, a new [setacl] command for setting an ACL on existing objects, and recursive and wildcard support for [put], [get], and [del]. --dry-run was enabled for [put], [get], and [sync]. Removal of non-empty buckets is allowed.
• A progress meter was implemented.
• New --include, --rinclude, and --(r)include-from options were added to override --exclude exclusions.
• A --add-header option was added along with a --list-md5 option for [ls].

What is new in version 0.9.8.4:

• This release restores access to upper-case named buckets.
• It has improved handling of filenames with Unicode characters.
• It avoids ZeroDivisionError on very fast links (for instance on Amazon EC2).
• It will reissue failed requests (e.g. connection errors, internal server errors, etc).
• Sync skips over files that can't be opened instead of terminating the sync completely.
• It doesn't run out of open files quota on sync with lots of files.