Use s3cmd for Managing Amazon S3 Storage from the CLI

Amazon S3 storage is dirt cheap. Whether it’s for work or personal use, S3 makes for a great repository for backups and other data you want to store. But most admins want a way to interact with S3 that doesn’t involve dealing with the Web-based interface. If you’re looking to script backups or other operations, you need a reliable CLI tool that works with S3–and s3cmd does the trick nicely.

You can find s3cmd on the S3 Tools site. Many Linux distros (like Debian and Ubuntu) package it for you.

After installing s3cmd, you must configure it with your S3 Amazon keys. Run s3cmd--configure and provide the keys to s3cmd. I probably don’t need to say this, but just in case: You might not want to use s3cmd on systems that you don’t control. So if you have a shell account on a system that someone else has root on, storing your Amazon S3 credentials on the system is something you might want to avoid.

You’ll also be prompted to provide an encryption password and asked if you want to use HTTPS. The default is no, but I enable HTTPS for my use.

Now what? Use s3cmd ls to display all of your buckets. If you don’t have any buckets, or you want to create a new one, you can create one using s3cmd mb s3://bucketname. List the contents of a bucket using s3 ls s3://bucketname.

If you want to upload or download individual files, use the put and get commands. The del command will (of course) delete a file.

Want to know how much you have stored in S3? Use the s3cmd du -H command. The -H is optional, but it provides the amounts in “human readable” form–so you see “3M” instead of “3293136” or “5G” instead of “5520830577.”

What’s really useful is the s3 sync command. Just use the s3cmd sync directory s3://bucketname. It works the other way too, of course, so you can easily sync directories stored on S3 back to local machines.

The only caveat I have for s3cmd is that it becomes really easy to start stuffing files into S3 for backup purposes. Watch your Amazon bill. While it’s dirt cheap to store files there (starting at $0.14 per GB for standard and $0.093 per GB for reduced redundancy), it will add up if you start getting into terabytes of data. Also remember that you are charged for requests as well, but at $0.01 per 1,000 PUT, COPY, POST and LIST requests, and $0.01 per 10,000 GET requests, it does take a while for those to add up.

Joe ‘Zonker’ Brockmeier is a freelance writer and editor with more than 10 years covering IT. Formerly the openSUSE Community Manager for Novell, Brockmeier has written for Linux Magazine, Sys Admin, Linux Pro Magazine, IBM developerWorks, Linux.com, CIO.com, Linux Weekly News, ZDNet, and many other publications. You can reach Zonker at jzb@zonker.net and follow him on Twitter.

Follow ServerWatch on Twitter