Use s3cmd for Managing Amazon S3 Storage from the CLI
Amazon S3 storage is dirt cheap. Whether it's for work or personal use, S3 makes for a great repository for backups and other data you want to store. But most admins want a way to interact with S3 that doesn't involve dealing with the Web-based interface. If you're looking to script backups or other operations, you need a reliable CLI tool that works with S3--and
s3cmd does the trick nicely.
- Navigating Your IT Career
- Exploring the Private Cloud for Your Organization
- IT Manager's Guide to Social Networking
You can find
s3cmd on the S3 Tools site. Many Linux distros (like Debian and Ubuntu) package it for you.
s3cmd, you must configure it with your S3 Amazon keys. Run
s3cmd--configure and provide the keys to
s3cmd. I probably don't need to say this, but just in case: You might not want to use
s3cmd on systems that you don't control. So if you have a shell account on a system that someone else has root on, storing your Amazon S3 credentials on the system is something you might want to avoid.
You'll also be prompted to provide an encryption password and asked if you want to use HTTPS. The default is no, but I enable HTTPS for my use.
Now what? Use
s3cmd ls to display all of your buckets. If you don't have any buckets, or you want to create a new one, you can create one using
s3cmd mb s3://bucketname. List the contents of a bucket using
s3 ls s3://bucketname.
If you want to upload or download individual files, use the
get commands. The
del command will (of course) delete a file.
Want to know how much you have stored in S3? Use the
s3cmd du -H command. The
-H is optional, but it provides the amounts in "human readable" form--so you see "3M" instead of "3293136" or "5G" instead of "5520830577."
What's really useful is the
s3 sync command. Just use the
s3cmd sync directory s3://bucketname. It works the other way too, of course, so you can easily sync directories stored on S3 back to local machines.
The only caveat I have for
s3cmd is that it becomes really easy to start stuffing files into S3 for backup purposes. Watch your Amazon bill. While it's dirt cheap to store files there (starting at $0.14 per GB for standard and $0.093 per GB for reduced redundancy), it will add up if you start getting into terabytes of data. Also remember that you are charged for requests as well, but at $0.01 per 1,000 PUT, COPY, POST and LIST requests, and $0.01 per 10,000 GET requests, it does take a while for those to add up.
Joe 'Zonker' Brockmeier is a freelance writer and editor with more than 10 years covering IT. Formerly the openSUSE Community Manager for Novell, Brockmeier has written for Linux Magazine, Sys Admin, Linux Pro Magazine, IBM developerWorks, Linux.com, CIO.com, Linux Weekly News, ZDNet, and many other publications. You can reach Zonker at firstname.lastname@example.org and follow him on Twitter.
Read more on "Open Source Software Spotlight" »