Tip of the Trade: cURL

Download the authoritative guide: Data Center Guide: Optimizing Your Data Center Strategy

Download the authoritative guide: Cloud Computing: Using the Cloud for Competitive Advantage

Need a Definition?

The "client for URLs," or cURL utility, is a deceptively simple download and copying utility that recognizes a host of transfer protocols. Its capabilities expand with the user's familiarity.

The "client for URLs," or cURL utility, is one of those deceptively simple, handy little programs. The more you dig into it, the more you learn about what it can do. cURL is a download and copying utility that recognizes a host of transfer protocols — HTTP, HTTPS, FTP, FTPS, TFTP, GopherR, Telnet, DICT, LDAP, and FILE. It also supports, and I quote from the cURL Web site:

"SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos ...), file transfer resume, proxy tunneling and a busload of other useful tricks."

Some folks compare cURL to wget, but they're really different beasts. wget is a mirroring tool and file downloader. cURL can do all manner of clever tricks in addition to merely downloading files. Its simplest invocation fetches Web pages, as this snippet shows:

$ curl http://www.serverwatch.com/ <b style="font-size:10pt;"></a href="http://cp.jupiterweb.com/index.php/3681_default">Winners announced June 5! </a></b></p>--> <P><span class=header>New on ServerWatch</span><p><a href="http://www.serverwatch.com/trends/article.php/3695711"><span class=topic>Greening Your Data Center — You May Have No Choice</span></a><br><span class=body>[August 22, 2007]</span>

As you can see, cURL fetches the raw page code. This could be useful if you don't want to use the "View page source" button in a graphical Web browser. But cURL can perform even more useful tasks, such as measuring download performance on a Web site:

$ $ curl -w 'nPreXfer time:t%{time_pretransfer}nStartXfer time:t%{time_starttransfer}nnTotal time:t%{time_total}n' -o /dev/null -s http://serverwatch.com/ PreXfer time: 0.138 StartXfer time: 0.286 Total time: 0.286

This shows that the file transfer starts almost as soon as the user connects, which is a good thing. You can fetch server banners and see what different sites are running, or see what your own server is revealing to the world:

$ curl -I www.serverwatch.com HTTP/1.1 200 OK Date: Thu, 23 Aug 2007 20:12:03 GMT Server: Apache Content-Type: application/x-httpd-php

That's a good banner. It doesn't give away too much information. Visit cURL for downloads and documentation and to learn hundreds of nifty ways to use cURL.