Tip of the Trade: cURL

The "client for URLs," or cURL utility, is one of those deceptively simple, handy little programs. The more you dig into it, the more you learn about what it can do. cURL is a download and copying utility that recognizes a host of transfer protocols — HTTP, HTTPS, FTP, FTPS, TFTP, GopherR, Telnet, DICT, LDAP, and FILE. It also supports, and I quote from the cURL Web site:

"SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos ...), file transfer resume, proxy tunneling and a busload of other useful tricks."

Some folks compare cURL to wget, but they're really different beasts. wget is a mirroring tool and file downloader. cURL can do all manner of clever tricks in addition to merely downloading files. Its simplest invocation fetches Web pages, as this snippet shows:

$ curl http://www.serverwatch.com/ 

<b style="font-size:10pt;"></a
 href="http://cp.jupiterweb.com/index.php/3681_default">Winners announced June 5!
 </a></b></p>--> <P><span class=header>New on
 ServerWatch</span><p><a
 href="http://www.serverwatch.com/trends/article.php/3695711"><span
 class=topic>Greening Your Data Center — You May Have No
 Choice</span></a><br><span class=body>[August 22,
 2007]</span>

As you can see, cURL fetches the raw page code. This could be useful if you don't want to use the "View page source" button in a graphical Web browser. But cURL can perform even more useful tasks, such as measuring download performance on a Web site:

$ $ curl -w 'nPreXfer time:t%{time_pretransfer}nStartXfer
 time:t%{time_starttransfer}nnTotal time:t%{time_total}n' -o /dev/null -s
 http://serverwatch.com/

PreXfer time:   0.138
StartXfer time: 0.286

Total time:     0.286

This shows that the file transfer starts almost as soon as the user connects, which is a good thing. You can fetch server banners and see what different sites are running, or see what your own server is revealing to the world:

$ curl -I www.serverwatch.com

HTTP/1.1 200 OK
Date: Thu, 23 Aug 2007 20:12:03 GMT
Server: Apache
Content-Type: application/x-httpd-php

That's a good banner. It doesn't give away too much information. Visit cURL for downloads and documentation and to learn hundreds of nifty ways to use cURL.