O’Reilly Network: Web Client Programming in Python

“Web client programming is a powerful technique for querying the Web. A web client is any program that
retrieves data from a web server using the Hyper Text Transfer Protocol (the http in your URLs). A web
browser is a client; so are web crawlers, programs that traverse the Web automatically to gather information.
You can also use web clients to take advantage of services offered by others on the Web and add dynamic
features to your own web site.”

Web client programming belongs in any developer’s toolbox. Perl aficionados have employed it for years. In
Python, the process reaches even higher levels of convenience and flexibility. Three modules provide most of
the functionality you will need: HTTPLIB, URLLIB, and a newer addition, XMLRPCLIB. In true
Pythonesque fashion, each module builds upon its predecessor, providing a solid, well-designed base for your
applications. We will cover the first two modules in this article, saving XMLRPCLIB for a later time.

“Web client programming belongs in any developer’s toolbox. Perl aficionados have employed it for years. In
Python, the process reaches even higher levels of convenience and flexibility. Three modules provide most of
the functionality you will need: HTTPLIB, URLLIB, and a newer addition, XMLRPCLIB. In true
Pythonesque fashion, each module builds upon its predecessor, providing a solid, well-designed base for your
applications. We will cover the first two modules in this article, saving XMLRPCLIB for a later time.”

“For our examples, we will use Meerkat. If you are like me, you invest time tracking trends and developments
in the open source community to give you a competitive edge. Meerkat is a tool that makes that task much
easier. It is an open wire service that collects and collates an enormous amount of information on open
source computing. Although its browser interface is flexible and customizable, using web client
programming we can scan, extract, and even store this information off-line for later use. We will first access
Meerkat using HTTPLIB interactively, and then move on to accessing Meerkat’s Open API via URLLIB to
create a customizable information-collecting tool.”