Log Analysis, Looking Beyond Web Servers

Log analysis tools for programs other than Web server logs aren't widely used. Perhaps because people don't know similar reports can be created for firewalls, proxy, mail, and FTP servers. FTP and mail servers secrete similar information to Web servers – IP addresses, files accessed, and actions taken. This article discusses the sort of information that can be gained by going beyond Web server logs. Our log analysis series wraps up with a look at how to best analyze logs from proxy, mail, and FTP servers.

When dealing with mail servers, one might like to know how many e-mail messages were delivered, or how many e-mails were rejected because they were flagged as spam, virus-laden, or addressed to nonexistent users. One of the first surprises

See all articles in our Log Analysis Series

often encountered when looking at mail logs, is the number of rejected e-mails sent to nonexistent users. This is normally a result of spammers trying common names, hoping to get lucky. Such dictionary attacks can be fun to graph, especially when placed alongside statistics about how many viruses and spam messages have been blocked.

Unfortunately, not many tools do for mail logs what they do for Web logs. Even fewer produce the nice Web pages we witnessed with Web logs. There are, however, countless tools that generate a text summary of information. These can be just as useful, albeit not as pleasing to the upper management when trying to justify expensive antivirus software. Qmail, Postfix, and Sendmail are all compatible with third-party programs that make short work of log files, usually displaying really pertinent summaries. One program, Awstats, can also parse mail logs, creating neat little summaries of data transferred, sender addresses, recipient addresses, messages processed, and error codes.

Qmail, Postfix, and Sendmail are all compatible with third-party programs that make short work of log files, usually displaying really pertinent summaries.

For some, FTP logs are as important as Web server logs. Most log-parsing tools work on the xferlog format produced by wu-ftpd and others. Awstats, Analog, and Webalizer can report who connected to the FTP server and accessed which files, as well as the important summaries of how much data was really transferred. The xferlog many FTP servers produce is the most common log format, but these tools can all be configured to work with most arbitrary log formats. The configuration requires the admin specify how the server logs to the file, so the parsing program can make sense of the data. This is oftentimes fairly critical, so focusing on log analysis tools that work with specific servers out of the box makes everyone's life easier.

Squid, a popular caching Web proxy, provides some very interesting information in the log files. Everything from the sites users are accessing to how much Web traffic is being used can be found there. The most popular tool, Calamaris, was shut down due to European software patents, but it can still be obtained in some corners of the Internet. Calamaris generated text or HTML reports including traffic, Web sites visited, and TCP statistics. Squidalyzer and Webalizer both display similar information, but squidalyzer is more focused on singling out users. When admins need to gather details about a specific user's Web browsing habits on the job, squidalyzer is an ideal go-to program.

Intrusion Detection System (IDS) programs, like snort, produce vast amounts of data. Even expensive commercial IDS programs and devices can produce massive amounts of data, which becomes useless until someone takes the time to make sense of it all. Quite a few analysis tools can parse the logs and generate text-based summaries for snort and popular firewall programs.

The essence of logs dictates the importance of examining them. Many valuable pearls of information can be gleaned from logs. When you suddenly realize your Internet connection is clogged, it is past the time to start thinking about implementing a usable system for log viewing.

Snort's logs can be quite daunting, assuming the tool is configured correctly. Log parsing tools can quickly help you figure out why a particular "problem" is showing up too much. The quick and easy way to view snort logs is with an HTML page, complete with pie charts. Snortalog is the most wide-used and highly regarded snort app. Snortalog makes large amounts of snort data much more manageable, and even provides links to some reports about specific attacks that it detected.

The essence of logs dictates the importance of examining them. Many valuable pearls of information can be gleaned from logs. When you suddenly realize your Internet connection is clogged, it is past the time to start thinking about implementing a usable system for log viewing. Web, firewall, FTP, and proxy logs can all (very clearly) show an administrator what is happening. Plagued with unexplainable data transfers? No problem, logs can easily identify which server is hosting unauthorized content and clogging the network by checking Web and FTP logs. In the more common cases, when a computer is infected and running an unauthorized Web server, we may have been able to notice that the machine was attacked by regularly looking at snort log reports.

It all comes down to taking time to watch the logs, and, unfortunately, this can consume the greater part of an administrator's day. With fancy reporting tools, singling out the problems becomes much easier, and frees up time to actually investigate and fix them.