Log Analysis, Looking Beyond Web Servers
Log analysis tools for programs other than Web server logs aren't widely used. Perhaps because people don't know similar reports can be created for firewalls, proxy, mail, and FTP servers. FTP and mail servers secrete similar information to Web servers IP addresses, files accessed, and actions taken. This article discusses the sort of information that can be gained by going beyond Web server logs.Our log analysis series wraps up with a look at how to best analyze logs from proxy, mail, and FTP servers.
When dealing with mail servers, one might like to know how many e-mail messages were delivered, or how many e-mails were rejected because they were flagged as spam, virus-laden, or addressed to nonexistent users. One of the first surprises
Unfortunately, not many tools do for mail logs what they do for Web logs. Even fewer produce the nice Web pages we witnessed with Web logs. There are, however, countless tools that generate a text summary of information. These can be just as useful, albeit not as pleasing to the upper management when trying to justify expensive antivirus software. Qmail, Postfix, and Sendmail are all compatible with third-party programs that make short work of log files, usually displaying really pertinent summaries. One program, Awstats, can also parse mail logs, creating neat little summaries of data transferred, sender addresses, recipient addresses, messages processed, and error codes.
Qmail, Postfix, and Sendmail are all compatible with third-party programs that make short work of log files, usually displaying really pertinent summaries.
Squid, a popular caching Web proxy, provides some very interesting information in the log files. Everything from the sites users are accessing to how much Web traffic is being used can be found there. The most popular tool, Calamaris, was shut down due to European software patents, but it can still be obtained in some corners of the Internet. Calamaris generated text or HTML reports including traffic, Web sites visited, and TCP statistics. Squidalyzer and Webalizer both display similar information, but squidalyzer is more focused on singling out users. When admins need to gather details about a specific user's Web browsing habits on the job, squidalyzer is an ideal go-to program.
Intrusion Detection System (IDS) programs, like snort, produce vast amounts of data. Even expensive commercial IDS programs and devices can produce massive amounts of data, which becomes useless until someone takes the time to make sense of it all. Quite a few analysis tools can parse the logs and generate text-based summaries for snort and popular firewall programs.
The essence of logs dictates the importance of examining them. Many valuable pearls of information can be gleaned from logs. When you suddenly realize your Internet connection is clogged, it is past the time to start thinking about implementing a usable system for log viewing.
The essence of logs dictates the importance of examining them. Many valuable pearls of information can be gleaned from logs. When you suddenly realize your Internet connection is clogged, it is past the time to start thinking about implementing a usable system for log viewing. Web, firewall, FTP, and proxy logs can all (very clearly) show an administrator what is happening. Plagued with unexplainable data transfers? No problem, logs can easily identify which server is hosting unauthorized content and clogging the network by checking Web and FTP logs. In the more common cases, when a computer is infected and running an unauthorized Web server, we may have been able to notice that the machine was attacked by regularly looking at snort log reports.
It all comes down to taking time to watch the logs, and, unfortunately, this can consume the greater part of an administrator's day. With fancy reporting tools, singling out the problems becomes much easier, and frees up time to actually investigate and fix them.