Advanced Logging Techniques With Apache

By Martin Brown (Send Email)
Posted Jul 2, 2004


Logs in Apache are more configurable than most people realize. Not only can you organize the fields in your logs, but you can also create formats and layouts. Access logs can be split and divided up to make them easier to process by reporting specific items or ignoring those items that have no relevance.

Logging in Apache

Contents
Logging in Apache
Creating Custom Log Formats
Conditional Logging
Subdividing the Logs
Logging Directly to a Database or Application
Speeding up the Logging Process
Striking a balance between the amount of information needed and the effect on performance to achieve it is a quandary all IT departments face. We separate the critical from the optional and explain how to configure your Apache server to provide this data.

When was the last time you seriously thought about the content of your Web logs — Not processing the logs and producing statistics, but the actual raw content and information the logs store?

Many people will never choose a format beyond the standard common log format most systems use. They will generate logs containing all the requests, errors, and other information. This can be wasteful on resources — both in terms of Apache reporting the information and the disk space required to store it. There are, however, ways in which you can simplify and customize the logging system to suit your needs.

Creating Custom Log Formats

Both versions of Apache allow you to create custom log formats with only the fields you want using the format and layout that you desire.

The custom log format system enables the clear definition of exactly which fields of information to include in the log and the surrounding format. This means you can use spaces, tabs, or other delimiters to specify the format of each line written to the file.

Individual fields within the log format are specified using a special format string. For example, %a is replaced in the actual log with the IP address of the remote host. For a full list of the fields available, see the Apache mod_log_config documentation.

To specify a format, use the LogFormat directive. For example, the Common Log Format would be set using the directive

LogFormat "%h %l %u %t \"%r\" %>s %b" common

The actual format of the information (i.e., the information surrounding the fields you include) is entirely up to you. For example, the following log format generates a log using an XML structure. It is formatted for clarity — the actual definition would have to be within a single line, but the embedded \n character will generate a new line when the log is written.

<programlisting><![CDATA[LogFormat "<logitem><host>%h</host>\n
<ident>%l</ident>\n
<user>%u</user>\n
<datetime>%t</datetime>\n
<url>%r</url>\n
<statuscode>%>s</statuscode>\n
<bytes>%b</bytes>\n</logitem>" commonxml

Formats can be given a name (as they are in both of the examples above), and we can use it when specifying a log for a host or virtual host to request the desired log format:

CustomLog logs/access_log common

It's possible to create a variety of log formats and for the individual logs to contain different information, so we can spread access, errors, scripts, referrer, and other information across a range of different files. We can also be selective about what information appears in each file by using the conditional logging system.

>> Conditional Logging

Page 1 of 4


Comment and Contribute

Your name/nickname

Your email

(Maximum characters: 1200). You have characters left.