What's All That Text Mean? Learn Linux Logging

You hear it all the time: Monitor your logs. When there is a problem, check the logs. And it's good advice, because system and application logs tell you anything you need to know, provided you actually look at them and understand what they are saying. Everyone tells you to check the logs when you think something's wrong with your Linux server, but few are forthcoming with what all that scrolling text means. Here's a helping hand.

See all articles in our Log Analysis Series

Viewing logfiles is merely a question of volition; understanding what they are telling you is a another kettle of clams entirely. Today we'll look at what you might find in your network activity logs, and what it all means. Then you'll know if you need to leap heroically into action, or if you can return to sipping your iced tea and catching up on your sadly-neglected napping.

The Mysterious -- MARK --
All newbie admins asks what this means in /var/log/messages:

Jul 1 16:04:53 windbag -- MARK --
Jul 1 16:24:53 windbag -- MARK --
Jul 1 16:44:53 windbag -- MARK --

That's just the syslog daemon letting you know that it is alive and well. You may set the interval to anything you like on Debian by editing /etc/init.d/sysklogd. This sets it to 60 minutes:

SYSLOGD="-m 60"

Then restart the syslog daemon:

# /etc/init.d/sysklogd restart

On Red Hat and derivatives, edit /etc/sysconfig/syslog :

SYSLOGD_OPTIONS="-m 60"

Restart syslogd with service syslog restart. Then watch it happen in realtime on any Linux with tail -f /var/log/messages. (You can monitor the changes in any text file with tail.)

Snort Logs
The Snort intrusion detection system is a fine piece of work, and if you're not using it to protect your network you should be. It is more than an intrusion detector; it is also an intrusion-preventer. Snort logs are uncommonly helpful, as this sample from /var/log/snort/alert shows:

06/26-12:22:18 [**] [1:2003:2] MS-SQL Worm propagation attempt [**] [Classification: Misc Attack] [Priority: 2] {UDP} X.X.162.5:1046 -> X.X.163.141:1434

This means the Slammer worm, released in January 2004, is still pounding the Internet, despite the wide, and widely-publicized, availability of a simple fix. Amusingly, or irritatingly, depending on your current mood, the vast majority of exploits you'll find in your logs are targeted at Microsoft products.

Also on System Logging Sawing Linux Logs with Simple Tools Scripting Clinic: Nagging Logs Make for Safe Networks Auditing on Windows 2000 Managing Windows 2000 Security Logs

The news and entertainment media like to present an image of dastardly skilled crackers (which they erroneously call "hackers", which is like calling a locksmith a burglar) toiling tirelessly day and night to penetrate your systems. If only it were so; the Internet would be a lot cleaner. Exploits are written by a few people, then propagate themselves upon the world in record time via huge networks of compromised Windows PCs, which are then used to spew mass quantities of spam, phishes, viruses, keyloggers, Trojan horses, spybots, backdoors, rootkits, and to connect to other botnets. It costs the crackers nothing to cast as wide a net as possible. It costs everyone else plenty, because even if your own systems are not compromised you still lose bandwidth. As the Honeynet Project discovered:

"...Windows XP and 2000 represent the most affected software versions." Why? Because freakin' Microsoft products are so easy to exploit: "These individuals demonstrate how even unskilled people can run and leverage a botnet." So much for Trusted Computing.

Getting back to reading Snort logs, this snippet tells you:

• The date the attack occurred
• The Snort signature ID
• A short text description of the alert
• The attack classification
• The alert priority- a lower number = higher priority
• The TCP/IP protocol used, the source IP and port, and the destination IP and port

The Snort signature ID is your shortcut to bales of useful information. In this example is it 1:2003. Take this number to the Snort database search page and look it up. You'll find out what the exploit does, what systems are vulnerable, what actions you need to take, and links to way more information.

Continued on page 2: A Glossary of Weird Log Entries