The Linux System Logging Daemon

I am regularly confronted by Windows NT users who are overwhelmed by how much
information they can collect and process using the Windows NT Event Viewer. It is so nice, they
maintain, that occurrences (events) are sorted by system, system security, and applications, and
that entries can be filtered and searched.

The problem is, that’s where it stops. With the exception of a few security-related events, what
you are able to log (or not log) is not configurable under Windows NT. You get whatever Microsoft
has decided is necessary, no more and no less.

With Linux, you can specify both what should be logged and where to log it. Windows NT, on the
other hand, always logs specific events to a specific file. Additionally, Windows NT differentiates
between only three different types of logs. This means you may need to wade through hundreds, if not
thousands, of entries looking for the right one.

Configuration from the GUI I find that editing files directly is a much easier and quicker way of configuring and monitoring syslogd than starting up a graphical utility and hunting through menus. However, for those of you who disagree with me, there is a solution in the form of Mark’s Administration Tool or MAT, which is available from http://sunsite.unc.edu. (Click on the LinSearch link and look for “MAT” or “syslog”). MAT is a graphical tool that allows you to configure and monitor a number of different aspects of your system, including users, e-mail, network services, and system logging. The screen shot below (right) shows you the configuration window for syslogd, which essentially shows you the contents of the /etc/syslog.conf file. Double-clicking on one of the entries brings you to a window like below (left). At the bottom are radio buttons for defining the actions to take. As you can see, you can define multiple facilities and priorities in the same way that you could by editing the syslog. conf file by hand. Clicking the “Apply” button makes the changes to the syslog.conf file and then restarts the daemon, thus activating the changes. Configure It: Double-clicking on an entry allows you to define actions (left). MAT’s configuration window (right).

syslogd – The Linux System Logger

The workhorse of the Linux logging system is the system logging daemon, or syslogd. This
daemon is normally started from the system start-up (rc) scripts when the system goes into
run level 1. (Run levels are numbers that specify your system’s current state. For details on the
various run levels, see Guru Guidance in September’s issue.) Once the daemon is running,
almost any part of the system, including applications, drivers, and other daemons, can make log
entries. Also, a command-line interface lets you make entries from shell scripts.

With Windows NT, each system maintains its own log files. There is no central location where
they are all stored. Although the Event Viewer can access event logs on other machines, this can
often take a great deal of time, especially when there are a lot of entries and you have a slow
connection.

syslogd can be configured to send all (or just some) of the messages to a remote machine,
which processes them and writes them to the necessary files. It is thus possible to store all of the
log messages of a particular type from all of the machines on your network in a single file on a
single machine, which makes accessing and administering the log messages much easier. syslogd
stores configuration information and log entries in text files, making it easy to write a script
that parses the entries and processes them in any way you see fit. Although it is possible for a
rogue program to write information in a nonstandard format, all system daemons and most programs
follow the standard, which is:

date time system facility message

System is the host name that generated the message. The facility is the
component of the system generating the message, which could be anything including the kernel itself,
system daemons, and even applications. Finally, there is the text of the message itself. Here are
two messages on the system jupiter. One is from syslogd and the other is from the
kernel:

Jun 5 09:20:52 jupiter syslogd 1.3-0: restart.

 Jun 5 09:20:55 jupiter kernel: VFS: Mounted root (ext2 filesystem) read only.

As you can see, even if you could not configure syslogd to separate the log entries into
different files, it would be fairly easy to separate them yourself using a script.

Configuring syslogd

The syslogd configuration file, syslog.conf, determines what can be done and when.
The file is usually in /etc. Syslogd. conf is a typical Linux configuration file with
one item (or rule) per line and comment lines beginning with a pound sign (#). Each rule
consists of the selector portion, which determines the events to which to react, and the action
portion, which determines what is to be done. The selector and action portions are separated by a
tab.

The selector portion is itself broken into two parts — facility and priority –
which are separated by a dot. The facility part says what aspect of the system is to be recorded,
and the priority says what level of messages to which to react. The selector has the general
syntax:

facility.priority

You can see a list of facilities in Table One and a list of the priorities in Table
Two.

For both facilities and priorities, there is a “wildcard,” represented by an asterisk
(*), that can be used to mean any facility or any priority. For example, *.emerg
means all emergency messages. mail.* means all messages coming from the mail facility.
Logically, *.* means all priorities of messages from all facilities.

The word none is used to refer to no priority for the specified facility. For example,
the selector mail.none would say not to perform the action for any mail event. At first,
this might not make sense to you. Why not simply leave off that facility? The answer lies in the
previous paragraph. Using the wildcard, you might want to log all messages with a priority of
info in a certain file. However, for obvious reasons, you might also want to log all of the
security messages (regardless of the priority) into another file.

Another possibility is to specify a subset of facilities, rather than all of them. This is done
by separating the facilities with a comma, with the priority following the last facility listed. For
example, to refer to information messages for mail, uucp, and news, the
selector entry would look like this:

mail,uucp,news.info

One thing I need to point out here is that when you specify a priority, you are actually
specifying everything at that priority or higher. Therefore, in this example, I am selecting all of
the priorities at info and higher.

You can perform three primary actions with these events. Probably the most common action is to
write them to a file. Because Linux (as well as other Unix dialects) treats devices as files, you
can also send the logging messages to a specific device, such as the system console
(/dev/console). It is a common practice to send emergency messages to the system console,
where someone will see the messages no matter what console terminal they are logged on. In other
cases, kernel messages are sent to one of the console terminals (e.g., /dev/tty7). You might
end with something like the following:

kernel.*        /dev/tty7

Writing to the hard disk takes time, which is why the system normally buffers a number of writes
before sending them all to disk. However, by default, syslogd will actually write the
information to the disk with each event. This ensures the entry actually makes it to the file if the
system should crash.

If overall system performance becomes an important factor with regard to logging, you can tell
syslogd not to sync the disk each time it writes to a log file. This is done by putting a
minus sign (-) in front of the file name, like this:

lpr.info    -/var/adm/printer.log

If you disable syncing the log file like this, one important thing to remember is that you stand
the chance of losing information. If the system goes down for some reason before the information is
written to the file, you may lose an important clue about why the system went down. One solution
would be to have a central log server where all of the information is sent and where you do not
disable syncing. That way, no matter what, you have a record of what happened.

Sending the log messages to another machine is done by using an at sign (@) in front of
the machine name as the action. For example:

*.emerg         @logserver

This sends all emergency messages to the machine logserver. I would suggest that you do
not create a logserver that is connected to the Internet. A malicious person might be able to
bring the system to a halt or at least affect its performance by flooding it with erroneous log
messages.

Another useful feature is the ability to send messages to named pipes. This is done by preceding
the name of the pipe by the pipe symbol (|). I find this a useful way
of sending log messages to other programs, where I can process them further. Named pipes are created
using the mkfifo(1) command and must already exist before syslogd starts up.

Another action is the ability to send messages to particular users, provided that they are
logged in at the moment. To do this for a single user you need simply to put the user-name as the
action. To send it to multiple users, separate the names by a comma. This might give you something
like the following:

*.emerg         root,jimmo

By using an asterisk in place of the list of usernames, you can send a message to everyone
logged in.

In some cases, you want multiple actions for a specific facility or priority. This is no
problem. You simply create multiple rules. One common example is broadcasting all of the emergency
messages to every user, as well as writing them to a log file and sending them to another server in
case the local machine crashes. This might be done like this:

*.emerg         /var/adm/messages

*.emerg         *
*.emerg         @logserver

Previously, I mentioned the ability to cause a single action based on the same kind of message
for multiple facilities. This is still an example of a single selector resulting in a specific
action. Taking this one step further, you might want multiple selectors all to result in a specific
action. Although it could be done with multiple rules, it is possible to have multiple selectors all
on the same line. This is done by separating the selectors with a semicolon (;).

*.emerg;kernel.critical        root,jimmo

This would notify the users root and jimmo of all emergency messages as well as
critical messages from the kernel facility.

The Linux syslogd has added a couple of functions that are not available in other
versions of Unix. By preceding a priority with an equal sign (=), you tell syslogd
only to react to that one priority. This is useful because syslogd normally reacts to
everything of that priority and higher. One place where this is useful is when you want all messages
of priority debug to be logged to a specific file, but everything else logged to another
file.

You can also explicitly exclude priorities by preceding them with an exclamation mark
(!). Note that this will exclude the priorities listed as well as anything higher. You can
combine the equal sign and exclamation mark to exclude a specific priority. If you do so, you need
to precede the equal sign with the exclamation mark, since what you are saying is not to include
anything that equals a particular priority.

All of these features can be combined in many different ways. For example, you can have multiple
selectors, which include as well as exclude specific priorities. For example:

*.warn;mail.=info;lpr.none;uucp.!crit       /dev/tty07

This would send warning messages from all priorities to the system console terminal
/dev/tty7, plus the mail log messages at only the info priority, no printer messages
at all, and all uucp messages except for the critical ones. Granted, this is a rather
contrived example, but it does show you how complex you can get.

Note that multiple selectors on a single line can cause some confusion when there are
conflicting components within a selector. The thing to keep in mind is that the last component takes
precedence. In the previous example, we specified warning messages for all facilities and then
overwrote portions of that for the mail, lpr, and uucp facilities.

Often, it is useful to log messages from scripts. This can done using the logger command
(usually found in /usr/ bin). Without any options, it takes the username as the facility and
notice as the priority. However, you can specify both a facility and priority from the
command line by using the -p option. For example:

logger -p kern.warning The kernel has been recompiled.

This would send the specified message to the same place other kernel messages are sent. For
details on the other options, see the logger(1)man page.

Managing Your Log Files

One common problem is figuring out what to do with all of the log messages. If you do a lot of
logging (particularly if everything is sent to a central server), you can fill up your filesystem
faster than you think. The most obvious and direct solution is to remove them after a specific
length of time or when they reach a particular size.

It is a fairly simple matter to write a shell script that is started from cron, and that
looks at the log files and takes specific actions. With Linux, this is not necessary. Linux provides
this functionality for you in the form of the logrotate command.

As its name implies, the goal of the logrotate program is to “rotate” log files. This
could be as simple as moving a log file to a different name and replacing the original with an empty
file. However, there is much more to it.

Two files define how logrotate behaves. The state file (located by default at
/var/state/logrotate and specified with the -s or –state option) basically
tells logrotate when the last actions were taken.

The configuration file tells logrotate when to rotate each of the respective files. If
necessary, you can have multiple configuration files that can all be specified on the same command
line, or you can include configuration files within another one.

The logrotate configuration file is broken into two parts.

At the beginning are the global configuration options, which apply to all of the log files.
Next, there are the configuration sections for each of the individual log files (the log file
definitions).

A very simple logrotate configuration file to rotate the /var/adm/messages might
look like this:

errors root@logserver

compress

/var/log/messages {

  rotate 4

  weekly

  postrotate

    /sbin/killall -HUP syslogd

  endscript

}

At the top are two global options, followed by a log file definition for /var/log/adm. In
this case, we could have included the global definitions within the log file definition. However,
there is normally more than one log file definition. It is also possible to specify an entire
directory rather than one log file. For example, you could rotate all of the Samba logs by
specifying the directory /var/log/ samba.d/*.

The first line says that all error messages are mailed to root at the logserver. The
second line says that log files are to be compressed after they are rotated.

The log file definition consists of the log file name and the directives to apply, which are
enclosed within curly brackets. The first line in the log file definition says to rotate the logs
four times before removing a file. The next line says to rotate the files once a week. Together,
these two lines mean that any given copy of the /var/log/messages file will be saved for four
weeks before it is removed.

The next three lines are actually a set of commands. The postrotate directive says that
what follows should be performed immediately after the log file has been rotated. In this case,
syslogd is sent an HUP signal to restart itself. There is also a prerotate
directive, which has the same basic functionality, but does everything before the log is
rotated.

In order to rotate logs based on their size, you would use the size= option. Setting
size=100K would rotate logs larger than 100 KB and size=100M would rotate logs
larger than 100 MB. There are an incredible number of additional options that you can use. Table
Three contains a list of options you can use with a brief explanation of each. For more details,
see the logrotate(1) man page.

James Mohr is the author of books on Linux, SCO Unix, Web site administration, and Unix-Windows
integration. He works in Coburg, Germany for a systems integrator, and can be reached at
[email protected].