syslog Overview

Last modified 08-Jul-2003
Comments to tbird

On this page:
Background
syslog Architecture
Quick Start
Troubleshooting
For More Information

Background

The goal of this document is to describe the use of UNIX syslog as a logging and auditing mechanism for operating systems and applications; to provide information on customizing your syslog configuration for local or remote log collection; and to document some possible syslog client configuration, for monitoring the health and well-being of a UNIX computer.We'll also establish naming conventions and sample network topologies for the other syslog configuration documents.

syslog Architecture

syslog is a consolidated audit mechanism for UNIX operating system and application messages. It’s designed to make it easier for application developers and system analysts to incorporate logging into their projects, and to provide system administrators with a single point of management for collecting, distributing and processing audit data. In most cases, a lot of different people influence the data that eventually ends up in a system log. The developer who wrote the operating system component or application starts the ball rolling, by deciding

The local system or application administrator gets to decide what actions to apply to the log data, based (in the default version of syslog) on the system component or application that generated the log entry, and the significance assigned to the log entry by its developers. It's a pretty coarse-grained system, which is why there are a lot of alternatives for collecting and acting on syslog data (see the Log Analysis web site's listing of syslog replacements). The administrator can choose amongst the following options:

The architecture is designed to maximize flexibility and ease-of-use. But there are few, if any, guidelines available for developers or system administrators, which means that implementing an enterprise logging system involves a lot of trial and error.

The behavior of syslog -- how data from different applications and subsystems on the host operating system, and of different levels of severity, is stored or distributed -- is controlled on a per machine basis by the file /etc/syslog.conf. This configuration file consists of multiple lines like this:

mail.info <Tab><Tab> /var/log/maillog

The format of configuration lines is:

facility.level <Tab><Tab> action

For historical reasons, the <Tab> key, not a simple blank space, is used to define white space between the selector on the left side of the line and the action on the right side. Throughout the Log Analysis configuration documents, we've used the <Tab> to remind you of this -- but of course, when you look at the file, you'll only see white space.

The facility is the application or operating system component that generates a log message. The level is the severity or significance of the message that’s been generated. The action defines what’s done with any newly-arrived message that matches the facility and level. This combination of facility and level, referred to as the selector, allows system administrators to customize message handling, based on which parts of the system are generating data, and how critical the data is.

The standard UNIX syslog facilities are

kern – kernel
user – application or user processes (this is the default if the application sending a message does not specify the facility)
mail/news/UUCP/cron – electronic mail/NNTP/UUCP/cron subsystems
daemon – system daemons
auth – authentication and authorization related commands
lpr – line printer spooling subsystem
mark – inserts timestamp into log data at regular intervals
local0-local7 – 8 facilities for customized auditing
syslog – internal messages generated by syslog itself
authpriv – non-system authorization messages
* -- on most versions of UNIX, refers to all facilities except mark

syslog levels are nominally defined as:

emerg – system is or will be unusable if situation is not resolved
alert – immediate action required
crit – critical situations
warning – recoverable errors
notice – unusual situation that merits investigation; a significant event that is typically part of normal day-to-day operation
info – informational messages
debug – verbose data for debugging

Unfortunately, there aren’t any clearly defined standards dictating how levels are determined for a particular application or computer subsystem. All too often, the best way to figure out how any given application determines its levels is to set it to log everything. Then read what gets logged as you perform different actions (logging in, changing configurations, etc).

syslog can perform these actions:

filename – write message to the specified file on the local machine
@hostname or @ipaddress – forward message to remote loghost
user1,user2,… -- write message to consoles of users named in list, if user is logged-in
* -- write message to all logged-in users

When a selector is defined in /etc/syslog.conf, it causes all messages from the specified facility and the specified level or higher to be subject to the defined action. So setting the following line:

kern.crit <Tab><Tab> root,@loghost

will cause messages matching kern.crit, kern.alert and kern.emerg to be written to the remote loghost and to the root user’s console (if root is logged in).

As a sample of how facilities, levels and actions can be used to customize a syslog installation, here’s a sample /etc/syslog.conf from RedHat Linux v7.1. The default configuration has been modified to write all messages of crit and higher level to the root user console, the remote loghost, and a local file:

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages


# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* /var/log/maillog


# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages, plus log
# them on another machine.
*.emerg *

# Save mail and news errors of level err and higher in a
# special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

# send messages level crit and higher to root console
# and loghost

*.crit /var/log/messages,@loghost,root

Remember that those empty spaces between the selector and the action are tabs, not spaces.

To verify that your syslog configuration is correct, use the UNIX utility logger to write arbitrary messages. The -p flag allows you to set the facility and level. Typing the following command:

# /usr/bin/logger –p local0.warning “This space intentionally left blank”

will cause the message This space intentionally left blank to be sent to the local system's syslog.

Quick Start

When you're getting started with centralized logging and monitoring in your network, you may choose either to look at the data being produced by the default logging configuration on your devices, or to configure things to produce lots and lots of log data so you can learn about your network's idiosyncracies in gory detail. Starting off with one or two computers in "log everything" mode gives you the chance to get comfortable with reading log files, so that's what we show here. After the operating system logging configuration is done, the next step is to configure your applications – like Apache Web server, FTP servers, or whatever else is running on the system that doesn’t log to syslog by default – to send data to the local syslog. There are links to configuration documents for a variety of operating systems and applications in the For More Information section below.

The following instructions use the word loghost to refer to the IP address or the valid DNS hostname of the central syslog server. Please substitute the appropriate address or name for your site. The <Tab> entries in the following lines are a reminder to use the <Tab> character to delimit white space between the selector and the facility, not blank spaces. White spaces in the following lines are added to improve the legibility of the information, and should not be included in your working syslog configuration file.

To send all syslog data from a Solaris host to the loghost, add the following line to /etc/syslog.conf:

*.debug <Tab><Tab> @loghost

and then restart syslog:

/etc/init.d/syslog stop ; /etc/init.d/syslog start

To send all syslog data from an OpenBSD host to the loghost, add the following line to /etc/syslog.conf:

*.* <Tab><Tab> @loghost

and then restart syslog:

kill –SIGHUP `cat /var/run/syslog.pid`

To send all syslog data from a Linux host to the loghost, add the following line to /etc/syslog.conf:

*.* <Tab><Tab> @loghost

and then restart syslog:

/etc/rc.d/init.d/syslog restart

On Nokia IPSO, the proprietary FreeBSD-based operating system used by Nokia FW-1 and RealSecure appliances, syslog configuration is controlled through the IPSO Voyager interface. Bring up Voyager, and select the Config radio button. Scroll through the list of options until you get to the System Configuration option; under System Configuration, select System Logging. Add the loghost's IP address as the new remote IP address to log to. Apply and then save your changes.

To integrate Windows NT and Windows 2000 into your syslog infrastructure, please see the separate Log Analysis link covering these operating systems.

Troubleshooting syslog

First, the syslog daemon must be running on the local computer, and the loghost must be on the network and ready to accept data. The local machine must have network access to the loghost. The local /etc/syslog.conf must contain the appropriate facilities and levels. If required, applications must also be configured to log to syslog. Here are some suggestions for checking each of these requirements.

1) Is syslog running on the local machine? If not, the most likely culprit is a stray whitespace in the configuration file. Verify that /etc/syslog.conf uses <Tab> and not whitespaces to separate selectors from actions. (The syslog daemon dies a lonely and depressing death without those tabs.)
2) Is the loghost on the network? Is it accepting data from other remote systems?
3) Does the local machine have network connectivity to the loghost? Verify network connectivity using ping or your own favorite network troubleshooting tools. Network sniffers (such as snoop or tcpdump, e.g. tcpdump -nli eth0 udp port 514) are invaluable here. If you can’t establish connectivity, are the machines on separate subnets on your network that require routing changes? Are there access control lists or firewall rules restricting traffic between the devices?
4) Is /etc/syslog.conf on the local machine configured to send the appropriate data to the loghost? Test this by sending the “This space intentionally left blank” message to the loghost, using the logger utility described above.
5) Are your applications configured to send data to syslog? This can be tricky to test – there’s no one mechanism for generating a log message that’s guaranteed to work for all applications – but you can try, for instance, stopping and starting a server, changing a configuration, or using the application, to generate log data. If the data is not appearing in syslog, check your vendor documentation to see if there’s support for syslog built-in. If the application does not support syslog by default, you can usually can use logger to get data into syslog. If the application produces binary logs, we need to find some way to decode them before invoking logger.

For More Information

For more information on syslog, including a lot more sample configurations, see