Softpanorama
May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Syslog Configuration and /etc/syslog.conf

News  Remote Syslog Recommended Links Syslog Messages Classification Loghost server and remote syslog Pipes in syslog
Configuration examples Logwatch Syslog Multitail Syslog configuration debugging logger utility Rsyslog Configuration
Log rotation AIX syslog Syslog for Windows Troubleshooting SFU 3.5 syslog Syslog Internals
syslog spoofing Syslog-ng  Horror Stories Tips Humor  Etc

One of the most neglected area of Unix is handling system events. Daily check for unusual system messages is crucial for the security and health conditions of a computer system. Syslog daemon was created as "afterthought" and is actually a result of Sendmail development, not Unix kernel development. Initial implementation was way too primitive and inflexible.

Some architectural decisions made in syslog are suboptimal. Still attempt to improve syslog such as syslog-ng and rsyslog proved to be only half-successful. The inertia of already established, critical,  even if highly deficient subsystem proved to be tremendous.

Typical Unix system logs contain much "noise" -- messages which have no importance -- and on the contrary few important events, which should not be lost in the volume of messages. With typical regular expression based analyzers like Logwatch (or its, less primitive, but similarly architecturally deficient commercial counterparts like  Splunk) it's difficult to select which messages we are interested in as generally you are limited only to past events "patterns". Here anomaly detection analyzers can probably be more useful. 

In classic syslog daemon a message can be sent to multiple  destinations based on the assigned facility/priority pair.

One important feature of SYSLOG is the ability to aggregate messages on a special server. Special BSD syslog protocol defined in RFC 3164  is used. It uses target UDP port 514.  RFC recommends that source port also be set to 514. A good summary of RFC can be found in Introduction to Syslog Protocol - MonitorWare

syslog uses the user datagram protocol (UDP) [1] as its underlying transport layer mechanism.  The UDP port that has been assigned to syslog is 514.

It is RECOMMENDED that the source port also be 514 to indicate that the message is from the syslog process of the sender,  but there have been cases seen where valid syslog messages have come  from a sender with a source port other than 514.  If the sender uses  a source port other than 514 then it is RECOMMENDED and has been  considered to be good form that subsequent messages are from a single
consistent port.

It its pure form syslogd provides an outdated, insecure (unless it is used with the central logging host, as it actually should be used in any modern enterprise environment) and rather primitive logging mechanism that lacks the flexibility of dynamically extending message classification scheme. 

A better scheme was proposed in 2004 syslog-1 but tremendous amount of inertia prevents any meaningful changes. Two recent implementations of syslog daemon (rsyslog and syslog-ng) managed somehow to lessen the severity of those shortcomings:

 Standard Solaris syslog is still "classic System V syslogd" and consists of the following components:

Linux uses all three implementations:

Classic /etc/syslog.conf  file

The key file that is influencing syslog behavior is /etc/syslog.conf  file. Traditionally it contains two columns called the selection and action.

NOTE: In classic syslogd  the syntax of /etc/syslog.conf is pretty restrictive. It does not permit using spaces as a separator, tabs should be used as a separator between two columns of syslog.conf.

Now let's discuss those two columns is some details. The selector field is a semicolon-separated list of priority specifications in the following format: facility.level; facility.level.

After making any changes to syslog.conf file, you need to ask the daemon to reread the configuration file with kill -HUP command, for example pkill -HUP syslogd. This is an operation that is often forgotten. It might make sense to implement "system configuration" attribute that can automatically send executes a command after closing of the file with such attribute if it was opened for writing (Unix has "command execution string" for scripts forever, for example #!/usr/bin/perl, so it can be used for configuration files). In the absence of such facility that would be a real paradise for absent minded people like me you probably will be better off creating a special script, like visyslog  that contains just two command: vi and pkill to ensure that you do not forget this operation; I often do and then face consequences)

The default Solaris syslog configuration (/etc/syslog.conf) is far from being optimal (any selector in /etc/syslog.conf means "this level and higher", for example mail.crit includes mail.emerg):

*.err;kern.notice;auth.notice                 /dev/sysmsg
*.err;kern.debug;daemon.notice;mail.crit      /var/adm/messages

*.alert;kern.err;daemon.err                   operator
*.alert                                       root
*.emerg                                       *

# if a non-loghost machine chooses to have authentication messages
# sent to the loghost machine, un-comment out the following line:
#auth.notice ifdef(`LOGHOST', /var/log/authlog, @loghost)

mail.debug ifdef(`LOGHOST', /var/log/syslog, @loghost)

#
# non-loghost machines will use the following lines to cause "user"
# log messages to be logged locally.
#
ifdef(`LOGHOST', ,
user.err                                     /dev/sysmsg
user.err                                     /var/adm/messages
user.alert                                   root, operator
user.emerg                                     *
)

For one thing, in Solaris AUTH messages by default don’t get logged to any logfiles. This is important if you want to know when people are trying to break into your system so such messages should be emailed at least to operator (may be operator and root) and written to /var/adm/authlog

*.emerg                          *
*.kernel.notice;*.alert          root, operator
*.err;kern.notice;auth.notice    /dev/sysmsg
*.notice                         /var/adm/messages 
auth.notice                      /var/adm/authlog, /var/log/messages

Each line of the file contains two parts:

!

You must use the tab character between the selector and the action field. If you use a space, it will look the same, but syslog will not read configuration file correctly.

Message selectors have two parts: a facility and a priority. kern.debug, for example, selects all debug messages (the priority) generated by the kernel (the facility). It also selects all priorities that are greater than debug. An asterisk in place of either the facility or the priority indicates "all." (That is, *.debug means all debug messages, while kern.* means all messages generated by the kernel.) You can also use commas to specify multiple facilities. Two or more selectors can be grouped together by using a semicolon. (See the earlier examples.)

The action field specifies one of five actions (some versions of syslog support additional actions, such as logging to a proprietary error management system):

  1. Log to a file or a device In this case, the action field consists of a filename (or device name), which must start with a forward slash (e.g., /var/adm/lpd-errs or /dev/console). Beware: logging to /dev/console creates the possibility of a denial of service attack. If you are logging to the console, an attacker can flood your console with log messages, rendering it unusable. If your system supports virtual consoles, as with Linux, you can usually safely log to one of the virtual consoles, and leave the others uncluttered.
     
  2. Send a message to a user. In this case, the action field consists of a username (e.g., root). You can specify multiple usernames by separating them with commas (e.g., root,nosmis). The message is written to each terminal where these users are shown to be logged in, according to the utmp file.
     
  3. Send a message to all users. In this case, the action field consists of an asterisk (*).
     
  4. Pipe the message to a program. In this case, the program is specified after the Unix pipe symbol (|). Note that the current Solaris version syslog does not support logging to programs. You need to use syslog-ng to achieve that.
     
  5. Send the message to the syslog on another host. In this case, the action field consists of a hostname preceded by an at sign (e.g., @LOGHOST).

With the following explanation, understanding the typical syslog.conf configuration file shown earlier becomes easy:

*.err;kern.debug;auth.notice /dev/console


This line causes all error messages, all kernel debug messages, and all notice messages generated by the authorization system to be printed on the system console. If your system console is a printing terminal, this process will generate a permanent hardcopy that you can file and use for later reference. (Note that kern.debug means all messages of priority debug and above.)

daemon,auth.notice /var/log/messages

This line causes all notice messages from either the system daemons or the authorization system to be appended to the file /var/log/messages. Note that this is the second line that mentions auth.notice messages. As a result, auth.notice messages will be sent to both the console and the messages file.

lpr.* /var/log/lpd-errs 

This line causes all messages from the line printer system to be appended to the /var/log/lpd-errs file.

auth.* root,nosmis 

This line causes all messages from the authorization system to be sent to the users root and nosmis. Note, however, that if the users are not logged in, the messages will be lost.

auth.* @LOGHOST

This line causes all authorization messages to be sent to the syslog daemon on the computer defined in /etc/hosts file as LOGHOST.

*.emerg *

This line causes all emergency messages to be displayed on every user's terminal.

*.alert | logpipe

This line causes all alert messages to be sent to a program called logpipe, which might forward them to some kind of monitoring system for display.

mark.* /dev/console

This line causes the time to be printed on the system console every 20 minutes. This is useful if you have other information being printed on the console, and you want a running clock on the printout.

Creating you own custom log files

Syslog is a daemon that mutated out of a sendmail debugging aid into a logfile-catchall for unix. A lot of applications send their log output to syslog, but they have to send it to syslog, otherwise syslog won't know about the stuff that is to be logged. To keep logs apart, during the evolution of syslog, facilities (nothing more than "categories" in syslog-speak) and severities got introduced. The actual filtering of what gets output where can be defined in syslogs /etc/syslog.conf file.

You can select messages from the daemon that interests you and send it to a separate file, That save time and effort of grepping messages file all the time

Syslog generally can receive messages in three ways:

The basic syntax of /etc/syslog.conf  is easy

  1. Empty lines and everything behind a hash mark (#) is ignored
  2. Rules are of the format
 <What>    <Goes Where>

Your basic "what" is a specification of a facility and a severity delimited by a period:

  <facility>.<severity>

This will catch all messages belonging to the given facility that have the given severity and higher.

If you only want to catch messages belonging to exactly the given severity, prefix the priority with an equation sign (=):

 <facility>.=<severity>

You can also negate the severity selection by prepending an exclamation sign (!):

 <facility>.!<severity>

This will select all messages belonging to the given facility and that have a severity lower than the one specified. Note that this also weeds out messages belonging to the given severity - which is logical, since the opposite of >= is <.

Of course this can make things tedious if you have to list all combinations of the 20 facilities and 9 severities by hand. So there are shortcuts, such as specifying an asterisk (*) as a catchall:

  <facility>.*  -> All messages belonging to <facility> 
  *.<severity>  -> All messages of the given <severity> 
  *.*                 -> All messages

And then, you can specify lists of "whats", where the "whats" are delimited by semicola (;):

 <facility>.<severity>;<facility>.<severity>

Or, if you want to process the same severities of different facilities, list the facilities using commas (,) first:

 <facility>,<facility>.<severity>

To make matters interesting, there is also a special severity called "none", which implies that no message of the given facility are to be logged with this rule:

  *.*;<facility>.none  -> Log all messages except those of the given facility

After the "What" part with all it's twists and turns, the "Where" is actually pretty simple:

 </path/to/logfile>

will log everything to the given logfile.

This logging is done with synchronous writes, which means that after each log entry, syslog waits for the operating system kernel to acknowledge that the data has indeed been written to the disk before writing its next entry. This can slow down your system 10-fold for services with extensive logging (especially mail servers!). This factor has been verified in the wild, so only if you can afford to write logs asynchronously, do so.

To indicate to syslog that you want log entries to be written asynchronously, prepend a minus (-) to the logfile:

 -</path/to/logfile>

This is basically what is needed in 99% of everyday life.

Note that you can specify the same "What" multiple times pointing to different "wheres" for each. The messages will then be logged to all "wheres" given.

Ok, the "Where" part isn't actually all that simple. You have a couple of other choices:

But again, these are things you don't need that often, and if you do, you'd better read up on them in the manpage first!

For more information see


Top updates

Bulletin Latest Past week Past month
Google Search


NEWS CONTENTS

Old News ;-)

[Nov 25, 2011] Linux syslog may be on way out By Brian Proffitt

It is naive to link binary format with better security. Instantly the hacking tools will be created to "sanitize" binary logs in sophisticated way. So instead of improvement of security we will get just the next level of intruders vs. defenders war. Bob Simpson in his comment below made an important point" the author of the proposal think locally rather the globally. And the key about logs security is thinking globally. But still it is true that the current syslog implementation which is coming from Sendmail  is outdated and can be improved. But is can be improved without abandoning text format and throwing the child with the dirty bathwater. Remote syslog is the only real security enhancing mechanism for syslog but "chain-signing" records with some certificate can also be useful on local level and complicates tampering.  The efforts spend on syslog probably be extended to shell history. Shell history also can benefit from some "anti-tampering" mechanisms. One easy thing is to make it "unshrinkable" via  special filesystem attributes mechanisms.
November 22, 2011 | ITworld

In an effort to foil crackers attempts to cover their tracks by altering text-based syslogs, as well as improve the syslog process as a whole, two Red Hat developers are proposing a new binary-based tool called The Journal that could replace the syslog daemon in as early as the Fedora 17 release.

And believe you me, some people are less than enthused by the proposed solution.

Developers Lennart Poettering and Kay Sievers are proposing that the current 30-year-old syslog system is inefficient and too easy to misread and hack to properly perform even its most basic function: store a log of system events on a given Linux box.

This is largely due to the free-form nature of the syslog, which basically accepts text strings in whatever form the application or daemon on the Linux system chooses to send. So, one daemon may send information about an event in one way, and another daemon in a completely different way, leaving it up to the human reader to parse the information in a useful manner. Automated log analyzer tools can help with this, but in a detailed description of The Journal, Poettering and Sievers wrote:

"The data logged is very free-form. Automated log-analyzers need to parse human language strings to a) identify message types, and b) parse parameters from them. This results in regex horrors, and a steady need to play catch-up with upstream developers who might tweak the human language log strings in new versions of their software. Effectively, in a away, in order not to break user-applied regular expressions all log messages become ABI of the software generating them, which is usually not intended by the developer."

That's just one of 14 points the two developers have highlighted as problems with the current syslog system. Others include:

And so on. Poettering and Sievers highlighted one very topical problem with the syslog system to drive their points about a needed change to syslog home:

"For example, the recent, much discussed kernel.org intrusion involved log file manipulation which was only detected by chance."

With these factors in mind, Sievers and Poettering have come up with The Journal daemon, which will store data from system events in binary--not text--form as a list of key-value pairs that includes hashing for additional security.

This is not the first time these two developers have proposed such sweeping changes to the Linux system infrastructure. Poettering is the developer who invented the systemd daemon that replaced the System V init daemon on Linux, as well as invented the PulseAudio sound server. Sievers was most recently one of the Fedora Project team members who proposed to move all executable files into the /usr/bin directory and their libraries into /usr/lib or /usr/lib64, as needed.

With this binary implementation, The Journal daemon can enable the addition of metadata to each system event, such as the process ID and name of the sender, user and group IDs, and other key system data.

"Inspired by udev events, journal entries resemble environment blocks. A number of key/value fields, separated by line breaks, with uppercase variable names. In comparison to udev device events and real environment blocks there's one major difference: while the focus is definitely on ASCII formatted strings, binary blobs as values are also supported--something which may be used to attach binary meta data such as ATA SMART health data, SCSI sense data, coredumps or firmware dumps. The code generating a journal entry can attach as many fields to an entry as he likes, which can be well-known ones, or service/subsystem/driver specific ones."

If all of this seems a bit familiar to developers, see if this rings a bell: a lot of the effort here by Poettering and Sievers was inspired by the key/value, hash, and metadata provided to developers who use the git version control system.

Not only will implementing The Journal make a Linux system more secure (as unauthorized log entries or unexpected data field entries will immediately be flagged by the journal daemon), its inventors hope to actually reduce the footprint of the logging system on Linux by unifying all log systems on a Linux machine and efficiently restructuring the data.

"It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that."

But not everyone is thrilled with the proposal. Poettering and Sievers anticipated that many developers and system admins would be unhappy with The Journal's use of UUIDs to identify messages--as evidenced by their tongue-in-cheek attention to the issue in the FAQ section of their proposal document.

But many of the objections voiced on Linux Weekly News, where the proposal was first highlighted, lament the replacement of a simple text-based system with a binary data format that will rely on one tool--The Journal--which in turn will only be available with the systemd daemon.

Several commenters picked up on this entry in The Journal proposal FAQ:

"Will the journal file format be standardized? Where can I find an explanation of the on-disk data structures?

"At this point we have no intention to standardize the format and we take the liberty to alter it as we see fit. We might document the on-disk format eventually, but at this point we don't want any other software to read, write or manipulate our journal files directly. The access is granted by a shared library and a command line tool. (But then again, it's Free Software, so you can always read the source code!)"

That entry, more than any other in the proposal document, generated a lot of controversy, as many LWN commenters objected to the idea of using a non-standard format for The Journal's data. Backwards compatibility was also a big point of concern.

"It's a shame that we will lose the simplicity of the plain-text syslog format. But syslogs are usually compressed using gzip anyway. So essentially for me, all this means is that I use <magic-lennart-tool> instead of gzcat as the first part of my shell command, wrote commenter C. McCabe. "The big issue that I see is that a lot of system administrators will treat this as magic security dust, and not realize that they need to periodically save those hashes to a remote (and secure!) system in order to get any security benefit.

"I also hope Lennart and co. realize the absolute necessity of backwards compatibility for the on-disk format," McCabe added. "It would really embitter a lot of system administrators if their old logs became unreadable after upgrading to the shiniest new version. But assuming this is managed well, I don't see any reason why this couldn't be a good idea."

How this plays out in the broader Linux community will be interesting, to be sure. I personally find it notable that Fedora (and its commercial parent Red Hat) now seems to be the project where many internal infrastructure changes to the Linux operating system are getting implemented, even as distros like Ubuntu focus on the interface and user space.

This is not a judgmental statement, but rather an observation. Linux is clearly undergoing some significant evolutionary changes and shedding some of its UNIX legacy. What remains to be seen is how these changes will affect Linux as it moves ahead. 

Selected comments

alevesely_tw273908198

I'm not sure why the design proposed would be definitely better than upgrading syslog() so as to cover recent standardizations such as

* RFC 5424: The Syslog Protocol * RFC 5674: Alarms in Syslog * RFC 5675: Mapping Simple Network Management Protocol (SNMP) Notifications to SYSLOG Messages * RFC 5676: Definitions of Managed Objects for Mapping SYSLOG Messages to Simple Network Management Protocol (SNMP) Notifications * RFC 5848: Signed Syslog Messages

Proposer of an enhancement should take the trouble to standardize it, beside coding, experimenting, and evangelizing it.

BobSimpson

Is this a joke? Or is it someone just trying to push their ideology of what they think should be done to the rest of the world to make their idea a standard?

Doing something like this would be a sure way for Linux to shoot itself in the foot. For evidence, one only needs to look as far as Microsoft who insists on doing it their special way and expecting everyone else to do what they deem as "good". The concept of syslog messages are that they are meant to be 'open' so disparate systems can read the data. How to you propose to integrate with large syslog reporting/analysis tools like LogZilla (http://www.logzilla.pro)?

The authors are correct that a format needs to be written so that parsing is easier. But how is their solution any "easier"? Instead, there is a much more effective solution available known as CEE (http://cee.mitre.org/) that proposes to include fields in the text.

> Syslog data is not authenticated. If you need that, then use TLS/certificates. when logging to a centralized host.

>Syslog is only one of many logging systems on a Linux machine. Surely you're aware of syslog-ng and rsyslog.

Access control to the syslogs is non-existent.

> To locally stored logs? Maybe (if you don't chown them to root?)

> But, if you are using syslog-ng or rsyslog and sending to a centralized host., then what is "local" to the system becomes irrelevant.

Disk usage limits are only applied at fixed intervals, leaving systems vulnerable to DDoS attacks. > Again, a moot point if admins are doing it correctly by centralizing with tools like syslog-ng, rsyslog and LogZilla.

>"For example, the recent, much discussed kernel.org intrusion involved log file manipulation which was only detected by chance." Oh, you mean they weren't managing their syslog properly so they got screwed and blamed their lack of management on the protocol itself. Ok, yeah, that makes sense.

They also noted in their paper that " In a later version we plan to extend the journal minimally to support live remote logging, in both PUSH and PULL modes always using a local journal as buffer for a store-and-forward logic" I can't understand how this would be an afterthought. They are clearly thinking "locally" rather than globally. Plus, if it is to eventually be able to send, what format will it use? Text? Ok, now they are back to their original complaint.

All of this really just makes me cringe. If RH/Fedora do this, there is no way for people that manage large system infrastructures to include those systems in their management. I am responsible for managing over 8,000 Cisco devices on top of several hundred linux systems. Am I supposed to log on to each linux server to get log information?

BlackSteven_YahHVZXTW

1. It's RedHat so if they actually go with it it will be forced to the vast majority of folks that work with Linux in a corporate environment.

2. It's RedHat so it doesn't need to be grep-able. They cater to current and former Windows users who call themselves "geeks" because they use an Operating System that is harder to use. (Linux doesn't have to be hard to use -- but don't tell that to RedHat.)

3. The guys behind this have been behind a number of really hideous ideas. One of them actually proposed dropping /usr. (Functionally dropping /usr by dropping /bin /sbin and /lib.) If I can't unmount /usr, then it doesn't _functionally_ exist.

4. It'll only be available with the "systemd" startup mechanism. This makes it totally pointless, as there's no way Debian derived distributions will be forcing folks to use that. It'll just be another piece of RedHat goofiness. Since prior RedHat goofiness kept me away from all RedHat derived distributions, this has little impact for me personally.

5. Many of the issues he has problems with are either generally useful -- human-readable logs readable with any tool -- or mostly an issue when not using syslog-ng and a log vault. Do you think your system was compromised? What does your log vault say?

6. If this system doesn't have a log vault facility it is only a matter of time before it is circumvented when root is compromised. When root is compromised anything touchable on the system is suspect. Their use of a shifting non-standard data format does nothing to make the data safer and it breaks in-place upgrades. What it means is that someone else will make a shared library that can read/write all versions of their logs and crackers will use that shared library. The fact that it is open-source and written in C means that having one library which links to multiple implementations of their "The Journal" is trivial. (Hello, C preprocessor!)

7. Remember that -- since this is RedHat -- they have a long history of not recommending in-place upgrades. If this breaks in-place upgrades or makes long-term archival of logs impossible there is no reason to think that it will stop them.

8. Portable software will need to support both the new system and continue to support SysLog. There's no way the BSDs would be migrating to this, even if the existing viable commercial Unix world decides to promptly oblige the whims of RedHat. More than that, basic syslog functionality can be easily faked on non-Unix-like environments without a syslog daemon and without breaking the log format. This means -- when using syslog -- the documentation for the product needs to mention the alternate location of the log but the actual documentation for log data is the same. This is not so with this new tool, where there's a different data format and different way to access the data.

Is it a good idea? Absolutely not. Will it stop them? They're RedHat -- they're too big to fail.

Here's another idea. What's the difference between forcing this additional data in to another tool which we will end up using through a text-based interface (so we can grep it) and actually proposing a standard for how to send log data to syslog which will support enhanced storage and retrieval on the backend? You could even offer a convenient C function to take the arguments supported by "The Journal" and pass the data in the correct format to syslog.

Brendan Edmonds

One of the big problems with implementing a binary base log file system is as mentioned already, only one tool can read it correctly. And you lose the security if someone finds out how the data is stored. It's the exact same thing "Problem" as a text based log file. The other problem, a lot has to change in order for programs to work with the new system, i'e dmesg needs to be changed.

Also I can not see how you can use the same methods as before to clean the logs, and to backup these logs. For example what happened if you wanted to view that log file in Windows, how would you do this. From a tool point of view you need the correct security tokens, etc.

You also have to remember routers use the syslogd protocol to send messages to a unix system. How will this be handled?

I don't like the move, it defeats the whole point of UNIX. Every thing in UNIX is a text file. Anyone can add to it and anyone can remove lines from it. It is up to the kernel to control who can do what and the program, who can do what. The point about the file being text only is mute as normally only root and write to the file and a binary file has the same "problem".

JoeGiles:

 "I don't like the move, it defeats the whole point of UNIX. Every thing in UNIX is a text file"

O'Rly?

When was the last time you took a look at wtmp or btmp or saXX files with vi? Not every file in Linux is a text file!

I think its a great idea as long as its executed properly!

NickG:

Personally, I love all the changes coming into Fedora. The filesystem is archaic, log files outright useless in their current implementation (kernel.org hack proves this), and systemd is chock full of advanced functionality. My only gripe is with the and the possibility of some logs becoming no longer backwards-compatible.

Other than that, if PulseAudio and systemd are any indication, we'll hopefully be seeing these changes filter into the other distros as well soon.

IFMAP_ORG_tw389617356:

A better approach would be to use an identity-centric open standard such as IF-MAP which is already used by many security vendors to integrate security information from multi-vendor environments.

See www.if-map.org

[Jun 5, 2008] Red Hat 5.1 now uses rsyslog

From release notes for Red Hat Enterprise Linux 5.2

rsyslog  is an enhanced multi-threaded syslogd  daemon that supports the following (among others):

rsyslog  is compatible with the stock syslogd, and can be used as a replacement in most cases. Its advanced features make it suitable for enterprise-class, encrypted syslog  relay chains; at the same time, its user-friendly interface is designed to make setup easy for novice users.

For more information about rsyslog, refer to http://www.rsyslog.com/.

[Dec 2, 2007] Perl newsyslog by Aaron Scarisbrick

freshmeat.net

newsyslog is a faithful Perl rewrite of the MIT newsyslog utility, with a number of features taken from the FreeBSD and NetBSD variants of newsyslog. It archives log files based on size, date or interval, and can optionally compress archives with gzip or bzip2. Complete documentation is available via "perldoc newsyslog.pl".

[Jun 28, 2007] rsyslog 1.14.0 (Development) by Rainer Gerhards

About: Rsyslog is an enhanced multi-threaded syslogd. Among other features, it offers support for reliable syslog over TCP and RFC 3195, writing to MySQL databases, fully configurable output formats (including great timestamps), the ability to filter on any part of the syslog message, and on-the-wire message compression. It is designed as a drop-in replacement for stock syslogd and thus is able to work with the same configuration file syntax. Of course, some enhanced features require changing the configuration file, but in general, this should be fairly easy.

Changes: Support for IPv6 has been added and some minor cleanups plus a fix for the Red Hat init script were applied. Currently, IPv6 is implemented for UDP only, TCP will follow shortly. IPv6 support should be considered experimental. It is not recommended that this release be used in production.

[Jan 5, 2007] An Overview of the syslog Protocol Overview of Syslog by  Anand Deveriya.

This chapter presents an overview of the syslog protocol and shows you how to deploy an end-to-end syslog system. You'll learn about the syslog architecture as well as the issues in deploying syslog servers in Linux and Windows OSs with a focus on their relevance in a Cisco environment.

Understanding system log files on a Solaris

Solaris systems use the /var directory to store logs and other local files so that the operating system can support other directories being mounted as read only, sometimes from file servers elsewhere on the network. The /var directory is thus often on a partition that is local to the system.

All of the log files described below can be found in subdirectories under /var. There may be other application-specific log files that you will also need to inspect. However, it is beyond the scope of this implementation to describe all of the log files that you might want to inspect for your specific Solaris installation.

Because log files often provide the only indication of an intrusion, intruders often attempt to erase any evidence of their activities by removing or modifying the log files. For this reason, it is very important that your log files be adequately protected to make it as difficult as possible for intruders to change or remove then. See the practice "Managing logging and other data collection mechanisms" for more information on this topic.

Newlogcheck

My central loghost machine uses a modified version of logcheck.sh that I wrote named (imaginatively) newlogcheck.sh. The modified version calls another script I wrote that sorts the output of the "logtail" by individual hosts into separate portions of the final report. The perl script attempts to avoid duplication of log messages by printing each log message only once, reporting how many times the event was reported.

This approach dramatically reduces the size of your logcheck reports, and sorting it by host makes it easy to read.

Check out a sample newlogcheck report

You need to configure your logcheck settings yourself, read the README that comes with logcheck from psionic.com, then the one included with my scripts. Copy my newlogcheck.sh and sort_logs.pl into your logcheck dir, and run newlogcheck.sh instead of logcheck.sh for reports.

If you're ready to go ahead, get the tarred/gzipped file here.

[Mar 25, 2003] Introduction to Syslog Protocol

MonitorWare

This paper is not an in depth paper about syslog. It simply gives you an overview and a broader picture about the Syslog Protocol and its architecture. If you are interested in in-depth details about Syslog, I would strongly suggest you to go through RFC: 3164.

What is Syslog?

Syslog is a protocol that allows a machine to send event notification messages across IP networks to event message collectors - also known as Syslog Servers or Syslog Daemons. In other words, a machine or a device can be configured in such a way that it generates a Syslog Message and forwards it to a specific Syslog Daemon (Server).

Syslog messages are based on the User Datagram Protocol (UDP) type of Internet Protocol (IP) communications. Syslog messages are received on UDP port 514. Syslog message text is generally no more than 1024 bytes in length. Since the UDP type of communication is connectionless, the sending or receiving host has no knowledge receipt for retransmission. If a UDP packet gets lost due to congestion on the network or due to resource unavailability, it will simply get lost - no one would know about it!!

What is Syslog Daemon?

A Syslog Daemon or Server is an entity that would listen to the Syslog messages that are sent to it. You cannot configure a Syslog Daemon to ask a specific device to send it Syslog Messages. If a specific device has no ability to generate Syslog Messages, then a Syslog Daemon cannot do anything about it. To make this thing clear, you can consider a Syslog Server or Syslog Daemon as a TV which can only display you the program that is currently running on a specific channel. You cannot ask another station to send a new program on that channel.

Some Background

Syslog Protocol was created for use by Unix Operating Systems. Using Syslog, a remote Unix host could, in effect, keep track of the general well being of any other Unix host. Any application can generate Syslog Compliant messages to send the information over the network. Since each process, application and operating system was written somewhat independently, there is little uniformity to the content of syslog messages. For this reason, no assumption is made upon the formatting or contents of the messages. The protocol is simply designed to transport these event messages. One of the fundamental design considerations of the syslog protocol was its simplicity. No stringent coordination is required between the transmitters and the receivers. Indeed, the transmission of syslog messages may be started on a device without a receiver being configured, or even actually physically present. Conversely, many devices will most likely be able to receive messages without explicit configuration or definitions. This simplicity has greatly aided the acceptance and deployment of syslog [1]

Format of a Syslog Packet

The full format of a Syslog message seen on the wire has three ditinct parts.

1. PRI

2. HEADER

3.MSG.

The total length of the packet cannot exceed 1,024 bytes, and there is no minimum length

1. PRI

The Priority part is a number that is enclosed in angle brackets. This represents both the Facility and Severity of the message. This number is an eight bit number. The first 3 least significant bits represent the Severity of the message (with 3 bits you can represent 8 different Severities) and the other 5 bits represent the Facility of the message. You can use the Facility and the Severity values to apply certain filters on the events in the Syslog Daemon. Note that Syslog Daemon cannot generate thse Priority and Facility values. They are generated by the applications on which the event is generated. Following are the codes for Severity and Facility. Please note that the codes written below are the recommended codes that the applicatoins should generate in the specified situations. You cannot, however, be 100 % sure if it really is the correct code sent by the application. For example: An application can generate a numerical code for severity as 0 (Emergency) when it should have generated 4 (Warning) instead. Syslog Daemon can not do anything about it!! It will simply receive the message as it is.

a) Severity Codes

The Severity code is the severity of the message that has been generated. Following are the codes for Severity:

Numerical Code Severity
0 Emergency: system is unusable
1 Alert: action must be taken immediately
2 Critical: critical conditions
3 Error: error conditions
4 Warning: warning conditions
5 Notice: normal but significant condition
6 Informational: informational messages
7 Debug: debug-level messages

b) Facility Codes

The facility is the application or operating system component that generates a log message.Following are the codes for facility:

Numerical Code Facility
0 kernel messages
1 user-level messages
2 mail system
3 system daemons
4 security/authorization messages
5 messages generated internally by syslogd
6 line printer subsystem
7 network news subsystem
8 UUCP subsystem
9 clock daemon
10 security/authorization messages
11 FTP daemon
12 NTP subsystem
13 log audit
14 log alert
15 clock daemon
16 local use 0
17 local use 1
18 local use 2
19 local use 3
20 local use 4
21 local use 5
22 local use 6
23 local use 7

1.1 Calculating Priority Value

The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. For example, a kernel message (Facility=0) with a Severity of Emergency (Severity=0) would have a Priority value of 0. Also, a "local use 4" message (Facility=20) with a Severity of Notice (Severity=5) would have a Priority value of 165. In the PRI part of a Syslog message, these values would be placed between the angle brackets as <0> and <165> respectively.

2. Header

The HEADER part contains the following things:

a) Timestamp -- The Time stamp is the date and time at which the message was generated. Be warned, that this timestamp is picked up from the system time and if the system time is not correct, you might get a packet with totally incorrect time stamp

b) Hostname or IP address of the device.

3. MSG

The MSG part will fill the remainder of the Syslog packet. This will usually contain some additional information of the process that generated the message, and then the text of the message. The MSG part has two fields:

a) TAG field

b) CONTENT field

The value in the TAG field will be the name of the program or process that generated the message. The CONTENT contains the details of the message.

Some Important Points

  1. As mentioned above, since Syslog protocol is UDP based, it is unreliable. It does not guarantee you the delivery of the messages. They may either be dropped through network congestion, or they may be maliciously intercepted and discarded.
  2. As mentioned above, since each process, application and operating system was written somewhat independently, there is little uniformity to the content of syslog messages. For this reason, no assumption is made upon the formatting or contents of the messages. The protocol is simply designed to transport these event messages.
  3. The receiver of a Syslog packet will not be able to ascertain that the message was indeed sent from the reported sender.
  4. One possible problem associated with the above mentioned point is of Authentication. A misconfigured machine may send syslog messages to a Syslog Daemon representing itself as another machine. The administrative staff may become confused because the status of the supposed sender of the messages may not be accurately reflected in the received messages.
  5. Another problem associated with point 2 is that an attacker may start sending fake messages indicating a problem on some machine. This may get the attention of the system administrators who will spend their time investigating the alleged problem. During this time, the attacker may be able to compromise a different machine, or a different process on the same machine. 
  6. The Syslog protocol do not ensure ordered delivery of packets.
  7. An attacker may record a set of messages that indicate normal activity of a machine. At a later time, that attacker may remove that machine from the network and replay the syslog messages to the Daemon.

Related Software

The MonitorWare line of products [2] can be used as Syslog Daemons for Windows Operating System to collect Syslog Messages from various devices (including Routers, Fire walls etc). They can also act as relaying servers and can forward the data from one Syslog Daemon to another.

References

[1] http://www.ietf.org/rfc/rfc3164.txt

[Jul 17, 2002] Sampling syslog software

Last week we discussed syslog, a system for handling status messages and logging and we looked briefly at the format of a syslog message. There's a lot more to the standard, and we encourage you to read the relevant IETF standard, RFC 3164, "The BSD syslog Protocol".

The syslog protocol is very useful, but be warned it has its deficiencies: It isn't secure; syslog messages are relatively easy to fake (sending syslog messages greater than the standard maximum of 1,024 bytes has been used in an exploit to hack syslog) and there's no sender validation. Anyway, we will forgo delving any deeper into the bowels of syslog and instead look at syslog products.

Recommended Links

Softpanorama Top Visited

Softpanorama Recommended

RFCs

RFC 3164 (rfc3164) - The BSD Syslog Protocol

rfc3195 Reliable Delivery for syslog

Supporting documents and discussions:

Troubleshooting

See also Syslog configuration debugging

Summary - configuring syslog.conf


   To all of the people who responded to my questions, many thanks..
     (There were just too many responses to thank everyone individually).
     
     Overall, the suggestions were similar.
     
     Don't use spaces, use tabs when configuring syslog.conf.
     After making changes, kill -HUP pid for syslog.conf.
     
     The message below is from Kai O'Yang who was one of may who forwarded 
     their syslog.conf files to share.
     
     I am now receiving auth.notice messages from a remote system to my 
     loghost(on both the console and authlog file.
     
     The only real problem I have that I haven't been able to resolve with 
     this is that the name of the remote host is not showing up.  Instead, 
     I am receiving "???" in its place, and garbage on the device name:
     
     Oct 23 14:44:32 ??? su:'su root' succeeded for mconroy on /dev/pts/3^m
     
     I am sure it is configured correctly in dns.  So I am at a lost.
     
     Any thought???
     
     Thanks again for everyone's help.
     
     Mark Conroy
     
     
     First add a loghost alias in /etc/hosts or nis table for the syslog 
     host. Here's my syslog.conf for all client machines.
     
     #ident  "@(#)syslog.conf        1.3     93/12/09 SMI"   /* SunOS 5.0 
     */ #
     # Copyright (c) 1991-1993, by Sun Microsystems, Inc. #
     # syslog configuration file.
     #
     # This file is processed by m4 so be careful to quote (`') names # 
     that match m4 reserved words.  Also, within ifdef's, arguments # 
     containing commas must be quoted.
     #
     # Note: Have to exclude user from most lines so that user.alert #      
      and user.emerg are not included, because old sendmails #       will 
     generate them for debugging information.  If you
     #       have no 4.2BSD based systems doing network logging, you #      
      can remove all the special cases for "user" logging.
     #
     *.err;kern.notice;auth.notice;user.none         /dev/console 
     *.err;kern.debug;daemon.notice;mail.crit;user.none      @loghost
     
     *.alert;kern.err;daemon.err;user.none           operator,@loghost 
     *.alert;user.none                               root,@loghost
     
     *.emerg;user.none                               @loghost auth.info     
                       @loghost
     mail.info                       @loghost daemon.info                   
       @loghost
     
     For the loghost:
     #ident  "@(#)syslog.conf        1.3     93/12/09 SMI"   /* SunOS 5.0 
     */ #
     # Copyright (c) 1991-1993, by Sun Microsystems, Inc. #
     # syslog configuration file.
     #
     # This file is processed by m4 so be careful to quote (`') names # 
     that match m4 reserved words.  Also, within ifdef's, arguments # 
     containing commas must be quoted.
     #
     # Note: Have to exclude user from most lines so that user.alert #      
      and user.emerg are not included, because old sendmails #       will 
     generate them for debugging information.  If you
     #       have no 4.2BSD based systems doing network logging, you #      
      can remove all the special cases for "user" logging.
     #
     *.err;kern.notice;auth.notice;user.none         /dev/console 
     *.err;kern.debug;daemon.notice;mail.crit;user.none      
     /var/adm/messages
     
     *.alert;kern.err;daemon.err;user.none           operator 
     *.alert;user.none                               root
     
     *.emerg;user.none                               * auth.info            
              /var/log/authlog mail.info                       
     /var/log/mlog
     #
     # Adding log to daemon information
     #
     daemon.info     /var/log/syslog



Etc

Society

Groupthink : Understanding Micromanagers and Control Freaks : Toxic Managers : BureaucraciesHarvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Two Party System as Polyarchy : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

Skeptical Finance : John Kenneth Galbraith : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Oscar Wilde : Talleyrand : Somerset Maugham : War and Peace : Marcus Aurelius : Eric Hoffer : Kurt Vonnegut : Otto Von Bismarck : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Oscar Wilde : Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 26, No.1 (January, 2013) Object-Oriented Cult : Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks: The efficient markets hypothesis : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

 

The Last but not Least


Copyright © 1996-2014 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine. This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting hosting of this site with different providers to distribute and speed up access. Currently there are two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.

Disclaimer:

The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: April 09, 2014