May the source be with you,
but remember the KISS principle ;-)
Key Softpanorama Topics
|About||Contents||Top Updates||Top Visited|
|Bulletin||Selected Papers||Softpanorama Bookshelf||History|
|News||Remote Syslog||Recommended Links||Syslog Messages Classification||Loghost server and remote syslog||Pipes in syslog|
|Configuration examples||Logwatch||Syslog Multitail||Syslog configuration debugging||logger utility|
|Log rotation||AIX syslog||Syslog for Windows||Troubleshooting||SFU 3.5 syslog||Syslog Internals|
|syslog spoofing||Syslog-ng||Horror Stories||Tips||Humor||Etc|
One of the most neglected area of Unix is handling system events. Daily check for unusual system messages is crucial for the security and health conditions of a computer system. Syslog daemon was created as "afterthought" and is actually a result of Sendmail development, not Unix kernel development. Initial implementation was way too primitive and inflexible. Some architectural decisions made are suboptimal. Still attempt to improve syslog such as syslog-ng and rsyslog proved to be only half-successful. The inertia of already established, even if highly deficient but critical subsystem proved to be tremendous.
Typical Unix system logs contain much "noise" -- messages which have no importance -- and on the contrary few important events, which should not be lost in the volume of messages. With typical regular expression based analyzers like Logwatch (or its, less primitive, but still architecturally deficient commercial counterparts like Splunk) it's difficult to select which messages we are interested in as generally you are limited only to past events "patterns". Here anomaly detection analyzers can probably be more useful.
In classic syslog daemon a message can be sent to multiple destinations based on the assigned facility/priority pair.
There are 12+8 (12 real and 8 local) predefined facilities (kernel, mail, daemon, news, syslog, lpr, auth, uucp, cron, local1-8, mark etc.). See Syslog Messages Classification. Fixed facilities is a big shortcoming of standard syslog (a design blunder) . Some of them are no longer used (uucp). Fixed facilities is the major weakness of classic syslog
There are eight different priorities (EMERG, ALERT, CRIT, ERR, WARN, NOTICE, INFO, DEBUG). Fixed priorities have a strange order and some of them (ALERT) have badly chosen names. For example, in first three (EMERG, ALERT, CRIT) priorities, why on the earth ALERT has higher priority then CRIT messages ? What is funny that older IBM mainframe severity classification is much better (talk about the progress after that ;-). Still even in its present form it is an OK solution.
One important feature of SYSLOG is the ability to aggregate messages on a special server. Special BSD syslog protocol defined in RFC 3164 is used. It uses target UDP port 514. RFC recommends that source port also be set to 514. A good summary of RFC can be found in Introduction to Syslog Protocol - MonitorWare
syslog uses the user datagram protocol (UDP)  as its underlying transport layer mechanism. The UDP port that has been assigned to syslog is 514.
It is RECOMMENDED that the source port also be 514 to indicate that the message is from the syslog process of the sender, but there have been cases seen where valid syslog messages have come from a sender with a source port other than 514. If the sender uses a source port other than 514 then it is RECOMMENDED and has been considered to be good form that subsequent messages are from a single
It its pure form syslogd provides an outdated, insecure (unless it is used with the central logging host, as it actually should be used in any modern enterprise environment) and rather primitive logging mechanism that lacks the flexibility of dynamically extending message classification scheme.
A better scheme was proposed in 2004 syslog-1 but tremendous amount of inertia prevents any meaningful changes. Two recent implementations of syslog daemon (rsyslog and syslog-ng) managed somehow to lessen the severity of those shortcomings:
rsyslog which is now installed by default in Red Hat (a drop-in replacement for syslogd which is also available for Solaris and Suse).The structure of rsyslog files is similar as it is "drop-in" replacement for classic syslogd.
Standard Solaris syslog is still "classic System V syslogd" and consists of the following components:
syslogd the system daemon used to receive and route system log events from syslog() calls and logger commands
/etc/syslog.conf the configuration file used to control the logging and routing of system log events
logger a UNIX command used to add single-line entries to the system log. See also ogger
syslog() an application program interface (API) referenced by several standard system utilities and available to anyone writing software in the C programming language, C++, Java and many scripting languages (Perl is one example).
Linux uses all three implementations:
RHEL uses rsyslog since version 5.1. Versions before that used GNU clone of classic syslogd.
SLES 9-11 uses syslog-ng by default, but latest version (SLES 11 SP2) can use rsyslog instead (DVD provides RPM package and /etc/init.d/syslog script checks if it installed and if so uses it instead of syslog-ng. It looks like SLES is moving toward replacement of syslog-ng with rsyslog. Older versions use classic syslogd clone.
The key file that is influencing syslog behavior is /etc/syslog.conf file. Traditionally it contains two columns called the selection and action.
NOTE: In classic syslogd the syntax of /etc/syslog.conf is pretty restrictive. It does not permit using spaces as a separator, tabs should be used as a separator between two columns of syslog.conf.
Now let's discuss those two columns is some details. The selector field is a semicolon-separated list of priority specifications in the following format: facility.level; facility.level.
emerg or 0 Panic conditions that are normally broadcast to all users
alert or 1 Conditions that should be corrected immediately, such as a corrupted system database. Only sysadmin of a particular server needs to be informed by mail or paged.
crit or 2 Warnings about critical conditions, such as hard device errors.
err or 3 Errors other than hard device errors
warning or 4 Warning messages, that generally does not interfere with normal operation.
notice or 5 Non-error conditions that might require special handling
info or 6 Purely informational messages (usually does not require any handling)
debug or 7 Messages that are normally used only when debugging a program
none or 8 Messages are not sent from the indicated facility to the selected file
After making any changes to syslog.conf file, you need to ask the daemon to reread the configuration file with kill -HUP command, for example pkill -HUP syslogd. This is an operation that is often forgotten. It might make sense to implement "system configuration" attribute that can automatically send executes a command after closing of the file with such attribute if it was opened for writing (Unix has "command execution string" for scripts forever, for example #!/usr/bin/perl, so it can be used for configuration files). In the absence of such facility that would be a real paradise for absent minded people like me you probably will be better off creating a special script, like visyslog that contains just two command: vi and pkill to ensure that you do not forget this operation; I often do and then face consequences)
The default Solaris syslog configuration (/etc/syslog.conf) is far from being optimal (any selector in /etc/syslog.conf means "this level and higher", for example mail.crit includes mail.emerg):
*.err;kern.notice;auth.notice /dev/sysmsg *.err;kern.debug;daemon.notice;mail.crit /var/adm/messages *.alert;kern.err;daemon.err operator *.alert root *.emerg * # if a non-loghost machine chooses to have authentication messages # sent to the loghost machine, un-comment out the following line: #auth.notice ifdef(`LOGHOST', /var/log/authlog, @loghost) mail.debug ifdef(`LOGHOST', /var/log/syslog, @loghost) # # non-loghost machines will use the following lines to cause "user" # log messages to be logged locally. # ifdef(`LOGHOST', , user.err /dev/sysmsg user.err /var/adm/messages user.alert root, operator user.emerg * )
For one thing, in Solaris AUTH messages by default dont get logged to any logfiles. This is important if you want to know when people are trying to break into your system so such messages should be emailed at least to operator (may be operator and root) and written to /var/adm/authlog
*.emerg * *.kernel.notice;*.alert root, operator *.err;kern.notice;auth.notice /dev/sysmsg *.notice /var/adm/messages auth.notice /var/adm/authlog, /var/log/messages
Each line of the file contains two parts:
A selector that specifies which kinds of messages to log (e.g., all error messages or all debugging messages from the kernel).
An action field that says what should be done with the message (e.g., put it in a file or send the message to a user's terminal).
Message selectors have two parts: a facility and a priority. kern.debug, for example, selects all debug messages (the priority) generated by the kernel (the facility). It also selects all priorities that are greater than debug. An asterisk in place of either the facility or the priority indicates "all." (That is, *.debug means all debug messages, while kern.* means all messages generated by the kernel.) You can also use commas to specify multiple facilities. Two or more selectors can be grouped together by using a semicolon. (See the earlier examples.)
The action field specifies one of five actions (some versions of syslog support additional actions, such as logging to a proprietary error management system):
With the following explanation, understanding the typical syslog.conf configuration file shown earlier becomes easy:
- *.err;kern.debug;auth.notice /dev/console
This line causes all error messages, all kernel debug messages, and all notice messages generated by the authorization system to be printed on the system console. If your system console is a printing terminal, this process will generate a permanent hardcopy that you can file and use for later reference. (Note that kern.debug means all messages of priority debug and above.)
- daemon,auth.notice /var/log/messages
This line causes all notice messages from either the system daemons or the authorization system to be appended to the file /var/log/messages. Note that this is the second line that mentions auth.notice messages. As a result, auth.notice messages will be sent to both the console and the messages file.
- lpr.* /var/log/lpd-errs
This line causes all messages from the line printer system to be appended to the /var/log/lpd-errs file.
- auth.* root,nosmis
This line causes all messages from the authorization system to be sent to the users root and nosmis. Note, however, that if the users are not logged in, the messages will be lost.
- auth.* @LOGHOST
This line causes all authorization messages to be sent to the syslog daemon on the computer defined in /etc/hosts file as LOGHOST.
- *.emerg *
This line causes all emergency messages to be displayed on every user's terminal.
- *.alert | logpipe
This line causes all alert messages to be sent to a program called logpipe, which might forward them to some kind of monitoring system for display.
- mark.* /dev/console
This line causes the time to be printed on the system console every 20 minutes. This is useful if you have other information being printed on the console, and you want a running clock on the printout.
Syslog is a daemon that mutated out of a sendmail debugging aid into a logfile-catchall for unix. A lot of applications send their log output to syslog, but they have to send it to syslog, otherwise syslog won't know about the stuff that is to be logged. To keep logs apart, during the evolution of syslog, facilities (nothing more than "categories" in syslog-speak) and severities got introduced. The actual filtering of what gets output where can be defined in syslogs /etc/syslog.conf file.
You can select messages from the daemon that interests you and send it to a separate file, That save time and effort of grepping messages file all the time
Syslog generally can receive messages in three ways:
The basic syntax of /etc/syslog.conf is easy
<What> <Goes Where>
Your basic "what" is a specification of a facility and a severity delimited by a period:
This will catch all messages belonging to the given facility that have the given severity and higher.
If you only want to catch messages belonging to exactly the given severity, prefix the priority with an equation sign (=):
You can also negate the severity selection by prepending an exclamation sign (!):
This will select all messages belonging to the given facility and that have a severity lower than the one specified. Note that this also weeds out messages belonging to the given severity - which is logical, since the opposite of >= is <
Of course this can make things tedious if you have to list all combinations of the 20 facilities and 9 severities by hand. So there are shortcuts, such as specifying an asterisk (*) as a catchall:
<facility>.* -> All messages belonging to <facility> *.<severity> -> All messages of the given <severity> *.* -> All messages
And then, you can specify lists of "whats", where the "whats" are delimited by semicola (;):
Or, if you want to process the same severities of different facilities, list the facilities using commas (,) first:
To make matters interesting, there is also a special severity called "none", which implies that no message of the given facility are to be logged with this rule:
*.*;<facility>.none -> Log all messages except those of the given facility
After the "What" part with all it's twists and turns, the "Where" is actually pretty simple:
will log everything to the given logfile.
This logging is done with synchronous writes, which means that after each log entry, syslog waits for the operating system kernel to acknowledge that the data has indeed been written to the disk before writing its next entry. This can slow down your system 10-fold for services with extensive logging (especially mail servers!). This factor has been verified in the wild, so only if you can afford to write logs asynchronously, do so.
To indicate to syslog that you want log entries to be written asynchronously, prepend a minus (-) to the logfile:
This is basically what is needed in 99% of everyday life.
Note that you can specify the same "What" multiple times pointing to different "wheres" for each. The messages will then be logged to all "wheres" given.
Ok, the "Where" part isn't actually all that simple. You have a couple of other choices:
|<path to fifo>
But again, these are things you don't need that often, and if you do, you'd better read up on them in the manpage first!
|Bulletin||Latest||Past week||Past month||
November 22, 2011 | ITworld
In an effort to foil crackers attempts to cover their tracks by altering text-based syslogs, as well as improve the syslog process as a whole, two Red Hat developers are proposing a new binary-based tool called The Journal that could replace the syslog daemon in as early as the Fedora 17 release.
And believe you me, some people are less than enthused by the proposed solution.
Developers Lennart Poettering and Kay Sievers are proposing that the current 30-year-old syslog system is inefficient and too easy to misread and hack to properly perform even its most basic function: store a log of system events on a given Linux box.
This is largely due to the free-form nature of the syslog, which basically accepts text strings in whatever form the application or daemon on the Linux system chooses to send. So, one daemon may send information about an event in one way, and another daemon in a completely different way, leaving it up to the human reader to parse the information in a useful manner. Automated log analyzer tools can help with this, but in a detailed description of The Journal, Poettering and Sievers wrote:
"The data logged is very free-form. Automated log-analyzers need to parse human language strings to a) identify message types, and b) parse parameters from them. This results in regex horrors, and a steady need to play catch-up with upstream developers who might tweak the human language log strings in new versions of their software. Effectively, in a away, in order not to break user-applied regular expressions all log messages become ABI of the software generating them, which is usually not intended by the developer."
That's just one of 14 points the two developers have highlighted as problems with the current syslog system. Others include:
- Syslog data is not authenticated.
- Syslog is only one of many logging systems on a Linux machine.
- Access control to the syslogs is non-existent.
- Disk usage limits are only applied at fixed intervals, leaving systems vulnerable to DDoS attacks.
And so on. Poettering and Sievers highlighted one very topical problem with the syslog system to drive their points about a needed change to syslog home:
"For example, the recent, much discussed kernel.org intrusion involved log file manipulation which was only detected by chance."
With these factors in mind, Sievers and Poettering have come up with The Journal daemon, which will store data from system events in binary--not text--form as a list of key-value pairs that includes hashing for additional security.
This is not the first time these two developers have proposed such sweeping changes to the Linux system infrastructure. Poettering is the developer who invented the systemd daemon that replaced the System V init daemon on Linux, as well as invented the PulseAudio sound server. Sievers was most recently one of the Fedora Project team members who proposed to move all executable files into the /usr/bin directory and their libraries into /usr/lib or /usr/lib64, as needed.
With this binary implementation, The Journal daemon can enable the addition of metadata to each system event, such as the process ID and name of the sender, user and group IDs, and other key system data.
"Inspired by udev events, journal entries resemble environment blocks. A number of key/value fields, separated by line breaks, with uppercase variable names. In comparison to udev device events and real environment blocks there's one major difference: while the focus is definitely on ASCII formatted strings, binary blobs as values are also supported--something which may be used to attach binary meta data such as ATA SMART health data, SCSI sense data, coredumps or firmware dumps. The code generating a journal entry can attach as many fields to an entry as he likes, which can be well-known ones, or service/subsystem/driver specific ones."
If all of this seems a bit familiar to developers, see if this rings a bell: a lot of the effort here by Poettering and Sievers was inspired by the key/value, hash, and metadata provided to developers who use the git version control system.
Not only will implementing The Journal make a Linux system more secure (as unauthorized log entries or unexpected data field entries will immediately be flagged by the journal daemon), its inventors hope to actually reduce the footprint of the logging system on Linux by unifying all log systems on a Linux machine and efficiently restructuring the data.
"It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that."
But not everyone is thrilled with the proposal. Poettering and Sievers anticipated that many developers and system admins would be unhappy with The Journal's use of UUIDs to identify messages--as evidenced by their tongue-in-cheek attention to the issue in the FAQ section of their proposal document.
But many of the objections voiced on Linux Weekly News, where the proposal was first highlighted, lament the replacement of a simple text-based system with a binary data format that will rely on one tool--The Journal--which in turn will only be available with the systemd daemon.
Several commenters picked up on this entry in The Journal proposal FAQ:
"Will the journal file format be standardized? Where can I find an explanation of the on-disk data structures?
"At this point we have no intention to standardize the format and we take the liberty to alter it as we see fit. We might document the on-disk format eventually, but at this point we don't want any other software to read, write or manipulate our journal files directly. The access is granted by a shared library and a command line tool. (But then again, it's Free Software, so you can always read the source code!)"
That entry, more than any other in the proposal document, generated a lot of controversy, as many LWN commenters objected to the idea of using a non-standard format for The Journal's data. Backwards compatibility was also a big point of concern.
"It's a shame that we will lose the simplicity of the plain-text syslog format. But syslogs are usually compressed using gzip anyway. So essentially for me, all this means is that I use <magic-lennart-tool> instead of gzcat as the first part of my shell command, wrote commenter C. McCabe. "The big issue that I see is that a lot of system administrators will treat this as magic security dust, and not realize that they need to periodically save those hashes to a remote (and secure!) system in order to get any security benefit.
"I also hope Lennart and co. realize the absolute necessity of backwards compatibility for the on-disk format," McCabe added. "It would really embitter a lot of system administrators if their old logs became unreadable after upgrading to the shiniest new version. But assuming this is managed well, I don't see any reason why this couldn't be a good idea."
How this plays out in the broader Linux community will be interesting, to be sure. I personally find it notable that Fedora (and its commercial parent Red Hat) now seems to be the project where many internal infrastructure changes to the Linux operating system are getting implemented, even as distros like Ubuntu focus on the interface and user space.
This is not a judgmental statement, but rather an observation. Linux is clearly undergoing some significant evolutionary changes and shedding some of its UNIX legacy. What remains to be seen is how these changes will affect Linux as it moves ahead.
I'm not sure why the design proposed would be definitely better than upgrading syslog() so as to cover recent standardizations such as
* RFC 5424: The Syslog Protocol * RFC 5674: Alarms in Syslog * RFC 5675: Mapping Simple Network Management Protocol (SNMP) Notifications to SYSLOG Messages * RFC 5676: Definitions of Managed Objects for Mapping SYSLOG Messages to Simple Network Management Protocol (SNMP) Notifications * RFC 5848: Signed Syslog Messages
Proposer of an enhancement should take the trouble to standardize it, beside coding, experimenting, and evangelizing it.
Is this a joke? Or is it someone just trying to push their ideology of what they think should be done to the rest of the world to make their idea a standard?
Doing something like this would be a sure way for Linux to shoot itself in the foot. For evidence, one only needs to look as far as Microsoft who insists on doing it their special way and expecting everyone else to do what they deem as "good". The concept of syslog messages are that they are meant to be 'open' so disparate systems can read the data. How to you propose to integrate with large syslog reporting/analysis tools like LogZilla (http://www.logzilla.pro)?
The authors are correct that a format needs to be written so that parsing is easier. But how is their solution any "easier"? Instead, there is a much more effective solution available known as CEE (http://cee.mitre.org/) that proposes to include fields in the text.
> Syslog data is not authenticated. If you need that, then use TLS/certificates. when logging to a centralized host.
>Syslog is only one of many logging systems on a Linux machine. Surely you're aware of syslog-ng and rsyslog.
Access control to the syslogs is non-existent.
> To locally stored logs? Maybe (if you don't chown them to root?)
> But, if you are using syslog-ng or rsyslog and sending to a centralized host., then what is "local" to the system becomes irrelevant.
Disk usage limits are only applied at fixed intervals, leaving systems vulnerable to DDoS attacks. > Again, a moot point if admins are doing it correctly by centralizing with tools like syslog-ng, rsyslog and LogZilla.
>"For example, the recent, much discussed kernel.org intrusion involved log file manipulation which was only detected by chance." Oh, you mean they weren't managing their syslog properly so they got screwed and blamed their lack of management on the protocol itself. Ok, yeah, that makes sense.
They also noted in their paper that " In a later version we plan to extend the journal minimally to support live remote logging, in both PUSH and PULL modes always using a local journal as buffer for a store-and-forward logic" I can't understand how this would be an afterthought. They are clearly thinking "locally" rather than globally. Plus, if it is to eventually be able to send, what format will it use? Text? Ok, now they are back to their original complaint.
All of this really just makes me cringe. If RH/Fedora do this, there is no way for people that manage large system infrastructures to include those systems in their management. I am responsible for managing over 8,000 Cisco devices on top of several hundred linux systems. Am I supposed to log on to each linux server to get log information?
1. It's RedHat so if they actually go with it it will be forced to the vast majority of folks that work with Linux in a corporate environment.
2. It's RedHat so it doesn't need to be grep-able. They cater to current and former Windows users who call themselves "geeks" because they use an Operating System that is harder to use. (Linux doesn't have to be hard to use -- but don't tell that to RedHat.)
3. The guys behind this have been behind a number of really hideous ideas. One of them actually proposed dropping /usr. (Functionally dropping /usr by dropping /bin /sbin and /lib.) If I can't unmount /usr, then it doesn't _functionally_ exist.
4. It'll only be available with the "systemd" startup mechanism. This makes it totally pointless, as there's no way Debian derived distributions will be forcing folks to use that. It'll just be another piece of RedHat goofiness. Since prior RedHat goofiness kept me away from all RedHat derived distributions, this has little impact for me personally.
5. Many of the issues he has problems with are either generally useful -- human-readable logs readable with any tool -- or mostly an issue when not using syslog-ng and a log vault. Do you think your system was compromised? What does your log vault say?
6. If this system doesn't have a log vault facility it is only a matter of time before it is circumvented when root is compromised. When root is compromised anything touchable on the system is suspect. Their use of a shifting non-standard data format does nothing to make the data safer and it breaks in-place upgrades. What it means is that someone else will make a shared library that can read/write all versions of their logs and crackers will use that shared library. The fact that it is open-source and written in C means that having one library which links to multiple implementations of their "The Journal" is trivial. (Hello, C preprocessor!)
7. Remember that -- since this is RedHat -- they have a long history of not recommending in-place upgrades. If this breaks in-place upgrades or makes long-term archival of logs impossible there is no reason to think that it will stop them.
8. Portable software will need to support both the new system and continue to support SysLog. There's no way the BSDs would be migrating to this, even if the existing viable commercial Unix world decides to promptly oblige the whims of RedHat. More than that, basic syslog functionality can be easily faked on non-Unix-like environments without a syslog daemon and without breaking the log format. This means -- when using syslog -- the documentation for the product needs to mention the alternate location of the log but the actual documentation for log data is the same. This is not so with this new tool, where there's a different data format and different way to access the data.
Is it a good idea? Absolutely not. Will it stop them? They're RedHat -- they're too big to fail.
Here's another idea. What's the difference between forcing this additional data in to another tool which we will end up using through a text-based interface (so we can grep it) and actually proposing a standard for how to send log data to syslog which will support enhanced storage and retrieval on the backend? You could even offer a convenient C function to take the arguments supported by "The Journal" and pass the data in the correct format to syslog.
One of the big problems with implementing a binary base log file system is as mentioned already, only one tool can read it correctly. And you lose the security if someone finds out how the data is stored. It's the exact same thing "Problem" as a text based log file. The other problem, a lot has to change in order for programs to work with the new system, i'e dmesg needs to be changed.
Also I can not see how you can use the same methods as before to clean the logs, and to backup these logs. For example what happened if you wanted to view that log file in Windows, how would you do this. From a tool point of view you need the correct security tokens, etc.
You also have to remember routers use the syslogd protocol to send messages to a unix system. How will this be handled?
I don't like the move, it defeats the whole point of UNIX. Every thing in UNIX is a text file. Anyone can add to it and anyone can remove lines from it. It is up to the kernel to control who can do what and the program, who can do what. The point about the file being text only is mute as normally only root and write to the file and a binary file has the same "problem".
"I don't like the move, it defeats the whole point of UNIX. Every thing in UNIX is a text file"
When was the last time you took a look at wtmp or btmp or saXX files with vi? Not every file in Linux is a text file!
I think its a great idea as long as its executed properly!
Personally, I love all the changes coming into Fedora. The filesystem is archaic, log files outright useless in their current implementation (kernel.org hack proves this), and systemd is chock full of advanced functionality. My only gripe is with the and the possibility of some logs becoming no longer backwards-compatible.
Other than that, if PulseAudio and systemd are any indication, we'll hopefully be seeing these changes filter into the other distros as well soon.
A better approach would be to use an identity-centric open standard such as IF-MAP which is already used by many security vendors to integrate security information from multi-vendor environments.
- From release notes for Red Hat Enterprise Linux 5.2
rsyslog is an enhanced multi-threaded syslogd daemon that supports the following (among others):
- RFC 3195
- permitted sender lists
- filtering on any message part
- more granular output format control
rsyslog is compatible with the stock syslogd, and can be used as a replacement in most cases. Its advanced features make it suitable for enterprise-class, encrypted syslog relay chains; at the same time, its user-friendly interface is designed to make setup easy for novice users.
For more information about rsyslog, refer to http://www.rsyslog.com/.
newsyslog is a faithful Perl rewrite of the MIT newsyslog utility, with a number of features taken from the FreeBSD and NetBSD variants of newsyslog. It archives log files based on size, date or interval, and can optionally compress archives with gzip or bzip2. Complete documentation is available via "perldoc newsyslog.pl".
About: Rsyslog is an enhanced multi-threaded syslogd. Among other features, it offers support for reliable syslog over TCP and RFC 3195, writing to MySQL databases, fully configurable output formats (including great timestamps), the ability to filter on any part of the syslog message, and on-the-wire message compression. It is designed as a drop-in replacement for stock syslogd and thus is able to work with the same configuration file syntax. Of course, some enhanced features require changing the configuration file, but in general, this should be fairly easy.
Changes: Support for IPv6 has been added and some minor cleanups plus a fix for the Red Hat init script were applied. Currently, IPv6 is implemented for UDP only, TCP will follow shortly. IPv6 support should be considered experimental. It is not recommended that this release be used in production.
This chapter presents an overview of the syslog protocol and shows you how to deploy an end-to-end syslog system. You'll learn about the syslog architecture as well as the issues in deploying syslog servers in Linux and Windows OSs with a focus on their relevance in a Cisco environment.
Solaris systems use the /var directory to store logs and other local files so that the operating system can support other directories being mounted as read only, sometimes from file servers elsewhere on the network. The /var directory is thus often on a partition that is local to the system.
All of the log files described below can be found in subdirectories under /var. There may be other application-specific log files that you will also need to inspect. However, it is beyond the scope of this implementation to describe all of the log files that you might want to inspect for your specific Solaris installation.
Because log files often provide the only indication of an intrusion, intruders often attempt to erase any evidence of their activities by removing or modifying the log files. For this reason, it is very important that your log files be adequately protected to make it as difficult as possible for intruders to change or remove then. See the practice "Managing logging and other data collection mechanisms" for more information on this topic.
My central loghost machine uses a modified version of logcheck.sh that I wrote named (imaginatively) newlogcheck.sh. The modified version calls another script I wrote that sorts the output of the "logtail" by individual hosts into separate portions of the final report. The perl script attempts to avoid duplication of log messages by printing each log message only once, reporting how many times the event was reported.
This approach dramatically reduces the size of your logcheck reports, and sorting it by host makes it easy to read.
Check out a sample newlogcheck report
You need to configure your logcheck settings yourself, read the README that comes with logcheck from psionic.com, then the one included with my scripts. Copy my newlogcheck.sh and sort_logs.pl into your logcheck dir, and run newlogcheck.sh instead of logcheck.sh for reports.
If you're ready to go ahead, get the tarred/gzipped file here.
This paper is not an in depth paper about syslog. It simply gives you an overview and a broader picture about the Syslog Protocol and its architecture. If you are interested in in-depth details about Syslog, I would strongly suggest you to go through RFC: 3164.
What is Syslog?
Syslog is a protocol that allows a machine to send event notification messages across IP networks to event message collectors - also known as Syslog Servers or Syslog Daemons. In other words, a machine or a device can be configured in such a way that it generates a Syslog Message and forwards it to a specific Syslog Daemon (Server).
Syslog messages are based on the User Datagram Protocol (UDP) type of Internet Protocol (IP) communications. Syslog messages are received on UDP port 514. Syslog message text is generally no more than 1024 bytes in length. Since the UDP type of communication is connectionless, the sending or receiving host has no knowledge receipt for retransmission. If a UDP packet gets lost due to congestion on the network or due to resource unavailability, it will simply get lost - no one would know about it!!
What is Syslog Daemon?
A Syslog Daemon or Server is an entity that would listen to the Syslog messages that are sent to it. You cannot configure a Syslog Daemon to ask a specific device to send it Syslog Messages. If a specific device has no ability to generate Syslog Messages, then a Syslog Daemon cannot do anything about it. To make this thing clear, you can consider a Syslog Server or Syslog Daemon as a TV which can only display you the program that is currently running on a specific channel. You cannot ask another station to send a new program on that channel.
Syslog Protocol was created for use by Unix Operating Systems. Using Syslog, a remote Unix host could, in effect, keep track of the general well being of any other Unix host. Any application can generate Syslog Compliant messages to send the information over the network. Since each process, application and operating system was written somewhat independently, there is little uniformity to the content of syslog messages. For this reason, no assumption is made upon the formatting or contents of the messages. The protocol is simply designed to transport these event messages. One of the fundamental design considerations of the syslog protocol was its simplicity. No stringent coordination is required between the transmitters and the receivers. Indeed, the transmission of syslog messages may be started on a device without a receiver being configured, or even actually physically present. Conversely, many devices will most likely be able to receive messages without explicit configuration or definitions. This simplicity has greatly aided the acceptance and deployment of syslog 
Format of a Syslog Packet
The full format of a Syslog message seen on the wire has three ditinct parts.
The total length of the packet cannot exceed 1,024 bytes, and there is no minimum length
The Priority part is a number that is enclosed in angle brackets. This represents both the Facility and Severity of the message. This number is an eight bit number. The first 3 least significant bits represent the Severity of the message (with 3 bits you can represent 8 different Severities) and the other 5 bits represent the Facility of the message. You can use the Facility and the Severity values to apply certain filters on the events in the Syslog Daemon. Note that Syslog Daemon cannot generate thse Priority and Facility values. They are generated by the applications on which the event is generated. Following are the codes for Severity and Facility. Please note that the codes written below are the recommended codes that the applicatoins should generate in the specified situations. You cannot, however, be 100 % sure if it really is the correct code sent by the application. For example: An application can generate a numerical code for severity as 0 (Emergency) when it should have generated 4 (Warning) instead. Syslog Daemon can not do anything about it!! It will simply receive the message as it is.
a) Severity Codes
The Severity code is the severity of the message that has been generated. Following are the codes for Severity:
Numerical Code Severity 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages
b) Facility Codes
The facility is the application or operating system component that generates a log message.Following are the codes for facility:
Numerical Code Facility 0 kernel messages 1 user-level messages 2 mail system 3 system daemons 4 security/authorization messages 5 messages generated internally by syslogd 6 line printer subsystem 7 network news subsystem 8 UUCP subsystem 9 clock daemon 10 security/authorization messages 11 FTP daemon 12 NTP subsystem 13 log audit 14 log alert 15 clock daemon 16 local use 0 17 local use 1 18 local use 2 19 local use 3 20 local use 4 21 local use 5 22 local use 6 23 local use 7
1.1 Calculating Priority Value
The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. For example, a kernel message (Facility=0) with a Severity of Emergency (Severity=0) would have a Priority value of 0. Also, a "local use 4" message (Facility=20) with a Severity of Notice (Severity=5) would have a Priority value of 165. In the PRI part of a Syslog message, these values would be placed between the angle brackets as <0> and <165> respectively.
The HEADER part contains the following things:
a) Timestamp -- The Time stamp is the date and time at which the message was generated. Be warned, that this timestamp is picked up from the system time and if the system time is not correct, you might get a packet with totally incorrect time stamp
b) Hostname or IP address of the device.
The MSG part will fill the remainder of the Syslog packet. This will usually contain some additional information of the process that generated the message, and then the text of the message. The MSG part has two fields:
a) TAG field
b) CONTENT field
The value in the TAG field will be the name of the program or process that generated the message. The CONTENT contains the details of the message.
Some Important Points
- As mentioned above, since Syslog protocol is UDP based, it is unreliable. It does not guarantee you the delivery of the messages. They may either be dropped through network congestion, or they may be maliciously intercepted and discarded.
- As mentioned above, since each process, application and operating system was written somewhat independently, there is little uniformity to the content of syslog messages. For this reason, no assumption is made upon the formatting or contents of the messages. The protocol is simply designed to transport these event messages.
- The receiver of a Syslog packet will not be able to ascertain that the message was indeed sent from the reported sender.
- One possible problem associated with the above mentioned point is of Authentication. A misconfigured machine may send syslog messages to a Syslog Daemon representing itself as another machine. The administrative staff may become confused because the status of the supposed sender of the messages may not be accurately reflected in the received messages.
- Another problem associated with point 2 is that an attacker may start sending fake messages indicating a problem on some machine. This may get the attention of the system administrators who will spend their time investigating the alleged problem. During this time, the attacker may be able to compromise a different machine, or a different process on the same machine.
- The Syslog protocol do not ensure ordered delivery of packets.
- An attacker may record a set of messages that indicate normal activity of a machine. At a later time, that attacker may remove that machine from the network and replay the syslog messages to the Daemon.
The MonitorWare line of products  can be used as Syslog Daemons for Windows Operating System to collect Syslog Messages from various devices (including Routers, Fire walls etc). They can also act as relaying servers and can forward the data from one Syslog Daemon to another.
Last week we discussed syslog, a system for handling status messages and logging and we looked briefly at the format of a syslog message. There's a lot more to the standard, and we encourage you to read the relevant IETF standard, RFC 3164, "The BSD syslog Protocol".
The syslog protocol is very useful, but be warned it has its deficiencies: It isn't secure; syslog messages are relatively easy to fake (sending syslog messages greater than the standard maximum of 1,024 bytes has been used in an exploit to hack syslog) and there's no sender validation. Anyway, we will forgo delving any deeper into the bowels of syslog and instead look at syslog products.
This chapter presents an overview of the syslog protocol and shows you how to deploy an end-to-end syslog system. You'll learn about the syslog architecture as well as the issues in deploying syslog servers in Linux and Windows OSs with a focus on their relevance in a Cisco environment.
RFC 3164 (rfc3164) - The BSD Syslog Protocol
rfc3195 Reliable Delivery for syslog
Supporting documents and discussions:
See also Syslog configuration debugging
Summary - configuring syslog.conf
To all of the people who responded to my questions, many thanks.. (There were just too many responses to thank everyone individually). Overall, the suggestions were similar. Don't use spaces, use tabs when configuring syslog.conf. After making changes, kill -HUP pid for syslog.conf. The message below is from Kai O'Yang who was one of may who forwarded their syslog.conf files to share. I am now receiving auth.notice messages from a remote system to my loghost(on both the console and authlog file. The only real problem I have that I haven't been able to resolve with this is that the name of the remote host is not showing up. Instead, I am receiving "???" in its place, and garbage on the device name: Oct 23 14:44:32 ??? su:'su root' succeeded for mconroy on /dev/pts/3^m I am sure it is configured correctly in dns. So I am at a lost. Any thought??? Thanks again for everyone's help. Mark Conroy First add a loghost alias in /etc/hosts or nis table for the syslog host. Here's my syslog.conf for all client machines. #ident "@(#)syslog.conf 1.3 93/12/09 SMI" /* SunOS 5.0 */ # # Copyright (c) 1991-1993, by Sun Microsystems, Inc. # # syslog configuration file. # # This file is processed by m4 so be careful to quote (`') names # that match m4 reserved words. Also, within ifdef's, arguments # containing commas must be quoted. # # Note: Have to exclude user from most lines so that user.alert # and user.emerg are not included, because old sendmails # will generate them for debugging information. If you # have no 4.2BSD based systems doing network logging, you # can remove all the special cases for "user" logging. # *.err;kern.notice;auth.notice;user.none /dev/console *.err;kern.debug;daemon.notice;mail.crit;user.none @loghost *.alert;kern.err;daemon.err;user.none operator,@loghost *.alert;user.none root,@loghost *.emerg;user.none @loghost auth.info @loghost mail.info @loghost daemon.info @loghost For the loghost: #ident "@(#)syslog.conf 1.3 93/12/09 SMI" /* SunOS 5.0 */ # # Copyright (c) 1991-1993, by Sun Microsystems, Inc. # # syslog configuration file. # # This file is processed by m4 so be careful to quote (`') names # that match m4 reserved words. Also, within ifdef's, arguments # containing commas must be quoted. # # Note: Have to exclude user from most lines so that user.alert # and user.emerg are not included, because old sendmails # will generate them for debugging information. If you # have no 4.2BSD based systems doing network logging, you # can remove all the special cases for "user" logging. # *.err;kern.notice;auth.notice;user.none /dev/console *.err;kern.debug;daemon.notice;mail.crit;user.none /var/adm/messages *.alert;kern.err;daemon.err;user.none operator *.alert;user.none root *.emerg;user.none * auth.info /var/log/authlog mail.info /var/log/mlog # # Adding log to daemon information # daemon.info /var/log/syslog
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : C++ Humor : ARE YOU A BBS ADDICT? : Object oriented programmers of all nations : C Humor : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor: Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : The Most Comprehensive Collection of Editor-related Humor : Microsoft plans to buy Catholic Church : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor : Best Russian Programmer Humor : Russian Musical Humor : The Perl Purity Test : Politically Incorrect Humor : GPL-related Humor : OFM Humor : IDS Humor : Real Programmers Humor : Scripting Humor : Web Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor :
The Last but not Least
|You can use PayPal to make a contribution, supporting hosting of this site with different providers to distribute and speed up access. Currently there are two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.|
The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Last modified: November 23, 2013