Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Remote Syslog

Old News

Recommended Links

Annotated Papers 

Man Pages

Syslog Syslog-ng
RHEL NTP configuration Troubleshooting NTP on Red Hat Linux Logs analysers Log management Log rotation Syslog viewers
Logwatch Tips Horror Stories Random Findings Humor Etc

The default Solaris syslogd daemon is a primitive (old style) daemon that is controlled by /etc/syslog.conf. It can send/receive messages from the remote hosts but the level of control is very primitive (they will be mixed with the logs from the LOGHOST server). syslog-ng is probably a step forward and can be used with Solaris.  In all cases usage of NTP for clock syncronization is a must. This is the most important prerequsite to implementing syslog aggregation on the "master" syslog server in the organization. See RHEL NTP configuration and Troubleshooting NTP on Red Hat Linux for the additional information.

The default syslog daemon uses UDP port 514 for forwarding messages to a centralized log host. There is no acknowledgement and messages are lost if the LOGHOST server is down, path to the master loghost server is not available or if its syslogd  daemon is down. Some nasty surprises are possible is a firewall filters out this port on the route tot he LOGHOST. 

In Solaris syslog is implemented using STEAMS log driver. The specific wording in syslogd(1M):

syslogd reads from the STREAMS log driver, /dev/log, and from any transport provider specified in /etc/netconfig, /etc/net/transport/hosts, and /etc/net/transport/services.

Solaris differs from other Unixes in the methods of sending remote logs: here is the table from syslog-ng reference manual

Platform Method
Linux A SOCK_STREAM unix socket named /dev/log, some of the distributions layer switched over to using SOCK_DGRAM, though applications still work with either method.
BSD flavors A SOCK_DGRAM unix socket named /var/run/log
Solaris (2.5 or below) An SVR4 style STREAMS device named /dev/log
Solaris (2.6 or above) In addition to the STREAMS device used in versions below 2.6, uses a new multithreaded IPC method called door. By default the door used by syslogd is /etc/.syslog_door

CPU load on a plain vanilla LOGHOST server is minimal and even an old entry level Sun server (V240) can integrated a couple of hundreds of hosts without much sweat. But that does not make sense as after a hundred hosts it is better to split them into groups representing major classes (webservers, database servers, etc) and use either syslog-ng or virtual machine (zones in Solaris 10) for each.

Also due to the mixture of logs from the remote hosts with LOGHOST server logs, generally you will be better off using Solaris for consolidating logs from Solaris servers, AIX for consolidating syslog from AIX servers, etc.

You can use remote syslog with or without encryption as well. As for encryption IPSEC is one option, Stunnel is an another one. But too much zeal is harmful here ;-)

The simplest remote syslog configuration is an unencrypted one. To activate such remote sysloging on Solaris you need:

Solaris uses macro processor M4 to preprocess syslog.conf and you can use all the power of M$ to make the configuration more flexible. Example in the standard syslogd configuration is a pretty weak one: it is based on checking if LOGHOST env. variable is defined (it is only if IP address of the server is defined as loghost in /etc/hosts) The ifdef (‘LOGHOST’, truefield, falsefield) command permits to select two variants of log forwarding depending on this. This permits any server to be used by loghost. Nice, but pretty useless in practice trick as LOGHOST server generally needs to be a specialized and more secure server.

Also you can replace syslogd with syslog-ng. The latter lets filter and route logs. The ability of syslog-ng to filter messages of various types provides the capability to distribute specific message types to people responsible for a given application/function. That makes the environment closer to VM/CMS system ;-). It is possible to invoke a script on the receipt of specific mail messages. If you use Perl the logic can be pretty sophisticated.

Creating a loghost for your servers can be invaluable both for troubleshooting and in the event of a security breach. If you are part of a cluster, make sure your machine logs to another loghost as well as itself. If someone breaks in and erases your log files, and you have only been logging locally, you will have no chance of figuring out where they came from or what they did. Intruders commonly use scripts to erase their presence from the log files. By sending your logs to another more secure machine, you have a better chance of at least tracking the intruder's activities.

Security precautions to take

Additional security precautions for LOGHOST

Protocol concerns

Implementation woes

Protocol gripes

  1. Firewalls & proxies:  need UDP port 514 proxy
  2. Unreliable:  No guarantee that a syslog packet will be received, and no facilities for retransmission (Contra: "Simple protocol, usually not needed on a typical LAN")
  3. Better Timestamping:  It should be a requirement that the timestamp specified in the protocol (both by the client, in the packet; and in the server, when the message is logged) are recorded in the logfile.
  4. Problems with log message formatting & structure:  "The structure of messages is really a mess."
  5. Standardisation:  syslog is not standardised, which makes it more difficult than it should be to produce interoperable implementations. (Contra: "Syslog is currently interoperable and actually more or less standardized between all flavours of Unix though there are some individual variations.")

Implementation woes

Syslog is still the most widespread and usable tool for event logging, esp. because it is a buil-in, low-cost protocol that can be easily added to network and embedded devices. Strategies for cautious use include:

Basic security enhancements


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Jul 2, 2008] Linux Configure Netconsole To Log Messages Over UDP Network by Vivek Gite

July 2, 2008

Linux can be configured to log dmesg output to another system via network using syslog. It is done using kernel level networking stuff ia UDP port 514. There is module called netconsole which logs kernel printk messages over udp allowing debugging of problem where disk logging fails and serial consoles are impractical. Most modern distro has this netconsole as a built-in module. netconsole initializes immediately after NIC cards. There are two steps to configure netconsole:

Step # 1: Configure Centralized syslogd

Login to syslogd.nixcraft.in server. Open syslogd configuration file. Different UNIX / Linux variant have different configuration files

Red Hat / CentOS / Fedora Linux Configuration

If you are using Red Hat / CentOS / Fedora Linux open /etc/sysconfig/syslog file and set SYSLOGD_OPTIONS option for udp logging.
# vi /etc/sysconfig/syslog
Configure syslogd option as follows:
SYSLOGD_OPTIONS="-m 0 -r -x"
Save and close the file. Restart syslogd, enter:
# service syslog restart

Debian / Ubuntu Linux Configuration

If you are using Debian / Ubuntu Linux open file /etc/default/syslogd set SYSLOGD option for udp logging.
# vi /etc/default/syslogd
Configure syslogd option as follows:
SYSLOGD_OPTIONS="-r"
# /etc/init.d/sysklogd restart

FreeBSD configuration

If you are using FreeBSD open /etc/rc.conf and set syslogd_flags option option for udp logging. Please note that FreeBSD by default accepts network connections. Please refer to syslogd man page for more information.

Firewall configuration

You may need to open UDP port 514 to allow network login. Sample iptables rules to open UDP port 514:
MYNET="192.168.1.0/24"
SLSERVER="192.168.1.100"
iptables -A INPUT -p udp -s $MYNET --sport 1024:65535 -d $SLSERVER --dport 514 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p udp -s $SLSERVER --sport 514 -d $MYNET --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT

Step # 2: Configure Linux Netconsole

You need to configure netconsole service. Once this service started, you are allowed a remote syslog daemon to record console output from local system. The local port number that the netconsole module will use 6666 (default). You need to set the IP address of the remote syslog server to send messages.

Open /etc/sysconfig/netconsole file under CentOS / RHEL / Fedora Linux, enter:
# vi /etc/sysconfig/netconsole
Set SYSLOGADDR to 192.168.1.100 (IP address of remote syslog server)
SYSLOGADDR=192.168.0.1
Save and close the file. Restart netconsole service, enter:
# /etc/init.d/netconsole restart

A note about Debian / Ubuntu Linux

Red Hat has netconsole init script. However, under Debian / Ubuntu Linux, you need to manually configure netconsole. Type the following command to start netconsole by loading kernel netconsole module, enter:
# modprobe netconsole 6666@192.168.1.5/eth0,514@192.168.1.100/00:19:D1:2A:BA:A8
Where,

You can add above modprobe line to /etc/rc.local to load module automatically. Another recommend option is create /etc/modprobe.d/netconsole file and append following text:
# echo 'options netconsole netconsole=6666@192.168.1.5/eth0,514@192.168.1.100/00:19:D1:2A:BA:A8 '> /etc/modprobe.d/netconsole

How do I verify netconsole is logging messages over UDP network?

Login to remote syslog udp server (i.e. 192.168.1.100 our sample syslogd system), enter:
# tail -f /var/log/messages
/var/log/messages is default log file under many distributions to log messages. Refer to /etc/syslog.conf for exact location of your file.

How do I use nc / netcat instead of messing with syslogd?

This is called one minute configuration. You can easily get output on 192.168.1.100 without using syslogd. All you have to do is run netcat (nc) command, on 192.168.1.100:
$ nc -l -p 30000 -u
Login to any other box, enter command:
# modprobe netconsole 6666@192.168.1.5/eth0,30000@192.168.1.100/00:19:D1:2A:BA:A8
Output should start to appear on 192.168.1.100 from 192.168.1.5 without configuring syslogd or anything else.

Further readings:

http://www.monitorware.com/Common/en/Ar ... server.php

A paper on optimizing syslog server performance. While it is written for our products, almost all of the same principles apply to generic syslog (except for the multi-threading discussion, because many syslogd's are single-threaded).

SecurityFocus SUN Re Better Syslog server

Date: Mon, 17 Mar 2003 14:08:08 -0500 (EST)
From: Jameel Akari <jakari@bithose.com>
To: Matt Harris <mdh@unix.si.edu>

On Mon, 17 Mar 2003, Matt Harris wrote: 

> I've been looking a bit on google/sourceforge/etc to try and find a more configurable and extensible syslog server, to no avail. Does anyone

There is syslog-ng, but IMO it's still lacking.

> # commands for syslog facility "XXX"
> Facility XXX {
> # Send message to stdin of a script which can send alerts, etc
> Severity emerg ACTION /usr/local/libexec/parse-emerg.sh
> # 192.168.47.9 - sample dhcp server
> Host "192.168.47.9" {
> # log stuff from the dhcp server to a seperate file
> Severity warn FILE /var/adm/dhcp.log
> }
> # send other stuff to a default file
> Severity DEFAULT FILE /var/adm/xxx.log
> }

Looks like a good idea to me.

> functionality I'd like - for example, piping out to a smart script,
> sending to different files based on host that sent the message, etc

Logging to a [My|Postgre]SQL backend instead of or in addition to file would be very cool. Stick a PHP frontend on it and make pretty charts and graphs.

Doing things like "tail -f /var/adm/messages | someparserscript.pl" kinda works, but is... non-optimal?

Syslog.org View topic - Server specs for enterprise syslog servers

There really isn't a good rule of thumb, because there are so many variables in the mix. You have 60 hosts going to 3 servers. I'm not sure if that means that 20 hosts will be logging to each, or if all 60 will be sending logs to all 3. The number of logs that each host will be sending per minute is an important item to know. Syslogd is a very very simple application that pulls the data off the wire and shoves it in a file. There are some implementations of syslogd that do write caching, and that will speed up performance.

Otherwise, it really is going to come down to disk speed & IO primarily. If you're able to get 20Mbps out of a traditional disk, that's going to theoretically give you the ability to process 20000 1024 bits syslog packets per second. Now, there are limitations that the OS, file system, etc will impose that will make that less, but it's going to be a pretty big number.

As a rule, drive speed and IO are paramount, then probably memory size and speed, then clock speed of the processor, in order of priority. Unless you are doing something out of the ordinary on the 60 hosts that are sending syslogs, I think you'll be fine with most any server class hardware that you would throw at it, with a lot of room to spare.

Post subject: RE: Server specs for enterprise syslog servers

Feb 19, 2005

Yeah that's the hard part...there won't be a constant flow of traffic, it all depends on what users changing things on the server at a specific time. It's going to be about 60 hosts per syslog server, so IO speed is the main thing and processor is the least then. I'm planning on piping the information into a mysql database as well.

BigAdmin Feature Article Creating a Centralized Secure Log Server with syslog-ng and Stunnel

UNIX system administrators are quite familiar with the syslog daemon, but information that it collects often remains unprocessed unless someone reports a problem. At any site with more than a handful of machines, no one devotes the time to log in and check multiple log files each day or even each month. Automated scripts that might correlate data across these machines are difficult to write because they must access each machine individually. To lessen the burden of both automatic and manual data processing, many sites implement a central log server that collects data for all machines (preferably running NTP to make time/date correlation easier) on a network, including UNIX servers, Windows and Mac desktops, and even networking equipment like routers and switches. Centralized logging is fairly trivial with most stock UNIX syslog daemons, but syslogd is little changed from early incarnations and has some shortcomings.

The standard UNIX syslog daemon transports messages in the clear over UDP, meaning that anyone can sniff potentially sensitive data. The facility.level model is also fairly restrictive, and the default /etc/syslog.conf files supplied with most operating systems neglect to log many messages administrators might be interested in. Parsing out the logs in a way that makes sense for human readability or automated data mining is non-trivial. The UNIX syslog daemon doesn't easily allow one to split log files out by host or match log messages by regular expression, for example. Therefore, most centralized log servers using the stock syslog daemon wind up with monolithic log files that are only processed after the syslogd processes close them.

As a result, most sites that centralize logging also wind up replacing the stock syslog daemon with something more secure and more flexible such as Metalog, msyslog, or something similar. One very popular syslog replacement is an open source program called syslog-ng. An organization can run syslog-ng on each UNIX host or just on the syslog server itself. If syslog-ng is run only on the log host, clients send data over UDP port 514 as usual, but better log organization and manipulation can be accomplished on the server.

The benefit of running syslog-ng on each UNIX host is the ability to encrypt the logging channel with IPSec or the utility Stunnel so that data is not readable by the casual sniffer. When combined with Stunnel as a transport mechanism, an organization has a secure way of centralizing log messages from all desired UNIX hosts for further processing. In the case of syslog-ng, Stunnel works by accepting log connections on a local port, wrapping them in SSL sessions, and then redirecting them to a secure port on the remote log host. The stunnel process on the remote log host then decrypts the SSL session and hands the information back to the syslog server on the standard port. Once on the log server, organization and parsing of the log files then takes place using the flexibility of syslog-ng.

Below I cover the installation and configuration of syslog-ng and Stunnel on machines running the Solaris 8 Operating System (SPARC Platform Edition), but the procedure also generally applies to older and newer versions of the Solaris OS on both SPARC and x86 platforms. Each of the reference machines discussed below comes installed with OpenSSL, tcp wrappers, the Solaris 8 /dev/urandom patch, and the GNU development environment (gcc, and so on) and several other freeware packages. The machine serving as the log server has also been thoroughly hardened since it will store sensitive and security-related information from all machines on the network. These reference machines all reside on the subnet 192.168.1, and the IP address of the log server is 192.168.1.10.

Central Loghost Mini-HOWTO

Aggregate Logs from Remote Sites

Monitoring the logs of a remote site or just a collocated server can often be overlooked when faced with the task of monitoring activity on your local network. You could use the traditional syslog facilities to send logging information from the remote network or systems, but since the syslog daemon uses UDP for sending to remote systems, this is not the ideal solution. UDP provides no reliability in its communications, and so you risk losing logging information. In addition, the traditional syslog daemon has no means to encrypt the traffic that it sends, so your logs might being viewable by anyone with access to the intermediary networks between you and your remote hosts or networks.

To get around these issues, you'll have to look beyond the syslog daemon that comes with your operating system and find a replacement. One such replacement syslog daemon is syslog-ng (http://www.balabit.com/products/syslog_ng/). syslog-ng is not only a fully functional replacement for the traditional syslog daemon, but also adds flexible message filtering capabilities, as well as support for logging to remote systems over TCP (in addition to support for the traditional UDP protocol). With the addition of TCP support, you can also employ stunnel or ssh to securely send the logs across untrusted networks.

To build syslog-ng, you will need the libol library package (http://www.balabit.com/downloads/syslog-ng/libol/) in addition to the syslog-ng package. After downloading these packages, unpack them and then build libol:

$ tar xfz libol-0.3.9.tar.gz
$ cd libol-0.3.9
$ ./configure && make

When you build syslog-ng you can have it statically link to libol, so there is no need to fully install the library.

And now to build syslog-ng:

$ tar xfz syslog-ng-1.5.26.tar.gz
$ cd syslog-ng-1.5.26
$ ./configure --with-libol=../libol-0.3.9
$ make

If you want to compile in TCP wrappers support, you can add the --enable-tcp-wrapper flag to the configure script. After syslog-ng is finished compiling, become root and run make install. This will install the syslog-ng binary and manpages. To configure the daemon, create the /usr/local/etc/syslog-ng directory and then create a syslog-ng.conf to put in it. To start off with, you can use one of the sample configuration files in the doc directory of the syslog-ng distribution.

There are five types of configuration file entries for syslog-ng, each of which begins with a specific keyword. The options entry allows you to tweak the behavior of the daemon, such as how often the daemon will sync the logs to the disk, whether the daemon will create directories automatically, and hostname expansion behavior. source entries tell syslog-ng where to collect log entries from. A source can include Unix sockets, TCP or UDP sockets, files, or pipes. destination entries allow you to specify possible places for syslog-ng to send logs to. You can specify files, pipes, Unix sockets, TCP or UDP sockets, TTYs, or programs. Sources and destinations are then combined with filters (using the filter keyword), which let you select syslog facilities and log levels. Finally, these are all used together in a log entry to define precisely where the information is logged. Thus you can arbitrarily combine any source, select what syslog facilities and levels you want from it, and then route it to any destination. This is what makes syslog-ng an incredibly powerful and flexible tool.

To set up syslog-ng on the remote end so that it can replace the syslogd on the system and send traffic to a remote syslog-ng, you'll first need to translate your syslog.conf to equivalent source, destination, and log entries.

Here's the syslog.conf for a FreeBSD system:

*.err;kern.debug;auth.notice;mail.crit          /dev/console
*.notice;kern.debug;lpr.info;mail.crit;news.err /var/log/messages
security.*                                      /var/log/security
auth.info;authpriv.info                         /var/log/auth.log
mail.info                                       /var/log/maillog
lpr.info                                        /var/log/lpd-errs
cron.*                                          /var/log/cron
*.emerg                                         *

First you'll need to configure a source. Under FreeBSD, /dev/log is a link to /var/run/log. The following source entry tells syslog-ng to read entries from this file:

source src { unix-dgram("/var/run/log"); internal( ); };

If you were using Linux, you would specify unix-stream and /dev/log like this:

source src { unix-stream("/dev/log"); internal( ) };

The internal() entry is for messages generated by syslog-ng itself. Notice that you can include multiple sources in a source entry. Next, include destination entries for each of the actual log files:

destination console { file("/dev/console"); };
destination messages { file("/var/log/messages"); };
destination security { file("/var/log/security"); };
destination authlog { file("/var/log/auth.log"); };
destination maillog { file("/var/log/maillog"); };
destination lpd-errs { file("/var/log/lpd-errs"); };
destination cron { file("/var/log/cron"); };
destination slip { file("/var/log/slip.log"); };
destination ppp { file("/var/log/ppp.log"); };
destination allusers { usertty("*"); };

In addition to these destinations, you'll also want to specify one for remote logging to another syslog-ng process. This can be done with a line similar to this:

destination loghost { tcp("192.168.0.2" port(5140)); };

The port number can be any available TCP port.

Defining the filters is straightforward. You can simply create one for each syslog facility and log level, or you can create them according to those used in your syslog.conf. If you do the latter, you will only have to specify one filter in each log statement, but it will still take some work to create your filters.

Here are example filters for the syslog facilities:

filter f_auth { facility(auth); };
filter f_authpriv { facility(authpriv); };
filter f_console { facility(console); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_ftp { facility(ftp); };
filter f_kern { facility(kern); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_news { facility(news); };
filter f_security { facility(security); };
filter f_user { facility(user); };
filter f_uucp { facility(uucp); };

and examples for the log levels:

filter f_emerg { level(emerg); };
filter f_alert { level(alert..emerg); };
filter f_crit { level(crit..emerg); };
filter f_err { level(err..emerg); };
filter f_warning { level(warning..emerg); };
filter f_notice { level(notice..emerg); };
filter f_info { level(info..emerg); };
filter f_debug { level(debug..emerg); };

Now you can combine the source with the proper filter and destination within the log entries:

# *.err;kern.debug;auth.notice;mail.crit                /dev/console
log { source(src); filter(f_err); destination(console); };
log { source(src); filter(f_kern); filter(f_debug); destination(console); };
log { source(src); filter(f_auth); filter(f_notice); destination(console); };
log { source(src); filter(f_mail); filter(f_crit); destination(console); };

# *.notice;kern.debug;lpr.info;mail.crit;news.err       /var/log/messages
log { source(src); filter(f_notice); destination(messages); };
log { source(src); filter(f_kern); filter(f_debug); destination(messages); };
log { source(src); filter(f_lpr); filter(f_info); destination(messages); };
log { source(src); filter(f_mail); filter(f_crit); destination(messages); };
log { source(src); filter(f_news); filter(f_err); destination(messages); };

# security.*                                            /var/log/security
log { source(src); filter(f_security); destination(security); };

# auth.info;authpriv.info                         /var/log/auth.log
log { source(src); filter(f_auth); filter(f_info); destination(authlog); };
log { source(src); filter(f_authpriv); filter(f_info); destination(authlog); };

# mail.info                                             /var/log/maillog
log { source(src); filter(f_mail); filter(f_info); destination(maillog); };

# lpr.info                                              /var/log/lpd-errs
log { source(src); filter(f_lpr); filter(f_info); destination(lpd-errs); };

# cron.*                                                /var/log/cron
log { source(src); filter(f_cron); destination(cron); };

# *.emerg                                               *
log { source(src); filter(f_emerg); destination(allusers); };

You can set up the machine that will be receiving the logs in much the same way as if you were replacing the currently used syslogd.

To configure syslog-ng to receive messages from a remote host, you must specify a source entry:

source r_src { tcp(ip("192.168.0.2") port(5140)); };

Alternatively, you can dump all the logs from the remote machines into the same destinations that you use for your local log entries. This is not really recommended, because it can be a nightmare to manage, but can be done by including multiple source drivers in the source entry that you use for your local logs:

source src { 
    unix-dgram("/var/run/log"); 
    tcp(ip("192.168.0.2") port(5140));
    internal( );
};

Now logs gathered from remote hosts will appear in any of the destinations that were combined with this source.

If you would like all logs from remote hosts to go into a separate file named for each host in /var/log, you could use a destination like this:

destination r_all { file("/var/log/$HOST"); };

syslog-ng will expand the $HOST macro to the hostname of the system sending it logs and create a file named after it in /var/log. An appropriate log entry to use with this would be:

log { source(r_src); destination(r_all); };

However, an even better method is to recreate all of the remote syslog-ng log files on your central log server. For instance, a destination for a remote machine's messages file would look like this:

destination fbsd_messages { file("/var/log/$HOST/messages"); };

Notice here that the $HOST macro is used in place of a directory name. If you are using a destination entry like this, be sure to create the directory beforehand, or use the create_dirs() option:

options { create_dirs(yes); };

syslog-ng's macros are a very powerful feature. For instance, if you wanted to separate logs by hostname and day, you could use a destination like this:

destination fbsd_messages {
    file("/var/log/$HOST/$YEAR.$MONTH.$DAY/messages"); 
};

You can combine the remote source with the appropriate destinations for the logs coming in from the network just as you did when configuring syslog-ng for local logging-j e with the proper destination and filters.

Another neat thing you can do with syslog-ng is collect logs from a number of remote hosts and then send all of those to yet another syslog-ng daemon. You can do this by combining a remote source and a remote destination with a log entry:

log { source(r_src); destination(loghost); };

Since syslog-ng is now using TCP ports, you can use any encrypting tunnel you like to secure the traffic between your syslog-ng daemons. You can use SSH port forwarding or stunnel to create an encrypted channel between each of your servers. By limiting connections on the listening port to include only localhost (using firewall rules, as in or ), you can eliminate the possibility of bogus log entries or denial-of-service attacks.

Server logs are among the most critical information that a system administrator needs to do her job. Using new tools and strong encryption, you can keep your valuable log data safe from prying eyes.

Recommended Links

Central Loghost Mini-HOWTO

Annotated Papers

ITS Unix Security Overview Solaris Logging

A. How does it work?

Syslogd is the daemon which does the actual logging. The configuration file for syslogd is /etc/syslog.conf. Basically, syslogd puts time-stamped messages into log files as designated by syslog.conf. The log files can then be viewed to construct a history of the events occurring on your system. Syslogd is started along with the other startup scripts in /etc/rc2.d.

Everybody should run syslogd.

B. Why use logs?

Log files can often provide early warning of an impending attack. Logs can also notify you of problems before your good backups are cycled out. Use logs to help keep track of what is happening on your system at all times. Syslogd records and dates things such as login attempts and failures, system errors, possible security problems, etc. It also alerts you to problems which you may otherwise not notice on a daily basis, such as a partition becoming full. In sum, logging will save you a lot of time and pain if it is done correctly.

C. /etc/syslog.conf

In syslog.conf, there are two fields, the selector and the action. The selector field tells what to log and the action field tells where to log it (i.e., to a certain file, or to a certain host on the network).

The selector has the form: facility.level

The action has the form of a path, e.g., /var/adm/messages or /var/log/syslog, or console, or a full host name preceded by an @ sign: @loghostname.caltech.edu

The facility and level have many possible combinations. But for security's sake, it is far easier to just log everything.

Hint: Send logging to one (or a few) specified central hosts to make things easier to manage in large domains.

D. Creating a loghost

Creating a loghost for your cluster of machines can be invaluable in the event of a security breach. If you are part of a cluster, make sure your machine logs to another loghost as well as itself. If someone breaks in and erases your log files, and you have only been logging locally, you will have no chance of figuring out where they came from or what they did. Intruders commonly use scripts to erase their presence from the log files. By sending your logs to another more secure machine, you have a better chance of at least tracking the intruder's activities.

Security precautions to take:

*.info /var/adm/messages

If you start getting too much of a certain message (say from sendmail), you can always bump that particular facility down by doing

*.info;mail.none /var/adm/messages

touch var/adm/loginlog

chmod 600 /var/adm/loginlog

chgrp sys /var/adm/loginlog

Additional security precautions to take:

/usr/sbin/syslogd

and change it to:

/usr/sbin/syslogd -r

*.info @logboy.caltech.edu

Once you restart syslogd, the client will send all of its log messages to logboy.caltech.edu, which will also then log the messages to its local files.

Be sure that the client is able to resolve logboy.caltech.edu at all times. A suggestion is to add a special entry in /etc/hosts so that logging is not interrupted if there is a resolving problem.

BigAdmin Feature Article Creating a Centralized Secure Log Server with syslog-ng and Stunnel

UNIX system administrators are quite familiar with the syslog daemon, but information that it collects often remains unprocessed unless someone reports a problem. At any site with more than a handful of machines, no one devotes the time to log in and check multiple log files each day or even each month. Automated scripts that might correlate data across these machines are difficult to write because they must access each machine individually. To lessen the burden of both automatic and manual data processing, many sites implement a central log server that collects data for all machines (preferably running NTP to make time/date correlation easier) on a network, including UNIX servers, Windows and Mac desktops, and even networking equipment like routers and switches. Centralized logging is fairly trivial with most stock UNIX syslog daemons, but syslogd is little changed from early incarnations and has some shortcomings.

The standard UNIX syslog daemon transports messages in the clear over UDP, meaning that anyone can sniff potentially sensitive data. The facility.level model is also fairly restrictive, and the default /etc/syslog.conf files supplied with most operating systems neglect to log many messages administrators might be interested in. Parsing out the logs in a way that makes sense for human readability or automated data mining is non-trivial. The UNIX syslog daemon doesn't easily allow one to split log files out by host or match log messages by regular expression, for example. Therefore, most centralized log servers using the stock syslog daemon wind up with monolithic log files that are only processed after the syslogd processes close them.

As a result, most sites that centralize logging also wind up replacing the stock syslog daemon with something more secure and more flexible such as Metalog, msyslog, or something similar. One very popular syslog replacement is an open source program called syslog-ng. An organization can run syslog-ng on each UNIX host or just on the syslog server itself. If syslog-ng is run only on the log host, clients send data over UDP port 514 as usual, but better log organization and manipulation can be accomplished on the server.

The benefit of running syslog-ng on each UNIX host is the ability to encrypt the logging channel with IPSec or the utility Stunnel so that data is not readable by the casual sniffer. When combined with Stunnel as a transport mechanism, an organization has a secure way of centralizing log messages from all desired UNIX hosts for further processing. In the case of syslog-ng, Stunnel works by accepting log connections on a local port, wrapping them in SSL sessions, and then redirecting them to a secure port on the remote log host. The stunnel process on the remote log host then decrypts the SSL session and hands the information back to the syslog server on the standard port. Once on the log server, organization and parsing of the log files then takes place using the flexibility of syslog-ng.

Below I cover the installation and configuration of syslog-ng and Stunnel on machines running the Solaris 8 Operating System (SPARC Platform Edition), but the procedure also generally applies to older and newer versions of the Solaris OS on both SPARC and x86 platforms. Each of the reference machines discussed below comes installed with OpenSSL, tcp wrappers, the Solaris 8 /dev/urandom patch, and the GNU development environment (gcc, and so on) and several other freeware packages. The machine serving as the log server has also been thoroughly hardened since it will store sensitive and security-related information from all machines on the network. These reference machines all reside on the subnet 192.168.1, and the IP address of the log server is 192.168.1.10.

Security issues in network event logging (draft-syslog-bcp)

[Chapter 10] 10.5 The UNIX System Log (syslog) Facility

Enterprise Linux Magazine

LJ 11 Linux System Administration How to Log Friends and Influence People by Mark Komarinski

syslog programming

Using syslog as a Generic Event Transport Mechanism Kevin Steves

Securing Your Host -- syslog.conf

Syslog is a flexible logging facility that allows you to:

Syslog is event driven. Each event is comprised of two parts, a facility and a level. The notation is "facility.level". Normal operating facilities are:
user, kern, mail, daemon, auth, lpr, news, uucp, cron, mark

Levels are:
emerg, alert, crit, err, warning, notice, info, debug, none

A syslog event might look like "mail.debug" or "auth.notice". Any and all combinations are valid. As a historical note, syslog at one time only logged via a loglevel, defined as local0-7. These days not many systems use this type of logging facility. If you are one of the few, please see your systems man page.

Now ever entry in /etc/syslog.conf follows the same format:
event log_location

Note: inbetween the event and the log location is a tab, not a space

In many cases, the log location will be a filename. "syslogd", the syslog daemon, will append to the file all appropriate messages. The file must exist prior to logging, so if you are setting up a new filename make sure you "touch" the file before putting it in your configuration file.

Sample entries for a syslog file are:

One nice capability of the syslog daemon is it's ability to log to a loghost. A loghost is a place you want messages sent to. You can either duplicate important messages (such as the filesystem is full) that you write out to your normal file, or have all messages just go there.

Loghost is defined by default as the local host in /etc/hosts. You can change it to whatever machine you want to log, and then simple put in /etc/syslog.conf
event loghost

The second way to define a loghost is simply to specify it in the syslog.conf file by changing the log location to be @loghost.foo.bar.
An example is:
# event @myloghost.foo.bar
mail.debug @myloghost.foo.bar

Remember, myloghost.foo.bar must be accepting syslog notices in order to log messages generated from your machine.

Last, it is recommended that you create the file /var/adm/loginlog. If this file exists, all failed login attempts will be logged there in cleartext. This is a very useful, if undocumented, feature.

SurLink Global Networking, Inc. - FreeBSD SECURITY HOW-TO


syslogd

Many, if not most, serious applications use a facility known as syslog to write and distribute messages. The actual logging of these messages can occur on the local system, a remote system, be sent to specific files, be sent to the system operator, or sent to other users. In most of the following applications I describe in this paper you will see excerpts from the system log created by these applications.

syslog allows you to consolidate your consoles just as PROP does under VM. At NSW TAB we have two AIX-based firewalls which produce copious amounts of log messages tracking the access and usage of our Internet connection. Initially, each of these systems created their own logs. With syslog these messages could be consolidated on, at first, one of the firewalls; and, following the port of syslogd on to our VM/ESA system.

The ability of syslog to route messages of various types then allows us to distribute specific message types to people responsible for a given application/function. The message traffic can also be directed to the VM system operator for integration with PROP. Although we haven't implemented it, it would be possible for an action routine to be invoked by PROP on the receipt of specific messages which would initiate a process to control the remote system.

It is also possible to provide the reverse of this process. That is, VM messages could be sent to a remote *NIX system for consolidation and action. This would involve providing PROP (or the NetView *NCCF service) with routines which used the syslog service.

The following is the /etc/syslog.conf file from our VM/ESA system:

Figure 1. Local syslog Configuration File

# Critical messages go to VM System Operator
*.crit				*SYSOPR
# Dump debug messages to console
*.debug				CON
# News notices go to VM Users MAINT and NEALE
news.notice			MAINT,NEALE
# Syslog debug messages can be logged to file
syslog.debug			/home/neale/SYSLOGD/syslog.log
# Non-syslog debug messages can be logged to file
*.debug				/home/neale/SYSLOGD/debug.log
# News and Mail debug messages can be logged to own file
news.debug,mail.debug	/home/neale/SYSLOGD/newsmail.log


Figure 2. Remote syslog Configuration File

*.debug         /var/adm/messages
*.debug         @vma.tabnsw.com.au
*.crit          @vma.tabnsw.com.au

Man Pages

docs.sun.com man pages section 1M System Administration Commands

OPTIONS

Linux.com - Man Reference - syslog.conf

-usr-share-man-cat5-syslog.conf(0)

CHPTR014.HTM -- nice buttons ;-)

System Administration


Troubleshooting

[Sunhelp] Syslog.conf problems

> *.emerg;*.crit;*.alert;*.err                    @sysa
> 
> Where sysa is a valid hostname in the host table. did a kill -HUP on the 
> sylog pid.

what's in /etc/nsswitch.conf for "hosts"?  You can use "getent hosts sysa"
to see where the system really things "sysa" is.  

Also, the spaces between *.err and @sysa have to be tabs.
NEW QUESTION & SUMMARY syslog.conf
The new question is for the below message how can I determine what is the
name of the facility and level ?
*.debug logs this message, daemon.debug does not. What I would like to do
is to filter below messages to a seperate file.

>logmsg: pri 24, flags 2, from beta, msg Jun 25 14:38:41
>in.telnetd[23112]: refused connect from xxx.yyy.tr
>Logging to UNUSED

SUMMARY:

My original question was:

>
>I have following line in my /etc/syslog.conf
>*	/var/adm/test
>daemon.* /var/adm/test
>user.* /var/adm/test
>
>And when I do syslogd -d I track the message:
>
>logmsg: pri 24, flags 2, from beta, msg Jun 25 14:38:41
>in.telnetd[23112]: refused connect from xxx.yyy.tr
>Logging to UNUSED
>
>to UNUSED, I suppose this means it is discarded.
>Ofcourse /var/adm/test is not even created.
>What can I do to make it Logging to /var/adm/test ?

Summary of answers is as below:

You must create the file beforehand (touch //var/adm/test)
and make sure you use TABS, not spaces.


The wildcard entry is *.debug, not *.

Thanks to all who replied.

-cagri

mail Summary - configuring syslog.conf
To all of the people who responded to my questions, many thanks..
(There were just too many responses to thank everyone individually).

Overall, the suggestions were similar.

Don't use spaces, use tabs when configuring syslog.conf.
After making changes, kill -HUP pid for syslog.conf.

The message below is from Kai O'Yang who was one of may who forwarded
their syslog.conf files to share.

I am now receiving auth.notice messages from a remote system to my
loghost(on both the console and authlog file.

The only real problem I have that I haven't been able to resolve with
this is that the name of the remote host is not showing up. Instead,
I am receiving "???" in its place, and garbage on the device name:

Oct 23 14:44:32 ??? su:'su root' succeeded for mconroy on /dev/pts/3^m

I am sure it is configured correctly in dns. So I am at a lost.

Any thought???

Thanks again for everyone's help.

Mark Conroy


First add a loghost alias in /etc/hosts or nis table for the syslog
host. Here's my syslog.conf for all client machines.

#ident "@(#)syslog.conf 1.3 93/12/09 SMI" /* SunOS 5.0
*/ #
# Copyright (c) 1991-1993, by Sun Microsystems, Inc. #
# syslog configuration file.
#
# This file is processed by m4 so be careful to quote (`') names #
that match m4 reserved words. Also, within ifdef's, arguments #
containing commas must be quoted.
#
# Note: Have to exclude user from most lines so that user.alert #
and user.emerg are not included, because old sendmails # will
generate them for debugging information. If you
# have no 4.2BSD based systems doing network logging, you #
can remove all the special cases for "user" logging.
#
*.err;kern.notice;auth.notice;user.none /dev/console
*.err;kern.debug;daemon.notice;mail.crit;user.none @loghost

*.alert;kern.err;daemon.err;user.none operator,@loghost
*.alert;user.none root,@loghost

*.emerg;user.none @loghost auth.info
@loghost
mail.info @loghost daemon.info
@loghost

For the loghost:
#ident "@(#)syslog.conf 1.3 93/12/09 SMI" /* SunOS 5.0
*/ #
# Copyright (c) 1991-1993, by Sun Microsystems, Inc. #
# syslog configuration file.
#
# This file is processed by m4 so be careful to quote (`') names #
that match m4 reserved words. Also, within ifdef's, arguments #
containing commas must be quoted.
#
# Note: Have to exclude user from most lines so that user.alert #
and user.emerg are not included, because old sendmails # will
generate them for debugging information. If you
# have no 4.2BSD based systems doing network logging, you #
can remove all the special cases for "user" logging.
#
*.err;kern.notice;auth.notice;user.none /dev/console
*.err;kern.debug;daemon.notice;mail.crit;user.none
/var/adm/messages

*.alert;kern.err;daemon.err;user.none operator
*.alert;user.none root

*.emerg;user.none * auth.info
/var/log/authlog mail.info
/var/log/mlog
#
# Adding log to daemon information
#
daemon.info /var/log/syslog

Configuration Examples

syslog.conf.html

syslog.conf-server

syslog.conf-client

aixfaq

1.115: Where are the AIX log files kept?
From: dirk@kimosabi.ucsc.edu (Dirk Coldewey)

Such as the equivalent of the SunOS file '/var/log/syslog' and
'/var/adm/messages'.

If you want regular unix syslog files, you can configure syslogd
in /etc/syslog.conf. Here's the one that I use:

#
# See the supplied /etc/syslog.conf file for copious comments.
#
*.err;kern.debug;auth.notice;user.none /dev/console
*.err;kern.debug;daemon,auth.notice;mail.crit;user.none /var/adm/messages
lpr.debug /var/adm/lpd-errs

*.alert;kern.err;daemon.err;user.none operator
*.alert;user.none root

*.emerg;user.none *

# for loghost machines, to have authentication messages (su, login, etc.)
# logged to a file, un-comment out the following line and adjust the
# file name as appropriate.
#
# if a non-loghost machine chooses to have such messages
# sent to the loghost machine, un-comment out the following line.
#
auth.notice /var/log/authlog

mail.debug /var/log/syslog

# following line for compatibility with old sendmails. they will send
# messages with no facility code, which will be turned into "user" messages
# by the local syslog daemon. only the "loghost" machine needs the following
# line, to cause these old sendmail log messages to be logged in the
# mail syslog file.
#
user.alert /var/log/syslog
#
# non-loghost machines will use the following lines to cause "user"
# log messages to be logged locally.
#
user.err /dev/console
user.err /var/adm/messages
user.alert `root, operator'
user.emerg *


FOCUS on Linux Securing Linux

-- begin sample syslog.conf --

*.=info;*.=notice;*.warn;mail.none              /usr/adm/messages
*.=debug                                        /usr/adm/debug
*.warn;*.crit;*.emerg                           /usr/adm/syslog
*.warn;*.crit;*.emerg                           /usr/adm/messages

auth.*                                          /usr/adm/log.auth
cron.*                                          /usr/adm/log.cron
daemon.*                                        /usr/adm/log.daemon
kern.*                                          /usr/adm/log.kern
mail.*                                          /usr/adm/log.mail
mark.*                                          /usr/adm/log.mark
syslog.*                                        /usr/adm/log.syslog
user.*                                          /usr/adm/log.user

-- end sample syslog.conf --

Central Loghost Mini-HOWTO

This page is simply a collection of open source tools you can use to glue together your own centralized (syslog) loghost. Included are example configuration settings so that you can configure your loghost in a manner similar to mine.

There is very little that you need to read and understand in order to use these tools. Also, these tools are widely used and therefore easy to get help with on internet mailing lists.

I established a centralized location for syslog collection in order to facilitate:

  1. Log reporting
    • real time alerting
    • periodic (several times per day) summary reporting
  2. Log storage
    • long term archival for possible later analysis

Tools used:

Random Findings

passlogd - passive syslog capture daemon

passlogd is a purpose-built sniffer for capturing syslog messages in transit. this allows for backup logging to be performed on a machine with no open ports. useful if your log server is compromised or you don't want to sift through a terabyte of sniffer logs. passlogd is currently under development, so check back here for bugfixes, new features, etc. if you find bugs, please let me know, and by all means, send me a patch.

NOTE: the latest version (0.1e) contains important security fixes to the parser.



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: September, 12, 2017