Softpanorama May the source be with you, but remember the KISS principle ;-)
	Contents	Bulletin	Latest	Last year	Top visited	Scriptorama

Mon -- King of Simplicity among Unix Monitoring Packages

News	See also	Recommended Links	Perl
Mon	Spong	Big Sister	Nagios
SAR		Humor	Etc

mon (see also FSF page and kernel.org page) is the king of simplicity among monitoring products that we reviewed here. It was written by Jim Trocki and released under GPL.

Active development slowed considerably after 2001. Still there have been several new features added and bugs fixed in mon 1.2.0.

The author wisely resisted the overcomplexity trap and mon remains simple and capable product with the transparent code base, the codebase that can be easily understood by wide audience of system administrators. In one of his posts David Nolan aptly stated:

My best summary of Mon is that its monitoring for sysadmins. Its not pretty, its not designed for management, its designed to allow a sysadmin to automate the performance monitoring that might otherwise be done ad-hoc or with cron jobs.

It doesn't trivially provide the typical statistics gathering that many bean-counters are looking for, but its extensible and scalable in amazing ways. (See recent posts on this list about one company deploying a network of 2400 mon servers and 1200 locations, and my mon site which runs 500K monitoring tests a day, some of those on hostgroups with hundreds of hosts.)

This is really a quintessential monitoring package "by sysadmin for sysadmins" -- simple, very flexible (each probe can be heavily modified to suit your needs without much digging into documentation) but functional enough to provide great value. And in his reply to the David Nolan post the author of the mon stated:

... the design of mon is extremely flexible, and was purposefully built the way it was in order to leverage other tools which already exist. it follows the traditional Unix design philosophy, which i think is the most elegant system design in existence to this very day. it is all about having a mechanism to connect together lots of smaller tools which do one job very well in order to solve larger problems, rather than writing a large tool for each new problem.

you can also think of this design in terms of using natural language, words and grammar to phrase something you want to say. perl itself also follows this model. larry wall is a linguist (a cunning one at that, sorry couldn't resist the pun), and he applied that to perl.

for example, mon leverages fping, the net-snmp tools, traceroute, rrdtool, etc. another example of mon's flexibility is how an on-call notification system with escalation was added without changing anything in mon at all, it was just a matter of writing a custom alert and plugging it in to your mon configuration file with the correct grammar.

in order to get a good idea of how mon works, i would recommend reading the slides from this presentation:

ftp://ftp.kernel.org/pub/software/admin/mon/mon-talk-0.4.tar.gz

And that fact that is written in "simple Perl" without OO perversions only increases its value. One of the first versions (0.38j) is small enough to get basic ideas without getting headache and I recommend to get it first. You should use the latest version but to get the ideas you still need to look at code of 0.38j.

Due to its design philosophy mon remains a rather simple package and by-and-large escaped Christmas Tree effect that hunts other monitoring packages like Nagios. It's still completely written in Perl and you can study what it is doing in debugger. It is also relatively easy to understand how particular checks are programmed and that fact alone constitutes tremendous advantage as it permit easy customarization.

Due to its design philosophy mon remains a rather simple package and by-and-large escaped Christmas Tree effect that hunts other monitoring packages like Nagios.

Mon fundamentally is a scheduler which executes the monitors (each tests a specific condition or report specific data), and calls the appropriate alerts if the monitor fails.

It views resource monitoring as two separate tasks: the testing of a condition, and triggering an action upon failure. It is important to understand that mon implements the testing and action-taking tasks as separate, stand-alone programs: invocation of an alert is governed by logic which offers various "squelch" features and dependencies, configurable by the user.

The current version is available from ftp://ftp.kernel.org/pub/software/admin/mon/devel

mon was developed under Linux, but it is known to work under Solaris. As both the clients and server are written completely in Perl portability should not be an issue. I would be very surprised if there are serious portability issues with mon. It also is included as standard vendor provided package on many Linux distributions such as Suse and Ubuntu.

Configuration

On startup, mon reads a configuration file and then performs tests as scheduled in the configuration file.

Each mon monitor (called "service monitor" in documentation) is a Perl script located in /etc/mon/mon.d or any other directory specified in /etc/mon/mon.cf configuration file. For example Suse packages uses iether /usr/lib/mon/mod.d or /usr/lib64/mon/mod.d depending on whether you use 64-bit kernel or not.

Probes

Mon was designed to be open and extensible in the sense that Mon's monitor programs can be any executable. The relevant quote from the documentations is:

Monitor processes are invoked with the arguments specified in the configuration file, appended by the hosts from the applicable host group.

.... should return an exit status of 0 if it completed successfully (found no problems), or nonzero if a problem was detected. The first line of output from the monitor script has a special meaning: it is used as a brief summary of the exact failure which was detected, and is passed to the alert program. All remaining output is also passed to the alert program, but it has no required interpretation.

There are more then a dozen prepackaged monitors that come with mon and most can be easily customized to your own needs:

Disk space Disk space monitoring via NFS or local mounts. Disk space monitoring is probably the first and the most important task that sysadmin need to solve.
ICMP echo (ping) Efficient host reachability via ICMP echo, using fping, which multiplexes pings using a single process. If a group of hosts is pinged, and if more than one fails, then the collective timeout before the monitor script returns is what it would normally be if you're pinging only one host.
SMTP SMTP connectivity is tested by connecting to the SMTP port on one or more servers, listening for a "220" response, sending a "QUIT", and waiting for a "221" response and then a shutdown.
Telnet Waits for an actual login prompt to determine if the service is working. This uses Wietse Venema's tcp_scan utility.
FTP FTP connectivity is tested by connecting to the FTP server, sending a "quit", and waiting for the appropriate response.
NNTP Similar to SMTP testing, speaks the NNTP protocol.
HTTP Speaks the HTTP protocol to fetch a URL from an HTTP server. The http_t monitor can measure page delivery latency and report failures if a minimum transfer speed is not met. There are several variants of the HTTP monitor for use in different situations.
POP-3 Similar to SMTP, speaks POP-3 to connect and then log out.
IMAP Similar to POP-3, speaks IMAP to connect and then log out.
TCP-based services Any simple TCP-based service can be tested. mon comes with a "tcp.monitor" utility (which is a wrapper for Wietse Venema's tcp_monitor from the Satan distribution). This will send output to a server, and for a particular response, with a timeout.
Uptime via Asynchronous SNMP Calculate the uptime and downtime of any SNMP-capable host. Hundreds of hosts can be polled in a few seconds using the asynchronous method.
HP Printers via SNMP Using SNMP, hpnp.monitor polls HP printers with JetDirect interfaces and can detect paper outages, paper jams, toner cartridge problems, and more. When a failure is detected, it can report the message which is displayed on the printer's LCD, e.g. "TRAY 2 EMPTY".
Processes via SNMP process.monitor polls the UCD SNMP agents on some number of hosts and reports problems with particular processes. This monitor is useful to detect if a particular process has died off.
Network Appliance free space via SNMP netappfree.monitor polls Network Appliance filers for free space on each exported filesystem. It does not require that you mount the filesystems on the mon host. Per-filesystem low watermarks are configured in a separate configuration file for easy changing without resetting the mon server.
Network Appliance quota trees via SNMP na_quote.monitor can monitor free space on multiple quota trees. Currently a work-in-progress.
LDAP ldap.monitor searches an LDAP server for objects which match a particular filter.
DNS dns.monitor queries a zone master for base information, and then queries one or more secondaries, comparing the serial number found in the SOA record to that of the master.
Dial-in terminal servers dialin.monitor can periodically dial in to a terminal server or some other modem host and report whether or not a connection was successfully made to the remote host, and if a login prompt was obtained.
mSQL and MySQL databases The msql-mysql monitor connects to the database in question and list the tables to ensure that the host is responding to queries.
Network latency up_rtt.monitor uses the UDP or TCP echo service to measure the round-trip time between the monitoring server and a set of hosts. If the measured latency exceeds a minimum, then a failure is reported.
Sun RPC Services The RPC monitor can call the RPC NULL function to determine if a server is responding to queries. This method can be used to poll NIS, NFS, mountd, and other popular services which use Sun RPC.
Foundry chassis Report power supply, fan, temperature, module failures in Foundry BigIron and FastIron routers and bridges.
Compaq Presario chassis Report power supply, fan, and temperature problems with Presario systems via the "Insight Manager" SNMP agent.
Brocade SilkWorm FCAL switches Report PSU, fan, and temperature, PHY, and link problems.

If a test fails, Mon calls the alert as specified in the configuration file. For example (taken from version 038j):

use File::Df;

foreach (@ARGV) {
    ($path, $minavail) = split (/:/, $_, 2);

    ($fs_type, $fs_desc, $used, $avail, $fused, $favail) = df ($path);
    if (!defined ($used)) {
	push (@failures, "statfs error for $path: $!");
	next;
    }

    if ($avail < $minavail) {
    	push (@failures, sprintf ("%1.1fGB free on %s", $avail / 1024 / 1024,
	    $path));
    }
}

if (@failures) {
    print join (", ", @failures), "\n";
    exit 1;
}

exit 0;

Most are simple scripts that notify users about a particular problem via particular communication channel like example SMTP mail or instant messaging. For example mail.alert that channel alerts into SMTP mails is structured as following:

open (MAIL, "| /usr/lib/sendmail -oi -t") ||
    die "could not open pipe to mail: $!\n";
print MAIL ...

Alerts

Mon alerts are also Perl scripts. They are located in /etc/mon/alert.d. Several prepackaged alerts that can be easily adapted to the environment that come with mon:

Email notification. This is the most common type of modification those days and with the popularity of Blackberries the most portable.
Alphanumeric paging notification via modem, using QuickPage. Pagers are still used althouth not that much like 10 years ago.
Alphanumeric messaging to portable devices using SNPP.
Alphanumeric paging via email.
Traps can be delivered to remote mon servers.

Web interface

There is also several WEB interfaces for mon.

mon.cgi. This is the most powerful interface available. Implemented as a pretty big monolythic Perl script (152K or 3822 lines) originally written by Arthur K. Chan and for some time (Feb 2000-May 2001) maintained by Ryan Clark. The last version is mon.cgi-1.52.tar.gz dated 2001.
bbmon.cgi (BBmon, The BB-like Mon Web interface) Simplified Web status interface for the Mon resource monitoring system. bbmon.cgi was written to offer a BigBrother-like interface to Mon. It only returns a visual state of the monitored systems. As of today it doesn't offer any interaction, use mon.cgi instead.
monshow, which ships with mon in the clients/ subdirectory of the mon distribution. monshow is
a simple report-only tool which supports configurable "views" of the mon configuration. monshow also operates as a textmode report generator.
Minotaure by Gilles Lamiral's, which can be found at ftp://ftp.kernel.org/pub/software/admin/mon/contrib/ .

Old News ;-)

[Jan 15, 2010] INSTALL - Mon Wiki

The "mon" daemon uses Perl 5.n, where n >= 005_01.
Mon requires that *.ph be created from the system header files. If you try to
run mon and Perl complains with the "did you run h2ph?" message, then chances
are this step wasn't done, either by your package manager or manually after
Perl installation. You can fix it by doing the following, as root:
cd /usr/include
h2ph -r -l .
You'll need the following modules for the server to function, all of
which are available from your nearest CPAN archive. The listed
CPAN paths relative to /cpan/modules/by-authors/id/ -- versions of
modules on CPAN change quickly, so there may be newer versions available,
but the following are known to work:
Time::Period PRYAN/Period-1.20.tar.gz
Time::HiRes J/JH/JHI/Time-HiRes-1.59.tar.gz

2. INSTALLING THE PERL CLIENT MODULE
------------------------------------
The Perl client module is distributed as a separate package. It is named
"mon-client-*.tar.gz". Refer to that for installation instructions.
It is available on kernel.org mirrors in the /pub/software/admin/mon directory,
and in CVS on sourceforge.net. Be sure to match the version of mon-client with
the version of mon you are using. At this time, branch "mon-1-2-branch" of the
mon CVS module matches the "mon-client-1-2-branch" branch of the mon-client CVS
module. See http://sourceforge.net/projects/mon/ for information on CVS access.

3. COMPILING THE C CODE (optional)
----------------------------------
Some of the monitors included with mon are written in C and need to
be compiled for your system. If you want to use the RPC monitor or the
dialin.monitor wrapper,
cd mon.d
(edit Makefile)
make
make install
cd ..
Keep in mind that although this is known to work on Linux, Solaris, and AIX,
it may not compile on your system. It is not required for the operation of mon
itself.

4. MONITORS
-----------
All of the monitor and alert scripts that are packaged with mon are
actually *optional*. However, this is what you'll need for each special
monitor, with CPAN paths relative to /cpan/modules/by-author/id/
freespace.monitor - requires Filesys::Diskspace from CPAN,
in FTASSIN/Filesys-DiskSpace-0.05.tar.gz

fping.monitor - requires the 'fping' binary, from http://www.fping.com
RPM packages available at http://dag.wieers.com/packages/fping/
telnet.monitor - requires the Net::Telnet from CPAN,
in J/JR/JROGERS/Net-Telnet-3.03.tar.gz
reboot.monitor
asyncreboot.monitor
netappfree.monitor
process.monitor
snmpdiskspace.monitor
cpqhealth.monitor
foundry-chassis.monitor
na_quota.monitor
silkworm.monitor
snmpvar.monitor
xedia-ipsec-tunnel.monitor
hpnp.monitor
Use the 'net-snmp' package (formerly UCD SNMP), from
http://sourceforge.net/projects/net-snmp
with G.S. Marzot's Perl module G/GS/GSM/SNMP-4.2.0.tar.gz
ldap.monitor - requires Net::LDAPapi from CPAN,
CDONLEY/Net-LDAPapi-1.42.tar.gz
dialin.monitor - requires the Perl Expect module from CPAN,
R/RG/RGIERSIG/Expect-1.15.tar.gz
dns.monitor - requires Net::DNS from CPAN,
C/CR/CREIN/Net-DNS-0.47.tar.gz
msql-mysql.monitor
requires the MSQL/MySQL DBD module and DBI front-end.
There are a good number of useful monitors included which are not listed
above, so please take the time to read doc/README.monitors to get the
details.
5. MON.CF CUSTOMIZATION AND STARTUP
-----------------------------------
-Read the man page for "mon" and "moncmd" in the doc/ directory to get
an overview of the directories involved, i.e. the configuration,
alert, monitors, state, and run directories.
cd doc
nroff -man mon.8 | more
-read the "READMEs" in the doc/ directory for some useful
insight on system configuration.
-Be sure you have the required Perl modules, which are listed in this
file.
-Make your own mon.cf file, using the suppled "example.cf" (located
in the etc/ directory) as a template, or the m4-based "example.m4":
cp etc/example.cf mon.cf
or
cp etc/example.m4 mon.m4
-Edit the "auth.cf" file. This file controls which users can perform
what command. The default is pretty restrictive (read-only), but that's
only for safety. Currently, "moncmd", "monshow", and "mon.cgi" are the
only clients which are able to authenticate themselves to the server;
the 2-way pager interface does not yet perform authentication. However,
these programs work fine in read-only mode.

-Add the following lines to /etc/services:
mon 2583/tcp # MON
mon 2583/udp # MON traps
-You may want to make a DNS CNAME entry called "monhost" for your
host that will run "mon". You can then set the environment variable
MONHOST to be this host. "moncmd" uses this variable.
-The Perl scripts look for perl in /usr/bin. You might want to change
this. I'd advise keeping a locally-installed copy of Perl if you're
going to monitor network resources and you expect this stuff to work
when some component of the network is down.
-Test it by starting "mon" from the distribution directory. Use these
arguments if you chose the non-m4 config:
./mon -f -c mon.cf -b `pwd`
and these arguments for the m4-based config:
./mon -f -M -c mon.m4 -b `pwd`
To see if it's running on your machine:
./clients/moncmd -s localhost list pids
If you get some output, then things are probably OK. Check the
syslog for further diagnostics.
Mon doesn't really need to be installed in any special location. Just
keep it on the local disk of the machine which will be running the server.

WEB INTERFACE
-------------
This distribution contains two web interfaces: monshow and mon.cgi. monshow is
a simple report-only tool which supports configurable "views" of the mon
configuration. monshow also operates as a textmode report generator.
mon.cgi, however, supports the full functionality of mon, including the ability
to disable/enable groups and hosts and services, acknowledge failed services,
show alert and downtime history, authenticate users, among many other things.
To install monshow, simply copy clients/monshow into your web server's cgi-bin
path and name it "monshow.cgi". You may want to read the man page in the doc/
directory so that you can understand how to configure a "view" to your liking.
To install mon.cgi, follow the instructions found in doc/README.mon.cgi.

[Jan 14, 2010] Monitoring with MON by Muhammad Kamran Azeem

March 17, 2009 | WBITT!

This is an extract from an implementation I did on a heartbeat based system. Mon is an excellent monitoring and reporting tool.

It is assumed that there are two servers connected to each other over multple heartbeat links.

MON for heartbeat monitoring:
----------------------------
both servers:
--------------
wget ftp://ftp.kernel.org/pub/software/admin/mon/mon-1.2.0.tar.gz
wget ftp://ftp.kernel.org/pub/software/admin/mon/mon-client-1.2.0.tar.gz
tar xzf mon-1.2.0.tar.gz
tar xzf mon-client-1.2.0.tar.gz

cd mon-client-1.2.0
perl Makefile.PL
make && make install

[Sept 4, 2007] Letter to the editor from Jim Trocki

Error corrected. Thanks for feedback...

This states "Active development stopped approximately in 2001", which is incorrect. Things slowed down for a bit, but there have been many new features added and bugs fixed in mon 1.2.0 which was released at the end of June this year.
Thank you for the nice write-up, though.