Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Shadow IDS

SHADOW (Secondary Heuristic Analysis for Defensive Online Warfare) is a very simple combination of tcpdump and Perl that actually works as a rudimentary NIDS but can made more powerful by using snort and/or custom Perl processing scripts. Due to flexibility and power of Perl the architecture used by Shadow remains one of the best NIDS architectures and in my opinion is architecturally superior to Snort.  

The original Shadow code was developed by the US Navy as the result of the "Cooperative Intrusion Detection Evaluation and Response" (CIDER) project. Currently SHADOW codebase is unmaintained abandonware and the code base is largely obsolete. The latest published version (2003) was developed by NSWC.

Solaris is an excellent platform to run any NIDS and Shadow is not an exception. But most steps actually are OS independent. 

Shadow basically consists of two parts that can run on different boxes (although it is not strictly necessary):

• Sensor. The sensor station is the server that running tcpdump that records traffic to a file. If necessary any suitable transport mechanism can be used to transfer file to central "analysis station" (ssh sync, etc). With the current CPUs and disks speed one sensor can run several instanced of TCPdump without breaking sweet and thus can listen to several traffic interception points. 
   
• Analysis Station. The analysis station is optional and serves as the main processing power for recorded traffic (that implies that files needs to be transferred to this station prior to analysis or NFS would be used. That may require creation of a separate segment as it due to volume of traffic it can negatively affect other network users)
   
  • Recorded traffic is runs through a series of Perl filters. All filters are tuned to tcpdump format. As Perl is much more powerful then ad-hoc capabilities of Snort you can get more useful alerts. You can also use Snort for processing tcpdump file to do the dirty work and produce "first level alerts" and database file for further processing.
  • If you are smart the next step can be integration of multiple steams of alerts via Tivoli TEC or similar monitoring engine.  

While the idea is transparent and very good, the code supplied is not very transparent and is of limited usefulness for any knowledgeable Perl programmer. There are things that definitely can be simplified.

The key files are:

cleanup.pl
condense.pl
fetchem.pl
find_scan.pl
obfuscate.pl

There is also pretty convoluted file (sensor_driver.pl) to launch tcpdump and it probably can be discarded and replaced with a simple ksh script. 

Actually you need is the following simple loop that can be invoked from cron each selected sensor using preselected interval, for example 15 min:

1. Write file with tcpdump to the archive (possibly compress it if you nee to work with huge pipe).

2. Optionally transfer it to the "analysis station".

3. Reprocess this file with TCPdump working in "listing mode", feeding the output to Perl analyzer that deciphers TCP dump format and produced actual alerts.  Or better use Snort snort instead of TCPdump for doing this and reprocessed generated alerts with Perl (using Snort database format)

Old News

[Apr 27, 2005] NewsForge Build an IDS with Snort, Shadow, and ACID by Mark Bain.

It looks like the author made it all wrong.

Title		Build an IDS with Snort, Shadow, and ACID
Date		2005.04.27 11:00
Author		Mark Bain
Topic

http://software.newsforge.com/article.pl?sid=05/04/14/1535223

Every organization understands the importance of using a firewall to protect its assets. But what happens if someone finds a hole in the wall? What if the infiltrator is actually someone from within your organization who wants to access information that he shouldn't? To cover those contingencies, you need an intrusion detection system (IDS) to complement your firewall. Fortunately, with a minimum amount of time and money you can set up an IDS with open source tools such as Snort, Shadow, and ACID.

An IDS comprises sensors to do the actual monitoring and a central console to store and analyse the data collected on each sensor. A sensor can be a basic PC with an extra network card (to do the actual monitoring), whilst the central console should have a bit more processing power and a larger disk.

If Linux isn't already on the PCs you plan to use then you'll have to install it. You'll need only a minimal install because these boxes will be used only for the IDS. I recommend installing Bastille to make the IDS more secure. Another key thing to do is to put the monitoring network card on the sensor without an IP address; this makes it effectively invisible.

You can now start installing the software to do the actual intrusion detection. You can use Snort to monitor patterns of behaviour (or signatures) and Shadow to monitor packet headers. You'll need ACID to analyse the data collected by Snort (Shadow has its own analyser).

Let's begin with the sensor. The first piece of software to install is the client and develop RPMs for MySQL drivers, which are necessary because Snort store its output in a database on the central console. Once these drivers are in place, download the tar file for Snort, decompress it, and install it with the commands:

./configure --with-mysql
make
make install

The installation and setup instructions for Snort are clear and concise. Snort uses a set of rules to identify signatures (or patterns of behaviour) to identify illegal activity. You can download a default set of customisable rules from Snort's site, along with some excellent documentation. If you have a Windows client, you can manage the signatures using Activeworx's IDS Policy Manager.

Shadow

With Snort on the workstation, the next step is to install Shadow. The software comprises a set of Perl scripts that use tcpdump to collect data from packet headers on the network in order to identify any unusual activity. Shadow's Web site contains documentation that covers package dependencies (including Perl and tcpdump; the latter is supplied with Shadow), installation, and setup. To configure Shadow you need to modify three files -- sensor_init.sh, std.ph, and std.filter -- to reflect your local setup, such as the location of tcpdump. You also need to set up crontab to run sensor_driver.pl (the main script of the sensor) once an hour. The package supplies a suitable cron file -- /usr/local/SHADOW/sensor/sensor_crontab -- that also will synchronize the sensor time with a reliable Network Time Protocol source.

Finally, you must make sure that Shadow runs at system startup time by copying /usr/local/SHADOW/sensor_init.sh to your /etc/rc.d directory. Once Shadow is running, your sensor PC is ready to be connected to the network.

Shadow comes with its own analyser, which requires tcpdump, OpenSSH, and Apache. The Shadow documentation will lead you through the configuration process, tell you how to set up the Shadow CGI scripts, and explain how to set up crontab for automatic operation.

The console

With the sensor set up you can turn to the central console, where the actual analysis of the data will be done.

Snort has no analysis software itself -- it simply generates data on the sensor and then stores it in a MySQL database on the central console. You can use ACID (Analysis Console for Intrusion Databases) to analyse the data. ACID's Web site gives full details on software installation and the creation of the MySQL database for storing the Snort alerts.

At this point your IDS should be up and running with Snort, Shadow, and ACID. On the sensor Snort will be monitoring the traffic patterns and raising alerts to the database. Shadow will be reading packet headers and creating log files. On the central console you will be able to view the Web pages created by Shadow, and to use ACID to analyse the alerts created by Snort.

Now the real work begins. You must learn how to understand the traffic on your own network so that you can filter the false alarms from the real ones. To start learning, have a look at the Reference and Security Links sections of the Shadow Web site. Then congratulate yourself for giving your organization a real chance to catch an intruder before he can do any damage.

 

Links

1. "Bastille" - http://www.bastille-linux.org/
2. "Snort" - http://www.snort.org/
3. "Shadow" - http://www.nswc.navy.mil/ISSEC/CID
4. "ACID" - http://www.andrew.cmu.edu/user/rdanyliw/snort/snortacid.html
5. "IDS Policy Manager" - http://www.activeworx.org/programs/idspm/index.htm
6. "tcpdump" - http://www.tcpdump.org/
7. "Shadow Web site" - http://www.nswc.navy.mil/

Recommended Links

---

In case of broken links please try to use Google search. If you find the page please notify us about new location

• Current Shadow software - version 1.8, released April 2003
   
• Shadow Screen Shots
   
• Current Shadow documentation (PDF format)
   
• Current Shadow documentation (HTML format)
   
• SHADOW articles of interest
   
• INTRUSION DETECTION: Shadow Style
   
• Whitehats.ca - Shadow Sensor Configuration Policy some not very coherent attempt to use shadow along with SGUIL - The Analyst Console for Network Security Monitoring.

SHADOW Technical Analysis

• Coordinated Attacks and Probes
• Stealth and coordinated powerpoint
• Stealth Coordinated Attacks

Recommended Papers

ShadowCon '00 Presentations

ShadowCon '99 Presentations

Proposed 1999 DARPA Off-line Intrusion Detection Evaluation Plans