Softpanorama
May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Shadow IDS

News A Slightly Skeptical View on Snort Network-level Intrusion Detection Recommended Books Recommended Links  Recommended Papers
Snort   Log analyzers History Humor Etc

SHADOW (Secondary Heuristic Analysis for Defensive Online Warfare) is a very simple combination of tcpdump and Perl that actually works as a rudimentary NIDS but can made more powerful by using snort and/or custom Perl processing scripts. Due to flexibility and power of Perl the architecture used by Shadow remains one of the best NIDS architectures and in my opinion is architecturally superior to Snort.  

The original Shadow code was developed by the US Navy as the result of the "Cooperative Intrusion Detection Evaluation and Response" (CIDER) project. Currently SHADOW codebase is unmaintained abandonware and the code base is largely obsolete. The latest published version (2003) was developed by NSWC.

Solaris is an excellent platform to run any NIDS and Shadow is not an exception. But most steps actually are OS independent. 

Shadow basically consists of two parts that can run on different boxes (although it is not strictly necessary):

While the idea is transparent and very good, the code supplied is not very transparent and is of limited usefulness for any knowledgeable Perl programmer. There are things that definitely can be simplified.

The key files are:

cleanup.pl
condense.pl
fetchem.pl
find_scan.pl
obfuscate.pl

There is also pretty convoluted file (sensor_driver.pl) to launch tcpdump and it probably can be discarded and replaced with a simple ksh script. 

Actually you need is the following simple loop that can be invoked from cron each selected sensor using preselected interval, for example 15 min:

1. Write file with tcpdump to the archive (possibly compress it if you nee to work with huge pipe).

2. Optionally transfer it to the "analysis station".

3. Reprocess this file with TCPdump working in "listing mode", feeding the output to Perl analyzer that deciphers TCP dump format and produced actual alerts.  Or better use Snort snort instead of TCPdump for doing this and reprocessed generated alerts with Perl (using Snort database format)


Top updates

Bulletin Latest Past week Past month
Google Search


Old News ;-)

[Apr 27, 2005] NewsForge Build an IDS with Snort, Shadow, and ACID by Mark Bain.

It looks like the author made it all wrong.
Title   Build an IDS with Snort, Shadow, and ACID
Date   2005.04.27 11:00
Author   Mark Bain
Topic    
http://software.newsforge.com/article.pl?sid=05/04/14/1535223

Every organization understands the importance of using a firewall to protect its assets. But what happens if someone finds a hole in the wall? What if the infiltrator is actually someone from within your organization who wants to access information that he shouldn't? To cover those contingencies, you need an intrusion detection system (IDS) to complement your firewall. Fortunately, with a minimum amount of time and money you can set up an IDS with open source tools such as Snort, Shadow, and ACID.

An IDS comprises sensors to do the actual monitoring and a central console to store and analyse the data collected on each sensor. A sensor can be a basic PC with an extra network card (to do the actual monitoring), whilst the central console should have a bit more processing power and a larger disk.

If Linux isn't already on the PCs you plan to use then you'll have to install it. You'll need only a minimal install because these boxes will be used only for the IDS. I recommend installing Bastille to make the IDS more secure. Another key thing to do is to put the monitoring network card on the sensor without an IP address; this makes it effectively invisible.

You can now start installing the software to do the actual intrusion detection. You can use Snort to monitor patterns of behaviour (or signatures) and Shadow to monitor packet headers. You'll need ACID to analyse the data collected by Snort (Shadow has its own analyser).

Let's begin with the sensor. The first piece of software to install is the client and develop RPMs for MySQL drivers, which are necessary because Snort store its output in a database on the central console. Once these drivers are in place, download the tar file for Snort, decompress it, and install it with the commands:

./configure --with-mysql
make
make install

The installation and setup instructions for Snort are clear and concise. Snort uses a set of rules to identify signatures (or patterns of behaviour) to identify illegal activity. You can download a default set of customisable rules from Snort's site, along with some excellent documentation. If you have a Windows client, you can manage the signatures using Activeworx's IDS Policy Manager.

Shadow

With Snort on the workstation, the next step is to install Shadow. The software comprises a set of Perl scripts that use tcpdump to collect data from packet headers on the network in order to identify any unusual activity. Shadow's Web site contains documentation that covers package dependencies (including Perl and tcpdump; the latter is supplied with Shadow), installation, and setup. To configure Shadow you need to modify three files -- sensor_init.sh, std.ph, and std.filter -- to reflect your local setup, such as the location of tcpdump. You also need to set up crontab to run sensor_driver.pl (the main script of the sensor) once an hour. The package supplies a suitable cron file -- /usr/local/SHADOW/sensor/sensor_crontab -- that also will synchronize the sensor time with a reliable Network Time Protocol source.

Finally, you must make sure that Shadow runs at system startup time by copying /usr/local/SHADOW/sensor_init.sh to your /etc/rc.d directory. Once Shadow is running, your sensor PC is ready to be connected to the network.

Shadow comes with its own analyser, which requires tcpdump, OpenSSH, and Apache. The Shadow documentation will lead you through the configuration process, tell you how to set up the Shadow CGI scripts, and explain how to set up crontab for automatic operation.

The console

With the sensor set up you can turn to the central console, where the actual analysis of the data will be done.

Snort has no analysis software itself -- it simply generates data on the sensor and then stores it in a MySQL database on the central console. You can use ACID (Analysis Console for Intrusion Databases) to analyse the data. ACID's Web site gives full details on software installation and the creation of the MySQL database for storing the Snort alerts.

At this point your IDS should be up and running with Snort, Shadow, and ACID. On the sensor Snort will be monitoring the traffic patterns and raising alerts to the database. Shadow will be reading packet headers and creating log files. On the central console you will be able to view the Web pages created by Shadow, and to use ACID to analyse the alerts created by Snort.

Now the real work begins. You must learn how to understand the traffic on your own network so that you can filter the false alarms from the real ones. To start learning, have a look at the Reference and Security Links sections of the Shadow Web site. Then congratulate yourself for giving your organization a real chance to catch an intruder before he can do any damage.

 

Links

  1. "Bastille" - http://www.bastille-linux.org/
  2. "Snort" - http://www.snort.org/
  3. "Shadow" - http://www.nswc.navy.mil/ISSEC/CID
  4. "ACID" - http://www.andrew.cmu.edu/user/rdanyliw/snort/snortacid.html
  5. "IDS Policy Manager" - http://www.activeworx.org/programs/idspm/index.htm
  6. "tcpdump" - http://www.tcpdump.org/
  7. "Shadow Web site" - http://www.nswc.navy.mil/

Recommended Links

Softpanorama Top Visited

Softpanorama Recommended

SHADOW Technical Analysis

 

Recommended Papers

Malik ngrep Scripts Scripts for use with Shadow IDS - using ngrep and regular expressions

ShadowCon '00 Presentations

ShadowCon '99 Presentations

Proposed 1999 DARPA Off-line Intrusion Detection Evaluation Plans




Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2014 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting hosting of this site with different providers to distribute and speed up access. Currently there are two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.

Disclaimer:

The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last updated: July 07, 2013