Softpanorama
May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Architectural Issues of Intrusion Detection Infrastructure in Large Enterprises

Dr. Nikolai Bezroukov
(version 0.9)

Note: Earlier version was published in Softpanorama Bulletin, Vol. 17, No.3 (2005)

Introduction

NIDS and their problems

The problem of false positives

Webliography


The most serious problem in intrusion detection is the problem of distinguishing very weak useful signal in massive noise as well as related problem of data assessment: the need to correlate, evaluate and verify all this mass of junk events (sometimes called alerts ;-) that various IDS are generating. This is a difficult problem that is not solved well by most organizations so most IDS in "rich" network are of limited usefulness.

This problem of false positives is especially acute in Network IDS( NIDS). That's why many NIDS deployments actually have the status of "innocent fraud" to borrow the catch phrase used by famous economist John Kenneth Galbraith in the title of his last book "The Economics of Innocent Fraud".

In Gartner report "Hype Cycle for Information Security, 2003" published on 30 May 2003 Richard Stiennon, who was at this time, a VP of Research (6 years Gartner veteran at the time of publication) courageously stated that "king is naked" in just one short paragraph:

"Intrusion detection systems are a market failure. Vendors are now hyping intrusion prevention systems, which also have stalled. The functionality is moving into firewalls, which will perform deep packet inspection for content and malicious traffic blocking, as well as antivirus activities."

To be useful intrusion detection requires multi-level approach with different layers able to communicate using some kind of common protocol may be on the base of typical EMS system (for example Tivoli).

Introduction

Formally IDSs fall into two main groups: host-based and network-based:

NIDS and their problems

Classification notwithstanding when people are talking about IDS they usually mean NIDS. The latter operate at a Internet layer of TCP/IP protocol stack. And that means pretty low level -- level of fragmented datagrams. NIDS are trying to infer attacks against the network from traffic patterns as well as the content of the data stream (this involved attempts defragment datagrams as well as to reconstruct higher level protocols, for example HTTP).

Most NIDS are close relatives of virus scanners and are implemented as scanner of traffic with some (limited) reconstruction of higher level protocols. They share the same major problem. among them:

All-in all network IDSs are probably the most over-hyped and the least useful category of IDS. The return on investment on a typical signature based NIDS appliance in case of using generic signatures ("classic ISS appliances value proposition") is asymptotically close to zero.

For some unknown to me reason the whole industry became pretty rotten selling mostly hype and FUD. Still I need to admit that FUD sells well. The total size of the world market for network IDS is probably several hundred millions dollars and this market niche is occupied by a lot of snake oil salesmen:

Synergy Research Group reported that the worldwide network security market spending continued to be over the $1 billion in the fourth quarter of 2005, in all segments -- hybrid solutions (firewall/VPN, appliances, and hybrid software solutions), Intrusion Detection/Prevention Systems (IDS/IPS), and SSL VPN.

IDS/IPS sales increased seven percent for the quarter and were up 30 percent over 2004. Read article here.

Most money spent on IDS can be spent with much greater return on investment on host based detection and first of all on log analysis (which provides almost immediate return on investment), host based detection including integrity checking, ESM software as well as on improving rules in existing or installing additional firewalls. actually spending money of firewalls is more efficient then spending money on IDS and that fact was noted by Gartner in 2003.

In Gartner report "Hype Cycle for Information Security, 2003" published on 30 May 2003 Richard Stiennon, who was at this time, a VP of Research (6 years Gartner veteran at the time of publication) courageously stated that "king is naked" in just one short paragraph:

"Intrusion detection systems are a market failure. Vendors are now hyping intrusion prevention systems, which also have stalled. The functionality is moving into firewalls, which will perform deep packet inspection for content and malicious traffic blocking, as well as antivirus activities."

Here is the relevant part of the report.

Succumbing to vendor hype in the security management area can have expensive consequences. Enterprises should assess their security needs and evaluate the relative maturity of a security technology before adopting it.

... ... ...

4.6 Intrusion Detection Systems

Definition: Software running on a host or a network sensor that identifies malicious activity and creates an alert.

Time to Plateau/Adoption Speed: Obsolete before Plateau.

Justification for Hype Cycle Position/Adoption Speed: Intrusion detection systems are a market failure. Vendors are now hyping intrusion prevention systems, which also have stalled. The functionality is moving into firewalls, which will perform deep packet inspection for content and malicious traffic blocking, as well as antivirus activities.

Business Impact Areas: Security and network management.

Selected Vendors: Cisco, Enterasys Networks, Entercept, Internet Security Systems, Symantec and Tripwire.

Analysis by Richard Stiennon

And this analysis withstood the test of the time, despite the fact the Gartner changed its position to please influential subscribers. While NIDS are far from being completely useless paying money for them is a kind of spending that only large companies, and, especially, only powerful players in financial industry can afford. Still they are politically correct thing and if deployed with some thinking can provide some useful signal.

But that also means that network IDS area is a natural area where open source software is more competitive then any commercial software. Simplifying we can even state that the fact of acquisition of commercial IDS by any organization can be a sign or weak or incompetent management ( although reality is more complex and sometimes such an acquisition is just a reaction on pressures outside IT like compliances-related pressures; moreover some implementations were done under the premises of "loss leader" mentality under the motto "let those jerks who want it have this sucker" ).

Actually an organization that is spending money on NIDS without first creating a solid foundation implementing log analysis and deploying ESM commits what is called "innocent fraud" ;-). It does not matter what traffic you detect if you do not understand what exactly happening on your servers/workstations and view your traffic as an unstructured stream, a pond out of which IDS magically fish alerts.

In reality as most time IDS is crying wolf so often, that few useful alerts that they generate are buried in the noise. Also "real time" that is selling point of IDS does not really matter: most organization have no possibility to react promptly on alerts even if we assume that there are (very rare) cases when NIDS pick up useful signal instead on noise. And that means that hybrid appliances that provide also "blackbox/flight recorder" type of capabilities like Niksun appliances are more promising that ISS appliances that for some reason dominate the commercial segment. Sourcefire appliances are better then ISS as they are tunable but they lack "blackbox/flight recorder" capabilities

A good introduction to NIDS can be found at NIST Draft Special Publication 800-94, Guide to Intrusion Detection and Prevention (IDP) Systems (Adobe PDF (2,333 KB) Zipped PDF (1,844 KB) )

A typical network IDS (NIDS) uses network card(s) in promiscuous mode, sniffing all packets on each network segment the server is connected to. Installations usually consists of several sensors and a central console to aggregate and analyze data (for example Snort can be used as a sensor and Acid as central console). NIDS can be classified into several types:

The second important classification of NIDS is the placement:

Organizations rarely have the resources to investigate every "security" event. Instead, they must attempt to identify and address the top issues, using the tools they've been given. This is practically impossible if an IDS is listening to a large traffic stream with many different types of servers and protocols. In this case security personnel, if any, are being forced to practice triage: tackle the highest-impact problems first and move on from there. Eventually it is replaced with even more simple approach: ignore them all ;-). Of course much depends on how well signatures are tuned to particular network infrastructure. therefore another classification can be based on the type of signature used:

Even is case when you limit traffic to specific segment of the internal network (for example local sites in national or international corporation, which is probably the best NISD deployment strategy) the effectiveness of network IDS is low but definitely above zero. That can be marginally useful in this restricted environment. Moreover that might have value for network troubleshooting (especially if they also configured to act as a blackbox recorder for traffic; the latter can be easily done using TCPdump as the first stage and processing TCPdump results with Snort (say, each quarter of an hour) and then reprocessing alerts with Perl scripts. Snort stage is optional and Perl can be used directly as was done in Shadow. Please not that all those talks about real time detection are 99% is a pure security FUD. Nothing can be done in most large organizations in less then an hour ;-)

That's why many large enterprise customers (especially those who still staff that have some clue, despite all efforts spend on outsourcing) started to defect commercial IDS vendors approximately in 2003. See my IDS Whitepaper for details. In order to preserve their business (and revenue stream) IDS vendors started to hype intrusion prevention systems as the next generation of IDS. But IPS is a very questionable idea that mixes the role of firewall with the role of IDS sensor. It's not surprising that it backfired many times for early (and/or too enthusiastic) adopters (beta addicts).

It is very symptomatic and proves the point about "innocent fraud" that intrusion prevention usually is advertised on the base of its ability to detect mail viruses, network worms threats and Spyware. For any specialist it is evident that mail viruses actually should be detected on mail gateway and it is benign idiotism to try to detect then on the packet filter level. Still idiotism might be key to commercial success and most IDS vendors pay a lot of attention to the rules or signatures that provide positive PR and that automatically drives that into virus/worms detection wonderland. There are two very important points here:

May be things eventually improve, but right now I do not see how commercial IDS can justify the return on investment and NIDS looks like a perfect area for open source solutions. In this sense please consider this page a pretty naive (missing organizational dynamic and power grab issues in large organizations) attempt to counter "innocent fraud" to borrow the catch phrase used by famous economist John Kenneth Galbraith in the title of his last book "The Economics of Innocent Fraud".

Programmability of NIDS

Important criteria for NIDS is also the level of programmability:

It's rather difficult to place NISM in segments with large traffic. Mirroring port on the switches work in simple cases, but in complex cases where there are multiple virtual LANs that will not work as usually only one port can be mirrored. Also mirroring increase the load on the switch. Taps are additional component and are somewhat risky on high traffic segments unless they are designed to channel all the traffic in case of failure. Logically network IDS belongs to firewall and some commercial firewalls have rudimentary IDS functionality. Also personal firewall with NIDS component might be even be more attractive for most consumers as they provide some insight on what is happening. They also can be useful for troubleshooting. Their major market is small business and probably people connected by DSL or cable who fear that their home computers may be invaded by crackers.

Among open source network Intrusion Detection Systems (IDS) Snort is the most well developed and powerful solution. It covered in a separate page. But along with network-based intrusion detection, one probably should pay more attention to host-based IDS that uses log analysis and integrity checking. One should never put all eggs into one basket. The most popular integrity checker is Tripwire, but it's somewhat too primitive for the intrusion detection. See Softpanorama Integrity Checkers

The problem is that useful signal about probes on actual intrusions is usually buried under mountains of data and wrong signal may drive you in a wrong direction. A typical way to cope with information overload from network IDS is to rely more on the aggregation of data (for example, detect scans not single probes) and "anomaly detection" (imitate firewall detector or use statistical criteria for traffic aggregation). Misuse detection is more costly and more problematic that anomaly detection approach with the notable exception of honeypots. It might be beneficial to use a hybrid tools that combine honeypots and NIDS. Just as a sophisticated home security system might comprise both external cameras and sensors and internal monitoring equipment to watch for suspicious activity both outside and within the house - so should an intrusion detection system.

You may not know it, but a surprisingly large number of IDS vendors have license provisions that can prohibit you from communicating information about the quality and usability of their security software. Some vendors have used these software license provisions to file or threaten lawsuits to silence users who criticized software quality in places such as Web sites, Usenet newsgroups, user group bulletin boards, and the technical support boards maintained by software vendors themselves. Here open source has a definite advantage, because it may be not the best but at least it is open, has a reasonable quality (for example Snort is very competitive with most popular commercial solutions) or at least it is the cheapest alternative among several equally bad choices ;-).

IDS often (and wrongly) are considered to be the key component for the enterprise-level security. Often that is achieved by buying fashionable but mainly useless outsourced IDS services. Generally this idea has a questionable value proposition because of the level of false positives and problems with the internal infrastructure (often stupid misconfigurations on WEB server level, inability to apply patches in a timely manner, etc.) that far outweigh and IDS-inspired capabilities. If you are buying IDS, the good staring point is to ask to show what attacks they recently detected and negotiate one to six month trial before you pay the money ("try before you buy").

The problem of false positives

The problem of false positives for IDS is a very important problem that is rarely discussed on a sound technological level. I don't think there is a 'best' IDS. But here are some considerations:

You probably got the idea at this point: the IQ of the network/security administrators and the ability to adapt the solution to this organization is of primary importance in the IDS area, more important then in, say, virus protection (where precooked signatures sets rules despite being a "retrospective" tool, which in principle can't catch new strains as they are not in database; heuristics can slightly alleviate this problem, but without integrity checker component and pretty restrictive firewall rules, all this corporate activity is more about "waving dead chicken" then providing high level of security). That's why open source solution and commercial solution that permit signature tuning are vastly superior to alternatives. From this point of view ISS simply does not stand a change to compete.

All-in-all the architecture and the level of customarization of the rulebase are more important then the capabilities of the NIDS.

Dr. Nikolai Bezroukov

Webliography




Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2014 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting hosting of this site with different providers to distribute and speed up access. Currently there are two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.

Disclaimer:

The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Created: May 16, 2003; Last modified: July 18, 2014