Chapter 1
System and Network Attacks
1.1 Introduction:
Systems and networks are subject to electronic attacks. The increasingly frequent attacks on Internet- visible systems are attempts to breach information security requirements for protection of data. Vulnerability-assessment tools check systems and networks for system problems and configuration errors that represent security vulnerabilities. Intrusion-detection systems collect information from a variety of vantage points within computer systems and networks and analyze this information for symptoms of security breaches. Both intrusion-detection and vulnerability-assessment technologies allow organizations to protect themselves from losses associated with network security problems. The market for intrusion-detection products, driven by reports of steadily increasing computer security breaches, has grown from $40 million in 1997 to $100 million in 1998. Intrusion-detection is the logical complement to network firewalls, extending the security management capabilities of system administrators to include security audit, monitoring, attack recognition, and response.
1.2 Intrusion detection systems perform a variety of functions:
• Monitoring and analysis of user and system activity
• Auditing of system configurations and vulnerabilities
• Assessing the integrity of critical system and data files • Recognition of activity patterns reflecting known attacks
• Statistical analysis for abnormal activity patterns
• Operating-system audit-trail management, with recognition of user activity reflecting policy violations Benefits of intrusion-detection and vulnerability-assessment products include the following
• Improving integrity of other parts of the information security infrastructure
Fig 1.1 Geographic Distribution of Attack Sources.
• Improved system monitoring .Tracing user activity from the point of entry to point of exit or impact
• Recognizing and reporting alterations to data files
• Spotting errors of system configuration and sometimes correcting them
• Recognizing specific types of attack and alerting appropriate staff for defensive responses
• Keeping system management personnel up to date on recent corrections to programs
• Allowing non-expert staff to contribute to system security
• Providing guidelines in establishing information-security policies Unrealistic expectations about intrusion-detection and vulnerability assessment products must be corrected: these products are not silver bullets and they
• cannot compensate for weak identification and authentication mechanisms
• cannot conduct investigations of attacks without human intervention
• cannot intuit the contents of your organizational security policy
• cannot compensate for weaknesses in network protocols
• cannot compensate for problems in the quality or integrity of information the system provides
• cannot analyze all of the traffic on a busy network
• cannot always deal with problems involving packet-level attacks
• cannot deal with modern network hardware and features
1.3 Definitions:
1.3.1 Network Security
is the property of computer systems and networks that specifies that the systems
in question and their elements can be trusted to act as expected in safeguarding their owners’ and users’ information. The goals of security include confidentiality (ensuring only authorized users can read or copy a given file or object), control (only authorized users can decide when to allow access to information), integrity (only authorized users can alter or delete a given file or object), authenticity (correctness of attribution or description), availability (no unauthorized user can deny authorized users timely access to files or other system resources), and utility (fitness for a specified purpose).
1.3.2 Intrusion Detection
systems collect information from a variety of system and network sources, then analyze the information for signs of intrusion (attacks coming from outside the organization) and misuse (attacks originating inside the organization).
1.3.3 Vulnerability Assessment (scanners)
performs rigorous examinations of systems in order to locate problems that represent security vulnerabilities.
1.3.4 Security vulnerabilities
are features or errors in system software or configuration that increase the likelihood of damage from attackers, accidents or errors.
1.3.5 Security Policy
is the statement of an organization’s posture towards security. It states what an organization considers to be valuable, and specifies how the things of value are to be protected. In practical use, security policies are coarse grained (i.e., generalized statements that apply to the organization as a whole) and drive finer-grained procedures, guidelines, and practices, which specify how the policy is to be implemented at group, office, net- work, and system, and user levels.
Chapter 2
Introduction to Intrusion Detection Systems
2.1 What is intrusion detection?
Intrusion Detection is the art of detecting inappropriate, incorrect, or anomalous activity. An intrusion can include a network attack from the outside, or one that originates from the internal network (generally referred to as misuse). Intrusion Detection Systems (IDSs) help computer systems prepare for and deal with these types of attacks. They collect information from several points within computer systems and network sources, then analyze this information for signs of intrusion and misuse.
2.2 Who are the intruders?
Intruders are unauthorized users, and can be classified as follows:
· External - users not authorized to use the system, generally called intruders. External intruders are the focus of physical security and firewalls, for instance.
· Internal - users not authorized to use some resources, usually referred to a misuse
o masquerades - impersonate other users
o clandestine - evade auditing, and are a threat to weak operating systems and badly managed systems
· Misfeasors - include those users who misuse their privileges
So, as you can see, intrusion detection systems must protect the integrity of the system from several different angles and several different people. All users are possible threats, regardless of their origin or how they were authenticated.
2.3 How do people gain access to a system to issue an attack?
Whether the attack is classified as an intrusion or as misuse, there are three primary ways intruders can gain access to a computer system:
1. Physical Intrusion - an intruder has physical access to a machine and can either use the keyboard or actually take apart the system
2. System Intrusion - the intruder has a low-privileged user account on the system, and may be able to use a known exploit to gain additional administrative privileges
3. Remote Intrusion - an attempt to penetrate a system remotely across the network
Intruders are able to gain access to a system because of several reasons. Some of these reasons include:
· software bugs which are exploited in the server daemons, the client applications, the operating system and the network stack. Bugs are commonly classified into buffer overflows, unhandled input, unexpected combinations of commands, and race conditions where two programs need to access the same data at the same time.
· system configurations include default configurations which are easy to use and therefore easy to break in, and lazy administrators who don't configure a root password
· password cracking due to really weak passwords, dictionary attacks, and brute force attacks
· design flaws including TCP/IP protocol flaws such as IP Spoofing and SYN Floods, as well as Unix design flaws
2.4 What types of intrusion detection are there?
Intrusion detection uses various techniques to trace the unauthorized use of resources. These techniques include the analysis of audit trail data and network traffic, either real-time or off-line (after-the-fact). Since the goal of intrusion detection is to catch the intruder in the act, real-time testing of the audit data is the most prevalent. More information is available on specific intrusion detection techniques in Methodologies(chapter 3).
2.5 How much danger from intrusion is there?
If you've ever paid for anything on-line with a credit card, if you have any
financial records or your social security number held on your personal computer, you should be concerned about someone finding this information. Everyone has something they wish to keep private from strangers accessing your computer or network.
Legal liability is the issue here. If any damages were caused by a hacker gaining access to your machine, you may be held liable for those damages. You must be able to prove to a court that you took "reasonable" measures to defend yourself from hackers. For example, consider if you put a machine on a fast link (cable modem or DSL) and left administrator/root accounts open with no password. Then if a hacker breaks into that machine, then uses that machine to break into a bank, you may be held liable because you did not take the most obvious measures in securing the machine.
For information regarding ethical issues and intrusion detection, see Benefits and Drawbacks.
2.6 Are IDSs similar to firewalls?
Intrusion Detection is actually considered to be a complement to network firewalls, as they extend the security management capabilities of system administrators to include things such as:
· monitoring and analysis of user and system activity
· auditing of system configuration and vulnerabilities
· assessing the integrity of critical system files and data files
· recognizing patterns of activity that reflect known attacks
· statistical analysis for abnormal activity patterns
· operating system audit trail management, with recognition of user activity that violates company policy
Chapter 3
Intrusion Detection System Methodologies
Intrusion Detection Systems monitor network traffic for unauthorized use. These system generate alarms in the form of console messages, e-mail messages and pages, just to name a few.
3.1 Basic Premise of IDSs
IDSs use signature analysis and statistical profiling to detect unauthorized use.
· Signature analysis matches network traffic against known rules containing known attack traces and protocol uses. If a match is detected, the traffic is flagged. Implementation issues involve the maintenance of common attack databases, protocol usage and writing efficient algorithms to match the traffic with the rules.
· Statistical profiling is generally performed on the host-based intrusion detection systems. It monitors the characteristics of the users using the system, developing sophisticated profiles over time. Characteristics of the users include application, amount of data, time of usage, protocols used, source and destination addresses, etc. Once a profile is completed, subsequent uses by the user are compared to the original profile. If the system detects a change in the users activities, the user is flagged by the system. For example, a user that generally edits documents in MSWord suddenly opens vi to edit a remote host password file should be flagged. Statistical profiling can also be performed at the network level by developing the activity profiles of web servers, for example. If a web server starts to receive remote commands and file uploads, those are not normal processes and are thus flagged.
3.2 Physical Implementation of IDSs
Intrusion Detection Systems are generally implemented in one of two ways. These implementations include host-based and network-based intrusion detection.
3.2.1 Host-Based Intrusion Detection: Detection software is loaded directly on to the computer it will be monitoring, and the data is audited from a single host. Each computer on the system will need to have the intrusion detection software running in order to be effective in identifying an attack. Host-based IDSs typically monitor system, event and security logs on Windows NT and syslog in Unix environments. If any changes in these files are detected, the IDS compares the new log entry with attack signatures to see if there is a match. If a match is found, the system will alert the administrator as well as perform other calls to action. There are two classes of host-based intrusion detection software:
o host wrappers/personal firewalls - these are configured to look at all network packets, connection attempts or login attempts to the monitored machine. Personal firewalls can detect software on the host that may be trying to connect to the network as well. Examples of wrapper packages include TCPWrappers for Unix and NukeNabber.
o agent-based software - an agent is able to monitor accesses, changes to critical system files and changes in user privilege. Agent software includes System Integrity Verifiers and Log File Monitors, which are considered to be "tools" to be combined with other network security for maximum protection. Examples of agents include Cyber Safe and Tripwire.
o For smaller networks, a host-based IDS is the most cost-effective.
o Pros:
§ cost-effective for small numbers of hosts
§ unlikely to miss activity due to high traffic loads
§ probably won't require dedicated hardware
o Cons:
§ requires software for each host
§ somewhat vulnerable to an attack
§ costs are proportional to the number of hosts
uses the host's resources
Vendors include:
o Intrusion Detection
o Axent Technologies
o Trusted Information Systems
3.2.2 Network-Based Intrusion Detection (NIDS): Packets on the network and audit data from several hosts are monitored on the particular segment the NIDS is covering. As the packets pass the sensor, they are examined for their similarities to a signature. Network Intrusion Detection Systems are primarily concerned with remote intrusion from an external source outside of the network. If an attack is detected, the NIDS will notify the administrator, terminate the connection and/or record the session for forensic analysis and evidence collection.
Network-based IDS's are highly recommended.
o Pros:
§ protect every device on the network
§ detects problems quickly
§ not vulnerable to attack
§ one flat cost
o Cons:
§ expensive
§ can't monitor individual files on a specific host
§ dedicated hardware needed
§ high network load may result in missed traffic
Vendors include:
o Internet Security Systems
o Network Associates
o Cisco Systems
Two other implementations of intrusion detection systems include System Integrity Verifiers and Log File Monitors, although these are viewed as "additions" or "tools" to be used by the intrusion detection system.
3.2.3 System Integrity Verifiers (SIV):
monitor system files to find when a intruder has changed them (thereby leaving behind a backdoor).
may watch the Windows registry and chron configuration, in order to find well known signatures.
may detect when a normal user somehow acquires root/administrator level privileges.
Example: Tripwire, COPS, SmartWatch, Tiger
3.2.4 Log File Monitors (LFM):
monitor log files generated by network services look for patterns in the log files that suggest an intruder is attacking Example: swatch
3.4 Signatures:
Signatures represent a pattern of activity required to gain access to a computer network or system. They allow the intrusion detection system to verify if the sequence of events it is monitoring are a threat to the system's integrity. The following signatures exist:
· Attack signatures (a.k.a. string signatures): look for a specific, well known pattern of activity logged by the system that indicates malicious or suspicious intent. The most common attack signatures include:
o reconnaissance - These include ping sweeps, DNS zone transfers, e-mail recons, TCP or UDP port scans, and possibly indexing of public web servers to find cgi holes.
Common reconnaissance attacks include:
§
§ TCP scans - probes for open (listening) TCP ports
§ UDP scans - sending garbage UDP packets to the desired port, which actually reveals an open port
§ OS identification - sending illegal or strange ICMP or TCP packets can identify the OS by how it responds to the packets
§ Account scans - look for accounts with no passwords, passwords that are the same as the user name, etc.
o exploits - Intruders will take advantage of hidden features, holes or bugs to gain access to the system. Common exploits include:
§ CGI scripts - For example, the string pattern "/cgi-bin/phf?" might indicate someone is attempting to access this vulnerable CGI script on a web-server.
§ Web server attacks
§ Web browser attacks
§ SMTP attacks
§ IP Spoofing
§ Buffer overflows
§ DNS attacks
o denial-of-service (DoS) attacks - Where the intruder attempts to crash a service (or the machine), overload network links, overloaded the CPU, or fill up the disk. The intruder is not trying to gain information, but to simply act as a vandal to prevent you from making use of your machine.
Common DoS attacks include:
§
§ SYN Flood
§ WinNuke
· Port signatures: look for connection attempts to well known, frequently attacked ports
· Header condition signatures: look for illogical or dangerous combinations in packet headers
3.5 Solutions
"Defense in depth, and overkill paranoia, are your friends." -- Bennett Todd
3.5.1 General Security Suggestions
· Firewalls - Firewalls are generally considered to be a network's first line of defense. But consider the misuse of the system by those already inside the firewall, such as employees or hackers with physical access. Here are a few important distinctions:
o firewalls protect external access, and leave the network unprotected internally
o firewalls are unable to recognize an attack
o firewalls, by default, stop all forms of communication, then learn rules to know which network connections to permit/dismiss
o firewalls are not a dynamic defense system like IDSs are, and are most effective when used in conjunction with IDSs
Adding an IDS to your firewall can:
o double-check misconfigured firewalls
o catch attacks that firewalls legitimate allow through (such as attacks against web servers)
o catch attempts to enter the system that fail
o catch insider hacking (misuse)
· Authentication - scanners to find open accounts, password policies, integration of password systems, etc
· VPNs (Virtual Private Networks) create a secure connection over the Internet for remote access (e.g. for telecommuters). Encryption - e-mail encryption (PGP, SMIME), file encryption (PGP again), or file system encryption (BestCrypt, PGP again)
· lures/honeypots - These are programs that pretend to be a service, but do not advertise themselves. It can be something as simple as one of the many BackOrifice emulators (such as NFR's Back Officer Friendly), or as complex as an entire subnet of bogus systems installed for that purpose.
3.6 Benefits
3.6.1 Benefits of IDS Implementation
There are several benefits to having a well-maintained intrusion detection system. By their very nature, IDSs are quite efficient at keeping unwanted users out of a system.
The benefits of having an IDS include:
· IDSs provide a level of protection beyond the firewall by protecting the network from internal and external attacks and threats
·
·
·
·
·
·
· IDSs dramatically improve the security of a network
· IDSs enhance perimeter firewall protection by taking appropriate action on packets and flows that violate the security policy or represent malicious network activity
· IDSs provide additional visibility at intranet, extranet, and branch-office Internet perimeters
· IDSs provide robust protection against attacks on the network and allow the network administrator to automatically respond to threats from internal or external hosts
3.7 Drawbacks and Ethical Concerns
3.7.1 Drawbacks of IDS Implementation
Considering the following drawbacks, companies must be able to weigh the benefits with the costs. The question the hacker asks is, Why bother breaking into a well-guarded system of a large corporation when you can achieve the same result by robbing a few dozen little guys? IDSs can help turn a network into a well-guarded system when properly combined with other forms of network security.
· IDSs are not silver bullets, as they are only one part of the security infrastructure. Primary systems such as firewalls, encryption, and authentication should be implemented in conjunction with the IDS.
· None of the IDS solutions available offer you complete protection.
· Careful hackers have been known to be able to evade or disable the IDS.
· IDSs are difficult to implement on a switched network. It is difficult to decide where to plug in an IDS sensor so that it catches a sufficient amount of network traffic.
· If an IDS sits at a centralized location on the network, it must be able to keep up with, analyze and store information from several machines. If sensors are distributed, however, each IDS can analyze its own log files and any attack signatures can be correlated by the primary IDS.
· Depending on the network traffic load, the number of packets per second the IDS can handle is important to prevent a bottleneck.
· Sometimes normal network traffic can cause many false positives (cry wolf).
· IDSs can be attacked in the following ways:
o blinding the sensor - hackers can send so many packets across the portion of the network the sensor is monitoring, causing it to drop packets it does not have time to process. High traffic rated can actually shut down the sensor.
o blinding the event storage - some port scanning tools contain "decoy" scans that use spoofed source IP addresses, as well as the real IP address of the hacker. These abundant spoofed IPs make it difficult for the IDS to identify which IPs are real and which are decoys.
o Denial of Service (DoS) Attack - an IDS is susceptible to SYN Floods and Smurf Attacks. The numerous protocols IDSs analyze leave them open to outright crashes when unexpected traffic is seen.
Intrusion detection is not the only benefit of deploying a good IDS. What comes with it is the ability to perform vulnerability assessments of your systems and close the security breaches before they are exploited, a greater degree of integrity to the rest of your security system, and tools for gathering information for building a sound security policy (http://www.technologynews.net).
3.7.2 Ethical Considerations and Legal Implications
Two primary areas of concern are evident when considering intrusion detection and the protection of private information on a network:
1. The auditing performed by the IDS on the internal network - employee actions are recorded and watched, extensive user profiles are developed.
2. The monitoring of the external network with respect to the legalities of hacking and cracking.
3.8 Audits and Activity Monitoring
IDS are capable of watching and logging all activity across the network. Some organizations take the position that all information systems are subject to arbitrary monitoring at any time. This viewpoint has ethical implications that may be detrimental to the livelihood of the organization's employees. A common argument in favor of security monitoring is that it is there to protect it's people.
Monitoring employee activity is a complex area to justify, as one must consider both the organization's integrity as well as the staff. The desire to protect both parties can often result in conflict. Legal, regulatory, policy and organizational considerations are usually very complex.
The log files and user profiles generated by the IDS contain a lot of information about the people and systems they represent. Personal habits and actions are recorded, analyzed, and stored for future reference. The actions that are stored are fragments of a personality, a concept which threatens traditional notions of privacy. We often don't realize when and what machines of surveillance and data gathering are asking. Nor do we realize when and how we are telling them about ourselves. That is, we don't always have the control over our self-projections that we should have. These issues can allow a conclusion that IDSs have the potential to be intrusive themselves.
What if someone were to gain access to the log files an analysis reports? Who watches the watcher?
3.9 Hacking and Cracking
There are several classifications of hackers, and all of them pose a potential threat to computer systems. The type that cause the most damage are the serious attackers and electronic criminals. They have the technical skill to plan and execute complex attacks, and know how to hide their trails during and after execution. Often, they will hack into high-profile targets for political, social or financial reasons. Another small but growing segment of the cracker community is espionage agents who exploit systems to obtain sensitive business information. Frequently, these individuals are former members of the international intelligence community, or work in illegal areas of competitor intelligence. They may work for a specific company or offer their services as independent agents.
IDSs are implemented as a line of defense against these types of hackers. They help to protect the integrity of the network and keep private information away from unauthorized users.
3.10 Example of intrusion detection system
A simple example of an intrusion detection system that adheres to the AAFID (autonomous agents for intrusion detection) architecture is shown in Fig. 3(a). This figure shows the four components of the architecture: agents, filters, transceivers and monitors. We refer to each one of these components as AAFID entities or simply entities, and to the whole intrusion detection system constituted by them as an AAFID system. An AAFID system can be distributed over any number of hosts in a network. Each host can contain any number of agents that monitor for interesting events occurring in the host. Agents may use filters to obtain data in a system-independent manner. All the agents in a host report their findings to a single transceiver. Transceivers are per-host entities that oversee the operation of all the agents running in their host. They have the ability to start, stop and send configuration commands to agents. They may also perform data reduction on the data received from the agents. The transceivers report their results to one or more monitors. Each monitor oversees the operation of several transceivers. Monitors have access to network wide data, therefore they are able to perform higher-level correlation and detect intrusions that involve several hosts. Monitors can be organized in a hierarchical fashion such that a monitor may in turn report to a higher-level monitor. Also, a transceiver may report to more than one monitor to provide redundancy and resistance to the failure of one of the monitors. Ultimately, a monitor is responsible for providing information and getting control commands from a user interface. Fig. 3(b) shows the logical organization corresponding to the physical distribution depicted in Fig. 3(a).
3.10.1 Components of the architecture
· Agents
An agent is an independently-running entity that monitors certain aspects of a host, and reports to the appropriate transceiver. For example, an agent could be looking for a large
number of telnet connections to a protected host, and consider their occurrence as spicious.
The agent would generate a report that is sent to the appropriate transceiver.
The agent does not have the authority to directly generate an alarm. Usually, a transceiver or a monitor will generate an alarm for the user based on information received from agents. By combining the reports from different agents, transceivers build a picture of the status of their host, and monitors build a picture of the status of the network they are monitoring.
Agents do not communicate directly with each other in the AAFID architecture. Instead, they send all their messages to the transceiver. The transceiver decides what to do with the information based on agent configuration information. The architecture does not specify any requirements or limitations for the functionality of an agent. It may be a simple program that monitors a specific event (for example, counting the number of telnet connections within the last 5 min, which is an existing agent in the current AAFID
implementation), or a complex software system (for example, an instance of IDIOT [7] looking for a set of local intrusion patterns). As long as the agent produces its output in the appropriate format and sends it to the transceiver, it can be part of the AAFID system.
Agents may perform any functions they need. Some possibilities (which have not been used by any existing AAFID agents) are:
· Agents may evolve over time using genetic programming techniques.
· Agents may employ techniques to retain state between sessions, allowing them to detect
long term attacks or changes in behavior. Currently, the architecture does not specify any mechanisms for maintaining persistent state.
· Agents could migrate from host to host by combining the AAFID architecture with some existing mobile-agent architecture.
Agents can be written in any programming language.
Some functionality (e.g., reporting, communication and synchronization mechanisms) is
Fig. 3. Physical and logical representations of a sample intrusion detection system that follows the AAFID architecture (called an AAFID system). (a) Physical layout of the components in a sample AAFID system, showing agents, filters, transceivers and monitors, as well as the communication and control channels between them. (b) Logical organization of the same AAFID showing the communication hierarchy of the components. The bi-directional arrows represent both the control and data flow between the entities. Notice that the logical organization is independent of the physical location of the entities in the hosts.
common to all the agents, and can be provided through shared libraries or similar mechanisms.
Thus, a framework implementation can provide most of the tools and mechanisms necessary to make writing new agents a relatively simple task.
· Filters
Filters are intended to be both a data selection and a data abstraction layer for agents. In the original AAFID architecture, each agent was responsible for obtaining the data it needed.
When the first prototype was implemented, this approach showed the following problems:
· On a single system, there may be more than one agent that needs data from the same data source. This is common in Unix with multifunction log files (such as /var/adm/messages). Having each agent read the data on its own meant duplicating the work of reading the file, parsing it and discarding unnecessary records.
· There may be agents that can provide a useful function under different versions of Unix, or even under different architectures (such as Windows NT). However, the data needed by the agent may be located in different places in each system and may be stored in different formats. This meant having to write a different agent for each system, that knows where to find the data and how to read it. Both of these problems are solved through the introduction of Filters. Filters provide a subscription-based service to agents, and have two functions. Data selection: There exists only one Alter per data source, and multiple agents can subscribe to it. When an agent subscribes to a Alter, it specifies which records it needs (using some criteria like regular expressions), and the filter only sends to the agent records that match the given criteria. This eliminates duplicate work in reading and filtering data. Data abstraction layer: Filters implement all the architecture- and system-dependent mechanisms for obtaining the data that agents need. Therefore, the same agent can run under different architectures simply by connecting to the appropriate Filter. This makes it easier to reuse code and to run AAFID under different operating systems.
· Transceivers
Transceivers are the external communications interface of each host. They have two roles: control and data processing. For a host to be monitored by an AAFID system, there must be a transceiver running on that host. In its control role, a transceiver performs the following functions:
· Keeps track and controls execution of agents in its host. The instructions to start and stop agents can come from configuration information, from a monitor, or as a response to specific events (for example, a report from one agent may trigger the activation of other agents to perform a more detailed monitoring of the host).
· Responds to commands issued by its monitor by providing the appropriate information or performing the requested actions. In its data processing role, a transceiver has the
following duties:
· Receives reports generated by the agents running in its host.
· Does appropriate processing on the information received from agents.
· Distributes the information received from the agents, or the results of processing it, either to other agents or to a monitor, as appropriate.
· Monitors
Monitors are the highest-level entities in the AAFID architecture. They have control and data processing roles that are similar to those of the transceivers. The main difference is that monitors can control entities that are running in several different hosts whereas transceivers only control local agents. In their data processing role, monitors receive information from all the transceivers they control, and can do higher-level correlations and detect events that involve several different hosts. Monitors have the capability to detect events that may be unnoticed by the transceivers.
In their control role, monitors can receive instructions from other monitors and they can
control transceivers and other monitors. Monitors have the ability to communicate with a user interface and provide the access point for the whole AAFID system. Monitors implement an interface that includes mechanisms for accessing the information that the monitor has, for providing commands to the monitor, or to send commands to lower-level entities such as transceivers and agents. If two monitors control the same transceiver, mechanisms have to be employed to ensure consistency of information and behavior. The AAFID architecture does not currently specify the mechanisms for achieving this consistency.
· User interfaces
The most complex and feature-full intrusion detection system can be useless if it does not have good mechanisms for users to interact with it. The AAFID architecture clearly separates the user interface from the data collection and processing elements. A user interface has to interact with a monitor to request information and to provide instructions. This separation allows different user interface implementations to be used with an AAFID system. For example, a graphical user interface (GUI) could be used to provide interactive access to the intrusion detection system, while a command- line based interface could be used in scripts to automate some maintenance and reporting functions.
· Communication mechanisms
The transmission of messages between entities is a central part of the functionality of an AAFID system. Although the AAFID architecture does not specify which communication mechanisms have to be used, we consider the following to be some important points about the communication mechanisms used in an AAFID system:
· Appropriate mechanisms should be used for different communication needs. In particular, communication within a host may be established by different means than communication across the network.
· The communication mechanisms should be efficient and reliable in the sense that they should (a) not add significantly to the communications load imposed by regular host activities, and (b) provide reasonable expectations of messages getting to their destination quickly and without alterations.
· The communication mechanisms should be secure in the sense that they should (a) be resistant to attempts of rendering it unusable by flooding or overloading, and (b) provide some kind of authentication and confidentiality mechanism. The topics of secure communications, secure distributed computation and security in autonomous agents have been already studied , and possibly some previous work can be used in AAFID implementations to obtain communication channels that provide the necessary characteristics.
Chapter 4
Multidimensional Network Monitoring
for Intrusion Detection
An approach for real-time network monitoring in terms of numerical time dependant functions of protocol parameters is suggested. Applying complex systems
theory for information flow analysis of networks, the information traffic is described
as a trajectory in multi-dimensional parameter-time space with about 10-12 dimensions.
The network traffic description is synthesized by applying methods of theoretical
physics and complex systems theory, to provide a robust approach for network
monitoring that detects known intrusions, and supports developing real systems for
detection of unknown intrusions. The methods of data analysis and pattern recognition
presented are the basis of a technology study for an automatic intrusion detection
system that detects the attack in the reconnaissance stage.
4.1 Introduction
Understanding the behavior of an information network and describing its main
features are very important for information exchange protection on computerized
information systems. Existing approaches for the study of network attack
tolerance usually include the study of the dependence of network stability on
network complexity and topology signature-based analysis technique; and statistical analysis and modeling of network Traffic . Recently, methods to study spatial
traffic flows and correlation functions of irregular sequences of numbers occurring
in the operation of computer networks have been proposed.
Herein we discuss properties related to information exchange on the network
rather than network structure and topology. Using general properties of information
flow on a network we suggest a new approach for network monitoring
and intrusion detection, an approach based on complete network monitoring.
For detailed analysis of information exchange on a network we apply methods
used in physics to analyze complex systems. These methods are rather powerful
for general analysis and provide a guideline by which to apply the result for
practical purposes such as real time network monitoring, and possibly, solutions
for real-time intrusion detection.
4.2 Description of Information Flow
A careful analysis of information exchange on networks leads to the appropriate
method to describe information flow in terms of numerical functions. It gives us
a mathematical description of the information exchange processes, the basis for
network simulations and analysis.
To describe the information flow on a network, we work on the level of packet
exchange between computers. The structure of the packets and their sizes vary
and depend on the process. In general, each packet consists of a header and
attached (encapsulated) data. Since the data part does not affect packet propagation
through the network, we consider only information included in headers.
We recall that the header consists of encapsulated protocols related to different
layers of communications, from a link layer to an application layer. The information
contained in the headers controls all network traffic. To extract this
information one uses tcpdump utilities developed with the standard of LBNL's
Network Research Group. This information is used to analyze network
traffic to find a signature of abnormal network behavior and to detect possible
intrusions.
The important difference of the proposed approach from traditionally used
methods is the presentation of information contained in headers in terms of
well-defined numerical functions. To do that we have developed software to read
binary tcpdump files and to represent all protocol parameters as corresponding
time-dependent functions. This gives us the opportunity to analyze complete
information (or a chosen fraction of complete information that combines some
parameters) for a given time and time window. The ability to vary the time
window for the analysis is important since it makes possible extracting different
scales in the time dependence of the system. Since different time scales have
different sensitivities for particular modes of system behavior, the time scales
could be sensitive to different methods of intrusion.
As was done in reference paper, we divide the protocol parameters for
host-to-host communication into two separate groups with respect to the pre-
serving or changing their values during packet propagation through the network
(internet). We refer to these two groups of parameters as \dynamic" and
\static". The dynamic parameters may be changed during packet propagation.
For example, the \physical" address of a computer, which is the MAC parameter
of the Ethernet protocol, is a dynamic parameter because it can be changed if
the packet has been re-directed by a router. On the other hand, the source IP
address is an example of a static parameter because its value does not change
during packet propagation. To describe the information flow, we use only static
parameters since they may carry intrinsic properties of the information flow and
neglect the network (internet) structure. (It should be noted that the dynamic
parameters may be important for study of network structure properties. Dynamic
parameters will be considered separately).
Using packets as a fundamental object for information exchange on a network
and being able to describe packets in terms of functions of time for static parameters
to analyze network traffic, we can apply methods developed in physics
and applied mathematics to study dynamic complex systems. We present some
results obtained in references to demonstrate the power of these methods
and to recall important results for network monitoring applications.
It was shown that to describe information flow on a network one can use a small number
(10 - 12) of parameters. In other words, the dimension of the information flow space is less than or equal to 12 and the properties of information flow are practically independent of network structure, size and topology. To estimate the dimension of the information flow on the network one can apply the algorithm for analysis of observed chaotic data in physical systems. The main idea relates to the fact that any dynamic system with dimensionality
of N can be described by a set of N deferential equations of the second order
in configuration space or by a set of 2N deferential equations of first order in
phase space.
Assuming that the information flow can be described in terms of ordinary
differential equations (or by discrete-time evolution rules), for some unknown
functions in a (parametric) phase space, one can analyze a time dependence of
a given scalar parameter s(t) that is related to the system dynamics. Then one
can build d-dimensional vectors from the variable s as
yd(n) = [s(n); s(n + T); s(n + 2T); : : : ; s(n + T(d _ 1))] (1.1)
at equal-distant time intervals T: s(t) ! s(T _ n) _ s(n), where n is an integer number to numerate s values at different times. Now, one can calculate a number of nearest neighbors in the vicinity of each point in the vector space and plot the dependence of the number of false nearest neighbors (FNN) as a function of time. The FNN for the d-dimensional space are neighbors that move far away when we increase dimension from d to d + 1.
The typical behavior of a scalar parameter and corresponding FNN plot are shown in Figs. (4.1) and (4.2). From the last plot one can see that the number of FNN rapidly decreases up to about 10 or 12 dimensions. After that it shows a slow dependency on the dimension, if at all. Fig. (4.2) shows that by increasing the dimension d step-by-step, the number of FNN, which occur due to projection of far away parts of the trajectory in higher dimensional space is decreases to a level restricted by system noise that has infinite dimension. Therefore, for
a complete description of the information flow one needs not more than 12 independent parameters. The dynamics of information flow can be described as a trajectory in a phase space with the dimension of about 10 - 12. Since this dimension does not depend on the network topology, its size, and the operating systems involved in the network, this is a universal characteristic and may be applied for any network.
However, we cannot identify exactly these independent parameters. Due to
the complexity of the system it is natural that these unknown parameters which
are real dynamic degrees of freedom of the system would have a complicated relationship
with the parameters contained in the network protocols. Fortunately, the suggested technique provides very powerful methods to extract general information about the behavior of dynamic complex systems. For example, the obtained time dependence of only one parameter, the protocol ID shown on Fig.(4.1), is enough to reconstruct the trajectory of the information flow in its phase space. The reconstructed projection of the trajectory on
3-dimensional space is shown on Fig. (4.3). Therefore, one can see that the complete description of the network information traffic in terms of a small number of parameters
is possible. The important point is that this trajectory (usually called as an \attractor") is well-localized. Therefore, it can be used for detailed analysis and pattern recognition techniques. It should be noted that the attractor presented here is obtained from one parameter measurement only, for that being illustrative purposes. For real analysis we use multi-dimensional high accuracy reconstruction.
4.3 Real Time Network Monitoring and Detection of Known Intrusions
The proposed approach for network traffic description provides the possibility of
real-time network monitoring and detection of all known network attacks. This
is because one collects from tcpdump binary output the complete information
about network traffic at any given point in the network. All header parameters
are converted into time dependant numerical functions. Therefore, each packet
for host-to-host exchange corresponds to a point in the multidimensional parametric
phase space. The set of these points (the trajectory) completely describes
information transfer on the network. It is clear that this representation provides
not only the total description of the network traffic at the given point but also
a powerful tool for analysis in real time. Let us consider some possible scenarios
for the analysis.
Figure 4.1: Protocol type ID in the IP protocol as a function of time (in t = 5sec units).
Suppose we are looking for known network intrusions. The signature of an
intrusion is a special set of relationships among the header parameters. For example , the signature for the attempt to identify live hosts by those responding to the ACK scan includes a source address, an ACK and SYN flags from TCP protocol, a target address of the internal network, sequence numbers, and source and destination port numbers. The lone ACK flag set with identical source and destination ports is the signature for the ACK scan. This is because the lone ACK flag set should be found only as the final transmission of the three-way handshake, an acknowledgement of receiving data, or data that is transmitted where the entire sending buffer has not been emptied. From this example one can see that the intrusion signature could be easily formulated in terms of logic rules and corresponding equations. Then, collecting the header parameters (this is the initial phase of network monitoring) and testing sets of them against the signatures (functions in terms of the subset of the parameters) one can filter out all known intrusions. Since we can collect any set of the parameters and easily add any signature function, it provides the way for a continuous upgrading of the intrusion detection system (IDS) built on these principles. In other words, such an IDS is universal and can be used to detect all possible network intrusions by adding new filter functions or macros in the existing testing program. It is very flexible and easily upgradeable. The flexibility is important and can be achieved
even in existing \traditional" IDS's. What is out of scope of traditional approaches
is the mathematically optimized minimization of possible false alarms
and controlled sensitivity to intrusion signals. These properties are an intrinsic
feature of our approach.
The important feature of the approach is the presentation of the parameters
in terms of time dependant functions. This gives the opportunity to decrease as best as possible for the particular network the false alarm probability of the IDS. This can be done using sophisticated methods already developed for noise.
reduction in time series. Moreover, representation of the protocol parameters
as numerical functions provides the opportunity for detailed mathematical analysis
and for the optimization of the signal-to-noise ratio using not only time
series techniques but also numerical methods for analysis of multi-dimensional
functions. The combination of these methods provides the best possible way, in
terms of accuracy of the algorithms and reliability of the obtained information,
to detect of known intrusions in real time.
Also, the description of the information flow in terms of numerical functions
gives the opportunity to monitor network traffic for different time windows
without missing information and without overflowing storage facilities. One can
suggest ways to do it. One example is the use of a parallel computer environment
(such as low cost powerful Linux clusters) for the simultaneous analysis of
the decoded binary tcpdump output. In this case the numerical functions of the
header parameters being sent to different nodes of the cluster will be analyzed by
each node using similar algorithms but different scales for time averaging of signals
(or functions). Thus, each node has a separate time window and, therefore,
is sensitive to network behavior in the particular range of time. For example,
choosing time averaging scales for the nodes from microseconds to weeks, one can
trace and analyze network traffic independently and simultaneously in all these
8 Multidimensional Network Monitoring Table 4.1: The parameters involved in intrusion signatures as shown on Fig.(4.4).
Number Protocol Parameter Frequency time windows. It is worthwhile to remember that the optimal signal-to-noise ratio is achieved for each time window independently thereby providing the best possible level of information traffic analysis for the whole network. There are three obvious advantages for this approach. The first is the possibility to detect
intrusions developed on different time scales simultaneously and in real time.
The second is the automatic decreasing of noise from short time fluctuations for long time windows due to time averaging. This provides detailed information analysis in each time window without loss of information. At the same time, it discards all noise related information, drastically reducing the amount of information at the storage facilities. The third advantage is the possibility to use (in real time) the output from short time scale analyzed data as additional information for long time scale analysis.
4.4 Detection of Unknown Intrusions
The aforementioned approach could be considered a powerful and promising method for network monitoring and detection of known network intrusions.
However, the more important feature of the approach is the ability to detect previously unknown attacks on a network in a wide range of time scales. This ability is based on the method of describing information exchange on a network in terms of numerical functions of header parameters (or a trajectory in multidimensional phase space) as well as using methods of theoretical physics for the analysis of dynamics of complex systems. These methods lead to a very useful result for the small dimensionality of the information flow space. Since the number of parameters used in packet header is large (on the order of hundreds), the practical search for unknown (even very abnormal) signals would be a difficult problem, if not impossible. Therefore, the small dimension of the parametric
space of the information flow is a crucial point for the practical approach for the
detection of unknown intrusions.
To build a real time intrusion detection system that is capable of detecting
unknown attacks, we exploit the fact that we need to analyze only a small number of parameters. Furthermore, as is known from complex systems theory, the choice of the parameters is not important unless they are sensitive to system behavior. The last statement needs to be explained in more detail. Generally, hundreds different parameters could be encapsulated in the packet headers. The question is which parameters we need to choose for the right description of the information flow. Following the discussion in the previous section, one might surmise that we need to make our choice from the known quoted 17 parameters.
It may be a good guess. However, the number 17 is bigger than the dimension
of the phase space which we have in mind, and it could be that hackers will invent new attacks with new signature parameters that are not included in the set presented in the previous section. The right answer to these remarks follows from complex systems theory. For a complete system description one needs only the number of parameters equal to the phase space dimension (more precisely, the smallest integer number that is larger than fractal dimension of the phase space). It could be a set of any parameters that are sensitive to the system dynamics (and the 17 discussed parameters could be good candidates). We do not know, and do not suppose to know, the real set of parameters until the theory of network information flow is developed or a reliable model for information flow description is suggested. Nevertheless, a method developed to study non-linear complex systems provides tools to extract the essential information about the system from the analysis of even a small partial set of the \sensitive" parameters.
As an example, one can refer to the Fig.(4.3) which shows the 3-dimensional
projection of the reconstructed trajectory from the time dependent behavior of
only one parameter (the protocol ID shown on Fig.(4.1)). It means that the
complete description of the network information flow could be obtained even
from a small set of \sensitive" parameters.
One of the ways to implement this approach is to use the multi-window
method discussed in the previous section with the proper data analysis for each
time scale. This method of analysis is not within the scope of the current paper
and will be reported elsewhere. We will review only the general idea and the problems related to this analysis. To detect unknown attacks (unusual network behavior) we use a deviation of signals from the normal regular network behavior.
For these purposes one can use a pattern recognition technique to establish
patterns for normal behavior and to measure a possible deviation from this normal
behavior. However, the pattern recognition problem is quite difficult for this multidimensional analysis. According to our knowledge, it is technically impossible to achieve reliable efficiency in a pattern recognition for space with a rather large dimension, such as 10. On the other hand, the more parameters we analyze the better accuracy and reliability we can obtain. Therefore, we have to choose the optimal (compromise) solution that uses pattern recognition techniques in information flow subspaces with low dimensions. By applying appropriate constraints on some header parameters one can choose these subspaces as cross sections of the total phase space defined. In this case, we will have a reasonable ratio of signal-to-noise and will simplify the pattern recognition technique and improve its reliability. For a pattern recognition we suggest using a 2-3 dimension wavelet analysis chosen on the basis of detailed study of the information traffic on the set of networks. The wavelet approach is promising because it reduces drastically and simultaneously the computational time and memory requirements. This is important for multidimensional analysis because it can be used for an additional, effective noise reduction technique.
4.5 Conclusions
We suggest a new approach for multidimensional real time network monitoring that is based on the application of complex systems theory for information flow analysis of networks. Describing network traffic in terms of numerical time dependant functions and applying methods of theoretical physics for the study of complex systems provides a robust method for network monitoring to detect known intrusions and is promising for development of real systems to detect unknown intrusions. To effectively apply innovative technology approaches against practical attacks it is necessary to detect and identify the attack in a reconnaissance stage.
Based on new methods of data analysis and pattern recognition, we are studying a technology to build an automatic intrusion detection system. The system will be able to help maintain a high level of confidence in the protection of networks.
We thank the staff of the Advanced Solutions Group for its technical support.
This work was supported by the DARPA Information Assurance and Survivability
Program and is administered by the USAF Air Force Research Laboratory via grant
F30602-99-2-0513, as modified.
Bibliography
Websites:
· http://www.cs.usask.ca/undergrads/der850/project/ids/methodologies.shtml
· http://www.ipa.go.jp/STC/IDA/index.html
· http://secinf.net/info/ids/intrusion/
· http://www.robertgraham.com/pubs/network-intrusion-detection.html
· http://www.linuxsecurity.com/feature_stories/ftp-analysis-part1-printer.html
· http://www.daemonnews.org/199905/ids.html
· http://www.robertgraham.com/pubs/network-intrusion-detection.html
· http://www.cerias.purdue.edu/homes/aafid/
Reference Books:
· Reka, A., J. Hawoong and B. Albert-Laszlo, Nature 406 (2000), 378-381.
· Northcutt, S., J. Novak and D. McLachlan, Network Intrusion Detection.
· An Analyst's Handbook, New Riders Publishing, Indiapolis, IN (2001).
· Deri, L. and S. Suin, Computer Networks 34 (2000), 873-880.
IEEE Books:
· Network Intrusion Detection. An Analyst's handbook, 2nd ed. by Stephen Northcutt and Judy Novak New Riders 2001. 430 pages, index Softcover. ISBN 0-7357-1008-2. Reviewed by Robert Bruen April 13, 2001.
TO DOWN LOAD REPORT AND PPT
