Ourmon: A Technical Explanation of the Architecture and Filters

Ourmon 2.5


Table of Contents


Introduction

In this document we give an introduction and overview of Ourmon and it's individual filters.

Ourmon is an open-source tool for network monitoring and anomaly detection. It runs on FreeBSD, Linux, and Solaris. Goals include:

We should be able to make network monitoring work with cheap Pentium-class computers, running FreeBSD, or Linux, and centrally located, port-mirroring capable switches, and probably most important, Ethernet. SNMP is not used. Given that ourmon is a public domain tool, this avoids the purchase of expensive probes and/or WAN-based analysis gear. SNMP/RMON, WAN analysis tools, and proprietary GUI interfaces are not part of the picture. UNIX, Ethernet, cheap switches, RRDTOOL, the web, and open-source are part of the picture.

The front-end packet capture engine has three forms of filters: 1. hardwired (programmed in C), 2. user-programmable using the BSD Berkeley Packet Filter (BPF), and 3. top N style filters that produce sorted lists of largest flows, interesting IP hosts, or top talker TCP/UDP ports. User-programmed BPF filters or BPF filter sets may be added to the system by the user, and allow considerable customization of the system. For example, a user might choose to create one or more graphs that watch local subnets, local hosts, or local ports on local hosts. Hardwired, and BPF filters use the RRDTOOL package for the display and baselining of data. Top N filters use a supplied histogram-drawing program and log one week's worth of data (typically off-line in the ourmon/logs directory) for later analysis. Some summarization-style logs are available from the web pages.

The back-end system displays network information on the web using either:

RRDTOOL strip charts, familiar to network engineers using MRTG tools like cricket, are used in the case of the canned hardwired filters, and user-programmable BPF filters. (See
here for more information on Tobias Oetiker's RRDTOOL). For examples, see the pkts filter, fixed ip protocol filter, or user-programmable BPF port filter below. Top N filters use their own drawing tool, called drawtopn , which displays flows and other top N information as histograms, typically with a label and a number, sorting on the number. The top N IP flow monitor. shows the largest IP (all IP protocols) flows. The top N flow filter itself in addition to the IP histogram, also shows TCP, UDP, or ICMP flows within the 30 second sampling time period and is one of five current top N mechanisms within ourmon.

Recently ourmon has been modified to enhance its abilities to detect network anomalies associated with various forms of attacks including TCP syn scanning, UDP scanning, DOS attacks, and distributed zombie attacks. A number of features have been added, some of which use the BPF to give us an overall network view of TCP and ICMP control data, some of which use new top N filters to show information about particular host IP systems engaged in scanning, and some of which provide RRDTOOL graphs for carefully chosen and proven useful metadata. We will go over these particular filters below in a section of their own entitled anomaly detection filters.


Ourmon Architecture Overview





The ourmon architectural flow diagram is intended to give a rough analysis of data flow through the ourmon system. For network setup, the ourmon "probe" box is assumed to be directly connected to an Ethernet switch. The switch must have port mirroring turned on so that promiscuous mode filtering on the ourmon front-end "probe" can see all desired network packets passing through the Ethernet switch. Network packets are passed to the NIC card of the front-end probe, and in FreeBSD are stored in the BPF buffer. (In Linux, the details are different, but there is still a kernel buffer associated with packet flow to the front-end application).

Ourmon architecturally consists of two parts, known as the front-end probe, written in C, and the back-end graphics engine, mostly written in Perl. The front-end is a form of network analyzer that reads network packets from an Ethernet interface out of a kernel buffer, and runs each packet through a set of filters. (It should be noted that the front-end program is also called ourmon). For each filter, byte counts or packet counts are periodically stored in an output report file, and that information is passed to the back-end. The back-end then takes the output of the front-end and creates web-based graphics. It may optionally log top N flows to ourmon logs (that are similar to syslog in style) and produce some daily "top of the top" reports on the logs.

The front-end uses the Berkeley Packet Filter and pcap library to fetch packets from a promiscuous mode Ethernet device. The filters used are specified in an input configuration file, called ourmon.conf . One may add or remove filters as desired. 68 bytes maximum per packet (as with tcpdump) are captured. Thus only the protocol parts of packets are actually captured via the BPF. Ourmon at this time does not examine L7 data. This allows (L2) Ethernet and (L3) IP addresses, and L4 ports to be examined. This is an important trade-off. As a consequence, because it does not look at layer 7 data, ourmon can deal with higher speed flows when compared to an IDS signature-based tool like Snort. On the other hand, one can only use it to analyze up through layer 4, and no layer 7 analysis is currently possible. From a security point of view, it could easily make sense to run both ourmon and Snort side by side (but on different hosts please). This is because Snort might miss a new attack, like a high-speed DOS attack. Ourmon would show that something abnormal happened simply because it does NOT understand the data. For example, the pkts graph could show an unusual number of counted packets and/or drops. On the other hand, ourmon is certainly not going to see a known attack signature in the data part of a UDP packet.

Internally the front-end program looks at the Ethernet header, IP source and destination addresses, IP next protocol, and TCP and UDP ports, or in the case of ICMP, major and minor ICMP codes. Each configured-in filter is executed per packet. Thus if there are 5 filters in use, they will be executed in order, one after the other on each packet. In general, the average filter will count bytes or packets and represents this with integer counters. Top N filters keep hash lists of flows or IP addresses associated with counters. At the end of the sample period (every thirty seconds) the output is written out in a very simple format, called the mon.lite file, counters are reinitialized to zero and dynamically allocated lists are also freed. See mon.lite for an example. This file is then processed by the back-end to create various kinds of graphics available from a web server. The "mon.lite" file may be viewed as a summarization of a great deal of packet information into a small intermediate and condensed form.

The back-end is Perl-based. The back-end consists of several programs. The most important backend program is called: omupdate.pl. It creates, and updates graphs with the help of the RRDTOOL library. For top N display, it also employs a tool called drawtopn to draw histograms. There are some additional sorting and logging functions that will not be covered here.

Omupdate.pl both creates and updates per-filter RRD (RRDTOOL) databases, depending on which set of filters you wish to use. Omupdate.pl must be driven by a crontab entry (and is typically called twice a minute). It takes the current copy of the mon.lite file, processes it a bit, and stores information in various per-filter rrds (RRDTOOL log databases), as well as creates the current set of RRD-based graphics. It is possible, and probably a good idea, to run the front-end and back-end on two separate hosts. How "mon.lite" is copied from the front-end to the back-end is an "out of band" problem, and is not solved by the ourmon system. But it is an easy thing to do and ssh or the wget application can be used to solve the problem. One very nice side effect of using RRDTOOL is that all the RRD filters, including BPF-based filters, produce a year's worth of baseline data in graphical format. Thus per-filter, one gets current (daily), weekly, monthly, and yearly pictures.

In the next two sections, we will discuss most of the individual filters in detail. Please note that ourmon as a system supports three kinds of filters:

1. hardwired filters, that are done in C code. These have names and specific functions. There are not many of them.

2. user-programmable BPF filters. Ourmon supports arbitrary user-space (as opposed to kernel-space) Berkeley Packet Filter (BPF)-based filters. One can have 1-6 BPF-filters per RRDTOOL graph (more than 6 produces a cluttered graph, and in fact 6 is probably too many). These filters use the same language specification mechanism used by tcpdump. It is currently possible to have up to six BPF expressions per RRDTOOL style graph. At the time of writing the one deployed ourmon front-end system has 60 BPF expressions in it total and in general does not lose packets.

3. top N filter mechanisms that produce various kinds of top N information including a traditional flow monitors. Other top N lists exist focused on anomaly detection. These lists typically show information associated with individual IP source addresses.

In the next section we begin a detailed discussion of the various filters supplied by ourmon. Note that filters have names; for example, the filter that displays different IP level protocols is called "fixed_ipproto" . Filter names are important because they tie the inputs and outputs of the ourmon system together. Filters are named in the ourmon.conf file, and output appropriate to that name appears in the mon.lite file, and again appears in rrdtool libraries, log files, or png images created by the back end. It should also be pointed out that data associated with a filter presented by the back end, is usually interpreted as packets or bytes, and may therefore be presented as bits/sec, or pkts/sec. In some cases, a data item may be presented as data/period. Period means the ourmon sample period of 30 seconds.


Ourmon Web Page Overview

In this section we provide a brief overview of the web page layout. The layout is hierarchical with two levels consisting of a main page (index.html) and any number of secondary detail pages. The primary function of the main page is to provide current status -- the goal is to tell you what is going on now . Secondary pages which are accessed as links from the main page show more runtime statistical details. In general the graphics-engine back-end of ourmon makes reports and graphics. It does not (yet) dynamically update any html pages. The design and layout of the html pages can (and should at least when using the user-customizable BPFs) be altered according to your own needs.

The main page includes RRDTOOL current graphs for the current day in terms of hours, the "top-most" top N histogram graphs, and various reports including reports for the last sample period and some hourly/daily summarizations. For example, this would include the RRDTOOL basic pkts graph at the top layer. From the main page, find basic network information and look right underneath it for the probe #1 pkts/drops graph. Essentially the graph has a label and both graph and label are links as well. In general the label (when a hypertext link) will take you to the info.html page in order to provide more details on the features of the graph. The graph will take you in turn to a secondary details page that provides more runtime statistical details. Ourmon's notion of help consists of links that jump back and forth between the main index.html page, the info.html help page, and the secondary detailed statistics pages referenced by index.html. (This system works best with mozilla/firefox).

Top_n graphs are often presented as histograms. On the main page find the ASCII heading top talker (top_n) flows based on IP source . Underneath it is the histogram graph that shows the top 10 IP-based flows. It is a link too and if you click on it, you go the to related secondary pages that should show more pics with 10 bars per picture. The first graph shows the top 10 flows. The second graph shows the next 10 flows, etc.

The index.html page is structurally broken up into several sections including:

Here we only wish to discuss the quick jump directory as the other sections should prove self-explanatory. The quick jump directory consists of two tables of links into various parts of ourmon. The first table consists of links that are either security related or have bearing on whether or not ourmon is actually running. The second table entitled main page sections simply divides the main page up into various sub-sections and can be used to directly jump to those sub-sections. For example, in the first table we have the TCP-based "worm" portreport.txt report that gives information about noisy TCP hosts. One also finds the similar UDP-based udpreport.txt that gives information about noisy UDP hosts. Both of these links are very fundamental to security and may show current attacks. The following links will take you to the relevent "info" section below for various important security links in the two tables:

In general one uses a web browser to browse ourmon. There are a few things to note. The top-page comes in two flavors, index.html and indexstatic.html. index.html self-updates every 30 seconds (which may be annoying). indexstatic.html should have the same content but does NOT update every thirty seconds. All the second-level web pages do update every thirty seconds, but ASCII reports obviousally are not written in html and do not update themselves. You must use your web browser reload button to update an ASCII report. Just remember that the sample period is thirty seconds long, and an occasional refresh if looking at an ASCII page is helpful.

Now we will discuss the filters in detail.


Network Management

In this section, we will look at filters that may be deemed to be of general interest from the network monitoring point of view, as opposed to the anomaly detection view. This is really a false distinction in some ways, as very blatant attacks (for example the UDP slammer worm coming from one local host) can cause many of the graphs in ourmon to indicate an anomaly.

pkts

The pkts filter (which shows one or two input interfaces, both assumed to be in promiscuous mode) displays the number of total packets caught and/or dropped by the BPF mechanism. This particular filter is always included and is not specified in the ourmon configuration file. It may not be removed. Drops are nature's way of telling you that you have overloaded the ourmon system. How much the ourmon system can do is hard to say, and may take some tinkering on the part of the administrator. However if you are often losing 50% of your packets, that may be a sign that you need to do less or buy a faster computer for the front-end probe. Dropped packets can also occur during attacks. For example, if you are hit with a distributed TCP syn attack, you may drop packets depending on your front-end load. Small packets are a general problem for all "sniffing" based systems as there is simply not enough time between packets to do arbitrary amounts of computational processing.

For example, you can always choose to lose a filter. The top N flow filter without a doubt, is likely the biggest CPU consumer. It may also be better to have ourmon running with only one Ethernet device as opposed to two (even though two is possible). This may be the case due to BPF buffering schemes. Note that the pkts (caught and dropped) counters are zeroed out (in SNMP terms, pkts is a GAUGE), at the mon.lite write period time. All of the individual ourmon filters are currently zeroed out at mon.lite output time. Thus the counters start over from zero for the next round (and at the moment they are more or less all GAUGEs anyway). Typical "mon.lite" output is as follows:

pkts: caught:9440 : drops:0: caught2:0 : drop2:0


fixed_ipproto

The fixed_ipproto filter is very simple. It simply counts up TCP versus UDP versus ICMP bits, dumping any other IP protocol into the "xtra" bin. Typical "mon.lite" output is as follows:

fixed_ipproto: tcp:67402888 : udp:30976940 : icmp:23158 : xtra:47623:

A sneaky technique is used in the mon.lite file in that in some cases mon.lite counts bytes per period, and omupdate.pl converts bytes to bits per second.

fixed_tcp3

The fixed_tcp3 filter accepts two TCP dst OR src ports. It then counts packets displaying bits/sec with src/dst port1 versus port2 versus any remainder packets, dumping any others into the "xtra" bin. Note that xtra is all other bits, not just TCP (this may be a silly idea). Ourmon (the front-end) takes a configuration file. The entry for this filter might take the following form:

fixed_tcp3 119 80

What this means is that we are interested in capturing USENET NEWS (port 119) and HTTP traffic (port 80). As supplied, this filter captures email (25) versus web (80) traffic. Typical "mon.lite" output is as follows:

fixed_tcp3: 119:39908731 : 80:11033941 : xtra:47506497:


fixed_cast

The fixed_cast filter is performed at the Ethernet header layer, not the IP layer, if and only if the supplied link device is an Ethernet device. It displays bits/sec. It classifies packets as Ethernet multicast, Ethernet unicast, and Ethernet broadcast, based on the Ethernet destination address. Note that this filter is potentially useful for observing possible broadcast storms, whatever the cause, or multicast routing meltdowns. At this time, assuming Ethernet inputs, "xtra" packets should be 0. The configuration file looks as follows:

fixed_cast 127.0.0.0 255.0.0.0

This is because we require an IP net/mask pair, just in case one of the input interfaces is not Ethernet-based (the localhost device). These addresses are ignored if Ethernet is used, but they must still be supplied (just leave them alone in the supplied configuration).

The mon.lite output looks like this:

fixed_cast: mcast:191214 : ucast:98257955 : bcast:0 : xtra:1440:


fixed_size

The fixed_size filter is performed at the Ethernet header layer, not the IP layer, if and only if the supplied BPF device is an Ethernet device. It displays pkts/sec. This filter counts packets within four fixed byte bucket sizes, where the packet is <= 100 bytes (tiny), <= 500 (small), <= 1000 (medium), or <= 1500 bytes (big). These packets may include errors. The configuration file looks as follows:

fixed_size

The mon.lite output looks like this:

fixed_size: tiny:51732 : small:13795 : med:13544 : big:32876:


fixed_l2proto

The fixed_l2proto filter is performed at the Ethernet header layer, not the IP layer, if and only if the supplied link device is an Ethernet device. It displays pkts/sec. This filter counts packets according to the L2 ethernet header for IP protocols only, including IPv4, IPv6, and ARP. Other packet types are considered as "xtra". Note that this filter might show an ARP storm. The configuration file looks as follows:

fixed_l2proto

The mon.lite output looks like this:

fixed_l2proto: ip:647819 : ipv6:1 : arp:48 : xtra:19:


user designed RRDTOOL-based BPF graphs

The user-mode BPF filters are a powerful part of ourmon and allow programmable back-end RRD-based graphs. They allow the user to design his or her own RRDTOOL graphs. For example, we might have the following BPF filter set examples in our ourmon.conf filter specification file:

bpf-packets
bpf "protopkts" "ip" "ip"
bpf-next "tcp" "tcp"
bpf-next "udp" "udp"
bpf-next "icmp" "icmp"

bpf-bytes
bpf "ports" "ssh" "tcp port 22"
bpf-next "p2p" "port 1214 or port 6881 or port 4662 or port 6346 or port 6257 or port 6699 or port 6257"
bpf-next "web" "tcp port 80 or tcp port 443"
bpf-next "ftp" "tcp port 20 or tcp port 21"
bpf-next "email" "tcp port 25"
bpf-noxtra

In the ourmon.conf file, BPF filters default to displaying bits per second (bpf-bytes is used above to force this mode). The config file has two modes which can be toggled on and off between filter sets (but not inside a filter set) using the following two commands:

bpf-bytes
bpf-packets

These two commands cause an entire BPF filter set to produce either bits/sec or pkts/sec.

The first filter above is supplied with the default ourmon config file and serves to graph pkts/sec for basic IP protocols, including any IP protocol itself, TCP, UDP, ICMP, and "other" protocols not of an IP nature. We get one RRDTOOL graph with five lines in it. Any use of "bpf" as the first config token starts a new BPF graph. The name of the graph is "protopkts". "bpf" and "bpf-next" tags are used to introduce (label, BPF expression) pairs. For example,

bpf-next "tcp" "tcp"

means the label is "tcp" followed by the BPF expression which is also "tcp" in this case. Let us look at a more complex example.

The "ports" filter set also gives us one RRDTOOL graph with 5 lines in it. The name of the graph (BPF filter set) is "ports". The BPF tag called "bpf" indicates to ourmon that we are starting a new BPF filter set graph. "ports" is the name and label of the entire graph and is used in the backend for creating unique filenames both in the web and rrddata directories. This first bpf line also includes the first line label "ssh" and a BPF filter specification associated with that line label. The BPF expression "tcp port 22" is designed to capture all secure-shell traffic on port 22. The subsequent "bpf-next" lines add additional lines to the graph for p2p traffic (kazaa, bigtorrent, edonkey, gnutella, and the like -- by definition p2p apps can use whatever port they want, so this isn't perfect, it's just an informed guess), the web, ftp, and email. Each bpf-next line has a line label and filter specification.

A BPF graph is either terminated by another filter or by a BPF-noxtra label which tells ourmon to not collect and graph bytes that fail to match any of the BPF filter specifications in this graph. One may choose to have remaining bytes shown in the final "xtra" graph, or choose to ignore them (which means xtra is *still* shown in the graphs but has a zero value as nothing is ever stored in its counter).

The supplied ourmon.conf file has many examples of BPF filters, and you should make your own. For example, you could graph subnet traffic for 3 subnets, 10.0.1.0/16, 10.0.2.0/16, and 10.0.3.0/16 as follows:

bpf-bytes
bpf "subnets1" "net1" "net 10.0.1.0/16"
bpf-next "net2" "net 10.0.2.0/16"
bpf-next "net3" "net 10.0.3.0/16"
bpf-noxtra

Or you can easily make up graphs that might watch a local server using expressions like:

bpf "hostA" "total" "host 10.0.0.1"
bpf-next "email" "host 10.0.0.1 and port 25"
bpf-next "web" "host 10.0.0.1 and (tcp port 80 or tcp port 443)"
bpf-noxtra

See the INSTALL file for more information on BPF graph customization.


CBPF BPF Optimization

As a runtime optimization that only involves the front-end probe, we have developed a sub-system we all CBPF. The goal of CBPF is to allow the user to increase the number of BPF filter-set expressions while at the same decreasing the odds that packets will be lost. CBPF allows us to hand code C functions to replace some commonly used BPF expressions in the front-end. Unlike the BPF expression mechanism, CBPF is not general. There is only a limited set of expressions that can be used. The syntax in the config file is also slightly different and in some cases cumbersome. Therefore BPF expressions will exist that cannot be replaced by the existing set of CBPF functions. However the runtime advantage of a CBPF function over its equivalent BPF expression is significant. Typically CBPF is at least ten times faster. One can use CBPF to replace common subnet expressions (net IP) and also in our supplied BPF filter-sets for ports and P2P applications. (See the supplied ourmon.conf file). In the configuration file, CBPF tags where they exist may be dropped into the user BPF filter-set at any point, replacing an equivalent BPF expression. CBPF config tags can be used to replace either "bpf" or "bpf-next" as follows:

cbpf "ctag" "graph-label" "line-label" "cfilter"

cbpf-next "ctag" "per-filter-label" "cfilter"

The "ctag" is used to signal to the configuration the syntactic form of the CBPF filter. The "graph-label" and "line-labels" have the same function as with the pure BPF filter set syntax. However "ctag" and "cfilter" are totally different from BPF. Here are usage examples for the existing set of CBPF tags shown with equivalent BPF expressions:

1. network tag ("net")
cbpf "net" "subnets" "subnet8" "10.0.8.0/24"
bpf "subnets" "subnet8" "net 10.0.8.0/24"

cbpf-next "net" "subnet8" "10.0.8.0/24"
bpf-next "subnet8" "net 10.0.8.0/24"

2. tcp port range ("tcprange")
cbpf-next "tcprange" "bittorrent" "6881 6889"
bpf-next "bittorrent" "tcp and (port 6881 or port 6882 etc.)"

Commentary: the BPF lacks a range operator. BPF expressions can consist of logical ORs but this results (particular in an attempt to catch the bittorrent default port range, which may actually extend to port 6899) in a very inefficient set of BPF instructions. Keep in mind that the BPF is interpreted at runtime, and CBPF is compiled C code. In general, BPF expressions have linear cost. Longer expressions cost more at runtime. Note that this CBPF operator (as it true in similar cases) can be viewed as a set of logical ORs in terms of its operands.

3. udp port range ("udprange")
cbpf-next "udprange" "someports" "6881 6889"
bpf-next "someports" "udp and (port 6881 or port 6882 etc.)"

4. udp and tcp mixed single ports ("ports")
cbpf-next "ports" "edonkey" "u 4665 u 4672 t 4665 t 4661 t 4662"
bpf-next "edonkey" "(udp and port 4665 or udp port 4672) or (tcp and (port 4665 or port 4661 or port 4662))"

Commentary: Mixed sets of UDP and TCP ports can be OR'ed together. Ten pairs of (u port) or (t port) combined are allowed maximum.

5. tcpcontrol flags ("tcpflag")
cbpf-next "tcpflag" "syn" "s"
cbpf-next "tcpflag" "fin" "f"
cbpf-next "tcpflag" "rst" "r"

The above is the functional equivalent of the following:

bpf-next "syn" "tcp[tcpflags] & tcp-syn != 0"
bpf-next "fin" "tcp[tcpflags] & tcp-fin != 0"
bpf "tcpcontrol" "rst" "tcp[tcpflags] & tcp-rst != 0"

6. ping ("ping")
cbpf-next "ping" "pingline"
bpf-next "bpfping" "(icmp[icmptype] == icmp-echoreply) || (icmp[icmptype] == icmp-echo)"

Commentary: The CBPF ping tag takes no expression string.

7. icmp major and minor code ("icmp")
cbpf-next "icmp" "ccportun" "3 3"
bpf-next "bpfportun" "icmp[icmptype] == icmp-unreach && icmp[icmpcode] == 3"

8. ip next protocol ("ipproto")
cbpf-next "ipproto" "tcp" "tcp"
bpf-next "tcp" "tcp"

cbpf-next "ipproto" "esp" "50"
bpf-next "esp" "ip proto 50"

topn flow graphs

The following links show the outputs for the various topn flow graphs:
topn_ip
topn_tcp
topn_udp
topn_icmp

The topn_ip filter provides a traditional top N flow point of view for IP (any IP flow), TCP, UPD, and ICMP flows. It shows the top N flows in bits/sec. An IP flow is defined as a 5 tuple having this form: (IP src, IP dst, next IP protocol, L4 src port, L4 dst port). TCP and UDP flows of course do not have the next IP protocol field. ICMP flows display major and minor ICMP codes as opposed to L4 ports. The ICMP major value is displayed as the L4 "source port", that is, it is put on the left-hand side of the flow. The ICMP minor value is displayed as the L4 "source port" on the right-hand side of the flow.

The front-end ourmon program performs a hash function, and creates a dynamic hash list of flows at runtime. At report time, it sorts the list with quicksort, and writes out the top N total IP "flows" (src to destination), top N TCP, top N UDP, and top N ICMP flows. Note that the numbers are simply 30 second byte counts represented as bits/sec. The drawtopn program is used to create png histograms for the results. In addition, logging is done for the four flow types in separate logging files (this is true for all top N filters). (These syslog-style logs are available for analysis by the administrator, but they are not available on the web). In addition "top of top" reports are generated for the topn_ip, topn_udp, and topn_icmp filters, on an hourly or daily basis for a week. The sorting functionality provided allows one to see the top flows over a period of time. Top of top reports are also created for topn_ip, topn_udp, and topn_icmp that report on the biggest IP source and destination addresses flow generators for these three flow types.

The mon.lite output is as follows:

topn_ip : 6954 : 131.252.208.43.65529->131.252.120.170.119(tcp): 18320510 : 128.223.220.30.40165->131.252.208.43.119(tcp): ETC ...
topn_tcp : 5596 : 131.252.208.43.65529->131.252.120.170.119: 18320510 : ETC...
topn_udp : 1257 : 209.70.46.6.27968->131.252.77.153.6974: 269300 : ETC ...
topn_icmp: 2: 131.252.3.1.0->131.252.2.1.0: 5234: 131.252.2.1.8->131.252.3.1.0: 5234: ETC ...

Note that the number following the topn_ip tag value above is the count of distinct IP flows seen during the sample period. This is not the same as the top N flows shown as tuples in the mon.lite file. It is a count of the unique flows seen during the sample period, all of which have been stored in the hash list itself. But of course, not all of them are printed out if the number of flows exceeds the top N value (currently hardwired to 9, but this will change in the next release). The flow count itself is currently graphed in an RRDTOOL-style graph as it is very useful for anomaly detection. See below in the anomaly detection section for more information on that graph.


topn ports graph

The following links show the output for the topn ports graph:
topn_tcp_port
topn_udp_port

The topn_port filter displays the top N ports used in TCP and UDP flows. The top N ports are sorted by packet byte count and expressed in bits/sec. Up to 100 ports may be displayed with 10 ports per graph for each TCP and UDP graph (which are separate). The back-end topn_tcp_port graph has the following format in its graphs:

port/L4 src_count/L4 dst_count
mbits

The top port is displayed followed by the L4 src_count and dst_count in the "legend" or top part of the label on the graph. The port value may be either a src or destination L4 port. For each port below the legend, we present bits/sec which is used for sorting the tuples. (Effectively the port is the key, and the bit count is used for sorting). The src_count/dst_count denote how many times the L4 port was a source/destination *port*. Thus it may be possible to determine that a particular port is only being used as a destination (or source) port. Also the src and dst counter is a packet counter, not a byte counter.
The entry for this filter in the configuration file may be as follows:

topn_port 20

This indicates how many top port tuples should be written to the mon.lite file.

mon.lite output is as follows:

tcp_ports: 8472 : 80:56316509:49409:39209 : 6881:20155127:13459:13166 : ETC...
udp_ports: 2237 : 49156:3834617:3693:2758 : 49302:3834617:2758:3693 : ETC...

The number value followed the filter tag in the mon.lite output (tcp_ports : number : ...) represents the number of distinct port tuples seen during the sample period. N (e.g., 20) 4-tuples follow with the tuple format: (port, byte count, L4 src port packet count, L4 dst port packet count). omupdate.pl rearranges this information into a tuple more suitable for display.

Anomaly Detection

Ourmon provides a fair number of anomaly detection filters using both supplied BPF filters, top N filters, and a few carefully chosen meta-data RRDTOOL-based graphs. We will discuss the top N filters here first, including some interesting features, and then go on to the supplied BPF filters. It should be pointed out that in general the BPF filters look at the big picture. For example, the TCP control BPF filter set shows the overall number of TCP SYNS, FINS, and RESETS in a network. On the other hand, the top N TCP syn filter shows the top IP hosts sending out TCP SYNS. The former gives you a network-wide picture. The latter helps show individual hosts that may be taking part in an attack (or running gnutella).

There are a couple of important design ideas here driving this work. One is simply the observation that it can be useful to look at network control data to observe a baseline and then detect anomalies in that baseline. There is in general much LESS network control data than data itself (one hopes). There are less ICMP error messages, than UDP data (usually). There are less TCP control packets in terms of SYNS/FINS/RESETS compared to data itself.

As two more concrete examples of this notion, one would expect that TCP SYNS might somehow be related graphically to TCP FINS. Large numbers of SYNS that have nothing to do with the FIN count thus indicate that something abnormal is happening. One can also look at ICMP error data, and see that for example, large numbers of ICMP host unreachables are being returned because of an active UDP scanning system. Second-order data may also be of interest from the viewpoint of network control. For example, instead of focusing on individual top network flows, we can focus on the overall count of network flows. Perhaps one has on the order of 1000 ICMP flows per second. 100000 ICMP flows per second would then indicate that something was very wrong. (This actually happened due to an outbreak of the Welchia/Nachi worm on one campus in Fall 2003).

We also are in the process of a long-term experiment using various kinds of weights to detect scanners. There are three new top N list mechanisms that in some cases use weights to try and detect scanning systems, and in some cases simply sort on integer counts. We have a top N syn list that currently does both of these things. It sorts on the top N ip src sending TCP syn packets, and at the same time, produces various metrics that for example tell us the percentage of TCP control packets compared to the overall percentage of TCP packets sent and received by that host during the sample period. We call this weight the work weight . The work weight appears to be very useful as it allows us to rule out most P2P applications that often also send large numbers of TCP SYNS during the sample period, but in fact, generate less TCP SYNS + FINS + RESETS / total pkt count than most SYN scanning worms. The top N syn mechanism also produces several other outputs including the tworm graph , which represents a total count of anomalous ip sources in terms of SYN counts, and thus allows one to see distributed attacks occuring in parallel, tcpworm.txt report which can be viewed via several different outputs most notably via the so-called port signature report . Both of these topn syn list outputs are produced as a side effect of a weight metric where we only capture ip sources from the syn tuple when they satisfy the following worm metric :

TCP SYNS sent - TCP FINS returned > 20.

This mechanism seems to do a good job of showing distributed zombie attacks simply because the number of hosts matching the SYNS - FINS metric increases over some normal bound by a significant number. The metric can be viewed as a low-pass filter that leaves out "normal" applications that typically produce roughly equivalent amounts of SYNS and FINS in a sample period. We have also done some normalization experiments and can state that unusual numbers of TCP SYNS aimed at TCP ports 445 and 139 do not occur with normal Microsoft file sharing.

In other new top N filters, we also count the total number of ICMP error packets sent back to a host, and a weighted measure of UDP errors as well. We also have a conventional IP scan catcher that sorts on the top N IP source addresses with the most unique destination IP addresses. This scan filter also produces graphs that show the mapping of an IP source to N L4 TCP or UDP destination ports.

This work is "not done" and is only preliminary. We hope to improve it over time.


TCP Syn List Overview

The following links provide outputs for information derived from the topn tcp syn list:
topn_syn histograms
tworm RRD graphs
the TCP port signature report
detailed TCP port signature/tcpworm.txt report
hourly summarized port signature analysis report
the TCP port signature new scanner log
the p2p application version of the port report

In this section we discuss the TCP syn list tuple and various outputs associated with it. We will first present a general overview ot the TCP syn list mechanism, then give it configuration, and finally discuss each output presented above in turn.

Ourmon if so configured stores a TCP syn tuple with various counters that look at the two-way nature of TCP traffic from an IP source address. This syn tuple includes information about SYNS sent by an ip source, the total number of SYNS+ACK packets sent by an ip source, FINS returned to that ip source, RESETS returned, ICMP errors returned, total pkts sent from the IP source, total pkts returned to the IP source, counts of unique IP destinations, counts of unique L4 TCP ports, and a novel destination port sampling scheme. The port sampling scheme stores the first N TCP destination ports seen associated with SYNs and stores packet counts for SYN packets only sent to those ports. This is a sampling scheme as only the first N ports seen are stored. Not all destination ports are stored. The SYN+ACK count packets are only those packets from a given ip source that are sent as the second packet of the TCP 3-way handshake. In TCP state machine terms, these packets represent a service on a host. Thus it is fair to say that a percent score here of 100% means a host is a server.

As one example, the topn_syn list shows individual IP source addresses sending the most TCP syns in a period.

The following outputs are currently produced by the topn_syn list tuple:

Note that three different top N mechanism currently rely on each other. The IRC summarization mechanism uses the topn_syn work weight. The topn_syn list itself also uses data from the topn_scan IP and port lists. All three of this lists should be turned on together.

Two major weight functions are used in various places above. These are called the work weight and the worm weight respectively. The work weight approximates the percentage of TCP control packets captured divided by the total number of packets sent both ways. The work weight for each ip source is roughly:

SYNS sent + FINS sent + RESETS returned / total 2-way packet count.

Typical benign hosts score low work weights of 0%. A work weight of 100% means all control and no data and may be deemed truly "anomalous". Signicantly anomalous hosts may have a work weight between 50..100%. Intuitively we are measuring the amount of TCP control packets versus the total number of packets. If there are large amounts of data the work weight will be low. Obviousally a SYN scanner will score a 100% work weight. Of course an anomaly may represent a client that has no server. Or it may represent a badly coded client or poorly performing TCP application. For example it is not unusual to spot hosts using Gnutella with a high work weight because the Gnutella application is failing to find Gnutella peers. However in many months, we have seen only a handful of cases of such anomalies that were not worms, and 1000s of cases that were worms. More details on the work weight are given below when we talk about the port signature report.

The front-end collects a topn syn list which may of course have 1000s of "syn flows" in it. It sorts the syn list twice, once for the topn_syn report, and once for a second output file now by the ourmon front, which is the tcpworm.txt file in raw form. (Raw form means it is not yet processed for human readability which is done by the back-end). The information given in the tcpworm.txt file consists of that set of ip source addresses showing up with more syns than fins returned to them. The metric used there is the worm weight . At this point the worm weight is computed by saying

SYNS - FINS > 20.

If this weight is satisfied, an ip source tuple is added to the collection in the tcpworm.txt file. Note that this is a different view of the original syn list data when compared to the topn syn list itself. The topn syn list in the mon.lite file represents a sort of the syn list and gives us the top N syn sending systems. The tcpworm.txt file gives us the complete set of ip sources where there were more SYNS than FINS by some number N. As a result, the top N syn list is bounded as of course it is a top N list. The tcpworm.txt list is not bounded. However typically barring large distributed attacks, it is far smaller than the total number of IP sources in the complete topn N syn tuple list.


TCP Syn list ourmon.conf configuration

Let us first look at the setup in the ourmon.conf file and then we will consider the results in more detail:

topn_syn 60
topn_syn_wormfile /home/mrourmon/tmp
topn_syn_homeip 10.1.0.0 255.255.0.0
topn_syn_p2pfile /home/mrourmon/tmp
tworm_weight 80

The first command topn_syn turns on the basic topn_syn graph. The argument to topn_syn (60) specifies how many hosts you want to capture in terms of pictures. This argument should have the value 10, 20, 30.. to 100 maximum. 60 will produce six histogram graphs with the top 60 "syn" producing systems shown in two graphs with 10 ip source addresses per graph.

topn_syn_wormfile turns on the tworm RRDTOOL graph output and also the tcpworm.txt report. In other words, output based on the worm metric is produced. There are actually several front-end outputs including the tworm graph, which graphs integer counts showing the number of external ip sources and internal ip sources appearing in the tcpworm.txt file, and the tcpworm.txt file itself, which like the mon.lite file is passed to the back-end for further processing. Note that we assume you will use this function, and it is probably not wise to turn it off.

The second argument to topn_syn_wormfile is a directory path which specifies where to place the tcpworm.txt report in the front-end file system. This argument may be overridden with the -D option to the front-end ourmon probe. -D causes all ourmon probe output files to be placed in a specified directory, typically /home/mrourmon/tmp. If two systems are used, one as the front-end probe, and the other as the back-end graphics engine, the administrator must arrange that the tcpworm.txt file will be copied to the back-end, and further processed by shellscripts produced in the configuration process. Tcpworm.txt is the rough output that is further processed by the back-end and becomes the port signature report, and other outputs when processed by the back-end. We will discuss these reports more below. So tcpworm.txt is a rich source of information and at this time there are a number of back-end reports based on its data.

topn_syn_homeip specifies what you consider to be the "internal" subnet used for "us" in the tworm.html output. This allows the front-end to decide what is "us" versus "them" in terms of the tworm RRDTOOL graph. The first argument is a network IP and the second argument is a netmask. If you are using ourmon on a host-basis and not watching an entire network, you can set this value to be your host IP, and then use a /32 netmask.

topn_syn_p2pfile causes the creation of the p2pfile.txt probe output file. This file is the subset of all syn tuples that has an application flag set which more or less means the set of hosts using well-known P2P applications including Bittorrent, Gnutella, Kazaa, and Limewire/Morpheus. It also includes hosts using Internet Relay Chat. This file is processed by the back-end to create the p2p port report, which is viewable on the IRC web page .

Finally tworm_weight can be used to specify a value ranging from 0..100 percent that is used to filter the number of hosts shown in the tworm RRDTOOL graph. 70 is probably a reasonable number here as in our experience hosts with a higher value are very likely to be worm/viruses, although worm/viruses can appear with a lower weight. For example a common attack against one of the SQL-slammer ports (1433) may actually result in two-way data exchange. This is because SQL servers are returning Layer 7 data that is basically saying: "the password guess is wrong". As a result, the work metric for an attacking IP source may be lower if data (albeit negative data) is being returned by the attacked host. If this switch is not used, the default filter weight is 0%. This means no filtering is done. There is at this time no way to filter the portreport itself or the tcpworm.txt file as an input.


TCP syn list outputs - topn_syn graph

The topn_syn graph shows syns, fins, resets, and the work weight metric for individual IP source addresses sorted by the IP source sending the most syns in the sample period. The topn_syn back-end graph information per ip source address might be presented as follows for two ip addresses 10.0.0.1 and 10.0.0.2.

10.0.0.1 f: 193: r: 247 t: 7254 72%:w
4779

10.0.0.2 f: 0: r: 20 t: 2035 100%:W
2015

The IP address is the IP source address for the SYNS being sent. The SYN count is given below the other information and in the first case above is 4779 meaning 4779 syns in the sample period. f stands for "FIN". r stands for "RESET". These represent TCP FIN and RESET counts sent back to the IP source address. t represents the total TCP packets sent to and from the address in question. The work weight is also given as a percent with one flag, which may be 'w' or 'W'. As a visual aid, if the percent is greater than 90%, we give it a "W" flag. If the percent is between 50..90%, we give it a 'w' flag. With the work weight, false positives are possible. In general, one should use tcpdump or some other scanner to capture packets from a believed suspicious IP host and verify its behavior.

Although false positives are possible, we have observed the following about the work weight system:

The above weighting system is also used in the port signature report and its companion tcpworm.txt output file which has more details in it per ip source.

mon.lite output is as follows for topn_syn:

syn_list: 9582 : 10.251.188.65:915:0:896:896:21:4654:3604:0:0:0:12:1:1:1433,4654,: next tuple ...

Note that the raw syn tuple is the same in both the mon.lite topn syn list, the tcpworm.txt raw format file, and the p2pfile.txt files all produced by the front-end. The first number after the syn_list tag is the total number of syn entries in the hash list. It is followed by the specified number of top N syn tuples (60 say in our example). The syn tuple, gives (IP src, syn count, syn+ack count, fins returned, fins sent, reset count, total sent, total returned packets, icmp errors returned, application flags (see p2p below), a count of edonkey hits, total unique IP destinations sent, total unique TCP destination ports seen, number of syn L4 destination ports sampled, and then two-tuples that make up the port signature. Each tuple in the port signature is of the form (dst port, packet count). The packet count here tells us that some number of packets including SYNS and ordinary non-SYN packets were sent by the ip source in question to the L4 destination port.


TCP syn list outputs - The tworm graph

The tworm graph attempts to capture the number of "wormy" hosts according to hosts put in tcpworm.txt at one time. It is graphing the number of hosts that appear in the tcpworm.txt file (which is the same as the number of lines in the file). Hosts placed in this file are deemed to be "noisy" in that for some reason they generate more TCP SYNS than TCP FINS. Spikes in this curve may correspond to automated distributed bot attacks which may be performing a DOS attack or simply scanning for exploits in parallel. By default the tworm count information counts all IP sources appearing in the tcpworm.txt file and classifies them as to whether or not they appear to be from the internal network, or from an external network. (If internal versus external doesn't make sense, best to do something like make the internal network 10.0.0.0 with netmask 255.0.0.0, thus making all IPs external). The mon.lite config variable

tworm_weight 80

may optionally be used to filter the tworm count by the work metric. Thus one can approximate the number of "real" worms as opposed to noisy P2P hosts or noisy web servers.

mon.lite output for tworm appears as follows:

tworm: 9: 3: 6:

This tuple is placed in the mon.lite file and processed by omupdate.pl in the backend. This produces the tworm RRDTOOL graph. The three numbers in turn represent:

1. a total count of "worms" (by default the count here is the number of ip hosts found in the tcpworm.txt file.
2. the total count of systems in the home system that appear in the tcpworm.txt file.
3. the total count of external systems not in the home subnet.

We have observed external attacks in the 1000s made on one campus. These attacks are real and this mechanism is very useful. It is fair to view this graph as a botnet attack detector.


TCP syn list outputs - the port signature report

The ourmon system presents two processed versions of the front-end tcpworm.txt file called "tcpworm.txt" and "portreport.txt" respectively. These files represent different views of the TCP syn tuple information. (The back-end tcpworm.txt report actually includes the portreport.txt information as well, but for reasons of quick web lookup, the portreport is broken out into a separate report file.) Both files are updated every thirty seconds but as they are ASCII outputs, you must hit "reload" yourself on your web client.

The portreport may be regarded as a simple summary of the more verbose tcpworm.txt report. It is called the TCP "portreport" or "port signature report" because it includes a small sample of TCP destination ports from the IP source in question. Thus it gives a set of destination ports which we call a "port signature". This port signature may allow you to see a new virus at work and often some virus/worms have distinctive "port signatures". For example, see ports 5554 and 9898 as found below in a small example of real port report output. This is a signature of the dabber worm.

Here we only discuss the portreport or "port signature" report as the longer version is for the most part more detailed in terms of SYN tuple information. The port signature report has three sections. The first section is the port signature report itself which is is sorted in ascending order in terms of IP source address. The second section is based on a daily "db" file database that tells you if the current set of destination ports seen are "new" in this port sample, or "old". This database is started again every night at midnight. New simply means so far today we have not seen that particular set of ports before. "old" means we have seen that particular port set before. For example, note the destination ports given below (5554, 9898). This set might have appeared in previous portsignature reports for the day, or it might be new for this particular report. New port signatures are stored in the event log as well (see below for more information). The final chunk of information in the port signature report is a simple condensed help reminder in case you forget what the port signature fields mean or the meaning of the EWORM flags used as well.

With the main port signature, we sort first by IP source address from small to large. Thus one can thus easily note attacks that come from the same IP source "locale", including attacks from the same IP subnet possibly synchronized with some bot mechanism. Each line represents culled information for one IP source deemed anomalous in terms of its SYN output. Let's look at some examples:

sample TCP port report in table form
ip src: flags apps: work: SA/S: L3D/L4D: sdst/total: port signature
10.82.196.58 (WOM) 100: 0: 423/1 107/107: [5554,40][9898,59])
192.168.245.29 (O) B 6: 67: 622/438 277/411: [6880,0][6881,42][6883,6][... more]

The ip source address is given, followed respectively by the flags, application flags, work, SA/S, L3 destination and L4 destination, sampled dst/total dst, and port signature tuple fields.

The flags field attempts to convey whether the "work" in question is two-way or simply one way from the IP source and provides details about the nature of TCP control data as well. The total set of flags is as follows with a rough explanation given for each:

E - An anomalous amount of ICMP errors are being returned to the IP source.
W - The work weight is greater than or equal to 90%. A W means the IP source is highly suspicious.
w - The work weight is greater than or equal to 50% but less than 90%.
O - very few fins are being returned. O stands for "output" (or "ouch"?).
R - TCP resets are being returned.
M - no TCP packets are being returned to the IP src in question from receivers. So for example, M here means there is no 2-way exchange of data.

Scanners or worms commonly produce WORM or WOM, although if a network administrator chooses too, he or she might produce more RESETS from local routers and/or hosts, and this could be useful in detection of scans. The flags field is explained in more detail in the verbose tcpworm.txt output file.

The work weight as before roughly represents the number of TCP control packets divided by all TCP packets sent to and received from the IP source address in question. 100% means more or less all control and no data. From experience we suggest that work weights in the range of 60% or more should be deemed highly suspicious. Although there are rare cases of noisy clients that for some reason cannot reach their server OR email servers that are trying to reply to spam (which will always fail). We have observed over many months that in general anything with a weight over 60% is "anomalous" and in most cases not benign. Low work weights may mean you have a noisy web server that is sending large numbers of small packets per connection to a client. On the other hand, low work weights may be associated with P2P applications, Layer 7 password attacks on web servers (port 80 as a destination port may indeed be significant, but it could be an attack or a benign web client use), or irc server bots with a high rate of JOIN message churn. The latter very well may be a trojan. Thus it is important to note: A low work is not necessarily a benign indication.

The application flags field (which is also used in the p2p version of the port report and in fact defines that report) gives various hints in some cases derived from L7 scanning about the nature of the applications on the host in question. In general the P2P applications here (and IRC) indicate the start of a peer to peer exchange (not a client/server exchange). Thus it is possible that a host may appear to have a "worm" or TCP scanner when using Gnutella and not show a "G" flag simply because it is completely failing to ever find a real Gnutella peer and start a Gnutella peer to peer exchange. The hints flags currently include the following:

B - a Bittorrent application was detected.
G - A Gnutella application was detected.
K - a Kazaa application was detected.
M - a Limewire/Morpheus application was detected.
I - an IRC application was detected.
e - an Edonkey/emule application was detected.
E - is used to indicate that packets are sent to destination port 25. The goal here is to alert you to a possible spammer or a benign email server.
H - packets are being sent from port 80 or port 443. The sender could be a noisy web server or nmap for that matter.

It should be noted that in general the applications flag field does not usually find false positives, although the Edonkey flag is more prone to false positives than the other fields. Also note that the E for email and H for web server flags are only based on the existance of port 25 as a TCP port for email, and ports 80 and 443 as destination ports for web servers. An attacker may very well be scanning on those ports. The use of E and H based on TCP ports compared to B and G is a mixed metaphor, but we felt it was useful to have a mechanism that would give us possible application clues and that clues about email and web are important indeed.

The SA/S field expresses the total percent (0..100) of SYN+ACK packets typically sent as the second packet of the TCP 3-way initial handshake divided by the total number of SYN packets sent from the IP source in question. There are three possible thresholds here. 0 means the system in question is a client. 100 means the system in question is a server. A number in-between (which is commonly found with P2P systems) shows that the system in question has both server and client functionality. One should correlate this information with the port signature information, in particular when the sample space of 10 possible ports is filled up. A low work weight, 100% for SA/S and 10 out of 10 ports in the port signature typically indicates a server (which is typically either a web server or a server undergoing a L7 attack returning error messages like "404 not found", or "password failed"). One interesting facet of this field is that occasionally one will see a work weight of 100% and an SA/S value of 100%. This can only mean that the host in question is performing SYN/ACK scanning.

The L3D/L4D destination field is derived from the ourmon probe scanner module. The L3D field shows the exact number of unique L3 IP destinations during the sampled period. Although in practical experience this field is not misleading, it should be noted that there is no guarantee that the IP destinations are indeed TCP peers (they might be UDP peers). (This is a bug and will hopefully be fixed in a future release). The L4D field shows the unique count of L4 TCP ports seen during the sample period. These fields can often be used to give you an approximation of the behavior of the IP host in question especially if it is a scanner.

The sdst/total field gives some measure of how well the port signature mechanism worked. The sdst number represents the total number of sampled packets stored as counts in the port signature tuple. The port signature tuple samples the first 10 destination ports seen coming from the IP source in question during a sample period. Each tuple consists of a TCP destination port, and the packet count associated with that destination port. The total field consists of the total number of packets coming from that IP source. If the dst value is much less than the total and all 10 port tuple buckets are full, then this is a good indication that the port signature field failed to catch much in terms of its samples. These two numbers are not expressed as a percent because the two values can give you some idea of how many packets per sample period are being sent by the IP source in question. For example if you see that the work weight is 100%, and that packets are being sent to say 5554, 9898 as above, you can then estimate from the numbers given above that probably 3 syns per second are being sent.

The port signature field itself results from a possible sample size of 10 TCP destination ports captured for the IP source. The destination port is the first field in the port 2-tuple. Note that in order to help spot similar attacks, the ports are sorted in ascending order from left to right.

[5554,40][9898,59])

For example [5554,40] means destination port 5554 was being scanned. The second field in the port 2-tuple gives the overall percent or frequency of packets for that port in the total number of sampled ports. In this case 40% of the total port count of packets were aimed at port 5554, and 59% were aimed at port 9898. (For the actual numbers look in tcpworm.txt). It should be noted that this is only a sample. It is not unusual to see all 10 port tuples "full" of something that seems to be evenly distributed at 10%. Such occurances are often due to web-based "noisy" client/server relationships and may be benign. However in some cases this may represent a remote scanner that is simply walking the port space. Scanners may be spotted by looking in the scanning log as their port signatures will be "new" and will change over time. (Of course they may also show up in the top N scanner part of ourmon as well).

We can sometimes tell by the port number itself that an attack is occuring. For example, in general one should be suspicious of any of the following Microsoft ports in the UDP or TCP port signature report.

135-139, 445, 1433, 1434

In typical use one does not see large discrepencies in ordinary use of the Microsoft File Share distributed file system or with the use of a SQL server. The work weight here simply does not matter (and high is still bad!), and in fact with port 1433, a low work weight is a bad sign because it means hosts are your site may be responding at Layer 7 to attacks. Ports 80 and 443 of course may or may not represent attacks on web servers. A low work weight with port 80 might mean that attacks are being launched at a web server and it is returned failure messages at Layer 7. It might also mean that a web server is sending lots of small connections back to a web client and in that case is benign. Certain other port combinations represent well-known viruses as (5554, 9898) is an example. If you see new port signatures, it can be useful to search google for suspicious port combinations (as well as the dshield site and web pages offered up by various anti-virus vendors).


TCP syn list outputs - the tcpworm report

The tcpworm report is essentially an expanded "translated" version of the raw tcpworm.txt file. It is sorted by the syn list; that is IP sources with more syns will appear before IP sources with less syns. We may divide it up into three parts: 1. per IP source statistics, which include all the information the front-end has gathered about a specific IP source, 2. various summary statistics., and 3. the port signature report, which is also included in here. The port signature report is included so that one can use a text editor and simply pattern-match back and forth to learn more details about IP sources in the port signature report deemed of interest.

TCP syn list outputs - the p2p report

The p2p report (found on the irc page) is a separate front-end output and is another topn syn feature. It is functionally all syn list IP src-based tuples that have set the application flag field barring port flags for email and web services. In other words, it is the set of IP hosts seen during the sample period that initiated P2P exchanges for Bittorrent, Kazaa, Gnutella, Morpheus/Limewire. It also includes any host that sent or received IRC messages including JOIN, PRIVMSG, PING, or PONG. See the previous portreport section for more format information. This file uses the same format as the TCP portreport, but is an entirely different subset of IP sources taken from the entire set of hosts sending TCP SYN packets. Note that there is no port signature database associated with this file. The port signature database is only used with the TCP portreport.

TCP syn list outputs - the TCP port signature new scanner log

The TCP scanner log is a daily log that provides new port signatures -- where new means a TCP port signature that has not been observed since midnight of the current day. New port signatures are stored in a db database that is turned over every night at midnight. Port signatures are put in the scanning log with the following form:

Tue Nov 2 08:00:29 PST 2004: new worm signature from:192.168.186.238 [5554][9898][16881]

A TCP scanner seen scanning over a long period may have multiple entries in the log. Note that this log is potentially updated every 30 seconds.

TCP syn list outputs - batch port signature reports

The ourmon back-end generates an hourly report that is a summarization of the TCP portreport. The summarization uses a weight supplied in the back-end that averages all instances of the TCP work weight found so far during the day. Only IP sources with all tuples averaging at least equal to or above the supplied weight are shown in the summarization. The goal is to leave out sporadic weights (of say email servers) where once in a while an IP source will have a high work weight, but generally does not have a high work weight. If you wish to tune this weight modify the back-end bin/batchip.sh script where it calls ombatchsyn.pl to make this log. As this is a batch report, reverse DNS lookup is performed. The output filename is "allworm_today.txt".

Currently the batch report gives the total number of instances for the IP source in question, and also presents a small sample (5) of port signatures. The port signatures presented here are of the form [port_destination, packet count].


icmp and udp error list

The following links show various outputs associated with the topn icmperror list:
topn_icmperror
topn_udperror
udp port signature report
udpweight graph
ourmon system event log

The icmperror_list top n mechanism is specified in the ourmon.conf file as follows:

topn_icmperror 20

10, 20, 30 ... 100 individual IP hosts may be specified with 10 outputs per picture in units of 10.

topn_icmperror generates two separate top N lists, one for ICMP errors associated with individual IP source addresses, and an additional list for UDP-based IP hosts. The former is important for catching scanners generating ICMP errors (which may or may not be generating ICMP packets themselves). The latter is focused on UDP-based anomalies which may include UDP scanners, DOS attacks, or badly-behaved UDP-based applications. The rationale for these top N lists is the same -- they focus on hosts that generate large numbers of ICMP errors.

In the mon.lite file, these lists are shown as follows:

icmperror_list: 20779 : 10.0.0.10:984:0:4:984:984:0:0: 10.0.0.2:125:5416:3:128:106:0:19: ETC.
udperror_list: 13435 : 10.1.2.3:5922:464:401:35:0:144:104:2:53,293,32768,8: ETC.

As usual, the number of list entries in the hash list follows the tag. For the icmperror_list, tuples are as follows: (ip src address: icmp total error count: tcp pkt count: udp pkt count: ping pkt count: icmp unreachable count: icmp redirect count: icmp ttl exceeded count). This means that for a given IP source address, we show the total number of ICMP errors, where errors are the sum of ICMP unreachables, redirects, and TTL exceeded packets returned to that IP src during the sample period. We also attempt to give clues as to whether or not the system is sending TCP packets, UDP packets, or ICMP ping packets. The list is sorted by the total number of ICMP errors. In the graphical histogram representation for the icmperror_list we use a flag-based system as follows:

T: TCP packets sent by an IP source.
U: UDP packets sent by an IP source.
P: ICMP echo request packets sent by an IP source.
Un: ICMP unreachable packets sent back to that IP source.
R: ICMP redirect packets sent back to that IP source.
X: ICMP ttl eXceeded packets sent back to that IP source.

We only show a flag:count if the result for the IP source address is non-zero. The list again is sorted by the total number of ICMP errors. In the next few sections, we will discuss the UDP list in more detail as it has several features.


topn_udperror and udp port signature report

The UDP port signature report comes in two flavors. As the UDP port signature information is fundamentally a top N list we include a histogram-based graphic, which is simpler in form than its more verbose ASCII cousin. The former may be called the "topn_udperror list" (graphic) and the latter is simply called the "udpreport". The udpreport layout is somewhat similar to the TCP report, but it is sorted by weight, not IP address. However the UDP port signature field of the ASCII report is completely similar to the TCP port signature field.

In the mon.lite file udperror_list, the tuple means: (ip source, a computed weight: UDP pkts sent, UDP pkts returned, ICMP unreachables returned, ICMP Ping packets sent, L3 IP destination count, L4 UDP destination port count, and a port count plus port signature tuple). The port count indicates how many port signatures are in the port tuple, 1 to 10 total. Each tuple consists of a L4 UDP destination port followed by a packet count. Port signatures are a sampling technique; that is, we do not try to capture all ports, only the first N seen. L3 and L4 destination fields and the port signature fields are thus roughly similar to the same fields in the TCP portreport.txt report. Here however the weight is used for sorting. The weight is basically:

UDP pkts sent - UDP packets received * ICMP errors.

It is quadratic and we have seen instances of this value above one billion (a DOS attack). The ICMP errors consist of ICMP unreachables, redirects, and TTL exceeded pkts sent back to the IP source in question. ICMP (port) unreachables weigh slightly more and will raise the weight.

In the graphical representation for the udperror_list, we sort on the computed weight as mentioned before. The flag system in this case is as follows:

U: UDP packets sent by an IP source.
R: UDP packets sent back to that IP source.
Un: total ICMP unreachable errors sent back to that IP source.
P: total ICMP PING packets sent from the IP source.

As usual, the weight is shown below the label that starts with the IP source address. Tags are shown only if they have non-zero counts.

The ASCII UDP port report is created by omupdate.pl and is more verbose than the histogram-based graphical system. The report presents the per IP source information in a form similar to the TCP port signature report. Port packet tuples are of the form [destination port, frequency]. In other words, packet counts are not given here, but instead are given as a percent of the total packets captured in the port signature sample. For example, one might see the following:

sample UDP port report in table form
ip src: weight: udp_sent: udp_recv: redirect: ping: L3D/L4D: sdst/total: port signature
10.82.196.58 7672 138 0: 28 1 10/1 1: [137,100])

This example is likely a worm. 138 UDP packets were sent and none were returned. 28 ICMP unreachables were returned. The only port used is a Microsoft file share port, and we see that all 138 packets were aimed at port 137. Ports 137 and 1434 (SQL) should be viewed with suspicion. In real life, the UDP report suggests that scanning and other anomalies are not as common as TCP-based equivalents, but they do exist. Low-level scans (with low weights) for ports 137 and 1434 are common.

Given the apparent sporadic nature of UDP attacks (compared to 7x24 TCP attacks), we have three additional mechanism for help in analysis. This includes the udpweight graph discussed in the next section, the system system event log and the new trigger system, which can be setup to automatically capture the top UDP weight based on a pre-configured threshold. See triggers - automated packet capture for more information on the latter.

The system event log is automatically configured to capture any UDP anomaly in this list with a weight greater than 10,000,000. Basically omupdate.pl stores the first line of the ASCII UDP port signature report in the event log if such an event occurs. The threshold in question is configured in bin/omupdate.sh as a parameter (-u) passed to omupdate.pl and may be changed if so desired.


udpweight graph

The udp weight graph shows the weight of the single UDP error list top talker per sample period. This allows you to more easily estimate the time of a UDP attack and also to gain some feeling for the statistical variance in the weights shown in the UDP port signature report. Thus you may be able to decide to increase or decrease the UDP weight threshold passed to omupdate.pl or given to the ourmon probe to trigger automated packet storage. See the previous section for more information.

topn scans

The following links show output associated with the topn scan lists:
ip_scans
ip_portscan
tcp_portscan
udp_portscan

The topn_scans filter (producing graph ip_scan above) counts packets sent from a unique IP source to multiple IP destination addresses during the sample period. Therefore we can say it is 1-N in terms of IP source to IP destination mapping. Here we sort on the maximum unique IP destinations.

The topn_port_scans filter presents three separate graphs, but in general, looks at single IP sources sending packets to many unique L4 TCP and UDP destination ports. We sort on the maximum L4 destination ports. There are three graphs because the ip_portscan graph counts both TCP and UDP ports (and does not discriminate between the two), while the tcp_portscan and udp_portscan graphs only show TCP and UDP destinations respectively. Thus both topn_scans and topn_port_scans are 1-N in terms of their basic mapping.

The ourmon.conf configuration is setup as follows:

topn_scans 20
topn_port_scans 20

The number supplied should vary from 10..100 by values of 10. 20 will produce two topn scan graphs with the usual histogram. There are four mon.lite outputs (discounting STATS which are used for internal tuning). mon.lite output is as follows:

ip_scan: 18908 : 10.0.0.1: 2340 : 10.0.0.3 : 2295 : ETC.
ip_portscan: 18563 : 10.0.0.1: 2366 : 10.0.0.4 : 2242 : ETC.
tcp_portscan: 16303 : 10.0.0.1: 2366 : 10.0.0.4 : 2242 : ETC.
udp_portscan: 2526 : 10.0.0.2 : 242 : 10.0.0.5 : 182 : ETC.

The topn_scans filter produces one output called ip_scan. The topn_port_scans filter produces three outputs, called ip_portscan, tcp_portscan, and udp_portscan, respectively. Each output has the number of tuples following the tag, and each tuple is a 2-tuple of (ip source address, unique destination count). The destination count is the number of unique IP or L4 port destinations.

Therefore the graph representation is not any different from the mon.lite outputs. We simply present an IP source address, with its respective destination count. Results are sorted in order of unique destinations. It should be pointed out that information here may be correlated with information shown in the UDP errors or TCP syn top N mechanisms.


topn flows and anomaly detection

The following links show various aspects of the topn flow mechanism and anomaly detection:
topn_ip flow RRDTOOL count graph
topn_ip insert count
topn_icmp
topicmp_today.txt

The top N flow mechanism may also be of use for network security in terms of anomaly detection. There are fundamentally two different sets of graphs shown here. First of all, it has proven very useful to use a RRDTOOL graph to display the count of all four kinds of flows. (The count is the tag that follows the flow tag in the mon.lite file. It is the count of unique flows of that type, of which only the top N are provided as tuples in the mon.lite output). We call this the "flow_count" graph. This graph shows small and large attacks and classifies them as to whether they are TCP, UDP, or ICMP in origin. It shows the attacks simply because "scanning" means variation in IP destination, and/or L4 ports (or ICMP major/minor numbers) which are classified as separate flows as a result, even though they may be flows of one packet only. This graph has proved to be a particular good indicator of network-wide scanning attacks, including single instances of worms like the UDP slammer. The companion graph, "topnstat.html", show the counts of inserts in the flow hash list, which again are due to separate flows. Inserts result in malloc(3) calls and this graph also tracks attacks fairly well. (However long term it may prove to be redundant to the flow count graph).

In addition, it should go be pointed out that the ICMP top N flow graph may be useful. In particular, the ICMP flow "top of top" report has a tendency to reveal hosts that are engaged in long term scanning simply because they pile up extraordinary amounts of errors. (It may also show a long term ping of course). These scanners may be both TCP or UDP-based. In the TCP case, errors may occur due to administrative prohibited ICMP unreachables, ICMP redirects, or TTL exceeded errors. In the UDP case, UDP scanners, may in addition pile up ICMP port unreachable errors over time. As a result, the top of the top report here is useful for detecting scanning systems as well. ICMP unreachables, TTL exceeded, and routing redirects may be treated with suspicion.


BPF graphs and network anomaly detection

A number of BPF filter sets are provided in the default configuration that show overall network error and control information.

BPF network errors

The graph above shows a total count of TCP resets, ICMP unreachables, ICMP pings, and ICMP ttl exceeded errors. TCP resets often correlate with TCP scanning attacks. ICMP unreachables may correlate with a UDP attack.

BPF view of ICMP unreachable packets

The bpf-unreachable BPF filter set breaks out a number of different kinds of ICMP unreachable errors. Network, host, port, and administrative prohibited unreachable packet counts are shown. Keep in mind that UDP-based attacks may cause large numbers of port unreachables. TCP and UDP attacks may produce administrative unreachable errors if ACLs in Firewalls or Routers return those kinds of errors. Yes, Virginia, it might just be a good idea to do that to detect scanners, as opposed to returning nothing.

BPF view of TCP control packets

The bpf-tcpcontrol BPF filter set shows a breakdown of network-wide TCP control packets, including SYNS, FINS, and RESETS. This may be of use for spotting SYN anomalies as well as other kinds of anomalies. It can sometimes be possible to spot an attack on this graph, and then look through the topn_syn logs (as well as the tcpworm.txt and tworm graphs) to determine more information about the source of an attack, possibly including IP source addresses.

Triggers - automated packet capture

The automated trigger mechanism allows the probe to dynamically store a set of "interesting" packets which may be filtered internally with various BPF expressions and then written out to a tcpdump style dump file.

Typically a trigger event occurs when a runtime current value as seen with a graphics-engine graph exceeds some administratively set ourmon probe configuration-time threshold integer. The threshold integer is calculated in terms of back-end values, either in bits per second, or packets per second. A per trigger type dump directory must be specified (say /usr/dumps) and created before the ourmon probe is started. Each runtime trigger when enabled places a unique instance of a tcpdump dumpfile filled with the specified count of packets in an instance file. The instance file name is always of the form:

trigger_tag.timestamp.dmp.

The timestamp combined with the trigger_tag makes the filename unique. In general, only one trigger of each trigger type may be active at a time. In this release, there are two kinds of triggers. At this time we include a trigger for tworm events and an additional trigger for UDP weight (udp error) events. More triggers types may be released in the future.

Each trigger has some sort of implicit BPF expression capability associated with it, which is not user specifiable at this time. For example the UDP weight trigger dynamically determines the top IP host address, and stores count packets of the form: "host IP and (udp or icmp)", thus storing UDP packets and any ICMP errors associated with the host IP that caused the trigger to activate.

Current trigger types include:

tworm - a trigger may be set on the total worm count.

topn_icmperror (udplist) - a trigger may be set on the top udp error weight for a specific IP host.

bpf_trigger - a trigger may be set on any BPF expression in a BPF filter set.

drop-trigger - a trigger may be set on the pkts filter when the number of drops exceeds a certain threshold.

It is important to note that triggers may be more useful in terms of the packets captured if the particular BPF expression is more specific. This problem can be called the "signal to noise problem". For example, a BPF expression that captures all TCP packets may not show anything useful. On the other hand, the udp error weight trigger only captures UDP packets for a particular IP source. Thus the tcpdump capture file in question is more likely to show the exact nature of an attack.

In ourmon.conf, triggers in general should be placed at the end of the configuration file as they assume that certain filters exist and are turned on. They have the following general syntax:

trigger_name threshold packet_count dump_dirname.

threshold - specify a threshold in bits/sec or pkts/sec or units depending on the trigger type. When the named filter reaches that threshold, packet capture will begin. The threshold is always specified in terms of a backend graphics-engine value.

packet_count - terminate when count packets are stored.

dump_dirname - directory in which to store the dump file. This directory should be created before the ourmon probe is run.

Packet dumping is turned off when either the count is exceeded or the trigger threshold flow count is less than the current runtime value -- which ever comes first. Packet dumping after being triggered cannot recur between trigger on and trigger off times. For example assume the packet capture count is 10000, and the threshold value is 2000. If at mon.lite creation time, the value is 2000, a trigger_on event is started and 10000 packets may be stored. If the count is exhausted and the threshold is still > than the value, no more packets will be stored in the file. Thus it is not possible for one trigger_on event to cause more than one output file concurrently. When an event is retriggered, a separate file with a different timestamp is created.

Trigger_on and trigger_off messages are put into mon.lite as "elog" messages and are recorded in the back-end daily event log. As a result it is possible to easily see when trigger on and off events have occurred. The trigger dump filename is passed to the event log. Note that if the probe and back-end graphics engine run on separate computers, due to the imprecision of synchronization, a given on or off message may NOT appear. However usually both messages will appear, or at least the on or off message will appear. Trigger messages in the event log appear as follows:

Tue Mar 1 09:37:01 PST 2005: ourmon front-end event: tworm trigger on, current count: 45, threshold 40, dumpfile: /usr/dumps/tworm.<03.01.2005|09:36:53>.dmp

Tue Mar 1 09:38:01 PST 2005: ourmon front-end event: tworm trigger OFF, current count is 20, threshold: 40

The trigger on message means that the trigger for tworm is on because the current total "worm" count is 20 and has exceeded the configured threshold of 10. The trigger off message indicates that packet capture is done. The trigger will capture and store packets into the file:

/home/mrourmon/dumps/tworm.<03.01.2005|09:36:53>.dmp.

The contents of this file may be viewed with tcpdump. Since the timestamp typically includes shell metacharacters one can usually cut and paste the timestamp name in between double quotes (or just use * globbing) as follows:

# tcpdump -n -r "tworm.<03.01.2005|09:36:53>.dmp"

Note: at this time ourmon has no way to determine if the dynamically created file has been created successfully or if there is runtime storage room within the file system in question. If the file cannot be created an elog message will be sent. The ourmon probe will not exit.


tworm trigger

In the ourmon configuration file this trigger has the following syntax:

trigger_worm threshold packet_count dump_dirname

The trigger_worm trigger is associated with the tworm graph . One should determine a suitable configuration threshold by watching this graph over time. Note that the threshold applies to the tworm total count (not us, not them, the total ). If the total value for tworm counts (us+them) exceeds the supplied threshold value, a packet dump will begin for count packets in the dump_dirname in a dynamically created filename. Only TCP SYN packets are stored. The filename has the form:

tworm.< timestamp >.dmp.

It can be extremely useful to use the saved back-end portreport.txt file for analysis here (see logging below). The relevant TCP port report file here may give important clues about the nature of the attack in terms of TCP destination port numbers, the number of IP destination addresses, or the IP source addresses involved in the attack. This information may be useful in helping both to ignore irrelevant TCP syns gathered during the attack and for searching the tcpdump file for the IP addresses or ports in question (simply tack on a BPF expression at the end of the tcpdump search).

For example, assume that the portsignature.txt file shows that port 6666 was attacked, then one can do:

# tcpdump -n -r "dumpfile" tcp port 6666

This helps to narrow down the packet dump to a more relevant set of packets.


udp weight trigger

In the ourmon configuration file this trigger has the following syntax:

udperror_trigger threshold packet_count dump_dirname

This trigger is based on the udperror_list (topn_icmperror). More information can be found in the following sections:

icmp and udp error list
udp weight graph
udp port signature report

In terms of the back-end the threshold value is simply the UDP weight calculated for the first host in the udperror_list. This value is graphed in the udp weight graph . We suggest that the threshold be set at 10000000 to start. It should be tuned down or up depending upon the incidence of false positives versus "interesting events". In general UDP scanning does occur and this mechanism will catch outrageous incidents. It may also catch disfunctional P2P applications or multimedia games. Of course what you do with that information, will depend upon your security policy.

Note that this threshold should be set the same both in the front-end config file and in the backend omupdate.sh file. The latter file causes the offending line from the UDP port signature report to be placed in the elog event file. As a result it becomes simple to identify the IP source address of the offending UDP system. Also the UDP destination ports in question are also supplied.

The capture filename has the form:

topn_udp_err.< timestamp >.dump

This trigger will capture and store UDP packets sent to and from the top host in udperror_list. The internal BPF expression used is: "ip_src and (udp or icmp)".

The event log entries for topn_udperror will appear roughly as follows:

Fri Mar 4 01:17:33 PST 2005: udpweight threshold exceeded:10.0.0.1 14841330 7284 1194 1218 0 10: [1649,0][7674,80][22321,18][33068,0][40167,0][54156,0][55662,0][61131,0][64734,0][ 65021,0]

Fri Mar 4 01:17:33 PST 2005: ourmon front-end event: topn_udp_err: trigger on, current count: 14841330, threshold 10000000, dumpfile: /usr/dumps/topn_udp_err.<03.04.2005|01:17:05>.dmp

There are two event log messages associated with the UDP error event. The first message is simply the first line of the udp port report and is generated by the backend. The offending IP is given, along with various statistics taken from the UDP port reporting including ports. The second line is generated by the probe and gives the trigger capture filename. Note that since udp port signature files are also logged, it may sometimes be useful to refer to the port signature file at the time of the attack. Given that the port signature line for the incident in question is in the log this may not be the case. On the other hand, a remote multiple system attack is always a possibility and thus the port signature file could provide additional details.


bpf trigger

In the ourmon configuration file this trigger has the following syntax:

bpf_trigger "filterset_label" "bpf_line_label" threshold packet_count dump_dirname

The BPF trigger capability works with any individual BPF expression in a BPF filter set. The bpf_trigger that matches a BPF expression must be declared in the config file after the filter set declaration. The trigger expression first names the filter set and then names the BPF line label as a two-tuple - thus determining the BPF expression used for the trigger. All packets stored match the BPF expression. The threshold value is either packets or bytes depending upon the declaration of the filter set and as always is expressed in terms of back-end graphics output. If the user bpf filter set was declared as follows:

bpf-packets
bpf "protopkts" "ip" "ip"
bpf-next "tcp" "tcp"

and the desired target BPF expression is "tcp", then the bpf_trigger to match that expression would be:

bpf_trigger "protopkts" "tcp" 1000 10000 /usr/dumps

which means that the trigger should be turned on to store 10000 packets when a threshold of 1000 packets per second is reached.

The dump filename is created by concatenating the major and minor labels along with a prefix of "bpf" and a timestamp to make the dump file unique. For example, a dump for the above could have the following name:

bpf_protopkts_tcp.<06.03.2005|01:02:37>.dmp


drops trigger

In the ourmon configuration file this trigger has the following syntax:

drop_trigger threshold packet_count dump_dirname

The drop trigger is associated with the pkts graph . It is triggered by the (first) drops counter. The assumption here is that this trigger is not normally triggered, but only triggers due to large DOS attacks. It is possible that the trigger might thus catch a DOS attack (especially if the DOS attack is the major cause of packets). The supplied packet count value should probably not be small as it is quite possible that this filter might store "good" packets as opposed to bad packets.

The capture filename has the form:

drop.timestamp.dmp

The threshold for this trigger is based on packet counts.


IRC information

IRC information is presented on its own web page. IRC data is reported for two reasons, both security related. It has long been known that IRC may be used by "hackers" as the control plane for hosts that have been captured and become part of a remote botnet. Client botnet hosts may in turn be used for launching attacks on local (or remote) hosts including denial of service attacks, and "fan-out" attacks using various well-known exploits. Thus client IRC hosts may act as TCP-based scanning hosts (UDP activity is not ruled out). Second it is always possible that IRC may be used for the dissemination of "warez" (meaning multimedia data including video and audio). Whether this is allowed or not within an administrative domain depends on the local security policy. Ourmon looks at statistical data for IRC and does NOT look at PRIVMSG content. The main goal is to identify IRC channels that appear to be under the control of computers as opposed to people.

IRC information is enabled in the front-end probe by the ourmon.conf config tags:

topn_irc 0

topn_irc_file /home/mrourmon/tmp

These two switches should both be used and never be given separately. (0 is a placeholder for a possible future feature). The latter switch directs where the front-end output irc.txt file should be stored. It may be overridden by the use of the -D parameter with the ourmon probe. The ourmon probe uses a lightweight layer 7 scanner to look for certain IRC messages including JOIN, PRIVMSG, PING, and PONG and gathers a small set of RRDTOOL statistics based on global counts for those messages. It also gathers two sets of tuples including irc host tuples and channel tuples. The host tuple contains statistical information about IRC usage for a given host participating in IRC communication. The channel tuple contains statistics about IRC channels including the set of hosts participating in the channels. IRC information gathering is not based on any TCP destination port and ignores ports. The information produced also includes data taken from the topn Syn module and includes a TCP syn work weight for each IRC host. Thus it is possible to determine if an IRC host may be infected by a worm/scanner/bot. Tuple information for IRC hosts and channels is placed in the per sample period irc.txt file and handed over to the back-end for further processing.

The web page has several kinds of data including:

1. The p2p port signature report . See that section above for more information. The p2p report is included here simply because it also includes information about IRC hosts. IRC hosts are given an I as a flag in the application flags section. Note that this file thus contains all IRC hosts visible during the sample period.

2. The next section gives a processed version of the raw irc.txt file. (This work is done by bin/irc.pl). A 30-second sample period report is given, followed by a week's worth of daily summarizations. Note that the first summarization is for the current day and as usual is performed hourly during the current day. It is highly likely that the summarizations for the current day and yesterday are the most important items due to the sporadic and slow communication patterns for IRC as a whole. We will explain the file format for the IRC output report below in more detail.

3. RRD graphs for total IRC message counts are then provided. Note that JOIN spikes here may indicate some sort of IRC "war" amongst competing IRC groups.

The IRC summarization format may be broken down in the following way. First the file itself consists of three sub-sections including:

1. global statistics, 2. channel statistics, and 3. IRC host statistics.

Channel statistics are given in major sections including:

channels sorted by wormy hosts
channels sorted by max messages
channels with associated host IPs (and host stats)
channels with no PRIVMSGS, only JOINS
channels with any wormy hosts

Host statistics follow:

most busy server in terms of max messages
hosts with no PRIVMSGS, only JOINS
hosts with any sign of worminess

The statistical layout for various kinds of formatting is explained below:

Channel statistics sans host statistics:
channel msgs joins privmsgs ipcount wormyhosts evil?
exploit 3 0 3 5 4 E


We have a channel named "exploit" with a total of 3 messages (all PRIVMSGS). Five hosts appeared in the channel and 4/5 had a work weight greater than 60%. (The work weight in the IRC summarization is not an average but the maximum work weight seen as we are looking for any signs of suspicious behavior). An E flag is awarded if a channel has more than one wormy host and at least half of its hosts (discounting servers which may be botnet controllers) are infected. The small e flag is given if a channel only has one host with a "bad" work weight.

Channel statistics with host statistics:
channel ip_src tmsg tjoin tping tpong tprivmsg maxchans maxworm Server?
foo
10.0.0.1 161 30 67 63 1 1 99 S

The foo channel has one host in it with a total message count of 161 messages broken down into total counts for JOINS, PINGS, PONGS, and PRIVMSGS. The total number of channels for the host in question is shown. The maximum work weight for all instances is also shown. We also show whether we believe the host is an IRC server or IRC client. Note that a real channel/host breakdown minimally has at least two hosts in it (one client and one server).

The most important sections of the IRC summarization in security terms are probably the very first channel sub-section that sorts channels based on the number of wormy hosts and the very last part of the report that shows wormy hosts themselves. We call channels with too many wormy hosts "evil channels". This simply means a channel that may be botnet controlled. The second important security section appears at the absolute bottom of the irc summarization and gives a list of hosts that may be infected according to the work weight.

Channel names in IRC are case insensitive. Thus it is possible that a channel named "exploit" and "EXPLOIT" are the same channel. This is probably true if they have the same set of IP hosts associated with them. At this time our report mechanism makes no attempt to join these channels into one channel. This may be deemed a bug. However our reason for not doing so is to simply show how the channel name is being spelled. This may be fixed in a subsequent release.


Ourmon Logging

Logging in ourmon takes various forms.

1. RRD data is stored in the base back-end rrddata directory. The file rrddata/ourmon.log records RRD errors and should be consulted when one is attempting a new user BPF filter-set, especially when the filter-set graphics do not appear!. The RRD data gives a year's worth of baselined data for RRD graphics. Note that RRD data files reach their maximum size at creation time and do not grow larger over time.

2. The event log gives significant events in the life of the ourmon system including probe reboots and important security events. More information is available below.

3. Web summarizations represent hourly (and eventually daily) reports available via the ourmon web pages. For example various top N summarizations and IRC summarizations are available. Most summarizations are available at the bottom of the
main web page . IRC summarizations are on a separate page. In general one week's worth of summarizations are available.

4. Basic log information is stored in the back-end logs directory. This information includes top N logs, front-end files like mon.lite, and important security files including the TCP port signature report in the logs/portreport directory, and the UDP port signature report in the logs/udpreport directory. Back end summarization scripts run at hourly intervals and summarize this data displaying some (but not all) of it in the form of web-pages including IRC, top N, and other kinds of summarizations. Note that after running ourmon for one week, in general the logs directory does not grow any bigger as the current day is rolled over at midnight (and the new data is zeroed out). Note that log files are typically NOT web summarizations. They represent 30-second sample reports. Taken together they can be used to produce a web summarization which typically is done on an hourly or daily basis. At this time the back-end logs directory includes the following:

portreport - these are the back-end processed TCP port signature report files (portreport.txt) and may be crucial to security analysis.

udpreport - these are the back-end processed UDP port signature reports. These files may be useful for security analysis.

p2preport - these are the back-end processed TCP syn p2p portreport files and show which systems are doing IRC, and various forms of p2p activity.

daily files (Mon Tue etc) - these are top N log files and in some cases are used for daily summarization.

irc - these are the back-end processed per sample irc report files.

rawirc - these are the front-end irc.txt files and are used for the back-end web summarization.

mon.lite - these are the front-end mon.lite files.

tworm - these are the front-end tcpworm.txt files.


Event Log

The event log records important events in the life of ourmon including probe reboots and nasty back-end errors. It also icludes important security events that in general should not occur too often (we hope). At this time security events include:

1. tcp worm trigger events

2. udp weight trigger events

3. udp weight port signature threshold events which provide the IP source address of the UDP offender.

The event log is summarized in a weekly summary . The current daily event log is updated hourly and at midnight becomes the event log for the previous day.


Web summarizations

Web summarizations are produced for some top N data and other data as well. Summarizations are a form of logging in which statistics are aggregated together and summarized in various ways. Daily summaries are updated at the hour and typically rolled back a day at midnight producing a week's worth of summaries. Summarized data is stored in the ourmon web directory of course and not in the logs directory. However the logging directory data is used to produce the web summarization.

Summarized data at the bottom of the main web page includes:

Other summarizations exist and are discussed elsewhere including the IRC data and the TCP scanner log .

Security Analysis and Logging

A couple of tricks that can be used for searching the log directory data should be pointed out. For example assume that you have an IP source address of interest and you wish to search the TCP portreport logging directory to see if that IP source is mentioned. Assume that you know friday is also a day of interest.

# cd /home/mrourmon/logs/portreport/Fri
# find . | xargs grep 10.0.0.1

The use of xargs here combined with grep allows you to easily search the log directory for the IP source address 10.0.0.1. This technique of course can be used in other logging directories as well.

Another search technique may be of use specifically in the portreport directory. If a large external multiple host attack occurs one can easily find the associated port report simply by sorting on the size in lines of the port report file. This is because each IP source address in the port report gets its own line.

# cd /home/mrourmon/logs/portreport/Fri
# find . | xargs wc -l | sort

Wc is used to count the lines in the file, and sort will sort out the results showing you the biggest file.