NfSen - Netflow Sensor

What is NfSen?

NfSen is a graphical web based front end for the nfdump netflow tools.

NfSen allows you to:
Different tasks need different interfaces to your netflow data. NfSen allows you to keep all the convenient advantages of the command line using nfdump directly and gives you also a graphical overview over your netflow data.
NfSen is available at sourceforge and distributed under the BSD license.

Table of Content

  1. NfSen - Netflow Sensor
    1. What is NfSen?
    2. Table of Content
  2. Screenshots
  3. Installing NfSen
    1. Prerequisites:
    2. First Installation
      1. Master Config File:
      2. Importing existing netflow data
      3. Start-Stop NfSen
    3. Upgrading NfSen
  4. Working with NfSen
    1. Views
    2. Profiles
      1. Profile Types
      2. Creating profiles
      3. Managing Profiles
    3. Navigation
      1. Selecting a time slot or a time window
      2. Statistic Summary
      3. Netflow Processing
        1. Default Filters
    4. Bookmarks
    5. Command line tool 'nfsen'
  5. Plugins
    1. Installing Plugins
    2. Writing Plugins
    3. Testing Plugins
    4. Optional Modules
      1. Notification.pm

Screenshots

NfSen Overview Page NfSen - Overview Flows
NfSen - Netflow Navigation
NfSen - General Overview Page
NfSen - Flow Overview Page
NfSen - Navigation Page
NfSen - Processing output
NfSen - Profile Info
NfSen - Netflow Processing output
NfSen - Profile Info



Installing NfSen

Prerequisites:

First Installation

NfSen is installed in its own directory tree. This directory is called BASEDIR. All NfSen related data as well as all netflow data files are stored below this BASEDIR. So make sure you have enough disk space for BASEDIR, or at least for the profile directory.
Directory Structure
Directory Structure

If you have installed all prerequisites, change to the etc directory and copy the NfSen template config file nfsen-dist.conf to nfsen.conf.
Edit nfsen.conf according your needs and setup:

Master Config File:

#
# NfSen master config file
#
# Configuration of NfSen:
# Set all the values to fit your NfSen setup and run the 'install.pl'
# script from the nfsen distribution directory.
#
# You should not need to changes anything after NfSen is installed,
# besides the NfSen plugins at the bottom.
# When you make any changes in the plugins section, run 'nfsen reload'
# to make sure nfsen-run reads the new configuration.
#
# Whenever you change any other settings in this file
# you must rerun the install.pl script.
#
# The syntax in this file must conform to Perl syntax.
#
# NfSen base directory: Home of NfSen
# contains all scripts, modules and profile directories:
# bin, libexec, var, profiles
$BASEDIR = "/data/nfsen";

# nfdump tools path
$PREFIX = '/usr/local/bin';

#
#
# NfSen master config file
#
# $Id: NfSen.html 25 2005-04-01 12:08:09Z peter $
#
# Configuration of NfSen:
# Set all the values to fit your NfSen setup and run the 'install.pl'
# script from the nfsen distribution directory.
#
# You should not need to changes anything after NfSen is installed,
# besides the NfSen plugins at the bottom.
# When you make any changes in the plugins section, run 'nfsen reload'
# to make sure nfsen-run reads the new configuration.
#
# Whenever you change any other settings in this file
# you must rerun the install.pl script.
#
# The syntax must conform to Perl syntax.
#
# NfSen base directory: Home of NfSen
# contains all scripts, modules and profile directories:
# bin, libexec, var, profiles
$BASEDIR = "/data/nfsen";

# nfdump tools path
$PREFIX = '/usr/local/bin';

#
# Run nfcapd under this user
# This may be a different or the same uid than your web server.
$USER = "netflow";

# user and group of the web server process
# All netflow processing will be done with this user
$WWWUSER = "www";
$WWWGROUP = "www";

# Receive buffer size for nfcapd - see man page nfcapd(1)
$BUFFLEN = 200000;

# Netflow sources
# Define an ident string, port and colour per netflow source
# ident identifies this netflow source. e.g. the router name,
# Upstream provider name etc.
# port nfcapd listens on this port for netflow data for this source
# col colour in nfsen graphs for this source
#
# Syntax:
# 'ident' => { 'port' => '<portnum>', 'col' => '<colour>' }
# Ident strings must be 1 to 19 characters long, only containing characters [a-zA-Z0-9_].

%sources = (
'upstream1' => { 'port' => '9995', 'col' => '#0000ff' },
'peer1' => { 'port' => '9996', 'col' => '#ff0000' },
);

#
# NfSen html pages directory:
# All php scripts will be installed here.
# URL: Entry point for nfsen: http://<webserver>/nfsen/nfsen.php
$HTMLDIR = "/var/www/nfsen/";

#
# Low water mark: When expiring files, delete files until
# size = max size * low water mark
# typically 0.9
$low_water = 0.9;

#
# syslog facility for periodic jobs
# nfsen uses level 'debug', 'info', 'warning' and 'err'
# Note: nfsen is very chatty for level 'debug' and 'info'
# For normal operation, you may set the logging level in syslog.conf
# to warning or error unless you want to debug NfSen
$syslog_facility = 'local3';

#
# plugins
# plugins are run for each timeslot, after the roll over of new data files.
# A plugin may run for any profile or for a specific profile only.
# Syntax: [ 'profile', 'module' ]
# profile: ',' separated list of profiles, or "*' for any profile
# module: Module name.
# The module follows the standard Perl module conventions, with at least two
# additional functions: Init() and run(). See demoplugin.pm for a simple template.
@plugins = (
# profile # module
# [ '*', 'demoplugin' ],
);

#

#
# Notification module
# The Notification module is an optional module. If you want your plugins to
# notify any result by email, use this module.
# Make sure you have installed Mail::Internet before using the module
#
# Use this from address
$MAIL_FROM = 'your@from.example.net';

# Use these recipients
$RCPT_TO = 'any@example.net, another@example.net';

# Use this SMTP server
$SMTP_SERVER = 'localhost';

1;


When you are done with nfsen.conf, run the install.pl script in the NfSen distribution directory:
./install.pl etc/nfsen.conf

Running install.pl will:
After the installation, you will find the nfsen.conf file under BASEDIR/etc. This documentation, located in the doc directory is not installed by default. You may copy this directory into the HTMLDIR for direct access or keep a local copy on your desktop. If you want this document available as Help link, copy the doc directory into the HTML dir and uncomment line 18 in navigation.php:

// print "<a href='doc/NfSen.html' target='_blank' >Help</a>\n";

Importing existing netflow data


Import existing dataIf you have existing netflow data from nfdump, follow these steps: The live profile is now setup with your existing data. You can verify your profile: ./nfsen -l live. The status of the profile is set to 'rebuild' which will change automatically to 'OK' the first time the periodic task of nfsen-run is executed.

Make sure to disable your current start/stop script, as NfSen will provide it's own start/stop script to start all required nfcapd processes, as well as the nfsen-run background process. The nfdump cleanup script nfclean.pl isn't  needed either, as expiring the netflow data is fully integrated into NfSen.

Start-Stop NfSen

NfSen provides a start-stop script nfsen.rc in BASEDIR/bin/. You may create a soft link from your appropriate rc.d directory to this file.
To start NfSen:

BASEDIR/bin/nfsen.rc start
This starts all nfcapd processes to collect the netflow data and the nfsen-run background process to update your profiles, as new data becomes available. Point your web browser to nfsen.php. ( Typically http://yourserver/nfsen/nfsen.php ).

The background task nfsen-run as well as nfcapd log to syslog. nfsen-run is very chatty, when specifying 'info' or less as syslog priority. You may want to set the syslog priority to 'warning' for normal operation. For debugging purpose, use 'info' or 'debug'.


Upgrading NfSen

To upgrade your current NfSen installation:
Change to the directory where you unpacked the NfSen version and run:
./install.pl <path/to/your/existing/nfsen/etc/nfsen.conf>
then
BASEDIR/bin/nfsen reload

This will update your current NfSen installation without affecting your netflow data.


Working with NfSen

NfSen has two different user interfaces:
Most of the time you will want to use the web interface. However, you can do everything from the command line as well.

Views

NfSen views
Tab Navigation

NfSen offers different views.  Each of the views can be selected using the tabs at the top of the page. When you point your browser to the nfsen.php page, The  'Home' view is the default view and shows an overview of the currently selected profile. The three columns show the 'Flows', 'Packets' and 'Bytes' history. To select a different view, click into any graph or select the view in the tab, e.g. clicking into the column of the bytes history switches to the bytes view. Clicking into any graph in the 'Flows', 'Packets' or 'Bytes' view switches to the 'Detailed' view for a further analysis of the netflow data.
If the currently selected profile is a continuous profile, the history pages are automatically refreshed every 5 minutes to update the graphs. This allows you to have a browser window on your screen, with always up to date graphs.
Detailed information about the currently selected profile is available under the 'Stats' tab.

Profiles

A profile is a specific view on the netflow data. A profile is defined by its name, type and profile filter, which is any valid filter accepted by nfdump.
At least the profile 'live' is always available and is used to store your incoming netflow data without filtering. You can switch back and forth to any profile using the pull down menu in the upper right corner of the web page.

Select Profile
Profile Selection

Profile Types

A profile can be either of type History or Continuous.  A history profile starts and ends back in the past and remains static. It neither grows nor expires. A continuous profile may start in the past and is continually updated while new netflow data becomes available. It grows dynamically and may have its own expire values set. Old data expires after a given amount of time or when a certain profile size is reached.

Creating profiles

Select the "New profile ..." entry in the profile pull down menu.
New Profile FormComplete the 'New Profile' form to start building the profile. The profile type is automatically detected according the 'Start' and 'End' values you enter. The help text should guide you through the process of creating the profile.
As profiles are created from netflow data from profile 'live', the start and end of the profile must fall in the time range of the profile 'live'.

Profile Created
Successful creation of new profile.

When the profile is successfully created, the build process starts. Depending on how long back in the past the profile starts, this can take a considerable amount of time. You can follow the build process by clicking 'Continue' or at any time later by selecting the 'Stat' tab of the new profile. On the top of the Status information you will see a progress bar, showing you the percentage of completion.

Progress of Building a Profile

Progress of building the profile


Managing Profiles

Once a profile is created you can change the expire settings of a continuous profile. Select the 'Stat' tab of the profile and click on the edit icon Edit of the appropriate expire value. A continuous profile may expire due to the age of the data or the profile size used on disk. Expiring starts whenever one of the two limits is reached. Expiring ends at the configured value $low_water in the config file nfsen.conf.

Delete Profile

Delete Profile

To delete a profile, click on the trash can on the upper right corner of the profile info table. You will have to confirm to delete the profile:

Confirm Delete Profile
Confirm Delete Profile


Navigation

Detailed navigation and netflow processing is done in the 'Details' view. When entering this view, you will see the navigation display.

Netflow Navigation
Navigation Display
The page is divided into two parts: The upper part allows you to navigate through the netflow data as well as selecting a time slot or time window. The lower part contains all the controls to process the netflow data of the selected time slot or time window.
Clicking on any of the small protocol graphs will replace the main graphics with the selected protocol graph. You can switch back and forth and select the protocol for the main graph, which is appropriate for investigating your current situation. Clicking on the small type graphs on the right will replace the main graph as well as the protocol graphs with the selected type. Therefore you can switch to the 'Flows' 'Packets' or 'Bytes' graphics according your needs.

The time span of the graph can be changed using the pull down menu.
Graph Scale
Select Time Span

Selecting a time slot or a time window

A time slot starts at every 5 minutes cycle of the hour ( 0, 5, 10, 15 etc. ) and lasts 5 minutes. A time window consists of several time slots.When entering the 'Details view' a window scale of one days is selected so you will see the last 24 hours of the profile. The time cursor is placed in the middle of the begin and end of these 24 hours and the time window slot is set to one time slot. You will see the selected time slot or time window always in the title of the browser window as well as in the title of the main graph. Selecting a different time slot can be done in a number of ways:

The graphs are immediately updated, when selecting a different time slot.
Sometimes it is desirable to process more than a single 5 min timeslot. The time range of interest an be selected by extending the time window. First, place the time cursor on the left edge of he requested time window, using the methods described above. Then, select the right edge of the time window by:
Select right mark
Select 'right' Mark from the pull down menu  and click into the main graph.

or
Enter right boundary
Enter the the right boundary in the tend input field and press the enter key.

The main graph will be immediately updated with the selected time window:
Selected Time Window
Selected Time Window
Note:
To reset the window to the default size use the button below the tstart, tend input boxes.
When you move the timeslot towards the begin of the profile, a grayed out area in the main graph appears and shows the end of the available netflow data. No data exists in the gray area. This border moves dynamically, when data expires. The run of the flows, packets and bytes graph may still be available, as this data is stored in the RRD database.
Out of Data
Border of available Data

Statistic Summary

The statistic summary below the main graph gives you an overview about flows, packets and bytes of the selected time slot or time window.
Stat Summary
Each line corresponds to one configured netflow source. If you are interested in only some of the sources, you may remove the others by clicking  the checkboxes. This disables or enables this source in all graphs and in the statistics respectively.
Disabled Sources
Disabled sources 'Downstream' and 'Peer1'
As this rescales the graphs according the remaining sources, you get a more detailed graph and a different resolution on the y-axis.

Netflow Processing

Once you have selected the time window of interest, you can process and filter the netflow data according your needs, using the process form.
Processing Form
Click 'process'.

The sources, the filter as well as all options from the processing form are compiled into the appropriate nfdump command. For convenience a short description of the filter syntax and options follows. More details are available in the nfdump(1) man page.

Filter Syntax

The filter syntax is similar to the well known pcap library used by tcpdump.  The filter can span several lines. Anything after a '#'  is treated as a comment and ignored to the end of the line. There is virtually no limit in length of the filter expression. All keywords are case independent, unless otherwise noted.
Any filter consists of one or more expressions expr. Any number of expr can be linked together:
expr and expr, expr or expr, not expr, ( expr ).

Expr can be one of the following filter primitives:

Any
any Flows. Used as dummy filter. Use 'not any' to block all flows.

protocol
TCP, UDP, ICMP or PROTO num where num is the protocol number.

IP address
[SourceDestination] IP a.b.c.d or HOST a.b.c.d with a.b.c.d as any valid IP address.

[SourceDestination]
defines the IP address to be selected and can be SRC DST or any combination of SRC and|or DST.
Omitting SourceDestination is equivalent to SRC or DST.

network
[SourceDestination] NET a.b.c.d m.n.r.s
[SourceDestination] NET a.b.c.d/num with a.b.c.d as network number, m.n.r.s as netmask or num
as maskbits. The network may be given as a.b, a.b.c, where a B or C-class
equivalent netmask is assumed.

Port
[SourceDestination] PORT [comp] num with num as a valid port number. If comp is omitted, '=' is assumed.

Flags
flags tcpflags
With tcpflags as a combination of:
A ACK.
S SYN.
F FIN.
R Reset.
P Push.
U Urgent.
X All flags on.
The ordering of the flags is not relevant. Flags not mentioned are treated as don't care. In order
to get those flows with only the SYN flag set, use the syntax 'flags S and not flags AFRPU'.

TOS
tos value
Type of service: Value 0..255.

Packets
packets [comp] num
Limit the packet count in the netflow record.

Bytes
bytes [comp] num
Limit the byte count in the netflow record.

[comp]
The following comparators are supported:
=, ==, >, <, EQ, LT, GT. If comp is omitted, '=' is assumed.

AS
[SourceDestination] AS num with num as a valid AS number.

Examples:
tcp and ( src ip 172.16.17.18 or dst ip 172.16.17.19)

tcp and ( net 172.16/16 and src port > 1024 and dst port 80 ) and bytes > 2048

Options

The list of the following options are used to compile the nfdump command. See also the nfdump man page for further details.
 
List Options
List First N Flows List only the first N flows of the selected time slot
nfdump option: -c N
aggregated Aggregate the flows: SRC IP, DST IP, SRC Port, Dst Port
nfdump option: -a
time sorted Sort all flows according the time first seen, when reading data from multiple netflow sources. Important when tracking a certain period of time.
nfdump option: -m
long output Include TCP flags and tos field in output
nfdump option: -o long
Statistic options
Stat Top N Limit the statistics to the first N
nfdump option: -n N
Limit Packets/Bytes After creating the statistics, only show aggregated flow with more/less packets/flows than specified.
nfdump option: +/-l num +/- L num
Packets/Bytes Create the bytes and packets statistics
nfdump option: -S
long output Include TCP flags and tos field in output.
nfdump option: -o long
SRC IP Addr. Create statistics about selected parameter.
nfdump option: -s <stat type>

Processing Output

Note:
Depending on the size of your network, netflow processing may consume a lot of time and resources, when you select a large time window and multiple resources.

Default Filters

Frequently used processing filters can be stored in a file under BASEDIR/var/filters. These filters will be available in the processing form. The name of the filter in the menu corresponds to the file name. The filters use the standard nfdump syntax.
Set a Default Filter
The default filter is combined with the filter you enter in the text box. Both filters are linked with logical 'and'. If a new installed filter is not visible in the menu after installing you may update the application cache by switching to the 'Stat' tab and back again.

Bookmarks

While working with NfSen, you may want to bookmark the current situation for later use or to send it as a link to a friend. The bookmark link at the top right of the page, allows you to do that.
Bookmark Link
Clicking on the link places the bookmark URL into the URL input field of your browser, allowing you to add this link to your bookmark collection. Many browsers also allow you 'right click' a link to copy the link location for pasting it in another application.

Command line tool 'nfsen'

The command line tool 'nfsen' in the BASEDIR/bin directory works hand in hand with the frontend. It's used to create and manage profiles as you can do with the frontend in the 'Stat' tab. Use nfsen -h to see all options available for nfsen:

bin/nfsen [options]
-h This help

-V Version of nfsen

-l <name>List profile <name>. Use -l all for all profiles.

-a <name>Add new profile <name>
[-c description
[-B Begin of profile.] Default: time now: continuous profile.
Format yyyy-mm-dd-HH-MM, or yyyymmddHHMM
[-E End of profile.] Default: continuous profile.
Format yyyy-mm-dd-HH-MM, or yyyymmddHHMM
[-S <sourcelist>] Default: All available sources.
Format ':' separated list of sources.
[-e <time>] expire Default: no profile expire time
[-s <size>] Size Default: no profile max size.
[-f <filter] Read filer from file <filter>
'PROFILE FILTER' Profile filter. Required unless -f <filter>

-d <name> Delete profile <name>

-m <name> Modify profile <name>
[-e <time>] expire Default: no profile expire time
[-s <size>] Size Default: no profile max size.
[-r] Rebuild profile WARNING: use with care!
[-U] Unlock profile WARNING: use with care!
[-L] Lock profile WARNING: use with care!

reload Reload nfsen-run process

If you create or delete a profile with the command line tool, the changes may not be instantly visible in the profile menu. Switching to the 'Stat' tab
updates the application cache and the profile menu.

Plugins

Even if NfSen is a flexible tool, you may have different or additional needs to process the netflow data. This can be done using the plugin feature provided by NfSen. A plugin is installed in 'nfsen-run', the background daemon, which keeps track with all the profile updates and data expiring. The plugins are Perl modules which are loaded when NfSen is started and run at every 5 min interval, when nfcapd rolls over the data files. This allows you the process new netflow data as it becomes available and trigger any action of your choice. So far plugins are not linked to the web frontend and therefore can not display any data in the frontend pages directly.

Installing Plugins

A plugin is expected to be stored in the BASEDIR/libexec directory. New plugins must be configured in the BASEDIR/etc/nfsen.conf file. The plugin configuration section is at the bottom of this file:
#
# plugins
# plugins are run for each timeslot, after the roll over of new data files.
# A plugin may run for any profile or for a specific profile only.
# Syntax: [ 'profile', 'module' ]
# profile: ',' separated list of profiles, or "*' for any profile
# module: Module name.
# The module follows the standard Perl module conventions, with at least two
# additional functions: Init() and run(). See demoplugin.pm for a simple template.
@plugins = (
# profile # module
# [ '*', 'demoplugin' ],
);

Once your profile is tested, installed and configured, signal the 'nfsen-run' daemon to integrate the new plugin:
BASEDIR/bin/nfsen reload
Have a look at the syslog file for errors when loading the plugins.

Writing Plugins

Writing plugins is as easy as writing Perl modules. The template for a plugin may at least look like the example below:
#
#
package PluginName;

use strict;
use NfConf;

sub run {
my $profile = shift;
my $timeslot = shift; # Format: yyyymmddHHMM

# Do whatever you want to do.
}

sub Init {
return 1;
}

1;

The plugin has to provide at least two functions: Init() and run(). Init() is called, when the plugin is loaded. You may do any plugin specific initialization work. Return 1 for a successful initialization and to add your plugin to the plugin list. Returning 0 disables your plugin and prevents the plugin from running.
The run() function is called periodically every 5 min, when new data becomes available. The first parameter specifies the profile name, the second parameter the new timeslot in the format 'yyyymmddHHMM'.
Profile specific information can be retrieved using the NfSen.pm and NfConf.pm modules.  See the demo plugin 'incmod' included in the NfSen distribution.

Demo plugin demoplugin.pm:
#
# Demo plugin for NfSen
#
# This plugin demonstrates the use of plugins
# It searches for flows > 500000 packets and sends the output
# using the Notification module

package demoplugin;

use strict;
use NfSen;
use NfConf;

#
# The plugin may send any messages to syslog
# Do not initialize syslog, as this is done by
# the main process nfsen-run
use Sys::Syslog;
Sys::Syslog::setlogsock('unix');

# Use the optional Notification module
use Notification;

my ( $nfdump, $BASEDIR );

#
# Define a nice filter:
# We like to see flows containing more than 500000 packets
my $nf_filter = 'packets > 500000';

#
# Periodic function
# input: profilename
# timeslot. Format yyyymmddHHMM e.g. 200503031200
sub run {
my $profile = shift;
my $timeslot = shift;

syslog('debug', "demoplugin run: Profile: $profile, Time: $timeslot");

my %profileinfo = NfSen::ReadProfile($profile);
my $netflow_sources = "$BASEDIR/profiles/$profile/$profileinfo{'sourcelist'}";

#
# process all sources of this profile at once
my @output = `$nfdump -M $netflow_sources -r nfcapd.$timeslot '$nf_filter'`;

#
# Process the output and notify the duty team

# The 2nd but last line contains the stat info of the processed flow:
# e.g. Flows analysed: 61 matched: 61, Bytes read: 3168
if ( $output[-2] =~ /matched:\s+(\d+)/ ) {
my $matched = $1;
syslog('debug', "demoplugin run: $matched flows");
if ( $matched ) {
notify("Packets > 500000: Profile $profile, Timeslot $timeslot", \@output );
}
} else {
syslog('err', "Unparsable output line '$output[-2]'");
}


# .... more to come

}

sub Init {
syslog("info", "demoplugin: Init");

# Init some vars
$nfdump = "$NfConf::PREFIX/nfdump";
$BASEDIR = "$NfConf::BASEDIR";

return 1;
}

sub BEGIN {
syslog("info", "demoplugin BEGIN");
# Standard BEGIN Perl function - See Perl documentation
# not used here
}

sub END {
syslog("info", "demoplugin END");
# Standard END Perl function - See Perl documentation
# not used here
}


1;

Testing Plugins

Before installing your plugin in nfsen-run, test the plugin with the testPlugin  script available in BASEDIR/bin. The test scripts allows you to  test the plugin with any available profile and timeslice:
./testPlugin -p <pluginname> -P <profile> -t <timeslot>

Optional Modules

Optional modules are not required by NfSen. However, they make your life easier when writing plugins.

Notification.pm

Plugins may produce some output to be sent to a duty team for further analysis or for requesting some actions.
This module allows you to send emails with one single line: notify(<Subject>, <BODY>); The configuration parameters, such as From, To and SMTP server are defined in the master nfsen.conf file.

Example:
nfsen.conf:
....
#
# Notification module
# The Notification module is an optional module. If you want your plugins to
# notify any result by email, use this module.
# Make sure you have installed Mail::Internet before using the module
#
# Use this from address
$MAIL_FROM = 'your@from.example.net' ;

# Use these recipients
$RCPT_TO = 'any@example.net', another@example.net" ;

# Use this SMTP server
$SMTP_SERVER = 'localhost' ;


Somewhere in your plugin:

use Notification;
...

@output = some command;
notify("Suspicious Flows found", \@output);