CYBR 5940 - Threat

Intelligence and Incident

Response
NetFlow analysis
Drawbacks of full pcap capture
• Full pcap capture is not an option in many cases
• The amount of traffic can increase exponentially
• It could become very expensive to store full pcap
captures
• More expensive hardware may need to be
acquired
• Analyzing large amount of pcap files can also
become very cumbersome
What is NetFlow?
• Introduced by Cisco in 1996
• Helps collect statistical information of network
traffic
• Data generated by NetFlow helps analyze:
• Source and destination of traffic
• Class of service
• The causes of congestion

https://en.wikipedia.org/wiki/NetFlow
What is NetFlow? (cont’d.)
• NetFlow record defines a summary of a flow
• This flow is a unidirectional sequence of packets
• These packets all share seven values
• These fields define a unique key for the flow
• NetFlow does not store the content!
• Only the metadata of a traffic
• This helps reduce the size of NetFlow records
immensely!

https://en.wikipedia.org/wiki/NetFlow
What is NetFlow? (cont’d.)
• Fields in NetFlow version 5:
• Ingress interface
• Source IP address
• Destination IP address
• Protocol
• Source port for UDP/TCP
• 0 for other protocols
• Destination port for UDP/TCP
• Type and code for ICMP
• 0 for other protocols
• IP Type of Service

https://en.wikipedia.org/wiki/NetFlow
Forensics relevance of NetFlow
• Helps identify:
• IP addresses with which a host communicates the most
• Whether a host contacts malicious servers (C2C)
• Amount of data exchanged between hosts and servers
Advantages of NetFlow
• One of the challenges in Network Forensics is
encrypted traffic
• However, fields that NetFlow extracts are always
unencrypted
• This means, NetFlow gives summary of encrypted
traffic as well
• Also, since the size of NetFlow records is small,
• Data collection for long periods of time is possible
NetFlow versions
• NetFlow version 5
• Popular NetFlow version
• Limited to IPv4 flows
• Each packet can hold up to 30 flow records
• NetFlow version 9
• Template-based
• More than 79 data fields available
• Supports IPv6
• Each packet can hold up to 30 flow records
• Others
• Zeek’s conn.log file
• SOF-ELK
• etc.
NetFlow Architecture
• NetFlow Exporter
• NetFlow Collector
• Flow Storage
• Analysis Console

https://en.wikipedia.org/wiki/NetFlow
NetFlow Architecture
• NetFlow Exporter
• Aggregates packets into flows
• Exports flow records towards one or more flow collectors
• NetFlow Collector
• Receives flows from an exporter
• Pre-processes the flow data
• Stores flows in Flow Storage
• Analysis Console
• Analyzes received flow data for
• Intrusion detection
• Traffic profiling
• etc.

https://en.wikipedia.org/wiki/NetFlow
NetFlow version 5 Header Format
Bytes Contents Description
0-1 version NetFlow export format version number
2-3 count Number of flows exported in this packet (1-30)
4-7 sys_uptime Current time in milliseconds since the export device booted
8-11 unix_secs Current count of seconds EPOCH (exporter)
12-15 unix_nsecs Residual nanoseconds EPOCH (exporter)
16-19 flow_sequence Sequence counter of total flows seen (helps detect packet
loss)
20 engine_type Type of flow-switching engine (not always forensics relevant)
21 engine_id Slot number of the flow-switching engine (not always
forensics relevant)
22-23 sampling_interval First two bits hold the sampling mode; remaining 14 bits
hold value of sampling interval (interval in which records
collected)

https://netflow.caligare.com/netflow_v5.htm
NetFlow version 5 Record Format
Bytes Contents Description
0-3 srcaddr Source IP address
4-7 dstaddr Destination IP address
8-11 nexthop IP address of next hop router
12-13 input SNMP index of input interface
14-15 output SNMP index of output interface
16-19 dPkts Packets in the flow (packet count)
20-23 dOctets Total number of Layer 3 bytes in the packets of the flow
(byte count)
24-27 first SysUptime at start of flow
28-31 last SysUptime at the time the last packet of the flow was
received
32-33 srcport TCP/UDP source port number or equivalent

https://netflow.caligare.com/netflow_v5.htm
NetFlow version 5 Record Format
Bytes Contents Description
34-35 dstport TCP/UDP destination port number or equivalent
36 pad1 Unused (zero) bytes
37 tcp_flags Cumulative OR of TCP flags (not packet by packet, all flags
seen)
38 prot IP protocol type (for example, TCP = 6; UDP = 17)
39 tos IP type of service (ToS)
40-41 src_as Autonomous system number of the source, either origin or
peer
42-43 dst_as Autonomous system number of the destination, either origin
or peer
44 src_mask Source address prefix mask bits
45 dst_mask Destination address prefix mask bits
46-47 pad2 Unused (zero) bytes
https://netflow.caligare.com/netflow_v5.htm
NetFlow version 9 Header Format
Bytes Contents Description
0-1 version NetFlow export format version number
2-3 count Number of flow sets exported in this packet, both template
and data (1-30)
4-7 sys_uptime Current time in milliseconds since the export device booted
8-11 unix_secs Current count of seconds since 0000 UTC 1970
12-15 package_sequence Sequence counter of all export packets sent by the export
device. Helps detect if packets are lost
16-19 source_id A 32-bit value that is used to guarantee uniqueness for all
flows exported from a particular device

https://netflow.caligare.com/netflow_v9.htm
NetFlow version 9 Record Format
Field Description
flowset_id Template ID used for subsequent record
length The total length of this FlowSet
Record 1 Field 1 The values defined in the corresponding NetFlow template
… The values defined in the corresponding NetFlow template
Record N Field N The values defined in the corresponding NetFlow template

https://netflow.caligare.com/netflow_v5.htm
 CYBR 5940 - Threat
Intelligence and Incident
Response
Open-source NetFlow
analysis tools
softflowd
• Software implementation of the NetFlow Exporter
• Captures traffic and transfers it to the NetFlow
Collector

https ://ma ttjhay es. com/2018/08/19/c olle cting-ne tflow-with-nfcapd-a nd -nfdum p/

nfcapd
• Basically, the NetFlow Collector
• Receives records from NetFlow Exporter
(softflowd)
• Saves the data to regular or binary files

https ://ma ttjhay es. com/2018/08/19/c olle cting-ne tflow-with-nfcapd-a nd -nfdum p/

nfcapd (cont’d.)
• The filename format of the records generated by
nfcapd is:
• nfcapd.YYYYMMDDHHMM

https ://ma ttjhay es. com/2018/08/19/c olle cting-ne tflow-with-nfcapd-a nd -nfdum p/

nfdump
• Like tcpdump
• nfdump reads binary input from nfcapd
• Outputs the results in ASCII or binary format
• nfdump allows filtering NetFlow records
• Supports BPF like tcpdump
• Also supports some additional syntax
nfdump Syntax
• Use of filters that are the same as BPF
• host, net, port
• Directionalities are also the same as BPF
• src port, dst port, src host, etc.
• Standard logic are also the same as BPF
• and, or, not, ()
• Unlike BPF, to specify a protocol,
• proto tcp, proto udp, etc.
• Protocols supported: tcp, udp, icmp, gre, ah, 132
nfdump Options
• -r Read NetFlow record
• -R Read NetFlow records recursively
• -c Number of records to match
• -t To specify time-range
• -o Output file format
raw, line, long, extended, custom
• -q Do not print the header
• -O To order records (Use tstart to order by time)
• -a To aggregate records based on the 5 flow keys
• -A For custom aggregations
nfdump Output file format
• nfdump has four fixed output formats:
• raw
• line
• long
• extended
• The default format is line, unless otherwise
specified
• The user may specify any desired output format
using the custom output format fmt:..

http://nfdump.sourceforge.net
nfdump Output file format
• raw format
• Displays each record in
multiple lines
• Prints any available
information in the record
• This format is rarely used
• But contains any
information available for a
given record

http://nfdump.sourceforge.net
nfdump Output file format
• line format
• This is the default format
• Shows one NetFlow record per line
• The date and duration of the flow are given in
millisecond resolution
• The number of flows is always 1 unless flows are
aggregated

http://nfdump.sourceforge.net
nfdump Output file format
• long format
• This format contains additional information to line
format such as:
• TCP flags
• Type of Service (ToS)
• etc.

http://nfdump.sourceforge.net
nfdump Output file format
• extended format
• This format contains additional information to long
format
• The following are calculated and displayed for each
record
• pps (packets per second)
• bps (bits per second)
• bps (bytes per packet)
• This format gives most information in one line

http://nfdump.sourceforge.net
nfdump Output file format
• custom format
• This is the most flexible format
• You can specify yourself how the output looks like
• The output format is defined using element tags as well
as plain ASCII text

http://nfdump.sourceforge.net
nfdump Output file format
• custom format (cont’d.)
Tag Description Tag Description
%ts Start Time - first seen %das Destination AS
%te End Time - last seen %in Input Interface num
%td Duration %out Output Interface num
%pr Protocol %pkt Packets
%sa Source Address %byt Bytes
%da Destination Address %fl Flows
%sap Source Address:Port %flg TCP Flags
%dap Destination Address:Port %tos ToS
%sp Source Port %bps Bits per second
%dp Destination Port %pps Packets per second
%sas Source AS %bpp Bytes per package

http://nfdump.sourceforge.net
nfdump Output file format
• custom format (cont’d.)
• Example: The “long” format can be described as follows:
-o “fmt:%ts %td %pr %sap -> %dap %pkt %byt %fl”

http://nfdump.sourceforge.net
Lab 4
• Exercise the commands provided in the pdf file on
Blackboard
nfdump Statistics
• Determining who uses the most bandwidth can be
a very beneficial piece of information
• You may also want to determine IPs that generated
the most traffic, connected to the most hosts
• You do not necessarily need to use command line
tools to obtain statistical information
• nfdump provides the ability to generate statistical
values internally
• This helps increase efficiency when dealing with large
data
nfdump Statistics
• Basic syntax:
• nfdump -s stat/order
• e.g.,
• nfdump -s ip/bytes
• This command will sort each IP based on the number of bytes
transferred
• By default, 10 entries are shown
• -n parameter can be used to specify the number of lines
to be displayed
nfdump Statistics
• Available stat values:
• proto Protocol numbers
• record Aggregated NetFlow record count
• ip Any IP address (src or dst)
• srcip Source IP address
• dstip Destination IP address
• port Any port (src or dst)
• srcport Source port
• dstport Destination port
• as Autonomous system (AS) (src or dst)
nfdump Statistics
• Available stat values: (cont’d.)
• srcas Source AS numbers
• dstas Destination AS numbers
• if Any interface number (input or output)
• inif Input interface numbers
• outif Output interface numbers
nfdump Statistics
• Available order values:
• flows Number of flow records
• packets Number of packets
• bytes Total bytes
• pps Packets per second
• bps Bytes per second
• bpp Bytes per packet
nfdump Statistics
• “:p” is an optional parameter that can be used as
such:
• nfdump -s stat:p/order
• This parameter splits results by protocol
• Instead of reporting all packets with port number 53,
this option will make sure to print TCP and UDP packets
with port number 53 separately