yaf(1) - Linux man page

Name

yaf - Yet Another Flowmeter

Synopsis

yaf     [--in INPUT_SPECIFIER] [--out OUTPUT_SPECIFIER]
        [--live LIVE_TYPE] [--ipfix TRANSPORT_PROTOCOL]
        [--filter BPF_FILTER]
        [--rotate ROTATE_DELAY] [--lock] [--caplist]
        [--group SPREAD_GROUP_NAME(s)]
        [--groupby GROUPBY_TYPE]
        [--stats INTERVAL][--no-stats] [--noerror]
        [--dag-interface]
        [--napatech-interface]
        [--gre-decode] [--no-frag]
        [--max-frags FRAG_TABLE_MAX]
        [--ip4-only] [--ip6-only]
        [--idle-timeout IDLE_TIMEOUT]
        [--active-timeout ACTIVE_TIMEOUT]
        [--udp-temp-timeout TEMPLATE_TIMEOUT]
        [--max-payload PAYLOAD_OCTETS] [--udp-payload]
        [--max-flows FLOW_TABLE_MAX]
        [--export-payload] [--silk] [--udp-uniflow PORT]
        [--uniflow] [--mac] [--force-ip6-export]
        [--observation-domain DOMAIN_ID] [--entropy]
        [--applabel] [--applabel-rules RULES_FILE]
        [--ipfix-port PORT] [--tls] [--tls-ca CA_PEM_FILE]
        [--tls-cert CERT_PEM_FILE] [--tls-key KEY_PEM_FILE]
        [--become-user UNPRIVILEGED_USER]
        [--become-group UNPRIVILEGED_GROUP]
        [--log LOG_SPECIFIER] [--loglevel LOG_LEVEL]
        [--verbose] [--version]
        [--p0fprint] [--fpexport]
        [--plugin-name LIBPLUGIN_NAME[,LIBPLUGIN_NAME...]]
        [--plugin-opts "OPTIONS[,OPTIONS...]"]

Description

"yaf" is Yet Another Flowmeter and "yaf" is a suite of tools to do flow metering. "yaf" is used as a sensor to capture flow information on a network and export that information in IPFIX format. It reads packet data from pcap(3) dumpfiles as generated by tcpdump(1), from live capture from an interface using pcap(3), an Endace DAG capture device, or a Napatech adapter, aggregates these packets into flows, and exports flow records via IPFIX over SCTP , TCP or UDP , Spread, or into serialized IPFIX message streams ( IPFIX files) on the local file system.

Since "yaf" is designed to be deployed on white-box sensors attached to local network segments or span ports at symmetric routing points, it supports bidirectional flow assembly natively. Biflow export is done via the export method specified in RFC 5103 Bidirectional Flow Export using IPFIX . See the OUTPUT section below for information on this format.

"yaf" also supports experimental partial payload capture, specifically for banner-grabbing applications and protocol verification purposes.

The output of "yaf" is designed to be collected and manipulated by flow processing toolchains supporting IPFIX . The yafscii(1) tool, which is installed as part of YAF , can also be used to print "yaf" output in a human-readable format somewhat reminiscient of tcpdump(1). "yaf" output can also be analyzed using the SiLK suite, and the nafalize(1) tool, both available from the CERT NetSA group.

Options

Input Options

These options control where "yaf" will take its input from. "yaf" can read packets from a pcap dumpfile (as generated by "tcpdump -w") or live from an interface via libpcap, libdag, or libpcapexpress (Napatech Adapter library). By default, if no input options are given, "yaf" reads a pcap dumpfile on standard input.
--in INPUT_SPECIFIER: INPUT_SPECIFIER is an input specifier. If --live is given, this is the name of an interface (e.g. "eth0", "en0", "dag0") to capture packets from. Otherwise, it is a filename; the string - may be used to read from standard input (the default).
--caplist: If present, treat the filename in INPUT_SPECIFIER as an ordered newline-delimited list of pathnames to pcap(3) dumpfiles. Blank lines and lines beginning with the character '#' within this are ignored. All pathnames are evaluated with respect to the working directory yaf is run in. These dumpfiles are processed in order using the same flow table, so they must be listed in ascending time order. This option is intended to ease the use of yaf with rotated or otherwise split tcpdump(1) output.
--noerror: Used with the --caplist option. When present, this prevents "yaf" from exiting when processing a list of dumpfiles in the middle due to an error in a file. "yaf" will continue to process all files given in the INPUT_SPECIFIER despite errors within those files.
--live LIVE_TYPE: If present, capture packets from an interface named in the INPUT_SPECIFIER . LIVE_TYPE is one of pcap for packet capture via libpcap, or dag for packet capture via an Endace DAG interface using libdag, or napatech for packet capture via a Napatech Adapter using libpcapexpress. dag is only available if "yaf" was built with Endace DAG support. napatech is only available if "yaf" was built with libpcapexpress, a library maintained by nPulse Technologies for use with a Napatech adapter.
--dag-interface: If present, the interface on which a packet was received will be noted internally within "yaf". When flow records are exported from "yaf", an "ingressinterface" and an "egressinterface" set of fields will be added to the output. The "ingressinterface" field will be the physical interface which captured the packet while the "egressinterface" will be the physical interface | 0x100. This can be used to separate traffic based on DAG physical ports. This option requires building "yaf" with the "--enable-daginterface" switch.
--napatech-interface: Similar to --dag-interface, if present, the interface on which a packet was received will be noted internally within "yaf". When flow records are exported from "yaf", an "ingressinterface" and an "egressinterface" set of fields will be added to the output. The "ingressinterface" field will be the physical interface which captured the packet while the "egressinterface" will be the the physical interface | 0x100. This can be used to separate traffic based on Napatech physical ports. This option requires building "yaf" with the "--enable-napatechinterface" switch.
--filter BPF_FILTER: If present, enable Berkeley Packet Filtering ( BPF ) in "yaf" with FILTER_EXPRESSION as the incoming traffic filter. Syntax of FILTER_EXPRESSION follows the expression format described in the tcpdump(1) man page. This option is not currently supported if --live is set to dag or napatech as BPF filtering is implemented with libpcap. This option is also not supported on a Bivio due to the Bivio supplied version of libpcap that supports the zero-copy driver.

Output Options

These options control where "yaf" will send its output. "yaf" can write flows to an IPFIX file or export flows to an IPFIX collector over SCTP , TCP , UDP , or Spread. By default, if no output options are given, "yaf" writes an IPFIX file to standard output.
--out OUTPUT_SPECIFIER: OUTPUT_SPECIFIER is an output specifier. If --ipfix is present, the OUTPUT_SPECIFIER specifies the hostname or IP address of the collector to which the flows will be exported. Otherwise, if --rotate is present, OUTPUT_SPECIFIER is the prefix name of each output file to write to. If --ipfix is present and set to spread, then OUTPUT_SPECIFIER should be set to the name of the Spread daemon to connect to (See below examples of spread daemon names). Otherwise, OUTPUT_SPECIFIER is a filename in which the flows will be written; the string - may be used to write to standard output (the default).; Output to file; "--out flows.yaf"; Output to collector on port 18000 at IP address 1.2.3.4; "--out 1.2.3.4 --ipfix-port 18000 --ipfix tcp"; Connect to the Spread daemon on the local machine; "--out 4803 or --out 4803@localhost"; Connect to the machine identified by the domain name "host.domain.edu" on port 4803.; "--out 4803@host.domain.edu"; Connect to the machine identified by the IP address "x.y.123.45" on port 4803.; "--out 4803@x.y.123.45"
--ipfix TRANSPORT_PROTOCOL: If present, causes "yaf" to operate as an IPFIX exporter, sending IPFIX Messages via the specified transport protocol to the collector (e.g., SiLK's rwflowpack or flowcap facilities) named in the OUTPUT_SPECIFIER . Valid TRANSPORT_PROTOCOL values are tcp, udp, sctp, and spread; sctp is only available if "yaf" was built with SCTP support; spread is only available if "yaf" was built with Spread support. UDP is not recommended, as it is not a reliable transport protocol, and cannot guarantee delivery of messages. As per the recommendations in RFC 5101, "yaf" will retransmit templates three times within the template timeout period (configurable using --udp-temp-timeout or by default, 10 minutes). Use the --ipfix-port, --tls, --tls-ca, --tls-cert, --tls-key, --tls-pass, and --group options to further configure the connection to the IPFIX collector.
--rotate ROTATE_DELAY: If present, causes "yaf" to write output to multiple files, opening a new output file every ROTATE_DELAY seconds in the input data. Rotated files are named using the prefix given in the OUTPUT_SPECIFIER , followed by a suffix containing a timestamp in "YYYYMMDDhhmmss" format, a decimal serial number, and the file extension .yaf.
--lock: Use lockfiles for concurrent file access protection on output files. This is recommended for interoperating with the Airframe filedaemon facility.
--stats INTERVAL: If present, causes "yaf" to export process statistics every INTERVAL seconds. The default value for INTERVAL is 300 seconds or every 5 minutes. "yaf" uses IPFIX Options Templates and Records to export flow, fragment, and decoding statistics. If INTERVAL is set to zero, stats will not be exported.
--no-stats: If present, "yaf" will not export process statistics. "yaf" uses IPFIX Options Templates and Records to export flow, fragment, and decoding statistics. --no-stats takes precedence over --stats.

Decoder Options

These options are used to modify the "yaf" packet decoder's behavior. None of these options are required; the default behavior for each option when not present is noted.
--no-frag: If present, ignore all fragmented packets. By default, "yaf" will reassemble fragments with a 30 second fragment timeout.
--max-frags FRAG_TABLE_MAX: If present, limit the number of outstanding, not-yet reassembled fragments in the fragment table to FRAG_TABLE_MAX by prematurely expiring fragments from the table. This option is provided to limit "yaf" resource usage when operating on data from very large networks or networks with abnormal fragmentation. The fragment table may exceed this limit slightly due to limits on how often "yaf" prunes the fragment table (every 5 seconds). By default, there is no fragment table limit, and the fragment table can grow to resource exhaustion.
--ip4-only: If present, ignore all IPv6 packets and export IPv4 flows only. The default is to process both IPv4 and IPv6 packets.
--ip6-only: If present, ignore all IPv4 packets and export IPv6 flows only. The default is to process both IPv4 and IPv6 packets.
--gre-decode: If present, attempt to decode GRE version 0 encapsulated packets. Flows will be created from packets within the GRE tunnels. Undecodeable GRE packets will be dropped. Without this option, GRE traffic is exported as IP protocol 47 flows. This option is presently experimental.

Flow Table Options

These options are used to modify the flow table behavior within "yaf". None of these options are required; the default behavior for each option when not present is noted.

--idle-timeout IDLE_TIMEOUT

Set flow idle timeout in seconds. Flows are considered idle and flushed from the flow table if no packets are received for IDLE_TIMEOUT seconds. The default flow idle timeout is 300 seconds (5 minutes).

--active-timeout ACTIVE_TIMEOUT

Set flow active timeout in seconds. Any flow lasting longer than ACTIVE_TIMEOUT seconds will be flushed from the flow table. The default flow active timeout is 1800 seconds (30 minutes).

--udp-temp-timeout TEMPLATE_TIMEOUT

Set UDP template timeout in seconds if --ipfix is set to udp. As per RFC 5101 recommendations, "yaf" will attempt to export templates three times within TEMPLATE_TIMEOUT . The default template timeout period is 600 seconds (10 minutes).

--max-payload PAYLOAD_OCTETS

If present, capture at most PAYLOAD_OCTETS octets from the start of each direction of each flow. Non-TCP flows will only capture payload from the first packet unless --udp-payload is set. If not present, "yaf" will not attempt to capture payload. Payload capture must be enabled for payload export (--export-payload), application labeling (--applabel), and entropy evaluation (--entropy). Note that payload capture is still an experimental feature.

--max-flows FLOW_TABLE_MAX

If present, limit the number of open flows in the flow table to FLOW_TABLE_MAX by prematurely expiring the flows with the least recently received packets; this is analogous to an adaptive idle timeout. This option is provided to limit "yaf" resource usage when operating on data from large networks. By default, there is no flow table limit, and the flow table can grow to resource exhaustion.

--udp-payload

If present, capture at most PAYLOAD_OCTETS octets fom the start of each direction of each UDP flow, where PAYLOAD_OCTETS is set using the --max-payload flag.

--silk

If present, export flows in "SiLK mode". As of YAF 2.0, this will export TCP information (flags, ISN ) in the main flow record instead of within the SubTemplateMultiList. This flag must be used when exporting to SiLK for it to collect TCP flow information. This also introduces the following incompatibilities with standard IPFIX export:

• totalOctetCount and reverseTotalOctetCount are clamped to 32 bits. Any packet that would cause either of these counters to overflow 32 bits will cause the flow to close with flowEndReason 0x02 (active timeout), and will become the first packet of a new flow. This is analogous to forcing an active timeout when the octet counters overflow.

• The high-order bit of the flowEndReason IE is set on any flow created on a counter overflow, as above.

• The high-order bit of the flowEndReason IE is set on any flow created on an active timeout.

Since this changes the semantics of the exported flowEndReason IE , it should only be used when generating flows and exporting to rwflowpack, flowcap, or writing files for processing with rwipfix2silk.

Export Options

These options are used to modify the the data exported by "yaf".
--export-payload: If present, export at most PAYLOAD_OCTETS (the argument to --max-payload) octets from the start of each direction of each flow. Non-TCP flows will only export payload from the first packet. By default, "yaf" will not export flow payload.
--uniflow: If present, export biflows using the Record Adjacency method in section 3 of RFC 5103. This is useful when exporting to IPFIX Collecting Processes that are not biflow-aware.
--mac: If present, export MAC-layer information; presently, exports source and destination MAC addresses.
--force-ip6-export: If present, force IPv4 flows to be exported with IPv6-mapped IPv4 addresses in ::FFFF/96. This will cause all flows to appear to be IPv6 flows.
--observation-domain DOMAIN_ID: Set the observationDomainID on each exported IPFIX message to the given integer value. If not present, the observationDomainId defaults to 1. This value is also used as the exportingProcessId in the "yaf" statistics Option Record as a Scope Field.
--udp-uniflow PORT: If present, export each UDP packet on the given port (or 1 for all ports) as a single flow, with flowEndReason set to YAF_END_UDPFORCE (0x1F). This will not close the flow. The flow will stay open until it closes naturally by the idle and active timeouts. Most useful with --export-payload in order to export every UDP payload on a specific port.

Application Labeler Options

If "yaf" is built with application labeler support enabled (using the "--enable-applabel" option to "./configure" when "yaf" is built), then "yaf" can examine packet payloads and determine the application protocol in use within a flow, and export a 16-bit application label with each flow.
The exported application label uses the common port number for the protocol. For example, HTTP traffic, independent of what port the traffic is detected on, will be labeled with a value of 80, the default HTTP port. Labels and rules are taken from a configuration file read at "yaf" startup time.
Application labeling requires payload capture to be enabled with the --max-payload option. A minimum payload capture length of 384 octets is recommended for best results.
Application labeling is presently experimental. SiLK does support IPFIX import and translation of the application label via "rwflowpack", "flowcap", and "rwipfix2silk".
--applabel: If present, export application label data. Requires --max-payload to enable payload capture.
--applabel-rules RULES_FILE: Read application labeler rules from RULES_FILE . If not present, rules are read by default from /usr/etc/yafApplabelRules.conf.

Entropy Measurement

If "yaf" is built with entropy measurement enabled (using the "--enable-entropy" option to "./configure" when "yaf" is built,) then "yaf" can examine the packet payloads and determine a Shannon Entropy value for the payload. The entropy calculation does not include the network ( IP ) or transport ( UDP/TCP ) headers. The entropy is calculated in terms of bits per byte, (log base 2.) The calculation generates a real number value between 0.0 and 8.0. That number is then converted into an 8-bit integer value between 0 and 255. Roughly, numbers above 230 are generally compressed (or encrypted) and numbers centered around approximately 140 are English text. Lower numbers carry even less information content. Another useful piece of information is that SSL/TLS tends to zero pad its packets, which causes the entropy of those flows to drop quite low.
--entropy: If present, export the entropy values for both the forward and reverse payloads. Requires the --max-payload option to operate.

IPFIX Connection Options

These options are used to configure the connection to an IPFIX collector.
--ipfix-port PORT: If --ipfix is present, export flows to TCP , UDP , or SCTP port PORT . If not present, the default IPFIX port 4739 is used. If --tls is also present, the default secure IPFIX port 4740 is used.
--tls: If --ipfix is present, use TLS to secure the connection to the IPFIX collector. Requires the TRANSPORT_PROTOCOL to be tcp, as DTLS over UDP or SCTP is not yet supported. Requires the --tls-ca, --tls-cert, and --tls-key options to specify the X.509 certificate and TLS key information.
--tls-ca CA_PEM_FILE: Use the Certificate Authority or Authorities in CA_PEM_FILE to verify the remote IPFIX Collecting Process' X.509 certificate. The connection to the Collecting Process will fail if its certificate was not signed by this CA (or by a certificate signed by this CA , recursively); this prevents export to unauthorized Collecting Processes. Required if --tls is present.
--tls-cert CERT_PEM_FILE: Use the X.509 certificate in CERT_PEM_FILE to identify this IPFIX Exporting Process. This certificate should contain the public part of the private key in KEY_PEM_FILE . Required if --tls is present.
--tls-key KEY_PEM_FILE: Use the private key in KEY_PEM_FILE for this IPFIX Exporting Process. This key should contain the private part of the public key in CERT_PEM_FILE . Required if --tls is present. If the key is encrypted, the password must be present in the YAF_TLS_PASS environment variable.
--group SPREAD_GROUP_NAME: If --ipfix is present and set to spread, use --group to specify the spread group name(s) to publish output. It is possible to list more than one group name in a comma-seperated list. To use Spread as a manifold for different types of flows, use the format GROUP , GROUP_NAME:VALUE , GROUP_NAME:VALUE as the argument to --group and use the --groupby switch. This list should be contained in quotes if it contains spaces ("yaf" will ignore spaces in quotes). It is suggested to use one group as the catchall for all flows (no value listed) so flows are not lost. The --groupby switch must be used if --group uses GROUP:VALUE format. See the Spread Documentation, www.spread.org, for more details on Spread.
--groupby GROUPBY_TYPE: If --group is used with group values, use --groupby to specify what type of value should be used. Options are port, vlan, applabel, protocol, version. --groupby accepts only one argument. The port option is destination transport port of the flow. version is the IP version of the flow.

Privilege Options

These options are used to cause "yaf" to drop privileges when running as root for live capture purposes.
--become-user UNPRIVILEGED_USER: After opening the live capture device in --live mode, drop privilege to the named user. Using --become-user requires "yaf" to be run as root or setuid root. This option will cause all files written by "yaf" to be owned by the user UNPRIVILEGED_USER and the user's primary group; use --become-group as well to change the group "yaf" runs as for output purposes.
If running as root for live capture purposes and --become-user is not present, "yaf" will warn that privilege is not being dropped. We highly recommend the use of this option, especially in production environments, for security purposes.
--become-group UNPRIVILEGED_GROUP: --become-group can be used to change the group from the default of the user given in --become-user. This option has no effect if given without the --become-user option as well.

Logging Options

These options are used to specify how log messages are routed. "yaf" can log to standard error, regular files, or the UNIX syslog facility.
--log LOG_SPECIFIER: Specifies destination for log messages. LOG_SPECIFIER can be a syslog(3) facility name, the special value stderr for standard error, or the absolute path to a file for file logging. The default log specifier is stderr if available, user otherwise.
--loglevel LOG_LEVEL: Specify minimum level for logged messages. In increasing levels of verbosity, the supported log levels are quiet, error, critical, warning, message, info, and debug. The default logging level is warning.
--verbose: Equivalent to --loglevel debug.
--version: If present, print version and copyright information to standard error and exit.

Plugin Options

These options are used to load, configure, and run a "yaf" plugin.
--plugin-name LIBPLUGIN_NAME[,LIBPLUGIN_NAME...]: Specify the plugin to load. The loaded plugin must follow the "yaf" plugin framework. LIBPLUGIN_NAME must be the full path to the plugin library name. The only plugin currently supported is the Deep Packet Inspection plugin. This flag will only be recognized if "yaf" is configured with --enable-plugins. There are also configure options to export only DNS Authoritative and NXDomain responses. Read the document on the DPI Plugin for more information.
--plugin-opts "OPTIONS[,OPTIONS...]": Specify the arguments to the plugin given to --plugin-name. This flag will only be recognized if "yaf" is configured with --enable-plugins and --plugin-name is set to a valid plugin. For example, the DPI Plugin takes the well-known port of a protocol(s) to enable DPI (default for DPI is all protocols).

Passive OS Fingerprinting (p0f)

These options are used to enable p0f in "yaf". p0f is presently experimental. There is no support in "yafscii" or SiLK for printing p0f related data. Currently, "yaf" uses the p0f Version 2 SYN fingerprints (see p0f.fp).
--p0fprint: If present, export p0f data. This data consists of three related information elements; osName, osVersion, osFingerPrint. This flag requires "yaf" to be configured with --enable-p0fprinter.
--fpexport: If present, enable export of handshake headers for external OS fingerprinters. The related information elements are firstPacketBanner and secondPacketBanner. This flag requires "yaf" to be configured with --enable-fpexporter.

Output

Basic Flow Record

"yaf"'s output consists of an IPFIX message stream. "yaf" uses a variety of templates for IPFIX data records; the information elements that may appear in these templates are enumerated below. For further information about the IPFIX information model and IPFIX message stream, see RFC 5102, RFC 5101, and RFC 5103. As of "yaf" 2.0, "yaf" nests some templates in an IPFIX subTemplateMultiList. In order to retain compatibility with the SiLK Tools, use --silk to prevent "yaf" from nesting TCP Information Elements. Below are descriptions of each of the templates YAF will export. See the Internet-Draft Export of Structured Data in IPFIX for more information on IPFIX lists.
"yaf" assigns information element numbers to reverse flow elements in biflow capture based on the standard IPFIX PEN 29305. This applies only for information elements defined in the standard IPFIX Information Model ( RFC 5102) that do not have a reverse information element already defined. For information elements defined under the CERT PEN , a standard method is used to calculate their reverse element identifier. The method is that bit fourteen is set to one in the IE field, (e.g. 16384 + the forward IE number.)
flowStartMilliseconds IE 152, 8 octets: Flow start time in milliseconds since 1970-01-01 00:00:00 UTC . Always present.
flowEndMilliseconds IE 153, 8 octets: Flow end time in milliseconds since 1970-01-01 00:00:00 UTC . Always present.
octetTotalCount IE 85, 8 octets: Number of octets in packets in forward direction of flow. Always present. May be encoded in 4 octets using IPFIX reduced-length encoding.
reverseOctetTotalCount Reverse ( PEN 29305) IE 85, 8 octets: Number of octets in packets in reverse direction of flow. Present if flow has a reverse direction. May be encoded in 4 octets using IPFIX reduced-length encoding.
packetTotalCount IE 86, 8 octets: Number of packets in forward direction of flow. Always present. May be encoded in 4 octets using IPFIX reduced-length encoding.
reversePacketTotalCount Reverse ( PEN 29305) IE 86, 8 octets: Number of packets in reverse direction of flow. Present if flow has a reverse direction. May be encoded in 4 octets using IPFIX reduced-length encoding.
reverseFlowDeltaMilliseconds CERT ( PEN 6871) IE 21, 4 octets: Difference in time in milliseconds between first packet in forward direction and first packet in reverse direction. Correlates with (but does not necessarily represent) round-trip time. Present if flow has a reverse direction.
sourceIPv4Address IE 8, 4 octets: IPv4 address of flow source or biflow initiator. Present for IPv4 flows without IPv6-mapped addresses only.
destinationIPv4Address IE 12, 4 octets: IPv4 address of flow source or biflow responder. Present for IPv4 flows without IPv6-mapped addresses only.
sourceIPv6Address IE 27, 16 octets: IPv6 address of flow source or biflow initiator. Present for IPv6 flows or IPv6-mapped IPv4 flows only.
destinationIPv6Address IE 28, 16 octets: IPv6 address of flow source or biflow responder. Present for IPv6 flows or IPv6-mapped IPv4 flows only.
sourceTransportPort IE 7, 2 octets: TCP or UDP port on the flow source or biflow initiator endpoint. Always present.
destinationTransportPort IE 11, 2 octets: TCP or UDP port on the flow destination or biflow responder endpoint. Always present. For ICMP flows, contains ICMP type * 256 + ICMP code. This is non-standard, and an open issue in YAF .
flowAttributes CERT ( PEN 6871) IE 40, 2 octets: Miscellaneous flow attributes for the forward direction of the flow. Always present ( YAF 2.1 or later). If all packets in the forward direction have fixed size, flowAttributes will be set to 1.
reverseFlowAttributes CERT ( PEN 6871) IE 16424, 2 octets: Miscellaneous flow attributes for the reverse direction of the flow. Always present ( YAF 2.1 or later). If all packets in the reverse direction have fixed size, reverseFlowAttributes will be set to 1.
protocolIdentifier IE 4, 1 octet: IP protocol of the flow. Always present.
flowEndReason IE 136, 1 octet: Flow end reason code, as defined by the IPFIX Information Model. Always present. In --silk mode, the high-order bit is set if the flow was created by continuation.
silkAppLabel CERT ( PEN 6871) IE 33, 2 octets: Application label, defined as the primary well-known port associated with a given application. Present if the application labeler is enabled, and was able to determine the application protocol used within the flow.
vlanId IE 58, 2 octets: 802.1q VLAN tag of the first packet in the forward direction of the flow.
reverseVlanId Reverse ( PEN 29305) IE 58, 2 octets: 802.1q VLAN tag of the first packet in the reverse direction of the flow. Present if the flow has a reverse direction.
subTemplateMultiList IE 32767, variable length: Represents a list of zero or more instances of a structured data type, where the data type of each list element can be different and corresponds with different template definitions. The Information Element Number will change upon updates to the IPFIX lists specification and libfixbuf releases.

TCP Flow Template

The following six Information Elements will be exported as a template within the subTemplateMultiList unless --silk is used.
tcpSequenceNumber IE 184, 4 octets: Initial sequence number of the forward direction of the flow. Present if the flow's protocolIdentifier is 6 ( TCP ). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.
reverseTcpSequenceNumber Reverse ( PEN 29305) IE 184, 4 octets: Initial sequence number of the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 ( TCP ) and the flow has a reverse direction. This element is contained in the "yaf" TCP template within the subTemplateMultiList unless --silk is used.
initialTCPFlags CERT ( PEN 6871) IE 14, 1 octet: TCP flags of initial packet in the forward direction of the flow. Present if the flow's protocolIdentifier is 6 ( TCP ). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.
unionTCPFlags CERT ( PEN 6871) IE 15, 1 octet: Union of TCP flags of all packets other than the initial packet in the forward direction of the flow. Present if the flow's protocolIdentifier is 6 ( TCP ). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.
reverseInitialTCPFlags CERT ( PEN 6871) IE 16398, 1 octet: TCP flags of initial packet in the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 ( TCP ) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.
reverseUnionTCPFlags CERT ( PEN 6871) IE 16399, 1 octet: Union of TCP flags of all packets other than the initial packet in the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 ( TCP ) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

MAC Flow Template

The following two Information Elements will be exported as a template within the subTemplateMultiList.
sourceMacAddress, IE 56, 6 octets: Source MAC Address of the first packet in the forward direction of the flow. This element is contained in the yaf MAC template within the subTemplateMultiList.
destinationMacAddress, IE 80, 6 octets: Destination MAC Address of the first packet in the reverse direction of the flow. This element is contained in the yaf MAC template within the subTemplateMultiList.

Payload Flow Template

The following two Information Elements will be exported as a template within the subTemplateMultiList.
payload CERT ( PEN 6871) IE 18, variable-length: Initial n bytes of forward direction of flow payload. Present if payload collection is enabled and payload is present in the forward direction of the flow. This element is contained in the "yaf" Payload template within the subTemplateMultiList.
reversePayload CERT ( PEN 6871) IE 16402, variable-length: Initial n bytes of reverse direction of flow payload. Present if payload collection is enabled and payload is present in the reverse direction of the flow. This element is contained in the "yaf" Payload template within the subTemplateMultiList.

Entropy Flow Template

The following two Information Elements will be exported as a template within the subTemplateMultiList.
payloadEntropy CERT ( PEN 6871) IE 35, 1 octet: Shannon Entropy calculation of the forward payload data. This element is contained in the "yaf" Entropy template within the subTemplateMultiList.
reversePayloadEntropy CERT ( PEN 6871) IE 16419, 1 octet: Shannon Entropy calculation of the reverse payload data. This element is contained in the "yaf" Entropy template within the subTemplateMultiList.

p0f Flow Template

The following six Information Elements will be exported as a template within the subTemplateMultiList if present and only if p0f is enabled.
osName CERT ( PEN 6871) IE 36, variable-length: p0f OS Name for the forward flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the "yaf" p0f template within the subTemplateMultiList.
reverseOsName CERT ( PEN 6871) IE 16420, variable-length: p0f OS Name for the reverse flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the "yaf" p0f template within the subTemplateMultiList.
osVersion CERT ( PEN 6871) IE 37, variable-length: p0f OS Version for the forward flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the "yaf" p0f template within the subTemplateMultiList.
reverseOsVersion CERT ( PEN 6871) IE 16421, variable-length: p0f OS Version for the reverse flow based on the SYN packet and p0f SYN fingerprints. Present only if p0f is enabled. This element is contained in the "yaf" p0f template within the subTemplateMultiList.
osFingerPrint CERT ( PEN 6871) IE 107, variable-length: p0f OS Fingerprint for the forward flow based on the SYN packet and p0f SYN fingerprints. Present only if p0f is enabled. This element is contained in the "yaf" p0f template within the subTemplateMultiList.
reverseOsFingerPrint CERT ( PEN 6871) IE 16491, variable-length: p0f OS Fingerprint for the reverse flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the "yaf" p0f template within the subTemplateMultiList.

Fingerprint Exporting Template

The following four Information Elements will be exported as a template within the subTemplateMultiList if present and only if fpexport is enabled.
firstPacketBanner CERT ( PEN 6871) IE 38, variable-length: IP and transport headers for first packet in forward direction to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the "yaf" FPExport template within the subTemplateMultiList.
reverseFirstPacketBanner CERT ( PEN 6871) IE 16422, variable-length: IP and transport headers for first packet in reverse direction to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the "yaf" FPExport template within the subTemplateMultiList.
secondPacketBanner CERT ( PEN 6871) IE 39, variable-length: IP and transport headers for second packet in forward direction (third packet in sequence) to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the "yaf" FPExport template within the subTemplateMultiList.
reverseSecondPacketBanner CERT ( PEN 6871) IE 16423, variable-length: IP and transport headers for second packet in reverse direction (currently not used). Present only if fpexport is enabled. This element is contained in the "yaf" FPExport template within the subTemplateMultiList.

Hooks Templates

"yaf" can export other templates within the subTemplateMultiList if plugins are enabled in "yaf". See <a href=../yafdpi.html>"yafdpi"</a> for descriptions of the "yaf" Deep Packet Inspection Information Elements.

Statistics Option Template

"yaf" will export information about its process periodically using IPFIX Options Template Record. This record gives information about the status of the flow and fragment table, as well as decoding information. This can be turned off using the --no-stats option. The following Information Elements will be exported:
exportedFlowRecordTotalCount IE 42, 8 octets: Total amount of exported flows from "yaf" start time.
packetTotalCount IE 86, 8 octets: Total amount of packets processed by "yaf" from "yaf" start time.
droppedPacketTotalCount IE 135, 8 octets: Total amount of dropped packets according to statistics given by libpcap, libdag, or libpcapexpress.
ignoredPacketTotalCount IE 164, 8 octets: Total amount of packets ignored by the YAF packet decoder, such as unsupported packet types and incomplete headers, from "yaf" start time.
expiredFragmentCount CERT ( PEN 6871) IE 100, 4 octets: Total amount of fragments that have been expired since "yaf" start time.
assembledFragmentCount CERT ( PEN 6871) IE 101, 4 octets: Total number of packets that been assembled from a series of fragments since "yaf" start time.
flowTableFlushEventCount CERT ( PEN 6871) IE 104, 4 octets: Total number of times the "yaf" flow table has been flushed since "yaf" start time.
flowTablePeakCount CERT ( PEN 6871) IE 105, 4 octets: The maximum number of flows in the "yaf" flow table at any one time since "yaf" start time.
exporterIPv4Address IE 130, 4 octets: The IPv4 Address of the "yaf" flow sensor.
exportingProcessId IE 144, 4 octets: Set the ID of the "yaf" flow sensor by giving a value to --observation-domain. The default is 1.
meanFlowRate CERT ( PEN 6871) IE 102, 4 octets: The mean flow rate of the "yaf" flow sensor since "yaf" start time, rounded to the nearest integer.
meanPacketRate CERT ( PEN 6871) IE 103, 4 octets: The mean packet rate of the "yaf" flow sensor since "yaf" start time, rounded to the nearest integer.

Signals

"yaf" responds to SIGINT or SIGTERM by terminating input processing, flushing any pending flows to the current output, and exiting. If --verbose is given, "yaf" responds to SIGUSR1 by printing present flow and fragment table statistics to its log. All other signals are handled by the C runtimes in the default manner on the platform on which "yaf" is currently operating.

Examples

To generate flows from an pcap file into an IPFIX file:

"yaf --in packets.pcap --out flows.yaf"

To capture flows from a pcap interface and export them to files in the current directory rotated hourly:

"yaf --live pcap --in en1 --out en1_capture --rotate 3600"

To capture flows from an Endace DAG card and export them via IPFIX over TCP:

"yaf --live dag --in dag0 --ipfix tcp --out my-collector.example.com"

To convert a pcap formatted packet capture and convert that into IPFIX:

"yaf <packets.pcap >flows.yaf"

To publish to spread group TST_SPRD for a spread daemon running locally on port 4803:

"yaf --live pcap --in eth1 --out 4803@localhost --ipfix spread --group TST_SPRD"

To publish to spread groups based on application label for spread daemon running locally on port 4803:

"yaf --live pcap --in eth1 --out 4803@localhost --ipfix spread --group "SPRD_CATCHALL, SPRD_DNS:53, SPRD_HTTP:80, SPRD_SMTP:25" --groupby applabel --applabel --max-payload=400"

To run "yaf" with application labeling enabled and export via IPFIX over TCP:

"yaf --live pcap --in eth1 --out 127.0.0.1 --ipfix tcp --ipfix-port=18001 --applabel --applabel-rules=/usr/local/etc/yafApplabelRules.conf --max-payload=300"

To run "yaf" with BPF on UDP Port 53

"yaf --live pcap --in en1 --out /path/to/dst/ --rotate 120 --filter="udp port 53""

To run "yaf" with Deep Packet Inspection enabled for HTTP , IMAP , and DNS:

"yaf --in packets.pcap --out flows.yaf --applabel --max-payload=400 --plugin-name=/usr/lib/dpacketplugin.la --plugin-opts="80 143 53""

Known Issues

YAF BPF Filtering does not operate with the Bivio Zero-Copy Library.

YAF BPF Filtering is ignored when using --live dag or napatech because libpcap is not used.

YAF 2.x requires libfixbuf 1.0.0 or later.

YAF 2.0 will not interoperate with the SiLK tools unless --silk is used, due to the TCP Information Elements being nested in the subTemplateMultiList. YAF 2.0 must be used with an IPFIX Collecting Process that can handle IPFIX lists elements, especially the subTemplateMultiList Information Element in order to retrieve certain flow information. Older versions of YAF can read YAF 2.0 flow files, but will ignore anything contained in the subTemplateMultiList.

The plugin infrastructure has been modified in YAF 2.0 to export templates in YAF 's subTemplateMultiList element.

YAF 2.0 will export statistics in an Options Template and Options Data Records unless the --no-stats switch is given. The IPFIX Collecting Process should be able to differentiate between options records and flow records in order to prevent incorrect transcoding of statistics records into flow records.

YAF 0.7 does not interoperate with previous versions, because it no longer uses provisional information elements for the reverse direction of a biflow. YAF 0.7 must be used with an IPFIX Collecting Process that uses PEN 29305 for reverse information elements. For export to SiLK, this implies that the SiLK packer or rwipfix2silk utility must be built against libfixbuf 0.7.0 or later.

Presently, the destinationTransportPort information element contains ICMP type and code information for ICMP or ICMP6 flows; this is nonstandard and may not be interoperable with other IPFIX implementations.

Bug reports and feature requests may be sent via email to <netsa-help@cert.org>.

Authors

Brian Trammell, Chris Inacio <inacio@cert.org>, Michael Duggan <mwd@cert.org>, Emily Sarneso <ecoff@cert.org>, Dan Ruef <druef@cert.org>, and the CERT Network Situational Awareness Group Engineering Team, <http://www.cert.org/netsa>.

Referenced By

applabel(1), yafdpi(1)