xref: /dports/security/silktools/silk-3.19.1/src/rwstats/rwuniq.pod

=pod

=head1 NAME

B<rwuniq> - Bin SiLK Flow records by a key and print each bin's volume

=head1 SYNOPSIS

  rwuniq --fields=KEY [--values=VALUES]
        [{--threshold=MIN-MAX | --threshold=MIN}]
        [--presorted-input] [--sort-output]
        [{--bin-time=SECONDS | --bin-time}]
        [--timestamp-format=FORMAT] [--epoch-time]
        [--ip-format=FORMAT] [--integer-ips] [--zero-pad-ips]
        [--integer-sensors] [--integer-tcp-flags]
        [--no-titles] [--no-columns] [--column-separator=CHAR]
        [--no-final-delimiter] [{--delimited | --delimited=CHAR}]
        [--print-filenames] [--copy-input=PATH] [--output-path=PATH]
        [--pager=PAGER_PROG] [--temp-directory=DIR_PATH]
        [{--legacy-timestamps | --legacy-timestamps={1,0}}]
        [--all-counts] [{--bytes | --bytes=MIN | --bytes=MIN-MAX}]
        [{--packets | --packets=MIN | --packets=MIN-MAX}]
        [{--flows | --flows=MIN | --flows=MIN-MAX}]
        [--stime] [--etime]
        [{--sip-distinct | --sip-distinct=MIN | --sip-distinct=MIN-MAX}]
        [{--dip-distinct | --dip-distinct=MIN | --dip-distinct=MIN-MAX}]
        [--ipv6-policy={ignore,asv4,mix,force,only}]
        [--site-config-file=FILENAME]
        [--plugin=PLUGIN [--plugin=PLUGIN ...]]
        [--python-file=PATH [--python-file=PATH ...]]
        [--pmap-file=MAPNAME:PATH [--pmap-file=MAPNAME:PATH ...]]
        [--pmap-column-width=NUM]
        {[--xargs] | [--xargs=FILENAME] | [FILE [FILE ...]]}

  rwuniq [--pmap-file=MAPNAME:PATH [--pmap-file=MAPNAME:PATH ...]]
        [--plugin=PLUGIN ...] [--python-file=PATH ...] --help

  rwuniq [--pmap-file=MAPNAME:PATH [--pmap-file=MAPNAME:PATH ...]]
        [--plugin=PLUGIN ...] [--python-file=PATH ...] --help-fields

  rwuniq --version

=head1 DESCRIPTION

B<rwuniq> reads SiLK Flow records and groups them by a key composed of
user-specified attributes of the flows.  For each group (or bin), a
collection of user-specified I<aggregate values> is computed; these
values are typically related to the volume of the bin, such as the sum
of the bytes fields for all records that match the key.  Once all the
SiLK Flow records are read, the key fields and the aggregate values
are printed.  For some of the built-in aggregate values, it is
possible to limit the output to the bins where the aggregate value
meets a user-specified minimum and/or maximum.

There is no need to sort the input to B<rwuniq> since B<rwuniq>
normally rearranges the records as they are read.  To have B<rwuniq>
sort its output, use the B<--sort-output> switch.

B<rwuniq> reads SiLK Flow records from the files named on the command
line or from the standard input when no file names are specified and
B<--xargs> is not present.  To read the standard input in addition to
the named files, use C<-> or C<stdin> as a file name.  If an input
file name ends in C<.gz>, the file is uncompressed as it is read.
When the B<--xargs> switch is provided, B<rwuniq> reads the names of
the files to process from the named text file or from the standard
input if no file name argument is provided to the switch.  The input
to B<--xargs> must contain one file name per line.

The user must provide the B<--fields> switch to select the flow
attribute(s) (or field(s)) that comprise the key for each bin.  The
available fields are similar to those supported by B<rwcut(1)>; see
the description of the B<--fields> switch in the L</OPTIONS> section
below for the details.  The list of fields can be extended by loading
PySiLK files (see B<silkpython(3)>) or plug-ins (B<silk-plugin(3)>).
The fields are printed in the order in which they occur in the
B<--fields> switch.  The size of the key is limited to 256 octets.  A
larger key more quickly uses the available the memory leading to
slower performance.

The aggregate value(s) to compute for each bin are also chosen by the
user.  As with the key fields, the user can extend the list of
aggregate fields by using PySiLK or plug-ins.  Specify the aggregate
fields with the B<--values> switch; the aggregate fields are printed
in the order they occur in the B<--values> switch.  If the user does
not provide B<--values> or a B<--threshold> switch (described next),
B<rwuniq> defaults to computing the number of flow records for each
bin.  As with the key fields, requesting more aggregate values slows
performance.

The B<--threshold> switch (added in SiLK 3.17.0) allows the user to
print only bins where a value field is within a certain range.  The
switch's argument contains the name of the value field, an equals
sign, the minimum value (start of the range), and optionally a hyphen
and the maximum value (end of the range); e.g.,
C<--threshold=bytes=1000-2000>.  The upper bound is unlimited when no
maximum is specified.  The B<--threshold> switch may be repeated to
set multiple thresholds, and only those bins that meet all thresholds
are printed.  Each field named by B<--threshold> is appended to the
set of aggregate value fields unless that field was named in the
B<--values> switch.

The B<--presorted-input> switch may allow B<rwuniq> to process data
more efficiently by causing B<rwuniq> to assume the input has been
previously sorted with the B<rwsort(1)> command.  With this switch,
B<rwuniq> typically does not need large amounts of memory because it does not
bin each flow; instead, it keeps a running summation and outputs the
bin whenever the key changes.  For the output to be meaningful,
B<rwsort> and B<rwuniq> I<must> be invoked with the same B<--fields>
value.  When multiple input files are specified and
B<--presorted-input> is given, B<rwuniq> merge-sorts the flow
records from the input files.  B<rwuniq> typically runs faster if
you do I<not> include the B<--presorted-input> switch when counting
distinct values, even when reading sorted input.  Finally, you
may get unusual results with B<--presorted-input> when the B<--fields>
switch contains multiple time-related key fields (C<sTime>,
C<duration>, C<eTime>), or when the time-related key is not the final
key listed in B<--fields>; see the L</NOTES> section for details.

B<rwuniq> attempts to keep all key and aggregate value data in the
computer's memory.  If B<rwuniq> runs out of memory, the current key
and aggregate value data is written to a temporary file.  Once all
input has been processed, the data from the temporary files is merged
to produce the final output.  By default, these temporary files are
stored in the F</tmp> directory.  Because these files can be large, it
is strongly recommended that F</tmp> I<not> be used as the temporary
directory.  To modify the temporary directory used by B<rwuniq>,
provide the B<--temp-directory> switch, set the SILK_TMPDIR
environment variable, or set the TMPDIR environment variable.

=head1 OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an
exact match for an option.  A parameter to an option may be specified
as B<--arg>=I<param> or B<--arg> I<param>, though the first form is
required for options that take optional parameters.

The B<--fields> switch is required.  B<rwuniq> fails when it is
not provided.

=over 4

=item B<--fields>=I<KEY>

I<KEY> contains the list of flow attributes (a.k.a. fields or columns)
that make up the key into which flows are binned.  The columns are
displayed in the order the fields are specified.  Each field may be
specified once only.  I<KEY> is a comma separated list of field-names,
field-integers, and ranges of field-integers; a range is specified by
separating the start and end of the range with a hyphen (B<->).
Field-names are case insensitive.  Example:

 --fields=stime,10,1-5

There is no default value for the B<--fields> switch; the switch must
be specified.

The complete list of built-in fields that the SiLK tool suite supports
follows, though note that not all fields are present in all SiLK file
formats; when a field is not present, its value is 0.

=over 4

=item sIP,1

source IP address

=item dIP,2

destination IP address

=item sPort,3

source port for TCP and UDP, or equivalent

=item dPort,4

destination port for TCP and UDP, or equivalent.  See note at C<iType>.

=item protocol,5

IP protocol

=item packets,pkts,6

packet count

=item bytes,7

byte count

=item flags,8

bit-wise OR of TCP flags over all packets

=item sTime,9

starting time of flow (seconds resolution unless B<--bin-time>
includes fractional seconds). When the time-related fields
C<sTime>,C<duration>,C<eTime> are all in use, B<rwuniq> ignores the
final time field when binning the records.

=item duration,10

duration of flow (seconds resolution unless B<--bin-time> includes
fractional seconds).  This field is not adjusted by B<--bin-time>
unless B<--fields> includes both C<sTime> and C<eTime>.  See note at
C<sTime,9>.

=item eTime,11

end time of flow (seconds resolution unless B<--bin-time> includes
fractional seconds).  See note at C<sTime,9>.

=item sensor,12

name or ID of the sensor where the flow was collected

=item class,20

class assigned to the flow by B<rwflowpack(8)>.  Binning by C<class>
and/or C<type> equates to binning by the integer value used internally
to represent the class/type pair.  When B<--fields> contains C<class>
but not C<type>, B<rwuniq>'s output contains multiple rows with the
same value(s) for the key field(s).

=item type,21

type assigned to the flow by B<rwflowpack(8)>.  See note on previous
entry.

=item iType

the ICMP type value for ICMP or ICMPv6 flows and empty (numerically
zero) for non-ICMP flows.  Internally, SiLK stores the ICMP type and
code in the C<dPort> field.  To avoid getting very odd results, either
do not use the C<dPort> field when your key includes ICMP field(s) or
be certain to include the C<protocol> field as part of your key.  This
field was introduced in SiLK 3.8.1.

=item iCode

the ICMP code value for ICMP or ICMPv6 flows and empty for non-ICMP
flows.  See note at C<iType>.

=item icmpTypeCode,25

equivalent to C<iType>,C<iCode> when used in B<--fields>.  This field
may not be mixed with C<iType> or C<iCode>, and this field is
deprecated as of SiLK 3.8.1.  As of SiLK 3.8.1, C<icmpTypeCode> may no
longer be used as the argument to the C<Distinct:> value field; the
C<dPort> field provides an equivalent result as long as the input
is limited to ICMP flow records.

=back

Many SiLK file formats do not store the following fields and their
values are always be 0; they are listed here for completeness:

=over 4

=item in,13

router SNMP input interface or vlanId if packing tools were
configured to capture it (see B<sensor.conf(5)>)

=item out,14

router SNMP output interface or postVlanId

=item nhIP,15

router next hop IP

=back

SiLK can store flows generated by enhanced collection software that
provides more information than NetFlow v5.  These flows may support
some or all of these additional fields; for flows without this
additional information, the field's value is always 0.

=over 4

=item initialFlags,26

TCP flags on first packet in the flow

=item sessionFlags,27

bit-wise OR of TCP flags over all packets except the first in the flow

=item attributes,28

flow attributes set by the flow generator:

=over 4

=item C<S>

all the packets in this flow record are exactly the same size

=item C<F>

flow generator saw additional packets in this flow following a packet
with a FIN flag (excluding ACK packets)

=item C<T>

flow generator prematurely created a record for a long-running
connection due to a timeout.  (When the flow generator B<yaf(1)> is
run with the B<--silk> switch, it prematurely creates a flow and
mark it with C<T> if the byte count of the flow cannot be stored in a
32-bit value.)

=item C<C>

flow generator created this flow as a continuation of long-running
connection, where the previous flow for this connection met a timeout
(or a byte threshold in the case of B<yaf>).

=back

Consider a long-running ssh session that exceeds the flow generator's
I<active> timeout.  (This is the active timeout since the flow
generator creates a flow for a connection that still has activity).
The flow generator will create multiple flow records for this ssh
session, each spanning some portion of the total session.  The first
flow record will be marked with a C<T> indicating that it hit the
timeout.  The second through next-to-last records will be marked with
C<TC> indicating that this flow both timed out and is a continuation
of a flow that timed out.  The final flow will be marked with a C<C>,
indicating that it was created as a continuation of an active flow.

=item application,29

guess as to the content of the flow.  Some software that generates flow
records from packet data, such as B<yaf>, will inspect the contents of
the packets that make up a flow and use traffic signatures to label
the content of the flow.  SiLK calls this label the I<application>;
B<yaf> refers to it as the I<appLabel>.  The application is the port
number that is traditionally used for that type of traffic (see the
F</etc/services> file on most UNIX systems).  For example, traffic
that the flow generator recognizes as FTP will have a value of 21,
even if that traffic is being routed through the standard HTTP/web
S<port (80)>.

=back

The following fields provide a way to label the IPs or ports on a
record.  These fields require external files to provide the mapping
from the IP or port to the label:

=over 4

=item sType,16

for the source IP address, the value 0 if the address is non-routable,
1 if it is internal, or 2 if it is routable and external.  Uses the
mapping file specified by the SILK_ADDRESS_TYPES environment variable,
or the F<address_types.pmap> mapping file, as described in
B<addrtype(3)>.

=item dType,17

as B<sType> for the destination IP address

=item scc,18

for the source IP address, a two-letter country code abbreviation
denoting the country where that IP address is located.  Uses the
mapping file specified by the SILK_COUNTRY_CODES environment variable,
or the F<country_codes.pmap> mapping file, as described in
B<ccfilter(3)>.  The abbreviations are those defined by ISO 3166-1
(see for example L<https://www.iso.org/iso-3166-country-codes.html>
or L<https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2>) or the
following special codes: B<--> N/A (e.g. private and experimental
reserved addresses); B<a1> anonymous proxy; B<a2> satellite provider;
B<o1> other

=item dcc,19

as B<scc> for the destination IP

=item src-I<map-name>

label contained in the prefix map file associated with I<map-name>.
If the prefix map is for IP addresses, the label is that associated
with the source IP address.  If the prefix map is for protocol/port
pairs, the label is that associated with the protocol and source port.
See also the description of the B<--pmap-file> switch below and the
B<pmapfilter(3)> manual page.

=item dst-I<map-name>

as B<src-I<map-name>> for the destination IP address or the protocol
and destination port.

=item sval

as B<src-I<map-name>> when no map-name is associated with the prefix
map file

=item dval

as B<dst-I<map-name>> when no map-name is associated with the prefix
map file

=back

Finally, the list of built-in fields may be augmented by the run-time
loading of PySiLK code or plug-ins written in C (also called shared
object files or dynamic libraries), as described by the
B<--python-file> and B<--plugin> switches.

=for comment
##########################################################################
# Whew!  We've finally reached the end of the --fields help

=item B<--values>=I<VALUES>

Specify the aggregate values to compute for each bin as a comma
separated list of names.  Names are case insensitive.  When the
B<--threshold> switch specifies an aggregate value field that does
appear in I<VALUES>, that field is appended to I<VALUES>.  When
neither the B<--values> switch nor any B<--threshold> switch is
specified, B<rwuniq> counts the number of flow records for each bin.
The aggregate fields are printed in the order they occur in I<VALUES>.
The names of the built-in value fields follow.  This list can be
augmented through the use of PySiLK and plug-ins.

=over 4

=item Records

Count the number of flow records that mapped to each bin.

=item Packets

Sum the number of packets across all records that mapped to each bin.

=item Bytes

Sum the number of bytes across all records that mapped to each bin.

=item sTime-Earliest

Keep track of the earliest start time (minimum time) seen across all
records that mapped to each bin, in seconds resolution.  The
B<--bin-time> switch does not normally affect this value; however,
this value uses milliseconds resolution when B<--bin-time> includes
fractional seconds.

=item eTime-Latest

Keep track of the latest end time (maximum time) seen across all
records that mapped to each bin, in seconds resolution.  The
B<--bin-time> switch does not normally affect this value; however,
this value uses milliseconds resolution when B<--bin-time> includes
fractional seconds.

=item sIP-Distinct

Count the number of distinct source IP addresses that were seen for
each bin, an alias for Distinct:sIP.

=item dIP-Distinct

Count the number of distinct destination IP addresses that were seen
for each bin, an alias for Distinct:dIP.

=item Distinct:I<KEY_FIELD>

Count the number of distinct values for I<KEY_FIELD>, where
I<KEY_FIELD> is any field that can be used as an argument to
B<--fields> except C<icmpTypeCode>.  For example, C<Distinct:sPort>
counts the number of distinct source ports for each bin.  When this
aggregate value field is used, the specified I<KEY_FIELD> cannot be
present in the argument to B<--fields>.

=item Flows

Count the number of flow records that mapped to each bin; an alias for
Records.

=back

=item B<--plugin>=I<PLUGIN>

Augment the list of key fields and/or aggregate value fields by using
run-time loading of the plug-in (shared object) whose path is
I<PLUGIN>.  The switch may be repeated to load multiple plug-ins.  The
creation of plug-ins is described in the B<silk-plugin(3)> manual
page.  When I<PLUGIN> does not contain a slash (C</>), B<rwuniq>
attempts to find a file named I<PLUGIN> in the directories listed in
the L</FILES> section.  If B<rwuniq> finds the file, it uses that
path.  If I<PLUGIN> contains a slash or if B<rwuniq> does not find the
file, B<rwuniq> relies on your operating system's B<dlopen(3)> call to
find the file.  When the SILK_PLUGIN_DEBUG environment variable is
non-empty, B<rwuniq> prints status messages to the standard error as
it attempts to find and open each of its plug-ins.

=item B<--threshold>=I<VALUE_FIELD>B<=>I<MIN>B<->I<MAX>

=item B<--threshold>=I<VALUE_FIELD>B<=>I<MIN>

Limit the output of B<rwuniq> to the bins where the value of the
aggregate value field I<VALUE_FIELD> is not less than I<MIN> and not
more than I<MAX>.  If I<MAX> is not given, limit the output to the
bins where the value of I<VALUE_FIELD> is at least I<MIN>.  The
I<VALUE_FIELD> argument is case insensitive and may be abbreviated to
the shortest unique prefix.  This switch may be repeated to set
thresholds for multiple fields, and B<rwuniq> only prints bins that
meet all thresholds.  A I<MIN> of 0 is treated as 1.  If
I<VALUE_FIELD> is not present in the argument to the B<--values>
switch, it is appended to those aggregate values.  I<VALUE_FIELD> may
be B<Records> (or B<Flows)>, B<Packets>, B<Bytes>, B<sIP-Distinct>,
B<dIP-Distinct>, or B<Distinct:>I<KEY_FIELD>.  Setting thresholds for
aggregate value fields defined by plug-ins is not supported.  I<Since
SiLK 3.17.0.>

=back

Miscellaneous options:

=over 4

=item B<--presorted-input>

Cause B<rwuniq> to assume that it is reading sorted input; i.e., that
B<rwuniq>'s input file(s) were generated by B<rwsort(1)> using the
I<exact same> value for the B<--fields> switch.  When no distinct
counts are being computed, B<rwuniq> can process its input without
needing to write temporary files.  When multiple input files are
specified, B<rwuniq> merge-sorts the flow records from the input
files.  See the L</NOTES> section for issues that may occur when using
B<--presorted-input>.

=item B<--sort-output>

Cause B<rwuniq> to present the output in sorted numerical order.  The
key B<rwuniq> uses for sorting is the same key it uses to index each
bin.

=item B<--bin-time>=I<SECONDS>

=item B<--bin-time>

Adjust the times in the key fields C<sTime> and C<eTime> to appear on
I<SECONDS>-second boundaries (the floor of the time is used).  As of
SiLK 3.17.0, I<SECONDS> may be a fractional value of 0.001 or greater,
and B<rwuniq> uses millisecond timestamps when I<SECONDS> includes a
fractional value that is non-zero.  When this switch is not specified,
times appear on 1-second boundaries.  When the switch is used but no
argument is given, B<rwuniq> uses 60-second time bins.  (When the
start-time is the only key field and time binning is desired, consider
using B<rwcount(1)> instead.)

=item B<--timestamp-format>=I<FORMAT>

Specify the format and/or timezone to use when printing timestamps.
When this switch is not specified, the SILK_TIMESTAMP_FORMAT
environment variable is checked for a default format and/or timezone.
If it is empty or contains invalid values, timestamps are printed in
the default format, and the timezone is UTC unless SiLK was compiled
with local timezone support.  I<FORMAT> is a comma-separated list of a
format and/or a timezone.  The format is one of:

=over 4

=item default

Print the timestamps as C<I<YYYY>/I<MM>/I<DD>TI<hh>:I<mm>:I<ss>>.

=item iso

Print the timestamps as S<C<I<YYYY>-I<MM>-I<DD> I<hh>:I<mm>:I<ss>>>.

=item m/d/y

Print the timestamps as S<C<I<MM>/I<DD>/I<YYYY> I<hh>:I<mm>:I<ss>>>.

=item epoch

Print the timestamps as the number of seconds since 00:00:00 UTC on
1970-01-01.

=back

When a timezone is specified, it is used regardless of the default
timezone support compiled into SiLK.  The timezone is one of:

=over 4

=item utc

Use Coordinated Universal Time to print timestamps.

=item local

Use the TZ environment variable or the local timezone.

=back

=item B<--epoch-time>

Print timestamps as epoch time (number of seconds since midnight GMT
on 1970-01-01).  This switch is equivalent to
B<--timestamp-format=epoch>, it is deprecated as of SiLK 3.0.0, and it
will be removed in the SiLK 4.0 release.

=item B<--ip-format>=I<FORMAT>

Specify how IP addresses are printed, where I<FORMAT> is a
comma-separated list of the arguments described below.  When this
switch is not specified, the SILK_IP_FORMAT environment variable is
checked for a value and that format is used if it is valid.  The
default I<FORMAT> is C<canonical>.  I<Since SiLK 3.7.0.>

=over 4

=item canonical

Print IP addresses in the canonical format.  If the key only contains
IPv4 addresses, use dot-separated decimal (C<192.0.2.1>).  Otherwise,
use colon-separated hexadecimal (C<2001:db8::1>) or a mixed IPv4-IPv6
representation for IPv4-mapped IPv6 addresses (the ::ffff:0:0/96
netblock, e.g., C<::ffff:192.0.2.1>) and IPv4-compatible IPv6
addresses (the ::/96 netblock other than ::/127, e.g.,
C<::192.0.2.1>).

=item no-mixed

Print IP addresses in the canonical format (C<192.0.2.1> or
C<2001:db8::1>) but do not used the mixed IPv4-IPv6 representations.
For example, use C<::ffff:c000:201> instead of C<::ffff:192.0.2.1>.
I<Since SiLK 3.17.0>.

=item decimal

Print IP addresses as integers in decimal format.  For example, print
C<192.0.2.1> and C<2001:db8::1> as C<3221225985> and
C<42540766411282592856903984951653826561>, respectively.

=item hexadecimal

Print IP addresses as integers in hexadecimal format.  For example,
print C<192.0.2.1> and C<2001:db8::1> as C<c00000201> and
C<20010db8000000000000000000000001>, respectively.

=item zero-padded

Make all IP address strings contain the same number of characters by
padding numbers with leading zeros.  For example, print C<192.0.2.1>
and C<2001:db8::1> as C<192.000.002.001> and
C<2001:0db8:0000:0000:0000:0000:0000:0001>, respectively.  For IPv6
addresses, this setting implies C<no-mixed>, so that
C<::ffff:192.0.2.1> is printed as
C<0000:0000:0000:0000:0000:ffff:c000:0201>.  As of SiLK 3.17.0, may be
combined with any of the above, including C<decimal> and
C<hexadecimal>.

=back

The following arguments modify certain IP addresses prior to printing.
These arguments may be combined with the above formats.

=over 4

=item map-v4

Change IPv4 addresses to IPv4-mapped IPv6 addresses (addresses in the
::ffff:0:0/96 netblock) prior to formatting.  I<Since SiLK 3.17.0>.

=item unmap-v6

When the key contains IPv6 addresses, change any IPv4-mapped IPv6
addresses (addresses in the ::ffff:0:0/96 netblock) to IPv4 addresses
prior to formatting.  I<Since SiLK 3.17.0>.

=back

The following argument is also available:

=over 4

=item force-ipv6

Set I<FORMAT> to C<map-v4>,C<no-mixed>.

=back

=item B<--integer-ips>

Print IP addresses as integers.  This switch is equivalent to
B<--ip-format=decimal>, it is deprecated as of SiLK 3.7.0, and it will
be removed in the SiLK 4.0 release.

=item B<--zero-pad-ips>

Print IP addresses as fully-expanded, zero-padded values in their
canonical form.  This switch is equivalent to
B<--ip-format=zero-padded>, it is deprecated as of SiLK 3.7.0, and it
will be removed in the SiLK 4.0 release.

=item B<--integer-sensors>

Print the integer ID of the sensor rather than its name.

=item B<--integer-tcp-flags>

Print the TCP flag fields (flags, initialFlags, sessionFlags) as an
integer value.  Typically, the characters C<F,S,R,P,A,U,E,C> are used
to represent the TCP flags.

=item B<--no-titles>

Turn off column titles.  By default, titles are printed.

=item B<--no-columns>

Disable fixed-width columnar output.

=item B<--column-separator>=I<C>

Use specified character between columns and after the final column.
When this switch is not specified, the default of 'B<|>' is used.

=item B<--no-final-delimiter>

Do not print the column separator after the final column.  Normally a
delimiter is printed.

=item B<--delimited>

=item B<--delimited>=I<C>

Run as if B<--no-columns> B<--no-final-delimiter> B<--column-sep>=I<C>
had been specified.  That is, disable fixed-width columnar output; if
character I<C> is provided, it is used as the delimiter between
columns instead of the default 'B<|>'.

=item B<--print-filenames>

Print to the standard error the names of input files as they are
opened.

=item B<--copy-input>=I<PATH>

Copy all binary SiLK Flow records read as input to the specified file
or named pipe.  I<PATH> may be C<stdout> or C<-> to write flows to the
standard output as long as the B<--output-path> switch is specified to
redirect B<rwuniq>'s textual output to a different location.

=item B<--output-path>=I<PATH>

Write the textual output to I<PATH>, where I<PATH> is a filename, a
named pipe, the keyword C<stderr> to write the output to the standard
error, or the keyword C<stdout> or C<-> to write the output to the
standard output (and bypass the paging program).  If I<PATH> names an
existing file, B<rwuniq> exits with an error unless the SILK_CLOBBER
environment variable is set, in which case I<PATH> is overwritten.  If
this switch is not given, the output is either sent to the pager or
written to the standard output.

=item B<--pager>=I<PAGER_PROG>

When output is to a terminal, invoke the program I<PAGER_PROG> to view
the output one screen full at a time.  This switch overrides the
SILK_PAGER environment variable, which in turn overrides the PAGER
variable.  If the B<--output-path> switch is given or if the value of
the pager is determined to be the empty string, no paging is performed
and all output is written to the terminal.

=item B<--ipv6-policy>=I<POLICY>

Determine how IPv4 and IPv6 flows are handled when SiLK has been
compiled with IPv6 support.  When the switch is not provided, the
SILK_IPV6_POLICY environment variable is checked for a policy.  If it
is also unset or contains an invalid policy, the I<POLICY> is
B<mix>.  When SiLK has not been compiled with IPv6 support, IPv6
flows are always ignored, regardless of the value passed to this
switch or in the SILK_IPV6_POLICY variable.  The supported values for
I<POLICY> are:

=over 4

=item ignore

Ignore any flow record marked as IPv6, regardless of the IP addresses
it contains.

=item asv4

Convert IPv6 flow records that contain addresses in the ::ffff:0:0/96
netblock (that is, IPv4-mapped IPv6 addresses) to IPv4 and ignore all
other IPv6 flow records.

=item mix

Process the input as a mixture of IPv4 and IPv6 flow records.  When an
IP address is used as part of the key or value, this policy is
equivalent to B<force>.

=item force

Convert IPv4 flow records to IPv6, mapping the IPv4 addresses into the
::ffff:0:0/96 netblock.

=item only

Process only flow records that are marked as IPv6 and ignore IPv4 flow
records in the input.

=back

=item B<--temp-directory>=I<DIR_PATH>

Specify the name of the directory in which to store data files
temporarily when the memory is not large enough to store all the bins
and their aggregate values.  This switch overrides the directory
specified in the SILK_TMPDIR environment variable, which overrides the
directory specified in the TMPDIR variable, which overrides the
default, F</tmp>.

=item B<--site-config-file>=I<FILENAME>

Read the SiLK site configuration from the named file I<FILENAME>.
When this switch is not provided, B<rwuniq> searches for the site
configuration file in the locations specified in the L</FILES>
section.

=item B<--legacy-timestamps>

=item B<--legacy-timestamps>=I<NUM>

When I<NUM> is not specified or is 1, this switch is equivalent to
B<--timestamp-format=m/d/y>.  Otherwise, the switch has no effect.
This switch is deprecated as of SiLK 3.0.0, and it will be removed in
the SiLK 4.0 release.

=item B<--xargs>

=item B<--xargs>=I<FILENAME>

Read the names of the input files from I<FILENAME> or from the
standard input if I<FILENAME> is not provided.  The input is expected
to have one filename per line.  B<rwuniq> opens each named file in
turn and reads records from it as if the filenames had been listed on
the command line.

=item B<--help>

Print the available options and exit.  Specifying switches that add
new fields, values, or additional switches before B<--help> allows
the output to include descriptions of those fields or switches.

=item B<--help-fields>

Print the description and alias(es) of each field and value and exit.
Specifying switches that add new fields before B<--help-fields>
allows the output to include descriptions of those fields.

=item B<--version>

Print the version number and information about how SiLK was
configured, then exit the application.

=item B<--pmap-file>=I<PATH>

=item B<--pmap-file>=I<MAPNAME>:I<PATH>

Load the prefix map file located at I<PATH> and create fields named
src-I<map-name> and dst-I<map-name> where I<map-name> is either the
I<MAPNAME> part of the argument or the map-name specified when the
file was created (see B<rwpmapbuild(1)>).  If no map-name is
available, B<rwuniq> names the fields C<sval> and C<dval>.  Specify
I<PATH> as C<-> or C<stdin> to read from the standard input.  The
switch may be repeated to load multiple prefix map files, but each
prefix map must use a unique map-name.  The B<--pmap-file> switch(es)
must precede the B<--fields> switch.  See also B<pmapfilter(3)>.

=item B<--pmap-column-width>=I<NUM>

When printing a label associated with a prefix map, this switch gives
the maximum number of characters to use when displaying the textual
value of the field.

=item B<--python-file>=I<PATH>

When the SiLK Python plug-in is used, B<rwuniq> reads the Python code
from the file I<PATH> to define additional fields that can be used as
part of the key or as an aggregate value.  This file should call
B<register_field()> for each field it wishes to define.  For details
and examples, see the B<silkpython(3)> and B<pysilk(3)> manual pages.

=back

=head2 Deprecated volume switches

These options add the named aggregate field(s) to B<--values> if the
field is not present.  When an argument is specified, the switch is
equivalent to a B<--threshold> switch.  Use of these switches is
deprecated.

=over 4

=item B<--all-counts>

Append the following fields to the argument of the B<--values> switch
unless the field is already present: B<Bytes>, B<Packets>, B<Records>,
B<sTime-Earliest>, and B<eTime-Latest>.  Deprecated since SiLK 2.0.0.

=item B<--bytes>

Append B<Bytes> to the argument of the B<--values> switch unless it is
already present.  Deprecated since SiLK 2.0.0.

=item B<--bytes>=I<MIN>

Add B<--threshold=bytes>=I<MIN> to the options.  Deprecated since SiLK
3.17.0.

=item B<--bytes>=I<MIN>-I<MAX>

Add B<--threshold=bytes>=I<MIN>-I<MAX> to the options.  Deprecated
since SiLK 3.17.0.

=item B<--packets>

Append B<Packets> to the argument of the B<--values> switch unless it
is already present.  Deprecated since SiLK 2.0.0.

=item B<--packets>=I<MIN>

Add B<--threshold=packets>=I<MIN> to the options.  Deprecated since
SiLK 3.17.0.

=item B<--packets>=I<MIN>-I<MAX>

Add B<--threshold=packets>=I<MIN>-I<MAX> to the options.  Deprecated
since SiLK 3.17.0.

=item B<--flows>

Append B<Records> to the argument of the B<--values> switch unless it
is already present.  Deprecated since SiLK 2.0.0.

=item B<--flows>=I<MIN>

Add B<--threshold=records>=I<MIN> to the options.  Deprecated since
SiLK 3.17.0.

=item B<--flows>=I<MIN>-I<MAX>

Add B<--threshold=records>=I<MIN>-I<MAX> to the options.  Deprecated
since SiLK 3.17.0.

=item B<--sip-distinct>

Append B<Distinct:sIP> to the argument of the B<--values> switch
unless it is already present.  Deprecated since SiLK 2.0.0.

=item B<--sip-distinct>=I<MIN>

Add B<--threshold=distinct:sip>=I<MIN> to the options.  Deprecated
since SiLK 3.17.0.

=item B<--sip-distinct>=I<MIN>-I<MAX>

Add B<--threshold=distinct:sip>=I<MIN>-I<MAX> to the options.
Deprecated since SiLK 3.17.0.

=item B<--dip-distinct>

Append B<Distinct:dIP> to the argument of the B<--values> switch
unless it is already present.  Deprecated since SiLK 2.0.0.

=item B<--dip-distinct>=I<MIN>

Add B<--threshold=distinct:dip>=I<MIN> to the options.  Deprecated
since SiLK 3.17.0.

=item B<--dip-distinct>=I<MIN>-I<MAX>

Add B<--threshold=distinct:dip>=I<MIN>-I<MAX> to the options.
Deprecated since SiLK 3.17.0.

=item B<--stime>

Append B<sTime-Earliest> to the argument of the B<--values> switch
unless it is already present.  Deprecated since SiLK 2.0.0.

=item B<--etime>

Append B<eTime-Latest> to the argument of the B<--values> switch
unless it is already present.  Deprecated since SiLK 2.0.0.

=back

=head1 EXAMPLES

In these examples, the dollar sign (C<$>) represents the shell prompt
and a backslash (C<\>) is used to continue a line for better
readability.  Many examples assume previous B<rwfilter(1)> commands
have written data files named F<data.rw> and F<data-v6.rw>.

=for comment
The output for nearly all commands is generated from the "make check"
test data.  All commands assume data.rw only contains the incoming
data, that is "rwfilter --type=in,inweb".

The B<--fields> switch is required to specify which field(s) comprise
the key.  By default, B<rwuniq> counts the number of records for each
key.  This example uses the source port as the key.

 $ rwuniq --fields=sport data.rw | head
 sPort|   Records|
    53|     62216|
    22|     27994|
    67|      7807|
 29897|        78|
 28816|        24|
    80|     27044|
 28925|        22|
     0|      7801|
 29246|        63|

Notice how the keys are printed in an arbitrary order.  Use the
B<--sort-output> switch to arrange the keys from lowest to highest.

 $ rwuniq --fields=sport --sort-output data.rw | head
 sPort|   Records|
     0|      7801|
    22|     27994|
    25|     15568|
    53|     62216|
    67|      7807|
    80|     27044|
   123|      7741|
   443|      7917|
  8080|      3946|

To sort the output by a volume field (such as the number of records),
use B<rwstats(1)>.

 $ rwstats --fields=sport --count=10 data.rw
 INPUT: 250928 Records for 4739 Bins and 250928 Total Records
 OUTPUT: Top 10 Bins by Records
 sPort|   Records|  %Records|   cumul_%|
    53|     62216| 24.794363| 24.794363|
    22|     27994| 11.156188| 35.950552|
    80|     27044| 10.777594| 46.728145|
    25|     15568|  6.204170| 52.932315|
   443|      7917|  3.155088| 56.087404|
    67|      7807|  3.111251| 59.198655|
     0|      7801|  3.108860| 62.307515|
   123|      7741|  3.084949| 65.392463|
  8080|      3946|  1.572563| 66.965026|
 29921|       117|  0.046627| 67.011653|

Alternatively, process the textual output of B<rwuniq> with the UNIX
B<sort(1)> utility.

 $ rwuniq --fields=sport data.rw  \
   | sort -r -t '|' -k 2 | head
 sPort|   Records|
    53|     62216|
    22|     27994|
    80|     27044|
    25|     15568|
   443|      7917|
    67|      7807|
     0|      7801|
   123|      7741|
  8080|      3946|

Use the B<--values> field to change the volume that B<rwuniq> computes
for each key.  This example prints the byte-, packet-, and
record-counts for each protocol, sorting the results by protocol.

 $ rwuniq --fields=proto --values=bytes,packets,records --sort data.rw
 pro|               Bytes|        Packets|   Records|
   1|             5344836|          73473|      7801|
   6|         59945492930|       72127917|    165363|
  17|            17553593|          77764|     77764|

The B<--threshold> switch limits the output to rows where a value
field meets a minimum value or falls within a specific range.  For
example, print the number of records and packets seen for each source
port for bins having at least 1000 records.

 $ rwuniq --fields=sport --values=records,packets \
        --threshold=records=1000 data.rw
 sPort|   Records|        Packets|
    53|     62216|          62216|
    22|     27994|       23434615|
    67|      7807|           7807|
    80|     27044|        8271125|
     0|      7801|          73473|
   123|      7741|           7741|
    25|     15568|         427777|
   443|      7917|        2421124|
  8080|      3946|        1202528|

Multiple thresholds may be specified.

 $ rwuniq --fields=sport --values=records,packets                 \
        --threshold=records=1000-5000 --threshold=packets=1000000 \
        data.rw
 sPort|   Records|        Packets|
  8080|      3946|        1202528|

The B<--bin-time> switch adjusts the times used by the C<sTime> and
C<eTime> key fields.  An argument of 86400 moves the starting and
ending time to day boundaries.

 $ rwuniq --bin-time=86400 --fields=stime,etime data.rw
               sTime|              eTime|   Records|
 2009/02/12T00:00:00|2009/02/12T00:00:00|     82969|
 2009/02/12T00:00:00|2009/02/13T00:00:00|       360|
 2009/02/13T00:00:00|2009/02/13T00:00:00|     83594|
 2009/02/13T00:00:00|2009/02/14T00:00:00|       332|
 2009/02/14T00:00:00|2009/02/14T00:00:00|     83673|

The B<--bin-time> switch does not adjust the C<duration> value unless
both C<sTime> and C<eTime> are given.

 $ rwuniq --bin-time=86400 --fields=stime,dur --sort data.rw | head -6
               sTime|durat|   Records|
 2009/02/12T00:00:00|    0|     29523|
 2009/02/12T00:00:00|    1|      4312|
 2009/02/12T00:00:00|    2|      4376|
 2009/02/12T00:00:00|    3|      3986|
 2009/02/12T00:00:00|    4|       923|

 $ rwuniq --bin-time=86400 --fields=stime,dur,etime data.rw
               sTime|durat|              eTime|   Records|
 2009/02/12T00:00:00|    0|2009/02/12T00:00:00|     82969|
 2009/02/12T00:00:00|86400|2009/02/13T00:00:00|       360|
 2009/02/13T00:00:00|    0|2009/02/13T00:00:00|     83594|
 2009/02/13T00:00:00|86400|2009/02/14T00:00:00|       332|
 2009/02/14T00:00:00|    0|2009/02/14T00:00:00|     83673|

As of SiLK 3.17.0, the B<--bin-time> switch accepts a floating point
value.  When the fractional part is non-zero, B<rwuniq> uses
millisecond precision for the times and the duration.

 $ rwuniq --bin-time=0.001 --fields=duration data.rw | head -6
  duration|   Records|
     0.000|     85565|
  1791.045|         4|
     2.120|        19|
    22.263|         5|
    19.902|         3|

The B<--bin-time> does not adjust the C<sTime-Earliest> and
C<eTime-Latest> aggregate value fields, but it does determine whether
those fields maintain millisecond precision.

 $ rwuniq --bin-time=86400 --fields=stime --value=etime data.rw
               sTime|       eTime-Latest|
 2009/02/12T00:00:00|2009/02/12T00:29:59|
 2009/02/13T00:00:00|2009/02/13T00:29:58|
 2009/02/14T00:00:00|2009/02/14T00:29:59|

 $ rwuniq --bin-time=0.001 --fields=proto --value=stime,etime data.rw
 pro|         sTime-Earliest|           eTime-Latest|
  17|2009/02/12T00:00:02.745|1970/01/15T06:57:35.997|
   6|2009/02/12T00:00:03.004|1970/01/15T06:57:35.998|
   1|2009/02/12T00:00:20.601|1970/01/15T06:57:35.992|

With an input of both IPv4 and IPv6 records, B<rwuniq> maps the IPv4
records into the ::ffff:0:0/96 netblock.  The data is normally mapped
back to IPv4 on output.  Given this input:

 $ rwcut --fields=sip,packets /tmp/v4v6.rw
                                     sIP|   packets|
                                     ::1|        45|
                              192.0.2.22|        87|
                    ::ffff:203.0.113.113|      2662|
                  2001:db8:54:32:ab:cd::|       345|

The B<rwuniq> tool produces:

 $ rwuniq --fields=sip --values=packets /tmp/v4v6.rw
                                     sIP|        Packets|
                                     ::1|             45|
                              192.0.2.22|             87|
                           203.0.113.113|           2662|
                  2001:db8:54:32:ab:cd::|            345|

Set the B<--ip-format> to map-v4 to leave the values as IPv4-mapped
IPv6.  (Using an B<--ipv6-policy> of C<force-ipv6> has the same
effect.)

 $ rwuniq --fields=sip --values=packets --ip-format=map-v4 /tmp/v4v6.rw
                                     sIP|        Packets|
                                     ::1|             45|
                       ::ffff:192.0.2.22|             87|
                    ::ffff:203.0.113.113|           2662|
                  2001:db8:54:32:ab:cd::|            345|

Print the source addresses that sent more than 10,000,000 bytes, and
for each address print the number of unique destination hosts it
contacted:

 $ rwuniq --fields=sip --values=bytes,distinct:dip \
        --threshold=bytes=10000000 data-v6.rw
                       sIP|               Bytes|dIP-Distin|
      2001:db8:a:fd::90:bd|            14529210|         2|

Print the number of bytes that host shared with each destination
(first use B<rwfilter> to limit the input to that host):

 $ rwfilter --saddr=2001:db8:a:fd::90:bd --pass=- data-v6.rw        \
   | rwuniq --fields=dip --values=bytes
                       dIP|               Bytes|
     2001:db8:c0:a8::fa:5d|             7097847|
      2001:db8:c0:a8::dd:6|             7431363|

Print the packet and byte counts for each IPv4 source-destination
pair, where the prefix length is 16 (use B<rwnetmask(1)> on the input
to B<rwuniq>):

 $ rwnetmask --4sip-prefix=16 --4dip-prefix=16 data.rw      \
   | rwuniq --fields=sip,dip --values=packet,byte | head
            sIP|            dIP|  Packets|        Bytes|
     10.139.0.0|    192.168.0.0|    33490|     22950353|
      10.40.0.0|    192.168.0.0|      258|        18544|
     10.204.0.0|    192.168.0.0|   353233|    288736424|
     10.106.0.0|    192.168.0.0|    13051|      3843693|
      10.71.0.0|    192.168.0.0|     4355|      1391194|
      10.98.0.0|    192.168.0.0|     7312|      7328359|
     10.114.0.0|    192.168.0.0|     2538|      4137927|
     10.168.0.0|    192.168.0.0|    92094|     86883062|
     10.176.0.0|    192.168.0.0|   122101|    116555051|

Given a file of scan traffic, print the source of TCP traffic with no
more than 3 packets and which also appears at least 4 times.  First
use B<rwfilter> to limit the traffic to TCP and find the flow records
where the packet count in that flow record is no more than 3.

 $ rwfilter --proto=6 --packets=1-3 --pass=- scandata.rw          \
   | rwuniq --field=sip --values=flow,packets --threshold=flows=4 \
   | head -5
             sIP|   Records|        Packets|
   10.249.216.38|       256|            256|
    10.155.55.93|       256|            256|
   10.61.255.154|       256|            256|
    10.60.122.82|       256|            256|

The B<silkpython(3)> manual page provides examples that use PySiLK to
create arbitrary fields to use as part of the key for B<rwuniq>.

When using B<rwuniq> on input that contains both incoming and outgoing
flow records, consider using the B<int-ext-fields(3)> plug-in which
defines four additional fields representing the external IP address,
the external port, the internal IP address, and the internal port.
The plug-in requires the user to specify which class/type pairs are
incoming and which are outgoing.  See its manual page for additional
information.  As an example, here we run B<rwuniq> on a file
containing incoming and outgoing web traffic.

 $ rwuniq --fields=sip,sport,dip,dport --values=bytes \
        --sort-output data.rw | head -7
             sIP|sPort|            dIP|dPort|               Bytes|
     10.4.52.235|29631|192.168.233.171|   80|               18260|
    10.5.231.251|   80|192.168.226.129|28770|              536169|
     10.9.77.117|29906| 192.168.184.65|   80|               55386|
     10.11.88.88|   80|192.168.251.222|28902|              433198|
   10.14.110.214|29989| 192.168.249.96|   80|               25903|
    10.15.224.27|  443| 192.168.231.49|29779|              163759|

Here the B<int-ext-fields> plug-in is used:

 $ export INCOMING_FLOWTYPES=all/in,all/inweb
 $ export OUTGOING_FLOWTYPES=all/out,all/outweb
 $ rwuniq --plugin=int-ext-fields.so \
        --fields=ext-ip,ext-port,int-ip,int-port --value=bytes \
        --sort-output data.rw | head -7
          ext-ip|ext-p|         int-ip|int-p|               Bytes|
     10.4.52.235|29631|192.168.233.171|   80|              726111|
    10.5.231.251|   80|192.168.226.129|28770|              561654|
     10.9.77.117|29906| 192.168.184.65|   80|             1811738|
     10.11.88.88|   80|192.168.251.222|28902|              444277|
   10.14.110.214|29989| 192.168.249.96|   80|              393068|
    10.15.224.27|  443| 192.168.231.49|29779|              167696|

=head1 ENVIRONMENT

=over 4

=item SILK_IPV6_POLICY

This environment variable is used as the value for B<--ipv6-policy>
when that switch is not provided.

=item SILK_IP_FORMAT

This environment variable is used as the value for B<--ip-format> when
that switch is not provided.  I<Since SiLK 3.11.0.>

=item SILK_TIMESTAMP_FORMAT

This environment variable is used as the value for
B<--timestamp-format> when that switch is not provided.  I<Since SiLK
3.11.0.>

=item SILK_PAGER

When set to a non-empty string, B<rwuniq> automatically invokes this
program to display its output a screen at a time.  If set to an empty
string, B<rwuniq> does not automatically page its output.

=item PAGER

When set and SILK_PAGER is not set, B<rwuniq> automatically invokes
this program to display its output a screen at a time.

=item SILK_TMPDIR

When set and B<--temp-directory> is not specified, B<rwuniq> writes
the temporary files it creates to this directory.  SILK_TMPDIR
overrides the value of TMPDIR.

=item TMPDIR

When set and SILK_TMPDIR is not set, B<rwuniq> writes the temporary
files it creates to this directory.

=item PYTHONPATH

This environment variable is used by Python to locate modules.  When
B<--python-file> is specified, B<rwuniq> must load the Python files
that comprise the PySiLK package, such as F<silk/__init__.py>.  If
this F<silk/> directory is located outside Python's normal search path
(for example, in the SiLK installation tree), it may be necessary to
set or modify the PYTHONPATH environment variable to include the
parent directory of F<silk/> so that Python can find the PySiLK
module.

=item SILK_PYTHON_TRACEBACK

When set, Python plug-ins print traceback information on Python
errors to the standard error.

=item SILK_COUNTRY_CODES

This environment variable allows the user to specify the country code
mapping file that B<rwuniq> uses when computing the scc and dcc
fields.  The value may be a complete path or a file relative to the
SILK_PATH.  See the L</FILES> section for standard locations of this
file.

=item SILK_ADDRESS_TYPES

This environment variable allows the user to specify the address type
mapping file that B<rwuniq> uses when computing the sType and dType
fields.  The value may be a complete path or a file relative to the
SILK_PATH.  See the L</FILES> section for standard locations of this
file.

=item SILK_CLOBBER

The SiLK tools normally refuse to overwrite existing files.  Setting
SILK_CLOBBER to a non-empty value removes this restriction.

=item SILK_CONFIG_FILE

This environment variable is used as the value for the
B<--site-config-file> when that switch is not provided.

=item SILK_DATA_ROOTDIR

This environment variable specifies the root directory of data
repository.  As described in the L</FILES> section, B<rwuniq> may
use this environment variable when searching for the SiLK site
configuration file.

=item SILK_PATH

This environment variable gives the root of the install tree.  When
searching for configuration files and plug-ins, B<rwuniq> may use this
environment variable.  See the L</FILES> section for details.

=item TZ

When the argument to the B<--timestamp-format> switch includes
C<local> or when a SiLK installation is built to use the local
timezone, the value of the TZ environment variable determines the
timezone in which B<rwuniq> displays timestamps.  (If both of
those are false, the TZ environment variable is ignored.)  If the TZ
environment variable is not set, the machine's default timezone is
used.  Setting TZ to the empty string or 0 causes timestamps to be
displayed in UTC.  For system information on the TZ variable, see
B<tzset(3)> or B<environ(7)>.  (To determine if SiLK was built with
support for the local timezone, check the C<Timezone support> value in
the output of B<rwuniq --version>.)

=item SILK_PLUGIN_DEBUG

When set to 1, B<rwuniq> prints status messages to the standard error
as it attempts to find and open each of its plug-ins.  In addition,
when an attempt to register a field fails, B<rwuniq> prints a message
specifying the additional function(s) that must be defined to register
the field in B<rwuniq>.  Be aware that the output can be rather
verbose.

=item SILK_TEMPFILE_DEBUG

When set to 1, B<rwuniq> prints debugging messages to the standard
error as it creates, re-opens, and removes temporary files.

=item SILK_UNIQUE_DEBUG

When set to 1, the binning engine used by B<rwuniq> prints debugging
messages to the standard error.

=back

=head1 FILES

=over 4

=item F<${SILK_ADDRESS_TYPES}>

=item F<${SILK_PATH}/share/silk/address_types.pmap>

=item F<${SILK_PATH}/share/address_types.pmap>

=item F<@prefix@/share/silk/address_types.pmap>

=item F<@prefix@/share/address_types.pmap>

Possible locations for the address types mapping file required by the
sType and dType fields.

=item F<${SILK_CONFIG_FILE}>

=item F<${SILK_DATA_ROOTDIR}/silk.conf>

=item F<@SILK_DATA_ROOTDIR@/silk.conf>

=item F<${SILK_PATH}/share/silk/silk.conf>

=item F<${SILK_PATH}/share/silk.conf>

=item F<@prefix@/share/silk/silk.conf>

=item F<@prefix@/share/silk.conf>

Possible locations for the SiLK site configuration file which are
checked when the B<--site-config-file> switch is not provided.

=item F<${SILK_COUNTRY_CODES}>

=item F<${SILK_PATH}/share/silk/country_codes.pmap>

=item F<${SILK_PATH}/share/country_codes.pmap>

=item F<@prefix@/share/silk/country_codes.pmap>

=item F<@prefix@/share/country_codes.pmap>

Possible locations for the country code mapping file required by the
scc and dcc fields.

=item F<${SILK_PATH}/lib64/silk/>

=item F<${SILK_PATH}/lib64/>

=item F<${SILK_PATH}/lib/silk/>

=item F<${SILK_PATH}/lib/>

=item F<@prefix@/lib64/silk/>

=item F<@prefix@/lib64/>

=item F<@prefix@/lib/silk/>

=item F<@prefix@/lib/>

Directories that B<rwuniq> checks when attempting to load a plug-in.

=item F<${SILK_TMPDIR}/>

=item F<${TMPDIR}/>

=item F</tmp/>

Directory in which to create temporary files.

=back

=head1 NOTES

If multiple thresholds are given (e.g., C<--threshold=bytes=80
--threshold=flows=2>), the values must meet all thresholds before the
record is printed.  For example, if a given key saw a single 100-byte
flow, the entry would not printed given the switches above.

B<rwuniq> functionally replaces the combination of

 rwcut | sort | uniq -c

To get a list of unique IP addresses in a data set without the
counting or threshold abilities of B<rwuniq>, consider using the IPset
tools B<rwset(1)> and B<rwsetcat(1)> for improved performance:

 rwset --sip-set=stdout | rwsetcat --print-ips

For situations where the key and value are each a single field, the
Bag tools (B<rwbag(1)>, B<rwbagcat(1)>) often provide better
performance, especially when the key length is one or two bytes:

 rwbag --bag-file=sport,bytes,stdout | rwbagcat

To create a binary file that contains B<rwuniq>-like output, use
B<rwaggbag(1)> or B<rwaggbagbuild(1)>.  The content of these files may
be printed with B<rwaggbagcat(1)>.

B<rwgroup(1)> works similarly to B<rwuniq>, except the data remains in
the form of SiLK Flow records, and the next-hop-IP field is modified
to denote the records that form a bin.

B<rwstats(1)> can do the same binning as B<rwuniq>, and then sort the
data by an aggregate field.

When the B<--bin-time> switch is given and the three time fields
(starting-time (C<sTime>), ending-time (C<eTime>), and duration
(C<duration>)) are present in the key, the duration field's value will be
modified to be the difference between the ending and starting times.

When the three time-related key fields (C<sTime>,C<duration>,C<eTime>) are
all in use, B<rwuniq> will ignore the final time field when binning
the records, but the field will appear in the output.  Due to
truncation of the milliseconds values, B<rwuniq> will print a
different number of rows depending on the order in which those three
values appear in the B<--fields> switch.

B<rwuniq> supports counting distinct source and/or destination IPs.
To see the number of distinct sources for each 10 minute bin, run:

 rwuniq --fields=stime --values=distinct:sip --bin-time=600 --sort-output

When computing distinct counts over a field, the field may not be part
of the key; that is, you cannot have C<--fields=sip
--values=sip-distinct>.

Using the B<--presorted-input> switch sometimes introduces more issues
than it solves, and B<--presorted-input> is less necessary now that
B<rwuniq> can use temporary files while processing input.

When computing distinct IP counts, B<rwuniq> will typically run faster
if you do I<not> use the B<--presorted-input> switch, even if the data
was previously sorted.

When using the B<--presorted-input> switch, it is highly recommended
that you use no more than one time-related key field (C<sTime>,
C<duration>, C<eTime>) in the B<--fields> switch and that the time-related
key appear last in B<--fields>.  The issue is caused by B<rwsort>
considering the millisecond values on the times when sorting, while
B<rwuniq> truncates the millisecond value.  The result may be unsorted
output and multiple rows in the output that have the same values for
the key fields:

 $ rwsort --fields=stime,duration data.rw       \
   | rwuniq --fields=stime,dur --presorted
               sTime|durat|   Records|
 ...
 2009/02/12T00:00:57|    0|         2|
 2009/02/12T00:00:57|   29|         2|
 2009/02/12T00:00:57|    0|         2|
 2009/02/12T00:00:57|   13|         2|
 ...

B<rwuniq>'s strength is its ability to build arbitrary keys and
aggregate fields.  For a key of a single IP address, see
B<rwaddrcount(1)> and B<rwbag(1)>; for a key made up of a single CIDR
block (/8, /16, /24 only), a single port, or a single protocol, use
B<rwtotal(1)> or B<rwbag(1)>.

As of SiLK 3.17.0, fields that are specified with the legacy
thresholding switches (e.g., B<--bytes>) and not with B<--values> are
printed in the order in which those switches appear.  Previously, the
order was always bytes, packets, flows, stime, etime, sip-distinct,
dip-distinct.

=head1 SEE ALSO

B<rwfilter(1)>, B<rwbag(1)>, B<rwbagcat(1)>, B<rwaggbag(1)>,
B<rwaggbagbuild(1)>, B<rwaggbagcat(1)>, B<rwcut(1)>, B<rwset(1)>,
B<rwsetcat(1)>, B<rwaddrcount(1)>, B<rwgroup(1)>, B<rwstats(1)>,
B<rwnetmask(1)>, B<rwsort(1)>, B<rwtotal(1)>, B<rwcount(1)>,
B<rwpmapbuild(1)>, B<addrtype(3)>, B<ccfilter(3)>,
B<int-ext-fields(3)>, B<pmapfilter(3)>, B<pysilk(3)>,
B<silkpython(3)>, B<silk-plugin(3)>, B<sensor.conf(5)>,
B<rwflowpack(8)>, B<silk(7)>, B<yaf(1)>, B<dlopen(3)>, B<tzset(3)>,
B<environ(7)>

=cut

$SiLK: rwuniq.pod 861b66f000c2 2019-09-24 22:01:14Z mthomas $

Local Variables:
mode:text
indent-tabs-mode:nil
End: