• Home
  • History
  • Annotate
Name Date Size #Lines LOC

..03-May-2022-

Flow_Working/H07-Nov-2014-

logs/H07-Nov-2014-

tools/H03-May-2022-2,8892,043

FV.cgiH A D03-May-202210.4 KiB242166

FV_Relay.cgiH A D03-May-20222 KiB4229

FlowGrapher.cgiH A D03-May-20227 KiB190111

FlowGrapher_Analyze.cgiH A D03-May-202252.7 KiB1,165928

FlowGrapher_ColorsH A D07-Nov-20145 KiB154153

FlowGrapher_Main.cgiH A D03-May-202285.3 KiB2,0891,587

FlowGrapher_PortsH A D07-Nov-201459.7 KiB1,4481,447

FlowGrapher_Replay.cgiH A D03-May-202228.5 KiB675503

FlowGrapher_Sort.cgiH A D03-May-202221.9 KiB511386

FlowMonitor.cgiH A D03-May-20227.3 KiB190110

FlowMonitor_CollectorH A D03-May-202252.4 KiB1,219922

FlowMonitor_Dashboard.cgiH A D03-May-202219.8 KiB541399

FlowMonitor_Display.cgiH A D03-May-20225.9 KiB154102

FlowMonitor_DisplayPublic.cgiH A D03-May-20227.1 KiB177126

FlowMonitor_Dumper.cgiH A D03-May-20229.2 KiB247160

FlowMonitor_GrapherH A D03-May-202227.9 KiB632459

FlowMonitor_Group.cgiH A D03-May-202239.6 KiB1,089888

FlowMonitor_Main.cgiH A D03-May-202236.7 KiB813580

FlowMonitor_Management.cgiH A D03-May-202246.9 KiB1,070846

FlowMonitor_RecreateH A D03-May-202233.2 KiB738536

FlowMonitor_Replay.cgiH A D03-May-20224.7 KiB13282

FlowMonitor_ThumbnailH A D03-May-202215.5 KiB401301

FlowViewer.cgiH A D03-May-20227.3 KiB196113

FlowViewer.cssH A D07-Nov-201417.5 KiB923913

FlowViewer_CleanASCacheH A D03-May-20222.4 KiB7836

FlowViewer_CleanFilesH A D03-May-20222.8 KiB8633

FlowViewer_CleanHostCacheH A D03-May-20223.1 KiB9447

FlowViewer_CleanSiLKH A D03-May-20226 KiB191124

FlowViewer_Configuration.pmH A D03-May-202213 KiB284171

FlowViewer_Main.cgiH A D03-May-202274.6 KiB1,7011,325

FlowViewer_Replay.cgiH A D03-May-202236.9 KiB842664

FlowViewer_Save.cgiH A D03-May-202215.7 KiB450346

FlowViewer_SaveManage.cgiH A D03-May-20229.5 KiB241172

FlowViewer_Sort.cgiH A D03-May-202224.6 KiB569443

FlowViewer_UI.pmH A D03-May-2022101.9 KiB2,1781,857

FlowViewer_Utilities.pmH A D03-May-202294.6 KiB2,6752,097

NamedInterfaces_DevicesH A D07-Nov-2014897 4337

NamedInterfaces_ExportersH A D07-Nov-2014455 1915

READMEH A D07-Nov-201451.9 KiB1,0771,076

dscan.suppress.dstH A D07-Nov-20140

dscan.suppress.srcH A D07-Nov-20140

flowmonitor_grapher_recentH A D07-Nov-201419 KiB492358

README

1# README (this file) FlowViewer V4.5  Date: 11/02/2014
2#
3# FlowViewer is a set of three tools (FlowViewer, FlowGrapher,
4# FlowMonitor) that create text reports, graph reports, and
5# long-term monitor reports from flow-tools and SiLK captured
6# and stored netflow data. FlowViewer can run with both flow-tools
7# and SiLK simultaneously. Flow-tools can handle up to v7; SilK
8# can handle v5, v9, and IPFIX. The User's Guide is very helpful.
9#
10# Software Dependencies:
11#
12#   flow-tools  http://code.google.com/p/flow-tools (If collecting v5 only)
13#   SiLK        http://tools.netsa.cert.org/silk    (If collecting IPFIX)
14#   libfixbuf   http://tools.netsa.cert.org/silk    (If collecting IPFIX)
15#   gd          http://www.libgd.org/Downloads
16#   GD          http://search.cpan.org/~lds/GD-2.30
17#   GD::Graph   http://search.cpan.org/~mverb/GDGraph-1.43
18#   GD::Text    http://search.cpan.org/~mverb/GDTextUtil-0.86/Text
19#   RRDtool     http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/pub
20#
21# Quick Upgrade
22#
23#   0. If using SiLK, must upgrade to v3.8.0 or newer
24#   1. Untar the package into a new cgi-bin subdirectory
25#   2. Configure FlowViewer_Configuration.pm variables to your environment
26#      and create all necessary directories with proper permissions
27#   3. Replace old logos with new buttons (will be done automatically)
28#   4. Copy FlowViewer.css, FlowViewer.pdf to $reports_directory
29#   5. Configure FlowViewer_Configuration.pm to point to existing
30#      FlowMonitor_Filter and FlowMonitor_RRDtool directories
31#      [As of v4.5 you can rename these directories to:
32#      FlowMonitor_Filter and FlowMonitor_RRDtool or change the
33#      configuration variables to point to the old directories.]
34#   6. Configure new FlowViewer_Configuration.pm
35#   7. Stop old FlowMonitor_Collector and FlowMonitor_Grapher
36#      [Upgrading to v4.5 you will be stopping FlowMonitor_Collector
37#      and FlowMonitor_Grapher.]
38#   8. Start new FlowMonitor_Collector and FlowMonitor_Grapher
39#   9. Copy NamedInterfaces_Devices, names file, user logo to new directory
40#  10. (If upgrading from pre v4.0) Run convert_pre40_filters against existing filters
41#      (ie FlowViewer_SavedFilters)
42#  11. Use included 'User Relay' scripts if desired (recommended - see below)
43#
44# Quick Install
45#
46#   1. Untar into cgi-bin subdirectory
47#
48#   For netflow v5 and older (option):
49#
50#   2. Download, install, configure flow-tools
51#
52#   For IPFIX (e.g., v9 - also handles v5):
53#
54#   3. Download, install, configure SiLK (v3.8.0 or newer) and libfixbuf
55#
56#   For sflow
57#
58#   4. From SiLK FAQ:
59#
60#      "Support for sFlow v5 is available as of SiLK 3.9.0 when you configure
61#      and build SiLK to use v1.6.0 or later of the libfixbuf library."
62#
63#   For FlowViewer
64#
65#   5. Configure FlowViewer_Configuration.pm variables as necessary
66#   6. Create all necessary directories with proper permissions
67#   7. Copy FlowViewer.css, FlowViewer.pdf to $reports_directory
68#   8. Point browser to FV.cgi
69#
70#   For FlowGrapher
71#
72#   9. Install gd (C), GD (Perl), GD::Graph (Perl) GD::Text (Perl)
73#  10. Configure FlowViewer_Configuration.pm variables as necessary
74#  11. Point browser to FV.cgi
75#
76#   For FlowMonitor
77#
78#  12. Install RRDtool (at least version 1.4)
79#  13. Create FlowMonitor_Filter and FlowMonitor_RRDtool directories
80#  14. Configure FlowViewer_Configuration.pm variables as necessary
81#  15. Start FlowMonitor_Collector, FlowMonitor_Grapher in background
82#  16. Point browser to FV.cgi
83#
84#   For all FlowViewer tools
85#
86#  17. Review all FlowViewer directories and files for proper permissions
87#
88# Version 4.5 Release Notes
89#
90#  Version 4.5 resolves an unfortunate name clash in commercial space and
91#  renames FlowTracker to FlowMonitor. The situation where SiLK data is
92#  saved in UTC (GMT) time, but the system is left in local time has been
93#  fixed (thanks to Kees Leune.) A new configuration variable
94#  "$silk_compiled_localtime" has been added for the environment where SiLK
95#  has been comipled with the --enable-localtime switch. FlowGrapher_Analyze
96#  has been fixed to handle hyper-links to IPv6 hosts properly. SiLK IPsets
97#  can now be input through the various tool menus. A problem with
98#  multi-word Dashboards and Group creation has been fixed. Corrected
99#  flows/second initiated calculation. Added the ability to bypass the
100#  printing of pulldowns on the bottom service bar. Fixed an error with
101#  filtering on port equal to '0'. Fixed 'Len' field output for some
102#  FlowGrapher reports. New parameter: $ipfix_default_device allows IPFIX
103#  users to pre-select a primary device (e.g., using one sensor only.)
104#  Extended pie-charts to some Printed reports. A new parameter
105#  $site_config_file is added to make it easier to accomodate various
106#  SiLK stored data file structures.
107#
108#  New FlowViewer_Configuration.pm parameters in v4.5:
109#
110#  $silk_compiled_localtime   - "Y" if SiLK compiled with local timezone
111#  $ipset_directory           - Directory where IPsets can be found
112#  $use_bottom_pulldowns      - Will exclude pulldowns on bottom of UI
113#  $ipfix_default_device      - Controls the default in device_name pulldown
114#  $sensor_config_file        - Changed from $sensor_config_directory
115#  $site_config_file          - Left blank (= "";) will look in rootdir
116#
117#  Note: the rename of FlowTracker to FlowMonitor includes default names
118#  for FlowMonitor related directories. The defaults that will previal if
119#  no changes are made are:
120#
121#  $monitor_directory       = "/var/www/html/FlowMonitor";
122#  $monitor_short           = "/FlowMonitor";
123#  $filter_directory        = " ... /FlowMonitor_Files/FlowMonitor_Filters";
124#  $rrdtool_directory       = " ... /FlowMonitor_Files/FlowMonitor_RRDtool";
125#
126#  For users who are upgrading, these can be revised back to 'FlowTracker'
127#  (or whatever) with no problem. The alternative is to simply rename the
128#  existing directories.
129#
130# Version 4.4 Release Notes
131#
132#  Version 4.4 introduces two new significant capabilities; multiple
133#  dashboards and FlowGrapher Analysis. The user can now set up
134#  more than one active dashboard with links on every page to get
135#  to each dashboard. Uses include multiple networks, data centers,
136#  or multiple users. The new version also permits users to quickly
137#  de-aggregate FlowGraphs into the largest (3 to 10) source or
138#  destination IP or Port contributors. This is particularly useful in
139#  analyzing peaks in graphs based on "Flows" for DDOS and the like.
140#  The multiple dashboard changes touched most of the scripts. There
141#  is one new script, FlowGrapher_Analyze, and the script
142#  FlowMonitor_AltDashboard is deprecated. The code was cleansed of
143#  confusing 'prorated (_P)' distinctions. Version 4.4 also greatly
144#  improves the user's ability to interface with various SiLK
145#  configurations other than the 'flow-tools-like' one suggested in
146#  earlier documentation. Note: SiLK users must upgrade SiLK to
147#  version 3.8.0 or newer.
148#
149# Version 4.3 Release Notes
150#
151#  Version 4.3 introduces a new FlowViewer report called "Detect
152#  Scanning" which employs scanning detection software in both
153#  flow-tools (flow-dscan) and SiLK (rwscan.) The output from each
154#  report includes links to the individual scanner sources detected.
155#  The link will create a FlowGrapher report for that host. Modified
156#  field checks to permit 32 bit AS entries. However, this is for the
157#  future when SiLK begins to handle the fields. Thanks Veasna Long.
158#
159# Version 4.2.2 Release Notes
160#
161#  Version 4.2.2 is a quick emergency fix with limited changes. An
162#  array (@temp_ports) in FlowViewer_Utilities was not initialized
163#  and was acumulating contents over many FlowMonitor_Collector runs.
164#  This was causing the whole FlowMonitor_Collector run to slow down
165#  gradually over time as this array would have to get sorted for each
166#  SiLK FlowMonitor (this caused no problems for flow-tools only
167#  users.)
168#
169# Version 4.2.1 Release Notes
170#
171#  Version 4.2.1 extends the use of the recently discovered flow-report
172#  option "linear-interpolated-flows-octets-packets" to FlowMonitor
173#  processing (see User's Guide for further discussion.) This provides
174#  significant speed-ups for FlowMonitor_Recreate (above 75%) and
175#  FlowMonitor_Collector (which will now permit many more FlowMonitors.)
176#  The FlowMonitor_Recreate processing for SiLK was remarkably poorly
177#  implemented and this has been corrected introducing unspeakable
178#  performance gains there as well. The same use of SiLK prefiltering
179#  that was introduced for FlowMonitor in v4.2 has been applied to
180#  FlowGrapher for a speed-up there. A new capability is added to work
181#  with different international date formats. The distribution includes a
182#  new capability, FlowViewer_CleanSiLK, to monitor and adjust diskspace
183#  used by IPFIX devices addressing the SiLK deficiency of not having the
184#  valuable flow-capture feature (-E) of active diskspace usage control.
185#  Made the use of prefiltered files or CONCATs non-optional and fixed
186#  SiLK processing of flows and packets for FlowMonitor_Collector. Modest
187#  changes have been made to FlowViewer_CleanFiles, FV.cgi and
188#  flowmonitor_grapher_nonlazy. An error causing packets to be monitored as
189#  flows in some cases was fixed. A new tool, flowmonitor_grapher_recent
190#  will re-graph (nonlazy) only recently created FlowMonitors.
191#
192# Version 4.2 Release Notes
193#
194#  Version 4.2 incorporates the equivalent of "use existing concatenations"
195#  for SiLK based FlowMonitors. FlowViewer takes advantage of previously
196#  concatenated flow-tools files during FlowMonitor_Collector processing
197#  to speed up the whole run. Now it does this for SiLK files as well by
198#  performing rwfilter "INPUT" filtering only once for all FlowMonitors
199#  that are based on the same sensor/class combination. This is a pretty
200#  significant speed-up and will permit the user to have many more
201#  FlowMonitors. For example, our implementation, a combination of
202#  flow-tools and SiLK based data, now processes 250 FlowMonitors in 35
203#  seconds (prior to the new version this was taking 50 seconds.) The new
204#  version also corrects processing of exporters [M. Donnelly]. Excluded
205#  fields (e.g., protocols=-17) were being accepted for SiLK FlowMonitors
206#  despite SiLK not being able to handle them; this was fixed. The
207#  analyze_netflow_packets tool has been fixed for IPv6 addresses. Four new
208#  date conversion utilities have been added to the 'tools' subdirectory.
209#  FlowMonitor_Collector was modified in version 4.0 to be able to create
210#  FlowMonitors for flows and packets (as well as bits.) This was
211#  inadvertantly dropped in version 4.1 and is restored in this update
212#  to version 4.2 [7/31].
213#
214# Version 4.1 Release Notes
215#
216#  Version 4.1 includes a new FlowGrapher capability that creates reports
217#  3 to 4 times faster than previously. The detail lines are a little bit
218#  different. The previous capability is retained offering the user a choice
219#  from the input form interface. The new FlowGrapher report type is
220#  "Aggregated". It makes use of a heretofore missed flow-tools capability
221#  known as the flow-report "linear-interpolated-flows-octets-packets" option
222#  which aggregates flows, octets, or packets into time buckets. This moves
223#  that processing into the compiled "C" code of flow-tools. New FlowMonitor
224#  capabilities are added to monitor flows or packets as well as the previously
225#  available octets. Version 4.1 introduces the ability to maintain different
226#  dashboards for different users (please see the User's Guide for how to do
227#  this. The new version includes a new FV_Relay.cgi script. The new version
228#  fixes a flaw in FlowMonitor_Collector that erroneously monitored protocols,
229#  tcp_flags, and tos_fields when using SiLK (thanks C. Spitzlay.) It also
230#  includes some small fixes like making directory creation a little easier,
231#  fixes removing (and adding) Trackings from the Dashboard and removes some
232#  minor extraneous formatting. Fixed problem with FlowMonitor_Collector
233#  processing of SiLK interface filtering [07/09/13]. Fixed initialization
234#  of @ipfix_devices in FlowViewer_Configuration.pm [Thanks M. Donnelly.]
235#
236# Version 4.0 Release Notes
237#
238#  Version 4.0 is a major upgrade that enables FlowViewer to handle IPFIX
239#  netflow data (i.e., v9, etc.) The User Interface has been completely redone
240#  and now features a Dashboard. Aside from the new collector interface and user
241#  interface, version 4.0 introduces some new capabilities:
242#
243#   1. FlowViewer report sorting by column header
244#   2. Dashboard of thumbnail versions of selected FlowMonitor graphs
245#   3. Ability to 'recreate' FlowMonitors, starting at a time specified in the
246#      past
247#
248#  The distribution manifest has changed significantly.
249#
250#  Preserved Scripts, Files, and Tools:
251#
252#   FlowViewer.cgi                Modified for new user interface.
253#   FlowViewer_Main.cgi           Modified for new user interface.
254#   FlowViewer_Relay.cgi          No change.
255#   FlowViewer_Save.cgi           Significant modification.
256#   FlowGrapher.cgi               Modified for new user interface.
257#   FlowGrapher_Main.cgi          Modified for new interface.
258#   FlowGrapher_Colors            No change.
259#   FlowGrapher_Relay.cgi         No change.
260#   FlowGrapher_Sort.cgi          Significant modification.
261#   FlowMonitor.cgi               Modified for new user interface.
262#   FlowMonitor_Collector         Modified to process stored SiLK data.
263#   FlowMonitor_Grapher           Modified to update Thumbnails.
264#   FlowMonitor_Group             Modified for new user interface.
265#   FlowMonitor_Dumper            Modified for new user interface.
266#   FlowMonitor_Relay.cgi         No change.
267#   FlowViewer_CleanASCache       No change.
268#   FlowViewer_CleanFiles         Minor changes.
269#   FlowViewer_CleanHostCache     No change.
270#   FlowViewer_Configuration.pm   Modifications for SiLK and user interface.
271#   FlowViewer_Utilities.pm       Removed filter output processing.
272#   NamedInterfaces_Devices       No change.
273#   NamedInterfaces_Exporters     No change.
274#   flowcapture_restart           No change.
275#   flow-capture-table.conf       No change.
276#   flowmonitor_restart           No change.
277#   performance_check             Parse FlowMonitor logs and report performance
278#   rsync_flows                   Rsync all of raw flow data to backup host
279#   rsync_monitors               Rsync all of Tracking data to backup host
280#
281#  New Scripts, Files, and Tools
282#
283#   FlowViewer_Replay.cgi         Presents saved FlowViewer reports
284#   FlowViewer_SaveManage.cgi     Manages saved reports
285#   FlowViewer_Sort.cgi           Sorts FlowViewer reports
286#   FlowViewer_UI.cgi             Utilities for creating user interface
287#   FlowGrapher_Replay.cgi        Presents saved FlowGrapher reports
288#   FlowMonitor_Dashboard.cgi     Manages the Dashboard contents
289#   FlowMonitor_Display.cgi       Presents a FlowMonitor
290#   FlowMonitor_DisplayPublic.cgi Presents a FlowMonitor from Public list
291#   FlowMonitor_Management.cgi    Manages FlowMonitors (e.g., remove, etc.)
292#   FlowMonitor_Recreate          Background process to recreate FlowMonitors
293#   FlowMonitor_Thumbnail         Invoked to create a Thumbnail FlowMonitor
294#   FlowViewer.css                FlowViewer cascading style sheet
295#   FV_button.png                 New button link to FlowViewer from front page
296#   FG_button.png                 New button link to FlowGrapher from front page
297#   FM_button.png                 New button link to FlowMonitor from front page
298#   convert_pre40_filters         Converts old saved filters (pre version 4.0).
299#   flowmonitor_archive_restore   Restores archived FlowMonitors gone astray
300#   flowmonitor_grapher_nonlazy   Forces a re-graphing of all FlowMonitor graphs
301#   resize_rrdtools               Extends RRDtools created prior to 3-Year graph
302#   rwflowpack_start              One-line script starts SiLK collector
303#   analyze_netflow_packets       Script analyzes TCPDUMP captured netflow data
304#
305#  Removed Scripts and Files
306#
307#   FlowViewer_SavedFilters       File kept saved filters
308#
309#  General Notes:
310#
311#   This is a major upgrade of FlowViewer. The upgrade preserves this
312#   open-source option for netflow analysis in the age of IPFIX. The user
313#   is urged to read through the User's Guide for a better understanding
314#   of installation and configuration.
315#
316#   Those who upgrade can preserve all previous filters and reports easily.
317#   Saved reports are automatically available in the new version. The only
318#   manual change requires users to run the 'convert_pre40_filters' script
319#   from the command line to move saved filters into the new format. Example:
320#
321#   host>convert_pre40_filters .../FlowViewer_3.4/FlowViewer_SavedFilters
322#
323#   With Respect to SiLK: The SiLK tool suite, developed by the NetSA group
324#   at Carnegie Mellon, is excellent software with equally excellent
325#   documentation. Version 3.0 of SiLK together with libfixbuf v1.1.0 are their
326#   entree into IPFIX/v9 netflow capture and analysis supporting IPv6. Initially
327#   they have chosen to limit the number of IPFIX Information Elements (IE) that
328#   the SiLK software will process. They have chosen a set that matches what
329#   flow-tools has provided with the addition of IPv6 data, but sadly with the
330#   exception of autonomous system (AS) elements. I have requested that they add
331#   the AS Elements, but we'll see. They have mentioned a future overhaul
332#   (beyond v3.0) to handle the entire IE space through user configuration. As
333#   of Spring 2013, SiLK v3.x is not fully through the process required to make
334#   the software open-source to the general public but they are proceeding with
335#   getting the approval. It is currently freely available to US Federal
336#   agencies.
337#
338#   The FlowViewer_Configuration.pm file has changed:
339#
340#    New parameters (configurable):
341#
342#     $dashboard_directory     = "/var/www/html/FlowViewer_Dashboard";
343#     $dashboard_short         = "/FlowViewer_Dashboard";
344#     $silk_data_directory     = "/data/flows";
345#     $silk_bin_directory      = "/usr/local/bin";
346#     $sensor_config_directory = "/data/flows";
347#     @ipfix_devices           = ("Router_v9_1","Router_v9_2","Test_6509_v9");
348#     $sip_prefix_length       = "16";
349#     $dip_prefix_length       = "16";
350#     $silk_all_only           = "N";
351#     $left_title              = "Any Title You Like";
352#     $left_title_link         = "http://abc.com/";
353#     $right_title             = "Any Second Title You Like";
354#     $right_title_link        = "http://abc.com/";
355#     $recreate_cat_length     = 1*(60*60); # Time length of concatenated file
356#     $thumbnail_width         = 250; # probably should leave this alone
357#     $thumbnail_height        = 80;  # probably should leave this alone
358#     $filename_color          = "#CF7C29";
359#     $dig_forward             = "/usr/bin/dig +time=1 +tries=1 ";
360#     $default_identifier      = "DNS"; # "IP" for addresses; "DNS" for names
361#
362#    Removed Parameters
363#
364#     $bg_color                = "#FFFFFF";
365#     $text_color              = "#000000";
366#     $link_color              = "#000000";
367#     $vlink_color             = "#BF294D";
368#     $monitors_title          = "Your Company Name";
369#     $user_logo               = "Generic_Logo.jpg";
370#     $user_hyperlink          = "http://www.yourcompany.com/";
371#
372#   With respect to the "Relay" scripts, many of you may already have resolved
373#   this issue by setting up a generic 'FlowViewer' directory and simply
374#   re-linking it to the new version's directory. I've been told this is proper
375#   :-). It certainly makes good sense. Otherwise the "Relay" approach is best
376#   explained below in Version 3.4 Release Notes.
377#
378# Version 3.4 Release Notes
379#
380#  Update - 8/17/2011 - Fixed FlowViewer bug when requesting time periods just
381#  shy of midnight. This had already been fixed in FlowGrapher. Modifications
382#  were made to FlowViewer_Main.cgi.
383#
384#  Update - 5/20/2011 - Modifications have been made to FlowGrapher_Main.cgi to
385#  fix a problem caused by the new speed-up processing. The speed-up was not
386#  accounting for Daylight Savings considerations.
387#
388#  It's been awhile, so version 3.4 will fix a myriad of little problems which
389#  I mostly can't remember. The primary new capabilities include:
390#
391#   1. In most cases, the user may now switch the device without losing entered
392#      filter criteria
393#   2. The different tool logos now provide a link to the Saved Reports page
394#   3. Users can now provide a meaningful name for saved FlowViewer and
395#      FlowGrapher reports
396#   4. Fixes to an end-of-year problem have resulted in a 8% speed up of
397#      FlowGrapher in general
398#   5. Users can select to limit FlowGrapher stats to no-zero data points,
399#      if desired
400#   6. Fixed problems with sorting
401#   7. Corrected the graphing by 'flows' (was graphing 'flags' :-)
402#   8. Can now provide up to 20 source or destination IP address/address ranges
403#   9. Can now exclude specified IP addresses from a larger included address
404#      range
405#
406#  New Scripts and Files:
407#
408#   FlowGrapherM.png            New logo link points to Saved reports web page
409#   FlowGrapherS.png            Revised logo link for naming of Saved Reports
410#   FlowViewerM.png             New logo link points to Saved reports web page
411#   FlowViewerS.png             Revised logo link for naming of Saved Reports
412#   FlowMonitorM.png            New logo link points to Saved reports web page
413#   flowcapture_restart         Renamed script for restarting flow-captures
414#   flowmonitor_restart         New script for re-starting FlowMonitor_Collector
415#
416#  General Notes:
417#
418#   Remember to copy into the new directory (e.g.,
419#   /usr/lib/cgi-bin/FlowViewer_3.4) user logos, names file, as_names,
420#   NamedInterfaces_Devices, NamedInterface_Exporters, FlowViewer_SavedFilters,
421#   etc., from the old cgi-bin directory.
422#
423#   The simplest way to transition to the new version is to leave all
424#   FlowViewer_Configuration.pm settings alone except:
425#
426#    $reports_directory       = "/var/www/FlowViewer_3.4";
427#    $reports_short           = "/FlowViewer_3.4";
428#    $graphs_directory        = "/var/www/FlowGrapher_3.4";
429#    $graphs_short            = "/FlowGrapher_3.4";
430#    $monitor_directory       = "/var/www/FlowMonitor_3.4";
431#    $monitor_short           = "/FlowMonitor_3.4";
432#    $cgi_bin_directory       = "/usr/lib/cgi-bin/FlowViewer_3.4";
433#    $cgi_bin_short           = "/cgi-bin/FlowViewer_3.4";
434#    $work_directory          = "/usr/lib/cgi-bin/FlowViewer_3.4/Flow_Working";
435#    $names_directory         = "/usr/lib/cgi-bin/FlowViewer_3.4";
436#    $log_directory           = "/usr/lib/cgi-bin/FlowViewer_3.4"
437#
438#   The following can remain the same (or copy contents to the new directory):
439#
440#    $save_directory          = "/var/www/FlowViewer_Saves";
441#    $save_short              = "/FlowViewer_Saves";
442#    $filter_directory        = ".../FlowMonitor_Files/FlowMonitor_Filters";
443#    $rrdtool_directory       = ".../FlowMonitor_Files/FlowMonitor_RRDtool";
444#
445#   If this is an upgrade for you (e.g., from v3.3.1) I recommend using the
446#   FlowViewer_Relay.cgi, FlowGrapher_Relay.cgi, and the FlowMonitor_Relay.cgi
447#   scripts to alert users to the new version with links and a reminder to
448#   change their bookmarks. In each of the relay scripts tailor the following
449#   line to your environment (point to the new FlowViewer_Configuration.pm file):
450#
451#    require "/usr/lib/cgi-bin/FlowViewer_3.4/FlowViewer_Configuration.pm";
452#
453#   ... then, in your old cgi-bin directory (e.g., FlowViewer_3.3.1), copy the
454#   following:
455#
456#    cp FlowViewer_Relay.cgi FlowViewer.cgi
457#    cp FlowGrapher_Relay.cgi FlowGrapher.cgi
458#    cp FlowMonitor_Relay.cgi FlowMonitor.cgi
459#
460#   Now, when users go to their book-marked FlowViewer web page, they will be
461#   directed to the new one. FlowMonitor_Relay.cgi is particularly important
462#   this is an upgrade it copies over archived FlowMonitors which would be a
463#   bit tedious to copy by hand.
464#
465#   The rsync_flows and rsync_monitors scripts are useful for easily backing
466#   up all raw netflow data and FlowMonitor state information (Filters and
467#   RRDtool databases.) The FlowViewer_CleanFiles script is useful for deleting
468#   aging files that are not necessary anymore. I run it out of 'cron' once a
469#   day.
470#
471#   The performance_check script can be used from the command line to keep track
472#   of how well your implementation is performing. I run it against my
473#   Flowmonitor_Collector.log file to see how things are going. Here at the
474#   NASA Earth Observing System network I have over 200 FlowMonitors and
475#   they complete in an average of 44 seconds. FlowMonitor_Collector runs every
476#   five minutes and I watch for runs that take longer than five minutes.
477#   Even in those situations, however, FlowMonitor_Collector seems to continue
478#   on with no real visible effects.
479#
480# Version 3.3 Release Notes
481#
482# ### Version 3.3.1 fixes a FlowMonitor_Collector bug when using exporters
483#     Also fixes problem for users without devices at all. If you are not
484#     using any devices (or exporters) you will now have to set:
485#     $no_devices_or_exporters = "Y"; Fixes FlowGrapher sorting of host
486#     names. This version fixes the problem of links to Trackings embedded
487#     in Group graphs not lining up properly. Fixes problem with
488#     FlowMonitor_Grapher not printing out named interfaces. Fixes
489#     FlowGrapher graph and output to now have exporter name. Fixes
490#     problem with end-of-month graphs (missing days_in_month.)
491#
492# New Capabilities
493#
494#   1. Some devices will now have 'named interfaces' (thanks C. Kishimoto)
495#   2. The user can now save filters of interest and recall them later
496#   3. Data can now be analyzed by Exporter ID (in addition to device name)
497#   4. Users can now set thresholds on FlowMonitors, and be alerted
498#   5. Users can now sort FlowGrapher output based on column type
499#   6. FlowViewer now provides Pie Charts
500#   7. Capability added to apply a Sampling Multiplier to output
501#   8. FlowMonitors now have a '3 year' graph
502#   9. The user can now generate text listings of FlowMonitor output
503#  10. Filtering on next-hop has been added
504#  11. Logging has been made more flexible (e.g., less data)
505#  12. Preserve latest three notations (was keeping first three)
506#  13. Can now specify and display time-zones
507#  14. A hook has been provided for a User Logo with link out of FlowViewer
508#  15. New file cleanup scripts have been added
509#  16. Unit Conversion capability has been added (thanks C. Kishimoto)
510#  17. Can now graph Flows, Packets as well as Octets (thanks E.Lautenschlaeger)
511#  18. Improved AS name resolution (thanks S. Cardus)
512#  19. New saved_directory for storing saved Reports and Graphs.
513#
514# New Scripts and Files:
515#
516#  FlowGrapher_Sort.cgi        Sorts FlowGrapher Detail Lines by column
517#  FlowMonitor_Dumper.cgi      Invoked by link in Trackings, prints text values
518#  FlowViewer_Save.png         New logo with links for saving filters, reports
519#  Flowgrapher_Save.png        New logo with links for saving filters, reports
520#  FlowViewer_CleanASCache     Tool used to remove obsolete AS name resolutions
521#  FlowViewer_CleanFiles       Tool used to remove old intermediate files
522#  FlowViewer_CleanHostCache   Tool used to remove obsolete host name resolutions
523#  FlowViewer_Relay.cgi        Optional: points users to new version (see Notes)
524#  FlowGrapher_Relay.cgi       Optional: points users to new version (see Notes)
525#  FlowMonitor_Relay.cgi       Optional: points users to new version (see Notes)
526#  flowcap                     Optional start-up script for flow-tools and
527#                              FlowMonitor
528#
529#  NamedInterfaces_Devices     Holds interface names for SNMP device indices
530#  NamedInterfaces_Exporters   Holds interface names for SNMP index (exporters)
531#  FlowViewer_SavedFilters     Created during processing to hold saved filters
532#
533# Notes:
534#
535#  Many thanks to Carles Kishimoto, Eric Lautenschlaeger, and Sean Cardus for
536#  their ideas and code contributions. Thanks to Dario La Guardia for pointing
537#  out a graphing problem that turned out to be a rounding error in FlowGrapher.
538#  Credit to Peter Hoffswell for the idea of linking the tools.
539#
540#  There are no new software dependencies with FlowViewer version 3.3.1, however
541#  Named Interfaces now requires Javascript in the browser to operate.
542#
543#  If you are having trouble with creating Tracking Groups, you may have a
544#  problem with the installation of RRDs.pm. This needs to be placed in a
545#  library that Perl includes in it's @INC array. For a fix, see the FlowViewer
546#  FAQ on the web site.
547#
548#  Using the 'Relay' scripts (these are optional)
549#
550#   If you have other users and you would like to point them to the new version,
551#   copy the included 'Relay' scripts over the old FlowViewer.cgi,
552#   FlowGrapher.cgi and FlowMonitor.cgi scripts in the last version's directory.
553#
554#   For example:
555#
556#    In the old directory /htp/cgi-bin/FlowViewer_3.2:
557#
558#    mv FlowViewer_Relay.cgi  FlowViewer.cgi
559#    mv FlowGrapher_Relay.cgi FlowGrapher.cgi
560#    mv FlowMonitor_Relay.cgi FlowMonitor.cgi
561#
562#   Then, when the user goes to the old FlowViewer, he will be provided a link
563#   to the new FlowViewer, and asked to change his bookmarks.
564#
565#
566#  Setting up crontab file for cleaning FlowViewer files:
567#
568#   min   hr  dom  moy  dow    command
569#
570#     5   0    *    *    *     .../FlowViewer_3.4/FlowViewer_CleanFiles
571#                                > .../FlowViewer_3.4/cleanup.log
572#                                2 >> .../FlowViewer_3.4/cleanup.log
573#
574#  The file cleanup is controlled by parameters in FlowViewer_Configuration.pm:
575#
576#   $remove_workfiles_time   = 86400;
577#   $remove_graphfiles_time  = 7*86400;
578#   $remove_reportfiles_time = 7*86400;
579#
580#  Remember, whichever crontab account this is started from must have adequate
581#  permissions to remove files created by the web process owner (e.g., apache.)
582#
583# Documentation
584#
585# The FlowViewer User's Guide is available on the FlowViewer Website:
586#
587#   http://ensight.eos.nasa.gov/FlowViewer
588#
589# Dependencies
590#
591# - FlowGrapher requires the Perl GD and GD:Graph packages
592#   gd package. Thomas Boutrell's graphics package written in 'C'
593#   GD package: http://search.cpan.org/~lds/GD-2.30/
594#   GD::Graph package: http://search.cpan.org/~mverb/GDGraph-1.43/
595# - FlowViewer.cgi requires the GDBM or NDBM capability in Perl
596# - FlowMonitor requires RRDtool (at least version 1.2.12)
597#   RRDtool: http://oss.oetiker.ch/rrdtool
598#
599# Contents
600#
601# FlowViewer_Configuration.pm
602#
603# This file contains parameters that configure and control the
604# FlowViewer, FlowGrapher, and FlowMonitor environments. This package
605# should remain in the same directory that the CGI scripts are in.
606#
607# FlowViewer_Utilities.pm
608#
609# This file contains processing used by multiple programs (e.g., to
610# create the Report Parameters output for each tool, and other utilities
611# (e.g., 'epoch_to_date' which converts between typical date formats
612# and 'seconds since 1972') that are invoked by other scripts. This
613# package should be placed in the same directory as the CGI scripts.
614#
615# FlowViewer.cgi
616#
617# This script produces the web page which provides the user the form
618# for entering analysis selection criteria for FlowViewer. Version 3.0
619# reorganized the processing. FlowViewer.cgi is now the old
620# create_FlowViewer_webpage. This change permits the input date and time
621# to be updated with each invocation.
622#
623# FlowViewer_Main.cgi
624#
625# This script responds when the user completes the selection criteria
626# form and submits the 'Generate Report' command. The script creates a
627# flow-tools filter file based on the selection criteria. Based on the
628# input time period, the script concatenates the relevant flow-tools
629# data files for the selected device. The location of the flow-tools
630# raw data files is specified via the 'flow_data_directory' parameter.
631# The script then invokes the selected statistics/print report flow-tools
632# program and reformats the output into HTML. An option is available in
633# FlowViewer_Configuration to have this script use the NDBM capability
634# (for caching resolved host names) instead of the default GDBM
635# capability for users whose Perl distribution does not have GDBM.
636#
637# FlowGrapher.cgi
638#
639# This script produces the web page which provides the user the form
640# for entering analysis selection criteria for FlowGrapher. Version 3.0
641# reorganized the processing. FlowGrapher.cgi is now the old
642# create_FlowGrapher. This change permits the input date and time
643# to be updated with each invocation.
644#
645# FlowGrapher_Main.cgi
646#
647# This script responds when the user completes the FlowGrapher selection
648# criteria form and submits the 'Generate Graph' command. The script
649# creates intermediate processing files exactly like FlowViewer above.
650# The script then parses intermediate output, fills time buckets, and
651# generates a graphic image. Textual output accompanies the graph. An
652# option is available in FlowViewer_Configuration to have this script use
653# the NDBM capability (for caching resolved host names) instead of the
654# default GDBM capability for users whose Perl distribution does not have
655# GDBM.
656#
657# FlowGrapher_Sort.cgi
658#
659# This script is invoked when the user clicks on a column header for the
660# Detail Lines of a FlowGrapher report. The textual data on the page is
661# sorted and re-presented.
662#
663# FlowGrapher_Colors
664#
665# This file contains a translation between textual color names and their
666# RGB value counterparts.
667#
668# FlowMonitor.cgi
669#
670# This script produces the web page which provides the user the form
671# for entering analysis selection criteria for FlowMonitor. The script
672# also provides the user with the ability to review, revise, or remove
673# existing monitors. FlowMonitor was new in version 3.0.
674#
675# FlowMonitor_Main.cgi
676#
677# This script responds when the user completes the FlowMonitor selection
678# criteria form and submits the 'Establish Tracking' command. The script
679# responds to the users desire to create, remove, or revise a monitor.
680#
681# FlowMonitor_Group.cgi
682#
683# This script controls the building of groups from existing Individual
684# FlowMonitors. The user has the ability with FlowMonitor v3.2 to create
685# 'groups' from pre-defined Individual monitors. A Group Tracking has no
686# RRD database associated with it, but simply creates a multifaceted graph
687# from several existing monitors. The Group 'merges' the Individual graphs
688# onto a single graph.
689#
690# FlowMonitor_Dumper.cgi
691#
692# This script is invoked when the user clicks on a link within the
693# FlowMonitor graph labeled '[List values]'. The script dumps the
694# RRDtool contents onto a web page.
695#
696# FlowMonitor_Collector
697#
698# The script is started once by the user and placed in the 'background'.
699# The script will execute and then sleep for the duration of a five minute
700# period, essentially running every five minutes. For each existing monitor,
701# the script applies the associated filter to the flow data and extracts the
702# amount that occured during a 5-minute window approximately 30 miuntes
703# earlier. This is to permit long-running flows to have been exported and
704# available to the collector. The script then divides the total bits by
705# 300 seconds to get an average bits-per-second rate during the period.
706# The data point is then provided to RRDtool for storage. The script
707# should be started out of the cgi-bin directory.
708#
709# FlowMonitor_Grapher
710#
711# The script is started once by the user and placed in the 'background'.
712# The script will execute and then sleep for the duration of a five minute
713# period, essentially running every five minutes. The script runs the
714# RRDtool graph function for each existing monitor. Daily, weekly,
715# monthly, and yearly graphs are updated with the latest information. The
716# script creates an html page for each monitor that includes the filter
717# parameters and the four graphs. The script also creates an overall web
718# page ($monitor_webpage) that provides links to all active monitor pages.
719# The script should be started out of the cgi-bin directory.
720#
721# FlowViewerM.png
722#
723# The FlowViewerM logo with links. Leave this file in the 'cgi-bin_directory',
724# the FlowViewerM.cgi script will place a copy of the image in
725# 'html_directory'. This image contains mapped links to FlowGrapher and
726# FlowMonitor such that those input pages are pre-loaded with the filter
727# criteria from FlowViewer.
728#
729# FlowViewerS.png
730#
731# The FlowViewerS logo with links. Leave this file in the
732# 'cgi-bin_directory', the FlowViewerS.cgi script will place a copy of the
733# image in 'reports_directory'. This image contains mapped links to the other
734# tools as well as links for saving the filter used or the report generated.
735#
736# FlowGrapherM.png
737#
738# The FlowGrapherM logo with links. Leave this file in the 'cgi-bin_directory',
739# the FlowGrapherM.cgi script will place a copy of the image in
740# 'graphs_directory'. This image contains mapped links to FlowViewer and
741# FlowMonitor such that those input pages are pre-loaded with the filter
742# criteria from FlowGrapher.
743#
744# FlowGrapherS.png
745#
746# The FlowGrapherS logo with links. Leave this file in the
747# 'cgi-bin_directory', the FlowGrapherS.cgi script will place a copy of the
748# image in 'graphs_directory'. This image contains mapped links to the other
749# tools as well as links for saving the filter used or the report generated.
750#
751# FlowMonitorM.png
752#
753# The FlowMonitor logo with links. Leave this file in the 'cgi-bin_directory',
754# the FlowMonitorM.cgi script will place a copy of the image in
755# 'monitor_directory'. This image contains mapped links to FlowViewer and
756# FlowGrapher such that those input pages are pre-loaded with the filter
757# criteria from FlowMonitor.
758#
759# FlowViewer_Save.cgi
760#
761# This script moves temporary save files into a permanent residence
762# as defined by either the 'reports_directory' or 'graphs_directory'
763# environment variables.
764#
765# FlowViewer_CleanFiles
766#
767#  A utility for cleaning out temporary files that have been left
768#  over from debugging (e.g. $debug_files = 'Y'). Files older than
769#  the following configurable parameters are removed:
770#
771#   $remove_workfiles_time   = 86400;
772#   $remove_graphfiles_time  = 7*86400;
773#   $remove_reportfiles_time = 7*86400
774#
775#  See above for crontab settings for running this automatically.
776#
777# FlowViewer_CleanASCache
778#
779# A utility for cleaning out from the AS resolving cache ($as_file) a
780# resolved AS name that is no longer valid.
781#
782# FlowViewer_CleanHostCache
783#
784# A utility for cleaning out from the DNS resolving cache ($names_file)
785# a resolved host name that is no longer valid.
786#
787# FlowViewer_Relay.cgi, FlowGrapher_Relay.cgi, FlowMonitor_Relay.cgi
788#
789# Short scripts that refer users from version 3.3.1 to version 3.4. This
790# keeps you from having to notify users to go to a different web site.
791#
792# flowcapture_restart
793#
794# A shell script used for starting up and restarting flow-captures. Tailor
795# this for your environment.
796#
797# flowmonitor_restart
798#
799# A shell script used for starting up and restarting FlowMonitor_Collector
800# and FlowMonitor_Grapher. Tailor this for your environment.
801#
802# Generic_Logo.jpg
803#
804# This image is to be replaced by your own image that can point back to
805# anywhere (e.g., your overarching NMS system.)
806#
807# NamedInterfaces_Devices
808#
809# This file is used for SNMP index to named interface translation. This file
810# provides translation when you are saving data by individual devices. Examples
811# are provided.
812#
813# NamedInterfaces_Exporters
814#
815# This file is used for SNMP index to named interface translation. This file
816# provides translation when you are saving data into a single directory but
817# for (possibly) multiple devices differentiated by EXPORTER_ID. Examples are
818# provided.
819#
820# Configuration parameters
821#
822# The FlowViewer, FlowGrapher, and FlowMonitor scripts all use parameters
823# in the FlowViewer_Configuration.pm file to control the environment that
824# they run in. Here is a brief explanation of some of the relevant
825# parameters:
826#
827# $ENV(PATH) - modify as appropriate for your installation
828# $FlowViewer_server - IP address of server hosting this software
829# $FlowViewer_service - Either HTTP (port 80) or HTTPS (port 443)
830# $reports_directory - Directory to hold saved FlowViewer reports
831# $reports_short - Reports directory beginning from web server default
832# $graphs_directory - Directory to hold saved FlowGrapher reports
833# $graphs_short - Graphs directory beginning from web server default
834# $monitor_directory - Directory to hold FlowMonitor monitors
835# $monitor_short - Monitor directory beginning from web server default
836# $filter_directory - Directory in which to keep FlowMonitor filter files
837# $rrdtool_directory - Directory in which to keep FlowMonitor RRDtool files
838# $cgi_bin_directory - Directory which holds cgi scripts
839# $cgi_bin_short - cgi-bin directory beginning from web server default
840# $flow_data_directory - Directory that holds all flow-tools data files
841# $exporter_directory - Directory where netflow stored for multiple exporters
842# $flow_bin_directory - Directory where all flow-tools reside
843# $rrdtool_bin_directory - Location of RRDtool programs
844# $work_directory - Directory to store intermediate files
845# $names_directory - Directory to save permanent 'names' file
846# $flow_capture_interval - Interval beyond end point to capture all flows
847# $flow_file_length - Length (in seconds) of each of your flow files
848# $devices - List of device names exporting netflow (see #4 below)
849# $no_devices_or_exporters - Set to "Y" if you have no devices and no exporters
850# $N - Used to control directory organization (see #5 below)
851# $dig - Location of DNS utility 'dig' (set to nslookup if required)
852# $actives_webpage - Name of HTML file which will list your Trackings
853# $monitors_title - Title for HTML page which lists Trackings
854# $user_logo - Filename of image used for your logo
855# $user_hyperlink - Link associated with $user_logo
856# $use_even_hours - Will start default time periods at the top of the hour
857# $use_NDBM - Some Perls don't have GDBM (default), but do have NDBM
858# $start_offset - Offset from current time for beginning pre-loaded time period
859# $end_offset - Offset from current time for end of pre-loaded time period
860# $flow_capture_interval - Minutes beyond end period for collecting all flows
861# $flow_file_length - Size (minutes) of each flow-tools flow file (default = 15)
862# $labels_in_titles - Whether to print FlowMonitor title in the graph itself
863# $debug_files - If Yes, will not remove intermediate files
864# $collection_offset - Seconds into past to begin collection period
865# $collection_period - Period to examine for FlowMonitor (keep at 5 minutes!)
866# $use_existing_concats - DEPRECATED. Re-use concatenations (much faster)
867# $rrd_dir_perms - (And others) UNIX directory or file permissions
868#
869# The rest of this file contains basic parameters such as colors, etc. Each
870# parameter is dicussed in more detail in the User's Guide.
871#
872# Additional Considerations
873#
874# 1. Directory permissions for the subdirectories created for the
875# 'htdocs', 'work', 'names', 'cgi-bin' (e.g., FlowMonitor_Filter,
876# FlowMonitor_RRDtool) directories must permit the owner of the web
877# server process (e.g., apache) to write into these directories.
878# The directories may have been created by a different user. Version 3.0
879# introduced the use of $dir_perms. There are several of these included
880# in FlowViewer_Configuration.pm. These are the permissions that the
881# scripts will set your various FlowViewer files and directories to.
882# They default to '0777' which permits the open interaction between the
883# web server process owner and the FlowMonitor background process owner.
884# You may want to adjust these permissions differently according
885# to your security policies, and whether you use the same or different
886# accounts for the web and background processes.
887#
888# 2. FlowViewer and FlowGrapher offer the ability to save interesting
889# reports. To do this, the scripts save a temporary copy of every report
890# in advance of the user electing to save it permanently. These
891# intermediate files will accumulate in the 'work' directory specified
892# in the FlowViewer_Configuration file. These files could be removed
893# daily via a cron script to prevent unecessary use of disk space. When
894# the user elects to save a report, it is copied into either the
895# 'reports_directory', or the 'graphs_directory' depending on which
896# function he is running. See discussion of FlowViewer_CleanFiles above.
897#
898# 3. FlowViewer and FlowGrapher offer the ability to resolve NetFlow IP
899# addresses into their host names on the fly. This process is speeded
900# up by caching names into a 'names' file which resides in the directory
901# specified by the 'names_directory' parameter. This parameter defaults
902# to /tmp, but this may not be the best directory for you since it will
903# disappear with a reboot. As you are building up your 'names' file
904# with early runs, you will notice the speed increase dramatically
905# as the 'names' file is used more. The process of resolving names is
906# the primary reason for slower overall FlowViewer performance. You
907# should preferably use the GDBM array database which is fastest.
908# However, not all Perl distributions support GDBM but most do support
909# NDBM. The '$use_NDBM' flag in FlowViewer_Configuration.pm will
910# cause the FlowViewer_Main and FlowGrapher_Main scripts to use NDBM.
911#
912# 4. The FlowViewer and FlowGrapher reporting features use a flow-tools
913# data directory layout that has a particular device at the top. A
914# typical flow-tools directory looks like:
915#
916# /flows/router_1/2005/2005-07/2005-07-04
917#
918# The device name (router_1) is obtained from an array called 'devices'
919# in the FlowViewer_Configuration.pm file. Populate this array with your
920# device names. If your flow-data file structure does not include a
921# device name, for example you are collecting only from one device, set
922# the @devices array to empty (i.e., @devices = ("");) On the web page
923# you can ignore the Devices pulldown selection.
924#
925# As of version 3.3, users may now apply FlowViewer to directories that
926# collect from multiple sources, differentiated by EXPORTER_ID. If you
927# are using this method (i.e., all flow-captures going into a single
928# directory), simply set the $exporter_directory parameter to the
929# directory that is set up to store the flow-data files.
930#
931# 5. Different organizations store captured netflow data differently
932# according to the 'N" setting on the flow-capture statement. However,
933# there is a bug in the flow-tools documentation such that the default
934# value is truly '3' and not '0' as indicated. I have set $N = 3 to
935# reflect the more common setting. The directory structure associated
936# with $N = 3 is shown below:
937#
938# /flows/router_1/2005/2005-07/2005-07-04
939#
940# If you are not seeing output, please check this setting.
941#
942# 6. Version 3.2 introduces Groups. Intermediate RRDtool databases are
943# created on the fly in order to create a temporary FlowMonitor graph that
944# shows the user how the final graph will look. It uses the Perl RRDs.pm
945# RRDtool module to speed this up. Make sure your RRDtool distribution
946# has a compatible RRDs.pm module.
947#
948# Change Log
949#
950# Version 3.4 - March 17, 2011
951#
952# See Version 3.4 Release Notes above
953#
954# Version 3.3
955#
956# See Version 3.3 Release Notes above
957#
958# Version 3.2
959#
960# Version 3.2 introduces Group monitors which are simply a monitor graph
961# made up from the merging of several predefined Individual monitors onto
962# one graph. There are no permanent RRDtool databases associated with a Group.
963# In the construction of a group however, temporary RRDtool databases are
964# created to simulate how the Group will eventually look. The new script
965# Flowmonitor_Group uses RRDs.pm (comes with RRDtool) to generate these
966# transient databases quickly. Please ensure that your RRDs.pm module is
967# compatible with your RRDtool distribution (this should normally be the
968# case - but if you see "ERR: can't handle RRD file version 0003" in
969# DEBUG_GRAPHER, you'll need to upgrade your RRDs.pm.
970#
971# This version also includes a 'speed-up' for FlowMonitor_Collector which
972# now concatenates once for each device. This is controlled by
973# $use_existing_concats, which defaults to "Y".
974#
975# Version 3.2 will continue to work happily along with earlier version
976# exiting Filter and RRDtool files. There are no new FlowViewer_Configuration
977# parameters of consequence. There are now two types of monitors; Individual
978# and Group. The FlowMonitor input screen will default to Individual which is
979# the same as the existing monitors.
980#
981# Woj Kozicki has contributed an Autonomous System (AS) resolving capability
982# and it is included in v 3.2.
983#
984# New FlowViewer, FlowGrapher, and FlowMonitor logos have been developed for
985# version 3.2. These new logos provide embedded links to the other tools so that
986# the user can switch between them easily and retain input parameters.
987#
988# Version 3.1
989#
990# 1. Added MIN, MAX, AVG, 95th PCT to FlowGrapher
991# 2. Added ability to 'archive' monitors
992# 3. Added ability to enter port ranges separated by a colon (:)
993# 4. Can now use any mask length for networks (1 - 32)
994# 5. Added RRDgraph 'lazy-mode' option to speed up graphing
995# 6. Upgraded FlowViewer/Grapher ability to go back more than 30 days
996# 7. Added the ability to configure file permissions
997# 8. Improved speed of FlowGrapher for larger values of 'detail lines'
998# 9. Fixed $rrdtool_bin-directory variable name
999# 10. Added ability to retain intermediate files for debugging
1000# 11. Sorted list of Active Trackings
1001# 12. Fixed bug where non-zero 'cutoof lines' would supress some reports
1002# 13. Fixed FlowViewer rate output to calculate average from all flows
1003# 14. Fixed FlowMonitor_Collector log output (to collect_period_average)
1004#
1005# Version 3.0
1006#
1007# 1. Major new addition of FlowMonitor
1008# 2. Reorganized scripts so that the date and time fields are updated
1009#    with each invocation
1010# 3. Moved common code (e.g., filter creation) to FlowViewer_Utilities.pm
1011# 4. Improved Report Parameters output formatting
1012# 5. Provided host names capability for FlowGrapher (thanks Mark Foster)
1013# 6. Introduced debug and logging capabilities
1014# 7. Merged GDBM/NDBM into a single script (thanks Ed Ravin)
1015#
1016# Version 2.3
1017#
1018# 1. Modified FlowGrapher record processing to not call 'timelocal' for
1019#    epoch times. Other speed improvements. Result: up to 10 times faster.
1020# 2. FlowGrapher error leaving spikes is fixed (thanks Mark Foster)
1021# 3. Bug with concatenation when $N=0 fixed (thanks Dave Faught)
1022#
1023# Version 2.2
1024#
1025# 1. Added flow_select parameter to control which flows are considered
1026#    with respect to the specified time period
1027# 2. Removed Easterm Time (ET) notation. All times are system local
1028#
1029# Version 2.1
1030#
1031# 1. Fixed concatenation. Needs to start one flow file length before start time
1032# 2. Fixed end-of-year problem in FlowGrapher
1033# 3. Small problem for time requests that end just before midnight
1034#
1035# Version 2.0
1036#
1037# 1. Used pipe (|) instead of reading intermediate files (thanks Woj Kozicki!)
1038# 2. Introduced configurable variable $N specifies flow-directory nesting levels
1039# 3. Reduced default value of variable $flow_capture_interval to 1800
1040# 4. Created FlowViewer_NDBM.cgi for users whose Perl does not have GDBM
1041# 5. Created configurable 'work_directory' separate from cgi_bin_directory
1042# 6. Sped up concatenation for requests that cross day boundaries
1043# 7. Added filter fields: Protocol, TOS Field, TCP Flags
1044# 8. Added some more syntax checking
1045# 9. Added FlowGrapher capability (requires GD for Perl)
1046#
1047# Version 1.0 (Original)
1048#
1049#
1050# Vital Assistance
1051#
1052# Special thanks to those FlowViewer users who provided feedback and valuable
1053# suggestions, including Sejin Ahn, Mark Boolootian, Bogdan Ghita, Woj Kozicki,
1054# Ed Arvin, Alex Shepherd, Mike Smith, Scott Wingfield, Vali Magdalinoiu, Ed
1055# ravin, Eric Lautenschlaeger, Sean Cardus, Carles Kishimoto, Shigeki Taniguchi,
1056# Dave Faught, Peter Hofwell, Dario La Guardia, Mike Stowe, Chris Spitzlay and
1057# Mike Donnelly. Big thanks to fellow toiler in the NASA vineyard Mark Foster
1058# for some detailed testing, excellent suggestions, and code to go along with it.
1059# Thanks from all of us to NASA whose unending support of innovation in all fields
1060# has resulted in this toolset.
1061#
1062# Bugs, recommendations
1063#
1064# If you need help installing, have a question, discover a bug, or have a
1065# recommendation, please send an email to:
1066#
1067# Joe Loiacono
1068# jloiacon@csc.com
1069#
1070# FlowViewer is being developed at NASA by a contractor in the employ of the
1071# United States Federal Government in the course of his official duties.
1072# Pursuant to Title 17, Section 105 of the United States Code, this software is
1073# not subject to copyright protection and is in the public domain. FlowViewer is
1074# an experimental system. NASA assumes no responsibility whatsoever for its use
1075# by other parties, and makes no guarantees, expressed or implied, about its
1076# quality, reliability, or any other characteristic.
1077