fwanalog: A firewall log summarizer that uses Analog

http://tud.at/programm/fwanalog/

Bal�zs B�r�ny, balazs@tud.at

Current version: 0.6.9

This program summarizes firewall logs and creates reports from them.

There are lots of programs that do this. But they weren't good enough for me:
- I use OpenBSD and GNU/Linux, so I need a solution that can handle both.
- I want pretty reports, see recent attacks, host names instead of IP
  addresses, HTML output.
- I want a daily report mailed to me, of course in text format.

I use the excellent logfile analyzer Analog ( http://www.analog.cx/ ) a lot.
It is very flexible. So I thought I could convert firewall logs to web
server logs Analog can use.

This shell script does exactly that.

1. It parses the firewall log (I can choose which one) and converts it to a
logfile that Analog understands. The fields in this file are faked, of course;
e.g. the network interface name is the virtual host.

2. It calls Analog with some nice options and so creates different reports:
one about all data; one about the last week; one about the current day in
HTML format and a short one in ASCII format for a report e-mailed daily.

3. Optionally, it can create a separate report for each host and each
blocked packet so you can look at the actions of a "bad guy" or answer
the question "who scanned port 443?".

Requires: Perl, zegrep, awk, diff, sed, egrep, and of course Analog.
Most of these programs should be already installed on any Unix system.
You should really have the GNU versions somewhere and configure them in
fwanalog.opts.

Install Analog:
Debian GNU/Linux: 	"apt-get install analog"
Any modern BSD:		"cd /usr/ports/www/analog; make"
Other GNU/Linux:	probably there are RPMs for your distribution
Any other OS:		Download from http://www.analog.cx/ , compile, install

Installation
============
1. Decompress the distribution in some directory, e.g. /usr/local/fwanalog
2. Symlink, move or copy the fwanalog.opts.{your OS} to "fwanalog.opts"
3. Edit fwanalog.opts if necessary (most settings should be OK, though)
4. If your Analog version is not the newest stable one, find a language
   file for it in the langfiles/ directory and copy it over fwanalog.lng
5. On a non-free Unix (e.g. Solaris), modifiy the first line of the
   fwanalog.sh script to "#! /bin/bash" or where your bash or ksh shell
   is. Also, look if you have the GNU versions of the utilites listed in
   fwanalog.opts.
6. Execute ./fwanalog.sh
7. There should be some HTML and text reports in the directory you specified
   in fwanalog.opts ("$outdir").

Customizing
===========
You can edit fwanalog.analog.conf.local to suit your taste, e.g. add pretty
icons and style sheets, switch reports on and off (however, the deactivated
reports don't make much sense with firewall logs). It is better not to edit
the master fwanalog.analog.conf yourself as that file probably will be
updated by me in the next fwanalog release.

You can also edit fwanalog.sh and change the Analog command line options,
deactivate reports and create a conversion function for your firewall if it is
not supported. It's easy. If you think that your changes made the program
better, please send them to me so I can include them in the next version.

Troubleshooting
===============
Some frequent problems:
- The language file doesn't match the Analog version. This can happen with
  new installations or after an upgrade of Analog or fwanalog.
  Make sure that you use one the correct version of the language file; the
  major and the first minor version numbers must match. (E.g. analog 5.32
  works with the 5.3 langfile.)
- "It works perfectly when called from the command line but not when called
  from cron!" - Search for differences between your shell's and the cron
  shell's configuration. The cron path sometimes doesn't include
  /usr/local/bin where Analog can be etc.

If you have a problem with fwanalog, go to the homepage and read through the
mailing list archives. Many common problems are already solved there. If
not, subscribe to the mailing list and ask there so more people can help
you. I don't have time to answer e-mails with problems that can be solved by
reading the documentation and/or the knowledge in the mailing list archives.

"One host" mode
===============
You can set "onehost=" to true in fwanalog.opts if you are analyzing the
logs of only one host. This will cause fwanalog to show each packet source
host (i.e. attacker) with the ports it tried.
However, setting this option loses the information about the target IP
address. So don't set this if your firewall protects an entire network.
This feature is based on an idea by Kenneth Vestergaard Schmidt, who is
also the Debian maintainer for fwanalog.
There is also an option "onehost=dynip" based on an idea by Ralph Niere.
This is useful if the address of your firewall changes often, e.g. because
you are on a dial-up connection with dynamic IPs.

Creating separate reports of hosts and packets
==============================================
In fwanalog.opts, set sep_hosts and/or sep_packets to true. Note that this
will cause analog to run once for each host and each packet in the "current"
log. This shouldn't be a problem on a modern machine when fwanalog is run
periodically (e.g. once a day).
As this processes the current log, it will probably run for a long time when
you run fwanalog. If you update from an older version of fwanalog, it will
only process the new log entries since the last invocation, so only a few
packets and hosts will be linked in the reports. You can call analog with
"-a host" or "-p packet" to create a report for a host or a packet you are
interested in. In the future, this host or packet will always be linked in
the reports.

Services
========
fwanalog includes a services.conf file for Analog to convert port numbers
like 21 into service names like ftp. If you think that your services list
is better, feel free to use support/mkservices.conf.sh with your list.

There is also a well_known_services.conf file in support/. It includes lots
of port definitions of more-or-less well known ports. You can include that
file by simply appending it to services.conf or by editing
fwanalog.analog.conf.local to include it. However, so many aliases make
analog slower.

Creating conversion functions for unsupported firewall formats
==============================================================
If your firewall is not supported, please contribute a conversion routine.
It is not very hard:
1. Add your format in fwanalog.opts to the known ones.
2. Copy the ipf or iptables function in fwanalog.sh into a new function with
   the name of your firewall (the same you added to fwanalog.opts).
3. Grep the lines about blocked packets from your firewall log into the
   fwlog.current file in the output directory
4. Call mkdateconvscript and sed if your log file doesn't contain years.
5. Change the long perl regexp (or use any other tool if you like) so it
   changes all lines into the faked web server log format. It is not very
   hard if you know regular expressions. Be careful with \$!

Language files
==============
Language files define the strings in the fwanalog output.
Most versions of analog require language files of a matching version.
You might find a language file for your Analog version (and your language,
if you prefer) in the langfiles/ subdirectory of the fwanalog distribution.
If you have a currently unsupported version of Analog, try the mklangfile
scripts in the support/ directory of the fwanalog distribution. Please
contribute language files you have created by submitting them to the author.

Report mappings
===============
fwanalog renames some Analog reports. Not all reports are switched on by default.
(See http://www.analog.cx/docs/output.html for details on Analog's reports)

Analog report       fwanalog report     remarks
-------------       ---------------     -------
GENERAL             General summary     The first report, gives an overview
YEARLY              Yearly report       Makes sense if you have firewall logs for more than a year
QUARTERLY           Quarterly report    Makes sense if you have firewall logs for more than 3 months
MONTHLY             Monthly report
WEEKLY              Weekly report       See also the "WEEKBEGINSON" in fwanalog.analog.conf.local
DAILYREP            Daily report
DAILYSUM            Daily summary       Summary by weekdays
HOURLYREP           Hourly report
HOURLYSUM           Hourly summary      Summary by hour of day
WEEKHOUR            Hour of the Week    Summary
QUARTERREP          Quarter-hour report
QUARTERSUM          Quarter-hour summary
FIVEREP             Five-minute report
FIVESUM             Five-minute summary
HOST                Packet Source Host  Which hosts sent the packets that your firewall blocked
REDIRHOST           -                   Doesn't make sense with firewall logs
FAILHOST            -                   Doesn't make sense with firewall logs
ORGANISATION        Organization report
DOMAIN              Domain report       top level domains
REQUEST             -					Not used: the directory report is better
										suited for fwanalog.
DIRECTORY           Blocked Packet      Detailed report of blocked packets.
                                        If onehost=false, the target address;
                                        if onehost=true, the source address and the target port.
FILETYPE            -                   Doesn't make sense with firewall logs
SIZE                Packet Size         Not many variations with some firewall settings
PROCTIME            Processing time     Not very interesting
REDIR               -                   Doesn't make sense with firewall logs
FAILURE             -                   Doesn't make sense with firewall logs
REFERRER            Source Port         Sometimes interesting, e.g. with port 21
REFSITE             -                   Doesn't make sense with firewall logs
SEARCHQUERY         -                   Doesn't make sense with firewall logs
SEARCHWORD          -                   Doesn't make sense with firewall logs
INTSEARCHQUERY      -                   Doesn't make sense with firewall logs
INTSEARCHWORD       -                   Doesn't make sense with firewall logs
REDIRREF            -                   Doesn't make sense with firewall logs
FAILREF             -                   Doesn't make sense with firewall logs
BROWSERREP          -                   MAC Address report (if your firewall logs them)
BROWSERSUM          -                   Doesn't make sense with firewall logs
OSREP               -                   Would be nice, but no firewall logs it 8-(
VHOST               Interface Report    You can turn it off if you have only one interface
REDIRVHOST          -                   Doesn't make sense with firewall logs
FAILVHOST           -                   Doesn't make sense with firewall logs
USER                Log Prefix Report	Only with iptables, if you set a log prefix
										Analog ignores this if it sees no data.
REDIRUSER           -                   Doesn't make sense with firewall logs
FAILUSER            -                   Doesn't make sense with firewall logs
STATUS              -                   Doesn't make sense with firewall logs

OpenBSD 3.x problem
===================
The developers of the new OpenBSD firewall "pf" decided that they log
blocked packets in a binary format instead of a text as usual on Unix.
This file can be only read by the OpenBSD version of tcpdump. So,
fwanalog must run on the OpenBSD 3.x machine itself in order to process
OpenBSD 3.x logfiles.
All other logfiles can be handled on each architecture, e.g. a Linux 2.4
machine can process the logfiles of Solaris, or FreeBSD the logs of
Linux 2.2 etc.

Other documentation
===================
See README.firewall for hints on configuring your firewall.
See README.sudo for information about running fwanalog as a non-root user.

Please mail your suggestions, patches, bugfixes etc. to balazs@tud.at .

$Id: README,v 1.30 2004/03/18 16:40:17 bb Exp $

Configuring the firewall for fwanalog
=====================================
- Make sure that each dropped packet is logged, only dropped packets are
  logged (however, some firewalls log this info, so fwanalog can
  distinguish them itself), and each packet is only logged once. (If you
  like precise statistics, that is.)
  Note: The lines "last line repeated X times" in some logfiles are NOT
  processed by fwanalog. I know that this leads to lower numbers of
  blocked packets but can't really do anything about it - it would be
  too hard to parse this with shellscript only. I don't think that this
  is a huge problem because if a host sends you the same packet so
  quickly it will stand out in the logs anyway.

- It is a good idea to use "--log-prefix some_info_about_the_block" with
  iptables. Because of a limitation in Analog's username parsing, you
  can't use spaces in the log prefix. (You *can* use them but fwanalog
  will only use them until the first space. So "bad in" and "bad out"
  become "bad". Use "bad_in" and "bad_out".)

  (Does another firewall support this? I would gladly include this
  feature for the other ones.)

- Some versions of ipf offer to resolve IP addresses and port numbers to
  hostnames and service names. You shouldn't do this with fwanalog
  because analog can do it better (and fwanalog won't work at all with
  such logs because it expects IP addresses and numeric port names).

Alternative syslog implementations
==================================
There some alternatives to the good old syslog and they have possibly
differing log formats.
Fwanalog doesn't support those by default because that would mean supporting X
different firewall formats multiplied by Y syslog formats and the result would
be entirely unmaintable.
Here are a few hints on what you can do.

- Metalog: One colon (:) before the log message is missing. Find this colon in
  the regular expression of your firewall function, or pre-process your logs
  and add the colon on the right place.

How to setup syslog on a NETGEAR or ZyXEL Internet Gateway Router's ZyNOS
=========================================================================

By Matt Christian <mattc@visi.com>
Version 1.1

The below instructions assume that you are familiar with telnet and making
some Unix configuration changes.  If you aren't then you may want to ask
a knowledgeable friend for help.

1. Telnet into your router (default: 192.168.0.1 or 192.168.1.1)
  $ telnet 192.168.0.1

2. Login using your password (default: 1234)
  Password: ****

3. Navigate the following menus (type in the number and press enter/return)
  "24. System Maintenance" -> "3. Log and Trace" -> "2. UNIX Syslog"

4. You should see a menu similar to example below:

                 Menu 24.3.2 - System Maintenance - UNIX Syslog

                    Syslog:
                    Active= No
                    Syslog IP Address=
                    Log Facility= Local 1

                    Types:
                    CDR= No
                    Packet triggered= No

                    Filter log= No
                    PPP log= No

5. Set the following information (follow prompts at bottom of screen)
   Active = Yes, Syslog IP Address = fwAnalog machine,
   Log Facility = your choice, CDR = Yes, Packet triggered = Yes,
   Filter log = Yes, PPP log = Yes

6. At the prompt, press ENTER, ESC, ESC, 99 to exit

7. On the fwAnalog machine (the IP you put in for "Syslog IP Address"),
   setup your syslog.conf to log the syslog facility (you put in for
   "Log Facility") to a log file.  For example, if you used "Local 1" then
   your syslog.conf file should contain something like the following:

local1.*                        -/var/log/router.log

8. Restart the syslogd daemon, usually by sending a SIGHUP signal to it.

9. Modify the fwanalog.opts file to pick up this log file (or files if you
setup log rotation on this log file).

10. Enjoy!

Setting up logging for fwanalog on a Cisco PIX firewall
=======================================================

By Ric Moseley <ric@theplanet.com>

On the PIX firewall running version 6.22 I added the following commands
to turn logging on.

logging on
logging timestamp
logging console warnings
logging buffered warnings
logging trap warnings
logging history warnings
logging facility 20
logging host [<interface_name>] <ip_address> [tcp|udp/port#]

Add this to your syslog.conf on the logging host:
local4.debug                    /var/log/firewall

Setting up logging for fwanalog on a Watchguard Firebox System 6.1
==================================================================

By Ric Moseley <ric@theplanet.com>

Open up the policy manager and go to 'setup->logging'.
Choose the syslog tab and fill in the server IP and the facility.

Set up the logging host like for Cisco.


$Id: README.firewall,v 1.6 2004/03/18 16:34:45 bb Exp $

Running fwanalog as a normal user, using sudo

The problem: usually, only root can access the logfiles with the
firewall logs, so fwanalog.sh must be run by root. However, it is a
fairly complex shell script, bugs in it could be fatal if exploited. So
it would be nice if normal users could run fwanalog.sh.

Fortunately, there are some solutions for this.

Solution 1: add the user to the admin/wheel/whatever group that can read
the logfiles. However, this grants her/him more privileges than are
really necessary.

Solution 2: use Sudo to grant the user the permission to search for
firewall patterns in the system log.

As root, type "visudo", this edits /etc/sudoers or wherever it is on
your system. Be sure that you read "man sudo" and "man sudoers" before
so you know what you do.

Add the following lines:

# rules for people who can use fwanalog on this machine
User_Alias      FWANALOG_USERS = {username}
Cmnd_Alias      FWANALOG_ZEGREP = {zegrep command} {zegrep params} {logfiles}
FWANALOG_USERS  ALL = NOPASSWD: FWANALOG_ZEGREP

{username} should be the name of the user who runs fwanalog
{zegrep command} is your zegrep.
    If you want to be really safe, use the path name, e.g /bin/zegrep
{zegrep params} is what fwanalog uses for grepping your logfiles. Look
    into fwanalog.sh, locate the function that searches the patterns in
	your logfiles, and copy its command line parameters: -h and the
	pattern, without the quotes, and without "$inputfiles".
{logfiles} should be your logfiles, either as a shell pattern
    (e.g. /var/log/messages*) or specified directly (e.g.
	/var/log/messages /var/log/messages.0)

For example, my sudoers entry on a Linux 2.4 machine looks like this:

User_Alias      FWANALOG_USERS = bb
Cmnd_Alias      FWANALOG_ZEGREP = /bin/zegrep -h IN.+OUT.+SRC.+DST.+LEN.+TTL.+PROTO.+ /var/log/messages*
FWANALOG_USERS  ALL = NOPASSWD: FWANALOG_ZEGREP

Test it by executing "sudo /path/to/zegrep {params} {logfiles}" as the
user. If it works, you can modify the "zegrep=..." line in fwanalog.opts
to 'zegrep="sudo /path/to/zegrep"'.

$Id: README.sudo,v 1.1 2002/03/08 09:06:51 bb Exp $