1fwanalog: A firewall log summarizer that uses Analog 2 3http://tud.at/programm/fwanalog/ 4 5Bal�zs B�r�ny, balazs@tud.at 6 7Current version: 0.6.9 8 9This program summarizes firewall logs and creates reports from them. 10 11There are lots of programs that do this. But they weren't good enough for me: 12- I use OpenBSD and GNU/Linux, so I need a solution that can handle both. 13- I want pretty reports, see recent attacks, host names instead of IP 14 addresses, HTML output. 15- I want a daily report mailed to me, of course in text format. 16 17I use the excellent logfile analyzer Analog ( http://www.analog.cx/ ) a lot. 18It is very flexible. So I thought I could convert firewall logs to web 19server logs Analog can use. 20 21This shell script does exactly that. 22 231. It parses the firewall log (I can choose which one) and converts it to a 24logfile that Analog understands. The fields in this file are faked, of course; 25e.g. the network interface name is the virtual host. 26 272. It calls Analog with some nice options and so creates different reports: 28one about all data; one about the last week; one about the current day in 29HTML format and a short one in ASCII format for a report e-mailed daily. 30 313. Optionally, it can create a separate report for each host and each 32blocked packet so you can look at the actions of a "bad guy" or answer 33the question "who scanned port 443?". 34 35Requires: Perl, zegrep, awk, diff, sed, egrep, and of course Analog. 36Most of these programs should be already installed on any Unix system. 37You should really have the GNU versions somewhere and configure them in 38fwanalog.opts. 39 40Install Analog: 41Debian GNU/Linux: "apt-get install analog" 42Any modern BSD: "cd /usr/ports/www/analog; make" 43Other GNU/Linux: probably there are RPMs for your distribution 44Any other OS: Download from http://www.analog.cx/ , compile, install 45 46Installation 47============ 481. Decompress the distribution in some directory, e.g. /usr/local/fwanalog 492. Symlink, move or copy the fwanalog.opts.{your OS} to "fwanalog.opts" 503. Edit fwanalog.opts if necessary (most settings should be OK, though) 514. If your Analog version is not the newest stable one, find a language 52 file for it in the langfiles/ directory and copy it over fwanalog.lng 535. On a non-free Unix (e.g. Solaris), modifiy the first line of the 54 fwanalog.sh script to "#! /bin/bash" or where your bash or ksh shell 55 is. Also, look if you have the GNU versions of the utilites listed in 56 fwanalog.opts. 576. Execute ./fwanalog.sh 587. There should be some HTML and text reports in the directory you specified 59 in fwanalog.opts ("$outdir"). 60 61Customizing 62=========== 63You can edit fwanalog.analog.conf.local to suit your taste, e.g. add pretty 64icons and style sheets, switch reports on and off (however, the deactivated 65reports don't make much sense with firewall logs). It is better not to edit 66the master fwanalog.analog.conf yourself as that file probably will be 67updated by me in the next fwanalog release. 68 69You can also edit fwanalog.sh and change the Analog command line options, 70deactivate reports and create a conversion function for your firewall if it is 71not supported. It's easy. If you think that your changes made the program 72better, please send them to me so I can include them in the next version. 73 74Troubleshooting 75=============== 76Some frequent problems: 77- The language file doesn't match the Analog version. This can happen with 78 new installations or after an upgrade of Analog or fwanalog. 79 Make sure that you use one the correct version of the language file; the 80 major and the first minor version numbers must match. (E.g. analog 5.32 81 works with the 5.3 langfile.) 82- "It works perfectly when called from the command line but not when called 83 from cron!" - Search for differences between your shell's and the cron 84 shell's configuration. The cron path sometimes doesn't include 85 /usr/local/bin where Analog can be etc. 86 87If you have a problem with fwanalog, go to the homepage and read through the 88mailing list archives. Many common problems are already solved there. If 89not, subscribe to the mailing list and ask there so more people can help 90you. I don't have time to answer e-mails with problems that can be solved by 91reading the documentation and/or the knowledge in the mailing list archives. 92 93"One host" mode 94=============== 95You can set "onehost=" to true in fwanalog.opts if you are analyzing the 96logs of only one host. This will cause fwanalog to show each packet source 97host (i.e. attacker) with the ports it tried. 98However, setting this option loses the information about the target IP 99address. So don't set this if your firewall protects an entire network. 100This feature is based on an idea by Kenneth Vestergaard Schmidt, who is 101also the Debian maintainer for fwanalog. 102There is also an option "onehost=dynip" based on an idea by Ralph Niere. 103This is useful if the address of your firewall changes often, e.g. because 104you are on a dial-up connection with dynamic IPs. 105 106Creating separate reports of hosts and packets 107============================================== 108In fwanalog.opts, set sep_hosts and/or sep_packets to true. Note that this 109will cause analog to run once for each host and each packet in the "current" 110log. This shouldn't be a problem on a modern machine when fwanalog is run 111periodically (e.g. once a day). 112As this processes the current log, it will probably run for a long time when 113you run fwanalog. If you update from an older version of fwanalog, it will 114only process the new log entries since the last invocation, so only a few 115packets and hosts will be linked in the reports. You can call analog with 116"-a host" or "-p packet" to create a report for a host or a packet you are 117interested in. In the future, this host or packet will always be linked in 118the reports. 119 120Services 121======== 122fwanalog includes a services.conf file for Analog to convert port numbers 123like 21 into service names like ftp. If you think that your services list 124is better, feel free to use support/mkservices.conf.sh with your list. 125 126There is also a well_known_services.conf file in support/. It includes lots 127of port definitions of more-or-less well known ports. You can include that 128file by simply appending it to services.conf or by editing 129fwanalog.analog.conf.local to include it. However, so many aliases make 130analog slower. 131 132Creating conversion functions for unsupported firewall formats 133============================================================== 134If your firewall is not supported, please contribute a conversion routine. 135It is not very hard: 1361. Add your format in fwanalog.opts to the known ones. 1372. Copy the ipf or iptables function in fwanalog.sh into a new function with 138 the name of your firewall (the same you added to fwanalog.opts). 1393. Grep the lines about blocked packets from your firewall log into the 140 fwlog.current file in the output directory 1414. Call mkdateconvscript and sed if your log file doesn't contain years. 1425. Change the long perl regexp (or use any other tool if you like) so it 143 changes all lines into the faked web server log format. It is not very 144 hard if you know regular expressions. Be careful with \$! 145 146Language files 147============== 148Language files define the strings in the fwanalog output. 149Most versions of analog require language files of a matching version. 150You might find a language file for your Analog version (and your language, 151if you prefer) in the langfiles/ subdirectory of the fwanalog distribution. 152If you have a currently unsupported version of Analog, try the mklangfile 153scripts in the support/ directory of the fwanalog distribution. Please 154contribute language files you have created by submitting them to the author. 155 156Report mappings 157=============== 158fwanalog renames some Analog reports. Not all reports are switched on by default. 159(See http://www.analog.cx/docs/output.html for details on Analog's reports) 160 161Analog report fwanalog report remarks 162------------- --------------- ------- 163GENERAL General summary The first report, gives an overview 164YEARLY Yearly report Makes sense if you have firewall logs for more than a year 165QUARTERLY Quarterly report Makes sense if you have firewall logs for more than 3 months 166MONTHLY Monthly report 167WEEKLY Weekly report See also the "WEEKBEGINSON" in fwanalog.analog.conf.local 168DAILYREP Daily report 169DAILYSUM Daily summary Summary by weekdays 170HOURLYREP Hourly report 171HOURLYSUM Hourly summary Summary by hour of day 172WEEKHOUR Hour of the Week Summary 173QUARTERREP Quarter-hour report 174QUARTERSUM Quarter-hour summary 175FIVEREP Five-minute report 176FIVESUM Five-minute summary 177HOST Packet Source Host Which hosts sent the packets that your firewall blocked 178REDIRHOST - Doesn't make sense with firewall logs 179FAILHOST - Doesn't make sense with firewall logs 180ORGANISATION Organization report 181DOMAIN Domain report top level domains 182REQUEST - Not used: the directory report is better 183 suited for fwanalog. 184DIRECTORY Blocked Packet Detailed report of blocked packets. 185 If onehost=false, the target address; 186 if onehost=true, the source address and the target port. 187FILETYPE - Doesn't make sense with firewall logs 188SIZE Packet Size Not many variations with some firewall settings 189PROCTIME Processing time Not very interesting 190REDIR - Doesn't make sense with firewall logs 191FAILURE - Doesn't make sense with firewall logs 192REFERRER Source Port Sometimes interesting, e.g. with port 21 193REFSITE - Doesn't make sense with firewall logs 194SEARCHQUERY - Doesn't make sense with firewall logs 195SEARCHWORD - Doesn't make sense with firewall logs 196INTSEARCHQUERY - Doesn't make sense with firewall logs 197INTSEARCHWORD - Doesn't make sense with firewall logs 198REDIRREF - Doesn't make sense with firewall logs 199FAILREF - Doesn't make sense with firewall logs 200BROWSERREP - MAC Address report (if your firewall logs them) 201BROWSERSUM - Doesn't make sense with firewall logs 202OSREP - Would be nice, but no firewall logs it 8-( 203VHOST Interface Report You can turn it off if you have only one interface 204REDIRVHOST - Doesn't make sense with firewall logs 205FAILVHOST - Doesn't make sense with firewall logs 206USER Log Prefix Report Only with iptables, if you set a log prefix 207 Analog ignores this if it sees no data. 208REDIRUSER - Doesn't make sense with firewall logs 209FAILUSER - Doesn't make sense with firewall logs 210STATUS - Doesn't make sense with firewall logs 211 212OpenBSD 3.x problem 213=================== 214The developers of the new OpenBSD firewall "pf" decided that they log 215blocked packets in a binary format instead of a text as usual on Unix. 216This file can be only read by the OpenBSD version of tcpdump. So, 217fwanalog must run on the OpenBSD 3.x machine itself in order to process 218OpenBSD 3.x logfiles. 219All other logfiles can be handled on each architecture, e.g. a Linux 2.4 220machine can process the logfiles of Solaris, or FreeBSD the logs of 221Linux 2.2 etc. 222 223Other documentation 224=================== 225See README.firewall for hints on configuring your firewall. 226See README.sudo for information about running fwanalog as a non-root user. 227 228Please mail your suggestions, patches, bugfixes etc. to balazs@tud.at . 229 230$Id: README,v 1.30 2004/03/18 16:40:17 bb Exp $ 231