This is an experimental web server, meant to demonstrate how to write a
scalable web server using the IO and IOB API from libowfat
(http://www.fefe.de/libowfat/).

To compile this, you will need the current libowfat version, probably
even the current CVS version.

gatling now supports some anti denial of service code.

You can set a limit of how often each IP can request files from gatling
per minute.  gatling will then keep book about which IPs had how many
requests in the last minute, and if someone exceeds that limit, they
will only be allowed back in if they did not request anything for a
whole minute.

If someone gets caught at DOSsing you, their connection will be
tarpitted.  If gatling is run in no-timeout mode, the connection is
dropped immediately instead.  The idea behind the tarpit is to stop
accidental flooding (if we drop the connection, we might make an
accidental flood worse due to immediate retry).  On the down side the
tarpit allows others to consume resources on our box.  So it is still a
denial of service on our resources if someone opens a lot of connections
against us, because the state for the connections wastes our memory.

bindbench will create many sockets and then bind them to port 0 (i.e.
tell the operating system to assign a free port to them).

Ideally, this should be a constant time operation, but some
implementations may implement selecting the smallest free port by
traversing a list or array.

On i386, bindbench will read the task cycle counter instead of
gettimeofday and give the results in CPU cycles, not usec.  You can
divide the numbers by the CPU frequency to get comparable numbers.  As
there is no portable way to get the CPU frequency, bindbench does not
even pretend to do this.

Gatling now has primitive CGI support.

To use it, touch ".proxy" in the root of the virtual host, for example

  $ touch default/.proxy

and then start gatling with -C and a regex by which to detect CGIs:

  # gatling -C '\.cgi'

You can also tell gatling to consider all executable files CGI programs:

  # gatling -C+x

Then, even index.html will be run as CGI if it is executable, allowing
for example a dynamically generated homepage on http://example.com/
without index.html having to do a lame redirect.  In this mode, gatling
will do a primitive check and only run CGIs that have the ELF magic
(i.e. look like an ELF binary) or the Shebang (#!, i.e. look like a
shell/perl/whatever script).

forkbench will fork off many child processes (settable on the command
line with -c, default is 1000).

Each child will write a single character into a pipe, and the parent
will read that single character out of the pipe.  Then the child will
block and wait for SIGTERM.  The parent will take the time for creating
the child and receiving the character.

Ideally, this time should be more or less constant.

gatling now also speaks FTP, and it is enabled per default.
Disable it with -F.

Working around itojun's disabled IPv4-mapped IPv6 addresses is even
worse for FTP than it is for HTTP.  I'm not going to waste my time on
this for now.  Please ask itojun himself to prove how "easy" or even
"trivial" it is to do this, as he always claims it is.

Like HTTP will bind to port 80 if running as root, or 8000 otherwise,
FTP will bind to port 21 and 2121.  To specify the FTP port, use
-f -p [port].  Example:

  gatling -p 81 -f -p 2100

would run a HTTP server on port 81 and an FTP server on port 2100.  For
now, HTTP and FTP will always bind to the same IP number.

Please note:

  a) No TELNET sequences.
     These are _really_ obsolete, a pain in the ass to implement, and
     have even been used as means to avoid intrusion detection systems
     due to the obscurity.

  b) The path checking deliberately _allows_ to leave the file system,
     as long as you follow a symbolic link in the process.  That means,
     if you symlink out of the FTP file system (and the destination is in
     the chroot jail), gatling will allow FTP (and HTTP!) users to
     follow the symlink.  However, following a directory symlink and
     appending "/../" will not follow the .. directory entry from the
     target directory of the symlink, as an attacker may hope.

     The idea is to make it easy to create an FTP Server by putting a
     few symlinks to directories you want to export in an empty
     directory and starting gatling there.

  c) Like for HTTP, gatling will do virtual hosting, i.e. if the client
     connected to IP 10.1.1.23 on port 21, gatling will look for the
     exported data in the directory "10.1.1.23:21".

  d) gatling will not let users download files that are not world
     readable, even if the permissions would normally allow the gatling
     process to read the files.  This is to prevent accidental
     publication of sensitive files.

  e) gatling accepts uploads per anonymous FTP, but only to directories
     that are world writable.  Since there normally are no world
     writable directories, this should not pose much of a threat to
     anyone.  Please note that the files are created with mode 600,
     which means gatling will not let others download uploaded files, so
     it cannot be exploited as warez dump.

     You can disable uploads altogether with -U, or you can allow
     anonymous downloaders to download just uploaded files with -a.

  f) gatling's directory listings will always claim files are owned by
     root.  The local accounts on FTP sites are ignored by software and
     not normally useful to outsiders anyway, but revealing them may
     expose more of your organisation to FTP users than you want.

gatling supports very basic .htaccess handling.

If checks (in the current directory only!) for a .htaccess file, which
is expected to have the following syntax:

  Realm
  username:password
  username2:password2
  [...]

Realm is simply a string that is usually displayed by the browser when
prompting for the password.  username is the user name in plain text,
and password is the password in crypt(3) format (like in /etc/passwd).

gatling only supports basic authentication.

You can also password protect a whole server.  Just put a
.htaccess_global file in the directory of the server, same syntax.

gatling is a non-forking HTTP server.
It listens on port 80 when running as root, or port 8000 otherwise.
You can specify the port with -p, i.e.

  gatling -p 81

to bind to port 81.  Run gatling -h to get a list of supported command
line options.  Connection Keep-Alive and Pipelining are supported.

Please note:

  a) No HTTP/0.9 support

  b) gatling deliberately _allows_ to leave the file system, as long as
     the request follows a symbolic link in the process.  That means, if
     you symlink out of the HTTP file system (and the destination is in
     the chroot jail), gatling will allow HTTP (and FTP!) users to
     follow the symlink.  However, following a directory symlink and
     appending "/../" will not follow the .. directory entry from the
     target directory of the symlink, as an attacker may hope.

     The idea is to make it easy to create an FTP Server by putting a
     few symlinks to directories you want to export in an empty
     directory and starting gatling there.

httpbench is a latency measurement tool.

Call it like this:

  ./httpbench -k -c 1000 -i 10 -s 5 http://127.0.0.1:8000/README.httpbench

This means that httpbench should open 500 HTTP connections to
127.0.0.1:8000, and every 10 connections it should 5 times request "/"
from the web server there.  The -k tells httpbench to run these 5
requests over the same TCP connection, using HTTP keep-alive.

The output from httpbench looks something like this:

  connecting to 127.0.0.1 port 8000
  sample 69 229
  clat 58
  clat 59
  clat 51
  clat 50
  clat 52
  clat 49
  clat 50
  clat 50
  clat 63
  clat 50
  sample 50 389
  [...]

The first line is for informational purposes.

Lines starting with "sample" give two timings, first the latency for
opening the TCP connection (which should be pretty small unless the web
server has problems calling accept() as fast as incoming connections
arrive).  The second is the average latency for writing the HTTP request
and reading the answer.  The timings are in ?s.

Lines starting with "clat" give the normal TCP connect() latency.  Since
this does not incur any actual work for the HTTP server (besides getting
the event and calling accept()), this should be uniformly small.

Here are a few useful command lines to plot this with gnuplot:

  plot "linux.log" using ($2) title "Linux 2.6: connect latency" with histeps

  plot "linux.log" using ($3) title "Linux 2.6: response latency" with histeps

httpbench can also be used to measure throughput.  To do that, run it
with -c 1 and give the URL to a large file (an ISO image or movie file >
100 MB is good).  The output will look something like this:

  connecting to 127.0.0.1 port 80
  tput 21414
  tput 25054
  tput 25473
  tput 13086
  tput 22462
  tput 110
  tput 25104
  tput 23306
  tput 25026
  tput 25067
  tput 25453
  tput 22296
  tput 25709
  tput 25394
  tput 25141
  tput 23719
  tput 24419
  tput 25089
  tput 25459
  tput 23715
  tput 24156
  tput 25413
  tput 24975
  tput 23840
  tput 24563
  tput 25072
  tput 25451
  [...]

This is from my external firewire hard disk over the loopback interface.
There will be one tput line per megabyte downloaded.  The number is
1000000000 divided by the number of ?sec it took to download this
megabyte, i.e. the average number of kilobytes per second.

manymapbench will mmap the first page of many small files.  This is what
mmapbench was supposed to approximate, but I am told that it does not,
but it triggers the worst case of an optimization in the BSD VM instead.

You are supposed to create the data by running ./mktestdata instead,
with the same -c argument.  Please note that manymapbench does not prime
the cache first, so please run it twice and disregard the output of the
first run.

manymapbench will output three numbers in each line, for example

  6753 2338 3094

The first number is the latency for opening the file (probably not
useful, but measuring it does not cost anything).  The second number is
the latency for mmapping the page.  The third number is the latency for
reading one byte on the page (which ought to trigger a page fault unless
the VM reads the page anyway if it encompasses the whole file, which
Linux 2.6 does, judging from the number in this example).

On i386, manymapbench will read the task cycle counter instead of
gettimeofday and give the results in CPU cycles, not usec.  You can
divide the numbers by the CPU frequency to get comparable numbers.  As
there is no portable way to get the CPU frequency, manymapbench does not
even pretend to do this.

mmapbench will take one file as argument on the command line.
It will then map ever second page of the file, "count" times.
count is 25000 per default, (i.e. 100 megs are mapped, the file needs to
be at least 200 megs large).

The program will then prime the cache and read a few bytes from every
page that is mmapped later on, to make sure the pages are in the buffer
cache.

mmapbench will then mmap every other page and measure the latency of
that, and it will then read a byte of the page, which should force a
page fault in the OS and cause the OS to actually map the page to
memory.

On i386, mmapbench will read the task cycle counter instead of
gettimeofday and give the results in CPU cycles, not usec.  You can
divide the numbers by the CPU frequency to get comparable numbers.  As
there is no portable way to get the CPU frequency, mmapbench does not
even pretend to do this.

NOTE: if you get a bus error, you gave too small a file as argument.
The default settings from run-bench assume a file (or block device!) at
least 160 MB.  So the easiest way is to just point mmapbench to your
root partition, or use dd to create a large new file.

gatling uses the io and iob interfaces from libowfat and thus exploits
OS specific performance hacks on the following operating systems:

  Linux 2.4:
    O(1) event notification scalability through sigio
    zero-copy TCP via sendfile

  Linux 2.6:
    O(1) event notification scalability through epoll
    zero-copy TCP via sendfile

  FreeBSD 4 and 5:
    O(1) event notification scalability through kqueue
    zero-copy TCP via sendfile

  NetBSD 2:
    O(1) event notification scalability through kqueue
    zero-copy TCP via mmap+write

  OpenBSD 3.4:
    They have kqueue, but it's not O(1)
    They don't have sendfile, and to my knowledge their mmap+write is
      not zero-copy

  HP-UX 11:
    There were rumours of /dev/poll, but my test box didn't have it.
    So we have O(n) event notification scalability through poll(2)
    zero-copy TCP via sendfile

  IRIX 6.5:
    O(1) event notification scalability through /dev/poll
    Rumour has it that mmap+write is zero-copy in IRIX.
    I have no way of knowing, though

  Solaris 9:
    O(1) event notification scalability through /dev/poll
    zero-copy TCP via sendfile

  MacOS X:
    Uses kqueue, but I haven't benchmarked it yet to see if it's O(1)
    the headers in panther declare sendfile (protected with #if SENDFILE
    (not #ifdef, #if!)), but libc doesn't have it, and neither does any
    other library.

  AIX:
    AIX 5 has send_file.  I am not aware of any way to speed up poll on
    AIX.

gatling now supports SCGI and FastCGI and can thus be used to run, for
example, PHP scripts.

Here's how to use it.

  1. compile gatling with proxy mode (this is on by default).
  2. enable proxying for the virtual host you want to use:

       $ touch www.example.com:80/.proxy

  3. run php in FastCGI mode (adjust path to PHP as needed):

       $ PHP_FCGI_CHILDREN=16 /opt/php/bin/php-cgi -b 127.0.0.1:8001

  4. tell gatling to use this to run the PHP scripts:

       # gatling -O 'F/127.0.0.1/8001/\.php'

  5. now, you should be able to browse to

     http://www.example.com/t.php

Note that the physical t.php file must exist in your http root.  gatling
checks if it's there and tells php to parse it from there.  This file
needs to be there but it does not need to be world readable.  Gatling
will only serve files that are world readable.  It is thus a good idea
to make the php files only readable to the user or group php runs under,
and not to the world.  That way you cannot accidentally serve them via
gatling.  The same trick goes for include files or other files that the
php scripts may want to read but that do not need to be served by
gatling directly.

PolarSSL is a small SSL/TLS library suitable for embedded use.
You can find it at http://www.polarssl.org/

At the time of this writing (August 2011), the author has added several
patches I suggested in the SVN trunk version, so you need the trunk
version.

To compile polarssl with dietlibc, just go to polarssl/library and do
this:

  $ diet -Os gcc -nostdinc -c *.c -I ../include
  $ ar rsu libpolarssl.a *.o
  $ cp libpolarssl.a `diet -L gcc`
  $ mkdir -p /opt/diet/include/polarssl
  $ cp -p ../include/polarssl/*.h /opt/diet/include/polarssl

After that, in the gatling source tree, run

  $ make ptlsgatling

polarssl support is only lightly tested and should be considered
experimental.  However, here is why you might want to check it out:

-rwxr-xr-x    1 leitner  users      165944 Aug 19 02:57 gatling
-rwxr-xr-x    1 leitner  users     1258176 Aug 19 02:57 tlsgatling
-rwxr-xr-x    1 leitner  users      384584 Aug 19 02:57 ptlsgatling

gatling now has experimental prefetching support.
The code is experimental and enabled per default, disable it with -P0.

The code is only used for downloads larger than 1 MB.  What it does it
mmap and read the next n MB in the file.  Set the value with -P 2M (for
2 MB) or -P 1G (for 1 GB).  Useful range would be (depending on your
RAM and hard disk speed) 1 MB until maybe 10 MB.  The idea is that
modern disks are very fast (50 MB/sec) for linear reading (so
downloading one file is very fast) but they are very slow for moving the
read head (so two people downlading two ISO images is very slow because
the read head is always moving from image A to image B and reading a few
kilobytes).  Normally, the OS should do prefetching, in particular if we
use sendfile (which we do), but I have yet to see an OS that does.  The
solution would be to read larger parts of the file before sending it.
gatling mmaps the files, reads two megabytes (or whatever you
configured), which then stay in the OS buffer cache.  This ought to
reduce head movement.


Please note: since gatling is not threaded, no requests will be
services while gatling is prefetching data.  So if you set the prefetch
too big, gatling will stall during the prefetches.  If you set the
prefetch too small, the OS internal prefetching will prefetch more than
gatling does and thus there will be no effect.  If you serve data from
many different disks of different speed, set the prefetch to a value
that is good for the _slowest_ disk of the pack.


NB: prefetching does not help AT ALL if your files are fragmented on
disk.  This usually happens if you downloaded them with some P2P
application or download manager (BitTorrent 3.3 has some counter
measures here and should be safe).  I'll include the readfrag program
which you can use to defragment files (it will output the file on
stdout, if stdout is a terminal, it will just say how much head movement
it could save compared to a naive OS; Linux 2.6's internal disk I/O
scheduler is good enough so you can just use cp instead of readfrag).
readfrag uses a little documented Linux specific ioctl that is used by
LILO (the boot loader) normally; if you know how to port this to other
OSes, please tell me.

gatling has primitive CGI support which is implemented as a proxy.

The idea is that you specify a regex and an ip and a port.  If the regex
matches the requested URL, gatling will not answer the query itself but
make a TCP connection to the ip and the port.  So, in effect, gatling
can be used to quickly serve static data but let someone else (an Apache
maybe) serve the Java servlets.

In my tests I used fnord as backend to run a CGI program.  This ought to
combine the best of all worlds.

For testing, run gatling like this:

  ./gatling -O 127.0.0.1/8023/cgi$

And then run fnord in the same directory like this:

  tcpserver -v -u `id -u nobody` -RHl localhost 0 8023 fnord-cgi

Another idea I have is to not relay the requests synchronously but
through a request pool.  If there are more requests than slots in the
pool, gatling could prioritize the requests through an external program.
My current idea is that gatling would run an external program (think:
perl) that would get peer IP address and HTTP request on stdin and write
a priority on stdout.

You can have more than one -O statement.  The first matching will always
be used.  So you can have one backend for .jsp and a different one for
.cgi.

This CGI proxy mode will only be used if the file ".proxy" exists in the
root of the virtual server.

gatling can do HTTP redirects.

To do it, create a symlink like this:

  $ ln -s http://www.google.com/ search.html

Then, if someone tries to access search.html via HTTP, gatling will
detect the special broken symlink and send the user a HTTP redirect to
www.google.com.

To redirect an URL ending in /, create that directory and create a
symlink as above for index.html.



UPDATE: gatling can now also do fallback redirects.  If you have a farm
of mirrors, all replicating off a central one, you can tell gatling to
not generate 404s on the mirrors, but redirects.
SUPPORT_FALLBACK_REDIRECT must be #defined in gatling.c for this to be
compiled in.

If, for example, you run on your mirror machine

  gatling -r http://fallback.example.com

and someone requests

  http://mirror.example.com/notyetuploaded.html

gatling will generate a temporary redirect to

  http://fallback.example.com/notyetuploaded.html

gatling now has primitive SSL/TLS support using OpenSSL.
I took the code from the excellent qmail STARTTLS patch.
No support for much of anything yet, you just get an HTTPS server
using the certificate in "server.pem" in the gatling root directory.

If you want OpenSSL to verify client certs, put the CA cert in
"clientca.pem".  If you need a revocation list, use OpenSSL 0.9.7 or
later, and put it in "clientcrl.pem".

No way to communicate anything about the client cert to CGIs yet.


As of Sep 23 2008 gatling has support for ssh passthrough.  The idea is
the following.  Let's assume you run a server somewhere, and you want to
SSH to it, but you only get internet access through some restrictive
proxy firewall that lets you connect to port 443 because that's what
HTTPS uses.  So you bind a ssh to port 443 on your server.  Now you
want to run an SSL webserver, too.  It turns out, you can do both!
For TLS, the client connects and writes something.  For SSH, the client
connects and expects the server to write something.  So, gatling can
accept the connection, attempt an SSL handshake, but if the client does
not write anything for a few seconds, you pass the descriptor on to sshd
running in inetd mode.  That way, you can transparently use both SSL and
SSH on the same port.  You still risk losing SSL connections that come
from very slow connections, so this is not enabled by default.  To
enable it, run tlsgatling with

  -X "2,/opt/diet/sbin/sshd -u0"

where -X is the option to enable this, 2 is the timeout in seconds, and
the rest after the comma is the sshd command line you want gatling to
run.  Note that gatling auto-appends the -i option to this command line,
so you do not need to specify it here.

PLEASE NOTE: if you are planning to run gatling with SSL in chroot mode,
you need to make sure the needed files (by the SSL library) are there.
For example, for openssl, there needs to be a /dev/urandom inside the
chroot jail.  Typical error message:

  ssl_handshake_error 8 error:140B512D:SSL routines:SSL_GET_NEW_SESSION:ssl session id callback failed




UPDATE 2013-07-01: gatling now supports ephemeral Diffie Hellman to
enable Perfect Forward Secrecy.  This, in a nutshell, means that if
someone steals or subpoenas your server to obtain your secret key, they
can still not decrypt any previous data transfers because those used
ephemeral (temporary) keys.  Hint: YOU WANT THIS.  The downside is that
it uses up more CPU time.

You need to do two things to get this to work.  First, generate some
Diffie Hellman parameters, like so:

  openssl gendh -out dhparams.pem -rand - 2048

Append the resulting dhparams.pem to your server.pem file at the end, or
store them in a file called dhparams.pem in the same directory as
server.pem.

Please note that you should also set the OpenSSL ciphers.  As of
20130907 gatling will do this for you if you don't.  The set gatling
uses is

  HIGH:!DSS:!RC4:!MD5:!aNULL:!eNULL:@STRENGTH

Note: As of 20140101 gatling is hard-coded to not allow SSLv2 or SSLv3,
only TLS 1.0 and up.

This cipher string means: Use TLSv1 ciphers with high grade encryption,
do not allow SSLv2 or SSLv3, insist on TLSv1 or better, do not allow cipher
suites without authentication or encryption, and sort by strength so the
strongest common suite between client and server gets selected.

Note that this contains a compromise.  It still allows cipher suites
without perfect forward secrecy.  This means that if somebody steals the
secret key from your server, they can decrypt all earlier intercepted
transmissions that did not have perfect forward secrecy enabled.

You can set ciphers via the environment before running tlsgatling:

  TLSCIPHERS='HIGH:!aNULL:!eNULL:@STRENGTH' tlsgatling [arguments]

If you are using minit and serdo, put this in your script file before
running tlsgatling:

  export TLSCIPHERS=HIGH:!aNULL:!eNULL:@STRENGTH

If you don't want two files lying around, you can also append the
contents of dhparams.pem to your server.pem; gatling will also look
there.

UPDATE 2013-09-07: gatling now defaults to good cipher suites and will
specify a dhparams for you if you don't.  The Diffie Hellman parameters
are not a secret key and no harm ought to come from using the ones
gatling comes with, but it will probably still make you feel better if
you generate your own.

NOTE: If you use a site like ssllabs.com, it will give you a false
positive warning that is not actually true.  It says that the server
does not mitigate the BEAST attack.  If your version of OpenSSL is
current, this is flat out not true.

UPDATE 2014-07-15: The SSL initialization code will now abort if
/dev/urandom can't be opened for reading.  It is a common mistake to
forget to have /dev/urandom in chroot mode (-C option in gatling), and
in that case randomness could be impaired.


UPDATE 2014-10-03: The OpenSSL code now supports SNI. Clients supporting
SNI send the name of the server they are trying to reach as part of the
initial handshake.  OpenSSL can than give gatling a callback, which will
attempt to load the key specified in the handshake.  For example, for
https://localhost/, gatling would attempt to load server.pem (the
default key) and then via SNI callback localhost.pem.  All modern
browsers support it (on Windows you need Vista or newer for IE/Chrome).