WWWOFFLE - World Wide Web Offline Explorer - Version 2.9g
          =========================================================


The WWWOFFLE programs simplify World Wide Web browsing from computers that use
intermittent connections to the internet.

Description
-----------

The WWWOFFLE server is a proxy web server with special features for use with
intermittent internet links.  This means that it is possible to browse web pages
and read them without having to remain connected.

Basic Features
    - Caching of HTTP, FTP and finger protocols.
    - Allows the 'GET', 'HEAD', 'POST' and 'PUT' HTTP methods.
    - Interactive or command line control of online/offline/autodial status.
    - Highly configurable.
    - Low maintenance, start/stop and online/offline status can be automated.

While Online
    - Caching of pages that are viewed for later review.
    - Conditional fetching to only get pages that have changed.
        - Based on expiration date, time since last fetched or once per session.
    - Non cached support for SSL (Secure Socket Layer e.g. https).
    - Caching for https connections. (compile time option).
    - Can be used with one or more external proxies based on web page.
    - Control which pages cannot be accessed.
        - Allow replacement of blocked pages.
    - Control which pages are not to be stored in the cache.
    - Create backups of cached pages when server cannot be contacted.
        - Option to create backup when server sends back an error page.
    - Requests compressed pages from web servers (compile time option).
    - Requests chunked transfer encoding from web servers.

While Offline
    - Can be configured to use dial-on-demand for pages that are not cached.
    - Selection of pages to download next time online
        - Using normal browser to follow links.
        - Command line interface to select pages for downloading.
    - Control which pages can be requested when offline.
    - Provides non-cached access to intranet servers.

Automated Download
    - Downloading of specified pages non-interactively.
    - Options to automatically fetch objects in requested pages
        - Understands various types of pages
            - HTML 4.0, Java classes, VRML (partial), XML (partial).
        - Options to fetch different classes of objects
            - Images, Stylesheets, Frames, Scripts, Java or other objects.
        - Option to not fetch webbug images (images of 1 pixel square).
    - Automatically follows links for pages that have been moved.
    - Can monitor pages at regular intervals to fetch those that have changed.
    - Recursive fetching
        - To specified depth.
        - On any host or limited to same server or same directory.
        - Chosen from command line or from browser.
        - Control over which links can be fetched recursively.

Convenience
    - Optional information footer on HTML pages showing date cached and options.
    - Options to modify HTML pages
        - Remove scripts.
        - Remove Java applets.
        - Remove stylesheets.
        - Remove shockwave flash animations.
        - Indicate cached and uncached links.
        - Remove the blink tag.
        - Remove the marquee tag.
        - Remove refresh tags.
        - Remove links to pages that are in the DontGet list.
        - Remove inline frames (iframes) that are in the DontGet list.
        - Replace images that are in the DontGet list.
        - Replace webbug images (images of 1 pixel square).
        - Demoronise HTML character sets.
        - Fix mixed Cyrillic character sets.
        - Stop animated GIFs.
        - Remove Cookies in meta tags.
    - Provides information about cached pages
        - Headers, raw and modified.
        - Contents, images, links etc.
        - Source code unmodified by WWWOFFLE.
    - Automatic proxy configuration with Proxy Auto-Config file.
    - Searchable cache with the addition of the ht://Dig, mnoGoSearch
      (UdmSearch), Namazu or Hyper Estraier programs.
    - Built in simple web-server for local pages
        - HTTP and HTTPS access (compile time option).
        - Allows CGI scripts.
    - Timeouts to stop proxy lockups
        - DNS name lookups.
        - Remote server connection.
        - Data transfer.
    - Continue or stop downloads interrupted by client.
        - Based on file size of fraction downloaded.
    - Purging of pages from cache
        - Based on URL matching.
        - To keep the cache size below a specified limit.
        - To keep the free disk space above a specified limit.
        - Interactive or command line control.
        - Compression of cached pages based on age.
    - Provides compressed pages to web browser (compile time option).
    - Use chunked transfer-encoding to web browser.

Indexes
    - Multiple indexes of pages stored in cache
        - Servers for each protocol (http, ftp ...).
        - Pages on each server.
        - Pages waiting to be fetched.
        - Pages requested last time offline.
        - Pages fetched last time online.
        - Pages monitored on a regular basis.
    - Configurable indexes
        - Sorted by name, date, server domain name, type of file.
        - Options to delete, refresh or monitor pages.
        - Selection of complete list of pages or hide un-interesting pages.

Security
    - Works with pages that require basic username/password authentication.
    - Automates proxy authentication for external proxies that require it.
    - Control over access to the proxy
        - Defaults to local host access only.
        - Host access configured by hostname or IP address.
        - Optional proxy authentication for user level access control.
    - Optional password control for proxy management functions.
    - HTTPS access to all proxy management web pages (compile time option).
    - Can censor incoming and outgoing HTTP headers to maintain user privacy.

Configuration
    - All options controlled using a configuration file.
    - Interactive web page to allow editing of the configuration file.
    - User customisable error and information pages.
    - Log file or syslog reporting with user specified error level.


Configuring A Web Browser
-------------------------

To use the WWWOFFLE programs, requires that your web browser is set up to use it
as a proxy.  The proxy hostname will be 'localhost' (or the name of the host
that wwwoffled is running on), and the port number will be the one that is used
by wwwoffled (default 8080).

There are lots of different browsers and it is not possible to list all the ways
to configure them here.  There should be an option in one of the menus or
described in the manual for the browser that explains how to configure a proxy.

You will also need to disable the caching that the web browser performs itself
between sessions to get the best out of the program.

Depending on which browser you use and which version, it is possible to request
pages to be refreshed while offline.  This is done using the 'reload' or
'refresh' button or key on the browser.  On many browsers, there are two ways of
doing this, one forces the proxy to reload the page, and this is the one that
will cause the page to be refreshed.


Welcome Page
------------

There is a welcome page at URL 'http://localhost:8080/' that gives a very brief
description of the program and has links to the index pages, interactive control
page and the WWWOFFLE internet home pages.

The most important places to get information about WWWOFFLE are the WWWOFFLE
homepage which has information about WWWOFFLE in general:

http://www.gedanken.org.uk/software/wwwoffle/


Index Of Cached Files
---------------------

To get the index of cached files, use the URL 'http://localhost:8080/index/'.
There are sufficient links on each of the index pages to allow easy navigation.

The indexes provides several levels of information:
   A list of the requests in the outgoing directory.
   A list of the files fetched the last time that the program was online.
      And for the previous 5 times before that.
   A list of the files requested the last time that the program was offline.
      And for the previous 5 times before that.
   A list of the files that are being monitored.
   A list of all hosts for each of the protocols (http,ftp etc.).
      A list of all of the files on a particular host.

These indexes can be sorted in a number of ways:
   No sorting (directory order on disk).
   By time of last modification (update).
   By time of last access.
   By date of last update with markers for each day.
   Alphabetically.
   By file extension.
   Random.

For each of the pages that are cached there are options to delete the page,
refresh it, select the interactive refresh page with the URL already filled in
or add the page to the list that is monitored regularly.

It is also possible to specify in the configuration file what URLs are not to be
listed in the indexes.


Interactive Refresh Page
------------------------

Pages can be specified by using whatever method is provided by the browser that
is used or as an alternative there is an interactive refresh page.  This allows
the user to enter a URL and then fetch it if it is not currently cached or
refresh it if it is in the cache.  There is also the option here to recursively
fetch the pages that are linked to by the page that is specified.  This
recursive fetching can be limited to pages from the same host, narrowed down to
links in the same directory (or subdirectory) or widened to fetch pages from any
web server.  This functionality is also provided in the 'wwwoffle' command line
program.


Monitoring Web-Pages
--------------------

Pages can be specified that are to be checked at regular intervals.  This can
either be every time that WWWOFFLE is online or at user specifiable times.  The
page will be monitored when the four specified conditions are all met:
A month of the year that it can be fetched in (can be set to all months).
A day of the month that the page can be fetched on (can be set to all days).
A day of the week that the page can be fetched on (can be set to all days).
An hour of the day that the page should be fetched after (can be more than one).

For example to get a URL every Saturday morning, use the following:

Month of year: all
Day of Month : all
Day of week  : Saturday
Hour of day  : 0 (24hr clock)


Interactive Control Page
------------------------

The behaviour and mode of operation of the WWWOFFLE daemon can be controlled from
an interactive control page at 'http://localhost:8080/control/'.  This has a
number of buttons that change the mode of the proxy server.  These provide the
same functionality as the 'wwwoffle' command line program.  To provide security,
this page can be password protected.  There is also the facility to delete pages
from the cache or from the spooled outgoing requests directory.


Interactive Configuration File Editing Page
-------------------------------------------

The interactive configuration file editing page allows the configuration file
wwwoffle.conf to be edited.  This facility can be reached via the configuration
editing page 'http://localhost:8080/configuration/'.  Each item in the
configuration file has a separate web-page with a form in it that lists the
current entries in the configuration file and allows each entry to be edited
individually.  When an entry has been updated, the configuration file needs to
be re-read.


Searching the Cache
-------------------

The three web indexing programs ht://Dig, mnoGoSearch (UdmSearch), Namazu or
Hyper Estraier can be used to create an index of the pages in the WWWOFFLE cache
for later searching.

For ht://Dig version 3.1.0b4 or later is required, it can be found at
http://www.htdig.org/.

For mnoGoSearch (previously called UdmSearch) version 3.1.0 or later is
required, it can be found at http://mnogosearch.org/.

For Namazu version 2.0.0 or later is required, it can be found at
http://www.namazu.org/, also required is mknmz-wwwoffle which can be found at
http://www.naney.org/comp/distrib/mknmz-wwwoffle/.

For Hyper Estraier version 0.5.3 or later is required, it can be found at
http://hyperestraier.sourceforge.net/.

The search forms for these programs are 'http://localhost:8080/search/htdig/',
'http://localhost:8080/search/mnogosearch/',
'http://localhost:8080/search/namazu/', and
'http://localhost:8080/search/hyperestraier/'.  These allow the search part of
the programs to be run to find the cached web-pages that you want.

For more information about configuring these programs to work with WWWOFFLE you
should read the file README.htdig, README.mnogosearch, README.namazu, or
README.hyperestraier.


Built-In Web-Server
-------------------

Any URLs to WWWOFFLE on port 8080 that refer to files in the directory '/' refer
to the files that are stored in the 'html' subdirectory.  This directory also
contains the message templates that WWWOFFLE uses to generate the internal web
pages.  When a file is requested from either of these locations it is first
looked for in the language specific sub-directory specified in the browser's
request header.  If it is not found in that location then it is looked for in
the directory named 'default' which by default is a symbolic link to the English
language pages, but can be changed.  If it is not found in this location then it
is looked for in the English language directory (since that will have a full set
of pages).

Any URLs that refer to files in the directory '/local/' are taken from the files
in the 'local' sub-directory of the spool directory if they exist.  If they do
not exist then they are searched for in the language subdirectories of the
'html' directory as described above.  This allows for trivial web-pages to be
provided without a separate web-server.  CGI scripts are available but disabled
by the default configuration file.  The MIME type used for these files are those
that are specified in the configuration file.

Important: The local web-page server will follow symbolic links, but will only
           allow access to files that are world readable.  See the FAQ for
           security issues.


Deleting Requests
-----------------

If no password is used for the control pages then it is possible for anybody to
delete requests that are recorded.  If a password is assigned then users that
know this password can delete any request (or cached file or other thing).
Individual users that do not know the password can delete pages that they have
requested provided that they do it immediately that the "Will Get" page appears,
the "Delete" button on here has a once-only password that will delete that
request.


Backup Copies of Pages
----------------------

When a page is fetched while online a remote server error will overwrite any
existing web page.  In this case a backup copy of the page is made so that when
the error message has been read while offline the backup copy is placed back
into the cache.  This is automatic for all cases of files that have remote
server errors (and that do not use external proxies), no user intervention is
required.


Lock Files
----------

When one WWWOFFLE process is downloading a file any other WWWOFFLE process that
tries to read that file will not be able to until the first one has finished.
This removes the problem of an incomplete page being displayed in the second
browser, or a second copy of the page being fetched.  If the lock file is not
removed by the first process within a timeout period then the second process
will produce an error message indicating the problem.

This is now a configurable option, the default condition is that lock files are
not used.


HTTPS Access to Internal Pages
------------------------------

All of the web pages that are available through normal HTTP access on port 8080
(e.g. http://localhost:8080/*) are also available with secure HTTPS access on
port 8443 if WWWOFFLE is compiled with the libgnutls encryption library.  This
applies to all pages; indexes, built-in web server and control and configuration
pages.


Caching of HTTPS Web Pages
--------------------------

It is possible to configure WWWOFFLE so that it will intercept and cache
selected HTTPS connections.  This is disabled by default and there are three
steps to enable it.  WWWOFFLE must be compiled with encryption support, the
enable-caching option in the SSLOptions section of the configuration file must
be set true and the list of hosts to cache for must be set.

When WWWOFFLE is configured to cache an HTTPS web page it will request the page,
decrypt it, re-encrypt it and pass it to the browser.  The copy of the page that
is stored in the cache will be stored without encryption.  With this option all
other WWWOFFLE features like the DontGet section, the ModifyHTML section, the
OnlineOptions and others will be used.  Normally most of these options cannot be
applied to HTTPS pages because the exact URL is not known to WWWOFFLE and the
unencrypted contents are not visible.


HTTPS Server Certificates
-------------------------

To handle the encryption functions described above WWWOFFLE will create and
manage a set of server certificates.  One master certificate is used to sign all
of the other certificates that WWWOFFLE creates.  The created certificates are
either for the WWWOFFLE server HTTPS access pages or for a fake certificate that
is created for each server that is cached.  The certificates that are captured
by WWWOFFLE and stored are the certificates that are sent back by the real HTTPS
server.  The final set of certificates are the trusted certificates that
WWWOFFLE can use to confirm that the remote server is the one it claims to be.

The full set of certificates that WWWOFFLE stores can be seen through the
WWWOFFLE URL http://localhost:8080/certificates/ but is only available if
WWWOFFLE was compiled with encryption support.

To add trusted certificates to WWWOFFLE place the certificate file (in PEM
format) into the directory '/var/spool/wwwoffle/certificates/trusted' and
restart WWWOFFLE.


Spool Directory Layout
----------------------

In the spool directory there is a directory for each of the network protocols
that are handled.  In this directory there is a directory for each hostname that
has been contacted and has pages cached.  These directories have the name of the
host.  In each of these directories, there is an entry for each of the pages
that are cached, generated using a hashing function to give a constant length.
The entry consists of two files, one prefixed with 'D' that contains the data
and one prefixed with 'U' that contains the URL.

The outgoing directory is a single directory that all of the pending requests
are contained in, the format is the same with two files for each, but using 'O'
for the file containing the request instead of 'D' and one prefixed with 'U'
that contains the URL.

The lasttime (and prevtime*) directories are a single directory that contains an
entry for each of the files that were fetched the last time that the program was
online.  Each entry consists of two files, one prefixed with 'D' that is a
hard-link to the real file and one prefixed with 'U' that contains the URL.

The lastout (and prevout*) directories are a single directory that contains an
entry for each of the files that were requested the last time that the program
was offline.  Each entry consists of two files, one prefixed with 'D' that is a
hard-link to the real file and one prefixed with 'U' that contains the URL.

The monitor directory is a single directory that all of the regularly monitored
requests are contained in, the format is the same as the outgoing directory with
two files for each, using 'O' and 'U' prefixes.  There is also a file with an
'M' prefix that contains the information about when to monitor the URL.


The Programs and Configuration File
-----------------------------------

There are two programs that make up this utility, with three distinct functions.

wwwoffle  - A program to interact with and control the HTTP proxy daemon.

wwwoffled - A daemon process that acts as an HTTP proxy.
wwwoffles - A server that actually does the fetching of the web pages.

The wwwoffles function is combined with the wwwoffled function into the
wwwoffled program from version 1.1 onwards.  This is to simplify the procedure
of starting servers, and allow for future improvements.

The configuration file, called wwwoffle.conf by default contains all of the
parameters that are used to control the way the wwwoffled and wwwoffles
functions work.  The default installation location for this file is in the
directory /etc/wwwoffle.


WWWOFFLE - User control program
-------------------------------

The control program (wwwoffle) is used to control the action of the daemon
program (wwwoffled), or to request pages that are not in the cache.

The daemon program needs to know if the system is online or offline, when to
fetch the pages that have been previously requested and when to purge the cache
of old pages.


The first mode of operation is for controlling the daemon process.  These are the
functions that are also available on the interactive control page (except kill).

wwwoffle -online        Indicates to the daemon that the system is online.

wwwoffle -autodial      Indicates to the daemon that the system is in autodial
                        mode, this will use cached pages if they exist and use
                        the network as last resort, for dial-on-demand systems.

wwwoffle -offline       Indicates to the daemon that the system is offline.

wwwoffle -fetch         Commands the daemon to fetch the pages that were
                        requested by clients while the system was offline.
                        wwwoffle exits when the fetching is complete.
                        (This requires the daemon to be told it is online).

wwwoffle -config        Cause the configuration file for the daemon process to be
                        re-read.  The config file can also be re-read by sending
                        a HUP signal to the wwwoffled process.

wwwoffle -purge         Commands the daemon to purge from the cache the pages
                        that are older than the number of days specified in the
                        configuration file, using modification or access
                        time. Or if a maximum size is specified then delete the
                        oldest pages until the maximum size is not exceeded.

wwwoffle -status        Request from the wwwoffled proxy server the current
                        status of the program.  The online or offline mode, the
                        fetch and purge statuses, the number of current
                        processes and their PIDs are displayed.

wwwoffle -kill          Causes the daemon to exit cleanly at a convenient point.


The second mode of operation is to specify URLs to get.

wwwoffle <URL> .. <URL> Specifies to the daemon URLs that must be fetched.
                        If online then it is got immediately, else the request
                        is stored for a later fetch.

wwwoffle <filename> ... The specified HTML file is be read and all of the links
                        in it used as if they had been specified on the command
                        line.

wwwoffle -post <URL>    Send a request using the POST method, the data is read
                        from stdin and should be provided correctly url-encoded.

wwwoffle -put <URL>     Send a request using the PUT method, the data is read
                        from stdin and should be provided correctly url-encoded.


wwwoffle -F             Force the wwwoffle server to refresh the URL.
                        (Or fetch it if not cached.)

wwwoffle -g[Sisfo]      Specifies that the URLs when fetched are to be parsed
                        for Stylesheets (s), images (i), scripts (s), frames (f)
                        or objects (o) and these are also to be fetched.  Using
                        -g without any following letters will get none of them.

wwwoffle -r[<depth>]    Specifies that the URL when fetched is to have the links
                        followed and these pages also fetched (to a depth
                        specified by the optional depth parameter, default 1).
                        Only links on the same server are to be fetched.

wwwoffle -R[<depth>]    This is the same as the '-r' option except that all of
                        the links are to be followed, even those to other
                        servers.

wwwoffle -d[<depth>]    This is the same as the '-r' option except that links
                        are only followed if they are in the same directory or a
                        sub-directory.

                        (If the -F, -(d|r|R) or -g[Sisfo] options are set they
                        override the options in the FetchOptions section of the
                        config file and only the -g[Sisfo] options are fetched.)


The third mode of operation is to get a URL from the cache.

wwwoffle <URL>          Specifies the URL to get.

wwwoffle -o             Get the URL and output it on the standard output.
                        (Or request it if not already cached.)

wwwoffle -O             Get the URL and output it on the standard output
                        including the HTTP headers.
                        (Or request it if not already cached.)


The last mode of operation is to provide help in using the other modes.

wwwoffle -h             Gives help about the command line options.


With any of the first three modes of operation the WWWOFFLE server can be
specified in one of three different ways.

wwwoffle -c <config-file>
                        Can be used to specify the configuration file that
                        contains the port numbers, server hostname (the first
                        entry in the LocalHost section) and the password (if
                        required for the first mode of operation).  If there is
                        a password then this is the only way to specify it.

wwwoffle -p <host>[:<port>]
                        Can be used to specify the hostname and port number that
                        the daemon program listens to for control messages (first
                        mode) or proxy connections (second and third modes).

WWWOFFLE_PROXY          An environment variable that can be used to specify
                        either the argument to the -c option (must be the full
                        pathname) or the argument to the -p option.  (In this
                        case two ports can be specified, the first for the proxy
                        connection, the second for the control connection
                        e.g. 'localhost:8080:8081' or 'localhost:8080'.)


WWWOFFLED - Daemon program
--------------------------

The daemon program (wwwoffled) runs as an HTTP proxy and also accepts
connections from the control program (wwwoffle).

The daemon program needs to maintain the current state of the system, online or
offline, as well as the other parameters in the configuration file.

As HTTP proxy requests come in, the program forks a copy of itself (the
wwwoffles function) to handle the requests.  The server program can also be
forked in response to the wwwoffle program requesting pages to be fetched.


wwwoffled -c <config-file>      Starts the daemon with the named configuration
                                file.

wwwoffled -d [level]            Starts the daemon in debugging mode, i.e it does
                                not detach from the terminal and uses standard
                                error for the log messages.  The optional
                                numeric level (0 for none to 5 for all or 6 for
                                more) specifies the level of error messages for
                                standard error, if not specified then use
                                log-level from the config file.

wwwoffled -f                    Start the daemon in debugging mode (implies -d)
                                and when the first HTTP request comes in handle
                                it without creating a child process and then
                                exit.

wwwoffled -p                    Print the PID of the daemon on standard out
                                before detaching from the terminal.

wwwoffled -h                    Gives help about the command line options.


There are a number of error and informational messages that are generated by the
program as it runs.  By default (in the config file) these go to syslog, by
using the -d flag the daemon does not detach from the terminal and the errors
are also on standard error.

By using the run-uid and run-gid options in the config file, it is possible to
change the user id and group id that the program runs as.  This will require
that the program is started by root and that the specified user has read/write
access to the spool directory.


WWWOFFLES - Server program
--------------------------

The server (wwwoffles) starts by being forked from the daemon (wwwoffled) in one
of three different modes.

Real  - When the system is online and acting as a proxy for a client.
        All requests for web pages are handled by forking a new server which
        will connect to the remote host and fetch the page.  This page is then
        stored in the cache as well as being returned to the client.  If the
        page is already in the cache then the remote server is asked for a newer
        page if one exists, else the cache one is used.

SpoolOrReal - When the system is in autodial mode and we have not decided if we
        will go for Spool or Real mode.  Select Spool mode if already cached and
        Real mode otherwise as a last resort.

Fetch - When the system is online and fetching pages that have been requested.
        All web page requests in the outgoing directory are fetched by the
        server connecting to the remote host to get the page.  This page is then
        stored in the cache, there is no client active.  If the page has been
        moved then the link is followed and that one fetched.

Spool - When the system is offline and acting as a proxy for a client.
        All requests for web pages are handled by forking a server that will
        either return a cached page or store the request.  If the page is
        cached, it is returned to the client, else a dummy page is returned
        (and stored in the cache), and the outgoing request is stored.
        If the cached page refers to a page that failed to be downloaded then it
        will be deleted from the cache.

Depending on the existence of files in the spool and other conditions, the mode
can be changed to one of several other modes.

RealNoCache - For requests for pages on the server machine or those specified
        not to be cached in the configuration file.

RealRefresh - Used by the refresh button on the index or the wwwoffle program
        to re-fetch a page while the system is online.

RealNoPassword - Used when a password was provided and two copies of the page
        are required, one with and one without the password.

FetchNoPassword - Used when a password was provided and two copies of the page
        are required, one with and one without the password.

SpoolGet - Used when the page does not exist in the cache so a request needs to
        be stored for it in the outgoing directory.

SpoolRefresh - Used when the refresh button on the index or the wwwoffle program
        are used, the existing spooled page (if there is one) is not
        overwritten, but a request is stored.

SpoolPragma - Used when the client requests the cache to refresh the page
        using the 'Pragma: no-cache' header, the existing spooled page (if there
        is one) is not overwritten, but a request is stored.

InternalPage - Used when the program is generating a web-page internally or is
        spooling a web-page with modifications.


WWWOFFLE-TOOLS - Cache maintenance program
------------------------------------------

This is a quick hack program that I wrote to allow you to list the contents of
the cache or move files around in it.  The programs are all named after common
UNIX commands with a 'wwwoffle-' prefix.

All of the programs should be invoked from the spool directory.

wwwoffle-rm     - Delete the URL that is specified on the command line.
                  To delete all URLs from a host it is easier to use
                  'rm -r http/foo' than use this.

wwwoffle-mv     - To rename URLs under one path in the spool to another path.
                  Because the URL is encoded in the filename just renaming the
                  files or the directory will not work.  Instead of 'mv http/foo
                  http/bar' use 'wwwoffle-mv http/foo http/bar'.  Also works for
                  complex cases: 'wwwoffle-mv http://foo/bar http:///bar/foo'.

wwwoffle-ls     - To list a cached URL or the files in a cache directory in the
                  style of 'ls -l'.  As examples use 'wwwoffle-ls http/foo' to
                  list all of the URLs in the cache directory 'http/foo', use
                  'wwwoffle-ls http://foo/' to list the single URL 'http://foo/'
                  or use 'wwwoffle-ls outgoing' to list the outgoing requests.

wwwoffle-read   - Read data directly from the cache for the URL named on the
                  command line and output it on stdout.

wwwoffle-write  - Write data directly to the cache for the URL named on the
                  command line from stdin.  Note this requires a HTTP header to
                  be included first or clients may get confused.
                  (echo "HTTP/1.0 200 OK" ; echo "" ; cat bar.html ) | \
                  wwwoffle-write http://www.foo/bar.html

wwwoffle-hash   - Print WWWOFFLE's encoding of the submitted URL. This is
                  useful for scripts working on the WWWOFFLE cache.

wwwoffle-fsck   - Checks the WWWOFFLE cache for consistency, it will rename any
                  files where the filename does not match the hash of the URL.

wwwoffle-gzip   - Compress the contents of the cache so that they take less
                  space but WWWOFFLE can still read them.

wwwoffle-gunzip - Uncompress the contents of the cache.

All of the programs are the same executable and the name of the file determines
the function.  The wwwoffle-tools executable can also be used with a command
line parameter, for example 'wwwoffle-ls' is the same as 'wwwoffle-tools -ls'.

This program also accepts the '-c <config-file>' option and uses the
'WWWOFFLE_PROXY' environment variable so that the wwwoffle-write program uses
the correct permissions and uid/gid.

These are basically hacks and should not be considered as fully featured and
fully debugged programs.


audit-usage.pl - Perl script to check log files
-----------------------------------------------

The audit-usage.pl script (in the contrib directory) can be used to get audit
information from the output of the wwwoffled program.

If wwwoffled is started as

wwwoffled -c /etc/wwwoffle/wwwoffle.conf -d 4

Then on the standard error output will be generated information about the
program as it is run.  The debug level needs to be 4 so that the URL information
is output.

If this is captured into a log file then it can be analysed by the
audit-usage.pl program.  When run this will tell the host that the connection is
made from and the URL that is requested.  It also includes the timestamp
information and connections to the WWWOFFLE control connection.


Test Programs
-------------

In the testprogs directory are two test programs that can be compiled if
required.  They are not needed for WWWOFFLE to work, but if you are customising
the information pages for WWWOFFLE to use or trying to debug the HTML parser
then they will be of use.

These are even more hacks than the wwwoffle-tools programs, use at your own risk.


Author and Copyright
--------------------

The two programs wwwoffle and wwwoffled were written and are copyright by Andrew
M. Bishop 1996-2011.

The programs known as wwwoffle-tools were written and a copyright by Andrew
M. Bishop 1997-2011.

The Perl scripts update-config.pl and audit-usage.pl were written and are
copyright by Andrew M. Bishop 1998-2011.

They can be freely distributed according to the terms of the GNU General Public
License (see the file `COPYING').


Ht://Dig
- - - -

The htdig package is copyrighted by Andrew Scherpbier. The icons in the
html/search/htdig directory come from htdig as does the search form
html/search/htdig/search.html and configuration files in search/htdig/conf/*
(with modifications by myself).


mnoGoSearch (UdmSearch)
- - - - - - - - - - - -

The mnoGoSearch package is copyrighted by Lavtech.Com Corp and released under
the GPL.  The mnoGoSearch icon in the html/search/mnogosearch directory comes
from mnoGoSearch as does the search form html/search/mnogosearch/search.html and
configuration files in search/mnogosearch/conf/* (with modifications by myself).


Namazu
- - -

The Namazu package is copyrighted by the Namazu Project and mknmz-wwwoffle is
copyrighted by WATANABE Yoshimasa, both programs are released under the GPL.
The configuration files in search/namazu/conf/* come from Namazu (with
modifications by myself).


Hyper Estraier
- - - - - - -

The Hyper Estraier package is copyrighted by Mikio Hirabayashi and is released
under the LGPL.  The configuration files in search/hyperestraier/conf/* come
from Hyper Estraier (with modifications by myself).


With Source Code Contributions From
- - - - - - - - - - - - - - - - - -

Yannick Versley <sa6z225@public.uni-hamburg.de>
        Initial syslog code (much rewritten before inclusion).

Axel Rasmus Wienberg <2wienbe@informatik.uni-hamburg.de>
        Code to run wwwoffled as a specified uid/gid.

Andreas Dietrich <quasi@baccus.franken.de>
        Code to detach the program from the terminal like a *real* daemon.

Ullrich von Bassewitz <uz@wuschel.ibb.schwaben.com>
        Better handling of signals.
        Optimisation of the file handling in the outgoing directory.
        The log-level, max-servers and max-fetch-servers config options.

Tilman Bohn <tb@bohn.isdn.uni-heidelberg.de>
        Autodial mode.

Walter Pfannenmueller <pfn@online.de>
        Document parsing Java/VRML/XML some HTML.

Ben Winslow <rain@insane.loonybin.net>
        Configuration file DontGet section optional replacement Url.
        New FTP commands to get file size and modification time.

Ingo Kloecker <kloecker@math.u-bordeaux.fr>
        Disable animated GIFs (code now removed and rewritten).

David McNab <david@rebirthing.co.nz>
        Workaround winsock bug for cygwin (now lingering close on all systems).

Olaf Buddenhagen <olafbuddenhagen@web.de>
        A patch to do random sorting in the indexes.

Jan Lukoschus <jan+wof@lukoschus.de>
        The patch for wwwoffle-hash (for wwwoffle-tools).

Paul A. Rombouts <p.a.rombouts@home.nl>
        The patch to force re-requests of redirection URLs.
        The patch to allow wildcards to have more than two '*' characters.
        The patch to allow local CGI scripts to be run.
        The patch to keep the backup copy of a page in case of server error.

Marc Boucher <MarcB@box100.com>
        The patch to perform case insensitive matching of URL-SPECs.
        The patch to handle FTP requests made with a password (like HTTP).

Ilya Dogolazky <ilyad@math.uni-bonn.de>
        The patch for the fix-mixed-cyrillic option.

Dieter <netbsd@sopwith.solgatos.com>
        A patch with some 64-bit/32-bit compatibility fixes (that prompted me to
        go and find and fix a lot more).

Andreas Mohr <andi@rhlx01.fht-esslingen.de>
        A patch to add "const" to lots of structures and function parameters
        (this prompted me to go and do a lot more).

Nils Kassube <kassube@gmx.net>
        The patch for the referer-from option.


And Other Useful Contributions From
- - - - - - - - - - - - - - - - - -

Too many people to mention - (everybody that e-mailed me).
        Suggestions and bug reports.

          WWWOFFLE - World Wide Web Offline Explorer - Version 2.9
          ========================================================


There are several help files, this is a meta-help file that will point you to
the correct one to read.


If you want to know what WWWOFFLE is and does:

doc/README


If you have not used WWWOFFLE before:

doc/README
doc/INSTALL
doc/README.lang             [Non-English versions]
doc/README.CONF
doc/README.win32            [Win32 users]


If you have used WWWOFFLE before:

doc/NEWS
doc/CHANGES.CONF
doc/README.CONF


If you want to know how to install the program or change compile time options:

doc/INSTALL
doc/README.lang             [Non-English versions]
doc/README.htdig
doc/README.mnogosearch
doc/README.namazu
doc/README.hyperestraier
doc/README.win32            [Win32 users]


If you want to know how to use the programs when installed:

wwwoffle(1)                 [UNIX users]
wwwoffled(8)                [UNIX users]
doc/README
doc/README.htdig
doc/README.mnogosearch
doc/README.namazu
doc/README.hyperestraier


If you want to know how to configure the program:

wwwoffle.conf               [UNIX users]
wwwoffle.conf(5)            [UNIX users]
doc/README.CONF
doc/CHANGES.CONF
doc/README.htdig
doc/README.mnogosearch
doc/README.namazu
doc/README.hyperestraier


If you want to customise the error message and other pages:

html/$LANG/messages/README


If you want to do more with WWWOFFLE:

contrib/README          [UNIX users]
contrib-win32/README    [Win32 users]



The location and purpose of the sources of information are as listed below.

Files in the doc directory (see also the language specific subdirectories):

ANNOUNCE           The announcement release for this version of WWWOFFLE.

COPYING            The WWWOFFLE licence (the GNU General Public Licence).

NEWS               A list of the changes in functionality in previous versions.

INSTALL            Help on installing the program and the compile-time options.

README             The main README file, usual README stuff about the program.
README.1st         This file - a meta-index.

README.CONF        Help on the format of the config file.
CHANGES.CONF       A list of the config file changes from version 1.3 to now.

README.PWD         Information about how password protected pages are stored.
README.URL         Information about how URLs are processed (decoding/encoding).
README.compress    Information about the problems with using cache compression.
README.https       Information about https, SSL/TLS, certificates and trust.

README.lang        A description of the non-English language web-page versions.

README.htdig         How to use ht://Dig to search the cache.
README.mnogosearch   How to use mnoGoSearch to search the cache.
README.namazu        How to use namazu to search the cache.
README.hyperestraier How to use Hyper Estraier to search the cache.

README.win32       Information about the Win32 version of WWWOFFLE.


Manual pages (installed names listed, files are in doc directory):

wwwoffle(1)        The wwwoffle user interface program UNIX manual page.
wwwoffle.conf(5)   The wwwoffle.conf configuration file UNIX manual page.
wwwoffled(8)       The wwwoffled server demon program UNIX manual page.


Other files:

html/$LANG/messages/README Information on how to customise the error messages.

contrib/README        Some various scripts for use with WWWOFFLE.
contrib-win32/README  Some various scripts for use with WWWOFFLE.

              WWWOFFLE - Configuration File - Version 2.9
              ===========================================

Introduction
------------

The configuration file (wwwoffle.conf) specifies all of the parameters that
control the operation of the proxy server.  The file is split into sections
each containing a series of parameters as described below.  The file
CHANGES.CONF explains the changes in the configuration file between this
version of the program and previous ones.

The file is split into sections, each of which can be empty or contain one or
more lines of configuration information.  The sections are named and the order
that they appear in the file is not important.

The general format of each of the sections is the same.  The name of the
section is on a line by itself to mark the start.  The contents of the section
are enclosed between a pair of lines containing only the '{' and '}'
characters or the '[' and ']' characters.  When the '{' and '}' characters are
used the lines between contain configuration information.  When the '[' and
']' characters are used then there must only be a single non-empty line
between them that contains the name of a file (in the same directory)
containing the configuration information for the section.

Comments are marked by a '#' character at the start of the line and they are
ignored.  Blank lines are also allowed and ignored.

The phrases URL-SPECIFICATION (or URL-SPEC for short) and WILDCARD have
specific meanings in the configuration file and are described at the end.  Any
item enclosed in '(' and ')' in the descriptions means that it is a parameter
supplied by the user, anything enclosed in '[' and ']' is optional, the '|'
symbol is used to denote alternate choices.  Some options apply to specific
URLs only, this is indicated by having a URL-SPECIFICATION enclosed between
'<' & '>' in the option, the first URL-SPECIFICATION to match is used.  If no
URL-SPECIFICATION is given then it matches all URLs.

--------------------------------------------------------------------------------

StartUp
-------

This contains the parameters that are used when the program starts, changes to
these are ignored if the configuration file is re-read while the program is
running.

bind-ipv4 = (hostname) | (ip-address) | none
        Specify the hostname or IP address to bind the HTTP proxy and WWWOFFLE
        control port sockets to using IPv4 (default='0.0.0.0').  If 'none' is
        specified then no IPv4 socket is bound.  If this is changed from the
        default value then the first entry in the LocalHost section may need
        to be changed to match.

bind-ipv6 = (hostname) | (ip-address) | none
        Specify the hostname or IP address to bind the HTTP proxy and WWWOFFLE
        control port sockets to using IPv6 (default='::').  If 'none' is
        specified then no IPv6 socket is bound.  This requires the IPv6
        compilation option.  If this is changed from the default value then
        the first entry in the LocalHost section may need to be changed to
        match.

http-port = (port)
        An integer specifying the port number for connections to access the
        internal WWWOFFLE pages and for HTTP/HTTPS/FTP proxying
        (default=8080).  This is the port number that must be specified in the
        client to connect to the WWWOFFLE proxy for HTTP/HTTPS/FTP proxying.

https-port = (port)
        An integer specifying the port number for encrypted connections to
        access the internal WWWOFFLE pages and for HTTP/FTP proxying
        (default=8443).  Requires gnutls compilation option.

wwwoffle-port = (port)
        An integer specifying the port number for the WWWOFFLE control
        connections to use (default=8081).

spool-dir = (dir)
        The full pathname of the top level cache directory (spool directory)
        (default=/var/spool/wwwoffle or whatever was used when the program was
        compiled).

run-uid = (user) | (uid)
        The username or numeric uid to change to when the WWWOFFLE server is
        started (default=none).  This option only works if the server is
        started by the root user on UNIX-like systems.

run-gid = (group) | (gid)
        The group name or numeric gid to change to when the WWWOFFLE server is
        started (default=none).  This option only works if the server is
        started by the root user on UNIX-like systems.

use-syslog = yes | no
        Whether to use the syslog facility for messages or not (default=yes).

password = (word)
        The password used for authentication of the control pages, for
        deleting cached pages etc (default=none).  For the password to be
        secure the configuration file must be set so that only authorised
        users can read it.

max-servers = (integer)
        The maximum number of server processes that are started for online and
        automatic fetching (default=8).

max-fetch-servers = (integer)
        The maximum number of server processes that are started to fetch pages
        that were marked in offline mode (default=4).  This value must be less
        than max-servers or you will not be able to use WWWOFFLE interactively
        online while fetching.

--------------------------------------------------------------------------------

Options
-------

Options that control how the program works.

log-level = debug | info | important | warning | fatal
        The minimum log level for messages in syslog or stderr
        (default=important).

socket-timeout = (time)
        The time in seconds that WWWOFFLE will wait for data on a socket
        connection before giving up (default=120).

dns-timeout = (time)
        The time in seconds that WWWOFFLE will wait for a DNS (Domain Name
        Service) lookup before giving up (default=60).

connect-timeout = (time)
        The time in seconds that WWWOFFLE will wait for the socket connection
        to be made before giving up (default=30).

connect-retry = yes | no
        If a connection cannot be made to a remote server then WWWOFFLE should
        try again after a short delay (default=no).

dir-perm = (octal int)
        The directory permissions to use when creating spool directories
        (default=0755).  This option overrides the umask of the user and must
        be in octal starting with a '0'.

file-perm = (octal int)
        The file permissions to use when creating spool files (default=0644).
        This option overrides the umask of the user and must be in octal
        starting with a '0'.

run-online = (filename)
        The full pathname of a program to run when WWWOFFLE is switched to
        online mode (default=none).  The program is started in the background
        with a single parameter set to the current mode name "online".

run-offline = (filename)
        The full pathname of a program to run when WWWOFFLE is switched to
        offline mode (default=none).  The program is started in the background
        with a single parameter set to the current mode name "offline".

run-autodial = (filename)
        The full pathname of a program to run when WWWOFFLE is switched to
        autodial (default=none).  The program is started in the background with
        a single parameter set to the current mode name "fetch".

run-fetch = (filename)
        The full pathname of a program to run when a WWWOFFLE fetch starts or
        stops (default=none).  The program is started in the background with two
        parameters, the first is the word "fetch" and the second is "start" or
        "stop".

lock-files = yes | no
        Enable the use of lock files to stop more than one WWWOFFLE process from
        downloading the same URL at the same time (default=no).  Disabling the
        lock-files may result in incomplete pages being displayed or many copies
        being downloaded if multiple requests are made for the same URL at the
        same time.

reply-compressed-data = yes | no
        If the replies that are made to the client are to contain compressed
        data when requested (default=no).  Requires zlib compilation option.

reply-chunked-data = yes | no
        If the replies that are made to the client are to use chunked encoding
        when possible (default=yes).

exec-cgi = (pathname)
        Enable the use of CGI scripts for the local pages on the WWWOFFLE
        server that match the wildcard pathname (default=none).

--------------------------------------------------------------------------------

OnlineOptions
-------------

Options that control how WWWOFFLE behaves when it is online.

[<URL-SPEC>] pragma-no-cache = yes | no
        Whether to request a new copy of a page if the request from the client
        has 'Pragma: no-cache' (default=yes).  This option takes precedence
        over the request-changed and request-changed-once options.

[<URL-SPEC>] cache-control-no-cache = yes | no
        Whether to request a new copy of a page if the request from the client
        has 'Cache-Control: no-cache' (default=yes).  This option takes
        precedence over the request-changed and request-changed-once options.

[<URL-SPEC>] cache-control-max-age-0 = yes | no
        Whether to request a new copy of a page if the request from the client
        has 'Cache-Control: max-age=0' (default=yes).  This option takes
        precedence over the request-changed and request-changed-once options.

[<URL-SPEC>] cookies-force-refresh = yes | no
        Whether to force the refresh of a page if the request from the client
        contains a cookie (default=no).  This option takes precedence over
        the request-changed and request-changed-once options.

[<URL-SPEC>] request-changed = (time)
        While online pages will only be fetched if the cached version is older
        than this specified time in seconds (default=600).  Setting this value
        negative will indicate that cached pages are always used while online.
        Longer times can be specified with a 'm', 'h', 'd' or 'w' suffix for
        minutes, hours, days or weeks (e.g. 10m=600).

[<URL-SPEC>] request-changed-once = yes | no
        While online pages will only be fetched if the cached version has not
        already been fetched once this session online (default=yes).  This
        option takes precedence over the request-changed option.

[<URL-SPEC>] request-expired = yes | no
        While online pages that have expired will always be requested again
        (default=no).  This option takes precedence over the request-changed
        and request-changed-once options.

[<URL-SPEC>] request-no-cache = yes | no
        While online pages that ask not to be cached will always be requested
        again (default=no).  This option takes precedence over the
        request-changed and request-changed-once options.

[<URL-SPEC>] request-redirection = yes | no
        While online pages that redirect the client to another URL temporarily
        will be requested again. (default=no).  This option takes precedence
        over the request-changed and request-changed-once options.

[<URL-SPEC>] request-conditional = yes | no
        While online pages that are requested from the server will be
        conditional requests so that the server only sends data if the page
        has changed (default=yes).

[<URL-SPEC>] validate-with-etag = yes | no
        When making a conditional request to a server enable the use of the
        HTTP/1.1 cache validator 'Etag' as well as modification time
        'If-Modified-Since' (default=yes).  The request-conditional option
        must also be selected for this option to take effect.

[<URL-SPEC>] try-without-password = yes | no
        If a request is made for a URL that contains a username and password
        then a request is made for the same URL without a username and
        password specified (default=yes).  This allows for requests for the
        URL without a password to re-direct the client to the passworded
        version.

[<URL-SPEC>] intr-download-keep = yes | no
        If the client closes the connection while online the currently
        downloaded incomplete page should be kept (default=no).

[<URL-SPEC>] intr-download-size = (integer)
        If the client closes the connection while online the page should
        continue to download if it is smaller than this size in kB
        (default=1).

[<URL-SPEC>] intr-download-percent = (integer)
        If the client closes the connection while online the page should
        continue to download if it is more than this percentage complete
        (default=80).

[<URL-SPEC>] timeout-download-keep = yes | no
        If the server connection times out while reading then the currently
        downloaded incomplete page should be kept (default=no).

[<URL-SPEC>] keep-cache-if-not-found = yes | no
        If the remote server replies with an error message or a redirection
        while there is a cached version with status 200 the previously cached
        version should be kept (default=no).

[<URL-SPEC>] request-compressed-data = yes | no
        If the requests that are made to the server are to request compressed
        data (default=yes).  Requires zlib compilation option.

[<URL-SPEC>] request-chunked-data = yes | no
        If the requests that are made to the server are to request chunked
        encoding (default=yes).

--------------------------------------------------------------------------------

OfflineOptions
--------------

Options that control how WWWOFFLE behaves when it is offline.

[<URL-SPEC>] pragma-no-cache = yes | no
        Whether to request a new copy of a page if the request from the client
        has 'Pragma: no-cache' (default=yes).  This option should be set to
        'no' if when browsing offline all pages are re-requested by a 'broken'
        browser.

[<URL-SPEC>] cache-control-no-cache = yes | no
        Whether to request a new copy of a page if the request from the client
        has 'Cache-Control: no-cache' (default=yes).  This option should be
        set to 'no' if when browsing offline all pages are re-requested by a
        'broken' browser.

[<URL-SPEC>] cache-control-max-age-0 = yes | no
        Whether to request a new copy of a page if the request from the client
        has 'Cache-Control: max-age=0' (default=yes).  This option should be
        set to 'no' if when browsing offline all pages are re-requested by a
        'broken' browser.

[<URL-SPEC>] confirm-requests = yes | no
        Whether to return a page requiring user confirmation instead of
        automatically recording requests made while offline (default=no).

[<URL-SPEC>] dont-request = yes | no
        Do not request any URLs that match this when offline (default=no).

--------------------------------------------------------------------------------

SSLOptions
----------

Options that control how WWWOFFLE behaves when a connection is made to it for an
https or Secure Sockets Layer (SSL) server.  Normally only tunnelling (with no
decryption or caching of the data) is possible.  When WWWOFFLE is compiled with
the gnutls library it is possible configure WWWOFFLE to decrypt, cache and
re-encrypt the connections.

quick-key-gen = yes | no
        Normally generation of secret keys for the SSL/https functions uses the
        default GnuTLS option for random number source.  This can be slow on
        some machines so this option selects a quicker but less secure random
        number source (default = no).  Requires GnuTLS compilation option.

expiration-time = (age)
        The length of time after creation that each certificate will expire
        (default = 1y).  Requires GnuTLS compilation option.

enable-caching = yes | no
        If caching (involving decryption and re-encryption) of SSL/https
        server connections is allowed (default = no).  Requires GnuTLS
        compilation option.

allow-tunnel = (host[:port])
        A hostname and port number (a WILDCARD match) for an https/SSL server
        that can be connected to using WWWOFFLE as a tunnelling proxy (no
        caching or decryption of the data) (default is no hosts or ports
        allowed).  This option should be set to *:443 to allow https to the
        default port number.  There can be more than one option for other
        ports or hosts as required.  This option takes precedence over the
        allow-cache option.  The host value is matched against the URL as
        presented, no hostname to IP or IP to hostname lookups are performed
        to find alternative equivalent names.

disallow-tunnel = (host[:port])
        A hostname and port number (a WILDCARD match) for an https/SSL server
        that can not be connected to using WWWOFFLE as a tunnelling proxy.
        There can be more than one option for other ports or hosts as
        required.  This option takes precedence over the allow-tunnel option.
        The host value is matched against the URL as presented, no hostname to
        IP or IP to hostname lookups are performed to find alternative
        equivalent names.

allow-cache = (host[:port])
        A hostname and port number (a WILDCARD match) for an https/SSL server
        that can be connected to using WWWOFFLE as a caching proxy (decryption
        of the data) (default is no hosts or ports allowed).  This option
        should be set to *:443 to allow https to the default port number.
        There can be more than one option for other ports or hosts as
        required.  The host value is matched against the URL as presented, no
        hostname to IP or IP to hostname lookups are performed to find
        alternative equivalent names.  Requires GnuTLS compilation option.

disallow-cache = (host[:port])
        A hostname and port number (a WILDCARD match) for an https/SSL server
        that can not be connected to using WWWOFFLE as a caching proxy.  This
        option takes precedence over the allow-cache option.  The host value
        is matched against the URL as presented, no hostname to IP or IP to
        hostname lookups are performed to find alternative equivalent names.
        Requires GnuTLS compilation option.

--------------------------------------------------------------------------------

FetchOptions
------------

Options that control what linked elements are downloaded when fetching pages
that were requested while offline.

[<URL-SPEC>] stylesheets = yes | no
        If style sheets are to be fetched (default=no).

[<URL-SPEC>] images = yes | no
        If images are to be fetched (default=no).

[<URL-SPEC>] webbug-images = yes | no
        If images that are declared in the HTML to be 1 pixel square are also to
        be fetched, requires the images option to also be selected
        (default=yes).  If these images are not fetched then the
        replace-webbug-images option in the ModifyHTML section can be used to
        stop browsers requesting them.

[<URL-SPEC>] icon-images = yes | no
        If icons (also called favourite icons or shortcut icons) as used by
        browsers for bookmarks are to be fetched (default=no).

[<URL-SPEC>] only-same-host-images = yes | no
        If the only images that are fetched are the ones that are on the same
        host as the page that references them, requires the images option to
        also be selected (default=no).

[<URL-SPEC>] frames = yes | no
        If frames are to be fetched (default=no).

[<URL-SPEC>] iframes = yes | no
        If inline frames (iframes) are to be fetched (default=no).

[<URL-SPEC>] scripts = yes | no
        If scripts (e.g. Javascript) are to be fetched (default=no).

[<URL-SPEC>] objects = yes | no
        If objects (e.g. Java class files) are to be fetched (default=no).

--------------------------------------------------------------------------------

IndexOptions
------------

Options that control what is displayed in the indexes.

create-history-indexes = yes | no
        Enables creation of the lasttime/prevtime and lastout/prevout indexes
        (default=yes).  The cycling of the indexes is always performed and
        they will flush even if this option is disabled.

cycle-indexes-daily = yes | no
        Cycles the lasttime/prevtime and lastout/prevout indexes daily instead
        of each time online or fetching (default = no).

<URL-SPEC> list-outgoing = yes | no
        Choose if the URL is to be listed in the outgoing index (default=yes).

<URL-SPEC> list-latest = yes | no
        Choose if the URL is to be listed in the lasttime/prevtime and
        lastout/prevout indexes (default=yes).

<URL-SPEC> list-monitor = yes | no
        Choose if the URL is to be listed in the monitor index (default=yes).

<URL-SPEC> list-host = yes | no
        Choose if the URL is to be listed in the host indexes (default=yes).

<URL-SPEC> list-any = yes | no
        Choose if the URL is to be listed in any of the indexes (default=yes).

--------------------------------------------------------------------------------

ModifyHTML
----------

Options that control how the HTML that is provided from the cache is modified.

[<URL-SPEC>] enable-modify-html = yes | no
        Enable the HTML modifications in this section (default=no).  With this
        option disabled the following HTML options will not have any effect.
        With this option enabled there is a small speed penalty.

[<URL-SPEC>] add-cache-info = yes | no
        At the bottom of all of the spooled pages the date that the page was
        cached and some navigation buttons are to be added (default=no).

[<URL-SPEC>] anchor-cached-begin = (HTML code) |
        Anchors (links) in the spooled page that are in the cache are to have
        the specified HTML inserted at the beginning (default="").

[<URL-SPEC>] anchor-cached-end = (HTML code) |
        Anchors (links) in the spooled page that are in the cache are to have
        the specified HTML inserted at the end (default="").

[<URL-SPEC>] anchor-requested-begin = (HTML code) |
        Anchors (links) in the spooled page that are not in the cache but have
        been requested for download are to have the specified HTML inserted at
        the beginning (default="").

[<URL-SPEC>] anchor-requested-end = (HTML code) |
        Anchors (links) in the spooled page that are not in the cache but have
        been requested for download are to have the specified HTML inserted at
        the end (default="").

[<URL-SPEC>] anchor-not-cached-begin = (HTML code) |
        Anchors (links) in the spooled page that are not in the cache or
        requested are to have the specified HTML inserted at the beginning
        (default="").

[<URL-SPEC>] anchor-not-cached-end = (HTML code) |
        Anchors (links) in the spooled page that are not in the cache or
        requested are to have the specified HTML inserted at the end
        (default="").

[<URL-SPEC>] disable-script = yes | no
        Removes all scripts and scripted events (default=no).

[<URL-SPEC>] disable-applet = yes | no
        Removes all Java applets (default=no).

[<URL-SPEC>] disable-style = yes | no
        Removes all stylesheets and style references (default=no).

[<URL-SPEC>] disable-blink = yes | no
        Removes the <blink> tag from HTML but does not disable blink in
        stylesheets (default=no).

[<URL-SPEC>] disable-marquee = yes | no
        Removes the <marquee> tag from HTML to stop scrolling text
        (default=no).

[<URL-SPEC>] disable-flash = yes | no
        Removes any Shockwave Flash animations (default=no).

[<URL-SPEC>] disable-iframe = yes | no
        Removes any inline frames (the <iframe> tag) from HTML (default=no).

[<URL-SPEC>] disable-meta-refresh = yes | no
        Removes any meta tags in the HTML header that re-direct the client to
        change to another page after an optional delay (default=no).

[<URL-SPEC>] disable-meta-refresh-self = yes | no
        Removes any meta tags in the HTML header that re-direct the client to
        reload the same page after a delay (default=no).

[<URL-SPEC>] disable-meta-set-cookie = yes | no
        Removes any meta tags in the HTML header that cause cookies to be set
        (default=no).

[<URL-SPEC>] disable-dontget-links = yes | no
        Disables any links to URLs that are in the DontGet section of the
        configuration file (default=no).

[<URL-SPEC>] disable-dontget-iframes = yes | no
        Disables inline frame (iframe) URLs that are in the DontGet section of
        the configuration file (default=no).

[<URL-SPEC>] replace-dontget-images = yes | no
        Replaces image URLs that are in the DontGet section of the
        configuration file with a static URL (default=no).

[<URL-SPEC>] replacement-dontget-image = (URL)
        The replacement image to use for URLs that are in the DontGet section
        of the configuration file (default=/local/dontget/replacement.gif).

[<URL-SPEC>] replace-webbug-images = yes | no
        Replaces image URLs that are 1 pixel square with a static URL
        (default=no).  The webbug-images option in the FetchOptions section
        can be used to stop these images from being automatically downloaded.

[<URL-SPEC>] replacement-webbug-image = (URL)
        The replacement image to use for images that are 1 pixel square
        (default=/local/dontget/replacement.gif).

[<URL-SPEC>] demoronise-ms-chars = yes | no
        Replaces strange characters that some Microsoft applications put into
        HTML with character equivalents that most browsers can display
        (default=no).  The idea for this comes from the public domain
        Demoroniser perl script.

[<URL-SPEC>] fix-mixed-cyrillic = yes | no
        Replaces punctuation characters in cp-1251 encoding that are combined
        with text in koi-8 encoding that appears in some cyrillic web pages.

[<URL-SPEC>] disable-animated-gif = yes | no
        Disables the animation in animated GIF files (default=no).

--------------------------------------------------------------------------------

LocalHost
---------

A list of hostnames that the host running the WWWOFFLE server may be known by.
This is so that the proxy does not need to contact itself if the request has a
different name for the same server.

(host)
        A hostname or IP address that in connection with the port number (in
        the StartUp section) specifies the WWWOFFLE proxy HTTP server.  The
        hostnames must match exactly, it is not a WILDCARD match.  The first
        named host is used as the server name for several features so must be
        a name that will work from any client host on the network.  The
        entries can be hostnames, IPv4 addresses or IPv6 addresses enclosed
        within '[...]'.  None of the hosts named here are cached or fetched
        via a proxy.

--------------------------------------------------------------------------------

LocalNet
--------

A list of hostnames whose web servers are always accessible even when offline
and are not to be cached by WWWOFFLE because they are on a local network.

(host)
        A hostname or IP address that is always available and is not to be
        cached by WWWOFFLE.  The host name matching uses WILDCARDs.  A host
        can be excluded by appending a '!' to the start of the name.  The host
        value is matched against the URL as presented, no hostname to IP or IP
        to hostname lookups are performed to find alternative equivalent
        names.  The entries can be hostnames, IPv4 addresses or IPv6 addresses
        enclosed within '[...]'.  All entries here are assumed to be reachable
        even when offline.  None of the hosts named here are cached or fetched
        via a proxy.

--------------------------------------------------------------------------------

AllowedConnectHosts
-------------------

A list of client hostnames that are allowed to connect to the server.

(host)
        A hostname or IP address that is allowed to connect to the server.
        The host name matching uses WILDCARDs.  A host can be excluded by
        appending a '!' to the start of the name.  If the IP address or
        hostname (if available) of the machine connecting matches then it is
        allowed.  The entries can be hostnames, IPv4 addresses or IPv6
        addresses enclosed within '[...]'.  All of the hosts named in
        LocalHost are also allowed to connect.

--------------------------------------------------------------------------------

AllowedConnectUsers
-------------------

A list of the users that are allowed to connect to the server and their
passwords.

(username):(password)
        The username and password of the users that are allowed to connect to
        the server.  If this section is left empty then no user authentication
        is done.  The username and password are both stored in plaintext
        format.  This requires the use of clients that handle the HTTP/1.1
        proxy authentication standard.

--------------------------------------------------------------------------------

DontCache
---------

A list of URLs that are not to be cached by WWWOFFLE.

[!]URL-SPECIFICATION
        Do not cache any URLs that match this.  The URL-SPECIFICATION can be
        negated to allow matches to be cached.  The URLs that are not cached
        will not have requests recorded if offline or fetched automatically.

--------------------------------------------------------------------------------

DontGet
-------

A list of URLs that are not to be got by WWWOFFLE when it is fetching and not
to be served from the WWWOFFLE cache even if they exist.

[!]URL-SPECIFICATION
        Do not get any URLs that match this.  The URL-SPECIFICATION can be
        negated to allow matches to be got.

[<URL-SPEC>] replacement = (URL)
        The URL to use to replace any URLs that match the URL-SPECIFICATIONs
        instead of using the standard error message (default=none).  The URLs
        in /local/dontget/ are suggested replacements (e.g. replacement.gif or
        replacement.png which are 1x1 pixel transparent images or
        replacement.js which is an empty javascript file).

<URL-SPEC> get-recursive = yes | no
        Choose whether to get URLs that match this when doing a recursive
        fetch (default=yes).

<URL-SPEC> location-error = yes | no
        When a URL reply contains a 'Location' header that redirects to a URL
        that is not got (specified in this section) then the reply is modified
        to be an error message instead (default=no).  This will stop ISP
        proxies from redirecting users to adverts if the advert URLs are
        in this section.

--------------------------------------------------------------------------------

DontCompress
------------

A list of MIME types and file extensions that are not to be compressed by
WWWOFFLE (because they are already compressed or not worth compressing).
Requires zlib compilation option.

mime-type = (mime-type)/(subtype)
        The MIME type of a URL that is not to be compressed in the cache (when
        purging) or when providing pages to clients.

file-ext = .(file-ext)
        The file extension of a URL that is not to be requested compressed
        from a server.

--------------------------------------------------------------------------------

CensorHeader
------------

A list of HTTP header lines that are to be removed from the requests sent to
web servers and the replies that come back from them.

[<URL-SPEC>] (header) = yes | no | (string)
        A header field name (e.g. From, Cookie, Set-Cookie, User-Agent) and
        the string to replace the header value with (default=no).  The header
        is case sensitive, and does not have a ':' at the end.  The value of
        "no" means that the header is unmodified, "yes" or no string can be
        used to remove the header or a string can be used to replace the
        header.  This only replaces headers it finds, it does not add any new
        ones.  An option for Referer here will take precedence over the
        referer-self and referer-self-dir options.

[<URL-SPEC>] referer-self = yes | no
        Sets the Referer header to the same as the URL being requested
        (default=no).  This will add the Referer header if none is contained
        in the original request.

[<URL-SPEC>] referer-self-dir = yes | no
        Sets the Referer header to the directory name of the URL being
        requested (default=no).  This will add the Referer header if none is
        contained in the original request.  This option takes precedence over
        referer-self.

[<URL-SPEC>] referer-from = yes | no
        Removes the Referer header based on a match of the referring URL
        (default=no).

[<URL-SPEC>] force-user-agent = yes | no
        Forces a User-Agent header to be inserted into all requests that are
        made by WWWOFFLE (default=no).  This User-Agent is added only if there
        is not an existing User-Agent header and is set to the value
        WWWOFFLE/<version-number>.  This header is inserted before censoring
        and may be changed by the normal header censoring method.

[<URL-SPEC>] pass-url-unchanged = yes | no
        Forces WWWOFFLE to ignore the requirements on the correct formatting of
        URLs and to pass through to the server the URL that was passed to it by
        the browser (default=no).

--------------------------------------------------------------------------------

FTPOptions
----------

Options to use when fetching files using the ftp protocol.

anon-username = (string)
        The username to use for anonymous ftp (default=anonymous).

anon-password = (string)
        The password to use for anonymous ftp (default determined at run
        time).  If using a firewall then this may contain a value that is not
        valid to the FTP server and may need to be set to a different value.

<URL-SPEC> auth-username = (string)
        The username to use on a host instead of the default anonymous
        username.

<URL-SPEC> auth-password = (string)
        The password to use on a host instead of the default anonymous
        password.

--------------------------------------------------------------------------------

MIMETypes
---------

MIME Types to use when serving files that were not fetched using HTTP or for
files on the built-in web-server.

default = (mime-type)/(subtype)
        The default MIME type (default=text/plain).

.(file-ext) = (mime-type)/(subtype)
        The MIME type to associate with a file extension.  The '.' must be
        included in the file extension.  If more than one extension matches
        then the longest one is used.

--------------------------------------------------------------------------------

Proxy
-----

This contains the names of the HTTP (or other) proxies to use external to the
WWWOFFLE server machine.

[<URL-SPEC>] proxy = (host[:port])
        The hostname and port on it to use as the proxy.

<URL-SPEC> auth-username = (string)
        The username to use on a proxy host to authenticate WWWOFFLE to it.
        The URL-SPEC in this case refers to the proxy and not the URL being
        retrieved.

<URL-SPEC> auth-password = (string)
        The password to use on a proxy host to authenticate WWWOFFLE to it.
        The URL-SPEC in this case refers to the proxy and not the URL being
        retrieved.

[<URL-SPEC>] ssl = (host[:port])
        A proxy server that should be used for https or Secure Socket Layer
        (SSL) connections.  Note that for the <URL-SPEC> that only the host is
        checked and that the other parts must be '*' WILDCARDs.

--------------------------------------------------------------------------------

Alias
-----

A list of aliases that are used to replace the server name and path with
another server name and path.

URL-SPECIFICATION = URL-SPECIFICATION
        Any requests that match the first URL-SPECIFICATION are replaced by
        the second URL-SPECIFICATION.  The first URL-SPECIFICATION is a
        wildcard match for the protocol and host/port, the path must match the
        start of the requested URL exactly and includes all subdirectories.

--------------------------------------------------------------------------------

Purge
-----

The method to determine which pages to purge, the default age the host
specific maximum age of the pages in days, and the maximum cache size.

use-mtime = yes | no
        The method to use to decide which files to purge, last access time
        (atime) or last modification time (mtime) (default=no).

max-size = (size)
        The maximum size for the cache in MB after purging (default=-1).  A
        maximum cache size of -1 (or 0 for backwards compatibility) means
        there is no limit to the size.  If this and the min-free options are
        both used the smaller cache size is chosen.  This option take into
        account the URLs that are never purged when measuring the cache size
        but will not purge them.

min-free = (size)
        The minimum amount of free disk space in MB after purging
        (default=-1).  A minimum disk free of -1 (or 0) means there is no
        limit to the free space.  If this and the max-size options are both
        used the smaller cache size is chosen.  This option take into account
        the URLs that are never purged when measuring the cache size but will
        not purge them.

use-url = yes | no
        If true then use the URL to decide on the purge age, otherwise use the
        protocol and host only (default=no).

del-dontget = yes | no
        If true then delete the URLs that match the entries in the DontGet
        section (default=no).

del-dontcache = yes | no
        If true then delete the URLs that match the entries in the DontCache
        section (default=no).

[<URL-SPEC>] age = (age)
        The maximum age in the cache for URLs that match this (default=14).
        An age of zero means always to delete, negative means not to delete.
        The URL-SPECIFICATION matches only the protocol and host unless
        use-url is set to true. Longer times can be specified with a 'w', 'm'
        or 'y' suffix for weeks, months or years (e.g. 2w=14).

[<URL-SPEC>] compress-age = (age)
        The maximum age in the cache for URLs that match this to be stored
        uncompressed (default=-1).  Requires zlib compilation option.  An age
        of zero means always to compress, negative means never to compress.
        The URL-SPECIFICATION matches only the protocol and host unless
        use-url is set to true. Longer times can be specified with a 'w', 'm'
        or 'y' suffix for weeks, months or years (e.g. 2w=14).

--------------------------------------------------------------------------------

WILDCARD
--------

A WILDCARD match is one that uses the '*' character to represent any group of
characters.

This is basically the same as the command line file matching expressions in
DOS or the UNIX shell, except that the '*' can match the '/' character.

For example

*.gif       matches  foo.gif and bar.gif

*.foo.com   matches  www.foo.com and ftp.foo.com

/foo/*      matches  /foo/bar.html and /foo/bar/foobar.html

--------------------------------------------------------------------------------

URL-SPECIFICATION
-----------------

When specifying a host and protocol and pathname in many of the sections a
URL-SPECIFICATION can be used, this is a way of recognising a URL.

For the purposes of this explanation a URL is considered to be made up of five
parts.

proto          The protocol that is used (e.g. 'http', 'ftp')

host           The server hostname (e.g. 'www.gedanken.org.uk').

port           The port number on the host (e.g. default of 80 for HTTP).

path           The pathname on the host (e.g. '/bar.html') or a directory name
               (e.g. '/foo/').

args           Optional arguments with the URL used for CGI scripts etc.
               (e.g. 'search=foo').

For example the WWWOFFLE homepage: http://www.gedanken.org.uk/software/wwwoffle/
The protocol is 'http', the host is 'www.gedanken.org.uk', the port is
the default (in this case 80), and the pathname is '/software/wwwoffle/'.

In general this is written as (proto)://(host)[:(port)]/[(path)][?(args)]

Where [] indicates an optional feature, and () indicate a user supplied name
or number.

Some example URL-SPECIFICATION options are the following:

*://*/*             Any protocol, Any host, Any port, Any path, Any args
                    (This is the default for options that can have a <URL-SPEC>
                    prefix when none is specified).

*://*/(path)        Any protocol, Any host, Any port, Named path, Any args

*://*/*?            Any protocol, Any host, Any port, Any path, No args

*://*/(path)?*      Any protocol, Any host, Any port, Named path, Any args

*://(host)          Any protocol, Named host, Any port, Any path, Any args

(proto)://*/*       Named proto, Any host, Any port, Any path, Any args

(proto)://(host)/*  Named proto, Named host, Any port, Any path, Any args

(proto)://(host):/* Named proto, Named host, Default port, Any path, Any args

*://(host):(port)/* Any protocol, Named host, Named port, Any path, Any args

The matching of the host, the path and the args use the WILDCARD matching that
is described above.  The matching of the path has the special condition that a
WILDCARD of '/*/foo' will match '/foo' and '/any/path/foo', in other words it
matches any path prefix.

In some sections that accept URL-SPECIFICATIONs they can be negated by
inserting the '!' character before it.  This will mean that the comparison
of a URL with the URL-SPECIFICATION will return the logically opposite value
to what would be returned without the '!'.  If all of the URL-SPECIFICATIONs
in a section are negated and '*://*/*' is added to the end then the sense of
the whole section is negated.

In all sections that accept URL-SPECIFICATIONs the comparison can be made case
insensitive for the path and arguments part by inserting the '~' character
before it.  (The host and the protocol comparisons are always case
insensitive).