Description:
This Perl script downloads the URL for the current comic of various strips that
are available online, and outputs these URLs to an HTML file. This enables you
to view all of your favorite strips at once, instead of visiting several
different websites.

One feature that sets this program apart from the rest is the ability to operate
in 'local' mode. This mode, instead of writing the addresses to images in the
HTML file, actually downloads the image and saves it locally. This way, you do
not have to wait for each image to download - it's already on your hard drive.
This is most useful when dailystrips is run from a cron script. (See file
README.LOCAL for more information and examples)

Requirements:
dailystrips requires a reasonably current version of Perl - it was developed
with 5.6.0. The HTTP::Request, LWP::UserAgent, and POSIX modules are needed. If
you want to use --date, you will need Date::Time from the TimeDate package. See
www.cpan.org if you don't have them installed already. dailystrips was
developed on Linux but should be able to run on any *nix with that has the
necessary Perl modules installed. Running on Win32 is also possible.

For the AvantGo mode, ImageMagick is required.

Installation:
See the INSTALL file.

Usage:
'dailystrips [stripname(s)]' will print to STDOUT an HTML page with image links
to the latest strip. These links are to the strip's webserver.
'dailystrips --help' lists all available options. --list shows the available
strips and groups. Strip names can specified as listed. Groups must be preceeded
with an '@' symbol.

Adding new strips:
The strips.def file should be relatively self-explanatory. (see README.DEFS for
detailed information). If you are adding several strips from the same site that
share a common format, please create a class for that site. In addition, please
try to pick a method of determining the most current URL possible (i.e. don't
search if it's possible to predict - we don't want to get old strips if running
early in the morning and a site hasn't updated the static page yet) Also, when
you add a new strip, I'd appreciate it if you could email the defintion to me at
amedico@amedico.dhs.org so I can add it to the distribution.

Personal definition file:
Users may create a file called ".dailystrips.defs" in their home directory.
Syntax is exactly the same as the main strips.def file. Personal files will be
processed after the main file. This means that classes set in the main file are
available for use in users' files. Also note that any entries (classes, strips,
and groups) in users' files with the same name as entries in the main will take
precedence. Personal definition files can be ignored with the '--nopersonal'
option.

System-wide definitions file:
Administrators may wish to create definitions, classes, or groups available to
all users on a system. This may be done by creating a file /etc/dailystrips.defs
and placing the definitions there. This file is loaded after the default
definition file and so may be use to override provided definitions, if desired.
The system-wide definitions file can be ignored with the --nosystem option.

Definition upgrades:
Since full releases are not made available every time the definitions file
changes, the current CVS copy of the definitions file is available through a
link at http://dailystrips.sourceforge.net/download.html. Users may either
replace /usr/share/dailystrips/strips.def with this updated copy, or if a
package management system is used (Debian dpkg, RPM, etc.) placing the updated
definitions in /etc/dailystrips.defs is recommended.

*Notice*:
Keep in mind that this program is for personal use only, as making
the output publicly available on the internet constitutes copyright infringement
without permission from the strips' authors. If you're running it on a personal
webserver that can be accessed fron the internet (even if it's not specifically
public), make sure you set up restrictions so that only you have access to it -
some publishers (Keenspot, Exclusive Content) seem to be checking their
webserver logs for dailystrips users and will come after you in a rather nasty
fashion if it even looks like you're using dailystrips to make a public website.

Copyright info:
This program Copyright (C) 2001 Andrew Medico <amedico@amedico.dhs.org>.
All rights reserved. This program is free software; you may redistribute it
and/or modify it under the terms of the GNU GPL, Version 2.

This file describes in further detail how the strips.def file is contsructed.

Strips can be defined in one of two ways. The first is standalone. Also, strips
can be provide the image URL by generating it (as from the current date) or by
searching a web page for a URL. Let's look at an example of generating first:

strip badtech
	name Badtech
	artist James Sharman
	homepage http://www.badtech.com/
	type generate
	imageurl http://www.badtech.com/a/%-y/%-m/%-d.jpg
	provides any
end

In the first line, we specify the short name of the strip that will be used to
refer to it on the command line. This must be unique. Next, "name" specifies the
name of the strip to display in the HTML output. "artist" specifies the strip
artist's name, which will be displayed on the same line as the name of the
strip. "homepage" is the address of the strip's homepage, use for the link in
the output. "type" can be either "generate" or "search". Here we are using
"generate" to generate a URL. "imageurl" is the address of the image. You are
allowed to use a number of special variables. Single letters preceeded by the
"%" symbol, such as "%Y", "%d", "%m", etc. are interpreted as date variables
and passed to the strftime function for conversion. Date variables may be used
in: homepage, searchpage, searchpattern, imageurl, baseurl, and referer.
"date --help" provides a reference that is compatible. You can also use a "$"
followed by the name of the above variables, such as "$homepage". This will
simply subsititute "http://www.badtech.com" in place of "$homepage". You can
use named variables for name, homepage, searchpage, searchpattern, imageurl,
baseurl, referer variables on homepage, searchpage, searchpattern, imageurl,
baseurl, and referer lines.


The other type of URL generation, searching, is as follows:

strip joyoftech
	name The Joy of Tech
	homepage http://www.joyoftech.com/joyoftech/
	type search
	searchpattern <IMG.+?src="(joyimages/\d+\.gif)\"
	matchpart 1
	baseurl http://www.joyoftech.com/joyoftech/
	provides latest
end

"strip", "name", and "homepage" all function as above. The difference is the
"type search" line and the lines that follow. "searchpattern" is a Perl regular
expression that must be written to match the strip's URL. Not shown is
"searchpage", which would ordinarily go above "searchpattern". This is a URL to
a web page and is only needed if the URL to the strip image is not found on the
homepage. The same special variables listed above for "imageurl" may also be
used here. "matchpart" tells the script which parenthetical section (there must
be at least one) will contain the desired URL (see man perlre on $n variables
for more). Note that this line is only mandatory for values other than 1.
"baseurl" only needs to be specified if the "searchpattern" regular expression
does not match a full URL (that is, it does not start with "http://" and contain
the host). If specified, it is prepended to whatever "searchpattern" matched.
Not shown is "urlsuffix", which will be appended to the what "searchpattern"
matched. Finally, "imageurl" can also be used here in place of "baseurl" and
"urlsuffix". It is useful when addresses to the strip image must be constructed
from a known portion and a variable portion that is searched for. Simply specify
the URL template and insert "$match" wherever you wish the result of the search
to be put. To access multiple matches, use the $match[1..9] variables. See the
strips "8bit" and "pgs" for examples of how this works.

The "provides" line indicates which type of strips the definition can provide:
either "any" for a definition that can provide the strip for any given date
or "latest" for a definition that can only provide the current strip. This is
used so that the program can skip definitons that only provide the latest strip
when running with the --date option.

Two additional variables are not shown. First is "referer". Some webservers
insist that the HTTP_REFERER header be set to the address of the HTML page that
the image is on, or they will not return the image. This is to prevent other
sites from linking to the image and (presumably) scripts like this from
functioning. What the script does by default is set the HTTP_REFERER header to
the searchpage (if specified), or the homepage (if no specific searchpage was
specified). If the webserver for some reason needs a referer other than the
searchpage or homepage, it can be specified with this variable. The second
keyword is "prefetch". This was added because it seemed at one point that
sfgate required a certain page to be downloaded immediately before the strip
images could be loaded.  The syntax is simply "prefetch [URL]". Any URL
specified will be downloaded immediately before the strip image. If this URL
cannot be retrieved (error 404 from the webserver, etc), no attempt will be made
to download the strip image.

New feature: you can now put little snippets of Perl code right into the
definition. For example, the definition for The Norm uses this to generate the
day number for 14 days ago. The Norm website uses Javascript to generate the
image URL, so it couldn't be searched for and previously there was no way to
work with dates other than the current date. Here's how it works: just
insert <code:Perl code>. No need to quote the code, just put it where
"Perl code" is. Don't forget to escape any > that may happen to be in your
code.


The other method of specifying strips is to use classes. This method is used
when there are serveral strips provided by the same webserver that all have an
identical definition, except for some strip-specific elements. Classes work as
follows:

First, the class is declared:
class ucomics-srch
	homepage http://www.ucomics.com/$strip/view$1.htm
	type search
	searchpattern (/$1/(\d+)/$1(\d+)\.(gif|jpg))
	matchpart 1
	baseurl http://images.ucomics.com/comics
	provides latest
end

This is just like a strip definition, except "class" is the first line. The
value for "class" must be unique among other classes but will not conflict with
the names of strips. Strip-specific elements are specified using special
variables "$x", where "x" is a number from 0 to 9. When the definition file is
parsed, these variables are retrieved from the strip definition, shown below.

strip calvinandhobbes
	name Calvin and Hobbes
	useclass ucomics-srch
	$1 ch
end

This definition is like a normal definition except the second line is "useclass"
followed by the name of the class to use. Below that, the strip-specific "$x"
variables must be specified. Values already declared in the class can be
overridden (if necessary) by simply specifying them in the strip definition.


For your convenience, "groups" of strips may also be defined. These allow you to
use a single keyword on the command line to refer to a whole set of strips. The
construct is as follows:

>group favorites
>	desc My Favorite Comics
>	include peanuts
>	include foxtrot, userfriendly
>end

The group name must be unique among all groups, but will not conflict with
strips or classes (in fact it might be useful to have a group for each class - I
might even make that automatic). "desc" is a description of the group that will
be shown by --list. Everything after an "include" is added to the list of
strips. You may specify one or more strips per "include" line, whatever you
prefer. Groups are referenced on the command line by an "@" symbol followed by
the name (bare words not preceeded by an "@" symbol will be considered strips).
Finally, you may use "exclude [strips]" lines instead of "include" lines to
make the group contain everystrip except those specified.

Notes:

* As of 1.0.10, only date variables use the "%" symbol - everything else now
  uses "$"

* For classes, variables declared in the strip definition take precedence over
  those specified in the class, if there is any conflict

* You cannot use "$variablename" to refer to a variable below the current line
  (assuming that you use the standard order) if the referenced variable is a
  reference itself - the script only parses "$variablename" references once.
  This is a bug and is scheduled to be fixed.

* If no "searchpage" is specified for definitions of type "search", the value of
  "homepage" is used.

* If no "referer" is specified, the value of "searchpage" is used. If this has
  not been set (in the case of definitions that generate the URL or search
  definitions that use the homepage as the searchpage), the value of "homepage"
  is used.

* There may be additional problems lurking in the defintion file-parsing code.
  It currently works fairly well, but needs to be re-written properly.

* Group, strip, and class names can contain pretty much any character except
  semicolon, space, and pipe.

This file describes in further detail dailystrips' 'local' mode of
operation.

Normally, dailystrips generates the URLs to the selected strips and
displays them on STDOUT for redirection into a file (--output lets you
specify a file to save the HTML into automatically).  However, this is
usually inconvenient as you must first run the program  manually,  and
then wait for your browser to download the images. Local mode, instead
of putting the address of each image in the generated page,  downloads
the image and save a copy of it locally. The address in  the generated
page then refers to that downloaded image. With this,  you do not have
to wait for your browser to download the images from the 'net  -  they
are already on your hard drive (or your own webserver - another way to
set it up).  The time savings really add up when you want to view  the
page more than once.

Method 1: (one system)

Create a directory (such as /home/user/dailystrips/) in which to store
the downloaded images and output files.

Create a cron script like the following and put it in  /etc/cron.daily
(call it dailystrips.sh, etc)

#!/bin/sh
echo "dailystrips running:"
su user -c "dailystrips --basedir /home/user/dailystrips --local all"

Of course, change /home/user/dailystrips to the exact  directory  that
you created.  Also, change "su user.." to reflect an appropriate login
(your own is  best if you created a  directory in your home directory)
You should also change the "all"  to reflect just  the strips you want
to see - the over 200 strips would take a rather long time to download
even with a fast connection.

By default, the program calls the output file 'dailystrips-YYYY.MM.DD'
and creates 'index.html' as a symlink to this, so that you can make a
bookmark to index.html  in your web browser that will always take you
to the latest page.  If you need to change this,  you'll have to edit
the script.

For scheduling the time that dailystrips runs, you'll have to consider
your time zone.  I have found  that running at  0600 EST (-0500) works
well for my strips (see the contents of group 'andrew').  You may have
to experiment a  little to find the best time for the  specific strips
you download. One idea is to create a crontab entry to run dailystrips
early in the morning  and then a few hours later  if the strips aren't
all available at one time.


Method 2: (webserver/intranet)

This is the same as above  -  just locate the directory somewhere  in
your webserver's document root.  Keep in mind that this should  be for
personal  use only,  as making the  output publicly  available  on the
internet constitutes  copyright infringement, without permission  from
the strips' authors. As far as I know, though, it's within your rights
to use this for personal use.

Description:
In order to keep disk usage down, the dailystrips-clean script is provided to
automatically delete files that are older than a specified number of days, for
example, a week or a month.

Usage:
Operation is straightforward. In the directory where the downloaded strip and
HTML files are located, simply run the command:

'dailystrips-clean DAYS'

Where DAYS is the number of days to preserve. There are also a few simple
options available that may be viewed by running 'dailystrips-clean --help'.

Description:
The dailystrips-update script is provided to automatically retrieve updated
strip definitions from a central server, eliminating the need to manually
download and install new versions of the entire program.

It is recommened that the dailystrips-update script be run before each
execution of the main dailystrips program. The dailystrips-update script
has provisions to only check for updates if the current file has reached a
certain age, which maximizes efficiency and should allow users to run the
update script before each execution of dailystrips without overtaxing the
update server.

Usage:
Operation is straightforward. Simply execute dailystrips-update and the
updated definitions will be saved to ~/.dailystrips-updates.def, which will
automatically be loaded by the main dailystrips program.

There are also a few simple options available that may be viewed by running
'dailystrips-update --help'.