This is the framework of scripts and stylesheets for running any XSL
formatter on a set of test files and working with the test results.

The default configuration works with the xmlroff XSL Formatter.

This is part of the xmlroff XSL Formatter project.

-----------------------------------------------------------------------
xmlroff is a fast, free, high-quality, multi-platform XSL formatter
that aims to excel at DocBook formatting and that integrates easily
with other programs and with scripting languages.

The xmlroff XSL Formatter is a library -- libfo -- and a command-line
program -- xmlroff -- for formatting XML according to the XSL 1.0
Recommendation.

See http://xmlroff.org/.
-----------------------------------------------------------------------

The version numbers of 'testing' releases are the same as the version
of xmlroff that is current at the time of the 'testing' release.
However, any version of this package may be used with any version of
xmlroff, although using the newest 'testing' version is recommended
since new versions tend to have bug fixes and ease of use improvements
and, occasionally, better documentation.

1. Testing xmlroff
2. Requirements
3. Before you begin
4. Running the tests
5. Working with the results
6. Bundling the test results
7. When the test suite changes
8. Setting up a web server for updating results
9. Testing an alternative XSL processor
10. DocBook testdocs
11. FOP "layoutengine" Tests
12. Setting defaults in the test results file
13. Gory details
14. License


1. Testing xmlroff
==================

xmlroff testing uses:

 - Test cases from the XSL 1.0 CR test suite, NIST, and xmlroff

   See the xmlroff 'testsuite' package and
   http://www.w3.org/Style/XSL/TestSuite/.

 - XSLT stylesheet for making shell scripts to run xmlroff on the test
   cases

 - Perl scripts for running the test scripts and diffing the outputs

 - XSLT stylesheets for making summary and individual test result HTML
   files

 - Perl CGI script and XSLT stylesheet for optionally updating test
   result HTML files


2. Requirements
===============

Testing xmlroff requires:

 - Autotools

 - Perl

 - ImageMagick

 - xsltproc


3. Before you begin
===================

1. Download and install the xmlroff 'testsuite' package from
   http://xmlroff.org/ or clone the git repository at
   https://github.com/MenteaXML/xmlroff.git.

2. Make sure that testsuites.xml refers to the testsuite.xml file in
   the 'testsuite' package.

3. (Optional) Checkout the DocBook testdocs module from the Subversion
   repository of the DocBook SourceForge project, e.g., into
   '../testdocs':

      cd ..
      svn co \
         https://docbook.svn.sourceforge.net/svnroot/docbook/trunk/testdocs \
         testdocs

   In the 'testdocs' directory, run 'make-testsuite.pl' from this
   module to create a usable testsuite.xml file, e.g.:

      cd testdocs
      perl ../testing/make-testsuite.pl \
         --stylesheet \
         http://docbook.sourceforge.net/release/xsl/current/fo/docbook.xsl \
         . > testsuite.xml

   Make sure that testsuites.xml in this module refers to the
   testsuite.xml in the 'testdocs' directory.

   DocBook testing is described further in the 'DocBook testdocs'
   section below (and at http://xmlroff.org/wiki/DocBookTestdocs).

4. Run configure to generate the Makefile:

      ./configure

   The parameters for controlling the test are set as configure
   options.  Use './configure --help' to see the available options.

   Processing instructions in the test results document may override
   the default values of some paramters, but setting parameters as
   configure options overrides both.

   See below for configuring to use an alternative XSL processor.

   (If starting from a Subversion checkout, run './autogen.sh'.)

7. (Optional, Apache-only) Copy 'testing.conf' to your Apache2 conf.d
   directory, e.g., '/etc/apache2/conf.d' (or make a symbolic link).
   Modify the directory name in 'testing.conf' to match where you
   installed this package.  Restart Apache.

   If you needed to modify 'testing.conf' and are going to repeatedly
   run 'configure', modify 'testing.conf.in' instead so you won't have
   to keep redoing the changes.

8. (Optional, non-Apache) Install update-results.pl in the cgi-bin of
   your web server.

9. (Optional) Make sure that your web server can open an 'index.html'
   file in this directory and that the paths in update-results.pl are
   correct for your setup.

   Note that you may need to change the permissions on the current
   directory and its descendant files and directories so both you and
   the webserver UID (e.g., 'wwwrun' or 'www-data' for Apache2,
   depending on your configuration) can create and modify files.

   The simplistic solution (if you have the 'acl' option set for the
   current file system in '/etc/fstab') is:

      setfacl -m u:www-data:rwx .


4. Running the tests
====================

All at once
-----------

1. Run make:

      make -k

Individual steps
----------------

1. Run dotests.pl to run the tests:

      perl dotests.pl .

2. Compare results to reference versions and generate PNG versions of
   files:

      perl png+ref2stereo.pl .

3. Generate an XML summary of the reslts:

      perl testsuccess.pl . > testsuccess.xml

4. Generate HTML reports of the tests and their results:

      make index.html

   To make a 'static' report without the forms for updating results,
   use the '--disable-form' option when running configure.


5. Working with the results
===========================

1. Open index.html in your web browser:

      http://127.0.0.1/path/to/testing/index.html

   or:

      file:///path/to/testing/index.html

   or start the threaded standalone web server:

      perl threaded.pl

   or start the standalone web server:

      perl testing-server.pl

   and open the page it suggests, e.g.:

      http://localhost:8080/index.html

2. For any test, you can click on the "Results" link to see individual
   test results.

3. If you did not create 'static' reports and you have
   update-results.pl installed correctly, you can update the test
   results using the form in the individual test result page.

   At present, the test result page is updated to reflect your changes
   only when using 'threaded.pl'.  Otherwise, the page is updated only
   once you run 'testsuccess.xsl' again.


6. Bundling the test results
============================

1. Run the 'dist-results' target:

      make -k dist-results

   This regenerates the HTML output with 'static' reports and bundles
   all the reports and associated output and image files into a
   gzipped tar archive.

   The reports, when unbundled, can be viewed but not modified.

   The individual reports contain links to their input XML and XSLT
   files, and relative links will only work when the reports and the
   testsuites used when running the tests are installed in the same
   locations relative to each other as they were when the tests were
   run and the reports were generated.


7. When the test suite changes
==============================

1. Re-run test2sh.xsl to update the script that runs the tests:

      xsltproc --novalid test2sh.xsl test2sh.xsl

2. Create a new, temporary test results XML file:

      xsltproc --novalid testsuites2testresults.xsl testsuites.xml > \
         tmp.xml

3. Merge the generated <testresult> elements for the new tests into
   the existing testresults.xml file.

   'testresults.xml' is the default name.  If you specified a
   different filename when you ran 'configure', modify that file.

4. Run the tests, which will now include the new tests.


8. Setting up a web server for updating results
===============================================

There are currently three ways to run a web server so you can update
the test results using the form in the individual HTML reports:

 - threaded.pl

 - testing-server.pl

 - Apache

threaded.pl
-----------

Based on a 'HTTP::Daemon::Threaded' example, this supports updating
individual results' HTML.

testing-server.pl
-----------------

Based on 'HTTP::Daemon', this supports updating the main test results
XML file from individual reports but does not regenerate the
individual report's HTML at this time.

Apache and update-results.pl
----------------------------

Allowing the update-results.pl CGI script to modify files is probably
a security no-no, so it's probably best to use this setup on a machine
that isn't accessible to the wider Internet.

Suggestions on how to improve the setup are welcome, but the best way
that has been found so far is:

1. Copy 'testing.conf' to '/etc/apache2/conf.d' (or make a symbolic
   link to the file).

2. Adjust the paths in the 'testing.conf' copy as necessary to
   match your setup.

   The '--with-project-name' and '--with-basename' values are used in
   'testing.conf'.

3. Restart apache, e.g.:

      sudo /etc/init.d/apache2 restart

4. Make the group id sticky (i.e., 'chmod g+s ...') for the current
   directory and all the generated directories.

   Alternatively, a simplistic solution if you have the 'acl' option
   set for the current file system in '/etc/fstab' is:

      setfacl -m u:www-data:rwx .


A previous method that would require rolling back changes in
testsuccess.xsl and update-results.pl is:

1. Create a new group, e.g. 'testing'.

2. Make the user id that runs the web server (e.g., 'apache') a member
   of the group.

3. Make any users who will be running the scripts and stylesheets also
   members of the group.

4. Change the group id of the generated test result files and report
   files to the new group.

5. Change the mode of the generated test result files and report files
   to '664'.

6. Change the group id of the 'testing' directory to the new group.

7. Make the group id sticky (i.e., 'chmod g+s ...') for the current
   directory and all the generated directories.

It is probably simplest to create the group and just set the sticky
group id of the 'testing' directory *before* running any of the
stylesheets or scripts.


9. Testing an Alternative XSL Processor
=======================================

You can test XSL processors other than xmlroff by specifying the
--with-command-patterns parameter to 'configure'.

The value of --with-command-patterns is a list of one or more command
patterns, separated by semicolons.

A command pattern is the format of the command to run the XSL
processor, but with format strings, e.g., '%p', at the points in the
command where the testing system substitutes values.

The format strings are:

   %p   XSL processor

        Specified by --with-xsl-processor.

   %c   XSL processor flags

        Specified by --with-xsl-processor-flags.

        When --with-xsl-processor-flags is not specified, this format
        string is not substituted.

   %i   Input XML file

        Taken from the test definition, and potentially different for
        every test.

   %s   Stylesheet file

        Either taken from the test definition or, when not specified
        in the test definition and --with-default-stylesheet is
        specified, the default stylesheet.

        When neither a per-test stylesheet nor a default stylesheet is
        specified, this format string is not substituted.

   %o   Output file

        The filename is determined by the testing module and is (or
        should be) different for each test.

   %%   Literal '%'

If a command pattern contains unsubstituted format strings, e.g., if
the pattern contains '%c' but --with-xsl-processor-flags is
unspecified, then the command pattern is disregarded and the next
pattern is evaluated.

If none of the command patterns are completely substituted, then
things will probably not work.

For example, the default command patterns value is:

   %p -o %o %c %i %s;%p -o %o %c %i;%p -o %o %i %s;%p -o %o %i

The first pattern is disregarded when either %c or %s is unevaluated;
the second, when %c is unevaluated; the third, when %s is unevaluated;
and the fourth will always be evaluated since it contains no optional
format strings.

This example shows how to configure to test using the Antenna House
Formatter on Linux:

   ./configure --with-xsl-processor=/usr/AHFormatterV6_64/run.sh \
      --with-command-patterns="%p -o %o -d %i"

This example shows how to configure to test using FOP on Linux:

   ./configure --with-xsl-processor=fop \
      --with-command-patterns="%p -r -xml %i -xsl %s -pdf %o;%p -r -fo %i -pdf %o"


10. DocBook testdocs
===================

The DocBook SourceForge project maintains a set of test documents.

At the time of this writing, you have to check them out from the
DocBook Subversion repository if you want to use them:

   svn co \
      https://docbook.svn.sourceforge.net/svnroot/docbook/trunk/testdocs \
      testdocs

This checkout doesn't get you any 'callout' symbols that are usable
by the test documents.  The DocBook testdocs fit in with the rest of
the SourceForge DocBook project's setup, so you need to fake some of
the DocBook setup for the callout graphics to work:

   cd testdocs/tests
   svn co \
      https://docbook.svn.sourceforge.net/svnroot/docbook/trunk/xsl/images \
      images

The testdocs checkout also does not include a test definitions file
(i.e., a 'testsuite.xml' file) that is needed by this module, so you
need to make your own.

'make-testsuite.pl' in this module generates a usable 'testsuites.xml'
that includes a '<testcase>' element for each XML file in
subdirectories of the specified directory.  Since testdocs currently
includes some XML files that aren't DocBook files, some of the
generated test cases will always fail; you can edit those out of the
generated 'testsuite.xml', but they will be regenerated if you ever
regenerate the 'testsuite.xml' after updating your testdocs checkout.

The DocBook testdocs need a stylesheet if they are to be formatted
(which, in a nutshell, is the whole point of DocBook and the DocBook
SourceForge project).  There are two alternatives for specifying the
stylesheet:

 - If you are also testing FO documents in the same run, e.g., also
   testing the xmlroff testsuite, then you need to specify the
   stylesheet for each testcase by specifying it when running
   'make-testsuite.pl':

      perl ../testing/make-testsuite.pl \
         --stylesheet \
         http://docbook.sourceforge.net/release/xsl/current/fo/docbook.xsl \
         . > testsuite.xml

 - Alternatively, if you are testing only DocBook documents, you can
   specify the DocBook stylesheet as the default stylesheet when
   congiguring testing:

      perl ../testing/make-testsuite.pl . > testsuite.xml
      cd ../testing
      ./configure \
         --with-default-stylesheet=http://docbook.sourceforge.net/release/xsl/current/fo/docbook.xsl

   This could be useful if you want to test different versions of the
   DocBook stylesheets.


11. FOP "layoutengine" Tests
============================

make-fop-testsuite.pl creates a W3C-format testsuite file from FOP
"layoutengine" tests.

Usage:

   make-fop-testsuite.pl [--verbose] [--stylesheet Stylesheet] \
      [--title Title] TestsDir

where:

  Title         = Profile name in the generated testsuite
  Stylesheet    = Filename or URL of stylesheet for all tests
  TestsDir      = Directory containing the tests directories


12. Setting Defaults in the Test Results File
=============================================

Configuring a test run using many non-standard parameters -- e.g. when
testing an alternative XSL processor or testing a specific xmlroff
backend and output format -- can make a long 'configure' command line
that is not so easy to reconstruct.

Also, when you redo a test run with the same parameters, you typically
want to update the test results from the last run to reflect any
changes in the test output.

Since the test results are so closely related to the test parameters
used for the test run, this module supports storing some parameters as
processing instructions in the test results file and using those
parameters when setting up to re-run the tests.

The 'configure' options and their corresponding PIs (with "xt-" prefix
meaning "xmlroff testing") are:

 --with-basename              xt-basename
 --with-color                 xt-color
 --with-command-patterns      xt-command-patterns
 --with-default-stylesheet    xt-default-stylesheet
 --with-project-name          xt-project-name
 --with-server-port	      xt-server-port
 --with-test-subset           xt-test-subset
 --with-testsuites            xt-testsuites
 --with-title 		      xt-title
 --with-trac                  xt-trac
 --with-xsl-processor         xt-xsl-processor
 --with-xsl-processor-flags   xt-xsl-processor-flags

The options without a PI are:

 --with-changelog-oldest
 --with-convert
 --with-compare
 --with-composite
 --with-debug
 --with-identify
 --with-test-results
 --with-xsltproc

The priority for setting a parameter, from lowest to highest, is:

 - Default hardwired into 'configure'
 - PI in test results XML
 - 'configure' command-line options

When you run 'configure', the 'configure' script outputs a summary of
the settable parameters and where the parameter was set ('default',
'results', or 'option').


13. Gory Details
================

Test Cases
----------

The XSL Working Group (WG) had to demonstrate that every feature of
XSL had been implemented before XSL 1.0 advanced from Candidate
Recommendation (CR) to Proposed Recommendation (PR).

In 2001, the WG and NIST (National Institute of Science and Technology
in the U.S.) developed a DTD for both test suite descriptions and test
suite results, and NIST and the XSL formatter implementors contributed
both test suites and their descriptions.  The DTD was augmented with
more attribute valuess for new formatting objects for the XSL 1.1 CR. See
http://www.w3.org/Style/XSL/TestSuite/.

When working on the XSL 1.0 CR test suite results with Max Froumentin
of the W3C, we developed a simple, well-formed document type for
referring to multiple test suite descriptions.

The XSL 1.0 CR test suite was put on the web.  NIST continued to
develop tests and later produced an expanded set of test cases.  See
http://xw2k.sdct.itl.nist.gov/brady/xml/generate.asp?tech=../../carmelo/XSL.

The test cases currently available for testing xmlroff are the XSL 1.0
CR test cases, the NIST test cases, and some test cases developed
specifically for xmlroff.

make-fop-testsuite.pl creates a W3C-format testsuite file from FOP
"layoutengine" tests.


config.h, config.pl, config.xsl
-------------------------------

These specify parameters used by the XSLT stylesheets and Perl
programs that actually do the work.

The other XSLT stylesheets import config.xsl.

You can still override any of these parameters by specifying the
parameter on the xsltproc command line.

Some of the files that you specify in these parameters won't
necessarily exist the first time that you run the stylesheets.


test2sh.xsl
-----------

This should be run manually whenever the test cases are modified.

Usage:

   xsltproc --novalid test2sh.xsl test2sh.xsl

Applicable 'configure' parameters are:

 - '--with-testsuites' sets the filename of the XML document referring
   to multiple test suite descriptions.

 - '--with-test-subset' restricts tests to those testcases with a
   matching @profile value.

The stylesheet expects that the current directory will have a
subdirectory with the same name as the subdirectory of each of the
test suite description files referred to by the file referred to by
the 'testsuites' parameter.  I.e., if 'testsuites.xml' refers to
'../NIST/testsuite.xml', then test2sh.xsl expects to be able to write
to a 'NIST' subdirectory of the current directory.

In each subdirectory corresponding to the subdirectories of the
referred-to test suite descriptions, test2sh.xsl writes a 'test.sh'
file containing the shell commands to run xmlroff on each test case
described in the test suite description.

For each test case, test2sh.xsl writes shell commands to:

 - Run xmlroff on the test case input file or files, with output
   written to a PDF file named after the input file and error and log
   output written to a .log file also named after the input file.

The 'test.sh' file also runs a simple command-line Perl script on the
log files to remove process IDs from the messages in the .log files so
the files can be meaningfully diffed.


test2makefile.xsl
-----------------

This is a work in progress for letting Make work out when a test needs
to be redone.

Also used for copying testsuite samples to reference directory, and
vice-versa.


dotests.pl
----------

This Perl script executes the shell scripts created by test2sh.xsl.

It could be run nightly as part of a cron job.

Usage:

   perl dotests.pl .

This script is run from the directory containing the subdirectories
with the test.sh shell scripts.

The Perl script scans the subdirectories of the current directory for
test.sh files.

In each subdirectory containing a test.sh file, the Perl script:

 - Runs test.sh, with output written to 'test.out' in the current
   subdirectory.


png+ref2stereo.pl
-----------------

Diff the generated PDF against the corresponding reference PDF in the
'ref' directory, writing the diff output to the 'diff' directory.

If a PDF file has changed, run ImageMagick's 'convert' program to make
a PNG file of each page of the PDF and then diff the new PNG with the
corresponding reference PNG in the 'ref' directory.

Potentially runs ImageMagick's 'composite' program with the 'stereo'
argument on each generated PNG file for which there is a corresponding
PNG file in the 'ref' subdirectory.  The output, which is written to a
like-named file in the 'stereo' subdirectory, shows the differences
between the generated file and the reference version in the 'ref'
subdirectory.

The script only runs 'composite' if the 'diff' file size is non-zero
or missing and the 'diff' file is newer than the PNG, etc.


testsuccess.pl
--------------

This Perl script is run after dotests.pl.

Usage:

   perl testsuccess.pl . > testsuccess.xml

It generates an XML file that lists file names and file sizes of test
result files.  That XML file is used as input to the XSLT stylesheet
that generates the HTML reports.  Using this script saves having to
worry about trying to find and use XSLT processor-specific extensions
for accessing file details.

testsucess.pl scans the subdirectories containing the test results and
outputs a well-formed XML document listing the file names and file
sizes for all the files relating to each PDF file.


testresults.xml
---------------

This is the record of the processor's results for each test.  It
conforms to the DTD from the XSL WG.

You can specify a different results file using the --with-test-results
option of 'configure'.

This file should have a new entry added to it for every new test case
added to the test suite.


testsuites2testresults.xsl
--------------------------

This generates a new testresults.xml file from a testsuites.xml file.

The stylesheet doesn't preserve anything from an existing
testresults.xml file, so it's best to write the output to a temporary
file and then merge any new entries into an existing testresults.xml
file.

Usage:

   xsltproc --novalid testsuites2testresults.xsl testsuites.xml > tmp.xml


testsuccess.xsl
---------------

This is run after testsuccess.pl has generated the XML file listing
the test result file names and file sizes.

It generates HTML reports: one *large* report summarizing all the test
cases and their results, plus an individual report for each test case.

Usage:

   xsltproc -o testsuccess.html --novalid \
      --stringparam date "`date`" \
      testsuccess.xsl testsuccess.xml

The parameters controlling the transformation are set when you run
'configure'.

 - '--with-testsuites' sets the filename of the XML document referring
   to multiple test suite descriptions.

 - '--with-test-results' sets the filename of the XML document
   recording the test results.

 - '--disable-individual' disables generating an individual report for
   each test result

 - '--disable-form' disables generating a form in individual reports

 - '--disable-reference' disables including the reference and stereo
   images in individual reports

 - '--with-trac' sets the base URL for linking from ticket references
   (e.g., '#21') in test result descriptions to tickets in a Trac
   (e.g., http://xmlroff.org/ticket/21).

   '--without-trac' disables creating links for ticket references.

 - '--with-color' sets the base colour for the reports.  The colour
   and lighter shades of the colour are used for backgrounds in the
   generated HTML.

The summary report (and the individual reports) include information
from both the test case description and the xmlroff test results.

The test case description information is obtained by following file
references in the file referred to by the '--with-testsuites' option
to individual test suite description files.

The xmlroff test results information is obtained from both
testresults.xml and testsuccess.xml.

The summary report is a large table with one row for each test case.
Each row contains:

 - The test case ID.

 - An indication of the value of the test result attributes from
   testresults.xml for the test.

 - The test result's descriptive text from testresults.xml.

 - A link to the PDF output, if any.

 - A link to the report HTML for the individual test.

 - An indication of the number of output pages and, when individual
   reports are being produced, the types of the result files present.


result-doc.xsl
--------------

Imported by testsuccess.xsl and used to generate individual report
HTML for each test.

The individual reports contain:

 - The "path" to the test case

 - The test case information:

    - Test description

    - Which FO is being tested, which could be 'multiple'

    - Input XML file, which could be an FO file

    - Input XSLT stylesheet, if any

    - Sample result file, if any provided by test case author

 - The test result information:

    - Whether or not the test result agrees with the test case
      description and/or sample results

    - Whether or not the XSL formatter will in the future support the
      FOs and properties used in the test

    - Whether or not any differences between the test result and the
      test case description and/or sample results indicates a problem
      with interpreting the XSL 1.0 Recommendation

    - Whether or not any differences between the test result and the
      test case description and/or sample results indicates a problem
      with the test.

    - Any further textual description

   The test result information is usually presented as an HTML form so
   the results can be updated from the individual report.  Use the
   --disable-form option of 'configure' to generate reports with
   static tables instead of forms.

 - For each page in the test result PDF:

    - The PNG for the page

   and, unless --disable-reference was specified when running
   'configure':

    - If there are differences between the current PNG and the
      corresponding 'ref' PNG:

       - The PNG for the page from the 'ref' subdirectory

       - The PNG for the 'stereo' composite showing the differences
         between the two PNGs

    - If, instead, there are no differences between the current PNG
      and the corresponding 'ref' PNG:

       - text stating 'Reference is identical to current.'

Selecting the 'Update' button in the 'Result' form both updates the
test result information in testresults.xml and copies the current PNG
files to the 'ref' subdirectory.  Unfortunately, it does not update
the form because the current XSLT stylesheet generates both the
summary report and all the individual reports, all of which takes a
*long* time.  Ideally the summary report and the individual reports
should be XHTML that can be simply modified, not regenerated, whenever
a test's results are updated.

The summary report should also be broken into multiple HTML files, one
or more for each test suite, but that also hasn't been done yet.


update-results.pl and update-result.xsl
---------------------------------------

update-results.pl is the CGI script called when the 'Update' button in
an individual report is selected.  It modifies testresults.xml using
the update-result.xsl stylesheet, copies the current test result's PNG
files to its 'ref' subdirectory, zeroes the corresponding files in the
'diff' subdirectory, and removes the corresponding files in the
'stereo' subdirectory.


14. License
===========

The stylesheets and other content are licensed under a BSD license.
See the file COPYING for details.


Tony Graham
1 October 2011