This directory contains useful test code for testing various bits
of Apache functionality.  This stuff is for the developers only,
so we might remove it on public releases.

Apache httpd pytest suite
=========================
Using pytest (<https://docs.pytest.org/en/6.2.x/>) and a Python >= 3.8
for a more flexible testing of Apache httpd.

Install
-------
If not already installed, you will need to install 'pytest' and 'OpenSSL' for
python:
> apt install python3-pip
> pip install -U pytest
> pip install -U pyopenssl

And for 'h2load':
> apt install nghttp2-client


Usage
-----
In your httpd source checkout, do:

> make install
> pytest

and all tests defined run on the installed executable and modules.
> pytest test/modules/core
runs all the core tests. You can run tests also by name selection:
> pytest -k test_core
runs all test cases starting with 'test_core'. Similar:
> pytest -k test_core_001_04
runs the test cases starting with that.

Test output gets more verbose, by adding one or several '-v'. This
also raises the error log level of the tested modules.
> pytest -vvv -k test_h2_004_01
run the specific test with mod_http2 at log level TRACE2.

By default, test cases will configure httpd with mpm_event. You
can change that with the invocation:
> MPM=worker pytest test/modules/http2

Some tests rely on additional tools installed in your environment
and will 'skip' if those are not present. In a non-verbose run,
these will appear as 's' in the output. If you run pytest more
verbose, the skipped test cases will mention a reason for why
they are disabled.

For example, most tests in test/modules/md require an installation
of 'pebble', an ACME test server, and look for it in $PATH.


Workings
--------
All tests start httpd on their own and try hard to shut it down
afterwards. You can abort tests with ^C and they clean up.

httpd is started/stopped repeatedly in testing as configurations
for individual test cases are changed. This is a main difference to
the Perl test framework which starts the server with all possible
combinations configured that are needed by tests.

In test/gen/apache a server root is created with config, logs and htdocs
directories. test/gen/apache/logs/error_log will be the log.
Configs start in test/gen/apache/conf/httpd.conf. modules.conf is
dynamically created for the list of modules that a test suite needs.

Test cases write their specific setup in test.conf and reload/restart
the httpd process. This makes for a small configuration in a test case.


Development
-----------

Adding a test in an existing file is done by adding a method. Its name
must start with 'test_' and the common practise is to have the name
of the test suite there as well. All http2 tests start with 'test_h2_'.

Following this can be any characters. If you make test cases of a
certain feature with a common prefix, it is easier to invoke just
them using the '-k' selector on the command line.

You can also add just a new file to a test suite, if you do a new
range of test cases that do not fit anywhere else. A very simple
one is found in test/modules/http2/test_001_httpd_alive.py.

There is a python class defined with 2 methods. One is the test
method itself and the other one ' is

    @pytest.fixture(autouse=True, scope='class')
    def _class_scope(self, env):
        code

is marked as a pytest 'fixture'. This is some pytest magic.
'autouse=True' means that this fixture is run, even though
no test case uses it directly. scope='class' means that it
is run once for all test cases in this class.

As you see, this fixture gets a parameter named 'env' and
that is the name of another pytest fixture, defined in the
file 'conftest.py' in the same directory.

    @pytest.fixture(scope="package")
    def env(pytestconfig) -> H2TestEnv:
        code

This one runs one time per 'package', meaning for all test
cases in this directory. And it gets the 'pytestconfig'
as parameter which is a standard pytest fixture.

So, when you run 'pytest -k test_h2_004', pytest will look
at _all_ test cases defined and collect those with that
prefix. For each directory with test cases found, it will
process the 'conftest.py', boot-strapping the 'env' fixture,
and the process the files with active test cases.

As a result, if you invoke just a single test case, only
the fixtures needed for that test case are created. This
gives good turn around times when debugging a test case.

If you want to add a new test suite, create a new directory.
Add the files '__init__.py', 'conftest.py' and a first
'test_suitename_something.py'. test/modules/core is the
simplest example. 'test/modules/http2' shows how you load
other modules. 'test/modules/md' checks and starts external
processes (an ACME test server).


Infrastructure
--------------
The test cases rely on the classes provided in 'test/pyhttpd'
for common code in managing a httpd test instance and do
provide some base setups:
- a small set of virtual hosts with some files
- access to paths and definitions (env fixture)
- start/stop httpd and inspect the error log
- run clients like curl and nghttp
- create certificates for your hosts and make curl use
  the root certs (so no --insecure calls by curl).

Variables
---------

The Travis scripts use the following environment variables:

* APR_VERSION - if set, APR of this version is built and installed in
  $HOME/root/apr-$APR_VERSION - a value of "trunk" means trunk is
  used, "*.x" means a branch, otherwise a tagged version is implied.

* APR_CONFIG - arguments to pass when running APR's configure script
  if APR_VERSION is set

* APU_VERSION - if set, APR-util of this version is built and
  installed in $HOME/root/apr-util-$APU_VERSION - a value of "*.x"
  means a branch, otherwise a tagged version is implied.  (Since there
  is no "trunk" for apr-util, that value cannot be used here.)

* APU_CONFIG - arguments to pass when running APR-util's configure
  script if APU_VERSION is set

* CONFIG - arguments to pass to httpd's configure script.

* BUILDCONFIG - arguments to pass when running httpd's ./buildconf script

* MFLAGS - arguments to pass when running "make" for httpd.

* SKIP_TESTING - if set, the Perl test framework is not run for the
  build.

* TEST_UBSAN - set for job using UBSan ("Undefined Behaviour Sanitizer")

* TEST_MALLOC - set for job using enhanced malloc debugging.

* TEST_INSTALL - set for job testing "make install"

* TEST_VPATH - set for job testing srcdir!=builddir

* TEST_LDAP - set for job with slapd, running LDAP tests

* TEST_SSL - set for job with SSL/TLS testing variants

* TESTS - a list of Perl framework tests to run

* TEST_ARGS - arguments to pass to ./t/TEST in the Perl test framework

Caching
-------

Perl modules installed in $HOME/perl5 are cached.

Anything installed into the $HOME/root directory is cached - notably,
versions of APR/APR-util are installed here and cached across httpd
build jobs without needing to be rebuilt every time.

TODO list
---------

* MacOS build
* Windows build
* clang-on-Linux build
* Use containers for non-Ubuntu-based Linux testing
* sanity checks for use of APLOGNO() - empty arguments, accidental duplicates, etc.
 - not sure how exactly
* Known test failures
 - "apt-get install" timeout/fails - workaround by moving apt install to before_script phase?

Testing from a Feature Branch
-----------------------------

An SVN branch off trunk should be mirrored to github, and will be
tested in the same way that trunk is in Travis, so this workflow is
available for those familiar with using Subversion and the standard
ASF/httpd repository layout.

Tested branches are listed at: https://travis-ci.org/github/apache/httpd/branches

Travis will also run the tests for a PR filed against the httpd Github
repository at https://github.com/apache/httpd or from a fork of this
repository if enabled for the Travis user.

A workflow to enable testing would be as follows, substituting
$USERNAME for your github username:

  $ git clone https://github.com/apache/httpd
  $ cd httpd
  $ git remote add $USERNAME git@github.com:$USERNAME/httpd.git
  $ git checkout -b my-feature origin/trunk
  ... do some work ...
  $ git commit ...
  $ git push -u $USERNAME my-feature:my-feature

To enable testing for a fork, visit the settings page at
https://travis-ci.org/$USERNAME/httpd/settings - you may need to sync
your account via https://travis-ci.org/account/repositories for a
freshly created fork.

To create a Pull Request, go to a URL like:
https://github.com/apache/httpd/compare/trunk...$USERNAME:trunk

Once a PR has been created, travis will run the tests and link the
results from a PR comment. All tested PRs are listed here:
https://travis-ci.org/github/apache/httpd/pull_requests

To merge from github back to SVN trunk, create a patch from e.g.:

  $ git diff origin/trunk..my-feature

and then apply it in SVN.  To rebase a feature once trunk has
diverged, from a feature branch run:

  $ git pull
  $ git rebase -i origin/trunk

and follow the standard rebase steps.