1Octopus testsuite
2
3== Running tests ==
4
5* To run the testsuite, type "make check" in this directory or the main directory. "make check-full" will do the same but also include tests marked 'long-run'.
6* The runs will use MPI if the code was compiled with MPI, and either `which mpiexec` or the $MPIEXEC environment variable leads to a useable MPI.
7* When running with MPI, the tests use up to 5 MPI tasks. Therefore you should typically have at least 4 cores available.
8* To run in parallel with a scheduler, e.g. on a cluster or supercomputer with PBS, create an appropriate job script for the machine you are using, along the lines of the example below.
9
10Example job script for hopper.nersc.gov:
11
12#!/bin/bash
13#PBS -N pulpo
14#PBS -m ae
15#PBS -q regular
16#PBS -l mppwidth=48
17#PBS -l walltime=6:00:00
18
19cd $HOME/hopper/octopus
20echo $PBS_O_WORKDIR
21export MPIEXEC=`which aprun`
22
23export TEMPDIRPATH=$SCRATCH/tmp
24export OCT_TEST_NJOBS=15
25make check-full &> $PBS_O_WORKDIR/makecheck-full
26
27* Environment variables:
28 - TEMPDIRPATH: sets the scratch directory where temporary working directories will be created. Default is /tmp, but on some machines you may not have permission to write there.
29 This variable also determines the directory that will be automatically cleaned of files older than 10 hours by 'make clean' in the testsuite directory.
30 - MPIEXEC: sets the command for parallel runs. Default is `which mpiexec`. Note that mpiexec and mpirun are generally equivalent (except for Intel MPI, in which case mpirun is recommended). Set this if you are using a different command such as 'ibrun' (SGE parallel environment), 'runjob' (BlueGene), 'aprun' (Cray), or 'poe' (IBM); if you need to use a different mpiexec than the one in your path; or if some options need to be appended to the command. Depending on the value of this variable, four styles of command line are used: ibrun, runjob, mpirun/aprun, and poe.
31 - OCT_TEST_NJOBS: number of tests to run concurrently for job parallelism. Used by oct-run_testsuite.sh.
32 - OCT_TEST_MPI_NPROCS: set to overrule the number of processors listed in the test files. Useful for testing correctness of parallelization or if you don't have 4 cores. Some tests, however, may fail with an error because they set an explicit number of MPI tasks for each parallelization strategy.
33 - MACHINELIST: if set, will be added to command line for MPI runs. This is needed in some MPI implementations.
34 - EXEC: if set, will be added before name of executable. Used to run code through valgrind.
35* To run just one test, in its directory, run "../oct-run_regression_test.pl [-p] -D ../../bin -f [testname].test". Include the -p flag to preserve the working directory.
36
37Contents:
38(1) oct-run_testsuite.sh -- this script runs tests from filenames ending in *.test within this directory.
39(2) oct-run_regression_test.pl -- called from oct-run_testsuite.sh for each individual test.
40(5) queue_monitor.pl -- a Perl script for parallel tests with a SLURM or PBS scheduler, which submits a job and monitors the queue to see what happens. Used by the BuildBot. Not actually present in current distributions.
41(6) test directories.
42(7) fix_testsuite.py -- (for developers) can be used to create or update match reference values in test files.
43(8) no_hires.sh -- Some clusters will not have the Perl package Time::HiRes installed which is needed for timing in oct-run_regression_test.pl. You can just deactivate the usage with this script. Not actually present in current distributions.
44(9) buildbot_query.pl -- (for developers) can be used to find the range of values obtained on the buildslaves for a particular match, to set values in the test files.
45
46== Writing tests ==
47
48The test files consist of lines beginning with one of a set of tags, parsed by the Perl script.
49
50# A comment line. Will be ignored.
51Test : title
52 Write a title to output to identify the test. Should be the first tag in the file.
53Options:
54 Specify that this test can only be run if the code was compiled with certain options, e.g. scalapack, netcdf, etc. The code is run with "-c" argument to query for options.
55Options_MPI:
56 Like Options, except only applies if the code is being run with MPI, and overrules the Options tag. This is used when a parallel run would require e.g. scalapack or mpi2, but a serial run does not have any option requirement.
57Program :
58 Which executable will be used. Typically either "octopus" for normal runs or "oct-test" for special test runs. Anything that does not run properly for both serial and parallel executables is not acceptable.
59TestGroups : group-name, [group-name2, [...]]
60 The oct-run_testsuite.sh script can be run with argument "-g" and a list of groups. Then tests will only be run if there is a match between the argument list and the list in TestGroups. Current groups: each directory ("components", "finite_systems_1d", etc.), "long-run" (included only "make check-full"), and "short-run" (everything that is not "long-run". [This tag is actually read by oct-run_testsuite.sh rather than oct-run_regression_test.pl.]
61Enabled : Yes/No
62 If Yes, will be run; if No, will not be run. Use to turn off a test without deleting it from the repository.
63Processors : integer or "serial"
64 Number of processors to use. Default is 2. Ignored if mpiexec is not available. Should not be more than 5 under current guidelines. Will be run without MPI if "serial" is given.
65Precision : 1e-4
66 A floating point number, the tolerance for testing whether a match has passed or failed. Persists until next Precision tag. Default is 1e-4.
67Util : util-name
68 Perform a run of a utility in serial (e.g. oct-dielectric_function).
69Input : file-name
70 Perform a run, after copying the input file to working directory.
71match ; name ; COMMAND(..); reference-value
72 Extracts a calculated number from a run and tests it against the reference value. The name is an identifier printed to output. The number is extracted as the standard output from the listed COMMAND, after filtering to remove non-numeric results. Note that an internal comma in the filename can be escaped as '\,'. Also, '_COMMA_' will be converted to ',' and '_COLUMN_' to ';' in any parameters. The COMMAND is one of this set:
73 . GREPFIELD(filename, 'search-regex', field, offset)
74 Finds the first line in file containing 'search-regex' (a regular expression passed to the 'grep' command, and which MAY NOT contain a comma), and returns the specified field of that line. "Field" is meant in the sense used by 'awk', i.e. the line is parsed into white-space separated groups, indexed starting with 1. The optional 'offset' directs the use of that many lines after the line containing 'search-regex'. No offset specified is equivalent to offset = 0. If there are multiple occurrences of 'search-regex', the first one will be used. This is the most robust of the commands to changes in output formatting, and should be used when possible in preference to the others.
75 . GREP(filename, 'search-text', column, offset)
76 Like GREPFIELD except returns the specified column of that line, i.e. a string from the nth character to the next whitespace. Deprecated.
77 . SIZE(filename)
78 Returns the size of the specified file via 'ls -lt'. Useful for binary files whose contents cannot easily be checked.
79 . LINEFIELD(filename, line, field)
80 Returns the specified field of the specified line number from the file. Negative number means from the end of the file. Use GREPFIELD instead if possible.
81 . LINE(filename, line, column)
82 Returns the specified column of the specified line number from the file. Negative number means from the end of the file. Use GREP instead if possible. Deprecated.
83 . SHELL(shell-command)
84 The result is the standard output of the listed command. Deprecated; use GREP(FIELD) or LINE(FIELD) unless absolutely necessary.
85