inputs/dev/tests_policy.tex

%%%%%%%%%%%%%%%%%%%
% XLiFE++ is an extended library of finite elements written in C++
%     Copyright (C) 2014  Lunéville, Eric; Kielbasiewicz, Nicolas; Lafranche, Yvon; Nguyen, Manh-Ha; Chambeyron, Colin
%
%     This program is free software: you can redistribute it and/or modify
%     it under the terms of the GNU General Public License as published by
%     the Free Software Foundation, either version 3 of the License, or
%     (at your option) any later version.
%     This program is distributed in the hope that it will be useful,
%     but WITHOUT ANY WARRANTY; without even the implied warranty of
%     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
%     GNU General Public License for more details.
%     You should have received a copy of the GNU General Public License
%     along with this program.  If not, see <http://www.gnu.org/licenses/>.
%%%%%%%%%%%%%%%%%%%

\section{Test families and test functions}

\subsection{About test families}

There are several families of tests:

\begin{description}
\item[unit:] Unitary tests are aimed to test each function of each class separately. In fact, it is easier to say it than to do it. But with a very large set of unitary tests, one can avoid most of bugs.
\item[sys:] System tests are aimed to test some user applications but with size of the linear systems not too big.
\item[dev:] Developer tests are aimed to test new functionalities someone is developing. These tests are often like system tests, but are supposed to be deleted as soon as functionalities are fully developped. This family can also be used to check error convergence and cputime for larger problems.
\item[ext:] External tests are aimed to test wrappers to external libraries, such as \arpack and \umfpack.
\end{description}

Each family correspond to a sub-directory of {\em tests}.

\subsection{Definition of a test function}

A test function is a usual c++ function with a meaningful name \textbf{beginning by the family name it belongs to} defined in a file with the same name. The test function takes an optional boolean as input argument and returns a string (String class). For instance, for the unitary tests of the class Function, the file \emph{unit\_Function.cpp} in the \emph{unit} directory looks like:
\begin{lstlisting}[]{}
/*! \file unit_Function.cpp
    \author XXX
    \since YYY
    \date  ZZZ
*/
#include "xlife++-libs.h"
#include "testUtils.h"

String unit_Function(bool check=true)
{String report="unit_Function : OK";
 ...
 return report;
}
\end{lstlisting}
\vspace{.2cm}
and for the system test SauterSchwabIM:
\begin{lstlisting}[]{}
/*! \file sys_SauterSchwabIM.cpp
    \author
    \since
    \date
*/
#include "xlife++-libs.h"
#include "testUtils.h"

String sys_SauterSchwabIM(bool check=true)
{String report="sys_SauterSchwabIM : OK";
 ...
 return report;
}
\end{lstlisting}
\vspace{.2cm}
The string report has to contain information about the errors occurring during the test in
order to easily locate them. There is no writing format of the errors except \textbf{the first
line of the report} which has to be of the form:
\begin{itemize}
\item if the test succeeds: \textbf{name\_of\_the\_test : OK},
\item if the test fails: \textbf{name\_of\_the\_test : \emph{n} error(s)} with \emph{n>0}
the number of errors.
\end{itemize}
\textbf{It is up to the developers to write meaningful tests and more exhaustive tests of the
class functionalities as possible.}

\subsection{Development of a test function}

There are mainly two approaches to make a test:
\begin{itemize}
\item internal value testing: compare results to expected results inside the function (value
comparison),
\item external value testing: comparing results to reference results get previously and stored
in a file (ascii comparison).
\end{itemize}
The first one is better than the second one for two reasons: in one hand it avoids false positive
answers due to change of comment or variable answers such as the value of a pointer and in
the other hand it allows to distinguish some variations due to rounding errors which are probably
meaningless errors. Note that it is possible to use the file comparison if the file contains
only numerical values or if you read it in a way to recover meaningful values.\\

For instance, the internal value testing SauterSchwabIM test looks like:
\begin{lstlisting}[]{}
String sys_SauterSchwabIM(bool check=true)
{String report="sys_SauterSchwabIM : OK";
 ...
 real_t nSol= ...                //compute the norm of the solution
 real_t nSolExpected=0.12345678; //expected norm get previously
 real_t diff=abs(nSol-nSolExpected);
 if(diff>tol)                    //tol is a tolerance (10*epsMachine)
   {report="sys_SauterSchwabIM : 1 error";
    report+="\n * error 1 : difference between norm of the solution are different,";
    report+="'difference="+tostring(diff);
    return report;
   }
 if(diff!=0)     //almost the same
   {report="sys_SauterSchwabIM : 1 error";
    report+="\n * error 1 : norm of the solution are almost the same,";
    report+="difference="+tostring(diff);
    report+=" possibly a rounding error";
   }
 return report;
}
\end{lstlisting}
\vspace{.2cm}
and the external value testing looks like:
\begin{lstlisting}
String sys_SauterSchwabIM(bool check=true)
{String report="sys_SauterSchwabIM : OK";
 ...
 real_t nSol= ...                //compute the norm of the solution
 std::stringstream out;
 out<<"norm of the solution ="<<nSol;
 if(!check)   //save the result in a file
   {std::ofstream fout("sys_SauterSchwabIM.res");
    fout<<out;
    fout.close();
    report="sys_SauterSchwabIM : results save to file sys_SauterSchwabIM.res";
   }
  else      //compare the result
   {ifstream fin("sys_SauterSchwabIM.res");
    String resf;
    fin>>resf;
    if(resf!=out)     //lines are different
      {report="test_Laplace3D : 1 error";
       report+="\n * error 1 : difference between the lines";
       report+="\n   ref : "+resf;
       report+="\n   new : "+out;
      }
    return report;
   }
}
\end{lstlisting}
\vspace{.2cm}
This second approach has the advantage to have a unique file to construct the checked values
(check=false) and to performs the regression test (check=true)
but it is more sensitive. In order to help the developers, some useful functions are proposed
in the \emph{testUtils.cpp} file:
\begin{lstlisting}[]{}
String saveResToFile(std::stringstream& result,const String& rootname);
String diffResFile(std::stringstream& result,const String& rootname);
\end{lstlisting}
\vspace{.2cm}
where \emph{result} is a stringstream containing all the output of a test and \emph{rootname}
is the name of the test function. The function \emph{saveResToFile} write the results of the
test to the file \emph{rootname.res} and the function \emph{diffResFile} compares the results
of the test with the file, line by line. Both, the functions produce a compliant report. Using
the function, the previous example becomes:
\begin{lstlisting}[deletekeywords={[3] check}]
String sys_SauterSchwabIM(bool check=true)
{String report="sys_SauterSchwabIM : OK";
 ...
 real_t nSol= ...                //compute the norm of the solution
 std::stringstream out;
 out<<"norm of the solution ="<<nSol;
 String rootname="sys_SauterSchwabIM";
 if(check) return diffResFile(out,rootname);
 else      return saveResToFile(out,rootname);
}
\end{lstlisting}
\vspace{.2cm}
The external value testing is rather dedicated to the tests of classes and functionalities
which involve a lot of basic tests.

\subsection{Includes}

As you may have noticed in previous listings, a test file, whatever its family always includes 2 headers: {\em xlife++-libs.h} and {\em testUtils.h}.

\section{Test of a new version of the library}

This section addresses the process of global test involved before a minor or major upgrade
of the library. It concerns the administrators of the library.

\subsection{Compilation of tests}

When compiling tests, each individual test is compiled, but also one global test per family. There is also a specific global test running both unit and sys test families. When compiling a global test, \cmake sources all files in the corresponding directory prefixed by the family. If a source file does not respect this naming convention, it is ignored.

\subsection{Running tests}

The administrators have a special tool to run a lot of tests, \emph{xlifepp\_test\_runner.rb}. This is a {\tt Ruby} script that runs one executable without compiling it if it does not exist.

\begin{lstlisting}[language=]
xlifepp_test_runner.rb deals with all automatic tasks such as updating doxygen, generating
    commit history and generating snapshots and releases

SYNOPSIS:
    xlifepp_test_runner.rb -c [-n] <test>
    xlifepp_test_runner.rb -e [-n] <test>
    xlifepp_test_runner.rb -h

DESCRIPTION:
    XLiFE++ is a C++ library with a large set of tests. Each test can be launched in 2 modes :
      - the edit mode (default in XLiFE++ executables) makes tests generating results file, so called res file
      - the check mode (default in xlifepp_test_runner.rb) makes tests comparing results to the current res file

    As XLiFE++ compilation is based on CMake, it is not a good way to define running targets, because
    of generators (to Eclipse, XCode, Visual Studio, CodeBlocks, ...). On the one hand, the lesser targets
    there are the easier XLiFE++ is to compile and run, on the other hand targets in generators are not just
    compilation targets: you can run them, and pass parameters to them.
    However, XLiFE++ developers using cmake in command-line mode must have an easy way to run tests.
    That's what xlifepp_test_runner.rb is for !!!

OPTIONS:
    -h, --help                     shows the current help
    -c, --check                    test will be run in check mode (default)
    -cxx, --cxx-compiler           sets the compiler to use
    -e, --edit                     test will be run in edit mode
    <test>                         name of the test (Examples: all, unit_Mesh_gmsh, sys_1d, ...)
    -n                             activates dry-run mode
    -v                             activates verbose level (same as --verbose-level 1)
    --verbose-level <num>          allows to set the verbose level. Default is 0, none.
\end{lstlisting}