1%%%%%%%%%%%%%%%%%%% 2% XLiFE++ is an extended library of finite elements written in C++ 3% Copyright (C) 2014 Lunéville, Eric; Kielbasiewicz, Nicolas; Lafranche, Yvon; Nguyen, Manh-Ha; Chambeyron, Colin 4% 5% This program is free software: you can redistribute it and/or modify 6% it under the terms of the GNU General Public License as published by 7% the Free Software Foundation, either version 3 of the License, or 8% (at your option) any later version. 9% This program is distributed in the hope that it will be useful, 10% but WITHOUT ANY WARRANTY; without even the implied warranty of 11% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12% GNU General Public License for more details. 13% You should have received a copy of the GNU General Public License 14% along with this program. If not, see <http://www.gnu.org/licenses/>. 15%%%%%%%%%%%%%%%%%%% 16 17\section{Test families and test functions} 18 19\subsection{About test families} 20 21There are several families of tests: 22 23\begin{description} 24\item[unit:] Unitary tests are aimed to test each function of each class separately. In fact, it is easier to say it than to do it. But with a very large set of unitary tests, one can avoid most of bugs. 25\item[sys:] System tests are aimed to test some user applications but with size of the linear systems not too big. 26\item[dev:] Developer tests are aimed to test new functionalities someone is developing. These tests are often like system tests, but are supposed to be deleted as soon as functionalities are fully developped. This family can also be used to check error convergence and cputime for larger problems. 27\item[ext:] External tests are aimed to test wrappers to external libraries, such as \arpack and \umfpack. 28\end{description} 29 30Each family correspond to a sub-directory of {\em tests}. 31 32\subsection{Definition of a test function} 33 34A test function is a usual c++ function with a meaningful name \textbf{beginning by the family name it belongs to} defined in a file with the same name. The test function takes an optional boolean as input argument and returns a string (String class). For instance, for the unitary tests of the class Function, the file \emph{unit\_Function.cpp} in the \emph{unit} directory looks like: 35\begin{lstlisting}[]{} 36/*! \file unit_Function.cpp 37 \author XXX 38 \since YYY 39 \date ZZZ 40*/ 41#include "xlife++-libs.h" 42#include "testUtils.h" 43 44String unit_Function(bool check=true) 45{String report="unit_Function : OK"; 46 ... 47 return report; 48} 49\end{lstlisting} 50\vspace{.2cm} 51and for the system test SauterSchwabIM: 52\begin{lstlisting}[]{} 53/*! \file sys_SauterSchwabIM.cpp 54 \author 55 \since 56 \date 57*/ 58#include "xlife++-libs.h" 59#include "testUtils.h" 60 61String sys_SauterSchwabIM(bool check=true) 62{String report="sys_SauterSchwabIM : OK"; 63 ... 64 return report; 65} 66\end{lstlisting} 67\vspace{.2cm} 68The string report has to contain information about the errors occurring during the test in 69order to easily locate them. There is no writing format of the errors except \textbf{the first 70line of the report} which has to be of the form: 71\begin{itemize} 72\item if the test succeeds: \textbf{name\_of\_the\_test : OK}, 73\item if the test fails: \textbf{name\_of\_the\_test : \emph{n} error(s)} with \emph{n>0} 74the number of errors. 75\end{itemize} 76\textbf{It is up to the developers to write meaningful tests and more exhaustive tests of the 77class functionalities as possible.} 78 79\subsection{Development of a test function} 80 81There are mainly two approaches to make a test: 82\begin{itemize} 83\item internal value testing: compare results to expected results inside the function (value 84comparison), 85\item external value testing: comparing results to reference results get previously and stored 86in a file (ascii comparison). 87\end{itemize} 88The first one is better than the second one for two reasons: in one hand it avoids false positive 89answers due to change of comment or variable answers such as the value of a pointer and in 90the other hand it allows to distinguish some variations due to rounding errors which are probably 91meaningless errors. Note that it is possible to use the file comparison if the file contains 92only numerical values or if you read it in a way to recover meaningful values.\\ 93 94For instance, the internal value testing SauterSchwabIM test looks like: 95\begin{lstlisting}[]{} 96String sys_SauterSchwabIM(bool check=true) 97{String report="sys_SauterSchwabIM : OK"; 98 ... 99 real_t nSol= ... //compute the norm of the solution 100 real_t nSolExpected=0.12345678; //expected norm get previously 101 real_t diff=abs(nSol-nSolExpected); 102 if(diff>tol) //tol is a tolerance (10*epsMachine) 103 {report="sys_SauterSchwabIM : 1 error"; 104 report+="\n * error 1 : difference between norm of the solution are different,"; 105 report+="'difference="+tostring(diff); 106 return report; 107 } 108 if(diff!=0) //almost the same 109 {report="sys_SauterSchwabIM : 1 error"; 110 report+="\n * error 1 : norm of the solution are almost the same,"; 111 report+="difference="+tostring(diff); 112 report+=" possibly a rounding error"; 113 } 114 return report; 115} 116\end{lstlisting} 117\vspace{.2cm} 118and the external value testing looks like: 119\begin{lstlisting} 120String sys_SauterSchwabIM(bool check=true) 121{String report="sys_SauterSchwabIM : OK"; 122 ... 123 real_t nSol= ... //compute the norm of the solution 124 std::stringstream out; 125 out<<"norm of the solution ="<<nSol; 126 if(!check) //save the result in a file 127 {std::ofstream fout("sys_SauterSchwabIM.res"); 128 fout<<out; 129 fout.close(); 130 report="sys_SauterSchwabIM : results save to file sys_SauterSchwabIM.res"; 131 } 132 else //compare the result 133 {ifstream fin("sys_SauterSchwabIM.res"); 134 String resf; 135 fin>>resf; 136 if(resf!=out) //lines are different 137 {report="test_Laplace3D : 1 error"; 138 report+="\n * error 1 : difference between the lines"; 139 report+="\n ref : "+resf; 140 report+="\n new : "+out; 141 } 142 return report; 143 } 144} 145\end{lstlisting} 146\vspace{.2cm} 147This second approach has the advantage to have a unique file to construct the checked values 148(check=false) and to performs the regression test (check=true) 149but it is more sensitive. In order to help the developers, some useful functions are proposed 150in the \emph{testUtils.cpp} file: 151\begin{lstlisting}[]{} 152String saveResToFile(std::stringstream& result,const String& rootname); 153String diffResFile(std::stringstream& result,const String& rootname); 154\end{lstlisting} 155\vspace{.2cm} 156where \emph{result} is a stringstream containing all the output of a test and \emph{rootname} 157is the name of the test function. The function \emph{saveResToFile} write the results of the 158test to the file \emph{rootname.res} and the function \emph{diffResFile} compares the results 159of the test with the file, line by line. Both, the functions produce a compliant report. Using 160the function, the previous example becomes: 161\begin{lstlisting}[deletekeywords={[3] check}] 162String sys_SauterSchwabIM(bool check=true) 163{String report="sys_SauterSchwabIM : OK"; 164 ... 165 real_t nSol= ... //compute the norm of the solution 166 std::stringstream out; 167 out<<"norm of the solution ="<<nSol; 168 String rootname="sys_SauterSchwabIM"; 169 if(check) return diffResFile(out,rootname); 170 else return saveResToFile(out,rootname); 171} 172\end{lstlisting} 173\vspace{.2cm} 174The external value testing is rather dedicated to the tests of classes and functionalities 175which involve a lot of basic tests. 176 177\subsection{Includes} 178 179As you may have noticed in previous listings, a test file, whatever its family always includes 2 headers: {\em xlife++-libs.h} and {\em testUtils.h}. 180 181\section{Test of a new version of the library} 182 183This section addresses the process of global test involved before a minor or major upgrade 184of the library. It concerns the administrators of the library. 185 186\subsection{Compilation of tests} 187 188When compiling tests, each individual test is compiled, but also one global test per family. There is also a specific global test running both unit and sys test families. When compiling a global test, \cmake sources all files in the corresponding directory prefixed by the family. If a source file does not respect this naming convention, it is ignored. 189 190\subsection{Running tests} 191 192The administrators have a special tool to run a lot of tests, \emph{xlifepp\_test\_runner.rb}. This is a {\tt Ruby} script that runs one executable without compiling it if it does not exist. 193 194\begin{lstlisting}[language=] 195xlifepp_test_runner.rb deals with all automatic tasks such as updating doxygen, generating 196 commit history and generating snapshots and releases 197 198SYNOPSIS: 199 xlifepp_test_runner.rb -c [-n] <test> 200 xlifepp_test_runner.rb -e [-n] <test> 201 xlifepp_test_runner.rb -h 202 203DESCRIPTION: 204 XLiFE++ is a C++ library with a large set of tests. Each test can be launched in 2 modes : 205 - the edit mode (default in XLiFE++ executables) makes tests generating results file, so called res file 206 - the check mode (default in xlifepp_test_runner.rb) makes tests comparing results to the current res file 207 208 As XLiFE++ compilation is based on CMake, it is not a good way to define running targets, because 209 of generators (to Eclipse, XCode, Visual Studio, CodeBlocks, ...). On the one hand, the lesser targets 210 there are the easier XLiFE++ is to compile and run, on the other hand targets in generators are not just 211 compilation targets: you can run them, and pass parameters to them. 212 However, XLiFE++ developers using cmake in command-line mode must have an easy way to run tests. 213 That's what xlifepp_test_runner.rb is for !!! 214 215OPTIONS: 216 -h, --help shows the current help 217 -c, --check test will be run in check mode (default) 218 -cxx, --cxx-compiler sets the compiler to use 219 -e, --edit test will be run in edit mode 220 <test> name of the test (Examples: all, unit_Mesh_gmsh, sys_1d, ...) 221 -n activates dry-run mode 222 -v activates verbose level (same as --verbose-level 1) 223 --verbose-level <num> allows to set the verbose level. Default is 0, none. 224\end{lstlisting} 225