1%% 2%% 3 4\chapter{Bacula Regression Testing} 5\label{TheChapterStart8} 6\index{Testing!Bacula Regression} 7\index{Bacula Regression Testing} 8\addcontentsline{toc}{section}{Bacula Regression Testing} 9 10\section{Setting up Regession Testing} 11\index{Setting up Regession Testing} 12\addcontentsline{toc}{section}{Setting up Regression Testing} 13 14This document is intended mostly for developers who wish to ensure that their 15changes to Bacula don't introduce bugs in the base code. However, you 16don't need to be a developer to run the regression scripts, and we 17recommend them before putting your system into production, and before each 18upgrade, especially if you build from source code. They are 19simply shell scripts that drive Bacula through bconsole and then typically 20compare the input and output with {\bf diff}. 21 22You can find the existing regression scripts in the Bacula developer's 23{\bf git} repository on SourceForge. We strongly recommend that you {\bf 24clone} the repository because afterwards, you can easily get pull the 25updates that have been made. 26 27To get started, we recommend that you create a directory named {\bf 28bacula}, under which you will put the current source code and the current 29set of regression scripts. Below, we will describe how to set this up. 30 31The top level directory that we call {\bf bacula} can be named anything you 32want. Note, all the standard regression scripts run as non-root and can be 33run on the same machine as a production Bacula system (the developers run 34it this way). 35 36To create the directory structure for the current trunk and to 37clone the repository, do the following (note, we assume you 38are working in your home directory in a non-root account): 39 40\footnotesize 41\begin{verbatim} 42git clone http://git.bacula.org/bacula.git bacula 43\end{verbatim} 44\normalsize 45 46This will create the directory {\bf bacula} and populate it with 47three directories: {\bf bacula}, {\bf gui}, and {\bf regress}. 48{\bf bacula} contains the Bacula source code; {\bf gui} contains 49certain gui programs that you will not need, and {\bf regress} contains 50all the regression scripts. The above should be needed only 51once. Thereafter to update to the latest code, you do: 52 53\footnotesize 54\begin{verbatim} 55cd bacula 56git pull 57\end{verbatim} 58\normalsize 59 60If you want to test with SQLite and it is not installed on your system, 61you will need to download the latest depkgs release from Source Forge and 62unpack it into {\bf depkgs}, then simply: 63 64\footnotesize 65\index{Script!Running the Regression} 66\addcontentsline{toc}{section}{Running the Regression Script} 67 68There are a number of different tests that may be run, such as: the standard 69set that uses disk Volumes and runs under any userid; a small set of tests 70that write to tape; another set of tests where you must be root to run them. 71Normally, I run all my tests as non-root and very rarely run the root 72tests. The tests vary in length, and running the full tests including disk 73based testing, tape based testing, autochanger based testing, and multiple 74drive autochanger based testing can take 3 or 4 hours. 75 76\subsection{Setting the Configuration Parameters} 77\index{Setting the Configuration Parameters} 78\index{Parameters!Setting the Configuration} 79\addcontentsline{toc}{subsection}{Setting the Configuration Parameters} 80 81There is nothing you need to change in the source directory. 82 83To begin: 84 85\footnotesize 86\begin{verbatim} 87cd bacula/regress 88\end{verbatim} 89\normalsize 90 91 92The 93very first time you are going to run the regression scripts, you will 94need to create a custom config file for your system. 95We suggest that you start by: 96 97\footnotesize 98\begin{verbatim} 99cp prototype.conf config 100\end{verbatim} 101\normalsize 102 103Then you can edit the {\bf config} file directly. 104 105\footnotesize 106\begin{verbatim} 107 108# Where to get the source to be tested 109BACULA_SOURCE="${HOME}/bacula/bacula" 110 111# Where to send email !!!!! Change me !!!!!!! 112EMAIL=your-name@your-domain.com 113SMTP_HOST="localhost" 114 115# Full "default" path where to find sqlite (no quotes!) 116SQLITE3_DIR=${HOME}/depkgs/sqlite3 117SQLITE_DIR=${HOME}/depkgs/sqlite 118 119TAPE_DRIVE="/dev/nst0" 120# if you don't have an autochanger set AUTOCHANGER to /dev/null 121AUTOCHANGER="/dev/sg0" 122# For two drive tests -- set to /dev/null if you do not have it 123TAPE_DRIVE1="/dev/null" 124 125# This must be the path to the autochanger including its name 126AUTOCHANGER_PATH="/usr/sbin/mtx" 127 128# Set what backend to use "postresql" "mysql" or "sqlite3" 129DBTYPE="postgresql" 130 131# Set your database here 132#WHICHDB="--with-${DBTYPE}=${SQLITE3_DIR}" 133WHICHDB="--with-${DBTYPE}" 134 135# Set this to "--with-tcp-wrappers" or "--without-tcp-wrappers" 136TCPWRAPPERS="--with-tcp-wrappers" 137 138# Set this to "" to disable OpenSSL support, "--with-openssl=yes" 139# to enable it, or provide the path to the OpenSSL installation, 140# eg "--with-openssl=/usr/local" 141OPENSSL="--with-openssl" 142 143# You may put your real host name here, but localhost or 127.0.0.1 144# is valid also and it has the advantage that it works on a 145# non-networked machine 146HOST="localhost" 147 148\end{verbatim} 149\normalsize 150 151\begin{itemize} 152\item {\bf BACULA\_SOURCE} should be the full path to the Bacula source code 153 that you wish to test. It will be loaded configured, compiled, and 154 installed with the "make setup" command, which needs to be done only 155 once each time you change the source code. 156 157\item {\bf EMAIL} should be your email addres. Please remember to change this 158 or I will get a flood of unwanted messages. You may or may not want to see 159 these emails. In my case, I don't need them so I direct it to the bit bucket. 160 161\item {\bf SMTP\_HOST} defines where your SMTP server is. 162 163\item {\bf SQLITE\_DIR} should be the full path to the sqlite package, must 164 be build before running a Bacula regression, if you are using SQLite. This 165 variable is ignored if you are using MySQL or PostgreSQL. To use PostgreSQL, 166 edit the Makefile and change (or add) WHICHDB?=``\verb{--{with-postgresql''. For 167 MySQL use ``WHICHDB=''\verb{--{with-mysql``. 168 169 The advantage of using SQLite is that it is totally independent of any 170 installation you may have running on your system, and there is no 171 special configuration or authorization that must be done to run it. 172 With both MySQL and PostgreSQL, you must pre-install the packages, 173 initialize them and ensure that you have authorization to access the 174 database and create and delete tables. 175 176\item {\bf TAPE\_DRIVE} is the full path to your tape drive. The base set of 177 regression tests do not use a tape, so this is only important if you want to 178 run the full tests. Set this to /dev/null if you do not have a tape drive. 179 180\item {\bf TAPE\_DRIVE1} is the full path to your second tape drive, if 181 have one. The base set of 182 regression tests do not use a tape, so this is only important if you want to 183 run the full two drive tests. Set this to /dev/null if you do not have a 184 second tape drive. 185 186\item {\bf AUTOCHANGER} is the name of your autochanger control device. Set this to 187 /dev/null if you do not have an autochanger. 188 189\item {\bf AUTOCHANGER\_PATH} is the full path including the program name for 190 your autochanger program (normally {\bf mtx}. Leave the default value if you 191 do not have one. 192 193\item {\bf TCPWRAPPERS} defines whether or not you want the ./configure 194 to be performed with tcpwrappers enabled. 195 196\item {\bf OPENSSL} used to enable/disable SSL support for Bacula 197 communications and data encryption. 198 199\item {\bf HOST} is the hostname that it will use when building the 200 scripts. The Bacula daemons will be named <HOST>-dir, <HOST>-fd, 201 ... It is also the name of the HOST machine that to connect to the 202 daemons by the network. Hence the name should either be your real 203 hostname (with an appropriate DNS or /etc/hosts entry) or {\bf 204 localhost} as it is in the default file. 205 206\item {\bf bin} is the binary location. 207 208\item {\bf scripts} is the bacula scripts location (where we could find 209 database creation script, autochanger handler, etc.) 210 211\end{itemize} 212 213\subsection{Building the Test Bacula} 214\index{Building the Test Bacula} 215\index{Bacula!Building the Test} 216\addcontentsline{toc}{subsection}{Building the Test Bacula} 217 218Once the above variables are set, you can build the setup by entering: 219 220\footnotesize 221\begin{verbatim} 222make setup 223\end{verbatim} 224\normalsize 225 226This will setup the regression testing and you should not need to 227do this again unless you want to change the database or other regression 228configuration parameters. 229 230 231\subsection{Setting up your SQL engine} 232\index{Setting up your SQL engine} 233\addcontentsline{toc}{subsection}{Setting up your SQL engine} 234 235If you are using MySQL or PostgreSQL, you will need to establish an 236account with your database engine for the user name {\bf regress} and 237you will need to manually create a database named {\bf regress} that can be 238used by user name regress, which means you will have to give the user 239regress sufficient permissions to use the database named regress. 240There is no password on the regress account. 241 242You have probably already done this procedure for the user name and 243database named bacula. If not, the manual describes roughly how to 244do it, and the scripts in bacula/regress/build/src/cats named 245create\_mysql\_database, create\_postgresql\_database, grant\_mysql\_privileges, 246and grant\_postgresql\_privileges may be of a help to you. 247 248Generally, to do the above, you will need to run under root to 249be able to create databases and modify permissions within MySQL and 250PostgreSQL. 251 252It is possible to configure MySQL access for database accounts that 253require a password to be supplied. This can be done by creating a ~/.my.cnf 254file which supplies the credentials by default to the MySQL commandline 255utilities. 256 257\begin{verbatim} 258[client] 259host = localhost 260user = regress 261password = asecret 262\end{verbatim} 263 264A similar technique can be used PostgreSQL regression testing where the 265database is configured to require a password. The ~/.pgpass file should 266contain a line with the database connection properties. 267 268\begin{verbatim} 269hostname:port:database:username:password 270\end{verbatim} 271 272\subsection{Running the Disk Only Regression} 273\index{Regression!Running the Disk Only} 274\index{Running the Disk Only Regression} 275\addcontentsline{toc}{subsection}{Running the Disk Only Regression} 276 277The simplest way to copy the source code, configure it, compile it, link 278it, and run the tests is to use a helper script: 279 280\footnotesize 281\begin{verbatim} 282./do_disk 283\end{verbatim} 284\normalsize 285 286 287 288 289This will run the base set of tests using disk Volumes. 290If you are testing on a 291non-Linux machine several of the of the tests may not be run. In any case, 292as we add new tests, the number will vary. It will take about 1 hour 293and you don't need to be root 294to run these tests (I run under my regular userid). The result should be 295something similar to: 296 297\footnotesize 298\begin{verbatim} 299Test results 300 ===== auto-label-test OK 12:31:33 ===== 301 ===== backup-bacula-test OK 12:32:32 ===== 302 ===== bextract-test OK 12:33:27 ===== 303 ===== bscan-test OK 12:34:47 ===== 304 ===== bsr-opt-test OK 12:35:46 ===== 305 ===== compressed-test OK 12:36:52 ===== 306 ===== compressed-encrypt-test OK 12:38:18 ===== 307 ===== concurrent-jobs-test OK 12:39:49 ===== 308 ===== data-encrypt-test OK 12:41:11 ===== 309 ===== encrypt-bug-test OK 12:42:00 ===== 310 ===== fifo-test OK 12:43:46 ===== 311 ===== backup-bacula-fifo OK 12:44:54 ===== 312 ===== differential-test OK 12:45:36 ===== 313 ===== four-concurrent-jobs-test OK 12:47:39 ===== 314 ===== four-jobs-test OK 12:49:22 ===== 315 ===== incremental-test OK 12:50:38 ===== 316 ===== query-test OK 12:51:37 ===== 317 ===== recycle-test OK 12:53:52 ===== 318 ===== restore2-by-file-test OK 12:54:53 ===== 319 ===== restore-by-file-test OK 12:55:40 ===== 320 ===== restore-disk-seek-test OK 12:56:29 ===== 321 ===== six-vol-test OK 12:57:44 ===== 322 ===== span-vol-test OK 12:58:52 ===== 323 ===== sparse-compressed-test OK 13:00:00 ===== 324 ===== sparse-test OK 13:01:04 ===== 325 ===== two-jobs-test OK 13:02:39 ===== 326 ===== two-vol-test OK 13:03:49 ===== 327 ===== verify-vol-test OK 13:04:56 ===== 328 ===== weird-files2-test OK 13:05:47 ===== 329 ===== weird-files-test OK 13:06:33 ===== 330 ===== migration-job-test OK 13:08:15 ===== 331 ===== migration-jobspan-test OK 13:09:33 ===== 332 ===== migration-volume-test OK 13:10:48 ===== 333 ===== migration-time-test OK 13:12:59 ===== 334 ===== hardlink-test OK 13:13:50 ===== 335 ===== two-pool-test OK 13:18:17 ===== 336 ===== fast-two-pool-test OK 13:24:02 ===== 337 ===== two-volume-test OK 13:25:06 ===== 338 ===== incremental-2disk OK 13:25:57 ===== 339 ===== 2drive-incremental-2disk OK 13:26:53 ===== 340 ===== scratch-pool-test OK 13:28:01 ===== 341Total time = 0:57:55 or 3475 secs 342 343\end{verbatim} 344\normalsize 345 346and the working tape tests are run with 347 348\footnotesize 349\begin{verbatim} 350make full_test 351\end{verbatim} 352\normalsize 353 354 355\footnotesize 356\begin{verbatim} 357Test results 358 359 ===== Bacula tape test OK ===== 360 ===== Small File Size test OK ===== 361 ===== restore-by-file-tape test OK ===== 362 ===== incremental-tape test OK ===== 363 ===== four-concurrent-jobs-tape OK ===== 364 ===== four-jobs-tape OK ===== 365\end{verbatim} 366\normalsize 367 368Each separate test is self contained in that it initializes to run Bacula from 369scratch (i.e. newly created database). It will also kill any Bacula session 370that is currently running. In addition, it uses ports 8101, 8102, and 8103 so 371that it does not intefere with a production system. 372 373Alternatively, you can do the ./do\_disk work by hand with: 374 375\footnotesize 376\begin{verbatim} 377make setup 378\end{verbatim} 379\normalsize 380 381The above will then copy the source code within 382the regression tree (in directory regress/build), configure it, and build it. 383There should be no errors. If there are, please correct them before 384continuing. From this point on, as long as you don't change the Bacula 385source code, you should not need to repeat any of the above steps. If 386you pull down a new version of the source code, simply run {\bf make setup} 387again. 388 389 390Once Bacula is built, you can run the basic disk only non-root regression test 391by entering: 392 393\footnotesize 394\begin{verbatim} 395make test 396\end{verbatim} 397\normalsize 398 399 400\subsection{Other Tests} 401\index{Other Tests} 402\index{Tests!Other} 403\addcontentsline{toc}{subsection}{Other Tests} 404 405There are a number of other tests that can be run as well. All the tests are a 406simply shell script keep in the regress directory. For example the ''make 407test`` simply executes {\bf ./all-non-root-tests}. The other tests, which 408are invoked by directly running the script are: 409 410\begin{description} 411 412\item [all\_non-root-tests] 413 \index{all\_non-root-tests} 414 All non-tape tests not requiring root. This is the standard set of tests, 415that in general, backup some data, then restore it, and finally compares the 416restored data with the original data. 417 418\item [all-root-tests] 419 \index{all-root-tests} 420 All non-tape tests requiring root permission. These are a relatively small 421number of tests that require running as root. The amount of data backed up 422can be quite large. For example, one test backs up /usr, another backs up 423/etc. One or more of these tests reports an error -- I'll fix it one day. 424 425\item [all-non-root-tape-tests] 426 \index{all-non-root-tape-tests} 427 All tape test not requiring root. There are currently three tests, all run 428without being root, and backup to a tape. The first two tests use one volume, 429and the third test requires an autochanger, and uses two volumes. If you 430don't have an autochanger, then this script will probably produce an error. 431 432\item [all-tape-and-file-tests] 433 \index{all-tape-and-file-tests} 434 All tape and file tests not requiring root. This includes just about 435everything, and I don't run it very often. 436\end{description} 437 438\subsection{If a Test Fails} 439\index{Fails!If a Test} 440\index{If a Test Fails} 441\addcontentsline{toc}{subsection}{If a Test Fails} 442 443If you one or more tests fail, the line output will be similar to: 444 445\footnotesize 446\begin{verbatim} 447 !!!!! concurrent-jobs-test failed!!! !!!!! 448\end{verbatim} 449\normalsize 450 451If you want to determine why the test failed, you will need to rerun the 452script with the debug output turned on. You do so by defining the 453environment variable {\bf REGRESS\_DEBUG} with commands such as: 454 455\begin{verbatim} 456REGRESS_DEBUG=1 457export REGRESS_DEBUG 458\end{verbatim} 459 460Then from the "regress" directory (all regression scripts assume that 461you have "regress" as the current directory), enter: 462 463\begin{verbatim} 464tests/test-name 465\end{verbatim} 466 467where test-name should be the name of a test script -- for example: 468{\bf tests/backup-bacula-test}. 469 470\section{Testing a Binary Installation} 471\index{Test!Testing a Binary Installation} 472 473If you have installed your Bacula from a binary release such as (rpms or debs), 474you can still run regression tests on it. 475First, make sure that your regression {\bf config} file uses the same catalog backend as 476your installed binaries. Then define the variables \texttt{bin} and \texttt{scripts} variables 477in your config file. 478 479Example: 480\begin{verbatim} 481bin=/opt/bacula/bin 482scripts=/opt/bacula/scripts 483\end{verbatim} 484 485The \texttt{./scripts/prepare-other-loc} will tweak the regress scripts to use 486your binary location. You will need to run it manually once before you run any 487regression tests. 488 489\begin{verbatim} 490$ ./scripts/prepare-other-loc 491$ ./tests/backup-bacula-test 492... 493\end{verbatim} 494 495All regression scripts must be run by hand or by calling the test scripts. 496These are principally scripts that begin with {\bf all\_...} such as {\bf all\_disk\_tests}, 497{\bf ./all\_test} ... 498None of the 499{\bf ./do\_disk}, {\bf ./do\_all}, {\bf ./nightly...} scripts will work. 500 501If you want to switch back to running the regression scripts from source, first 502remove the {\bf bin} and {\bf scripts} variables from your {\bf config} file and 503rerun the {\bf make setup} step. 504 505\section{Running a Single Test} 506\index{Running a Single Test} 507\addcontentsline{toc}{section}{Running a Single Test} 508 509If you wish to run a single test, you can simply: 510 511\begin{verbatim} 512cd regress 513tests/<name-of-test> 514\end{verbatim} 515 516or, if the source code has been updated, you would do: 517 518\begin{verbatim} 519cd bacula 520git pull 521cd regress 522make setup 523tests/backup-to-null 524\end{verbatim} 525 526 527\section{Writing a Regression Test} 528\index{Test!Writing a Regression} 529\index{Writing a Regression Test} 530\addcontentsline{toc}{section}{Writing a Regression Test} 531 532Any developer, who implements a major new feature, should write a regression 533test that exercises and validates the new feature. Each regression test is a 534complete test by itself. It terminates any running Bacula, initializes the 535database, starts Bacula, then runs the test by using the console program. 536 537\subsection{Running the Tests by Hand} 538\index{Hand!Running the Tests by} 539\index{Running the Tests by Hand} 540\addcontentsline{toc}{subsection}{Running the Tests by Hand} 541 542You can run any individual test by hand by cd'ing to the {\bf regress} 543directory and entering: 544 545\footnotesize 546\begin{verbatim} 547tests/<test-name> 548\end{verbatim} 549\normalsize 550 551\subsection{Directory Structure} 552\index{Structure!Directory} 553\index{Directory Structure} 554\addcontentsline{toc}{subsection}{Directory Structure} 555 556The directory structure of the regression tests is: 557 558\footnotesize 559\begin{verbatim} 560 regress - Makefile, scripts to start tests 561 |------ scripts - Scripts and conf files 562 |-------tests - All test scripts are here 563 | 564 |------------------ -- All directories below this point are used 565 | for testing, but are created from the 566 | above directories and are removed with 567 | "make distclean" 568 | 569 |------ bin - This is the install directory for 570 | Bacula to be used testing 571 |------ build - Where the Bacula source build tree is 572 |------ tmp - Most temp files go here 573 |------ working - Bacula working directory 574 |------ weird-files - Weird files used in two of the tests. 575\end{verbatim} 576\normalsize 577 578\subsection{Adding a New Test} 579\index{Adding a New Test} 580\index{Test!Adding a New} 581\addcontentsline{toc}{subsection}{Adding a New Test} 582 583If you want to write a new regression test, it is best to start with one of 584the existing test scripts, and modify it to do the new test. 585 586When adding a new test, be extremely careful about adding anything to any of 587the daemons' configuration files. The reason is that it may change the prompts 588that are sent to the console. For example, adding a Pool means that the 589current scripts, which assume that Bacula automatically selects a Pool, will 590now be presented with a new prompt, so the test will fail. If you need to 591enhance the configuration files, consider making your own versions. 592 593\subsection{Running a Test Under The Debugger} 594\index{Debugger} 595\addcontentsline{toc}{subsection}{Running a Test Under The Debugger} 596You can run a test under the debugger (actually run a Bacula daemon 597under the debugger) by first setting the environment variable 598{\bf REGRESS\_WAIT} with commands such as: 599 600\begin{verbatim} 601REGRESS_WAIT=1 602export REGRESS_WAIT 603\end{verbatim} 604 605Then executing the script. When the script prints the following line: 606 607\begin{verbatim} 608Start Bacula under debugger and enter anything when ready ... 609\end{verbatim} 610 611You start the Bacula component you want to run under the debugger in a 612different shell window. For example: 613 614\begin{verbatim} 615cd .../regress/bin 616gdb bacula-sd 617(possibly set breakpoints, ...) 618run -s -f 619\end{verbatim} 620 621Then enter any character in the window with the above message. 622An error message will appear saying that the daemon you are debugging 623is already running, which is the case. You can simply ignore the 624error message. 625