1 <chapter id=gettingup> 2 <title>Getting DejaGnu up and running</title> 3<para>This chapter was originally written by Niklaus Giger (ngiger@mus.ch) because he lost a week to figure out how DejaGnu works and how to write a first test.</para> 4 5<para>Follow these instructions as closely a possible in order get a good insight into how DejaGnu works, else you might run into a lot of subtle problems. You have been warned.</para> 6<para>It should be no big problems installing DejaGnu using your package manager or from the source code. Under a Debian/GNU/Linux systems just type (as root) <programlisting>apt-get dejagnu</programlisting>. These examples were run on a primary machine with a AMD K6 and a Mac Powerbook G3 serving as a remote target.</para> 7 8<para> The tests for Windows were run under Windows NT using the actual Cygwin version (1.3.x as of October 2001). It's target system was a PPC embedded system running vxWorks. 9</para> 10 11<sect1> 12<title>Test your installation</title> 13 14<para>Create a new user called "dgt" (DejaGnuTest), which uses bash as it login shell. PS1 must be set to '\u:\w\$ ' in its ~/.bashrc. Login as this user, create an empty directory and change the working directory to it. e.g</para> 15 16<programlisting> 17dgt:~$ mkdir ~/dejagnu.test 18dgt:~$ cd ~/dejagnu.test 19</programlisting> 20 21<para>Now you are ready to test DejaGnu's main program called runtest. The expecteted output is shown</para> 22 23<example> 24<title>Runtest output in a empty directory 25</title> 26 27<programlisting> 28dgt:~/dejagnu.test$ runtest 29WARNING: Couldn't find the global config file. 30WARNING: No tool specified Test 31Run By dgt on Sun Nov 25 17:07:03 2001 Native configuration is i586-pc-linux-gnu 32=== tests === 33Schedule of variations: unix 34Running target unix Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target. 35Using /usr/share/dejagnu/config/unix.exp as generic interface file for target. 36ERROR: Couldn't find tool config file for unix. 37=== Summary ===</programlisting> 38 39<para>We will show you later how to get rid of all the WARNING- and ERROR-messages. The files testrun.sum and testrun.log have been created, which do not interest us at this point. Let's remove them.</para> 40 41<programlisting>:~/dejagnu.test$ rm testrun.sum testrun.log 42</programlisting> 43</example> 44 45<sect2> 46<title>Windows</title> 47 48<para>On Windows systems DejaGnu is part of a port of a lot of Unix tools to the Windows OS, called Cygwin. Cygwin may be downloaded and installed from a mirror of http://www.cygwin.com/. All examples were also run on Windows NT. If nothing is said, you can assume that you should get the same output as on a Unix system.</para> 49 50<para>You will need a telnet daemon if you want to use a Windows box as a remote target. There seems to be a freeware telnet daemon at http://www.fictional.net/.</para> 51</sect2> 52 53<sect2> 54<title>Getting the source code for the calc example</title> 55 56<para>If you are running a Debian distribution you can find the examples under /usr/share/doc/dejagnu/examples. 57These examples seem to be missing in Red Hat's RPM. 58In this case download the sources of DejaGnu and adjust the pathes to the DejaGnu examples accordingly.</para> 59</sect2> 60</sect1> 61 62<sect1> 63<title>Create a minimal project, e.g. calc</title> 64 65<para>In this section you will to start a small project, 66using the sample application calc, which is part of your DejaGnu distribution</para> 67 68<sect2><title>A simple project without the GNU autotools</title> 69 70<para>The runtest program can be run standalone. All the autoconf/automake support is just cause those programs are commonly used for other GNU applications. The key to running runtest standalone is having the local site.exp file setup correctly, which automake does.</para> 71 72<para>The generated site.exp should like like:</para> 73<programlisting> 74set tool calc 75set srcdir . 76set objdir /home/dgt/dejagnu.test 77</programlisting></sect2> 78 79<sect2> 80<title>Using autoconf/autoheader/automake</title> 81 82<para>We have to prepare some input file in order to run autocon and automake. There is book “GNU autoconf, automake and libtool” by Garry V. Vaughan, et al. NewRider, ISBN 1-57870-190-2 which describes this process thoroughly.</para> 83 84<para>From the calc example distributed with the DejaGnu documentation you should copy the program file itself (calc.c) and some additional files, which you might examine a little bit close to derive their meanings.</para> 85 86<programlisting> 87dgt:~/dejagnu.test$ cp -r /usr/share/doc/dejagnu/examples/calc/\ 88{configure.in,Makefile.am,calc.c,testsuite} . 89</programlisting> 90 91<para>In Makemake.am note the presence of the AUTOMAKE_OPTIONS = dejagnu. This option is needed.</para> 92 93<para>Run aclocal to generate aclocal.m4, which is a collection of macros needed by configure.in</para> 94 95<programlisting> 96dgt:~/dejagnu.test$ aclocal 97</programlisting> 98 99<para>autoconf is another part of the auto-tools. 100Run it to generate configure based on information contained in configure.in.</para> 101 102<programlisting> 103dgt:~/dejagnu.test$ autoconf 104</programlisting> 105 106<para>autoheader is another part of the auto-tools. 107Run it to generate calc.h.in. </para> 108 109<programlisting> 110dgt:~/dejagnu.test$ autoheader 111</programlisting> 112 113<para>The Makefile.am of this example was developed as port of the DejaGnu 114distribution. 115Adapt Makefile.am for this test. Replace the line 116“#noinst_PROGRAMS = calc” to 117“bin_PROGRAMS = calc”. 118Change the RUNTESTDEFAULTFLAGS from 119“$$srcdir/testsuite” to 120“./testsuite”.</para> 121 122<para>Running automake at this point contains a series of warning in its output as shown in the following example:</para> 123 124<example> 125<title>Sample output of automake with missing files</title> 126<programlisting> 127dgt:~/dejagnu.test$ automake --add-missing 128automake: configure.in: installing `./install-sh' 129automake: configure.in: installing `./mkinstalldirs' 130automake: configure.in: installing `./missing' 131automake: Makefile.am: installing `./INSTALL' 132automake: Makefile.am: required file `./NEWS' not found 133automake: Makefile.am: required file `./README' not found 134automake: Makefile.am: installing `./COPYING' 135automake: Makefile.am: required file `./AUTHORS' not found 136automake: Makefile.am: required file `./ChangeLog' not found 137configure.in: 4: required file `./calc.h.in' not found 138Makefile.am:6: required directory ./doc does not exist 139</programlisting> 140</example> 141 142<para>Create a empty directory doc and empty files 143INSTALL, NEWS, README, AUTHORS, ChangeLog and COPYING. 144The default COPYING will point to the GNU Public License (GPL). 145In a real project it would be time to add some meaningfull text in each file. 146</para> 147 148<para>Adapt calc to your environment by calling configure.</para> 149<example> 150<title>Sample output of configure 151</title> 152<programlisting> 153dgt:~/dejagnu.test$ ./configure 154creating cache ./config.cache 155checking whether to enable maintainer-specific portions of Makefiles... no 156checking for a BSD compatible install... /usr/bin/install -c 157checking whether build environment is sane... yes 158checking whether make sets ${MAKE}... yes 159checking for working aclocal... found 160checking for working autoconf... found 161checking for working automake... found 162checking for working autoheader... found 163checking for working makeinfo... found 164checking for gcc... gcc checking whether the C compiler (gcc ) works... yes 165checking whether the C compiler (gcc ) is a cross-compiler... no 166checking whether we are using GNU C... yes 167checking whether gcc accepts -g... yes 168checking for a BSD compatible install... /usr/bin/install -c 169checking how to run the C preprocessor... gcc -E 170checking for stdlib.h... yes 171checking for strcmp... yes 172updating cache ./config.cache 173creating ./config.status 174creating Makefile creating calc.h 175</programlisting> 176</example> 177 178<para>If you are familiar with GNU software, 179this output should not contain any surprise to you. 180Any errors should be easy to fix for such a simple program.</para> 181 182<para>Build the calc executable:</para> 183 184<example> 185<title>Sample output building calc 186</title> 187<programlisting> 188dgt:~/dejagnu.test$ make 189gcc -DHAVE_CONFIG_H -I. -I. -I. -g -O2 -c calc.c 190gcc -g -O2 -o calc calc.o 191</programlisting> 192</example> 193 194<para>You prepared a few files and then called some commands. 195Respecting the right order assures a automatic and correctly compiled calc program. The following example resumes the correct order.</para> 196 197<example> 198<title>Creating the calc program using the GNU autotools</title> 199<programlisting> 200dgt:~/dejagnu.test$ aclocal 201dgt:~/dejagnu.test$ autoconf 202dgt:~/dejagnu.test$ autoheader 203dgt:~/dejagnu.test$ automake --add-missing 204dgt:~/dejagnu.test$ ./configure 205dgt:~/dejagnu.test$ make 206 207</programlisting> 208</example> 209 210<para>Play with calc and verify whether it works correctly. 211A sample session might look like this:</para> 212 213<programlisting> 214dgt:~/dejagnu.test$ ./calc 215calc: version 216Version: 1.1 217calc:<emphasis> </emphasis>add 3 4 2187 219calc: multiply 3 4<emphasis> </emphasis> 22012 221calc: multiply 2 4<emphasis> </emphasis> 22212 223calc: quit 224 225</programlisting> 226 227<para>Look at the intentional bug that 2 times 4 equals 12.</para> 228<para>The tests run by DejaGnu need a file called site.exp, 229which is automatically generated if we call “make site.exp”. 230This was the purpose of the “AUTOMAKE_OPTIONS = dejagnu” in Makefile.am.</para> 231 232<example> 233<title>Sample output generating a site.exp</title> 234<programlisting> 235dgt: make site.exp 236dgt:~/dejagnu.test$ make site.exp 237Making a new site.exp file... 238</programlisting> 239</example> 240 241 242</sect2> 243</sect1> 244 245<sect1> 246<title>Our first automated tests</title> 247<sect2><title>Running the test for the calc example</title> 248 249<para>Now we are ready to call the automated tests </para> 250 251<example> 252<title>Sample output of runtest in a configured directory</title> 253 254<programlisting> 255dgt:~/dejagnu.test$ make check 256make check-DEJAGNU 257make[1]: Entering directory `/home/dgt/dejagnu.test' srcdir=`cd . && pwd`; export srcdir; \ 258EXPECT=expect; export EXPECT; \ runtest=runtest; \ 259if /bin/sh -c "$runtest --version" > /dev/null 2>&1; then \ 260$runtest --tool calc CALC=`pwd`/calc --srcdir ./testsuite ; \ 261else echo "WARNING: could not find \`runtest'" 1>&2; :;\ 262fi 263WARNING: Couldn't find the global config file. 264WARNING: Couldn't find tool init file 265Test Run By dgt on Sun Nov 25 21:42:21 2001 266Native configuration is i586-pc-linux-gnu 267 268 === calc tests === 269 270Schedule of variations: 271 unix 272 273Running target unix 274Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target. 275Using /usr/share/dejagnu/config/unix.exp as generic interface file for target. 276Using ./testsuite/config/unix.exp as tool-and-target-specific interface file. 277Running ./testsuite/calc.test/calc.exp ... 278FAIL: multiply2 (bad match) 279 280=== calc Summary === 281 282# of expected passes 5 283# of unexpected failures 1 284/home/Dgt/dejagnu.test/calc version Version: 1.1 285make[1]: *** [check-DEJAGNU] Fehler 1 286make[1]: Leaving directory `/home/Dgt/dejagnu.test' make: *** [check-am] Fehler 2 287</programlisting> 288</example> 289 290<para>Did you see the line “FAIL:“? The test cases for calc catch the bug in the calc.c file. Fix the error in calc.c later as the following examples assume a unchanged calc.c.</para> 291 292<para>Examine the output files calc.sum and calc.log. 293Try to understand the testcases written in ~/dejagnu.test/testsuite/calc.test/calc.exp. 294To understand Expect you might take a look at the book "Exploring Expect", 295which is an excellent resource for learning and using Expect. (Pub: O'Reilly, ISBN 1-56592-090-2) 296The book contains hundreds of examples and also includes a tutorial on Tcl. 297Exploring Expect is 602 pages long. </para> 298</sect2> 299 300 301<sect2><title>The various config files or how to avoid warnings</title> 302 303<para>DejaGnu may be customized by each user. It first searches for a file called ~/.dejagnurc. Create the file ~/.dejagnurc and insert the following line:</para> 304 305<programlisting> 306puts "I am ~/.dejagnurc" 307</programlisting> 308 309<para>Rerun make check. Test whether the output contains "I am ~/.dejagnurc". 310Create ~/my_dejagnu.exp and insert the following line:</para> 311 312<programlisting> 313puts "I am ~/my_dejagnu.exp" 314</programlisting> 315 316<para>In a Bash-Shell enter</para> 317 318<programlisting> 319dgt:~/dejagnu.test$ export DEJAGNU=~/my_dejagnu.exp 320</programlisting> 321 322<para>Run “make check” again. The output should not contain 323“WARNING: Couldn't find the global config file.”. 324Create the sub-directory lib. Create the file “calc.exp” in it and insert the following line:</para> 325 326<programlisting> 327puts "I am lib/calc.exp" 328</programlisting> 329 330<para>The last warning “WARNING: Couldn't find tool init file” 331should not be part of the output of make check. 332Create the directory ˜/boards. Create the file ˜/boards/standard.exp and insert the following line:</para> 333 334<programlisting> 335puts "I am boards/standard.exp" 336</programlisting> 337 338<para>If the variable DEJAGNU is still not empty then the (abbreviated) output of “make check” should look like this:</para> 339 340<example> 341<title>Sample output of runtest with the usual configuration files</title> 342<programlisting> 343dgt:~/dejagnu.test$ make check 344<...> 345fi 346I am ~/.dejagnurc 347I am ~/my_dejagnu.exp 348I am lib/calc.exp 349Test Run By dgt on Sun Nov 25 22:19:14 2001 350Native configuration is i586-pc-linux-gnu 351 352 === calc tests === 353Using /home/Dgt/boards/standard.exp as standard board description\ 354file for build. 355I am ~/boards/standard.exp 356Using /home/Dgt/boards/standard.exp as standard board description\ 357 file for host. 358I am ~/boards/standard.exp 359 360Schedule of variations: 361 unix 362 363Running target unix 364Using /home/Dgt/boards/standard.exp as standard board description\ 365 file for target. 366I am ~/boards/standard.exp 367Using /usr/share/dejagnu/baseboards/unix.exp as board description file\ 368for target. 369<...> 370</programlisting> 371</example> 372 373<para>It is up to you to decide when and where to use any of the above 374mentioned config files for customizing. 375This chapters showed you where and in which order the different config 376files are run.</para> 377</sect2> 378 379<sect2> 380<title>When trouble strikes</title> 381 382<para>Calling runtest with the '-v'-flag shows you in even more details which files are searched in which order. Passing it several times gives more and more details. </para> 383<example> 384<title>Displaying details about runtest execution</title> 385<programlisting> 386runtest -v -v -v --tool calc CALC=`pwd`/calc --srcdir ./testsuite 387</programlisting> 388</example> 389 390<para>Calling runtest with the '--debug'-flag logs a lot of details to dbg.log where you can analyse it afterwards. </para> 391 392<para>In all test cases you can temporary adjust the verbosity of information by adding the following Tcl-command to any tcl file that gets loaded by 393dejagnu, for instance, ~/.dejagnurc:</para> 394 395<programlisting> 396set verbose 9 397</programlisting> 398 399</sect2> 400 401<sect2> 402<title>Testing “Hello world” locally</title> 403 404<para>This test checks, whether the built-in shell command “echo Hello world” 405 will really write “Hello world” on the console. 406Create the file ~/dejagnu.test/testsuite/calc.test/local_echo.exp. 407It should contain the following lines</para> 408 409<example> 410<title>A first (local) test case</title> 411<programlisting> 412set test "Local Hello World" 413send "echo Hello World" 414expect { 415 -re "Hello World" { pass "$test" } 416} 417</programlisting> 418</example> 419 420<para>Run runtest again and verify the output “calc.log”</para> 421</sect2> 422</sect1> 423 424<sect1> 425<title>A first remote test</title> 426 427<para>Testing remote targets is a lot trickier especially if you are using an 428 embedded target 429which has no built in support for things like a compiler, ftp server or a Bash-shell. 430Before you can test calc on a remote target you have to acquire a few basics skills.</para> 431 432<sect2> 433<title>Setup telnet to your own host</title> 434<para>The easiest remote host is usually the host you are working on. 435In this example we will use telnet to login in your own workstation. 436For security reason you should never have a telnet deamon running on 437machine connected on the internet, as password and usernames are transmitted 438 in clear text. 439We assume you know how to setup your machine for a telnet daemon.</para> 440 441<para>Next try whether you may login in your own host by issuing the 442command “telnet localhost.1”. In order to be able to 443distinguish between a normal session an a telnet login add the following lines to /home/dgt/.bashrc.</para> 444 445<programlisting> 446if [ "$REMOTEHOST" ] 447then 448 PS1='remote:\w\$ ' 449fi 450</programlisting> 451 452<para>Now on the machine a “remote” login looks like this:</para> 453 454<example> 455<title>Sample log of a telnet login to localhost</title> 456<programlisting> 457dgt:~/dejagnu.test$ telnet localhost 458Trying 127.0.0.1... 459Connected to 127.0.0.1. 460Escape character is '^]'. 461Debian GNU/Linux testing/unstable Linux 462K6Linux login: dgt 463Password: 464Last login: Sun Nov 25 22:46:34 2001 from localhost on pts/4 465Linux K6Linux 2.4.14 #1 Fre Nov 16 19:28:25 CET 2001 i586 unknown 466No mail. 467remote:~$ exit 468logout 469Connection closed by foreign host. 470</programlisting> 471</example> 472</sect2> 473 474<sect2> 475<title>A test case for login via telnet</title> 476<para>In order to define a correct setup we have add a line containing 477“set target unix” either to ~/.dejagnurc or to ~/my_dejagnu.exp. 478In ~/boards/standard.exp add the following four lines to define a few patterns for the DejaGnu telnet login procedure.</para> 479 480<example> 481<title>Defining a remote target board</title> 482<programlisting> 483set_board_info shell_prompt "remote:" 484set_board_info telnet_username "dgt" 485set_board_info telnet_password "top_secret" 486set_board_info hostname "localhost" 487 488</programlisting> 489</example> 490 491<para>As DejaGnu will be parsing the telnet session output for some well 492known pattern the output there are a lot of things that can go wrong. 493If you have any problems verify your setup:</para> 494<itemizedlist> 495 496<listitem> 497<para>Is <filename>/etc/motd</filename> empty?</para></listitem> 498 499<listitem> 500<para>Is <filename>/etc/issue.net</filename> empty?</para></listitem> 501 502<listitem> 503<para>Exists a empty <filename>~/.hushlogin</filename>?</para></listitem> 504 505<listitem> 506<para>The LANG environment variable must be either empty or set to “C”. </para></listitem> 507 508</itemizedlist> 509<para>To test the login via telnet write a sample test case. 510Create the file ~/dejagnu.test/testsuite/calc.test/remote_echo.exp and 511add the following few lines:</para> 512 513<example> 514<title>DejaGnu script for logging in into a remote target</title> 515<programlisting> 516puts "this is remote_echo.exp target for $target " 517target_info $target 518#set verbose 9 519set shell_id [remote_open $target] 520set test "Remote login to $target" 521#set verbose 0 522puts "Spawn id for remote shell is $shell_id" 523if { $shell_id > 0 } { 524 pass "$test" 525} else { 526 fail "Remote open to $target" 527} 528</programlisting> 529</example> 530 531<para>In the runtest output you should find something like:</para> 532 533<programlisting> 534Running ./testsuite/calc.test/local_echo.exp ... 535Running ./testsuite/calc.test/remote_echoo.exp ... 536this is remote_echo.exp target is unix 537Spawn id for remote shell is exp7 538</programlisting> 539 540<para>Have again a look at calc.log to get a feeling how DejaGnu and expect 541parse the input. </para></sect2> 542 543<sect2> 544<title>Remote testing “Hello world”</title> 545 546<para>Next you will transform the above “hello world” example to 547its remote equivalent. 548This can be done by adding the following lines to our file remote_echo.exp.</para> 549 550<example> 551<title>A first (local) remote "Hello world" test</title> 552<programlisting> 553set test "Remote_send Hello World" 554set status [remote_send $target "echo \"Hello\" \"World\"\n" ] 555pass "$test" 556set test "Remote_expect Hello World" 557remote_expect $target 5 { 558 -re "Hello World" { pass "$test" } 559} 560</programlisting> 561</example> 562 563<para>Call make check. The output should contain 564“# of expected passes 9” and “# of unexcpected failures 1”.</para> 565 566<para>Have a look at the procedures in /usr/share/dejagnu/remote.exp to have an overview of the offered procedures and their features. </para> 567 568<para>Now setup a real target. 569In the following example we assume as target a PowerBook running Debian. 570As above add a test user "dgt", install telnet and FTP servers. 571In order to distinguish it from the host add the line 572<programlisting>PS1='test:>'</programlisting> to /home/dgt/.bash_profile. 573Also add a corresponding entry "powerbook" to /etc/hosts and verify that you 574are able to ping, telnet and ftp to the target "powerbook".</para> 575 576<para>In order to let runtest run its test on the "powerbook" target change the following lines in ~/boards/standard.exp:</para> 577 578<example> 579<title>Board definition for a remote target</title> 580<programlisting> 581set_board_info protocol "telnet" 582set_board_info telnet_username "dgt" 583set_board_info telnet_password "top_secret" 584set_board_info shell_prompt "test:> " 585set_board_info hostname "powerbook" 586</programlisting> 587</example> 588 589<para>Now call runtest again with the same arguments and verify whether all went okay by taking a close look at calc.log.</para> 590</sect2> 591 592 593<sect2> 594<title>Transferring files from/to the target</title> 595 596<para>A simple procedure like this will do the job for you:</para> 597 598<example> 599<title>Test script to transfer a file to a remote target</title> 600<programlisting> 601set test "Remote_download" 602puts "Running Remote_download" 603# set verbose 9 604set remfile /home/dgt/dejagnu2 605 606set status [remote_download $target /home/dgt/.dejagnurc $remfile] 607if { "$status" == "" } { 608 fail "Remote download to $remfile on $target" 609} else { 610 pass "$test" 611} 612 613puts "status of remote_download ist $status" 614# set verbose 0 615</programlisting> 616</example> 617 618<para>After running runtest again, check whether the file dejagnu2 exists on the target. 619 620This example will only work if the rcp command works with your target. 621 622If you have a working FTP-server on the target you can use it by adding the 623following lines to ~/boards/standard.exp:</para> 624 625<example> 626<title>Defining a board to use FTP as file transport</title> 627<programlisting> 628set_board_info file_transfer "ftp" 629set_board_info ftp_username "dgt" 630set_board_info ftp_password "1234" 631</programlisting> 632</example> 633 634</sect2> 635 636<sect2> 637<title>Preparing for crosscompilation</title> 638 639<para>For crosscompiling you need working binutils, gcc and a base library like 640libc or glib for your target. 641It is beyond the scope of this document to describe how to get it working. 642The following examples assume a cross compiler for PowerPC which is called linux-powerpc-gcc. 643</para> 644 645<para>Add AC_CANONICAL_TARGET in dejagnu.test/configure.in at the following location. Copy config.guess from /usr/share/automake to dejagnu.test.</para> 646 647<programlisting> 648AM_CONFIG_HEADER(calc.h) 649AC_CANONICAL_TARGET([]) 650AM_INIT_AUTOMAKE(calc, 1.1) 651</programlisting> 652 653<para>You need to run automake 2.5 or later. 654Depending on your installation calling autoconf2.5 instead of autoconf is not needed. 655The sequence to regenerate all files is:</para> 656 657<example> 658<title>Using autotools for cross development</title> 659<programlisting> 660$ autoconf2.5 661$ autoheader 662$ automake 663$ ./configure --host=powerpc-linux --target=powerpc-linux 664configure: WARNING: If you wanted to set the --build type, don't use --host. 665 If a cross compiler is detected then cross compile mode will be used. 666checking build system type... ./config.guess: ./config.guess: No such file or directory 667configure: error: cannot guess build type; you must specify one 668$ cp /usr/share/automake/config.guess . 669$ ./configure --host=powerpc-linux --target=powerpc-linux 670configure: WARNING: If you wanted to set the --build type, don't use --host. 671If a cross compiler is detected then cross compile mode will be used. \ 672checking build system type... i586-pc-linux-gnu 673checking host system type... powerpc-unknown-linux-gnu 674<...> 675checking whether we are cross compiling... yes 676<...> 677Configuration: 678Source code location: . 679C Compiler: powerpc-linux-gcc 680C Compiler flags: -g -O2 681 682</programlisting> 683</example> 684 685<para>Everything should be ready to recompile for the target:</para> 686 687<programlisting>$ make 688powerpc-linux-gcc -DHAVE_CONFIG_H -I. -I. -I. -g -O2 -c calc.c 689powerpc-linux-gcc -g -O2 -o calc calc.o 690 691</programlisting> 692</sect2> 693 694<sect2> 695<title>Remote testing of calc</title> 696<para>Not yet written, as I have problem getting libc6-dev-powerpc to work. Probably I first have to build my cross compiler. </para> 697</sect2> 698 699<sect2> 700<title>Using Windows as host and vxWorks as target</title> 701 702<para>A more thorough walk-through will be written in a few weeks.</para> 703 704<para>In order to test the vxWorks as a target I changed boards/standards.exp to reflect my settings (IP, username, password). Then I reconfigured vxWorks to include a FTP and telnet server (using the same username/password combination ad in boards/standard.exp).</para> 705 706<para>With this setup and some minor modification (e.g. replacing echo by printf) in my test cases I could test my vxWorks system. It sure does not seem to be a correct setup by DejaGnu standard. For instance, it still loading /usr/share/dejagnu/baseboards/unix.exp instead of vxWorks. In any case I found that (at least under Windows) I did not find out how the command line would let me override settings in my personal config files.</para> 707 708</sect2> 709</sect1> 710</chapter> 711 <chapter id=runningtests> 712 <title>Running Tests</title> 713 714 <para>There are two ways to execute a testsuite. The most 715 common way is when there is existing support in the 716 <filename>Makefile</filename>. This support consists of a 717 <emphasis>check</emphasis> target. The other way is to execute the 718 <command>runtest</command> program directly. To run 719 <command>runtest</command> directcly from the command line requires 720 either all the correct options, or the <xref linkend=local> must be setup 721 correctly.</para> 722 723 <sect1 id=makecheck xreflabel="Make Check"> 724 <title>Make check</title> 725 726 <para>To run tests from an existing collection, first use 727 <command>configure</command> as usual to set up the 728 build directory. Then try typing:</para> 729 730 <screen> 731 make check 732 </screen> 733 734 <para>If the <emphasis>check</emphasis> target exists, it 735 usually saves you some trouble. For instance, it can set up any 736 auxiliary programs or other files needed by the tests. The most 737 common file the check builds is the 738 <emphasis>site.exp</emphasis>. The site.exp file contains 739 various variables that DejaGnu used to dertermine the 740 configuration of the program being tested. This is mostly for 741 supporting remote testing.</para> 742 743 <para>The <emphasis>check</emphasis> target is supported by GNU 744 <productname>Automake</productname>. To have DejaGnu support added to your 745 generated <filename>Makefile.in</filename>, just add the keyword 746 dejagnu to the AUTOMAKE_OPTIONS variable in your 747 <filename>Makefile.am</filename> file.</para> 748 749 <para>Once you have run <emphasis>make check</emphasis> to build 750 any auxiliary files, you can invoke the test driver 751 <command>runtest</command> directly to repeat the tests. 752 You will also have to execute <command>runtest</command> 753 directly for test collections with no 754 <emphasis>check</emphasis> target in the 755 <filename>Makefile</filename>.</para> 756 757 </sect1> 758 759 <sect1 id=runtest xreflabel="Runtest"> 760 <title>Runtest</title> 761 762 <para><command>runtest</command> is the executable test driver 763 for DejaGnu. You can specify two kinds of things on the 764 <command>runtest</command> command line: command line options, 765 and Tcl variables for the test scripts. The options are listed 766 alphabetically below.</para> 767 768 <para><command>runtest</command> returns an exit code of 769 <emphasis>1</emphasis> if any test has an unexpected result; otherwise 770 (if all tests pass or fail as expected) it returns <emphasis>0</emphasis> 771 as the exit code.</para> 772 773 <sect2 id=outputs xreflabel="Output States"> 774 <title>Output States</title> 775 776 <para><filename>runtest</filename> flags the outcome of each 777 test as one of these cases. <xref linkend=posix> for a 778 discussion of how POSIX specifies the meanings of these 779 cases.</para> 780 781 <variablelist> 782 <varlistentry> 783 <term>PASS</term> 784 <listitem><para>The most desirable outcome: the test succeeded, and 785 was expected to succeed.</para></listitem> 786 </varlistentry> 787 788 <varlistentry> 789 <term>XPASS</term> 790 <listitem><para>A pleasant kind of failure: a test was expected to 791 fail, but succeeded. This may indicate progress; inspect the test 792 case to determine whether you should amend it to stop expecting 793 failure.</para></listitem> 794 </varlistentry> 795 796 <varlistentry> 797 <term>FAIL</term> 798 <listitem><para>A test failed, although it was expected to succeed. 799 This may indicate regress; inspect the test case and the failing 800 software to ocate the bug.</para></listitem> 801 </varlistentry> 802 803 <varlistentry> 804 <term>XFAIL</term> 805 <listitem><para>A test failed, but it was expected to fail. This 806 result indicates no change in a known bug. If a test fails because 807 the operating system where the test runs lacks some facility required 808 by the test, the outcome is <emphasis>UNSUPPORTED</emphasis> 809 instead.</para></listitem> 810 </varlistentry> 811 812 <varlistentry> 813 <term>UNRESOLVED</term> 814 <listitem><para>Output from a test requires manual inspection; the 815 testsuite could not automatically determine the outcome. For 816 example, your tests can report this outcome is when a test does not 817 complete as expected.</para></listitem> 818 </varlistentry> 819 820 <varlistentry> 821 <term>UNTESTED</term> 822 <listitem><para>A test case is not yet complete, and in particular 823 cannot yet produce a <emphasis>PASS</emphasis> or 824 <emphasis>FAIL</emphasis>. You can also use this outcome in dummy 825 ``tests'' that note explicitly the absence of a real test case for a 826 particular property.</para></listitem> 827 </varlistentry> 828 829 <varlistentry> 830 <term>UNSUPPORTED</term> 831 <listitem><para>A test depends on a conditionally available feature 832 that does not exist (in the configured testing environment). For 833 example, you can use this outcome to report on a test case that does 834 not work on a particular target because its operating system support 835 does not include a required subroutine.</para></listitem> 836 </varlistentry> 837 </variablelist> 838 839 <para>runtest may also display the following messages:</para> 840 841 <variablelist> 842 <varlistentry> 843 <term>ERROR</term> 844 <listitem><para>Indicates a major problem (detected by the test case 845 itself) in running the test. This is usually an unrecoverable error, 846 such as a missing file or loss of communication to the target. (POSIX 847 testsuites should not emit this message; use 848 <emphasis>UNSUPPORTED</emphasis>, <emphasis>UNTESTED</emphasis>, or 849 <emphasis>UNRESOLVED</emphasis> instead, as 850 appropriate.)</para></listitem> 851 </varlistentry> 852 853 <varlistentry> 854 <term>WARNING</term> 855 <listitem><para>Indicates a possible problem in running the 856 test. Usually warnings correspond to recoverable errors, or display 857 an important message about the following tests.</para></listitem> 858 </varlistentry> 859 860 <varlistentry> 861 <term>NOTE</term> 862 <listitem><para>An informational message about the test 863 case.</para></listitem> 864 </varlistentry> 865 </variablelist> 866 867 </sect2> 868 869 <sect2 id=invoking xreflabel="Invoking Runtest"> 870 <title>Invoking Runtest</title> 871 872 <para>This is the full set of command line options that 873 <filename>runtest</filename> recognizes. Arguments may be 874 abbreviated to the shortest unique string.</para> 875 876 <variablelist> 877 <varlistentry> 878 <term><option>--all</option> (-a)</term> 879 <listitem><para>Display all test output. By default, 880 <emphasis>runtest</emphasis> shows only the output of tests that 881 produce unexpected results; that is, tests with status 882 <emphasis>FAIL</emphasis> (unexpected failure), 883 <emphasis>XPASS</emphasis> (unexpected success), or 884 <emphasis>ERROR</emphasis> (a severe error in the test case 885 itself). Specify <emphasis>--all</emphasis> to see output for tests 886 with status <emphasis>PASS</emphasis> (success, as expected) 887 <emphasis>XFAIL</emphasis> (failure, as expected), or 888 <emphasis>WARNING</emphasis> (minor error in the test case 889 itself).</para></listitem> 890 </varlistentry> 891 892 <varlistentry> 893 <term><option>--build [string]</option></term> 894 <listitem><para><emphasis>string</emphasis> is a full configuration 895 ``triple'' name as used by <command>configure</command>. This 896 is the type of machine DejaGnu and the tools to be tested are built 897 on. For a normal cross this is the same as the host, but for a 898 canadian cross, they are seperate.</para></listitem> 899 </varlistentry> 900 901 <varlistentry> 902 <term><option>--host [string]</option></term> 903 <listitem><para><symbol>string</symbol> is a full configuration 904 ``triple'' name as used by <emphasis>configure</emphasis>. Use this 905 option to override the default string recorded by your 906 configuration's choice of host. This choice does not change how 907 anything is actually configured unless --build is also specified; it 908 affects <emphasis>only</emphasis> DejaGnu procedures that compare the 909 host string with particular values. The procedures 910 <emphasis>ishost</emphasis>, <emphasis>istarget</emphasis>, 911 <emphasis>isnative</emphasis>, and <emphasis>setup</emphasis>xfail} 912 are affected by <emphasis>--host</emphasis>. In this usage, 913 <emphasis>host</emphasis> refers to the machine that the tests are to 914 be run on, which may not be the same as the 915 <emphasis>build</emphasis> machine. If <emphasis>--build</emphasis> 916 is also specified, then <emphasis>--host</emphasis> refers to the 917 machine that the tests wil, be run on, not the machine DejaGnu is run 918 on.</para></listitem> 919 </varlistentry> 920 921 <varlistentry> 922 <term><option>--host_board [name]</option></term> 923 <listitem><para>The host board to use.</para></listitem> 924 </varlistentry> 925 926 <varlistentry> 927 <term><option>--target [string]</option></term> 928 <listitem><para>Use this option to override the default setting 929 (running native tests). <emphasis>string</emphasis> is a full 930 configuration ``triple'' name of the form 931 <emphasis>cpu-vendor-os</emphasis> as used by 932 <command>configure</command>. This option changes the 933 configuration <emphasis>runtest</emphasis> uses for the default tool 934 names, and other setup information.</para></listitem> 935 </varlistentry> 936 937 <varlistentry> 938 <term><option>--debug</option> (-de)</term> 939 <listitem><para>Turns on the <emphasis>expect</emphasis> internal 940 debugging output. Debugging output is displayed as part of the 941 <emphasis>runtest</emphasis> output, and logged to a file called 942 <filename>dbg.log</filename>. The extra debugging output does 943 <emphasis>not</emphasis> appear on standard output, unless the 944 verbose level is greater than 2 (for instance, to see debug output 945 immediately, specify <emphasis>--debug</emphasis>-v -v}). The 946 debugging output shows all attempts at matching the test output of 947 the tool with the scripted patterns describing expected output. The 948 output generated with <emphasis>--strace</emphasis> also goes into 949 <filename>dbg.log</filename>.</para></listitem> 950 </varlistentry> 951 952 <varlistentry> 953 <term><option>--help</option> (-he)</term> 954 <listitem><para>Prints out a short summary of the 955 <emphasis>runtest</emphasis> options, then exits (even if you also 956 specify other options).</para></listitem> 957 </varlistentry> 958 959 <varlistentry> 960 <term><option>--ignore [name(s)] </option></term> 961 <listitem><para>The names of specific tests to 962 ignore.</para></listitem> 963 </varlistentry> 964 965 <varlistentry> 966 <term><option>--objdir [path]</option></term> 967 <listitem><para>Use <emphasis>path</emphasis> as the top directory 968 containing any auxiliary compiled test code. This defaults to 969 <filename>.</filename>. Use this option to locate pre-compiled test 970 code. You can normally prepare any auxiliary files needed with 971 <emphasis>make</emphasis>.</para></listitem> 972 </varlistentry> 973 974 <varlistentry> 975 <term><option>--outdir [path]</option></term> 976 <listitem><para>Write output logs in directory 977 <filename>path</filename>. The default is <emphasis>.}, 978 the</emphasis> directory where you start 979 <emphasis>runtest</emphasis>. This option affects only the summary 980 and the detailed log files 981 <filename>tool.sum</filename> and 982 <filename>tool.log</filename>. The DejaGnu debug 983 log <filename>dbg.log</filename> always appears (when requested) in 984 the local directory.</para></listitem> 985 </varlistentry> 986 987 <varlistentry> 988 <term><option>--reboot [name]</option></term> 989 <listitem><para>Reboot the target board when 990 <emphasis>runtest</emphasis> initializes. Usually, when running tests 991 on a separate target board, it is safer to reboot the target to be 992 certain of its state. However, when developing test scripts, 993 rebooting takes a lot of time.</para></listitem> 994 </varlistentry> 995 996 <varlistentry> 997 <term><option>--srcdir [path]</option></term> 998 <listitem><para>Use <filename>path</filename> as the top directory 999 for test scripts to run. <emphasis>runtest</emphasis> looks in this 1000 directory for any subdirectory whose name begins with the toolname 1001 (specified with <emphasis>--tool</emphasis>). For instance, with 1002 <emphasis>--tool</emphasis>gdb}, <emphasis>runtest</emphasis> uses 1003 tests in subdirectories <filename>gdb.*</filename> (with the usual 1004 shell-like filename expansion). If you do not use 1005 <emphasis>--srcdir</emphasis>, <emphasis>runtest</emphasis> looks for 1006 test directories under the current working 1007 directory.</para></listitem> 1008 </varlistentry> 1009 1010 <varlistentry> 1011 <term><option>--strace [number]</option></term> 1012 <listitem><para>Turn on internal tracing for 1013 <emphasis>expect</emphasis>, to n levels deep. By adjusting the 1014 level, you can control the extent to which your output expands 1015 multi-level Tcl statements. This allows you to ignore some levels of 1016 <emphasis>case</emphasis> or <emphasis>if</emphasis> statements. 1017 Each procedure call or control structure counts as one ``level''. The 1018 output is recorded in the same file, <filename>dbg.log</filename>, 1019 used for output from <emphasis>--debug</emphasis>.</para></listitem> 1020 </varlistentry> 1021 1022 <varlistentry> 1023 <term><option>--connect [program]</option></term> 1024 <listitem><para>Connect to a target testing environment as specified 1025 by <emphasis>type</emphasis>, if the target is not the computer 1026 running <emphasis>runtest</emphasis>. For example, use 1027 <emphasis>--connect</emphasis> to change the program used to connect 1028 to a ``bare board'' boot monitor. The choices for 1029 <emphasis>type</emphasis> in the DejaGnu 1.4 distribution are 1030 <emphasis>rlogin</emphasis>, <emphasis>telnet</emphasis>, 1031 <emphasis>rsh</emphasis>, <emphasis>tip</emphasis>, 1032 <emphasis>kermit</emphasis>, and <emphasis>mondfe</emphasis>.</para> 1033 1034 <para>The default for this option depends on the configuration most 1035 convenient communication method available, but often other 1036 alternatives work as well; you may find it useful to try alternative 1037 connect methods if you suspect a communication problem with your 1038 testing target.</para></listitem> 1039 </varlistentry> 1040 1041 <varlistentry> 1042 <term><option>--baud [number]</option></term> 1043 <listitem><para>Set the default baud rate to something other than 1044 9600. (Some serial interface programs, like <emphasis>tip</emphasis>, 1045 use a separate initialization file instead of this 1046 value.)</para></listitem> 1047 </varlistentry> 1048 1049 <varlistentry> 1050 <term><option>--target_board [name(s)]</option></term> 1051 <listitem><para>The list of target boards to run tests 1052 on.</para></listitem> 1053 </varlistentry> 1054 1055 <varlistentry id=tool-opt> 1056 <term><option>--tool[name(s)]</option></term> 1057 <listitem><para>Specifies which testsuite to run, and what 1058 initialization module to use. <option>--tool</option> is used 1059 <emphasis>only</emphasis> for these two purposes. It is 1060 <emphasis>not</emphasis> used to name the executable program to 1061 test. Executable tool names (and paths) are recorded in 1062 <filename>site.exp</filename> and you can override them by specifying 1063 Tcl variables on the command line.</para> 1064 1065 <para>For example, including "<option>--tool</option> gcc" on the 1066 <emphasis>runtest</emphasis> command line runs tests from all test 1067 subdirectories whose names match <filename>gcc.*</filename>, and uses 1068 one of the initialization modules named 1069 <filename>config/*-gcc.exp</filename>. To specify the name of the 1070 compiler (perhaps as an alternative path to what 1071 <emphasis>runtest</emphasis> would use by default), use 1072 <emphasis>GCC=binname</emphasis> on the <emphasis>runtest</emphasis> 1073 command line.</para></listitem> 1074 </varlistentry> 1075 1076 <varlistentry> 1077 <term><option>--tool_exec [name]</option></term> 1078 <listitem><para>The path to the tool executable to 1079 test.</para></listitem> 1080 </varlistentry> 1081 1082 <varlistentry> 1083 <term><option>--tool_opts [options]</option></term> 1084 <listitem><para>A list of additional options to pass to the 1085 tool.</para></listitem> 1086 </varlistentry> 1087 1088 <varlistentry> 1089 <term><option>--verbose</option> (-v)</term> 1090 <listitem><para>Turns on more output. Repeating this option increases 1091 the amount of output displayed. Level one (<emphasis>-v</emphasis>) 1092 is simply test output. Level two (<emphasis>-v</emphasis>-v}) shows 1093 messages on options, configuration, and process control. Verbose 1094 messages appear in the detailed (<filename>*.log</filename>) log 1095 file, but not in the summary (<filename>*.sum</filename>) log 1096 file.</para></listitem> 1097 </varlistentry> 1098 1099 <varlistentry> 1100 <term><option>--version</option> (-V)</term> 1101 <listitem><para>Prints out the version numbers of DejaGnu, 1102 <emphasis>expect</emphasis> and Tcl, and exits without running any 1103 tests.</para></listitem> 1104 </varlistentry> 1105 1106 <varlistentry> 1107 <term><option>--D[0-1]</option></term> 1108 <listitem><para>Start the internal Tcl debugger. The Tcl debugger 1109 supports breakpoints, single stepping, and other common debugging 1110 activities. See the document "Debugger for Tcl Applications" by Don 1111 Libes. (Distributed in PostScript form with 1112 <emphasis>expect</emphasis> as the file 1113 <filename>expect/tcl-debug.ps.</filename>. If you specify 1114 <emphasis>-D1</emphasis>, the <emphasis>expect</emphasis> shell stops 1115 at a breakpoint as soon as DejaGnu invokes it. If you specify 1116 <emphasis>-D0</emphasis>, DejaGnu starts as usual, but you can enter 1117 the debugger by sending an interrupt (e.g. by typing 1118 <keycombo><keycap>C</keycap><keycap>c</keycap></keycombo>). 1119 </para></listitem> 1120 </varlistentry> 1121 1122 <varlistentry> 1123 <term><filename>testfile</filename>.exp[=arg(s)]</term> 1124 <listitem><para>Specify the names of testsuites to run. By default, 1125 <emphasis>runtest</emphasis> runs all tests for the tool, but you can 1126 restrict it to particular testsuites by giving the names of the 1127 <emphasis>.exp expect</emphasis> scripts that control 1128 them. <emphasis>testsuite</emphasis>.exp may not include path 1129 information; use plain filenames.</para></listitem> 1130 </varlistentry> 1131 1132 <varlistentry> 1133 <term><filename>testfile</filename>.exp="testfile1 ..."</term> 1134 <listitem><para>Specify a subset of tests in a suite to run. For 1135 compiler or assembler tests, which often use a single 1136 <emphasis>.exp</emphasis> script covering many different source 1137 files, this option allows you to further restrict the tests by 1138 listing particular source files to compile. Some tools even support 1139 wildcards here. The wildcards supported depend upon the tool, but 1140 typically they are <emphasis>?</emphasis>, <emphasis>*</emphasis>, 1141 and <emphasis>[chars]</emphasis>.</para></listitem> 1142 </varlistentry> 1143 1144 <varlistentry> 1145 <term><symbol>tclvar</symbol>=value</term> 1146 <listitem><para>You can define Tcl variables for use by your test 1147 scripts in the same style used with <emphasis>make</emphasis> for 1148 environment variables. For example, <emphasis>runtest 1149 GDB=gdb.old</emphasis> defines a variable called 1150 <command>GDB</command>; when your scripts refer to 1151 <symbol>$GDB</symbol> in this run, they use the value 1152 <emphasis>gdb.old</emphasis>.</para> 1153 1154 <para>The default Tcl variables used for most tools are defined in 1155 the main DejaGnu <emphasis>Makefile</emphasis>; their values are 1156 captured in the <filename>site.exp</filename> file.</para></listitem> 1157 </varlistentry> 1158 </variablelist> 1159 </sect2> 1160 1161 <sect2 id=common xreflabel="Common Operations"> 1162 <title>Common Options</title> 1163 1164 <para>Typically, you don't need must to use any command-line options. 1165 <option>--tool</option> used is only required when there are more than 1166 one testsuite in the same directory. The default options are in the 1167 local site.exp file, created by "make site.exp".</para> 1168 1169 <para>For example, if the directory <filename>gdb/testsuite</filename> 1170 contains a collection of DejaGnu tests for GDB, you can run them like 1171 this:</para> 1172 1173 <screen> 1174 eg$ cd gdb/testsuite 1175 eg$ runtest --tool gdb 1176 </screen> 1177 1178 <para>Test output follows, ending with:</para> 1179 1180 <screen> 1181 === gdb Summary === 1182 1183 # of expected passes 508 1184 # of expected failures 103 1185 /usr/latest/bin/gdb version 4.14.4 -nx 1186 </screen> 1187 1188 <para>You can use the option <emphasis>--srcdir</emphasis> to point to 1189 some other directory containing a collection of tests:</para> 1190 1191 <screen> 1192 eg$ runtest--srcdir /devo/gdb/testsuite 1193 </screen> 1194 1195 <para>By default, <command>runtest</command> prints only the 1196 names of the tests it runs, output from any tests that have unexpected 1197 results, and a summary showing how many tests passed and how many 1198 failed. To display output from all tests (whether or not they behave 1199 as expected), use the <emphasis>--all</emphasis> option. For more 1200 verbose output about processes being run, communication, and so on, use 1201 <emphasis>--verbose</emphasis>. To see even more output, use multiple 1202 <emphasis>--verbose</emphasis> options. for a more detailed explanation 1203 of each <command>runtest</command> option.</para> 1204 1205 <para>Test output goes into two files in your current directory: 1206 summary output in <filename>tool.sum</filename>, 1207 and detailed output in <filename> 1208 tool.log</filename>. (<emphasis>tool</emphasis> 1209 refers to the collection of tests; for example, after a run with 1210 <emphasis>--tool</emphasis> gdb, look for output files 1211 <filename>gdb.sum</filename> and 1212 <filename>gdb.log</filename>.)</para> 1213 </sect2> 1214 </sect1> 1215 1216 <sect1 id=outputfiles xreflabel="Output Files"> 1217 1218 <title>The files DejaGnu produces.</title> 1219 1220 <para>DejaGnu always writes two kinds of output files: summary 1221 logs and detailed logs. The contents of both of these are 1222 determined by your tests.</para> 1223 1224 <para>For troubleshooting, a third kind of output file is useful: 1225 use <option>--debug</option> to request an output file showing 1226 details of what <productname>Expect</productname> is doing 1227 internally.</para> 1228 1229 <sect2 id=sum xreflabel="Summary File"> 1230 <title>Summary File</title> 1231 1232 <para>DejaGnu always produces a summary output file 1233 <filename>tool.sum</filename>. This summary shows the names of 1234 all test files run; for each test file, one line of output from 1235 each <command>pass</command> command (showing status 1236 <emphasis>PASS</emphasis> or <emphasis>XPASS</emphasis>) or 1237 <command>fail</command> command (status 1238 <emphasis>FAIL</emphasis> or <emphasis>XFAIL</emphasis>); 1239 trailing summary statistics that count passing and failing tests 1240 (expected and unexpected); and the full pathname and version 1241 number of the tool tested. (All possible outcomes, and all 1242 errors, are always reflected in the summary output file, 1243 regardless of whether or not you specify 1244 <option>--all</option>.)</para> 1245 1246 <para>If any of your tests use the procedures 1247 <command>unresolved</command>, <command>unsupported</command>, 1248 or <command>runtested</command>, the summary output also 1249 tabulates the corresponding outcomes.</para> 1250 1251 <para>For example, after <command>runtest --tool 1252 binutils</command>, look for a summary log in 1253 <filename>binutils.sum</filename>. Normally, DejaGnu writes this 1254 file in your current working directory; use the 1255 <option>--outdir</option> option to select a different 1256 directory.</para> 1257 1258 <example> 1259 <title>Here is a short sample summary log</title> 1260 1261 <screen> 1262 Test Run By rob on Mon May 25 21:40:57 PDT 1992 1263 === gdb tests === 1264 Running ./gdb.t00/echo.exp ... 1265 PASS: Echo test 1266 Running ./gdb.all/help.exp ... 1267 PASS: help add-symbol-file 1268 PASS: help aliases 1269 PASS: help breakpoint "bre" abbreviation 1270 FAIL: help run "r" abbreviation 1271 Running ./gdb.t10/crossload.exp ... 1272 PASS: m68k-elf (elf-big) explicit format; loaded 1273 XFAIL: mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed C types 1274 === gdb Summary === 1275 # of expected passes 5 1276 # of expected failures 1 1277 # of unexpected failures 1 1278 /usr/latest/bin/gdb version 4.6.5 -q 1279 </screen> 1280 </example> 1281 1282 </sect2> 1283 1284 <sect2 id=log xreflabel="Log File"> 1285 <title>Log File</title> 1286 1287 <para>DejaGnu also saves a detailed log file 1288 <filename>tool.log</filename>, showing any output generated by 1289 tests as well as the summary output. For example, after 1290 <command>runtest --tool binutils</command>, look for a detailed 1291 log in <filename>binutils.log</filename>. Normally, DejaGnu 1292 writes this file in your current working directory; use the 1293 <option>--outdir</option> option to select a different 1294 directory.</para> 1295 1296 1297 <example> 1298 <title>Here is a brief example showing a detailed log for 1299 <productname>G++</productname> tests</title> 1300 1301 <screen> 1302 Test Run By rob on Mon May 25 21:40:43 PDT 1992 1303 1304 === g++ tests === 1305 1306 --- Running ./g++.other/t01-1.exp --- 1307 PASS: operate delete 1308 1309 --- Running ./g++.other/t01-2.exp --- 1310 FAIL: i960 bug EOF 1311 p0000646.C: In function `int warn_return_1 ()': 1312 p0000646.C:109: warning: control reaches end of non-void function 1313 p0000646.C: In function `int warn_return_arg (int)': 1314 p0000646.C:117: warning: control reaches end of non-void function 1315 p0000646.C: In function `int warn_return_sum (int, int)': 1316 p0000646.C:125: warning: control reaches end of non-void function 1317 p0000646.C: In function `struct foo warn_return_foo ()': 1318 p0000646.C:132: warning: control reaches end of non-void function 1319 1320 --- Running ./g++.other/t01-4.exp --- 1321 FAIL: abort 1322 900403_04.C:8: zero width for bit-field `foo' 1323 --- Running ./g++.other/t01-3.exp --- 1324 FAIL: segment violation 1325 900519_12.C:9: parse error before `;' 1326 900519_12.C:12: Segmentation violation 1327 /usr/latest/bin/gcc: Internal compiler error: program cc1plus got fatal signal 1328 1329 === g++ Summary === 1330 1331 # of expected passes 1 1332 # of expected failures 3 1333 /usr/latest/bin/g++ version cygnus-2.0.1 1334 </screen> 1335 </example> 1336 1337 </sect2> 1338 1339 <sect2 id=debugfile xreflabel="Debug Log File"> 1340 <title>Debug Log File</title> 1341 1342 <para>With the <option>--debug</option> option, you can request 1343 a log file showing the output from 1344 <productname>Expect</productname> itself, running in debugging 1345 mode. This file (<filename>dbg.log</filename>, in the directory 1346 where you start <command>runtest</command>) shows each pattern 1347 <productname>Expect</productname> considers in analyzing test 1348 output.</para> 1349 1350 <para>This file reflects each <command>send</command> command, 1351 showing the string sent as input to the tool under test; and 1352 each <productname>Expect</productname> command, showing each 1353 pattern it compares with the tool output.</para> 1354 1355 <example> 1356 <title>The log messages begin with a message of the form</title> 1357 1358 <screen> 1359 1360 expect: does {<symbol>tool output</symbol>} (spawn_id <symbol>n</symbol>) 1361 match pattern {<emphasis>expected pattern</emphasis>}? 1362 1363 </screen> 1364 </example> 1365 1366 <para>For every unsuccessful match, 1367 <productname>Expect</productname> issues a 1368 <emphasis>no</emphasis> after this message; if other patterns 1369 are specified for the same <productname>Expect</productname> 1370 command, they are reflected also, but without the first part of 1371 the message (<emphasis>expect... match pattern</emphasis>).</para> 1372 1373 <para>When <productname>Expect</productname> finds a match, the 1374 log for the successful match ends with <emphasis>yes</emphasis>, 1375 followed by a record of the <productname>Expect</productname> 1376 variables set to describe a successful match.</para> 1377 1378 <example> 1379 <title>Here is an excerpt from the debugging log for a 1380 <productname>GDB</productname> test:</title> 1381 1382 <screen> 1383 send: sent {break gdbme.c:34\n} to spawn id 6 1384 expect: does {} (spawn_id 6) match pattern {Breakpoint.*at.* file 1385 gdbme.c, line 34.*\(gdb\) $}? no 1386 {.*\(gdb\) $}? no 1387 expect: does {} (spawn_id 0) match pattern {return} ? no 1388 {\(y or n\) }? no 1389 {buffer_full}? no 1390 {virtual}? no 1391 {memory}? no 1392 {exhausted}? no 1393 {Undefined}? no 1394 {command}? no 1395 break gdbme.c:34 1396 Breakpoint 8 at 0x23d8: file gdbme.c, line 34. 1397 (gdb) expect: does {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8: 1398 file gdbme.c, line 34.\r\n(gdb) } (spawn_id 6) match pattern 1399 {Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $}? yes 1400 expect: set expect_out(0,start) {18} 1401 expect: set expect_out(0,end) {71} 1402 expect: set expect_out(0,string) {Breakpoint 8 at 0x23d8: file 1403 gdbme.c, line 34.\r\n(gdb) } 1404 epect: set expect_out(spawn_id) {6} 1405 expect: set expect_out(buffer) {break gdbme.c:34\r\nBreakpoint 8 1406 at 0x23d8: file gdbme.c, line 34.\r\n(gdb) } 1407 PASS: 70 0 breakpoint line number in file 1408 </screen> 1409 </example> 1410 1411 <para>This example exhibits three properties of 1412 <productname>Expect</productname> and 1413 <productname>DejaGnu</productname> that might be surprising at 1414 first glance:</para> 1415 1416 <itemizedlist mark="bullet"> 1417 <listitem><para>Empty output for the first attempted match. The 1418 first set of attempted matches shown ran against the output 1419 <emphasis>{}</emphasis> --- that is, no 1420 output. <productname>Expect</productname> begins 1421 attempting to match the patterns supplied immediately; often, 1422 the first pass is against incomplete output (or completely 1423 before all output, as in this case).</para></listitem> 1424 1425 <listitem><para>Interspersed tool output. The beginning of 1426 the log entry for the second attempted match may be hard to 1427 spot: this is because the prompt <emphasis>{(gdb) }</emphasis> 1428 appears on the same line, just before the 1429 <emphasis>expect:</emphasis> that marks the beginning of the 1430 log entry.</para></listitem> 1431 1432 <listitem><para>Fail-safe patterns. Many of the patterns 1433 tested are fail-safe patterns provided by 1434 <productname>GDB</productname> testing utilities, to reduce 1435 possible indeterminacy. It is useful to anticipate potential 1436 variations caused by extreme system conditions 1437 (<productname>GDB</productname> might issue the message 1438 <emphasis>virtual memory exhausted</emphasis> in rare 1439 circumstances), or by changes in the tested program 1440 (<emphasis>Undefined command</emphasis> is the likeliest 1441 outcome if the name of a tested command changes).</para> 1442 1443 <para>The pattern <emphasis>{return}</emphasis> is a 1444 particularly interesting fail-safe to notice; it checks for an 1445 unexpected <keycap>RET</keycap> prompt. This may happen, 1446 for example, if the tested tool can filter output through a 1447 pager.</para> 1448 1449 <para>These fail-safe patterns (like the debugging log itself) 1450 are primarily useful while developing test scripts. Use the 1451 <command>error</command> procedure to make the actions for 1452 fail-safe patterns produce messages starting with 1453 <emphasis>ERROR</emphasis> on standard output, and in the 1454 detailed log file.</para></listitem> 1455 </itemizedlist> 1456 </sect2> 1457 </sect1> 1458 </chapter> 1459 1460 <chapter id=Customizing xreflabel="Customizing DejaGnu"> 1461 <title>Customizing DejaGnu</title> 1462 1463 <para>The site configuration file, <filename>site.exp</filename>, 1464 captures configuration-dependent values and propagates them to the 1465 DejaGnu test environment using Tcl variables. This ties the 1466 DejaGnu test scripts into the <command>configure</command> and 1467 <command>make</command> programs. If this file is setup correctly, 1468 it is possible to execute a testsuite merely by typing 1469 <command>runtest</command>.</para> 1470 1471 <para>DejaGnu supports two <filename>site.exp</filename> 1472 files. The multiple instances of <filename>site.exp</filename> are 1473 loaded in a fixed order built into DejaGnu. The first file loaded 1474 is the local file <filename>site.exp</filename>, and then the 1475 optional global <filename>site.exp</filename> file as 1476 pointed to by the <symbol>DEJAGNU</symbol> environment 1477 variable.</para> 1478 1479 <para>There is an optional <emphasis>master</emphasis> 1480 <filename>site.exp</filename>, capturing configuration values that 1481 apply to DejaGnu across the board, in each configuration-specific 1482 subdirectory of the DejaGnu library directory. 1483 <command>runtest</command> loads these values first. The master 1484 <filename>site.exp</filename> contains the default values for all 1485 targets and hosts supported by DejaGnu. This master file is 1486 identified by setting the environment variable 1487 <symbol>DEJAGNU</symbol> to the name of the file. This is also 1488 refered to as the ``global'' config file.</para> 1489 1490 <para>Any directory containing a configured testsuite also has a 1491 local <filename>site.exp</filename>, capturing configuration values 1492 specific to the tool under test. Since <command>runtest</command> 1493 loads these values last, the individual test configuration can 1494 either rely on and use, or override, any of the global values from 1495 the global <filename>site.exp</filename> file.</para> 1496 1497 <para>You can usually generate or update the testsuite's local 1498 <filename>site.exp</filename> by typing <command>make 1499 site.exp</command> in the testsuite directory, after the test 1500 suite is configured.</para> 1501 1502 <para>You can also have a file in your home directory called 1503 <filename>.dejagnurc</filename>. This gets loaded first before the 1504 other config files. Usually this is used for personal stuff, like 1505 setting the <symbol>all_flag</symbol> so all the output gets 1506 printed, or your own verbosity levels. This file is usually 1507 restricted to setting command line options.</para> 1508 1509 <para>You can further override the default values in a 1510 user-editable section of any <filename>site.exp</filename>, or by 1511 setting variables on the <command>runtest</command> command 1512 line.</para> 1513 1514 <sect1 id=local xreflabel="Local Config File"> 1515 <title>Local Config File</title> 1516 1517 <para>It is usually more convenient to keep these <emphasis>manual 1518 overrides</emphasis> in the <filename>site.exp</filename> 1519 local to each test directory, rather than in the global 1520 <filename>site.exp</filename> in the installed DejaGnu 1521 library. This file is mostly for supplying tool specific info 1522 that is required by the testsuite.</para> 1523 1524 <para>All local <filename>site.exp</filename> files have 1525 two sections, separated by comment text. The first section is 1526 the part that is generated by <command>make</command>. It is 1527 essentially a collection of Tcl variable definitions based on 1528 <filename>Makefile</filename> environment variables. Since they 1529 are generated by <command>make</command>, they contain the 1530 values as specified by <command>configure</command>. (You can 1531 also customize these values by using the <option>--site</option> 1532 option to <command>configure</command>.) In particular, this 1533 section contains the <filename>Makefile</filename> 1534 variables for host and target configuration data. Do not edit 1535 this first section; if you do, your changes are replaced next 1536 time you run <command>make</command>.</para> 1537 1538 <example> 1539 <title>The first section starts with</title> 1540 1541 <programlisting> 1542 ## these variables are automatically generated by make ## 1543 # Do not edit here. If you wish to override these values 1544 # add them to the last section 1545 </programlisting> 1546 </example> 1547 1548 <para>In the second section, you can override any default values 1549 (locally to DejaGnu) for all the variables. The second section 1550 can also contain your preferred defaults for all the command 1551 line options to <command>runtest</command>. This allows you to 1552 easily customize <command>runtest</command> for your preferences 1553 in each configured test-suite tree, so that you need not type 1554 options repeatedly on the command line. (The second section may 1555 also be empty, if you do not wish to override any defaults.)</para> 1556 1557 <example> 1558 <title>The first section ends with this line</title> 1559 1560 <programlisting> 1561 ## All variables above are generated by configure. Do Not Edit ## 1562 </programlisting> 1563 </example> 1564 1565 <para>You can make any changes under this line. If you wish to 1566 redefine a variable in the top section, then just put a 1567 duplicate value in this second section. Usually the values 1568 defined in this config file are related to the configuration of 1569 the test run. This is the ideal place to set the variables 1570 <symbol>host_triplet</symbol>, <symbol>build_triplet</symbol>, 1571 <symbol>target_triplet</symbol>. All other variables are tool 1572 dependant, i.e., for testing a compiler, the value for 1573 <symbol>CC</symbol> might be set to a freshly built binary, as 1574 opposed to one in the user's path.</para> 1575 1576 <para>Here's an example local site.exp file, as used for 1577 <productname>GCC/G++</productname> testing.</para> 1578 1579 <example> 1580 <title>Local Config File</title> 1581 1582 <programlisting> 1583 ## these variables are automatically generated by make ## 1584 # Do not edit here. If you wish to override these values 1585 # add them to the last section 1586 set rootme "/build/devo-builds/i586-pc-linux-gnulibc1/gcc" 1587 set host_triplet i586-pc-linux-gnulibc1 1588 set build_triplet i586-pc-linux-gnulibc1 1589 set target_triplet i586-pc-linux-gnulibc1 1590 set target_alias i586-pc-linux-gnulibc1 1591 set CFLAGS "" 1592 set CXXFLAGS "-isystem /build/devo-builds/i586-pc-linux-gnulibc1/gcc/../libio -isystem $srcdir/../libg++/src -isystem $srcdir/../libio -isystem $srcdir/../libstdc++ -isystem $srcdir/../libstdc++/stl -L/build/devo-builds/i586-pc-linux-gnulibc1/gcc/../libg++ -L/build/devo-builds/i586-pc-linux-gnulibc1/gcc/../libstdc++" 1593 append LDFLAGS " -L/build/devo-builds/i586-pc-linux-gnulibc1/gcc/../ld" 1594 set tmpdir /build/devo-builds/i586-pc-linux-gnulibc1/gcc/testsuite 1595 set srcdir "${srcdir}/testsuite" 1596 ## All variables above are generated by configure. Do Not Edit ## 1597 1598 </programlisting> 1599 </example> 1600 1601 <para>This file defines the required fields for a local config 1602 file, namely the three config triplets, and the srcdir. It also 1603 defines several other Tcl variables that are used exclusivly by 1604 the GCC testsuite. For most test cases, the CXXFLAGS and LDFLAGS 1605 are supplied by DejaGnu itself for cross testing, but to test a 1606 compiler, GCC needs to manipulate these itself.</para> 1607 1608 </sect1> 1609 <sect1 id=global xreflabel="Global Config File"> 1610 <title>Global Config File</title> 1611 1612 <para>The master config file is where all the target specific 1613 config variables for a whole site get set. The idea is 1614 that for a centralized testing lab where people have to share a 1615 target between multiple developers. There are settings for both 1616 remote targets and remote hosts. Here's an example of a Master 1617 Config File (also called the Global config file) for a 1618 <emphasis>canadian cross</emphasis>. A canadian cross is when 1619 you build and test a cross compiler on a machine other than the 1620 one it's to be hosted on.</para> 1621 1622 <para>Here we have the config settings for our California 1623 office. Note that all config values are site dependant. Here we 1624 have two sets of values that we use for testing m68k-aout cross 1625 compilers. As both of these target boards has a different 1626 debugging protocol, we test on both of them in sequence.</para> 1627 1628 <example> 1629 <title>Global Config file</title> 1630 1631 <programlisting> 1632 1633 # Make sure we look in the right place for the board description files. 1634 if ![info exists boards_dir] { 1635 set boards_dir {} 1636 } 1637 lappend boards_dir "/nfs/cygint/s1/cygnus/dejagnu/boards" 1638 1639 verbose "Global Config File: target_triplet is $target_triplet" 2 1640 global target_list 1641 1642 case "$target_triplet" in { 1643 { "native" } { 1644 set target_list "unix" 1645 } 1646 { "sparc64-*elf" } { 1647 set target_list "sparc64-sim" 1648 } 1649 { "mips-*elf" } { 1650 set target_list "mips-sim wilma barney" 1651 } 1652 { "mips-lsi-elf" } { 1653 set target_list "mips-lsi-sim{,soft-float,el}" 1654 } 1655 { "sh-*hms" } { 1656 set target_list { "sh-hms-sim" "bloozy" } 1657 } 1658 } 1659 </programlisting> 1660 </example> 1661 1662 <para>In this case, we have support for several cross compilers, 1663 that all run on this host. For testing on operating systems that 1664 don't support Expect, DejaGnu can be run on the local build 1665 machine, and it can connect to the remote host and run all the 1666 tests for this cross compiler on that host. All the remote OS 1667 requires is a working telnetd.</para> 1668 1669 <para>As you can see, all one does is set the variable 1670 <symbol>target_list</symbol> to the list of targets and options to 1671 test. The simple settings, like for 1672 <emphasis>sparc64-elf</emphasis> only require setting the name of 1673 the single board config file. The <emphasis>mips-elf</emphasis> 1674 target is more complicated. Here it sets the list to three target 1675 boards. One is the default mips target, and both 1676 <emphasis>wilma</emphasis> <emphasis>barney</emphasis> are 1677 symbolic names for other mips boards. Symbolic names are covered 1678 in the <xref linkend=addboard> chapter. The more complicated 1679 example is the one for <emphasis>mips-lsi-elf</emphasis>. This one 1680 runs the tests with multiple iterations using all possible 1681 combinations of the <option>--soft-float</option> and the 1682 <option>--el</option> (little endian) option. Needless to say, 1683 this last feature is mostly compiler specific.</para> 1684 1685 </sect1> 1686 1687 <sect1 id=boardconfig xreflabel="Board Config File"> 1688 <title>Board Config File</title> 1689 1690 <para>The board config file is where board specfic config data 1691 is stored. A board config file contains all the higher-level 1692 configuration settings. There is a rough inheritance scheme, where it is 1693 possible to base a new board description file on an existing one. There 1694 are also collections of custom procedures for common environments. For 1695 more information on adding a new board config file, go to the <xref 1696 linkend=addboard> chapter. </para> 1697 1698 <para>An example board config file for a GNU simulator is as 1699 follows. <function>set_board_info</function> is a procedure that sets the 1700 field name to the specified value. The procedures in square brackets 1701 <emphasis>[]</emphasis> are <emphasis>helper procedures</emphasis>. Thes 1702 are used to find parts of a tool chain required to build an executable 1703 image that may reside in various locations. This is mostly of use for 1704 when the startup code, the standard C lobraries, or the tool chain itself 1705 is part of your build tree.</para> 1706 1707 <example> 1708 <title>Board Config File</title> 1709 1710 <programlisting> 1711 # This is a list of toolchains that are supported on this board. 1712 set_board_info target_install {sparc64-elf} 1713 1714 # Load the generic configuration for this board. This will define any 1715 # routines needed by the tool to communicate with the board. 1716 load_generic_config "sim" 1717 1718 # We need this for find_gcc and *_include_flags/*_link_flags. 1719 load_base_board_description "basic-sim" 1720 1721 # Use long64 by default. 1722 process_multilib_options "long64" 1723 1724 setup_sim sparc64 1725 1726 # We only support newlib on this target. We assume that all multilib 1727 # options have been specified before we get here. 1728 set_board_info compiler "[find_gcc]" 1729 set_board_info cflags "[libgloss_include_flags] [newlib_include_flags]" 1730 set_board_info ldflags "[libgloss_link_flags] [newlib_link_flags]" 1731 # No linker script. 1732 set_board_info ldscript ""; 1733 1734 # Used by a few gcc.c-torture testcases to delimit how large the 1735 # stack can be. 1736 set_board_info gcc,stack_size 16384 1737 # The simulator doesn't return exit statuses and we need to indicate this 1738 # the standard GCC wrapper will work with this target. 1739 set_board_info needs_status_wrapper 1 1740 # We can't pass arguments to programs. 1741 set_board_info noargs 1 1742 </programlisting> 1743 </example> 1744 1745 <para>There are five helper procedures used in this example. The first 1746 one, <function>find gcc</function> looks for a copy of the GNU compiler in 1747 your build tree, or it uses the one in your path. This will also return 1748 the proper transformed name for a cross compiler if you whole build tree 1749 is configured for one. The next helper procedures are 1750 <function>libgloss_include_flags</function> & 1751 <function>libgloss_link_flags</function>. These return the proper flags to 1752 compiler and link an executable image using <xref 1753 linkend=libgloss>, the GNU BSP (Board Support Package). The final 1754 procedures are <function>newlib_include_flag</function> & 1755 <function>newlib_include_flag</function>. These find the Newlib C 1756 library, which is a reentrant standard C library for embedded systems 1757 comprising of non GPL'd code.</para> 1758 1759 </sect1> 1760 1761 <sect1 id=releng xreflabel="Remote Host Testing"> 1762 <title>Remote Host Testing</title> 1763 1764 <note><para>Thanks to Dj Delorie for the original paper that 1765 this section is based on.</para></note> 1766 1767 <para>DejaGnu also supports running the tests on a remote 1768 host. To set this up, the remote host needs an ftp server, and a 1769 telnet server. Currently foreign operating systems used as 1770 remote hosts are VxWorks, VRTX, DOS/Windows 3.1, MacOS and Windows.</para> 1771 1772 <para>The recommended source for a Windows-based FTP 1773 server is to get IIS (either IIS 1 or Personal Web Server) from 1774 <ulink 1775 URL="http://www.microsoft.com">http://www.microsoft.com</ulink>. 1776 When you install it, make sure you install the FTP server - it's 1777 not selected by default. Go into the IIS manager and change the 1778 FTP server so that it does not allow anonymous FTP. Set the home 1779 directory to the root directory (i.e. c:\) of a suitable 1780 drive. Allow writing via FTP.</para> 1781 1782 <para>It will create an account like IUSR_FOOBAR where foobar is 1783 the name of your machine. Go into the user editor and give that 1784 account a password that you don't mind hanging around in the 1785 clear (i.e. not the same as your admin or personal 1786 passwords). Also, add it to all the various permission groups.</para> 1787 1788 <para>You'll also need a telnet server. For Windows, go 1789 to the <ulink URL="http://ataman.com">Ataman</ulink> web site, 1790 pick up the Ataman Remote Logon Services for Windows, and 1791 install it. You can get started on the eval period anyway. Add 1792 IUSR_FOOBAR to the list of allowed users, set the HOME directory 1793 to be the same as the FTP default directory. Change the Mode 1794 prompt to simple.</para> 1795 1796 <para>Ok, now you need to pick a directory name to do all the 1797 testing in. For the sake of this example, we'll call it piggy 1798 (i.e. c:\piggy). Create this directory.</para> 1799 1800 <para>You'll need a unix machine. Create a directory for the 1801 scripts you'll need. For this example, we'll use 1802 /usr/local/swamp/testing. You'll need to have a source tree 1803 somewhere, say /usr/src/devo. Now, copy some files from 1804 releng's area in SV to your machine:</para> 1805 1806 <example> 1807 <title>Remote host setup</title> 1808 1809 <screen> 1810 cd /usr/local/swamp/testing 1811 mkdir boards 1812 scp darkstar.welcomehome.org:/dejagnu/cst/bin/MkTestDir . 1813 scp darkstar.welcomehome.org:/dejagnu/site.exp . 1814 scp darkstar.welcomehome.org:/dejagnu/boards/useless98r2.exp boards/foobar.exp 1815 export DEJAGNU=/usr/local/swamp/testing/site.exp 1816 1817 </screen> 1818 </example> 1819 1820 <para>You must edit the boards/foobar.exp file to reflect your 1821 machine; change the hostname (foobar.com), username 1822 (iusr_foobar), password, and ftp_directory (c:/piggy) to match 1823 what you selected.</para> 1824 1825 <para>Edit the global <filename> site.exp</filename> to reflect your 1826 boards directory:</para> 1827 1828 <example> 1829 <title>Add The Board Directory</title> 1830 1831 <programlisting> 1832 lappend boards_dir "/usr/local/swamp/testing/boards" 1833 </programlisting> 1834 </example> 1835 1836 <para>Now run MkTestDir, which is in the contrib 1837 directory. The first parameter is the toolchain prefix, the 1838 second is the location of your devo tree. If you are testing a 1839 cross compiler (ex: you have sh-hms-gcc.exe in your PATH on 1840 the PC), do something like this:</para> 1841 1842 <example> 1843 <title>Setup Cross Remote Testing</title> 1844 1845 <programlisting> 1846 ./MkTestDir sh-hms /usr/dejagnu/src/devo 1847 </programlisting> 1848 </example> 1849 1850 <para>If you are testing a native PC compiler (ex: you have 1851 gcc.exe in your PATH on the PC), do this:</para> 1852 1853 <example> 1854 <title>Setup Native Remote Testing</title> 1855 1856 <programlisting> 1857 ./MkTestDir '' /usr/dejagnu/src/devo 1858 </programlisting> 1859 </example> 1860 1861 <para>To test the setup, <command>ftp</command> to your PC 1862 using the username (iusr_foobar) and password you selected. CD 1863 to the test directory. Upload a file to the PC. Now telnet to 1864 your PC using the same username and password. CD to the test 1865 directory. Make sure the file is there. Type "set" and/or "gcc 1866 -v" (or sh-hms-gcc -v) and make sure the default PATH contains 1867 the installation you want to test.</para> 1868 1869 <example> 1870 <title>Run Test Remotely</title> 1871 1872 <programlisting> 1873 cd /usr/local/swamp/testing 1874 make -k -w check RUNTESTFLAGS="--host_board foobar --target_board foobar -v -v" > check.out 2>&1 1875 </programlisting> 1876 </example> 1877 1878 <para>To run a specific test, use a command like this (for 1879 this example, you'd run this from the gcc directory that 1880 MkTestDir created):</para> 1881 1882 <example> 1883 <title>Run a Test Remotely</title> 1884 1885 <programlisting> 1886 make check RUNTESTFLAGS="--host_board sloth --target_board sloth -v compile.exp=921202-1.c" 1887 </programlisting> 1888 </example> 1889 1890 <para>Note: if you are testing a cross-compiler, put in the 1891 correct target board. You'll also have to download more .exp 1892 files and modify them for your local configuration. The -v's 1893 are optional.</para> 1894 1895 </sect1> 1896 1897 <sect1 id=configfile xreflabel="Config File Values"> 1898 <title>Config File Values</title> 1899 1900 <para>DejaGnu uses a named array in Tcl to hold all the info for 1901 each machine. In the case of a canadian cross, this means host 1902 information as well as target information. The named array is 1903 called <symbol>target_info</symbol>, and it has two indices. The 1904 following fields are part of the array.</para> 1905 1906 <sect2 id=optiondefs xreflabel="Option Variables"> 1907 <title>Command Line Option Variables</title> 1908 1909 <para>In the user editable second section of the <xref 1910 linkend=personal> you can not only override the configuration 1911 variables captured in the first section, but also specify 1912 default values for all on the <command>runtest</command> 1913 command line options. Save for <option>--debug</option>, 1914 <option>--help</option>, and <option>--version</option>, each 1915 command line option has an associated Tcl variable. Use the 1916 Tcl <command>set</command> command to specify a new default 1917 value (as for the configuration variables). The following 1918 table describes the correspondence between command line 1919 options and variables you can set in 1920 <filename>site.exp</filename>. <xref linkend=invoking>, for 1921 explanations of the command-line options.</para> 1922 1923 <para><table frame=all rowsep=0 colsep=0> 1924 <title>Tcl Variables For Command Line Options</title> 1925 1926 <tgroup cols=3 align="char" rowsep=1 colsep=0> 1927 <thead><row> 1928 <entry>runtest</entry><entry>Tcl</entry> 1929 <entry>option</entry><entry>variable</entry><entry>description</entry> 1930 </row></thead> 1931 <tbody> 1932 1933 <row> 1934 <entry>--all</entry> 1935 <entry>all_flag</entry> 1936 <entry>display all test results if set</entry> 1937 </row> 1938 1939 <row> 1940 <entry>--baud</entry> 1941 <entry>baud</entry> 1942 <entry>set the default baud rate to something other than 1943 9600.</entry> 1944 </row> 1945 1946 <row> 1947 <entry>--connect</entry> 1948 <entry>connectmode</entry> 1949 <entry><command>rlogin</command>, 1950 <command>telnet</command>, <command>rsh</command>, 1951 <command>kermit</command>, <command>tip</command>, or 1952 <command>mondfe</command></entry> 1953 </row> 1954 1955 <row> 1956 <entry>--outdir</entry> 1957 <entry>outdir</entry> 1958 <entry>directory for <filename>tool.sum</filename> and 1959 <filename>tool.log.</filename></entry> 1960 </row> 1961 1962 <row> 1963 <entry>--objdir</entry> 1964 <entry>objdir</entry> 1965 <entry>directory for pre-compiled binaries</entry> 1966 </row> 1967 1968 <row> 1969 <entry>--reboot</entry> 1970 <entry>reboot</entry> 1971 <entry>reboot the target if set to 1972 <emphasis>"1"</emphasis>; do not reboot if set to 1973 <emphasis>"0"</emphasis> (the default).</entry> 1974 </row> 1975 1976 <row> 1977 <entry>--srcdir</entry> 1978 <entry>srcdir</entry> 1979 <entry>directory of test subdirectories</entry> 1980 </row> 1981 1982 <row> 1983 <entry>--strace</entry> 1984 <entry>tracelevel</entry> 1985 <entry>a number: Tcl trace depth</entry> 1986 </row> 1987 1988 <row> 1989 <entry>--tool</entry> 1990 <entry>tool</entry> 1991 <entry>name of tool to test; identifies init, test subdir</entry> 1992 </row> 1993 1994 <row> 1995 <entry>--verbose</entry> 1996 <entry>verbose</entry> 1997 <entry>verbosity level. As option, use multiple times; as 1998 variable, set a number, 0 or greater.</entry> 1999 </row> 2000 2001 <row> 2002 <entry>--target</entry> 2003 <entry>target_triplet</entry> 2004 <entry>The canonical configuration string for the target.</entry> 2005 </row> 2006 2007 <row> 2008 <entry>--host</entry> 2009 <entry>host_triplet</entry> 2010 <entry>The canonical configuration string for the host.</entry> 2011 </row> 2012 2013 <row> 2014 <entry>--build</entry> 2015 <entry>build_triplet</entry> 2016 <entry>The canonical configuration string for the build 2017 host.</entry> 2018 </row> 2019 2020 <row> 2021 <entry>--mail</entry> 2022 <entry>address</entry> 2023 <entry>Email the output log to the specified address.</entry> 2024 </row> 2025 2026 </tbody> 2027 </tgroup> 2028 </table> 2029 </para> 2030 2031 </sect2> 2032 2033 <sect2 id=personal xreflabel="Personal Config File"> 2034 <title>Personal Config File</title> 2035 2036 <para>The personal config file is used to customize 2037 <command>runtest's</command> behaviour for each person. It's 2038 typically used to set the user prefered setting for verbosity, 2039 and any experimental Tcl procedures. My personal 2040 <filename>~/.dejagnurc</filename> file looks like:</para> 2041 2042 <example> 2043 <title>Personal Config File</title> 2044 2045 <programlisting> 2046 set all_flag 1 2047 set RLOGIN /usr/ucb/rlogin 2048 set RSH /usr/local/sbin/ssh 2049 </programlisting> 2050 </example> 2051 2052 <para>Here I set <symbol>all_flag</symbol> so I see all the test 2053 cases that PASS along with the ones that FAIL. I also set 2054 <symbol>RLOGIN</symbol> to the BSD version. I have 2055 <productname>Kerberos</productname> installed, and when I rlogin 2056 to a target board, it usually isn't supported. So I use the non 2057 secure version rather than the default that's in my path. I also 2058 set <symbol>RSH</symbol> to the <productname>SSH</productname> 2059 secure shell, as rsh is mostly used to test unix 2060 machines within a local network here.</para> 2061 2062 </sect2> 2063 </sect1> 2064 2065 </chapter> 2066 2067 <chapter id=Extending xreflabel="Extending DejaGnu"> 2068 <title>Extending DejaGnu</title> 2069 2070 <sect1 id=addsuite xreflabel="Adding a new Testsuite"> 2071 <title>Adding A New Testsuite</title> 2072 2073 <para>The testsuite for a new tool should always be located in that tools 2074 source directory. DejaGnu require the directory be named 2075 <filename>testsuite</filename>. Under this directory, the test cases go 2076 in a subdirectory whose name begins with the tool name. For example, for 2077 a tool named <emphasis>flubber</emphasis>, each subdirectory containing 2078 testsuites must start with <emphasis>"flubber."</emphasis>.</para> 2079 2080 </sect1> 2081 2082 <sect1 id=addtool xreflabel="Adding A New Tool"> 2083 <title>Adding A New Tool</title> 2084 2085 <para>In general, the best way to learn how to write (code or even prose) 2086 is to read something similar. This principle applies to test cases and 2087 to testsuites. Unfortunately, well-established testsuites have a way 2088 of developing their own conventions: as test writers become more 2089 experienced with DejaGnu and with Tcl, they accumulate more utilities, 2090 and take advantage of more and more features of 2091 <productname>Expect</productname> and <productname>Tcl</productname> in 2092 general.</para> 2093 2094 <para>Inspecting such established testsuites may make the prospect of 2095 creating an entirely new testsuite appear overwhelming. Nevertheless, 2096 it is quite straightforward to get a new testsuite going.</para> 2097 2098 <para>There is one testsuite that is guaranteed not to grow more 2099 elaborate over time: both it and the tool it tests were created expressly 2100 to illustrate what it takes to get started with DejaGnu. The 2101 <filename>example/</filename> directory of the DejaGnu distribution 2102 contains both an interactive tool called <command>calc</command>, and a 2103 testsuite for it. Reading this testsuite, and experimenting with it, 2104 is a good way to supplement the information in this section. (Thanks to 2105 Robert Lupton for creating calc and its testsuite---and also the first 2106 version of this section of the manual!)</para> 2107 2108 <para>To help orient you further in this task, here is an outline of the 2109 steps to begin building a testsuite for a program example.</para> 2110 2111 <itemizedlist mark=bullet> 2112 2113 <listitem><para>Create or select a directory to contain your new 2114 collection of tests. Change into that directory (shown here as 2115 <filename>testsuite</filename>):</para> 2116 2117 <para>Create a <filename>configure.in</filename> file in this directory, 2118 to control configuration-dependent choices for your tests. So far as 2119 DejaGnu is concerned, the important thing is to set a value for the 2120 variable <symbol>target_abbrev</symbol>; this value is the link to the 2121 init file you will write soon. (For simplicity, we assume the 2122 environment is Unix, and use <emphasis>unix</emphasis> as the 2123 value.)</para> 2124 2125 <para>What else is needed in <filename>configure.in</filename> depends on 2126 the requirements of your tool, your intended test environments, and which 2127 configure system you use. This example is a minimal configure.in for use 2128 with <productname>GNU Autoconf</productname>. </para></listitem> 2129 2130 <listitem><para>Create <filename>Makefile.in</filename> (if you are using 2131 Autoconf), or <filename>Makefile.am</filename>(if you are using 2132 Automake), the source file used by configure to build your 2133 <filename>Makefile</filename>. If you are using GNU Automake.just add the 2134 keyword <emphasis>dejagnu</emphasis> to the 2135 <emphasis>AUTOMAKE_OPTIONS</emphasis> variable in your 2136 <filename>Makefile.am</filename> file. This will add all the Makefile 2137 support needed to run DejaGnu, and support the <xref linkend=makecheck> 2138 target.</para> 2139 2140 <para>You also need to include two targets important to DejaGnu: 2141 <emphasis>check</emphasis>, to run the tests, and 2142 <emphasis>site.exp</emphasis>, to set up the Tcl copies of 2143 configuration-dependent values. This is called the <xref linkend=local> 2144 The check target must run the <command>runtest</command> program to 2145 execute the tests.</para> 2146 2147 <para>The <filename>site.exp</filename> target should usually set up 2148 (among other things) the $tool variable for the name of your program. If 2149 the local site.exp file is setup correctly, it is possible to execute the 2150 tests by merely typing <command>runtest</command> on the command 2151 line.</para> 2152 2153 <example> 2154 <title>Sample Makefile.in Fragment</title> 2155 2156 <programlisting> 2157 # Look for a local version of DejaGnu, otherwise use one in the path 2158 RUNTEST = `if test -f $(top_srcdir)/../dejagnu/runtest; then \ 2159 echo $(top_srcdir) ../dejagnu/runtest; \ 2160 else \ 2161 echo runtest; \ 2162 fi` 2163 2164 # The flags to pass to runtest 2165 RUNTESTFLAGS = 2166 2167 # Execute the tests 2168 check: site.exp all 2169 $(RUNTEST) $(RUNTESTFLAGS) \ 2170 --tool <symbol>${example}</symbol> --srcdir $(srcdir) 2171 2172 # Make the local config file 2173 site.exp: ./config.status Makefile 2174 @echo "Making a new config file..." 2175 -@rm -f ./tmp? 2176 @touch site.exp 2177 2178 -@mv site.exp site.bak 2179 @echo "## these variables are automatically\ 2180 generated by make ##" > ./tmp0 2181 @echo "# Do not edit here. If you wish to\ 2182 override these values" >> ./tmp0 2183 @echo "# add them to the last section" >> ./tmp0 2184 @echo "set host_os ${host_os}" >> ./tmp0 2185 @echo "set host_alias ${host_alias}" >> ./tmp0 2186 @echo "set host_cpu ${host_cpu}" >> ./tmp0 2187 @echo "set host_vendor ${host_vendor}" >> ./tmp0 2188 @echo "set target_os ${target_os}" >> ./tmp0 2189 @echo "set target_alias ${target_alias}" >> ./tmp0 2190 @echo "set target_cpu ${target_cpu}" >> ./tmp0 2191 @echo "set target_vendor ${target_vendor}" >> ./tmp0 2192 @echo "set host_triplet ${host_canonical}" >> ./tmp0 2193 @echo "set target_triplet ${target_canonical}">>./tmp0 2194 @echo "set tool binutils" >> ./tmp0 2195 @echo "set srcdir ${srcdir}" >> ./tmp0 2196 @echo "set objdir `pwd`" >> ./tmp0 2197 @echo "set <symbol>${examplename}</symbol> <symbol>${example}</symbol>" >> ./tmp0 2198 @echo "## All variables above are generated by\ 2199 configure. Do Not Edit ##" >> ./tmp0 2200 @cat ./tmp0 > site.exp 2201 @sed < site.bak \ 2202 -e '1,/^## All variables above are.*##/ d' \ 2203 >> site.exp 2204 -@rm -f ./tmp? 2205 2206 </programlisting> 2207 </example> 2208 </listitem> 2209 2210 <listitem><para>Create a directory (in <filename>testsuite</filename>) 2211 called <filename>config</filename>. Make a <emphasis>Tool Init 2212 File</emphasis> in this directory. Its name must start with the 2213 <symbol>target_abbrev</symbol> value, or be named 2214 <filename>default.exp</filename> so call it 2215 <filename>config/unix.exp</filename> for our Unix based example. This 2216 is the file that contains the target-dependent procedures. 2217 Fortunately, on Unix, most of them do not have to do very much in 2218 order for <command>runtest</command> to run.</para> 2219 2220 <para>If the program being tested is not interactive, you can get 2221 away with this minimal <filename>unix.exp</filename> to begin 2222 with:</para> 2223 2224 <example> 2225 <title>Simple Batch Program Tool Init File</title> 2226 2227 <programlisting> 2228 2229 proc foo_exit {} {} 2230 proc foo_version {} {} 2231 2232 </programlisting> 2233 </example> 2234 2235 <para>If the program being tested is interactive, however, you might 2236 as well define a <emphasis>start</emphasis> routine and invoke it by 2237 using an init file like this:</para> 2238 2239 <example> 2240 <title>Simple Interactive Program Tool Init File</title> 2241 2242 <programlisting> 2243 2244 proc foo_exit {} {} 2245 proc foo_version {} {} 2246 2247 proc foo_start {} { 2248 global ${examplename} 2249 spawn ${examplename} 2250 expect { 2251 -re "" {} 2252 } 2253 } 2254 2255 # Start the program running we want to test 2256 foo_start 2257 2258 </programlisting> 2259 </example> 2260 </listitem> 2261 2262 <listitem><para>Create a directory whose name begins with your tool's 2263 name, to contain tests. For example, if your tool's name is 2264 <emphasis>gcc</emphasis>, then the directories all need to start with 2265 <emphasis>"gcc."</emphasis>.</para></listitem> 2266 2267 <listitem><para>Create a sample test file. Its name must end with 2268 <filename>.exp</filename>. You can use 2269 <filename>first-try.exp</filename>. To begin with, just write there a 2270 line of Tcl code to issue a message.</para> 2271 2272 <example> 2273 <title>Testing A New Tool Config</title> 2274 2275 <programlisting> 2276 2277 send_user "Testing: one, two...\n" 2278 2279 </programlisting> 2280 </example> 2281 </listitem> 2282 2283 <listitem><para>Back in the <filename>testsuite</filename> (top 2284 level) directory, run <command>configure</command>. Typically you do 2285 this while in the build directory. You may have to specify more of a 2286 path, if a suitable configure is not available in your execution 2287 path.</para></listitem> 2288 2289 <listitem><para>e now ready to triumphantly type <command>make 2290 check</command> or <command>runtest</command>. You should see 2291 something like this:</para> 2292 2293 <example> 2294 <title>Example Test Case Run</title> 2295 2296 <screen> 2297 Test Run By rhl on Fri Jan 29 16:25:44 EST 1993 2298 2299 === example tests === 2300 2301 Running ./example.0/first-try.exp ... 2302 Testing: one, two... 2303 2304 === example Summary === 2305 2306 </screen> 2307 </example> 2308 2309 <para>There is no output in the summary, because so far the example 2310 does not call any of the procedures that establish a test 2311 outcome.</para></listitem> 2312 2313 <listitem><para>Write some real tests. For an interactive tool, you 2314 should probably write a real exit routine in fairly short order. In 2315 any case, you should also write a real version routine 2316 soon. </para></listitem> 2317 2318 </itemizedlist> 2319 2320 </sect1> 2321 2322 <sect1 id=addtarget xreflabel="Adding A New Target"> 2323 <title>Adding A New Target</title> 2324 2325 <para>DejaGnu has some additional requirements for target support, beyond 2326 the general-purpose provisions of configure. DejaGnu must actively 2327 communicate with the target, rather than simply generating or managing 2328 code for the target architecture. Therefore, each tool requires an 2329 initialization module for each target. For new targets, you must supply 2330 a few Tcl procedures to adapt DejaGnu to the target. This permits 2331 DejaGnu itself to remain target independent.</para> 2332 2333 <para>Usually the best way to write a new initialization module is to 2334 edit an existing initialization module; some trial and error will be 2335 required. If necessary, you can use the @samp{--debug} option to see what 2336 is really going on.</para> 2337 2338 <para>When you code an initialization module, be generous in printing 2339 information controlled by the <function>verbose</function> 2340 procedure.</para> 2341 2342 <para>For cross targets, most of the work is in getting the 2343 communications right. Communications code (for several situations 2344 involving IP networks or serial lines) is available in a DejaGnu library 2345 file.</para> 2346 2347 <para>If you suspect a communication problem, try running the connection 2348 interactively from <productname>Expect</productname>. (There are three 2349 ways of running <productname>Expect</productname> as an interactive 2350 interpreter. You can run <productname>Expect</productname> with no 2351 arguments, and control it completely interactively; or you can use 2352 <command>expect -i</command> together with other command-line options and 2353 arguments; or you can run the command <command>interpreter</command> from 2354 any <productname>Expect</productname> procedure. Use 2355 <command>return</command> to get back to the calling procedure (if any), 2356 or <command>return -tcl</command> to make the calling procedure itself 2357 return to its caller; use <command>exi</command>t or end-of-file to leave 2358 Expect altogether.) Run the program whose name is recorded in 2359 <symbol>$connectmode</symbol>, with the arguments in 2360 <symbol>$targetname</symbol>, to establish a connection. You should at 2361 least be able to get a prompt from any target that is physically 2362 connected.</para> 2363 2364 </sect1> 2365 2366 <sect1 id=addboard xreflabel="Adding A New Board"> 2367 <title>Adding A New Board</title> 2368 2369 <para>Adding a new board consists of creating a new board config 2370 file. Examples are in 2371 <filename>dejagnu/baseboards</filename>. Usually to make a new 2372 board file, it's easiest to copy an existing one. It is also 2373 possible to have your file be based on a 2374 <emphasis>baseboard</emphasis> file with only one or two 2375 changes needed. Typically, this can be as simple as just 2376 changing the linker script. Once the new baseboard file is done, 2377 add it to the <symbol>boards_DATA</symbol> list in the 2378 <filename>dejagnu/baseboards/Makefile.am</filename>, and regenerate the 2379 Makefile.in using automake. Then just rebuild and install DejaGnu. You 2380 can test it by:</para> 2381 2382 <para>There is a crude inheritance scheme going on with board files, so 2383 you can include one board file into another, The two main procedures used 2384 to do this are <function>load_generic_config</function> and 2385 <function>load_base_board_description</function>. The generic config file 2386 contains other procedures used for a certain class of target. The 2387 board description file is where the board specfic settings go. Commonly 2388 there are similar target environments with just different 2389 processors.</para> 2390 2391 <example> 2392 <title>Testing a New Board Config File</title> 2393 2394 <screen> 2395 make check RUNTESTFLAGS="--target_board=<emphasis>newboardfile</emphasis>". 2396 </screen> 2397 </example> 2398 2399 <para>Here's an example of a board config file. There are 2400 several <emphasis>helper procedures</emphasis> used in this 2401 example. A helper procedure is one that look for a tool of files 2402 in commonly installed locations. These are mostly used when 2403 testing in the build tree, because the executables to be tested 2404 are in the same tree as the new dejagnu files. The helper 2405 procedures are the ones in square braces 2406 <emphasis>[]</emphasis>, which is the Tcl execution characters.</para> 2407 2408 <example> 2409 <title>Example Board Config File</title> 2410 2411 <programlisting> 2412 2413 # Load the generic configuration for this board. This will define a basic 2414 # set of routines needed by the tool to communicate with the board. 2415 load_generic_config "sim" 2416 2417 # basic-sim.exp is a basic description for the standard Cygnus simulator. 2418 load_base_board_description "basic-sim" 2419 2420 # The compiler used to build for this board. This has *nothing* to do 2421 # with what compiler is tested if we're testing gcc. 2422 set_board_info compiler "[find_gcc]" 2423 2424 # We only support newlib on this target. 2425 # However, we include libgloss so we can find the linker scripts. 2426 set_board_info cflags "[newlib_include_flags] [libgloss_include_flags]" 2427 set_board_info ldflags "[newlib_link_flags]" 2428 2429 # No linker script for this board. 2430 set_board_info ldscript "-Tsim.ld"; 2431 2432 # The simulator doesn't return exit statuses and we need to indicate this. 2433 set_board_info needs_status_wrapper 1 2434 2435 # Can't pass arguments to this target. 2436 set_board_info noargs 1 2437 2438 # No signals. 2439 set_board_info gdb,nosignals 1 2440 2441 # And it can't call functions. 2442 set_board_info gdb,cannot_call_functions 1 2443 2444 </programlisting> 2445 </example> 2446 2447 </sect1> 2448 2449 <sect1 id=boarddefs xreflabel="Board File Values"> 2450 <title>Board Config File Values</title> 2451 2452 <para>These fields are all in the <symbol>board_info</symbol> These are 2453 all set by using the <function>set_board_info</function> procedure. The 2454 parameters are the field name, followed by the value to set the field 2455 to.</para> 2456 2457 <para><table frame=all rowsep=0 colsep=0> 2458 <title>Common Board Info Fields</title> 2459 2460 <tgroup cols=3 align="char" rowsep=1 colsep=0> 2461 <thead><row> 2462 <entry>Field</entry> 2463 <entry>Sample Value</entry> 2464 <entry>Description</entry> 2465 </row></thead> 2466 <tbody> 2467 2468 <row> 2469 <entry>compiler</entry> 2470 <entry>"[find_gcc]"</entry> 2471 <entry>The path to the compiler to use.</entry> 2472 </row> 2473 2474 <row> 2475 <entry>cflags</entry> 2476 <entry>"-mca"</entry> 2477 <entry>Compilation flags for the compiler.</entry> 2478 </row> 2479 2480 <row> 2481 <entry>ldflags</entry> 2482 <entry>"[libgloss_link_flags] [newlib_link_flags]"</entry> 2483 <entry>Linking flags for the compiler.</entry> 2484 </row> 2485 2486 <row> 2487 <entry>ldscript</entry> 2488 <entry>"-Wl,-Tidt.ld"</entry> 2489 <entry>The linker script to use when cross compiling.</entry> 2490 </row> 2491 2492 <row> 2493 <entry>libs</entry> 2494 <entry>"-lgcc"</entry> 2495 <entry>Any additional libraries to link in.</entry> 2496 </row> 2497 2498 <row> 2499 <entry>shell_prompt</entry> 2500 <entry>"cygmon>"</entry> 2501 <entry>The command prompt of the remote shell.</entry> 2502 </row> 2503 2504 <row> 2505 <entry>hex_startaddr</entry> 2506 <entry>"0xa0020000"</entry> 2507 <entry>The Starting address as a string.</entry> 2508 </row> 2509 2510 <row> 2511 <entry>start_addr</entry> 2512 <entry>0xa0008000</entry> 2513 <entry>The starting address as a value.</entry> 2514 </row> 2515 2516 <row> 2517 <entry>startaddr</entry> 2518 <entry>"a0020000"</entry> 2519 <entry></entry> 2520 </row> 2521 2522 <row> 2523 <entry>exit_statuses_bad</entry> 2524 <entry>1</entry> 2525 <entry>Whether there is an accurate exit status.</entry> 2526 </row> 2527 2528 <row> 2529 <entry>reboot_delay</entry> 2530 <entry>10</entry> 2531 <entry>The delay between power off and power on.</entry> 2532 </row> 2533 2534 <row> 2535 <entry>unreliable</entry> 2536 <entry>1</entry> 2537 <entry>Whether communication with the board is unreliable.</entry> 2538 </row> 2539 2540 <row> 2541 <entry>sim</entry> 2542 <entry>[find_sim]</entry> 2543 <entry>The path to the simulator to use.</entry> 2544 </row> 2545 2546 <row> 2547 <entry>objcopy</entry> 2548 <entry>$tempfil</entry> 2549 <entry>The path to the <command>objcopy</command> program.</entry> 2550 </row> 2551 2552 <row> 2553 <entry>support_libs</entry> 2554 <entry>"${prefix_dir}/i386-coff/"</entry> 2555 <entry>Support libraries needed for cross compiling.</entry> 2556 </row> 2557 2558 <row> 2559 <entry>addl_link_flags</entry> 2560 <entry>"-N"</entry> 2561 <entry>Additional link flags, rarely used.</entry> 2562 </row> 2563 2564 </tbody> 2565 </tgroup> 2566 </table> 2567 </para> 2568 2569 <para>These fields are used by the GCC and GDB tests, and are mostly 2570 only useful to somewhat trying to debug a new board file for one of 2571 these tools. Many of these are used only by a few testcases, and their 2572 purpose is esoteric. These are listed with sample values as a guide to 2573 better guessing if you need to change any of these.</para> 2574 2575 <para><table frame=all rowsep=0 colsep=0> 2576 <title>Board Info Fields For GCC & GDB</title> 2577 2578 <tgroup cols=3 align="char" rowsep=1 colsep=0> 2579 <thead><row> 2580 <entry>Field</entry> 2581 <entry>Sample Value</entry> 2582 <entry>Description</entry> 2583 </row></thead> 2584 <tbody> 2585 2586 <row> 2587 <entry>strip</entry> 2588 <entry>$tempfile</entry> 2589 <entry>Strip the executable of symbols.</entry> 2590 </row> 2591 2592 <row> 2593 <entry>gdb_load_offset</entry> 2594 <entry>"0x40050000"</entry> 2595 </row> 2596 2597 <row> 2598 <entry>gdb_protocol</entry> 2599 <entry>"remote"</entry> 2600 <entry>The GDB debugging protocol to use.</entry> 2601 </row> 2602 2603 <row> 2604 <entry>gdb_sect_offset</entry> 2605 <entry>"0x41000000";</entry> 2606 </row> 2607 2608 <row> 2609 <entry>gdb_stub_ldscript</entry> 2610 <entry>"-Wl,-Teva-stub.ld"</entry> 2611 <entry>The linker script to use with a GDB stub.</entry> 2612 </row> 2613 2614 <row> 2615 <entry>gdb_init_command</entry> 2616 <entry>"set mipsfpu none"</entry> 2617 </row> 2618 2619 <row> 2620 <entry>gdb,cannot_call_functions</entry> 2621 <entry>1</entry> 2622 <entry>Whether GDB can call functions on the target,</entry> 2623 </row> 2624 2625 <row> 2626 <entry>gdb,noargs</entry> 2627 <entry>1</entry> 2628 <entry>Whether the target can take command line arguments.</entry> 2629 </row> 2630 2631 <row> 2632 <entry>gdb,nosignals</entry> 2633 <entry>1</entry> 2634 <entry>Whether there are signals on the target.</entry> 2635 </row> 2636 2637 <row> 2638 <entry>gdb,short_int</entry> 2639 <entry>1</entry> 2640 </row> 2641 2642 <row> 2643 <entry>gdb,start_symbol</entry> 2644 <entry>"_start";</entry> 2645 <entry>The starting symbol in the executable.</entry> 2646 </row> 2647 2648 <row> 2649 <entry>gdb,target_sim_options</entry> 2650 <entry>"-sparclite"</entry> 2651 <entry>Special options to pass to the simulator.</entry> 2652 </row> 2653 2654 <row> 2655 <entry>gdb,timeout</entry> 2656 <entry>540</entry> 2657 <entry>Timeout value to use for remote communication.</entry> 2658 </row> 2659 2660 <row> 2661 <entry>gdb_init_command</entry> 2662 <entry>"print/x \$fsr = 0x0"</entry> 2663 </row> 2664 2665 <row> 2666 <entry>gdb_load_offset</entry> 2667 <entry>"0x12020000"</entry> 2668 </row> 2669 2670 <row> 2671 <entry>gdb_opts</entry> 2672 <entry>"--command gdbinit"</entry> 2673 </row> 2674 2675 <row> 2676 <entry>gdb_prompt</entry> 2677 <entry>"\\(gdb960\\)"</entry> 2678 <entry>The prompt GDB is using.</entry> 2679 </row> 2680 2681 <row> 2682 <entry>gdb_run_command</entry> 2683 <entry>"jump start"</entry> 2684 </row> 2685 2686 <row> 2687 <entry>gdb_stub_offset</entry> 2688 <entry>"0x12010000"</entry> 2689 </row> 2690 2691 <row> 2692 <entry>use_gdb_stub</entry> 2693 <entry>1</entry> 2694 <entry>Whether to use a GDB stub.</entry> 2695 </row> 2696 2697 <row> 2698 <entry>use_vma_offset</entry> 2699 <entry>1</entry> 2700 </row> 2701 2702 <row> 2703 <entry>wrap_m68k_aout</entry> 2704 <entry>1</entry> 2705 </row> 2706 2707 <row> 2708 <entry>gcc,no_label_values</entry> 2709 <entry>1</entry> 2710 </row> 2711 2712 <row> 2713 <entry>gcc,no_trampolines</entry> 2714 <entry>1</entry> 2715 </row> 2716 2717 <row> 2718 <entry>gcc,no_varargs</entry> 2719 <entry>1</entry> 2720 </row> 2721 2722 <row> 2723 <entry>gcc,stack_size</entry> 2724 <entry>16384</entry> 2725 <entry>Stack size to use with some GCC testcases.</entry> 2726 </row> 2727 2728 <row> 2729 <entry>ieee_multilib_flags</entry> 2730 <entry>"-mieee";</entry> 2731 </row> 2732 2733 <row> 2734 <entry>is_simulator</entry> 2735 <entry>1</entry> 2736 </row> 2737 2738 <row> 2739 <entry>needs_status_wrapper</entry> 2740 <entry>1</entry> 2741 </row> 2742 2743 <row> 2744 <entry>no_double</entry> 2745 <entry>1</entry> 2746 </row> 2747 2748 <row> 2749 <entry>no_long_long</entry> 2750 <entry>1</entry> 2751 </row> 2752 2753 <row> 2754 <entry>noargs</entry> 2755 <entry>1</entry> 2756 </row> 2757 2758 <row> 2759 <entry>nullstone,lib</entry> 2760 <entry>"mips-clock.c"</entry> 2761 </row> 2762 2763 <row> 2764 <entry>nullstone,ticks_per_sec</entry> 2765 <entry>3782018</entry> 2766 </row> 2767 2768 <row> 2769 <entry>sys_speed_value</entry> 2770 <entry>200</entry> 2771 </row> 2772 2773 <row> 2774 <entry>target_install</entry> 2775 <entry>{sh-hms}</entry> 2776 </row> 2777 2778 </tbody> 2779 </tgroup> 2780 </table> 2781 </para> 2782 2783 </sect1> 2784 2785 <sect1 id=writing xreflabel="Writing A Test Case"> 2786 <title>Writing A Test Case</title> 2787 2788 <para>The easiest way to prepare a new test case is to base it 2789 on an existing one for a similar situation. There are two major 2790 categories of tests: batch or interactive. Batch oriented tests 2791 are usually easier to write.</para> 2792 2793 <para>The GCC tests are a good example of batch oriented tests. 2794 All GCC tests consist primarily of a call to a single common 2795 procedure, Since all the tests either have no output, or only 2796 have a few warning messages when successfully compiled. Any 2797 non-warning output is a test failure. All the C code needed is 2798 kept in the test directory. The test driver, written in Tcl, 2799 need only get a listing of all the C files in the directory, and 2800 compile them all using a generic procedure. This procedure and a 2801 few others supporting for these tests are kept in the library 2802 module <filename>lib/c-torture.exp</filename> in the GCC test 2803 suite. Most tests of this kind use very few 2804 <productname>expect</productname> features, and are coded almost 2805 purely in Tcl.</para> 2806 2807 <para>Writing the complete suite of C tests, then, consisted of 2808 these steps:</para> 2809 2810 <itemizedlist mark=bullet> 2811 <listitem><para>Copying all the C code into the test directory. 2812 These tests were based on the C-torture test created by Torbjorn 2813 Granlund (on behalf of the Free Software Foundation) for GCC 2814 development.</para></listitem> 2815 2816 <listitem><para>Writing (and debugging) the generic Tcl procedures for 2817 compilation.</para></listitem> 2818 2819 <listitem><para>Writing the simple test driver: its main task is to 2820 search the directory (using the Tcl procedure 2821 <emphasis>glob</emphasis> for filename expansion with wildcards) 2822 and call a Tcl procedure with each filename. It also checks for 2823 a few errors from the testing procedure.</para></listitem> 2824 </itemizedlist> 2825 2826 <para>Testing interactive programs is intrinsically more 2827 complex. Tests for most interactive programs require some trial 2828 and error before they are complete.</para> 2829 2830 <para>However, some interactive programs can be tested in a 2831 simple fashion reminiscent of batch tests. For example, prior 2832 to the creation of DejaGnu, the GDB distribution already 2833 included a wide-ranging testing procedure. This procedure was 2834 very robust, and had already undergone much more debugging and 2835 error checking than many recent DejaGnu test cases. 2836 Accordingly, the best approach was simply to encapsulate the 2837 existing GDB tests, for reporting purposes. Thereafter, new GDB 2838 tests built up a family of Tcl procedures specialized for GDB 2839 testing.</para> 2840 2841 </sect1> 2842 2843 <sect1 id=debugging xreflabel="Debugging A Test Case"> 2844 <title>Debugging A Test Case</title> 2845 2846 <para>These are the kinds of debugging information available 2847 from DejaGnu:</para> 2848 2849 <itemizedlist mark=bullet> 2850 2851 <listitem><para>Output controlled by test scripts themselves, 2852 explicitly allowed for by the test author. This kind of 2853 debugging output appears in the detailed output recorded in the 2854 DejaGnu log file. To do the same for new tests, use the 2855 <command>verbose</command> procedure (which in turn uses the 2856 variable also called <emphasis>verbose</emphasis>) to control 2857 how much output to generate. This will make it easier for other 2858 people running the test to debug it if necessary. Whenever 2859 possible, if <emphasis>$verbose</emphasis> is 2860 <emphasis>0</emphasis>, there should be no output other than the 2861 output from <emphasis>pass</emphasis>, 2862 <emphasis>fail</emphasis>, <emphasis>error</emphasis>, and 2863 <emphasis>warning</emphasis>. Then, to whatever extent is 2864 appropriate for the particular test, allow successively higher 2865 values of <emphasis>$verbose</emphasis> to generate more 2866 information. Be kind to other programmers who use your tests: 2867 provide for a lot of debugging information.</para></listitem> 2868 2869 <listitem><para>Output from the internal debugging functions of 2870 Tcl and <productname>Expect</productname>. There is a command 2871 line options for each; both forms of debugging output are 2872 recorded in the file <filename>dbg.log</filename> in the current 2873 directory.</para> 2874 2875 <para>Use <option>--debug</option> for information from the 2876 expect level; it generates displays of the expect attempts to 2877 match the tool output with the patterns specified. This output 2878 can be very helpful while developing test scripts, since it 2879 shows precisely the characters received. Iterating between the 2880 latest attempt at a new test script and the corresponding 2881 <filename>dbg.log</filename> can allow you to create the final 2882 patterns by ``cut and paste''. This is sometimes the best way 2883 to write a test case.</para></listitem> 2884 2885 <listitem><para>Use <option>--strace</option> to see more 2886 detail at the Tcl level; this shows how Tcl procedure 2887 definitions expand, as they execute. The associated number 2888 controls the depth of definitions expanded.</para></listitem> 2889 2890 <listitem><para>Finally, if the value of 2891 <emphasis>verbose</emphasis> is 3 or greater,DejaGnu turns on 2892 the expect command <command>log_user</command>. This command 2893 prints all expect actions to the expect standard output, to the 2894 detailed log file, and (if <option>--debug</option> is on) to 2895 <filename>dbg.log</filename>.</para></listitem> 2896 </itemizedlist> 2897 2898 </sect1> 2899 2900 <sect1 id=adding xreflabel="Adding A Test Case To A Testsuite"> 2901 <title>Adding A Test Case To A Testsuite.</title> 2902 2903 <para>There are two slightly different ways to add a test 2904 case. One is to add the test case to an existing directory. The 2905 other is to create a new directory to hold your test. The 2906 existing test directories represent several styles of testing, 2907 all of which are slightly different; examine the directories for 2908 the tool of interest to see which (if any) is most suitable.</para> 2909 2910 <para>Adding a GCC test can be very simple: just add the C code 2911 to any directory beginning with <filename>gcc</filename>. and it 2912 runs on the next <programlisting>runtest --tool 2913 gcc</programlisting>.</para> 2914 2915 <para>To add a test to GDB, first add any source code you will 2916 need to the test directory. Then you can either create a new 2917 expect file, or add your test to an existing one (any 2918 file with a <emphasis>.exp</emphasis> suffix). Creating a new 2919 .exp file is probably a better idea if the test is significantly 2920 different from existing tests. Adding it as a separate file also 2921 makes upgrading easier. If the C code has to be already compiled 2922 before the test will run, then you'll have to add it to the 2923 <filename>Makefile.in</filename> file for that test directory, 2924 then run <command>configure</command> and 2925 <command>make</command>.</para> 2926 2927 <para>Adding a test by creating a new directory is very 2928 similar:</para> 2929 2930 <itemizedlist mark=bullet> 2931 2932 <listitem><para>Create the new directory. All subdirectory names 2933 begin with the name of the tool to test; e.g. G++ tests might be 2934 in a directory called <filename>g++.other</filename>. There can 2935 be multiple test directories that start with the same tool name 2936 (such as <emphasis>g++</emphasis>).</para></listitem> 2937 2938 <listitem><para>Add the new directory name to the 2939 <symbol>configdirs</symbol> definition in the 2940 <filename>configure.in</filename> file for the testsuite 2941 directory. This way when <command>make</command> and 2942 <command>configure</command> next run, they include the new 2943 directory.</para></listitem> 2944 2945 <listitem><para>Add the new test case to the directory, as 2946 above. </para></listitem> 2947 2948 <listitem><para>To add support in the new directory for 2949 configure and make, you must also create a 2950 <filename>Makefile.in</filename> and a 2951 <filename>configure.in</filename>.</para></listitem> 2952 </itemizedlist> 2953 2954 </sect1> 2955 2956 <sect1 id=hints xreflabel="Hints On Writing A Test Case"> 2957 <title>Hints On Writing A Test Case</title> 2958 2959 <para>It is safest to write patterns that match all the output 2960 generated by the tested program; this is called closure. 2961 If a pattern does not match the entire output, any output that 2962 remains will be examined by the next <command>expect</command> 2963 command. In this situation, the precise boundary that determines 2964 which <command>expect</command> command sees what is very 2965 sensitive to timing between the Expect task and the task running 2966 the tested tool. As a result, the test may sometimes appear to 2967 work, but is likely to have unpredictable results. (This problem 2968 is particularly likely for interactive tools, but can also 2969 affect batch tools---especially for tests that take a long time 2970 to finish.) The best way to ensure closure is to use the 2971 <option>-re</option> option for the <command>expect</command> 2972 command to write the pattern as a full regular expressions; then 2973 you can match the end of output using a <emphasis>$</emphasis>. 2974 It is also a good idea to write patterns that match all 2975 available output by using <emphasis>.*\</emphasis> after the 2976 text of interest; this will also match any intervening blank 2977 lines. Sometimes an alternative is to match end of line using 2978 <emphasis>\r</emphasis> or <emphasis>\n</emphasis>, but this is 2979 usually too dependent on terminal settings.</para> 2980 2981 <para>Always escape punctuation, such as <emphasis>(</emphasis> 2982 or <emphasis>"</emphasis>, in your patterns; for example, write 2983 <emphasis>\(</emphasis>. If you forget to escape punctuation, 2984 you will usually see an error message like <programlisting>extra 2985 characters after close-quote.</programlisting></para> 2986 2987 <para>If you have trouble understanding why a pattern does not 2988 match the program output, try using the <option>--debug</option> 2989 option to <command>runtest</command>, and examine the debug log 2990 carefully.</para> 2991 2992 <para>Be careful not to neglect output generated by setup rather 2993 than by the interesting parts of a test case. For example, 2994 while testing GDB, I issue a send <emphasis>set height 2995 0\n</emphasis> command. The purpose is simply to make sure GDB 2996 never calls a paging program. The <emphasis>set 2997 height</emphasis> command in GDB does not generate any 2998 output; but running any command makes GDB issue a new 2999 <emphasis>(gdb) </emphasis> prompt. If there were no 3000 <command>expect</command> command to match this prompt, the 3001 output <emphasis>(gdb) </emphasis> begins the text seen by the 3002 next <command>expect</command> command---which might make that 3003 pattern fail to match.</para> 3004 3005 <para>To preserve basic sanity, I also recommended that no test 3006 ever pass if there was any kind of problem in the test case. To 3007 take an extreme case, tests that pass even when the tool will 3008 not spawn are misleading. Ideally, a test in this sort of 3009 situation should not fail either. Instead, print an error 3010 message by calling one of the DejaGnu procedures 3011 <command>error</command> or <command>warning</command>.</para> 3012 3013 </sect1> 3014 3015 <sect1 id=tvariables xreflabel="Test Case Variables"> 3016 <title>Special variables used by test cases.</title> 3017 3018 <para>There are special variables used by test cases. These contain 3019 other information from DejaGnu. Your test cases can use these variables, 3020 with conventional meanings (as well as the variables saved in 3021 <filename>site.exp</filename>. You can use the value of these variables, 3022 but they should never be changed.</para> 3023 3024 <variablelist> 3025 <varlistentry> 3026 <term>$prms_id</term> 3027 <listitem><para>The tracking system (e.g. GNATS) number identifying 3028 a corresponding bugreport. (<emphasis>0</emphasis>} if you do not 3029 specify it in the test script.)</para></listitem> 3030 </varlistentry> 3031 3032 <varlistentry> 3033 <term>$item bug_id</term> 3034 <listitem><para>An optional bug id; may reflect a bug 3035 identification from another organization. (<emphasis>0</emphasis> 3036 if you do not specify it.)</para></listitem> 3037 </varlistentry> 3038 3039 <varlistentry> 3040 <term>$subdir</term> 3041 <listitem><para>The subdirectory for the current test 3042 case.</para></listitem> 3043 </varlistentry> 3044 3045 <varlistentry> 3046 <term>$expect_out(buffer)</term> 3047 <listitem><para>The output from the last command. This is an 3048 internal variable set by Expect. More information can be found in 3049 the Expect manual.</para></listitem> 3050 </varlistentry> 3051 3052 <varlistentry> 3053 <term>$exec_output</term> 3054 <listitem><para>This is the output from a 3055 <function>${tool}_load</function> command. This only applies to 3056 tools like GCC and GAS which produce an object file that must in 3057 turn be executed to complete a test.</para></listitem> 3058 </varlistentry> 3059 3060 <varlistentry> 3061 <term>$comp_output</term> 3062 <listitem><para>This is the output from a 3063 <function>${tool}_start</function> command. This is conventionally 3064 used for batch oriented programs, like GCC and GAS, that may 3065 produce interesting output (warnings, errors) without further 3066 interaction.</para></listitem> 3067 </varlistentry> 3068 </variablelist> 3069 3070 </sect1> 3071 3072</chapter> 3073 3074 <chapter id=unit xreflabel="Unit Testing"> 3075 <title>Unit Testing</title> 3076 3077 <sect1 id=unittest xreflabel="What Is Unit Testing ?"> 3078 <title>What Is Unit Testing ?</title> 3079 3080 <para>Most regression testing as done by DejaGnu is system 3081 testing. This is the complete application is tested all at 3082 once. Unit testing is for testing single files, or small 3083 libraries. In this case, each file is linked with a test case in 3084 C or C++, and each function or class and method is tested in 3085 series, with the test case having to check private data or 3086 global variables to see if the function or method worked.</para> 3087 3088 <para>This works particularly well for testing APIs and at level 3089 where it is easier to debug them, than by needing to trace through 3090 the entire appication. Also if there is a specification for the 3091 API to be tested, the testcase can also function as a compliance 3092 test.</para> 3093 3094 </sect1> 3095 3096 <sect1 id=djh xreflabel="The dejagnu.h Header File"> 3097 <title>The dejagnu.h Header File</title> 3098 3099 <para>DejaGnu uses a single header file to assist in unit 3100 testing. As this file also produces it's one test state output, 3101 it can be run standalone, which is very useful for testing on 3102 embedded systems. This header file has a C and C++ API for the 3103 test states, with simple totals, and standardized 3104 output. Because the output has been standardized, DejaGnu can be 3105 made to work with this test case, without writing almost any 3106 Tcl. The library module, dejagnu.exp, will look for the output 3107 messages, and then merge them into DejaGnu's.</para> 3108 3109 </sect1> 3110 3111</chapter> 3112 3113<!-- Keep this comment at the end of the file 3114Local variables: 3115mode: sgml 3116sgml-omittag:t 3117sgml-shorttag:t 3118sgml-namecase-general:t 3119sgml-general-insert-case:lower 3120sgml-minimize-attributes:nil 3121sgml-always-quote-attributes:t 3122sgml-indent-step:1 3123sgml-indent-data:nil 3124sgml-parent-document:nil 3125sgml-exposed-tags:nil 3126sgml-local-catalogs:nil 3127sgml-local-ecat-files:nil 3128End: 3129--> 3130