1c243e490SMarcel MoolenaarMajor changes between releases Automated Testing Framework 2c243e490SMarcel Moolenaar=========================================================================== 3c243e490SMarcel Moolenaar 4c243e490SMarcel Moolenaar 51a61beb0SJulio MerinoChanges in version 0.20 61a61beb0SJulio Merino*********************** 71a61beb0SJulio Merino 81a61beb0SJulio MerinoExperimental version released on February 7th, 2014. 91a61beb0SJulio Merino 101a61beb0SJulio MerinoThis is the first release without the code for the deprecated tools. If 111a61beb0SJulio Merinoyou require such code, please fetch a copy of the 0.19 release and extract 121a61beb0SJulio Merinothe 'tools' directory for your own consumption. 131a61beb0SJulio Merino 141a61beb0SJulio Merino* Removed the deprecated tools. This includes atf-config, atf-report, 151a61beb0SJulio Merino atf-run and atf-version. 161a61beb0SJulio Merino 171a61beb0SJulio Merino 181a61beb0SJulio MerinoChanges in version 0.19 191a61beb0SJulio Merino*********************** 201a61beb0SJulio Merino 211a61beb0SJulio MerinoExperimental version released on February 7th, 2014. 221a61beb0SJulio Merino 231a61beb0SJulio MerinoThis is the last release to bundle the code for the deprecated tools. 241a61beb0SJulio MerinoThe next release will drop their code and will stop worrying about 251a61beb0SJulio Merinobackwards compatibility between the ATF libraries and what the old tools 261a61beb0SJulio Merinomay or may not support. 271a61beb0SJulio Merino 281a61beb0SJulio MerinoIf you still require the old tools for some reason, grab a copy of the 291a61beb0SJulio Merino'tools' directory now. The code in this directory is standalone and 301a61beb0SJulio Merinodoes not depend on any internal details of atf-c++ any longer. 311a61beb0SJulio Merino 321a61beb0SJulio Merino* Various fixes and improvements to support running as part of the FreeBSD 331a61beb0SJulio Merino test suite. 341a61beb0SJulio Merino 351a61beb0SJulio Merino* Project hosting moved from Google Code (as a subproject of Kyua) to 361a61beb0SJulio Merino GitHub (as a first-class project). The main reason for the change is 371a61beb0SJulio Merino the suppression of binary downloads in Google Code on Jan 15th, 2014. 381a61beb0SJulio Merino See https://github.com/jmmv/atf/ 391a61beb0SJulio Merino 401a61beb0SJulio Merino* Removed builtin help from atf-sh(1) and atf-check(1) for simplicity 411a61beb0SJulio Merino reasons. In other words, their -h option is gone. 421a61beb0SJulio Merino 431a61beb0SJulio Merino* Moved the code of the deprecated tools into a 'tools' directory and 441a61beb0SJulio Merino completely decoupled their code from the internals of atf-c++. The 451a61beb0SJulio Merino reason for this is to painlessly allow a third-party to maintain a 461a61beb0SJulio Merino copy of these tools after we delete them because upcoming changes to 471a61beb0SJulio Merino atf-c++ would break the stale tools. 481a61beb0SJulio Merino 491a61beb0SJulio Merino 50a18eacbeSJulio MerinoChanges in version 0.18 51a18eacbeSJulio Merino*********************** 52a18eacbeSJulio Merino 53a18eacbeSJulio MerinoExperimental version released on November 16th, 2013. 54a18eacbeSJulio Merino 55a18eacbeSJulio Merino* Issue 45: Added require.memory support in atf-run for FreeBSD. 56a18eacbeSJulio Merino 57a18eacbeSJulio Merino* Fixed an issue with the handling of cin with libc++. 58a18eacbeSJulio Merino 59a18eacbeSJulio Merino* Issue 64: Fixed various mandoc formatting warnings. 60a18eacbeSJulio Merino 61a18eacbeSJulio Merino* NetBSD PR bin/48284: Made atf-check flush its progress message to 62a18eacbeSJulio Merino stdout so that an interrupted test case always shows the last message 63a18eacbeSJulio Merino being executed. 64a18eacbeSJulio Merino 65a18eacbeSJulio Merino* NetBSD PR bin/48285: Fixed atf_check examples in atf-sh-api(3). 66a18eacbeSJulio Merino 67a18eacbeSJulio Merino 68a18eacbeSJulio MerinoChanges in version 0.17 69a18eacbeSJulio Merino*********************** 70a18eacbeSJulio Merino 71a18eacbeSJulio MerinoExperimental version released on February 14th, 2013. 72a18eacbeSJulio Merino 73a18eacbeSJulio Merino* Added the atf_utils_cat_file, atf_utils_compare_file, 74a18eacbeSJulio Merino atf_utils_copy_file, atf_utils_create_file, atf_utils_file_exists, 75a18eacbeSJulio Merino atf_utils_fork, atf_utils_grep_file, atf_utils_grep_string, 76a18eacbeSJulio Merino atf_utils_readline, atf_utils_redirect and atf_utils_wait utility 77a18eacbeSJulio Merino functions to atf-c-api. Documented the already-public 78a18eacbeSJulio Merino atf_utils_free_charpp function. 79a18eacbeSJulio Merino 80a18eacbeSJulio Merino* Added the cat_file, compare_file, copy_file, create_file, file_exists, 81a18eacbeSJulio Merino fork, grep_collection, grep_file, grep_string, redirect and wait 82a18eacbeSJulio Merino functions to the atf::utils namespace of atf-c++-api. These are 83a18eacbeSJulio Merino wrappers around the same functions added to the atf-c-api library. 84a18eacbeSJulio Merino 85a18eacbeSJulio Merino* Added the ATF_CHECK_MATCH, ATF_CHECK_MATCH_MSG, ATF_REQUIRE_MATCH and 86a18eacbeSJulio Merino ATF_REQUIRE_MATCH_MSG macros to atf-c to simplify the validation of a 87a18eacbeSJulio Merino string against a regular expression. 88a18eacbeSJulio Merino 89a18eacbeSJulio Merino* Miscellaneous fixes for manpage typos and compilation problems with 90a18eacbeSJulio Merino clang. 91a18eacbeSJulio Merino 92a18eacbeSJulio Merino* Added caching of the results of those configure tests that rely on 93a18eacbeSJulio Merino executing a test program. This should help crossbuild systems by 94a18eacbeSJulio Merino providing a mechanism to pre-specify what the results should be. 95a18eacbeSJulio Merino 96a18eacbeSJulio Merino* PR bin/45690: Make atf-report convert any non-printable characters to 97a18eacbeSJulio Merino a plain-text representation (matching their corresponding hexadecimal 98a18eacbeSJulio Merino entities) in XML output files. This is to prevent the output of test 99a18eacbeSJulio Merino cases from breaking xsltproc later. 100a18eacbeSJulio Merino 101a18eacbeSJulio Merino 102c243e490SMarcel MoolenaarChanges in version 0.16 103c243e490SMarcel Moolenaar*********************** 104c243e490SMarcel Moolenaar 105c243e490SMarcel MoolenaarExperimental version released on July 10th, 2012. 106c243e490SMarcel Moolenaar 107c243e490SMarcel Moolenaar* Added a --enable-tools flag to configure to request the build of the 108c243e490SMarcel Moolenaar deprecated ATF tools, whose build is now disabled by default. In order 109c243e490SMarcel Moolenaar to continue running tests, you should migrate to Kyua instead of enabling 110c243e490SMarcel Moolenaar the build of the deprecated tools. The kyua-atf-compat package provides 111c243e490SMarcel Moolenaar transitional compatibility versions of atf-run and atf-report built on 112c243e490SMarcel Moolenaar top of Kyua. 113c243e490SMarcel Moolenaar 114c243e490SMarcel Moolenaar* Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can 115c243e490SMarcel Moolenaar detect defined but unused test cases. 116c243e490SMarcel Moolenaar 117c243e490SMarcel Moolenaar* PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and 118c243e490SMarcel Moolenaar tp-time XML tags leaking into the generated HTML file. Also improved 119c243e490SMarcel Moolenaar the CSS file slightly to correct alignment and color issues with the 120c243e490SMarcel Moolenaar timestamps column. 121c243e490SMarcel Moolenaar 122c243e490SMarcel Moolenaar* Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during 123c243e490SMarcel Moolenaar compilation with GNU G++. 124c243e490SMarcel Moolenaar 125c243e490SMarcel Moolenaar* Flipped the default to building shared libraries for atf-c and atf-c++, 126c243e490SMarcel Moolenaar and started versioning them. As a side-effect, this removes the 127c243e490SMarcel Moolenaar --enable-unstable-shared flag from configure that appears to not work any 128c243e490SMarcel Moolenaar more (under NetBSD). Additionally, some distributions require the use of 129c243e490SMarcel Moolenaar shared libraries for proper dependency tracking (e.g. Fedora), so it is 130c243e490SMarcel Moolenaar better if we do the right versioning upstream. 131c243e490SMarcel Moolenaar 132c243e490SMarcel Moolenaar* Project hosting moved from an adhoc solution (custom web site and 133c243e490SMarcel Moolenaar Monotone repository) to Google Code (standard wiki and Git). ATF now 134c243e490SMarcel Moolenaar lives in a subcomponent of the Kyua project. 135c243e490SMarcel Moolenaar 136c243e490SMarcel Moolenaar 137c243e490SMarcel MoolenaarChanges in version 0.15 138c243e490SMarcel Moolenaar*********************** 139c243e490SMarcel Moolenaar 140c243e490SMarcel MoolenaarExperimental version released on January 16th, 2012. 141c243e490SMarcel Moolenaar 142c243e490SMarcel Moolenaar* Respect stdin in atf-check. The previous release silenced stdin for any 143c243e490SMarcel Moolenaar processes spawned by atf, not only test programs, which caused breakage 144c243e490SMarcel Moolenaar in tests that pipe data through atf-check. 145c243e490SMarcel Moolenaar 146c243e490SMarcel Moolenaar* Performance improvements to atf-sh. 147c243e490SMarcel Moolenaar 148c243e490SMarcel Moolenaar* Enabled detection of unused parameters and variables in the code and 149c243e490SMarcel Moolenaar fixed all warnings. 150c243e490SMarcel Moolenaar 151c243e490SMarcel Moolenaar* Changed the behavior of "developer mode". Compiler warnings are now 152c243e490SMarcel Moolenaar enabled unconditionally regardless of whether we are in developer mode or 153c243e490SMarcel Moolenaar not; developer mode is now only used to perform strict warning checks and 154c243e490SMarcel Moolenaar to enable assertions. Additionally, developer mode is now only 155c243e490SMarcel Moolenaar automatically enabled when building from the repository, not for formal 156c243e490SMarcel Moolenaar releases. 157c243e490SMarcel Moolenaar 158c243e490SMarcel Moolenaar* Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and 159c243e490SMarcel Moolenaar ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag 160c243e490SMarcel Moolenaar in configure scripts and detecting the presence of any of the ATF 161c243e490SMarcel Moolenaar bindings. Note that ATF_CHECK_SH was already introduced in 0.14, but it 162c243e490SMarcel Moolenaar has now been modified to also honor --with-atf if instantiated. 163c243e490SMarcel Moolenaar 164c243e490SMarcel Moolenaar* Added timing support to atf-run / atf-report. 165c243e490SMarcel Moolenaar 166c243e490SMarcel Moolenaar* Added support for a 'require.memory' property, to specify the minimum 167c243e490SMarcel Moolenaar amount of physical memory needed by the test case to yield valid results. 168c243e490SMarcel Moolenaar 169c243e490SMarcel Moolenaar* PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by 170c243e490SMarcel Moolenaar atf-report so that invalid data in the output of test cases does not 171c243e490SMarcel Moolenaar mangle our report. 172c243e490SMarcel Moolenaar 173c243e490SMarcel Moolenaar 174c243e490SMarcel MoolenaarChanges in version 0.14 175c243e490SMarcel Moolenaar*********************** 176c243e490SMarcel Moolenaar 177c243e490SMarcel MoolenaarExperimental version released on June 14th, 2011. 178c243e490SMarcel Moolenaar 179c243e490SMarcel Moolenaar* Added a pkg-config file for atf-sh and an aclocal file to ease the 180c243e490SMarcel Moolenaar detection of atf-sh from autoconf scripts. 181c243e490SMarcel Moolenaar 182c243e490SMarcel Moolenaar* Made the default test case body defined by atf_sh fail. This is to 183c243e490SMarcel Moolenaar ensure that test cases are properly defined in test programs and helps 184c243e490SMarcel Moolenaar in catching typos in the names of the body functions. 185c243e490SMarcel Moolenaar 186c243e490SMarcel Moolenaar* PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero. 187c243e490SMarcel Moolenaar This provides more consistent results with "normal" execution (in 188c243e490SMarcel Moolenaar particular, when tests are executed detached from a terminal). 189c243e490SMarcel Moolenaar 190c243e490SMarcel Moolenaar* Made atf-run hardcode TZ=UTC for test cases. It used to undefine TZ, but 191c243e490SMarcel Moolenaar that does not take into account that libc determines the current timezone 192c243e490SMarcel Moolenaar from a configuration file. 193c243e490SMarcel Moolenaar 194c243e490SMarcel Moolenaar* All test programs will now print a warning when they are not run through 195c243e490SMarcel Moolenaar atf-run(1) stating that this is unsupported and may deliver incorrect 196c243e490SMarcel Moolenaar results. 197c243e490SMarcel Moolenaar 198c243e490SMarcel Moolenaar* Added support for the 'require.files' test-case property. This allows 199c243e490SMarcel Moolenaar test cases to specify installed files that must be present for the test 200c243e490SMarcel Moolenaar case to run. 201c243e490SMarcel Moolenaar 202c243e490SMarcel Moolenaar 203c243e490SMarcel MoolenaarChanges in version 0.13 204c243e490SMarcel Moolenaar*********************** 205c243e490SMarcel Moolenaar 206c243e490SMarcel MoolenaarExperimental version released on March 31st, 2011. 207c243e490SMarcel Moolenaar 208c243e490SMarcel MoolenaarThis is the first release after the creation of the Kyua project, a more 209c243e490SMarcel Moolenaarmodular and reliable replacement for ATF. From now on, ATF will change to 210c243e490SMarcel Moolenaaraccomodate the transition to this new codebase, but ATF will still continue 211c243e490SMarcel Moolenaarto see development in the short/medium term. Check out the project page at 212c243e490SMarcel Moolenaarhttp://code.google.com/p/kyua/ for more details. 213c243e490SMarcel Moolenaar 214c243e490SMarcel MoolenaarThe changes in this release are: 215c243e490SMarcel Moolenaar 216c243e490SMarcel Moolenaar* Added support to run the tests with the Kyua runtime engine (kyua-cli), a 217c243e490SMarcel Moolenaar new package that aims to replace atf-run and atf-report. The ATF tests 218c243e490SMarcel Moolenaar can be run with the new system by issuing a 'make installcheck-kyua' from 219c243e490SMarcel Moolenaar the top-level directory of the project (assuming the 'kyua' binary is 220c243e490SMarcel Moolenaar available during the configuration stage of ATF). 221c243e490SMarcel Moolenaar 222c243e490SMarcel Moolenaar* atf-run and atf-report are now in maintenance mode (but *not* deprecated 223c243e490SMarcel Moolenaar yet!). Kyua already implements a new, much more reliable runtime engine 224c243e490SMarcel Moolenaar that provides similar features to these tools. That said, it is not 225c243e490SMarcel Moolenaar complete yet so all development efforts should go towards it. 226c243e490SMarcel Moolenaar 227c243e490SMarcel Moolenaar* If GDB is installed, atf-run dumps the stack trace of crashing test 228c243e490SMarcel Moolenaar programs in an attempt to aid debugging. Contributed by Antti Kantee. 229c243e490SMarcel Moolenaar 230c243e490SMarcel Moolenaar* Reverted default timeout change in previous release and reset its value 231c243e490SMarcel Moolenaar to 5 minutes. This was causing several issues, specially when running 232c243e490SMarcel Moolenaar the existing NetBSD test suite in qemu. 233c243e490SMarcel Moolenaar 234c243e490SMarcel Moolenaar* Fixed the 'match' output checker in atf-check to properly validate the 235c243e490SMarcel Moolenaar last line of a file even if it does not have a newline. 236c243e490SMarcel Moolenaar 237c243e490SMarcel Moolenaar* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to 238c243e490SMarcel Moolenaar check for the presence (or lack thereof) of an element in a collection. 239c243e490SMarcel Moolenaar 240c243e490SMarcel Moolenaar* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run 241c243e490SMarcel Moolenaar when the cleanup of a test case triggered asynchronous modifications to 242c243e490SMarcel Moolenaar its work directory (e.g. killing a daemon process that cleans up a pid 243c243e490SMarcel Moolenaar file in the work directory). 244c243e490SMarcel Moolenaar 245c243e490SMarcel Moolenaar* PR bin/44301: Fixed the sample XSLT file to report bogus test programs 246c243e490SMarcel Moolenaar instead of just listing them as having 0 test cases. 247c243e490SMarcel Moolenaar 248c243e490SMarcel Moolenaar 249c243e490SMarcel MoolenaarChanges in version 0.12 250c243e490SMarcel Moolenaar*********************** 251c243e490SMarcel Moolenaar 252c243e490SMarcel MoolenaarExperimental version released on November 7th, 2010. 253c243e490SMarcel Moolenaar 254c243e490SMarcel Moolenaar* Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as 255c243e490SMarcel Moolenaar ATF_REQUIRE_THROW but allows checking for the validity of the exception's 256c243e490SMarcel Moolenaar error message by means of a regular expression. 257c243e490SMarcel Moolenaar 258c243e490SMarcel Moolenaar* Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a 259c243e490SMarcel Moolenaar regular expression match in a string. 260c243e490SMarcel Moolenaar 261c243e490SMarcel Moolenaar* Changed the default timeout for test cases from 5 minutes to 30 seconds. 262c243e490SMarcel Moolenaar 30 seconds is long enough for virtually all tests to complete, and 5 263c243e490SMarcel Moolenaar minutes is a way too long pause in a test suite where a single test case 264c243e490SMarcel Moolenaar stalls. 265c243e490SMarcel Moolenaar 266c243e490SMarcel Moolenaar* Deprecated the use.fs property. While this seemed like a good idea in 267c243e490SMarcel Moolenaar the first place to impose more control on what test cases can do, it 268c243e490SMarcel Moolenaar turns out to be bad. First, use.fs=false prevents bogus test cases 269c243e490SMarcel Moolenaar from dumping core so after-the-fact debugging is harder. Second, 270c243e490SMarcel Moolenaar supporting use.fs adds a lot of unnecessary complexity. atf-run will 271c243e490SMarcel Moolenaar now ignore any value provided to use.fs and will allow test cases to 272c243e490SMarcel Moolenaar freely access the file system if they wish to. 273c243e490SMarcel Moolenaar 274c243e490SMarcel Moolenaar* Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c 275c243e490SMarcel Moolenaar library. The 'text' module became private in 0.11 but was being used 276c243e490SMarcel Moolenaar externally to simplify the parsing of configuration variables. 277c243e490SMarcel Moolenaar 278c243e490SMarcel Moolenaar* Made atf-run recognize the 'unprivileged-user' configuration variable 279c243e490SMarcel Moolenaar and automatically drop root privileges when a test case sets 280c243e490SMarcel Moolenaar require.user=unprivileged. Note that this is, by no means, done for 281c243e490SMarcel Moolenaar security purposes; this is just for user convenience; tests should, in 282c243e490SMarcel Moolenaar general, not be blindly run as root in the first place. 283c243e490SMarcel Moolenaar 284c243e490SMarcel Moolenaar 285c243e490SMarcel MoolenaarChanges in version 0.11 286c243e490SMarcel Moolenaar*********************** 287c243e490SMarcel Moolenaar 288c243e490SMarcel MoolenaarExperimental version released on October 20th, 2010. 289c243e490SMarcel Moolenaar 290c243e490SMarcel Moolenaar* The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match 291c243e490SMarcel Moolenaar their counterparts in atf-c. 292c243e490SMarcel Moolenaar 293c243e490SMarcel Moolenaar* Clearly separated the modules in atf-c that are supposed to be public 294c243e490SMarcel Moolenaar from those that are implementation details. The header files for the 295c243e490SMarcel Moolenaar internal modules are not installed any more. 296c243e490SMarcel Moolenaar 297c243e490SMarcel Moolenaar* Made the atf-check tool private. It is only required by atf-sh and being 298c243e490SMarcel Moolenaar public has the danger of causing confusion. Also, making it private 299c243e490SMarcel Moolenaar simplifies the public API of atf. 300c243e490SMarcel Moolenaar 301c243e490SMarcel Moolenaar* Changed atf-sh to enable per-command error checking (set -e) by default. 302c243e490SMarcel Moolenaar This catches many cases in which a test case is broken but it is not 303c243e490SMarcel Moolenaar reported as such because execution continues. 304c243e490SMarcel Moolenaar 305c243e490SMarcel Moolenaar* Fixed the XSTL and CSS stylesheets to support expected failures. 306c243e490SMarcel Moolenaar 307c243e490SMarcel Moolenaar 308c243e490SMarcel MoolenaarChanges in version 0.10 309c243e490SMarcel Moolenaar*********************** 310c243e490SMarcel Moolenaar 311c243e490SMarcel MoolenaarExperimental version released on July 2nd, 2010. 312c243e490SMarcel Moolenaar 313c243e490SMarcel MoolenaarMiscellaneous features 314c243e490SMarcel Moolenaar 315c243e490SMarcel Moolenaar* Added expected failures support to test cases and atf-run. These 316c243e490SMarcel Moolenaar include, for example, expected clean exits, expected reception of fatal 317c243e490SMarcel Moolenaar signals, expected timeouts and expected errors in condition checks. 318c243e490SMarcel Moolenaar These statuses can be used to denote test cases that are known to fail 319c243e490SMarcel Moolenaar due to a bug in the code they are testing. atf-report reports these 320c243e490SMarcel Moolenaar tests separately but they do not count towards the failed test cases 321c243e490SMarcel Moolenaar amount. 322c243e490SMarcel Moolenaar 323c243e490SMarcel Moolenaar* Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to 324c243e490SMarcel Moolenaar allow easy checking of call failures that update errno. 325c243e490SMarcel Moolenaar 326c243e490SMarcel Moolenaar* Added the has.cleanup meta-data property to test caes that specifies 327c243e490SMarcel Moolenaar whether the test case has a cleanup routine or not; its value is 328c243e490SMarcel Moolenaar automatically set. This property is read by atf-run to know if it has to 329c243e490SMarcel Moolenaar run the cleanup routine; skipping this run for every test case 330c243e490SMarcel Moolenaar significantly speeds up the run time of test suites. 331c243e490SMarcel Moolenaar 332c243e490SMarcel Moolenaar* Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to 333c243e490SMarcel Moolenaar take the expected exception as the first argument and the statement to 334c243e490SMarcel Moolenaar execute as the second argument. 335c243e490SMarcel Moolenaar 336c243e490SMarcel MoolenaarChanges in atf-check 337c243e490SMarcel Moolenaar 338c243e490SMarcel Moolenaar* Changed atf-check to support negating the status and output checks by 339c243e490SMarcel Moolenaar prefixing them with not- and added support to specify multiple checkers 340c243e490SMarcel Moolenaar for stdout and stderr, not only one. 341c243e490SMarcel Moolenaar 342c243e490SMarcel Moolenaar* Added the match output checker to atf-check to look for regular 343c243e490SMarcel Moolenaar expressions in the stdout and stderr of commands. 344c243e490SMarcel Moolenaar 345c243e490SMarcel Moolenaar* Modified the exit checks in atf-check to support checking for the 346c243e490SMarcel Moolenaar reception of signals. 347c243e490SMarcel Moolenaar 348c243e490SMarcel MoolenaarCode simplifications and cleanups 349c243e490SMarcel Moolenaar 350c243e490SMarcel Moolenaar* Removed usage messages from test programs to simplify the 351c243e490SMarcel Moolenaar implementation of every binding by a significant amount. They just now 352c243e490SMarcel Moolenaar refer the user to the appropriate manual page and do not attempt to wrap 353c243e490SMarcel Moolenaar lines on terminal boundaries. Test programs are not supposed to be run 354c243e490SMarcel Moolenaar by users directly so this minor interface regression is not important. 355c243e490SMarcel Moolenaar 356c243e490SMarcel Moolenaar* Removed the atf-format internal utility, which is unused after the 357c243e490SMarcel Moolenaar change documented above. 358c243e490SMarcel Moolenaar 359c243e490SMarcel Moolenaar* Removed the atf-cleanup internal utility. It has been unused since the 360c243e490SMarcel Moolenaar test case isolation was moved to atf-run in 0.8 361c243e490SMarcel Moolenaar 362c243e490SMarcel Moolenaar* Splitted the Makefile.am into smaller files for easier maintenance and 363c243e490SMarcel Moolenaar dropped the use of M4. Only affects users building from the repository 364c243e490SMarcel Moolenaar sources. 365c243e490SMarcel Moolenaar 366c243e490SMarcel Moolenaar* Intermixed tests with the source files in the source tree to provide 367c243e490SMarcel Moolenaar them more visibility and easier access. The tests directory is gone from 368c243e490SMarcel Moolenaar the source tree and tests are now suffixed by _test, not prefixed by t_. 369c243e490SMarcel Moolenaar 370c243e490SMarcel Moolenaar* Simplifications to the atf-c library: removed the io, tcr and ui 371c243e490SMarcel Moolenaar modules as they had become unnecessary after all simplifications 372c243e490SMarcel Moolenaar introduced since the 0.8 release. 373c243e490SMarcel Moolenaar 374c243e490SMarcel Moolenaar* Removed the application/X-atf-tcr format introduced in 0.8 release. 375c243e490SMarcel Moolenaar Tests now print a much simplified format that is easy to parse and nicer 376c243e490SMarcel Moolenaar to read by end users. As a side effect, the default for test cases is 377c243e490SMarcel Moolenaar now to print their results to stdout unless otherwise stated by providing 378c243e490SMarcel Moolenaar the -r flag. 379c243e490SMarcel Moolenaar 380c243e490SMarcel Moolenaar* Removed XML distribution documents and replaced them with plain-text 381c243e490SMarcel Moolenaar documents. They provided little value and introduced a lot of complexity 382c243e490SMarcel Moolenaar to the build system. 383c243e490SMarcel Moolenaar 384c243e490SMarcel Moolenaar* Simplified the output of atf-version by not attempting to print a 385c243e490SMarcel Moolenaar revision number when building form a distfile. Makes the build system 386c243e490SMarcel Moolenaar easier to maintain. 387c243e490SMarcel Moolenaar 388c243e490SMarcel Moolenaar 389c243e490SMarcel MoolenaarChanges in version 0.9 390c243e490SMarcel Moolenaar********************** 391c243e490SMarcel Moolenaar 392c243e490SMarcel MoolenaarExperimental version released on June 3rd, 2010. 393c243e490SMarcel Moolenaar 394c243e490SMarcel Moolenaar* Added atf-sh, an interpreter to process test programs written using 395c243e490SMarcel Moolenaar the shell API. This is not really a shell interpreter by itself though: 396c243e490SMarcel Moolenaar it is just a wrapper around the system shell that eases the loading of 397c243e490SMarcel Moolenaar the necessary ATF libraries. 398c243e490SMarcel Moolenaar 399c243e490SMarcel Moolenaar* Removed atf-compile in favour of atf-sh. 400c243e490SMarcel Moolenaar 401c243e490SMarcel Moolenaar* Added the use.fs metadata property to test case, which is used to 402c243e490SMarcel Moolenaar specify which test cases require file system access. This is to 403c243e490SMarcel Moolenaar highlight dependencies on external resources more clearly and to speed up 404c243e490SMarcel Moolenaar the execution of test suites by skipping the creation of many unnecessary 405c243e490SMarcel Moolenaar work directories. 406c243e490SMarcel Moolenaar 407c243e490SMarcel Moolenaar* Fixed test programs to get a sane default value for their source 408c243e490SMarcel Moolenaar directory. This means that it should not be necessary any more to pass 409c243e490SMarcel Moolenaar -s when running test programs that do not live in the current directory. 410c243e490SMarcel Moolenaar 411c243e490SMarcel Moolenaar* Defining test case headers became optional. This is trivial to achieve 412c243e490SMarcel Moolenaar in shell-based tests but a bit ugly in C and C++. In C, use the new 413c243e490SMarcel Moolenaar ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use 414c243e490SMarcel Moolenaar ATF_TEST_CASE_WITHOUT_HEAD. 415c243e490SMarcel Moolenaar 416c243e490SMarcel Moolenaar 417c243e490SMarcel MoolenaarChanges in version 0.8 418c243e490SMarcel Moolenaar********************** 419c243e490SMarcel Moolenaar 420c243e490SMarcel MoolenaarExperimental version released on May 7th, 2010. 421c243e490SMarcel Moolenaar 422c243e490SMarcel Moolenaar* Test programs no longer run several test cases in a row. The execution 423c243e490SMarcel Moolenaar of a test program now requires a test case name, and that single test 424c243e490SMarcel Moolenaar case is executed. To execute several test cases, use the atf-run utility 425c243e490SMarcel Moolenaar as usual. 426c243e490SMarcel Moolenaar 427c243e490SMarcel Moolenaar* Test programs no longer fork a subprocess to isolate the execution of 428c243e490SMarcel Moolenaar test cases. They run the test case code in-process, and a crash of the 429c243e490SMarcel Moolenaar test case will result in a crash of the test program. This is to ease 430c243e490SMarcel Moolenaar debugging of faulty test cases. 431c243e490SMarcel Moolenaar 432c243e490SMarcel Moolenaar* Test programs no longer isolate their test cases. This means that they 433c243e490SMarcel Moolenaar will not create temporary directories nor sanitize the environment any 434c243e490SMarcel Moolenaar more. Yes: running a test case that depends on system state by hand will 435c243e490SMarcel Moolenaar most likely yield different results depending on where (machine, 436c243e490SMarcel Moolenaar directory, user environment, etc.) it is run. Isolation has been moved 437c243e490SMarcel Moolenaar to atf-run. 438c243e490SMarcel Moolenaar 439c243e490SMarcel Moolenaar* Test programs no longer print a cryptic format (application/X-atf-tcs) 440c243e490SMarcel Moolenaar on a special file channel. They can now print whatever they want on the 441c243e490SMarcel Moolenaar screen. Because test programs can now only run one test case every time, 442c243e490SMarcel Moolenaar providing controlled output is not necessary any more. 443c243e490SMarcel Moolenaar 444c243e490SMarcel Moolenaar* Test programs no longer write their status into a special file 445c243e490SMarcel Moolenaar descriptor. Instead, they create a file with the results, which is later 446c243e490SMarcel Moolenaar parsed by atf-run. This changes the semantics of the -r flag. 447c243e490SMarcel Moolenaar 448c243e490SMarcel Moolenaar* atf-run has been adjusted to perform the test case isolation. As a 449c243e490SMarcel Moolenaar result, there is now a single canonical place that implements the 450c243e490SMarcel Moolenaar isolation of test caes. In previous releases, the three language 451c243e490SMarcel Moolenaar bindings (C, C++ and shell) had to be kept in sync with each other (read: 452c243e490SMarcel Moolenaar not a nice thing to do at all). As a side effect of this change, writing 453c243e490SMarcel Moolenaar bindings for other languages will be much, much easier from now on. 454c243e490SMarcel Moolenaar 455c243e490SMarcel Moolenaar* atf-run forks test programs on a test case basis, instead of on a test 456c243e490SMarcel Moolenaar program basis as it did before. This is to provide the test case 457c243e490SMarcel Moolenaar isolation that was before implemented by the test programs themselves. 458c243e490SMarcel Moolenaar 459c243e490SMarcel Moolenaar* Removed the atf-exec tool. This was used to implement test case 460c243e490SMarcel Moolenaar isolation in atf-sh, but it is now unnecessary. 461c243e490SMarcel Moolenaar 462c243e490SMarcel Moolenaar* It is now optional to define the descr meta-data property. It has been 463c243e490SMarcel Moolenaar proven to be mostly useless, because test cases often carry a descriptive 464c243e490SMarcel Moolenaar name of their own. 465c243e490SMarcel Moolenaar 466c243e490SMarcel Moolenaar 467c243e490SMarcel MoolenaarChanges in version 0.7 468c243e490SMarcel Moolenaar********************** 469c243e490SMarcel Moolenaar 470c243e490SMarcel MoolenaarExperimental version released on December 22nd, 2009. 471c243e490SMarcel Moolenaar 472c243e490SMarcel Moolenaar* Added build-time checks to atf-c and atf-c++. A binding for atf-sh 473c243e490SMarcel Moolenaar will come later. 474c243e490SMarcel Moolenaar 475c243e490SMarcel Moolenaar* Migrated all build-time checks for header files to proper ATF tests. 476c243e490SMarcel Moolenaar This demonstrates the use of the new feature described above. 477c243e490SMarcel Moolenaar 478c243e490SMarcel Moolenaar* Added an internal API for child process management. 479c243e490SMarcel Moolenaar 480c243e490SMarcel Moolenaar* Converted all plain-text distribution documents to a Docbook canonical 481c243e490SMarcel Moolenaar version, and include pre-generated plain text and HTML copies in the 482c243e490SMarcel Moolenaar distribution file. 483c243e490SMarcel Moolenaar 484c243e490SMarcel Moolenaar* Simplified the contents of the Makefile.am by regenerating it from a 485c243e490SMarcel Moolenaar canonical Makefile.am.m4 source. As a side-effect, some dependency 486c243e490SMarcel Moolenaar specifications were fixed. 487c243e490SMarcel Moolenaar 488c243e490SMarcel Moolenaar* Migrated all checks from the check target to installcheck, as these 489c243e490SMarcel Moolenaar require ATF to be installed. 490c243e490SMarcel Moolenaar 491c243e490SMarcel Moolenaar* Fixed sign comparison mismatches triggered by the now-enabled 492c243e490SMarcel Moolenaar -Wsign-compare. 493c243e490SMarcel Moolenaar 494c243e490SMarcel Moolenaar* Fixed many memory and object leaks. 495c243e490SMarcel Moolenaar 496c243e490SMarcel Moolenaar 497c243e490SMarcel MoolenaarChanges in version 0.6 498c243e490SMarcel Moolenaar********************** 499c243e490SMarcel Moolenaar 500c243e490SMarcel MoolenaarExperimental version released on January 18th, 2009. 501c243e490SMarcel Moolenaar 502c243e490SMarcel Moolenaar* Make atf-exec be able to kill its child process after a certain period 503c243e490SMarcel Moolenaar of time; this is controlled through the new -t option. 504c243e490SMarcel Moolenaar 505c243e490SMarcel Moolenaar* Change atf-sh to use atf-exec's -t option to control the test case's 506c243e490SMarcel Moolenaar timeouts, instead of doing it internally. Same behavior as before, but 507c243e490SMarcel Moolenaar noticeably faster. 508c243e490SMarcel Moolenaar 509c243e490SMarcel Moolenaar* atf-exec's -g option and atf-killpg are gone due to the previous 510c243e490SMarcel Moolenaar change. 511c243e490SMarcel Moolenaar 512c243e490SMarcel Moolenaar* Added the atf-check(1) tool, a program that executes a given command 513c243e490SMarcel Moolenaar and checks its exit code against a known value and allows the management 514c243e490SMarcel Moolenaar of stdout and stderr in multiple ways. This replaces the previous 515c243e490SMarcel Moolenaar atf_check function in the atf-sh library and exposes this functionality 516c243e490SMarcel Moolenaar to both atf-c and atf-c++. 517c243e490SMarcel Moolenaar 518c243e490SMarcel Moolenaar* Added the ATF_REQUIRE family of macros to the C interface. These help 519c243e490SMarcel Moolenaar in checking for fatal test conditions. The old ATF_CHECK macros now 520c243e490SMarcel Moolenaar perform non-fatal checks only. I.e. by using ATF_CHECK, the test case 521c243e490SMarcel Moolenaar can now continue its execution and the failures will not be reported 522c243e490SMarcel Moolenaar until the end of the whole run. 523c243e490SMarcel Moolenaar 524c243e490SMarcel Moolenaar* Extended the amount of ATF_CHECK_* C macros with new ones to provide 525c243e490SMarcel Moolenaar more features to the developer. These also have their corresponding 526c243e490SMarcel Moolenaar counterparts in the ATF_REQUIRE_* family. The new macros (listing the 527c243e490SMarcel Moolenaar suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and 528c243e490SMarcel Moolenaar _STREQ_MSG. 529c243e490SMarcel Moolenaar 530c243e490SMarcel Moolenaar 531c243e490SMarcel MoolenaarChanges in version 0.5 532c243e490SMarcel Moolenaar********************** 533c243e490SMarcel Moolenaar 534c243e490SMarcel MoolenaarExperimental version released on May 1st, 2008. 535c243e490SMarcel Moolenaar 536c243e490SMarcel Moolenaar* Clauses 3 and 4 of the BSD license used by the project were dropped. 537c243e490SMarcel Moolenaar All the code is now under a 2-clause BSD license compatible with the GNU 538c243e490SMarcel Moolenaar General Public License (GPL). 539c243e490SMarcel Moolenaar 540c243e490SMarcel Moolenaar* Added a C-only binding so that binary test programs do not need to be 541c243e490SMarcel Moolenaar tied to C++ at all. This binding is now known as the atf-c library. 542c243e490SMarcel Moolenaar 543c243e490SMarcel Moolenaar* Renamed the C++ binding to atf-c++ for consistency with the new atf-c. 544c243e490SMarcel Moolenaar 545c243e490SMarcel Moolenaar* Renamed the POSIX shell binding to atf-sh for consistency with the new 546c243e490SMarcel Moolenaar atf-c and atf-c++. 547c243e490SMarcel Moolenaar 548c243e490SMarcel Moolenaar* Added a -w flag to test programs through which it is possible to 549c243e490SMarcel Moolenaar specify the work directory to be used. This was possible in prior 550c243e490SMarcel Moolenaar releases by defining the workdir configuration variable (-v workdir=...), 551c243e490SMarcel Moolenaar but was a conceptually incorrect mechanism. 552c243e490SMarcel Moolenaar 553c243e490SMarcel Moolenaar* Test programs now preserve the execution order of test cases when they 554c243e490SMarcel Moolenaar are given in the command line. Even those mentioned more than once are 555c243e490SMarcel Moolenaar executed multiple times to comply with the user's requests. 556c243e490SMarcel Moolenaar 557c243e490SMarcel Moolenaar 558c243e490SMarcel MoolenaarChanges in version 0.4 559c243e490SMarcel Moolenaar********************** 560c243e490SMarcel Moolenaar 561c243e490SMarcel MoolenaarExperimental version released on February 4th, 2008. 562c243e490SMarcel Moolenaar 563c243e490SMarcel Moolenaar* Added two new manual pages, atf-c++-api and atf-sh-api, describing the 564c243e490SMarcel Moolenaar C++ and POSIX shell interfaces used to write test programs. 565c243e490SMarcel Moolenaar 566c243e490SMarcel Moolenaar* Added a pkg-config file, useful to get the flags to build against the 567c243e490SMarcel Moolenaar C++ library or to easily detect the presence of ATF. 568c243e490SMarcel Moolenaar 569c243e490SMarcel Moolenaar* Added a way for test cases to require a specific architecture and/or 570c243e490SMarcel Moolenaar machine type through the new 'require.arch' and 'require.machine' 571c243e490SMarcel Moolenaar meta-data properties, respectively. 572c243e490SMarcel Moolenaar 573c243e490SMarcel Moolenaar* Added the 'timeout' property to test cases, useful to set an 574c243e490SMarcel Moolenaar upper-bound limit for the test's run time and thus prevent global test 575c243e490SMarcel Moolenaar program stalls due to the test case's misbehavior. 576c243e490SMarcel Moolenaar 577c243e490SMarcel Moolenaar* Added the atf-exec(1) internal utility, used to execute a command 578c243e490SMarcel Moolenaar after changing the process group it belongs to. 579c243e490SMarcel Moolenaar 580c243e490SMarcel Moolenaar* Added the atf-killpg(1) internal utility, used to kill process groups. 581c243e490SMarcel Moolenaar 582c243e490SMarcel Moolenaar* Multiple portability fixes. Of special interest, full support for 583c243e490SMarcel Moolenaar SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12 584c243e490SMarcel Moolenaar C++ compiler. 585c243e490SMarcel Moolenaar 586c243e490SMarcel Moolenaar* Fixed a serious bug that prevented atf-run(1) from working at all 587c243e490SMarcel Moolenaar under Fedora 8 x86_64. Due to the nature of the bug, other platforms 588c243e490SMarcel Moolenaar were likely affected too. 589c243e490SMarcel Moolenaar 590c243e490SMarcel Moolenaar 591c243e490SMarcel MoolenaarChanges in version 0.3 592c243e490SMarcel Moolenaar********************** 593c243e490SMarcel Moolenaar 594c243e490SMarcel MoolenaarExperimental version released on November 11th, 2007. 595c243e490SMarcel Moolenaar 596c243e490SMarcel Moolenaar* Added XML output support to atf-report. This is accompanied by a DTD 597c243e490SMarcel Moolenaar for the format's structure and sample XSLT/CSS files to post-process this 598c243e490SMarcel Moolenaar output and convert it to a plain HTML report. 599c243e490SMarcel Moolenaar 600c243e490SMarcel Moolenaar* Changed atf-run to add system information to the report it generates. 601c243e490SMarcel Moolenaar This is currently used by atf-report's XML output only, and is later 602c243e490SMarcel Moolenaar printed in the HTML reports in a nice and useful summary table. The user 603c243e490SMarcel Moolenaar and system administrator are allowed to tune this feature by means of 604c243e490SMarcel Moolenaar hooks. 605c243e490SMarcel Moolenaar 606c243e490SMarcel Moolenaar* Removed the test cases' 'isolated' property. This was intended to 607c243e490SMarcel Moolenaar avoid touching the file system at all when running the related test case, 608c243e490SMarcel Moolenaar but this has not been true for a long while: some control files are 609c243e490SMarcel Moolenaar unconditionally required for several purposes, and we cannot easily get 610c243e490SMarcel Moolenaar rid of them. This way we remove several critical and delicate pieces of 611c243e490SMarcel Moolenaar code. 612c243e490SMarcel Moolenaar 613c243e490SMarcel Moolenaar* Improved atf-report's CSV output format to include information about 614c243e490SMarcel Moolenaar test programs too. 615c243e490SMarcel Moolenaar 616c243e490SMarcel Moolenaar* Fixed the tests that used atf-compile to not require this tool as a 617c243e490SMarcel Moolenaar helper. Avoids systems without build-time utilities to skip many tests 618c243e490SMarcel Moolenaar that could otherwise be run. (E.g. NetBSD without the comp.tgz set 619c243e490SMarcel Moolenaar installed.) 620c243e490SMarcel Moolenaar 621c243e490SMarcel Moolenaar* Many general cleanups: Fixed many pieces of code marked as ugly and/or 622c243e490SMarcel Moolenaar incomplete. 623c243e490SMarcel Moolenaar 624c243e490SMarcel Moolenaar 625c243e490SMarcel MoolenaarChanges in version 0.2 626c243e490SMarcel Moolenaar********************** 627c243e490SMarcel Moolenaar 628c243e490SMarcel MoolenaarExperimental version released on September 20th, 2007. 629c243e490SMarcel Moolenaar 630c243e490SMarcel Moolenaar* Test cases now get a known umask on entry. 631c243e490SMarcel Moolenaar 632c243e490SMarcel Moolenaar* atf-run now detects many unexpected failures caused by test programs and 633c243e490SMarcel Moolenaar reports them as bogus tests. atf-report is able to handle these new 634c243e490SMarcel Moolenaar errors and nicely reports them to the user. 635c243e490SMarcel Moolenaar 636c243e490SMarcel Moolenaar* All the data formats read and written by the tools have been 637c243e490SMarcel Moolenaar documented and cleaned up. These include those grammars that define how 638c243e490SMarcel Moolenaar the different components communicate with each other as well as the 639c243e490SMarcel Moolenaar format of files written by the developers and users: the Atffiles and the 640c243e490SMarcel Moolenaar configuration files. 641c243e490SMarcel Moolenaar 642c243e490SMarcel Moolenaar* Added the atf-version tool, a utility that displays information about 643c243e490SMarcel Moolenaar the currently installed version of ATF. 644c243e490SMarcel Moolenaar 645c243e490SMarcel Moolenaar* Test cases can now define an optional cleanup routine to undo their 646c243e490SMarcel Moolenaar actions regardless of their exit status. 647c243e490SMarcel Moolenaar 648c243e490SMarcel Moolenaar* atf-report now summarizes the list of failed (bogus) test programs 649c243e490SMarcel Moolenaar when using the ticker output format. 650c243e490SMarcel Moolenaar 651c243e490SMarcel Moolenaar* Test programs now capture some termination signals and clean up any 652c243e490SMarcel Moolenaar temporary files before exiting the program. 653c243e490SMarcel Moolenaar 654c243e490SMarcel Moolenaar* Multiple bug fixes and improvements all around. 655c243e490SMarcel Moolenaar 656c243e490SMarcel Moolenaar 657c243e490SMarcel MoolenaarChanges in version 0.1 658c243e490SMarcel Moolenaar********************** 659c243e490SMarcel Moolenaar 660c243e490SMarcel MoolenaarExperimental version released on August 20th, 2007. 661c243e490SMarcel Moolenaar 662c243e490SMarcel Moolenaar* First public version. This was released coinciding with the end of the 663c243e490SMarcel Moolenaar Google Summer of Code 2007 program. 664c243e490SMarcel Moolenaar 665c243e490SMarcel Moolenaar 666c243e490SMarcel Moolenaar=========================================================================== 667c243e490SMarcel Moolenaarvim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2 668