1c243e490SMarcel MoolenaarMajor changes between releases Automated Testing Framework 2c243e490SMarcel Moolenaar=========================================================================== 3c243e490SMarcel Moolenaar 4c243e490SMarcel Moolenaar 5a18eacbeSJulio MerinoChanges in version 0.18 6a18eacbeSJulio Merino*********************** 7a18eacbeSJulio Merino 8a18eacbeSJulio MerinoExperimental version released on November 16th, 2013. 9a18eacbeSJulio Merino 10a18eacbeSJulio Merino* Issue 45: Added require.memory support in atf-run for FreeBSD. 11a18eacbeSJulio Merino 12a18eacbeSJulio Merino* Fixed an issue with the handling of cin with libc++. 13a18eacbeSJulio Merino 14a18eacbeSJulio Merino* Issue 64: Fixed various mandoc formatting warnings. 15a18eacbeSJulio Merino 16a18eacbeSJulio Merino* NetBSD PR bin/48284: Made atf-check flush its progress message to 17a18eacbeSJulio Merino stdout so that an interrupted test case always shows the last message 18a18eacbeSJulio Merino being executed. 19a18eacbeSJulio Merino 20a18eacbeSJulio Merino* NetBSD PR bin/48285: Fixed atf_check examples in atf-sh-api(3). 21a18eacbeSJulio Merino 22a18eacbeSJulio Merino 23a18eacbeSJulio MerinoChanges in version 0.17 24a18eacbeSJulio Merino*********************** 25a18eacbeSJulio Merino 26a18eacbeSJulio MerinoExperimental version released on February 14th, 2013. 27a18eacbeSJulio Merino 28a18eacbeSJulio Merino* Added the atf_utils_cat_file, atf_utils_compare_file, 29a18eacbeSJulio Merino atf_utils_copy_file, atf_utils_create_file, atf_utils_file_exists, 30a18eacbeSJulio Merino atf_utils_fork, atf_utils_grep_file, atf_utils_grep_string, 31a18eacbeSJulio Merino atf_utils_readline, atf_utils_redirect and atf_utils_wait utility 32a18eacbeSJulio Merino functions to atf-c-api. Documented the already-public 33a18eacbeSJulio Merino atf_utils_free_charpp function. 34a18eacbeSJulio Merino 35a18eacbeSJulio Merino* Added the cat_file, compare_file, copy_file, create_file, file_exists, 36a18eacbeSJulio Merino fork, grep_collection, grep_file, grep_string, redirect and wait 37a18eacbeSJulio Merino functions to the atf::utils namespace of atf-c++-api. These are 38a18eacbeSJulio Merino wrappers around the same functions added to the atf-c-api library. 39a18eacbeSJulio Merino 40a18eacbeSJulio Merino* Added the ATF_CHECK_MATCH, ATF_CHECK_MATCH_MSG, ATF_REQUIRE_MATCH and 41a18eacbeSJulio Merino ATF_REQUIRE_MATCH_MSG macros to atf-c to simplify the validation of a 42a18eacbeSJulio Merino string against a regular expression. 43a18eacbeSJulio Merino 44a18eacbeSJulio Merino* Miscellaneous fixes for manpage typos and compilation problems with 45a18eacbeSJulio Merino clang. 46a18eacbeSJulio Merino 47a18eacbeSJulio Merino* Added caching of the results of those configure tests that rely on 48a18eacbeSJulio Merino executing a test program. This should help crossbuild systems by 49a18eacbeSJulio Merino providing a mechanism to pre-specify what the results should be. 50a18eacbeSJulio Merino 51a18eacbeSJulio Merino* PR bin/45690: Make atf-report convert any non-printable characters to 52a18eacbeSJulio Merino a plain-text representation (matching their corresponding hexadecimal 53a18eacbeSJulio Merino entities) in XML output files. This is to prevent the output of test 54a18eacbeSJulio Merino cases from breaking xsltproc later. 55a18eacbeSJulio Merino 56a18eacbeSJulio Merino 57c243e490SMarcel MoolenaarChanges in version 0.16 58c243e490SMarcel Moolenaar*********************** 59c243e490SMarcel Moolenaar 60c243e490SMarcel MoolenaarExperimental version released on July 10th, 2012. 61c243e490SMarcel Moolenaar 62c243e490SMarcel Moolenaar* Added a --enable-tools flag to configure to request the build of the 63c243e490SMarcel Moolenaar deprecated ATF tools, whose build is now disabled by default. In order 64c243e490SMarcel Moolenaar to continue running tests, you should migrate to Kyua instead of enabling 65c243e490SMarcel Moolenaar the build of the deprecated tools. The kyua-atf-compat package provides 66c243e490SMarcel Moolenaar transitional compatibility versions of atf-run and atf-report built on 67c243e490SMarcel Moolenaar top of Kyua. 68c243e490SMarcel Moolenaar 69c243e490SMarcel Moolenaar* Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can 70c243e490SMarcel Moolenaar detect defined but unused test cases. 71c243e490SMarcel Moolenaar 72c243e490SMarcel Moolenaar* PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and 73c243e490SMarcel Moolenaar tp-time XML tags leaking into the generated HTML file. Also improved 74c243e490SMarcel Moolenaar the CSS file slightly to correct alignment and color issues with the 75c243e490SMarcel Moolenaar timestamps column. 76c243e490SMarcel Moolenaar 77c243e490SMarcel Moolenaar* Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during 78c243e490SMarcel Moolenaar compilation with GNU G++. 79c243e490SMarcel Moolenaar 80c243e490SMarcel Moolenaar* Flipped the default to building shared libraries for atf-c and atf-c++, 81c243e490SMarcel Moolenaar and started versioning them. As a side-effect, this removes the 82c243e490SMarcel Moolenaar --enable-unstable-shared flag from configure that appears to not work any 83c243e490SMarcel Moolenaar more (under NetBSD). Additionally, some distributions require the use of 84c243e490SMarcel Moolenaar shared libraries for proper dependency tracking (e.g. Fedora), so it is 85c243e490SMarcel Moolenaar better if we do the right versioning upstream. 86c243e490SMarcel Moolenaar 87c243e490SMarcel Moolenaar* Project hosting moved from an adhoc solution (custom web site and 88c243e490SMarcel Moolenaar Monotone repository) to Google Code (standard wiki and Git). ATF now 89c243e490SMarcel Moolenaar lives in a subcomponent of the Kyua project. 90c243e490SMarcel Moolenaar 91c243e490SMarcel Moolenaar 92c243e490SMarcel MoolenaarChanges in version 0.15 93c243e490SMarcel Moolenaar*********************** 94c243e490SMarcel Moolenaar 95c243e490SMarcel MoolenaarExperimental version released on January 16th, 2012. 96c243e490SMarcel Moolenaar 97c243e490SMarcel Moolenaar* Respect stdin in atf-check. The previous release silenced stdin for any 98c243e490SMarcel Moolenaar processes spawned by atf, not only test programs, which caused breakage 99c243e490SMarcel Moolenaar in tests that pipe data through atf-check. 100c243e490SMarcel Moolenaar 101c243e490SMarcel Moolenaar* Performance improvements to atf-sh. 102c243e490SMarcel Moolenaar 103c243e490SMarcel Moolenaar* Enabled detection of unused parameters and variables in the code and 104c243e490SMarcel Moolenaar fixed all warnings. 105c243e490SMarcel Moolenaar 106c243e490SMarcel Moolenaar* Changed the behavior of "developer mode". Compiler warnings are now 107c243e490SMarcel Moolenaar enabled unconditionally regardless of whether we are in developer mode or 108c243e490SMarcel Moolenaar not; developer mode is now only used to perform strict warning checks and 109c243e490SMarcel Moolenaar to enable assertions. Additionally, developer mode is now only 110c243e490SMarcel Moolenaar automatically enabled when building from the repository, not for formal 111c243e490SMarcel Moolenaar releases. 112c243e490SMarcel Moolenaar 113c243e490SMarcel Moolenaar* Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and 114c243e490SMarcel Moolenaar ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag 115c243e490SMarcel Moolenaar in configure scripts and detecting the presence of any of the ATF 116c243e490SMarcel Moolenaar bindings. Note that ATF_CHECK_SH was already introduced in 0.14, but it 117c243e490SMarcel Moolenaar has now been modified to also honor --with-atf if instantiated. 118c243e490SMarcel Moolenaar 119c243e490SMarcel Moolenaar* Added timing support to atf-run / atf-report. 120c243e490SMarcel Moolenaar 121c243e490SMarcel Moolenaar* Added support for a 'require.memory' property, to specify the minimum 122c243e490SMarcel Moolenaar amount of physical memory needed by the test case to yield valid results. 123c243e490SMarcel Moolenaar 124c243e490SMarcel Moolenaar* PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by 125c243e490SMarcel Moolenaar atf-report so that invalid data in the output of test cases does not 126c243e490SMarcel Moolenaar mangle our report. 127c243e490SMarcel Moolenaar 128c243e490SMarcel Moolenaar 129c243e490SMarcel MoolenaarChanges in version 0.14 130c243e490SMarcel Moolenaar*********************** 131c243e490SMarcel Moolenaar 132c243e490SMarcel MoolenaarExperimental version released on June 14th, 2011. 133c243e490SMarcel Moolenaar 134c243e490SMarcel Moolenaar* Added a pkg-config file for atf-sh and an aclocal file to ease the 135c243e490SMarcel Moolenaar detection of atf-sh from autoconf scripts. 136c243e490SMarcel Moolenaar 137c243e490SMarcel Moolenaar* Made the default test case body defined by atf_sh fail. This is to 138c243e490SMarcel Moolenaar ensure that test cases are properly defined in test programs and helps 139c243e490SMarcel Moolenaar in catching typos in the names of the body functions. 140c243e490SMarcel Moolenaar 141c243e490SMarcel Moolenaar* PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero. 142c243e490SMarcel Moolenaar This provides more consistent results with "normal" execution (in 143c243e490SMarcel Moolenaar particular, when tests are executed detached from a terminal). 144c243e490SMarcel Moolenaar 145c243e490SMarcel Moolenaar* Made atf-run hardcode TZ=UTC for test cases. It used to undefine TZ, but 146c243e490SMarcel Moolenaar that does not take into account that libc determines the current timezone 147c243e490SMarcel Moolenaar from a configuration file. 148c243e490SMarcel Moolenaar 149c243e490SMarcel Moolenaar* All test programs will now print a warning when they are not run through 150c243e490SMarcel Moolenaar atf-run(1) stating that this is unsupported and may deliver incorrect 151c243e490SMarcel Moolenaar results. 152c243e490SMarcel Moolenaar 153c243e490SMarcel Moolenaar* Added support for the 'require.files' test-case property. This allows 154c243e490SMarcel Moolenaar test cases to specify installed files that must be present for the test 155c243e490SMarcel Moolenaar case to run. 156c243e490SMarcel Moolenaar 157c243e490SMarcel Moolenaar 158c243e490SMarcel MoolenaarChanges in version 0.13 159c243e490SMarcel Moolenaar*********************** 160c243e490SMarcel Moolenaar 161c243e490SMarcel MoolenaarExperimental version released on March 31st, 2011. 162c243e490SMarcel Moolenaar 163c243e490SMarcel MoolenaarThis is the first release after the creation of the Kyua project, a more 164c243e490SMarcel Moolenaarmodular and reliable replacement for ATF. From now on, ATF will change to 165c243e490SMarcel Moolenaaraccomodate the transition to this new codebase, but ATF will still continue 166c243e490SMarcel Moolenaarto see development in the short/medium term. Check out the project page at 167c243e490SMarcel Moolenaarhttp://code.google.com/p/kyua/ for more details. 168c243e490SMarcel Moolenaar 169c243e490SMarcel MoolenaarThe changes in this release are: 170c243e490SMarcel Moolenaar 171c243e490SMarcel Moolenaar* Added support to run the tests with the Kyua runtime engine (kyua-cli), a 172c243e490SMarcel Moolenaar new package that aims to replace atf-run and atf-report. The ATF tests 173c243e490SMarcel Moolenaar can be run with the new system by issuing a 'make installcheck-kyua' from 174c243e490SMarcel Moolenaar the top-level directory of the project (assuming the 'kyua' binary is 175c243e490SMarcel Moolenaar available during the configuration stage of ATF). 176c243e490SMarcel Moolenaar 177c243e490SMarcel Moolenaar* atf-run and atf-report are now in maintenance mode (but *not* deprecated 178c243e490SMarcel Moolenaar yet!). Kyua already implements a new, much more reliable runtime engine 179c243e490SMarcel Moolenaar that provides similar features to these tools. That said, it is not 180c243e490SMarcel Moolenaar complete yet so all development efforts should go towards it. 181c243e490SMarcel Moolenaar 182c243e490SMarcel Moolenaar* If GDB is installed, atf-run dumps the stack trace of crashing test 183c243e490SMarcel Moolenaar programs in an attempt to aid debugging. Contributed by Antti Kantee. 184c243e490SMarcel Moolenaar 185c243e490SMarcel Moolenaar* Reverted default timeout change in previous release and reset its value 186c243e490SMarcel Moolenaar to 5 minutes. This was causing several issues, specially when running 187c243e490SMarcel Moolenaar the existing NetBSD test suite in qemu. 188c243e490SMarcel Moolenaar 189c243e490SMarcel Moolenaar* Fixed the 'match' output checker in atf-check to properly validate the 190c243e490SMarcel Moolenaar last line of a file even if it does not have a newline. 191c243e490SMarcel Moolenaar 192c243e490SMarcel Moolenaar* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to 193c243e490SMarcel Moolenaar check for the presence (or lack thereof) of an element in a collection. 194c243e490SMarcel Moolenaar 195c243e490SMarcel Moolenaar* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run 196c243e490SMarcel Moolenaar when the cleanup of a test case triggered asynchronous modifications to 197c243e490SMarcel Moolenaar its work directory (e.g. killing a daemon process that cleans up a pid 198c243e490SMarcel Moolenaar file in the work directory). 199c243e490SMarcel Moolenaar 200c243e490SMarcel Moolenaar* PR bin/44301: Fixed the sample XSLT file to report bogus test programs 201c243e490SMarcel Moolenaar instead of just listing them as having 0 test cases. 202c243e490SMarcel Moolenaar 203c243e490SMarcel Moolenaar 204c243e490SMarcel MoolenaarChanges in version 0.12 205c243e490SMarcel Moolenaar*********************** 206c243e490SMarcel Moolenaar 207c243e490SMarcel MoolenaarExperimental version released on November 7th, 2010. 208c243e490SMarcel Moolenaar 209c243e490SMarcel Moolenaar* Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as 210c243e490SMarcel Moolenaar ATF_REQUIRE_THROW but allows checking for the validity of the exception's 211c243e490SMarcel Moolenaar error message by means of a regular expression. 212c243e490SMarcel Moolenaar 213c243e490SMarcel Moolenaar* Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a 214c243e490SMarcel Moolenaar regular expression match in a string. 215c243e490SMarcel Moolenaar 216c243e490SMarcel Moolenaar* Changed the default timeout for test cases from 5 minutes to 30 seconds. 217c243e490SMarcel Moolenaar 30 seconds is long enough for virtually all tests to complete, and 5 218c243e490SMarcel Moolenaar minutes is a way too long pause in a test suite where a single test case 219c243e490SMarcel Moolenaar stalls. 220c243e490SMarcel Moolenaar 221c243e490SMarcel Moolenaar* Deprecated the use.fs property. While this seemed like a good idea in 222c243e490SMarcel Moolenaar the first place to impose more control on what test cases can do, it 223c243e490SMarcel Moolenaar turns out to be bad. First, use.fs=false prevents bogus test cases 224c243e490SMarcel Moolenaar from dumping core so after-the-fact debugging is harder. Second, 225c243e490SMarcel Moolenaar supporting use.fs adds a lot of unnecessary complexity. atf-run will 226c243e490SMarcel Moolenaar now ignore any value provided to use.fs and will allow test cases to 227c243e490SMarcel Moolenaar freely access the file system if they wish to. 228c243e490SMarcel Moolenaar 229c243e490SMarcel Moolenaar* Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c 230c243e490SMarcel Moolenaar library. The 'text' module became private in 0.11 but was being used 231c243e490SMarcel Moolenaar externally to simplify the parsing of configuration variables. 232c243e490SMarcel Moolenaar 233c243e490SMarcel Moolenaar* Made atf-run recognize the 'unprivileged-user' configuration variable 234c243e490SMarcel Moolenaar and automatically drop root privileges when a test case sets 235c243e490SMarcel Moolenaar require.user=unprivileged. Note that this is, by no means, done for 236c243e490SMarcel Moolenaar security purposes; this is just for user convenience; tests should, in 237c243e490SMarcel Moolenaar general, not be blindly run as root in the first place. 238c243e490SMarcel Moolenaar 239c243e490SMarcel Moolenaar 240c243e490SMarcel MoolenaarChanges in version 0.11 241c243e490SMarcel Moolenaar*********************** 242c243e490SMarcel Moolenaar 243c243e490SMarcel MoolenaarExperimental version released on October 20th, 2010. 244c243e490SMarcel Moolenaar 245c243e490SMarcel Moolenaar* The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match 246c243e490SMarcel Moolenaar their counterparts in atf-c. 247c243e490SMarcel Moolenaar 248c243e490SMarcel Moolenaar* Clearly separated the modules in atf-c that are supposed to be public 249c243e490SMarcel Moolenaar from those that are implementation details. The header files for the 250c243e490SMarcel Moolenaar internal modules are not installed any more. 251c243e490SMarcel Moolenaar 252c243e490SMarcel Moolenaar* Made the atf-check tool private. It is only required by atf-sh and being 253c243e490SMarcel Moolenaar public has the danger of causing confusion. Also, making it private 254c243e490SMarcel Moolenaar simplifies the public API of atf. 255c243e490SMarcel Moolenaar 256c243e490SMarcel Moolenaar* Changed atf-sh to enable per-command error checking (set -e) by default. 257c243e490SMarcel Moolenaar This catches many cases in which a test case is broken but it is not 258c243e490SMarcel Moolenaar reported as such because execution continues. 259c243e490SMarcel Moolenaar 260c243e490SMarcel Moolenaar* Fixed the XSTL and CSS stylesheets to support expected failures. 261c243e490SMarcel Moolenaar 262c243e490SMarcel Moolenaar 263c243e490SMarcel MoolenaarChanges in version 0.10 264c243e490SMarcel Moolenaar*********************** 265c243e490SMarcel Moolenaar 266c243e490SMarcel MoolenaarExperimental version released on July 2nd, 2010. 267c243e490SMarcel Moolenaar 268c243e490SMarcel MoolenaarMiscellaneous features 269c243e490SMarcel Moolenaar 270c243e490SMarcel Moolenaar* Added expected failures support to test cases and atf-run. These 271c243e490SMarcel Moolenaar include, for example, expected clean exits, expected reception of fatal 272c243e490SMarcel Moolenaar signals, expected timeouts and expected errors in condition checks. 273c243e490SMarcel Moolenaar These statuses can be used to denote test cases that are known to fail 274c243e490SMarcel Moolenaar due to a bug in the code they are testing. atf-report reports these 275c243e490SMarcel Moolenaar tests separately but they do not count towards the failed test cases 276c243e490SMarcel Moolenaar amount. 277c243e490SMarcel Moolenaar 278c243e490SMarcel Moolenaar* Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to 279c243e490SMarcel Moolenaar allow easy checking of call failures that update errno. 280c243e490SMarcel Moolenaar 281c243e490SMarcel Moolenaar* Added the has.cleanup meta-data property to test caes that specifies 282c243e490SMarcel Moolenaar whether the test case has a cleanup routine or not; its value is 283c243e490SMarcel Moolenaar automatically set. This property is read by atf-run to know if it has to 284c243e490SMarcel Moolenaar run the cleanup routine; skipping this run for every test case 285c243e490SMarcel Moolenaar significantly speeds up the run time of test suites. 286c243e490SMarcel Moolenaar 287c243e490SMarcel Moolenaar* Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to 288c243e490SMarcel Moolenaar take the expected exception as the first argument and the statement to 289c243e490SMarcel Moolenaar execute as the second argument. 290c243e490SMarcel Moolenaar 291c243e490SMarcel MoolenaarChanges in atf-check 292c243e490SMarcel Moolenaar 293c243e490SMarcel Moolenaar* Changed atf-check to support negating the status and output checks by 294c243e490SMarcel Moolenaar prefixing them with not- and added support to specify multiple checkers 295c243e490SMarcel Moolenaar for stdout and stderr, not only one. 296c243e490SMarcel Moolenaar 297c243e490SMarcel Moolenaar* Added the match output checker to atf-check to look for regular 298c243e490SMarcel Moolenaar expressions in the stdout and stderr of commands. 299c243e490SMarcel Moolenaar 300c243e490SMarcel Moolenaar* Modified the exit checks in atf-check to support checking for the 301c243e490SMarcel Moolenaar reception of signals. 302c243e490SMarcel Moolenaar 303c243e490SMarcel MoolenaarCode simplifications and cleanups 304c243e490SMarcel Moolenaar 305c243e490SMarcel Moolenaar* Removed usage messages from test programs to simplify the 306c243e490SMarcel Moolenaar implementation of every binding by a significant amount. They just now 307c243e490SMarcel Moolenaar refer the user to the appropriate manual page and do not attempt to wrap 308c243e490SMarcel Moolenaar lines on terminal boundaries. Test programs are not supposed to be run 309c243e490SMarcel Moolenaar by users directly so this minor interface regression is not important. 310c243e490SMarcel Moolenaar 311c243e490SMarcel Moolenaar* Removed the atf-format internal utility, which is unused after the 312c243e490SMarcel Moolenaar change documented above. 313c243e490SMarcel Moolenaar 314c243e490SMarcel Moolenaar* Removed the atf-cleanup internal utility. It has been unused since the 315c243e490SMarcel Moolenaar test case isolation was moved to atf-run in 0.8 316c243e490SMarcel Moolenaar 317c243e490SMarcel Moolenaar* Splitted the Makefile.am into smaller files for easier maintenance and 318c243e490SMarcel Moolenaar dropped the use of M4. Only affects users building from the repository 319c243e490SMarcel Moolenaar sources. 320c243e490SMarcel Moolenaar 321c243e490SMarcel Moolenaar* Intermixed tests with the source files in the source tree to provide 322c243e490SMarcel Moolenaar them more visibility and easier access. The tests directory is gone from 323c243e490SMarcel Moolenaar the source tree and tests are now suffixed by _test, not prefixed by t_. 324c243e490SMarcel Moolenaar 325c243e490SMarcel Moolenaar* Simplifications to the atf-c library: removed the io, tcr and ui 326c243e490SMarcel Moolenaar modules as they had become unnecessary after all simplifications 327c243e490SMarcel Moolenaar introduced since the 0.8 release. 328c243e490SMarcel Moolenaar 329c243e490SMarcel Moolenaar* Removed the application/X-atf-tcr format introduced in 0.8 release. 330c243e490SMarcel Moolenaar Tests now print a much simplified format that is easy to parse and nicer 331c243e490SMarcel Moolenaar to read by end users. As a side effect, the default for test cases is 332c243e490SMarcel Moolenaar now to print their results to stdout unless otherwise stated by providing 333c243e490SMarcel Moolenaar the -r flag. 334c243e490SMarcel Moolenaar 335c243e490SMarcel Moolenaar* Removed XML distribution documents and replaced them with plain-text 336c243e490SMarcel Moolenaar documents. They provided little value and introduced a lot of complexity 337c243e490SMarcel Moolenaar to the build system. 338c243e490SMarcel Moolenaar 339c243e490SMarcel Moolenaar* Simplified the output of atf-version by not attempting to print a 340c243e490SMarcel Moolenaar revision number when building form a distfile. Makes the build system 341c243e490SMarcel Moolenaar easier to maintain. 342c243e490SMarcel Moolenaar 343c243e490SMarcel Moolenaar 344c243e490SMarcel MoolenaarChanges in version 0.9 345c243e490SMarcel Moolenaar********************** 346c243e490SMarcel Moolenaar 347c243e490SMarcel MoolenaarExperimental version released on June 3rd, 2010. 348c243e490SMarcel Moolenaar 349c243e490SMarcel Moolenaar* Added atf-sh, an interpreter to process test programs written using 350c243e490SMarcel Moolenaar the shell API. This is not really a shell interpreter by itself though: 351c243e490SMarcel Moolenaar it is just a wrapper around the system shell that eases the loading of 352c243e490SMarcel Moolenaar the necessary ATF libraries. 353c243e490SMarcel Moolenaar 354c243e490SMarcel Moolenaar* Removed atf-compile in favour of atf-sh. 355c243e490SMarcel Moolenaar 356c243e490SMarcel Moolenaar* Added the use.fs metadata property to test case, which is used to 357c243e490SMarcel Moolenaar specify which test cases require file system access. This is to 358c243e490SMarcel Moolenaar highlight dependencies on external resources more clearly and to speed up 359c243e490SMarcel Moolenaar the execution of test suites by skipping the creation of many unnecessary 360c243e490SMarcel Moolenaar work directories. 361c243e490SMarcel Moolenaar 362c243e490SMarcel Moolenaar* Fixed test programs to get a sane default value for their source 363c243e490SMarcel Moolenaar directory. This means that it should not be necessary any more to pass 364c243e490SMarcel Moolenaar -s when running test programs that do not live in the current directory. 365c243e490SMarcel Moolenaar 366c243e490SMarcel Moolenaar* Defining test case headers became optional. This is trivial to achieve 367c243e490SMarcel Moolenaar in shell-based tests but a bit ugly in C and C++. In C, use the new 368c243e490SMarcel Moolenaar ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use 369c243e490SMarcel Moolenaar ATF_TEST_CASE_WITHOUT_HEAD. 370c243e490SMarcel Moolenaar 371c243e490SMarcel Moolenaar 372c243e490SMarcel MoolenaarChanges in version 0.8 373c243e490SMarcel Moolenaar********************** 374c243e490SMarcel Moolenaar 375c243e490SMarcel MoolenaarExperimental version released on May 7th, 2010. 376c243e490SMarcel Moolenaar 377c243e490SMarcel Moolenaar* Test programs no longer run several test cases in a row. The execution 378c243e490SMarcel Moolenaar of a test program now requires a test case name, and that single test 379c243e490SMarcel Moolenaar case is executed. To execute several test cases, use the atf-run utility 380c243e490SMarcel Moolenaar as usual. 381c243e490SMarcel Moolenaar 382c243e490SMarcel Moolenaar* Test programs no longer fork a subprocess to isolate the execution of 383c243e490SMarcel Moolenaar test cases. They run the test case code in-process, and a crash of the 384c243e490SMarcel Moolenaar test case will result in a crash of the test program. This is to ease 385c243e490SMarcel Moolenaar debugging of faulty test cases. 386c243e490SMarcel Moolenaar 387c243e490SMarcel Moolenaar* Test programs no longer isolate their test cases. This means that they 388c243e490SMarcel Moolenaar will not create temporary directories nor sanitize the environment any 389c243e490SMarcel Moolenaar more. Yes: running a test case that depends on system state by hand will 390c243e490SMarcel Moolenaar most likely yield different results depending on where (machine, 391c243e490SMarcel Moolenaar directory, user environment, etc.) it is run. Isolation has been moved 392c243e490SMarcel Moolenaar to atf-run. 393c243e490SMarcel Moolenaar 394c243e490SMarcel Moolenaar* Test programs no longer print a cryptic format (application/X-atf-tcs) 395c243e490SMarcel Moolenaar on a special file channel. They can now print whatever they want on the 396c243e490SMarcel Moolenaar screen. Because test programs can now only run one test case every time, 397c243e490SMarcel Moolenaar providing controlled output is not necessary any more. 398c243e490SMarcel Moolenaar 399c243e490SMarcel Moolenaar* Test programs no longer write their status into a special file 400c243e490SMarcel Moolenaar descriptor. Instead, they create a file with the results, which is later 401c243e490SMarcel Moolenaar parsed by atf-run. This changes the semantics of the -r flag. 402c243e490SMarcel Moolenaar 403c243e490SMarcel Moolenaar* atf-run has been adjusted to perform the test case isolation. As a 404c243e490SMarcel Moolenaar result, there is now a single canonical place that implements the 405c243e490SMarcel Moolenaar isolation of test caes. In previous releases, the three language 406c243e490SMarcel Moolenaar bindings (C, C++ and shell) had to be kept in sync with each other (read: 407c243e490SMarcel Moolenaar not a nice thing to do at all). As a side effect of this change, writing 408c243e490SMarcel Moolenaar bindings for other languages will be much, much easier from now on. 409c243e490SMarcel Moolenaar 410c243e490SMarcel Moolenaar* atf-run forks test programs on a test case basis, instead of on a test 411c243e490SMarcel Moolenaar program basis as it did before. This is to provide the test case 412c243e490SMarcel Moolenaar isolation that was before implemented by the test programs themselves. 413c243e490SMarcel Moolenaar 414c243e490SMarcel Moolenaar* Removed the atf-exec tool. This was used to implement test case 415c243e490SMarcel Moolenaar isolation in atf-sh, but it is now unnecessary. 416c243e490SMarcel Moolenaar 417c243e490SMarcel Moolenaar* It is now optional to define the descr meta-data property. It has been 418c243e490SMarcel Moolenaar proven to be mostly useless, because test cases often carry a descriptive 419c243e490SMarcel Moolenaar name of their own. 420c243e490SMarcel Moolenaar 421c243e490SMarcel Moolenaar 422c243e490SMarcel MoolenaarChanges in version 0.7 423c243e490SMarcel Moolenaar********************** 424c243e490SMarcel Moolenaar 425c243e490SMarcel MoolenaarExperimental version released on December 22nd, 2009. 426c243e490SMarcel Moolenaar 427c243e490SMarcel Moolenaar* Added build-time checks to atf-c and atf-c++. A binding for atf-sh 428c243e490SMarcel Moolenaar will come later. 429c243e490SMarcel Moolenaar 430c243e490SMarcel Moolenaar* Migrated all build-time checks for header files to proper ATF tests. 431c243e490SMarcel Moolenaar This demonstrates the use of the new feature described above. 432c243e490SMarcel Moolenaar 433c243e490SMarcel Moolenaar* Added an internal API for child process management. 434c243e490SMarcel Moolenaar 435c243e490SMarcel Moolenaar* Converted all plain-text distribution documents to a Docbook canonical 436c243e490SMarcel Moolenaar version, and include pre-generated plain text and HTML copies in the 437c243e490SMarcel Moolenaar distribution file. 438c243e490SMarcel Moolenaar 439c243e490SMarcel Moolenaar* Simplified the contents of the Makefile.am by regenerating it from a 440c243e490SMarcel Moolenaar canonical Makefile.am.m4 source. As a side-effect, some dependency 441c243e490SMarcel Moolenaar specifications were fixed. 442c243e490SMarcel Moolenaar 443c243e490SMarcel Moolenaar* Migrated all checks from the check target to installcheck, as these 444c243e490SMarcel Moolenaar require ATF to be installed. 445c243e490SMarcel Moolenaar 446c243e490SMarcel Moolenaar* Fixed sign comparison mismatches triggered by the now-enabled 447c243e490SMarcel Moolenaar -Wsign-compare. 448c243e490SMarcel Moolenaar 449c243e490SMarcel Moolenaar* Fixed many memory and object leaks. 450c243e490SMarcel Moolenaar 451c243e490SMarcel Moolenaar 452c243e490SMarcel MoolenaarChanges in version 0.6 453c243e490SMarcel Moolenaar********************** 454c243e490SMarcel Moolenaar 455c243e490SMarcel MoolenaarExperimental version released on January 18th, 2009. 456c243e490SMarcel Moolenaar 457c243e490SMarcel Moolenaar* Make atf-exec be able to kill its child process after a certain period 458c243e490SMarcel Moolenaar of time; this is controlled through the new -t option. 459c243e490SMarcel Moolenaar 460c243e490SMarcel Moolenaar* Change atf-sh to use atf-exec's -t option to control the test case's 461c243e490SMarcel Moolenaar timeouts, instead of doing it internally. Same behavior as before, but 462c243e490SMarcel Moolenaar noticeably faster. 463c243e490SMarcel Moolenaar 464c243e490SMarcel Moolenaar* atf-exec's -g option and atf-killpg are gone due to the previous 465c243e490SMarcel Moolenaar change. 466c243e490SMarcel Moolenaar 467c243e490SMarcel Moolenaar* Added the atf-check(1) tool, a program that executes a given command 468c243e490SMarcel Moolenaar and checks its exit code against a known value and allows the management 469c243e490SMarcel Moolenaar of stdout and stderr in multiple ways. This replaces the previous 470c243e490SMarcel Moolenaar atf_check function in the atf-sh library and exposes this functionality 471c243e490SMarcel Moolenaar to both atf-c and atf-c++. 472c243e490SMarcel Moolenaar 473c243e490SMarcel Moolenaar* Added the ATF_REQUIRE family of macros to the C interface. These help 474c243e490SMarcel Moolenaar in checking for fatal test conditions. The old ATF_CHECK macros now 475c243e490SMarcel Moolenaar perform non-fatal checks only. I.e. by using ATF_CHECK, the test case 476c243e490SMarcel Moolenaar can now continue its execution and the failures will not be reported 477c243e490SMarcel Moolenaar until the end of the whole run. 478c243e490SMarcel Moolenaar 479c243e490SMarcel Moolenaar* Extended the amount of ATF_CHECK_* C macros with new ones to provide 480c243e490SMarcel Moolenaar more features to the developer. These also have their corresponding 481c243e490SMarcel Moolenaar counterparts in the ATF_REQUIRE_* family. The new macros (listing the 482c243e490SMarcel Moolenaar suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and 483c243e490SMarcel Moolenaar _STREQ_MSG. 484c243e490SMarcel Moolenaar 485c243e490SMarcel Moolenaar 486c243e490SMarcel MoolenaarChanges in version 0.5 487c243e490SMarcel Moolenaar********************** 488c243e490SMarcel Moolenaar 489c243e490SMarcel MoolenaarExperimental version released on May 1st, 2008. 490c243e490SMarcel Moolenaar 491c243e490SMarcel Moolenaar* Clauses 3 and 4 of the BSD license used by the project were dropped. 492c243e490SMarcel Moolenaar All the code is now under a 2-clause BSD license compatible with the GNU 493c243e490SMarcel Moolenaar General Public License (GPL). 494c243e490SMarcel Moolenaar 495c243e490SMarcel Moolenaar* Added a C-only binding so that binary test programs do not need to be 496c243e490SMarcel Moolenaar tied to C++ at all. This binding is now known as the atf-c library. 497c243e490SMarcel Moolenaar 498c243e490SMarcel Moolenaar* Renamed the C++ binding to atf-c++ for consistency with the new atf-c. 499c243e490SMarcel Moolenaar 500c243e490SMarcel Moolenaar* Renamed the POSIX shell binding to atf-sh for consistency with the new 501c243e490SMarcel Moolenaar atf-c and atf-c++. 502c243e490SMarcel Moolenaar 503c243e490SMarcel Moolenaar* Added a -w flag to test programs through which it is possible to 504c243e490SMarcel Moolenaar specify the work directory to be used. This was possible in prior 505c243e490SMarcel Moolenaar releases by defining the workdir configuration variable (-v workdir=...), 506c243e490SMarcel Moolenaar but was a conceptually incorrect mechanism. 507c243e490SMarcel Moolenaar 508c243e490SMarcel Moolenaar* Test programs now preserve the execution order of test cases when they 509c243e490SMarcel Moolenaar are given in the command line. Even those mentioned more than once are 510c243e490SMarcel Moolenaar executed multiple times to comply with the user's requests. 511c243e490SMarcel Moolenaar 512c243e490SMarcel Moolenaar 513c243e490SMarcel MoolenaarChanges in version 0.4 514c243e490SMarcel Moolenaar********************** 515c243e490SMarcel Moolenaar 516c243e490SMarcel MoolenaarExperimental version released on February 4th, 2008. 517c243e490SMarcel Moolenaar 518c243e490SMarcel Moolenaar* Added two new manual pages, atf-c++-api and atf-sh-api, describing the 519c243e490SMarcel Moolenaar C++ and POSIX shell interfaces used to write test programs. 520c243e490SMarcel Moolenaar 521c243e490SMarcel Moolenaar* Added a pkg-config file, useful to get the flags to build against the 522c243e490SMarcel Moolenaar C++ library or to easily detect the presence of ATF. 523c243e490SMarcel Moolenaar 524c243e490SMarcel Moolenaar* Added a way for test cases to require a specific architecture and/or 525c243e490SMarcel Moolenaar machine type through the new 'require.arch' and 'require.machine' 526c243e490SMarcel Moolenaar meta-data properties, respectively. 527c243e490SMarcel Moolenaar 528c243e490SMarcel Moolenaar* Added the 'timeout' property to test cases, useful to set an 529c243e490SMarcel Moolenaar upper-bound limit for the test's run time and thus prevent global test 530c243e490SMarcel Moolenaar program stalls due to the test case's misbehavior. 531c243e490SMarcel Moolenaar 532c243e490SMarcel Moolenaar* Added the atf-exec(1) internal utility, used to execute a command 533c243e490SMarcel Moolenaar after changing the process group it belongs to. 534c243e490SMarcel Moolenaar 535c243e490SMarcel Moolenaar* Added the atf-killpg(1) internal utility, used to kill process groups. 536c243e490SMarcel Moolenaar 537c243e490SMarcel Moolenaar* Multiple portability fixes. Of special interest, full support for 538c243e490SMarcel Moolenaar SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12 539c243e490SMarcel Moolenaar C++ compiler. 540c243e490SMarcel Moolenaar 541c243e490SMarcel Moolenaar* Fixed a serious bug that prevented atf-run(1) from working at all 542c243e490SMarcel Moolenaar under Fedora 8 x86_64. Due to the nature of the bug, other platforms 543c243e490SMarcel Moolenaar were likely affected too. 544c243e490SMarcel Moolenaar 545c243e490SMarcel Moolenaar 546c243e490SMarcel MoolenaarChanges in version 0.3 547c243e490SMarcel Moolenaar********************** 548c243e490SMarcel Moolenaar 549c243e490SMarcel MoolenaarExperimental version released on November 11th, 2007. 550c243e490SMarcel Moolenaar 551c243e490SMarcel Moolenaar* Added XML output support to atf-report. This is accompanied by a DTD 552c243e490SMarcel Moolenaar for the format's structure and sample XSLT/CSS files to post-process this 553c243e490SMarcel Moolenaar output and convert it to a plain HTML report. 554c243e490SMarcel Moolenaar 555c243e490SMarcel Moolenaar* Changed atf-run to add system information to the report it generates. 556c243e490SMarcel Moolenaar This is currently used by atf-report's XML output only, and is later 557c243e490SMarcel Moolenaar printed in the HTML reports in a nice and useful summary table. The user 558c243e490SMarcel Moolenaar and system administrator are allowed to tune this feature by means of 559c243e490SMarcel Moolenaar hooks. 560c243e490SMarcel Moolenaar 561c243e490SMarcel Moolenaar* Removed the test cases' 'isolated' property. This was intended to 562c243e490SMarcel Moolenaar avoid touching the file system at all when running the related test case, 563c243e490SMarcel Moolenaar but this has not been true for a long while: some control files are 564c243e490SMarcel Moolenaar unconditionally required for several purposes, and we cannot easily get 565c243e490SMarcel Moolenaar rid of them. This way we remove several critical and delicate pieces of 566c243e490SMarcel Moolenaar code. 567c243e490SMarcel Moolenaar 568c243e490SMarcel Moolenaar* Improved atf-report's CSV output format to include information about 569c243e490SMarcel Moolenaar test programs too. 570c243e490SMarcel Moolenaar 571c243e490SMarcel Moolenaar* Fixed the tests that used atf-compile to not require this tool as a 572c243e490SMarcel Moolenaar helper. Avoids systems without build-time utilities to skip many tests 573c243e490SMarcel Moolenaar that could otherwise be run. (E.g. NetBSD without the comp.tgz set 574c243e490SMarcel Moolenaar installed.) 575c243e490SMarcel Moolenaar 576c243e490SMarcel Moolenaar* Many general cleanups: Fixed many pieces of code marked as ugly and/or 577c243e490SMarcel Moolenaar incomplete. 578c243e490SMarcel Moolenaar 579c243e490SMarcel Moolenaar 580c243e490SMarcel MoolenaarChanges in version 0.2 581c243e490SMarcel Moolenaar********************** 582c243e490SMarcel Moolenaar 583c243e490SMarcel MoolenaarExperimental version released on September 20th, 2007. 584c243e490SMarcel Moolenaar 585c243e490SMarcel Moolenaar* Test cases now get a known umask on entry. 586c243e490SMarcel Moolenaar 587c243e490SMarcel Moolenaar* atf-run now detects many unexpected failures caused by test programs and 588c243e490SMarcel Moolenaar reports them as bogus tests. atf-report is able to handle these new 589c243e490SMarcel Moolenaar errors and nicely reports them to the user. 590c243e490SMarcel Moolenaar 591c243e490SMarcel Moolenaar* All the data formats read and written by the tools have been 592c243e490SMarcel Moolenaar documented and cleaned up. These include those grammars that define how 593c243e490SMarcel Moolenaar the different components communicate with each other as well as the 594c243e490SMarcel Moolenaar format of files written by the developers and users: the Atffiles and the 595c243e490SMarcel Moolenaar configuration files. 596c243e490SMarcel Moolenaar 597c243e490SMarcel Moolenaar* Added the atf-version tool, a utility that displays information about 598c243e490SMarcel Moolenaar the currently installed version of ATF. 599c243e490SMarcel Moolenaar 600c243e490SMarcel Moolenaar* Test cases can now define an optional cleanup routine to undo their 601c243e490SMarcel Moolenaar actions regardless of their exit status. 602c243e490SMarcel Moolenaar 603c243e490SMarcel Moolenaar* atf-report now summarizes the list of failed (bogus) test programs 604c243e490SMarcel Moolenaar when using the ticker output format. 605c243e490SMarcel Moolenaar 606c243e490SMarcel Moolenaar* Test programs now capture some termination signals and clean up any 607c243e490SMarcel Moolenaar temporary files before exiting the program. 608c243e490SMarcel Moolenaar 609c243e490SMarcel Moolenaar* Multiple bug fixes and improvements all around. 610c243e490SMarcel Moolenaar 611c243e490SMarcel Moolenaar 612c243e490SMarcel MoolenaarChanges in version 0.1 613c243e490SMarcel Moolenaar********************** 614c243e490SMarcel Moolenaar 615c243e490SMarcel MoolenaarExperimental version released on August 20th, 2007. 616c243e490SMarcel Moolenaar 617c243e490SMarcel Moolenaar* First public version. This was released coinciding with the end of the 618c243e490SMarcel Moolenaar Google Summer of Code 2007 program. 619c243e490SMarcel Moolenaar 620c243e490SMarcel Moolenaar 621c243e490SMarcel Moolenaar=========================================================================== 622c243e490SMarcel Moolenaarvim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2 623