1c243e490SMarcel MoolenaarMajor changes between releases Automated Testing Framework 2c243e490SMarcel Moolenaar=========================================================================== 3c243e490SMarcel Moolenaar 4c243e490SMarcel Moolenaar 5c243e490SMarcel MoolenaarChanges in version 0.16 6c243e490SMarcel Moolenaar*********************** 7c243e490SMarcel Moolenaar 8c243e490SMarcel MoolenaarExperimental version released on July 10th, 2012. 9c243e490SMarcel Moolenaar 10c243e490SMarcel Moolenaar* Added a --enable-tools flag to configure to request the build of the 11c243e490SMarcel Moolenaar deprecated ATF tools, whose build is now disabled by default. In order 12c243e490SMarcel Moolenaar to continue running tests, you should migrate to Kyua instead of enabling 13c243e490SMarcel Moolenaar the build of the deprecated tools. The kyua-atf-compat package provides 14c243e490SMarcel Moolenaar transitional compatibility versions of atf-run and atf-report built on 15c243e490SMarcel Moolenaar top of Kyua. 16c243e490SMarcel Moolenaar 17c243e490SMarcel Moolenaar* Tweaked the ATF_TEST_CASE macro of atf-c++ so that the compiler can 18c243e490SMarcel Moolenaar detect defined but unused test cases. 19c243e490SMarcel Moolenaar 20c243e490SMarcel Moolenaar* PR bin/45859: Fixed some XSLT bugs that resulted in the tc-time and 21c243e490SMarcel Moolenaar tp-time XML tags leaking into the generated HTML file. Also improved 22c243e490SMarcel Moolenaar the CSS file slightly to correct alignment and color issues with the 23c243e490SMarcel Moolenaar timestamps column. 24c243e490SMarcel Moolenaar 25c243e490SMarcel Moolenaar* Optimized atf-c++/macros.hpp so that GNU G++ consumes less memory during 26c243e490SMarcel Moolenaar compilation with GNU G++. 27c243e490SMarcel Moolenaar 28c243e490SMarcel Moolenaar* Flipped the default to building shared libraries for atf-c and atf-c++, 29c243e490SMarcel Moolenaar and started versioning them. As a side-effect, this removes the 30c243e490SMarcel Moolenaar --enable-unstable-shared flag from configure that appears to not work any 31c243e490SMarcel Moolenaar more (under NetBSD). Additionally, some distributions require the use of 32c243e490SMarcel Moolenaar shared libraries for proper dependency tracking (e.g. Fedora), so it is 33c243e490SMarcel Moolenaar better if we do the right versioning upstream. 34c243e490SMarcel Moolenaar 35c243e490SMarcel Moolenaar* Project hosting moved from an adhoc solution (custom web site and 36c243e490SMarcel Moolenaar Monotone repository) to Google Code (standard wiki and Git). ATF now 37c243e490SMarcel Moolenaar lives in a subcomponent of the Kyua project. 38c243e490SMarcel Moolenaar 39c243e490SMarcel Moolenaar 40c243e490SMarcel MoolenaarChanges in version 0.15 41c243e490SMarcel Moolenaar*********************** 42c243e490SMarcel Moolenaar 43c243e490SMarcel MoolenaarExperimental version released on January 16th, 2012. 44c243e490SMarcel Moolenaar 45c243e490SMarcel Moolenaar* Respect stdin in atf-check. The previous release silenced stdin for any 46c243e490SMarcel Moolenaar processes spawned by atf, not only test programs, which caused breakage 47c243e490SMarcel Moolenaar in tests that pipe data through atf-check. 48c243e490SMarcel Moolenaar 49c243e490SMarcel Moolenaar* Performance improvements to atf-sh. 50c243e490SMarcel Moolenaar 51c243e490SMarcel Moolenaar* Enabled detection of unused parameters and variables in the code and 52c243e490SMarcel Moolenaar fixed all warnings. 53c243e490SMarcel Moolenaar 54c243e490SMarcel Moolenaar* Changed the behavior of "developer mode". Compiler warnings are now 55c243e490SMarcel Moolenaar enabled unconditionally regardless of whether we are in developer mode or 56c243e490SMarcel Moolenaar not; developer mode is now only used to perform strict warning checks and 57c243e490SMarcel Moolenaar to enable assertions. Additionally, developer mode is now only 58c243e490SMarcel Moolenaar automatically enabled when building from the repository, not for formal 59c243e490SMarcel Moolenaar releases. 60c243e490SMarcel Moolenaar 61c243e490SMarcel Moolenaar* Added new Autoconf M4 macros (ATF_ARG_WITH, ATF_CHECK_C and 62c243e490SMarcel Moolenaar ATF_CHECK_CXX) to provide a consistent way of defining a --with-arg flag 63c243e490SMarcel Moolenaar in configure scripts and detecting the presence of any of the ATF 64c243e490SMarcel Moolenaar bindings. Note that ATF_CHECK_SH was already introduced in 0.14, but it 65c243e490SMarcel Moolenaar has now been modified to also honor --with-atf if instantiated. 66c243e490SMarcel Moolenaar 67c243e490SMarcel Moolenaar* Added timing support to atf-run / atf-report. 68c243e490SMarcel Moolenaar 69c243e490SMarcel Moolenaar* Added support for a 'require.memory' property, to specify the minimum 70c243e490SMarcel Moolenaar amount of physical memory needed by the test case to yield valid results. 71c243e490SMarcel Moolenaar 72c243e490SMarcel Moolenaar* PR bin/45690: Force an ISO-8859-1 encoding in the XML files generated by 73c243e490SMarcel Moolenaar atf-report so that invalid data in the output of test cases does not 74c243e490SMarcel Moolenaar mangle our report. 75c243e490SMarcel Moolenaar 76c243e490SMarcel Moolenaar 77c243e490SMarcel MoolenaarChanges in version 0.14 78c243e490SMarcel Moolenaar*********************** 79c243e490SMarcel Moolenaar 80c243e490SMarcel MoolenaarExperimental version released on June 14th, 2011. 81c243e490SMarcel Moolenaar 82c243e490SMarcel Moolenaar* Added a pkg-config file for atf-sh and an aclocal file to ease the 83c243e490SMarcel Moolenaar detection of atf-sh from autoconf scripts. 84c243e490SMarcel Moolenaar 85c243e490SMarcel Moolenaar* Made the default test case body defined by atf_sh fail. This is to 86c243e490SMarcel Moolenaar ensure that test cases are properly defined in test programs and helps 87c243e490SMarcel Moolenaar in catching typos in the names of the body functions. 88c243e490SMarcel Moolenaar 89c243e490SMarcel Moolenaar* PR bin/44882: Made atf-run connect the stdin of test cases to /dev/zero. 90c243e490SMarcel Moolenaar This provides more consistent results with "normal" execution (in 91c243e490SMarcel Moolenaar particular, when tests are executed detached from a terminal). 92c243e490SMarcel Moolenaar 93c243e490SMarcel Moolenaar* Made atf-run hardcode TZ=UTC for test cases. It used to undefine TZ, but 94c243e490SMarcel Moolenaar that does not take into account that libc determines the current timezone 95c243e490SMarcel Moolenaar from a configuration file. 96c243e490SMarcel Moolenaar 97c243e490SMarcel Moolenaar* All test programs will now print a warning when they are not run through 98c243e490SMarcel Moolenaar atf-run(1) stating that this is unsupported and may deliver incorrect 99c243e490SMarcel Moolenaar results. 100c243e490SMarcel Moolenaar 101c243e490SMarcel Moolenaar* Added support for the 'require.files' test-case property. This allows 102c243e490SMarcel Moolenaar test cases to specify installed files that must be present for the test 103c243e490SMarcel Moolenaar case to run. 104c243e490SMarcel Moolenaar 105c243e490SMarcel Moolenaar 106c243e490SMarcel MoolenaarChanges in version 0.13 107c243e490SMarcel Moolenaar*********************** 108c243e490SMarcel Moolenaar 109c243e490SMarcel MoolenaarExperimental version released on March 31st, 2011. 110c243e490SMarcel Moolenaar 111c243e490SMarcel MoolenaarThis is the first release after the creation of the Kyua project, a more 112c243e490SMarcel Moolenaarmodular and reliable replacement for ATF. From now on, ATF will change to 113c243e490SMarcel Moolenaaraccomodate the transition to this new codebase, but ATF will still continue 114c243e490SMarcel Moolenaarto see development in the short/medium term. Check out the project page at 115c243e490SMarcel Moolenaarhttp://code.google.com/p/kyua/ for more details. 116c243e490SMarcel Moolenaar 117c243e490SMarcel MoolenaarThe changes in this release are: 118c243e490SMarcel Moolenaar 119c243e490SMarcel Moolenaar* Added support to run the tests with the Kyua runtime engine (kyua-cli), a 120c243e490SMarcel Moolenaar new package that aims to replace atf-run and atf-report. The ATF tests 121c243e490SMarcel Moolenaar can be run with the new system by issuing a 'make installcheck-kyua' from 122c243e490SMarcel Moolenaar the top-level directory of the project (assuming the 'kyua' binary is 123c243e490SMarcel Moolenaar available during the configuration stage of ATF). 124c243e490SMarcel Moolenaar 125c243e490SMarcel Moolenaar* atf-run and atf-report are now in maintenance mode (but *not* deprecated 126c243e490SMarcel Moolenaar yet!). Kyua already implements a new, much more reliable runtime engine 127c243e490SMarcel Moolenaar that provides similar features to these tools. That said, it is not 128c243e490SMarcel Moolenaar complete yet so all development efforts should go towards it. 129c243e490SMarcel Moolenaar 130c243e490SMarcel Moolenaar* If GDB is installed, atf-run dumps the stack trace of crashing test 131c243e490SMarcel Moolenaar programs in an attempt to aid debugging. Contributed by Antti Kantee. 132c243e490SMarcel Moolenaar 133c243e490SMarcel Moolenaar* Reverted default timeout change in previous release and reset its value 134c243e490SMarcel Moolenaar to 5 minutes. This was causing several issues, specially when running 135c243e490SMarcel Moolenaar the existing NetBSD test suite in qemu. 136c243e490SMarcel Moolenaar 137c243e490SMarcel Moolenaar* Fixed the 'match' output checker in atf-check to properly validate the 138c243e490SMarcel Moolenaar last line of a file even if it does not have a newline. 139c243e490SMarcel Moolenaar 140c243e490SMarcel Moolenaar* Added the ATF_REQUIRE_IN and ATF_REQUIRE_NOT_IN macros to atf-c++ to 141c243e490SMarcel Moolenaar check for the presence (or lack thereof) of an element in a collection. 142c243e490SMarcel Moolenaar 143c243e490SMarcel Moolenaar* PR bin/44176: Fixed a race condition in atf-run that would crash atf-run 144c243e490SMarcel Moolenaar when the cleanup of a test case triggered asynchronous modifications to 145c243e490SMarcel Moolenaar its work directory (e.g. killing a daemon process that cleans up a pid 146c243e490SMarcel Moolenaar file in the work directory). 147c243e490SMarcel Moolenaar 148c243e490SMarcel Moolenaar* PR bin/44301: Fixed the sample XSLT file to report bogus test programs 149c243e490SMarcel Moolenaar instead of just listing them as having 0 test cases. 150c243e490SMarcel Moolenaar 151c243e490SMarcel Moolenaar 152c243e490SMarcel MoolenaarChanges in version 0.12 153c243e490SMarcel Moolenaar*********************** 154c243e490SMarcel Moolenaar 155c243e490SMarcel MoolenaarExperimental version released on November 7th, 2010. 156c243e490SMarcel Moolenaar 157c243e490SMarcel Moolenaar* Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as 158c243e490SMarcel Moolenaar ATF_REQUIRE_THROW but allows checking for the validity of the exception's 159c243e490SMarcel Moolenaar error message by means of a regular expression. 160c243e490SMarcel Moolenaar 161c243e490SMarcel Moolenaar* Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a 162c243e490SMarcel Moolenaar regular expression match in a string. 163c243e490SMarcel Moolenaar 164c243e490SMarcel Moolenaar* Changed the default timeout for test cases from 5 minutes to 30 seconds. 165c243e490SMarcel Moolenaar 30 seconds is long enough for virtually all tests to complete, and 5 166c243e490SMarcel Moolenaar minutes is a way too long pause in a test suite where a single test case 167c243e490SMarcel Moolenaar stalls. 168c243e490SMarcel Moolenaar 169c243e490SMarcel Moolenaar* Deprecated the use.fs property. While this seemed like a good idea in 170c243e490SMarcel Moolenaar the first place to impose more control on what test cases can do, it 171c243e490SMarcel Moolenaar turns out to be bad. First, use.fs=false prevents bogus test cases 172c243e490SMarcel Moolenaar from dumping core so after-the-fact debugging is harder. Second, 173c243e490SMarcel Moolenaar supporting use.fs adds a lot of unnecessary complexity. atf-run will 174c243e490SMarcel Moolenaar now ignore any value provided to use.fs and will allow test cases to 175c243e490SMarcel Moolenaar freely access the file system if they wish to. 176c243e490SMarcel Moolenaar 177c243e490SMarcel Moolenaar* Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c 178c243e490SMarcel Moolenaar library. The 'text' module became private in 0.11 but was being used 179c243e490SMarcel Moolenaar externally to simplify the parsing of configuration variables. 180c243e490SMarcel Moolenaar 181c243e490SMarcel Moolenaar* Made atf-run recognize the 'unprivileged-user' configuration variable 182c243e490SMarcel Moolenaar and automatically drop root privileges when a test case sets 183c243e490SMarcel Moolenaar require.user=unprivileged. Note that this is, by no means, done for 184c243e490SMarcel Moolenaar security purposes; this is just for user convenience; tests should, in 185c243e490SMarcel Moolenaar general, not be blindly run as root in the first place. 186c243e490SMarcel Moolenaar 187c243e490SMarcel Moolenaar 188c243e490SMarcel MoolenaarChanges in version 0.11 189c243e490SMarcel Moolenaar*********************** 190c243e490SMarcel Moolenaar 191c243e490SMarcel MoolenaarExperimental version released on October 20th, 2010. 192c243e490SMarcel Moolenaar 193c243e490SMarcel Moolenaar* The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match 194c243e490SMarcel Moolenaar their counterparts in atf-c. 195c243e490SMarcel Moolenaar 196c243e490SMarcel Moolenaar* Clearly separated the modules in atf-c that are supposed to be public 197c243e490SMarcel Moolenaar from those that are implementation details. The header files for the 198c243e490SMarcel Moolenaar internal modules are not installed any more. 199c243e490SMarcel Moolenaar 200c243e490SMarcel Moolenaar* Made the atf-check tool private. It is only required by atf-sh and being 201c243e490SMarcel Moolenaar public has the danger of causing confusion. Also, making it private 202c243e490SMarcel Moolenaar simplifies the public API of atf. 203c243e490SMarcel Moolenaar 204c243e490SMarcel Moolenaar* Changed atf-sh to enable per-command error checking (set -e) by default. 205c243e490SMarcel Moolenaar This catches many cases in which a test case is broken but it is not 206c243e490SMarcel Moolenaar reported as such because execution continues. 207c243e490SMarcel Moolenaar 208c243e490SMarcel Moolenaar* Fixed the XSTL and CSS stylesheets to support expected failures. 209c243e490SMarcel Moolenaar 210c243e490SMarcel Moolenaar 211c243e490SMarcel MoolenaarChanges in version 0.10 212c243e490SMarcel Moolenaar*********************** 213c243e490SMarcel Moolenaar 214c243e490SMarcel MoolenaarExperimental version released on July 2nd, 2010. 215c243e490SMarcel Moolenaar 216c243e490SMarcel MoolenaarMiscellaneous features 217c243e490SMarcel Moolenaar 218c243e490SMarcel Moolenaar* Added expected failures support to test cases and atf-run. These 219c243e490SMarcel Moolenaar include, for example, expected clean exits, expected reception of fatal 220c243e490SMarcel Moolenaar signals, expected timeouts and expected errors in condition checks. 221c243e490SMarcel Moolenaar These statuses can be used to denote test cases that are known to fail 222c243e490SMarcel Moolenaar due to a bug in the code they are testing. atf-report reports these 223c243e490SMarcel Moolenaar tests separately but they do not count towards the failed test cases 224c243e490SMarcel Moolenaar amount. 225c243e490SMarcel Moolenaar 226c243e490SMarcel Moolenaar* Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to 227c243e490SMarcel Moolenaar allow easy checking of call failures that update errno. 228c243e490SMarcel Moolenaar 229c243e490SMarcel Moolenaar* Added the has.cleanup meta-data property to test caes that specifies 230c243e490SMarcel Moolenaar whether the test case has a cleanup routine or not; its value is 231c243e490SMarcel Moolenaar automatically set. This property is read by atf-run to know if it has to 232c243e490SMarcel Moolenaar run the cleanup routine; skipping this run for every test case 233c243e490SMarcel Moolenaar significantly speeds up the run time of test suites. 234c243e490SMarcel Moolenaar 235c243e490SMarcel Moolenaar* Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to 236c243e490SMarcel Moolenaar take the expected exception as the first argument and the statement to 237c243e490SMarcel Moolenaar execute as the second argument. 238c243e490SMarcel Moolenaar 239c243e490SMarcel MoolenaarChanges in atf-check 240c243e490SMarcel Moolenaar 241c243e490SMarcel Moolenaar* Changed atf-check to support negating the status and output checks by 242c243e490SMarcel Moolenaar prefixing them with not- and added support to specify multiple checkers 243c243e490SMarcel Moolenaar for stdout and stderr, not only one. 244c243e490SMarcel Moolenaar 245c243e490SMarcel Moolenaar* Added the match output checker to atf-check to look for regular 246c243e490SMarcel Moolenaar expressions in the stdout and stderr of commands. 247c243e490SMarcel Moolenaar 248c243e490SMarcel Moolenaar* Modified the exit checks in atf-check to support checking for the 249c243e490SMarcel Moolenaar reception of signals. 250c243e490SMarcel Moolenaar 251c243e490SMarcel MoolenaarCode simplifications and cleanups 252c243e490SMarcel Moolenaar 253c243e490SMarcel Moolenaar* Removed usage messages from test programs to simplify the 254c243e490SMarcel Moolenaar implementation of every binding by a significant amount. They just now 255c243e490SMarcel Moolenaar refer the user to the appropriate manual page and do not attempt to wrap 256c243e490SMarcel Moolenaar lines on terminal boundaries. Test programs are not supposed to be run 257c243e490SMarcel Moolenaar by users directly so this minor interface regression is not important. 258c243e490SMarcel Moolenaar 259c243e490SMarcel Moolenaar* Removed the atf-format internal utility, which is unused after the 260c243e490SMarcel Moolenaar change documented above. 261c243e490SMarcel Moolenaar 262c243e490SMarcel Moolenaar* Removed the atf-cleanup internal utility. It has been unused since the 263c243e490SMarcel Moolenaar test case isolation was moved to atf-run in 0.8 264c243e490SMarcel Moolenaar 265c243e490SMarcel Moolenaar* Splitted the Makefile.am into smaller files for easier maintenance and 266c243e490SMarcel Moolenaar dropped the use of M4. Only affects users building from the repository 267c243e490SMarcel Moolenaar sources. 268c243e490SMarcel Moolenaar 269c243e490SMarcel Moolenaar* Intermixed tests with the source files in the source tree to provide 270c243e490SMarcel Moolenaar them more visibility and easier access. The tests directory is gone from 271c243e490SMarcel Moolenaar the source tree and tests are now suffixed by _test, not prefixed by t_. 272c243e490SMarcel Moolenaar 273c243e490SMarcel Moolenaar* Simplifications to the atf-c library: removed the io, tcr and ui 274c243e490SMarcel Moolenaar modules as they had become unnecessary after all simplifications 275c243e490SMarcel Moolenaar introduced since the 0.8 release. 276c243e490SMarcel Moolenaar 277c243e490SMarcel Moolenaar* Removed the application/X-atf-tcr format introduced in 0.8 release. 278c243e490SMarcel Moolenaar Tests now print a much simplified format that is easy to parse and nicer 279c243e490SMarcel Moolenaar to read by end users. As a side effect, the default for test cases is 280c243e490SMarcel Moolenaar now to print their results to stdout unless otherwise stated by providing 281c243e490SMarcel Moolenaar the -r flag. 282c243e490SMarcel Moolenaar 283c243e490SMarcel Moolenaar* Removed XML distribution documents and replaced them with plain-text 284c243e490SMarcel Moolenaar documents. They provided little value and introduced a lot of complexity 285c243e490SMarcel Moolenaar to the build system. 286c243e490SMarcel Moolenaar 287c243e490SMarcel Moolenaar* Simplified the output of atf-version by not attempting to print a 288c243e490SMarcel Moolenaar revision number when building form a distfile. Makes the build system 289c243e490SMarcel Moolenaar easier to maintain. 290c243e490SMarcel Moolenaar 291c243e490SMarcel Moolenaar 292c243e490SMarcel MoolenaarChanges in version 0.9 293c243e490SMarcel Moolenaar********************** 294c243e490SMarcel Moolenaar 295c243e490SMarcel MoolenaarExperimental version released on June 3rd, 2010. 296c243e490SMarcel Moolenaar 297c243e490SMarcel Moolenaar* Added atf-sh, an interpreter to process test programs written using 298c243e490SMarcel Moolenaar the shell API. This is not really a shell interpreter by itself though: 299c243e490SMarcel Moolenaar it is just a wrapper around the system shell that eases the loading of 300c243e490SMarcel Moolenaar the necessary ATF libraries. 301c243e490SMarcel Moolenaar 302c243e490SMarcel Moolenaar* Removed atf-compile in favour of atf-sh. 303c243e490SMarcel Moolenaar 304c243e490SMarcel Moolenaar* Added the use.fs metadata property to test case, which is used to 305c243e490SMarcel Moolenaar specify which test cases require file system access. This is to 306c243e490SMarcel Moolenaar highlight dependencies on external resources more clearly and to speed up 307c243e490SMarcel Moolenaar the execution of test suites by skipping the creation of many unnecessary 308c243e490SMarcel Moolenaar work directories. 309c243e490SMarcel Moolenaar 310c243e490SMarcel Moolenaar* Fixed test programs to get a sane default value for their source 311c243e490SMarcel Moolenaar directory. This means that it should not be necessary any more to pass 312c243e490SMarcel Moolenaar -s when running test programs that do not live in the current directory. 313c243e490SMarcel Moolenaar 314c243e490SMarcel Moolenaar* Defining test case headers became optional. This is trivial to achieve 315c243e490SMarcel Moolenaar in shell-based tests but a bit ugly in C and C++. In C, use the new 316c243e490SMarcel Moolenaar ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use 317c243e490SMarcel Moolenaar ATF_TEST_CASE_WITHOUT_HEAD. 318c243e490SMarcel Moolenaar 319c243e490SMarcel Moolenaar 320c243e490SMarcel MoolenaarChanges in version 0.8 321c243e490SMarcel Moolenaar********************** 322c243e490SMarcel Moolenaar 323c243e490SMarcel MoolenaarExperimental version released on May 7th, 2010. 324c243e490SMarcel Moolenaar 325c243e490SMarcel Moolenaar* Test programs no longer run several test cases in a row. The execution 326c243e490SMarcel Moolenaar of a test program now requires a test case name, and that single test 327c243e490SMarcel Moolenaar case is executed. To execute several test cases, use the atf-run utility 328c243e490SMarcel Moolenaar as usual. 329c243e490SMarcel Moolenaar 330c243e490SMarcel Moolenaar* Test programs no longer fork a subprocess to isolate the execution of 331c243e490SMarcel Moolenaar test cases. They run the test case code in-process, and a crash of the 332c243e490SMarcel Moolenaar test case will result in a crash of the test program. This is to ease 333c243e490SMarcel Moolenaar debugging of faulty test cases. 334c243e490SMarcel Moolenaar 335c243e490SMarcel Moolenaar* Test programs no longer isolate their test cases. This means that they 336c243e490SMarcel Moolenaar will not create temporary directories nor sanitize the environment any 337c243e490SMarcel Moolenaar more. Yes: running a test case that depends on system state by hand will 338c243e490SMarcel Moolenaar most likely yield different results depending on where (machine, 339c243e490SMarcel Moolenaar directory, user environment, etc.) it is run. Isolation has been moved 340c243e490SMarcel Moolenaar to atf-run. 341c243e490SMarcel Moolenaar 342c243e490SMarcel Moolenaar* Test programs no longer print a cryptic format (application/X-atf-tcs) 343c243e490SMarcel Moolenaar on a special file channel. They can now print whatever they want on the 344c243e490SMarcel Moolenaar screen. Because test programs can now only run one test case every time, 345c243e490SMarcel Moolenaar providing controlled output is not necessary any more. 346c243e490SMarcel Moolenaar 347c243e490SMarcel Moolenaar* Test programs no longer write their status into a special file 348c243e490SMarcel Moolenaar descriptor. Instead, they create a file with the results, which is later 349c243e490SMarcel Moolenaar parsed by atf-run. This changes the semantics of the -r flag. 350c243e490SMarcel Moolenaar 351c243e490SMarcel Moolenaar* atf-run has been adjusted to perform the test case isolation. As a 352c243e490SMarcel Moolenaar result, there is now a single canonical place that implements the 353c243e490SMarcel Moolenaar isolation of test caes. In previous releases, the three language 354c243e490SMarcel Moolenaar bindings (C, C++ and shell) had to be kept in sync with each other (read: 355c243e490SMarcel Moolenaar not a nice thing to do at all). As a side effect of this change, writing 356c243e490SMarcel Moolenaar bindings for other languages will be much, much easier from now on. 357c243e490SMarcel Moolenaar 358c243e490SMarcel Moolenaar* atf-run forks test programs on a test case basis, instead of on a test 359c243e490SMarcel Moolenaar program basis as it did before. This is to provide the test case 360c243e490SMarcel Moolenaar isolation that was before implemented by the test programs themselves. 361c243e490SMarcel Moolenaar 362c243e490SMarcel Moolenaar* Removed the atf-exec tool. This was used to implement test case 363c243e490SMarcel Moolenaar isolation in atf-sh, but it is now unnecessary. 364c243e490SMarcel Moolenaar 365c243e490SMarcel Moolenaar* It is now optional to define the descr meta-data property. It has been 366c243e490SMarcel Moolenaar proven to be mostly useless, because test cases often carry a descriptive 367c243e490SMarcel Moolenaar name of their own. 368c243e490SMarcel Moolenaar 369c243e490SMarcel Moolenaar 370c243e490SMarcel MoolenaarChanges in version 0.7 371c243e490SMarcel Moolenaar********************** 372c243e490SMarcel Moolenaar 373c243e490SMarcel MoolenaarExperimental version released on December 22nd, 2009. 374c243e490SMarcel Moolenaar 375c243e490SMarcel Moolenaar* Added build-time checks to atf-c and atf-c++. A binding for atf-sh 376c243e490SMarcel Moolenaar will come later. 377c243e490SMarcel Moolenaar 378c243e490SMarcel Moolenaar* Migrated all build-time checks for header files to proper ATF tests. 379c243e490SMarcel Moolenaar This demonstrates the use of the new feature described above. 380c243e490SMarcel Moolenaar 381c243e490SMarcel Moolenaar* Added an internal API for child process management. 382c243e490SMarcel Moolenaar 383c243e490SMarcel Moolenaar* Converted all plain-text distribution documents to a Docbook canonical 384c243e490SMarcel Moolenaar version, and include pre-generated plain text and HTML copies in the 385c243e490SMarcel Moolenaar distribution file. 386c243e490SMarcel Moolenaar 387c243e490SMarcel Moolenaar* Simplified the contents of the Makefile.am by regenerating it from a 388c243e490SMarcel Moolenaar canonical Makefile.am.m4 source. As a side-effect, some dependency 389c243e490SMarcel Moolenaar specifications were fixed. 390c243e490SMarcel Moolenaar 391c243e490SMarcel Moolenaar* Migrated all checks from the check target to installcheck, as these 392c243e490SMarcel Moolenaar require ATF to be installed. 393c243e490SMarcel Moolenaar 394c243e490SMarcel Moolenaar* Fixed sign comparison mismatches triggered by the now-enabled 395c243e490SMarcel Moolenaar -Wsign-compare. 396c243e490SMarcel Moolenaar 397c243e490SMarcel Moolenaar* Fixed many memory and object leaks. 398c243e490SMarcel Moolenaar 399c243e490SMarcel Moolenaar 400c243e490SMarcel MoolenaarChanges in version 0.6 401c243e490SMarcel Moolenaar********************** 402c243e490SMarcel Moolenaar 403c243e490SMarcel MoolenaarExperimental version released on January 18th, 2009. 404c243e490SMarcel Moolenaar 405c243e490SMarcel Moolenaar* Make atf-exec be able to kill its child process after a certain period 406c243e490SMarcel Moolenaar of time; this is controlled through the new -t option. 407c243e490SMarcel Moolenaar 408c243e490SMarcel Moolenaar* Change atf-sh to use atf-exec's -t option to control the test case's 409c243e490SMarcel Moolenaar timeouts, instead of doing it internally. Same behavior as before, but 410c243e490SMarcel Moolenaar noticeably faster. 411c243e490SMarcel Moolenaar 412c243e490SMarcel Moolenaar* atf-exec's -g option and atf-killpg are gone due to the previous 413c243e490SMarcel Moolenaar change. 414c243e490SMarcel Moolenaar 415c243e490SMarcel Moolenaar* Added the atf-check(1) tool, a program that executes a given command 416c243e490SMarcel Moolenaar and checks its exit code against a known value and allows the management 417c243e490SMarcel Moolenaar of stdout and stderr in multiple ways. This replaces the previous 418c243e490SMarcel Moolenaar atf_check function in the atf-sh library and exposes this functionality 419c243e490SMarcel Moolenaar to both atf-c and atf-c++. 420c243e490SMarcel Moolenaar 421c243e490SMarcel Moolenaar* Added the ATF_REQUIRE family of macros to the C interface. These help 422c243e490SMarcel Moolenaar in checking for fatal test conditions. The old ATF_CHECK macros now 423c243e490SMarcel Moolenaar perform non-fatal checks only. I.e. by using ATF_CHECK, the test case 424c243e490SMarcel Moolenaar can now continue its execution and the failures will not be reported 425c243e490SMarcel Moolenaar until the end of the whole run. 426c243e490SMarcel Moolenaar 427c243e490SMarcel Moolenaar* Extended the amount of ATF_CHECK_* C macros with new ones to provide 428c243e490SMarcel Moolenaar more features to the developer. These also have their corresponding 429c243e490SMarcel Moolenaar counterparts in the ATF_REQUIRE_* family. The new macros (listing the 430c243e490SMarcel Moolenaar suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and 431c243e490SMarcel Moolenaar _STREQ_MSG. 432c243e490SMarcel Moolenaar 433c243e490SMarcel Moolenaar 434c243e490SMarcel MoolenaarChanges in version 0.5 435c243e490SMarcel Moolenaar********************** 436c243e490SMarcel Moolenaar 437c243e490SMarcel MoolenaarExperimental version released on May 1st, 2008. 438c243e490SMarcel Moolenaar 439c243e490SMarcel Moolenaar* Clauses 3 and 4 of the BSD license used by the project were dropped. 440c243e490SMarcel Moolenaar All the code is now under a 2-clause BSD license compatible with the GNU 441c243e490SMarcel Moolenaar General Public License (GPL). 442c243e490SMarcel Moolenaar 443c243e490SMarcel Moolenaar* Added a C-only binding so that binary test programs do not need to be 444c243e490SMarcel Moolenaar tied to C++ at all. This binding is now known as the atf-c library. 445c243e490SMarcel Moolenaar 446c243e490SMarcel Moolenaar* Renamed the C++ binding to atf-c++ for consistency with the new atf-c. 447c243e490SMarcel Moolenaar 448c243e490SMarcel Moolenaar* Renamed the POSIX shell binding to atf-sh for consistency with the new 449c243e490SMarcel Moolenaar atf-c and atf-c++. 450c243e490SMarcel Moolenaar 451c243e490SMarcel Moolenaar* Added a -w flag to test programs through which it is possible to 452c243e490SMarcel Moolenaar specify the work directory to be used. This was possible in prior 453c243e490SMarcel Moolenaar releases by defining the workdir configuration variable (-v workdir=...), 454c243e490SMarcel Moolenaar but was a conceptually incorrect mechanism. 455c243e490SMarcel Moolenaar 456c243e490SMarcel Moolenaar* Test programs now preserve the execution order of test cases when they 457c243e490SMarcel Moolenaar are given in the command line. Even those mentioned more than once are 458c243e490SMarcel Moolenaar executed multiple times to comply with the user's requests. 459c243e490SMarcel Moolenaar 460c243e490SMarcel Moolenaar 461c243e490SMarcel MoolenaarChanges in version 0.4 462c243e490SMarcel Moolenaar********************** 463c243e490SMarcel Moolenaar 464c243e490SMarcel MoolenaarExperimental version released on February 4th, 2008. 465c243e490SMarcel Moolenaar 466c243e490SMarcel Moolenaar* Added two new manual pages, atf-c++-api and atf-sh-api, describing the 467c243e490SMarcel Moolenaar C++ and POSIX shell interfaces used to write test programs. 468c243e490SMarcel Moolenaar 469c243e490SMarcel Moolenaar* Added a pkg-config file, useful to get the flags to build against the 470c243e490SMarcel Moolenaar C++ library or to easily detect the presence of ATF. 471c243e490SMarcel Moolenaar 472c243e490SMarcel Moolenaar* Added a way for test cases to require a specific architecture and/or 473c243e490SMarcel Moolenaar machine type through the new 'require.arch' and 'require.machine' 474c243e490SMarcel Moolenaar meta-data properties, respectively. 475c243e490SMarcel Moolenaar 476c243e490SMarcel Moolenaar* Added the 'timeout' property to test cases, useful to set an 477c243e490SMarcel Moolenaar upper-bound limit for the test's run time and thus prevent global test 478c243e490SMarcel Moolenaar program stalls due to the test case's misbehavior. 479c243e490SMarcel Moolenaar 480c243e490SMarcel Moolenaar* Added the atf-exec(1) internal utility, used to execute a command 481c243e490SMarcel Moolenaar after changing the process group it belongs to. 482c243e490SMarcel Moolenaar 483c243e490SMarcel Moolenaar* Added the atf-killpg(1) internal utility, used to kill process groups. 484c243e490SMarcel Moolenaar 485c243e490SMarcel Moolenaar* Multiple portability fixes. Of special interest, full support for 486c243e490SMarcel Moolenaar SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12 487c243e490SMarcel Moolenaar C++ compiler. 488c243e490SMarcel Moolenaar 489c243e490SMarcel Moolenaar* Fixed a serious bug that prevented atf-run(1) from working at all 490c243e490SMarcel Moolenaar under Fedora 8 x86_64. Due to the nature of the bug, other platforms 491c243e490SMarcel Moolenaar were likely affected too. 492c243e490SMarcel Moolenaar 493c243e490SMarcel Moolenaar 494c243e490SMarcel MoolenaarChanges in version 0.3 495c243e490SMarcel Moolenaar********************** 496c243e490SMarcel Moolenaar 497c243e490SMarcel MoolenaarExperimental version released on November 11th, 2007. 498c243e490SMarcel Moolenaar 499c243e490SMarcel Moolenaar* Added XML output support to atf-report. This is accompanied by a DTD 500c243e490SMarcel Moolenaar for the format's structure and sample XSLT/CSS files to post-process this 501c243e490SMarcel Moolenaar output and convert it to a plain HTML report. 502c243e490SMarcel Moolenaar 503c243e490SMarcel Moolenaar* Changed atf-run to add system information to the report it generates. 504c243e490SMarcel Moolenaar This is currently used by atf-report's XML output only, and is later 505c243e490SMarcel Moolenaar printed in the HTML reports in a nice and useful summary table. The user 506c243e490SMarcel Moolenaar and system administrator are allowed to tune this feature by means of 507c243e490SMarcel Moolenaar hooks. 508c243e490SMarcel Moolenaar 509c243e490SMarcel Moolenaar* Removed the test cases' 'isolated' property. This was intended to 510c243e490SMarcel Moolenaar avoid touching the file system at all when running the related test case, 511c243e490SMarcel Moolenaar but this has not been true for a long while: some control files are 512c243e490SMarcel Moolenaar unconditionally required for several purposes, and we cannot easily get 513c243e490SMarcel Moolenaar rid of them. This way we remove several critical and delicate pieces of 514c243e490SMarcel Moolenaar code. 515c243e490SMarcel Moolenaar 516c243e490SMarcel Moolenaar* Improved atf-report's CSV output format to include information about 517c243e490SMarcel Moolenaar test programs too. 518c243e490SMarcel Moolenaar 519c243e490SMarcel Moolenaar* Fixed the tests that used atf-compile to not require this tool as a 520c243e490SMarcel Moolenaar helper. Avoids systems without build-time utilities to skip many tests 521c243e490SMarcel Moolenaar that could otherwise be run. (E.g. NetBSD without the comp.tgz set 522c243e490SMarcel Moolenaar installed.) 523c243e490SMarcel Moolenaar 524c243e490SMarcel Moolenaar* Many general cleanups: Fixed many pieces of code marked as ugly and/or 525c243e490SMarcel Moolenaar incomplete. 526c243e490SMarcel Moolenaar 527c243e490SMarcel Moolenaar 528c243e490SMarcel MoolenaarChanges in version 0.2 529c243e490SMarcel Moolenaar********************** 530c243e490SMarcel Moolenaar 531c243e490SMarcel MoolenaarExperimental version released on September 20th, 2007. 532c243e490SMarcel Moolenaar 533c243e490SMarcel Moolenaar* Test cases now get a known umask on entry. 534c243e490SMarcel Moolenaar 535c243e490SMarcel Moolenaar* atf-run now detects many unexpected failures caused by test programs and 536c243e490SMarcel Moolenaar reports them as bogus tests. atf-report is able to handle these new 537c243e490SMarcel Moolenaar errors and nicely reports them to the user. 538c243e490SMarcel Moolenaar 539c243e490SMarcel Moolenaar* All the data formats read and written by the tools have been 540c243e490SMarcel Moolenaar documented and cleaned up. These include those grammars that define how 541c243e490SMarcel Moolenaar the different components communicate with each other as well as the 542c243e490SMarcel Moolenaar format of files written by the developers and users: the Atffiles and the 543c243e490SMarcel Moolenaar configuration files. 544c243e490SMarcel Moolenaar 545c243e490SMarcel Moolenaar* Added the atf-version tool, a utility that displays information about 546c243e490SMarcel Moolenaar the currently installed version of ATF. 547c243e490SMarcel Moolenaar 548c243e490SMarcel Moolenaar* Test cases can now define an optional cleanup routine to undo their 549c243e490SMarcel Moolenaar actions regardless of their exit status. 550c243e490SMarcel Moolenaar 551c243e490SMarcel Moolenaar* atf-report now summarizes the list of failed (bogus) test programs 552c243e490SMarcel Moolenaar when using the ticker output format. 553c243e490SMarcel Moolenaar 554c243e490SMarcel Moolenaar* Test programs now capture some termination signals and clean up any 555c243e490SMarcel Moolenaar temporary files before exiting the program. 556c243e490SMarcel Moolenaar 557c243e490SMarcel Moolenaar* Multiple bug fixes and improvements all around. 558c243e490SMarcel Moolenaar 559c243e490SMarcel Moolenaar 560c243e490SMarcel MoolenaarChanges in version 0.1 561c243e490SMarcel Moolenaar********************** 562c243e490SMarcel Moolenaar 563c243e490SMarcel MoolenaarExperimental version released on August 20th, 2007. 564c243e490SMarcel Moolenaar 565c243e490SMarcel Moolenaar* First public version. This was released coinciding with the end of the 566c243e490SMarcel Moolenaar Google Summer of Code 2007 program. 567c243e490SMarcel Moolenaar 568c243e490SMarcel Moolenaar 569c243e490SMarcel Moolenaar=========================================================================== 570c243e490SMarcel Moolenaarvim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2 571