1Major changes between releases Automated Testing Framework 2=========================================================================== 3 4 5Changes in version 0.12 6*********************** 7 8Experimental version released on November 7th, 2010. 9 10* Added the ATF_REQUIRE_THROW_RE to atf-c++, which is the same as 11 ATF_REQUIRE_THROW but allows checking for the validity of the exception's 12 error message by means of a regular expression. 13 14* Added the ATF_REQUIRE_MATCH to atf-c++, which allows checking for a 15 regular expression match in a string. 16 17* Changed the default timeout for test cases from 5 minutes to 30 seconds. 18 30 seconds is long enough for virtually all tests to complete, and 5 19 minutes is a way too long pause in a test suite where a single test case 20 stalls. 21 22* Deprecated the use.fs property. While this seemed like a good idea in 23 the first place to impose more control on what test cases can do, it 24 turns out to be bad. First, use.fs=false prevents bogus test cases 25 from dumping core so after-the-fact debugging is harder. Second, 26 supporting use.fs adds a lot of unnecessary complexity. atf-run will 27 now ignore any value provided to use.fs and will allow test cases to 28 freely access the file system if they wish to. 29 30* Added the atf_tc_get_config_var_as_{bool,long}{,_wd} functions to the atf-c 31 library. The 'text' module became private in 0.11 but was being used 32 externally to simplify the parsing of configuration variables. 33 34* Made atf-run recognize the 'unprivileged-user' configuration variable 35 and automatically drop root privileges when a test case sets 36 require.user=unprivileged. Note that this is, by no means, done for 37 security purposes; this is just for user convenience; tests should, in 38 general, not be blindly run as root in the first place. 39 40 41Changes in version 0.11 42*********************** 43 44Experimental version released on October 20th, 2010. 45 46* The ATF_CHECK* macros in atf-c++ were renamed to ATF_REQUIRE* to match 47 their counterparts in atf-c. 48 49* Clearly separated the modules in atf-c that are supposed to be public 50 from those that are implementation details. The header files for the 51 internal modules are not installed any more. 52 53* Made the atf-check tool private. It is only required by atf-sh and being 54 public has the danger of causing confusion. Also, making it private 55 simplifies the public API of atf. 56 57* Changed atf-sh to enable per-command error checking (set -e) by default. 58 This catches many cases in which a test case is broken but it is not 59 reported as such because execution continues. 60 61* Fixed the XSTL and CSS stylesheets to support expected failures. 62 63 64Changes in version 0.10 65*********************** 66 67Experimental version released on July 2nd, 2010. 68 69Miscellaneous features 70 71* Added expected failures support to test cases and atf-run. These 72 include, for example, expected clean exits, expected reception of fatal 73 signals, expected timeouts and expected errors in condition checks. 74 These statuses can be used to denote test cases that are known to fail 75 due to a bug in the code they are testing. atf-report reports these 76 tests separately but they do not count towards the failed test cases 77 amount. 78 79* Added the ATF_CHECK_ERRNO and ATF_REQUIRE_ERRNO to the C library to 80 allow easy checking of call failures that update errno. 81 82* Added the has.cleanup meta-data property to test caes that specifies 83 whether the test case has a cleanup routine or not; its value is 84 automatically set. This property is read by atf-run to know if it has to 85 run the cleanup routine; skipping this run for every test case 86 significantly speeds up the run time of test suites. 87 88* Reversed the order of the ATF_CHECK_THROW macro in the C++ binding to 89 take the expected exception as the first argument and the statement to 90 execute as the second argument. 91 92Changes in atf-check 93 94* Changed atf-check to support negating the status and output checks by 95 prefixing them with not- and added support to specify multiple checkers 96 for stdout and stderr, not only one. 97 98* Added the match output checker to atf-check to look for regular 99 expressions in the stdout and stderr of commands. 100 101* Modified the exit checks in atf-check to support checking for the 102 reception of signals. 103 104Code simplifications and cleanups 105 106* Removed usage messages from test programs to simplify the 107 implementation of every binding by a significant amount. They just now 108 refer the user to the appropriate manual page and do not attempt to wrap 109 lines on terminal boundaries. Test programs are not supposed to be run 110 by users directly so this minor interface regression is not important. 111 112* Removed the atf-format internal utility, which is unused after the 113 change documented above. 114 115* Removed the atf-cleanup internal utility. It has been unused since the 116 test case isolation was moved to atf-run in 0.8 117 118* Splitted the Makefile.am into smaller files for easier maintenance and 119 dropped the use of M4. Only affects users building from the repository 120 sources. 121 122* Intermixed tests with the source files in the source tree to provide 123 them more visibility and easier access. The tests directory is gone from 124 the source tree and tests are now suffixed by _test, not prefixed by t_. 125 126* Simplifications to the atf-c library: removed the io, tcr and ui 127 modules as they had become unnecessary after all simplifications 128 introduced since the 0.8 release. 129 130* Removed the application/X-atf-tcr format introduced in 0.8 release. 131 Tests now print a much simplified format that is easy to parse and nicer 132 to read by end users. As a side effect, the default for test cases is 133 now to print their results to stdout unless otherwise stated by providing 134 the -r flag. 135 136* Removed XML distribution documents and replaced them with plain-text 137 documents. They provided little value and introduced a lot of complexity 138 to the build system. 139 140* Simplified the output of atf-version by not attempting to print a 141 revision number when building form a distfile. Makes the build system 142 easier to maintain. 143 144 145Changes in version 0.9 146********************** 147 148Experimental version released on June 3rd, 2010. 149 150* Added atf-sh, an interpreter to process test programs written using 151 the shell API. This is not really a shell interpreter by itself though: 152 it is just a wrapper around the system shell that eases the loading of 153 the necessary ATF libraries. 154 155* Removed atf-compile in favour of atf-sh. 156 157* Added the use.fs metadata property to test case, which is used to 158 specify which test cases require file system access. This is to 159 highlight dependencies on external resources more clearly and to speed up 160 the execution of test suites by skipping the creation of many unnecessary 161 work directories. 162 163* Fixed test programs to get a sane default value for their source 164 directory. This means that it should not be necessary any more to pass 165 -s when running test programs that do not live in the current directory. 166 167* Defining test case headers became optional. This is trivial to achieve 168 in shell-based tests but a bit ugly in C and C++. In C, use the new 169 ATF_TC_WITHOUT_HEAD macro to define the test case, and in C++ use 170 ATF_TEST_CASE_WITHOUT_HEAD. 171 172 173Changes in version 0.8 174********************** 175 176Experimental version released on May 7th, 2010. 177 178* Test programs no longer run several test cases in a row. The execution 179 of a test program now requires a test case name, and that single test 180 case is executed. To execute several test cases, use the atf-run utility 181 as usual. 182 183* Test programs no longer fork a subprocess to isolate the execution of 184 test cases. They run the test case code in-process, and a crash of the 185 test case will result in a crash of the test program. This is to ease 186 debugging of faulty test cases. 187 188* Test programs no longer isolate their test cases. This means that they 189 will not create temporary directories nor sanitize the environment any 190 more. Yes: running a test case that depends on system state by hand will 191 most likely yield different results depending on where (machine, 192 directory, user environment, etc.) it is run. Isolation has been moved 193 to atf-run. 194 195* Test programs no longer print a cryptic format (application/X-atf-tcs) 196 on a special file channel. They can now print whatever they want on the 197 screen. Because test programs can now only run one test case every time, 198 providing controlled output is not necessary any more. 199 200* Test programs no longer write their status into a special file 201 descriptor. Instead, they create a file with the results, which is later 202 parsed by atf-run. This changes the semantics of the -r flag. 203 204* atf-run has been adjusted to perform the test case isolation. As a 205 result, there is now a single canonical place that implements the 206 isolation of test caes. In previous releases, the three language 207 bindings (C, C++ and shell) had to be kept in sync with each other (read: 208 not a nice thing to do at all). As a side effect of this change, writing 209 bindings for other languages will be much, much easier from now on. 210 211* atf-run forks test programs on a test case basis, instead of on a test 212 program basis as it did before. This is to provide the test case 213 isolation that was before implemented by the test programs themselves. 214 215* Removed the atf-exec tool. This was used to implement test case 216 isolation in atf-sh, but it is now unnecessary. 217 218* It is now optional to define the descr meta-data property. It has been 219 proven to be mostly useless, because test cases often carry a descriptive 220 name of their own. 221 222 223Changes in version 0.7 224********************** 225 226Experimental version released on December 22nd, 2009. 227 228* Added build-time checks to atf-c and atf-c++. A binding for atf-sh 229 will come later. 230 231* Migrated all build-time checks for header files to proper ATF tests. 232 This demonstrates the use of the new feature described above. 233 234* Added an internal API for child process management. 235 236* Converted all plain-text distribution documents to a Docbook canonical 237 version, and include pre-generated plain text and HTML copies in the 238 distribution file. 239 240* Simplified the contents of the Makefile.am by regenerating it from a 241 canonical Makefile.am.m4 source. As a side-effect, some dependency 242 specifications were fixed. 243 244* Migrated all checks from the check target to installcheck, as these 245 require ATF to be installed. 246 247* Fixed sign comparison mismatches triggered by the now-enabled 248 -Wsign-compare. 249 250* Fixed many memory and object leaks. 251 252 253Changes in version 0.6 254********************** 255 256Experimental version released on January 18th, 2009. 257 258* Make atf-exec be able to kill its child process after a certain period 259 of time; this is controlled through the new -t option. 260 261* Change atf-sh to use atf-exec's -t option to control the test case's 262 timeouts, instead of doing it internally. Same behavior as before, but 263 noticeably faster. 264 265* atf-exec's -g option and atf-killpg are gone due to the previous 266 change. 267 268* Added the atf-check(1) tool, a program that executes a given command 269 and checks its exit code against a known value and allows the management 270 of stdout and stderr in multiple ways. This replaces the previous 271 atf_check function in the atf-sh library and exposes this functionality 272 to both atf-c and atf-c++. 273 274* Added the ATF_REQUIRE family of macros to the C interface. These help 275 in checking for fatal test conditions. The old ATF_CHECK macros now 276 perform non-fatal checks only. I.e. by using ATF_CHECK, the test case 277 can now continue its execution and the failures will not be reported 278 until the end of the whole run. 279 280* Extended the amount of ATF_CHECK_* C macros with new ones to provide 281 more features to the developer. These also have their corresponding 282 counterparts in the ATF_REQUIRE_* family. The new macros (listing the 283 suffixes only) are: _EQ (replaces _EQUAL), _EQ_MSG, _STREQ and 284 _STREQ_MSG. 285 286 287Changes in version 0.5 288********************** 289 290Experimental version released on May 1st, 2008. 291 292* Clauses 3 and 4 of the BSD license used by the project were dropped. 293 All the code is now under a 2-clause BSD license compatible with the GNU 294 General Public License (GPL). 295 296* Added a C-only binding so that binary test programs do not need to be 297 tied to C++ at all. This binding is now known as the atf-c library. 298 299* Renamed the C++ binding to atf-c++ for consistency with the new atf-c. 300 301* Renamed the POSIX shell binding to atf-sh for consistency with the new 302 atf-c and atf-c++. 303 304* Added a -w flag to test programs through which it is possible to 305 specify the work directory to be used. This was possible in prior 306 releases by defining the workdir configuration variable (-v workdir=...), 307 but was a conceptually incorrect mechanism. 308 309* Test programs now preserve the execution order of test cases when they 310 are given in the command line. Even those mentioned more than once are 311 executed multiple times to comply with the user's requests. 312 313 314Changes in version 0.4 315********************** 316 317Experimental version released on February 4th, 2008. 318 319* Added two new manual pages, atf-c++-api and atf-sh-api, describing the 320 C++ and POSIX shell interfaces used to write test programs. 321 322* Added a pkg-config file, useful to get the flags to build against the 323 C++ library or to easily detect the presence of ATF. 324 325* Added a way for test cases to require a specific architecture and/or 326 machine type through the new 'require.arch' and 'require.machine' 327 meta-data properties, respectively. 328 329* Added the 'timeout' property to test cases, useful to set an 330 upper-bound limit for the test's run time and thus prevent global test 331 program stalls due to the test case's misbehavior. 332 333* Added the atf-exec(1) internal utility, used to execute a command 334 after changing the process group it belongs to. 335 336* Added the atf-killpg(1) internal utility, used to kill process groups. 337 338* Multiple portability fixes. Of special interest, full support for 339 SunOS (Solaris Express Developer Edition 2007/09) using the Sun Studio 12 340 C++ compiler. 341 342* Fixed a serious bug that prevented atf-run(1) from working at all 343 under Fedora 8 x86_64. Due to the nature of the bug, other platforms 344 were likely affected too. 345 346 347Changes in version 0.3 348********************** 349 350Experimental version released on November 11th, 2007. 351 352* Added XML output support to atf-report. This is accompanied by a DTD 353 for the format's structure and sample XSLT/CSS files to post-process this 354 output and convert it to a plain HTML report. 355 356* Changed atf-run to add system information to the report it generates. 357 This is currently used by atf-report's XML output only, and is later 358 printed in the HTML reports in a nice and useful summary table. The user 359 and system administrator are allowed to tune this feature by means of 360 hooks. 361 362* Removed the test cases' 'isolated' property. This was intended to 363 avoid touching the file system at all when running the related test case, 364 but this has not been true for a long while: some control files are 365 unconditionally required for several purposes, and we cannot easily get 366 rid of them. This way we remove several critical and delicate pieces of 367 code. 368 369* Improved atf-report's CSV output format to include information about 370 test programs too. 371 372* Fixed the tests that used atf-compile to not require this tool as a 373 helper. Avoids systems without build-time utilities to skip many tests 374 that could otherwise be run. (E.g. NetBSD without the comp.tgz set 375 installed.) 376 377* Many general cleanups: Fixed many pieces of code marked as ugly and/or 378 incomplete. 379 380 381Changes in version 0.2 382********************** 383 384Experimental version released on September 20th, 2007. 385 386* Test cases now get a known umask on entry. 387 388* atf-run now detects many unexpected failures caused by test programs and 389 reports them as bogus tests. atf-report is able to handle these new 390 errors and nicely reports them to the user. 391 392* All the data formats read and written by the tools have been 393 documented and cleaned up. These include those grammars that define how 394 the different components communicate with each other as well as the 395 format of files written by the developers and users: the Atffiles and the 396 configuration files. 397 398* Added the atf-version tool, a utility that displays information about 399 the currently installed version of ATF. 400 401* Test cases can now define an optional cleanup routine to undo their 402 actions regardless of their exit status. 403 404* atf-report now summarizes the list of failed (bogus) test programs 405 when using the ticker output format. 406 407* Test programs now capture some termination signals and clean up any 408 temporary files before exiting the program. 409 410* Multiple bug fixes and improvements all around. 411 412 413Changes in version 0.1 414********************** 415 416Experimental version released on August 20th, 2007. 417 418* First public version. This was released coinciding with the end of the 419 Google Summer of Code 2007 program. 420 421 422=========================================================================== 423vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2 424