1Changes in version 0.7 2====================== 3 4Experimental version released on October 18th, 2013. 5 6* Made failures from testers more resilent. If a tester fails, the 7 corresponding test case will be marked as broken instead of causing 8 kyua to exit. 9 10* Added the '--results-filter' option to the 'report-html' command and 11 set its default value to skip passed results from HTML reports. This 12 is to keep these reports more succint and to avoid generating tons of 13 detail files that will be, in general, useless. 14 15* Switched to use Lutok 0.3 to gain compatibility with Lua 5.2. 16 17* Issue 69: Cope with the lack of AM_PROG_AR in configure.ac, which 18 first appeared in Automake 1.11.2. Fixes a problem in Ubuntu 10.04 19 LTS, which appears stuck in 1.11.1. 20 21 22Changes in version 0.6 23====================== 24 25Experimental version released on February 22nd, 2013. 26 27* Issue 36: Changed 'kyua help' to not fail when the configuration file 28 is bogus. Help should always work. 29 30* Issue 37: Simplified the syntax() calls in configuration and Kyuafile 31 files to only specify the requested version instead of also the format 32 name. The format name is implied by the file being loaded, so there 33 is no use in the caller having to specify it. The version number of 34 these file formats has been bumped to 2. 35 36* Issue 39: Added per-test-case metadata values to the HTML reports. 37 38* Issue 40: Rewrote the documentation as manual pages and removed the 39 previous GNU Info document. 40 41* Issue 47: Started using the independent testers in the kyua-testers 42 package to run the test cases. Kyua does not implement the logic to 43 invoke test cases any more, which provides for better modularity, 44 extensibility and robustness. 45 46* Issue 57: Added support to specify arbitrary metadata properties for 47 test programs right from the Kyuafile. This is to make plain test 48 programs more versatile, by allowing them to specify any of the 49 requirements (allowed architectures, required files, etc.) supported 50 by Kyua. 51 52* Reduced automatic screen line wrapping of messages to the 'help' 53 command and the output of tables by 'db-exec'. Wrapping any other 54 messages (specially anything going to stderr) was very annoying 55 because it prevented natural copy/pasting of text. 56 57* Increased the granularity of the error codes returned by kyua(1) to 58 denote different error conditions. This avoids the overload of '1' to 59 indicate both "expected" errors from specific subcommands and 60 unexpected errors caused by the internals of the code. The manual now 61 correctly explain how the exit codes behave on a command basis. 62 63* Optimized the database schema to make report generation almost 64 instantaneous. 65 66* Bumped the database schema to 2. The database now records the 67 metadata of both test programs and test cases generically, without 68 knowledge of their interface. 69 70* Added the 'db-migrate' command to provide a mechanism to upgrade a 71 database with an old schema to the current schema. 72 73* Removed the GDB build-time configuration variable. This is now part 74 of the kyua-testers package. 75 76* Issue 31: Rewrote the Kyuafile parsing code in C++, which results in 77 a much simpler implementation. As a side-effect, this gets rid of the 78 external Lua files required by 'kyua', which in turn make the tool 79 self-contained. 80 81* Added caching of various configure test results (particularly in those 82 tests that need to execute a test program) so that cross-compilers can 83 predefine the results of the tests without having to run the 84 executables. 85 86 87Changes in version 0.5 88====================== 89 90Experimental version released on July 10th, 2012. 91 92* Issue 15: Added automatic stacktrace gathering of crashing test cases. 93 This relies on GDB and is a best-effort operation. 94 95* Issue 32: Added the '--build-root' option to the debug, list and test 96 commands. This allows executing test programs from a different 97 directory than where the Kyuafile scripts live. See the 'Build roots' 98 section in the manual for more details. 99 100* Issue 33: Removed the kyuaify.sh script. This has been renamed to 101 atf2kyua and moved to the kyua-atf-compat module, where it ships as a 102 first-class utility (with a manual page and tests). 103 104* Issue 34: Changed the HTML reports to include the stdout and stderr of 105 every test case. 106 107* Fixed the build when using a "build directory" and a clean source tree 108 from the repository. 109 110 111Changes in version 0.4 112====================== 113 114Experimental version released on June 6th, 2012. 115 116* Added the 'report-html' command to generate HTML reports of the 117 execution of any recorded action. 118 119* Changed the '--output' flag of the 'report' command to only take a 120 path to the target file, not its format. Different formats are better 121 supported by implementing different subcommands, as the options they 122 may receive will vary from format to format. 123 124* Added a '--with-atf' flag to the configure script to control whether 125 the ATF tests get built or not. May be useful for packaging systems 126 that do not have ATF in them yet. Disabling ATF also cuts down the 127 build time of Kyua significantly, but with the obvious drawbacks. 128 129* Grouped 'kyua' subcommands by topic both in the output of 'help' and 130 in the documentation. In general, the user needs to be aware of 131 commands that rely on a current project and those commands that rely 132 purely on the database to generate reports. 133 134* Made 'help' print the descriptions of options and commands properly 135 tabulated. 136 137* Changed most informational messages to automatically wrap on screen 138 boundaries. 139 140* Rewrote the configuration file parsing module for extensibility. This 141 will allow future versions of Kyua to provide additional user-facing 142 options in the configuration file. 143 144 No syntax changes have been made, so existing configuration files 145 (version 1) will continue to be parsed without problems. There is one 146 little exception though: all variables under the top-level 147 'test_suites' tree must be declared as strings. 148 149 Similarly, the '-v' and '--variable' flags to the command line must 150 now carry a 'test_suites.' prefix when referencing any variables under 151 such tree. 152 153 154Changes in version 0.3 155====================== 156 157Experimental version released on February 24th, 2012. 158 159* Made the 'test' command record the results of the executed test 160 cases into a SQLite database. As a side effect, 'test' now supports a 161 '--store' option to indicate where the database lives. 162 163* Added the 'report' command to generate plain-text reports of the 164 test results stored in the database. The interface of this command is 165 certainly subject to change at this point. 166 167* Added the 'db-exec' command to directly interact with the store 168 database. 169 170* Issue 28: Added support for the 'require.memory' test case property 171 introduced in ATF 0.15. 172 173* Renamed the user-specific configuration file from ~/.kyuarc to 174 ~/.kyua/kyua.conf for consistency with other files stored in the 175 ~/.kyua/ subdirectory. 176 177* Switched to use Lutok instead of our own wrappers over the Lua C 178 library. Lutok is just what used to be our own utils::lua module, but 179 is now distributed separately. 180 181* Removed the 'Atffile's from the source tree. Kyua is stable enough 182 to generate trustworthy reports, and we do not want to give the 183 impression that atf-run / atf-report are still supported. 184 185* Enabled logging to stderr for our own test programs. This makes it 186 slightly easier to debug problems in our own code when we get a 187 failing test. 188 189 190Changes in version 0.2 191====================== 192 193Experimental version released on August 24th, 2011. 194 195The biggest change in this release is the ability for Kyua to run test 196programs implemented using different frameworks. What this means is 197that, now, a Kyua test suite can include not only ATF-based test 198programs, but also "legacy" (aka plain) test programs that do not use 199any framework. I.e. if you have tests that are simple programs that 200exit with 0 on success and 1 on failure, you can plug them in into a 201Kyua test suite. 202 203Other than this, there have been several user-visible changes. The most 204important are the addition of the new 'config' and 'debug' subcommands 205to the 'kyua' binary. The former can be used to inspect the runtime 206configuration of Kyua after parsing, and the latter is useful to 207interact with failing tests cases in order to get more data about the 208failure itself. 209 210Without further ado, here comes the itemized list of changes: 211 212* Generalized the run-time engine to support executing test programs 213 that implement different interfaces. Test programs that use the ATF 214 libraries are just a special case of this. (Issue 18.) 215 216* Added support to the engine to run "plain" test programs: i.e. test 217 programs that do not use any framework and report their pass/fail 218 status as an exit code. This is to simplify the integration of legacy 219 test programs into a test suite, and also to demonstrate that the 220 run-time engine is generic enough to support different test 221 interfaces. (Issue 18.) 222 223* Added the 'debug' subcommand. This command allows end users to tweak 224 the execution of a specific test case and to poke into the behavior of 225 its execution. At the moment, all this command allows is to view the 226 stdout and stderr of the command in real time (which the 'test' 227 command currently completely hides). 228 229* Added the 'config' subcommand. This command allows the end user to 230 inspect the current configuration variables after evaluation, without 231 having to read through configuration files. (Issue 11.) 232 233* Removed the test_suites_var function from configuration files. This 234 was used to set the value of test-suite-sepecific variables, but it 235 was ugly-looking. It is now possible to use the more natural syntax 236 'test_suites.<test-suite-name>.<variable> = <value>'. (Issue 11.) 237 238* Added a mechanism to disable the loading of configuration files 239 altogether. Needed for testing purposes and for scriptability. 240 Available by passing the '--config=none' flag. 241 242* Enabled detection of unused parameters and variables in the code and 243 fixed all warnings. (Issue 23.) 244 245* Changed the behavior of "developer mode". Compiler warnings are now 246 enabled unconditionally regardless of whether we are in developer mode 247 or not; developer mode is now only used to perform strict warning 248 checks and to enable assertions. Additionally, developer mode is now 249 only automatically enabled when building from the repository, not for 250 formal releases. (Issue 22.) 251 252* Fixed many build and portability problems to Debian sid with GCC 4.6.3 253 and Ubuntu 10.04.1 LTS. (Issues 20, 21, 26.) 254 255 256Changes in version 0.1 257====================== 258 259Experimental version released on June 23rd, 2011. 260 261This is the first public release of the kyua-cli package. 262 263The scope of this release is to provide functional replacement for the 264'atf-run' utility included in the atf package. At this point, 'kyua' 265can reliably run the NetBSD 5.99.53 test suite delivering the same 266results as 'atf-run'. 267 268The reporting facilities of this release are quite limited. There is 269no replacement for 'atf-report' yet, and there is no easy way of 270debugging failing test programs other than running them by hand. These 271features will mark future milestones and therefore be part of other 272releases. 273 274Be aware that this release has suffered very limited field testing. 275The test suite for kyua-cli is quite comprehensive, but some bugs may 276be left in any place. 277