1% Testing the JDK 2 3## Using the run-test framework 4 5This new way of running tests is developer-centric. It assumes that you have 6built a JDK locally and want to test it. Running common test targets is simple, 7and more complex ad-hoc combination of tests is possible. The user interface is 8forgiving, and clearly report errors it cannot resolve. 9 10The main target "run-test" uses the jdk-image as the tested product. There is 11also an alternate target "exploded-run-test" that uses the exploded image 12instead. Not all tests will run successfully on the exploded image, but using 13this target can greatly improve rebuild times for certain workflows. 14 15Some example command-lines: 16 17 $ make run-test-tier1 18 $ make run-test-jdk_lang JTREG="JOBS=8" 19 $ make run-test TEST=jdk_lang 20 $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" 21 $ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" 22 $ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java" 23 $ make exploded-run-test TEST=tier2 24 25### Configuration 26 27To be able to run JTReg tests, `configure` needs to know where to find the 28JTReg test framework. If it is not picked up automatically by configure, use 29the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework. 30Note that this option should point to the JTReg home, i.e. the top directory, 31containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME` 32environment variable to point to the JTReg home before running `configure`.) 33 34## Test selection 35 36All functionality is available using the run-test make target. In this use 37case, the test or tests to be executed is controlled using the `TEST` variable. 38To speed up subsequent test runs with no source code changes, run-test-only can 39be used instead, which do not depend on the source and test image build. 40 41For some common top-level tests, direct make targets have been generated. This 42includes all JTReg test groups, the hotspot gtest, and custom tests (if 43present). This means that `make run-test-tier1` is equivalent to `make run-test 44TEST="tier1"`, but the latter is more tab-completion friendly. For more complex 45test runs, the `run-test TEST="x"` solution needs to be used. 46 47The test specifications given in `TEST` is parsed into fully qualified test 48descriptors, which clearly and unambigously show which tests will be run. As an 49example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 50jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 51jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can 52always submit a list of fully qualified test descriptors in the `TEST` variable 53if you want to shortcut the parser. 54 55### JTReg 56 57JTReg tests can be selected either by picking a JTReg test group, or a selection 58of files or directories containing JTReg tests. 59 60JTReg test groups can be specified either without a test root, e.g. `:tier1` 61(or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`, 62`test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test 63root can be specified either as an absolute path, or a path relative to the 64JDK top directory, or the `test` directory. For simplicity, the hotspot 65JTReg test root, which really is `hotspot/jtreg` can be abbreviated as 66just `hotspot`. 67 68When specified without a test root, all matching groups from all test roots 69will be added. Otherwise, only the group from the specified test root will be 70added. 71 72Individual JTReg tests or directories containing JTReg tests can also be 73specified, like `test/hotspot/jtreg/native_sanity/JniVersion.java` or 74`hotspot/jtreg/native_sanity`. Just like for test root selection, you can 75either specify an absolute path (which can even point to JTReg tests outside 76the source tree), or a path relative to either the JDK top directory or the 77`test` directory. `hotspot` can be used as an alias for `hotspot/jtreg` here as 78well. 79 80As long as the test groups or test paths can be uniquely resolved, you do not 81need to enter the `jtreg:` prefix. If this is not possible, or if you want to 82use a fully qualified test descriptor, add `jtreg:`, e.g. 83`jtreg:test/hotspot/jtreg/native_sanity`. 84 85### Gtest 86 87Since the Hotspot Gtest suite is so quick, the default is to run all tests. 88This is specified by just `gtest`, or as a fully qualified test descriptor 89`gtest:all`. 90 91If you want, you can single out an individual test or a group of tests, for 92instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This 93can be particularly useful if you want to run a shaky test repeatedly. 94 95For Gtest, there is a separate test suite for each JVM variant. The JVM variant 96is defined by adding `/<variant>` to the test descriptor, e.g. 97`gtest:Log/client`. If you specify no variant, gtest will run once for each JVM 98variant present (e.g. server, client). So if you only have the server JVM 99present, then `gtest:all` will be equivalent to `gtest:all/server`. 100 101## Test results and summary 102 103At the end of the test run, a summary of all tests run will be presented. This 104will have a consistent look, regardless of what test suites were used. This is 105a sample summary: 106 107 ============================== 108 Test summary 109 ============================== 110 TEST TOTAL PASS FAIL ERROR 111 >> jtreg:jdk/test:tier1 1867 1865 2 0 << 112 jtreg:langtools/test:tier1 4711 4711 0 0 113 jtreg:nashorn/test:tier1 133 133 0 0 114 ============================== 115 TEST FAILURE 116 117Tests where the number of TOTAL tests does not equal the number of PASSed tests 118will be considered a test failure. These are marked with the `>> ... <<` marker 119for easy identification. 120 121The classification of non-passed tests differs a bit between test suites. In 122the summary, ERROR is used as a catch-all for tests that neither passed nor are 123classified as failed by the framework. This might indicate test framework 124error, timeout or other problems. 125 126In case of test failures, `make run-test` will exit with a non-zero exit value. 127 128All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`, 129where TEST_ID is a path-safe conversion from the fully qualified test 130descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is 131`jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the 132test run. 133 134Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For 135some frameworks, this directory might contain information that is useful in 136determining the cause of a failed test. 137 138## Test suite control 139 140It is possible to control various aspects of the test suites using make control 141variables. 142 143These variables use a keyword=value approach to allow multiple values to be 144set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTReg 145concurrency level to 1 and the timeout factor to 8. This is equivalent to 146setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that 147the `JTREG` variable is parsed and verified for correctness, so 148`JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just 149pass unnoticed. 150 151To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell 152normally eats `;`, the recommended usage is to write the assignment inside 153qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved, 154as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`. 155 156(Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`. 157Also, as a special technique, the string `%20` will be replaced with space for 158certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`. 159This can be useful if you have layers of scripts and have trouble getting 160proper quoting of command line arguments through.) 161 162As far as possible, the names of the keywords have been standardized between 163test suites. 164 165### General keywords (TEST_OPTS) 166 167Some keywords are valid across different test suites. If you want to run 168tests from multiple test suites, or just don't want to care which test suite specific 169control variable to use, then you can use the general TEST_OPTS control variable. 170 171There are also some keywords that applies globally to the test runner system, 172not to any specific test suites. These are also available as TEST_OPTS keywords. 173 174#### JOBS 175 176Currently only applies to JTReg. 177 178#### TIMEOUT_FACTOR 179 180Currently only applies to JTReg. 181 182#### VM_OPTIONS 183 184Applies to JTReg, GTest and Micro. 185 186#### JAVA_OPTIONS 187 188Applies to JTReg, GTest and Micro. 189 190#### AOT_MODULES 191 192Applies to JTReg and GTest. 193 194### JTReg keywords 195 196#### JOBS 197The test concurrency (`-concurrency`). 198 199Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to 200JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but 201never more than 12. 202 203#### TIMEOUT_FACTOR 204The timeout factor (`-timeoutFactor`). 205 206Defaults to 4. 207 208#### TEST_MODE 209The test mode (`-agentvm`, `-samevm` or `-othervm`). 210 211Defaults to `-agentvm`. 212 213#### ASSERT 214Enable asserts (`-ea -esa`, or none). 215 216Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except 217for hotspot. 218 219#### VERBOSE 220The verbosity level (`-verbose`). 221 222Defaults to `fail,error,summary`. 223 224#### RETAIN 225What test data to retain (`-retain`). 226 227Defaults to `fail,error`. 228 229#### MAX_MEM 230Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none). 231 232Limit memory consumption for JTReg test framework and VM under test. Set to 0 233to disable the limits. 234 235Defaults to 512m, except for hotspot, where it defaults to 0 (no limit). 236 237#### KEYWORDS 238 239JTReg kewords sent to JTReg using `-k`. Please be careful in making sure that 240spaces and special characters (like `!`) are properly quoted. To avoid some 241issues, the special value `%20` can be used instead of space. 242 243#### EXTRA_PROBLEM_LISTS 244 245Use additional problem lists file or files, in addition to the default 246ProblemList.txt located at the JTReg test roots. 247 248If multiple file names are specified, they should be separated by space (or, to 249help avoid quoting issues, the special value `%20`). 250 251The file names should be either absolute, or relative to the JTReg test root of 252the tests to be run. 253 254 255#### OPTIONS 256Additional options to the JTReg test framework. 257 258Use `JTREG="OPTIONS=--help all"` to see all available JTReg options. 259 260#### JAVA_OPTIONS 261Additional Java options to JTReg (`-javaoption`). 262 263#### VM_OPTIONS 264Additional VM options to JTReg (`-vmoption`). 265 266#### AOT_MODULES 267 268Generate AOT modules before testing for the specified module, or set of 269modules. If multiple modules are specified, they should be separated by space 270(or, to help avoid quoting issues, the special value `%20`). 271 272### Gtest keywords 273 274#### REPEAT 275The number of times to repeat the tests (`--gtest_repeat`). 276 277Default is 1. Set to -1 to repeat indefinitely. This can be especially useful 278combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent 279problem. 280 281#### OPTIONS 282Additional options to the Gtest test framework. 283 284Use `GTEST="OPTIONS=--help"` to see all available Gtest options. 285 286#### AOT_MODULES 287 288Generate AOT modules before testing for the specified module, or set of 289modules. If multiple modules are specified, they should be separated by space 290(or, to help avoid quoting issues, the special value `%20`). 291 292## Notes for Specific Tests 293 294### Docker Tests 295 296Docker tests with default parameters may fail on systems with glibc versions not 297compatible with the one used in the default docker image (e.g., Oracle Linux 7.6 for x86). 298For example, they pass on Ubuntu 16.04 but fail on Ubuntu 18.04 if run like this on x86: 299 300 $ make run-test TEST="jtreg:test/hotspot/jtreg/containers/docker" 301 302To run these tests correctly, additional parameters for the correct docker image are 303required on Ubuntu 18.04 by using `JAVA_OPTIONS`. 304 305 $ make run-test TEST="jtreg:test/hotspot/jtreg/containers/docker" JTREG="JAVA_OPTIONS=-Djdk.test.docker.image.name=ubuntu -Djdk.test.docker.image.version=latest" 306 307### Client UI Tests 308 309Some Client UI tests use key sequences which may be reserved by the operating 310system. Usually that causes the test failure. So it is highly recommended to disable 311system key shortcuts prior testing. The steps to access and disable system key shortcuts 312for various platforms are provided below. 313 314#### MacOS 315Choose Apple menu; System Preferences, click Keyboard, then click Shortcuts; 316select or deselect desired shortcut. 317 318For example, test/jdk/javax/swing/TooltipManager/JMenuItemToolTipKeyBindingsTest/JMenuItemToolTipKeyBindingsTest.java fails 319on MacOS because it uses `CTRL + F1` key sequence to show or hide tooltip message 320but the key combination is reserved by the operating system. To run the test correctly 321the default global key shortcut should be disabled using the steps described above, and then deselect 322"Turn keyboard access on or off" option which is responsible for `CTRL + F1` combination. 323 324#### Linux 325Open the Activities overview and start typing Settings; Choose Settings, click Devices, 326then click Keyboard; set or override desired shortcut. 327 328#### Windows 329Type `gpedit` in the Search and then click Edit group policy; navigate to 330User Configuration -> Administrative Templates -> Windows Components -> File Explorer; 331in the right-side pane look for "Turn off Windows key hotkeys" and double click on it; 332enable or disable hotkeys. 333 334Note: restart is required to make the settings take effect. 335 336--- 337# Override some definitions in the global css file that are not optimal for 338# this document. 339header-includes: 340 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>' 341--- 342