1# Testing 2 3Given the multiple API languages, processes, and dependencies, 4testing FOG is a matter of choosing the right tool for the situation. 5 6## One Big Command 7 8To run all the things, here's the tl;dr: 9 10`MOZ_LOG="timestamp,sync,glean::*:5,fog::*:5,fog_control::*:5,glean_core::*:5" 11./mach build && ./mach lint -Ww -o --fix 12&& ./mach lint --linter clippy toolkit/components/glean/api/src 13&& ./mach rusttests && ./mach gtest FOG* 14&& python3 ./mach python-test toolkit/components/glean/pytest 15&& ./mach test toolkit/components/glean/xpcshell 16&& ./mach telemetry-tests-client toolkit/components/telemetry/tests/marionette/tests/client/test_fog* --gecko-log "-"` 17 18## Logging 19 20An often-overlooked first line of testing is "what do the logs say?". 21To turn on logging for FOG, use any of the following: 22* Run Firefox with `RUST_LOG="fog_control,fog,glean_core"`. 23 * On some platforms this will use terminal colours to indicate log level. 24* Run Firefox with `MOZ_LOG="timestamp,sync,glean::*:5,fog::*:5,fog_control::*:5,glean_core::*:5"`. 25* Set the following prefs: 26 * `logging.config.timestamp` to `true` 27 * `logging.config.sync` to `true` 28 * `logging.fog_control::*` to `5` 29 * `logging.fog::*` to `5` 30 * `logging.glean::*` to `5` 31 * `logging.glean_core::*` to `5` 32 33For more information on logging in Firefox Desktop, see the 34[Gecko Logging docs](https://developer.mozilla.org/docs/Mozilla/Developer_guide/Gecko_Logging). 35 36## `about:glean` 37 38`about:glean` is a page in a running Firefox that allows you to 39[debug the Glean SDK](https://mozilla.github.io/glean/book/user/debugging/index.html) 40in Firefox Desktop. 41It does this through the displayed user interface (just follow the instructions). 42 43## Linting 44 45To keep in accordance with Mozilla's various and several Coding Styles, 46we rely on `mach lint`. 47 48To lint the code in the "usual" way, automatically fixing where possible, run: 49`./mach lint -Ww -o --fix` 50 51This should keep you from checking in code that will automatically be backed out. 52 53In addition, we need to run the Rust formatter `clippy` on the `fog` crate: 54`./mach lint --linter clippy toolkit/components/glean/api/src` 55 56This will ensure that clippy-only builds will have all the symbols they need to lint our code. 57 58## Rust 59 60Not all of our Rust code can be tested in a single fashion, unfortunately. 61 62### Using `rusttests` (Treeherder symbol `Br` (a build task)) 63 64If the crate you're testing has no Gecko symbols you can write standard 65[Rust tests](https://doc.rust-lang.org/book/ch11-01-writing-tests.html). 66 67This supports both unit tests 68(inline in the file under test) and integration tests 69(in the `tests/` folder in the crate root). 70Metric type tests are currently written as unit tests inline in the file, 71as they require access to the metric ID, which should only be exposed in tests. 72 73To run FOG's `rusttests` suite use `mach rusttests` 74 75If the crate uses only a few Gecko symbols, they may use the 76`with_gecko` feature to conditionally use them. 77This allows the crate to test its non-Gecko-adjacent code using Rust tests. 78(You will need to cover the Gecko-adjacent code via another means.) 79 80**Note:** Some FOG rusttests panic on purpose. They print stack traces to stdout. 81If the rusttests fail and you see a stack trace, 82double-check it isn't from a purposefully-panicking test. 83 84**Note:** If a test fails, it is very likely they'll poison the test lock. 85This will cause all subsequent tests that attempt to take the test lock 86(which is all of them) 87to also fail due to `PoisonError`s. They can be safely ignored. 88 89### Using `gtest` (Treeherder symbol `GTest` (a build task)) 90 91Because Gecko symbols aren't built for the 92`rusttests` build, 93any test that is written for code that uses Gecko symbols should be written as a 94[`gtest`](https://github.com/google/googletest) 95in `toolkit/components/glean/gtest/`. 96You can write the actual test code in Rust. 97It needs to be accompanied by a C++ GTest that calls a C FFI-exported Rust function. 98See [Testing & Debugging Rust Code](/testing-rust-code/) for more. 99See [`toolkit/components/glean/gtest/TestFog.cpp`](https://searchfox.org/mozilla-central/source/toolkit/components/glean/gtest/TestFog.cpp) 100and [`toolkit/components/glean/gtest/test.rs`](https://searchfox.org/mozilla-central/source/toolkit/components/glean/gtest/test.rs) 101for an example. 102 103By necessity these can only be integration tests against the compiled crate. 104 105**Note:** When adding a new test file, don't forget to add it to 106`toolkit/components/glean/gtest/moz.build` and use the 107`FOG` prefix in your test names 108(e.g. `TEST(FOG, YourTestName) { ... }`). 109 110To run FOG's Rust `gtest` suite use `mach gtest FOG.*` 111 112## Python (Treeherder symbol `py3(fp)` aka `source-test-python-fog`) 113 114The [Glean Parser](https://github.com/mozilla/glean_parser/) 115has been augmented to generate FOG-specific APIs for Glean metrics. 116This augmentation is tested by running: 117 118`mach test toolkit/components/glean/pytest` 119 120These tests require Python 3+. 121If your default Python is Python 2, you may need to instead run: 122 123`python3 mach python-test toolkit/components/glean/pytest` 124 125These tests check the code generator output against known good file contents. 126If you change the code generator the files will need an update. 127Run the test suite with the `UPDATE_EXPECT` environment variable set to do that automatically: 128 129`UPDATE_EXPECT=1 mach test toolkit/components/glean/pytest` 130 131## C++ (Treeherder symbol `GTest` (a build task)) 132 133To test the C++ parts of FOG's implementation 134(like metric types) 135you should use `gtest`. 136FOG's `gtest` tests are in 137[`gtest/`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/gtest/). 138 139You can either add a test case to an existing file or add a new file. 140If you add a new file, remember to add it to the 141[`moz.build`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/gtest/moz.build)) 142or the test runner won't be able to find it. 143 144All tests should start with `FOG` so that all tests are run with 145`./mach gtest FOG*`. 146 147## JS (Treeherder symbol `X(Xn)` for some number `n`) 148 149To test the JS parts of FOG's implementation 150(like metric types) 151you should use `xpcshell`. 152FOG's `xpcshell` tests are in 153[`xpcshell/`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/xpcshell). 154 155You can either add a test case to an existing file or add a new file. 156If you add a new file, remember to add it to the 157[`xpcshell.ini`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/glean/xpcshell/xpcshell.ini) 158or the test runner will not be able to find it. 159 160To run FOG's JS tests, run: 161`./mach test toolkit/components/glean/xpcshell` 162 163## Integration (Marionette, borrowing `telemetry-tests-client` Treeherder symbol `tt(c)`) 164 165To test pings (See [bug 1681742](https://bugzilla.mozilla.org/show_bug.cgi?id=1681742)) 166or anything that requires one or more full browsers running, 167we use the `telemetry-tests-client` suite in 168[`toolkit/components/telemetry/tests/marionette/tests/client/`](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/tests/marionette/tests/client/). 169 170For more information on this suite, look to 171[Firefox Telemetry's Test Documentation](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/internals/tests.html#integration-tests-telemetry-tests-client-and-telemetry-integration-tests) 172and 173[Marionette's Documentation](/testing/marionette/Testing.md). 174 175To run these integration tests, run: 176`./mach telemetry-tests-client toolkit/components/telemetry/tests/marionette/tests/client/` 177 178To capture the Firefox under test's logs, use the `--gecko-log` parameter. 179For example, to echo to stdout: 180`./mach telemetry-tests-client toolkit/components/telemetry/tests/marionette/tests/client/test_fog* --gecko-log "-"` 181 182**Note:** Running the `tt(c)` suite in this way ignored skip directives in the manifest. 183This means that you might run tests that are not expected to succeed on your platform. 184Check `toolkit/components/telemetry/tests/marionette/tests/client/manifest.ini` for details. 185