1---
2title: "testthat 3e"
3output: rmarkdown::html_vignette
4vignette: >
5  %\VignetteIndexEntry{testthat 3e}
6  %\VignetteEngine{knitr::rmarkdown}
7  %\VignetteEncoding{UTF-8}
8---
9
10```{r, include = FALSE}
11knitr::opts_chunk$set(
12  collapse = TRUE,
13  comment = "#>"
14)
15```
16
17testthat 3.0.0 introduces the idea of an "edition" of testthat.
18An edition is a bundle of behaviours that you have to explicitly choose to use, allowing us to make otherwise backward incompatible changes.
19This is particularly important for testthat since it has a very large number of packages that use it (almost 5,000 at last count).
20Choosing to use the 3rd edition allows you to use our latest recommendations for ongoing and new work, while historical packages continue to use the old behaviour.
21
22(We don't anticipate creating new editions very often, and they'll always be matched with major version, i.e. if there's another edition, it'll be the fourth edition and will come with testthat 4.0.0.)
23
24This vignette shows you how to activate the 3rd edition, introduces the main features, and discusses common challenges when upgrading a package.
25If you have a problem that this vignette doesn't cover, please let me know, as it's likely that the problem also affects others.
26
27```{r, message = FALSE}
28library(testthat)
29local_edition(3)
30```
31
32## Activating
33
34The usual way to activate the 3rd edition is to add a line to your `DESCRIPTION`:
35
36    Config/testthat/edition: 3
37
38This will activate the 3rd edition for every test in your package.
39
40You can also control the edition used for individual tests with `testthat::local_edition()`:
41
42```{r}
43test_that("I can use the 3rd edition", {
44  local_edition(3)
45  expect_true(TRUE)
46})
47```
48
49This is also useful if you've switched to the 3rd edition and have a couple of tests that fail.
50You can use `local_edition(2)` to revert back to the old behaviour, giving you some breathing room to figure out the underlying issue.
51
52```{r}
53test_that("I want to use the 2nd edition", {
54  local_edition(2)
55  expect_true(TRUE)
56})
57```
58
59## Changes
60
61There are three major changes in the 3rd edition:
62
63-   A number of outdated functions are now **deprecated**, so you'll be warned about them every time you run your tests (but they won't cause `R CMD check` to fail).
64
65-   testthat no longer silently swallows **messages**; you now need to deliberately handle them.
66
67-   `expect_equal()` and `expect_identical()` now use the [**waldo**](https://waldo.r-lib.org/) package instead of `identical()` and `all.equal()`.
68    This makes them more consistent and provides an enhanced display of differences when a test fails.
69
70### Deprecations
71
72A number of outdated functions have been deprecated.
73Most of these functions have not been recommended for a number of years, but before the introduction of the edition idea, I didn't have a good way of preventing people from using them without breaking a lot of code on CRAN.
74
75-   `context()` is formally deprecated.
76    testthat has been moving away from `context()` in favour of file names for quite some time, and now you'll be strongly encouraged remove these calls from your tests.
77
78-   `expect_is()` is deprecated in favour of the more specific `expect_type()`, `expect_s3_class()`, and `expect_s4_class()`.
79    This ensures that you check the expected class along with the expected OO system.
80
81-   The very old `expect_that()` syntax is now deprecated.
82    This was an overly clever API that I regretted even before the release of testthat 1.0.0.
83
84-   `expect_equivalent()` has been deprecated since it is now equivalent (HA HA) to `expect_equal(ignore_attr = TRUE)`.
85    The main difference is that it won't ignore names; so you'll need an explicit `unname()` if you deliberately want to ignore names.
86
87-   `setup()` and `teardown()` are deprecated in favour of test fixtures.
88    See `vignette("test-fixtures")` for details.
89
90-   `expect_known_output()`, `expect_known_value()`, `expect_known_hash()`, and `expect_equal_to_reference()` are all deprecated in favour of `expect_snapshot_output()` and `expect_snapshot_value()`.
91
92-   `with_mock()` and `local_mock()` are deprecated; please use the [mockr](https://krlmlr.github.io/mockr/) or [mockery](https://github.com/r-lib/mockery#mockery) instead.
93
94Fixing these deprecation warnings should be straightforward.
95
96### Warnings
97
98In the second edition, `expect_warning()` swallows all warnings regardless of whether or not they match the `regexp` or `class`:
99
100```{r}
101f <- function() {
102  warning("First warning")
103  warning("Second warning")
104  warning("Third warning")
105}
106
107local_edition(2)
108expect_warning(f(), "First")
109```
110
111In the third edition, `expect_warning()` captures at most one warning so the others will bubble up:
112
113```{r}
114local_edition(3)
115expect_warning(f(), "First")
116```
117
118You can either add additional expectations to catch these warnings, or silence them all with `supressWarnings()`:
119
120```{r}
121f() %>%
122  expect_warning("First") %>%
123  expect_warning("Second") %>%
124  expect_warning("Third")
125
126f() %>%
127  expect_warning("First") %>%
128  suppressWarnings()
129```
130
131Alternatively, you might want to capture them all in a snapshot test:
132
133```{r}
134test_that("f() produces expected outputs/messages/warnings", {
135  expect_snapshot(f())
136})
137```
138
139The same principle also applies to `expect_message()`, but message handling has changed in a more radical way, as described next.
140
141### Messages
142
143For reasons that I can no longer remember, testthat silently ignores all messages.
144This is inconsistent with other types of output, so as of the 3rd edition, they now bubble up to your test results.
145You'll have to explicit ignore them with `supressMesssages()`, or if they're important, test for their presence with `expect_message()`.
146
147### waldo
148
149Probably the biggest day-to-day difference (and the biggest reason to upgrade!) is the use of [`waldo::compare()`](https://waldo.r-lib.org/reference/compare.html) inside of `expect_equal()` and `expect_identical()`.
150The goal of waldo is to find and concisely describe the difference between a pair of R objects, and it's designed specifically to help you figure out what's gone wrong in your unit tests.
151
152```{r, error = TRUE}
153f1 <- factor(letters[1:3])
154f2 <- ordered(letters[1:3], levels = letters[1:4])
155
156local_edition(2)
157expect_equal(f1, f2)
158
159local_edition(3)
160expect_equal(f1, f2)
161```
162
163waldo looks even better in your console because it carefully uses colours to help highlight the differences.
164
165The use of waldo also makes precise the difference between `expect_equal()` and `expect_identical()`: `expect_equal()` sets `tolerance` so that waldo will ignore small numerical differences arising from floating point computation.
166Otherwise the functions are identical (HA HA).
167
168This change is likely to result in the most work during an upgrade, because waldo can give slightly different results to both `identical()` and `all.equal()` in moderately common situations.
169I believe on the whole the differences are meaningful and useful, so you'll need to handle them by tweaking your tests.
170The following changes are most likely to affect you:
171
172-   `expect_equal()` previously ignored the environments of formulas and functions.
173    This is most like to arise if you are testing models.
174    It's worth thinking about what the correct values should be, but if that is to annoying you can opt out of the comparison with `ignore_function_env` or `ignore_formula_env`.
175
176-   `expect_equal()` used a combination of `all.equal()` and a home-grown `testthat::compare()` which unfortunately used a slightly different definition of tolerance.
177    Now `expect_equal()` always uses the same defintion of tolerance everywhere, which may require tweaks to your exising tolerance values.
178
179-   `expect_equal()` previously ignored timezone differences when one object had the current timezone set implicitly (with `""`) and the other had it set explictly:
180
181    ```{r, error = TRUE}
182    dt1 <- dt2 <- ISOdatetime(2020, 1, 2, 3, 4, 0)
183    attr(dt1, "tzone") <- ""
184    attr(dt2, "tzone") <- Sys.timezone()
185
186    local_edition(2)
187    expect_equal(dt1, dt2)
188
189    local_edition(3)
190    expect_equal(dt1, dt2)
191    ```
192
193### Reproducible output
194
195In the third edition, `test_that()` automatically calls `local_reproducible_output()` which automatically sets a number of options and environment variables to ensure output is as reproducible across systems.
196This includes setting:
197
198-   `options(crayon.enabled = FALSE)` and `options(cli.unicode = FALSE)` so that the crayon and cli packages produce raw ASCII output.
199
200-   `Sys.setLocale("LC_COLLATE" = "C")` so that sorting a character vector returns the same order regardless of the system language.
201
202-   `options(width = 80)` so print methods always generate the same output regardless of your actual console width.
203
204See the documentation for more details.
205
206## Alternatives
207
208You might wonder why we came up with the idea of an "edition", rather than creating a new package like testthat3.
209We decided against making a new package because the 2nd and 3rd edition share a very large amount of code, so making a new package would have substantially increased the maintenance burden: the majority of bugs would've needed to be fixed in two places.
210
211If you're a programmer in other languages, you might wonder why we can't rely on [semantic versioning](https://semver.org).
212The main reason is that CRAN checks all packages that use testthat with the latest version of testthat, so simply incrementing the major version number doesn't actually help with reducing R CMD check failures on CRAN.
213