README.md
1# Guide on end-to-end testing
2
3This directory hosts the end-to-end tests we run on DevTools.
4These tests open a target page and a DevTools frontend page, for which the DevTools frontend connects to the target page over CDP.
5We use [Puppeteer] to talk over CDP and all functionality of Puppeteer is available to you as well when writing end-to-end tests.
6We use [Mocha] as testing framework.
7
8The goal of these end-to-end tests is to implement core user journeys throughout the application.
9As such, the tests you write should read like a little story that you can read, even if you don't know how it is implemented.
10
11The tests therefore have a dual purpose:
121. Verify that core user stories are working as intended and are not broken by a particular DevTools frontend change.
131. Serve as documentation and reference point for how DevTools is intended to be used.
14
15## Running tests
16All tests: `npm run e2etest` (note, this requires python2 to be the default python binary!)
17Some additional, optional, helpful flags:
18`npm run e2etest -- --chrome-binary=[LOCATION] --chrome-features=[FEATURES]`
19LOCATION is a path to the chrome executable
20FEATURES is a comma separated list of chrome features passed as `--enable-features=[FEATURES]` to the chrome binary.
21
22If you only want to run a single test or testsuite, use respectively `it.only` or `describe.only`.
23
24You can also run all tests in one file with the `--test-file` option, e.g.
25`npm run e2etest -- --test-file=console/console-clear_test`. The path is relative to the generated test/e2e/
26directory. File extensions are not needed and are ignored.
27
28## Debugging tests
29To see what the test script does, run `npm run debug-e2etest`. This will bring up the chrome window and stop just
30before your test script is about to execute. The test will then run to completion and exit. You can add an infinite
31await `await new Promise(() => {});` at the end of your test to give you some time to examine the result of your
32test script.
33
34The `it.repeat` helper is useful for reproducing a flaky test failure. e.g.
35
36```js
37it.repeat(20, 'find element', async () => {...});
38```
39
40`it.repeat` behaves like `it.only` in that it will cause just that single test to be run.
41
42## General implementation details
43
44To that end, the "what" from the "how" are separate in end-to-end tests.
45The "what" is the actual end-to-end test.
46The "how" are functions in [helpers](helpers/) that implement the interaction with DevTools to perform a particular action.
47
48For example, an end-to-end test might read like this:
49
50```js
51it('can show newly created snippets show up in command menu', async () => {
52 await openSourcesPanel();
53 await openSnippetsSubPane();
54 await createNewSnippet('New snippet');
55
56 await openCommandMenu();
57 await showSnippetsAutocompletion();
58
59 assert.deepEqual(await getAvailableSnippets(), [
60 'New snippet\u200B',
61 ]);
62});
63```
64
65The test describes the user journey and then describes what actions the user takes to fulfill that journey.
66For example, when the user wants to "open the command menu", that is an action performed.
67The implementation is separated in a helper that implements "how" the user performs that action.
68
69The separation of the two concepts allows us to change the underlying implementation of the action while making sure that the user journey remains intact.
70
71For example, this is the implementation of `openCommandMenu()`:
72
73```js
74export const openCommandMenu = async () => {
75 const {frontend} = getBrowserAndPages();
76
77 switch (platform) {
78 case 'mac':
79 await frontend.keyboard.down('Meta');
80 await frontend.keyboard.down('Shift');
81 break;
82
83 case 'linux':
84 case 'win32':
85 await frontend.keyboard.down('Control');
86 await frontend.keyboard.down('Shift');
87 break;
88 }
89
90 await frontend.keyboard.press('P');
91
92 switch (platform) {
93 case 'mac':
94 await frontend.keyboard.up('Meta');
95 await frontend.keyboard.up('Shift');
96 break;
97
98 case 'linux':
99 case 'win32':
100 await frontend.keyboard.up('Control');
101 await frontend.keyboard.up('Shift');
102 break;
103 }
104
105 await waitFor(QUICK_OPEN_SELECTOR);
106};
107```
108
109As you can see, the way the user opens the command menu is via key-bindings.
110We don't "bypass" the functionality by calling functions on components or on our models directly; we instruct Puppeteer to do exactly what a user would do.
111Doing so, we are certain that we don't test our component abstractions and potentially lose track of integration issues.
112
113Secondly, the function has a `waitFor`, which waits for the command menu (found by the `QUICK_OPEN_SELECTOR`) to appear.
114For every action that is performed in DevTools, there must be a corresponding user-visible change in the UI.
115This means that you always have to wait for something to happen and you can't assume that, as soon as you have performed an action, the UI has updated accordingly.
116
117**Note: Because of the async rendering of DevTools, content might not be strictly visible when DOM Nodes are appended to the DOM.**
118As such, be aware of the functionality you are testing and relying on, as it could render differently than you originally assumed.
119
120To summarize:
1211. Separate the "what" from the "how".
1221. Use real actions (clicking, using key-bindings, typing) instead of "bypassing" via components/models.
1231. Every action must be observed by a change in the UI and must be waited for.
1241. Be aware of the async rendering of DevTools
125
126## Helpers
127
128There are two kinds of helpers:
1291. Helpers written as part of the end-to-end test, implementing the "how" of user actions.
1301. Helpers supplied to interact with a page and abstract away the way the tests are run.
131
132The former are implemented in [helpers](helpers/), written by the DevTools maintainers and are specific to the implementation of the DevTools frontend.
133The latter are implemented in [../shared](../shared/), written by the Puppeteer maintainers and are predominantly DevTools-agnostic (apart from the handling of a target page + CDP connection).
134
135In general, the e2e/helpers make use of the shared helpers.
136See [../shared/README.md](../shared/README.md) for more documentation on the shared helpers.
137
138[Puppeteer]: https://pptr.dev/
139[Mocha]: https://mochajs.org
140