1################## 2Raptor Browsertime 3################## 4 5.. contents:: 6 :depth: 2 7 :local: 8 9Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and Firefox for Android and GeckoView-based vehicles. 10 11Source code: 12 13- Our current Browsertime version uses the canonical repo. 14- In-tree: https://searchfox.org/mozilla-central/source/tools/browsertime and https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/browsertime.sh 15 16Running Locally 17--------------- 18 19**Prerequisites** 20 21- A local mozilla repository clone with a `successful Firefox build <https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions>`_ completed 22 23Setup 24----- 25 26Note that if you are running Raptor-Browsertime then it will get installed automatically and also updates itself. 27 28- ``./mach browsertime --clobber --setup --install-vismet-reqs`` 29 30This will automatically check your setup, which will output something like this: 31 32:: 33 34 ffmpeg: OK 35 convert: OK 36 compare: OK 37 Pillow: OK 38 SSIM: OK 39 40- To manually check your setup, run ``./mach browsertime --check`` 41 42Known Issues 43^^^^^^^^^^^^ 44 45**If you aren't running visual metrics, then failures in** ``Pillow`` **and** ``SSIM`` **can be ignored.** 46 47`Bug 1735410: [meta] browsertime visual metrics dependencies not installing correctly <https://bugzilla.mozilla.org/show_bug.cgi?id=1735410>`_ 48 49Currently there are issues on all platforms installing browsertime vismet dependencies. There is a fix for Linux (`Bug 1746208 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746208>`__) but not on Windows (`Bug 1746206 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746206>`__) or OSX (`Bug 1746207 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746207>`__) 50 51Linux 52""""" 53`Bug 1746208 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746208>`__ **(resolved)** 54 55If ``ffmpeg`` is listed as FAIL, try `downloading ffmpeg manually <https://ffmpeg.org/>`_ and adding it to your PATH 56 57OSX 58""" 59 60`Bug 1746207 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746207>`__ **(unresolved)** 61 62**Current Status**: ``convert`` and ``compare`` fail to install. Rebuilding Firefox and running browsertime setup has not shown to resolve this issue. 63 64Windows 65""""""" 66 67`Bug 1746206 <https://bugzilla.mozilla.org/show_bug.cgi?id=1746206>`__ **(unresolved)** 68 69If the ImageMagick URL returns a 404 during setup, please `file a bug like this <https://bugzilla.mozilla.org/show_bug.cgi?id=1735540>_` to have the URL updated. 70 71**Current Status**: ``convert``, ``compare``, and ``ffmpeg`` fail to install. Neither adding ``ffmpeg`` to the PATH, nor rebuilding Firefox have shown to resolve this issue. 72 73 74- For other issues, try deleting the ``~/.mozbuild/browsertime`` folder and re-running the browsertime setup command. 75 76- If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the :ref: `Running on Android` section) 77 78- **If you encounter any issues not mentioned here, please** `file a bug <https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Raptor>`_ **in the** ``Testing::Raptor`` **component.** 79 80 81Running on Firefox Desktop 82-------------------------- 83 84Page-load tests 85--------------- 86There are two ways to run performance tests through browsertime listed below. 87 88**Note that** ``./mach browsertime`` **should not be used when debugging performance issues with profiles as it does not do symbolication.** 89 90* Raptor-Browsertime (recommended): 91 92:: 93 94 ./mach raptor --browsertime -t google-search 95 96* Browsertime-"native": 97 98:: 99 100 ./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/{userdir}/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox' 101 102Benchmark tests 103--------------- 104* Raptor-wrapped: 105 106:: 107 108 ./mach raptor -t raptor-speedometer --browsertime 109 110Running on Android 111------------------ 112Running on Raptor-Browsertime (recommended): 113 114* Running on Fenix 115 116:: 117 118 ./mach raptor --browsertime -t amazon --app fenix --binary org.mozilla.fenix 119 120* Running on Geckoview 121 122:: 123 124 ./mach raptor --browsertime -t amazon --app geckoview --binary org.mozilla.geckoview_example 125 126Running on vanilla Browsertime: 127 128* Running on Fenix/Firefox Preview 129 130:: 131 132 ./mach browsertime --android --browser firefox --firefox.android.package org.mozilla.fenix.debug --firefox.android.activity org.mozilla.fenix.IntentReceiverActivity https://www.sitespeed.io 133 134* Running on the GeckoView Example app 135 136:: 137 138 ./mach browsertime --android --browser firefox https://www.sitespeed.io 139 140Running on Google Chrome 141------------------------ 142Chrome releases are tied to a specific version of ChromeDriver -- you will need to ensure the two are aligned. 143 144There are two ways of doing this: 145 1461. Download the ChromeDriver that matches the chrome you wish to run from https://chromedriver.chromium.org/ and specify the path: 147 148:: 149 150 ./mach browsertime https://www.sitespeed.io -b chrome --chrome.chromedriverPath <PATH/TO/VERSIONED/CHROMEDRIVER> 151 1522. Upgrade the ChromeDriver version in ``tools/browsertime/package-lock.json`` (see https://www.npmjs.com/package/@sitespeed.io/chromedriver for versions). 153 154Run ``npm install``. 155 156Launch vanilla Browsertime as follows: 157 158:: 159 160 ./mach browsertime https://www.sitespeed.io -b chrome 161 162Or for Raptor-Browsertime (use ``chrome`` for desktop, and ``chrome-m`` for mobile): 163 164:: 165 166 ./mach raptor --browsertime -t amazon --app chrome --browsertime-chromedriver <PATH/TO/CHROMEDRIVER> 167 168More Examples 169------------- 170 171`Browsertime docs <https://github.com/mozilla/browsertime/tree/master/docs/examples>`_ 172 173Passing Additional Arguments to Browertime 174------------------------------------------ 175 176Browsertime has many command line flags to configure its usage, see `Browsertime configuration <https://www.sitespeed.io/documentation/browsertime/configuration/>`_. 177 178We do not currently support passing additional arguments, this work can be tracked in `Bug 1750976 <https://bugzilla.mozilla.org/show_bug.cgi?id=1750976>`_. 179 180There are two options to work around this and enable additional arguments. 181 1821. Define additional arguments in `testing/raptor/raptor/browsertime/base.py <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/browsertime/base.py#220-252>`_. 183 1842. Add a ``browsertime_args`` entry to the appropriate manifest with the desired arguments, i.e. `browsertime-tp6.ini <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/tp6/desktop/browsertime-tp6.ini>`_ for desktop page load tests. `Example of browsertime_args format <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/custom/browsertime-process-switch.ini#27>`_. 185 186 187Running Browsertime on Try 188-------------------------- 189You can run all of our browsertime pageload tests through ``./mach try fuzzy --full``. We use chimera mode in these tests which means that both cold and warm pageload variants are running at the same time. 190 191For example: 192 193:: 194 195 ./mach try fuzzy -q "'g5 'imdb 'geckoview 'vismet '-wr 'shippable" 196 197Retriggering Browsertime Visual Metrics Tasks 198--------------------------------------------- 199 200You can retrigger Browsertime tasks just like you retrigger any other tasks from Treeherder (using the retrigger buttons, add-new-jobs, retrigger-multiple, etc.). 201 202When you retrigger the Browsertime test task, it will trigger a new vismet task as well. If you retrigger a Browsertime vismet task, then it will cause the test task to be retriggered and a new vismet task will be produced from there. This means that both of these tasks are treated as "one" task when it comes to retriggering them. 203 204There is only one path that still doesn't work for retriggering Browsertime tests and that happens when you use ``--rebuild X`` in a try push submission. 205 206For details on how we previously retriggered visual metrics tasks see `VisualMetrics <https://wiki.mozilla.org/TestEngineering/Performance/Raptor/VisualMetrics>`_ (this will stay here for a few months just in case). 207 208Gecko Profiling with Browsertime 209-------------------------------- 210 211To run gecko profiling using Raptor-Browsertime you can add the ``--gecko-profile`` flag to any command and you will get profiles from the test (with the profiler page opening in the browser automatically). This method also performs symbolication for you. For example: 212 213:: 214 215 ./mach raptor --browsertime -t amazon --gecko-profile 216 217Note that vanilla Browsertime does support Gecko Profiling but **it does not symbolicate the profiles** so it is **not recommended** to use for debugging performance regressions/improvements. 218 219Upgrading Browsertime In-Tree 220----------------------------- 221To upgrade the browsertime version used in-tree you can run, then commit the changes made to ``package.json`` and ``package-lock.json``: 222 223:: 224 225 ./mach browsertime --update-upstream-url <TARBALL-URL> 226 227Here is a sample URL that we can update to: https://github.com/sitespeedio/browsertime/tarball/89771a1d6be54114db190427dbc281582cba3d47 228 229To test the upgrade, run a raptor test locally (with and without visual-metrics ``--browsertime-visualmetrics`` if possible) and test it on try with at least one test on desktop and mobile. 230 231Finding the Geckodriver Being Used 232---------------------------------- 233If you're looking for the latest geckodriver being used there are two ways: 234* Find the latest one from here: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=geckodriver 235* Alternatively, if you're trying to figure out which geckodriver a given CI task is using, you can click on the browsertime task in treeherder, and then click on the ``Task`` id in the bottom left of the pop-up interface. Then in the window that opens up, click on `See more` in the task details tab on the left, this will show you the dependent tasks with the latest toolchain-geckodriver being used. There's an Artifacts drop down on the right hand side for the toolchain-geckodriver task that you can find the latest geckodriver in. 236 237If you're trying to test Browsertime with a new geckodriver, you can do either of the following: 238* Request a new geckodriver build in your try run (i.e. through ``./mach try fuzzy``). 239* Trigger a new geckodriver in a try push, then trigger the browsertime tests which will then use the newly built version in the try push. 240 241Comparing Before/After Browsertime Videos 242----------------------------------------- 243 244We have some scripts that can produce side-by-side comparison videos for you of the worst pairing of videos. You can find the script here: https://github.com/mozilla/mozperftest-tools#browsertime-side-by-side-video-comparisons 245 246Once the side-by-side comparison is produced, the video on the left is the old/base video, and the video on the right is the new video. 247