1######### 2Debugging 3######### 4 5.. contents:: 6 :depth: 2 7 :local: 8 9Debugging Desktop Product Failures 10********************************** 11 12As of now, there is no easy way to do this. Raptor was not built for debugging functional failures. Hitting these in Raptor is indicative that we lack functional test coverage so regression tests should be added for those failures after they are fixed. 13 14To debug a functional failure in Raptor you can follow these steps: 15 16#. If bug 1653617 has not landed yet, apply the patch. 17#. Add the --verbose flag to the extra-options list `here <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml#98-101>`__. 18#. If the --setenv doesn't exist yet (`bug 1494669 <https://bugzilla.mozilla.org/show_bug.cgi?id=1494669>`_), then add your MOZ_LOG environment variables to give you additional logging `here <https://searchfox.org/mozilla-central/source/testing/raptor/raptor/webextension/desktop.py#42>`_. 19#. If the flag does exist, then you can add the MOZ_LOG variables to the `raptor.yml <https://searchfox.org/mozilla-central/source/taskcluster/ci/test/raptor.yml>`_ configuration file. 20#. Push to try if you can't reproduce the failure locally. 21 22You can follow `bug 1655554 <https://bugzilla.mozilla.org/show_bug.cgi?id=1655554>`_ as we work on improving this workflow. 23 24In some cases, you might not be able to get logging for what you are debugging (browser console logging for instance). In this case, you should make your own debug prints with printf or something along those lines (`see :agi's debugging work for an example <https://matrix.to/#/!LfXZSWEroPFPMQcYmw:mozilla.org/$r_azj7OipkgDzQ75SCns2QIayp4260PIMHLWLApJJNg?via=mozilla.org&via=matrix.org&via=rduce.org>`_). 25 26Debugging the Raptor Web Extension 27********************************** 28 29When developing on Raptor and debugging, there's often a need to look at the output coming from the `Raptor Web Extension <https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor>`_. Here are some pointers to help. 30 31Raptor Debug Mode 32----------------- 33 34The easiest way to debug the Raptor web extension is to run the Raptor test locally and invoke debug mode, i.e. for Firefox: 35 36:: 37 38 ./mach raptor --test raptor-tp6-amazon-firefox --debug-mode 39 40Or on Chrome, for example: 41 42:: 43 44 ./mach raptor --test raptor-tp6-amazon-chrome --app=chrome --binary="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --debug-mode 45 46Running Raptor with debug mode will: 47 48* Automatically set the number of test page-cycles to 2 maximum 49* Reduce the 30 second post-browser startup delay from 30 seconds to 3 seconds 50* On Firefox, the devtools browser console will automatically open, where you can view all of the console log messages generated by the Raptor web extension 51* On Chrome, the devtools console will automatically open 52* The browser will remain open after the Raptor test has finished; you will be prompted in the terminal to manually shutdown the browser when you're finished debugging. 53 54Manual Debugging on Firefox Desktop 55----------------------------------- 56 57The main Raptor runner is '`runner.js <https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor/runner.js>`_' which is inside the web extension. The code that actually captures the performance measures is in the web extension content code '`measure.js <https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor/measure.js>`_'. 58 59In order to retrieve the console.log() output from the Raptor runner, do the following: 60 61#. Invoke Raptor locally via ``./mach raptor`` 62#. During the 30 second Raptor pause which happens right after Firefox has started up, in the ALREADY OPEN current tab, type "about:debugging" for the URL. 63#. On the debugging page that appears, make sure "Add-ons" is selected on the left (default). 64#. Turn ON the "Enable add-on debugging" check-box 65#. Then scroll down the page until you see the Raptor web extension in the list of currently-loaded add-ons. Under "Raptor" click the blue "Debug" link. 66#. A new window will open in a minute, and click the "console" tab 67 68To retrieve the console.log() output from the Raptor content 'measure.js' code: 69 70#. As soon as Raptor opens the new test tab (and the test starts running / or the page starts loading), in Firefox just choose "Tools => Web Developer => Web Console", and select the "console' tab. 71 72Raptor automatically closes the test tab and the entire browser after test completion; which will close any open debug consoles. In order to have more time to review the console logs, Raptor can be temporarily hacked locally in order to prevent the test tab and browser from being closed. Currently this must be done manually, as follows: 73 74#. In the Raptor web extension runner, comment out the line that closes the test tab in the test clean-up. That line of `code is here <https://searchfox.org/mozilla-central/rev/3c85ea2f8700ab17e38b82d77cd44644b4dae703/testing/raptor/webext/raptor/runner.js#357>`_. 75#. Add a return statement at the top of the Raptor control server method that shuts-down the browser, the browser shut-down `method is here <https://searchfox.org/mozilla-central/rev/924e3d96d81a40d2f0eec1db5f74fc6594337128/testing/raptor/raptor/control_server.py#120>`_. 76 77For **benchmark type tests** (i.e. speedometer, motionmark, etc.) Raptor doesn't inject 'measure.js' into the test page content; instead it injects '`benchmark-relay.js <https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor/benchmark-relay.js>`_' into the benchmark test content. Benchmark-relay is as it sounds; it basically relays the test results coming from the benchmark test, to the Raptor web extension runner. Viewing the console.log() output from benchmark-relay is done the same was as noted for the 'measure.js' content above. 78 79Note, `Bug 1470450 <https://bugzilla.mozilla.org/show_bug.cgi?id=1470450>`_ is on file to add a debug mode to Raptor that will automatically grab the web extension console output and dump it to the terminal (if possible) that will make debugging much easier. 80 81Debugging TP6 and Killing the Mitmproxy Server 82---------------------------------------------- 83 84Regarding debugging Raptor pageload tests that use Mitmproxy (i.e. tp6, gdocs). If Raptor doesn't finish naturally and doesn't stop the Mitmproxy tool, the next time you attempt to run Raptor it might fail out with this error: 85 86:: 87 88 INFO - Error starting proxy server: OSError(48, 'Address already in use') 89 NFO - raptor-mitmproxy Aborting: mitmproxy playback process failed to start, poll returned: 1 90 91That just means the Mitmproxy server was already running before so it couldn't startup. In this case, you need to kill the Mitmproxy server processes, i.e: 92 93:: 94 95 mozilla-unified rwood$ ps -ax | grep mitm 96 5439 ttys000 0:00.09 /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/mitmdump -k -q -s /Users/rwood/mozilla-unified/testing/raptor/raptor/playback/alternate-server-replay.py /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/amazon.mp 97 440 ttys000 0:01.64 /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/mitmdump -k -q -s /Users/rwood/mozilla-unified/testing/raptor/raptor/playback/alternate-server-replay.py /Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.7.0/testing/raptor/amazon.mp 98 5509 ttys000 0:00.01 grep mitm 99 100Then just kill the first mitm process in the list and that's sufficient: 101 102:: 103 104 mozilla-unified rwood$ kill 5439 105 106Now when you run Raptor again, the Mitmproxy server will be able to start. 107 108Manual Debugging on Firefox Android 109----------------------------------- 110 111Be sure to read the above section first on how to debug the Raptor web extension when running on Firefox Desktop. 112 113When running Raptor tests on Firefox on Android (i.e. geckoview), to see the console.log() output from the Raptor web extension, do the following: 114 115#. With your android device (i.e. Google Pixel 2) all set up and connected to USB, invoke the Raptor test normally via ``./mach raptor`` 116#. Start up a local copy of the Firefox Nightly Desktop browser 117#. In Firefox Desktop choose "Tools => Web Developer => WebIDE" 118#. In the Firefox WebIDE dialog that appears, look under "USB Devices" listed on the top right. If your device is not there, there may be a link to install remote device tools - if that link appears click it and let that install. 119#. Under "USB Devices" on the top right your android device should be listed (i.e. "Firefox Custom on Android Pixel 2" - click on your device. 120#. The debugger opens. On the left side click on "Main Process", and click the "console" tab below - and the Raptor runner output will be included there. 121#. On the left side under "Tabs" you'll also see an option for the active tab/page; select that and the Raptor content console.log() output should be included there. 122 123Also note: When debugging Raptor on Android, the 'adb logcat' is very useful. More specifically for 'geckoview', the output (including for Raptor) is prefixed with "GeckoConsole" - so this command is very handy: 124 125:: 126 127 adb logcat | grep GeckoConsole 128 129Manual Debugging on Google Chrome 130--------------------------------- 131 132Same as on Firefox desktop above, but use the Google Chrome console: View ==> Developer ==> Developer Tools. 133