1Extensions & Experiments 2======================== 3 4This document describes address bar extensions and experiments: what they are, 5how to run them, how to write them, and the processes involved in each. 6 7The primary purpose right now for writing address bar extensions is to run 8address bar experiments. But extensions are useful outside of experiments, and 9not all experiments use extensions. 10 11Like all Firefox extensions, address bar extensions use the WebExtensions_ 12framework. 13 14.. _WebExtensions: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions 15 16.. toctree:: 17 :caption: Table of Contents 18 19 experiments 20 21WebExtensions 22------------- 23 24**WebExtensions** is the name of Firefox's extension architecture. The "web" 25part of the name hints at the fact that Firefox extensions are built using Web 26technologies: JavaScript, HTML, CSS, and to a certain extent the DOM. 27 28Individual extensions themselves often are referred to as *WebExtensions*. For 29clarity and conciseness, this document will refer to WebExtensions as 30*extensions*. 31 32Why are we interested in extensions? Mainly because they're a powerful way to 33run experiments in Firefox. See Experiments_ for more on that. In addition, we'd 34also like to build up a robust set of APIs useful to extension authors, although 35right now the API can only be used by Mozilla extensions. 36 37WebExtensions are introduced and discussed in detail on `MDN 38<WebExtensions_>`__. You'll need a lot of that knowledge in order to build 39address bar extensions. 40 41Developing Address Bar Extensions 42--------------------------------- 43 44Overview 45~~~~~~~~ 46 47The address bar WebExtensions API currently lives in two API namespaces, 48``browser.urlbar`` and ``browser.experiments.urlbar``. The reason for this is 49historical and is discussed in the `Developing Address Bar Extension APIs`_ 50section. As a consumer of the API, there are only two important things you need 51to know: 52 53* There's no meaningful difference between the APIs of the two namespaces. 54 Their kinds of functions, events, and properties are similar. You should 55 think of the address bar API as one single API that happens to be split into 56 two namespaces. 57 58* However, there is a big difference between the two when it comes to setting up 59 your extension to use them. This is discussed next. 60 61The ``browser.urlbar`` API namespace is built into Firefox. It's a 62**privileged API**, which means that only Mozilla-signed and temporarily 63installed extensions can use it. The only thing your Mozilla extension needs to 64do in order to use it is to request the ``urlbar`` permission in its 65manifest.json, as illustrated `here <urlbarPermissionExample_>`__. 66 67In contrast, the ``browser.experiments.urlbar`` API namespace is bundled inside 68your extension. APIs that are bundled inside extensions are called 69**experimental APIs**, and the extensions in which they're bundled are called 70**WebExtension experiments**. As with privileged APIs, experimental APIs are 71available only to Mozilla-signed and temporarily installed extensions. 72("WebExtension experiments" is a term of art and shouldn't be confused with the 73general notion of experiments that happen to use extensions.) For more on 74experimental APIs and WebExtension experiments, see the `WebExtensions API 75implementation documentation <webextAPIImplBasicsDoc_>`__. 76 77Since ``browser.experiments.urlbar`` is bundled inside your extension, you'll 78need to include it in your extension's repo by doing the following: 79 801. The implementation consists of two files, api.js_ and schema.json_. In your 81 extension repo, create a *experiments/urlbar* subdirectory and copy the 82 files there. See `this repo`__ for an example. 83 842. Add the following ``experiment_apis`` key to your manifest.json (see here__ 85 for an example in context):: 86 87 "experiment_apis": { 88 "experiments_urlbar": { 89 "schema": "experiments/urlbar/schema.json", 90 "parent": { 91 "scopes": ["addon_parent"], 92 "paths": [["experiments", "urlbar"]], 93 "script": "experiments/urlbar/api.js" 94 } 95 } 96 } 97 98As mentioned, only Mozilla-signed and temporarily installed extensions can use 99these two API namespaces. For information on running the extensions you develop 100that use these namespaces, see `Running Address Bar Extensions`_. 101 102.. _urlbarPermissionExample: https://github.com/0c0w3/urlbar-top-sites-experiment/blob/ac1517118bb7ee165fb9989834514b1082575c10/src/manifest.json#L24 103.. _webextAPIImplBasicsDoc: https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/basics.html 104.. _api.js: https://searchfox.org/mozilla-central/source/browser/components/urlbar/tests/ext/api.js 105.. _schema.json: https://searchfox.org/mozilla-central/source/browser/components/urlbar/tests/ext/schema.json 106__ https://github.com/0c0w3/dynamic-result-type-extension/tree/master/src/experiments/urlbar 107__ https://github.com/0c0w3/dynamic-result-type-extension/blob/0987da4b259b9fcb139b31d771883a2f822712b5/src/manifest.json#L28 108 109browser.urlbar 110~~~~~~~~~~~~~~ 111 112Currently the only documentation for ``browser.urlbar`` is its `schema 113<urlbar.json_>`__. Fortunately WebExtension schemas are JSON and aren't too hard 114to read. If you need help understanding it, see the `WebExtensions API 115implementation documentation <webextAPIImplDoc_>`__. 116 117For examples on using the API, see the Cookbook_ section. 118 119.. _urlbar.json: https://searchfox.org/mozilla-central/source/browser/components/extensions/schemas/urlbar.json 120 121browser.experiments.urlbar 122~~~~~~~~~~~~~~~~~~~~~~~~~~ 123 124As with ``browser.urlbar``, currently the only documentation for 125``browser.experiments.urlbar`` is its schema__. For examples on using the API, 126see the Cookbook_ section. 127 128__ https://searchfox.org/mozilla-central/source/browser/components/urlbar/tests/ext/schema.json 129 130Workflow 131~~~~~~~~ 132 133The web-ext_ command-line tool makes the extension-development workflow very 134simple. Simply start it with the *run* command, passing it the location of the 135Firefox binary you want to use. web-ext will launch your Firefox and remain 136running until you stop it, watching for changes you make to your extension's 137files. When it sees a change, it automatically reloads your extension — in 138Firefox, in the background — without your having to do anything. It's really 139nice. 140 141The `web-ext documentation <web-ext commands_>`__ lists all its options, but 142here are some worth calling out for the *run* command: 143 144``--browser-console`` 145 Automatically open the browser console when Firefox starts. Very useful for 146 watching your extension's console logging. (Make sure "Show Content Messages" 147 is checked in the console.) 148 149``-p`` 150 This option lets you specify a path to a profile directory. 151 152``--keep-profile-changes`` 153 Normally web-ext doesn't save any changes you make to the profile. Use this 154 option along with ``-p`` to reuse the same profile again and again. 155 156``--verbose`` 157 web-ext suppresses Firefox messages in the terminal unless you pass this 158 option. If you've added some ``dump`` calls in Firefox because you're working 159 on a new ``browser.urlbar`` API, for example, you won't see them without this. 160 161web-ext also has a *build* command that packages your extension's files into a 162zip file. The following *build* options are useful: 163 164``--overwrite-dest`` 165 Without this option, web-ext won't overwrite a zip file it previously created. 166 167web-ext can load its configuration from your extension's package.json. That's 168the recommended way to configure it. Here's an example__. 169 170Finally, web-ext can also sign extensions, but if you're developing your 171extension for an experiment, you'll use a different process for signing. See 172`The Experiment Development Process`_. 173 174.. _web-ext: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Getting_started_with_web-ext 175.. _web-ext commands: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/web-ext_command_reference 176__ https://github.com/0c0w3/urlbar-top-sites-experiment/blob/6681a7126986bc2565d036b888cb5b8807397ce5/package.json#L7 177 178Automated Tests 179~~~~~~~~~~~~~~~ 180 181It's possible to write `browser chrome mochitests`_ for your extension the same 182way we write tests for Firefox. One of the example extensions linked throughout 183this document includes a test_, for instance. 184 185See the readme in the example-addon-experiment_ repo for a workflow. 186 187.. _browser chrome mochitests: https://developer.mozilla.org/en-US/docs/Mozilla/Browser_chrome_tests 188.. _test: https://github.com/0c0w3/urlbar-top-sites-experiment/blob/master/tests/tests/browser/browser_urlbarTopSitesExtension.js 189 190Cookbook 191~~~~~~~~ 192 193*To be written.* For now, you can find example uses of ``browser.experiments.urlbar`` and ``browser.urlbar`` in the following repos: 194 195* https://github.com/mozilla-extensions/firefox-quick-suggest-weather 196* https://github.com/0c0w3/urlbar-tips-experiment 197* https://github.com/0c0w3/urlbar-top-sites-experiment 198* https://github.com/0c0w3/urlbar-search-interventions-experiment 199 200Further Reading 201~~~~~~~~~~~~~~~ 202 203`WebExtensions on MDN <WebExtensions_>`__ 204 The place to learn about developing WebExtensions in general. 205 206`Getting started with web-ext <web-ext_>`__ 207 MDN's tutorial on using web-ext. 208 209`web-ext command reference <web-ext commands_>`__ 210 MDN's documentation on web-ext's commands and their options. 211 212Developing Address Bar Extension APIs 213------------------------------------- 214 215Built-In APIs vs. Experimental APIs 216~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 217 218Originally we developed the address bar extension API in the ``browser.urlbar`` 219namespace, which is built into Firefox as discussed above. By "built into 220Firefox," we mean that the API is developed in `mozilla-central 221<urlbar.json_>`__ and shipped inside Firefox just like any other Firefox 222feature. At the time, that seemed like the right thing to do because we wanted 223to build an API that ultimately could be used by all extension authors, not only 224Mozilla. 225 226However, there were a number of disadvantages to this development model. The 227biggest was that it tightly coupled our experiments to specific versions of 228Firefox. For example, if we were working on an experiment that targeted Firefox 22972, then any APIs used by that experiment needed to land and ship in 72. If we 230weren't able to finish an API by the time 72 shipped, then the experiment would 231have to be postponed until 73. Our experiment development timeframes were always 232very short because we always wanted to ship our experiments ASAP. Often we 233targeted the Firefox version that was then in Nightly; sometimes we even 234targeted the version in Beta. Either way, it meant that we were always uplifting 235patch after patch to Beta. This tight coupling between Firefox versions and 236experiments erased what should have been a big advantage of implementing 237experiments as extensions in the first place: the ability to ship experiments 238outside the usual cyclical release process. 239 240Another notable disadvantage of this model was just the cognitive weight of the 241idea that we were developing APIs not only for ourselves and our experiments but 242potentially for all extensions. This meant that not only did we have to design 243APIs to meet our immediate needs, we also had to imagine use cases that could 244potentially arise and then design for them as well. 245 246For these reasons, we stopped developing ``browser.urlbar`` and created the 247``browser.experiments.urlbar`` experimental API. As discussed earlier, 248experimental APIs are APIs that are bundled inside extensions. Experimental APIs 249can do anything that built-in APIs can do with the added flexibility of not 250being tied to specific versions of Firefox. 251 252Adding New APIs 253~~~~~~~~~~~~~~~ 254 255All new address bar APIs should be added to ``browser.experiments.urlbar``. 256Although this API does not ship in Firefox, it's currently developed in 257mozilla-central, in `browser/components/urlbar/tests/ext/ <extDirectory_>`__ -- 258note the "tests" subdirectory. Developing it in mozilla-central lets us take 259advantage of our usual build and testing infrastructure. This way we have API 260tests running against each mozilla-central checkin, against all versions of 261Firefox that are tested on Mozilla's infrastructure, and we're alerted to any 262breaking changes we accidentally make. When we start a new extension repo, we 263copy schema.json and api.js to it as described earlier (or clone an example repo 264with up-to-date copies of these files). 265 266Generally changes to the API should be reviewed by someone on the address bar 267team and someone on the WebExtensions team. Shane (mixedpuppy) is a good 268contact. 269 270.. _extDirectory: https://searchfox.org/mozilla-central/source/browser/components/urlbar/tests/ext/ 271 272Anatomy of an API 273~~~~~~~~~~~~~~~~~ 274 275Roughly speaking, a WebExtensions API implementation comprises three different 276pieces: 277 278Schema 279 The schema declares the functions, properties, events, and types that the API 280 makes available to extensions. Schemas are written in JSON. 281 282 The ``browser.experiments.urlbar`` schema is schema.json_, and the 283 ``browser.urlbar`` schema is urlbar.json_. 284 285 For reference, the schemas of built-in APIs are in 286 `browser/components/extensions/schemas`_ and 287 `toolkit/components/extensions/schemas`_. 288 289 .. _browser/components/extensions/schemas: https://searchfox.org/mozilla-central/source/browser/components/extensions/schemas/ 290 .. _toolkit/components/extensions/schemas: https://searchfox.org/mozilla-central/source/toolkit/components/extensions/schemas/ 291 292Internals 293 Every API hooks into some internal part of Firefox. For the address bar API, 294 that's the Urlbar implementation in `browser/components/urlbar`_. 295 296 .. _browser/components/urlbar: https://searchfox.org/mozilla-central/source/browser/components/urlbar/ 297 298Glue 299 Finally, there's some glue code that implements everything declared in the 300 schema. Essentially, this code mediates between the previous two pieces. It 301 translates the function calls, property accesses, and event listener 302 registrations made by extensions using the public-facing API into terms that 303 the Firefox internals understand, and vice versa. 304 305 For ``browser.experiments.urlbar``, this is api.js_, and for 306 ``browser.urlbar``, it's ext-urlbar.js_. 307 308 For reference, the implementations of built-in APIs are in 309 `browser/components/extensions`_ and `toolkit/components/extensions`_, in the 310 *parent* and *child* subdirecties. As you might guess, code in *parent* runs 311 in the main process, and code in *child* runs in the extensions process. 312 Address bar APIs deal with browser chrome and their implementations therefore 313 run in the parent process. 314 315 .. _ext-urlbar.js: https://searchfox.org/mozilla-central/source/browser/components/extensions/parent/ext-urlbar.js 316 .. _browser/components/extensions: https://searchfox.org/mozilla-central/source/browser/components/extensions/ 317 .. _toolkit/components/extensions: https://searchfox.org/mozilla-central/source/toolkit/components/extensions/ 318 319Keep in mind that extensions run in a different process from the main process. 320That has implications for your APIs. They'll generally need to be async, for 321example. 322 323Further Reading 324~~~~~~~~~~~~~~~ 325 326`WebExtensions API implementation documentation <webextAPIImplDoc_>`__ 327 Detailed info on implementing a WebExtensions API. 328 329.. _webextAPIImplDoc: https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/ 330 331Running Address Bar Extensions 332------------------------------ 333 334As discussed above, ``browser.experiments.urlbar`` and ``browser.urlbar`` are 335privileged APIs. There are two different points to consider when it comes to 336running an extension that uses privileged APIs: loading the extension in the 337first place, and granting it access to privileged APIs. There's a certain bar 338for loading any extension regardless of its API usage that depends on its signed 339state and the Firefox build you want to run it in. There's yet a higher bar for 340granting it access to privileged APIs. This section discusses how to load 341extensions so that they can access privileged APIs. 342 343Since we're interested in extensions primarily for running experiments, there 344are three particular signed states relevant to us: 345 346Unsigned 347 There are two ways to run unsigned extensions that use privileged APIs. 348 349 They can be loaded temporarily using a Firefox Nightly build or 350 Developer Edition but not Beta or Release [source__], and the 351 ``extensions.experiments.enabled`` preference must be set to true [source__]. 352 You can load extensions temporarily by visiting 353 about:debugging#/runtime/this-firefox and clicking "Load Temporary Add-on." 354 `web-ext <Workflow_>`__ also loads extensions temporarily. 355 356 __ https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/toolkit/components/extensions/Extension.jsm#1884 357 __ https://searchfox.org/mozilla-central/rev/014fe72eaba26dcf6082fb9bbaf208f97a38594e/toolkit/mozapps/extensions/internal/AddonSettings.jsm#93 358 359 They can be also be loaded normally (not temporarily) in a custom build where 360 the build-time setting ``AppConstants.MOZ_REQUIRE_SIGNING`` [source__, source__] 361 and ``xpinstall.signatures.required`` pref are both false. As in the previous 362 paragraph, such builds include Nightly and Developer Edition but not Beta or 363 Release [source__]. In addition, your custom build must modify the 364 ``Extension.isPrivileged`` getter__ to return true. This getter determines 365 whether an extension can access privileged APIs. 366 367 __ https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/toolkit/mozapps/extensions/internal/XPIProvider.jsm#2382 368 __ https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/toolkit/mozapps/extensions/internal/AddonSettings.jsm#36 369 __ https://searchfox.org/mozilla-central/search?q=MOZ_REQUIRE_SIGNING&case=false®exp=false&path= 370 __ https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/toolkit/components/extensions/Extension.jsm#1874 371 372 Extensions remain unsigned as you develop them. See the Workflow_ section for 373 more. 374 375Signed for testing (Signed for QA) 376 Signed-for-testing extensions that use privileged APIs can be run using the 377 same techniques for running unsigned extensions. 378 379 They can also be loaded normally (not temporarily) if you use a Firefox build 380 where the build-time setting ``AppConstants.MOZ_REQUIRE_SIGNING`` is false and 381 you set the ``xpinstall.signatures.dev-root`` pref to true 382 [source__]. ``xpinstall.signatures.dev-root`` does not exist by default and 383 must be created. 384 385 __ https://searchfox.org/mozilla-central/rev/053826b10f838f77c27507e5efecc96e34718541/toolkit/mozapps/extensions/internal/XPIInstall.jsm#262 386 387 You encounter extensions that are signed for testing when you are writing 388 extensions for experiments. See the Experiments_ section for details. 389 390 "Signed for QA" is another way of referring to this signed state. 391 392Signed for release 393 Signed-for-release extensions that use privileged APIs can be run in any 394 Firefox build with no special requirements. 395 396 You encounter extensions that are signed for release when you are writing 397 extensions for experiments. See the Experiments_ section for details. 398 399.. important:: 400 To see console logs from extensions in the browser console, select the "Show 401 Content Messages" option in the console's settings. This is necessary because 402 extensions run outside the main process. 403 404Experiments 405----------- 406 407**Experiments** let us try out ideas in Firefox outside the usual release cycle 408and on particular populations of users. 409 410For example, say we have a hunch that the top sites shown on the new-tab page 411aren't very discoverable, so we want to make them more visible. We have one idea 412that might work — show them every time the user begins an interaction with the 413address bar — but we aren't sure how good an idea it is. So we test it. We write 414an extension that does just that, make sure it collects telemetry that will help 415us answer our question, ship it outside the usual release cycle to a small 416percentage of Beta users, collect and analyze the telemetry, and determine 417whether the experiment was successful. If it was, then we might want to ship the 418feature to all Firefox users. 419 420Experiments sometimes are also called **studies** (not to be confused with *user 421studies*, which are face-to-face interviews with users conducted by user 422researchers). 423 424There are two types of experiments: 425 426Pref-flip experiments 427 Pref-flip experiments are simple. If we have a fully baked feature in the 428 browser that's preffed off, a pref-flip experiment just flips the pref on, 429 enabling the feature for users running the experiment. No code is required. 430 We tell the experiments team the name of the pref we want to flip, and they 431 handle it. 432 433 One important caveat to pref-flip studies is that they're currently capable of 434 flipping only a single pref. There's an extension called Multipreffer_ that 435 can flip multiple prefs, though. 436 437 .. _Multipreffer: https://github.com/mozilla/multipreffer 438 439Add-on experiments 440 Add-on experiments are much more complex but much more powerful. (Here 441 *add-on* is a synonym for extension.) They're the type of experiments that 442 this document has been discussing all along. 443 444 An add-on experiment is shipped as an extension that we write and that 445 implements the experimental feature we want to test. To reiterate, the 446 extension is a WebExtension and uses WebExtensions APIs. If the current 447 WebExtensions APIs do not meet the needs of your experiment, then you must 448 create either experimental or built-in APIs so that your extension can use 449 them. If necessary, you can make any new built-in APIs privileged so that they 450 are available only to Mozilla extensions. 451 452 An add-on experiment can collect additional telemetry that's not collected in 453 the product by using the privileged ``browser.telemetry`` WebExtensions API, 454 and of course the product will continue to collect all the telemetry it 455 usually does. The telemetry pings from users running the experiment will be 456 correlated with the experiment with no extra work on our part. 457 458A single experiment can deliver different UXes to different groups of users 459running the experiment. Each group or UX within an experiment is called a 460**branch**. Experiments often have two branches, control and treatment. The 461**control branch** actually makes no UX changes. It may capture additional 462telemetry, though. Think of it as the control in a science experiment. It's 463there so we can compare it to data from the **treatment branch**, which does 464make UX changes. Some experiments may require multiple treatment branches, in 465which case the different branches will have different names. Add-on experiments 466can implement all branches in the same extension or each branch in its own 467extension. 468 469Experiments are delivered to users by a system called **Normandy**. Normandy 470comprises a client side that lives in Firefox and a server side. In Normandy, 471experiments are defined server-side in files called **recipes**. Recipes include 472information about the experiment like the Firefox release channel and version 473that the experiment targets, the number of users to be included in the 474experiment, the branches in the experiment, the percentage of users on each 475branch, and so on. 476 477Experiments are tracked by Mozilla project management using a system called 478Experimenter_. 479 480Finally, there was an older version of the experiments program called 481**Shield**. Experiments under this system were called **Shield studies** and 482could be be shipped as extensions too. 483 484.. _Experimenter: https://experimenter.services.mozilla.com/ 485 486Further Reading 487~~~~~~~~~~~~~~~ 488 489`Pref-Flip and Add-On Experiments <https://mana.mozilla.org/wiki/pages/viewpage.action?spaceKey=FIREFOX&title=Pref-Flip+and+Add-On+Experiments>`__ 490 A comprehensive document on experiments from the Experimenter team. See the 491 child pages in the sidebar, too. 492 493`Client Implementation Guidelines for Experiments <https://docs.telemetry.mozilla.org/cookbooks/client_guidelines.html>`_ 494 Relevant documentation from the telemetry team. 495 496#ask-experimenter Slack channel 497 A friendly place to get answers to your experiment questions. 498 499The Experiment Development Process 500---------------------------------- 501 502This section describes an experiment's life cycle. 503 5041. Experiments usually originate with product management and UX. They're 505 responsible for identifying a problem, deciding how an experiment should 506 approach it, the questions we want to answer, the data we need to answer 507 those questions, the user population that should be enrolled in the 508 experiment, the definition of success, and so on. 509 5102. UX makes a spec that describes what the extension looks like and how it 511 behaves. 512 5133. There's a kickoff meeting among the team to introduce the experiment and UX 514 spec. It's an opportunity for engineering to ask questions of management, UX, 515 and data science. It's really important for engineering to get a precise and 516 accurate understanding of how the extension is supposed to behave — right 517 down to the UI changes — so that no one makes erroneous assumptions during 518 development. 519 5204. At some point around this time, the team (usually management) creates a few 521 artifacts to track the work and facilitate communication with outside teams 522 involved in shipping experiments. They include: 523 524 * A page on `Experimenter <Experiments_>`__ 525 * A QA PI (product integrity) request so that QA resources are allocated 526 * A bug in `Data Science :: Experiment Collaboration`__ so that data science 527 can track the work and discuss telemetry (engineering might file this one) 528 529 __ https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_ignored=0&bug_severity=normal&bug_status=NEW&bug_type=task&cf_firefox_messaging_system=---&cf_fx_iteration=---&cf_fx_points=---&comment=%23%23%20Brief%20Description%20of%20the%20request%20%28required%29%3A%0D%0A%0D%0A%23%23%20Business%20purpose%20for%20this%20request%20%28required%29%3A%0D%0A%0D%0A%23%23%20Requested%20timelines%20for%20the%20request%20or%20how%20this%20fits%20into%20roadmaps%20or%20critical%20decisions%20%28required%29%3A%0D%0A%0D%0A%23%23%20Links%20to%20any%20assets%20%28e.g%20Start%20of%20a%20PHD%2C%20BRD%3B%20any%20document%20that%20helps%20describe%20the%20project%29%3A%0D%0A%0D%0A%23%23%20Name%20of%20Data%20Scientist%20%28If%20Applicable%29%3A%0D%0A%0D%0A%2APlease%20note%20if%20it%20is%20found%20that%20not%20enough%20information%20has%20been%20given%20this%20will%20delay%20the%20triage%20of%20this%20request.%2A&component=Experiment%20Collaboration&contenttypemethod=list&contenttypeselection=text%2Fplain&filed_via=standard_form&flag_type-4=X&flag_type-607=X&flag_type-800=X&flag_type-803=X&flag_type-936=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=--&product=Data%20Science&rep_platform=Unspecified&target_milestone=---&version=unspecified 530 5315. Engineering breaks down the work and files bugs. There's another engineering 532 meeting to discuss the breakdown, or it's discussed asynchronously. 533 5346. Engineering sets up a GitHub repo for the extension. See `Implementing 535 Experiments`_ for an example repo you can clone to get started. Disable 536 GitHub Issues on the repo so that QA will file bugs in Bugzilla instead of 537 GitHub. There's nothing wrong with GitHub Issues, but our team's project 538 management tracks all work through Bugzilla. If it's not there, it's not 539 captured. 540 5417. Engineering or management fills out the Add-on section of the Experimenter 542 page as much as possible at this point. "Active Experiment Name" isn't 543 necessary, and "Signed Release URL" won't be available until the end of the 544 process. 545 5468. Engineering implements the extension and any new WebExtensions APIs it 547 requires. 548 5499. When the extension is done, engineering or management clicks the "Ready for 550 Sign-Off" button on the Experimenter page. That changes the page's status 551 from "Draft" to "Ready for Sign-Off," which allows QA and other teams to sign 552 off on their portions of the experiment. 553 55410. Engineering requests the extension be signed "for testing" (or "for 555 QA"). Michael (mythmon) from the Experiments team and Rehan (rdalal) from 556 Services Engineering are good contacts. Build the extension zip file using 557 web-ext as discussed in Workflow_. Attach it to a bug (a metabug for 558 implementing the extension, for example), needinfo Michael or Rehan, and ask 559 him to sign it. He'll attach the signed version to the bug. If neither 560 Michael nor Rehan is available, try asking in the #ask-experimenter Slack 561 channel. 562 56311. Engineering sends QA the link to the signed extension and works with them to 564 resolve bugs they find. 565 56612. When QA signs off, engineering asks Michael to sign the extension "for 567 release" using the same needinfo process described earlier. 568 56913. Paste the URL of the signed extension in the "Signed Release URL" textbox of 570 the Add-on section of the Experimenter page. 571 57214. Other teams sign off as they're ready. 573 57415. The experiment ships! 575 576 577Implementing Experiments 578------------------------ 579 580This section discusses how to implement add-on experiments. Pref-flip 581experiments are much simpler and don't need a lot of explanation. You should be 582familiar with the concepts discussed in the `Developing Address Bar Extensions`_ 583and `Running Address Bar Extensions`_ sections before reading this one. 584 585The most salient thing about add-on experiments is that they're implemented 586simply as privileged extensions. Other than being privileged and possibly 587containing bundled experimental APIs, they're similar to all other extensions. 588 589The `top-sites experiment extension <topSites_>`__ is an example of a real, 590shipped experiment. 591 592.. _topSites: https://github.com/0c0w3/urlbar-top-sites-experiment 593 594Setup 595~~~~~ 596 597example-addon-experiment_ is a repo you can clone to get started. It's geared 598toward urlbar extensions and includes the stub of a browser chrome mochitest. 599 600.. _example-addon-experiment: https://github.com/0c0w3/example-addon-experiment 601 602browser.normandyAddonStudy 603~~~~~~~~~~~~~~~~~~~~~~~~~~ 604 605As discussed in Experiments_, an experiment typically has more than one branch 606so that it can test different UXes. The experiment's extension(s) needs to know 607the branch the user is enrolled in so that it can behave appropriately for the 608branch: show the user the proper UX, collect the proper telemetry, and so on. 609 610This is the purpose of the ``browser.normandyAddonStudy`` WebExtensions API. 611Like ``browser.urlbar``, it's a privileged API available only to Mozilla 612extensions. 613 614Its schema is normandyAddonStudy.json_. 615 616It's a very simple API. The primary function is ``getStudy``, which returns the 617study the user is currently enrolled in or null if there isn't one. (Recall that 618*study* is a synonym for *experiment*.) One of the first things an experiment 619extension typically does is to call this function. 620 621The Normandy client in Firefox will keep an experiment extension installed only 622while the experiment is active. Therefore, ``getStudy`` should always return a 623non-null study object. Nevertheless, the study object has an ``active`` boolean 624property that's trivial to sanity check. (The example extension does.) 625 626The more important property is ``branch``, the name of the branch that the user 627is enrolled in. Your extension should use it to determine the appropriate UX. 628 629Finally, there's an ``onUnenroll`` event that's fired when the user is 630unenrolled in the study. It's not quite clear in what cases an extension would 631need to listen for this event given that Normandy automatically uninstalls 632extensions on unenrollment. Maybe if they create some persistent state that's 633not automatically undone on uninstall by the WebExtensions framework? 634 635If your extension itself needs to unenroll the user for some reason, call 636``endStudy``. 637 638.. _normandyAddonStudy.json: https://searchfox.org/mozilla-central/source/browser/components/extensions/schemas/normandyAddonStudy.json 639 640Telemetry 641~~~~~~~~~ 642 643Experiments can capture telemetry in two places: in the product itself and 644through the privileged ``browser.telemetry`` WebExtensions API. The API schema 645is telemetry.json_. 646 647The telemetry pings from users running experiments are automatically correlated 648with those experiments, no extra work required. That's true regardless of 649whether the telemetry is captured in the product or though 650``browser.telemetry``. 651 652The address bar has some in-product, preffed off telemetry that we want to 653enable for all our experiments — at least that's the thinking as of August 2019. 654It's called `engagement event telemetry`_, and it records user *engagements* 655with and *abandonments* of the address bar [source__]. We added a 656BrowserSetting_ on ``browser.urlbar`` just to let us flip the pref and enable 657this telemetry in our experiment extensions. Call it like this:: 658 659 await browser.urlbar.engagementTelemetry.set({ value: true }); 660 661.. _telemetry.json: https://searchfox.org/mozilla-central/source/toolkit/components/extensions/schemas/telemetry.json 662.. _engagement event telemetry: https://bugzilla.mozilla.org/show_bug.cgi?id=1559136 663__ https://searchfox.org/mozilla-central/rev/7088fc958db5935eba24b413b1f16d6ab7bd13ea/browser/components/urlbar/UrlbarController.jsm#598 664.. _BrowserSetting: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/types/BrowserSetting 665 666Engineering Best Practices 667~~~~~~~~~~~~~~~~~~~~~~~~~~ 668 669Clear up questions with your UX person early and often. There's often a gap 670between what they have in their mind and what you have in yours. Nothing wrong 671with that, it's just the nature of development. But misunderstandings can cause 672big problems when they're discovered late. This is especially true of UX 673behaviors, as opposed to visuals or styling. It's no fun to realize at the end 674of a release cycle that you've designed the wrong WebExtensions API because some 675UX detail was overlooked. 676 677~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 678 679Related to the previous point, make builds of your extension for your UX person 680so they can test it. 681 682~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 683 684Taking the previous point even further, if your experiment will require a 685substantial new API(s), you might think about prototyping the experiment 686entirely in a custom Firefox build before designing the API at all. Give it to 687your UX person. Let them disect it and tell you all the problems with it. Fill 688in all the gaps in your understanding, and then design the API. We've never 689actually done this, though. 690 691~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 692 693It's a good idea to work on the extension as you're designing and developing the 694APIs it'll use. You might even go as far as writing the first draft of the 695extension before even starting to implement the APIs. That lets you spot 696problems that may not be obvious were you to design the API in isolation. 697 698~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 699 700Your extension's ID should end in ``@shield.mozilla.org``. QA will flag it if it 701doesn't. 702 703~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 704 705Set ``"hidden": true`` in your extension's manifest.json. That hides it on 706about:addons. (It can still be seen on about:studies.) QA will spot this if you 707don't. 708 709~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 710 711There are drawbacks of hiding features behind prefs and enabling them in 712experiment extensions. Consider not doing that if feasible, or at least weigh 713these drawbacks against your expected benefits. 714 715* Prefs stay flipped on in private windows, but experiments often have special 716 requirements around private-browsing mode (PBM). Usually, they shouldn't be 717 active in PBM at all, unless of course the point of the experiment is to test 718 PBM. Extensions also must request PBM access ("incognito" in WebExtensions 719 terms), and the user can disable access at any time. The result is that part 720 of your experiment could remain enabled — the part behind the pref — while 721 other parts are disabled. 722 723* Prefs stay flipped on in safe mode, even though your extension (like all 724 extensions) will be disabled. This might be a bug__ in the WebExtensions 725 framework, though. 726 727 __ https://bugzilla.mozilla.org/show_bug.cgi?id=1576997 728