1.. role:: html(code) 2 :language: html 3 4.. role:: js(code) 5 :language: javascript 6 7============================= 8Fluent for Firefox Developers 9============================= 10 11 12This tutorial is intended for Firefox engineers already familiar with the previous 13localization systems offered by Gecko - `DTD`_ and `StringBundle`_ - and assumes 14prior experience with those systems. 15 16For a more hands-on tutorial of understanding Fluent from the ground up, try 17following the `Fluent DOMLocalization Tutorial`__, which provides some background on 18how Fluent works and walks you through creating a basic web project from scratch that 19uses Fluent for localization. 20 21__ https://projectfluent.org/dom-l10n-documentation/ 22 23Using Fluent in Gecko 24===================== 25 26`Fluent`_ is a modern localization system introduced into 27the Gecko platform with a focus on quality, performance, maintenance and completeness. 28 29The legacy DTD system is deprecated, and Fluent should be used where possible. 30 31Getting a Review 32---------------- 33 34If you work on any patch that touches FTL files, you'll need to get a review 35from `fluent-reviewers`__. There's a Herald hook that automatically sets 36that group as a blocking reviewer. 37 38__ https://phabricator.services.mozilla.com/tag/fluent-reviewers/ 39 40Guidelines for the review process are available `here`__. 41 42__ ./fluent_review.html 43 44To lighten the burden on reviewers, please take a moment to review some 45best practices before submitting your patch for review. 46 47- `ProjectFluent Good Practices for Developers`_ 48- `Mozilla Localization Best Practices For Developers`_ 49 50.. _ProjectFluent Good Practices for Developers: https://github.com/projectfluent/fluent/wiki/Good-Practices-for-Developers 51.. _Mozilla Localization Best Practices For Developers: https://mozilla-l10n.github.io/documentation/localization/dev_best_practices.html 52 53Major Benefits 54============== 55 56Fluent `ties tightly`__ into the domain of internationalization 57through `Unicode`_, `CLDR`_ and `ICU`_. 58 59__ https://github.com/projectfluent/fluent/wiki/Fluent-and-Standards 60 61More specifically, the most observable benefits for each group of consumers are 62 63 64Developers 65---------- 66 67 - Support for XUL, XHTML, HTML, Web Components, React, JS, Python and Rust 68 - Strings are available in a single, unified localization context available for both DOM and runtime code 69 - Full internationalization (i18n) support: date and time formatting, number formatting, plurals, genders etc. 70 - Strong focus on `declarative API via DOM attributes`__ 71 - Extensible with custom formatters, Mozilla-specific APIs etc. 72 - `Separation of concerns`__: localization details, and the added complexity of some languages, don't leak onto the source code and are no concern for developers 73 - Compound messages link a single translation unit to a single UI element 74 - `DOM Overlays`__ allow for localization of DOM fragments 75 - Simplified build system model 76 - No need for pre-processing instructions 77 - Support for pseudolocalization 78 79__ https://github.com/projectfluent/fluent/wiki/Get-Started 80__ https://github.com/projectfluent/fluent/wiki/Design-Principles 81__ https://github.com/projectfluent/fluent.js/wiki/DOM-Overlays 82 83 84Product Quality 85------------------ 86 87 - A robust, multilevel, `error fallback system`__ prevents XML errors and runtime errors 88 - Simplified l10n API reduces the amount of l10n specific code and resulting bugs 89 - Runtime localization allows for dynamic language changes and updates over-the-air 90 - DOM Overlays increase localization security 91 92__ https://github.com/projectfluent/fluent/wiki/Error-Handling 93 94 95Fluent Translation List - FTL 96============================= 97 98Fluent introduces a file format designed specifically for easy readability 99and the localization features offered by the system. 100 101At first glance the format is a simple key-value store. It may look like this: 102 103.. code-block:: fluent 104 105 home-page-header = Home Page 106 107 # The label of a button opening a new tab 108 new-tab-open = Open New Tab 109 110But the FTL file format is significantly more powerful and the additional features 111quickly add up. In order to familiarize yourself with the basic features, 112consider reading through the `Fluent Syntax Guide`_ to understand 113a more complex example like: 114 115.. code-block:: fluent 116 117 ### These messages correspond to security and privacy user interface. 118 ### 119 ### Please choose simple and non-threatening language when localizing 120 ### to help user feel in control when interacting with the UI. 121 122 ## General Section 123 124 -brand-short-name = Firefox 125 .gender = masculine 126 127 pref-pane = 128 .title = 129 { PLATFORM() -> 130 [windows] Options 131 *[other] Preferences 132 } 133 .accesskey = C 134 135 # Variables: 136 # $tabCount (Number) - number of container tabs to be closed 137 containers-disable-alert-ok-button = 138 { $tabCount -> 139 [one] Close { $tabCount } Container Tab 140 *[other] Close { $tabCount } Container Tabs 141 } 142 143 update-application-info = 144 You are using { -brand-short-name } Version: { $version }. 145 Please read the <a>privacy policy</a>. 146 147The above, of course, is a particular selection of complex strings intended to exemplify 148the new features and concepts introduced by Fluent. 149 150.. important:: 151 152 While in Fluent it’s possible to use both lowercase and uppercase characters in message 153 identifiers, the naming convention in Gecko is to use lowercase and hyphens, avoiding 154 CamelCase and underscores. For example, `allow-button` should be preferred to 155 `allow_button` or `allowButton`, unless there are technically constraints – like 156 identifiers generated at run-time from external sources – that make this impractical. 157 158In order to ensure the quality of the output, a lot of checks and tooling 159is part of the build system. 160`Pontoon`_, the main localization tool used to translate Firefox, also supports 161Fluent and its features to help localizers in their work. 162 163 164.. _fluent-tutorial-social-contract: 165 166Social Contract 167=============== 168 169Fluent uses the concept of a `social contract` between developer and localizers. 170This contract is established by the selection of a unique identifier, called :js:`l10n-id`, 171which carries a promise of being used in a particular place to carry a particular meaning. 172 173The use of unique identifiers is shared with legacy localization systems in 174Firefox. 175 176.. important:: 177 178 An important part of the contract is that the developer commits to treat the 179 localization output as `opaque`. That means that no concatenations, replacements 180 or splitting should happen after the translation is completed to generate the 181 desired output. 182 183In return, localizers enter the social contract by promising to provide an accurate 184and clean translation of the messages that match the request. 185 186In Fluent, the developer is not to be bothered with inner logic and complexity that the 187localization will use to construct the response. Whether `declensions`__ or other 188variant selection techniques are used is up to a localizer and their particular translation. 189From the developer perspective, Fluent returns a final string to be presented to 190the user, with no l10n logic required in the running code. 191 192__ https://en.wikipedia.org/wiki/Declension 193 194 195Markup Localization 196=================== 197 198To localize an element in Fluent, the developer adds a new message to 199an FTL file and then has to associate an :js:`l10n-id` with the element 200by defining a :js:`data-l10n-id` attribute: 201 202.. code-block:: html 203 204 <h1 data-l10n-id="home-page-header" /> 205 206 <button data-l10n-id="pref-pane" /> 207 208Fluent will take care of the rest, populating the element with the message value 209in its content and all localizable attributes if defined. 210 211The developer provides only a single message to localize the whole element, 212including the value and selected attributes. 213 214The value can be a whole fragment of DOM: 215 216.. code-block:: html 217 218 <p data-l10n-id="update-application-info" data-l10n-args='{"version": "60.0"}'> 219 <a data-l10n-name="privacy-url" href="http://www.mozilla.org/privacy" /> 220 </p> 221 222.. code-block:: fluent 223 224 -brand-short-name = Firefox 225 update-application-info = 226 You are using { -brand-short-name } Version: { $version }. 227 Please read the <a data-l10n-name="privacy-url">privacy policy</a>. 228 229 230Fluent will overlay the translation onto the source fragment preserving attributes like 231:code:`class` and :code:`href` from the source and adding translations for the elements 232inside. The resulting localized content will look like this: 233 234.. code-block:: 235 236 <p data-l10n-id="update-application-info" data-l10n-args='{"version": "60.0"}'"> 237 You are using Firefox Version: 60.0. 238 Please read the <a href="http://www.mozilla.org/privacy">privacy policy</a>. 239 </p> 240 241 242This operation is sanitized, and Fluent takes care of selecting which elements and 243attributes can be safely provided by the localization. 244The list of allowed elements and attributes is `maintained by the W3C`__, and if 245the developer needs to allow for localization of additional attributes, they can 246allow them using :code:`data-l10n-attrs` list: 247 248.. code-block:: html 249 250 <label data-l10n-id="search-input" data-l10n-attrs="style" /> 251 252The above example adds an attribute :code:`style` to be allowed on this 253particular :code:`label` element. 254 255 256External Arguments 257------------------ 258 259Notice in the previous example the attribute :code:`data-l10n-args`, which is 260a JSON object storing variables exposed by the developer to the localizer. 261 262This is the main channel for the developer to provide additional variables 263to be used in the localization. 264 265Arguments are rarely needed for situations where it’s currently possible to use 266DTD, since such variables would need to be computed from the code at runtime. 267It's worth noting that, when the :code:`l10n-args` are set in 268the runtime code, they are in fact encoded as JSON and stored together with 269:code:`l10n-id` as an attribute of the element. 270 271__ https://www.w3.org/TR/2011/WD-html5-20110525/text-level-semantics.html 272 273 274Runtime Localization 275==================== 276 277In almost every case the JS runtime code will operate on a particular document, either 278XUL, XHTML or HTML. 279 280If the document has its markup already localized, then Fluent exposes a new 281attribute on the :js:`document` element - :js:`document.l10n`. 282 283This property is an object of type :js:`DOMLocalization` which maintains the main 284localization context for this document and exposes it to runtime code as well. 285 286With a focus on `declarative localization`__, the primary method of localization is 287to alter the localization attributes in the DOM. Fluent provides a method to facilitate this: 288 289.. code-block:: javascript 290 291 document.l10n.setAttributes(element, "new-panel-header"); 292 293This will set the :code:`data-l10n-id` on the element and translate it before the next 294animation frame. 295 296The reason to use this API over manually setting the attribute is that it also 297facilitates encoding l10n arguments as JSON: 298 299.. code-block:: javascript 300 301 document.l10n.setAttributes(element, "containers-disable-alert-ok-button", { 302 tabCount: 5 303 }); 304 305__ https://github.com/projectfluent/fluent/wiki/Good-Practices-for-Developers 306 307 308Non-Markup Localization 309----------------------- 310 311In rare cases, when the runtime code needs to retrieve the translation and not 312apply it onto the DOM, Fluent provides an API to retrieve it: 313 314.. code-block:: javascript 315 316 let [ msg ] = await document.l10n.formatValues([ 317 {id: "remove-containers-description"} 318 ]); 319 320 alert(msg); 321 322This model is heavily discouraged and should be used only in cases where the 323DOM annotation is not possible. 324 325.. note:: 326 327 This API is available as asynchronous. In case of Firefox, 328 the only non-DOM localizable calls are used where the output goes to 329 a third-party like Bluetooth, Notifications etc. 330 All those cases should already be asynchronous. If you can't avoid synchronous 331 access, you can use ``mozILocalization.formatMessagesSync`` with synchronous IO. 332 333 334Internationalization 335==================== 336 337The majority of internationalization issues are implicitly handled by Fluent without 338any additional requirement. Full Unicode support, `bidirectionality`__, and 339correct number formatting work without any action required from either 340developer or localizer. 341 342__ https://github.com/projectfluent/fluent/wiki/BiDi-in-Fluent 343 344.. code-block:: javascript 345 346 document.l10n.setAttributes(element, "welcome-message", { 347 userName: "اليسع", 348 count: 5 349 }); 350 351A message like this localized to American English will correctly wrap the user 352name in directionality marks, allowing the layout engine to determine how to 353display the bidirectional text. 354 355On the other hand, the same message localized to Arabic will use the Eastern Arabic 356numeral for number "5". 357 358 359Plural Rules 360------------ 361 362The most common localization feature is the ability to provide different variants 363of the same string depending on plural categories. Fluent ties into the Unicode CLDR 364standard called `Plural Rules`_. 365 366In order to allow localizers to use it, all the developer has to do is to pass 367an external argument number: 368 369.. code-block:: javascript 370 371 document.l10n.setAttributes(element, "unread-warning", { unreadCount: 5 }); 372 373Localizers can use the argument to build a multi variant message if their 374language requires that: 375 376.. code-block:: fluent 377 378 unread-warning = 379 { $unreadCount -> 380 [one] You have { $unreadCount } unread message 381 *[other] You have { $unreadCount } unread messages 382 } 383 384If the variant selection is performed based on a number, Fluent matches that 385number against literal numbers as well as its `plural category`__. 386 387If the given translation doesn't need pluralization for the string (for example 388Japanese often will not), the localizer can replace it with: 389 390.. code-block:: fluent 391 392 unread-warning = You have { $unreadCount } unread messages 393 394and the message will preserve the social contract. 395 396One additional feature is that the localizer can further improve the message by 397specifying variants for particular values: 398 399.. code-block:: fluent 400 401 unread-warning = 402 { $unreadCount -> 403 [0] You have no unread messages 404 [1] You have one unread message 405 *[other] You have { $unreadCount } unread messages 406 } 407 408The advantage here is that per-locale choices don't leak onto the source code 409and the developer is not affected. 410 411 412.. note:: 413 414 There is an important distinction between a variant keyed on plural category 415 `one` and digit `1`. Although in English the two are synonymous, in other 416 languages category `one` may be used for other numbers. 417 For example in `Bosnian`__, category `one` is used for numbers like `1`, `21`, `31` 418 and so on, and also for fractional numbers like `0.1`. 419 420__ https://unicode.org/cldr/charts/latest/supplemental/language_plural_rules.html 421__ https://unicode.org/cldr/charts/latest/supplemental/language_plural_rules.html#bs 422 423Partially-formatted variables 424----------------------------- 425 426When it comes to formatting data, Fluent allows the developer to provide 427a set of parameters for the formatter, and the localizer can fine tune some of them. 428This technique is called `partially-formatted variables`__. 429 430For example, when formatting a date, the developer can just pass a JS :js:`Date` object, 431but its default formatting will be pretty expressive. In most cases, the developer 432may want to use some of the :js:`Intl.DateTimeFormat` options to select the default 433representation of the date in string: 434 435.. code-block:: javascript 436 437 document.l10n.formatValue("welcome-message", { 438 startDate: FluentDateTime(new Date(), { 439 year: "numeric", 440 month: "long", 441 day: "numeric" 442 }) 443 }); 444 445.. code-block:: fluent 446 447 welcome-message = Your session will start date: { $startDate } 448 449In most cases, that will be enough and the date would get formatted in the current 450Firefox as `February 28, 2018`. 451 452But if in some other locale the string would get too long, the localizer can fine 453tune the options as well: 454 455.. code-block:: fluent 456 457 welcome-message = Początek Twojej sesji: { DATETIME($startDate, month: "short") } 458 459This will adjust the length of the month token in the message to short and get formatted 460in Polish as `28 lut 2018`. 461 462At the moment Fluent supports two formatters that match JS Intl API counterparts: 463 464 * **NUMBER**: `Intl.NumberFormat`__ 465 * **DATETIME**: `Intl.DateTimeFormat`__ 466 467With time more formatters will be added. Also, this feature is not exposed 468to ``setAttributes`` at this point, as that serializes to JSON. 469 470__ https://projectfluent.org/fluent/guide/functions.html#partially-formatted-variables 471__ https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/NumberFormat 472__ https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat 473 474Registering New L10n Files 475========================== 476 477Fluent uses a wildcard statement, packaging all localization resources into 478their component's `/localization/` directory. 479 480That means that, if a new file is added to a component of Firefox already 481covered by Fluent like `browser`, it's enough to add the new file to the 482repository in a path like `browser/locales/en-US/browser/component/file.ftl`, and 483the toolchain will package it into `browser/localization/browser/component/file.ftl`. 484 485At runtime Firefox uses a special registry for all localization data. It will 486register the browser's `/localization/` directory and make all files inside it 487available to be referenced. 488 489To make the document localized using Fluent, all the developer has to do is add 490localizable resources for Fluent API to use: 491 492.. code-block:: html 493 494 <link rel="localization" href="branding/brand.ftl"/> 495 <link rel="localization" href="browser/preferences/preferences.ftl"/> 496 497The URI provided to the :html:`<link/>` element are relative paths within the localization 498system. 499 500 501Custom Localizations 502==================== 503 504The above method creates a single localization context per document. 505In almost all scenarios that's sufficient. 506 507In rare edge cases where the developer needs to fetch additional resources, or 508the same resources in another language, it is possible to create additional 509Localization object manually using the `Localization` class: 510 511.. code-block:: javascript 512 513 const myL10n = new Localization([ 514 "branding/brand.ftl", 515 "browser/preferences/preferences.ftl" 516 ]); 517 518 519 let [isDefaultMsg, isNotDefaultMsg] = 520 await myL10n.formatValues({id: "is-default"}, {id: "is-not-default"}); 521 522 523.. admonition:: Example 524 525 An example of a use case is the Preferences UI in Firefox, which uses the 526 main context to localize the UI but also to build a search index. 527 528 It is common to build such search index both in a current language and additionally 529 in English, since a lot of documentation and online help exist only in English. 530 531 A developer may create manually a new context with the same resources as the main one, 532 but hardcode it to `en-US` and then build the search index using both contexts. 533 534 535By default, all `Localization` contexts are asynchronous. It is possible to create a synchronous 536one by passing an `sync = false` argument to the constructor, or calling the `SetIsSync(bool)` method 537on the class. 538 539 540.. code-block:: javascript 541 542 const myL10n = new Localization([ 543 "branding/brand.ftl", 544 "browser/preferences/preferences.ftl" 545 ], false); 546 547 548 let [isDefaultMsg, isNotDefaultMsg] = 549 myL10n.formatValuesSync({id: "is-default"}, {id: "is-not-default"}); 550 551 552Synchronous contexts should be always avoided as they require synchronous I/O. If you think your use case 553requires a synchronous localization context, please consult Gecko, Performance and L10n Drivers teams. 554 555 556Designing Localizable APIs 557========================== 558 559When designing localizable APIs, the most important rule is to resolve localization as 560late as possible. That means that instead of resolving strings somewhere deep in the 561codebase and then passing them on, or even caching, it is highly recommended to pass 562around :code:`l10n-id` or :code:`[l10n-id, l10n-args]` pairs until the top-most code 563resolves them or applies them onto the DOM element. 564 565 566Testing 567======= 568 569When writing tests that involve both I18n and L10n, the general rule is that 570result strings are opaque. That means that the developer should not assume any particular 571value and should never test against it. 572 573In case of raw i18n the :js:`resolvedOptions` method on all :js:`Intl.*` formatters 574makes it relatively easy. In case of localization, the recommended way is to test that 575the code sets the right :code:`l10n-id`/:code:`l10n-args` attributes like this: 576 577.. code-block:: javascript 578 579 testedFunction(); 580 581 const l10nAttrs = document.l10n.getAttributes(element); 582 583 deepEquals(l10nAttrs, { 584 id: "my-expected-id", 585 args: { 586 unreadCount: 5 587 } 588 }); 589 590If the code really has to test for particular values in the localized UI, it is 591always better to scan for a variable: 592 593.. code-block:: javascript 594 595 testedFunction(); 596 597 equals(element.textContent.contains("John")); 598 599.. important:: 600 601 Testing against whole values is brittle and will break when we insert Unicode 602 bidirectionality marks into the result string or adapt the output in other ways. 603 604 605Pseudolocalization 606================== 607 608When working with a Fluent-backed UI, the developer gets a new tool to test their UI 609against several classes of problems. 610 611Pseudolocalization is a mechanism which transforms messages on the fly, using 612specific logic to help emulate how the UI will look once it gets localized. 613 614The three classes of potential problems that this can help with are: 615 616 - Hardcoded strings. 617 618 Turning on pseudolocalization should expose any strings that were left 619 hardcoded in the source, since they won't get transformed. 620 621 622 - UI space not adapting to longer text. 623 624 Many languages use longer strings than English. For example, German strings 625 may be 30% longer (or more). Turning on pseudolocalization is a quick way to 626 test how the layout handles such locales. Strings that don't fit the space 627 available are truncated and pseudolocalization can also help with detecting them. 628 629 630 - Bidi adaptation. 631 632 For many developers, testing the UI in right-to-left mode is hard. 633 Pseudolocalization shows how a right-to-left locale will look like. 634 635To turn on pseudolocalization, add a new string pref :js:`intl.l10n.pseudo` and 636select the strategy to be used: 637 638 - :js:`accented` - [Ȧȧƈƈḗḗƞŧḗḗḓ Ḗḗƞɠŀīīşħ] 639 640 This strategy replaces all Latin characters with their accented equivalents, 641 and duplicates some vowels to create roughly 30% longer strings. Strings are 642 wrapped in markers (square brackets), which help with detecting truncation. 643 644 645 - :js:`bidi` - ɥsıʅƃuƎ ıpıԐ 646 647 This strategy replaces all Latin characters with their 180 degree rotated versions 648 and enforces right to left text flow using Unicode UAX#9 `Explicit Directional Embeddings`__. 649 In this mode, the UI directionality will also be set to right-to-left. 650 651__ https://www.unicode.org/reports/tr9/#Explicit_Directional_Embeddings 652 653Inner Structure of Fluent 654========================= 655 656The inner structure of Fluent in Gecko is out of scope of this tutorial, but 657since the class and file names may show up during debugging or profiling, 658below is a list of major components, each with a corresponding file in `/intl/l10n` 659modules in Gecko. 660 661For more hands-on experience with some of the concepts below, try 662following the `Fluent DOMLocalization Tutorial`__, which provides some 663background on how Fluent works and walks you through creating a basic 664web project from scratch that uses Fluent for localization. 665 666__ https://projectfluent.org/dom-l10n-documentation/overview.html 667 668FluentBundle 669-------------- 670 671FluentBundle is the lowest level API. It's fully synchronous, contains a parser for the 672FTL file format and a resolver for the logic. It is not meant to be used by 673consumers directly. 674 675In the future we intend to offer this layer for standardization and it may become 676part of the :js:`mozIntl.*` or even :js:`Intl.*` API sets. 677 678That part of the codebase is also the first that we'll be looking to port to Rust. 679 680 681Localization 682------------ 683 684Localization is a higher level API which uses :js:`FluentBundle` internally but 685provides a full layer of compound message formatting and robust error fall-backing. 686 687It is intended for use in runtime code and contains all fundamental localization 688methods. 689 690 691DOMLocalization 692--------------- 693 694DOMLocalization extends :js:`Localization` with functionality to operate on HTML, XUL 695and the DOM directly including DOM Overlays and Mutation Observers. 696 697DocumentL10n 698------------ 699 700DocumentL10n implements the DocumentL10n WebIDL API and allows Document to 701communicate with DOMLocalization. 702 703L10nRegistry 704------------ 705 706L10nRegistry is our resource management service. It 707maintains the state of resources packaged into the build and language packs, 708providing an asynchronous iterator of :js:`FluentBundle` objects for a given locale set 709and resources that the :js:`Localization` class uses. 710 711 712.. _Fluent: https://projectfluent.org/ 713.. _DTD: https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XUL/Tutorial/Localization 714.. _StringBundle: https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XUL/Tutorial/Property_Files 715.. _Firefox Preferences: https://bugzilla.mozilla.org/show_bug.cgi?id=1415730 716.. _Unprivileged Contexts: https://bugzilla.mozilla.org/show_bug.cgi?id=1407418 717.. _System Add-ons: https://bugzilla.mozilla.org/show_bug.cgi?id=1425104 718.. _CLDR: http://cldr.unicode.org/ 719.. _ICU: http://site.icu-project.org/ 720.. _Unicode: https://www.unicode.org/ 721.. _Fluent Syntax Guide: https://projectfluent.org/fluent/guide/ 722.. _Pontoon: https://pontoon.mozilla.org/ 723.. _Plural Rules: http://cldr.unicode.org/index/cldr-spec/plural-rules 724