1
2 Firefox Sync Support in Epiphany
3
4 Overview
5 --------
6
7 Firefox Sync [0] is a browser synchronization feature developed by Mozilla
8 and is currently used to synchronize data between desktop and mobile Firefox
9 browsers. The way it is designed ensures data security in such a manner that
10 not even Mozilla can read the user data on the storage servers.
11
12 To synchronize data via Firefox Sync, users must own a Firefox account.
13
14 Behind the Scenes
15 -----------------
16
17 Firefox Sync relies on the existence of a server to store the synchronized
18 records. This server is called the Sync Storage Server [1] and the current
19 version of its API is 1.5 [2], which is being used since 2014. Each record
20 that is synchronized (a.k.a. BSO - Basic Storage Object) is grouped with
21 other related records into collections. Besides the default collections
22 (i.e. bookmarks, history, passwords, tabs), there are additional collections
23 (clients, crypto, meta) used for management purposes.
24
25 After a quick glance at the API, you may notice that clients access the stored
26 records on the server relatively easy, via HTTP requests. However, there are a
27 couple of aspects that need to be taken into consideration before being able
28 to communicate with the Sync Storage Server:
29
30 1) What is the URL of the Sync Storage Server and how do you obtain it?
31 Moreover, since every request sent to the Sync Storage Server needs to
32 be authenticated with Hawk [3], how do you obtain the Hawk id and key?
33
34 2) How do you decrypt the response so that you can actually "see" the stored
35 record? As mentioned in the Mozilla docs, the only job of the Sync Storage
36 Server is to store the records (it does not alter or delete them).
37 So it falls into the responsibility of the clients to encrypt the records
38 that are uploaded to the server.
39
40 These aspects will be covered in the following chapters.
41
42 Obtaining the Storage Credentials
43 ---------------------------------
44
45 To obtain the storage credentials (i.e. the URL of the Sync Storage Server and
46 the Hawk id and key), the sync clients have to interact with the Token Server
47 [4][5]. In the context of Firefox Sync, the role of the Token Server is to
48 share the URL of the Sync Storage Server together with a pair of Hawk id and
49 key in exchange for a BrowserID assertion [6]. However, the Hawk id and key
50 have a limited expiry time, so clients need to take that into consideration
51 and request new credentials when the previous ones have expired.
52
53 So the next question that raises is: how do you make the BrowserID assertion?
54 More exactly, since every BrowserID assertion requires a signed identity
55 certificate, where do you get the certificate from?
56
57 The certificate is obtained from the Firefox Accounts Server [7][8], more
58 specifically from the /certificate/sign endpoint. As you can see in the API,
59 requests to this endpoint have to be Hawk-authenticated based on a so-called
60 sessionToken (a 32 bytes token) that is obtained from the /account/login
61 endpoint (this endpoint does not require Hawk authentication). Details about
62 how the email and the password of the user's Firefox account are stretched to
63 obtain the request body of the login request can be found here [9], however it
64 is of no big interest since this is made automatically by the Firefox Accounts
65 Content Server [10] (how Epiphany uses the Firefox Accounts Content Server to
66 do the login will be explained later). What is important to know is how the
67 sessionToken is derived to obtain the Hawk id and key that will be used to
68 authenticate the request to the certificate endpoint. The process is explained
69 here [11] and it involves the sessionToken being fed into HKDF [12] to obtain
70 the Hawk id and key.
71
72 To summarize, the steps of obtaining the storage credentials are:
73
74 1. Login with the Firefox account and obtain a sessionToken. This is a one
75 time step since the sessionToken lasts forever until revoked by a password
76 change or explicit revocation command (via the /session/destroy endpoint of
77 the Firefox Accounts Server) and can be used an unlimited number of times.
78
79 2. Based on the sessionToken, obtain a signed identity certificate from the
80 /certificate/sign endpoint of the Firefox Accounts Server. The certificate
81 has a limited lifetime of 24 hours.
82
83 3. Create the BrowserID assertion from the previously obtained certificate.
84
85 4. Send a request to the Token Server with the Authorization HTTP header set
86 to the BrowserID assertion. The Token Server will respond with the URL of
87 the Sync Storage Server and the Hawk id and key together with the validity
88 duration.
89
90 5. When the validity duration has expired, repeat from step 2.
91
92 Encrypting and Decrypting Records
93 ---------------------------------
94
95 Every collection on the Sync Storage Server has a key bundle associated
96 formed of two keys: a symmetric encryption key and an HMAC key. The former
97 is used to encrypt and decrypt the records with AES-256, while the latter
98 is used to verify the records using HMAC hashing. Both keys are 32 bytes.
99 The hashing algorithm used is SHA-256. Besides the bundles associated with
100 each collection, there is also a default key bundle which is supposed to be
101 used when handling records belonging to a collection that has no key bundle
102 associated.
103
104 All the key bundles (including the default one) are stored in the crypto/keys
105 record [13] on the Sync Storage Server. This is a normal record, but with a
106 special meaning: being a record that holds information about the keys used to
107 encrypt/decrypt all the other records, it cannot be encrypted with any of
108 those keys for obvious reasons. Therefore it is encrypted and verified with a
109 different key bundle derived from the Sync Key or the Master Sync Key.
110
111 The Master Sync Key is a 32 bytes token available only to sync clients and
112 is obtained from the Firefox Accounts Server via the /account/keys endpoint.
113 This endpoint enforces the requests to be Hawk-authenticated based on a
114 keyFetchToken. The keyFetchToken is also a 32 bytes token and is obtained at
115 login along with the sessionToken. The process of deriving the keyFetchToken
116 into the Hawk id and key used to authenticate the request and the process of
117 extracting the Master Sync Key from response bundle are thoroughly explained
118 here [14]. Note that the Master Sync Key is referred there as kB. In short,
119 the keyFetchToken is used to derive four other tokens through two HKDF
120 processes. The first HKDF process derives the Hawk id and key together with
121 a new keying material that serves as input for the second HKDF process.
122 The second HKDF process derives a response HMAC key and a response XOR key.
123 In response to the Hawk request, the server sends in return a bundle which
124 holds a ciphertext and a pre-calculated HMAC value. Clients use the response
125 HMAC key to compute the HMAC value of the ciphertext to validate it. After
126 that, the ciphertext is XORed with the response XOR key to obtain a 64 bytes
127 token. The first 32 bytes represent kA which is left unused. The last 32 bytes
128 are XORed with unwrapBKey to obtain kB (a.k.a the Master Sync Key). unwrapBKey
129 is yet another 32 bytes token returned at login together with sessionToken
130 and keyFetchToken). The hashing algorithm used is SHA-256.
131
132 Note that the Master Sync Key is an immutable token which is generated when
133 the Firefox account is created. Therefore, it should be considered a secret
134 and not ever be displayed in plain text as it could lead to the account's
135 data on the Sync Storage Server being compromised.
136
137 Having the Master Sync Key, deriving the key bundle that is used to encrypt
138 and verify the crypto/keys record is rather trivial: just perform a two-step
139 HKDF with an all-zeros salt. T(1) will represent the AES encryption key and
140 T(2) will represent the HMAC key. Having this key bundle, the crypto/keys
141 record can be decrypted to extract the default key bundle together with the
142 per-collection key bundles.
143
144 After that, clients are ready to upload/download records to/from the Sync
145 Storage Server. The flow when uploading a record is:
146
147 1. Serialize the object representing a bookmark/history/password/tab into
148 a JSON object. The stringified JSON object will represent the cleartext.
149
150 2. Encrypt the cleartext with AES-256 using the encryption key from the
151 corresponding collection's key bundle or from the default key bundle if
152 the collection does not have a key bundle associated. As an initialization
153 vector (IV) for AES-256, a random 16 bytes token will be used. AES-256
154 will output the ciphertext which will be base64 encoded afterward.
155
156 3. Compute the HMAC value of the base64 encoded ciphertext using the HMAC
157 key from the corresponding collection's key bundle or from the default
158 key bundle if the collection does not have a key bundle associated.
159 The hashing algorithm used is SHA-256.
160
161 4. Create a JSON object containing the base64 encoded ciphertext, the
162 base64 encoded initialization vector and the hex encoded HMAC value.
163 The stringified JSON object will represent the payload of the BSO
164 that will be uploaded to the Sync Storage Server.
165
166 5. Create a JSON object containing the id of the record and the previously
167 computed payload. The id of the record is a base64 URL-safe string that
168 is randomly generated (however when updating a record the id must be
169 preserved) and is usually 12 characters long. The stringified JSON object
170 will represent the body of the request sent to the Sync Storage Server.
171 Of course, the request will be Hawk-authenticated with the Hawk id and key
172 obtained from the Token Server.
173
174 The flow is reversed when downloading a record.
175
176 More details about the cryptography of the Sync Storage Server can be found
177 here [15].
178
179 Firefox Objects Formats
180 -----------------------
181
182 The Firefox format of the objects uploaded to the Sync Storage Server is
183 described here [16]. You will notice there are multiple versions described
184 for each collection, however, all collections currently use version 1, except
185 for bookmarks which use version 2. All formats are JSON objects.
186
187 In Epiphany, these formats are described by the GObject properties of the
188 objects that are synchronized:
189
190 * EphyBookmark for bookmarks
191 * EphyPasswordRecord for passwords
192 * EphyHistoryRecord for history
193 * EphyOpenTabsRecord for tabs
194
195 All these objects implement the JsonSerializable interface which allows
196 GObjects to be serialized into JSON strings and to be constructed from JSON
197 strings based on the properties of the object. They also implement the
198 EphySynchronizable interface which describes an object viable to become a BSO.
199
200 Signing in to Firefox Sync in Epiphany
201 --------------------------------------
202
203 As mentioned in the previous chapters, a vital part of Firefox Sync is
204 signing in with the email and password of the Firefox account and obtaining
205 a sessionToken, a keyFetchToken and an unwrapBKey that are further used to
206 access data on the Sync Storage Server.
207
208 In Epiphany, users can sign in with an existing Firefox account or create
209 a new one via the Sync tab in the preferences dialog. After the sign in has
210 completed successfully, users can customize their sync experience by choosing
211 what collections to sync (bookmarks, history, password and open tabs), whether
212 to sync or not with Firefox and the sync frequency.
213
214 However, it is important to understand what happens behind the scenes when
215 users sign in. The preferences dialog displays a Firefox iframe where users
216 type their email and password of their Firefox account and click Sign In.
217 The Firefox iframe is actually the web interface of the Firefox Accounts
218 Server, called the Firefox Accounts Content Server. This is the preferred
219 way of communicating sync related information from the Firefox Accounts
220 Server to Firefox Sync clients (the other way is by direct requests to the
221 login endpoint of the Firefox Accounts Server but that has been pretty much
222 restricted from public access). Communication with the Firefox Accounts
223 Content Server is made via the WebChannels flow [17] which is briefly
224 described further.
225
226 The Firefox iframe is loaded in a WebKitWebView inside the Sync tab of the
227 preferences dialog. A JavaScript that listens to WebChannelMessageToChrome
228 events is added to the WebKitUserContentManager of the WebKitWebView.
229 WebChannelMessageToChrome events are messages that come from the Firefox
230 Accounts Content Server to the web browser. When such an event is received,
231 the JavaScript forwards it to the WebKitUserContentManager via the
232 "script-message-received" signal. The callback connected to this signal in
233 the preferences dialog will parse the message and respond with a
234 WebChannelMessageToContent event through webkit_web_view_run_javascript()
235 if necessary. WebChannelMessageToContent events are messages that go from
236 the web browser to the Firefox Accounts Content Server.
237 Both WebChannelMessageToChrome and WebChannelMessageToContent events have a
238 "detail" JSON object that has the following members: the id of the WebChannel
239 and a "message" JSON object.
240
241 The "message" JSON object has the following members:
242
243 * command: string, one of "fxaccounts:can_link_account", "fxaccounts:login",
244 "fxaccounts:delete_account", "fxaccounts:change_password",
245 "fxaccounts:loaded", "profile:change".
246
247 * messageId: string, a unique identifier that should be kept the same when
248 responding to a message.
249
250 * data: JSON object, optional, carries the actual data of the message.
251
252 The WebChannelMessageToChrome/WebChannelMessageToContent sign in flow is:
253
254 1. "fxaccounts:loaded" command is received via WebChannelMessageToChrome when
255 the Firefox iframe is loaded. No data is sent and no response is expected.
256
257 2. "fxaccounts:can_link_account" command is received via
258 WebChannelMessageToChrome when the user has entered the credentials and
259 submitted the form. The data received contains the email of the user.
260 A response with an "ok" field is expected.
261
262 3. A WebChannelMessageToContent message is sent in response to the previous
263 command. The fields detail.message.command, detail.message.messageId and
264 detail.id are kept the same. The field detail.message.data is set to
265 {ok: true}.
266
267 4. "fxaccounts:login" command is received via WebChannelMessageToChrome.
268 No response is expected. The data field contains the sessionToken,
269 keyFetchToken and unwrapBKey tokens amongst other. Now the client has
270 everything it needs and the sync can begin.
271
272 After the sync tokens have been received via "fxaccounts:login" command in
273 the preferences dialog, they are passed to EphySyncService via the _sign_in()
274 function. After that, EphySyncService takes care of the rest:
275
276 * It retrieves the Master Sync Key.
277
278 * It verifies the version of the Sync Storage Server. EphySyncService only
279 supports version 1.5 so the sign in will fail in case it detects a lower
280 version. The version is kept on the Sync Storage Server in the meta/global
281 record [18]. This is a special record that is not encrypted and contains
282 general information about the state of the Sync Storage Server. In case
283 the meta/global record is missing (this happens when the Firefox account
284 is newly created), EphySyncService will generate and upload a new one.
285
286 * It retrieves account's crypto keys from the Sync Storage Server. In case
287 the crypto/keys record does not exist (this happens when the Firefox
288 account is newly created), EphySyncService will generate and upload a new
289 crypto/keys record that contains a randomly generated default key bundle.
290
291 * It registers the current device in the clients collection on the Sync
292 Storage Server.
293
294 * It stores the sync secrets (sessionToken, uid, Master Sync Key, crypto
295 keys) in the sync SecretSchema. They are kept there until sign out when
296 the SecretSchema is cleared. At this point, the sign in is considered
297 complete and the Firefox iframe is replaced with the panel that displays
298 the sync configuration options.
299
300 Note that the above steps are chronological. If any failure happens at any
301 point, EphySyncService will abort and report a sign in error next to the
302 Firefox iframe.
303
304 Sync Modules in Epiphany
305 ------------------------
306
307 I. EphySyncService
308
309 Synchronization in Epiphany is made via EphySyncService, which is a
310 singleton object residing in EphyShell, being accessible anywhere in
311 src/ via ephy_shell_get_sync_service(). However, EphySyncService is designed
312 to operate mostly on its own so you probably won't have to deal with it in
313 newly written code. EphySyncService handles all aspects of the communication
314 with the Sync Storage Server (uploads and downloads records), Token Server
315 (gets the storage credentials) and Firefox Accounts Server (gets account keys,
316 gets certificates, destroys the session). It also schedules and does
317 periodical synchronizations via every collection's manager.
318
319 The requests to the Sync Storage Server are sent internally via
320 ephy_sync_service_queue_storage_request(). This checks whether the storage
321 credentials are expired or not. If not expired, the request is sent directly
322 via ephy_sync_service_send_storage_request(), otherwise, the request is put
323 in a message queue and new storage credentials are obtained. When the new
324 storage credentials have been obtained, the queue will be emptied and all
325 requests will be sent.
326
327 The API of EphySyncService is rather simple. It contains functions to sign in,
328 sign out, start periodical synchronization and do a synchronization. Besides
329 these, there are four other functions:
330
331 * _new(). This is the constructor. It receives a boolean that says whether
332 this sync service should do periodical synchronizations. This is needed
333 because passwords are saved from EphyWebExtension which runs in another
334 process (the web process). For that, EphyWebExtension needs to have its
335 own EphySyncService that will upload/update/delete passwords once they are
336 saved/modified/deleted but will not do any periodical synchronizations.
337 Periodical synchronizations (which synchronize all records from all
338 collections) will only be made by the EphySyncService that belongs to the
339 UI process.
340
341 * _register_device(). This function adds the current device with the given
342 name to the clients collection on the Sync Storage Server. The clients
343 collection is a special collection which stores data about the devices
344 connected to the Firefox account. It is worth mentioning that every device
345 is identified by a device id and a device name. The device id is randomly
346 generated at login by EphySyncService and cannot be changed by users.
347 The device name can be edited by users from the preferences dialog.
348 If no device name is chosen, then the default is used: "@username's
349 Epiphany on @hostname".
350
351 * _register_manager() and _unregister_manager(). These will be explained in
352 the context of EphySynchronizableManager.
353
354 The sign out function is more of a cleanup function: it stops the periodical
355 synchronization, unregisters the device by deleting the associated record
356 from the clients collection, deletes the associated record from the tabs
357 collection, destroys the session, clears the message queue, unregisters
358 all managers and clears the SecretSchema that contains the sync secrets.
359
360 II. EphySynchronizableManager
361
362 EphySynchronizableManager is an interface that describes the common
363 functionality of every collection and is implemented by every collection
364 manager: EphyBookmarksManager, EphyPasswordManager, EphyHistoryManager and
365 EphyOpenTabsManager. All these managers are also singleton objects residing
366 in EphyShell and are accessible in src/ via ephy_shell_get_<manager>().
367 The main reason why such an interface is needed is that EphySyncService is
368 defined under lib/ which makes it unable to access objects from src/ (i.e.
369 EphyBookmark and EphyBookmarksManager) so it needs to access them through
370 a delegate interface. EphySyncService is defined under lib/ because it is
371 required by EphyWebExtension which is defined under embed/. (See section
372 LAYERING in HACKING). For the same reason, EphySyncService has functions to
373 register/unregister EphySynchronizableManagers. Managers are registered in
374 EphyShell when EphySyncService is created based on the user preferences stored
375 in the GSettings schema. Managers are also registered and unregistered when
376 users toggle the check buttons that say whether a collection should be
377 synchronized or not in the preferences dialog.
378
379 The API of EphySynchronizableManager is described in the source code via
380 documentation comments. EphySynchronizableManager provides two signals:
381 "synchronizable-modified" and "synchronizable-deleted". The implementations
382 of EphySynchronizableManager trigger the first signal when an object has
383 been added or modified so it needs to be uploaded to be Sync Storage Server
384 and the second signal when an object has been deleted locally so it needs to
385 be deleted from the Sync Storage Server too. EphySyncService connects a
386 callback to these signals for every manager that has been registered to it
387 and disconnects it when the manager is unregistered. This way any local
388 changes to the synchronized objects will be mirrored instantly on the Sync
389 Storage Server. However, in case EphySyncService finds a newer version of
390 the object on the server, it will download it.
391
392 Note that when a record is deleted from the Sync Storage Server, it does not
393 disappear from the server. It is only marked as deleted with a "deleted" flag
394 set to true. This way other sync clients will know that the record has been
395 deleted by another client and will delete it too from their local collection.
396 See ephy_sync_debug_delete_record() vs ephy_sync_debug_erase_record() for more
397 details about this.
398
399 The synchronization merge logic of every collection is described in the
400 source code comments of every manager's merge function.
401
402 III. EphySynchronizable
403
404 EphySynchronizable is another delegate interface that describes the
405 objects that are uploaded and downloaded from the Sync Storage Server.
406 It is implemented by EphyBookmark, EphyPasswordRecord, EphyHistoryRecord
407 and EphyOpenTabsRecord which also implement the JsonSerializable interface
408 so that they can be converted to BSOs. EphySynchronizable objects are managed
409 by the associated EphySynchronizableManager. The API of EphySynchronizable is
410 described in the source code via documentation comments.
411
412 IV. EphySyncCrypto
413
414 EphySyncCrypto is a helper module that handles all the cryptographic stuff.
415 Its API include:
416
417 * _derive_session_token(). Derives the Hawk id and key from a sessionToken.
418
419 * _derive_key_fetch_token(). Derives the Hawk id and key, the response HMAC
420 key and the response XOR key from a keyFetchToken.
421
422 * _derive_master_keys(). Derives the Master Sync Key from the unwrapBKey
423 token, the bundle returned by the /accounts/keys endpoint of the Firefox
424 Accounts Server, the response HMAC key and the response XOR key. kB is
425 the Master Sync Key and kA is left unused.
426
427 * _derive_master_bundle(). Derives the key bundle from the Master Sync Key.
428
429 * _generate_crypto_keys(). Generates a new crypto/keys record.
430
431 * _encrypt_record(). Encrypts a cleartext into a BSO payload.
432
433 * _decrypt_record(). Decrypts a BSO payload into a cleartext.
434
435 * _rsa_key_pair_new(). Generates an RSA key pair. This is needed when
436 obtaining an identity certificate and creating the BrowserID assertion.
437
438 * _create_assertion(). Creates a BrowserID assertion from a certificate, an
439 audience and an RSA key pair.
440
441 * _hawk_header_new(). Creates a Hawk header that is used to authenticate
442 Hawk requests. Unfortunately, there isn't any C library for creating Hawk
443 headers so the code has been reproduced from a Python library [19].
444
445 V. EphySyncDebug
446
447 EphySyncDebug is a helper module for debugging purposes. All its functions
448 use only synchronous API calls and should not be used in production code.
449 Its API is described in the source code via documentation comments.
450
451 References
452 ----------
453
454 [0] https://wiki.mozilla.org/Services/Sync
455 [1] https://github.com/mozilla-services/server-syncstorage
456 [2] https://mozilla-services.readthedocs.io/en/latest/storage/apis-1.5.html
457 [3] https://github.com/hueniverse/hawk/blob/master/README.md
458 [4] https://mozilla-services.readthedocs.io/en/latest/token/index.html
459 [5] https://github.com/mozilla-services/tokenserver
460 [6] https://fuller.li/posts/how-does-browserid-work/
461 [7] https://mozilla-services.readthedocs.io/en/latest/fxa/index.html
462 [8] https://github.com/mozilla/fxa-auth-server/
463 [9] https://github.com/mozilla/fxa-auth-server/wiki/onepw-protocol#login-obtaining-the-sessiontoken
464 [10] https://github.com/mozilla/fxa-content-server/
465 [11] https://github.com/mozilla/fxa-auth-server/wiki/onepw-protocol#signing-certificates
466 [12] https://tools.ietf.org/html/rfc5869
467 [13] https://mozilla-services.readthedocs.io/en/latest/sync/storageformat5.html#crypto-keys-record
468 [14] https://github.com/mozilla/fxa-auth-server/wiki/onepw-protocol#-fetching-sync-keys
469 [15] https://mozilla-services.readthedocs.io/en/latest/sync/storageformat5.html#cryptography
470 [16] https://mozilla-services.readthedocs.io/en/latest/sync/objectformats.html
471 [17] https://github.com/mozilla/fxa-content-server/blob/master/docs/relier-communication-protocols/fx-webchannel.md
472 [18] https://mozilla-services.readthedocs.io/en/latest/sync/storageformat5.html#metaglobal-record
473 [19] https://github.com/mozilla/PyHawk
474