1# IOUtils Migration Guide
2
3**Improving performance through a new file API**
4
5---
6
7## What is IOUtils?
8
9`IOUtils` is a privileged JavaScript API for performing file I/O in the Firefox frontend.
10It was developed as a replacement for `OS.File`, addressing
11[bug 1231711](https://bugzilla.mozilla.org/show_bug.cgi?id=1231711).
12It is *not to be confused* with the unprivileged
13[DOM File API](https://developer.mozilla.org/en-US/docs/Web/API/File).
14
15`IOUtils` provides a minimal API surface to perform common
16I/O tasks via a collection of static methods inspired from `OS.File`.
17It is implemented in C++, and exposed to JavaScript via WebIDL bindings.
18
19The most up-to-date API can always be found in
20[IOUtils.webidl](https://searchfox.org/mozilla-central/source/dom/chrome-webidl/IOUtils.webidl).
21
22## Differences from `OS.File`
23
24`IOUtils` has a similar API to `OS.File`, but one should keep in mind some key differences.
25
26### No `File` instances (except `SyncReadFile` in workers)
27
28Most of the `IOUtils` methods only operate on absolute path strings, and don't expose a file handle to the caller.
29The exception to this rule is the `openFileForSyncReading` API, which is only available in workers.
30
31Furthermore, `OS.File` was exposing platform-specific file descriptors through the
32[`fd`](https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules/OSFile.jsm/OS.File_for_workers#Attributes)
33attribute. `IOUtils` does not expose file descriptors.
34
35### WebIDL has no `Date` type
36
37`IOUtils` is written in C++ and exposed to JavaScript through WebIDL.
38Many uses of `OS.File` concern themselves with obtaining or manipulating file metadata,
39like the last modified time, however the `Date` type does not exist in WebIDL.
40Using `IOUtils`,
41these values are returned to the caller as the number of milliseconds since
42`1970-01-01T00:00:00Z`.
43`Date`s can be safely constructed from these values if needed.
44
45For example, to obtain the last modification time of a file and update it to the current time:
46
47```js
48let { lastModified } = await IOUtils.stat(path);
49
50let lastModifiedDate = new Date(lastModified);
51
52let now = new Date();
53
54await IOUtils.touch(path, now.valueOf());
55```
56
57### Some methods are not implemented
58
59For various reasons
60(complexity, safety, availability of underlying system calls, usefulness, etc.)
61the following `OS.File` methods have no analogue in IOUtils.
62They also will **not** be implemented.
63
64-   void unixSymlink(in string targetPath, in string createPath)
65-   string getCurrentDirectory(void)
66-   void setCurrentDirectory(in string path)
67-   object open(in string path)
68-   object openUnique(in string path)
69
70### Errors are reported as `DOMException`s
71
72When an `OS.File` method runs into an error,
73it will throw/reject with a custom
74[`OS.File.Error`](https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules/OSFile.jsm/OS.File.Error).
75These objects have custom attributes that can be checked for common error cases.
76
77`IOUtils` has similar behaviour, however its methods consistently reject with a
78[`DOMException`](https://developer.mozilla.org/en-US/docs/Web/API/DOMException)
79whose name depends on the failure:
80
81| Exception Name | Reason for exception |
82| -------------- | -------------------- |
83| `NotFoundError` | A file at the specified path could not be found on disk. |
84| `NotAllowedError` | Access to a file at the specified path was denied by the operating system. |
85| `NotReadableError` | A file at the specified path could not be read for some reason. It may have been too big to read, or it was corrupt, or some other reason. The exception message should have more details. |
86| `ReadOnlyError` | A file at the specified path is read only and could not be modified. |
87| `NoModificationAllowedError` | A file already exists at the specified path and could not be overwritten according to the specified options. The exception message should have more details. |
88| `OperationError` | Something went wrong during the I/O operation. E.g. failed to allocate a buffer. The exception message should have more details. |
89| `UnknownError` | An unknown error occurred in the implementation. An nsresult error code should be included in the exception message to assist with debugging and improving `IOUtils` internal error handling. |
90
91### `IOUtils` is mostly async-only
92
93`OS.File` provided an asynchronous front-end for main-thread consumers,
94and a synchronous front-end for workers.
95`IOUtils` only provides an asynchronous API for the vast majority of its API surface.
96These asynchronous methods can be called from both the main thread and from chrome-privileged worker threads.
97
98The one exception to this rule is `openFileForSyncReading`, which allows synchronous file reading in workers.
99
100## `OS.File` vs `IOUtils`
101
102Some methods and options of `OS.File` keep the same name and underlying behaviour in `IOUtils`,
103but others have been renamed.
104The following is a detailed comparison with examples of the methods and options in each API.
105
106### Reading a file
107
108`IOUtils` provides the following methods to read data from a file. Like
109`OS.File`, they accept an `options` dictionary.
110
111Note: The maximum file size that can be read is `UINT32_MAX` bytes. Attempting
112to read a file larger will result in a `NotReadableError`.
113
114```idl
115Promise<Uint8Array> read(DOMString path, ...);
116
117Promise<DOMString> readUTF8(DOMString path, ...);
118
119Promise<any> readJSON(DOMString path, ...);
120
121// Workers only:
122SyncReadFile openFileForSyncReading(DOMString path);
123```
124
125#### Options
126
127| `OS.File` option   | `IOUtils` option             | Description                                                                                                               |
128| ------------------ | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
129| bytes: number?     | maxBytes: number?            | If specified, read only up to this number of bytes. Otherwise, read the entire file. Default is null.                         |
130| compression: 'lz4' | decompress: boolean          | If true, read the file and return the decompressed LZ4 stream. Otherwise, just read the file byte-for-byte. Default is false. |
131| encoding: 'utf-8'  | N/A; use `readUTF8` instead. | Interprets the file as UTF-8 encoded text, and returns a string to the caller.                                                |
132
133#### Examples
134
135##### Read raw (unsigned) byte values
136
137**`OS.File`**
138```js
139let bytes = await OS.File.read(path); // Uint8Array
140```
141**`IOUtils`**
142```js
143let bytes = await IOUtils.read(path); // Uint8Array
144```
145
146##### Read UTF-8 encoded text
147
148**`OS.File`**
149```js
150let utf8 = await OS.File.read(path, { encoding: 'utf-8' }); // string
151```
152**`IOUtils`**
153```js
154let utf8 = await IOUtils.readUTF8(path); // string
155```
156
157##### Read JSON file
158
159**`IOUtils`**
160```js
161let obj = await IOUtils.readJSON(path); // object
162```
163
164##### Read LZ4 compressed file contents
165
166**`OS.File`**
167```js
168// Uint8Array
169let bytes = await OS.File.read(path, { compression: 'lz4' });
170// string
171let utf8 = await OS.File.read(path, {
172    encoding: 'utf-8',
173    compression: 'lz4',
174});
175```
176**`IOUtils`**
177```js
178let bytes = await IOUtils.read(path, { decompress: true }); // Uint8Array
179let utf8 = await IOUtils.readUTF8(path, { decompress: true }); // string
180```
181
182##### Synchronously read a fragment of a file into a buffer, from a worker
183
184**`OS.File`**
185```js
186// Read 64 bytes at offset 128, workers only:
187let file = OS.File.open(path, { read: true });
188file.setPosition(128);
189let bytes = file.read({ bytes: 64 }); // Uint8Array
190file.close();
191```
192**`IOUtils`**
193```js
194// Read 64 bytes at offset 128, workers only:
195let file = IOUtils.openFileForSyncReading(path);
196let bytes = new Uint8Array(64);
197file.readBytesInto(bytes, 128);
198file.close();
199```
200
201### Writing to a file
202
203IOUtils provides the following methods to write data to a file. Like
204OS.File, they accept an options dictionary.
205
206```idl
207Promise<unsigned long long> write(DOMString path, Uint8Array data, ...);
208
209Promise<unsigned long long> writeUTF8(DOMString path, DOMString string, ...);
210
211Promise<unsigned long long> writeJSON(DOMString path, any value, ...);
212```
213
214#### Options
215
216| `OS.File` option     | `IOUtils` option              | Description                                                                                                                                                               |
217| -------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
218| backupTo: string?    | backupFile: string?           | Identifies the path to backup the target file to before performing the write operation. If unspecified, no backup will be performed. Default is null.                     |
219| tmpPath: string?     | tmpPath: string?              | Identifies a path to write to first, before performing a move to overwrite the target file. If unspecified, the target file will be written to directly. Default is null. |
220| noOverwrite: boolean | noOverwrite: boolean          | If true, fail if the destination already exists. Default is false.                                                                                                        |
221| flush: boolean       | flush: boolean                | If true, force the OS to flush its internal buffers to disk. Default is false.                                                                                            |
222| encoding: 'utf-8'    | N/A; use `writeUTF8` instead. | Allows the caller to supply a string to be encoded as utf-8 text on disk.                                                                                                 |
223
224#### Examples
225##### Write raw (unsigned) byte values
226
227**`OS.File`**
228```js
229let bytes = new Uint8Array();
230await OS.File.writeAtomic(path, bytes);
231```
232
233**`IOUtils`**
234```js
235let bytes = new Uint8Array();
236await IOUtils.write(path, bytes);
237```
238
239##### Write UTF-8 encoded text
240
241**`OS.File`**
242```js
243let str = "";
244await OS.File.writeAtomic(path, str, { encoding: 'utf-8' });
245```
246
247**`IOUtils`**
248```js
249let str = "";
250await IOUtils.writeUTF8(path, str);
251```
252
253##### Write A JSON object
254
255**`IOUtils`**
256```js
257let obj = {};
258await IOUtils.writeJSON(path, obj);
259```
260
261##### Write with LZ4 compression
262
263**`OS.File`**
264```js
265let bytes = new Uint8Array();
266await OS.File.writeAtomic(path, bytes, { compression: 'lz4' });
267let str = "";
268await OS.File.writeAtomic(path, str, {
269    compression: 'lz4',
270});
271```
272
273**`IOUtils`**
274```js
275let bytes = new Uint8Array();
276await IOUtils.write(path, bytes, { compress: true });
277let str = "";
278await IOUtils.writeUTF8(path, str, { compress: true });
279```
280
281### Move a file
282
283`IOUtils` provides the following method to move files on disk.
284Like `OS.File`, it accepts an options dictionary.
285
286```idl
287Promise<void> move(DOMString sourcePath, DOMString destPath, ...);
288```
289
290#### Options
291
292| `OS.File` option     | `IOUtils` option                    | Description                                                                                                                                                               |
293| -------------------- | ----------------------------------- | ---------------------------------------------------------------------------- |
294| noOverwrite: boolean | noOverwrite: boolean                | If true, fail if the destination already exists. Default is false.           |
295| noCopy: boolean      | N/A; will not be implemented        | This option is not implemented in `IOUtils`, and will be ignored if provided |
296
297#### Example
298
299**`OS.File`**
300```js
301await OS.File.move(srcPath, destPath);
302```
303
304**`IOUtils`**
305```js
306await IOUtils.move(srcPath, destPath);
307```
308
309### Remove a file
310
311`IOUtils` provides *one* method to remove files from disk.
312`OS.File` provides several methods.
313
314```idl
315Promise<void> remove(DOMString path, ...);
316```
317
318#### Options
319
320| `OS.File` option                                           | `IOUtils` option      | Description                                                                                                      |
321| ---------------------------------------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------- |
322| ignoreAbsent: boolean                                      | ignoreAbsent: boolean | If true, and the destination does not exist, then do not raise an error. Default is true.                        |
323| N/A; `OS.File` has dedicated methods for directory removal | recursive: boolean    | If true, and the target is a directory, recursively remove the directory and all its children. Default is false. |
324
325#### Examples
326
327##### Remove a file
328
329**`OS.File`**
330```js
331await OS.File.remove(path, { ignoreAbsent: true });
332```
333
334**`IOUtils`**
335```js
336await IOUtils.remove(path);
337```
338
339##### Remove a directory and all its contents
340
341**`OS.File`**
342```js
343await OS.File.removeDir(path, { ignoreAbsent: true });
344```
345
346**`IOUtils`**
347```js
348await IOUtils.remove(path, { recursive: true });
349```
350
351##### Remove an empty directory
352
353**`OS.File`**
354```js
355await OS.File.removeEmptyDir(path); // Will throw an exception if `path` is not empty.
356```
357
358**`IOUtils`**
359```js
360await IOUtils.remove(path); // Will throw an exception if `path` is not empty.
361```
362
363### Make a directory
364
365`IOUtils` provides the following method to create directories on disk.
366Like `OS.File`, it accepts an options dictionary.
367
368```idl
369Promise<void> makeDirectory(DOMString path, ...);
370```
371
372#### Options
373
374| `OS.File` option        | `IOUtils` option         | Description                                                                                                                                                                                                     |
375| ----------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
376| ignoreExisting: boolean | ignoreExisting: boolean  | If true, succeed even if the target directory already exists. Default is true.                                                                                                                                  |
377| from: string            | createAncestors: boolean | If true, `IOUtils` will create all missing ancestors in a path. Default is true. This option differs from `OS.File`, which requires the caller to specify a root path from which to create missing directories. |
378| unixMode                | N/A                      | `IOUtils` does not support setting a custom directory mode on unix.                                                                                                                                             |
379| winSecurity             | N/A                      | `IOUtils` does not support setting custom directory security settings on Windows.                                                                                                                               |
380
381#### Example
382
383**`OS.File`**
384```js
385await OS.File.makeDir(srcPath, destPath);
386```
387**`IOUtils`**
388```js
389await IOUtils.makeDirectory(srcPath, destPath);
390```
391
392### Update a file's modification time
393
394`IOUtils` provides the following method to update a file's modification time.
395
396```idl
397Promise<void> setModificationTime(DOMString path, optional long long modification);
398```
399
400#### Example
401
402**`OS.File`**
403```js
404await OS.File.setDates(path, new Date(), new Date());
405```
406
407**`IOUtils`**
408```js
409await IOUtils.setModificationTime(path, new Date().valueOf());
410```
411
412### Get file metadata
413
414`IOUtils` provides the following method to query file metadata.
415
416```idl
417Promise<void> stat(DOMString path);
418```
419
420#### Example
421
422**`OS.File`**
423```js
424let fileInfo = await OS.File.stat(path);
425```
426
427**`IOUtils`**
428```js
429let fileInfo = await IOUtils.stat(path);
430```
431
432### Copy a file
433
434`IOUtils` provides the following method to copy a file on disk.
435Like `OS.File`, it accepts an options dictionary.
436
437```idl
438Promise<void> copy(DOMString path, ...);
439```
440
441#### Options
442
443| `OS.File` option                                                    | `IOUtils` option     | Description                                                    |
444| ------------------------------------------------------------------- | -------------------- | -------------------------------------------------------------------|
445| noOverwrite: boolean                                                | noOverwrite: boolean | If true, fail if the destination already exists. Default is false. |
446| N/A; `OS.File` does not appear to support recursively copying files | recursive: boolean   | If true, copy the source recursively.                              |
447
448#### Examples
449
450##### Copy a file
451
452**`OS.File`**
453```js
454await OS.File.copy(srcPath, destPath);
455```
456
457**`IOUtils`**
458```js
459await IOUtils.copy(srcPath, destPath);
460```
461
462##### Copy a directory recursively
463
464**`OS.File`**
465```js
466// Not easy to do.
467```
468
469**`IOUtils`**
470```js
471await IOUtils.copy(srcPath, destPath, { recursive: true });
472```
473
474### Iterate a directory
475
476At the moment, `IOUtils` does not have a way to expose an iterator for directories.
477This is blocked by
478[bug 1577383](https://bugzilla.mozilla.org/show_bug.cgi?id=1577383).
479As a stop-gap for this functionality,
480one can get all the children of a directory and iterate through the returned path array using the following method.
481
482```idl
483Promise<sequence<DOMString>> getChildren(DOMString path);
484```
485
486#### Example
487
488**`OS.File`**
489```js
490for await (const { path } of new OS.FileDirectoryIterator(dirName)) {</p>
491  ...
492}
493```
494
495**`IOUtils`**
496```js
497for (const path of await IOUtils.getChildren(dirName)) {
498  ...
499}
500```
501
502### Check if a file exists
503
504`IOUtils` provides the following method analogous to the `OS.File` method of the same name.
505
506```idl
507Promise<boolean> exists(DOMString path);
508```
509
510#### Example
511
512**`OS.File`**
513```js
514if (await OS.File.exists(path)) {
515  ...
516}
517```
518
519**`IOUtils`**
520```js
521if (await IOUtils.exists(path)) {
522  ...
523}
524```
525
526### Set the permissions of a file
527
528`IOUtils` provides the following method analogous to the `OS.File` method of the same name.
529
530```idl
531Promise<void> setPermissions(DOMString path, unsigned long permissions, optional boolean honorUmask = true);
532```
533
534#### Options
535
536| `OS.File` option        | `IOUtils` option          | Description                                                                                                                          |
537| ----------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
538| unixMode: number        | permissions: usigned long | The UNIX file mode representing the permissions. Required in IOUtils.                                                                |
539| unixHonorUmask: boolean | honorUmask: boolean       | If omitted or true, any UNIX file mode is modified by the permissions. Otherwise the exact value of the permissions will be applied. |
540
541#### Example
542
543**`OS.File`**
544```js
545await OS.File.setPermissions(path, { unixMode: 0o600 });
546```
547
548**`IOUtils`**
549```js
550await IOUtils.setPermissions(path, 0o600);
551```
552
553## FAQs
554
555**Why should I use `IOUtils` instead of `OS.File`?**
556
557[Bug 1231711](https://bugzilla.mozilla.org/show_bug.cgi?id=1231711)
558provides some good context, but some reasons include:
559* reduced cache-contention,
560* faster startup, and
561* less memory usage.
562
563Additionally, `IOUtils` benefits from a native implementation,
564which assists in performance-related work for
565[Project Fission](https://hacks.mozilla.org/2021/05/introducing-firefox-new-site-isolation-security-architecture/).
566
567We are actively working to migrate old code usages of `OS.File`
568to analogous `IOUtils` calls, so new usages of `OS.File`
569should not be introduced at this time.
570
571**Do I need to import anything to use this API?**
572
573Nope! It's available via the `IOUtils` global in JavaScript (`ChromeOnly` context).
574
575**Can I use this API from C++ or Rust?**
576
577Currently usage is geared exclusively towards JavaScript callers,
578and all C++ methods are private except for the Web IDL bindings.
579However given sufficient interest,
580it should be easy to expose ergonomic public methods for C++ and/or Rust.
581
582**Why isn't `IOUtils` written in Rust?**
583
584At the time of writing,
585support for Web IDL bindings was more mature for C++ oriented tooling than it was for Rust.
586
587**Is `IOUtils` feature complete? When will it be available?**
588
589`IOUtils` is considered feature complete as of Firefox 83.
590