1# IOUtils Migration Guide 2 3**Improving performance through a new file API** 4 5--- 6 7## What is IOUtils? 8 9`IOUtils` is a privileged JavaScript API for performing file I/O in the Firefox frontend. 10It was developed as a replacement for `OS.File`, addressing 11[bug 1231711](https://bugzilla.mozilla.org/show_bug.cgi?id=1231711). 12It is *not to be confused* with the unprivileged 13[DOM File API](https://developer.mozilla.org/en-US/docs/Web/API/File). 14 15`IOUtils` provides a minimal API surface to perform common 16I/O tasks via a collection of static methods inspired from `OS.File`. 17It is implemented in C++, and exposed to JavaScript via WebIDL bindings. 18 19The most up-to-date API can always be found in 20[IOUtils.webidl](https://searchfox.org/mozilla-central/source/dom/chrome-webidl/IOUtils.webidl). 21 22## Differences from `OS.File` 23 24`IOUtils` has a similar API to `OS.File`, but one should keep in mind some key differences. 25 26### No `File` instances (except `SyncReadFile` in workers) 27 28Most of the `IOUtils` methods only operate on absolute path strings, and don't expose a file handle to the caller. 29The exception to this rule is the `openFileForSyncReading` API, which is only available in workers. 30 31Furthermore, `OS.File` was exposing platform-specific file descriptors through the 32[`fd`](https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules/OSFile.jsm/OS.File_for_workers#Attributes) 33attribute. `IOUtils` does not expose file descriptors. 34 35### WebIDL has no `Date` type 36 37`IOUtils` is written in C++ and exposed to JavaScript through WebIDL. 38Many uses of `OS.File` concern themselves with obtaining or manipulating file metadata, 39like the last modified time, however the `Date` type does not exist in WebIDL. 40Using `IOUtils`, 41these values are returned to the caller as the number of milliseconds since 42`1970-01-01T00:00:00Z`. 43`Date`s can be safely constructed from these values if needed. 44 45For example, to obtain the last modification time of a file and update it to the current time: 46 47```js 48let { lastModified } = await IOUtils.stat(path); 49 50let lastModifiedDate = new Date(lastModified); 51 52let now = new Date(); 53 54await IOUtils.touch(path, now.valueOf()); 55``` 56 57### Some methods are not implemented 58 59For various reasons 60(complexity, safety, availability of underlying system calls, usefulness, etc.) 61the following `OS.File` methods have no analogue in IOUtils. 62They also will **not** be implemented. 63 64- void unixSymlink(in string targetPath, in string createPath) 65- string getCurrentDirectory(void) 66- void setCurrentDirectory(in string path) 67- object open(in string path) 68- object openUnique(in string path) 69 70### Errors are reported as `DOMException`s 71 72When an `OS.File` method runs into an error, 73it will throw/reject with a custom 74[`OS.File.Error`](https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules/OSFile.jsm/OS.File.Error). 75These objects have custom attributes that can be checked for common error cases. 76 77`IOUtils` has similar behaviour, however its methods consistently reject with a 78[`DOMException`](https://developer.mozilla.org/en-US/docs/Web/API/DOMException) 79whose name depends on the failure: 80 81| Exception Name | Reason for exception | 82| -------------- | -------------------- | 83| `NotFoundError` | A file at the specified path could not be found on disk. | 84| `NotAllowedError` | Access to a file at the specified path was denied by the operating system. | 85| `NotReadableError` | A file at the specified path could not be read for some reason. It may have been too big to read, or it was corrupt, or some other reason. The exception message should have more details. | 86| `ReadOnlyError` | A file at the specified path is read only and could not be modified. | 87| `NoModificationAllowedError` | A file already exists at the specified path and could not be overwritten according to the specified options. The exception message should have more details. | 88| `OperationError` | Something went wrong during the I/O operation. E.g. failed to allocate a buffer. The exception message should have more details. | 89| `UnknownError` | An unknown error occurred in the implementation. An nsresult error code should be included in the exception message to assist with debugging and improving `IOUtils` internal error handling. | 90 91### `IOUtils` is mostly async-only 92 93`OS.File` provided an asynchronous front-end for main-thread consumers, 94and a synchronous front-end for workers. 95`IOUtils` only provides an asynchronous API for the vast majority of its API surface. 96These asynchronous methods can be called from both the main thread and from chrome-privileged worker threads. 97 98The one exception to this rule is `openFileForSyncReading`, which allows synchronous file reading in workers. 99 100## `OS.File` vs `IOUtils` 101 102Some methods and options of `OS.File` keep the same name and underlying behaviour in `IOUtils`, 103but others have been renamed. 104The following is a detailed comparison with examples of the methods and options in each API. 105 106### Reading a file 107 108`IOUtils` provides the following methods to read data from a file. Like 109`OS.File`, they accept an `options` dictionary. 110 111Note: The maximum file size that can be read is `UINT32_MAX` bytes. Attempting 112to read a file larger will result in a `NotReadableError`. 113 114```idl 115Promise<Uint8Array> read(DOMString path, ...); 116 117Promise<DOMString> readUTF8(DOMString path, ...); 118 119Promise<any> readJSON(DOMString path, ...); 120 121// Workers only: 122SyncReadFile openFileForSyncReading(DOMString path); 123``` 124 125#### Options 126 127| `OS.File` option | `IOUtils` option | Description | 128| ------------------ | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | 129| bytes: number? | maxBytes: number? | If specified, read only up to this number of bytes. Otherwise, read the entire file. Default is null. | 130| compression: 'lz4' | decompress: boolean | If true, read the file and return the decompressed LZ4 stream. Otherwise, just read the file byte-for-byte. Default is false. | 131| encoding: 'utf-8' | N/A; use `readUTF8` instead. | Interprets the file as UTF-8 encoded text, and returns a string to the caller. | 132 133#### Examples 134 135##### Read raw (unsigned) byte values 136 137**`OS.File`** 138```js 139let bytes = await OS.File.read(path); // Uint8Array 140``` 141**`IOUtils`** 142```js 143let bytes = await IOUtils.read(path); // Uint8Array 144``` 145 146##### Read UTF-8 encoded text 147 148**`OS.File`** 149```js 150let utf8 = await OS.File.read(path, { encoding: 'utf-8' }); // string 151``` 152**`IOUtils`** 153```js 154let utf8 = await IOUtils.readUTF8(path); // string 155``` 156 157##### Read JSON file 158 159**`IOUtils`** 160```js 161let obj = await IOUtils.readJSON(path); // object 162``` 163 164##### Read LZ4 compressed file contents 165 166**`OS.File`** 167```js 168// Uint8Array 169let bytes = await OS.File.read(path, { compression: 'lz4' }); 170// string 171let utf8 = await OS.File.read(path, { 172 encoding: 'utf-8', 173 compression: 'lz4', 174}); 175``` 176**`IOUtils`** 177```js 178let bytes = await IOUtils.read(path, { decompress: true }); // Uint8Array 179let utf8 = await IOUtils.readUTF8(path, { decompress: true }); // string 180``` 181 182##### Synchronously read a fragment of a file into a buffer, from a worker 183 184**`OS.File`** 185```js 186// Read 64 bytes at offset 128, workers only: 187let file = OS.File.open(path, { read: true }); 188file.setPosition(128); 189let bytes = file.read({ bytes: 64 }); // Uint8Array 190file.close(); 191``` 192**`IOUtils`** 193```js 194// Read 64 bytes at offset 128, workers only: 195let file = IOUtils.openFileForSyncReading(path); 196let bytes = new Uint8Array(64); 197file.readBytesInto(bytes, 128); 198file.close(); 199``` 200 201### Writing to a file 202 203IOUtils provides the following methods to write data to a file. Like 204OS.File, they accept an options dictionary. 205 206```idl 207Promise<unsigned long long> write(DOMString path, Uint8Array data, ...); 208 209Promise<unsigned long long> writeUTF8(DOMString path, DOMString string, ...); 210 211Promise<unsigned long long> writeJSON(DOMString path, any value, ...); 212``` 213 214#### Options 215 216| `OS.File` option | `IOUtils` option | Description | 217| -------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | 218| backupTo: string? | backupFile: string? | Identifies the path to backup the target file to before performing the write operation. If unspecified, no backup will be performed. Default is null. | 219| tmpPath: string? | tmpPath: string? | Identifies a path to write to first, before performing a move to overwrite the target file. If unspecified, the target file will be written to directly. Default is null. | 220| noOverwrite: boolean | noOverwrite: boolean | If true, fail if the destination already exists. Default is false. | 221| flush: boolean | flush: boolean | If true, force the OS to flush its internal buffers to disk. Default is false. | 222| encoding: 'utf-8' | N/A; use `writeUTF8` instead. | Allows the caller to supply a string to be encoded as utf-8 text on disk. | 223 224#### Examples 225##### Write raw (unsigned) byte values 226 227**`OS.File`** 228```js 229let bytes = new Uint8Array(); 230await OS.File.writeAtomic(path, bytes); 231``` 232 233**`IOUtils`** 234```js 235let bytes = new Uint8Array(); 236await IOUtils.write(path, bytes); 237``` 238 239##### Write UTF-8 encoded text 240 241**`OS.File`** 242```js 243let str = ""; 244await OS.File.writeAtomic(path, str, { encoding: 'utf-8' }); 245``` 246 247**`IOUtils`** 248```js 249let str = ""; 250await IOUtils.writeUTF8(path, str); 251``` 252 253##### Write A JSON object 254 255**`IOUtils`** 256```js 257let obj = {}; 258await IOUtils.writeJSON(path, obj); 259``` 260 261##### Write with LZ4 compression 262 263**`OS.File`** 264```js 265let bytes = new Uint8Array(); 266await OS.File.writeAtomic(path, bytes, { compression: 'lz4' }); 267let str = ""; 268await OS.File.writeAtomic(path, str, { 269 compression: 'lz4', 270}); 271``` 272 273**`IOUtils`** 274```js 275let bytes = new Uint8Array(); 276await IOUtils.write(path, bytes, { compress: true }); 277let str = ""; 278await IOUtils.writeUTF8(path, str, { compress: true }); 279``` 280 281### Move a file 282 283`IOUtils` provides the following method to move files on disk. 284Like `OS.File`, it accepts an options dictionary. 285 286```idl 287Promise<void> move(DOMString sourcePath, DOMString destPath, ...); 288``` 289 290#### Options 291 292| `OS.File` option | `IOUtils` option | Description | 293| -------------------- | ----------------------------------- | ---------------------------------------------------------------------------- | 294| noOverwrite: boolean | noOverwrite: boolean | If true, fail if the destination already exists. Default is false. | 295| noCopy: boolean | N/A; will not be implemented | This option is not implemented in `IOUtils`, and will be ignored if provided | 296 297#### Example 298 299**`OS.File`** 300```js 301await OS.File.move(srcPath, destPath); 302``` 303 304**`IOUtils`** 305```js 306await IOUtils.move(srcPath, destPath); 307``` 308 309### Remove a file 310 311`IOUtils` provides *one* method to remove files from disk. 312`OS.File` provides several methods. 313 314```idl 315Promise<void> remove(DOMString path, ...); 316``` 317 318#### Options 319 320| `OS.File` option | `IOUtils` option | Description | 321| ---------------------------------------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------- | 322| ignoreAbsent: boolean | ignoreAbsent: boolean | If true, and the destination does not exist, then do not raise an error. Default is true. | 323| N/A; `OS.File` has dedicated methods for directory removal | recursive: boolean | If true, and the target is a directory, recursively remove the directory and all its children. Default is false. | 324 325#### Examples 326 327##### Remove a file 328 329**`OS.File`** 330```js 331await OS.File.remove(path, { ignoreAbsent: true }); 332``` 333 334**`IOUtils`** 335```js 336await IOUtils.remove(path); 337``` 338 339##### Remove a directory and all its contents 340 341**`OS.File`** 342```js 343await OS.File.removeDir(path, { ignoreAbsent: true }); 344``` 345 346**`IOUtils`** 347```js 348await IOUtils.remove(path, { recursive: true }); 349``` 350 351##### Remove an empty directory 352 353**`OS.File`** 354```js 355await OS.File.removeEmptyDir(path); // Will throw an exception if `path` is not empty. 356``` 357 358**`IOUtils`** 359```js 360await IOUtils.remove(path); // Will throw an exception if `path` is not empty. 361``` 362 363### Make a directory 364 365`IOUtils` provides the following method to create directories on disk. 366Like `OS.File`, it accepts an options dictionary. 367 368```idl 369Promise<void> makeDirectory(DOMString path, ...); 370``` 371 372#### Options 373 374| `OS.File` option | `IOUtils` option | Description | 375| ----------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | 376| ignoreExisting: boolean | ignoreExisting: boolean | If true, succeed even if the target directory already exists. Default is true. | 377| from: string | createAncestors: boolean | If true, `IOUtils` will create all missing ancestors in a path. Default is true. This option differs from `OS.File`, which requires the caller to specify a root path from which to create missing directories. | 378| unixMode | N/A | `IOUtils` does not support setting a custom directory mode on unix. | 379| winSecurity | N/A | `IOUtils` does not support setting custom directory security settings on Windows. | 380 381#### Example 382 383**`OS.File`** 384```js 385await OS.File.makeDir(srcPath, destPath); 386``` 387**`IOUtils`** 388```js 389await IOUtils.makeDirectory(srcPath, destPath); 390``` 391 392### Update a file's modification time 393 394`IOUtils` provides the following method to update a file's modification time. 395 396```idl 397Promise<void> setModificationTime(DOMString path, optional long long modification); 398``` 399 400#### Example 401 402**`OS.File`** 403```js 404await OS.File.setDates(path, new Date(), new Date()); 405``` 406 407**`IOUtils`** 408```js 409await IOUtils.setModificationTime(path, new Date().valueOf()); 410``` 411 412### Get file metadata 413 414`IOUtils` provides the following method to query file metadata. 415 416```idl 417Promise<void> stat(DOMString path); 418``` 419 420#### Example 421 422**`OS.File`** 423```js 424let fileInfo = await OS.File.stat(path); 425``` 426 427**`IOUtils`** 428```js 429let fileInfo = await IOUtils.stat(path); 430``` 431 432### Copy a file 433 434`IOUtils` provides the following method to copy a file on disk. 435Like `OS.File`, it accepts an options dictionary. 436 437```idl 438Promise<void> copy(DOMString path, ...); 439``` 440 441#### Options 442 443| `OS.File` option | `IOUtils` option | Description | 444| ------------------------------------------------------------------- | -------------------- | -------------------------------------------------------------------| 445| noOverwrite: boolean | noOverwrite: boolean | If true, fail if the destination already exists. Default is false. | 446| N/A; `OS.File` does not appear to support recursively copying files | recursive: boolean | If true, copy the source recursively. | 447 448#### Examples 449 450##### Copy a file 451 452**`OS.File`** 453```js 454await OS.File.copy(srcPath, destPath); 455``` 456 457**`IOUtils`** 458```js 459await IOUtils.copy(srcPath, destPath); 460``` 461 462##### Copy a directory recursively 463 464**`OS.File`** 465```js 466// Not easy to do. 467``` 468 469**`IOUtils`** 470```js 471await IOUtils.copy(srcPath, destPath, { recursive: true }); 472``` 473 474### Iterate a directory 475 476At the moment, `IOUtils` does not have a way to expose an iterator for directories. 477This is blocked by 478[bug 1577383](https://bugzilla.mozilla.org/show_bug.cgi?id=1577383). 479As a stop-gap for this functionality, 480one can get all the children of a directory and iterate through the returned path array using the following method. 481 482```idl 483Promise<sequence<DOMString>> getChildren(DOMString path); 484``` 485 486#### Example 487 488**`OS.File`** 489```js 490for await (const { path } of new OS.FileDirectoryIterator(dirName)) {</p> 491 ... 492} 493``` 494 495**`IOUtils`** 496```js 497for (const path of await IOUtils.getChildren(dirName)) { 498 ... 499} 500``` 501 502### Check if a file exists 503 504`IOUtils` provides the following method analogous to the `OS.File` method of the same name. 505 506```idl 507Promise<boolean> exists(DOMString path); 508``` 509 510#### Example 511 512**`OS.File`** 513```js 514if (await OS.File.exists(path)) { 515 ... 516} 517``` 518 519**`IOUtils`** 520```js 521if (await IOUtils.exists(path)) { 522 ... 523} 524``` 525 526### Set the permissions of a file 527 528`IOUtils` provides the following method analogous to the `OS.File` method of the same name. 529 530```idl 531Promise<void> setPermissions(DOMString path, unsigned long permissions, optional boolean honorUmask = true); 532``` 533 534#### Options 535 536| `OS.File` option | `IOUtils` option | Description | 537| ----------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | 538| unixMode: number | permissions: usigned long | The UNIX file mode representing the permissions. Required in IOUtils. | 539| unixHonorUmask: boolean | honorUmask: boolean | If omitted or true, any UNIX file mode is modified by the permissions. Otherwise the exact value of the permissions will be applied. | 540 541#### Example 542 543**`OS.File`** 544```js 545await OS.File.setPermissions(path, { unixMode: 0o600 }); 546``` 547 548**`IOUtils`** 549```js 550await IOUtils.setPermissions(path, 0o600); 551``` 552 553## FAQs 554 555**Why should I use `IOUtils` instead of `OS.File`?** 556 557[Bug 1231711](https://bugzilla.mozilla.org/show_bug.cgi?id=1231711) 558provides some good context, but some reasons include: 559* reduced cache-contention, 560* faster startup, and 561* less memory usage. 562 563Additionally, `IOUtils` benefits from a native implementation, 564which assists in performance-related work for 565[Project Fission](https://hacks.mozilla.org/2021/05/introducing-firefox-new-site-isolation-security-architecture/). 566 567We are actively working to migrate old code usages of `OS.File` 568to analogous `IOUtils` calls, so new usages of `OS.File` 569should not be introduced at this time. 570 571**Do I need to import anything to use this API?** 572 573Nope! It's available via the `IOUtils` global in JavaScript (`ChromeOnly` context). 574 575**Can I use this API from C++ or Rust?** 576 577Currently usage is geared exclusively towards JavaScript callers, 578and all C++ methods are private except for the Web IDL bindings. 579However given sufficient interest, 580it should be easy to expose ergonomic public methods for C++ and/or Rust. 581 582**Why isn't `IOUtils` written in Rust?** 583 584At the time of writing, 585support for Web IDL bindings was more mature for C++ oriented tooling than it was for Rust. 586 587**Is `IOUtils` feature complete? When will it be available?** 588 589`IOUtils` is considered feature complete as of Firefox 83. 590