README.md
1[![CI](https://github.com/WebAssembly/binaryen/workflows/CI/badge.svg?branch=master&event=push)](https://github.com/WebAssembly/binaryen/actions?query=workflow%3ACI)
2
3# Binaryen
4
5Binaryen is a compiler and toolchain infrastructure library for WebAssembly,
6written in C++. It aims to make [compiling to WebAssembly] **easy, fast, and
7effective**:
8
9 * **Easy**: Binaryen has a simple [C API] in a single header, and can also be
10 [used from JavaScript][JS_API]. It accepts input in [WebAssembly-like
11 form][compile_to_wasm] but also accepts a general [control flow graph] for
12 compilers that prefer that.
13
14 * **Fast**: Binaryen's internal IR uses compact data structures and is designed
15 for completely parallel codegen and optimization, using all available CPU
16 cores. Binaryen's IR also compiles down to WebAssembly extremely easily and
17 quickly because it is essentially a subset of WebAssembly.
18
19 * **Effective**: Binaryen's optimizer has many passes (see an overview later
20 down) that can improve code size and speed. These optimizations aim to make
21 Binaryen powerful enough to be used as a [compiler backend][backend] by
22 itself. One specific area of focus is on WebAssembly-specific optimizations
23 (that general-purpose compilers might not do), which you can think of as
24 wasm [minification], similar to minification for JavaScript, CSS, etc., all
25 of which are language-specific.
26
27Compilers using Binaryen include:
28
29 * [`AssemblyScript`](https://github.com/AssemblyScript/assemblyscript) which compiles a subset of TypeScript to WebAssembly
30 * [`wasm2js`](https://github.com/WebAssembly/binaryen/blob/master/src/wasm2js.h) which compiles WebAssembly to JS
31 * [`Asterius`](https://github.com/tweag/asterius) which compiles Haskell to WebAssembly
32 * [`Grain`](https://github.com/grain-lang/grain) which compiles Grain to WebAssembly
33
34Binaryen also provides a set of **toolchain utilities** that can
35
36 * **Parse** and **emit** WebAssembly. In particular this lets you load
37 WebAssembly, optimize it using Binaryen, and re-emit it, thus implementing a
38 wasm-to-wasm optimizer in a single command.
39 * **Interpret** WebAssembly as well as run the WebAssembly spec tests.
40 * Integrate with **[Emscripten](http://emscripten.org)** in order to provide a
41 complete compiler toolchain from C and C++ to WebAssembly.
42 * **Polyfill** WebAssembly by running it in the interpreter compiled to
43 JavaScript, if the browser does not yet have native support (useful for
44 testing).
45
46Consult the [contributing instructions](Contributing.md) if you're interested in
47participating.
48
49## Binaryen IR
50
51Binaryen's internal IR is designed to be
52
53 * **Flexible and fast** for optimization.
54 * **As close as possible to WebAssembly** so it is simple and fast to convert
55 it to and from WebAssembly.
56
57There are a few differences between Binaryen IR and the WebAssembly language:
58
59 * Tree structure
60 * Binaryen IR [is a tree][binaryen_ir], i.e., it has hierarchical structure,
61 for convenience of optimization. This differs from the WebAssembly binary
62 format which is a stack machine.
63 * Consequently Binaryen's text format allows only s-expressions.
64 WebAssembly's official text format is primarily a linear instruction list
65 (with s-expression extensions). Binaryen can't read the linear style, but
66 it can read a wasm text file if it contains only s-expressions.
67 * Binaryen uses Stack IR to optimize "stacky" code (that can't be
68 represented in structured form).
69 * When stacky code must be represented in Binaryen IR, such as with
70 multivalue instructions and blocks, it is represented with tuple types that
71 do not exist in the WebAssembly language. In addition to multivalue
72 instructions, locals and globals can also have tuple types in Binaryen IR
73 but not in WebAssembly.
74 * Types and unreachable code
75 * WebAssembly limits block/if/loop types to none and the concrete value types
76 (i32, i64, f32, f64). Binaryen IR has an unreachable type, and it allows
77 block/if/loop to take it, allowing [local transforms that don't need to
78 know the global context][unreachable]. As a result, Binaryen's default
79 text output is not necessarily valid wasm text. (To get valid wasm text,
80 you can do `--generate-stack-ir --print-stack-ir`, which prints Stack IR,
81 this is guaranteed to be valid for wasm parsers.)
82 * Binaryen ignores unreachable code when reading WebAssembly binaries. That
83 means that if you read a wasm file with unreachable code, that code will be
84 discarded as if it were optimized out (often this is what you want anyhow,
85 and optimized programs have no unreachable code anyway, but if you write an
86 unoptimized file and then read it, it may look different). The reason for
87 this behavior is that unreachable code in WebAssembly has corner cases that
88 are tricky to handle in Binaryen IR (it can be very unstructured, and
89 Binaryen IR is more structured than WebAssembly as noted earlier). Note
90 that Binaryen does support unreachable code in .wat text files, since as we
91 saw Binaryen only supports s-expressions there, which are structured.
92 * Blocks
93 * Binaryen IR has only one node that contains a variable-length list of
94 operands: the block. WebAssembly on the other hand allows lists in loops,
95 if arms, and the top level of a function. Binaryen's IR has a single
96 operand for all non-block nodes; this operand may of course be a block.
97 The motivation for this property is that many passes need special code
98 for iterating on lists, so having a single IR node with a list simplifies
99 them.
100 * As in wasm, blocks and loops may have names. Branch targets in the IR are
101 resolved by name (as opposed to nesting depth). This has 2 consequences:
102 * Blocks without names may not be branch targets.
103 * Names are required to be unique. (Reading .wat files with duplicate names
104 is supported; the names are modified when the IR is constructed).
105 * As an optimization, a block that is the child of a loop (or if arm, or
106 function toplevel) and which has no branches targeting it will not be
107 emitted when generating wasm. Instead its list of operands will be directly
108 used in the containing node. Such a block is sometimes called an "implicit
109 block".
110 * Multivalue
111 * Binaryen will not represent multivalue instructions and values directly.
112 Binaryen's main focus is on optimization of wasm, and therefore the question
113 of whether we should have multivalue in the main IR is whether it justifes
114 the extra complexity there. Experiments show that the shrinking of code
115 size thanks to multivalue is useful but small, just 1-3% or so. Given that,
116 we prefer to keep the main IR simple, and focus on multivalue optimizations
117 in Stack IR, which is more suitable for such things.
118 * Binaryen does still need to implement the "ABI" level of multivalue, that
119 is, we need multivalue calls because those may cross module boundaries,
120 and so they are observable externally. To support that, Binaryen may use
121 `push` and `pop` as mentioned earlier; another option is to add LLVM-like
122 `extractvalue/composevalue` instructions.
123
124As a result, you might notice that round-trip conversions (wasm => Binaryen IR
125=> wasm) change code a little in some corner cases.
126
127 * When optimizing Binaryen uses an additional IR, Stack IR (see
128 `src/wasm-stack.h`). Stack IR allows a bunch of optimizations that are
129 tailored for the stack machine form of WebAssembly's binary format (but Stack
130 IR is less efficient for general optimizations than the main Binaryen IR). If
131 you have a wasm file that has been particularly well-optimized, a simple
132 round-trip conversion (just read and write, without optimization) may cause
133 more noticeable differences, as Binaryen fits it into Binaryen IR's more
134 structured format. If you also optimize during the round-trip conversion then
135 Stack IR opts will be run and the final wasm will be better optimized.
136
137Notes when working with Binaryen IR:
138
139 * As mentioned above, Binaryen IR has a tree structure. As a result, each
140 expression should have exactly one parent - you should not "reuse" a node by
141 having it appear more than once in the tree. The motivation for this
142 limitation is that when we optimize we modify nodes, so if they appear more
143 than once in the tree, a change in one place can appear in another
144 incorrectly.
145 * For similar reasons, nodes should not appear in more than one functions.
146
147## Tools
148
149This repository contains code that builds the following tools in `bin/`:
150
151 * **wasm-opt**: Loads WebAssembly and runs Binaryen IR passes on it.
152 * **wasm-as**: Assembles WebAssembly in text format (currently S-Expression
153 format) into binary format (going through Binaryen IR).
154 * **wasm-dis**: Un-assembles WebAssembly in binary format into text format
155 (going through Binaryen IR).
156 * **wasm2js**: A WebAssembly-to-JS compiler. This is used by Emscripten to
157 generate JavaScript as an alternative to WebAssembly.
158 * **wasm-reduce**: A testcase reducer for WebAssembly files. Given a wasm file
159 that is interesting for some reason (say, it crashes a specific VM),
160 wasm-reduce can find a smaller wasm file that has the same property, which is
161 often easier to debug. See the
162 [docs](https://github.com/WebAssembly/binaryen/wiki/Fuzzing#reducing)
163 for more details.
164 * **wasm-shell**: A shell that can load and interpret WebAssembly code. It can
165 also run the spec test suite.
166 * **wasm-emscripten-finalize**: Takes a wasm binary produced by llvm+lld and
167 performs emscripten-specific passes over it.
168 * **wasm-ctor-eval**: A tool that can execute C++ global constructors ahead of
169 time. Used by Emscripten.
170 * **binaryen.js**: A standalone JavaScript library that exposes Binaryen methods for [creating and optimizing WASM modules](https://github.com/WebAssembly/binaryen/blob/master/test/binaryen.js/hello-world.js). For builds, see [binaryen.js on npm](https://www.npmjs.com/package/binaryen) (or download it directly from [github](https://raw.githubusercontent.com/AssemblyScript/binaryen.js/master/index.js), [rawgit](https://cdn.rawgit.com/AssemblyScript/binaryen.js/master/index.js), or [unpkg](https://unpkg.com/binaryen@latest/index.js)).
171
172Usage instructions for each are below.
173
174## Binaryen Optimizations
175
176Binaryen contains
177[a lot of optimization passes](https://github.com/WebAssembly/binaryen/tree/master/src/passes)
178to make WebAssembly smaller and faster. You can run the Binaryen optimizer by
179using ``wasm-opt``, but also they can be run while using other tools, like
180``wasm2js`` and ``wasm-metadce``.
181
182* The default optimization pipeline is set up by functions like
183 [`addDefaultFunctionOptimizationPasses`](https://github.com/WebAssembly/binaryen/blob/369b8bdd3d9d49e4d9e0edf62e14881c14d9e352/src/passes/pass.cpp#L396).
184* There are various
185 [pass options](https://github.com/WebAssembly/binaryen/blob/369b8bdd3d9d49e4d9e0edf62e14881c14d9e352/src/pass.h#L85)
186 that you can set, to adjust the optimization and shrink levels, whether to
187 ignore unlikely traps, inlining heuristics, fast-math, and so forth. See
188 ``wasm-opt --help`` for how to set them and other details.
189
190See each optimization pass for details of what it does, but here is a quick
191overview of some of the relevant ones:
192
193* **CoalesceLocals** - Key “register allocation” pass. Does a live range
194 analysis and then reuses locals in order to minimize their number, as well as
195 to remove copies between them.
196* **CodeFolding** - Avoids duplicate code by merging it (e.g. if two `if` arms
197 have some shared instructions at their end).
198* **CodePushing** - “Pushes” code forward past branch operations, potentially
199 allowing the code to not be run if the branch is taken.
200* **DeadArgumentElimination** - LTO pass to remove arguments to a function if it
201 is always called with the same constants.
202* **DeadCodeElimination**
203* **Directize** - Turn an indirect call into a normal call, when the table index
204 is constant.
205* **DuplicateFunctionElimination** - LTO pass.
206* **Inlining** - LTO pass.
207* **LocalCSE** - Simple local common subexpression elimination.
208* **LoopInvariantCodeMotion**
209* **MemoryPacking** - Key "optimize data segments" pass that combines segments,
210 removes unneeded parts, etc.
211* **MergeBlocks** - Merge a `block` to an outer one where possible, reducing
212 their number.
213* **MergeLocals** - When two locals have the same value in part of their
214 overlap, pick in a way to help CoalesceLocals do better later (split off from
215 CoalesceLocals to keep the latter simple).
216* **MinifyImportsAndExports** - Minifies them to “a”, “b”, etc.
217* **OptimizeAddedConstants** - Optimize a load/store with an added constant into
218 a constant offset.
219* **OptimizeInstructions** - Key peephole optimization pass with a constantly
220 increasing list of patterns.
221* **PickLoadSigns** - Adjust whether a load is signed or unsigned in order to
222 avoid sign/unsign operations later.
223* **Precompute** - Calculates constant expressions at compile time, using the
224 built-in interpreter (which is guaranteed to be able to handle any constant
225 expression).
226* **ReReloop** - Transforms wasm structured control flow to a CFG and then goes
227 back to structured form using the Relooper algorithm, which may find more
228 optimal shapes.
229* **RedundantSetElimination** - Removes a `local.set` of a value that is already
230 present in a local. (Overlaps with CoalesceLocals; this achieves the specific
231 operation just mentioned without all the other work CoalesceLocals does, and
232 therefore is useful in other places in the optimization pipeline.)
233* **RemoveUnsedBrs** - Key “minor control flow optimizations” pass, including
234 jump threading and various transforms that can get rid of a `br` or `br_table`
235 (like turning a `block` with a `br` in the middle into an `if` when possible).
236* **RemoveUnusedModuleElements** - “Global DCE”, an LTO pass that removes
237 imports, functions, globals, etc., when they are not used.
238* **ReorderFunctions** - Put more-called functions first, potentially allowing
239 the LEB emitted to call them to be smaller (in a very large program).
240* **ReorderLocals** - Put more-used locals first, potentially allowing the LEB
241 emitted to use them to be smaller (in a very large function). After the
242 sorting, it also removes locals not used at all.
243* **SimplifyGlobals** - Optimizes globals in various ways, for example,
244 coalescing them, removing mutability from a global never modified, applying a
245 constant value from an immutable global, etc.
246* **SimplifyLocals** - Key “`local.get/set/tee`” optimization pass, doing things
247 like replacing a set and a get with moving the set’s value to the get (and
248 creating a tee) where possible. Also creates `block/if/loop` return values
249 instead of using a local to pass the value.
250* **Vacuum** - Key “remove silly unneeded code” pass, doing things like removing
251 an `if` arm that has no contents, a drop of a constant value with no side
252 effects, a `block` with a single child, etc.
253
254“LTO” in the above means an optimization is Link Time Optimization-like in that
255it works across multiple functions, but in a sense Binaryen is always “LTO” as
256it usually is run on the final linked wasm.
257
258Advanced optimization techniques in the Binaryen optimizer include
259[SSAification](https://github.com/WebAssembly/binaryen/blob/master/src/passes/SSAify.cpp),
260[Flat IR](https://github.com/WebAssembly/binaryen/blob/master/src/ir/flat.h), and
261[Stack/Poppy IR](https://github.com/WebAssembly/binaryen/blob/master/src/ir/stack-utils.h).
262
263Binaryen also contains various passes that do other things than optimizations,
264like
265[legalization for JavaScript](https://github.com/WebAssembly/binaryen/blob/master/src/passes/LegalizeJSInterface.cpp),
266[Asyncify](https://github.com/WebAssembly/binaryen/blob/master/src/passes/Asyncify.cpp),
267etc.
268
269## Building
270
271```
272cmake . && make
273```
274
275A C++14 compiler is required. Note that you can also use `ninja` as your generator: `cmake -G Ninja . && ninja`.
276
277Binaryen.js can be built using Emscripten, which can be installed via [the SDK](http://kripken.github.io/emscripten-site/docs/getting_started/downloads.html)).
278
279```
280emcmake cmake . && emmake make binaryen_js
281```
282
283### Visual C++
284
2851. Using the Microsoft Visual Studio Installer, install the "Visual C++ tools for CMake" component.
286
2871. Generate the projects:
288
289 ```
290 mkdir build
291 cd build
292 "%VISUAL_STUDIO_ROOT%\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" ..
293 ```
294
295 Substitute VISUAL_STUDIO_ROOT with the path to your Visual Studio
296 installation. In case you are using the Visual Studio Build Tools, the path
297 will be "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools".
298
2991. From the Developer Command Prompt, build the desired projects:
300
301 ```
302 msbuild binaryen.vcxproj
303 ```
304
305 CMake generates a project named "ALL_BUILD.vcxproj" for conveniently building all the projects.
306
307## Running
308
309### wasm-opt
310
311Run
312
313````
314bin/wasm-opt [.wasm or .wat file] [options] [passes, see --help] [--help]
315````
316
317The wasm optimizer receives WebAssembly as input, and can run transformation
318passes on it, as well as print it (before and/or after the transformations). For
319example, try
320
321````
322bin/wasm-opt test/passes/lower-if-else.wat --print
323````
324
325That will pretty-print out one of the test cases in the test suite. To run a
326transformation pass on it, try
327
328````
329bin/wasm-opt test/passes/lower-if-else.wat --print --lower-if-else
330````
331
332The `lower-if-else` pass lowers if-else into a block and a break. You can see
333the change the transformation causes by comparing the output of the two print
334commands.
335
336It's easy to add your own transformation passes to the shell, just add `.cpp`
337files into `src/passes`, and rebuild the shell. For example code, take a look at
338the [`lower-if-else` pass](https://github.com/WebAssembly/binaryen/blob/master/src/passes/LowerIfElse.cpp).
339
340Some more notes:
341
342 * See `bin/wasm-opt --help` for the full list of options and passes.
343 * Passing `--debug` will emit some debugging info.
344
345### wasm2js
346
347Run
348
349```
350bin/wasm2js [input.wasm file]
351```
352
353This will print out JavaScript to the console.
354
355For example, try
356
357```
358$ bin/wasm2js test/hello_world.wat
359```
360
361That output contains
362
363```
364 function add(x, y) {
365 x = x | 0;
366 y = y | 0;
367 return x + y | 0 | 0;
368 }
369```
370
371as a translation of
372
373```
374 (func $add (; 0 ;) (type $0) (param $x i32) (param $y i32) (result i32)
375 (i32.add
376 (local.get $x)
377 (local.get $y)
378 )
379 )
380```
381
382wasm2js's output is in ES6 module format - basically, it converts a wasm
383module into an ES6 module (to run on older browsers and Node.js versions
384you can use Babel etc. to convert it to ES5). Let's look at a full example
385of calling that hello world wat; first, create the main JS file:
386
387```javascript
388// main.mjs
389import { add } from "./hello_world.mjs";
390console.log('the sum of 1 and 2 is:', add(1, 2));
391```
392
393The run this (note that you need a new enough Node.js with ES6 module
394support):
395
396```shell
397$ bin/wasm2js test/hello_world.wat -o hello_world.mjs
398$ node --experimental-modules main.mjs
399the sum of 1 and 2 is: 3
400```
401
402Things keep to in mind with wasm2js's output:
403
404 * You should run wasm2js with optimizations for release builds, using `-O`
405 or another optimization level. That will optimize along the entire pipeline
406 (wasm and JS). It won't do everything a JS minifer would, though, like
407 minify whitespace, so you should still run a normal JS minifer afterwards.
408 * It is not possible to match WebAssembly semantics 100% precisely with fast
409 JavaScript code. For example, every load and store may trap, and to make
410 JavaScript do the same we'd need to add checks everywhere, which would be
411 large and slow. Instead, wasm2js assumes loads and stores do not trap, that
412 int/float conversions do not trap, and so forth. There may also be slight
413 differences in corner cases of conversions, like non-trapping float to int.
414
415## Testing
416
417```
418./check.py
419```
420
421(or `python check.py`) will run `wasm-shell`, `wasm-opt`, etc. on the testcases in `test/`, and verify their outputs.
422
423The `check.py` script supports some options:
424
425```
426./check.py [--interpreter=/path/to/interpreter] [TEST1] [TEST2]..
427```
428
429 * If an interpreter is provided, we run the output through it, checking for
430 parse errors.
431 * If tests are provided, we run exactly those. If none are provided, we run
432 them all. To see what tests are available, run `./check.py --list-suites`.
433 * Some tests require `emcc` or `nodejs` in the path. They will not run if the
434 tool cannot be found, and you'll see a warning.
435 * We have tests from upstream in `tests/spec`, in git submodules. Running
436 `./check.py` should update those.
437
438### Setting up dependencies
439
440```
441./third_party/setup.py [mozjs|v8|wabt|all]
442```
443
444(or `python third_party/setup.py`) installs required dependencies like the SpiderMonkey JS shell, the V8 JS shell
445and WABT in `third_party/`. Other scripts automatically pick these up when installed.
446
447### Fuzzing
448
449```
450./scripts/fuzz_opt.py [--binaryen-bin=build/bin]
451```
452
453(or `python scripts/fuzz_opt.py`) will run various fuzzing modes on random inputs with random passes until it finds
454a possible bug. See [the wiki page](https://github.com/WebAssembly/binaryen/wiki/Fuzzing) for all the details.
455
456## Design Principles
457
458 * **Interned strings for names**: It's very convenient to have names on nodes,
459 instead of just numeric indices etc. To avoid most of the performance
460 difference between strings and numeric indices, all strings are interned,
461 which means there is a single copy of each string in memory, string
462 comparisons are just a pointer comparison, etc.
463 * **Allocate in arenas**: Based on experience with other
464 optimizing/transformating toolchains, it's not worth the overhead to
465 carefully track memory of individual nodes. Instead, we allocate all elements
466 of a module in an arena, and the entire arena can be freed when the module is
467 no longer needed.
468
469## FAQ
470
471* Why the weird name for the project?
472
473"Binaryen" is a combination of **binary** - since WebAssembly is a binary format
474for the web - and **Emscripten** - with which it can integrate in order to
475compile C and C++ all the way to WebAssembly, via asm.js. Binaryen began as
476Emscripten's WebAssembly processing library (`wasm-emscripten`).
477
478"Binaryen" is pronounced [in the same manner](http://www.makinggameofthrones.com/production-diary/2011/2/11/official-pronunciation-guide-for-game-of-thrones.html) as "[Targaryen](https://en.wikipedia.org/wiki/List_of_A_Song_of_Ice_and_Fire_characters#House_Targaryen)": *bi-NAIR-ee-in*. Or something like that? Anyhow, however Targaryen is correctly pronounced, they should rhyme. Aside from pronunciation, the Targaryen house words, "Fire and Blood", have also inspired Binaryen's: "Code and Bugs."
479
480* Does it compile under Windows and/or Visual Studio?
481
482Yes, it does. Here's a step-by-step [tutorial][win32] on how to compile it
483under **Windows 10 x64** with with **CMake** and **Visual Studio 2015**. Help
484would be appreciated on Windows and OS X as most of the core devs are on Linux.
485
486[compiling to WebAssembly]: https://github.com/WebAssembly/binaryen/wiki/Compiling-to-WebAssembly-with-Binaryen
487[win32]: https://github.com/brakmic/bazaar/blob/master/webassembly/COMPILING_WIN32.md
488[C API]: https://github.com/WebAssembly/binaryen/wiki/Compiling-to-WebAssembly-with-Binaryen#c-api-1
489[control flow graph]: https://github.com/WebAssembly/binaryen/wiki/Compiling-to-WebAssembly-with-Binaryen#cfg-api
490[JS_API]: https://github.com/WebAssembly/binaryen/wiki/binaryen.js-API
491[compile_to_wasm]: https://github.com/WebAssembly/binaryen/wiki/Compiling-to-WebAssembly-with-Binaryen#what-do-i-need-to-have-in-order-to-use-binaryen-to-compile-to-webassembly
492[backend]: https://kripken.github.io/talks/binaryen.html#/9
493[minification]: https://kripken.github.io/talks/binaryen.html#/2
494[unreachable]: https://github.com/WebAssembly/binaryen/issues/903
495[binaryen_ir]: https://github.com/WebAssembly/binaryen/issues/663
496