1Parser for Rust source code
2===========================
3
4[<img alt="github" src="https://img.shields.io/badge/github-dtolnay/syn-8da0cb?style=for-the-badge&labelColor=555555&logo=github" height="20">](https://github.com/dtolnay/syn)
5[<img alt="crates.io" src="https://img.shields.io/crates/v/syn.svg?style=for-the-badge&color=fc8d62&logo=rust" height="20">](https://crates.io/crates/syn)
6[<img alt="docs.rs" src="https://img.shields.io/badge/docs.rs-syn-66c2a5?style=for-the-badge&labelColor=555555&logoColor=white&logo=data:image/svg+xml;base64,PHN2ZyByb2xlPSJpbWciIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgdmlld0JveD0iMCAwIDUxMiA1MTIiPjxwYXRoIGZpbGw9IiNmNWY1ZjUiIGQ9Ik00ODguNiAyNTAuMkwzOTIgMjE0VjEwNS41YzAtMTUtOS4zLTI4LjQtMjMuNC0zMy43bC0xMDAtMzcuNWMtOC4xLTMuMS0xNy4xLTMuMS0yNS4zIDBsLTEwMCAzNy41Yy0xNC4xIDUuMy0yMy40IDE4LjctMjMuNCAzMy43VjIxNGwtOTYuNiAzNi4yQzkuMyAyNTUuNSAwIDI2OC45IDAgMjgzLjlWMzk0YzAgMTMuNiA3LjcgMjYuMSAxOS45IDMyLjJsMTAwIDUwYzEwLjEgNS4xIDIyLjEgNS4xIDMyLjIgMGwxMDMuOS01MiAxMDMuOSA1MmMxMC4xIDUuMSAyMi4xIDUuMSAzMi4yIDBsMTAwLTUwYzEyLjItNi4xIDE5LjktMTguNiAxOS45LTMyLjJWMjgzLjljMC0xNS05LjMtMjguNC0yMy40LTMzLjd6TTM1OCAyMTQuOGwtODUgMzEuOXYtNjguMmw4NS0zN3Y3My4zek0xNTQgMTA0LjFsMTAyLTM4LjIgMTAyIDM4LjJ2LjZsLTEwMiA0MS40LTEwMi00MS40di0uNnptODQgMjkxLjFsLTg1IDQyLjV2LTc5LjFsODUtMzguOHY3NS40em0wLTExMmwtMTAyIDQxLjQtMTAyLTQxLjR2LS42bDEwMi0zOC4yIDEwMiAzOC4ydi42em0yNDAgMTEybC04NSA0Mi41di03OS4xbDg1LTM4Ljh2NzUuNHptMC0xMTJsLTEwMiA0MS40LTEwMi00MS40di0uNmwxMDItMzguMiAxMDIgMzguMnYuNnoiPjwvcGF0aD48L3N2Zz4K" height="20">](https://docs.rs/syn)
7[<img alt="build status" src="https://img.shields.io/github/workflow/status/dtolnay/syn/CI/master?style=for-the-badge" height="20">](https://github.com/dtolnay/syn/actions?query=branch%3Amaster)
8
9Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree
10of Rust source code.
11
12Currently this library is geared toward use in Rust procedural macros, but
13contains some APIs that may be useful more generally.
14
15- **Data structures** — Syn provides a complete syntax tree that can represent
16  any valid Rust source code. The syntax tree is rooted at [`syn::File`] which
17  represents a full source file, but there are other entry points that may be
18  useful to procedural macros including [`syn::Item`], [`syn::Expr`] and
19  [`syn::Type`].
20
21- **Derives** — Of particular interest to derive macros is [`syn::DeriveInput`]
22  which is any of the three legal input items to a derive macro. An example
23  below shows using this type in a library that can derive implementations of a
24  user-defined trait.
25
26- **Parsing** — Parsing in Syn is built around [parser functions] with the
27  signature `fn(ParseStream) -> Result<T>`. Every syntax tree node defined by
28  Syn is individually parsable and may be used as a building block for custom
29  syntaxes, or you may dream up your own brand new syntax without involving any
30  of our syntax tree types.
31
32- **Location information** — Every token parsed by Syn is associated with a
33  `Span` that tracks line and column information back to the source of that
34  token. These spans allow a procedural macro to display detailed error messages
35  pointing to all the right places in the user's code. There is an example of
36  this below.
37
38- **Feature flags** — Functionality is aggressively feature gated so your
39  procedural macros enable only what they need, and do not pay in compile time
40  for all the rest.
41
42[`syn::File`]: https://docs.rs/syn/1.0/syn/struct.File.html
43[`syn::Item`]: https://docs.rs/syn/1.0/syn/enum.Item.html
44[`syn::Expr`]: https://docs.rs/syn/1.0/syn/enum.Expr.html
45[`syn::Type`]: https://docs.rs/syn/1.0/syn/enum.Type.html
46[`syn::DeriveInput`]: https://docs.rs/syn/1.0/syn/struct.DeriveInput.html
47[parser functions]: https://docs.rs/syn/1.0/syn/parse/index.html
48
49*Version requirement: Syn supports rustc 1.31 and up.*
50
51[*Release notes*](https://github.com/dtolnay/syn/releases)
52
53<br>
54
55## Resources
56
57The best way to learn about procedural macros is by writing some. Consider
58working through [this procedural macro workshop][workshop] to get familiar with
59the different types of procedural macros. The workshop contains relevant links
60into the Syn documentation as you work through each project.
61
62[workshop]: https://github.com/dtolnay/proc-macro-workshop
63
64<br>
65
66## Example of a derive macro
67
68The canonical derive macro using Syn looks like this. We write an ordinary Rust
69function tagged with a `proc_macro_derive` attribute and the name of the trait
70we are deriving. Any time that derive appears in the user's code, the Rust
71compiler passes their data structure as tokens into our macro. We get to execute
72arbitrary Rust code to figure out what to do with those tokens, then hand some
73tokens back to the compiler to compile into the user's crate.
74
75[`TokenStream`]: https://doc.rust-lang.org/proc_macro/struct.TokenStream.html
76
77```toml
78[dependencies]
79syn = "1.0"
80quote = "1.0"
81
82[lib]
83proc-macro = true
84```
85
86```rust
87use proc_macro::TokenStream;
88use quote::quote;
89use syn::{parse_macro_input, DeriveInput};
90
91#[proc_macro_derive(MyMacro)]
92pub fn my_macro(input: TokenStream) -> TokenStream {
93    // Parse the input tokens into a syntax tree
94    let input = parse_macro_input!(input as DeriveInput);
95
96    // Build the output, possibly using quasi-quotation
97    let expanded = quote! {
98        // ...
99    };
100
101    // Hand the output tokens back to the compiler
102    TokenStream::from(expanded)
103}
104```
105
106The [`heapsize`] example directory shows a complete working implementation of a
107derive macro. It works on any Rust compiler 1.31+. The example derives a
108`HeapSize` trait which computes an estimate of the amount of heap memory owned
109by a value.
110
111[`heapsize`]: examples/heapsize
112
113```rust
114pub trait HeapSize {
115    /// Total number of bytes of heap memory owned by `self`.
116    fn heap_size_of_children(&self) -> usize;
117}
118```
119
120The derive macro allows users to write `#[derive(HeapSize)]` on data structures
121in their program.
122
123```rust
124#[derive(HeapSize)]
125struct Demo<'a, T: ?Sized> {
126    a: Box<T>,
127    b: u8,
128    c: &'a str,
129    d: String,
130}
131```
132
133<br>
134
135## Spans and error reporting
136
137The token-based procedural macro API provides great control over where the
138compiler's error messages are displayed in user code. Consider the error the
139user sees if one of their field types does not implement `HeapSize`.
140
141```rust
142#[derive(HeapSize)]
143struct Broken {
144    ok: String,
145    bad: std::thread::Thread,
146}
147```
148
149By tracking span information all the way through the expansion of a procedural
150macro as shown in the `heapsize` example, token-based macros in Syn are able to
151trigger errors that directly pinpoint the source of the problem.
152
153```console
154error[E0277]: the trait bound `std::thread::Thread: HeapSize` is not satisfied
155 --> src/main.rs:7:5
156  |
1577 |     bad: std::thread::Thread,
158  |     ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `HeapSize` is not implemented for `std::thread::Thread`
159```
160
161<br>
162
163## Parsing a custom syntax
164
165The [`lazy-static`] example directory shows the implementation of a
166`functionlike!(...)` procedural macro in which the input tokens are parsed using
167Syn's parsing API.
168
169[`lazy-static`]: examples/lazy-static
170
171The example reimplements the popular `lazy_static` crate from crates.io as a
172procedural macro.
173
174```rust
175lazy_static! {
176    static ref USERNAME: Regex = Regex::new("^[a-z0-9_-]{3,16}$").unwrap();
177}
178```
179
180The implementation shows how to trigger custom warnings and error messages on
181the macro input.
182
183```console
184warning: come on, pick a more creative name
185  --> src/main.rs:10:16
186   |
18710 |     static ref FOO: String = "lazy_static".to_owned();
188   |                ^^^
189```
190
191<br>
192
193## Testing
194
195When testing macros, we often care not just that the macro can be used
196successfully but also that when the macro is provided with invalid input it
197produces maximally helpful error messages. Consider using the [`trybuild`] crate
198to write tests for errors that are emitted by your macro or errors detected by
199the Rust compiler in the expanded code following misuse of the macro. Such tests
200help avoid regressions from later refactors that mistakenly make an error no
201longer trigger or be less helpful than it used to be.
202
203[`trybuild`]: https://github.com/dtolnay/trybuild
204
205<br>
206
207## Debugging
208
209When developing a procedural macro it can be helpful to look at what the
210generated code looks like. Use `cargo rustc -- -Zunstable-options
211--pretty=expanded` or the [`cargo expand`] subcommand.
212
213[`cargo expand`]: https://github.com/dtolnay/cargo-expand
214
215To show the expanded code for some crate that uses your procedural macro, run
216`cargo expand` from that crate. To show the expanded code for one of your own
217test cases, run `cargo expand --test the_test_case` where the last argument is
218the name of the test file without the `.rs` extension.
219
220This write-up by Brandon W Maister discusses debugging in more detail:
221[Debugging Rust's new Custom Derive system][debugging].
222
223[debugging]: https://quodlibetor.github.io/posts/debugging-rusts-new-custom-derive-system/
224
225<br>
226
227## Optional features
228
229Syn puts a lot of functionality behind optional features in order to optimize
230compile time for the most common use cases. The following features are
231available.
232
233- **`derive`** *(enabled by default)* — Data structures for representing the
234  possible input to a derive macro, including structs and enums and types.
235- **`full`** — Data structures for representing the syntax tree of all valid
236  Rust source code, including items and expressions.
237- **`parsing`** *(enabled by default)* — Ability to parse input tokens into a
238  syntax tree node of a chosen type.
239- **`printing`** *(enabled by default)* — Ability to print a syntax tree node as
240  tokens of Rust source code.
241- **`visit`** — Trait for traversing a syntax tree.
242- **`visit-mut`** — Trait for traversing and mutating in place a syntax tree.
243- **`fold`** — Trait for transforming an owned syntax tree.
244- **`clone-impls`** *(enabled by default)* — Clone impls for all syntax tree
245  types.
246- **`extra-traits`** — Debug, Eq, PartialEq, Hash impls for all syntax tree
247  types.
248- **`proc-macro`** *(enabled by default)* — Runtime dependency on the dynamic
249  library libproc_macro from rustc toolchain.
250
251<br>
252
253## Proc macro shim
254
255Syn operates on the token representation provided by the [proc-macro2] crate
256from crates.io rather than using the compiler's built in proc-macro crate
257directly. This enables code using Syn to execute outside of the context of a
258procedural macro, such as in unit tests or build.rs, and we avoid needing
259incompatible ecosystems for proc macros vs non-macro use cases.
260
261In general all of your code should be written against proc-macro2 rather than
262proc-macro. The one exception is in the signatures of procedural macro entry
263points, which are required by the language to use `proc_macro::TokenStream`.
264
265The proc-macro2 crate will automatically detect and use the compiler's data
266structures when a procedural macro is active.
267
268[proc-macro2]: https://docs.rs/proc-macro2/1.0/proc_macro2/
269
270<br>
271
272#### License
273
274<sup>
275Licensed under either of <a href="LICENSE-APACHE">Apache License, Version
2762.0</a> or <a href="LICENSE-MIT">MIT license</a> at your option.
277</sup>
278
279<br>
280
281<sub>
282Unless you explicitly state otherwise, any contribution intentionally submitted
283for inclusion in this crate by you, as defined in the Apache-2.0 license, shall
284be dual licensed as above, without any additional terms or conditions.
285</sub>
286