README.md
1goldmark
2==========================================
3
4[![https://pkg.go.dev/github.com/yuin/goldmark](https://pkg.go.dev/badge/github.com/yuin/goldmark.svg)](https://pkg.go.dev/github.com/yuin/goldmark)
5[![https://github.com/yuin/goldmark/actions?query=workflow:test](https://github.com/yuin/goldmark/workflows/test/badge.svg?branch=master&event=push)](https://github.com/yuin/goldmark/actions?query=workflow:test)
6[![https://coveralls.io/github/yuin/goldmark](https://coveralls.io/repos/github/yuin/goldmark/badge.svg?branch=master)](https://coveralls.io/github/yuin/goldmark)
7[![https://goreportcard.com/report/github.com/yuin/goldmark](https://goreportcard.com/badge/github.com/yuin/goldmark)](https://goreportcard.com/report/github.com/yuin/goldmark)
8
9> A Markdown parser written in Go. Easy to extend, standards-compliant, well-structured.
10
11goldmark is compliant with CommonMark 0.30.
12
13Motivation
14----------------------
15I needed a Markdown parser for Go that satisfies the following requirements:
16
17- Easy to extend.
18 - Markdown is poor in document expressions compared to other light markup languages such as reStructuredText.
19 - We have extensions to the Markdown syntax, e.g. PHP Markdown Extra, GitHub Flavored Markdown.
20- Standards-compliant.
21 - Markdown has many dialects.
22 - GitHub-Flavored Markdown is widely used and is based upon CommonMark, effectively mooting the question of whether or not CommonMark is an ideal specification.
23 - CommonMark is complicated and hard to implement.
24- Well-structured.
25 - AST-based; preserves source position of nodes.
26- Written in pure Go.
27
28[golang-commonmark](https://gitlab.com/golang-commonmark/markdown) may be a good choice, but it seems to be a copy of [markdown-it](https://github.com/markdown-it).
29
30[blackfriday.v2](https://github.com/russross/blackfriday/tree/v2) is a fast and widely-used implementation, but is not CommonMark-compliant and cannot be extended from outside of the package, since its AST uses structs instead of interfaces.
31
32Furthermore, its behavior differs from other implementations in some cases, especially regarding lists: [Deep nested lists don't output correctly #329](https://github.com/russross/blackfriday/issues/329), [List block cannot have a second line #244](https://github.com/russross/blackfriday/issues/244), etc.
33
34This behavior sometimes causes problems. If you migrate your Markdown text from GitHub to blackfriday-based wikis, many lists will immediately be broken.
35
36As mentioned above, CommonMark is complicated and hard to implement, so Markdown parsers based on CommonMark are few and far between.
37
38Features
39----------------------
40
41- **Standards-compliant.** goldmark is fully compliant with the latest [CommonMark](https://commonmark.org/) specification.
42- **Extensible.** Do you want to add a `@username` mention syntax to Markdown?
43 You can easily do so in goldmark. You can add your AST nodes,
44 parsers for block-level elements, parsers for inline-level elements,
45 transformers for paragraphs, transformers for the whole AST structure, and
46 renderers.
47- **Performance.** goldmark's performance is on par with that of cmark,
48 the CommonMark reference implementation written in C.
49- **Robust.** goldmark is tested with [go-fuzz](https://github.com/dvyukov/go-fuzz), a fuzz testing tool.
50- **Built-in extensions.** goldmark ships with common extensions like tables, strikethrough,
51 task lists, and definition lists.
52- **Depends only on standard libraries.**
53
54Installation
55----------------------
56```bash
57$ go get github.com/yuin/goldmark
58```
59
60
61Usage
62----------------------
63Import packages:
64
65```go
66import (
67 "bytes"
68 "github.com/yuin/goldmark"
69)
70```
71
72
73Convert Markdown documents with the CommonMark-compliant mode:
74
75```go
76var buf bytes.Buffer
77if err := goldmark.Convert(source, &buf); err != nil {
78 panic(err)
79}
80```
81
82With options
83------------------------------
84
85```go
86var buf bytes.Buffer
87if err := goldmark.Convert(source, &buf, parser.WithContext(ctx)); err != nil {
88 panic(err)
89}
90```
91
92| Functional option | Type | Description |
93| ----------------- | ---- | ----------- |
94| `parser.WithContext` | A `parser.Context` | Context for the parsing phase. |
95
96Context options
97----------------------
98
99| Functional option | Type | Description |
100| ----------------- | ---- | ----------- |
101| `parser.WithIDs` | A `parser.IDs` | `IDs` allows you to change logics that are related to element id(ex: Auto heading id generation). |
102
103
104Custom parser and renderer
105--------------------------
106```go
107import (
108 "bytes"
109 "github.com/yuin/goldmark"
110 "github.com/yuin/goldmark/extension"
111 "github.com/yuin/goldmark/parser"
112 "github.com/yuin/goldmark/renderer/html"
113)
114
115md := goldmark.New(
116 goldmark.WithExtensions(extension.GFM),
117 goldmark.WithParserOptions(
118 parser.WithAutoHeadingID(),
119 ),
120 goldmark.WithRendererOptions(
121 html.WithHardWraps(),
122 html.WithXHTML(),
123 ),
124 )
125var buf bytes.Buffer
126if err := md.Convert(source, &buf); err != nil {
127 panic(err)
128}
129```
130
131| Functional option | Type | Description |
132| ----------------- | ---- | ----------- |
133| `goldmark.WithParser` | `parser.Parser` | This option must be passed before `goldmark.WithParserOptions` and `goldmark.WithExtensions` |
134| `goldmark.WithRenderer` | `renderer.Renderer` | This option must be passed before `goldmark.WithRendererOptions` and `goldmark.WithExtensions` |
135| `goldmark.WithParserOptions` | `...parser.Option` | |
136| `goldmark.WithRendererOptions` | `...renderer.Option` | |
137| `goldmark.WithExtensions` | `...goldmark.Extender` | |
138
139Parser and Renderer options
140------------------------------
141
142### Parser options
143
144| Functional option | Type | Description |
145| ----------------- | ---- | ----------- |
146| `parser.WithBlockParsers` | A `util.PrioritizedSlice` whose elements are `parser.BlockParser` | Parsers for parsing block level elements. |
147| `parser.WithInlineParsers` | A `util.PrioritizedSlice` whose elements are `parser.InlineParser` | Parsers for parsing inline level elements. |
148| `parser.WithParagraphTransformers` | A `util.PrioritizedSlice` whose elements are `parser.ParagraphTransformer` | Transformers for transforming paragraph nodes. |
149| `parser.WithASTTransformers` | A `util.PrioritizedSlice` whose elements are `parser.ASTTransformer` | Transformers for transforming an AST. |
150| `parser.WithAutoHeadingID` | `-` | Enables auto heading ids. |
151| `parser.WithAttribute` | `-` | Enables custom attributes. Currently only headings supports attributes. |
152
153### HTML Renderer options
154
155| Functional option | Type | Description |
156| ----------------- | ---- | ----------- |
157| `html.WithWriter` | `html.Writer` | `html.Writer` for writing contents to an `io.Writer`. |
158| `html.WithHardWraps` | `-` | Render newlines as `<br>`.|
159| `html.WithXHTML` | `-` | Render as XHTML. |
160| `html.WithUnsafe` | `-` | By default, goldmark does not render raw HTML or potentially dangerous links. With this option, goldmark renders such content as written. |
161
162### Built-in extensions
163
164- `extension.Table`
165 - [GitHub Flavored Markdown: Tables](https://github.github.com/gfm/#tables-extension-)
166- `extension.Strikethrough`
167 - [GitHub Flavored Markdown: Strikethrough](https://github.github.com/gfm/#strikethrough-extension-)
168- `extension.Linkify`
169 - [GitHub Flavored Markdown: Autolinks](https://github.github.com/gfm/#autolinks-extension-)
170- `extension.TaskList`
171 - [GitHub Flavored Markdown: Task list items](https://github.github.com/gfm/#task-list-items-extension-)
172- `extension.GFM`
173 - This extension enables Table, Strikethrough, Linkify and TaskList.
174 - This extension does not filter tags defined in [6.11: Disallowed Raw HTML (extension)](https://github.github.com/gfm/#disallowed-raw-html-extension-).
175 If you need to filter HTML tags, see [Security](#security).
176 - If you need to parse github emojis, you can use [goldmark-emoji](https://github.com/yuin/goldmark-emoji) extension.
177- `extension.DefinitionList`
178 - [PHP Markdown Extra: Definition lists](https://michelf.ca/projects/php-markdown/extra/#def-list)
179- `extension.Footnote`
180 - [PHP Markdown Extra: Footnotes](https://michelf.ca/projects/php-markdown/extra/#footnotes)
181- `extension.Typographer`
182 - This extension substitutes punctuations with typographic entities like [smartypants](https://daringfireball.net/projects/smartypants/).
183
184### Attributes
185The `parser.WithAttribute` option allows you to define attributes on some elements.
186
187Currently only headings support attributes.
188
189**Attributes are being discussed in the
190[CommonMark forum](https://talk.commonmark.org/t/consistent-attribute-syntax/272).
191This syntax may possibly change in the future.**
192
193
194#### Headings
195
196```
197## heading ## {#id .className attrName=attrValue class="class1 class2"}
198
199## heading {#id .className attrName=attrValue class="class1 class2"}
200```
201
202```
203heading {#id .className attrName=attrValue}
204============
205```
206
207### Table extension
208The Table extension implements [Table(extension)](https://github.github.com/gfm/#tables-extension-), as
209defined in [GitHub Flavored Markdown Spec](https://github.github.com/gfm/).
210
211Specs are defined for XHTML, so specs use some deprecated attributes for HTML5.
212
213You can override alignment rendering method via options.
214
215| Functional option | Type | Description |
216| ----------------- | ---- | ----------- |
217| `extension.WithTableCellAlignMethod` | `extension.TableCellAlignMethod` | Option indicates how are table cells aligned. |
218
219### Typographer extension
220
221The Typographer extension translates plain ASCII punctuation characters into typographic-punctuation HTML entities.
222
223Default substitutions are:
224
225| Punctuation | Default entity |
226| ------------ | ---------- |
227| `'` | `‘`, `’` |
228| `"` | `“`, `”` |
229| `--` | `–` |
230| `---` | `—` |
231| `...` | `…` |
232| `<<` | `«` |
233| `>>` | `»` |
234
235You can override the default substitutions via `extensions.WithTypographicSubstitutions`:
236
237```go
238markdown := goldmark.New(
239 goldmark.WithExtensions(
240 extension.NewTypographer(
241 extension.WithTypographicSubstitutions(extension.TypographicSubstitutions{
242 extension.LeftSingleQuote: []byte("‚"),
243 extension.RightSingleQuote: nil, // nil disables a substitution
244 }),
245 ),
246 ),
247)
248```
249
250### Linkify extension
251
252The Linkify extension implements [Autolinks(extension)](https://github.github.com/gfm/#autolinks-extension-), as
253defined in [GitHub Flavored Markdown Spec](https://github.github.com/gfm/).
254
255Since the spec does not define details about URLs, there are numerous ambiguous cases.
256
257You can override autolinking patterns via options.
258
259| Functional option | Type | Description |
260| ----------------- | ---- | ----------- |
261| `extension.WithLinkifyAllowedProtocols` | `[][]byte` | List of allowed protocols such as `[][]byte{ []byte("http:") }` |
262| `extension.WithLinkifyURLRegexp` | `*regexp.Regexp` | Regexp that defines URLs, including protocols |
263| `extension.WithLinkifyWWWRegexp` | `*regexp.Regexp` | Regexp that defines URL starting with `www.`. This pattern corresponds to [the extended www autolink](https://github.github.com/gfm/#extended-www-autolink) |
264| `extension.WithLinkifyEmailRegexp` | `*regexp.Regexp` | Regexp that defines email addresses` |
265
266Example, using [xurls](https://github.com/mvdan/xurls):
267
268```go
269import "mvdan.cc/xurls/v2"
270
271markdown := goldmark.New(
272 goldmark.WithRendererOptions(
273 html.WithXHTML(),
274 html.WithUnsafe(),
275 ),
276 goldmark.WithExtensions(
277 extension.NewLinkify(
278 extension.WithLinkifyAllowedProtocols([][]byte{
279 []byte("http:"),
280 []byte("https:"),
281 }),
282 extension.WithLinkifyURLRegexp(
283 xurls.Strict,
284 ),
285 ),
286 ),
287)
288```
289
290### Footnotes extension
291
292The Footnote extension implements [PHP Markdown Extra: Footnotes](https://michelf.ca/projects/php-markdown/extra/#footnotes).
293
294This extension has some options:
295
296| Functional option | Type | Description |
297| ----------------- | ---- | ----------- |
298| `extension.WithFootnoteIDPrefix` | `[]byte` | a prefix for the id attributes.|
299| `extension.WithFootnoteIDPrefixFunction` | `func(gast.Node) []byte` | a function that determines the id attribute for given Node.|
300| `extension.WithFootnoteLinkTitle` | `[]byte` | an optional title attribute for footnote links.|
301| `extension.WithFootnoteBacklinkTitle` | `[]byte` | an optional title attribute for footnote backlinks. |
302| `extension.WithFootnoteLinkClass` | `[]byte` | a class for footnote links. This defaults to `footnote-ref`. |
303| `extension.WithFootnoteBacklinkClass` | `[]byte` | a class for footnote backlinks. This defaults to `footnote-backref`. |
304| `extension.WithFootnoteBacklinkHTML` | `[]byte` | a class for footnote backlinks. This defaults to `↩︎`. |
305
306Some options can have special substitutions. Occurrences of “^^” in the string will be replaced by the corresponding footnote number in the HTML output. Occurrences of “%%” will be replaced by a number for the reference (footnotes can have multiple references).
307
308`extension.WithFootnoteIDPrefix` and `extension.WithFootnoteIDPrefixFunction` are useful if you have multiple Markdown documents displayed inside one HTML document to avoid footnote ids to clash each other.
309
310`extension.WithFootnoteIDPrefix` sets fixed id prefix, so you may write codes like the following:
311
312```go
313for _, path := range files {
314 source := readAll(path)
315 prefix := getPrefix(path)
316
317 markdown := goldmark.New(
318 goldmark.WithExtensions(
319 NewFootnote(
320 WithFootnoteIDPrefix([]byte(path)),
321 ),
322 ),
323 )
324 var b bytes.Buffer
325 err := markdown.Convert(source, &b)
326 if err != nil {
327 t.Error(err.Error())
328 }
329}
330```
331
332`extension.WithFootnoteIDPrefixFunction` determines an id prefix by calling given function, so you may write codes like the following:
333
334```go
335markdown := goldmark.New(
336 goldmark.WithExtensions(
337 NewFootnote(
338 WithFootnoteIDPrefixFunction(func(n gast.Node) []byte {
339 v, ok := n.OwnerDocument().Meta()["footnote-prefix"]
340 if ok {
341 return util.StringToReadOnlyBytes(v.(string))
342 }
343 return nil
344 }),
345 ),
346 ),
347)
348
349for _, path := range files {
350 source := readAll(path)
351 var b bytes.Buffer
352
353 doc := markdown.Parser().Parse(text.NewReader(source))
354 doc.Meta()["footnote-prefix"] = getPrefix(path)
355 err := markdown.Renderer().Render(&b, source, doc)
356}
357```
358
359You can use [goldmark-meta](https://github.com/yuin/goldmark-meta) to define a id prefix in the markdown document:
360
361
362```markdown
363---
364title: document title
365slug: article1
366footnote-prefix: article1
367---
368
369# My article
370
371```
372
373Security
374--------------------
375By default, goldmark does not render raw HTML or potentially-dangerous URLs.
376If you need to gain more control over untrusted contents, it is recommended that you
377use an HTML sanitizer such as [bluemonday](https://github.com/microcosm-cc/bluemonday).
378
379Benchmark
380--------------------
381You can run this benchmark in the `_benchmark` directory.
382
383### against other golang libraries
384
385blackfriday v2 seems to be the fastest, but as it is not CommonMark compliant, its performance cannot be directly compared to that of the CommonMark-compliant libraries.
386
387goldmark, meanwhile, builds a clean, extensible AST structure, achieves full compliance with
388CommonMark, and consumes less memory, all while being reasonably fast.
389
390```
391goos: darwin
392goarch: amd64
393BenchmarkMarkdown/Blackfriday-v2-12 326 3465240 ns/op 3298861 B/op 20047 allocs/op
394BenchmarkMarkdown/GoldMark-12 303 3927494 ns/op 2574809 B/op 13853 allocs/op
395BenchmarkMarkdown/CommonMark-12 244 4900853 ns/op 2753851 B/op 20527 allocs/op
396BenchmarkMarkdown/Lute-12 130 9195245 ns/op 9175030 B/op 123534 allocs/op
397BenchmarkMarkdown/GoMarkdown-12 9 113541994 ns/op 2187472 B/op 22173 allocs/op
398```
399
400### against cmark (CommonMark reference implementation written in C)
401
402```
403----------- cmark -----------
404file: _data.md
405iteration: 50
406average: 0.0037760639 sec
407go run ./goldmark_benchmark.go
408------- goldmark -------
409file: _data.md
410iteration: 50
411average: 0.0040964230 sec
412```
413
414As you can see, goldmark's performance is on par with cmark's.
415
416Extensions
417--------------------
418
419- [goldmark-meta](https://github.com/yuin/goldmark-meta): A YAML metadata
420 extension for the goldmark Markdown parser.
421- [goldmark-highlighting](https://github.com/yuin/goldmark-highlighting): A syntax-highlighting extension
422 for the goldmark markdown parser.
423- [goldmark-emoji](https://github.com/yuin/goldmark-emoji): An emoji
424 extension for the goldmark Markdown parser.
425- [goldmark-mathjax](https://github.com/litao91/goldmark-mathjax): Mathjax support for the goldmark markdown parser
426- [goldmark-pdf](https://github.com/stephenafamo/goldmark-pdf): A PDF renderer that can be passed to `goldmark.WithRenderer()`.
427- [goldmark-hashtag](https://github.com/abhinav/goldmark-hashtag): Adds support for `#hashtag`-based tagging to goldmark.
428- [goldmark-wikilink](https://github.com/abhinav/goldmark-wikilink): Adds support for `[[wiki]]`-style links to goldmark.
429- [goldmark-toc](https://github.com/abhinav/goldmark-toc): Adds support for generating tables-of-contents for goldmark documents.
430- [goldmark-mermaid](https://github.com/abhinav/goldmark-mermaid): Adds support for renderng [Mermaid](https://mermaid-js.github.io/mermaid/) diagrams in goldmark documents.
431
432goldmark internal(for extension developers)
433----------------------------------------------
434### Overview
435goldmark's Markdown processing is outlined in the diagram below.
436
437```
438 <Markdown in []byte, parser.Context>
439 |
440 V
441 +-------- parser.Parser ---------------------------
442 | 1. Parse block elements into AST
443 | 1. If a parsed block is a paragraph, apply
444 | ast.ParagraphTransformer
445 | 2. Traverse AST and parse blocks.
446 | 1. Process delimiters(emphasis) at the end of
447 | block parsing
448 | 3. Apply parser.ASTTransformers to AST
449 |
450 V
451 <ast.Node>
452 |
453 V
454 +------- renderer.Renderer ------------------------
455 | 1. Traverse AST and apply renderer.NodeRenderer
456 | corespond to the node type
457
458 |
459 V
460 <Output>
461```
462
463### Parsing
464Markdown documents are read through `text.Reader` interface.
465
466AST nodes do not have concrete text. AST nodes have segment information of the documents, represented by `text.Segment` .
467
468`text.Segment` has 3 attributes: `Start`, `End`, `Padding` .
469
470(TBC)
471
472**TODO**
473
474See `extension` directory for examples of extensions.
475
476Summary:
477
4781. Define AST Node as a struct in which `ast.BaseBlock` or `ast.BaseInline` is embedded.
4792. Write a parser that implements `parser.BlockParser` or `parser.InlineParser`.
4803. Write a renderer that implements `renderer.NodeRenderer`.
4814. Define your goldmark extension that implements `goldmark.Extender`.
482
483
484Donation
485--------------------
486BTC: 1NEDSyUmo4SMTDP83JJQSWi1MvQUGGNMZB
487
488License
489--------------------
490MIT
491
492Author
493--------------------
494Yusuke Inuzuka
495