Name | Date | Size | #Lines | LOC | ||
---|---|---|---|---|---|---|
.. | 23-Jun-2021 | - | ||||
LICENSE | H A D | 23-Jun-2021 | 11.1 KiB | 202 | 169 | |
README.md | H A D | 23-Jun-2021 | 2.7 KiB | 92 | 57 | |
doc.go | H A D | 23-Jun-2021 | 1.7 KiB | 46 | 1 | |
maketesttables.go | H A D | 23-Jun-2021 | 5.4 KiB | 220 | 178 | |
segment.go | H A D | 23-Jun-2021 | 9.8 KiB | 285 | 162 | |
segment_fuzz.go | H A D | 23-Jun-2021 | 817 | 23 | 9 | |
segment_words.go | H A D | 23-Jun-2021 | 752.4 KiB | 19,543 | 19,244 | |
segment_words.rl | H A D | 23-Jun-2021 | 8.9 KiB | 286 | 243 | |
segment_words_prod.go | H A D | 23-Jun-2021 | 2.6 MiB | 173,644 | 170,086 |
README.md
1# segment 2 3A Go library for performing Unicode Text Segmentation 4as described in [Unicode Standard Annex #29](http://www.unicode.org/reports/tr29/) 5 6## Features 7 8* Currently only segmentation at Word Boundaries is supported. 9 10## License 11 12Apache License Version 2.0 13 14## Usage 15 16The functionality is exposed in two ways: 17 181. You can use a bufio.Scanner with the SplitWords implementation of SplitFunc. 19The SplitWords function will identify the appropriate word boundaries in the input 20text and the Scanner will return tokens at the appropriate place. 21 22 scanner := bufio.NewScanner(...) 23 scanner.Split(segment.SplitWords) 24 for scanner.Scan() { 25 tokenBytes := scanner.Bytes() 26 } 27 if err := scanner.Err(); err != nil { 28 t.Fatal(err) 29 } 30 312. Sometimes you would also like information returned about the type of token. 32To do this we have introduce a new type named Segmenter. It works just like Scanner 33but additionally a token type is returned. 34 35 segmenter := segment.NewWordSegmenter(...) 36 for segmenter.Segment() { 37 tokenBytes := segmenter.Bytes()) 38 tokenType := segmenter.Type() 39 } 40 if err := segmenter.Err(); err != nil { 41 t.Fatal(err) 42 } 43 44## Choosing Implementation 45 46By default segment does NOT use the fastest runtime implementation. The reason is that it adds approximately 5s to compilation time and may require more than 1GB of ram on the machine performing compilation. 47 48However, you can choose to build with the fastest runtime implementation by passing the build tag as follows: 49 50 -tags 'prod' 51 52## Generating Code 53 54Several components in this package are generated. 55 561. Several Ragel rules files are generated from Unicode properties files. 572. Ragel machine is generated from the Ragel rules. 583. Test tables are generated from the Unicode test files. 59 60All of these can be generated by running: 61 62 go generate 63 64## Fuzzing 65 66There is support for fuzzing the segment library with [go-fuzz](https://github.com/dvyukov/go-fuzz). 67 681. Install go-fuzz if you haven't already: 69 70 go get github.com/dvyukov/go-fuzz/go-fuzz 71 go get github.com/dvyukov/go-fuzz/go-fuzz-build 72 732. Build the package with go-fuzz: 74 75 go-fuzz-build github.com/blevesearch/segment 76 773. Convert the Unicode provided test cases into the initial corpus for go-fuzz: 78 79 go test -v -run=TestGenerateWordSegmentFuzz -tags gofuzz_generate 80 814. Run go-fuzz: 82 83 go-fuzz -bin=segment-fuzz.zip -workdir=workdir 84 85## Status 86 87 88[![Build Status](https://travis-ci.org/blevesearch/segment.svg?branch=master)](https://travis-ci.org/blevesearch/segment) 89 90[![Coverage Status](https://img.shields.io/coveralls/blevesearch/segment.svg)](https://coveralls.io/r/blevesearch/segment?branch=master) 91 92[![GoDoc](https://godoc.org/github.com/blevesearch/segment?status.svg)](https://godoc.org/github.com/blevesearch/segment)