1# Chinese support in RediSearch
2
3Support for adding documents in Chinese is available starting at version 0.99.0.
4
5Chinese support allows Chinese documents to be added and tokenized using segmentation
6rather than simple tokenization using whitespace and/or punctuation.
7
8Indexing a Chinese document is different than indexing a document in most other
9languages because of how tokens are extracted. While most languages can have
10their tokens distinguished by separation characters and whitespace, this
11is not common in Chinese.
12
13Chinese tokenization is done by scanning the input text and checking every
14character or sequence of characters against a dictionary of predefined terms
15and determining the most likely (based on the surrounding terms and characters)
16match.
17
18RediSearch makes use of the [Friso](https://github.com/lionsoul2014/friso)
19Chinese tokenization library for this purpose. This is largely transparent to
20the user and often no additional configuration is required.
21
22## Example: Using Chinese in RediSearch
23
24In pseudo-code:
25
26```
27FT.CREATE idx SCHEMA txt TEXT
28FT.ADD idx docCn 1.0 LANGUAGE chinese FIELDS txt "Redis支持主从同步。数据可以从主服务器向任意数量的从服务器上同步，从服务器可以是关联其他从服务器的主服务器。这使得Redis可执行单层树复制。从盘可以有意无意的对数据进行写操作。由于完全实现了发布/订阅机制，使得从数据库在任何地方同步树时，可订阅一个频道并接收主服务器完整的消息发布记录。同步对读取操作的可扩展性和数据冗余很有帮助。[8]"
29FT.SEARCH idx "数据" LANGUAGE chinese HIGHLIGHT SUMMARIZE
30# Outputs:
31# <b>数据</b>?... <b>数据</b>进行写操作。由于完全实现了发布... <b>数据</b>冗余很有帮助。[8...
32```
33
34Using the Python Client:
35
36```
37# -*- coding: utf-8 -*-
38
39from redisearch.client import Client, Query
40from redisearch import TextField
41
42client = Client('idx')
43try:
44    client.drop_index()
45except:
46    pass
47
48client.create_index([TextField('txt')])
49
50# Add a document
51client.add_document('docCn1',
52                    txt='Redis支持主从同步。数据可以从主服务器向任意数量的从服务器上同步从服务器可以是关联其他从服务器的主服务器。这使得Redis可执行单层树复制。从盘可以有意无意的对数据进行写操作。由于完全实现了发布/订阅机制，使得从数据库在任何地方同步树时，可订阅一个频道并接收主服务器完整的消息发布记录。同步对读取操作的可扩展性和数据冗余很有帮助。[8]',
53                    language='chinese')
54print client.search(Query('数据').summarize().highlight().language('chinese')).docs[0].txt
55```
56
57Prints:
58
59```
60<b>数据</b>?... <b>数据</b>进行写操作。由于完全实现了发布... <b>数据</b>冗余很有帮助。[8...
61```
62
63## Using custom dictionaries
64
65If you wish to use a custom dictionary, you can do so at the module level when
66loading the module. The `FRISOINI` setting can point to the location of a
67`friso.ini` file which contains the relevant settings and paths to the dictionary
68files.
69
70Note that there is no "default" friso.ini file location. RediSearch comes with
71its own `friso.ini` and dictionary files which are compiled into the module
72binary at build-time.