Searched refs:spaces_in_training (Results 1 – 1 of 1) sorted by relevance
20892 bool spaces_in_training = false; in train_tokenizer() local20906 if (unilib::unicode::category(chr) & unilib::unicode::Zs) spaces_in_training = true; in train_tokenizer()20950 …bool allow_spaces = spaces_in_training; if (!option_bool(tokenizer, "allow_spaces", allow_spaces, … in train_tokenizer()