Searched refs:tokens_buffer (Results 1 – 1 of 1) sorted by relevance
4356 vector<token_range> tokens_buffer; member in ufal::udpipe::morphodita::unicode_tokenizer14148 vector<token_range>& tokens = tokens_ptr ? *tokens_ptr : tokens_buffer; in next_sentence()