Searched defs:detokenizer (Results 1 – 4 of 4) sorted by relevance
| /dports/misc/py-gluonnlp/gluon-nlp-0.10.0/docs/examples/machine_translation/ |
| H A D | utils.py | 28 def evaluate(model, data_loader, test_loss_function, translator, tgt_vocab, detokenizer, context): argument
74 def translate(translator, src_seq, src_vocab, tgt_vocab, detokenizer, ctx): argument
|
| /dports/misc/py-gluonnlp/gluon-nlp-0.10.0/scripts/machine_translation/ |
| H A D | train_transformer.py | 208 detokenizer = nlp.data.SacreMosesDetokenizer() variable
|
| H A D | inference_transformer.py | 137 detokenizer = nlp.data.SacreMosesDetokenizer() variable
|
| /dports/textproc/py-ufal.udpipe/ufal.udpipe-1.2.0.3/udpipe/ |
| H A D | udpipe.cpp | 19360 class detokenizer { class
19426 detokenizer::detokenizer(const string& plain_text) in detokenizer() function in ufal::udpipe::detokenizer
20884 unique_ptr<detokenizer> detokenizer; in train_tokenizer() local
|