lmp.tknzr
#
Tokenizer module.
- lmp.tknzr.ALL_TKNZRS#
All available tokenizers.
- Type
- lmp.tknzr.TKNZR_OPTS#
Mapping tokenizer name
tknzr_name
to tokenizer class.- Type
Examples
Get lmp.tknzr.CharTknzr
by its name.
>>> from lmp.tknzr import CharTknzr, TKNZR_OPTS
>>> CharTknzr.tknzr_name in TKNZR_OPTS
True
>>> TKNZR_OPTS[CharTknzr.tknzr_name] == CharTknzr
True