lmp.tknzr

Tokenizer module.

lmp.tknzr.ALL_TKNZRS

All available tokenizers.

Type

list[lmp.tknzr.BaseTknzr]

lmp.tknzr.TKNZR_OPTS

Mapping tokenizer name tknzr_name to tokenizer class.

Type

Final[dict[str, lmp.tknzr.BaseTknzr]]

Examples

Get lmp.tknzr.CharTknzr by its name.

>>> from lmp.tknzr import CharTknzr, TKNZR_OPTS
>>> CharTknzr.tknzr_name in TKNZR_OPTS
True
>>> TKNZR_OPTS[CharTknzr.tknzr_name] == CharTknzr
True

Submodules:

• lmp.tknzr._base
• lmp.tknzr._bpe
• lmp.tknzr._char
• lmp.tknzr._ws