lmp

Language model playground source code.

This module provide utilities for training language model, perplexity evaluation and continual text generation.

• lmp.dset contains all available datasets.

• lmp.model contains all available language models.

• lmp.script provide scripts for training and inference on all tokenizers and language models.

• lmp.tknzr contains all available tokenizers.

• lmp.util contains utilities shared throughout this project.

Submodules:

• lmp.dset
• lmp.infer
• lmp.model
• lmp.script
• lmp.tknzr
• lmp.util
• lmp.vars