Welcome to Language Model Playground’s documentation!#

Language model playground is a tutorial about “How to implement neural network based language models”. We use Pytorch to implement language models.

We have implemented several language models including:

Elman Net. (See ElmanNet.)
LSTM and its variations. (See LSTM1997, LSTM2000 and LSTM2002.)
Transformer encoder. (See TransEnc)
And more to come!

You can easily create these models instance using module lmp.model. You can also train these models directly using CLI script lmp.script.train_model.

import lmp.model

model = lmp.model.ElmanNet(...)  # parameters go in here.
model = lmp.model.LSTM1997(...)  # parameters go in here.
model = lmp.model.TransEnc(...)  # parameters go in here.

See also

lmp.model: All available language models.

We have written serveral scripts to demonstrate typical training pipline of language models and demonstrate furthur usage on language models:

Use lmp.script.sample_dset to take a look at available datasets.
Use lmp.script.train_tknzr to train tokenizers.
Use lmp.script.tknz_txt to tokenize text with pre-trained tokenizers.
Use lmp.script.train_model to train language models.
Use lmp.script.eval_txt_ppl to calculate perplexity on the given sample with pre-trained language model checkpoint.
Use lmp.script.eval_dset_ppl to calculate perplexity on dataset over a range of pre-trained language model checkpoints.
Use lmp.script.gen_txt to generate continual text with pre-trained language model checkpoint.

See quick start for typical language model training pipline, or jump directly to contents you are interesting in!

Table of Contents:

Welcome to Language Model Playground’s documentation!#

Indices and tables#