Evaluate language model perplexity on a dataset#

Use pre-trained language model checkpoints to calculate perplexity on a dataset.

One must first run the script lmp.script.train_model before running this script. Use pipenv run tensorboard to launch tensorboard and open browser with URL http://localhost:6006/ to see evaluation results over selected model checkpoints.

See also

lmp.model: All available language models.
lmp.script.eval_txt_ppl: Use pre-trained language model to calculate perplexity on given text.
lmp.script.train_model: Train language model.

Examples

The following example evaluate language model experiment my_model_exp on WikiText2 dataset with version valid. It evaluates checkpoints whose numbers are larger than or equal to 5000.

python -m lmp.script.eval_dset_ppl wiki-text-2 \
  --batch_size 32 \
  --first_ckpt 5000 \
  --exp_name my_model_exp \
  --ver valid

The following example only evaluate on the last checkpoint.

python -m lmp.script.eval_dset_ppl wiki-text-2 \
  --batch_size 32 \
  --first_ckpt -1 \
  --exp_name my_model_exp \
  --ver valid

There are many checkpoints to be evaluated. One can specify checkpoint range one want to evaluate.

python -m lmp.script.eval_dset_ppl wiki-text-2 \
  --batch_size 32 \
  --first_ckpt 5000 \
  --exp_name my_model_exp \
  --last_ckpt 10000 \
  --ver valid

Since evaluation do not need to construct tensor graph when perform forward pass, model will consume less memory than training. Thus we can use larger batch size to accelerate evaluation process.

python -m lmp.script.eval_dset_ppl wiki-text-2 \
  --batch_size 128 \
  --ckpt -1 \
  --exp_name my_model_exp \
  --ver valid

You can use -h or --help options to get a list of supported CLI arguments.

python -m lmp.script.eval_dset_ppl -h

lmp.script.eval_dset_ppl.main(argv: List[str]) → None[source]#

Script entry point.

Parameters: argv (list[str]) – List of CLI arguments.
Return type: None

lmp.script.eval_dset_ppl.parse_args(argv: List[str]) → Namespace[source]#

Parse CLI arguments.

Parameters: argv (list[str]) – List of CLI arguments.