lmp.script.eval_dset_ppl
#
Use pre-trained language model checkpoints to calculate perplexity on a dataset.
One must first run the script lmp.script.train_model before running this script.
Use pipenv run tensorboard
to launch tensorboard and open browser with URL http://localhost:6006/ to see evaluation
results over selected model checkpoints.
See also
- lmp.model
All available language models.
- lmp.script.eval_txt_ppl
Use pre-trained language model to calculate perplexity on given text.
- lmp.script.train_model
Train language model.
Examples
The following example evaluate language model experiment my_model_exp
on WikiText2
dataset
with version valid
.
It evaluates checkpoints whose numbers are larger than or equal to 5000
.
python -m lmp.script.eval_dset_ppl wiki-text-2 \
--batch_size 32 \
--first_ckpt 5000 \
--exp_name my_model_exp \
--ver valid
The following example only evaluate on the last checkpoint.
python -m lmp.script.eval_dset_ppl wiki-text-2 \
--batch_size 32 \
--first_ckpt -1 \
--exp_name my_model_exp \
--ver valid
There are many checkpoints to be evaluated. One can specify checkpoint range one want to evaluate.
python -m lmp.script.eval_dset_ppl wiki-text-2 \
--batch_size 32 \
--first_ckpt 5000 \
--exp_name my_model_exp \
--last_ckpt 10000 \
--ver valid
Since evaluation do not need to construct tensor graph when perform forward pass, model will consume less memory than training. Thus we can use larger batch size to accelerate evaluation process.
python -m lmp.script.eval_dset_ppl wiki-text-2 \
--batch_size 128 \
--ckpt -1 \
--exp_name my_model_exp \
--ver valid
You can use -h
or --help
options to get a list of supported CLI arguments.
python -m lmp.script.eval_dset_ppl -h