Inference methods#
Overview#
An inference method can let a language model generate continual text of given text. One must provide a language model, its paired tokenizer and a text to an inference method. The generated results will returned as text.
See also
- lmp.model
All available language models.
- lmp.script.gen_txt
Continual text generation script.
- lmp.tknzr
All available tokenizers.
Import inference method module#
All inference method classes are collectively gathered under the module lmp.infer
.
One can import inference method module as usual Python module:
import lmp.infer
Create inference method instances#
After importing lmp.infer
, one can create inference method instance through the class attributes of lmp.infer
.
For example, one can create top-1 inference method Top1Infer
and top-K inference method TopKInfer
as follow:
import lmp.infer
# Create top-1 inference method instance.
top_1_infer = lmp.infer.Top1Infer()
# Create top-K inference method instance.
top_k_infer = lmp.infer.TopKInfer()
The max_seq_len
parameters is provided to avoid non-stopping generation.
The maximum value of max_seq_len
is 1024
.
import lmp.infer
# Create top-1 inference method instance.
top_1_infer = lmp.infer.Top1Infer(max_seq_len=128)
# Create top-K inference method instance.
top_k_infer = lmp.infer.TopKInfer(max_seq_len=128)
Different inference methods have different hyperparameters.
For example, top-K inference method has a parameter k
which represent the first k
possible tokens to sample.
import lmp.infer
# Create top-K inference method instance.
top_k_infer = lmp.infer.TopKInfer(k=5)
Generate continual text#
To generate continual text on a given text, one must provide a language model, its paired tokenizer and a text to an inference method. Meaningful generation result can only be achieve through pre-trained language models. The following example demonstrate the usage of generation without pre-training a language model.
import lmp.infer
import lmp.model
import lmp.tknzr
inference = lmp.infer.Top1Infer()
tokenizer = lmp.tknzr.CharTknzr()
model = lmp.model.ElmanNet(tknzr=tokenizer)
# Generate continual text.
generated_txt = inference.gen(model=model, tknzr=tokenizer, txt='abc')