lm_polygraph.estimators.p_true_sampling module

class lm_polygraph.estimators.p_true_sampling.PTrueSampling[source]

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “P(True)” as provided in the paper https://arxiv.org/abs/2207.05221 and model samples. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

On input question q, model generation a and several text samples s, the method uses the following prompt:

Question: {q} Here are some ideas that were brainstormed: {s} Possible answer: {a} Is the possible answer:

  1. True

  2. False

The possible answer is:

and calculates the log-probability of True token with minus sign.