lm_polygraph.estimators.p_true module

class lm_polygraph.estimators.p_true.PTrue[source]

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “P(True)” as provided in the paper https://arxiv.org/abs/2207.05221. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

On input question q and model generation a, the method uses the following prompt:

Question: {q} Possible answer: {a} Is the possible answer:

  1. True

  2. False

The possible answer is:

and calculates the log-probability of ‘True’ token with minus sign.