lm_polygraph.estimators.linguistic_1s module

class lm_polygraph.estimators.linguistic_1s.Linguistic1S(expressions: Dict[str, float], name_postfix='')

Bases: Estimator

Estimates sequence-level uncertainty of a language model by extracting the confidence estimate from the model’s answer using a provided regex. Model is expected to output confidence using one of the provided expressions, i.e. “Highly Likely”, “Probable” etc. Mapping between expressions of confidence and their corresponding numerical values has to be provided. To use this estimator, model has to be correctly prompted to output it’s confidence in the answer. Adapted from the original implementation in the paper https://arxiv.org/abs/2305.14975