lm_polygraph.estimators.verbalized_2s module

class lm_polygraph.estimators.verbalized_2s.Verbalized2S(confidence_prompt: str, confidence_regex: str = '', max_new_tokens: int = 10, name_postfix='')

Bases: Estimator

Asks model to output it’s confidence in a provided follow-up prompt and extracts the confidence estimate from the model’s answer using a provided regex. Only usabe for instruct-finetuned models with chat template support. Adapted from the original implementation in the paper https://arxiv.org/abs/2305.14975