lm_polygraph.estimators.deg_mat module

class lm_polygraph.estimators.deg_mat.DegMat(similarity_score: Literal['NLI_score', 'Jaccard_score'] = 'NLI_score', affinity: Literal['entail', 'contra'] = 'entail', verbose: bool = False)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “The Degree Matrix” as provided in the paper https://arxiv.org/abs/2305.19187. Works with both whitebox and blackbox models (initialized using lm_polygraph.utils.model.BlackboxModel/WhiteboxModel).

Elements on diagonal of matrix D are sums of similarities between the particular number (position in matrix) and other answers. Thus, it is an average pairwise distance (lower values indicated smaller distance between answers which means greater uncertainty).

U_DegMat(i, stats)