lm_polygraph.estimators.semantic_entropy module

class lm_polygraph.estimators.semantic_entropy.SemanticEntropy(verbose: bool = False, class_probability_estimation: str = 'sum', entropy_estimation: str = 'mean', use_unique_responses: bool = False)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “Semantic entropy” as provided in the paper https://arxiv.org/abs/2302.09664. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

This method calculates the generation entropy estimations merged by semantic classes using Monte-Carlo. The number of samples is controlled by lm_polygraph.stat_calculators.sample.SamplingGenerationCalculator ‘samples_n’ parameter.

The entropy_estimation parameter supports two methods: - “mean”: Mean entropy estimation from the original paper (https://arxiv.org/abs/2302.09664). - “direct”: Direct entropy estimation from the proper estimator in the SDLG paper

(https://arxiv.org/pdf/2406.04306).

batched_call(hyps_list: List[List[str]], loglikelihoods_list: List[List[float]] | None, log_weights: List[List[float]] | None = None) → array

get_unique_hypos_by_class(batch_i, hyps_list)