lm_polygraph.estimators package

Submodules

lm_polygraph.estimators.claim_conditioned_probability module

class lm_polygraph.estimators.claim_conditioned_probability.ClaimConditionedProbability

Bases: Estimator

lm_polygraph.estimators.common module

lm_polygraph.estimators.common.compute_sim_score(answers, affinity, similarity_score)

lm_polygraph.estimators.conditional_pointwise_mutual_information module

class lm_polygraph.estimators.conditional_pointwise_mutual_information.ConditionalPointwiseMutualInformation(tau: float = 0.0656, lambd: float = 3.599)

Bases: Estimator

Estimates the token-level uncertainty of a language model following the method of Conditional Pointwise Mutual Information (CPMI) as provided in the paper https://arxiv.org/abs/2210.13210. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

class lm_polygraph.estimators.conditional_pointwise_mutual_information.MeanConditionalPointwiseMutualInformation(tau: float = 0.0656, lambd: float = 3.599)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of Conditional Pointwise Mutual Information (CPMI) as provided in the paper https://arxiv.org/abs/2210.13210. The sequence-level estimation is calculated as average token-level CPMI estimations. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

lm_polygraph.estimators.deg_mat module

class lm_polygraph.estimators.deg_mat.DegMat(similarity_score: Literal['NLI_score', 'Jaccard_score'] = 'NLI_score', affinity: Literal['entail', 'contra'] = 'entail', verbose: bool = False)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “The Degree Matrix” as provided in the paper https://arxiv.org/abs/2305.19187. Works with both whitebox and blackbox models (initialized using lm_polygraph.utils.model.BlackboxModel/WhiteboxModel).

Elements on diagonal of matrix D are sums of similarities between the particular number (position in matrix) and other answers. Thus, it is an average pairwise distance (lower values indicated smaller distance between answers which means greater uncertainty).

U_DegMat(i, stats)

lm_polygraph.estimators.eccentricity module

class lm_polygraph.estimators.eccentricity.Eccentricity(similarity_score: Literal['NLI_score', 'Jaccard_score'] = 'NLI_score', affinity: Literal['entail', 'contra'] = 'entail', verbose: bool = False, thres: float = 0.9)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “Eccentricity” as provided in the paper https://arxiv.org/abs/2305.19187. Works with both whitebox and blackbox models (initialized using lm_polygraph.utils.model.BlackboxModel/WhiteboxModel).

Method calculates a frobenious (euclidian) norm between all eigenvectors that are informative embeddings of graph Laplacian (lower norm -> closer embeddings -> higher eigenvectors -> greater uncertainty).

U_Eccentricity(i, stats)

lm_polygraph.estimators.eig_val_laplacian module

class lm_polygraph.estimators.eig_val_laplacian.EigValLaplacian(similarity_score: Literal['NLI_score', 'Jaccard_score'] = 'NLI_score', affinity: Literal['entail', 'contra'] = 'entail', verbose: bool = False)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “Sum of Eigenvalues of the Graph Laplacian” as provided in the paper https://arxiv.org/abs/2305.19187. Works with both whitebox and blackbox models (initialized using lm_polygraph.utils.model.BlackboxModel/WhiteboxModel).

A continuous analogue to the number of semantic sets (higher values means greater uncertainty).

U_EigVal_Laplacian(i, stats)

lm_polygraph.estimators.ensemble_sequence_measures module

class lm_polygraph.estimators.ensemble_sequence_measures.EPSrmi

Bases: Estimator

class lm_polygraph.estimators.ensemble_sequence_measures.EPSrmiabs

Bases: Estimator

class lm_polygraph.estimators.ensemble_sequence_measures.EPStu

Bases: Estimator

class lm_polygraph.estimators.ensemble_sequence_measures.PESrmi

Bases: Estimator

class lm_polygraph.estimators.ensemble_sequence_measures.PESrmiabs

Bases: Estimator

class lm_polygraph.estimators.ensemble_sequence_measures.PEStu

Bases: Estimator

lm_polygraph.estimators.ensemble_sequence_measures.all_ep_estimators()

lm_polygraph.estimators.ensemble_sequence_measures.all_pe_estimators()

lm_polygraph.estimators.ensemble_sequence_measures.get_seq_level_ue(sequence_level_data: Dict[str, Tensor]) → Dict[str, ndarray]

lm_polygraph.estimators.ensemble_token_measures module

class lm_polygraph.estimators.ensemble_token_measures.EPEnsembleEstimator

Bases: EnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.EPTdu

Bases: EPEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.EPTent10

Bases: EPEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.EPTent15

Bases: EPEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.EPTent5

Bases: EPEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.EPTepkl

Bases: EPEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.EPTmi

Bases: EPEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.EPTrmi

Bases: EPEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.EPTtu

Bases: EPEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.EnsembleEstimator

Bases: Estimator

class lm_polygraph.estimators.ensemble_token_measures.PEEnsembleEstimator

Bases: EnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.PETdu

Bases: PEEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.PETent10

Bases: PEEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.PETent15

Bases: PEEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.PETent5

Bases: PEEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.PETepkl

Bases: PEEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.PETmi

Bases: PEEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.PETrmi

Bases: PEEnsembleEstimator

class lm_polygraph.estimators.ensemble_token_measures.PETtu

Bases: PEEnsembleEstimator

lm_polygraph.estimators.ensemble_token_measures.aggregate(posterior_mode, ue_name, token_level_data) → ndarray

lm_polygraph.estimators.ensemble_token_measures.all_token_estimators()

lm_polygraph.estimators.estimator module

class lm_polygraph.estimators.estimator.Estimator(**kwargs)

Bases: ABC

Abstract estimator class, which estimates the uncertainty of a language model.

lm_polygraph.estimators.fisher_rao module

class lm_polygraph.estimators.fisher_rao.FisherRao(verbose: bool = False, temperature: float = 2)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “FisherRao” as provided in the paper https://arxiv.org/pdf/2212.09171.pdf. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

This method calculates the generation Fisher-Rao distance between probability distribution for each token and uniform distribution. Code adapted from https://github.com/icannos/Todd/blob/master/Todd/itscorers.py

lm_polygraph.estimators.lexical_similarity module

class lm_polygraph.estimators.lexical_similarity.LexicalSimilarity(metric: str = 'rougeL')

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “Lexical Similarity” as provided in the paper https://arxiv.org/abs/2302.09664. Works with both whitebox and blackbox models (initialized using lm_polygraph.utils.model.BlackboxModel/WhiteboxModel).

The method calculates mean similarity between all pairs of sampled generations with minus sign. The number of samples is controlled by lm_polygraph.stat_calculators.sample.SamplingGenerationCalculator ‘samples_n’ parameter.

lm_polygraph.estimators.mahalanobis_distance module

class lm_polygraph.estimators.mahalanobis_distance.MahalanobisDistanceSeq(embeddings_type: str = 'decoder', parameters_path: str | None = None, normalize: bool = False)

Bases: Estimator

lm_polygraph.estimators.mahalanobis_distance.compute_inv_covariance(centroids, train_features, jitters=None)

This function computes inverse covariance matrix that is required by Mahalanobis distance: MD = sqrt((h(x) - mu)^{T} Sigma^{-1} (h(x) - mu))

lm_polygraph.estimators.mahalanobis_distance.create_cuda_tensor_from_numpy(array)

lm_polygraph.estimators.mahalanobis_distance.mahalanobis_distance_with_known_centroids_sigma_inv(centroids, centroids_mask, sigma_inv, eval_features)

• This function takes in centroids, centroids_mask, sigma_inv, and eval_features.

• tensor of Mahalanobis distances is returned.

lm_polygraph.estimators.max_probability module

class lm_polygraph.estimators.max_probability.MaximumSequenceProbability

Bases: Estimator

Estimates the sequence-level uncertainty of a language model by calculating the log-probability of the generation with minus sign. It is calculated as the sum of log-probabilities in each token. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

class lm_polygraph.estimators.max_probability.MaximumTokenProbability

Bases: Estimator

Estimates the token-level uncertainty of a language model by calculating the log-probability for each token during autoregressive generation.

lm_polygraph.estimators.monte_carlo_normalized_sequence_entropy module

class lm_polygraph.estimators.monte_carlo_normalized_sequence_entropy.MonteCarloNormalizedSequenceEntropy

Bases: Estimator

lm_polygraph.estimators.monte_carlo_sequence_entropy module

class lm_polygraph.estimators.monte_carlo_sequence_entropy.MonteCarloSequenceEntropy

Bases: Estimator

lm_polygraph.estimators.num_sem_sets module

class lm_polygraph.estimators.num_sem_sets.NumSemSets(verbose: bool = False)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “Number of Semantic Sets” as provided in the paper https://arxiv.org/abs/2305.19187. Works with both whitebox and blackbox models (initialized using lm_polygraph.utils.model.BlackboxModel/WhiteboxModel).

U_NumSemSets(i, stats)

find_connected_components(graph)

lm_polygraph.estimators.p_true module

class lm_polygraph.estimators.p_true.PTrue

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “P(True)” as provided in the paper https://arxiv.org/abs/2207.05221. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

On input question q and model generation a, the method uses the following prompt:

Question: {q} Possible answer: {a} Is the possible answer:

1. True

2. False

The possible answer is:

and calculates the log-probability of ‘True’ token with minus sign.

lm_polygraph.estimators.p_true_sampling module

class lm_polygraph.estimators.p_true_sampling.PTrueSampling

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “P(True)” as provided in the paper https://arxiv.org/abs/2207.05221 and model samples. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

On input question q, model generation a and several text samples s, the method uses the following prompt:

Question: {q} Here are some ideas that were brainstormed: {s} Possible answer: {a} Is the possible answer:

1. True

2. False

The possible answer is:

and calculates the log-probability of True token with minus sign.

lm_polygraph.estimators.perplexity module

class lm_polygraph.estimators.perplexity.Perplexity

Bases: Estimator

lm_polygraph.estimators.pointwise_mutual_information module

class lm_polygraph.estimators.pointwise_mutual_information.MeanPointwiseMutualInformation

Bases: Estimator

Estimates the sequence-level uncertainty of a language model using Pointwise Mutual Information. The sequence-level estimation is calculated as average token-level PMI estimations. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

class lm_polygraph.estimators.pointwise_mutual_information.PointwiseMutualInformation

Bases: Estimator

Estimates the token-level uncertainty of a language model using Pointwise Mutual Information. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

lm_polygraph.estimators.ppl_md module

class lm_polygraph.estimators.ppl_md.PPLMDSeq(embeddings_type: str = 'decoder', md_type: str = 'MD', parameters_path: str | None = None, normalize: bool = False)

Bases: Estimator

lm_polygraph.estimators.ppl_md.load_array(filename)

lm_polygraph.estimators.ppl_md.rank(target_array, source_array)

lm_polygraph.estimators.ppl_md.save_array(array, filename)

lm_polygraph.estimators.rde module

lm_polygraph.estimators.rde.MCD_covariance(X, y=None, label=None, seed=42)

class lm_polygraph.estimators.rde.RDESeq(embeddings_type: str = 'decoder', parameters_path: str | None = None, normalize: bool = False)

Bases: Estimator

The RDE method improves over MD by reducing the dimensionality of h(x) via PCA decomposition. It also computes the covariance matrix in a robust way using the Minimum Covariance Determinant estimate (Rousseeuw, 1984).

load_mcd()

load_pca()

save_mcd()

save_pca()

lm_polygraph.estimators.rde.load_array(filename)

lm_polygraph.estimators.rde.save_array(array, filename)

lm_polygraph.estimators.relative_mahalanobis_distance module

class lm_polygraph.estimators.relative_mahalanobis_distance.RelativeMahalanobisDistanceSeq(embeddings_type: str = 'decoder', parameters_path: str | None = None, normalize: bool = False)

Bases: Estimator

Ren et al. (2023) showed that it might be useful to adjust the Mahalanobis distance score by subtracting from it the other Mahalanobis distance MD_0(x) computed for some large general purpose dataset covering many domain. RMD(x) = MD(x) - MD_0(x)

lm_polygraph.estimators.relative_mahalanobis_distance.load_array(filename)

lm_polygraph.estimators.relative_mahalanobis_distance.save_array(array, filename)

lm_polygraph.estimators.renyi_neg module

class lm_polygraph.estimators.renyi_neg.RenyiNeg(verbose: bool = False, alpha: float = 0.5, temperature: float = 2)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “RenyiNeg” as provided in the paper https://arxiv.org/pdf/2212.09171.pdf. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

This method calculates the generation Rényi divergence between probability distribution for each token and uniform distribution. Code adapted from https://github.com/icannos/Todd/blob/master/Todd/itscorers.py

lm_polygraph.estimators.sar module

class lm_polygraph.estimators.sar.SAR(verbose: bool = False)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “Token SAR” as provided in the paper https://arxiv.org/abs/2307.01379. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

This method calculates the sum of corrected probability using tokenSAR of the generated text and text relevance relative to all other generations.

lm_polygraph.estimators.semantic_entropy module

class lm_polygraph.estimators.semantic_entropy.SemanticEntropy(verbose: bool = False, class_probability_estimation: str = 'sum')

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “Semantic entropy” as provided in the paper https://arxiv.org/abs/2302.09664. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

This method calculates the generation entropy estimations merged by semantic classes using Monte-Carlo. The number of samples is controlled by lm_polygraph.stat_calculators.sample.SamplingGenerationCalculator ‘samples_n’ parameter.

batched_call(hyps_list: List[List[str]], loglikelihoods_list: List[List[float]] | None, log_weights: List[List[float]] | None = None) → array

lm_polygraph.estimators.semantic_entropy_token module

class lm_polygraph.estimators.semantic_entropy_token.SemanticEntropyToken(tokenizer_path: str, tokenizer_save_path: str, semantic_bert_path: str = 'sentence-transformers/bert-base-nli-mean-tokens', batch_size: int = 10)

Bases: Estimator

lm_polygraph.estimators.semantic_entropy_token.split_classes(tokens: List[str], batch_size: int, semantic_bert_path: str = 'sentence-transformers/bert-base-nli-mean-tokens', sim_threshold: float = 0.85) → ndarray

lm_polygraph.estimators.sentence_sar module

class lm_polygraph.estimators.sentence_sar.SentenceSAR(verbose: bool = False)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “Sentence SAR” as provided in the paper https://arxiv.org/abs/2307.01379. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

This method calculates the sum of the probability of the generated text and text relevance relative to all other generations.

lm_polygraph.estimators.token_entropy module

class lm_polygraph.estimators.token_entropy.MeanTokenEntropy

Bases: Estimator

Estimates the sequence-level uncertainty of a language model by calculating the mean entropy among all tokens in the generation. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

class lm_polygraph.estimators.token_entropy.TokenEntropy

Bases: Estimator

Estimates the token-level uncertainty of a language model by calculating the entropy for each token in the generation. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

lm_polygraph.estimators.token_sar module

class lm_polygraph.estimators.token_sar.TokenSAR(verbose: bool = False)

Bases: Estimator

Estimates the sequence-level uncertainty of a language model following the method of “Token SAR” as provided in the paper https://arxiv.org/abs/2307.01379. Works only with whitebox models (initialized using lm_polygraph.utils.model.WhiteboxModel).

This method calculates the weighted sum of log_likelihoods with weights computed using token relevance.

Module contents