lm_polygraph.stat_calculators package

Submodules

lm_polygraph.stat_calculators.bart_score module

class lm_polygraph.stat_calculators.bart_score.BartScoreCalculator(device=None, max_length=256, checkpoint='facebook/bart-large-cnn')

Bases: StatCalculator

load(path=None)

Load model from paraphrase finetuning

multi_ref_score(srcs, tgts: List[List[str]], agg='mean', batch_size=4)

score(srcs, tgts, batch_size=4)

Score a batch of examples

test(batch_size=3)

Test

lm_polygraph.stat_calculators.cross_encoder_similarity module

class lm_polygraph.stat_calculators.cross_encoder_similarity.CrossEncoderSimilarityMatrixCalculator(nli_model)

Bases: StatCalculator

Calculates the cross-encoder similarity matrix for generation samples using RoBERTa model.

lm_polygraph.stat_calculators.embeddings module

class lm_polygraph.stat_calculators.embeddings.EmbeddingsCalculator

Bases: StatCalculator

lm_polygraph.stat_calculators.embeddings.aggregate(x, aggregation_method, axis)

lm_polygraph.stat_calculators.embeddings.get_embeddings_from_output(output, batch, model_type, hidden_state: List[str] = ['encoder', 'decoder'], ignore_padding: bool = True, use_averaging: bool = True, all_layers: bool = False, aggregation_method: str = 'mean')

lm_polygraph.stat_calculators.ensemble_token_data module

class lm_polygraph.stat_calculators.ensemble_token_data.EnsembleTokenLevelDataCalculator

Bases: StatCalculator

lm_polygraph.stat_calculators.entropy module

class lm_polygraph.stat_calculators.entropy.EntropyCalculator

Bases: StatCalculator

Calculates entropy of probabilities at each token position in the generation of a Whitebox model.

lm_polygraph.stat_calculators.extract_claims module

class lm_polygraph.stat_calculators.extract_claims.Claim(claim_text: str, sentence: str, aligned_token_ids: List[int])

Bases: object

aligned_token_ids: List[int]

claim_text: str

sentence: str

class lm_polygraph.stat_calculators.extract_claims.ClaimsExtractor(openai_chat: OpenAIChat, sent_separators: str = '.?!。？！\n', language: str = 'en')

Bases: StatCalculator

Extracts claims from the text of the model generation.

claims_from_text(text: str, tokens: List[int], tokenizer) → List[Claim]

lm_polygraph.stat_calculators.greedy_alternatives_nli module

class lm_polygraph.stat_calculators.greedy_alternatives_nli.GreedyAlternativesFactPrefNLICalculator(nli_model)

Bases: StatCalculator

class lm_polygraph.stat_calculators.greedy_alternatives_nli.GreedyAlternativesNLICalculator(nli_model)

Bases: StatCalculator

lm_polygraph.stat_calculators.greedy_lm_probs module

class lm_polygraph.stat_calculators.greedy_lm_probs.GreedyLMProbsCalculator

Bases: StatCalculator

Calculates probabilities of the model generations without input texts. Used to calculate P(y_t|y_<t) subtrahend in PointwiseMutualInformation.

lm_polygraph.stat_calculators.greedy_probs module

class lm_polygraph.stat_calculators.greedy_probs.BlackboxGreedyTextsCalculator

Bases: StatCalculator

Calculates generation texts for Blackbox model (lm_polygraph.BlackboxModel).

class lm_polygraph.stat_calculators.greedy_probs.GreedyProbsCalculator(n_alternatives: int = 10)

Bases: StatCalculator

For Whitebox model (lm_polygraph.WhiteboxModel), at input texts batch calculates: * generation texts * tokens of the generation texts * probabilities distribution of the generated tokens * attention masks across the model (if applicable) * embeddings from the model

lm_polygraph.stat_calculators.model_score module

class lm_polygraph.stat_calculators.model_score.ModelScoreCalculator(prompt: str = 'Paraphrase "{}": ', batch_size: int = 10)

Bases: StatCalculator

lm_polygraph.stat_calculators.prompt module

class lm_polygraph.stat_calculators.prompt.PromptCalculator(prompt: str, expected: str, method: str, input_text_dependency: str = 'input_texts', sample_text_dependency: str | None = 'sample_texts', generation_text_dependency: str = 'greedy_texts')

Bases: StatCalculator

Calculates the probability for a specific token to be generated from the specific prompt. Used for P(True)-based methods.

lm_polygraph.stat_calculators.sample module

class lm_polygraph.stat_calculators.sample.BlackboxSamplingGenerationCalculator(samples_n: int = 10)

Bases: StatCalculator

Calculates several sampled texts for Blackbox model (lm_polygraph.BlackboxModel).

class lm_polygraph.stat_calculators.sample.SamplingGenerationCalculator(samples_n: int = 10)

Bases: StatCalculator

For Whitebox model (lm_polygraph.WhiteboxModel), at input texts batch calculates: * sampled texts * tokens of the sampled texts * probabilities of the sampled tokens generation

lm_polygraph.stat_calculators.semantic_matrix module

class lm_polygraph.stat_calculators.semantic_matrix.SemanticMatrixCalculator(nli_model)

Bases: StatCalculator

Calculates the NLI semantic matrix for generation samples using DeBERTa model.

lm_polygraph.stat_calculators.stat_calculator module

class lm_polygraph.stat_calculators.stat_calculator.StatCalculator(**kwargs)

Bases: ABC

Abstract class for some particular statistics calculation. Used to re-use same statistics across different uncertainty estimators at lm_polygraph.estimators. See the list of available calculators at lm_polygraph/stat_calculators/__init__.py.

While estimators specify stats_dependencies to re-use these StatCalculator calculations, calculators can also specify dependencies on other calculators.

UEManager at lm_polygraph.utils.manager will order all the needed calculators and estimators to be called in the correct order. Any cycle dependencies among calculators will be spotted by UEManager and end with an exception.

Each new StatCalculator needs to be registered at lm_polygraph/stat_calculators/__init__.py to be seen be UEManager.

property stat_dependencies: List[str]

Returns:

List[str]: Names of statistics dependencies which this class needs at __call__.

property stats: List[str]

Returns:

List[str]: Names of statistics which can be calculated by this class.

Module contents