lm_polygraph package

Subpackages

• lm_polygraph.estimators package
  • Submodules
    • lm_polygraph.estimators.attention_score module
      • AttentionScore
      • unpad_attentions()
    • lm_polygraph.estimators.boostedprob_score module
      • BoostedProbSequence
    • lm_polygraph.estimators.claim_conditioned_probability module
      • ClaimConditionedProbability
    • lm_polygraph.estimators.cocoa module
      • CocoaMSP
      • CocoaMTE
      • CocoaPPL
    • lm_polygraph.estimators.common module
      • compute_sim_score()
    • lm_polygraph.estimators.conditional_pointwise_mutual_information module
      • ConditionalPointwiseMutualInformation
      • MeanConditionalPointwiseMutualInformation
    • lm_polygraph.estimators.csl module
      • CSL
    • lm_polygraph.estimators.deg_mat module
      • DegMat
    • lm_polygraph.estimators.eccentricity module
      • Eccentricity
    • lm_polygraph.estimators.eig_val_laplacian module
      • EigValLaplacian
    • lm_polygraph.estimators.eigenscore module
      • EigenScore
    • lm_polygraph.estimators.ensemble_sequence_measures module
      • EPSrmi
      • EPSrmiabs
      • EPStu
      • PESrmi
      • PESrmiabs
      • PEStu
      • all_ep_estimators()
      • all_pe_estimators()
      • get_seq_level_ue()
    • lm_polygraph.estimators.ensemble_token_measures module
      • EPEnsembleEstimator
      • EPTdu
      • EPTent10
      • EPTent15
      • EPTent5
      • EPTepkl
      • EPTmi
      • EPTrmi
      • EPTtu
      • EnsembleEstimator
      • PEEnsembleEstimator
      • PETdu
      • PETent10
      • PETent15
      • PETent5
      • PETepkl
      • PETmi
      • PETrmi
      • PETtu
      • aggregate()
      • all_token_estimators()
    • lm_polygraph.estimators.estimator module
      • Estimator
    • lm_polygraph.estimators.fisher_rao module
      • FisherRao
    • lm_polygraph.estimators.focus module
      • Focus
      • IDFStats
      • calcu_idf()
      • entropy2()
      • load_idf()
      • token_level_focus_scores()
    • lm_polygraph.estimators.kernel_language_entropy module
      • KernelLanguageEntropy
      • heat_kernel()
      • laplacian_matrix()
      • normalize_kernel()
      • scale_entropy()
      • vn_entropy()
    • lm_polygraph.estimators.label_prob module
      • LabelProb
    • lm_polygraph.estimators.lexical_similarity module
      • LexicalSimilarity
    • lm_polygraph.estimators.linguistic_1s module
      • Linguistic1S
    • lm_polygraph.estimators.luq module
      • LUQ
    • lm_polygraph.estimators.mahalanobis_distance module
      • MahalanobisDistanceSeq
      • compute_inv_covariance()
      • create_cuda_tensor_from_numpy()
      • mahalanobis_distance_with_known_centroids_sigma_inv()
    • lm_polygraph.estimators.max_probability module
      • MaximumSequenceProbability
      • MaximumTokenProbability
    • lm_polygraph.estimators.monte_carlo_normalized_sequence_entropy module
      • MonteCarloNormalizedSequenceEntropy
    • lm_polygraph.estimators.monte_carlo_sequence_entropy module
      • MonteCarloSequenceEntropy
    • lm_polygraph.estimators.num_sem_sets module
      • NumSemSets
    • lm_polygraph.estimators.p_true module
      • PTrue
    • lm_polygraph.estimators.p_true_empirical module
      • PTrueEmpirical
    • lm_polygraph.estimators.p_true_sampling module
      • PTrueSampling
    • lm_polygraph.estimators.perplexity module
      • Perplexity
    • lm_polygraph.estimators.pointwise_mutual_information module
      • MeanPointwiseMutualInformation
      • PointwiseMutualInformation
    • lm_polygraph.estimators.ppl_md module
      • PPLMDSeq
      • load_array()
      • rank()
      • save_array()
    • lm_polygraph.estimators.rauq module
      • RAUQ
    • lm_polygraph.estimators.rde module
      • MCD_covariance()
      • RDESeq
      • load_array()
      • save_array()
    • lm_polygraph.estimators.relative_mahalanobis_distance module
      • RelativeMahalanobisDistanceSeq
      • load_array()
      • save_array()
    • lm_polygraph.estimators.renyi_neg module
      • RenyiNeg
    • lm_polygraph.estimators.sar module
      • SAR
    • lm_polygraph.estimators.self_certainty module
      • SelfCertainty
    • lm_polygraph.estimators.semantic_density module
      • SemanticDensity
    • lm_polygraph.estimators.semantic_entropy module
      • SemanticEntropy
    • lm_polygraph.estimators.semantic_entropy_token module
      • SemanticEntropyToken
      • split_classes()
    • lm_polygraph.estimators.sentence_sar module
      • SentenceSAR
    • lm_polygraph.estimators.token_entropy module
      • MeanTokenEntropy
      • TokenEntropy
    • lm_polygraph.estimators.token_sar module
      • TokenSAR
      • token_level_sar_scores()
    • lm_polygraph.estimators.verbalized_1s module
      • Verbalized1S
    • lm_polygraph.estimators.verbalized_2s module
      • Verbalized2S
• lm_polygraph.generation_metrics package
  • Submodules
    • lm_polygraph.generation_metrics.accuracy module
      • AccuracyMetric
    • lm_polygraph.generation_metrics.aggregated_metric module
      • AggregatedMetric
    • lm_polygraph.generation_metrics.alignscore module
      • AlignScore
    • lm_polygraph.generation_metrics.alignscore_utils module
      • AlignScorer
      • BERTAlignModel
      • ElectraDiscriminatorPredictions
      • Inferencer
      • ModelOutput
    • lm_polygraph.generation_metrics.bart_score module
      • BartScoreSeqMetric
    • lm_polygraph.generation_metrics.bert_score module
      • BertScoreMetric
    • lm_polygraph.generation_metrics.bleu module
      • BLEUMetric
    • lm_polygraph.generation_metrics.comet module
      • Comet
    • lm_polygraph.generation_metrics.generation_metric module
      • GenerationMetric
    • lm_polygraph.generation_metrics.model_score module
      • ModelScoreSeqMetric
      • ModelScoreTokenwiseMetric
    • lm_polygraph.generation_metrics.openai_fact_check module
      • OpenAIFactCheck
    • lm_polygraph.generation_metrics.preprocess_output_target module
      • PreprocessOutputTarget
    • lm_polygraph.generation_metrics.rouge module
      • RougeMetric
    • lm_polygraph.generation_metrics.sbert module
      • SbertMetric
• lm_polygraph.model_adapters package
  • Submodules
    • lm_polygraph.model_adapters.visual_whitebox_model module
      • VisualWhiteboxModel
    • lm_polygraph.model_adapters.whitebox_model_basic module
      • WhiteboxModelBasic
    • lm_polygraph.model_adapters.whitebox_model_vllm module
      • WhiteboxModelvLLM
• lm_polygraph.stat_calculators package
  • Submodules
    • lm_polygraph.stat_calculators.attention_eliciting_prompt module
      • AttentionElicitingPromptCalculator
      • BaseAttentionPromptCalculator
    • lm_polygraph.stat_calculators.attention_forward_pass module
      • AttentionForwardPassCalculator
    • lm_polygraph.stat_calculators.attention_forward_pass_visual module
      • AttentionForwardPassCalculatorVisual
    • lm_polygraph.stat_calculators.bart_score module
      • BartScoreCalculator
    • lm_polygraph.stat_calculators.claim_level_prompts module
    • lm_polygraph.stat_calculators.cross_encoder_similarity module
      • CrossEncoderSimilarityMatrixCalculator
    • lm_polygraph.stat_calculators.cross_encoder_visual_similarity module
      • CrossEncoderSimilarityMatrixVisualCalculator
    • lm_polygraph.stat_calculators.embeddings module
      • EmbeddingsCalculator
      • aggregate()
      • get_embeddings_from_output()
    • lm_polygraph.stat_calculators.ensemble_token_data module
      • EnsembleTokenLevelDataCalculator
    • lm_polygraph.stat_calculators.entropy module
      • EntropyCalculator
    • lm_polygraph.stat_calculators.extract_claims module
      • Claim
      • ClaimsExtractor
    • lm_polygraph.stat_calculators.greedy_alternatives_nli module
      • GreedyAlternativesFactPrefNLICalculator
      • GreedyAlternativesNLICalculator
      • eval_nli_model()
    • lm_polygraph.stat_calculators.greedy_cross_encoder_similarity module
      • GreedyCrossEncoderSimilarityMatrixCalculator
    • lm_polygraph.stat_calculators.greedy_lm_probs module
      • GreedyLMProbsCalculator
    • lm_polygraph.stat_calculators.greedy_lm_visual_probs module
      • GreedyLMProbsVisualCalculator
    • lm_polygraph.stat_calculators.greedy_probs module
      • GreedyProbsCalculator
    • lm_polygraph.stat_calculators.greedy_probs_blackbox module
      • BlackboxGreedyTextsCalculator
    • lm_polygraph.stat_calculators.greedy_semantic_matrix module
      • BaseGreedySemanticMatrixCalculator
      • ConcatGreedySemanticMatrixCalculator
      • GreedySemanticMatrixCalculator
    • lm_polygraph.stat_calculators.greedy_visual_probs module
      • GreedyProbsVisualCalculator
    • lm_polygraph.stat_calculators.infer_causal_lm_calculator module
      • InferCausalLMCalculator
    • lm_polygraph.stat_calculators.initial_state module
      • InitialStateCalculator
    • lm_polygraph.stat_calculators.model_score module
      • ModelScoreCalculator
    • lm_polygraph.stat_calculators.prompt module
      • BasePromptCalculator
      • ClaimPromptCalculator
      • PromptCalculator
      • SamplingPromptCalculator
    • lm_polygraph.stat_calculators.prompt_visual module
      • BasePromptVisualCalculator
      • ClaimPromptVisualCalculator
      • PromptVisualCalculator
      • SamplingPromptVisualCalculator
    • lm_polygraph.stat_calculators.raw_input module
      • RawInputCalculator
    • lm_polygraph.stat_calculators.sample module
      • BlackboxSamplingGenerationCalculator
      • OutputWrapper
      • SamplingGenerationCalculator
    • lm_polygraph.stat_calculators.sample_visual module
      • OutputWrapper
      • SamplingGenerationVisualCalculator
    • lm_polygraph.stat_calculators.semantic_classes module
      • SemanticClassesCalculator
    • lm_polygraph.stat_calculators.semantic_classes_claim_to_samples module
      • SemanticClassesClaimToSamplesCalculator
    • lm_polygraph.stat_calculators.semantic_matrix module
      • SemanticMatrixCalculator
    • lm_polygraph.stat_calculators.stat_calculator module
      • StatCalculator
    • lm_polygraph.stat_calculators.statistic_extraction module
      • TrainingStatisticExtractionCalculator
    • lm_polygraph.stat_calculators.statistic_extraction_visual module
      • TrainingStatisticExtractionCalculatorVisual
    • lm_polygraph.stat_calculators.vllm_logprobs_extraction module
      • VLLMLogprobsExtractionCalculator
• lm_polygraph.ue_metrics package
  • Submodules
    • lm_polygraph.ue_metrics.ece module
      • ECE
    • lm_polygraph.ue_metrics.isotonic_pcc module
      • IsotonicPCC
    • lm_polygraph.ue_metrics.kendalltau module
      • KendallTauCorrelation
    • lm_polygraph.ue_metrics.pr_auc module
      • PRAUC
    • lm_polygraph.ue_metrics.pred_rej_area module
      • PredictionRejectionArea
    • lm_polygraph.ue_metrics.rev_pairs_prop module
      • ReversedPairsProportion
    • lm_polygraph.ue_metrics.risk_cov_curve module
      • RiskCoverageCurveAUC
    • lm_polygraph.ue_metrics.roc_auc module
      • ROCAUC
    • lm_polygraph.ue_metrics.spearmanr module
      • SpearmanRankCorrelation
    • lm_polygraph.ue_metrics.ue_metric module
      • UEMetric
      • get_random_scores()
      • normalize()
      • normalize_metric()
      • skip_target_nans()
• lm_polygraph.utils package
  • Subpackages
    • lm_polygraph.utils.ensemble_utils package
      • Submodules
  • Submodules
    • lm_polygraph.utils.api_with_uncertainty module
      • APILogprobData
      • APIWithUncertainty
      • convert_api_logprobs()
    • lm_polygraph.utils.builder_enviroment_stat_calculator module
      • BuilderEnvironmentBase
      • BuilderEnvironmentStatCalculator
    • lm_polygraph.utils.builder_stat_calculator_simple module
      • load_stat_calculator()
    • lm_polygraph.utils.causal_lm_with_uncertainty module
      • CausalLMWithUncertainty
      • GenerateDecoderOnlyOutputWithUncertainty
    • lm_polygraph.utils.cir_model module
      • CenteredIsotonicRegression
    • lm_polygraph.utils.common module
      • flatten_results()
      • load_external_module()
      • load_image()
      • load_processor()
      • polygraph_module_init()
      • seq_man_key()
    • lm_polygraph.utils.dataset module
      • Dataset
    • lm_polygraph.utils.deberta module
      • Deberta
      • MultilingualDeberta
    • lm_polygraph.utils.estimate_uncertainty module
      • UncertaintyOutput
      • estimate_uncertainty()
    • lm_polygraph.utils.factory_estimator module
      • FactoryEstimator
      • load_simple_estimators()
    • lm_polygraph.utils.factory_stat_calculator module
      • FactoryStatCalculator
      • StatCalculatorContainer
      • create_stat_calculator()
    • lm_polygraph.utils.generation_parameters module
      • GenerationParameters
      • GenerationParametersFactory
    • lm_polygraph.utils.manager module
      • UEManager
      • order_calculators()
    • lm_polygraph.utils.model module
      • BlackboxModel
      • Model
      • WhiteboxModel
      • create_ensemble()
    • lm_polygraph.utils.normalize module
      • filter_nans()
      • get_mans_ues_metrics()
    • lm_polygraph.utils.ood_detection module
      • calculate_ood_from_mans()
    • lm_polygraph.utils.openai_chat module
      • OpenAIChat
    • lm_polygraph.utils.processor module
      • Logger
      • Processor
    • lm_polygraph.utils.token_restoration module
      • collect_sample_token_level_uncertainties()
      • collect_token_level_uncertainties()
      • get_collect_fn()
      • update_token_level_scores()
    • lm_polygraph.utils.vllm_with_uncertainty module