lm_polygraph package

Subpackages

• lm_polygraph.app package
  • Submodules
  • lm_polygraph.app.parsers module
    • parse_ensemble()
    • parse_model()
    • parse_seq_ue_method()
    • parse_tok_ue_method()
  • lm_polygraph.app.service module
  • lm_polygraph.app.service_helpers module
  • lm_polygraph.app.service_modal module
  • lm_polygraph.app.test_service module
  • Module contents
• lm_polygraph.estimators package
  • Submodules
  • lm_polygraph.estimators.claim_conditioned_probability module
    • ClaimConditionedProbability
  • lm_polygraph.estimators.common module
    • compute_sim_score()
  • lm_polygraph.estimators.conditional_pointwise_mutual_information module
    • ConditionalPointwiseMutualInformation
    • MeanConditionalPointwiseMutualInformation
  • lm_polygraph.estimators.deg_mat module
    • DegMat
      • DegMat.U_DegMat()
  • lm_polygraph.estimators.eccentricity module
    • Eccentricity
      • Eccentricity.U_Eccentricity()
  • lm_polygraph.estimators.eig_val_laplacian module
    • EigValLaplacian
      • EigValLaplacian.U_EigVal_Laplacian()
  • lm_polygraph.estimators.ensemble_sequence_measures module
    • EPSrmi
    • EPSrmiabs
    • EPStu
    • PESrmi
    • PESrmiabs
    • PEStu
    • all_ep_estimators()
    • all_pe_estimators()
    • get_seq_level_ue()
  • lm_polygraph.estimators.ensemble_token_measures module
    • EPEnsembleEstimator
    • EPTdu
    • EPTent10
    • EPTent15
    • EPTent5
    • EPTepkl
    • EPTmi
    • EPTrmi
    • EPTtu
    • EnsembleEstimator
    • PEEnsembleEstimator
    • PETdu
    • PETent10
    • PETent15
    • PETent5
    • PETepkl
    • PETmi
    • PETrmi
    • PETtu
    • aggregate()
    • all_token_estimators()
  • lm_polygraph.estimators.estimator module
    • Estimator
  • lm_polygraph.estimators.fisher_rao module
    • FisherRao
  • lm_polygraph.estimators.lexical_similarity module
    • LexicalSimilarity
  • lm_polygraph.estimators.mahalanobis_distance module
    • MahalanobisDistanceSeq
    • compute_inv_covariance()
    • create_cuda_tensor_from_numpy()
    • mahalanobis_distance_with_known_centroids_sigma_inv()
  • lm_polygraph.estimators.max_probability module
    • MaximumSequenceProbability
    • MaximumTokenProbability
  • lm_polygraph.estimators.monte_carlo_normalized_sequence_entropy module
    • MonteCarloNormalizedSequenceEntropy
  • lm_polygraph.estimators.monte_carlo_sequence_entropy module
    • MonteCarloSequenceEntropy
  • lm_polygraph.estimators.num_sem_sets module
    • NumSemSets
      • NumSemSets.U_NumSemSets()
      • NumSemSets.find_connected_components()
  • lm_polygraph.estimators.p_true module
    • PTrue
  • lm_polygraph.estimators.p_true_sampling module
    • PTrueSampling
  • lm_polygraph.estimators.perplexity module
    • Perplexity
  • lm_polygraph.estimators.pointwise_mutual_information module
    • MeanPointwiseMutualInformation
    • PointwiseMutualInformation
  • lm_polygraph.estimators.ppl_md module
    • PPLMDSeq
    • load_array()
    • rank()
    • save_array()
  • lm_polygraph.estimators.rde module
    • MCD_covariance()
    • RDESeq
      • RDESeq.load_mcd()
      • RDESeq.load_pca()
      • RDESeq.save_mcd()
      • RDESeq.save_pca()
    • load_array()
    • save_array()
  • lm_polygraph.estimators.relative_mahalanobis_distance module
    • RelativeMahalanobisDistanceSeq
    • load_array()
    • save_array()
  • lm_polygraph.estimators.renyi_neg module
    • RenyiNeg
  • lm_polygraph.estimators.sar module
    • SAR
  • lm_polygraph.estimators.semantic_entropy module
    • SemanticEntropy
      • SemanticEntropy.batched_call()
  • lm_polygraph.estimators.semantic_entropy_token module
    • SemanticEntropyToken
    • split_classes()
  • lm_polygraph.estimators.sentence_sar module
    • SentenceSAR
  • lm_polygraph.estimators.token_entropy module
    • MeanTokenEntropy
    • TokenEntropy
  • lm_polygraph.estimators.token_sar module
    • TokenSAR
  • Module contents
• lm_polygraph.generation_metrics package
  • Submodules
  • lm_polygraph.generation_metrics.accuracy module
    • AccuracyMetric
  • lm_polygraph.generation_metrics.aggregated_metric module
    • AggregatedMetric
  • lm_polygraph.generation_metrics.alignscore module
    • AlignScore
  • lm_polygraph.generation_metrics.alignscore_utils module
    • AlignScorer
      • AlignScorer.score()
    • BERTAlignModel
      • BERTAlignModel.configure_optimizers()
      • BERTAlignModel.electra_forward()
      • BERTAlignModel.forward()
      • BERTAlignModel.mse_loss()
      • BERTAlignModel.training_step()
      • BERTAlignModel.training_step_end()
      • BERTAlignModel.validation_epoch_end()
      • BERTAlignModel.validation_step()
      • BERTAlignModel.validation_step_end()
    • ElectraDiscriminatorPredictions
      • ElectraDiscriminatorPredictions.forward()
    • Inferencer
      • Inferencer.batch_tokenize()
      • Inferencer.chunks()
      • Inferencer.inference()
      • Inferencer.inference_example_batch()
      • Inferencer.inference_per_example()
      • Inferencer.inference_reg()
      • Inferencer.nlg_eval()
      • Inferencer.smart_doc()
      • Inferencer.smart_l()
      • Inferencer.smart_n()
    • ModelOutput
      • ModelOutput.all_loss
      • ModelOutput.attentions
      • ModelOutput.hidden_states
      • ModelOutput.loss
      • ModelOutput.loss_nums
      • ModelOutput.prediction_logits
      • ModelOutput.reg_label_logits
      • ModelOutput.seq_relationship_logits
      • ModelOutput.tri_label_logits
  • lm_polygraph.generation_metrics.bart_score module
    • BartScoreSeqMetric
      • BartScoreSeqMetric.load()
      • BartScoreSeqMetric.score()
      • BartScoreSeqMetric.test()
  • lm_polygraph.generation_metrics.bert_score module
    • BertScoreMetric
  • lm_polygraph.generation_metrics.bleu module
    • BLEUMetric
  • lm_polygraph.generation_metrics.comet module
    • Comet
  • lm_polygraph.generation_metrics.generation_metric module
    • GenerationMetric
  • lm_polygraph.generation_metrics.model_score module
    • ModelScoreSeqMetric
    • ModelScoreTokenwiseMetric
  • lm_polygraph.generation_metrics.openai_fact_check module
    • OpenAIFactCheck
  • lm_polygraph.generation_metrics.rouge module
    • RougeMetric
  • lm_polygraph.generation_metrics.sbert module
    • SbertMetric
  • Module contents
• lm_polygraph.stat_calculators package
  • Submodules
  • lm_polygraph.stat_calculators.bart_score module
    • BartScoreCalculator
      • BartScoreCalculator.load()
      • BartScoreCalculator.multi_ref_score()
      • BartScoreCalculator.score()
      • BartScoreCalculator.test()
  • lm_polygraph.stat_calculators.cross_encoder_similarity module
    • CrossEncoderSimilarityMatrixCalculator
  • lm_polygraph.stat_calculators.embeddings module
    • EmbeddingsCalculator
    • aggregate()
    • get_embeddings_from_output()
  • lm_polygraph.stat_calculators.ensemble_token_data module
    • EnsembleTokenLevelDataCalculator
  • lm_polygraph.stat_calculators.entropy module
    • EntropyCalculator
  • lm_polygraph.stat_calculators.extract_claims module
    • Claim
      • Claim.aligned_token_ids
      • Claim.claim_text
      • Claim.sentence
    • ClaimsExtractor
      • ClaimsExtractor.claims_from_text()
  • lm_polygraph.stat_calculators.greedy_alternatives_nli module
    • GreedyAlternativesFactPrefNLICalculator
    • GreedyAlternativesNLICalculator
  • lm_polygraph.stat_calculators.greedy_lm_probs module
    • GreedyLMProbsCalculator
  • lm_polygraph.stat_calculators.greedy_probs module
    • BlackboxGreedyTextsCalculator
    • GreedyProbsCalculator
  • lm_polygraph.stat_calculators.model_score module
    • ModelScoreCalculator
  • lm_polygraph.stat_calculators.prompt module
    • PromptCalculator
  • lm_polygraph.stat_calculators.sample module
    • BlackboxSamplingGenerationCalculator
    • SamplingGenerationCalculator
  • lm_polygraph.stat_calculators.semantic_matrix module
    • SemanticMatrixCalculator
  • lm_polygraph.stat_calculators.stat_calculator module
    • StatCalculator
      • StatCalculator.stat_dependencies
      • StatCalculator.stats
  • Module contents
• lm_polygraph.ue_metrics package
  • Submodules
  • lm_polygraph.ue_metrics.kendalltau module
    • KendallTauCorrelation
  • lm_polygraph.ue_metrics.pr_auc module
    • PRAUC
      • PRAUC.preprocess_inf()
  • lm_polygraph.ue_metrics.pred_rej_area module
    • PredictionRejectionArea
  • lm_polygraph.ue_metrics.rev_pairs_prop module
    • ReversedPairsProportion
  • lm_polygraph.ue_metrics.risk_cov_curve module
    • RiskCoverageCurveAUC
  • lm_polygraph.ue_metrics.roc_auc module
    • ROCAUC
      • ROCAUC.is_ood_metric
      • ROCAUC.preprocess_inf()
  • lm_polygraph.ue_metrics.spearmanr module
    • SpearmanRankCorrelation
  • lm_polygraph.ue_metrics.ue_metric module
    • UEMetric
    • get_random_scores()
    • normalize()
    • normalize_metric()
    • skip_target_nans()
  • Module contents
• lm_polygraph.utils package
  • Subpackages
    • lm_polygraph.utils.ensemble_utils package
      • Submodules
      • lm_polygraph.utils.ensemble_utils.dropout module
      • lm_polygraph.utils.ensemble_utils.ensemble_beam module
      • lm_polygraph.utils.ensemble_utils.ensemble_generator module
      • lm_polygraph.utils.ensemble_utils.ensemble_greedy module
      • lm_polygraph.utils.ensemble_utils.ensemble_sample module
      • Module contents
    • lm_polygraph.utils.prompt_templates package
      • Submodules
      • lm_polygraph.utils.prompt_templates.llama module
      • lm_polygraph.utils.prompt_templates.vicuna module
      • Module contents
  • Submodules
  • lm_polygraph.utils.cir_model module
    • CenteredIsotonicRegression
      • CenteredIsotonicRegression.fit()
      • CenteredIsotonicRegression.set_fit_request()
      • CenteredIsotonicRegression.set_predict_request()
      • CenteredIsotonicRegression.set_score_request()
      • CenteredIsotonicRegression.set_transform_request()
  • lm_polygraph.utils.common module
    • load_external_module()
    • polygraph_module_init()
    • seq_man_key()
  • lm_polygraph.utils.dataset module
    • Dataset
      • Dataset.from_csv()
      • Dataset.from_datasets()
      • Dataset.load()
      • Dataset.load_hf_dataset()
      • Dataset.select()
      • Dataset.subsample()
      • Dataset.train_test_split()
  • lm_polygraph.utils.deberta module
    • Deberta
      • Deberta.deberta
      • Deberta.deberta_tokenizer
      • Deberta.setup()
      • Deberta.to()
    • MultilingualDeberta
      • MultilingualDeberta.setup()
  • lm_polygraph.utils.generation_parameters module
    • GenerationParameters
      • GenerationParameters.allow_newlines
      • GenerationParameters.do_sample
      • GenerationParameters.generate_until
      • GenerationParameters.num_beams
      • GenerationParameters.presence_penalty
      • GenerationParameters.repetition_penalty
      • GenerationParameters.temperature
      • GenerationParameters.top_k
      • GenerationParameters.top_p
  • lm_polygraph.utils.manager module
    • UEManager
      • UEManager.calculate()
      • UEManager.estimate()
      • UEManager.load()
      • UEManager.save()
    • UncertaintyOutput
      • UncertaintyOutput.estimator
      • UncertaintyOutput.generation_text
      • UncertaintyOutput.generation_tokens
      • UncertaintyOutput.input_text
      • UncertaintyOutput.model_path
      • UncertaintyOutput.uncertainty
    • estimate_uncertainty()
  • lm_polygraph.utils.model module
    • BlackboxModel
      • BlackboxModel.from_huggingface()
      • BlackboxModel.from_openai()
      • BlackboxModel.generate()
      • BlackboxModel.generate_texts()
      • BlackboxModel.tokenizer()
    • Model
      • Model.generate()
      • Model.generate_texts()
    • WhiteboxModel
      • WhiteboxModel.device()
      • WhiteboxModel.from_pretrained()
      • WhiteboxModel.generate()
      • WhiteboxModel.generate_texts()
      • WhiteboxModel.get_stopping_criteria()
      • WhiteboxModel.tokenize()
    • create_ensemble()
  • lm_polygraph.utils.normalize module
    • filter_nans()
    • get_mans_ues_metrics()
  • lm_polygraph.utils.ood_detection module
    • calculate_ood_from_mans()
  • lm_polygraph.utils.openai_chat module
    • OpenAIChat
      • OpenAIChat.ask()
  • lm_polygraph.utils.processor module
    • Logger
      • Logger.on_batch()
      • Logger.on_eval()
    • Processor
      • Processor.on_batch()
      • Processor.on_eval()
  • lm_polygraph.utils.register_stat_calculators module
    • register_stat_calculators()
  • lm_polygraph.utils.token_restoration module
    • collect_sample_token_level_uncertainties()
    • collect_token_level_uncertainties()
    • get_collect_fn()
    • update_token_level_scores()
  • Module contents

Module contents