lm_polygraph.utils.estimate_uncertainty module

class lm_polygraph.utils.estimate_uncertainty.UncertaintyOutput(uncertainty: float | List[float], input_text: str, generation_text: str, generation_tokens: List[int], model_path: str, estimator: str)

Bases: object

Uncertainty estimator output.

Parameters:

uncertainty (float): uncertainty estimation. input_text (str): text used as model input. generation_text (str): text generated by the model. model_path (str): path to the model used in generation.

estimator: str

generation_text: str

generation_tokens: List[int]

input_text: str

model_path: str

uncertainty: float | List[float]

lm_polygraph.utils.estimate_uncertainty.estimate_uncertainty(model: Model, estimator: Estimator, input_text: str, input_image: str | Path | Image | None = None) → UncertaintyOutput

Estimated uncertainty of the model generation using the provided esitmator.

Parameters:

model (Model): model to estimate uncertainty of. Either lm_polygraph.WhiteboxModel or

lm_polygraph.BlackboxModel model can be used.

estimator (Estimator): uncertainty estimation method to use. Can be any of the methods at

lm_polygraph.estimators.

input_text (str): text to estimate uncertainty of.

Returns:

UncertaintyOutput: uncertainty estimation float along with supporting info.

Examples:

`python >>> from lm_polygraph import WhiteboxModel >>> from lm_polygraph.estimators import LexicalSimilarity >>> model = WhiteboxModel.from_pretrained( ...     'bigscience/bloomz-560m', ...     device='cpu', ... ) >>> estimator = LexicalSimilarity('rougeL') >>> estimate_uncertainty(model, estimator, input_text='Who is George Bush?') UncertaintyOutput(uncertainty=-0.9176470588235295, input_text='Who is George Bush?', generation_text=' President of the United States', model_path='bigscience/bloomz-560m') `

`python >>> from lm_polygraph import BlackboxModel >>> from lm_polygraph.estimators import EigValLaplacian >>> model = BlackboxModel.from_openai( ...     'YOUR_OPENAI_TOKEN', ...     'gpt-3.5-turbo' ... ) >>> estimator = EigValLaplacian() >>> estimate_uncertainty(model, estimator, input_text='When did Albert Einstein die?') UncertaintyOutput(uncertainty=1.0022274826855433, input_text='When did Albert Einstein die?', generation_text='Albert Einstein died on April 18, 1955.', model_path='gpt-3.5-turbo') `