lm_polygraph.utils.processor module

class lm_polygraph.utils.processor.Logger

Bases: Processor

Processor logging batch information to stdout.

on_batch(batch_stats: Dict[str, ndarray], batch_gen_metrics: Dict[Tuple[str, str], List[float]], batch_estimations: Dict[Tuple[str, str], List[float]])

Outputs statistics from batch_stats, batch_gen_metrics and batch_estimations to stdout.

on_eval(metrics: Dict[Tuple[str, str, str, str], float], bad_estimators: Dict[Estimator, int])

Outputs statistics from metrics and failed estimators to stdout.

class lm_polygraph.utils.processor.Processor

Bases: object

Abstract class to perform actions after processing new texts batch.

on_batch(batch_stats: Dict[str, ndarray], batch_gen_metrics: Dict[Tuple[str, str], List[float]], batch_estimations: Dict[Tuple[str, str], List[float]])

Processes new batch.

Parameters:

batch_stats (Dict[str, np.ndarray]): Dictionary of statistics calculated with stat_calculators. batch_gen_metrics (Dict[Tuple[str, str], List[float]]): Dictionary of generation metrics calculated

for the batch. Dictionary keys consist of UE level (sequence or token) and generation metrics name.

batch_estimations (Dict[Tuple[str, str], List[float]]): Dictionary of UE estimations calculated

for the batch. Dictionary keys consist of UE level (sequence or token) and UE estimator name.

on_eval(metrics: Dict[Tuple[str, str, str, str], float])

Processes newly calculated evaluation metrics.

Parameters:

metrics (Dict[Tuple[str, str, str, str], float]: metrics calculated using ue_metrics on the batch which

was considered at the last on_batch call. Dictionary keys consist of UE level, estimator name, generation metrics name and ue_metrics name which was used to calculate quality metrics between this estimator’s uncertainty estimations and generation metric outputs.