lm_polygraph.model_adapters.whitebox_model_vllm module

class lm_polygraph.model_adapters.whitebox_model_vllm.WhiteboxModelvLLM(model, sampling_params, generation_parameters: GenerationParameters = GenerationParameters(temperature=1.0, top_k=50, top_p=1.0, do_sample=False, num_beams=1, presence_penalty=0.0, repetition_penalty=1.0, stop_strings=None, allow_newlines=True, max_new_tokens=100), device: str = 'cuda', instruct: bool = False)

Bases: Model

Basic whitebox model adapter for using vLLM in stat calculators and uncertainty estimators.

device()

generate(*args, **kwargs)

Abstract method. Generates the model output with scores from batch formed by HF Tokenizer. Not implemented for black-box models.

generate_texts(input_texts: List[str], **args)

Abstract method. Generates a list of model answers using input texts batch.

Parameters:

input_texts (List[str]): input texts batch.

Return:

List[str]: corresponding model generations. Have the same length as input_texts.

post_processing(outputs)

tokenize(texts)