Score a text a local model did not write

Computes the log-probability of each token of continuation given the context before it, whether or not the model would have produced that text. The continuation is tokenized on its own and appended to the tokenized context; each token is scored by one request whose prompt is the context and the tokens before it, read from the model's distribution at that point (forcing the token does not change its value).

Usage

llm_score(
  config,
  continuation,
  messages = NULL,
  prompt = NULL,
  prefill = NULL,
  top_n = 5,
  thinking = NULL,
  cache = FALSE
)

Arguments

config: An llm_config() for provider "llamacpp".
continuation: The text to score, one string.
messages: Chat messages, as in call_llm(); rendered with the model's chat template. Supply this or prompt.
prompt: Raw text, used as written (for base models, or to control the context exactly).
prefill: Text appended to the context; with messages, the start of the assistant's turn (see llm_render_template()).
top_n: Number of most probable alternatives to record at each position.
thinking: See llm_render_template().
cache: If TRUE, let the server reuse the shared prefix between requests (see Details).

Value

A tibble with one row per token of continuation: pos, id, token, bytes, logprob, prob, rank (the token's rank among the alternatives, NA when below top_n), and top (a list of tibbles of the alternatives). Attributes total_logprob and mean_logprob give the sum and mean, and llmr_local the provenance and context.

Details

The values come from processing each prefix afresh. The log-probabilities llama.cpp reports while generating the same tokens come from processing them one at a time, and the two differ by floating-point effects: by about 1e-3 on Apple GPUs and by up to about 0.1 on the CPU. cache = TRUE reuses the previous request's work, which reproduces the generation values and is faster.

Examples

if (llm_local_ready("smollm2-135m")) {
  cfg <- llm_config("llamacpp", "smollm2-135m")
  llm_local_start("smollm2-135m", threads = 2)
  s <- llm_score(cfg, " Lyon.", prompt = "The capital of France is")
  attr(s, "total_logprob")
  llm_local_stop()
}