Probabilities a local model gives to a set of options

Scores each option as a continuation of the same context with llm_score() and returns its total log-probability and its probability renormalized over the options. Longer options have lower total probabilities by construction, so options of similar length, such as "Yes" and "No", compare most directly; n_tokens is reported for that reason.

Usage

llm_choice_probs(
  config,
  options,
  messages = NULL,
  prompt = NULL,
  prefill = NULL,
  thinking = NULL
)

Arguments

config

An llm_config() for provider "llamacpp".

options

Character vector of options, each scored as a continuation. Write each as the model would produce it: after a prompt that ends mid-sentence, a word usually begins with a space (" Paris"); at the start of a chat reply it does not ("Yes").

messages

Chat messages, as in call_llm(); rendered with the model's chat template. Supply this or prompt.

prompt

Raw text, used as written (for base models, or to control the context exactly).

prefill

Text appended to the context; with messages, the start of the assistant's turn (see llm_render_template()).

thinking

See llm_render_template().

Value

A tibble with option, logprob, n_tokens, and prob.

See also

llm_score()

Examples

if (llm_local_ready("smollm2-135m")) {
  cfg <- llm_config("llamacpp", "smollm2-135m")
  llm_local_start("smollm2-135m", threads = 2)
  llm_choice_probs(cfg, c(" Paris", " Lyon"), prompt = "The capital of France is")
  llm_local_stop()
}