The next-token distribution of a local model

Returns the model's probability distribution over the next token after a context: the chat template applied to messages, or the raw prompt, followed by prefill. The probabilities are the model's own, before any sampling setting (temperature, top-k, top-p) acts, so they do not depend on the configuration's sampling parameters.

Usage

llm_next_token(
  config,
  messages = NULL,
  prompt = NULL,
  prefill = NULL,
  top_n = 20,
  full = FALSE,
  thinking = NULL
)

Arguments

config: An llm_config() for provider "llamacpp".
messages: Chat messages, as in call_llm(); rendered with the model's chat template. Supply this or prompt.
prompt: Raw text, used as written (for base models, or to control the context exactly).
prefill: Text appended to the context; with messages, the start of the assistant's turn (see llm_render_template()).
top_n: Number of most probable tokens to return.
full: If TRUE, return the whole vocabulary.
thinking: See llm_render_template().

Value

A tibble with rank, id, token, logprob, prob, and bytes (the token's raw bytes, which matter for tokens that are parts of a multi-byte character). The attribute llmr_local records the model, its file's sha256, the llama.cpp build, the device, and the context text.

Details

llama.cpp computes the probabilities in single precision: the whole vocabulary sums to 1 within about 5e-4, and a value can differ by about 3e-4 with the number of tokens requested. The values also depend on the device (see llm_local_start()).

Examples

if (llm_local_ready("smollm2-135m")) {
  cfg <- llm_config("llamacpp", "smollm2-135m")
  llm_local_start("smollm2-135m", threads = 2)
  llm_next_token(cfg, prompt = "The capital of France is", top_n = 5)
  llm_local_stop()
}