Continue a reply, or any text, with a local model

Generates continuations of a context whose end the researcher writes: the chat template applied to messages with the assistant's turn starting with prefill, or the raw prompt followed by prefill. This shows how the model goes on from words put in its mouth, and with n > 1 gives a sample of continuations from the same point.

Usage

llm_continue(
  config,
  messages = NULL,
  prompt = NULL,
  prefill = "",
  n = 1,
  max_tokens = 256,
  seed = NULL,
  temperature = NULL,
  top_p = NULL,
  top_k = NULL,
  min_p = NULL,
  thinking = NULL,
  top_n = 0,
  ...
)

Arguments

config: An llm_config() for provider "llamacpp".
messages: Chat messages, as in call_llm(); rendered with the model's chat template. Supply this or prompt.
prompt: Raw text, used as written (for base models, or to control the context exactly).
prefill: Text appended to the context; with messages, the start of the assistant's turn (see llm_render_template()).
n: Number of continuations.
max_tokens: Largest number of tokens to generate for each.
seed: NULL for unseeded sampling, or a whole number from 0 to 4294967294: continuation i then uses seed seed + i - 1, so the sample can be reproduced on the same llama.cpp build and device.
temperature, top_p, top_k, min_p: Sampling settings; top_k limits sampling to the top_k most probable tokens, with 0 meaning no limit.
thinking: See llm_render_template().
top_n: If above 0, also return, for each generated token, its log-probability and the top_n most probable alternatives. llama.cpp omits these for a token that is part of a multi-byte character.
...: Further llama.cpp /completion fields, sent as given (for example stop, grammar, or logit_bias).

Value

A tibble with one row per continuation: sample, seed, prefill, continuation, text (prefill followed by the continuation), finish_reason ("stop" or "length"), n_tokens, and, with top_n > 0, a list column logprobs. The attribute llmr_local records the model, build, device, context, and sampling settings.

Details

Unless the arguments or the configuration set them, the model samples from its full distribution: temperature 1, with no top-k, top-p, or min-p truncation, no repetition penalties, and no other sampler that would alter the distribution. llama.cpp's own defaults (top-k 40, top-p 0.95, min-p 0.05) are not used. Of the configuration's settings, temperature, top_p, top_k, and min_p are used when the arguments are not given; other generation settings in the configuration are not.

Examples

if (llm_local_ready("smollm2-135m")) {
  cfg <- llm_config("llamacpp", "smollm2-135m")
  llm_local_start("smollm2-135m", threads = 2)
  llm_continue(cfg, messages = "Is coffee good for you?",
               prefill = "Honestly, I am not sure, but", n = 2,
               max_tokens = 20, seed = 110)
  llm_local_stop()
}