Start, stop, and inspect local model servers

Each installed local model runs in its own llama.cpp server process on this computer. A server starts by itself on the first call that needs it, so llm_local_start() is needed only to choose settings other than the defaults, such as the context length, the number of CPU threads, or CPU instead of GPU. A server listens only on 127.0.0.1, accepts only requests that carry a key generated when it starts (only its /health status is open), and stops when R exits.

Usage

llm_local_start(
  model,
  ctx = NULL,
  device = c("auto", "cpu", "gpu"),
  threads = NULL,
  parallel = NULL,
  timeout = 120,
  server = NULL,
  args = character(),
  dir = llm_local_dir()
)

llm_local_stop(model = NULL)

llm_local_status(dir = llm_local_dir())

Arguments

model

Name of an installed model; see llm_local_models().

ctx

Context length in tokens; by default the model's registry value (4096 for most chat models, 8192 for embedding models).

device

"auto" (the GPU when the installed build has one, otherwise the CPU), "cpu", or "gpu".

threads

Number of CPU threads; by default the llmr.local_threads option, or else llama.cpp's own choice.

parallel

Number of requests served at once. Chat models use 1 so that requests are handled in order; embedding models default to 4.

timeout

Seconds to wait for the server to load the model.

server

Path to a llama-server program to use instead of the installed one, for systems the official builds do not support.

args

Further command-line arguments for llama-server.

dir

The local model directory; see llm_local_dir().

Value

llm_local_start() returns the server's address, invisibly. llm_local_stop() returns the names of the stopped servers, invisibly. llm_local_status() returns an llmr_local_status list with the installed server build, the installed models, and the running servers.

Details

Results depend on where the model runs. On the same model, log-probabilities computed on the GPU and on the CPU differ in the first decimal place (by up to about 0.2 in tests with Qwen3 0.6B), and on the CPU they also depend on settings such as flash attention, which LLMR turns off. Repeated requests on the same device and build give identical values. LLMR records the device with every result of the token-level functions; numbers meant to be compared should come from the same device and, for the CPU, the same type of machine.

See also

llm_local_install()

Examples

llm_local_status()
if (llm_local_ready("smollm2-135m")) {
  llm_local_start("smollm2-135m", threads = 2)
  llm_local_stop("smollm2-135m")
}