Provider batch APIs run large jobs asynchronously at a reduced price
(typically half) in exchange for delayed delivery (minutes up to 24 hours).
llm_batch_submit() packages one request per element of messages and
submits the job; llm_batch_status() polls it; llm_batch_fetch()
retrieves results as a tidy tibble aligned with the inputs.
Arguments
- config
An llm_config for a generative model.
- messages
An unnamed character vector (each element becomes one request's user message); a named character vector like
c(system = ..., user = ...)(a single multi-role request); or a list with one element per request, where each element is any messages form accepted bycall_llm(). Multimodal file parts are not supported in batch jobs.- state_path
Optional file path; when given, the job object is also saved there as RDS (and the path is remembered for convenience).
Details
Supported providers: "openai" and "groq" (Files + Batches protocol),
"anthropic" (Message Batches), and "gemini" (batchGenerateContent
with inline requests; developer API, not Vertex). All request-shaping
features of llm_config() apply: sampling parameters, structured output,
tools, and hooks shape each request exactly as a live call_llm() would.
The returned job object contains no secrets (the config stores an
environment-variable reference, not the key), so it can be saved to disk,
shared, and fetched later or from another machine with the same
environment variables set. Pass state_path to save it automatically.
Batching, chunking, and row packing
"Batch" appears in three distinct senses in LLMR, and they are easy to keep
apart once you note which path each belongs to. (1) The asynchronous provider
Batch API (llm_batch_submit() and friends) defers a whole job for later
delivery at a reduced price; it is the only deferred path here. (2) In the
embedding path, get_batched_embeddings()'s batch_size sets how many texts
go in one synchronous embedding request, bounded by the provider's per-call
limit. (3) In the generative path, the .rows_per_prompt argument here packs
several data rows into one generative prompt and parses them back into rows.
Senses (2) and (3) are synchronous: batch_size and .rows_per_prompt
control how work is grouped into requests, not the per-token rate, so the
synchronous helpers do not themselves apply a provider discount. Pricing is a
separate axis: several providers bill batched embeddings at a reduced rate
through a dedicated async/batch tier (the embedding analogue of (1)), so
embeddings are not categorically full-price; that discount is a property of
the endpoint, not of batch_size. Row packing (3) can still lower total
tokens by amortizing shared prompt overhead across rows, again without
changing the rate.
See also
llm_batch_status(), llm_batch_fetch(), llm_batch_cancel();
and, for the two synchronous senses of "batch", get_batched_embeddings()
(embedding chunking) and llm_mutate() (.rows_per_prompt row packing).
Examples
if (FALSE) { # \dontrun{
cfg <- llm_config("groq", "openai/gpt-oss-20b", temperature = 0)
job <- llm_batch_submit(cfg, c("2+2?", "Capital of Chile?"),
state_path = "my_batch.rds")
llm_batch_status(job)
# ... later, possibly in a new session:
res <- llm_batch_fetch("my_batch.rds")
} # }