Run an audit grid — audit_run • LLMRcontent

Codes every data unit under each combination in an audit plan and evaluates the plan's estimator for each grid cell.

Usage

audit_run(plan, .runner = NULL, ...)

Arguments

plan: An audit_plan() containing at least one model.
.runner: A function accepting (experiments, ...). It must return the experiment rows with their identifying columns and a nonmissing response_text column. If it returns success, every value must be TRUE. The default runner is LLMR::call_llm_par().
...: Arguments passed to the runner.

Value

A list of class audit with the following elements.

cells: A tibble with one row per grid cell.
units: The unit-level label and response-ID table returned by audit_units().
plan: The supplied audit plan.

The cells columns have the following meanings.

cell, the grid-cell number.
prompt, the prompt variant name.
model, the model name in the plan.
label_order, the label-order setting.
temperature, the temperature setting.
estimate, the estimator's value, or NA.
estimate_status, the outcome of evaluating the estimator.
estimate_note, a description of an unsuccessful evaluation, or NA for a usable estimate.
n_labels, the number of parsed labels the estimator received.
parse_failures, the number of unparsed labels.
tokens, the subtotal of known token totals.
tokens_unknown, the number of calls that reported no usage.

A finite numeric or logical scalar gives status "ok"; logical values are converted to numbers. A scalar NA or NaN gives "missing". An estimator error gives "error", and another return value gives "invalid". Errors and invalid returns also produce a warning.

Details

The grid crosses prompt variants with models, label orders, and temperatures. Cell 1 uses the first value of each dimension and is the default reference for audit_fragility().

Tokens are counted as in tune_protocol(): each cell's tokens sums the calls that reported usage and is NA when none did.

Failed calls

The grid is sent to the runner as one table. When the runner reports a call as unsuccessful (success not TRUE) or returns a missing response_text, no estimate is computed and the function stops with an error of class llmrcontent_run_failure. Its completed field is a tibble of the runner rows that succeeded, and its failed field gives the cell, the unit_id (the row of the plan's data), and the runner's error_message, when there is one, of each failed call. The completed rows can be kept with

calls <- tryCatch(
  audit_run(plan),
  llmrcontent_run_failure = function(e) e$completed)

Examples

if (FALSE) { # \dontrun{
speeches <- data.frame(
  text = c("cut taxes now", "deregulate markets",
           "fund the schools", "expand care"))
plan <- audit_plan(
  data = speeches, text = "text",
  estimator = function(d) mean(d$label == "conservative", na.rm = TRUE),
  labels = c("conservative", "progressive"),
  prompt = "Classify as one of: {labels}.\n\n{text}\n\nLabel:")
plan <- audit_add_models(plan,
  list(oss = LLMR::llm_config("groq", "openai/gpt-oss-20b", temperature = 0)))
audit <- audit_run(plan)
audit
audit_stability(audit)
audit_fragility(audit)
head(audit_units(audit))
} # }

# The `.runner` seam answers the grid without a provider, for tests or for
# a deterministic or external coder. The same plan, scored offline:
speeches <- data.frame(
  text = c("cut taxes now", "deregulate markets",
           "fund the schools", "expand care"))
plan <- audit_plan(
  data = speeches, text = "text",
  estimator = function(d) mean(d$label == "conservative", na.rm = TRUE),
  labels = c("conservative", "progressive"),
  prompt = "Classify as one of: {labels}.\n\n{text}\n\nLabel:")
plan <- audit_add_models(plan,
  list(oss = LLMR::llm_config("groq", "openai/gpt-oss-20b", temperature = 0)))
keyword_coder <- function(experiments, ...) {
  msg <- vapply(experiments$messages, `[[`, "", "user")
  experiments$response_text <- ifelse(grepl("taxes|deregulate", msg),
                                      "conservative", "progressive")
  experiments
}
audit <- audit_run(plan, .runner = keyword_coder)
audit_stability(audit)