Run a robustness battery

Runs a procedure across the cross-product of perturbation axes and reports, by axis, how much the result moves. The procedure is your run_fn; the perturbations reach it through an optional third argument, perturb, so you write the procedure once and the battery varies its inputs.

Usage

agent_robustness(
  run_fn,
  design = NULL,
  reps = 1L,
  vary = list(),
  measure = NULL,
  baseline = "first",
  .config = NULL,
  parallel = FALSE,
  quiet = TRUE,
  ...
)

Arguments

run_fn

The procedure, function(cond, rep[, perturb]), returning a result whose stability is assessed via measure.

design

Optional baseline conditions (a data frame); one empty condition if NULL.

reps

Replications per cell.

vary

A named list of axes: bare level vectors or axis specs from vary_models() and related helpers.

measure

A function result -> scalar (or a field name) producing the quantity whose stability is assessed. If NULL, the result is used when it is already scalar.

baseline

Which level of each axis is the reference ("first").

.config

A base config (used to build perturb$config).

parallel

Passed to agent_experiment().

quiet

Passed to agent_experiment().

...

Passed to agent_experiment().

Value

An object of class agent_robustness: a list with cells (the full perturbed design with measure_value), by_axis (one row per axis-level with instability, dispersion, agreement_alpha, failure_rate, flips_vs_baseline, delta_mean), and overall (a fragile flag).

Details

run_fn may be function(cond, rep) (the existing agent_experiment() contract) or function(cond, rep, perturb). perturb is a list with config (the base config with this cell's model/temperature applied), persona(x) (apply this cell's persona variant to a base persona), prompt(x) (apply this cell's prompt variant), and reorder(options) (apply this cell's option permutation). A two-argument run_fn still runs; then only the model/temperature axes affect the run.

See also

agent_experiment(), LLMR::llm_replicate(), LLMR::llm_agreement()

Examples

if (FALSE) { # \dontrun{
batt <- agent_robustness(
  run_fn = function(cond, rep, perturb) {
    a <- agent("S", perturb$config, persona = perturb$persona("A cautious voter."))
    a$reply(perturb$prompt("Do you support the policy? yes/no."))
  },
  vary = list(temperature = c(0, 1), model = c("openai/gpt-oss-20b")),
  measure = function(r) tolower(trimws(r))
)
batt$by_axis
} # }