Runs a procedure across the cross-product of perturbation axes and reports,
by axis, how much the result moves. The procedure is your run_fn; the
perturbations reach it through an optional third argument, perturb, so you
write the procedure once and the battery varies its inputs.
Usage
agent_robustness(
run_fn,
design = NULL,
reps = 1L,
vary = list(),
measure = NULL,
baseline = "first",
config = NULL,
parallel = FALSE,
quiet = TRUE,
...
)Arguments
- run_fn
The procedure,
function(cond, rep[, perturb]), returning a result whose stability is assessed viameasure.- design
Optional baseline conditions (a data frame); one empty condition if
NULL.- reps
Replications per cell.
- vary
A named list of axes: bare level vectors or axis specs from
vary_models()and related helpers.- measure
A function
result -> scalar(or a field name) producing the quantity whose stability is assessed. IfNULL, the result is used when it is already scalar.- baseline
Which level of each axis is the reference (
"first").- config
A base config (used to build
perturb$config).- parallel
Passed to
agent_experiment().- quiet
Passed to
agent_experiment().- ...
Passed to
agent_experiment().
Value
An object of class agent_robustness: a list with cells (the full
perturbed design with measure_value and measure_error), by_axis (one
row per axis-level with instability, dispersion, agreement_alpha,
failure_rate, flips_vs_baseline, delta_mean), and overall (a
fragile flag). A cell counts as failed when its run raised an error or
when measure raised an error or returned anything other than one non-NA
value; failed cells are counted in failure_rate and excluded from the
other statistics. Agreement pairs replicate r at one level with replicate r
at another, so every replicate is used. When no cell could be measured,
dispersion, max_instability, and fragile are NA.
Details
run_fn may be function(cond, rep) (the existing agent_experiment()
contract) or function(cond, rep, perturb). perturb is a list with
config (the base config with this cell's model/temperature applied),
persona(x) (apply this cell's persona variant to a base persona), prompt(x)
(apply this cell's prompt variant), and reorder(options) (apply this cell's
option permutation). A two-argument run_fn still runs; then only the
model/temperature axes affect the run.
Examples
if (FALSE) { # \dontrun{
batt <- agent_robustness(
run_fn = function(cond, rep, perturb) {
a <- agent("S", perturb$config, persona = perturb$persona("A cautious voter."))
a$reply(perturb$prompt("Do you support the policy? yes/no."))
},
vary = list(temperature = c(0, 1), model = c("openai/gpt-oss-20b")),
measure = function(r) tolower(trimws(r))
)
batt$by_axis
} # }