The fragility index is the smallest number of grid dimensions (prompt,
model, label order, temperature) that must change from the reference
cell before the conclusion – as defined by the flip rule – changes.
Inf when no cell in the grid flips, which is a statement about this
grid, not a proof of robustness.
Usage
audit_fragility(audit, reference = 1L, flip = sign_flip())Arguments
- audit
An
audit_run()result.- reference
Reference cell (default 1).
- flip
A flip rule (see flip_rules); default
sign_flip().
Value
An audit_fragility object with fragility (an integer or Inf),
flipping_cells (the nearest flipping cell IDs), status ("ok",
"no_flip", or "reference_failed"), reference, and
reference_estimate. A failed reference has no conclusion to flip, so
its fragility is Inf and its flipping cells are an empty integer vector.
Examples
plan <- audit_plan(data.frame(text = c("a", "b")), "text",
function(d) mean(d$label == "yes") - 0.25, c("yes", "no"), "{text}")
plan <- audit_add_models(plan,
list(demo = LLMR::llm_config("groq", "demo", temperature = 0)))
runner <- function(x, ...) transform(x, response_text = c("yes", "no"))
audit <- audit_run(plan, .runner = runner)
audit_fragility(audit)