Stability of the estimate across the grid

Usage

audit_stability(audit, reference = 1L)

Arguments

audit: An audit_run() result.
reference: Cell number whose estimate anchors the sign comparison (default 1, the baseline cell).

Value

A one-row tibble: n_cells, reference_estimate, sign_agreement (share of cells whose estimate has the reference's sign), min, median, max, iqr, n_failed_cells (estimator returned NA), and status ("ok", or "no_valid_estimates" when no cell produced an estimate, or "reference_failed" when the reference cell's estimate is NA, in which case the summary columns are NA).

Examples

plan <- audit_plan(data.frame(text = c("a", "b")), "text",
  function(d) mean(d$label == "yes"), c("yes", "no"), "{text}")
plan <- audit_add_models(plan,
  list(demo = LLMR::llm_config("groq", "demo", temperature = 0)))
runner <- function(x, ...) transform(x, response_text = c("yes", "no"))
audit <- audit_run(plan, .runner = runner)
audit_stability(audit)