How small a measurement change flips the conclusion? — audit

The fragility index is the smallest number of grid dimensions (prompt, model, label order, temperature) that must change from the reference cell before the conclusion – as defined by the flip rule – changes. Inf when no cell in the grid flips, which is a statement about this grid, not a proof of robustness.

Usage

audit_fragility(audit, reference = 1L, flip = sign_flip())

Arguments

audit: An audit_run() result.
reference: Reference cell (default 1).
flip: A flip rule (see flip_rules); default sign_flip().

Value

Integer (or Inf), with the flipping cells in a flipping_cells attribute. When the reference cell's estimate is NA the value is Inf with a status = "reference_failed" attribute, since a failed reference has no sign or threshold to flip against.