Declare a measurement-robustness audit

Usage

audit_plan(data, text, estimator, labels, prompt)

Arguments

data: The data frame whose texts are measured.
text: Name of the text column.
estimator: The estimand as code: a function that receives data with one added character column label (the LLM measurement under one grid cell) and returns a single numeric value – a share, a regression coefficient, a difference in means. Whatever number the paper reports, this function computes it. Two contract points: label may contain NA (parse failures), and your estimator must decide what they mean – exclude, impute, or treat as a category – explicitly; and if the estimator errors in a cell, audit_run() records NA for that cell and counts it, rather than aborting the grid.
labels: Valid labels, in their baseline order. The prompt may reference them via the {labels} placeholder, which is what the label-order perturbation reorders.
prompt: Baseline prompt template; must contain {text}, may contain {labels}. Placeholders are substituted literally, so braces in the measured text are safe.

Value

An audit_plan with a single prompt variant ("base"), no models, label order "as_given", and temperature 0. Add the grid with audit_add_models(), audit_add_prompts(), audit_add_perturbations().

Examples

plan <- audit_plan(
  data = data.frame(text = c("cut taxes now", "fund the schools")),
  text = "text",
  estimator = function(d) mean(d$label == "conservative"),
  labels = c("conservative", "progressive"),
  prompt = "Classify the slogan as one of: {labels}.\n\n{text}\n\nLabel:"
)
plan