Calibrate LLM/agent labels for valid downstream inference

A calibration helper: combine plentiful low-cost LLM (or agent) labels with a small human-labeled GOLD sample to produce a design-based / prediction-powered estimate of a mean, proportion, or OLS coefficient that is valid even when the model agrees with humans only 80-90% of the time. Plugging predicted labels straight into an estimator is biased; this function estimates the model's bias on the labeled units, subtracts it off (the rectifier), and propagates the extra uncertainty into the standard error.

Usage

agent_calibrate(
  predictions,
  gold,
  ...,
  method = c("dsl", "ppi", "naive"),
  estimand = c("mean", "proportion", "ols"),
  formula = NULL,
  data = NULL,
  label = NULL,
  id = NULL,
  level = 0.95,
  attach_to = NULL
)

Arguments

predictions: The model's predictions on all units. The common path is a bare numeric / logical / character vector. Also accepted: a tibble (then label names the column) or an Agent / agent run (then the call-level response_text is used).
gold: The held-out human labels on the labeled subset, with the model's predictions on exactly those units. Two forms: (a) a list or tibble with $gold (the true labels) and $pred_on_gold (the predictions on the same units); or (b) a named vector or 2-column (id, value) tibble together with id, the id vector aligned to all-units predictions. For estimand = "ols", gold is the gold outcome on the labeled rows (a vector), and the labeled rows are identified by id (a logical/integer index into data).
...: Unused; reserved.
method: One of "dsl", "ppi", or "naive". For the mean/proportion, "dsl" and "ppi" coincide under random sampling.
estimand: One of "mean", "proportion", or "ols". "mean" and "proportion" share the rectified-mean machinery (a proportion is the mean of a 0/1 label).
formula: For estimand = "ols", the model formula. Write it with the response on the left (e.g. y ~ x1 + x2); the response column does not need to be in data (the gold outcome on the labeled rows is supplied via gold, and the predicted outcome on all units via predictions). Only the covariates must exist in data for every unit. A one-sided formula (~ x1 + x2) is also accepted.
data: For estimand = "ols", a data frame of the covariates for all units. The outcome columns are not read from data: the predicted outcome comes from predictions and the gold outcome on the labeled rows from gold.
label: When predictions is a tibble, the name of the prediction column.
id: For the mean/proportion id-aligned form, the id vector for all units. For OLS, a logical or integer index marking the labeled rows of data (aligned to gold).
level: Confidence level for the interval (default 0.95).
attach_to: Optionally an agent run; when supplied, the returned object carries an attached_to_run_id attribute. Attaching to the run itself is a separate, explicit step via attach_calibration().

Value

An object of class agent_calibration: a list with estimate (a tibble of term, estimate, std_error, conf_low, conf_high, method, estimand), naive (the same shape for the plug-in), agreement (accuracy and Krippendorff alpha on the labeled set), n_labeled, n_total, rectifier, calibrated = TRUE, method, estimand, and a manifest_patch to fold into a run's design via attach_calibration().

Estimators

For estimand = "mean" or "proportion":

method = "ppi": the prediction-powered rectified mean of Angelopoulos et al. (2023). With f_all the predictions on all N units and Y, f_lab the gold and the predictions on the n labeled units, the estimate is mean(f_all) + mean(Y - f_lab) with variance var(f_all)/N + var(Y - f_lab)/n.
method = "dsl": design-based supervised learning (Egami et al. 2023). Under a simple random gold sample the point estimate coincides with PPI's rectified mean, so it is implemented as PPI here. Two caveats: the variance reported is PPI's (a superpopulation form that treats the prediction frame as random), not the finite-population design variance, so for a fixed prediction frame it is mildly conservative; and DSL's generalization to non-random or weighted gold samples (inverse-probability weighting, which also corrects the point estimate) is not implemented in this thin local estimator. For weighted or non-random sampling, use a full implementation (e.g. via the LLMRcontent bridge, see as_llmrcontent_validation()).
method = "naive": the biased plug-in mean(f_all) with variance var(f_all)/N. Provided for comparison only; do not report it.

For estimand = "ols":

method = "ppi": the one-step debiased OLS. With f_all the predicted outcome on all units and Y the gold outcome on the labeled subset, fit beta_f (OLS of f_all on the covariates over all units) and the rectifier (X_lab'X_lab)^-1 X_lab'(Y_lab - f_lab) over the labeled units; the estimate is beta_f + rectifier. Standard errors add an HC0 sandwich for beta_f over all units to an HC0 sandwich for the rectifier over the labeled units. This is a deliberately simple inference: it omits the cross-covariance from the labeled rows being a subset of the all-units frame, and uses no finite-sample or leverage correction, so it can under-cover with a small gold sample or high-leverage designs. Treat the OLS intervals as approximate; for careful inference use a full PPI/DSL implementation.
method = "naive": OLS of the predicted outcome on the covariates, sandwich SEs, for comparison.

References

Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., & Zrnic, T. (2023). Prediction-powered inference. Science, 382(6671), 669-674.

Egami, N., Hinck, M., Stewart, B. M., & Wei, H. (2023). Using imperfect surrogates for downstream inference: Design-based supervised learning for social science. NeurIPS 36.

Examples

# The clearest contract: hand over both sides explicitly. `gold` is the human
# labels on the subset; `pred_on_gold` is the model's labels on those same
# units. No id matching to get wrong.
set.seed(110)
truth <- rbinom(2000, 1, 0.4)
pred  <- ifelse(runif(2000) < 0.15, 1 - truth, truth)   # ~85% accurate
lab   <- sample(2000, 200)
cal <- agent_calibrate(
  predictions = pred,
  gold = list(gold = truth[lab], pred_on_gold = pred[lab]),
  method = "ppi", estimand = "proportion")
cal$estimate

# The id-aligned contract: `id` is the id of EVERY unit (so length(id) ==
# length(predictions)); the ids in `gold` are matched into it to find the
# labeled rows. A frequent mistake is to pass only the subset's ids as `id`.
preds <- c(1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0)            # model: all 12 units
gold  <- tibble::tibble(id = 1:6, value = c(1, 0, 1, 0, 1, 0))  # human: first 6
agent_calibrate(preds, gold = gold, id = 1:12,            # id = ALL 12 unit ids
                method = "ppi", estimand = "proportion")$estimate

if (FALSE) { # \dontrun{
# The common path: an agent labels every unit, humans label a subset.
preds <- vapply(texts, function(t) a$chat(t), character(1))
cal <- agent_calibrate(preds, gold = list(gold = human, pred_on_gold = preds[idx]),
                       method = "dsl", estimand = "proportion")
} # }