Validate a coding protocol against gold labels — validate

Applies one protocol to a gold-set split and compares its modal labels with the gold labels.

Usage

validate_protocol(protocol, gold, split = NULL, .runner = NULL, ...)

Arguments

protocol: A protocol(). It must be locked for evaluation on the gold set's declared holdout split.
gold: A gold_set().
split: Name of the split to evaluate. NULL selects the gold set's declared holdout split.
.runner: A function accepting (experiments, ...). It must return the experiment rows with their identifying columns and a nonmissing response_text column. If it returns success, every value must be TRUE. The default runner is LLMR::call_llm_par().
...: Arguments passed to the runner.

Value

A list of class protocol_validation with the following elements.

protocol, the protocol label.
protocol_hash, the stored protocol hash.
split, the evaluated split name.
holdout, the declared holdout split name.
n, the number of scored units.
accuracy, the proportion of correct labels.
acc_lo and acc_hi, the exact 95 percent binomial confidence limits.
macro_f1, the mean category F1 score, with undefined category scores counted as zero.
parse_failures, the number of missing modal labels.
tokens, the subtotal of known token totals.
calls, the number of model calls, replicates included.
tokens_unknown, the number of calls that reported no usage.
per_category, the category-level score table.
confusion, the table of gold and predicted labels.
ledger_entries, the number of ledger entries after the call, including this evaluation when it was recorded.

Details

Evaluation on the declared holdout split requires a locked protocol. A locked protocol must still match its stored hash. Evaluation on another split may use an unlocked protocol.

An evaluation on the holdout split is appended to the gold set's ledger when that split is sealed.

Tokens are counted as in tune_protocol(): tokens sums the calls that reported usage and is NA when none did.

Failed calls

A run stops with an error when the runner reports a call as unsuccessful (success not TRUE) or returns a missing response_text. The error has class llmrcontent_run_failure and keeps the work done before it in two fields.

completed: A tibble of the runner rows that succeeded: every row of earlier replicate passes and the successful rows of the pass that failed. Its columns are those the runner returned, with replicate.
failed: A tibble with one row per failed call: protocol_id (the position in the list of protocols), unit_id (the position of the text among the coded texts, which for tuning and validation are the rows of gold_split(gold, split)), replicate, and error_message when the runner returns one.

When the units have identifiers (the id column given to code_corpus() or gold_set()), both tables carry them as their first column, under the same name. An identifier column named like one of the tables' own columns, such as unit_id, takes that column's place. The completed rows can be kept with

calls <- tryCatch(
  code_corpus(corpus, p, "text", id = "doc_id"),
  llmrcontent_run_failure = function(e) e$completed)

and e$failed names the calls to run again.

Examples

if (FALSE) { # \dontrun{
cb <- codebook("tone", "one sentence",
  list(cb_category("positive", "Approving."),
       cb_category("negative", "Critical.")))
gold_data <- data.frame(
  text  = c(paste("clear benefit", 1:10), paste("serious harm", 1:10)),
  label = rep(c("positive", "negative"), each = 10))
set.seed(110)
g <- gold_set(gold_data, text = "text", label = "label",
              split = c(test = 1))
p <- protocol_lock(protocol(cb, LLMR::llm_config("groq", "openai/gpt-oss-20b")))
validate_protocol(p, g)
gold_ledger(g)   # the evaluation is on the record
} # }