Tune candidate protocols on the development split

Runs every protocol over the gold set's split rows (default "dev") and scores each protocol's modal label across its configured replicates against the gold labels: accuracy with an exact binomial CI, macro-F1, and parse failures. This is the tuning loop: iterate freely here; the holdout split waits, sealed, for the one protocol you lock.

Usage

tune_protocol(protocols, gold, split = "dev", .runner = NULL, ...)

Arguments

protocols

A list of protocol() objects (or a single one).

gold

A gold_set().

split

Which split to evaluate on; the gold set's holdout split ("test" unless gold_set() was given another holdout name) is refused here. That is validate_protocol()'s job, and it leaves a ledger entry.

.runner

Offline runner seam: a function (experiments, ...) that receives a data frame with config and messages list-columns and returns those rows with at least response_text. Default LLMR::call_llm_par().

...

Passed to the runner (e.g. tries, progress).

Value

A protocol_tuning object with table (one row per protocol: protocol, n, accuracy, acc_lo, acc_hi, macro_f1, parse_failures, and tokens), per_category (named per-protocol detail), and split.

Examples

if (FALSE) { # \dontrun{
cb <- codebook("tone", "one sentence",
  list(cb_category("positive", "Approving."),
       cb_category("negative", "Critical.")))
gold_data <- data.frame(
  text  = c(paste("clear benefit", 1:10), paste("serious harm", 1:10)),
  label = rep(c("positive", "negative"), each = 10))
g  <- gold_set(gold_data, text = "text", label = "label",
               split = c(dev = 0.5, test = 0.5))
cfg <- LLMR::llm_config("groq", "openai/gpt-oss-20b", temperature = 0)
tune_protocol(list(protocol(cb, cfg, label = "baseline")), g)
} # }