Compare coding protocols on a development split — tune

Applies candidate protocols to a gold-set split and compares their labels with the gold labels. Each protocol uses its configured replicate count and modal-label rule.

Usage

tune_protocol(protocols, gold, split = "dev", .runner = NULL, ...)

Arguments

protocols: A list of protocol() objects, or one protocol.
gold: A gold_set().
split: Name of the split to evaluate. The default is "dev". The gold set's declared holdout split is refused.
.runner: A function accepting (experiments, ...). It must return the experiment rows with their identifying columns and a nonmissing response_text column. If it returns success, every value must be TRUE. The default runner is LLMR::call_llm_par().
...: Arguments passed to the runner.

Value

A list of class protocol_tuning with the following elements.

table

A tibble with one row per protocol, ordered by decreasing accuracy. The columns have the following meanings.

protocol, the protocol label, made unique when needed.
n, the number of scored units.
accuracy, the proportion of correct labels.
acc_lo and acc_hi, the limits of the exact 95 percent binomial confidence interval.
macro_f1, the mean category F1 score, with undefined category scores counted as zero.
parse_failures, the number of missing modal labels.
tokens, the subtotal of known token totals.
protocol_id, the protocol's position in protocols, which identifies it whatever its label.

per_category

Category-level scores, named by the protocol labels used in table.

split

The evaluated split name.

usage

A tibble in the same protocol order as table, with columns protocol, calls (model calls, replicates included), tokens_unknown (calls that reported no usage), and protocol_id.

Details

The gold set's declared holdout split cannot be used for tuning.

A call's tokens are the runner's total_tokens, or sent_tokens plus rec_tokens when both are reported. tokens sums the calls that reported usage and is NA when none did; counts that are not nonnegative whole numbers are treated as unreported, with a warning.

Failed calls

A run stops with an error when the runner reports a call as unsuccessful (success not TRUE) or returns a missing response_text. The error has class llmrcontent_run_failure and keeps the work done before it in two fields.

completed: A tibble of the runner rows that succeeded: every row of earlier replicate passes and the successful rows of the pass that failed. Its columns are those the runner returned, with replicate.
failed: A tibble with one row per failed call: protocol_id (the position in the list of protocols), unit_id (the position of the text among the coded texts, which for tuning and validation are the rows of gold_split(gold, split)), replicate, and error_message when the runner returns one.

When the units have identifiers (the id column given to code_corpus() or gold_set()), both tables carry them as their first column, under the same name. An identifier column named like one of the tables' own columns, such as unit_id, takes that column's place. The completed rows can be kept with

calls <- tryCatch(
  code_corpus(corpus, p, "text", id = "doc_id"),
  llmrcontent_run_failure = function(e) e$completed)

and e$failed names the calls to run again.

Examples

cb <- codebook("tone", "one sentence",
  list(cb_category("positive", "Approving."),
       cb_category("negative", "Critical.")))
gold_data <- data.frame(
  text  = c(paste("clear benefit", 1:20), paste("serious harm", 1:20)),
  label = rep(c("positive", "negative"), each = 20))
set.seed(110)
g  <- gold_set(gold_data, text = "text", label = "label",
               split = c(dev = 0.5, test = 0.5))
cfg <- LLMR::llm_config("groq", "openai/gpt-oss-20b", temperature = 0)
if (FALSE) { # \dontrun{
tune_protocol(list(protocol(cb, cfg, label = "baseline")), g)
} # }

# Offline, with a keyword coder in place of the model; it codes correctly
# only under the terse prompt. The two prompt variants on one model share
# the default label, so the table shows them as "groq/openai/gpt-oss-20b"
# and "groq/openai/gpt-oss-20b.1", and `protocol_id` selects the winner.
candidates <- list(
  protocol(cb, cfg),
  protocol(cb, cfg, prompt = "{codebook}\n\n{text}\nLabel:"))
keyword_coder <- function(experiments, ...) {
  terse <- grepl("Label:$", vapply(experiments$messages, `[[`, "", "user"))
  experiments$response_text <- ifelse(
    terse & grepl("benefit", experiments$text), "positive", "negative")
  experiments
}
tuning <- tune_protocol(candidates, g, .runner = keyword_coder)
tuning$table[c("protocol", "protocol_id", "accuracy")]
winner <- protocol_lock(candidates[[tuning$table$protocol_id[[1]]]])