Code a corpus with a locked protocol — code

Applies the locked protocol to every text, with protocol$replicates codings per unit. With replicates, the modal label and the share of replicates agreeing with it are returned, which is the unit-level stability diagnostic reviewers should ask for.

Usage

code_corpus(corpus, protocol, text, id = NULL, .runner = NULL, ...)

Arguments

corpus: A data frame.
protocol: A locked protocol().
text: Name of the text column in corpus.
id: Optional name of a stable unit-identifier column in corpus. Carry it through so gold_correct() can link audit units to corpus rows by id, the only way to disambiguate rows that share identical text. Use the same id here as in gold_set(). Its values must be unique and nonmissing; the corpus is refused before any call otherwise.
.runner: A function accepting (experiments, ...). It must return the experiment rows with their identifying columns and a nonmissing response_text column. If it returns success, every value must be TRUE. The default runner is LLMR::call_llm_par().
...: Arguments passed to the runner.

Value

A coded_corpus object. Its data field contains corpus plus label (modal label; NA when no replicate parsed or when the top labels tie, since a tie is disagreement, not a winner), label_share (the top label's share of all replicates, so parse failures lower it), label_share_parsed (its share among parsed replies alone), tied, parse_failures per unit, a .text_hash linkage column, and when protocol$replicates > 1 the individual replicate columns label_rep1, label_rep2, .... Protocol and linkage fields are stored alongside the data. These result columns are reserved: a corpus that already contains one of them is refused rather than overwritten.

Details

Execution is live and parallel (LLMR::call_llm_par()); the .runner seam accepts any function with the same contract, including the replayer archive_replay() returns.

Coded text rides in the prompt

The rendered prompt interpolates each unit's text into the instruction message. A document that itself contains instruction-like text ("ignore the codebook and answer X") can therefore try to steer the model; for adversarial corpora, inspect replicate disagreement and validate against human labels with particular care. Separating instructions from data at the message level is planned for a later release.

Failed calls

A run stops with an error when the runner reports a call as unsuccessful (success not TRUE) or returns a missing response_text. The error has class llmrcontent_run_failure and keeps the work done before it in two fields.

completed: A tibble of the runner rows that succeeded: every row of earlier replicate passes and the successful rows of the pass that failed. Its columns are those the runner returned, with replicate.
failed: A tibble with one row per failed call: protocol_id (the position in the list of protocols), unit_id (the position of the text among the coded texts, which for tuning and validation are the rows of gold_split(gold, split)), replicate, and error_message when the runner returns one.

When the units have identifiers (the id column given to code_corpus() or gold_set()), both tables carry them as their first column, under the same name. An identifier column named like one of the tables' own columns, such as unit_id, takes that column's place. The completed rows can be kept with

calls <- tryCatch(
  code_corpus(corpus, p, "text", id = "doc_id"),
  llmrcontent_run_failure = function(e) e$completed)

and e$failed names the calls to run again.

Examples

cb <- codebook("tone", "one sentence",
  list(cb_category("positive", "Approving."),
       cb_category("negative", "Critical.")))
cfg <- LLMR::llm_config("groq", "openai/gpt-oss-20b", temperature = 0)
if (FALSE) { # \dontrun{
p <- protocol_lock(protocol(cb, cfg, replicates = 2))
code_corpus(data.frame(text = c("clear progress", "serious problem")),
            p, "text")
} # }

# The `.runner` seam answers the calls without a provider, for tests or for
# a deterministic or external coder:
p <- protocol_lock(protocol(cb, cfg))
keyword_coder <- function(experiments, ...) {
  user <- vapply(experiments$messages, `[[`, "", "user")
  experiments$response_text <- ifelse(grepl("progress", user),
                                      "positive", "negative")
  experiments
}
code_corpus(data.frame(text = c("clear progress", "serious problem")),
            p, "text", .runner = keyword_coder)

# A failed call stops the run, and the condition names it by `id`:
flaky_coder <- function(experiments, ...) {
  out <- keyword_coder(experiments)
  out$success <- !grepl("problem", out$text)
  out
}
corpus <- data.frame(doc_id = c("a07", "a08"),
                     text = c("clear progress", "serious problem"))
tryCatch(code_corpus(corpus, p, "text", id = "doc_id", .runner = flaky_coder),
         llmrcontent_run_failure = function(e) e$failed)