LLMRcontent treats model labels as measured variables. A study defines the instrument, develops candidate protocols on designated rows, validates one locked protocol against held-out human labels, and carries the remaining classification error into corpus estimates. Robustness checks show how specified measurement choices affect the estimand, while replication archives connect results to recorded calls.
Coding with held-out validation
The codebook states the construct and its category boundaries.
cb <- codebook(
name = "policy appraisal",
unit = "one statement",
categories = list(
cb_category("supportive", "Presents the proposal as beneficial."),
cb_category("critical", "Presents the proposal as harmful or deficient.")
),
version = "1.0"
)
cb
#> <codebook 'policy appraisal' v1.0 | unit: one statement | 2 categories | 6bd60ff4b3f0>
#> - supportive: Presents the proposal as beneficial.
#> - critical: Presents the proposal as harmful or deficient.The study begins with a finite corpus: every statement whose category
shares it will report, each with a stable identifier. In this simulated
corpus a fifth of the statements are worded ambiguously, and the vector
truth stands in for the judgment of a trained human coder.
It labels the statements drawn for the audit, and at the end it gives
the share the estimates are meant to recover.
set.seed(110)
n <- 120
truth <- sample(c("supportive", "critical"), n, replace = TRUE,
prob = c(0.4, 0.6))
wording <- ifelse(truth == "supportive", "clear public benefit",
"serious public harm")
wording[sample(n, 24)] <- "mixed evidence"
corpus <- data.frame(
id = sprintf("s%03d", seq_len(n)),
text = paste(wording, "in statement", seq_len(n)),
stringsAsFactors = FALSE
)The audited statements are a simple random sample of the corpus, and
human coders label them. Before labels are collected,
gold_size() can compare candidate sample sizes for a target
width of the agreement interval.
audited <- sample(nrow(corpus), 40)
gold_data <- data.frame(
corpus[audited, c("id", "text")],
label = truth[audited],
stringsAsFactors = FALSE
)The gold set divides the audited statements into a development split,
used to choose a protocol, and a test split, which is sealed for
evaluating the chosen one. By default gold_set() stratifies
the split by label, so that both splits have the same class composition.
That suits validation. A split stratified by the human label is close to
a simple random sample of the corpus, but not exactly one: rounding the
allocation within each label gives statements of different labels
slightly different chances of entering the test split. With
stratify = FALSE the test split is a simple random sample
of the audited statements, and therefore of the corpus, which is what
the correction below asserts.
gold <- gold_set(
gold_data,
text = "text",
label = "label",
id = "id",
split = c(dev = 0.5, test = 0.5),
stratify = FALSE
)
gold
#> <gold_set | 40 units | dev=20, test=20 | holdout 'test' SEALED | 0 ledgered evaluation(s)>Execution functions accept a .runner, a function that
receives request rows and returns them with a response_text
column. This offline response function labels by keyword without
contacting a provider. It codes every ambiguous statement as critical,
except under a prompt that ends with “Label:”, where it codes them as
supportive.
offline_coder <- function(requests, ...) {
prompt <- vapply(requests$messages, `[[`, "", "user")
requests$response_text <- ifelse(
grepl("benefit", requests$text, fixed = TRUE), "supportive", "critical"
)
ambiguous <- grepl("mixed evidence", requests$text, fixed = TRUE)
requests$response_text[ambiguous & grepl("Label:$", prompt)] <- "supportive"
requests
}Candidate protocols are compared on the development split. The tuning
table is sorted by accuracy, and its protocol_id column
gives each candidate’s position in the list, which selects the winner
whatever its label. Locking the winner fixes its hash over the codebook,
prompt, model configuration, and parser; the same locked object is used
for validation and for coding the corpus.
cfg <- LLMR::llm_config("groq", "offline-demo", temperature = 0)
candidates <- list(
protocol(cb, cfg, label = "full codebook"),
protocol(cb, cfg, prompt = "{codebook}\n\n{text}\nLabel:", label = "terse")
)
tuning <- tune_protocol(candidates, gold, .runner = offline_coder)
tuning$table[c("protocol", "protocol_id", "accuracy", "acc_lo", "acc_hi")]
#> # A tibble: 2 × 5
#> protocol protocol_id accuracy acc_lo acc_hi
#> <chr> <int> <dbl> <dbl> <dbl>
#> 1 full codebook 1 0.95 0.751 0.999
#> 2 terse 2 0.9 0.683 0.988
locked <- protocol_lock(candidates[[tuning$table$protocol_id[[1]]]])
locked
#> <coding_protocol 'full codebook' | groq/offline-demo | codebook 'policy appraisal' v1.0 | LOCKED 769e2be85904>The locked protocol is evaluated once on the sealed test split, and the evaluation is recorded in the gold-set ledger.
validation <- validate_protocol(
locked,
gold,
.runner = offline_coder
)
validation
#> <protocol_validation | full codebook | split 'test' | n = 20>
#> accuracy 0.800 [0.563, 0.943] | macro-F1 0.780 | parse failures 0
#> token usage unknown for all 20 call(s)
#> test-split evaluations ledgered so far: 1The validation result reports holdout accuracy, its exact interval, category-level performance, and parse failures. These quantities describe agreement with the human labels. They do not establish that the codebook captures the construct.
The same protocol then codes the whole corpus, audited statements included.
coded <- code_corpus(
corpus,
locked,
"text",
id = "id",
.runner = offline_coder
)
coded_data <- tibble::as_tibble(coded)
head(coded_data[c("id", "label")])
#> # A tibble: 6 × 2
#> id label
#> <chr> <chr>
#> 1 s001 supportive
#> 2 s002 critical
#> 3 s003 supportive
#> 4 s004 supportive
#> 5 s005 supportive
#> 6 s006 criticalgold_correct() pairs the human and model labels of the
test-split statements, which it finds in the coded corpus by
id. For each category, the corrected share is the corpus
share of model labels in that category plus the mean, over the audited
pairs, of the difference between the human indicator and the model
indicator for the category. This is the difference estimator of survey
sampling. Its standard error is sqrt((1 - n/N) * s2 / n),
where s2 is the variance of the paired differences,
n the number of pairs, and N the number of
coded statements with a parsed label; the corpus share adds no sampling
error, because the corpus is the population whose shares are estimated.
When model and human labels mostly agree, the differences vary little,
and the corrected share is more precise than the share of human labels
among the audited statements alone.
The estimator and its standard error assume that the audited
statements are a simple random sample of the coded corpus, so the call
requires that assertion: design = "srs" states it, and the
default refuses to compute a correction from an undeclared audit.
corrected <- gold_correct(coded, gold, design = "srs")
corrected
#> <gold_correction | naive vs corrected category prevalences>
#> supportive: naive 0.292 | corrected 0.492 | SE 0.084 | 95% CI [0.327, 0.656]
#> critical: naive 0.708 | corrected 0.508 | SE 0.084 | 95% CI [0.344, 0.673]
#> audit n = 20 | audit accuracy 0.800 | corpus parse failures = 0 | protocol 769e2be85904
#> The audit uses the sealed holdout split ('test'), linked into the corpus by id.
#> Declared design: audited units are a random subsample of the corpus.
mean(truth == "supportive")
#> [1] 0.4The model codes every ambiguous statement as critical, so the naive supportive share falls below the simulated share of 0.40. The interval for the corrected share covers 0.40, but with 20 audited pairs the point estimate is noisy: here it lies above the simulated share by about as much as the naive share lies below it. The development statements that chose the protocol are part of the corpus as well, while the test split is a random sample of the rest of it. Because the protocol was chosen for its accuracy on the development statements, the correction can understate the model’s error among them, by an amount that shrinks with their share of the corpus. Interpretation remains conditional on the codebook and the human labels.
LLMR::report() combines the validation result with the
locked protocol and gold set. The report includes the holdout ledger,
which now records the validation and correction.
methods <- LLMR::report(
validation,
gold = gold,
protocol = locked
)
methods
#> MEASUREMENT. Units (one statement) were coded into 2 categories (supportive, critical) using codebook 'policy appraisal' v1.0 (SHA-256 6bd60ff4b3f0).
#> PROTOCOL. Coding was performed by full codebook under locked protocol 769e2be85904 (temperature and all inference settings are part of the hash).
#> GOLD SET. 40 human-labeled units (dev = 20, test = 20).
#> VALIDATION (split 'test', n = 20). Accuracy 0.800 (95% CI 0.563-0.943); macro-F1 0.780; parse failures 0.
#> TEST-SPLIT LEDGER. The sealed split was evaluated 2 time(s) -- all evaluations are listed below, in order.
#> 2026-10-30T14:43:23+0000 | protocol 769e2be85904 (full codebook) | n = 20 | accuracy 0.8
#> 2026-10-30T14:43:24+0000 | protocol 769e2be85904 (full codebook) | n = 20 | accuracy 0.8Robustness and replication
A robustness audit recomputes an estimand across prompt wording, label order, and temperature. Here the estimand is the supportive share centered at 0.5.
plan <- audit_plan(
data = corpus,
text = "text",
estimator = function(d) {
mean(d$label == "supportive", na.rm = TRUE) - 0.5
},
labels = c("supportive", "critical"),
prompt = paste(
"Classify as one of: {labels}.",
"",
"{text}",
"",
"Label:",
sep = "\n"
)
)
plan <- audit_add_models(plan, list(keyword = cfg))
plan <- audit_add_prompts(
plan,
concise = "Choose one label from {labels} for this statement: {text}"
)
plan <- audit_add_perturbations(
plan,
label_order = "reversed",
temperature = 0.7
)
audit <- audit_run(plan, .runner = offline_coder)
audit_stability(audit)
#> # A tibble: 1 × 9
#> n_cells reference_estimate sign_agreement min median max iqr
#> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 8 -0.00833 1 -0.208 -0.108 -0.00833 0.2
#> # ℹ 2 more variables: n_failed_cells <int>, status <chr>
audit_fragility(audit)
#> <audit_fragility | fragility Inf | status no_flip | 0 flipping cell(s)>audit_stability() describes variation inside the
specified grid, not across all possible measurement choices.
audit_fragility() measures the smallest number of changed
grid dimensions needed to alter the stated conclusion. Here the concise
prompt moves the ambiguous statements from supportive to critical, and
the estimate moves with it; a substantive study should include
defensible prompt and model alternatives.
The offline response function writes no LLMR call log, so one compact JSON record supplies the fields needed to demonstrate archive construction.
log_path <- tempfile(fileext = ".jsonl")
writeLines(paste0(
'{"ts":"2026-07-01T10:00:01+0000","schema_version":"1.0","kind":"call",',
'"provider":"groq","model":"offline-demo","status":200,',
'"request":{"messages":[{"role":"user","content":"Classify statement 1"}],',
'"temperature":0},"usage":{"sent":12,"rec":1},',
'"response_id":"offline-1","text":"supportive"}'
), log_path)
archive <- archive_seal(
archive_build(log_path, name = "offline-study")
)
archive_check(
archive,
results = data.frame(response_id = "offline-1")
)
#> <archive_check | 1 record(s) | integrity: INTACT | records: VALID | seal: VALID | content root: VALID>
#> completeness: 1/1 result rows map to a logged callThe check compares the stored record with its content hash and seal, then matches the result identifier to the logged call. A successful check establishes integrity and linkage; it does not promise the same reply from a changed model or provider.
archive_write() stores the records and manifest in a
directory. After archive_read(),
archive_check() verifies the copy and
LLMR::report() summarizes it.
archive_dir <- tempfile("offline-study-")
archive_write(archive, archive_dir)
copy <- archive_read(archive_dir)
archive_check(copy)
#> <archive_check | 1 record(s) | integrity: INTACT | records: VALID | seal: VALID | content root: VALID>
LLMR::report(copy)
#> REPLICATION ARCHIVE 'offline-study'.
#> SEAL. Root 613759066da610f8c53903a27f4a79275dbf9333b35ced741f0752e6d075d56e over 1 record(s), sealed 2026-10-30T14:43:25+0000.
#> CALLS. 1 total between 2026-07-01T10:00:01+0000 and 2026-07-01T10:00:01+0000; 0 failure(s).
#> TOKENS. 12 sent, 1 received (as reported by providers).
#> VERIFIABILITY HORIZON.
#> offline-demo groq 1 call(s) api-contingent
#> ENVIRONMENT. R 4.6.2; LLMR 0.9.0; log schema 1.0; built 2026-10-30T14:43:25+0000.