LLMRcontent treats model labels as measured variables. A study defines the instrument, develops candidate protocols on designated rows, validates one locked protocol against held-out human labels, and carries the remaining classification error into corpus estimates. Robustness checks show how specified measurement choices affect the estimand, while replication archives connect results to recorded calls.
Coding with held-out validation
The codebook states the construct and its category boundaries.
cb <- codebook(
name = "policy appraisal",
unit = "one statement",
categories = list(
cb_category("supportive", "Presents the proposal as beneficial."),
cb_category("critical", "Presents the proposal as harmful or deficient.")
),
version = "1.0"
)
cb
#> <codebook 'policy appraisal' v1.0 | unit: one statement | 2 categories | 6bd60ff4b3f0>
#> - supportive: Presents the proposal as beneficial.
#> - critical: Presents the proposal as harmful or deficient.The gold set stores its split assignment and seals the test split.
Stable unit identifiers provide the later link between audited rows and
the coded corpus. Before collecting labels, gold_size() can
compare candidate sample sizes for a target agreement-interval width.
The development split supports protocol selection; the test split is
reserved for evaluating the selected protocol.
gold_data <- data.frame(
id = sprintf("g%02d", seq_len(40)),
text = c(
paste("clear public benefit case", seq_len(20)),
paste("serious public harm case", seq_len(20))
),
label = rep(c("supportive", "critical"), each = 20),
stringsAsFactors = FALSE
)
set.seed(110)
gold <- gold_set(
gold_data,
text = "text",
label = "label",
id = "id",
split = c(dev = 0.5, test = 0.5)
)
gold
#> <gold_set | 40 units | dev=20, test=20 | holdout 'test' SEALED | 0 ledgered evaluation(s)>A locked protocol hashes the codebook, prompt, model configuration,
and parser. In a full study, tune_protocol() compares
candidates on the development split before one is locked; this compact
example fixes one protocol so the holdout result and corrected
prevalence remain the focus. The same locked object must be used for
validation and corpus coding.
cfg <- LLMR::llm_config("groq", "offline-demo", temperature = 0)
locked <- protocol_lock(
protocol(cb, cfg, label = "keyword protocol")
)
locked
#> <coding_protocol 'keyword protocol' | groq/offline-demo | codebook 'policy appraisal' v1.0 | LOCKED a353e551db05>Execution functions accept a .runner through an
interface in which a function receives request rows and returns response
rows; this offline response function applies a fixed keyword rule
without contacting a provider.
offline_coder <- function(requests, ...) {
text <- tolower(requests$text)
requests$response_text <- ifelse(
grepl("benefit", text, fixed = TRUE), "supportive", "critical"
)
requests$response_text[grepl("benefit case (3|7|11|15)$", text)] <- "critical"
requests
}The locked protocol is evaluated on the sealed test split. This access is recorded in the gold-set ledger.
validation <- validate_protocol(
locked,
gold,
.runner = offline_coder
)
validation
#> <protocol_validation | keyword protocol | split 'test' | n = 20>
#> accuracy 0.900 [0.750, 1.000] | macro-F1 0.899 | parse failures 0
#> test-split evaluations ledgered so far: 1The validation result reports holdout accuracy, its interval, category-level performance, and parse failures. These quantities describe agreement with the specified human labels. They do not establish that the codebook captures the construct, and repeated holdout use appears in the gold-set ledger.
The corpus includes the audited units and new statements. The same response function and locked protocol produce the model labels.
new_corpus <- data.frame(
id = sprintf("c%02d", seq_len(20)),
text = c(
paste("clear public benefit new", seq_len(8)),
paste("serious public harm new", seq_len(8)),
paste("mixed evidence new", seq_len(4))
),
stringsAsFactors = FALSE
)
corpus <- rbind(gold_data[c("id", "text")], new_corpus)
coded <- code_corpus(
corpus,
locked,
"text",
id = "id",
.runner = offline_coder
)
coded_data <- tibble::as_tibble(coded)
head(coded_data[c("id", "label")])
#> # A tibble: 6 × 2
#> id label
#> <chr> <chr>
#> 1 g01 supportive
#> 2 g02 supportive
#> 3 g03 critical
#> 4 g04 supportive
#> 5 g05 supportive
#> 6 g06 supportivegold_correct() compares the model and human labels in
the holdout sample. It uses those differences to correct corpus category
shares and calculate their standard errors.
corrected <- gold_correct(coded, gold)
tibble::as_tibble(corrected)
#> # A tibble: 2 × 6
#> category share_naive share_corrected se ci_lo ci_hi
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 supportive 0.4 0.5 0.0562 0.390 0.610
#> 2 critical 0.6 0.5 0.0562 0.390 0.610The corrected estimates use the holdout confusion matrix to adjust the corpus shares. Their standard errors carry uncertainty from the finite holdout into the prevalence estimates. Interpretation remains conditional on the codebook, the human labels, and the match between holdout and corpus coding errors.
LLMR::report() combines the validation result with the
locked protocol and gold set. The report includes the holdout ledger,
which now records the validation and correction.
methods <- LLMR::report(
validation,
gold = gold,
protocol = locked
)
methods
#> MEASUREMENT. Units (one statement) were coded into 2 categories (supportive, critical) using codebook 'policy appraisal' v1.0 (SHA-256 6bd60ff4b3f0).
#> PROTOCOL. Coding was performed by keyword protocol under locked protocol a353e551db05 (temperature and all inference settings are part of the hash).
#> GOLD SET. 40 human-labeled units (dev = 20, test = 20).
#> VALIDATION (split 'test', n = 20). Accuracy 0.900 (95% CI 0.750-1.000); macro-F1 0.899; parse failures 0.
#> TEST-SPLIT LEDGER. The sealed split was evaluated 2 time(s) -- all evaluations are listed below, in order.
#> 2026-07-27T18:55:15+0000 | protocol a353e551db05 (keyword protocol) | n = 20 | accuracy 0.9
#> 2026-07-27T18:55:15+0000 | protocol a353e551db05 (keyword protocol) | n = 20 | accuracy 0.9Robustness and replication
A robustness audit recomputes an estimand across prompt wording, label order, and temperature. Here the estimand is the supportive share centered at 0.5.
plan <- audit_plan(
data = new_corpus,
text = "text",
estimator = function(d) {
mean(d$label == "supportive", na.rm = TRUE) - 0.5
},
labels = c("supportive", "critical"),
prompt = paste(
"Classify as one of: {labels}.",
"",
"{text}",
"",
"Label:",
sep = "\n"
)
)
plan <- audit_add_models(plan, list(keyword = cfg))
plan <- audit_add_prompts(
plan,
concise = "Choose one label from {labels} for this statement: {text}"
)
plan <- audit_add_perturbations(
plan,
label_order = "reversed",
temperature = 0.7
)
audit <- audit_run(plan, .runner = offline_coder)
audit_stability(audit)
#> # A tibble: 1 × 9
#> n_cells reference_estimate sign_agreement min median max iqr
#> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 8 -0.1 1 -0.1 -0.1 -0.1 0
#> # ℹ 2 more variables: n_failed_cells <int>, status <chr>
audit_fragility(audit)
#> <audit_fragility | fragility Inf | status no_flip | 0 flipping cell(s)>audit_stability() describes variation inside the
specified grid, not across all possible measurement choices.
audit_fragility() measures the smallest number of changed
grid dimensions needed to alter the stated conclusion. The fixed
response function used here isolates the audit structure; a substantive
study should include defensible prompt and model alternatives.
The offline response function writes no LLMR call log, so one compact JSON record supplies the fields needed to demonstrate archive construction.
log_path <- tempfile(fileext = ".jsonl")
writeLines(paste0(
'{"ts":"2026-07-01T10:00:01+0000","schema_version":"1.0","kind":"call",',
'"provider":"groq","model":"offline-demo","status":200,',
'"request":{"messages":[{"role":"user","content":"Classify statement 1"}],',
'"temperature":0},"usage":{"sent":12,"rec":1},',
'"response_id":"offline-1","text":"supportive"}'
), log_path)
archive <- archive_seal(
archive_build(log_path, name = "offline-study")
)
archive_check(
archive,
results = data.frame(response_id = "offline-1")
)
#> <archive_check | 1 record(s) | integrity: INTACT | records: VALID | seal: VALID>
#> completeness: 1/1 result rows map to a logged callThe check compares the stored record with its content hash and seal, then matches the result identifier to the logged call. A successful check establishes integrity and linkage; it does not promise the same reply from a changed model or provider.
archive_write() stores the records and manifest in a
directory. After archive_read(),
archive_check() verifies the copy and
LLMR::report() summarizes it.
archive_dir <- tempfile("offline-study-")
archive_write(archive, archive_dir)
copy <- archive_read(archive_dir)
archive_check(copy)
#> <archive_check | 1 record(s) | integrity: INTACT | records: VALID | seal: VALID>
LLMR::report(copy)
#> REPLICATION ARCHIVE 'offline-study'.
#> SEAL. Root 3d150c327fcea94e341939b0d3feead1eeb5558739e73340af3ff65776194444 over 1 record(s), sealed 2026-07-27T18:55:16+0000.
#> CALLS. 1 total between 2026-07-01T10:00:01+0000 and 2026-07-01T10:00:01+0000; 0 failure(s).
#> TOKENS. 12 sent, 1 received (as reported by providers).
#> VERIFIABILITY HORIZON.
#> offline-demo groq 1 call(s) api-contingent
#> ENVIRONMENT. R 4.6.1; LLMR 0.8.11; log schema 1.0; built 2026-07-27T18:55:16+0000.