Compare panel responses with a human benchmark

Compares closed-item response shares with human benchmark shares. The comparison uses responses that parsed and were not reported as execution failures. It does not alter responses or adjust their shares.

Usage

panel_benchmark(responses, benchmark, benchmark_name = "benchmark")

Arguments

responses

A panel_administer() result.

benchmark

A data frame with item_id, response, and share columns. Shares must be numeric probabilities in [0, 1] without missing values. The function raises an error when an item's share total differs from 1 by more than 0.01. Each item-response pair must occur once, and covered response labels must be offered options. An offered option that the benchmark omits for a covered item is taken as a known human share of zero, not as a missing value.

benchmark_name

Name used to identify the benchmark in reports.

Value

responses with its benchmark field populated.

• table contains the panel share, human share, and their difference for each covered item-response pair.

• nonresponse contains execution-failure and nonresponse rates by item.

• items_covered and items_total record closed-item coverage.

• mean_abs_dev and max_dev summarize absolute deviations.

• benchmark_name and ts identify the comparison and its time.

Details

Without a benchmark, response shares describe the configured model under the supplied personas.

The function refuses conjoint responses because profile assignments differ across personas. It also refuses a covered item when every execution failed. If executions occurred but no response parsed for an item, its panel shares and deviations are NA.

Examples

panel <- panel_from_margins(list(group = c(A = 1)), n = 4)
instrument <- panel_instrument(
  item_choice("fund", "Choose an investment.", c("transit", "roads")),
  randomize = character(0)
)
cfg <- LLMR::llm_config("groq", "example-model")
runner <- function(experiments, ...) {
  experiments$response_text <- "transit"
  experiments
}
responses <- panel_administer(
  panel, instrument, cfg, .runner = runner
)
benchmark <- data.frame(
  item_id = "fund",
  response = c("transit", "roads"),
  share = c(.6, .4)
)
panel_benchmark(responses, benchmark, "Example human benchmark")