Summarize execution failures, parse failures, and first-option sensitivity

Counts execution and parse failures by item. For closed items administered with randomized option order, it also applies a chi-squared test to the chosen response and the option shown first. The test does not use the full option permutation.

Usage

panel_bias_audit(responses)

Arguments

responses

A panel_administer() result.

Value

A tibble: item_id, n, parse_failures, execution_failures, order_effect_p, and order_test_note. order_effect_p is the first-option chi-squared p-value. When it is NA, order_test_note says why: the item is open, its option order did not vary, fewer than four responses parsed, every parsed response is the same, one option was shown first in every parsed response, the expected cell counts are below five, or the test itself failed, in which case a warning is also issued. order_test_note is NA when a p-value is reported.

Examples

panel <- panel_from_margins(list(group = c(A = 1)), n = 4)
instrument <- panel_instrument(
  item_choice("pick", "Choose one.", c("A", "B")),
  randomize = character(0))
config <- LLMR::llm_config("groq", "example-model")
runner <- function(experiments, ...) {
  experiments$response_text <- "A"
  experiments$success <- TRUE
  experiments
}
responses <- panel_administer(panel, instrument, config, .runner = runner)
panel_bias_audit(responses)