Skip to contents

Agreement among coders or replicates

Source: R/coder_metrics.R
coder_agreement.Rd

A thin, shape-aware wrapper around LLMR::llm_agreement(). Pass either a gold_set() whose coders columns hold individual human codings (the human-human reliability a methods section must report), or any data frame plus the label columns to compare (e.g. model replicates from code_corpus(), or one human column and one model column).

Usage

coder_agreement(x, cols = NULL)

Arguments

x

A gold_set(), a code_corpus() result, or a data frame.

cols

For data frames: character vector of label columns. Ignored for gold sets (their coders columns are used).

Value

An llmr_agreement object (mean pairwise agreement, Krippendorff's alpha, per-unit majorities); see LLMR::llm_agreement().

Examples

labels <- data.frame(
  coder_a = c("yes", "no", "yes", "no"),
  coder_b = c("yes", "no", "no", "no"))
coder_agreement(labels, c("coder_a", "coder_b"))