Skip to contents

LLM-as-a-Judge Evaluation

Source: R/LLM_parallel_utils.R
llm_judge.Rd

Evaluates outputs in a target column against a custom prompt using llm_mutate_tags() for clean tag-based extraction. The target column value is available in the prompt template as {.target}.

Usage

llm_judge(
  .data,
  .target,
  .config,
  prompt,
  .tags = c("reasoning", "score"),
  .output = "judge_res",
  ...
)

Arguments

.data

Data frame of experiment results.

.target

Bare column name containing the output to evaluate.

.config

The judge llm_config.

prompt

Evaluation prompt template. Use {.target} to reference the target column value (other data columns are also available).

.tags

Tags to extract from the judge response. Defaults to c("reasoning", "score").

.output

Name of the column that receives the judge's raw response. Default "judge_res".

...

Passed to llm_mutate_tags().

Value

.data with judge output columns appended.

See also

llm_mutate_tags(), llm_parse_tags()

Examples

if (FALSE) { # \dontrun{
results |>
  llm_judge(
    .target = response_text,
    .config = judge_cfg,
    prompt = "Rate this answer on a 1-5 scale:\n{.target}",
    .tags = c("reasoning", "score")
  )
} # }