Run Arms of a Focus Group from One Shared Prefix

Runs a session up to an injection point once, then continues the full remainder of the session (every later question and turn, and the final summary) once per arm, so that every arm shares each message before the injection. Comparing the arms then compares sessions that differ only in what was injected and in what followed from it.

Usage

run_focus_group_arms(
  topic,
  config,
  arms,
  n_participants = 6,
  guide = c(Opening = 1, Icebreaker = 1, Engagement = 2, Exploration = 3, Closing = 1),
  demographics = NULL,
  survey_responses = NULL,
  flow = "desire_based",
  seed = NULL,
  message_mode = c("roleflip", "flat"),
  verbose = TRUE,
  max_participant_responses = NULL,
  max_calls = 120L,
  confirm = FALSE,
  max_tokens_config = NULL,
  reasoning_effort = NULL,
  on_desire_failure = c("error", "random"),
  probes = NULL,
  moderator_verbatim = FALSE,
  call_on_silent = c("opening", "midpoint", "end"),
  call_on_text = NULL,
  handles = NULL,
  final_summary = TRUE,
  .runner = NULL
)

Arguments

topic: Character. Focus group topic. Or a recorded session to continue, a FocusGroup or a focus_group_result; see "Continuing a recorded session".
config: An explicit LLMR::llm_config used by all agents and group-level model tasks.
arms: A named list of at least two arms, as described in Details.
n_participants: Integer. Number of participants, excluding the moderator.
guide: A named numeric vector or named list. The names are phases, among "Opening", "Icebreaker", "Engagement", "Exploration", and "Closing", each at most once and in that order. Numeric values select that many moderator instructions from the phase banks. Character vectors supply the ordered instructions directly. The default is a compact session (one opening, one icebreaker, two engagement and three exploration questions, one closing); with the default flow, participants, response limit, and call-ons it plans at most 111 model outputs, inside the default max_calls of 120. Larger guides multiply quickly: each question round costs one output per response plus, under "desire_based", one scoring output per participant. NULL runs a minimal script: an opening, one open discussion of topic, and a closing.
demographics: Optional participant demographics, one row per participant. With this, survey_responses, and personas all absent, the participants are drawn from the bundled ANES 2024 records.
survey_responses: Optional participant survey answers, one row per participant.
flow: Character. One of "round_robin", "probabilistic", or "desire_based".
seed: Optional whole number (a fraction such as 1.7 is an error) governing in-package sampling: the respondents drawn from the bundled records and the random choices of the turn-taking flow, such as the speaker draws of "probabilistic" and the tie-breaks of "desire_based". The random waits of a provider's retries do not draw from this stream. The caller's random-number state is restored when the function returns. The option focusgroup.seed, when set, fixes the respondent draw instead.
message_mode: Character. "roleflip" or "flat".
verbose: Logical. Print session progress.
max_participant_responses: Optional integer maximum number of participant responses per moderator question. NULL, the default, uses the option focusgroup.max_participant_responses, which is 2 unless you set it.
max_calls: Integer. Maximum estimated live model outputs allowed without confirmation; a finite whole number.
confirm: Logical. Permit a live run whose estimate exceeds max_calls.
max_tokens_config: Optional named list of token limits by call type, as for FocusGroup$new(): utterance, moderator, desire, desire_retry, summary, and interim_summary. A model that reasons before it answers needs desire and desire_retry large enough to hold its reasoning and the score.
reasoning_effort: Optional reasoning-effort setting passed to the provider: one string, such as "low", for every call, or a named vector by call type ("moderator", "participant", "desire", "summary", "probe"). It is written over the reasoning_effort of config for those calls; NULL leaves config as it is. LLMR forwards it on OpenAI-compatible providers.
on_desire_failure: Character. Under "desire_based", what happens when a participant's desire score cannot be obtained after its retry. "error" (the default) stops the session with an error of class focusgroup_desire_failure, returned as a partial session as described in Details. "random" chooses the speaker among the participants with valid scores by the usual rule, and uniformly among the eligible participants only when none has a valid score; the failure is recorded in the selected message's metadata$desire_scoring_failure.
probes: Optional fg_probe() or list of probes. A probe at or before the branch point is asked once, in the prefix; a later one is asked in every arm.
moderator_verbatim: FALSE (the default) has the moderator's model render each guide item as a line of its own. TRUE delivers every guide item exactly as written, with no model call, and a vector of item numbers delivers those items so. A verbatim message carries metadata$verbatim = TRUE. The opening and closing instructions of the numeric phase banks are instructions to the moderator, so verbatim delivery suits a guide written as the lines to be said.
call_on_silent: Where the moderator calls on the participants who have not spoken since the previous such point, as described in Details: any of "opening", "midpoint", and "end" (the default is all three), guide item numbers, or "none" (or FALSE) for no call-ons.
call_on_text: The call-on line. It must contain {participant}, which is replaced by the participant's handle; NULL uses "{participant}, I'd like to hear from you as well. What are your thoughts?".
handles: The labels under which the agents see each other: NULL (the default) for "P1", "P2", ... and "Moderator", "ids" for the agent IDs, or a character vector of handles named by agent ID, as for FocusGroup$new().
final_summary: Logical. TRUE (the default) ends the session with a model-written summary; FALSE makes no summary call and leaves summary empty.
.runner: Optional function used instead of live model calls. It receives a data frame with config and messages list-columns and returns those rows with at least response_text. Token columns such as sent_tokens, rec_tokens, and total_tokens are optional; when they are absent, usage is unknown.

Value

A focus_group_arms object, a list with:

arms: A named list of focus_group_result objects, one per arm, each a complete session as returned by run_focus_group(). Its transcript, calls, and records begin with the shared prefix; its usage counts the whole session, prefix included; and its metadata$branch records the arm, the branch point, the number of prefix messages and calls, and prefix_hash, a hash of the prefix transcript that is the same in every arm.
prefix: The shared prefix as a focus_group_result with metadata$complete set to FALSE and no summary.
branch_point: The branch point, as written, such as "fg_after_item(4)".
injections: A data frame of every arm's injections, as in run_focus_group().
probes: A data frame of every probe answer: those asked in the prefix once, with arm set to NA, and those asked in each arm.
usage: The token usage of all the calls made: the prefix once and each arm's remainder. Adding up the arms' own usage would count the prefix once per arm.
metadata: A list with topic, branch_point, arms (the arm names), estimated_calls, seed, prefix_messages, prefix_hash, complete, and failure.

When a call fails, the run stops with an error of class focusgroup_incomplete_arms whose partial field holds the focus_group_arms object built from what was completed, with the arm that failed, if any, as a partial session; its arm field names that arm and its parent field holds the original error.

Details

Each arm is NULL, which continues the prefix with nothing injected, one fg_injection(), or a list of injections. The first injection of every arm that has one must name the same point; that point is where the session branches. An arm's later injections may name later points. A neutral injection at the branch point, rather than NULL, gives a control arm that also receives a message there.

The prefix is run once, with its probes, and a copy of the whole session (agents, turn-taking state, records, and token counters) is taken at the branch point; each arm continues its own copy. Each arm also starts from the random-number state at the branch point, so the random draws that follow, such as tie-breaks and probabilistic speaker choices, use the same stream in every arm; a provider's retries, which wait a random time, do not draw from it. With seed, the whole run is reproducible; without it, the arms still share the stream from the branch point on.

A branch point must lie within the guide: fg_after_item(k) needs k no larger than the number of guide items, and fg_before_turn(t) needs t no larger than the number of participant turns the guide allows. A point the session still does not reach, as when a custom flow chooses no speaker, stops the run with an error of class focusgroup_injection_unreached.

The estimate checked against max_calls is that of one session times the number of arms, an upper bound, since the prefix is paid once.

Continuing a recorded session

When topic is a recorded session, a FocusGroup or a focus_group_result (for example one read from a session file by focus_group_from_transcript()), no prefix is run: each arm continues the record from a participant turn. The branch point must be fg_before_turn(t), where turn t is the t-th message by a participant. The messages before it are the shared prefix, taken from the record as they are, and the record's turn t and everything after it are replaced by each arm's continuation. An arm fg_injection(text, at = fg_before_turn(t), from = "P2") has P2 say text as turn t (see fg_injection()); a NULL arm regenerates turn t and the rest; an outside voice or a moderator line can also be injected there.

The session is rebuilt as it stood just before turn t: the guide item and its stage, the turns already given on it, the call-on queue, a participant the moderator's last line addressed, the question in force, the interim summary in force, and the turn-taking rule's state, with the record's personas, handles, guide, settings, templates, standing rules, and message construction. config and .runner are given to every agent; reasoning_effort and max_tokens_config, when given, replace the record's. The arguments that the record fixes (n_participants, guide, demographics, survey_responses, flow, message_mode, max_participant_responses, on_desire_failure, moderator_verbatim, call_on_silent, call_on_text, and handles) must be left out. A probe at the branch point itself is asked once, on the rebuilt session, before the arms, so that every arm shares its answers, as a probe there is asked in the prefix of a new session. A probe or an injection at a point the record has already passed is an error, and so is a transcript without moderator lines, which has no guide item to continue, as are a custom turn-taking flow, whose state cannot be rebuilt, a record without a topic, and a participant who has only the placeholder persona of an imported transcript; give personas with focus_group_from_transcript(x, personas = ...). Everything is checked before any call. Under round robin with a model that replies the same way to the same messages, a NULL arm repeats the record. Each arm's calls and usage hold the record's calls before the cut and the arm's own after it. With seed, the caller's random-number state is restored afterwards.

A continuation of a human transcript is a model's continuation of a human discussion: the later moderator lines are the recorded ones, delivered as written, and the participants' turns are generated from the personas given. Arms are comparable with one another, not with the people who took part.

Examples

cfg <- LLMR::llm_config("groq", "openai/gpt-oss-20b",
                        api_key = LLMR::llm_api_key_env("GROQ_API_KEY"))
scripted <- function(experiments, ...) {
  experiments$response_text <- "Evening hours matter most to people who work."
  experiments
}
guide <- list(Opening = "Welcome the participants.",
              Engagement = c("Which hours matter most?",
                             "Who is left out by the current hours?"),
              Exploration = "What would you change first?",
              Closing = "Thank the participants and close.")
arms <- list(
  objection = fg_injection(
    "Extended hours would cost the branch two weekday programs.",
    at = fg_after_item(3), from = "Guest expert"),
  control = NULL)
out <- run_focus_group_arms(
  "Library hours", cfg, arms = arms, n_participants = 2, guide = guide,
  flow = "round_robin", verbose = FALSE,
  probes = fg_probe("From 1 to 7, how strongly do you favor longer hours?",
                    at = list(fg_after_item(3), "end")),
  .runner = scripted)
out
#> <focus_group_arms>
#> Topic: Library hours
#> Branch point: fg_after_item(3) after 6 shared message(s)
#>   objection: 13 messages
#>   control: 12 messages
#> Tokens: unknown
#> Token usage is incomplete for 25 call(s).
out$probes[, c("arm", "point", "agent_id", "answer")]
#>         arm            point agent_id
#> 1      <NA> fg_after_item(3)       P1
#> 2      <NA> fg_after_item(3)       P2
#> 3 objection              end       P1
#> 4 objection              end       P2
#> 5   control              end       P1
#> 6   control              end       P2
#>                                          answer
#> 1 Evening hours matter most to people who work.
#> 2 Evening hours matter most to people who work.
#> 3 Evening hours matter most to people who work.
#> 4 Evening hours matter most to people who work.
#> 5 Evening hours matter most to people who work.
#> 6 Evening hours matter most to people who work.

# Continue a recorded session from its third participant turn: as recorded,
# and with P2 saying something else there.
session <- run_focus_group("Library hours", cfg, n_participants = 2,
                           guide = guide, flow = "round_robin",
                           verbose = FALSE, .runner = scripted)
path <- tempfile(fileext = ".fg.jsonl")
writeLines(format(session), path)
again <- run_focus_group_arms(
  focus_group_from_transcript(path), cfg,
  arms = list(regenerated = NULL,
              changed = fg_injection("Mornings matter more to me.",
                                     at = fg_before_turn(3), from = "P2")),
  verbose = FALSE, .runner = scripted)
again$arms$changed$transcript[, c("speaker_id", "text")]
#>    speaker_id                                          text
#> 1      System          Participants: P1, P2; Moderator: MOD
#> 2         MOD Evening hours matter most to people who work.
#> 3         MOD Evening hours matter most to people who work.
#> 4          P1 Evening hours matter most to people who work.
#> 5          P2 Evening hours matter most to people who work.
#> 6         MOD Evening hours matter most to people who work.
#> 7          P2                   Mornings matter more to me.
#> 8          P1 Evening hours matter most to people who work.
#> 9         MOD Evening hours matter most to people who work.
#> 10         P2 Evening hours matter most to people who work.
#> 11         P1 Evening hours matter most to people who work.
#> 12        MOD Evening hours matter most to people who work.