Provenance and archiving: an agent run as evidence

When a language model produces the data you will analyze, the model is an instrument. Instruments need a record: what was asked, which model answered, how many tokens it cost, and whether anything failed. Without that record a result cannot be checked, and a study that cannot be checked is hard to defend. This vignette shows how each LLMRagent run records itself, and how to write an archive a reader can open months later.

library(LLMRagent)
cfg <- LLMR::llm_config("groq", "openai/gpt-oss-20b", temperature = 0.7)

One object, five views

Anything you run, whether a single chat, a deliberation, or an experiment, converts to one object through as_agent_run(). That object answers five questions, and you ask each by naming a level:

panel <- list(
  agent("Optimist", cfg, persona = "You see opportunity first."),
  agent("Skeptic",  cfg, persona = "You probe for hidden costs.")
)
debate <- deliberate(panel, "Should the department adopt a four-day week?",
                     rounds = 1)

run <- as_agent_run(debate)

The levels are deliberately plain tibbles, so the rest of your analysis is ordinary data work:

tibble::as_tibble(run, level = "utterance")  # who said what, in order
tibble::as_tibble(run, level = "call")       # one row per model call
tibble::as_tibble(run, level = "event")      # the full event graph
tibble::as_tibble(run, level = "tool")       # every tool invocation
tibble::as_tibble(run, level = "state")      # each agent's memory at the end

The utterance level is the transcript: the thing you usually read. The call level is the instrument log: every model call as a row, with the served model version, the token counts, a success flag, and a request_hash that uniquely identifies the exact request that produced it. The agent recorded all of this while it answered.

Diagnostics for a run

diagnostics() reads the call level and returns a one-row summary. The numbers come from the same place the wider ‘LLMR’ ecosystem reads them, so they agree across tools rather than reporting two different figures:

diagnostics(run)

A high failure count, an unexpected model version, a token bill larger than the work warrants – these surface here before they surface in your conclusions.

Reporting within scope

report() writes a short methods paragraph and, importantly, refuses to overstate. An agent run is model-conditioned output, not a sample from a human population, and report() says so unless you have attached a calibration (see the validity vignette). The report carries that scope condition unless a calibration is attached.

report(run)

The manifest: identity without the transcript

Two runs are “the same study” when their design is the same: same personas, same tools, same turn policy, same model and settings. They are emphatically not the same when any of those change, even if the transcripts happen to read alike. agent_manifest() captures design identity as a set of content hashes, keeping outcome (the transcript) separate from apparatus (the design):

m <- agent_manifest(run)
m$manifest_hash    # changes iff the design changes

A reworded persona, a changed tool body, or a different temperature moves the hash. Re-run the identical design and it stays put. This is the same split ‘LLMR’ makes between a request’s identity and its content, lifted to the level of a whole study.

Archiving: seal the evidence

archive_agent_study() writes a self-describing directory. The transcript, the event graph, the manifest, the artifacts (votes, verdicts, designs), a drafted methods note, and a checksum file that seals the whole thing. The centerpiece is calls.jsonl, which is a genuine ‘LLMR’ audit log: you can read it back with ‘LLMR’ itself.

dir <- file.path(tempdir(), "four_day_week_study")
archive_agent_study(run, dir)
list.files(dir)

The archive keeps the original hashes. The hashes are computed over the original text, so they match the live run even if you redact the stored copy for privacy. And the request hash in calls.jsonl joins to the request_hash you saw at the call level. That join is the integrity check: it lets a skeptic confirm that the archived log describes the run it claims to describe.

calls   <- tibble::as_tibble(run, level = "call")
archived <- LLMR::llm_log_read(file.path(dir, "calls.jsonl"))

# every archived call is one of the calls the run actually made
all(archived$manifest$request_hash %in% calls$request_hash)

If you keep a live audit log during the run (call LLMR::llm_log_enable("calls.jsonl") before you start), the archive copies that real file, filtered to this run’s calls, rather than reconstructing it, and so preserves the log’s own record hashes.

Redaction without lying

Survey responses and interview text often cannot be stored verbatim. redact scrubs the free-text columns of the stored copy, while the hashes stay computed over the original. The archive remains internally consistent: it does not claim a request body it did not send, and the join still holds.

archive_agent_study(run, file.path(tempdir(), "redacted_study"),
                    redact = c("four-day", "salary"))

For privacy that must omit the request entirely, pass include_messages = FALSE: the archive then carries metadata and hashes but no message text at all.

Why archive

A reviewer asks which model you used and whether you cherry-picked. A co-author wants to re-run last month’s study and get a comparable result. A replication team, a year on, has only your files. In each case the answer is the archive: the run, the manifest, and the log that joins them. The archive makes a model-generated result something you can inspect, rerun, and defend.