What Agent Transcripts Reveal About Coding Agent Behavior

A coding agent transcript is the full record of what the model did during a task: every prompt, every tool call, every file it opened, every command it ran, in order. Most teams throw these away once they've checked whether the task passed. They throw away the only record of how the model actually got there. Benchmark scores tell you pass or fail. Transcripts tell you why, and why is where the useful engineering decisions live. Teams that skip transcript review are optimizing blind, and the gap between benchmark performance and production usefulness is bigger than most of them think.

The industry mostly evaluates coding agents through benchmark suites: SWE-bench for repository-level coding tasks, WebArena for browsing agents. These produce a single number, aggregated across a batch of runs, and that number is fine for comparing models at a glance. But it can't tell you an agent burned forty tool calls circling a bug it had already diagnosed correctly on call twelve. It can't tell you the agent read a file, forgot what it read, and read it again a few turns later. Token counts and exit status carry the same blindness: they count what happened without saying what kind of thing happened.

METR's work on SWE-bench Verified makes the stakes concrete. Roughly half of the test-passing pull requests generated by recent agents wouldn't actually get merged by the maintainers of the repositories they targeted. Passing the test suite and being useful in production are different bars, and a benchmark score clears the first while badly missing the second. Improving agent behavior means looking at the sequence of decisions that produced the result, not the outcome alone, and transcripts are the only instrument built for that.

What a coding agent transcript contains

Structurally, a transcript is a sequence of turns, comprising user prompts, assistant responses, the model's internal reasoning traces where those get exposed, tool calls, and the results those calls return. From each turn you can pull how many tokens got used, which tool got called and with what arguments, which file paths got touched, which shell commands ran and what they returned.

Zoomed out, a transcript works like an observability trace for an autonomous process. Tool selection, the arguments handed to the tool, the model's response to the tool's output, any reads or writes to a scratch file, any branch into a different plan: all of it gets captured as structured data. That's what lets someone reconstruct, after the fact, what the agent did, in what order, with what inputs and outputs at each stage.

The volume of transcript data now available is what turns this from a nice idea into an actual practice. As of April 2026, the SWE-chat dataset holds close to 6,000 real agentic coding sessions, more than 63,000 user prompts, and over 355,000 tool calls, pulled from more than 200 public repositories. At that scale, patterns become visible across many transcripts that are not visible in one. Idle loops are a good example: an agent re-confirms a task is done, over and over, without doing anything that moves it forward. A pass/fail metric can't catch that, since the task might still resolve eventually. The transcript shows turn after turn of confirmation with no new state produced.

How agents behave in real sessions, versus how we assume they behave

A behavior-grounded study by Gao & Chen (2026) combined the SWE-chat dataset (557 real agentic coding sessions) with AIDev (33,097 agentic pull requests) and extracted 94,813 development events, including 3,033 documentation interactions and 690,260 file-level change records. Most of what teams assume when they design agent harnesses turns out to be wrong, and the data says so directly.

Start with what agents actually read. Agent-facing artifacts, meaning instruction files like AGENTS.md or CLAUDE.md plus working notes such as plans, scratch directories, and verification logs, account for 60.5% of all documentation interactions. Classical technical documentation, the kind humans write for humans, accounts for just 10.6%. API references are 1.3%. Agents mostly consult material written for agents, not the reference docs teams spend most of their maintenance time on.

The assumed sequence of "read documentation, then write code" falls apart too. The transition probability from reading a doc straight to editing code is 0.002, functionally zero. What follows a documentation read is usually more reasoning or another documentation read, not an edit. Coding patterns split into two camps rather than blending: in 41% of sessions the agent writes essentially all the committed code, and in 23% the human writes all of it. The middle ground, where human and agent trade off authorship within one task, is rarer than most "pair programming with an AI" mental models assume.

Two more numbers round this out. Only 44% of agent-produced code survives into a user's final commit, so more than half gets discarded somewhere along the way. And users push back against agent output, correcting it, interrupting it, flagging a mistake, in 44% of all turns. That's close to a coin flip on any given turn, which is a lot more friction than the smooth-assistant framing most teams carry around in their heads.

Reading failure patterns: what transcripts reveal that metrics cannot

Quantitative metadata tells you a run failed. It doesn't tell you why. Grounded, behavior-level work fills that gap by breaking a broad failure like "the agent keeps acting without checking its own work" into specific, traceable sub-behaviors: ignoring its own earlier output, retrying an action without re-checking whether the underlying problem changed. Each sub-behavior ties back to a specific step in a specific transcript, with the actual text quoted as evidence.

A handful of structured approaches to building this kind of failure taxonomy have emerged. AgentErrorTaxonomy breaks failures into five operational modules. MAST applies grounded theory by hand across a corpus of multi-agent traces and produces a structured failure taxonomy. AutoTraceGT automates that grounded-theory process across six separate trajectory corpora and, in testing, recovers between 73% and 91% of the failure modes already identified in human-built taxonomies, while surfacing additional patterns those taxonomies missed.

Tool brittleness deserves its own category, and it's the one most teams get wrong first. A 2026 analysis manually categorized more than 3,800 publicly reported bugs across Claude Code, Codex CLI, and Gemini CLI. Developer frustration with these tools spans a range of failure modes, including problems at the seam between the model and its environment: how a tool gets invoked, how a shell command runs, how configuration and environment state get handled. That's a harness problem, not a model problem, and it becomes visible only when someone reads the transcripts closely enough to see exactly where the seam tears.

Transcript scanning has also caught integrity problems that spot checks would have missed. Automated scanners built in 2026 flagged 23 cases of test-data misuse across five different agents, a category of failure that random manual sampling almost certainly wouldn't have surfaced by luck.

None of this is post-mortem paperwork. Reading failures at the transcript level is the mechanism that connects a specific harness design choice to the outcome pattern it produces.

The method problem: why reading transcripts at scale requires a systematic approach

Agent trajectories can run hundreds of steps, and full evaluation runs can burn through hundreds of millions of tokens across a task set. Reading all of that by hand doesn't scale, and it definitely doesn't generalize to a behavior nobody anticipated in advance.

Three obvious approaches each fall short in their own way. Lightweight metadata scales fine but explains almost nothing about process. Full manual review by a person is interpretable but gets prohibitively expensive once trajectories get long and numerous. Pre-built behavioral classifiers are efficient but rigid, since they only detect categories of failure someone already thought to define, so they miss anything genuinely new.

Grounded theory starts from a different place. It's inductive: categories of behavior emerge from the data itself, based on observed patterns rather than a prior hypothesis about what failure ought to look like. It has a defined stopping point, theoretical saturation, the moment where sampling more transcripts stops adding new structure. And it leaves an auditable trail from the raw transcript up to the behavioral category it lands in. Because it's inductive, it stays open to behaviors nobody predicted going in, which happens to be exactly the category of failure a pass/fail metric can't catch by design.

AutoTraceGT is a working example of this at scale: a multi-agent pipeline that runs open coding, axial coding, and theoretical coding on agent trajectories automatically, continuing until it hits saturation, and produces a behavioral taxonomy specific to the task at hand. The codebooks it produces beat both zero-shot and few-shot LLM baselines on downstream failure prediction. No team needs to run grounded theory by hand to take the lesson from it: transcript analysis needs a real stopping criterion, not spot checks done whenever something happens to break.

How transcript analysis connects to harness design decisions

An agent is a model plus a harness, where the harness comprises the runtime loop, the tool set, the context management strategy, the safety rails, and the orchestration logic that wraps around the model and connects it to the world. Changing the harness changes the agent's behavior even with the exact same model underneath it, because the harness governs how the model's outputs get executed and interpreted; this matters more than model choice in most of the cases where it's been measured.

LangChain found that a harness change alone moved a coding agent from 52.8% to 66.5% on Terminal Bench, with no change to the model. Princeton's CORE-Bench found a single model scoring 42% under one scaffold and 78% under another. Vercel found that stripping 80% of an agent's available tools took success rate from 80% to 100% on the same underlying model, more than halved token use, and cut latency from 724 seconds to 141. Together these cases point at one conclusion: the harness, not the model, is usually the bigger lever, and teams that spend their budget chasing the next model release are pulling the weaker one.

The Vercel case matters most here because the fix was removing tools, and figuring out which tools to cut is exactly the judgment transcript analysis supports. A transcript shows which tools get called often and succeed, which get called and fail, and which barely get touched at all: dead weight sitting in the tool set, adding confusion and nothing else. When an agent has more tools than it can use well, transcripts show the damage directly, in the form of repeated calls that return nothing useful, malformed parameters, and brittleness right at the seam between tool and environment.

Context-window limits are visible in transcripts the same way. In long trajectories, the context window becomes a hard wall eventually, and transcripts make visible both the moment it gets hit and what changes after: the agent losing track of an earlier decision, or redoing work it already finished. A harness that skips transcript analysis can't tell the difference between a design that still fits and one that's turned into brittle scaffolding wrapped around a model that quietly outgrew it.

What transcript analysis reveals about agent productivity beyond self-reporting

METR ran a transcript-based productivity study in February 2026 covering 5,305 Claude Code transcripts from 7 METR technical staff during January 2026. An LLM judge estimated, for each task in the transcripts, how long it would have taken without AI help, then compared that estimate against the actual time the task took.

The result: an estimated time-savings factor ranging from about 1.5x to 13x depending on the staff member, with individual variation running from roughly 2x to past 10x. Self-reported daily estimates from the same staff came in consistently lower than what the transcript analysis measured. People doing the work underestimated their own gains by a wide enough margin that self-reporting looks structurally unreliable here, not just a little noisy.

Even the transcript-derived number has limits, and it functions more as an upper bound than a settled figure. Task substitution plays a role: agents make some work possible that wouldn't have happened otherwise, and that new work tends to carry lower value than the tasks it displaces. Task selection matters too, since transcripts only capture work that already went through an AI assistant and miss everything that didn't. Individual differences across staff members add more noise on top. And the LLM judge itself was checked against only 34 ground-truth labels, a small check propping up a very large measurement.

Task substitution deserves separate treatment, because it's a behavioral signal, not just a statistical caveat. Transcripts can show when agents get pointed at low-value, nice-to-have work instead of the tasks that actually matter, which says something about how a team chooses to deploy its agents, apart entirely from how well the agent performs once deployed. Line the three measurement approaches up: benchmarks tend to overstate real-world usefulness, self-reports tend to understate actual gains, and transcript analysis is in between, more granular than a benchmark and more honest than a memory of how the week went.

How agents treat their own instruction files

Go back to the Gao et al. (2026) documentation numbers and split them further. Agent instruction files, AGENTS.md and CLAUDE.md specifically, account for 35.4% of documentation interactions on their own. Agent working notes, plans, thoughts directories, verification logs, add another 25.1%. Combined, that's the 60.5% of everything an agent reads that counts as documentation.

That carries a direct implication for harness design: the files agents pay the most attention to are exactly the files the harness team controls. Instruction files are the primary source of information the agent actually works from. The quality of that file, how specific it is, how current it stays as the codebase moves, matters far more than most teams currently treat it. A stale AGENTS.md is a direct hit on the agent's decision quality on every single task that touches it. It's a direct hit on the agent's decision quality on every single task that touches it.

Agents also write documentation at close to the same rate they consult it, though documentation still trails code in practice: in multi-commit pull requests, code gets touched several times more often than documentation does. And one assumption baked into a lot of harness design falls apart under transcript inspection. Agents don't show a consistent pattern of checking their own output against a documented spec before calling a task done. The validation loop a lot of teams assume runs quietly in the background simply isn't present in the data.

Trimming, restructuring, or just keeping an instruction file current, based on what transcripts actually show an agent consulting rather than what a harness designer assumes it consults, is a cheap and direct lever on behavior. It belongs in the same category as pruning an unused tool from the tool set: a small, targeted change with an outsized effect on what the agent does next.

Building a practical transcript analysis practice for engineering teams

The signal in transcript data lives in the variation across sessions, not inside any single one. A team that reads one transcript closely gets an anecdote. A team that reads hundreds gets a pattern, and that difference should shape how a transcript analysis practice actually gets built.

Start with what to instrument quantitatively. Tool call frequency and outcome, broken down by individual tool, shows which tools work, which fail, and which sit unused: dead skills a harness carries around for no reason. Retry and loop detection, meaning repeated identical or near-identical tool calls with no real change in state between them, flags an agent stuck cycling through a failure mode. Context consumption over the length of a trajectory shows where sessions hit hard limits and what behavior shifts once they do. Human pushback events, corrections and interruptions logged in the transcript, work as a rough proxy for output quality that never made it as far as a commit.

Some of this only becomes visible by actually reading the text. Reasoning traces around a failure point matter because they show not just what the agent did but what it believed it was doing, which is often where the real disconnect sits. Instruction-file access patterns, which sections get pulled up and at what point in a task, deserve a close look of their own too.

Put together, this becomes a loop: analyze transcripts, find the tools that are dead or brittle, fix or remove them, run again, and compare the new transcripts against the old ones. That loop moves faster than waiting on a full benchmark re-evaluation cycle. In a high-throughput setting, transcript signals pulled from CI failures and ongoing agent pull request activity can feed straight into merge queue decisions, turning transcript data into something operational rather than purely diagnostic.

None of this demands reading every transcript end to end. Automated grounded-theory pipelines and LLM-as-judge scoring both show that systematic coverage doesn't require exhaustive human review, as long as the method matches the scale of the deployment. The goal is sampling proportional to that scale, not full coverage for its own sake: enough of it that failure categories emerge from the data itself, rather than getting reconstructed after the fact from whatever incident happened to get noticed.