Transcript Analysis Pipelines for AI Coding Agents

Coding agents don't just answer prompts anymore. They run for minutes or hours, calling tools, checking their own output, backtracking when something breaks, and eventually landing on a build that either passes or fails. That shift, from single-turn response to long-running autonomous loop, changes what "accountability" even means for a team running these agents in production. A green build tells you the agent got there. It says nothing about how, and transcript analysis pipelines exist to close that gap.

A transcript, in this context, is a precise thing: the full sequential record of an agent's steps within a task. Tool calls, the observations that come back from those calls, self-corrections, moments of backtracking, and the final output, all in order. That's a different object than an infrastructure log. Logs record what the system did. Transcripts record what the agent decided, one step after another, and that distinction is the whole reason this discipline exists.

What a transcript analysis pipeline produces

A pipeline isn't a dashboard. It isn't a log aggregator sitting behind a nice UI, either. It's a structured measurement loop: raw transcripts go in one end, and actionable signal comes out the other.

What comes out, specifically, breaks into a few categories. Skill usage distribution shows which tools get called, how often, and whether calling them correlates with a task actually succeeding. Stall and backtrack maps show where in execution an agent loops on itself, gives up on an approach, or pivots to something else entirely. Success and failure signatures capture the behavioral patterns that tend to precede a green or red outcome, patterns that are invisible if you only look at the final verdict. Pass@k (best-case performance across k attempts) tells a very different story than pass^k (the probability that all k attempts succeed). An agent that succeeds 70% of the time per trial looks like it's succeeding a much higher share of the time on pass@3, but its pass^3 figure sits far lower. That gap between "can do it once" and "does it reliably every time" is often the real story a team needs to see.

None of this works without a shared vocabulary. A task is one test, with defined inputs and explicit success criteria. A trial is a single attempt at that task. A grader is whatever scoring logic, deterministic code or a calibrated model, converts a trial into a verdict. And pass^k is the consistency view: not "can this succeed," but "will this keep succeeding."

The pipeline is not a one-time audit; it is not transcript transcription either. Nobody is converting audio to text here. This operates on structured execution records, and the whole point is that it runs continuously, with production traces feeding directly into the next round of test cases.

Collecting and compressing transcripts at scale

Evaluation runs at this scale can burn through 100 million to a billion tokens per task. Reading that manually isn't an option, so compression becomes the first real engineering problem in the pipeline, not an afterthought.

METR's February 2026 prototype offers a concrete model for how this gets done. The team collected 5,305 Claude Code transcripts from seven METR technical staff over the course of a month in January 2026. Their compression method summarized assistant actions and outputs, using GPT-4o to do the summarizing, while keeping code diffs fully intact before passing the compressed transcript to a judge. The principle behind that choice is more important than the specific tool: compression has to preserve whatever signal you actually plan to score. Code diffs stayed whole because they carry the net successful output of the run, and summarizing them away would have deleted exactly the thing being measured.

Stripping the right material out matters just as much as preserving the right material. Failed subtasks that a user explicitly rejected don't reflect agent skill on the path that actually succeeded, so they get cut. Agent overhead, the work of spinning up subagents or managing sessions, is setup noise that wouldn't exist without the AI layer at all and says nothing about task capability. Abandoned work, where a user pivots from one approach to another mid-task, leaves behind an execution record for the abandoned path that's just noise for outcome analysis. And self-explanation, an agent narrating what it just did, adds no new information about behavior since the code diff already shows it.

Decisions made at collection time echo through everything downstream. METR counted overlapping ten-minute windows only once, specifically to avoid overcounting when sessions ran in parallel. The team also had to distinguish human-typed messages from model-generated ones in more advanced workflows, a real risk in multi-agent setups where one agent can end up impersonating a user turn. A spot check of 100 randomly sampled "human-typed" messages from METR's most advanced user found no obvious misclassifications, which is reassuring but not proof the risk disappears at scale.

None of this requires a massive dataset to start. Anthropic's guidance recommends beginning with 20 to 50 tasks pulled from real production failures. Early-stage agents show large effect sizes per change, so a small, well-chosen set is enough to start iterating honestly.

Grader design: deterministic checks versus LLM judges

Once transcripts are collected and compressed, something has to score them, and the choice isn't really a choice between two camps. It's a question of which grader fits which kind of failure, and most working pipelines end up needing both.

Code-based graders, string matching, regex, static analysis, direct outcome verification, are fast, cheap, and deterministic. No calibration step required. They're the right tool when success criteria are genuinely fixed: tests pass or they don't, output matches a schema or it doesn't, the linter is clean or it isn't. Model-based graders, by contrast, handle rubric scoring, pairwise comparison, and multi-judge consensus. They're flexible enough to deal with open-ended behavior, but that flexibility comes with a cost: they need calibration before anyone should trust their output.

Code-based graders fall apart on long-horizon tasks where "correct" isn't a fixed string, or where the path an agent took matters as much as where it ended up. Model-based judgment becomes necessary specifically for detecting stall patterns, for judging whether a given tool call made sense given the surrounding context, and for scoring reasoning quality across a multi-step execution. Docent, a transcript analysis tool described in Meng et al. (2025), demonstrates exactly this: it identified benchmarks containing several unsolvable problems and models quietly ignoring their instructions. Neither failure would have shown up in a code-based check, because both live in the texture of the transcript rather than in the final output.

The design principle behind this is simple to state and easy to violate: grade what you actually intend to improve. A grader that only looks at final output will never surface a mid-execution failure mode, no matter how sophisticated the grader gets.

Calibrating the LLM judge so scores mean something

An uncalibrated judge is worse than no judge at all, because it produces numbers confident enough to optimize a dashboard while quietly diverging from what an expert reviewer would say. The pipeline looks healthy. The pipeline looks healthy, but it isn't, because a flawed judge produces numbers confident enough to optimize a dashboard while quietly diverging from what an expert reviewer would say.

METR's study on NanoGPT expenditure horizons found exactly this kind of divergence. The LLM judge underestimated human contributor effort by roughly 37% on totals and about 32% by geometric mean, with the bias concentrated on tasks requiring considerable experimentation, which happen to be the tasks a team most wants to understand well. METR applied a correction factor of α = 1.58 (the ratio of total contributor-estimated hours to total judge-estimated hours), with a bootstrap 95% confidence interval of [1.07, 2.62]. Without that kind of correction, time-savings figures pulled from transcript analysis function as a soft upper bound rather than a real productivity number.

Judges carry other known biases: position bias, where earlier options in a pairwise comparison get preferred regardless of merit; verbosity bias; self-preference; format bias; and score drift that creeps in across long evaluation runs. None of these are exotic. They're the standard failure modes of using one model to grade another, and they appear whether or not a team goes looking for them.

Calibration, then, is a process, not a step. It starts with a human-labeled gold set (METR's validation set ran 34 ground-truth labels, small, but sufficient for spotting the direction and rough size of a bias). From there, agreement between judge and human reviewers needs tracking on an ongoing basis. Where judge and human diverge systematically, the task type usually explains why: METR found the widest gaps on tasks that required heavy experimentation and were hard to interpret from outputs alone.

The PostTrainBench work, published on arXiv, makes the calibration challenge concrete in a different domain. An LLM-as-a-judge scanner flagged 23 contamination cases involving test data misuse across five agents, but the scanner's accuracy swung substantially depending on the benchmark, the criteria, and the underlying model being judged. Calibration doesn't transfer automatically from one setting to the next. It has to be redone per benchmark.

Connecting transcript signals to CI gates and merge decisions

A merge queue exists to keep the main branch green, by testing every pull request against the future state of main rather than the stale point where the branch started. Transcript analysis tells a team which agent behaviors are the ones putting that invariant at risk.

There's a timing problem: by the time a human gets an automated email saying CI broke, the agent's own context on that run is gone. By the time a human gets an automated email saying CI broke, the agent's own context on that run is gone. The window where a fix is actually cheap to make is while the agent is still running, not after. exe.dev described a practical pattern for this in June 2026: build a merge queue command, then build a second command that runs everything the merge queue runs except the actual merge, and hand that second command to agents directly. That lets an agent catch and fix failing tests inside its own active context window, before the code ever lands.

Transcript data adds detail build logs simply don't carry. It shows whether an agent attempted a real fix and self-corrected, or stalled out and shipped output anyway despite the stall. It shows which tools got called immediately before a failure, the kind of precursor pattern that starts to predict red builds before they happen. And it shows whether a given failure signature is showing up across multiple agents, which points to a systemic issue, or is isolated to one harness configuration, which points to something narrower.

The gating strategy that actually holds up blocks on eval score regression rather than on any single run failing. That's the difference between a CI gate that's flaky and annoying, and one that's actually carrying signal. And the payoff isn't just cleaner builds: routing routine, well-understood failures to an automated fix with human approval at merge time frees up human judgment for the failures that actually need it.

Using transcript patterns to identify dead skills and tune the agent harness

The harness, the scaffolding that decides which tools an agent can see, how those tools get described, and what extra context rides along with each observation, is widely recognized as one of the biggest levers on agent performance. Research on harness configurations has found substantial accuracy gaps across different setups running the same underlying model. That's not a rounding error. That's the harness deciding the outcome more than the model does.

Skill usage distribution data, pulled straight from transcripts, is what makes tuning that harness an evidence-based exercise instead of a guessing game. Some tools get called often and correlate strongly with successful outcomes; those are the load-bearing skills to invest further in. Others get called often but show no real correlation with success, candidates for removal or a redesign. And some tools never get called at all, dead weight sitting in the action space, doing nothing but making every decision the agent makes slightly harder.

Removing skills matters just as much as adding them, maybe more. A bloated skill set forces the model to reason over a larger action space at every single step, and that has a real cost even when the extra tools are individually harmless. The LIFE-HARNESS paper from May 2026, "Adapting the Interface, Not the Model," showed that changing the runtime interface alone, with no change to model weights at all, changes agent behavior measurably. The same logic runs in reverse: trimming the interface changes behavior too, often for the better.

Transcripts also reveal a quieter failure mode in tool environment shift. Tools get updated over time, their invocation protocols change while the underlying function stays the same, and transcripts show exactly where an agent is still calling a tool the old way. That looks like agent error on the surface. It's really a harness-maintenance gap. Kim et al. Kim et al. found that under a harness with minimal design effort, post-training performance drops sharply as tool environment shift gets stronger, which means transcript analysis of harness behavior is, in effect, a leading indicator for post-training outcomes.

Stall patterns tell their own story. Repeated tool calls with no change in state mean the agent is stuck and the harness isn't giving it a clean way out. Backtracking to an earlier step right after a tool failure points to missing error-handling scaffolding. METR's time-horizon research put the correlation between task length and agent success rate at R² = 0.83. Stall patterns are most predictive, and most costly, on exactly the longest tasks.

Turning the pipeline into a continuous feedback loop rather than a periodic audit

Tracing and evaluation answer two different questions. Tracing tells a team what the agent did. Evaluation tells them whether that was correct. The pipelines that actually hold up over time share a single data layer between production traces and offline evaluation cases, rather than treating them as separate systems that occasionally sync.

That single layer is what lets failing live scorers grow the golden dataset automatically. Every production run that trips a known bad signature becomes a new test case without anyone manually triaging it into the set. METR framed its own transcript analysis prototype, the 5,305-transcript effort from February 2026, explicitly as a cheaper substitute for running full human uplift studies. That framing matters: the feedback loop doesn't just improve signal quality, it drives down the cost of measurement as the agent fleet grows, which is the only way this kind of analysis stays viable at scale.

A loop running continuously catches things a single snapshot never will. Drift appears in agent behavior shifting across model updates without a matching change to the harness. Regressions appear too, in a tuning change that helps one cluster of skills while quietly degrading another. And emergent stall patterns appear once volume hits a certain scale or once a particular harness combination meets a particular task type.

The infrastructure behind this works best when it's built from composable pieces rather than one monolithic system, purpose-built components for ingesting transcripts, running graders, and feeding results back into the eval set, wired together rather than welded into a single black box. That composability is what lets a team swap out a grader, retune a harness, or expand a gold set without tearing down the whole pipeline to do it. Given how much of the actual engineering work in agent development already is in evaluation rather than in writing new code, that flexibility isn't optional. It's the whole point.