Benchmarking AI Coding Agent Quality From Transcript Data

SWE-bench Verified is the number every coding-agent vendor puts in its launch post. It's also the number platform engineers should stop treating as a stand-in for their own quality bar. Claude Mythos Preview scores 93.9% on it. Claude Opus 4.7 scores 87.6%. GPT-5.3 Codex is 85%. Real results, on a real benchmark, but a 2025 analysis of the top 30 leaderboard entries found that 19.78% of "solved" cases were semantically wrong: the patch passed tests by coincidence, or through reward hacking, not because the code did what the issue asked. Even the genuinely solved cases come from 2,294 tasks pulled from 12 popular Python repositories, so unless your codebase happens to be one of those 12, the score tells you almost nothing about how the agent handles your actual work.

That gap is what this piece is about. External benchmarks measure a model inside a box someone else built. Transcript data, the full record of what your agent actually did across your actual tasks, is the only benchmark that reflects your harness, your codebase, and your task mix. What follows covers what public benchmarks are good for and where they stop, why the harness deserves more scrutiny than the model, what a transcript captures that no benchmark can, which metrics are worth pulling from that data, and how to turn the exercise into a repeatable practice instead of a post-mortem after a bad sprint.

What public benchmarks measure, and where their scope ends

The benchmark landscape isn't thin. SWE-bench, SWE-bench Pro, Terminal-Bench, Aider Polyglot, GAIA, OSWorld, Tau-Bench, and WebArena each isolate a different slice of the job, and none of them claim to cover the whole thing.

SWE-bench's format is narrow by design: take a real GitHub issue, snapshot the repo at that commit, score whether the generated patch passes the hidden test suite. That's a reasonable test of single-session patch generation. It says almost nothing about multi-file changes, work that spans several sessions, or a long-horizon refactor that takes an engineer three days and six false starts.

SWE-bench Pro tries to close that gap, drawing on 1,865 problems from actively maintained repositories and demanding larger patches with more surrounding context than the original benchmark asks for. That's a real step toward enterprise-scale difficulty, and it would be a stronger one if the dataset held up under scrutiny. A 2026 OpenAI audit raised concerns about roughly 30% of it, citing broken test cases and tests stricter than the issue actually required. A meaningful chunk of the "difficulty" in SWE-bench Pro is dataset noise, not genuine task complexity, and that distinction matters if you're using the score to decide anything.

Terminal-Bench comes at the problem from a different angle: how well does an agent operate inside a sandboxed command line, compiling code, standing up servers, debugging a failure it caused itself. That's closer to how coding agents actually spend their time than a single-patch benchmark. Terminal-Bench 2.1 adds continuous validation, pushing it further toward measuring sustained competence rather than one lucky pass. None of these benchmarks are wrong to run. They're just answering a narrower question than most teams assume when they quote the score in a planning meeting.

Why the harness, not the model, is the variable you can control

A coding agent is a model wrapped in a harness, and the harness is doing most of the operational work. It decides which files the model gets to see, how tool calls get dispatched, what happens when an edit fails, how the prompt cache gets managed, when the loop is allowed to stop. Two teams running the identical model can end up with wildly different agents, because almost none of the behavior that matters day to day comes from the model itself.

The cost data makes this concrete. The same model, run through two different harnesses on the same task, produced a cost spread of 32 times, from $0.07 to $2.26, with code quality that came out nearly identical. Thirty-two times is not a rounding error. That spread comes from the harness deciding how many redundant tool calls happen, how aggressively context gets pruned, how many retries get burned before the agent gives up or succeeds.

Quality moves the same way, and the evidence for it is longitudinal rather than anecdotal. A 2026 study tracked 35 sequential harness releases of Qwen Code on SWE-bench Verified, holding the underlying model fixed across every release. The score still moved meaningfully across those 35 versions, while the model itself stayed unchanged. So the thing platform engineers actually have leverage over, the harness, is also the thing public benchmarks are worst at isolating. SWE-bench reports a single number for "the agent." It can't tell you whether that number came from the model or from the retry logic wrapped around it, and treating the two as interchangeable is where most benchmark-driven decisions go wrong.

What transcript data captures that no external benchmark can

A transcript, sometimes called a trajectory, is the complete session record of an agent run: every tool call, every model response, every retry, every intermediate state, in the order it happened. Raw sequence, nothing smoothed over.

What it captures is specific: your actual codebase, your actual task distribution, your actual harness behavior. Your actual codebase. Your actual task distribution, the real mix of bug fixes, refactors, feature work, and infrastructure changes your team ships in a given month. Your actual harness behavior: which tools got called, in what sequence, how many retries fired, where the agent visibly stalled or looped without progress. And your actual skill inventory in use, meaning which capabilities in the harness got exercised and which ones got invoked but returned nothing useful.

The field is already converging on transcripts as a shared unit of measurement. Letta's "trajectory" library normalizes coding-agent sessions from Claude Code, Codex, Letta Code, and other harnesses into one token-efficient format, built so agents can learn from past sessions instead of starting from zero every time. That design choice is a useful proof point on its own: cross-session memory and analysis can be built on filesystem traces a team already has sitting around, with no new infrastructure required to generate them.

The metrics worth extracting from agent transcripts

Metrics belong at every layer of the harness, not just at the finish line. A task that ends in a merged PR can still have burned three times the retries it should have, and a task that fails outright can still show exactly where the harness broke down. Split the metrics into three loops.

The inner loop covers what the agent did during the run itself. Tool call success rate tracks what fraction of invocations came back with usable output versus an error, a timeout, or an empty result. Retry depth counts how many iterations the agent needed to reach a terminal state, and a high number usually signals context confusion, a tool interface the model doesn't understand well, or a task spec that was ambiguous from the start. Skill utilization tracks which tools in the harness actually got invoked across a batch of sessions, since a skill that never fires is still costing tokens and adding noise to every tool-selection decision the model makes. Context consumption pattern shows where in the window the agent's behavior started to shift, and degradation that appears consistently near the context limit points to a harness design problem, not a model that got worse. Self-correction events track whether the agent caught its own mistakes through linting or test feedback mid-run, or whether the failure only surfaced later, at PR review or in CI.

The middle loop covers the harness's own policy behavior. Policy trigger rate shows how often automated checks stepped in, and at which stage. Revision cycle count tracks how many review-revise loops a task went through before it reached something mergeable. Conflict introduction rate flags whether agent-generated changes created logical conflicts with other work already in flight.

The outer loop ties transcripts back to production outcomes. PR acceptance rate for agent-authored changes, split against human-authored changes and broken out by task type, shows where the agent is pulling real weight versus where it's creating review overhead. Post-merge revert rate is the cleanest signal of the three: it reflects a real-world judgment that the change was wrong, made after the code already looked good enough to ship. CI failure origin, traced backward through the transcript, tells you whether a given failure came from a flaky test, a harness defect, or an actual bug the agent wrote. Three very different problems, and a red CI badge alone can't tell them apart.

Turning transcript metrics into a repeatable benchmarking practice

One engineer reading through session logs after a rough week is a post-mortem, not a benchmark, and post-mortems don't compound the way a repeatable practice does.

A real benchmark needs consistent task sampling, a scoring rubric that doesn't drift from run to run, a baseline to measure against, and a fixed cadence. Skipping any one of those four causes the exercise to slide back into ad hoc log-reading with extra steps.

Start by normalizing the transcripts. Without a shared schema, comparing a session from one harness against a session from another is manual work, and manual comparison doesn't survive contact with a growing team or a growing task volume. This infrastructure doesn't need to be built from scratch: Letta's "trajectory" library already normalizes output from more than 15 harness formats, so the adoption cost is closer to configuration than construction.

Then define the task sample on purpose, not by convenience. A transcript benchmark only means something if the sample reflects the workload the agent actually handles day to day. Stratify by task type, bug fix, refactor, new feature, infrastructure change, and by complexity tier, and watch for the sample quietly drifting toward the easy end over time. Easy tasks make every number look better without making the agent any better, and that drift is the single easiest way to fool yourself with your own data.

How transcript benchmarking surfaces the two highest-leverage improvement actions

Once the metrics run on a real cadence, two improvement actions tend to outweigh everything else, and both do more for agent quality than swapping in a newer model.

Skill pruning comes first, and it's the one teams underrate most. Transcript data shows exactly which tools in the harness get invoked but never produce a usable result across a large batch of sessions. A bloated skill inventory doesn't sit there harmlessly: it inflates context, adds token cost to every run, and introduces noise into how the model chooses which tool to call next. A tool that appears in the harness but never returns a successful call across many sessions is either misconfigured, redundant with something already in the toolkit, or built for tasks the workload simply doesn't generate. Cutting it is a quality improvement, not a capability loss, and transcript frequency data is what makes the case visible, since a skill can look essential on paper while never once earning its keep in practice.

Harness failure diagnosis is the second, and it reframes where the blame usually belongs. The paper "From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws" makes a point that's easy to miss without transcript evidence in front of you: production failures frequently trace back to harness components (context management, tool dispatch, retry logic) rather than to limits in the model's reasoning. The diagnostic pattern is specific and repeatable. When a transcript shows the agent reaching the correct intermediate state and then looping, stalling, or issuing a malformed tool call right after, that's a harness failure signature. The model found the right move. The harness couldn't execute it cleanly. That distinction decides where engineering time should actually go, and no benchmark score, however high it climbs, can draw that line for you.