Agent Evaluation Benchmark Design for Software Factory Teams

If you run a software factory, generic coding benchmarks measure the wrong unit of work. A pass/fail score on one issue cannot show you an agent that has to coordinate changes across dozens of files, respond to layered feedback, and behave sanely in a CI pipeline. The result is that teams optimize for leaderboard positions while their actual production pipelines degrade.

Why generic coding benchmarks fail software factory teams

The dominant benchmark design, give an agent a GitHub issue and see if a patch passes tests, was built for LLM code generation. It was never designed for composite agentic systems inside CI pipelines, where shipping depends on whether a correct answer was arrived at soundly. Mainstream coding benchmarks share the same collapse: they fold the full system into a single end-to-end score with no signal at the level of individual components. A system that spends half its context window on useless retrieval is judged exactly as good as one that routes context sensibly, because nothing in the patch result records that difference.

Some benchmarks do provide sub-component breakdowns. LiveCodeBench scores four distinct scenarios, code generation, self-repair, code execution, and test output prediction, at three difficulty levels. BigCodeBench reports separate Complete and Instruct scores across seven domains. These are genuine improvements over pure end-to-end scoring, yet even they do not capture the pipeline-embedded, multi-signal environments where software factory agents actually operate. The gap between a lab task and a factory task is not one of difficulty. It is one of structure.

What factory-scale agentic work actually looks like, versus what benchmarks simulate

Factory-scale agentic development means coordinated, multi-target changes that span dozens of files and multiple feedback loops, and existing benchmarks do not capture that structure. RoadmapBench documents the gap concretely: most benchmarks center on oracle solutions of only dozens of lines of code, while real version upgrades require modifications spanning thousands of lines across 51 files. The task models are different in kind. A single-issue bug fix is a closed problem. A version upgrade is an open set of interdependent changes whose correctness only emerges from their combination.

RoadmapBench grounds this in real open-source version upgrades. Each task starts an agent on a source-version repository snapshot and asks it to implement the full behavior added in the next release, using multi-target roadmaps with a median of five subtasks, each verified by its own test suite, so the score is continuous and weighted, not a binary pass/fail. Factory systems add another structural layer on top of this. Gorinova et al. at Tessl describe two levels: agent harnesses executing single tasks, and orchestration harnesses that decompose higher-level goals, dispatch work, and route outputs through layered feedback loops. Those loops operate at three levels: inner-loop signals such as tests, linting, and type-checking. They include middle-loop signals from policy auditors and reviewer-agent critique. And they include outer-loop signals including PR acceptance, incident rate, revert rate, and customer feedback. A benchmark that stops at the first green test run measures only the shallowest slice of this system.

How outcome-only scoring hides the process failures that actually break pipelines

Even with realistic tasks, if you measure only final correctness, you miss the process that makes a factory reliable. If an agent reaches a correct answer through reckless trial-and-error, it is less reliable in a production pipeline than one that follows sound engineering process, yet every major existing benchmark scores them identically. RigorBench formalizes this gap with a survey of nine major AI coding benchmarks, finding that none evaluate the engineering process through which agents arrive at solutions.

The illustrative case is straightforward. Agent A reads an error trace, forms a hypothesis, writes a targeted fix, adds a regression test, and verifies. Agent B tries five patches in sequence until one passes, with no understanding and no verification. Under every existing benchmark, the two receive identical scores. The difference becomes visible only when the agents enter a pipeline. RigorBench introduces seven process pillars to capture this dimension: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, Atomic Transition Integrity, Test Assertion Density, and Exploration Efficiency, aggregated into a composite RigorScore via weighted sum. For factory teams, the implication is operational rather than academic. An agent running doom-loop retry behavior in a merge queue does not just fail its own task. It burns CI capacity and blocks every PR behind it in the queue.

Why merge queue behavior is the most expensive place to discover an agent's evaluation gaps

When agents are not evaluated on CI behavior and retry discipline, the merge queue is where that gap becomes a pipeline-level failure rather than a single-task failure. An agent that has been evaluated only on outcome correctness can be instructed to get the PR merged and will re-queue the same change repeatedly. Each attempt burns a full CI run and re-triggers cascading failures for every PR behind it in the queue. Retry loops that were mildly wasteful at human frequency become a denial-of-service at agent frequency, and queueing theory is unforgiving when arrival rate approaches CI service capacity.

The production data bears this out. Atlassian's Jira team, with 800-plus developers and more than 300 merges per day, found that without disciplined queue management the merge itself caused semantic failures at a meaningful rate, so they got weekly firefights and constant coordination overhead. Adopting a merge queue brought failures to near-zero and established their first merge SLO. BMW's merge pipeline data shows over 400 changes merged daily, and PRs wait a substantial time before processing begins, plus a further hour of processing time. That queue is already near capacity before agent-generated PR volume is added. The process-discipline argument from the prior section now has a concrete cost attached to it. An agent with high Exploration Efficiency and strong Recovery scores in RigorBench's framework is precisely the agent that does not doom-loop in a production queue, because it knows when to stop probing and when to flag that a task exceeds its reliable operating range. If an evaluation framework omits CI behavior, queue interaction, and retry discipline, it will optimize for the wrong thing and produce agents whose factory impact is actively negative even when their benchmark scores are high.

What benchmark designs are emerging to close the gap between lab tasks and factory conditions

Several benchmark designs published in 2026 address distinct factory-relevant dimensions that single-task outcome benchmarks miss, but no one benchmark covers all of them. That absence is itself a signal for how factory teams should think about evaluation. RoadmapBench addresses long-horizon task fidelity with 115 tasks grounded in real open-source version upgrades, spanning multiple repositories and programming languages, with a multi-target roadmap structure and per-subtask verification. Even the strongest evaluated model, Claude-Opus-4.7, resolves only a minority of tasks, so long-horizon software development remains largely unsolved.

RigorBench addresses process discipline through a set of tasks across five categories, including planning before building, verifying rather than dying, escaping doom loops, knowing when to fold, and keeping the build unbroken. Tasks are scored on the seven process pillars via trajectory analysis of every plan, edit, test invocation, error recovery, and commit. AlphaEval addresses production-groundedness, sourcing tasks from seven companies deploying agents in core business operations and evaluating complete agent products rather than isolated models, using multiple scoring paradigms including LLM-as-judge, reference-driven metrics, formal verification, rubric-based assessment, and automated UI testing. Gorinova et al. at Tessl add the architectural requirement: factory-relevant benchmarks must treat correctness as grounded in independent behavioral specifications rather than single reference solutions, and must expose signal at the level of individual components rather than collapsing everything into one end-to-end score. Trajectory logs, tool calls, intermediate states, and retries, are emerging as primary evaluation artifacts in this framework, not supplementary data.

The four evaluation dimensions factory teams should design benchmarks around

The first dimension is CI behavior and pipeline integration. Evaluation should ask how agents interact with CI: do they surface failures cleanly, avoid retry loops, and pass merge-gate checks without burning excess runner capacity? This maps directly to the merge queue thrashing problem, since an agent that passes a task-correctness benchmark but doom-loops in CI is not a passing agent in factory terms. RigorBench's Recovery Efficiency and Atomic Transition Integrity pillars are the closest existing proxy measures for this dimension. Its Exploration Efficiency and Abstention Quality pillars measure a related question: does the agent know when to stop, when to flag uncertainty, and when a task is outside its reliable operating range?

The second dimension is long-horizon task completion with partial credit. Binary pass/fail is inadequate for multi-target factory tasks, where partial progress must be scored as a continuous value. RoadmapBench's multi-target roadmap structure with per-subtask verification is the design model: each subtask is independently verified and contributes a weighted score, so the evaluation captures where in a long task an agent degrades. The third dimension is process fidelity, not just outcome correctness. Evaluation should examine the trajectory, not only the artifact: did the agent plan before editing, verify before committing, write regression tests, and recover gracefully from errors? RigorBench's trajectory-based scoring framework, which analyzes every plan, edit, test invocation, error recovery, and commit, is the operational model. The fourth dimension is evaluation paradigm composition. AlphaEval's production-grounded multi-paradigm approach, combining rubric-based assessment, formal verification, and LLM-as-judge, shows that no single scoring mechanism is sufficient. Factory teams should compose evaluation paradigms rather than relying on one, the same composable-over-monolithic logic that applies to infrastructure applying to evaluation design.

How agent transcript analysis turns evaluation from a one-time score into a continuous improvement loop

Workflow structure, branching, revision, and stopping rules should be documented so teams can attribute benchmark gains to specific components, design more controlled comparisons, and produce agent evaluations that are reproducible and auditable. Gorinova et al. at Tessl make this argument directly, and its consequence is that trajectory logs, tool calls, intermediate states, and retries, become the primary evaluation artifacts. The final pass/fail is a downstream summary, not the diagnostic input.

In practice, transcript analysis surfaces which skills agents invoke versus which sit idle, where in a long task agents begin to degrade, which recovery patterns succeed and which enter doom loops, and how often agents correctly abstain versus attempt tasks outside their reliable range. The outer-loop signals that Gorinova et al. identify, PR acceptance rate, incident rate, revert rate, and code longevity, become meaningful only when connected back to the trajectory data that explains them. Without transcript analysis, teams see outcomes but cannot act on causes. The transcript is where evaluation stops being a one-time gate and becomes a continuous improvement loop.