CI Failure Triage Inside a Merge Queue

A merge queue turns ordinary CI failures into a different category of problem, because the queue's job is to keep main green by testing every pull request against a future state that other pull requests are still building. That structural difference, not any flaw in the tests themselves, is what this piece is about: how teams need to triage failures inside a merge queue, and why classification has to happen before a human ever gets paged.

The structural cause is called a semantic conflict. Git's merge algorithm operates on text: it detects a conflict only when the same lines were modified by two different branches. A function rename in one pull request and a call site update in another touch entirely different lines, so git merges them cleanly and CI passes each branch in isolation against the same stale base. The trunk breaks because neither branch was ever tested against the other's changes combined. Requiring branches to be up to date does not fix this because "up to date" means the branch includes the current tip of main, not the other in-flight pull requests that haven't landed yet. Two branches can each be current with main, pass CI individually, and still produce a broken combination the moment both merge.

Why does main break even when every PR passes CI and there's no git conflict?

A merge queue exists to enforce a single invariant: main is always green. It does this by testing each pull request not against the stale commit where the branch was cut, but against the current tip of main plus everything already queued ahead of it. That speculative construction is what makes the queue powerful, and it's also what makes a queue failure a fundamentally different event from a feature-branch failure.

Queue throughput follows simple, unforgiving arithmetic. If merge-group CI takes 30 minutes end to end, a strictly ordered queue can land at most two pull requests per hour. As arrival rates climb toward that ceiling, wait times don't rise gradually. They climb sharply and then break. That throughput ceiling is set by test suite duration, runner capacity, flake rate, and queue mechanics, none of which changed when AI agents started producing code faster.

This is where the queue's atomicity becomes consequential. A failure on a feature branch costs that branch a re-run. A failure inside the queue costs more, because the queue doesn't just block the pull request that failed: it resets the speculative state that every pull request behind it was tested against. One bad test run doesn't just stall one change. It invalidates the work the queue already did on everyone else's behalf.

Agentic PR volume turning queue failures from expensive to catastrophic

That invalidation problem used to be rare enough to treat as a nuisance. Agentic development pushed it into the center of daily operations. As AI agents push pull request volume past what the queue was built to absorb, the ejection cascade becomes the dominant event the queue has to deal with.

Teams with high AI adoption are merging nearly twice as many pull requests as they used to, and their delivery metrics barely move, because the queue is still sized for how fast a person types, not how fast an agent proposes changes. A single developer directing several agents across parallel worktrees can generate pull requests at a rate untethered from headcount, but the queue's capacity to validate those changes still depends on the same fixed resources: runners, flake rate, and queue mechanics.

The failure mode compounds when an agent is simply told to get a pull request merged. Left to its own devices, it will re-queue the same change repeatedly, burning a full CI run on every attempt and re-triggering the cascade for every pull request sitting behind it. A retry loop that was mildly wasteful when a human kicked it off by hand turns into something close to a denial-of-service attack at the speed an agent operates. Teams that expected autonomous development to simply go faster instead found overloaded test runners, queues backlogged with hundreds of unvetted pull requests, and cloud infrastructure bills jumping into four or five figures overnight.

Agents are always new to the parts of a codebase they haven't touched before, they can regress areas they weren't paying attention to, and by the time an automated failure notification reaches anyone, the agent's working context on that change is already gone. The agent needed to resolve the failure before handing it off, not after.

The ejection cascade and the real cost that per-PR flake-rate arithmetic understates

The ejection cascade is the mechanism that makes all of this expensive rather than merely annoying. One infrastructural failure doesn't cost a single pull request and a single CI run. It costs that pull request plus a forced re-run of every queued entry sitting behind it, and each of those re-runs gets its own fresh shot at hitting some unrelated flake, and each of those unrelated failures resets the queue all over again.

A post-mortem from corvmc.org, dated 2026-09-08, documents this directly. When the leading pull request in the queue was ejected, GitHub reset five queued pull requests behind it (#777, #772, #779, #784, #781) onto a new base and re-ran all of them. One of those, PR #772, had already passed its earlier queue run at 02:57:52. It was only the re-run on the fresh ref, triggered by someone else's ejection, that hit a timing flake in Action.svelte.spec.ts, producing a second ejection that reset the queue again. The failure had nothing to do with #772's own code. It was a byproduct of the first ejection, amplified by the queue's mechanics into a second one.

This is the arithmetic that per-PR flake-rate calculations miss. The number that matters isn't failures per pull request, it's failures per queue-entry re-run, and re-runs are generated continuously by other people's rejections. A rejection landing shortly after another pull request was ejected deserves suspicion on its face: there's a real chance it's downstream of that ejection rather than an independent finding.

The runner-hour cost of the same dynamic appears in a separate case from the Apache DataFusion Comet project, dated 2026-09-15. A single transient job failure at 16:26 UTC left the merge-group run going for another hour, with nine other jobs still executing, producing results the queue was always going to discard once the failure was known. The group of pull requests queued behind that run also had to be rebuilt and rerun from scratch. The issue filed against that project traces the structural cause back to the CI configuration itself: ci.yml has no fail-fast behavior across jobs, and the only required check is an aggregate job that runs with if: always(), reporting only once every upstream job has finished. The queue has no way to learn about a failure until the entire run completes, so it keeps paying for work it already knows is wasted.

Why classification must happen before any human gets paged

These cascades point to a design requirement beyond monitoring. The first question a queue failure raises is whether the pull request caused the failure or whether the infrastructure did, and that question has to be answered before anyone gets paged, because paging first routinely misattributes systemic failures to pull requests and spends a team's limited attention on the wrong events.

Every queue failure falls into one of two classes: PR-caused, where the change itself introduced a regression, or systemic, covering flaky tests, runner outages, dependency fetch errors, or an ejection cascade set off by someone else's failure. Treating a systemic failure as PR-caused produces three distinct kinds of waste: the author spends time debugging code that was never broken, the pull request gets blocked from re-queuing when re-queuing was the correct move all along, and the queue accumulates a false signal suggesting the pull request's quality is worse than it is.

The corvmc.org post-mortem shows how this misattribution happens in practice. The automated guard's comment named a specific set of failing band spec files, but that list was silently truncated, hiding the fact that 250 of 254 tests had actually failed, and the real cause was that the web server never started. Neither pull request under review was at fault: the ejected pull request's own code went on to pass end-to-end tests in five later queue runs stacked on top of it. The guard's list was the artifact that triage actually read, and because it was truncated, it produced the same wrong impression twice, independently, including for a second session reviewing the identical rejection.

Classification-first triage changes the re-queue decision itself. A systemic failure should produce an immediate re-queue recommendation before any developer even opens the pull request. A PR-caused failure should produce a fix-first recommendation that names the specific failing step. Those are two different workflows, and sending a pull request down the wrong one wastes exactly the kind of attention teams can least afford to waste during a cascade.

The structure of a classification-first triage process inside a live queue

A triage process built for a live merge queue has to follow a fixed order: find the first failing step, classify it as PR-caused or systemic, check whether it's downstream of a prior ejection, and only then issue a recommendation to re-queue or fix first, all before a human ever opens the pull request.

The first step matters because not every failing job is the root cause. The job that fails first in execution order is the one that counts, and anything that fails after it may simply be a cascade artifact of that earlier failure.

The second step, classification, relies on a set of concrete signals. Systemic indicators include failures that reproduce across pull requests whose diffs never touch the failing area, runner-level errors such as disk exhaustion, I/O throttling, or rate-limit 429 responses on dependency downloads, web server or environment startup failures, and known flaky tests already flagged by prior statistical tracking. PR-caused indicators point the other way: the failing test exercises code the pull request actually modified, and the failure never showed up in queue runs that didn't include this pull request.

The third step checks ejection context directly. If another pull request was ejected shortly before this run, the failure should be treated as potentially downstream of that ejection until proven otherwise.

All four steps run under a timing constraint that the queue itself imposes. Once the merge-group check reports failure, the queue ejects the pull request and tears down its gh-readonly-queue ref, so a triage process that takes minutes cannot run as a job gated on that failed check. It would be racing the queue's own teardown.

The DataFusion Comet issue also cancels the entire merge-group workflow run the moment any single job fails, so the pull request is dequeued immediately and its runners return to the pool rather than sitting idle for an hour. The queue only needs to know pass or fail, and the first failure already answers that question. Fail-fast behavior on the first job failure is a prerequisite for triage that can actually respond in time. A queue that waits for every job to finish before reporting a failure it already knows about delays both triage and re-queuing by the full length of the run.

Automated triage without holding the queue open

Turning that process into an automated workflow means solving two problems at the same time: running after the queue has already ejected the pull request, so the triage step never holds the queue open, and producing a verdict that's tamper-resistant and always present, even in the case where the model that generated it comes back with nothing usable.

An implementation from the shakenfist project, dated September 2026, solves the timing half of that problem by triggering on workflow_run rather than running as a job inside the CI suite itself. The reasoning is direct: a job gated on the can_merge check having failed is racing the queue, because the moment that check reports failure, the queue ejects the pull request and tears down its gh-readonly-queue ref, and a triage job that takes several minutes would be sitting in exactly that closing window. By running as a separate workflow triggered after the fact, the triage step reads a failure that has already happened rather than trying to catch one mid-collapse. The workflow also never checks out or executes the code it's triaging. It reads the failed run through gh and checks out the default branch instead, which is what makes it safe to hand the workflow a writable token while it's reading an untrusted pull request's diff.

The verdict itself is built to survive a model that fails to produce anything coherent. A document gets written every time: a model that only returns prose, or returns nothing at all, yields a verdict marked unknown along with an attached error, so a triage pass that reached no conclusion remains distinguishable from one that never ran. The envelope around that verdict isn't model output either. The repository, run ID, and pull request number are written from what GitHub itself reported, and those fields overwrite whatever the model happened to put there, so a verdict can never end up filed against the wrong failure. Any tracking issue the model cites has to actually exist and reference the failed run, or it gets dropped from the verdict with the reason recorded alongside the evidence.

A dry run of this system against a real failure classified it as systemic and recommended a re-queue, the same conclusion manual triage reached independently, and the whole pass took about two minutes.

Deterministic code and AI in the triage pipeline

This raises a design decision: where AI belongs in the pipeline and where it doesn't. The most durable triage architectures confine AI to the one stage that actually requires human-like causal reasoning, grouping failures and working out their root cause, and run every other stage on deterministic code. Log cleaning, report assembly, ticket filing, and publishing results don't need a model's judgment, and running them through one just adds a point where the pipeline can produce a confidently wrong answer instead of a correct one. A control point that has to behave the same way every time shouldn't be handed to a system that doesn't. Reserving AI for the one step where judgment is genuinely required, and letting fixed code handle everything that surrounds it, is what keeps a triage pipeline's verdicts something a team can actually trust when the queue is moving fast and the cost of a wrong call is another full cascade.