CI Failure Triage Inside a Merge Queue

Two PRs pass CI, both get approved, both merge clean without a single line of textual conflict. One renames a function. The other calls that function by its old name. Git sees no overlap, merges both without complaint, and main breaks the moment the second PR lands. This isn't a bug in git or a lapse in code review. It's a structural blind spot in how continuous integration checks code before merge, and the fix most teams reach for first, requiring branches to be up to date before merge, doesn't actually close it.

Across a large sample of merges spanning 477 organizations, a PR that had already passed CI still broke main a small but measurable fraction of the time at teams of 2 to 5 engineers. At teams past 40 engineers, that number climbs substantially: roughly one merge in eight. At that scale, a broken trunk stops being an incident with a postmortem and becomes background noise, a recurring tax on every engineer who pulls from main expecting it to build.

Requiring branches to be up to date works fine if exactly one person merges at a time. With ten PRs in flight, whoever merges first invalidates the other nine's "up to date" status the instant their merge lands, so the check passes at approval time and means nothing by merge time. Manual rebase loops turn into a full-time job for someone, because the actual problem is serialization, and serialization done by hand doesn't scale past a handful of engineers, a pressure point that platforms like Vex, which runs an agentic merge queue for AI-driven pipelines, and CircleCI are each responding to in different ways. CircleCI's 2026 State of Software Delivery report, covering 28 million workflows, shows the consequence at the macro level: throughput up 59% year over year, main-branch success rates down to 70.8% (a five-year low), recovery times up 13%. Teams are shipping faster into a trunk that breaks more often and takes longer to fix. A merge queue exists to close the gap between "CI passed" and "safely merged," and that gap is the right place to start if you want to understand why queues need to do more than just order merges.

What a merge queue actually does to the merge lifecycle

A merge queue sits between "PR approved" and "PR merged." It functions as a gate, not a label, and it operates in four distinct stages.

Entry comes first. A PR that's been approved and passed its own CI run gets checked for eligibility: required checks present, not a draft, branch reasonably current. If it clears those bars, the queue assigns it a position based on priority and arrival time.

Next comes building the test branch, and this is the part that does the real work. The queue constructs a temporary branch made of the latest main, plus every PR ahead of this one in the queue, plus this PR's own changes. That's not a snapshot of what main looks like right now. It's a forecast of what main will look like after this PR merges, which is the exact condition the two-PR renaming example above needed and never got.

Then queue CI runs on that constructed branch, separate from whatever CI ran on the PR in isolation. GitHub Actions fires this through a distinct merge_group event, and any third-party CI system needs to watch branches prefixed gh-readonly-queue/{base_branch} or it'll never see these runs at all. CI that doesn't respond to merge_group events leaves queued PRs without the signal they need to proceed, which looks like a queue malfunction but is actually a misconfigured integration.

Finally, the queue either merges or fails. On success, it merges using whatever strategy the team configured (merge commit, squash, rebase, fast-forward). On failure, the PR gets ejected, main stays untouched, the author gets notified, and every PR still behind it in line gets re-evaluated against the corrected version of what main will look like without the ejected change.

That last step is the actual design insight. A failure doesn't jam the queue the way a bad merge jams a serial process; the offending PR gets removed, and everything behind it revalidates against a combination that no longer includes it. GitHub's native queue, generally available since 2023, exposes knobs for merge method, group size (1 to 100), build concurrency (1 to 100), and check timeout. GitLab's equivalent, merge trains, ships in Premium and Ultimate tiers and exposes a CI_MERGE_REQUEST_EVENT_TYPE=merge_train variable, which teams use to run heavier test suites only during train runs rather than on every single push. When a merge request gets removed from a GitLab train, every pipeline queued behind it restarts, and redundant pipelines get canceled outright.

There's a ceiling built into all of this, though. Serial operation with a 30-minute CI pipeline tops out around 48 PRs a day, full stop. That constraint is exactly why batching and speculative execution exist: without them, a queue is safe but slow, and at high velocity, slow is its own kind of failure.

How agentic development breaks the assumptions CI pipelines were built on

CI pipelines assume a human writes a PR, waits for feedback, and fixes it by hand. Agentic development doesn't work that way, and the assumptions underneath CI crack once the author on the other end isn't a person waiting on a notification.

GitHub's Copilot Coding Agent, launched in May 2025, runs inside GitHub Actions and gets triggered by an issue, a PR comment, or a schedule. It reads the codebase, writes code, runs tests, and opens a PR, and a human reviews the result the same way they'd review any other PR. That's a meaningfully different workflow from autocomplete-style tools: an agent takes a task description like "add rate limiting to our API gateway" and executes the whole chain autonomously, analysis, implementation across multiple modules, test writing, test running, PR submission.

Adoption moved fast. An empirical analysis of over 129,000 public GitHub projects (Robbes et al., 2026) found that 16% to 23% of them had integrated agentic coding tools within months of general availability. Separately, METR's research shows the length of task an AI agent can complete autonomously has been doubling roughly every seven months since 2019, a trajectory that pushes agents from finishing a function today toward finishing a whole feature tomorrow.

The volume problem this creates is concrete, and it's the part most teams underestimate. An agent writes a change, opens a PR, and CI picks it up. If the build fails, the agent pushes a fix and CI runs again. If a test fails, another fix, another run. Each individual iteration is fast, but the agent only discovers a failure after crossing the PR boundary, and multiplied across many agents working in parallel, that turns into bursts. The consequence is predictable: queues built to test one PR at a time face sustained pressure when bursts of agent-generated PRs arrive faster than CI can clear them.

The security dimension is not small, either. Apiiro's analysis from September 2025 found that AI-generated code introduced more than 10,000 new security findings per month by June 2025 across the repositories studied, a tenfold increase from December 2024. Separate research puts the share of AI-generated code containing security vulnerabilities at a substantial fraction of all output. And a large share of companies planning to deploy AI agents have found their existing security tooling was never built to handle autonomous code execution in the first place. That's a structural mismatch between the tools and the thing they're now being asked to watch, not a configuration gap teams can patch around.

Put together, human-in-the-loop review of every CI failure becomes the bottleneck at agent scale. The merge queue is the natural place to automate that triage, because it's the one place in the pipeline that already has the context: which PR failed, what it was tested against, and what changed underneath it.

The three categories of CI failure a queue can distinguish

A standalone CI run tells you pass or fail. A queue, because it holds causally ordered context, can tell you why, and that distinction is the whole basis for triage. Treating all three categories below as one undifferentiated "CI failed" event is the mistake that makes triage impossible before it even starts, and it's the mistake most teams are still making today.

Integration failures make up the first category. The PR's code is correct on its own but conflicts with a change sitting ahead of it in the queue, and that conflict is invisible until the two get tested together. The right response is ejection and a re-queue once the conflicting change lands, not a code fix, because there's nothing wrong with the code to fix.

Regression failures are the second category. Here, the PR itself introduces a genuine defect that only shows up against the current state of main rather than the stale base it was developed against. The right response is ejection paired with author notification that includes the diff context exposing the problem, because without that context the author is debugging blind.

Infrastructure and environmental failures round out the third category: flaky tests, a slow CI runner timing out, a transient network blip inside the test environment. The right response here is retry, not ejection. Treating an environmental failure like a code failure wastes queue throughput and sends the author chasing a bug that doesn't exist.

The cost of misclassifying these runs in opposite directions, and that asymmetry is the whole reason classification matters more than detection. Eject on a flaky failure, and a perfectly valid PR sits stuck waiting for a human to notice and re-queue it. Retry on a genuine regression, and a broken change slips onto main. Between the two, silent ejection is the costlier mistake, because it erodes trust in the queue itself: engineers start manually re-queuing PRs "just in case," which quietly reintroduces the serialization problem the queue exists to remove.

The queue's repeatability is what makes any of this tractable. Running the same constructed test branch twice produces a flakiness signal that a single isolated PR-level CI run simply can't produce, since there's no second run to compare against. Current native tooling doesn't go this far, though. GitHub's queue ejects and rebuilds, but it doesn't classify why a check failed or distinguish a flake from a regression. That work gets left to the author, or to whatever's layered on top.

What systematic triage looks like at each stage of the queue lifecycle

Good triage starts before a PR ever reaches the queue, not after it fails inside one.

At entry, the first gate is eligibility itself: approved, required checks present, not a draft, branch not wildly stale. Beyond that, any test already known to be flaky should be quarantined or suppressed in queue CI before the queue even turns on, since a probabilistic failure has no business ejecting a valid PR. A two-step CI structure helps here too: run fast checks (lint, unit tests, type check) on every push, and only let PRs that clear those reach the queue, where the full and more expensive suite runs. That keeps queue CI budget from getting spent on changes that were always going to fail trivially.

During CI, the queue needs to watch for the difference between a timeout and an outright failure, because those are different signals that deserve different handling, not identical ejection. Speculative execution adds another wrinkle: if a PR in position 2 of a five-PR queue fails, the test branches built for positions 3 through 5 are now invalid, since they were built assuming position 2 would merge cleanly. Rebuilding those branches immediately, rather than waiting for their CI to finish running against a base that's already wrong, is what keeps the queue from wasting cycles on doomed runs. GitLab's CI_MERGE_REQUEST_EVENT_TYPE=merge_train variable lets teams distinguish train runs from ordinary push runs, which teams use to run heavier test suites only during train runs rather than on every push.

At the point of failure, the response has to follow from the classification. An integration failure means ejecting the later PR, not the earlier one, and automatically re-queuing it once the conflicting change merges, with a notification that names the specific upstream change responsible rather than a generic "CI failed." A regression failure means ejecting and notifying with the combined-state diff that actually surfaces the defect, so the author can address it before re-queuing. An environmental failure means retrying rather than immediately ejecting, so that a PR which never had a real defect isn't silently removed from the queue.

Batching adds a layer of scale. Grouping several PRs into a single CI run cuts the total number of runs substantially; at a 30-minute CI pipeline, batches of four meaningfully change the throughput math. But when a batch fails, something has to isolate which PR inside it is actually responsible, and that's what automatic bisection does: split the batch, re-test the halves, narrow down to the culprit. Without bisection, one bad PR strands every other PR riding in the same batch, which defeats the purpose of batching in the first place.

Priority queuing belongs in this same conversation, even though it's easy to treat as a separate feature. A hotfix needs to jump the line; an experimental or low-priority change can wait. A hotfix stuck behind a failing experimental PR isn't a scheduling quirk, it's a triage failure, and any queue design that doesn't treat priority as part of failure response is missing something basic.

Why agent-operated queues are the natural next step, not an optional enhancement

Merge throughput is already outrunning human triage capacity at high-velocity teams, and that gap only widens as agents generate more of the PR volume. At the throughput levels agentic development drives, a human triaging every CI failure by hand consumes time that directly competes with the engineering work generating those PRs in the first place. Treating agent-operated triage as a nice-to-have misreads where the volume is already headed. Full stop: teams that wait for volume to force the issue will be building this under pressure, with an already-backed-up queue, instead of on their own schedule.

The information an agent needs to triage a failure is already sitting inside the queue: which PR failed, what it was tested against, what the diff contains, whether the failure is new or something that's happened before. An agent doesn't need to go find that context, the queue already surfaces it, which is precisely why the queue is the right place to automate the response rather than bolting triage on somewhere else in the pipeline.

An agent can classify a failure immediately against the three-category taxonomy without waiting for someone to open a log file. It can retry environmental failures within seconds of detection, re-queue an integration failure the instant the blocking upstream PR lands, and open a fix PR for a regression, tag it with the failure context, and add it back to the queue, closing the loop without a human handoff in the middle. None of that is speed a human triager can match, no matter how disciplined the on-call rotation is.

Watching how agents actually use their skills inside queue CI workflows, through transcript analysis, becomes the feedback mechanism that tells a team which automated triage steps are firing correctly and which are quietly accumulating retries or misclassifying failures. The transcript is the audit trail. Without it, an automated triage layer is a black box that either works or doesn't, with no way to tell why.

The design principle that falls out of all this is straightforward: humans should intervene on judgment calls, not on mechanical decisions that already have a deterministic answer. A regression that needs an architectural discussion, or a security finding buried in AI-generated code, deserves a human's attention. A flaky test retry does not, and routing it to a person anyway is what burns out the on-call engineer meant to be watching for the failures that actually matter. Cursor Origin, built by the team behind Graphite's stacked-PR merge queue, is aimed at exactly this: queues designed to keep CI green with automated fixes for failing builds, rather than routing every failure to a person. GitLab has moved in a similar direction with CI/CD AI Root Cause Analysis for failed jobs, native AI-assisted diagnosis built directly into pipeline runs. Even platform-native tooling is starting to treat automated triage as core functionality, not an add-on.

Flaky tests as the failure category that defeats triage systems before they start

Flakiness is the queue's own worst enemy, and it's the category that breaks every triage system built without it in mind. A merge queue re-runs tests on every single PR that passes through it, which means a test that fails only occasionally in isolation can block the queue repeatedly at scale, with no code anywhere being actually wrong. Any triage system that can't tell a flake from a regression will eventually eject good code for bad reasons, and that failure mode undermines the entire case for building triage in the first place.

GitHub's native queue has no visibility into what's happening inside CI itself. If tests are flaky, slow, or stuck retrying, the queue can't tell that apart from a genuine failure. The ejection looks identical either way, and the author gets the same unhelpful notification regardless of which one actually happened.

Agent-generated code adds a wrinkle that classic flakiness detection was never built for. AI agents are probabilistic by nature: they produce something closer to "Y-ish" than a clean "Y." Testing that kind of output with a binary assertion like Assert X == Y produces failures that look exactly like flakes on the surface but actually reflect genuine non-determinism in how the agent behaves. That's not a flakiness problem CI can quarantine its way out of. It's a sign the CI contract itself, built around deterministic pass-fail assertions, needs rethinking for a world where some of the code under test was never meant to be deterministic in the first place.

Practically, known flaky tests need to be quarantined before the queue even turns on, since a test in quarantine can't eject a valid PR. Raising the check timeout is not a fix, no matter how tempting it looks: a flake given more time is still a flake, it just fails less often and takes longer to do it. Whatever detects flakiness has to sit inside the queue itself, watching for tests that fail inconsistently across otherwise-identical test branches, because that's the one place in the pipeline with enough repeated context to catch the pattern before it costs another engineer a re-queued PR and an afternoon of confusion.