Flaky Test Quarantine in High-Throughput Merge Queues

A merge queue exists to keep mainline green: every PR gets tested against the latest code before it lands. A flaky test breaks that promise, and "flaky" is too soft a word for what actually happens. On a feature branch, a flaky test costs one developer an afternoon. Inside a merge queue, that same test stalls every PR sitting behind it, and the cost compounds with each one waiting in line. Most teams still file this under tooling gaps. That diagnosis is wrong: it's architectural, and treating it as anything smaller guarantees it keeps getting worse.

The mechanism is worth being precise about. A standard merge queue tests approved PRs speculatively: each one gets checked not just against current mainline, but against every other PR already ahead of it in line. If PR 4 assumes PRs 1 through 3 have already merged clean, and CI passes, PR 4 clears to go. That assumption is what makes the queue fast, since it doesn't wait for each PR to actually land before starting the next one's checks. But it also means every PR's result depends on a chain of assumptions about the PRs in front of it. Break one link, and everything downstream has to be re-verified from scratch.

That's exactly what a flaky test does when it fires inside the queue. It doesn't just fail one check. It invalidates a chain.

Before merge queues, mainline stability was often worse than teams admit out loud. Uber's iOS mainline was reportedly green only about half the time before the company built a merge queue system, meaning the build was broken about as often as it worked. That's the instability merge queues exist to fix. But Uber's own SubmitQueue then ran into a second, subtler problem: flaky tests caused the queue to keep re-running CI jobs, backing everything up and burning enormous amounts of compute and developer time. Fixing mainline instability introduced a new failure mode stacked right on top of it. That two-layer problem, queue instability solved by a queue that then gets destabilized by flakiness, is the shape of the issue this piece covers.

None of the underlying math has gotten easier. Merge throughput is governed by test suite duration, runner capacity, flake rate, and queue mechanics, and none of those four variables improved just because AI coding agents started opening pull requests at a pace no human team ever sustained. Higher PR volume just means the queue hits its flakiness math faster and more often.

How flakiness propagates through queue mechanics to produce exponential CI waste

Walk through one cascade event, start to finish. A flaky test fires on PR N while it sits in the queue. PR N gets ejected, and its speculative merge state, the assumption that it could sit cleanly on top of everything ahead of it, gets thrown out. Every PR behind N, meaning N+1, N+2, and so on, was tested against a world that included N's changes. Once N is pulled, those speculative states stop being valid. The queue restarts from the ejection point, and every trailing PR re-enters and re-runs its full CI suite.

Each re-run costs real money: suite duration times runner cost, repeated for every PR knocked out of its speculative state. The fix most teams reach for first, just re-run the failed tests, makes things worse, not better, because it doubles the execution cost of every flaky test that fires. Retry overhead alone has been estimated to eat 20 to 30% of total CI compute at organizations dealing with this at scale.

Batching is supposed to fix throughput here: instead of testing PRs one at a time, test five together in a single run and get roughly five times the throughput. It works, right up until one flaky failure hits the batch. Then the queue has to bisect the batch to find the culprit, or eject the whole thing and rebuild it, which erases the throughput gain batching was supposed to deliver. Batching is a bet that flakiness won't show up, and at any real scale, that bet loses. Teams leaning on batching as their primary throughput strategy, with no quarantine underneath it, are building on a foundation that fails exactly when volume makes it matter most. That's not a hypothetical. It's the default outcome once PR count climbs.

At the PR volumes where teams typically decide they need a merge queue at all, a single flaky test can stall the queue for hours. That's not coincidence. It's the exact point where flakiness stops being an annoyance and starts being catastrophic, because the number of downstream PRs caught in any one cascade grows with queue depth.

The scope of this, once actually measured, tends to run bigger than teams expect. Slack has reported a CI failure rate from flaky tests as high as 56.76% before it built dedicated remediation tooling. Atlassian attributed 21% of its master branch build failures to flaky tests and put the annual cost at 150,000 developer hours. Numbers like that aren't outliers. They're what happens when a mechanism built on trustworthy pass/fail signals gets fed signals that are, some fraction of the time, just noise.

The behavioral fallout matters just as much as the compute bill, maybe more. Developers facing a red build they don't trust develop a habit fast: click re-run until it goes green. This dynamic is sometimes called Broken Window Syndrome, borrowing from the theory that visible small failures normalize bigger ones. Once "just re-run it" becomes the default response to red, a real regression gets exactly the same treatment. Nobody investigates, because investigating red has stopped being the norm. That's the quieter cost sitting underneath the compute waste: once CI gets routinely dismissed as noise, the queue's entire reason for existing, the guarantee that only verified code merges, collapses without anyone actually deciding to collapse it.

The cascade math is worth stating plainly, because it's counterintuitive. Cost doesn't scale with how often a test flakes. It's multiplicative. Every ejection invalidates not one PR's work but the speculative state of every PR behind it, so the cost of a single flaky test scales with queue depth at the moment it fires, not with the test's flake rate in isolation. A test that flakes once a month can still be the most expensive test in the suite, if it happens to fire when the queue is fifteen PRs deep.

What makes a test flaky in CI specifically, causes that matter for queue context

Flaky tests don't come from one bug pattern. They come from a handful of recurring mismatches between how tests get written and how CI actually runs them.

The most basic mismatch is the CI-runner gap. A developer's laptop and a CI runner are different machines with different resource profiles: RAM, disk speed, available network ports. A test that passes fine on a sixteen-core workstation can choke on a constrained runner sharing resources with a dozen other jobs. Async timing violations make this worse. A hardcoded sleep() call assumes the application responds at a fixed speed, but under CI load it doesn't, and the test fails not because the code is wrong but because the wait window was too short that particular run.

Shared state causes just as much trouble. Parallel jobs reusing the same ports, file paths, or database instances step on each other, and a test that leaks state from its setup or teardown contaminates whatever runs next. Closely related: non-deterministic test ordering, where a test passes fine in isolation but fails only when it runs after another test that left data behind, an order dependency invisible until parallel or shuffled execution exposes it.

Tests that call out to live external services inherit whatever variability those services have that day: network latency, rate throttling, a partial outage somewhere in a third-party API. Tests built on non-deterministic data, unseeded random generators, hardcoded dates that quietly expire, produce different inputs run to run, some of which land on edge cases the test was never built to handle.

Resource contention adds a wrinkle specific to queues. Memory leaks from earlier runs can slow later tests past their timeout thresholds, and that effect scales directly with queue depth and batch size. This is exactly why these causes bite harder in a merge queue than in an ordinary CI pipeline: parallel speculative execution, running multiple PRs' test suites at once, amplifies precisely the shared-state and resource-contention failures that stay rare or dormant in sequential runs.

None of this is exotic. Roughly 16% of Google's test suite has been reported to show flaky behavior at some point, and a majority of developers, in the neighborhood of 59% by some measures, say they hit flaky tests at least monthly. The causes above aren't edge cases in someone's unusual test suite. They're the default failure modes of running a lot of tests in parallel, under resource constraints, against systems that were never fully deterministic to begin with. Teams that treat flakiness as a code-quality problem to be shamed out of existence are misreading the data. Most of it is environmental, and no amount of stern code review catches a timing bug that only fires under runner contention.

Quarantine as an infrastructure pattern, what it does and what it deliberately does not do

Quarantine has a precise definition here: it pulls a test out of the required-check set without pulling it out of the suite. The test keeps running. Its result keeps getting recorded. A failure from that test just stops being able to block a merge.

That distinction, quarantine versus skip, is the whole architecture, and conflating the two is the single most common mistake teams make when they first adopt this pattern. Skipping a test means it stops running entirely, and the signal disappears: nobody knows if the underlying behavior it checks still works. Quarantine keeps the test executing and reporting, and only strips its authority over the merge decision. That's what makes quarantine an infrastructure-level response to flakiness, not a quiet cut to test coverage dressed up in nicer language.

Quarantine exists for one specific situation, and it's narrower than people assume: a test that's flaky, but where there's no evidence the underlying code is actually broken. Nobody can prove the test wrong, so it shouldn't block merges while someone looks into it, but it also shouldn't get deleted or ignored, because it might still be catching something real.

The boundary matters more than the mechanism. A test failing consistently is usually telling the truth, and quarantining a consistently-failing test automatically should never happen, full stop. Doing so risks hiding a genuine regression behind the same label used for tests that are merely unreliable. That call needs to be made by a person, not inferred by a classifier chasing a pattern in the data.

Quarantine also isn't meant to be permanent, and any team treating it that way has already lost the thread. The pattern only holds up if there's a governance contract behind it: the team that owns the flaky test gets a defined window, enforced by automated tooling, to either fix the root cause or delete the test outright. Without that deadline, quarantine becomes a place where tests go to be forgotten, and the queue slowly piles up dead weight nobody's accountable for.

The reporting side counts just as much as the gating side. Quarantined test failures should route to an asynchronous, non-blocking job, and surface somewhere visible: a health dashboard, a dedicated channel, something a team actually checks. That's what keeps Broken Window Syndrome from sneaking back in through the quarantine mechanism itself. A quarantined test that fails silently, with nobody watching, produces the exact same "nobody looks at red anymore" culture that quarantine was supposed to prevent. The failure has to stay visible. It just stops being a gate.

Put together, this changes what actually happens the moment a flaky test fires inside a merge queue. Without quarantine, that failure ejects the PR, invalidates every speculative state behind it, and kicks off the re-run chain described above. With quarantine in place, the same failure produces a soft result: logged, visible, and the queue keeps moving.

Detection at scale, what reliable flakiness classification actually requires

Quarantine only works if the system knows, with real confidence, which tests are actually flaky. Getting that right is harder than it sounds, and the definition has to be exact: a test is flaky when it's been observed both passing and failing on the same commit hash, with no code changes between runs. That's the actual bar, and detection tooling has to enforce it, not approximate it with something looser like "failed more than once this week."

A single failure proves nothing on its own. Reliable classification needs a history across multiple commits, weighted so recent results count more than old ones. A test that just started failing consistently this week isn't flaky, it's broken, and treating it like a genuinely intermittent test risks quarantining a real regression right when it matters most.

That split creates two categories that call for opposite handling. A test that passes and fails on the same commit is a fair candidate for automatic quarantine. A test that fails consistently, especially in its most recent runs, needs a deliberate, opt-in decision from a human before quarantine, because odds are it's reporting something true.

Two operational metrics decide how much weight any given verdict should carry. Confidence measures how much data the detection system actually has behind a classification: low confidence means the verdict could still flip, so early classifications shouldn't trigger irreversible moves like auto-deleting a test. Impact measures something else entirely, the count of failed executions a given test has actually caused. Sort by impact instead of raw flake rate, and something useful falls out: a small number of tests are usually responsible for most of the red builds a team sees, and those are the ones worth fixing first, regardless of how "flaky" they technically score.

None of this works without good data feeding it. Detection needs the actual failure reports, not just a record of the passing runs, so any CI pipeline that discards or suppresses failed test output before it gets uploaded has effectively disabled its own ability to catch flakiness.

There's also an upstream layer worth building separately from post-merge detection: running new or modified tests multiple times, sometimes five to ten runs, as part of PR checks before they ever get to merge. That catches flaky tests before they enter mainline at all, rather than after they've already started cascading through the queue. Detection, done right, works on two timescales at once: pre-merge, to keep new flakiness out, and post-merge, to catch what's already snuck in.

Scale reveals something else here. Uber's internal system, called Testopedia, has reportedly detected hundreds of flaky tests out of hundreds of thousands in its Go monorepo, a slice well under 1%. But at that scale, a small fraction is still hundreds of tests needing active tracking, fixes, and someone accountable for each one. Flakiness doesn't need to be common to be a real operational burden. It just needs to happen at a scale large enough to multiply.

That scale is also exactly why manually maintained quarantine lists fall apart, and any team still running one by spreadsheet or sticky ticket is already behind. Tests drift in and out of flaky behavior as code changes, as environments shift, as dependencies get upgraded underneath them. A list that depends on someone noticing and filing a ticket is stale by the time anyone reads it, and a queue processing dozens of PRs a day moves faster than manual curation could ever hope to.

How the main quarantine-capable platforms handle this in practice

The tooling landscape splits along one line that actually matters: whether flakiness detection is built into the merge queue itself, so a quarantine decision changes queue behavior in real time, or whether detection lives off in a separate system the queue never talks to. That distinction decides whether quarantine actually solves the cascade problem or just writes a report about it after the damage is done, and it's the question worth asking before adopting any tool in this space, ahead of feature checklists or pricing. Bolt-on flake dashboards without queue integration are a partial fix at best, and teams that pick tooling based on dashboard polish instead of this one architectural question end up paying the cascade cost anyway.

Queue-native detection can spot when a test fails speculatively but then passes further back in the same queue, a strong signal the failure was environmental rather than a real regression, and keep the queue moving on that basis. When detection and queue live in separate systems instead, a flaky failure still ejects the PR and forces the speculation chain to rebuild from scratch, even if some dashboard elsewhere correctly flags the test as flaky. The information exists. It just arrives too late to save the CI cycles already spent.

Some platforms take an ingestion-based approach: pulling in JUnit XML output from whatever test framework is already in use, no changes needed to the test code itself, just an upload step tacked onto the CI job. Auto-quarantine, where it's offered, tends to work as a per-repository setting: once turned on, flaky tests get quarantined automatically as detection finds them, with no human needing to file a ticket first. Quarantined failures then act as soft signals, meaning the overall CI check reports green as long as the only failures came from already-quarantined tests, while the underlying queue configuration doesn't need to change at all.

Command-line interfaces built around this pattern let a test's health get queried directly: healthy, flaky, or broken, returned as something machine-readable. That detail matters more than it looks, because it's what lets an automated agent, human or otherwise, decide whether to retry a failure or escalate it, without a person manually staring at a dashboard first. Auto-retry, where these systems offer it, is typically scoped narrowly to tests already confirmed flaky, rather than applied across the whole suite, which keeps retry logic from quietly masking a real bug behind a "just try again" default.

Broken-test quarantine, where a test fails consistently rather than intermittently, tends to stay a separate, deliberate, opt-in action across these tools, distinct from the automatic flaky-test path. That separation preserves the line between "this is unreliable" and "this is telling you something is wrong," and it's not a line worth blurring for convenience, no matter how tempting the automation looks on paper.

Elsewhere in the ecosystem, merge queue products that automate the rebase process and guarantee each PR tests against the latest codebase have started building or integrating flakiness detection directly, so intermittently failing tests get quarantined before they block a PR, and parallel CI runs can test multiple PRs at once. That parallelism is a real throughput gain, but it comes with a tradeoff worth stating outright: more concurrency only speeds up the queue if flaky tests are already getting quarantined. Otherwise higher parallelism just means more simultaneous chances for a shared-state flake to fire, and teams chasing parallelism numbers without quarantine in place are optimizing the wrong variable entirely.

The broader landscape as of 2026 breaks into a few distinct layers. CI-native test intelligence features sit built directly into major CI platforms and native merge queue tooling. Dedicated flake-management platforms are built specifically around detection and quarantine workflows. Test observability platforms correlate flaky failures with underlying infrastructure metrics, useful when the root cause turns out to be the runner environment rather than the test itself. Framework-level retry and isolation tools sit one layer down, handling individual test frameworks rather than queue-wide behavior. A newer category of AI-driven root-cause tools sorts failures by likely cause (timing issues, environment mismatches, network flakiness, bad assertions) and ranks tests by failure rate and time impact instead of raw count, which pushes teams toward fixing the tests actually costing the most CI time rather than the ones that merely fail most often.

Some observability tooling built around device and browser testing pushes this further, using test history analysis to tell flaky failures apart from consistently-failing ones and from environment-specific failures, which matters when the flake ties to a particular browser or device configuration rather than the code.

Across all of it, the dividing line holds steady. Detection that talks to the queue changes what happens the moment a flaky test fires. Detection that doesn't just documents the damage after the cascade has already run its course.

Implementing quarantine without degrading confidence in the overall test signal

Quarantine solves a real problem, but implemented carelessly it introduces a new one: a merge gate that quietly stops meaning what everyone still assumes it means. Getting this right means treating quarantine as a governed process, not a switch flipped once and forgotten. Teams that skip the governance and just flip the switch are the ones who end up, six months later, with a quarantine list nobody remembers building.

That defined window matters more than it looks like it should. Without a hard deadline, quarantined tests pile up indefinitely, and a growing quarantine list is itself a warning sign: either the test suite has systemic reliability problems, or teams are using quarantine to dodge fixing tests they don't want to deal with. Either way, that list needs an owner and a clock, not just a shelf to sit on indefinitely.

Confidence thresholds need to gate automatic action, not just inform it after the fact. A test with only two or three observed runs shouldn't get auto-quarantined with the same certainty as one with a hundred runs across multiple commits, because the verdict on a thin data set can still flip on the next run. Systems that quarantine aggressively on early, low-confidence signals risk burying real regressions under the same label used for genuinely unreliable tests, which is exactly the failure mode the flaky-versus-broken distinction was built to prevent.

Visibility has to survive the move into quarantine, not evaporate the moment a test stops gating. A quarantined test that fails silently, its result buried somewhere nobody checks, has effectively been skipped in every way that matters, even though technically it's still running. The health dashboards, Slack-style notifications, and API-exposed test statistics some platforms have added are worth taking seriously for exactly this reason: they're what stops quarantine from becoming skip by another name.

The line between flaky and broken deserves to stay a line a human crosses on purpose, not a default some classifier reaches on its own. Auto-quarantine for confirmed flaky tests is defensible, and at queue scale, close to necessary. Auto-quarantine for consistently-failing tests is a different decision entirely, one that risks laundering a real regression into a category built for tests that were never actually wrong to begin with. Any platform that blurs this distinction for the sake of a cleaner automation story is selling convenience at the expense of the exact guarantee the merge queue exists to provide, and that's a trade worth refusing even when the automation looks impressive in a demo. Some infrastructure tools built for AI-driven pipelines, like Vex with its agentic merge queue, are designed around this constraint from the start rather than adding quarantine governance as an afterthought.

Handled this way, quarantine doesn't weaken the merge queue's guarantee. It protects it, by making sure the only thing gating a merge is a test the team has real reason to trust, while everything else keeps reporting honestly in the background until someone fixes it, or finally deletes it for good.