Human Escalation Policies in Agent-Managed Merge Queues

Merge queues sit at the exact point where AI-generated code volume collides with production risk, and the policies that decide when an agent must stop and ask a human are becoming the real bottleneck in software delivery. The coding part got fast. What happens after the code is written, the testing, the merging, the governance layer between a PR and main, has not caught up, and escalation policy is the mechanism that determines whether that gap gets closed or gets worse.

Developer survey data from 2025 puts the scale of this in plain view: 84% of developers already use AI tools or plan to, and 51% use them daily. That's a nontrivial shift in how code gets written. It's a structural one, and it means the volume of pull requests hitting a merge queue is no longer paced by how fast a person can type. Merge queues were built on the opposite assumption.

What agent-operated queues actually do, and where their decision authority currently stops

A merge queue's job sounds simple: it sits between an approved pull request and the main branch, tests each PR against the state main will actually have at merge time (including whatever's ahead of it in line), and only lets the merge through once CI passes. That serialization is the entire point. It's also why queues stall: at high merge volumes, a single flaky test can jam the line, forcing a cascade of manual re-runs even when nothing is actually broken. The most common reason a queue grinds to a halt at scale is a false alarm mistaken for a real failure. It's noise that looks like one.

Research on agentic CI/CD systems (Barnes, Ghaleb, and Hassan, "From Assistance to Agency") draws a line worth borrowing directly: data-plane authority versus control-plane authority. Data-plane actions are localized, things like generating a patch, rerunning a test, resolving a conflict inside a single PR. Control-plane actions change the rules themselves: pipeline configuration, deployment policy, approval gates, the stuff that redraws an organization's risk boundary. Current systems keep agents almost entirely in the data plane, and that's not caution for its own sake. It's where escalation policy earns its keep, because control-plane changes are exactly the moves a human still has to sign off on.

What happens when that boundary gets fuzzy has already played out in production. In April 2026, GitHub's native merge queue, which constructs the merged state itself rather than just checking it, silently reverted already-shipped code across 2,092 pull requests and 230 repositories, in some cases wiping out thousands of lines of code. That's a real, non-hypothetical edge case. It's a concrete failure that shows native queuing can buckle under agent-heavy load in ways a human-paced workflow never surfaced.

Agent authorship alone does not explain the reliability problem. Data from Mergify's 2026 State of Merge Queues report, drawn from 200,000 merges across 477 organizations, shows AI-assisted PRs broke main about half as often as human-written ones, 1.9% versus 4.4%. Scale, not skill, is the issue. Lower failure rates times a much bigger volume still adds up to more absolute failures moving through the pipe. Barnes and colleagues note three patterns holding across current agentic CI/CD deployments: constrained autonomy as the default design, safety coming from the surrounding governance infrastructure rather than any guarantee baked into the agent itself, and deployment moving faster than anyone's ability to evaluate it. Escalation policy is that governance infrastructure. It's not a fallback for when the agent screws up. It's a designed property of the system, or it should be.

Why confidence scores alone cannot define when to escalate

The obvious first instinct is a threshold: set a confidence cutoff, say 90%, and route anything below it to a human. Some playbooks suggest 60 to 70% as the floor. It reads clean on a whiteboard, and it's wrong in practice.

Large language models are notoriously bad at knowing what they don't know. Models trained with reinforcement learning from human feedback tend to run systematically overconfident, and their stated confidence correlates poorly with whether they're actually right. A model claiming 90% confidence, per one production analysis, can end up closer to 75% accurate in the real world. That gap doesn't stay put, either, it compounds. Chain three agents together, each miscalibrated by something like 15 percentage points, and each individually claiming 90% confidence, and the realistic joint probability that all three steps are correct drops to somewhere around 42%. A merge queue running a planner agent, a conflict-resolution agent, and a CI-triage agent in sequence is exactly this setup. Three confident agents in a row does not mean a confident outcome.

So confidence is the wrong signal to build a policy around, full stop. The right signals are action type, reversibility, and scope, properties you can check without asking the model to grade its own homework. Decagon's framework for escalation calibration puts the tension well: an overly cautious policy floods human queues with work the agent could've handled fine, and an overly loose one leaves the decisions that actually matter without anyone watching. Getting the threshold right is a measurement problem, solved by tracking outcomes over time, not a matter of picking a number that feels safe.

A four-tier action-risk classification for merge queue decisions

If confidence won't do the job, reversibility and scope will. A four-tier model sorts agent actions by what happens if they're wrong, not by how sure the agent claims to be.

Tier 1 is read-only: the agent observes, analyzes, reports, no approval needed anywhere in the loop. Reading queue state, flagging a flakiness pattern, summarizing what's stuck, all fine on autopilot. Tier 2 covers reversible writes, actions the agent can take and undo inside a defined window, with human review happening async and after the fact: reordering the queue, retrying a failed CI run, rebasing against main. Tier 3 is where things leave the agent's own boundary, triggering a deploy, pinging an external system, granting a policy exception, and here escalation has to happen synchronously, before the action, not after. Tier 4 is the ceiling: mandatory human sign-off no matter what the agent's confidence says, covering things like merging to a protected branch under non-standard conditions, touching approval-gate configuration, deleting data, or changing who has access to what.

That maps cleanly onto the data-plane and control-plane split from earlier. Tiers 1 and 2 stay in the data plane. Tiers 3 and 4 cross into control-plane territory, and that's where governance stops belong. Applied to a real queue: the agent handles flaky-test detection and retry on its own (Tier 2), it surfaces a merge conflict but waits for a human to confirm before touching shared infrastructure code (Tier 3), and it never touches the queue's own approval-gate settings without an explicit human sign-off (Tier 4).

One more wrinkle worth building into the thresholds: Mergify's 2026 data shows private repos broke main 4.5 times more often than open-source ones. That's a signal, not noise, and it means the tier boundaries for an internal codebase should sit more conservatively than they would for a public one. Same four tiers, different risk appetite baked in.

Structuring the escalation trigger set: which conditions belong in policy

The tier model sorts actions by risk. The trigger set is what actually fires the escalation, and getting that list right matters as much as the tiers themselves. Research on human-AI collaboration (Safin and Balta) frames the underlying principle well: escalation makes sense exactly when an agent's reliability varies across inputs in ways that can't be specified ahead of time. The trigger set is that principle turned into working rules.

A few categories are worth wiring up directly, drawing on published escalation design frameworks. Action-type triggers cover any Tier 3 or Tier 4 move, unconditional, no confidence score can override it. Scope-breach triggers fire when a PR touches files outside what the agent is supposed to be handling, pipeline configuration, deployment manifests, security policy files. Conflict-complexity triggers kick in when a merge conflict spans more files than some defined limit, or touches code that other in-flight PRs are also modifying at the same time. Loop-detection triggers catch an agent that's retried the same CI failure past a set number of attempts without resolving it, the same logic customer-service escalation systems use to catch a bot stuck in a circle. Novelty triggers fire when a failure doesn't match any pattern the queue has seen before; unknown failure modes aren't agent territory, full stop. And explicit policy-rule triggers route certain file paths, dependency types, or author categories straight to a human, set by security or platform teams and not up for override by anything the agent thinks.

What shouldn't be on that list matters just as much. A PR being AI-authored is not, on its own, a reason to escalate. Mergify's numbers, 1.9% failure for AI-assisted PRs against 4.4% for human-written ones, make that case on their own. Treating authorship as inherently risky defeats the entire purpose of the queue and rebuilds the exact bottleneck it was meant to remove.

These policies also need to live as configuration, not folklore. Best practice recommends version-controlling escalation rules and reviewing them whenever agent capabilities change, since a change in what the agent can actually do changes where the thresholds should sit. A distinction worth keeping straight in a multi-agent queue is that escalation hands a decision to a human, while a lateral handoff routes it to another sub-agent working inside its own defined scope. Some things that look like escalations in a multi-agent setup are really lateral handoffs, and they need their own authority boundaries, not a blanket route to a person.

The handoff package: what a human reviewer needs to act without re-investigating

Escalating to a human only works if the human can actually do something with what lands in front of them. Dumping a raw JSON payload or a full diff on a reviewer produces one of two outcomes: a rubber-stamped approval, or a slow one while the reviewer reconstructs context the agent already had. Neither is the point of having a human in the loop at all.

There's data backing up how bad this gets over time. A 2026 paper, "Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code," reflects how reviewers tend to approve agent-generated code more automatically and scrutinize it less over time. That's a direct warning: a human gate without a well-built context package isn't really a gate, it's a formality that decays the moment reviewers get used to seeing green checkmarks.

A handoff package that actually works needs a handful of things. The trigger, stated in plain language, not an internal rule ID nobody outside the platform team can parse. The proposed action, in a form the reviewer can evaluate without opening three other tools. The risk surface, meaning which files, systems, or downstream processes are actually touched. The agent's reasoning, summarized in plain terms, enough to judge the logic without reading a full transcript. The decision options laid out explicitly, approve, reject, or modify, with what each one actually does. And relevant history: has this exact PR or failure pattern shown up before, and how did it get resolved that time.

A useful distinction is between a warm transfer, where context arrives before the human engages, and a cold one, where the reviewer gets dropped into the queue with nothing and has to start from scratch. Escalation policy should specify which mode applies to which trigger, because not every escalation deserves the same level of ceremony. And the whole thing needs to be built async-first. Synchronous approval runs straight into gateway timeouts, expired tokens, stale cursors, the ordinary mess of production infrastructure, so a durable, state-managed interrupt with idempotency keys is what actually survives contact with a real system.

That matters more than it might seem, because Mergify's 2026 data puts p99 wait times in private queues at roughly 20 hours. A PR parked that long is exactly the case where context rots fastest. The handoff package has to be self-contained enough that a reviewer picking it up nearly a day later can still decide without rebuilding the whole picture from memory or from digging through logs.

Measuring whether the escalation policy is calibrated correctly

One industry analysis puts the pilot-to-production failure rate for AI systems at 88%, and pins the cause on teams relying on observability instead of enforcement, watching what happens instead of setting hard rules about what's allowed to happen. Escalation policy is the enforcement layer. Measuring it is how a team finds out whether it's actually enforcing anything or just generating dashboards nobody acts on.

Escalation rate is the core metric, and Decagon's framework is right to insist it gets tracked by trigger type, not lumped into one aggregate number that hides everything interesting. A high rate on action-type triggers points to thresholds that are miscalibrated, or an agent whose skill set doesn't match what the queue is throwing at it. A high rate on loop-detection points to a recurring CI failure the agent genuinely can't handle, a case for either expanding what the agent knows how to do or writing an explicit policy rule to cover it. A high rate of manual, explicit-override escalations, meaning reviewers stepping in on things the policy never caught, means the policy has holes. And a very low escalation rate isn't automatically good news; it can just as easily mean the agent is quietly making calls it shouldn't own, which calls for an audit, not applause.

Outcome tracking after the fact matters just as much as the rate itself. Did the human approve the agent's proposed action, reject it, or change it? A trigger type that gets rejected consistently means the agent is making the wrong call in that category, and the threshold needs to move. One that gets approved consistently might mean the threshold is too cautious and could loosen. Broken-main rate is the ground-truth check underneath all of it: Mergify's data shows that rate climbing 16 times over as team size grows, and a policy that's actually working should decouple that curve from headcount rather than just track alongside it.

Mergify's 2026 numbers also show 94% of teams still merging one PR at a time instead of batching, which suggests most organizations haven't yet hit the point where escalation policy becomes the thing actually limiting their throughput. For teams that have hit that point, the measurement discipline above is what separates a policy that stays alive from one that quietly rots into a document nobody rereads. Harness-level transcript analysis, watching which skills fire, which trigger escalations, which show up on cases outside what the agent was supposed to handle, feeds straight back into both the policy and the harness itself. That's the loop: not a one-time calibration, but a running check on whether the line between agent authority and human judgment is still drawn in the right place.