Skill Set Bloat and AI Agent Performance Degradation

When an AI agent's performance starts to slip as its skill library grows, the first instinct among engineering teams is to blame the context window: too much loaded at once, too much noise for the model to sort through. The research says this instinct points at the wrong target. A Databricks study by Song and Wei split agent degradation into two separate effects, skill shadowing and context overhead, and it measured each one on its own terms. Skill shadowing happens when an agent picks the wrong skill more often simply because the library has gotten bigger, and the study found this effect grows with library size and drives most of the performance drop. Context overhead, the effect of a stuffed context window degrading execution even when the right skill gets picked, turned out to be small and statistically indistinguishable from zero. That distinction changes what an engineering team should actually do about it. Shrinking prompts or upgrading to a model with a bigger context window leaves the real failure untouched, because the failure happens earlier, at the moment the agent decides which skill to use.

How skills differ from tools

Tools break one way and skills break another, so when shadowing hits, it does more damage than a bad tool call ever could. A tool call that goes wrong tends to announce itself: a type mismatch, a missing parameter, an API error code, because a typed schema limits how badly things can go off the rails. A skill selected in error doesn't trip any such alarm. It injects natural-language reasoning straight into the agent's working context, and the model treats that reasoning as legitimate guidance, not as something to question. Nothing in the pipeline flags it, because nothing about it looks malformed. It reads like advice, because it is written like advice, and the agent acts on it accordingly.

Skills carry broad scope and plain language meant for humans to read, so their descriptions end up blurring into each other. A skill library built for maximum coverage ends up full of descriptions that partially overlap, so a query aimed at one skill can just as easily land on another with a similarly worded summary. Skills are routed on their names and descriptions, the level-1 metadata, but the SkillRouter research found that most of the routing signal lives in the skill body, not the name and description the router actually sees, so the description carries only a fraction of what the router needs, an assumption that breaks down as the library expands.

The deeper problem sits in what the router can actually see. Skills get routed by name and description, the level-1 metadata that sits at the top of the file, but research on skill routing (the SkillRouter work) found that most of the signal a router would need to make a good decision lives further down, in the body of the skill itself. The router never sees that body before it decides. It has to make a high-stakes call on a sliver of the available information, and every new skill added to the library adds another thin, overlapping description competing for the same narrow slice of attention. The mario-coin-counting case, described later, shows how this plays out when two descriptions converge on the same query. The router isn't confused by too much information; it has too little to go on, every time, and that gets worse with every skill added to the shelf.

Degradation compounding as the library grows: the monotonic drop and the abandonment endpoint

Song and Wei's experiment started with a small set of skills already known to help with the task at hand, the oracle set, then added noise skills one batch at a time while tracking the pass rate. The pattern held across two different models: pass rate fell steadily as the library grew, with no plateau and no point where the trend reversed. At 52 skills, the agent lost 8 points of pass rate. At 102 skills, it lost 14. At 202 skills, it lost 21, averaged across the two models tested. The library size drove the decline directly, not task difficulty and not model capability, because the only thing changing between runs was how many skills sat on the shelf competing for selection.

A case from the study, built around a task involving mario coin counting, shows how this mechanism works. The library contained a skill named video-frame-extraction, and its description happened to match the query's wording more closely on the surface than the actual skill meant to help with the task. The agent picked the wrong skill in every single run, across every trajectory tested, because the router had no way to see past the surface match to the skill that would have actually solved the problem.

The most consequential finding in the study is what happens when the agent stops trying to route. The fraction of runs where the agent invokes no skill whatsoever and just does the work by hand, skipping the library entirely, rises from a modest share when the skill set is small and useful to more than a third of all runs once the library reaches 202 skills. In a production log, this doesn't look like a routing failure. It looks identical to a case where no relevant skill existed in the first place, so nobody flags it, nobody investigates it, and the library keeps growing while this silent abandonment rate climbs alongside it. A team watching pass rates alone might notice the 21-point drop and still miss the fact that more than a third of their runs have quietly given up on the skill system altogether.

Why teams keep adding skills anyway: the collaboration paradox and the velocity illusion

Skill libraries keep growing because the signals teams rely on to judge agent health point in the wrong direction. Velocity metrics climb as agents take on more tasks, individual skills look fine when reviewed one at a time, and the slow accumulation of degradation gets pinned on the model or on harder tasks rather than on the size of the library doing the routing. Each new skill, viewed in isolation, seems like a reasonable addition. The damage appears only in aggregate figures, which are much harder to trace back to a cause than a single bad skill would be.

Anthropic's 2026 Agentic Coding Trends Report captures a version of this same blind spot at the human level. Engineers say they use AI for roughly 60% of their work and feel meaningful productivity gains from it, but they also say they can fully delegate only a small fraction of tasks without stepping back in to fix something. When perceived gain outpaces actual reliable delegation by this much, you have to wonder how much brittleness the productivity numbers are masking. Teams feel faster. This same routing fragility accumulates in skill libraries, where throughput looks healthy right up until the moment someone checks whether the work being produced is actually sound.

Skill shadowing's propagation into CI/CD pipelines and merge queues

Skill shadowing doesn't stay contained to a single agent run. Once an agent picks the wrong skill, or abandons skill selection and does the work by hand, the result is lower-quality code or a faulty action that still gets submitted into the pipeline, where it consumes CI capacity, produces flaky results, and adds pressure to the merge queue, eating into the gains that justified adopting agents in the first place.

Teams with heavy AI adoption are merging nearly twice as many pull requests as they used to, but their delivery metrics barely budge, because every one of those PRs still has to pass through a review pipeline and merge queue sized for how fast humans used to type, not how fast agents now produce code. Linear reported its test suites nearly quadrupled in 2026 as coding agents sped up how fast the team could ship, so it had to rework its CI setup just to keep feedback loops fast enough to be useful. An agent told to get a pull request merged will simply retry on failure, burning a full CI run each time and retriggering the queue for every PR stacked behind it. At human frequency, that kind of retry loop was a minor inefficiency. At agent frequency, it becomes a capacity problem.

Anthropic's own account of its CI infrastructure makes the scale of this visible: CI job volume grew 25-fold over six months as Claude came to write the majority of the company's code, which forced Anthropic to redesign how it chooses which tests to run on each change. And the root cause sits upstream of the pipeline: agents producing worse output because of skill shadowing generate more failed runs, more retries, and more merge conflicts. Skill library hygiene isn't only a matter of agent accuracy benchmarks; it's a direct input into how much CI and merge queue capacity a team needs to keep running.

Keeping libraries below the shadowing threshold

The interventions that have actually worked share a common logic: stop handing the router the entire library at once, and make skill discovery something that happens dynamically, either by trimming what's always loaded or by fetching the right skill only when the task signals that it's needed.

Progressive disclosure is the structural version of this fix. Loading only the lightweight level-1 metadata at startup, pulling in level-2 runtime detail once a skill is triggered, and reserving level-3 reference material for on-demand lookup keeps the router's decision space small enough that shadowing can't compound the way it does in a flat, fully loaded library, while still preserving each skill's full capability the moment it's genuinely called for.

What doesn't work is adding more runners, more compute, or switching to a model with a larger context window. Those moves treat the symptom, slow or failed runs, but they never touch the selection failure that caused them. Attach a faster engine to a broken steering system, and you still end up in the same ditch, just sooner.

Transcript analysis as a continuous practice for skill auditing

A single cleanup pass through a bloated skill library fixes the problem that exists on the day of the cleanup. It does nothing to stop the library from growing back past the same threshold six months later, because without visibility into which skills an agent actually selects in production, as opposed to which ones it ignores, shadows, or quietly abandons, there's no signal telling a team when the danger line has been crossed again.

The abandonment signal described earlier is the clearest example of why this visibility has to be ongoing rather than periodic. You have to analyze the full agent transcript, tracking what gets invoked across many runs over time, if you want to separate an actual routing failure from a genuine gap in capability. That distinction matters enormously for what a team does next: one calls for trimming or reorganizing the skill library, the other calls for building a new skill.

Transcript analysis also finds two categories of waste that no other method catches as cleanly. It finds the skills sitting in the library that never get selected in any production run, dead weight that still competes for routing attention without ever contributing a capability. And it finds the skills that are being selected, but for the wrong tasks, which is skill shadowing directly visible in the record rather than inferred from a pass-rate drop. At the scale of a software factory, where agents are producing pull requests continuously and CI runs around the clock, the transcript record is the only place where the full pattern of what gets invoked and what gets ignored is actually visible. Teams that skip this analysis are operating without sight of the single most consequential lever they have for agent performance.

The composability principle: why purpose-built, narrow skill libraries outperform monolithic harnesses

Every line of evidence here points toward the same conclusion. The evidence from skill shadowing research, the GitHub and Anthropic interventions, and the AGENTS.md optimization literature all converge on one architectural principle: narrow, composable skill sets assembled on demand outperform monolithic harnesses loaded in full, and the gap widens monotonically as libraries grow.

This isn't a pattern unique to agent skills. You can see the same logic reshaping platform engineering in 2026, where composable internal developer platforms, assembled from best-of-breed components connected by thin integration layers, win out over monolithic internal developer platforms that try to do everything in one package. The underlying argument is identical at both layers: a system built from narrow, well-defined parts connected loosely degrades more gracefully as it scales than one giant system trying to cover every case by itself.

Some will object that trimming a skill library means losing coverage, that fewer skills must mean less capability. The research answers that objection directly. A monolithic AGENTS.md file, loaded in full, degrades agent accuracy to somewhere between 60% and 70%, research has found, while a skill the agent never correctly selects contributes nothing to capability and actively drags down performance by shadowing the skills the agent would otherwise have used correctly. When a skill never gets selected right, you don't weaken the library by removing it. It strengthens it, because every skill left in the router's path is competing for the same narrow slice of attention, and a library of skills that actually get chosen correctly will always outperform a larger one where half the entries exist only to confuse the router.