Skill Set Bloat and AI Agent Performance Degradation

Skill set bloat is a measurable engineering failure, not a vague sense that an agent has gotten sloppier. When an AI agent's skill library grows past a certain point, performance drops in ways that follow a predictable pattern: the agent picks the wrong skill more often, and the sheer bulk of loaded context degrades execution even when it picks correctly. Research from Databricks shows performance falls by up to 21% scaling from a small set of helpful skills to a 202-skill library. The instinct most teams have when they notice something's off, adding more retrieval logic, more disambiguation, more rules, makes the underlying problem worse, not better.

Skill libraries are collections of loadable, natural-language instructions an agent selects from at runtime to handle domain-specific tasks. They differ from typed tool calls in a way that matters more than it first appears. A tool call has a schema: wrong parameter, missing field, malformed type, and the failure surfaces immediately as an error the system can catch. A skill has no such contract. It's a paragraph of prose describing when and how to act, and when the agent loads the wrong one, nothing throws an exception. The agent just proceeds, treating a plausible-but-wrong set of instructions as authoritative. That's the core asymmetry: tool failures are loud, skill failures are quiet.

Libraries grow the way most software grows when nobody's incentivized to prune it. Fix a bug, add a rule. Support a new use case, add a skill. Almost nobody deletes anything, because deletion feels risky in a way that addition doesn't. The library only ever moves in one direction, and that direction turns out to be the direction of decay.

How performance degrades: the two mechanisms in detail

Databricks researchers Hongwen Song and Song (Vinson) Wei, in a paper submitted in May 2026 (arXiv:2605.24050), isolated two distinct failure modes as skill libraries scale. The first is skill shadowing: as the library grows, the retrieval signal for the correct skill gets crowded out by semantically similar neighbors, and the rate of wrong-skill selection rises monotonically with library size. The second is context overhead: even when the correct skill gets selected, a longer, denser context degrades the agent's execution of it.

The two mechanisms are not equally responsible for the damage, and that should reorient how teams think about fixing this. The paper found these two mechanisms are not equally responsible for the damage. Skill shadowing scales with library size and is the primary bottleneck. Context overhead, in their data, stayed small enough to be statistically indistinguishable from zero. In plain terms: the agent is failing because it's grabbing the wrong instruction off the shelf. It's failing because it's grabbing the wrong instruction off the shelf.

The scale of that failure mode isn't gentle. A tool-selection analogy from the broader literature shows accuracy above 90% with fewer than 30 candidate tools, collapsing to 13.6% at 11,100 options. That's not a gradual slope, but a cliff.

A companion benchmark, SameCapRisk-Bench (arXiv:2606.10388), sharpens the picture further with what its authors call the "right family, wrong skill" problem. Retrievers in that study found the correct capability family at a Recall@3 of 0.910 to 0.936, which sounds like strong performance. But the same retrievers co-retrieved a risky sibling skill, one from the same family but with a conflicting contract, at a Harmful Sibling Rate (HSR@3) of 0.386 to 0.402. High recall, in other words, does not mean safe retrieval. Across four conflict types the benchmark tracks, resource conflicts (stale location data), procedure conflicts (a step meant for a different context), and applicability or output contract mismatches, 95.0% to 95.7% of the "helpful" top-three results also contained the conflicting sibling sitting right next to it.

The behavioral consequence is straightforward: conflicting or redundant instructions don't cause the model to throw an error, they just increase output variance. The model makes a judgment call. That call is unpredictable, and unpredictability at the instruction layer is what a production agent can't afford. There's a secondary cost too, more mechanical: every bloated skill file eats into the token budget that should be going toward the actual task, the current codebase, or the conversation history.

The quantified cost: how much performance a large library destroys

Put the 21% figure from Databricks in a real operating context. That's the difference between an agent that completes tasks reliably and one that fails roughly one time in five more often than it used to, not an abstract benchmark number. For a CI agent or a merge-queue agent running hundreds of operations a day, that's not a rounding error buried in a dashboard. That's a visible reliability regression that someone on the team is going to have to explain.

The more uncomfortable finding comes from SkillsBench (arXiv:2605.19576, May 2026), and it cuts against the instinct to just build more skills faster. LLM-authored skills, generated automatically without human review, delivered a 0.0 percentage point improvement over a no-skill baseline. Human-curated skills, by contrast, delivered a 16.2 percentage point improvement. The gap between those numbers isn't about quantity, it's entirely about source and curation. More skills does not mean a smarter agent. It can just as easily mean a slower one that performs identically to having no skills.

The same research direction goes a step further: low-quality skills don't just fail to help, they can degrade performance below the no-skill floor, and a team that's auto-generating skills and never pruning them isn't treading water. A team that's auto-generating skills and never pruning them is very possibly performing worse than if it had shipped no skill library at all, while still paying the full context and latency cost of maintaining one. It's very possibly performing worse than if it had shipped no skill library at all, while still paying the full context and latency cost of maintaining one.

A 2026 survey covering more than 20 self-evolving skill library systems (arXiv:2605.19576) found that lifecycle management, versioning, conflict detection, deprecation, is "largely neglected" across nearly all of them. Libraries grow. Agents, on the whole, do not get better.

Context rot and prompt bloat: what degradation looks like in practice

MindStudio's analysis of Claude Code skill files coined a useful term for this: context rot, the gradual degradation in agent performance caused by skill files that have grown too large, too dense, or too cluttered with information the agent doesn't need for the task in front of it. It's a name developers landed on independently because they kept running into the same thing and needed a word for it. It's a name developers landed on independently because they kept running into the same thing and needed a word for it.

Prompt bloat is the sibling concept, and it's counterintuitive in a way that trips up a lot of teams: the more you stuff into a system prompt, the worse the agent tends to perform. A week of iteration can leave a team with a system prompt loaded with conditionals, caveats, and special-case handling, and the agent at the end of that week is less reliable than the one they started with.

The symptom list from MindStudio's harness bloat audit guide identifies symptoms that often get misdiagnosed:

Response quality declines even though nobody touched the model. The agent contradicts itself mid-conversation, following an early instruction and then a later one that conflicts with it. Instructions get silently ignored, especially ones placed lower in the prompt or ones that are redundant with something stated elsewhere. Latency creeps upward, because more tokens mean more processing time and a higher bill per call. Tool or skill calls fire when they shouldn't, invoked speculatively because the descriptions are vague or overlapping. And onboarding a new engineer turns into an archaeology project, digging through commit history to figure out why some instruction is even there.

The trap is what teams do next. These symptoms get read as model limitations, so the fix becomes adding more instructions, more guardrails, more skills to patch the gap. That response accelerates the exact decay it's meant to solve.

There's a mechanical reason Claude Code users in particular should care about file hygiene. CLAUDE.md and skill definition files load into context at the start of every session, they are not retrieved on demand. That means a bloated skill file costs its full token count on every single inference, whether or not the agent needs it that turn. File size in that architecture isn't a style preference, it's a direct operating cost. And there's the "lost in the middle" effect layered on top: information placed in the middle of a long context gets recalled less reliably than content at the beginning or end, so a critical instruction buried mid-file is systematically underweighted, quietly, without any error to flag it.

Why smarter retrieval on top of a bloated library does not solve the problem

The obvious fix, when a team notices skill selection failures, is to add retrieval-based pre-filtering, sharpen the descriptions to reduce ambiguity, or build a learned router on top. The research on skill library failure modes acknowledges these as mitigations worth exploring. But mitigation isn't cure, and the SameCapRisk-Bench numbers show why.

Four public neural retrievers in that benchmark hit Recall@3 between 0.910 and 0.936, genuinely strong at finding the right capability family. Their Harmful Selection Rate at the same cutoff stayed at 0.386 to 0.402. That means even with good retrievers, a risky sibling skill rides along with the correct one more than a third of the time. Test against a more realistic, larger public skill pool with a single query and the number gets worse: HSR@3 climbs to 0.808 to 0.845. The risky sibling shows up in the top three more than 80% of the time. Retrieval is excellent at matching capability. It has no mechanism for discriminating between two skills from the same family whose contracts, resource assumptions, procedural steps, applicability conditions, actually conflict.

The skill library research (arXiv:2605.19576) closes the loop on why this isn't fixable by retrieval alone: as the library grows without pruning, retrieval quality falls. As the library grows, retrieval quality falls. So the project of building a smarter retriever is a project of running uphill against a hill that keeps getting taller.

A cleaner way to picture it: a better search engine pointed at a landfill still returns landfill results. The volume of low-quality material is the problem, and no amount of search sophistication changes what's actually in the pile. That doesn't mean retrieval improvements are worthless; they're a legitimate complement once the pile has been reduced. But they're a complement to pruning, not a replacement for it, and treating them as a replacement is how teams end up building increasingly elaborate routing logic around a library that keeps getting worse underneath it.

Running a harness audit

Deletion feels risky. The research is blunt about the fact that it's less risky than the alternative: keeping dead weight in the harness carries a higher cost than removing something and being wrong about it. The fix for that anxiety is documentation. Note what gets removed and why, and rolling back becomes trivial if something breaks.

MindStudio's guide describes a full harness audit as going through every one of these components on purpose: the system prompt, few-shot examples, tool definitions and function calls, knowledge documents and retrieval context, guardrails and rules, memory and conversation history, and output format instructions. None of these get a pass just because they've been there a while.

Five passes make up the audit. First, a contradiction sweep: find instructions that fight each other. A system prompt that says "always be concise" in one section and "provide detailed explanations with examples" in another isn't giving the model two preferences, it's giving the model an unresolvable conflict, and the model will resolve it unpredictably every time. One of those instructions has to go.

Second, a redundancy sweep: find the same intent stated three different ways in three different places and collapse it to one canonical version. Third, a relevance sweep: flag anything, instructions, skills, knowledge documents, that was built for a feature or workflow that no longer exists. These are the easiest deletions in the whole audit, and usually the ones nobody's gotten around to.

Fourth, a skill quality pass. Given that SkillsBench found LLM-authored skills deliver a flat 0.0 percentage points of improvement on average, any skill generated without human review deserves to be treated as unverified until it's actually tested against outcomes. Human-curated, outcome-tested skills are where the library's real value sits, and those deserve protection rather than identical treatment to everything else. Sound lifecycle management means versioning, conflict detection, and explicit deprecation, not just an ever-growing pile with an add button.

Fifth, a same-capability family check, using the SameCapRisk-Bench taxonomy as the mental model. Where the library holds multiple skills covering overlapping ground, check specifically for resource conflicts (stale data one skill assumes is current), procedure conflicts (a step written for a different context), and applicability or output contract mismatches. These are exactly the conflicts retrieval can't catch on its own.

What comes out the other end should be a versioned, documented library where every entry has a stated purpose, a test result, an owner, and a condition under which it gets retired. Not just a list that only ever gets longer.

How transcript analysis surfaces what static review misses

Reading a skill file tells you what it's supposed to do. It tells you nothing about whether the agent is actually using it correctly, using it when it shouldn't, or ignoring it. That gap is the static review blind spot, and it's why the audit passes above need a second, ongoing instrument sitting alongside them: execution data.

Transcripts expose four things static review structurally cannot. Dead skills, ones that never fire across a meaningful sample of runs, become immediately visible and are the clearest deletion candidates in the whole system. Over-invoked skills, called speculatively in situations they don't belong, point to descriptions that are too vague or too overlapping with something else in the library. Wrong-family selections make the skill shadowing effect concrete: an agent picking a plausible neighbor instead of the actual correct skill, visible in the log even when the final output still looks fine. And reasoning contamination, where a wrong skill got loaded and quietly shaped the agent's reasoning chain even though the output passed a surface check, is close to invisible any other way.

Given that lifecycle management is "largely neglected" across self-evolving skill library systems surveyed in 2026, most teams simply have no instrumentation at this layer. Transcript analysis is the practical way to build that instrumentation without standing up a full governance framework from scratch.

In an agentic CI/CD setup where agents are opening pull requests, running test suites, and managing merge queues, skill invocation patterns are production signals in the same way CI failure rates and deployment error rates are. They belong in the same observability stack, not shuffled off into a separate manual review queue that happens once a quarter. Treated that way, transcript analysis becomes a pruning loop: skills with dead invocation patterns over a rolling window become deprecation candidates, and skills with high same-family collision rates get flagged for disambiguation or consolidation.

MLflow's guide to production agents, published in May 2026, frames the general principle this applies to: observability drives continuous improvement, and drift detection creates the feedback loop that keeps an agent reliable over time. Transcript analysis at the skill layer is that same feedback loop, just aimed at the part of the system most teams have never thought to watch.