Dead Skill Detection in AI Coding Agent Harnesses

Skills are the dependency layer of an AI coding agent's harness: SKILL.md files that bundle natural-language instructions, executable scripts, and tool configurations, loaded into context only when a user request semantically matches the skill's description. That conditional loading is the whole point, and it's also why a dead skill can sit in a catalog for months doing nothing, or worse, doing something quietly wrong, without anyone noticing. Detecting them requires reading session transcripts systematically, not periodic manual review and not gut feel.

What a dead skill is, and why it is harder to spot than dead code

Dead code has a long research history. Studies across open-source Java projects have found close to 16% of methods effectively dead, meaning unreachable in practice despite compiling and passing tests. Separate analysis of tens of thousands of web pages found a median of around 70% of JavaScript functions on a page go unused, and stripping them cut payload size by as much as 60%. These numbers aren't a direct proxy for skill catalogs, but they're a reasonable baseline for how much dead surface area accumulates in any system that grows by addition and rarely subtracts.

Dead skills come in more than one flavor, and conflating them is where most detection efforts go wrong.

The first is never-matched: the skill sits in the catalog, but its semantic trigger never fires because nobody's real request lines up with the description written for it. The second, and more dangerous, is matched but inert. Published analysis of Vercel's evaluation suite describes a Next.js documentation skill that went uninvoked in 56% of eval cases. A skill that never reliably reaches the agent is a different problem than one that never wakes up.

There's a third failure mode that's easy to miss entirely: proxy invocation. A session digest from the bdfinst/agentic-dev-team repository showed a "test-improve" skill with the heaviest edit churn of any skill in the whole sample, yet it appeared on the never-invoked list. The actual work was happening, just not through the skill's own path. Counting invocations alone would have called this one dead. It wasn't; the trigger conditions were broken, but the underlying capability was very much alive, routed around its own front door.

None of this is visible the way dead code is visible. Static analysis catches unreachable functions before a program runs. A dead skill only reveals itself across enough live sessions to establish a pattern, which makes it closer to silent data corruption than to a compiler warning: the system keeps producing plausible output, nothing crashes, and the rot just sits there compounding. Infrequent isn't the same as dead. A skill built for incident response or a major version migration can go quiet for a long stretch and then be the exact skill that resolves the one incident where it's needed. Any detection method that can't tell rare from dead will end up deleting things it shouldn't.

How dead skills degrade agent performance in ways that are easy to misattribute

A bloated skill catalog costs more than storage. Every skill in the catalog is a candidate for semantic matching, and even the ones that never load still add overhead to that matching process, competing for relevance signal against the skills that should be loading. Then there's the subtler cost: skills that occasionally match on a weak or overly broad trigger inject irrelevant procedure into a session, pulling the agent toward instructions that don't actually apply to the task at hand. That's noise in the orchestration layer, and it looks, from the outside, exactly like a model having an off day.

Which is the real danger. When a harness underperforms, teams tend to reach for a model upgrade or a prompt rewrite before they think to audit the skill catalog, because the skill layer is invisible during normal work. On Terminal-Bench 2.0, a third-party harness (Letta Code) scored 59.1% against Claude Code's 41.6% on the same underlying model. Sebastian Raschka's read on that gap was that harness quality "can often be the distinguishing factor that makes one LLM work better than another," which means teams evaluating model quality are frequently evaluating harness quality without knowing it. A catalog full of dead weight obscures which skills are actually driving good outcomes, so when things go sideways, the fix gets applied to the wrong layer.

Stale state can be worse than no state at all. Anthropic's April 2026 postmortem on a Claude Code quality regression traced the degradation to three separate causes: a reasoning-effort downgrade, a bug where thinking history got silently dropped across stale sessions, and a change to the verbosity system prompt. None of those three was a skill problem specifically, but the pattern generalizes cleanly: stale harness state accumulates quietly and produces exactly the kind of degraded output that gets blamed on the model. Research on the Knowledge Activation framework surveyed 67 engineers and found a mean of 2.6 hours saved per engineer per week under a properly structured, governance-aware skill architecture. Run that number backward and the implication is plain: an ungoverned catalog isn't neutral, it's actively costing time somewhere, even if nobody can point to exactly where.

The observability layer is where detection has to happen

Harness engineering is increasingly treated as its own discipline, following prompt engineering and context engineering as the third phase of AI engineering maturity. Within a production-grade harness, the observability layer sits alongside tool orchestration, and it's the one that matters here: it's where invocation logs live, cross-referenced against everything the orchestration layer actually dispatched.

Session-level tooling, of the kind surveyed across agent harness platforms in 2026, captures full transcripts: tool calls, model invocations, cost, latency, the works. That transcript is the ground truth a static catalog can never provide. A skill listed in the catalog but absent from the transcript record across a wide enough window is a candidate for removal, full stop; the catalog can tell you what exists, but only the transcript can tell you what's actually used.

The same bdfinst/agentic-dev-team session digest that surfaced session digest also flagged 28 agents that were never dispatched in the sampled window, and more than 80 skills that were never invoked at all. Systematic transcript review produced that as a real output. But the digest also shows the limits of detection on its own: it can tell you a skill never fired, it cannot tell you whether that skill should be deleted, re-triggered, or simply left alone as a rarely-needed resource. That judgment call needs something the raw transcript doesn't carry.

That's where a framework like SkillHone comes in. Its focus is on preserving the decision context around a skill after deployment, meaning the diagnoses, the revisions tried, the evidence gathered, the outcomes observed. Without that history, a never-matched dead skill and a rarely-needed valid one look identical in a transcript. Teams that haven't built structured observability into their harness simply can't run this kind of analysis; the detection method is only as reliable as the record it's reading from.

A systematic transcript-driven process for finding dead skill candidates

Start with the observation window. Too short a sample and infrequent-but-valid skills get flagged as dead alongside genuinely dead ones; retained decision context about each skill is what makes a given window defensible rather than arbitrary.

From there, pull the invocation index: for every skill in the catalog, extract total invocation count, the timestamp of last invocation, and the session IDs that triggered it. Zero-invocation skills become the first-pass candidate list, but that list is a starting point, not a verdict.

Next comes the outcome cross-check. For skills that did fire, look at whether their invocation correlates with successful task completion or with failures and retries. A skill that fires reliably but keeps appearing in failed sessions is miscalibrated rather than dead, and that requires adjusting the skill, not deleting it.

Then run the proxy invocation audit, using the proxy-routing pattern as a model. For every never-invoked skill, check whether the work it's supposed to do is happening anyway, through direct model action or through another skill's path. If the answer is yes, the trigger conditions are broken and need repair; the capability itself is fine.

Apply the rarely-needed filter to anything with low but non-zero invocation counts. Is the pattern tied to a genuinely infrequent workflow, like incident response or a major version migration? SkillHone-style decision records help answer that question directly; without them, a rare skill and a dead one produce the same thin invocation trail.

Finally, classify. Remove skills with no invocations, no proxy work, and no documented case for infrequency. Re-trigger skills where the work is happening but not through the skill's own path. Archive with a decision record anything that's valid but rare, so a future audit doesn't have to re-litigate the same question from scratch.

The standard the evidence points toward is a structured eval suite comparing skill-present against skill-absent performance, not an engineer reading through a few transcripts and forming an impression. The discipline is in running evals.

Why intuition and periodic manual audits consistently fail to catch dead skills

Skills only enter context when they're invoked, which means a developer working alongside the agent never sees the skills that were considered and passed over. That invisibility is structural. There's nothing to notice, because the thing that would tell you a skill is dead never appears on screen.

Periodic manual audits run into the same wall from a different angle. They tend to review skill content, asking whether a SKILL.md reads as useful, rather than checking whether it's actually being used. Reading Vercel's Next.js skill file would never have surfaced the 56% non-invocation rate; only running the eval did that. Content review and usage evidence are simply answering different questions, and only one of them is the question that matters.

Then there's survivorship bias. Teams remember the sessions where a skill clearly helped. Nobody keeps a running memory of the sessions where a dead skill fired quietly and injected noise, or the sessions where it should have fired and silently didn't. That asymmetry means intuition will always overestimate a catalog's health.

The underlying issue can be framed as advisory versus deterministic enforcement. Telling an agent to "follow coding standards" in a prompt is probabilistic, a suggestion the model might or might not honor; wiring a linter that blocks the pull request outright is deterministic. The same gap sits between a line in a SKILL.md that says "deprecated" and an automated gate that actually acts on invocation telemetry. Some production skill libraries, like one widely used agent-skills project, already encode deprecation-aware workflows as skills in their own right, distinguishing compulsory from advisory deprecation. That's a genuine step forward, but it solves how to respond, not how to detect. It gives you a procedure to run once a dead skill has been found; it does nothing to find it.

At the scale the ecosystem has reached, manual review isn't a viable long-term strategy regardless. One platform alone reported a large number of skills, amounting to several times the count of repositories, spread across fewer than 9,000 repositories as of January 2026. Nobody is reading through that by hand on a quarterly basis.

Which skill should replace the removed one's slot, and how to handle the decision record

Deleting a skill without documenting why invites a specific failure: a future team member sees the capability described somewhere, decides it sounds useful, and re-adds it without knowing it was already tried and cut. That's the skill-layer version of reverting a dead-code removal because the function name sounds important. The fix isn't caution about deleting, it's discipline about recording.

SkillHone's decision-history format is built for exactly this handoff: preserve the diagnoses, the revisions tried, the evidence gathered, and the outcomes that led to removal, so the next person inherits the reasoning along with the decision. Removing a skill without that record just resets the clock until someone reintroduces the same dead weight.

The Knowledge Activation framework's model is useful here too. It treats units of knowledge as nodes in a composable graph rather than standalone entries, which reframes what removal should actually look like: not a void where a skill used to be, but a rerouted relationship, with the surviving skills that cover adjacent territory absorbing the work the dead one never really did. If a removed skill encoded a genuine workflow that still matters, that knowledge has to land somewhere: either folded into the AGENTS.md file for always-available context, or consolidated into a surviving skill with overlapping scope.

There's a real upside to trimming beyond just cleanliness. Those 80-plus never-invoked skills from the bdfinst digest represent context budget that could go toward the skills actually driving outcomes. Removal isn't only about cutting noise, it sharpens the signal from whatever's left. A skill catalog assembled deliberately, one skill at a time for a specific need, deserves the same deliberateness when it's time to take something back out. It shouldn't function as an append-only log that nobody's willing to edit.

Building a continuous dead skill detection practice rather than a one-time cleanup

Catalogs aren't shrinking, and the underlying activity generating them keeps accelerating. Within projects tracked in the World of Code V2604 snapshot that had at least one AI-attributed commit, AI-attributed commit share grew from 1.6% of non-bot activity in December 2025 to 6.7% by March 2026. At that pace, a skill catalog is a moving target that has to be watched continuously rather than cleaned once with the ticket closed.

Cadence should scale with velocity rather than follow a fixed calendar. High-velocity teams running frequent agent-authored pull requests and adding to their catalogs often benefit from rolling transcript review tied directly into CI pipeline instrumentation. Moderate-velocity teams can anchor to a quarterly invocation audit, using the same observation-window discipline described earlier so the results are actually defensible. And every team, regardless of pace, needs a trigger-based review whenever the harness itself changes meaningfully: a new agent version, a model upgrade, a significant shift in workflow. Any of those can silently change which skills get matched and which stop firing.

None of it works without instrumentation in place first. A team not capturing session transcripts with tool-call-level detail has no ground truth to check candidates against, and stands up the observability layer as a prerequisite, not a nice-to-have added later. projections that 40% of enterprise applications will carry task-specific AI agents by the end of 2026, up from under 5% in 2025, points toward a lot more agent workflows arriving soon, each dragging its own skill set into the shared catalog. The teams that build the transcript record now will have something to audit later. The teams that don't will be doing forensics blind.