5 Software Disciplines That Keep AI Skills From Rotting
Skills are reusable units of agent context, so they can decay like code. Here are five engineering practices that keep them alive.
I have spent a lot of words lately on how to write a good skill: the authoring patterns, the principles that separate a useful skill from a markdown essay, and the collections worth installing. They all answer how to make a skill that works today, but that is the easy half.
As skills move from personal snippets into shared repositories, maintenance becomes the harder problem. A skill quietly decays as models change, APIs move, and infrastructure drifts. Left untouched, it can stop helping and start steering the agent in the wrong direction.
Shared collections already accumulate duplicates, stale instructions, uncertain provenance, and overlapping capabilities. Teams need to know what is installed, where it came from, who maintains it, and what is affected if it is compromised. We learned to fight code rot with linters, tests, scanners, package managers, and monitoring. Skills need the same treatment, from the cheapest offline check to the richest production signal.
1. Skill Linting: Catch Problems Before a Skill Runs
A skill can be broken before anyone runs it: an underspecified description may fail to trigger it, oversized instructions can crowd the context window, required fields may be missing for the target format, and unsafe instructions may omit guardrails. Without an offline check, you discover these problems the expensive way, when the agent misfires in front of a user.
Static analysis inspects a skill without executing it, and it scales from shallow to deep as it does for code. The shallow end is deterministic linting: does the skill have the required fields, valid references, and a description specific enough to be selected? A layer up sits best-practice review, matching it against published authoring guidance for concision, progressive disclosure, and clear trigger cues. The deep end is substance review, with a model reading the skill the way a senior engineer reads a pull request and asking whether it does a sensible thing at all. The rules already exist in guidance and in the skill authoring patterns; the discipline is turning them into repeatable checks.
Related tools.
Field and schema linters that validate skill structure
Best-practice reviewers that score against authoring guidance
Generic LLM reviewers for substance, in the spirit of an automated code review
When to use. Run deterministic schema, reference, and style checks on every change. Schedule model-based substance review according to cost and risk.
The main trade-off. Static analysis can inspect structure, semantics, dependencies, and likely risks, but it cannot observe runtime behavior. A skill can pass those checks and still fail in a real agent, model, tool, and repository environment.
2. Skill Evals: Test How a Skill Behaves
Linting tells you a skill is well formed. It cannot show how the skill behaves on a task. Running representative tasks provides behavioral evidence, though it cannot prove that a skill works universally. Without that evidence, every change is guesswork, the same trap as code with no tests.
A useful, intentionally imperfect analogy is the test pyramid. Skill evals isolate one skill. Project evals combine a repository‘s context and can replay a past pull request. Broader system evals exercise a production-like setup end to end. Any of those scenario tiers can be run across target models when the question is model choice. A smaller model can make an isolated run cheaper, but passing it means only that no regression was observed in those scenarios. Scenario quality, repeated runs, and target-model coverage matter more than a comforting coverage number.
Related tools.
OpenAI’s Codex eval pattern captures a run’s trace and artifacts, scores deterministic checks, and adds schema-constrained model grading for qualitative criteria
Eval harnesses that run an agent through a defined task environment
Scenario extractors that turn a past pull request into a repeatable test
When to use. As soon as a skill is doing something you would be afraid to change. That is the moment you need a test.
The main trade-off. Evals provide behavioral evidence, but environments and judges cost time to build and maintain, and an unvalidated judge can create false confidence.
3. Skill Scanning: Scan Skills Before You Trust Them
Skills are not inert documents. Depending on the client, an agent may follow their instructions, execute bundled scripts, or invoke tools under the harness‘s permissions. That makes third-party skills a supply-chain surface. The risks fall into three broad groups: malicious skills designed to cause harm, negligent skills that omit safety boundaries, and vulnerable skills that expose secrets or unsafe flows. Skills are already part of the attack surface, not merely the next one.
Open-source scanners already cover several layers. NVIDIA SkillSpector and Cisco Skill Scanner combine static detection with optional semantic analysis; SkillWard adds sandbox execution for suspicious skills; Snyk Agent Scan can discover installed agent components as well as inspect an individual skill. These projects use different combinations of signatures, data-flow analysis, dependency checks, model-assisted review, inventory, and sandbox execution. That variety matters because no one technique covers the whole attack surface. Underneath them all is supply-chain security: knowing what you run and where it came from, so a compromised skill is traceable rather than invisible.
Related tools.
Static and semantic scanners integrated into publishing or installation
Sandboxed execution and red-teaming harnesses that probe runtime behavior
Provenance and supply-chain tracking for installed skills
When to use. Before any third-party skill enters your workspace, and continuously on whatever is already installed.
The main trade-off. Scanning can flag known risk classes, but false positives, false negatives, and runtime-only behavior remain. It reduces risk without proving a skill safe.
4. Skill Dependencies: Make Hidden Relationships Explicit
Skills can depend on tools, scripts, packages and, in clients that support programmatic skill invocation, other skills. An incident-response workflow might use one capability to gather logs, another to run root-cause analysis, and a third to file the ticket. The open Agent Skills format defines the skill folder, but it does not standardize dependency declarations or skill-to-skill calls, so many of these relationships remain implicit.
Package managers now exist, so this practice is no longer hypothetical. Microsoft’s Agent Package Manager is one concrete implementation: it uses apm.yml for declarations and apm.lock.yaml for resolved commits and content hashes, with transitive dependency resolution and policy gates. The ecosystem still lacks one universally adopted manager, but the shape is becoming clear: discovery, versioned releases, a project manifest, a lockfile, provenance, and install-time quality and security controls. That is what turns the shared-repository swamp into something a large organization can govern.
Related tools.
Skill registries for discovery and publishing
Project manifests that declare packages and lockfiles that pin resolved versions and hashes
Plugin bundlers that package related skills together
When to use. The moment more than one person shares skills, or one skill starts depending on another.
The main trade-off. A package manager buys governance and traceability at the cost of process friction, and asking a culture that currently copies a markdown file to adopt manifests and version gates is a real adoption hurdle.
5. Skill Observability: Learn From Skills in Production
Offline checks cannot reproduce every production workload. You can lint, test, scan, and version a skill carefully and still miss how it behaves against real users, repositories, and drift. Runtime traces, user feedback, and experiments add another kind of evidence.
As a runtime complement to pre-install scanning, Microsoft’s Agent Governance Toolkit wraps agent tool calls with policy checks and records allow-or-deny decisions in an audit trail. That trail shows what the agent attempted and what policy decided; it does not prove the real-world outcome.
The discipline is to monitor agents at runtime and feed what you learn back into the other four. Observability closes the loop in two directions. It turns representative failures into candidate eval scenarios, so production teaches the test suite. And it clusters recurring behavior that may reveal a missing, misfiring, or redundant skill. Those are hypotheses for review, not automatic deletion decisions: a skill that never fires may have a weak description or cover a rare but important case. This is the move that can turn maintenance from a chore into an optimization loop.
Related tools.
Agent-run logging and policy-decision audit trails across real sessions
Scenario extraction that promotes a production failure into an eval
Trace clustering that proposes skills to add, fix, or review for removal
When to use. Once skills are live in front of real work, and especially before you trust a shared repository at scale.
The main trade-off. Production telemetry is rich but requires runtime instrumentation and access to potentially sensitive agent logs, creating real retention, access, and privacy responsibilities.
The takeaway
These five disciplines borrow established software-engineering practices and point them at a new artifact. If skills are treated as code-like context, they deserve comparable lifecycle controls.
The broader discipline is a context development lifecycle: generate context, evaluate and test it, optimize it, distribute it, consume it safely, observe what happens, and repeat. The five disciplines support that loop; they are not identical to its stages. Guy Podjarny‘s Skills Are the New Code talk was the spark for this post. The treatment here is my synthesis of that lifecycle argument with current specifications, eval practice, security tools, package management, and runtime operations.