Skill Engineering, Part 4: Evolution
This is Part 4 of a six-part series on Skill Engineering. Part 1 — Foundations, Part 2 — Creation, Part 3 — Governance.
Chapter 17 — Skill Rot
A skill that worked yesterday will fail next month.
This is the central observation of Part IV, and it is worth stating plainly before any of the machinery that responds to it. A skill is prose that describes a procedure against a system. The system changes. The procedure changes with it. The prose, unless something keeps it current, does not.
The decay has a name in this book: skill rot. The name is borrowed from “bit rot”, the slow corruption of stored data through environmental degradation. Bit rot is a misnomer in modern storage — the actual mechanism is more often filesystem corruption or hardware failure — but the term captures the right intuition: things that look static decay if nothing tends them. Skill rot is the same intuition, applied to skills that describe systems.
The three causes of rot
Skill rot has three distinct causes. The team that confuses them ends up fixing the wrong thing.
API drift. The system the skill describes changes its API. An endpoint moves; a field is renamed; a response format gains a new wrapper; an authentication mechanism is replaced. The skill’s instructions reference the old shape and produce wrong actions against the new shape.
API drift is the most mechanical and the most attributable cause of rot. The change is concrete; the agent’s error is concrete; the fix is concrete. A skill that fails because the API moved is a skill the team can repair in a single editing session, once they know what changed.
The challenge is that API drift often happens silently. A vendor upgrades the system; nobody on the team is paying attention to the upgrade notes; the skill quietly breaks. The skill’s first failure may be days or weeks after the upgrade, during a routine invocation by a user who has no idea that anything has changed. The user blames the agent; the team investigates and discovers that the system, not the agent, changed.
Environment drift. The skill works correctly against the system, but the user’s environment has changed in a way that breaks the skill’s assumptions. A CLI tool has been upgraded and its flags have changed. A configuration file has been moved. A required network path is now firewalled. A dependency that the skill assumed would be present has been replaced.
Environment drift is harder to attribute than API drift, because the system being addressed is correct. The user’s environment is the problem, and environments are heterogeneous: a skill may work on one user’s machine and fail on another’s, depending on which CLI versions each has installed.
Domain drift. The team’s processes, vocabulary, and judgement criteria change. The system has not changed; the environment has not changed; what has changed is what the team considers the right answer to the user’s question.
A skill that filed bugs against project DC00 last year may need to file them against project DC01 this year, because the team has restructured. A skill that diagnosed data product issues by checking three layers may now need to check four, because a new layer has been added to the architecture. A skill that recommended emergency access for incidents may now need to refuse such recommendations, because the team’s incident-response process has changed.
Domain drift is the slowest of the three causes and the most consequential. It moves at the pace of organisational change, which is slow but irreversible. A skill that was forged against last year’s processes and never updated against this year’s processes is producing actions that the team no longer endorses. The agent is still doing what the skill says; the skill is just no longer right.
The silent failure mode
The most insidious property of skill rot is that it often fails silently. The agent loads the skill, follows the instructions, makes the API calls the skill said to make, and produces a response that looks plausible. The user sees the response, does not have the context to know it is wrong, and moves on.
The wrong response propagates. The user takes action based on it, the action affects something downstream, the downstream failure shows up days later in an unrelated context. By the time the team traces the failure back to its source, the original skill invocation is long forgotten, and the connection between the rot and the eventual harm is hard to establish.
Silent failure is what makes skill rot dangerous in a way that crashing failure is not. A skill that crashes is a skill the team can fix; a skill that quietly produces wrong answers is a skill that erodes trust in the entire library, because users start to second-guess every response.
The machinery in the remaining chapters of Part IV is, in large part, about catching silent failures before they become eroding events. The trace capture (Chapter 18) records what the agent did so that wrong actions are visible. The trace analysis (Chapter 19) distils traces into evidence so that patterns become legible. The evolver (Chapter 20) responds to evidence with proposed changes. The dual-track validation (Chapter 21) ensures that the changes do not introduce new silent failures while fixing existing ones.
The “wait for the user to complain” anti-pattern
The default response to skill rot, in teams that have not built any machinery, is to wait for a user to complain. The user says “this skill stopped working”; the team investigates; the team fixes the skill; the cycle repeats.
This pattern produces a library that is reactively maintained, in a state of permanent partial decay. At any given moment, some non-trivial number of skills have rotted; the team is unaware of the rot because no user has noticed yet; the rot is silently producing wrong answers until somebody complains.
The pattern has two specific failure modes worth naming.
The first is detection lag. The time between the rot occurring and the team learning about it is the time during which the skill is silently producing wrong actions. Detection lag is determined by the frequency of skill use and the alertness of users. A skill used twice a week, by users who do not have the context to notice subtle wrongness, may have a detection lag measured in months.
The second is attribution error. When a user complains about a skill, the user often does not know which skill was responsible for the wrong action. They know the agent gave them a wrong answer; they do not know which skill the agent loaded to produce it. The team has to reverse-engineer the attribution from the trace, if a trace exists. If no trace exists — and most teams do not capture them — the team has to ask the user to reproduce the failure, hope that the failure is reproducible, and watch the agent’s behaviour to identify the skill.
Both failure modes compound the cost of reactive maintenance. The team spends much more time on each user-reported failure than they would spend on a proactively-caught failure, and the team’s library quality is bounded by the patience of its most vigilant users.
Chapter 18 — Trace Capture
The evolution loop runs on evidence. The evidence is the record of what the agent actually did, in the form of session traces — the structured logs of every prompt, every action, every tool call, every response — that the harness produces during normal operation. Without traces, the loop has nothing to reason over.
What a trace contains
A session trace, in the form this book recommends, is a JSON-Lines file in which each line is one event in the agent’s session. The events include:
- User messages. What the user typed.
- Assistant messages. What the agent said in response.
- Tool calls. Each invocation of a tool: the tool’s name, the arguments, the result.
- Skill references. Markers indicating which skills were loaded during the turn.
- Errors. Tool failures, parsing errors, agent self-correction events.
A reasonably long session — twenty user turns, fifty tool calls — produces a trace of a few hundred kilobytes. Across a team of half a dozen users, a month of normal use produces a few hundred megabytes of trace data, which is small by modern standards but large enough that the next chapter’s distillation matters.
The trace is the closest the team can get to a video recording of the agent’s behaviour. Reading a trace tells you what the agent saw, what it decided, and what it did. Aggregating traces across many sessions tells you how the agent behaves in the wild.
Why traces are safe to record
Trace data, in most systems that record agent behaviour, is dangerous. The agent’s context window contains every credential, every personal identifier, every piece of sensitive content that came up during the session. A trace that captures the context window captures all of it, and storing the trace then means storing the credentials.
The zero-trust proxy of Chapter 3 changes this calculus. With the proxy in place, credentials never enter the agent’s context window. They live in the proxy; the agent sees only “make a request to this URL”, and the proxy injects the credential at the moment of the actual HTTP call. The trace captures the agent’s view, which is the credential-free view. Recording the trace is therefore safe by construction.
This is the property that makes Part IV viable. Without it, every team building agents would face a choice between recording everything (and accepting the credential leakage) or recording nothing (and giving up the evolution loop). With it, the choice is moot: traces are recorded, and they contain nothing that would be dangerous to inspect.
The recording mechanism
The harness produces traces as a side effect of normal operation. In Claude Code, the mechanism is a hook that fires after each turn and writes a line to the session’s trace file. The reference repository includes a hook configuration that enables this; teams using the repository inherit the configuration on installation.
The hook writes traces to ~/.claude/traces/<session_id>.jsonl. The directory is gitignored by default; traces stay on the user’s machine and are not committed to the repository. The reference repository defaults to the local-only mode. The evolution loop runs against the user’s own traces. The user’s library improves through the user’s own usage patterns.
Chapter 19 — The Three-Layer Trace Pipeline
A month of traces from a single user contains thousands of events. No analyst — and certainly no LLM operating within a context window — is going to read all of them and produce useful conclusions. The data has to be distilled before it can be reasoned over.
The pipeline has three layers. Each layer takes the output of the previous layer and produces something smaller and more focused. By the end of the pipeline, the team has a per-skill view of what happened, ready to feed into the evolver of the next chapter.
Layer 1 — Raw trajectories. The bottom layer is the raw trace data. The pipeline filters on time range, skill references, and sanity. The filtered set is the input to the next layer. A typical analysis window contains a few hundred traces.
Layer 2 — Causal extraction. The second layer extracts, from each raw trace, the causal chain of actions and feedback. For each session, the pipeline produces a list of steps, each with the shape: the step index, the active skill, a compact action summary (tool name + key arguments + key result, ≤ 400 characters), a compact feedback summary, and a causal link indicating whether this step’s outcome influenced subsequent steps.
The extraction is mechanical. The pipeline does not need an LLM to do it; a few hundred lines of Python with the right summarisation rules suffice. The four-hundred-character budget per field is a deliberate choice: small enough that a hundred-step session fits in roughly eighty kilobytes of text, well within the context window of any modern LLM, but large enough to capture the essential outcome of any action.
Layer 3 — Evidence grouping. The third layer groups sessions by the skill they referenced. For each skill in the library, the pipeline produces an evidence group: the set of sessions that loaded the skill, with aggregate statistics and recurring patterns extracted.
1 | G(diagnose-data-product-status): |
The evidence group is the unit the evolver reasons over. For each skill, it answers: how often was this skill invoked, what fraction of invocations succeeded, what patterns recurred, and what cases failed.
The empty group, G(∅), collects sessions where the agent answered without invoking any skill. This is the gaps view: patterns in G(∅) are the things the agent has been doing without skill support. The evolver reasons over G(∅) to propose new skills.
Chapter 20 — The Evolver Agent
The evidence groups from the previous chapter are the input. Proposed skill changes are the output. The agent that performs this translation is the evolver: an LLM-driven subagent that reasons over a skill’s evidence and produces one of four actions for that skill.
The four actions
refine — edit the skill’s prose to address an observed failure or to incorporate a learned constraint. The edit is targeted: a specific section is changed, the rest is left alone. This is the most common action and the safest.
create — propose a new skill. This action applies to the G(∅) group, where the evidence shows the agent doing something that no skill currently covers. The evolver drafts a new skill following the forging pipeline of Part II.
optimize_description — modify only the description field. The skill’s prose is fine, but its description is causing it to trigger for prompts it should not, or to fail to trigger for prompts it should. Description-only changes are cheaper than body changes and often resolve mismatched-routing problems.
skip — do nothing. The evidence is insufficient to justify a change. The skill is working well enough; or the failures observed are not the skill’s fault; or the patterns are not clear enough to act on.
The four-action menu is the SkillClaw paper’s contribution, with the names lightly adapted. The discipline of choosing among them — rather than always editing — is what prevents the evolver from accumulating churn. An evolver that defaults to refinement on every group will produce a library that drifts unpredictably; an evolver that respects skip will produce a library that improves only where improvement is warranted.
A library that goes through an evolution cycle with most skills receiving skip is a library that is operating well. The cycles where most skills receive refine are early-stage cycles; the cycles where most skills receive skip are mature-stage cycles. The transition is a sign of progress.
The conservative-editing constraints
The evolver is an LLM, and LLMs left to their own devices will rewrite more than they need to, replace correct content with adjacent content that sounds plausible, and generally produce changes that fix the local problem while introducing global ones. The constraints below prevent this.
- Treat the current skill as the source of truth, not a draft. The skill was forged carefully and has been working for some users; the evolver’s job is to extend or refine it, not to rewrite it.
- Preserve sections that successful sessions support. If an evidence-group session succeeded and the success used a particular section of the skill, that section is doing its job.
- Edit narrowly when failures are corner cases. A failure that occurred in two of fifty sessions is a corner case. Add the constraint or the troubleshooting note; do not restructure the skill around the corner case.
- Do not casually change API contracts, ports, endpoints, output paths, payload formats, or required filenames. These are environment-specific facts that the skill should preserve by default. Replacing the team’s memory with the evolver’s guess is a regression.
- Do not remove core capabilities, API references, command patterns, or tool-usage examples unrelated to the observed failures. Keep what worked.
- Do not rewrite the whole skill from scratch. Targeted edits only.
Distinguishing skill problems from agent problems
The most important judgement the evolver has to make is whether a failure is the skill’s fault. The taxonomy:
Skill problem. The skill is wrong, incomplete, or misleading. Fix: edit the skill.
Agent problem. The skill is correct, but the agent did not follow it. Fix: do not edit the skill. The skill is fine; the agent’s misbehaviour is addressed by improving the harness or the model, not the skill.
Environment problem. The skill and the agent both did the right thing, but the environment failed. Fix: usually nothing in the skill; possibly a brief note about how the environment is known to behave.
The critical anti-pattern: when the skill already contains correct environment information (API endpoints, ports, payload formats) and the agent failed because it did not use that information, the failure is an agent problem, not a skill problem. The evolver must not delete the correct information and replace it with instructions like “consult the source code for the correct API”. This is the most damaging edit an evolver can make: it removes the institutional memory and replaces it with a deferral that will fail the next time someone tries to use the skill.
Chapter 21 — Dual-Track Validation
The evolver has proposed a change. The remaining question is whether the change is correct: does it fix the failures it was meant to fix, and does it leave intact the successes that were already working?
The validation mechanism has a name: dual-track validation. The two tracks are invariant and target. The invariant track verifies that previously-working behaviour still works. The target track verifies that previously-failing behaviour now works. A proposal that passes both tracks is accepted; a proposal that fails either is rejected.
The invariant track
The invariant track is the set of sessions that succeeded with the previous version of the skill. Any candidate new version must continue to produce correct results for them.
The track’s name reflects its purpose: it captures the invariants that the skill must preserve. A successful evolution adds capability; it does not remove capability. The invariant track is the mechanism by which “does not remove capability” becomes a check rather than a hope.
The target track
The target track is the set of sessions that failed with the previous version. The candidate skill should now produce correct results for them.
For each failing session, the target track requires: the agent’s response indicates the correct outcome; the actions the agent takes are consistent with what the corrected skill prescribes; any specific failure mode observed in the original session does not recur.
The third point is where absent_regex assertions earn their keep. The original failure was a specific pattern in the trace; the corrected skill should prevent the pattern from appearing in the new trace.
The decision rule
The decision rule for accepting a candidate is simple: both tracks must pass.
- Invariant pass + target pass → accept. The candidate fixes the bug without breaking anything. The evolution is sound.
- Invariant pass + target fail → reject. The candidate does not actually fix the bug. The proposed change was off-target.
- Invariant fail + target pass → reject. The candidate fixes the bug but breaks something else. The proposed change introduced a regression.
- Invariant fail + target fail → reject. The candidate is worse than the original on multiple dimensions.
The strict rule is what makes the loop monotonic. A library that only accepts changes that pass both tracks cannot regress: every accepted change is, by construction, at least as good as the previous version. Over many cycles, the library accumulates improvements without accumulating bugs.
What happens to rejected candidates
A rejected candidate is not deleted. It is preserved in a rejected/ directory alongside the skill’s history. The rejection is recorded, with the candidate’s prose, the rationale the evolver gave, and the validator’s report explaining why the candidate failed. This preserves three pieces of information: the failure mode the evolver was trying to address, the specific attempt that failed, and the validation evidence.
The rejected directory is not loaded by the harness; it does not affect the library’s runtime behaviour. It is purely an evolutionary archive, for the team’s own use when reasoning about the skill’s history.
The role of the eval suite
The skill’s eval suite (Chapter 9) provides assertions that the validator uses for both tracks. Evolutions that add new absent_regex assertions are an example of the evolution loop closing on itself: the failure that drove the evolution becomes a permanent regression test for the skill. The next evolver that proposes a similar mistake will see the assertion fail and the proposal rejected.
This is the property the SkillClaw paper calls “monotonic deployment”. The library’s quality only goes up over time, because every accepted change is verified to preserve what was working, and every new test added in response to a failure prevents the same failure from recurring.
Chapter 22 — The History Ledger
Every accepted change leaves a trail. The trail must be precise enough that a reader can answer, six months later, “why does this skill look like this?” and find the answer in the file system rather than in someone’s memory.
What the ledger records
The history ledger lives in each skill’s history/ directory. Every accepted evolution produces two files there:
history/v<N>.md: a verbatim copy of the skill’sSKILL.mdcontent at version N, before the change that produced version N+1.history/v<N>_evidence.md: a structured note explaining what changed between version N and version N+1, why, and based on what evidence.
A well-formed evidence file has sections for: decision summary (action, target, why), session evidence (specific sessions and their outcomes), historical comparison (what the previous version did and why that is no longer sufficient), the edit plan (what was changed, what was preserved), new eval assertions added, and open questions.
Why versions, not dates
The reference repository uses version-based filenames (v1.md, v2.md) rather than date-based ones (2026-06-15.md). The choice is deliberate.
Evolutions can happen multiple times per day. Date-based filenames would have to add timestamps to disambiguate, producing names like 2026-06-15T09:32.md. The names become long, the lexicographic order becomes confusing, and the reader has to mentally translate timestamps to version numbers anyway, because the version number is what appears in the manifest and the frontmatter.
Version-based filenames are short, unambiguous, and align with the rest of the ledger. A team that wants the date for human readability can include it inside the file.
Reading history before editing
A discipline that the evolver should observe, and that human authors should observe equally: read the history before editing.
Before making a substantive change to a skill, read every prior v*_evidence.md file. Understand what changed in each prior version and why, what evidence drove each change, which assertions were added at each step, whether any prior change is being reverted or contradicted by the proposed change.
A change that contradicts a prior change is not necessarily wrong — circumstances change, prior decisions get superseded — but it should be made deliberately, with the contradiction acknowledged. A change made in ignorance of a prior decision is likely to re-introduce a problem the prior decision was solving.
Reverting
Sometimes a change, after merging, turns out to be wrong. The repository can revert. The revert is itself an evolution, and it produces a new history entry: the bad state is preserved as v<N+1>.md; the v<N+1>_evidence.md explains why the revert was performed. The version number does not go backwards.
This is a subtle convention but important. Reverts are forward motion in the version sequence; the history ledger preserves the bad state as evidence of what went wrong, and the team can read forward through the ledger and understand the full trajectory.
Chapter 23 — Running the Loop
The command is make evolve. What that command should do, when the team should invoke it, what should happen when it produces a result, and what should happen when the team stops trusting it — those are the subjects of this chapter.
1 | evolve: |
The cadence question
Too often: the team runs make evolve after every session. Most sessions produced no useful evidence — the skill worked, the evolver has nothing to change. The evolver dutifully proposes nothing, or worse, proposes cosmetic changes to fill the silence. The ritual stops being run.
Too rarely: the team runs make evolve once a quarter. The evidence has accumulated across dozens of sessions, and the evolver is asked to produce a single change that fixes everything. The proposal is large, the validator’s verdict is uncertain, and the human reviewer cannot untangle which piece of evidence motivated which piece of the edit.
The interval that produces good evolution is usually weekly to bi-weekly, keyed to the team’s release cadence for the skill. The heuristic to watch: if the average evidence group in the pipeline has three or more sessions and covers a coherent failure pattern, the interval is right.
The human in the loop
Three postures are possible.
Auto-accept. The validator says accept, the change is applied, no human is consulted. Appropriate for well-understood skills with mature eval suites and a long track record of successful evolution.
Reviewed-accept. The validator says accept, and the change is opened as a pull request. A human reviews the proposal, the evidence, and the eval delta, then merges. The reviewer’s job is not to re-do the validator’s work; it is to check whether the change fits the team’s evolving intent for the skill, and whether the evidence is being interpreted charitably. The review is usually five to ten minutes.
Escalate-accept. The validator says accept, but the reviewer disagrees or wants a second opinion. This posture should be rare; if it becomes common, either the evolver’s proposals are systematically wrong or the team’s intent for the skill is drifting.
A team new to the loop should start in reviewed-accept for every skill, then move individual skills to auto-accept as trust accumulates.
Failure modes of the loop itself
The loop can hallucinate progress. The evolver produces proposals, the validator accepts most of them, the history ledger fills up — and the skills are not measurably better. The symptom is that user-reported issues do not decline even as the ledger grows. The response is to look at the traces, not at the evolver.
The loop can stagnate. The evolver proposes small changes, the validator accepts them, the skill drifts toward local optima. The proposals get smaller over time and never touch the skill’s structural assumptions. The response is periodic redrafting — an occasional deliberate rewrite from scratch, informed by everything in the history but not constrained by it.
The loop can outpace the team. The evolver runs faster than the team can review, proposals queue up, and the team stops reading evidence files. The response is to slow the cadence, not speed the review.
The evolution loop is running well when: make evolve runs to completion without manual intervention on more than half of runs; the evidence files, read a month later, still make sense to their author; the team’s reported friction with skills declines over quarters; the history ledger for each skill tells a coherent story.
Series footer:
Part 4 of 6 — Skill Engineering
Part 1: Foundations · Part 2: Creation · Part 3: Governance · Part 4: Evolution · Part 5: Outlook · Part 6: Appendices