Skill Engineering, Part 5: Outlook
This is Part 5 of a six-part series on Skill Engineering. This final part steps back from the mechanics and asks where the discipline goes next.
Chapter 24 — Collective Evolution
Every chapter so far has assumed a single team using a single library, running the loop on that team’s own traces. This is a defensible assumption for a book that wants to describe a discipline rather than a research direction. It is also, in the long run, the least interesting configuration of the machinery.
The reason skills exist at all is that experience compounds. One person learns that the deployment API returns stale data for thirty seconds after a full-sync, and the next person does not have to learn it. A skill is the artefact that carries that lesson forward. Every skill in a working library is a message from someone’s past self to someone else’s present self, and the value of the library is the value of the messages.
Collective evolution is the observation that this compounding does not stop at the team boundary. If one team learns something and encodes it in their skill, another team using the same underlying system could benefit from the same lesson.
The SkillClaw observation
The SkillClaw paper’s central claim, as it applies here, is that skills evolve faster and reach higher quality when the evidence base is pooled. A skill maintained by one user with one user’s failure traces will improve; a skill maintained across many users’ failure traces will improve faster, and its improvements will generalise better because the evidence base is broader.
Every mature open-source library benefits from the same effect: bug reports come from many users, patterns emerge that no single user could have seen, and the library becomes more robust than any individual contributor could have made it.
The redaction problem
The first and largest problem with collective evolution: traces contain sensitive data. Enterprise skills touch real systems with real credentials, real customer names, real internal URLs, real ticket numbers that map to real incidents. Pooling raw traces across users is a data-handling incident waiting to happen.
Three approaches exist on a spectrum from “cheap and permissive” to “expensive and safe”.
Redact on capture. The trace-capture step applies a redaction pass before writing to disk. Regexes strip known credential patterns; named-entity recognition flags person names and organisation names. The resulting trace is safer but the redaction is imperfect.
Structural redaction. Instead of preserving trace text and redacting sensitive spans, capture only the trace’s structure: which skill was invoked, which tool calls it made, what arguments (redacted to types rather than values), what the outcome was. This approach loses some fidelity but the residual risk is much lower.
Opt-in per trace. The user reviews each trace before it enters the collective pool. This is the safest approach but also the slowest, and it depends on the user’s ongoing attention.
A mature deployment probably combines all three: automated redaction on capture as the first line, structural extraction for the pool, and per-trace opt-in for the residual cases.
What actually gets pooled
The useful middle: pool the version-to-version deltas along with structural evidence summaries. For each accepted evolution, the pool contains what changed in the skill (the diff between vN.md and vN+1.md), why it changed (a de-identified summary of the evidence), and what assertion was added to prevent regression.
This is the level at which the SkillClaw paper’s results were most encouraging. Skills that were exposed to cross-user evidence improved along axes that single-user evidence would not have surfaced, and the improvements survived validation on holdout tasks.
Trust and the reviewer network
Collective evolution introduces a governance question that single-team evolution avoided: whose review is authoritative?
In a collective model, an evolver reasoning over pooled evidence might propose a change to a skill that Team A owns based on evidence contributed by Teams B and C. Team A’s reviewer is the authority; the evolution should not land in Team A’s library without Team A’s sign-off, no matter how strong the pooled evidence is.
This is the same pattern as upstream open-source contribution. The skill’s maintainer decides what merges. The collective is a source of evidence and proposals, not a source of decisions.
Trust builds over time. A team that has repeatedly received useful proposals from the collective will grow more willing to auto-accept from it; a team that has received noisy proposals will grow more skeptical.
What the discipline gains
Rot detection is faster. A skill that works for User A may already be failing for User B, whose environment has drifted in a way A has not encountered. Pooled traces surface these failures before they hit A’s environment.
Generalisation is checked. A change that looks right based on a single team’s traces might not survive contact with another team’s environment. When the evolver reasons over pooled evidence, proposals that would generalise poorly are visible earlier.
The discipline itself matures. Every team’s history ledger becomes visible to every other team. Patterns of what makes a good evidence file, a good invariant assertion, a good version-bump decision become learnable by reading across teams.
What the discipline risks
Pooled evidence can homogenise. If every team’s skill for querying Jira is evolving against the same pooled trace stream, the skills converge on a single design that fits the average tenant well and no tenant perfectly. The response is to keep skills team-scoped by default and pool evidence rather than skills.
Pooled evidence can lie. A team may contribute evidence in good faith that reflects a misconfiguration on their side. The response is provenance — every piece of evidence traces back to a specific team’s trace, and reviewers can weight or discount evidence based on its source.
Pooled evidence can centralise. A large team with many contributors dominates the pool by volume. The response is deliberate weighting — evidence is not one-trace-one-vote but weighted for pattern diversity across contributors.
An honest scope
The single-user, single-team model of Parts I–IV is production-ready. Teams can adopt it today. The collective model of this chapter is not yet production-ready in the same sense; the infrastructure exists in research prototypes and early industry deployments, but the discipline of running it at scale is still being learned.
Chapter 25 — Cross-Domain Skill Transfer
The book has been about skills for agentic coding harnesses. Claude Code was the working example, sap-jira and catalog-bdc-pkg-missing were the running references, and every principle was tested against the question “does this help someone writing skills for their team’s library today?” This chapter widens the frame. Nothing about the discipline is intrinsic to coding, or to SAP, or to Claude Code. The pieces travel.
The skill graph as a first-class object
Twenty skills interlink. Fifty skills interlink more. A hundred skills form a structure that no single reader can hold in memory, and the questions the reader wants to ask — which skills depend on this one? which skills would break if this connector’s contract changed? which skills use the same substrate? — become graph queries in earnest.
A well-shaped library looks a specific way when the graph is drawn. There are a few nodes with no dependencies (the substrate). There are many nodes that depend on the substrate and on nothing else (the connectors). There are a moderate number of nodes that compose connectors (the primitives). And there are workflows at the top of the graph, each depending on a small number of primitives.
When the graph looks different — when it is flat, when everything depends on everything, when workflows depend on other workflows, when the substrate has fifteen layers — the library is telling the team something. The graph is the library’s structural x-ray. Reading it well is a skill in itself.
Skills outside coding
The Claude Code assumption is convenient but not necessary. The discipline described in this book — versioned procedural knowledge, executable by an agent, governed by dependencies, evolved by evidence — applies wherever an agent operates against a stable environment. Three domains are already visible on the near horizon.
Customer service. Support agents resolve tickets by following a procedure that varies with the customer, the product, and the state of the world. That procedure is a skill. Today it lives in a runbook or a support wiki, written for humans and consumed by humans. Rewritten as a skill, it would be versioned, executable, and evolvable.
Compliance and audit. A compliance officer’s job includes running periodic checks against enterprise systems — verifying that certain access controls are in place, that certain data-retention policies are followed, that certain approvals are recorded. Every one of these is a procedure that can be encoded as a skill and executed by an agent. The failure modes are different (a compliance skill that fails silently is much worse than a data-product-diagnosis skill that fails silently), which affects the eval design and the evolution cadence.
Scientific workflows. A researcher running a data-analysis pipeline follows a procedure — load the data, clean it, apply a set of transformations, produce plots, write up the results. Much of that procedure is stable across studies and could be encoded as skills. The scientific-workflow community has been building this machinery for decades (Snakemake, Nextflow, Make) but has not connected it to the agent-native world.
The structural improvements that generalise
Some observations from Parts I–IV are specific to coding harnesses. Others generalise.
The four roles generalise. Every domain that has an agentic layer will have a substrate, connectors, primitives, and workflows. The names may change; the four levels probably do not.
The three phases of creation generalise. Explore, crystallise, harden. A support-workflow skill or a compliance skill is written badly if the author skips exploration or skips hardening.
The evolution loop generalises. Traces, evidence, evolver, validator, ledger. The specifics of each piece are domain-dependent — what a “trace” is in a support-agent context is not the same as in a code context — but the loop’s shape is the same.
The dependency graph generalises. Every mature library will have one. Making it explicit, machine-readable, and enforceable pays off in every domain.
Open problems
Automated eval synthesis. Chapter 9 spent effort on the manual authoring of eval assertions. Automating this — deriving assertions from traces, from the skill’s own body, from adjacent skills — is a live research direction where progress would make Harden dramatically cheaper.
Cross-team merge conflicts. When two teams evolve the same skill in incompatible directions, whose version wins? Better merge semantics for skills — perhaps semantic-diff-aware, perhaps evidence-aware — is a problem the discipline will hit before the collective model matures.
Evaluating the evolver itself. The evolver is a system with its own quality dimension. Building an eval harness for the evolver — the meta-eval — is an open research problem that becomes more pressing as the loop’s autonomy widens.
What the book has tried to be
A closing note about the book’s own ambition.
The book has tried to be short. Twenty-five chapters, roughly two thousand words each, is fifty thousand words — a working practitioner can read it in a weekend. The alternative — a comprehensive treatise on agentic engineering — would be longer and less useful.
The book has tried to be opinionated. Every chapter has taken positions: absent_regex is the highest-leverage assertion; version-based history filenames beat date-based ones; the four roles matter and the words for them matter; the loop should be run weekly, not daily; auto-accept is earned, not granted. A book without opinions is a manual, and manuals are for tools, not disciplines.
The book has tried to be practical. Every chapter has assumed a reader who wants to write skills tomorrow, not a reader who wants to read about skills forever. The forge-skill script exists, the manifest schema exists, the eval runner exists, the evolution loop exists. Everything described in this book has a corresponding artefact that someone could clone and run.
Whether the discipline described here becomes what “skill engineering” means in five years is not something the book can control. What it can do is offer the working practitioner a coherent way to think about the problem, a coherent set of practices to try, and a coherent vocabulary to reason with. If those three things travel, the book has done its job.
Coda
A book about skills for agents ends with a small procedural note. The library that inspired this book — awesome-sap-skills — is public. The forge-skill, the evolver, the eval runner, the manifest schema, the migration playbook: all of them live there, and all of them are open to being cloned, forked, or improved.
If the practices in this book are worth anything, they are worth trying against a real library that a real team is running. If they are not — if some chapter is wrong, or some vocabulary fails in practice, or some assumption breaks in a domain the book has not considered — the response is not another book. The response is a pull request. The discipline improves the way the skills it describes improve: by evidence, by review, and by patient revision.
The end of a book is the start of the practice. Good luck, and — as the ledger will remember — write the evidence carefully.
Series footer:
Part 5 of 6 — Skill Engineering
Part 1: Foundations · Part 2: Creation · Part 3: Governance · Part 4: Evolution · Part 5: Outlook · Part 6: Appendices