Skill Engineering, Part 2: Creation
This is Part 2 of a six-part series on Skill Engineering. Part 1 — Foundations covers what skills are and how they are structured.
Chapter 6 — The 3x Rule
Before writing any skill, ask whether the skill is worth writing. The answer depends on two numbers: how many times you have done the task, and how expensive each invocation is.
The rule: if you have done the task at least three times, and the cumulative cost of the three invocations exceeds the cost of forging and maintaining the skill, the skill is worth writing.
Why three?
One is insufficient evidence that the task is recurring. Two means you have done it twice — possibly a coincidence, possibly a recurring pattern. Three is the minimum that separates “this happened multiple times” from “this is the kind of thing that keeps happening.” Three also provides enough evidence to identify the pitfalls: by the third time, you have seen at least two different failure modes.
The number is a heuristic, not a law. A task that is done twice but whose next recurrence is nearly certain (a weekly maintenance procedure, a standard diagnostic sequence) can justify a skill at two. A task that is done four times but whose next invocations are uncertain (a one-off migration that might need reversing, might not) might not.
Estimating cost
The cost calculation has two sides.
Invocation cost. How long does one manual invocation take? Include the time to gather context, navigate to the right systems, remember the specific pitfalls, execute the procedure, and verify the result. A thirty-minute diagnostic procedure costs thirty minutes per invocation. Ten invocations cost five hours. The cost of a week’s worth of invocations is visible.
Skill cost. How long does it take to forge and maintain the skill? Exploration (one guided invocation, taking notes): sixty to ninety minutes. Crystallisation (writing the SKILL.md): sixty to ninety minutes. Hardening (writing the evals): thirty to sixty minutes. Shipping (registration, first history entry, PR): thirty minutes. Total: three to five hours for a typical skill.
If ten invocations cost five hours and a skill costs four hours to forge, the skill pays for itself in invocation eight — the first two invocations after the forging cost is covered. Everything beyond that is pure leverage.
The cost of not writing the skill
The calculation above counts the direct invocation cost. It does not count the indirect costs: the colleague who asks the same question and spends their own thirty minutes; the degraded result when the informal version of the procedure is applied without the pitfalls in mind; the institutional knowledge that stays in one person’s head and vanishes when that person is on leave or leaves the team.
Skills are how teams make their members replaceable in the good sense: anyone on the team can run the procedure correctly, because the procedure is not in anyone’s head.
Chapter 7 — Explore
The Explore phase is the first of four phases in the forging pipeline. Its purpose is to produce the raw material for the skill: a documented procedure, a list of pitfalls, and a clear statement of what the skill must and must not do.
The discipline: run the procedure once as an exploration. Not fast, not from memory. Run it slowly, note every decision point, document every pitfall, record every place where the first attempt was wrong and a second attempt was needed.
What the exploration produces
The exploration produces a explore.md file — not committed, not beautiful, not intended for anyone else — with:
- The procedure, as you actually ran it, step by step.
- Pitfalls: places where a naive agent would fail, clearly labeled.
- Constraints: non-negotiable rules (never do X; always verify Y before Z).
- Open questions: things you are not sure about that might matter.
The pitfall ledger is the most important output. Each pitfall becomes an absent_regex assertion in the eval file (Chapter 9). Each one is a documented mistake that the agent must not repeat.
The pitfall ledger
A pitfall is not a constraint. A constraint is a rule the skill imposes (“always wait thirty seconds after triggering a full-sync”). A pitfall is a mistake the environment traps you in if you are not careful (“the Loki query requires the internal tenant UUID, not the customer-visible tenant name — do not query with the name”).
The distinction matters because the responses are different. A constraint goes in the Constraints section of the skill. A pitfall goes in the Constraints section and generates an absent_regex assertion. A pitfall without an assertion is an undocumented risk.
Constraints vs. anti-patterns
Not everything that belongs in the skill belongs in Explore. The things that belong in Explore are the things you discover empirically, by running the procedure: the edge cases, the failure modes, the environment-specific quirks.
The things that do not belong in Explore are the things you know in advance: general best practices, generic error handling, agent-runtime concerns. Those go in the skill if they are relevant to this specific procedure; they do not come from the exploration.
The question to ask for each item in the exploration notes: “would an agent following a generic, well-formed version of this procedure get this right?” If yes, it is a general practice and does not need special mention. If no — if the agent would fail without this specific guidance — it is a pitfall.
What to do with open questions
Open questions belong at the end of explore.md and are not resolved during the Explore phase. Some of them will be resolved during Crystallisation (when you write the skill and have to commit to a specific approach). Some will be left as notes in the Troubleshooting section. Some will become the first item in the skill’s history/v0_evidence.md under “open questions”.
The discipline is not to delay the Explore phase by trying to resolve open questions. Run the procedure, take the notes, move forward. The open questions are feedback to the next phase, not blockers of this one.
Chapter 8 — Crystallize
Crystallisation is the writing phase. The exploration is done; the pitfall ledger is complete; now the skill is written. The discipline of this phase is specific: write for the agent reader, not for yourself.
Write frontmatter first
Start with the frontmatter. The name, version, layer, status, owners, tags, description. Write the description last within the frontmatter, because the description is the hardest field to write well — it is the routing signal the harness uses to decide whether this skill applies to a given user request.
A good description:
- Describes the task in terms the user would use (“diagnose why a data product is stuck in activating”), not in terms of the implementation (“query Loki and correlate with Grafana”).
- Covers the full range of triggers, not just the happy path (“including cases where the pipeline run has not started yet”).
- Is specific enough to distinguish this skill from related skills (“…in the activating state” distinguishes it from a skill about products in the deploying state).
A bad description is one that either matches nothing (too specific) or matches everything (too general). The most common failure mode is the latter: descriptions that say “help with data products” will trigger the skill for any data product question, including ones the skill cannot handle.
Read sibling skills before writing
Before writing the skill body, read two or three existing skills in the same library. Not to copy their content — to absorb their conventions. Every team’s library develops a house style: how Constraints sections are phrased, whether the Instructions section uses numbered steps or prose paragraphs, how the Self-Test is structured.
A new skill that violates the house style is harder to maintain, because the team’s automated habits — the mental models for “how skills work here” — do not apply to it. Matching the style is not aesthetic conformism; it is consistency that reduces maintenance cost.
Write for the agent reader
The agent reader is not a human. Several important differences:
The agent cannot ask questions. If the skill says “wait an appropriate amount of time,” the agent will wait for what it thinks is appropriate, which may be zero seconds or five minutes. Write “wait thirty seconds.” Be specific.
The agent reads everything. A human skims. The agent reads the entire skill prose before acting. This means that a skill body that buries the most important constraint at the end is a skill whose constraint will be read too late. Put constraints before instructions.
The agent cannot infer unstated context. A human reading “query the stage log” knows to use the tenant-specific endpoint and to filter by product ID. The agent does not know this unless you say it. Write the steps as if explaining to a competent but brand-new colleague who has never seen your system before.
Prose density matters. A dense, highly structured skill is harder for an agent to parse than a skill with clear section headings, numbered lists, and one idea per sentence. Write as you would write for a technical audience that is reading quickly.
The scaffold order
For a workflow skill:
- Write the frontmatter (all fields).
- Write the Overview (what does this skill do, when should it be invoked, what are the primary dependencies).
- Write the Constraints (pitfalls from the exploration, non-negotiable rules).
- Write the Instructions (the procedure, numbered, one action per step).
- Write the Self-Test (prerequisite check + live test).
- Write the Troubleshooting section (known failure patterns and their responses).
For a connector skill, the scaffold is simpler: Overview, then the API endpoints and how to call them, then the auth pattern, then common errors. The connector does not need elaborate Instructions because it does not direct the agent through a multi-step procedure; it exposes an API.
Progressive disclosure
A skill body should be ≤ 500 lines. When detail threatens to push it past that, move the detail to references/ files and instruct the agent to consult them when needed (“for the full list of stage-log error codes, see references/stage-log-codes.md“).
This is progressive disclosure: the skill presents the essential flow, and the detail is available on demand. An agent that reads a 500-line skill is an agent that processes what it needs; an agent that reads a 2000-line skill is an agent that has loaded so much context that the signal is diluted by the noise.
Chapter 9 — Harden
A skill is not done when it is written. A skill is done when it is verified. Hardening is the phase that produces the verification: the eval suite, the Self-Test, the CI integration.
The three assertion types
The reference eval framework supports three assertion types. Each has a distinct role.
substring — fires when a literal string appears in the agent’s trace. Use it to assert that the agent said or did something specific: “routing to bdc-team,” “querying stage log,” “found stale reference.”
regex — fires when a pattern matches the trace. Use it for assertions about shape rather than exact content: the agent produced a Jira key of the form DC00-\d+; the agent waited some positive number of seconds; the agent referenced an ISO-8601 date.
absent_regex — fires when a pattern does not appear in the trace. This is the highest-leverage assertion type. Use it to encode pitfalls: the agent must not query with a customer-visible tenant name; the agent must not route to the ops team for an application-layer failure; the agent must not produce a raw Bearer token in its output.
The absent_regex is a scar: every entry corresponds to a mistake that was made once and must not be made again. A mature skill’s eval file reads like a history lesson.
How many assertions?
The minimum gate for a Harden-ready skill: at least three cases, at least one absent_regex per case, evals passing on the current agent, CI configured to run them.
Three cases is not many. The cases should cover:
- The happy path (the common successful invocation).
- A known failure mode (a case where the naive version would fail; the skill should succeed).
- A constraint case (a case that tests a specific constraint from the Constraints section).
More cases are better, especially for skills that are invoked often or whose failures have downstream consequences. But three is the minimum that demonstrates that the skill has been tested beyond “I ran it once and it worked.”
The Self-Test
The Self-Test is not an eval. It is a runnable check written in the skill’s prose body, intended to be run by the user or the agent when something feels wrong.
It has two tiers. The first is a prerequisite check: does sig login succeed? is the required CLI tool installed? are the environment variables set? These checks run in under a second and fail fast if the environment is not ready.
The second is a live test: given a known-good input (a specific tenant, a specific resource that is known to exist), does the skill return a sensible result? This check actually invokes the procedure against the live system; it takes longer but catches a different class of failure (the environment exists but the procedure is wrong).
The Self-Test is the skill’s equivalent of a smoke test. It is not a substitute for the eval suite; it is a complement — the thing you run interactively when you need to confirm the skill is working right now, not in the CI environment an hour ago.
The CI gate
The eval suite is not useful if it is not run. Wire it into CI: on every commit that touches the skill’s directory, the CI system runs scripts/run-evals.py and reports the result. A skill that breaks its own evals cannot merge.
This is the discipline made enforceable. Without the CI gate, the eval suite is documentation. With it, the eval suite is a contract.
Chapter 10 — Ship
The Harden phase is complete: the skill is written, the evals pass, the CI is wired. Ship is the phase that moves the skill from “a file I have on my machine” to “a skill the team uses.”
Installation
Install the skill by symlinking the source directory into ~/.claude/skills/:
1 | make install SKILL=diagnose-product-status |
The make install target reads manifest.yaml‘s distribution.symlink_target field and creates the symlink. The source of truth stays in the git-tracked directory; the symlink points to it. Changes to the skill in git are immediately reflected in the installed version.
If the skill is a project-scoped skill (Chapter 15), the symlink goes into .claude/skills/ in the project directory rather than ~/.claude/skills/.
The first history entry
Write history/v0_evidence.md. This is the skill’s creation rationale — a brief prose note explaining why the skill was created, what decisions were made during forging, and what open questions remain.
The v0 evidence file is not elaborate. A few sentences on motivation (the 3x rule was satisfied), the significant design decisions (why this layer? why these dependencies?), and the open questions that will be revisited after the skill has been used. The discipline of Chapter 22 applies here: write it as a message to your future self, who will be reading this six months from now and wants to understand why the skill looks the way it does.
The PR
The PR for a new skill contains:
- The skill directory, complete with
SKILL.md,manifest.yaml,evals/, andhistory/v0_evidence.md. - The exploration file (
explore.md), as a reference. It does not need to be a polished document; it is the record of the exploration work. - A screenshot of the eval runner’s output (all cases passing).
- A note in the PR description about the 3x calculation: what task, how many times, what the cost estimate was.
Two reviewers. The review is not a rubber stamp: the reviewers confirm that the skill’s description is correct (does it match or overlap existing skills?), that the layer assignment is right, that the dependencies are declared, and that the evals test the right things.
Announcing the skill
A short message to the team’s channel: “new skill: diagnose-product-status. Use it when a customer’s data product is stuck in activating. It handles the three most common failure modes automatically.” Include the skill’s name (the harness will route to it on that name), a one-sentence description of when to use it, and the most important thing it handles that users would otherwise do manually.
The announcement is not optional. A skill that is installed but not known does not get used. The team’s awareness of what is in the library is the mechanism by which the library’s leverage is realised.
Chapter 11 — Anti-Patterns
Eight patterns, observed in practice, that produce libraries that look active but accumulate debt.
Mixed-role skills. A skill that is part connector (exposes a system) and part workflow (makes business decisions) is two skills that have not been separated. The connector becomes hard to reuse because it carries workflow logic. The workflow becomes hard to test in isolation because it contains API-specific code. Separate them.
Credential-bearing skills. A skill that contains a credential — even a “test” credential, even a “low-privilege” credential — is a skill that will eventually leak the credential. The zero-trust proxy exists for this reason. If a skill cannot be written without an embedded credential, the fix is to add a sigcli provider for the system, not to embed the credential.
Prose-stuffed skills. A skill whose Instructions section contains 800 words of dense prose, no numbered steps, and no clear structure. The agent can read it; the agent cannot navigate it. Skills are navigational documents as much as they are informational ones. Structure helps the agent as much as content does.
The corrupting edit. A skill that already contains correct environment information (API endpoints, payload formats, required fields) is edited to remove that information because a session failed. The failure was an agent problem (the agent did not use the information), not a skill problem (the information was wrong). The edit removes the institutional memory and replaces it with nothing. The next agent will fail for the same reason.
The disappearing API. A skill that says “consult the API documentation for the current endpoint structure.” The documentation link is broken six months later. The skill is useless without the link. The correct move is to include the relevant API shape in the skill itself (or in references/) and to update it when the API changes. The skill is the documentation.
The eternal experimental. A skill that has been in status: experimental for eight months and has been used by the team for six of them. Experimental status is meant to signal “do not depend on this; it may change.” A skill that is depended on but still marked experimental is sending a false signal. Promote it.
Trigger-phrase scarcity. A skill whose description is so narrow that the harness routes to it only when the user uses an exact phrase. The user asks “why is the data product not activating?” and the skill is not invoked because the description says “diagnose stuck-in-activating data products.” Write the description to cover the range of ways a user might phrase the request.
The over-engineered first version. A skill written to handle every possible variant, edge case, and future requirement — on the first version, before any invocations have revealed which variants actually occur. The skill is long, the evals test hypothetical cases, and the maintenance cost is immediately high. Write the minimal correct version. Let evolution (Part IV) add what usage reveals is needed.
Series footer:
Part 2 of 6 — Skill Engineering
Part 1: Foundations · Part 2: Creation · Part 3: Governance · Part 4: Evolution · Part 5: Outlook · Part 6: Appendices