Skill Engineering, Part 6: Appendices

Posted on Edited on In

This is Part 6 of a six-part series on Skill Engineering. This final part contains the reference material: the manifest schema, eval assertion semantics, a worked end-to-end example, the migration playbook, and the glossary.

Appendix A — Manifest Schema Reference

Chapter 16 introduced manifest.yaml as the machine-readable mirror of a skill’s frontmatter. This appendix documents the schema in full.

Full schema (YAML, abbreviated)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
$schema: "https://awesome-sap-skills.dev/schemas/manifest.v1.json"

name: string                    # kebab-case, must match directory name
version: string                 # semver, e.g. "2.3.0"

layer: enum                     # infrastructure | connector | primitive | workflow
status: enum                    # experimental | stable | deprecated | replaced | archived
tags: [string]

owners: [string]
maintainer: string

auth:
  providers: [string]           # sigcli provider names required
  cli: [string]                 # external CLI binaries required
  env: [string]                 # environment variables required

depends_on:
  <skill-name>: <semver-range>

deprecated_by: string | null
deprecated_reason: string | null

distribution:
  scope: enum                   # global | project | private
  symlink_target: string

description: string             # matches frontmatter; ≤ 500 words

created_at: string              # ISO 8601
last_evolved_at: string         # ISO 8601

Key field notes

name — kebab-case, ASCII lowercase letters, digits, hyphens only, ≤ 40 characters, must equal the containing directory name. Rename requires a migration playbook (Appendix D) — never rename in place.

version — semver 2.0.0 (MAJOR.MINOR.PATCH); pre-release suffixes allowed (-alpha, -rc.1); + build metadata not used. Bump rules covered in Chapter 12.

layer — enforced by the resolver’s composition check: a skill of layer L may only depend on skills of the same or lower layer.

status — state transitions: experimental → stable → (deprecated or replaced) → archived. Reverse transitions require an evidence file.

auth.providers — each entry is a sigcli provider name. The resolver runs a pre-flight auth check before invoking the skill; a skill missing a required provider fails fast with a helpful error.

depends_on — keys must resolve to installed skills; values are semver ranges (>=1.4, ^2.0, >=1.4 <3.0).

deprecated_by — set only when status=replaced; must resolve to an installed skill; may not be self.

description — must match the SKILL.md frontmatter description byte-for-byte. The CI check enforces this.

Timestamps — managed by the forge-skill script (creation) and the record-history script (evolution). Do not hand-edit.

Worked example — a workflow skill

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
name: diagnose-product-status
version: 2.3.1
layer: workflow
status: stable
tags: [catalog, bdc, debugging, data-products]

owners: [pylon-peng, "@bdc-team"]
maintainer: pylon-peng

auth:
  providers: [cic-stabi, sap-jira]
  cli: [python3, jq]
  env: []

depends_on:
  sap-loki: ">=2.0"
  sap-jira: ">=1.4"
  catalog-support-route: ">=1.2"

deprecated_by: null
deprecated_reason: null

distribution:
  scope: project
  symlink_target: "~/.claude/skills/diagnose-product-status"

description: |
  Diagnose why a data product is stuck in the activating state. Queries
  the stage log, correlates with the pipeline run, checks known-failure
  patterns from history, and routes to the responsible team if the
  pattern is not covered by an existing recovery skill.

created_at: "2026-04-12T10:14:22Z"
last_evolved_at: "2026-06-28T15:47:03Z"

Consistency check

The CI runs scripts/validate-manifest.py on every commit. Failures include: description mismatch, version behind the latest history entry, unresolvable depends_on references, status=replaced with null deprecated_by, maintainer not in owners, and layer composition violations.

Appendix B — Eval Assertion Reference

Chapter 9 introduced the three assertion types the reference framework supports. This appendix documents the semantics of each, common patterns, and the file format the eval runner consumes.

The eval file format

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
version: 1

cases:
  - name: happy-path
    fixture: fixtures/happy-input.json
    assertions:
      - type: substring
        value: "product status is activating"
      - type: regex
        pattern: "waited \\d+ seconds"
      - type: absent_regex
        pattern: "TODO|FIXME|placeholder"

  - name: timing-constraint
    fixture: fixtures/quick-check.json
    assertions:
      - type: substring
        value: "wait at least 10 seconds"
      - type: absent_regex
        pattern: "verifying.*immediately"

A concrete evals.json example for sap-jira: three assertion types applied to a real eval case

substring

Fires when the assertion’s value string appears literally in the trace. Case-sensitive by default; add case_insensitive: true to relax.

Use substring when the correct behaviour of the skill produces a distinctive literal phrase that a wrong behaviour would not produce: a specific tool call, a specific instruction being honoured, a specific error message being surfaced.

Avoid substring when the phrase you want to check for is common English that could appear by coincidence. Assert on the distinctive part, not the generic part.

regex

Fires when the assertion’s pattern matches somewhere in the trace (Python’s re.search). Backslashes must be doubled in YAML (\\d for \d).

Use regex when the correct behaviour produces a shape rather than a literal: a positive integer number of seconds, a Jira key of the form DC00-\d+, an ISO-8601 timestamp.

Common patterns:

1
2
3
4
5
6
7
8
- type: regex
  pattern: "waited \\d+ seconds?"

- type: regex
  pattern: "\\b(DC00|DW101|DS00)-\\d+\\b"

- type: regex
  pattern: "\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}"

Avoid regex when a substring assertion would do the job. Regexes are more expressive but harder to read.

absent_regex

Fires when the assertion’s pattern does not match anywhere in the trace. The assertion passes when the pattern is absent; it fails when the pattern is present.

Use absent_regex when you want to check that the skill does not do something it used to do or that a naive implementation might do:

  • A prior version verified immediately without waiting: absent_regex: "verifying.*immediately".
  • The skill should route to bdc-team, not ops-team: absent_regex: "routing to ops-team".
  • Credentials must never surface: absent_regex: "Bearer \\w+".

The absent_regex file in a mature skill reads like a history lesson. Each entry corresponds to a mistake that was made once and must not be made again:

1
2
3
4
5
6
7
8
- type: absent_regex
  pattern: "verifying.*immediately"     # v1.2 timing bug
- type: absent_regex
  pattern: "routing to ops-team"        # v2.0 mis-routing bug
- type: absent_regex
  pattern: "TODO|FIXME"                 # never let placeholders ship
- type: absent_regex
  pattern: "Bearer [A-Za-z0-9\\-_]+"    # never surface tokens

Each line is a scar. The scars are the discipline made visible.

The gate

For a Harden-ready skill: at least three cases, at least one absent_regex per case, evals passing on the current agent, CI configured to run them.

Appendix C — Worked Example: diagnose-product-status

This appendix walks the skill that the book has used as its running example end-to-end: from the incident that motivated its creation, through the four forging phases, into its first evolution round.

The precipitating incident

A colleague pings: “customer’s data product has been stuck in activating for two hours, they’re escalating, what do you check first?” I know the general shape of the answer — the stage log, then the pipeline run, then the specific transform that is stalling — but I have not written the sequence down. I open the customer’s tenant in the DWC console, tail the Loki logs, cross-reference the pipeline run in Grafana, and forty minutes later the customer’s product is activating.

An hour later, another colleague asks the same question about a different customer.

The 3x rule has been satisfied. The workflow is repeatable, I have run it three times in a week, and each run is expensive enough that the cost of writing the skill is repaid in fewer than three future runs.

Phase 1 — Explore

I run the procedure a fourth time, this time as a deliberate exploration. Notes in explore.md:

1
2
3
4
5
6
7
8
9
10
11
12
13
## Attempt 1: check the stage log first

- Loki query: {service="stage-manager"} | json |
  tenant_id="acme-prod" | product_id="pd_abc123"
- Expected: a chain of stage transitions ending in "activating".
- Actual: no results. Query returned empty.

Cause: the tenant_id label is not "acme-prod" — it's the internal
UUID. The customer-visible tenant name only appears in the Jira
ticket, not the logs.

Pitfall: never assume customer-visible IDs match log labels.
The mapping is a separate lookup.

The exploration produces six such pitfalls. Each becomes an absent_regex in the eval file.

Phase 2 — Crystallize

I read three sibling skills first: sap-loki, catalog-support-route, and an existing workflow whose shape I want to match. I write the skill’s frontmatter first, then the Overview, Constraints, Instructions, Self-Test, and Troubleshooting. Two hours of focused writing. The skill is at v0.1.0, experimental.

Phase 3 — Harden

Three eval cases derived from real incidents:

  1. happy-path: straightforward activation failure caused by a missing source connection.
  2. timing-constraint: the most recent sync was under 30 seconds ago; the agent must wait.
  3. ambiguous-input: the customer name maps to two tenant UUIDs; the agent must ask the user to disambiguate.

A representative sample from happy-path:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
assertions:
  - type: substring
    value: "resolve customer name to tenant UUID"
  - type: substring
    value: "querying stage log"
  - type: regex
    pattern: "missing source connection: [a-z0-9\\-]+"
  - type: substring
    value: "routing to bdc-team"
  - type: absent_regex
    pattern: "routing to ops-team"
  - type: absent_regex
    pattern: "acme-prod|customer_name"
  - type: absent_regex
    pattern: "TODO|FIXME|placeholder"

I run scripts/run-evals.py. Two cases pass; timing-constraint fails. The agent is not waiting before verification. I add an explicit “wait 30 seconds after a full-sync” line to the Constraints section, re-run: all three cases pass.

Phase 4 — Ship

Symlinked into ~/.claude/skills/ via make install. First history entry: history/v0_evidence.md. PR opened with exploration file, v0 evidence, and eval screenshot. Two reviewers approve; the PR merges.

First evolution round

Six weeks pass. Forty-seven invocations. Twelve failures. I run make evolve SKILL=diagnose-product-status WINDOW=6w.

The pipeline produces four evidence groups:

  1. Group A (5 sessions): routes to bdc-team for failures that are actually upstream ops-team outages (distinguishing symptom: entire pipeline stalled, not just this product).
  2. Group B (3 sessions): the 30-second timing constraint is too aggressive for new-landscape tenants, where sync completes in 8-12s.
  3. Group C (2 sessions): the ambiguous-tenant case is now handleable autonomously — the tenant directory API has added a last_active_at field.
  4. Group D (2 sessions): flaky, no coherent pattern. Evolver ignores by design.

The evolver proposes three edits, one per actionable group. The validator runs both tracks and returns accept. I review — five minutes, because the evidence file is clear. I merge.

The history ledger records: history/v1.md (snapshot of the pre-edit skill), history/v1_evidence.md (the structured evidence file). The skill is now at v2.0.0 (MAJOR bump because Group A’s routing change is a behaviour change users can observe). Three new eval assertions added, one per proposal group.

Appendix D — Migration Playbook

A team with an existing collection of skills cannot adopt the full framework in a single afternoon. The migration is deliberate, staged, and reversible.

This playbook assumes the existing library is around ten to fifty skills, in git, with per-skill directories but no manifest, no evals, no version fields, and no history ledger.

The playbook is ordered by increasing invasiveness. Steps 1-3 are safe and produce immediate value; steps 4-6 change how the team works; steps 7-8 open the evolution loop and depend on trust that has to accumulate.

Step 1 — Inventory (day 1)

Before changing anything, produce a list of every skill in the library, its owner, its rough purpose in one sentence, and its state (active, deprecated-but-not-marked, abandoned). Resolve ambiguities — skills no one remembers writing, duplicate skills — before proceeding.

Deliverable: docs/library-inventory.md, checked in.

Step 2 — Adopt the frontmatter schema (week 1)

For each active skill, add the extended frontmatter with: name (from directory name), version (1.0.0 for every skill on adoption day — a convention, not a claim about maturity), layer (read each skill briefly; use Chapter 4’s definitions), status (stable for active, deprecated for abandoned), owners (from the inventory), description (reuse existing; do not rewrite yet).

Skip fields that require thought: depends_on, auth.providers, deprecated_by. Fill them lazily.

Deliverable: every active skill has frontmatter with the minimum fields.

Step 3 — Adopt the manifest (week 2)

Generate manifest.yaml alongside each SKILL.md using scripts/backfill-manifests.py. Enable the CI check that enforces manifest-frontmatter consistency. This is the first step that gates commits.

Deliverable: every active skill has a manifest; CI enforces consistency.

Step 4 — Declare dependencies (weeks 3-4)

Read every skill and populate depends_on. This is manual work and it is where the team’s implicit knowledge becomes explicit. The exercise surfaces surprises: skills that were thought to be independent turn out to share a connector; “connectors” that turn out to be primitives because they compose two systems.

Enable the resolver’s composition check at the end of this step. From this point forward, a commit that introduces a cycle or a layer violation cannot merge.

Deliverable: the dependency graph is complete and machine-checkable; make graph produces a Mermaid diagram.

Step 5 — Write evals for the top-quartile skills (weeks 5-8)

Start with the skills the team invokes most often. For each, follow the Chapter 9 discipline: collect three to five historical failure cases, each becoming an absent_regex; write two to three positive cases with substring and regex; add a Self-Test section; wire evals into CI.

Skills below the top quartile can wait. The pattern is to add evals as failures occur.

Deliverable: the top quartile has eval coverage; CI runs evals on every commit.

Step 6 — Establish the history ledger (week 9)

Add history/v0_evidence.md to every active skill, retrospectively. The content is minimal — a few sentences summarising why the skill exists. Add scripts/record-history.py to the repository.

Deliverable: every skill has a history/ directory with at least v0_evidence.md.

Step 7 — Run the evolution loop for one skill (weeks 10-12)

Pick one high-value, high-activity skill. Enable trace capture. Let two weeks of usage accumulate. Run make evolve SKILL=<pick> WINDOW=2w. Review the proposal and either accept or reject. Do this for two or three iterations on the same skill.

Deliverable: at least one skill has gone through the evolution loop end-to-end; the team knows the shape of the process.

Step 8 — Widen the loop (month 4 onwards)

Adopt evolution for more skills, one or two at a time. Move well-understood skills to auto-accept as trust accumulates. Enable fleet-wide runs (make evolve-all) once the team is comfortable reviewing aggregate output.

Anti-patterns to avoid during migration

Big-bang adoption. The team’s attention runs out before the discipline is internalised, and the shortcuts taken during rush-adoption become debt. Ship one step at a time.

Skipping the inventory. The inventory step surfaces surprises that change the plan in every migration. Do not skip.

Retro-writing elaborate v0 evidence files. Write them short. The effort is better spent on step-5 evals and step-7 loop runs.

Enforcing every check at once. The CI checks land incrementally. Enforcing everything on day 1 blocks every commit.

Migrating deprecated skills. Delete them, not migrate them. If they turn out to be needed, un-delete from git history.

Realistic timing

For a ten-person team with fifty skills: Steps 1-3 take three to four weeks. Step 4 (dependency declaration) takes another three to four weeks — it is the hardest step because it forces classification decisions. Step 5 takes six to eight weeks. Steps 6-7 take four to five weeks combined. Step 8 is ongoing.

Total: roughly four to five months from start to end-of-playbook.

Glossary (Selected Terms)

Absent regex. An eval assertion that fires when a given pattern does not appear in the trace. Encodes pitfalls: each entry is a mistake that was made once and must not be made again.

Connector. A skill whose role is to expose a single external system in a form the agent can compose. Depends on the substrate; depended on by primitives and workflows.

Dependency graph. The directed graph whose nodes are skills and whose edges are declared depends_on relationships. Enforced by the resolver.

Description. The routing signal the harness uses to decide which skill applies to a user’s request. A skill’s most important frontmatter field.

Evolver. The agent (usually another Claude Code invocation with a specialised system prompt) that reads evidence groups, proposes edits to a skill, and hands the proposals to the validator.

Forge, forging. The four-phase process by which a skill comes into being: Explore, Crystallize, Harden, Ship.

History ledger. The sequence of history/vN.md and history/vN_evidence.md files that record a skill’s evolution over time. The library’s institutional memory.

Invariant track. The half of dual-track validation that checks a proposed change against the skill’s existing eval suite.

Pitfall. A documented failure mode that a skill must avoid. Recorded during Explore, encoded as an absent_regex assertion in Harden.

Primitive. A skill whose role is to encapsulate a reusable domain operation. Composes connectors; consumed by workflows.

Rot. The tendency of a skill to become less correct over time as its environment changes.

Skill. A versioned, executable description of how an agent should accomplish a recurring task in a specific environment.

Substrate. A synonym for infrastructure layer, used when emphasising the “everything rests on this” property. sigcli is the substrate for authentication.

Target track. The half of dual-track validation that checks a proposed change against synthetic cases derived from the evidence that motivated the change.

Workflow. A skill whose role is to compose primitives to solve a specific user-facing problem. The top tier of the four-role hierarchy.

Series footer:

Part 6 of 6 — Skill Engineering
Part 1: Foundations · Part 2: Creation · Part 3: Governance · Part 4: Evolution · Part 5: Outlook · Part 6: Appendices