SPB Git

spb/ultra-sharp-agent-skills Public

Ultra-Sharp Agent Skills — a research-first skill-authoring system + 72 production-ready skills for AI agents.

aiagentsskillsclaude
Python 100%
7.4 KB
View source Raw Blame History

# VALIDATION-REPORT.md — Test Prompts, Outputs, and Checklist Results

Author: Simon-Pierre Boucher Contact: contact@spboucher.ai Date: 2026-08-05

Validation of the two Phase-2 example skills per the CLAUDE.md protocol, plus mechanical verification of the whole repository with tools/validate_skills.py (the deterministic linter that enforces the Sharp Skill Checklist).

# Skill 1 — profiling-csv-data (deterministic / verifiable)

# Test prompts

# Prompt Expected Result
1 "Profile customers.csv and tell me about its data quality" TRIGGER ✅ matches "profile", "data quality", "csv" in description
2 "What columns and missing values are in validation/fixtures/customers.csv?" TRIGGER ✅ matches "columns", "missing/null values", "CSV"
3 "Convert customers.csv to JSON" NO TRIGGER ✅ description explicitly excludes "converting, transforming" — falls through to processing-csv / direct answer

# Execution evidence (real runs, not simulated)

  • python3 skill-1-profiling-csv-data/scripts/profile_csv.py validation/fixtures/customers.csv → valid JSON: 8 rows, 6 columns, types inferred correctly (integer/string/date/float), 1 null in age, 1 null in city, 1 duplicate row detected. ✅
  • Rendered report per references/report-format.md: see validation/fixtures/customers-profile.md — verdict rule 2 applied (🟡, 3 issues), naming rule respected (customers.csv → customers-profile.md). ✅
  • Edge cases:
    • ragged.csv → ragged_rows: 2 detected ✅
    • headeronly.csv → rows: 0, exit 0 (→ ⚠️ verdict path) ✅
    • missing.csv → clear error on stderr, exit code 2 ✅

# Sharp Skill Checklist

Item Result Note
A1 what + when in description ✅
A2 literal user phrases ✅ "profile", "what's in this CSV", "duplicate rows"
A3 third person, directive, ≤1024 chars ✅ 441 chars
A4 no false-positive neighbors ✅ conversion/editing/plotting explicitly excluded
A5 name gerund, lowercase-hyphens ✅ profiling-csv-data
B1 body <500 lines ✅ 44 lines
B2–B3 no known-content, one default ✅ script is the single source of truth
B4 checklist workflow + validation loop ✅ Step 5 validates rendering
B5 output fully specified ✅ template + <stem>-profile.md naming + destination
B6 examples where format matters ✅ exact template in reference file
B7 consistent terms, no time-sensitive info ✅
B8 failure modes addressed ✅ missing/empty/ragged/encoding
C1 references one level deep ✅ one reference file
C2 TOC for >100-line references ✅ (N/A) reference is 62 lines
C3 execute-vs-read explicit ✅ "execute, do not read"
C4 scripts solve-don't-defer, constants justified ✅ all 4 constants commented; errors handled
C5 dependencies stated ✅ stdlib-only
D1–D4 tests incl. negative ✅ above
E1 author headers ✅ linter-verified

Initial failures found and fixed during validation:

  1. Linter flagged name != folder because CLAUDE.md mandates skill-1- folder prefixes — resolved by encoding the project convention into the linter (frontmatter name must equal folder minus skill-N- prefix). Re-checked: pass.

# Skill 2 — writing-release-notes (stylistic / subjective)

# Test prompts

# Prompt Expected Result
1 "Write release notes for v2.8 from these commits: [list]" TRIGGER ✅ matches "write release notes"
2 "Draft the what's new section for our app update" TRIGGER ✅ matches "'what's new' section", "version announcement"
3 "Write a commit message for this diff" NO TRIGGER ✅ description explicitly excludes git commit messages

# Simulated run (prompt 1, commit list from references/examples.md)

Input: 6 commits (SAML SSO, query-planner perf, avatar NPE fix, CI bump, billing refactor, breaking /v1/auth removal). Output produced by following the skill:

markdown
## Acme 2.8.0 — 2026-08-05

> **Breaking:** The legacy `/v1/auth` endpoint is removed. Switch integrations to `/v2/auth` before upgrading.

Faster searches and single sign-on headline this release.

### Added
- Sign in with your company account: SAML single sign-on is now available on all Team plans.

### Improved
- Search is dramatically faster — most queries now return in about 0.2 seconds instead of nearly a second.

### Fixed
- Fixed a crash when opening a profile that has no avatar.

Verified against the five house rules: benefit-first ✅, plain language ✅ (no "leverage/robust/seamless", no ticket IDs), exact three-heading grouping ✅, breaking change at top as blockquote ✅, noise cut ✅ (CI bump and refactor correctly absent).

# Sharp Skill Checklist

Item Result Note
A1–A3 description what+when, literal phrases, 3rd person, ≤1024 ✅ 418 chars
A4 no false positives ✅ commit messages / PR descriptions / API docs excluded
A5 name ✅ writing-release-notes (gerund)
B1 body <500 lines ✅ 68 lines
B3 one default ✅ single format, single grouping scheme
B4 workflow + self-review loop ✅ step 5 re-checks the five rules
B5 output fully specified ✅ template + delivery rule (chat vs CHANGELOG.md)
B6 positive AND negative examples ✅ in body + references/examples.md
B8 edge cases ✅ no user-visible changes / ambiguity / unknown version
C1–C2 references one level deep, TOC ✅ examples.md has TOC
C3–C4 scripts ✅ (N/A) prose-only skill by design (subjective domain)
D1–D4 tests incl. negative ✅ above
E1 headers ✅ linter-verified

Boundary-condition check (mandated for the stylistic skill): the skill defines both an exclusion list AND a tie-breaker procedure (ask "end users or engineers?") for ambiguous audience — pass.

# Repository-wide mechanical validation

python3 tools/validate_skills.py . enforces: frontmatter parses; name rules (≤64 chars, charset, reserved words, folder match); description rules (≤1024, "Use when" trigger clause, "Do not use" boundary, third person, no XML tags); author header in every .md/.py/.sh file; body ≤500 lines; no broken reference links; no backslash paths.

# Final run (after all six collections were built)

text
$ python3 tools/validate_skills.py .
checked 72 skill(s)
all checks passed

Coverage: 2 example skills (Phase 2), 10 doc-skills, 10 frontend-skills, 10 db-skills, 20 backend-skills, 10 writing-skills, 10 finance-skills — 72 skills, 157 markdown/Python files, every one carrying the author header and every description carrying both a "Use when …" trigger clause and a "Do not use …" boundary.

Failures caught by the linter during the build (all fixed):

  1. name vs mandated skill-N- folder prefix conflict → linter updated to encode the project convention (see Skill 1 section).
  2. Several transient broken-reference failures while build agents were mid-write → re-checked after completion; all resolved.

Trigger de-confliction: each collection README documents a pairwise boundary table; every skill description names its nearest neighboring intent and explicitly excludes it, so no request should plausibly fire two skills.