SPB Git

spb/ultra-sharp-agent-skills Public

Ultra-Sharp Agent Skills — a research-first skill-authoring system + 72 production-ready skills for AI agents.

aiagentsskillsclaude
Python 100%

Ultra-Sharp Agent Skills: research-first authoring system + 72 skills

Research synthesis (15 principles, Sharp Skill Checklist, SKILL.md
template), 2 validated example skills, 7 collections (documents,
frontend design, databases, backend, writing, US/CA tax & accounting),
a stdlib-only linter enforcing the checklist repo-wide, and a
validation report with real execution evidence.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 5 days ago (Aug 5, 2026)

Showing 161 changed files with +14,370 and −0

added .gitignore +2 −0
@@ -0,0 +1,2 @@
1 +.claude/
2 +.DS_Store
added CLAUDE.md +103 −0
@@ -0,0 +1,103 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# CLAUDE.md — Skill Authoring Protocol (Research-First, Ultra-Sharp Skills)
7 +
8 +**Author:** Simon-Pierre Boucher
9 +**Contact:** contact@spboucher.ai
10 +
11 +---
12 +
13 +## Mission
14 +
15 +You (Claude) will become an expert in writing **skills for AI agents** — skills that are *ultra sharp* and *fine-pointed*: minimal surface, maximal precision, zero ambiguity, perfect triggering. You will do this in two mandatory phases: **(1) Ultra-intensive web research**, then **(2) Build 2 example skills in English to validate what you learned.**
16 +
17 +Do NOT skip Phase 1. Do NOT write any skill before the research is complete and synthesized.
18 +
19 +---
20 +
21 +## Phase 1 — Ultra-Intensive Web Research (MANDATORY FIRST)
22 +
23 +Perform a deep, exhaustive web research campaign (minimum 10–15 distinct searches + page fetches) on how to write high-quality skills for AI agents. Cover ALL of the following angles:
24 +
25 +1. **Official Anthropic documentation on Agent Skills** — SKILL.md format, YAML frontmatter (`name`, `description`), progressive disclosure (metadata → body → bundled resources), folder anatomy (`scripts/`, `references/`, `assets/`).
26 +2. **Skill triggering mechanics** — how descriptions drive activation, why skills under-trigger, how to write "pushy" but precise descriptions that fire on the right user phrases and never on the wrong ones.
27 +3. **Best practices & anti-patterns** — ideal SKILL.md length (<500 lines), when to split into reference files, table of contents for large references, "principle of least surprise," deterministic scripts vs. prose instructions.
28 +4. **Prompt engineering research applied to skills** — instruction clarity, positive/negative examples, output-format specification, step ordering, failure-mode handling.
29 +5. **Community & ecosystem knowledge** — GitHub repos of real skills (e.g., anthropics/skills), blog posts, engineering write-ups, comparisons with OpenAI/LLM tool-use instructions, MCP-adjacent patterns.
30 +6. **Evaluation & iteration** — how to test a skill (trigger evals, task evals), how to measure trigger rate, how to iterate on descriptions without overfitting.
31 +
32 +### Research rules
33 +- Use `web_search` broadly first, then `web_fetch` the highest-quality primary sources (Anthropic docs, official repos, engineering blogs).
34 +- Take structured notes as you go. Prefer primary sources over aggregators.
35 +- At the end of Phase 1, produce a written synthesis file: **`RESEARCH-SYNTHESIS.md`** containing:
36 + - The 10–15 core principles of an ultra-sharp skill (each in 1–2 sentences).
37 + - A checklist ("Sharp Skill Checklist") you will apply to every skill you write.
38 + - A template of the ideal SKILL.md structure.
39 +- Only when `RESEARCH-SYNTHESIS.md` is complete may you proceed to Phase 2.
40 +
41 +---
42 +
43 +## Phase 2 — Build 2 Example Skills (in English) to Test the Method
44 +
45 +Using ONLY the principles from your synthesis, create **two complete example skills, written entirely in English**, each in its own folder with a proper `SKILL.md`:
46 +
47 +### Skill 1 — Deterministic / verifiable domain
48 +A skill with objectively checkable output (e.g., data extraction, file transformation, structured report generation). It must include:
49 +- YAML frontmatter with a sharp, trigger-optimized `description`.
50 +- A step-by-step workflow with explicit output format.
51 +- At least one bundled resource (`references/` or `scripts/`) demonstrating progressive disclosure.
52 +
53 +### Skill 2 — Stylistic / subjective domain
54 +A skill governing style or judgment (e.g., a house writing style, a code-review playbook). It must include:
55 +- Positive AND negative examples (do this / never do this).
56 +- Clear boundary conditions: when the skill applies and when it must NOT.
57 +
58 +### Validation step
59 +For each skill, after writing it:
60 +1. Write 3 realistic test prompts (some that SHOULD trigger it, at least one that should NOT).
61 +2. Simulate/run the skill against the triggering prompts and show the outputs.
62 +3. Apply your "Sharp Skill Checklist" line by line and report pass/fail.
63 +4. Fix anything that fails, then re-check.
64 +
65 +---
66 +
67 +## Non-Negotiable File Header Rule
68 +
69 +**EVERY file you create in this project** — `CLAUDE.md`, `RESEARCH-SYNTHESIS.md`, every `SKILL.md`, every reference file, every script — MUST begin with this header (adapted to the file's comment syntax):
70 +
71 +```
72 +Author: Simon-Pierre Boucher
73 +Contact: contact@spboucher.ai
74 +```
75 +
76 +- Markdown files: use an HTML comment block or visible header lines at the very top.
77 +- Python/shell scripts: use `#` comment lines at the very top (after any shebang).
78 +- YAML frontmatter files (SKILL.md): place the header comment immediately after the frontmatter block, or as `#` comments before it if the format allows.
79 +
80 +No file ships without this header. Verify it before delivering anything.
81 +
82 +---
83 +
84 +## Definition of "Ultra Sharp / Fine Point"
85 +
86 +A skill qualifies as ultra sharp only if:
87 +- Its description triggers on the exact intended intents and nothing else.
88 +- Every instruction is actionable — no vague verbs ("handle," "deal with") without a concrete procedure.
89 +- The SKILL.md is as short as possible but no shorter; anything long lives in `references/`.
90 +- Output format is fully specified (structure, naming, location).
91 +- Failure modes and edge cases are addressed explicitly.
92 +- It passed the validation step above.
93 +
94 +---
95 +
96 +## Deliverables Summary
97 +
98 +1. `RESEARCH-SYNTHESIS.md` (with header)
99 +2. `skill-1-<name>/SKILL.md` + resources (all with headers)
100 +3. `skill-2-<name>/SKILL.md` + resources (all with headers)
101 +4. Validation report (test prompts, outputs, checklist results)
102 +
103 +Work in this exact order. Research first. Sharp skills second. Headers everywhere.
added README.md +213 −0
@@ -0,0 +1,213 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +<div align="center">
7 +
8 +# ⚡ Ultra-Sharp Agent Skills
9 +
10 +### A research-first skill-authoring system + 72 production-ready skills for AI agents
11 +
12 +[![Skills](https://img.shields.io/badge/skills-72-blueviolet?style=for-the-badge)](#-the-collections)
13 +[![Collections](https://img.shields.io/badge/collections-7-blue?style=for-the-badge)](#-the-collections)
14 +[![Lint](https://img.shields.io/badge/sharp--skill--lint-passing-brightgreen?style=for-the-badge)](#-quality-gate)
15 +[![Standard](https://img.shields.io/badge/Agent%20Skills-SKILL.md-orange?style=for-the-badge)](https://agentskills.io)
16 +
17 +[![Research First](https://img.shields.io/badge/method-research--first-8A2BE2)](RESEARCH-SYNTHESIS.md)
18 +[![Validated](https://img.shields.io/badge/validation-real%20execution%20evidence-success)](VALIDATION-REPORT.md)
19 +[![Python](https://img.shields.io/badge/tooling-python3%20stdlib%20only-3776AB?logo=python&logoColor=white)](tools/validate_skills.py)
20 +[![Claude Code](https://img.shields.io/badge/works%20with-Claude%20Code-D97757)](https://code.claude.com)
21 +[![Author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-informational)](mailto:contact@spboucher.ai)
22 +
23 +*Every skill: a trigger-optimized description, one default per decision, a validation-loop workflow,*
24 +*and pairwise-exclusive boundaries so no request ever fires two skills.*
25 +
26 +</div>
27 +
28 +---
29 +
30 +## 🧭 What is this?
31 +
32 +**Skills** are folders of instructions (`SKILL.md` + resources) that AI agents load on demand.
33 +This repository was built in two phases, per [CLAUDE.md](CLAUDE.md):
34 +
35 +1. 🔬 **Research** — 20+ primary-source searches & fetches (Anthropic docs, `anthropics/skills`, engineering blogs, eval guides) distilled into **[RESEARCH-SYNTHESIS.md](RESEARCH-SYNTHESIS.md)**: 15 core principles, the **Sharp Skill Checklist**, and the ideal SKILL.md template.
36 +2. 🗡️ **Build** — 72 skills across 7 collections, each mechanically verified by a custom linter and validated with positive **and negative** trigger tests ([VALIDATION-REPORT.md](VALIDATION-REPORT.md)).
37 +
38 +## 🚀 Quickstart
39 +
40 +```bash
41 +# install any skill into Claude Code (personal scope)
42 +cp -r doc-skills/processing-pdf ~/.claude/skills/
43 +
44 +# or project scope
45 +cp -r backend-skills/designing-rest-apis .claude/skills/
46 +
47 +# verify the whole repo (stdlib only, no dependencies)
48 +python3 tools/validate_skills.py .
49 +# checked 72 skill(s)
50 +# all checks passed
51 +```
52 +
53 +---
54 +
55 +## 📚 The Collections
56 +
57 +### 📄 doc-skills — Document Processing ![10 skills](https://img.shields.io/badge/skills-10-blue)
58 +
59 +Create / read / modify every major document type. One default library per operation, re-parse validation after every write.
60 +
61 +| Skill | What it does |
62 +|---|---|
63 +| 📊 [`processing-xlsx`](doc-skills/processing-xlsx/) | Excel workbooks with openpyxl — values, formulas, formatting, multi-sheet |
64 +| 📝 [`processing-docx`](doc-skills/processing-docx/) | Word documents with python-docx — paragraphs, tables, styles + pandoc reading |
65 +| 🎞️ [`processing-pptx`](doc-skills/processing-pptx/) | PowerPoint decks with python-pptx — slides, layouts, tables, charts |
66 +| 📕 [`processing-pdf`](doc-skills/processing-pdf/) | Extract text/tables, create, merge, split, watermark, encrypt, fill forms |
67 +| 🧾 [`processing-json`](doc-skills/processing-json/) | Create, modify, validate, and query JSON / JSON Lines |
68 +| 🗂️ [`processing-csv`](doc-skills/processing-csv/) | CSV/TSV create, edit, convert to/from JSON |
69 +| 🏷️ [`processing-xml`](doc-skills/processing-xml/) | XML create, edit, validate, XPath queries — namespace-safe |
70 +| ⚙️ [`processing-yaml`](doc-skills/processing-yaml/) | YAML configs — safe_load discipline, round-trip edits, Norway-problem proof |
71 +| ✍️ [`processing-markdown`](doc-skills/processing-markdown/) | Markdown create, restructure, convert via pandoc |
72 +| 🌐 [`processing-html`](doc-skills/processing-html/) | Local HTML — extract text/tables/links, edit with BeautifulSoup |
73 +
74 +### 🎨 frontend-skills — Frontend Design ![10 skills](https://img.shields.io/badge/skills-10-ff69b4)
75 +
76 +Grounded in Anthropic's `frontend-design` skill + 2026 standards (WCAG 2.2 AA · LCP ≤2.5s · INP ≤200ms · CLS ≤0.1).
77 +
78 +| Skill | What it does |
79 +|---|---|
80 +| 🏗️ [`structuring-semantic-html`](frontend-skills/structuring-semantic-html/) | Landmarks, heading hierarchy, meaning-bearing markup, SEO/OG meta |
81 +| 📐 [`designing-responsive-layouts`](frontend-skills/designing-responsive-layouts/) | Mobile-first flexbox/grid, container queries, clamp() fluid sizing |
82 +| 🎨 [`theming-design-tokens`](frontend-skills/theming-design-tokens/) | Token tiers, dark mode, spacing scales, contrast-safe palettes |
83 +| 🔤 [`choosing-typography`](frontend-skills/choosing-typography/) | Typeface pairing per brief, modular scales, font loading |
84 +| ♿ [`ensuring-accessibility`](frontend-skills/ensuring-accessibility/) | WCAG 2.2 AA audits — keyboard, focus, ARIA, contrast, targets |
85 +| ✨ [`crafting-ui-animations`](frontend-skills/crafting-ui-animations/) | Purposeful motion, transform/opacity only, reduced-motion always |
86 +| 🧾 [`designing-forms`](frontend-skills/designing-forms/) | Labels, validation timing, error states, multi-step patterns |
87 +| ⚛️ [`building-react-components`](frontend-skills/building-react-components/) | Props APIs, composition, state placement, hooks |
88 +| 🚀 [`optimizing-web-performance`](frontend-skills/optimizing-web-performance/) | Core Web Vitals budgets — images, fonts, splitting, measurement |
89 +| 🎯 [`creating-landing-pages`](frontend-skills/creating-landing-pages/) | Hero thesis, CTA rhythm, one signature element, benefit-first copy |
90 +
91 +### 🗄️ db-skills — Database Management ![10 skills](https://img.shields.io/badge/skills-10-336791)
92 +
93 +PostgreSQL-first with MySQL/SQLite deviations noted. Measure before changing anything.
94 +
95 +| Skill | What it does |
96 +|---|---|
97 +| 🔍 [`writing-sql-queries`](db-skills/writing-sql-queries/) | Correct, readable, injection-safe SQL — CTEs, window functions, NULL traps |
98 +| 📋 [`designing-database-schemas`](db-skills/designing-database-schemas/) | 3NF-first modeling, keys, types, constraints, naming |
99 +| ⚡ [`optimizing-sql-performance`](db-skills/optimizing-sql-performance/) | EXPLAIN ANALYZE, indexing strategy, N+1, keyset pagination |
100 +| 🔀 [`managing-database-migrations`](db-skills/managing-database-migrations/) | Versioned, immutable, reversible; zero-downtime expand→contract |
101 +| 💾 [`backing-up-databases`](db-skills/backing-up-databases/) | Scheduled backups, PITR, 3-2-1 rule, restore drills |
102 +| 🔐 [`securing-databases`](db-skills/securing-databases/) | Least-privilege roles, parameterized-only queries, TLS, RLS, auditing |
103 +| 🐘 [`administering-postgresql`](db-skills/administering-postgresql/) | Roles, config knobs with starting values, autovacuum, monitoring |
104 +| 🪶 [`managing-sqlite`](db-skills/managing-sqlite/) | WAL mode, pragmas, bulk-insert transactions, safe backups |
105 +| 📦 [`modeling-nosql-data`](db-skills/modeling-nosql-data/) | Access-pattern-first design, embed vs reference, Redis key design |
106 +| 🚨 [`troubleshooting-databases`](db-skills/troubleshooting-databases/) | 5-stage incident triage runbook with the exact query per stage |
107 +
108 +### 🔧 backend-skills — Backend Development ![20 skills](https://img.shields.io/badge/skills-20-success)
109 +
110 +| Skill | What it does |
111 +|---|---|
112 +| 🛣️ [`designing-rest-apis`](backend-skills/designing-rest-apis/) | Resources, status codes, pagination, versioning, problem+json, OpenAPI |
113 +| 🕸️ [`designing-graphql-apis`](backend-skills/designing-graphql-apis/) | Schema-first, DataLoader vs N+1, cursor connections, complexity limits |
114 +| 📡 [`designing-webhooks`](backend-skills/designing-webhooks/) | HMAC signing, backoff retries, idempotent delivery, DLQ |
115 +| 📤 [`handling-file-uploads`](backend-skills/handling-file-uploads/) | Presigned URLs, magic-byte validation, streaming, resumable uploads |
116 +| 🔑 [`implementing-authentication`](backend-skills/implementing-authentication/) | argon2id, sessions vs JWT, OAuth2/OIDC + PKCE, MFA, safe resets |
117 +| 🛂 [`implementing-authorization`](backend-skills/implementing-authorization/) | RBAC/ABAC, deny-by-default, IDOR prevention, tenant isolation |
118 +| 🧰 [`validating-input`](backend-skills/validating-input/) | Boundary schemas, allowlists, canonicalization, structured 422s |
119 +| 🛡️ [`securing-backend-services`](backend-skills/securing-backend-services/) | Security headers, TLS, secrets, SSRF/CSRF defenses, safe errors |
120 +| 💥 [`handling-errors`](backend-skills/handling-errors/) | Error taxonomy, problem+json, retries with jitter, circuit breakers |
121 +| 🔭 [`instrumenting-observability`](backend-skills/instrumenting-observability/) | Structured logs, correlation IDs, RED metrics, OpenTelemetry |
122 +| 🧪 [`testing-backend-services`](backend-skills/testing-backend-services/) | Test pyramid, testcontainers, contract tests, zero flake |
123 +| 🎛️ [`managing-configuration`](backend-skills/managing-configuration/) | 12-factor env config, typed startup validation, secret separation |
124 +| ⏳ [`writing-background-jobs`](backend-skills/writing-background-jobs/) | Idempotent jobs, backoff, DLQ, timeouts, queue observability |
125 +| 📨 [`handling-async-messaging`](backend-skills/handling-async-messaging/) | Transactional outbox, idempotent consumers, schema versioning |
126 +| 🗃️ [`caching-strategies`](backend-skills/caching-strategies/) | Cache-aside, TTL discipline, invalidation, stampede protection |
127 +| 🚦 [`limiting-request-rates`](backend-skills/limiting-request-rates/) | Token bucket, 429 + Retry-After, tiered limits, load shedding |
128 +| 🐳 [`containerizing-services`](backend-skills/containerizing-services/) | Multi-stage builds, non-root, pinned images, healthchecks |
129 +| 🚢 [`shipping-with-ci-cd`](backend-skills/shipping-with-ci-cd/) | Build-once promotion, canary/blue-green, automated rollback |
130 +| 🧩 [`architecting-service-boundaries`](backend-skills/architecting-service-boundaries/) | Modular monolith default, data ownership, sagas, strangler |
131 +| 📈 [`scaling-backend-services`](backend-skills/scaling-backend-services/) | Statelessness, pooling, load balancing, autoscaling on the right metric |
132 +
133 +### ✒️ writing-skills — Writing ![10 skills](https://img.shields.io/badge/skills-10-yellow)
134 +
135 +Stylistic skills: every rule ships with a ✅/❌ pair; every workflow ends in a self-review pass.
136 +
137 +| Skill | What it does |
138 +|---|---|
139 +| 📖 [`writing-technical-documentation`](writing-skills/writing-technical-documentation/) | READMEs, architecture docs, runbooks — quickstart first, scannable |
140 +| 🔌 [`writing-api-documentation`](writing-skills/writing-api-documentation/) | Endpoint reference with runnable examples, errors documented like successes |
141 +| 🧑‍🏫 [`writing-tutorials`](writing-skills/writing-tutorials/) | Step-by-step guides — every step has a verifiable checkpoint |
142 +| 📰 [`writing-blog-posts`](writing-skills/writing-blog-posts/) | One specific claim, a real hook, real code and numbers |
143 +| 🤖 [`writing-agent-skills`](writing-skills/writing-agent-skills/) | **The meta-skill** — author SKILL.md files with the Sharp Skill method |
144 +| 📧 [`writing-professional-emails`](writing-skills/writing-professional-emails/) | Ask-first structure, decision-ready subject lines |
145 +| 📊 [`writing-executive-summaries`](writing-skills/writing-executive-summaries/) | BLUF, one page, numbers over adjectives, a clear pick |
146 +| 🤝 [`writing-proposals`](writing-skills/writing-proposals/) | Their problem first, scope exclusions, tiered pricing |
147 +| 🗒️ [`writing-meeting-notes`](writing-skills/writing-meeting-notes/) | Decisions & actions first, every action has owner + deadline |
148 +| 🪄 [`editing-and-proofreading`](writing-skills/editing-and-proofreading/) | Four fixed passes: structure → paragraphs → sentences → mechanics |
149 +
150 +### 💰 finance-skills — US/Canada Tax & Accounting ![10 skills](https://img.shields.io/badge/skills-10-gold)
151 +
152 +🇺🇸 🇨🇦 Tax-year-2026 figures with official verification sources. **Educational, legal planning only** — every skill refuses evasion and refers complex cases to a CPA.
153 +
154 +| Skill | What it does |
155 +|---|---|
156 +| 🧾 [`preparing-us-personal-tax-returns`](finance-skills/preparing-us-personal-tax-returns/) | Form 1040 — documents, schedules, deadlines, safe harbors |
157 +| 🍁 [`preparing-canadian-personal-tax-returns`](finance-skills/preparing-canadian-personal-tax-returns/) | T1 — slips, deductions vs credits, NETFILE, instalments |
158 +| 💸 [`optimizing-us-personal-taxes`](finance-skills/optimizing-us-personal-taxes/) | 401(k)/IRA/HSA priority, Roth decisions, loss harvesting, bunching |
159 +| 🇨🇦 [`optimizing-canadian-personal-taxes`](finance-skills/optimizing-canadian-personal-taxes/) | RRSP vs TFSA vs FHSA, legal splitting, superficial-loss rule |
160 +| 🌉 [`handling-cross-border-taxation`](finance-skills/handling-cross-border-taxation/) | Residency tests, treaty tie-breakers, FBAR/8938, TFSA-for-US-persons trap |
161 +| 📒 [`bookkeeping-for-small-businesses`](finance-skills/bookkeeping-for-small-businesses/) | Double-entry, chart of accounts, monthly close, reconciliation |
162 +| 📑 [`preparing-financial-statements`](finance-skills/preparing-financial-statements/) | Income statement, balance sheet, cash flow — with cross-statement ties |
163 +| 🏢 [`filing-us-business-taxes`](finance-skills/filing-us-business-taxes/) | Schedule C / 1065 / 1120-S / 1120, estimated taxes, payroll, 1099s |
164 +| 🏬 [`filing-canadian-business-taxes`](finance-skills/filing-canadian-business-taxes/) | T2125, T2, GST/HST, payroll remittances, T4/T5 |
165 +| 🧮 [`optimizing-business-taxes`](finance-skills/optimizing-business-taxes/) | S-corp analysis (US) · SBD protection, salary vs dividends (Canada) |
166 +
167 +### 🏅 The founding examples ![2 skills](https://img.shields.io/badge/skills-2-lightgrey)
168 +
169 +| Skill | What it does |
170 +|---|---|
171 +| 🔬 [`skill-1-profiling-csv-data`](skill-1-profiling-csv-data/) | Deterministic domain — CSV data-quality reports via a bundled stdlib script (executed & verified) |
172 +| 📣 [`skill-2-writing-release-notes`](skill-2-writing-release-notes/) | Stylistic domain — benefit-first release notes with do/never examples |
173 +
174 +---
175 +
176 +## ✅ Quality Gate
177 +
178 +![frontmatter](https://img.shields.io/badge/frontmatter-validated-brightgreen)
179 +![descriptions](https://img.shields.io/badge/descriptions-Use%20when%20%2B%20Do%20not%20use-brightgreen)
180 +![headers](https://img.shields.io/badge/author%20headers-100%25-brightgreen)
181 +![links](https://img.shields.io/badge/reference%20links-0%20broken-brightgreen)
182 +![budget](https://img.shields.io/badge/SKILL.md%20bodies-%E2%89%A4500%20lines-brightgreen)
183 +
184 +[`tools/validate_skills.py`](tools/validate_skills.py) (stdlib-only) mechanically enforces the Sharp Skill Checklist on every skill: valid frontmatter · name rules (≤64 chars, charset, reserved words, folder match) · description rules (≤1024 chars, literal "Use when …" trigger clause, "Do not use …" boundary, third person, no XML tags) · author header in every file · body ≤500 lines · zero broken reference links · forward-slash paths only.
185 +
186 +## 🧬 Design invariants (all 72 skills)
187 +
188 +1. 🎯 **Description = trigger** — WHAT + literal "Use when …" phrases + "Do not use …" boundary
189 +2. 🧱 **Pairwise-exclusive boundaries** per collection — no request can plausibly fire two skills
190 +3. 1️⃣ **One default per decision**, escape hatch named with its install command
191 +4. 🔁 **Every workflow ends with a validation step** — malformed input is reported, never guessed around
192 +5. 🪜 **Progressive disclosure** — lean SKILL.md, depth one level deep in `references/` with a TOC
193 +
194 +## 📁 Repository map
195 +
196 +```text
197 +.
198 +├── RESEARCH-SYNTHESIS.md # the method: 15 principles + Sharp Skill Checklist + template
199 +├── VALIDATION-REPORT.md # trigger tests, execution evidence, checklist results
200 +├── skill-1-profiling-csv-data/ skill-2-writing-release-notes/
201 +├── doc-skills/ (10) ├── frontend-skills/ (10) ├── db-skills/ (10)
202 +├── backend-skills/ (20) ├── writing-skills/ (10) ├── finance-skills/ (10)
203 +├── tools/validate_skills.py # the quality gate
204 +└── validation/ # fixtures + real outputs
205 +```
206 +
207 +<div align="center">
208 +
209 +**Author:** Simon-Pierre Boucher · **Contact:** [contact@spboucher.ai](mailto:contact@spboucher.ai)
210 +
211 +*Built research-first with [Claude Code](https://claude.com/claude-code) — every figure sourced, every skill linted, every trigger tested.*
212 +
213 +</div>
added RESEARCH-SYNTHESIS.md +167 −0
@@ -0,0 +1,167 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# RESEARCH-SYNTHESIS.md — How to Write Ultra-Sharp Skills for AI Agents
7 +
8 +**Author:** Simon-Pierre Boucher
9 +**Contact:** contact@spboucher.ai
10 +**Date:** 2026-08-05
11 +**Basis:** 16 distinct web searches + page fetches across official Anthropic documentation, the anthropics/skills repository, Anthropic engineering/blog posts, and community engineering write-ups. Sources listed at the end.
12 +
13 +---
14 +
15 +## Part 1 — The 15 Core Principles of an Ultra-Sharp Skill
16 +
17 +1. **The description IS the trigger.** At startup only `name` + `description` are loaded (~100 tokens/skill); Claude matches the user's request against the description to decide whether to fire. A vague description is the #1 cause of skill failure.
18 +
19 +2. **A description states WHAT + WHEN, with literal user phrases.** Include both what the skill does and the concrete trigger contexts — the exact words a user would type ("PDFs, forms, document extraction") — because an abstract description won't match a concrete request.
20 +
21 +3. **Write descriptions in third person, directive voice.** Descriptions are injected into the system prompt; "I can help you…" or "You can use this…" causes discovery problems. Directive phrasing ("Use when…") measurably raises activation rates versus purely descriptive phrasing.
22 +
23 +4. **Descriptions live in a shared, finite budget.** In Claude Code all skill descriptions share a character budget (~15,000 chars default) and overflow is silently dropped — so every description must be dense and short, and each skill you add taxes all the others.
24 +
25 +5. **Progressive disclosure is the architecture.** Three levels: metadata (always loaded, ~100 tokens), SKILL.md body (loaded on trigger, keep <500 lines / <5k tokens), bundled resources (zero context cost until read/executed). Design every skill around these levels.
26 +
27 +6. **Concise is key — the context window is a public good.** Assume Claude is already smart: cut anything Claude already knows, and challenge every paragraph with "does this justify its token cost?"
28 +
29 +7. **Match degrees of freedom to task fragility.** Fragile, order-dependent operations get exact scripts and "do not modify" guardrails (low freedom); judgment tasks get heuristics and trust (high freedom). Narrow bridge → rails; open field → direction.
30 +
31 +8. **One default, not a menu.** Give a single recommended tool/approach with an explicit escape hatch for the exception ("Use pdfplumber; for scanned PDFs use OCR instead") — offering many options confuses execution.
32 +
33 +9. **Prefer deterministic scripts over prose for deterministic work.** Bundled scripts are more reliable than generated code, cost zero context (only their output enters context), and must solve errors themselves rather than defer to Claude; no unexplained "voodoo constants."
34 +
35 +10. **Make execution intent explicit.** Say "Run `scripts/x.py`" (execute) vs "See `scripts/x.py` for the algorithm" (read) — ambiguity here wastes tokens or produces re-implementation.
36 +
37 +11. **Keep references one level deep, with a table of contents past ~100 lines.** Nested reference chains get partially read (`head -100`); every reference file should link directly from SKILL.md, and long ones need a TOC so partial reads still reveal scope.
38 +
39 +12. **Fully specify the output.** Templates, exact file naming, destination paths, and input→output example pairs (3–5, covering edge cases) convey format and style better than any amount of description — this is standard prompt-engineering practice applied to skills.
40 +
41 +13. **Use workflows with checklists and validation loops.** Break complex tasks into numbered steps Claude can check off, and close the loop: run validator → fix → re-validate → only then proceed.
42 +
43 +14. **Build evals BEFORE writing extensive documentation.** Establish a baseline without the skill, write ≥3 test scenarios including **negative prompts that must NOT trigger the skill**, run multiple trials (behavior is nondeterministic), and grade outcomes, not paths.
44 +
45 +15. **Iterate from observed behavior, not assumptions.** Use the two-Claude loop (Claude A authors, Claude B executes real tasks), watch how the agent actually navigates the files (ignored files, missed links, over-read sections), and refine the description first whenever triggering misfires.
46 +
47 +**Hygiene constants (from the spec):** `name` ≤64 chars, lowercase/numbers/hyphens only, no reserved words ("anthropic", "claude"), gerund form preferred; `description` ≤1,024 chars, non-empty, no XML tags; consistent terminology throughout; no time-sensitive facts; forward-slash paths only; never assume packages are installed.
48 +
49 +---
50 +
51 +## Part 2 — The Sharp Skill Checklist
52 +
53 +Apply line by line to EVERY skill before delivery.
54 +
55 +### A. Triggering (the point of the spear)
56 +- [ ] A1. Description states WHAT the skill does AND WHEN to use it
57 +- [ ] A2. Description contains the literal key terms/phrases a user would type
58 +- [ ] A3. Description is third person, directive ("Use when…"), no XML tags, ≤1,024 chars
59 +- [ ] A4. Description would NOT match plausible neighboring requests (no false positives)
60 +- [ ] A5. Name is gerund-form (or clear noun phrase), lowercase/hyphens, ≤64 chars, not vague (`helper`, `utils`)
61 +
62 +### B. Body sharpness
63 +- [ ] B1. SKILL.md body <500 lines; anything long lives in `references/`
64 +- [ ] B2. No content Claude already knows; every paragraph earns its tokens
65 +- [ ] B3. Exactly one recommended default per operation, with explicit escape hatch
66 +- [ ] B4. Workflow is numbered steps (checklist if >3 steps); validation loop for quality-critical output
67 +- [ ] B5. Output format fully specified: structure, file naming, destination
68 +- [ ] B6. Concrete input→output examples where style/format matters
69 +- [ ] B7. Consistent terminology; no time-sensitive info; no vague verbs without a procedure
70 +- [ ] B8. Failure modes and edge cases explicitly addressed
71 +
72 +### C. Resources & structure
73 +- [ ] C1. All references link one level deep from SKILL.md; descriptive file names
74 +- [ ] C2. Reference files >100 lines start with a table of contents
75 +- [ ] C3. Scripts: execution vs read-as-reference intent is explicit
76 +- [ ] C4. Scripts handle their own errors ("solve, don't defer"); all constants justified
77 +- [ ] C5. Dependencies stated explicitly (or stdlib-only); forward-slash paths only
78 +
79 +### D. Validation
80 +- [ ] D1. ≥3 test prompts written: triggering AND at least one non-triggering
81 +- [ ] D2. Skill executed/simulated against triggering prompts; outputs verified
82 +- [ ] D3. Negative prompt confirmed NOT to activate the skill
83 +- [ ] D4. Checklist re-run after any fix
84 +
85 +### E. Project rule
86 +- [ ] E1. Every file carries the author header (Author: Simon-Pierre Boucher / Contact: contact@spboucher.ai)
87 +
88 +---
89 +
90 +## Part 3 — Template of the Ideal SKILL.md
91 +
92 +```markdown
93 +---
94 +name: doing-the-thing # gerund, lowercase-hyphens, ≤64 chars
95 +description: <What it does in one clause with key nouns>. Use when the user asks to <literal trigger phrases>, mentions <key terms>, or <concrete context>. Do not use for <nearest non-target intent>.
96 +---
97 +
98 +<!--
99 +Author: Simon-Pierre Boucher
100 +Contact: contact@spboucher.ai
101 +-->
102 +
103 +# Doing the Thing
104 +
105 +## When to use / when NOT to use
106 +- Use for: <precise intents>
107 +- Do NOT use for: <neighboring intents that belong to other skills or plain answers>
108 +
109 +## Workflow
110 +Copy this checklist and check off items as you complete them:
111 +
112 +- [ ] Step 1: <action> (run `scripts/tool.py <args>` — execute, do not read)
113 +- [ ] Step 2: <action>
114 +- [ ] Step 3: Validate: <check>. If it fails, fix and repeat Step 2.
115 +- [ ] Step 4: Produce output exactly per "Output format" below.
116 +
117 +## Output format
118 +Save to `<naming-rule>`. Use this exact structure:
119 +<template block>
120 +
121 +## Examples
122 +**Input:** <realistic input> → **Output:** <exact desired output>
123 +**Input:** <edge case> → **Output:** <exact desired output>
124 +
125 +## Edge cases & failure modes
126 +- <case> → <exact behavior>
127 +- <case> → <exact behavior>
128 +
129 +## References (one level deep)
130 +- Advanced details: see [references/advanced.md](references/advanced.md)
131 +```
132 +
133 +Directory anatomy:
134 +
135 +```text
136 +doing-the-thing/
137 +├── SKILL.md # <500 lines, loaded on trigger
138 +├── references/ # loaded only when needed; TOC if >100 lines
139 +│ └── advanced.md
140 +├── scripts/ # executed via bash; only output enters context
141 +│ └── tool.py
142 +└── assets/ # templates, images, data (optional)
143 +```
144 +
145 +---
146 +
147 +## Sources
148 +
149 +Primary (fetched in full):
150 +- [Skill authoring best practices — Claude Platform Docs](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices.md)
151 +- [Agent Skills overview — Claude Platform Docs](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview)
152 +- [Equipping agents for the real world with Agent Skills — Anthropic Engineering](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)
153 +- [skill-creator SKILL.md — anthropics/skills (GitHub)](https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md)
154 +- [Extend Claude with skills — Claude Code Docs](https://code.claude.com/docs/en/skills)
155 +- [Improving skill-creator: test, measure, and refine Agent Skills — Anthropic blog](https://claude.com/blog/improving-skill-creator-test-measure-and-refine-agent-skills)
156 +- [Practical Guide to Evaluating and Testing Agent Skills — Philipp Schmid](https://www.philschmid.de/testing-skills)
157 +- [Claude Code skills not triggering? It might not see them — blog.fsck.com](https://blog.fsck.com/2025/12/17/claude-code-skills-not-triggering/)
158 +
159 +Secondary (search-level):
160 +- [anthropics/skills repository](https://github.com/anthropics/skills)
161 +- [Why Claude Code Skills Don't Trigger (2026) — DEV Community](https://dev.to/lizechengnet/why-claude-code-skills-dont-trigger-and-how-to-fix-them-in-2026-o7h)
162 +- [How to Activate Claude Skills Automatically — DEV Community](https://dev.to/oluwawunmiadesewa/claude-code-skills-not-triggering-2-fixes-for-100-activation-3b57)
163 +- [Prompt engineering best practices for 2026 — Claude blog](https://claude.com/blog/best-practices-for-prompt-engineering)
164 +- [Testing Agent Skills Systematically with Evals — OpenAI Developers](https://developers.openai.com/blog/eval-skills)
165 +- [Function calling — OpenAI API docs](https://developers.openai.com/api/docs/guides/function-calling)
166 +- [Agent Skills: Progressive Disclosure as a System Design Pattern — SwirlAI](https://www.newsletter.swirlai.com/p/agent-skills-progressive-disclosure)
167 +- [Introducing Task Evals — Tessl](https://tessl.io/blog/introducing-task-evals-measure-whether-your-skills-actually-work/)
added VALIDATION-REPORT.md +166 −0
@@ -0,0 +1,166 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# VALIDATION-REPORT.md — Test Prompts, Outputs, and Checklist Results
7 +
8 +**Author:** Simon-Pierre Boucher
9 +**Contact:** contact@spboucher.ai
10 +**Date:** 2026-08-05
11 +
12 +Validation of the two Phase-2 example skills per the CLAUDE.md protocol, plus
13 +mechanical verification of the whole repository with `tools/validate_skills.py`
14 +(the deterministic linter that enforces the Sharp Skill Checklist).
15 +
16 +---
17 +
18 +## Skill 1 — `profiling-csv-data` (deterministic / verifiable)
19 +
20 +### Test prompts
21 +
22 +| # | Prompt | Expected | Result |
23 +|---|---|---|---|
24 +| 1 | "Profile customers.csv and tell me about its data quality" | TRIGGER | ✅ matches "profile", "data quality", "csv" in description |
25 +| 2 | "What columns and missing values are in validation/fixtures/customers.csv?" | TRIGGER | ✅ matches "columns", "missing/null values", "CSV" |
26 +| 3 | "Convert customers.csv to JSON" | NO TRIGGER | ✅ description explicitly excludes "converting, transforming" — falls through to processing-csv / direct answer |
27 +
28 +### Execution evidence (real runs, not simulated)
29 +
30 +- `python3 skill-1-profiling-csv-data/scripts/profile_csv.py validation/fixtures/customers.csv`
31 + → valid JSON: 8 rows, 6 columns, types inferred correctly (integer/string/date/float),
32 + 1 null in `age`, 1 null in `city`, 1 duplicate row detected. ✅
33 +- Rendered report per `references/report-format.md`: see
34 + [validation/fixtures/customers-profile.md](validation/fixtures/customers-profile.md)
35 + — verdict rule 2 applied (🟡, 3 issues), naming rule respected
36 + (`customers.csv` → `customers-profile.md`). ✅
37 +- Edge cases:
38 + - `ragged.csv` → `ragged_rows: 2` detected ✅
39 + - `headeronly.csv` → `rows: 0`, exit 0 (→ ⚠️ verdict path) ✅
40 + - `missing.csv` → clear error on stderr, exit code 2 ✅
41 +
42 +### Sharp Skill Checklist
43 +
44 +| Item | Result | Note |
45 +|---|---|---|
46 +| A1 what + when in description | ✅ | |
47 +| A2 literal user phrases | ✅ | "profile", "what's in this CSV", "duplicate rows" |
48 +| A3 third person, directive, ≤1024 chars | ✅ | 441 chars |
49 +| A4 no false-positive neighbors | ✅ | conversion/editing/plotting explicitly excluded |
50 +| A5 name gerund, lowercase-hyphens | ✅ | `profiling-csv-data` |
51 +| B1 body <500 lines | ✅ | 44 lines |
52 +| B2–B3 no known-content, one default | ✅ | script is the single source of truth |
53 +| B4 checklist workflow + validation loop | ✅ | Step 5 validates rendering |
54 +| B5 output fully specified | ✅ | template + `<stem>-profile.md` naming + destination |
55 +| B6 examples where format matters | ✅ | exact template in reference file |
56 +| B7 consistent terms, no time-sensitive info | ✅ | |
57 +| B8 failure modes addressed | ✅ | missing/empty/ragged/encoding |
58 +| C1 references one level deep | ✅ | one reference file |
59 +| C2 TOC for >100-line references | ✅ (N/A) | reference is 62 lines |
60 +| C3 execute-vs-read explicit | ✅ | "execute, do not read" |
61 +| C4 scripts solve-don't-defer, constants justified | ✅ | all 4 constants commented; errors handled |
62 +| C5 dependencies stated | ✅ | stdlib-only |
63 +| D1–D4 tests incl. negative | ✅ | above |
64 +| E1 author headers | ✅ | linter-verified |
65 +
66 +**Initial failures found and fixed during validation:**
67 +1. Linter flagged `name != folder` because CLAUDE.md mandates `skill-1-` folder
68 + prefixes — resolved by encoding the project convention into the linter
69 + (frontmatter name must equal folder minus `skill-N-` prefix). Re-checked: pass.
70 +
71 +---
72 +
73 +## Skill 2 — `writing-release-notes` (stylistic / subjective)
74 +
75 +### Test prompts
76 +
77 +| # | Prompt | Expected | Result |
78 +|---|---|---|---|
79 +| 1 | "Write release notes for v2.8 from these commits: [list]" | TRIGGER | ✅ matches "write release notes" |
80 +| 2 | "Draft the what's new section for our app update" | TRIGGER | ✅ matches "'what's new' section", "version announcement" |
81 +| 3 | "Write a commit message for this diff" | NO TRIGGER | ✅ description explicitly excludes git commit messages |
82 +
83 +### Simulated run (prompt 1, commit list from references/examples.md)
84 +
85 +Input: 6 commits (SAML SSO, query-planner perf, avatar NPE fix, CI bump,
86 +billing refactor, breaking `/v1/auth` removal).
87 +Output produced by following the skill:
88 +
89 +```markdown
90 +## Acme 2.8.0 — 2026-08-05
91 +
92 +> **Breaking:** The legacy `/v1/auth` endpoint is removed. Switch integrations to `/v2/auth` before upgrading.
93 +
94 +Faster searches and single sign-on headline this release.
95 +
96 +### Added
97 +- Sign in with your company account: SAML single sign-on is now available on all Team plans.
98 +
99 +### Improved
100 +- Search is dramatically faster — most queries now return in about 0.2 seconds instead of nearly a second.
101 +
102 +### Fixed
103 +- Fixed a crash when opening a profile that has no avatar.
104 +```
105 +
106 +Verified against the five house rules: benefit-first ✅, plain language ✅
107 +(no "leverage/robust/seamless", no ticket IDs), exact three-heading grouping ✅,
108 +breaking change at top as blockquote ✅, noise cut ✅ (CI bump and refactor
109 +correctly absent).
110 +
111 +### Sharp Skill Checklist
112 +
113 +| Item | Result | Note |
114 +|---|---|---|
115 +| A1–A3 description what+when, literal phrases, 3rd person, ≤1024 | ✅ | 418 chars |
116 +| A4 no false positives | ✅ | commit messages / PR descriptions / API docs excluded |
117 +| A5 name | ✅ | `writing-release-notes` (gerund) |
118 +| B1 body <500 lines | ✅ | 68 lines |
119 +| B3 one default | ✅ | single format, single grouping scheme |
120 +| B4 workflow + self-review loop | ✅ | step 5 re-checks the five rules |
121 +| B5 output fully specified | ✅ | template + delivery rule (chat vs CHANGELOG.md) |
122 +| B6 positive AND negative examples | ✅ | in body + references/examples.md |
123 +| B8 edge cases | ✅ | no user-visible changes / ambiguity / unknown version |
124 +| C1–C2 references one level deep, TOC | ✅ | examples.md has TOC |
125 +| C3–C4 scripts | ✅ (N/A) | prose-only skill by design (subjective domain) |
126 +| D1–D4 tests incl. negative | ✅ | above |
127 +| E1 headers | ✅ | linter-verified |
128 +
129 +**Boundary-condition check (mandated for the stylistic skill):** the skill
130 +defines both an exclusion list AND a tie-breaker procedure (ask "end users or
131 +engineers?") for ambiguous audience — pass.
132 +
133 +---
134 +
135 +## Repository-wide mechanical validation
136 +
137 +`python3 tools/validate_skills.py .` enforces: frontmatter parses; name rules
138 +(≤64 chars, charset, reserved words, folder match); description rules (≤1024,
139 +"Use when" trigger clause, "Do not use" boundary, third person, no XML tags);
140 +author header in every .md/.py/.sh file; body ≤500 lines; no broken reference
141 +links; no backslash paths.
142 +
143 +### Final run (after all six collections were built)
144 +
145 +```
146 +$ python3 tools/validate_skills.py .
147 +checked 72 skill(s)
148 +all checks passed
149 +```
150 +
151 +Coverage: 2 example skills (Phase 2), 10 doc-skills, 10 frontend-skills,
152 +10 db-skills, 20 backend-skills, 10 writing-skills, 10 finance-skills —
153 +72 skills, 157
154 +markdown/Python files, every one carrying the author header and every
155 +description carrying both a "Use when …" trigger clause and a
156 +"Do not use …" boundary.
157 +
158 +**Failures caught by the linter during the build (all fixed):**
159 +1. `name` vs mandated `skill-N-` folder prefix conflict → linter updated to
160 + encode the project convention (see Skill 1 section).
161 +2. Several transient broken-reference failures while build agents were
162 + mid-write → re-checked after completion; all resolved.
163 +
164 +**Trigger de-confliction:** each collection README documents a pairwise
165 +boundary table; every skill description names its nearest neighboring intent
166 +and explicitly excludes it, so no request should plausibly fire two skills.
added backend-skills/README.md +63 −0
@@ -0,0 +1,63 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# backend-skills — Backend Development Skill Collection
7 +
8 +**Author:** Simon-Pierre Boucher
9 +**Contact:** contact@spboucher.ai
10 +
11 +Twenty ultra-sharp backend skills following the method in
12 +[../RESEARCH-SYNTHESIS.md](../RESEARCH-SYNTHESIS.md). Database-specific work
13 +lives in [../db-skills/](../db-skills/) — this collection deliberately
14 +excludes it. All pass `python3 ../tools/validate_skills.py`.
15 +
16 +## The collection and its boundaries
17 +
18 +**API design**
19 +| Skill | Handles | Explicitly does NOT handle |
20 +|---|---|---|
21 +| `designing-rest-apis` | resources, methods, status codes, pagination, versioning, problem+json, OpenAPI | GraphQL; webhooks; auth mechanics |
22 +| `designing-graphql-apis` | schema design, DataLoader/N+1, cursor pagination, complexity limits | REST; subscriptions infra |
23 +| `designing-webhooks` | payloads, HMAC signing, retries, idempotent delivery | consuming webhooks' business logic; internal messaging |
24 +| `handling-file-uploads` | presigned URLs, validation, safe naming, streaming, resumable | serving files; image processing; CDNs |
25 +
26 +**Security (defensive)**
27 +| Skill | Handles | Explicitly does NOT handle |
28 +|---|---|---|
29 +| `implementing-authentication` | password storage, sessions vs JWT, OAuth2/OIDC + PKCE, MFA, resets | permissions → `implementing-authorization`; DB creds → db-skills |
30 +| `implementing-authorization` | RBAC/ABAC, deny-by-default, IDOR, tenant isolation, policy centralization | login/identity → `implementing-authentication` |
31 +| `validating-input` | boundary schemas, allowlists, canonicalization, 422 responses | authn/authz; deep business rules |
32 +| `securing-backend-services` | headers, TLS, dependencies, secrets, SSRF/CSRF, least privilege | login, permissions, input validation, DB security (see above) |
33 +
34 +**Reliability & operations**
35 +| Skill | Handles | Explicitly does NOT handle |
36 +|---|---|---|
37 +| `handling-errors` | error taxonomy, problem+json mapping, retries, circuit breakers | validation responses; observability pipelines |
38 +| `instrumenting-observability` | structured logs, correlation IDs, RED metrics, OpenTelemetry, alerting | error-handling code; incident process |
39 +| `testing-backend-services` | test pyramid, testcontainers, contract tests, determinism | UI testing; load testing |
40 +| `managing-configuration` | 12-factor env config, startup validation, secrets separation | infra provisioning; CI/CD config |
41 +
42 +**Async & performance**
43 +| Skill | Handles | Explicitly does NOT handle |
44 +|---|---|---|
45 +| `writing-background-jobs` | idempotent jobs, retries/backoff, DLQ, timeouts | inter-service messaging; system cron |
46 +| `handling-async-messaging` | outbox pattern, event versioning, ordering, poison messages | in-process job queues; customer webhooks |
47 +| `caching-strategies` | cache-aside, TTL discipline, invalidation, stampede protection | CDN/edge config; SQL tuning → db-skills |
48 +| `limiting-request-rates` | token bucket, 429/Retry-After, tiered limits, load shedding | authorization; capacity planning |
49 +
50 +**Infrastructure & architecture**
51 +| Skill | Handles | Explicitly does NOT handle |
52 +|---|---|---|
53 +| `containerizing-services` | multi-stage builds, non-root, layer caching, health checks | Kubernetes; pipeline design |
54 +| `shipping-with-ci-cd` | stages, build-once-promote, deploy strategies, rollback | writing tests; Dockerfiles |
55 +| `architecting-service-boundaries` | modular monolith default, data ownership, sagas, strangler | messaging mechanics; endpoint design |
56 +| `scaling-backend-services` | statelessness, pooling, load balancing, autoscaling, replicas | query tuning; caching; rate limiting |
57 +
58 +## Shared conventions
59 +
60 +- Description = WHAT + "Use when …" (literal phrases) + "Do not use for …"
61 +- One default per decision with an escape hatch; concrete values over adjectives
62 +- Every workflow ends with a validation step (replay a job twice, boot with a missing var, kill an instance mid-deploy)
63 +- `SKILL.md` <150 lines; depth in `references/patterns.md` (with TOC)
added backend-skills/architecting-service-boundaries/SKILL.md +46 −0
@@ -0,0 +1,46 @@
1 +---
2 +name: architecting-service-boundaries
3 +description: Decides how to split (or not split) a backend into services — modular monolith by default, boundaries along business capabilities, data ownership per service, sagas over distributed transactions, strangler-pattern extraction. Use when the user asks whether to adopt microservices, how to split a monolith, where service boundaries belong, how services should share data, or how to handle transactions across services. Do not use for message-broker mechanics (handling-async-messaging) or REST endpoint design (designing-rest-apis).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Architecting Service Boundaries
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** monolith-vs-services decisions, drawing/reviewing service boundaries, cross-service data and transaction design, extraction plans.
15 +- **Do NOT use for:** broker/queue mechanics (handling-async-messaging), endpoint design (designing-rest-apis), or infra scaling (scaling-backend-services).
16 +
17 +## Core rules
18 +
19 +1. **Default to a modular monolith.** One deployable with enforced internal module boundaries. Split a module out only on a proven trigger: independent scaling need, independent deploy cadence, or team-ownership friction — never "because microservices."
20 +2. **Boundaries follow business capabilities, not technical layers.**
21 + - ✅ `billing`, `catalog`, `fulfillment`
22 + - ❌ `api-service`, `business-logic-service`, `database-service`
23 +3. **Each service owns its data.** No other service reads its tables; integration happens through its API or published events. A shared database is one service wearing several trench coats.
24 +4. **Keep synchronous call chains shallow — ~2 hops max.** Request → A → B is acceptable; A → B → C → D couples four uptimes and multiplies latency. Deeper flows go asynchronous via events.
25 +5. **Distributed transactions are sagas.** A sequence of local transactions with explicit compensating actions for each step. Never reach for two-phase commit; design the compensation before the happy path.
26 +6. **Contracts are versioned and backward compatible.** Additive changes only within a version; breaking changes get a new version with a deprecation window. A consumer must never be forced to deploy in lockstep with a provider.
27 +7. **Extract with the strangler pattern.** Route a slice of traffic through the new service while the monolith still serves the rest; retire monolith code only after parity is proven. Big-bang rewrites forfeit the rollback path.
28 +8. **Watch for the distributed monolith.** Services that must deploy together, share a schema, or break when one is down have all of the costs of microservices and none of the benefits — re-merge or re-draw the boundary.
29 +
30 +## Workflow
31 +
32 +1. List business capabilities and the teams that own them; sketch candidate boundaries there.
33 +2. For each candidate split, name the concrete trigger from rule 1. No trigger → stays a module.
34 +3. For each boundary: define the owned data, the exposed contract, and the events published.
35 +4. Map every cross-boundary write flow as a saga with compensations (rule 5).
36 +5. Plan extraction via strangler routing with a rollback switch (rule 7).
37 +6. Validate the design: for each service, confirm it can deploy alone with every other service frozen, and survive (degraded, not down) any single dependency being offline. Any "no" is a rule-8 smell — redraw before building.
38 +
39 +## Edge cases & failure modes
40 +- **Two services keep changing in the same PRs** → boundary is wrong; merge them or move the shared concept to one owner.
41 +- **A service needs another's data constantly** → replicate via events into a local read model instead of chatty sync calls.
42 +- **Reporting needs to join across services** → dedicated analytics store fed by events; never grant cross-service table access.
43 +- **Saga compensation impossible** (e.g., email already sent) → make the step last, or design it to be semantically reversible ("correction" message).
44 +
45 +## References
46 +Decision tables, saga and strangler sketches: see [references/patterns.md](references/patterns.md).
added backend-skills/architecting-service-boundaries/references/patterns.md +115 −0
@@ -0,0 +1,115 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Architecting Service Boundaries
7 +
8 +## Contents
9 +- Split/don't-split decision table
10 +- Modular monolith enforcement
11 +- Saga with compensations (order placement)
12 +- Local read model via events
13 +- Strangler extraction plan
14 +- Distributed-monolith smell test
15 +- Gotchas
16 +
17 +## Split/don't-split decision table
18 +
19 +| Signal | Verdict |
20 +|---|---|
21 +| Module needs 10× the replicas of the rest | Split (scaling trigger) |
22 +| Module ships weekly while the rest ships daily and blocks it | Split (cadence trigger) |
23 +| Two teams keep merge-conflicting in one module | Split (ownership trigger) |
24 +| "It would be cleaner as a service" | Don't split |
25 +| Junior team, no platform/on-call maturity | Don't split yet |
26 +| Module is called by everything, in-process, latency-sensitive | Don't split |
27 +
28 +## Modular monolith enforcement
29 +
30 +Enforce boundaries inside one deployable so extraction stays cheap:
31 +
32 +```
33 +app/
34 +├── billing/ # public API: billing/api.py only
35 +├── catalog/
36 +├── fulfillment/
37 +└── shared/ # pure utilities only — no business logic
38 +```
39 +
40 +- Each module exposes one façade (`billing/api.py`); cross-module imports of internals fail CI (import-linter / ArchUnit / eslint-boundaries).
41 +- Each module gets its own schema/namespace in the database from day one — table ownership is then already settled when a split comes.
42 +
43 +## Saga with compensations (order placement)
44 +
45 +```
46 +Step Compensation
47 +1. reserve inventory → release reservation
48 +2. charge payment → refund payment
49 +3. create shipment → cancel shipment
50 +4. send confirmation → (terminal; deliberately last — cannot be unsent)
51 +```
52 +
53 +```python
54 +SAGA = [
55 + (reserve_inventory, release_reservation),
56 + (charge_payment, refund_payment),
57 + (create_shipment, cancel_shipment),
58 + (send_confirmation, None), # irreversible step goes last
59 +]
60 +
61 +def run_saga(order):
62 + done = []
63 + for step, compensate in SAGA:
64 + try:
65 + step(order)
66 + done.append(compensate)
67 + except StepFailed:
68 + for comp in reversed(done): # unwind in reverse order
69 + if comp:
70 + comp(order)
71 + raise
72 +```
73 +
74 +Orchestration (one coordinator, above) is the default — easier to trace and test.
75 +Choreography (each service reacts to events) is the escape hatch when the
76 +coordinator itself becomes a coupling point.
77 +
78 +## Local read model via events
79 +
80 +Instead of `fulfillment` calling `catalog` on every shipment:
81 +
82 +```
83 +catalog --publishes--> product.updated {id, weight, dims}
84 +fulfillment --consumes--> upserts into its own product_dimensions table
85 +```
86 +
87 +Fulfillment reads locally (fast, survives catalog downtime); staleness is
88 +bounded by event lag and must be acceptable for the use case — if it isn't,
89 +the data wasn't yours to cache and the boundary may be wrong.
90 +
91 +## Strangler extraction plan
92 +
93 +1. Freeze feature work in the monolith module being extracted.
94 +2. Stand up the new service; dual-write or replay events to fill its store.
95 +3. Route N% of read traffic via a routing layer flag; compare responses (shadow diff).
96 +4. Ramp N → 100 with the rollback switch live at every step.
97 +5. Cut writes over behind the same flag; verify; delete monolith code path within a sprint (a dormant duplicate path rots into rule-8 territory).
98 +
99 +## Distributed-monolith smell test
100 +
101 +Answer for each service; any "yes" means redraw:
102 +
103 +- Must it deploy in the same release train as another service?
104 +- Does another service read or write its tables?
105 +- Does one service's outage take it fully down (not degraded)?
106 +- Do contract changes require synchronized PRs across repos?
107 +
108 +## Gotchas
109 +
110 +- **Entity-shaped services** (`user-service` that everyone calls for everything) recreate the shared database over HTTP; split by capability (identity vs profile vs preferences).
111 +- **Shared libraries with business logic** are a hidden shared schema — a change forces lockstep upgrades. Keep shared libs to pure utilities.
112 +- **Sagas are not ACID**: intermediate states are visible. Model them explicitly (`PENDING_PAYMENT`), don't pretend isolation.
113 +- **Compensation ≠ undo**: refund is a new transaction, not an erasure — books must show both.
114 +- **Event schemas are contracts too** — version them like APIs (rule 6), with consumers tolerant of unknown fields.
115 +- **The routing flag is load-bearing** during strangler cuts; it needs tests and an owner like any production code.
added backend-skills/caching-strategies/SKILL.md +50 −0
@@ -0,0 +1,50 @@
1 +---
2 +name: caching-strategies
3 +description: Designs application-level caching — cache-aside pattern, key naming, TTL policy, invalidation strategy, and stampede protection with Redis or in-process caches. Use when the user asks to add a cache, cache API responses or query results, speed up repeated reads, fix stale-cache or cache-invalidation bugs, or prevent cache stampedes. Do not use for HTTP/CDN edge caching configuration or for database query and index tuning.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Caching Strategies
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** adding or reviewing application caches (Redis, Memcached, in-process): what to cache, key design, TTLs, invalidation, failure behavior.
15 +- **Do NOT use for:** CDN/`Cache-Control` edge configuration, database tuning (→ db-skills/optimizing-sql-performance), or memoizing pure functions in code.
16 +
17 +## Core rules
18 +
19 +1. **Measure before caching.** Cache only reads that are demonstrably hot AND expensive; record the baseline latency and expected hit rate first. A cache below ~80% hit rate usually adds complexity for nothing.
20 +2. **Cache-aside is the default pattern:** read cache → miss → read origin → write cache with TTL. Write-through/write-behind only when a measured write-path need exists.
21 +3. **Every key has a TTL — no immortal keys.** TTL is the invalidation of last resort; without it, every bug becomes permanent.
22 + - ✅ `SETEX product:v1:42 300 …`
23 + - ❌ `SET product:42 …` (lives until someone remembers it exists)
24 +4. **Choose the invalidation strategy per data class, at design time:**
25 + - tolerates staleness → **TTL-only** (pick the tolerance as the TTL)
26 + - must reflect writes → **purge/update on write** (delete the key in the write path)
27 + - broad derived data → **versioned keys** (bump `v` in the key; old entries age out)
28 + Mixing strategies ad hoc is how stale-forever bugs are born.
29 +5. **Key schema is part of the design:** `entity:version:id[:variant]`, e.g. `product:v2:42:fr`. Every dimension that changes the value appears in the key — locale, currency, role.
30 +6. **Never serve one user's data from a shared key.** Per-user data gets the user ID in the key; better, don't cache authorization decisions at all.
31 + - ❌ `GET profile:current` — whoever primed it wins
32 +7. **Protect against stampedes:** jitter TTLs (±10%) so keys don't expire in sync, and use a single-flight lock so one process rebuilds a hot key while others serve slightly-stale or wait.
33 +8. **Cache down ≠ site down.** Wrap cache calls with a short timeout (~50 ms) and fall through to origin on any cache error; a cache outage becomes a latency event, not an availability event.
34 +
35 +## Workflow
36 +
37 +1. Identify the hot, expensive read; record baseline latency and expected hit rate.
38 +2. Classify its data (staleness tolerance) and pick the invalidation strategy from rule 4.
39 +3. Define the key schema and TTL (+ jitter); implement cache-aside with origin fallback on cache errors.
40 +4. Add hit/miss metrics per key family.
41 +5. **Validate:** write to the origin and confirm the read path reflects it within the chosen tolerance; kill the cache and confirm requests still succeed from origin; check hit rate after a warm-up period against the target.
42 +
43 +## Edge cases & failure modes
44 +- **Caching negative results** (not-found) — allowed with a SHORT TTL (~30 s) to absorb miss storms, but must be purged on create.
45 +- **Large values** (>100 KB in Redis) — compress or split; big values evict everything else.
46 +- **Cold start after deploy/flush** — expect an origin load spike; single-flight (rule 7) is what keeps it survivable.
47 +- **Two caches for one datum** (in-process + Redis) — layered TTLs multiply staleness; keep the in-process layer very short (~1–5 s).
48 +
49 +## References
50 +Snippets for single-flight, jitter, and key-schema helpers: see [references/patterns.md](references/patterns.md).
added backend-skills/caching-strategies/references/patterns.md +124 −0
@@ -0,0 +1,124 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Caching
7 +
8 +## Contents
9 +- Cache-aside with fallback
10 +- TTL jitter
11 +- Single-flight rebuild
12 +- Versioned keys
13 +- Purge-on-write
14 +- Negative caching
15 +- Metrics
16 +- Gotchas
17 +
18 +## Cache-aside with fallback
19 +
20 +```python
21 +import json, random
22 +
23 +CACHE_TIMEOUT_S = 0.05 # cache slower than 50 ms is worse than origin
24 +TTL_S = 300 # staleness tolerance for this data class
25 +
26 +def get_product(pid: int):
27 + key = f"product:v2:{pid}"
28 + try:
29 + raw = redis.get(key) # timeout=CACHE_TIMEOUT_S on the client
30 + if raw is not None:
31 + return json.loads(raw)
32 + except RedisError:
33 + pass # cache down → serve from origin
34 + value = db.fetch_product(pid)
35 + try:
36 + redis.setex(key, jittered(TTL_S), json.dumps(value))
37 + except RedisError:
38 + pass # failing to cache is not an error
39 + return value
40 +```
41 +
42 +## TTL jitter
43 +
44 +```python
45 +def jittered(ttl: int) -> int:
46 + return int(ttl * random.uniform(0.9, 1.1)) # ±10% desynchronizes expiry
47 +```
48 +
49 +## Single-flight rebuild
50 +
51 +```python
52 +LOCK_TTL_S = 10 # > rebuild p99 so a crashed builder's lock self-clears
53 +
54 +def get_report(rid: str):
55 + key = f"report:v1:{rid}"
56 + raw = redis.get(key)
57 + if raw is not None:
58 + return json.loads(raw)
59 + if redis.set(f"lock:{key}", "1", nx=True, ex=LOCK_TTL_S):
60 + value = build_report(rid) # only this process rebuilds
61 + redis.setex(key, jittered(600), json.dumps(value))
62 + redis.delete(f"lock:{key}")
63 + return value
64 + time.sleep(0.1) # others: brief wait then retry once
65 + raw = redis.get(key)
66 + return json.loads(raw) if raw else build_report(rid) # last resort: origin
67 +```
68 +
69 +## Versioned keys
70 +
71 +```python
72 +# Invalidate a whole family by bumping the version constant in code —
73 +# no scan-and-delete, old entries simply age out via TTL.
74 +PRODUCT_CACHE_V = 3
75 +key = f"product:v{PRODUCT_CACHE_V}:{pid}:{locale}"
76 +```
77 +
78 +## Purge-on-write
79 +
80 +```python
81 +def update_product(pid: int, fields: dict):
82 + db.update_product(pid, fields)
83 + try:
84 + redis.delete(f"product:v2:{pid}") # delete, don't rewrite: the next
85 + except RedisError: # read repopulates from fresh origin
86 + log.warning("purge failed for %s — TTL is the backstop", pid)
87 +```
88 +
89 +Delete (not set) after write: writing the new value here races concurrent
90 +readers repopulating from a stale read replica.
91 +
92 +## Negative caching
93 +
94 +```python
95 +NEG_TTL_S = 30 # short: absorbs miss storms without delaying creates for long
96 +
97 +if value is None:
98 + redis.setex(key, NEG_TTL_S, "__miss__")
99 +# On create: redis.delete(key) so the sentinel dies immediately.
100 +```
101 +
102 +## Metrics
103 +
104 +```python
105 +metrics.incr(f"cache.{family}.{'hit' if raw is not None else 'miss'}")
106 +```
107 +
108 +Alert when hit rate for a family drops below its target (~80%) — usually a
109 +key-schema change or an invalidation bug, not traffic.
110 +
111 +## Gotchas
112 +
113 +- **Caching the serialized response of another cache-user** stacks TTLs;
114 + staleness = sum of layers, not max.
115 +- **`KEYS pattern*` for invalidation** blocks Redis; that need is the signal
116 + to switch to versioned keys.
117 +- **Thundering read-repair after purge-on-write**: hot keys need
118 + single-flight even with purge-on-write.
119 +- **Objects that serialize differently across app versions** poison shared
120 + caches during deploys — version the key (rule 5) on format changes.
121 +- **Read replicas + purge-on-write**: purge, then repopulate-on-read may
122 + read a stale replica and resurrect old data; short TTL bounds the damage.
123 +- **In-process caches in autoscaled fleets** are N independent staleness
124 + bubbles; keep them ≤5 s or accept per-instance divergence.
added backend-skills/containerizing-services/SKILL.md +50 −0
@@ -0,0 +1,50 @@
1 +---
2 +name: containerizing-services
3 +description: Containerizes backend services with production-grade Dockerfiles — multi-stage builds, pinned base images, non-root users, healthchecks, and small images. Use when the user asks to write or review a Dockerfile, dockerize or containerize a service or app, shrink a Docker image, or fix container build or security issues. Do not use for Kubernetes manifests or orchestration, docker-compose service topology, or CI pipeline design (shipping-with-ci-cd).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Containerizing Services
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** writing or reviewing Dockerfiles, image size/security hardening, build-cache problems.
15 +- **Do NOT use for:** Kubernetes/orchestration, compose topology, CI pipelines (shipping-with-ci-cd), or app code changes.
16 +
17 +## Core rules
18 +
19 +1. **Multi-stage: build heavy, run slim.** Compile/install in a build stage; copy only artifacts into the runtime stage.
20 + - ✅ `FROM node:22.4-slim AS build` … `FROM node:22.4-slim` + `COPY --from=build /app/dist ./dist`
21 + - ❌ One stage that ships compilers, dev deps, and source history.
22 +2. **Pin base images.** Exact tag minimum; digest for production.
23 + - ✅ `FROM python:3.12.4-slim@sha256:…`
24 + - ❌ `FROM python:latest`
25 +3. **Run as non-root.** Create a user and switch: `USER app`. Root-only containers fail most cluster policies and widen every exploit.
26 +4. **Order layers for cache.** Copy dependency manifests and install BEFORE copying source, so code edits don't bust the dependency layer.
27 + - ✅ `COPY package*.json ./` → `RUN npm ci` → `COPY . .`
28 + - ❌ `COPY . .` first (every commit reinstalls everything).
29 +5. **One process per container.** No supervisord bundles; sidecars belong to the orchestrator. Logs go to stdout/stderr only — never files inside the container.
30 +6. **Ship a `.dockerignore`.** At minimum: `.git`, `node_modules`/venvs, `.env*`, secrets, test fixtures. Never `COPY` a secret; pass at runtime.
31 +7. **Define `HEALTHCHECK`** (or document the orchestrator probe) hitting a real readiness endpoint, not `/`.
32 +8. **Default slim/distroless; alpine only knowingly.** musl breaks some native wheels/binaries — use alpine only after the app is verified on it.
33 +
34 +## Workflow
35 +
36 +1. Pick runtime base (`<lang>:<exact-version>-slim` default; distroless escape hatch for static binaries).
37 +2. Write `.dockerignore` before the Dockerfile.
38 +3. Write multi-stage Dockerfile per rules 1–8.
39 +4. Build: `docker build -t svc:dev .` — rebuild after touching one source file and confirm the dependency layer is cached (`CACHED` in output).
40 +5. Validate: `docker run --rm svc:dev id -u` returns non-zero UID; `docker inspect --format='{{.Config.Healthcheck}}' svc:dev` is set; `docker run --rm --read-only svc:dev` starts (add tmpfs mounts if the app needs scratch dirs).
41 +6. Check size: `docker images svc:dev` — if the runtime image exceeds ~2× the artifact size, find what leaked in (`docker history svc:dev`).
42 +
43 +## Edge cases & failure modes
44 +- **Native deps fail on slim** → install build tools in the build stage only (`apt-get install -y --no-install-recommends build-essential`), never in runtime.
45 +- **Secrets needed at build time** → `RUN --mount=type=secret,id=npm_token …` (BuildKit); never `ARG` a secret (it persists in history).
46 +- **Image works locally, fails in cluster as non-root** → files owned by root; `COPY --chown=app:app`.
47 +- **Timezone/CA errors in distroless** → use the `:debug` or cc variant, or copy `ca-certificates` from the build stage.
48 +
49 +## References
50 +Deeper recipes and gotchas: see [references/patterns.md](references/patterns.md).
added backend-skills/containerizing-services/references/patterns.md +135 −0
@@ -0,0 +1,135 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Containerizing Services
7 +
8 +## Contents
9 +- Node.js multi-stage Dockerfile
10 +- Python multi-stage Dockerfile
11 +- Go static binary + distroless
12 +- .dockerignore baseline
13 +- Build-time secrets (BuildKit)
14 +- Runtime hardening flags
15 +- Gotchas
16 +
17 +## Node.js multi-stage Dockerfile
18 +
19 +```dockerfile
20 +# syntax=docker/dockerfile:1
21 +FROM node:22.4-slim AS build
22 +WORKDIR /app
23 +COPY package*.json ./
24 +RUN npm ci
25 +COPY . .
26 +RUN npm run build && npm prune --omit=dev
27 +
28 +FROM node:22.4-slim
29 +ENV NODE_ENV=production
30 +WORKDIR /app
31 +RUN useradd --uid 10001 --create-home app
32 +COPY --from=build --chown=app:app /app/node_modules ./node_modules
33 +COPY --from=build --chown=app:app /app/dist ./dist
34 +USER app
35 +EXPOSE 3000
36 +HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
37 + CMD node -e "fetch('http://127.0.0.1:3000/healthz').then(r=>{if(!r.ok)process.exit(1)}).catch(()=>process.exit(1))"
38 +CMD ["node", "dist/server.js"]
39 +```
40 +
41 +## Python multi-stage Dockerfile
42 +
43 +```dockerfile
44 +# syntax=docker/dockerfile:1
45 +FROM python:3.12.4-slim AS build
46 +WORKDIR /app
47 +RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
48 + && rm -rf /var/lib/apt/lists/*
49 +COPY requirements.txt .
50 +RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
51 +
52 +FROM python:3.12.4-slim
53 +WORKDIR /app
54 +RUN useradd --uid 10001 --create-home app
55 +COPY --from=build /install /usr/local
56 +COPY --chown=app:app . .
57 +USER app
58 +EXPOSE 8000
59 +HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
60 + CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/healthz').status==200 else 1)"
61 +CMD ["gunicorn", "-b", "0.0.0.0:8000", "app.wsgi:application"]
62 +```
63 +
64 +## Go static binary + distroless
65 +
66 +```dockerfile
67 +# syntax=docker/dockerfile:1
68 +FROM golang:1.23.1 AS build
69 +WORKDIR /src
70 +COPY go.mod go.sum ./
71 +RUN go mod download
72 +COPY . .
73 +RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /out/svc ./cmd/svc
74 +
75 +FROM gcr.io/distroless/static-debian12:nonroot
76 +COPY --from=build /out/svc /svc
77 +EXPOSE 8080
78 +ENTRYPOINT ["/svc"]
79 +```
80 +
81 +Distroless has no shell — the orchestrator's HTTP probe replaces `HEALTHCHECK`.
82 +
83 +## .dockerignore baseline
84 +
85 +```
86 +.git
87 +.gitignore
88 +.env*
89 +*.md
90 +node_modules
91 +__pycache__
92 +*.pyc
93 +.venv
94 +dist
95 +coverage
96 +tests/fixtures
97 +Dockerfile
98 +```
99 +
100 +Remove `dist` from the ignore list if you COPY prebuilt artifacts instead of building in-image.
101 +
102 +## Build-time secrets (BuildKit)
103 +
104 +```dockerfile
105 +RUN --mount=type=secret,id=pip_index \
106 + PIP_INDEX_URL=$(cat /run/secrets/pip_index) pip install -r requirements.txt
107 +```
108 +
109 +```bash
110 +docker build --secret id=pip_index,src=.pip_index_url .
111 +```
112 +
113 +Never `ARG TOKEN` — args are recoverable via `docker history`.
114 +
115 +## Runtime hardening flags
116 +
117 +```bash
118 +docker run --rm \
119 + --read-only --tmpfs /tmp \
120 + --cap-drop ALL \
121 + --security-opt no-new-privileges \
122 + -p 8000:8000 svc:prod
123 +```
124 +
125 +Start from all-dropped and add back only what breaks.
126 +
127 +## Gotchas
128 +
129 +- **`COPY --from` keeps root ownership** unless `--chown` is given — the classic "works as root, crashes as USER app" cause.
130 +- **alpine + Python wheels**: musl forces source builds of numpy/psycopg2 etc.; use `-slim` (glibc) unless you've verified alpine.
131 +- **`EXPOSE` documents, it does not publish** — publishing is `-p`/orchestrator config.
132 +- **apt cache bloat**: always `rm -rf /var/lib/apt/lists/*` in the same `RUN` as the install, or the cache lands in the layer anyway.
133 +- **CMD shell form (`CMD node server.js`) wraps in `/bin/sh`** → PID 1 is sh, signals (SIGTERM) never reach the app → 10s kill delay on every deploy. Use exec form (JSON array).
134 +- **`HEALTHCHECK` in Dockerfile is ignored by Kubernetes** — it uses its own probes; keep both consistent.
135 +- **Bind-mounting over image content in dev** hides image bugs; test the real image before shipping.
added backend-skills/designing-graphql-apis/SKILL.md +48 −0
@@ -0,0 +1,48 @@
1 +---
2 +name: designing-graphql-apis
3 +description: Designs GraphQL APIs with schema-first typing, Relay-style cursor connections, DataLoader batching against N+1 queries, union result types for expected errors, and depth/complexity limits. Use when the user asks to design, review, or refactor a GraphQL schema or API, write type definitions or resolvers, add pagination to a GraphQL query, fix N+1 resolver performance, or structure mutations. Do not use for REST APIs (designing-rest-apis) or for real-time subscription transport infrastructure.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Designing GraphQL APIs
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** GraphQL schema design, type/resolver structure, connections, mutations, error modeling, query cost control.
15 +- **Do NOT use for:** REST endpoint design → `designing-rest-apis`; WebSocket/subscription transport setup; client-side query writing.
16 +
17 +## Core rules
18 +
19 +1. **Schema first, domain-shaped.** Design the SDL from the domain and client needs, then implement resolvers. Never mirror database tables 1:1 — expose what clients consume.
20 + - ✅ `type Order { total: Money!, placedAt: DateTime! }`
21 + - ❌ `type OrdersTbl { amt_cents: Int, created_ts: Int }`
22 +2. **Nullable by default; non-null (`!`) only when guaranteed.** A non-null field that fails nulls out its whole parent chain. Reserve `!` for IDs and fields the resolver can never fail to produce.
23 +3. **Paginate lists with Relay connections.** Any list that can grow gets `Connection`/`Edge`/`PageInfo` with `first`/`after` cursor args — never a bare unbounded `[Order!]!`.
24 +4. **DataLoader is the default N+1 fix.** Every resolver that fetches by ID goes through a per-request DataLoader that batches and caches. One query per collection of parents, not one per parent.
25 +5. **Mutations: verb-object names, one input, one payload.** `createOrder(input: CreateOrderInput!): CreateOrderPayload!`. Payload contains the changed object plus expected-error fields — never return the bare object.
26 +6. **Expected errors are union result types; the `errors` array is for exceptions only.**
27 + - ✅ `union CreateOrderResult = Order | ValidationError | OutOfStockError`
28 + - ❌ throwing for "email already taken" so clients parse `errors[0].message`
29 +7. **Cap query cost.** Enforce a depth limit (default 10) and a complexity budget (points per field × list multipliers); reject over-budget queries before execution.
30 +8. **Additive evolution only.** Add fields freely; never remove or change a type in place — `@deprecated(reason: "...")` first, remove in a coordinated major cycle.
31 +
32 +## Workflow
33 +
34 +1. List client use cases; sketch queries clients should be able to write.
35 +2. Write the SDL: types, connections (rule 3), mutations (rule 5), error unions (rule 6).
36 +3. Apply nullability discipline (rule 2) type by type.
37 +4. Plan resolvers: mark every by-ID fetch as a DataLoader (rule 4).
38 +5. Set depth/complexity limits (rule 7).
39 +6. **Validate:** run the schema through a linter (`npx graphql-schema-linter schema.graphql` — install with `npm i -g graphql-schema-linter` if missing), execute the sketched queries from step 1 against a stub server, and log SQL for one nested query to confirm no N+1 (query count must be O(depth), not O(rows)). Fix and repeat.
40 +
41 +## Edge cases & failure modes
42 +- **A field that is expensive for some parents** → split it (`Order.summary` cheap, `Order.analytics` separate type) so cheap queries don't pay.
43 +- **Polymorphic lists** → interfaces when types share fields, unions when they don't; always include `__typename` handling in examples.
44 +- **File uploads** → do not tunnel through GraphQL; issue a presigned URL via mutation and upload out-of-band.
45 +- **Global object identity** → give every fetchable type a globally unique `id: ID!` (base64 `Type:dbId`) so caches and `node(id:)` refetching work.
46 +
47 +## References
48 +Deeper patterns (connection SDL, DataLoader implementation, complexity scoring): see [references/patterns.md](references/patterns.md).
added backend-skills/designing-graphql-apis/references/patterns.md +127 −0
@@ -0,0 +1,127 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Designing GraphQL APIs
7 +
8 +## Contents
9 +- Connection SDL (Relay style)
10 +- Mutation with union result
11 +- DataLoader implementation
12 +- Depth and complexity limits
13 +- Deprecation flow
14 +- Gotchas
15 +
16 +## Connection SDL (Relay style)
17 +
18 +```graphql
19 +type Query {
20 + orders(first: Int = 20, after: String, status: OrderStatus): OrderConnection!
21 +}
22 +
23 +type OrderConnection {
24 + edges: [OrderEdge!]!
25 + pageInfo: PageInfo!
26 + totalCount: Int # nullable: expensive, resolve only when asked
27 +}
28 +
29 +type OrderEdge {
30 + node: Order!
31 + cursor: String!
32 +}
33 +
34 +type PageInfo {
35 + hasNextPage: Boolean!
36 + endCursor: String
37 +}
38 +```
39 +Clamp `first` to a max of 100 in the resolver (same reasoning as REST page
40 +limits); reject negative values with a validation error.
41 +
42 +## Mutation with union result
43 +
44 +```graphql
45 +input CreateOrderInput {
46 + customerId: ID!
47 + items: [OrderItemInput!]!
48 +}
49 +
50 +union CreateOrderResult = CreateOrderSuccess | ValidationError | OutOfStockError
51 +
52 +type CreateOrderSuccess { order: Order! }
53 +type ValidationError { fields: [FieldError!]! }
54 +type OutOfStockError { itemIds: [ID!]!, message: String! }
55 +
56 +type Mutation {
57 + createOrder(input: CreateOrderInput!): CreateOrderResult!
58 +}
59 +```
60 +
61 +Client handles each case via `... on` fragments; the top-level `errors` array
62 +stays reserved for genuine faults (auth failure, internal error).
63 +
64 +## DataLoader implementation
65 +
66 +```python
67 +# Python (aiodataloader); same shape in JS's dataloader package.
68 +from aiodataloader import DataLoader
69 +
70 +class CustomerLoader(DataLoader):
71 + async def batch_load_fn(self, ids):
72 + rows = await db.fetch(
73 + "SELECT * FROM customers WHERE id = ANY($1)", ids)
74 + by_id = {r["id"]: r for r in rows}
75 + # Must return results in the SAME ORDER as ids, None for misses.
76 + return [by_id.get(i) for i in ids]
77 +
78 +# Create ONE loader per request (in context), never a module-level singleton —
79 +# its cache would leak data across users.
80 +async def resolve_customer(order, info):
81 + return await info.context["customer_loader"].load(order["customer_id"])
82 +```
83 +
84 +## Depth and complexity limits
85 +
86 +```javascript
87 +// graphql-depth-limit + graphql-query-complexity (npm)
88 +validationRules: [
89 + depthLimit(10), // pathological nesting stops here
90 + createComplexityRule({
91 + maximumComplexity: 1000, // ~1 point per field
92 + listFactor: 10, // lists multiply child cost
93 + onComplete: (c) => log.info({ complexity: c }),
94 + }),
95 +]
96 +```
97 +Return the budget in the rejection message so clients can adapt:
98 +`"Query complexity 2140 exceeds maximum 1000"`.
99 +
100 +## Deprecation flow
101 +
102 +```graphql
103 +type Order {
104 + total: Money!
105 + amount: Int @deprecated(reason: "Use total; amount is cents-only and removed after 2027-01-01.")
106 +}
107 +```
108 +1. Add the replacement field. 2. Deprecate with a reason that names the
109 +replacement and a date. 3. Monitor field usage (most gateways report it).
110 +4. Remove only when usage is zero or the date passes.
111 +
112 +## Gotchas
113 +
114 +- **Non-null cascade:** an error in `Order.customer: Customer!` nulls the
115 + entire `order` — with `Customer` (nullable) only the field nulls. This is
116 + why rule 2 defaults to nullable.
117 +- **DataLoader order contract:** `batch_load_fn` must return exactly
118 + len(ids) results in input order; returning a dict or short list corrupts
119 + unrelated resolvers silently.
120 +- **Enums over booleans:** `status: OrderStatus` beats `isActive/isArchived`
121 + boolean pairs that can contradict each other.
122 +- **Introspection in production:** disable for public APIs unless the API is
123 + deliberately open; it enumerates your entire attack surface.
124 +- **`totalCount` on large tables** is a full COUNT(*) per query — make it
125 + nullable and resolve lazily, or return an estimate and say so.
126 +- **Cursor stability:** cursors must encode the ORDER BY key, not row
127 + position, or pagination skips/duplicates under concurrent writes.
added backend-skills/designing-rest-apis/SKILL.md +48 −0
@@ -0,0 +1,48 @@
1 +---
2 +name: designing-rest-apis
3 +description: Designs REST APIs with correct resource naming, HTTP method and status-code semantics, mandatory pagination, day-one versioning, and RFC 9457 problem+json errors, with OpenAPI as the source of truth. Use when the user asks to design, review, or refactor a REST API, define endpoints or routes, choose status codes, add pagination or versioning, or write an OpenAPI/Swagger spec. Do not use for GraphQL APIs (designing-graphql-apis), webhook delivery (designing-webhooks), or authentication mechanics (implementing-authentication).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Designing REST APIs
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** designing or reviewing REST endpoints, URL structure, status codes, pagination, versioning, error bodies, OpenAPI specs.
15 +- **Do NOT use for:** GraphQL schemas → `designing-graphql-apis`; webhook delivery → `designing-webhooks`; login/tokens → `implementing-authentication`; rate-limit policy → `limiting-request-rates`.
16 +
17 +## Core rules
18 +
19 +1. **Plural nouns, never verbs, in paths.** Actions come from HTTP methods; non-CRUD actions become sub-resources.
20 + - ✅ `POST /orders`, `POST /orders/42/cancellation`
21 + - ❌ `POST /createOrder`, `GET /getOrders`
22 +2. **Method semantics are non-negotiable.** GET is safe and cacheable; PUT replaces and is idempotent; PATCH partially updates; DELETE is idempotent (second call → 404 or 204, never 500). Never mutate on GET.
23 +3. **Status codes carry meaning — use the right one.** 200 read/update, 201 + `Location` header on create, 204 no body, 400 malformed syntax, 401 unauthenticated, 403 unauthorized, 404 absent (also for hiding resources), 409 state conflict, 422 valid syntax but failed business validation, 429 rate limited, 500 only for genuine server faults.
24 + - ✅ `422` for "email already registered"
25 + - ❌ `200 {"success": false}`
26 +4. **Every collection paginates from day one.** Default `limit=20`, hard max `limit=100`; return cursor pagination (`next_cursor`) by default, offset only for small, static datasets.
27 +5. **Version from the first commit.** Path prefix `/v1/` is the default. A breaking change (removed/renamed field, changed type or semantics) requires `/v2/`; additive changes do not.
28 +6. **Errors are structured, uniform, and safe.** Use RFC 9457 problem+json: `type`, `title`, `status`, `detail`, `instance` (+ per-field `errors` array for 422). Never leak stack traces, SQL, or internal class names.
29 +7. **The OpenAPI spec is the source of truth.** Write or update the spec with every endpoint change; generated docs and clients follow the spec, not the code.
30 +8. **Filtering and sorting are query parameters with one convention.** `?status=active&sort=-created_at` (leading `-` = descending). Unknown parameters → 400, don't ignore silently.
31 +
32 +## Workflow
33 +
34 +1. List the resources (nouns) and their relationships before any URL exists.
35 +2. Map each operation to method + path per rules 1–2; define request/response bodies.
36 +3. Assign status codes per rule 3, including every failure path.
37 +4. Add pagination, filtering, versioning per rules 4–5, 8.
38 +5. Write the OpenAPI spec (or update it) and define the problem+json error schema once, referenced everywhere.
39 +6. **Validate:** lint the spec (`npx @redocly/cli lint openapi.yaml` — install with `npm i -g @redocly/cli` if missing) and walk one full CRUD cycle checking each response code against rule 3. Fix and re-lint until clean.
40 +
41 +## Edge cases & failure modes
42 +- **Long-running operations** → `202 Accepted` + status resource (`GET /operations/{id}`), never a 30 s blocking request.
43 +- **Bulk operations** → dedicated resource (`POST /orders/batch`) returning per-item results with individual statuses (207-style body), not first-error-aborts.
44 +- **Retries on create** → accept an `Idempotency-Key` header on POST; same key + same body → same response, no duplicate.
45 +- **Deprecating a field** → mark `deprecated: true` in the spec and announce a sunset date; removal only in the next major version.
46 +
47 +## References
48 +Deeper patterns (pagination envelopes, problem+json schemas, OpenAPI skeleton): see [references/patterns.md](references/patterns.md).
added backend-skills/designing-rest-apis/references/patterns.md +147 −0
@@ -0,0 +1,147 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Designing REST APIs
7 +
8 +## Contents
9 +- URL and method matrix
10 +- Pagination envelope (cursor)
11 +- problem+json error schemas
12 +- Idempotency-Key handling
13 +- OpenAPI 3.1 skeleton
14 +- Gotchas
15 +
16 +## URL and method matrix
17 +
18 +```
19 +GET /v1/orders list (paginated) 200
20 +POST /v1/orders create 201 + Location
21 +GET /v1/orders/{id} read 200 / 404
22 +PUT /v1/orders/{id} full replace (idempotent) 200 / 404
23 +PATCH /v1/orders/{id} partial update 200 / 404 / 409
24 +DELETE /v1/orders/{id} delete (idempotent) 204 / 404
25 +POST /v1/orders/{id}/refund non-CRUD action 201 / 409
26 +GET /v1/customers/{id}/orders nested read-only view 200
27 +```
28 +Nest at most one level; deeper relations get top-level resources with filters
29 +(`/v1/orders?customer_id=7`), not `/customers/7/orders/3/items/9`.
30 +
31 +## Pagination envelope (cursor)
32 +
33 +```json
34 +{
35 + "data": [{ "id": "ord_01J8", "status": "active" }],
36 + "pagination": {
37 + "next_cursor": "eyJpZCI6Im9yZF8wMUo4In0",
38 + "has_more": true,
39 + "limit": 20
40 + }
41 +}
42 +```
43 +Request: `GET /v1/orders?limit=20&cursor=eyJpZCI6...`. The cursor is opaque
44 +(base64 of the last sort key) — clients must not parse it. `limit` above the
45 +max of 100 → clamp to 100, do not error.
46 +
47 +## problem+json error schemas
48 +
49 +Header: `Content-Type: application/problem+json`
50 +
51 +```json
52 +{
53 + "type": "https://api.example.com/errors/validation",
54 + "title": "Validation failed",
55 + "status": 422,
56 + "detail": "2 fields are invalid.",
57 + "instance": "/v1/orders",
58 + "errors": [
59 + { "field": "email", "message": "must be a valid email address" },
60 + { "field": "quantity", "message": "must be between 1 and 99" }
61 + ]
62 +}
63 +```
64 +
65 +Conflict example:
66 +
67 +```json
68 +{
69 + "type": "https://api.example.com/errors/state-conflict",
70 + "title": "Order already shipped",
71 + "status": 409,
72 + "detail": "Order ord_01J8 cannot be cancelled after shipment.",
73 + "instance": "/v1/orders/ord_01J8/cancellation"
74 +}
75 +```
76 +
77 +Define the schema once under `components/schemas/Problem` and `$ref` it from
78 +every 4xx/5xx response.
79 +
80 +## Idempotency-Key handling
81 +
82 +```python
83 +# Pseudocode for POST endpoints that create resources.
84 +# Keys expire after 24h — long enough for client retry storms, short enough
85 +# to bound storage.
86 +def create_order(request):
87 + key = request.headers.get("Idempotency-Key")
88 + if key:
89 + cached = idempotency_store.get(key)
90 + if cached:
91 + if cached.request_hash != hash(request.body):
92 + return problem(422, "Idempotency-Key reused with different body")
93 + return cached.response # replay, no side effect
94 + response = do_create(request.body) # single side effect
95 + if key:
96 + idempotency_store.put(key, hash(request.body), response, ttl_hours=24)
97 + return response
98 +```
99 +
100 +## OpenAPI 3.1 skeleton
101 +
102 +```yaml
103 +openapi: 3.1.0
104 +info: { title: Orders API, version: 1.0.0 }
105 +servers: [{ url: https://api.example.com/v1 }]
106 +paths:
107 + /orders:
108 + get:
109 + parameters:
110 + - { name: limit, in: query, schema: { type: integer, maximum: 100, default: 20 } }
111 + - { name: cursor, in: query, schema: { type: string } }
112 + responses:
113 + "200": { $ref: "#/components/responses/OrderList" }
114 + "429": { $ref: "#/components/responses/Problem" }
115 + post:
116 + parameters:
117 + - { name: Idempotency-Key, in: header, schema: { type: string } }
118 + responses:
119 + "201": { $ref: "#/components/responses/Order" }
120 + "422": { $ref: "#/components/responses/Problem" }
121 +components:
122 + schemas:
123 + Problem:
124 + type: object
125 + properties:
126 + type: { type: string }
127 + title: { type: string }
128 + status: { type: integer }
129 + detail: { type: string }
130 + instance: { type: string }
131 +```
132 +
133 +## Gotchas
134 +
135 +- **404 vs 403 leaks existence.** If callers must not learn a resource exists,
136 + return 404 for both missing and forbidden.
137 +- **PUT with partial body silently nulls fields** — that is correct replace
138 + semantics; if clients expect merge, they need PATCH. Document which you offer.
139 +- **Offset pagination drifts** when rows are inserted mid-scan; deep offsets
140 + also get slow. Cursors fix both — default to them.
141 +- **Trailing slashes**: `/orders` and `/orders/` must not be two resources.
142 + Redirect or normalize one to the other.
143 +- **Enum widening is breaking for clients that switch exhaustively.** Adding
144 + an enum value is only additive if the spec documents "unknown values may
145 + appear" from v1.
146 +- **Date-times**: always RFC 3339 UTC (`2026-08-05T14:30:00Z`); epoch
147 + integers and local times cause silent client bugs.
added backend-skills/designing-webhooks/SKILL.md +47 −0
@@ -0,0 +1,47 @@
1 +---
2 +name: designing-webhooks
3 +description: Designs outbound webhook systems with signed deliveries, exponential-backoff retries, delivery IDs for consumer idempotency, endpoint verification, and dead-letter handling under explicit at-least-once semantics. Use when the user asks to design, build, or review webhooks or event notifications to external consumers, sign or verify webhook payloads, add webhook retries, or document webhook delivery guarantees. Do not use for internal service-to-service queues or event buses (handling-async-messaging) or for the business logic of consuming a third party's webhooks.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Designing Webhooks
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** designing the producer side of webhooks (payloads, signing, retries, delivery semantics, subscriber management) and the receiver checklist you publish to consumers.
15 +- **Do NOT use for:** internal async messaging between your own services → `handling-async-messaging`; what a consumer's handler should do with a specific third-party event.
16 +
17 +## Core rules
18 +
19 +1. **Events are facts, named `resource.action` in past tense.** `invoice.paid`, `user.deleted` — not commands (`send_email`) and not ambiguous (`update`).
20 +2. **Default to thin payloads.** Send `id`, `type`, `created_at`, and the resource ID; the consumer fetches the current state via API. Fat payloads only for non-sensitive, low-churn data — a fat payload of stale or sensitive data is a liability.
21 + - ✅ `{"type": "invoice.paid", "data": {"invoice_id": "inv_42"}}`
22 + - ❌ full invoice with customer PII in every delivery
23 +3. **Sign every delivery: HMAC-SHA256 over `timestamp + "." + body`,** sent as a header (`Webhook-Signature: t=...,v1=...`). Receivers must reject signatures older than 5 minutes (replay window) and compare in constant time.
24 +4. **State the guarantee plainly: at-least-once, unordered.** Consumers WILL receive duplicates and out-of-order events. Publish this in the docs; never promise exactly-once or ordering you can't enforce.
25 +5. **Every delivery carries a unique `delivery_id`** (stable across retries of the same event). Consumers deduplicate on it; your docs must show the dedup pattern.
26 +6. **Retry on failure with exponential backoff + jitter, bounded.** Non-2xx or >10 s timeout → retry at ~1 min, 5 min, 30 min, 2 h, 12 h (5 attempts). After the last failure, park the delivery in a dead-letter store and surface it in the dashboard/API — never drop silently.
27 +7. **Verify endpoints before sending real events.** On subscription, send a challenge the consumer must echo (or a signed ping they must 2xx). Auto-disable endpoints failing >7 days and notify the owner.
28 +8. **2xx means accepted, nothing else.** Consumers should enqueue and return 200 immediately; producers treat 3xx/4xx/5xx and slow responses identically — as failures to retry.
29 +
30 +## Workflow
31 +
32 +1. Enumerate events (rule 1) and choose thin/fat per event (rule 2).
33 +2. Define the envelope: `delivery_id`, `type`, `created_at`, `data`.
34 +3. Implement signing (rule 3) and the verification handshake (rule 7).
35 +4. Implement the retry schedule and dead-letter store (rule 6).
36 +5. Write the consumer documentation: signature check, 5-minute skew rejection, dedup on `delivery_id`, at-least-once warning (rules 3–5).
37 +6. **Validate:** deliver a test event to a sample receiver, then force-redeliver the same event — the receiver's side effect must occur exactly once (dedup works) and both deliveries must verify the signature. Break the secret and confirm rejection.
38 +
39 +## Edge cases & failure modes
40 +- **Consumer endpoint is down for a day** → retries cover
41 +~14.5 h; dead-letter after that with manual/API redelivery available.
42 +- **Secret rotation** → support two active secrets per endpoint (`v1` old + `v1` new signatures during overlap) so consumers rotate without dropped events.
43 +- **Event storm (bulk import)** → per-endpoint delivery rate cap and warn subscribers; do not interleave retries ahead of fresh events indefinitely — cap queue age.
44 +- **Consumer needs ordering** → they must reorder on `created_at`/sequence in their own store; offer a `GET /events?after=` reconciliation API as the source of truth.
45 +
46 +## References
47 +Deeper patterns (envelope JSON, signing/verification code, retry table, receiver checklist): see [references/patterns.md](references/patterns.md).
added backend-skills/designing-webhooks/references/patterns.md +114 −0
@@ -0,0 +1,114 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Designing Webhooks
7 +
8 +## Contents
9 +- Delivery envelope
10 +- Signing (producer) and verification (consumer)
11 +- Retry schedule
12 +- Consumer receiver checklist
13 +- Dead-letter and redelivery API
14 +- Gotchas
15 +
16 +## Delivery envelope
17 +
18 +```json
19 +{
20 + "delivery_id": "whd_01J8ZC4T9",
21 + "type": "invoice.paid",
22 + "created_at": "2026-08-05T14:30:00Z",
23 + "api_version": "2026-06-01",
24 + "data": { "invoice_id": "inv_42" }
25 +}
26 +```
27 +`delivery_id` is unique per event and STABLE across retries — it is the
28 +consumer's dedup key. `api_version` pins the payload shape per subscriber.
29 +
30 +## Signing (producer) and verification (consumer)
31 +
32 +Producer:
33 +
34 +```python
35 +import hmac, hashlib, time
36 +
37 +def sign(secret: str, body: bytes) -> str:
38 + ts = str(int(time.time()))
39 + msg = ts.encode() + b"." + body
40 + sig = hmac.new(secret.encode(), msg, hashlib.sha256).hexdigest()
41 + return f"t={ts},v1={sig}"
42 +# Header: Webhook-Signature: t=1754404200,v1=5257a86...
43 +```
44 +
45 +Consumer:
46 +
47 +```python
48 +# 300 s = 5 min replay window: generous for clock skew, tight for replays.
49 +MAX_SKEW = 300
50 +
51 +def verify(secret, header, body) -> bool:
52 + parts = dict(p.split("=", 1) for p in header.split(","))
53 + ts, their_sig = parts["t"], parts["v1"]
54 + if abs(time.time() - int(ts)) > MAX_SKEW:
55 + return False
56 + msg = ts.encode() + b"." + body
57 + ours = hmac.new(secret.encode(), msg, hashlib.sha256).hexdigest()
58 + return hmac.compare_digest(ours, their_sig) # constant time
59 +```
60 +
61 +Verify against the RAW request body bytes — any re-serialization (JSON parse
62 +then dump) changes the bytes and fails the check.
63 +
64 +## Retry schedule
65 +
66 +```
67 +attempt delay cumulative
68 +1 immediate 0
69 +2 1 min 1 min
70 +3 5 min 6 min
71 +4 30 min 36 min
72 +5 2 h ~2.6 h
73 +6 12 h ~14.6 h → dead-letter after this
74 +```
75 +Add ±20% jitter to every delay so a consumer recovering from an outage is not
76 +hit by a synchronized thundering herd. Timeout per attempt: 10 s.
77 +
78 +## Consumer receiver checklist (publish this in your docs)
79 +
80 +```
81 +1. Verify Webhook-Signature before parsing (raw body, constant-time compare).
82 +2. Reject t older than 5 minutes.
83 +3. Return 200 immediately; process async (queue). Slow handlers get retried
84 + and you will process duplicates.
85 +4. Deduplicate on delivery_id (store processed IDs ≥ 24 h).
86 +5. Treat events as unordered; fetch current state from the API when it matters.
87 +6. Do not whitelist our IPs as your only security — verify signatures.
88 +```
89 +
90 +## Dead-letter and redelivery API
91 +
92 +```
93 +GET /v1/webhook-deliveries?status=failed&endpoint_id=we_7
94 +POST /v1/webhook-deliveries/{delivery_id}/redeliver
95 +```
96 +Keep failed deliveries ≥ 30 days. Auto-disable endpoints failing every
97 +delivery for 7 consecutive days; email the owner at 24 h, 72 h, and on
98 +disable.
99 +
100 +## Gotchas
101 +
102 +- **Signing the parsed-then-reserialized body** — key ordering and whitespace
103 + differ; always HMAC the raw bytes you send/receive.
104 +- **Redirects:** don't follow 3xx on delivery (SSRF vector + signature is now
105 + going somewhere unverified). Treat as failure.
106 +- **SSRF on subscription:** validate subscriber URLs (https only, no private
107 + IP ranges, resolve-and-check at send time too — DNS can change).
108 +- **Retrying 4xx forever:** a 410 Gone should disable the endpoint
109 + immediately; a 401/403 after working deliveries usually means the consumer
110 + rotated secrets — alert, don't hammer.
111 +- **Ordering promises creep into docs via examples** — audit docs so every
112 + example shows dedup + unordered handling.
113 +- **One shared secret for all endpoints of a customer** — breach of one
114 + endpoint burns all; scope secrets per endpoint.
added backend-skills/handling-async-messaging/SKILL.md +45 −0
@@ -0,0 +1,45 @@
1 +---
2 +name: handling-async-messaging
3 +description: Designs event-driven messaging between services — topics/queues, transactional outbox, idempotent consumers, event schema versioning, ordering, and poison-message handling (Kafka, RabbitMQ, SNS/SQS, NATS). Use when the user asks to publish or consume events between services, decouple services with a message broker, design event schemas or topics, fix lost/duplicated/out-of-order events, or implement the outbox pattern. Do not use for in-process job queues inside one application (writing-background-jobs) or webhook delivery to external customers (designing-webhooks).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Handling Async Messaging
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** events and messages BETWEEN services: broker/topic design, producer and consumer contracts, delivery semantics, schema evolution.
15 +- **Do NOT use for:** background jobs within one app (→ writing-background-jobs), webhooks to external customers (→ designing-webhooks), or choosing broker infrastructure sizing.
16 +
17 +## Core rules
18 +
19 +1. **The contract is at-least-once + idempotent consumers.** End-to-end exactly-once is a myth once side effects leave the broker; every consumer deduplicates on the event ID or a natural key.
20 + - ✅ `if seen(event.id): ack(); return`
21 + - ❌ assuming the broker's "exactly-once" flag makes handlers safe
22 +2. **Publish through a transactional outbox.** Writing the DB and publishing to the broker are two systems — without an outbox one of them will lie after a crash. Same transaction: business write + outbox row; a relay publishes and marks it sent.
23 +3. **Events are past-tense facts, named for what happened:** `order_placed`, `payment_failed`. An event commanding another service (`create_shipment`) is an RPC in disguise — if the producer needs a response or cares who handles it, make a direct call instead.
24 +4. **Schema changes are additive only.** Add optional fields freely; renaming, retyping, or removing fields means a NEW topic/version (`order_placed.v2`) with both published during migration. Include `event_id`, `occurred_at`, and `schema_version` in every envelope.
25 +5. **Ordering exists only per key.** Brokers guarantee order per partition/key at best; key by the entity (`order_id`) when sequence matters, and make consumers tolerate reordering across keys.
26 +6. **Poison messages go to a DLQ after N attempts** (default 5) — never block the partition retrying forever, and alert on DLQ arrivals.
27 +7. **Consumers own their offset/ack discipline:** ack only after side effects are durable. Ack-then-process converts every crash into silent data loss.
28 +8. **Design for replay.** New consumers or bug fixes will re-read history; rule 1 makes replays safe, and time-sensitive handlers must check `occurred_at` before acting (don't send a "your order shipped" email from a 2-year-old event).
29 +
30 +## Workflow
31 +
32 +1. List the facts to publish (past-tense names), their producers, consumers, and the ordering key per topic.
33 +2. Define envelopes: `event_id`, `occurred_at`, `schema_version`, payload of IDs + stable facts.
34 +3. Implement the producer with a transactional outbox; implement consumers with dedup + ack-after-durable-side-effect.
35 +4. Configure retries → DLQ (5 attempts) with alerting.
36 +5. **Validate:** publish the same event twice and confirm one side effect; kill the consumer between side effect and ack, restart, and confirm no loss and no duplicate effect.
37 +
38 +## Edge cases & failure modes
39 +- **Consumer needs data the event lacks** → refetch from the producer's API by ID; do not fatten events into full snapshots reflexively.
40 +- **Broker down at publish time** → the outbox absorbs it; the relay catches up. This is the pattern's main payoff.
41 +- **Two consumers in one service want the same topic** → separate consumer groups; sharing a group splits the stream between them.
42 +- **Burst of replayed events floods a downstream dependency** → consumers apply their own concurrency/rate limits.
43 +
44 +## References
45 +Outbox schema, envelope template, and dedup snippets: see [references/patterns.md](references/patterns.md).
added backend-skills/handling-async-messaging/references/patterns.md +140 −0
@@ -0,0 +1,140 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Async Messaging
7 +
8 +## Contents
9 +- Event envelope
10 +- Transactional outbox
11 +- Idempotent consumer with dedup table
12 +- Ack discipline
13 +- Schema evolution
14 +- DLQ routing
15 +- Gotchas
16 +
17 +## Event envelope
18 +
19 +```json
20 +{
21 + "event_id": "01J4QG8Z3V9K6W2N8P5R7T1X4C",
22 + "event_type": "order_placed",
23 + "schema_version": 1,
24 + "occurred_at": "2026-08-05T14:03:22Z",
25 + "producer": "orders-service",
26 + "payload": {
27 + "order_id": "ord_8842",
28 + "customer_id": "cus_311",
29 + "total_cents": 12900,
30 + "currency": "USD"
31 + }
32 +}
33 +```
34 +
35 +ULIDs for `event_id`: sortable by creation time, globally unique.
36 +
37 +## Transactional outbox
38 +
39 +```sql
40 +CREATE TABLE outbox (
41 + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
42 + event_id text NOT NULL UNIQUE,
43 + topic text NOT NULL,
44 + envelope jsonb NOT NULL,
45 + published_at timestamptz -- NULL = pending
46 +);
47 +```
48 +
49 +```python
50 +# Producer: business write + outbox row, one transaction
51 +with db.transaction():
52 + create_order(order)
53 + db.execute("INSERT INTO outbox (event_id, topic, envelope) VALUES (%s,%s,%s)",
54 + (evt.event_id, "orders", evt.json()))
55 +
56 +# Relay: poll-and-publish (or CDC/Debezium as the escape hatch at scale)
57 +BATCH = 100 # small batches keep publish latency and crash re-sends bounded
58 +rows = db.fetch("""SELECT * FROM outbox WHERE published_at IS NULL
59 + ORDER BY id LIMIT %s FOR UPDATE SKIP LOCKED""", (BATCH,))
60 +for r in rows:
61 + broker.publish(r.topic, r.envelope, key=r.envelope["payload"]["order_id"])
62 + db.execute("UPDATE outbox SET published_at = now() WHERE id = %s", (r.id,))
63 +```
64 +
65 +The relay is at-least-once (crash between publish and UPDATE → re-publish);
66 +consumer dedup absorbs it.
67 +
68 +## Idempotent consumer with dedup table
69 +
70 +```python
71 +def handle(envelope):
72 + with db.transaction():
73 + inserted = db.execute(
74 + """INSERT INTO consumed_events (consumer, event_id)
75 + VALUES (%s, %s) ON CONFLICT DO NOTHING""",
76 + ("shipping-service", envelope["event_id"])).rowcount
77 + if not inserted:
78 + return # duplicate or replay — already applied
79 + apply_side_effects(envelope) # same transaction where possible
80 + ack() # only after commit
81 +```
82 +
83 +Prune `consumed_events` older than the broker's retention window.
84 +
85 +## Ack discipline
86 +
87 +```python
88 +# ❌ ack-then-process: crash after ack = event lost forever
89 +msg = consume(); ack(msg); process(msg)
90 +
91 +# ✅ process-then-ack: crash before ack = redelivery, dedup absorbs it
92 +msg = consume(); process(msg); ack(msg)
93 +```
94 +
95 +## Schema evolution
96 +
97 +Additive (same topic, bump minor): add optional field with a default.
98 +
99 +Breaking (new topic): publish both during migration.
100 +
101 +```python
102 +broker.publish("order_placed", v1_envelope) # until last v1 consumer migrates
103 +broker.publish("order_placed.v2", v2_envelope)
104 +```
105 +
106 +Never: rename/retype a field in place, or repurpose an existing field.
107 +
108 +## DLQ routing
109 +
110 +```python
111 +MAX_ATTEMPTS = 5 # transient issues resolve well before 5 spaced retries
112 +
113 +def consume_loop(msg):
114 + try:
115 + handle(msg)
116 + except Exception:
117 + if msg.delivery_count >= MAX_ATTEMPTS:
118 + broker.publish("orders.dlq", msg.envelope,
119 + headers={"error": traceback.format_exc(limit=3)})
120 + ack(msg) # remove poison message from the main stream
121 + alert("orders.dlq received a message")
122 + else:
123 + nack(msg) # broker redelivers with backoff
124 +```
125 +
126 +## Gotchas
127 +
128 +- **Outbox relay + `UPDATE` in one transaction with the publish** is
129 + impossible — the broker isn't in your DB transaction. Accept relay
130 + at-least-once; dedup downstream.
131 +- **Keying by random UUID** destroys per-entity ordering; key by the entity
132 + whose sequence matters.
133 +- **Consumer group rebalances** redeliver in-flight messages — another
134 + duplicate source the dedup table must absorb.
135 +- **Fat events as API snapshots** rot: consumers act on stale fields.
136 + Carry IDs + the facts of the event; refetch the rest.
137 +- **Retention < replay need**: if history matters, size retention (or an
138 + archive) before the first consumer bug, not after.
139 +- **One shared DLQ for all topics** makes triage impossible; one DLQ per
140 + topic, each with its own alert.
added backend-skills/handling-errors/SKILL.md +47 −0
@@ -0,0 +1,47 @@
1 +---
2 +name: handling-errors
3 +description: Designs error handling for backend services — error taxonomy, HTTP problem+json responses (RFC 9457), retry semantics with backoff and jitter, circuit breakers, and fail-fast startup. Use when the user asks how to handle, structure, or standardize errors or exceptions in an API or service, design error responses, add retries or a circuit breaker, or fix swallowed/double-logged exceptions. Do not use for validating request input (validating-input) or for logging/metrics/tracing pipelines (instrumenting-observability).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Handling Errors
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** structuring errors and exceptions in backend code, API error responses, retry/circuit-breaker policy, startup failure behavior.
15 +- **Do NOT use for:** request-validation rules and 400-level field errors (validating-input) or log/metric/trace plumbing (instrumenting-observability).
16 +
17 +## Core rules
18 +
19 +1. **Expected failures are values, exceptional failures are exceptions.** "User not found" is a normal outcome — return it. A broken DB socket is exceptional — raise it. Follow the language idiom (Result/Either in Rust/TS-fp, exceptions in Python/Java).
20 +2. **Keep the taxonomy small.** 4–6 error classes mapped to HTTP: `ValidationError`→400, `AuthError`→401/403, `NotFound`→404, `Conflict`→409, `RateLimited`→429, everything else→500. New error types must argue their way in.
21 +3. **API errors are problem+json (RFC 9457).**
22 + - ✅ `{"type":"https://api.example.com/errors/quota","title":"Quota exceeded","status":429,"detail":"Plan allows 100 reports/day.","instance":"/reports/123"}`
23 + - ❌ `{"error": "something went wrong"}` or a raw stack trace.
24 +4. **Log or re-raise — never both.** Handling an exception twice produces double logs and masks the real failure point. Catch only where you can act; add context and re-raise otherwise.
25 + - ✅ `except PaymentError as e: raise OrderError(order_id=id) from e`
26 + - ❌ `except Exception: logger.error(e); raise` at every layer.
27 +5. **User-facing messages say what to do next; internals stay in logs.** ✅ "Payment declined — try another card." ❌ "psycopg2.OperationalError: connection refused at 10.0.3.7:5432".
28 +6. **Retry only idempotent operations,** with exponential backoff + full jitter (base 200 ms, factor 2, cap 30 s, max 5 attempts — bounded work, no thundering herd) and respect `Retry-After` when present.
29 +7. **Wrap flaky dependencies in a circuit breaker** (open after 5 consecutive failures, half-open probe after 30 s) so one dead downstream doesn't exhaust your threads.
30 +8. **Fail fast at startup.** Missing config, unreachable migrations, bad credentials → crash with a named cause before serving traffic. Never boot into a half-working state.
31 +
32 +## Workflow
33 +
34 +1. List the operation's failure modes; split expected vs exceptional.
35 +2. Map each to the taxonomy (rule 2) and its HTTP status; add a new class only if none fits.
36 +3. Implement handlers at the boundary layer only (HTTP middleware / job wrapper), converting taxonomy → problem+json.
37 +4. Add retry/breaker policy for each external dependency (rules 6–7).
38 +5. Validate: trigger each failure mode in a test — assert the status code, the problem+json shape, exactly one log entry per failure, and no stack trace in the response body.
39 +
40 +## Edge cases & failure modes
41 +- **Partial failure in a batch** → return 207-style per-item results or a summary object; never fail the whole batch silently.
42 +- **Retryable error during a non-idempotent call** → do not retry; surface it. Make the call idempotent first (idempotency keys) if retries are required.
43 +- **Error while handling an error** (e.g., logger down) → last-resort handler writes to stderr and returns a static 500 body; never raise from the handler.
44 +- **Timeout vs failure ambiguity** → treat timeouts as unknown-outcome: only retry with an idempotency key.
45 +
46 +## References
47 +Deeper recipes and gotchas: see [references/patterns.md](references/patterns.md)
added backend-skills/handling-errors/references/patterns.md +116 −0
@@ -0,0 +1,116 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Handling Errors
7 +
8 +## Contents
9 +- Error taxonomy skeleton (Python)
10 +- Boundary handler → problem+json
11 +- Retry with exponential backoff + full jitter
12 +- Circuit breaker
13 +- Fail-fast startup
14 +- Gotchas
15 +
16 +## Error taxonomy skeleton (Python)
17 +
18 +```python
19 +class AppError(Exception):
20 + status = 500
21 + title = "Internal error"
22 + def __init__(self, detail="", **ctx):
23 + super().__init__(detail)
24 + self.detail, self.ctx = detail, ctx
25 +
26 +class ValidationError(AppError): status, title = 400, "Invalid request"
27 +class AuthError(AppError): status, title = 401, "Authentication required"
28 +class Forbidden(AppError): status, title = 403, "Not allowed"
29 +class NotFound(AppError): status, title = 404, "Resource not found"
30 +class Conflict(AppError): status, title = 409, "Conflict"
31 +class RateLimited(AppError): status, title = 429, "Too many requests"
32 +```
33 +
34 +## Boundary handler → problem+json
35 +
36 +One handler at the HTTP edge; nothing below it formats responses.
37 +
38 +```python
39 +@app.exception_handler(AppError)
40 +def app_error(request, exc):
41 + # Exactly one log line per failure, with context, at the boundary.
42 + logger.warning("request_failed", status=exc.status,
43 + error=type(exc).__name__, **exc.ctx)
44 + return JSONResponse(status_code=exc.status, media_type="application/problem+json",
45 + content={"type": f"https://api.example.com/errors/{type(exc).__name__}",
46 + "title": exc.title, "status": exc.status,
47 + "detail": exc.detail, "instance": str(request.url.path)})
48 +
49 +@app.exception_handler(Exception)
50 +def unexpected(request, exc):
51 + logger.exception("unhandled_error") # full trace to logs only
52 + return JSONResponse(status_code=500, media_type="application/problem+json",
53 + content={"title": "Internal error", "status": 500,
54 + "detail": "Unexpected error. Retry or contact support."})
55 +```
56 +
57 +## Retry with exponential backoff + full jitter
58 +
59 +```python
60 +import random, time
61 +
62 +# base 0.2s, factor 2, cap 30s, 5 attempts: worst-case wait ~ <60s total.
63 +def retry(fn, retryable=(TimeoutError, ConnectionError),
64 + attempts=5, base=0.2, cap=30.0):
65 + for n in range(attempts):
66 + try:
67 + return fn()
68 + except retryable:
69 + if n == attempts - 1:
70 + raise
71 + time.sleep(random.uniform(0, min(cap, base * 2 ** n))) # full jitter
72 +```
73 +
74 +Honor server hints: if the response carries `Retry-After: N`, sleep `N` seconds instead of the computed backoff.
75 +
76 +## Circuit breaker
77 +
78 +```python
79 +class Breaker:
80 + # 5 consecutive failures opens; probe after 30s (half-open).
81 + def __init__(self, threshold=5, reset_after=30.0):
82 + self.fail, self.threshold, self.reset_after = 0, threshold, reset_after
83 + self.opened_at = None
84 + def call(self, fn):
85 + if self.opened_at is not None:
86 + if time.monotonic() - self.opened_at < self.reset_after:
87 + raise DependencyDown("circuit open")
88 + self.opened_at = None # half-open: allow one probe
89 + try:
90 + out = fn()
91 + except Exception:
92 + self.fail += 1
93 + if self.fail >= self.threshold:
94 + self.opened_at = time.monotonic()
95 + raise
96 + self.fail = 0
97 + return out
98 +```
99 +
100 +## Fail-fast startup
101 +
102 +```python
103 +def main():
104 + cfg = load_config() # raises with the missing key named
105 + db.ping(cfg.database_url) # unreachable DB -> crash now, not at first request
106 + run_pending_migration_check(cfg)
107 + serve(cfg)
108 +```
109 +
110 +## Gotchas
111 +- `except Exception: pass` hides bugs for months — if a failure is truly ignorable, log it at DEBUG with a reason string.
112 +- Re-raising with `raise NewError(...) from e` preserves the chain; bare `raise NewError(...)` destroys the original traceback.
113 +- Retrying a POST without an idempotency key can double-charge/double-create — timeouts are *unknown outcome*, not failure.
114 +- Breakers per dependency, not global — one dead cache must not open the DB breaker.
115 +- problem+json `type` URLs should be stable identifiers; they don't have to resolve, but never reuse one for a different meaning.
116 +- 500 bodies must be static — rendering them from the exception risks leaking internals and can itself fail.
added backend-skills/handling-file-uploads/SKILL.md +45 −0
@@ -0,0 +1,45 @@
1 +---
2 +name: handling-file-uploads
3 +description: Designs safe, scalable file-upload handling — presigned direct-to-storage URLs for large files, multipart for small ones, magic-byte content validation, server-generated storage names, streaming instead of buffering, and resumable chunked uploads. Use when the user asks to implement or review file, image, video, or document uploads, add an upload endpoint, validate uploaded files, or generate presigned S3/GCS upload URLs. Do not use for serving or downloading files, image processing/resizing, or CDN configuration.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Handling File Uploads
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** upload endpoint design, presigned-URL flows, upload validation and limits, storage naming, resumable uploads, virus-scanning hook points.
15 +- **Do NOT use for:** download/serving paths, thumbnailing or transcoding pipelines, CDN/cache setup — handle those separately.
16 +
17 +## Core rules
18 +
19 +1. **Files >5 MB go direct to storage via presigned URL — never through the app.** The API issues a short-lived presigned PUT (15 min expiry, content-length-range enforced), the client uploads to S3/GCS, then confirms. App servers proxy only small multipart uploads.
20 + - ✅ `POST /uploads` → `{upload_url, file_id}` → client PUTs to storage → `POST /uploads/{file_id}/complete`
21 + - ❌ 500 MB video buffered through the web server's memory
22 +2. **Enforce size limits before buffering.** Reject on `Content-Length` first, and hard-cap the stream anyway (clients lie). Set limits per file type: images 10 MB, documents 25 MB, video via resumable only — and put the numbers in the error message.
23 +3. **Validate content, not just extension.** Check magic bytes (e.g. `%PDF`, `\x89PNG`) against the claimed type; reject mismatches. Extension and `Content-Type` header are client-controlled hints, nothing more.
24 +4. **Never trust the client filename.** Generate the storage key server-side (`uploads/2026/08/{uuid4}.pdf`); keep the original name as escaped metadata only. This kills path traversal (`../../etc/cron.d/x`) and collision attacks.
25 +5. **Stream, don't buffer.** Small-path multipart parsing goes chunk-by-chunk to disk/storage; memory use must be O(chunk), not O(file).
26 +6. **Uploads are pending until scanned and confirmed.** New objects land in a quarantine prefix/bucket with a `pending` status; a scanning hook (ClamAV, provider malware scan) promotes to `clean` or deletes. Serve nothing from quarantine.
27 +7. **Very large files use chunked/resumable uploads** (S3 multipart or tus): 8 MB parts, per-part checksums and retries, abort-and-clean incomplete uploads older than 24 h (lifecycle rule).
28 +
29 +## Workflow
30 +
31 +1. Classify expected uploads: types, size ranges, volume → pick path per rule 1/7.
32 +2. Define the presigned flow endpoints (`create`, `complete`) or the multipart endpoint with limits (rule 2).
33 +3. Implement validation: magic bytes (rule 3), server-side naming (rule 4).
34 +4. Wire the quarantine + scan hook (rule 6) and lifecycle cleanup (rule 7).
35 +5. **Validate:** upload (a) an oversized file → clean 413 with the limit named; (b) an `.exe` renamed `.png` → rejected by magic bytes; (c) a filename `../../x` → stored under the generated UUID key with metadata escaped; (d) a happy-path file → reaches `clean` status and only then is retrievable.
36 +
37 +## Edge cases & failure modes
38 +- **Client never calls `complete`** → the `pending` record and quarantine object are garbage-collected by the 24 h lifecycle rule.
39 +- **Duplicate uploads** → hash the content (SHA-256) at scan time; either dedupe by hash or at least record it for later dedup/audit.
40 +- **Zip/archive uploads** → beware zip bombs: cap decompressed size and entry count before extracting anything.
41 +- **SVG uploads** → they are executable XML (scripts, external entities); sanitize or serve with `Content-Disposition: attachment` + strict CSP, never inline from your origin.
42 +- **Presigned URL leaks** → 15-minute expiry plus `content-length-range` bounds the damage; never sign without both.
43 +
44 +## References
45 +Deeper patterns (presigned flow code, magic-byte table, tus/multipart setup, lifecycle rules): see [references/patterns.md](references/patterns.md).
added backend-skills/handling-file-uploads/references/patterns.md +129 −0
@@ -0,0 +1,129 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Handling File Uploads
7 +
8 +## Contents
9 +- Presigned direct-to-storage flow
10 +- Magic-byte validation table
11 +- Streaming multipart limit enforcement
12 +- Quarantine and scan promotion
13 +- Resumable uploads and lifecycle cleanup
14 +- Gotchas
15 +
16 +## Presigned direct-to-storage flow
17 +
18 +```python
19 +# 1) Client asks to upload
20 +# POST /v1/uploads {"filename": "report.pdf", "content_type": "application/pdf", "size": 8123456}
21 +import boto3, uuid
22 +
23 +# 15 min: enough for a slow client on the declared size, short enough that a
24 +# leaked URL is a bounded liability.
25 +PRESIGN_TTL = 900
26 +
27 +def create_upload(req):
28 + validate_declared(req.content_type, req.size) # rule 2 pre-check
29 + file_id = str(uuid.uuid4())
30 + key = f"quarantine/{file_id}.pdf" # rule 4: server key
31 + url = s3.generate_presigned_post(
32 + Bucket="uploads", Key=key,
33 + Fields={"Content-Type": req.content_type},
34 + Conditions=[
35 + {"Content-Type": req.content_type},
36 + ["content-length-range", 1, 26_214_400], # 25 MB doc cap
37 + ],
38 + ExpiresIn=PRESIGN_TTL)
39 + db.insert("uploads", id=file_id, key=key, status="pending",
40 + original_name=req.filename, declared_size=req.size)
41 + return {"file_id": file_id, "upload": url}
42 +
43 +# 2) Client PUT/POSTs the file to `url`
44 +# 3) POST /v1/uploads/{file_id}/complete → server HEADs the object, verifies
45 +# size matches, kicks the scan job, returns {"status": "scanning"}
46 +```
47 +
48 +## Magic-byte validation table
49 +
50 +```python
51 +MAGIC = {
52 + "image/png": [b"\x89PNG\r\n\x1a\n"],
53 + "image/jpeg": [b"\xff\xd8\xff"],
54 + "image/webp": [b"RIFF"], # + b"WEBP" at offset 8
55 + "application/pdf": [b"%PDF"],
56 + "application/zip": [b"PK\x03\x04"], # also docx/xlsx/pptx containers
57 +}
58 +
59 +def sniff_ok(claimed: str, head: bytes) -> bool:
60 + sigs = MAGIC.get(claimed)
61 + return bool(sigs) and any(head.startswith(s) for s in sigs)
62 +# Read the first 16 bytes from storage (ranged GET) — never rely on the
63 +# client's Content-Type alone.
64 +```
65 +
66 +## Streaming multipart limit enforcement
67 +
68 +```python
69 +# FastAPI/Starlette example — same idea in any framework: consume the stream
70 +# in chunks and abort the moment the cap is crossed.
71 +CHUNK = 64 * 1024
72 +MAX_IMAGE = 10 * 1024 * 1024
73 +
74 +async def save_stream(stream, dest, cap=MAX_IMAGE):
75 + written = 0
76 + with open(dest, "wb") as f:
77 + async for chunk in stream:
78 + written += len(chunk)
79 + if written > cap:
80 + f.close(); os.unlink(dest)
81 + raise Payload413(f"file exceeds {cap // 1_048_576} MB limit")
82 + f.write(chunk)
83 + return written
84 +```
85 +
86 +## Quarantine and scan promotion
87 +
88 +```
89 +quarantine/{uuid}.{ext} status=pending (not servable)
90 + │ scan job (ClamAV / provider malware scan)
91 + ├── clean → copy to files/{uuid}.{ext}, status=clean, delete quarantine copy
92 + └── infected → delete object, status=rejected, notify uploader
93 +```
94 +Record `sha256` at scan time for dedup/audit. The serving layer reads only
95 +`files/` and only rows with `status=clean`.
96 +
97 +## Resumable uploads and lifecycle cleanup
98 +
99 +```python
100 +# S3 multipart: 8 MB parts — large enough to keep part count low (10k max),
101 +# small enough that a retry wastes little.
102 +PART_SIZE = 8 * 1024 * 1024
103 +```
104 +
105 +Bucket lifecycle rules (Terraform-style):
106 +
107 +```hcl
108 +rule { id = "abort-incomplete" abort_incomplete_multipart_upload_days = 1 }
109 +rule { id = "purge-quarantine" prefix = "quarantine/" expiration_days = 1 }
110 +```
111 +
112 +For browser-based resumable uploads prefer tus (tusd server or provider
113 +equivalent) over hand-rolled chunk endpoints.
114 +
115 +## Gotchas
116 +
117 +- **`Content-Length` is a claim, not a fact** — always cap the actual stream
118 + too (rule 2 belt-and-braces).
119 +- **docx/xlsx/pptx sniff as zip** (`PK\x03\x04`) — accept the zip signature,
120 + then check the internal `[Content_Types].xml` if you must distinguish.
121 +- **Filename header injection:** original filenames go into
122 + `Content-Disposition` later; store them escaped and serve with
123 + `filename*=UTF-8''...` encoding, or attackers smuggle header content.
124 +- **Presigning with no Content-Type condition** lets an attacker upload
125 + `text/html` to your bucket and phish from your domain.
126 +- **EXIF in images** may carry GPS/PII — strip metadata at scan/processing
127 + time if images are re-served publicly.
128 +- **Multipart ETag ≠ MD5** on S3 multipart uploads — use explicit per-part
129 + checksums (`ChecksumSHA256`) for integrity, not ETag comparison.
added backend-skills/implementing-authentication/SKILL.md +48 −0
@@ -0,0 +1,48 @@
1 +---
2 +name: implementing-authentication
3 +description: Implements identity verification for backend services — password storage, session cookies vs JWT, OAuth2/OIDC login flows, MFA, and password reset. Use when the user asks to add login, signup, authentication, sessions, JWTs, OAuth/OIDC or social login, password hashing, or password reset to a service. Do not use for permission checks after login (implementing-authorization) or for database credentials and roles (securing-databases).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Implementing Authentication
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** login/signup flows, password storage, session or token management, OAuth2/OIDC integration, MFA, password reset.
15 +- **Do NOT use for:** deciding what a logged-in user may do (implementing-authorization), database users/roles (db-skills/securing-databases), or general service hardening (securing-backend-services).
16 +
17 +## Core rules
18 +
19 +1. **Hash passwords with argon2id.** bcrypt (cost ≥12) is the acceptable fallback when argon2 is unavailable.
20 + - ✅ `argon2id(password)` via a maintained library
21 + - ❌ `sha256(password + salt)` — fast hashes are crackable at scale; never MD5/SHA-family alone
22 +2. **First-party web apps default to server-side sessions in cookies**, not JWTs.
23 + - Cookie flags always: `HttpOnly; Secure; SameSite=Lax` (Strict for admin surfaces).
24 + - JWTs are for service-to-service and mobile/SPA APIs: expiry ≤15 min, paired with rotating refresh tokens, revocation list for logout.
25 +3. **Third-party login uses OAuth2 authorization code + PKCE.** Never the implicit flow; never roll your own OAuth client if the framework has one.
26 +4. **Rate-limit credential endpoints** (login, signup, reset): per-IP and per-account. Check new passwords against a breach corpus (e.g. haveibeenpwned k-anonymity API).
27 +5. **Password reset:** single-use token, expiry ≤1 hour, stored hashed, sent by email link only.
28 + - ✅ Response is identical whether the account exists or not ("If that address exists, we sent a link.")
29 + - ❌ "No account with that email" — user enumeration
30 +6. **MFA hooks:** TOTP as default second factor; enforce at login and before sensitive actions (payout, email change). Recovery codes generated once, stored hashed.
31 +7. **Never log or echo credentials, tokens, or session IDs.** Rotate the session ID on privilege change (login, MFA pass) to block session fixation.
32 +
33 +## Workflow
34 +
35 +1. Pick the mechanism with the decision rule in rule 2 (sessions vs JWT vs OIDC).
36 +2. Implement storage: user table with `password_hash` (argon2id), no plaintext or reversible encryption anywhere.
37 +3. Implement the flow with the framework's primitives (e.g. FastAPI + `authlib`, Express + `passport`); wire rate limits (rule 4).
38 +4. Add reset + MFA per rules 5–6.
39 +5. **Validate:** attempt login with wrong password (must fail generically), replay an expired/rotated token (must fail), inspect the Set-Cookie header for `HttpOnly; Secure; SameSite`, and confirm the reset flow gives identical responses for existing and unknown emails. Fix and re-test until all four pass.
40 +
41 +## Edge cases & failure modes
42 +- **Existing weak hashes (MD5/SHA1)** → rehash transparently on next successful login; force reset for dormant accounts.
43 +- **Clock skew with JWTs** → allow ≤60 s leeway on `exp`/`nbf`, never more.
44 +- **Lockout abuse** (attacker locking victims out) → prefer progressive delays + CAPTCHA over hard lockout.
45 +- **OAuth provider returns unverified email** → treat as unverified; require confirmation before linking accounts.
46 +
47 +## References
48 +Copy-paste patterns and gotchas: see [references/patterns.md](references/patterns.md).
added backend-skills/implementing-authentication/references/patterns.md +133 −0
@@ -0,0 +1,133 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Implementing Authentication
7 +
8 +## Contents
9 +- Password hashing (argon2id)
10 +- Session cookie setup
11 +- JWT issuance and verification
12 +- OAuth2 authorization code + PKCE
13 +- Password reset flow
14 +- TOTP MFA
15 +- Gotchas
16 +
17 +## Password hashing (argon2id)
18 +
19 +```python
20 +from argon2 import PasswordHasher
21 +from argon2.exceptions import VerifyMismatchError
22 +
23 +ph = PasswordHasher() # library defaults follow current OWASP guidance
24 +
25 +def hash_password(password: str) -> str:
26 + return ph.hash(password)
27 +
28 +def verify_password(stored: str, candidate: str) -> bool:
29 + try:
30 + ph.verify(stored, candidate)
31 + except VerifyMismatchError:
32 + return False
33 + # Transparent upgrade if parameters changed since hashing
34 + if ph.check_needs_rehash(stored):
35 + return True # caller should rehash and store
36 + return True
37 +```
38 +
39 +Fallback: `bcrypt.hashpw(pw, bcrypt.gensalt(rounds=12))` — bcrypt truncates at 72 bytes; reject longer passwords explicitly.
40 +
41 +## Session cookie setup
42 +
43 +```python
44 +# Flask example — equivalents exist in every framework
45 +app.config.update(
46 + SESSION_COOKIE_HTTPONLY=True,
47 + SESSION_COOKIE_SECURE=True,
48 + SESSION_COOKIE_SAMESITE="Lax",
49 + PERMANENT_SESSION_LIFETIME=timedelta(hours=12),
50 +)
51 +
52 +# On every privilege change (login, MFA pass):
53 +session.regenerate() # or: logout_user(); new session id — blocks fixation
54 +```
55 +
56 +Store sessions server-side (Redis/DB) so logout and admin revocation are real.
57 +
58 +## JWT issuance and verification
59 +
60 +```python
61 +import jwt, datetime as dt
62 +
63 +ACCESS_TTL = dt.timedelta(minutes=15) # short: leaked tokens age out fast
64 +LEEWAY = 60 # seconds of clock-skew tolerance, max
65 +
66 +def issue(sub: str, secret: str) -> str:
67 + now = dt.datetime.now(dt.timezone.utc)
68 + return jwt.encode(
69 + {"sub": sub, "iat": now, "exp": now + ACCESS_TTL, "iss": "api"},
70 + secret, algorithm="HS256")
71 +
72 +def verify(token: str, secret: str) -> dict:
73 + # Pin the algorithm list — never accept the header's alg claim blindly
74 + return jwt.decode(token, secret, algorithms=["HS256"],
75 + issuer="api", leeway=LEEWAY)
76 +```
77 +
78 +Refresh tokens: opaque random strings, stored hashed, rotated on every use; reuse of a rotated token revokes the whole family (theft signal).
79 +
80 +## OAuth2 authorization code + PKCE
81 +
82 +```python
83 +# authlib example (FastAPI/Starlette)
84 +oauth.register(
85 + name="google",
86 + server_metadata_url="https://accounts.google.com/.well-known/openid-configuration",
87 + client_id=..., client_secret=...,
88 + client_kwargs={"scope": "openid email profile", "code_challenge_method": "S256"},
89 +)
90 +# Callback: verify state, exchange code, then check id_token claims:
91 +# - aud == your client_id
92 +# - email_verified is True before trusting the email
93 +```
94 +
95 +## Password reset flow
96 +
97 +```python
98 +def start_reset(email: str):
99 + user = find_user(email)
100 + if user:
101 + raw = secrets.token_urlsafe(32)
102 + store_reset(user.id, sha256(raw), expires=now() + timedelta(hours=1))
103 + send_email(email, link_with(raw))
104 + return "If that address exists, we sent a link." # identical either way
105 +
106 +def finish_reset(raw: str, new_password: str):
107 + row = pop_reset(sha256(raw)) # single-use: delete on read
108 + if row is None or row.expired():
109 + raise ResetInvalid # generic error, no detail
110 + set_password(row.user_id, hash_password(new_password))
111 + revoke_all_sessions(row.user_id) # kill attacker's live sessions
112 +```
113 +
114 +## TOTP MFA
115 +
116 +```python
117 +import pyotp
118 +
119 +secret = pyotp.random_base32() # store encrypted, show QR once
120 +totp = pyotp.TOTP(secret)
121 +ok = totp.verify(code, valid_window=1) # ±30 s window, no more
122 +# Recovery codes: 8-10 random codes, stored hashed, single-use.
123 +```
124 +
125 +## Gotchas
126 +
127 +- **`alg: none` / algorithm confusion** — always pin `algorithms=[...]` when decoding JWTs; never trust the token header.
128 +- **bcrypt 72-byte truncation** — `password[:72]` collisions; validate length or pre-hash with SHA-256+base64 before bcrypt.
129 +- **SameSite=Lax still sends cookies on top-level GET navigation** — state-changing endpoints must be POST with CSRF protection.
130 +- **Timing side channel on login** — run the hash verify even when the user doesn't exist (hash a dummy) so response time doesn't reveal account existence.
131 +- **JWT logout is not logout** — without a revocation list or short TTL, "logged out" tokens keep working until expiry.
132 +- **OAuth `state` skipped** — omitting the state check re-opens CSRF on the callback; PKCE does not replace it for web apps.
133 +- **Storing TOTP secrets in plaintext** — encrypt at rest; a DB dump otherwise defeats MFA entirely.
added backend-skills/implementing-authorization/SKILL.md +46 −0
@@ -0,0 +1,46 @@
1 +---
2 +name: implementing-authorization
3 +description: Implements permission decisions in backend services — RBAC/ABAC models, resource-ownership checks, IDOR prevention, multi-tenant isolation, and centralized policy enforcement. Use when the user asks to add roles, permissions, access control, admin-only routes, ownership checks, tenant isolation, or asks why a user can see another user's data. Do not use for identity verification and login flows (implementing-authentication) or OS and file-system permissions.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Implementing Authorization
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** deciding what an authenticated caller may do — roles, permissions, ownership checks, tenant isolation, policy middleware, privileged-action auditing.
15 +- **Do NOT use for:** logging users in (implementing-authentication), database GRANTs (db-skills/securing-databases), or OS/file permissions.
16 +
17 +## Core rules
18 +
19 +1. **Authorize on every request, server-side.** Client-side hiding of buttons is UX, not security.
20 +2. **Deny by default.** Routes without an explicit policy are rejected, not allowed.
21 + - ✅ `@require(perm="reports:read")` on each route; unannotated routes 403 in middleware
22 + - ❌ "Only add checks to the sensitive endpoints"
23 +3. **RBAC is the default model** (user → roles → permissions). Reach for ABAC (attribute rules) only when decisions depend on resource attributes (owner, status, amount thresholds).
24 +4. **Authentication is not authorization — check ownership.** Fetching by ID must scope to the caller.
25 + - ✅ `SELECT ... WHERE id = :id AND owner_id = :caller`
26 + - ❌ `SELECT ... WHERE id = :id` after login — classic IDOR
27 +5. **Multi-tenant: scope every query by `tenant_id`,** derived from the session/token — never from the request body or URL. Use Postgres row-level security as a backstop where available.
28 +6. **Centralize policy in one module/middleware.** Scattered `if user.role == "admin"` checks drift and rot; route handlers call `authorize(caller, action, resource)` and nothing else.
29 +7. **Audit privileged actions** (role grants, data exports, deletions): who, what, when, from where — to an append-only log.
30 +
31 +## Workflow
32 +
33 +1. Enumerate actions and resources; write the permission matrix (roles × actions) before code.
34 +2. Implement the central `authorize()` + deny-by-default middleware (rule 2, 6).
35 +3. Add ownership/tenant scoping at the data layer (rules 4–5) so a missed route check cannot leak cross-tenant data.
36 +4. Wire the audit log for privileged actions (rule 7).
37 +5. **Validate:** as user A, request user B's resource by ID (expect 403/404); as a role without the permission, call each privileged route (expect 403); confirm an unannotated test route is rejected by default. All three must pass before shipping.
38 +
39 +## Edge cases & failure modes
40 +- **403 vs 404:** return 404 for resources the caller must not know exist (cross-tenant); 403 within a tenant where existence is not secret. Pick per resource and stay consistent.
41 +- **Role changes mid-session** → re-read roles per request (or short cache ≤60 s); revoke sessions on demotion.
42 +- **Background jobs and internal services** → they get their own scoped identities, never a shared "system = superuser" that skips `authorize()`.
43 +- **Batch endpoints** → authorize each item, not just the endpoint; report per-item denials.
44 +
45 +## References
46 +Copy-paste patterns and gotchas: see [references/patterns.md](references/patterns.md).
added backend-skills/implementing-authorization/references/patterns.md +109 −0
@@ -0,0 +1,109 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Implementing Authorization
7 +
8 +## Contents
9 +- Permission matrix and RBAC tables
10 +- Central authorize() and deny-by-default middleware
11 +- Ownership scoping (IDOR prevention)
12 +- Multi-tenant isolation with RLS backstop
13 +- Audit logging
14 +- Gotchas
15 +
16 +## Permission matrix and RBAC tables
17 +
18 +```sql
19 +CREATE TABLE roles (id serial PRIMARY KEY, name text UNIQUE NOT NULL);
20 +CREATE TABLE permissions (id serial PRIMARY KEY, name text UNIQUE NOT NULL); -- "reports:read"
21 +CREATE TABLE role_permissions (
22 + role_id int REFERENCES roles, permission_id int REFERENCES permissions,
23 + PRIMARY KEY (role_id, permission_id));
24 +CREATE TABLE user_roles (
25 + user_id bigint REFERENCES users, role_id int REFERENCES roles,
26 + PRIMARY KEY (user_id, role_id));
27 +```
28 +
29 +Name permissions `resource:action`; keep the matrix in a checked-in doc so
30 +reviews see permission changes as diffs.
31 +
32 +## Central authorize() and deny-by-default middleware
33 +
34 +```python
35 +class Forbidden(Exception): ...
36 +
37 +def authorize(caller, action: str, resource=None):
38 + if action not in caller.permissions:
39 + raise Forbidden(action)
40 + if resource is not None and hasattr(resource, "owner_id"):
41 + if resource.owner_id != caller.id and "override:ownership" not in caller.permissions:
42 + raise Forbidden(f"{action} on foreign resource")
43 +
44 +# Deny-by-default: routes must declare a policy or be rejected.
45 +@app.middleware("http")
46 +async def enforce_policy(request, call_next):
47 + endpoint = request.scope.get("endpoint")
48 + if endpoint is None or not getattr(endpoint, "_policy", None):
49 + return JSONResponse({"detail": "no policy declared"}, status_code=403)
50 + return await call_next(request)
51 +
52 +def require(perm: str):
53 + def deco(fn):
54 + fn._policy = perm
55 + @wraps(fn)
56 + async def inner(request, *a, **kw):
57 + authorize(request.state.caller, perm)
58 + return await fn(request, *a, **kw)
59 + return inner
60 + return deco
61 +```
62 +
63 +## Ownership scoping (IDOR prevention)
64 +
65 +```python
66 +# Scope in the query itself — a forgotten route check then returns 404, not a leak.
67 +def get_document(db, doc_id: int, caller):
68 + row = db.execute(
69 + "SELECT * FROM documents WHERE id = :id AND owner_id = :owner",
70 + {"id": doc_id, "owner": caller.id}).fetchone()
71 + if row is None:
72 + raise NotFound # don't reveal existence of others' docs
73 + return row
74 +```
75 +
76 +## Multi-tenant isolation with RLS backstop
77 +
78 +```python
79 +# tenant_id comes from the verified session/token — NEVER from the request.
80 +tenant_id = caller.tenant_id
81 +db.execute("SET app.tenant_id = :t", {"t": tenant_id})
82 +```
83 +
84 +```sql
85 +ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
86 +CREATE POLICY tenant_isolation ON invoices
87 + USING (tenant_id = current_setting('app.tenant_id')::bigint);
88 +-- Backstop: even a query missing the WHERE clause cannot cross tenants.
89 +```
90 +
91 +## Audit logging
92 +
93 +```python
94 +def audit(caller, action, target, request):
95 + audit_log.insert( # append-only table / stream
96 + actor=caller.id, action=action, target=target,
97 + ip=request.client.host, at=utcnow())
98 +
99 +# Call on: role grants, exports, deletions, impersonation, settings changes.
100 +```
101 +
102 +## Gotchas
103 +
104 +- **Checking the route but not the query** — a second entry point (GraphQL, admin API, background job) reuses the unscoped query and leaks. Scope at the data layer.
105 +- **tenant_id taken from the URL/body** — attacker just edits it. Only the token/session is trusted.
106 +- **Caching authorization results too long** — demoted admins keep power; cap policy caches at ~60 s or bust on role change.
107 +- **`is_admin` boolean creep** — one flag becomes god-mode everywhere and can't be audited; use named permissions even for admins.
108 +- **RLS silently disabled for table owners** — Postgres table owners bypass RLS unless `FORCE ROW LEVEL SECURITY` is set; app roles must not own the tables.
109 +- **404-vs-403 inconsistency** — mixing them per route lets attackers map which IDs exist; decide per resource class and enforce in one place.
added backend-skills/instrumenting-observability/SKILL.md +45 −0
@@ -0,0 +1,45 @@
1 +---
2 +name: instrumenting-observability
3 +description: Instruments backend services with structured JSON logs, correlation IDs, RED metrics, and OpenTelemetry traces, plus alerting on symptoms and golden-signal dashboards. Use when the user asks to add or improve logging, metrics, tracing, monitoring, alerts, or dashboards for a service, propagate request IDs, or pick log levels. Do not use for structuring error-handling code itself (handling-errors) or for incident-response process and runbooks.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Instrumenting Observability
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** logging strategy, metrics, distributed tracing, alert and dashboard design for backend services.
15 +- **Do NOT use for:** how code raises/maps errors (handling-errors) or writing incident runbooks/postmortems.
16 +
17 +## Core rules
18 +
19 +1. **Logs are structured JSON, one event per line,** with consistent field names across all services (`ts`, `level`, `service`, `request_id`, `event`, then context).
20 + - ✅ `{"ts":"2026-08-05T14:02:11Z","level":"info","service":"orders","request_id":"r-9f2","event":"order_created","order_id":123}`
21 + - ❌ `print(f"created order {id}!!")`
22 +2. **Every request gets a correlation ID** — accept inbound `X-Request-ID` (else generate one), attach it to every log line via context, and forward it on every outbound call.
23 +3. **Log levels have contracts.** ERROR = a human should act; WARN = degraded but self-healing; INFO = significant state change; DEBUG = development detail, disabled in production. If nobody would act on it, it is not ERROR.
24 +4. **Emit RED metrics per endpoint** — Rate, Errors, Duration (as a histogram, not an average) — plus the handful of business metrics that matter (orders_created, payments_failed).
25 +5. **OpenTelemetry is the default for traces** (and its semantic conventions for names). Auto-instrument HTTP/DB clients first; add manual spans only around meaningful units of work.
26 +6. **Never log secrets or PII.** Maintain a denylist (password, token, authorization, card fields), scrub at the logger layer, and review new log statements for payload dumps.
27 +7. **Alert on symptoms, not causes** — SLO burn rate, error ratio, p99 latency. CPU at 80% is not a page; users receiving 500s is.
28 +8. **One dashboard per service, golden signals first** (latency, traffic, errors, saturation), business metrics second. If a panel never changed a decision, delete it.
29 +
30 +## Workflow
31 +
32 +1. Add the shared logging setup (JSON formatter + context injection) and the request-ID middleware.
33 +2. Instrument RED metrics on every route and consumer; histogram buckets sized to the SLO.
34 +3. Enable OpenTelemetry auto-instrumentation; verify trace context propagates across one full request path.
35 +4. Define 2–4 symptom alerts tied to SLOs; wire dashboards with golden signals.
36 +5. Validate: make one request and confirm the same `request_id` appears in every service's logs and on the trace; grep a log sample for denylisted keys (`grep -iE "password|token|authorization"` must return nothing).
37 +
38 +## Edge cases & failure modes
39 +- **High-cardinality label explosion** (user_id, URL-with-ID as metric labels) → metrics store meltdown; keep IDs in logs/traces, out of metric labels.
40 +- **Log volume spikes** (tight retry loop logging per attempt) → log the first failure and the final outcome, count the rest in a metric.
41 +- **Sampling** → 100% traces is fine at low traffic; above ~100 rps, head-sample (e.g., 10%) but always keep error traces.
42 +- **Clock skew across services** → rely on trace spans for ordering, not log timestamps.
43 +
44 +## References
45 +Deeper recipes and gotchas: see [references/patterns.md](references/patterns.md)
added backend-skills/instrumenting-observability/references/patterns.md +122 −0
@@ -0,0 +1,122 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Instrumenting Observability
7 +
8 +## Contents
9 +- Structured logging setup (Python)
10 +- Request-ID middleware and propagation
11 +- RED metrics (Prometheus)
12 +- OpenTelemetry tracing setup
13 +- Symptom alerts (SLO burn)
14 +- Gotchas
15 +
16 +## Structured logging setup (Python)
17 +
18 +```python
19 +import structlog
20 +
21 +structlog.configure(
22 + processors=[
23 + structlog.contextvars.merge_contextvars, # injects request_id
24 + structlog.processors.add_log_level,
25 + structlog.processors.TimeStamper(fmt="iso", key="ts"),
26 + structlog.processors.JSONRenderer(),
27 + ],
28 +)
29 +logger = structlog.get_logger(service="orders")
30 +logger.info("order_created", order_id=123, total_cents=4999)
31 +```
32 +
33 +Scrub secrets before rendering:
34 +
35 +```python
36 +DENYLIST = {"password", "token", "authorization", "card_number", "cvv"}
37 +def scrub(logger, method, event_dict):
38 + for k in list(event_dict):
39 + if k.lower() in DENYLIST:
40 + event_dict[k] = "[redacted]"
41 + return event_dict
42 +# add `scrub` before JSONRenderer in processors
43 +```
44 +
45 +## Request-ID middleware and propagation
46 +
47 +```python
48 +import uuid, structlog
49 +
50 +@app.middleware("http")
51 +async def request_id(request, call_next):
52 + rid = request.headers.get("x-request-id") or f"r-{uuid.uuid4().hex[:12]}"
53 + structlog.contextvars.bind_contextvars(request_id=rid)
54 + response = await call_next(request)
55 + response.headers["x-request-id"] = rid # echo for the caller
56 + return response
57 +
58 +# outbound: always forward
59 +httpx.get(url, headers={"x-request-id": rid})
60 +```
61 +
62 +## RED metrics (Prometheus)
63 +
64 +```python
65 +from prometheus_client import Counter, Histogram
66 +
67 +REQS = Counter("http_requests_total", "requests", ["route", "method", "status"])
68 +# Buckets bracket the SLO (e.g. 300ms target): resolution where it matters.
69 +LAT = Histogram("http_request_seconds", "latency", ["route"],
70 + buckets=[.025, .05, .1, .2, .3, .5, 1, 2, 5])
71 +
72 +@app.middleware("http")
73 +async def metrics(request, call_next):
74 + with LAT.labels(request.url.path).time():
75 + resp = await call_next(request)
76 + REQS.labels(request.url.path, request.method, resp.status_code).inc()
77 + return resp
78 +```
79 +
80 +Label values must be low-cardinality: route *templates* (`/orders/{id}`), never raw paths.
81 +
82 +## OpenTelemetry tracing setup
83 +
84 +```python
85 +from opentelemetry import trace
86 +from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
87 +from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
88 +
89 +FastAPIInstrumentor.instrument_app(app) # inbound spans + context extraction
90 +HTTPXClientInstrumentor().instrument() # outbound propagation
91 +
92 +tracer = trace.get_tracer("orders")
93 +with tracer.start_as_current_span("price_cart") as span: # manual span: meaningful unit only
94 + span.set_attribute("cart.items", len(items))
95 + total = price(items)
96 +```
97 +
98 +Export via OTLP to the collector; configure endpoint with `OTEL_EXPORTER_OTLP_ENDPOINT`.
99 +
100 +## Symptom alerts (SLO burn)
101 +
102 +```yaml
103 +# Page when error budget burns 14.4x too fast over 1h AND 5m (multiwindow).
104 +- alert: HighErrorBurn
105 + expr: >
106 + (sum(rate(http_requests_total{status=~"5.."}[5m]))
107 + / sum(rate(http_requests_total[5m]))) > 14.4 * 0.001
108 + and
109 + (sum(rate(http_requests_total{status=~"5.."}[1h]))
110 + / sum(rate(http_requests_total[1h]))) > 14.4 * 0.001
111 + labels: {severity: page}
112 +```
113 +
114 +p99 latency against SLO: `histogram_quantile(0.99, sum(rate(http_request_seconds_bucket[5m])) by (le))`.
115 +
116 +## Gotchas
117 +- Averages hide pain: a 50 ms mean can coexist with a 5 s p99 — always use histograms/percentiles.
118 +- `user_id` as a metric label = cardinality bomb; it belongs in log fields and span attributes.
119 +- Logging inside a hot retry loop can 100x volume — first failure + final outcome, counter for the middle.
120 +- Forgetting to *echo* X-Request-ID back means clients can't report the ID for support.
121 +- OTel context does not cross thread/process pools automatically — use the context-propagation helpers.
122 +- DEBUG left on in prod both leaks payloads and doubles log cost; enforce level via env config.
added backend-skills/limiting-request-rates/SKILL.md +45 −0
@@ -0,0 +1,45 @@
1 +---
2 +name: limiting-request-rates
3 +description: Designs rate limiting and backpressure for APIs and services — token-bucket limits, limit keys, 429 responses with Retry-After, tiered and weighted limits, distributed enforcement, and load shedding. Use when the user asks to add rate limiting or throttling, protect an API from abuse or overload, return proper 429 responses, set per-user or per-API-key quotas, or handle traffic spikes with backpressure. Do not use for authorization and permissions (implementing-authorization) or capacity planning and autoscaling (scaling-backend-services).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Limiting Request Rates
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** request-rate limits, quotas, throttling, and load shedding on APIs and internal services.
15 +- **Do NOT use for:** who-may-do-what decisions (→ implementing-authorization), fleet sizing (→ scaling-backend-services), or client-side retry logic in isolation.
16 +
17 +## Core rules
18 +
19 +1. **Token bucket is the default algorithm:** refill rate = sustained limit, bucket size = allowed burst (start: burst 2× the per-second rate). Fixed windows create boundary spikes; sliding-log costs memory — use them only with a measured reason.
20 +2. **Choose the limit key deliberately.** Default: per API key/user ID (identity-based). Per-IP only as an anti-abuse fallback for unauthenticated routes — corporate NATs and CGNAT put thousands of users behind one IP.
21 + - ✅ `ratelimit:{api_key}:{endpoint_class}`
22 + - ❌ one global per-IP limit in front of a login page used by offices
23 +3. **A rejected request gets `429` + headers:** `Retry-After` (seconds), `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` — on EVERY response, not just rejections, so clients can pace themselves before hitting the wall.
24 +4. **Weight endpoints by cost.** One search = many token units, one health check = fewer. A single uniform limit either strangles cheap calls or lets expensive ones melt the backend (start: expensive endpoints 10× the base cost).
25 +5. **Distributed enforcement uses a central store (Redis) with a local fallback:** on store failure, each node enforces `limit / node_count` locally rather than dropping enforcement entirely.
26 +6. **Fail-open by default, fail-closed on sensitive routes.** If the limiter breaks, serve traffic (availability first) — EXCEPT login, signup, password reset, and payment endpoints, which fail-closed because unlimited tries there is the actual attack.
27 +7. **Load-shed before collapse:** when saturated (queue depth/latency past threshold), reject early with `503` + `Retry-After` at the edge; a fast no is kinder than a timeout after 30 s of held resources.
28 +8. **Document the limits** where API consumers read (limits, windows, headers, expected backoff). An undocumented limit is indistinguishable from an outage to the client.
29 +
30 +## Workflow
31 +
32 +1. Inventory endpoints; group into cost classes and mark sensitive routes (fail-closed set).
33 +2. Set sustained rate + burst per class per key type; pick limit keys (rule 2).
34 +3. Implement token bucket in Redis (atomic Lua/`INCR`+`EXPIRE` pattern) with the local fallback; emit the rule-3 headers everywhere.
35 +4. Add metrics: rejections per key/class, top offenders, limiter latency.
36 +5. **Validate:** hammer one key past its limit and confirm `429` + correct `Retry-After` while a second key sails through; kill Redis and confirm the fallback behavior matches rule 6 per route.
37 +
38 +## Edge cases & failure modes
39 +- **Legitimate burst (batch import, retry storm after your own outage)** → allow temporary overrides per key rather than raising the global limit.
40 +- **Clock skew across nodes** breaks window math → keep all timing in the central store, not node clocks.
41 +- **429 storms from misbehaving clients that don't back off** → escalate: exponential penalty windows per repeat offender.
42 +- **Internal service-to-service calls** need limits too (a runaway internal loop looks exactly like an attack) — but budget them separately from customer quotas.
43 +
44 +## References
45 +Redis token-bucket implementation and header helpers: see [references/patterns.md](references/patterns.md).
added backend-skills/limiting-request-rates/references/patterns.md +124 −0
@@ -0,0 +1,124 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Rate Limiting
7 +
8 +## Contents
9 +- Token bucket in Redis (atomic)
10 +- Response headers
11 +- Cost-weighted limits
12 +- Local fallback
13 +- Penalty escalation
14 +- Load shedding
15 +- Gotchas
16 +
17 +## Token bucket in Redis (atomic)
18 +
19 +```lua
20 +-- KEYS[1]=bucket key ARGV: rate_per_s, burst, now_ms, cost
21 +-- Returns {allowed(0/1), remaining, retry_after_s}
22 +local rate, burst = tonumber(ARGV[1]), tonumber(ARGV[2])
23 +local now, cost = tonumber(ARGV[3]), tonumber(ARGV[4])
24 +local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
25 +local tokens = tonumber(b[1]) or burst
26 +local ts = tonumber(b[2]) or now
27 +tokens = math.min(burst, tokens + (now - ts) / 1000 * rate)
28 +local allowed = tokens >= cost and 1 or 0
29 +if allowed == 1 then tokens = tokens - cost end
30 +redis.call('HMSET', KEYS[1], 'tokens', tokens, 'ts', now)
31 +redis.call('PEXPIRE', KEYS[1], math.ceil(burst / rate * 2000)) -- self-clean idle keys
32 +local retry = allowed == 1 and 0 or math.ceil((cost - tokens) / rate)
33 +return {allowed, math.floor(tokens), retry}
34 +```
35 +
36 +```python
37 +allowed, remaining, retry = redis.evalsha(
38 + SHA, 1, f"rl:{api_key}:{cls}", RATE, BURST, now_ms(), cost)
39 +```
40 +
41 +One Lua call = read-modify-write with no race; never split GET/SET
42 +across the network.
43 +
44 +## Response headers
45 +
46 +```python
47 +def limit_headers(limit, remaining, reset_epoch):
48 + return {
49 + "X-RateLimit-Limit": str(limit),
50 + "X-RateLimit-Remaining": str(max(0, remaining)),
51 + "X-RateLimit-Reset": str(reset_epoch),
52 + }
53 +
54 +# On rejection add:
55 +# Retry-After: <seconds> and body: {"error": "rate_limited", "retry_after": n}
56 +```
57 +
58 +Send the X-RateLimit-* trio on 200s too — clients pace themselves only if
59 +they can see the meter.
60 +
61 +## Cost-weighted limits
62 +
63 +```python
64 +COSTS = { # units per call; base=1. Weigh by measured backend cost.
65 + "search": 10, # fans out to the search cluster
66 + "export": 25, # long-running, memory heavy
67 + "read": 1,
68 + "health": 0, # never throttle probes
69 +}
70 +cost = COSTS.get(endpoint_class, 1)
71 +```
72 +
73 +## Local fallback
74 +
75 +```python
76 +NODE_SHARE = RATE // max(node_count(), 1) # conservative split when Redis is down
77 +
78 +def check(key, cost):
79 + try:
80 + return redis_bucket(key, cost)
81 + except RedisError:
82 + if key_route_is_sensitive(key): # login/signup/reset/payment
83 + return REJECT # fail-closed
84 + return local_bucket(key, cost, rate=NODE_SHARE) # fail-open, degraded
85 +```
86 +
87 +## Penalty escalation
88 +
89 +```python
90 +# Repeat offenders get exponentially longer cool-downs.
91 +strikes = redis.incr(f"rl:strikes:{api_key}")
92 +redis.expire(f"rl:strikes:{api_key}", 3600)
93 +if strikes > 3:
94 + penalty = min(2 ** (strikes - 3) * 60, 3600) # 1min → 1h cap
95 + redis.setex(f"rl:block:{api_key}", penalty, "1")
96 +```
97 +
98 +## Load shedding
99 +
100 +```python
101 +QUEUE_DEPTH_MAX = 100 # ≈ p99 concurrency × safety factor 2
102 +def middleware(request):
103 + if executor.queue_depth() > QUEUE_DEPTH_MAX:
104 + return Response(503, headers={"Retry-After": "5"})
105 + ...
106 +```
107 +
108 +Shed at the cheapest point in the stack (edge/middleware), before auth and
109 +DB work — the point of shedding is to spend nothing on rejected requests.
110 +
111 +## Gotchas
112 +
113 +- **Fixed windows double-dose at boundaries**: 100/min allows 200 requests
114 + in the 2 s straddling the minute mark; token bucket doesn't.
115 +- **Limiting after authentication** spends a DB call on every rejected
116 + request — put coarse anti-abuse limits before auth, fine per-user limits
117 + after.
118 +- **`Retry-After: 0`** (rounding down) makes clients hammer instantly;
119 + always `ceil`.
120 +- **One bucket for reads and writes** lets a read storm starve writes;
121 + split classes.
122 +- **Missing `PEXPIRE` on buckets** leaks a key per client forever.
123 +- **Health checks and load-balancer probes** must bypass limits or the LB
124 + will mark healthy nodes dead during an attack — exactly when you need them.
added backend-skills/managing-configuration/SKILL.md +47 −0
@@ -0,0 +1,47 @@
1 +---
2 +name: managing-configuration
3 +description: Structures backend service configuration the 12-factor way — environment variables into one typed config object validated at startup, secrets kept out of code and VCS, per-environment overrides without per-environment code. Use when the user asks how to manage config, environment variables, .env files, secrets in a service, add a config setting, or fix per-environment behavior. Do not use for infrastructure provisioning (Terraform/CloudFormation) or CI/CD pipeline configuration.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Managing Configuration
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** application-level configuration: env vars, .env files, secrets handling, typed config objects, feature flags vs config.
15 +- **Do NOT use for:** provisioning infrastructure (Terraform etc.) or CI/CD pipeline YAML — different lifecycles and tools.
16 +
17 +## Core rules
18 +
19 +1. **All config comes from the environment** (12-factor). Code reads env vars; nothing environment-specific is baked into the artifact — the same image runs in dev, staging, and prod.
20 +2. **One typed config object, validated at startup.** Parse env once into a frozen, typed structure; every missing/invalid key crashes the boot with the key named.
21 + - ✅ `Settings()` raises `DATABASE_URL: field required` before serving
22 + - ❌ `os.environ["DATABASE_URL"]` scattered across 30 files, failing at first use in prod.
23 +3. **Secrets are not config.** Passwords, API keys, signing keys come from a secret manager or injected env at deploy time — never committed, never in Docker images, never logged. `.env` is for local dev only and gitignored.
24 +4. **No per-environment code paths.** Behavior differences are driven by config *values*, not environment *names*.
25 + - ✅ `if settings.payments_sandbox:` (set true in staging's env)
26 + - ❌ `if ENV == "staging": use_sandbox()` sprinkled through the codebase.
27 +5. **Defaults are safe for dev, explicit for prod.** Local dev works out of the box (localhost DB, DEBUG level); production values must be provided — a prod boot with dev defaults should fail validation (e.g., `SECRET_KEY` has no default).
28 +6. **`.env.example` is the single, current catalog** of every variable: name, purpose, example value, required/optional. A new variable isn't merged without its line.
29 +7. **Feature flags are not settings.** Flags are runtime-togglable and short-lived (removed after rollout); config is boot-time and long-lived. Don't grow a flag system inside your config object.
30 +
31 +## Workflow
32 +
33 +1. Inventory every `os.environ` / `process.env` access; move each into the central typed config object.
34 +2. Classify each key: plain config vs secret; route secrets to the secret manager / injected env.
35 +3. Set dev defaults where safe; mark prod-critical keys as required (no default).
36 +4. Update `.env.example` with every key; gitignore `.env`; purge any committed secrets (and rotate them — history remembers).
37 +5. Validate: boot with one required var unset → assert the process exits naming that key; run `git log -p | grep -iE "api_key|secret|password"` and a repo scan (`gitleaks detect`) → must be clean.
38 +
39 +## Edge cases & failure modes
40 +- **A secret was committed** → rotating the secret is mandatory; rewriting git history is optional. Treat it as leaked.
41 +- **Config needed before the config system loads** (log level for the config parser itself) → read that one var directly, document the exception.
42 +- **Multi-tenant / per-customer settings** → that's data, not config: store in the database, not env vars.
43 +- **Large config values** (PEM certs) → mount as files and put the *path* in env; multiline env vars break tooling.
44 +- **Different values per process in one deploy** (worker vs web concurrency) → separate variables (`WEB_CONCURRENCY`, `WORKER_CONCURRENCY`), not conditionals on process type.
45 +
46 +## References
47 +Deeper recipes and gotchas: see [references/patterns.md](references/patterns.md)
added backend-skills/managing-configuration/references/patterns.md +107 −0
@@ -0,0 +1,107 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Managing Configuration
7 +
8 +## Contents
9 +- Typed settings with pydantic-settings
10 +- .env.example catalog format
11 +- Secret manager integration
12 +- Config-value-driven behavior (no env-name branches)
13 +- Startup validation test
14 +- Gotchas
15 +
16 +## Typed settings with pydantic-settings
17 +
18 +```python
19 +# pip install pydantic-settings
20 +from pydantic import Field, PostgresDsn, SecretStr
21 +from pydantic_settings import BaseSettings
22 +
23 +class Settings(BaseSettings):
24 + # Required in every environment — no default means boot fails without it.
25 + database_url: PostgresDsn
26 + secret_key: SecretStr # SecretStr never repr()s its value
27 +
28 + # Safe dev defaults; prod overrides via env.
29 + log_level: str = "DEBUG"
30 + payments_sandbox: bool = True
31 + request_timeout_s: float = Field(10.0, gt=0) # upstream p99 ~3s; 10s bounds hangs
32 +
33 + model_config = {"env_file": ".env", "frozen": True}
34 +
35 +settings = Settings() # raises with every missing/invalid key named, at import
36 +```
37 +
38 +Node equivalent: `znv`/`zod` — parse `process.env` through a schema once, export the frozen result.
39 +
40 +## .env.example catalog format
41 +
42 +```bash
43 +# --- Required -----------------------------------------------------------
44 +DATABASE_URL=postgresql://app:app@localhost:5432/app # primary database
45 +SECRET_KEY=change-me # session signing; generate: openssl rand -hex 32
46 +
47 +# --- Optional (defaults shown) ------------------------------------------
48 +LOG_LEVEL=DEBUG # DEBUG|INFO|WARN|ERROR; prod: INFO
49 +PAYMENTS_SANDBOX=true # false only in production
50 +REQUEST_TIMEOUT_S=10
51 +```
52 +
53 +Copy to `.env` for local dev; `.env` stays in `.gitignore`.
54 +
55 +## Secret manager integration
56 +
57 +Injected-env pattern (works with AWS/GCP/Vault/Doppler — the app stays ignorant):
58 +
59 +```bash
60 +# deploy layer resolves secrets into env; app just reads env
61 +aws secretsmanager get-secret-value --secret-id prod/app --query SecretString ...
62 +# or: doppler run -- python -m app / vault agent + envconsul
63 +```
64 +
65 +Direct-fetch escape hatch (only when the platform can't inject):
66 +
67 +```python
68 +def load_secret(name: str) -> str:
69 + import boto3
70 + return boto3.client("secretsmanager").get_secret_value(SecretId=name)["SecretString"]
71 +```
72 +
73 +Rules either way: fetched at boot, held in memory only, never written to disk or logs.
74 +
75 +## Config-value-driven behavior
76 +
77 +```python
78 +# ❌ environment-name branching — untestable matrix, staging drift
79 +if os.environ.get("ENV") == "staging":
80 + client = SandboxPayments()
81 +
82 +# ✅ capability flag — set PAYMENTS_SANDBOX=true wherever sandbox is wanted
83 +client = SandboxPayments() if settings.payments_sandbox else LivePayments()
84 +```
85 +
86 +## Startup validation test
87 +
88 +```python
89 +import subprocess, sys
90 +
91 +def test_boot_fails_without_database_url(monkeypatch):
92 + env = {k: v for k, v in os.environ.items() if k != "DATABASE_URL"}
93 + proc = subprocess.run([sys.executable, "-c", "import app.settings"],
94 + env=env, capture_output=True, text=True)
95 + assert proc.returncode != 0
96 + assert "database_url" in proc.stderr.lower() # the key is NAMED
97 +```
98 +
99 +Secret hygiene scan in CI: `gitleaks detect --no-banner` (fails the build on committed secrets).
100 +
101 +## Gotchas
102 +- Env vars are strings: `DEBUG=False` is truthy as a raw string — always parse through the schema, never `bool(os.environ.get(...))`.
103 +- `.env` loaded in production shadows real env injection and hides misconfiguration; enable env_file only outside prod or ensure real env wins (pydantic-settings: real env takes precedence by default).
104 +- Default-then-override dicts (`config.update(prod_config)`) make the effective value untraceable; one flat schema, one source.
105 +- Printing the settings object at boot is a classic secret leak — use `SecretStr`/redacted repr.
106 +- Rotating a secret must not require a rebuild — if it does, the secret is baked into the image (wrong layer).
107 +- Feature-flag creep: a "flag" older than one quarter is config wearing a costume — promote it or delete it.
added backend-skills/scaling-backend-services/SKILL.md +46 −0
@@ -0,0 +1,46 @@
1 +---
2 +name: scaling-backend-services
3 +description: Scales backend services to handle load — bottleneck measurement first, stateless app tiers, connection pooling, load balancing, autoscaling on the saturating metric, queue-based load leveling, and read replicas. Use when the user asks how to scale a service or API, handle more traffic or concurrent users, fix connection exhaustion, size a connection pool, add a load balancer or autoscaling, or prepare for a traffic spike. Do not use for SQL query tuning (db-skills/optimizing-sql-performance), cache design (caching-strategies), or rate limiting (limiting-request-rates).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Scaling Backend Services
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** capacity planning, horizontal/vertical scaling decisions, pool sizing, load balancing, autoscaling, spike preparation.
15 +- **Do NOT use for:** query-level tuning (db-skills/optimizing-sql-performance), caching design (caching-strategies), rate limiting (limiting-request-rates), or frontend performance.
16 +
17 +## Core rules
18 +
19 +1. **Measure the bottleneck before scaling anything.** Load-test with realistic traffic shape and identify the saturating resource (CPU, memory, DB connections, IO, downstream API). Scaling the wrong tier spends money to move the queue.
20 + - ✅ "p99 collapses at 800 RPS; DB connections pinned at max" → fix pooling.
21 + - ❌ "It's slow, add more pods."
22 +2. **Vertical first when it's cheaper than complexity.** Doubling instance size is a config change; sharding is a project. Buy headroom while you build the durable fix.
23 +3. **Stateless app tier is the precondition for horizontal scaling.** Sessions, uploads, and locks live in external stores (Redis/object storage/DB) so any instance can serve any request and instances can die freely.
24 +4. **Pool every connection.** App→DB and app→downstream. DB pool starting point: `cores × 2` for CPU-bound work; more only for IO-wait-heavy loads. Many app replicas × pool size must stay under the DB's max — put a server-side pooler (pgbouncer) in front of Postgres once replicas multiply.
25 +5. **Load balance with health checks and connection draining.** Instances that fail readiness stop receiving traffic; deploys drain in-flight requests before termination — zero-downtime is a MUST-pass test, not a hope.
26 +6. **Autoscale on the saturating metric, not CPU by reflex.** Queue depth, p95 latency, or connections — whatever rule 1 found. Scale up fast, down slowly (thrash guard), and cap max instances below what the database can survive (rule 4 arithmetic).
27 +7. **Level spiky writes through a queue.** Accept fast (202 + job id), process at a sustainable rate, make consumers idempotent. The queue absorbs the spike; the worker pool sets the drain rate.
28 +8. **Read replicas for read-heavy loads — with lag eyes open.** Route reads that tolerate staleness to replicas; read-your-own-writes flows stay on the primary.
29 +
30 +## Workflow
31 +
32 +1. Load-test to current breaking point; record the saturating metric and p95/p99 at each load step.
33 +2. Externalize any instance state found (rule 3).
34 +3. Size pools by rule 4 arithmetic across ALL replicas; add pgbouncer if the sum approaches DB max_connections.
35 +4. Configure LB health checks + draining; set autoscaling on the measured metric with up-fast/down-slow policies and a hard max.
36 +5. Queue spiky write paths (rule 7).
37 +6. Validate: re-run the load test — the previous breaking point passes; kill one instance at full load and confirm zero failed requests; verify autoscaler adds and (slowly) removes instances.
38 +
39 +## Edge cases & failure modes
40 +- **Thundering herd after downtime** → LB slow-start / gradual traffic ramp for recovering instances; jittered client retries.
41 +- **Autoscaler scales app until the DB dies** → rule 6 cap; the database's ceiling is the fleet's ceiling.
42 +- **One hot key/tenant saturates a single shard/instance** → consistent hashing with hot-key detection; isolate the noisy tenant.
43 +- **Long-lived connections (websockets) defeat draining** → set max connection age; force periodic reconnect so deploys can complete.
44 +
45 +## References
46 +Pool math, autoscaling configs, load-test sketches: see [references/patterns.md](references/patterns.md).
added backend-skills/scaling-backend-services/references/patterns.md +139 −0
@@ -0,0 +1,139 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Scaling Backend Services
7 +
8 +## Contents
9 +- Connection-pool arithmetic
10 +- pgbouncer minimal config
11 +- Kubernetes HPA on a custom metric
12 +- Load balancer health + draining (nginx)
13 +- Queue-based load leveling sketch
14 +- Load-test skeleton (k6)
15 +- Gotchas
16 +
17 +## Connection-pool arithmetic
18 +
19 +```
20 +per-instance pool = cores × 2 # CPU-bound starting point
21 +fleet demand = replicas × per-instance pool
22 +must satisfy : fleet demand < db max_connections − superuser_reserved
23 +
24 +Example: 12 replicas × 10 pool = 120 > Postgres default 100 → pgbouncer required.
25 +```
26 +
27 +IO-wait-heavy workloads (slow downstreams inside transactions) justify larger
28 +pools — but first shorten the transaction, don't widen the pool.
29 +
30 +## pgbouncer minimal config
31 +
32 +```ini
33 +[databases]
34 +app = host=10.0.0.5 port=5432 dbname=app
35 +
36 +[pgbouncer]
37 +listen_port = 6432
38 +auth_type = scram-sha-256
39 +pool_mode = transaction ; multiplexes: many clients, few server conns
40 +default_pool_size = 20 ; server-side connections per db/user pair
41 +max_client_conn = 2000
42 +server_idle_timeout = 60
43 +```
44 +
45 +`pool_mode = transaction` breaks session state (prepared statements, advisory
46 +locks, `SET`) — verify the driver's compatibility mode before enabling.
47 +
48 +## Kubernetes HPA on a custom metric
49 +
50 +```yaml
51 +apiVersion: autoscaling/v2
52 +kind: HorizontalPodAutoscaler
53 +spec:
54 + scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
55 + minReplicas: 3
56 + maxReplicas: 24 # capped by DB pool arithmetic, not by budget alone
57 + metrics:
58 + - type: Pods
59 + pods:
60 + metric: { name: http_p95_latency_ms }
61 + target: { type: AverageValue, averageValue: "250" }
62 + behavior:
63 + scaleUp:
64 + stabilizationWindowSeconds: 0 # react to spikes immediately
65 + policies: [{ type: Percent, value: 100, periodSeconds: 60 }]
66 + scaleDown:
67 + stabilizationWindowSeconds: 300 # shrink slowly — thrash guard
68 + policies: [{ type: Pods, value: 1, periodSeconds: 120 }]
69 +```
70 +
71 +## Load balancer health + draining (nginx)
72 +
73 +```nginx
74 +upstream api {
75 + least_conn;
76 + server 10.0.1.10:8000 max_fails=3 fail_timeout=10s;
77 + server 10.0.1.11:8000 max_fails=3 fail_timeout=10s;
78 + server 10.0.1.12:8000 slow_start=30s; # ramp recovered instances gradually
79 +}
80 +```
81 +
82 +App side (draining): on SIGTERM, fail the readiness endpoint, keep serving
83 +in-flight requests, exit after they finish or a 30 s deadline:
84 +
85 +```python
86 +def handle_sigterm(*_):
87 + ready.set_unhealthy() # LB stops sending new traffic
88 + server.shutdown(grace=30) # finish in-flight, then exit
89 +```
90 +
91 +## Queue-based load leveling sketch
92 +
93 +```python
94 +# ingest: accept fast, defer work
95 +@app.post("/imports")
96 +def create_import(req):
97 + job_id = queue.enqueue("imports", req.body, idempotency_key=req.headers["Idempotency-Key"])
98 + return 202, {"job_id": job_id, "status_url": f"/imports/{job_id}"}
99 +
100 +# worker pool drains at a sustainable rate; consumers are idempotent
101 +def worker():
102 + for job in queue.consume("imports", prefetch=1):
103 + if already_processed(job.idempotency_key): # replay-safe
104 + job.ack(); continue
105 + process(job)
106 + mark_processed(job.idempotency_key)
107 + job.ack()
108 +```
109 +
110 +Worker count — not producer rate — sets DB write pressure; scale workers only
111 +while the DB stays under its ceiling.
112 +
113 +## Load-test skeleton (k6)
114 +
115 +```javascript
116 +import http from 'k6/http';
117 +export const options = {
118 + stages: [
119 + { duration: '2m', target: 200 }, // ramp
120 + { duration: '5m', target: 200 }, // steady — realistic think time below
121 + { duration: '2m', target: 800 }, // find the knee
122 + ],
123 + thresholds: { http_req_duration: ['p(95)<300'] },
124 +};
125 +export default function () {
126 + http.get('https://staging.example.com/api/orders');
127 +}
128 +```
129 +
130 +Test against staging with production-shaped data; an empty database lies.
131 +
132 +## Gotchas
133 +
134 +- **Sticky sessions are hidden state** — they break draining and uneven-load the fleet; externalize the session instead.
135 +- **Autoscaling on CPU while blocked on IO** does nothing: instances idle at 20% CPU while every request waits on the pool. Scale on the metric that saturates.
136 +- **`least_conn` beats round-robin** once request costs vary, but health checks matter more than the algorithm.
137 +- **Replica lag is not constant** — it spikes exactly when you're overloaded, i.e., when you rerouted reads there. Monitor lag and fail reads back gracefully.
138 +- **Local caches per instance multiply cold starts** during scale-up; a scale-out event can stampede the DB (see caching-strategies for stampede locks).
139 +- **Max-instance caps get deleted in incidents** — document WHY the cap exists (DB arithmetic) next to the setting.
added backend-skills/securing-backend-services/SKILL.md +46 −0
@@ -0,0 +1,46 @@
1 +---
2 +name: securing-backend-services
3 +description: Hardens backend services against common attacks — security headers, TLS, secret management, dependency scanning, SSRF and CSRF defenses, and safe error responses. Use when the user asks to secure a service or API, add security headers or CSP, manage secrets, run a security review of a backend, fix an SSRF/CSRF finding, or prepare for a pentest. Do not use for login flows (implementing-authentication), permission models (implementing-authorization), request schemas (validating-input), or database security (securing-databases).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Securing Backend Services
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** OWASP-aligned service hardening — headers, TLS, secrets, dependencies, SSRF/CSRF, error hygiene, service-account privilege.
15 +- **Do NOT use for:** authentication flows, authorization models, input schemas, or database roles/encryption — each has its own skill. This skill is the layer around them.
16 +
17 +## Core rules
18 +
19 +1. **Security headers on every response:**
20 + `Strict-Transport-Security: max-age=31536000; includeSubDomains`, `X-Content-Type-Options: nosniff`, `Content-Security-Policy` (start `default-src 'self'`), `frame-ancestors 'none'` (or the one legit embedder), `Referrer-Policy: strict-origin-when-cross-origin`.
21 +2. **TLS everywhere, including service-to-service.** Redirect HTTP→HTTPS at the edge; internal traffic gets mTLS or a private mesh — "internal" networks are not trusted.
22 +3. **Secrets never live in code or VCS.** Environment variables or a secret manager; distinct secrets per environment; rotate immediately on any exposure and treat the git history as public once pushed.
23 +4. **Dependencies: lockfile + scanner + cadence.** Commit the lockfile, run `pip-audit`/`npm audit`/Dependabot in CI, patch criticals within days not quarters.
24 +5. **SSRF: user-supplied URLs are hostile.**
25 + - ✅ Allowlist destination hosts; resolve DNS and reject private/link-local ranges (10.x, 172.16–31, 192.168, 169.254, ::1); disable redirects or re-check after each hop
26 + - ❌ `requests.get(user_url)` — hello, cloud metadata endpoint
27 +6. **CSRF tokens for every cookie-authenticated state change.** SameSite helps but is not sufficient (top-level POST exemptions, old clients); use the framework's CSRF middleware.
28 +7. **Errors never leak internals.** Map exceptions to problem+json with a correlation ID; stack traces, SQL, and versions go to logs only.
29 +8. **Service accounts get least privilege** — scoped API keys, no wildcard IAM, one identity per service so revocation is surgical.
30 +
31 +## Workflow
32 +
33 +1. Inventory: endpoints, secrets, outbound URL fetches, cookie-authenticated mutations, third-party dependencies.
34 +2. Apply rules 1–2 at the middleware/edge layer (one place, not per route).
35 +3. Move any in-code secrets out (rule 3) and rotate them — moving without rotating fixes nothing.
36 +4. Add SSRF/CSRF defenses where the inventory found exposure (rules 5–6); wire the error mapper (rule 7).
37 +5. **Validate:** `curl -sI` each surface and confirm the rule-1 headers; run the dependency scanner to zero criticals; request a known-bad internal URL through any fetch feature (expect rejection); trigger an exception and confirm the response body contains no stack trace. All four must pass.
38 +
39 +## Edge cases & failure modes
40 +- **CSP breaks inline scripts** → prefer nonces (`'nonce-...'`) over loosening to `unsafe-inline`.
41 +- **Header set twice (app + proxy)** → duplicated CSP is intersected by browsers; set each header in exactly one layer.
42 +- **Webhooks/health checks behind mTLS** → give external callers a dedicated ingress with its own auth, don't weaken the default.
43 +- **Secret manager outage** → cache secrets in memory with TTL; never fall back to a baked-in default secret.
44 +
45 +## References
46 +Copy-paste patterns and gotchas: see [references/patterns.md](references/patterns.md).
added backend-skills/securing-backend-services/references/patterns.md +118 −0
@@ -0,0 +1,118 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Securing Backend Services
7 +
8 +## Contents
9 +- Security-headers middleware
10 +- SSRF-safe URL fetcher
11 +- CSRF setup
12 +- Problem+json error mapper
13 +- Secrets loading
14 +- Dependency scanning in CI
15 +- Gotchas
16 +
17 +## Security-headers middleware
18 +
19 +```python
20 +SECURITY_HEADERS = {
21 + "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
22 + "X-Content-Type-Options": "nosniff",
23 + "Content-Security-Policy": "default-src 'self'; frame-ancestors 'none'",
24 + "Referrer-Policy": "strict-origin-when-cross-origin",
25 + "Cache-Control": "no-store", # for API responses carrying user data
26 +}
27 +
28 +@app.middleware("http")
29 +async def add_security_headers(request, call_next):
30 + resp = await call_next(request)
31 + for k, v in SECURITY_HEADERS.items():
32 + resp.headers.setdefault(k, v) # setdefault: edge/proxy may own some
33 + return resp
34 +```
35 +
36 +## SSRF-safe URL fetcher
37 +
38 +```python
39 +import ipaddress, socket
40 +from urllib.parse import urlparse
41 +
42 +ALLOWED_HOSTS = {"api.partner.example"} # allowlist beats any blocklist
43 +
44 +def assert_public(host: str):
45 + for info in socket.getaddrinfo(host, None):
46 + ip = ipaddress.ip_address(info[4][0])
47 + if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_reserved:
48 + raise ValueError(f"blocked address {ip}")
49 +
50 +def safe_fetch(url: str):
51 + p = urlparse(url)
52 + if p.scheme != "https" or p.hostname not in ALLOWED_HOSTS:
53 + raise ValueError("destination not allowed")
54 + assert_public(p.hostname) # resolve NOW, at request time
55 + return requests.get(url, timeout=10, allow_redirects=False) # re-check per hop if following
56 +```
57 +
58 +## CSRF setup
59 +
60 +```python
61 +# Use the framework middleware — do not hand-roll token comparison.
62 +# Django: CsrfViewMiddleware (default). Flask: flask-wtf CSRFProtect(app).
63 +# FastAPI cookie-auth: double-submit cookie via starlette-csrf.
64 +# Token in a custom header (X-CSRF-Token) read from a non-HttpOnly cookie;
65 +# SameSite=Lax remains as defense-in-depth, not the defense.
66 +```
67 +
68 +## Problem+json error mapper
69 +
70 +```python
71 +@app.exception_handler(Exception)
72 +async def unhandled(request, exc):
73 + cid = request.state.correlation_id
74 + logger.exception("unhandled error cid=%s", cid) # full detail → logs
75 + return JSONResponse(status_code=500, content={ # zero detail → client
76 + "type": "about:blank", "title": "Internal error",
77 + "status": 500, "correlation_id": cid,
78 + })
79 +```
80 +
81 +## Secrets loading
82 +
83 +```python
84 +import os
85 +
86 +class Settings:
87 + def __init__(self):
88 + self.db_url = self._req("DATABASE_URL")
89 + self.signing_key = self._req("SIGNING_KEY")
90 +
91 + @staticmethod
92 + def _req(name: str) -> str:
93 + val = os.environ.get(name)
94 + if not val:
95 + raise RuntimeError(f"missing required secret {name}") # fail fast, no defaults
96 + return val
97 +```
98 +
99 +Pre-commit guard: `gitleaks protect --staged` blocks accidental secret commits.
100 +
101 +## Dependency scanning in CI
102 +
103 +```yaml
104 +# GitHub Actions
105 +- run: pip install pip-audit && pip-audit --strict # Python
106 +- run: npm audit --audit-level=high # Node
107 +# Fail the build on criticals; schedule a weekly run so quiet repos still alert.
108 +```
109 +
110 +## Gotchas
111 +
112 +- **HSTS on a domain still serving plain HTTP paths** locks users out for max-age; deploy HTTPS fully first, then add the header, then preload.
113 +- **CSP `report-only` left on forever** — attackers are unaffected by reports; graduate to enforcing after a week of clean reports.
114 +- **SSRF via DNS rebinding** — validate the resolved IP at request time (as above), not in a separate pre-check the attacker can race.
115 +- **`allow_redirects=True` after an allowlist check** — the first hop is allowed, the redirect goes to the metadata IP; disable or re-validate per hop.
116 +- **Rotating a secret without invalidating derived artifacts** — old JWTs signed with the leaked key stay valid; rotate key AND revoke issued tokens.
117 +- **One shared "backend" IAM role** — a compromise anywhere is a compromise everywhere; one identity per service.
118 +- **Error middleware ordered after routers** — exceptions in earlier middleware bypass the mapper; register it outermost.
added backend-skills/shipping-with-ci-cd/SKILL.md +46 −0
@@ -0,0 +1,46 @@
1 +---
2 +name: shipping-with-ci-cd
3 +description: Designs CI/CD pipelines that ship safely — fail-fast stages, build-once artifact promotion, rolling/blue-green/canary deploy strategies, automated rollback, and backward-compatible migrations. Use when the user asks to set up or fix a CI/CD pipeline, GitHub Actions or GitLab CI workflow, deployment process, release automation, rollback strategy, or asks how to deploy to staging and production. Do not use for writing the tests themselves (testing-backend-services) or authoring Dockerfiles (containerizing-services).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Shipping with CI/CD
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** pipeline design/review, deploy strategy choice, environment promotion, release/rollback automation.
15 +- **Do NOT use for:** test authoring (testing-backend-services), Dockerfile content (containerizing-services), or infra provisioning.
16 +
17 +## Core rules
18 +
19 +1. **Main is always deployable.** Trunk-based development by default: short-lived branches, merge behind a green pipeline. A red main is the team's top priority.
20 +2. **Fail fast, in cost order.** Stage sequence: lint/typecheck → unit tests → build → integration tests → security scan → deploy. Cheap checks first so failures cost seconds, not minutes.
21 +3. **Build once, promote the artifact.** The exact image/binary tested in staging is byte-identical in production — tag by commit SHA.
22 + - ✅ `svc:3f2a91c` promoted staging → prod
23 + - ❌ Rebuilding "the same" code per environment (different deps, different artifact).
24 +4. **Pick the deploy strategy by risk, not fashion.** Rolling is the default; blue-green when you need instant rollback; canary (1–5% traffic, then ramp) for high-risk changes. Anything beyond rolling must justify its infra cost.
25 +5. **Rollback is automated, not heroic.** Health checks/SLO probes gate each deploy step; regression triggers automatic rollback to the previous artifact. If rollback requires a human running commands from memory, it isn't a rollback plan.
26 +6. **Migrations deploy before code and stay one version backward-compatible.** Old code must run against the new schema (expand → deploy → contract). Never couple a destructive migration to the deploy that needs it.
27 +7. **Secrets come from the platform** (environment/secret manager, OIDC cloud auth) — never committed in pipeline YAML, never echoed in logs.
28 +8. **Keep the pipeline under ~10 minutes** commit-to-verdict. Parallelize test shards, cache dependencies; a slow pipeline silently kills trunk-based flow.
29 +
30 +## Workflow
31 +
32 +1. Map stages in cost order (rule 2); wire caching for dependency steps.
33 +2. Emit one artifact tagged with the commit SHA; push to the registry once.
34 +3. Define environments (staging auto-deploys on main; production gated by promotion of the same artifact).
35 +4. Add deploy gates: health-check verification after each batch, automatic rollback on failure (rule 5).
36 +5. Wire migrations as a separate pre-deploy step with a rehearsed rollback (rule 6).
37 +6. Validate: run a deliberately failing commit (lint error) and confirm the pipeline stops at stage 1; deploy a canary/rolling change and kill an instance mid-deploy — confirm zero failed requests and automatic recovery.
38 +
39 +## Edge cases & failure modes
40 +- **Flaky tests** → quarantine tagged-flaky tests to a non-blocking stage the same day; a pipeline people retry until green is a pipeline nobody trusts.
41 +- **Hotfix while main is red** → fix forward on main; the deployable-main rule makes dedicated hotfix branches unnecessary.
42 +- **Long-running migration locks a hot table** → split into batched backfills; see db-skills/managing-database-migrations.
43 +- **Rollback needed but migration was destructive** → this is a rule-6 violation; restore path is a backup drill, so keep contract phases a release behind.
44 +
45 +## References
46 +Pipeline templates and gotchas: see [references/patterns.md](references/patterns.md).
added backend-skills/shipping-with-ci-cd/references/patterns.md +134 −0
@@ -0,0 +1,134 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Shipping with CI/CD
7 +
8 +## Contents
9 +- GitHub Actions: fail-fast pipeline with artifact promotion
10 +- Environment promotion job
11 +- Canary deploy sketch
12 +- Automated rollback gate
13 +- Migration-before-code ordering
14 +- Gotchas
15 +
16 +## GitHub Actions: fail-fast pipeline with artifact promotion
17 +
18 +```yaml
19 +name: ci
20 +on:
21 + push:
22 + branches: [main]
23 + pull_request:
24 +
25 +concurrency:
26 + group: ${{ github.workflow }}-${{ github.ref }}
27 + cancel-in-progress: true # stale runs waste the <10 min budget
28 +
29 +jobs:
30 + lint:
31 + runs-on: ubuntu-24.04
32 + steps:
33 + - uses: actions/checkout@v4
34 + - uses: actions/setup-node@v4
35 + with: { node-version: 22, cache: npm }
36 + - run: npm ci
37 + - run: npm run lint && npm run typecheck
38 +
39 + test:
40 + needs: lint # fail-fast ordering
41 + runs-on: ubuntu-24.04
42 + strategy:
43 + matrix: { shard: [1, 2, 3, 4] } # parallel shards keep wall-clock low
44 + steps:
45 + - uses: actions/checkout@v4
46 + - uses: actions/setup-node@v4
47 + with: { node-version: 22, cache: npm }
48 + - run: npm ci
49 + - run: npm test -- --shard=${{ matrix.shard }}/4
50 +
51 + build:
52 + needs: test
53 + if: github.ref == 'refs/heads/main'
54 + runs-on: ubuntu-24.04
55 + permissions: { id-token: write, contents: read } # OIDC, no long-lived keys
56 + steps:
57 + - uses: actions/checkout@v4
58 + - run: docker build -t registry.example.com/svc:${{ github.sha }} .
59 + - run: docker push registry.example.com/svc:${{ github.sha }}
60 +```
61 +
62 +## Environment promotion job
63 +
64 +```yaml
65 + deploy-staging:
66 + needs: build
67 + environment: staging
68 + runs-on: ubuntu-24.04
69 + steps:
70 + - run: ./deploy.sh registry.example.com/svc:${{ github.sha }} staging
71 +
72 + deploy-prod:
73 + needs: deploy-staging
74 + environment: production # requires reviewer approval in repo settings
75 + runs-on: ubuntu-24.04
76 + steps:
77 + # SAME artifact — promotion, not rebuild
78 + - run: ./deploy.sh registry.example.com/svc:${{ github.sha }} production
79 +```
80 +
81 +## Canary deploy sketch
82 +
83 +```bash
84 +# deploy.sh <image> production — canary ramp with health gates
85 +set -euo pipefail
86 +IMAGE=$1
87 +for PCT in 5 25 100; do
88 + set_traffic_split "$IMAGE" "$PCT"
89 + sleep 120 # observation window per step
90 + ERR=$(error_rate_last_2m)
91 + if (( $(echo "$ERR > 0.01" | bc -l) )); then # >1% errors aborts the ramp
92 + set_traffic_split "$PREVIOUS_IMAGE" 100
93 + echo "canary failed at ${PCT}% (err=${ERR}), rolled back" >&2
94 + exit 1
95 + fi
96 +done
97 +```
98 +
99 +## Automated rollback gate
100 +
101 +```yaml
102 + - name: verify and rollback
103 + run: |
104 + for i in $(seq 1 30); do
105 + if curl -fsS https://svc.example.com/healthz; then exit 0; fi
106 + sleep 10
107 + done
108 + ./deploy.sh "$PREVIOUS_SHA" production # 5 min without health = revert
109 + exit 1
110 +```
111 +
112 +Record `PREVIOUS_SHA` before deploying — rollback needs a target, not a rebuild.
113 +
114 +## Migration-before-code ordering
115 +
116 +```yaml
117 + migrate:
118 + needs: build
119 + runs-on: ubuntu-24.04
120 + steps:
121 + - run: ./run-migrations.sh # expand-phase only; contract ships a release later
122 +
123 + deploy-staging:
124 + needs: migrate # code never deploys onto an unmigrated schema
125 +```
126 +
127 +## Gotchas
128 +
129 +- **`cancel-in-progress` on main deploys** can abort a half-finished rollout — scope the concurrency group to PRs, or use a deploy queue for main.
130 +- **Cache poisoning:** keyed only on lockfile hash, a cache never picks up new OS packages; include the base-image tag in the key when builds depend on it.
131 +- **`environment:` protection is the gate; branch protection is not** — a promoted artifact needs its own approval step.
132 +- **OIDC beats stored cloud keys**: `permissions: id-token: write` + cloud trust policy removes the leakable secret entirely.
133 +- **Rollback ≠ redeploy old branch**: it is re-pointing to the previous *artifact*; rebuilding old code can produce a different binary than what ran yesterday.
134 +- **Migrations in the same job as deploy** hide ordering failures — separate jobs make "schema first" visible and retryable.
added backend-skills/testing-backend-services/SKILL.md +47 −0
@@ -0,0 +1,47 @@
1 +---
2 +name: testing-backend-services
3 +description: Designs and writes tests for backend services — unit, integration with real dependencies in containers, contract tests, and minimal E2E — with deterministic, parallel-safe practices. Use when the user asks to test an API or service, write unit/integration/contract tests, fix flaky backend tests, set up testcontainers, or decide what to mock. Do not use for frontend or UI testing, or for load and performance testing.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Testing Backend Services
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** test strategy and test code for services and APIs: unit, integration, contract, E2E; de-flaking; test data design.
15 +- **Do NOT use for:** browser/UI testing, or load/perf/soak testing — different tooling and goals.
16 +
17 +## Core rules
18 +
19 +1. **Shape tests as a pyramid.** Many milliseconds-fast unit tests on pure logic; a solid layer of integration tests; one happy-path E2E per critical flow. Inverting it (E2E-heavy) buys slow, flaky suites.
20 +2. **Integration tests hit real dependencies in containers.** Spin up Postgres/Redis/broker via testcontainers.
21 + - ✅ test repository code against a real Postgres container
22 + - ❌ mock the database driver and assert SQL strings — that tests your mock.
23 +3. **Mock only what you don't own** (third-party HTTP APIs, clocks, randomness) — and pin those mocks with contract tests where possible.
24 +4. **Test behavior, not implementation.** Assert on outputs, state changes, and emitted events — not on which internal methods were called. Refactors must not break green tests.
25 +5. **No sleeps.** Poll with a timeout for async effects; freeze the clock for time logic; seed randomness. A test that needs `sleep(2)` is a race you scheduled.
26 + - ✅ `wait_until(lambda: outbox.count() == 1, timeout=5)`
27 + - ❌ `time.sleep(2); assert outbox.count() == 1`
28 +6. **Each test is independent and parallel-safe:** owns its data (unique IDs per test), never depends on execution order, cleans up via transaction rollback or per-test schema.
29 +7. **Factories over shared fixtures.** `make_user(email=...)` with overridable defaults beats a giant `fixtures.sql` that every test secretly depends on.
30 +8. **Contract tests guard API boundaries:** provider verifies it still satisfies consumer expectations (Pact or OpenAPI-based) on every CI run — cheaper than E2E across repos.
31 +
32 +## Workflow
33 +
34 +1. Classify the change: pure logic → unit; touches DB/broker/HTTP edge → integration; crosses service boundary → contract; business-critical flow → one E2E.
35 +2. Write the test first at the lowest level that can catch the bug.
36 +3. Build test data with factories; give every entity a per-test unique key.
37 +4. Replace any sleep/order dependency with polling, frozen clocks, seeded RNG.
38 +5. Validate: run the suite twice — full run and `--last-failed` in random order (`pytest -p randomly`); both must pass. Run the new test 20× (`pytest --count=20 -x`) to prove it's not flaky.
39 +
40 +## Edge cases & failure modes
41 +- **Test passes locally, fails in CI** → almost always shared state or timing; check for fixed ports, shared DB rows, real clock usage.
42 +- **Container startup dominates runtime** → reuse one container per session with per-test transactions/schemas, not one container per test.
43 +- **Untestable code** (network calls in constructors, global singletons) → refactor for injection first; don't monkey-patch around design problems.
44 +- **Non-determinism sources** (UUIDs, now(), env) → inject them; asserting on wall-clock values is a flake factory.
45 +
46 +## References
47 +Deeper recipes and gotchas: see [references/patterns.md](references/patterns.md)
added backend-skills/testing-backend-services/references/patterns.md +120 −0
@@ -0,0 +1,120 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Testing Backend Services
7 +
8 +## Contents
9 +- Testcontainers setup (pytest + Postgres)
10 +- Per-test isolation via transaction rollback
11 +- Polling instead of sleeping
12 +- Frozen clock and seeded randomness
13 +- Data factories
14 +- Contract test sketch
15 +- Gotchas
16 +
17 +## Testcontainers setup (pytest + Postgres)
18 +
19 +```python
20 +# pip install testcontainers[postgres] pytest
21 +import pytest
22 +from testcontainers.postgres import PostgresContainer
23 +
24 +@pytest.fixture(scope="session")
25 +def pg_url():
26 + # One container per session: startup (~2-5s) is paid once, not per test.
27 + with PostgresContainer("postgres:17") as pg:
28 + yield pg.get_connection_url()
29 +```
30 +
31 +## Per-test isolation via transaction rollback
32 +
33 +```python
34 +@pytest.fixture
35 +def db(pg_url):
36 + engine = get_engine(pg_url)
37 + conn = engine.connect()
38 + tx = conn.begin()
39 + yield conn # test runs inside the transaction
40 + tx.rollback() # everything the test wrote vanishes
41 + conn.close()
42 +```
43 +
44 +Escape hatch: code under test that commits internally needs per-test schemas
45 +(`CREATE SCHEMA test_{uuid}`) instead of rollback.
46 +
47 +## Polling instead of sleeping
48 +
49 +```python
50 +import time
51 +
52 +def wait_until(predicate, timeout=5.0, interval=0.05):
53 + # 50ms interval: fast feedback without hammering; 5s cap: fail loudly.
54 + deadline = time.monotonic() + timeout
55 + while time.monotonic() < deadline:
56 + if predicate():
57 + return
58 + time.sleep(interval)
59 + raise AssertionError(f"condition not met within {timeout}s")
60 +
61 +publish(event)
62 +wait_until(lambda: repo.count(status="processed") == 1)
63 +```
64 +
65 +## Frozen clock and seeded randomness
66 +
67 +```python
68 +# pip install freezegun
69 +from freezegun import freeze_time
70 +
71 +@freeze_time("2026-08-05T12:00:00Z")
72 +def test_subscription_expires():
73 + sub = make_subscription(days=30)
74 + assert sub.expires_at == datetime(2026, 9, 4, 12, tzinfo=UTC)
75 +
76 +# conftest.py — same failures every run
77 +import random
78 +random.seed(1337)
79 +```
80 +
81 +## Data factories
82 +
83 +```python
84 +import itertools
85 +_seq = itertools.count()
86 +
87 +def make_user(**over):
88 + n = next(_seq)
89 + defaults = dict(email=f"u{n}@test.local", name=f"User {n}", plan="free")
90 + return User(**{**defaults, **over})
91 +
92 +def make_order(user=None, **over):
93 + user = user or make_user()
94 + defaults = dict(user_id=user.id, total_cents=1000, status="pending")
95 + return Order(**{**defaults, **over})
96 +```
97 +
98 +Unique-per-call defaults keep parallel tests from colliding on unique constraints.
99 +
100 +## Contract test sketch
101 +
102 +Consumer publishes expectations; provider CI verifies against the real app:
103 +
104 +```python
105 +# consumer side (pact-python): "GET /users/9 returns id+email"
106 +pact.given("user 9 exists").upon_receiving("get user") \
107 + .with_request("GET", "/users/9") \
108 + .will_respond_with(200, body={"id": 9, "email": Like("a@b.c")})
109 +```
110 +
111 +Lighter alternative: validate provider responses in integration tests against
112 +the OpenAPI schema (`schemathesis` or response-validation middleware).
113 +
114 +## Gotchas
115 +- Mock-heavy tests rot: they keep passing while real integration breaks — the DB mock never raises `UniqueViolation`.
116 +- `scope="session"` containers + tests that commit = cross-test contamination; pair session containers with rollback/schema isolation.
117 +- Asserting exact timestamps or auto-increment IDs couples tests to execution order.
118 +- Parallel runners (pytest-xdist) expose hidden shared state — fixed ports, `/tmp` paths, same S3 bucket keys.
119 +- A retried-until-green test is a deleted test with extra steps; fix the race instead of adding `--reruns`.
120 +- E2E through the public API only — reaching into another service's DB in a test welds the deploy order together.
added backend-skills/validating-input/SKILL.md +45 −0
@@ -0,0 +1,45 @@
1 +---
2 +name: validating-input
3 +description: Validates and sanitizes incoming request data at the service boundary — schema validation, type/length/range/format checks, allowlists, unknown-field rejection, and structured 422 error responses. Use when the user asks to validate request bodies, query params, headers, or uploads, add a pydantic/zod/joi schema, prevent malformed or malicious payloads, or fix a path-traversal or oversized-input issue. Do not use for login and permission decisions (implementing-authentication, implementing-authorization) or for business rules deeper than the request boundary.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Validating Input
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** request-boundary validation of bodies, params, headers, and file/path inputs; schema definitions; validation error responses.
15 +- **Do NOT use for:** authn/authz decisions, domain/business invariants (those live in the domain layer), or output encoding (XSS is an output concern — encode at render time).
16 +
17 +## Core rules
18 +
19 +1. **Validate at the boundary with the stack's schema library** — pydantic (Python), zod (TS), joi (Node legacy). Hand-rolled `if` chains drift.
20 +2. **Every field gets type + length + range + format.** Unbounded strings and numbers are bugs.
21 + - ✅ `name: str, min_length=1, max_length=200`
22 + - ❌ `name: str` — a 10 MB name is now valid
23 +3. **Allowlist over blocklist.** Enumerate what is valid (enum, regex anchored `^...$`, closed set); never try to enumerate evil.
24 +4. **Reject unknown fields** (`extra="forbid"` / `.strict()`). Silent extras become mass-assignment holes.
25 +5. **Canonicalize before validating:** trim whitespace, NFC-normalize unicode, lowercase emails — then check. Validating pre-canonical data lets `admin ` ≠ `admin` bypasses through.
26 +6. **Client-side validation is UX only.** The server re-validates everything, always.
27 +7. **Fail with a structured 422** naming every invalid field and why — one pass, not first-error-only.
28 +8. **Path and file inputs:** resolve to an absolute path and require it start with the allowed base directory; validate content-type by sniffing magic bytes, not the filename; cap upload size before reading the body.
29 +
30 +## Workflow
31 +
32 +1. Define one schema per endpoint request (body, query, path params) with rule-2 constraints on every field.
33 +2. Enable unknown-field rejection and canonicalization hooks (rules 4–5).
34 +3. Wire the validation-error handler to the structured 422 format (rule 7).
35 +4. Add path/upload guards where files are involved (rule 8).
36 +5. **Validate the validator:** send a malformed payload per field class (wrong type, over-length, out-of-range, unknown field, path `../../etc/passwd`) and confirm each yields a 422/400 naming the field — never a 500. Fix and repeat until all pass.
37 +
38 +## Edge cases & failure modes
39 +- **Numbers as strings** (`"42"`) → decide once: coerce (default for query params) or reject (default for JSON bodies); stay consistent.
40 +- **Empty vs missing vs null** → distinguish explicitly in the schema; PATCH semantics need "absent = unchanged".
41 +- **Arrays** → cap length (e.g. ≤1000 items) and validate every element, not just the container.
42 +- **Validation library throws on deeply nested payloads** → cap request size and nesting depth at the web-server layer first.
43 +
44 +## References
45 +Copy-paste patterns and gotchas: see [references/patterns.md](references/patterns.md).
added backend-skills/validating-input/references/patterns.md +114 −0
@@ -0,0 +1,114 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Validating Input
7 +
8 +## Contents
9 +- Pydantic request schema (strict, canonicalizing)
10 +- Zod equivalent
11 +- Structured 422 handler
12 +- Path traversal guard
13 +- Upload validation
14 +- Gotchas
15 +
16 +## Pydantic request schema (strict, canonicalizing)
17 +
18 +```python
19 +from pydantic import BaseModel, ConfigDict, EmailStr, Field, field_validator
20 +import unicodedata
21 +
22 +class CreateUser(BaseModel):
23 + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
24 +
25 + name: str = Field(min_length=1, max_length=200)
26 + email: EmailStr
27 + age: int = Field(ge=13, le=130)
28 + role: Literal["member", "editor"] # closed set — allowlist
29 + tags: list[str] = Field(default_factory=list, max_length=50)
30 +
31 + @field_validator("name")
32 + @classmethod
33 + def canonicalize(cls, v: str) -> str:
34 + return unicodedata.normalize("NFC", v) # canonicalize BEFORE checks
35 +
36 + @field_validator("email")
37 + @classmethod
38 + def lower_email(cls, v: str) -> str:
39 + return v.lower()
40 +```
41 +
42 +## Zod equivalent
43 +
44 +```ts
45 +const CreateUser = z.object({
46 + name: z.string().trim().min(1).max(200),
47 + email: z.string().email().toLowerCase(),
48 + age: z.number().int().min(13).max(130),
49 + role: z.enum(["member", "editor"]),
50 + tags: z.array(z.string().max(50)).max(50).default([]),
51 +}).strict(); // reject unknown fields
52 +```
53 +
54 +## Structured 422 handler
55 +
56 +```python
57 +# FastAPI: collect ALL field errors in one response.
58 +@app.exception_handler(RequestValidationError)
59 +async def on_validation_error(request, exc):
60 + return JSONResponse(status_code=422, content={
61 + "type": "https://example.com/errors/validation",
62 + "title": "Request validation failed",
63 + "errors": [
64 + {"field": ".".join(map(str, e["loc"])), "reason": e["msg"]}
65 + for e in exc.errors()
66 + ],
67 + })
68 +```
69 +
70 +```json
71 +{"title": "Request validation failed",
72 + "errors": [{"field": "body.age", "reason": "Input should be less than or equal to 130"}]}
73 +```
74 +
75 +## Path traversal guard
76 +
77 +```python
78 +from pathlib import Path
79 +
80 +BASE = Path("/srv/app/uploads").resolve()
81 +
82 +def safe_path(user_supplied: str) -> Path:
83 + candidate = (BASE / user_supplied).resolve()
84 + if not candidate.is_relative_to(BASE): # blocks ../.. and absolute paths
85 + raise ValueError("path escapes base directory")
86 + return candidate
87 +```
88 +
89 +## Upload validation
90 +
91 +```python
92 +MAX_UPLOAD = 10 * 1024 * 1024 # 10 MB — set per product need, never unlimited
93 +
94 +MAGIC = {b"\x89PNG": "image/png", b"\xff\xd8\xff": "image/jpeg", b"%PDF": "application/pdf"}
95 +
96 +def check_upload(stream, declared_type: str):
97 + head = stream.read(8); stream.seek(0)
98 + sniffed = next((t for magic, t in MAGIC.items() if head.startswith(magic)), None)
99 + if sniffed is None or sniffed != declared_type:
100 + raise ValueError("content does not match declared type") # never trust filename/Content-Type
101 +```
102 +
103 +Enforce `MAX_UPLOAD` at the web server (nginx `client_max_body_size`) as well —
104 +before the app buffers anything.
105 +
106 +## Gotchas
107 +
108 +- **`extra="ignore"` (the common default)** silently drops attacker fields today and mass-assigns them after the next model refactor; always `forbid`.
109 +- **Unanchored regex** — `re.search("[a-z]+")` passes `"$(rm -rf /)abc"`; anchor `^...$` and prefer `fullmatch`.
110 +- **Unicode homoglyphs after validation** — normalize (NFC/NFKC) first or `café` and `café` (combining accent) count as different users.
111 +- **`int` coercion of booleans** — in Python `True` is an `int`; pydantic v2 `strict=True` or explicit `StrictInt` where it matters.
112 +- **Trusting Content-Length** — read with a hard cap; a lying client otherwise OOMs the worker.
113 +- **Validating after parsing huge JSON** — depth/size bombs hit the parser first; cap body size and nesting at the server layer.
114 +- **First-error-only responses** — clients fix one field per round-trip; return all errors at once (see handler above).
added backend-skills/writing-background-jobs/SKILL.md +47 −0
@@ -0,0 +1,47 @@
1 +---
2 +name: writing-background-jobs
3 +description: Designs and implements background jobs and task queues that survive retries, crashes, and duplicate delivery — idempotency, backoff, dead-letter queues, timeouts, and job observability. Use when the user asks to write a background job, worker, or task queue, move work out of a request (send emails, process images, sync data), add retries to a job, or fix duplicate/stuck jobs (Celery, Sidekiq, BullMQ, RQ, or hand-rolled workers). Do not use for inter-service event/messaging architecture (handling-async-messaging) or cron-style system administration.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing Background Jobs
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** designing or reviewing background jobs, workers, and task queues inside one application: enqueueing, retries, failure handling, monitoring.
15 +- **Do NOT use for:** events between services (→ handling-async-messaging), customer-facing webhook delivery (→ designing-webhooks), or OS-level cron administration.
16 +
17 +## Core rules
18 +
19 +1. **Every job is idempotent.** Queues deliver at-least-once; your job WILL run twice. Guard side effects with a natural key or an idempotency record.
20 + - ✅ `INSERT ... ON CONFLICT (payment_id) DO NOTHING`, then act only on inserted rows
21 + - ❌ `charge_card(amount)` executed unconditionally at the top of the job
22 +2. **Payloads carry IDs, not objects.** Refetch current state at run time; serialized objects go stale between enqueue and execution.
23 + - ✅ `enqueue(send_invoice, invoice_id=42)`
24 + - ❌ `enqueue(send_invoice, invoice=invoice.to_json())`
25 +3. **Explicit retry policy on every job:** exponential backoff with jitter, capped attempts. Default: base 30 s, factor 2, full jitter, max 5 attempts. Distinguish retryable (network, 5xx) from permanent (validation) failures — permanent failures skip retries and go straight to the dead-letter queue.
26 +4. **Dead-letter queue with an alert.** Exhausted jobs land in a DLQ that pages someone; a silent DLQ is a data-loss buffer.
27 +5. **Timeout on every job.** No default-infinite jobs: set an explicit per-job timeout slightly above p99 runtime, and make the handler kill-safe (rule 1 covers the rerun).
28 +6. **No shared mutable state between jobs.** Workers run concurrently across processes and hosts; coordinate through the database (row locks, `SELECT ... FOR UPDATE SKIP LOCKED`) — never through process memory or files.
29 +7. **Long work = chain of short jobs.** Split anything over ~1 minute into resumable steps (paginate by cursor, one page per job). Short jobs retry cheaply; hour-long jobs lose an hour per crash.
30 +8. **Instrument jobs like endpoints:** duration, success/failure counters per job type, queue depth and oldest-message age with alerts. Queue depth growing while workers are idle means a poison job or a crashed consumer.
31 +
32 +## Workflow
33 +
34 +1. Define the job: trigger, payload (IDs only), side effects, and the idempotency key for each side effect.
35 +2. Classify failures as retryable vs permanent; set backoff (30 s base, ×2, jitter, 5 attempts) and timeout (≈ p99 runtime + margin).
36 +3. Implement with the project's existing queue library — do not introduce a new one if any queue is already present.
37 +4. Wire the DLQ and its alert; add duration/failure metrics.
38 +5. **Validate:** run the job twice with the same payload and confirm exactly one side effect; kill the worker mid-job and confirm the retry completes cleanly.
39 +
40 +## Edge cases & failure modes
41 +- **Job depends on uncommitted data** → enqueue after commit (transactional enqueue or on-commit hook), or the worker races the transaction and sees nothing.
42 +- **Poison message** (crashes the worker every time) → attempts cap sends it to the DLQ; never retry forever.
43 +- **Queue backlog after outage** → workers must tolerate a thundering herd: keep rule 1, add concurrency limits per job type.
44 +- **Scheduled (cron-like) jobs double-fire** on overlapping schedules → take a distributed lock keyed by job name + period before running.
45 +
46 +## References
47 +Deeper recipes and library-specific snippets: see [references/patterns.md](references/patterns.md).
added backend-skills/writing-background-jobs/references/patterns.md +135 −0
@@ -0,0 +1,135 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Background Jobs
7 +
8 +## Contents
9 +- Idempotency record
10 +- Backoff with full jitter
11 +- Transactional enqueue (enqueue after commit)
12 +- Chunked long-running job
13 +- Worker coordination with SKIP LOCKED
14 +- Distributed lock for scheduled jobs
15 +- Observability queries
16 +- Gotchas
17 +
18 +## Idempotency record
19 +
20 +```python
21 +# One row per logical operation; the INSERT is the guard.
22 +def send_invoice_email(invoice_id: int):
23 + with db.transaction():
24 + inserted = db.execute(
25 + """INSERT INTO job_runs (job_key)
26 + VALUES (%s) ON CONFLICT (job_key) DO NOTHING""",
27 + (f"invoice-email:{invoice_id}",),
28 + ).rowcount
29 + if not inserted:
30 + return # duplicate delivery — already done or in progress
31 + invoice = db.fetch_invoice(invoice_id) # refetch fresh state
32 + mailer.send(invoice.email, render(invoice))
33 +```
34 +
35 +## Backoff with full jitter
36 +
37 +```python
38 +import random
39 +
40 +BASE_S = 30 # first retry ~30 s: transient blips resolve in seconds
41 +FACTOR = 2
42 +MAX_ATTEMPTS = 5 # ~30s..8min window; beyond that it's an outage, use the DLQ
43 +
44 +def next_delay(attempt: int) -> float:
45 + return random.uniform(0, BASE_S * FACTOR ** attempt) # full jitter
46 +```
47 +
48 +```python
49 +# Celery equivalent
50 +@app.task(bind=True, max_retries=5, retry_backoff=30,
51 + retry_backoff_max=600, retry_jitter=True,
52 + autoretry_for=(TransientError,), time_limit=120)
53 +def sync_account(self, account_id): ...
54 +```
55 +
56 +## Transactional enqueue (enqueue after commit)
57 +
58 +```python
59 +# ❌ enqueue inside the transaction: worker may run before COMMIT
60 +# ✅ enqueue on commit
61 +with db.transaction() as tx:
62 + order_id = create_order(tx)
63 + tx.on_commit(lambda: queue.enqueue(process_order, order_id=order_id))
64 +```
65 +
66 +If the queue library has no on-commit hook, write the job to an
67 +`outbox`-style table in the same transaction and let a relay enqueue it.
68 +
69 +## Chunked long-running job
70 +
71 +```python
72 +PAGE = 500 # one page ≈ seconds of work: cheap to retry, no timeout risk
73 +
74 +def reindex_products(cursor: int = 0):
75 + rows = db.fetch("SELECT id FROM products WHERE id > %s ORDER BY id LIMIT %s",
76 + (cursor, PAGE))
77 + for r in rows:
78 + index(r.id)
79 + if len(rows) == PAGE:
80 + queue.enqueue(reindex_products, cursor=rows[-1].id) # resume point
81 +```
82 +
83 +## Worker coordination with SKIP LOCKED
84 +
85 +```sql
86 +-- Homemade queue table: safe concurrent pickup, no double-claim
87 +UPDATE jobs SET state = 'running', locked_at = now()
88 +WHERE id = (
89 + SELECT id FROM jobs
90 + WHERE state = 'pending' AND run_at <= now()
91 + ORDER BY run_at
92 + FOR UPDATE SKIP LOCKED
93 + LIMIT 1
94 +)
95 +RETURNING *;
96 +```
97 +
98 +## Distributed lock for scheduled jobs
99 +
100 +```python
101 +# Prevents double-fire when two schedulers overlap.
102 +def run_nightly_report():
103 + got = redis.set("lock:nightly-report:2026-08-05", worker_id,
104 + nx=True, ex=3600) # ex ≈ expected runtime + margin
105 + if not got:
106 + return
107 + ...
108 +```
109 +
110 +## Observability queries
111 +
112 +```python
113 +# Emit per job type
114 +metrics.timing(f"job.{name}.duration_ms", elapsed)
115 +metrics.incr(f"job.{name}.{'ok' if success else 'failed'}")
116 +```
117 +
118 +Alert on: DLQ size > 0 (page), oldest pending message age > 5× expected
119 +latency (warn), failure rate > 5% over 10 min (warn).
120 +
121 +## Gotchas
122 +
123 +- **`retry_jitter` off by default** in several libraries — synchronized
124 + retries stampede the dependency that just recovered.
125 +- **Visibility timeout < job timeout** (SQS-style queues) → the message
126 + reappears while the first worker still runs it; keep visibility ≥ job
127 + timeout + margin.
128 +- **Enqueue-then-crash before commit** → job references data that never
129 + existed; see transactional enqueue above.
130 +- **Serialized enums/dataclasses** break old in-flight jobs on deploy;
131 + IDs-only payloads (rule 2) sidestep the whole class of errors.
132 +- **DLQ replays must go through the same idempotency guard** — a replay is
133 + just one more duplicate delivery.
134 +- **`SELECT ... FOR UPDATE` without `SKIP LOCKED`** serializes all workers
135 + on one row: a queue with one effective consumer.
added db-skills/README.md +40 −0
@@ -0,0 +1,40 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# db-skills — Database Management Skill Collection
7 +
8 +**Author:** Simon-Pierre Boucher
9 +**Contact:** contact@spboucher.ai
10 +
11 +Ten ultra-sharp database-management skills following the method in
12 +[../RESEARCH-SYNTHESIS.md](../RESEARCH-SYNTHESIS.md), grounded in current
13 +practice (3NF-first design, measure-before-tuning with EXPLAIN ANALYZE /
14 +pg_stat_statements, index every foreign key, expand→migrate→contract
15 +zero-downtime migrations, 3-2-1 backups with restore drills). PostgreSQL is
16 +the default dialect; deviations for MySQL/SQLite are noted where they matter.
17 +All pass `python3 ../tools/validate_skills.py`.
18 +
19 +## The collection and its boundaries
20 +
21 +| Skill | Handles | Explicitly does NOT handle |
22 +|---|---|---|
23 +| `writing-sql-queries` | correct, readable, parameterized SQL; CTEs; window functions; NULL traps | schema design → `designing-database-schemas`; tuning → `optimizing-sql-performance` |
24 +| `designing-database-schemas` | domain modeling, 3NF, keys, types, constraints, naming | query writing; migration mechanics; NoSQL → `modeling-nosql-data` |
25 +| `optimizing-sql-performance` | EXPLAIN ANALYZE, indexes, plans, N+1, keyset pagination | incidents → `troubleshooting-databases`; fresh schema design |
26 +| `managing-database-migrations` | versioned, immutable, reversible migrations; zero-downtime patterns | target-schema design; one-off data fixes |
27 +| `backing-up-databases` | scheduled backups, logical vs physical, PITR, 3-2-1, restore drills | HA/replication setup; migrations |
28 +| `securing-databases` | least-privilege roles, injection defense, secrets, TLS, RLS, auditing | app-level auth (sessions/JWT); general firewalls |
29 +| `administering-postgresql` | roles, config knobs, autovacuum, monitoring, upgrades | app query tuning; backup strategy |
30 +| `managing-sqlite` | WAL, pragmas, transactions, safe backup, type affinity | server databases; generic SQL writing |
31 +| `modeling-nosql-data` | access-pattern-first modeling, embed vs reference, key design, TTL | relational schemas; JSON columns in SQL databases |
32 +| `troubleshooting-databases` | incident triage runbook: connections → locks → slow queries → disk → replication | proactive tuning; server configuration |
33 +
34 +## Shared conventions
35 +
36 +- Description = WHAT + "Use when …" (literal phrases) + "Do not use for …"
37 +- Measure before changing anything; capture evidence before restarting anything
38 +- One default per decision (tool, key type, journal mode) with an escape hatch
39 +- Every workflow ends with a validation step (test restore, rollback rehearsal, EXPLAIN before/after)
40 +- `SKILL.md` <150 lines; depth in `references/patterns.md` (with TOC)
added db-skills/administering-postgresql/SKILL.md +48 −0
@@ -0,0 +1,48 @@
1 +---
2 +name: administering-postgresql
3 +description: Administers PostgreSQL servers — roles and privileges, configuration tuning with sane starting values, autovacuum, monitoring with pg_stat_statements, extensions, and version upgrades. Use when the user asks to configure or tune a Postgres server, create roles or grant permissions, fix autovacuum or bloat, monitor a Postgres instance, install an extension, or plan a Postgres upgrade. Do not use for writing or tuning application queries, backup strategy, or other database engines.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Administering PostgreSQL
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** server-side PostgreSQL work — roles/privileges, `postgresql.conf` tuning, autovacuum, monitoring, extensions, upgrades.
15 +- **Do NOT use for:** writing application SQL (`writing-sql-queries`), query tuning (`optimizing-sql-performance`), backup/restore strategy (`backing-up-databases`), or MySQL/SQLite.
16 +
17 +## Core rules
18 +
19 +1. **Roles: one login role per human/service, privileges via group roles.**
20 + - ✅ `CREATE ROLE app_rw NOLOGIN; GRANT app_rw TO svc_api;`
21 + - ❌ Granting table privileges directly to a dozen login roles.
22 +2. **New objects need default privileges, not just `GRANT`.** `GRANT` covers existing tables only; run `ALTER DEFAULT PRIVILEGES IN SCHEMA app GRANT SELECT ON TABLES TO app_ro;` for future ones.
23 +3. **Config starting values, then measure:** `shared_buffers` = 25% of RAM (cap ~8GB before measuring), `effective_cache_size` = 50–75% of RAM, `work_mem` = 16–64MB — it is **per sort/hash per query**, so `max_connections × work_mem` must fit in RAM.
24 +4. **Never raise `max_connections` to fix connection errors — add a pooler.** PgBouncer in transaction mode with `max_connections` ≤ 200 beats 2000 raw connections.
25 + - ✅ App → PgBouncer (pool_size 20) → Postgres.
26 + - ❌ `max_connections = 5000`.
27 +5. **Never disable autovacuum.** For hot tables, tune per-table instead: `ALTER TABLE events SET (autovacuum_vacuum_scale_factor = 0.02);`
28 +6. **Measure before touching anything:** enable `pg_stat_statements` (`shared_preload_libraries`), and use `pg_stat_activity` / bloat queries from the reference file.
29 +7. **Extensions go through migrations,** not psql one-offs: `CREATE EXTENSION IF NOT EXISTS pg_trgm;` in a versioned migration so every environment matches.
30 +8. **Upgrades: `pg_upgrade --link` for same-host major upgrades; logical replication when downtime must be near zero.** Always run `ANALYZE` after either.
31 +
32 +## Workflow
33 +
34 +1. State the goal (new role, config change, slow server, extension, upgrade) and capture current state first: version, `SELECT * FROM pg_settings WHERE source <> 'default';`, top queries from `pg_stat_statements`.
35 +2. Apply the smallest change that addresses the goal (one knob or one grant at a time).
36 +3. Reload or restart as required: `SELECT pg_reload_conf();` for reloadable settings; note in your reply if a restart is required (`SELECT name FROM pg_settings WHERE pending_restart;`).
37 +4. Validate: re-run the measurement from step 1 and confirm the change took effect (`SHOW shared_buffers;`, `\du`, `\l+`) and improved the metric.
38 +5. Record what changed and why in the project's migration/ops notes.
39 +
40 +## Edge cases & failure modes
41 +- **`pg_stat_statements` missing** → it needs `shared_preload_libraries = 'pg_stat_statements'` + restart, then `CREATE EXTENSION pg_stat_statements;`.
42 +- **Permission denied after GRANT** → almost always missing `GRANT USAGE ON SCHEMA` or missing default privileges (rule 2).
43 +- **Config change has no effect** → check `pg_settings.pending_restart`; some knobs (e.g. `shared_buffers`) need a full restart.
44 +- **Managed Postgres (RDS/Cloud SQL)** → no filesystem or `postgresql.conf`; use parameter groups / flags, and superuser is unavailable — use the provider's admin role.
45 +- **Out-of-disk from bloat** → do NOT run `VACUUM FULL` on a hot table in peak hours (exclusive lock); use `pg_repack` or schedule a window.
46 +
47 +## References
48 +Copy-paste monitoring SQL, role templates, and upgrade commands: see [references/patterns.md](references/patterns.md).
added db-skills/administering-postgresql/references/patterns.md +144 −0
@@ -0,0 +1,144 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# PostgreSQL Administration — Patterns
7 +
8 +## Contents
9 +- Roles and privileges
10 +- Configuration starting points
11 +- Autovacuum tuning
12 +- Monitoring queries
13 +- Extensions
14 +- Upgrades
15 +- Gotchas
16 +
17 +## Roles and privileges
18 +
19 +```sql
20 +-- Group roles hold privileges; login roles are members.
21 +CREATE ROLE app_ro NOLOGIN;
22 +CREATE ROLE app_rw NOLOGIN;
23 +
24 +GRANT USAGE ON SCHEMA app TO app_ro, app_rw;
25 +GRANT SELECT ON ALL TABLES IN SCHEMA app TO app_ro;
26 +GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA app TO app_rw;
27 +GRANT USAGE ON ALL SEQUENCES IN SCHEMA app TO app_rw;
28 +
29 +-- Future objects too (run as the role that creates the tables):
30 +ALTER DEFAULT PRIVILEGES IN SCHEMA app GRANT SELECT ON TABLES TO app_ro;
31 +ALTER DEFAULT PRIVILEGES IN SCHEMA app
32 + GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_rw;
33 +
34 +-- Login roles:
35 +CREATE ROLE svc_api LOGIN PASSWORD '...' IN ROLE app_rw;
36 +CREATE ROLE analyst_anne LOGIN PASSWORD '...' IN ROLE app_ro;
37 +```
38 +
39 +Audit: `\du+` (roles), `\dp app.*` (table privileges).
40 +
41 +## Configuration starting points
42 +
43 +```ini
44 +# postgresql.conf — starting values for a dedicated 16GB server; measure after.
45 +shared_buffers = 4GB # 25% of RAM
46 +effective_cache_size = 12GB # 75% of RAM (planner hint, not allocation)
47 +work_mem = 32MB # per sort/hash per query — keep conns × work_mem « RAM
48 +maintenance_work_mem = 512MB # vacuum/index builds
49 +max_connections = 200 # use PgBouncer instead of raising this
50 +wal_compression = on
51 +shared_preload_libraries = 'pg_stat_statements'
52 +```
53 +
54 +Reload vs restart:
55 +
56 +```sql
57 +SELECT pg_reload_conf(); -- reloadable knobs
58 +SELECT name FROM pg_settings WHERE pending_restart; -- needs restart?
59 +SELECT name, setting, source FROM pg_settings WHERE source <> 'default';
60 +```
61 +
62 +## Autovacuum tuning
63 +
64 +```sql
65 +-- Default scale factor 0.2 = vacuum after 20% dead rows — too lazy for hot tables.
66 +ALTER TABLE app.events SET (
67 + autovacuum_vacuum_scale_factor = 0.02, -- vacuum at 2% dead rows
68 + autovacuum_analyze_scale_factor = 0.01
69 +);
70 +-- Check autovacuum activity:
71 +SELECT relname, last_autovacuum, n_dead_tup, n_live_tup
72 +FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 10;
73 +```
74 +
75 +## Monitoring queries
76 +
77 +```sql
78 +-- What is running now (and what it waits on):
79 +SELECT pid, state, wait_event_type, wait_event, now() - query_start AS runtime,
80 + left(query, 80) AS query
81 +FROM pg_stat_activity WHERE state <> 'idle' ORDER BY runtime DESC;
82 +
83 +-- Top queries by total time (needs pg_stat_statements):
84 +SELECT round(total_exec_time) AS ms, calls, rows, left(query, 100) AS query
85 +FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;
86 +
87 +-- Table sizes incl. indexes and toast:
88 +SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) AS total
89 +FROM pg_catalog.pg_statio_user_tables
90 +ORDER BY pg_total_relation_size(relid) DESC LIMIT 10;
91 +
92 +-- Unused indexes (candidates for removal — check replicas first):
93 +SELECT indexrelname, idx_scan, pg_size_pretty(pg_relation_size(indexrelid))
94 +FROM pg_stat_user_indexes WHERE idx_scan = 0
95 +ORDER BY pg_relation_size(indexrelid) DESC;
96 +
97 +-- Cache hit ratio (want > 0.99 on OLTP):
98 +SELECT sum(blks_hit)::float / nullif(sum(blks_hit) + sum(blks_read), 0)
99 +FROM pg_stat_database;
100 +```
101 +
102 +## Extensions
103 +
104 +```sql
105 +-- In a versioned migration, never ad hoc:
106 +CREATE EXTENSION IF NOT EXISTS pg_trgm; -- fuzzy text search
107 +CREATE EXTENSION IF NOT EXISTS pgcrypto; -- gen_random_uuid() pre-v13
108 +SELECT extname, extversion FROM pg_extension; -- installed
109 +SELECT name, default_version FROM pg_available_extensions -- available
110 +WHERE name LIKE 'pg%' LIMIT 20;
111 +```
112 +
113 +## Upgrades
114 +
115 +```bash
116 +# Same-host major upgrade (minutes of downtime, hard links, no data copy):
117 +pg_upgrade --link \
118 + --old-datadir /var/lib/postgresql/15/main \
119 + --new-datadir /var/lib/postgresql/16/main \
120 + --old-bindir /usr/lib/postgresql/15/bin \
121 + --new-bindir /usr/lib/postgresql/16/bin
122 +# Then ALWAYS:
123 +vacuumdb --all --analyze-in-stages
124 +```
125 +
126 +Near-zero-downtime alternative: logical replication — create publication on old,
127 +subscription on new, wait for sync, switch the application, drop subscription.
128 +Statistics are NOT migrated by either path — `ANALYZE` is mandatory.
129 +
130 +## Gotchas
131 +
132 +- `work_mem` is per **operation**, not per connection — a single query with 4
133 + sorts can use 4 × work_mem. This is the classic OOM cause.
134 +- `GRANT ALL ON ALL TABLES` does not cover tables created later — you need
135 + `ALTER DEFAULT PRIVILEGES` (and it only applies to objects created by the
136 + role that ran it).
137 +- `shared_buffers` beyond ~8GB often yields nothing — the OS page cache does
138 + the rest; measure before going higher.
139 +- PgBouncer transaction mode breaks session state: no `SET`, no advisory locks,
140 + no `LISTEN/NOTIFY`, no prepared statements (before PgBouncer 1.21).
141 +- `VACUUM FULL` takes an ACCESS EXCLUSIVE lock and rewrites the table — it is
142 + an outage, not maintenance. Prefer `pg_repack`.
143 +- On managed services (RDS, Cloud SQL), `shared_preload_libraries` is set via
144 + parameter group + reboot, and there is no true superuser.
added db-skills/backing-up-databases/SKILL.md +46 −0
@@ -0,0 +1,46 @@
1 +---
2 +name: backing-up-databases
3 +description: Designs and implements database backup and restore strategies including scheduled dumps, point-in-time recovery, retention, and restore drills. Use when the user asks to back up a database, set up pg_dump or mysqldump, configure WAL archiving or point-in-time recovery, restore a database or a single table, define backup retention, or verify that backups actually work. Do not use for high-availability or replication setup, or for schema migrations (managing-database-migrations).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Backing Up Databases
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** backup scheduling, dump commands, point-in-time recovery (PITR), restore procedures, retention policies, restore drills, backup encryption and monitoring.
15 +- **Do NOT use for:** replication/failover architecture, schema migrations (`managing-database-migrations`), or access control (`securing-databases`).
16 +
17 +## Core rules
18 +
19 +1. **A backup is only real once it has been restored.** An untested backup is a hope, not a backup. Every strategy MUST include a scheduled restore drill (monthly at minimum) into a scratch instance, with row-count spot checks.
20 +2. **Automate or it doesn't exist.** Backups run from cron/systemd timers/managed schedules — never "someone runs pg_dump sometimes."
21 + - ✅ systemd timer + failure alert
22 + - ❌ README saying "remember to back up before releases"
23 +3. **Pick logical vs physical by size and RPO.** Default: logical dumps (`pg_dump -Fc`) for databases < ~100 GB or when per-table restore matters; physical base backup + WAL archiving when the database is larger or the recovery-point objective is minutes, not hours.
24 +4. **PITR needs both halves.** A base backup without continuous WAL/binlog archiving cannot restore to "5 minutes before the bad DELETE." If the user needs that sentence to be true, set up WAL archiving.
25 +5. **Follow 3-2-1.** Three copies, two media/locations, one off-site (different cloud region or provider). The backup living on the same disk as the database counts as zero copies.
26 +6. **Encrypt at rest, and keep the key OUT of the backup location.** `age`/`gpg` on dump files or server-side encryption on the bucket.
27 +7. **Failures must be loud.** Alert on job failure AND on "no successful backup in N hours" (dead-man switch) — a silently disabled cron job is the classic disaster.
28 +8. **Retention is a policy, not an accident.** Default: 7 daily + 4 weekly + 12 monthly; adjust to compliance needs and state it in the backup script header.
29 +
30 +## Workflow
31 +
32 +1. Establish requirements: database engine, size, acceptable data loss (RPO) and downtime (RTO), compliance retention.
33 +2. Choose the mechanism by rule 3; write the backup script/config with encryption (rule 6) and retention pruning (rule 8).
34 +3. Schedule it and wire both failure and dead-man alerts (rule 7).
35 +4. Write the restore procedure as a runnable script or step list — not prose in someone's head.
36 +5. **Validate:** perform the restore into a scratch database NOW, verify with row counts on the 3 largest tables and one known record. Record the restore duration (that is your real RTO). The task is not done until this restore succeeds.
37 +
38 +## Edge cases & failure modes
39 +- **Database too big to dump within the window** → switch to physical backups + WAL archiving; never let dumps overlap.
40 +- **Restore drill has no scratch environment** → a temporary Docker container of the same engine version is the minimum; same major version is mandatory for physical restores.
41 +- **`pg_dump` version mismatch** → always dump with the NEWER client version; restore with `pg_restore` matching the target server.
42 +- **Managed databases (RDS/Cloud SQL)** → provider snapshots + PITR cover rules 3–4, but still run logical dumps for off-provider copies (rule 5) and still do restore drills (rule 1).
43 +- **Backup contains secrets/PII** → encryption (rule 6) is non-negotiable; restrict bucket access to the backup role only.
44 +
45 +## References
46 +Runnable scripts and PITR walkthrough: see [references/patterns.md](references/patterns.md)
added db-skills/backing-up-databases/references/patterns.md +139 −0
@@ -0,0 +1,139 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Backup & Restore Patterns — Recipes
7 +
8 +## Contents
9 +- Nightly logical backup (PostgreSQL)
10 +- Restore from a logical dump
11 +- Point-in-time recovery (PITR) setup
12 +- PITR restore walkthrough
13 +- MySQL equivalents
14 +- Retention pruning
15 +- Dead-man monitoring
16 +- Gotchas
17 +
18 +## Nightly logical backup (PostgreSQL)
19 +
20 +```bash
21 +#!/usr/bin/env bash
22 +# nightly-backup.sh — custom-format dump, encrypted, uploaded off-site.
23 +# Retention: 7 daily / 4 weekly / 12 monthly (prune step below).
24 +set -euo pipefail
25 +
26 +DB="appdb"
27 +STAMP=$(date +%F)
28 +OUT="/var/backups/pg/${DB}-${STAMP}.dump.age"
29 +
30 +# -Fc = custom format: compressed, supports parallel & per-table restore
31 +pg_dump -Fc --no-owner "$DB" \
32 + | age -r "$BACKUP_PUBLIC_KEY" > "$OUT"
33 +
34 +# off-site copy (different region/provider than the DB)
35 +aws s3 cp "$OUT" "s3://acme-db-backups/${DB}/" --only-show-errors
36 +
37 +# dead-man ping: monitoring alerts if this URL isn't hit every 24h
38 +curl -fsS "https://hc-ping.com/${HEALTHCHECK_ID}" > /dev/null
39 +```
40 +
41 +Schedule with a systemd timer (survives reboots, logs to journal), not user cron.
42 +
43 +## Restore from a logical dump
44 +
45 +```bash
46 +age -d -i backup_key.txt appdb-2026-08-05.dump.age > appdb.dump
47 +createdb appdb_restore
48 +pg_restore -d appdb_restore --no-owner -j 4 appdb.dump # -j 4: parallel
49 +
50 +# verification minimum
51 +psql appdb_restore -c "SELECT count(*) FROM orders;"
52 +psql appdb_restore -c "SELECT count(*) FROM users;"
53 +psql appdb_restore -c "SELECT email FROM users WHERE id = 1;" # known record
54 +```
55 +
56 +Single table only: `pg_restore -d appdb_restore -t orders appdb.dump`
57 +
58 +## Point-in-time recovery (PITR) setup
59 +
60 +postgresql.conf:
61 +
62 +```ini
63 +wal_level = replica
64 +archive_mode = on
65 +# archive to object storage; %p = file path, %f = file name
66 +archive_command = 'wal-g wal-push %p' # or: aws s3 cp %p s3://bucket/wal/%f
67 +```
68 +
69 +Base backup (weekly, plus WAL stream between):
70 +
71 +```bash
72 +wal-g backup-push /var/lib/postgresql/16/main
73 +# or without wal-g:
74 +pg_basebackup -D /var/backups/pg/base-$(date +%F) -Ft -z -Xs -P
75 +```
76 +
77 +## PITR restore walkthrough
78 +
79 +Recover to just before a bad statement at 14:32:10:
80 +
81 +```bash
82 +# 1. stop postgres, move the broken data dir aside
83 +# 2. restore the latest base backup BEFORE the target time into the data dir
84 +wal-g backup-fetch /var/lib/postgresql/16/main LATEST
85 +# 3. tell recovery where to stop
86 +cat > /var/lib/postgresql/16/main/postgresql.auto.conf <<'EOF'
87 +restore_command = 'wal-g wal-fetch %f %p'
88 +recovery_target_time = '2026-08-05 14:32:00+00'
89 +recovery_target_action = 'promote'
90 +EOF
91 +touch /var/lib/postgresql/16/main/recovery.signal
92 +# 4. start postgres; it replays WAL to the target time, then promotes
93 +```
94 +
95 +## MySQL equivalents
96 +
97 +```bash
98 +# logical dump, single transaction = consistent without locking InnoDB
99 +mysqldump --single-transaction --routines --triggers appdb | gzip > appdb.sql.gz
100 +# PITR half: enable binlog (log_bin=ON), archive binlogs off-site
101 +# restore: load dump, then replay binlogs to a point:
102 +mysqlbinlog --stop-datetime="2026-08-05 14:32:00" binlog.0000* | mysql appdb
103 +```
104 +
105 +## Retention pruning
106 +
107 +```bash
108 +# keep 7 daily; weekly (Sunday) kept 28 days; monthly (1st) kept 365 days
109 +find /var/backups/pg -name '*.dump.age' -mtime +7 \
110 + ! -newermt "$(date -d 'last sunday' +%F)" -delete 2>/dev/null || true
111 +# simplest robust option: let the object store do it — S3 lifecycle rules
112 +# per prefix daily/ weekly/ monthly/, and upload into the matching prefix.
113 +```
114 +
115 +Prefer bucket lifecycle policies over local `find` arithmetic when possible.
116 +
117 +## Dead-man monitoring
118 +
119 +Alert on absence, not just failure:
120 +
121 +- Push a ping (healthchecks.io, Cronitor, PagerDuty heartbeat) as the LAST
122 + line of the backup script (only reached on success).
123 +- Second check: a daily job that fails if the newest object in the backup
124 + bucket is older than 26 h (24 h schedule + 2 h grace).
125 +
126 +## Gotchas
127 +
128 +- **`pg_dump` while DDL runs** can fail mid-dump with "relation changed";
129 + schedule dumps away from migration windows.
130 +- **Physical restores require the same major version and architecture**;
131 + logical dumps are the portable path across versions.
132 +- **`--no-owner`** on dump/restore avoids failures when the scratch instance
133 + lacks the original roles.
134 +- **WAL archiving fills the disk if `archive_command` fails** — PostgreSQL keeps
135 + WAL until archived. Alert on `pg_stat_archiver.failed_count`.
136 +- **Snapshots of a running DB without filesystem/DB coordination** can be
137 + torn on multi-volume setups; use `pg_basebackup`/provider snapshots instead.
138 +- **Testing restores against the production instance** — never; always a
139 + scratch instance or container.
added db-skills/designing-database-schemas/SKILL.md +58 −0
@@ -0,0 +1,58 @@
1 +---
2 +name: designing-database-schemas
3 +description: Designs relational database schemas — tables, primary keys, column types, constraints, naming, and normalization decisions. Use when the user asks to design or review a database schema, create tables for a new feature or app, choose primary keys or column types, model entities and relationships, or decide about normalization or soft deletes. Do not use for writing application queries, tuning existing query performance, writing migration files, or modeling NoSQL documents — separate skills cover those.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Designing Database Schemas
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** designing or reviewing tables, keys, types, constraints, relationships, and normalization for a relational database.
15 +- **Do NOT use for:** query authoring (`writing-sql-queries`), performance tuning (`optimizing-sql-performance`), migration mechanics (`managing-database-migrations`), document/key-value modeling (`modeling-nosql-data`).
16 +
17 +Default dialect: PostgreSQL.
18 +
19 +## Core rules
20 +
21 +1. **Model the domain honestly.** One table per entity, one row per instance of that entity; name the grain in a comment if it is not obvious.
22 +
23 +2. **Start at 3NF; denormalize only after a measured bottleneck.** Duplicate a column or add a materialized view only when `EXPLAIN ANALYZE` on a real query proves the join is the problem — never on a hunch.
24 +
25 +3. **Default primary key: surrogate `bigint GENERATED ALWAYS AS IDENTITY`.** Use `uuid` (v7 if available) when IDs are generated client-side, exposed publicly, or merged across databases. Natural keys only for genuinely immutable identifiers (ISO country code) — emails and usernames change.
26 + - ✅ `id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY`
27 + - ❌ `email text PRIMARY KEY`
28 +
29 +4. **Types carry meaning:**
30 + - ✅ `timestamptz` — ❌ `timestamp` (naive, timezone bugs)
31 + - ✅ `numeric(12,2)` or integer cents for money — ❌ `float`/`real` (rounding errors)
32 + - ✅ `text` + `CHECK`/enum — ❌ `varchar(255)` cargo-cult limits
33 + - ✅ `boolean` — ❌ `char(1)` Y/N flags
34 +
35 +5. **Constraints are documentation that cannot go stale.** Every column `NOT NULL` unless NULL has a defined meaning; `CHECK` for invariants (`price_cents >= 0`); `UNIQUE` for business uniqueness; every relationship a real `FOREIGN KEY` with an explicit `ON DELETE` decision (no default cascades by accident).
36 +
37 +6. **Index every foreign key at creation time.** PostgreSQL does not do this automatically; unindexed FKs cause full-table scans on joins and cascaded deletes.
38 +
39 +7. **One naming convention, applied everywhere:** `snake_case`, singular table names (`user_order`), PK `id`, FK `<table>_id`, timestamps `created_at`/`updated_at`. If the existing schema uses plural, match it — consistency beats preference.
40 +
41 +8. **Soft deletes are a tradeoff, not a default.** `deleted_at timestamptz NULL` keeps history but every query and every `UNIQUE` constraint must account for it (use partial unique indexes). Choose hard delete + audit table when history, not resurrection, is the need.
42 +
43 +## Workflow
44 +
45 +1. List entities, relationships (1-1, 1-N, N-N), and the grain of each table.
46 +2. Draft `CREATE TABLE` statements applying rules 3–7; junction tables for N-N.
47 +3. Add constraints for every stated business rule.
48 +4. Validate: run the DDL in a scratch database, then insert one valid row and one row violating each constraint — every violation must fail. Fix and re-run until it does.
49 +5. Deliver DDL plus a one-line rationale per non-obvious decision.
50 +
51 +## Edge cases & failure modes
52 +- **Existing schema present** → inspect it first and match its conventions; flag (don't silently fix) inconsistencies.
53 +- **Polymorphic references** ("commentable_id + type") → prefer one FK column per target table or a supertype table; plain polymorphic columns can't have FK constraints.
54 +- **Multi-tenancy** → decide row-level (`tenant_id` on every table, composite indexes leading with it) vs schema-per-tenant before writing DDL.
55 +- **Very wide tables** (>30 columns) → usually two entities in disguise; split by update frequency or ownership.
56 +
57 +## References
58 +DDL templates, junction/soft-delete/audit patterns, dialect notes: see [references/patterns.md](references/patterns.md).
added db-skills/designing-database-schemas/references/patterns.md +126 −0
@@ -0,0 +1,126 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Designing Database Schemas
7 +
8 +## Contents
9 +- Canonical table template
10 +- One-to-many and many-to-many
11 +- Partial unique index with soft delete
12 +- Audit table (hard-delete alternative)
13 +- Enum strategies
14 +- updated_at trigger
15 +- Dialect deviations (MySQL, SQLite)
16 +- Gotchas
17 +
18 +## Canonical table template
19 +
20 +```sql
21 +CREATE TABLE app_user (
22 + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
23 + email text NOT NULL UNIQUE,
24 + display_name text NOT NULL,
25 + status text NOT NULL DEFAULT 'active'
26 + CHECK (status IN ('active', 'suspended', 'closed')),
27 + created_at timestamptz NOT NULL DEFAULT now(),
28 + updated_at timestamptz NOT NULL DEFAULT now()
29 +);
30 +```
31 +
32 +## One-to-many and many-to-many
33 +
34 +```sql
35 +CREATE TABLE user_order (
36 + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
37 + user_id bigint NOT NULL REFERENCES app_user (id) ON DELETE RESTRICT,
38 + total_cents integer NOT NULL CHECK (total_cents >= 0),
39 + created_at timestamptz NOT NULL DEFAULT now()
40 +);
41 +-- Rule 6: index the FK immediately.
42 +CREATE INDEX user_order_user_id_idx ON user_order (user_id);
43 +
44 +-- N-N: junction table, composite PK, both FKs indexed (PK covers the first).
45 +CREATE TABLE order_tag (
46 + order_id bigint NOT NULL REFERENCES user_order (id) ON DELETE CASCADE,
47 + tag_id bigint NOT NULL REFERENCES tag (id) ON DELETE CASCADE,
48 + PRIMARY KEY (order_id, tag_id)
49 +);
50 +CREATE INDEX order_tag_tag_id_idx ON order_tag (tag_id);
51 +```
52 +
53 +`ON DELETE` decision: `RESTRICT` (default choice — force explicit cleanup),
54 +`CASCADE` only for true child rows (junction rows, line items), `SET NULL`
55 +when the relationship is optional history.
56 +
57 +## Partial unique index with soft delete
58 +
59 +```sql
60 +ALTER TABLE app_user ADD COLUMN deleted_at timestamptz NULL;
61 +
62 +-- Plain UNIQUE(email) would block re-registering a deleted email:
63 +DROP INDEX IF EXISTS app_user_email_key;
64 +CREATE UNIQUE INDEX app_user_email_live_key
65 + ON app_user (email) WHERE deleted_at IS NULL;
66 +```
67 +
68 +Every live-row query now needs `WHERE deleted_at IS NULL` — encode it in a view
69 +if the application layer can't be trusted to remember.
70 +
71 +## Audit table (hard-delete alternative)
72 +
73 +```sql
74 +CREATE TABLE app_user_audit (
75 + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
76 + user_id bigint NOT NULL, -- no FK: row may be gone
77 + action text NOT NULL CHECK (action IN ('insert','update','delete')),
78 + old_row jsonb,
79 + changed_at timestamptz NOT NULL DEFAULT now()
80 +);
81 +```
82 +
83 +## Enum strategies
84 +
85 +Default — `text` + `CHECK` (cheap to extend: one `ALTER TABLE … DROP/ADD CONSTRAINT`):
86 +
87 +```sql
88 +status text NOT NULL CHECK (status IN ('draft','published','archived'))
89 +```
90 +
91 +Escape hatch — native `CREATE TYPE … AS ENUM` when many tables share the set;
92 +note that removing enum values is painful.
93 +Lookup table when values carry attributes (label, sort order) or are user-editable.
94 +
95 +## updated_at trigger
96 +
97 +```sql
98 +CREATE OR REPLACE FUNCTION set_updated_at() RETURNS trigger AS $$
99 +BEGIN
100 + NEW.updated_at = now();
101 + RETURN NEW;
102 +END $$ LANGUAGE plpgsql;
103 +
104 +CREATE TRIGGER app_user_touch BEFORE UPDATE ON app_user
105 +FOR EACH ROW EXECUTE FUNCTION set_updated_at();
106 +```
107 +
108 +## Dialect deviations (MySQL, SQLite)
109 +
110 +| PostgreSQL | MySQL 8 | SQLite |
111 +|---|---|---|
112 +| `bigint GENERATED ALWAYS AS IDENTITY` | `BIGINT AUTO_INCREMENT` | `INTEGER PRIMARY KEY` (rowid) |
113 +| `timestamptz` | `TIMESTAMP` (stored UTC) — store app-side UTC | `TEXT` ISO-8601 UTC |
114 +| `text` freely | prefer `VARCHAR(n)`; `TEXT` can't be fully indexed | `TEXT` |
115 +| partial indexes | none — emulate with generated column | supported |
116 +| `CHECK` enforced | enforced 8.0.16+ | enforced, but FKs need `PRAGMA foreign_keys = ON` |
117 +
118 +## Gotchas
119 +
120 +- PostgreSQL folds unquoted identifiers to lowercase — never create quoted CamelCase names.
121 +- `UNIQUE` allows multiple NULLs (PostgreSQL < 15 semantics); use `UNIQUE NULLS NOT DISTINCT` (15+) or `NOT NULL` if that's wrong for the domain.
122 +- FK constraints don't create indexes (rule 6) — the referenced side's PK is indexed, the referencing column is not.
123 +- `serial` is legacy; `GENERATED ALWAYS AS IDENTITY` is the standard-conforming replacement.
124 +- Random UUIDv4 PKs fragment B-tree indexes at scale; prefer UUIDv7 (time-ordered) when using uuid.
125 +- `varchar(255)` has no performance benefit over `text` in PostgreSQL — the limit is only a constraint.
126 +- Money as `float` fails equality checks and loses cents in aggregation — `numeric` or integer cents, always.
added db-skills/managing-database-migrations/SKILL.md +48 −0
@@ -0,0 +1,48 @@
1 +---
2 +name: managing-database-migrations
3 +description: Writes and manages versioned database schema migrations with safe rollbacks and zero-downtime deployment patterns. Use when the user asks to write, create, review, or fix a migration, add or drop a column/table/index, rename a column safely, run a backfill, or deploy a schema change without downtime, or mentions Alembic, Prisma Migrate, Rails migrations, or Flyway. Do not use for designing the target schema itself (designing-database-schemas) or for one-off production data fixes.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Managing Database Migrations
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** authoring, reviewing, sequencing, or fixing schema migrations; zero-downtime schema changes; backfills tied to schema changes; rollback strategy.
15 +- **Do NOT use for:** deciding what the schema should look like (that is `designing-database-schemas`), ad-hoc data corrections, or backup/restore work (`backing-up-databases`).
16 +
17 +## Core rules
18 +
19 +1. **One logical change per migration.** A migration that adds a table AND renames a column elsewhere cannot be partially rolled back.
20 + - ✅ `20260805_add_invoices_table.py` + `20260805_rename_users_phone.py`
21 + - ❌ `20260805_misc_schema_updates.py`
22 +2. **Applied migrations are immutable.** Never edit a migration that has run anywhere beyond your machine — write a new one that corrects it.
23 +3. **Every migration has a down path — or documents why not.** If `down` is impossible (data-destroying), state it in the migration header and require an explicit flag/confirmation to run.
24 +4. **Use the project's framework-native tool.** Detect it before writing anything: `alembic.ini` → Alembic, `prisma/schema.prisma` → Prisma Migrate, `db/migrate/` → Rails, `flyway.conf`/`sql/V*.sql` → Flyway. Only fall back to raw SQL files + a version table if the project has no tool.
25 +5. **Never mix schema changes and data changes in one migration.** Schema migrations run in DDL transactions; backfills are long-running DML. Split them: schema → backfill → schema.
26 +6. **Zero-downtime changes follow expand → migrate → contract.** The app must work with both old and new schema between steps.
27 + - ✅ add nullable column → deploy code writing both → backfill → add `NOT NULL` → remove old column later
28 + - ❌ `ALTER TABLE users RENAME COLUMN phone TO phone_number;` in one release
29 +7. **Long locks are outages.** On PostgreSQL: `CREATE INDEX CONCURRENTLY` (outside a transaction), add `NOT NULL` via `CHECK ... NOT VALID` + `VALIDATE CONSTRAINT`, batch backfills (see references).
30 +8. **Test the rollback, not just the migration.** `up` then `down` then `up` again on a scratch database must succeed before review.
31 +
32 +## Workflow
33 +
34 +1. Detect the migration tool (rule 4) and read the two most recent migrations to match naming and style.
35 +2. Classify the change: additive (safe), destructive (needs expand/contract), or data-touching (needs a separate backfill migration).
36 +3. Write the migration(s) — smallest possible units, down paths included.
37 +4. For destructive or locking changes, write the deployment sequence as comments at the top of the migration (which app version must be live before/after).
38 +5. **Validate:** run `up` → `down` → `up` against a scratch/dev database and paste the tool's output. A migration is not done until this passes.
39 +
40 +## Edge cases & failure modes
41 +- **No scratch DB available** → say so explicitly and mark the migration untested; never claim validation that didn't run.
42 +- **Migration already applied and wrong** → write a corrective follow-up migration; never edit history (rule 2).
43 +- **Backfill on a large table** → batch by primary-key range with pauses (see references); a single `UPDATE table SET ...` locks the table.
44 +- **Divergent heads (two branches added migrations)** → merge with the tool's mechanism (e.g. `alembic merge`), never renumber existing files.
45 +- **MySQL** → no transactional DDL: a failed migration leaves partial state; make each statement idempotent (`IF EXISTS`/`IF NOT EXISTS`).
46 +
47 +## References
48 +Deeper recipes and lock-safe SQL: see [references/patterns.md](references/patterns.md)
added db-skills/managing-database-migrations/references/patterns.md +124 −0
@@ -0,0 +1,124 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Migration Patterns — Recipes
7 +
8 +## Contents
9 +- Expand → migrate → contract (column rename)
10 +- Adding NOT NULL without a long lock (PostgreSQL)
11 +- Index creation without blocking writes
12 +- Batched backfill
13 +- Safe destructive migrations
14 +- Divergent heads
15 +- Gotchas
16 +
17 +## Expand → migrate → contract (column rename)
18 +
19 +Renaming `users.phone` → `users.phone_number` across releases:
20 +
21 +```sql
22 +-- Migration 1 (expand): release N
23 +ALTER TABLE users ADD COLUMN phone_number text;
24 +
25 +-- Release N code: write both columns, read phone_number
26 +-- COALESCE(phone_number, phone) during transition.
27 +
28 +-- Migration 2 (backfill): separate migration, batched (see below)
29 +-- Migration 3 (contract): release N+1, after all rows copied
30 +ALTER TABLE users DROP COLUMN phone;
31 +```
32 +
33 +Down paths: 1 → `DROP COLUMN phone_number`; 3 is irreversible — document it.
34 +
35 +## Adding NOT NULL without a long lock (PostgreSQL)
36 +
37 +A plain `ALTER TABLE ... SET NOT NULL` scans the whole table under an
38 +ACCESS EXCLUSIVE lock. Split it:
39 +
40 +```sql
41 +-- Step 1: instant, does not validate existing rows
42 +ALTER TABLE orders ADD CONSTRAINT orders_customer_id_nn
43 + CHECK (customer_id IS NOT NULL) NOT VALID;
44 +
45 +-- Step 2: scans without blocking writes (SHARE UPDATE EXCLUSIVE)
46 +ALTER TABLE orders VALIDATE CONSTRAINT orders_customer_id_nn;
47 +
48 +-- Step 3 (PG 12+): SET NOT NULL is instant because the valid CHECK proves it
49 +ALTER TABLE orders ALTER COLUMN customer_id SET NOT NULL;
50 +ALTER TABLE orders DROP CONSTRAINT orders_customer_id_nn;
51 +```
52 +
53 +## Index creation without blocking writes
54 +
55 +```sql
56 +-- PostgreSQL: must run OUTSIDE a transaction block
57 +CREATE INDEX CONCURRENTLY idx_orders_customer_id ON orders (customer_id);
58 +```
59 +
60 +- Alembic: `op.create_index(..., postgresql_concurrently=True)` with
61 + `with op.get_context().autocommit_block():`
62 +- If it fails it leaves an INVALID index — drop it and retry:
63 + `DROP INDEX CONCURRENTLY idx_orders_customer_id;`
64 +- MySQL (InnoDB): plain `CREATE INDEX` is already online (`ALGORITHM=INPLACE`).
65 +
66 +## Batched backfill
67 +
68 +Never one giant UPDATE. Batch by primary key with a pause per batch:
69 +
70 +```sql
71 +-- Repeat until 0 rows affected (driver loop or DO block):
72 +UPDATE users
73 +SET phone_number = phone
74 +WHERE id IN (
75 + SELECT id FROM users
76 + WHERE phone_number IS NULL AND phone IS NOT NULL
77 + ORDER BY id
78 + LIMIT 5000 -- 5000 keeps each transaction < ~1s on typical rows
79 +);
80 +-- sleep 100ms between batches to let replication and vacuum breathe
81 +```
82 +
83 +Run as a repeatable script or data migration — separate from schema migrations.
84 +
85 +## Safe destructive migrations
86 +
87 +Dropping a table/column:
88 +
89 +1. Confirm zero references: grep the codebase AND check
90 + `pg_stat_user_tables.seq_scan/idx_scan` deltas over a week if possible.
91 +2. Rename first, drop later (rename is instantly reversible):
92 +
93 +```sql
94 +-- Release N: soft-drop
95 +ALTER TABLE legacy_events RENAME TO legacy_events_dropped_20260805;
96 +-- Release N+2 (weeks later): real drop, after nothing broke
97 +DROP TABLE legacy_events_dropped_20260805;
98 +```
99 +
100 +## Divergent heads
101 +
102 +Two branches each added a migration on the same parent:
103 +
104 +- Alembic: `alembic merge -m "merge heads" <rev1> <rev2>`
105 +- Rails/Prisma/Flyway: re-timestamp the UNAPPLIED migration only
106 + (`prisma migrate dev` handles resolution; Flyway: bump the `V` number).
107 +- Never renumber a migration that has been applied to any shared environment.
108 +
109 +## Gotchas
110 +
111 +- **PostgreSQL DDL is transactional; MySQL DDL is not.** A failed multi-statement
112 + MySQL migration leaves partial state — write idempotent statements
113 + (`ADD COLUMN IF NOT EXISTS`) so re-running converges.
114 +- **`CREATE INDEX CONCURRENTLY` inside a transaction fails** — migration tools
115 + wrap migrations in transactions by default; use the tool's autocommit escape.
116 +- **Default values:** PG 11+ adds `ADD COLUMN ... DEFAULT ...` instantly
117 + (metadata-only) for constant defaults; volatile defaults (`now()`, `uuid()`)
118 + still rewrite the table — add the column first, set the default after.
119 +- **Down migrations that drop data are lies** — a `down` for `DROP COLUMN`
120 + restores the column, not its contents. Say so in the migration header.
121 +- **Foreign keys on busy tables:** add as `NOT VALID`, then `VALIDATE CONSTRAINT`
122 + separately (same trick as NOT NULL).
123 +- **Renaming with views/triggers attached** — PG updates them automatically;
124 + MySQL does not for triggers referencing the old name. Check `information_schema`.
added db-skills/managing-sqlite/SKILL.md +49 −0
@@ -0,0 +1,49 @@
1 +---
2 +name: managing-sqlite
3 +description: Sets up and operates SQLite in applications — choosing when SQLite fits, WAL mode, required pragmas, fast bulk inserts, safe backups, and type-affinity pitfalls. Use when the user asks to add or configure SQLite, mentions a .db or .sqlite file, hits "database is locked" errors, needs to back up or speed up an SQLite database, or asks whether SQLite is the right choice. Do not use for server databases like PostgreSQL or MySQL, or for generic SQL query writing.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Managing SQLite
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** embedding, configuring, operating, or debugging SQLite in an application (`.db`/`.sqlite`/`.sqlite3` files).
15 +- **Do NOT use for:** PostgreSQL/MySQL administration (`administering-postgresql`) or general SQL authoring (`writing-sql-queries`).
16 +
17 +## Core rules
18 +
19 +1. **Pick SQLite deliberately.** Right when: embedded/desktop/mobile/CLI apps, local-first data, one writer at a time, read-heavy sites, datasets well under ~1TB. Wrong when: many concurrent writers, network access from multiple hosts, or you need roles/permissions — use a server database.
20 +2. **WAL mode is the default choice for any app.** Readers stop blocking the writer.
21 + - ✅ `PRAGMA journal_mode=WAL;` (persistent, set once per database)
22 + - ❌ Shipping with the default rollback journal and retro-fitting after lock errors.
23 +3. **Foreign keys are OFF by default — enable per connection.** `PRAGMA foreign_keys=ON;` on every connection open, or FK constraints silently do nothing.
24 +4. **Set `busy_timeout` on every connection** (e.g. `PRAGMA busy_timeout=5000;`) so concurrent writes wait instead of instantly failing with `SQLITE_BUSY`.
25 +5. **Wrap bulk inserts in one transaction.** One-insert-per-transaction is 100–1000× slower because each commit is an fsync.
26 + - ✅ `BEGIN; INSERT ×10 000; COMMIT;`
27 + - ❌ 10 000 autocommit inserts.
28 +6. **Never copy a live database file.** Use `VACUUM INTO 'backup.db'` (SQL, ≥3.27) or the online backup API / `.backup` in the CLI; copying mid-write yields a corrupt file, and WAL data lives in `-wal` until checkpoint.
29 +7. **Types are affinities, not constraints.** `INSERT INTO t(age) VALUES ('abc')` succeeds on an `INTEGER` column. Use `STRICT` tables (≥3.37) when types must be enforced.
30 +8. **One writer at a time — design for it.** Serialize writes through a single connection/queue in the app rather than relying on retries.
31 +
32 +## Workflow
33 +
34 +1. On every connection open, apply the pragma set: `journal_mode=WAL` (once), `foreign_keys=ON`, `busy_timeout=5000`, and `synchronous=NORMAL` for WAL databases.
35 +2. Create schema with explicit `STRICT` tables when type safety matters.
36 +3. Route all writes through one connection; use transactions for any multi-statement or bulk operation.
37 +4. Back up with `VACUUM INTO` on a schedule; test the backup by opening it and running `PRAGMA integrity_check;`.
38 +5. Validate the setup: `PRAGMA journal_mode;` returns `wal`, `PRAGMA foreign_keys;` returns `1`, and `PRAGMA integrity_check;` returns `ok`.
39 +
40 +## Edge cases & failure modes
41 +- **`database is locked`** → missing `busy_timeout`, a long-running write transaction, or two processes writing; fix in that order.
42 +- **FKs "not working"** → the connection that inserted skipped `PRAGMA foreign_keys=ON` (it is per-connection, not per-database).
43 +- **Growing `-wal` file** → no checkpoints because a reader holds a long transaction; close it or run `PRAGMA wal_checkpoint(TRUNCATE);`.
44 +- **Corrupt database** → `PRAGMA integrity_check;`, then `.recover` in the sqlite3 CLI; restore from backup if recovery fails. Never keep using a file that fails integrity_check.
45 +- **Network filesystem (NFS/SMB)** → file locking is unreliable there; keep SQLite files on local disks only.
46 +- **No dependency needed** → SQLite ships in Python's stdlib (`import sqlite3`); the CLI is preinstalled on macOS, else `brew install sqlite`.
47 +
48 +## References
49 +Connection templates, backup commands, STRICT tables, and WAL details: see [references/patterns.md](references/patterns.md).
added db-skills/managing-sqlite/references/patterns.md +125 −0
@@ -0,0 +1,125 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# SQLite — Patterns
7 +
8 +## Contents
9 +- Connection setup (Python)
10 +- Fast bulk inserts
11 +- STRICT tables and type safety
12 +- Backups
13 +- WAL maintenance
14 +- Single-writer pattern
15 +- Gotchas
16 +
17 +## Connection setup (Python)
18 +
19 +```python
20 +import sqlite3
21 +
22 +def connect(path):
23 + conn = sqlite3.connect(path, timeout=5.0) # timeout ≈ busy_timeout
24 + conn.execute("PRAGMA journal_mode=WAL") # persistent; readers don't block writer
25 + conn.execute("PRAGMA foreign_keys=ON") # per-connection, off by default
26 + conn.execute("PRAGMA busy_timeout=5000") # wait 5s on lock instead of failing
27 + conn.execute("PRAGMA synchronous=NORMAL") # safe with WAL, much faster than FULL
28 + conn.row_factory = sqlite3.Row
29 + return conn
30 +```
31 +
32 +## Fast bulk inserts
33 +
34 +```python
35 +rows = [(i, f"name-{i}") for i in range(100_000)]
36 +with connect("app.db") as conn: # context manager = one transaction
37 + conn.executemany("INSERT INTO users (id, name) VALUES (?, ?)", rows)
38 +# One COMMIT (one fsync) instead of 100k. Order-of-magnitude speedup.
39 +```
40 +
41 +## STRICT tables and type safety
42 +
43 +```sql
44 +-- Without STRICT, 'abc' inserts fine into an INTEGER column (affinity only).
45 +CREATE TABLE users (
46 + id INTEGER PRIMARY KEY, -- alias for rowid: fast, auto-increment
47 + name TEXT NOT NULL,
48 + age INTEGER CHECK (age >= 0)
49 +) STRICT; -- SQLite ≥ 3.37: types are enforced
50 +```
51 +
52 +`INTEGER PRIMARY KEY` is the rowid — use it instead of `AUTOINCREMENT`
53 +(AUTOINCREMENT adds overhead and is almost never needed).
54 +
55 +## Backups
56 +
57 +```sql
58 +-- Online, consistent, from SQL (SQLite ≥ 3.27):
59 +VACUUM INTO '/backups/app-2026-08-05.db';
60 +```
61 +
62 +```bash
63 +# CLI equivalent:
64 +sqlite3 app.db ".backup '/backups/app.db'"
65 +# Verify every backup:
66 +sqlite3 /backups/app.db "PRAGMA integrity_check;" # must print: ok
67 +```
68 +
69 +```python
70 +# Python online backup API (works while the app runs):
71 +src = sqlite3.connect("app.db")
72 +dst = sqlite3.connect("backup.db")
73 +with dst:
74 + src.backup(dst)
75 +```
76 +
77 +Never `cp app.db backup.db` while the app can write — and remember WAL means
78 +recent commits live in `app.db-wal`, not the main file.
79 +
80 +## WAL maintenance
81 +
82 +```sql
83 +PRAGMA wal_checkpoint(TRUNCATE); -- merge -wal into the db and truncate it
84 +PRAGMA journal_mode; -- confirm: wal
85 +PRAGMA wal_autocheckpoint; -- default 1000 pages (~4MB)
86 +```
87 +
88 +A `-wal` file that grows without bound means a long-lived read transaction is
89 +pinning it — find and close that reader.
90 +
91 +## Single-writer pattern
92 +
93 +```python
94 +# One writer thread owns the write connection; others enqueue.
95 +import queue, threading
96 +
97 +write_q = queue.Queue()
98 +
99 +def writer(path):
100 + conn = connect(path)
101 + while True:
102 + sql, params = write_q.get()
103 + with conn:
104 + conn.execute(sql, params)
105 +```
106 +
107 +Readers can each have their own connection — WAL lets them run concurrently
108 +with the single writer.
109 +
110 +## Gotchas
111 +
112 +- `PRAGMA foreign_keys=ON` is **per connection**. Pools/ORMs must set it in a
113 + connect hook (SQLAlchemy: `event.listens_for(engine, "connect")`).
114 +- `journal_mode=WAL` is per **database** (persistent), but `synchronous`,
115 + `busy_timeout`, `foreign_keys` are per connection.
116 +- Python's `sqlite3` opens implicit transactions around DML and holds them —
117 + pass `isolation_level=None` (autocommit) and manage `BEGIN`/`COMMIT`
118 + explicitly if lock durations surprise you.
119 +- `REAL` stores IEEE-754 doubles — money should be integer cents, not REAL.
120 +- Date/time types don't exist; store ISO-8601 TEXT or unix-epoch INTEGER and
121 + pick one convention per database.
122 +- WAL databases can't live on read-only media; use `PRAGMA query_only=ON` or
123 + rollback mode for read-only deployments.
124 +- Dropping columns needs SQLite ≥ 3.35; before that it's create-new-table,
125 + copy, rename.
added db-skills/modeling-nosql-data/SKILL.md +49 −0
@@ -0,0 +1,49 @@
1 +---
2 +name: modeling-nosql-data
3 +description: Models data for document and key-value stores — access-pattern-first design, embed vs reference decisions, controlled denormalization, Redis key design, cursor pagination, and document schema versioning. Use when the user asks to design a MongoDB or DynamoDB schema, model documents or collections, structure Redis keys, decide between embedding and referencing, or migrate a relational model to NoSQL. Do not use for relational schema design or for SQL databases with JSON columns, which follow relational rules first.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Modeling NoSQL Data
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** designing document models (MongoDB, Firestore, DynamoDB) and key-value layouts (Redis, DynamoDB keys).
15 +- **Do NOT use for:** relational schemas (`designing-database-schemas`) — and a Postgres table with a JSONB column is still relational: normalize first, JSONB for the truly schemaless remainder.
16 +
17 +## Core rules
18 +
19 +1. **Queries first, schema second.** List every access pattern (screen/endpoint → fields → frequency) BEFORE shaping documents — the inverse of relational design. A model that can't be listed against its queries isn't done.
20 +2. **Embed vs reference decision rule:**
21 + - **Embed** when data is 1:few, read together, and updated together (order + its line items).
22 + - **Reference** when 1:many grows unbounded, is shared by many parents, or updates independently (user ← posts; product ← reviews).
23 + - ✅ `{order_id, items: [{sku, qty}]}` — ❌ embedding a user's entire post history in the user document.
24 +3. **Denormalize on purpose, with ONE source of truth per fact.** Copying `author_name` into posts is fine — but document where the canonical value lives and how copies get refreshed (on write, or accept staleness with a TTL).
25 +4. **Design far below size limits.** MongoDB caps documents at 16MB; treat ~1MB as the design ceiling. Unbounded arrays are the failure smell — bucket them (one document per day/100 items) or reference.
26 +5. **Redis/KV keys: colon-namespaced, predictable, with a TTL policy.**
27 + - ✅ `user:42:session`, `cache:product:9f3e` + `EXPIRE`
28 + - ❌ `data_42_final`, keys without owner/type, cache keys with no TTL.
29 +6. **Paginate with cursors, never offsets.** Sort by an indexed, unique (or tie-broken) field and continue from the last seen value: `find({created_at: {$lt: cursor}}).limit(20)`.
30 +7. **Version documents.** Every document carries `schema_version: 3`; readers handle N and N-1, writers upgrade on write (lazy migration).
31 +8. **Model transactions around aggregates.** Put data that must change atomically in ONE document; cross-document transactions exist but are the escape hatch, not the design.
32 +
33 +## Workflow
34 +
35 +1. Write the access-pattern table: operation, fields needed, reads/sec vs writes/sec, consistency need.
36 +2. Group into aggregates (what changes together) → those become documents; everything else becomes references or copies.
37 +3. For each copy created by denormalization, record the source of truth and the refresh rule.
38 +4. Define keys/indexes per access pattern (compound indexes matching sort + filter; Redis key format + TTL per type).
39 +5. Validate: walk EVERY access pattern from step 1 against the model — each must resolve to one indexed query or one key lookup. Any pattern needing a scan or N+1 fetches → reshape and re-check.
40 +
41 +## Edge cases & failure modes
42 +- **Many-to-many** → reference both ways only if both directions are queried; otherwise store the relation on the side that's queried.
43 +- **Hot documents** (one doc absorbing all writes, e.g. a global counter) → shard the key (`counter:{0..15}`) and sum on read.
44 +- **Search across fields** → document stores are poor at ad-hoc search; pair with a search index rather than contorting the model.
45 +- **Strong consistency needed across entities** → that's the signal you may be in relational territory; say so instead of forcing it.
46 +- **Relational→NoSQL migration** → do NOT port tables 1:1 into collections; restart from access patterns (rule 1).
47 +
48 +## References
49 +Worked examples (blog, orders, sessions), bucket pattern, cursor pagination code, Redis key catalog: see [references/patterns.md](references/patterns.md).
added db-skills/modeling-nosql-data/references/patterns.md +128 −0
@@ -0,0 +1,128 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# NoSQL Data Modeling — Patterns
7 +
8 +## Contents
9 +- Access-pattern table template
10 +- Embed vs reference — worked examples
11 +- Bucket pattern for unbounded arrays
12 +- Cursor pagination
13 +- Redis key catalog
14 +- Schema versioning / lazy migration
15 +- Gotchas
16 +
17 +## Access-pattern table template
18 +
19 +| # | Operation | Fields needed | Freq | Consistency |
20 +|---|---|---|---|---|
21 +| 1 | Load order page | order, items, shipping status | 500 r/s | read-your-writes |
22 +| 2 | Add item to cart | cart items | 50 w/s | strong (same doc) |
23 +| 3 | Seller sales list | order ids, totals by seller | 5 r/s | eventual ok |
24 +
25 +Every pattern must map to one indexed query or key lookup in the final model.
26 +
27 +## Embed vs reference — worked examples
28 +
29 +```js
30 +// EMBED: order + line items (1:few, read & updated together, bounded)
31 +{
32 + _id: "ord_81",
33 + user_id: "u_42", // reference — user updates independently
34 + user_name: "Alice", // denormalized copy; truth: users collection,
35 + // refreshed: never (historical snapshot is correct here)
36 + status: "shipped",
37 + items: [ { sku: "A-1", qty: 2, unit_price_cents: 1250 } ],
38 + schema_version: 2
39 +}
40 +
41 +// REFERENCE: user ← posts (unbounded growth)
42 +{ _id: "u_42", name: "Alice", schema_version: 1 }
43 +{ _id: "p_777", author_id: "u_42", author_name: "Alice", // copy, refreshed on user rename via async job
44 + title: "...", created_at: ISODate("2026-08-01") }
45 +// Index to serve "posts by user, newest first":
46 +db.posts.createIndex({ author_id: 1, created_at: -1 })
47 +```
48 +
49 +## Bucket pattern for unbounded arrays
50 +
51 +```js
52 +// Instead of one sensor doc with an ever-growing readings[] array:
53 +{
54 + _id: "sensor_9:2026-08-05", // one bucket per sensor per day
55 + sensor_id: "sensor_9",
56 + day: "2026-08-05",
57 + count: 1440,
58 + readings: [ { t: "00:00", v: 20.1 }, /* ≤ 1440 */ ]
59 +}
60 +// Bounded documents, efficient range reads, no 16MB ceiling risk.
61 +```
62 +
63 +## Cursor pagination
64 +
65 +```js
66 +// Page 1:
67 +db.posts.find({ author_id: uid })
68 + .sort({ created_at: -1, _id: -1 }) // _id breaks timestamp ties
69 + .limit(20)
70 +// Next page — cursor = (created_at, _id) of last item:
71 +db.posts.find({
72 + author_id: uid,
73 + $or: [
74 + { created_at: { $lt: cursor.created_at } },
75 + { created_at: cursor.created_at, _id: { $lt: cursor._id } }
76 + ]
77 +}).sort({ created_at: -1, _id: -1 }).limit(20)
78 +// Offsets (skip) degrade linearly and break when rows shift — never for feeds.
79 +```
80 +
81 +## Redis key catalog
82 +
83 +Document every key family in one table; every cache key has a TTL.
84 +
85 +| Key pattern | Type | TTL | Notes |
86 +|---|---|---|---|
87 +| `user:{id}:session` | hash | 30d sliding | auth token data |
88 +| `cache:product:{id}` | string(json) | 300s | invalidate on product write |
89 +| `rate:{ip}:{minute}` | int (INCR) | 120s | rate limiting window |
90 +| `queue:emails` | list | none | worker queue (durable store elsewhere) |
91 +| `counter:orders:{0..15}` | int | none | sharded hot counter; SUM on read |
92 +
93 +```
94 +SET cache:product:9f3e '{"name":...}' EX 300
95 +INCR rate:203.0.113.7:202608051211
96 +EXPIRE rate:203.0.113.7:202608051211 120 NX
97 +```
98 +
99 +## Schema versioning / lazy migration
100 +
101 +```js
102 +// Reader handles current and previous version:
103 +function readUser(doc) {
104 + if (doc.schema_version === 1) {
105 + doc.full_name = doc.name; // v1 → v2 shape
106 + doc.schema_version = 2;
107 + }
108 + return doc;
109 +}
110 +// Writer persists upgraded shape on next write ("lazy migration").
111 +// Backfill job optional once v1 read-rate ≈ 0.
112 +```
113 +
114 +## Gotchas
115 +
116 +- MongoDB's 16MB limit includes field names — long repeated keys in big arrays
117 + waste real space; short names matter at scale.
118 +- An index on `{a: 1, b: 1}` serves filters on `a` and `a+b`, NOT `b` alone
119 + (prefix rule) — order compound indexes by equality → sort → range.
120 +- `$lookup` (joins) run on unsharded/local data paths and get slow fast — a
121 + frequent `$lookup` is a modeling smell: embed or copy instead.
122 +- DynamoDB: model EVERYTHING around partition key + sort key up front; there is
123 + no ad-hoc query escape hatch, only GSIs (each with its own cost).
124 +- Redis `KEYS pattern` blocks the server — always `SCAN` in production.
125 +- Firestore charges and limits per document read — bucket smallness matters
126 + differently: many tiny docs can cost more than fewer medium ones.
127 +- Eventual consistency of denormalized copies must be a product decision
128 + ("name may be stale ≤5 min"), never an accident.
added db-skills/optimizing-sql-performance/SKILL.md +61 −0
@@ -0,0 +1,61 @@
1 +---
2 +name: optimizing-sql-performance
3 +description: Diagnoses and fixes slow SQL queries through query plans, indexing strategy, and query rewrites. Use when the user says a query or page is slow, asks to read or interpret EXPLAIN or EXPLAIN ANALYZE output, asks which index to create, mentions N+1 queries, slow pagination, missing indexes, or asks to speed up a report or endpoint backed by SQL. Do not use for operational incidents such as locks, deadlocks, connection exhaustion, or replication lag, and not for designing new schemas from scratch — separate skills cover those.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Optimizing SQL Performance
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** slow queries, index design, query-plan analysis, N+1 elimination, pagination performance.
15 +- **Do NOT use for:** locks/deadlocks/replication incidents (`troubleshooting-databases`), new-schema design (`designing-database-schemas`), plain query authoring (`writing-sql-queries`).
16 +
17 +Default dialect: PostgreSQL.
18 +
19 +## Core rules
20 +
21 +1. **Measure first — never optimize on a hunch.** Get the real query, run `EXPLAIN (ANALYZE, BUFFERS)`, and on servers check `pg_stat_statements` for the workload's actual top offenders before touching anything.
22 +
23 +2. **Read the plan for the two classic tells:**
24 + - a **Seq Scan** on a large table filtered by a selective predicate → missing index;
25 + - a **row-estimate mismatch** (estimated 3 rows, actual 30,000) → stale statistics: run `ANALYZE table` before designing indexes around a lie.
26 +
27 +3. **Composite index column order: equality columns first, then the range/sort column.**
28 + - ✅ `CREATE INDEX ON orders (user_id, created_at)` for `WHERE user_id = $1 AND created_at >= $2`
29 + - ❌ `(created_at, user_id)` — range first makes the equality column unusable for narrowing.
30 +
31 +4. **Covering index when the same hot query still heap-fetches:** `INCLUDE` the selected columns to enable index-only scans. Reserve for measured hot paths — every index taxes writes.
32 +
33 +5. **Drop unused indexes.** Check `pg_stat_user_indexes.idx_scan = 0` over a representative period; an unused index is pure write overhead and storage.
34 +
35 +6. **Kill N+1 at the application boundary:** one query per collection, not per row.
36 + - ✅ `WHERE user_id = ANY($1)` then group in app code, or JOIN
37 + - ❌ loop issuing `SELECT … WHERE user_id = $1` per user
38 +
39 +7. **Keyset pagination for deep pages.**
40 + - ✅ `WHERE (created_at, id) < ($1, $2) ORDER BY created_at DESC, id DESC LIMIT 20`
41 + - ❌ `OFFSET 100000 LIMIT 20` — scans and discards 100k rows every page.
42 +
43 +8. **Materialized view only when it earns its keep:** an aggregation proven slow by rule 1, tolerating staleness, refreshed on a schedule (`REFRESH MATERIALIZED VIEW CONCURRENTLY`). Otherwise fix the query/index.
44 +
45 +## Workflow
46 +
47 +1. Capture the exact slow query with real parameter values.
48 +2. Baseline: `EXPLAIN (ANALYZE, BUFFERS)` — record total time and the dominant node.
49 +3. Apply ONE change (index, rewrite, `ANALYZE`) chosen from the rules above.
50 +4. Re-run the same `EXPLAIN (ANALYZE, BUFFERS)`; keep the change only if the dominant node improved and total time dropped meaningfully.
51 +5. Repeat 3–4 until the target is met; report before/after timings and every index added or dropped.
52 +
53 +## Edge cases & failure modes
54 +- **Cannot run EXPLAIN ANALYZE on prod writes** → wrap in `BEGIN; EXPLAIN ANALYZE …; ROLLBACK;`.
55 +- **Fast in psql, slow in app** → parameter-sensitive plan or connection/ORM overhead; compare with `PREPARE`/generic plan and log ORM SQL.
56 +- **Index exists but unused** → type mismatch (`text` vs `varchar` cast), function on the column (`lower(email)` needs an expression index), or the planner is right (low selectivity).
57 +- **LIKE '%term%'** → B-tree can't help; needs `pg_trgm` GIN index or full-text search.
58 +- **Everything is slow, not one query** → out of scope here; hand to `troubleshooting-databases`.
59 +
60 +## References
61 +Plan-reading walkthrough, index recipes, N+1 and pagination rewrites: see [references/patterns.md](references/patterns.md).
added db-skills/optimizing-sql-performance/references/patterns.md +145 −0
@@ -0,0 +1,145 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Optimizing SQL Performance
7 +
8 +## Contents
9 +- Reading EXPLAIN ANALYZE
10 +- Finding the workload's worst queries
11 +- Index recipes
12 +- N+1 rewrites
13 +- Keyset pagination
14 +- Materialized views
15 +- Dialect deviations (MySQL, SQLite)
16 +- Gotchas
17 +
18 +## Reading EXPLAIN ANALYZE
19 +
20 +```sql
21 +EXPLAIN (ANALYZE, BUFFERS)
22 +SELECT id, total_cents FROM orders
23 +WHERE user_id = 42 AND created_at >= '2026-01-01';
24 +```
25 +
26 +Read inner-most node first; the fix usually targets the node with the largest
27 +`actual time`. Checklist per node:
28 +
29 +| Signal | Meaning | Action |
30 +|---|---|---|
31 +| `Seq Scan` + `Rows Removed by Filter` huge | selective filter, no usable index | add index (see recipes) |
32 +| `rows=3` estimated vs `rows=30000` actual | stale statistics | `ANALYZE orders;` then re-plan |
33 +| `Sort Method: external merge Disk` | sort spills to disk | index matching ORDER BY, or raise `work_mem` for the session |
34 +| `Nested Loop` with huge outer side | join misestimate | fix stats; check join column indexes |
35 +| `Heap Fetches` high on Index Only Scan | visibility map stale | `VACUUM orders;` |
36 +
37 +## Finding the workload's worst queries
38 +
39 +```sql
40 +-- requires: CREATE EXTENSION pg_stat_statements;
41 +SELECT calls, mean_exec_time::int AS mean_ms,
42 + (calls * mean_exec_time)::int AS total_ms, query
43 +FROM pg_stat_statements
44 +ORDER BY calls * mean_exec_time DESC
45 +LIMIT 10;
46 +```
47 +
48 +Optimize by `total_ms` (aggregate cost), not by single-query time.
49 +
50 +## Index recipes
51 +
52 +```sql
53 +-- Equality-then-range composite (rule 3):
54 +CREATE INDEX orders_user_created_idx ON orders (user_id, created_at);
55 +
56 +-- Covering index for an index-only scan:
57 +CREATE INDEX orders_user_created_cov_idx
58 + ON orders (user_id, created_at) INCLUDE (total_cents);
59 +
60 +-- Expression index (query must use the same expression):
61 +CREATE INDEX users_email_lower_idx ON users (lower(email));
62 +
63 +-- Partial index for a hot subset:
64 +CREATE INDEX orders_pending_idx ON orders (created_at)
65 + WHERE status = 'pending';
66 +
67 +-- Trigram index for LIKE '%term%':
68 +CREATE EXTENSION IF NOT EXISTS pg_trgm;
69 +CREATE INDEX users_name_trgm_idx ON users USING gin (display_name gin_trgm_ops);
70 +
71 +-- Build without blocking writes (production):
72 +CREATE INDEX CONCURRENTLY ...;
73 +
74 +-- Find unused indexes:
75 +SELECT indexrelid::regclass, idx_scan
76 +FROM pg_stat_user_indexes
77 +WHERE idx_scan = 0 AND indexrelid::regclass::text NOT LIKE '%_pkey';
78 +```
79 +
80 +## N+1 rewrites
81 +
82 +```python
83 +# ❌ one query per user
84 +for uid in user_ids:
85 + cur.execute("SELECT * FROM orders WHERE user_id = %s", (uid,))
86 +
87 +# ✅ one query, group in app code
88 +cur.execute(
89 + "SELECT user_id, id, total_cents FROM orders WHERE user_id = ANY(%s)",
90 + (user_ids,))
91 +```
92 +
93 +Aggregate-per-parent variant in one round trip:
94 +
95 +```sql
96 +SELECT u.id, COALESCE(SUM(o.total_cents), 0) AS spend
97 +FROM users u
98 +LEFT JOIN orders o ON o.user_id = u.id
99 +WHERE u.id = ANY($1)
100 +GROUP BY u.id;
101 +```
102 +
103 +## Keyset pagination
104 +
105 +```sql
106 +-- page 1
107 +SELECT id, created_at FROM orders
108 +ORDER BY created_at DESC, id DESC LIMIT 20;
109 +
110 +-- next page: pass the last row's (created_at, id)
111 +SELECT id, created_at FROM orders
112 +WHERE (created_at, id) < ($1, $2)
113 +ORDER BY created_at DESC, id DESC LIMIT 20;
114 +
115 +CREATE INDEX orders_created_id_idx ON orders (created_at DESC, id DESC);
116 +```
117 +
118 +Tie-break with `id` is mandatory — `created_at` alone skips/duplicates rows on
119 +equal timestamps. Tradeoff: no random page jumps.
120 +
121 +## Materialized views
122 +
123 +```sql
124 +CREATE MATERIALIZED VIEW daily_revenue AS
125 +SELECT created_at::date AS day, SUM(total_cents) AS revenue_cents
126 +FROM orders GROUP BY created_at::date;
127 +
128 +CREATE UNIQUE INDEX daily_revenue_day_idx ON daily_revenue (day);
129 +REFRESH MATERIALIZED VIEW CONCURRENTLY daily_revenue; -- needs the unique index
130 +```
131 +
132 +## Dialect deviations (MySQL, SQLite)
133 +
134 +- MySQL: `EXPLAIN ANALYZE` (8.0.18+); no partial/expression indexes before 8.0.13 (functional indexes); use `performance_schema` instead of `pg_stat_statements`; no `INCLUDE` — add columns to the index key.
135 +- SQLite: `EXPLAIN QUERY PLAN`; run `ANALYZE;` to populate stats; partial indexes supported; no concurrent index builds.
136 +
137 +## Gotchas
138 +
139 +- `CREATE INDEX` (without `CONCURRENTLY`) takes a write lock on the table.
140 +- `EXPLAIN` without `ANALYZE` shows estimates only — plans, not reality.
141 +- Casts defeat indexes: `WHERE id::text = $1` seq-scans; cast the parameter instead.
142 +- Low-selectivity indexes (boolean flags) are usually ignored by the planner — a partial index on the rare value works.
143 +- `random_page_cost` default (4.0) is tuned for spinning disks; on SSDs the planner may wrongly prefer seq scans — typical SSD setting is 1.1.
144 +- After bulk loads, run `ANALYZE` (and `VACUUM`) before judging any plan.
145 +- ORMs hide N+1: enable SQL logging before believing "the query is slow" — often it's 500 queries.
added db-skills/securing-databases/SKILL.md +48 −0
@@ -0,0 +1,48 @@
1 +---
2 +name: securing-databases
3 +description: Hardens database security through least-privilege roles, injection-proof query patterns, secret management, TLS, encryption, and audit logging. Use when the user asks to secure a database, create database users/roles/grants, prevent SQL injection, review database credentials or connection strings, enable TLS for database connections, set up row-level security for multi-tenant data, or audit database access. Do not use for application-level authentication (sessions, JWT, OAuth) or general network firewall configuration beyond database exposure.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Securing Databases
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** roles and grants, SQL-injection defense, credential/secret handling, connection TLS, at-rest encryption, row-level security (RLS), audit logging, database network exposure.
15 +- **Do NOT use for:** app-level auth (sessions/JWT/OAuth), OS hardening, or firewall design beyond keeping the DB port private. Backup encryption lives in `backing-up-databases`.
16 +
17 +## Core rules
18 +
19 +1. **Least privilege, three roles minimum.** Owner (runs migrations, owns objects) ≠ app role (DML only) ≠ read-only role (analytics/humans). The app NEVER connects as owner or superuser.
20 + - ✅ `GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA app TO app_rw;`
21 + - ❌ `postgres://postgres:...` in the application config
22 +2. **Parameterized queries are the ONLY injection defense.** Escaping, sanitizing, or quoting by hand is always a finding.
23 + - ✅ `cur.execute("SELECT * FROM users WHERE email = %s", (email,))`
24 + - ❌ `f"SELECT * FROM users WHERE email = '{email}'"` — even for "trusted" input
25 + - Identifiers (table/column names) can't be parameters: allow-list them from a fixed set.
26 +3. **Secrets live in the environment or a secret manager — never in code, VCS, or logs.** Rotate any credential that ever hit a repository; deletion does not un-leak it.
27 +4. **TLS on every connection that leaves the host.** PostgreSQL: `sslmode=verify-full` in the DSN (not the default `prefer`, which downgrades silently).
28 +5. **The database port is never public.** Private network/VPC only; humans reach it via bastion or SSH tunnel; app reaches it via internal network. `0.0.0.0` bindings and `pg_hba.conf` `host all all 0.0.0.0/0` entries are findings.
29 +6. **Multi-tenant data gets row-level security.** RLS with a `tenant_id` policy enforced in the database beats every "remember the WHERE clause" convention.
30 +7. **Privileged access is logged.** At minimum: log DDL and role changes (`log_statement = 'ddl'`), plus `pgaudit` (or the engine's equivalent) when compliance requires read auditing.
31 +8. **Encrypt sensitive data at rest.** Default: full-disk/volume encryption; column-level (`pgcrypto`) only for fields needing protection from DB admins themselves.
32 +
33 +## Workflow
34 +
35 +1. Inventory: current roles and grants, where credentials are stored, how connections reach the DB (network path, TLS), what data is sensitive.
36 +2. Fix the highest-severity gaps in this order: public exposure (rule 5) → superuser app connections (rule 1) → injection patterns (rule 2) → plaintext secrets (rule 3) → TLS (rule 4).
37 +3. Apply RLS/audit/encryption (rules 6–8) as the data model requires.
38 +4. **Validate:** connect as the app role and confirm a privileged action FAILS (`CREATE TABLE`, `DROP TABLE`, reading another tenant's rows under RLS). Grep the codebase for string-built SQL (`f"SELECT`, `"+ sql`, `format(` near queries) and report every hit. The task is not done until both checks run.
39 +
40 +## Edge cases & failure modes
41 +- **ORM in use** → ORMs parameterize by default, but `raw()`/`text()`/`WHERE` string fragments reintroduce injection; audit those call sites specifically.
42 +- **Legacy app owns everything as one role** → migrate incrementally: create the new roles, move the app connection first, keep owner for migrations only.
43 +- **RLS and the owner role** → table owners and superusers BYPASS RLS unless the policy role is `FORCE`d; test RLS as the app role, never as owner.
44 +- **Secret already committed to git** → rotate immediately; treat history rewriting as cleanup, not remediation.
45 +- **Managed databases** → provider handles disk encryption and network; rules 1–4, 6–7 remain fully yours.
46 +
47 +## References
48 +Grants, RLS policies, TLS DSNs, and audit setup: see [references/patterns.md](references/patterns.md)
added db-skills/securing-databases/references/patterns.md +156 −0
@@ -0,0 +1,156 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Database Security Patterns — Recipes
7 +
8 +## Contents
9 +- Role setup (owner / app / read-only)
10 +- Default-deny grants for new tables
11 +- Parameterized queries by language
12 +- Safe dynamic identifiers
13 +- Row-level security for multi-tenancy
14 +- TLS connection strings
15 +- Audit logging
16 +- Gotchas
17 +
18 +## Role setup (owner / app / read-only)
19 +
20 +```sql
21 +-- PostgreSQL. Run as an admin role once per database.
22 +CREATE ROLE app_owner NOLOGIN; -- owns schema, runs migrations
23 +CREATE ROLE app_rw LOGIN PASSWORD :'pw_rw'; -- the application
24 +CREATE ROLE app_ro LOGIN PASSWORD :'pw_ro'; -- analytics / humans
25 +
26 +CREATE SCHEMA app AUTHORIZATION app_owner;
27 +
28 +GRANT USAGE ON SCHEMA app TO app_rw, app_ro;
29 +GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA app TO app_rw;
30 +GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA app TO app_rw;
31 +GRANT SELECT ON ALL TABLES IN SCHEMA app TO app_ro;
32 +```
33 +
34 +Migrations connect as a LOGIN role that is `SET ROLE app_owner` (or a login
35 +owner role); the app connects as `app_rw` only.
36 +
37 +## Default-deny grants for new tables
38 +
39 +Grants above cover EXISTING tables only. Make future tables inherit:
40 +
41 +```sql
42 +ALTER DEFAULT PRIVILEGES FOR ROLE app_owner IN SCHEMA app
43 + GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_rw;
44 +ALTER DEFAULT PRIVILEGES FOR ROLE app_owner IN SCHEMA app
45 + GRANT SELECT ON TABLES TO app_ro;
46 +ALTER DEFAULT PRIVILEGES FOR ROLE app_owner IN SCHEMA app
47 + GRANT USAGE, SELECT ON SEQUENCES TO app_rw;
48 +-- and revoke the PUBLIC default on the database itself:
49 +REVOKE ALL ON DATABASE appdb FROM PUBLIC;
50 +```
51 +
52 +## Parameterized queries by language
53 +
54 +```python
55 +# psycopg (Python)
56 +cur.execute("SELECT * FROM users WHERE email = %s AND status = %s",
57 + (email, status))
58 +```
59 +
60 +```javascript
61 +// node-postgres
62 +await pool.query("SELECT * FROM users WHERE email = $1", [email]);
63 +```
64 +
65 +```python
66 +# SQLAlchemy raw text — parameters, never f-strings
67 +conn.execute(text("SELECT * FROM users WHERE email = :email"), {"email": email})
68 +```
69 +
70 +Injection audit greps (each hit needs review):
71 +
72 +```bash
73 +grep -rnE 'f"(SELECT|INSERT|UPDATE|DELETE)' --include='*.py' .
74 +grep -rnE '"\s*\+\s*\w+\s*\+?\s*"?.*(WHERE|FROM|VALUES)' --include='*.js' .
75 +grep -rn '\.format(.*SELECT' --include='*.py' .
76 +```
77 +
78 +## Safe dynamic identifiers
79 +
80 +Parameters cannot bind table/column names. Allow-list, never interpolate input:
81 +
82 +```python
83 +SORTABLE = {"created_at", "total", "status"} # fixed set, not user-defined
84 +if sort_col not in SORTABLE:
85 + raise ValueError(f"unsortable column: {sort_col}")
86 +cur.execute(f"SELECT * FROM orders ORDER BY {sort_col} LIMIT %s", (limit,))
87 +```
88 +
89 +psycopg also offers `sql.Identifier()` for quoting — still allow-list first.
90 +
91 +## Row-level security for multi-tenancy
92 +
93 +```sql
94 +ALTER TABLE app.orders ENABLE ROW LEVEL SECURITY;
95 +ALTER TABLE app.orders FORCE ROW LEVEL SECURITY; -- applies to owner too
96 +
97 +CREATE POLICY tenant_isolation ON app.orders
98 + USING (tenant_id = current_setting('app.tenant_id')::uuid);
99 +```
100 +
101 +Per-request, after taking a connection from the pool:
102 +
103 +```sql
104 +SET app.tenant_id = '4fa2...'; -- SET LOCAL inside a transaction is safer
105 +```
106 +
107 +Use `SET LOCAL` + transaction per request so a pooled connection can never
108 +leak the previous request's tenant.
109 +
110 +## TLS connection strings
111 +
112 +```bash
113 +# verify-full = encrypt AND verify hostname against the server cert
114 +postgres://app_rw:pw@db.internal:5432/appdb?sslmode=verify-full&sslrootcert=/etc/ssl/rds-ca.pem
115 +
116 +# mysql client equivalent
117 +mysql --ssl-mode=VERIFY_IDENTITY --ssl-ca=/etc/ssl/ca.pem ...
118 +```
119 +
120 +`sslmode=require` encrypts but does NOT verify the server — MITM-able; use
121 +`verify-full` for anything crossing a network you don't own.
122 +
123 +## Audit logging
124 +
125 +```ini
126 +# postgresql.conf — minimum viable audit
127 +log_statement = 'ddl' # every schema/role change
128 +log_connections = on
129 +log_disconnections = on
130 +```
131 +
132 +```sql
133 +-- pgaudit for compliance-grade auditing
134 +CREATE EXTENSION pgaudit;
135 +ALTER SYSTEM SET pgaudit.log = 'ddl, role, write';
136 +SELECT pg_reload_conf();
137 +```
138 +
139 +Ship logs off-host (the attacker who owns the DB host owns its logs).
140 +
141 +## Gotchas
142 +
143 +- **Superusers and table owners bypass RLS** unless `FORCE ROW LEVEL SECURITY`
144 + is set — and superusers bypass it regardless. Test policies as `app_rw`.
145 +- **`GRANT ... ON ALL TABLES` is a snapshot**, not a subscription — without
146 + `ALTER DEFAULT PRIVILEGES`, every migration-created table is silently
147 + inaccessible (or worse, PUBLIC-readable).
148 +- **`sslmode=prefer` (the default) silently falls back to plaintext.**
149 +- **Connection pools + `SET app.tenant_id`** leak across requests unless you
150 + use `SET LOCAL` in a transaction or reset on checkout.
151 +- **`.pgpass`, shell history, and process lists** (`psql -c` with inline
152 + passwords, `ps` showing DSNs) are the classic secret leaks alongside VCS.
153 +- **`pg_hba.conf` `trust` entries** mean password-less login for anyone who
154 + can reach the socket — audit for them explicitly.
155 +- **Column encryption with `pgcrypto`** kills indexes on that column
156 + (equality possible via deterministic digest column; range queries are gone).
added db-skills/troubleshooting-databases/SKILL.md +57 −0
@@ -0,0 +1,57 @@
1 +---
2 +name: troubleshooting-databases
3 +description: Diagnoses live database incidents with a fixed triage runbook — connection exhaustion, lock contention and blocking queries, newly slow queries, disk and bloat, and replication lag — with the exact diagnostic query for each stage and safe kill procedures. Use when the user reports a database outage or incident, "too many connections", queries hanging or timing out, "database is slow" right now, deadlocks, or replica lag. Do not use for proactive query tuning or server configuration work.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Troubleshooting Databases
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** live incidents — the database is slow, hanging, erroring, or lagging RIGHT NOW (examples are PostgreSQL; the triage order applies to any engine).
15 +- **Do NOT use for:** proactive query tuning (`optimizing-sql-performance`) or configuration/maintenance work (`administering-postgresql`).
16 +
17 +## Prime directive
18 +
19 +**Capture evidence before changing anything.** Restarting clears the very state (`pg_stat_activity`, locks, stats) that explains the incident. Snapshot first (Workflow step 1), then act.
20 +
21 +## Triage runbook — run IN ORDER, stop at the first stage that explains the symptom
22 +
23 +1. **Connections / pool exhaustion** — symptom: "too many connections", app timeouts on connect.
24 + `SELECT state, count(*) FROM pg_stat_activity GROUP BY state;`
25 + Many `idle` → app leaks connections / missing pooler. Many `idle in transaction` → code holds transactions open; find and fix the caller, kill the worst offenders.
26 +2. **Locks / blocking queries** — symptom: queries hang but CPU is quiet.
27 + Run the blocker query (reference file) joining `pg_locks` to `pg_stat_activity`; it prints who blocks whom. Kill the ROOT blocker only.
28 +3. **Slow queries just deployed** — symptom: gradual or post-deploy slowdown.
29 + `SELECT round(total_exec_time) ms, calls, left(query,100) FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;` — compare against the last deploy; a new top entry is your suspect.
30 +4. **Disk / IO / bloat** — symptom: everything slow, writes failing.
31 + `df -h` on the data volume; dead-tuple and size queries from the reference file. Out of disk → free WAL/logs first, never delete files inside the data directory.
32 +5. **Replication lag** — symptom: stale reads on replicas.
33 + `SELECT client_addr, state, replay_lag FROM pg_stat_replication;` (primary) / `SELECT now() - pg_last_xact_replay_timestamp();` (replica). Long-running replica queries or primary write bursts are the usual causes.
34 +
35 +## Kill safely
36 +
37 +- ✅ First `SELECT pg_cancel_backend(pid);` (cancels the query, keeps the connection). Only if it doesn't die: `pg_terminate_backend(pid)`.
38 +- ❌ Never `kill -9` a Postgres backend — it forces a full crash-recovery restart of the whole server.
39 +- Record every pid you kill, with its query text, in the incident notes.
40 +
41 +## Workflow
42 +
43 +1. **Snapshot evidence:** dump `pg_stat_activity`, the blocker query output, and top `pg_stat_statements` into a timestamped file BEFORE any intervention.
44 +2. Run the triage runbook in order; stop at the first stage whose check explains the symptom.
45 +3. Apply the smallest intervention for that stage (cancel one pid, fix one caller, free disk).
46 +4. Validate recovery: symptom gone, connection states normal, no waiting locks, lag shrinking.
47 +5. Write the post-incident note: stage, root cause, evidence file, intervention, and the follow-up fix that prevents recurrence (pooler, index, code fix) — routed to the appropriate skill.
48 +
49 +## Edge cases & failure modes
50 +- **Deadlock errors** → Postgres already resolved it (one victim). Read the two queries in the error detail; the fix is consistent lock ordering in application code, not a server change.
51 +- **Can't even connect to diagnose** → connect to a different database on the instance, or use the reserved superuser slot; on managed services use the provider's performance dashboard.
52 +- **`idle in transaction` recidivism** → set `idle_in_transaction_session_timeout` (e.g. `60s`) as a guardrail after the incident.
53 +- **Everything checks out but app is slow** → the bottleneck is app-side (pool config, N+1, network) — say so explicitly rather than tuning the database blindly.
54 +- **Managed replicas lagging with no visible cause** → check for vacuum conflicts (`max_standby_streaming_delay`) and instance-class IO limits.
55 +
56 +## References
57 +Copy-paste diagnostic SQL for every stage, blocker tree query, evidence-snapshot script: see [references/patterns.md](references/patterns.md).
added db-skills/troubleshooting-databases/references/patterns.md +159 −0
@@ -0,0 +1,159 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Database Troubleshooting — Diagnostic Patterns
7 +
8 +## Contents
9 +- Stage 1: connections
10 +- Stage 2: locks and blockers
11 +- Stage 3: slow queries
12 +- Stage 4: disk, IO, bloat
13 +- Stage 5: replication lag
14 +- Safe kill procedure
15 +- Evidence snapshot script
16 +- Gotchas
17 +
18 +## Stage 1: connections
19 +
20 +```sql
21 +SELECT state, count(*) FROM pg_stat_activity GROUP BY state ORDER BY 2 DESC;
22 +
23 +-- Worst 'idle in transaction' offenders (these hold locks and block vacuum):
24 +SELECT pid, usename, application_name,
25 + now() - xact_start AS xact_age, left(query, 80) AS last_query
26 +FROM pg_stat_activity
27 +WHERE state = 'idle in transaction'
28 +ORDER BY xact_age DESC LIMIT 10;
29 +```
30 +
31 +## Stage 2: locks and blockers
32 +
33 +```sql
34 +-- Who blocks whom (root blockers have blocked_by = {}):
35 +SELECT a.pid,
36 + a.pid AS blocked_pid,
37 + pg_blocking_pids(a.pid) AS blocked_by,
38 + a.wait_event_type,
39 + now() - a.query_start AS waiting_for,
40 + left(a.query, 80) AS query
41 +FROM pg_stat_activity a
42 +WHERE cardinality(pg_blocking_pids(a.pid)) > 0
43 +ORDER BY waiting_for DESC;
44 +
45 +-- Detail on the root blocker:
46 +SELECT pid, state, now() - xact_start AS xact_age, left(query, 120) AS query
47 +FROM pg_stat_activity WHERE pid = <root_pid>;
48 +```
49 +
50 +Kill the ROOT of the tree, not the waiters — they resolve on their own.
51 +
52 +## Stage 3: slow queries
53 +
54 +```sql
55 +-- Top by cumulative time:
56 +SELECT round(total_exec_time) AS total_ms, calls,
57 + round(mean_exec_time, 1) AS mean_ms, rows,
58 + left(query, 100) AS query
59 +FROM pg_stat_statements
60 +ORDER BY total_exec_time DESC LIMIT 10;
61 +
62 +-- Reset AFTER capturing, to watch fresh accumulation during the incident:
63 +SELECT pg_stat_statements_reset();
64 +
65 +-- Currently running long queries:
66 +SELECT pid, now() - query_start AS runtime, state, left(query, 100)
67 +FROM pg_stat_activity
68 +WHERE state = 'active' AND now() - query_start > interval '30 seconds'
69 +ORDER BY runtime DESC;
70 +```
71 +
72 +## Stage 4: disk, IO, bloat
73 +
74 +```bash
75 +df -h /var/lib/postgresql # data volume
76 +du -sh /var/lib/postgresql/16/main/pg_wal # runaway WAL?
77 +```
78 +
79 +```sql
80 +-- Dead tuples (vacuum debt):
81 +SELECT relname, n_dead_tup, n_live_tup, last_autovacuum
82 +FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 10;
83 +
84 +-- Biggest relations:
85 +SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
86 +FROM pg_statio_user_tables
87 +ORDER BY pg_total_relation_size(relid) DESC LIMIT 10;
88 +```
89 +
90 +Out of disk: free space OUTSIDE the data dir first (old logs, temp dumps).
91 +Runaway `pg_wal` usually means a dead replication slot:
92 +
93 +```sql
94 +SELECT slot_name, active, pg_size_pretty(
95 + pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
96 +FROM pg_replication_slots;
97 +-- Inactive slot retaining GBs → confirm the consumer is truly gone, then:
98 +SELECT pg_drop_replication_slot('dead_slot');
99 +```
100 +
101 +## Stage 5: replication lag
102 +
103 +```sql
104 +-- On the primary:
105 +SELECT client_addr, state, sent_lsn, replay_lsn,
106 + write_lag, flush_lag, replay_lag
107 +FROM pg_stat_replication;
108 +
109 +-- On the replica:
110 +SELECT now() - pg_last_xact_replay_timestamp() AS lag;
111 +```
112 +
113 +## Safe kill procedure
114 +
115 +```sql
116 +SELECT pg_cancel_backend(<pid>); -- 1) cancel the query
117 +-- wait 5-10s; if still there:
118 +SELECT pg_terminate_backend(<pid>); -- 2) drop the connection
119 +```
120 +
121 +Never `kill -9` a backend: Postgres treats it as a crash and restarts with
122 +full recovery, turning one bad query into a full outage.
123 +
124 +## Evidence snapshot script
125 +
126 +```bash
127 +#!/bin/sh
128 +# Author: Simon-Pierre Boucher
129 +# Contact: contact@spboucher.ai
130 +# snapshot.sh — capture incident evidence before intervening.
131 +TS=$(date +%Y%m%dT%H%M%S)
132 +OUT="incident-$TS.txt"
133 +for Q in \
134 + "SELECT now()" \
135 + "SELECT state, count(*) FROM pg_stat_activity GROUP BY state" \
136 + "SELECT pid, state, now()-query_start rt, wait_event_type, left(query,120) FROM pg_stat_activity WHERE state <> 'idle' ORDER BY rt DESC" \
137 + "SELECT a.pid, pg_blocking_pids(a.pid), left(a.query,100) FROM pg_stat_activity a WHERE cardinality(pg_blocking_pids(a.pid)) > 0" \
138 + "SELECT round(total_exec_time) ms, calls, left(query,100) FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 15"
139 +do
140 + echo "== $Q" >> "$OUT"; psql -X -c "$Q" >> "$OUT" 2>&1
141 +done
142 +echo "evidence saved to $OUT"
143 +```
144 +
145 +## Gotchas
146 +
147 +- `pg_stat_activity.query` shows the LAST query for idle sessions — an
148 + `idle in transaction` session's displayed query already finished; the
149 + transaction is what's still open.
150 +- `pg_blocking_pids()` is Postgres ≥ 9.6; on older versions use the classic
151 + pg_locks self-join.
152 +- `pg_stat_statements` normalizes literals (`WHERE id = $1`) — you cannot
153 + recover the exact parameter values from it; check application logs for those.
154 +- Lock waits don't consume CPU — "server looks idle but everything hangs" is
155 + the lock-stage signature, not a reason to skip to stage 4.
156 +- On replicas, long SELECTs conflict with WAL replay; lag with an idle-looking
157 + replica often traces to one analytics query.
158 +- After ANY intervention, re-run the stage check — a killed blocker often
159 + reveals a second blocker behind it.
added db-skills/writing-sql-queries/SKILL.md +65 −0
@@ -0,0 +1,65 @@
1 +---
2 +name: writing-sql-queries
3 +description: Writes correct, readable, injection-safe SQL — explicit columns and joins, CTEs, window functions, NULL-safe predicates, parameterized queries. Use when the user asks to write, fix, or review a SQL query, SELECT/INSERT/UPDATE/DELETE statement, join, aggregation, ranking, or running total, or asks "query the database for X". Do not use for designing tables or schemas, tuning slow queries or indexes, or writing schema migrations — separate skills cover those.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing SQL Queries
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** authoring or reviewing SQL statements — selects, joins, aggregations, window functions, DML.
15 +- **Do NOT use for:** table/schema design (`designing-database-schemas`), performance tuning (`optimizing-sql-performance`), migration files (`managing-database-migrations`).
16 +
17 +Default dialect: PostgreSQL. Note deviations only when the user names another engine.
18 +
19 +## Core rules
20 +
21 +1. **Explicit column lists in production code — never `SELECT *`.**
22 + - ✅ `SELECT id, email, created_at FROM users;`
23 + - ❌ `SELECT * FROM users;` (breaks on schema change, over-fetches)
24 + `SELECT *` is fine for interactive exploration only.
25 +
26 +2. **Explicit `JOIN … ON`, never comma joins.**
27 + - ✅ `FROM orders o JOIN users u ON u.id = o.user_id`
28 + - ❌ `FROM orders o, users u WHERE u.id = o.user_id`
29 +
30 +3. **CTEs over nested subqueries once there is more than one level.**
31 + - ✅ `WITH recent AS (SELECT …) SELECT … FROM recent JOIN …`
32 + - ❌ `SELECT … FROM (SELECT … FROM (SELECT …) a) b`
33 +
34 +4. **Window functions for ranking and running totals — not self-joins.**
35 + - ✅ `ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY created_at DESC)`
36 + - ❌ correlated subquery counting rows "before this one"
37 +
38 +5. **NULL is not a value.**
39 + - ✅ `WHERE deleted_at IS NULL` / `WHERE id NOT IN (SELECT … WHERE x IS NOT NULL)` or use `NOT EXISTS`
40 + - ❌ `WHERE deleted_at = NULL` (always false) · ❌ `NOT IN` against a set containing NULL (returns no rows)
41 + Default to `NOT EXISTS` over `NOT IN` for subqueries.
42 +
43 +6. **Every non-aggregated selected column appears in `GROUP BY`.** Aggregate everything else explicitly; filter groups with `HAVING`, rows with `WHERE`.
44 +
45 +7. **Parameterized queries ALWAYS — never string interpolation.**
46 + - ✅ `cur.execute("SELECT id FROM users WHERE email = %s", (email,))`
47 + - ❌ `f"SELECT id FROM users WHERE email = '{email}'"` (SQL injection)
48 +
49 +8. **Format for review:** keywords UPPERCASE, one clause per line, short meaningful aliases (`users u`, not `users a`).
50 +
51 +## Workflow
52 +
53 +1. Restate what the query must return (columns, grain, filters) in one line.
54 +2. Write the query following the rules above.
55 +3. Validate: run it (or `EXPLAIN` it if data is unavailable) against the target engine; check the row grain with a `LIMIT 10` sample and, for aggregates, a known-total sanity check.
56 +4. If it fails or returns the wrong grain, fix and re-run before delivering.
57 +
58 +## Edge cases & failure modes
59 +- **Unknown schema** → inspect first (`\d table` / `information_schema.columns`); never guess column names.
60 +- **Dialect mismatch** (e.g. `LIMIT` vs `TOP`, `||` vs `CONCAT`) → confirm engine, adjust per notes in references.
61 +- **Division** → guard with `NULLIF(denominator, 0)`.
62 +- **Timezones** → compare `timestamptz` in UTC; never compare naive and aware timestamps.
63 +
64 +## References
65 +Copy-paste patterns and dialect notes: see [references/patterns.md](references/patterns.md).
added db-skills/writing-sql-queries/references/patterns.md +144 −0
@@ -0,0 +1,144 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Writing SQL Queries
7 +
8 +## Contents
9 +- CTE pipelines
10 +- Window functions (ranking, running totals, deduplication)
11 +- NULL-safe predicates
12 +- Upserts
13 +- Aggregation patterns
14 +- Parameterized queries by language
15 +- Dialect deviations (MySQL, SQLite)
16 +- Gotchas
17 +
18 +## CTE pipelines
19 +
20 +```sql
21 +WITH recent_orders AS (
22 + SELECT user_id, total_cents, created_at
23 + FROM orders
24 + WHERE created_at >= now() - INTERVAL '30 days'
25 +),
26 +user_totals AS (
27 + SELECT user_id, SUM(total_cents) AS spend_cents
28 + FROM recent_orders
29 + GROUP BY user_id
30 +)
31 +SELECT u.id, u.email, t.spend_cents
32 +FROM users u
33 +JOIN user_totals t ON t.user_id = u.id
34 +ORDER BY t.spend_cents DESC;
35 +```
36 +
37 +## Window functions
38 +
39 +Latest row per group (deduplication):
40 +
41 +```sql
42 +SELECT id, user_id, status
43 +FROM (
44 + SELECT o.*,
45 + ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY created_at DESC) AS rn
46 + FROM orders o
47 +) ranked
48 +WHERE rn = 1;
49 +```
50 +
51 +Running total and rank:
52 +
53 +```sql
54 +SELECT created_at::date AS day,
55 + SUM(total_cents) AS day_total,
56 + SUM(SUM(total_cents)) OVER (ORDER BY created_at::date) AS running_total,
57 + RANK() OVER (ORDER BY SUM(total_cents) DESC) AS day_rank
58 +FROM orders
59 +GROUP BY created_at::date;
60 +```
61 +
62 +`ROW_NUMBER` = no ties · `RANK` = ties skip numbers · `DENSE_RANK` = ties don't skip.
63 +
64 +## NULL-safe predicates
65 +
66 +```sql
67 +-- Anti-join: rows in users with no orders. Prefer NOT EXISTS.
68 +SELECT u.id
69 +FROM users u
70 +WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.user_id = u.id);
71 +
72 +-- NULL-safe equality (PostgreSQL):
73 +WHERE a IS NOT DISTINCT FROM b
74 +
75 +-- Safe division:
76 +SELECT paid_cents::numeric / NULLIF(total_cents, 0) AS paid_ratio;
77 +```
78 +
79 +## Upserts
80 +
81 +```sql
82 +INSERT INTO settings (user_id, key, value)
83 +VALUES ($1, $2, $3)
84 +ON CONFLICT (user_id, key)
85 +DO UPDATE SET value = EXCLUDED.value, updated_at = now();
86 +```
87 +
88 +## Aggregation patterns
89 +
90 +Conditional aggregation (pivot-lite):
91 +
92 +```sql
93 +SELECT user_id,
94 + COUNT(*) FILTER (WHERE status = 'paid') AS paid_count,
95 + COUNT(*) FILTER (WHERE status = 'refunded') AS refunded_count
96 +FROM orders
97 +GROUP BY user_id;
98 +```
99 +
100 +`HAVING` filters groups, `WHERE` filters rows — apply `WHERE` first for less work:
101 +
102 +```sql
103 +SELECT user_id, COUNT(*) AS n
104 +FROM orders
105 +WHERE created_at >= '2026-01-01'
106 +GROUP BY user_id
107 +HAVING COUNT(*) >= 5;
108 +```
109 +
110 +## Parameterized queries by language
111 +
112 +```python
113 +# psycopg (PostgreSQL)
114 +cur.execute("SELECT id FROM users WHERE email = %s", (email,))
115 +# sqlite3
116 +cur.execute("SELECT id FROM users WHERE email = ?", (email,))
117 +```
118 +
119 +```javascript
120 +// node-postgres
121 +await pool.query('SELECT id FROM users WHERE email = $1', [email]);
122 +```
123 +
124 +Identifiers (table/column names) cannot be parameters — validate them against an allowlist if dynamic.
125 +
126 +## Dialect deviations (MySQL, SQLite)
127 +
128 +| PostgreSQL | MySQL | SQLite |
129 +|---|---|---|
130 +| `ON CONFLICT … DO UPDATE` | `ON DUPLICATE KEY UPDATE` | `ON CONFLICT … DO UPDATE` (3.24+) |
131 +| `COUNT(*) FILTER (WHERE …)` | `SUM(CASE WHEN … THEN 1 ELSE 0 END)` | same as MySQL |
132 +| `now() - INTERVAL '30 days'` | `NOW() - INTERVAL 30 DAY` | `datetime('now', '-30 days')` |
133 +| `::type` cast | `CAST(x AS type)` | `CAST(x AS type)` |
134 +| `string \|\| string` | `CONCAT(a, b)` (`\|\|` needs PIPES_AS_CONCAT) | `\|\|` |
135 +
136 +## Gotchas
137 +
138 +- `NOT IN (subquery)` returns zero rows if the subquery yields any NULL — use `NOT EXISTS`.
139 +- `COUNT(col)` skips NULLs; `COUNT(*)` counts rows. Different numbers on nullable columns.
140 +- `UNION` deduplicates (and sorts on many engines); `UNION ALL` is what you usually want.
141 +- Integer division truncates in PostgreSQL: `1/2 = 0`. Cast one side to `numeric`.
142 +- `ORDER BY` in a subquery/CTE is not guaranteed to survive to the outer query — order at the outermost level.
143 +- `BETWEEN a AND b` is inclusive on both ends; for timestamp ranges use `>= a AND < b_next` to avoid double-counting boundaries.
144 +- Window functions cannot appear in `WHERE`/`HAVING` — wrap in a subquery (see deduplication pattern).
added doc-skills/README.md +42 −0
@@ -0,0 +1,42 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# doc-skills — Document-Type Skill Collection
7 +
8 +**Author:** Simon-Pierre Boucher
9 +**Contact:** contact@spboucher.ai
10 +
11 +Ten ultra-sharp skills, one per document type, each covering **create, read,
12 +and modify** with a single default tool per operation, an explicit escape
13 +hatch, a validation step, and a `references/recipes.md` for deep recipes.
14 +All skills follow the principles in [../RESEARCH-SYNTHESIS.md](../RESEARCH-SYNTHESIS.md)
15 +and pass `python3 ../tools/validate_skills.py`.
16 +
17 +## The collection and its boundaries
18 +
19 +| Skill | Handles | Explicitly does NOT handle |
20 +|---|---|---|
21 +| `processing-xlsx` | .xlsx/.xlsm/.xltx spreadsheets | .csv/.tsv → `processing-csv`; profiling |
22 +| `processing-csv` | .csv/.tsv create/read/edit/convert | quality audits → `profiling-csv-data`; Excel files |
23 +| `processing-docx` | Word .docx/.dotx | PDFs, spreadsheets, Google Docs, Markdown |
24 +| `processing-pptx` | PowerPoint .pptx/.potx | Word docs, PDFs, Google Slides |
25 +| `processing-pdf` | .pdf read/create/merge/split/forms | Word docs, images, Office→PDF export |
26 +| `processing-json` | .json/.jsonl create/read/edit/validate/query | YAML/TOML configs; API design |
27 +| `processing-yaml` | .yaml/.yml incl. configs | JSON files; CI pipeline logic |
28 +| `processing-xml` | .xml create/read/edit/validate/XPath | HTML pages; OOXML internals of Office files |
29 +| `processing-markdown` | .md create/read/edit/convert | release-notes house style; HTML files |
30 +| `processing-html` | .html create/read/extract/edit | live web fetching; XML data files |
31 +
32 +Boundaries are deliberately pairwise-exclusive so that no user request can
33 +plausibly trigger two skills at once — the #1 defense against false-positive
34 +triggering in a large collection.
35 +
36 +## Shared conventions
37 +
38 +- Frontmatter description = WHAT + "Use when …" (literal phrases, extensions) + "Do not use for …"
39 +- One default library per operation; escape hatch named with its install command
40 +- Every write is followed by a re-parse/re-open validation step
41 +- Malformed input → report the parser's error verbatim; never guess or regex-patch
42 +- `SKILL.md` <150 lines; depth lives in `references/recipes.md` (with TOC)
added doc-skills/processing-csv/SKILL.md +72 −0
@@ -0,0 +1,72 @@
1 +---
2 +name: processing-csv
3 +description: Creates, reads, modifies, and converts CSV and TSV files — filtering rows, adding or renaming columns, changing delimiters, and converting to or from JSON. Use when the user asks to read, write, edit, filter, sort, clean, split, or convert a .csv or .tsv file, mentions comma- or tab-separated data, or asks to turn CSV into JSON or JSON into CSV. Do not use for data-quality profiling or auditing a CSV (a separate profiling-csv-data skill covers that) and not for .xlsx Excel files.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Processing CSV
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** any task where a `.csv` or `.tsv` file is the input or output — creating, editing, filtering, converting.
15 +- **Do NOT use for:** profiling/auditing data quality (use `profiling-csv-data`) or `.xlsx` Excel files (use `processing-xlsx`).
16 +
17 +## Quick reference — one default per operation
18 +
19 +**Read — stdlib `csv`, always `newline=""`:**
20 +```python
21 +import csv
22 +with open("in.csv", newline="", encoding="utf-8") as f:
23 + rows = list(csv.DictReader(f)) # rows as dicts keyed by header
24 +```
25 +
26 +**Write:**
27 +```python
28 +with open("out.csv", "w", newline="", encoding="utf-8") as f:
29 + w = csv.DictWriter(f, fieldnames=["id", "name"])
30 + w.writeheader()
31 + w.writerows(rows)
32 +```
33 +
34 +**Unknown delimiter — sniff it:**
35 +```python
36 +with open("in.csv", newline="", encoding="utf-8") as f:
37 + dialect = csv.Sniffer().sniff(f.read(4096)) # 4 KB is plenty to detect the delimiter
38 + f.seek(0)
39 + rows = list(csv.DictReader(f, dialect=dialect))
40 +```
41 +
42 +**TSV:** same code with `delimiter="\t"` passed to the reader/writer.
43 +
44 +**Escape hatch — pandas** for large files (>~1 GB) or typed/numeric operations:
45 +```python
46 +import pandas as pd # pip install pandas
47 +df = pd.read_csv("in.csv", dtype=str) # dtype=str avoids silent type coercion
48 +```
49 +
50 +## Rules
51 +- Always open CSV files with `newline=""` — omitting it doubles line breaks on Windows.
52 +- The `csv` module quotes fields containing delimiters/quotes/newlines automatically — never hand-assemble CSV with string joins.
53 +- Keep the header row unless the user explicitly wants it gone.
54 +- Modify via read-all → transform → write-new (or temp file + `os.replace` to edit in place); never patch a CSV with regex.
55 +
56 +## Workflow
57 +1. Read the input with `DictReader` (sniff the dialect if the delimiter is unknown).
58 +2. Transform in Python (filter/map on the list of dicts).
59 +3. Write the result with `DictWriter`, explicit `fieldnames`.
60 +4. **Validate:** re-open the output, check the header matches `fieldnames` and the row count equals what the transform should produce. Fix and rewrite if not.
61 +5. Report output path, row count in → row count out, and columns changed.
62 +
63 +## Edge cases & failure modes
64 +- **Missing dependency** → only pandas is third-party: `pip install pandas`.
65 +- **Malformed row (wrong field count)** → `DictReader` puts extras under `None`/fills missing with `None`; count such rows, report them, and ask before dropping — don't silently discard data.
66 +- **Non-UTF-8 file** → `UnicodeDecodeError`; retry with `encoding="latin-1"` and tell the user which encoding was used.
67 +- **Huge file** → stream row-by-row (iterate the reader, write as you go) instead of `list(...)`; or use pandas with `chunksize`.
68 +- **Empty file / header-only** → report it plainly; output a header-only file if a transform was requested.
69 +- **Values with leading zeros or big IDs** (postal codes, phone numbers) → keep them as strings; that's why the pandas escape hatch uses `dtype=str`.
70 +
71 +## References
72 +Deeper recipes (filter/sort/dedupe, column ops, csv↔json, delimiter conversion, gotchas): see [references/recipes.md](references/recipes.md).
added doc-skills/processing-csv/references/recipes.md +157 −0
@@ -0,0 +1,157 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# CSV Recipes
7 +
8 +## Contents
9 +- Create
10 +- Read / filter / sort / dedupe
11 +- Modify columns
12 +- Convert (csv ↔ json, delimiter change, split/merge files)
13 +- Streaming large files
14 +- Gotchas
15 +
16 +## Create
17 +
18 +**From a list of dicts:**
19 +```python
20 +import csv
21 +rows = [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
22 +with open("out.csv", "w", newline="", encoding="utf-8") as f:
23 + w = csv.DictWriter(f, fieldnames=list(rows[0]))
24 + w.writeheader(); w.writerows(rows)
25 +```
26 +
27 +**From lists of lists (positional):**
28 +```python
29 +with open("out.csv", "w", newline="", encoding="utf-8") as f:
30 + csv.writer(f).writerows([["id", "name"], [1, "Alice"]])
31 +```
32 +
33 +## Read / filter / sort / dedupe
34 +
35 +**Filter rows:**
36 +```python
37 +with open("in.csv", newline="", encoding="utf-8") as f:
38 + rows = [r for r in csv.DictReader(f) if r["status"] == "active"]
39 +```
40 +
41 +**Sort (numeric column — cast, or you get lexicographic "10" < "9"):**
42 +```python
43 +rows.sort(key=lambda r: float(r["amount"]), reverse=True)
44 +```
45 +
46 +**Dedupe on a key, keeping first occurrence:**
47 +```python
48 +seen, unique = set(), []
49 +for r in rows:
50 + if r["email"] not in seen:
51 + seen.add(r["email"]); unique.append(r)
52 +```
53 +
54 +**Count malformed rows without crashing:**
55 +```python
56 +with open("in.csv", newline="", encoding="utf-8") as f:
57 + reader = csv.reader(f)
58 + header = next(reader)
59 + bad = sum(1 for row in reader if len(row) != len(header))
60 +```
61 +
62 +## Modify columns
63 +
64 +**Add a computed column:**
65 +```python
66 +for r in rows:
67 + r["total"] = f'{float(r["price"]) * int(r["qty"]):.2f}'
68 +fieldnames = list(rows[0]) # includes the new column
69 +```
70 +
71 +**Rename / drop columns:**
72 +```python
73 +RENAME = {"e-mail": "email"}
74 +DROP = {"internal_id"}
75 +rows = [{RENAME.get(k, k): v for k, v in r.items() if k not in DROP} for r in rows]
76 +```
77 +
78 +**In-place edit, atomically:**
79 +```python
80 +import os, tempfile
81 +fd, tmp = tempfile.mkstemp(dir=".", suffix=".csv")
82 +with os.fdopen(fd, "w", newline="", encoding="utf-8") as f:
83 + w = csv.DictWriter(f, fieldnames=fieldnames)
84 + w.writeheader(); w.writerows(rows)
85 +os.replace(tmp, "in.csv")
86 +```
87 +
88 +## Convert
89 +
90 +**CSV → JSON array:**
91 +```python
92 +import json
93 +with open("in.csv", newline="", encoding="utf-8") as f:
94 + rows = list(csv.DictReader(f))
95 +json.dump(rows, open("out.json", "w", encoding="utf-8"), indent=2, ensure_ascii=False)
96 +```
97 +
98 +**JSON array → CSV** (union of keys so ragged records don't crash `DictWriter`):
99 +```python
100 +records = json.load(open("in.json", encoding="utf-8"))
101 +fields = sorted({k for r in records for k in r})
102 +with open("out.csv", "w", newline="", encoding="utf-8") as f:
103 + w = csv.DictWriter(f, fieldnames=fields)
104 + w.writeheader(); w.writerows(records)
105 +```
106 +
107 +**CSV → TSV (or any delimiter change):**
108 +```python
109 +with open("in.csv", newline="", encoding="utf-8") as fin, \
110 + open("out.tsv", "w", newline="", encoding="utf-8") as fout:
111 + csv.writer(fout, delimiter="\t").writerows(csv.reader(fin))
112 +```
113 +
114 +**CSV ↔ Excel:** hand off to the `processing-xlsx` skill; the boundary belongs there.
115 +
116 +**Split one big CSV into N-row chunks:**
117 +```python
118 +CHUNK = 50_000 # ~50k rows keeps each part loadable in spreadsheets
119 +with open("in.csv", newline="", encoding="utf-8") as f:
120 + reader = csv.reader(f); header = next(reader)
121 + part, buf = 1, []
122 + for row in reader:
123 + buf.append(row)
124 + if len(buf) == CHUNK:
125 + with open(f"part-{part:03d}.csv", "w", newline="", encoding="utf-8") as out:
126 + w = csv.writer(out); w.writerow(header); w.writerows(buf)
127 + part, buf = part + 1, []
128 + if buf:
129 + with open(f"part-{part:03d}.csv", "w", newline="", encoding="utf-8") as out:
130 + w = csv.writer(out); w.writerow(header); w.writerows(buf)
131 +```
132 +
133 +## Streaming large files
134 +
135 +Transform row-by-row without holding the file in memory:
136 +```python
137 +with open("in.csv", newline="", encoding="utf-8") as fin, \
138 + open("out.csv", "w", newline="", encoding="utf-8") as fout:
139 + reader = csv.DictReader(fin)
140 + writer = csv.DictWriter(fout, fieldnames=reader.fieldnames)
141 + writer.writeheader()
142 + for row in reader: # one row in memory at a time
143 + if row["country"] == "CA":
144 + writer.writerow(row)
145 +```
146 +
147 +pandas alternative: `for chunk in pd.read_csv("in.csv", dtype=str, chunksize=100_000): ...`
148 +
149 +## Gotchas
150 +- **`newline=""` is not optional.** Without it the `csv` module's `\r\n` handling stacks with Python's, producing blank lines between rows on Windows.
151 +- **Never build CSV by `",".join(...)`** — a single value containing a comma, quote, or newline corrupts the file. The writer quotes correctly for free.
152 +- **Excel mangles CSVs:** strips leading zeros, converts big numbers to scientific notation, and reinterprets `1/2` as a date. Keep identifier-like columns as strings and warn users who round-trip through Excel.
153 +- **Excel needs a BOM to detect UTF-8:** if the file is destined for Excel, write with `encoding="utf-8-sig"`.
154 +- **`csv.Sniffer` can misfire on single-column files** or quoted samples — wrap in try/except and fall back to `,`.
155 +- **Sorting strings numerically** gives `"10" < "9"`; cast before sorting.
156 +- **`DictReader` with duplicate headers** silently keeps only the last column of that name — check `reader.fieldnames` for dupes when auditing unknown files.
157 +- **Line numbers vs row numbers** differ when fields contain embedded newlines; use `reader.line_num` for error reporting, not your own counter.
added doc-skills/processing-docx/SKILL.md +70 −0
@@ -0,0 +1,70 @@
1 +---
2 +name: processing-docx
3 +description: Creates, reads, and modifies Word documents (.docx, .dotx) with python-docx — paragraphs, headings, tables, images, styles — and reads them as Markdown via pandoc. Use when the user asks to create, open, read, edit, or fix a Word document, mentions .docx/.dotx files, or wants a report, memo, letter, or template in Word format. Do not use for PDFs, spreadsheets, Google Docs, or Markdown files.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Processing DOCX
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** creating, reading, or modifying `.docx` and `.dotx` files — text, headings, tables, images, styles.
15 +- **Do NOT use for:** PDFs, spreadsheets, Google Docs (different API), or plain Markdown/text files.
16 +
17 +## Quick reference
18 +
19 +Default library: **python-docx**. Reading: **pandoc** first, python-docx as fallback. Escape hatch: unzip + edit `word/document.xml` + rezip for what python-docx can't do (tracked changes, comments).
20 +
21 +**Create:**
22 +```python
23 +from docx import Document
24 +doc = Document()
25 +doc.add_heading("Quarterly Report", level=1)
26 +doc.add_paragraph("Revenue grew 12% quarter over quarter.")
27 +t = doc.add_table(rows=2, cols=2)
28 +t.style = "Table Grid"
29 +t.rows[0].cells[0].text = "Region"
30 +doc.save("report.docx")
31 +```
32 +
33 +**Read (pandoc default, python-docx fallback):**
34 +```bash
35 +pandoc -t markdown report.docx -o report.md
36 +```
37 +```python
38 +# fallback if pandoc is missing
39 +from docx import Document
40 +text = "\n".join(p.text for p in Document("report.docx").paragraphs)
41 +```
42 +
43 +**Modify:**
44 +```python
45 +from docx import Document
46 +doc = Document("report.docx")
47 +for p in doc.paragraphs:
48 + if "12%" in p.text:
49 + for run in p.runs:
50 + run.text = run.text.replace("12%", "14%")
51 +doc.save("report.docx")
52 +```
53 +
54 +## Workflow
55 +1. Classify the task: create / read / modify.
56 +2. Read: try `pandoc -t markdown file.docx`; if pandoc is not installed, fall back to python-docx text extraction (note: fallback loses images and most formatting fidelity).
57 +3. Create/modify with python-docx. When editing, reuse the document's existing styles (`doc.styles`) instead of hardcoding fonts.
58 +4. For tracked changes or comments, python-docx cannot help: unzip the `.docx`, edit `word/document.xml`, rezip with the original file layout (see recipes).
59 +5. Validate: re-open the saved file with `Document(path)` — if it raises, fix before delivering. Confirm expected paragraph/table counts.
60 +6. Report the output path and what changed.
61 +
62 +## Edge cases & failure modes
63 +- **python-docx missing** → `pip install python-docx`. **pandoc missing** → `brew install pandoc` (macOS) / `apt-get install pandoc`; or use the python-docx fallback.
64 +- **Corrupt / not a zip** → `Document()` raises `PackageNotFoundError`; report the file is not a valid docx, stop.
65 +- **Password-protected document** → python-docx cannot decrypt; ask the user for an unprotected copy.
66 +- **`.doc` (legacy binary)** → not supported by python-docx; convert first: `soffice --headless --convert-to docx file.doc`.
67 +- **Large documents (hundreds of pages)** → python-docx loads the whole XML tree; fine to ~10k paragraphs, but prefer targeted XML edits for bulk find-and-replace across huge files.
68 +
69 +## References
70 +Deeper recipes (styles, images, headers/footers, find-and-replace across runs, XML escape hatch, conversion, gotchas): see [references/recipes.md](references/recipes.md).
added doc-skills/processing-docx/references/recipes.md +161 −0
@@ -0,0 +1,161 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# DOCX Recipes — python-docx (+ pandoc, raw XML)
7 +
8 +## Contents
9 +- Create with formatting (styles, fonts, page setup, images, headers/footers)
10 +- Tables
11 +- Read / extract (text, tables, structure)
12 +- Modify (find-and-replace across runs, insert/delete paragraphs)
13 +- Raw-XML escape hatch (tracked changes, comments)
14 +- Convert (docx ↔ markdown/pdf)
15 +- Gotchas
16 +
17 +## Create with formatting
18 +
19 +```python
20 +from docx import Document
21 +from docx.shared import Pt, Inches, RGBColor
22 +from docx.enum.text import WD_ALIGN_PARAGRAPH
23 +
24 +doc = Document()
25 +
26 +# Page setup — US Letter (python-docx defaults to the template's size)
27 +section = doc.sections[0]
28 +section.page_width, section.page_height = Inches(8.5), Inches(11)
29 +section.left_margin = section.right_margin = Inches(1)
30 +
31 +# Built-in styles: use them so a table of contents works
32 +doc.add_heading("Title of Report", level=0) # style "Title"
33 +doc.add_heading("Introduction", level=1) # style "Heading 1"
34 +
35 +p = doc.add_paragraph("Body text with ")
36 +run = p.add_run("bold emphasis")
37 +run.bold = True
38 +run.font.size = Pt(11)
39 +run.font.color.rgb = RGBColor(0x44, 0x72, 0xC4)
40 +p.alignment = WD_ALIGN_PARAGRAPH.JUSTIFY
41 +
42 +# Bullets and numbers come from styles, never literal characters
43 +doc.add_paragraph("First point", style="List Bullet")
44 +doc.add_paragraph("Step one", style="List Number")
45 +
46 +# Image, sized by width (height scales proportionally)
47 +doc.add_picture("chart.png", width=Inches(5))
48 +
49 +# Header/footer
50 +doc.sections[0].header.paragraphs[0].text = "Confidential"
51 +doc.sections[0].footer.paragraphs[0].text = "Page footer"
52 +
53 +doc.save("report.docx")
54 +```
55 +
56 +## Tables
57 +
58 +```python
59 +table = doc.add_table(rows=1, cols=3)
60 +table.style = "Table Grid" # built-in style name
61 +hdr = table.rows[0].cells
62 +for i, h in enumerate(["Region", "Q1", "Q2"]):
63 + hdr[i].text = h
64 + hdr[i].paragraphs[0].runs[0].bold = True
65 +for region, q1, q2 in [("East", "100", "150")]:
66 + row = table.add_row().cells
67 + row[0].text, row[1].text, row[2].text = region, q1, q2
68 +
69 +# Merge cells
70 +a = table.cell(0, 0); b = table.cell(0, 1)
71 +merged = a.merge(b)
72 +```
73 +
74 +## Read / extract
75 +
76 +```python
77 +from docx import Document
78 +doc = Document("report.docx")
79 +
80 +# All body text in order (paragraphs only — table text is separate)
81 +text = "\n".join(p.text for p in doc.paragraphs)
82 +
83 +# Tables → list of rows
84 +tables = [[[cell.text for cell in row.cells] for row in t.rows] for t in doc.tables]
85 +
86 +# Structure: headings with levels
87 +outline = [(p.style.name, p.text) for p in doc.paragraphs if p.style.name.startswith("Heading")]
88 +```
89 +
90 +Full-fidelity read: `pandoc -t markdown report.docx` (keeps headings, lists, tables, links).
91 +
92 +## Modify
93 +
94 +**Find-and-replace — the run-splitting problem.** Word splits a paragraph's text into runs at arbitrary points, so a target string often spans runs. Safe pattern: operate at paragraph level, rebuild runs only when the paragraph actually matches.
95 +
96 +```python
97 +def replace_in_paragraph(p, old, new):
98 + if old not in p.text:
99 + return
100 + # Concatenate, replace, put everything in the first run, empty the rest.
101 + # Trade-off: intra-paragraph formatting collapses to the first run's format.
102 + full = p.text.replace(old, new)
103 + for run in p.runs:
104 + run.text = ""
105 + if p.runs:
106 + p.runs[0].text = full
107 + else:
108 + p.add_run(full)
109 +
110 +doc = Document("report.docx")
111 +for p in doc.paragraphs:
112 + replace_in_paragraph(p, "FY2025", "FY2026")
113 +for t in doc.tables:
114 + for row in t.rows:
115 + for cell in row.cells:
116 + for p in cell.paragraphs:
117 + replace_in_paragraph(p, "FY2025", "FY2026")
118 +doc.save("report.docx")
119 +```
120 +
121 +**Insert/delete paragraphs:**
122 +```python
123 +# Insert before an existing paragraph
124 +target = doc.paragraphs[3]
125 +new_p = target.insert_paragraph_before("Inserted text", style="Normal")
126 +
127 +# Delete: python-docx has no API — remove the XML element
128 +p = doc.paragraphs[5]
129 +p._element.getparent().remove(p._element)
130 +```
131 +
132 +## Raw-XML escape hatch
133 +
134 +For tracked changes (`w:ins`/`w:del`), comments, or anything python-docx lacks:
135 +
136 +```bash
137 +mkdir unpacked && cd unpacked && unzip -o ../report.docx
138 +# edit word/document.xml (and word/comments.xml for comments)
139 +zip -r ../report-edited.docx . -x '.*' # zip from inside so paths stay relative
140 +```
141 +
142 +Accept all tracked changes = keep `w:ins` content (strip the wrapper tag), delete `w:del` elements entirely. Validate the result opens: `python3 -c "from docx import Document; Document('report-edited.docx')"`.
143 +
144 +## Convert
145 +
146 +```bash
147 +pandoc report.md -o report.docx # markdown → docx
148 +pandoc -t markdown report.docx -o report.md # docx → markdown
149 +soffice --headless --convert-to pdf report.docx # docx → pdf
150 +soffice --headless --convert-to docx legacy.doc # .doc → .docx
151 +```
152 +
153 +## Gotchas
154 +
155 +- **python-docx cannot read or write tracked changes, comments, or fields** (page numbers, TOC field codes) — use the XML escape hatch.
156 +- **A TOC inserted programmatically shows empty until Word/LibreOffice refreshes fields**; headings must use built-in `Heading N` styles for it to populate.
157 +- **Runs split unpredictably** — never assume one run per paragraph; see the find-and-replace pattern above.
158 +- **`doc.paragraphs` skips text inside tables, headers, footers, and text boxes** — iterate those containers separately.
159 +- **New documents inherit the bundled default template** (Calibri, A4 in some builds); set page size and margins explicitly when layout matters.
160 +- **Style names are English built-ins** ("Heading 1", "Table Grid") regardless of Word's UI language; a missing custom style raises `KeyError` on use.
161 +- **`.dotx` templates:** open normally, but save as `.docx` unless the user wants a template back.
added doc-skills/processing-html/SKILL.md +76 −0
@@ -0,0 +1,76 @@
1 +---
2 +name: processing-html
3 +description: Creates, reads, and modifies local HTML files — extracting text, tables, and links, or editing markup with BeautifulSoup. Use when the user asks to parse, scrape data out of, edit, clean up, or generate an .html or .htm file, extract a table or links from saved HTML, or build a static HTML page. Do not use for fetching live web pages (that is web browsing/scraping) or for XML data files.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Processing HTML
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** local `.html`/`.htm` files — extracting text, tables, or links; targeted markup edits; generating static pages from scratch.
15 +- **Do NOT use for:** fetching live web pages (that is web browsing/scraping — get the file first, then this skill applies) or XML data files (use the XML skill — XML parsers are strict, HTML parsers are tolerant).
16 +
17 +## Quick reference
18 +
19 +**Default:** BeautifulSoup4 with the built-in `html.parser` (`pip install beautifulsoup4`). **Escape hatch:** the `lxml` parser (`pip install lxml`) for large or badly malformed files — faster and more lenient.
20 +
21 +```python
22 +from bs4 import BeautifulSoup
23 +
24 +soup = BeautifulSoup(open("page.html", encoding="utf-8"), "html.parser")
25 +
26 +# Read / extract
27 +title = soup.title.get_text(strip=True) if soup.title else ""
28 +links = [(a.get_text(strip=True), a["href"]) for a in soup.find_all("a", href=True)]
29 +rows = [[c.get_text(strip=True) for c in tr.find_all(["td", "th"])]
30 + for tr in soup.select("table tr")] # lists-of-lists
31 +
32 +# Modify — targeted, leave everything else untouched
33 +for img in soup.find_all("img", src=True):
34 + if not img.get("alt"):
35 + img["alt"] = ""
36 +open("page.html", "w", encoding="utf-8").write(str(soup))
37 +```
38 +
39 +**Create** — write HTML5 directly, no library:
40 +
41 +```html
42 +<!DOCTYPE html>
43 +<html lang="en">
44 +<head>
45 + <meta charset="utf-8">
46 + <meta name="viewport" content="width=device-width, initial-scale=1">
47 + <title>Page title</title>
48 +</head>
49 +<body>
50 + <main>…</main>
51 +</body>
52 +</html>
53 +```
54 +
55 +## Rules
56 +- **Never regex-parse HTML.** Use the soup — selectors and tree navigation, always.
57 +- Edit surgically: modify the matched tags only; do not re-indent or `prettify()` an existing file (it rewrites every text node's whitespace).
58 +- New pages are HTML5: doctype, `<meta charset="utf-8">`, semantic tags (`main`, `nav`, `article`), `lang` attribute.
59 +- Extract tables as lists-of-lists; flag `rowspan`/`colspan` cells instead of silently mis-aligning columns.
60 +
61 +## Workflow
62 +1. Identify the operation: extract / modify / create.
63 +2. Parse with `html.parser`; if the file is huge (>5 MB) or the tree looks wrong (missing siblings, truncated body), reparse with `"lxml"`.
64 +3. Perform the operation with the narrowest selector that matches (recipes in references/recipes.md).
65 +4. Write back with the file's original encoding.
66 +5. **Validate:** re-parse the written file and confirm the edit landed (query the changed element) and the element count of untouched regions is unchanged. Fix and repeat until clean.
67 +
68 +## Edge cases & failure modes
69 +- **bs4/lxml missing** → `pip install beautifulsoup4` / `pip install lxml`.
70 +- **Malformed HTML** → parsers auto-repair rather than error; if extraction returns nothing, the repaired tree may differ from the source — try the `"lxml"` parser and inspect `soup.prettify()[:2000]` to see the actual tree before concluding data is absent.
71 +- **Encoding** → check `<meta charset>` before assuming UTF-8; on `UnicodeDecodeError`, reopen with that charset or pass raw bytes to BeautifulSoup and let it detect.
72 +- **JavaScript-rendered content** → if the data isn't in the file, it never was: say so — no parser will find DOM built at runtime.
73 +- **Fragments** (no `<html>`/`<body>`) → parse and write back as-is; do not let output gain wrapper tags the input lacked (html.parser doesn't add them; lxml does).
74 +
75 +## References
76 +Deeper copy-paste recipes (selectors, table→CSV, rewriting links, sanitizing): see [references/recipes.md](references/recipes.md).
added doc-skills/processing-html/references/recipes.md +136 −0
@@ -0,0 +1,136 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# HTML Recipes
7 +
8 +## Contents
9 +- Selecting elements (CSS selectors vs find_all)
10 +- Extract: tables → CSV, links, clean text
11 +- Modify: rewrite links, insert/remove/replace elements
12 +- Sanitize: strip scripts and inline handlers
13 +- Convert HTML → Markdown
14 +- Gotchas (parser differences, encoding, whitespace)
15 +
16 +## Selecting elements (CSS selectors vs find_all)
17 +
18 +```python
19 +from bs4 import BeautifulSoup
20 +soup = BeautifulSoup(open("page.html", encoding="utf-8"), "html.parser")
21 +
22 +soup.select("div.card > h2") # CSS — best for structural paths
23 +soup.select_one("#main table") # first match or None
24 +soup.find_all("a", href=True) # find_all — best for attribute filters
25 +soup.find_all("img", alt=False) # images missing alt
26 +soup.find("h2", string="Pricing") # exact text match
27 +```
28 +
29 +`select()` covers most CSS: descendants, `>`, `[attr=val]`, `:nth-of-type()`. It does not run JavaScript-era pseudo-classes like `:visible`.
30 +
31 +## Extract: tables → CSV, links, clean text
32 +
33 +Table to CSV, with span detection:
34 +
35 +```python
36 +import csv
37 +
38 +def table_to_csv(table, out_path):
39 + if table.find(attrs={"rowspan": True}) or table.find(attrs={"colspan": True}):
40 + print("warning: table uses rowspan/colspan; columns may misalign")
41 + rows = [[c.get_text(strip=True) for c in tr.find_all(["td", "th"])]
42 + for tr in table.find_all("tr")]
43 + with open(out_path, "w", newline="", encoding="utf-8") as f:
44 + csv.writer(f).writerows(rows)
45 +
46 +for i, table in enumerate(soup.find_all("table")):
47 + table_to_csv(table, f"table_{i}.csv")
48 +```
49 +
50 +Links with absolute resolution against a known base:
51 +
52 +```python
53 +from urllib.parse import urljoin
54 +base = "https://example.com/docs/"
55 +links = [(a.get_text(strip=True), urljoin(base, a["href"]))
56 + for a in soup.find_all("a", href=True)]
57 +```
58 +
59 +Readable text without script/style noise:
60 +
61 +```python
62 +for tag in soup(["script", "style", "noscript"]):
63 + tag.decompose()
64 +text = soup.get_text(separator="\n", strip=True)
65 +```
66 +
67 +## Modify: rewrite links, insert/remove/replace elements
68 +
69 +```python
70 +# Rewrite links (http → https, or path migration)
71 +for a in soup.find_all("a", href=True):
72 + if a["href"].startswith("http://example.com"):
73 + a["href"] = a["href"].replace("http://", "https://", 1)
74 +
75 +# Insert: add a class, append a child, insert a sibling
76 +div = soup.select_one("div.content")
77 +div["class"] = div.get("class", []) + ["highlight"]
78 +new_p = soup.new_tag("p")
79 +new_p.string = "Appended paragraph."
80 +div.append(new_p)
81 +div.insert_after(soup.new_tag("hr"))
82 +
83 +# Remove vs unwrap
84 +soup.select_one("aside.ad").decompose() # delete element and children
85 +for span in soup.find_all("span", class_="tracking"):
86 + span.unwrap() # keep children, drop the tag
87 +
88 +# Replace
89 +old = soup.select_one("center")
90 +new = soup.new_tag("div", attrs={"style": "text-align:center"})
91 +new.extend(list(old.contents))
92 +old.replace_with(new)
93 +
94 +open("page.html", "w", encoding="utf-8").write(str(soup))
95 +```
96 +
97 +## Sanitize: strip scripts and inline handlers
98 +
99 +For untrusted HTML that will be displayed, remove active content:
100 +
101 +```python
102 +for tag in soup(["script", "iframe", "object", "embed"]):
103 + tag.decompose()
104 +for tag in soup.find_all(True):
105 + for attr in list(tag.attrs):
106 + if attr.lower().startswith("on"): # onclick, onload, ...
107 + del tag[attr]
108 + if tag.get("href", "").lstrip().lower().startswith("javascript:"):
109 + del tag["href"]
110 +```
111 +
112 +(For production-grade sanitizing use the `bleach` library — this covers the common cases.)
113 +
114 +## Convert HTML → Markdown
115 +
116 +```bash
117 +pandoc page.html -t gfm -o page.md # brew install pandoc
118 +```
119 +
120 +Or in Python, `pip install markdownify`:
121 +
122 +```python
123 +from markdownify import markdownify
124 +md = markdownify(open("page.html", encoding="utf-8").read(), heading_style="ATX")
125 +```
126 +
127 +## Gotchas (parser differences, encoding, whitespace)
128 +
129 +- **Parsers repair differently.** `html.parser` leaves fragments bare; `lxml` wraps them in `<html><body>`; `html5lib` (`pip install html5lib`) repairs exactly like a browser but is slow. If output gained wrapper tags the input lacked, you switched parsers mid-task.
130 +- **Misnested tags relocate.** A `<table>` with stray `</div>` inside may have its rows moved outside the table in the repaired tree — "the selector finds nothing" often means the repair changed the structure, not that the data is missing. Inspect `soup.prettify()` on a slice.
131 +- **`str(soup)` vs `soup.prettify()`.** `str()` preserves the whitespace as parsed; `prettify()` re-indents every node and inserts newlines *inside text*, which can change rendering (whitespace matters around inline elements). Never prettify an existing file.
132 +- **Encoding detection.** BeautifulSoup guesses from `<meta charset>` and byte patterns when given bytes; given a mis-decoded *string*, it can't fix it. When in doubt, pass `open(path, "rb")`.
133 +- **`.string` vs `.get_text()`.** `.string` is `None` when a tag has multiple children; `.get_text()` always returns the concatenated text. Use `.get_text(strip=True)` for extraction.
134 +- **Attribute multi-values.** `class` comes back as a *list* (`["btn", "large"]`); `id` as a string. Appending a class means list-append, not string concat.
135 +- **Entities are decoded on parse** (`&amp;` → `&`) and re-encoded minimally on output; byte-identical round-trips of untouched regions are not guaranteed, only semantically identical ones — diff renders, not bytes.
136 +- **Comments** are `Comment` nodes, invisible to `get_text()`; find them with `soup.find_all(string=lambda s: isinstance(s, Comment))` (`from bs4 import Comment`).
added doc-skills/processing-json/SKILL.md +74 −0
@@ -0,0 +1,74 @@
1 +---
2 +name: processing-json
3 +description: Creates, reads, modifies, validates, and queries JSON and JSON Lines files. Use when the user asks to read, write, parse, edit, update, merge, validate, pretty-print, or query a .json or .jsonl file, mentions JSON data or JSON Lines, or asks to fix invalid JSON. Do not use for YAML/TOML config files or for designing APIs.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Processing JSON
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** any task where a `.json` or `.jsonl` file is the input or output — creating, parsing, editing, validating, querying.
15 +- **Do NOT use for:** YAML/TOML config files or API design discussions.
16 +
17 +## Quick reference — one default per operation
18 +
19 +**Read / write — Python stdlib `json`:**
20 +```python
21 +import json
22 +with open("data.json", encoding="utf-8") as f:
23 + data = json.load(f) # load() IS the syntax validator
24 +
25 +with open("data.json", "w", encoding="utf-8") as f:
26 + json.dump(data, f, indent=2, ensure_ascii=False)
27 +```
28 +
29 +**JSON Lines — one object per line, never json.load the whole file:**
30 +```python
31 +records = [json.loads(line) for line in open("data.jsonl", encoding="utf-8") if line.strip()]
32 +with open("out.jsonl", "w", encoding="utf-8") as f:
33 + for r in records:
34 + f.write(json.dumps(r, ensure_ascii=False) + "\n")
35 +```
36 +
37 +**Modify — always atomically** (temp file + rename; a crash mid-write can't corrupt the original):
38 +```python
39 +import json, os, tempfile
40 +with open("data.json", encoding="utf-8") as f:
41 + data = json.load(f)
42 +data["version"] = 2
43 +fd, tmp = tempfile.mkstemp(dir=os.path.dirname(os.path.abspath("data.json")))
44 +with os.fdopen(fd, "w", encoding="utf-8") as f:
45 + json.dump(data, f, indent=2, ensure_ascii=False)
46 +os.replace(tmp, "data.json")
47 +```
48 +
49 +**Query — escape hatch for large files:** `jq '.items[] | select(.active)' big.json` (requires jq installed: `brew install jq`). For everything else, load and filter in Python.
50 +
51 +**Schema validation:** `pip install jsonschema`, then `jsonschema.validate(instance=data, schema=schema)`.
52 +
53 +## Rules
54 +- **Never edit JSON with regex or string replacement.** Parse → mutate → dump. Always.
55 +- Preserve key order: Python dicts keep insertion order; do not pass `sort_keys=True` unless asked.
56 +- `indent=2, ensure_ascii=False` for human-facing files; single-line compact only for machine-to-machine output.
57 +
58 +## Workflow
59 +1. Load the input (`json.load` / line-by-line for `.jsonl`). A parse error here is a finding, not a failure — see edge cases.
60 +2. Apply the change/query in Python on the parsed structure.
61 +3. Write atomically (recipe above).
62 +4. **Validate:** re-open and `json.load` the written file; for `.jsonl`, re-parse every line. Only then report success.
63 +5. Report the output path and what changed (keys touched, records added/removed).
64 +
65 +## Edge cases & failure modes
66 +- **Malformed JSON** → `json.JSONDecodeError` includes line and column; report it verbatim (e.g. "Expecting ',' delimiter: line 12 column 3") and show the offending line. Common causes: trailing commas, single quotes, comments — fix precisely, don't guess.
67 +- **`NaN`/`Infinity` in input** → stdlib accepts them but they are NOT valid JSON; re-emit with `json.dump(..., allow_nan=False)` after replacing them with `null` (confirm with the user).
68 +- **Missing dependency** → only third-party need: `pip install jsonschema` (schema validation) or `brew install jq`.
69 +- **Huge file (>500 MB)** → if it's `.jsonl`, stream line-by-line; if a single JSON document, use `jq` rather than loading into Python.
70 +- **Empty file** → report "file is empty — not valid JSON (an empty JSON file should contain `{}` or `[]`)" and ask which the user wants.
71 +- **Duplicate keys** → `json.load` silently keeps the last one; when auditing, parse with `object_pairs_hook=list` to detect them.
72 +
73 +## References
74 +Deeper recipes (merging, diffing, flattening, jsonl↔json, encoding traps): see [references/recipes.md](references/recipes.md).
added doc-skills/processing-json/references/recipes.md +154 −0
@@ -0,0 +1,154 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# JSON Recipes
7 +
8 +## Contents
9 +- Create
10 +- Read / query
11 +- Modify
12 +- Convert (json ↔ jsonl, json → CSV)
13 +- Validate
14 +- Gotchas
15 +
16 +## Create
17 +
18 +**New file with non-ASCII content kept readable:**
19 +```python
20 +import json
21 +data = {"name": "Café Müller", "items": [1, 2, 3]}
22 +with open("out.json", "w", encoding="utf-8") as f:
23 + json.dump(data, f, indent=2, ensure_ascii=False)
24 + f.write("\n") # trailing newline: POSIX-friendly diffs
25 +```
26 +
27 +**Compact machine-to-machine output:**
28 +```python
29 +json.dumps(data, separators=(",", ":"), ensure_ascii=False)
30 +```
31 +
32 +## Read / query
33 +
34 +**Safe nested access:**
35 +```python
36 +value = data.get("config", {}).get("db", {}).get("host") # None if any level missing
37 +```
38 +
39 +**Filter a list of records:**
40 +```python
41 +active = [r for r in data["users"] if r.get("active")]
42 +```
43 +
44 +**jq equivalents for large files (jq must be installed):**
45 +```bash
46 +jq '.users[] | select(.active) | .email' big.json # filter + project
47 +jq 'length' big.json # count
48 +jq -r '.items[].id' big.json # raw strings, no quotes
49 +```
50 +
51 +**Detect duplicate keys while parsing:**
52 +```python
53 +def no_dupes(pairs):
54 + keys = [k for k, _ in pairs]
55 + dupes = {k for k in keys if keys.count(k) > 1}
56 + if dupes:
57 + raise ValueError(f"duplicate keys: {sorted(dupes)}")
58 + return dict(pairs)
59 +
60 +data = json.load(open("in.json", encoding="utf-8"), object_pairs_hook=no_dupes)
61 +```
62 +
63 +## Modify
64 +
65 +**Atomic in-place update (the only sanctioned write pattern for existing files):**
66 +```python
67 +import json, os, tempfile
68 +
69 +def update_json(path, mutate):
70 + with open(path, encoding="utf-8") as f:
71 + data = json.load(f)
72 + mutate(data)
73 + fd, tmp = tempfile.mkstemp(dir=os.path.dirname(os.path.abspath(path)))
74 + with os.fdopen(fd, "w", encoding="utf-8") as f:
75 + json.dump(data, f, indent=2, ensure_ascii=False)
76 + f.write("\n")
77 + os.replace(tmp, path) # atomic on POSIX
78 +
79 +update_json("config.json", lambda d: d.setdefault("features", {}).update(dark_mode=True))
80 +```
81 +
82 +**Deep merge (dict-over-dict, right side wins):**
83 +```python
84 +def deep_merge(base, override):
85 + out = dict(base)
86 + for k, v in override.items():
87 + out[k] = deep_merge(out[k], v) if isinstance(out.get(k), dict) and isinstance(v, dict) else v
88 + return out
89 +```
90 +
91 +**Shallow diff of two objects:**
92 +```python
93 +def diff(a, b):
94 + keys = a.keys() | b.keys()
95 + return {k: (a.get(k), b.get(k)) for k in keys if a.get(k) != b.get(k)}
96 +```
97 +
98 +## Convert
99 +
100 +**jsonl → json array:**
101 +```python
102 +records = [json.loads(l) for l in open("in.jsonl", encoding="utf-8") if l.strip()]
103 +json.dump(records, open("out.json", "w", encoding="utf-8"), indent=2, ensure_ascii=False)
104 +```
105 +
106 +**json array → jsonl (streams better, appends safely):**
107 +```python
108 +with open("out.jsonl", "w", encoding="utf-8") as f:
109 + for r in json.load(open("in.json", encoding="utf-8")):
110 + f.write(json.dumps(r, ensure_ascii=False) + "\n")
111 +```
112 +
113 +**Flat records → CSV** (nested values must be flattened or stringified first):
114 +```python
115 +import csv
116 +rows = json.load(open("in.json", encoding="utf-8"))
117 +fields = sorted({k for r in rows for k in r})
118 +with open("out.csv", "w", newline="", encoding="utf-8") as f:
119 + w = csv.DictWriter(f, fieldnames=fields)
120 + w.writeheader(); w.writerows(rows)
121 +```
122 +
123 +**Flatten nested keys with dots:**
124 +```python
125 +def flatten(d, prefix=""):
126 + out = {}
127 + for k, v in d.items():
128 + key = f"{prefix}{k}"
129 + out.update(flatten(v, key + ".")) if isinstance(v, dict) else out.setdefault(key, v)
130 + return out
131 +```
132 +
133 +## Validate
134 +
135 +**Syntax:** `python3 -m json.tool file.json > /dev/null` — prints the error with line/column, exit 1 on failure.
136 +
137 +**Schema:**
138 +```python
139 +# pip install jsonschema
140 +from jsonschema import validate, ValidationError
141 +try:
142 + validate(instance=data, schema=schema)
143 +except ValidationError as e:
144 + print(f"invalid at {list(e.absolute_path)}: {e.message}")
145 +```
146 +
147 +## Gotchas
148 +- **Trailing commas, single quotes, comments** are the top three causes of `JSONDecodeError` — they are JavaScript habits, not JSON. Fix the exact character the error points at.
149 +- **`NaN`, `Infinity`**: `json.dumps` emits them by default but no strict parser accepts them. Use `allow_nan=False` to force the error at write time, then substitute `null`.
150 +- **Encoding:** JSON files are UTF-8 by spec; always pass `encoding="utf-8"` — Windows defaults to cp1252 and corrupts round-trips.
151 +- **Float precision:** `json.load` gives you binary floats (`0.1 + 0.2 != 0.3`); for money, parse with `json.load(f, parse_float=decimal.Decimal)`.
152 +- **Large ints** round-trip fine in Python but break JavaScript beyond 2^53 − 1; stringify IDs above that when the consumer is JS.
153 +- **`sort_keys=True` rewrites the whole file's order** — a huge diff for a one-key change. Leave order alone unless asked.
154 +- **BOM:** files from Windows tools may start with U+FEFF; open with `encoding="utf-8-sig"` if `json.load` fails on character 0.
added doc-skills/processing-markdown/SKILL.md +59 −0
@@ -0,0 +1,59 @@
1 +---
2 +name: processing-markdown
3 +description: Creates, reads, restructures, and converts Markdown files. Use when the user asks to write, edit, reorganize, lint, or fix a .md file, update a specific section of a README or docs page, generate a table of contents, or convert Markdown to or from HTML, DOCX, or PDF with pandoc. Do not use for writing user-facing release notes in house style (the writing-release-notes skill covers that) and not for HTML files.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Processing Markdown
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** creating or editing `.md` files (READMEs, docs, notes), structure-aware edits ("update the Installation section"), generating tables of contents, and converting md ↔ html/docx/pdf.
15 +- **Do NOT use for:** user-facing release notes in house style (use the writing-release-notes skill) or HTML files (use the HTML skill).
16 +
17 +## Quick reference
18 +
19 +**Default:** Markdown is plain text — create and modify by writing/editing the file directly. No library needed. **Conversion:** pandoc (`brew install pandoc`; PDF output also needs a LaTeX engine: `brew install basictex`).
20 +
21 +```bash
22 +# Convert
23 +pandoc README.md -o README.html # md → HTML
24 +pandoc report.md -o report.docx # md → Word
25 +pandoc report.md -o report.pdf # md → PDF (needs LaTeX)
26 +pandoc page.html -t gfm -o page.md # HTML → md
27 +pandoc document.docx -t gfm -o document.md # Word → md
28 +```
29 +
30 +**Structure-aware edit** — locate sections by heading lines, then edit only that slice:
31 +
32 +```python
33 +lines = open("README.md").read().splitlines(keepends=True)
34 +starts = [i for i, l in enumerate(lines) if l.startswith("#")]
35 +# a section runs from its heading to the next heading of same-or-higher level
36 +```
37 +
38 +## Rules
39 +- **Match the file's existing conventions** when editing: heading style (`#` vs underline), bullet marker (`-` vs `*`), emphasis (`_` vs `*`), code-fence style. Never reformat untouched sections.
40 +- One H1 (`#`) per document, at the top; sections descend without skipping levels (`##` → `###`).
41 +- Fenced code blocks always carry a language tag (```python, ```bash, ```text for plain).
42 +- Blank line before and after headings, lists, and code fences — most renderers require it.
43 +
44 +## Workflow
45 +1. Identify the operation: create / read-extract / section edit / convert.
46 +2. For edits, read the file first and note its conventions (heading style, bullets, fence style).
47 +3. Perform the edit on the smallest possible region (recipes in references/recipes.md).
48 +4. For conversion, verify every relative link and image path referenced in the file exists before running pandoc; report missing targets.
49 +5. **Validate:** re-read the result — heading hierarchy has no skipped levels, all fences are closed (even count of ``` lines), links/images resolve. For conversions, confirm the output file exists and is non-empty. Fix and repeat until clean.
50 +
51 +## Edge cases & failure modes
52 +- **pandoc missing** → `brew install pandoc` (macOS) / `apt-get install pandoc` (Linux); PDF errors about `pdflatex` → `brew install basictex`.
53 +- **Unclosed code fence** → everything after it renders as code; check for an odd number of ``` lines before editing by heading.
54 +- **Duplicate section names** → confirm with the user which occurrence to edit; never guess.
55 +- **Markdown flavor mismatch** (tables, task lists, footnotes) → target GitHub-Flavored Markdown (`-t gfm` in pandoc) unless the user states another renderer.
56 +- **Huge files (>5,000 lines)** → edit by line-range around the located heading; never rewrite the whole file for a one-section change.
57 +
58 +## References
59 +Deeper copy-paste recipes (section replace, TOC generation, link checking, pandoc options): see [references/recipes.md](references/recipes.md).
added doc-skills/processing-markdown/references/recipes.md +134 −0
@@ -0,0 +1,134 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Markdown Recipes
7 +
8 +## Contents
9 +- Parse document structure (headings, sections)
10 +- Replace one section in place
11 +- Generate a table of contents
12 +- Check links and images
13 +- Extract tables and code blocks
14 +- Pandoc conversion options
15 +- Gotchas (flavors, fences, whitespace)
16 +
17 +## Parse document structure (headings, sections)
18 +
19 +Heading detection must ignore ``` fenced regions — `#` inside code is not a heading:
20 +
21 +```python
22 +def headings(path):
23 + """Yield (line_no, level, title) for real headings only."""
24 + in_fence = False
25 + for i, line in enumerate(open(path, encoding="utf-8")):
26 + if line.lstrip().startswith("```"):
27 + in_fence = not in_fence
28 + elif not in_fence and line.startswith("#"):
29 + level = len(line) - len(line.lstrip("#"))
30 + if level <= 6 and line[level:level + 1] == " ":
31 + yield i, level, line[level:].strip()
32 +```
33 +
34 +## Replace one section in place
35 +
36 +A section spans from its heading to the next heading of the same or higher level:
37 +
38 +```python
39 +def replace_section(path, title, new_body):
40 + lines = open(path, encoding="utf-8").read().splitlines(keepends=True)
41 + hs = list(headings(path))
42 + match = [h for h in hs if h[2] == title]
43 + if len(match) != 1:
44 + raise ValueError(f"'{title}': found {len(match)} occurrences, need exactly 1")
45 + start_line, level, _ = match[0]
46 + later = [h for h in hs if h[0] > start_line and h[1] <= level]
47 + end_line = later[0][0] if later else len(lines)
48 + new = lines[:start_line + 1] + [new_body.rstrip() + "\n\n"] + lines[end_line:]
49 + open(path, "w", encoding="utf-8").writelines(new)
50 +```
51 +
52 +## Generate a table of contents
53 +
54 +GitHub anchor rule: lowercase, spaces → `-`, strip everything except word chars and hyphens:
55 +
56 +```python
57 +import re
58 +
59 +def toc(path, max_level=3):
60 + out = []
61 + for _, level, title in headings(path):
62 + if 2 <= level <= max_level: # skip the H1 itself
63 + anchor = re.sub(r"[^\w\- ]", "", title).strip().lower().replace(" ", "-")
64 + out.append(f"{' ' * (level - 2)}- [{title}](#{anchor})")
65 + return "\n".join(out)
66 +```
67 +
68 +Duplicate titles get `-1`, `-2` suffixes on GitHub — deduplicate with a counter if titles repeat.
69 +
70 +## Check links and images
71 +
72 +Run before any pandoc conversion; missing images abort PDF builds:
73 +
74 +```python
75 +import os, re
76 +
77 +def broken_refs(path):
78 + text = open(path, encoding="utf-8").read()
79 + base = os.path.dirname(os.path.abspath(path))
80 + broken = []
81 + for target in re.findall(r"!?\[[^\]]*\]\(([^)#\s]+)[^)]*\)", text):
82 + if not target.startswith(("http://", "https://", "mailto:")):
83 + if not os.path.exists(os.path.join(base, target)):
84 + broken.append(target)
85 + return broken
86 +```
87 +
88 +(External URLs: check only if the user asks — needs network calls.)
89 +
90 +## Extract tables and code blocks
91 +
92 +```python
93 +import re
94 +
95 +text = open("doc.md", encoding="utf-8").read()
96 +
97 +# Fenced code blocks with language → [(lang, code), ...]
98 +blocks = re.findall(r"```(\w*)\n(.*?)```", text, flags=re.S)
99 +
100 +# Pipe tables → rows of cells (skip the |---| separator line)
101 +rows = [[c.strip() for c in line.strip().strip("|").split("|")]
102 + for line in text.splitlines()
103 + if line.lstrip().startswith("|") and not re.match(r"^\s*\|[\s:|-]+\|\s*$", line)]
104 +```
105 +
106 +## Pandoc conversion options
107 +
108 +```bash
109 +# Standalone HTML with title metadata (otherwise pandoc emits a fragment)
110 +pandoc README.md -s --metadata title="README" -o README.html
111 +
112 +# GFM input explicitly (tables, task lists, strikethrough)
113 +pandoc -f gfm README.md -o out.docx
114 +
115 +# PDF with margins and a TOC
116 +pandoc report.md --toc -V geometry:margin=1in -o report.pdf
117 +
118 +# Word → Markdown, keep images
119 +pandoc report.docx --extract-media=./media -t gfm -o report.md
120 +
121 +# Custom Word styling: reuse an existing doc's styles
122 +pandoc report.md --reference-doc=template.docx -o styled.docx
123 +```
124 +
125 +## Gotchas (flavors, fences, whitespace)
126 +
127 +- **Flavor differences.** Tables, task lists (`- [ ]`), footnotes, and strikethrough are GFM/extensions — original Markdown renderers ignore them. Pandoc's default input is *pandoc markdown*, not GFM: pass `-f gfm` when the source came from GitHub.
128 +- **Fences must balance.** An unclosed ``` swallows the rest of the document. Count fence lines before structural edits (the `headings()` recipe already guards this).
129 +- **Indented code blocks.** 4-space-indented lines are code in classic Markdown — a list continuation indented 4+ spaces can silently become a code block. Prefer fenced blocks everywhere.
130 +- **Two trailing spaces = line break.** Invisible but meaningful; don't strip trailing whitespace blindly in an existing file.
131 +- **Bare URLs** don't auto-link in all renderers — wrap in `<...>` or `[text](url)`.
132 +- **HTML inside Markdown** passes through most renderers but is stripped by some (and by pandoc to some formats) — flag it when converting.
133 +- **Setext headings** (`Title\n=====`) are equivalent to `#`/`##`; the structure parser above misses them — normalize them first if a file mixes both styles, but only with the user's agreement (it rewrites lines).
134 +- **Anchor links differ across renderers.** The GitHub rule in the TOC recipe does not match GitLab/MkDocs exactly for punctuation-heavy titles; verify on the target platform.
added doc-skills/processing-pdf/SKILL.md +63 −0
@@ -0,0 +1,63 @@
1 +---
2 +name: processing-pdf
3 +description: Reads, creates, and modifies PDF files — extracts text and tables, builds new PDFs, merges, splits, rotates, watermarks, encrypts/decrypts, and fills PDF forms. Use when the user asks to read, extract, parse, create, generate, merge, combine, split, rotate, watermark, password-protect, decrypt, or fill a PDF, mentions a .pdf file, or asks to pull text or tables out of a PDF. Do not use for Word documents, images, or converting Office files to PDF (use the corresponding Office skill and export).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Processing PDF
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** any operation where a `.pdf` file is the input or the output — extraction, creation, page manipulation, forms, encryption.
15 +- **Do NOT use for:** Word documents (`.docx`), images, or converting Office files to PDF — use the corresponding Office skill and export instead.
16 +
17 +## Quick reference — one default per operation
18 +
19 +**Extract text/tables — pdfplumber:**
20 +```python
21 +import pdfplumber
22 +with pdfplumber.open("in.pdf") as pdf:
23 + text = "\n".join(p.extract_text() or "" for p in pdf.pages)
24 + tables = pdf.pages[0].extract_tables()
25 +```
26 +Escape hatch: if output is garbled or misordered, use `pdftotext -layout in.pdf out.txt` (poppler-utils). If pages yield no text at all, the PDF is scanned — OCR it with pytesseract + pdf2image (see recipes).
27 +
28 +**Page operations (merge/split/rotate/encrypt) and form filling — pypdf:**
29 +```python
30 +from pypdf import PdfReader, PdfWriter
31 +writer = PdfWriter()
32 +for path in ["a.pdf", "b.pdf"]:
33 + writer.append(path) # merge
34 +writer.write("merged.pdf")
35 +```
36 +
37 +**Create new PDFs — reportlab (Platypus):**
38 +```python
39 +from reportlab.lib.pagesizes import letter
40 +from reportlab.platypus import SimpleDocTemplate, Paragraph
41 +from reportlab.lib.styles import getSampleStyleSheet
42 +styles = getSampleStyleSheet()
43 +SimpleDocTemplate("out.pdf", pagesize=letter).build(
44 + [Paragraph("Title", styles["Title"]), Paragraph("Body text.", styles["Normal"])])
45 +```
46 +
47 +## Workflow
48 +1. Identify the operation (extract / create / modify / form-fill) and pick the default tool above.
49 +2. Run the operation with the minimal code needed; write output next to the input unless the user names a path.
50 +3. **Validate:** re-open the output with `PdfReader("out.pdf")` and check `len(reader.pages)` matches expectations; for extraction, confirm the text/tables are non-empty before reporting success.
51 +4. If validation fails, fix and repeat step 2 — never deliver an unverified file.
52 +5. Report the output path and a one-line summary (page count, or rows/chars extracted).
53 +
54 +## Edge cases & failure modes
55 +- **Missing dependency** → install exactly: `pip install pdfplumber pypdf reportlab`.
56 +- **Corrupt/malformed PDF** → report the parser's error message verbatim (it names the object/offset); do not guess at contents.
57 +- **Encrypted PDF** → `PdfReader(path)` raises or `reader.is_encrypted` is True; call `reader.decrypt(password)` — ask the user for the password, never brute-force.
58 +- **Scanned PDF (no text layer)** → `extract_text()` returns None/empty; switch to OCR (recipes) and tell the user accuracy depends on scan quality.
59 +- **Huge PDF** → process page-by-page (pdfplumber pages are lazy); never load all text into one string above ~1,000 pages.
60 +- **Empty file (0 bytes)** → report "file is empty, not a valid PDF" and stop.
61 +
62 +## References
63 +Deeper copy-paste recipes (watermark, split, encrypt, forms, OCR, conversion): see [references/recipes.md](references/recipes.md).
added doc-skills/processing-pdf/references/recipes.md +164 −0
@@ -0,0 +1,164 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# PDF Recipes
7 +
8 +## Contents
9 +- Read / extract (text, tables, metadata, images)
10 +- Create (Platypus documents, Canvas overlays)
11 +- Modify (split, rotate, watermark, encrypt/decrypt, forms)
12 +- Convert (PDF → text/CSV, images → PDF, OCR)
13 +- Gotchas
14 +
15 +## Read / extract
16 +
17 +**Text, page by page (memory-safe on large files):**
18 +```python
19 +import pdfplumber
20 +with pdfplumber.open("in.pdf") as pdf:
21 + for i, page in enumerate(pdf.pages, 1):
22 + text = page.extract_text() or "" # None on empty/scanned pages
23 + print(f"--- page {i} ---\n{text}")
24 +```
25 +
26 +**Tables → CSV:**
27 +```python
28 +import csv, pdfplumber
29 +with pdfplumber.open("in.pdf") as pdf, open("out.csv", "w", newline="") as f:
30 + w = csv.writer(f)
31 + for page in pdf.pages:
32 + for table in page.extract_tables():
33 + w.writerows(table)
34 +```
35 +
36 +**Metadata and page count:**
37 +```python
38 +from pypdf import PdfReader
39 +r = PdfReader("in.pdf")
40 +print(len(r.pages), r.metadata) # metadata may be None
41 +```
42 +
43 +**Extract embedded images:** `pdfimages -png in.pdf img_prefix` (poppler-utils).
44 +
45 +## Create
46 +
47 +**Structured document (Platypus):**
48 +```python
49 +from reportlab.lib.pagesizes import letter
50 +from reportlab.lib.units import inch
51 +from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer, Table, TableStyle
52 +from reportlab.lib.styles import getSampleStyleSheet
53 +from reportlab.lib import colors
54 +
55 +styles = getSampleStyleSheet()
56 +story = [
57 + Paragraph("Quarterly Report", styles["Title"]),
58 + Spacer(1, 0.2 * inch),
59 + Paragraph("Summary paragraph.", styles["Normal"]),
60 + Table([["Region", "Revenue"], ["East", "1,200"], ["West", "980"]],
61 + style=TableStyle([("GRID", (0, 0), (-1, -1), 0.5, colors.grey),
62 + ("BACKGROUND", (0, 0), (-1, 0), colors.lightgrey)])),
63 +]
64 +SimpleDocTemplate("report.pdf", pagesize=letter).build(story)
65 +```
66 +
67 +**Low-level drawing (Canvas)** — for precise placement (stamps, labels):
68 +```python
69 +from reportlab.pdfgen import canvas
70 +c = canvas.Canvas("stamp.pdf")
71 +c.setFont("Helvetica", 36)
72 +c.saveState(); c.translate(300, 400); c.rotate(45)
73 +c.setFillGray(0.5, 0.3) # 30% opacity grey watermark text
74 +c.drawCentredString(0, 0, "CONFIDENTIAL")
75 +c.restoreState(); c.save()
76 +```
77 +
78 +## Modify
79 +
80 +**Split — one file per page:**
81 +```python
82 +from pypdf import PdfReader, PdfWriter
83 +r = PdfReader("in.pdf")
84 +for i, page in enumerate(r.pages, 1):
85 + w = PdfWriter(); w.add_page(page); w.write(f"page-{i:03d}.pdf")
86 +```
87 +
88 +**Extract a page range (1-based, inclusive):**
89 +```python
90 +w = PdfWriter()
91 +w.append("in.pdf", pages=(4, 10)) # pypdf's range is 0-based, end-exclusive: pages 5–10
92 +w.write("excerpt.pdf")
93 +```
94 +
95 +**Rotate all pages 90° clockwise:**
96 +```python
97 +r = PdfReader("in.pdf"); w = PdfWriter()
98 +for page in r.pages:
99 + w.add_page(page.rotate(90)) # rotate() mutates and returns the page
100 +w.write("rotated.pdf")
101 +```
102 +
103 +**Watermark every page** (build `stamp.pdf` with the Canvas recipe above):
104 +```python
105 +r = PdfReader("in.pdf"); stamp = PdfReader("stamp.pdf").pages[0]
106 +w = PdfWriter()
107 +for page in r.pages:
108 + page.merge_page(stamp) # stamp drawn over the page content
109 + w.add_page(page)
110 +w.write("watermarked.pdf")
111 +```
112 +
113 +**Encrypt / decrypt:**
114 +```python
115 +w = PdfWriter(); w.append("in.pdf")
116 +w.encrypt(user_password="secret", algorithm="AES-256")
117 +w.write("locked.pdf")
118 +
119 +r = PdfReader("locked.pdf")
120 +if r.is_encrypted:
121 + r.decrypt("secret") # returns PasswordType; 0 means wrong password
122 +```
123 +
124 +**Fill form fields:**
125 +```python
126 +r = PdfReader("form.pdf"); w = PdfWriter(); w.append(r)
127 +print(r.get_fields().keys()) # inspect field names FIRST — they rarely match labels
128 +w.update_page_form_field_values(w.pages[0], {"name": "Alice", "date": "2026-08-05"})
129 +w.write("filled.pdf")
130 +```
131 +
132 +## Convert
133 +
134 +**PDF → plain text preserving layout:** `pdftotext -layout in.pdf out.txt`
135 +
136 +**Images → PDF:**
137 +```python
138 +from PIL import Image
139 +pages = [Image.open(p).convert("RGB") for p in ["a.png", "b.png"]]
140 +pages[0].save("out.pdf", save_all=True, append_images=pages[1:])
141 +```
142 +
143 +**OCR a scanned PDF:**
144 +```python
145 +# pip install pytesseract pdf2image ; also: brew install tesseract poppler
146 +import pytesseract
147 +from pdf2image import convert_from_path
148 +# 300 dpi is the standard OCR sweet spot: below it accuracy drops, above it is slow
149 +text = "\n".join(pytesseract.image_to_string(img)
150 + for img in convert_from_path("scan.pdf", dpi=300))
151 +```
152 +
153 +## Gotchas
154 +- **pdfplumber on scanned PDFs** returns `None`/empty text — that is a signal to OCR, not a bug.
155 +- **`extract_text()` can return `None`**; always `or ""` before joining.
156 +- **pypdf `append(pages=...)` is 0-based and end-exclusive** while users speak 1-based inclusive — convert deliberately.
157 +- **`page.rotate()` mutates in place** (and returns the page); don't rotate twice by chaining carelessly.
158 +- **Form fields keep old values visually** in some viewers unless you also set
159 + `w.set_need_appearances_writer(True)` before writing.
160 +- **Encryption:** pypdf's default RC4 is weak — always pass `algorithm="AES-256"`.
161 +- **Watermark order:** `page.merge_page(stamp)` draws the stamp *over* content; to put it under, merge the page onto the stamp instead.
162 +- **reportlab coordinates** start at the bottom-left in points (1 pt = 1/72 inch).
163 +- **Table extraction** depends on ruled lines; borderless tables need
164 + `extract_tables({"vertical_strategy": "text", "horizontal_strategy": "text"})`.
added doc-skills/processing-pptx/SKILL.md +81 −0
@@ -0,0 +1,81 @@
1 +---
2 +name: processing-pptx
3 +description: Creates, reads, and modifies PowerPoint presentations (.pptx, .potx) with python-pptx — slides, layouts, text, tables, images, charts. Use when the user asks to create, build, read, edit, or fix a presentation, deck, or slides, or mentions .pptx/.potx files or PowerPoint. Do not use for Word documents, PDFs, or Google Slides.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Processing PPTX
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** creating, reading, or modifying `.pptx` and `.potx` files — slides, text, tables, images, charts, speaker notes.
15 +- **Do NOT use for:** Word documents, PDFs, or Google Slides (different API).
16 +
17 +## Quick reference
18 +
19 +Default library: **python-pptx**. Visual verification: **LibreOffice** render to PDF/images when layout correctness matters.
20 +
21 +**Create:**
22 +```python
23 +from pptx import Presentation
24 +from pptx.util import Inches, Pt
25 +
26 +prs = Presentation() # 4:3 default; set 16:9 explicitly:
27 +prs.slide_width, prs.slide_height = Inches(13.333), Inches(7.5)
28 +
29 +slide = prs.slides.add_slide(prs.slide_layouts[0]) # 0 = title slide
30 +slide.shapes.title.text = "Q3 Results"
31 +slide.placeholders[1].text = "Finance Team — 2026"
32 +prs.save("deck.pptx")
33 +```
34 +
35 +**Read:**
36 +```python
37 +from pptx import Presentation
38 +for i, slide in enumerate(Presentation("deck.pptx").slides, 1):
39 + for shape in slide.shapes:
40 + if shape.has_text_frame:
41 + print(i, shape.text_frame.text)
42 +```
43 +
44 +**Modify:**
45 +```python
46 +prs = Presentation("deck.pptx")
47 +slide = prs.slides[1]
48 +for shape in slide.shapes:
49 + if shape.has_text_frame and "draft" in shape.text_frame.text.lower():
50 + shape.text_frame.text = shape.text_frame.text.replace("Draft", "Final")
51 +prs.save("deck.pptx")
52 +```
53 +
54 +**Visual check (when layout matters):**
55 +```bash
56 +soffice --headless --convert-to pdf deck.pptx # then inspect pages for overflow/overlap
57 +```
58 +
59 +## Design rules
60 +- Safe fonts only: Arial or Calibri.
61 +- Keep ≥0.5" margins; body text ≥14pt, titles ≥28pt.
62 +- No text-only slides in a finished deck — add an image, chart, or table per slide where content allows.
63 +- Build a new deck from an existing template (`Presentation("template.potx")`) when the user has one; reuse its layouts instead of drawing boxes manually.
64 +
65 +## Workflow
66 +1. Classify the task: create / read / modify.
67 +2. Creating: start from the user's template if provided, else default template with 16:9 size. Use slide layouts and placeholders, not free-floating text boxes, so themes apply.
68 +3. Modifying: read all slides first; edit in place, preserving each shape's position and formatting.
69 +4. Validate: re-open with `Presentation(path)` — if it raises, fix before delivering. Count slides and confirm expected titles.
70 +5. If the deck is a final deliverable, render via `soffice --headless --convert-to pdf` and check for text overflow, overlap, and contrast.
71 +6. Report the output path, slide count, and what changed.
72 +
73 +## Edge cases & failure modes
74 +- **python-pptx missing** → `pip install python-pptx`. **LibreOffice missing** → `brew install --cask libreoffice` (only needed for visual checks/conversion).
75 +- **Corrupt / not a zip** → `Presentation()` raises `PackageNotFoundError`; report invalid pptx, stop.
76 +- **Password-protected file** → python-pptx cannot decrypt; ask for an unprotected copy.
77 +- **`.ppt` (legacy binary)** → unsupported; convert first: `soffice --headless --convert-to pptx file.ppt`.
78 +- **Text overflowing its box** → python-pptx cannot measure rendered text; shorten text or reduce font size, then re-render to verify.
79 +
80 +## References
81 +Deeper recipes (bullets, tables, images, charts, speaker notes, template reuse, per-slide deletion, conversion, gotchas): see [references/recipes.md](references/recipes.md).
added doc-skills/processing-pptx/references/recipes.md +173 −0
@@ -0,0 +1,173 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# PPTX Recipes — python-pptx (+ LibreOffice)
7 +
8 +## Contents
9 +- Create: layouts, bullets, formatting
10 +- Tables
11 +- Images
12 +- Charts
13 +- Speaker notes
14 +- Read / extract (all text, per-slide, notes)
15 +- Modify (find-and-replace, delete a slide, reorder)
16 +- Template reuse
17 +- Convert / render
18 +- Gotchas
19 +
20 +## Create: layouts, bullets, formatting
21 +
22 +```python
23 +from pptx import Presentation
24 +from pptx.util import Inches, Pt
25 +from pptx.dml.color import RGBColor
26 +from pptx.enum.text import PP_ALIGN
27 +
28 +prs = Presentation()
29 +prs.slide_width, prs.slide_height = Inches(13.333), Inches(7.5) # 16:9
30 +
31 +# Default template layout indexes: 0 title, 1 title+content, 5 title only, 6 blank
32 +slide = prs.slides.add_slide(prs.slide_layouts[1])
33 +slide.shapes.title.text = "Agenda"
34 +
35 +body = slide.placeholders[1].text_frame
36 +body.text = "Overview" # first bullet, level 0
37 +for txt, lvl in [("Results", 0), ("By region", 1), ("Next steps", 0)]:
38 + p = body.add_paragraph()
39 + p.text, p.level = txt, lvl # levels give indented bullets — never type "•"
40 +
41 +# Run-level formatting
42 +run = body.paragraphs[0].runs[0]
43 +run.font.name = "Arial"
44 +run.font.size = Pt(18)
45 +run.font.bold = True
46 +run.font.color.rgb = RGBColor(0x44, 0x72, 0xC4)
47 +body.paragraphs[0].alignment = PP_ALIGN.LEFT
48 +
49 +prs.save("deck.pptx")
50 +```
51 +
52 +## Tables
53 +
54 +```python
55 +rows, cols = 3, 3
56 +# 0.5" margins on a 13.333" slide → 12.333" usable width
57 +tbl = slide.shapes.add_table(rows, cols, Inches(0.5), Inches(2), Inches(12.333), Inches(3)).table
58 +tbl.columns[0].width = Inches(4)
59 +for c, h in enumerate(["Region", "Q1", "Q2"]):
60 + cell = tbl.cell(0, c)
61 + cell.text = h
62 + cell.text_frame.paragraphs[0].runs[0].font.bold = True
63 +tbl.cell(1, 0).text = "East"
64 +```
65 +
66 +## Images
67 +
68 +```python
69 +# Size by width only — height scales to keep aspect ratio
70 +slide.shapes.add_picture("chart.png", Inches(7), Inches(1.5), width=Inches(5.8))
71 +```
72 +
73 +## Charts
74 +
75 +```python
76 +from pptx.chart.data import CategoryChartData
77 +from pptx.enum.chart import XL_CHART_TYPE
78 +
79 +data = CategoryChartData()
80 +data.categories = ["East", "West"]
81 +data.add_series("Q1", (100, 90))
82 +data.add_series("Q2", (150, 120))
83 +slide.shapes.add_chart(XL_CHART_TYPE.COLUMN_CLUSTERED,
84 + Inches(0.5), Inches(1.5), Inches(6), Inches(4.5), data)
85 +```
86 +
87 +## Speaker notes
88 +
89 +```python
90 +slide.notes_slide.notes_text_frame.text = "Mention the supply-chain caveat here."
91 +```
92 +
93 +## Read / extract
94 +
95 +```python
96 +from pptx import Presentation
97 +prs = Presentation("deck.pptx")
98 +
99 +# All text, slide by slide (walks groups too)
100 +def shape_texts(shapes):
101 + for sh in shapes:
102 + if sh.shape_type == 6: # group shape — recurse
103 + yield from shape_texts(sh.shapes)
104 + elif sh.has_text_frame:
105 + yield sh.text_frame.text
106 +
107 +for i, slide in enumerate(prs.slides, 1):
108 + print(f"--- slide {i}: {list(shape_texts(slide.shapes))}")
109 +
110 +# Tables
111 +for slide in prs.slides:
112 + for sh in slide.shapes:
113 + if sh.has_table:
114 + rows = [[c.text for c in r.cells] for r in sh.table.rows]
115 +
116 +# Notes
117 +notes = [s.notes_slide.notes_text_frame.text if s.has_notes_slide else "" for s in prs.slides]
118 +```
119 +
120 +## Modify
121 +
122 +```python
123 +# Find-and-replace preserving run formatting where possible
124 +def replace_text(prs, old, new):
125 + for slide in prs.slides:
126 + for sh in slide.shapes:
127 + if not sh.has_text_frame:
128 + continue
129 + for p in sh.text_frame.paragraphs:
130 + for run in p.runs:
131 + if old in run.text:
132 + run.text = run.text.replace(old, new)
133 + if old in p.text: # spans runs — collapse to first run
134 + full = p.text.replace(old, new)
135 + for r in p.runs: r.text = ""
136 + if p.runs: p.runs[0].text = full
137 +
138 +# Delete a slide (no public API — drop the XML relationship)
139 +def delete_slide(prs, index):
140 + xml_slides = prs.slides._sldIdLst
141 + xml_slides.remove(list(xml_slides)[index])
142 +
143 +# Reorder: remove and reinsert the sldId element at the target position
144 +```
145 +
146 +## Template reuse
147 +
148 +```python
149 +prs = Presentation("corporate-template.potx") # keeps theme, fonts, layouts
150 +for layout in prs.slide_masters[0].slide_layouts:
151 + print(layout.name) # pick layouts by name, indexes vary per template
152 +slide = prs.slides.add_slide(prs.slide_masters[0].slide_layouts[1])
153 +prs.save("deck.pptx") # save as .pptx, not .potx
154 +```
155 +
156 +## Convert / render
157 +
158 +```bash
159 +soffice --headless --convert-to pdf deck.pptx # visual QA
160 +pdftoppm -png -r 80 deck.pdf slide # slide-1.png … for inspection
161 +soffice --headless --convert-to pptx legacy.ppt # legacy conversion
162 +```
163 +
164 +## Gotchas
165 +
166 +- **Placeholder indexes are template-specific** — `placeholders[1]` on the default template is the body, but custom templates differ; iterate `slide.placeholders` and check `.placeholder_format.idx`/`.name`.
167 +- **`prs.slides` has no `add_slide`-at-position, no delete, no copy** — deletion/reorder require the `_sldIdLst` XML manipulation above; copying slides between files is not supported (rebuild instead).
168 +- **First paragraph already exists** in a text frame: set `tf.text` for it, use `add_paragraph()` only for the rest, or you get a leading empty line.
169 +- **python-pptx cannot measure rendered text** — overflow is invisible until you render with LibreOffice; always visual-check final decks.
170 +- **Colors:** `RGBColor(0x44, 0x72, 0xC4)` — no alpha channel; theme colors need `run.font.color.theme_color`.
171 +- **Charts added by python-pptx embed an xlsx part** — replacing chart data later requires `chart.replace_data(new_data)`, not editing cells.
172 +- **`.potx` templates open fine but must be saved as `.pptx`** to be presentable; saving over the `.potx` alters the user's template.
173 +- **Group shapes hide their children** from a flat `slide.shapes` text walk — recurse into `shape_type == 6` (see reader recipe).
added doc-skills/processing-xlsx/SKILL.md +63 −0
@@ -0,0 +1,63 @@
1 +---
2 +name: processing-xlsx
3 +description: Creates, reads, and modifies Excel workbooks (.xlsx, .xlsm, .xltx) with openpyxl — cell values, formulas, formatting, multiple sheets. Use when the user asks to create, open, read, edit, update, or fix an Excel file or spreadsheet, mentions .xlsx/.xlsm/.xltx files, workbooks, worksheets, or Excel formulas. Do not use for .csv/.tsv files (plain-text tabular data) or for data-quality profiling.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Processing XLSX
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** creating, reading, or modifying `.xlsx`, `.xlsm`, `.xltx` workbooks — values, formulas, formatting, sheets.
15 +- **Do NOT use for:** `.csv`/`.tsv` files (handle as plain text), data-quality profiling, or Google Sheets (different API).
16 +
17 +## Quick reference
18 +
19 +Default library: **openpyxl**. Escape hatch: **pandas** (`read_excel`/`to_excel`) only for bulk data I/O with no formulas or formatting.
20 +
21 +**Create:**
22 +```python
23 +from openpyxl import Workbook
24 +wb = Workbook()
25 +ws = wb.active
26 +ws.title = "Sales"
27 +ws.append(["Region", "Revenue"])
28 +ws.append(["East", 1200])
29 +ws["B3"] = "=SUM(B2:B2)" # formulas as strings, never hardcoded results
30 +wb.save("sales.xlsx")
31 +```
32 +
33 +**Read — always two passes:**
34 +```python
35 +from openpyxl import load_workbook
36 +wb_f = load_workbook("sales.xlsx") # pass 1: formula strings
37 +wb_v = load_workbook("sales.xlsx", data_only=True) # pass 2: cached values
38 +```
39 +`data_only=True` returns `None` for formulas if the file was never opened/recalculated by Excel or LibreOffice — report that, don't guess values.
40 +
41 +**Modify:**
42 +```python
43 +wb = load_workbook("sales.xlsx") # never data_only when re-saving: cached-only load discards formulas
44 +wb["Sales"]["B2"] = 1500
45 +wb.save("sales.xlsx")
46 +```
47 +
48 +## Workflow
49 +1. Classify the task: create / read / modify. For bulk dataframe dumps with zero formatting, use pandas; otherwise openpyxl.
50 +2. When modifying, first read the file (two passes) and match its existing conventions: sheet names, header row, number formats, fonts.
51 +3. Write formulas as strings (`'=SUM(B2:B9)'`); never compute a result in Python and hardcode it where a formula belongs.
52 +4. Save, then validate: `load_workbook(path)` on the output — if it raises, fix before delivering. List sheet names and dimensions to confirm expected structure.
53 +5. Report the output path and what changed (sheets touched, ranges written).
54 +
55 +## Edge cases & failure modes
56 +- **openpyxl missing** → `pip install openpyxl` (pandas path additionally needs `pip install pandas`).
57 +- **Corrupt / not a zip** → `load_workbook` raises `BadZipFile` or `InvalidFileException`; report the file is not a valid xlsx, stop.
58 +- **Password-protected workbook** → openpyxl cannot decrypt; tell the user to remove the password (openpyxl has no decryption support).
59 +- **Large file (>50 MB or >1M cells)** → read with `read_only=True`, write with `write_only=True`; both stream instead of loading everything in memory.
60 +- **`.xlsm` macros** → open with `keep_vba=True` and save as `.xlsm`, otherwise macros are silently stripped.
61 +
62 +## References
63 +Deeper recipes (formatting, charts, merged cells, find-and-replace, conversion, gotchas): see [references/recipes.md](references/recipes.md).
added doc-skills/processing-xlsx/references/recipes.md +163 −0
@@ -0,0 +1,163 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# XLSX Recipes — openpyxl
7 +
8 +## Contents
9 +- Create with formatting (styles, header row, merged cells, column widths)
10 +- Charts
11 +- Read / extract (all sheets, one sheet, used range, tables)
12 +- Modify (insert/delete rows and columns, find-and-replace, add a sheet)
13 +- Convert (CSV ↔ xlsx, xlsx → pandas)
14 +- Gotchas
15 +
16 +## Create with formatting
17 +
18 +```python
19 +from openpyxl import Workbook
20 +from openpyxl.styles import Font, PatternFill, Alignment, Border, Side
21 +from openpyxl.utils import get_column_letter
22 +
23 +wb = Workbook()
24 +ws = wb.active
25 +ws.title = "Report"
26 +
27 +# Header row
28 +headers = ["Region", "Q1", "Q2", "Total"]
29 +ws.append(headers)
30 +header_font = Font(name="Arial", bold=True, color="FFFFFF")
31 +header_fill = PatternFill("solid", fgColor="4472C4")
32 +for cell in ws[1]:
33 + cell.font = header_font
34 + cell.fill = header_fill
35 + cell.alignment = Alignment(horizontal="center")
36 +
37 +# Data + formula per row
38 +for row in [["East", 100, 150], ["West", 90, 120]]:
39 + ws.append(row)
40 +for r in range(2, ws.max_row + 1):
41 + ws.cell(row=r, column=4).value = f"=SUM(B{r}:C{r})"
42 +
43 +# Merged title above the table: insert row first, then merge
44 +ws.insert_rows(1)
45 +ws["A1"] = "Quarterly Sales"
46 +ws.merge_cells("A1:D1")
47 +ws["A1"].font = Font(size=14, bold=True)
48 +
49 +# Column widths (openpyxl never auto-sizes)
50 +for col in range(1, 5):
51 + ws.column_dimensions[get_column_letter(col)].width = 14
52 +
53 +# Number format
54 +for r in range(3, ws.max_row + 1):
55 + for c in range(2, 5):
56 + ws.cell(row=r, column=c).number_format = "#,##0"
57 +
58 +wb.save("report.xlsx")
59 +```
60 +
61 +## Charts
62 +
63 +```python
64 +from openpyxl.chart import BarChart, Reference
65 +
66 +chart = BarChart()
67 +chart.title = "Revenue by Region"
68 +data = Reference(ws, min_col=2, max_col=3, min_row=2, max_row=ws.max_row) # includes header row for series names
69 +cats = Reference(ws, min_col=1, min_row=3, max_row=ws.max_row)
70 +chart.add_data(data, titles_from_data=True)
71 +chart.set_categories(cats)
72 +ws.add_chart(chart, "F3") # anchor = top-left cell of the chart
73 +wb.save("report.xlsx")
74 +```
75 +
76 +## Read / extract
77 +
78 +```python
79 +from openpyxl import load_workbook
80 +
81 +wb = load_workbook("report.xlsx", data_only=True)
82 +
83 +# All sheets → list of rows
84 +for name in wb.sheetnames:
85 + ws = wb[name]
86 + rows = [[c.value for c in row] for row in ws.iter_rows()]
87 +
88 +# Used range only (skips trailing empty rows/cols)
89 +ws = wb["Report"]
90 +data = [[c.value for c in row] for row in ws.iter_rows(min_row=1, max_row=ws.max_row, max_col=ws.max_column)]
91 +
92 +# Streaming read for large files
93 +wb_big = load_workbook("big.xlsx", read_only=True, data_only=True)
94 +for row in wb_big["Sheet1"].iter_rows(values_only=True):
95 + pass # process row tuple
96 +wb_big.close() # read_only keeps the file handle open — always close
97 +```
98 +
99 +## Modify
100 +
101 +```python
102 +from openpyxl import load_workbook
103 +wb = load_workbook("report.xlsx")
104 +ws = wb["Report"]
105 +
106 +# Insert / delete
107 +ws.insert_rows(2) # one row above row 2
108 +ws.delete_cols(3) # delete column C
109 +ws.insert_cols(3, amount=2)
110 +
111 +# Find-and-replace (string cells only)
112 +for row in ws.iter_rows():
113 + for cell in row:
114 + if isinstance(cell.value, str) and "East" in cell.value:
115 + cell.value = cell.value.replace("East", "North-East")
116 +
117 +# Add a sheet at a position
118 +summary = wb.create_sheet("Summary", 0) # index 0 = first tab
119 +summary["A1"] = "=Report!D3"
120 +
121 +wb.save("report.xlsx")
122 +```
123 +
124 +Insert/delete shifts cells but does **not** rewrite formulas that referenced the shifted range — check formulas after structural edits.
125 +
126 +## Convert
127 +
128 +```python
129 +# CSV → xlsx
130 +import csv
131 +from openpyxl import Workbook
132 +wb = Workbook(); ws = wb.active
133 +with open("data.csv", newline="") as f:
134 + for row in csv.reader(f):
135 + ws.append(row)
136 +wb.save("data.xlsx")
137 +
138 +# xlsx → CSV (one sheet)
139 +import csv
140 +from openpyxl import load_workbook
141 +ws = load_workbook("data.xlsx", data_only=True).active
142 +with open("out.csv", "w", newline="") as f:
143 + csv.writer(f).writerows([c if c is not None else "" for c in row]
144 + for row in ws.iter_rows(values_only=True))
145 +
146 +# Bulk I/O escape hatch
147 +import pandas as pd
148 +df = pd.read_excel("data.xlsx", sheet_name="Report") # needs openpyxl installed
149 +df.to_excel("out.xlsx", index=False) # loses all formulas/formatting
150 +```
151 +
152 +Formula recalculation without Excel: `soffice --headless --convert-to xlsx --outdir /tmp file.xlsx` (LibreOffice recalculates and writes cached values).
153 +
154 +## Gotchas
155 +
156 +- **openpyxl never computes formulas.** `data_only=True` returns the cached value from the last save by Excel/LibreOffice; a file created by openpyxl and never opened elsewhere has no cached values (`None`).
157 +- **Loading with `data_only=True` and saving destroys all formulas** — they are replaced by their cached values. Never re-save a data_only load.
158 +- **`ws.max_row`/`max_column` count formatted-but-empty cells**, so they can overshoot the real data; trim trailing `None` rows when extracting.
159 +- **Merged cells:** only the top-left cell holds the value; the rest read `None`. Unmerge with `ws.unmerge_cells(...)` before editing the range.
160 +- **Dates** come back as `datetime` objects; number formats (e.g. `"YYYY-MM-DD"`) control display only.
161 +- **Colors are 8-digit ARGB hex** (`"FF4472C4"`) or 6-digit RGB — no `#` prefix.
162 +- **`keep_vba=True`** is required to round-trip `.xlsm`; saving an `.xlsm` load as `.xlsx` drops macros without warning.
163 +- **Styles are copied by assignment, not reference:** to reuse a style on many cells, assign `Font(...)`/`PatternFill(...)` objects per cell or use `NamedStyle`.
added doc-skills/processing-xml/SKILL.md +69 −0
@@ -0,0 +1,69 @@
1 +---
2 +name: processing-xml
3 +description: Creates, reads, modifies, validates, and queries XML files with Python. Use when the user asks to parse, edit, generate, validate, or extract data from an .xml file, mentions XML elements, attributes, namespaces, XPath queries, or XSD validation, or needs to transform XML data. Do not use for HTML pages (different parser tolerance) or for the internal XML of .docx/.xlsx/.pptx files (use the corresponding Office skill).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Processing XML
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** creating, reading, editing, validating, or querying standalone `.xml` files (data feeds, configs, sitemaps, SVG as data, exports).
15 +- **Do NOT use for:** HTML pages (use the HTML skill — HTML parsers tolerate malformed markup, XML parsers do not) or the internal XML inside `.docx`/`.xlsx`/`.pptx` archives (use the corresponding Office skill).
16 +
17 +## Quick reference
18 +
19 +**Default:** stdlib `xml.etree.ElementTree` (no install). **Untrusted input:** `defusedxml` (`pip install defusedxml`) — same API, blocks entity-expansion attacks. **Escape hatch:** `lxml` (`pip install lxml`) only when you need full XPath 1.0, XSD validation, or pretty-printing on Python <3.9.
20 +
21 +```python
22 +import xml.etree.ElementTree as ET
23 +
24 +# Read
25 +tree = ET.parse("data.xml")
26 +root = tree.getroot()
27 +
28 +# Query (namespaced documents need an explicit map)
29 +ns = {"a": "http://example.com/ns"}
30 +items = root.findall(".//a:item", ns)
31 +
32 +# Modify
33 +for item in items:
34 + item.set("status", "done")
35 +
36 +# Write — always preserve the declaration and encoding
37 +tree.write("data.xml", xml_declaration=True, encoding="utf-8")
38 +```
39 +
40 +**Create:**
41 +
42 +```python
43 +root = ET.Element("catalog")
44 +book = ET.SubElement(root, "book", id="1")
45 +ET.SubElement(book, "title").text = "Example"
46 +ET.indent(root) # pretty-print, Python 3.9+
47 +ET.ElementTree(root).write("out.xml", xml_declaration=True, encoding="utf-8")
48 +```
49 +
50 +## Rules
51 +- **Never regex-edit XML.** Parse, modify the tree, re-serialize — always.
52 +- Handle namespaces explicitly with a namespace map; register prefixes before writing to avoid `ns0:` pollution.
53 +- Always write with `xml_declaration=True, encoding="utf-8"` so the declaration survives round-trips.
54 +
55 +## Workflow
56 +1. Identify the operation (create / read / modify / validate / query) and whether input is untrusted (→ defusedxml).
57 +2. Parse the file; on `ET.ParseError`, report the message with its line/column verbatim and stop — do not guess at fixes.
58 +3. Perform the operation per the Quick reference (deeper recipes in references/recipes.md).
59 +4. Write with declaration + encoding preserved.
60 +5. **Validate:** re-parse the written file (`ET.parse(output)`); if an XSD was given, validate against it with lxml. Fix and repeat step 3 until it parses clean.
61 +
62 +## Edge cases & failure modes
63 +- **`lxml`/`defusedxml` missing** → `pip install lxml` / `pip install defusedxml`; fall back to stdlib ElementTree if the feature allows.
64 +- **Malformed XML** → relay the parser's error (line, column) to the user; never hand-patch text and retry silently.
65 +- **Encoding mismatch** (declaration says one thing, bytes say another) → parse from bytes, not decoded text; report if it still fails.
66 +- **Huge files (>100 MB)** → stream with `ET.iterparse(path, events=("end",))` and `elem.clear()` after each record instead of loading the whole tree.
67 +
68 +## References
69 +Deeper copy-paste recipes (XPath, XSD validation, namespace round-trips, conversion): see [references/recipes.md](references/recipes.md).
added doc-skills/processing-xml/references/recipes.md +139 −0
@@ -0,0 +1,139 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# XML Recipes
7 +
8 +## Contents
9 +- Create a document with namespaces
10 +- Read and extract (XPath)
11 +- Modify: insert, remove, rename, move
12 +- Validate against an XSD
13 +- Streaming huge files
14 +- Convert XML ↔ dict/JSON
15 +- Gotchas
16 +
17 +## Create a document with namespaces
18 +
19 +```python
20 +import xml.etree.ElementTree as ET
21 +
22 +NS = "http://example.com/catalog"
23 +ET.register_namespace("", NS) # default namespace, no prefix in output
24 +
25 +root = ET.Element(f"{{{NS}}}catalog")
26 +book = ET.SubElement(root, f"{{{NS}}}book", {"id": "bk101"})
27 +ET.SubElement(book, f"{{{NS}}}title").text = "XML Developer's Guide"
28 +ET.SubElement(book, f"{{{NS}}}price").text = "44.95"
29 +
30 +ET.indent(root) # 2-space pretty print (3.9+)
31 +ET.ElementTree(root).write("catalog.xml",
32 + xml_declaration=True, encoding="utf-8")
33 +```
34 +
35 +## Read and extract (XPath)
36 +
37 +ElementTree supports a *subset* of XPath — child paths, `//`, `[@attr]`, `[tag='text']`, positional `[1]`:
38 +
39 +```python
40 +tree = ET.parse("catalog.xml")
41 +root = tree.getroot()
42 +ns = {"c": "http://example.com/catalog"}
43 +
44 +first = root.find("c:book[1]", ns) # first book
45 +cheap = root.findall(".//c:book[c:price='44.95']", ns)
46 +ids = [b.get("id") for b in root.findall(".//c:book", ns)]
47 +text = root.findtext(".//c:book/c:title", default="", namespaces=ns)
48 +```
49 +
50 +Full XPath 1.0 (functions, `contains()`, axes) needs lxml:
51 +
52 +```python
53 +from lxml import etree
54 +root = etree.parse("catalog.xml").getroot()
55 +titles = root.xpath("//c:book[contains(c:title,'Guide')]/c:title/text()",
56 + namespaces={"c": "http://example.com/catalog"})
57 +```
58 +
59 +## Modify: insert, remove, rename, move
60 +
61 +```python
62 +tree = ET.parse("catalog.xml")
63 +root = tree.getroot()
64 +ns = {"c": "http://example.com/catalog"}
65 +
66 +# Insert after an existing child (ElementTree has no insert-after: use index)
67 +books = root.findall("c:book", ns)
68 +new = ET.Element(f"{{http://example.com/catalog}}book", {"id": "bk102"})
69 +root.insert(list(root).index(books[-1]) + 1, new)
70 +
71 +# Remove — you must remove from the PARENT
72 +for bad in root.findall("c:book[@id='bk101']", ns):
73 + root.remove(bad)
74 +
75 +# Rename a tag
76 +for el in root.iter(f"{{http://example.com/catalog}}price"):
77 + el.tag = f"{{http://example.com/catalog}}cost"
78 +
79 +tree.write("catalog.xml", xml_declaration=True, encoding="utf-8")
80 +```
81 +
82 +To find a parent when you only matched the child (stdlib has no `getparent()`):
83 +
84 +```python
85 +parents = {c: p for p in root.iter() for c in p}
86 +parents[child].remove(child)
87 +```
88 +
89 +## Validate against an XSD
90 +
91 +Requires lxml (`pip install lxml`):
92 +
93 +```python
94 +from lxml import etree
95 +
96 +schema = etree.XMLSchema(etree.parse("catalog.xsd"))
97 +doc = etree.parse("catalog.xml")
98 +if not schema.validate(doc):
99 + for err in schema.error_log:
100 + print(f"line {err.line}, col {err.column}: {err.message}")
101 +```
102 +
103 +Report every error with line/column; fix the tree, re-serialize, re-validate.
104 +
105 +## Streaming huge files
106 +
107 +Load-then-clear keeps memory flat regardless of file size:
108 +
109 +```python
110 +import xml.etree.ElementTree as ET
111 +
112 +for event, elem in ET.iterparse("huge.xml", events=("end",)):
113 + if elem.tag.endswith("record"):
114 + process(elem)
115 + elem.clear() # release children; keeps memory bounded
116 +```
117 +
118 +## Convert XML ↔ dict/JSON
119 +
120 +No stdlib one-liner exists; for flat, repetitive data write it explicitly:
121 +
122 +```python
123 +rows = [{"id": b.get("id"), "title": b.findtext("c:title", "", ns)}
124 + for b in root.findall(".//c:book", ns)]
125 +import json; json.dump(rows, open("books.json", "w"), indent=2)
126 +```
127 +
128 +For arbitrary nesting, `pip install xmltodict` and `xmltodict.parse(open("f.xml", "rb"))` — but beware: single children become dicts, repeated children become lists, so downstream code must handle both shapes.
129 +
130 +## Gotchas
131 +
132 +- **`ns0:` prefix pollution.** ElementTree invents `ns0:` prefixes on write unless you call `ET.register_namespace(prefix, uri)` *before* parsing/writing. Register `""` for a default namespace.
133 +- **Truthiness trap.** An element with no children is falsy: `if elem:` is False even when the element exists. Always use `if elem is not None:`.
134 +- **`find()` vs `findall()` with no namespace map** silently return nothing on namespaced documents — the tag you search must be `{uri}tag` or use the `ns` map. "It finds nothing" almost always means a missing namespace map.
135 +- **No parent pointers** in stdlib ElementTree — removal requires the parent (see recipe above); lxml elements have `.getparent()`.
136 +- **Comments and processing instructions are dropped** by the default ElementTree parser. If they must survive a round-trip, use lxml, which preserves them.
137 +- **`ET.tostring()` omits the XML declaration** unless `xml_declaration=True` — and returns bytes when an encoding is given.
138 +- **Entity attacks (billion laughs, external entities).** Never parse untrusted XML with plain ElementTree/lxml defaults; use defusedxml drop-ins (`defusedxml.ElementTree.parse`).
139 +- **`ET.indent()` mutates text nodes** — don't use it on documents where whitespace inside elements is significant (mixed content).
added doc-skills/processing-yaml/SKILL.md +68 −0
@@ -0,0 +1,68 @@
1 +---
2 +name: processing-yaml
3 +description: Creates, reads, modifies, and validates YAML files with Python. Use when the user asks to parse, edit, generate, or fix a .yaml or .yml file, mentions YAML syntax errors, or works with YAML-based config files such as docker-compose.yml, Kubernetes manifests, or GitHub Actions workflow files. Do not use for JSON files or for authoring the logic of CI pipelines — only the YAML file mechanics.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Processing YAML
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** creating, reading, editing, or validating `.yaml`/`.yml` files — app configs, docker-compose, Kubernetes manifests, CI workflow files (the file mechanics).
15 +- **Do NOT use for:** JSON files (use `json` directly) or designing the *logic* of CI pipelines — this skill covers YAML file handling, not what the pipeline should do.
16 +
17 +## Quick reference
18 +
19 +**Default:** PyYAML (`pip install pyyaml`). **Escape hatch:** ruamel.yaml (`pip install ruamel.yaml`) only when comments and formatting must survive a round-trip edit — PyYAML discards them.
20 +
21 +```python
22 +import yaml
23 +
24 +# Read — safe_load ONLY
25 +with open("config.yaml") as f:
26 + data = yaml.safe_load(f) # returns dict/list/scalars
27 +
28 +# Modify
29 +data["server"]["port"] = 8080
30 +
31 +# Write
32 +with open("config.yaml", "w") as f:
33 + yaml.safe_dump(data, f, default_flow_style=False,
34 + sort_keys=False, allow_unicode=True)
35 +```
36 +
37 +**Comment-preserving edit (ruamel.yaml):**
38 +
39 +```python
40 +from ruamel.yaml import YAML
41 +y = YAML() # round-trip mode by default
42 +doc = y.load(open("config.yaml"))
43 +doc["server"]["port"] = 8080
44 +y.dump(doc, open("config.yaml", "w"))
45 +```
46 +
47 +## Rules
48 +- **NEVER `yaml.load()` without `SafeLoader`** — it executes arbitrary Python object constructors. `safe_load`/`safe_dump` always.
49 +- Quote strings that YAML would reinterpret: `"no"`, `"on"`, `"yes"`, version strings like `"1.10"`, country codes like `"NO"`.
50 +- 2-space indentation; never tabs (tabs are a YAML syntax error).
51 +- If the file has comments the user wants kept, use ruamel.yaml — a PyYAML round-trip silently deletes them.
52 +
53 +## Workflow
54 +1. Determine the operation and whether comments/formatting must survive (→ ruamel.yaml).
55 +2. Parse with `yaml.safe_load`; on `yaml.YAMLError`, report its message with line/column verbatim and stop — do not guess.
56 +3. Modify the loaded structure (recipes in references/recipes.md), quoting ambiguous scalars.
57 +4. Write with `safe_dump(default_flow_style=False, sort_keys=False, allow_unicode=True)` to keep block style, key order, and non-ASCII text.
58 +5. **Validate:** re-parse the written file with `safe_load`; for multi-document files confirm document count is unchanged. Fix and repeat step 3 until clean.
59 +
60 +## Edge cases & failure modes
61 +- **PyYAML missing** → `pip install pyyaml`; ruamel.yaml missing → `pip install ruamel.yaml`.
62 +- **Malformed YAML** → relay the parser error (it includes line/column); common causes to mention: tabs, unquoted `:` in values, bad indentation.
63 +- **Multi-document files** (`---` separators) → use `yaml.safe_load_all()` / `yaml.safe_dump_all()`; plain `safe_load` raises on the second document.
64 +- **Encoding** → open files as UTF-8; `allow_unicode=True` on dump prevents `\uXXXX` escaping of accented text.
65 +- **Huge files** → YAML has no practical streaming parser; for >50 MB data files, question whether the data belongs in YAML at all and suggest JSON/CSV.
66 +
67 +## References
68 +Deeper copy-paste recipes (anchors, multi-doc, schema validation, JSON conversion): see [references/recipes.md](references/recipes.md).
added doc-skills/processing-yaml/references/recipes.md +135 −0
@@ -0,0 +1,135 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# YAML Recipes
7 +
8 +## Contents
9 +- Create a config from scratch
10 +- Read: multi-document files and defaults
11 +- Modify: nested keys, lists, comment-preserving edits
12 +- Validate against a schema
13 +- Convert YAML ↔ JSON
14 +- Gotchas (Norway problem, implicit typing, anchors)
15 +
16 +## Create a config from scratch
17 +
18 +```python
19 +import yaml
20 +
21 +config = {
22 + "server": {"host": "0.0.0.0", "port": 8080},
23 + "features": ["auth", "metrics"],
24 + "welcome": "Bienvenue à Montréal",
25 +}
26 +with open("config.yaml", "w") as f:
27 + yaml.safe_dump(config, f,
28 + default_flow_style=False, # block style, not {a: 1}
29 + sort_keys=False, # keep insertion order
30 + allow_unicode=True) # keep accented chars readable
31 +```
32 +
33 +Force quotes on an ambiguous scalar:
34 +
35 +```python
36 +yaml.safe_dump({"country": "NO", "version": "1.10"}, f,
37 + default_style=None) # safe_dump already quotes these on output
38 +# When hand-writing YAML text, quote them yourself: country: "NO"
39 +```
40 +
41 +## Read: multi-document files and defaults
42 +
43 +```python
44 +import yaml
45 +
46 +# Single document
47 +data = yaml.safe_load(open("config.yaml")) or {} # empty file → None, so `or {}`
48 +
49 +# Multi-document (--- separators), e.g. Kubernetes manifests
50 +docs = list(yaml.safe_load_all(open("manifests.yaml")))
51 +
52 +# Nested read with defaults
53 +port = (data.get("server") or {}).get("port", 8080)
54 +```
55 +
56 +## Modify: nested keys, lists, comment-preserving edits
57 +
58 +PyYAML (comments will be lost):
59 +
60 +```python
61 +data = yaml.safe_load(open("config.yaml"))
62 +data.setdefault("server", {})["port"] = 9090
63 +data.setdefault("features", []).append("tracing")
64 +yaml.safe_dump(data, open("config.yaml", "w"),
65 + default_flow_style=False, sort_keys=False, allow_unicode=True)
66 +```
67 +
68 +ruamel.yaml (comments, quotes, key order all survive):
69 +
70 +```python
71 +from ruamel.yaml import YAML
72 +
73 +y = YAML() # round-trip mode
74 +y.indent(mapping=2, sequence=4, offset=2) # match common k8s/compose style
75 +doc = y.load(open("docker-compose.yml"))
76 +doc["services"]["web"]["ports"] = ["8080:80"]
77 +y.dump(doc, open("docker-compose.yml", "w"))
78 +```
79 +
80 +Multi-document write:
81 +
82 +```python
83 +yaml.safe_dump_all(docs, open("manifests.yaml", "w"),
84 + default_flow_style=False, sort_keys=False)
85 +```
86 +
87 +## Validate against a schema
88 +
89 +For structural guarantees use jsonschema (`pip install jsonschema`) on the loaded data — YAML loads to the same shapes JSON Schema describes:
90 +
91 +```python
92 +import yaml, jsonschema
93 +
94 +schema = {
95 + "type": "object",
96 + "required": ["server"],
97 + "properties": {
98 + "server": {
99 + "type": "object",
100 + "required": ["port"],
101 + "properties": {"port": {"type": "integer",
102 + "minimum": 1, "maximum": 65535}},
103 + }
104 + },
105 +}
106 +data = yaml.safe_load(open("config.yaml"))
107 +jsonschema.validate(data, schema) # raises ValidationError with a JSON path
108 +```
109 +
110 +## Convert YAML ↔ JSON
111 +
112 +```python
113 +import json, yaml
114 +
115 +# YAML → JSON
116 +json.dump(yaml.safe_load(open("config.yaml")), open("config.json", "w"),
117 + indent=2, ensure_ascii=False)
118 +
119 +# JSON → YAML
120 +yaml.safe_dump(json.load(open("config.json")), open("config.yaml", "w"),
121 + default_flow_style=False, sort_keys=False, allow_unicode=True)
122 +```
123 +
124 +Caveat: JSON has no equivalent for YAML anchors, multi-doc streams, or non-string keys — conversion resolves/loses them.
125 +
126 +## Gotchas (Norway problem, implicit typing, anchors)
127 +
128 +- **The Norway problem.** In YAML 1.1 (what PyYAML implements), unquoted `no`, `yes`, `on`, `off`, `y`, `n` parse as booleans — `country: NO` becomes `country: False`. Quote them. Same family: `port: 022` is octal 18, `version: 1.10` is the float `1.1`.
129 +- **Timestamps auto-convert.** `date: 2026-08-05` loads as `datetime.date`, not a string. Quote if you need the text.
130 +- **`safe_load` of an empty file returns `None`**, not `{}` — guard with `or {}`.
131 +- **Duplicate keys don't error** in PyYAML — the last one silently wins. ruamel.yaml raises; use it when duplicates would be a bug.
132 +- **Anchors/aliases (`&base`, `*base`, `<<:` merge)** load fine but PyYAML re-dumps them expanded (aliases only re-emitted when the same object identity repeats). If the user's file relies on anchors for maintainability, edit with ruamel.yaml.
133 +- **`sort_keys` defaults to True** in `safe_dump` — it silently alphabetizes configs; always pass `sort_keys=False` when round-tripping.
134 +- **Long strings get folded** with line breaks on dump; pass `width=4096` to keep long values (URLs, tokens) on one line.
135 +- **YAML 1.2 vs 1.1.** ruamel.yaml follows 1.2 (only `true`/`false` are booleans); PyYAML follows 1.1. The same file can load differently across the two libraries — pick one per task and stay with it.
added finance-skills/README.md +52 −0
@@ -0,0 +1,52 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# finance-skills — US/Canada Tax & Accounting Skill Collection
7 +
8 +**Author:** Simon-Pierre Boucher
9 +**Contact:** contact@spboucher.ai
10 +
11 +Ten ultra-sharp skills covering US and Canadian taxation, small-business
12 +accounting, financial statements, and legal tax optimization, following the
13 +method in [../RESEARCH-SYNTHESIS.md](../RESEARCH-SYNTHESIS.md) and grounded
14 +in web research with tax-year-2026 figures. All pass
15 +`python3 ../tools/validate_skills.py`.
16 +
17 +## Collection-wide safety rules
18 +
19 +Every skill in this collection enforces three limits:
20 +1. **Educational help, not professional advice** — complex situations are
21 + referred to a CPA/EA (US) or CPA (Canada).
22 +2. **Figures rot** — every dollar amount, rate, and deadline is stamped with
23 + its tax year and must be verified against irs.gov / canada.ca before use.
24 +3. **Legal planning only** — evasion (unreported income, fabricated
25 + deductions, falsified records) is out of scope and refused.
26 +
27 +## The collection and its boundaries
28 +
29 +**Personal taxation**
30 +| Skill | Handles | Explicitly does NOT handle |
31 +|---|---|---|
32 +| `preparing-us-personal-tax-returns` | Form 1040, schedules, deadlines, estimated-tax safe harbors | Canadian returns; business entities; planning strategy |
33 +| `preparing-canadian-personal-tax-returns` | T1, slips, deductions vs credits, deadlines, instalments | US returns; corporate T2; planning strategy |
34 +| `optimizing-us-personal-taxes` | account priority, Roth vs traditional, harvesting, bunching | return prep; business planning; Canadian planning |
35 +| `optimizing-canadian-personal-taxes` | RRSP/TFSA/FHSA decisions, legal splitting, superficial-loss | T1 prep; corporate planning; US planning |
36 +| `handling-cross-border-taxation` | residency tests, treaty tie-breakers, FTC, FBAR/8938, TFSA-for-US-persons trap | single-country returns; corporate structuring |
37 +
38 +**Business accounting & taxation**
39 +| Skill | Handles | Explicitly does NOT handle |
40 +|---|---|---|
41 +| `bookkeeping-for-small-businesses` | chart of accounts, double entry, monthly close, reconciliation | formal statements; tax filings |
42 +| `preparing-financial-statements` | income statement, balance sheet, cash flow; tie-out; GAAP/ASPE basis | bookkeeping; tax filings; audited public reporting |
43 +| `filing-us-business-taxes` | Schedule C / 1065 / 1120-S / 1120, estimated taxes, payroll forms, 1099s | personal-only 1040; Canadian filings; planning |
44 +| `filing-canadian-business-taxes` | T2125, T2, GST/HST, payroll remittances, T4/T5 slips | personal-only T1; US filings; planning |
45 +| `optimizing-business-taxes` | US: entity election, retirement plans, depreciation timing · Canada: SBD, salary vs dividends, CCA | filing the returns; personal planning |
46 +
47 +## Shared conventions
48 +
49 +- Description = WHAT + "Use when …" (literal phrases) + "Do not use for …"
50 +- "Current figures (tax year 2026 — verify before use)" table with official source URLs in every applicable skill
51 +- Every workflow ends with a validation step (checklist completeness, statements tie-out, both-scenarios math)
52 +- `SKILL.md` <150 lines; depth in `references/reference.md` (with TOC)
added finance-skills/bookkeeping-for-small-businesses/SKILL.md +54 −0
@@ -0,0 +1,54 @@
1 +---
2 +name: bookkeeping-for-small-businesses
3 +description: Sets up and maintains double-entry books for a small business — chart of accounts, journal entries, debits/credits, monthly close, and reconciliation. Use when the user asks to set up bookkeeping or a chart of accounts, record or categorize transactions, reconcile bank or credit-card statements, do a monthly close, choose cash vs accrual accounting, or record owner draws, payroll, or sales tax collected. Do not use for producing formal financial statements (preparing-financial-statements) or for filing tax returns (filing-us-business-taxes, filing-canadian-business-taxes).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Bookkeeping for Small Businesses
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** setting up books, recording/categorizing transactions, reconciliations, monthly close, cash-vs-accrual decisions, owner-compensation entries, sales-tax bookkeeping (US and Canada).
15 +- **Do NOT use for:** building the income statement / balance sheet / cash flow (→ `preparing-financial-statements`) or preparing tax filings (→ the filing skills).
16 +
17 +## Important limits
18 +- Educational help, not professional accounting or tax advice — complex or high-stakes cases go to a CPA.
19 +- Figures and deadlines are tax-year-stamped and MUST be verified against irs.gov / canada.ca before use.
20 +- Never assist with evasion, backdating, or falsified records; refuse and explain.
21 +
22 +## Core rules
23 +
24 +1. **Chart of accounts: 5 root types, numbered ranges.** 1000s assets, 2000s liabilities, 3000s equity, 4000s revenue, 5000s+ expenses. Start small (~25 accounts); add only when a category is reused.
25 +2. **Debit/credit rules, stated plainly.** Debits increase assets and expenses; credits increase liabilities, equity, and revenue. Every entry balances: total debits = total credits, no exceptions.
26 + - ✅ `Dr Equipment 2,000 / Cr Cash 2,000`
27 + - ❌ A one-sided "expense" line with no offsetting account
28 +3. **Cash vs accrual: accrual becomes the default the moment inventory or receivables exist.** Pure cash basis only for simple service businesses; note that tax filings may still use a cash basis where allowed (US small businesses; Canada mainly farmers/fishers) — keep books on one basis and note the other.
29 +4. **Separate business and personal money absolutely.** Dedicated bank account + card from day one. Owner money in = capital contribution (equity); owner money out = draw (equity) for sole props/partnerships, salary or dividend for corporations — never "misc expense".
30 + - ✅ `Dr Owner's Draw / Cr Cash` for a sole prop owner withdrawal
31 + - ❌ Booking the owner's groceries to Office Expense
32 +5. **Sales tax collected is a LIABILITY, not revenue.** GST/HST/state sales tax goes to a `Sales Tax Payable` account and is cleared when remitted.
33 + - ✅ Sale $100 + $13 HST → `Dr Cash 113 / Cr Revenue 100 / Cr HST Payable 13`
34 + - ❌ `Cr Revenue 113`
35 +6. **Reconcile monthly, to the penny.** Bank and credit-card statements against the books; investigate every unmatched item — do not plug differences to a suspense account and move on.
36 +7. **Keep the paper.** Receipt/invoice for every entry. Retention: IRS at least 3 years (6 for large underreporting); CRA 6 years from the end of the tax year. Digital copies acceptable in both countries.
37 +
38 +## Monthly close workflow
39 +
40 +1. Import/enter all transactions; sweep the uncategorized list to zero.
41 +2. Reconcile every bank and credit-card account to its statement.
42 +3. Review AR aging (chase >30 days) and AP aging (schedule payments).
43 +4. Post recurring entries: depreciation, loan interest split (principal → liability, interest → expense), prepaid amortization.
44 +5. Verify sales-tax payable matches the filing-period report.
45 +6. **Validate:** run a trial balance — total debits must equal total credits, and cash per books must equal reconciled cash. If either fails, fix before closing the month.
46 +
47 +## Edge cases & failure modes
48 +- **Mixed personal/business card in the past** → reclassify personal items to draws/contributions; do not delete transactions.
49 +- **Missing receipts** → record the transaction anyway with a note; flag for the owner to source documentation.
50 +- **Loan payments** → never expense the full payment; split principal/interest per the amortization schedule.
51 +- **Refunds** → reverse against the original revenue/expense account, not a new one.
52 +
53 +## References
54 +Sample chart of accounts, worked journal entries, and gotchas: see [references/reference.md](references/reference.md).
added finance-skills/bookkeeping-for-small-businesses/references/reference.md +119 −0
@@ -0,0 +1,119 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Reference — Bookkeeping for Small Businesses
7 +
8 +## Contents
9 +- Sample chart of accounts
10 +- Debit/credit cheat table
11 +- Worked journal entries
12 +- Owner compensation by entity type
13 +- Retention rules (US / Canada)
14 +- Gotchas
15 +
16 +## Sample chart of accounts
17 +
18 +| # | Account | Type |
19 +|---|---|---|
20 +| 1000 | Cash — Operating | Asset |
21 +| 1100 | Accounts Receivable | Asset |
22 +| 1200 | Inventory | Asset |
23 +| 1300 | Prepaid Expenses | Asset |
24 +| 1500 | Equipment | Asset |
25 +| 1510 | Accumulated Depreciation — Equipment | Contra-asset |
26 +| 2000 | Accounts Payable | Liability |
27 +| 2100 | Credit Card Payable | Liability |
28 +| 2200 | Sales Tax Payable (GST/HST/state) | Liability |
29 +| 2300 | Payroll Liabilities | Liability |
30 +| 2500 | Loan Payable | Liability |
31 +| 3000 | Owner's Capital / Common Shares | Equity |
32 +| 3100 | Owner's Draws / Dividends Declared | Equity |
33 +| 3900 | Retained Earnings | Equity |
34 +| 4000 | Sales Revenue | Revenue |
35 +| 4900 | Refunds and Discounts | Contra-revenue |
36 +| 5000 | Cost of Goods Sold | Expense |
37 +| 6000 | Rent | Expense |
38 +| 6100 | Salaries and Wages | Expense |
39 +| 6200 | Payroll Taxes (employer share) | Expense |
40 +| 6300 | Software and Subscriptions | Expense |
41 +| 6400 | Professional Fees | Expense |
42 +| 6500 | Insurance | Expense |
43 +| 6600 | Depreciation Expense | Expense |
44 +| 6700 | Interest Expense | Expense |
45 +
46 +## Debit/credit cheat table
47 +
48 +| Account type | Increase | Decrease | Normal balance |
49 +|---|---|---|---|
50 +| Asset | Debit | Credit | Debit |
51 +| Expense | Debit | Credit | Debit |
52 +| Liability | Credit | Debit | Credit |
53 +| Equity | Credit | Debit | Credit |
54 +| Revenue | Credit | Debit | Credit |
55 +
56 +## Worked journal entries
57 +
58 +**Credit sale with 13% HST (Ontario):**
59 +```text
60 +Dr 1100 Accounts Receivable 1,130
61 + Cr 4000 Sales Revenue 1,000
62 + Cr 2200 Sales Tax Payable 130
63 +```
64 +
65 +**Customer pays the invoice:**
66 +```text
67 +Dr 1000 Cash 1,130
68 + Cr 1100 Accounts Receivable 1,130
69 +```
70 +
71 +**Remitting the sales tax:**
72 +```text
73 +Dr 2200 Sales Tax Payable 130
74 + Cr 1000 Cash 130
75 +```
76 +
77 +**Monthly loan payment $500 ($420 principal, $80 interest):**
78 +```text
79 +Dr 2500 Loan Payable 420
80 +Dr 6700 Interest Expense 80
81 + Cr 1000 Cash 500
82 +```
83 +
84 +**Monthly depreciation (straight line, $12,000 equipment / 5 years):**
85 +```text
86 +Dr 6600 Depreciation Expense 200
87 + Cr 1510 Accumulated Depreciation 200
88 +```
89 +
90 +**Prepaid annual insurance $1,200, monthly recognition:**
91 +```text
92 +At payment: Dr 1300 Prepaid Expenses 1,200 / Cr 1000 Cash 1,200
93 +Each month: Dr 6500 Insurance 100 / Cr 1300 Prepaid Expenses 100
94 +```
95 +
96 +## Owner compensation by entity type
97 +
98 +| Entity | Money out is recorded as | Notes |
99 +|---|---|---|
100 +| US sole prop / SMLLC | Owner's Draw (equity) | Not an expense; not payroll |
101 +| US partnership | Partner Draw (equity) per partner | Guaranteed payments are an expense |
102 +| US S-corp | Salary (payroll expense) + distributions (equity) | Reasonable salary required before distributions |
103 +| US C-corp | Salary (expense) and/or dividends (equity) | Dividends are not deductible |
104 +| Canadian sole prop | Owner's Draw (equity) | Taxed via T2125 on the T1 regardless of draws |
105 +| Canadian corporation (CCPC) | Salary (expense, T4) or dividends (equity, T5) | Choice is a planning question — see optimizing-business-taxes |
106 +
107 +## Retention rules
108 +
109 +| Country | Rule | Source |
110 +|---|---|---|
111 +| US | Keep records ≥3 years from filing (6 if income underreported >25%) | irs.gov — Recordkeeping |
112 +| Canada | Keep records 6 years from the end of the last tax year they relate to | canada.ca — Keeping records |
113 +
114 +## Gotchas
115 +- **Transfers between own accounts are not income or expense** — book as transfers, or revenue is overstated.
116 +- **Credit-card payments are not expenses** — the expense was recorded at purchase; the payment clears 2100.
117 +- **Sales-tax input credits (Canada):** GST/HST paid on purchases is recoverable for registrants — track in an ITC account, don't bury it in the expense.
118 +- **Negative liability or negative asset balances** in the trial balance almost always mean a mis-signed entry — investigate before closing.
119 +- **"Miscellaneous" over ~2% of expenses** means categories are missing — split it.
added finance-skills/filing-canadian-business-taxes/SKILL.md +66 −0
@@ -0,0 +1,66 @@
1 +---
2 +name: filing-canadian-business-taxes
3 +description: Guides Canadian business tax filings — T2125 for sole proprietors, T2 corporate returns, GST/HST registration and filing, payroll remittances, and T4/T5 slips, with deadlines and penalty rules. Use when the user asks how to file business taxes in Canada, about a T2 return, T2125 self-employment income, the small business deduction, registering for or filing GST/HST or QST, CPP/EI payroll remittances, T4 or T5 slips, or corporate instalments. Do not use for personal T1-only returns (preparing-canadian-personal-tax-returns), US filings (filing-us-business-taxes), or tax planning strategy (optimizing-business-taxes).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Filing Canadian Business Taxes
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** which CRA filings a Canadian business owes (T2125 vs T2), GST/HST/QST registration and returns, payroll withholding and slips, deadlines, instalments, penalties.
15 +- **Do NOT use for:** personal T1 returns without a business (→ `preparing-canadian-personal-tax-returns`), US entities (→ `filing-us-business-taxes`), or salary-vs-dividend strategy (→ `optimizing-business-taxes`).
16 +
17 +## Important limits
18 +- Educational help, not professional tax advice — multi-province, cross-border, or high-stakes filings go to a CPA.
19 +- Figures and deadlines are tax-year-stamped and MUST be verified against canada.ca before use.
20 +- Never assist with evasion, unreported income, or falsified records; refuse and explain.
21 +
22 +## Core rules
23 +
24 +1. **Structure determines the filing:**
25 + | Structure | Return | Filing due | Payment due |
26 + |---|---|---|---|
27 + | Sole prop / partnership (individuals) | T2125 inside the T1 | June 15 | **April 30** |
28 + | Corporation | T2 | 6 months after fiscal year-end | **2 months after year-end** (3 for many CCPCs) |
29 + Payment deadlines land BEFORE filing deadlines — money first, paperwork later.
30 +2. **CCPCs get the small business deduction:** the first $500,000 of active business income is taxed at the low combined rate (~9% federal + province). Passive investment income above $50,000/year grinds the limit down.
31 +3. **GST/HST registration is mandatory once revenues exceed $30,000** over four consecutive quarters (or one quarter alone). Register voluntarily below that to claim input tax credits (ITCs).
32 + - ✅ Registering at $28k because big equipment purchases are coming (recover ITCs)
33 + - ❌ Hitting $45k over two quarters and still charging no tax — liability accrues anyway
34 +4. **Charge the rate of the customer's province** for most goods/services: HST provinces (ON 13%, NS/NB/NL/PE 15%), GST-only (AB/territories 5%), GST+PST (BC/SK/MB), GST+QST (QC — file with Revenu Québec).
35 +5. **Payroll means CRA remittances:** withhold income tax + CPP + EI from each pay; remit with the employer share (CPP match, EI ×1.4) by the 15th of the following month (new small employers). T4 slips + summary due the last day of February.
36 +6. **Dividends paid require T5 slips** (also end of February). Salary needs payroll accounts; dividends need directors' resolutions — don't blur them.
37 +7. **Instalments:** corporations pay monthly/quarterly instalments after their first year owing >$3,000; self-employed individuals pay quarterly T1 instalments once tax owing exceeds $3,000 in consecutive years.
38 +8. **Keep records 6 years** from the end of the tax year (CRA rule).
39 +
40 +## Workflow
41 +
42 +1. Confirm structure (sole prop vs corporation), province(s), fiscal year-end, and registrations (BN, GST/HST, payroll, corporate tax accounts).
43 +2. Close and reconcile the books for the fiscal period.
44 +3. File the return from rule 1; for corporations include the GIFI (financial-statement codes) mapping.
45 +4. File the GST/HST return for the period: tax collected − ITCs = net remittance (or refund).
46 +5. Verify slips: T4s match payroll ledger; T5s match dividends declared.
47 +6. **Validate:** deadline calendar cross-checked against canada.ca; payment dates (which precede filing dates) scheduled; GST/HST collected account cleared by the remittance.
48 +
49 +## Current figures (tax year 2026 — verify before use)
50 +
51 +| Item | Figure | Source |
52 +|---|---|---|
53 +| Small business deduction limit | $500,000 active income (CCPC) | canada.ca |
54 +| Passive-income grind threshold | $50,000/year | canada.ca |
55 +| GST/HST registration threshold | $30,000 / 4 consecutive quarters | canada.ca/gst-hst |
56 +| GST rate / HST examples | 5% GST; ON 13%; Atlantic 15% | canada.ca/gst-hst-rates |
57 +| Instalment threshold | >$3,000 owing (fed) | canada.ca |
58 +
59 +## Edge cases & failure modes
60 +- **Quebec** files separately with Revenu Québec (QST, provincial return, payroll) — flag it explicitly.
61 +- **First corporate year** → no instalments required, but the full balance is due 2–3 months after year-end; cash-plan for it.
62 +- **Late T4/T5 slips** → per-slip penalties; file late rather than never.
63 +- **Zero-rated vs exempt sales:** zero-rated (exports, basic groceries) still allow ITCs; exempt (residential rent, most financial services) do not — misclassification changes refunds.
64 +
65 +## References
66 +Deadline/rate tables, GST/HST worked example, and gotchas: see [references/reference.md](references/reference.md).
added finance-skills/filing-canadian-business-taxes/references/reference.md +82 −0
@@ -0,0 +1,82 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Reference — Filing Canadian Business Taxes
7 +
8 +## Contents
9 +- Deadline master table (tax year 2026)
10 +- Sales tax by province
11 +- GST/HST worked example
12 +- Payroll remittance table
13 +- Corporate tax rate sketch
14 +- Penalty table
15 +- Gotchas
16 +
17 +## Deadline master table (tax year 2026 — verify at canada.ca)
18 +
19 +| Filing | Who | Due |
20 +|---|---|---|
21 +| T1 + T2125 | Self-employed individuals | File June 15; **pay April 30** |
22 +| T2 | Corporations | File 6 months after year-end; **pay 2 months** (3 for eligible CCPCs) |
23 +| GST/HST return | Registrants | Annual: 3 months after year-end (self-employed: June 15/pay Apr 30); quarterly/monthly: 1 month after period |
24 +| T4 slips + summary | Employers | Last day of February |
25 +| T5 slips + summary | Corporations paying dividends | Last day of February |
26 +| Payroll remittance | New/small employers | 15th of the month following payday |
27 +| T1 instalments | Individuals owing >$3,000 | Mar 15 / Jun 15 / Sep 15 / Dec 15 |
28 +| RRSP deduction cutoff | Individuals | ~60 days into the new year (Mar 2, 2026 for 2025) |
29 +
30 +## Sales tax by province (2026 — verify)
31 +
32 +| Province | Regime | Rate |
33 +|---|---|---|
34 +| ON | HST | 13% |
35 +| NB, NS, NL, PE | HST | 15% |
36 +| AB, NT, NU, YT | GST only | 5% |
37 +| BC | GST + PST | 5% + 7% |
38 +| SK | GST + PST | 5% + 6% |
39 +| MB | GST + RST | 5% + 7% |
40 +| QC | GST + QST | 5% + 9.975% (file with Revenu Québec) |
41 +
42 +## GST/HST worked example (Ontario, quarterly filer)
43 +
44 +- Sales in quarter: $50,000 + 13% HST collected = $6,500
45 +- Purchases: $12,000 + $1,560 HST paid (ITCs)
46 +- **Net remittance = $6,500 − $1,560 = $4,940**, due one month after quarter-end.
47 +
48 +Books: HST collected accumulates in `Sales Tax Payable`; ITCs in `GST/HST Recoverable`; the remittance clears both.
49 +
50 +## Payroll remittance per pay run (2026 figures — verify)
51 +
52 +| Item | Employee | Employer |
53 +|---|---|---|
54 +| Income tax | withheld per TD1 tables | — |
55 +| CPP | withheld up to annual max | matched 1.0× |
56 +| EI | withheld up to annual max | 1.4× employee premium |
57 +
58 +Remit all of the above together by the 15th of the following month (accelerated schedules for larger remitters).
59 +
60 +## Corporate tax rate sketch (2026 — verify exact provincial rates)
61 +
62 +| Income | Federal | Combined typical |
63 +|---|---|---|
64 +| Active income ≤ $500k (CCPC, SBD) | 9% | ~11–12% with province |
65 +| Active income > $500k | 15% | ~25–31% |
66 +| Passive investment income | ~38.7% refundable regime | refunds on dividend payout (RDTOH) |
67 +
68 +## Penalty table (verify at canada.ca)
69 +
70 +| Failure | Penalty |
71 +|---|---|
72 +| Late T1/T2 filing | 5% of balance + 1%/month (max 12); doubles for repeat offenders |
73 +| Late GST/HST | 1% of amount owing + 0.25%/month (max 12) |
74 +| Late slips (T4/T5) | Per-slip penalty scaled by count and days late |
75 +| Missed payroll remittance | 3–10% of the amount, escalating |
76 +
77 +## Gotchas
78 +- **Payment before filing:** corporate balances are due months before the T2 itself; treating the filing date as the payment date accrues interest silently.
79 +- **The $30,000 GST/HST threshold is rolling** (four consecutive quarters), not calendar-year — check every quarter.
80 +- **Place-of-supply rules:** an Alberta business selling to Ontario customers generally charges 13% HST, not 5%.
81 +- **Director liability:** unremitted payroll and GST/HST amounts are personal liabilities of directors — prioritize trust amounts over all other debts.
82 +- **GIFI codes:** the T2 requires financial statements mapped to GIFI; keep the chart of accounts mappable.
added finance-skills/filing-us-business-taxes/SKILL.md +66 −0
@@ -0,0 +1,66 @@
1 +---
2 +name: filing-us-business-taxes
3 +description: Guides US business tax filings by entity type — Schedule C, Form 1065, 1120-S, 1120 — plus estimated taxes, payroll forms, and 1099s, with deadlines and penalty rules. Use when the user asks how to file business taxes in the US, which form their LLC, S-corp, C-corp, partnership, or sole proprietorship files, about quarterly estimated taxes, self-employment tax, K-1s, 941/940 payroll filings, W-2s, 1099-NEC, or business tax extensions. Do not use for personal 1040-only returns (preparing-us-personal-tax-returns), Canadian filings (filing-canadian-business-taxes), or tax planning strategy (optimizing-business-taxes).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Filing US Business Taxes
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** which federal return an entity files, filing/payment deadlines, estimated taxes, payroll and contractor reporting, extensions and penalties.
15 +- **Do NOT use for:** personal returns without a business (→ `preparing-us-personal-tax-returns`), Canadian entities (→ `filing-canadian-business-taxes`), or choosing/optimizing structure (→ `optimizing-business-taxes`).
16 +
17 +## Important limits
18 +- Educational help, not professional tax advice — multi-state, international, or high-stakes filings go to a CPA/EA.
19 +- Figures and deadlines are tax-year-stamped and MUST be verified against irs.gov before use.
20 +- Never assist with evasion, unreported income, or falsified records; refuse and explain.
21 +
22 +## Core rules
23 +
24 +1. **The entity determines the return:**
25 + | Entity | Return | Owner receives | Federal due date |
26 + |---|---|---|---|
27 + | Sole prop / single-member LLC | Schedule C on Form 1040 (+ Schedule SE) | — | April 15 |
28 + | Partnership / multi-member LLC | Form 1065 | Schedule K-1 | March 15 |
29 + | S-corporation | Form 1120-S | Schedule K-1 (+ W-2 salary) | March 15 |
30 + | C-corporation | Form 1120 (21% flat rate) | W-2 and/or dividends | April 15 |
31 +2. **Pass-through returns are due a month early (March 15)** so K-1s reach owners before their 1040s. Missing a 1065/1120-S deadline triggers a per-partner, per-month penalty even with no tax due.
32 +3. **Self-employment tax rides with Schedule C:** 15.3% (Social Security + Medicare) on net self-employment earnings via Schedule SE, on top of income tax; half is deductible.
33 +4. **S-corp owners who work in the business must take a reasonable W-2 salary** through payroll before distributions.
34 + - ✅ $70k salary + $50k distributions with payroll filings
35 + - ❌ $0 salary, all distributions — a classic audit trigger
36 +5. **Estimated taxes are quarterly:** April 15, June 15, September 15, January 15. Safe harbor: pay 100% of last year's tax (110% if AGI >$150k) or 90% of the current year to avoid penalties.
37 +6. **Employees mean payroll filings:** Form 941 quarterly (withholding + FICA), Form 940 annually (FUTA), W-2s to employees and the SSA by January 31, plus state equivalents.
38 +7. **Contractors paid ≥$600 get a 1099-NEC by January 31.** Collect a W-9 before first payment, not at year-end.
39 +8. **Extensions extend filing, never payment.** Form 7004 (businesses) / 4868 (individuals) gives ~6 months to file; tax owed is still due at the original deadline. Late-FILE penalties (5%/month, max 25%) dwarf late-PAY (0.5%/month) — **always file on time even if you cannot pay.**
40 +
41 +## Workflow
42 +
43 +1. Confirm the entity type and any elections (e.g., LLC taxed as S-corp via Form 2553).
44 +2. Close the books for the tax year; reconcile income to bank deposits and books.
45 +3. Select the return from the table (rule 1) and its state counterpart(s).
46 +4. Compute and schedule estimated payments for the coming year (rule 5).
47 +5. Verify information returns: W-2s/941s consistent with wage expense; 1099-NECs issued.
48 +6. **Validate before filing:** deadline calendar cross-checked against irs.gov; book income reconciled to return income (Schedule M-1 for corporations); all K-1s issued.
49 +
50 +## Current figures (tax year 2026 — verify before use)
51 +
52 +| Item | Figure | Source |
53 +|---|---|---|
54 +| C-corp federal rate | 21% flat | irs.gov |
55 +| SE tax rate | 15.3% on net SE earnings | irs.gov/schedule-se |
56 +| 1099-NEC threshold | $600 | irs.gov/form-1099-nec |
57 +| Estimated-tax safe harbor | 100%/110% prior year or 90% current | irs.gov/form-1040-es |
58 +
59 +## Edge cases & failure modes
60 +- **LLC ≠ a tax status.** An LLC files as sole prop, partnership, S-corp, or C-corp depending on members/elections — always ask which.
61 +- **First year, no prior-year tax** → safe harbor based on prior year is $0; still project and pay 90% current-year to be safe.
62 +- **Missed 1099s** → file late anyway; penalties scale with lateness.
63 +- **State taxes vary widely** (franchise taxes, gross-receipts taxes) — flag the state and direct to its authority; this skill covers federal.
64 +
65 +## References
66 +Entity/form/deadline tables, penalty table, and gotchas: see [references/reference.md](references/reference.md).
added finance-skills/filing-us-business-taxes/references/reference.md +73 −0
@@ -0,0 +1,73 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Reference — Filing US Business Taxes
7 +
8 +## Contents
9 +- Form and deadline master table (tax year 2026)
10 +- Penalty table
11 +- Estimated-tax worksheet pattern
12 +- Payroll filing calendar
13 +- Common elections
14 +- Gotchas
15 +
16 +## Form and deadline master table (tax year 2026 — verify at irs.gov)
17 +
18 +| Filing | Who | Due | Extension (via 7004/4868) |
19 +|---|---|---|---|
20 +| Form 1040 + Schedule C + SE | Sole props, SMLLCs | April 15 | Oct 15 (file only) |
21 +| Form 1065 + K-1s | Partnerships, MMLLCs | March 15 | Sept 15 |
22 +| Form 1120-S + K-1s | S-corps | March 15 | Sept 15 |
23 +| Form 1120 | C-corps (calendar year) | April 15 | Oct 15 |
24 +| Form 941 | Employers, quarterly | Apr 30 / Jul 31 / Oct 31 / Jan 31 |— |
25 +| Form 940 (FUTA) | Employers, annual | Jan 31 | — |
26 +| W-2 / W-3 | Employers | Jan 31 (to employees and SSA) | — |
27 +| 1099-NEC | Payers of contractors ≥$600 | Jan 31 | — |
28 +| Estimated taxes (1040-ES / 1120-W) | Self-employed, corps | Apr 15 / Jun 15 / Sep 15 / Jan 15 | — |
29 +
30 +Fiscal-year C-corps: return due the 15th day of the 4th month after year-end.
31 +
32 +## Penalty table (verify current amounts at irs.gov)
33 +
34 +| Failure | Penalty |
35 +|---|---|
36 +| Late filing (income tax) | 5% of unpaid tax per month, max 25% |
37 +| Late payment | 0.5% per month, max 25% (plus interest) |
38 +| Late 1065/1120-S | Per-partner/shareholder, per-month flat penalty — applies even with zero tax |
39 +| Missing/late 1099 or W-2 | Tiered per-form penalty rising with lateness |
40 +| Payroll deposit late | 2–15% of the deposit depending on days late |
41 +
42 +Rule of thumb encoded in the skill: filing on time without paying costs 10× less than not filing.
43 +
44 +## Estimated-tax worksheet pattern
45 +
46 +1. Project net income for the year (books YTD × remaining months, adjusted for seasonality).
47 +2. Compute income tax + SE tax (15.3% up to the Social Security wage base; 2.9% Medicare above; 0.9% additional Medicare over $200k single).
48 +3. Subtract withholding (if any W-2 income).
49 +4. Divide by 4; adjust remaining vouchers if income shifts mid-year.
50 +5. Safe harbor check: payments ≥ 100% (110% high earners) of last year's total tax avoids penalties regardless of this year's outcome.
51 +
52 +## Payroll filing calendar (employer with staff)
53 +
54 +| When | What |
55 +|---|---|
56 +| Each pay run | Withhold federal income tax + employee FICA; accrue employer FICA |
57 +| Semiweekly/monthly | Deposit withheld taxes (schedule set by lookback period) |
58 +| Quarterly | Form 941 |
59 +| Jan 31 | Form 940, W-2s to employees + SSA, 1099-NECs |
60 +
61 +## Common elections
62 +
63 +| Election | Form | Deadline |
64 +|---|---|---|
65 +| LLC → S-corp taxation | Form 2553 | ~2.5 months into the tax year it takes effect |
66 +| LLC → C-corp taxation | Form 8832 | Effective date within 75 days back / 12 months forward |
67 +
68 +## Gotchas
69 +- **K-1 timing:** partners cannot file accurate 1040s until the 1065/1120-S is done — a late business return cascades into late personal returns.
70 +- **Distributions are not payroll:** S-corp distributions without reasonable salary invite reclassification + back payroll taxes + penalties.
71 +- **Home-state nexus:** hiring a remote employee in another state usually creates payroll and possibly income-tax obligations there.
72 +- **Q2 estimated payment is June 15** — only two months after Q1, not three; cash-flow plan for it.
73 +- **Schedule M-1/M-2:** book-tax differences must reconcile on corporate returns; "plug" numbers get flagged.
added finance-skills/handling-cross-border-taxation/SKILL.md +56 −0
@@ -0,0 +1,56 @@
1 +---
2 +name: handling-cross-border-taxation
3 +description: Guides US-Canada cross-border personal tax situations — residency determination, treaty tie-breakers, foreign tax credits, dual filing for US citizens in Canada, FBAR and Form 8938 reporting, TFSA/RRSP treatment, and snowbird rules. Use when the user asks about working, moving, or living across the US-Canada border, dual citizenship taxes, tax residency in two countries, double taxation, FBAR, the substantial presence test, or whether a TFSA or RRSP is taxed by the IRS. Do not use for single-country returns (preparing-us-personal-tax-returns, preparing-canadian-personal-tax-returns) or for corporate cross-border structuring.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Handling US-Canada Cross-Border Taxation
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** determining which country can tax whom, sequencing dual filings, foreign tax credits, US information reporting (FBAR/8938), registered-account treatment across the border, snowbirds and remote workers.
15 +- **Do NOT use for:** preparing either country's return mechanics (use the preparing-* skills once obligations are established), corporate structuring, immigration advice, or third-country situations.
16 +
17 +## Important limits
18 +- Educational guidance, not professional tax advice — cross-border is the highest-referral area: renunciation, departure tax, PFIC holdings, and treaty elections go to a cross-border CPA/EA.
19 +- All figures are stated for **tax year 2026** and MUST be verified against [irs.gov](https://www.irs.gov) and [canada.ca](https://www.canada.ca/en/revenue-agency.html) before use.
20 +- Never assist with evasion — hiding accounts or income from either tax authority is out of scope; disclosure programs (IRS Streamlined, CRA Voluntary Disclosures) are the legal path for past non-compliance.
21 +
22 +## Workflow
23 +1. **Determine residency first — everything follows from it.**
24 + - US: citizen or green-card holder → US tax resident regardless of location (citizenship-based taxation). Otherwise apply the substantial presence test: days this year + ⅓ last year + ⅙ year before ≥ 183 (and ≥ 31 days this year).
25 + - Canada: facts-and-circumstances residential ties — home, spouse/dependants in Canada are primary; secondary ties (accounts, licences, health card) accumulate.
26 +2. If resident of **both**, apply the treaty tie-breaker in strict order: permanent home → centre of vital interests → habitual abode → citizenship → competent authority.
27 +3. Establish the filing set: US citizens/green-card holders in Canada file **both** every year (1040 + T1). Treaty non-residents may still owe source-country filings (e.g., 1040-NR for US-source income).
28 +4. Eliminate double tax with **foreign tax credits, claimed in the residence country, per income category** — compute the source-country return first, then credit those taxes on the residence return.
29 +5. Screen US information reporting: FBAR (FinCEN 114) if aggregate non-US accounts exceeded $10,000 at any moment; Form 8938 at its higher thresholds; Form 3520/3520-A risk for TFSA/FHSA/RESP.
30 +6. Screen Canadian side: T1135 for foreign property over $100,000 cost; departure/arrival year rules (deemed disposition on emigration).
31 +7. **Validate:** residency conclusion documented with the facts used; every income item appears on the correct return(s); FTC claimed only in the residence country; all information forms listed with their deadlines; anything involving PFICs, departure tax, or renunciation flagged for a professional.
32 +
33 +## Current figures (tax year 2026 — verify before use)
34 +| Item | Value | Source |
35 +|---|---|---|
36 +| Substantial presence | 183-day weighted 3-year formula (min 31 current-year days) | irs.gov |
37 +| Closer connection (Form 8840) | Escape valve if < 183 current-year days + closer ties to Canada | irs.gov |
38 +| FBAR (FinCEN 114) | Aggregate non-US accounts > $10,000 at any time; filed with FinCEN, auto-extension to Oct 15 | fincen.gov |
39 +| Form 8938 | $50k/$100k (single/joint) US-resident year-end; $200k/$400k living abroad | irs.gov |
40 +| US filing abroad | Automatic extension to June 15 (interest from Apr 15) | irs.gov |
41 +| T1135 (Canada) | Foreign property cost > CAD $100,000 | canada.ca |
42 +
43 +## Registered accounts across the border (flag loudly)
44 +- **RRSP/RRIF:** treaty-recognized — US tax deferred; still reportable on FBAR/8938.
45 +- **TFSA/FHSA:** **NOT treaty-protected** — income is taxable currently on the US return, with possible Form 3520/3520-A trust filings. A US person holding a TFSA is usually a mistake; raise it immediately.
46 +- **Canadian mutual funds/ETFs held by US persons:** PFIC regime — punitive; refer to a professional.
47 +- **401(k)/IRA for Canadian residents:** treaty-deferred in Canada; withdrawals sourced to the US with withholding.
48 +
49 +## Edge cases & failure modes
50 +- **Snowbirds** near 183 weighted days → Form 8840 closer-connection statement, filed on time, every year.
51 +- **Remote worker in Canada for a US employer** → likely Canadian-resident: T1 on worldwide income, US obligations depend on citizenship/source; watch payroll-withholding mismatch.
52 +- **Departure from Canada** → deemed disposition (departure tax) on most property; date-of-departure return; refer out if large unrealized gains.
53 +- **Past non-compliance discovered** → IRS Streamlined Foreign Offshore / CRA Voluntary Disclosures — never advise silent catch-up filing.
54 +
55 +## References
56 +Treaty tie-breaker detail, filing matrices by profile, FTC mechanics, and reporting-form table: see [references/reference.md](references/reference.md).
added finance-skills/handling-cross-border-taxation/references/reference.md +92 −0
@@ -0,0 +1,92 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Reference — US-Canada Cross-Border Personal Taxation
7 +
8 +Figures are **tax year 2026** — verify at [irs.gov](https://www.irs.gov) / [canada.ca](https://www.canada.ca/en/revenue-agency.html) before use. Treaty = Canada-US Income Tax Convention.
9 +
10 +## Contents
11 +- Residency tests in detail
12 +- Treaty tie-breaker
13 +- Filing matrix by profile
14 +- Foreign tax credit mechanics
15 +- US information-reporting table
16 +- Canadian reporting and departure rules
17 +- Registered accounts matrix
18 +- Worked mini-examples
19 +- Gotchas
20 +
21 +## Residency tests in detail
22 +
23 +**US substantial presence test:** resident if current-year days ≥ 31 AND (current days + ⅓ × prior-year days + ⅙ × second-prior days) ≥ 183. Days present for a medical condition arising in the US, commuting days, and certain exempt statuses (F/J students within limits) don't count — check the exceptions before concluding.
24 +
25 +**Closer connection exception (Form 8840):** available only if current-year days < 183, a tax home in Canada exists, and a closer connection to Canada is shown. Must be filed timely each year — it is not automatic.
26 +
27 +**Canadian residency:** no day-count statute for ordinary residency; primary ties (dwelling available, spouse, dependants) dominate; secondary ties (bank accounts, provincial health, driver's licence, memberships) accumulate. A person with a home and family in Canada is resident almost regardless of travel. Sojourner rule: 183+ days physically in Canada in a year → deemed resident (before treaty).
28 +
29 +## Treaty tie-breaker (Article IV) — strict order, stop at first decisive step
30 +1. Permanent home available in only one country → resident there.
31 +2. Both/neither → centre of vital interests (personal + economic relations).
32 +3. Unclear → habitual abode.
33 +4. Both/neither → citizenship.
34 +5. Both/neither → competent-authority agreement.
35 +
36 +A treaty tie-break to Canada does not remove US **citizen** filing obligations — citizenship-based taxation survives the tie-breaker (saving clause).
37 +
38 +## Filing matrix by profile
39 +
40 +| Profile | US filings | Canadian filings |
41 +|---|---|---|
42 +| US citizen living in Canada | 1040 worldwide + FBAR/8938 (+ 3520s if TFSA/RESP) | T1 worldwide |
43 +| Canadian working in US (resident by SPT, no US citizenship) | 1040 worldwide | Departure-year T1 or non-resident T1 for Canadian-source income |
44 +| Canadian with US rental/investment income only | 1040-NR (US-source) | T1 worldwide + FTC for US tax |
45 +| Snowbird under thresholds | Form 8840 only (no 1040) | T1 worldwide |
46 +| Cross-border commuter (lives CA, works US) | 1040-NR on US wages | T1 worldwide + FTC |
47 +
48 +## Foreign tax credit mechanics
49 +- Claim the credit in the **residence** country for tax paid to the **source** country; prepare the source return first.
50 +- US side: Form 1116 per category (general, passive); credit limited to US tax on that foreign income; excess carries back 1 / forward 10 years.
51 +- Canadian side: federal + provincial foreign tax credit, computed per country; limited to Canadian tax on that income.
52 +- Social security taxes are handled by the **Totalization Agreement** (pay into one system, not both) — not by FTC.
53 +
54 +## US information-reporting table
55 +
56 +| Form | Trigger | Where/when | Penalty exposure |
57 +|---|---|---|---|
58 +| FBAR (FinCEN 114) | Aggregate non-US accounts > $10,000 any time | FinCEN, Apr 15 + auto Oct 15 | Severe, per-account; willful much worse |
59 +| Form 8938 | $50k/$100k resident; $200k/$400k abroad (year-end; higher any-time thresholds) | With 1040 | $10,000+ |
60 +| Form 3520/3520-A | Foreign trusts — TFSA/RESP risk | Separate deadlines | $10,000+ each |
61 +| Form 8621 | PFIC (Canadian mutual funds/ETFs) | With 1040 | Punitive tax regime itself |
62 +| Form 8833 | Treaty-based return positions | With 1040 | $1,000 |
63 +
64 +## Canadian reporting and departure rules
65 +- **T1135:** foreign property with total cost > CAD $100,000 (excludes personal-use property and registered accounts).
66 +- **Emigration:** deemed disposition of most capital property at FMV on departure (departure tax); exceptions for Canadian real property, RRSPs; election to defer with security. Date-of-departure T1 marks residency change.
67 +- **Immigration to Canada:** cost basis steps up to FMV on arrival — document valuations on entry day.
68 +
69 +## Registered accounts matrix
70 +
71 +| Account | Canada view | US view (US person) |
72 +|---|---|---|
73 +| RRSP/RRIF | Deferred | Treaty-deferred; report on FBAR/8938 |
74 +| TFSA | Tax-free | Fully taxable annually; likely 3520/3520-A |
75 +| FHSA | Deductible + tax-free | No treaty protection — taxable; trust-filing risk |
76 +| RESP | Deferred + grants | Taxable to US-person subscriber; 3520 risk |
77 +| 401(k)/IRA | Treaty-deferred for Canadian residents | Deferred |
78 +
79 +## Worked mini-examples
80 +
81 +**Example 1 — SPT arithmetic.** 130 days in 2026, 120 in 2025, 90 in 2024 → 130 + 40 + 15 = 185 ≥ 183 → US resident by SPT unless Form 8840 closer connection (130 < 183 ✓) is filed.
82 +
83 +**Example 2 — FTC direction.** US citizen resident in Canada earns Canadian salary. Canada taxes first (source + residence); the US 1040 reports the salary and claims Form 1116 credit for Canadian tax — usually reducing US tax to zero, but the return is still mandatory.
84 +
85 +**Example 3 — TFSA flag.** Dual citizen holds a $40,000 TFSA of Canadian ETFs: US-taxable income annually + PFIC (8621) + possible 3520 — three problems in one account; refer to a cross-border professional and consider unwinding.
86 +
87 +## Gotchas
88 +- The **saving clause** lets the US tax its citizens as if the treaty didn't exist (limited exceptions) — never tell a US citizen the tie-breaker ends their 1040 duty.
89 +- FBAR aggregates **all** accounts (chequing, TFSA, RRSP, even signing authority) — the $10,000 trigger is total, not per account.
90 +- Currency: US forms in USD (Treasury year-end/average rates), Canadian in CAD — convert consistently and note the rate used.
91 +- Provincial health-card and driver's-licence renewals are residential ties — snowbirds chasing 182 days can still be Canadian-resident (that's usually the goal) but may trip US state rules separately.
92 +- Streamlined/VDP eligibility can be lost once the authority contacts you first — timing of disclosure matters.
added finance-skills/optimizing-business-taxes/SKILL.md +68 −0
@@ -0,0 +1,68 @@
1 +---
2 +name: optimizing-business-taxes
3 +description: Plans legal small-business tax reduction in the US and Canada - US entity and S-corp election analysis, retirement plans, depreciation timing; Canadian CCPC small business deduction protection, salary vs dividends modeling, CCA and GST/HST input credits; year-end planning for both. Use when the user asks how to reduce business taxes, whether to elect S-corp status, salary vs dividends from a corporation, about the small business deduction, Section 179 or CCA timing, or year-end business tax moves in the US or Canada. Do not use for filing the returns themselves (filing-us-business-taxes, filing-canadian-business-taxes) or personal non-business planning (optimizing-us-personal-taxes, optimizing-canadian-personal-taxes).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Optimizing Business Taxes (US & Canada)
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** legal tax planning for a small business or its owner-manager — entity/status choices, owner compensation mix, expense and depreciation timing, registered plans.
15 +- **Do NOT use for:** preparing/filing returns, payroll processing mechanics, or personal planning unrelated to the business.
16 +
17 +## Important limits
18 +- Educational planning help, not professional advice — the user must verify strategies with a CPA or tax professional before acting.
19 +- All figures are tax-year-stamped and MUST be verified against [irs.gov](https://www.irs.gov) / [canada.ca](https://www.canada.ca/en/revenue-agency.html) before use.
20 +- Legal avoidance only — never assist with unreported income, fabricated expenses, or sham transactions. Refuse and say why.
21 +
22 +## Core strategies — US
23 +
24 +1. **Review S-corp election once net profit clears roughly $50–80k.** Pay yourself a reasonable salary; remaining profit flows as distributions free of self-employment tax. Weigh payroll/admin costs (~$1–3k/yr) and reduced retirement-plan base.
25 + ✅ $120k profit: $70k reasonable salary + $50k distributions ≈ $7,000+/yr SE-tax savings.
26 + ❌ $10k salary on $150k profit — "unreasonably low" salary is the classic audit trigger.
27 +2. **Open a Solo 401(k) (or SEP-IRA) before year-end** — employee deferral + ~25% employer contribution shelters far more than a personal IRA.
28 +3. **Time equipment purchases with Section 179 / bonus depreciation** — deduct in the high-income year, not by habit; verify current-year percentages on irs.gov.
29 +4. **Check QBI-type pass-through deductions** against current law and income thresholds before assuming eligibility.
30 +5. **Keep home-office and vehicle logs contemporaneously** — the deduction is legal; the reconstruction-in-audit version is not defensible.
31 +
32 +## Core strategies — Canada
33 +
34 +6. **Protect the small business deduction (SBD):** ~9% federal rate on the first $500,000 of CCPC active income; corporate passive investment income above $50,000/yr grinds the limit ($5 of limit lost per $1 of passive income).
35 + ✅ Move surplus corporate investments toward the owner's RRSP/TFSA (via salary) or corporate-owned exempt insurance before crossing $50k passive.
36 + ❌ Accumulate a large passive portfolio in the operating company and lose the SBD.
37 +7. **Model salary vs dividends every December — never default.** Salary: corporate deduction, RRSP room, CPP; dividends: no payroll, no RRSP room; integration makes totals roughly neutral, so the decision rides on RRSP room, CPP value, cash needs, and provincial rates.
38 + ✅ Enough salary to max RRSP room ($33,810 needs ≈ $187,800 salary) and CPP, dividends above that.
39 + ❌ 100% dividends for years, then discovering zero RRSP room and no CPP.
40 +8. **Time income and CCA:** defer invoices/bonus accruals across year-end when next year's rate is lower; claim CCA strategically (it's optional each year — skip it in loss years to preserve it); capture every GST/HST input tax credit (registration mandatory over $30,000 revenue in four consecutive quarters).
41 +
42 +## Workflow
43 +
44 +1. Identify: country, entity type (sole prop / LLC / S-corp / C-corp / CCPC), fiscal year-end, expected profit, owner cash needs, existing salary/dividend mix.
45 +2. Apply the relevant country's strategies in order; skip inapplicable ones and say why.
46 +3. For each move, state the hard deadline (most: fiscal year-end, often Dec 31; US S-corp election: March 15 for current-year effect; RRSP-driving salary: paid by Dec 31).
47 +4. **Validate:** model at least two scenarios with real numbers (e.g., sole-prop vs S-corp total tax; salary vs dividend mix showing corporate + personal tax and RRSP/CPP effects). Present the comparison table and the winner.
48 +5. Remind: document the business purpose of every planning move; never let the tax tail wag the business dog.
49 +
50 +## Current figures (tax year 2026 — verify before use)
51 +
52 +| Item | Amount | Source |
53 +|---|---|---|
54 +| US SE tax rate | 15.3% on ~92.35% of net SE income (SS portion capped) | irs.gov |
55 +| US Solo 401(k) deferral | $24,500 employee + employer % | irs.gov |
56 +| CA federal SBD rate / limit | ~9% on first $500,000 active income | canada.ca |
57 +| CA passive-income grind | starts $50,000; SBD gone by $150,000 | canada.ca |
58 +| CA salary for max RRSP room | ≈ $187,800 (18% → $33,810) | canada.ca |
59 +| GST/HST registration threshold | $30,000 over four consecutive quarters | canada.ca |
60 +
61 +## Edge cases
62 +- **Multi-state / multi-province operations** → nexus and allocation rules; flag for professional review.
63 +- **US LLC owned by a Canadian (or vice versa)** → hybrid-entity mismatches can double-tax; route to cross-border specialist immediately.
64 +- **Losses** → different playbook (carrybacks, skipping CCA, no S-corp benefit); say so rather than applying profit strategies.
65 +- **User asks to deduct personal expenses as business or skim cash** → refuse, state it is illegal, offer the legal alternatives above.
66 +
67 +## References
68 +Decision tables, worked math, and traps: see [references/reference.md](references/reference.md).
added finance-skills/optimizing-business-taxes/references/reference.md +105 −0
@@ -0,0 +1,105 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Reference — Business Tax Optimization (US & Canada)
7 +
8 +## Contents
9 +- US: S-corp vs sole proprietorship worked math
10 +- US: retirement plan comparison
11 +- US: home office and vehicle deductions done right
12 +- Canada: salary vs dividends worked math
13 +- Canada: passive-income SBD grind table
14 +- Canada: common CCA classes
15 +- Year-end checklist (both countries)
16 +- Gotchas
17 +
18 +## US: S-corp vs sole proprietorship worked math
19 +
20 +Assumptions: single owner, all profit is SE income, SS wage base not exceeded, ignoring state tax and the QBI interaction (model those before deciding). SE tax ≈ 15.3% on 92.35% of net profit; S-corp pays 15.3% FICA only on salary.
21 +
22 +| Net profit | Sole prop SE tax | S-corp (reasonable salary) | FICA on salary | Payroll/admin cost | Approx. annual saving |
23 +|---|---|---|---|---|---|
24 +| $60,000 | ≈ $8,478 | salary $45,000 | ≈ $6,885 | $1,500 | ≈ $93 — **not worth it** |
25 +| $120,000 | ≈ $16,955 | salary $70,000 | ≈ $10,710 | $1,500 | ≈ **$4,745** |
26 +| $200,000 | ≈ $23,000 (SS capped) | salary $90,000 | ≈ $13,770 | $1,500 | ≈ **$7,700** |
27 +
28 +Break-even sits around $50–80k profit. "Reasonable salary" = what you'd pay a stranger for the same role — document comparables (BLS data, job postings).
29 +
30 +Election deadline: Form 2553 by March 15 for the election to apply to the current calendar year (late-election relief exists but don't plan on it).
31 +
32 +## US: retirement plan comparison
33 +
34 +| Plan | 2026 shelter potential | Best for |
35 +|---|---|---|
36 +| Solo 401(k) | $24,500 deferral + ~25% of compensation employer piece | Owner-only, wants max shelter at moderate income |
37 +| SEP-IRA | ~25% of compensation only | Simplicity; no employee deferral piece |
38 +| Traditional IRA | $7,500 | Fallback only |
39 +
40 +At $70,000 S-corp salary: Solo 401(k) ≈ $24,500 + $17,500 = **$42,000 sheltered** vs SEP ≈ $17,500. The S-corp salary choice directly caps the employer piece — factor it into the salary decision.
41 +
42 +## US: home office and vehicle deductions done right
43 +
44 +**Home office** (self-employed; exclusive + regular use required):
45 +- Simplified method: $5/sq ft up to 300 sq ft = max $1,500 — zero recordkeeping beyond square footage.
46 +- Actual method: business-use % × (rent or depreciation, utilities, insurance, repairs). A 150 sq ft office in a 1,500 sq ft home = 10% of eligible costs — usually beats simplified once annual home costs exceed ~$15,000, but adds depreciation-recapture complexity for owners.
47 +
48 +**Vehicle:**
49 +- Standard mileage rate (verify the current rate on irs.gov) vs actual expenses × business-use % — pick per vehicle, but standard-mileage must be chosen in year 1 to keep the option.
50 +- The log is the deduction: date, destination, purpose, miles, kept contemporaneously. Commuting from home to a regular workplace is never business mileage.
51 +
52 +## Canada: salary vs dividends worked math
53 +
54 +CCPC in a ~9%-federal SBD province, owner needs $80,000 pre-personal-tax cash, corporate pre-tax profit $150,000. Illustrative combined rates — model the actual province.
55 +
56 +**All salary ($80,000):** corporation deducts it (saves ~12% combined corporate ≈ $9,600 on that slice); owner pays personal tax + CPP (~$4,000 employee+employer, half deductible); owner earns $14,400 RRSP room.
57 +
58 +**All dividends ($80,000 non-eligible):** corporation first pays ~12% corporate tax, dividends carry a gross-up + credit designed so the combined bill lands within ~1–2% of the salary route (integration). No CPP cost — and no CPP benefit, **zero RRSP room**.
59 +
60 +**Decision drivers, not totals:** RRSP room (salary only), CPP disability/retirement value vs its cost, provincial integration gaps, income smoothing (dividends flexible), mortgage-qualification preferences. Default blended pattern: salary to the RRSP-max level (≈ $187,800 for full room — or lower per cash reality), dividends for the remainder.
61 +
62 +## Canada: passive-income SBD grind table
63 +
64 +SBD limit reduction = 5 × (passive investment income − $50,000).
65 +
66 +| Corporate passive income | SBD limit remaining |
67 +|---|---|
68 +| ≤ $50,000 | $500,000 |
69 +| $75,000 | $375,000 |
70 +| $100,000 | $250,000 |
71 +| $150,000+ | $0 — all active income at the general rate (~15% federal) |
72 +
73 +Each $1 of passive income above $50k costs $5 of limit ≈ up to ~$0.30 extra corporate tax. Mitigations: pay salary/dividends out and invest personally (RRSP/TFSA), buy back active capacity, corporate-class funds deferring income realization.
74 +
75 +## Canada: common CCA classes
76 +
77 +Verify class assignments and rates on canada.ca — these are the frequent ones:
78 +
79 +| Class | Rate | Typical assets |
80 +|---|---|---|
81 +| 8 | 20% | Furniture, equipment, tools ≥ $500 |
82 +| 10 / 10.1 | 30% | Vehicles (10.1 caps luxury-car cost — no terminal loss) |
83 +| 12 | 100% | Small tools < $500, some software |
84 +| 50 | 55% | Computer hardware |
85 +| 14.1 | 5% | Goodwill and other intangibles |
86 +
87 +Mechanics: declining balance on the class pool; the half-year rule limits the first-year claim to half the addition (accelerated first-year rules have varied — verify current status); CCA is optional each year — skipping it in loss years preserves the pool for profitable ones.
88 +
89 +## Year-end checklist (both countries)
90 +
91 +1. Project profit to Dec 31 while there is still time to act (start in November).
92 +2. US: confirm reasonable salary run through payroll; fund Solo 401(k) deferral by Dec 31; place equipment in service before year-end if deducting this year; Q4 estimated payment Jan 15.
93 +3. Canada: set salary/bonus by Dec 31 (bonus accrued now, payable within 180 days); check passive income vs $50k; decide this year's CCA claim; confirm GST/HST ITCs all captured.
94 +4. Both: document the business purpose of each move in writing, dated now — not at audit time.
95 +
96 +## Gotchas
97 +
98 +- **Unreasonably low S-corp salary** is the IRS's top S-corp audit issue; distributions reclassified as wages arrive with penalties and interest.
99 +- **S-corp reduces the retirement base:** employer 401(k)/SEP contributions key off salary, not distributions — aggressive salary minimization can cost more shelter than it saves in FICA.
100 +- **Integration is provincial:** salary-vs-dividend "neutrality" varies ±2-3% by province and income type — always compute, never assume.
101 +- **Bonus accrual trap (Canada):** an accrued bonus unpaid within 180 days of year-end is denied as a deduction until paid.
102 +- **CCA is optional per year** — claiming it in a loss year wastes it; carry the pool forward instead.
103 +- **GST/HST on the $30,000 threshold:** registration is mandatory from the quarter you cross it — late registration means remitting tax you never collected.
104 +- **Personal expenses through the corporation** (Canada) trigger shareholder-benefit inclusion at full rates with no corporate deduction — the worst of both worlds, plus penalties.
105 +- **Paper trails beat intentions:** logs, minutes, and comparables written contemporaneously are what survive an audit.
added finance-skills/optimizing-canadian-personal-taxes/SKILL.md +67 −0
@@ -0,0 +1,67 @@
1 +---
2 +name: optimizing-canadian-personal-taxes
3 +description: Plans legal Canadian personal income-tax reduction - RRSP vs TFSA vs FHSA priority, legal income splitting, capital-gains timing with the superficial-loss rule, asset location, donation and RESP strategies. Use when the user asks how to lower or optimize their Canadian personal taxes, whether to contribute to RRSP or TFSA or FHSA, about spousal RRSPs, pension splitting, the superficial-loss rule, RESP grants, or year-end Canadian tax moves. Do not use for preparing the T1 return itself (preparing-canadian-personal-tax-returns), corporate or owner-manager planning (optimizing-business-taxes), or US planning (optimizing-us-personal-taxes).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Optimizing Canadian Personal Taxes
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** planning moves that legally reduce a Canadian individual's income tax — registered-account priority, splitting, harvesting, asset location, donations, RESP.
15 +- **Do NOT use for:** completing the T1 and schedules, CCPC owner compensation, US taxes, or Québec-specific mechanics beyond noting QC differs.
16 +
17 +## Important limits
18 +- Educational planning help, not professional advice — the user must verify strategies with a CPA or tax professional before acting.
19 +- All figures are tax-year-stamped and MUST be verified against [canada.ca](https://www.canada.ca/en/revenue-agency.html) before use; limits and brackets change annually.
20 +- Legal avoidance only — never assist with unreported income, fabricated expenses, or sham transactions. Refuse and say why.
21 +
22 +## Core strategies
23 +
24 +1. **Capture any employer RPP/group-RRSP match first** — guaranteed return before any other move.
25 +2. **RRSP vs TFSA by marginal-rate comparison.** RRSP wins when today's marginal rate exceeds the expected retirement rate (deduct high, withdraw low); TFSA wins otherwise, or when flexibility matters (withdrawals restore room the following year).
26 + ✅ A $130k earner (26%+ federal) prioritizes RRSP; a $45k earner prioritizes TFSA.
27 + ❌ Default to RRSP at low income, converting a 14%-rate deduction into 20%+-rate withdrawals later.
28 +3. **First-time homebuyer → max the FHSA before non-matched RRSP.** It is deductible like an RRSP AND tax-free on qualifying withdrawal like a TFSA — strictly better for a house down payment.
29 + ✅ $8,000/yr to FHSA (lifetime $40,000), then RRSP/TFSA.
30 + ❌ Use the RRSP Home Buyers' Plan first while FHSA room sits unused.
31 +4. **Split income only through legal channels:** spousal RRSP (higher earner deducts, lower earner withdraws after the 3-year attribution window), pension income splitting (up to 50% at 65+), giving the spouse money to fund their own TFSA (no attribution in a TFSA).
32 + ✅ Spousal RRSP to equalize retirement incomes.
33 + ❌ Sprinkle private-corporation dividends to family — TOSI taxes it at the top rate.
34 +5. **Harvest losses respecting the superficial-loss rule:** no repurchase of the identical property 30 days before/after by you, your spouse, or accounts you control (RRSP/TFSA included) — the loss is denied and added to the repurchaser's cost base.
35 + ✅ Sell the losing Canadian equity ETF, buy a different-index ETF the same day.
36 + ❌ Sell for the loss while the spouse's TFSA buys the same fund that week.
37 +6. **Asset location:** shelter interest-bearing assets (fully taxed) inside RRSP/TFSA first; keep Canadian eligible-dividend and capital-gains assets in taxable accounts (dividend tax credit, 50% gain inclusion).
38 +7. **Donations:** pool spouses' donations on one return and consider carrying forward (up to 5 years) to clear the ~$200 threshold where the credit rate jumps.
39 +8. **RESP: contribute $2,500/child/year to capture the 20% CESG ($500/yr, lifetime $7,200)** before any additional TFSA/RRSP beyond the match.
40 +
41 +## Workflow
42 +
43 +1. Collect: province, income by type, marginal rate, RRSP/TFSA/FHSA room (from CRA My Account), family situation (spouse income, kids, first-home status).
44 +2. Place the user on the 2026 federal bracket table; note the provincial layer exists.
45 +3. Apply strategies 1–8 in order; skip inapplicable ones and say why.
46 +4. State each deadline: TFSA anytime (room restores Jan 1); RRSP deduction deadline = 60 days into the next year; harvesting = settlement by Dec 31; RESP = Dec 31 for that year's grant.
47 +5. **Validate:** model the tax outcome with and without the moves using actual numbers and the user's real contribution room; confirm no limit is exceeded and no superficial-loss window is violated. Present both scenarios.
48 +
49 +## Current figures (tax year 2026 — verify before use)
50 +
51 +| Item | Amount | Source |
52 +|---|---|---|
53 +| Federal brackets | 14% to $58,523; 20.5% to $117,045; 26% to $181,440; 29% to $258,482; 33% above | [canada.ca current rates](https://www.canada.ca/en/revenue-agency/services/tax/individuals/tax-rates-brackets/current-year.html) |
54 +| RRSP limit | 18% of 2025 earned income, max $33,810, + unused room | canada.ca |
55 +| TFSA annual | $7,000 (cumulative $109,000 if eligible since 2009) | canada.ca |
56 +| FHSA | $8,000/yr, $40,000 lifetime | canada.ca |
57 +| RRSP deadline for 2025 deduction | March 2, 2026 | canada.ca |
58 +| CESG | 20% of RESP contributions, $500/yr, $7,200 lifetime | canada.ca |
59 +
60 +## Edge cases
61 +- **Québec residents** → provincial return and rates differ substantially; flag it.
62 +- **US citizens in Canada** → TFSA/FHSA/RESP have US tax complications; route to cross-border professional advice.
63 +- **Attribution rules** → money gifted to a spouse for taxable investing attributes income back; only the TFSA/spousal-RRSP channels above are clean.
64 +- **User asks to hide income or fabricate expenses** → refuse, state it is illegal, offer the legal alternatives above.
65 +
66 +## References
67 +Decision tables, worked math, and traps: see [references/reference.md](references/reference.md).
added finance-skills/optimizing-canadian-personal-taxes/references/reference.md +103 −0
@@ -0,0 +1,103 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Reference — Canadian Personal Tax Optimization
7 +
8 +## Contents
9 +- RRSP vs TFSA decision table
10 +- Worked example: RRSP vs TFSA at two rate profiles
11 +- Worked example: FHSA for a first home
12 +- Worked example: superficial-loss harvesting
13 +- Worked example: donation pooling and carry-forward
14 +- Asset-location placement table
15 +- Retirement decumulation notes (RRIF, OAS)
16 +- Contribution-room mechanics and deadlines
17 +- Gotchas
18 +
19 +## RRSP vs TFSA decision table
20 +
21 +| Situation (tax year 2026) | Pick | Why |
22 +|---|---|---|
23 +| Marginal rate now > expected retirement rate | RRSP | Deduct at the high rate, withdraw at the low rate |
24 +| Marginal rate now < expected retirement rate | TFSA | No deduction needed now; withdrawals never taxed |
25 +| Income < ~$58k (14% federal bracket) | TFSA | Save RRSP room for higher-earning years — room carries forward |
26 +| Expects OAS/GIS in retirement | TFSA | TFSA withdrawals don't count as income, so no clawback |
27 +| First home purchase planned | FHSA first | RRSP-style deduction + TFSA-style withdrawal |
28 +| Needs the money before retirement | TFSA | Withdrawals restore room next Jan 1; RRSP withdrawals lose room forever |
29 +
30 +## Worked example: RRSP vs TFSA at two rate profiles
31 +
32 +$10,000 of gross salary invested 25 years at 7%/yr (≈5.43× growth).
33 +
34 +**Case A — 40% marginal now, 25% in retirement:**
35 +- RRSP: full $10,000 in (deduction refunds the tax) → $54,300; taxed 25% out → **$40,725 net**
36 +- TFSA: $6,000 after tax in → $32,580 tax-free → **$32,580 net**
37 +- RRSP wins by ~$8,100 per $10k — the 15-point rate drop is the entire win.
38 +
39 +**Case B — 25% now, 35% effective in retirement (OAS clawback zone):**
40 +- RRSP: $54,300 × 0.65 → **$35,295 net**
41 +- TFSA: $7,500 in → $40,725 tax-free → **$40,725 net**
42 +- TFSA wins by ~$5,400. Withdrawal-side clawbacks can push the effective retirement rate above the statutory bracket — always model them.
43 +
44 +## Worked example: FHSA for a first home
45 +
46 +Buyer at 30% marginal rate contributes $8,000/yr for 5 years ($40,000 lifetime max):
47 +- Deductions refund 30% × $40,000 = **$12,000** along the way.
48 +- Suppose the account grows to $48,000 → withdrawn **tax-free** for a qualifying first home.
49 +- The same money in a TFSA: no $12,000 refund. Via RRSP + Home Buyers' Plan: withdrawal must be repaid over 15 years or it becomes taxable income. FHSA dominates for this goal.
50 +
51 +## Worked example: superficial-loss harvesting
52 +
53 +$12,000 loss on a Canadian index ETF in November:
54 +1. Sell with settlement before Dec 31; buy a different-index ETF the same day to stay invested.
55 +2. The capital loss offsets capital gains this year; unused losses carry back 3 years (T1A request) or forward indefinitely.
56 +3. At a 50% inclusion rate and 40% marginal rate, offsetting $12,000 of gains saves ≈ **$2,400**.
57 +4. Check ±30 days for purchases of the identical fund by you, your spouse, your RRSP/TFSA, or a corporation you control — a match denies the loss (and in registered accounts the denied loss is gone permanently, no basis bump).
58 +
59 +## Worked example: donation pooling and carry-forward
60 +
61 +Couple each gives $150/yr to charity, claimed separately every year.
62 +- Federal credit: 15% on the first $200, 29% above — separate $150 claims never reach the higher tier.
63 +- **Pooled and carried:** accumulate 5 years of both spouses' donations ($1,500) and claim once on the higher earner's return: 15% × $200 + 29% × $1,300 = **$407 federal** (plus provincial), vs $225 claimed annually-and-separately.
64 +- Rule of thumb: pool spouses always; carry forward (max 5 years) whenever annual totals are small.
65 +
66 +## Asset-location placement table
67 +
68 +Fill registered room with the worst-taxed assets first.
69 +
70 +| Asset type | Taxable-account treatment | Priority for RRSP/TFSA shelter |
71 +|---|---|---|
72 +| Interest (bonds, GICs, HISA) | 100% at full marginal rate | **Highest** |
73 +| Foreign dividends | Full rate + possible withholding | High (US withholding exempt in RRSP under treaty, NOT in TFSA) |
74 +| Canadian eligible dividends | Dividend tax credit — low effective rate | Low — fine in taxable |
75 +| Capital gains | 50% inclusion, deferrable until sale | Lowest — fine in taxable |
76 +
77 +## Retirement decumulation notes (RRIF, OAS)
78 +
79 +- RRSP converts to a RRIF by end of the year the holder turns 71; mandatory minimum withdrawals begin the next year and rise with age.
80 +- Withdrawals are ordinary income: large RRSP balances can push retirees into OAS clawback (15% on income above the threshold — verify the current threshold on canada.ca).
81 +- Planning levers: draw RRSP down early in low-income retirement years (before OAS/CPP start), base RRIF minimums on the younger spouse's age, and shift surplus withdrawals into the TFSA.
82 +
83 +## Contribution-room mechanics and deadlines
84 +
85 +| Account | 2026 figure | Deadline | Mechanics |
86 +|---|---|---|---|
87 +| RRSP | 18% of prior-year earned income, max $33,810 | Mar 2, 2026 for 2025 deduction | Room carries forward; deduction can also be deferred to a higher-income year |
88 +| TFSA | $7,000/yr; $109,000 cumulative since 2009 | none | Withdrawals restore room the following Jan 1 — recontributing the same year over-contributes |
89 +| FHSA | $8,000/yr; $40,000 lifetime | Dec 31 | Only $8,000 of unused room carries forward; must open the account to start accruing |
90 +| RESP | $2,500/child/yr for full CESG | Dec 31 | CESG 20%, $500/yr, $7,200 lifetime; catch-up limited to one extra year at a time |
91 +
92 +Verify personal room in CRA My Account — never estimate it from salary alone (pension adjustments reduce RRSP room).
93 +
94 +## Gotchas
95 +
96 +- **RRSP over-contribution:** 1%/month penalty tax on amounts more than $2,000 over your limit — file T1-OVP if it happens; withdraw the excess promptly.
97 +- **TFSA same-year recontribution:** withdrawing $10k in March and redepositing in June over-contributes unless room remained; the room comes back Jan 1.
98 +- **Spousal RRSP attribution:** withdrawals within 3 calendar years of any spousal contribution attribute back to the contributor.
99 +- **TOSI:** dividends/gains from a related private corporation to family members are taxed at the top rate unless an exclusion applies (e.g., 20+ hrs/week active work) — this killed income sprinkling.
100 +- **Superficial loss includes registered accounts:** repurchasing inside an RRSP/TFSA denies the loss with no cost-base adjustment — the worst outcome.
101 +- **OAS clawback:** retirement income above the annual threshold claws back OAS at 15% — RRSP/RRIF withdrawals count, TFSA withdrawals don't; model it before large RRSP balances build.
102 +- **Deferring the RRSP deduction:** contributing now but deducting in a future higher-rate year is legal and often forgotten.
103 +- **Québec:** separate return, different rates and credits — federal-only math understates everything.
added finance-skills/optimizing-us-personal-taxes/SKILL.md +75 −0
@@ -0,0 +1,75 @@
1 +---
2 +name: optimizing-us-personal-taxes
3 +description: Plans legal US personal income-tax reduction - retirement and HSA contribution priority, traditional vs Roth choice, bracket management, tax-loss harvesting, deduction bunching, and capital-gains timing. Use when the user asks how to lower or optimize their US personal taxes, whether to prioritize 401(k), Roth, IRA, or HSA contributions, about tax-loss harvesting, the wash-sale rule, itemizing vs the standard deduction, or year-end US tax moves. Do not use for preparing the return itself (preparing-us-personal-tax-returns), business or self-employment planning (optimizing-business-taxes), or Canadian planning (optimizing-canadian-personal-taxes).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Optimizing US Personal Taxes
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** planning moves that legally reduce a US individual's federal income tax — account priority, Roth vs traditional, harvesting, bunching, gain timing.
15 +- **Do NOT use for:** filling out Form 1040 and schedules, business/SE tax strategy, Canadian taxes, or state-specific planning beyond noting that states differ.
16 +
17 +## Important limits
18 +- Educational planning help, not professional advice — the user must verify strategies with a CPA or tax professional before acting.
19 +- All figures are tax-year-stamped and MUST be verified against [irs.gov](https://www.irs.gov) before use; limits change annually.
20 +- Legal avoidance only — never assist with unreported income, fabricated expenses, or sham transactions. Refuse and say why.
21 +
22 +## Core strategies
23 +
24 +1. **Contribute in priority order: employer match → HSA → 401(k)/IRA → taxable.**
25 + ✅ Capture the full employer 401(k) match first — it is an instant 50–100% return.
26 + ❌ Max a taxable brokerage account while leaving match dollars unclaimed.
27 +2. **Treat the HSA as a retirement account (triple advantage: deductible in, growing tax-free, tax-free out for medical).**
28 + ✅ Max the HSA and pay small medical bills out of pocket, keeping receipts.
29 + ❌ Treat the HSA as a spending float that stays near $0.
30 +3. **Traditional vs Roth by marginal-rate comparison.** Traditional wins when today's marginal rate is higher than the expected retirement rate; Roth wins when it is lower (early career, gap years).
31 + ✅ A resident earning $60k picks Roth; a peak earner at 35% picks traditional.
32 + ❌ Pick Roth "because tax-free sounds better" without comparing rates.
33 +4. **Manage the bracket edge.** Defer income (bonus timing, retirement contributions) when just above a bracket threshold; realize income (Roth conversions, gain harvesting) in low-income years.
34 + ✅ Convert traditional→Roth during a sabbatical year at 12%.
35 + ❌ Realize a large gain in the same year as a signing bonus.
36 +5. **Tax-loss harvest, respecting the wash-sale rule (no repurchase of a substantially identical security 30 days before or after — across all accounts including IRAs and a spouse's).**
37 + ✅ Sell the losing S&P 500 ETF and buy a total-market ETF the same day.
38 + ❌ Rebuy the identical ETF within 30 days, or have the IRA auto-reinvest into it.
39 +6. **Hold for long-term rates.** Gains on assets held >1 year get preferential rates.
40 + ✅ Wait two more weeks to cross the 1-year mark before selling a winner.
41 + ❌ Sell at 11 months and pay ordinary rates without checking the calendar.
42 +7. **Bunch itemized deductions in alternating years when near the standard deduction.**
43 + ✅ Stack two years of charitable gifts into one year (donor-advised fund), itemize that year, take the standard deduction the next.
44 + ❌ Donate the same amount every year while never clearing the standard deduction.
45 +8. **Prefer credits over deductions when eligible** (child tax credit, education credits, saver's credit, energy credits) — a credit reduces tax dollar-for-dollar.
46 + ✅ Check credit eligibility before hunting for marginal deductions.
47 + ❌ Ignore a $2,000 credit while optimizing a $500 deduction.
48 +
49 +## Workflow
50 +
51 +1. Collect: filing status, expected income by type (wages, SE, interest, dividends, gains), current contributions, itemizable expenses, state.
52 +2. Place the user on the 2026 bracket table; note distance to the nearest bracket edge.
53 +3. Apply strategies 1–8 in order; skip any that do not apply and say why.
54 +4. For each recommended move, state the deadline (most contributions and harvesting: Dec 31; IRA and HSA: the April filing deadline).
55 +5. **Validate:** model the tax outcome with and without the moves using actual numbers; confirm no contribution limit is exceeded and no wash-sale window is violated. Present both scenarios.
56 +
57 +## Current figures (tax year 2026 — verify before use)
58 +
59 +| Item | Amount | Source |
60 +|---|---|---|
61 +| Bracket rates | 10, 12, 22, 24, 32, 35, 37% | [irs.gov newsroom](https://www.irs.gov/newsroom) |
62 +| Standard deduction | $16,100 single / $32,200 MFJ | irs.gov |
63 +| 401(k) elective deferral | $24,500 (+$8,000 catch-up 50+; $11,250 ages 60–63) | [irs.gov 401(k) limits](https://www.irs.gov/newsroom/401k-limit-increases-to-24500-for-2026-ira-limit-increases-to-7500) |
64 +| IRA | $7,500 (+$1,100 catch-up 50+) | irs.gov |
65 +| HSA | $4,400 self / $8,750 family | irs.gov |
66 +| Long-term gain holding period | >1 year | irs.gov |
67 +
68 +## Edge cases
69 +- **Income too high for direct Roth IRA** → note the backdoor Roth exists but has pro-rata traps; flag for CPA review, don't improvise.
70 +- **Equity compensation (RSU/ISO)** → AMT and withholding traps; flag for professional review.
71 +- **State taxes** → strategies above are federal; state treatment differs (e.g., HSA in CA/NJ). Say so explicitly.
72 +- **User asks to hide income or invent deductions** → refuse, state it is illegal, offer the legal alternatives above.
73 +
74 +## References
75 +Decision tables, worked math, and traps: see [references/reference.md](references/reference.md).
added finance-skills/optimizing-us-personal-taxes/references/reference.md +109 −0
@@ -0,0 +1,109 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Reference — US Personal Tax Optimization
7 +
8 +## Contents
9 +- Traditional vs Roth decision table
10 +- Worked example: traditional vs Roth at two rates
11 +- Worked example: Roth conversion in a gap year
12 +- Worked example: bunching deductions
13 +- Worked example: tax-loss harvesting
14 +- Common credits worth checking (2026)
15 +- Estimated-tax safe harbor
16 +- Contribution mechanics and deadlines
17 +- Gotchas
18 +
19 +## Traditional vs Roth decision table
20 +
21 +| Situation (tax year 2026) | Pick | Why |
22 +|---|---|---|
23 +| Marginal rate now > expected retirement rate | Traditional | Deduct at the high rate, withdraw at the low rate |
24 +| Marginal rate now < expected retirement rate | Roth | Pay the low rate now, withdraw tax-free |
25 +| Rates roughly equal | Roth (slight edge) | Tax-free growth hedges future rate increases; no RMDs on Roth IRA |
26 +| Low-income year (sabbatical, grad school, early retirement) | Roth conversions | Fill the 10–12% brackets with converted dollars |
27 +| Needs the deduction to qualify for credits/subsidies | Traditional | AGI reduction can unlock saver's credit, ACA subsidies |
28 +
29 +## Worked example: traditional vs Roth at two rates
30 +
31 +$10,000 of salary directed to a 401(k), invested 25 years at 7%/yr (≈5.43× growth).
32 +
33 +**Case A — 32% now, 22% in retirement:**
34 +- Traditional: $10,000 grows to $54,300; taxed 22% at withdrawal → **$42,354 net**
35 +- Roth: $6,800 after tax grows to $36,924 tax-free → **$36,924 net**
36 +- Traditional wins by ~$5,400 per $10k contributed.
37 +
38 +**Case B — 12% now, 22% in retirement:**
39 +- Traditional: $54,300 × 0.78 → **$42,354 net**
40 +- Roth: $8,800 after tax grows to $47,784 tax-free → **$47,784 net**
41 +- Roth wins by ~$5,400. The comparison is symmetric: only the rate differential matters (plus RMD/flexibility considerations).
42 +
43 +## Worked example: Roth conversion in a gap year
44 +
45 +Single filer takes an unpaid sabbatical in 2026; only income is $15,000 of freelance work.
46 +
47 +1. Standard deduction $16,100 wipes out ordinary income — taxable income ≈ $0.
48 +2. Convert traditional IRA dollars to Roth up to the top of the 12% bracket. Roughly: $16,100 (deduction) + 12%-bracket ceiling − $15,000 existing income of conversion headroom taxed at only 10–12%.
49 +3. Those dollars would have been taxed at 24%+ in a working year — each $10,000 converted saves ≈ $1,200+ in lifetime tax.
50 +4. Caveats: conversion income can affect ACA premium subsidies (a cliff-like phase-out — model it first), and conversions cannot be undone (recharacterization of conversions was eliminated).
51 +
52 +## Worked example: bunching deductions
53 +
54 +Married couple, tax year 2026, standard deduction $32,200. Annual itemizables: $10,000 state/local tax (capped), $8,000 mortgage interest, $12,000 charity = $30,000 — below the standard deduction every year, so charity yields zero tax benefit.
55 +
56 +**Bunched:** give $24,000 to a donor-advised fund in year 1, $0 in year 2.
57 +- Year 1 itemized: $10,000 + $8,000 + $24,000 = $42,000 → $9,800 above standard
58 +- Year 2: standard $32,200
59 +- Two-year deductions: $74,200 vs $64,400 unbunched → **$9,800 more deducted**; at 24% ≈ **$2,352 saved** for the same giving.
60 +
61 +## Worked example: tax-loss harvesting
62 +
63 +Holding shows a $15,000 loss in November.
64 +1. Sell; buy a similar-but-not-substantially-identical fund the same day (S&P 500 → total market) to stay invested.
65 +2. Offset $15,000 of realized gains; if gains < losses, deduct up to $3,000 against ordinary income and carry the rest forward indefinitely.
66 +3. At 24% + state, the $3,000 ordinary offset alone ≈ $720+ this year.
67 +4. Check every account (both spouses, IRAs, DRIPs) for purchases of the sold security within ±30 days — any match wash-sales that portion of the loss.
68 +
69 +## Common credits worth checking (2026)
70 +
71 +Credits reduce tax dollar-for-dollar; check these before optimizing deductions. Verify amounts and phase-outs on irs.gov — they move yearly.
72 +
73 +| Credit | Who typically qualifies | Notes |
74 +|---|---|---|
75 +| Child Tax Credit | Parents of children under 17 | Partially refundable; income phase-out |
76 +| Child & Dependent Care | Working parents paying for care | Percentage of qualifying expenses |
77 +| Saver's Credit | Low/moderate income + retirement contributions | Stacks ON TOP of the deduction — tiered by AGI, cliff edges |
78 +| American Opportunity / Lifetime Learning | Tuition payers | AOTC partially refundable, 4-year limit; LLC unlimited years |
79 +| Energy credits (home efficiency, clean vehicle) | Qualifying purchases | Verify current-law status before promising anything |
80 +| Foreign Tax Credit | Foreign investment income / expats | Form 1116; avoids double taxation |
81 +
82 +## Estimated-tax safe harbor
83 +
84 +Freelancers and investors avoid underpayment penalties by paying, through withholding + quarterly estimates, the smaller of:
85 +- 90% of the current year's tax, or
86 +- 100% of last year's tax (110% if prior-year AGI > $150,000).
87 +
88 +Quarterly due dates: Apr 15, Jun 15, Sep 15, Jan 15. A December W-2 withholding bump counts as paid evenly through the year — the cleanest late-year fix for an estimate shortfall.
89 +
90 +## Contribution mechanics and deadlines
91 +
92 +| Account | 2026 limit | Deadline for tax year 2026 | Notes |
93 +|---|---|---|---|
94 +| 401(k) employee deferral | $24,500 | Dec 31, 2026 (payroll) | Catch-up 50+: +$8,000; ages 60–63: +$11,250 |
95 +| IRA (traditional/Roth) | $7,500 | April filing deadline 2027 | Catch-up 50+: +$1,100; Roth has income phase-outs — check irs.gov |
96 +| HSA | $4,400 / $8,750 | April filing deadline 2027 | Requires HDHP coverage; 55+ catch-up +$1,000 |
97 +| Harvesting / gain realization | — | Dec 31, 2026 (trade date) | Settlement date does not matter for US equities |
98 +
99 +Traditional-IRA deductibility phases out when covered by a workplace plan — verify the current phase-out bands on irs.gov before recommending.
100 +
101 +## Gotchas
102 +
103 +- **Wash sale across accounts:** an IRA repurchase of the harvested security permanently destroys the loss (no basis adjustment). Automatic dividend reinvestment is the classic accidental trigger.
104 +- **"Substantially identical":** same-index ETFs from different providers are risky; different-index funds (S&P 500 → total market) are the accepted swap.
105 +- **Roth income limits:** direct Roth IRA contributions phase out at high AGI; an ineligible contribution accrues a 6% excise per year until fixed.
106 +- **Backdoor Roth pro-rata trap:** existing pre-tax IRA balances make the conversion mostly taxable — the pro-rata rule looks at all IRAs on Dec 31.
107 +- **Bracket myths:** crossing a bracket only taxes the marginal dollars — never advise refusing income to "stay in a lower bracket"; but do watch cliff-based credits (ACA subsidies pre-2026 rules, saver's credit tiers) where $1 can cost hundreds.
108 +- **HSA state nonconformity:** CA and NJ tax HSA earnings — the triple advantage is federal.
109 +- **Short-term vs long-term lot selection:** specify lots (SpecID) when selling; default FIFO can realize short-term gains unnecessarily.
added finance-skills/preparing-canadian-personal-tax-returns/SKILL.md +51 −0
@@ -0,0 +1,51 @@
1 +---
2 +name: preparing-canadian-personal-tax-returns
3 +description: Helps organize and prepare a Canadian personal income tax return (T1) — slips checklist, deductions vs credits, federal brackets, deadlines, NETFILE, and instalments. Use when the user asks for help preparing, organizing, or checking their Canadian tax return, T1, T4/T5/T3 slips, RRSP deduction, CRA filing deadlines, refund estimation, or which credits apply. Do not use for US returns (preparing-us-personal-tax-returns), corporate T2 returns (filing-canadian-business-taxes), or tax planning strategy (optimizing-canadian-personal-taxes).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Preparing Canadian Personal Tax Returns
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** organizing slips, mapping income and deductions to the T1, distinguishing deductions from credits, deadline and payment questions, NETFILE readiness, instalment rules for an individual return.
15 +- **Do NOT use for:** US 1040 returns, corporate T2, GST/HST business filings, or planning strategy for future years. Quebec residents also file a provincial TP-1 — flag it, then stay on the federal T1.
16 +
17 +## Important limits
18 +- Educational preparation help, not professional tax advice — complex situations (departure tax, foreign property T1135, audits) go to a CPA.
19 +- All figures below are stated for **tax year 2026** and MUST be verified against [canada.ca](https://www.canada.ca/en/revenue-agency.html) before use.
20 +- Never assist with evasion — unreported income or fabricated deductions are out of scope; if asked, decline and explain the legal alternative.
21 +
22 +## Workflow
23 +1. Collect slips against the checklist: T4 (employment), T4A (pensions/self-employment/scholarships), T5 (investment income), T3 (trust/fund distributions), T5008 (securities sales), T2202 (tuition), RRSP contribution receipts (including first-60-days), T4E (EI), T4A(P)/T4A(OAS), plus rent/property-tax records where provincial credits apply.
24 +2. Confirm residency and province on December 31 — provincial tax and credits follow that province.
25 +3. Map income to T1 lines (see the slip map in [references/reference.md](references/reference.md)); capital gains from T5008 need cost-basis reconciliation.
26 +4. Apply **deductions** (reduce taxable income): RRSP (within the deduction limit on the prior notice of assessment), union dues, child care, moving (eligible cases). Keep the distinction from credits precise.
27 +5. Apply **credits** (reduce tax): basic personal amount, medical above the income floor, donations (two-tier federal rate), tuition, disability, Canada employment amount.
28 +6. Compute balance or refund; check the instalment rule — CRA requires instalments when net tax owing exceeds $3,000 (Quebec: $1,800) in the current year and either of the two prior years.
29 +7. **Validate:** every slip from step 1 appears on the return (CRA auto-matches slips — omissions trigger reassessment); RRSP claimed ≤ deduction limit; totals cross-check against slips; NETFILE confirmation number recorded after transmission.
30 +
31 +## Current figures (tax year 2026 — verify before use)
32 +| Item | Value | Source |
33 +|---|---|---|
34 +| Federal brackets | 14% ≤ $58,523 · 20.5% ≤ $117,045 · 26% ≤ $181,440 · 29% ≤ $258,482 · 33% above | canada.ca |
35 +| Provincial tax | Stacks on top — separate bracket table per province | provincial sites |
36 +| Filing deadline | April 30 (self-employed: June 15, but payment still April 30) | canada.ca |
37 +| RRSP deduction deadline | ~60 days into the following year (Mar 2, 2026 for TY2025) | canada.ca |
38 +| RRSP dollar limit | $33,810 (and ≤18% of prior-year earned income + carryforward) | canada.ca |
39 +| TFSA annual room | $7,000 (no deduction — contributions are after-tax) | canada.ca |
40 +| FHSA | $8,000/yr, $40,000 lifetime (deductible like RRSP) | canada.ca |
41 +| Instalment threshold | Net owing > $3,000 in current + one of two prior years | canada.ca |
42 +
43 +## Edge cases & failure modes
44 +- **Missing slip** → pull it from CRA My Account (Auto-fill my return); never estimate silently.
45 +- **Error found after filing** → ReFILE or T1 adjustment (T1-ADJ); do not file a second return.
46 +- **First-time filer** → NETFILE may require paper filing or CRA account setup first.
47 +- **Can't pay** → file by the deadline anyway: the late-filing penalty is 5% + 1%/month (up to 12), separate from interest on the balance.
48 +- **Foreign property > $100,000 cost** → T1135 required; steep penalties — flag and refer out if complex.
49 +
50 +## References
51 +Full slip map, credit tables, deadline calendar, penalty structure, and worked examples: see [references/reference.md](references/reference.md).
added finance-skills/preparing-canadian-personal-tax-returns/references/reference.md +90 −0
@@ -0,0 +1,90 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Reference — Canadian Personal Tax Return Preparation (T1)
7 +
8 +All dollar figures are **tax year 2026** unless noted — verify at [canada.ca](https://www.canada.ca/en/revenue-agency.html) before use.
9 +
10 +## Contents
11 +- Slip map
12 +- Deductions vs credits (the core distinction)
13 +- Credit table
14 +- Deadline calendar
15 +- Penalty and interest structure
16 +- Instalments
17 +- Worked mini-examples
18 +- Gotchas
19 +
20 +## Slip map
21 +
22 +| Slip | What it reports | T1 destination |
23 +|---|---|---|
24 +| T4 | Employment income, CPP/EI, tax withheld | Employment income lines |
25 +| T4A | Pension, self-employment box 048, scholarships | Varies by box — box 048 → T2125 |
26 +| T5 | Interest, dividends (eligible/other) | Investment income; dividends are grossed-up then credited |
27 +| T3 | Trust/mutual-fund distributions, capital gains | Investment income / Schedule 3 |
28 +| T5008 | Securities dispositions | Schedule 3 — needs adjusted cost base from your records |
29 +| T2202 | Tuition | Tuition credit (transferable in part) |
30 +| RRSP receipts | Contributions (incl. first-60-days) | RRSP deduction |
31 +| T4E | EI benefits | Other income; possible clawback |
32 +| T4A(P) / T4A(OAS) | CPP / OAS | Pension lines; OAS clawback above the recovery threshold |
33 +| T5013 | Partnership income | Per-box mapping |
34 +
35 +## Deductions vs credits (the core distinction)
36 +
37 +- **Deduction** reduces taxable income — worth your **marginal rate**. RRSP/FHSA contributions, union/professional dues, child-care expenses (lower-income spouse, with limits), moving expenses (40 km closer to work/school), carrying charges.
38 +- **Non-refundable credit** reduces tax at the **lowest federal rate** (14% in 2026) regardless of income — basic personal amount, age amount, medical, tuition, disability (DTC), Canada employment amount.
39 +- **Donations** are two-tier federally: low rate on the first $200, high rate above.
40 +- **Refundable credits/benefits** pay out even with zero tax: GST/HST credit, Canada Workers Benefit, provincial credits — filing is worthwhile even with no income.
41 +
42 +## Credit table
43 +
44 +| Credit | Notes |
45 +|---|---|
46 +| Basic personal amount | Indexed annually; reduced at very high income — verify current value |
47 +| Medical expenses | Only above min(3% of net income, indexed floor); pick any 12-month period ending in the year; pool family expenses on the lower-income return |
48 +| Tuition (T2202) | Student claims first; up to $5,000 transferable to parent/spouse; rest carries forward |
49 +| Disability (DTC) | Requires approved T2201; opens RDSP and other doors |
50 +| Donations | Two-tier; can pool spouses and carry forward 5 years |
51 +| Home buyers' amount | First-time buyers; fixed amount |
52 +| Canada caregiver | For dependants with impairment |
53 +
54 +## Deadline calendar (for tax year 2026)
55 +
56 +| Date | Event |
57 +|---|---|
58 +| Feb (late) 2027 | NETFILE opens; slips due to you by end of Feb |
59 +| ~Mar 1, 2027 | RRSP/FHSA deadline for deducting against 2026 |
60 +| Apr 30, 2027 | Filing + payment deadline (all balances, incl. self-employed) |
61 +| Jun 15, 2027 | Filing deadline if you or spouse are self-employed (payment was Apr 30) |
62 +| Mar 15 / Jun 15 / Sep 15 / Dec 15 | Instalment due dates when required |
63 +
64 +## Penalty and interest structure
65 +
66 +- **Late filing:** 5% of balance owing + 1% per full month late (max 12). Repeat offenders (late in prior 3 years with demand): 10% + 2%/month (max 20).
67 +- **Interest:** compounds daily on balances from May 1 at the CRA prescribed rate.
68 +- **Repeated failure to report income:** federal/provincial penalty when income is omitted twice within 4 years.
69 +- **T1135 failure:** $25/day (min $100, max $2,500) and up — flag foreign property early.
70 +
71 +## Instalments
72 +
73 +Required when net tax owing exceeds **$3,000** (Quebec residents: $1,800) in the current year **and** either of the two preceding years. Three CRA calculation options (no-calc reminders, prior-year, current-year); paying the amounts on CRA's instalment reminders is safe-harbor even if too low.
74 +
75 +## Worked mini-examples
76 +
77 +**Example 1 — deduction vs credit value.** Income $120,000 (26% federal bracket + province). A $5,000 RRSP deduction saves ≈ $5,000 × (26% + provincial rate). A $5,000 medical *credit base* saves only ≈ $5,000 × 14% federally — never present credits as if they were deductions.
78 +
79 +**Example 2 — dividends.** T5 shows $1,000 eligible dividends → gross up to taxable amount, then apply the dividend tax credit; effective rate depends on province and bracket — compute, don't guess.
80 +
81 +**Example 3 — instalments.** Owed $4,200 in 2025 and expects $4,500 for 2026 → both years above $3,000 → quarterly instalments required for 2026; missing them accrues instalment interest.
82 +
83 +## Gotchas
84 +- CRA slip-matching is automatic — a forgotten T5 of $60 still triggers reassessment; use Auto-fill my return.
85 +- RRSP **contribution room** ≠ **deduction claimed**: you may contribute now and deduct in a later, higher-income year; over-contribution beyond the $2,000 cushion costs 1%/month.
86 +- First-60-days RRSP receipts belong on the **prior** year's return (reported, even if deduction deferred).
87 +- TFSA contributions are never deductible and withdrawals re-add room only the **next** January 1 — re-contributing the same year over-contributes.
88 +- Self-employed June 15 deadline is filing-only; interest on any balance runs from May 1.
89 +- Provincial credits differ widely (rent credits, political donations) — check the province's schedule before declaring the return complete.
90 +- Capital gains need your own ACB records; T5008 often lacks or misstates cost basis.
added finance-skills/preparing-financial-statements/SKILL.md +52 −0
@@ -0,0 +1,52 @@
1 +---
2 +name: preparing-financial-statements
3 +description: Builds the three financial statements — income statement, balance sheet, and cash flow statement — from a trial balance, with correct classification and cross-statement ties. Use when the user asks to prepare, draft, or review financial statements, a P&L or income statement, a balance sheet, a cash flow statement, year-end or month-end statements, or asks which framework applies (US GAAP, IFRS, or Canadian ASPE). Do not use for day-to-day bookkeeping (bookkeeping-for-small-businesses), tax filings (filing-us-business-taxes, filing-canadian-business-taxes), or public-company audited reporting.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Preparing Financial Statements
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** producing an income statement, balance sheet, and cash flow statement from closed books/trial balance; classifying accounts; checking that statements tie; choosing/disclosing the framework (GAAP / IFRS / ASPE).
15 +- **Do NOT use for:** recording transactions (→ `bookkeeping-for-small-businesses`), tax returns (→ the filing skills), or audited public-company reporting — that requires a licensed auditor.
16 +
17 +## Important limits
18 +- Educational help, not professional accounting advice — statements for lenders, investors, or audits go through a CPA.
19 +- Framework guidance is tax-year/standard-stamped and MUST be verified against current GAAP/IFRS/ASPE pronouncements.
20 +- Never assist with misstating results or falsifying records; refuse and explain.
21 +
22 +## Core rules
23 +
24 +1. **Start from a reconciled trial balance.** If debits ≠ credits, stop and fix the books first — statements built on an unbalanced trial balance are fiction.
25 +2. **Income statement structure is fixed:** Revenue − COGS = Gross profit; − Operating expenses = Operating income; ± Interest/other = Pre-tax income; − Income tax = **Net income**.
26 + - ✅ COGS separated from operating expenses
27 + - ❌ One undifferentiated "Expenses" block
28 +3. **Balance sheet MUST balance:** Assets = Liabilities + Equity, exactly. Classify current (≤12 months) vs non-current on both sides. Accumulated depreciation shown as a contra-asset.
29 +4. **Cash flow statement: indirect method by default.** Operating = net income + non-cash addbacks (depreciation/amortization) ± working-capital changes (AR↑ subtracts, AP↑ adds); Investing = asset purchases/sales; Financing = loans and owner contributions/draws/dividends.
30 +5. **The statements must tie.** Net income flows into retained earnings (ending RE = beginning RE + net income − dividends/draws); cash flow ending cash = balance sheet cash. If either tie fails, a statement is wrong.
31 +6. **Name the framework on the statements.** US private companies: US GAAP. Canadian private companies: ASPE by default (IFRS if public or required by lenders). State the basis and whether statements are unaudited (e.g., "Notice to Reader / compilation").
32 +7. **Show comparatives.** Present the prior period alongside the current one; explain material swings in the notes.
33 +8. **Notes for anything material:** accounting basis, depreciation method/useful lives, loan terms, related-party transactions, commitments.
34 +
35 +## Workflow
36 +
37 +1. Obtain the closed, reconciled trial balance for the period (and prior period for comparatives).
38 +2. Map each account to its statement line (use the classification table in the reference).
39 +3. Build the income statement top-down; compute net income.
40 +4. Build the balance sheet; roll retained earnings forward with net income and draws/dividends.
41 +5. Build the cash flow statement (indirect); derive ending cash.
42 +6. **Validate the three ties:** (a) balance sheet balances, (b) retained-earnings roll-forward uses the income statement's net income, (c) cash flow ending cash equals balance sheet cash. All three must pass before delivery.
43 +7. Add framework/basis note, comparatives, and material-item notes.
44 +
45 +## Edge cases & failure modes
46 +- **Balance sheet off by exactly net income** → retained earnings not rolled forward.
47 +- **Cash tie fails** → most often a missed financing item (owner draw/loan principal) or a working-capital sign error.
48 +- **Negative equity** → present it plainly; do not net it away — flag going-concern language to a CPA.
49 +- **Cash-basis books** → say so on the statements ("cash basis") rather than silently presenting them as GAAP/ASPE.
50 +
51 +## References
52 +Statement templates with a fully tied worked example and classification tables: see [references/reference.md](references/reference.md).
added finance-skills/preparing-financial-statements/references/reference.md +82 −0
@@ -0,0 +1,82 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Reference — Preparing Financial Statements
7 +
8 +## Contents
9 +- Account classification table
10 +- Worked example: trial balance → three tied statements
11 +- Framework comparison (US GAAP / IFRS / ASPE)
12 +- Gotchas
13 +
14 +## Account classification table
15 +
16 +| Account | Statement | Line |
17 +|---|---|---|
18 +| Cash, AR, inventory, prepaid | Balance sheet | Current assets |
19 +| Equipment, vehicles, leaseholds | Balance sheet | Non-current assets |
20 +| Accumulated depreciation | Balance sheet | Contra to non-current assets |
21 +| AP, credit cards, sales tax payable, payroll liabilities | Balance sheet | Current liabilities |
22 +| Loan principal due ≤12 months | Balance sheet | Current portion of long-term debt |
23 +| Loan principal due >12 months | Balance sheet | Non-current liabilities |
24 +| Capital, draws, retained earnings | Balance sheet | Equity |
25 +| Sales, refunds/discounts | Income statement | Revenue (net) |
26 +| COGS | Income statement | Cost of goods sold |
27 +| Rent, wages, software, insurance, depreciation | Income statement | Operating expenses |
28 +| Interest expense | Income statement | Below operating income |
29 +
30 +## Worked example (year 1, small consultancy)
31 +
32 +**Trial balance (Dec 31, 2026):**
33 +
34 +| Account | Dr | Cr |
35 +|---|---|---|
36 +| Cash | 34,000 | |
37 +| Accounts receivable | 6,000 | |
38 +| Equipment | 10,000 | |
39 +| Accumulated depreciation | | 2,000 |
40 +| Accounts payable | | 3,000 |
41 +| Loan payable | | 8,000 |
42 +| Owner's capital | | 5,000 |
43 +| Owner's draws | 20,000 | |
44 +| Revenue | | 90,000 |
45 +| Operating expenses | 36,000 | |
46 +| Depreciation expense | 2,000 | |
47 +| **Totals** | **108,000** | **108,000** |
48 +
49 +**Income statement:** Revenue 90,000 − Operating expenses 36,000 − Depreciation 2,000 = **Net income 52,000**
50 +
51 +**Balance sheet:**
52 +- Assets: Cash 34,000 + AR 6,000 + Equipment 10,000 − Accum. dep. 2,000 = **48,000**
53 +- Liabilities: AP 3,000 + Loan 8,000 = 11,000
54 +- Equity: Capital 5,000 + Net income 52,000 − Draws 20,000 = 37,000
55 +- Liabilities + Equity = **48,000** ✔ balances
56 +
57 +**Cash flow (indirect):**
58 +- Operating: 52,000 + 2,000 depreciation − 6,000 AR increase + 3,000 AP increase = 51,000
59 +- Investing: −10,000 equipment
60 +- Financing: +5,000 capital +8,000 loan −20,000 draws = −7,000
61 +- Net change: 34,000; beginning cash 0 → **ending cash 34,000** ✔ ties to balance sheet
62 +
63 +All three ties hold: balance sheet balances; RE roll-forward uses 52,000; ending cash matches.
64 +
65 +## Framework comparison
66 +
67 +| Aspect | US GAAP | IFRS | ASPE (Canada, private) |
68 +|---|---|---|---|
69 +| Who must use | US companies (public: SEC GAAP) | Canadian public companies; optional elsewhere | Default for Canadian private enterprises |
70 +| Complexity | High | High | Simplified, made-in-Canada |
71 +| Revaluation of fixed assets | No | Allowed | No |
72 +| Development costs | Expensed (mostly) | Capitalize if criteria met | Policy choice |
73 +| Statement titles | Flexible | "Statement of financial position" etc. | Traditional titles common |
74 +
75 +Disclose the basis on the statements (e.g., "Prepared in accordance with ASPE, unaudited").
76 +
77 +## Gotchas
78 +- **Draws/dividends never touch the income statement** — equity only. A P&L with "owner draw expense" overstates costs.
79 +- **Working-capital signs** in the cash flow trip everyone: asset increases consume cash (subtract); liability increases provide cash (add).
80 +- **Current portion of long-term debt** must be split out yearly or current liabilities are understated.
81 +- **Refunds** net against revenue, not expenses.
82 +- **Comparative columns must use the same account mapping** — a reclassified account needs the prior year restated or a note.
added finance-skills/preparing-us-personal-tax-returns/SKILL.md +51 −0
@@ -0,0 +1,51 @@
1 +---
2 +name: preparing-us-personal-tax-returns
3 +description: Helps organize and prepare a US federal individual income tax return (Form 1040) — document checklist, filing status, standard vs itemized decision, schedule mapping, credits, deadlines, and estimated-tax safe harbors. Use when the user asks for help preparing, organizing, or checking their US personal tax return, 1040, W-2/1099 documents, IRS filing deadlines, refund estimation, or which schedules and credits apply. Do not use for Canadian returns (preparing-canadian-personal-tax-returns), business entity returns (filing-us-business-taxes), or tax planning strategy (optimizing-us-personal-taxes).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Preparing US Personal Tax Returns
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** organizing documents, mapping income and deductions to Form 1040 and its schedules, choosing filing status, standard-vs-itemized comparison, credit eligibility screening, deadline and payment questions for an individual federal return.
15 +- **Do NOT use for:** Canadian T1 returns, business entity returns (1120/1120-S/1065), state returns (mention that a state return likely also applies, then stop), or planning strategy for future years.
16 +
17 +## Important limits
18 +- Educational preparation help, not professional tax advice — complex situations (multi-state, equity compensation, trusts, audits) go to a CPA or EA.
19 +- All figures below are stated for **tax year 2026** and MUST be verified against [irs.gov](https://www.irs.gov) before use.
20 +- Never assist with evasion — unreported income or fabricated deductions are out of scope; if asked, decline and explain the legal alternative.
21 +
22 +## Workflow
23 +1. Collect documents against the checklist: W-2s (every employer), 1099s (NEC, MISC, INT, DIV, B, R, K, G, SSA), 1098 (mortgage) / 1098-T (tuition) / 1098-E (student loan), HSA forms (1099-SA, 5498-SA), prior-year return, IP PIN if issued.
24 +2. Determine filing status (Single, MFJ, MFS, Head of Household, Qualifying Surviving Spouse) — marital status on December 31 controls; HoH requires an unmarried filer paying >half the cost of a home for a qualifying person.
25 +3. Map each income item to its schedule (see the schedule map in [references/reference.md](references/reference.md)).
26 +4. Compare standard deduction vs itemized (Schedule A: SALT, mortgage interest, medical above the AGI floor, donations). Pick the larger; note MFS must both itemize or both take standard.
27 +5. Screen credits: Child Tax Credit, EITC, education credits (AOTC/LLC), child & dependent care, Saver's Credit — check phase-outs in the reference file.
28 +6. Compute balance due or refund; if balance due is large, check the estimated-tax safe harbor for next year (90% of current-year tax, or 100% of prior-year tax — 110% if prior-year AGI exceeded $150,000).
29 +7. **Validate:** every document from step 1 appears on the return; totals on the return match source-document totals; filing status is consistent across the return; flag any 1099 income with no matching entry.
30 +
31 +## Current figures (tax year 2026 — verify before use)
32 +| Item | Value | Source |
33 +|---|---|---|
34 +| Brackets | 10 / 12 / 22 / 24 / 32 / 35 / 37% | irs.gov |
35 +| Standard deduction | $16,100 single · $32,200 MFJ | irs.gov |
36 +| Filing deadline | April 15, 2027 (TY2026) | irs.gov |
37 +| Extension (Form 4868) | To Oct 15 — extends filing, NOT payment | irs.gov |
38 +| Estimated-tax safe harbor | 90% current / 100% prior (110% if AGI > $150k) | irs.gov |
39 +| 401(k) elective deferral | $24,500 (+$8,000 catch-up 50+) | irs.gov |
40 +| IRA limit | $7,500 (+$1,100 catch-up) | irs.gov |
41 +| HSA limit | $4,400 self / $8,750 family | irs.gov |
42 +
43 +## Edge cases & failure modes
44 +- **Missing W-2/1099** → request a wage and income transcript from the IRS; never estimate silently.
45 +- **Error found after filing** → amend with Form 1040-X; do not re-file a second original.
46 +- **First-time filer** → no prior-year AGI for e-file identity check; use $0.
47 +- **Can't pay** → file anyway: the failure-to-file penalty (5%/month) is 10× the failure-to-pay penalty (0.5%/month); set up an IRS payment plan.
48 +- **Marketplace health coverage (1095-A)** → Form 8962 is mandatory; the return will reject without it.
49 +
50 +## References
51 +Full schedule map, credit phase-out table, deadline calendar, penalty structure, and worked examples: see [references/reference.md](references/reference.md).
added finance-skills/preparing-us-personal-tax-returns/references/reference.md +103 −0
@@ -0,0 +1,103 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Reference — US Personal Tax Return Preparation (Form 1040)
7 +
8 +All dollar figures are **tax year 2026** unless noted — verify at [irs.gov](https://www.irs.gov) before use.
9 +
10 +## Contents
11 +- Schedule and form map
12 +- Common income documents
13 +- Standard vs itemized decision
14 +- Credit table with phase-out notes
15 +- Deadline calendar
16 +- Penalty structure
17 +- Worked mini-examples
18 +- Gotchas
19 +
20 +## Schedule and form map
21 +
22 +| Situation | Where it goes |
23 +|---|---|
24 +| Wages (W-2) | 1040 line 1 |
25 +| Interest / ordinary dividends | Schedule B (if > $1,500), else 1040 directly |
26 +| Self-employment / gig income (1099-NEC, 1099-K) | Schedule C → SE tax on Schedule SE → Schedule 2 |
27 +| Capital gains, crypto/stock sales (1099-B) | Form 8949 → Schedule D |
28 +| Rental income, royalties, K-1 pass-through | Schedule E |
29 +| Adjustments (student loan interest, HSA, IRA deduction, ½ SE tax) | Schedule 1 Part II |
30 +| Other income (unemployment 1099-G, prizes, hobby) | Schedule 1 Part I |
31 +| Additional taxes (SE tax, early-withdrawal penalty, NIIT, AMT) | Schedule 2 |
32 +| Nonrefundable/refundable credits beyond CTC/EITC | Schedule 3 |
33 +| Itemized deductions | Schedule A |
34 +| Premium Tax Credit reconciliation (1095-A) | Form 8962 |
35 +| Retirement distributions (1099-R) | 1040 lines 4–5; Form 5329 if early |
36 +
37 +## Common income documents
38 +
39 +- **W-2** — wages; one per employer; box 12 codes matter (D = 401(k), W = HSA employer contribution).
40 +- **1099-NEC** — contractor income ≥ $600 → Schedule C, even without the form.
41 +- **1099-INT / 1099-DIV** — interest/dividends; qualified dividends get capital-gains rates.
42 +- **1099-B** — broker sales; check basis was reported; reconcile on Form 8949.
43 +- **1099-R** — retirement distributions; code in box 7 drives taxability and penalty.
44 +- **1099-K** — payment platforms; report gross then back out personal/nontaxable items on Schedule 1.
45 +- **SSA-1099** — up to 85% of Social Security may be taxable depending on combined income.
46 +
47 +## Standard vs itemized decision
48 +
49 +Itemize on Schedule A only if the sum beats the standard deduction ($16,100 single / $32,200 MFJ):
50 +- State and local taxes (SALT) — capped; verify the current cap for the tax year.
51 +- Home mortgage interest (1098) — acquisition-debt limits apply.
52 +- Medical expenses — only the portion above 7.5% of AGI.
53 +- Charitable donations — cash and fair-market-value of goods; receipts required at $250+.
54 +
55 +Rule: compute both, take the larger, and record the comparison so the choice is auditable.
56 +
57 +## Credit table
58 +
59 +| Credit | Type | Key conditions |
60 +|---|---|---|
61 +| Child Tax Credit | Partially refundable | Qualifying child under 17 with SSN; phases out at higher AGI |
62 +| EITC | Refundable | Earned income + AGI limits by family size; investment-income cap; frequent audit target — document eligibility |
63 +| AOTC (education) | Partially refundable | First 4 years post-secondary, per student, needs 1098-T |
64 +| Lifetime Learning | Nonrefundable | Any post-secondary; per return |
65 +| Child & Dependent Care | Nonrefundable | Care enabling work; needs provider EIN/SSN |
66 +| Saver's Credit | Nonrefundable | Retirement contributions at lower AGI |
67 +| Premium Tax Credit | Refundable | Marketplace coverage; reconcile with Form 8962 — mandatory |
68 +
69 +Verify every phase-out threshold for the tax year at irs.gov; do not quote from memory.
70 +
71 +## Deadline calendar (for tax year 2026)
72 +
73 +| Date | Event |
74 +|---|---|
75 +| Jan 15, 2027 | Q4 2026 estimated payment |
76 +| Jan 31, 2027 | W-2s and most 1099s due to recipients |
77 +| Apr 15, 2027 | Filing + payment deadline; Q1 2027 estimate; IRA/HSA contribution deadline for 2026 |
78 +| Jun 15, 2027 | Q2 estimate; automatic 2-month filing extension for taxpayers abroad (interest still runs) |
79 +| Sep 15, 2027 | Q3 estimate |
80 +| Oct 15, 2027 | Extended filing deadline (Form 4868 filed by Apr 15) |
81 +
82 +## Penalty structure
83 +
84 +- **Failure to file:** 5% of unpaid tax per month, max 25%. Minimum penalty applies after 60 days.
85 +- **Failure to pay:** 0.5% per month, max 25%; drops to 0.25% under an installment agreement.
86 +- Both apply → file penalty is reduced by pay penalty for the overlap; filing on time is always worth it.
87 +- **Estimated-tax underpayment:** interest-based (Form 2210); avoided via safe harbor (90% current / 100% prior / 110% if prior AGI > $150k) or owing < $1,000.
88 +
89 +## Worked mini-examples
90 +
91 +**Example 1 — standard vs itemized.** Single filer: SALT $9,000 (under cap), mortgage interest $5,500, donations $1,200 → itemized $15,700 < standard $16,100 → take standard.
92 +
93 +**Example 2 — safe harbor.** 2025 total tax $18,000, AGI $120,000; 2026 income jumps. Paying 100% × $18,000 = $18,000 through withholding/estimates avoids the underpayment penalty regardless of the 2026 bill.
94 +
95 +**Example 3 — gig worker.** 1099-NEC $22,000, expenses $4,000 → Schedule C net $18,000 → SE tax ≈ 15.3% × (18,000 × 0.9235) on Schedule SE; half of SE tax deducts on Schedule 1.
96 +
97 +## Gotchas
98 +- An extension extends **filing only** — estimate and pay by April 15 or penalties/interest run.
99 +- 1099-K gross ≠ income: refunds, personal transfers, and basis must be adjusted, not ignored.
100 +- Dependents with jobs: they file their own return but must check "can be claimed as a dependent" — double-claiming causes e-file rejects.
101 +- Backdoor/IRA basis: Form 8606 must be filed every year with nondeductible contributions or basis is lost.
102 +- MFS traps: loses EITC, education credits, and usually the child-care credit; both spouses must match standard/itemized choice.
103 +- The IRS already has your W-2s/1099s — mismatches trigger automated CP2000 notices; reconcile every document, even small ones.
added frontend-skills/README.md +38 −0
@@ -0,0 +1,38 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# frontend-skills — Frontend Design Skill Collection
7 +
8 +**Author:** Simon-Pierre Boucher
9 +**Contact:** contact@spboucher.ai
10 +
11 +Ten ultra-sharp frontend-design skills following the method in
12 +[../RESEARCH-SYNTHESIS.md](../RESEARCH-SYNTHESIS.md), grounded in Anthropic's
13 +open-source `frontend-design` skill and 2026 web standards (WCAG 2.2 AA,
14 +INP ≤200 ms, LCP ≤2.5 s, CLS ≤0.1, mobile-first). All pass
15 +`python3 ../tools/validate_skills.py`.
16 +
17 +## The collection and its boundaries
18 +
19 +| Skill | Handles | Explicitly does NOT handle |
20 +|---|---|---|
21 +| `structuring-semantic-html` | landmarks, heading hierarchy, meaning-bearing markup, SEO meta | ARIA/keyboard work → `ensuring-accessibility`; editing HTML files → `processing-html` |
22 +| `designing-responsive-layouts` | flexbox/grid, breakpoints, container queries, fluid sizing | page composition → `creating-landing-pages`; tokens → `theming-design-tokens` |
23 +| `theming-design-tokens` | token tiers, dark mode, spacing/shadow scales, contrast-safe palettes | typefaces/type scales → `choosing-typography` |
24 +| `choosing-typography` | display/body pairing, type scale, measure, font loading | color/spacing tokens → `theming-design-tokens`; copywriting |
25 +| `ensuring-accessibility` | WCAG 2.2 AA, keyboard/focus, ARIA, contrast, targets | semantic structure → `structuring-semantic-html`; visual design |
26 +| `crafting-ui-animations` | micro-interactions, orchestration, transform/opacity rule, reduced motion | a11y audits → `ensuring-accessibility`; load perf → `optimizing-web-performance` |
27 +| `designing-forms` | labels, validation timing, error states, grouping, multi-step | backend validation; a11y audits → `ensuring-accessibility` |
28 +| `building-react-components` | props API, composition, state placement, hooks | CSS layout → `designing-responsive-layouts`; non-React frameworks |
29 +| `optimizing-web-performance` | Core Web Vitals budgets, images, fonts, splitting, measurement | animation smoothness → `crafting-ui-animations`; backend latency |
30 +| `creating-landing-pages` | hero thesis, CTA rhythm, section flow, signature element, copy | layout mechanics → `designing-responsive-layouts`; tokens → `theming-design-tokens` |
31 +
32 +## Shared conventions
33 +
34 +- Description = WHAT + "Use when …" (literal phrases) + "Do not use for …"
35 +- Core rules are numbered and actionable, with ✅/❌ pairs where style matters
36 +- Concrete values over adjectives (contrast ratios, ms durations, px targets)
37 +- Every workflow ends with a self-review/validation step
38 +- `SKILL.md` <150 lines; depth in `references/patterns.md` (with TOC)
added frontend-skills/building-react-components/SKILL.md +62 −0
@@ -0,0 +1,62 @@
1 +---
2 +name: building-react-components
3 +description: Designs and writes React components with clean props APIs, composition over configuration, correct state placement, and hooks-based logic reuse. Use when the user asks to create, refactor, or review a React component, design a component API, decide where state should live, extract a custom hook, or fix prop drilling or unnecessary re-renders. Do not use for CSS layout mechanics (designing-responsive-layouts) or non-React frameworks — adapt the principles manually there.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Building React Components
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** creating or refactoring React components, designing props APIs, placing state, extracting custom hooks, reviewing component structure.
15 +- **Do NOT use for:** CSS layout/breakpoint work (designing-responsive-layouts), styling systems (theming-design-tokens), Vue/Svelte/Angular components, or backend data modeling.
16 +
17 +## Core rules
18 +
19 +1. **Narrow, typed props — no boolean explosions.** One `variant` union beats three flags that can contradict each other.
20 + - ✅ `variant: 'primary' | 'danger' | 'ghost'`
21 + - ❌ `isPrimary`, `isDanger`, `isGhost` as three separate booleans
22 +
23 +2. **Composition over configuration.** When a component grows a prop per content slot, switch to `children` or slot components.
24 + - ✅ `<Card><Card.Header>…</Card.Header><Card.Body>…</Card.Body></Card>`
25 + - ❌ `<Card headerText="…" headerIcon="…" bodyContent={…} footerButtons={…} />`
26 +
27 +3. **State lives at the lowest component that needs it; lift only when shared.** Before `useState`, pick the state's home: server data → query library cache; shareable/bookmarkable → URL; everything else → local state closest to use.
28 + - ✅ search text in the `SearchBox`, results in the query cache, filters in the URL
29 + - ❌ every field of a page hoisted into one context "to be safe"
30 +
31 +4. **Pick controlled or uncontrolled per input and stay consistent.** Controlled (`value` + `onChange`) when other UI reacts per keystroke; uncontrolled (`defaultValue` + read on submit) for plain forms.
32 + - ❌ `value` without `onChange`, or switching between the two mid-lifecycle
33 +
34 +5. **Extract reusable logic into custom hooks, not wrapper components.** A hook named `useX` returning plain values beats render-props or HOC indirection.
35 + - ✅ `const { data, error } = usePolling(url, 5000)`
36 + - ❌ `<PollingProvider render={(data) => …} />`
37 +
38 +6. **Memoize only after measuring.** No `React.memo`/`useMemo`/`useCallback` by default; add them when the Profiler shows a real re-render cost, and comment why.
39 + - ❌ wrapping every callback in `useCallback` "for performance"
40 +
41 +7. **One component per file, named exports, file named after the component.** `UserMenu.tsx` exports `UserMenu`; its private subcomponents stay in the same file until reused elsewhere.
42 +
43 +8. **Derive, don't sync.** Values computable from existing props/state are computed during render — never mirrored into state with an effect.
44 + - ✅ `const fullName = first + ' ' + last`
45 + - ❌ `useEffect(() => setFullName(first + ' ' + last), [first, last])`
46 +
47 +## Workflow
48 +
49 +1. Name the component and write its props type first — if the type needs more than ~7 props or any boolean pair, redesign with rules 1–2.
50 +2. Decide each piece of state's home (rule 3) before writing any `useState`.
51 +3. Implement render logic; derive values instead of syncing state (rule 8).
52 +4. Extract any logic used twice into a custom hook (rule 5).
53 +5. Self-review: re-check every prop against rules 1–2, every `useState` against rules 3 and 8, every memoization against rule 6. Fix violations before delivering.
54 +
55 +## Edge cases & failure modes
56 +- **Existing codebase conventions conflict with these rules** → match the codebase; note the divergence in one sentence, don't refactor uninvited.
57 +- **Class components in the file being edited** → keep the class style for small edits; propose (don't perform) a hooks migration.
58 +- **Prop drilling more than 2 levels** → prefer composition (pass the composed element down) before reaching for context.
59 +- **Server components (Next.js/RSC)** → hooks and state are client-only; add `'use client'` only at the interactive leaf, not the page root.
60 +
61 +## References
62 +Copy-paste patterns and gotchas: see [references/patterns.md](references/patterns.md)
added frontend-skills/building-react-components/references/patterns.md +133 −0
@@ -0,0 +1,133 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Building React Components
7 +
8 +## Contents
9 +- Props API: variant unions and discriminated props
10 +- Composition: compound components
11 +- State placement: the decision table
12 +- Controlled vs uncontrolled inputs
13 +- Custom hooks
14 +- Measured memoization
15 +- Gotchas
16 +
17 +## Props API: variant unions and discriminated props
18 +
19 +```tsx
20 +type ButtonProps = {
21 + variant?: 'primary' | 'danger' | 'ghost'; // one axis, one prop
22 + size?: 'sm' | 'md' | 'lg';
23 + disabled?: boolean; // true independent boolean is fine
24 + onClick: () => void;
25 + children: React.ReactNode;
26 +};
27 +
28 +export function Button({ variant = 'primary', size = 'md', ...rest }: ButtonProps) { /* … */ }
29 +```
30 +
31 +Discriminated union when props only make sense together:
32 +
33 +```tsx
34 +type AlertProps =
35 + | { kind: 'info'; message: string }
36 + | { kind: 'error'; message: string; retry: () => void }; // retry exists only for errors
37 +```
38 +
39 +## Composition: compound components
40 +
41 +```tsx
42 +export function Card({ children }: { children: React.ReactNode }) {
43 + return <section className="card">{children}</section>;
44 +}
45 +Card.Header = function CardHeader({ children }: { children: React.ReactNode }) {
46 + return <header className="card-header">{children}</header>;
47 +};
48 +Card.Body = function CardBody({ children }: { children: React.ReactNode }) {
49 + return <div className="card-body">{children}</div>;
50 +};
51 +
52 +// Usage — caller controls content, Card controls chrome:
53 +<Card>
54 + <Card.Header><h2>Invoices</h2></Card.Header>
55 + <Card.Body><InvoiceTable rows={rows} /></Card.Body>
56 +</Card>
57 +```
58 +
59 +Escape hatch: if slots must be validated/reordered, accept named element props
60 +(`header={<h2>…</h2>}`) — still elements, not strings-plus-config.
61 +
62 +## State placement: the decision table
63 +
64 +| The value is… | Home | Tool |
65 +|---|---|---|
66 +| Fetched from an API | Query cache | TanStack Query / SWR |
67 +| Shareable via link (filters, tab, page) | URL | `useSearchParams` |
68 +| Form input mid-edit | Local component | `useState` / form library |
69 +| Theme, auth, locale (rarely changes, read widely) | Context | `createContext` |
70 +| Everything else | Lowest component that uses it | `useState` |
71 +
72 +## Controlled vs uncontrolled inputs
73 +
74 +```tsx
75 +// Controlled — other UI reacts per keystroke
76 +const [query, setQuery] = useState('');
77 +<input value={query} onChange={(e) => setQuery(e.target.value)} />
78 +<Results filter={query} />
79 +
80 +// Uncontrolled — value only needed on submit
81 +<form onSubmit={(e) => {
82 + e.preventDefault();
83 + const data = new FormData(e.currentTarget);
84 + save(data.get('email'));
85 +}}>
86 + <input name="email" defaultValue={user.email} />
87 +</form>
88 +```
89 +
90 +## Custom hooks
91 +
92 +```tsx
93 +// Reusable logic = hook. Name starts with `use`, returns plain values.
94 +function useDebouncedValue<T>(value: T, delayMs = 300) {
95 + const [debounced, setDebounced] = useState(value);
96 + useEffect(() => {
97 + const id = setTimeout(() => setDebounced(value), delayMs);
98 + return () => clearTimeout(id);
99 + }, [value, delayMs]);
100 + return debounced;
101 +}
102 +
103 +const debouncedQuery = useDebouncedValue(query); // any component, same behavior
104 +```
105 +
106 +## Measured memoization
107 +
108 +```tsx
109 +// 1. Measure first: React DevTools Profiler → record → find components
110 +// re-rendering with unchanged props AND non-trivial render cost.
111 +// 2. Then, and only then:
112 +const Row = React.memo(function Row({ item }: { item: Item }) { /* … */ });
113 +// Parent must keep prop identities stable for memo to work:
114 +const onSelect = useCallback((id: string) => setSelected(id), []);
115 +// Comment the reason so the next reader knows it's load-bearing:
116 +// memo: 5k rows re-rendered on every keystroke before this (Profiler 2026-08).
117 +```
118 +
119 +## Gotchas
120 +
121 +- **`useEffect` for derivation** is the top React bug source — if the effect only
122 + calls `setState` from other state/props, delete it and compute during render.
123 +- **Index as `key`** breaks reordering/deletion — use a stable id; index is
124 + acceptable only for static, never-reordered lists.
125 +- **Object/array literals in JSX props** (`style={{…}}`, `options={[…]}`) defeat
126 + `React.memo` children — hoist or memoize them only where memo is in play.
127 +- **Context triggers re-render of every consumer** on any value change — split
128 + contexts (state vs dispatch) or keep fast-changing values out of context.
129 +- **`useCallback` without a memoized child** is pure overhead — it saves nothing.
130 +- **Stale closure in intervals/subscriptions**: include deps or use a ref;
131 + an empty dep array freezes captured state at mount time.
132 +- **`'use client'` at the page root** opts the whole tree out of server
133 + rendering — place it at the interactive leaf component instead.
added frontend-skills/choosing-typography/SKILL.md +54 −0
@@ -0,0 +1,54 @@
1 +---
2 +name: choosing-typography
3 +description: Chooses and systematizes web typography — display/body typeface pairing matched to the brief, modular type scales with clamp(), line-height and line-length rules, and performant font loading with woff2 and font-display. Use when the user asks to pick or pair fonts, set up a type scale or heading sizes, fix readability or line length, load web fonts, or use variable fonts. Do not use for color, spacing, or theming token systems (theming-design-tokens) or for writing the copy itself.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Choosing Typography
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** typeface selection/pairing, type scales, line-height/measure, web-font loading, variable fonts.
15 +- **Do NOT use for:** color/spacing tokens (→ theming-design-tokens); page layout (→ designing-responsive-layouts); the words themselves — copy is not typography.
16 +
17 +## Core rules
18 +
19 +1. **Pair for the brief, not from the AI-default shelf.** Pick one display face with character matched to the subject and one quiet, workhorse body face; justify the choice in one sentence tied to the brief.
20 + - ✅ "Fintech dashboard → IBM Plex Sans (technical heritage) + Plex Mono for figures"
21 + - ❌ Playfair Display + Inter on cream, or any pairing you'd reach for on *every* project
22 +
23 +2. **Two families maximum (display + body); get hierarchy from weight/size/case, not more fonts.**
24 +
25 +3. **Modular scale, fluid at the top.** Body fixed at 1rem (16px minimum — never smaller); headings on a ratio (1.25 default; 1.333+ for editorial drama), made fluid with `clamp()`.
26 + - ✅ `h1 { font-size: clamp(2rem, 1.3rem + 3vw, 3.5rem); }`
27 + - ❌ `h1 { font-size: 56px; }` (desktop-only) or `font-size: 5vw` (unbounded, breaks zoom)
28 +
29 +4. **Line-height by role:** body 1.5–1.7, headings 1.1–1.25, buttons/labels 1. Unitless values only.
30 +
31 +5. **Measure 45–75 characters:** `max-width: 65ch` on prose containers. Long lines are the most common readability failure.
32 +
33 +6. **Loading discipline:** self-hosted `woff2` only, `font-display: swap`, preload the one or two files used above the fold, and set `size-adjust`-matched fallbacks to keep CLS ≤ 0.1.
34 +
35 +7. **Variable fonts when you need >2 weights of one family** — one file replaces four; animate weight sparingly and never on body text.
36 +
37 +8. **Numbers in tables/dashboards get `font-variant-numeric: tabular-nums`** so columns align.
38 +
39 +## Workflow
40 +
41 +1. Read the brief; write one sentence naming the personality the type must carry.
42 +2. Choose display + body per rules 1–2 (with licensing/availability check — Google Fonts, Fontshare, or the client's licensed faces).
43 +3. Build the scale and roles (rules 3–5) as CSS custom properties.
44 +4. Set up loading (rule 6): woff2 subset, preload, fallback stack with metric overrides.
45 +5. Validate: body ≥16px; measure within 45–75ch at 360px and 1280px; headings don't wrap awkwardly at 360px; toggle network throttling — text visible immediately (swap) and no visible layout jump when the web font lands.
46 +
47 +## Edge cases & failure modes
48 +- **Brand mandates a display-only face for body text:** refuse silently by scoping it to headings/pull-quotes; pick a compatible body face and note the substitution.
49 +- **Font file unavailable/offline build:** system stack fallback — `system-ui, -apple-system, "Segoe UI", Roboto, sans-serif` — and say so; never hotlink a foundry's CDN without license.
50 +- **Multilingual content:** confirm the chosen faces cover the required scripts/diacritics before committing; fall back per-script with `unicode-range`.
51 +- **User zoom/large-text settings:** everything in `rem` (rule 3) — px-based type breaks zoom accessibility.
52 +
53 +## References
54 +Scale sheet, @font-face/preload boilerplate, metric-matched fallbacks, pairing shortlists by genre: see [references/patterns.md](references/patterns.md)
added frontend-skills/choosing-typography/references/patterns.md +125 −0
@@ -0,0 +1,125 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Typography Systems
7 +
8 +## Contents
9 +- Type-scale token sheet
10 +- @font-face + preload boilerplate
11 +- Metric-matched fallback (CLS-safe)
12 +- Variable font setup
13 +- Prose defaults
14 +- Pairing shortlists by genre
15 +- Gotchas
16 +
17 +## Type-scale token sheet
18 +
19 +```css
20 +:root {
21 + --font-display: "Fraunces", var(--font-fallback-serif);
22 + --font-body: "Inter", var(--font-fallback-sans);
23 + --font-fallback-sans: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
24 + --font-fallback-serif: Georgia, "Times New Roman", serif;
25 +
26 + /* Ratio 1.25 (major third); h1 fluid, the rest step down */
27 + --text-sm: 0.875rem;
28 + --text-base: 1rem; /* never below 16px */
29 + --text-lg: 1.25rem;
30 + --text-xl: 1.563rem;
31 + --text-2xl: clamp(1.75rem, 1.3rem + 1.8vw, 1.953rem);
32 + --text-3xl: clamp(2rem, 1.3rem + 3vw, 2.441rem);
33 +
34 + --leading-body: 1.6;
35 + --leading-heading: 1.15;
36 +}
37 +
38 +h1 { font: 700 var(--text-3xl)/var(--leading-heading) var(--font-display); }
39 +h2 { font: 700 var(--text-2xl)/var(--leading-heading) var(--font-display); }
40 +h3 { font: 600 var(--text-xl)/1.25 var(--font-body); }
41 +body { font: 400 var(--text-base)/var(--leading-body) var(--font-body); }
42 +```
43 +
44 +## @font-face + preload boilerplate
45 +
46 +```html
47 +<!-- Preload ONLY above-the-fold weights (usually body 400 + display 700) -->
48 +<link rel="preload" href="/fonts/inter-400.woff2" as="font" type="font/woff2" crossorigin>
49 +<link rel="preload" href="/fonts/fraunces-700.woff2" as="font" type="font/woff2" crossorigin>
50 +```
51 +
52 +```css
53 +@font-face {
54 + font-family: "Inter";
55 + src: url("/fonts/inter-400.woff2") format("woff2");
56 + font-weight: 400;
57 + font-style: normal;
58 + font-display: swap; /* text visible immediately in fallback */
59 +}
60 +```
61 +
62 +## Metric-matched fallback (CLS-safe)
63 +
64 +```css
65 +/* Tune size-adjust until fallback and web font occupy the same space.
66 + Tools output these (e.g. fontaine, capsize); values below fit Inter/Arial. */
67 +@font-face {
68 + font-family: "Inter-fallback";
69 + src: local("Arial");
70 + size-adjust: 107%;
71 + ascent-override: 90%;
72 + descent-override: 22.5%;
73 + line-gap-override: 0%;
74 +}
75 +body { font-family: "Inter", "Inter-fallback", sans-serif; }
76 +```
77 +
78 +## Variable font setup
79 +
80 +```css
81 +@font-face {
82 + font-family: "Fraunces";
83 + src: url("/fonts/fraunces-vf.woff2") format("woff2-variations");
84 + font-weight: 300 900; /* the range the file supports */
85 + font-display: swap;
86 +}
87 +.hero-title { font-variation-settings: "opsz" 72; font-weight: 640; }
88 +```
89 +
90 +## Prose defaults
91 +
92 +```css
93 +.prose {
94 + max-width: 65ch; /* measure: 45–75ch */
95 + text-wrap: pretty; /* fewer orphans where supported */
96 +}
97 +.prose h1, .prose h2 { text-wrap: balance; }
98 +table.data { font-variant-numeric: tabular-nums; }
99 +```
100 +
101 +## Pairing shortlists by genre
102 +
103 +Starting points — always re-justify against the brief (rule 1), never copy blindly:
104 +
105 +| Brief genre | Display | Body |
106 +|---|---|---|
107 +| Fintech / data product | IBM Plex Sans | IBM Plex Sans + Plex Mono (figures) |
108 +| Editorial / longform | Fraunces or Newsreader | Source Serif 4 |
109 +| Developer tool | Space Grotesk | Inter + JetBrains Mono (code) |
110 +| Fashion / portfolio | Canela-like high-contrast serif | Neue Haas–style grotesque |
111 +| Government / civic | Public Sans | Public Sans |
112 +| Playful consumer | Bricolage Grotesque | Nunito Sans |
113 +
114 +Licensing quick check: Google Fonts and Fontshare = free incl. commercial;
115 +foundry faces (Canela, Neue Haas) require a license — confirm before specifying.
116 +
117 +## Gotchas
118 +
119 +- **`font-display: swap` trades FOIT for layout shift** — pair it with metric-matched fallbacks or CLS suffers; `optional` is the strictest-CLS choice for non-brand fonts.
120 +- **Preloading every weight defeats the purpose** — each preload competes with LCP-critical resources; 2 files max.
121 +- **`vw`-only font sizes break pinch-zoom and text-only zoom** — clamp() with a rem term keeps zoom working.
122 +- **`ch` unit varies per font** — 65ch in the fallback ≠ 65ch in the web font; check measure after fonts load.
123 +- **Google Fonts CSS API adds a render-blocking third-party hop** — self-host the woff2 files instead.
124 +- **Faux bold/italic:** if a weight/style file isn't declared, browsers synthesize it (badly) — declare every weight you use, or use a variable font.
125 +- **Line-height with units inherits computed pixels** — always unitless (`1.6`, not `1.6em`).
added frontend-skills/crafting-ui-animations/SKILL.md +55 −0
@@ -0,0 +1,55 @@
1 +---
2 +name: crafting-ui-animations
3 +description: Designs and implements purposeful UI motion — micro-interactions, page-load and scroll-reveal sequences, transitions, and keyframe or Web Animations API animations that stay smooth and respect reduced-motion preferences. Use when the user asks to add an animation, transition, hover or press effect, loading indicator, scroll reveal, or page-load sequence, or to fix janky or excessive motion. Do not use for accessibility audits (ensuring-accessibility) or loading-performance work (optimizing-web-performance).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Crafting UI Animations
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** adding or refining motion — micro-interactions, enter/exit transitions, scroll reveals, load sequences, loading states — and fixing jank or motion overload.
15 +- **Do NOT use for:** WCAG audits (ensuring-accessibility), bundle/loading speed (optimizing-web-performance), or pure layout work.
16 +
17 +## Core rules
18 +
19 +1. **Motion serves purpose.** Each animation must communicate state, direct attention, or express identity — one orchestrated moment beats scattered effects. If you can't name its purpose, delete it.
20 +
21 +2. **Animate only `transform` and `opacity`** (compositor-friendly); never animate `width`, `height`, `top`, `left`, or `margin`.
22 + - ✅ `transform: translateY(8px) → none`
23 + - ❌ `top: 8px → 0` (forces layout every frame)
24 +
25 +3. **Never move layout after load** — reserve space; animate overlays and transforms so CLS stays ≤0.1.
26 + - ✅ banner slides over content, or space is pre-reserved
27 + - ❌ banner insertion pushes the page down
28 +
29 +4. **Duration/easing tokens:** UI feedback 150–300ms; enter `ease-out`, exit `ease-in`; larger scene changes ≤500ms. Define once, reuse everywhere.
30 +
31 +5. **Choose the lightest tool that works:** two states → CSS `transition`; multi-step/looping → CSS `@keyframes`; runtime-computed values, sequencing, or interruption → Web Animations API. No animation library for what these three cover.
32 +
33 +6. **Always honor `prefers-reduced-motion`** — gate every non-essential animation, swap movement for opacity or nothing.
34 +
35 +7. **Hierarchy of restraint:** micro-interactions everywhere are fine (≤200ms, subtle); attention-seeking motion (bounce, pulse) at most one element per view.
36 +
37 +## Workflow
38 +
39 +1. Name the purpose of each requested animation (state feedback / attention / identity). Cut anything purposeless.
40 +2. Define or reuse motion tokens (durations, easings) as CSS custom properties.
41 +3. Implement with the lightest tool per rule 5, animating only transform/opacity.
42 +4. Add the `prefers-reduced-motion` fallback for every animation you wrote.
43 +5. **Validate:** trigger each animation — no layout shift (DevTools → Performance → Layout Shift regions), steady 60fps (no long purple layout bars), interruption behaves (rapid hover on/off doesn't stutter).
44 +6. **Reduced-motion check:** enable "Emulate CSS prefers-reduced-motion" in DevTools Rendering panel and re-run the flow — nothing essential may be lost.
45 +
46 +## Edge cases & failure modes
47 +
48 +- **Animating `display: none` → visible:** `display` can't transition; use `@starting-style` + `transition-behavior: allow-discrete`, or WAAPI with a visibility swap.
49 +- **Scroll reveals below the fold:** use `IntersectionObserver`, reveal once, and ensure content is visible without JS (progressive enhancement — never leave `opacity: 0` as the no-JS state).
50 +- **Infinite/looping animation** (spinners excepted) → must pause when not visible (`animation-play-state`, or stop the WAAPI animation) and under reduced motion.
51 +- **Jank persists despite transform/opacity** → check for unintentionally huge paint areas; promote with `will-change: transform` sparingly and remove it after the animation.
52 +- **Autoplaying motion >5s** must have a pause control (WCAG 2.2.2) — flag this instead of shipping it silently.
53 +
54 +## References
55 +Copy-paste patterns (motion tokens, enter/exit, scroll reveal, WAAPI sequencing, reduced-motion): see [references/patterns.md](references/patterns.md)
added frontend-skills/crafting-ui-animations/references/patterns.md +157 −0
@@ -0,0 +1,157 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# UI Animation Patterns — Copy-Paste Reference
7 +
8 +## Contents
9 +- Motion tokens
10 +- Micro-interaction (button press/hover)
11 +- Enter/exit transition (dialog, toast)
12 +- Scroll reveal with IntersectionObserver
13 +- Page-load sequence (staggered)
14 +- Web Animations API sequencing
15 +- Reduced-motion gate
16 +- Gotchas
17 +
18 +## Motion tokens
19 +
20 +```css
21 +:root {
22 + --ease-out: cubic-bezier(0.16, 1, 0.3, 1); /* enter: fast start, soft landing */
23 + --ease-in: cubic-bezier(0.7, 0, 0.84, 0); /* exit: accelerate away */
24 + --dur-fast: 150ms; /* hover, press feedback */
25 + --dur-base: 250ms; /* enters, exits, toggles */
26 + --dur-slow: 450ms; /* scene-level changes */
27 +}
28 +```
29 +
30 +## Micro-interaction (button press/hover)
31 +
32 +```css
33 +.btn {
34 + transition: transform var(--dur-fast) var(--ease-out),
35 + box-shadow var(--dur-fast) var(--ease-out);
36 +}
37 +.btn:hover { transform: translateY(-1px); }
38 +.btn:active { transform: translateY(0) scale(0.98); }
39 +```
40 +
41 +## Enter/exit transition (dialog, toast)
42 +
43 +Enter with ease-out, exit faster with ease-in.
44 +
45 +```css
46 +.toast {
47 + transition: transform var(--dur-base) var(--ease-out),
48 + opacity var(--dur-base) var(--ease-out);
49 +}
50 +.toast[data-state="closed"] {
51 + transform: translateY(8px);
52 + opacity: 0;
53 + transition-duration: var(--dur-fast);
54 + transition-timing-function: var(--ease-in);
55 +}
56 +```
57 +
58 +Animating from `display: none` (modern CSS):
59 +
60 +```css
61 +dialog[open] {
62 + opacity: 1;
63 + transform: none;
64 + transition: opacity var(--dur-base) var(--ease-out),
65 + transform var(--dur-base) var(--ease-out),
66 + display var(--dur-base) allow-discrete;
67 + @starting-style { opacity: 0; transform: translateY(12px); }
68 +}
69 +```
70 +
71 +## Scroll reveal with IntersectionObserver
72 +
73 +Content is fully visible without JS; the class only *enables* the hidden start state.
74 +
75 +```html
76 +<section data-reveal>…</section>
77 +```
78 +
79 +```css
80 +.js [data-reveal] { opacity: 0; transform: translateY(16px); }
81 +.js [data-reveal].is-shown {
82 + opacity: 1; transform: none;
83 + transition: opacity var(--dur-slow) var(--ease-out),
84 + transform var(--dur-slow) var(--ease-out);
85 +}
86 +```
87 +
88 +```js
89 +document.documentElement.classList.add('js');
90 +const io = new IntersectionObserver((entries) => {
91 + for (const e of entries) if (e.isIntersecting) {
92 + e.target.classList.add('is-shown');
93 + io.unobserve(e.target); // reveal once
94 + }
95 +}, { threshold: 0.15 }); // fire when 15% visible
96 +document.querySelectorAll('[data-reveal]').forEach(el => io.observe(el));
97 +```
98 +
99 +## Page-load sequence (staggered)
100 +
101 +One orchestrated moment: hero elements enter in order, 60ms apart.
102 +
103 +```css
104 +.hero > * {
105 + opacity: 0;
106 + transform: translateY(12px);
107 + animation: enter var(--dur-slow) var(--ease-out) forwards;
108 + animation-delay: calc(var(--i) * 60ms);
109 +}
110 +@keyframes enter { to { opacity: 1; transform: none; } }
111 +```
112 +
113 +```html
114 +<div class="hero">
115 + <h1 style="--i:0">…</h1>
116 + <p style="--i:1">…</p>
117 + <a style="--i:2" class="btn">…</a>
118 +</div>
119 +```
120 +
121 +## Web Animations API sequencing
122 +
123 +For interruption-safe, runtime-computed motion.
124 +
125 +```js
126 +const panel = document.querySelector('.panel');
127 +const anim = panel.animate(
128 + [{ transform: 'translateX(100%)' }, { transform: 'none' }],
129 + { duration: 250, easing: 'cubic-bezier(0.16,1,0.3,1)', fill: 'forwards' }
130 +);
131 +// interruptible: reverse mid-flight instead of restarting
132 +closeBtn.onclick = () => anim.reverse();
133 +await anim.finished; // sequencing point
134 +```
135 +
136 +## Reduced-motion gate
137 +
138 +```css
139 +@media (prefers-reduced-motion: reduce) {
140 + [data-reveal], .hero > * { opacity: 1 !important; transform: none !important;
141 + animation: none !important; transition: none !important; }
142 +}
143 +```
144 +
145 +```js
146 +const reduced = matchMedia('(prefers-reduced-motion: reduce)').matches;
147 +if (!reduced) el.animate(/* … */);
148 +```
149 +
150 +## Gotchas
151 +
152 +- `transition: all` invites accidental layout animations and future regressions — always list properties.
153 +- `animation-fill-mode: forwards` keeps the element on its final keyframe, which can trap it in a stale state after class changes; prefer transitioning real end states.
154 +- `will-change` left permanently on many elements consumes GPU memory — add before heavy animation, remove after.
155 +- Staggers multiply duration: 10 items × 100ms delay = a 1s+ sequence. Cap total sequence time near 700ms.
156 +- `height: auto` can't transition; animate `grid-template-rows: 0fr → 1fr` on a wrapper, or use WAAPI with measured pixel values.
157 +- IntersectionObserver with `threshold: 1.0` never fires on elements taller than the viewport.
added frontend-skills/creating-landing-pages/SKILL.md +58 −0
@@ -0,0 +1,58 @@
1 +---
2 +name: creating-landing-pages
3 +description: Composes landing pages that convert — hero thesis, section flow, one primary call to action, disciplined visual boldness, and benefit-first copy. Use when the user asks to create, design, or rewrite a landing page, homepage, product page, marketing page, hero section, or signup/waitlist page, or asks why a page is not converting. Do not use for CSS layout mechanics (designing-responsive-layouts), design-token systems (theming-design-tokens), or user-facing release notes (writing-release-notes).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Creating Landing Pages
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** landing pages, homepages, product/marketing pages, hero sections, waitlist and signup pages — structure, visual direction, and copy.
15 +- **Do NOT use for:** grid/flexbox/breakpoint mechanics (designing-responsive-layouts), token/theming systems (theming-design-tokens), full app UI, or documentation pages.
16 +
17 +## Core rules
18 +
19 +1. **Open with a hero thesis, not a template.** The first screen shows the page's single most characteristic element — a headline, product image, demo, or interactive moment chosen for THIS subject.
20 + - ✅ a live 10-second product demo as the hero for a screen-recording tool
21 + - ❌ generic gradient background + stat trio + stock illustration
22 +
23 +2. **One primary CTA per page, repeated in rhythm.** Same verb, same destination, after the hero and after each proof section; secondary actions are visually quiet links.
24 + - ✅ "Start recording — free" (hero, mid-page, footer)
25 + - ❌ "Sign up", "Book a demo", "Learn more", and "Contact us" all competing above the fold
26 +
27 +3. **Section flow: problem → proof → product → action.** Name the visitor's pain, prove you solve it (numbers, logos, testimonials), show the product doing it, then ask once more.
28 +
29 +4. **Spend boldness in ONE signature element.** One memorable move — an unusual type treatment, a signature interaction, a striking image — and keep everything around it quiet and disciplined.
30 + - ❌ animated gradient + parallax + marquee + glassmorphism on one page
31 +
32 +5. **Avoid the AI-default palettes.** Cream background + serif display + terracotta accents, and near-black + acid-green/vermilion, read as templated. Derive the palette from the product's actual subject and brand.
33 +
34 +6. **Benefit-first, specific copy in active voice.** Headlines state what the visitor gets, with concrete nouns and numbers; button labels are specific verbs.
35 + - ✅ "Ship your changelog in 5 minutes" / button: "Create my changelog"
36 + - ❌ "Empowering seamless productivity solutions" / button: "Submit"
37 +
38 +7. **Social proof is placed, not piled.** Logos directly under the hero; one strong testimonial (with name, face, role) per proof section; numbers only when true and impressive.
39 +
40 +8. **Every section earns scroll.** If a section restates the previous one or exists "because landing pages have one" (generic feature grid, filler FAQ), cut it.
41 +
42 +## Workflow
43 +
44 +1. Write the one-sentence thesis: who the page is for and the single action they should take.
45 +2. Draft the section outline per rule 3; assign each section one job and one message.
46 +3. Define the visual direction: palette (4–6 named colors, rule 5), display + body typefaces, and the ONE signature element (rule 4).
47 +4. Write copy headline-first (rule 6); place CTAs and proof (rules 2, 7).
48 +5. Build the page (hand off layout mechanics to designing-responsive-layouts if needed).
49 +6. Self-critique pass: check the hero against rule 1, count CTAs (rule 2), hunt template smells (rules 4–5), cut dead sections (rule 8). Revise anything that reads templated rather than subject-specific.
50 +
51 +## Edge cases & failure modes
52 +- **No brand or palette exists** → derive 4–6 colors from the product's domain (a finance tool and a kids' app must not share a palette); never fall back to the defaults in rule 5.
53 +- **Multiple audiences with different CTAs** → split into separate landing pages; one page, one primary action.
54 +- **No social proof yet (pre-launch)** → substitute concrete specificity: exact feature claims, a real demo, founder's note; never fabricate logos, counts, or testimonials.
55 +- **Client insists on many CTAs** → keep one primary visually dominant; demote the rest to text links, and say why.
56 +
57 +## References
58 +Section templates, copy formulas, and gotchas: see [references/patterns.md](references/patterns.md)
added frontend-skills/creating-landing-pages/references/patterns.md +119 −0
@@ -0,0 +1,119 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Creating Landing Pages
7 +
8 +## Contents
9 +- Page skeleton (problem → proof → product → action)
10 +- Hero patterns by product type
11 +- Copy formulas
12 +- CTA hierarchy markup
13 +- Social proof blocks
14 +- Visual direction worksheet
15 +- Gotchas
16 +
17 +## Page skeleton (problem → proof → product → action)
18 +
19 +```html
20 +<main>
21 + <section class="hero"> <!-- thesis + primary CTA -->
22 + <section class="logos"> <!-- proof: who already trusts it -->
23 + <section class="problem"> <!-- name the pain in the visitor's words -->
24 + <section class="product"> <!-- show it solving the pain (demo/screens) -->
25 + <section class="proof"> <!-- testimonial + numbers, CTA repeat -->
26 + <section class="objections"> <!-- pricing/security/FAQ — only real questions -->
27 + <section class="closer"> <!-- restate thesis, final CTA -->
28 +</main>
29 +```
30 +
31 +Each section: one job, one message, max one CTA.
32 +
33 +## Hero patterns by product type
34 +
35 +| Product | Hero thesis element |
36 +|---|---|
37 +| Dev tool / API | Working code snippet or terminal session, real output |
38 +| Visual product (design, video) | The product's own output, full-bleed |
39 +| SaaS dashboard | One annotated screenshot of the "aha" screen — not a collage |
40 +| AI product | Live or looped input→output demo, honest speed |
41 +| Service / agency | The single best result with the number that proves it |
42 +
43 +Headline sits with the element, max ~10 words, states the benefit.
44 +
45 +## Copy formulas
46 +
47 +Headline (pick one, fill with concrete nouns):
48 +- Outcome + time: "Ship your changelog in 5 minutes"
49 +- Kill the pain: "Never write release notes by hand again"
50 +- Capability + audience: "Session replay built for mobile teams"
51 +
52 +Subheadline: one sentence — how it works + for whom.
53 +Button labels: verb + object, first person works well — "Create my changelog",
54 +"Start recording — free". Never "Submit", "Learn more" (as primary), "Get started" (default-smell).
55 +
56 +Testimonial pull: quote the *result*, not the compliment —
57 +✅ "Cut our onboarding time from 3 weeks to 4 days." — Dana K., Head of Ops
58 +❌ "Great tool, love the team!"
59 +
60 +## CTA hierarchy markup
61 +
62 +```html
63 +<div class="cta-group">
64 + <a class="cta-primary" href="/signup">Start recording — free</a>
65 + <a class="cta-secondary" href="/demo">Watch the 2-min demo</a>
66 +</div>
67 +```
68 +
69 +```css
70 +.cta-primary { /* filled, brand color, largest tap target on screen */ }
71 +.cta-secondary { /* text link or ghost — visibly subordinate */ }
72 +```
73 +
74 +Same `.cta-primary` label + destination everywhere it repeats.
75 +
76 +## Social proof blocks
77 +
78 +```html
79 +<!-- Logos: 4–6, grayscale, one row, directly under hero -->
80 +<section class="logos" aria-label="Trusted by">
81 + <img src="logo-a.svg" alt="Acme" height="28">
82 + …
83 +</section>
84 +
85 +<!-- One strong testimonial beats six weak ones -->
86 +<figure class="testimonial">
87 + <blockquote>Cut our onboarding time from 3 weeks to 4 days.</blockquote>
88 + <figcaption><img src="dana.jpg" alt="" width="40" height="40">
89 + Dana K. — Head of Ops, Acme</figcaption>
90 +</figure>
91 +```
92 +
93 +Numbers block: only true, only impressive, max three. "12,400 teams" beats "many customers"; if the true number is small, use a different proof type.
94 +
95 +## Visual direction worksheet
96 +
97 +Before building, write down (one line each):
98 +1. Palette: 4–6 named hex colors derived from the subject (not cream/terracotta, not black/acid-green).
99 +2. Type: display face + body face, chosen for this brief; type scale (e.g. 1.25 ratio).
100 +3. Layout concept in one sentence + rough ASCII wireframe of the hero.
101 +4. THE signature element — the one bold move. Everything else stays quiet.
102 +
103 +Then critique: "Would this exact plan fit any other product?" If yes, revise until it wouldn't.
104 +
105 +## Gotchas
106 +
107 +- **The feature-grid reflex**: a 3×2 grid of icon+blurb usually restates the
108 + product section with less proof — cut or merge it.
109 +- **Carousel/slider testimonials** hide proof behind interaction — static, one
110 + per section.
111 +- **Fake urgency** (countdowns, "3 spots left") destroys trust with technical
112 + audiences instantly.
113 +- **Above-the-fold obsession**: long pages convert when each section earns the
114 + scroll; don't cram everything into the first screen.
115 +- **Lorem ipsum in review**: never review layout with placeholder copy — copy
116 + IS the design decision on a landing page.
117 +- **Hero video autoplay with sound** — muted, looped, short, with poster; or
118 + don't autoplay.
119 +- **Dark pattern CTAs** ("No thanks, I hate saving money") — never.
added frontend-skills/designing-forms/SKILL.md +57 −0
@@ -0,0 +1,57 @@
1 +---
2 +name: designing-forms
3 +description: Designs and builds web forms with correct labels, input types, autocomplete attributes, inline validation timing, error messaging, and layout — including multi-step forms and submit states. Use when the user asks to create or improve a form, sign-up or checkout flow, add form validation or error messages, fix form UX, or build a multi-step wizard. Do not use for backend validation logic or for general accessibility audits (ensuring-accessibility).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Designing Forms
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** creating or improving forms — field markup, labels, validation timing, error messages, layout, submit states, multi-step flows.
15 +- **Do NOT use for:** server-side validation rules or business logic, and full WCAG audits (ensuring-accessibility) — though forms built here must already follow its basics.
16 +
17 +## Core rules
18 +
19 +1. **Every field has a visible `<label>`; placeholder never replaces it.** Placeholders are optional format hints only.
20 + - ✅ `<label for="phone">Phone</label><input id="phone" placeholder="514-555-0100">`
21 + - ❌ `<input placeholder="Phone">`
22 +
23 +2. **Use the right `type` and `autocomplete`** so mobile keyboards and autofill work: `type="email" autocomplete="email"`, `type="tel" autocomplete="tel"`, `autocomplete="given-name"`, `"postal-code"`, `"cc-number"`, etc.
24 +
25 +3. **Validation timing: validate on blur; after a field first errors, re-validate on every input.** Never validate on every keystroke of an untouched field, never only on submit.
26 +
27 +4. **Error messages are specific and adjacent:** placed next to the field, saying what's wrong AND how to fix it, wired with `aria-describedby` + `aria-invalid="true"`.
28 + - ✅ "Enter an email address with an @, like name@example.com."
29 + - ❌ "Invalid input."
30 +
31 +5. **Single-column layout.** Related short fields (city/postal code) may share a row; everything else stacks. Group related fields with `<fieldset><legend>`.
32 +
33 +6. **Ask for the minimum.** Every field must justify itself; mark the exception ("optional") rather than decorating everything with asterisks when most fields are required.
34 +
35 +7. **Submit button states:** descriptive label ("Create account", not "Submit"); disable only *during* submission with a pending indicator; on failure re-enable and show a summarized error (`role="alert"`) that links to the first invalid field.
36 +
37 +8. **Multi-step forms:** one topic per step, visible progress ("Step 2 of 4"), Back never loses data, validate per-step, final review step before irreversible submission.
38 +
39 +## Workflow
40 +
41 +1. List the data actually needed; cut or mark-optional everything else (rule 6).
42 +2. Write the markup: label + correct type/autocomplete per field, fieldsets for groups, single-column layout.
43 +3. Implement validation with rule-3 timing and rule-4 messages (constraint attributes first — `required`, `minlength`, `pattern` — JS only on top).
44 +4. Wire submit states (rule 7); for multi-step flows apply rule 8.
45 +5. **Error-state walkthrough:** submit empty, then fix fields one by one — each error appears next to its field, is announced (aria wiring), disappears on fix, and focus lands on the first invalid field after a failed submit.
46 +6. **Autofill test:** trigger browser autofill; every field must fill correctly (wrong fills = wrong `autocomplete` values).
47 +
48 +## Edge cases & failure modes
49 +
50 +- **Password fields:** allow paste, provide show/hide toggle, state the rules up front (not only as errors), `autocomplete="new-password"` vs `"current-password"`.
51 +- **Server-side failure after client-side pass** → show a `role="alert"` summary at the top with per-field errors re-injected; never lose the user's input.
52 +- **Select with >10 options** → use a searchable combobox or grouped options; >2–4 radio options → use a select.
53 +- **Date inputs:** `type="date"` unless the design demands a custom picker; always allow keyboard typing.
54 +- **Names, addresses, phone numbers vary globally** → no restrictive patterns (e.g., don't reject accents or 5+ digit postal codes) unless the business rule is explicit.
55 +
56 +## References
57 +Copy-paste patterns (full field markup, validation JS, error summary, multi-step skeleton): see [references/patterns.md](references/patterns.md)
added frontend-skills/designing-forms/references/patterns.md +147 −0
@@ -0,0 +1,147 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Form Patterns — Copy-Paste Reference
7 +
8 +## Contents
9 +- Complete field markup
10 +- Common autocomplete values
11 +- Validation timing (blur, then input)
12 +- Error summary after failed submit
13 +- Submit button pending state
14 +- Multi-step form skeleton
15 +- Gotchas
16 +
17 +## Complete field markup
18 +
19 +```html
20 +<div class="field">
21 + <label for="email">Email</label>
22 + <input id="email" name="email" type="email" required
23 + autocomplete="email" placeholder="name@example.com"
24 + aria-describedby="email-hint email-error">
25 + <p id="email-hint" class="hint">We only use this for receipts.</p>
26 + <p id="email-error" class="error" hidden></p>
27 +</div>
28 +```
29 +
30 +The error node exists from the start (hidden) so `aria-describedby` stays stable.
31 +
32 +## Common autocomplete values
33 +
34 +| Field | type | autocomplete |
35 +|---|---|---|
36 +| Email | email | email |
37 +| Phone | tel | tel |
38 +| First / last name | text | given-name / family-name |
39 +| Street / city / postal | text | address-line1 / address-level2 / postal-code |
40 +| Country | (select) | country-name |
41 +| Card number / expiry / CVC | text | cc-number / cc-exp / cc-csc |
42 +| New / current password | password | new-password / current-password |
43 +| One-time code | text | one-time-code |
44 +
45 +## Validation timing (blur, then input)
46 +
47 +```js
48 +const messages = {
49 + valueMissing: (label) => `Enter your ${label.toLowerCase()}.`,
50 + typeMismatch: () => 'Enter an email address with an @, like name@example.com.',
51 + tooShort: (label, input) => `${label} must be at least ${input.minLength} characters.`,
52 +};
53 +
54 +function validate(input) {
55 + const label = input.labels[0].textContent;
56 + const errEl = document.getElementById(input.getAttribute('aria-describedby')
57 + .split(' ').find(id => id.endsWith('-error')));
58 + let msg = '';
59 + for (const key of Object.keys(messages)) {
60 + if (input.validity[key]) { msg = messages[key](label, input); break; }
61 + }
62 + input.setAttribute('aria-invalid', msg ? 'true' : 'false');
63 + errEl.textContent = msg;
64 + errEl.hidden = !msg;
65 + return !msg;
66 +}
67 +
68 +document.querySelectorAll('input, select, textarea').forEach((input) => {
69 + input.addEventListener('blur', () => validate(input), { once: false });
70 + input.addEventListener('input', () => {
71 + // re-validate live only after the field has already errored
72 + if (input.getAttribute('aria-invalid') === 'true') validate(input);
73 + });
74 +});
75 +```
76 +
77 +Pair with `<form novalidate>` so these messages replace the browser bubbles.
78 +
79 +## Error summary after failed submit
80 +
81 +```js
82 +form.addEventListener('submit', (e) => {
83 + const fields = [...form.querySelectorAll('input, select, textarea')];
84 + const invalid = fields.filter((f) => !validate(f));
85 + if (invalid.length) {
86 + e.preventDefault();
87 + const summary = document.getElementById('form-alert');
88 + summary.innerHTML = `<p>Fix ${invalid.length} error(s):</p><ul>` +
89 + invalid.map((f) => `<li><a href="#${f.id}">${f.labels[0].textContent}</a></li>`).join('') +
90 + '</ul>';
91 + invalid[0].focus();
92 + }
93 +});
94 +```
95 +
96 +```html
97 +<div id="form-alert" role="alert" tabindex="-1"></div>
98 +```
99 +
100 +## Submit button pending state
101 +
102 +```js
103 +form.addEventListener('submit', async (e) => {
104 + e.preventDefault();
105 + const btn = form.querySelector('button[type="submit"]');
106 + btn.disabled = true;
107 + btn.dataset.label = btn.textContent;
108 + btn.textContent = 'Creating account…';
109 + try {
110 + await submit(new FormData(form));
111 + } catch (err) {
112 + showServerErrors(err); // role="alert" summary; inputs keep their values
113 + } finally {
114 + btn.disabled = false;
115 + btn.textContent = btn.dataset.label;
116 + }
117 +});
118 +```
119 +
120 +Disable only during the request — a permanently disabled submit hides *why* the form can't be sent.
121 +
122 +## Multi-step form skeleton
123 +
124 +```html
125 +<form id="wizard">
126 + <p class="progress" aria-live="polite">Step <span id="step-n">1</span> of 3</p>
127 + <fieldset data-step="1"><legend>Your details</legend>…</fieldset>
128 + <fieldset data-step="2" hidden><legend>Shipping</legend>…</fieldset>
129 + <fieldset data-step="3" hidden><legend>Review &amp; confirm</legend>…</fieldset>
130 + <button type="button" id="back" hidden>Back</button>
131 + <button type="button" id="next">Continue</button>
132 + <button type="submit" id="finish" hidden>Place order</button>
133 +</form>
134 +```
135 +
136 +Rules encoded above: steps are fieldsets shown one at a time (data survives because
137 +nothing unmounts), per-step validation runs on "Continue", the last step is a
138 +read-only review, and the progress line is announced on change.
139 +
140 +## Gotchas
141 +
142 +- `<form novalidate>` disables browser bubbles but NOT the `validity` API — exactly what you want for custom messages.
143 +- Disabled inputs are skipped on submit AND invisible to autofill/screen readers in some browsers — prefer `readonly` for locked-but-submitted values.
144 +- `pattern` is anchored (implicit `^…$`) and silently fails on typos in the regex; always pair with a `title`/error message that states the format.
145 +- iOS zooms into inputs with font-size <16px — keep inputs ≥16px.
146 +- Numeric codes (postal, OTP): use `inputmode="numeric" pattern="[0-9]*"`, NOT `type="number"` (which strips leading zeros and adds spinners).
147 +- Autofocus on page load disorients screen-reader users — reserve `autofocus` for single-purpose pages (search, login).
added frontend-skills/designing-responsive-layouts/SKILL.md +59 −0
@@ -0,0 +1,59 @@
1 +---
2 +name: designing-responsive-layouts
3 +description: Designs responsive CSS layouts — mobile-first breakpoints, flexbox vs grid decisions, container queries, fluid sizing with clamp(), and intrinsic auto-fit/minmax patterns. Use when the user asks to make a page or component responsive, build a layout or grid, fix overflow or squished content on mobile, choose between flexbox and grid, or add breakpoints or container queries. Do not use for page content composition and conversion structure (creating-landing-pages) or for color/spacing token systems (theming-design-tokens).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Designing Responsive Layouts
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** layout mechanics — grids, columns, wrapping, breakpoints, container queries, fluid sizing, overflow fixes.
15 +- **Do NOT use for:** what content goes where on a marketing page (→ creating-landing-pages); token scales and theming (→ theming-design-tokens); typography scales (→ choosing-typography).
16 +
17 +## Core rules
18 +
19 +1. **Mobile-first: base styles are the narrow layout; media queries only add width.**
20 + - ✅ `.cards { display: grid; } @media (min-width: 48rem) { .cards { grid-template-columns: 1fr 1fr; } }`
21 + - ❌ Desktop styles first, then `@media (max-width: …)` overrides undoing them
22 +
23 +2. **Flexbox vs grid decision rule: one dimension → flexbox; two dimensions or explicit placement → grid.** Nav bars, button rows, media objects = flex. Card grids, page shells, dashboards = grid.
24 +
25 +3. **Prefer intrinsic (no-breakpoint) patterns before adding media queries.**
26 + - ✅ `grid-template-columns: repeat(auto-fit, minmax(min(16rem, 100%), 1fr));`
27 + - ❌ Three hand-written breakpoints to go 1→2→3 columns
28 +
29 +4. **Breakpoints in `rem`, chosen where the content breaks — not at device names.** Defaults when nothing else is known: 48rem (768px) and 80rem (1280px); always verify at 360px, 768px, 1280px.
30 +
31 +5. **Components respond to their container, not the viewport.**
32 + - ✅ `.sidebar { container-type: inline-size; } @container (min-width: 24rem) { .card { flex-direction: row; } }`
33 + - ❌ A viewport media query that breaks the card when it's placed in a narrow sidebar
34 +
35 +6. **Fluid values with `clamp()`, not stepped jumps:** `padding: clamp(1rem, 3vw, 2.5rem);` for space that scales; reserve breakpoints for structural change.
36 +
37 +7. **Never fix heights on text containers; let content size the box.**
38 + - ✅ `min-height: 20rem;` (or nothing)
39 + - ❌ `height: 20rem; overflow: hidden;` — clips translated/user content and causes CLS
40 +
41 +8. **Kill accidental horizontal scroll at the source:** media `max-width: 100%; height: auto;`, `min-width: 0` on flex/grid children that must shrink, `overflow-wrap: break-word` on long strings.
42 +
43 +## Workflow
44 +
45 +1. Identify the layout's dimensionality per rule 2 and pick flex or grid.
46 +2. Build the narrow (360px) layout first with intrinsic patterns (rule 3).
47 +3. Add `@container` or `@media (min-width)` steps only where the content visibly breaks.
48 +4. Replace remaining fixed values with `clamp()`/`minmax()` where they should flex.
49 +5. Validate: test at 360px, 768px, 1280px, and one in-between width; check no horizontal scrollbar, no clipped text, images scale, and layout shift stays visually stable (CLS ≤ 0.1 target).
50 +
51 +## Edge cases & failure modes
52 +- **Flex children overflowing:** flex items default to `min-width: auto` — set `min-width: 0` on the shrinking child.
53 +- **`auto-fit` collapsing with one item stretched full width:** use `auto-fill` when empty tracks should be preserved.
54 +- **Container queries need a named/typed container:** without `container-type: inline-size` on an ancestor, `@container` silently never matches.
55 +- **`100vw` causes a scrollbar-width overflow on Windows:** use `100%` or `100dvw`.
56 +- **Legacy browser support required:** container queries and `dvh/dvw` need a media-query fallback — state the assumption before using them.
57 +
58 +## References
59 +Copy-paste layout recipes (shells, card grids, sidebars, holy grail, media objects): see [references/patterns.md](references/patterns.md)
added frontend-skills/designing-responsive-layouts/references/patterns.md +116 −0
@@ -0,0 +1,116 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Responsive Layouts
7 +
8 +## Contents
9 +- Auto-responsive card grid (no breakpoints)
10 +- Page shell (header/content/footer)
11 +- Sidebar + content (collapses on narrow)
12 +- Container-query card
13 +- Media object
14 +- Fluid space and size
15 +- Overflow-proofing checklist
16 +- Gotchas
17 +
18 +## Auto-responsive card grid (no breakpoints)
19 +
20 +```css
21 +.cards {
22 + display: grid;
23 + /* min(16rem, 100%) prevents overflow when the container is narrower than 16rem */
24 + grid-template-columns: repeat(auto-fit, minmax(min(16rem, 100%), 1fr));
25 + gap: 1rem;
26 +}
27 +```
28 +
29 +## Page shell (header/content/footer, footer pinned)
30 +
31 +```css
32 +body {
33 + min-height: 100dvh; /* dvh avoids mobile URL-bar jump; fallback: 100vh */
34 + display: grid;
35 + grid-template-rows: auto 1fr auto;
36 +}
37 +```
38 +
39 +## Sidebar + content (collapses on narrow)
40 +
41 +```css
42 +/* Flexbox version — wraps naturally, no media query */
43 +.with-sidebar {
44 + display: flex;
45 + flex-wrap: wrap;
46 + gap: 1.5rem;
47 +}
48 +.with-sidebar > .sidebar { flex: 1 1 15rem; } /* basis = collapse point */
49 +.with-sidebar > .content { flex: 999 1 60%; min-width: 0; }
50 +```
51 +
52 +```css
53 +/* Grid version — explicit breakpoint */
54 +.shell { display: grid; gap: 1.5rem; }
55 +@media (min-width: 48rem) {
56 + .shell { grid-template-columns: 15rem minmax(0, 1fr); }
57 +}
58 +```
59 +
60 +## Container-query card
61 +
62 +```css
63 +.card-slot { container-type: inline-size; }
64 +
65 +.card { display: flex; flex-direction: column; gap: 0.75rem; }
66 +
67 +@container (min-width: 24rem) {
68 + .card { flex-direction: row; align-items: center; }
69 + .card img { max-width: 40%; }
70 +}
71 +```
72 +
73 +## Media object (avatar + text)
74 +
75 +```css
76 +.media { display: flex; gap: 0.75rem; align-items: flex-start; }
77 +.media > img { flex-shrink: 0; width: 3rem; height: 3rem; border-radius: 50%; }
78 +.media > .body { min-width: 0; } /* lets long words/URLs shrink instead of overflow */
79 +```
80 +
81 +## Fluid space and size
82 +
83 +```css
84 +:root {
85 + /* 1rem at 320px viewport → 2.5rem at 1280px, linear between */
86 + --space-section: clamp(1rem, 0.5rem + 2.5vw, 2.5rem);
87 + --content-max: 72rem;
88 +}
89 +.section { padding-block: var(--space-section); }
90 +.container {
91 + width: min(100% - 2rem, var(--content-max));
92 + margin-inline: auto;
93 +}
94 +```
95 +
96 +## Overflow-proofing checklist
97 +
98 +```css
99 +img, video, canvas, svg { max-width: 100%; height: auto; }
100 +.flex-child, .grid-child { min-width: 0; } /* on anything that must shrink */
101 +.prose { overflow-wrap: break-word; } /* long URLs, tokens */
102 +table { display: block; overflow-x: auto; } /* wide data tables scroll alone */
103 +```
104 +
105 +Test sequence: 360px → 768px → 1280px → one odd width (e.g. 913px). Look for:
106 +horizontal scrollbar, clipped text, stretched images, wrap orphans.
107 +
108 +## Gotchas
109 +
110 +- **`minmax(16rem, 1fr)` overflows below 16rem** — always wrap the minimum in `min(16rem, 100%)`.
111 +- **`auto-fit` vs `auto-fill`:** auto-fit collapses empty tracks (items stretch); auto-fill keeps them (items stay small). One word, opposite layouts.
112 +- **Flex `min-width: auto` default** means text children refuse to shrink — the single most common "why is this overflowing" cause.
113 +- **Percentage heights need a sized ancestor**; prefer grid rows (`1fr`) or `min-height` chains.
114 +- **`gap` on flexbox** is fine everywhere modern; if supporting old Safari (<14.1), fall back to margins.
115 +- **Nested containers:** `@container` resolves against the *nearest* ancestor with `container-type` — name containers (`container-name: card-slot`) when nesting.
116 +- **`100dvh` vs `100vh`:** dvh tracks the dynamic mobile toolbar; vh can leave a gap or cause scroll. Provide `100vh` first as fallback on the previous line.
added frontend-skills/ensuring-accessibility/SKILL.md +61 −0
@@ -0,0 +1,61 @@
1 +---
2 +name: ensuring-accessibility
3 +description: Audits and fixes web UI for WCAG 2.2 AA compliance — keyboard navigation, visible focus, ARIA usage, alt text, form labels, color contrast, target sizes, and reduced motion. Use when the user asks to make a page or component accessible, run an accessibility or a11y audit, fix WCAG violations, add ARIA or alt text, improve keyboard or screen-reader support, or check color contrast. Do not use for general semantic markup structure (structuring-semantic-html) or visual design choices.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Ensuring Accessibility
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** auditing or fixing UI against WCAG 2.2 AA — keyboard access, focus, ARIA, alt text, labels, contrast, target sizes, motion preferences.
15 +- **Do NOT use for:** choosing heading/landmark structure for its own sake (structuring-semantic-html), visual design or theming decisions, or backend logic.
16 +
17 +## Core rules
18 +
19 +1. **Semantic HTML first, ARIA last.** ARIA only when no native element can express the role (first rule of ARIA).
20 + - ✅ `<button onclick="…">Save</button>`
21 + - ❌ `<div role="button" tabindex="0" onclick="…">Save</div>`
22 +
23 +2. **Everything interactive works by keyboard alone,** in a logical Tab order, no traps, no drag-only interactions (WCAG 2.5.7).
24 + - ✅ sortable list also offers "Move up/Move down" buttons
25 + - ❌ reorder only via drag-and-drop
26 +
27 +3. **Focus must be visible** with ≥3:1 contrast against adjacent colors; never remove it without a replacement.
28 + - ✅ `:focus-visible { outline: 2px solid var(--focus); outline-offset: 2px; }`
29 + - ❌ `:focus { outline: none; }`
30 +
31 +4. **Contrast minimums:** 4.5:1 for normal text, 3:1 for large text (≥24px or ≥18.7px bold) and for UI components/graphics.
32 +
33 +5. **Targets ≥24×24 CSS px** (WCAG 2.2), 44×44px for primary touch targets.
34 +
35 +6. **Every image has an `alt`:** descriptive for informative images, `alt=""` for decorative ones — never omit the attribute.
36 + - ✅ `<img src="chart.png" alt="Revenue grew 40% from Q1 to Q4">`
37 + - ❌ `<img src="chart.png" alt="chart">`
38 +
39 +7. **Every form control has a programmatically associated label** (`<label for>` or `aria-labelledby`); errors linked via `aria-describedby` + `aria-invalid="true"`.
40 +
41 +8. **Honor `prefers-reduced-motion`:** disable non-essential animation, parallax, and autoplay when set.
42 +
43 +## Workflow
44 +
45 +1. Inventory interactive elements and images on the page/component in scope.
46 +2. Apply rules 1–8, fixing violations directly in the markup/CSS (smallest diff that fixes the violation).
47 +3. Run automated checks if available (`npx axe-cli <url>` or Lighthouse accessibility category); fix every reported violation.
48 +4. **Keyboard-only walkthrough:** Tab through the whole flow — every control reachable, operable (Enter/Space/arrows), focus always visible, no traps.
49 +5. **Screen-reader pass** (VoiceOver: Cmd+F5 on macOS): headings/landmarks announce sensibly, images and controls have accessible names, errors are announced.
50 +6. Report remaining issues you cannot fix in code (e.g., brand color fails contrast) with the exact measured ratio and a compliant alternative.
51 +
52 +## Edge cases & failure modes
53 +
54 +- **Brand color fails contrast** → do not silently change the brand; report the ratio (e.g., "3.2:1, needs 4.5:1") and propose the nearest compliant shade.
55 +- **Third-party widget is inaccessible** → wrap with an accessible trigger where possible; otherwise flag it as a blocker, don't fake ARIA on top.
56 +- **Icon-only buttons** → require `aria-label`; a tooltip alone is not an accessible name.
57 +- **Dynamic content updates** (toasts, async results) → announce with `aria-live="polite"` (or `role="alert"` for errors only).
58 +- **axe/Lighthouse unavailable** → say so and rely on the manual walkthroughs; never claim "audit passed" on rules 1–8 alone.
59 +
60 +## References
61 +Copy-paste patterns (skip links, focus styles, live regions, accessible modals, contrast tokens): see [references/patterns.md](references/patterns.md)
added frontend-skills/ensuring-accessibility/references/patterns.md +141 −0
@@ -0,0 +1,141 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Accessibility Patterns — Copy-Paste Reference
7 +
8 +## Contents
9 +- Skip link
10 +- Focus styles
11 +- Accessible form field with error
12 +- Icon-only button
13 +- Live regions (async updates)
14 +- Accessible modal dialog
15 +- Reduced motion
16 +- Contrast tokens
17 +- Gotchas
18 +
19 +## Skip link
20 +
21 +First focusable element on the page; visible only on focus.
22 +
23 +```html
24 +<a class="skip-link" href="#main">Skip to main content</a>
25 +<main id="main" tabindex="-1">…</main>
26 +```
27 +
28 +```css
29 +.skip-link {
30 + position: absolute;
31 + left: -9999px;
32 +}
33 +.skip-link:focus {
34 + left: 8px;
35 + top: 8px;
36 + z-index: 100;
37 +}
38 +```
39 +
40 +## Focus styles
41 +
42 +`:focus-visible` shows the ring for keyboard users without flashing it on every mouse click.
43 +
44 +```css
45 +:focus-visible {
46 + outline: 2px solid var(--color-focus, #1a56db);
47 + outline-offset: 2px;
48 +}
49 +/* never do: :focus { outline: none; } without a replacement */
50 +```
51 +
52 +## Accessible form field with error
53 +
54 +```html
55 +<label for="email">Email address</label>
56 +<input id="email" name="email" type="email" autocomplete="email"
57 + aria-describedby="email-error" aria-invalid="true">
58 +<p id="email-error" class="error">Enter an email address with an @, like name@example.com.</p>
59 +```
60 +
61 +Remove `aria-invalid` and the error node (or empty it) once the field validates.
62 +
63 +## Icon-only button
64 +
65 +```html
66 +<button type="button" aria-label="Close dialog">
67 + <svg aria-hidden="true" focusable="false">…</svg>
68 +</button>
69 +```
70 +
71 +`aria-hidden` on the SVG keeps screen readers from announcing the icon twice.
72 +
73 +## Live regions (async updates)
74 +
75 +```html
76 +<!-- polite: status updates, search-result counts -->
77 +<div aria-live="polite" class="visually-hidden" id="status"></div>
78 +
79 +<!-- assertive, errors only -->
80 +<div role="alert" id="form-alert"></div>
81 +```
82 +
83 +```js
84 +document.getElementById('status').textContent = '12 results found';
85 +```
86 +
87 +The live region must exist in the DOM before you write into it — injecting a new `aria-live` node with text is not announced reliably.
88 +
89 +## Accessible modal dialog
90 +
91 +Native `<dialog>` gives focus trapping and Esc for free — prefer it.
92 +
93 +```html
94 +<dialog id="confirm" aria-labelledby="confirm-title">
95 + <h2 id="confirm-title">Delete file?</h2>
96 + <button type="button" id="cancel">Cancel</button>
97 + <button type="button" id="ok">Delete</button>
98 +</dialog>
99 +```
100 +
101 +```js
102 +const dlg = document.getElementById('confirm');
103 +dlg.showModal(); // traps focus, Esc closes
104 +dlg.addEventListener('close', () => opener.focus()); // return focus to the trigger
105 +```
106 +
107 +## Reduced motion
108 +
109 +```css
110 +@media (prefers-reduced-motion: reduce) {
111 + *, *::before, *::after {
112 + animation-duration: 0.01ms !important;
113 + animation-iteration-count: 1 !important;
114 + transition-duration: 0.01ms !important;
115 + scroll-behavior: auto !important;
116 + }
117 +}
118 +```
119 +
120 +## Contrast tokens
121 +
122 +Bake compliance into tokens so components can't ship a failing pair.
123 +
124 +```css
125 +:root {
126 + --text-on-light: #1f2937; /* 14.7:1 on #ffffff */
127 + --text-muted: #4b5563; /* 7.6:1 on #ffffff — still AA for body text */
128 + --color-focus: #1a56db; /* 3.6:1 vs #ffffff — passes 3:1 UI minimum */
129 +}
130 +```
131 +
132 +Check ratios with a tool (e.g., `npx wcag-contrast 4b5563 ffffff`) rather than by eye.
133 +
134 +## Gotchas
135 +
136 +- `display: none` and `visibility: hidden` hide content from screen readers too; use a `.visually-hidden` clip-pattern class for screen-reader-only text.
137 +- `tabindex` values >0 break natural focus order — only ever use `0` and `-1`.
138 +- `role="button"` on a div does NOT add Enter/Space handling; you must write the keydown handler yourself (another reason to use `<button>`).
139 +- Placeholder text is not a label and usually fails contrast; see designing-forms.
140 +- `aria-label` overrides visible text — if a button says "Save" but `aria-label="Submit"`, voice-control users saying "click Save" fail (WCAG 2.5.3 label-in-name).
141 +- Automated tools (axe, Lighthouse) catch roughly a third of WCAG issues; the keyboard and screen-reader walkthroughs are not optional.
added frontend-skills/optimizing-web-performance/SKILL.md +65 −0
@@ -0,0 +1,65 @@
1 +---
2 +name: optimizing-web-performance
3 +description: Optimizes web page loading and interactivity against Core Web Vitals budgets — LCP, INP, CLS, JS weight — via image/font optimization, code-splitting, and script loading. Use when the user asks to make a page or site faster, fix Core Web Vitals, improve LCP, INP, CLS, or Lighthouse/PageSpeed scores, reduce bundle size, optimize images or fonts, or diagnose slow page loads. Do not use for animation smoothness or jank during interactions (crafting-ui-animations) or for backend/API latency.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Optimizing Web Performance
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** page-load and interactivity performance — Core Web Vitals, bundle size, images, fonts, script loading, hydration cost.
15 +- **Do NOT use for:** animation frame-rate work (crafting-ui-animations), server/database latency, or React render architecture (building-react-components).
16 +
17 +## Budgets (2026 targets, p75 real-user)
18 +
19 +| Metric | Budget |
20 +|---|---|
21 +| LCP (Largest Contentful Paint) | ≤ 2.5 s |
22 +| INP (Interaction to Next Paint) | ≤ 200 ms |
23 +| CLS (Cumulative Layout Shift) | ≤ 0.1 |
24 +| JavaScript, gzipped, interactive page | ≤ 400 KB |
25 +
26 +## Core rules
27 +
28 +1. **Measure before optimizing.** Run Lighthouse (lab) and check real-user data (CrUX / RUM) first; fix the worst failing metric, not the easiest one.
29 + - ❌ "minify everything" before knowing whether LCP or INP is the problem
30 +
31 +2. **The LCP element loads first, eagerly.** Preload the hero image, serve it in AVIF/WebP, and never lazy-load it.
32 + - ✅ `<link rel="preload" as="image" href="hero.avif">` + `<img fetchpriority="high" …>`
33 + - ❌ `<img loading="lazy">` on the hero
34 +
35 +3. **Every image ships sized, modern, and responsive.** `width`/`height` attributes (prevents CLS), `srcset`/`sizes` for viewports, `loading="lazy"` below the fold only.
36 +
37 +4. **Fonts: self-host, subset, `font-display: swap`, preload the one used above the fold.** Two families maximum; variable font when more than two weights are needed.
38 +
39 +5. **Ship less JavaScript.** Code-split by route, `import()` heavy widgets on interaction, prefer server rendering with minimal hydration; audit with a bundle analyzer before adding any dependency over ~10 KB gz.
40 + - ✅ `const Chart = lazy(() => import('./Chart'))` mounted when scrolled into view
41 + - ❌ charting + date + animation libraries in the entry bundle
42 +
43 +6. **Nothing render-blocking, no third-party scripts in the critical path.** `defer` all scripts; load analytics/chat/ads after load or on idle; inline only the critical CSS.
44 +
45 +7. **Reserve space for everything that arrives late.** Ads, embeds, banners, and skeletons get fixed dimensions or `aspect-ratio` so nothing shifts (CLS).
46 +
47 +8. **Long tasks break INP — chunk them.** Split main-thread work over 50 ms with `scheduler.yield()`/`setTimeout`, debounce input handlers, move pure computation to a Web Worker.
48 +
49 +## Workflow
50 +
51 +1. Baseline: run Lighthouse on the target page (mobile, throttled) and record LCP/INP/CLS/JS-weight against the budget table.
52 +2. Identify the single worst offender per failing metric (LCP element, longest task, largest shift source, biggest bundle chunk).
53 +3. Apply the matching rule (2–8) to that offender only.
54 +4. Re-run Lighthouse; confirm the metric moved and no other metric regressed.
55 +5. Repeat 2–4 until all budgets pass, then verify with real-user data after deploy.
56 +
57 +## Edge cases & failure modes
58 +- **Lab passes, field fails** → trust field data; test on a low-end device profile and slow 4G; check geographic latency to origin (CDN).
59 +- **LCP element is text** → the font is the bottleneck: preload it, subset it, check `font-display`.
60 +- **Third-party script is required by the business** → load it after `load` event via a facade (static placeholder that loads the real widget on interaction); it cannot live in the critical path.
61 +- **Framework hydration dominates JS cost** → move non-interactive parts to server components/static rendering; hydrate islands only.
62 +- **No RUM available** → use CrUX (public, origin-level) as the field proxy; add `web-vitals` (npm) reporting when possible.
63 +
64 +## References
65 +Copy-paste snippets and gotchas: see [references/patterns.md](references/patterns.md)
added frontend-skills/optimizing-web-performance/references/patterns.md +153 −0
@@ -0,0 +1,153 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Optimizing Web Performance
7 +
8 +## Contents
9 +- LCP: hero image and critical path
10 +- Images: responsive, modern, shift-free
11 +- Fonts
12 +- JavaScript: splitting and on-interaction loading
13 +- Third-party scripts: facades and idle loading
14 +- INP: breaking long tasks
15 +- Measuring
16 +- Gotchas
17 +
18 +## LCP: hero image and critical path
19 +
20 +```html
21 +<head>
22 + <link rel="preconnect" href="https://cdn.example.com">
23 + <link rel="preload" as="image" href="hero-1200.avif"
24 + imagesrcset="hero-800.avif 800w, hero-1200.avif 1200w" imagesizes="100vw">
25 + <style>/* inlined critical CSS: layout + above-the-fold only */</style>
26 + <script src="/app.js" defer></script>
27 +</head>
28 +<body>
29 + <img src="hero-1200.avif"
30 + srcset="hero-800.avif 800w, hero-1200.avif 1200w" sizes="100vw"
31 + width="1200" height="600" fetchpriority="high" alt="…">
32 +```
33 +
34 +## Images: responsive, modern, shift-free
35 +
36 +```html
37 +<picture>
38 + <source type="image/avif" srcset="chart-400.avif 400w, chart-800.avif 800w">
39 + <source type="image/webp" srcset="chart-400.webp 400w, chart-800.webp 800w">
40 + <img src="chart-800.jpg" srcset="chart-400.jpg 400w, chart-800.jpg 800w"
41 + sizes="(max-width: 600px) 100vw, 50vw"
42 + width="800" height="500" loading="lazy" decoding="async" alt="…">
43 +</picture>
44 +```
45 +
46 +- `width`/`height` (or CSS `aspect-ratio`) on every `img`/`video`/`iframe` — CLS zero-cost insurance.
47 +- `loading="lazy"` only below the fold; the browser handles the rest.
48 +
49 +## Fonts
50 +
51 +```html
52 +<link rel="preload" as="font" type="font/woff2" href="/fonts/inter-var.woff2" crossorigin>
53 +```
54 +
55 +```css
56 +@font-face {
57 + font-family: 'Inter';
58 + src: url('/fonts/inter-var.woff2') format('woff2');
59 + font-weight: 100 900; /* one variable file replaces 5 static weights */
60 + font-display: swap; /* text visible immediately with fallback */
61 +}
62 +/* Reduce swap-induced shift: size-adjusted fallback */
63 +@font-face {
64 + font-family: 'Inter-fallback';
65 + src: local('Arial');
66 + size-adjust: 107%; /* match Inter's metrics; tune per family */
67 +}
68 +```
69 +
70 +Subset with `pyftsubset` (fonttools) to the scripts actually used — typically 30–100 KB → under 15 KB.
71 +
72 +## JavaScript: splitting and on-interaction loading
73 +
74 +```tsx
75 +// Route-level splitting (React Router / Next.js does this per page by default)
76 +const Settings = lazy(() => import('./Settings'));
77 +
78 +// Widget on interaction — nothing loads until the user needs it
79 +button.addEventListener('click', async () => {
80 + const { openChart } = await import('./chart.js');
81 + openChart(data);
82 +}, { once: true });
83 +```
84 +
85 +```bash
86 +# Find what's actually in the bundle before removing anything
87 +npx source-map-explorer dist/assets/*.js
88 +```
89 +
90 +## Third-party scripts: facades and idle loading
91 +
92 +```html
93 +<!-- Facade: static thumbnail replaces a 500 KB embed until clicked -->
94 +<button class="yt-facade" data-id="VIDEO_ID"
95 + style="background: url('https://i.ytimg.com/vi/VIDEO_ID/hqdefault.jpg')">▶</button>
96 +<script>
97 + document.querySelector('.yt-facade').addEventListener('click', (e) => {
98 + const iframe = document.createElement('iframe');
99 + iframe.src = `https://www.youtube.com/embed/${e.currentTarget.dataset.id}?autoplay=1`;
100 + iframe.width = 560; iframe.height = 315; iframe.allow = 'autoplay';
101 + e.currentTarget.replaceWith(iframe);
102 + }, { once: true });
103 +</script>
104 +
105 +<!-- Analytics after everything else -->
106 +<script>
107 + addEventListener('load', () => {
108 + requestIdleCallback(() => import('/analytics.js'));
109 + });
110 +</script>
111 +```
112 +
113 +## INP: breaking long tasks
114 +
115 +```js
116 +// Yield to the main thread between chunks of work (>50 ms tasks block input)
117 +async function processRows(rows) {
118 + for (const chunk of chunks(rows, 200)) { // 200 rows ≈ stays under 50 ms
119 + renderChunk(chunk);
120 + await (scheduler.yield?.() ?? new Promise(r => setTimeout(r)));
121 + }
122 +}
123 +```
124 +
125 +Pure computation (parsing, diffing, search indexing) → Web Worker; the main thread only renders.
126 +
127 +## Measuring
128 +
129 +```bash
130 +npx lighthouse https://example.com --preset=perf --form-factor=mobile --view
131 +```
132 +
133 +```js
134 +// Field data from real users
135 +import { onLCP, onINP, onCLS } from 'web-vitals';
136 +[onLCP, onINP, onCLS].forEach(fn => fn(m => navigator.sendBeacon('/vitals', JSON.stringify(m))));
137 +```
138 +
139 +## Gotchas
140 +
141 +- **Lazy-loading the LCP image** is the single most common self-inflicted LCP
142 + regression — audit every `loading="lazy"` above the fold.
143 +- **`preload` overuse** starves the network of bandwidth for actual critical
144 + resources — preload at most the LCP image and one font.
145 +- **`display: none` fonts still download** if declared in CSS — subset instead.
146 +- **CLS from late-loading banners/toolbars**: reserve the slot with
147 + `min-height` even when content is conditional.
148 +- **Debounce vs INP**: debouncing helps continuous input, but a *slow handler*
149 + needs chunking (see INP section) — debounce doesn't shorten the task.
150 +- **Bundle analyzer lies about tree-shaking** until you build in production
151 + mode — always analyze the production build.
152 +- **CDN cache misses** dominate TTFB for global users — check `cf-cache-status`
153 + / `x-cache` headers before optimizing the payload.
added frontend-skills/structuring-semantic-html/SKILL.md +61 −0
@@ -0,0 +1,61 @@
1 +---
2 +name: structuring-semantic-html
3 +description: Structures HTML documents semantically — landmarks, heading hierarchy, meaningful lists/tables/figures, and SEO/meta/Open Graph tags. Use when the user asks to write or review the HTML structure of a page, fix heading levels, add landmarks, improve SEO markup or social previews, or decide between a div and a semantic element. Do not use for ARIA attributes, keyboard, or screen-reader work (ensuring-accessibility) or for parsing/editing existing HTML files programmatically (processing-html).
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Structuring Semantic HTML
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** authoring or reviewing page structure — landmarks, headings, sectioning, lists/tables/figures, `<head>` metadata, Open Graph/Twitter cards.
15 +- **Do NOT use for:** ARIA roles/states, focus management, screen-reader testing (→ ensuring-accessibility); programmatic parsing or bulk editing of HTML files (→ processing-html); visual styling.
16 +
17 +## Core rules
18 +
19 +1. **One `<main>` per page, every byte of content inside a landmark.**
20 + - ✅ `<header>` → `<nav>` → `<main>` → `<footer>`, asides in `<aside>`
21 + - ❌ Content floating in `<body>` between landmark regions
22 +
23 +2. **Exactly one `<h1>`; heading levels never skip down.**
24 + - ✅ `h1 → h2 → h3`, next section starts back at `h2`
25 + - ❌ `h1 → h3` because the h3 "looks right" — fix size with CSS, not level
26 +
27 +3. **Element by meaning, div only when no element carries the meaning.**
28 + - ✅ `<button>` for actions, `<a href>` for navigation, `<time datetime="2026-08-05">`, `<address>`, `<dl>` for key–value pairs
29 + - ❌ `<div class="button" onclick=…>`, `<span class="date">`
30 +
31 +4. **Tables for data, never for layout; always `<caption>` + `<th scope>`.**
32 + - ✅ `<table><caption>Q2 revenue</caption><thead><tr><th scope="col">…`
33 + - ❌ A grid of divs presenting tabular data, or a table used to position content
34 +
35 +5. **Sectioning: `<article>` = self-contained/syndicatable, `<section>` = titled thematic group (must contain a heading), `<div>` = styling hook only.**
36 +
37 +6. **`<figure>` + `<figcaption>` for any image/chart/code the text refers to.**
38 + - ✅ `<figure><img src="chart.png" alt="Revenue grew 40% in Q2"><figcaption>Fig 1. Quarterly revenue</figcaption></figure>`
39 + - ❌ An image and an italic paragraph pretending to be a caption
40 +
41 +7. **Minimum viable `<head>`:** `<meta charset="utf-8">`, `<meta name="viewport" content="width=device-width, initial-scale=1">`, unique `<title>` (≤60 chars, page-specific first), `<meta name="description">` (≤160 chars), canonical URL. Add Open Graph (`og:title`, `og:description`, `og:image` 1200×630, `og:url`, `og:type`) and `<meta name="twitter:card" content="summary_large_image">` for any shareable page.
42 +
43 +8. **Navigation is a list.**
44 + - ✅ `<nav aria-label="Main"><ul><li><a …>` (the `aria-label` here names the landmark; deeper ARIA belongs to ensuring-accessibility)
45 + - ❌ A row of bare `<a>` tags or divs
46 +
47 +## Workflow
48 +
49 +1. Outline the content hierarchy first (what is the one h1; what are the sections) — before writing any tags.
50 +2. Lay down landmarks, then headings, then flow content, choosing elements by rule 3.
51 +3. Fill the `<head>` per rule 7.
52 +4. Self-review: exactly one `<h1>`? No skipped heading levels (grep `<h[1-6]` and read the sequence)? All content inside landmarks? Every `<section>` has a heading? Title and description unique and within length?
53 +
54 +## Edge cases & failure modes
55 +- **Single-page apps:** the rules apply to the rendered DOM — verify the hydrated output, not just the template.
56 +- **Multiple h1s from a CMS/theme:** demote all but the page's main topic; adjust sizes in CSS.
57 +- **No semantic element fits** (pure styling wrapper): use `<div>` without guilt — forcing `<section>` everywhere is as wrong as div-soup.
58 +- **Legacy layout tables:** convert to CSS layout only when asked; otherwise flag it and move on.
59 +
60 +## References
61 +Copy-paste page skeletons, meta blocks, and data-table patterns: see [references/patterns.md](references/patterns.md)
added frontend-skills/structuring-semantic-html/references/patterns.md +151 −0
@@ -0,0 +1,151 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Semantic HTML Structure
7 +
8 +## Contents
9 +- Full page skeleton
10 +- Complete head block (SEO + social)
11 +- Article with sections
12 +- Data table
13 +- Figure, time, address, definition list
14 +- Breadcrumbs
15 +- Gotchas
16 +
17 +## Full page skeleton
18 +
19 +```html
20 +<!doctype html>
21 +<html lang="en">
22 +<head>…see head block below…</head>
23 +<body>
24 + <header>
25 + <a href="/" class="logo">Acme</a>
26 + <nav aria-label="Main">
27 + <ul>
28 + <li><a href="/products">Products</a></li>
29 + <li><a href="/pricing">Pricing</a></li>
30 + </ul>
31 + </nav>
32 + </header>
33 +
34 + <main>
35 + <h1>Page topic — the only h1</h1>
36 + <section>
37 + <h2>First theme</h2>
38 + <p>…</p>
39 + </section>
40 + <aside aria-label="Related links">
41 + <h2>Related</h2>
42 + <ul>…</ul>
43 + </aside>
44 + </main>
45 +
46 + <footer>
47 + <p><small>© 2026 Acme</small></p>
48 + </footer>
49 +</body>
50 +</html>
51 +```
52 +
53 +## Complete head block (SEO + social)
54 +
55 +```html
56 +<head>
57 + <meta charset="utf-8">
58 + <meta name="viewport" content="width=device-width, initial-scale=1">
59 + <title>Pricing — Acme</title> <!-- ≤60 chars, specific first -->
60 + <meta name="description" content="Acme plans from $9/mo. Compare Starter, Pro, and Team features."> <!-- ≤160 chars -->
61 + <link rel="canonical" href="https://acme.com/pricing">
62 +
63 + <!-- Open Graph -->
64 + <meta property="og:title" content="Pricing — Acme">
65 + <meta property="og:description" content="Acme plans from $9/mo.">
66 + <meta property="og:image" content="https://acme.com/og/pricing.png"> <!-- 1200×630 -->
67 + <meta property="og:url" content="https://acme.com/pricing">
68 + <meta property="og:type" content="website">
69 +
70 + <!-- Twitter -->
71 + <meta name="twitter:card" content="summary_large_image">
72 +</head>
73 +```
74 +
75 +## Article with sections
76 +
77 +```html
78 +<article>
79 + <header>
80 + <h1>How we cut LCP to 1.8s</h1>
81 + <p>By <address style="display:inline">Jane Doe</address> ·
82 + <time datetime="2026-08-05">August 5, 2026</time></p>
83 + </header>
84 + <section>
85 + <h2>The problem</h2>
86 + <p>…</p>
87 + </section>
88 + <section>
89 + <h2>What we changed</h2>
90 + <h3>Images</h3>
91 + <p>…</p>
92 + </section>
93 + <footer>
94 + <p>Filed under <a href="/tags/perf">performance</a></p>
95 + </footer>
96 +</article>
97 +```
98 +
99 +## Data table
100 +
101 +```html
102 +<table>
103 + <caption>Revenue by region, Q2 2026</caption>
104 + <thead>
105 + <tr><th scope="col">Region</th><th scope="col">Revenue</th></tr>
106 + </thead>
107 + <tbody>
108 + <tr><th scope="row">EMEA</th><td>$4.2M</td></tr>
109 + <tr><th scope="row">APAC</th><td>$3.1M</td></tr>
110 + </tbody>
111 +</table>
112 +```
113 +
114 +## Figure, time, address, definition list
115 +
116 +```html
117 +<figure>
118 + <img src="chart.png" alt="Revenue grew 40% quarter over quarter in Q2 2026">
119 + <figcaption>Fig 1. Quarterly revenue</figcaption>
120 +</figure>
121 +
122 +<time datetime="2026-08-05T14:30">Aug 5, 2:30 PM</time>
123 +
124 +<dl>
125 + <dt>Plan</dt><dd>Pro</dd>
126 + <dt>Seats</dt><dd>25</dd>
127 +</dl>
128 +```
129 +
130 +## Breadcrumbs
131 +
132 +```html
133 +<nav aria-label="Breadcrumb">
134 + <ol>
135 + <li><a href="/">Home</a></li>
136 + <li><a href="/docs">Docs</a></li>
137 + <li aria-current="page">Installation</li>
138 + </ol>
139 +</nav>
140 +```
141 +
142 +## Gotchas
143 +
144 +- **`<section>` without a heading is meaningless** — if it has no heading, it should be a `<div>`.
145 +- **`<article>` vs `<section>`:** ask "would this make sense in an RSS feed alone?" Yes → article.
146 +- **Multiple `<nav>` landmarks need distinguishing `aria-label`s** ("Main", "Breadcrumb", "Footer") — otherwise they announce identically.
147 +- **`<time>` requires machine-readable `datetime`** when the text isn't already ISO format.
148 +- **`og:image` must be an absolute URL**; relative paths silently break social previews.
149 +- **`<title>` and `og:title` can differ** — title is for tabs/SERP (include brand), og:title for the share card.
150 +- **Skipping `<thead>`/`<th scope>`** makes header cells unassociable; screen readers and copy-paste both degrade.
151 +- **`<b>`/`<i>` vs `<strong>`/`<em>`:** the former are stylistic offsets, the latter carry emphasis semantics — pick by meaning.
added frontend-skills/theming-design-tokens/SKILL.md +55 −0
@@ -0,0 +1,55 @@
1 +---
2 +name: theming-design-tokens
3 +description: Builds design-token systems with CSS custom properties — primitive/semantic/component tiers, dark mode via prefers-color-scheme with a data-theme override, spacing/radius/shadow scales, and contrast-safe color palettes. Use when the user asks to set up design tokens or CSS variables, add dark mode or theme switching, define a color palette or spacing scale, or make theme colors meet contrast requirements. Do not use for typeface selection and type scales (choosing-typography) or for one-off page styling that no system will reuse.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Theming with Design Tokens
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** token architecture, palettes, dark mode/theming, spacing/radius/shadow scales, migrating hard-coded values to tokens.
15 +- **Do NOT use for:** font pairing and type scales (→ choosing-typography); layout mechanics (→ designing-responsive-layouts); styling a single element nothing else will reuse — just style it.
16 +
17 +## Core rules
18 +
19 +1. **Three tiers, referenced downward only: primitive → semantic → component.** Components consume semantic tokens; semantic tokens reference primitives; nothing skips upward.
20 + - ✅ `--color-blue-600: #2563eb;` → `--color-accent: var(--color-blue-600);` → `--button-bg: var(--color-accent);`
21 + - ❌ `--button-bg: #2563eb;` (component pinned to a raw hex)
22 +
23 +2. **Semantic names describe role, never appearance.**
24 + - ✅ `--color-bg-surface`, `--color-text-muted`, `--color-border-danger`
25 + - ❌ `--light-gray-2`, `--dark-blue-bg` (breaks the moment the theme flips)
26 +
27 +3. **Dark mode = redefining semantic tokens only.** Primitives and components never change per theme.
28 +
29 +4. **Both mechanisms, always:** `@media (prefers-color-scheme: dark)` for the default, `[data-theme="dark"]` / `[data-theme="light"]` for the user override; plus `color-scheme: light dark` on `:root` so form controls and scrollbars follow.
30 +
31 +5. **Contrast is enforced in the token definitions, not per usage:** every `text`/`bg` semantic pair ≥ 4.5:1 (WCAG AA), every UI-component/border pair ≥ 3:1. Verify pairs when defining them — then usage is safe by construction.
32 +
33 +6. **Scales, not ad-hoc values.** Spacing on a base-4/8 scale (`0.25/0.5/0.75/1/1.5/2/3/4rem`), 3–4 radii, 3 shadow levels. A new value must join the scale or justify itself.
34 + - ✅ `padding: var(--space-4) var(--space-6);`
35 + - ❌ `padding: 13px 22px;`
36 +
37 +7. **Token count stays small:** ~10–20 semantic color tokens covers most products. If you have 60, roles are duplicated — merge before adding.
38 +
39 +## Workflow
40 +
41 +1. Define primitives: a 3–5 step neutral ramp + 1 accent ramp (+ success/warn/danger primitives), spacing/radius/shadow scales.
42 +2. Define semantic tokens for both themes (rule 2–4), checking each text/bg pair's contrast ratio as you go (rule 5).
43 +3. Wire components to semantic tokens only.
44 +4. Add the theme toggle contract: `data-theme` attribute on `<html>`, persisted; absence = follow system.
45 +5. Validate: toggle both themes; confirm every visible pairing passes 4.5:1 (text) / 3:1 (UI); grep the stylesheet for stray hex/rgb values outside the primitives block — there should be none.
46 +
47 +## Edge cases & failure modes
48 +- **Brand color fails contrast on light bg:** keep the brand primitive for large/decorative use; add a darkened `-text-safe` variant for text/small UI, and note the pair it passes against.
49 +- **Images/illustrations in dark mode:** don't invert; reduce brightness slightly (`filter: brightness(0.9)`) or provide themed assets.
50 +- **Shadows invisible on dark surfaces:** in dark themes, convey elevation with a subtly lighter surface token instead of larger shadows.
51 +- **Third-party widgets ignore tokens:** scope overrides in one `@layer third-party` block rather than scattering `!important`.
52 +- **Flash of wrong theme on load:** set `data-theme` in a tiny inline script in `<head>` before CSS paints.
53 +
54 +## References
55 +Full starter token sheet (light + dark), toggle script, and contrast-checked palette: see [references/patterns.md](references/patterns.md)
added frontend-skills/theming-design-tokens/references/patterns.md +153 −0
@@ -0,0 +1,153 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Patterns — Design Tokens & Theming
7 +
8 +## Contents
9 +- Starter token sheet (primitives)
10 +- Semantic tokens, light + dark
11 +- Component tier example
12 +- Theme toggle (no-flash)
13 +- Contrast quick-check
14 +- Gotchas
15 +
16 +## Starter token sheet (primitives)
17 +
18 +```css
19 +:root {
20 + /* Neutral ramp (5 steps is enough to start) */
21 + --gray-50: #f8fafc;
22 + --gray-200: #e2e8f0;
23 + --gray-500: #64748b;
24 + --gray-800: #1e293b;
25 + --gray-950: #0b1220;
26 +
27 + /* Accent ramp */
28 + --blue-400: #60a5fa;
29 + --blue-600: #2563eb;
30 + --blue-700: #1d4ed8;
31 +
32 + /* Status */
33 + --green-600: #16a34a;
34 + --amber-600: #d97706;
35 + --red-600: #dc2626;
36 +
37 + /* Spacing — base-4 scale */
38 + --space-1: 0.25rem; --space-2: 0.5rem; --space-3: 0.75rem;
39 + --space-4: 1rem; --space-6: 1.5rem; --space-8: 2rem;
40 + --space-12: 3rem; --space-16: 4rem;
41 +
42 + /* Radius & shadows — 3 levels each */
43 + --radius-sm: 0.25rem; --radius-md: 0.5rem; --radius-full: 9999px;
44 + --shadow-1: 0 1px 2px rgb(0 0 0 / 0.06);
45 + --shadow-2: 0 4px 8px rgb(0 0 0 / 0.10);
46 + --shadow-3: 0 12px 24px rgb(0 0 0 / 0.14);
47 +
48 + color-scheme: light dark;
49 +}
50 +```
51 +
52 +## Semantic tokens, light + dark
53 +
54 +```css
55 +/* Light (default) */
56 +:root, [data-theme="light"] {
57 + --color-bg-page: var(--gray-50);
58 + --color-bg-surface: #ffffff;
59 + --color-text: var(--gray-800); /* on bg-page: 12.6:1 ✓ */
60 + --color-text-muted: var(--gray-500); /* on surface: 4.8:1 ✓ */
61 + --color-border: var(--gray-200);
62 + --color-accent: var(--blue-600); /* as text on surface: 5.2:1 ✓ */
63 + --color-accent-hover: var(--blue-700);
64 + --color-danger: var(--red-600);
65 +}
66 +
67 +/* Dark via system preference (only when no explicit choice) */
68 +@media (prefers-color-scheme: dark) {
69 + :root:not([data-theme]) {
70 + --color-bg-page: var(--gray-950);
71 + --color-bg-surface: var(--gray-800);
72 + --color-text: var(--gray-50);
73 + --color-text-muted: #94a3b8; /* on gray-800: 5.9:1 ✓ */
74 + --color-border: #334155;
75 + --color-accent: var(--blue-400); /* lighter accent for dark bg: 6.6:1 ✓ */
76 + --color-accent-hover: #93c5fd;
77 + }
78 +}
79 +
80 +/* Dark via explicit user choice — identical block, attribute-scoped */
81 +[data-theme="dark"] {
82 + --color-bg-page: var(--gray-950);
83 + --color-bg-surface: var(--gray-800);
84 + --color-text: var(--gray-50);
85 + --color-text-muted: #94a3b8;
86 + --color-border: #334155;
87 + --color-accent: var(--blue-400);
88 + --color-accent-hover: #93c5fd;
89 +}
90 +```
91 +
92 +## Component tier example
93 +
94 +```css
95 +.button-primary {
96 + background: var(--color-accent);
97 + color: var(--color-bg-surface);
98 + padding: var(--space-2) var(--space-4);
99 + border-radius: var(--radius-md);
100 + box-shadow: var(--shadow-1);
101 +}
102 +.button-primary:hover { background: var(--color-accent-hover); }
103 +
104 +.card {
105 + background: var(--color-bg-surface);
106 + border: 1px solid var(--color-border);
107 + border-radius: var(--radius-md);
108 + padding: var(--space-6);
109 +}
110 +```
111 +
112 +## Theme toggle (no-flash)
113 +
114 +```html
115 +<!-- In <head>, BEFORE any stylesheet, so first paint is themed -->
116 +<script>
117 + const t = localStorage.getItem("theme"); // "light" | "dark" | null
118 + if (t) document.documentElement.dataset.theme = t; // absent = follow system
119 +</script>
120 +```
121 +
122 +```js
123 +function setTheme(next) { // next: "light" | "dark" | "system"
124 + if (next === "system") {
125 + delete document.documentElement.dataset.theme;
126 + localStorage.removeItem("theme");
127 + } else {
128 + document.documentElement.dataset.theme = next;
129 + localStorage.setItem("theme", next);
130 + }
131 +}
132 +```
133 +
134 +## Contrast quick-check
135 +
136 +Compute ratio = (L1 + 0.05) / (L2 + 0.05) with relative luminance, or verify in
137 +devtools (element → color picker shows the ratio). Thresholds to enforce **in
138 +the token sheet**:
139 +
140 +| Pair | Minimum |
141 +|---|---|
142 +| Body/label text on its background | 4.5:1 (AA) |
143 +| Large text ≥24px (or 18.7px bold) | 3:1 |
144 +| UI component borders, icons, focus rings | 3:1 |
145 +
146 +## Gotchas
147 +
148 +- **`prefers-color-scheme` block must exclude explicit choices** — scope it with `:root:not([data-theme])` or a user who picked "light" gets system-dark anyway.
149 +- **Forgetting `color-scheme: light dark`** leaves scrollbars, form controls, and default UA styles stuck in light mode.
150 +- **Muted text is the #1 AA failure** — grays around `#999` fail on white (2.8:1). Nothing lighter than `#767676` for body-size text on white.
151 +- **Same accent in both themes usually fails one of them** — accents need a per-theme value (darker for light bg, lighter for dark bg).
152 +- **`rgb(0 0 0 / 0.5)`-style translucent text** has no fixed contrast ratio (depends on what's behind) — use opaque token colors for text.
153 +- **Tokens in shadows:** define whole shadows as tokens, not just colors — dark themes often need different alpha, not different hue.
added skill-1-profiling-csv-data/SKILL.md +36 −0
@@ -0,0 +1,36 @@
1 +---
2 +name: profiling-csv-data
3 +description: Profiles CSV files and produces a structured data-quality report covering column types, missing values, duplicates, outliers, and summary statistics. Use when the user asks to profile, audit, inspect, or check the quality of a CSV file, asks "what's in this CSV", or asks about columns, missing/null values, duplicate rows, or basic statistics of a CSV dataset. Do not use for converting, transforming, editing, or plotting CSV data.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Profiling CSV Data
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** profiling, auditing, or summarizing the contents and quality of one or more `.csv` files.
15 +- **Do NOT use for:** format conversion (CSV→JSON/Excel), editing or cleaning data, plotting/visualization, or non-CSV files. Answer those directly or with the appropriate tool.
16 +
17 +## Workflow
18 +Copy this checklist and check off items as you complete them:
19 +
20 +- [ ] Step 1: Run the profiler (execute, do not read): `python3 scripts/profile_csv.py <path/to/file.csv>`
21 +- [ ] Step 2: If the script exits non-zero, report its error message to the user verbatim and stop.
22 +- [ ] Step 3: Render the JSON output into a Markdown report following [references/report-format.md](references/report-format.md) exactly.
23 +- [ ] Step 4: Save the report as `<input-stem>-profile.md` in the same directory as the input CSV (e.g., `sales.csv` → `sales-profile.md`).
24 +- [ ] Step 5: Validate: every column in the JSON appears in the report table, and the three verdict rules from the report format were applied. If not, fix and repeat Step 3.
25 +- [ ] Step 6: Reply to the user with the report file path and the one-line verdict only. Do not paste the full report into chat unless asked.
26 +
27 +## Rules
28 +- The script is the single source of truth for all numbers. Never recompute or estimate statistics yourself.
29 +- Profile at most 100,000 rows (the script enforces this and sets `"truncated": true`). If truncated, the report MUST state it.
30 +- Multiple CSV files → one report per file, same naming rule.
31 +
32 +## Edge cases & failure modes
33 +- **File not found / not readable** → script exits 2; relay its message, stop.
34 +- **Empty file or header-only file** → script exits 0 with `"rows": 0`; produce the report anyway and set verdict ⚠️ with note "file contains no data rows".
35 +- **Ragged rows** (inconsistent column counts) → reported in `ragged_rows`; list the count in the Issues section.
36 +- **Non-UTF-8 encoding** → the script falls back to latin-1 and sets `"encoding_fallback": true`; mention it in Issues.
added skill-1-profiling-csv-data/references/report-format.md +52 −0
@@ -0,0 +1,52 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Report Format — CSV Data-Quality Profile
7 +
8 +Render the profiler's JSON into exactly this Markdown structure. Do not add,
9 +remove, or reorder sections.
10 +
11 +```markdown
12 +# Data Profile: <file name>
13 +
14 +**Verdict:** <one of the three verdict lines below>
15 +
16 +## Overview
17 +| Metric | Value |
18 +|---|---|
19 +| Rows profiled | <rows> (+ " (truncated at 100,000)" if truncated) |
20 +| Columns | <number of columns> |
21 +| Duplicate rows | <duplicate_rows> |
22 +| Ragged rows | <ragged_rows> |
23 +
24 +## Columns
25 +| Column | Type | Nulls | Unique | Stats |
26 +|---|---|---|---|---|
27 +| <name> | <type> | <nulls> | <unique> | <stats cell — see rule below> |
28 +
29 +## Issues
30 +- <one bullet per detected issue; write "None detected." if empty>
31 +```
32 +
33 +## Cell and verdict rules
34 +
35 +**Stats cell:**
36 +- numeric columns → `min=<min>, max=<max>, mean=<mean>, median=<median>` plus `, outliers=<n>` when the key is present and > 0
37 +- string columns → `top: <value> (<count>), <value> (<count>), ...` from `top_values`
38 +- boolean/date/empty columns → `—`
39 +
40 +**Issue bullets** (include each only when its condition is true):
41 +- `nulls > 0` in a column → "`<column>` has <n> missing values (<percent of rows>%)"
42 +- `duplicate_rows > 0` → "<n> duplicate rows"
43 +- `ragged_rows > 0` → "<n> rows have an inconsistent number of fields"
44 +- `outliers` present and > 0 → "`<column>` has <n> outliers (>3σ from mean)"
45 +- `encoding_fallback` is true → "file is not valid UTF-8; profiled using latin-1 fallback"
46 +- `truncated` is true → "profile limited to the first 100,000 rows"
47 +- `rows == 0` → "file contains no data rows"
48 +
49 +**Verdict line** (pick exactly one, in this priority order):
50 +1. `rows == 0` OR `ragged_rows > 0` → `⚠️ Needs attention — <short reason>`
51 +2. any other issue bullet present → `🟡 Usable with caveats — <n> issue(s) found`
52 +3. no issue bullets → `✅ Clean — no data-quality issues detected`
added skill-1-profiling-csv-data/scripts/profile_csv.py +167 −0
@@ -0,0 +1,167 @@
1 +#!/usr/bin/env python3
2 +# Author: Simon-Pierre Boucher
3 +# Contact: contact@spboucher.ai
4 +#
5 +# profile_csv.py — deterministic CSV profiler. Stdlib only.
6 +# Usage: python3 profile_csv.py <file.csv>
7 +# Prints a JSON profile to stdout. Exit codes: 0 = OK, 2 = unreadable input.
8 +
9 +import csv
10 +import json
11 +import statistics
12 +import sys
13 +from collections import Counter
14 +from pathlib import Path
15 +
16 +# Cap keeps runtime and memory bounded on very large files; 100k rows is
17 +# enough for stable summary statistics on any practical column.
18 +MAX_ROWS = 100_000
19 +
20 +# A column is typed integer/float/date/boolean only if at least 95% of its
21 +# non-empty values parse as that type; below that, mixed content is safer
22 +# reported as "string" than as a false-precision numeric column.
23 +TYPE_THRESHOLD = 0.95
24 +
25 +# Top-N most frequent values reported for string columns; 5 shows the
26 +# dominant categories without bloating the report.
27 +TOP_VALUES = 5
28 +
29 +# Values beyond 3 standard deviations from the mean are counted as outliers
30 +# (classic three-sigma rule).
31 +OUTLIER_SIGMAS = 3
32 +
33 +
34 +def is_int(s):
35 + try:
36 + int(s)
37 + return True
38 + except ValueError:
39 + return False
40 +
41 +
42 +def is_float(s):
43 + try:
44 + float(s)
45 + return True
46 + except ValueError:
47 + return False
48 +
49 +
50 +def is_bool(s):
51 + return s.strip().lower() in ("true", "false", "yes", "no", "0", "1")
52 +
53 +
54 +def is_date(s):
55 + s = s.strip()
56 + for sep in ("-", "/"):
57 + parts = s.split(sep)
58 + if len(parts) == 3 and all(p.isdigit() for p in parts):
59 + return True
60 + return False
61 +
62 +
63 +def infer_type(values):
64 + """Return the dominant type of non-empty values per TYPE_THRESHOLD."""
65 + if not values:
66 + return "empty"
67 + n = len(values)
68 + for name, pred in (("integer", is_int), ("float", is_float),
69 + ("boolean", is_bool), ("date", is_date)):
70 + if sum(1 for v in values if pred(v)) / n >= TYPE_THRESHOLD:
71 + return name
72 + return "string"
73 +
74 +
75 +def read_rows(path):
76 + """Read the CSV, falling back to latin-1 if UTF-8 fails."""
77 + fallback = False
78 + try:
79 + with open(path, newline="", encoding="utf-8") as f:
80 + rows = list(csv.reader(f))
81 + except UnicodeDecodeError:
82 + fallback = True
83 + with open(path, newline="", encoding="latin-1") as f:
84 + rows = list(csv.reader(f))
85 + return rows, fallback
86 +
87 +
88 +def profile_column(name, values):
89 + non_empty = [v for v in values if v.strip() != ""]
90 + col = {
91 + "name": name,
92 + "type": infer_type(non_empty),
93 + "nulls": len(values) - len(non_empty),
94 + "unique": len(set(non_empty)),
95 + }
96 + if col["type"] in ("integer", "float"):
97 + nums = [float(v) for v in non_empty if is_float(v)]
98 + if nums:
99 + col["min"] = min(nums)
100 + col["max"] = max(nums)
101 + col["mean"] = round(statistics.fmean(nums), 4)
102 + col["median"] = statistics.median(nums)
103 + if len(nums) > 1:
104 + sd = statistics.stdev(nums)
105 + col["stdev"] = round(sd, 4)
106 + if sd > 0:
107 + m = statistics.fmean(nums)
108 + col["outliers"] = sum(
109 + 1 for x in nums if abs(x - m) > OUTLIER_SIGMAS * sd)
110 + elif col["type"] == "string" and non_empty:
111 + col["top_values"] = Counter(non_empty).most_common(TOP_VALUES)
112 + return col
113 +
114 +
115 +def main():
116 + if len(sys.argv) != 2:
117 + print("usage: profile_csv.py <file.csv>", file=sys.stderr)
118 + sys.exit(2)
119 + path = Path(sys.argv[1])
120 + if not path.is_file():
121 + print(f"error: file not found or not a regular file: {path}",
122 + file=sys.stderr)
123 + sys.exit(2)
124 +
125 + try:
126 + rows, encoding_fallback = read_rows(path)
127 + except OSError as e:
128 + print(f"error: cannot read {path}: {e}", file=sys.stderr)
129 + sys.exit(2)
130 +
131 + profile = {
132 + "file": str(path),
133 + "encoding_fallback": encoding_fallback,
134 + "truncated": False,
135 + "rows": 0,
136 + "columns": [],
137 + "duplicate_rows": 0,
138 + "ragged_rows": 0,
139 + }
140 +
141 + if not rows:
142 + print(json.dumps(profile, indent=2))
143 + return
144 +
145 + header, data = rows[0], rows[1:]
146 + if len(data) > MAX_ROWS:
147 + data = data[:MAX_ROWS]
148 + profile["truncated"] = True
149 +
150 + width = len(header)
151 + profile["ragged_rows"] = sum(1 for r in data if len(r) != width)
152 + # Ragged rows are padded/clipped so every column still gets profiled.
153 + normalized = [(r + [""] * width)[:width] for r in data]
154 +
155 + profile["rows"] = len(normalized)
156 + profile["duplicate_rows"] = len(normalized) - len(
157 + {tuple(r) for r in normalized})
158 + profile["columns"] = [
159 + profile_column(name, [r[i] for r in normalized])
160 + for i, name in enumerate(header)
161 + ]
162 +
163 + print(json.dumps(profile, indent=2))
164 +
165 +
166 +if __name__ == "__main__":
167 + main()
added skill-2-writing-release-notes/SKILL.md +70 −0
@@ -0,0 +1,70 @@
1 +---
2 +name: writing-release-notes
3 +description: Writes user-facing release notes and changelog entries in the house style — benefit-first, plain language, grouped by Added/Improved/Fixed. Use when the user asks to write, draft, or edit release notes, a changelog entry, a "what's new" section, or a version announcement for end users. Do not use for git commit messages, internal PR descriptions, or API reference documentation.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing Release Notes
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** release notes, changelog entries, "what's new" copy, version announcements — anything a **customer or end user** will read about a release.
15 +- **Do NOT use for:** git commit messages, PR titles/descriptions, internal engineering changelogs, API reference docs, or marketing landing pages. Those have different audiences and conventions — write them normally.
16 +- If the audience is ambiguous, ask one question: "Is this for end users or for engineers?" Apply this skill only for end users.
17 +
18 +## House style — the five rules
19 +
20 +1. **Lead with the user benefit, not the implementation.**
21 + - ✅ "Search results now load twice as fast."
22 + - ❌ "Migrated the search index to OpenSearch 2.x with query caching."
23 +
24 +2. **Plain language, present tense, second person where natural.**
25 + - ✅ "You can now export reports as PDF."
26 + - ❌ "PDF export functionality has been implemented."
27 + - Never use: "leverage", "utilize", "robust", "seamless", "enhanced UX", ticket IDs, or internal codenames.
28 +
29 +3. **Group entries under exactly three headings, in this order: `### Added`, `### Improved`, `### Fixed`.** Omit a heading only if it has no entries. One bullet per change, max 2 sentences per bullet.
30 +
31 +4. **Every breaking change gets a `> **Breaking:**` blockquote at the TOP of the notes,** before any heading, stating what breaks and the one-line migration action. Never bury a breaking change in a bullet.
32 +
33 +5. **Cut the noise.** Never include: dependency bumps with no user impact, internal refactors, test/CI changes, or "various bug fixes and improvements" filler. If a change has no observable effect for the user, it does not appear.
34 +
35 +## Workflow
36 +
37 +1. Collect the changes (diff, commit list, or user's description). Discard anything rule 5 excludes.
38 +2. Classify each remaining change as Added / Improved / Fixed; identify breaking changes.
39 +3. Rewrite each change benefit-first (rule 1) in plain language (rule 2).
40 +4. Assemble in the output format below.
41 +5. Self-review against the five rules; fix violations before delivering.
42 +
43 +## Output format
44 +
45 +```markdown
46 +## <Product name> <version> — <YYYY-MM-DD>
47 +
48 +> **Breaking:** <what breaks + migration action> ← only if applicable
49 +
50 +<One-sentence summary of the release theme.>
51 +
52 +### Added
53 +- <bullet>
54 +
55 +### Improved
56 +- <bullet>
57 +
58 +### Fixed
59 +- <bullet>
60 +```
61 +
62 +Deliver in chat by default; write to `CHANGELOG.md` (prepended above previous entries) only if the user names a file or the repo already has one.
63 +
64 +## Edge cases
65 +- **No user-visible changes at all** → say so and propose the one-liner: "This release contains internal improvements only." Do not invent benefits.
66 +- **Unclear whether a change is user-visible** → include it under Improved, phrased by observable effect; flag your assumption to the user.
67 +- **Version or date unknown** → use placeholders `<version>` / `<date>` and tell the user to fill them in.
68 +
69 +## More examples
70 +Full positive/negative example pairs, including a complete before/after release: see [references/examples.md](references/examples.md).
added skill-2-writing-release-notes/references/examples.md +97 −0
@@ -0,0 +1,97 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Examples — Release Notes House Style
7 +
8 +## Contents
9 +- Bullet-level pairs (do / never)
10 +- Breaking-change placement
11 +- Complete before/after release
12 +
13 +## Bullet-level pairs (do / never)
14 +
15 +**Pair 1 — benefit first**
16 +- ✅ "Dashboards now refresh automatically every 30 seconds — no more manual reloads."
17 +- ❌ "Implemented WebSocket-based polling refactor for dashboard state (JIRA-4521)."
18 +
19 +**Pair 2 — plain language**
20 +- ✅ "Fixed a bug where exported CSVs dropped the last row."
21 +- ❌ "Resolved an edge-case regression in the serialization pipeline impacting terminal record emission."
22 +
23 +**Pair 3 — noise cut**
24 +- ✅ (entry omitted entirely)
25 +- ❌ "Bumped lodash from 4.17.20 to 4.17.21."
26 +
27 +**Pair 4 — no filler**
28 +- ✅ "Fixed three crashes on startup affecting Windows users with non-Latin usernames."
29 +- ❌ "Various bug fixes and performance improvements."
30 +
31 +## Breaking-change placement
32 +
33 +✅ Correct — top of the notes, before any heading:
34 +
35 +```markdown
36 +## Acme CLI 3.0.0 — 2026-08-01
37 +
38 +> **Breaking:** `acme deploy` now requires `--env`. Add `--env production` to existing scripts.
39 +
40 +This release focuses on safer deployments.
41 +
42 +### Added
43 +- ...
44 +```
45 +
46 +❌ Wrong — buried as a bullet:
47 +
48 +```markdown
49 +### Improved
50 +- Deploy command now requires an environment flag (breaking).
51 +```
52 +
53 +## Complete before/after release
54 +
55 +**Input (engineer's commit list):**
56 +
57 +```
58 +- feat: add SSO via SAML (ENG-201)
59 +- perf: rewrite query planner, p95 latency 840ms -> 210ms
60 +- fix: null pointer when profile avatar missing
61 +- chore: upgrade CI runners to ubuntu-24.04
62 +- refactor: extract billing module
63 +- feat!: remove legacy /v1/auth endpoint
64 +```
65 +
66 +**❌ Never ship this (implementation-first, noise included, breaking buried):**
67 +
68 +```markdown
69 +## 2.8.0
70 +- Added SAML SSO support (ENG-201)
71 +- Rewrote query planner
72 +- Fixed NPE on missing avatar
73 +- Upgraded CI to ubuntu-24.04
74 +- Extracted billing module
75 +- Removed /v1/auth (breaking)
76 +```
77 +
78 +**✅ House style:**
79 +
80 +```markdown
81 +## Acme 2.8.0 — 2026-08-05
82 +
83 +> **Breaking:** The legacy `/v1/auth` endpoint is removed. Switch integrations to `/v2/auth` before upgrading.
84 +
85 +Faster searches and single sign-on headline this release.
86 +
87 +### Added
88 +- Sign in with your company account: SAML single sign-on is now available on all Team plans.
89 +
90 +### Improved
91 +- Search is dramatically faster — most queries now return in about 0.2 seconds instead of nearly a second.
92 +
93 +### Fixed
94 +- Fixed a crash when opening a profile that has no avatar.
95 +```
96 +
97 +Note: the CI upgrade and billing refactor are correctly absent (no user impact).
added tools/validate_skills.py +144 −0
@@ -0,0 +1,144 @@
1 +#!/usr/bin/env python3
2 +# Author: Simon-Pierre Boucher
3 +# Contact: contact@spboucher.ai
4 +#
5 +# validate_skills.py — deterministic linter enforcing the Sharp Skill Checklist
6 +# (see RESEARCH-SYNTHESIS.md) on every skill in this repository. Stdlib only.
7 +#
8 +# Usage: python3 tools/validate_skills.py [repo_root]
9 +# Exit codes: 0 = all skills pass, 1 = at least one failure, 2 = usage error.
10 +
11 +import re
12 +import sys
13 +from pathlib import Path
14 +
15 +# Spec limits from Anthropic's Agent Skills documentation.
16 +NAME_MAX = 64
17 +DESC_MAX = 1024
18 +BODY_MAX_LINES = 500
19 +
20 +NAME_RE = re.compile(r"^[a-z0-9-]+$")
21 +# Reserved words are forbidden anywhere in the skill name per the spec.
22 +RESERVED = ("anthropic", "claude")
23 +
24 +HEADER_AUTHOR = "Author: Simon-Pierre Boucher"
25 +HEADER_CONTACT = "Contact: contact@spboucher.ai"
26 +
27 +# First/second-person openings that indicate a wrong point of view in a
28 +# description, which the docs warn causes discovery problems.
29 +BAD_POV = re.compile(r"\b(I can|I will|you can use this)\b", re.IGNORECASE)
30 +
31 +MD_LINK = re.compile(r"\]\(([^)#>][^)]*)\)")
32 +
33 +
34 +def parse_frontmatter(text):
35 + """Return (fields dict, body) or (None, text) if no frontmatter."""
36 + if not text.startswith("---\n"):
37 + return None, text
38 + end = text.find("\n---", 4)
39 + if end == -1:
40 + return None, text
41 + fields = {}
42 + for line in text[4:end].splitlines():
43 + if ":" in line and not line.startswith((" ", "\t", "#")):
44 + k, v = line.split(":", 1)
45 + fields[k.strip()] = v.strip().strip("\"'")
46 + return fields, text[end + 4:]
47 +
48 +
49 +def check_header(path, text):
50 + """Every project file must carry the author header near the top."""
51 + head = "\n".join(text.splitlines()[:12])
52 + return HEADER_AUTHOR in head and HEADER_CONTACT in head
53 +
54 +
55 +def lint_skill(skill_md, errors):
56 + text = skill_md.read_text(encoding="utf-8")
57 + rel = skill_md
58 + fm, body = parse_frontmatter(text)
59 +
60 + if fm is None:
61 + errors.append(f"{rel}: missing or unterminated YAML frontmatter")
62 + return
63 +
64 + name = fm.get("name", "")
65 + desc = fm.get("description", "")
66 +
67 + if not name:
68 + errors.append(f"{rel}: frontmatter missing 'name'")
69 + else:
70 + if len(name) > NAME_MAX:
71 + errors.append(f"{rel}: name exceeds {NAME_MAX} chars")
72 + if not NAME_RE.match(name):
73 + errors.append(f"{rel}: name must be lowercase letters/numbers/hyphens")
74 + if any(w in name for w in RESERVED):
75 + errors.append(f"{rel}: name contains a reserved word {RESERVED}")
76 + # CLAUDE.md mandates 'skill-N-<name>' folders for the example skills;
77 + # the frontmatter name must match the folder minus that prefix.
78 + folder = re.sub(r"^skill-\d+-", "", skill_md.parent.name)
79 + if name != folder:
80 + errors.append(f"{rel}: name '{name}' != folder '{folder}'")
81 +
82 + if not desc:
83 + errors.append(f"{rel}: frontmatter missing 'description'")
84 + else:
85 + if len(desc) > DESC_MAX:
86 + errors.append(f"{rel}: description exceeds {DESC_MAX} chars ({len(desc)})")
87 + if "<" in desc and ">" in desc:
88 + errors.append(f"{rel}: description may contain XML tags")
89 + if "Use when" not in desc and "use when" not in desc:
90 + errors.append(f"{rel}: description lacks a 'Use when …' trigger clause")
91 + if "Do not use" not in desc and "do not use" not in desc:
92 + errors.append(f"{rel}: description lacks a 'Do not use for …' boundary")
93 + if BAD_POV.search(desc):
94 + errors.append(f"{rel}: description not in third person")
95 +
96 + if not check_header(skill_md.parent / "SKILL.md", body):
97 + errors.append(f"{rel}: author header missing after frontmatter")
98 +
99 + n_lines = len(body.splitlines())
100 + if n_lines > BODY_MAX_LINES:
101 + errors.append(f"{rel}: body has {n_lines} lines (max {BODY_MAX_LINES})")
102 +
103 + for target in MD_LINK.findall(body):
104 + if target.startswith(("http://", "https://", "mailto:")):
105 + continue
106 + if "\\" in target:
107 + errors.append(f"{rel}: backslash path in link '{target}'")
108 + continue
109 + if not (skill_md.parent / target).exists():
110 + errors.append(f"{rel}: broken reference link '{target}'")
111 +
112 + for sub in skill_md.parent.rglob("*"):
113 + if sub.is_file() and sub != skill_md and sub.suffix in (".md", ".py", ".sh"):
114 + if not check_header(sub, sub.read_text(encoding="utf-8", errors="replace")):
115 + errors.append(f"{sub}: author header missing")
116 +
117 +
118 +def main():
119 + root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(".")
120 + if not root.is_dir():
121 + print(f"error: not a directory: {root}", file=sys.stderr)
122 + sys.exit(2)
123 +
124 + skill_files = sorted(root.rglob("SKILL.md"))
125 + if not skill_files:
126 + print(f"error: no SKILL.md files found under {root}", file=sys.stderr)
127 + sys.exit(2)
128 +
129 + errors = []
130 + for skill_md in skill_files:
131 + lint_skill(skill_md, errors)
132 +
133 + print(f"checked {len(skill_files)} skill(s)")
134 + if errors:
135 + for e in errors:
136 + print(f"FAIL {e}")
137 + print(f"{len(errors)} failure(s)")
138 + sys.exit(1)
139 + print("all checks passed")
140 + sys.exit(0)
141 +
142 +
143 +if __name__ == "__main__":
144 + main()
added validation/fixtures/customers-profile.md +31 −0
@@ -0,0 +1,31 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Data Profile: customers.csv
7 +
8 +**Verdict:** 🟡 Usable with caveats — 3 issue(s) found
9 +
10 +## Overview
11 +| Metric | Value |
12 +|---|---|
13 +| Rows profiled | 8 |
14 +| Columns | 6 |
15 +| Duplicate rows | 1 |
16 +| Ragged rows | 0 |
17 +
18 +## Columns
19 +| Column | Type | Nulls | Unique | Stats |
20 +|---|---|---|---|---|
21 +| id | integer | 0 | 7 | min=1.0, max=8.0, mean=4.375, median=4.5 |
22 +| name | string | 0 | 7 | top: Eve (2), Alice (1), Bob (1), Carol (1), Dan (1) |
23 +| age | integer | 1 | 6 | min=27.0, max=52.0, mean=37.0, median=38.0 |
24 +| city | string | 1 | 3 | top: Montreal (4), Toronto (2), Vancouver (1) |
25 +| signup_date | date | 0 | 7 | — |
26 +| score | float | 0 | 7 | min=60.2, max=999.9, mean=309.5875, median=84.9 |
27 +
28 +## Issues
29 +- `age` has 1 missing values (12.5%)
30 +- `city` has 1 missing values (12.5%)
31 +- 1 duplicate rows
added validation/fixtures/customers.csv +9 −0
@@ -0,0 +1,9 @@
1 +id,name,age,city,signup_date,score
2 +1,Alice,34,Montreal,2025-01-15,88.5
3 +2,Bob,29,Toronto,2025-02-20,92.1
4 +3,Carol,,Montreal,2025-03-05,75.0
5 +4,Dan,41,Vancouver,2025-03-18,60.2
6 +5,Eve,38,Montreal,2025-04-02,999.9
7 +5,Eve,38,Montreal,2025-04-02,999.9
8 +7,Grace,27,Toronto,2025-05-11,81.3
9 +8,Heidi,52,,2025-06-30,79.8
added validation/fixtures/headeronly.csv +1 −0
@@ -0,0 +1 @@
1 +col1,col2
added validation/fixtures/ragged.csv +3 −0
@@ -0,0 +1,3 @@
1 +a,b,c
2 +1,2
3 +3,4,5,6
added writing-skills/README.md +43 −0
@@ -0,0 +1,43 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# writing-skills — Writing Skill Collection
7 +
8 +**Author:** Simon-Pierre Boucher
9 +**Contact:** contact@spboucher.ai
10 +
11 +Ten ultra-sharp stylistic writing skills following the method in
12 +[../RESEARCH-SYNTHESIS.md](../RESEARCH-SYNTHESIS.md). As stylistic skills,
13 +every one carries ✅/❌ (do/never) example pairs, boundary conditions, and a
14 +self-review workflow. User-facing release notes remain in
15 +[../skill-2-writing-release-notes/](../skill-2-writing-release-notes/).
16 +All pass `python3 ../tools/validate_skills.py`.
17 +
18 +## The collection and its boundaries
19 +
20 +**Technical writing**
21 +| Skill | Handles | Explicitly does NOT handle |
22 +|---|---|---|
23 +| `writing-technical-documentation` | READMEs, architecture docs, runbooks, ADRs | API reference → `writing-api-documentation`; tutorials; release notes |
24 +| `writing-api-documentation` | endpoint/SDK reference, runnable examples, error tables | conceptual guides; designing the API itself |
25 +| `writing-tutorials` | step-by-step learning content with checkpoints | reference docs; conceptual overviews |
26 +| `writing-blog-posts` | technical posts: one idea, hook, specifics, skimmable | documentation; tutorials; marketing copy |
27 +| `writing-agent-skills` | authoring SKILL.md files per the Sharp Skill method | prompts/system prompts; CLAUDE.md; MCP tool definitions |
28 +
29 +**Business writing**
30 +| Skill | Handles | Explicitly does NOT handle |
31 +|---|---|---|
32 +| `writing-professional-emails` | subject-line asks, BLUF body, next steps | marketing campaigns; chat; formal letters |
33 +| `writing-executive-summaries` | BLUF, one page, numbers over adjectives, recommendation | full reports; blog posts; meeting notes |
34 +| `writing-proposals` | their-problem-first, scope + exclusions, options, next step | internal exec summaries; contracts; grants |
35 +| `writing-meeting-notes` | decisions/actions first, owner+deadline, 24h distribution | exec summaries; status reports; transcription |
36 +| `editing-and-proofreading` | fixed four-pass revision: structure → paragraphs → sentences → mechanics | writing from scratch; translation; code review |
37 +
38 +## Shared conventions
39 +
40 +- Description = WHAT + "Use when …" (literal phrases) + "Do not use for …"
41 +- House rules are numbered, each with a ✅/❌ pair
42 +- Every workflow ends with a self-review pass against the rules
43 +- `SKILL.md` <150 lines; extended before/after examples in `references/examples.md` (with TOC)
added writing-skills/editing-and-proofreading/SKILL.md +64 −0
@@ -0,0 +1,64 @@
1 +---
2 +name: editing-and-proofreading
3 +description: Edits and proofreads existing prose through a fixed four-pass method — structure, then paragraphs, then sentences, then mechanics — cutting 10-20% while preserving the author's voice. Use when the user asks to edit, revise, proofread, tighten, polish, or shorten a draft, document, article, or any written text they provide. Do not use for writing new content from scratch, translation, or code review.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Editing and Proofreading
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** revising an existing draft the user provides — documents, articles, reports, web copy, any prose.
15 +- **Do NOT use for:** writing new content from scratch (use the matching writing skill), translation, or code review. If the "draft" is three bullet points, that's writing, not editing.
16 +
17 +## House rules
18 +
19 +1. **Never edit in one pass.** Four passes, in fixed order — structure, paragraphs, sentences, mechanics. Fixing commas in a paragraph you'll later delete is wasted work.
20 + - ✅ Pass 1 moves section 4 before section 2; commas wait for pass 4.
21 + - ❌ Perfecting the wording of the intro before checking whether the intro should exist.
22 +
23 +2. **Pass 1 — structure.** Does the order serve the reader? Lead with what the reader needs; every section earns its place or goes.
24 + - ✅ "The recommendation is in §5; it becomes §1. Background §1–2 compress into one paragraph."
25 + - ❌ Keeping the author's discovery order because it's already written.
26 +
27 +3. **Pass 2 — paragraphs.** One idea per paragraph; first sentences alone should carry the whole argument (the first-sentence skim test).
28 + - ✅ Reading only first sentences yields a coherent summary.
29 + - ❌ Paragraphs that change topic midway, or open with throat-clearing ("It is worth noting that…").
30 +
31 +4. **Pass 3 — sentences.** Active voice; verbs over nominalizations; cut hedges and intensifiers.
32 + - ✅ "The team missed the deadline because the API changed."
33 + - ❌ "The deadline was unfortunately not met due to the occurrence of changes in the API." ("occurrence of changes" → "changes"; who missed it?)
34 +
35 +5. **Cut 10–20% on principle.** Every draft is padded; if nothing feels cuttable, the cuts are hiding in redundant pairs ("each and every"), double qualifiers, and repeated points.
36 + - ✅ 1,000 words in → ~850 out, same content.
37 + - ❌ Returning the same length with synonyms swapped.
38 +
39 +6. **Preserve the author's voice.** Edit for clarity, not to your taste. Judgment calls (tone, humor, a deliberate fragment) get flagged as comments, not silently rewritten.
40 + - ✅ "[flag] This joke may not land with the exec audience — keep or cut?"
41 + - ❌ Rewriting a casual post into corporate neutral.
42 +
43 +7. **Pass 4 — mechanics, with verification.** Spelling, punctuation, and consistency (terms, capitalization, number formats) — and verify every number, name, and link against the source or flag it as unverified.
44 + - ✅ "Total says $84k here, $48k in the table — which is correct?"
45 + - ❌ Proofreading around a number that contradicts itself.
46 +
47 +8. **Read the key passages aloud.** Openings, transitions, and anything rhythmic: if you stumble reading it, the reader will too.
48 +
49 +## Workflow
50 +
51 +1. Ask/infer the target audience and any length limit; note the author's register (formal/casual) to preserve it.
52 +2. Pass 1 (structure): reorder/cut sections; confirm big moves with the user if they change meaning or scope.
53 +3. Pass 2 (paragraphs): one idea each; run the first-sentence skim test.
54 +4. Pass 3 (sentences): active voice, strong verbs, cut hedges/fillers; track the word count toward the 10–20% cut.
55 +5. Pass 4 (mechanics): spelling, punctuation, consistency; verify or flag every number, name, and link.
56 +6. Deliver: the edited text, plus a short list of judgment-call flags and the before/after word count. Do not list every mechanical fix.
57 +
58 +## Edge cases
59 +- **The draft needs rewriting, not editing** (structure unsalvageable) → say so, propose the new outline, and get agreement before rewriting.
60 +- **User asks for proofreading only** → run pass 4 alone; note (once, briefly) if a structural problem is glaring, then respect the request.
61 +- **Text in a style you'd never choose** → their voice wins; edit only what impedes the reader.
62 +
63 +## References
64 +Extended before/after examples: see [references/examples.md](references/examples.md).
added writing-skills/editing-and-proofreading/references/examples.md +101 −0
@@ -0,0 +1,101 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Examples — Editing and Proofreading House Style
7 +
8 +## Contents
9 +- Sentence-level pairs (pass 3)
10 +- The hedge/intensifier cut list
11 +- Complete worked example (four passes on one paragraph)
12 +- Voice-preservation pairs
13 +- Gotchas
14 +
15 +## Sentence-level pairs (pass 3)
16 +
17 +| ❌ Before | ✅ After |
18 +|---|---|
19 +| A decision was made by the committee to postpone the launch. | The committee postponed the launch. |
20 +| We conducted an analysis of the results. | We analyzed the results. |
21 +| It is important to note that costs have seen an increase. | Costs rose 12%. |
22 +| There are three factors that determine pricing. | Three factors determine pricing. |
23 +| The implementation of the migration was completed successfully. | The migration is done. |
24 +
25 +## The hedge/intensifier cut list
26 +
27 +Cut on sight unless load-bearing: *very, really, quite, rather, somewhat,
28 +fairly, basically, essentially, actually, arguably, it seems that, I think
29 +that* (in formal prose), *in order to* (→ to), *due to the fact that* (→
30 +because), *at this point in time* (→ now), *each and every* (→ every).
31 +
32 +## Complete worked example — four passes on one paragraph
33 +
34 +**Original (87 words):**
35 +
36 +> It is worth noting that, over the course of the past several months, our
37 +> team has been engaged in the process of evaluating a number of different
38 +> options with respect to our hosting infrastructure. Basically, after a
39 +> really thorough analysis was conducted, it was determined by the team that
40 +> a migration to the new provider would potentially be capable of delivering
41 +> significant cost savings. The security implications were also looked at.
42 +> It is our belief that this migration should be undertaken in Q4.
43 +
44 +- **Pass 1 (structure):** recommendation last → moves first.
45 +- **Pass 2 (paragraph):** one idea = "migrate in Q4 because it saves money
46 + and is secure"; evaluation history is background, compresses to a clause.
47 +- **Pass 3 (sentences):** passives → active; "engaged in the process of
48 + evaluating" → "evaluated"; cut *basically, really, potentially, it is our
49 + belief that*.
50 +- **Pass 4 (mechanics):** "significant cost savings" has no number — flag:
51 + `[verify: how much?]`.
52 +
53 +**Edited (34 words, −61%):**
54 +
55 +> We should migrate to the new hosting provider in Q4. After evaluating four
56 +> options over three months, the team found it saves [verify: how much?] per
57 +> year and passed our security review.
58 +
59 +## Paragraph-level example (pass 2 — first-sentence skim test)
60 +
61 +**❌ Fails the skim test** (first sentences: "There are several things to
62 +consider." / "Another point is worth mentioning." / "Finally, some context."):
63 +the skim yields nothing.
64 +
65 +**✅ Passes** — first sentences alone tell the story:
66 +
67 +> Migrating in Q4 costs less than waiting. […]
68 +> The security review found no blockers. […]
69 +> The one real risk is the December code freeze. […]
70 +
71 +If the skim summary and the document's argument differ, either the first
72 +sentences are weak (fix them) or the paragraphs argue something the document
73 +doesn't claim (restructure — back to pass 1).
74 +
75 +## Voice-preservation pairs
76 +
77 +**❌ Voice flattened (over-edit):**
78 +- Author: "Look — nobody wakes up excited about expense reports."
79 +- Editor rewrite: "Expense reporting is generally considered tedious."
80 +
81 +**✅ Voice kept, clarity edited:**
82 +- Author: "Look — nobody wakes up excited about expense reports, and that's
83 + precisely the exact reason we automated every single part of it end-to-end."
84 +- Edit: "Look — nobody wakes up excited about expense reports. That's why we
85 + automated all of it." *(rhythm kept, redundancy cut)*
86 +
87 +## Gotchas
88 +
89 +- **Editing quotes** — never touch quoted speech beyond bracketed
90 + clarifications; flag suspected transcription errors instead.
91 +- **The 10–20% cut on already-tight text** — if a text genuinely resists
92 + cutting, say so; the target is a heuristic, not a quota to fake.
93 +- **"Passive voice is always wrong"** — no: "The suspect was arrested"
94 + beats "Officers arrested the suspect" when the actor is irrelevant. Cut
95 + passives that hide accountability, keep passives that focus correctly.
96 +- **Consistency traps for pass 4:** e-mail/email, USD 5k/$5,000/5 000 $,
97 + Oxford comma on/off, capitalized job titles. Pick the file's dominant
98 + convention and enforce it; don't import your own.
99 +- **Grammar-checker artifacts** — mechanical tools flag long correct
100 + sentences and miss wrong numbers. Pass 4 is human judgment plus
101 + verification, not a spell-check transcript.
added writing-skills/writing-agent-skills/SKILL.md +59 −0
@@ -0,0 +1,59 @@
1 +---
2 +name: writing-agent-skills
3 +description: Authors ultra-sharp SKILL.md skills for AI agents using the Sharp Skill method — trigger-optimized descriptions, progressive disclosure, validation-loop workflows, and eval-first testing. Use when the user asks to create, write, improve, or review a skill, a SKILL.md file, a slash command, or asks why a skill is not triggering. Do not use for writing prompts or system prompts, CLAUDE.md project instructions, or MCP tool definitions.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing Agent Skills
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** creating new skills, sharpening existing SKILL.md files, fixing skills that under- or over-trigger, reviewing a skill against the checklist.
15 +- **Do NOT use for:** prompts/system prompts, CLAUDE.md instructions, MCP tool schemas, or agent definitions — different formats with different rules.
16 +
17 +## House rules
18 +
19 +1. **The description is the trigger — write it as WHAT + WHEN + NOT.** Third person, directive, ≤1024 chars, with the literal phrases a user would type.
20 + - ✅ "Profiles CSV files and produces a data-quality report. Use when the user asks to profile, audit, or inspect a CSV… Do not use for converting or editing CSV data."
21 + - ❌ "Helps with CSV files."
22 +
23 +2. **One default per decision, with an escape hatch.**
24 + - ✅ "Use pdfplumber for extraction; for scanned PDFs use OCR (pytesseract) instead."
25 + - ❌ "You can use pypdf, pdfplumber, PyMuPDF, or…"
26 +
27 +3. **Progressive disclosure:** body under 500 lines (aim far lower); depth in `references/` linked one level deep; a table of contents in any reference over 100 lines; scripts for deterministic work with intent explicit ("Run scripts/x.py" vs "See scripts/x.py for the algorithm").
28 +
29 +4. **Workflows end in validation.** Numbered steps; the last step checks the output and loops back on failure.
30 + - ✅ "Step 5: re-parse the output file; if invalid, fix and repeat step 3."
31 + - ❌ A list of steps that ends at "deliver".
32 +
33 +5. **Assume the agent is smart.** Cut anything a strong model already knows; every paragraph must justify its token cost.
34 + - ✅ Straight to the library call and the project-specific rule.
35 + - ❌ "PDF (Portable Document Format) files are a common format that…"
36 +
37 +6. **Specify the output completely:** structure, naming, destination, and input→output examples wherever style matters.
38 +
39 +7. **Boundaries are pairwise-exclusive.** Name the nearest neighboring intent and exclude it explicitly, so no request plausibly fires two skills.
40 +
41 +8. **Eval-first:** before shipping, write 3+ test prompts — at least one that must NOT trigger — and run/simulate them.
42 +
43 +## Workflow
44 +
45 +1. Define the skill's single intent in one sentence; name it in gerund form (lowercase, hyphens, ≤64 chars, no reserved words).
46 +2. Draft the description: WHAT clause + "Use when …" with literal trigger phrases + "Do not use for …" naming the nearest non-target intent.
47 +3. Write the body: when/not-when, rules or quick reference (one default per decision), numbered workflow ending in validation, edge cases, one-level-deep references.
48 +4. Move anything long into `references/` (TOC if over 100 lines); write scripts for deterministic steps and make execute-vs-read intent explicit.
49 +5. Write 3+ test prompts (≥1 negative); simulate each against the description; adjust wording until positives fire and negatives don't.
50 +6. Final gate: run the Sharp Skill Checklist (see references) line by line; fix every failure and re-run before delivering.
51 +
52 +## Edge cases
53 +- **Skill under-triggers** → rewrite the description with the user's literal phrasing; check the collection's total description budget (Claude Code truncates the listing) — shorten siblings if needed.
54 +- **Skill over-triggers** → add the "Do not use for" clause naming the intents it's stealing; move generic keywords out of the description.
55 +- **Two skills fight over requests** → redraw the boundary in BOTH descriptions so they exclude each other by name.
56 +- **Skill too long** → split by domain into reference files, or split into two skills if it has two intents.
57 +
58 +## References
59 +The full checklist, description formula, and worked examples: see [references/examples.md](references/examples.md).
added writing-skills/writing-agent-skills/references/examples.md +114 −0
@@ -0,0 +1,114 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Examples — Writing Agent Skills
7 +
8 +## Contents
9 +- The description formula, worked
10 +- Complete worked example: a skill sharpened (bad → house style)
11 +- The Sharp Skill Checklist (final gate)
12 +- Trigger-eval examples
13 +- Gotchas
14 +
15 +## The description formula, worked
16 +
17 +`[WHAT it does, one clause with key nouns]. Use when [literal user phrases,
18 +file extensions, key terms]. Do not use for [nearest neighboring intent].`
19 +
20 +✅ "Writes user-facing release notes and changelog entries in the house
21 +style — benefit-first, grouped by Added/Improved/Fixed. Use when the user
22 +asks to write release notes, a changelog entry, a what's-new section, or a
23 +version announcement. Do not use for git commit messages, PR descriptions,
24 +or API reference documentation."
25 +
26 +❌ "A skill for helping with release-related writing tasks." (no triggers,
27 +no boundary, would also under-trigger — nothing matches what users type)
28 +
29 +## Complete worked example: a skill sharpened
30 +
31 +**❌ Before:**
32 +
33 +```markdown
34 +---
35 +name: json-helper
36 +description: Helps with JSON.
37 +---
38 +
39 +# JSON Helper
40 +
41 +This skill helps you work with JSON files. JSON (JavaScript Object
42 +Notation) is a popular data format. You can use many tools to work with
43 +it, such as Python's json module, jq, or JavaScript. Depending on the
44 +situation, choose the best tool and be careful with edge cases.
45 +```
46 +
47 +Failures: vague name; description has no WHAT/WHEN/NOT; explains what JSON
48 +is; offers a menu with no default; "be careful" is not a procedure; no
49 +workflow, no validation, no output spec.
50 +
51 +**✅ After (house style):**
52 +
53 +```markdown
54 +---
55 +name: processing-json
56 +description: Creates, reads, edits, validates, and queries JSON and JSON
57 + Lines files. Use when the user asks to fix, format, merge, or query a
58 + .json or .jsonl file, or mentions invalid JSON. Do not use for YAML or
59 + TOML config files or for designing APIs.
60 +---
61 +
62 +# Processing JSON
63 +
64 +## Quick reference
65 +Read/write: stdlib `json` — `json.dump(data, f, indent=2,
66 +ensure_ascii=False)`. Large-file queries: `jq` (must be installed).
67 +
68 +## Workflow
69 +1. Parse the input; on failure report the parser's line/column verbatim.
70 +2. Apply the change in Python (never regex-edit JSON).
71 +3. Write atomically (temp file, then rename).
72 +4. Validate: re-parse the written file; if it fails, fix and repeat step 2.
73 +
74 +## Edge cases
75 +- Trailing commas/NaN: stdlib rejects them — report, don't "repair" silently.
76 +```
77 +
78 +## The Sharp Skill Checklist (final gate)
79 +
80 +Triggering: description has WHAT + "Use when" + "Do not use"; literal user
81 +phrases; third person; ≤1024 chars; name gerund-form, lowercase-hyphens.
82 +Body: <500 lines; no known-content; one default per decision; numbered
83 +workflow ending in validation; output fully specified; concrete examples;
84 +edge cases addressed. Resources: references one level deep; TOC over 100
85 +lines; execute-vs-read explicit; scripts handle own errors; dependencies
86 +stated. Validation: 3+ test prompts incl. ≥1 negative, run before shipping.
87 +
88 +## Trigger-eval examples
89 +
90 +For `processing-json` above:
91 +
92 +| Prompt | Expected | Why |
93 +|---|---|---|
94 +| "This config.json won't parse, fix it" | TRIGGER | "fix", ".json", "invalid JSON" |
95 +| "Merge these two .jsonl exports" | TRIGGER | "merge", ".jsonl" |
96 +| "Convert this YAML to nicer formatting" | NO TRIGGER | YAML excluded by boundary |
97 +
98 +A negative prompt that triggers means the description keywords are too
99 +broad — tighten nouns, add the exclusion by name.
100 +
101 +## Gotchas
102 +
103 +- **Descriptions are a shared budget:** in Claude Code all skill listings
104 + share a character budget and overflow is dropped silently — a bloated
105 + description can knock *other* skills out of context.
106 +- **First person kills discovery** ("I can help you…") — the description is
107 + injected into a system prompt written in third person.
108 +- **Nested references get partially read** (`head -100`); keep every
109 + reference file linked directly from SKILL.md.
110 +- **Time-sensitive facts** ("before v2, do X") rot; move legacy notes into a
111 + collapsed "old patterns" block or delete them.
112 +- **A skill that documents imagined problems** stays unused; run the task
113 + without the skill first, and write down only what the agent actually
114 + missed — that gap is the skill.
added writing-skills/writing-api-documentation/SKILL.md +63 −0
@@ -0,0 +1,63 @@
1 +---
2 +name: writing-api-documentation
3 +description: Writes API endpoint and SDK reference documentation with complete per-endpoint entries, copy-paste runnable examples, and errors documented as thoroughly as successes. Use when the user asks to document an API, write endpoint reference docs, an API reference page, SDK docs, or improve OpenAPI/Swagger descriptions. Do not use for conceptual guides and READMEs, tutorials, or designing the API itself — separate skills cover those.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing API Documentation
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** endpoint reference pages, SDK method reference, OpenAPI descriptions, error catalogs, API changelogs.
15 +- **Do NOT use for:** conceptual guides and READMEs (writing-technical-documentation), step-by-step tutorials (writing-tutorials), or designing the API's shape (backend-skills/designing-rest-apis).
16 +
17 +## House rules
18 +
19 +1. **Auth section first.** It's the #1 lookup; put "Authentication" before any endpoint.
20 + - ✅ "All requests require `Authorization: Bearer sk_live_…`. Get a key at Settings → API."
21 + - ❌ Auth explained under a FAQ at the bottom.
22 +
23 +2. **Every endpoint gets the full block, no exceptions:** method + path, one-line purpose, auth requirement, parameter table, request example, response example, error table.
24 + - ✅ `POST /v1/invoices` documented with all seven parts.
25 + - ❌ "Works like the orders endpoint but for invoices."
26 +
27 +3. **Examples are copy-paste runnable.** curl by default, complete headers, real-looking fake data.
28 + - ✅ `curl https://api.acme.com/v1/invoices -H "Authorization: Bearer sk_test_51H..." -d amount=1999 -d currency=usd`
29 + - ❌ `curl <endpoint> -d foo=bar`
30 +
31 +4. **Parameter tables with five columns:** name / type / required / default / constraints.
32 + - ✅ `| amount | integer | yes | — | cents, 50–999999 |`
33 + - ❌ "Takes an amount and an optional currency."
34 +
35 +5. **Document errors as thoroughly as successes.** Every error code: when it happens and how the caller fixes it.
36 + - ✅ `| 402 | card_declined | Card was declined | Ask the customer for another card |`
37 + - ❌ "Returns standard HTTP error codes."
38 +
39 +6. **Show the response, not a description of it.** Full JSON body with realistic values, fields explained inline or in a table.
40 + - ✅ A complete `200` JSON example plus a field table.
41 + - ❌ "Returns the created invoice object."
42 +
43 +7. **Breaking changes go in a changelog with dates and migration notes.**
44 + - ✅ "2026-06-01 — `total` renamed to `amount_total`. Both returned until 2026-09-01."
45 + - ❌ Silent renames discovered in production.
46 +
47 +8. **OpenAPI is the source of truth for shapes; prose carries concepts** (pagination, idempotency, rate limits) once, linked everywhere.
48 +
49 +## Workflow
50 +
51 +1. Inventory: list every endpoint/method to document; flag undocumented errors by reading the handler code if available.
52 +2. Write the shared sections once: Authentication (first), pagination, idempotency, rate limits, error format.
53 +3. For each endpoint, fill the full block from rule 2 — write the error table before polishing the happy path.
54 +4. Run every example against a test environment (or validate against the OpenAPI spec if no environment exists) and paste real output as the response example.
55 +5. Self-review: scan for "foo/bar" data, missing defaults in parameter tables, and error tables with fewer than two entries; fix before delivering.
56 +
57 +## Edge cases
58 +- **No test environment to run examples** → validate request/response examples against the OpenAPI schema and mark them "generated from spec".
59 +- **Endpoint with side effects** (payments, deletion) → example uses test-mode keys/sandbox data and says so.
60 +- **Huge API** → document by resource, one page per resource, identical block structure; never summarize "similar" endpoints together.
61 +
62 +## References
63 +Extended before/after examples: see [references/examples.md](references/examples.md).
added writing-skills/writing-api-documentation/references/examples.md +124 −0
@@ -0,0 +1,124 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Examples — API Documentation House Style
7 +
8 +## Contents
9 +- Complete worked example: one endpoint (bad → house style)
10 +- Error-table examples
11 +- Shared-concepts section pattern
12 +- Changelog entries
13 +- Gotchas
14 +
15 +## Complete worked example: one endpoint
16 +
17 +**❌ Before:**
18 +
19 +```markdown
20 +### Create invoice
21 +
22 +Creates an invoice. Send a POST request with the invoice data.
23 +Returns the invoice object, or an error code if something goes wrong.
24 +
25 + curl <api-url>/invoices -d foo=bar
26 +```
27 +
28 +**✅ After (house style):**
29 +
30 +```markdown
31 +### Create an invoice
32 +
33 +`POST /v1/invoices` — creates a draft invoice for a customer.
34 +
35 +Requires: `Authorization: Bearer` with an `invoices:write` key.
36 +
37 +**Parameters**
38 +
39 +| Name | Type | Required | Default | Constraints |
40 +|---|---|---|---|---|
41 +| `customer_id` | string | yes | — | existing customer, `cus_` prefix |
42 +| `amount` | integer | yes | — | cents, 50–999999 |
43 +| `currency` | string | no | `usd` | ISO 4217, lowercase |
44 +| `due_date` | string | no | +30 days | ISO 8601 date, future |
45 +
46 +**Request**
47 +
48 + curl https://api.acme.com/v1/invoices \
49 + -H "Authorization: Bearer sk_test_51HxTmA..." \
50 + -d customer_id=cus_9XKzR2 \
51 + -d amount=1999 \
52 + -d currency=usd
53 +
54 +**Response — 201**
55 +
56 + {
57 + "id": "inv_7GtQpN",
58 + "customer_id": "cus_9XKzR2",
59 + "amount": 1999,
60 + "currency": "usd",
61 + "status": "draft",
62 + "due_date": "2026-09-04",
63 + "created_at": "2026-08-05T14:03:22Z"
64 + }
65 +
66 +**Errors**
67 +
68 +| Status | Code | When | Fix |
69 +|---|---|---|---|
70 +| 400 | `amount_out_of_range` | amount below 50 or above 999999 | adjust the amount |
71 +| 401 | `invalid_api_key` | missing/revoked key | check the key and its prefix (test vs live) |
72 +| 403 | `missing_scope` | key lacks `invoices:write` | create a key with the scope |
73 +| 404 | `customer_not_found` | unknown `customer_id` | create the customer first |
74 +| 429 | `rate_limited` | over 100 req/min | back off per `Retry-After` |
75 +```
76 +
77 +What changed: full seven-part block; five-column parameter table; runnable
78 +curl with realistic fake data; actual JSON response; errors with when + fix.
79 +
80 +## Error-table examples
81 +
82 +✅ Each row actionable:
83 +`| 409 | idempotency_conflict | same Idempotency-Key with a different body | use a new key or resend the original body |`
84 +
85 +❌ Non-actionable:
86 +`| 409 | conflict | conflict occurred | — |`
87 +
88 +## Shared-concepts section pattern
89 +
90 +Write once, link everywhere:
91 +
92 +```markdown
93 +## Pagination
94 +
95 +List endpoints return at most `limit` items (default 20, max 100) and a
96 +`next_cursor`. Pass it as `cursor` to fetch the next page. Cursors expire
97 +after 24 h.
98 +
99 + curl "https://api.acme.com/v1/invoices?limit=50&cursor=eyJpZCI6..."
100 +```
101 +
102 +Every list endpoint then says: "Paginated — see [Pagination](#pagination)."
103 +Never re-explain pagination per endpoint (copies drift).
104 +
105 +## Changelog entries
106 +
107 +✅ "2026-06-01 — **Breaking:** `total` renamed to `amount_total` on invoice
108 +objects. Both fields returned until 2026-09-01; update readers before then."
109 +
110 +❌ "June: minor improvements to invoice responses."
111 +
112 +## Gotchas
113 +
114 +- **"foo/bar" sample data** signals the example was never run; use realistic
115 + prefixed IDs (`cus_`, `inv_`) and plausible amounts.
116 +- **Documenting the spec, not the behavior:** if the server returns fields
117 + the spec omits, the docs are wrong until reconciled — test against the
118 + real API.
119 +- **Error tables copied between endpoints** rot instantly; generate them per
120 + endpoint or verify each row.
121 +- **Auth examples with live-mode keys** get copy-pasted into scripts; always
122 + show `sk_test_` keys.
123 +- **Undocumented defaults** force callers to reverse-engineer; every optional
124 + parameter states its default, even when it's "empty".
added writing-skills/writing-blog-posts/SKILL.md +61 −0
@@ -0,0 +1,61 @@
1 +---
2 +name: writing-blog-posts
3 +description: Writes technical blog posts built on one specific claim, a three-sentence hook, and real code, numbers, and failures. Use when the user asks to write a blog post, an engineering blog article, a technical write-up of a project or incident, or a dev.to/Medium-style post. Do not use for documentation, tutorials, or marketing landing-page copy — separate skills cover those.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing Blog Posts
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** technical blog posts, engineering write-ups, postmortem narratives for a public audience, "how we built X" articles.
15 +- **Do NOT use for:** documentation (writing-technical-documentation), tutorials (writing-tutorials), API reference (writing-api-documentation), or landing-page copy (frontend-skills/creating-landing-pages).
16 +
17 +## House rules
18 +
19 +1. **One idea per post, stated as a specific claim in the title.**
20 + - ✅ "We cut our CI bill 60% by deleting half our integration tests"
21 + - ❌ "Thoughts on testing"
22 +
23 +2. **Hook in the first 3 sentences** — the problem, a surprising fact, or the result. No throat-clearing.
24 + - ✅ "Our CI bill hit $40k/month. The fix wasn't caching or bigger runners — it was deleting 4,000 tests. Here's how we decided which ones."
25 + - ❌ "Testing is a very important topic in software engineering. In this post, I will share some thoughts…"
26 +
27 +3. **Real code, real numbers, real failures.** Specifics earn trust; sanitize secrets, keep the mess.
28 + - ✅ "The first attempt shaved 4% — nowhere near worth the two weeks it took."
29 + - ❌ "After some optimization, performance improved significantly."
30 +
31 +4. **Subheadings tell the story alone.** A skimmer reading only headings gets the arc.
32 + - ✅ "The $40k bill → What the tests actually covered → Deleting with a safety net → Results after 90 days"
33 + - ❌ "Introduction → Background → Discussion → Conclusion"
34 +
35 +5. **Personal experience over generic advice.** "We did X and Y happened" beats "you should X".
36 + - ✅ "We tried contract tests first; they caught 2 of the 17 regressions."
37 + - ❌ "Consider using contract tests, which can catch regressions."
38 +
39 +6. **Honest limitations section** — where this wouldn't work, what you'd do differently.
40 + - ✅ "This only worked because our unit coverage was already strong; with flaky units, deleting integration tests would be reckless."
41 + - ❌ Presenting the approach as universal.
42 +
43 +7. **At most one call-to-action**, at the end.
44 + - ✅ One "we're hiring" or one "try the tool" link.
45 + - ❌ Newsletter box, three product links, and a webinar pitch mid-article.
46 +
47 +## Workflow
48 +
49 +1. Write the claim as one sentence; if it needs "and", split into two posts.
50 +2. Collect the evidence: numbers, code snippets, timelines, failed attempts. No evidence → no post yet.
51 +3. Outline as story-telling subheadings (rule 4); check the headings alone carry the arc.
52 +4. Draft: hook first (rule 2), then the story, then limitations, then the single CTA.
53 +5. Self-review pass: delete every sentence that could appear unchanged in anyone else's post on the topic; verify every number has a source; confirm exactly one CTA. Fix and re-check.
54 +
55 +## Edge cases
56 +- **No hard numbers available** → use concrete qualitative specifics (timeline, error messages, before/after code); never invent or round up.
57 +- **Company-sensitive details** → replace absolute revenue-like numbers with ratios/percentages, and say you did.
58 +- **Post is really a tutorial in disguise** (reader follows steps) → switch to writing-tutorials.
59 +
60 +## References
61 +Extended before/after examples: see [references/examples.md](references/examples.md).
added writing-skills/writing-blog-posts/references/examples.md +102 −0
@@ -0,0 +1,102 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Examples — Blog Post House Style
7 +
8 +## Contents
9 +- Complete worked example: opening (bad → house style)
10 +- Title pairs
11 +- Subheading arcs
12 +- Limitations sections
13 +- Gotchas
14 +
15 +## Complete worked example: opening
16 +
17 +**❌ Before:**
18 +
19 +```markdown
20 +# Some Thoughts on Database Migrations
21 +
22 +Database migrations are one of the most important yet often overlooked
23 +aspects of modern software development. Every developer will eventually
24 +need to deal with them. In today's fast-paced world, it's crucial to have
25 +a robust migration strategy. In this post, I will share some thoughts and
26 +best practices that I have learned over the years.
27 +```
28 +
29 +(Four sentences in: no claim, no evidence, nothing the reader can't predict.)
30 +
31 +**✅ After (house style):**
32 +
33 +```markdown
34 +# A 40-character migration took our API down for 11 minutes
35 +
36 +The migration was one line: `ALTER TABLE orders ADD COLUMN note text`.
37 +It locked 90 million rows on our primary at 14:02 on a Tuesday.
38 +This post reconstructs the incident minute by minute, and shows the
39 +expand-migrate-contract pattern we now enforce in CI so it can't recur.
40 +
41 +## 14:02 — the deploy that looked harmless
42 +...
43 +```
44 +
45 +What changed: title is a specific claim with numbers; hook is the problem +
46 +the surprise in three sentences; timeline promises evidence, not opinions.
47 +
48 +## Title pairs
49 +
50 +- ✅ "Keyset pagination made our deep pages 200× faster" / ❌ "Pagination best practices"
51 +- ✅ "We replaced 14 microservices with a monolith and cut p99 by half" / ❌ "Microservices vs monoliths"
52 +- ✅ "Everything we learned fuzzing our YAML parser for 30 days" / ❌ "An introduction to fuzzing"
53 +
54 +## Subheading arcs
55 +
56 +✅ Story alone in headings:
57 +
58 +```
59 +The bill nobody questioned
60 +What 4,000 integration tests actually tested
61 +The deletion protocol: score, quarantine, delete
62 +What broke (two things) and what didn't (everything else)
63 +90 days later: numbers
64 +Where this would have gone wrong
65 +```
66 +
67 +❌ Template headings: `Introduction / Background / Approach / Results /
68 +Conclusion` — interchangeable with any post ever written.
69 +
70 +## Limitations sections
71 +
72 +✅ "This worked because (1) unit coverage was 85%+ before we started,
73 +(2) our services share one language, so contract drift is rare, and
74 +(3) we could tolerate a staging-only canary for two weeks. Team with a
75 +polyglot stack or thin unit coverage: the quarantine step alone is still
76 +worth stealing; the mass deletion is not."
77 +
78 +❌ "Of course, your mileage may vary." (says nothing)
79 +
80 +## Hook patterns (three that work)
81 +
82 +- **The problem:** "Our CI bill hit $40k/month and nobody could say why."
83 +- **The surprising fact:** "Deleting tests made our deploys safer. Here's
84 + the data."
85 +- **The result:** "p99 went from 840 ms to 210 ms with a one-line change —
86 + after three weeks of wrong turns. The wrong turns are the useful part."
87 +
88 +All three commit to specifics in sentence one; none begin with the topic's
89 +importance.
90 +
91 +## Gotchas
92 +
93 +- **The generic-sentence test is brutal but works:** if a sentence could
94 + close any post on the topic ("testing is a journey, not a destination"),
95 + it adds nothing — delete it.
96 +- **Rounded-up numbers get called out** in comments ("60%" that's really
97 + 48% costs all credibility); use the real figure.
98 +- **Burying the result** to build suspense loses skimmers; the hook may
99 + reveal the ending — the story is *how*, not *whether*.
100 +- **Code screenshots** aren't searchable or copyable; always fenced blocks.
101 +- **Two ideas in one post** each get half the depth; the split versions
102 + both outperform the original.
added writing-skills/writing-executive-summaries/SKILL.md +61 −0
@@ -0,0 +1,61 @@
1 +---
2 +name: writing-executive-summaries
3 +description: Writes one-page executive summaries in a bottom-line-up-front house style — recommendation in sentence one, quantified impact, options with a clear pick. Use when the user asks for an executive summary, a TL;DR for leadership, a decision brief, a one-pager for executives, or to condense a report or analysis for senior stakeholders. Do not use for full reports or analyses, blog posts, or meeting notes.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing Executive Summaries
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** decision briefs, one-pagers, leadership TL;DRs, the summary section at the top of a longer document.
15 +- **Do NOT use for:** the full report itself, blog posts, or meeting minutes (a separate writing-meeting-notes skill covers those). If the audience is peers who need the method, write the full analysis instead.
16 +
17 +## House rules
18 +
19 +1. **BLUF — bottom line up front.** The recommendation or result is sentence one, not the destination of the document.
20 + - ✅ "Recommendation: migrate to vendor B, saving $210k/yr with a 6-week transition."
21 + - ❌ "This document examines our current vendor landscape and evaluates several options…"
22 +
23 +2. **Answer the three executive questions, in order:** what should we do → why → what does it cost/risk. Nothing else earns space on the page.
24 + - ✅ Sections: Recommendation / Rationale / Cost & Risk.
25 + - ❌ Sections: Background / Methodology / Stakeholders consulted / Timeline of events.
26 +
27 +3. **One page maximum.** If it doesn't fit, the summary isn't done; cut until it fits.
28 + - ✅ 250–400 words, 3–5 headed blocks.
29 + - ❌ "Executive summary (pages 1–4)".
30 +
31 +4. **Numbers over adjectives, with confidence named.** Quantify impact; state how sure you are.
32 + - ✅ "Cuts onboarding time from 12 days to 4 (measured across 40 accounts; high confidence)."
33 + - ❌ "Significantly improves onboarding efficiency."
34 +
35 +5. **Options come with a pick.** Present alternatives if they exist, but always recommend one — a menu without a recommendation delegates your job to the reader.
36 + - ✅ "Option A (recommended): … Option B: … We recommend A because the risk in B is unbounded."
37 + - ❌ "Options A, B and C are presented for consideration."
38 +
39 +6. **Risks stated plainly, each with a mitigation.** Naming risks builds credibility; hiding them destroys it in the Q&A.
40 + - ✅ "Risk: key engineer departure mid-migration → mitigation: 2-week shadow period, docs by day 5."
41 + - ❌ Omitting risks, or "some challenges may arise."
42 +
43 +7. **No methodology before conclusions.** How you got the answer goes in an appendix or the full report.
44 + - ✅ "Details and methodology: appendix B."
45 + - ❌ Opening with survey design and sample sizes.
46 +
47 +## Workflow
48 +
49 +1. Write the recommendation sentence first, alone. If you can't, the analysis isn't finished — stop and say so.
50 +2. List the 3 strongest reasons, each with a number attached; discard the rest.
51 +3. Add cost, timeline, and the top 2–3 risks with mitigations.
52 +4. Assemble: Recommendation → Rationale → Cost & Risk → Next step (owner + date).
53 +5. Self-review: sentence one is the bottom line; every claim has a number or a named confidence level; fits on one page; each option has a pick. Fix violations and re-check.
54 +
55 +## Edge cases
56 +- **The data doesn't support a recommendation yet** → say exactly that in sentence one, state what decision-relevant information is missing and the date you'll have it. Never fake certainty.
57 +- **Two options are genuinely tied** → recommend the reversible one and say that's the tiebreaker.
58 +- **Hostile audience for the conclusion** → the bottom line still goes first; add the single strongest counterargument and your answer to it in the rationale.
59 +
60 +## References
61 +Extended before/after examples: see [references/examples.md](references/examples.md).
added writing-skills/writing-executive-summaries/references/examples.md +95 −0
@@ -0,0 +1,95 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Examples — Executive Summary House Style
7 +
8 +## Contents
9 +- Opening-sentence pairs
10 +- Complete worked example (bad → house style)
11 +- The "no recommendation yet" variant
12 +- Gotchas
13 +
14 +## Opening-sentence pairs
15 +
16 +| ❌ Never | ✅ House style |
17 +|---|---|
18 +| "This report presents the findings of our infrastructure review." | "Recommendation: consolidate to one cloud region, saving $84k/yr at negligible latency cost." |
19 +| "Over the past quarter, the team has been analyzing churn." | "Churn doubled in Q2 (4.1% → 8.3%); the driver is the March pricing change, and we recommend reverting it for accounts under 50 seats." |
20 +| "Several options were considered for the mobile strategy." | "We should ship the responsive web app now and defer the native app 12 months; this captures 80% of the value at 20% of the cost." |
21 +
22 +## Complete worked example
23 +
24 +**❌ Bad version (methodology-first, adjectives, no pick):**
25 +
26 +> **Executive Summary**
27 +>
28 +> As part of our ongoing commitment to operational excellence, a
29 +> cross-functional working group was convened to evaluate our customer
30 +> support tooling. The group conducted stakeholder interviews, reviewed
31 +> industry benchmarks, and assessed three leading platforms across a
32 +> comprehensive evaluation matrix. Each option presents meaningful benefits.
33 +> Platform A offers robust automation. Platform B provides significant
34 +> flexibility. Platform C is a strong contender on price. The working group
35 +> presents these options for leadership consideration. Adoption of a new
36 +> platform is expected to substantially improve response times and deliver
37 +> considerable cost efficiencies.
38 +
39 +**✅ House-style version:**
40 +
41 +> **Executive Summary**
42 +>
43 +> **Recommendation:** adopt Platform A for customer support, replacing
44 +> Zendesk in Q4. Net saving: $61k/yr; payback in 7 months (high confidence —
45 +> pricing is contracted, migration quoted).
46 +>
47 +> **Why:**
48 +> - Automation resolves an estimated 38% of tier-1 tickets (measured in our
49 +> 3-week pilot: 41% on 1,200 tickets).
50 +> - First-response time drops from 6.2h to under 1h for automated categories.
51 +> - Only option meeting our EU data-residency requirement out of the box.
52 +>
53 +> **Cost & risk:**
54 +> - $45k migration (one-time) + $9k/mo licence (vs $14k/mo today).
55 +> - Risk: agent adoption → mitigation: 2-week parallel run, champions per team.
56 +> - Risk: data-migration loss → mitigation: vendor-managed migration with
57 +> row-count verification; Zendesk kept read-only for 90 days.
58 +>
59 +> **Next step:** sign by Aug 15 (quote expiry); IT kickoff Aug 18. Owner: S. Boucher.
60 +>
61 +> *Methodology and full scoring matrix: appendix B.*
62 +
63 +## The "no recommendation yet" variant
64 +
65 +> **Executive Summary**
66 +>
67 +> **We cannot yet recommend a platform:** the two finalists are tied on cost
68 +> and features, and the deciding factor — EU data-residency certification for
69 +> Platform B — is unconfirmed. Vendor answer due Aug 12; we will issue the
70 +> recommendation Aug 13. If forced to decide today, Platform A (the
71 +> reversible choice: month-to-month contract).
72 +
73 +## Condensing an existing report (summary-at-the-top variant)
74 +
75 +When the summary heads a longer document, it must stand alone — assume most
76 +readers stop after it.
77 +
78 +| ❌ Never | ✅ House style |
79 +|---|---|
80 +| "As detailed in section 4…" (forces the jump) | State the number here; cite the section in parentheses for the curious. |
81 +| "See below for our recommendation." | The recommendation is sentence one of the summary itself. |
82 +| Summarizing section by section in document order | Reorganize into Recommendation → Rationale → Cost & Risk regardless of the report's structure. |
83 +| New information that appears only in the summary | Everything in the summary exists in the body; the summary selects, never adds. |
84 +
85 +## Gotchas
86 +
87 +- **The summary written before the analysis ends** becomes advocacy. Write
88 + the recommendation sentence only when you can defend its number.
89 +- **"High/medium/low confidence" without basis** is an adjective in disguise —
90 + attach what makes it high ("contracted pricing", "n=1,200 pilot").
91 +- **Percentage without base** ("38% of tickets") invites the first hostile
92 + question; give the base once ("of 1,200 pilot tickets").
93 +- **Next step without an owner and date** means no next step.
94 +- **Executive summaries of bad news** follow the same BLUF rule — burying
95 + "we lost the account" in paragraph three reads as spin, not tact.
added writing-skills/writing-meeting-notes/SKILL.md +60 −0
@@ -0,0 +1,60 @@
1 +---
2 +name: writing-meeting-notes
3 +description: Writes meeting notes and minutes in a decisions-first house style — decisions and action items at the top, every action with an owner and deadline, neutral discussion summary. Use when the user asks to write up, summarize, or clean up meeting notes, minutes, action items, or a recap from a transcript, recording notes, or their own rough notes. Do not use for executive summaries, project status reports, or verbatim transcription.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing Meeting Notes
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** minutes, recaps, and action-item write-ups from any meeting input — rough notes, a transcript, or a verbal debrief.
15 +- **Do NOT use for:** executive summaries (writing-executive-summaries), project status reports, or verbatim transcription. Write those normally.
16 +
17 +## House rules
18 +
19 +1. **Decisions and actions first, discussion second.** Most readers need only the first screen; the discussion summary exists for the few who ask "why".
20 + - ✅ Order: Decisions → Action items → Open items → Discussion summary.
21 + - ❌ Chronological replay of the meeting ending with "next steps".
22 +
23 +2. **Every action item = owner + deadline + verifiable deliverable.** One name (not a team), one date, one thing that either exists or doesn't.
24 + - ✅ "Dana: send signed vendor contract to finance — by Aug 8."
25 + - ❌ "Team to look into vendor options." / "Marketing to follow up."
26 +
27 +3. **Decisions carry one line of rationale.** The why prevents the decision being relitigated in three weeks.
28 + - ✅ "Decision: ship without the export feature — it blocks 0 of the 12 beta customers, and waiting costs the launch window."
29 + - ❌ "Decision: ship without the export feature."
30 +
31 +4. **Header: attendees, date, purpose.** Three lines; whoever wasn't there can tell in five seconds whether to read on.
32 + - ✅ "Aug 5 · Pricing review · Dana, Alex, Simon (notes) · Goal: pick the Q4 price point."
33 + - ❌ No header, or a header listing only the meeting title.
34 +
35 +5. **Neutral discussion summary.** Report positions without adjectives or editorializing; attribute contested claims.
36 + - ✅ "Alex argued the discount cannibalizes annual plans; Dana disagreed, citing the 2025 cohort data."
37 + - ❌ "Alex made a good point…" / "After some pointless back-and-forth…"
38 +
39 +6. **Unresolved items are flagged OPEN with a next step.** Silence on an unresolved item is how it dies.
40 + - ✅ "OPEN: EU pricing — needs legal's VAT answer. Next: Simon asks legal by Aug 7, decision at next week's sync."
41 + - ❌ Leaving the unresolved topic out because nothing was decided.
42 +
43 +7. **Distribute within 24 hours.** Value decays fast; a perfect recap on day 3 loses to a good one within the day.
44 + - ✅ Send the same day with "corrections by tomorrow noon, then this stands as the record."
45 + - ❌ Polishing for three days.
46 +
47 +## Workflow
48 +
49 +1. From the raw input, extract three lists: decisions made, actions agreed, items raised but unresolved.
50 +2. For each action, pin owner + deadline + deliverable — if any is missing from the meeting, mark it "(owner/date TBC — flagged)" rather than inventing one.
51 +3. Write the 3-line header, then Decisions (with rationale), Actions, OPEN items (with next step), then a neutral discussion summary of 3–8 bullets.
52 +4. Self-review: every action has one name and a date; no adjectives about people or opinions; nothing decided appears only in the discussion section. Fix and deliver with the corrections-deadline line.
53 +
54 +## Edge cases
55 +- **Nothing was decided** → say so at the top ("No decisions — discussion only"), keep actions and OPEN items; that emptiness is information.
56 +- **Sensitive/HR content came up** → note "a personnel matter was discussed offline" without detail; minutes are distributable documents.
57 +- **Conflicting recollections in the input** → record both versions attributed, flag OPEN for confirmation.
58 +
59 +## References
60 +Extended before/after examples: see [references/examples.md](references/examples.md).
added writing-skills/writing-meeting-notes/references/examples.md +106 −0
@@ -0,0 +1,106 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Examples — Meeting Notes House Style
7 +
8 +## Contents
9 +- Action-item pairs
10 +- Complete worked example (bad → house style)
11 +- The "no decisions" meeting
12 +- Gotchas
13 +
14 +## Action-item pairs
15 +
16 +| ❌ Never | ✅ House style |
17 +|---|---|
18 +| Team to look into hosting costs | Alex: cost comparison of the 3 hosting options in a shared sheet — by Aug 12 |
19 +| Marketing to follow up with the agency | Priya: get revised banner quotes from the agency, post in #launch — by Thu |
20 +| Everyone review the spec | Each of Dana/Alex/Sam: comments on spec v3 in the doc — by Mon 10am |
21 +| Discuss pricing next time | OPEN: pricing tier for EDU — Simon adds it to next sync's agenda with legal's VAT answer |
22 +
23 +## Complete worked example
24 +
25 +**Raw input (chronological ramble):**
26 +
27 +> talked about launch date, marketing wants sept, eng says risky. went back
28 +> and forth on the export feature. decided to cut it. sarah raised the EU
29 +> question again, nobody had the legal answer. mike will do something about
30 +> the beta emails. next meeting tuesday.
31 +
32 +**❌ Bad version (chronological, ownerless):**
33 +
34 +> Meeting Notes
35 +>
36 +> We started by discussing the launch date. Marketing prefers September but
37 +> engineering expressed concerns. There was a long discussion about the
38 +> export feature and it was eventually decided to cut it. Sarah raised the
39 +> EU question again. Mike will handle the beta emails. Next meeting Tuesday.
40 +
41 +**✅ House-style version:**
42 +
43 +> **Aug 5 · Launch sync · Sarah, Mike, Priya, Simon (notes) · Goal: lock scope for Sept launch**
44 +>
45 +> **Decisions**
46 +> 1. Export feature cut from launch scope — blocks 0 of 12 beta customers;
47 +> keeping it risks the September window. (Revisit for the October release.)
48 +>
49 +> **Action items**
50 +> - Mike: send beta-cohort email announcing the export change — by Aug 7
51 +> - Priya: confirm Sept 15 with the agency and book assets — by Aug 8
52 +> - Simon: circulate these notes; corrections by Aug 6 noon
53 +>
54 +> **OPEN**
55 +> - EU availability at launch — blocked on legal's VAT answer.
56 +> Next: Sarah asks legal by Aug 7; decision at Tuesday's sync.
57 +>
58 +> **Discussion summary**
59 +> - Marketing (Priya) holds Sept 15; engineering (Mike) called the date
60 +> "tight but doable" once export was cut.
61 +> - Sarah noted the EU question has now slipped twice — hence the OPEN flag
62 +> with a dated next step.
63 +>
64 +> Next sync: Tue Aug 12, same time.
65 +
66 +## The "no decisions" meeting
67 +
68 +> **Aug 6 · Architecture brainstorm · Full eng team · Goal: explore options (no decision expected)**
69 +>
70 +> **No decisions — discussion only.**
71 +>
72 +> **Action items**
73 +> - Dana: write up option A (event-sourced) as a 1-pager — by Fri
74 +> - Sam: write up option B (CRUD + audit log) as a 1-pager — by Fri
75 +>
76 +> **Discussion summary:** [3–5 neutral bullets]
77 +
78 +## Recurring-meeting variant (carry-forward discipline)
79 +
80 +For weekly syncs, open with last week's actions and their status — an action
81 +item that never gets re-read never gets done.
82 +
83 +> **Carried from Aug 5:**
84 +> - ✅ Mike: beta-cohort email — sent Aug 7
85 +> - ❌ Priya: agency confirmation — slipped; new date Aug 14 (2nd slip — flagged)
86 +>
87 +> **Decisions** …
88 +
89 +Rules: a slipped action keeps its history ("2nd slip") rather than silently
90 +getting a fresh deadline; done items appear once with ✅ then drop off; an
91 +action slipping three times goes to the meeting agenda as a blocker, not
92 +back onto the list.
93 +
94 +## Gotchas
95 +
96 +- **"Owner: everyone"** means owner: no one. Split it per person or assign
97 + one collector.
98 +- **Deadlines like "next week"** rot instantly — convert to a date while the
99 + meeting context is fresh.
100 +- **Decisions hidden in the discussion section** get missed by the
101 + decisions-only readers; if it was decided, it appears under Decisions,
102 + full stop.
103 +- **Recording who was late/absent-with-excuse** is surveillance, not minutes —
104 + list attendees, skip the commentary.
105 +- **The corrections window** ("stands as the record after tomorrow noon")
106 + is what turns notes into an authoritative artifact; don't skip it.
added writing-skills/writing-professional-emails/SKILL.md +65 −0
@@ -0,0 +1,65 @@
1 +---
2 +name: writing-professional-emails
3 +description: Writes and edits professional workplace emails in a direct, scannable house style — ask-first structure, decision-ready subject lines, explicit next steps. Use when the user asks to write, draft, reply to, or improve a work email, a follow-up, a request to a colleague or client, or an announcement sent by email. Do not use for marketing or newsletter email campaigns, chat or Slack messages, or formal letters.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing Professional Emails
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** workplace emails — requests, replies, follow-ups, status updates, announcements to colleagues, clients, or partners.
15 +- **Do NOT use for:** marketing/newsletter campaigns (different goals and regulations), chat/Slack messages (different register), or formal letters (different conventions). Write those normally.
16 +
17 +## House rules
18 +
19 +1. **Subject line = the ask or the news.** The reader should be able to triage from the inbox list alone.
20 + - ✅ `Approval needed by Fri: Q3 vendor contract ($48k)`
21 + - ❌ `Quick question` / `Following up`
22 +
23 +2. **The ask in the first two sentences.** Context comes after the request, never before.
24 + - ✅ "Can you approve the attached contract by Friday? Background below."
25 + - ❌ Three paragraphs of history ending with "...so I was wondering if you could approve it."
26 +
27 +3. **One email = one topic.** A second topic gets a second email with its own subject.
28 + - ✅ Separate emails for "contract approval" and "office move dates".
29 + - ❌ "Also, while I have you..." burying a second decision in paragraph four.
30 +
31 +4. **Scannable body.** Bullets for multiple items; bold the deadline or the single most important fact.
32 + - ✅ "Three things need your input: • budget cap • start date • vendor choice. **Deadline: Thursday 5pm.**"
33 + - ❌ A single 12-line paragraph containing all three questions.
34 +
35 +5. **Explicit next step with owner and date.** End with who does what by when.
36 + - ✅ "Next step: I'll send the revised draft by Wednesday; you review by Friday."
37 + - ❌ "Let me know what you think." / "Hope this helps!"
38 +
39 +6. **Mirror the counterpart's formality.** Match their greeting, sign-off, and register; when unknown, start neutral-professional ("Hi <name>," / "Best,").
40 + - ✅ Client writes "Dear Mr. Boucher" → reply "Dear Ms. Chen".
41 + - ❌ Replying "Hey!" to a formal first contact.
42 +
43 +7. **Reply-all discipline.** Reply-all only when every recipient needs the answer; move people to BCC with a note when releasing them from a thread.
44 + - ✅ "Moving the team to BCC to spare inboxes — Dana and I will close this out."
45 + - ❌ Reply-all "Thanks!" to 40 people.
46 +
47 +8. **No passive-aggressive markers.** State the fact or re-ask plainly.
48 + - ✅ "Resending — I need your answer by tomorrow to hold the price."
49 + - ❌ "As previously mentioned…" / "Per my last email…" / "Just circling back again…"
50 +
51 +## Workflow
52 +
53 +1. Identify the single goal of the email (decision, information, action) and the one reader who must act.
54 +2. Write the subject line as the ask or the news.
55 +3. Write the ask in sentences one/two; add only the context needed to act on it.
56 +4. Format for scanning (bullets, bolded deadline); add the next-step line (owner + date).
57 +5. Self-review against the eight rules above; fix any violation, then check the recipient list (To = must act, CC = informed only) before delivering.
58 +
59 +## Edge cases
60 +- **Bad-news email** → the news still goes first, stated plainly; follow with cause and remedy in that order. Never soften into ambiguity.
61 +- **Escalation email** → facts and dates only, no adjectives about people; state what you need from the recipient to unblock.
62 +- **Very short reply suffices** → send the short reply ("Approved — go ahead."). The rules exist for the reader's time, not for length.
63 +
64 +## References
65 +Extended before/after examples: see [references/examples.md](references/examples.md).
added writing-skills/writing-professional-emails/references/examples.md +110 −0
@@ -0,0 +1,110 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Examples — Professional Email House Style
7 +
8 +## Contents
9 +- Subject-line pairs
10 +- Complete worked example: request email (bad → house style)
11 +- Complete worked example: bad-news email (bad → house style)
12 +- Follow-up sequence without passive aggression
13 +- Gotchas
14 +
15 +## Subject-line pairs
16 +
17 +| ❌ Never | ✅ House style |
18 +|---|---|
19 +| Quick question | Need your call: ship Friday or wait for QA? |
20 +| Update | Launch moved to Sept 12 — no action needed |
21 +| Meeting | Can you join a 30-min pricing review Thu 2pm? |
22 +| Following up | Reminder: contract signature due tomorrow |
23 +| Important!!! | Server costs up 40% — decision needed by EOM |
24 +
25 +## Complete worked example: request email
26 +
27 +**❌ Bad version (context-first, buried ask, no next step):**
28 +
29 +> Subject: Question
30 +>
31 +> Hi Sarah,
32 +>
33 +> Hope you're doing well! As you know, we've been working on the vendor
34 +> evaluation for the past few weeks. The team looked at five options and
35 +> narrowed it down to two. There were a lot of considerations around pricing,
36 +> support SLAs, and integration effort, and honestly it was a close call.
37 +> After several discussions we're leaning toward Acme, though DataCo has
38 +> some advantages too. Anyway, I was wondering if you might have some time
39 +> to maybe look this over when you get a chance?
40 +>
41 +> Thanks!
42 +
43 +**✅ House-style version:**
44 +
45 +> Subject: Approval needed by Thu: Acme as our data vendor ($36k/yr)
46 +>
47 +> Hi Sarah,
48 +>
49 +> Can you approve Acme as our data vendor by **Thursday 5pm**? Their quote
50 +> expires Friday.
51 +>
52 +> Why Acme over DataCo:
53 +> - 20% cheaper at our volume ($36k vs $45k/yr)
54 +> - 24h support SLA vs business-hours only
55 +> - Integration estimated at 2 days vs 2 weeks
56 +>
57 +> Full comparison attached. Next step: on your approval I'll sign Thursday
58 +> and kick off integration Monday.
59 +>
60 +> Best,
61 +> Simon
62 +
63 +## Complete worked example: bad-news email
64 +
65 +**❌ Bad version (buried, vague, blame-shifting):**
66 +
67 +> Subject: Project update
68 +>
69 +> Hi all, lots of moving parts lately! The team has been heads-down and
70 +> we've made real progress on several fronts. That said, given some
71 +> dependencies outside our control and a few unexpected challenges, the
72 +> timeline may need to be revisited somewhat...
73 +
74 +**✅ House-style version:**
75 +
76 +> Subject: Launch slips 2 weeks to Oct 3 — mitigation below
77 +>
78 +> Hi all,
79 +>
80 +> The launch moves from Sept 19 to **Oct 3**. Cause: the payment-provider
81 +> API migration returned breaking changes we discovered in integration
82 +> testing on Tuesday.
83 +>
84 +> Mitigation:
85 +> - Beta customers keep the Sept 19 date on the legacy flow
86 +> - Marketing holds the announcement until Sept 26
87 +>
88 +> Next step: I'll confirm the new date is solid at Friday's checkpoint and
89 +> report here by 3pm.
90 +
91 +## Follow-up sequence without passive aggression
92 +
93 +- Follow-up 1 (day 3): "Resending — I need your sign-off by Wed to keep the
94 + Friday ship date."
95 +- Follow-up 2 (day 5): "Deadline risk: without sign-off today, ship moves to
96 + next sprint. Reply 'go' and I'll handle the rest."
97 +- Escalation (day 6, new thread, manager CC'd): facts and dates only:
98 + "Sign-off requested Aug 1, followed up Aug 4 and Aug 6; ship date at risk."
99 +
100 +## Gotchas
101 +
102 +- **"Hope you're well" openers** — harmless once, filler when the ask is
103 + urgent. If the email fits on one screen with it, keep it; otherwise cut.
104 +- **Bold more than one thing** and nothing is bold. One bolded element per email.
105 +- **Questions hidden in statements** ("It would be great to know the budget")
106 + get no answers. Ask directly with a question mark.
107 +- **CC as pressure** reads as hostile. CC managers only with a stated reason
108 + ("CC Dana for visibility on the date change").
109 +- **Attachment amnesia** — name the attachment in the body ("comparison
110 + attached") so its absence is caught before sending.
added writing-skills/writing-proposals/SKILL.md +62 −0
@@ -0,0 +1,62 @@
1 +---
2 +name: writing-proposals
3 +description: Writes project and business proposals in a client-first house style — their problem restated first, outcome before activities, explicit scope exclusions, tiered pricing with a recommendation. Use when the user asks to write or improve a proposal, a statement of work, a pitch document, a quote with scope, or a response to an RFP for a client or internal sponsor. Do not use for executive summaries of internal work, contracts or legal terms, or grant applications.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing Proposals
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** client project proposals, statements of work, internal investment pitches, RFP responses, quotes that carry scope.
15 +- **Do NOT use for:** executive summaries of completed internal work (writing-executive-summaries), contracts/legal terms (route to legal), or grant applications (funder-specific formats). Write those normally.
16 +
17 +## House rules
18 +
19 +1. **Open with THEIR problem, restated in their words.** The first section proves you listened; it contains no mention of you.
20 + - ✅ "Your support team answers the same 30 questions 400 times a month, and response time slipped past your 4-hour SLA in March."
21 + - ❌ "Founded in 2019, our agency specializes in cutting-edge automation solutions."
22 +
23 +2. **Outcome before activities.** Sell the destination, then the itinerary.
24 + - ✅ "Outcome: tier-1 tickets resolved in under 5 minutes, without hiring. How we get there: …"
25 + - ❌ Leading with a 14-row table of tasks, hours, and deliverables.
26 +
27 +3. **Scope with explicit exclusions.** What is NOT included prevents every future dispute.
28 + - ✅ "Not included: content migration from the legacy wiki, training beyond the two included sessions, ongoing maintenance (available separately)."
29 + - ❌ A scope section that only lists inclusions.
30 +
31 +4. **Price in tiers with a recommendation.** Good/better/best beats take-it-or-leave-it; anchor and recommend.
32 + - ✅ "Essential $18k / **Recommended: Standard $27k** / Premium $41k — Standard because the analytics module pays for itself by month 3."
33 + - ❌ A single number with no framing: "Total: $27,400."
34 +
35 +5. **Risks and assumptions, each one accountable.** State what you assume; each assumption is billable if wrong — say so.
36 + - ✅ "Assumption: API access granted by day 3. If access slips, timeline shifts day-for-day and re-planning is billed at the day rate."
37 + - ❌ No assumptions section, or "subject to unforeseen circumstances."
38 +
39 +6. **Concrete next step with an expiry date.** A proposal without a deadline negotiates against itself forever.
40 + - ✅ "To start Sept 2: countersign by Aug 22. Pricing valid until Aug 29."
41 + - ❌ "We look forward to hearing from you at your convenience."
42 +
43 +7. **Social proof after the solution, not before.** Case studies support a claim the reader already understands.
44 + - ✅ Solution section, then: "We did this for Nordica (case: 62% ticket deflection in 8 weeks)."
45 + - ❌ Opening page of logos and testimonials before the problem is stated.
46 +
47 +## Workflow
48 +
49 +1. Extract the client's problem statement from their own words (call notes, RFP, emails); write section 1 from it.
50 +2. Write the outcome section: the measurable after-state, with numbers where defensible.
51 +3. Draft approach/deliverables, then the exclusions list (ask: "what will they assume is included?").
52 +4. Build three price tiers; mark the recommended one and justify the pick in one sentence.
53 +5. List assumptions and risks with owner/consequence; add timeline and next step with expiry.
54 +6. Self-review against the seven rules; check every number appears once and is consistent; then deliver.
55 +
56 +## Edge cases
57 +- **RFP with a mandated structure** → follow their structure exactly (compliance beats style), but apply the rules inside each mandated section.
58 +- **No budget signal from the client** → tiers do the discovery; widen the spread (×2–3 between lowest and highest).
59 +- **Internal proposal (no price)** → tiers become effort/scope options; the "price" is headcount-weeks and opportunity cost. All other rules hold.
60 +
61 +## References
62 +Extended before/after examples: see [references/examples.md](references/examples.md).
added writing-skills/writing-proposals/references/examples.md +109 −0
@@ -0,0 +1,109 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Examples — Proposal House Style
7 +
8 +## Contents
9 +- Opening-section pairs
10 +- Scope exclusion patterns
11 +- Complete worked example (bad → house style, condensed)
12 +- Tier table pattern
13 +- Gotchas
14 +
15 +## Opening-section pairs
16 +
17 +**❌ Never (us-first):**
18 +> About Us: With over a decade of combined experience, our team delivers
19 +> best-in-class digital transformation solutions to clients across industries.
20 +
21 +**✅ House style (them-first):**
22 +> The problem. Your checkout abandonment sits at 71% — 11 points above your
23 +> category median — and your team traced most drop-offs to the forced
24 +> account-creation step. You told us: "we lose them at the login wall."
25 +
26 +## Scope exclusion patterns
27 +
28 +✅ Every scope section carries three lists:
29 +
30 +```markdown
31 +### Included
32 +- Checkout flow redesign (5 screens) with two revision rounds
33 +- A/B test setup and 4-week measurement
34 +
35 +### Not included
36 +- Payment-provider migration or new payment methods
37 +- Copywriting beyond the checkout screens
38 +- Changes to the mobile app (web only)
39 +
40 +### Available separately
41 +- Post-launch conversion monitoring: $2k/mo
42 +```
43 +
44 +❌ Never: an "Included" list alone — the client's imagination writes the
45 +other two lists, in their favor.
46 +
47 +## Complete worked example (condensed)
48 +
49 +**❌ Bad version:**
50 +
51 +> Proposal for Services
52 +>
53 +> About us: [4 paragraphs of credentials and logos]
54 +> Our approach: We will conduct a discovery phase, followed by design
55 +> sprints, followed by implementation and QA. Deliverables include wireframes,
56 +> design system tokens, and up to 40 hours of development.
57 +> Investment: $27,400.
58 +> We're excited about the opportunity and look forward to hearing from you!
59 +
60 +**✅ House-style version:**
61 +
62 +> ## Cutting checkout abandonment at Northshoe
63 +>
64 +> **The problem.** Checkout abandonment is 71% against a 60% category median;
65 +> your analytics show the account-creation step loses half of them. Every
66 +> point of abandonment is ≈$9k/mo in lost orders at current traffic.
67 +>
68 +> **The outcome.** Guest checkout live in 6 weeks; target abandonment ≤63%
69 +> within 60 days of launch, measured by your own GA4 funnel — worth ≈$70k/yr
70 +> at the low end.
71 +>
72 +> **How.** Week 1–2 flow redesign (5 screens, 2 revision rounds); week 3–5
73 +> build behind a feature flag; week 6 A/B rollout at 50%.
74 +>
75 +> **Scope.** Included / Not included / Available separately: [three lists]
76 +>
77 +> **Investment.** Essential $18k · **Standard $27k (recommended)** ·
78 +> Premium $41k. Standard adds the A/B measurement that turns the launch
79 +> into evidence; Premium adds the mobile app.
80 +>
81 +> **Assumptions.** GA4 access by day 2; feature-flag infra exists (it does —
82 +> LaunchDarkly). If either slips, timeline moves day-for-day.
83 +>
84 +> **Proof.** Same intervention at Nordica: 71% → 58% in 9 weeks (case attached).
85 +>
86 +> **Next step.** Countersign by Aug 22 to hold the Sept 2 start. Pricing
87 +> valid until Aug 29.
88 +
89 +## Tier table pattern
90 +
91 +| | Essential $18k | **Standard $27k ✓** | Premium $41k |
92 +|---|---|---|---|
93 +| Checkout redesign | ✅ | ✅ | ✅ |
94 +| A/B test + measurement | — | ✅ | ✅ |
95 +| Mobile app screens | — | — | ✅ |
96 +
97 +One sentence under the table says why the middle tier is the pick.
98 +
99 +## Gotchas
100 +
101 +- **Effort listed in hours** invites hourly negotiation; sell deliverables
102 + and outcomes, keep hours internal.
103 +- **"Investment" vs "cost"** — use "investment" only when an ROI number backs
104 + it; otherwise it reads as spin.
105 +- **Unpriced Premium tier** ("contact us") breaks the anchor; price all tiers.
106 +- **Case studies without numbers** are decoration; every proof point carries
107 + a metric and a timeframe.
108 +- **The expiry date you don't enforce** trains clients to ignore the next one.
109 + Pick a date you'll honor.
added writing-skills/writing-technical-documentation/SKILL.md +61 −0
@@ -0,0 +1,61 @@
1 +---
2 +name: writing-technical-documentation
3 +description: Writes READMEs, architecture docs, and runbooks that lead with purpose, put a working quickstart first, and stay ruthlessly scannable. Use when the user asks to write or improve a README, project documentation, an architecture document, a design doc, a runbook, or internal engineering docs. Do not use for API endpoint reference docs, step-by-step tutorials, or user-facing release notes — separate skills cover those.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing Technical Documentation
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** READMEs, architecture/design docs, runbooks, onboarding docs, internal engineering documentation.
15 +- **Do NOT use for:** API endpoint reference (writing-api-documentation), step-by-step tutorials (writing-tutorials), release notes (writing-release-notes), or blog posts (writing-blog-posts).
16 +
17 +## House rules
18 +
19 +1. **Lead with what it is and who it's for — first two sentences.**
20 + - ✅ "payment-router routes card transactions to the cheapest eligible processor. It's used by the checkout service; most engineers only need the client library."
21 + - ❌ "Welcome to the payment-router repository! This document describes the project."
22 +
23 +2. **Quickstart before reference.** A copy-paste working example must appear in the first screen of a README.
24 + - ✅ Install command + minimal working snippet within the first 30 lines.
25 + - ❌ Three screens of architecture background before the first runnable command.
26 +
27 +3. **Document the why, not just the what** (ADR mini-pattern: Context → Decision → Consequences).
28 + - ✅ "We chose SQS over Kafka because volume is under 100 msg/s and we already pay for AWS. Revisit if we need replay."
29 + - ❌ "The system uses SQS."
30 +
31 +4. **Docs live next to the code and change in the same PR.**
32 + - ✅ "Updated `docs/runbook.md` in the PR that changed the alert threshold."
33 + - ❌ A wiki page last edited two majors ago.
34 +
35 +5. **Ruthless scannability.** A heading roughly every 10 lines; tables for enumerable facts; every code block tested before committing.
36 + - ✅ A "Configuration" table with name/default/effect columns.
37 + - ❌ Configuration options described across four paragraphs of prose.
38 +
39 +6. **No marketing adjectives.** Claims must be verifiable.
40 + - ✅ "Handles 2,000 requests/s on one c7g.large."
41 + - ❌ "Blazingly fast, robust, and seamless."
42 +
43 +7. **Date-stamp anything that rots.** Benchmarks, dependency lists, screenshots, on-call contacts get an "as of YYYY-MM" marker.
44 + - ✅ "Benchmarks (as of 2026-08): …"
45 + - ❌ Undated numbers that outlive their truth.
46 +
47 +## Workflow
48 +
49 +1. Identify the doc type (README / architecture doc / runbook) and the primary reader; write both at the top as the first two sentences.
50 +2. Draft the skeleton: purpose → quickstart → how it works → reference → operations. For runbooks: symptom → diagnosis → fix → escalation.
51 +3. Write the quickstart first and actually run every command in it.
52 +4. Fill remaining sections; convert any enumerable prose into tables.
53 +5. Self-review against the seven house rules; delete every unverifiable adjective; add date stamps; fix violations before delivering.
54 +
55 +## Edge cases
56 +- **Existing doc to improve** → preserve its structure where sound; apply rules 1–2 first (purpose and quickstart), then scannability. List what you changed.
57 +- **No runnable quickstart possible** (pure library of internals, docs-only repo) → substitute a "minimal usage" code snippet or a "start here" reading path.
58 +- **Audience is mixed** (users + contributors) → split: README for users, CONTRIBUTING/ARCHITECTURE for contributors. Never interleave.
59 +
60 +## References
61 +Extended before/after examples: see [references/examples.md](references/examples.md).
added writing-skills/writing-technical-documentation/references/examples.md +132 −0
@@ -0,0 +1,132 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Examples — Technical Documentation House Style
7 +
8 +## Contents
9 +- Complete worked example: README (bad → house style)
10 +- ADR mini-pattern examples
11 +- Runbook entry (bad → house style)
12 +- Scannability conversions
13 +- Gotchas
14 +
15 +## Complete worked example: README
16 +
17 +**❌ Before (typical drifted README):**
18 +
19 +```markdown
20 +# payment-router
21 +
22 +Welcome to payment-router! This repository contains the source code for the
23 +payment routing system. Payment routing is an important part of our
24 +infrastructure. It is a robust, scalable, high-performance solution.
25 +
26 +## Background
27 +
28 +Historically, payments were processed by the monolith. In 2023 the team
29 +decided to extract routing. There were many discussions about the
30 +architecture. Kafka was considered...
31 +
32 +(three more screens before any command)
33 +```
34 +
35 +**✅ After (house style):**
36 +
37 +```markdown
38 +# payment-router
39 +
40 +Routes card transactions to the cheapest eligible processor. Used by the
41 +checkout service — most engineers only need the client library below.
42 +
43 +## Quickstart
44 +
45 + pip install payment-router-client
46 +
47 + from payment_router import route
48 + decision = route(amount_cents=1999, currency="USD", card_bin="424242")
49 + print(decision.processor) # "stripe"
50 +
51 +## How it works
52 +
53 +Each transaction is scored against processor fee tables (refreshed hourly)
54 +and eligibility rules (`rules/*.yaml`). Highest-margin eligible processor
55 +wins. Ties break on historical auth rate.
56 +
57 +## Why SQS, not Kafka
58 +
59 +Context: volume is under 100 msg/s and we already run on AWS.
60 +Decision: SQS for retry queues.
61 +Consequences: no replay; revisit if volume exceeds 1,000 msg/s.
62 +
63 +## Configuration
64 +
65 +| Variable | Default | Effect |
66 +|---|---|---|
67 +| `PR_FEE_REFRESH_MIN` | `60` | Minutes between fee-table refreshes |
68 +| `PR_FALLBACK` | `stripe` | Processor when no rule matches |
69 +
70 +## Operations
71 +
72 +Benchmarks (as of 2026-08): 2,000 req/s on one c7g.large, p99 = 11 ms.
73 +Runbook: [docs/runbook.md](docs/runbook.md)
74 +```
75 +
76 +What changed: purpose + audience in two sentences; runnable quickstart in the
77 +first screen; decision documented with its why; config as a table; verifiable
78 +numbers with a date stamp; zero marketing adjectives.
79 +
80 +## ADR mini-pattern examples
81 +
82 +✅ "Context: we need per-tenant encryption and already use KMS.
83 +Decision: one KMS key per tenant, cached data keys (5 min TTL).
84 +Consequences: +$1/tenant/month; key deletion gives crypto-shredding for free."
85 +
86 +❌ "The system encrypts data per tenant using KMS." (decision with no why —
87 +the next engineer re-litigates it from scratch)
88 +
89 +## Runbook entry (bad → house style)
90 +
91 +**❌ Before:** "If the queue backs up, investigate the consumers and restart
92 +if needed."
93 +
94 +**✅ After:**
95 +
96 +```markdown
97 +## Symptom: `router_queue_depth > 10k` alert
98 +
99 +1. Diagnose: `kubectl logs deploy/router-consumer --since=10m | grep ERROR`
100 + - `FeeTableStale` → refresh job failed, go to step 2
101 + - `ProcessorTimeout` → upstream incident, escalate to #payments-oncall
102 +2. Fix: `kubectl create job --from=cronjob/fee-refresh manual-refresh`
103 +3. Verify: queue depth falling within 5 minutes.
104 +4. Escalate: if still rising after 15 min, page payments-primary.
105 +```
106 +
107 +## Scannability conversions
108 +
109 +Enumerable prose → table, always:
110 +
111 +❌ "The service supports three modes. In strict mode it rejects unknown
112 +fields. In lenient mode it ignores them. In log mode it accepts them but
113 +logs a warning."
114 +
115 +✅
116 +| Mode | Unknown fields |
117 +|---|---|
118 +| `strict` | rejected (400) |
119 +| `lenient` | ignored |
120 +| `log` | accepted, warning logged |
121 +
122 +## Gotchas
123 +
124 +- **Untested code blocks** are the #1 trust killer — a quickstart that errors
125 + on line one discredits the whole doc. Run every command before committing.
126 +- **"Simply" and "just"** hide missing steps; delete them and add the step.
127 +- **Architecture docs that describe the aspiration**, not the system as built:
128 + mark aspirational sections explicitly ("Planned, not implemented").
129 +- **Screenshots** rot fastest of all; prefer text output, and date-stamp any
130 + screenshot you must include.
131 +- **Duplicated content** between README and wiki always diverges; pick one
132 + home and link from the other.
added writing-skills/writing-tutorials/SKILL.md +59 −0
@@ -0,0 +1,59 @@
1 +---
2 +name: writing-tutorials
3 +description: Writes step-by-step tutorials and how-to guides where every step has a verifiable checkpoint, full commands, and expected output. Use when the user asks to write a tutorial, a how-to guide, a getting-started guide, a walkthrough, or onboarding steps for learners. Do not use for reference documentation, conceptual overviews and READMEs, or API endpoint docs — separate skills cover those.
4 +---
5 +
6 +<!--
7 +Author: Simon-Pierre Boucher
8 +Contact: contact@spboucher.ai
9 +-->
10 +
11 +# Writing Tutorials
12 +
13 +## When to use / when NOT to use
14 +- **Use for:** tutorials, how-to guides, getting-started walkthroughs, workshop material — content a learner follows top to bottom.
15 +- **Do NOT use for:** reference docs (writing-api-documentation), conceptual overviews/READMEs (writing-technical-documentation), or blog posts (writing-blog-posts).
16 +
17 +## House rules
18 +
19 +1. **State the destination upfront:** end result, prerequisites (with versions), and honest time estimate — before step 1.
20 + - ✅ "By the end you'll have a deployed webhook receiver. Prerequisites: Python 3.12, an ngrok account. Time: ~20 minutes."
21 + - ❌ Diving into `mkdir project` with no destination stated.
22 +
23 +2. **One path only.** No forks, options, or "alternatively…" mid-tutorial; link alternatives at the end.
24 + - ✅ "We'll use SQLite. (Using Postgres instead? See the appendix link at the end.)"
25 + - ❌ "You can use SQLite, Postgres, or MySQL here — configure accordingly."
26 +
27 +3. **Every step ends with a verifiable checkpoint.**
28 + - ✅ "Run `curl localhost:8000/health` — you should see `{"status":"ok"}`."
29 + - ❌ Three file edits in a row with no way to know they worked.
30 +
31 +4. **Full commands, full expected output.** Never `cd <your-project>`; never truncate output the learner must compare against.
32 + - ✅ The exact command plus the exact lines it prints.
33 + - ❌ "Run the usual migration commands."
34 +
35 +5. **Never skip steps that "everyone knows."** Activating the virtualenv, exporting the variable, saving the file — write them.
36 + - ✅ "Save the file, then in the same terminal run…"
37 + - ❌ "Simply configure your environment."
38 +
39 +6. **Troubleshoot the 3 most likely failures inline**, at the step where they occur — not in a distant appendix.
40 + - ✅ "If you see `Address already in use`, another process holds port 8000: run `lsof -i :8000`…"
41 + - ❌ A generic "Troubleshooting" section listing every possible error.
42 +
43 +7. **End with "what you built + where to go next":** recap the result and give 2–3 concrete next links.
44 +
45 +## Workflow
46 +
47 +1. Do the task yourself start to finish; record every command, output, and mistake you hit (mistakes become inline troubleshooting).
48 +2. Write the header: end result, prerequisites with versions, time estimate.
49 +3. Convert your recording into numbered steps, each with command → expected output → checkpoint.
50 +4. Add inline troubleshooting for the 3 most likely failures at the exact steps they occur.
51 +5. Validate: replay the tutorial verbatim in a clean environment (fresh directory/venv/container); any deviation between the doc and reality is a bug — fix it and replay until it runs clean.
52 +
53 +## Edge cases
54 +- **Can't provide a clean environment for replay** → state the tested environment explicitly ("Tested on macOS 15, Python 3.12.4") and flag untested paths.
55 +- **Long tutorial (over ~15 steps)** → split into parts, each part ending at a working state a learner can stop at.
56 +- **Steps involving paid/external services** → say the cost and offer the free-tier route as the single main path.
57 +
58 +## References
59 +Extended before/after examples: see [references/examples.md](references/examples.md).
added writing-skills/writing-tutorials/references/examples.md +111 −0
@@ -0,0 +1,111 @@
1 +<!--
2 +Author: Simon-Pierre Boucher
3 +Contact: contact@spboucher.ai
4 +-->
5 +
6 +# Examples — Tutorial House Style
7 +
8 +## Contents
9 +- Complete worked example: tutorial opening + one step (bad → house style)
10 +- Checkpoint patterns
11 +- Inline troubleshooting pattern
12 +- Ending pattern
13 +- Gotchas
14 +
15 +## Complete worked example
16 +
17 +**❌ Before:**
18 +
19 +```markdown
20 +# Webhooks tutorial
21 +
22 +In this tutorial we will learn about webhooks. Webhooks are a way for
23 +services to notify each other. First, set up your environment and create
24 +a project. You can use Flask, FastAPI, or Django. Then write a handler
25 +for the webhook and test it works.
26 +```
27 +
28 +**✅ After (house style):**
29 +
30 +```markdown
31 +# Receive Stripe webhooks locally
32 +
33 +By the end you'll have a local endpoint that verifies and logs Stripe
34 +events, reachable from the internet.
35 +
36 +**Prerequisites:** Python 3.12, a free Stripe test account, ngrok 3.x
37 +installed. **Time:** ~20 minutes.
38 +
39 +## Step 1 — Create the project
40 +
41 + mkdir stripe-webhooks && cd stripe-webhooks
42 + python3 -m venv .venv
43 + source .venv/bin/activate
44 + pip install fastapi==0.115.0 uvicorn==0.30.0 stripe==10.5.0
45 +
46 +Expected output ends with:
47 +
48 + Successfully installed fastapi-0.115.0 stripe-10.5.0 uvicorn-0.30.0
49 +
50 +**Checkpoint:** `python -c "import fastapi, stripe; print('ok')"` prints `ok`.
51 +
52 +> If you see `command not found: python3`, install Python 3.12 from
53 +> python.org, then restart this step in a new terminal.
54 +```
55 +
56 +What changed: destination + prerequisites + time before step 1; one framework
57 +(no menu); full commands including venv activation; pinned versions; expected
58 +output; a checkpoint; the likeliest failure handled inline.
59 +
60 +## Checkpoint patterns
61 +
62 +✅ Observable and exact:
63 +- "You should see `{"status":"ok"}`."
64 +- "The dashboard now lists one endpoint with a green Active badge."
65 +- "`ls migrations/` shows one file ending in `_init.py`."
66 +
67 +❌ Unverifiable:
68 +- "Make sure everything is configured correctly."
69 +- "The server should now be working."
70 +
71 +## Inline troubleshooting pattern
72 +
73 +Place at the step, as a quote block, most-likely first:
74 +
75 +```markdown
76 +> **`Address already in use`** — another process holds port 8000:
77 +> `lsof -i :8000`, stop it, rerun.
78 +> **`401 Unauthorized` from Stripe** — you exported the live key;
79 +> re-export the one starting with `sk_test_`.
80 +```
81 +
82 +Cap at 3 per step; more than that means the step itself needs splitting.
83 +
84 +## Ending pattern
85 +
86 +```markdown
87 +## What you built
88 +
89 +A verified Stripe webhook receiver: signature checking, event logging,
90 +and a public URL via ngrok.
91 +
92 +## Where to go next
93 +
94 +- Handle `invoice.paid` and update your database — see Persisting events
95 +- Deploy the receiver — see Deploying FastAPI
96 +- Full event catalog: Stripe docs, Webhook events
97 +```
98 +
99 +## Gotchas
100 +
101 +- **The author's environment leaks in** (aliases, globally installed tools,
102 + exported variables from earlier work). Only a clean-environment replay
103 + catches these — do it, every time.
104 +- **Unpinned versions** make tutorials rot silently; pin every install and
105 + date-stamp the tested versions.
106 +- **"Simply", "just", "obviously"** mark exactly the places learners get
107 + stuck; delete the word, add the missing step.
108 +- **Screenshots of terminals** can't be copy-pasted or diffed; use text
109 + blocks for anything the learner must compare.
110 +- **Checkpoint drift:** when you edit a step, its expected output usually
111 + changes too — replay from the edited step onward, not just the step.
112