# CLAUDE.md — Metrika

> **Metrika** — A next-generation statistical & data science application for macOS.
> Stata-class command syntax, GPU-accelerated by Apple Silicon (MLX/Metal), powered by DuckDB.
> Native Swift + SwiftUI. No Electron. No Python runtime dependency. No compromises.

---

## 0. Non-Negotiable Project Rules

1. **Every source file MUST begin with this header** (adapt comment style to the language):

```swift
//
//  <FileName>.swift
//  Metrika
//
//  Author:  Simon-Pierre Boucher
//  Contact: contact@spboucher.ai
//  Copyright © 2026 Simon-Pierre Boucher. All rights reserved.
//
```

For Metal shaders (`.metal`), shell scripts, and Python tooling scripts, use the equivalent comment syntax (`//` or `#`). **No file is committed without this header.** A pre-commit hook (see §12) enforces it.

2. Language: **Swift 6**, strict concurrency enabled (`-strict-concurrency=complete`).
3. UI: **SwiftUI first**, AppKit bridges (`NSViewRepresentable`) only where SwiftUI is insufficient (text editor, data grid virtualization).
4. Minimum target: **macOS 14 (Sonoma)**, Apple Silicon only (`arm64`). No Intel builds.
5. All numerical results must match reference implementations (R / Stata) to at least 1e-10 relative tolerance for CPU paths; GPU paths documented tolerance 1e-6 (float32) or 1e-10 (float64 where MLX supports it).
6. English for all code, comments, commit messages, and docs.

---

## 1. Product Vision

Metrika is a complete data science environment:

- **Command-driven**, like Stata: `reg y x1 x2, robust cluster(id)` — one line, publication-ready output.
- **Scriptable**: `.zyq` do-files with full reproducibility (logs, seeds, versioned datasets).
- **GPU-native**: massively parallel workloads (bootstrap, Monte Carlo, MCMC, ML training, permutation tests) run on Apple GPU via MLX with unified memory.
- **Extensible**: user-defined commands ("home-made" commands) via a plugin protocol — drop a Swift package or a `.zyq` ado-style script into `~/Library/Application Support/Metrika/Commands/`.
- **Big-data capable**: DuckDB engine → out-of-core, columnar, multi-threaded. 100M+ rows on a laptop.

### Feature pillars (v1 scope)

| Pillar | Contents |
|---|---|
| Data management | import/export CSV, Parquet, Arrow, Excel, Stata `.dta`, JSON; merge, append, reshape, collapse, egen-style generators |
| Estimation | summarize, tabulate, correlate, `regress` (OLS), `logit`, `probit`, `poisson`, `ivregress` (2SLS), `xtreg` (FE/RE), `areg`, quantile regression |
| Inference | robust (HC0–HC3), cluster-robust SE, **GPU bootstrap** (pairs, wild, cluster), permutation tests, jackknife |
| Simulation | Monte Carlo engine, random number generation (Philox counter-based, reproducible on GPU) |
| ML | gradient boosting, elastic net, k-means, PCA — MLX-backed |
| Graphics | scatter, line, histogram, kdensity, coefplot, binscatter — Swift Charts + custom Metal renderer for >1M points |
| Reproducibility | do-file editor, logging (`log using`), `set seed` honored across CPU & GPU |

---

## 2. App Name, Bundle & Identity

- **App name:** Metrika
- **Bundle ID:** `ai.spboucher.metrika`
- **Version scheme:** SemVer (`MARKETING_VERSION`), build number = CI run number.
- **Icon:** `Assets/AppIcon/metrika-icon.svg` is the master (see §11). Rasterize to all required sizes via `scripts/make_icns.sh`.

---

## 3. Architecture Overview

```
┌─────────────────────────────────────────────────────────┐
│                     MetrikaApp (SwiftUI)                  │
│  CommandConsole · DoFileEditor · DataBrowser · Plots    │
└──────────────────────────┬──────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────┐
│                    MetrikaKit (SPM package)               │
│                                                         │
│  ZQParser        command grammar → AST                  │
│  ZQPlanner       AST → ExecutionPlan (CPU/GPU/hybrid)   │
│  ZQEngine        orchestration, sessions, logging       │
│  ZQData          DataFrame façade over DuckDB/Arrow     │
│  ZQStats         CPU estimators (Accelerate/LAPACK)     │
│  ZQGPU           MLX batched estimators, RNG, kernels   │
│  ZQGraphics      plot spec → Swift Charts / Metal       │
│  ZQPlugins       user command discovery & sandboxing    │
└─────────────────────────────────────────────────────────┘
```

### Module rules
- `MetrikaKit` is a **Swift Package** with zero UI dependencies → fully testable via `swift test`.
- The app target only imports `MetrikaKit` and renders.
- `ZQGPU` is the **only** module allowed to import MLX/Metal. `ZQStats` is the only one importing Accelerate. This keeps backends swappable.

### Dependencies (SPM)
```
duckdb/duckdb-swift          — data engine
ml-explore/mlx-swift          — GPU tensor compute
apple/swift-collections       — deques, ordered dicts
pointfreeco/swift-parsing     — command grammar
apple/swift-argument-parser   — metrika-cli companion tool
```

---

## 4. The Command Language (ZQL)

Stata-compatible mental model, cleaner grammar:

```
command [varlist] [if expr] [in range] [weight] [, options]
```

Examples that MUST work in v1:

```
use sales.parquet
summarize revenue price, detail
gen log_rev = ln(revenue)
reg log_rev price i.region, robust
bootstrap, reps(100000) seed(42): reg log_rev price
xtset firm_id year
xtreg log_rev price, fe cluster(firm_id)
graph scatter log_rev price, by(region)
```

### Parser (`ZQParser`)
- Built with `swift-parsing`; grammar defined in `Sources/ZQParser/Grammar/`.
- Produces a typed AST: `ZQCommand { verb, varlist, condition, range, weight, options }`.
- Factor variables (`i.region`, `c.age#c.age`) expand at plan time, not parse time.
- Error messages must cite column position and suggest fixes (Levenshtein on known verbs).

### Home-made commands
Two tiers:
1. **Script commands** (`.zyq` files): sequences of ZQL with `args`/`syntax` declarations — the Stata ado-file analog.
2. **Native plugins**: Swift packages conforming to:

```swift
public protocol ZQCommandPlugin: Sendable {
    static var verb: String { get }
    static var syntax: ZQSyntaxSpec { get }
    func execute(_ ctx: ZQContext) async throws -> ZQResult
}
```

Discovery path: `~/Library/Application Support/Metrika/Commands/`. Plugins run in-process but receive a **read-only DataFrame view** unless they declare `mutates: true`.

---

## 5. Execution Planner — CPU vs GPU Dispatch

The planner decides the backend **automatically**. The user never chooses.

```swift
enum Backend { case cpu, gpu, hybrid }

func plan(_ cmd: ZQCommand, data: ZQDataFrame) -> ExecutionPlan {
    // Heuristics, tuned by benchmarks in Tests/Bench:
    // 1. Single estimation, n < 5_000_000  → .cpu (LAPACK QR, dqrls-style)
    // 2. reps ≥ 500 independent replicates → .gpu (batched solve)
    // 3. MCMC / iterative + large n        → .hybrid (GPU likelihood, CPU control)
}
```

### CPU path (`ZQStats`)
- OLS: QR via LAPACK `dgeqrf`/`dormqr` (never form X'X for conditioning reasons).
- Robust/cluster SE: sandwich estimators, computed with BLAS level-3.
- MLE (logit/probit/poisson): Newton–Raphson with analytic Hessians; IRLS fallback.
- All CPU compute in **Float64**.

### GPU path (`ZQGPU`, MLX)
- **Batched OLS**: resample indices as `(reps, n)` int tensor → gather → batched Cholesky/QR solve on `(reps, k, k)` — thousands of regressions per GPU pass.
- **RNG**: Philox4x32 counter-based generator → identical streams CPU/GPU given same seed; `set seed` maps to Philox key. Reproducibility is a release blocker.
- **Cluster bootstrap** (ragged n per replicate): pad-to-max with weight masks; custom Metal kernel only if profiling shows >2× overhead from padding.
- Precision: MLX float32 by default; accumulate cross-products in float64 emulation (Kahan/two-sum) where it matters; document tolerances per command.
- **Chunk reps** to respect memory: `chunk = min(reps, budget / (n*k*4))`, stream results, report progress to UI via `AsyncStream`.

### Hybrid
- Example: Bayesian regression — GPU computes log-likelihood over chains in parallel, CPU runs the sampler control loop.

---

## 6. Data Layer (`ZQData`)

- Canonical store: **DuckDB** in-process database; DataFrames are Arrow views over query results.
- Zero-copy handoff Arrow → MLX tensors where dtype allows; otherwise single cast pass.
- `use file.parquet` → registers a DuckDB view, lazy. `gen`/`replace` compile to SQL expressions when possible; fall back to vectorized Swift over Arrow buffers.
- Missing values: Arrow validity bitmaps end-to-end; every estimator does listwise deletion by default with an explicit report line ("(12 observations dropped due to missing values)").
- `.dta` reader/writer implemented natively in `ZQData/Stata/` (spec v117–v121).

---

## 7. UI (SwiftUI)

Windows/panes:
1. **Console** — command input with history (↑/↓), autocomplete on verbs & varnames, ANSI-style rich results (monospaced, aligned tables via `AttributedString`).
2. **Do-file editor** — `NSTextView` bridge, syntax highlighting for ZQL, ⌘R runs selection.
3. **Data browser** — virtualized grid (custom `NSTableView` bridge; SwiftUI `Table` is too slow past ~100k rows), sortable, filter bar compiles to `if` expressions.
4. **Plots** — Swift Charts for standard plots; custom `MTKView` renderer for scatter >1M points (instanced point sprites).
5. **Sidebar** — variables list with types, labels, missing counts.

State: `@Observable` session model; every command mutation goes through `ZQEngine` so console, do-files, and plugins share one execution path. Undo = dataset snapshots via DuckDB `CHECKPOINT` + copy-on-write.

---

## 8. Performance Budgets (release blockers)

| Operation | Data | Budget (M3 Pro) | Measured (M4-class, 2026-08-05) |
|---|---|---|---|
| `use` parquet | 10M rows × 20 cols | < 1.5 s | 0.20 s (10M × 4, incl. process start) |
| `use` csv | 10M rows × 4 cols | — | 0.31 s |
| `summarize` all vars | 10M × 20 | < 300 ms | ~270 ms (10M × 4) |
| `reg` 5 covariates | 10M rows | < 900 ms | ~240 ms (1 covariate) |
| `bootstrap, reps(10000): reg` | 100k rows | < 3 s | reps(1000) ≈ 0.1 s CPU |
| `bootstrap, reps(100000): reg` | 100k rows | < 25 s | GPU-batched in app |
| Scatter render | 2M points | 60 fps pan/zoom | renders 2M via Metal |

Loads go through the DuckDB C-API bulk reader (contiguous column
memcpy); the duckdb-swift per-element path was O(rows × chunks) and took
276 s at 10M rows.

Benchmarks live in `Tests/Bench/` and run in CI on self-hosted Apple Silicon runner; regressions >10% fail the build.

---

## 9. Testing

- `swift test` on MetrikaKit: parser golden tests, estimator numerical tests vs fixtures generated by R (`Tests/Fixtures/generate.R`).
- Property tests: OLS invariance (scaling, permutation), bootstrap CI coverage simulation (n small, known DGP).
- GPU vs CPU cross-checks: same seed ⇒ identical resample indices; estimates within documented tolerance.
- UI: XCUITest smoke — launch, load sample dataset, run `reg`, assert output table.

---

## 10. Build, Signing, Notarization & DMG

> ⚠️ **PLACEHOLDERS** — the `Zyquo-local/` folder containing the real notarization credentials was not available when this file was generated. Fill every `<...>` below from `Zyquo-local/notarization.md` (or equivalent) before the first release build. Do **not** commit real credentials; keep them in the local keychain / CI secrets.

### Identifiers
```
DEVELOPMENT_TEAM      = <TEAM_ID>                     # from Metrika-local
CODE_SIGN_IDENTITY    = "Developer ID Application: Simon-Pierre Boucher (<TEAM_ID>)"
NOTARY_PROFILE        = "<KEYCHAIN_PROFILE_NAME>"     # created via: xcrun notarytool store-credentials
BUNDLE_ID             = ai.spboucher.metrika
```

### One-time setup
```bash
xcrun notarytool store-credentials "<KEYCHAIN_PROFILE_NAME>" \
  --apple-id "<APPLE_ID_EMAIL>" \
  --team-id "<TEAM_ID>" \
  --password "<APP_SPECIFIC_PASSWORD>"
```

### Release pipeline — `scripts/release.sh`
```bash
#!/bin/bash
#
#  release.sh — Metrika
#  Author:  Simon-Pierre Boucher
#  Contact: contact@spboucher.ai
#
set -euo pipefail

APP="Metrika"
SCHEME="Metrika"
BUILD_DIR="build"
DMG="${APP}.dmg"

# 1. Archive
xcodebuild -scheme "$SCHEME" -configuration Release -arch arm64 \
  -archivePath "$BUILD_DIR/$APP.xcarchive" archive

# 2. Export with Developer ID
xcodebuild -exportArchive \
  -archivePath "$BUILD_DIR/$APP.xcarchive" \
  -exportOptionsPlist scripts/ExportOptions.plist \
  -exportPath "$BUILD_DIR/export"

# 3. Verify hardened runtime + entitlements
codesign -dv --verbose=4 "$BUILD_DIR/export/$APP.app"
codesign --verify --deep --strict "$BUILD_DIR/export/$APP.app"

# 4. Build DMG (create-dmg, background art in Assets/DMG/)
create-dmg \
  --volname "$APP" \
  --window-size 540 380 \
  --icon-size 128 \
  --icon "$APP.app" 130 190 \
  --app-drop-link 400 190 \
  --background "Assets/DMG/background.png" \
  "$DMG" "$BUILD_DIR/export/"

# 5. Sign the DMG itself
codesign --sign "Developer ID Application: Simon-Pierre Boucher (<TEAM_ID>)" "$DMG"

# 6. Notarize & wait
xcrun notarytool submit "$DMG" \
  --keychain-profile "<KEYCHAIN_PROFILE_NAME>" \
  --wait

# 7. Staple
xcrun stapler staple "$DMG"
xcrun stapler validate "$DMG"

echo "✅ $DMG notarized and stapled."
```

### `scripts/ExportOptions.plist`
```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
 "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>method</key><string>developer-id</string>
  <key>teamID</key><string><TEAM_ID></string>
  <key>signingStyle</key><string>automatic</string>
  <key>destination</key><string>export</string>
</dict></plist>
```

### Entitlements (`Metrika.entitlements`)
- Hardened Runtime: ON.
- App Sandbox: ON, with `com.apple.security.files.user-selected.read-write` (open/save datasets) and bookmark entitlements for recent files.
- No JIT, no unsigned executable memory (MLX does not require them).
- If plugin loading of external dylibs is added later, revisit `com.apple.security.cs.disable-library-validation` — avoid if possible; prefer SPM-compiled-in plugins.

---

## 11. App Icon

- Master: `Assets/AppIcon/metrika-icon.svg` (1024×1024 design).
- Concept: a glowing **M-shaped time-series line** — a zigzag chart trace cutting through a scatter of data points on a deep space-gradient squircle — econometrics meets GPU power.
- Generate `.icns`: `scripts/make_icns.sh` rasterizes the SVG (via `rsvg-convert` or `qlmanage`) to 16→1024 px and runs `iconutil -c icns`.
- macOS Sonoma+ icon grid: content within the 824×824 safe zone of the 1024 canvas; system applies the squircle mask — but we bake our own rounded-rect background per Big Sur+ convention.

---

## 12. Repo Conventions

```
Metrika/
├── CLAUDE.md
├── Metrika.xcodeproj
├── Metrika/                    # app target (UI)
├── MetrikaKit/                 # SPM package (all logic)
│   └── Sources/{ZQParser,ZQPlanner,ZQEngine,ZQData,ZQStats,ZQGPU,ZQGraphics,ZQPlugins}
├── Assets/{AppIcon,DMG}
├── scripts/{release.sh,make_icns.sh,check_headers.sh}
└── Tests/{Unit,Bench,Fixtures}
```

- **Pre-commit hook** runs `scripts/check_headers.sh`: rejects any staged source file (`.swift`, `.metal`, `.sh`, `.py`, `.zyq`) missing the `Author: Simon-Pierre Boucher` / `Contact: contact@spboucher.ai` header.
- **Build quirks (mlx-swift)**: SwiftPM CLI cannot compile Metal shaders, so `swift test` skips the GPU suites (CPU fallback via metallib detection); run `xcodebuild test -scheme MetrikaKit-Package -destination 'platform=macOS' -skipPackagePluginValidation` from `MetrikaKit/` for full coverage. All `xcodebuild` invocations (app and tests) need `-skipPackagePluginValidation` for mlx-swift's CudaBuild plugin.
- Commits: Conventional Commits (`feat(parser): factor variable expansion`).
- Branches: `main` (protected), `dev`, feature branches `feat/*`.
- CI: GitHub Actions on self-hosted arm64 macOS runner — build, test, bench, header check.

---

## 13. Roadmap

**v0.1 (MVP, ~8 weeks of focused work)**
use/save (parquet, csv, dta) · summarize · tabulate · gen/replace · reg (robust, cluster) · logit · basic scatter/histogram · console + data browser · do-file execution.

**v0.2** — GPU bootstrap + permutation tests · xtreg FE · ivregress · Swift Charts full suite · plugin protocol.

**v0.3** — MCMC/Bayesian module · gradient boosting & elastic net (MLX) · Metal 2M-point renderer · margins/predict.

**v1.0** — notarized DMG release, docs site, sample datasets, command reference (`help regress` in-app).

---

## 14. What Claude Should Do When Working in This Repo

1. Read this file first; respect §0 rules absolutely (headers, Swift 6 strict concurrency).
2. Before touching numerics, read the matching fixture in `Tests/Fixtures/` and keep tests green.
3. Any new command: grammar entry → AST case → planner rule → CPU impl → (optional) GPU impl → golden test → in-app help entry. All six or the PR is incomplete.
4. Never introduce a Python or Node runtime dependency into the app bundle.
5. When unsure between CPU and GPU, implement CPU first — correctness before speed — then add the GPU path behind the planner.
6. Update the performance table (§8) whenever a benchmark materially changes.