spb/metrika Public

1# CLAUDE.md — Metrika23> **Metrika** — A next-generation statistical & data science application for macOS.4> Stata-class command syntax, GPU-accelerated by Apple Silicon (MLX/Metal), powered by DuckDB.5> Native Swift + SwiftUI. No Electron. No Python runtime dependency. No compromises.67---89## 0. Non-Negotiable Project Rules10111. **Every source file MUST begin with this header** (adapt comment style to the language):1213```swift14//15//  <FileName>.swift16//  Metrika17//18//  Author:  Simon-Pierre Boucher19//  Contact: contact@spboucher.ai20//  Copyright © 2026 Simon-Pierre Boucher. All rights reserved.21//22```2324For Metal shaders (`.metal`), shell scripts, and Python tooling scripts, use the equivalent comment syntax (`//` or `#`). **No file is committed without this header.** A pre-commit hook (see §12) enforces it.25262. Language: **Swift 6**, strict concurrency enabled (`-strict-concurrency=complete`).273. UI: **SwiftUI first**, AppKit bridges (`NSViewRepresentable`) only where SwiftUI is insufficient (text editor, data grid virtualization).284. Minimum target: **macOS 14 (Sonoma)**, Apple Silicon only (`arm64`). No Intel builds.295. All numerical results must match reference implementations (R / Stata) to at least 1e-10 relative tolerance for CPU paths; GPU paths documented tolerance 1e-6 (float32) or 1e-10 (float64 where MLX supports it).306. English for all code, comments, commit messages, and docs.3132---3334## 1. Product Vision3536Metrika is a complete data science environment:3738- **Command-driven**, like Stata: `reg y x1 x2, robust cluster(id)` — one line, publication-ready output.39- **Scriptable**: `.zyq` do-files with full reproducibility (logs, seeds, versioned datasets).40- **GPU-native**: massively parallel workloads (bootstrap, Monte Carlo, MCMC, ML training, permutation tests) run on Apple GPU via MLX with unified memory.41- **Extensible**: user-defined commands ("home-made" commands) via a plugin protocol — drop a Swift package or a `.zyq` ado-style script into `~/Library/Application Support/Metrika/Commands/`.42- **Big-data capable**: DuckDB engine → out-of-core, columnar, multi-threaded. 100M+ rows on a laptop.4344### Feature pillars (v1 scope)4546| Pillar | Contents |47|---|---|48| Data management | import/export CSV, Parquet, Arrow, Excel, Stata `.dta`, JSON; merge, append, reshape, collapse, egen-style generators |49| Estimation | summarize, tabulate, correlate, `regress` (OLS), `logit`, `probit`, `poisson`, `ivregress` (2SLS), `xtreg` (FE/RE), `areg`, quantile regression |50| Inference | robust (HC0–HC3), cluster-robust SE, **GPU bootstrap** (pairs, wild, cluster), permutation tests, jackknife |51| Simulation | Monte Carlo engine, random number generation (Philox counter-based, reproducible on GPU) |52| ML | gradient boosting, elastic net, k-means, PCA — MLX-backed |53| Graphics | scatter, line, histogram, kdensity, coefplot, binscatter — Swift Charts + custom Metal renderer for >1M points |54| Reproducibility | do-file editor, logging (`log using`), `set seed` honored across CPU & GPU |5556---5758## 2. App Name, Bundle & Identity5960- **App name:** Metrika61- **Bundle ID:** `ai.spboucher.metrika`62- **Version scheme:** SemVer (`MARKETING_VERSION`), build number = CI run number.63- **Icon:** `Assets/AppIcon/metrika-icon.svg` is the master (see §11). Rasterize to all required sizes via `scripts/make_icns.sh`.6465---6667## 3. Architecture Overview6869```70┌─────────────────────────────────────────────────────────┐71│                     MetrikaApp (SwiftUI)                  │72│  CommandConsole · DoFileEditor · DataBrowser · Plots    │73└──────────────────────────┬──────────────────────────────┘74                           │75┌──────────────────────────▼──────────────────────────────┐76│                    MetrikaKit (SPM package)               │77│                                                         │78│  ZQParser        command grammar → AST                  │79│  ZQPlanner       AST → ExecutionPlan (CPU/GPU/hybrid)   │80│  ZQEngine        orchestration, sessions, logging       │81│  ZQData          DataFrame façade over DuckDB/Arrow     │82│  ZQStats         CPU estimators (Accelerate/LAPACK)     │83│  ZQGPU           MLX batched estimators, RNG, kernels   │84│  ZQGraphics      plot spec → Swift Charts / Metal       │85│  ZQPlugins       user command discovery & sandboxing    │86└─────────────────────────────────────────────────────────┘87```8889### Module rules90- `MetrikaKit` is a **Swift Package** with zero UI dependencies → fully testable via `swift test`.91- The app target only imports `MetrikaKit` and renders.92- `ZQGPU` is the **only** module allowed to import MLX/Metal. `ZQStats` is the only one importing Accelerate. This keeps backends swappable.9394### Dependencies (SPM)95```96duckdb/duckdb-swift          — data engine97ml-explore/mlx-swift          — GPU tensor compute98apple/swift-collections       — deques, ordered dicts99pointfreeco/swift-parsing     — command grammar100apple/swift-argument-parser   — metrika-cli companion tool101```102103---104105## 4. The Command Language (ZQL)106107Stata-compatible mental model, cleaner grammar:108109```110command [varlist] [if expr] [in range] [weight] [, options]111```112113Examples that MUST work in v1:114115```116use sales.parquet117summarize revenue price, detail118gen log_rev = ln(revenue)119reg log_rev price i.region, robust120bootstrap, reps(100000) seed(42): reg log_rev price121xtset firm_id year122xtreg log_rev price, fe cluster(firm_id)123graph scatter log_rev price, by(region)124```125126### Parser (`ZQParser`)127- Built with `swift-parsing`; grammar defined in `Sources/ZQParser/Grammar/`.128- Produces a typed AST: `ZQCommand { verb, varlist, condition, range, weight, options }`.129- Factor variables (`i.region`, `c.age#c.age`) expand at plan time, not parse time.130- Error messages must cite column position and suggest fixes (Levenshtein on known verbs).131132### Home-made commands133Two tiers:1341. **Script commands** (`.zyq` files): sequences of ZQL with `args`/`syntax` declarations — the Stata ado-file analog.1352. **Native plugins**: Swift packages conforming to:136137```swift138public protocol ZQCommandPlugin: Sendable {139    static var verb: String { get }140    static var syntax: ZQSyntaxSpec { get }141    func execute(_ ctx: ZQContext) async throws -> ZQResult142}143```144145Discovery path: `~/Library/Application Support/Metrika/Commands/`. Plugins run in-process but receive a **read-only DataFrame view** unless they declare `mutates: true`.146147---148149## 5. Execution Planner — CPU vs GPU Dispatch150151The planner decides the backend **automatically**. The user never chooses.152153```swift154enum Backend { case cpu, gpu, hybrid }155156func plan(_ cmd: ZQCommand, data: ZQDataFrame) -> ExecutionPlan {157    // Heuristics, tuned by benchmarks in Tests/Bench:158    // 1. Single estimation, n < 5_000_000  → .cpu (LAPACK QR, dqrls-style)159    // 2. reps ≥ 500 independent replicates → .gpu (batched solve)160    // 3. MCMC / iterative + large n        → .hybrid (GPU likelihood, CPU control)161}162```163164### CPU path (`ZQStats`)165- OLS: QR via LAPACK `dgeqrf`/`dormqr` (never form X'X for conditioning reasons).166- Robust/cluster SE: sandwich estimators, computed with BLAS level-3.167- MLE (logit/probit/poisson): Newton–Raphson with analytic Hessians; IRLS fallback.168- All CPU compute in **Float64**.169170### GPU path (`ZQGPU`, MLX)171- **Batched OLS**: resample indices as `(reps, n)` int tensor → gather → batched Cholesky/QR solve on `(reps, k, k)` — thousands of regressions per GPU pass.172- **RNG**: Philox4x32 counter-based generator → identical streams CPU/GPU given same seed; `set seed` maps to Philox key. Reproducibility is a release blocker.173- **Cluster bootstrap** (ragged n per replicate): pad-to-max with weight masks; custom Metal kernel only if profiling shows >2× overhead from padding.174- Precision: MLX float32 by default; accumulate cross-products in float64 emulation (Kahan/two-sum) where it matters; document tolerances per command.175- **Chunk reps** to respect memory: `chunk = min(reps, budget / (n*k*4))`, stream results, report progress to UI via `AsyncStream`.176177### Hybrid178- Example: Bayesian regression — GPU computes log-likelihood over chains in parallel, CPU runs the sampler control loop.179180---181182## 6. Data Layer (`ZQData`)183184- Canonical store: **DuckDB** in-process database; DataFrames are Arrow views over query results.185- Zero-copy handoff Arrow → MLX tensors where dtype allows; otherwise single cast pass.186- `use file.parquet` → registers a DuckDB view, lazy. `gen`/`replace` compile to SQL expressions when possible; fall back to vectorized Swift over Arrow buffers.187- Missing values: Arrow validity bitmaps end-to-end; every estimator does listwise deletion by default with an explicit report line ("(12 observations dropped due to missing values)").188- `.dta` reader/writer implemented natively in `ZQData/Stata/` (spec v117–v121).189190---191192## 7. UI (SwiftUI)193194Windows/panes:1951. **Console** — command input with history (↑/↓), autocomplete on verbs & varnames, ANSI-style rich results (monospaced, aligned tables via `AttributedString`).1962. **Do-file editor** — `NSTextView` bridge, syntax highlighting for ZQL, ⌘R runs selection.1973. **Data browser** — virtualized grid (custom `NSTableView` bridge; SwiftUI `Table` is too slow past ~100k rows), sortable, filter bar compiles to `if` expressions.1984. **Plots** — Swift Charts for standard plots; custom `MTKView` renderer for scatter >1M points (instanced point sprites).1995. **Sidebar** — variables list with types, labels, missing counts.200201State: `@Observable` session model; every command mutation goes through `ZQEngine` so console, do-files, and plugins share one execution path. Undo = dataset snapshots via DuckDB `CHECKPOINT` + copy-on-write.202203---204205## 8. Performance Budgets (release blockers)206207| Operation | Data | Budget (M3 Pro) | Measured (M4-class, 2026-08-05) |208|---|---|---|---|209| `use` parquet | 10M rows × 20 cols | < 1.5 s | 0.20 s (10M × 4, incl. process start) |210| `use` csv | 10M rows × 4 cols | — | 0.31 s |211| `summarize` all vars | 10M × 20 | < 300 ms | ~270 ms (10M × 4) |212| `reg` 5 covariates | 10M rows | < 900 ms | ~240 ms (1 covariate) |213| `bootstrap, reps(10000): reg` | 100k rows | < 3 s | reps(1000) ≈ 0.1 s CPU |214| `bootstrap, reps(100000): reg` | 100k rows | < 25 s | GPU-batched in app |215| Scatter render | 2M points | 60 fps pan/zoom | renders 2M via Metal |216217Loads go through the DuckDB C-API bulk reader (contiguous column218memcpy); the duckdb-swift per-element path was O(rows × chunks) and took219276 s at 10M rows.220221Benchmarks live in `Tests/Bench/` and run in CI on self-hosted Apple Silicon runner; regressions >10% fail the build.222223---224225## 9. Testing226227- `swift test` on MetrikaKit: parser golden tests, estimator numerical tests vs fixtures generated by R (`Tests/Fixtures/generate.R`).228- Property tests: OLS invariance (scaling, permutation), bootstrap CI coverage simulation (n small, known DGP).229- GPU vs CPU cross-checks: same seed ⇒ identical resample indices; estimates within documented tolerance.230- UI: XCUITest smoke — launch, load sample dataset, run `reg`, assert output table.231232---233234## 10. Build, Signing, Notarization & DMG235236> ⚠️ **PLACEHOLDERS** — the `Zyquo-local/` folder containing the real notarization credentials was not available when this file was generated. Fill every `<...>` below from `Zyquo-local/notarization.md` (or equivalent) before the first release build. Do **not** commit real credentials; keep them in the local keychain / CI secrets.237238### Identifiers239```240DEVELOPMENT_TEAM      = <TEAM_ID>                     # from Metrika-local241CODE_SIGN_IDENTITY    = "Developer ID Application: Simon-Pierre Boucher (<TEAM_ID>)"242NOTARY_PROFILE        = "<KEYCHAIN_PROFILE_NAME>"     # created via: xcrun notarytool store-credentials243BUNDLE_ID             = ai.spboucher.metrika244```245246### One-time setup247```bash248xcrun notarytool store-credentials "<KEYCHAIN_PROFILE_NAME>" \249  --apple-id "<APPLE_ID_EMAIL>" \250  --team-id "<TEAM_ID>" \251  --password "<APP_SPECIFIC_PASSWORD>"252```253254### Release pipeline — `scripts/release.sh`255```bash256#!/bin/bash257#258#  release.sh — Metrika259#  Author:  Simon-Pierre Boucher260#  Contact: contact@spboucher.ai261#262set -euo pipefail263264APP="Metrika"265SCHEME="Metrika"266BUILD_DIR="build"267DMG="${APP}.dmg"268269# 1. Archive270xcodebuild -scheme "$SCHEME" -configuration Release -arch arm64 \271  -archivePath "$BUILD_DIR/$APP.xcarchive" archive272273# 2. Export with Developer ID274xcodebuild -exportArchive \275  -archivePath "$BUILD_DIR/$APP.xcarchive" \276  -exportOptionsPlist scripts/ExportOptions.plist \277  -exportPath "$BUILD_DIR/export"278279# 3. Verify hardened runtime + entitlements280codesign -dv --verbose=4 "$BUILD_DIR/export/$APP.app"281codesign --verify --deep --strict "$BUILD_DIR/export/$APP.app"282283# 4. Build DMG (create-dmg, background art in Assets/DMG/)284create-dmg \285  --volname "$APP" \286  --window-size 540 380 \287  --icon-size 128 \288  --icon "$APP.app" 130 190 \289  --app-drop-link 400 190 \290  --background "Assets/DMG/background.png" \291  "$DMG" "$BUILD_DIR/export/"292293# 5. Sign the DMG itself294codesign --sign "Developer ID Application: Simon-Pierre Boucher (<TEAM_ID>)" "$DMG"295296# 6. Notarize & wait297xcrun notarytool submit "$DMG" \298  --keychain-profile "<KEYCHAIN_PROFILE_NAME>" \299  --wait300301# 7. Staple302xcrun stapler staple "$DMG"303xcrun stapler validate "$DMG"304305echo "✅ $DMG notarized and stapled."306```307308### `scripts/ExportOptions.plist`309```xml310<?xml version="1.0" encoding="UTF-8"?>311<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"312 "http://www.apple.com/DTDs/PropertyList-1.0.dtd">313<plist version="1.0"><dict>314  <key>method</key><string>developer-id</string>315  <key>teamID</key><string><TEAM_ID></string>316  <key>signingStyle</key><string>automatic</string>317  <key>destination</key><string>export</string>318</dict></plist>319```320321### Entitlements (`Metrika.entitlements`)322- Hardened Runtime: ON.323- App Sandbox: ON, with `com.apple.security.files.user-selected.read-write` (open/save datasets) and bookmark entitlements for recent files.324- No JIT, no unsigned executable memory (MLX does not require them).325- If plugin loading of external dylibs is added later, revisit `com.apple.security.cs.disable-library-validation` — avoid if possible; prefer SPM-compiled-in plugins.326327---328329## 11. App Icon330331- Master: `Assets/AppIcon/metrika-icon.svg` (1024×1024 design).332- Concept: a glowing **M-shaped time-series line** — a zigzag chart trace cutting through a scatter of data points on a deep space-gradient squircle — econometrics meets GPU power.333- Generate `.icns`: `scripts/make_icns.sh` rasterizes the SVG (via `rsvg-convert` or `qlmanage`) to 16→1024 px and runs `iconutil -c icns`.334- macOS Sonoma+ icon grid: content within the 824×824 safe zone of the 1024 canvas; system applies the squircle mask — but we bake our own rounded-rect background per Big Sur+ convention.335336---337338## 12. Repo Conventions339340```341Metrika/342├── CLAUDE.md343├── Metrika.xcodeproj344├── Metrika/                    # app target (UI)345├── MetrikaKit/                 # SPM package (all logic)346│   └── Sources/{ZQParser,ZQPlanner,ZQEngine,ZQData,ZQStats,ZQGPU,ZQGraphics,ZQPlugins}347├── Assets/{AppIcon,DMG}348├── scripts/{release.sh,make_icns.sh,check_headers.sh}349└── Tests/{Unit,Bench,Fixtures}350```351352- **Pre-commit hook** runs `scripts/check_headers.sh`: rejects any staged source file (`.swift`, `.metal`, `.sh`, `.py`, `.zyq`) missing the `Author: Simon-Pierre Boucher` / `Contact: contact@spboucher.ai` header.353- **Build quirks (mlx-swift)**: SwiftPM CLI cannot compile Metal shaders, so `swift test` skips the GPU suites (CPU fallback via metallib detection); run `xcodebuild test -scheme MetrikaKit-Package -destination 'platform=macOS' -skipPackagePluginValidation` from `MetrikaKit/` for full coverage. All `xcodebuild` invocations (app and tests) need `-skipPackagePluginValidation` for mlx-swift's CudaBuild plugin.354- Commits: Conventional Commits (`feat(parser): factor variable expansion`).355- Branches: `main` (protected), `dev`, feature branches `feat/*`.356- CI: GitHub Actions on self-hosted arm64 macOS runner — build, test, bench, header check.357358---359360## 13. Roadmap361362**v0.1 (MVP, ~8 weeks of focused work)**363use/save (parquet, csv, dta) · summarize · tabulate · gen/replace · reg (robust, cluster) · logit · basic scatter/histogram · console + data browser · do-file execution.364365**v0.2** — GPU bootstrap + permutation tests · xtreg FE · ivregress · Swift Charts full suite · plugin protocol.366367**v0.3** — MCMC/Bayesian module · gradient boosting & elastic net (MLX) · Metal 2M-point renderer · margins/predict.368369**v1.0** — notarized DMG release, docs site, sample datasets, command reference (`help regress` in-app).370371---372373## 14. What Claude Should Do When Working in This Repo3743751. Read this file first; respect §0 rules absolutely (headers, Swift 6 strict concurrency).3762. Before touching numerics, read the matching fixture in `Tests/Fixtures/` and keep tests green.3773. Any new command: grammar entry → AST case → planner rule → CPU impl → (optional) GPU impl → golden test → in-app help entry. All six or the PR is incomplete.3784. Never introduce a Python or Node runtime dependency into the app bundle.3795. When unsure between CPU and GPU, implement CPU first — correctness before speed — then add the GPU path behind the planner.3806. Update the performance table (§8) whenever a benchmark materially changes.381