# CLAUDE.md — Metrika

Metrika — A next-generation statistical & data science application for macOS. Stata-class command syntax, GPU-accelerated by Apple Silicon (MLX/Metal), powered by DuckDB. Native Swift + SwiftUI. No Electron. No Python runtime dependency. No compromises.

# 0. Non-Negotiable Project Rules

1. Every source file MUST begin with this header (adapt comment style to the language):

//
//  <FileName>.swift
//  Metrika
//
//  Author:  Simon-Pierre Boucher
//  Contact: contact@spboucher.ai
//  Copyright © 2026 Simon-Pierre Boucher. All rights reserved.
//

For Metal shaders (.metal), shell scripts, and Python tooling scripts, use the equivalent comment syntax (// or #). No file is committed without this header. A pre-commit hook (see §12) enforces it.

2. Language: Swift 6, strict concurrency enabled (-strict-concurrency=complete).
3. UI: SwiftUI first, AppKit bridges (NSViewRepresentable) only where SwiftUI is insufficient (text editor, data grid virtualization).
4. Minimum target: macOS 14 (Sonoma), Apple Silicon only (arm64). No Intel builds.
5. All numerical results must match reference implementations (R / Stata) to at least 1e-10 relative tolerance for CPU paths; GPU paths documented tolerance 1e-6 (float32) or 1e-10 (float64 where MLX supports it).
6. English for all code, comments, commit messages, and docs.

# 1. Product Vision

Metrika is a complete data science environment:

• Command-driven, like Stata: reg y x1 x2, robust cluster(id) — one line, publication-ready output.
• Scriptable: .zyq do-files with full reproducibility (logs, seeds, versioned datasets).
• GPU-native: massively parallel workloads (bootstrap, Monte Carlo, MCMC, ML training, permutation tests) run on Apple GPU via MLX with unified memory.
• Extensible: user-defined commands ("home-made" commands) via a plugin protocol — drop a Swift package or a .zyq ado-style script into ~/Library/Application Support/Metrika/Commands/.
• Big-data capable: DuckDB engine → out-of-core, columnar, multi-threaded. 100M+ rows on a laptop.

# Feature pillars (v1 scope)

# 2. App Name, Bundle & Identity

• App name: Metrika
• Bundle ID: ai.spboucher.metrika
• Version scheme: SemVer (MARKETING_VERSION), build number = CI run number.
• Icon: Assets/AppIcon/metrika-icon.svg is the master (see §11). Rasterize to all required sizes via scripts/make_icns.sh.

# 3. Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                     MetrikaApp (SwiftUI)                  │
│  CommandConsole · DoFileEditor · DataBrowser · Plots    │
└──────────────────────────┬──────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────┐
│                    MetrikaKit (SPM package)               │
│                                                         │
│  ZQParser        command grammar → AST                  │
│  ZQPlanner       AST → ExecutionPlan (CPU/GPU/hybrid)   │
│  ZQEngine        orchestration, sessions, logging       │
│  ZQData          DataFrame façade over DuckDB/Arrow     │
│  ZQStats         CPU estimators (Accelerate/LAPACK)     │
│  ZQGPU           MLX batched estimators, RNG, kernels   │
│  ZQGraphics      plot spec → Swift Charts / Metal       │
│  ZQPlugins       user command discovery & sandboxing    │
└─────────────────────────────────────────────────────────┘

# Module rules

• MetrikaKit is a Swift Package with zero UI dependencies → fully testable via swift test.
• The app target only imports MetrikaKit and renders.
• ZQGPU is the only module allowed to import MLX/Metal. ZQStats is the only one importing Accelerate. This keeps backends swappable.

# Dependencies (SPM)

duckdb/duckdb-swift          — data engine
ml-explore/mlx-swift          — GPU tensor compute
apple/swift-collections       — deques, ordered dicts
pointfreeco/swift-parsing     — command grammar
apple/swift-argument-parser   — metrika-cli companion tool

# 4. The Command Language (ZQL)

Stata-compatible mental model, cleaner grammar:

command [varlist] [if expr] [in range] [weight] [, options]

Examples that MUST work in v1:

use sales.parquet
summarize revenue price, detail
gen log_rev = ln(revenue)
reg log_rev price i.region, robust
bootstrap, reps(100000) seed(42): reg log_rev price
xtset firm_id year
xtreg log_rev price, fe cluster(firm_id)
graph scatter log_rev price, by(region)

# Parser (ZQParser)

• Built with swift-parsing; grammar defined in Sources/ZQParser/Grammar/.
• Produces a typed AST: ZQCommand { verb, varlist, condition, range, weight, options }.
• Factor variables (i.region, c.age#c.age) expand at plan time, not parse time.
• Error messages must cite column position and suggest fixes (Levenshtein on known verbs).

# Home-made commands

Two tiers:

1. Script commands (.zyq files): sequences of ZQL with args/syntax declarations — the Stata ado-file analog.
2. Native plugins: Swift packages conforming to:

public protocol ZQCommandPlugin: Sendable {
    static var verb: String { get }
    static var syntax: ZQSyntaxSpec { get }
    func execute(_ ctx: ZQContext) async throws -> ZQResult
}

Discovery path: ~/Library/Application Support/Metrika/Commands/. Plugins run in-process but receive a read-only DataFrame view unless they declare mutates: true.

# 5. Execution Planner — CPU vs GPU Dispatch

The planner decides the backend automatically. The user never chooses.

enum Backend { case cpu, gpu, hybrid }

func plan(_ cmd: ZQCommand, data: ZQDataFrame) -> ExecutionPlan {
    // Heuristics, tuned by benchmarks in Tests/Bench:
    // 1. Single estimation, n < 5_000_000  → .cpu (LAPACK QR, dqrls-style)
    // 2. reps ≥ 500 independent replicates → .gpu (batched solve)
    // 3. MCMC / iterative + large n        → .hybrid (GPU likelihood, CPU control)
}

# CPU path (ZQStats)

• OLS: QR via LAPACK dgeqrf/dormqr (never form X'X for conditioning reasons).
• Robust/cluster SE: sandwich estimators, computed with BLAS level-3.
• MLE (logit/probit/poisson): Newton–Raphson with analytic Hessians; IRLS fallback.
• All CPU compute in Float64.

# GPU path (ZQGPU, MLX)

• Batched OLS: resample indices as (reps, n) int tensor → gather → batched Cholesky/QR solve on (reps, k, k) — thousands of regressions per GPU pass.
• RNG: Philox4x32 counter-based generator → identical streams CPU/GPU given same seed; set seed maps to Philox key. Reproducibility is a release blocker.
• Cluster bootstrap (ragged n per replicate): pad-to-max with weight masks; custom Metal kernel only if profiling shows >2× overhead from padding.
• Precision: MLX float32 by default; accumulate cross-products in float64 emulation (Kahan/two-sum) where it matters; document tolerances per command.
• Chunk reps to respect memory: chunk = min(reps, budget / (n*k*4)), stream results, report progress to UI via AsyncStream.

# Hybrid

• Example: Bayesian regression — GPU computes log-likelihood over chains in parallel, CPU runs the sampler control loop.

# 6. Data Layer (ZQData)

• Canonical store: DuckDB in-process database; DataFrames are Arrow views over query results.
• Zero-copy handoff Arrow → MLX tensors where dtype allows; otherwise single cast pass.
• use file.parquet → registers a DuckDB view, lazy. gen/replace compile to SQL expressions when possible; fall back to vectorized Swift over Arrow buffers.
• Missing values: Arrow validity bitmaps end-to-end; every estimator does listwise deletion by default with an explicit report line ("(12 observations dropped due to missing values)").
• .dta reader/writer implemented natively in ZQData/Stata/ (spec v117–v121).

# 7. UI (SwiftUI)

Windows/panes:

1. Console — command input with history (↑/↓), autocomplete on verbs & varnames, ANSI-style rich results (monospaced, aligned tables via AttributedString).
2. Do-file editor — NSTextView bridge, syntax highlighting for ZQL, ⌘R runs selection.
3. Data browser — virtualized grid (custom NSTableView bridge; SwiftUI Table is too slow past ~100k rows), sortable, filter bar compiles to if expressions.
4. Plots — Swift Charts for standard plots; custom MTKView renderer for scatter >1M points (instanced point sprites).
5. Sidebar — variables list with types, labels, missing counts.

State: @Observable session model; every command mutation goes through ZQEngine so console, do-files, and plugins share one execution path. Undo = dataset snapshots via DuckDB CHECKPOINT + copy-on-write.

# 8. Performance Budgets (release blockers)

Loads go through the DuckDB C-API bulk reader (contiguous column memcpy); the duckdb-swift per-element path was O(rows × chunks) and took 276 s at 10M rows.

Benchmarks live in Tests/Bench/ and run in CI on self-hosted Apple Silicon runner; regressions >10% fail the build.

# 9. Testing

• swift test on MetrikaKit: parser golden tests, estimator numerical tests vs fixtures generated by R (Tests/Fixtures/generate.R).
• Property tests: OLS invariance (scaling, permutation), bootstrap CI coverage simulation (n small, known DGP).
• GPU vs CPU cross-checks: same seed ⇒ identical resample indices; estimates within documented tolerance.
• UI: XCUITest smoke — launch, load sample dataset, run reg, assert output table.

# 10. Build, Signing, Notarization & DMG

⚠️ PLACEHOLDERS — the Zyquo-local/ folder containing the real notarization credentials was not available when this file was generated. Fill every <...> below from Zyquo-local/notarization.md (or equivalent) before the first release build. Do not commit real credentials; keep them in the local keychain / CI secrets.

# Identifiers

DEVELOPMENT_TEAM      = <TEAM_ID>                     # from Metrika-local
CODE_SIGN_IDENTITY    = "Developer ID Application: Simon-Pierre Boucher (<TEAM_ID>)"
NOTARY_PROFILE        = "<KEYCHAIN_PROFILE_NAME>"     # created via: xcrun notarytool store-credentials
BUNDLE_ID             = ai.spboucher.metrika

# One-time setup

xcrun notarytool store-credentials "<KEYCHAIN_PROFILE_NAME>" \
  --apple-id "<APPLE_ID_EMAIL>" \
  --team-id "<TEAM_ID>" \
  --password "<APP_SPECIFIC_PASSWORD>"

# Release pipeline — scripts/release.sh

#!/bin/bash
#
#  release.sh — Metrika
#  Author:  Simon-Pierre Boucher
#  Contact: contact@spboucher.ai
#
set -euo pipefail

APP="Metrika"
SCHEME="Metrika"
BUILD_DIR="build"
DMG="${APP}.dmg"

# 1. Archive
xcodebuild -scheme "$SCHEME" -configuration Release -arch arm64 \
  -archivePath "$BUILD_DIR/$APP.xcarchive" archive

# 2. Export with Developer ID
xcodebuild -exportArchive \
  -archivePath "$BUILD_DIR/$APP.xcarchive" \
  -exportOptionsPlist scripts/ExportOptions.plist \
  -exportPath "$BUILD_DIR/export"

# 3. Verify hardened runtime + entitlements
codesign -dv --verbose=4 "$BUILD_DIR/export/$APP.app"
codesign --verify --deep --strict "$BUILD_DIR/export/$APP.app"

# 4. Build DMG (create-dmg, background art in Assets/DMG/)
create-dmg \
  --volname "$APP" \
  --window-size 540 380 \
  --icon-size 128 \
  --icon "$APP.app" 130 190 \
  --app-drop-link 400 190 \
  --background "Assets/DMG/background.png" \
  "$DMG" "$BUILD_DIR/export/"

# 5. Sign the DMG itself
codesign --sign "Developer ID Application: Simon-Pierre Boucher (<TEAM_ID>)" "$DMG"

# 6. Notarize & wait
xcrun notarytool submit "$DMG" \
  --keychain-profile "<KEYCHAIN_PROFILE_NAME>" \
  --wait

# 7. Staple
xcrun stapler staple "$DMG"
xcrun stapler validate "$DMG"

echo "✅ $DMG notarized and stapled."

# scripts/ExportOptions.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
 "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>method</key><string>developer-id</string>
  <key>teamID</key><string><TEAM_ID></string>
  <key>signingStyle</key><string>automatic</string>
  <key>destination</key><string>export</string>
</dict></plist>

# Entitlements (Metrika.entitlements)

• Hardened Runtime: ON.
• App Sandbox: ON, with com.apple.security.files.user-selected.read-write (open/save datasets) and bookmark entitlements for recent files.
• No JIT, no unsigned executable memory (MLX does not require them).
• If plugin loading of external dylibs is added later, revisit com.apple.security.cs.disable-library-validation — avoid if possible; prefer SPM-compiled-in plugins.

# 11. App Icon

• Master: Assets/AppIcon/metrika-icon.svg (1024×1024 design).
• Concept: a glowing M-shaped time-series line — a zigzag chart trace cutting through a scatter of data points on a deep space-gradient squircle — econometrics meets GPU power.
• Generate .icns: scripts/make_icns.sh rasterizes the SVG (via rsvg-convert or qlmanage) to 16→1024 px and runs iconutil -c icns.
• macOS Sonoma+ icon grid: content within the 824×824 safe zone of the 1024 canvas; system applies the squircle mask — but we bake our own rounded-rect background per Big Sur+ convention.

# 12. Repo Conventions

Metrika/
├── CLAUDE.md
├── Metrika.xcodeproj
├── Metrika/                    # app target (UI)
├── MetrikaKit/                 # SPM package (all logic)
│   └── Sources/{ZQParser,ZQPlanner,ZQEngine,ZQData,ZQStats,ZQGPU,ZQGraphics,ZQPlugins}
├── Assets/{AppIcon,DMG}
├── scripts/{release.sh,make_icns.sh,check_headers.sh}
└── Tests/{Unit,Bench,Fixtures}

• Pre-commit hook runs scripts/check_headers.sh: rejects any staged source file (.swift, .metal, .sh, .py, .zyq) missing the Author: Simon-Pierre Boucher / Contact: contact@spboucher.ai header.
• Build quirks (mlx-swift): SwiftPM CLI cannot compile Metal shaders, so swift test skips the GPU suites (CPU fallback via metallib detection); run xcodebuild test -scheme MetrikaKit-Package -destination 'platform=macOS' -skipPackagePluginValidation from MetrikaKit/ for full coverage. All xcodebuild invocations (app and tests) need -skipPackagePluginValidation for mlx-swift's CudaBuild plugin.
• Commits: Conventional Commits (feat(parser): factor variable expansion).
• Branches: main (protected), dev, feature branches feat/*.
• CI: GitHub Actions on self-hosted arm64 macOS runner — build, test, bench, header check.

# 13. Roadmap

v0.1 (MVP, ~8 weeks of focused work) use/save (parquet, csv, dta) · summarize · tabulate · gen/replace · reg (robust, cluster) · logit · basic scatter/histogram · console + data browser · do-file execution.

v0.2 — GPU bootstrap + permutation tests · xtreg FE · ivregress · Swift Charts full suite · plugin protocol.

v0.3 — MCMC/Bayesian module · gradient boosting & elastic net (MLX) · Metal 2M-point renderer · margins/predict.

v1.0 — notarized DMG release, docs site, sample datasets, command reference (help regress in-app).

# 14. What Claude Should Do When Working in This Repo

1. Read this file first; respect §0 rules absolutely (headers, Swift 6 strict concurrency).
2. Before touching numerics, read the matching fixture in Tests/Fixtures/ and keep tests green.
3. Any new command: grammar entry → AST case → planner rule → CPU impl → (optional) GPU impl → golden test → in-app help entry. All six or the PR is incomplete.
4. Never introduce a Python or Node runtime dependency into the app bundle.
5. When unsure between CPU and GPU, implement CPU first — correctness before speed — then add the GPU path behind the planner.
6. Update the performance table (§8) whenever a benchmark materially changes.