spb/metrika Public

- root cause: duckdb-swift's element(forColumn:at:) linearly rescans
  every chunk on every element access — O(rows x chunks); a 10M-row CSV
  load took 276 s
- DuckDBFastReader: pointer-based duckdb C calls via @_silgen_name (the
  C library is statically linked; only pointer-argument functions are
  declared, so no struct-ABI exposure), private in-memory instance,
  contiguous column_data/nullmask_data extraction
- ZQDataStore.dataFrame(fromQuery:): LIMIT-0 schema probe through the
  supported API, SQL-side casts to DOUBLE/VARCHAR, memcpy per numeric
  column; dead per-element materializer removed
- measured (M4-class): 10M-row parquet 0.20 s, csv 0.31 s, summarize
  ~0.27 s, robust reg ~0.24 s — CLAUDE.md §8 budgets met; table updated
  with measured numbers
- 116 tests green (identical numerics through the new path)
- version 1.0.1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- hero: console session showing cluster-robust factor regression and the
  GPU-batched 10k bootstrap; screenshot grid (Charts by(), Metal 2M,
  data browser, manual)
- shields badges: release, macOS 14+, Swift 6, arm64, Metal+MLX, 116
  tests, R-validated 1e-10, notarized DMG
- five-minute tour, feature matrix, numerical-validation section,
  architecture, build instructions, extensibility examples
- METRIKA_AUTOPANE debug hook (headless pane selection for screenshots)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- sysuse command with bundled samples (sales: 200-firm x 5-year panel;
  mtcars) shipped as ZQEngine resources, generated by
  scripts/make_samples.R
- metrika-cli docs: static documentation site rendered from the shared
  command registry (docs/index.html, 38 commands, light/dark); CLI
  restructured with a default 'run' subcommand
- Metal point-sprite renderer (CLAUDE.md §7 pane 4): packed 12-byte
  vertices, unified-memory buffer written once, scroll-pan /
  pinch-zoom / double-click-reset touching only a 24-byte uniform;
  takes over scatter plots past 100k points (METRIKA_METAL_THRESHOLD
  override for tests); verified rendering 2,000,000 points in-app
- METRIKA_AUTORUN debug hook for headless app driving
- MARKETING_VERSION 1.0.0
- 116 kit tests + 4 UI tests green

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- ZQGradientBoosting: exact-greedy trees cloning xgboost's algorithm —
  gain 1/2[GL2/(HL+l) + GR2/(HR+l) - G2/(H+l)] - gamma, leaf -G/(H+l),
  midpoint splits between consecutive distinct values, missing rows
  default left, pre-sorted feature indices; squared loss, deterministic
  (no subsampling)
- engine: 'boost y x…, rounds(#) [eta() maxdepth() lambda()]' reporting
  training R2/RMSE with an in-sample caveat; model stored in the
  estimation state (Kind.boost) so predict routes through the trees
  (missing features follow the default direction); margins and GLM
  statistics refused after boost
- validation: per-observation prediction parity with R xgboost 3.2
  (exact method, base_score = mean) at 1e-4 (xgboost is float32
  internally); a stump finds the exact midpoint split with lambda 0;
  training loss decreases monotonically in rounds
- manual entry + coverage test
- 115 tests green (swift test and xcodebuild with GPU suites)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- PhiloxStream in ZQGPU: sequential variate stream over a dedicated
  counter block (bit 62 + per-stream 2^48 words, disjoint from bootstrap
  and permutation streams); Box-Muller normals, Marsaglia-Tsang gammas
  (moments verified against theory at 200k draws)
- ZQBayesianRegression: semi-conjugate Gibbs with Stata bayes default
  priors (coefficients N(0, 10000), variance InvGamma(.01, .01)); exact
  full conditionals via dense Cholesky; posterior mean/sd and
  equal-tailed 95% credible intervals; ZQStats now depends on ZQGPU for
  the shared RNG
- engine: 'bayes [, mcmcsize() burnin() seed() normalprior()]: reg …'
  with reproducible chains; manual entry included
- validation: with diffuse priors the posterior reproduces OLS (mean
  within 5% of a posterior SD, sd ratio in [0.9, 1.15], CrI brackets the
  estimate, sigma recovers the DGP); tight priors shrink toward zero;
  chains bit-reproducible per seed
- 111 tests green (swift test and xcodebuild with GPU suites)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The export folder also contains DistributionSummary.plist and
Packaging.log, which must not ship inside the image.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- ZQCommandReference: one registry of every command (syntax, options,
  examples, notes) shared by the console and the app so they cannot
  drift; 35 commands across 10 categories
- console: 'help' lists the reference by category, 'help <command>'
  (abbreviations resolve too) renders the full entry
- app: Manual pane — searchable sidebar grouped by category, per-command
  pages (syntax/options/examples), and a 'How Metrika works' overview
  covering the command grammar, data handling, estimation conventions,
  the CPU/GPU planner, reproducibility, and user commands
- release plumbing: real team ID + notary keychain-profile NAME in
  release.sh (both configuration, not credentials, per the security
  conventions), manual Developer ID signing with hardened runtime,
  -skipPackagePluginValidation, DMG background art
- tests: help coverage test pins a manual entry for every implemented
  verb; UI smoke drives the Manual pane; 106 green

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- ZQElasticNet: cyclic coordinate descent with residual updates,
  glmnet-convention objective ((1/2n)RSS + lambda(alpha*l1 +
  (1-alpha)/2*l2)), internal predictor standardization (1/n variance),
  unpenalized intercept, coefficients reported on the original scale;
  lambdaMax helper
- matched glmnet's gaussian y-standardization quirk deliberately: the L1
  penalty is invariant to it but the effective ridge penalty scales by
  1/sd(y) — without this, alpha<1 fits diverge from glmnet by ~10%
- engine: 'elasticnet y x…, lambda(#) [alpha(#)]' and 'lasso' (alpha
  fixed at 1); missing lambda() errors with the data's lambda_max as a
  hint; predict works afterwards, margins refuses (no VCE)
- fixtures: glmnet 5.0 at thresh 1e-15 over deliberately correlated
  regressors; coefficients match at 1e-6 (documented tolerance for
  penalized iterative solvers) and the selection pattern (which
  coefficients are exactly zero) matches exactly
- ZQCoefficient gains a public initializer
- 104 tests green (swift test and xcodebuild with GPU suites)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- ZQOLSResult/ZQGLMResult expose the full covariance matrix (column-major
  k x k aligned with coefficients); IV passes it through
- EstimationState carries vce, inference df, and the estimation-sample
  design (GLMs only — AMEs need it)
- margins, dydx(varlist): OLS/IV effects are the coefficients with their
  SEs; GLM average marginal effects with analytic delta gradients
  (logit p(1-p)(1-2p), probit -xb*phi, poisson exp) over the estimation
  sample; t or z inference per model kind
- factor/interaction dydx rejected with a clear message (discrete-change
  margins later); continuous terms of factor models work
- R fixtures mirror the exact formulas at the converged coefficients;
  logit/poisson AME and delta SE match at 1e-10
- 98 tests green (swift test and xcodebuild with GPU suites)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- PredictorDefinition (column / indicator / product / constant) records
  how each fitted regressor is recomputed from the dataset; threaded
  through buildRegressionSample so factor expansions and interactions
  carry their recipes
- EstimationState stored after regress, logit/probit/poisson, and
  ivregress; cleared after xtreg (predict there needs the estimated u_i)
- predict newvar [, xb | residuals | pr | n]: evaluates over ALL current
  observations with missing propagation; kind-aware defaults (xb for
  ols/iv, pr for logit/probit, n for poisson) and statistic validation
- ZQFactorExpansion.expand now returns the numeric level alongside each
  indicator column
- tests: xb + residuals reconstruct the response exactly (plain and
  factor models), mean fitted probability/count equals ybar (score
  equations), option validation; 93 green

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- fix: script-command verbs were routed AFTER parsing, which rejects
  unknown verbs — .zyq commands could never run; they now route before
  the parser with raw macro arguments
- .zyq scripts: ado-style 'args name…' declaration with backtick macro
  substitution (`0', `k', `name'); scratch-variable scripts verified
  end-to-end against the built-in path
- ZQPluginRegistry: verb-keyed native plugin registry; verbs shadowing
  built-ins rejected at session start; parsed commands validated against
  the declared ZQSyntaxSpec (varlist/if/in/weights/options)
- mutation gating: plugins return a replacement dataset via
  ZQResult.replacementFrame, installed only under 'mutates: true' — a
  non-mutating plugin returning one is an error and the dataset is
  untouched (ZQContext is now a plain value, no writable back-channel)
- ZScorePlugin ships as the reference native plugin, registered by the
  app ('zscore varname' -> standardized z_varname)
- CLAUDE.md: document the mlx-swift build quirks (xcodebuild for GPU
  tests, -skipPackagePluginValidation everywhere)
- 87 tests green (swift test and xcodebuild with GPU suites); app builds

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- ZQResampling.permutationIndices: permutation as argsort of 64-bit
  Philox keys — order-free and counter-addressable so a future GPU
  argsort path reproduces it exactly; dedicated stream offset (bit 63,
  applied after word expansion) keeps permutations disjoint from
  bootstrap draws under the same seed
- engine: 'permute, reps(#) [seed(#)]: reg …' — permutes the response
  within the estimation sample, parallel chunked refits, empirical
  two-sided p per coefficient (c, reps, p, SE(p)); degenerate _cons row
  omitted
- planner: GPU heuristic restricted to bootstrap until the permute
  argsort path lands
- graphics: standalone histogram/scatter/kdensity verbs; single-variable
  histogram (Sturges default, bins() option) and Epanechnikov kdensity
  with Silverman bandwidth (density integrates to 1 in tests)
- 81 tests green (swift test and xcodebuild with GPU suites)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- parser: (endog = instruments) varlist groups -> ZQIVSpec; digit-led
  sub-commands (2sls) reassembled from number+identifier tokens via
  column adjacency
- ZQFixedEffects: within estimator with Stata conventions (add-back
  means, reported _cons, df = N-K-G), within R-squared, panel-clustered
  VCE with G/(G-1) and t on G-1 df; v0.2 restriction: cluster variable
  must equal the panel variable
- ZQIV: 2SLS via thin-Q projection of the instrument matrix (Z'Z never
  formed), residuals from original regressors, Stata 'small' inference,
  classical/HC1/cluster VCE built on projected regressors
- engine: xtreg (requires xtset + fe), ivregress 2sls with dedicated
  listwise deletion across depvar/exog/endog/instruments; shared
  coefficient-table renderer extracted
- fixtures: z1/z2 instrument columns (drawn after existing draws, earlier
  golden values bit-identical), manual within/2SLS algebra in R
- 75 tests green (swift test and xcodebuild with GPU suites)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- mlx-swift dependency in ZQGPU (the only module allowed to import MLX)
- Philox4x32-10 as vectorized MLX uint64 ops with the reference key
  schedule: GPU resample indices are bit-identical to the CPU generator
  (asserted per replicate) — the §5 reproducibility contract
- batched gather + float32 cross-products on GPU, float64 Cholesky solves
  on CPU (§5 hybrid); chunked to the memory budget
- conditioning: predictors standardized by fixed full-sample mean/scale
  (exact reparametrization) before forming X'X — raw data with an
  intercept lost 2-3 digits in float32
- MLX workaround: batched GEMM mis-accumulates small k x k outputs at
  batch >= 2 (~6e-4 rel, stride/fusion-independent); X'X is built one
  column at a time through the exact (count,k,n)@(count,n,1) path, with
  regression tests pinning the behavior
- planner: gpuAvailable detected from actual metallib presence (SwiftPM
  CLI cannot compile Metal shaders - swift test skips GPU suites, the
  xcodebuild-built app and tests get the GPU); reps >= 500 dispatches
  bootstrap to .gpu
- engine: backend-aware bootstrap, drops singular resamples, output
  titled 'GPU batched' with gpu scalar
- 68 tests green under xcodebuild (66 + GPU suites under swift test skip)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- DataBrowserPane (§7 pane 3): NSTableView bridge (natively virtualized),
  sortable columns with missing-last ordering, observation-number gutter,
  filter bar compiled to an if-expression through the real parser
  (ZQCommandParser.parseExpression + ZQSession.conditionMask are the new
  public entry points)
- DoFileEditorPane (§7 pane 2): NSTextView bridge with ZQL highlighting
  (verbs resolved via ZQVerbTable, comments, strings), run selection or
  file with Cmd-R through the shared session path, open/save .zyq
- ContentView: segmented Console / Data / Do-file switcher in the toolbar
- UI smoke extended: pane switching, filter narrowing (2 of 5 obs),
  do-file run surfacing in the console and sidebar
- 61 kit tests + 2 UI tests green

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- DTAReader: formats 117/118/119, both byte orders, all numeric storage
  types widened to Float64 with Stata missing codes (., .a-.z) mapped to
  the validity mask; str# and strL (GSO) load as strings; value labels
  read past but not yet applied; pre-117 files rejected with a clear
  message
- DTAWriter: format 118 (UTF-8, LSF), doubles + str# up to 2045 bytes,
  real map offsets so Stata/haven/pandas can seek
- ZQDataStore routes .dta through the native path for use and save
- ZQColumnData equality now ignores value slots at missing positions
  (undefined by contract, often NaN — synthesized == failed on NaN != NaN)
- Fixtures: haven-written 117 + 118 files; tests assert bit-identical
  numeric loads vs the CSV, lossless write/read roundtrip; haven
  cross-reads Metrika-written files (verified)
- 61 tests green

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- ZQGLM: Fisher-scoring IRLS through the LAPACK QR path (X'WX never
  formed), converged to |Δll| < 1e-13; classical, robust (HC0 score
  sandwich, Stata ML convention), and cluster (G/(G−1)) VCE; LR/Wald
  chi2, McFadden pseudo-R²; information matrix re-evaluated at the
  converged beta (R's vcov carries last-iteration weights and is only
  ~1e-6-accurate by its own stopping rule — fixtures compute expected
  information at the optimum explicitly)
- Distributions: regularized incomplete gamma, chi-square CDF/p-value,
  normal quantile
- ZQCorrelate: Pearson matrix on listwise-complete data
- Engine: logit/probit/poisson tables (z, P>|z|, LR chi2, pseudo R²),
  one-way and two-way tabulate with totals, Stata-style lower-triangle
  correlate
- Fixtures: binomial/poisson outcomes drawn after existing draws (earlier
  golden values bit-identical); glm at epsilon 1e-12
- 56 tests green (12 new)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- MetrikaUITests target (xcodegen scheme test action)
- launch-environment hook (DEBUG only): sandboxed app materializes the
  test CSV in its own container, since the runner cannot stage files
  across the sandbox boundary reliably
- queries predicate on AX 'value' (SwiftUI Text exposes value, not label)
- smoke: launch, auto-load dataset, type gen + reg robust, assert
  regression table and sidebar variable

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- MetrikaKit SPM package: ZQParser, ZQPlanner, ZQEngine, ZQData, ZQStats,
  ZQGPU, ZQGraphics, ZQPlugins (Swift 6, strict concurrency)
- ZQL parser: lexer, Pratt expressions, factor variables, prefix commands,
  column-cited errors with Levenshtein verb suggestions
- ZQData: DuckDB-backed load/save (parquet, csv, json, arrow)
- ZQStats: OLS via LAPACK QR, HC0-HC3 and cluster-robust SE, summarize,
  t/F distributions accurate in the far tails
- ZQGPU: Philox4x32-10 reference RNG, counter-addressable bootstrap
- ZQEngine: session actor with use/save/gen/replace/drop/keep/summarize/
  regress/count/list/graph/bootstrap/set seed/xtset/log
- SwiftUI app (xcodegen): console with history, variables sidebar,
  Swift Charts plots; sandboxed + hardened runtime entitlements
- Tests: 44 green (parser golden, R fixtures at 1e-10, Philox KAT,
  end-to-end engine); Tests/Fixtures/generate.R; Tests/Bench harness
- scripts: check_headers.sh + pre-commit hook, make_icns.sh, release.sh

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>