SPB Git

Repository Public

checklist.md 11.3 KB

# Release checklist

The quality gates of a release, in the order they are run, each with the command that proves it. A gate is passed when the command exits 0 and prints the stated evidence; a statement in a document is not evidence. All commands are run from the repository root.

Details of the last three stages: signing, notarization, distribution. Test strategy: testing. Build requirements: setup.

flowchart LR
    A["1 Source<br/>clean and scanned"] --> B["2 Tests<br/>all packages"]
    B --> C["3 Strings and docs<br/>consistent"]
    C --> D["4 Release build<br/>signed and verified"]
    D --> E["5 Notarized<br/>and stapled"]
    E --> F["6 Artifacts<br/>checked"]
    F --> G["7 Publication<br/>owner only"]

SVG fallback: Order of the release gates

Gates 1 to 4 run offline (after the one-time DuckDB download) and change nothing outside build/, dist/ and docs/reports/. Gate 5 uploads the application to Apple. Gate 7 is never automated.

# 1. Source

  • Working tree is what will be released. git status --short prints nothing (or only files deliberately left out), and git log --oneline -1 is the intended commit.
  • No secrets in tracked or untracked, non-ignored files. Scripts/secret-scan.sh → secret scan: clean (<n> files), exit 0. The patterns cover private-key blocks, AWS/Anthropic/OpenAI/GitHub/Slack token shapes, generic password|secret|api_key|token = <16+ chars> assignments and notarytool store-credentials … --password; it also fails on .p8, .p12, .pem, .mobileprovision, .cer, .keychain and .env files. Test fixtures, .xcstrings catalogs and documentation lines containing "example" or "placeholder" are excluded. It is a pattern scan, not a proof of absence.
  • Version is set. grep -E "MARKETING_VERSION|CURRENT_PROJECT_VERSION" project.yml shows the version and build number to release.
  • The project file matches project.yml. Scripts/build.sh generate → ORIXSheet.xcodeproj generated (also runs Scripts/fetch-duckdb.sh, which verifies the pinned SHA-256 when it downloads).

# 2. Tests

  • Every package test suite passes. Scripts/test-all.sh (or Scripts/build.sh test) → last line TOTAL … 0 failed package(s), exit 0; the machine-readable result is written to docs/reports/tests.json ("failedPackages": 0). Per-package logs are in build/test-logs/<Package>.log. A package is counted as failed if its log lacks a "passed" summary line or contains error: or signal code. The on-device model evaluation is opt-in (ORIX_AI_EVAL=1) and is not part of this run. Recorded run: 22 packages, 1,270 test functions, "failedPackages": 0 (2026-11-21T08:47:01Z).
  • Counts are taken from the file, not from memory. Quote test numbers only from docs/reports/tests.json of this run.
  • Optional, env-gated suites that the default run skips (run when the area changed): cd Packages/SheetIntelligence && ORIX_AI_EVAL=1 swift test (model evaluation, minutes long, needs the on-device model, rewrites EVALUATION.md); ORIX_DATA_LARGE_TESTS, ORIX_BENCH (see testing and performance for what each enables).
  • App integration tests. xcodebuild test -project ORIXSheet.xcodeproj -scheme ORIXSheet -destination 'platform=macOS,arch=arm64' → ** TEST SUCCEEDED **. Recorded run of the ORIXSheetTests target (App/Tests): 9 tests, 0 failures (docs/reports/app-tests.json, 2026-11-21T08:48:35Z). One of the nine asserts that the menu bar has no duplicate key equivalent.
  • Debug app builds. Scripts/build.sh debug → ** BUILD SUCCEEDED ** and no error: line. (The script filters warnings out of its output.)

# 3. Strings and documentation

  • String catalogs of the shared UI and the app are complete and regenerated. python3 Scripts/localize.py --check → one (checked) line per catalog, exit 0. It fails on a key without translation, mismatched format specifiers, a stale catalog or an interpolated key. Observed on 2026-11-21 after the template chooser and the new Settings privacy text were added: sheetui: 346 keys (checked), app: 218 keys, 9 problem(s), exit 1 — eight keys of App/Sources/TemplateChooser.swift and App/Sources/AppSettings.swift have no fr-CA translation and App/Resources/Localizable.xcstrings is out of date. This gate is not passed until Scripts/localize.py is run with the missing translations added.
  • Feature packages' catalogs. Each feature package has its own string tool (Packages/FeatureCharts/Scripts/strings.py, Packages/FeatureAnalysis/Scripts/strings.py --all, tools under Packages/FeatureAudit/Tools, FeatureIntelligence/Tools, FeatureAutomation/Tools). The Charts and Analysis tools exit 1 on a missing or mismatched translation, but they also rewrite their catalog; there is no verify-only mode. Run them, then check that git status --short shows no change.
  • Documentation links and images resolve. python3 Scripts/check-docs.py → 0 problem(s).
  • Every diagram has an up-to-date SVG fallback. python3 docs/assets/diagrams/make_diagrams.py --check → no stale: or error: line, exit 0.
  • Documents quote measured values only. Benchmarks referenced by the documents exist in docs/benchmarks/ and in the package BENCHMARKS.md files with their machine, build and date.

# 4. Release build and signature

  • Signed universal build. Scripts/sign.sh (or Scripts/build.sh release) → ends with RESULT: dist/ORIX Sheet.app is built and signed. The script itself enforces, and prints:

    • codesign --verify --deep --strict passes;
    • architectures x86_64 arm64;
    • authority Developer ID Application: …, a Timestamp= line, flags=0x10000(runtime);
    • no get-task-allow entitlement;
    • every Mach-O in the bundle signed by team 3YM54G49SN with the hardened runtime;
    • the embedded tool answers: a line -- embedded CLI: orix-sheet <version> (…).
  • Independent re-check of the bundle (does not rebuild):

    bash
    codesign --verify --deep --strict --verbose=2 "dist/ORIX Sheet.app"
    codesign -dvv "dist/ORIX Sheet.app" 2>&1 | grep -E "Authority|TeamIdentifier|Timestamp|flags"
    codesign -d --entitlements - --xml "dist/ORIX Sheet.app"      # expect an empty <dict/>
  • The bundle contains what the current project links. ls "dist/ORIX Sheet.app/Contents/Frameworks" lists libduckdb.dylib, and codesign -dvv "dist/ORIX Sheet.app/Contents/Frameworks/libduckdb.dylib" 2>&1 | grep TeamIdentifier prints 3YM54G49SN.

  • The command-line tool is embedded, signed and runs. "dist/ORIX Sheet.app/Contents/MacOS/orix-sheet" version prints the version line, and codesign -dvv "dist/ORIX Sheet.app/Contents/MacOS/orix-sheet" 2>&1 | grep -E "TeamIdentifier|flags" prints 3YM54G49SN and flags=0x10000(runtime). Scheduled workflows depend on this file (automation).

  • The signed app launches and opens a document. Launch dist/ORIX Sheet.app by hand; the launch arguments used for automated screenshots (-ORIXOpen <file> -ORIXShot <png>) are described in setup. No script asserts this for the Release bundle.

# 5. Notarization (uploads to Apple; owner's credentials)

  • App and DMG accepted and stapled. Scripts/notarize.sh → exit 0 and RESULT: dist/ORIX Sheet.app and dist/ORIXSheet-<version>.dmg are notarized by Apple and stapled. Exit 2 = app not accepted, exit 3 = DMG not accepted; logs in build/notary/.

  • Tickets validate. xcrun stapler validate "dist/ORIX Sheet.app" and xcrun stapler validate dist/ORIXSheet-<version>.dmg → The validate action worked!

  • Gatekeeper accepts both.

    bash
    spctl --assess --type execute --verbose=2 "dist/ORIX Sheet.app"
    spctl --assess --type open --context context:primary-signature --verbose=2 dist/ORIXSheet-<version>.dmg

    → accepted, source=Notarized Developer ID.

# 6. Artifacts

  • Checksums match. cd dist && shasum -a 256 -c ORIXSheet-<version>.sha256 → OK twice.
  • The DMG has the designed layout (background, app at the left, Applications link at the right). If package.sh printed create-dmg unavailable or failed: building a plain disk image, the fallback image was produced: decide whether that is acceptable. Checked by opening the DMG.
  • The DMG is signed. codesign -dvv dist/ORIXSheet-<version>.dmg 2>&1 | grep Authority.
  • First-launch test from a download on a Mac other than the build machine (quarantine set by the browser): the app opens without a Gatekeeper warning beyond the standard "downloaded from the Internet" confirmation. Manual; see RELEASE-REPORT.md for whether it was done.
  • Manual accessibility pass, if claimed. docs/development/accessibility.md lists what was not verified (VoiceOver was not listened to, display options not switched on during a run). Do not describe these as verified unless they have been.

# 7. Publication (owner's authorization required)

  • Scripts/secret-scan.sh run again immediately before pushing.
  • git push origin main to the private repository (gitsrv:srv/git-private/orix-sheet.git).
  • Tag, release notes and any download location: decided and performed by the owner. Nothing in the repository automates them; see distribution.

# Gates observed on 2026-11-21

Gate Command Result
Package tests Scripts/test-all.sh 22 packages, 1,270 test functions, 0 failed packages (docs/reports/tests.json, 2026-11-21T08:47:01Z)
App integration tests xcodebuild test -project ORIXSheet.xcodeproj -scheme ORIXSheet -destination 'platform=macOS,arch=arm64' 9 tests, 0 failures (docs/reports/app-tests.json, 2026-11-21T08:48:35Z)
Signed universal build Scripts/sign.sh, then codesign -dvv "dist/ORIX Sheet.app" and codesign --verify --deep --strict Developer ID Application, team 3YM54G49SN, flags=0x10000(runtime), x86_64 arm64, timestamp Nov 21, 2026 at 3:50:32 AM, empty entitlements (no get-task-allow), verification passes
Bundle content ls, codesign -dvv, lipo -archs on the nested files Contents/MacOS/orix-sheet and Contents/Frameworks/libduckdb.dylib present, universal, signed by the same team with the hardened runtime
Secret scan Scripts/secret-scan.sh secret scan: clean (1071 files), exit 0
String catalogs python3 Scripts/localize.py --check fails: sheetui: 346 keys checked, app: 218 keys, 9 problems (8 missing fr-CA translations, app catalog out of date), exit 1
Documentation python3 Scripts/check-docs.py, python3 docs/assets/diagrams/make_diagrams.py --check run after the documentation refresh; see the tools' output
Notarization, stapling, checksums, Gatekeeper Scripts/notarize.sh and the commands of gates 5 and 6 see RELEASE-REPORT.md

Source: docs/reports/tests.json, docs/reports/app-tests.json, and read-only inspection of dist/ORIX Sheet.app, Scripts/secret-scan.sh and Scripts/localize.py --check run on 2026-11-21 after the release commit 1a190fd.

Files main
Menu

Theme