SPB Git forge

spb/hfmarketdata

Public

Open high-frequency market data platform — FirstRate full-history downloader, DuckDB/Parquet lake, open REST API and React docs platform (www.hfmarketdata.io)

127commits 1branches 0releases
24.7 MBsize
maindefault branch
11 days agolast push
JavaScript 53.7% Python 38.3% CSS 4.6% TypeScript 3.1%

fundamentals: script d'attente pour lancer migrate_coverage après le backfill (survit au rsync --delete du déploiement)

Simon-Pierre Boucher committed 18 days ago (Sep 7, 2026) parent 02cd79d

1 changed file +12 −0

added scripts/run_migrate_coverage_when_idle.sh +12 −0
@@ -0,0 +1,12 @@
1 +#!/bin/zsh
2 +# Waits for the EDGAR backfill (concurrent writer of fund_statements) to stop, then converts
3 +# fund_statements.coverage → fund_coverage_blob dictionary (resumable). Run from the deployment dir on the node:
4 +# nohup scripts/run_migrate_coverage_when_idle.sh > ~/firstratedata/state/migrate_coverage.log 2>&1 &
5 +cd "$(dirname "$0")/.."
6 +export HFMD_DATA_ROOT=${HFMD_DATA_ROOT:-$HOME/firstratedata}
7 +export HFMD_STATE_DB=${HFMD_STATE_DB:-$HFMD_DATA_ROOT/state/hfmd.db}
8 +export HFMD_REDIS_URL=${HFMD_REDIS_URL:-redis://127.0.0.1:6379/0}
9 +export HFMD_ENV=${HFMD_ENV:-production}
10 +while pm2 jlist 2>/dev/null | python3 -c "import sys,json; sys.exit(0 if any(p['name']=='hfmarketdata-edgar-backfill' and p['pm2_env']['status']=='online' for p in json.load(sys.stdin)) else 1)"; do sleep 120; done
11 +echo "$(date) backfill finished, converting coverage"
12 +exec venv/bin/python scripts/migrate_coverage.py --sleep 0.05 "$@"
13