spb/hfmarketdata

Public

Open high-frequency market data platform — FirstRate full-history downloader, DuckDB/Parquet lake, open REST API and React docs platform (www.hfmarketdata.io)

127commits 1branches 0releases

24.7 MBsize

maindefault branch

11 days agolast push

JavaScript 53.7% Python 38.3% CSS 4.6% TypeScript 3.1%

919 B · 13 lines shellscript

Raw Blame History

1#!/bin/zsh2# Waits for the EDGAR backfill (concurrent writer of fund_statements) to stop, then converts3# fund_statements.coverage → fund_coverage_blob dictionary (resumable). Run from the deployment dir on the node:4#   nohup scripts/run_migrate_coverage_when_idle.sh > ~/firstratedata/state/migrate_coverage.log 2>&1 &5cd "$(dirname "$0")/.."6export HFMD_DATA_ROOT=${HFMD_DATA_ROOT:-$HOME/firstratedata}7export HFMD_STATE_DB=${HFMD_STATE_DB:-$HFMD_DATA_ROOT/state/hfmd.db}8export HFMD_REDIS_URL=${HFMD_REDIS_URL:-redis://127.0.0.1:6379/0}9export HFMD_ENV=${HFMD_ENV:-production}10while pm2 jlist 2>/dev/null | python3 -c "import sys,json; sys.exit(0 if any(p['name']=='hfmarketdata-edgar-backfill' and p['pm2_env']['status']=='online' for p in json.load(sys.stdin)) else 1)"; do sleep 120; done11echo "$(date) backfill finished, converting coverage"12exec venv/bin/python scripts/migrate_coverage.py --sleep 0.05 "$@"13