SPB Git forge

spb/hfmarketdata

Public

Open high-frequency market data platform — FirstRate full-history downloader, DuckDB/Parquet lake, open REST API and React docs platform (www.hfmarketdata.io)

127commits 1branches 0releases
24.7 MBsize
maindefault branch
11 days agolast push
JavaScript 53.7% Python 38.3% CSS 4.6% TypeScript 3.1%

Fall back to bsdtar for Deflate64 zip archives (options 2023_q1/q3)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
simon-pierre boucher committed 1 mo ago (Aug 10, 2026) parent dd1eb9a

1 changed file +9 −2

modified frd_downloader.py +9 −2
@@ -64,6 +64,7 @@ import json
64 64 import logging
65 65 import os
66 66 import shutil
67 +import subprocess
67 68 import sys
68 69 import tempfile
69 70 import threading
@@ -430,8 +431,14 @@ def extract_and_convert(zip_path: Path, dest_dir: Path, asset_type: str) -> tupl
430 431 def _extract_recursive(zip_path: Path, dest: Path, depth: int = 0) -> None:
431 432 if depth > 3:
432 433 return
433 − with zipfile.ZipFile(zip_path) as zf:
434 − zf.extractall(dest)
434 + try:
435 + with zipfile.ZipFile(zip_path) as zf:
436 + zf.extractall(dest)
437 + except (NotImplementedError, RuntimeError, zipfile.BadZipFile):
438 + # Some FirstRate archives use Deflate64, which the stdlib zipfile
439 + # cannot decompress — bsdtar (libarchive) handles it.
440 + subprocess.run(["tar", "-xf", str(zip_path), "-C", str(dest)],
441 + check=True, capture_output=True)
435 442 for nested in list(dest.rglob("*.zip")):
436 443 sub = nested.with_suffix("")
437 444 sub.mkdir(exist_ok=True)
438 445