SPB Git

spb/hfmarketdata Public

Open high-frequency market data platform — FirstRate full-history downloader, DuckDB/Parquet lake, open REST API and React docs platform (www.hfmarketdata.io)

Python 46.7% JavaScript 37.4% CSS 14.9% HTML 0.9%

Fall back to bsdtar for Deflate64 zip archives (options 2023_q1/q3)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
simon-pierre boucher committed 19 h ago (Aug 10, 2026) parent dd1eb9a

Showing 1 changed file with +9 and −2

modified frd_downloader.py +9 −2
@@ -64,6 +64,7 @@ import json
64 64 import logging
65 65 import os
66 66 import shutil
67 +import subprocess
67 68 import sys
68 69 import tempfile
69 70 import threading
@@ -430,8 +431,14 @@ def extract_and_convert(zip_path: Path, dest_dir: Path, asset_type: str) -> tupl
430 431 def _extract_recursive(zip_path: Path, dest: Path, depth: int = 0) -> None:
431 432 if depth > 3:
432 433 return
433 with zipfile.ZipFile(zip_path) as zf:
434 zf.extractall(dest)
434 + try:
435 + with zipfile.ZipFile(zip_path) as zf:
436 + zf.extractall(dest)
437 + except (NotImplementedError, RuntimeError, zipfile.BadZipFile):
438 + # Some FirstRate archives use Deflate64, which the stdlib zipfile
439 + # cannot decompress — bsdtar (libarchive) handles it.
440 + subprocess.run(["tar", "-xf", str(zip_path), "-C", str(dest)],
441 + check=True, capture_output=True)
435 442 for nested in list(dest.rglob("*.zip")):
436 443 sub = nested.with_suffix("")
437 444 sub.mkdir(exist_ok=True)
438 445