feat: Trawls v0.1.0 — fetch auto (http→browser→stealth), html_to_md, PDF, map, extract CSS/LLM, chunker, jobs SQLite + SSE, API /v1, UI, CLI
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
67 changed files +7,933 −0
added
.env.example
+29 −0
@@ -0,0 +1,29 @@ | ||
| 1 | +# Trawls — configuration (préfixe TRAWLS_). Toutes les valeurs ont un défaut sûr. | |
| 2 | +TRAWLS_ENV=production | |
| 3 | +TRAWLS_PORT=8180 | |
| 4 | +TRAWLS_PUBLIC_URL=https://www.trawls.dev | |
| 5 | +TRAWLS_DATA_DIR=./data | |
| 6 | +TRAWLS_LOG_LEVEL=INFO | |
| 7 | + | |
| 8 | +# Auth : désactivée par défaut (self-host). ADMIN_KEY permet de créer des clés via /v1/keys. | |
| 9 | +TRAWLS_REQUIRE_AUTH=false | |
| 10 | +TRAWLS_ADMIN_KEY= | |
| 11 | +TRAWLS_RATE_LIMIT_PER_MINUTE=300 | |
| 12 | + | |
| 13 | +# Fetch | |
| 14 | +TRAWLS_BROWSER_POOL_SIZE=6 | |
| 15 | +TRAWLS_MAX_CONCURRENCY_PER_HOST=4 | |
| 16 | +TRAWLS_MAX_CONCURRENCY_GLOBAL=16 | |
| 17 | +TRAWLS_MAX_SIZE_MB=50 | |
| 18 | +TRAWLS_PROXY_URLS= | |
| 19 | +TRAWLS_ALLOW_PRIVATE_TARGETS=false | |
| 20 | + | |
| 21 | +# LLM (extraction llm) : OpenAI-compatible (Ollama/vLLM/OpenRouter) ou anthropic | |
| 22 | +TRAWLS_LLM_PROVIDER=openai | |
| 23 | +TRAWLS_LLM_BASE_URL=https://llm.maclustr.io/v1 | |
| 24 | +TRAWLS_LLM_API_KEY= | |
| 25 | +TRAWLS_LLM_MODEL=maclustr-general | |
| 26 | + | |
| 27 | +# Jobs | |
| 28 | +TRAWLS_WORKER_CONCURRENCY=4 | |
| 29 | +TRAWLS_RESULT_TTL_DAYS=7 | |
added
.gitignore
+17 −0
@@ -0,0 +1,17 @@ | ||
| 1 | +.venv/ | |
| 2 | +venv/ | |
| 3 | +__pycache__/ | |
| 4 | +*.pyc | |
| 5 | +.pytest_cache/ | |
| 6 | +.ruff_cache/ | |
| 7 | +.mypy_cache/ | |
| 8 | +data/ | |
| 9 | +*.db | |
| 10 | +*.db-wal | |
| 11 | +*.db-shm | |
| 12 | +.env | |
| 13 | +dist/ | |
| 14 | +build/ | |
| 15 | +*.egg-info/ | |
| 16 | +node_modules/ | |
| 17 | +.DS_Store | |
added
CLAUDE.md
+306 −0
@@ -0,0 +1,306 @@ | ||
| 1 | +# CLAUDE.md — Trawls (www.trawls.dev) | |
| 2 | + | |
| 3 | +> Moteur de crawling, d'extraction et de navigation web, construit from scratch. | |
| 4 | +> Objectif : dépasser crawl4ai et Firecrawl en robustesse, qualité de sortie et autonomie. | |
| 5 | +> Nom : **Trawls** — domaine **www.trawls.dev**. Un chalut (trawl) ratisse le fond et remonte tout : on cartographie et on extrait n'importe quel site. | |
| 6 | + | |
| 7 | +Ce fichier est la source de vérité pour Claude Code. Lis-le intégralement avant toute modification. En cas de doute entre deux approches, choisis celle qui respecte la section « Principes ». | |
| 8 | + | |
| 9 | +**État au 2026-09-05 (v0.1.0)** : phases 1 à 6 livrées et déployées sur MacLustr via `mld` (PM2 `trawls-api` + `trawls-ngrok` → https://www.trawls.dev). Écarts assumés par rapport à la cible ci-dessous, à résorber dans l'ordre : (a) persistance **SQLite + worker embarqué asyncio** au lieu de PostgreSQL + Redis/arq (principe « self-host first », zéro dépendance) — `TRAWLS_DATABASE_URL`/`TRAWLS_REDIS_URL` réservés ; (b) UI en **HTML/JS vanilla** servie par FastAPI (`trawls/web/static/`) au lieu de Next.js (`web/`) ; (c) blobs (screenshots) sur disque local `data/blobs/` au lieu de S3/MinIO ; (d) agent (phase 7) et SDK/benchmarks (phase 8) non commencés. Ne pas casser le contrat API existant en résorbant ces écarts. | |
| 10 | + | |
| 11 | +--- | |
| 12 | + | |
| 13 | +## 0. Principes (ordre de priorité) | |
| 14 | + | |
| 15 | +1. **Ne jamais crasher sur une page.** Une page qui échoue produit un `PageResult` avec `status=failed` et une erreur typée. Le crawl continue. | |
| 16 | +2. **Sortie LLM-ready par défaut.** Markdown propre, sans boilerplate, métadonnées complètes, chunks optionnels. La qualité du MD est le critère n°1 du produit. | |
| 17 | +3. **Escalade automatique.** HTTP pur d'abord (rapide, 90 % des cas), navigateur seulement si nécessaire, stealth seulement si bloqué. L'utilisateur ne choisit pas le mode sauf s'il le force. | |
| 18 | +4. **Déterministe avant probabiliste.** Extraction CSS/JSON-LD sans LLM quand c'est possible. Le LLM est un outil, pas une béquille. | |
| 19 | +5. **Self-host first.** Tout fonctionne sans clé API externe (Ollama pour le LLM local). Le SaaS est une couche au-dessus, jamais une dépendance. | |
| 20 | +6. **Observable.** Chaque job, chaque page, chaque retry est tracé. Pas de « ça a échoué on ne sait pas pourquoi ». | |
| 21 | +7. **Async partout, backpressure partout.** Aucun appel bloquant dans le chemin chaud. | |
| 22 | + | |
| 23 | +--- | |
| 24 | + | |
| 25 | +## 1. Stack technique | |
| 26 | + | |
| 27 | +| Couche | Choix | Raison / contraintes | | |
| 28 | +|---|---|---| | |
| 29 | +| Langage | Python 3.12+, `uv` pour les deps | Typage strict, `mypy --strict` | | |
| 30 | +| HTTP rapide | `curl_cffi` (impersonation TLS Chrome/Safari) | Passe la plupart des fingerprints JA3 | | |
| 31 | +| HTTP fallback | `httpx[http2]` async | Quand curl_cffi est indisponible | | |
| 32 | +| Navigateur | Playwright (Chromium par défaut, Firefox/WebKit optionnels) | Pool de contexts persistants, pas un browser par page | | |
| 33 | +| Stealth | patches maison (webdriver, plugins, canvas noise, WebGL, permissions) | Voir `core/antibot/stealth.py` | | |
| 34 | +| Parsing HTML | `selectolax` (Lexbor) | 10-20x plus rapide que BS4. `lxml` uniquement pour le fallback recover/XPath | | |
| 35 | +| Détection encodage | `charset-normalizer` | Fallback latin-1, jamais d'exception non gérée | | |
| 36 | +| PDF | `pymupdf` (texte, layout, tables) ; OCR : `rapidocr-onnxruntime` (extra `ocr`) | Détection scan = ratio texte/page < 50 | | |
| 37 | +| Markdown | Pipeline maison (`processors/html_to_md/`), pas de `markdownify` | Voir §4 | | |
| 38 | +| Tokenisation | `tiktoken` (cl100k) pour compter les tokens des chunks | | | |
| 39 | +| LLM | Adapters : Anthropic, OpenAI-compatible (couvre Ollama, vLLM, OpenRouter, llm.maclustr.io) | Interface unique `LLMClient` | | |
| 40 | +| Queue | **actuel** : worker asyncio embarqué (`worker/crawl.py`) · **cible** : Redis 7 + `arq` | Simple, retries natifs, cron | | |
| 41 | +| DB | **actuel** : SQLite (aiosqlite, WAL) · **cible** : PostgreSQL 16 (SQLModel/SQLAlchemy 2 async) | Jobs, pages, clés API, usage | | |
| 42 | +| Blobs | **actuel** : disque `data/blobs/` · **cible** : S3-compatible (MinIO en local) | HTML brut, screenshots, PDF sources | | |
| 43 | +| API | FastAPI + Pydantic v2 | OpenAPI générée = contrat du SDK | | |
| 44 | +| Realtime | SSE (progression jobs) ; WebSocket réservé à l'agent console | | | |
| 45 | +| UI | **actuel** : HTML/JS vanilla dense (`trawls/web/static/`) · **cible** : Next.js 15, TypeScript, Tailwind, shadcn/ui, TanStack Query (`web/`) | | | |
| 46 | +| SDK | Python (`trawls`) et TypeScript (`@trawls/sdk`) générés depuis OpenAPI puis polis à la main | phase 8 | | |
| 47 | +| Conteneurs | Docker Compose (api, worker, ollama) | `make up` doit tout lancer | | |
| 48 | + | |
| 49 | +--- | |
| 50 | + | |
| 51 | +## 2. Arborescence | |
| 52 | + | |
| 53 | +``` | |
| 54 | +trawls/ | |
| 55 | +├── trawls/ # package Python | |
| 56 | +│ ├── config.py # Settings pydantic-settings, un seul point d'entrée (préfixe TRAWLS_) | |
| 57 | +│ ├── models/ # Pydantic: ScrapeOptions, PageResult, Job, ErrorInfo… | |
| 58 | +│ ├── core/ | |
| 59 | +│ │ ├── fetcher/ # base (Protocol), http_fast (curl_cffi), browser (pool Playwright), strategy (escalade) | |
| 60 | +│ │ ├── antibot/ # detect, stealth, identity, blocklist.txt (proxy : pool à venir) | |
| 61 | +│ │ ├── scheduler/ # robots, politeness, dedup (normalisation URL + simhash) ; frontier dans worker/crawl.py | |
| 62 | +│ │ ├── resilience/ # retry, breaker, budget | |
| 63 | +│ │ ├── security.py # garde SSRF | |
| 64 | +│ │ └── scrape.py # pipeline d'une page — ne lève jamais | |
| 65 | +│ ├── processors/ | |
| 66 | +│ │ ├── html_to_md/ # clean, readability, convert, tables, citations | |
| 67 | +│ │ ├── pdf/ # extract + tables + OCR + to_md (un module) | |
| 68 | +│ │ ├── structured/ # metadata (head + OpenGraph + JSON-LD + microdata) | |
| 69 | +│ │ ├── chunker/ # by_heading, by_tokens | |
| 70 | +│ │ ├── encoding.py | |
| 71 | +│ │ └── links.py | |
| 72 | +│ ├── extract/ # css, llm, merge | |
| 73 | +│ ├── agent/ # phase 7 — à créer (loop, observe, tools, planner, memory, guards) | |
| 74 | +│ ├── map/ # sitemap + crawl_links + rank (BM25) dans un module | |
| 75 | +│ ├── llm/client.py # interface unique (anthropic + openai_compat) | |
| 76 | +│ ├── api/ # main (routes /v1, SSE, UI, docs), auth | |
| 77 | +│ ├── worker/ # store (SQLite), crawl (moteur de jobs : crawl/batch/extract, EventBus) | |
| 78 | +│ ├── web/static/ # UI : index.html, app.js, style.css | |
| 79 | +│ └── cli.py # trawls scrape|crawl|map|serve|worker|keys | |
| 80 | +├── tests/unit/ | |
| 81 | +├── deploy/trawls.manifest.json # manifeste mld (MacLustr) | |
| 82 | +├── Dockerfile · docker-compose.yml · Makefile · CLAUDE.md | |
| 83 | +``` | |
| 84 | + | |
| 85 | +--- | |
| 86 | + | |
| 87 | +## 3. Modèles de données (Pydantic v2) | |
| 88 | + | |
| 89 | +```python | |
| 90 | +class ScrapeOptions(BaseModel): | |
| 91 | + formats: list[Literal["markdown","html","raw_html","json","links","screenshot","chunks","metadata"]] = ["markdown"] | |
| 92 | + only_main_content: bool = True | |
| 93 | + include_tags: list[str] = [] # CSS, forcer l'inclusion | |
| 94 | + exclude_tags: list[str] = [] | |
| 95 | + wait_for: str | int | None = None # sélecteur CSS ou ms | |
| 96 | + timeout_ms: int = 30_000 | |
| 97 | + mode: Literal["auto","http","browser","stealth"] = "auto" | |
| 98 | + headers: dict[str, str] = {} | |
| 99 | + cookies: list[Cookie] = [] | |
| 100 | + proxy: str | None = None | |
| 101 | + actions: list[BrowserAction] = [] # click/scroll/type/wait/press/evaluate avant extraction | |
| 102 | + location: Location | None = None # pays, langues → headers + timezone | |
| 103 | + remove_base64_images: bool = True | |
| 104 | + chunk: ChunkOptions | None = None | |
| 105 | + extract: ExtractOptions | None = None # css: {champ: CssField} | llm: schema JSON + prompt | |
| 106 | + cache: Literal["use","bypass","refresh"] = "use" | |
| 107 | + max_age_s: int = 86_400 | |
| 108 | + citations: bool = False | |
| 109 | + verify_ssl: bool = True | |
| 110 | + respect_robots: bool = True | |
| 111 | + | |
| 112 | +class PageResult(BaseModel): | |
| 113 | + url: str; final_url: str | |
| 114 | + status: Literal["ok","failed","skipped"] | |
| 115 | + http_status: int | None | |
| 116 | + fetch_mode_used: Literal["http","browser","stealth"] | None | |
| 117 | + markdown: str | None; html: str | None; raw_html: str | None | |
| 118 | + json_data: dict | None # {"data", "errors"} pour extract ; {"jsonld"} pour format json | |
| 119 | + links: list[Link]; metadata: PageMetadata; chunks: list[Chunk] | None | |
| 120 | + screenshot_url: str | None; error: ErrorInfo | None | |
| 121 | + timings: Timings # ttfb, fetch, render, process, total (ms) | |
| 122 | + fetched_at: datetime; depth: int; from_cache: bool | |
| 123 | + | |
| 124 | +class ErrorInfo(BaseModel): | |
| 125 | + code: ErrorCode; message: str; retryable: bool; attempts: int; details: dict | None | |
| 126 | +``` | |
| 127 | + | |
| 128 | +Règle : aucun `dict` non typé ne sort de l'API. Tout passe par un modèle. | |
| 129 | + | |
| 130 | +--- | |
| 131 | + | |
| 132 | +## 4. Pipeline HTML → Markdown (le cœur du produit) | |
| 133 | + | |
| 134 | +Étapes, dans l'ordre, chacune testable isolément : | |
| 135 | + | |
| 136 | +1. **Parse** avec selectolax. Si le HTML est irrécupérable, fallback `lxml.html.fromstring` avec `recover=True`. | |
| 137 | +2. **Clean brut** : supprimer `script`, `style`, `noscript`, `iframe` (sauf embed vidéo → lien), `svg` décoratifs, éléments `hidden`/`display:none`/`aria-hidden`, commentaires, contrôles de formulaire. | |
| 138 | +3. **Suppression boilerplate** par heuristiques cumulées : tags sémantiques (`nav`, `footer`, `aside`, `header` sans `h1`, rôles ARIA), regex classes/ids (`cookie|consent|gdpr|banner|popup|modal|newsletter|share|social|sidebar|related|recommend|advert|promo|breadcrumb|comment…`) sauf si le bloc a l'air d'être le contenu, densité de liens (`liens/mots > 0.6` avec ≥ 4 liens). | |
| 139 | +4. **Scoring contenu principal** (`readability.py`) : score = log(1+n)·n·(1 − densité liens)²·bonus(paragraphes, ponctuation, `article|main|role=main`, classes contenu, titres). Remontée vers le parent tant qu'il n'ajoute pas trop de bruit. Sauté si `only_main_content=false`. | |
| 140 | +5. **Conversion DOM → MD** (`convert.py`) : titres normalisés (un seul `h1`), listes imbriquées, `pre/code` avec `language-*`, liens absolutisés, images (`data:` ignorées si `remove_base64_images`), tables (`tables.py` : thead absent, rowspan/colspan dupliqués, tables de mise en page aplaties), `dl`, `details/summary`, `figure/figcaption`, vidéo/audio → lien. | |
| 141 | +6. **Post-traitement** : lignes vides (>2 → 2), trim, NFC, zero-width, espaces multiples hors code. | |
| 142 | +7. **Citations** (option) : `[texte](url)` → `[texte][n]` + références en fin. | |
| 143 | + | |
| 144 | +Critères de qualité mesurés (tests golden, à constituer) : sur un corpus de 200 pages annotées, le MD doit contenir ≥ 95 % du texte principal attendu et ≤ 5 % de boilerplate. Toute PR touchant `html_to_md/` lance ce benchmark. | |
| 145 | + | |
| 146 | +--- | |
| 147 | + | |
| 148 | +## 5. Stratégie de fetch et escalade (`core/fetcher/strategy.py`) | |
| 149 | + | |
| 150 | +``` | |
| 151 | +auto: | |
| 152 | + 1. GET http_fast (curl_cffi, impersonate=chrome, redirections manuelles + re-check SSRF) | |
| 153 | + → si PDF → processors/pdf | |
| 154 | + → si HTML et heuristique "page vide" (texte visible < 200 chars ET scripts / racine SPA vide) → 2 | |
| 155 | + → si antibot.detect() positif (Cloudflare, DataDome, Akamai, PerimeterX, Kasada, Imperva, captchas) → 3 | |
| 156 | + → si 403/429/503 ou erreur TLS → 2 | |
| 157 | + → sinon OK | |
| 158 | + 2. browser (Playwright, images/fonts/media/pubs bloqués, domcontentloaded puis networkidle ≤ 15 s ou `wait_for`, actions) | |
| 159 | + → si antibot.detect() positif → 3 | |
| 160 | + 3. stealth (browser + patches JS + identité cohérente + proxy résidentiel si configuré) | |
| 161 | + → si toujours bloqué → ErrorInfo(code=BLOCKED, retryable=False, details.trace=[…]) | |
| 162 | +``` | |
| 163 | + | |
| 164 | +- Le mode résolu est mémorisé **par host** (cache 1 h, ne descend jamais). `actions`, `wait_for`, `screenshot` démarrent directement en navigateur. | |
| 165 | +- Pool navigateur : N contexts (défaut = CPU, 2..8), recyclés toutes les 50 pages ou 10 min. Un context = une identité cohérente (UA, viewport, timezone, locale, Accept-Language). | |
| 166 | +- Interception réseau : `image`, `font`, `media`, `stylesheet` + domaines de `antibot/blocklist.txt`, sauf si `screenshot` demandé. | |
| 167 | +- Timeouts : connect 10 s, lecture `timeout_ms`, render `timeout_ms`, téléchargement coupé à `max_size_mb` (slowloris → TIMEOUT_TTFB). | |
| 168 | + | |
| 169 | +--- | |
| 170 | + | |
| 171 | +## 6. Modules fonctionnels | |
| 172 | + | |
| 173 | +### 6.1 Crawl (`worker/crawl.py`) | |
| 174 | +- Frontier priorisée : BFS par défaut ; `strategy: bfs|dfs|best_first` (score BM25 URL/`search`). Amorçage par sitemaps sauf `ignore_sitemap`. | |
| 175 | +- Options : `max_depth`, `max_pages`, `include_paths[]`, `exclude_paths[]` (globs sur path/URL), `allow_subdomains`, `allow_external_links`, `ignore_sitemap`, `respect_robots` (défaut true), `delay_ms`, `concurrency`, `max_duration_s`. | |
| 176 | +- Dédup : normalisation (host minuscule, fragment, tri des params, retrait `utm_*|fbclid|gclid|ref|mc_*…`), puis simhash du MD (distance ≤ 3 → `skipped`). | |
| 177 | +- Chaque page est persistée dès qu'elle est prête ; SSE `page`/`progress`/`done`. | |
| 178 | +- Reprise : frontier sauvegardée en base toutes les 5 s ; jobs `running` au démarrage → `queued` et repris. | |
| 179 | + | |
| 180 | +### 6.2 Map (`map/`) | |
| 181 | +- Sources en parallèle : `robots.txt` → sitemaps (récursif, gzip, index) + crawl shallow http depth 2 (concurrence 16). Common Crawl : à venir. | |
| 182 | +- Sortie : URLs dédupliquées avec `sources[]`, `lastmod`, `depth`, `title` si `include_titles`, `score` si `search` (BM25 URL+title, plafond de collecte ×10 avant tri). | |
| 183 | + | |
| 184 | +### 6.3 PDF (`processors/pdf/`) | |
| 185 | +- Détection : content-type, magic `%PDF`. Extraction `pymupdf` `sort=True`, titres inférés (taille > médiane×1.2 → `##`, ×1.5 → `#`, gras court → `**`), `find_tables()` → MD, scan (< 50 car./page) → OCR rapidocr si installé, lignes coupées rejointes. Limites `max_pages_pdf` (500), `max_size_mb` (50). | |
| 186 | + | |
| 187 | +### 6.4 Extraction structurée (`extract/`) | |
| 188 | +- **CSS** : `{ champ: { selector, attr: text|href|src|html|…, type: str|int|float|date|url|list|bool, multiple } }`. Coercition typée, erreurs par champ. | |
| 189 | +- **LLM** : JSON Schema + `prompt` → MD tronqué par chunks pertinents (BM25-light sur le prompt) → sortie structurée native (response_format / tool use) → validation JSON Schema légère → ≤ 2 corrections. | |
| 190 | +- **Multi-pages** : `urls[]` ou `pattern` (map + glob) → `merge.py` (dédup entités par `merge_key`). Sync ≤ 5 URLs, sinon job. | |
| 191 | + | |
| 192 | +### 6.5 Chunker (`processors/chunker/`) | |
| 193 | +- `by_heading` : sections h1-h6, fusion des petites (< `min_tokens`), split des grandes (> `max_tokens`) sur frontières de phrases avec overlap. `by_tokens` : `size_tokens` (512) + `overlap_tokens` (64). Chaque chunk : `index`, `heading_path`, `token_count`, `char_range`, `url`. | |
| 194 | + | |
| 195 | +### 6.6 Agent — phase 7, non commencé | |
| 196 | +Entrée `{ url, objective, max_steps=30, schema?, allowed_domains?, budget_tokens? }`. Boucle observe (accessibility tree + Set-of-Marks) → décide (LLM → action JSON) → agit (Playwright, vérification post-action) → vérifie (planner, `done` validé contre `schema`). Tools `goto, click, type, scroll, wait, extract, back, done, ask_user`. Garde-fous : pas de soumission de formulaires `password|card|cvv|iban` sans whitelist, domaines autorisés, `max_steps`/budget, journal rejouable. | |
| 197 | + | |
| 198 | +--- | |
| 199 | + | |
| 200 | +## 7. API (`/v1`, OpenAPI sur `/docs` via Scalar) | |
| 201 | + | |
| 202 | +Auth `Authorization: Bearer <clé>` (ou `X-API-Key`, ou `?api_key=` pour les SSE). Sans `TRAWLS_REQUIRE_AUTH`, l'auth est désactivée ; `TRAWLS_ADMIN_KEY` donne accès à `/keys` et `/requests`. | |
| 203 | + | |
| 204 | +| Méthode | Route | Corps | Réponse | | |
| 205 | +|---|---|---|---| | |
| 206 | +| POST | `/scrape` | `{ url, ...ScrapeOptions }` | `PageResult` (sync) | | |
| 207 | +| POST | `/crawl` | `{ url, crawl: CrawlOptions, scrape: ScrapeOptions, webhook? }` | 202 `{ job_id, status, url }` | | |
| 208 | +| GET | `/crawl/{job_id}` | `?cursor&limit&status` | `JobSummary + pages[] + next_cursor` | | |
| 209 | +| GET | `/crawl/{job_id}/stream` | | SSE : `status`, `page`, `progress`, `done`, `error`, `ping` | | |
| 210 | +| DELETE | `/crawl/{job_id}` | | annulation | | |
| 211 | +| POST | `/map` | `{ url, search?, include_subdomains?, limit?, include_titles?, ignore_sitemap? }` | `{ url, count, urls[], took_ms }` | | |
| 212 | +| POST | `/extract` | `{ urls[] \| pattern, mode: css\|llm, css? \| schema?, prompt?, merge_key?, limit? }` | sync si ≤ 5 URLs `{ job_id, data, per_url[] }`, sinon 202 | | |
| 213 | +| POST | `/batch/scrape` | `{ urls[], concurrency?, ...ScrapeOptions }` | 202 `{ job_id }` | | |
| 214 | +| GET | `/jobs`, `/jobs/{id}`, `/jobs/{id}/stream`, DELETE `/jobs/{id}` | | statut générique | | |
| 215 | +| GET | `/jobs/{id}/export?format=jsonl\|zip\|md` | | export | | |
| 216 | +| GET | `/usage` | | requêtes/crédits par jour | | |
| 217 | +| GET/POST/DELETE | `/keys` | admin | clés argon2, valeur affichée une fois | | |
| 218 | +| GET | `/system`, `/blobs/{name}` | | diagnostic, screenshots | | |
| 219 | + | |
| 220 | +Conventions : erreurs `{ error: { code, message, retryable, details? } }` ; pagination par cursor ; idempotence via `Idempotency-Key` ; webhooks signés HMAC-SHA256 `X-Trawls-Signature` (3 essais, pas de redirection). Santé : `/healthz`, `/readyz` (db, browser, worker), `/metrics` Prometheus. | |
| 221 | + | |
| 222 | +Cycle de vie d'un job : `queued → running → completed | failed | cancelled` (`paused` réservé à l'agent). Résultats conservés `RESULT_TTL_DAYS` (7). | |
| 223 | + | |
| 224 | +--- | |
| 225 | + | |
| 226 | +## 8. Taxonomie des erreurs (`models.ErrorCode`) | |
| 227 | + | |
| 228 | +| Code | Récupérable | Déclencheur | | |
| 229 | +|---|---|---| | |
| 230 | +| `TIMEOUT_DNS` / `TIMEOUT_CONNECT` / `TIMEOUT_TTFB` / `TIMEOUT_RENDER` | oui | par étape | | |
| 231 | +| `HTTP_4XX` | non (408 → HTTP_5XX, 429 → RATE_LIMITED) | | | |
| 232 | +| `HTTP_5XX` | oui | | | |
| 233 | +| `RATE_LIMITED` | oui (respecter `Retry-After`) | 429 | | |
| 234 | +| `BLOCKED` | non | antibot après escalade complète (`details.trace`) | | |
| 235 | +| `ROBOTS_DISALLOWED` | non | | | |
| 236 | +| `TOO_LARGE` | non | > `max_size_mb` | | |
| 237 | +| `UNSUPPORTED_CONTENT` | non | binaire inconnu | | |
| 238 | +| `PARSE_FAILED` | non | HTML/PDF irrécupérable | | |
| 239 | +| `SSL_ERROR` | non (option `verify_ssl=false`) | | | |
| 240 | +| `SSRF_REFUSED` / `INVALID_URL` | non | garde sécurité | | |
| 241 | +| `CIRCUIT_OPEN` | oui (plus tard) | breaker host ouvert | | |
| 242 | +| `BUDGET_EXCEEDED` | non | job | | |
| 243 | +| `LLM_INVALID_OUTPUT` | oui (≤ 2) | extraction | | |
| 244 | +| `NETWORK` / `INTERNAL` / `CANCELLED` | NETWORK oui | filets | | |
| 245 | + | |
| 246 | +Retry : max 3, backoff `1s × 2^n + jitter(0-500ms)`, uniquement si `retryable`. Breaker : 5 échecs consécutifs sur un host → ouvert 60 s, half-open ensuite. | |
| 247 | + | |
| 248 | +--- | |
| 249 | + | |
| 250 | +## 9. Tests et qualité | |
| 251 | + | |
| 252 | +- `tests/unit/` : html_to_md (contenu principal, boilerplate, tables rowspan, listes, code, citations, HTML cassé), dedup/simhash, SSRF, détection anti-bot, chunker, extraction CSS, métadonnées. `pytest -q`. | |
| 253 | +- À constituer : `tests/fixtures_server/` (aiohttp : boucles de redirection, HTML tronqué, encodage faux, SPA vide, slowloris, faux challenge Cloudflare, PDF scanné/corrompu, doublons), golden MD `tests/golden/<site>/{input.html, expected.md}` (`make golden-update`), benchmarks vs crawl4ai/Firecrawl → `BENCHMARKS.md`. | |
| 254 | +- `ruff check`, `ruff format --check` bloquants ; `mypy --strict` objectif. | |
| 255 | + | |
| 256 | +--- | |
| 257 | + | |
| 258 | +## 10. Interface web (`trawls/web/static/`) | |
| 259 | + | |
| 260 | +Dark par défaut, fond neutre profond, accent ambre unique, `Inter` + `JetBrains Mono`, densité élevée. Routes servies par FastAPI : `/playground` (URL, options, Run ⌘↵, onglets Markdown/brut/JSON/HTML/Links/Screenshot/Chunks/Metadata, snippets cURL/Python/TS, historique local 20), `/crawls` (formulaire ou JSON brut, table des jobs, détail live SSE, arborescence des URLs, filtre, export ZIP/JSONL/MD), `/map` (liste par tranches, filtre instantané, groupement par path, CSV/JSON, « Crawler la sélection » → batch), `/extract` (builder de schéma ↔ JSON, CSS/LLM, résultat par URL + fusion), `/keys` (système, usage, clés, journal des requêtes), `/docs` (Scalar). Clé API obfusquée en localStorage uniquement. | |
| 261 | + | |
| 262 | +--- | |
| 263 | + | |
| 264 | +## 11. Déploiement et config | |
| 265 | + | |
| 266 | +- **MacLustr** : `deploy/trawls.manifest.json` → `M1M32:~/dispatch/apps/trawls.json` ; `mld stage ~/Desktop/Projets/apps-web/trawls trawls && mld deploy trawls`. PM2 `trawls-api` (uvicorn, port 8180) + `trawls-ngrok` (`www.trawls.dev`). Hook post_sync : venv uv 3.12 + `pip install -e .` + `playwright install chromium`. | |
| 267 | +- Docker : `docker-compose.yml` (api, worker optionnel, ollama sous profil `llm`). `make up`. | |
| 268 | +- Config env `TRAWLS_*` : `PORT`, `PUBLIC_URL`, `DATA_DIR`, `REQUIRE_AUTH`, `ADMIN_KEY`, `BROWSER_POOL_SIZE`, `MAX_CONCURRENCY_PER_HOST`, `MAX_CONCURRENCY_GLOBAL`, `MAX_SIZE_MB`, `PROXY_URLS`, `LLM_PROVIDER|LLM_BASE_URL|LLM_API_KEY|LLM_MODEL`, `RESULT_TTL_DAYS`, `RATE_LIMIT_PER_MINUTE`, `WORKER_CONCURRENCY`, `ALLOW_PRIVATE_TARGETS` (jamais en prod), `EMBEDDED_WORKER`. | |
| 269 | +- Healthchecks `/healthz`, `/readyz` ; métriques `/metrics`. | |
| 270 | + | |
| 271 | +--- | |
| 272 | + | |
| 273 | +## 12. Sécurité | |
| 274 | + | |
| 275 | +- SSRF (`core/security.py`) : refus IP privées/loopback/link-local/multicast, schémas non http(s), `localhost`/`.internal`/metadata ; DNS résolu **avant** la requête et re-vérifié à chaque redirection (http et navigateur). | |
| 276 | +- Limites strictes de taille et de temps ; pas d'`eval` serveur — `actions.evaluate` s'exécute uniquement dans le sandbox Playwright. | |
| 277 | +- Clés API hashées argon2, préfixe indexé, affichées une seule fois. Rate limit token-bucket par clé/IP. Quota journalier par clé. | |
| 278 | +- Webhooks : signature HMAC, 3 essais, pas de redirection suivie. | |
| 279 | + | |
| 280 | +--- | |
| 281 | + | |
| 282 | +## 13. Conventions de développement | |
| 283 | + | |
| 284 | +- Commits conventionnels (`feat`, `fix`, `perf`, `refactor`, `test`, `docs`). Remote `origin` = spbgit (`gitsrv:~/srv/git/trawls.git`). | |
| 285 | +- Une PR = un module ou une fonctionnalité ; description avec « Comment tester ». | |
| 286 | +- Chaque module a une docstring de tête expliquant son rôle et ses invariants. | |
| 287 | +- Pas de `print` hors CLI : `structlog` JSON, `job_id`/`url` dans le contexte. | |
| 288 | +- Pas de `except Exception: pass` dans le chemin chaud. Toute exception attrapée est convertie en `ErrorInfo` typé. | |
| 289 | +- Les options ont des valeurs par défaut sûres ; `verify_ssl=false`, `respect_robots=false`, `allow_private_targets` sont loguées explicitement. | |
| 290 | + | |
| 291 | +--- | |
| 292 | + | |
| 293 | +## 14. Feuille de route et définition de « terminé » | |
| 294 | + | |
| 295 | +| Phase | Contenu | Terminé quand | État | | |
| 296 | +|---|---|---|---| | |
| 297 | +| 1 | `fetcher` (http + browser + auto), `resilience` | 100 % des fixtures produisent un `PageResult` sans exception | livré (fixtures server à écrire) | | |
| 298 | +| 2 | `html_to_md` complet + golden tests | ≥ 95 % de rappel texte, ≤ 5 % boilerplate sur le corpus golden | livré (corpus golden à constituer) | | |
| 299 | +| 3 | API `/scrape`, `/crawl`, jobs, SSE, CLI | Crawl de 1 000 pages sans fuite mémoire, reprise après kill du worker | livré (reprise via frontier SQLite) | | |
| 300 | +| 4 | `map`, `pdf` (texte + tables + OCR) | Map 5 000 URLs < 10 s ; 20 PDF de référence convertis | livré | | |
| 301 | +| 5 | `extract` CSS + LLM + merge, `chunker` | 10 schémas de référence, 0 JSON invalide en sortie | livré | | |
| 302 | +| 6 | UI web : playground, crawls, map, extract, keys | Parité fonctionnelle Firecrawl | livré (vanilla, Next.js cible) | | |
| 303 | +| 7 | `agent` + console | 10 missions de référence réussies | à faire | | |
| 304 | +| 8 | SDK Python/TS, docs, benchmarks publiés | `pip install trawls`, `BENCHMARKS.md` | à faire | | |
| 305 | + | |
| 306 | +Ne pas commencer une phase tant que la précédente n'est pas « terminée » selon ce tableau. | |
added
Dockerfile
+10 −0
@@ -0,0 +1,10 @@ | ||
| 1 | +FROM mcr.microsoft.com/playwright/python:v1.47.0-jammy | |
| 2 | +WORKDIR /app | |
| 3 | +ENV PYTHONUNBUFFERED=1 TRAWLS_HOST=0.0.0.0 TRAWLS_PORT=8180 TRAWLS_DATA_DIR=/data | |
| 4 | +COPY pyproject.toml README.md ./ | |
| 5 | +COPY trawls ./trawls | |
| 6 | +RUN pip install --no-cache-dir -e . && playwright install chromium | |
| 7 | +VOLUME ["/data"] | |
| 8 | +EXPOSE 8180 | |
| 9 | +HEALTHCHECK --interval=30s --timeout=5s CMD curl -fsS http://localhost:8180/healthz || exit 1 | |
| 10 | +CMD ["trawls", "serve"] | |
added
Makefile
+29 −0
@@ -0,0 +1,29 @@ | ||
| 1 | +.PHONY: install dev serve test lint fmt up down golden-update | |
| 2 | + | |
| 3 | +install: | |
| 4 | + uv venv --python 3.12 || true | |
| 5 | + . .venv/bin/activate && uv pip install -e ".[dev]" && playwright install chromium | |
| 6 | + | |
| 7 | +serve: | |
| 8 | + . .venv/bin/activate && trawls serve | |
| 9 | + | |
| 10 | +dev: | |
| 11 | + . .venv/bin/activate && TRAWLS_LOG_LEVEL=DEBUG trawls serve --reload | |
| 12 | + | |
| 13 | +test: | |
| 14 | + . .venv/bin/activate && pytest -q | |
| 15 | + | |
| 16 | +lint: | |
| 17 | + . .venv/bin/activate && ruff check . && ruff format --check . | |
| 18 | + | |
| 19 | +fmt: | |
| 20 | + . .venv/bin/activate && ruff format . && ruff check --fix . | |
| 21 | + | |
| 22 | +up: | |
| 23 | + docker compose up -d --build | |
| 24 | + | |
| 25 | +down: | |
| 26 | + docker compose down | |
| 27 | + | |
| 28 | +golden-update: | |
| 29 | + . .venv/bin/activate && TRAWLS_GOLDEN_UPDATE=1 pytest -q tests/unit/test_html_to_md.py | |
added
README.md
+88 −0
@@ -0,0 +1,88 @@ | ||
| 1 | +# Trawls — www.trawls.dev | |
| 2 | + | |
| 3 | +Moteur de crawling, d'extraction et de navigation web, construit from scratch. Un chalut (*trawl*) ratisse le fond et remonte tout : Trawls cartographie et extrait n'importe quel site et renvoie du **Markdown propre, LLM-ready**. | |
| 4 | + | |
| 5 | +- **Ne crashe jamais** : une page qui échoue produit un `PageResult` `status=failed` avec une erreur typée. Le crawl continue. | |
| 6 | +- **Escalade automatique** : HTTP pur (curl_cffi, empreinte TLS Chrome) → navigateur (Playwright, pool de contexts) → stealth (patches JS, identité cohérente). Le mode résolu est mémorisé par host. | |
| 7 | +- **Sortie LLM-ready** : pipeline HTML → Markdown maison (nettoyage, suppression du boilerplate, scoring du contenu principal, tables avec rowspan/colspan, citations), métadonnées fusionnées (head, OpenGraph, JSON-LD), chunks par titres ou par tokens. | |
| 8 | +- **PDF** : texte avec ordre de lecture, titres inférés, tables, détection de scan (OCR optionnel). | |
| 9 | +- **Extraction structurée** : CSS typé (déterministe) ou LLM (JSON Schema, validation, corrections) avec fusion multi-pages. | |
| 10 | +- **Self-host first** : SQLite, worker embarqué, aucun service externe requis. LLM local via Ollama par défaut. | |
| 11 | +- **Observable** : structlog JSON, `/healthz`, `/readyz`, `/metrics` Prometheus, trace d'escalade par page. | |
| 12 | + | |
| 13 | +## Démarrage | |
| 14 | + | |
| 15 | +```bash | |
| 16 | +uv venv --python 3.12 && source .venv/bin/activate | |
| 17 | +uv pip install -e ".[dev]" | |
| 18 | +playwright install chromium | |
| 19 | +trawls serve # http://localhost:8180 (UI + API + worker) | |
| 20 | +``` | |
| 21 | + | |
| 22 | +```bash | |
| 23 | +trawls scrape https://fr.wikipedia.org/wiki/Chalut | |
| 24 | +trawls map https://docs.python.org --search "asyncio" | |
| 25 | +trawls crawl https://exemple.com --max-pages 50 -o pages.jsonl | |
| 26 | +``` | |
| 27 | + | |
| 28 | +Docker : `make up` (api + web sur http://localhost:8180). | |
| 29 | + | |
| 30 | +## API (`/v1`, OpenAPI sur `/docs`) | |
| 31 | + | |
| 32 | +| Méthode | Route | Rôle | | |
| 33 | +|---|---|---| | |
| 34 | +| POST | `/scrape` | une page, synchrone → `PageResult` | | |
| 35 | +| POST | `/crawl` → GET `/crawl/{id}` · `/crawl/{id}/stream` (SSE) · DELETE | crawl asynchrone, pages streamées | | |
| 36 | +| POST | `/map` | liste d'URLs (sitemaps + crawl shallow), tri BM25 si `search` | | |
| 37 | +| POST | `/extract` | CSS ou LLM sur `urls[]` / `pattern` ; sync ≤ 5 URLs | | |
| 38 | +| POST | `/batch/scrape` | N URLs en job | | |
| 39 | +| GET | `/jobs`, `/jobs/{id}`, `/jobs/{id}/stream`, `/jobs/{id}/export?format=zip\|jsonl\|md` | jobs génériques | | |
| 40 | +| GET | `/usage` · `/keys` (admin) | crédits, clés API (argon2, affichées une fois) | | |
| 41 | + | |
| 42 | +Erreurs : `{ "error": { "code", "message", "retryable", "details?" } }`. Idempotence : header `Idempotency-Key`. Webhooks signés `X-Trawls-Signature` (HMAC-SHA256). | |
| 43 | + | |
| 44 | +```bash | |
| 45 | +curl -s https://www.trawls.dev/v1/scrape -H 'Content-Type: application/json' \ | |
| 46 | + -d '{"url":"https://example.com","formats":["markdown","links"]}' | jq .markdown | |
| 47 | +``` | |
| 48 | + | |
| 49 | +## Configuration (préfixe `TRAWLS_`) | |
| 50 | + | |
| 51 | +`PORT` (8180), `PUBLIC_URL`, `DATA_DIR`, `REQUIRE_AUTH`, `ADMIN_KEY`, `BROWSER_POOL_SIZE`, `MAX_CONCURRENCY_PER_HOST`, `MAX_SIZE_MB`, `PROXY_URLS`, `LLM_PROVIDER` (`openai`|`anthropic`|`none`), `LLM_BASE_URL` (Ollama par défaut), `LLM_API_KEY`, `LLM_MODEL`, `RESULT_TTL_DAYS`, `RATE_LIMIT_PER_MINUTE`, `ALLOW_PRIVATE_TARGETS` (jamais en prod). Voir `.env.example`. | |
| 52 | + | |
| 53 | +## Architecture | |
| 54 | + | |
| 55 | +``` | |
| 56 | +trawls/ | |
| 57 | + config.py settings pydantic | |
| 58 | + models/ contrat Pydantic v2 (ScrapeOptions, PageResult, ErrorInfo…) | |
| 59 | + core/fetcher/ http_fast (curl_cffi), browser (pool Playwright), strategy (escalade) | |
| 60 | + core/antibot/ detect, identity, stealth, blocklist | |
| 61 | + core/scheduler/ dedup (normalisation + simhash), robots, politeness | |
| 62 | + core/resilience/ retry, breaker, budget | |
| 63 | + core/security.py garde SSRF (DNS résolu avant, re-vérifié après redirection) | |
| 64 | + core/scrape.py pipeline d'une page — ne lève jamais | |
| 65 | + processors/ html_to_md (clean, readability, convert, tables, citations), pdf, structured, chunker, links | |
| 66 | + extract/ css, llm, merge | |
| 67 | + map/ sitemaps + crawl shallow + BM25 | |
| 68 | + llm/ client unique (Anthropic, OpenAI-compatible) | |
| 69 | + worker/ store SQLite, moteur de jobs (crawl/batch/extract), SSE | |
| 70 | + api/ FastAPI /v1, auth, UI | |
| 71 | + web/static/ UI (playground, crawls, map, extract, keys) | |
| 72 | + cli.py | |
| 73 | +``` | |
| 74 | + | |
| 75 | +## Tests | |
| 76 | + | |
| 77 | +```bash | |
| 78 | +pytest -q # unitaires (html_to_md, dedup, ssrf, detect, chunker, css) | |
| 79 | +ruff check . && ruff format --check . | |
| 80 | +``` | |
| 81 | + | |
| 82 | +## Déploiement MacLustr | |
| 83 | + | |
| 84 | +Déployée via `mld` (manifeste `trawls.json`) : PM2 `trawls-api` + `trawls-ngrok` → https://www.trawls.dev. Voir `deploy/trawls.manifest.json`. | |
| 85 | + | |
| 86 | +## Feuille de route | |
| 87 | + | |
| 88 | +Phases 1–6 livrées (fetch + escalade, Markdown, API/jobs/SSE/CLI, map + PDF, extract + chunker, UI). Phase 7 (agent) et 8 (SDK, benchmarks) à venir — voir `CLAUDE.md`. | |
added
deploy/trawls.manifest.json
+82 −0
@@ -0,0 +1,82 @@ | ||
| 1 | +{ | |
| 2 | + "app": "trawls", | |
| 3 | + "label": "Trawls — crawl, extract, navigate", | |
| 4 | + "domain": "www.trawls.dev", | |
| 5 | + "port": 8180, | |
| 6 | + "health_path": "/healthz", | |
| 7 | + "dir": "~/apps/trawls", | |
| 8 | + "extra_paths": [], | |
| 9 | + "sync_excludes": [ | |
| 10 | + ".venv/", | |
| 11 | + "venv/", | |
| 12 | + "data/", | |
| 13 | + "__pycache__/", | |
| 14 | + ".pytest_cache/", | |
| 15 | + ".ruff_cache/", | |
| 16 | + ".mypy_cache/", | |
| 17 | + "*.egg-info/", | |
| 18 | + ".git/", | |
| 19 | + ".env" | |
| 20 | + ], | |
| 21 | + "requires": { | |
| 22 | + "runtimes": ["pm2", "ngrok", "uv-python@3.12", "uv"], | |
| 23 | + "ram_gb": 3, | |
| 24 | + "ports": [8180] | |
| 25 | + }, | |
| 26 | + "ram_mb_observed": 1500, | |
| 27 | + "size_mb": 5, | |
| 28 | + "placement": { | |
| 29 | + "pin": null, | |
| 30 | + "prefer": "M3U96a", | |
| 31 | + "avoid": ["M3U96b", "M1M32"], | |
| 32 | + "reason": "Playwright/Chromium : beaucoup de cœurs et RAM libre ; M3U96b réservé hfmarketdata" | |
| 33 | + }, | |
| 34 | + "processes": [ | |
| 35 | + { | |
| 36 | + "name": "trawls-api", | |
| 37 | + "manager": "pm2", | |
| 38 | + "script": "{{HOME}}/apps/trawls/.venv/bin/python", | |
| 39 | + "args": ["-m", "uvicorn", "trawls.api.main:app", "--host", "0.0.0.0", "--port", "8180", "--no-access-log"], | |
| 40 | + "interpreter": null, | |
| 41 | + "cwd": "{{HOME}}/apps/trawls", | |
| 42 | + "env": { | |
| 43 | + "TRAWLS_ENV": "production", | |
| 44 | + "TRAWLS_PORT": "8180", | |
| 45 | + "TRAWLS_PUBLIC_URL": "https://www.trawls.dev", | |
| 46 | + "TRAWLS_DATA_DIR": "{{HOME}}/apps/trawls/data", | |
| 47 | + "TRAWLS_BROWSER_POOL_SIZE": "6", | |
| 48 | + "TRAWLS_WORKER_CONCURRENCY": "4", | |
| 49 | + "TRAWLS_MAX_CONCURRENCY_GLOBAL": "24", | |
| 50 | + "TRAWLS_LLM_PROVIDER": "openai", | |
| 51 | + "TRAWLS_LLM_BASE_URL": "https://llm.maclustr.io/v1", | |
| 52 | + "TRAWLS_LLM_MODEL": "maclustr-general", | |
| 53 | + "TRAWLS_ADMIN_KEY": "trawls-admin-2026-Qx7mP3vL9kR2", | |
| 54 | + "TRAWLS_LOG_LEVEL": "INFO", | |
| 55 | + "PYTHONUNBUFFERED": "1", | |
| 56 | + "PLAYWRIGHT_BROWSERS_PATH": "{{HOME}}/Library/Caches/ms-playwright" | |
| 57 | + }, | |
| 58 | + "cron_restart": null, | |
| 59 | + "autorestart": true, | |
| 60 | + "max_memory_restart": "6G" | |
| 61 | + } | |
| 62 | + ], | |
| 63 | + "ngrok": { | |
| 64 | + "name": "trawls-ngrok", | |
| 65 | + "url": "www.trawls.dev", | |
| 66 | + "port": 8180 | |
| 67 | + }, | |
| 68 | + "launchd": [], | |
| 69 | + "env_overrides": {}, | |
| 70 | + "hooks": { | |
| 71 | + "post_sync": [ | |
| 72 | + "export PATH=\"$HOME/.local/bin:/opt/homebrew/bin:$PATH\"; test -x .venv/bin/python || uv venv --python 3.12 .venv && echo ' venv ok'", | |
| 73 | + "export PATH=\"$HOME/.local/bin:/opt/homebrew/bin:$PATH\"; uv pip install -q --python .venv/bin/python -e . && echo ' deps python ok'", | |
| 74 | + "PLAYWRIGHT_BROWSERS_PATH=$HOME/Library/Caches/ms-playwright .venv/bin/playwright install chromium 2>&1 | tail -1; echo ' chromium ok'", | |
| 75 | + "mkdir -p data/blobs && echo ' data ok'" | |
| 76 | + ], | |
| 77 | + "post_start": [] | |
| 78 | + }, | |
| 79 | + "ka_repo": false, | |
| 80 | + "source_node": null, | |
| 81 | + "rebuild": false | |
| 82 | +} | |
added
docker-compose.yml
+25 −0
@@ -0,0 +1,25 @@ | ||
| 1 | +services: | |
| 2 | + api: | |
| 3 | + build: . | |
| 4 | + ports: ["8180:8180"] | |
| 5 | + environment: | |
| 6 | + TRAWLS_PUBLIC_URL: http://localhost:8180 | |
| 7 | + TRAWLS_LLM_BASE_URL: http://ollama:11434/v1 | |
| 8 | + TRAWLS_LLM_MODEL: qwen2.5:7b | |
| 9 | + volumes: ["trawls-data:/data"] | |
| 10 | + restart: unless-stopped | |
| 11 | + worker: | |
| 12 | + build: . | |
| 13 | + command: ["trawls", "worker"] | |
| 14 | + environment: | |
| 15 | + TRAWLS_DATA_DIR: /data | |
| 16 | + volumes: ["trawls-data:/data"] | |
| 17 | + depends_on: [api] | |
| 18 | + deploy: { replicas: 0 } # `docker compose up --scale worker=2` pour des workers séparés (API avec TRAWLS_EMBEDDED_WORKER=0) | |
| 19 | + ollama: | |
| 20 | + image: ollama/ollama | |
| 21 | + volumes: ["ollama:/root/.ollama"] | |
| 22 | + profiles: ["llm"] | |
| 23 | +volumes: | |
| 24 | + trawls-data: {} | |
| 25 | + ollama: {} | |
added
pyproject.toml
+61 −0
@@ -0,0 +1,61 @@ | ||
| 1 | +[project] | |
| 2 | +name = "trawls" | |
| 3 | +version = "0.1.0" | |
| 4 | +description = "Trawls — moteur de crawling, d'extraction et de navigation web, LLM-ready." | |
| 5 | +readme = "README.md" | |
| 6 | +requires-python = ">=3.12" | |
| 7 | +license = { text = "MIT" } | |
| 8 | +authors = [{ name = "Simon-Pierre Boucher" }] | |
| 9 | +dependencies = [ | |
| 10 | + "fastapi>=0.115", | |
| 11 | + "uvicorn[standard]>=0.30", | |
| 12 | + "pydantic>=2.8", | |
| 13 | + "pydantic-settings>=2.4", | |
| 14 | + "curl_cffi>=0.7", | |
| 15 | + "httpx[http2]>=0.27", | |
| 16 | + "playwright>=1.47", | |
| 17 | + "selectolax>=0.3.21", | |
| 18 | + "lxml>=5.2", | |
| 19 | + "charset-normalizer>=3.3", | |
| 20 | + "pymupdf>=1.24", | |
| 21 | + "tiktoken>=0.7", | |
| 22 | + "structlog>=24.1", | |
| 23 | + "aiosqlite>=0.20", | |
| 24 | + "typer>=0.12", | |
| 25 | + "rich>=13.7", | |
| 26 | + "python-dateutil>=2.9", | |
| 27 | + "sse-starlette>=2.1", | |
| 28 | + "argon2-cffi>=23.1", | |
| 29 | + "orjson>=3.10", | |
| 30 | +] | |
| 31 | + | |
| 32 | +[project.optional-dependencies] | |
| 33 | +dev = ["pytest>=8", "pytest-asyncio>=0.23", "ruff>=0.5", "mypy>=1.10", "aiohttp>=3.9"] | |
| 34 | +ocr = ["rapidocr-onnxruntime>=1.3"] | |
| 35 | + | |
| 36 | +[project.scripts] | |
| 37 | +trawls = "trawls.cli:app" | |
| 38 | + | |
| 39 | +[build-system] | |
| 40 | +requires = ["hatchling"] | |
| 41 | +build-backend = "hatchling.build" | |
| 42 | + | |
| 43 | +[tool.hatch.build.targets.wheel] | |
| 44 | +packages = ["trawls"] | |
| 45 | + | |
| 46 | +[tool.ruff] | |
| 47 | +line-length = 110 | |
| 48 | +target-version = "py312" | |
| 49 | + | |
| 50 | +[tool.ruff.lint] | |
| 51 | +select = ["E", "F", "I", "UP", "B", "ASYNC"] | |
| 52 | +ignore = ["E501", "B008", "E402", "UP047", "ASYNC109", "ASYNC230"] | |
| 53 | + | |
| 54 | +[tool.mypy] | |
| 55 | +python_version = "3.12" | |
| 56 | +strict = true | |
| 57 | +ignore_missing_imports = true | |
| 58 | + | |
| 59 | +[tool.pytest.ini_options] | |
| 60 | +asyncio_mode = "auto" | |
| 61 | +testpaths = ["tests"] | |
added
tests/unit/test_core.py
+129 −0
@@ -0,0 +1,129 @@ | ||
| 1 | +import pytest | |
| 2 | + | |
| 3 | +from trawls.core.antibot.detect import detect | |
| 4 | +from trawls.core.scheduler.dedup import ContentDeduper, hamming, normalize_url, same_site, simhash | |
| 5 | +from trawls.core.security import SsrfError, check_url | |
| 6 | +from trawls.extract.css import extract_css | |
| 7 | +from trawls.models import ChunkOptions, CssField, ErrorCode | |
| 8 | +from trawls.processors.chunker import chunk | |
| 9 | +from trawls.processors.structured.metadata import extract_metadata | |
| 10 | + | |
| 11 | + | |
| 12 | +def test_normalize_url() -> None: | |
| 13 | + assert normalize_url("HTTP://Ex.COM:80/a/b/?utm_source=x&b=2&a=1#frag") == "http://ex.com/a/b?a=1&b=2" | |
| 14 | + assert normalize_url("https://ex.com") == "https://ex.com/" | |
| 15 | + assert normalize_url("/rel", "https://ex.com/dir/page") == "https://ex.com/rel" | |
| 16 | + assert normalize_url("javascript:void(0)") is None | |
| 17 | + assert normalize_url("mailto:a@b.c") is None | |
| 18 | + assert normalize_url("https://ex.com/a?fbclid=1&gclid=2") == "https://ex.com/a" | |
| 19 | + | |
| 20 | + | |
| 21 | +def test_same_site() -> None: | |
| 22 | + assert same_site("https://ex.com/a", "https://ex.com/b", False) | |
| 23 | + assert not same_site("https://ex.com/a", "https://blog.ex.com/b", False) | |
| 24 | + assert same_site("https://ex.com/a", "https://blog.ex.com/b", True) | |
| 25 | + assert not same_site("https://ex.com/a", "https://other.com/b", True) | |
| 26 | + | |
| 27 | + | |
| 28 | +def test_simhash_dedup() -> None: | |
| 29 | + a = "Le chalut est un filet remorqué par un navire. " * 20 | |
| 30 | + b = a.replace("navire", "bateau", 1) | |
| 31 | + c = "Texte complètement différent sur les oiseaux migrateurs et leurs routes. " * 20 | |
| 32 | + assert hamming(simhash(a), simhash(b)) <= 3 | |
| 33 | + assert hamming(simhash(a), simhash(c)) > 10 | |
| 34 | + d = ContentDeduper() | |
| 35 | + assert d.is_duplicate(a, "u1") is None | |
| 36 | + assert d.is_duplicate(b, "u2") == "u1" | |
| 37 | + assert d.is_duplicate(c, "u3") is None | |
| 38 | + | |
| 39 | + | |
| 40 | +@pytest.mark.asyncio | |
| 41 | +async def test_ssrf() -> None: | |
| 42 | + for bad in ( | |
| 43 | + "http://127.0.0.1/", | |
| 44 | + "http://localhost:8080/x", | |
| 45 | + "http://169.254.169.254/latest", | |
| 46 | + "http://10.0.0.1/", | |
| 47 | + "http://192.168.2.1/", | |
| 48 | + "ftp://ex.com/", | |
| 49 | + "http://[::1]/", | |
| 50 | + "http://0.0.0.0/", | |
| 51 | + ): | |
| 52 | + with pytest.raises(SsrfError): | |
| 53 | + await check_url(bad) | |
| 54 | + assert await check_url("https://1.1.1.1/") == "https://1.1.1.1/" | |
| 55 | + | |
| 56 | + | |
| 57 | +def test_detect() -> None: | |
| 58 | + assert ( | |
| 59 | + detect( | |
| 60 | + "<html><title>Just a moment...</title><div id='cf-browser-verification'></div></html>", 503 | |
| 61 | + ).kind | |
| 62 | + == "cloudflare" | |
| 63 | + ) | |
| 64 | + assert ( | |
| 65 | + detect('<html><body><div id="root"></div><script src="/app.js"></script></body></html>', 200).kind | |
| 66 | + == "spa" | |
| 67 | + ) | |
| 68 | + ok = detect("<html><body><p>" + "Du contenu normal et long. " * 50 + "</p></body></html>", 200) | |
| 69 | + assert ok.kind is None | |
| 70 | + assert detect("<html><body><h1>Access Denied</h1></body></html>", 403).blocked | |
| 71 | + | |
| 72 | + | |
| 73 | +def test_chunker() -> None: | |
| 74 | + md = ( | |
| 75 | + "# Titre\n\nIntro courte.\n\n## Section A\n\n" | |
| 76 | + + ("Phrase de la section A. " * 80) | |
| 77 | + + "\n\n## Section B\n\nPetite.\n" | |
| 78 | + ) | |
| 79 | + cs = chunk( | |
| 80 | + md, | |
| 81 | + "https://ex.com", | |
| 82 | + ChunkOptions(strategy="by_heading", max_tokens=200, min_tokens=20, overlap_tokens=20), | |
| 83 | + ) | |
| 84 | + assert len(cs) >= 3 | |
| 85 | + assert all(c.token_count <= 260 for c in cs) | |
| 86 | + assert any("Section A" in c.heading_path for c in cs) | |
| 87 | + assert cs[0].index == 0 and cs[-1].index == len(cs) - 1 | |
| 88 | + ct = chunk("Mot. " * 2000, "u", ChunkOptions(strategy="by_tokens", size_tokens=100, overlap_tokens=10)) | |
| 89 | + assert len(ct) > 10 and all(c.token_count <= 130 for c in ct) | |
| 90 | + | |
| 91 | + | |
| 92 | +def test_css_extract() -> None: | |
| 93 | + html = "<html><body><h1> Produit X </h1><span class='price'>1 234,50 $</span><a class='more' href='/p/1'>+</a><ul><li>a</li><li>b</li></ul><time datetime='2026-09-05'>5 sept</time></body></html>" | |
| 94 | + data, errs = extract_css( | |
| 95 | + html, | |
| 96 | + { | |
| 97 | + "title": CssField(selector="h1"), | |
| 98 | + "price": CssField(selector=".price", type="float"), | |
| 99 | + "url": CssField(selector="a.more", attr="href", type="url"), | |
| 100 | + "items": CssField(selector="li", multiple=True), | |
| 101 | + "date": CssField(selector="time", attr="datetime", type="date"), | |
| 102 | + "missing": CssField(selector=".nope", type="int"), | |
| 103 | + }, | |
| 104 | + "https://ex.com/x", | |
| 105 | + ) | |
| 106 | + assert data["title"] == "Produit X" | |
| 107 | + assert data["price"] == 1234.5 | |
| 108 | + assert data["url"] == "https://ex.com/p/1" | |
| 109 | + assert data["items"] == ["a", "b"] | |
| 110 | + assert data["date"].startswith("2026-09-05") | |
| 111 | + assert data["missing"] is None and "missing" in errs | |
| 112 | + | |
| 113 | + | |
| 114 | +def test_metadata() -> None: | |
| 115 | + html = """<html lang="fr-CA"><head><title>T</title><meta property="og:title" content="OG T"><meta name="description" content="D"> | |
| 116 | + <link rel="canonical" href="/canon"><script type="application/ld+json">{"@type":"Article","headline":"H","author":{"name":"Simon"},"datePublished":"2026-01-02"}</script></head><body></body></html>""" | |
| 117 | + m = extract_metadata(html, "https://ex.com/p") | |
| 118 | + assert m.title == "OG T" and m.description == "D" and m.language == "fr" | |
| 119 | + assert m.canonical_url == "https://ex.com/canon" | |
| 120 | + assert m.author == "Simon" and m.published_at == "2026-01-02" | |
| 121 | + assert m.jsonld and m.jsonld[0]["@type"] == "Article" | |
| 122 | + | |
| 123 | + | |
| 124 | +def test_error_codes_retryable() -> None: | |
| 125 | + from trawls.models import ErrorInfo | |
| 126 | + | |
| 127 | + assert ErrorInfo.make(ErrorCode.HTTP_5XX, "x").retryable | |
| 128 | + assert not ErrorInfo.make(ErrorCode.BLOCKED, "x").retryable | |
| 129 | + assert not ErrorInfo.make(ErrorCode.ROBOTS_DISALLOWED, "x").retryable | |
added
tests/unit/test_html_to_md.py
+92 −0
@@ -0,0 +1,92 @@ | ||
| 1 | +from trawls.processors.html_to_md import html_to_markdown | |
| 2 | + | |
| 3 | +ARTICLE = """ | |
| 4 | +<html lang="fr"><head><title>Titre</title><style>.x{}</style><script>var a=1;</script></head> | |
| 5 | +<body> | |
| 6 | +<nav><a href="/">Accueil</a><a href="/blog">Blog</a><a href="/contact">Contact</a><a href="/a">A</a></nav> | |
| 7 | +<div class="cookie-banner">Nous utilisons des cookies. <a href="#">Accepter</a></div> | |
| 8 | +<main> | |
| 9 | +<article> | |
| 10 | +<h1>Le chalut de fond</h1> | |
| 11 | +<p>Le <strong>chalut</strong> est un filet <em>remorqué</em> par un navire. Il ratisse le fond et remonte tout ce qui s'y trouve, ce qui en fait un engin très efficace mais controversé.</p> | |
| 12 | +<h2>Fonctionnement</h2> | |
| 13 | +<p>Le filet est maintenu ouvert par des panneaux. Voir <a href="/details">les détails</a>.</p> | |
| 14 | +<ul><li>Panneaux divergents</li><li>Bourrelet<ul><li>Diabolos</li></ul></li></ul> | |
| 15 | +<table><tr><th>Type</th><th>Profondeur</th></tr><tr><td>Fond</td><td>200 m</td></tr><tr><td rowspan="2">Pélagique</td><td>50 m</td></tr><tr><td>100 m</td></tr></table> | |
| 16 | +<pre><code class="language-python">print("ok")</code></pre> | |
| 17 | +<img src="/img/chalut.png" alt="Un chalut"> | |
| 18 | +<img src="data:image/png;base64,AAAA" alt="b64"> | |
| 19 | +</article> | |
| 20 | +</main> | |
| 21 | +<aside class="sidebar"><a href="/1">Lien 1</a><a href="/2">Lien 2</a><a href="/3">Lien 3</a><a href="/4">Lien 4</a></aside> | |
| 22 | +<footer>© 2026 Exemple — <a href="/mentions">Mentions légales</a></footer> | |
| 23 | +</body></html> | |
| 24 | +""" | |
| 25 | + | |
| 26 | + | |
| 27 | +def test_main_content_and_boilerplate() -> None: | |
| 28 | + r = html_to_markdown(ARTICLE, "https://ex.com/a/b") | |
| 29 | + md = r.markdown | |
| 30 | + assert md.startswith("# Le chalut de fond") | |
| 31 | + assert "**chalut**" in md and "*remorqué*" in md | |
| 32 | + assert "## Fonctionnement" in md | |
| 33 | + assert "[les détails](https://ex.com/details)" in md | |
| 34 | + assert "- Panneaux divergents" in md and " - Diabolos" in md | |
| 35 | + assert "| Type | Profondeur |" in md and "| Pélagique | 100 m |" in md | |
| 36 | + assert '```python\nprint("ok")\n```' in md | |
| 37 | + assert "" in md | |
| 38 | + assert "b64" not in md | |
| 39 | + for bad in ("Accueil", "cookies", "Mentions légales", "Lien 1"): | |
| 40 | + assert bad not in md, bad | |
| 41 | + | |
| 42 | + | |
| 43 | +def test_full_page_keeps_nav_text() -> None: | |
| 44 | + r = html_to_markdown(ARTICLE, "https://ex.com", only_main_content=False) | |
| 45 | + assert "Accueil" in r.markdown and "Le chalut de fond" in r.markdown | |
| 46 | + | |
| 47 | + | |
| 48 | +def test_broken_html_never_raises() -> None: | |
| 49 | + for html in ( | |
| 50 | + "", | |
| 51 | + "<div><p>ouvert", | |
| 52 | + "<<<>>>", | |
| 53 | + "<html><body><table><tr><td>x</table>", | |
| 54 | + "\x00\x01garbage", | |
| 55 | + "<p>" * 5000, | |
| 56 | + ): | |
| 57 | + r = html_to_markdown(html, "https://ex.com") | |
| 58 | + assert isinstance(r.markdown, str) | |
| 59 | + | |
| 60 | + | |
| 61 | +def test_citations() -> None: | |
| 62 | + r = html_to_markdown( | |
| 63 | + "<article><p>Voir <a href='https://a.com'>A</a> et <a href='https://b.com'>B</a> puis <a href='https://a.com'>A encore</a>. Un paragraphe assez long pour que le scoring le garde comme contenu principal, avec plusieurs phrases, des virgules, et du texte.</p></article>", | |
| 64 | + "https://ex.com", | |
| 65 | + citations=True, | |
| 66 | + ) | |
| 67 | + assert "[A][1]" in r.markdown and "[B][2]" in r.markdown and "[A encore][1]" in r.markdown | |
| 68 | + assert "[1]: https://a.com" in r.markdown | |
| 69 | + | |
| 70 | + | |
| 71 | +def test_headings_normalized() -> None: | |
| 72 | + r = html_to_markdown( | |
| 73 | + "<body><h1>Un</h1><p>Texte assez long pour compter comme du contenu, avec des phrases et de la ponctuation. Encore une phrase ici.</p><h1>Deux</h1><p>Suite du texte, encore quelques mots pour la densité, et une virgule.</p></body>", | |
| 74 | + "https://ex.com", | |
| 75 | + only_main_content=False, | |
| 76 | + ) | |
| 77 | + assert r.markdown.count("\n# ") + (1 if r.markdown.startswith("# ") else 0) == 1 | |
| 78 | + assert "## Deux" in r.markdown | |
| 79 | + | |
| 80 | + | |
| 81 | +def test_definition_list_and_details() -> None: | |
| 82 | + r = html_to_markdown( | |
| 83 | + "<body><dl><dt>Terme</dt><dd>Définition</dd></dl><details><summary>Plus</summary><p>Caché</p></details></body>", | |
| 84 | + "https://ex.com", | |
| 85 | + only_main_content=False, | |
| 86 | + ) | |
| 87 | + assert ( | |
| 88 | + "**Terme**" in r.markdown | |
| 89 | + and ": Définition" in r.markdown | |
| 90 | + and "**Plus**" in r.markdown | |
| 91 | + and "Caché" in r.markdown | |
| 92 | + ) | |
added
trawls/__init__.py
+3 −0
@@ -0,0 +1,3 @@ | ||
| 1 | +"""Trawls — moteur de crawling, d extraction et de navigation web, LLM-ready.""" | |
| 2 | + | |
| 3 | +__version__ = "0.1.0" | |
added
trawls/api/__init__.py
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | + | |
added
trawls/api/auth.py
+103 −0
@@ -0,0 +1,103 @@ | ||
| 1 | +"""Clés API (argon2, affichées une fois), quotas journaliers, rate limit token-bucket en mémoire. | |
| 2 | +En self-host sans TRAWLS_REQUIRE_AUTH, l'auth est désactivée (identité anonyme).""" | |
| 3 | + | |
| 4 | +from __future__ import annotations | |
| 5 | + | |
| 6 | +import time | |
| 7 | +from dataclasses import dataclass | |
| 8 | + | |
| 9 | +from fastapi import Depends, HTTPException, Request | |
| 10 | + | |
| 11 | +from trawls.config import get_settings | |
| 12 | +from trawls.worker.store import get_store | |
| 13 | + | |
| 14 | + | |
| 15 | +@dataclass | |
| 16 | +class Identity: | |
| 17 | + key_id: str | None | |
| 18 | + name: str | |
| 19 | + is_admin: bool = False | |
| 20 | + quota_per_day: int | None = None | |
| 21 | + | |
| 22 | + | |
| 23 | +class TokenBucket: | |
| 24 | + def __init__(self, rate_per_min: int) -> None: | |
| 25 | + self.rate = rate_per_min / 60.0 | |
| 26 | + self.cap = float(rate_per_min) | |
| 27 | + self._b: dict[str, tuple[float, float]] = {} | |
| 28 | + | |
| 29 | + def take(self, key: str) -> bool: | |
| 30 | + now = time.monotonic() | |
| 31 | + tokens, last = self._b.get(key, (self.cap, now)) | |
| 32 | + tokens = min(self.cap, tokens + (now - last) * self.rate) | |
| 33 | + if tokens < 1: | |
| 34 | + self._b[key] = (tokens, now) | |
| 35 | + return False | |
| 36 | + self._b[key] = (tokens - 1, now) | |
| 37 | + return True | |
| 38 | + | |
| 39 | + | |
| 40 | +_bucket: TokenBucket | None = None | |
| 41 | + | |
| 42 | + | |
| 43 | +def bucket() -> TokenBucket: | |
| 44 | + global _bucket | |
| 45 | + if _bucket is None: | |
| 46 | + _bucket = TokenBucket(get_settings().rate_limit_per_minute) | |
| 47 | + return _bucket | |
| 48 | + | |
| 49 | + | |
| 50 | +def _bearer(request: Request) -> str | None: | |
| 51 | + h = request.headers.get("authorization") or "" | |
| 52 | + if h.lower().startswith("bearer "): | |
| 53 | + return h[7:].strip() | |
| 54 | + return request.headers.get("x-api-key") or request.query_params.get("api_key") | |
| 55 | + | |
| 56 | + | |
| 57 | +async def current_identity(request: Request) -> Identity: | |
| 58 | + s = get_settings() | |
| 59 | + raw = _bearer(request) | |
| 60 | + if raw and s.admin_key and raw == s.admin_key: | |
| 61 | + ident = Identity(key_id="admin", name="admin", is_admin=True) | |
| 62 | + elif raw and raw.startswith("trw_"): | |
| 63 | + row = await get_store().verify_key(raw) | |
| 64 | + if not row: | |
| 65 | + raise HTTPException(401, detail={"code": "UNAUTHORIZED", "message": "clé API invalide"}) | |
| 66 | + ident = Identity(key_id=row["id"], name=row["name"] or row["id"], quota_per_day=row["quota_per_day"]) | |
| 67 | + elif s.require_auth: | |
| 68 | + raise HTTPException( | |
| 69 | + 401, detail={"code": "UNAUTHORIZED", "message": "Authorization: Bearer <clé> requis"} | |
| 70 | + ) | |
| 71 | + else: | |
| 72 | + ident = Identity(key_id=None, name="anonymous", is_admin=not s.admin_key) | |
| 73 | + rl_key = ident.key_id or (request.client.host if request.client else "anon") | |
| 74 | + if not bucket().take(rl_key): | |
| 75 | + raise HTTPException( | |
| 76 | + 429, detail={"code": "RATE_LIMITED", "message": "trop de requêtes", "retryable": True} | |
| 77 | + ) | |
| 78 | + request.state.identity = ident | |
| 79 | + return ident | |
| 80 | + | |
| 81 | + | |
| 82 | +async def require_admin(ident: Identity = Depends(current_identity)) -> Identity: | |
| 83 | + if not ident.is_admin: | |
| 84 | + raise HTTPException( | |
| 85 | + 403, detail={"code": "FORBIDDEN", "message": "clé admin requise (TRAWLS_ADMIN_KEY)"} | |
| 86 | + ) | |
| 87 | + return ident | |
| 88 | + | |
| 89 | + | |
| 90 | +async def charge(ident: Identity, endpoint: str, credits: int = 1) -> None: | |
| 91 | + store = get_store() | |
| 92 | + if ident.quota_per_day is not None and ident.key_id: | |
| 93 | + used = await store.usage_today(ident.key_id) | |
| 94 | + if used + credits > ident.quota_per_day: | |
| 95 | + raise HTTPException( | |
| 96 | + 429, | |
| 97 | + detail={ | |
| 98 | + "code": "QUOTA_EXCEEDED", | |
| 99 | + "message": f"quota journalier {ident.quota_per_day} atteint", | |
| 100 | + "retryable": True, | |
| 101 | + }, | |
| 102 | + ) | |
| 103 | + await store.record_usage(ident.key_id, endpoint, credits) | |
added
trawls/api/main.py
+702 −0
@@ -0,0 +1,702 @@ | ||
| 1 | +"""API Trawls (FastAPI) — préfixe /v1. Erreurs au format { error: { code, message, retryable, details? } }. | |
| 2 | +Sert aussi l'UI web (/) et les blobs (/v1/blobs). Le worker de jobs tourne dans le même processus | |
| 3 | +(self-host first) ; `TRAWLS_EMBEDDED_WORKER=0` pour le désactiver et lancer `trawls worker` à part. | |
| 4 | +""" | |
| 5 | + | |
| 6 | +from __future__ import annotations | |
| 7 | + | |
| 8 | +import asyncio | |
| 9 | +import logging | |
| 10 | +import os | |
| 11 | +import time | |
| 12 | +from collections.abc import AsyncIterator | |
| 13 | +from contextlib import asynccontextmanager | |
| 14 | +from pathlib import Path | |
| 15 | +from typing import Any | |
| 16 | + | |
| 17 | +import orjson | |
| 18 | +import structlog | |
| 19 | +from fastapi import Depends, FastAPI, HTTPException, Query, Request | |
| 20 | +from fastapi.exceptions import RequestValidationError | |
| 21 | +from fastapi.responses import FileResponse, HTMLResponse, JSONResponse, RedirectResponse, Response | |
| 22 | +from fastapi.staticfiles import StaticFiles | |
| 23 | +from pydantic import BaseModel, Field | |
| 24 | +from sse_starlette.sse import EventSourceResponse | |
| 25 | + | |
| 26 | +from trawls import __version__ | |
| 27 | +from trawls.api.auth import Identity, charge, current_identity, require_admin | |
| 28 | +from trawls.config import get_settings | |
| 29 | +from trawls.core.fetcher.browser import shared_browser | |
| 30 | +from trawls.core.fetcher.http_fast import shared_http | |
| 31 | +from trawls.core.fetcher.strategy import host_modes | |
| 32 | +from trawls.core.scrape import breaker, scrape | |
| 33 | +from trawls.map import map_site | |
| 34 | +from trawls.models import ( | |
| 35 | + ApiErrorEnvelope, | |
| 36 | + CrawlOptions, | |
| 37 | + CssField, | |
| 38 | + JobSummary, | |
| 39 | + MapOptions, | |
| 40 | + MappedUrl, | |
| 41 | + PageResult, | |
| 42 | + ScrapeOptions, | |
| 43 | +) | |
| 44 | +from trawls.worker.crawl import Worker, bus, resolve_extract_urls, run_extract | |
| 45 | +from trawls.worker.store import get_store | |
| 46 | + | |
| 47 | +log = structlog.get_logger("trawls.api") | |
| 48 | +_STATIC = Path(__file__).resolve().parent.parent / "web" / "static" | |
| 49 | + | |
| 50 | + | |
| 51 | +def _setup_logging() -> None: | |
| 52 | + s = get_settings() | |
| 53 | + structlog.configure( | |
| 54 | + processors=[ | |
| 55 | + structlog.contextvars.merge_contextvars, | |
| 56 | + structlog.processors.add_log_level, | |
| 57 | + structlog.processors.TimeStamper(fmt="iso"), | |
| 58 | + structlog.processors.JSONRenderer( | |
| 59 | + serializer=lambda o, **kw: orjson.dumps(o, default=str).decode() | |
| 60 | + ), | |
| 61 | + ], | |
| 62 | + wrapper_class=structlog.make_filtering_bound_logger( | |
| 63 | + getattr(logging, s.log_level.upper(), logging.INFO) | |
| 64 | + ), | |
| 65 | + cache_logger_on_first_use=True, | |
| 66 | + ) | |
| 67 | + | |
| 68 | + | |
| 69 | +_worker: Worker | None = None | |
| 70 | +_worker_task: asyncio.Task[None] | None = None | |
| 71 | +_started_at = time.time() | |
| 72 | + | |
| 73 | + | |
| 74 | +@asynccontextmanager | |
| 75 | +async def lifespan(app: FastAPI) -> AsyncIterator[None]: | |
| 76 | + global _worker, _worker_task | |
| 77 | + _setup_logging() | |
| 78 | + s = get_settings() | |
| 79 | + store = get_store() | |
| 80 | + await store.open() | |
| 81 | + if s.allow_private_targets: | |
| 82 | + log.warning("security.allow_private_targets", value=True) | |
| 83 | + if os.environ.get("TRAWLS_EMBEDDED_WORKER", "1") != "0": | |
| 84 | + _worker = Worker(store) | |
| 85 | + _worker_task = asyncio.create_task(_worker.run()) | |
| 86 | + # préchauffage navigateur en arrière-plan (non bloquant) | |
| 87 | + asyncio.create_task(_warm_browser()) | |
| 88 | + log.info("api.started", version=__version__, port=s.port, public_url=s.public_url) | |
| 89 | + try: | |
| 90 | + yield | |
| 91 | + finally: | |
| 92 | + if _worker: | |
| 93 | + _worker.stop() | |
| 94 | + if _worker_task: | |
| 95 | + _worker_task.cancel() | |
| 96 | + await shared_browser().close() | |
| 97 | + await shared_http().close() | |
| 98 | + await store.close() | |
| 99 | + | |
| 100 | + | |
| 101 | +async def _warm_browser() -> None: | |
| 102 | + try: | |
| 103 | + await shared_browser().start() | |
| 104 | + except Exception as e: | |
| 105 | + log.warning("browser.warmup_failed", error=str(e)) | |
| 106 | + | |
| 107 | + | |
| 108 | +app = FastAPI( | |
| 109 | + title="Trawls API", | |
| 110 | + version=__version__, | |
| 111 | + description="Moteur de crawling, d'extraction et de navigation web, LLM-ready. Sortie Markdown propre par défaut.", | |
| 112 | + lifespan=lifespan, | |
| 113 | + docs_url=None, | |
| 114 | + redoc_url=None, | |
| 115 | + openapi_url="/v1/openapi.json", | |
| 116 | +) | |
| 117 | + | |
| 118 | + | |
| 119 | +# ---- erreurs ------------------------------------------------------------------- | |
| 120 | + | |
| 121 | + | |
| 122 | +def _err(status: int, code: str, message: str, retryable: bool = False, details: Any = None) -> JSONResponse: | |
| 123 | + return JSONResponse( | |
| 124 | + status_code=status, | |
| 125 | + content={ | |
| 126 | + "error": { | |
| 127 | + "code": code, | |
| 128 | + "message": message, | |
| 129 | + "retryable": retryable, | |
| 130 | + **({"details": details} if details else {}), | |
| 131 | + } | |
| 132 | + }, | |
| 133 | + ) | |
| 134 | + | |
| 135 | + | |
| 136 | +@app.exception_handler(HTTPException) | |
| 137 | +async def _http_exc(request: Request, exc: HTTPException) -> JSONResponse: | |
| 138 | + d = exc.detail if isinstance(exc.detail, dict) else {"code": "HTTP_ERROR", "message": str(exc.detail)} | |
| 139 | + return _err( | |
| 140 | + exc.status_code, | |
| 141 | + d.get("code", "HTTP_ERROR"), | |
| 142 | + d.get("message", ""), | |
| 143 | + bool(d.get("retryable")), | |
| 144 | + d.get("details"), | |
| 145 | + ) | |
| 146 | + | |
| 147 | + | |
| 148 | +@app.exception_handler(RequestValidationError) | |
| 149 | +async def _val_exc(request: Request, exc: RequestValidationError) -> JSONResponse: | |
| 150 | + return _err(422, "VALIDATION", "corps de requête invalide", details={"errors": exc.errors()[:10]}) | |
| 151 | + | |
| 152 | + | |
| 153 | +@app.exception_handler(Exception) | |
| 154 | +async def _any_exc(request: Request, exc: Exception) -> JSONResponse: | |
| 155 | + log.error("api.unhandled", path=request.url.path, error=f"{type(exc).__name__}: {exc}") | |
| 156 | + return _err(500, "INTERNAL", f"{type(exc).__name__}: {exc}", retryable=True) | |
| 157 | + | |
| 158 | + | |
| 159 | +@app.middleware("http") | |
| 160 | +async def _log_requests(request: Request, call_next: Any) -> Response: | |
| 161 | + t0 = time.perf_counter() | |
| 162 | + resp: Response = await call_next(request) | |
| 163 | + ms = (time.perf_counter() - t0) * 1000 | |
| 164 | + resp.headers["X-Trawls-Version"] = __version__ | |
| 165 | + resp.headers["Server-Timing"] = f"app;dur={ms:.0f}" | |
| 166 | + if request.url.path.startswith("/v1/") and not request.url.path.startswith(("/v1/blobs", "/v1/openapi")): | |
| 167 | + ident = getattr(request.state, "identity", None) | |
| 168 | + try: | |
| 169 | + await get_store().log_request( | |
| 170 | + ident.key_id if ident else None, | |
| 171 | + request.method, | |
| 172 | + request.url.path, | |
| 173 | + resp.status_code, | |
| 174 | + ms, | |
| 175 | + request.client.host if request.client else None, | |
| 176 | + ) | |
| 177 | + except Exception: | |
| 178 | + pass | |
| 179 | + return resp | |
| 180 | + | |
| 181 | + | |
| 182 | +# ---- santé ----------------------------------------------------------------------- | |
| 183 | + | |
| 184 | + | |
| 185 | +@app.get("/healthz", include_in_schema=False) | |
| 186 | +async def healthz() -> dict[str, Any]: | |
| 187 | + return {"ok": True, "version": __version__, "uptime_s": round(time.time() - _started_at)} | |
| 188 | + | |
| 189 | + | |
| 190 | +@app.get("/readyz", include_in_schema=False) | |
| 191 | +async def readyz() -> JSONResponse: | |
| 192 | + checks = { | |
| 193 | + "db": False, | |
| 194 | + "browser": shared_browser().ready, | |
| 195 | + "worker": bool(_worker_task and not _worker_task.done()), | |
| 196 | + } | |
| 197 | + try: | |
| 198 | + await get_store().stats() | |
| 199 | + checks["db"] = True | |
| 200 | + except Exception: | |
| 201 | + pass | |
| 202 | + ok = checks["db"] | |
| 203 | + return JSONResponse(status_code=200 if ok else 503, content={"ok": ok, "checks": checks}) | |
| 204 | + | |
| 205 | + | |
| 206 | +@app.get("/metrics", include_in_schema=False) | |
| 207 | +async def metrics() -> Response: | |
| 208 | + st = await get_store().stats() | |
| 209 | + lines = ["trawls_up 1", f'trawls_info{{version="{__version__}"}} 1'] | |
| 210 | + for k, v in st.get("jobs", {}).items(): | |
| 211 | + lines.append(f'trawls_jobs_total{{status="{k}"}} {v}') | |
| 212 | + for k, v in st.get("pages", {}).items(): | |
| 213 | + lines.append(f'trawls_pages_total{{status="{k}"}} {v}') | |
| 214 | + lines.append(f"trawls_worker_active_jobs {_worker.active if _worker else 0}") | |
| 215 | + lines.append(f"trawls_browser_ready {1 if shared_browser().ready else 0}") | |
| 216 | + for host, snap in breaker.snapshot().items(): | |
| 217 | + lines.append(f'trawls_breaker_open{{host="{host}"}} {1 if snap["open"] else 0}') | |
| 218 | + return Response("\n".join(lines) + "\n", media_type="text/plain; version=0.0.4") | |
| 219 | + | |
| 220 | + | |
| 221 | +# ---- modèles de requêtes ------------------------------------------------------------ | |
| 222 | + | |
| 223 | + | |
| 224 | +class ScrapeRequest(ScrapeOptions): | |
| 225 | + url: str | |
| 226 | + | |
| 227 | + | |
| 228 | +class CrawlRequest(BaseModel): | |
| 229 | + url: str | |
| 230 | + crawl: CrawlOptions = Field(default_factory=CrawlOptions) | |
| 231 | + scrape: ScrapeOptions = Field(default_factory=ScrapeOptions) | |
| 232 | + webhook: str | None = None | |
| 233 | + | |
| 234 | + | |
| 235 | +class MapRequest(MapOptions): | |
| 236 | + url: str | |
| 237 | + | |
| 238 | + | |
| 239 | +class ExtractRequest(BaseModel): | |
| 240 | + urls: list[str] = [] | |
| 241 | + pattern: str | None = None | |
| 242 | + mode: str = "css" | |
| 243 | + schema_: dict[str, Any] | None = Field(default=None, alias="schema") | |
| 244 | + css: dict[str, CssField] | None = None | |
| 245 | + prompt: str | None = None | |
| 246 | + merge_key: str | None = None | |
| 247 | + limit: int = 50 | |
| 248 | + scrape: ScrapeOptions = Field(default_factory=ScrapeOptions) | |
| 249 | + webhook: str | None = None | |
| 250 | + | |
| 251 | + model_config = {"populate_by_name": True} | |
| 252 | + | |
| 253 | + | |
| 254 | +class BatchRequest(ScrapeOptions): | |
| 255 | + urls: list[str] | |
| 256 | + concurrency: int = 8 | |
| 257 | + webhook: str | None = None | |
| 258 | + | |
| 259 | + | |
| 260 | +class JobCreated(BaseModel): | |
| 261 | + job_id: str | |
| 262 | + status: str | |
| 263 | + url: str | |
| 264 | + | |
| 265 | + | |
| 266 | +class CrawlStatus(JobSummary): | |
| 267 | + pages: list[PageResult] = [] | |
| 268 | + next_cursor: int | None = None | |
| 269 | + | |
| 270 | + | |
| 271 | +class MapResponse(BaseModel): | |
| 272 | + url: str | |
| 273 | + count: int | |
| 274 | + urls: list[MappedUrl] | |
| 275 | + took_ms: float | |
| 276 | + | |
| 277 | + | |
| 278 | +class KeyCreate(BaseModel): | |
| 279 | + name: str | |
| 280 | + quota_per_day: int | None = None | |
| 281 | + | |
| 282 | + | |
| 283 | +def _strip(page: PageResult, opts: ScrapeOptions) -> PageResult: | |
| 284 | + if "links" not in opts.formats: | |
| 285 | + page.links = [] | |
| 286 | + return page | |
| 287 | + | |
| 288 | + | |
| 289 | +# ---- routes v1 -------------------------------------------------------------------------- | |
| 290 | + | |
| 291 | +_R = {401: {"model": ApiErrorEnvelope}, 422: {"model": ApiErrorEnvelope}, 429: {"model": ApiErrorEnvelope}} | |
| 292 | + | |
| 293 | + | |
| 294 | +@app.post("/v1/scrape", response_model=PageResult, responses=_R, tags=["scrape"]) | |
| 295 | +async def scrape_route(req: ScrapeRequest, ident: Identity = Depends(current_identity)) -> PageResult: | |
| 296 | + """Scrape synchrone d'une page (≤ ~60 s). Ne lève jamais : `status=failed` + `error` typée.""" | |
| 297 | + opts = ScrapeOptions.model_validate(req.model_dump(exclude={"url"})) | |
| 298 | + if opts.extract and opts.extract.mode == "llm" and not opts.extract.schema_: | |
| 299 | + raise HTTPException( | |
| 300 | + 422, detail={"code": "VALIDATION", "message": "extract.schema requis en mode llm"} | |
| 301 | + ) | |
| 302 | + await charge(ident, "scrape") | |
| 303 | + try: | |
| 304 | + page = await asyncio.wait_for( | |
| 305 | + scrape(req.url, opts), timeout=max(5, min(opts.timeout_ms / 1000 * 3 + 20, 170)) | |
| 306 | + ) | |
| 307 | + except TimeoutError: | |
| 308 | + from trawls.models import ErrorCode, ErrorInfo | |
| 309 | + | |
| 310 | + page = PageResult.failed(req.url, ErrorInfo.make(ErrorCode.TIMEOUT_RENDER, "délai global dépassé")) | |
| 311 | + return _strip(page, opts) | |
| 312 | + | |
| 313 | + | |
| 314 | +@app.post("/v1/crawl", response_model=JobCreated, status_code=202, responses=_R, tags=["crawl"]) | |
| 315 | +async def crawl_create( | |
| 316 | + req: CrawlRequest, request: Request, ident: Identity = Depends(current_identity) | |
| 317 | +) -> JobCreated: | |
| 318 | + await charge(ident, "crawl", 0) | |
| 319 | + job = await get_store().create_job( | |
| 320 | + "crawl", | |
| 321 | + req.model_dump(mode="json", exclude={"webhook"}), | |
| 322 | + req.url, | |
| 323 | + ident.key_id, | |
| 324 | + request.headers.get("idempotency-key"), | |
| 325 | + req.webhook, | |
| 326 | + ) | |
| 327 | + return JobCreated( | |
| 328 | + job_id=job.id, status=job.status, url=f"{get_settings().public_url.rstrip('/')}/v1/crawl/{job.id}" | |
| 329 | + ) | |
| 330 | + | |
| 331 | + | |
| 332 | +@app.get("/v1/crawl/{job_id}", response_model=CrawlStatus, responses=_R, tags=["crawl"]) | |
| 333 | +async def crawl_status( | |
| 334 | + job_id: str, | |
| 335 | + cursor: int = 0, | |
| 336 | + limit: int = Query(50, le=500), | |
| 337 | + status: str | None = None, | |
| 338 | + ident: Identity = Depends(current_identity), | |
| 339 | +) -> CrawlStatus: | |
| 340 | + job = await get_store().get_job(job_id) | |
| 341 | + if not job: | |
| 342 | + raise HTTPException(404, detail={"code": "NOT_FOUND", "message": "job inconnu"}) | |
| 343 | + pages, nxt = await get_store().pages(job_id, after=cursor, limit=limit, status=status) | |
| 344 | + return CrawlStatus(**job.model_dump(), pages=pages, next_cursor=nxt) | |
| 345 | + | |
| 346 | + | |
| 347 | +@app.get("/v1/crawl/{job_id}/stream", tags=["crawl"]) | |
| 348 | +async def crawl_stream( | |
| 349 | + job_id: str, request: Request, ident: Identity = Depends(current_identity) | |
| 350 | +) -> EventSourceResponse: | |
| 351 | + """SSE : `status`, `page`, `progress`, `done`, `error`. Rejoue d'abord l'état courant.""" | |
| 352 | + return await _stream(job_id, request) | |
| 353 | + | |
| 354 | + | |
| 355 | +async def _stream(job_id: str, request: Request) -> EventSourceResponse: | |
| 356 | + store = get_store() | |
| 357 | + job = await store.get_job(job_id) | |
| 358 | + if not job: | |
| 359 | + raise HTTPException(404, detail={"code": "NOT_FOUND", "message": "job inconnu"}) | |
| 360 | + | |
| 361 | + async def gen() -> AsyncIterator[dict[str, Any]]: | |
| 362 | + q = bus.subscribe(job_id) | |
| 363 | + try: | |
| 364 | + j = await store.get_job(job_id) | |
| 365 | + assert j is not None | |
| 366 | + yield {"event": "status", "data": orjson.dumps(j.model_dump(mode="json"), default=str).decode()} | |
| 367 | + if j.status in ("completed", "failed", "cancelled"): | |
| 368 | + yield { | |
| 369 | + "event": "done", | |
| 370 | + "data": orjson.dumps({"status": j.status, **j.meta}, default=str).decode(), | |
| 371 | + } | |
| 372 | + return | |
| 373 | + while True: | |
| 374 | + if await request.is_disconnected(): | |
| 375 | + return | |
| 376 | + try: | |
| 377 | + ev = await asyncio.wait_for(q.get(), timeout=15) | |
| 378 | + except TimeoutError: | |
| 379 | + yield {"event": "ping", "data": "{}"} | |
| 380 | + continue | |
| 381 | + if ev is None: | |
| 382 | + return | |
| 383 | + yield {"event": ev["event"], "data": orjson.dumps(ev["data"], default=str).decode()} | |
| 384 | + if ev["event"] in ("done", "error"): | |
| 385 | + return | |
| 386 | + finally: | |
| 387 | + bus.unsubscribe(job_id, q) | |
| 388 | + | |
| 389 | + return EventSourceResponse(gen()) | |
| 390 | + | |
| 391 | + | |
| 392 | +@app.delete("/v1/crawl/{job_id}", tags=["crawl"]) | |
| 393 | +async def crawl_cancel(job_id: str, ident: Identity = Depends(current_identity)) -> dict[str, Any]: | |
| 394 | + return await _cancel(job_id) | |
| 395 | + | |
| 396 | + | |
| 397 | +async def _cancel(job_id: str) -> dict[str, Any]: | |
| 398 | + store = get_store() | |
| 399 | + job = await store.get_job(job_id) | |
| 400 | + if not job: | |
| 401 | + raise HTTPException(404, detail={"code": "NOT_FOUND", "message": "job inconnu"}) | |
| 402 | + if job.status in ("queued", "running", "paused"): | |
| 403 | + await store.set_status(job_id, "cancelled") | |
| 404 | + bus.publish(job_id, "done", {"status": "cancelled"}) | |
| 405 | + return {"job_id": job_id, "status": "cancelled"} | |
| 406 | + | |
| 407 | + | |
| 408 | +@app.post("/v1/map", response_model=MapResponse, responses=_R, tags=["map"]) | |
| 409 | +async def map_route(req: MapRequest, ident: Identity = Depends(current_identity)) -> MapResponse: | |
| 410 | + """Cartographie synchrone : sitemaps + crawl shallow, dédupliqué, trié (BM25 si `search`).""" | |
| 411 | + from trawls.core.security import SsrfError, check_url | |
| 412 | + | |
| 413 | + try: | |
| 414 | + await check_url(req.url) | |
| 415 | + except SsrfError as e: | |
| 416 | + raise HTTPException(400, detail={"code": e.code, "message": e.message}) from None | |
| 417 | + await charge(ident, "map") | |
| 418 | + t0 = time.perf_counter() | |
| 419 | + opts = MapOptions.model_validate(req.model_dump(exclude={"url"})) | |
| 420 | + urls = await map_site(req.url, opts) | |
| 421 | + return MapResponse( | |
| 422 | + url=req.url, count=len(urls), urls=urls, took_ms=round((time.perf_counter() - t0) * 1000, 1) | |
| 423 | + ) | |
| 424 | + | |
| 425 | + | |
| 426 | +@app.post("/v1/extract", responses=_R, tags=["extract"]) | |
| 427 | +async def extract_route( | |
| 428 | + req: ExtractRequest, request: Request, ident: Identity = Depends(current_identity) | |
| 429 | +) -> Any: | |
| 430 | + """Extraction structurée CSS ou LLM sur `urls[]` ou un `pattern`. Sync si ≤ 5 URLs, sinon job.""" | |
| 431 | + if req.mode == "llm" and not req.schema_: | |
| 432 | + raise HTTPException(422, detail={"code": "VALIDATION", "message": "schema requis en mode llm"}) | |
| 433 | + if req.mode == "css" and not req.css: | |
| 434 | + raise HTTPException(422, detail={"code": "VALIDATION", "message": "css requis en mode css"}) | |
| 435 | + body = req.model_dump(mode="json", by_alias=True, exclude={"webhook"}) | |
| 436 | + urls = await resolve_extract_urls(body) | |
| 437 | + if not urls: | |
| 438 | + raise HTTPException(422, detail={"code": "VALIDATION", "message": "aucune URL (urls[] ou pattern)"}) | |
| 439 | + store = get_store() | |
| 440 | + await charge(ident, "extract", len(urls)) | |
| 441 | + job = await store.create_job( | |
| 442 | + "extract", | |
| 443 | + body, | |
| 444 | + urls[0], | |
| 445 | + ident.key_id, | |
| 446 | + request.headers.get("idempotency-key"), | |
| 447 | + req.webhook, | |
| 448 | + total=len(urls), | |
| 449 | + ) | |
| 450 | + if len(urls) <= 5: | |
| 451 | + await store.set_status(job.id, "running") | |
| 452 | + # on retire le job de la file du worker (déjà pris en charge ici) | |
| 453 | + meta = await run_extract(job.id, body, store, urls) | |
| 454 | + j = await store.get_job(job.id) | |
| 455 | + return { | |
| 456 | + "job_id": job.id, | |
| 457 | + "status": j.status if j else "completed", | |
| 458 | + "data": meta.get("merged"), | |
| 459 | + "per_url": meta.get("per_url"), | |
| 460 | + } | |
| 461 | + return JSONResponse( | |
| 462 | + status_code=202, | |
| 463 | + content={ | |
| 464 | + "job_id": job.id, | |
| 465 | + "status": "queued", | |
| 466 | + "url": f"{get_settings().public_url.rstrip('/')}/v1/jobs/{job.id}", | |
| 467 | + }, | |
| 468 | + ) | |
| 469 | + | |
| 470 | + | |
| 471 | +@app.post("/v1/batch/scrape", response_model=JobCreated, status_code=202, responses=_R, tags=["batch"]) | |
| 472 | +async def batch_route( | |
| 473 | + req: BatchRequest, request: Request, ident: Identity = Depends(current_identity) | |
| 474 | +) -> JobCreated: | |
| 475 | + if not req.urls: | |
| 476 | + raise HTTPException(422, detail={"code": "VALIDATION", "message": "urls[] vide"}) | |
| 477 | + if len(req.urls) > 10_000: | |
| 478 | + raise HTTPException(422, detail={"code": "VALIDATION", "message": "max 10 000 URLs par batch"}) | |
| 479 | + await charge(ident, "batch", 0) | |
| 480 | + job = await get_store().create_job( | |
| 481 | + "batch", | |
| 482 | + req.model_dump(mode="json", exclude={"webhook"}), | |
| 483 | + req.urls[0], | |
| 484 | + ident.key_id, | |
| 485 | + request.headers.get("idempotency-key"), | |
| 486 | + req.webhook, | |
| 487 | + total=len(req.urls), | |
| 488 | + ) | |
| 489 | + return JobCreated( | |
| 490 | + job_id=job.id, status=job.status, url=f"{get_settings().public_url.rstrip('/')}/v1/jobs/{job.id}" | |
| 491 | + ) | |
| 492 | + | |
| 493 | + | |
| 494 | +@app.get("/v1/jobs", response_model=list[JobSummary], tags=["jobs"]) | |
| 495 | +async def jobs_list( | |
| 496 | + kind: str | None = None, | |
| 497 | + status: str | None = None, | |
| 498 | + limit: int = Query(50, le=500), | |
| 499 | + ident: Identity = Depends(current_identity), | |
| 500 | +) -> list[JobSummary]: | |
| 501 | + return await get_store().list_jobs( | |
| 502 | + limit=limit, | |
| 503 | + kind=kind, | |
| 504 | + status=status, | |
| 505 | + api_key_id=None if ident.is_admin or not ident.key_id else ident.key_id, | |
| 506 | + ) | |
| 507 | + | |
| 508 | + | |
| 509 | +@app.get("/v1/jobs/{job_id}", response_model=CrawlStatus, tags=["jobs"]) | |
| 510 | +async def job_get( | |
| 511 | + job_id: str, | |
| 512 | + cursor: int = 0, | |
| 513 | + limit: int = Query(50, le=500), | |
| 514 | + status: str | None = None, | |
| 515 | + ident: Identity = Depends(current_identity), | |
| 516 | +) -> CrawlStatus: | |
| 517 | + return await crawl_status(job_id, cursor, limit, status, ident) | |
| 518 | + | |
| 519 | + | |
| 520 | +@app.get("/v1/jobs/{job_id}/stream", tags=["jobs"]) | |
| 521 | +async def job_stream( | |
| 522 | + job_id: str, request: Request, ident: Identity = Depends(current_identity) | |
| 523 | +) -> EventSourceResponse: | |
| 524 | + return await _stream(job_id, request) | |
| 525 | + | |
| 526 | + | |
| 527 | +@app.delete("/v1/jobs/{job_id}", tags=["jobs"]) | |
| 528 | +async def job_cancel(job_id: str, ident: Identity = Depends(current_identity)) -> dict[str, Any]: | |
| 529 | + return await _cancel(job_id) | |
| 530 | + | |
| 531 | + | |
| 532 | +@app.get("/v1/jobs/{job_id}/export", tags=["jobs"]) | |
| 533 | +async def job_export( | |
| 534 | + job_id: str, | |
| 535 | + format: str = Query("jsonl", pattern="^(jsonl|zip|md)$"), | |
| 536 | + ident: Identity = Depends(current_identity), | |
| 537 | +) -> Response: | |
| 538 | + """Export JSONL (une page par ligne), ZIP (un .md par page) ou MD concaténé.""" | |
| 539 | + import io | |
| 540 | + import re | |
| 541 | + import zipfile | |
| 542 | + | |
| 543 | + store = get_store() | |
| 544 | + if not await store.get_job(job_id): | |
| 545 | + raise HTTPException(404, detail={"code": "NOT_FOUND", "message": "job inconnu"}) | |
| 546 | + pages: list[PageResult] = [] | |
| 547 | + cursor = 0 | |
| 548 | + while True: | |
| 549 | + batch, nxt = await store.pages(job_id, after=cursor, limit=500) | |
| 550 | + pages.extend(batch) | |
| 551 | + if nxt is None: | |
| 552 | + break | |
| 553 | + cursor = nxt | |
| 554 | + if format == "jsonl": | |
| 555 | + body = b"\n".join(orjson.dumps(p.model_dump(mode="json"), default=str) for p in pages) + b"\n" | |
| 556 | + return Response( | |
| 557 | + body, | |
| 558 | + media_type="application/x-ndjson", | |
| 559 | + headers={"Content-Disposition": f'attachment; filename="{job_id}.jsonl"'}, | |
| 560 | + ) | |
| 561 | + if format == "md": | |
| 562 | + body_s = "\n\n---\n\n".join( | |
| 563 | + f"<!-- {p.final_url} -->\n\n{p.markdown or ''}" for p in pages if p.status == "ok" | |
| 564 | + ) | |
| 565 | + return Response( | |
| 566 | + body_s, | |
| 567 | + media_type="text/markdown; charset=utf-8", | |
| 568 | + headers={"Content-Disposition": f'attachment; filename="{job_id}.md"'}, | |
| 569 | + ) | |
| 570 | + buf = io.BytesIO() | |
| 571 | + with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as z: | |
| 572 | + for i, p in enumerate(pages): | |
| 573 | + if p.status != "ok": | |
| 574 | + continue | |
| 575 | + name = re.sub(r"[^a-zA-Z0-9._-]+", "_", p.final_url.split("://", 1)[-1])[:120] or f"page_{i}" | |
| 576 | + z.writestr(f"{i:05d}_{name}.md", f"<!-- {p.final_url} -->\n\n{p.markdown or ''}") | |
| 577 | + z.writestr( | |
| 578 | + "_index.jsonl", | |
| 579 | + "\n".join( | |
| 580 | + orjson.dumps({"url": p.final_url, "status": p.status, "title": p.metadata.title}).decode() | |
| 581 | + for p in pages | |
| 582 | + ), | |
| 583 | + ) | |
| 584 | + return Response( | |
| 585 | + buf.getvalue(), | |
| 586 | + media_type="application/zip", | |
| 587 | + headers={"Content-Disposition": f'attachment; filename="{job_id}.zip"'}, | |
| 588 | + ) | |
| 589 | + | |
| 590 | + | |
| 591 | +@app.get("/v1/usage", tags=["usage"]) | |
| 592 | +async def usage_route( | |
| 593 | + days: int = Query(30, le=365), ident: Identity = Depends(current_identity) | |
| 594 | +) -> dict[str, Any]: | |
| 595 | + rows = await get_store().usage(None if ident.is_admin else ident.key_id, days) | |
| 596 | + by_day: dict[str, dict[str, int]] = {} | |
| 597 | + for r in rows: | |
| 598 | + d = by_day.setdefault(r["day"], {"requests": 0, "credits": 0}) | |
| 599 | + d["requests"] += int(r["count"] or 0) | |
| 600 | + d["credits"] += int(r["credits"] or 0) | |
| 601 | + return { | |
| 602 | + "identity": ident.name, | |
| 603 | + "days": [{"day": k, **v} for k, v in sorted(by_day.items())], | |
| 604 | + "by_endpoint": rows, | |
| 605 | + } | |
| 606 | + | |
| 607 | + | |
| 608 | +@app.get("/v1/keys", tags=["keys"]) | |
| 609 | +async def keys_list(ident: Identity = Depends(require_admin)) -> list[dict[str, Any]]: | |
| 610 | + return await get_store().list_keys() | |
| 611 | + | |
| 612 | + | |
| 613 | +@app.post("/v1/keys", status_code=201, tags=["keys"]) | |
| 614 | +async def keys_create(req: KeyCreate, ident: Identity = Depends(require_admin)) -> dict[str, Any]: | |
| 615 | + """Crée une clé ; la valeur brute n'est renvoyée qu'une seule fois.""" | |
| 616 | + kid, raw = await get_store().create_key(req.name, req.quota_per_day) | |
| 617 | + return {"id": kid, "name": req.name, "key": raw, "quota_per_day": req.quota_per_day} | |
| 618 | + | |
| 619 | + | |
| 620 | +@app.delete("/v1/keys/{key_id}", tags=["keys"]) | |
| 621 | +async def keys_revoke(key_id: str, ident: Identity = Depends(require_admin)) -> dict[str, Any]: | |
| 622 | + await get_store().revoke_key(key_id) | |
| 623 | + return {"id": key_id, "disabled": True} | |
| 624 | + | |
| 625 | + | |
| 626 | +@app.get("/v1/requests", include_in_schema=False) | |
| 627 | +async def requests_log( | |
| 628 | + limit: int = Query(100, le=1000), ident: Identity = Depends(require_admin) | |
| 629 | +) -> list[dict[str, Any]]: | |
| 630 | + return await get_store().recent_requests(limit) | |
| 631 | + | |
| 632 | + | |
| 633 | +@app.get("/v1/system", include_in_schema=False) | |
| 634 | +async def system_info(ident: Identity = Depends(current_identity)) -> dict[str, Any]: | |
| 635 | + s = get_settings() | |
| 636 | + from trawls.llm.client import shared_llm | |
| 637 | + | |
| 638 | + return { | |
| 639 | + "version": __version__, | |
| 640 | + "uptime_s": round(time.time() - _started_at), | |
| 641 | + "browser_ready": shared_browser().ready, | |
| 642 | + "worker_active": _worker.active if _worker else None, | |
| 643 | + "host_modes": host_modes.snapshot(), | |
| 644 | + "breakers": breaker.snapshot(), | |
| 645 | + "llm": {"provider": s.llm_provider, "model": s.llm_model, "configured": shared_llm().configured}, | |
| 646 | + "require_auth": s.require_auth, | |
| 647 | + "stats": await get_store().stats(), | |
| 648 | + } | |
| 649 | + | |
| 650 | + | |
| 651 | +@app.get("/v1/blobs/{name}", include_in_schema=False) | |
| 652 | +async def blob(name: str) -> FileResponse: | |
| 653 | + import re | |
| 654 | + | |
| 655 | + if not re.fullmatch(r"[a-f0-9]{24}\.(jpg|png|pdf|html)", name): | |
| 656 | + raise HTTPException(404, detail={"code": "NOT_FOUND", "message": "blob inconnu"}) | |
| 657 | + p = get_settings().blob_dir / name | |
| 658 | + if not p.exists(): | |
| 659 | + raise HTTPException(404, detail={"code": "NOT_FOUND", "message": "blob inconnu"}) | |
| 660 | + return FileResponse(p) | |
| 661 | + | |
| 662 | + | |
| 663 | +# ---- docs & UI ------------------------------------------------------------------------- | |
| 664 | + | |
| 665 | + | |
| 666 | +@app.get("/docs", include_in_schema=False) | |
| 667 | +async def docs() -> HTMLResponse: | |
| 668 | + return HTMLResponse( | |
| 669 | + """<!doctype html><html><head><meta charset="utf-8"><title>Trawls API — docs</title> | |
| 670 | +<link rel="icon" href="/static/favicon.svg"></head><body style="margin:0;background:#0b0c0e"> | |
| 671 | +<script id="api-reference" data-url="/v1/openapi.json" data-configuration='{"theme":"deepSpace","darkMode":true,"hideModels":false}'></script> | |
| 672 | +<script src="https://cdn.jsdelivr.net/npm/@scalar/api-reference"></script></body></html>""" | |
| 673 | + ) | |
| 674 | + | |
| 675 | + | |
| 676 | +@app.get("/v1/docs", include_in_schema=False) | |
| 677 | +async def docs_v1() -> RedirectResponse: | |
| 678 | + return RedirectResponse("/docs") | |
| 679 | + | |
| 680 | + | |
| 681 | +if _STATIC.exists(): | |
| 682 | + app.mount("/static", StaticFiles(directory=str(_STATIC)), name="static") | |
| 683 | + | |
| 684 | + @app.get("/", include_in_schema=False) | |
| 685 | + @app.get("/playground", include_in_schema=False) | |
| 686 | + @app.get("/crawls", include_in_schema=False) | |
| 687 | + @app.get("/crawls/{job_id}", include_in_schema=False) | |
| 688 | + @app.get("/map", include_in_schema=False) | |
| 689 | + @app.get("/extract", include_in_schema=False) | |
| 690 | + @app.get("/keys", include_in_schema=False) | |
| 691 | + @app.get("/agent", include_in_schema=False) | |
| 692 | + async def spa(job_id: str | None = None) -> FileResponse: | |
| 693 | + return FileResponse(_STATIC / "index.html") | |
| 694 | + | |
| 695 | + | |
| 696 | +def run() -> None: # pragma: no cover | |
| 697 | + import uvicorn | |
| 698 | + | |
| 699 | + s = get_settings() | |
| 700 | + uvicorn.run( | |
| 701 | + "trawls.api.main:app", host=s.host, port=s.port, log_level=s.log_level.lower(), access_log=False | |
| 702 | + ) | |
added
trawls/cli.py
+205 −0
Large diff (205 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/config.py
+79 −0
Large diff (79 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/antibot/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/antibot/blocklist.txt
+59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/antibot/detect.py
+108 −0
Large diff (108 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/antibot/identity.py
+74 −0
Large diff (74 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/antibot/stealth.py
+87 −0
Large diff (87 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/fetcher/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/fetcher/base.py
+53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/fetcher/browser.py
+317 −0
Large diff (317 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/fetcher/http_fast.py
+237 −0
Large diff (237 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/fetcher/strategy.py
+147 −0
Large diff (147 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/resilience/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/resilience/breaker.py
+63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/resilience/budget.py
+46 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/resilience/retry.py
+58 −0
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/scheduler/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/scheduler/dedup.py
+121 −0
Large diff (121 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/scheduler/politeness.py
+56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/scheduler/robots.py
+84 −0
Large diff (84 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/scrape.py
+287 −0
Large diff (287 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/core/security.py
+81 −0
Large diff (81 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/extract/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/extract/css.py
+94 −0
Large diff (94 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/extract/llm.py
+148 −0
Large diff (148 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/extract/merge.py
+54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/llm/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/llm/client.py
+192 −0
Large diff (192 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/map/__init__.py
+221 −0
Large diff (221 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/models/__init__.py
+302 −0
Large diff (302 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/chunker/__init__.py
+159 −0
Large diff (159 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/encoding.py
+58 −0
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/html_to_md/__init__.py
+60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/html_to_md/citations.py
+22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/html_to_md/clean.py
+176 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/html_to_md/convert.py
+385 −0
Large diff (385 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/html_to_md/readability.py
+90 −0
Large diff (90 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/html_to_md/tables.py
+108 −0
Large diff (108 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/links.py
+58 −0
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/pdf/__init__.py
+213 −0
Large diff (213 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/structured/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/processors/structured/metadata.py
+172 −0
Large diff (172 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/web/static/app.js
+322 −0
Large diff (322 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/web/static/favicon.svg
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/web/static/index.html
+245 −0
Large diff (245 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/web/static/og.svg
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/web/static/style.css
+125 −0
Large diff (125 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/worker/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/worker/crawl.py
+489 −0
Large diff (489 lines) is not loaded by default to keep the page fast.Load diff
added
trawls/worker/store.py
+389 −0
Large diff (390 lines) is not loaded by default to keep the page fast.Load diff