SPB Git

spb/tendril Public

Tendril — web ingestion platform (scrape/crawl/map/search) on macOS Apple Silicon: WebKit fidelity, authenticated pages, deterministic testable extraction. A self-hosted Firecrawl alternative.

web-scrapingcrawlerwebkitmacostypescriptfirecrawlswiftfastify
JavaScript 82.6% TypeScript 11.8% HTML 5.3%

feat: scaffold Tendril monorepo with Phase 1 foundations

Web ingestion platform (scrape/crawl/map/search) for macOS Apple Silicon,
following CLAUDE.md. Phase 1 (§24) delivered and fully tested (74 tests):

- shared: Result<T,E>, §15 error taxonomy (single source), types, Pino
  logger with redaction, URL normalization (§10.1)
- router: pure shouldEscalate escalation decision (§3.4)
- egress: SSRF validation with resolve-then-validate + IP pinning (§16.5)
- fetcher-http: Tier 0 undici client, Safari header order, manual redirects
  with per-hop SSRF re-validation, meta-refresh, gzip/br (§3)
- extract: deterministic pipeline (§8) — JSON-LD/OG/Twitter harvest,
  Readability + density fallback, Turndown rules (GFM tables, fenced code,
  figures, dl), link extraction
- api: Fastify POST /v1/scrape + /healthz, errors.ts HTTP mapping (§15)

TypeScript strict (noUncheckedIndexedAccess, exactOptionalPropertyTypes),
fetchers/extractors return Result, author header on every code file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 10 h ago (Aug 10, 2026)

Showing 60 changed files with +5,669 and −0

added .gitignore +14 −0
@@ -0,0 +1,14 @@
1 +node_modules/
2 +dist/
3 +*.tsbuildinfo
4 +coverage/
5 +.DS_Store
6 +.env
7 +.env.*
8 +!.env.example
9 +blobs/
10 +*.log
11 +.turbo/
12 +native/**/.build/
13 +
14 +.claude/settings.local.json
added CLAUDE.md +999 −0
@@ -0,0 +1,999 @@
1 +# CLAUDE.md — Tendril
2 +
3 +> Web ingestion platform (search / scrape / crawl / map) running on macOS Apple Silicon.
4 +> Built to beat Firecrawl on three axes: **stealth**, **access to authenticated pages**, and
5 +> **deterministic extraction quality**.
6 +
7 +This file is the project's persistent context. Read it fully before making any change.
8 +
9 +**Production host:** `m3u96a` (Mac, M3, 96 GB unified memory) — single node, self-hosted.
10 +**Public endpoint:** `https://www.ten-dril.com` via ngrok reserved domain.
11 +
12 +---
13 +
14 +## Table of contents
15 +
16 +| § | Section |
17 +|---|---|
18 +| 0 | Name & identity |
19 +| 1 | Product thesis |
20 +| 2 | Architecture — three-tier router |
21 +| 3 | Tier 0 spec — HTTP |
22 +| 4 | Tier 1 spec — WKWebView pool |
23 +| 5 | Tier 2 spec — real Safari |
24 +| 6 | Proxy & identity layer |
25 +| 7 | Session profiles |
26 +| 8 | Extraction engine (deterministic) |
27 +| 9 | Non-HTML content |
28 +| 10 | Crawl frontier |
29 +| 11 | Cache & storage |
30 +| 12 | Data model |
31 +| 13 | Queue & job lifecycle |
32 +| 14 | API contracts |
33 +| 15 | Error taxonomy |
34 +| 16 | Deployment — m3u96a + ngrok |
35 +| 17 | macOS prerequisites |
36 +| 18 | Observability |
37 +| 19 | Performance targets |
38 +| 20 | Testing |
39 +| 21 | Code conventions |
40 +| 22 | SDKs & docs |
41 +| 23 | Repository layout |
42 +| 24 | Roadmap |
43 +| 25 | Anti-patterns |
44 +| 26 | Legal & ethical |
45 +
46 +---
47 +
48 +## 0. Name & identity
49 +
50 +Codename: **Tendril** — the climbing shoot that latches on and grows.
51 +Domain: `ten-dril.com`.
52 +
53 +If the name changes, update: `package.json` names, Swift bundle IDs (`com.tendril.worker`), Redis key prefix (`tdr:*`), Postgres schema name, ngrok config, API key prefix (`tdr_live_` / `tdr_test_`), LaunchAgent labels, and the User-Agent string.
54 +
55 +---
56 +
57 +## 1. Product thesis
58 +
59 +Firecrawl runs in the cloud, on datacenter IPs, in headless Chromium. Tendril runs on a **residential machine**, in **WebKit** (Safari's actual engine).
60 +
61 +**There is no local LLM in this system.** Extraction is deterministic: parsers, selectors, and structural inference. This is a deliberate choice with real consequences, so be honest about them:
62 +
63 +- ✅ Predictable cost, predictable latency, reproducible output. The same page always yields the same result — you can write golden-file tests against extraction, which is impossible with a sampling model.
64 +- ✅ No 20 GB resident model, no 40 s cold start, no GPU contention with the WebView pool.
65 +- ❌ No zero-shot extraction from arbitrary prose. If a user wants "the CEO's name" from an unstructured about-page, deterministic parsing will not find it reliably.
66 +
67 +The gap in ❌ is covered by **BYOK** (§8.7): the user supplies their own LLM API key, Tendril passes the cleaned markdown through and returns the result. Tendril never pays for inference and never stores the key beyond the request.
68 +
69 +So the three real differentiators are:
70 +
71 +1. **Browser fidelity.** WebKit on macOS ARM behind a residential IP is not a headless Chromium signature. Sites that block Firecrawl outright often do not block Tendril at all.
72 +2. **Authenticated pages.** Tier 2 drives a real Safari profile with real logged-in sessions. No cloud service can offer this.
73 +3. **Extraction you can test.** Deterministic output, versioned rules, golden files. Enterprise buyers care about reproducibility more than about magic.
74 +
75 +**Accepted constraint:** one machine, one IP, no native horizontal scaling.
76 +Tendril is not built to crawl 10M pages/day. It is built to crawl 100k pages/day *that nobody else can reach*, with output you can regression-test.
77 +
78 +---
79 +
80 +## 2. Architecture — three-tier router
81 +
82 +The core of the system is an **escalation router**. Always start at the cheapest tier; escalate only on evidence of failure.
83 +
84 +```
85 + ┌──────────────┐
86 + request ──────────► │ Router │
87 + └──────┬───────┘
88 + │ shouldEscalate()
89 + ┌─────────────────────┼─────────────────────┐
90 + ▼ ▼ ▼
91 + ┌─────────┐ ┌───────────┐ ┌──────────────┐
92 + │ TIER 0 │ │ TIER 1 │ │ TIER 2 │
93 + │ undici │─fail────►│ WKWebView │─fail──►│ Safari │
94 + │ HTTP │ │ pool │ │ safaridriver │
95 + └────┬────┘ └─────┬─────┘ └──────┬───────┘
96 + │ │ │
97 + └─────────────────────┴─────────────────────┘
98 + │ raw HTML + metadata
99 + ▼
100 + ┌──────────────────┐
101 + │ Extraction engine │
102 + └──────────────────┘
103 +```
104 +
105 +| | Tier 0 | Tier 1 | Tier 2 |
106 +|---|---|---|---|
107 +| Engine | undici | WKWebView | Safari + safaridriver |
108 +| Latency p50 | 50 ms | 500 ms | 3-8 s |
109 +| Concurrency | 200 | 24 | **1** |
110 +| JS execution | no | yes | yes |
111 +| Cookies/session | per-request | per-profile | real user profile |
112 +| Share of traffic | 80% | 18% | 2% |
113 +| Cost per page | ~0 | ~15 MB·s RAM | a human-scale amount of time |
114 +
115 +**Escalation is one-way and capped.** A request escalates at most twice. Every escalation is recorded with its reason so you can see, per domain, which tier actually works — and then pin that domain in `domain_hints` so future requests skip straight to it. The pinning table is what makes the system fast over time; without it you pay the escalation cost forever.
116 +
117 +---
118 +
119 +## 3. Tier 0 spec — HTTP (`packages/fetcher-http`)
120 +
121 +### 3.1 Client configuration
122 +
123 +`undici.Agent` with:
124 +
125 +```ts
126 +{
127 + connections: 64,
128 + pipelining: 1, // pipelining breaks on many CDNs, leave at 1
129 + keepAliveTimeout: 30_000,
130 + keepAliveMaxTimeout: 120_000,
131 + connect: { timeout: 8_000, rejectUnauthorized: true },
132 + maxRedirections: 5, // handled manually, see 3.3
133 + bodyTimeout: 20_000,
134 + headersTimeout: 10_000,
135 +}
136 +```
137 +
138 +### 3.2 Header fidelity
139 +
140 +Header **order** is a fingerprint. Send them in Safari's order, not alphabetically, and not in the order a JS object happens to iterate:
141 +
142 +```
143 +Host
144 +Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
145 +Accept-Language en-US,en;q=0.9 (or match the target's likely locale)
146 +Accept-Encoding gzip, deflate, br
147 +User-Agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 ...
148 + + " Tendril/1.0 (+https://www.ten-dril.com/bot)"
149 +Connection keep-alive
150 +Sec-Fetch-Dest document
151 +Sec-Fetch-Mode navigate
152 +Sec-Fetch-Site none
153 +Sec-Fetch-User ?1
154 +Upgrade-Insecure-Requests 1
155 +```
156 +
157 +The bot suffix in the UA is non-negotiable (§26). Stealth here means *not looking broken*, not lying about who you are.
158 +
159 +TLS: Node's default cipher order differs from Safari's, which is a JA3-visible signal. If a target fingerprints TLS, that target belongs on Tier 1 anyway — do not try to fix it at Tier 0.
160 +
161 +### 3.3 Redirect handling
162 +
163 +Handle redirects manually rather than letting undici follow them, because you need to:
164 +
165 +- record the full chain in `metadata.redirects[]` (users need it for canonicalization);
166 +- re-run SSRF validation on **every** hop, not just the first (§16.5);
167 +- detect meta-refresh and JS redirects in the body, which undici cannot see;
168 +- stop on a redirect loop (same normalized URL twice) with `ERR_REDIRECT_LOOP`.
169 +
170 +### 3.4 Escalation decision
171 +
172 +`shouldEscalate(response): TierDecision` — a pure function, unit tested, no I/O.
173 +
174 +| Signal | Threshold | Weight |
175 +|---|---|---|
176 +| Status 403 / 429 / 503 | — | decisive |
177 +| Body < 2 KB with empty `#root`, `#__next`, `#app`, `[ng-version]` | — | decisive |
178 +| Body contains `challenge-platform`, `cf-browser-verification`, `_Incapsula_`, `px-captcha`, `datadome` | — | decisive |
179 +| Text/HTML ratio | < 0.05 | strong |
180 +| `<noscript>` containing "enable JavaScript" / "JavaScript is required" | — | strong |
181 +| Zero `<a href>` on a page over 10 KB | — | moderate |
182 +| `<title>` matching `/^(just a moment|attention required|access denied)/i` | — | decisive |
183 +| Content-Type is not HTML | — | route to §9, do not escalate |
184 +
185 +Decisive → escalate. Two or more strong/moderate → escalate. Return the reason string; it goes in `timings.escalationReason` and into the metrics label.
186 +
187 +---
188 +
189 +## 4. Tier 1 spec — WKWebView pool (`native/tendril-worker/`)
190 +
191 +**This is the heart of the system.** A Swift daemon exposing HTTP on `127.0.0.1:8787`.
192 +
193 +### 4.1 Why native Swift, not Playwright-WebKit
194 +
195 +Playwright ships a patched WebKit build with detectable artifacts: modified `navigator.webdriver` handling, injected bindings, altered timing. A native `WKWebView` is the *same binary* as Safari — same CFNetwork stack, same TLS, same font cache, same media codecs. That fidelity is the product.
196 +
197 +### 4.2 Process model
198 +
199 +- **One process, N WebViews.** The daemon **must** run as a real AppKit app (`NSApplication` with `.accessory` activation policy), not a plain CLI, or WebKit refuses to lay out and render.
200 +- Pool of **24** on m3u96a. Each WebView is 150-400 MB resident. Measure with `footprint` before raising it; the number is bounded by memory, not cores.
201 +- The pool is a Swift `actor`. Acquire/release with an async semaphore, FIFO waiters, 30 s acquisition timeout → `ERR_POOL_EXHAUSTED`.
202 +- **Recycle aggressively.** Destroy and recreate a WebView after 50 renders or 10 minutes of life, whichever comes first. WebKit leaks slowly under adversarial pages, and a recycled view has a clean JS heap.
203 +- Hard per-render watchdog: if a render exceeds `timeout + 5 s`, tear the WebView down rather than waiting. A hung page must never hold a slot.
204 +
205 +### 4.3 Isolation
206 +
207 +Each WebView gets its own `WKWebsiteDataStore`:
208 +
209 +- `.nonPersistent()` by default → disposable profile, no cookie bleed between customers. This is a **security boundary**, not an optimization.
210 +- Named persistent stores for `profileId` (§7).
211 +- Between renders on a non-persistent store, call `removeData(ofTypes:modifiedSince:)` with the full type set anyway. Belt and braces.
212 +
213 +### 4.4 Render-complete detection
214 +
215 +Never rely on `didFinish` alone — it fires at DOM ready, long before an SPA has content. Combine:
216 +
217 +1. `webView(_:didFinish:)` fires, **and**
218 +2. network idle: zero in-flight requests for 500 ms (track via a `WKURLSchemeHandler` shim or a `PerformanceObserver` injected at document start), **and**
219 +3. DOM quiet: a `MutationObserver` reports no mutations for 300 ms, **and**
220 +4. optional `waitFor`: a CSS selector present, or a fixed delay, or a JS predicate returning true.
221 +
222 +Cap the whole thing at `timeout`. On cap, **return what you have** with `metadata.renderTimedOut: true` rather than failing — a partially rendered page is usually still useful, and the caller can decide.
223 +
224 +Many sites poll forever (chat widgets, analytics beacons, websockets). Condition 2 must ignore requests to known-noisy hosts, or it never settles. Maintain a blocklist and apply it as a `WKContentRuleList` — which also speeds up every render substantially.
225 +
226 +### 4.5 Injected user script (`.atDocumentStart`, main frame + subframes)
227 +
228 +- Stub `Notification.requestPermission`, `navigator.geolocation`, `navigator.mediaDevices` so permission prompts never appear (a prompt blocks the run loop).
229 +- Neutralize `window.alert/confirm/prompt` — otherwise a modal deadlocks the WebView.
230 +- Install the `MutationObserver` and `PerformanceObserver` used by §4.4.
231 +- Capture `window.onerror` and `console.*` into an array the daemon reads back. Return it in `console[]`; it is the single most useful debugging field for "why is this page empty".
232 +- Do **not** try to spoof fingerprinting surfaces (canvas, WebGL, audio). WebKit's real values are the asset here. Spoofing creates inconsistencies that are *more* detectable than the truth.
233 +
234 +### 4.6 Daemon API
235 +
236 +```
237 +POST /render
238 + { url, waitFor?, timeout?, profileId?, actions?, blockResources?, viewport?, userAgent? }
239 + → { html, status, finalURL, redirects[], console[], timings, renderTimedOut }
240 +
241 +POST /screenshot
242 + { url, fullPage?, width?, height?, format?, quality? } → image/png | image/jpeg
243 +
244 +POST /pdf
245 + { url, paperSize? } → application/pdf
246 +
247 +GET /health
248 + → { pool: { total, busy, idle, recycled }, rss, uptime, rendersTotal }
249 +
250 +POST /profile { id, cookies[], userAgent? } → creates/updates a persistent data store
251 +DELETE /profile/:id → wipes it
252 +```
253 +
254 +`actions` mini-DSL, executed in order after initial load:
255 +
256 +```jsonc
257 +[
258 + { "click": "#accept-cookies" },
259 + { "wait": 1500 },
260 + { "type": { "selector": "#search", "text": "query" } },
261 + { "press": "Enter" },
262 + { "scroll": 2000 },
263 + { "scrollToBottom": { "maxScrolls": 10, "delay": 500 } },
264 + { "waitForSelector": ".results-loaded" },
265 + { "evaluate": "document.querySelector('.more').click()" }
266 +]
267 +```
268 +
269 +`scrollToBottom` is the workhorse for infinite-scroll pages; implement it as scroll → wait → compare `scrollHeight` → repeat until stable or `maxScrolls`.
270 +
271 +### 4.7 Transport between Node and the daemon
272 +
273 +HTTP over loopback with keep-alive. Not Unix sockets (harder to debug), not stdio (framing pain).
274 +Node client (`packages/fetcher-webkit`) must implement: 2 s connect timeout, single retry on `ECONNREFUSED` after a 1 s delay, circuit breaker that opens after 5 consecutive failures and routes everything to Tier 0 with a loud alert. **If the daemon is dead, the API must degrade rather than 500 across the board.**
275 +
276 +---
277 +
278 +## 5. Tier 2 spec — real Safari (`packages/fetcher-safari`)
279 +
280 +Last resort, for targets that defeat Tier 1 or require the real user's logged-in state.
281 +
282 +### 5.1 Two modes
283 +
284 +- **`safaridriver`** (W3C WebDriver) — preferred. Stable, scriptable, proper timeouts. **Hard limit: one session at a time on the machine.** Safari Technology Preview provides a second, independent session; that is the ceiling — two, total.
285 +- **AppleScript** via `osascript` — fallback. Serial, fragile, breaks on focus changes, but drives the *real* Safari with the *real* user profile and all its logged-in sessions.
286 +
287 +### 5.2 Operating rules
288 +
289 +- Its own BullMQ queue, `concurrency: 1`, hard 60 s timeout, max 200 jobs/day by default.
290 +- Never reachable via `tier: "auto"` for anonymous requests. Tier 2 requires either an explicit `tier: "safari"` or a `profileId` bound to the real profile.
291 +- Between jobs: close the tab, do **not** clear cookies (that is the point), but do clear the URL bar and history entry if the customer is not the profile owner.
292 +- Fails closed. If safaridriver reports a session conflict, queue the job rather than falling back to AppleScript mid-flight — mixing the two corrupts state.
293 +- Log every Tier 2 job with the requesting API key. This is the audit trail you will need if a target complains.
294 +
295 +---
296 +
297 +## 6. Proxy & identity layer (`packages/egress`)
298 +
299 +One machine means one IP means one ban ends everything. Design for this from day one.
300 +
301 +- **Tier 0 and Tier 1 egress through rotating residential proxies.** Tier 2 uses the host IP directly — that is precisely its value.
302 +- Proxy selection is **sticky per host** for the duration of a crawl job: switching IPs mid-crawl of one site looks far more suspicious than staying put.
303 +- Health-check the proxy pool every 60 s against a known-good endpoint. Eject on 3 consecutive failures, re-test after 10 min.
304 +- Track per-(proxy, host) success rate in Redis. Stop routing a proxy to a host once its rate drops below 50% over the last 20 attempts — it is burned there.
305 +- Geo-pin when the target is geo-sensitive: `egressRegion` option on the API, defaulting to the account's region.
306 +- WKWebView proxy configuration is set per-`WKWebsiteDataStore` via `proxyConfigurations` (macOS 14+). It cannot be changed on a live store, so the pool must hold **buckets of WebViews per proxy** and acquire from the matching bucket. Plan the pool shape accordingly: e.g. 4 proxies × 6 WebViews rather than 24 undifferentiated ones.
307 +
308 +---
309 +
310 +## 7. Session profiles (`packages/profiles`)
311 +
312 +A profile is a named, persistent browser identity: cookies, localStorage, and a pinned UA.
313 +
314 +- Created by `POST /v1/profiles` with either an explicit cookie array, or an interactive login flow (§7.2).
315 +- Stored as a `WKWebsiteDataStore` keyed by UUID on disk, plus a Postgres row holding metadata (owner, target domains, created/last-used, expiry).
316 +- **Cookies are secrets.** Encrypt the Postgres-side copy with a key from the macOS Keychain, never log them, redact them from error payloads, and exclude the profile directory from any backup that leaves the machine unencrypted.
317 +- Profiles expire: default 30 days idle, then wiped. Re-auth is the user's job.
318 +- A profile is bound to one account. Cross-account use is a hard error, not a warning.
319 +
320 +### 7.2 Interactive login flow
321 +
322 +For targets where cookie import is impractical: `POST /v1/profiles/:id/login` opens a visible WKWebView on `m3u96a`, the operator logs in by hand, the daemon detects navigation to a success URL and seals the profile. This is a manual, low-volume operation and that is fine — it is what makes Tier 2 valuable.
323 +
324 +---
325 +
326 +## 8. Extraction engine (`packages/extract`)
327 +
328 +Deterministic, versioned, testable. **No model inference anywhere in this package.**
329 +
330 +### 8.1 Pipeline
331 +
332 +```
333 +raw HTML
334 + → encoding detection (charset header → <meta> → BOM → heuristic)
335 + → parse with linkedom
336 + → sanitize: strip <script> <style> <svg> <noscript>, ad iframes, tracking pixels
337 + → structured-data harvest (§8.3) ─────────┐
338 + → boilerplate removal (§8.4) │
339 + → relative → absolute URL rewriting ├─► metadata
340 + → Turndown with custom rules (§8.5) │
341 + → post-process │
342 + → markdown ────────────────────────────────┘
343 +```
344 +
345 +Every stage is a pure function `(input, options) => output`. The whole pipeline is testable offline against saved HTML fixtures, which is the entire reason for choosing determinism.
346 +
347 +### 8.2 Boilerplate removal
348 +
349 +Two modes, selected by `onlyMainContent`:
350 +
351 +- **Readability** (`@mozilla/readability`) for article-shaped pages. Reliable on news, blogs, docs.
352 +- **Density-based fallback** when Readability returns under 200 characters — which it does on listing pages, product pages, and dashboards. Implement a simple text-density scorer: for each block element, `textLength / (1 + linkTextLength)`, pick the subtree maximizing the sum. Do not skip this fallback; Readability silently failing to a near-empty result is the single most common extraction bug.
353 +
354 +Always strip, in both modes: `<header>`, `<footer>`, `<nav>`, `[role=navigation]`, `[aria-hidden=true]`, cookie banners (match on common class patterns), newsletter modals, and elements whose class matches `/(^|[-_])(ad|ads|advert|sponsor|promo|share|social|related|comment)([-_]|$)/i`.
355 +
356 +### 8.3 Structured data harvest
357 +
358 +This is where deterministic extraction earns its keep. Harvest, in priority order, and merge:
359 +
360 +1. **JSON-LD** (`<script type="application/ld+json">`) — parse all blocks, handle `@graph`, resolve `@type` to a normalized shape. Covers Article, Product, Recipe, Event, JobPosting, BreadcrumbList, Organization, FAQPage.
361 +2. **Microdata** (`itemscope`/`itemprop`) and **RDFa** — older but still common on e-commerce.
362 +3. **OpenGraph** and **Twitter cards** — title, description, image, type, publish time.
363 +4. **Standard meta** — `description`, `author`, `keywords`, `canonical`, `hreflang` alternates.
364 +5. **Heuristics** — `<time datetime>` for dates, `<h1>` for title, first `<p>` over 100 chars for description.
365 +
366 +Output as `metadata` plus a normalized `structured` object. A Product page should yield price, currency, availability, SKU, and images **without any model involved**. Roughly 60% of commercially interesting pages carry usable structured data; this is the highest-leverage code in the repo.
367 +
368 +### 8.4 Selector-based extraction (`/v1/extract` core)
369 +
370 +Users supply a schema where each field maps to an extractor:
371 +
372 +```jsonc
373 +{
374 + "schema": {
375 + "title": { "selector": "h1", "type": "text" },
376 + "price": { "selector": ".price", "type": "number", "clean": "currency" },
377 + "images": { "selector": "img.product", "type": "attr", "attr": "src", "multiple": true },
378 + "specs": { "selector": "table.specs", "type": "table" },
379 + "sku": { "jsonld": "$.sku" },
380 + "body": { "selector": "#desc", "type": "markdown" }
381 + }
382 +}
383 +```
384 +
385 +Types: `text`, `html`, `markdown`, `number`, `date`, `url`, `attr`, `table`, `list`, `boolean` (presence).
386 +Cleaners: `currency`, `whitespace`, `trim`, `stripTags`, `parseDate` (with locale hint).
387 +Sources: `selector` (CSS), `xpath`, `jsonld` (JSONPath into harvested JSON-LD), `regex`, `meta`.
388 +
389 +Field resolution order: `jsonld` → `meta` → `selector` → `xpath` → `regex`. First non-empty wins. This lets one schema work across sites with different markup.
390 +
391 +### 8.5 Repeated-structure inference (`/v1/extract` on list pages)
392 +
393 +For listing pages, users should not have to write selectors. Implement auto-detection:
394 +
395 +1. Compute a structural signature for every element: tag path + class set, normalized.
396 +2. Find the deepest parent whose children contain **≥ 5 siblings sharing a signature**.
397 +3. That parent is the list container; the siblings are the records.
398 +4. Within one record, each leaf position becomes a candidate field; label it from `itemprop`, `class`, or a nearby `<dt>`/`<th>`.
399 +
400 +This handles search results, product grids, and job boards without configuration. Return the inferred selectors alongside the data so users can pin and refine them — inference that cannot be inspected and overridden is worse than useless.
401 +
402 +### 8.6 Markdown conversion rules
403 +
404 +Turndown with hand-written rules. Write these; the defaults are not good enough:
405 +
406 +| Input | Output |
407 +|---|---|
408 +| `<pre><code class="language-x">` | fenced block with language tag |
409 +| `<pre>` without language | fenced block, attempt language detection from content |
410 +| `<table>` | GFM table, pipes escaped, empty cells preserved, `<th>` → header row |
411 +| nested/spanning tables | HTML passthrough (GFM cannot express them) |
412 +| `<figure>` + `<figcaption>` | image followed by italic caption |
413 +| `<dl>/<dt>/<dd>` | bold term + indented definition |
414 +| heading anchors (`#`, `¶`, `.anchor`) | removed |
415 +| `<img>` with `data-src`/`srcset` only | resolve to the highest-resolution real URL |
416 +| `<br>` inside a table cell | `<br>` kept (newline breaks the table) |
417 +| inline `<svg>` | removed, unless it has `<title>` → alt-text image reference |
418 +| `<abbr title>` | text + parenthetical |
419 +| MathML / `.katex` | `$...$` / `$$...$$` |
420 +
421 +Post-process: collapse 3+ blank lines to 2, trim trailing whitespace, normalize list markers to `-`, deduplicate consecutive identical links.
422 +
423 +### 8.7 BYOK LLM pass-through (optional, `packages/byok`)
424 +
425 +Thin and stateless. `/v1/extract` accepts:
426 +
427 +```jsonc
428 +{ "llm": { "provider": "anthropic|openai|...", "apiKey": "sk-...", "model": "...", "prompt": "..." } }
429 +```
430 +
431 +Tendril runs its normal deterministic pipeline, then forwards the resulting markdown plus the user's JSON schema to their provider with their key, and returns the response.
432 +
433 +Rules: the key is held in memory for the request only, never written to disk, never logged, never included in error payloads; chunk markdown over ~50k characters by heading and merge results; on provider error, return the deterministic result with `llm: { error }` rather than failing the whole request. Tendril is a pass-through, not an LLM vendor — do not build a model router, do not cache completions, do not offer a Tendril-supplied key.
434 +
435 +### 8.8 Versioning
436 +
437 +The extraction pipeline has a `pipelineVersion` (semver) stored with every cached result. Bumping it invalidates derived markdown but **not** the raw HTML cache, so you can re-extract the entire corpus after a rules improvement. This is the single most valuable property of keeping raw HTML forever.
438 +
439 +---
440 +
441 +## 9. Non-HTML content (`packages/extract/formats`)
442 +
443 +Content-Type routing, before any HTML logic:
444 +
445 +| Type | Handling |
446 +|---|---|
447 +| `application/pdf` | text layer extraction; if under 100 chars, flag `needsOCR` and skip (do not silently return empty) |
448 +| `application/json` | pretty-print; if it looks like an API response, return as `json` directly |
449 +| `text/plain`, `text/markdown` | pass through |
450 +| `text/csv` | parse to a GFM table, cap at 500 rows |
451 +| `application/xml`, RSS, Atom | parse feeds into items; this makes `/map` much stronger on blogs |
452 +| `image/*` | metadata only, plus the blob; no OCR |
453 +| `application/zip`, archives | reject with `ERR_UNSUPPORTED_TYPE` |
454 +| anything > `maxSizeBytes` (default 20 MB) | reject before download completes, via streaming size check |
455 +
456 +Never let a Content-Type mismatch reach the HTML parser. A 200 MB video streamed into linkedom will take the process down.
457 +
458 +---
459 +
460 +## 10. Crawl frontier (`packages/frontier`)
461 +
462 +Non-negotiable rules:
463 +
464 +1. **Normalize before anything.** Lowercase host, strip default port, sort query params, remove `utm_*`/`fbclid`/`gclid`/`ref`/`mc_cid`/`_ga`, drop fragment (unless `#!` hashbang), resolve `..`, consistent trailing slash, punycode hosts. Dedup runs on the normalized form. Keep the original for output.
465 +2. **Two-stage dedup.** In-memory Bloom filter (1% FP, sized `limit × 3`) → Postgres confirmation on hit. Avoids a DB round-trip per discovered link, which at 8 concurrent renders is thousands per minute.
466 +3. **Per-host rate limiting**, never global. Token bucket in Redis, configurable `delayMs`, honor robots.txt `Crawl-delay` when higher. A crawl of one site must not slow another customer's job.
467 +4. **Priority.** Strict BFS by depth, then heuristics: `/blog/`, `/docs/`, `/article/`, `/product/` ahead of `/tag/`, `/author/`, `/page/47`, `?sort=`, `?filter=`.
468 +5. **Trap detection.** If more than 200 URLs on one host share a path pattern with only a varying numeric or date segment, sample instead of exhausting. Infinite calendars, paginated tag archives, and faceted-search URL explosions are the three classics; each can generate unbounded URLs.
469 +6. **Backoff.** 429 or 503 → pause the entire host, exponential 2^n s, capped at 5 min, 3 attempts, then mark the host degraded for the job.
470 +7. **Budgets.** Page count *and* wall-clock *and* total bytes. First limit reached ends the job cleanly with `stopReason`.
471 +8. **Canonical awareness.** If a page declares `<link rel=canonical>` pointing elsewhere on the same host, record the mapping and do not crawl both. Saves 20-40% on most e-commerce sites.
472 +9. **Sitemap seeding.** Always attempt sitemap discovery first, even for a crawl — starting from a complete URL list beats link-following, and it lets you report accurate progress instead of an unknown denominator.
473 +10. **Resumability.** Frontier state lives in Postgres, not memory. A restart mid-crawl resumes; it does not start over.
474 +
475 +### 10.1 robots.txt
476 +
477 +Full RFC 9309 parsing: `User-agent` groups with correct specificity matching, `Allow`/`Disallow` with longest-match-wins, wildcards `*` and `$`, `Sitemap` directives, `Crawl-delay` (non-standard but honored). Cache per host for 24 h. On fetch failure treat as allow-all, but log it; on 5xx treat as disallow-all for 1 h, per the spec's intent.
478 +
479 +---
480 +
481 +## 11. Cache & storage (`packages/cache`)
482 +
483 +- **Content-addressed blob store.** `blobs/<sha256[0:2]>/<sha256[2:4]>/<sha256>`. Raw HTML, screenshots, PDFs. Deduplicates automatically across customers — the same page fetched by two accounts is stored once.
484 +- **Metadata in Postgres**, pointing at blob hashes. Never store blobs in Postgres.
485 +- **Cache key** = `sha256(normalizedURL + tier + profileId + relevantOptions)`. Options that do not change the fetch (output format, extraction schema) must **not** be in the key, or the hit rate collapses.
486 +- `maxAge` on the request decides whether a hit is served. Default 0 (always fetch) for `/scrape`, 3600 for `/crawl` sub-pages.
487 +- Respect `ETag` and `Last-Modified`: on a stale hit, revalidate with a conditional request. A 304 costs almost nothing and refreshes the entry.
488 +- **Eviction:** LRU by last access, triggered when the blob store exceeds a configured ceiling (default 200 GB). Run it as a scheduled job, not inline.
489 +- GDPR: cache entries carry a retention date; a purge job runs nightly (§26).
490 +
491 +---
492 +
493 +## 12. Data model
494 +
495 +Postgres 16, schema `tendril`, migrations via drizzle. Core tables:
496 +
497 +```
498 +accounts id, name, plan, created_at, egress_region
499 +api_keys id, account_id, hash (argon2id), prefix, scopes[], last_used_at, revoked_at
500 +jobs id, account_id, type(scrape|crawl|map|search|extract), status, params jsonb,
501 + created_at, started_at, finished_at, stop_reason, error_code
502 +pages id, job_id, url, normalized_url, canonical_url, depth, status_code,
503 + tier_used, escalation_reason, html_hash, markdown_hash, screenshot_hash,
504 + metadata jsonb, structured jsonb, pipeline_version, fetched_at, duration_ms
505 +frontier job_id, normalized_url, depth, priority, state(pending|active|done|failed),
506 + attempts, next_attempt_at -- PK (job_id, normalized_url)
507 +domain_hints host, preferred_tier, requires_profile, crawl_delay_ms, success_rate,
508 + last_updated -- the learned routing table
509 +profiles id, account_id, name, domains[], data_store_path, cookies_enc,
510 + created_at, last_used_at, expires_at
511 +robots_cache host, body, fetched_at, expires_at
512 +usage account_id, day, pages_by_tier jsonb, bytes_out, llm_passthrough_calls
513 +webhooks id, job_id, url, events[], secret, deliveries jsonb
514 +```
515 +
516 +Indexes that matter: `frontier(job_id, state, priority)` for the dequeue path, `pages(job_id, fetched_at)` for pagination, `pages(html_hash)` for dedup lookups, partial index on `frontier(next_attempt_at) WHERE state='failed'`.
517 +
518 +---
519 +
520 +## 13. Queue & job lifecycle
521 +
522 +BullMQ on Redis. Queues:
523 +
524 +| Queue | Concurrency | Notes |
525 +|---|---|---|
526 +| `fetch:http` | 32 | Tier 0 |
527 +| `fetch:webkit` | 24 | matches the pool size exactly |
528 +| `fetch:safari` | **1** | never raise this |
529 +| `crawl:control` | 8 | one job per active crawl, manages its frontier |
530 +| `extract` | 16 | CPU-bound, pure |
531 +| `webhook` | 8 | with retries |
532 +| `maintenance` | 1 | cache eviction, purges, robots refresh |
533 +
534 +Job lifecycle: `queued → running → (succeeded | failed | cancelled)`.
535 +Retries: 3 attempts, exponential backoff with jitter, only for transient error codes (§15). Never retry `ERR_ROBOTS_DENIED`, `ERR_SSRF_BLOCKED`, `ERR_UNSUPPORTED_TYPE`, or any 4xx other than 408/429.
536 +
537 +**Idempotency:** `/scrape`, `/crawl`, `/extract` accept an `Idempotency-Key` header. Store key → jobId for 24 h; a repeat returns the original job rather than starting a new one. Customers with retry logic will otherwise double-bill themselves and blame you.
538 +
539 +Cancellation is cooperative: set a flag in Redis, workers check it between pages, in-flight renders complete. Never kill a WebView to cancel — it corrupts the pool.
540 +
541 +---
542 +
543 +## 14. API contracts
544 +
545 +Base: `https://www.ten-dril.com/v1`. Auth: `Authorization: Bearer tdr_live_...`.
546 +All responses: `{ success: boolean, data?, error?: { code, message, details? } }`.
547 +All list responses are cursor-paginated: `{ data: [...], next?: "cursor" }`. No offset pagination.
548 +
549 +### 14.1 `POST /v1/scrape`
550 +
551 +```jsonc
552 +{
553 + "url": "https://example.com/article",
554 + "formats": ["markdown", "html", "links", "screenshot", "structured", "extract"],
555 + "extractSchema": { /* §8.4 */ },
556 + "llm": { /* §8.7, optional */ },
557 + "tier": "auto", // auto | http | webkit | safari
558 + "profileId": "uuid", // forces tier >= 1
559 + "actions": [ /* §4.6 */ ],
560 + "onlyMainContent": true,
561 + "includeTags": ["article", "main"],
562 + "excludeTags": [".sidebar", "#comments"],
563 + "waitFor": 0,
564 + "timeout": 30000,
565 + "maxAge": 0,
566 + "headers": { /* custom, merged over defaults */ },
567 + "egressRegion": "us",
568 + "blockResources": ["image", "media", "font"]
569 +}
570 +```
571 +
572 +Response `data`: `{ markdown, html, rawHtml, links[], structured, extract, screenshot, metadata, tierUsed, cached, timings }`.
573 +
574 +`timings`: `{ total, dns, connect, ttfb, download, render, extract, escalations[] }`. Expose this — customers debugging slow crawls will otherwise blame you for their target's latency.
575 +
576 +`links[]` is `{ url, text, rel, isInternal }`, deduplicated, absolute.
577 +
578 +### 14.2 `POST /v1/crawl`
579 +
580 +```jsonc
581 +{
582 + "url": "https://example.com",
583 + "limit": 1000,
584 + "maxDepth": 4,
585 + "maxDurationSeconds": 3600,
586 + "maxBytes": 5368709120,
587 + "includePaths": ["^/blog/.*"],
588 + "excludePaths": ["\\.pdf$", "^/tag/"],
589 + "allowSubdomains": false,
590 + "allowBackwardLinks": false,
591 + "allowExternalLinks": false,
592 + "concurrency": 8,
593 + "delayMs": 250,
594 + "respectRobots": true,
595 + "deduplicateSimilar": true,
596 + "scrapeOptions": { /* §14.1 minus url */ },
597 + "webhook": { "url": "...", "events": ["page","completed","failed"], "secret": "..." }
598 +}
599 +```
600 +
601 +→ `202 { jobId, statusUrl, streamUrl }`.
602 +
603 +- `GET /v1/crawl/:id` → `{ status, total, completed, failed, creditsUsed, data[], next }`
604 +- `GET /v1/crawl/:id/stream` → SSE, one event per page plus a terminal event. Heartbeat every 15 s or ngrok will drop the connection.
605 +- `DELETE /v1/crawl/:id` → cooperative cancel, returns partial results.
606 +- `GET /v1/crawl/:id/errors` → per-URL failures with codes. Customers need this and Firecrawl handles it poorly.
607 +
608 +Webhook payloads are signed: `X-Tendril-Signature: sha256=<hmac(secret, body)>`, plus `X-Tendril-Timestamp` for replay protection. Retry 5 times with exponential backoff, then mark the webhook degraded.
609 +
610 +### 14.3 `POST /v1/map`
611 +
612 +URL discovery **without rendering**. Target: under 3 s for a normal site.
613 +Sources merged and deduplicated: `sitemap.xml` (recursive through sitemap indexes, `.gz` support, cap at 50k URLs), `robots.txt` Sitemap directives, homepage links, `/llms.txt`, RSS/Atom feeds.
614 +Options: `search` (substring filter, ranked by match position), `limit`, `includeSubdomains`, `sitemapOnly`, `ignoreSitemap`.
615 +Returns `{ links: [{ url, title?, lastModified?, source }] }` — `source` tells the user where each URL came from, which builds trust in the result.
616 +
617 +### 14.4 `POST /v1/search`
618 +
619 +```jsonc
620 +{ "query": "...", "limit": 10, "lang": "en", "country": "us", "timeRange": "month",
621 + "scrapeResults": false, "scrapeOptions": {} }
622 +```
623 +
624 +Pipeline: SearXNG (self-hosted, §16.1) → dedup by domain+normalized title → rank → optionally scrape the top N in parallel (Tier 0 forced, 8 s timeout, partial failures tolerated and reported per-result).
625 +
626 +Be explicit in the docs: Tendril has no index of its own. Search quality is SearXNG's quality. Under-promise here.
627 +
628 +### 14.5 `POST /v1/extract`
629 +
630 +```jsonc
631 +{
632 + "urls": ["https://example.com/products/*"], // wildcards expand via /map
633 + "schema": { /* §8.4 */ },
634 + "inferList": true, // §8.5 auto-detection
635 + "llm": { /* §8.7, optional */ },
636 + "scrapeOptions": {}
637 +}
638 +```
639 +
640 +Returns one record per matched URL, or one aggregated array when `inferList` finds a list page. Include the resolved selectors in the response so users can pin them.
641 +
642 +### 14.6 Ancillary
643 +
644 +- `POST /v1/profiles`, `GET /v1/profiles`, `DELETE /v1/profiles/:id`, `POST /v1/profiles/:id/login`
645 +- `GET /v1/usage?from=&to=` — pages by tier, bytes, cost
646 +- `GET /healthz` — unauthenticated, shallow
647 +- `GET /v1/status` — authenticated, per-tier health (§18.3)
648 +
649 +Rate limits returned on every response: `X-RateLimit-Limit`, `-Remaining`, `-Reset`.
650 +
651 +---
652 +
653 +## 15. Error taxonomy
654 +
655 +One file maps codes to HTTP statuses and retry-ability. Never invent an ad-hoc string.
656 +
657 +| Code | HTTP | Retryable | Meaning |
658 +|---|---|---|---|
659 +| `ERR_INVALID_URL` | 400 | no | malformed or unsupported scheme |
660 +| `ERR_SSRF_BLOCKED` | 400 | no | resolved to a private or forbidden address |
661 +| `ERR_UNSUPPORTED_TYPE` | 415 | no | content type not handled |
662 +| `ERR_TOO_LARGE` | 413 | no | exceeded `maxSizeBytes` |
663 +| `ERR_UNAUTHORIZED` | 401 | no | bad or revoked key |
664 +| `ERR_QUOTA_EXCEEDED` | 402 | no | plan limit hit |
665 +| `ERR_RATE_LIMITED` | 429 | yes | our limit, not the target's |
666 +| `ERR_ROBOTS_DENIED` | 403 | no | robots.txt disallows |
667 +| `ERR_TARGET_BLOCKED` | 502 | maybe | target returned a challenge at every tier |
668 +| `ERR_TARGET_4XX` / `_5XX` | 502 | 5xx only | upstream status passed through |
669 +| `ERR_TIER_TIMEOUT` | 504 | yes | render exceeded timeout |
670 +| `ERR_POOL_EXHAUSTED` | 503 | yes | no WebView available in 30 s |
671 +| `ERR_DAEMON_DOWN` | 503 | yes | Swift daemon unreachable, circuit open |
672 +| `ERR_SAFARI_BUSY` | 503 | yes | Tier 2 session conflict |
673 +| `ERR_PROFILE_EXPIRED` | 409 | no | session profile needs re-auth |
674 +| `ERR_EXTRACT_FAILED` | 422 | no | schema produced no fields |
675 +| `ERR_REDIRECT_LOOP` | 502 | no | same URL twice in a chain |
676 +| `ERR_INTERNAL` | 500 | yes | a bug; page it |
677 +
678 +Every error response carries a `requestId` that appears in the logs. Support requests without one are unanswerable.
679 +
680 +---
681 +
682 +## 16. Deployment — node `m3u96a` + ngrok
683 +
684 +Single host. No Kubernetes, no cloud VM. Ingress is an ngrok tunnel on a reserved domain.
685 +
686 +### 16.1 Topology
687 +
688 +```
689 + Internet
690 + │
691 + ▼
692 + ngrok edge ──── TLS terminated here, certificate managed by ngrok
693 + www.ten-dril.com
694 + │ (encrypted tunnel, outbound connection initiated by the Mac)
695 + ▼
696 + ┌──────────────────────── m3u96a ────────────────────────┐
697 + │ ngrok agent (LaunchAgent) │
698 + │ │ │
699 + │ ▼ │
700 + │ Caddy :8080 ── reverse proxy, routing, access logs │
701 + │ ├─► Fastify API :3000 │
702 + │ └─► /healthz │
703 + │ │
704 + │ Internal only, bound to 127.0.0.1, never tunneled: │
705 + │ tendril-worker (Swift) :8787 │
706 + │ Redis :6379 │
707 + │ PostgreSQL :5432 │
708 + │ SearXNG :8888 │
709 + │ Prometheus / Grafana :9090 / :3001 │
710 + └────────────────────────────────────────────────────────┘
711 +```
712 +
713 +Caddy sits between ngrok and the API deliberately: one place for routing, access logs, and local rate limiting, and only one port is ever tunneled.
714 +
715 +### 16.2 ngrok configuration
716 +
717 +`deploy/ngrok.yml`:
718 +
719 +```yaml
720 +version: "3"
721 +agent:
722 + authtoken: ${NGROK_AUTHTOKEN}
723 + log: /usr/local/var/log/ngrok.log
724 + log_level: info
725 + connect_timeout: 10s
726 +endpoints:
727 + - name: tendril-api
728 + url: https://www.ten-dril.com
729 + upstream:
730 + url: 8080
731 + traffic_policy:
732 + inbound:
733 + - actions:
734 + - type: rate-limit
735 + config:
736 + name: global
737 + algorithm: sliding_window
738 + capacity: 600
739 + rate: 60s
740 + - expressions:
741 + - "req.url.path.startsWith('/internal')"
742 + actions:
743 + - type: deny
744 + config:
745 + status_code: 404
746 +```
747 +
748 +DNS: `CNAME` for `www.ten-dril.com` → the target ngrok returns when you reserve the domain. Reserve the apex `ten-dril.com` too and 301 it to `www` **at the ngrok edge**, not in the application.
749 +
750 +A reserved domain requires a paid ngrok plan; free-tier URLs rotate on restart and are unusable for a customer-facing API. Confirm current plan tiers and Traffic Policy feature availability on ngrok's site before building against a specific capability — this changes.
751 +
752 +### 16.3 Process supervision
753 +
754 +LaunchAgents (not LaunchDaemons — WebKit requires a user GUI context), in `~/Library/LaunchAgents/`:
755 +
756 +| Label | Process | Notes |
757 +|---|---|---|
758 +| `com.tendril.caffeinate` | `caffeinate -dimsu` | first to start, mandatory |
759 +| `com.tendril.postgres` | postgres | |
760 +| `com.tendril.redis` | redis-server | |
761 +| `com.tendril.worker` | Swift daemon | needs GUI session, `KeepAlive: true` |
762 +| `com.tendril.searxng` | uvicorn/docker | |
763 +| `com.tendril.api` | `node apps/api/dist/main.js` | |
764 +| `com.tendril.caddy` | `caddy run` | |
765 +| `com.tendril.ngrok` | `ngrok start --all` | **last**, gated on health |
766 +
767 +Set `SoftResourceLimits: { NumberOfFiles: 65536 }` in every plist, or the pool saturates around 40 connections.
768 +
769 +Start ordering: `com.tendril.ngrok` runs a wrapper that polls `http://127.0.0.1:8080/healthz` until healthy (max 120 s) before starting the tunnel. Exposing an unhealthy API is worse than a short outage.
770 +
771 +### 16.4 Deploy procedure (`scripts/deploy.sh`)
772 +
773 +1. `git pull` on a deploy branch, never on a dirty tree
774 +2. `pnpm install --frozen-lockfile && pnpm build && swift build -c release`
775 +3. `pnpm db:migrate` — migrations must be backward-compatible for one version, so a rollback does not need a down-migration
776 +4. Drain: stop accepting new jobs, wait up to 60 s for in-flight ones
777 +5. `launchctl kickstart -k` each agent in dependency order
778 +6. `./scripts/health.sh` — all tiers, all services
779 +7. **Smoke test through the public URL**, not localhost: scrape a known page via `https://www.ten-dril.com/v1/scrape` and assert the markdown contains an expected string
780 +8. On failure: `git checkout` previous tag, repeat from 2. Keep the last 3 builds on disk.
781 +
782 +### 16.5 Security posture
783 +
784 +Exposing a home machine to the internet deserves more care than a cloud VM.
785 +
786 +- **Only port 8080 is tunneled.** 8787, 6379, 5432, 8888, 9090 bind to `127.0.0.1` explicitly. Verify with `lsof -iTCP -sTCP:LISTEN` after every deploy.
787 +- API keys hashed with argon2id, prefix stored in clear for identification, full key shown once at creation.
788 +- Quotas enforced in-app **and** a global rate limit at the ngrok edge, so a burst never reaches the Mac.
789 +- Grafana and any admin surface: IP allowlist in Traffic Policy, or a separate non-public tunnel.
790 +- **SSRF is the top risk of any scraping API.** Users control the target URL. Enforce: block RFC1918, loopback, link-local, `169.254.169.254`, IPv6 ULA and mapped-IPv4; allow only `http`/`https`; allow only ports 80/443/8080/8443. **Resolve DNS first and validate the resolved IP**, then connect to that IP with the Host header set — otherwise DNS rebinding walks straight into Redis and Postgres. Re-validate on every redirect hop.
791 +- Redact from logs: `Authorization`, `Cookie`, `Set-Cookie`, BYOK keys, webhook secrets, profile cookie blobs.
792 +- macOS firewall on, `Remote Login` restricted to key auth on a non-default port, FileVault enabled.
793 +
794 +### 16.6 Failure modes specific to this setup
795 +
796 +These are what will actually take the service down. Treat them as design constraints.
797 +
798 +- **The tunnel is a single point of failure.** Sleep, Wi-Fi loss, or reboot means total outage. External uptime monitor on `https://www.ten-dril.com/healthz` every 60 s, alerting to your phone. Nothing else in this document matters if you find out from a customer.
799 +- **Residential upload bandwidth is the real ceiling.** Screenshots and large HTML saturate a typical uplink long before CPU. Gzip everything (Caddy), cap screenshot dimensions, and serve large blobs via short-lived signed links rather than inline base64.
800 +- **Reboots break the GUI session.** Without auto-login, Tiers 1 and 2 die while Tier 0 keeps returning bot-blocked pages. That looks like a quality regression, not an outage — which is why health checks must probe each tier independently (§18.3).
801 +- **ngrok reconnects silently but not instantly.** Expect 5-30 s gaps on network flaps. SDKs must retry with backoff; document it.
802 +- **One IP, one ban surface.** See §6.
803 +- **Thermal throttling.** 24 WebViews rendering continuously will heat an M3 and reduce clocks. Monitor `powermetrics` and cap the pool lower if sustained throughput drops.
804 +- **Backups.** Postgres and the blob CAS live on one SSD. Nightly `pg_dump` plus a CAS rsync to an external encrypted volume, offsite weekly, **and a tested restore**. An untested backup is not a backup.
805 +
806 +---
807 +
808 +## 17. macOS prerequisites
809 +
810 +Scripted in `scripts/setup-mac.sh`, idempotent, but several steps require a human.
811 +
812 +- **A GUI session is mandatory.** WKWebView and Safari will not render without a window server session. SSH alone is insufficient. Enable auto-login; connect via Screen Sharing *after* the graphical session exists.
813 +- **`caffeinate -dimsu`** as a LaunchAgent. Sleep kills the pool.
814 +- **Screen lock and screen saver disabled.** A locked screen suspends rendering in some WebViews.
815 +- **Safari → Develop → "Allow JavaScript from Apple Events"** must be ticked by hand. Not scriptable.
816 +- **`safaridriver --enable`** once, with an admin password.
817 +- **Automation and Accessibility permissions:** the first `osascript` triggers a TCC dialog requiring a click. No clean bypass. Document it in the runbook.
818 +- **Full Disk Access** for the terminal and the daemon if touching the Safari profile directory.
819 +- Disable Spotlight indexing on the blob store (`mdutil -i off`) — it will otherwise index millions of files.
820 +- Increase `kern.maxfiles` and `kern.maxfilesperproc` via a `sysctl` plist.
821 +
822 +---
823 +
824 +## 18. Observability
825 +
826 +### 18.1 Logs
827 +
828 +Pino, JSON, to stdout and a rotated file. Every line carries `{ requestId, jobId, accountId, url, tier }` where applicable. No `console.log` anywhere, enforced by lint rule.
829 +
830 +### 18.2 Metrics (Prometheus, scraped locally)
831 +
832 +- `tendril_requests_total{endpoint,status}`
833 +- `tendril_fetch_duration_seconds{tier}` — histogram
834 +- `tendril_escalations_total{from,to,reason}` — **the most important metric in the system**
835 +- `tendril_pool_slots{state}` — gauge: busy/idle/recycling
836 +- `tendril_extraction_empty_total{reason}` — pages yielding under 200 chars; a rising line here means a silent quality regression
837 +- `tendril_proxy_success_rate{proxy,host}`
838 +- `tendril_tunnel_up` — 0/1 from the agent's local API
839 +- `tendril_queue_depth{queue}`, `tendril_job_duration_seconds{type}`
840 +
841 +### 18.3 Health checks
842 +
843 +`GET /v1/status` probes each tier independently against a known-stable control URL and returns per-tier latency and last success. `/healthz` stays shallow and fast for the uptime monitor.
844 +
845 +Alert on: tunnel down, daemon circuit open, escalation rate above 40% (a target changed its defenses), extraction-empty rate above 10%, queue depth growing for 10 minutes, disk above 85%.
846 +
847 +---
848 +
849 +## 19. Performance targets
850 +
851 +Measured on m3u96a, `pnpm bench` against 100 reference URLs. Regressions block a release.
852 +
853 +| Metric | Target |
854 +|---|---|
855 +| Tier 0 p50 / p95 | 60 ms / 400 ms |
856 +| Tier 1 p50 / p95 | 550 ms / 2.5 s |
857 +| Escalation rate | < 25% |
858 +| Extraction (HTML → markdown), 500 KB page | < 120 ms |
859 +| `/map` on a 5k-URL sitemap | < 3 s |
860 +| Sustained crawl throughput | 40 pages/s Tier 0, 12 pages/s Tier 1 |
861 +| Memory, full pool, steady state | < 14 GB |
862 +| Cold start to first successful render | < 45 s |
863 +
864 +---
865 +
866 +## 20. Testing
867 +
868 +- **Unit:** URL normalization, `shouldEscalate`, Turndown rules, robots parsing, SSRF validation, structural inference. Fast, hermetic, no network. These are the majority of the suite.
869 +- **Fixtures:** `test/fixtures/html/` with ~60 saved real pages — React SPA, Next.js, WordPress, Shopify, static docs, a Cloudflare interstitial, a paginated listing, a page with nested tables, a page with JSON-LD, one with broken encoding. Extraction is tested only here.
870 +- **Golden files** for markdown output. Any rule change must show its diff and be reviewed.
871 +- **Contract tests** on the API schemas — the OpenAPI document is generated from Zod, so schema drift breaks the build.
872 +- **E2E** against 20 stable public sites, tagged `@slow`, excluded from CI. Asserts escalation rates and non-empty extraction, never exact content.
873 +- **Deploy smoke test** through the public URL (§16.4). Half of all outages are ingress, not application.
874 +- **Chaos:** kill the Swift daemon mid-crawl and assert the circuit breaker degrades to Tier 0 instead of failing the job. Do this in CI with a fake daemon.
875 +
876 +Coverage target: 85% on `packages/`, no target on `apps/`.
877 +
878 +---
879 +
880 +## 21. Code conventions
881 +
882 +- TypeScript strict, `noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`. No `any`; `unknown` plus narrowing.
883 +- **No exceptions for normal control flow.** Fetchers and extractors return `Result<T, TendrilError>`. Exceptions signal bugs and are caught only at the top-level handler.
884 +- Every error carries a code from §15. The code → HTTP mapping lives in exactly one file.
885 +- Pure functions wherever possible; I/O confined to `fetcher-*` and `cache`. This is what makes the fixture-based tests viable.
886 +- No comments restating code. Comment the *why*, especially for site-specific workarounds — and date them, they rot within months.
887 +- Swift: `async/await` only, no completion handlers. The pool is an `actor`. No force-unwraps outside tests.
888 +- Commits: conventional commits. A commit touching extraction rules must include the golden-file diff.
889 +
890 +---
891 +
892 +## 22. SDKs & docs
893 +
894 +- **TypeScript SDK** — generated types from the OpenAPI schema, hand-written ergonomics on top. Must handle: automatic retry with backoff (for tunnel flaps, §16.6), SSE streaming for crawls with reconnect, idempotency keys generated by default.
895 +- **Python SDK** — same surface, sync and async clients.
896 +- Docs site with runnable examples per endpoint, plus a **migration guide from Firecrawl** mapping their parameter names to yours. Compatibility of naming where it costs nothing is worth more than elegance.
897 +- Publish the OpenAPI document at `https://www.ten-dril.com/openapi.json`.
898 +
899 +---
900 +
901 +## 23. Repository layout
902 +
903 +```
904 +tendril/
905 +├── CLAUDE.md
906 +├── package.json # pnpm workspaces
907 +├── turbo.json
908 +├── apps/
909 +│ ├── api/ # Fastify — public surface
910 +│ │ ├── src/routes/{scrape,crawl,map,search,extract,profiles,usage}.ts
911 +│ │ ├── src/schemas/ # Zod → generated OpenAPI
912 +│ │ ├── src/auth/ # keys, quotas, rate limits
913 +│ │ └── src/errors.ts # §15 mapping, single source of truth
914 +│ └── worker/ # BullMQ consumers
915 +├── packages/
916 +│ ├── router/ # tier decision, escalation, domain_hints
917 +│ ├── fetcher-http/ # Tier 0
918 +│ ├── fetcher-webkit/ # Swift daemon client + circuit breaker
919 +│ ├── fetcher-safari/ # Tier 2
920 +│ ├── egress/ # proxy pool, SSRF validation
921 +│ ├── profiles/ # session management
922 +│ ├── extract/ # deterministic pipeline
923 +│ │ ├── src/structured/ # JSON-LD, microdata, OG
924 +│ │ ├── src/boilerplate/ # Readability + density fallback
925 +│ │ ├── src/markdown/ # Turndown rules
926 +│ │ ├── src/selectors/ # schema-driven extraction
927 +│ │ ├── src/infer/ # repeated-structure detection
928 +│ │ └── src/formats/ # PDF, CSV, XML, feeds
929 +│ ├── byok/ # optional LLM pass-through
930 +│ ├── frontier/ # crawl logic, robots, sitemaps
931 +│ ├── cache/ # CAS + metadata
932 +│ └── shared/ # types, errors, logger, Result
933 +├── native/
934 +│ └── tendril-worker/
935 +│ ├── Package.swift
936 +│ └── Sources/TendrilWorker/{main,Pool,Renderer,Server,Actions,Profiles}.swift
937 +├── deploy/
938 +│ ├── ngrok.yml
939 +│ ├── Caddyfile
940 +│ ├── launchagents/*.plist
941 +│ └── runbook.md
942 +├── sdks/{typescript,python}/
943 +├── test/fixtures/
944 +└── scripts/{setup-mac,deploy,health,backup}.sh
945 +```
946 +
947 +---
948 +
949 +## 24. Roadmap
950 +
951 +**Phase 1 — foundations (2 wks).** Tier 0, extraction pipeline (§8.1-8.3, 8.6), `/scrape`, CAS cache, error taxonomy. Must handle 80% of the fixture set. Ship nothing publicly yet.
952 +
953 +**Phase 2 — rendering (3 wks).** Swift daemon, pool, escalation router, `domain_hints`, `/screenshot`. Targets: escalation under 25%, Tier 1 p95 under 2.5 s.
954 +
955 +**Phase 3 — crawl (2 wks).** Frontier, BullMQ, `/crawl` with SSE and signed webhooks, `/map`, resumability.
956 +
957 +**Phase 4 — extraction depth (3 wks).** Selector schemas (§8.4), structure inference (§8.5), non-HTML formats (§9), `/extract`. **This is the phase that differentiates the product — do not cut it.** Optional BYOK at the end.
958 +
959 +**Phase 5 — stealth & identity (2 wks).** Proxy layer, Tier 2 Safari, session profiles, `/search` via SearXNG.
960 +
961 +**Phase 6 — production (2 wks).** Deploy on `m3u96a`, ngrok reserved domain, quotas, billing, observability, SDKs, docs, migration guide.
962 +
963 +Do not start Phase 4 before Phase 2's escalation rate target is met. Extraction quality on pages you cannot fetch is worth nothing.
964 +
965 +---
966 +
967 +## 25. Anti-patterns — do not do this
968 +
969 +- ❌ Driving Safari via AppleScript for volume. Serial, breaks on focus change. A last resort, not a foundation.
970 +- ❌ Spawning a `WKWebView` per request. Init costs ~800 ms. Pool or nothing.
971 +- ❌ Using jsdom. linkedom is 5-10× faster on this workload and the difference is the whole extraction budget.
972 +- ❌ Storing markdown without the source HTML. You will never re-extract after improving rules (§8.8).
973 +- ❌ Putting output-format options in the cache key. Hit rate collapses.
974 +- ❌ A global rate limit instead of per-host. You will get banned from one site while crawling another fast.
975 +- ❌ Trusting `networkidle` alone as a render signal. Many pages poll forever.
976 +- ❌ Letting Readability's empty result pass through silently. Always run the density fallback.
977 +- ❌ Ignoring robots.txt by default. Explicit opt-out, logged per client, never implicit.
978 +- ❌ Putting Tier 2 in the general queue. One stuck session freezes everything.
979 +- ❌ Validating the hostname instead of the resolved IP for SSRF. That is a rebinding hole straight into your database.
980 +- ❌ Tunneling any port other than 8080.
981 +- ❌ Assuming the tunnel is up. Verify after every deploy, monitor continuously.
982 +- ❌ Adding a local model "just for the hard cases". That decision reverses §1 and turns a testable system into a probabilistic one. If it becomes necessary, do it as an explicit, documented rewrite — not as a quiet addition.
983 +
984 +---
985 +
986 +## 26. Legal & ethical
987 +
988 +Not decorative. This is real product risk.
989 +
990 +- `respectRobots: true` is the default. Disabling requires an explicit flag, is logged per client, and is refused entirely for domains on an internal blocklist.
991 +- Honest rate limiting: minimum 100 ms `delayMs` per host, not overridable.
992 +- Identifiable User-Agent with a contact URL (`+https://www.ten-dril.com/bot`) and a published page explaining who you are and how to block you. Stealth means matching a real browser's fingerprint, not hiding your identity.
993 +- Scraping authenticated pages (Tier 2) puts the user on the hook for the target's ToS. State it plainly and require acceptance at signup.
994 +- **No CAPTCHA solving, no paywall bypass, no login-credential stuffing.** Commercial red lines as much as ethical ones.
995 +- Personal data collected incidentally has a retention period; the nightly purge job enforces it. Support deletion requests by URL and by domain.
996 +- Honor `noindex`/`nofollow` as a signal for crawl scope even though they are indexing directives — it costs little and is defensible.
997 +- Running a public service from a residential connection may conflict with your ISP's terms on operating servers. Check the contract before going live on `www.ten-dril.com`.
998 +
999 +I am not a lawyer. Have this framework reviewed by one before commercializing, particularly regarding the CFAA and state computer-crime statutes (US), the Database Directive and GDPR (EU), and the terms of the specific high-value targets you intend to support.
added README.md +73 −0
@@ -0,0 +1,73 @@
1 +# Tendril
2 +
3 +Web ingestion platform (search / scrape / crawl / map) for macOS Apple Silicon.
4 +See [`CLAUDE.md`](./CLAUDE.md) for the full product and architecture spec.
5 +
6 +> **Author:** Simon-Pierre Boucher — <contact@spboucher.ai>
7 +
8 +## Status — Phase 1 (foundations)
9 +
10 +Per the roadmap (`CLAUDE.md` §24), Phase 1 delivers Tier 0 fetching, the
11 +deterministic extraction pipeline, `/scrape`, and the error taxonomy. This is
12 +what currently exists and is tested:
13 +
14 +| Area | Spec | State |
15 +|---|---|---|
16 +| Error taxonomy (single source) | §15 | ✅ `packages/shared/src/errors.ts` |
17 +| `Result<T,E>` + typed errors | §21 | ✅ `packages/shared` |
18 +| URL normalization | §10 rule 1, §10.1 | ✅ `packages/shared/src/url.ts` |
19 +| Escalation decision `shouldEscalate` (pure) | §3.4 | ✅ `packages/router` |
20 +| SSRF validation (resolve-then-validate) | §16.5 | ✅ `packages/egress` |
21 +| Tier 0 HTTP fetch (header order, manual redirects, IP pinning) | §3 | ✅ `packages/fetcher-http` |
22 +| Extraction: structured-data harvest | §8.3 | ✅ `packages/extract` |
23 +| Extraction: boilerplate (Readability + density fallback) | §8.2 | ✅ |
24 +| Extraction: Turndown rules (tables, code, figures, dl…) | §8.6 | ✅ |
25 +| `POST /v1/scrape` + `/healthz` | §14.1 | ✅ `apps/api` |
26 +
27 +**Not yet built** (later phases): Tier 1 WKWebView daemon (§4), Tier 2 Safari
28 +(§5), proxy layer (§6), profiles (§7), selector/inference extraction (§8.4–8.5),
29 +non-HTML formats (§9), crawl frontier (§10), cache (§11), Postgres/Redis/BullMQ
30 +(§12–13), `/crawl` `/map` `/search` `/extract`, deployment (§16), SDKs (§22).
31 +
32 +Because Tiers 1–2 do not exist yet, a page that `shouldEscalate` flags under
33 +`tier: "auto"` **fails closed** with `ERR_TARGET_BLOCKED` and the decisive
34 +reason, rather than returning a challenge/interstitial page as if it were
35 +content. Force `tier: "http"` to extract Tier 0 output regardless.
36 +
37 +## Layout
38 +
39 +Monorepo (pnpm workspaces) following `CLAUDE.md` §23:
40 +
41 +```
42 +packages/shared Result, error taxonomy, types, logger, URL normalization
43 +packages/router shouldEscalate (§3.4)
44 +packages/egress SSRF validation + IP classification (§16.5)
45 +packages/fetcher-http Tier 0 undici client (§3)
46 +packages/extract deterministic pipeline (§8): structured, boilerplate, markdown
47 +apps/api Fastify public surface (§14)
48 +test/fixtures/html saved pages for offline extraction tests (§20)
49 +```
50 +
51 +## Develop
52 +
53 +```bash
54 +pnpm install
55 +pnpm test # 74 unit/contract tests, hermetic, no network
56 +pnpm build # tsc -b, strict, emits dist/
57 +pnpm dev:api # Fastify on 127.0.0.1:3000
58 +```
59 +
60 +Try it:
61 +
62 +```bash
63 +curl -s -X POST http://127.0.0.1:3000/v1/scrape \
64 + -H 'content-type: application/json' \
65 + -d '{"url":"https://example.com","formats":["markdown","links","structured"]}'
66 +```
67 +
68 +## Conventions
69 +
70 +- TypeScript strict, `noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`, no `any`.
71 +- Fetchers/extractors return `Result<T, TendrilError>`; exceptions only signal bugs (§21).
72 +- Every code file starts with an author header comment.
73 +- Extraction stages are pure functions tested offline against fixtures (§8, §20).
added apps/api/package.json +20 −0
@@ -0,0 +1,20 @@
1 +{
2 + "name": "@tendril/api",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "main": "./src/main.ts",
7 + "scripts": {
8 + "dev": "tsx watch src/main.ts",
9 + "start": "node dist/main.js"
10 + },
11 + "dependencies": {
12 + "@tendril/shared": "workspace:*",
13 + "@tendril/router": "workspace:*",
14 + "@tendril/egress": "workspace:*",
15 + "@tendril/fetcher-http": "workspace:*",
16 + "@tendril/extract": "workspace:*",
17 + "fastify": "^5.2.0",
18 + "zod": "^3.24.1"
19 + }
20 +}
added apps/api/src/app.test.ts +42 −0
@@ -0,0 +1,42 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { afterAll, beforeAll, describe, expect, it } from "vitest";
3 +import type { FastifyInstance } from "fastify";
4 +import { buildApp } from "./app.js";
5 +
6 +let app: FastifyInstance;
7 +
8 +beforeAll(async () => {
9 + app = buildApp();
10 + await app.ready();
11 +});
12 +
13 +afterAll(async () => {
14 + await app.close();
15 +});
16 +
17 +describe("api", () => {
18 + it("serves a shallow healthz", async () => {
19 + const res = await app.inject({ method: "GET", url: "/healthz" });
20 + expect(res.statusCode).toBe(200);
21 + expect(res.json()).toMatchObject({ success: true, data: { status: "ok" } });
22 + });
23 +
24 + it("rejects an invalid scrape body with ERR_INVALID_URL and a requestId", async () => {
25 + const res = await app.inject({ method: "POST", url: "/v1/scrape", payload: { url: "not-a-url" } });
26 + expect(res.statusCode).toBe(400);
27 + const body = res.json();
28 + expect(body.success).toBe(false);
29 + expect(body.error.code).toBe("ERR_INVALID_URL");
30 + expect(body.requestId).toMatch(/^tdr_req_/);
31 + expect(res.headers["x-request-id"]).toMatch(/^tdr_req_/);
32 + });
33 +
34 + it("rejects unknown fields (strict schema)", async () => {
35 + const res = await app.inject({
36 + method: "POST",
37 + url: "/v1/scrape",
38 + payload: { url: "https://example.com", bogus: true },
39 + });
40 + expect(res.statusCode).toBe(400);
41 + });
42 +});
added apps/api/src/app.ts +59 −0
@@ -0,0 +1,59 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { randomUUID } from "node:crypto";
3 +import Fastify, { type FastifyInstance } from "fastify";
4 +import { tendrilError } from "@tendril/shared";
5 +import { toApiError } from "./errors.js";
6 +import { ScrapeRequestSchema } from "./schemas.js";
7 +import { runScrape } from "./scrape.js";
8 +
9 +function requestId(): string {
10 + return `tdr_req_${randomUUID()}`;
11 +}
12 +
13 +export function buildApp(): FastifyInstance {
14 + const app = Fastify({
15 + logger: { level: process.env["LOG_LEVEL"] ?? "info" },
16 + genReqId: requestId,
17 + bodyLimit: 2 * 1024 * 1024,
18 + });
19 +
20 + app.addHook("onSend", async (req, reply) => {
21 + reply.header("X-Request-Id", req.id);
22 + });
23 +
24 + app.get("/healthz", async () => ({ success: true, data: { status: "ok" } }));
25 +
26 + app.post("/v1/scrape", async (req, reply) => {
27 + const parsed = ScrapeRequestSchema.safeParse(req.body);
28 + if (!parsed.success) {
29 + const { status, body } = toApiError(
30 + tendrilError("ERR_INVALID_URL", {
31 + message: "Invalid request body",
32 + details: { issues: parsed.error.issues.map((i) => ({ path: i.path, message: i.message })) },
33 + }),
34 + req.id,
35 + );
36 + return reply.status(status).send(body);
37 + }
38 +
39 + const result = await runScrape(parsed.data);
40 + if (!result.ok) {
41 + const { status, body } = toApiError(result.error, req.id);
42 + req.log.warn({ code: result.error.code, url: parsed.data.url }, "scrape failed");
43 + return reply.status(status).send(body);
44 + }
45 +
46 + return reply.status(200).send({ success: true, data: result.value, requestId: req.id });
47 + });
48 +
49 + app.setErrorHandler((error, req, reply) => {
50 + req.log.error({ err: error }, "unhandled error");
51 + const { status, body } = toApiError(
52 + tendrilError("ERR_INTERNAL", { message: "Internal error" }),
53 + req.id,
54 + );
55 + return reply.status(status).send(body);
56 + });
57 +
58 + return app;
59 +}
added apps/api/src/errors.ts +32 −0
@@ -0,0 +1,32 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { httpStatusFor, type ErrorCode, type TendrilError } from "@tendril/shared";
3 +
4 +export interface ApiErrorBody {
5 + readonly success: false;
6 + readonly error: {
7 + readonly code: ErrorCode;
8 + readonly message: string;
9 + readonly details?: Readonly<Record<string, unknown>>;
10 + };
11 + readonly requestId: string;
12 +}
13 +
14 +/**
15 + * The §15 mapping applied at the HTTP boundary — the single place that turns a
16 + * TendrilError into an HTTP status + response envelope. The status table itself
17 + * lives in @tendril/shared ERROR_TAXONOMY; this function is its only consumer at
18 + * the API edge.
19 + */
20 +export function toApiError(error: TendrilError, requestId: string): { status: number; body: ApiErrorBody } {
21 + const status = httpStatusFor(error.code);
22 + const body: ApiErrorBody = {
23 + success: false,
24 + error: {
25 + code: error.code,
26 + message: error.message,
27 + ...(error.details !== undefined ? { details: error.details } : {}),
28 + },
29 + requestId,
30 + };
31 + return { status, body };
32 +}
added apps/api/src/main.ts +17 −0
@@ -0,0 +1,17 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { buildApp } from "./app.js";
3 +
4 +const PORT = Number(process.env["PORT"] ?? 3000);
5 +const HOST = process.env["HOST"] ?? "127.0.0.1";
6 +
7 +const app = buildApp();
8 +
9 +app
10 + .listen({ port: PORT, host: HOST })
11 + .then((address) => {
12 + app.log.info({ address }, "tendril api listening");
13 + })
14 + .catch((error: unknown) => {
15 + app.log.error({ err: error }, "failed to start");
16 + process.exit(1);
17 + });
added apps/api/src/schemas.ts +30 −0
@@ -0,0 +1,30 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { z } from "zod";
3 +
4 +export const OutputFormatSchema = z.enum([
5 + "markdown",
6 + "html",
7 + "rawHtml",
8 + "links",
9 + "screenshot",
10 + "structured",
11 + "extract",
12 +]);
13 +
14 +export const ScrapeRequestSchema = z
15 + .object({
16 + url: z.string().url(),
17 + formats: z.array(OutputFormatSchema).default(["markdown"]),
18 + tier: z.enum(["auto", "http", "webkit", "safari"]).default("auto"),
19 + onlyMainContent: z.boolean().default(true),
20 + includeTags: z.array(z.string()).optional(),
21 + excludeTags: z.array(z.string()).optional(),
22 + waitFor: z.number().int().min(0).max(60_000).default(0),
23 + timeout: z.number().int().min(1_000).max(120_000).default(30_000),
24 + maxAge: z.number().int().min(0).default(0),
25 + headers: z.record(z.string()).optional(),
26 + maxBytes: z.number().int().min(1).optional(),
27 + })
28 + .strict();
29 +
30 +export type ScrapeRequest = z.infer<typeof ScrapeRequestSchema>;
added apps/api/src/scrape.ts +112 −0
@@ -0,0 +1,112 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { performance } from "node:perf_hooks";
3 +import { err, ok, tendrilError, type FetchTimings, type Result, type Tier } from "@tendril/shared";
4 +import { shouldEscalate } from "@tendril/router";
5 +import { httpFetch } from "@tendril/fetcher-http";
6 +import { extract, PIPELINE_VERSION, type ExtractResult } from "@tendril/extract";
7 +import type { ScrapeRequest } from "./schemas.js";
8 +
9 +export interface ScrapeData {
10 + markdown?: string;
11 + html?: string;
12 + rawHtml?: string;
13 + links?: ExtractResult["links"];
14 + structured?: ExtractResult["structured"];
15 + metadata: ExtractResult["metadata"] & { statusCode: number; sourceURL: string; pipelineVersion: string };
16 + tierUsed: Tier;
17 + cached: boolean;
18 + timings: FetchTimings;
19 +}
20 +
21 +function nonHtmlMarkdown(contentType: string, body: string): Result<string> {
22 + const ct = contentType.toLowerCase();
23 + if (ct.includes("application/json")) {
24 + try {
25 + return ok("```json\n" + JSON.stringify(JSON.parse(body), null, 2) + "\n```");
26 + } catch {
27 + return ok(body);
28 + }
29 + }
30 + if (ct.includes("text/plain") || ct.includes("text/markdown") || ct.includes("text/csv") || ct.includes("xml")) {
31 + return ok(body);
32 + }
33 + return err(tendrilError("ERR_UNSUPPORTED_TYPE", { details: { contentType } }));
34 +}
35 +
36 +/**
37 + * Orchestrate a single scrape (§14.1): Tier 0 fetch → escalation decision →
38 + * deterministic extraction. Tiers 1/2 are not yet built, so a page that demands
39 + * escalation under `tier: "auto"` fails closed with ERR_TARGET_BLOCKED and the
40 + * decisive reason, rather than returning a challenge page as if it were content.
41 + */
42 +export async function runScrape(req: ScrapeRequest): Promise<Result<ScrapeData>> {
43 + const started = performance.now();
44 + const timings: FetchTimings = { total: 0, escalations: [] };
45 +
46 + const fetched = await httpFetch(req.url, {
47 + timeout: req.timeout,
48 + ...(req.headers !== undefined ? { headers: req.headers } : {}),
49 + ...(req.maxBytes !== undefined ? { maxBytes: req.maxBytes } : {}),
50 + });
51 + if (!fetched.ok) return fetched;
52 + const page = fetched.value;
53 +
54 + const decision = shouldEscalate({ status: page.status, contentType: page.contentType, body: page.body });
55 +
56 + if (decision.action === "non-html") {
57 + const md = nonHtmlMarkdown(page.contentType, page.body);
58 + if (!md.ok) return md;
59 + timings.total = performance.now() - started;
60 + return ok({
61 + ...(req.formats.includes("markdown") ? { markdown: md.value } : {}),
62 + ...(req.formats.includes("rawHtml") ? { rawHtml: page.body } : {}),
63 + metadata: {
64 + statusCode: page.status,
65 + sourceURL: page.finalUrl,
66 + pipelineVersion: PIPELINE_VERSION,
67 + },
68 + tierUsed: "http",
69 + cached: false,
70 + timings,
71 + });
72 + }
73 +
74 + if (decision.action === "escalate" && req.tier === "auto") {
75 + timings.escalations.push({ from: "http", to: "webkit", reason: decision.reason });
76 + return err(
77 + tendrilError("ERR_TARGET_BLOCKED", {
78 + message: "Tier 0 insufficient and higher tiers are not yet available",
79 + details: { reason: decision.reason, status: page.status },
80 + }),
81 + );
82 + }
83 +
84 + const extractStart = performance.now();
85 + const extracted = extract(page.body, {
86 + url: page.finalUrl,
87 + onlyMainContent: req.onlyMainContent,
88 + ...(req.includeTags !== undefined ? { includeTags: req.includeTags } : {}),
89 + ...(req.excludeTags !== undefined ? { excludeTags: req.excludeTags } : {}),
90 + });
91 + if (!extracted.ok) return extracted;
92 + const e = extracted.value;
93 + timings.extract = performance.now() - extractStart;
94 + timings.total = performance.now() - started;
95 +
96 + return ok({
97 + ...(req.formats.includes("markdown") ? { markdown: e.markdown } : {}),
98 + ...(req.formats.includes("html") ? { html: e.html } : {}),
99 + ...(req.formats.includes("rawHtml") ? { rawHtml: page.body } : {}),
100 + ...(req.formats.includes("links") ? { links: e.links } : {}),
101 + ...(req.formats.includes("structured") ? { structured: e.structured } : {}),
102 + metadata: {
103 + ...e.metadata,
104 + statusCode: page.status,
105 + sourceURL: page.finalUrl,
106 + pipelineVersion: PIPELINE_VERSION,
107 + },
108 + tierUsed: "http",
109 + cached: false,
110 + timings,
111 + });
112 +}
added apps/api/tsconfig.json +31 −0
@@ -0,0 +1,31 @@
1 +{
2 + "extends": "../../tsconfig.base.json",
3 + "compilerOptions": {
4 + "rootDir": "src",
5 + "outDir": "dist",
6 + "noEmit": false
7 + },
8 + "references": [
9 + {
10 + "path": "../../packages/shared"
11 + },
12 + {
13 + "path": "../../packages/router"
14 + },
15 + {
16 + "path": "../../packages/egress"
17 + },
18 + {
19 + "path": "../../packages/fetcher-http"
20 + },
21 + {
22 + "path": "../../packages/extract"
23 + }
24 + ],
25 + "include": [
26 + "src/**/*.ts"
27 + ],
28 + "exclude": [
29 + "src/**/*.test.ts"
30 + ]
31 +}
added package.json +36 −0
@@ -0,0 +1,36 @@
1 +{
2 + "name": "tendril",
3 + "version": "0.1.0",
4 + "private": true,
5 + "description": "Web ingestion platform (search / scrape / crawl / map) on macOS Apple Silicon",
6 + "type": "module",
7 + "packageManager": "pnpm@11.1.2",
8 + "engines": {
9 + "node": ">=22"
10 + },
11 + "scripts": {
12 + "build": "tsc -b apps/api",
13 + "typecheck": "tsc -b apps/api",
14 + "test": "vitest run",
15 + "test:watch": "vitest",
16 + "dev:api": "tsx watch apps/api/src/main.ts",
17 + "lint": "eslint ."
18 + },
19 + "devDependencies": {
20 + "@types/node": "^22.10.0",
21 + "@types/turndown": "^5.0.5",
22 + "tsx": "^4.19.2",
23 + "typescript": "^5.7.2",
24 + "vite-tsconfig-paths": "^5.1.4",
25 + "vitest": "^2.1.8"
26 + },
27 + "dependencies": {
28 + "@mozilla/readability": "^0.5.0",
29 + "fastify": "^5.2.0",
30 + "linkedom": "^0.18.6",
31 + "pino": "^9.5.0",
32 + "turndown": "^7.2.0",
33 + "undici": "^7.2.0",
34 + "zod": "^3.24.1"
35 + }
36 +}
added packages/egress/package.json +12 −0
@@ -0,0 +1,12 @@
1 +{
2 + "name": "@tendril/egress",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "main": "./src/index.ts",
7 + "types": "./src/index.ts",
8 + "exports": { ".": "./src/index.ts" },
9 + "dependencies": {
10 + "@tendril/shared": "workspace:*"
11 + }
12 +}
added packages/egress/src/index.ts +3 −0
@@ -0,0 +1,3 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +export { isBlockedIp, ipToBigInt } from "./ip.js";
3 +export { validateEgress, type SafeTarget, type Resolver } from "./ssrf.js";
added packages/egress/src/ip.test.ts +54 −0
@@ -0,0 +1,54 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { describe, expect, it } from "vitest";
3 +import { isBlockedIp } from "./ip.js";
4 +
5 +describe("isBlockedIp", () => {
6 + it("blocks loopback", () => {
7 + expect(isBlockedIp("127.0.0.1")).toBe(true);
8 + expect(isBlockedIp("127.10.20.30")).toBe(true);
9 + expect(isBlockedIp("::1")).toBe(true);
10 + });
11 +
12 + it("blocks RFC1918 ranges", () => {
13 + expect(isBlockedIp("10.0.0.1")).toBe(true);
14 + expect(isBlockedIp("172.16.5.4")).toBe(true);
15 + expect(isBlockedIp("172.31.255.255")).toBe(true);
16 + expect(isBlockedIp("192.168.1.1")).toBe(true);
17 + });
18 +
19 + it("does not block 172.32.x (outside the /12)", () => {
20 + expect(isBlockedIp("172.32.0.1")).toBe(false);
21 + });
22 +
23 + it("blocks the cloud metadata address", () => {
24 + expect(isBlockedIp("169.254.169.254")).toBe(true);
25 + });
26 +
27 + it("blocks link-local and CGNAT", () => {
28 + expect(isBlockedIp("169.254.1.1")).toBe(true);
29 + expect(isBlockedIp("100.64.0.1")).toBe(true);
30 + });
31 +
32 + it("blocks IPv6 ULA and link-local", () => {
33 + expect(isBlockedIp("fc00::1")).toBe(true);
34 + expect(isBlockedIp("fd12:3456::1")).toBe(true);
35 + expect(isBlockedIp("fe80::1")).toBe(true);
36 + });
37 +
38 + it("unwraps IPv4-mapped IPv6 and blocks private embeds", () => {
39 + expect(isBlockedIp("::ffff:127.0.0.1")).toBe(true);
40 + expect(isBlockedIp("::ffff:10.0.0.1")).toBe(true);
41 + });
42 +
43 + it("allows public addresses", () => {
44 + expect(isBlockedIp("8.8.8.8")).toBe(false);
45 + expect(isBlockedIp("1.1.1.1")).toBe(false);
46 + expect(isBlockedIp("93.184.216.34")).toBe(false);
47 + expect(isBlockedIp("2606:4700:4700::1111")).toBe(false);
48 + });
49 +
50 + it("blocks garbage input", () => {
51 + expect(isBlockedIp("not-an-ip")).toBe(true);
52 + expect(isBlockedIp("999.1.1.1")).toBe(true);
53 + });
54 +});
added packages/egress/src/ip.ts +137 −0
@@ -0,0 +1,137 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { isIP } from "node:net";
3 +
4 +interface Cidr {
5 + readonly base: bigint;
6 + readonly bits: number;
7 + readonly family: 4 | 6;
8 +}
9 +
10 +function ipv4ToBigInt(ip: string): bigint | null {
11 + const parts = ip.split(".");
12 + if (parts.length !== 4) return null;
13 + let acc = 0n;
14 + for (const part of parts) {
15 + if (!/^\d{1,3}$/.test(part)) return null;
16 + const n = Number(part);
17 + if (n > 255) return null;
18 + acc = (acc << 8n) | BigInt(n);
19 + }
20 + return acc;
21 +}
22 +
23 +function ipv6ToBigInt(ip: string): bigint | null {
24 + let addr = ip;
25 + const zone = addr.indexOf("%");
26 + if (zone !== -1) addr = addr.slice(0, zone);
27 +
28 + let mappedSuffix = 0n;
29 + let embeddedV4 = false;
30 + const lastColon = addr.lastIndexOf(":");
31 + const tail = addr.slice(lastColon + 1);
32 + if (tail.includes(".")) {
33 + const v4 = ipv4ToBigInt(tail);
34 + if (v4 === null) return null;
35 + mappedSuffix = v4;
36 + embeddedV4 = true;
37 + addr = addr.slice(0, lastColon + 1) + "0:0";
38 + }
39 +
40 + const halves = addr.split("::");
41 + if (halves.length > 2) return null;
42 + const head = halves[0] === "" || halves[0] === undefined ? [] : halves[0].split(":");
43 + const tailGroups = halves.length === 2 ? (halves[1] === "" ? [] : (halves[1] ?? "").split(":")) : [];
44 +
45 + const groups: string[] = [];
46 + if (halves.length === 2) {
47 + const missing = 8 - head.length - tailGroups.length;
48 + if (missing < 0) return null;
49 + groups.push(...head, ...Array<string>(missing).fill("0"), ...tailGroups);
50 + } else {
51 + groups.push(...head);
52 + }
53 + if (groups.length !== 8) return null;
54 +
55 + let acc = 0n;
56 + for (const g of groups) {
57 + if (!/^[0-9a-fA-F]{1,4}$/.test(g)) return null;
58 + acc = (acc << 16n) | BigInt(parseInt(g, 16));
59 + }
60 + if (embeddedV4) {
61 + acc = (acc & ~0xffffffffn) | mappedSuffix;
62 + }
63 + return acc;
64 +}
65 +
66 +export function ipToBigInt(ip: string, family: 4 | 6): bigint | null {
67 + return family === 4 ? ipv4ToBigInt(ip) : ipv6ToBigInt(ip);
68 +}
69 +
70 +function cidr(spec: string, family: 4 | 6): Cidr {
71 + const [addr, bitsRaw] = spec.split("/");
72 + const bits = Number(bitsRaw);
73 + const base = ipToBigInt(addr ?? "", family);
74 + if (base === null) throw new Error(`bad cidr ${spec}`);
75 + return { base, bits, family };
76 +}
77 +
78 +const V4_BLOCKED: readonly Cidr[] = [
79 + "0.0.0.0/8",
80 + "10.0.0.0/8",
81 + "100.64.0.0/10",
82 + "127.0.0.0/8",
83 + "169.254.0.0/16",
84 + "172.16.0.0/12",
85 + "192.0.0.0/24",
86 + "192.0.2.0/24",
87 + "192.168.0.0/16",
88 + "198.18.0.0/15",
89 + "198.51.100.0/24",
90 + "203.0.113.0/24",
91 + "224.0.0.0/4",
92 + "240.0.0.0/4",
93 +].map((s) => cidr(s, 4));
94 +
95 +const V6_BLOCKED: readonly Cidr[] = [
96 + "::1/128",
97 + "::/128",
98 + "fc00::/7",
99 + "fe80::/10",
100 + "ff00::/8",
101 + "2001:db8::/32",
102 +].map((s) => cidr(s, 6));
103 +
104 +const V4_FULL_BITS = 32;
105 +const V6_FULL_BITS = 128;
106 +const V4_MAPPED_HIGH = 0xffffn; // high 96 bits of ::ffff:a.b.c.d
107 +
108 +function inCidr(value: bigint, c: Cidr): boolean {
109 + const full = c.family === 4 ? V4_FULL_BITS : V6_FULL_BITS;
110 + if (c.bits === 0) return true;
111 + const shift = BigInt(full - c.bits);
112 + return value >> shift === c.base >> shift;
113 +}
114 +
115 +/**
116 + * True when an IP literal points at a private, loopback, link-local, multicast,
117 + * or otherwise forbidden destination (§16.5). Pure — safe to unit test. IPv4-mapped
118 + * IPv6 (`::ffff:a.b.c.d`) is unwrapped and evaluated as IPv4.
119 + */
120 +export function isBlockedIp(ip: string): boolean {
121 + const family = isIP(ip);
122 + if (family === 0) return true;
123 +
124 + if (family === 4) {
125 + const v = ipv4ToBigInt(ip);
126 + if (v === null) return true;
127 + return V4_BLOCKED.some((c) => inCidr(v, c));
128 + }
129 +
130 + const v = ipv6ToBigInt(ip);
131 + if (v === null) return true;
132 + if (v >> 32n === V4_MAPPED_HIGH) {
133 + const embedded = v & 0xffffffffn;
134 + return V4_BLOCKED.some((c) => inCidr(embedded, c));
135 + }
136 + return V6_BLOCKED.some((c) => inCidr(v, c));
137 +}
added packages/egress/src/ssrf.test.ts +54 −0
@@ -0,0 +1,54 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { describe, expect, it } from "vitest";
3 +import { validateEgress, type Resolver } from "./ssrf.js";
4 +
5 +const publicResolver: Resolver = async () => ["93.184.216.34"];
6 +const privateResolver: Resolver = async () => ["10.0.0.5"];
7 +
8 +describe("validateEgress", () => {
9 + it("accepts a public https URL on an allowed port", async () => {
10 + const r = await validateEgress("https://example.com/x", publicResolver);
11 + expect(r.ok).toBe(true);
12 + if (r.ok) {
13 + expect(r.value.port).toBe(443);
14 + expect(r.value.addresses).toEqual(["93.184.216.34"]);
15 + }
16 + });
17 +
18 + it("rejects non-http schemes", async () => {
19 + const r = await validateEgress("ftp://example.com", publicResolver);
20 + expect(r.ok).toBe(false);
21 + if (!r.ok) expect(r.error.code).toBe("ERR_INVALID_URL");
22 + });
23 +
24 + it("rejects disallowed ports before resolving", async () => {
25 + const r = await validateEgress("http://example.com:22/", publicResolver);
26 + expect(r.ok).toBe(false);
27 + if (!r.ok) expect(r.error.code).toBe("ERR_SSRF_BLOCKED");
28 + });
29 +
30 + it("rejects a literal private host without DNS", async () => {
31 + const r = await validateEgress("http://127.0.0.1/", publicResolver);
32 + expect(r.ok).toBe(false);
33 + if (!r.ok) expect(r.error.code).toBe("ERR_SSRF_BLOCKED");
34 + });
35 +
36 + it("rejects when DNS resolves to a private address (rebinding defense)", async () => {
37 + const r = await validateEgress("https://evil.example/", privateResolver);
38 + expect(r.ok).toBe(false);
39 + if (!r.ok) expect(r.error.code).toBe("ERR_SSRF_BLOCKED");
40 + });
41 +
42 + it("rejects if any resolved address is private", async () => {
43 + const mixed: Resolver = async () => ["93.184.216.34", "10.0.0.5"];
44 + const r = await validateEgress("https://example.com/", mixed);
45 + expect(r.ok).toBe(false);
46 + });
47 +
48 + it("allows port 8080 and 8443", async () => {
49 + for (const p of [8080, 8443]) {
50 + const r = await validateEgress(`http://example.com:${p}/`, publicResolver);
51 + expect(r.ok).toBe(true);
52 + }
53 + });
54 +});
added packages/egress/src/ssrf.ts +82 −0
@@ -0,0 +1,82 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { lookup } from "node:dns/promises";
3 +import { isIP } from "node:net";
4 +import { err, ok, tendrilError, type Result } from "@tendril/shared";
5 +import { isBlockedIp } from "./ip.js";
6 +
7 +const ALLOWED_PORTS = new Set([80, 443, 8080, 8443]);
8 +const DEFAULT_PORT: Readonly<Record<string, number>> = { "http:": 80, "https:": 443 };
9 +
10 +export interface SafeTarget {
11 + readonly url: URL;
12 + readonly host: string;
13 + readonly port: number;
14 + /** All resolved addresses, all verified public. Pin these to defeat DNS rebinding. */
15 + readonly addresses: readonly string[];
16 +}
17 +
18 +export type Resolver = (host: string) => Promise<readonly string[]>;
19 +
20 +const defaultResolver: Resolver = async (host) => {
21 + const results = await lookup(host, { all: true });
22 + return results.map((r) => r.address);
23 +};
24 +
25 +/**
26 + * Validate a URL for egress (§16.5). Rejects non-http(s) schemes and disallowed
27 + * ports up front, then resolves DNS and rejects if *any* resolved address is
28 + * private/loopback/link-local. The returned `addresses` should be pinned by the
29 + * caller when connecting, so a rebind between validation and fetch cannot slip
30 + * a private IP through.
31 + */
32 +export async function validateEgress(
33 + raw: string,
34 + resolver: Resolver = defaultResolver,
35 +): Promise<Result<SafeTarget>> {
36 + let url: URL;
37 + try {
38 + url = new URL(raw);
39 + } catch {
40 + return err(tendrilError("ERR_INVALID_URL", { details: { url: raw } }));
41 + }
42 +
43 + if (url.protocol !== "http:" && url.protocol !== "https:") {
44 + return err(tendrilError("ERR_INVALID_URL", { details: { scheme: url.protocol } }));
45 + }
46 +
47 + const port = url.port !== "" ? Number(url.port) : DEFAULT_PORT[url.protocol] ?? 0;
48 + if (!ALLOWED_PORTS.has(port)) {
49 + return err(tendrilError("ERR_SSRF_BLOCKED", { message: "Port not allowed", details: { port } }));
50 + }
51 +
52 + const host = url.hostname.toLowerCase();
53 +
54 + const literal = host.startsWith("[") ? host.slice(1, -1) : host;
55 + const hostIsIp = isIP(literal) !== 0;
56 + if (hostIsIp && isBlockedIp(literal)) {
57 + return err(tendrilError("ERR_SSRF_BLOCKED", { details: { host } }));
58 + }
59 +
60 + let addresses: readonly string[];
61 + if (hostIsIp) {
62 + addresses = [literal];
63 + } else {
64 + try {
65 + addresses = await resolver(literal);
66 + } catch (cause) {
67 + return err(tendrilError("ERR_INVALID_URL", { message: "DNS resolution failed", details: { host }, cause }));
68 + }
69 + }
70 +
71 + if (addresses.length === 0) {
72 + return err(tendrilError("ERR_INVALID_URL", { message: "No addresses resolved", details: { host } }));
73 + }
74 +
75 + for (const addr of addresses) {
76 + if (isBlockedIp(addr)) {
77 + return err(tendrilError("ERR_SSRF_BLOCKED", { details: { host, resolved: addr } }));
78 + }
79 + }
80 +
81 + return ok({ url, host, port, addresses });
82 +}
added packages/egress/tsconfig.json +19 −0
@@ -0,0 +1,19 @@
1 +{
2 + "extends": "../../tsconfig.base.json",
3 + "compilerOptions": {
4 + "rootDir": "src",
5 + "outDir": "dist",
6 + "noEmit": false
7 + },
8 + "references": [
9 + {
10 + "path": "../shared"
11 + }
12 + ],
13 + "include": [
14 + "src/**/*.ts"
15 + ],
16 + "exclude": [
17 + "src/**/*.test.ts"
18 + ]
19 +}
added packages/extract/package.json +15 −0
@@ -0,0 +1,15 @@
1 +{
2 + "name": "@tendril/extract",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "main": "./src/index.ts",
7 + "types": "./src/index.ts",
8 + "exports": { ".": "./src/index.ts" },
9 + "dependencies": {
10 + "@tendril/shared": "workspace:*",
11 + "@mozilla/readability": "^0.5.0",
12 + "linkedom": "^0.18.6",
13 + "turndown": "^7.2.0"
14 + }
15 +}
added packages/extract/src/boilerplate.ts +63 −0
@@ -0,0 +1,63 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { Readability } from "@mozilla/readability";
3 +import { parse, stripChrome } from "./dom.js";
4 +
5 +const MIN_READABILITY_CHARS = 200;
6 +
7 +function textLen(el: Element): number {
8 + return (el.textContent ?? "").replace(/\s+/g, " ").trim().length;
9 +}
10 +
11 +function linkTextLen(el: Element): number {
12 + let sum = 0;
13 + for (const a of Array.from(el.querySelectorAll("a"))) sum += textLen(a);
14 + return sum;
15 +}
16 +
17 +/**
18 + * Density-based main-content fallback (§8.2). For each candidate block, score it
19 + * by `textLength^2 / (1 + linkTextLength)` — favouring large, low-link subtrees —
20 + * and return the winner's HTML. This is the safety net for listing/product pages
21 + * where Readability silently returns almost nothing.
22 + */
23 +export function densityExtract(document: Document): string {
24 + const candidates = Array.from(document.querySelectorAll("article, main, section, div, td"));
25 + let best: Element | null = document.body;
26 + let bestScore = -1;
27 + for (const el of candidates) {
28 + const t = textLen(el);
29 + if (t < 100) continue;
30 + const score = (t * t) / (1 + linkTextLen(el));
31 + if (score > bestScore) {
32 + bestScore = score;
33 + best = el;
34 + }
35 + }
36 + return (best ?? document.body).innerHTML;
37 +}
38 +
39 +export interface MainContent {
40 + readonly html: string;
41 + readonly usedReadability: boolean;
42 +}
43 +
44 +export function extractMainContent(html: string, onlyMainContent: boolean): MainContent {
45 + if (!onlyMainContent) {
46 + const { document } = parse(html);
47 + return { html: document.body.innerHTML, usedReadability: false };
48 + }
49 +
50 + try {
51 + const { document } = parse(html);
52 + const article = new Readability(document as unknown as Document, { charThreshold: MIN_READABILITY_CHARS }).parse();
53 + if (article && (article.textContent ?? "").trim().length >= MIN_READABILITY_CHARS && article.content) {
54 + return { html: article.content, usedReadability: true };
55 + }
56 + } catch {
57 + /* fall through to density */
58 + }
59 +
60 + const { document } = parse(html);
61 + stripChrome(document);
62 + return { html: densityExtract(document), usedReadability: false };
63 +}
added packages/extract/src/dom.ts +135 −0
@@ -0,0 +1,135 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { parseHTML } from "linkedom";
3 +
4 +export interface Dom {
5 + readonly document: Document;
6 + readonly window: Window;
7 +}
8 +
9 +export function parse(html: string): Dom {
10 + const { document, window } = parseHTML(html) as unknown as { document: Document; window: Window };
11 + return { document, window };
12 +}
13 +
14 +/**
15 + * Parse an HTML fragment (e.g. Readability output or an element's innerHTML),
16 + * guaranteeing the content lands in `document.body`. linkedom leaves bare
17 + * fragments outside `<body>`, which silently empties the pipeline otherwise.
18 + */
19 +export function parseFragment(fragment: string): Dom {
20 + return parse(`<!doctype html><html><body>${fragment}</body></html>`);
21 +}
22 +
23 +const DROP_TAGS = ["script", "style", "svg", "noscript", "template", "iframe", "object", "embed"];
24 +
25 +const AD_CLASS_RE = /(^|[-_ ])(ad|ads|advert|sponsor|promo|share|social|related|comment)([-_ ]|$)/i;
26 +const CHROME_SELECTORS = [
27 + "header",
28 + "footer",
29 + "nav",
30 + "[role=navigation]",
31 + "[role=banner]",
32 + "[role=complementary]",
33 + "[aria-hidden=true]",
34 + "[hidden]",
35 +];
36 +
37 +export function sanitize(document: Document): void {
38 + for (const tag of DROP_TAGS) {
39 + for (const el of Array.from(document.querySelectorAll(tag))) el.remove();
40 + }
41 + for (const img of Array.from(document.querySelectorAll("img"))) {
42 + const w = img.getAttribute("width");
43 + const h = img.getAttribute("height");
44 + if ((w === "1" && h === "1") || img.getAttribute("aria-hidden") === "true") img.remove();
45 + }
46 +}
47 +
48 +export function stripChrome(document: Document): void {
49 + for (const sel of CHROME_SELECTORS) {
50 + for (const el of Array.from(document.querySelectorAll(sel))) el.remove();
51 + }
52 + for (const el of Array.from(document.querySelectorAll("[class]"))) {
53 + const cls = el.getAttribute("class") ?? "";
54 + if (AD_CLASS_RE.test(cls)) el.remove();
55 + }
56 +}
57 +
58 +export function dropSelectors(document: Document, selectors: readonly string[]): void {
59 + for (const sel of selectors) {
60 + let matches: Element[];
61 + try {
62 + matches = Array.from(document.querySelectorAll(sel));
63 + } catch {
64 + continue;
65 + }
66 + for (const el of matches) el.remove();
67 + }
68 +}
69 +
70 +export function keepOnly(document: Document, selectors: readonly string[]): void {
71 + const kept: Element[] = [];
72 + for (const sel of selectors) {
73 + try {
74 + kept.push(...Array.from(document.querySelectorAll(sel)));
75 + } catch {
76 + continue;
77 + }
78 + }
79 + if (kept.length === 0) return;
80 + const container = document.createElement("div");
81 + for (const el of kept) container.appendChild(el.cloneNode(true) as Node);
82 + const body = document.body;
83 + body.textContent = "";
84 + body.appendChild(container);
85 +}
86 +
87 +export function absolutizeUrls(document: Document, base: string): void {
88 + const attrs: Array<[string, string]> = [
89 + ["a", "href"],
90 + ["img", "src"],
91 + ["source", "src"],
92 + ["link", "href"],
93 + ];
94 + for (const [tag, attr] of attrs) {
95 + for (const el of Array.from(document.querySelectorAll(tag))) {
96 + const raw = el.getAttribute(attr);
97 + if (raw === null || raw === "") continue;
98 + if (/^(data:|javascript:|mailto:|tel:|#)/i.test(raw)) continue;
99 + try {
100 + el.setAttribute(attr, new URL(raw, base).toString());
101 + } catch {
102 + continue;
103 + }
104 + }
105 + }
106 + for (const img of Array.from(document.querySelectorAll("img"))) {
107 + resolveLazyImage(img, base);
108 + }
109 +}
110 +
111 +function resolveLazyImage(img: Element, base: string): void {
112 + if ((img.getAttribute("src") ?? "") !== "") return;
113 + const lazy = img.getAttribute("data-src") ?? bestFromSrcset(img.getAttribute("srcset"));
114 + if (lazy === null) return;
115 + try {
116 + img.setAttribute("src", new URL(lazy, base).toString());
117 + } catch {
118 + /* ignore malformed lazy URL */
119 + }
120 +}
121 +
122 +function bestFromSrcset(srcset: string | null): string | null {
123 + if (srcset === null || srcset.trim() === "") return null;
124 + const candidates = srcset.split(",").map((part) => {
125 + const [url, size] = part.trim().split(/\s+/);
126 + const width = size?.endsWith("w") ? Number.parseInt(size, 10) : 0;
127 + return { url: url ?? "", width };
128 + });
129 + candidates.sort((a, b) => b.width - a.width);
130 + return candidates[0]?.url ?? null;
131 +}
132 +
133 +export function textContentLength(document: Document): number {
134 + return (document.body.textContent ?? "").replace(/\s+/g, " ").trim().length;
135 +}
added packages/extract/src/fixtures.test.ts +47 −0
@@ -0,0 +1,47 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { readFileSync } from "node:fs";
3 +import { fileURLToPath } from "node:url";
4 +import { dirname, resolve } from "node:path";
5 +import { describe, expect, it } from "vitest";
6 +import { extract } from "./pipeline.js";
7 +
8 +const here = dirname(fileURLToPath(import.meta.url));
9 +const fixture = (name: string): string =>
10 + readFileSync(resolve(here, "../../../test/fixtures/html", name), "utf8");
11 +
12 +describe("fixture: shopify-product", () => {
13 + const html = fixture("shopify-product.html");
14 + const r = extract(html, { url: "https://gardenco.example/products/trellis-frame", onlyMainContent: false });
15 + if (!r.ok) throw new Error("extraction failed");
16 + const d = r.value;
17 +
18 + it("harvests the Product JSON-LD without a model", () => {
19 + const product = d.structured.jsonld.find(
20 + (n): n is Record<string, unknown> => typeof n === "object" && n !== null && (n as Record<string, unknown>)["@type"] === "Product",
21 + );
22 + expect(product?.["sku"]).toBe("TRL-2026");
23 + const offers = product?.["offers"] as Record<string, unknown> | undefined;
24 + expect(offers?.["price"]).toBe("49.00");
25 + });
26 +
27 + it("prefers canonical and OpenGraph metadata", () => {
28 + expect(d.metadata.canonical).toBe("https://gardenco.example/products/trellis-frame");
29 + expect(d.metadata.title).toBe("Trellis Climbing Frame");
30 + expect(d.metadata.siteName).toBe("GardenCo");
31 + });
32 +
33 + it("renders the spec table as a GFM table", () => {
34 + expect(d.markdown).toContain("| Spec | Value |");
35 + expect(d.markdown).toContain("| Material | Powder-coated steel |");
36 + });
37 +
38 + it("resolves lazy data-src images to absolute URLs", () => {
39 + expect(d.html).toContain("https://gardenco.example/img/trellis-800.jpg");
40 + });
41 +
42 + it("absolutizes and classifies links", () => {
43 + const reviews = d.links.find((l) => l.url.includes("/reviews"));
44 + expect(reviews?.url).toBe("https://gardenco.example/products/trellis-frame/reviews");
45 + expect(reviews?.isInternal).toBe(true);
46 + });
47 +});
added packages/extract/src/index.ts +14 −0
@@ -0,0 +1,14 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +export { extract } from "./pipeline.js";
3 +export { htmlToMarkdown, createTurndown, postProcess } from "./markdown.js";
4 +export { harvestStructured } from "./structured.js";
5 +export { extractMainContent, densityExtract } from "./boilerplate.js";
6 +export { extractLinks } from "./links.js";
7 +export {
8 + PIPELINE_VERSION,
9 + type ExtractOptions,
10 + type ExtractResult,
11 + type PageMetadata,
12 + type StructuredData,
13 + type PageLink,
14 +} from "./types.js";
added packages/extract/src/links.ts +35 −0
@@ -0,0 +1,35 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import type { PageLink } from "./types.js";
3 +
4 +export function extractLinks(document: Document, base: string): PageLink[] {
5 + let baseHost: string;
6 + try {
7 + baseHost = new URL(base).hostname.toLowerCase();
8 + } catch {
9 + baseHost = "";
10 + }
11 +
12 + const seen = new Set<string>();
13 + const links: PageLink[] = [];
14 + for (const a of Array.from(document.querySelectorAll("a[href]"))) {
15 + const raw = a.getAttribute("href");
16 + if (raw === null || raw === "" || /^(javascript:|mailto:|tel:|#)/i.test(raw)) continue;
17 + let abs: URL;
18 + try {
19 + abs = new URL(raw, base);
20 + } catch {
21 + continue;
22 + }
23 + if (abs.protocol !== "http:" && abs.protocol !== "https:") continue;
24 + const url = abs.toString();
25 + if (seen.has(url)) continue;
26 + seen.add(url);
27 + links.push({
28 + url,
29 + text: (a.textContent ?? "").replace(/\s+/g, " ").trim(),
30 + rel: a.getAttribute("rel"),
31 + isInternal: abs.hostname.toLowerCase() === baseHost,
32 + });
33 + }
34 + return links;
35 +}
added packages/extract/src/markdown.test.ts +57 −0
@@ -0,0 +1,57 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { describe, expect, it } from "vitest";
3 +import { htmlToMarkdown } from "./markdown.js";
4 +
5 +describe("htmlToMarkdown", () => {
6 + it("uses ATX headings and dash bullets", () => {
7 + const md = htmlToMarkdown("<h1>Title</h1><ul><li>a</li><li>b</li></ul>");
8 + expect(md).toContain("# Title");
9 + expect(md).toContain("- a");
10 + expect(md).toContain("- b");
11 + });
12 +
13 + it("emits fenced code blocks with a language tag", () => {
14 + const md = htmlToMarkdown('<pre><code class="language-ts">const x = 1;</code></pre>');
15 + expect(md).toContain("```ts");
16 + expect(md).toContain("const x = 1;");
17 + });
18 +
19 + it("converts tables to GFM with a header separator", () => {
20 + const md = htmlToMarkdown(
21 + "<table><thead><tr><th>Name</th><th>Age</th></tr></thead><tbody><tr><td>Ada</td><td>36</td></tr></tbody></table>",
22 + );
23 + expect(md).toContain("| Name | Age |");
24 + expect(md).toContain("| --- | --- |");
25 + expect(md).toContain("| Ada | 36 |");
26 + });
27 +
28 + it("escapes pipes inside table cells", () => {
29 + const md = htmlToMarkdown("<table><tr><th>a|b</th></tr><tr><td>c|d</td></tr></table>");
30 + expect(md).toContain("a\\|b");
31 + expect(md).toContain("c\\|d");
32 + });
33 +
34 + it("renders figure + figcaption as image plus italic caption", () => {
35 + const md = htmlToMarkdown('<figure><img src="https://x/y.png" alt="chart"><figcaption>Fig 1</figcaption></figure>');
36 + expect(md).toContain("![chart](https://x/y.png)");
37 + expect(md).toContain("_Fig 1_");
38 + });
39 +
40 + it("removes heading anchor links", () => {
41 + const md = htmlToMarkdown('<h2>Sec <a class="headerlink" href="#sec">¶</a></h2>');
42 + expect(md).toContain("## Sec");
43 + expect(md).not.toContain("¶");
44 + expect(md).not.toContain("#sec");
45 + });
46 +
47 + it("renders definition lists", () => {
48 + const md = htmlToMarkdown("<dl><dt>Term</dt><dd>Meaning</dd></dl>");
49 + expect(md).toContain("**Term**");
50 + expect(md).toContain(": Meaning");
51 + });
52 +
53 + it("collapses 3+ blank lines to 2", () => {
54 + const md = htmlToMarkdown("<p>a</p><br><br><br><p>b</p>");
55 + expect(md).not.toMatch(/\n{3,}/);
56 + });
57 +});
added packages/extract/src/markdown.ts +160 −0
@@ -0,0 +1,160 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import TurndownService from "turndown";
3 +
4 +interface MinimalEl {
5 + textContent: string | null;
6 + getAttribute(name: string): string | null;
7 + querySelectorAll(sel: string): ArrayLike<MinimalEl>;
8 + querySelector(sel: string): MinimalEl | null;
9 +}
10 +
11 +function asEl(node: unknown): MinimalEl {
12 + return node as MinimalEl;
13 +}
14 +
15 +function cellText(el: MinimalEl): string {
16 + return (el.textContent ?? "")
17 + .replace(/\r?\n/g, " ")
18 + .replace(/\s+/g, " ")
19 + .trim()
20 + .replace(/\|/g, "\\|");
21 +}
22 +
23 +function buildTable(node: MinimalEl): string {
24 + const rows = Array.from(node.querySelectorAll("tr"));
25 + if (rows.length === 0) return "";
26 + const grid: string[][] = [];
27 + for (const row of rows) {
28 + const cells = Array.from(asEl(row).querySelectorAll("th, td"));
29 + grid.push(cells.map((c) => cellText(c)));
30 + }
31 + const firstRow = grid[0];
32 + if (firstRow === undefined) return "";
33 + const width = grid.reduce((m, r) => Math.max(m, r.length), 0);
34 + const pad = (r: string[]): string[] => {
35 + const copy = r.slice();
36 + while (copy.length < width) copy.push("");
37 + return copy;
38 + };
39 + const header = pad(firstRow);
40 + const sep = header.map(() => "---");
41 + const body = grid.slice(1).map((r) => pad(r));
42 + const line = (r: string[]): string => `| ${r.join(" | ")} |`;
43 + return ["", line(header), line(sep), ...body.map(line), ""].join("\n");
44 +}
45 +
46 +function detectLanguage(node: MinimalEl): string {
47 + const code = node.querySelector("code");
48 + const cls = (code ?? node).getAttribute("class") ?? "";
49 + const m = /language-([a-z0-9+#-]+)/i.exec(cls) ?? /lang-([a-z0-9+#-]+)/i.exec(cls);
50 + return m?.[1] ?? "";
51 +}
52 +
53 +export function createTurndown(): TurndownService {
54 + const td = new TurndownService({
55 + headingStyle: "atx",
56 + codeBlockStyle: "fenced",
57 + bulletListMarker: "-",
58 + emDelimiter: "_",
59 + hr: "---",
60 + linkStyle: "inlined",
61 + });
62 +
63 + td.remove(["script", "style", "noscript"]);
64 +
65 + td.addRule("removeAnchorLinks", {
66 + filter: (node) => {
67 + const el = asEl(node);
68 + if ((node as { nodeName?: string }).nodeName !== "A") return false;
69 + const href = el.getAttribute("href") ?? "";
70 + const cls = el.getAttribute("class") ?? "";
71 + const text = (el.textContent ?? "").trim();
72 + return href.startsWith("#") && (/anchor|headerlink|permalink/i.test(cls) || text === "" || text === "¶" || text === "#");
73 + },
74 + replacement: () => "",
75 + });
76 +
77 + td.addRule("fencedCodeWithLang", {
78 + filter: (node) => (node as { nodeName?: string }).nodeName === "PRE",
79 + replacement: (_content, node) => {
80 + const el = asEl(node);
81 + const code = el.querySelector("code") ?? el;
82 + const text = code.textContent ?? "";
83 + const lang = detectLanguage(el);
84 + return `\n\n\`\`\`${lang}\n${text.replace(/\n$/, "")}\n\`\`\`\n\n`;
85 + },
86 + });
87 +
88 + td.addRule("gfmTable", {
89 + filter: (node) => (node as { nodeName?: string }).nodeName === "TABLE",
90 + replacement: (_content, node) => {
91 + const el = asEl(node);
92 + const nested = Array.from(el.querySelectorAll("table")).some((x) => x !== node);
93 + if (nested) {
94 + return `\n\n${(node as { outerHTML?: string }).outerHTML ?? ""}\n\n`;
95 + }
96 + return `\n${buildTable(el)}\n`;
97 + },
98 + });
99 +
100 + td.addRule("figureCaption", {
101 + filter: (node) => (node as { nodeName?: string }).nodeName === "FIGURE",
102 + replacement: (_content, node) => {
103 + const el = asEl(node);
104 + const img = el.querySelector("img");
105 + const cap = el.querySelector("figcaption");
106 + const src = img?.getAttribute("src") ?? "";
107 + const alt = img?.getAttribute("alt") ?? "";
108 + const caption = (cap?.textContent ?? "").trim();
109 + const image = src !== "" ? `![${alt}](${src})` : "";
110 + return caption !== "" ? `\n\n${image}\n\n_${caption}_\n\n` : `\n\n${image}\n\n`;
111 + },
112 + });
113 +
114 + td.addRule("definitionList", {
115 + filter: (node) => (node as { nodeName?: string }).nodeName === "DL",
116 + replacement: (_content, node) => {
117 + const el = asEl(node);
118 + const parts: string[] = [];
119 + const children = Array.from(el.querySelectorAll("dt, dd"));
120 + for (const child of children) {
121 + const name = (child as { nodeName?: string }).nodeName;
122 + const text = (child.textContent ?? "").replace(/\s+/g, " ").trim();
123 + if (text === "") continue;
124 + parts.push(name === "DT" ? `\n**${text}**` : `\n: ${text}`);
125 + }
126 + return `\n\n${parts.join("").trim()}\n\n`;
127 + },
128 + });
129 +
130 + td.addRule("strikethrough", {
131 + filter: ["del", "s"],
132 + replacement: (content) => `~~${content}~~`,
133 + });
134 +
135 + return td;
136 +}
137 +
138 +const sharedTurndown = createTurndown();
139 +
140 +export function postProcess(markdown: string): string {
141 + const lines = markdown
142 + .split("\n")
143 + .map((l) => l.replace(/[ \t]+$/, ""))
144 + .map((l) => l.replace(/^(\s*)[-*+][ \t]+/, "$1- "));
145 + const deduped: string[] = [];
146 + const linkRe = /^\s*\[[^\]]*\]\([^)]*\)\s*$/;
147 + for (const line of lines) {
148 + const prev = deduped[deduped.length - 1];
149 + if (linkRe.test(line) && prev !== undefined && prev === line) continue;
150 + deduped.push(line);
151 + }
152 + return deduped
153 + .join("\n")
154 + .replace(/\n{3,}/g, "\n\n")
155 + .trim();
156 +}
157 +
158 +export function htmlToMarkdown(html: string): string {
159 + return postProcess(sharedTurndown.turndown(html));
160 +}
added packages/extract/src/pipeline.test.ts +65 −0
@@ -0,0 +1,65 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { describe, expect, it } from "vitest";
3 +import { extract } from "./pipeline.js";
4 +
5 +const article = `<!doctype html><html lang="en"><head>
6 + <title>How Tendrils Climb</title>
7 + <meta name="description" content="A study of climbing plants.">
8 + <script type="application/ld+json">{"@type":"Article","headline":"How Tendrils Climb","author":{"name":"Botanist"}}</script>
9 +</head><body>
10 + <header><nav><a href="/home">Home</a></nav></header>
11 + <article>
12 + <h1>How Tendrils Climb</h1>
13 + <p>${"Tendrils are specialized stems that coil around supports to hoist the plant upward. ".repeat(6)}</p>
14 + <p>${"They respond to touch through a process called thigmotropism which is fascinating. ".repeat(6)}</p>
15 + <a href="/deep-dive">Read the deep dive</a>
16 + </article>
17 + <footer><a href="/privacy">Privacy</a></footer>
18 +</body></html>`;
19 +
20 +describe("extract pipeline", () => {
21 + it("returns markdown, metadata, structured and links", () => {
22 + const r = extract(article, { url: "https://plants.example/guide", onlyMainContent: true });
23 + expect(r.ok).toBe(true);
24 + if (!r.ok) return;
25 + const d = r.value;
26 + expect(d.metadata.title).toBe("How Tendrils Climb");
27 + expect(d.metadata.author).toBe("Botanist");
28 + expect(d.structured.jsonld.length).toBe(1);
29 + expect(d.markdown).toContain("thigmotropism");
30 + expect(d.textLength).toBeGreaterThan(200);
31 + });
32 +
33 + it("absolutizes links and flags internal vs external", () => {
34 + const r = extract(article, { url: "https://plants.example/guide" });
35 + if (!r.ok) throw new Error("expected ok");
36 + const deep = r.value.links.find((l) => l.url.endsWith("/deep-dive"));
37 + expect(deep?.url).toBe("https://plants.example/deep-dive");
38 + expect(deep?.isInternal).toBe(true);
39 + });
40 +
41 + it("drops chrome (nav/footer) content from markdown in onlyMainContent mode", () => {
42 + const r = extract(article, { url: "https://plants.example/guide", onlyMainContent: true });
43 + if (!r.ok) throw new Error("expected ok");
44 + expect(r.value.markdown).not.toContain("Privacy");
45 + });
46 +
47 + it("honors excludeTags", () => {
48 + const html = `<html><body><main><p>${"keep this important content here for density ".repeat(8)}</p>` +
49 + `<div class="promo-box">buy now discount offer</div></main></body></html>`;
50 + const r = extract(html, { url: "https://x.example/", onlyMainContent: false, excludeTags: [".promo-box"] });
51 + if (!r.ok) throw new Error("expected ok");
52 + expect(r.value.markdown).not.toContain("buy now");
53 + expect(r.value.markdown).toContain("keep this");
54 + });
55 +
56 + it("uses density fallback on a listing page with little prose", () => {
57 + const items = Array.from({ length: 8 }, (_, i) => `<li><a href="/p/${i}">Product ${i}</a> $${i}0.00 in stock</li>`).join("");
58 + const html = `<html><body><nav><a href="/">nav</a></nav><main><ul class="grid">${items}</ul>` +
59 + `<p>${"Some descriptive listing text that gives the container real density weight. ".repeat(4)}</p></main></body></html>`;
60 + const r = extract(html, { url: "https://shop.example/list", onlyMainContent: true });
61 + if (!r.ok) throw new Error("expected ok");
62 + expect(r.value.markdown).toContain("Product 0");
63 + expect(r.value.markdown).toContain("Product 7");
64 + });
65 +});
added packages/extract/src/pipeline.ts +48 −0
@@ -0,0 +1,48 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { ok, type Result } from "@tendril/shared";
3 +import { absolutizeUrls, dropSelectors, keepOnly, parse, parseFragment, sanitize, textContentLength } from "./dom.js";
4 +import { extractMainContent } from "./boilerplate.js";
5 +import { harvestStructured } from "./structured.js";
6 +import { extractLinks } from "./links.js";
7 +import { htmlToMarkdown } from "./markdown.js";
8 +import type { ExtractOptions, ExtractResult } from "./types.js";
9 +
10 +/**
11 + * The deterministic extraction pipeline (§8.1). Pure: `(html, options) => result`,
12 + * no I/O, safe to run offline against saved fixtures. Structured data is harvested
13 + * from the full document before boilerplate removal, so JSON-LD in <head> survives.
14 + */
15 +export function extract(html: string, options: ExtractOptions): Result<ExtractResult> {
16 + const base = options.url;
17 +
18 + const full = parse(html);
19 + const { metadata, structured } = harvestStructured(full.document);
20 + const links = extractLinks(full.document, base);
21 +
22 + const main = extractMainContent(html, options.onlyMainContent ?? true);
23 + const content = parseFragment(main.html);
24 +
25 + if (options.includeTags && options.includeTags.length > 0) {
26 + keepOnly(content.document, options.includeTags);
27 + }
28 + if (options.excludeTags && options.excludeTags.length > 0) {
29 + dropSelectors(content.document, options.excludeTags);
30 + }
31 +
32 + sanitize(content.document);
33 + absolutizeUrls(content.document, base);
34 +
35 + const cleanedHtml = content.document.body.innerHTML;
36 + const markdown = htmlToMarkdown(cleanedHtml);
37 + const textLength = textContentLength(content.document);
38 +
39 + const result: ExtractResult = {
40 + markdown,
41 + html: cleanedHtml,
42 + links,
43 + metadata,
44 + structured,
45 + textLength,
46 + };
47 + return ok(result);
48 +}
added packages/extract/src/structured.test.ts +77 −0
@@ -0,0 +1,77 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { describe, expect, it } from "vitest";
3 +import { parse } from "./dom.js";
4 +import { harvestStructured } from "./structured.js";
5 +
6 +describe("harvestStructured", () => {
7 + it("harvests JSON-LD, resolving @graph", () => {
8 + const html = `<html><head>
9 + <script type="application/ld+json">
10 + {"@context":"https://schema.org","@graph":[
11 + {"@type":"Article","headline":"The Headline","datePublished":"2026-01-01","author":{"@type":"Person","name":"Ada"}},
12 + {"@type":"Organization","name":"Acme"}
13 + ]}</script>
14 + </head><body><h1>Fallback</h1></body></html>`;
15 + const { metadata, structured } = harvestStructured(parse(html).document);
16 + expect(structured.jsonld.length).toBe(2);
17 + expect(metadata.title).toBe("The Headline");
18 + expect(metadata.publishedTime).toBe("2026-01-01");
19 + expect(metadata.author).toBe("Ada");
20 + });
21 +
22 + it("prefers JSON-LD over OpenGraph over meta over heuristics", () => {
23 + const html = `<html><head>
24 + <title>Doc Title</title>
25 + <meta name="description" content="meta desc">
26 + <meta property="og:title" content="OG Title">
27 + <meta property="og:description" content="og desc">
28 + <script type="application/ld+json">{"@type":"Article","headline":"LD Title"}</script>
29 + </head><body><h1>H1 Title</h1></body></html>`;
30 + const { metadata } = harvestStructured(parse(html).document);
31 + expect(metadata.title).toBe("LD Title");
32 + expect(metadata.description).toBe("og desc");
33 + });
34 +
35 + it("falls back to og:title then <title> then h1", () => {
36 + const ogOnly = harvestStructured(
37 + parse('<html><head><meta property="og:title" content="OG"><title>T</title></head><body><h1>H</h1></body></html>')
38 + .document,
39 + );
40 + expect(ogOnly.metadata.title).toBe("OG");
41 + const titleOnly = harvestStructured(parse("<html><head><title>T</title></head><body><h1>H</h1></body></html>").document);
42 + expect(titleOnly.metadata.title).toBe("T");
43 + const h1Only = harvestStructured(parse("<html><body><h1>H</h1></body></html>").document);
44 + expect(h1Only.metadata.title).toBe("H");
45 + });
46 +
47 + it("harvests OpenGraph and Twitter cards", () => {
48 + const html = `<html><head>
49 + <meta property="og:site_name" content="Acme">
50 + <meta property="og:image" content="https://x/og.png">
51 + <meta name="twitter:image" content="https://x/tw.png">
52 + </head><body></body></html>`;
53 + const { metadata, structured } = harvestStructured(parse(html).document);
54 + expect(structured.openGraph["site_name"]).toBe("Acme");
55 + expect(metadata.siteName).toBe("Acme");
56 + expect(metadata.image).toBe("https://x/og.png");
57 + expect(structured.twitter["image"]).toBe("https://x/tw.png");
58 + });
59 +
60 + it("ignores malformed JSON-LD without throwing", () => {
61 + const { structured } = harvestStructured(
62 + parse('<html><head><script type="application/ld+json">{bad json,</script></head><body></body></html>').document,
63 + );
64 + expect(structured.jsonld).toEqual([]);
65 + });
66 +
67 + it("splits keywords and reads canonical + lang", () => {
68 + const html = `<html lang="fr"><head>
69 + <meta name="keywords" content="a, b ,c">
70 + <link rel="canonical" href="https://x/canon">
71 + </head><body></body></html>`;
72 + const { metadata } = harvestStructured(parse(html).document);
73 + expect(metadata.keywords).toEqual(["a", "b", "c"]);
74 + expect(metadata.canonical).toBe("https://x/canon");
75 + expect(metadata.lang).toBe("fr");
76 + });
77 +});
added packages/extract/src/structured.ts +127 −0
@@ -0,0 +1,127 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import type { PageMetadata, StructuredData } from "./types.js";
3 +
4 +function flattenJsonLd(node: unknown, out: unknown[]): void {
5 + if (Array.isArray(node)) {
6 + for (const item of node) flattenJsonLd(item, out);
7 + return;
8 + }
9 + if (node !== null && typeof node === "object") {
10 + const obj = node as Record<string, unknown>;
11 + if ("@graph" in obj) {
12 + flattenJsonLd(obj["@graph"], out);
13 + const rest = { ...obj };
14 + delete rest["@graph"];
15 + delete rest["@context"];
16 + if (Object.keys(rest).length > 0) out.push(rest);
17 + return;
18 + }
19 + out.push(obj);
20 + }
21 +}
22 +
23 +function harvestJsonLd(document: Document): unknown[] {
24 + const out: unknown[] = [];
25 + for (const el of Array.from(document.querySelectorAll('script[type="application/ld+json"]'))) {
26 + const raw = el.textContent ?? "";
27 + if (raw.trim() === "") continue;
28 + try {
29 + flattenJsonLd(JSON.parse(raw), out);
30 + } catch {
31 + continue;
32 + }
33 + }
34 + return out;
35 +}
36 +
37 +function harvestMetaPrefixed(document: Document, attr: "property" | "name", prefix: string): Record<string, string> {
38 + const map: Record<string, string> = {};
39 + for (const el of Array.from(document.querySelectorAll(`meta[${attr}^="${prefix}"]`))) {
40 + const key = (el.getAttribute(attr) ?? "").slice(prefix.length);
41 + const content = el.getAttribute("content");
42 + if (key !== "" && content !== null && !(key in map)) map[key] = content;
43 + }
44 + return map;
45 +}
46 +
47 +function metaContent(document: Document, name: string): string | undefined {
48 + const el = document.querySelector(`meta[name="${name}" i]`);
49 + const content = el?.getAttribute("content");
50 + return content !== null && content !== undefined && content !== "" ? content : undefined;
51 +}
52 +
53 +function firstDefined(...values: Array<string | undefined>): string | undefined {
54 + for (const v of values) if (v !== undefined && v !== "") return v;
55 + return undefined;
56 +}
57 +
58 +function jsonLdString(nodes: unknown[], types: string[], keys: string[]): string | undefined {
59 + for (const node of nodes) {
60 + if (node === null || typeof node !== "object") continue;
61 + const obj = node as Record<string, unknown>;
62 + const t = obj["@type"];
63 + const typeStr = Array.isArray(t) ? t.map(String) : typeof t === "string" ? [t] : [];
64 + if (types.length > 0 && !typeStr.some((x) => types.includes(x))) continue;
65 + for (const key of keys) {
66 + const val = obj[key];
67 + if (typeof val === "string" && val !== "") return val;
68 + if (Array.isArray(val) && typeof val[0] === "string" && val[0] !== "") return val[0];
69 + if (val !== null && typeof val === "object") {
70 + const obj2 = val as Record<string, unknown>;
71 + for (const nested of ["name", "url"]) {
72 + const nv = obj2[nested];
73 + if (typeof nv === "string" && nv !== "") return nv;
74 + }
75 + }
76 + }
77 + }
78 + return undefined;
79 +}
80 +
81 +export function harvestStructured(document: Document): { metadata: PageMetadata; structured: StructuredData } {
82 + const jsonld = harvestJsonLd(document);
83 + const openGraph = harvestMetaPrefixed(document, "property", "og:");
84 + const twitter = harvestMetaPrefixed(document, "name", "twitter:");
85 + const structured: StructuredData = { jsonld, openGraph, twitter };
86 +
87 + const htmlLang = document.querySelector("html")?.getAttribute("lang") ?? undefined;
88 + const canonical = document.querySelector('link[rel="canonical"]')?.getAttribute("href") ?? undefined;
89 + const docTitle = firstDefined(document.querySelector("title")?.textContent?.trim());
90 + const h1 = firstDefined(document.querySelector("h1")?.textContent?.trim());
91 +
92 + let firstPara: string | undefined;
93 + for (const p of Array.from(document.querySelectorAll("p"))) {
94 + const text = (p.textContent ?? "").trim();
95 + if (text.length > 100) {
96 + firstPara = text;
97 + break;
98 + }
99 + }
100 +
101 + const metadata: PageMetadata = {};
102 + const set = <K extends keyof PageMetadata>(key: K, value: PageMetadata[K] | undefined): void => {
103 + if (value !== undefined) metadata[key] = value;
104 + };
105 +
106 + set("title", firstDefined(jsonLdString(jsonld, [], ["headline", "name"]), openGraph["title"], docTitle, h1));
107 + set(
108 + "description",
109 + firstDefined(jsonLdString(jsonld, [], ["description"]), openGraph["description"], metaContent(document, "description"), firstPara),
110 + );
111 + set("author", firstDefined(jsonLdString(jsonld, [], ["author"]), metaContent(document, "author")));
112 + set("image", firstDefined(jsonLdString(jsonld, [], ["image"]), openGraph["image"], twitter["image"]));
113 + set("siteName", firstDefined(openGraph["site_name"]));
114 + set("type", firstDefined(openGraph["type"]));
115 + set("publishedTime", firstDefined(jsonLdString(jsonld, [], ["datePublished"]), openGraph["article:published_time"]));
116 + set("modifiedTime", firstDefined(jsonLdString(jsonld, [], ["dateModified"]), openGraph["article:modified_time"]));
117 + set("canonical", canonical);
118 + set("lang", htmlLang);
119 +
120 + const keywords = metaContent(document, "keywords");
121 + if (keywords !== undefined) {
122 + const list = keywords.split(",").map((k) => k.trim()).filter((k) => k !== "");
123 + if (list.length > 0) metadata.keywords = list;
124 + }
125 +
126 + return { metadata, structured };
127 +}
added packages/extract/src/types.ts +45 −0
@@ -0,0 +1,45 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +export interface PageMetadata {
3 + title?: string;
4 + description?: string;
5 + author?: string;
6 + canonical?: string;
7 + lang?: string;
8 + image?: string;
9 + siteName?: string;
10 + type?: string;
11 + publishedTime?: string;
12 + modifiedTime?: string;
13 + keywords?: string[];
14 +}
15 +
16 +export interface StructuredData {
17 + jsonld: unknown[];
18 + openGraph: Record<string, string>;
19 + twitter: Record<string, string>;
20 +}
21 +
22 +export interface ExtractOptions {
23 + readonly url: string;
24 + readonly onlyMainContent?: boolean;
25 + readonly includeTags?: readonly string[];
26 + readonly excludeTags?: readonly string[];
27 +}
28 +
29 +export interface ExtractResult {
30 + readonly markdown: string;
31 + readonly html: string;
32 + readonly links: PageLink[];
33 + readonly metadata: PageMetadata;
34 + readonly structured: StructuredData;
35 + readonly textLength: number;
36 +}
37 +
38 +export interface PageLink {
39 + readonly url: string;
40 + readonly text: string;
41 + readonly rel: string | null;
42 + readonly isInternal: boolean;
43 +}
44 +
45 +export const PIPELINE_VERSION = "0.1.0";
added packages/extract/tsconfig.json +23 −0
@@ -0,0 +1,23 @@
1 +{
2 + "extends": "../../tsconfig.base.json",
3 + "compilerOptions": {
4 + "rootDir": "src",
5 + "outDir": "dist",
6 + "noEmit": false,
7 + "lib": [
8 + "ES2023",
9 + "DOM"
10 + ]
11 + },
12 + "references": [
13 + {
14 + "path": "../shared"
15 + }
16 + ],
17 + "include": [
18 + "src/**/*.ts"
19 + ],
20 + "exclude": [
21 + "src/**/*.test.ts"
22 + ]
23 +}
added packages/fetcher-http/package.json +15 −0
@@ -0,0 +1,15 @@
1 +{
2 + "name": "@tendril/fetcher-http",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "main": "./src/index.ts",
7 + "types": "./src/index.ts",
8 + "exports": { ".": "./src/index.ts" },
9 + "dependencies": {
10 + "@tendril/shared": "workspace:*",
11 + "@tendril/egress": "workspace:*",
12 + "@tendril/router": "workspace:*",
13 + "undici": "^7.2.0"
14 + }
15 +}
added packages/fetcher-http/src/fetch.ts +192 −0
@@ -0,0 +1,192 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { Agent, request } from "undici";
3 +import type { LookupFunction } from "node:net";
4 +import { brotliDecompressSync, gunzipSync, inflateSync } from "node:zlib";
5 +import {
6 + err,
7 + normalizeUrl,
8 + ok,
9 + tendrilError,
10 + type FetchResult,
11 + type RedirectHop,
12 + type Result,
13 +} from "@tendril/shared";
14 +import { validateEgress, type Resolver } from "@tendril/egress";
15 +import { buildHeaders } from "./headers.js";
16 +
17 +const MAX_REDIRECTS = 5;
18 +const DEFAULT_MAX_BYTES = 20 * 1024 * 1024;
19 +
20 +const pinnedAddresses = new Map<string, string>();
21 +
22 +const pinnedLookup: LookupFunction = (hostname, options, callback) => {
23 + const pinned = pinnedAddresses.get(hostname);
24 + if (pinned === undefined) {
25 + callback(new Error(`no pinned address for ${hostname}`) as NodeJS.ErrnoException, "");
26 + return;
27 + }
28 + const family = pinned.includes(":") ? 6 : 4;
29 + if (options.all === true) {
30 + callback(null, [{ address: pinned, family }]);
31 + } else {
32 + callback(null, pinned, family);
33 + }
34 +};
35 +
36 +const agent = new Agent({
37 + connections: 64,
38 + pipelining: 1,
39 + keepAliveTimeout: 30_000,
40 + keepAliveMaxTimeout: 120_000,
41 + bodyTimeout: 20_000,
42 + headersTimeout: 10_000,
43 + connect: {
44 + timeout: 8_000,
45 + rejectUnauthorized: true,
46 + lookup: pinnedLookup,
47 + },
48 +});
49 +
50 +export interface HttpFetchOptions {
51 + readonly timeout?: number;
52 + readonly maxBytes?: number;
53 + readonly headers?: Readonly<Record<string, string>>;
54 + readonly userAgent?: string;
55 + readonly resolver?: Resolver;
56 +}
57 +
58 +function decompress(buf: Buffer, encoding: string | undefined): Buffer {
59 + try {
60 + switch ((encoding ?? "").toLowerCase()) {
61 + case "gzip":
62 + return gunzipSync(buf);
63 + case "deflate":
64 + return inflateSync(buf);
65 + case "br":
66 + return brotliDecompressSync(buf);
67 + default:
68 + return buf;
69 + }
70 + } catch {
71 + return buf;
72 + }
73 +}
74 +
75 +async function readCapped(body: AsyncIterable<Buffer>, max: number): Promise<Buffer | null> {
76 + const chunks: Buffer[] = [];
77 + let total = 0;
78 + for await (const chunk of body) {
79 + total += chunk.length;
80 + if (total > max) return null;
81 + chunks.push(chunk);
82 + }
83 + return Buffer.concat(chunks);
84 +}
85 +
86 +const META_REFRESH_RE = /<meta[^>]+http-equiv=["']?refresh["']?[^>]*content=["'][^"']*url=([^"'>\s]+)/i;
87 +
88 +function firstHeader(value: string | string[] | undefined): string | undefined {
89 + if (Array.isArray(value)) return value[0];
90 + return value;
91 +}
92 +
93 +/**
94 + * Tier 0 HTTP fetch (§3). Handles redirects manually so it can re-validate SSRF
95 + * on every hop (§16.5), record the full chain, detect meta-refresh redirects the
96 + * transport cannot see, and stop on a redirect loop. Connections are pinned to
97 + * the SSRF-validated IP to defeat DNS rebinding between validation and connect.
98 + */
99 +export async function httpFetch(rawUrl: string, options: HttpFetchOptions = {}): Promise<Result<FetchResult>> {
100 + const maxBytes = options.maxBytes ?? DEFAULT_MAX_BYTES;
101 + const redirects: RedirectHop[] = [];
102 + const seen = new Set<string>();
103 +
104 + let current = rawUrl;
105 + for (let hop = 0; hop <= MAX_REDIRECTS; hop++) {
106 + const norm = normalizeUrl(current);
107 + if (!norm.ok) return norm;
108 + if (seen.has(norm.value)) return err(tendrilError("ERR_REDIRECT_LOOP", { details: { url: current } }));
109 + seen.add(norm.value);
110 +
111 + const safe = await validateEgress(current, options.resolver);
112 + if (!safe.ok) return safe;
113 + const address = safe.value.addresses[0];
114 + if (address === undefined) return err(tendrilError("ERR_SSRF_BLOCKED", { details: { host: safe.value.host } }));
115 + pinnedAddresses.set(safe.value.host, address);
116 +
117 + let res: Awaited<ReturnType<typeof request>>;
118 + try {
119 + res = await request(current, {
120 + dispatcher: agent,
121 + method: "GET",
122 + headersTimeout: options.timeout ?? 10_000,
123 + bodyTimeout: options.timeout ?? 20_000,
124 + headers: buildHeaders(safe.value.host, options.headers, options.userAgent),
125 + });
126 + } catch (cause) {
127 + return err(tendrilError("ERR_TARGET_5XX", { message: "Transport error", details: { url: current }, cause }));
128 + }
129 +
130 + const status = res.statusCode;
131 + const location = firstHeader(res.headers["location"]);
132 +
133 + if (status >= 300 && status < 400 && location !== undefined && location !== "") {
134 + res.body.dump().catch(() => undefined);
135 + if (hop === MAX_REDIRECTS) return err(tendrilError("ERR_REDIRECT_LOOP", { details: { url: current } }));
136 + let next: string;
137 + try {
138 + next = new URL(location, current).toString();
139 + } catch {
140 + return err(tendrilError("ERR_INVALID_URL", { details: { location } }));
141 + }
142 + redirects.push({ from: current, to: next, status });
143 + current = next;
144 + continue;
145 + }
146 +
147 + const rawBody = await readCapped(res.body, maxBytes);
148 + if (rawBody === null) return err(tendrilError("ERR_TOO_LARGE", { details: { max: maxBytes } }));
149 +
150 + const contentType = (firstHeader(res.headers["content-type"]) ?? "").toLowerCase();
151 + const encoding = firstHeader(res.headers["content-encoding"]);
152 + const decoded = decompress(rawBody, encoding);
153 + const bodyText = decoded.toString("utf8");
154 +
155 + const isHtml = contentType.includes("text/html") || contentType.includes("xhtml") || contentType === "";
156 + if (isHtml && hop < MAX_REDIRECTS) {
157 + const meta = META_REFRESH_RE.exec(bodyText);
158 + if (meta?.[1] !== undefined) {
159 + let next: string;
160 + try {
161 + next = new URL(meta[1], current).toString();
162 + } catch {
163 + next = "";
164 + }
165 + if (next !== "" && !seen.has(normalizeUrl(next).ok ? (normalizeUrl(next) as { value: string }).value : next)) {
166 + redirects.push({ from: current, to: next, status: 200 });
167 + current = next;
168 + continue;
169 + }
170 + }
171 + }
172 +
173 + const headers: Record<string, string> = {};
174 + for (const [key, value] of Object.entries(res.headers)) {
175 + headers[key] = Array.isArray(value) ? value.join(", ") : (value ?? "");
176 + }
177 +
178 + const result: FetchResult = {
179 + tier: "http",
180 + status,
181 + finalUrl: current,
182 + contentType,
183 + body: bodyText,
184 + bodyBytes: decoded.length,
185 + redirects,
186 + headers,
187 + };
188 + return ok(result);
189 + }
190 +
191 + return err(tendrilError("ERR_REDIRECT_LOOP", { details: { url: rawUrl } }));
192 +}
added packages/fetcher-http/src/headers.test.ts +37 −0
@@ -0,0 +1,37 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { describe, expect, it } from "vitest";
3 +import { buildHeaders, BOT_SUFFIX } from "./headers.js";
4 +
5 +describe("buildHeaders", () => {
6 + it("emits headers in Safari order", () => {
7 + const keys = Object.keys(buildHeaders("example.com"));
8 + expect(keys).toEqual([
9 + "Host",
10 + "Accept",
11 + "Accept-Language",
12 + "Accept-Encoding",
13 + "User-Agent",
14 + "Connection",
15 + "Sec-Fetch-Dest",
16 + "Sec-Fetch-Mode",
17 + "Sec-Fetch-Site",
18 + "Sec-Fetch-User",
19 + "Upgrade-Insecure-Requests",
20 + ]);
21 + });
22 +
23 + it("always appends the bot suffix to the UA (§26)", () => {
24 + expect(buildHeaders("example.com")["User-Agent"]).toContain(BOT_SUFFIX);
25 + expect(buildHeaders("example.com", {}, "Custom/9")["User-Agent"]).toBe("Custom/9" + BOT_SUFFIX);
26 + });
27 +
28 + it("sets the Host header from the argument", () => {
29 + expect(buildHeaders("shop.example")["Host"]).toBe("shop.example");
30 + });
31 +
32 + it("merges overrides over defaults without reordering base keys", () => {
33 + const h = buildHeaders("example.com", { "Accept-Language": "fr-CA,fr;q=0.9", "X-Extra": "1" });
34 + expect(h["Accept-Language"]).toBe("fr-CA,fr;q=0.9");
35 + expect(h["X-Extra"]).toBe("1");
36 + });
37 +});
added packages/fetcher-http/src/headers.ts +36 −0
@@ -0,0 +1,36 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +export const SAFARI_UA =
3 + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15";
4 +
5 +export const BOT_SUFFIX = " Tendril/1.0 (+https://www.ten-dril.com/bot)";
6 +
7 +export const DEFAULT_USER_AGENT = SAFARI_UA + BOT_SUFFIX;
8 +
9 +/**
10 + * Build request headers in Safari's emission order (§3.2). Header order is a
11 + * fingerprint, so this returns an insertion-ordered object; do not sort it.
12 + * The bot suffix in the UA is non-negotiable (§26).
13 + */
14 +export function buildHeaders(
15 + host: string,
16 + overrides?: Readonly<Record<string, string>>,
17 + userAgent?: string,
18 +): Record<string, string> {
19 + const headers: Record<string, string> = {
20 + Host: host,
21 + Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
22 + "Accept-Language": "en-US,en;q=0.9",
23 + "Accept-Encoding": "gzip, deflate, br",
24 + "User-Agent": (userAgent ?? SAFARI_UA) + BOT_SUFFIX,
25 + Connection: "keep-alive",
26 + "Sec-Fetch-Dest": "document",
27 + "Sec-Fetch-Mode": "navigate",
28 + "Sec-Fetch-Site": "none",
29 + "Sec-Fetch-User": "?1",
30 + "Upgrade-Insecure-Requests": "1",
31 + };
32 + if (overrides) {
33 + for (const [key, value] of Object.entries(overrides)) headers[key] = value;
34 + }
35 + return headers;
36 +}
added packages/fetcher-http/src/index.ts +3 −0
@@ -0,0 +1,3 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +export { httpFetch, type HttpFetchOptions } from "./fetch.js";
3 +export { buildHeaders, DEFAULT_USER_AGENT, SAFARI_UA, BOT_SUFFIX } from "./headers.js";
added packages/fetcher-http/tsconfig.json +25 −0
@@ -0,0 +1,25 @@
1 +{
2 + "extends": "../../tsconfig.base.json",
3 + "compilerOptions": {
4 + "rootDir": "src",
5 + "outDir": "dist",
6 + "noEmit": false
7 + },
8 + "references": [
9 + {
10 + "path": "../shared"
11 + },
12 + {
13 + "path": "../egress"
14 + },
15 + {
16 + "path": "../router"
17 + }
18 + ],
19 + "include": [
20 + "src/**/*.ts"
21 + ],
22 + "exclude": [
23 + "src/**/*.test.ts"
24 + ]
25 +}
added packages/router/package.json +12 −0
@@ -0,0 +1,12 @@
1 +{
2 + "name": "@tendril/router",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "main": "./src/index.ts",
7 + "types": "./src/index.ts",
8 + "exports": { ".": "./src/index.ts" },
9 + "dependencies": {
10 + "@tendril/shared": "workspace:*"
11 + }
12 +}
added packages/router/src/escalate.test.ts +75 −0
@@ -0,0 +1,75 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { describe, expect, it } from "vitest";
3 +import { shouldEscalate } from "./escalate.js";
4 +
5 +const goodHtml =
6 + "<html><head><title>Article</title></head><body>" +
7 + "<h1>Heading</h1>" +
8 + "<p>" +
9 + "Lorem ipsum dolor sit amet consectetur adipiscing elit sed do eiusmod. ".repeat(20) +
10 + "</p>" +
11 + '<a href="/next">next</a></body></html>';
12 +
13 +describe("shouldEscalate", () => {
14 + it("proceeds on a healthy article page", () => {
15 + expect(shouldEscalate({ status: 200, contentType: "text/html", body: goodHtml })).toEqual({
16 + action: "proceed",
17 + });
18 + });
19 +
20 + it("routes non-HTML content to §9 without escalating", () => {
21 + const d = shouldEscalate({ status: 200, contentType: "application/pdf", body: "%PDF-1.7" });
22 + expect(d.action).toBe("non-html");
23 + });
24 +
25 + it("escalates on 403/429/503 (decisive)", () => {
26 + for (const status of [403, 429, 503]) {
27 + const d = shouldEscalate({ status, contentType: "text/html", body: goodHtml });
28 + expect(d.action).toBe("escalate");
29 + if (d.action === "escalate") expect(d.reason).toContain(`status:${status}`);
30 + }
31 + });
32 +
33 + it("escalates on a Cloudflare challenge title (decisive)", () => {
34 + const d = shouldEscalate({
35 + status: 200,
36 + contentType: "text/html",
37 + body: "<html><head><title>Just a moment...</title></head><body></body></html>",
38 + });
39 + expect(d.action).toBe("escalate");
40 + if (d.action === "escalate") expect(d.reason).toContain("challenge-title");
41 + });
42 +
43 + it("escalates on known challenge markers (decisive)", () => {
44 + const d = shouldEscalate({
45 + status: 200,
46 + contentType: "text/html",
47 + body: `<html><body>${goodHtml}<script>datadome</script></body></html>`,
48 + });
49 + expect(d.action).toBe("escalate");
50 + if (d.action === "escalate") expect(d.reason).toContain("challenge:datadome");
51 + });
52 +
53 + it("escalates on an empty SPA root under 2KB (decisive)", () => {
54 + const d = shouldEscalate({
55 + status: 200,
56 + contentType: "text/html",
57 + body: '<html><body><div id="root"></div></body></html>',
58 + });
59 + expect(d.action).toBe("escalate");
60 + if (d.action === "escalate") expect(d.reason).toContain("empty-spa-root");
61 + });
62 +
63 + it("does not escalate on a single strong signal alone", () => {
64 + const body = `<html><body><noscript>Please enable JavaScript to continue</noscript>${goodHtml}</body></html>`;
65 + const d = shouldEscalate({ status: 200, contentType: "text/html", body });
66 + expect(d.action).toBe("proceed");
67 + });
68 +
69 + it("escalates on two weak signals together", () => {
70 + const filler = "x".repeat(11_000);
71 + const body = `<html><body><noscript>JavaScript is required</noscript><!--${filler}--></body></html>`;
72 + const d = shouldEscalate({ status: 200, contentType: "text/html", body });
73 + expect(d.action).toBe("escalate");
74 + });
75 +});
added packages/router/src/escalate.ts +104 −0
@@ -0,0 +1,104 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +export interface EscalationInput {
3 + readonly status: number;
4 + readonly contentType: string;
5 + readonly body: string;
6 +}
7 +
8 +export type TierDecision =
9 + | { readonly action: "proceed" }
10 + | { readonly action: "escalate"; readonly reason: string }
11 + | { readonly action: "non-html"; readonly reason: string };
12 +
13 +type Severity = "decisive" | "strong" | "moderate";
14 +interface Signal {
15 + readonly severity: Severity;
16 + readonly reason: string;
17 +}
18 +
19 +const CHALLENGE_MARKERS = [
20 + "challenge-platform",
21 + "cf-browser-verification",
22 + "_Incapsula_",
23 + "px-captcha",
24 + "datadome",
25 +];
26 +
27 +const EMPTY_ROOT_RE = /<[^>]+\bid=["']?(root|__next|app)["']?[^>]*>\s*<\/[a-z]+>/i;
28 +const NOSCRIPT_JS_RE = /<noscript[^>]*>[\s\S]*?(enable\s+javascript|javascript\s+is\s+required)[\s\S]*?<\/noscript>/i;
29 +const TITLE_RE = /<title[^>]*>\s*(just a moment|attention required|access denied)/i;
30 +const HREF_RE = /<a\s[^>]*href\s*=/i;
31 +
32 +function isHtml(contentType: string): boolean {
33 + const ct = contentType.toLowerCase();
34 + return ct.includes("text/html") || ct.includes("application/xhtml+xml") || ct === "";
35 +}
36 +
37 +function textHtmlRatio(html: string): number {
38 + if (html.length === 0) return 0;
39 + const text = html
40 + .replace(/<script[\s\S]*?<\/script>/gi, "")
41 + .replace(/<style[\s\S]*?<\/style>/gi, "")
42 + .replace(/<[^>]+>/g, "")
43 + .replace(/\s+/g, " ")
44 + .trim();
45 + return text.length / html.length;
46 +}
47 +
48 +/**
49 + * Decide whether a Tier 0 response should escalate to Tier 1 (§3.4). Pure, no I/O.
50 + * Decisive signals escalate on their own; two or more strong/moderate signals
51 + * escalate together. Non-HTML content types route to §9 instead of escalating.
52 + */
53 +export function shouldEscalate(input: EscalationInput): TierDecision {
54 + if (!isHtml(input.contentType)) {
55 + return { action: "non-html", reason: `content-type:${input.contentType || "unknown"}` };
56 + }
57 +
58 + const signals: Signal[] = [];
59 + const body = input.body;
60 + const bytes = Buffer.byteLength(body, "utf8");
61 +
62 + if (input.status === 403 || input.status === 429 || input.status === 503) {
63 + signals.push({ severity: "decisive", reason: `status:${input.status}` });
64 + }
65 +
66 + if (bytes < 2048 && EMPTY_ROOT_RE.test(body)) {
67 + signals.push({ severity: "decisive", reason: "empty-spa-root" });
68 + }
69 +
70 + for (const marker of CHALLENGE_MARKERS) {
71 + if (body.includes(marker)) {
72 + signals.push({ severity: "decisive", reason: `challenge:${marker}` });
73 + break;
74 + }
75 + }
76 +
77 + if (TITLE_RE.test(body)) {
78 + signals.push({ severity: "decisive", reason: "challenge-title" });
79 + }
80 +
81 + if (textHtmlRatio(body) < 0.05) {
82 + signals.push({ severity: "strong", reason: "low-text-ratio" });
83 + }
84 +
85 + if (NOSCRIPT_JS_RE.test(body)) {
86 + signals.push({ severity: "strong", reason: "noscript-js-required" });
87 + }
88 +
89 + if (bytes > 10_240 && !HREF_RE.test(body)) {
90 + signals.push({ severity: "moderate", reason: "no-links" });
91 + }
92 +
93 + const decisive = signals.find((s) => s.severity === "decisive");
94 + if (decisive) {
95 + return { action: "escalate", reason: signals.map((s) => s.reason).join(",") };
96 + }
97 +
98 + const weak = signals.filter((s) => s.severity === "strong" || s.severity === "moderate");
99 + if (weak.length >= 2) {
100 + return { action: "escalate", reason: weak.map((s) => s.reason).join(",") };
101 + }
102 +
103 + return { action: "proceed" };
104 +}
added packages/router/src/index.ts +2 −0
@@ -0,0 +1,2 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +export { shouldEscalate, type EscalationInput, type TierDecision } from "./escalate.js";
added packages/router/tsconfig.json +19 −0
@@ -0,0 +1,19 @@
1 +{
2 + "extends": "../../tsconfig.base.json",
3 + "compilerOptions": {
4 + "rootDir": "src",
5 + "outDir": "dist",
6 + "noEmit": false
7 + },
8 + "references": [
9 + {
10 + "path": "../shared"
11 + }
12 + ],
13 + "include": [
14 + "src/**/*.ts"
15 + ],
16 + "exclude": [
17 + "src/**/*.test.ts"
18 + ]
19 +}
added packages/shared/package.json +14 −0
@@ -0,0 +1,14 @@
1 +{
2 + "name": "@tendril/shared",
3 + "version": "0.1.0",
4 + "private": true,
5 + "type": "module",
6 + "main": "./src/index.ts",
7 + "types": "./src/index.ts",
8 + "exports": {
9 + ".": "./src/index.ts"
10 + },
11 + "dependencies": {
12 + "pino": "^9.5.0"
13 + }
14 +}
added packages/shared/src/errors.test.ts +44 −0
@@ -0,0 +1,44 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { describe, expect, it } from "vitest";
3 +import { ERROR_TAXONOMY, httpStatusFor, isRetryable, tendrilError } from "./errors.js";
4 +
5 +describe("error taxonomy", () => {
6 + it("maps every code to a valid HTTP status", () => {
7 + for (const [code, spec] of Object.entries(ERROR_TAXONOMY)) {
8 + expect(spec.http, code).toBeGreaterThanOrEqual(400);
9 + expect(spec.http, code).toBeLessThan(600);
10 + expect(spec.message.length, code).toBeGreaterThan(0);
11 + }
12 + });
13 +
14 + it("marks non-retryable terminal codes correctly", () => {
15 + expect(isRetryable("ERR_ROBOTS_DENIED")).toBe(false);
16 + expect(isRetryable("ERR_SSRF_BLOCKED")).toBe(false);
17 + expect(isRetryable("ERR_UNSUPPORTED_TYPE")).toBe(false);
18 + expect(isRetryable("ERR_TARGET_4XX")).toBe(false);
19 + });
20 +
21 + it("marks transient codes as retryable", () => {
22 + expect(isRetryable("ERR_TIER_TIMEOUT")).toBe(true);
23 + expect(isRetryable("ERR_POOL_EXHAUSTED")).toBe(true);
24 + expect(isRetryable("ERR_DAEMON_DOWN")).toBe(true);
25 + });
26 +
27 + it("builds errors with default and custom messages", () => {
28 + expect(tendrilError("ERR_INVALID_URL").message).toBe(ERROR_TAXONOMY.ERR_INVALID_URL.message);
29 + const e = tendrilError("ERR_TOO_LARGE", { message: "custom", details: { size: 42 } });
30 + expect(e.message).toBe("custom");
31 + expect(e.details).toEqual({ size: 42 });
32 + });
33 +
34 + it("omits optional fields when not provided (exactOptionalPropertyTypes)", () => {
35 + const e = tendrilError("ERR_INTERNAL");
36 + expect("details" in e).toBe(false);
37 + expect("cause" in e).toBe(false);
38 + });
39 +
40 + it("exposes HTTP status via helper", () => {
41 + expect(httpStatusFor("ERR_UNAUTHORIZED")).toBe(401);
42 + expect(httpStatusFor("ERR_RATE_LIMITED")).toBe(429);
43 + });
44 +});
added packages/shared/src/errors.ts +86 −0
@@ -0,0 +1,86 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +export type ErrorCode =
3 + | "ERR_INVALID_URL"
4 + | "ERR_SSRF_BLOCKED"
5 + | "ERR_UNSUPPORTED_TYPE"
6 + | "ERR_TOO_LARGE"
7 + | "ERR_UNAUTHORIZED"
8 + | "ERR_QUOTA_EXCEEDED"
9 + | "ERR_RATE_LIMITED"
10 + | "ERR_ROBOTS_DENIED"
11 + | "ERR_TARGET_BLOCKED"
12 + | "ERR_TARGET_4XX"
13 + | "ERR_TARGET_5XX"
14 + | "ERR_TIER_TIMEOUT"
15 + | "ERR_POOL_EXHAUSTED"
16 + | "ERR_DAEMON_DOWN"
17 + | "ERR_SAFARI_BUSY"
18 + | "ERR_PROFILE_EXPIRED"
19 + | "ERR_EXTRACT_FAILED"
20 + | "ERR_REDIRECT_LOOP"
21 + | "ERR_INTERNAL";
22 +
23 +export interface ErrorSpec {
24 + readonly http: number;
25 + readonly retryable: boolean;
26 + readonly message: string;
27 +}
28 +
29 +/**
30 + * The single source of truth for the §15 error taxonomy: code → HTTP status,
31 + * retry-ability, and a default human message. The API layer reads `http`; the
32 + * queue layer reads `retryable`. Do not duplicate this mapping anywhere else.
33 + */
34 +export const ERROR_TAXONOMY: Readonly<Record<ErrorCode, ErrorSpec>> = {
35 + ERR_INVALID_URL: { http: 400, retryable: false, message: "Malformed or unsupported scheme" },
36 + ERR_SSRF_BLOCKED: { http: 400, retryable: false, message: "Resolved to a private or forbidden address" },
37 + ERR_UNSUPPORTED_TYPE: { http: 415, retryable: false, message: "Content type not handled" },
38 + ERR_TOO_LARGE: { http: 413, retryable: false, message: "Exceeded maxSizeBytes" },
39 + ERR_UNAUTHORIZED: { http: 401, retryable: false, message: "Bad or revoked key" },
40 + ERR_QUOTA_EXCEEDED: { http: 402, retryable: false, message: "Plan limit hit" },
41 + ERR_RATE_LIMITED: { http: 429, retryable: true, message: "Rate limited" },
42 + ERR_ROBOTS_DENIED: { http: 403, retryable: false, message: "robots.txt disallows" },
43 + ERR_TARGET_BLOCKED: { http: 502, retryable: true, message: "Target returned a challenge at every tier" },
44 + ERR_TARGET_4XX: { http: 502, retryable: false, message: "Upstream 4xx status" },
45 + ERR_TARGET_5XX: { http: 502, retryable: true, message: "Upstream 5xx status" },
46 + ERR_TIER_TIMEOUT: { http: 504, retryable: true, message: "Render exceeded timeout" },
47 + ERR_POOL_EXHAUSTED: { http: 503, retryable: true, message: "No WebView available in 30s" },
48 + ERR_DAEMON_DOWN: { http: 503, retryable: true, message: "Swift daemon unreachable, circuit open" },
49 + ERR_SAFARI_BUSY: { http: 503, retryable: true, message: "Tier 2 session conflict" },
50 + ERR_PROFILE_EXPIRED: { http: 409, retryable: false, message: "Session profile needs re-auth" },
51 + ERR_EXTRACT_FAILED: { http: 422, retryable: false, message: "Schema produced no fields" },
52 + ERR_REDIRECT_LOOP: { http: 502, retryable: false, message: "Same URL twice in a redirect chain" },
53 + ERR_INTERNAL: { http: 500, retryable: true, message: "Internal error" },
54 +};
55 +
56 +export interface TendrilError {
57 + readonly code: ErrorCode;
58 + readonly message: string;
59 + readonly details?: Readonly<Record<string, unknown>>;
60 + readonly cause?: unknown;
61 +}
62 +
63 +export function tendrilError(
64 + code: ErrorCode,
65 + overrides?: { message?: string; details?: Record<string, unknown>; cause?: unknown },
66 +): TendrilError {
67 + const base: TendrilError = {
68 + code,
69 + message: overrides?.message ?? ERROR_TAXONOMY[code].message,
70 + };
71 + const details = overrides?.details;
72 + const cause = overrides?.cause;
73 + return {
74 + ...base,
75 + ...(details !== undefined ? { details } : {}),
76 + ...(cause !== undefined ? { cause } : {}),
77 + };
78 +}
79 +
80 +export function httpStatusFor(code: ErrorCode): number {
81 + return ERROR_TAXONOMY[code].http;
82 +}
83 +
84 +export function isRetryable(code: ErrorCode): boolean {
85 + return ERROR_TAXONOMY[code].retryable;
86 +}
added packages/shared/src/index.ts +6 −0
@@ -0,0 +1,6 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +export * from "./result.js";
3 +export * from "./errors.js";
4 +export * from "./types.js";
5 +export * from "./url.js";
6 +export { logger, childLogger, type Logger } from "./logger.js";
added packages/shared/src/logger.ts +32 −0
@@ -0,0 +1,32 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { pino, type Logger } from "pino";
3 +
4 +const REDACT_PATHS = [
5 + "req.headers.authorization",
6 + "req.headers.cookie",
7 + "res.headers['set-cookie']",
8 + "headers.authorization",
9 + "headers.cookie",
10 + "*.apiKey",
11 + "*.cookies",
12 + "*.cookies_enc",
13 + "*.secret",
14 + "llm.apiKey",
15 +];
16 +
17 +export const logger: Logger = pino({
18 + level: process.env["LOG_LEVEL"] ?? "info",
19 + base: { service: "tendril" },
20 + redact: { paths: REDACT_PATHS, censor: "[redacted]" },
21 + formatters: {
22 + level(label) {
23 + return { level: label };
24 + },
25 + },
26 +});
27 +
28 +export type { Logger };
29 +
30 +export function childLogger(bindings: Record<string, unknown>): Logger {
31 + return logger.child(bindings);
32 +}
added packages/shared/src/result.ts +34 −0
@@ -0,0 +1,34 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import type { TendrilError } from "./errors.js";
3 +
4 +export type Ok<T> = { readonly ok: true; readonly value: T };
5 +export type Err<E> = { readonly ok: false; readonly error: E };
6 +export type Result<T, E = TendrilError> = Ok<T> | Err<E>;
7 +
8 +export function ok<T>(value: T): Ok<T> {
9 + return { ok: true, value };
10 +}
11 +
12 +export function err<E>(error: E): Err<E> {
13 + return { ok: false, error };
14 +}
15 +
16 +export function isOk<T, E>(r: Result<T, E>): r is Ok<T> {
17 + return r.ok;
18 +}
19 +
20 +export function isErr<T, E>(r: Result<T, E>): r is Err<E> {
21 + return !r.ok;
22 +}
23 +
24 +export function map<T, U, E>(r: Result<T, E>, f: (value: T) => U): Result<U, E> {
25 + return r.ok ? ok(f(r.value)) : r;
26 +}
27 +
28 +export function mapErr<T, E, F>(r: Result<T, E>, f: (error: E) => F): Result<T, F> {
29 + return r.ok ? r : err(f(r.error));
30 +}
31 +
32 +export function unwrapOr<T, E>(r: Result<T, E>, fallback: T): T {
33 + return r.ok ? r.value : fallback;
34 +}
added packages/shared/src/types.ts +54 −0
@@ -0,0 +1,54 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +export type Tier = "http" | "webkit" | "safari";
3 +export type TierRequest = Tier | "auto";
4 +
5 +export type OutputFormat =
6 + | "markdown"
7 + | "html"
8 + | "rawHtml"
9 + | "links"
10 + | "screenshot"
11 + | "structured"
12 + | "extract";
13 +
14 +export interface RedirectHop {
15 + readonly from: string;
16 + readonly to: string;
17 + readonly status: number;
18 +}
19 +
20 +export interface FetchTimings {
21 + total: number;
22 + dns?: number;
23 + connect?: number;
24 + ttfb?: number;
25 + download?: number;
26 + render?: number;
27 + extract?: number;
28 + escalations: EscalationRecord[];
29 +}
30 +
31 +export interface EscalationRecord {
32 + readonly from: Tier;
33 + readonly to: Tier;
34 + readonly reason: string;
35 +}
36 +
37 +/** Raw output of a fetcher tier, before extraction. */
38 +export interface FetchResult {
39 + readonly tier: Tier;
40 + readonly status: number;
41 + readonly finalUrl: string;
42 + readonly contentType: string;
43 + readonly body: string;
44 + readonly bodyBytes: number;
45 + readonly redirects: readonly RedirectHop[];
46 + readonly headers: Readonly<Record<string, string>>;
47 +}
48 +
49 +export interface PageLink {
50 + readonly url: string;
51 + readonly text: string;
52 + readonly rel: string | null;
53 + readonly isInternal: boolean;
54 +}
added packages/shared/src/url.test.ts +69 −0
@@ -0,0 +1,69 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { describe, expect, it } from "vitest";
3 +import { normalizeUrl } from "./url.js";
4 +
5 +function norm(raw: string, base?: string): string {
6 + const r = normalizeUrl(raw, base);
7 + if (!r.ok) throw new Error(`expected ok, got ${r.error.code}`);
8 + return r.value;
9 +}
10 +
11 +describe("normalizeUrl", () => {
12 + it("lowercases the host but not the path", () => {
13 + expect(norm("https://EXAMPLE.com/Path")).toBe("https://example.com/Path");
14 + });
15 +
16 + it("strips default ports", () => {
17 + expect(norm("http://example.com:80/a")).toBe("http://example.com/a");
18 + expect(norm("https://example.com:443/a")).toBe("https://example.com/a");
19 + });
20 +
21 + it("keeps non-default ports", () => {
22 + expect(norm("https://example.com:8443/a")).toBe("https://example.com:8443/a");
23 + });
24 +
25 + it("removes tracking params but keeps real ones", () => {
26 + expect(norm("https://e.com/?utm_source=x&q=1&fbclid=z&_ga=2")).toBe("https://e.com/?q=1");
27 + });
28 +
29 + it("sorts query params deterministically", () => {
30 + expect(norm("https://e.com/?b=2&a=1&a=0")).toBe("https://e.com/?a=0&a=1&b=2");
31 + });
32 +
33 + it("drops the fragment unless hashbang", () => {
34 + expect(norm("https://e.com/p#section")).toBe("https://e.com/p");
35 + expect(norm("https://e.com/p#!/route")).toBe("https://e.com/p#!/route");
36 + });
37 +
38 + it("removes trailing slash except at root", () => {
39 + expect(norm("https://e.com/foo/")).toBe("https://e.com/foo");
40 + expect(norm("https://e.com/")).toBe("https://e.com/");
41 + });
42 +
43 + it("resolves dot segments", () => {
44 + expect(norm("https://e.com/a/b/../c")).toBe("https://e.com/a/c");
45 + });
46 +
47 + it("punycodes international hosts", () => {
48 + expect(norm("https://münchen.de/")).toBe("https://xn--mnchen-3ya.de/");
49 + });
50 +
51 + it("resolves relative URLs against a base", () => {
52 + expect(norm("../x", "https://e.com/a/b")).toBe("https://e.com/x");
53 + });
54 +
55 + it("rejects non-http schemes", () => {
56 + const r = normalizeUrl("ftp://e.com/x");
57 + expect(r.ok).toBe(false);
58 + if (!r.ok) expect(r.error.code).toBe("ERR_INVALID_URL");
59 + });
60 +
61 + it("rejects malformed input", () => {
62 + const r = normalizeUrl("not a url");
63 + expect(r.ok).toBe(false);
64 + });
65 +
66 + it("produces identical output for equivalent URLs", () => {
67 + expect(norm("https://E.com:443/a/../b/?utm_x=1&z=9#frag")).toBe(norm("https://e.com/b?z=9"));
68 + });
69 +});
added packages/shared/src/url.ts +71 −0
@@ -0,0 +1,71 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { err, ok, type Result } from "./result.js";
3 +import { tendrilError } from "./errors.js";
4 +
5 +const DEFAULT_PORTS: Readonly<Record<string, string>> = {
6 + "http:": "80",
7 + "https:": "443",
8 +};
9 +
10 +const TRACKING_PARAMS = new Set(["fbclid", "gclid", "gclsrc", "ref", "mc_cid", "mc_eid", "_ga", "_gl"]);
11 +
12 +function isTrackingParam(key: string): boolean {
13 + const lower = key.toLowerCase();
14 + return lower.startsWith("utm_") || TRACKING_PARAMS.has(lower);
15 +}
16 +
17 +/**
18 + * Canonicalize a URL for dedup and cache keys (§10 rule 1, §10.1). The result is
19 + * intended for comparison only — keep the original string for output. Returns
20 + * ERR_INVALID_URL for anything that is not an absolute http(s) URL.
21 + */
22 +export function normalizeUrl(raw: string, base?: string): Result<string> {
23 + let u: URL;
24 + try {
25 + u = base !== undefined ? new URL(raw, base) : new URL(raw);
26 + } catch {
27 + return err(tendrilError("ERR_INVALID_URL", { details: { url: raw } }));
28 + }
29 +
30 + if (u.protocol !== "http:" && u.protocol !== "https:") {
31 + return err(tendrilError("ERR_INVALID_URL", { details: { url: raw, scheme: u.protocol } }));
32 + }
33 +
34 + u.hostname = u.hostname.toLowerCase();
35 +
36 + if (u.port !== "" && DEFAULT_PORTS[u.protocol] === u.port) {
37 + u.port = "";
38 + }
39 +
40 + const kept: Array<[string, string]> = [];
41 + for (const [key, value] of u.searchParams.entries()) {
42 + if (!isTrackingParam(key)) kept.push([key, value]);
43 + }
44 + kept.sort((a, b) => (a[0] === b[0] ? (a[1] < b[1] ? -1 : a[1] > b[1] ? 1 : 0) : a[0] < b[0] ? -1 : 1));
45 + u.search = "";
46 + for (const [key, value] of kept) u.searchParams.append(key, value);
47 +
48 + const hashbang = u.hash.startsWith("#!");
49 + if (!hashbang) u.hash = "";
50 +
51 + if (u.pathname.length > 1 && u.pathname.endsWith("/")) {
52 + u.pathname = u.pathname.replace(/\/+$/, "");
53 + if (u.pathname === "") u.pathname = "/";
54 + }
55 +
56 + return ok(u.toString());
57 +}
58 +
59 +export interface ParsedUrl {
60 + readonly url: URL;
61 + readonly host: string;
62 +}
63 +
64 +export function parseHost(raw: string): Result<ParsedUrl> {
65 + try {
66 + const url = new URL(raw);
67 + return ok({ url, host: url.hostname.toLowerCase() });
68 + } catch {
69 + return err(tendrilError("ERR_INVALID_URL", { details: { url: raw } }));
70 + }
71 +}
added packages/shared/tsconfig.json +14 −0
@@ -0,0 +1,14 @@
1 +{
2 + "extends": "../../tsconfig.base.json",
3 + "compilerOptions": {
4 + "rootDir": "src",
5 + "outDir": "dist",
6 + "noEmit": false
7 + },
8 + "include": [
9 + "src/**/*.ts"
10 + ],
11 + "exclude": [
12 + "src/**/*.test.ts"
13 + ]
14 +}
added pnpm-lock.yaml +1879 −0
@@ -0,0 +1,1879 @@
1 +lockfileVersion: '9.0'
2 +
3 +settings:
4 + autoInstallPeers: true
5 + excludeLinksFromLockfile: false
6 +
7 +importers:
8 +
9 + .:
10 + dependencies:
11 + '@mozilla/readability':
12 + specifier: ^0.5.0
13 + version: 0.5.0
14 + fastify:
15 + specifier: ^5.2.0
16 + version: 5.11.3
17 + linkedom:
18 + specifier: ^0.18.6
19 + version: 0.18.13
20 + pino:
21 + specifier: ^9.5.0
22 + version: 9.14.0
23 + turndown:
24 + specifier: ^7.2.0
25 + version: 7.2.4
26 + undici:
27 + specifier: ^7.2.0
28 + version: 7.29.0
29 + zod:
30 + specifier: ^3.24.1
31 + version: 3.25.76
32 + devDependencies:
33 + '@types/node':
34 + specifier: ^22.10.0
35 + version: 22.20.1
36 + '@types/turndown':
37 + specifier: ^5.0.5
38 + version: 5.0.6
39 + tsx:
40 + specifier: ^4.19.2
41 + version: 4.23.11
42 + typescript:
43 + specifier: ^5.7.2
44 + version: 5.9.3
45 + vite-tsconfig-paths:
46 + specifier: ^5.1.4
47 + version: 5.1.4(typescript@5.9.3)(vite@5.4.21(@types/node@22.20.1))
48 + vitest:
49 + specifier: ^2.1.8
50 + version: 2.1.9(@types/node@22.20.1)
51 +
52 + apps/api:
53 + dependencies:
54 + '@tendril/egress':
55 + specifier: workspace:*
56 + version: link:../../packages/egress
57 + '@tendril/extract':
58 + specifier: workspace:*
59 + version: link:../../packages/extract
60 + '@tendril/fetcher-http':
61 + specifier: workspace:*
62 + version: link:../../packages/fetcher-http
63 + '@tendril/router':
64 + specifier: workspace:*
65 + version: link:../../packages/router
66 + '@tendril/shared':
67 + specifier: workspace:*
68 + version: link:../../packages/shared
69 + fastify:
70 + specifier: ^5.2.0
71 + version: 5.11.3
72 + zod:
73 + specifier: ^3.24.1
74 + version: 3.25.76
75 +
76 + packages/egress:
77 + dependencies:
78 + '@tendril/shared':
79 + specifier: workspace:*
80 + version: link:../shared
81 +
82 + packages/extract:
83 + dependencies:
84 + '@mozilla/readability':
85 + specifier: ^0.5.0
86 + version: 0.5.0
87 + '@tendril/shared':
88 + specifier: workspace:*
89 + version: link:../shared
90 + linkedom:
91 + specifier: ^0.18.6
92 + version: 0.18.13
93 + turndown:
94 + specifier: ^7.2.0
95 + version: 7.2.4
96 +
97 + packages/fetcher-http:
98 + dependencies:
99 + '@tendril/egress':
100 + specifier: workspace:*
101 + version: link:../egress
102 + '@tendril/router':
103 + specifier: workspace:*
104 + version: link:../router
105 + '@tendril/shared':
106 + specifier: workspace:*
107 + version: link:../shared
108 + undici:
109 + specifier: ^7.2.0
110 + version: 7.29.0
111 +
112 + packages/router:
113 + dependencies:
114 + '@tendril/shared':
115 + specifier: workspace:*
116 + version: link:../shared
117 +
118 + packages/shared:
119 + dependencies:
120 + pino:
121 + specifier: ^9.5.0
122 + version: 9.14.0
123 +
124 +packages:
125 +
126 + '@esbuild/aix-ppc64@0.21.5':
127 + resolution: {integrity: sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==}
128 + engines: {node: '>=12'}
129 + cpu: [ppc64]
130 + os: [aix]
131 +
132 + '@esbuild/aix-ppc64@0.28.2':
133 + resolution: {integrity: sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==}
134 + engines: {node: '>=18'}
135 + cpu: [ppc64]
136 + os: [aix]
137 +
138 + '@esbuild/android-arm64@0.21.5':
139 + resolution: {integrity: sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==}
140 + engines: {node: '>=12'}
141 + cpu: [arm64]
142 + os: [android]
143 +
144 + '@esbuild/android-arm64@0.28.2':
145 + resolution: {integrity: sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==}
146 + engines: {node: '>=18'}
147 + cpu: [arm64]
148 + os: [android]
149 +
150 + '@esbuild/android-arm@0.21.5':
151 + resolution: {integrity: sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==}
152 + engines: {node: '>=12'}
153 + cpu: [arm]
154 + os: [android]
155 +
156 + '@esbuild/android-arm@0.28.2':
157 + resolution: {integrity: sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==}
158 + engines: {node: '>=18'}
159 + cpu: [arm]
160 + os: [android]
161 +
162 + '@esbuild/android-x64@0.21.5':
163 + resolution: {integrity: sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==}
164 + engines: {node: '>=12'}
165 + cpu: [x64]
166 + os: [android]
167 +
168 + '@esbuild/android-x64@0.28.2':
169 + resolution: {integrity: sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==}
170 + engines: {node: '>=18'}
171 + cpu: [x64]
172 + os: [android]
173 +
174 + '@esbuild/darwin-arm64@0.21.5':
175 + resolution: {integrity: sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==}
176 + engines: {node: '>=12'}
177 + cpu: [arm64]
178 + os: [darwin]
179 +
180 + '@esbuild/darwin-arm64@0.28.2':
181 + resolution: {integrity: sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==}
182 + engines: {node: '>=18'}
183 + cpu: [arm64]
184 + os: [darwin]
185 +
186 + '@esbuild/darwin-x64@0.21.5':
187 + resolution: {integrity: sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==}
188 + engines: {node: '>=12'}
189 + cpu: [x64]
190 + os: [darwin]
191 +
192 + '@esbuild/darwin-x64@0.28.2':
193 + resolution: {integrity: sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==}
194 + engines: {node: '>=18'}
195 + cpu: [x64]
196 + os: [darwin]
197 +
198 + '@esbuild/freebsd-arm64@0.21.5':
199 + resolution: {integrity: sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==}
200 + engines: {node: '>=12'}
201 + cpu: [arm64]
202 + os: [freebsd]
203 +
204 + '@esbuild/freebsd-arm64@0.28.2':
205 + resolution: {integrity: sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==}
206 + engines: {node: '>=18'}
207 + cpu: [arm64]
208 + os: [freebsd]
209 +
210 + '@esbuild/freebsd-x64@0.21.5':
211 + resolution: {integrity: sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==}
212 + engines: {node: '>=12'}
213 + cpu: [x64]
214 + os: [freebsd]
215 +
216 + '@esbuild/freebsd-x64@0.28.2':
217 + resolution: {integrity: sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==}
218 + engines: {node: '>=18'}
219 + cpu: [x64]
220 + os: [freebsd]
221 +
222 + '@esbuild/linux-arm64@0.21.5':
223 + resolution: {integrity: sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==}
224 + engines: {node: '>=12'}
225 + cpu: [arm64]
226 + os: [linux]
227 +
228 + '@esbuild/linux-arm64@0.28.2':
229 + resolution: {integrity: sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==}
230 + engines: {node: '>=18'}
231 + cpu: [arm64]
232 + os: [linux]
233 +
234 + '@esbuild/linux-arm@0.21.5':
235 + resolution: {integrity: sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==}
236 + engines: {node: '>=12'}
237 + cpu: [arm]
238 + os: [linux]
239 +
240 + '@esbuild/linux-arm@0.28.2':
241 + resolution: {integrity: sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==}
242 + engines: {node: '>=18'}
243 + cpu: [arm]
244 + os: [linux]
245 +
246 + '@esbuild/linux-ia32@0.21.5':
247 + resolution: {integrity: sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==}
248 + engines: {node: '>=12'}
249 + cpu: [ia32]
250 + os: [linux]
251 +
252 + '@esbuild/linux-ia32@0.28.2':
253 + resolution: {integrity: sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==}
254 + engines: {node: '>=18'}
255 + cpu: [ia32]
256 + os: [linux]
257 +
258 + '@esbuild/linux-loong64@0.21.5':
259 + resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==}
260 + engines: {node: '>=12'}
261 + cpu: [loong64]
262 + os: [linux]
263 +
264 + '@esbuild/linux-loong64@0.28.2':
265 + resolution: {integrity: sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==}
266 + engines: {node: '>=18'}
267 + cpu: [loong64]
268 + os: [linux]
269 +
270 + '@esbuild/linux-mips64el@0.21.5':
271 + resolution: {integrity: sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==}
272 + engines: {node: '>=12'}
273 + cpu: [mips64el]
274 + os: [linux]
275 +
276 + '@esbuild/linux-mips64el@0.28.2':
277 + resolution: {integrity: sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==}
278 + engines: {node: '>=18'}
279 + cpu: [mips64el]
280 + os: [linux]
281 +
282 + '@esbuild/linux-ppc64@0.21.5':
283 + resolution: {integrity: sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==}
284 + engines: {node: '>=12'}
285 + cpu: [ppc64]
286 + os: [linux]
287 +
288 + '@esbuild/linux-ppc64@0.28.2':
289 + resolution: {integrity: sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==}
290 + engines: {node: '>=18'}
291 + cpu: [ppc64]
292 + os: [linux]
293 +
294 + '@esbuild/linux-riscv64@0.21.5':
295 + resolution: {integrity: sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==}
296 + engines: {node: '>=12'}
297 + cpu: [riscv64]
298 + os: [linux]
299 +
300 + '@esbuild/linux-riscv64@0.28.2':
301 + resolution: {integrity: sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==}
302 + engines: {node: '>=18'}
303 + cpu: [riscv64]
304 + os: [linux]
305 +
306 + '@esbuild/linux-s390x@0.21.5':
307 + resolution: {integrity: sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==}
308 + engines: {node: '>=12'}
309 + cpu: [s390x]
310 + os: [linux]
311 +
312 + '@esbuild/linux-s390x@0.28.2':
313 + resolution: {integrity: sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==}
314 + engines: {node: '>=18'}
315 + cpu: [s390x]
316 + os: [linux]
317 +
318 + '@esbuild/linux-x64@0.21.5':
319 + resolution: {integrity: sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==}
320 + engines: {node: '>=12'}
321 + cpu: [x64]
322 + os: [linux]
323 +
324 + '@esbuild/linux-x64@0.28.2':
325 + resolution: {integrity: sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==}
326 + engines: {node: '>=18'}
327 + cpu: [x64]
328 + os: [linux]
329 +
330 + '@esbuild/netbsd-arm64@0.28.2':
331 + resolution: {integrity: sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==}
332 + engines: {node: '>=18'}
333 + cpu: [arm64]
334 + os: [netbsd]
335 +
336 + '@esbuild/netbsd-x64@0.21.5':
337 + resolution: {integrity: sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==}
338 + engines: {node: '>=12'}
339 + cpu: [x64]
340 + os: [netbsd]
341 +
342 + '@esbuild/netbsd-x64@0.28.2':
343 + resolution: {integrity: sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==}
344 + engines: {node: '>=18'}
345 + cpu: [x64]
346 + os: [netbsd]
347 +
348 + '@esbuild/openbsd-arm64@0.28.2':
349 + resolution: {integrity: sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==}
350 + engines: {node: '>=18'}
351 + cpu: [arm64]
352 + os: [openbsd]
353 +
354 + '@esbuild/openbsd-x64@0.21.5':
355 + resolution: {integrity: sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==}
356 + engines: {node: '>=12'}
357 + cpu: [x64]
358 + os: [openbsd]
359 +
360 + '@esbuild/openbsd-x64@0.28.2':
361 + resolution: {integrity: sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==}
362 + engines: {node: '>=18'}
363 + cpu: [x64]
364 + os: [openbsd]
365 +
366 + '@esbuild/openharmony-arm64@0.28.2':
367 + resolution: {integrity: sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==}
368 + engines: {node: '>=18'}
369 + cpu: [arm64]
370 + os: [openharmony]
371 +
372 + '@esbuild/sunos-x64@0.21.5':
373 + resolution: {integrity: sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==}
374 + engines: {node: '>=12'}
375 + cpu: [x64]
376 + os: [sunos]
377 +
378 + '@esbuild/sunos-x64@0.28.2':
379 + resolution: {integrity: sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==}
380 + engines: {node: '>=18'}
381 + cpu: [x64]
382 + os: [sunos]
383 +
384 + '@esbuild/win32-arm64@0.21.5':
385 + resolution: {integrity: sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==}
386 + engines: {node: '>=12'}
387 + cpu: [arm64]
388 + os: [win32]
389 +
390 + '@esbuild/win32-arm64@0.28.2':
391 + resolution: {integrity: sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==}
392 + engines: {node: '>=18'}
393 + cpu: [arm64]
394 + os: [win32]
395 +
396 + '@esbuild/win32-ia32@0.21.5':
397 + resolution: {integrity: sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==}
398 + engines: {node: '>=12'}
399 + cpu: [ia32]
400 + os: [win32]
401 +
402 + '@esbuild/win32-ia32@0.28.2':
403 + resolution: {integrity: sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==}
404 + engines: {node: '>=18'}
405 + cpu: [ia32]
406 + os: [win32]
407 +
408 + '@esbuild/win32-x64@0.21.5':
409 + resolution: {integrity: sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==}
410 + engines: {node: '>=12'}
411 + cpu: [x64]
412 + os: [win32]
413 +
414 + '@esbuild/win32-x64@0.28.2':
415 + resolution: {integrity: sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==}
416 + engines: {node: '>=18'}
417 + cpu: [x64]
418 + os: [win32]
419 +
420 + '@fastify/ajv-compiler@4.0.6':
421 + resolution: {integrity: sha512-NtuzM0SfaMJbGlnjr9LWQUN5LzgSrbB8tf/wRZNas+4E1O/Nmzl53e7ruT61HDZyRCJGC6FxIogmNZO1c5ETBA==}
422 +
423 + '@fastify/error@4.2.0':
424 + resolution: {integrity: sha512-RSo3sVDXfHskiBZKBPRgnQTtIqpi/7zhJOEmAxCiBcM7d0uwdGdxLlsCaLzGs8v8NnxIRlfG0N51p5yFaOentQ==}
425 +
426 + '@fastify/fast-json-stringify-compiler@5.1.0':
427 + resolution: {integrity: sha512-PxcYtKLbQ8Z+yApiqjK8FwxIwvEj38k2OiLc17u8dkJSlmfi2wHHPaSnaoqBPQqtvF8YVsDgDpP2snDCfFrpfw==}
428 +
429 + '@fastify/forwarded@3.0.2':
430 + resolution: {integrity: sha512-NE8HgKLgYejV9lDpqkEFaDKMLYelJBVfHekhB0UKvX0ghagXRJqg68feg8er1NPXxG4N9i6vPxzt8E+3wHfcmA==}
431 +
432 + '@fastify/merge-json-schemas@0.2.1':
433 + resolution: {integrity: sha512-OA3KGBCy6KtIvLf8DINC5880o5iBlDX4SxzLQS8HorJAbqluzLRn80UXU0bxZn7UOFhFgpRJDasfwn9nG4FG4A==}
434 +
435 + '@fastify/proxy-addr@5.1.0':
436 + resolution: {integrity: sha512-INS+6gh91cLUjB+PVHfu1UqcB76Sqtpyp7bnL+FYojhjygvOPA9ctiD/JDKsyD9Xgu4hUhCSJBPig/w7duNajw==}
437 +
438 + '@jridgewell/sourcemap-codec@1.5.5':
439 + resolution: {integrity: sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og==}
440 +
441 + '@mixmark-io/domino@2.2.0':
442 + resolution: {integrity: sha512-Y28PR25bHXUg88kCV7nivXrP2Nj2RueZ3/l/jdx6J9f8J4nsEGcgX0Qe6lt7Pa+J79+kPiJU3LguR6O/6zrLOw==}
443 +
444 + '@mozilla/readability@0.5.0':
445 + resolution: {integrity: sha512-Z+CZ3QaosfFaTqvhQsIktyGrjFjSC0Fa4EMph4mqKnWhmyoGICsV/8QK+8HpXut6zV7zwfWwqDmEjtk1Qf6EgQ==}
446 + engines: {node: '>=14.0.0'}
447 +
448 + '@napi-rs/lzma-linux-x64-gnu@1.5.1':
449 + resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==}
450 + engines: {node: ^22.20 || ^24.12 || >=25}
451 + cpu: [x64]
452 + os: [linux]
453 + libc: [glibc]
454 +
455 + '@pinojs/redact@0.4.0':
456 + resolution: {integrity: sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg==}
457 +
458 + '@rollup/rollup-android-arm-eabi@4.62.4':
459 + resolution: {integrity: sha512-RrPokAb7dmbxFoeO3TloqHyOjgye8RkBhSqmp4aJMIex4c9r46ZstPnleDQOq1t46VOVjwIuwNogIqbodV1Vvg==}
460 + cpu: [arm]
461 + os: [android]
462 +
463 + '@rollup/rollup-android-arm64@4.62.4':
464 + resolution: {integrity: sha512-JKuJc+pnpks2pjy7L/N3v/cAkZxYlnmuZoD840ldbMI5KDbC4iO9NKwPKYdjYFCMAIIlBzYSFHxIJVYzRo2/8A==}
465 + cpu: [arm64]
466 + os: [android]
467 +
468 + '@rollup/rollup-darwin-arm64@4.62.4':
469 + resolution: {integrity: sha512-krw5uS2STmvJ02x0uTXHbqQNuz+9eZ1iw+qXk9dmW2gvV4jV7O2hEoOnuhFrpOPiel1mBFtqbxYZZtC46hXLOw==}
470 + cpu: [arm64]
471 + os: [darwin]
472 +
473 + '@rollup/rollup-darwin-x64@4.62.4':
474 + resolution: {integrity: sha512-wsTxtgApb4PrOsNJIm0FZ1h3WvCC+k9uxLJ4ad75hgoS4NiRes2SoJFlDAyMwiUY8IssDqGcHbXuN0sx1tfF1A==}
475 + cpu: [x64]
476 + os: [darwin]
477 +
478 + '@rollup/rollup-freebsd-arm64@4.62.4':
479 + resolution: {integrity: sha512-GUOnQlyZe3yAXhWOtOMsn5Qkrv5E5mZXa0thbARWi5Ei2szlVXJFQhddZ4HbAzh8q92w5twp+CQvs/eFanz9YQ==}
480 + cpu: [arm64]
481 + os: [freebsd]
482 +
483 + '@rollup/rollup-freebsd-x64@4.62.4':
484 + resolution: {integrity: sha512-/Y7f3QuxjzPKsjA/rfEDa3+0vXqyjmJ50Ln8dPpCmWkKTrUoWHG1cWhTqaAMLob2m2nESWuC7yGrREz019Ztqg==}
485 + cpu: [x64]
486 + os: [freebsd]
487 +
488 + '@rollup/rollup-linux-arm-gnueabihf@4.62.4':
489 + resolution: {integrity: sha512-81wiiX3v7aqy+T+bT61TJ78yJjRquqFFTTbAPt08imfQQzkPIW8t6aJbkTagtCCrXMNc9D66+geqlK7ydLPNqA==}
490 + cpu: [arm]
491 + os: [linux]
492 + libc: [glibc]
493 +
494 + '@rollup/rollup-linux-arm-musleabihf@4.62.4':
495 + resolution: {integrity: sha512-9kmDIvNZqdoHOBZgNtpTBeLWYO/LVipM3H/j62P8848/l/VPEQL6N3uxU9pvP1oZAsXyC2MEnFP3ovRjo7WYNQ==}
496 + cpu: [arm]
497 + os: [linux]
498 + libc: [musl]
499 +
500 + '@rollup/rollup-linux-arm64-gnu@4.62.4':
501 + resolution: {integrity: sha512-CcnXHWnXg69g+DX5VWL3FHts3qMRN2uVEHX+BZvGLdd07/gXkn3ePjYtO1LDJvxkGKVHMclKBRa1QUTH+6toYQ==}
502 + cpu: [arm64]
503 + os: [linux]
504 + libc: [glibc]
505 +
506 + '@rollup/rollup-linux-arm64-musl@4.62.4':
507 + resolution: {integrity: sha512-iFOibiHnTRuhrWLlRsOQFdZJJIa7S8OwkneJr4ocALP16u5yk6lWLINFwhHaEqBFMsKDUZofLkGos7+CPzGB3g==}
508 + cpu: [arm64]
509 + os: [linux]
510 + libc: [musl]
511 +
512 + '@rollup/rollup-linux-loong64-gnu@4.62.4':
513 + resolution: {integrity: sha512-XnWYMI7euHlb5a871xPja+Gm7DRCFU+FGRrtS2sMq9N8FvqtpagUy6gD4YOemC5MRk9xbh8+jYMEJbigFQwsgA==}
514 + cpu: [loong64]
515 + os: [linux]
516 + libc: [glibc]
517 +
518 + '@rollup/rollup-linux-loong64-musl@4.62.4':
519 + resolution: {integrity: sha512-qGDAlO0U8xedCcsdRm9oaoQY8DAx/QT7uIxJWhCdx0ceIWX783UC9QSYkdpzAe29wNiVfp24+bZdQmn49o45SQ==}
520 + cpu: [loong64]
521 + os: [linux]
522 + libc: [musl]
523 +
524 + '@rollup/rollup-linux-ppc64-gnu@4.62.4':
525 + resolution: {integrity: sha512-ru4H6ezD7ysA5EiEK6qkkaEb4modH8CTej6kUy/gQi20u3kB3G7Zn8snXXkeJSCOFKG/rbPPtM/+9Wgas1961w==}
526 + cpu: [ppc64]
527 + os: [linux]
528 + libc: [glibc]
529 +
530 + '@rollup/rollup-linux-ppc64-musl@4.62.4':
531 + resolution: {integrity: sha512-2W4MO5WQVJnbJaZdvDb9rhBDuFU1nKIepPFpJUBsTh2k1YY2g+ODViaWuyOAjQ5cOP7NvrvLzt3wvHOoiAvc7w==}
532 + cpu: [ppc64]
533 + os: [linux]
534 + libc: [musl]
535 +
536 + '@rollup/rollup-linux-riscv64-gnu@4.62.4':
537 + resolution: {integrity: sha512-+fxjfuoAmVMCYV5QyjoIpu0cp5DOiOTeqYFk1AVaxGr+/ravWLX89XfQmptsoWcaVy/TGf2hexzbUOrCQIL1CQ==}
538 + cpu: [riscv64]
539 + os: [linux]
540 + libc: [glibc]
541 +
542 + '@rollup/rollup-linux-riscv64-musl@4.62.4':
543 + resolution: {integrity: sha512-jTn8JfHGL4djjFxPuM06LmNUJDsst2jeVlsd9OmIH6zc5sC9K6rIuO4YajXatLUpBmBKl6b35ro1QZocLi+tcA==}
544 + cpu: [riscv64]
545 + os: [linux]
546 + libc: [musl]
547 +
548 + '@rollup/rollup-linux-s390x-gnu@4.62.4':
549 + resolution: {integrity: sha512-oCJCJL4pXsoDcP2QZ+JVlPTIRc6266zsIaeJJsWImmF7HO0W8nb6HuSgZlMWxJwaPf8ehbSw8yo0EUw925hKsA==}
550 + cpu: [s390x]
551 + os: [linux]
552 + libc: [glibc]
553 +
554 + '@rollup/rollup-linux-x64-gnu@4.62.4':
555 + resolution: {integrity: sha512-W69hukhZ3KKNRCaMIEzKvcFye42hh0FE1+YoYaf5+Ikacuftoco6yO/xouz0hc5d5W/s3yBro5jRiuEE/Q5vUw==}
556 + cpu: [x64]
557 + os: [linux]
558 + libc: [glibc]
559 +
560 + '@rollup/rollup-linux-x64-musl@4.62.4':
561 + resolution: {integrity: sha512-qiXbGG2jkjXhzXpsFZSR2Xpb8DN/UaxYsbb/STbuR/6fpaDgRmmaq1B/LmtF2wQFOFOSsK2jdE0RZ3a0zHn4QA==}
562 + cpu: [x64]
563 + os: [linux]
564 + libc: [musl]
565 +
566 + '@rollup/rollup-openbsd-x64@4.62.4':
567 + resolution: {integrity: sha512-nWeM//hxv8mIo6jD7Hu4o48DVmV9pbV6gsKaWU+4NFyqHoPKwrkRiZGLKUhOBk8qNmDmpwFtPKg80Bo/Tn4xiQ==}
568 + cpu: [x64]
569 + os: [openbsd]
570 +
571 + '@rollup/rollup-openharmony-arm64@4.62.4':
572 + resolution: {integrity: sha512-s62SQ/vgsRSvMwDkOEfTqfgASF0f26ZNaQuTA6Aok5lrikf89yI2W0gFHvZb2Jpgc6N8JnOKZgCK2iciO3CsxQ==}
573 + cpu: [arm64]
574 + os: [openharmony]
575 +
576 + '@rollup/rollup-win32-arm64-msvc@4.62.4':
577 + resolution: {integrity: sha512-J6wGf8TVGbXJq+HH+ttTvrcfNKPbuZecV6KT1B8I18BC5IURUh5kl4Yl5OEP5eFIUoI5BWxCsyYMhFsDx8kekw==}
578 + cpu: [arm64]
579 + os: [win32]
580 +
581 + '@rollup/rollup-win32-ia32-msvc@4.62.4':
582 + resolution: {integrity: sha512-zmfrQd/0wu6oJs8Vq8KwY/YtsKSsLtKe/HwAP4Wqy8LhWjeT55fHRAkOhYQ12wI3ayS4Tt12d5CDRD7N96SAYQ==}
583 + cpu: [ia32]
584 + os: [win32]
585 +
586 + '@rollup/rollup-win32-x64-gnu@4.62.4':
587 + resolution: {integrity: sha512-qPzHqdj9rfUD+w79dtE07zi/kFwKyCJqplp5K5ygeLTp7jLpAoc16OAH39HSmRC9UpozaecsleI8uAdEj6v2yw==}
588 + cpu: [x64]
589 + os: [win32]
590 +
591 + '@rollup/rollup-win32-x64-msvc@4.62.4':
592 + resolution: {integrity: sha512-zD6NdeWEByGE9QF9vCrlJ5YQB4oq9q91kPZS37Jwj5hOkvR1lTBSpsKhKDw4IJtbQ35LsTS1HD9DZYGKIshU1Q==}
593 + cpu: [x64]
594 + os: [win32]
595 +
596 + '@types/estree@1.0.9':
597 + resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==}
598 +
599 + '@types/node@22.20.1':
600 + resolution: {integrity: sha512-EANqOCF9QFyra+4pfxUcX9STKJpCLjMbObVzljIJomAWSnuSIEAvyzEU53GaajbXJEgdh0iEcPL+DGvpUd4k1Q==}
601 +
602 + '@types/turndown@5.0.6':
603 + resolution: {integrity: sha512-ru00MoyeeouE5BX4gRL+6m/BsDfbRayOskWqUvh7CLGW+UXxHQItqALa38kKnOiZPqJrtzJUgAC2+F0rL1S4Pg==}
604 +
605 + '@vitest/expect@2.1.9':
606 + resolution: {integrity: sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==}
607 +
608 + '@vitest/mocker@2.1.9':
609 + resolution: {integrity: sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==}
610 + peerDependencies:
611 + msw: ^2.4.9
612 + vite: ^5.0.0
613 + peerDependenciesMeta:
614 + msw:
615 + optional: true
616 + vite:
617 + optional: true
618 +
619 + '@vitest/pretty-format@2.1.9':
620 + resolution: {integrity: sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==}
621 +
622 + '@vitest/runner@2.1.9':
623 + resolution: {integrity: sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==}
624 +
625 + '@vitest/snapshot@2.1.9':
626 + resolution: {integrity: sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==}
627 +
628 + '@vitest/spy@2.1.9':
629 + resolution: {integrity: sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==}
630 +
631 + '@vitest/utils@2.1.9':
632 + resolution: {integrity: sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==}
633 +
634 + abstract-logging@2.0.1:
635 + resolution: {integrity: sha512-2BjRTZxTPvheOvGbBslFSYOUkr+SjPtOnrLP33f+VIWLzezQpZcqVg7ja3L4dBXmzzgwT+a029jRx5PCi3JuiA==}
636 +
637 + ajv-formats@3.0.1:
638 + resolution: {integrity: sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==}
639 + peerDependencies:
640 + ajv: ^8.0.0
641 + peerDependenciesMeta:
642 + ajv:
643 + optional: true
644 +
645 + ajv@8.20.0:
646 + resolution: {integrity: sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==}
647 +
648 + assertion-error@2.0.1:
649 + resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==}
650 + engines: {node: '>=12'}
651 +
652 + atomic-sleep@1.0.0:
653 + resolution: {integrity: sha512-kNOjDqAh7px0XWNI+4QbzoiR/nTkHAWNud2uvnJquD1/x5a7EQZMJT0AczqK0Qn67oY/TTQ1LbUKajZpp3I9tQ==}
654 + engines: {node: '>=8.0.0'}
655 +
656 + avvio@9.3.0:
657 + resolution: {integrity: sha512-g2tQ7LE7oOSqDfwEm3M+ZCMTJc7KiZCdJ4UwyZJb5ckTKyYu50OYmvv0mCFXPuYXoM4zkSt8zM9XQ9KCvxA74A==}
658 +
659 + boolbase@2.0.0:
660 + resolution: {integrity: sha512-DkVaaQHymRhpYEYo9x1oo7Q7B0Y6KJUsjm3c9eTyFDby4MHLBTwZ6ZDWBel5zrYxj1WsZgC5oLpiz+93MluXeA==}
661 + engines: {node: '>=20.19.0'}
662 +
663 + cac@6.7.14:
664 + resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==}
665 + engines: {node: '>=8'}
666 +
667 + chai@5.3.3:
668 + resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==}
669 + engines: {node: '>=18'}
670 +
671 + check-error@2.1.3:
672 + resolution: {integrity: sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==}
673 + engines: {node: '>= 16'}
674 +
675 + cookie@1.1.1:
676 + resolution: {integrity: sha512-ei8Aos7ja0weRpFzJnEA9UHJ/7XQmqglbRwnf2ATjcB9Wq874VKH9kfjjirM6UhU2/E5fFYadylyhFldcqSidQ==}
677 + engines: {node: '>=18'}
678 +
679 + css-select@7.0.0:
680 + resolution: {integrity: sha512-snmjEVXy+1LnwXdxhYvTMj1d9tOh4HxkA1YmoayVBeeyR2C14Pum7fcxJIm4SswYspVy866eYNwlH6xC3/VH5g==}
681 + engines: {node: '>=20.19.0'}
682 +
683 + css-what@8.0.0:
684 + resolution: {integrity: sha512-DH0Bqq3DNp5tdOReuNyAA+Ev4Y2GS5FMbZpeTLP6C4CDi0h5nL0BmUPChXw3o/qbHLDWHl49sbNqQVY7bMSDdw==}
685 + engines: {node: '>=20.19.0'}
686 +
687 + cssom@0.5.0:
688 + resolution: {integrity: sha512-iKuQcq+NdHqlAcwUY0o/HL69XQrUaQdMjmStJ8JFmUaiiQErlhrmuigkg/CU4E2J0IyUKUrMAgl36TvN67MqTw==}
689 +
690 + debug@4.4.3:
691 + resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==}
692 + engines: {node: '>=6.0'}
693 + peerDependencies:
694 + supports-color: '*'
695 + peerDependenciesMeta:
696 + supports-color:
697 + optional: true
698 +
699 + deep-eql@5.0.2:
700 + resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==}
701 + engines: {node: '>=6'}
702 +
703 + dequal@2.0.3:
704 + resolution: {integrity: sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==}
705 + engines: {node: '>=6'}
706 +
707 + dom-serializer@2.0.0:
708 + resolution: {integrity: sha512-wIkAryiqt/nV5EQKqQpo3SToSOV9J0DnbJqwK7Wv/Trc92zIAYZ4FlMu+JPFW1DfGFt81ZTCGgDEabffXeLyJg==}
709 +
710 + dom-serializer@3.1.1:
711 + resolution: {integrity: sha512-4MEa38/QexBob6gFNwu+EGdWvhJ1OKuNwdYY3Y3NyeWDQfnGeDYQUDfIRzWu5B5gsv03so2Uxd28YC6zrsx3Lw==}
712 + engines: {node: '>=20.19.0'}
713 +
714 + domelementtype@2.3.0:
715 + resolution: {integrity: sha512-OLETBj6w0OsagBwdXnPdN0cnMfF9opN69co+7ZrbfPGrdpPVNBUj02spi6B1N7wChLQiPn4CSH/zJvXw56gmHw==}
716 +
717 + domelementtype@3.0.0:
718 + resolution: {integrity: sha512-umCQid3jKbDmVjx8jGaW7uUykm4DEUeyV21hPxNMo2nV955DhUThwqyOIDtreepP31hl84X7G5U9ZfsWvIB3Pg==}
719 + engines: {node: '>=20.19.0'}
720 +
721 + domhandler@5.0.3:
722 + resolution: {integrity: sha512-cgwlv/1iFQiFnU96XXgROh8xTeetsnJiDsTc7TYCLFd9+/WNkIqPTxiM/8pSd8VIrhXGTf1Ny1q1hquVqDJB5w==}
723 + engines: {node: '>= 4'}
724 +
725 + domhandler@6.0.1:
726 + resolution: {integrity: sha512-gYzvtM72ZtxQO0T048kd6HWSbbGCNOUwcnfQ01cqIJ4X2IYKFFHZ5mKvrQETcFXxsRObZulDaKmy//R7TPtsBg==}
727 + engines: {node: '>=20.19.0'}
728 +
729 + domutils@3.2.2:
730 + resolution: {integrity: sha512-6kZKyUajlDuqlHKVX1w7gyslj9MPIXzIFiz/rGu35uC1wMi+kMhQwGhl4lt9unC9Vb9INnY9Z3/ZA3+FhASLaw==}
731 +
732 + domutils@4.0.2:
733 + resolution: {integrity: sha512-qI4JLRKnSzqFqr7hAlS5xQDusBCjKSEG4t4+7aNrIQMHBcsC2TGEhuyABJdYkgSewL57PNLYEiibY2iPKhKpaA==}
734 + engines: {node: '>=20.19.0'}
735 +
736 + entities@4.5.0:
737 + resolution: {integrity: sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw==}
738 + engines: {node: '>=0.12'}
739 +
740 + entities@7.0.1:
741 + resolution: {integrity: sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA==}
742 + engines: {node: '>=0.12'}
743 +
744 + entities@8.0.0:
745 + resolution: {integrity: sha512-zwfzJecQ/Uej6tusMqwAqU/6KL2XaB2VZ2Jg54Je6ahNBGNH6Ek6g3jjNCF0fG9EWQKGZNddNjU5F1ZQn/sBnA==}
746 + engines: {node: '>=20.19.0'}
747 +
748 + es-module-lexer@1.7.0:
749 + resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==}
750 +
751 + esbuild@0.21.5:
752 + resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==}
753 + engines: {node: '>=12'}
754 + hasBin: true
755 +
756 + esbuild@0.28.2:
757 + resolution: {integrity: sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==}
758 + engines: {node: '>=18'}
759 + hasBin: true
760 +
761 + estree-walker@3.0.3:
762 + resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==}
763 +
764 + expect-type@1.4.0:
765 + resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
766 + engines: {node: '>=12.0.0'}
767 +
768 + fast-decode-uri-component@1.0.1:
769 + resolution: {integrity: sha512-WKgKWg5eUxvRZGwW8FvfbaH7AXSh2cL+3j5fMGzUMCxWBJ3dV3a7Wz8y2f/uQ0e3B6WmodD3oS54jTQ9HVTIIg==}
770 +
771 + fast-deep-equal@3.1.3:
772 + resolution: {integrity: sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==}
773 +
774 + fast-json-stringify@7.0.1:
775 + resolution: {integrity: sha512-eRSayARSbbwlBjpP4vnTTIRD5QPcIrmihPxDeN1DtKnHPg66UuJLx+8hlK1kaFdjvzyQ/dzALoi4vwAQ+T+iZA==}
776 +
777 + fast-querystring@1.1.2:
778 + resolution: {integrity: sha512-g6KuKWmFXc0fID8WWH0jit4g0AGBoJhCkJMb1RmbsSEUNvQ+ZC8D6CUZ+GtF8nMzSPXnhiePyyqqipzNNEnHjg==}
779 +
780 + fast-uri@3.1.5:
781 + resolution: {integrity: sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw==}
782 +
783 + fast-uri@4.1.2:
784 + resolution: {integrity: sha512-TyGmBcbDTZXcb2cj5MV89DrF42DKvb3y5DDUNh95iO+IMeAzMkVSxK1PZRrRIpc9yg8U2GhGdbofNa0LS/a4Bw==}
785 +
786 + fastify@5.11.3:
787 + resolution: {integrity: sha512-W6hzDP8s0iSeL7LGwY6Oc/ZxuXWOvFEMs6p2L0Si415YRo27W5pBKdOTXxhemBDeSTAcpYf5evRA9onF2OYhPA==}
788 +
789 + fastq@1.20.1:
790 + resolution: {integrity: sha512-GGToxJ/w1x32s/D2EKND7kTil4n8OVk/9mycTc4VDza13lOvpUZTGX3mFSCtV9ksdGBVzvsyAVLM6mHFThxXxw==}
791 +
792 + find-my-way@9.7.0:
793 + resolution: {integrity: sha512-f2JHn75x2JlwUwLenZypgczR7YWMb/uO9BvUXtus+JMgkbIkLADd38cI4EiV+OQqrGo1Zlq6V8wnqMJ8e62wUQ==}
794 + engines: {node: '>=20'}
795 +
796 + fsevents@2.3.3:
797 + resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
798 + engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
799 + os: [darwin]
800 +
801 + globrex@0.1.2:
802 + resolution: {integrity: sha512-uHJgbwAMwNFf5mLst7IWLNg14x1CkeqglJb/K3doi4dw6q2IvAAmM/Y81kevy83wP+Sst+nutFTYOGg3d1lsxg==}
803 +
804 + html-escaper@3.0.3:
805 + resolution: {integrity: sha512-RuMffC89BOWQoY0WKGpIhn5gX3iI54O6nRA0yC124NYVtzjmFWBIiFd8M0x+ZdX0P9R4lADg1mgP8C7PxGOWuQ==}
806 +
807 + htmlparser2@10.1.0:
808 + resolution: {integrity: sha512-VTZkM9GWRAtEpveh7MSF6SjjrpNVNNVJfFup7xTY3UpFtm67foy9HDVXneLtFVt4pMz5kZtgNcvCniNFb1hlEQ==}
809 +
810 + ipaddr.js@2.5.0:
811 + resolution: {integrity: sha512-aq+t5NAc+cS6rZQQVWC2x98CPqGtKKTMDd4Gaodv0wShnItdKg/51djkGJ1hqH+Oy0ivDftCbSLCQob8zso01w==}
812 + engines: {node: '>= 10'}
813 +
814 + json-schema-ref-resolver@3.0.0:
815 + resolution: {integrity: sha512-hOrZIVL5jyYFjzk7+y7n5JDzGlU8rfWDuYyHwGa2WA8/pcmMHezp2xsVwxrebD/Q9t8Nc5DboieySDpCp4WG4A==}
816 +
817 + json-schema-traverse@1.0.0:
818 + resolution: {integrity: sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==}
819 +
820 + light-my-request@6.6.0:
821 + resolution: {integrity: sha512-CHYbu8RtboSIoVsHZ6Ye4cj4Aw/yg2oAFimlF7mNvfDV192LR7nDiKtSIfCuLT7KokPSTn/9kfVLm5OGN0A28A==}
822 +
823 + linkedom@0.18.13:
824 + resolution: {integrity: sha512-ES/o9qotMpzpN2MHs+Iq/JcVoOj8Fa5wiQYrTdFpvAnwXL0g66XHHUc9WUMk6nAlBtGsFQ24ne+SYnvnaQ2FSw==}
825 + engines: {node: '>=16'}
826 + peerDependencies:
827 + canvas: '>= 2'
828 + peerDependenciesMeta:
829 + canvas:
830 + optional: true
831 +
832 + loupe@3.2.1:
833 + resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==}
834 +
835 + magic-string@0.30.21:
836 + resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
837 +
838 + ms@2.1.3:
839 + resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
840 +
841 + nanoid@3.3.18:
842 + resolution: {integrity: sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w==}
843 + engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1}
844 + hasBin: true
845 +
846 + nth-check@3.0.1:
847 + resolution: {integrity: sha512-GX0gsdbGVCgnRgbeGaubfjpBXyYRWOOCVeYh08bSQvDZqxz5ndXs1OTfAt/h36G1xvI94YIspsI0sVFqAV9+RQ==}
848 + engines: {node: '>=20.19.0'}
849 +
850 + on-exit-leak-free@2.1.2:
851 + resolution: {integrity: sha512-0eJJY6hXLGf1udHwfNftBqH+g73EU4B504nZeKpz1sYRKafAghwxEJunB2O7rDZkL4PGfsMVnTXZ2EjibbqcsA==}
852 + engines: {node: '>=14.0.0'}
853 +
854 + pathe@1.1.2:
855 + resolution: {integrity: sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==}
856 +
857 + pathval@2.0.1:
858 + resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==}
859 + engines: {node: '>= 14.16'}
860 +
861 + picocolors@1.1.1:
862 + resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
863 +
864 + pino-abstract-transport@2.0.0:
865 + resolution: {integrity: sha512-F63x5tizV6WCh4R6RHyi2Ml+M70DNRXt/+HANowMflpgGFMAym/VKm6G7ZOQRjqN7XbGxK1Lg9t6ZrtzOaivMw==}
866 +
867 + pino-std-serializers@7.1.0:
868 + resolution: {integrity: sha512-BndPH67/JxGExRgiX1dX0w1FvZck5Wa4aal9198SrRhZjH3GxKQUKIBnYJTdj2HDN3UQAS06HlfcSbQj2OHmaw==}
869 +
870 + pino@9.14.0:
871 + resolution: {integrity: sha512-8OEwKp5juEvb/MjpIc4hjqfgCNysrS94RIOMXYvpYCdm/jglrKEiAYmiumbmGhCvs+IcInsphYDFwqrjr7398w==}
872 + hasBin: true
873 +
874 + postcss@8.5.26:
875 + resolution: {integrity: sha512-u82N74LFzG8ca+dD8puPnplTXoGH4fTPpVGuIbt36G3qvNlkvfD0lEAZSxaly3KX8TS/L1A1gsCEmvKmBcVbkQ==}
876 + engines: {node: ^10 || ^12 || >=14}
877 +
878 + process-warning@4.0.1:
879 + resolution: {integrity: sha512-3c2LzQ3rY9d0hc1emcsHhfT9Jwz0cChib/QN89oME2R451w5fy3f0afAhERFZAwrbDU43wk12d0ORBpDVME50Q==}
880 +
881 + process-warning@5.1.0:
882 + resolution: {integrity: sha512-jQSaVHsPgtyw60e1rQ/A+/ArPEj/S8pS/vFnyGa/gYFXrKk/6RuDkoqVDQ5NI5MmS01698ltlAk0NoDBNLujRw==}
883 +
884 + quick-format-unescaped@4.0.4:
885 + resolution: {integrity: sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg==}
886 +
887 + real-require@0.2.0:
888 + resolution: {integrity: sha512-57frrGM/OCTLqLOAh0mhVA9VBMHd+9U7Zb2THMGdBUoZVOtGbJzjxsYGDJ3A9AYYCP4hn6y1TVbaOfzWtm5GFg==}
889 + engines: {node: '>= 12.13.0'}
890 +
891 + require-from-string@2.0.2:
892 + resolution: {integrity: sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==}
893 + engines: {node: '>=0.10.0'}
894 +
895 + ret@0.5.0:
896 + resolution: {integrity: sha512-I1XxrZSQ+oErkRR4jYbAyEEu2I0avBvvMM5JN+6EBprOGRCs63ENqZ3vjavq8fBw2+62G5LF5XelKwuJpcvcxw==}
897 + engines: {node: '>=10'}
898 +
899 + reusify@1.1.0:
900 + resolution: {integrity: sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw==}
901 + engines: {iojs: '>=1.0.0', node: '>=0.10.0'}
902 +
903 + rfdc@1.4.1:
904 + resolution: {integrity: sha512-q1b3N5QkRUWUl7iyylaaj3kOpIT0N2i9MqIEQXP73GVsN9cw3fdx8X63cEmWhJGi2PPCF23Ijp7ktmd39rawIA==}
905 +
906 + rollup@4.62.4:
907 + resolution: {integrity: sha512-RXOqwaPsBGjMNMa4sQjDjHieHEZDFoj/Rdr46l2MU5DfEs16wHJPC2RPTPHWhNl+M3aI472LLqFkFKut4SblOg==}
908 + engines: {node: '>=18.0.0', npm: '>=8.0.0'}
909 + hasBin: true
910 +
911 + safe-regex2@5.1.1:
912 + resolution: {integrity: sha512-mOSBvHGDZMuIEZMdOz/aCEYDCv0E7nfcNsIhUF+/P+xC7Hyf3FkvymqgPbg9D1EdSGu+uKbJgy09K/RKKc7kJA==}
913 + hasBin: true
914 +
915 + safe-stable-stringify@2.5.0:
916 + resolution: {integrity: sha512-b3rppTKm9T+PsVCBEOUR46GWI7fdOs00VKZ1+9c1EWDaDMvjQc6tUwuFyIprgGgTcWoVHSKrU8H31ZHA2e0RHA==}
917 + engines: {node: '>=10'}
918 +
919 + secure-json-parse@4.1.0:
920 + resolution: {integrity: sha512-l4KnYfEyqYJxDwlNVyRfO2E4NTHfMKAWdUuA8J0yve2Dz/E/PdBepY03RvyJpssIpRFwJoCD55wA+mEDs6ByWA==}
921 +
922 + semver@7.8.5:
923 + resolution: {integrity: sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==}
924 + engines: {node: '>=10'}
925 + hasBin: true
926 +
927 + set-cookie-parser@2.7.2:
928 + resolution: {integrity: sha512-oeM1lpU/UvhTxw+g3cIfxXHyJRc/uidd3yK1P242gzHds0udQBYzs3y8j4gCCW+ZJ7ad0yctld8RYO+bdurlvw==}
929 +
930 + siginfo@2.0.0:
931 + resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==}
932 +
933 + sonic-boom@4.2.1:
934 + resolution: {integrity: sha512-w6AxtubXa2wTXAUsZMMWERrsIRAdrK0Sc+FUytWvYAhBJLyuI4llrMIC1DtlNSdI99EI86KZum2MMq3EAZlF9Q==}
935 +
936 + source-map-js@1.2.1:
937 + resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==}
938 + engines: {node: '>=0.10.0'}
939 +
940 + split2@4.2.0:
941 + resolution: {integrity: sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==}
942 + engines: {node: '>= 10.x'}
943 +
944 + stackback@0.0.2:
945 + resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==}
946 +
947 + std-env@3.10.0:
948 + resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==}
949 +
950 + thread-stream@3.2.0:
951 + resolution: {integrity: sha512-zLBvqpwr4Esa0kRjcrzGU6zL25lePWaCLMx0RQFrmteozIfeNdaMLpG5U7PeHzvlFkAWaRKA9/KVW4F60iB+qw==}
952 +
953 + tinybench@2.9.0:
954 + resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==}
955 +
956 + tinyexec@0.3.2:
957 + resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==}
958 +
959 + tinypool@1.1.1:
960 + resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==}
961 + engines: {node: ^18.0.0 || >=20.0.0}
962 +
963 + tinyrainbow@1.2.0:
964 + resolution: {integrity: sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==}
965 + engines: {node: '>=14.0.0'}
966 +
967 + tinyspy@3.0.2:
968 + resolution: {integrity: sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==}
969 + engines: {node: '>=14.0.0'}
970 +
971 + toad-cache@3.7.4:
972 + resolution: {integrity: sha512-m1TdR/rvT7kgGJZhspNtXdsdYk0fddFpJJFlG5s+UkPFo6lkLoZ3YLOaovPYjq1R75NP5JfeTlSHaOsE09peCg==}
973 + engines: {node: '>=20'}
974 +
975 + tsconfck@3.1.6:
976 + resolution: {integrity: sha512-ks6Vjr/jEw0P1gmOVwutM3B7fWxoWBL2KRDb1JfqGVawBmO5UsvmWOQFGHBPl5yxYz4eERr19E6L7NMv+Fej4w==}
977 + engines: {node: ^18 || >=20}
978 + deprecated: unmaintained
979 + hasBin: true
980 + peerDependencies:
981 + typescript: ^5.0.0
982 + peerDependenciesMeta:
983 + typescript:
984 + optional: true
985 +
986 + tsx@4.23.11:
987 + resolution: {integrity: sha512-Ry2oTEUnhBdeEdWIztY8kf3/nBGnPnjMLVGL0YfdRXMORuPER5NlKmayqxtxRxwB1xBN+RivRaJfe7PM1rtiyw==}
988 + engines: {node: '>=18.0.0'}
989 + hasBin: true
990 +
991 + turndown@7.2.4:
992 + resolution: {integrity: sha512-I8yFsfRzmzK0WV1pNNOA4A7y4RDfFxPRxb3t+e3ui14qSGOxGtiSP6GjeX+Y6CHb7HYaFj7ECUD7VE5kQMZWGQ==}
993 + engines: {node: '>=18', npm: '>=9'}
994 +
995 + typescript@5.9.3:
996 + resolution: {integrity: sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==}
997 + engines: {node: '>=14.17'}
998 + hasBin: true
999 +
1000 + uhyphen@0.2.0:
1001 + resolution: {integrity: sha512-qz3o9CHXmJJPGBdqzab7qAYuW8kQGKNEuoHFYrBwV6hWIMcpAmxDLXojcHfFr9US1Pe6zUswEIJIbLI610fuqA==}
1002 +
1003 + undici-types@6.21.0:
1004 + resolution: {integrity: sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==}
1005 +
1006 + undici@7.29.0:
1007 + resolution: {integrity: sha512-IDxfleLmmbSskfWSUATiN1nfn2rDuvnMOqb5CWR92iIfojA0Ud+ulOAAEQ57LPr9rWmsreUyf5lwyao+7GNNVw==}
1008 + engines: {node: '>=20.18.1'}
1009 +
1010 + vite-node@2.1.9:
1011 + resolution: {integrity: sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==}
1012 + engines: {node: ^18.0.0 || >=20.0.0}
1013 + hasBin: true
1014 +
1015 + vite-tsconfig-paths@5.1.4:
1016 + resolution: {integrity: sha512-cYj0LRuLV2c2sMqhqhGpaO3LretdtMn/BVX4cPLanIZuwwrkVl+lK84E/miEXkCHWXuq65rhNN4rXsBcOB3S4w==}
1017 + peerDependencies:
1018 + vite: '*'
1019 + peerDependenciesMeta:
1020 + vite:
1021 + optional: true
1022 +
1023 + vite@5.4.21:
1024 + resolution: {integrity: sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==}
1025 + engines: {node: ^18.0.0 || >=20.0.0}
1026 + hasBin: true
1027 + peerDependencies:
1028 + '@types/node': ^18.0.0 || >=20.0.0
1029 + less: '*'
1030 + lightningcss: ^1.21.0
1031 + sass: '*'
1032 + sass-embedded: '*'
1033 + stylus: '*'
1034 + sugarss: '*'
1035 + terser: ^5.4.0
1036 + peerDependenciesMeta:
1037 + '@types/node':
1038 + optional: true
1039 + less:
1040 + optional: true
1041 + lightningcss:
1042 + optional: true
1043 + sass:
1044 + optional: true
1045 + sass-embedded:
1046 + optional: true
1047 + stylus:
1048 + optional: true
1049 + sugarss:
1050 + optional: true
1051 + terser:
1052 + optional: true
1053 +
1054 + vitest@2.1.9:
1055 + resolution: {integrity: sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==}
1056 + engines: {node: ^18.0.0 || >=20.0.0}
1057 + hasBin: true
1058 + peerDependencies:
1059 + '@edge-runtime/vm': '*'
1060 + '@types/node': ^18.0.0 || >=20.0.0
1061 + '@vitest/browser': 2.1.9
1062 + '@vitest/ui': 2.1.9
1063 + happy-dom: '*'
1064 + jsdom: '*'
1065 + peerDependenciesMeta:
1066 + '@edge-runtime/vm':
1067 + optional: true
1068 + '@types/node':
1069 + optional: true
1070 + '@vitest/browser':
1071 + optional: true
1072 + '@vitest/ui':
1073 + optional: true
1074 + happy-dom:
1075 + optional: true
1076 + jsdom:
1077 + optional: true
1078 +
1079 + why-is-node-running@2.3.0:
1080 + resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==}
1081 + engines: {node: '>=8'}
1082 + hasBin: true
1083 +
1084 + zod@3.25.76:
1085 + resolution: {integrity: sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==}
1086 +
1087 +snapshots:
1088 +
1089 + '@esbuild/aix-ppc64@0.21.5':
1090 + optional: true
1091 +
1092 + '@esbuild/aix-ppc64@0.28.2':
1093 + optional: true
1094 +
1095 + '@esbuild/android-arm64@0.21.5':
1096 + optional: true
1097 +
1098 + '@esbuild/android-arm64@0.28.2':
1099 + optional: true
1100 +
1101 + '@esbuild/android-arm@0.21.5':
1102 + optional: true
1103 +
1104 + '@esbuild/android-arm@0.28.2':
1105 + optional: true
1106 +
1107 + '@esbuild/android-x64@0.21.5':
1108 + optional: true
1109 +
1110 + '@esbuild/android-x64@0.28.2':
1111 + optional: true
1112 +
1113 + '@esbuild/darwin-arm64@0.21.5':
1114 + optional: true
1115 +
1116 + '@esbuild/darwin-arm64@0.28.2':
1117 + optional: true
1118 +
1119 + '@esbuild/darwin-x64@0.21.5':
1120 + optional: true
1121 +
1122 + '@esbuild/darwin-x64@0.28.2':
1123 + optional: true
1124 +
1125 + '@esbuild/freebsd-arm64@0.21.5':
1126 + optional: true
1127 +
1128 + '@esbuild/freebsd-arm64@0.28.2':
1129 + optional: true
1130 +
1131 + '@esbuild/freebsd-x64@0.21.5':
1132 + optional: true
1133 +
1134 + '@esbuild/freebsd-x64@0.28.2':
1135 + optional: true
1136 +
1137 + '@esbuild/linux-arm64@0.21.5':
1138 + optional: true
1139 +
1140 + '@esbuild/linux-arm64@0.28.2':
1141 + optional: true
1142 +
1143 + '@esbuild/linux-arm@0.21.5':
1144 + optional: true
1145 +
1146 + '@esbuild/linux-arm@0.28.2':
1147 + optional: true
1148 +
1149 + '@esbuild/linux-ia32@0.21.5':
1150 + optional: true
1151 +
1152 + '@esbuild/linux-ia32@0.28.2':
1153 + optional: true
1154 +
1155 + '@esbuild/linux-loong64@0.21.5':
1156 + optional: true
1157 +
1158 + '@esbuild/linux-loong64@0.28.2':
1159 + optional: true
1160 +
1161 + '@esbuild/linux-mips64el@0.21.5':
1162 + optional: true
1163 +
1164 + '@esbuild/linux-mips64el@0.28.2':
1165 + optional: true
1166 +
1167 + '@esbuild/linux-ppc64@0.21.5':
1168 + optional: true
1169 +
1170 + '@esbuild/linux-ppc64@0.28.2':
1171 + optional: true
1172 +
1173 + '@esbuild/linux-riscv64@0.21.5':
1174 + optional: true
1175 +
1176 + '@esbuild/linux-riscv64@0.28.2':
1177 + optional: true
1178 +
1179 + '@esbuild/linux-s390x@0.21.5':
1180 + optional: true
1181 +
1182 + '@esbuild/linux-s390x@0.28.2':
1183 + optional: true
1184 +
1185 + '@esbuild/linux-x64@0.21.5':
1186 + optional: true
1187 +
1188 + '@esbuild/linux-x64@0.28.2':
1189 + optional: true
1190 +
1191 + '@esbuild/netbsd-arm64@0.28.2':
1192 + optional: true
1193 +
1194 + '@esbuild/netbsd-x64@0.21.5':
1195 + optional: true
1196 +
1197 + '@esbuild/netbsd-x64@0.28.2':
1198 + optional: true
1199 +
1200 + '@esbuild/openbsd-arm64@0.28.2':
1201 + optional: true
1202 +
1203 + '@esbuild/openbsd-x64@0.21.5':
1204 + optional: true
1205 +
1206 + '@esbuild/openbsd-x64@0.28.2':
1207 + optional: true
1208 +
1209 + '@esbuild/openharmony-arm64@0.28.2':
1210 + optional: true
1211 +
1212 + '@esbuild/sunos-x64@0.21.5':
1213 + optional: true
1214 +
1215 + '@esbuild/sunos-x64@0.28.2':
1216 + optional: true
1217 +
1218 + '@esbuild/win32-arm64@0.21.5':
1219 + optional: true
1220 +
1221 + '@esbuild/win32-arm64@0.28.2':
1222 + optional: true
1223 +
1224 + '@esbuild/win32-ia32@0.21.5':
1225 + optional: true
1226 +
1227 + '@esbuild/win32-ia32@0.28.2':
1228 + optional: true
1229 +
1230 + '@esbuild/win32-x64@0.21.5':
1231 + optional: true
1232 +
1233 + '@esbuild/win32-x64@0.28.2':
1234 + optional: true
1235 +
1236 + '@fastify/ajv-compiler@4.0.6':
1237 + dependencies:
1238 + ajv: 8.20.0
1239 + ajv-formats: 3.0.1(ajv@8.20.0)
1240 + fast-uri: 4.1.2
1241 +
1242 + '@fastify/error@4.2.0': {}
1243 +
1244 + '@fastify/fast-json-stringify-compiler@5.1.0':
1245 + dependencies:
1246 + fast-json-stringify: 7.0.1
1247 +
1248 + '@fastify/forwarded@3.0.2': {}
1249 +
1250 + '@fastify/merge-json-schemas@0.2.1':
1251 + dependencies:
1252 + dequal: 2.0.3
1253 +
1254 + '@fastify/proxy-addr@5.1.0':
1255 + dependencies:
1256 + '@fastify/forwarded': 3.0.2
1257 + ipaddr.js: 2.5.0
1258 +
1259 + '@jridgewell/sourcemap-codec@1.5.5': {}
1260 +
1261 + '@mixmark-io/domino@2.2.0': {}
1262 +
1263 + '@mozilla/readability@0.5.0': {}
1264 +
1265 + '@napi-rs/lzma-linux-x64-gnu@1.5.1':
1266 + optional: true
1267 +
1268 + '@pinojs/redact@0.4.0': {}
1269 +
1270 + '@rollup/rollup-android-arm-eabi@4.62.4':
1271 + optional: true
1272 +
1273 + '@rollup/rollup-android-arm64@4.62.4':
1274 + optional: true
1275 +
1276 + '@rollup/rollup-darwin-arm64@4.62.4':
1277 + optional: true
1278 +
1279 + '@rollup/rollup-darwin-x64@4.62.4':
1280 + optional: true
1281 +
1282 + '@rollup/rollup-freebsd-arm64@4.62.4':
1283 + optional: true
1284 +
1285 + '@rollup/rollup-freebsd-x64@4.62.4':
1286 + optional: true
1287 +
1288 + '@rollup/rollup-linux-arm-gnueabihf@4.62.4':
1289 + optional: true
1290 +
1291 + '@rollup/rollup-linux-arm-musleabihf@4.62.4':
1292 + optional: true
1293 +
1294 + '@rollup/rollup-linux-arm64-gnu@4.62.4':
1295 + optional: true
1296 +
1297 + '@rollup/rollup-linux-arm64-musl@4.62.4':
1298 + optional: true
1299 +
1300 + '@rollup/rollup-linux-loong64-gnu@4.62.4':
1301 + optional: true
1302 +
1303 + '@rollup/rollup-linux-loong64-musl@4.62.4':
1304 + optional: true
1305 +
1306 + '@rollup/rollup-linux-ppc64-gnu@4.62.4':
1307 + optional: true
1308 +
1309 + '@rollup/rollup-linux-ppc64-musl@4.62.4':
1310 + optional: true
1311 +
1312 + '@rollup/rollup-linux-riscv64-gnu@4.62.4':
1313 + optional: true
1314 +
1315 + '@rollup/rollup-linux-riscv64-musl@4.62.4':
1316 + optional: true
1317 +
1318 + '@rollup/rollup-linux-s390x-gnu@4.62.4':
1319 + optional: true
1320 +
1321 + '@rollup/rollup-linux-x64-gnu@4.62.4':
1322 + optional: true
1323 +
1324 + '@rollup/rollup-linux-x64-musl@4.62.4':
1325 + optional: true
1326 +
1327 + '@rollup/rollup-openbsd-x64@4.62.4':
1328 + optional: true
1329 +
1330 + '@rollup/rollup-openharmony-arm64@4.62.4':
1331 + optional: true
1332 +
1333 + '@rollup/rollup-win32-arm64-msvc@4.62.4':
1334 + optional: true
1335 +
1336 + '@rollup/rollup-win32-ia32-msvc@4.62.4':
1337 + optional: true
1338 +
1339 + '@rollup/rollup-win32-x64-gnu@4.62.4':
1340 + optional: true
1341 +
1342 + '@rollup/rollup-win32-x64-msvc@4.62.4':
1343 + optional: true
1344 +
1345 + '@types/estree@1.0.9': {}
1346 +
1347 + '@types/node@22.20.1':
1348 + dependencies:
1349 + undici-types: 6.21.0
1350 +
1351 + '@types/turndown@5.0.6': {}
1352 +
1353 + '@vitest/expect@2.1.9':
1354 + dependencies:
1355 + '@vitest/spy': 2.1.9
1356 + '@vitest/utils': 2.1.9
1357 + chai: 5.3.3
1358 + tinyrainbow: 1.2.0
1359 +
1360 + '@vitest/mocker@2.1.9(vite@5.4.21(@types/node@22.20.1))':
1361 + dependencies:
1362 + '@vitest/spy': 2.1.9
1363 + estree-walker: 3.0.3
1364 + magic-string: 0.30.21
1365 + optionalDependencies:
1366 + vite: 5.4.21(@types/node@22.20.1)
1367 +
1368 + '@vitest/pretty-format@2.1.9':
1369 + dependencies:
1370 + tinyrainbow: 1.2.0
1371 +
1372 + '@vitest/runner@2.1.9':
1373 + dependencies:
1374 + '@vitest/utils': 2.1.9
1375 + pathe: 1.1.2
1376 +
1377 + '@vitest/snapshot@2.1.9':
1378 + dependencies:
1379 + '@vitest/pretty-format': 2.1.9
1380 + magic-string: 0.30.21
1381 + pathe: 1.1.2
1382 +
1383 + '@vitest/spy@2.1.9':
1384 + dependencies:
1385 + tinyspy: 3.0.2
1386 +
1387 + '@vitest/utils@2.1.9':
1388 + dependencies:
1389 + '@vitest/pretty-format': 2.1.9
1390 + loupe: 3.2.1
1391 + tinyrainbow: 1.2.0
1392 +
1393 + abstract-logging@2.0.1: {}
1394 +
1395 + ajv-formats@3.0.1(ajv@8.20.0):
1396 + optionalDependencies:
1397 + ajv: 8.20.0
1398 +
1399 + ajv@8.20.0:
1400 + dependencies:
1401 + fast-deep-equal: 3.1.3
1402 + fast-uri: 3.1.5
1403 + json-schema-traverse: 1.0.0
1404 + require-from-string: 2.0.2
1405 +
1406 + assertion-error@2.0.1: {}
1407 +
1408 + atomic-sleep@1.0.0: {}
1409 +
1410 + avvio@9.3.0:
1411 + dependencies:
1412 + '@fastify/error': 4.2.0
1413 + fastq: 1.20.1
1414 +
1415 + boolbase@2.0.0: {}
1416 +
1417 + cac@6.7.14: {}
1418 +
1419 + chai@5.3.3:
1420 + dependencies:
1421 + assertion-error: 2.0.1
1422 + check-error: 2.1.3
1423 + deep-eql: 5.0.2
1424 + loupe: 3.2.1
1425 + pathval: 2.0.1
1426 +
1427 + check-error@2.1.3: {}
1428 +
1429 + cookie@1.1.1: {}
1430 +
1431 + css-select@7.0.0:
1432 + dependencies:
1433 + boolbase: 2.0.0
1434 + css-what: 8.0.0
1435 + domhandler: 6.0.1
1436 + domutils: 4.0.2
1437 + nth-check: 3.0.1
1438 +
1439 + css-what@8.0.0: {}
1440 +
1441 + cssom@0.5.0: {}
1442 +
1443 + debug@4.4.3:
1444 + dependencies:
1445 + ms: 2.1.3
1446 +
1447 + deep-eql@5.0.2: {}
1448 +
1449 + dequal@2.0.3: {}
1450 +
1451 + dom-serializer@2.0.0:
1452 + dependencies:
1453 + domelementtype: 2.3.0
1454 + domhandler: 5.0.3
1455 + entities: 4.5.0
1456 +
1457 + dom-serializer@3.1.1:
1458 + dependencies:
1459 + domelementtype: 3.0.0
1460 + domhandler: 6.0.1
1461 + entities: 8.0.0
1462 +
1463 + domelementtype@2.3.0: {}
1464 +
1465 + domelementtype@3.0.0: {}
1466 +
1467 + domhandler@5.0.3:
1468 + dependencies:
1469 + domelementtype: 2.3.0
1470 +
1471 + domhandler@6.0.1:
1472 + dependencies:
1473 + domelementtype: 3.0.0
1474 +
1475 + domutils@3.2.2:
1476 + dependencies:
1477 + dom-serializer: 2.0.0
1478 + domelementtype: 2.3.0
1479 + domhandler: 5.0.3
1480 +
1481 + domutils@4.0.2:
1482 + dependencies:
1483 + dom-serializer: 3.1.1
1484 + domelementtype: 3.0.0
1485 + domhandler: 6.0.1
1486 +
1487 + entities@4.5.0: {}
1488 +
1489 + entities@7.0.1: {}
1490 +
1491 + entities@8.0.0: {}
1492 +
1493 + es-module-lexer@1.7.0: {}
1494 +
1495 + esbuild@0.21.5:
1496 + optionalDependencies:
1497 + '@esbuild/aix-ppc64': 0.21.5
1498 + '@esbuild/android-arm': 0.21.5
1499 + '@esbuild/android-arm64': 0.21.5
1500 + '@esbuild/android-x64': 0.21.5
1501 + '@esbuild/darwin-arm64': 0.21.5
1502 + '@esbuild/darwin-x64': 0.21.5
1503 + '@esbuild/freebsd-arm64': 0.21.5
1504 + '@esbuild/freebsd-x64': 0.21.5
1505 + '@esbuild/linux-arm': 0.21.5
1506 + '@esbuild/linux-arm64': 0.21.5
1507 + '@esbuild/linux-ia32': 0.21.5
1508 + '@esbuild/linux-loong64': 0.21.5
1509 + '@esbuild/linux-mips64el': 0.21.5
1510 + '@esbuild/linux-ppc64': 0.21.5
1511 + '@esbuild/linux-riscv64': 0.21.5
1512 + '@esbuild/linux-s390x': 0.21.5
1513 + '@esbuild/linux-x64': 0.21.5
1514 + '@esbuild/netbsd-x64': 0.21.5
1515 + '@esbuild/openbsd-x64': 0.21.5
1516 + '@esbuild/sunos-x64': 0.21.5
1517 + '@esbuild/win32-arm64': 0.21.5
1518 + '@esbuild/win32-ia32': 0.21.5
1519 + '@esbuild/win32-x64': 0.21.5
1520 +
1521 + esbuild@0.28.2:
1522 + optionalDependencies:
1523 + '@esbuild/aix-ppc64': 0.28.2
1524 + '@esbuild/android-arm': 0.28.2
1525 + '@esbuild/android-arm64': 0.28.2
1526 + '@esbuild/android-x64': 0.28.2
1527 + '@esbuild/darwin-arm64': 0.28.2
1528 + '@esbuild/darwin-x64': 0.28.2
1529 + '@esbuild/freebsd-arm64': 0.28.2
1530 + '@esbuild/freebsd-x64': 0.28.2
1531 + '@esbuild/linux-arm': 0.28.2
1532 + '@esbuild/linux-arm64': 0.28.2
1533 + '@esbuild/linux-ia32': 0.28.2
1534 + '@esbuild/linux-loong64': 0.28.2
1535 + '@esbuild/linux-mips64el': 0.28.2
1536 + '@esbuild/linux-ppc64': 0.28.2
1537 + '@esbuild/linux-riscv64': 0.28.2
1538 + '@esbuild/linux-s390x': 0.28.2
1539 + '@esbuild/linux-x64': 0.28.2
1540 + '@esbuild/netbsd-arm64': 0.28.2
1541 + '@esbuild/netbsd-x64': 0.28.2
1542 + '@esbuild/openbsd-arm64': 0.28.2
1543 + '@esbuild/openbsd-x64': 0.28.2
1544 + '@esbuild/openharmony-arm64': 0.28.2
1545 + '@esbuild/sunos-x64': 0.28.2
1546 + '@esbuild/win32-arm64': 0.28.2
1547 + '@esbuild/win32-ia32': 0.28.2
1548 + '@esbuild/win32-x64': 0.28.2
1549 +
1550 + estree-walker@3.0.3:
1551 + dependencies:
1552 + '@types/estree': 1.0.9
1553 +
1554 + expect-type@1.4.0: {}
1555 +
1556 + fast-decode-uri-component@1.0.1: {}
1557 +
1558 + fast-deep-equal@3.1.3: {}
1559 +
1560 + fast-json-stringify@7.0.1:
1561 + dependencies:
1562 + '@fastify/merge-json-schemas': 0.2.1
1563 + ajv: 8.20.0
1564 + ajv-formats: 3.0.1(ajv@8.20.0)
1565 + fast-uri: 4.1.2
1566 + json-schema-ref-resolver: 3.0.0
1567 + rfdc: 1.4.1
1568 +
1569 + fast-querystring@1.1.2:
1570 + dependencies:
1571 + fast-decode-uri-component: 1.0.1
1572 +
1573 + fast-uri@3.1.5: {}
1574 +
1575 + fast-uri@4.1.2: {}
1576 +
1577 + fastify@5.11.3:
1578 + dependencies:
1579 + '@fastify/ajv-compiler': 4.0.6
1580 + '@fastify/error': 4.2.0
1581 + '@fastify/fast-json-stringify-compiler': 5.1.0
1582 + '@fastify/proxy-addr': 5.1.0
1583 + abstract-logging: 2.0.1
1584 + avvio: 9.3.0
1585 + fast-json-stringify: 7.0.1
1586 + find-my-way: 9.7.0
1587 + light-my-request: 6.6.0
1588 + pino: 9.14.0
1589 + process-warning: 5.1.0
1590 + rfdc: 1.4.1
1591 + secure-json-parse: 4.1.0
1592 + semver: 7.8.5
1593 + toad-cache: 3.7.4
1594 +
1595 + fastq@1.20.1:
1596 + dependencies:
1597 + reusify: 1.1.0
1598 +
1599 + find-my-way@9.7.0:
1600 + dependencies:
1601 + fast-deep-equal: 3.1.3
1602 + fast-querystring: 1.1.2
1603 + safe-regex2: 5.1.1
1604 +
1605 + fsevents@2.3.3:
1606 + optional: true
1607 +
1608 + globrex@0.1.2: {}
1609 +
1610 + html-escaper@3.0.3: {}
1611 +
1612 + htmlparser2@10.1.0:
1613 + dependencies:
1614 + domelementtype: 2.3.0
1615 + domhandler: 5.0.3
1616 + domutils: 3.2.2
1617 + entities: 7.0.1
1618 +
1619 + ipaddr.js@2.5.0: {}
1620 +
1621 + json-schema-ref-resolver@3.0.0:
1622 + dependencies:
1623 + dequal: 2.0.3
1624 +
1625 + json-schema-traverse@1.0.0: {}
1626 +
1627 + light-my-request@6.6.0:
1628 + dependencies:
1629 + cookie: 1.1.1
1630 + process-warning: 4.0.1
1631 + set-cookie-parser: 2.7.2
1632 +
1633 + linkedom@0.18.13:
1634 + dependencies:
1635 + css-select: 7.0.0
1636 + cssom: 0.5.0
1637 + html-escaper: 3.0.3
1638 + htmlparser2: 10.1.0
1639 + uhyphen: 0.2.0
1640 +
1641 + loupe@3.2.1: {}
1642 +
1643 + magic-string@0.30.21:
1644 + dependencies:
1645 + '@jridgewell/sourcemap-codec': 1.5.5
1646 +
1647 + ms@2.1.3: {}
1648 +
1649 + nanoid@3.3.18: {}
1650 +
1651 + nth-check@3.0.1:
1652 + dependencies:
1653 + boolbase: 2.0.0
1654 +
1655 + on-exit-leak-free@2.1.2: {}
1656 +
1657 + pathe@1.1.2: {}
1658 +
1659 + pathval@2.0.1: {}
1660 +
1661 + picocolors@1.1.1: {}
1662 +
1663 + pino-abstract-transport@2.0.0:
1664 + dependencies:
1665 + split2: 4.2.0
1666 +
1667 + pino-std-serializers@7.1.0: {}
1668 +
1669 + pino@9.14.0:
1670 + dependencies:
1671 + '@pinojs/redact': 0.4.0
1672 + atomic-sleep: 1.0.0
1673 + on-exit-leak-free: 2.1.2
1674 + pino-abstract-transport: 2.0.0
1675 + pino-std-serializers: 7.1.0
1676 + process-warning: 5.1.0
1677 + quick-format-unescaped: 4.0.4
1678 + real-require: 0.2.0
1679 + safe-stable-stringify: 2.5.0
1680 + sonic-boom: 4.2.1
1681 + thread-stream: 3.2.0
1682 +
1683 + postcss@8.5.26:
1684 + dependencies:
1685 + nanoid: 3.3.18
1686 + picocolors: 1.1.1
1687 + source-map-js: 1.2.1
1688 +
1689 + process-warning@4.0.1: {}
1690 +
1691 + process-warning@5.1.0: {}
1692 +
1693 + quick-format-unescaped@4.0.4: {}
1694 +
1695 + real-require@0.2.0: {}
1696 +
1697 + require-from-string@2.0.2: {}
1698 +
1699 + ret@0.5.0: {}
1700 +
1701 + reusify@1.1.0: {}
1702 +
1703 + rfdc@1.4.1: {}
1704 +
1705 + rollup@4.62.4:
1706 + dependencies:
1707 + '@types/estree': 1.0.9
1708 + optionalDependencies:
1709 + '@napi-rs/lzma-linux-x64-gnu': 1.5.1
1710 + '@rollup/rollup-android-arm-eabi': 4.62.4
1711 + '@rollup/rollup-android-arm64': 4.62.4
1712 + '@rollup/rollup-darwin-arm64': 4.62.4
1713 + '@rollup/rollup-darwin-x64': 4.62.4
1714 + '@rollup/rollup-freebsd-arm64': 4.62.4
1715 + '@rollup/rollup-freebsd-x64': 4.62.4
1716 + '@rollup/rollup-linux-arm-gnueabihf': 4.62.4
1717 + '@rollup/rollup-linux-arm-musleabihf': 4.62.4
1718 + '@rollup/rollup-linux-arm64-gnu': 4.62.4
1719 + '@rollup/rollup-linux-arm64-musl': 4.62.4
1720 + '@rollup/rollup-linux-loong64-gnu': 4.62.4
1721 + '@rollup/rollup-linux-loong64-musl': 4.62.4
1722 + '@rollup/rollup-linux-ppc64-gnu': 4.62.4
1723 + '@rollup/rollup-linux-ppc64-musl': 4.62.4
1724 + '@rollup/rollup-linux-riscv64-gnu': 4.62.4
1725 + '@rollup/rollup-linux-riscv64-musl': 4.62.4
1726 + '@rollup/rollup-linux-s390x-gnu': 4.62.4
1727 + '@rollup/rollup-linux-x64-gnu': 4.62.4
1728 + '@rollup/rollup-linux-x64-musl': 4.62.4
1729 + '@rollup/rollup-openbsd-x64': 4.62.4
1730 + '@rollup/rollup-openharmony-arm64': 4.62.4
1731 + '@rollup/rollup-win32-arm64-msvc': 4.62.4
1732 + '@rollup/rollup-win32-ia32-msvc': 4.62.4
1733 + '@rollup/rollup-win32-x64-gnu': 4.62.4
1734 + '@rollup/rollup-win32-x64-msvc': 4.62.4
1735 + fsevents: 2.3.3
1736 +
1737 + safe-regex2@5.1.1:
1738 + dependencies:
1739 + ret: 0.5.0
1740 +
1741 + safe-stable-stringify@2.5.0: {}
1742 +
1743 + secure-json-parse@4.1.0: {}
1744 +
1745 + semver@7.8.5: {}
1746 +
1747 + set-cookie-parser@2.7.2: {}
1748 +
1749 + siginfo@2.0.0: {}
1750 +
1751 + sonic-boom@4.2.1:
1752 + dependencies:
1753 + atomic-sleep: 1.0.0
1754 +
1755 + source-map-js@1.2.1: {}
1756 +
1757 + split2@4.2.0: {}
1758 +
1759 + stackback@0.0.2: {}
1760 +
1761 + std-env@3.10.0: {}
1762 +
1763 + thread-stream@3.2.0:
1764 + dependencies:
1765 + real-require: 0.2.0
1766 +
1767 + tinybench@2.9.0: {}
1768 +
1769 + tinyexec@0.3.2: {}
1770 +
1771 + tinypool@1.1.1: {}
1772 +
1773 + tinyrainbow@1.2.0: {}
1774 +
1775 + tinyspy@3.0.2: {}
1776 +
1777 + toad-cache@3.7.4: {}
1778 +
1779 + tsconfck@3.1.6(typescript@5.9.3):
1780 + optionalDependencies:
1781 + typescript: 5.9.3
1782 +
1783 + tsx@4.23.11:
1784 + dependencies:
1785 + esbuild: 0.28.2
1786 + optionalDependencies:
1787 + fsevents: 2.3.3
1788 +
1789 + turndown@7.2.4:
1790 + dependencies:
1791 + '@mixmark-io/domino': 2.2.0
1792 +
1793 + typescript@5.9.3: {}
1794 +
1795 + uhyphen@0.2.0: {}
1796 +
1797 + undici-types@6.21.0: {}
1798 +
1799 + undici@7.29.0: {}
1800 +
1801 + vite-node@2.1.9(@types/node@22.20.1):
1802 + dependencies:
1803 + cac: 6.7.14
1804 + debug: 4.4.3
1805 + es-module-lexer: 1.7.0
1806 + pathe: 1.1.2
1807 + vite: 5.4.21(@types/node@22.20.1)
1808 + transitivePeerDependencies:
1809 + - '@types/node'
1810 + - less
1811 + - lightningcss
1812 + - sass
1813 + - sass-embedded
1814 + - stylus
1815 + - sugarss
1816 + - supports-color
1817 + - terser
1818 +
1819 + vite-tsconfig-paths@5.1.4(typescript@5.9.3)(vite@5.4.21(@types/node@22.20.1)):
1820 + dependencies:
1821 + debug: 4.4.3
1822 + globrex: 0.1.2
1823 + tsconfck: 3.1.6(typescript@5.9.3)
1824 + optionalDependencies:
1825 + vite: 5.4.21(@types/node@22.20.1)
1826 + transitivePeerDependencies:
1827 + - supports-color
1828 + - typescript
1829 +
1830 + vite@5.4.21(@types/node@22.20.1):
1831 + dependencies:
1832 + esbuild: 0.21.5
1833 + postcss: 8.5.26
1834 + rollup: 4.62.4
1835 + optionalDependencies:
1836 + '@types/node': 22.20.1
1837 + fsevents: 2.3.3
1838 +
1839 + vitest@2.1.9(@types/node@22.20.1):
1840 + dependencies:
1841 + '@vitest/expect': 2.1.9
1842 + '@vitest/mocker': 2.1.9(vite@5.4.21(@types/node@22.20.1))
1843 + '@vitest/pretty-format': 2.1.9
1844 + '@vitest/runner': 2.1.9
1845 + '@vitest/snapshot': 2.1.9
1846 + '@vitest/spy': 2.1.9
1847 + '@vitest/utils': 2.1.9
1848 + chai: 5.3.3
1849 + debug: 4.4.3
1850 + expect-type: 1.4.0
1851 + magic-string: 0.30.21
1852 + pathe: 1.1.2
1853 + std-env: 3.10.0
1854 + tinybench: 2.9.0
1855 + tinyexec: 0.3.2
1856 + tinypool: 1.1.1
1857 + tinyrainbow: 1.2.0
1858 + vite: 5.4.21(@types/node@22.20.1)
1859 + vite-node: 2.1.9(@types/node@22.20.1)
1860 + why-is-node-running: 2.3.0
1861 + optionalDependencies:
1862 + '@types/node': 22.20.1
1863 + transitivePeerDependencies:
1864 + - less
1865 + - lightningcss
1866 + - msw
1867 + - sass
1868 + - sass-embedded
1869 + - stylus
1870 + - sugarss
1871 + - supports-color
1872 + - terser
1873 +
1874 + why-is-node-running@2.3.0:
1875 + dependencies:
1876 + siginfo: 2.0.0
1877 + stackback: 0.0.2
1878 +
1879 + zod@3.25.76: {}
added pnpm-workspace.yaml +3 −0
@@ -0,0 +1,3 @@
1 +packages:
2 + - "apps/*"
3 + - "packages/*"
added test/fixtures/html/shopify-product.html +49 −0
@@ -0,0 +1,49 @@
1 +<!doctype html>
2 +<html lang="en">
3 +<head>
4 + <meta charset="utf-8">
5 + <title>Trellis Climbing Frame — GardenCo</title>
6 + <link rel="canonical" href="https://gardenco.example/products/trellis-frame">
7 + <meta name="description" content="A powder-coated steel trellis for climbing plants.">
8 + <meta property="og:title" content="Trellis Climbing Frame">
9 + <meta property="og:image" content="https://cdn.gardenco.example/trellis.jpg">
10 + <meta property="og:site_name" content="GardenCo">
11 + <script type="application/ld+json">
12 + {
13 + "@context": "https://schema.org",
14 + "@type": "Product",
15 + "name": "Trellis Climbing Frame",
16 + "sku": "TRL-2026",
17 + "image": "https://cdn.gardenco.example/trellis.jpg",
18 + "offers": {
19 + "@type": "Offer",
20 + "priceCurrency": "USD",
21 + "price": "49.00",
22 + "availability": "https://schema.org/InStock"
23 + }
24 + }
25 + </script>
26 +</head>
27 +<body>
28 + <header><nav><a href="/">Home</a> <a href="/shop">Shop</a></nav></header>
29 + <main>
30 + <article class="product">
31 + <h1>Trellis Climbing Frame</h1>
32 + <p class="price">$49.00</p>
33 + <img class="product" data-src="/img/trellis-800.jpg" alt="Trellis frame">
34 + <div id="desc">
35 + <p>The Trellis Climbing Frame is a powder-coated steel lattice that gives tendrils and vines a sturdy support to climb. It resists rust for many seasons of outdoor use.</p>
36 + <table class="specs">
37 + <thead><tr><th>Spec</th><th>Value</th></tr></thead>
38 + <tbody>
39 + <tr><td>Material</td><td>Powder-coated steel</td></tr>
40 + <tr><td>Height</td><td>180 cm</td></tr>
41 + </tbody>
42 + </table>
43 + </div>
44 + <a href="/products/trellis-frame/reviews">Read 24 reviews</a>
45 + </article>
46 + </main>
47 + <footer><a href="/privacy">Privacy policy</a></footer>
48 +</body>
49 +</html>
added tsconfig.base.json +34 −0
@@ -0,0 +1,34 @@
1 +{
2 + "$schema": "https://json.schemastore.org/tsconfig",
3 + "compilerOptions": {
4 + "target": "ES2022",
5 + "module": "NodeNext",
6 + "moduleResolution": "NodeNext",
7 + "lib": ["ES2023"],
8 + "types": ["node"],
9 + "strict": true,
10 + "noUncheckedIndexedAccess": true,
11 + "exactOptionalPropertyTypes": true,
12 + "noImplicitOverride": true,
13 + "noFallthroughCasesInSwitch": true,
14 + "noUnusedLocals": true,
15 + "noUnusedParameters": true,
16 + "verbatimModuleSyntax": true,
17 + "isolatedModules": true,
18 + "esModuleInterop": true,
19 + "forceConsistentCasingInFileNames": true,
20 + "skipLibCheck": true,
21 + "declaration": true,
22 + "declarationMap": true,
23 + "sourceMap": true,
24 + "composite": true,
25 + "baseUrl": ".",
26 + "paths": {
27 + "@tendril/shared": ["packages/shared/src/index.ts"],
28 + "@tendril/router": ["packages/router/src/index.ts"],
29 + "@tendril/egress": ["packages/egress/src/index.ts"],
30 + "@tendril/fetcher-http": ["packages/fetcher-http/src/index.ts"],
31 + "@tendril/extract": ["packages/extract/src/index.ts"]
32 + }
33 + }
34 +}
added tsconfig.json +7 −0
@@ -0,0 +1,7 @@
1 +{
2 + "extends": "./tsconfig.base.json",
3 + "compilerOptions": {
4 + "noEmit": true
5 + },
6 + "include": ["packages/*/src/**/*.ts", "apps/*/src/**/*.ts"]
7 +}
added vitest.config.ts +16 −0
@@ -0,0 +1,16 @@
1 +// author: simon-pierre boucher <contact@spboucher.ai>
2 +import { defineConfig } from "vitest/config";
3 +import tsconfigPaths from "vite-tsconfig-paths";
4 +
5 +export default defineConfig({
6 + plugins: [tsconfigPaths({ projects: ["./tsconfig.base.json"] })],
7 + test: {
8 + include: ["packages/**/*.test.ts", "apps/**/*.test.ts"],
9 + environment: "node",
10 + coverage: {
11 + provider: "v8",
12 + include: ["packages/**/src/**"],
13 + thresholds: { lines: 85, functions: 85, branches: 75, statements: 85 },
14 + },
15 + },
16 +});
17