Commit
Connector layer expansion: SDK (capabilities, domain policies, circuit breaker, adapters Shopify/WooCommerce/sitemap/RSS/PDF/schema.org), resumable backfills, schema-drift + health metrics, certificates provenance, multi-currency views, source registry (443 sources), connector:new scaffolder, admin explorer + coverage dashboard, docs; +119 connectors (212 total) across cards EU/JP, grading cert lookups, Shopify fleets NA/intl, numismatics, comics/toys/games, auction houses NA + EU/APAC, Asia marketplaces, watches, gated eBay/Etsy/Trade Me/Rakuten/Yahoo/Numista/CardTrader
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
modified
CLAUDE.md
+1 −1
@@ -183,7 +183,7 @@ One canonical asset graph containing every collectible, every important variatio | ||
| 183 | 183 | |
| 184 | 184 | - Package manager `pnpm`; `pnpm typecheck`, `pnpm test`, `pnpm build`. Workers run with `tsx`. |
| 185 | 185 | - Secrets only in `.env` (git-ignored) or the `mld` manifest on the cluster. Never in code or fixtures. |
| 186 | −- Add a connector: `connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts}` + `data/fixtures/<id>/*.json`, then `pnpm tsx scripts/build-registry.ts` and `pnpm db:seed`. | |
| 186 | +- Add a connector: `pnpm connector:new <id> --url … [--adapter shopify|woocommerce]` → `connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts,README.md}` + real fixtures in `data/fixtures/<id>/`, source entry in `data/sources/entries/<group>.json`, host policy in `connectors/domains.d/<group>.json`; then `pnpm registry`, `pnpm sources:build`, `pnpm db:seed`. Guide: `docs/connectors/ADDING_A_CONNECTOR.md`; catalogue: `docs/connectors/SOURCES.md`. | |
| 187 | 187 | - DB changes: edit `packages/database/src/schema/*`, run `pnpm db:generate`, commit the SQL migration. |
| 188 | 188 | - UI: dense, neutral, tabular numbers (`num` utility), `<Unavailable/>` for missing evidence, every number with its sample size / confidence / freshness when it is an estimate. |
| 189 | 189 | - Member accounts: email + password, email verification code and multi-factor code delivered through Resend (`RESEND_API_KEY`, `EMAIL_FROM`). |
modified
apps/web/src/app/admin/connectors/[id]/page.tsx
+107 −7
@@ -4,6 +4,7 @@ import { requireAdmin } from '@/lib/admin/auth'; | ||
| 4 | 4 | import { connectorDetail } from '@/lib/admin/queries'; |
| 5 | 5 | import { connectorAction, updateConnectorConfig } from '@/lib/admin/actions'; |
| 6 | 6 | import { AdminShell, ActionButton, StatusPill, Kpi, fmtTs, n } from '@/components/admin/shell'; |
| 7 | +import { HealthLabel, SourceLogo, Progress, Chip, Json } from '@/components/admin/connector-bits'; | |
| 7 | 8 | import { Table, th, td, tdNum } from '@/components/ui/primitives'; |
| 8 | 9 | |
| 9 | 10 | export default async function ConnectorInspectPage({ params }: { params: Promise<{ id: string }> }) { |
@@ -14,32 +15,119 @@ export default async function ConnectorInspectPage({ params }: { params: Promise | ||
| 14 | 15 | const c = d.connector; |
| 15 | 16 | const health = (d.health?.health ?? {}) as Record<string, unknown>; |
| 16 | 17 | const outputs = (d.outputs ?? {}) as Record<string, unknown>; |
| 18 | + const meta = (c.meta ?? {}) as Record<string, unknown>; | |
| 19 | + const activeBackfill = d.backfills.find((b) => b.status === 'running' || b.status === 'paused'); | |
| 20 | + const fieldStats = d.fieldStats as Array<{ field: string; day: string; total: number; nulls: number }>; | |
| 21 | + const today = new Date().toISOString().slice(0, 10); | |
| 22 | + const nullRates = Object.values( | |
| 23 | + fieldStats.reduce<Record<string, { field: string; today: [number, number]; base: [number, number] }>>((acc, r) => { | |
| 24 | + const e = (acc[r.field] ??= { field: r.field, today: [0, 0], base: [0, 0] }); | |
| 25 | + const t = r.day === today ? e.today : e.base; | |
| 26 | + t[0] += Number(r.total); | |
| 27 | + t[1] += Number(r.nulls); | |
| 28 | + return acc; | |
| 29 | + }, {}), | |
| 30 | + ); | |
| 31 | + const actions = [ | |
| 32 | + ['probe', 'Test', 'neutral'], | |
| 33 | + ['run', 'Run now', 'neutral'], | |
| 34 | + ['backfill', activeBackfill ? 'Continue backfill' : 'Backfill', 'neutral'], | |
| 35 | + ...(activeBackfill?.status === 'running' ? [['backfill_pause', 'Pause backfill', 'neutral']] : []), | |
| 36 | + ...(activeBackfill ? [['backfill_reset', 'Reset backfill', 'danger']] : []), | |
| 37 | + ['retry', 'Retry from cursor', 'neutral'], | |
| 38 | + c.status === 'active' ? ['pause', 'Pause', 'danger'] : ['resume', 'Resume', 'primary'], | |
| 39 | + c.status === 'maintenance' ? ['resume', 'End maintenance', 'primary'] : ['maintenance', 'Maintenance', 'neutral'], | |
| 40 | + ] as const; | |
| 17 | 41 | return ( |
| 18 | 42 | <AdminShell |
| 19 | 43 | current="/admin/connectors" |
| 20 | 44 | title={String(c.display_name)} |
| 21 | − subtitle={<>{String(c.id)} · source <span className="font-mono">{String(c.source_id)}</span> · {(c.engine_priority as string[]).join(' → ')} · categories {(c.categories as string[]).join(', ')} · <Link href="/admin/connectors" className="underline">back</Link></>} | |
| 45 | + subtitle={ | |
| 46 | + <> | |
| 47 | + <SourceLogo domain={String(meta.domain ?? '')} name={String(c.display_name)} /> {String(c.id)} · source <span className="font-mono">{String(c.source_id)}</span> · {String(meta.domain ?? '')} · {String(meta.country ?? (c.regions as string[])?.[0] ?? 'global')} · {(c.engine_priority as string[]).join(' → ')} · {String(meta.acquisitionMethod ?? '')} · categories {(c.categories as string[]).join(', ')} · <Link href="/admin/connectors" className="underline">back</Link> | |
| 48 | + </> | |
| 49 | + } | |
| 22 | 50 | actions={ |
| 23 | 51 | <div className="flex flex-wrap gap-1"> |
| 24 | − {(['run', 'probe', 'recrawl', 'retry', c.status === 'paused' ? 'resume' : 'pause'] as const).map((a) => ( | |
| 52 | + {actions.map(([a, label, tone]) => ( | |
| 25 | 53 | <form key={a} action={connectorAction}> |
| 26 | 54 | <input type="hidden" name="id" value={String(c.id)} /> |
| 27 | 55 | <input type="hidden" name="action" value={a} /> |
| 28 | − <ActionButton label={a} small={false} tone={a === 'pause' ? 'danger' : a === 'resume' ? 'primary' : 'neutral'} /> | |
| 56 | + <ActionButton label={label} small={false} tone={tone as 'neutral' | 'danger' | 'primary'} /> | |
| 29 | 57 | </form> |
| 30 | 58 | ))} |
| 31 | 59 | </div> |
| 32 | 60 | } |
| 33 | 61 | > |
| 62 | + <div className="mb-3 flex flex-wrap gap-1"> | |
| 63 | + {((meta.capabilities as string[]) ?? []).map((cap) => ( | |
| 64 | + <Chip key={cap} tone="index">{cap}</Chip> | |
| 65 | + ))} | |
| 66 | + {((meta.requires as string[]) ?? []).map((r) => ( | |
| 67 | + <Chip key={r} tone="alert">requires {r}</Chip> | |
| 68 | + ))} | |
| 69 | + <Chip>refresh {String(meta.refreshClass ?? '')} · {n(c.refresh_frequency_minutes)} min</Chip> | |
| 70 | + <Chip>history {String(meta.historicalDepth ?? 'none')}</Chip> | |
| 71 | + {meta.termsUrl ? ( | |
| 72 | + <a href={String(meta.termsUrl)} target="_blank" rel="noreferrer noopener" className="text-[10px] underline text-subtle">terms</a> | |
| 73 | + ) : null} | |
| 74 | + </div> | |
| 34 | 75 | <div className="grid grid-cols-2 gap-3 md:grid-cols-4 xl:grid-cols-6"> |
| 35 | − <Kpi label="Status" value={<StatusPill status={String(c.status)} />} sub={<>health <StatusPill status={(d.health?.status as string) ?? 'unknown'} /></>} /> | |
| 76 | + <Kpi label="Status" value={<StatusPill status={String(c.status)} />} sub={<>health <HealthLabel status={(d.health?.status as string) ?? 'unknown'} /></>} /> | |
| 36 | 77 | <Kpi label="Success 24h" value={health.success_rate_24h != null ? `${Math.round(Number(health.success_rate_24h) * 100)}%` : '—'} sub={`${n(health.pages_success)}/${n(health.pages_attempted)} pages`} /> |
| 37 | 78 | <Kpi label="Firecrawl / Scrapfly" value={`${health.firecrawl_success_rate != null ? Math.round(Number(health.firecrawl_success_rate) * 100) + '%' : '—'} / ${health.scrapfly_fallback_rate != null ? Math.round(Number(health.scrapfly_fallback_rate) * 100) + '%' : '—'}`} sub="success / fallback share" /> |
| 38 | 79 | <Kpi label="Parse failures" value={health.parse_failure_rate != null ? `${(Number(health.parse_failure_rate) * 100).toFixed(1)}%` : '—'} sub={`${n(health.duplicates_24h)} duplicates 24h`} /> |
| 39 | 80 | <Kpi label="Outputs" value={`${n(outputs.sales)} sales`} sub={`${n(outputs.listings)} listings · ${n(outputs.observations)} observations`} /> |
| 40 | − <Kpi label="Last success" value={fmtTs(c.last_success_at).slice(0, 16)} sub={`next ${fmtTs(c.next_run_at).slice(0, 16)}`} /> | |
| 81 | + <Kpi label="Last success" value={fmtTs(c.last_success_at).slice(0, 16)} sub={`next ${fmtTs(c.next_run_at).slice(0, 16)} · fresh ${health.data_freshness ? fmtTs(health.data_freshness).slice(0, 10) : '—'}`} /> | |
| 82 | + <Kpi label="Requests 24h" value={n(health.requests_24h)} sub={`${health.latency_ms_avg != null ? n(health.latency_ms_avg) + ' ms avg' : '—'} · ${n(health.challenges_24h)} challenges · ${n(health.rate_limited_24h)}× 429`} tone={Number(health.challenges_24h) > 0 ? 'alert' : undefined} /> | |
| 83 | + <Kpi label="HTTP errors 24h" value={Object.entries((health.http_errors as Record<string, number>) ?? {}).map(([k, v]) => `${k}×${v}`).join(' ') || '0'} sub={`${n(health.circuit_refusals_24h)} refused by circuit`} tone={Object.keys((health.http_errors as Record<string, number>) ?? {}).length ? 'loss' : undefined} /> | |
| 84 | + <Kpi label="Volume baseline" value={health.records_7d_daily_avg != null ? `${n(health.records_7d_daily_avg)}/day` : '—'} sub={`${n(health.records_24h)} raw in 24h`} tone={((health.anomalies as string[]) ?? []).some((a) => a.startsWith('result_count_collapse')) ? 'alert' : undefined} /> | |
| 85 | + <Kpi label="Certificates" value={n(outputs.certificates)} sub="cert numbers seen via this source" /> | |
| 41 | 86 | </div> |
| 42 | 87 | |
| 88 | + {activeBackfill || d.backfills.length ? ( | |
| 89 | + <section className="card mt-6"> | |
| 90 | + <header className="border-b border-border px-4 py-2.5"><h2 className="text-sm font-semibold">Historical backfill campaigns</h2></header> | |
| 91 | + <Table> | |
| 92 | + <thead><tr><th className={th}>Started</th><th className={th}>Status</th><th className={th}>Progress</th><th className={`${th} text-right`}>Pages</th><th className={`${th} text-right`}>Items</th><th className={th}>Reached date</th><th className={`${th} text-right`}>Runs</th><th className={`${th} text-right`}>Errors</th><th className={th}>Last cursor</th><th className={th}>Last error</th></tr></thead> | |
| 93 | + <tbody> | |
| 94 | + {d.backfills.map((b) => ( | |
| 95 | + <tr key={String(b.id)}> | |
| 96 | + <td className={td}>{fmtTs(b.started_at)}</td> | |
| 97 | + <td className={td}><StatusPill status={String(b.status)} /></td> | |
| 98 | + <td className={td}><Progress percent={b.percent as number | null} status={String(b.status)} /></td> | |
| 99 | + <td className={tdNum}>{n(b.pages_processed)}{b.total_pages ? ` / ${n(b.total_pages)}` : ''}</td> | |
| 100 | + <td className={tdNum}>{n(b.items_processed)}</td> | |
| 101 | + <td className={td}>{b.reached_date ? String(b.reached_date) : '—'}</td> | |
| 102 | + <td className={tdNum}>{n(b.runs)}</td> | |
| 103 | + <td className={tdNum}>{n(b.errors)}</td> | |
| 104 | + <td className={`${td} max-w-[260px] truncate font-mono text-[10px]`} title={JSON.stringify(b.last_cursor)}>{JSON.stringify(b.last_cursor)}</td> | |
| 105 | + <td className={`${td} max-w-[220px] truncate text-loss`} title={String(b.last_error ?? '')}>{String(b.last_error ?? '')}</td> | |
| 106 | + </tr> | |
| 107 | + ))} | |
| 108 | + </tbody> | |
| 109 | + </Table> | |
| 110 | + </section> | |
| 111 | + ) : null} | |
| 112 | + | |
| 113 | + {nullRates.length ? ( | |
| 114 | + <section className="card mt-6"> | |
| 115 | + <header className="border-b border-border px-4 py-2.5"><h2 className="text-sm font-semibold">Field presence (schema-drift monitor)</h2></header> | |
| 116 | + <div className="grid grid-cols-2 gap-x-6 gap-y-1 px-4 py-3 text-xs md:grid-cols-4 xl:grid-cols-6"> | |
| 117 | + {nullRates.map((f) => { | |
| 118 | + const tRate = f.today[0] ? f.today[1] / f.today[0] : null; | |
| 119 | + const bRate = f.base[0] ? f.base[1] / f.base[0] : null; | |
| 120 | + const drift = tRate !== null && bRate !== null && tRate - bRate >= 0.3; | |
| 121 | + return ( | |
| 122 | + <div key={f.field} className={drift ? 'text-alert' : ''}> | |
| 123 | + <span className="font-mono">{f.field}</span> <span className="text-subtle">null today {tRate === null ? '—' : `${Math.round(tRate * 100)}%`} · 7d {bRate === null ? '—' : `${Math.round(bRate * 100)}%`}</span> | |
| 124 | + </div> | |
| 125 | + ); | |
| 126 | + })} | |
| 127 | + </div> | |
| 128 | + </section> | |
| 129 | + ) : null} | |
| 130 | + | |
| 43 | 131 | {d.anomalies.length ? ( |
| 44 | 132 | <section className="card mt-6"> |
| 45 | 133 | <header className="border-b border-border px-4 py-2.5"><h2 className="text-sm font-semibold text-alert">Anomalies (recent runs)</h2></header> |
@@ -81,7 +169,10 @@ export default async function ConnectorInspectPage({ params }: { params: Promise | ||
| 81 | 169 | <li key={String(r.id)} className="px-4 py-2"> |
| 82 | 170 | <div className="flex items-center justify-between gap-2"><span className="font-mono text-[11px]">{String(r.kind)} · {String(r.engine)} · {String(r.http_status ?? '')}</span><span className="text-subtle">{fmtTs(r.fetched_at)}</span></div> |
| 83 | 171 | <a href={String(r.url)} target="_blank" rel="noreferrer noopener" className="block truncate text-muted hover:text-fg">{String(r.url)}</a> |
| 84 | − {r.process_error ? <p className="text-loss">{String(r.process_error)}</p> : <p className="text-[10px] text-subtle">{r.processed_at ? `processed ${fmtTs(r.processed_at)}` : 'unprocessed'}</p>} | |
| 172 | + <div className="flex items-center justify-between"> | |
| 173 | + {r.process_error ? <p className="text-loss">{String(r.process_error)}</p> : <p className="text-[10px] text-subtle">{r.processed_at ? `processed ${fmtTs(r.processed_at)}` : 'unprocessed'}</p>} | |
| 174 | + <Link href={`/admin/connectors/${String(c.id)}/raw/${String(r.id)}`} className="text-[10px] underline">view payload</Link> | |
| 175 | + </div> | |
| 85 | 176 | </li> |
| 86 | 177 | ))} |
| 87 | 178 | </ul> |
@@ -93,7 +184,10 @@ export default async function ConnectorInspectPage({ params }: { params: Promise | ||
| 93 | 184 | {d.normalizedSample.map((r) => ( |
| 94 | 185 | <li key={String(r.id)} className="px-4 py-2"> |
| 95 | 186 | <div className="flex items-center justify-between gap-2"><span className="truncate">{String(r.raw_title ?? '')}</span><StatusPill status={String(r.status)} /></div> |
| 96 | − <p className="text-[10px] text-subtle">{String(r.kind)} · {r.price ? `${String(r.price)} ${String(r.currency ?? '')}` : ''} · {r.match_method ? `${String(r.match_method)} ${r.match_confidence ? Math.round(Number(r.match_confidence) * 100) + '%' : ''}` : ''} {r.reject_reason ? `· ${String(r.reject_reason)}` : ''}</p> | |
| 187 | + <div className="flex items-center justify-between"> | |
| 188 | + <p className="text-[10px] text-subtle">{String(r.kind)} · {r.price ? `${String(r.price)} ${String(r.currency ?? '')}` : ''} · {r.match_method ? `${String(r.match_method)} ${r.match_confidence ? Math.round(Number(r.match_confidence) * 100) + '%' : ''}` : ''} {r.reject_reason ? `· ${String(r.reject_reason)}` : ''}</p> | |
| 189 | + <Link href={`/admin/connectors/${String(c.id)}/record/${String(r.id)}`} className="text-[10px] underline">view record</Link> | |
| 190 | + </div> | |
| 97 | 191 | </li> |
| 98 | 192 | ))} |
| 99 | 193 | </ul> |
@@ -128,6 +222,12 @@ export default async function ConnectorInspectPage({ params }: { params: Promise | ||
| 128 | 222 | </form> |
| 129 | 223 | </section> |
| 130 | 224 | </div> |
| 225 | + | |
| 226 | + <section className="card mt-6"> | |
| 227 | + <header className="border-b border-border px-4 py-2.5"><h2 className="text-sm font-semibold">Access & compliance notes (meta.json)</h2></header> | |
| 228 | + <div className="px-4 py-3 text-xs leading-relaxed text-muted">{String(meta.accessNotes ?? 'No accessNotes recorded — add them to meta.json (SPEC §36).')}</div> | |
| 229 | + <details className="border-t border-border px-4 py-2 text-xs"><summary className="cursor-pointer text-subtle">registry entry</summary><Json value={meta} className="mt-2" /></details> | |
| 230 | + </section> | |
| 131 | 231 | </AdminShell> |
| 132 | 232 | ); |
| 133 | 233 | } |
added
apps/web/src/app/admin/connectors/[id]/raw/[rawId]/page.tsx
+33 −0
@@ -0,0 +1,33 @@ | ||
| 1 | +import Link from 'next/link'; | |
| 2 | +import { notFound } from 'next/navigation'; | |
| 3 | +import { requireAdmin } from '@/lib/admin/auth'; | |
| 4 | +import { rawRecordDetail } from '@/lib/admin/queries'; | |
| 5 | +import { AdminShell, fmtTs } from '@/components/admin/shell'; | |
| 6 | +import { Json } from '@/components/admin/connector-bits'; | |
| 7 | + | |
| 8 | +/** "View last payload" (SPEC §29): the immutable raw capture with its provenance (engine, parser/connector versions, snapshot ref). */ | |
| 9 | +export default async function RawRecordPage({ params }: { params: Promise<{ id: string; rawId: string }> }) { | |
| 10 | + await requireAdmin(); | |
| 11 | + const { id, rawId } = await params; | |
| 12 | + const r = await rawRecordDetail(rawId); | |
| 13 | + if (!r || String(r.connector_id) !== id) notFound(); | |
| 14 | + const payload = r.payload as Record<string, unknown> | null; | |
| 15 | + const snapshot = payload && typeof payload === 'object' && typeof payload.snapshot === 'string' ? (payload.snapshot as string) : null; | |
| 16 | + const rest = snapshot && payload ? Object.fromEntries(Object.entries(payload).filter(([k]) => k !== 'snapshot')) : payload; | |
| 17 | + return ( | |
| 18 | + <AdminShell current="/admin/connectors" title={`Raw payload ${rawId}`} subtitle={<><Link href={`/admin/connectors/${id}`} className="underline">{id}</Link> · {String(r.kind)} · engine {String(r.engine)} · HTTP {String(r.http_status ?? '—')} · fetched {fmtTs(r.fetched_at)} · parser v{String(r.parser_version)} · connector v{String(r.connector_version)} · run {String(r.run_id ?? '—')}</>}> | |
| 19 | + <div className="mb-3 text-xs text-muted"> | |
| 20 | + <a href={String(r.url)} target="_blank" rel="noreferrer noopener" className="underline">{String(r.url)}</a> · external id <span className="font-mono">{String(r.external_id ?? '—')}</span> · hash <span className="font-mono">{String(r.content_hash).slice(0, 16)}…</span> | |
| 21 | + {r.snapshot_ref ? <> · snapshot on disk: <span className="font-mono">{String(r.snapshot_ref)}</span></> : null} | |
| 22 | + {r.process_error ? <div className="mt-1 text-loss">process error: {String(r.process_error)}</div> : <div className="mt-1 text-subtle">{r.processed_at ? `normalised ${fmtTs(r.processed_at)}` : 'not yet normalised'}</div>} | |
| 23 | + </div> | |
| 24 | + <Json value={rest} /> | |
| 25 | + {snapshot ? ( | |
| 26 | + <details className="mt-4 text-xs"> | |
| 27 | + <summary className="cursor-pointer text-subtle">inline snapshot ({snapshot.length.toLocaleString()} chars)</summary> | |
| 28 | + <Json value={snapshot.slice(0, 200_000)} className="mt-2" /> | |
| 29 | + </details> | |
| 30 | + ) : null} | |
| 31 | + </AdminShell> | |
| 32 | + ); | |
| 33 | +} | |
added
apps/web/src/app/admin/connectors/[id]/record/[recId]/page.tsx
+25 −0
@@ -0,0 +1,25 @@ | ||
| 1 | +import Link from 'next/link'; | |
| 2 | +import { notFound } from 'next/navigation'; | |
| 3 | +import { requireAdmin } from '@/lib/admin/auth'; | |
| 4 | +import { normalizedRecordDetail } from '@/lib/admin/queries'; | |
| 5 | +import { AdminShell, StatusPill, fmtTs } from '@/components/admin/shell'; | |
| 6 | +import { Json } from '@/components/admin/connector-bits'; | |
| 7 | + | |
| 8 | +/** "View last parsed record" (SPEC §29): the canonical normalised record and its resolution decision. */ | |
| 9 | +export default async function NormalizedRecordPage({ params }: { params: Promise<{ id: string; recId: string }> }) { | |
| 10 | + await requireAdmin(); | |
| 11 | + const { id, recId } = await params; | |
| 12 | + const r = await normalizedRecordDetail(recId); | |
| 13 | + if (!r || String(r.connector_id) !== id) notFound(); | |
| 14 | + return ( | |
| 15 | + <AdminShell current="/admin/connectors" title={`Normalised record ${recId}`} subtitle={<><Link href={`/admin/connectors/${id}`} className="underline">{id}</Link> · {String(r.kind)} · <StatusPill status={String(r.status)} /> · created {fmtTs(r.created_at)} · from raw <Link href={`/admin/connectors/${id}/raw/${String(r.raw_record_id)}`} className="underline">{String(r.raw_record_id)}</Link></>}> | |
| 16 | + <div className="mb-3 grid gap-1 text-xs text-muted md:grid-cols-2"> | |
| 17 | + <div>match: <span className="font-mono">{String(r.match_method ?? '—')}</span> {r.match_confidence ? `(${Math.round(Number(r.match_confidence) * 100)}%)` : ''}</div> | |
| 18 | + <div>asset: {r.asset_id ? <Link href={`/admin/data-quality?asset=${String(r.asset_id)}`} className="font-mono underline">{String(r.asset_id)}</Link> : '—'} · variant <span className="font-mono">{String(r.variant_id ?? '—')}</span></div> | |
| 19 | + <div>target: <span className="font-mono">{String(r.target_id ?? '—')}</span></div> | |
| 20 | + <div className={r.reject_reason ? 'text-loss' : ''}>reason: {String(r.reject_reason ?? '—')}</div> | |
| 21 | + </div> | |
| 22 | + <Json value={r.payload} /> | |
| 23 | + </AdminShell> | |
| 24 | + ); | |
| 25 | +} | |
modified
apps/web/src/app/admin/connectors/page.tsx
+184 −42
@@ -3,75 +3,210 @@ import { requireAdmin } from '@/lib/admin/auth'; | ||
| 3 | 3 | import { connectorsOverview } from '@/lib/admin/queries'; |
| 4 | 4 | import { connectorAction } from '@/lib/admin/actions'; |
| 5 | 5 | import { AdminShell, ActionButton, StatusPill, fmtTs, n } from '@/components/admin/shell'; |
| 6 | +import { HealthLabel, SourceLogo, Pct, Progress, Chip } from '@/components/admin/connector-bits'; | |
| 6 | 7 | import { Table, th, td, tdNum } from '@/components/ui/primitives'; |
| 7 | 8 | |
| 8 | −function pct(v: unknown): string { | |
| 9 | − const x = Number(v); | |
| 10 | − return Number.isFinite(x) ? `${Math.round(x * 100)}%` : '—'; | |
| 9 | +type Row = Record<string, unknown>; | |
| 10 | + | |
| 11 | +function str(v: unknown): string { | |
| 12 | + return v === null || v === undefined ? '' : String(v); | |
| 11 | 13 | } |
| 12 | 14 | |
| 13 | −export default async function ConnectorsPage() { | |
| 15 | +/** Connector explorer (SPEC §29): every connector with source, geography, acquisition, health, usage, backfill and actions. */ | |
| 16 | +export default async function ConnectorsPage({ searchParams }: { searchParams: Promise<Record<string, string | undefined>> }) { | |
| 14 | 17 | await requireAdmin(); |
| 15 | − const rows = await connectorsOverview(); | |
| 18 | + const sp = await searchParams; | |
| 19 | + const rowsAll = (await connectorsOverview()) as Row[]; | |
| 20 | + const q = (sp.q ?? '').toLowerCase(); | |
| 21 | + const status = sp.status ?? ''; | |
| 22 | + const category = sp.category ?? ''; | |
| 23 | + const country = sp.country ?? ''; | |
| 24 | + const rows = rowsAll.filter((r) => { | |
| 25 | + const meta = (r.meta ?? {}) as Row; | |
| 26 | + if (status && String(r.health_status ?? 'unknown') !== status && String(r.status) !== status) return false; | |
| 27 | + if (category && !((r.categories as string[]) ?? []).includes(category)) return false; | |
| 28 | + if (country && String(meta.country ?? (r.regions as string[])?.[0] ?? '') !== country) return false; | |
| 29 | + if (q && !`${str(r.id)} ${str(r.display_name)} ${str(meta.domain)} ${((r.categories as string[]) ?? []).join(' ')}`.toLowerCase().includes(q)) return false; | |
| 30 | + return true; | |
| 31 | + }); | |
| 32 | + const counts = rowsAll.reduce<Record<string, number>>((acc, r) => { | |
| 33 | + const k = String(r.health_status ?? 'unknown'); | |
| 34 | + acc[k] = (acc[k] ?? 0) + 1; | |
| 35 | + return acc; | |
| 36 | + }, {}); | |
| 37 | + const categories = [...new Set(rowsAll.flatMap((r) => (r.categories as string[]) ?? []))].sort(); | |
| 38 | + const countries = [...new Set(rowsAll.map((r) => String(((r.meta ?? {}) as Row).country ?? (r.regions as string[])?.[0] ?? 'global')))].sort(); | |
| 39 | + | |
| 16 | 40 | return ( |
| 17 | − <AdminShell current="/admin/connectors" title="Connector control center" subtitle={`${rows.length} connectors registered · actions enqueue jobs on the crawl.run queue consumed by the worker`}> | |
| 18 | − <div className="card"> | |
| 41 | + <AdminShell | |
| 42 | + current="/admin/connectors" | |
| 43 | + title="Connector explorer" | |
| 44 | + subtitle={ | |
| 45 | + <> | |
| 46 | + {rowsAll.length} connectors · {Object.entries(counts).map(([k, v]) => `${v} ${k}`).join(' · ')} · <Link href="/admin/coverage" className="underline">coverage dashboard</Link> · <Link href="/api/admin/connectors" className="underline">JSON</Link> | |
| 47 | + </> | |
| 48 | + } | |
| 49 | + actions={ | |
| 50 | + <form className="flex flex-wrap items-center gap-1 text-xs" method="get"> | |
| 51 | + <input name="q" defaultValue={sp.q ?? ''} placeholder="search id, domain, category" className="rounded-md border border-border bg-sunken px-2 py-1 text-xs" /> | |
| 52 | + <select name="status" defaultValue={status} className="rounded-md border border-border bg-sunken px-2 py-1 text-xs"> | |
| 53 | + <option value="">any health</option> | |
| 54 | + {['healthy', 'degraded', 'failing', 'disabled', 'paused', 'maintenance', 'unknown'].map((s) => ( | |
| 55 | + <option key={s} value={s}>{s}</option> | |
| 56 | + ))} | |
| 57 | + </select> | |
| 58 | + <select name="category" defaultValue={category} className="rounded-md border border-border bg-sunken px-2 py-1 text-xs"> | |
| 59 | + <option value="">any category</option> | |
| 60 | + {categories.map((c) => ( | |
| 61 | + <option key={c} value={c}>{c}</option> | |
| 62 | + ))} | |
| 63 | + </select> | |
| 64 | + <select name="country" defaultValue={country} className="rounded-md border border-border bg-sunken px-2 py-1 text-xs"> | |
| 65 | + <option value="">any country</option> | |
| 66 | + {countries.map((c) => ( | |
| 67 | + <option key={c} value={c}>{c}</option> | |
| 68 | + ))} | |
| 69 | + </select> | |
| 70 | + <button type="submit" className="rounded-md border border-border px-2 py-1 text-xs hover:bg-inset">Filter</button> | |
| 71 | + </form> | |
| 72 | + } | |
| 73 | + > | |
| 74 | + <div className="card overflow-x-auto"> | |
| 19 | 75 | <Table> |
| 20 | 76 | <thead> |
| 21 | 77 | <tr> |
| 22 | − <th className={th}>Connector</th> | |
| 78 | + <th className={th}>Source</th> | |
| 79 | + <th className={th}>Category · country</th> | |
| 23 | 80 | <th className={th}>Health</th> |
| 24 | 81 | <th className={th}>Status</th> |
| 25 | − <th className={`${th} text-right`}>Pages/min</th> | |
| 82 | + <th className={th}>Acquisition</th> | |
| 83 | + <th className={th}>Last sync</th> | |
| 84 | + <th className={`${th} text-right`}>Items 24h</th> | |
| 26 | 85 | <th className={`${th} text-right`}>Errors</th> |
| 27 | − <th className={th}>Last crawl</th> | |
| 28 | − <th className={`${th} text-right`}>Records 24h</th> | |
| 29 | − <th className={`${th} text-right`}>Dup.</th> | |
| 30 | − <th className={`${th} text-right`}>FC rate</th> | |
| 31 | − <th className={`${th} text-right`}>SF rate</th> | |
| 32 | − <th className={`${th} text-right`}>Parser conf.</th> | |
| 33 | − <th className={th}>Schema</th> | |
| 86 | + <th className={`${th} text-right`}>FC · SF credits</th> | |
| 87 | + <th className={th}>Rate limit</th> | |
| 88 | + <th className={th}>Priority</th> | |
| 89 | + <th className={th}>Backfill</th> | |
| 34 | 90 | <th className={th}>Actions</th> |
| 35 | 91 | </tr> |
| 36 | 92 | </thead> |
| 37 | 93 | <tbody> |
| 38 | − {rows.length === 0 ? <tr><td className={td} colSpan={13}><span className="text-subtle">No connectors in the registry yet. Add meta.json files under connectors/ and run pnpm db:seed.</span></td></tr> : null} | |
| 94 | + {rows.length === 0 ? ( | |
| 95 | + <tr> | |
| 96 | + <td className={td} colSpan={13}> | |
| 97 | + <span className="text-subtle">No connector matches. Add meta.json files under connectors/, run pnpm registry && pnpm db:seed.</span> | |
| 98 | + </td> | |
| 99 | + </tr> | |
| 100 | + ) : null} | |
| 39 | 101 | {rows.map((r) => { |
| 40 | − const health = (r.health ?? {}) as Record<string, unknown>; | |
| 102 | + const meta = (r.meta ?? {}) as Row; | |
| 103 | + const health = (r.health ?? {}) as Row; | |
| 41 | 104 | const anomalies = Array.isArray(r.last_anomalies) ? (r.last_anomalies as string[]) : []; |
| 42 | − const drift = anomalies.some((a) => /schema|selector|missing_field|redesign/.test(a)); | |
| 105 | + const drift = ((health.schema_drift as string[]) ?? []).length > 0 || anomalies.some((a) => /schema_drift|selector|pagination_failure|result_count_collapse/.test(a)); | |
| 106 | + const httpErrors = Object.values((health.http_errors as Record<string, number>) ?? {}).reduce((a, b) => a + b, 0); | |
| 107 | + const cats = (r.categories as string[]) ?? []; | |
| 108 | + const id = str(r.id); | |
| 109 | + const domain = str(meta.domain); | |
| 110 | + const country = str(meta.country ?? (r.regions as string[])?.[0] ?? 'global'); | |
| 111 | + const refresh = Number(r.refresh_frequency_minutes); | |
| 112 | + const refreshClass = str(meta.refreshClass) || (refresh <= 5 ? 'hot' : refresh <= 60 ? 'active' : refresh <= 1440 ? 'normal' : 'archive'); | |
| 113 | + const missing = (health.missing_requirements as string[]) ?? []; | |
| 43 | 114 | return ( |
| 44 | − <tr key={String(r.id)} className="align-top"> | |
| 115 | + <tr key={id} className="align-top"> | |
| 116 | + <td className={td}> | |
| 117 | + <div className="flex items-center gap-2"> | |
| 118 | + <SourceLogo domain={domain} name={str(r.display_name)} logoUrl={str(meta.logoUrl) || null} /> | |
| 119 | + <div> | |
| 120 | + <Link href={`/admin/connectors/${id}`} className="font-medium hover:underline">{str(r.display_name)}</Link> | |
| 121 | + <div className="text-[11px] text-subtle"> | |
| 122 | + {domain ? ( | |
| 123 | + <a href={`https://${domain}`} target="_blank" rel="noreferrer noopener" className="hover:underline">{domain}</a> | |
| 124 | + ) : ( | |
| 125 | + id | |
| 126 | + )} | |
| 127 | + {' · '} | |
| 128 | + {id} | |
| 129 | + </div> | |
| 130 | + </div> | |
| 131 | + </div> | |
| 132 | + </td> | |
| 133 | + <td className={td}> | |
| 134 | + <div className="flex max-w-[220px] flex-wrap gap-1"> | |
| 135 | + {cats.slice(0, 3).map((c) => ( | |
| 136 | + <Chip key={c}>{c}</Chip> | |
| 137 | + ))} | |
| 138 | + {cats.length > 3 ? <Chip>+{cats.length - 3}</Chip> : null} | |
| 139 | + <Chip tone="index">{country}</Chip> | |
| 140 | + </div> | |
| 141 | + </td> | |
| 142 | + <td className={td}> | |
| 143 | + <HealthLabel status={str(r.health_status) || 'unknown'} /> | |
| 144 | + <div className="text-[10px] text-subtle"> | |
| 145 | + {health.success_rate_24h != null ? <>success <Pct value={health.success_rate_24h} /></> : 'no runs 24h'} | |
| 146 | + {r.health_at ? ` · ${fmtTs(r.health_at).slice(5, 16)}` : ''} | |
| 147 | + </div> | |
| 148 | + {drift ? <div className="text-[10px] text-alert">schema drift?</div> : null} | |
| 149 | + {missing.length ? <div className="text-[10px] text-alert">needs {missing.join(', ')}</div> : null} | |
| 150 | + </td> | |
| 151 | + <td className={td}> | |
| 152 | + <StatusPill status={str(r.status)} /> | |
| 153 | + {r.last_run_status ? <div className="text-[10px] text-subtle">run: {str(r.last_run_status)}</div> : null} | |
| 154 | + </td> | |
| 155 | + <td className={td}> | |
| 156 | + <div className="text-xs">{str(meta.acquisitionMethod) || (r.engine_priority as string[]).join(' → ')}</div> | |
| 157 | + <div className="text-[10px] text-subtle">{(r.engine_priority as string[]).join('→')} · {refreshClass} · every {n(refresh)} min</div> | |
| 158 | + </td> | |
| 45 | 159 | <td className={td}> |
| 46 | − <Link href={`/admin/connectors/${String(r.id)}`} className="font-medium hover:underline">{String(r.display_name)}</Link> | |
| 47 | − <div className="text-[11px] text-subtle">{String(r.id)} · {(r.engine_priority as string[]).join('→')} · {String(r.priority)} · every {n(r.refresh_frequency_minutes)} min</div> | |
| 160 | + {fmtTs(r.last_started_at)} | |
| 161 | + <div className="text-[10px] text-subtle">ok: {fmtTs(r.last_success_at).slice(0, 16)}</div> | |
| 162 | + <div className="text-[10px] text-subtle">fresh: {health.data_freshness ? fmtTs(health.data_freshness).slice(0, 10) : '—'}</div> | |
| 163 | + </td> | |
| 164 | + <td className={tdNum}> | |
| 165 | + {n(r.records_24h)} | |
| 166 | + <div className="text-[10px] text-subtle">{n(r.normalized_24h)} norm. · {n(r.duplicates_24h)} dup.</div> | |
| 167 | + <div className="text-[10px] text-subtle">{n(r.pages_per_min, 1)} pages/min</div> | |
| 168 | + </td> | |
| 169 | + <td className={`${tdNum} ${r.last_error || httpErrors ? 'text-loss' : ''}`} title={str(r.last_error)}> | |
| 170 | + {httpErrors || (r.last_error ? 1 : 0)} | |
| 171 | + {anomalies.length ? <div className="text-[10px] text-alert">{anomalies.length} anomal.</div> : null} | |
| 172 | + {Number(health.challenges_24h) > 0 ? <div className="text-[10px] text-alert">{n(health.challenges_24h)} challenges</div> : null} | |
| 173 | + </td> | |
| 174 | + <td className={tdNum}> | |
| 175 | + {n(r.fc_credits_24h)} · {n(r.sf_credits_24h)} | |
| 176 | + <div className="text-[10px] text-subtle"> | |
| 177 | + FC <Pct value={health.firecrawl_success_rate} /> · SF <Pct value={health.scrapfly_fallback_rate} /> | |
| 178 | + </div> | |
| 48 | 179 | </td> |
| 49 | − <td className={td}><StatusPill status={(r.health_status as string) ?? 'unknown'} />{r.health_at ? <div className="text-[10px] text-subtle">{fmtTs(r.health_at)}</div> : null}</td> | |
| 50 | − <td className={td}><StatusPill status={String(r.status)} />{r.last_run_status ? <div className="text-[10px] text-subtle">run: {String(r.last_run_status)}</div> : null}</td> | |
| 51 | − <td className={tdNum}>{n(r.pages_per_min, 1)}</td> | |
| 52 | − <td className={`${tdNum} ${r.last_error ? 'text-loss' : ''}`} title={String(r.last_error ?? '')}>{r.last_error ? '1' : '0'}{anomalies.length ? <div className="text-[10px] text-alert">{anomalies.length} anomal.</div> : null}</td> | |
| 53 | − <td className={td}>{fmtTs(r.last_started_at)}<div className="text-[10px] text-subtle">ok: {fmtTs(r.last_success_at)}</div></td> | |
| 54 | − <td className={tdNum}>{n(r.records_24h)}<div className="text-[10px] text-subtle">{n(r.normalized_24h)} norm.</div></td> | |
| 55 | − <td className={tdNum}>{n(r.duplicates_24h)}</td> | |
| 56 | − <td className={tdNum}>{pct(health.firecrawl_success_rate)}</td> | |
| 57 | − <td className={tdNum}>{pct(health.scrapfly_fallback_rate)}</td> | |
| 58 | − <td className={tdNum}>{pct(r.parser_confidence)}</td> | |
| 59 | − <td className={td}>v{String(r.schema_version)}{drift ? <div className="text-[10px] text-alert">drift?</div> : null}</td> | |
| 60 | 180 | <td className={td}> |
| 61 | − <div className="flex flex-wrap gap-1"> | |
| 62 | − {(['run', 'probe', 'recrawl', 'retry'] as const).map((a) => ( | |
| 181 | + <div className="text-xs">{health.latency_ms_avg != null ? `${n(health.latency_ms_avg)} ms` : '—'}</div> | |
| 182 | + <div className="text-[10px] text-subtle">{Number(health.rate_limited_24h) > 0 ? `${n(health.rate_limited_24h)}× 429` : 'no 429'} · {n(health.requests_24h)} req</div> | |
| 183 | + </td> | |
| 184 | + <td className={td}> | |
| 185 | + <Chip tone={str(r.priority) === 'high' ? 'index' : 'neutral'}>{str(r.priority)}</Chip> | |
| 186 | + <div className="text-[10px] text-subtle">trust {source_trust(meta)}</div> | |
| 187 | + </td> | |
| 188 | + <td className={td}>{r.backfill_id ? <Progress percent={r.backfill_percent as number | null} status={str(r.backfill_status)} /> : <span className="text-[10px] text-subtle">{str(meta.historicalDepth) && str(meta.historicalDepth) !== 'none' ? `history: ${str(meta.historicalDepth)}` : '—'}</span>}</td> | |
| 189 | + <td className={td}> | |
| 190 | + <div className="flex max-w-[210px] flex-wrap gap-1"> | |
| 191 | + {( | |
| 192 | + [ | |
| 193 | + ['probe', 'Test'], | |
| 194 | + ['run', 'Run now'], | |
| 195 | + ['backfill', 'Backfill'], | |
| 196 | + ] as const | |
| 197 | + ).map(([a, label]) => ( | |
| 63 | 198 | <form key={a} action={connectorAction}> |
| 64 | − <input type="hidden" name="id" value={String(r.id)} /> | |
| 199 | + <input type="hidden" name="id" value={id} /> | |
| 65 | 200 | <input type="hidden" name="action" value={a} /> |
| 66 | − <ActionButton label={a} /> | |
| 201 | + <ActionButton label={label} /> | |
| 67 | 202 | </form> |
| 68 | 203 | ))} |
| 69 | 204 | <form action={connectorAction}> |
| 70 | − <input type="hidden" name="id" value={String(r.id)} /> | |
| 71 | − <input type="hidden" name="action" value={r.status === 'paused' ? 'resume' : 'pause'} /> | |
| 72 | − <ActionButton label={r.status === 'paused' ? 'resume' : 'pause'} tone={r.status === 'paused' ? 'primary' : 'danger'} /> | |
| 205 | + <input type="hidden" name="id" value={id} /> | |
| 206 | + <input type="hidden" name="action" value={r.status === 'active' ? 'pause' : 'resume'} /> | |
| 207 | + <ActionButton label={r.status === 'active' ? 'Pause' : 'Resume'} tone={r.status === 'active' ? 'danger' : 'primary'} /> | |
| 73 | 208 | </form> |
| 74 | − <Link href={`/admin/connectors/${String(r.id)}`} className="rounded-md border border-border px-2 py-0.5 text-[11px] font-medium hover:bg-inset">inspect</Link> | |
| 209 | + <Link href={`/admin/connectors/${id}`} className="rounded-md border border-border px-2 py-0.5 text-[11px] font-medium hover:bg-inset">Logs</Link> | |
| 75 | 210 | </div> |
| 76 | 211 | </td> |
| 77 | 212 | </tr> |
@@ -80,7 +215,14 @@ export default async function ConnectorsPage() { | ||
| 80 | 215 | </tbody> |
| 81 | 216 | </Table> |
| 82 | 217 | </div> |
| 83 | − <p className="mt-3 text-[11px] text-subtle">run = incremental crawl · probe = 25-record smoke test · recrawl = backfill mode · retry = re-run from the last cursor. FC = Firecrawl success rate, SF = share of pages that needed Scrapfly (from the 24 h health snapshot).</p> | |
| 218 | + <p className="mt-3 text-[11px] text-subtle"> | |
| 219 | + Test = 25-record probe · Run now = incremental crawl · Backfill = resumable historical campaign (progress in the Backfill column) · Pause/Resume flips scheduling; Maintenance is available on the connector page. FC/SF = Firecrawl/Scrapfly credits spent in 24 h and success/fallback rates. Health labels: UP · DEGRADED · BROKEN · DISABLED · MAINTENANCE. | |
| 220 | + </p> | |
| 84 | 221 | </AdminShell> |
| 85 | 222 | ); |
| 86 | 223 | } |
| 224 | + | |
| 225 | +function source_trust(meta: Row): string { | |
| 226 | + const t = Number(meta.trustScore); | |
| 227 | + return Number.isFinite(t) ? `${Math.round(t * 100)}%` : '—'; | |
| 228 | +} | |
added
apps/web/src/app/api/admin/connectors/route.ts
+58 −0
@@ -0,0 +1,58 @@ | ||
| 1 | +import { NextResponse } from 'next/server'; | |
| 2 | +import { connectorsOverview } from '@/lib/admin/queries'; | |
| 3 | + | |
| 4 | +export const runtime = 'nodejs'; | |
| 5 | +export const dynamic = 'force-dynamic'; | |
| 6 | + | |
| 7 | +const LABEL: Record<string, string> = { healthy: 'UP', degraded: 'DEGRADED', failing: 'BROKEN', paused: 'DISABLED', disabled: 'DISABLED', maintenance: 'MAINTENANCE', unknown: 'UNKNOWN' }; | |
| 8 | + | |
| 9 | +/** Connector health API (SPEC §12): one object per connector with status, usage, errors, freshness and backfill progress. */ | |
| 10 | +export async function GET() { | |
| 11 | + const rows = await connectorsOverview(); | |
| 12 | + return NextResponse.json({ | |
| 13 | + as_of: new Date().toISOString(), | |
| 14 | + connectors: rows.map((r) => { | |
| 15 | + const h = (r.health ?? {}) as Record<string, unknown>; | |
| 16 | + const m = (r.meta ?? {}) as Record<string, unknown>; | |
| 17 | + return { | |
| 18 | + id: r.id, | |
| 19 | + name: r.display_name, | |
| 20 | + source: r.source_id, | |
| 21 | + domain: m.domain ?? null, | |
| 22 | + country: m.country ?? null, | |
| 23 | + categories: r.categories, | |
| 24 | + capabilities: m.capabilities ?? [], | |
| 25 | + acquisition: m.acquisitionMethod ?? (r.engine_priority as string[]).join('>'), | |
| 26 | + engines: r.engine_priority, | |
| 27 | + priority: r.priority, | |
| 28 | + refresh_minutes: r.refresh_frequency_minutes, | |
| 29 | + refresh_class: m.refreshClass ?? null, | |
| 30 | + status: r.status, | |
| 31 | + health: r.health_status ?? 'unknown', | |
| 32 | + health_label: LABEL[String(r.health_status ?? 'unknown')] ?? 'UNKNOWN', | |
| 33 | + health_at: r.health_at, | |
| 34 | + success_rate_24h: h.success_rate_24h ?? null, | |
| 35 | + requests_24h: h.requests_24h ?? null, | |
| 36 | + items_24h: r.records_24h ?? 0, | |
| 37 | + normalized_24h: r.normalized_24h ?? 0, | |
| 38 | + duplicates_24h: r.duplicates_24h ?? 0, | |
| 39 | + latest_success: r.last_success_at, | |
| 40 | + last_run: r.last_started_at, | |
| 41 | + last_run_status: r.last_run_status, | |
| 42 | + last_error: r.last_error, | |
| 43 | + http_errors: h.http_errors ?? {}, | |
| 44 | + parse_failure_rate: h.parse_failure_rate ?? null, | |
| 45 | + schema_drift: h.schema_drift ?? [], | |
| 46 | + latency_ms_avg: h.latency_ms_avg ?? null, | |
| 47 | + rate_limited_24h: h.rate_limited_24h ?? 0, | |
| 48 | + challenges_24h: h.challenges_24h ?? 0, | |
| 49 | + data_freshness: h.data_freshness ?? null, | |
| 50 | + firecrawl: { credits_24h: r.fc_credits_24h ?? 0, success_rate: h.firecrawl_success_rate ?? null }, | |
| 51 | + scrapfly: { credits_24h: r.sf_credits_24h ?? 0, fallback_rate: h.scrapfly_fallback_rate ?? null }, | |
| 52 | + missing_requirements: h.missing_requirements ?? [], | |
| 53 | + backfill: r.backfill_id ? { id: r.backfill_id, status: r.backfill_status, percent: r.backfill_percent, pages: r.backfill_pages, items: r.backfill_items, reached_date: r.backfill_reached_date, updated_at: r.backfill_updated_at } : null, | |
| 54 | + anomalies: r.last_anomalies ?? [], | |
| 55 | + }; | |
| 56 | + }), | |
| 57 | + }); | |
| 58 | +} | |
added
apps/web/src/components/admin/connector-bits.tsx
+52 −0
@@ -0,0 +1,52 @@ | ||
| 1 | +import type { ReactNode } from 'react'; | |
| 2 | +import { cn } from '@/lib/format'; | |
| 3 | + | |
| 4 | +/** Operator labels for health statuses (SPEC §12): UP · DEGRADED · BROKEN · DISABLED · MAINTENANCE. */ | |
| 5 | +const LABELS: Record<string, { label: string; cls: string }> = { | |
| 6 | + healthy: { label: 'UP', cls: 'bg-gain-bg text-gain' }, | |
| 7 | + degraded: { label: 'DEGRADED', cls: 'bg-alert-bg text-alert' }, | |
| 8 | + failing: { label: 'BROKEN', cls: 'bg-loss-bg text-loss' }, | |
| 9 | + paused: { label: 'DISABLED', cls: 'bg-inset text-muted' }, | |
| 10 | + disabled: { label: 'DISABLED', cls: 'bg-inset text-muted' }, | |
| 11 | + maintenance: { label: 'MAINTENANCE', cls: 'bg-index-bg text-index' }, | |
| 12 | + unknown: { label: 'UNKNOWN', cls: 'bg-inset text-subtle' }, | |
| 13 | +}; | |
| 14 | + | |
| 15 | +export function HealthLabel({ status, small = true }: { status: string | null | undefined; small?: boolean }) { | |
| 16 | + const l = LABELS[status ?? 'unknown'] ?? LABELS.unknown!; | |
| 17 | + return <span className={cn('inline-flex rounded-sm font-semibold tracking-wide', small ? 'px-1.5 py-0.5 text-[10px]' : 'px-2 py-1 text-xs', l.cls)}>{l.label}</span>; | |
| 18 | +} | |
| 19 | + | |
| 20 | +/** Source favicon (Google s2 service, falls back to an initial). */ | |
| 21 | +export function SourceLogo({ domain, name, logoUrl, size = 18 }: { domain: string | null | undefined; name: string; logoUrl?: string | null; size?: number }) { | |
| 22 | + const src = logoUrl ?? (domain ? `https://www.google.com/s2/favicons?domain=${encodeURIComponent(domain)}&sz=${size * 2}` : null); | |
| 23 | + if (!src) return <span className="inline-flex h-[18px] w-[18px] items-center justify-center rounded-sm bg-inset text-[10px] font-semibold text-muted">{name.slice(0, 1).toUpperCase()}</span>; | |
| 24 | + // eslint-disable-next-line @next/next/no-img-element | |
| 25 | + return <img src={src} alt="" width={size} height={size} loading="lazy" className="inline-block rounded-sm bg-white/5" />; | |
| 26 | +} | |
| 27 | + | |
| 28 | +export function Pct({ value, digits = 0 }: { value: unknown; digits?: number }) { | |
| 29 | + const x = Number(value); | |
| 30 | + return <>{Number.isFinite(x) ? `${(x * 100).toFixed(digits)}%` : '—'}</>; | |
| 31 | +} | |
| 32 | + | |
| 33 | +/** Thin progress bar for backfill campaigns. */ | |
| 34 | +export function Progress({ percent, status }: { percent: number | null | undefined; status?: string | null }) { | |
| 35 | + const p = percent === null || percent === undefined ? null : Math.max(0, Math.min(100, Number(percent))); | |
| 36 | + return ( | |
| 37 | + <div className="min-w-[72px]"> | |
| 38 | + <div className="h-1.5 w-full overflow-hidden rounded-full bg-inset"> | |
| 39 | + <div className={cn('h-full rounded-full', status === 'completed' ? 'bg-gain' : status === 'failed' ? 'bg-loss' : status === 'paused' ? 'bg-alert' : 'bg-index')} style={{ width: `${p ?? (status === 'running' ? 8 : 0)}%` }} /> | |
| 40 | + </div> | |
| 41 | + <div className="mt-0.5 text-[10px] text-subtle">{p === null ? (status ?? '—') : `${p.toFixed(0)}% · ${status ?? ''}`}</div> | |
| 42 | + </div> | |
| 43 | + ); | |
| 44 | +} | |
| 45 | + | |
| 46 | +export function Json({ value, className }: { value: unknown; className?: string }) { | |
| 47 | + return <pre className={cn('max-h-[70vh] overflow-auto rounded-md border border-border bg-sunken p-3 font-mono text-[11px] leading-relaxed', className)}>{typeof value === 'string' ? value : JSON.stringify(value, null, 2)}</pre>; | |
| 48 | +} | |
| 49 | + | |
| 50 | +export function Chip({ children, tone = 'neutral' }: { children: ReactNode; tone?: 'neutral' | 'index' | 'alert' }) { | |
| 51 | + return <span className={cn('inline-flex rounded-sm px-1.5 py-0.5 text-[10px]', tone === 'index' ? 'bg-index-bg text-index' : tone === 'alert' ? 'bg-alert-bg text-alert' : 'bg-inset text-muted')}>{children}</span>; | |
| 52 | +} | |
modified
apps/web/src/components/admin/shell.tsx
+2 −1
@@ -5,6 +5,7 @@ import { cn } from '@/lib/format'; | ||
| 5 | 5 | export const ADMIN_NAV = [ |
| 6 | 6 | { href: '/admin', label: 'Overview' }, |
| 7 | 7 | { href: '/admin/connectors', label: 'Connectors' }, |
| 8 | + { href: '/admin/coverage', label: 'Coverage' }, | |
| 8 | 9 | { href: '/admin/data-quality', label: 'Data quality' }, |
| 9 | 10 | { href: '/admin/costs', label: 'Costs' }, |
| 10 | 11 | { href: '/admin/taxonomy', label: 'Taxonomy' }, |
@@ -41,7 +42,7 @@ export function AdminShell({ children, title, subtitle, actions, current }: { ch | ||
| 41 | 42 | |
| 42 | 43 | export function StatusPill({ status }: { status: string | null | undefined }) { |
| 43 | 44 | const s = status ?? 'unknown'; |
| 44 | − const tone: Record<string, string> = { healthy: 'bg-gain-bg text-gain', active: 'bg-gain-bg text-gain', success: 'bg-gain-bg text-gain', valid: 'bg-gain-bg text-gain', degraded: 'bg-alert-bg text-alert', partial: 'bg-alert-bg text-alert', paused: 'bg-alert-bg text-alert', flagged: 'bg-alert-bg text-alert', pending: 'bg-alert-bg text-alert', running: 'bg-index-bg text-index', failing: 'bg-loss-bg text-loss', failed: 'bg-loss-bg text-loss', excluded: 'bg-loss-bg text-loss', disabled: 'bg-inset text-muted', unknown: 'bg-inset text-muted' }; | |
| 45 | + const tone: Record<string, string> = { maintenance: 'bg-index-bg text-index', completed: 'bg-gain-bg text-gain', healthy: 'bg-gain-bg text-gain', active: 'bg-gain-bg text-gain', success: 'bg-gain-bg text-gain', valid: 'bg-gain-bg text-gain', degraded: 'bg-alert-bg text-alert', partial: 'bg-alert-bg text-alert', paused: 'bg-alert-bg text-alert', flagged: 'bg-alert-bg text-alert', pending: 'bg-alert-bg text-alert', running: 'bg-index-bg text-index', failing: 'bg-loss-bg text-loss', failed: 'bg-loss-bg text-loss', excluded: 'bg-loss-bg text-loss', disabled: 'bg-inset text-muted', unknown: 'bg-inset text-muted' }; | |
| 45 | 46 | return <span className={cn('inline-flex rounded-sm px-1.5 py-0.5 text-[11px] font-medium', tone[s] ?? 'bg-inset text-muted')}>{s}</span>; |
| 46 | 47 | } |
| 47 | 48 | |
modified
apps/web/src/lib/admin/actions.ts
+15 −6
@@ -4,7 +4,7 @@ import { cookies } from 'next/headers'; | ||
| 4 | 4 | import { redirect } from 'next/navigation'; |
| 5 | 5 | import { revalidatePath } from 'next/cache'; |
| 6 | 6 | import { z } from 'zod'; |
| 7 | −import { getDb, connectors, connectorRuns, sales, normalizedRecords, taxonomyProposals, categories, auditLog, events, eq, sql } from '@rareindex/database'; | |
| 7 | +import { getDb, connectors, connectorRuns, connectorBackfills, sales, normalizedRecords, taxonomyProposals, categories, auditLog, events, eq, and, inArray, sql } from '@rareindex/database'; | |
| 8 | 8 | import { newId } from '@rareindex/shared'; |
| 9 | 9 | import { ADMIN_COOKIE, adminCookieValue, isAdmin, tokenMatches } from './auth'; |
| 10 | 10 | import { enqueue, QUEUES } from './queue'; |
@@ -33,9 +33,12 @@ export async function adminLogout(): Promise<void> { | ||
| 33 | 33 | redirect('/admin/login'); |
| 34 | 34 | } |
| 35 | 35 | |
| 36 | −const ConnectorAction = z.enum(['run', 'probe', 'recrawl', 'retry', 'pause', 'resume']); | |
| 36 | +const ConnectorAction = z.enum(['run', 'probe', 'recrawl', 'backfill', 'backfill_pause', 'backfill_reset', 'retry', 'pause', 'resume', 'maintenance']); | |
| 37 | 37 | |
| 38 | −/** Connector control-center actions (§144): run/recrawl/probe enqueue `crawl.run`; pause/resume flip status. */ | |
| 38 | +/** | |
| 39 | + * Connector control-center actions (§144, SPEC §29): run/probe/backfill enqueue `crawl.run`; | |
| 40 | + * pause/resume/maintenance flip status; backfill_pause/backfill_reset manage the resumable campaign. | |
| 41 | + */ | |
| 39 | 42 | export async function connectorAction(formData: FormData): Promise<void> { |
| 40 | 43 | await guard(); |
| 41 | 44 | const id = String(formData.get('id') ?? ''); |
@@ -43,11 +46,17 @@ export async function connectorAction(formData: FormData): Promise<void> { | ||
| 43 | 46 | const db = getDb(); |
| 44 | 47 | const [c] = await db.select().from(connectors).where(eq(connectors.id, id)).limit(1); |
| 45 | 48 | if (!c) throw new Error(`unknown connector ${id}`); |
| 46 | − if (action === 'pause' || action === 'resume') { | |
| 47 | − await db.update(connectors).set({ status: action === 'pause' ? 'paused' : 'active', updatedAt: new Date() }).where(eq(connectors.id, id)); | |
| 49 | + if (action === 'pause' || action === 'resume' || action === 'maintenance') { | |
| 50 | + await db.update(connectors).set({ status: action === 'pause' ? 'paused' : action === 'maintenance' ? 'maintenance' : 'active', updatedAt: new Date() }).where(eq(connectors.id, id)); | |
| 48 | 51 | await audit('connector', id, action, `admin ${action}`); |
| 52 | + } else if (action === 'backfill_pause') { | |
| 53 | + await db.update(connectorBackfills).set({ status: 'paused', updatedAt: new Date() }).where(and(eq(connectorBackfills.connectorId, id), eq(connectorBackfills.status, 'running'))); | |
| 54 | + await audit('connector', id, action, 'admin paused backfill campaign'); | |
| 55 | + } else if (action === 'backfill_reset') { | |
| 56 | + await db.update(connectorBackfills).set({ status: 'failed', lastError: 'reset by admin', finishedAt: new Date(), updatedAt: new Date() }).where(and(eq(connectorBackfills.connectorId, id), inArray(connectorBackfills.status, ['running', 'paused']))); | |
| 57 | + await audit('connector', id, action, 'admin reset backfill campaign'); | |
| 49 | 58 | } else { |
| 50 | − const mode = action === 'recrawl' ? 'backfill' : action === 'probe' ? 'probe' : 'incremental'; | |
| 59 | + const mode = action === 'recrawl' || action === 'backfill' ? 'backfill' : action === 'probe' ? 'probe' : 'incremental'; | |
| 51 | 60 | const payload: Record<string, unknown> = { connectorId: id, mode, trigger: action === 'retry' ? 'retry' : 'manual', requestedAt: new Date().toISOString() }; |
| 52 | 61 | if (action === 'probe') payload.limit = 25; |
| 53 | 62 | if (action === 'retry') { |
modified
apps/web/src/lib/admin/queries.ts
+63 −5
@@ -43,17 +43,25 @@ export async function connectorsOverview() { | ||
| 43 | 43 | greatest(1, extract(epoch from (max(coalesce(finished_at, now())) - min(started_at))) / 60.0) as minutes |
| 44 | 44 | from connector_runs where started_at >= now() - interval '24 hours' group by connector_id), |
| 45 | 45 | conf as ( |
| 46 | − select connector_id, avg((payload->>'confidence')::float) as parser_confidence, count(*)::int as normalized_24h from normalized_records where created_at >= now() - interval '24 hours' group by connector_id) | |
| 47 | − select c.id, c.display_name, c.source_id, c.status, c.priority, c.engine_priority, c.categories, c.refresh_frequency_minutes, c.schema_version, c.connector_version, c.last_run_at, c.last_success_at, c.next_run_at, | |
| 46 | + select connector_id, avg((payload->>'confidence')::float) as parser_confidence, count(*)::int as normalized_24h from normalized_records where created_at >= now() - interval '24 hours' group by connector_id), | |
| 47 | + spend as ( | |
| 48 | + select connector_id, sum(credits) filter (where kind = 'firecrawl')::float as fc_credits_24h, sum(credits) filter (where kind = 'scrapfly')::float as sf_credits_24h from costs where occurred_at >= now() - interval '24 hours' and connector_id is not null group by connector_id), | |
| 49 | + bf as ( | |
| 50 | + select distinct on (connector_id) connector_id, id as backfill_id, status as backfill_status, percent as backfill_percent, pages_processed as backfill_pages, items_processed as backfill_items, updated_at as backfill_updated_at, reached_date as backfill_reached_date from connector_backfills order by connector_id, started_at desc) | |
| 51 | + select c.id, c.display_name, c.source_id, c.status, c.priority, c.engine_priority, c.categories, c.regions, c.refresh_frequency_minutes, c.schema_version, c.connector_version, c.last_run_at, c.last_success_at, c.next_run_at, c.meta, | |
| 48 | 52 | h.status as health_status, h.computed_at as health_at, h.health, |
| 49 | 53 | lr.id as last_run_id, lr.status as last_run_status, lr.error as last_error, lr.started_at as last_started_at, lr.finished_at as last_finished_at, lr.pages_attempted as last_pages, lr.records_raw as last_records, lr.anomalies as last_anomalies, |
| 50 | 54 | d.pages as pages_24h, d.records as records_24h, d.duplicates as duplicates_24h, d.credits as credits_24h, (coalesce(d.pages,0) / coalesce(d.minutes,1)) as pages_per_min, |
| 51 | − cf.parser_confidence, cf.normalized_24h | |
| 55 | + cf.parser_confidence, cf.normalized_24h, | |
| 56 | + sp.fc_credits_24h, sp.sf_credits_24h, | |
| 57 | + bf.backfill_id, bf.backfill_status, bf.backfill_percent, bf.backfill_pages, bf.backfill_items, bf.backfill_updated_at, bf.backfill_reached_date | |
| 52 | 58 | from connectors c |
| 53 | 59 | left join connector_health h on h.connector_id = c.id |
| 54 | 60 | left join last_run lr on lr.connector_id = c.id |
| 55 | 61 | left join day d on d.connector_id = c.id |
| 56 | 62 | left join conf cf on cf.connector_id = c.id |
| 63 | + left join spend sp on sp.connector_id = c.id | |
| 64 | + left join bf on bf.connector_id = c.id | |
| 57 | 65 | order by c.priority = 'high' desc, c.id`); |
| 58 | 66 | } |
| 59 | 67 | |
@@ -67,9 +75,59 @@ export async function connectorDetail(id: string) { | ||
| 67 | 75 | const normalized = await run(sql`select status, count(*)::int as n from normalized_records where connector_id = ${id} group by status`); |
| 68 | 76 | const normalizedSample = await run(sql`select id, kind, status, match_method, match_confidence, asset_id, reject_reason, created_at, payload->>'rawTitle' as raw_title, payload->>'price' as price, payload->>'currency' as currency from normalized_records where connector_id = ${id} order by created_at desc limit 10`); |
| 69 | 77 | const costs = await run(sql`select date_trunc('day', occurred_at)::date as day, kind, sum(credits)::float as credits, sum(usd_est)::float as usd from costs where connector_id = ${id} and occurred_at >= now() - interval '30 days' group by 1, 2 order by 1 desc`); |
| 70 | − const outputs = await one(sql`select (select count(*)::int from sales where connector_id = ${id}) as sales, (select count(*)::int from listings where connector_id = ${id}) as listings, (select count(*)::int from price_observations where connector_id = ${id}) as observations`); | |
| 78 | + const outputs = await one(sql`select (select count(*)::int from sales where connector_id = ${id}) as sales, (select count(*)::int from listings where connector_id = ${id}) as listings, (select count(*)::int from price_observations where connector_id = ${id}) as observations, (select count(*)::int from certificates where ${id} = any(source_ids) or last_source_url like '%' || ${connector.source_id as string} || '%') as certificates`); | |
| 79 | + const backfills = await run(sql`select * from connector_backfills where connector_id = ${id} order by started_at desc limit 10`); | |
| 80 | + const fieldStats = await run(sql`select field, day::text as day, total, nulls from connector_field_stats where connector_id = ${id} and day >= (current_date - interval '7 days')::date order by day desc, field`); | |
| 71 | 81 | const anomalies = runs.flatMap((r) => (Array.isArray(r.anomalies) ? (r.anomalies as string[]) : [])).slice(0, 30); |
| 72 | − return { connector, source, health, runs, rawSample, normalized, normalizedSample, costs, outputs, anomalies }; | |
| 82 | + return { connector, source, health, runs, rawSample, normalized, normalizedSample, costs, outputs, anomalies, backfills, fieldStats }; | |
| 83 | +} | |
| 84 | + | |
| 85 | +export async function rawRecordDetail(id: string) { | |
| 86 | + return one(sql`select id, connector_id, source_id, run_id, engine, url, external_id, kind, fetched_at, content_hash, http_status, payload, snapshot_ref, parser_version, connector_version, processed_at, process_error from raw_records where id = ${id}`); | |
| 87 | +} | |
| 88 | + | |
| 89 | +export async function normalizedRecordDetail(id: string) { | |
| 90 | + return one(sql`select n.*, r.url as raw_url, r.engine as raw_engine from normalized_records n left join raw_records r on r.id = n.raw_record_id where n.id = ${id}`); | |
| 91 | +} | |
| 92 | + | |
| 93 | +/** Global coverage dashboard (SPEC §30). */ | |
| 94 | +export async function coverage() { | |
| 95 | + const totals = await one(sql`select | |
| 96 | + (select count(*)::int from connectors) as connectors, | |
| 97 | + (select count(*)::int from connectors where status = 'active') as connectors_active, | |
| 98 | + (select count(*)::int from listings where availability = 'available') as live_listings, | |
| 99 | + (select count(*)::int from sales) as sales, | |
| 100 | + (select count(*)::int from sales where sale_date < now() - interval '1 year') as sales_older_1y, | |
| 101 | + (select min(sale_date) from sales) as oldest_sale, | |
| 102 | + (select count(*)::int from auction_lots) as auction_lots, | |
| 103 | + (select count(distinct auction_house) from auctions) as auction_houses, | |
| 104 | + (select count(distinct grader) from certificates) as graders_with_certs, | |
| 105 | + (select count(*)::int from certificates) as certificates, | |
| 106 | + (select count(*)::int from population_reports) as population_reports, | |
| 107 | + (select count(*)::int from assets) as assets, | |
| 108 | + (select count(*)::int from asset_variants) as variants, | |
| 109 | + (select count(*)::int from price_observations) as observations, | |
| 110 | + (select count(*)::int from raw_records where fetched_at >= now() - interval '24 hours') as raw_24h, | |
| 111 | + (select count(*)::int from normalized_records where created_at >= now() - interval '24 hours') as normalized_24h, | |
| 112 | + (select count(*)::int from sales where created_at >= now() - interval '24 hours') as sales_24h, | |
| 113 | + (select count(*)::int from listings where first_seen_at >= now() - interval '24 hours') as listings_24h, | |
| 114 | + (select count(*)::int from connector_backfills where status = 'running') as backfills_running, | |
| 115 | + (select count(*)::int from connector_backfills where status = 'completed') as backfills_completed`); | |
| 116 | + const byStatus = await run(sql`select coalesce(h.status, 'unknown') as status, count(*)::int as n from connectors c left join connector_health h on h.connector_id = c.id group by 1 order by n desc`); | |
| 117 | + const byCategory = await run(sql`select cat as category, count(distinct c.id)::int as connectors, count(distinct c.id) filter (where h.status = 'healthy')::int as healthy from connectors c cross join unnest(c.categories) as cat left join connector_health h on h.connector_id = c.id group by 1 order by connectors desc, 1`); | |
| 118 | + const byCountry = await run(sql`select coalesce(c.meta->>'country', c.regions[1], 'global') as country, count(*)::int as connectors from connectors c group by 1 order by connectors desc`); | |
| 119 | + const bySourceType = await run(sql`select s.source_type, count(*)::int as connectors from connectors c join sources s on s.id = c.source_id group by 1 order by 2 desc`); | |
| 120 | + const byEngine = await run(sql`select engine_priority[1] as engine, count(*)::int as connectors from connectors group by 1 order by 2 desc`); | |
| 121 | + const byCapability = await run(sql`select cap as capability, count(*)::int as connectors from connectors c cross join jsonb_array_elements_text(coalesce(c.meta->'capabilities', '[]'::jsonb)) as cap group by 1 order by 2 desc`); | |
| 122 | + const salesByFamily = await run(sql`select a.family_slug, count(*)::int as sales, count(*) filter (where s.sale_date >= now() - interval '30 days')::int as sales_30d, min(s.sale_date)::date as oldest from sales s join assets a on a.id = s.asset_id group by 1 order by sales desc`); | |
| 123 | + const listingsByFamily = await run(sql`select a.family_slug, count(*)::int as live from listings l join assets a on a.id = l.asset_id where l.availability = 'available' group by 1 order by live desc`); | |
| 124 | + const daily = await run(sql`select d::date as day, | |
| 125 | + (select count(*)::int from raw_records r where r.fetched_at >= d and r.fetched_at < d + interval '1 day') as raw, | |
| 126 | + (select count(*)::int from sales s where s.created_at >= d and s.created_at < d + interval '1 day') as sales, | |
| 127 | + (select count(*)::int from listings l where l.first_seen_at >= d and l.first_seen_at < d + interval '1 day') as listings | |
| 128 | + from generate_series(current_date - interval '13 days', current_date, interval '1 day') d order by 1`); | |
| 129 | + const salesByYear = await run(sql`select extract(year from sale_date)::int as year, count(*)::int as sales from sales group by 1 order by 1`); | |
| 130 | + return { totals, byStatus, byCategory, byCountry, bySourceType, byEngine, byCapability, salesByFamily, listingsByFamily, daily, salesByYear }; | |
| 73 | 131 | } |
| 74 | 132 | |
| 75 | 133 | export async function costsOverview(days = 30) { |
modified
connectors/README.md
+4 −0
@@ -16,3 +16,7 @@ Rules: never invent data, keep source quirks inside the connector, store the sou | ||
| 16 | 16 | keep the native currency, respect robots/terms/rate limits, and never bypass access controls (§179). |
| 17 | 17 | |
| 18 | 18 | Module contract: `export default (meta: ConnectorMeta) => RareIndexConnector`. |
| 19 | + | |
| 20 | +Full guide: `docs/connectors/ADDING_A_CONNECTOR.md` (scaffold with `pnpm connector:new`, adapters for Shopify/ | |
| 21 | +WooCommerce/sitemaps/RSS/PDF/JSON-LD, per-host policies in `domains.json` + `domains.d/`, source catalogue in | |
| 22 | +`data/sources/entries/`, resumable backfills, health and drift monitoring). | |
added
connectors/api/1stdibs/README.md
+13 −0
@@ -0,0 +1,13 @@ | ||
| 1 | +# 1stdibs — 1stDibs marketplace (dealer listings) | |
| 2 | + | |
| 3 | +Browse pages (`/furniture/seating/`, `/jewelry/`, `/jewelry/watches/wrist-watches/`, | |
| 4 | +`/fashion/handbags-purses-bags/`, `/art/prints-works-on-paper/` … `?page=N`, 60 items, ≤ 50 pages) embed | |
| 5 | +the server-rendered Relay store in `<script id="serverVars_data">`. `parseBrowseHtml` dereferences the | |
| 6 | +`Item` records: title, `serviceId`, display prices in 10 currencies (USD kept), seller company/id, | |
| 7 | +designers, location, period/style line, materials, measurements, photos, sold/hold flags. | |
| 8 | + | |
| 9 | +- One raw record per page (trimmed items, not the 3 MB store), one `listing` per item. | |
| 10 | +- Category from seed vertical + category code + attribute line/title (`design_furniture` / `antiques` / | |
| 11 | + decorative-object slugs / jewelry / watch brands / `luxury_handbags` / art). | |
| 12 | +- Cursor `{seedIndex, page}` with seed rotation; 5 s politeness. | |
| 13 | +- Identifiers: `firstdibs_item_id` (`f_51602042`), `firstdibs_seller_id`. | |
added
connectors/api/1stdibs/index.test.ts
+54 −0
@@ -0,0 +1,54 @@ | ||
| 1 | +import { describe, expect, it } from 'vitest'; | |
| 2 | +import metaJson from './meta.json' with { type: 'json' }; | |
| 3 | +import { loadFixture, runFixtureSuite } from '@rareindex/connectors/testing'; | |
| 4 | +import { localMeta } from '../_lib/local-meta.js'; | |
| 5 | +import createConnector, { makeDeref, pageUrl, parseBrowseHtml } from './index.js'; | |
| 6 | + | |
| 7 | +const connector = createConnector(localMeta(metaJson)); | |
| 8 | + | |
| 9 | +describe('1stdibs', () => { | |
| 10 | + runFixtureSuite(connector, it, expect); | |
| 11 | + | |
| 12 | + it('parses Item records out of the serverVars_data Relay store', () => { | |
| 13 | + const html = String((loadFixture('1stdibs', 'seating-chairs-p1').raw.payload as { snapshot?: string }).snapshot ?? ''); | |
| 14 | + const p = parseBrowseHtml(html); | |
| 15 | + expect(p).not.toBeNull(); | |
| 16 | + expect(p!.items.length).toBeGreaterThanOrEqual(2); | |
| 17 | + expect(p!.totalResults).toBeGreaterThan(1000); | |
| 18 | + expect(p!.maxPages).toBe(50); | |
| 19 | + const it0 = p!.items[0]!; | |
| 20 | + expect(it0.serviceId).toMatch(/^f_\d+$/); | |
| 21 | + expect(it0.url).toMatch(/^https:\/\/www\.1stdibs\.com\/furniture\/.+\/id-f_\d+\/$/); | |
| 22 | + expect(it0.prices.USD).toBeGreaterThan(0); | |
| 23 | + expect(it0.prices.EUR).toBeGreaterThan(0); | |
| 24 | + expect(it0.seller.serviceId).toMatch(/^f_\d+$/); | |
| 25 | + expect(it0.images[0]).toMatch(/1stdibscdn\.com/); | |
| 26 | + expect(it0.categoryCode).toMatch(/^F_/); | |
| 27 | + expect(pageUrl('/furniture/seating/', 2)).toBe('https://www.1stdibs.com/furniture/seating/?page=2'); | |
| 28 | + }); | |
| 29 | + | |
| 30 | + it('dereferences __ref / __refs chains', () => { | |
| 31 | + const store = { a: { __typename: 'A', b: { __ref: 'b' }, list: { __refs: ['c', 'c'] } }, b: { __typename: 'B', v: 1 }, c: { __typename: 'C', v: 2 } }; | |
| 32 | + const deref = makeDeref(store as never); | |
| 33 | + expect(deref(store.a)).toEqual({ __typename: 'A', b: { __typename: 'B', v: 1 }, list: [{ __typename: 'C', v: 2 }, { __typename: 'C', v: 2 }] }); | |
| 34 | + expect(parseBrowseHtml('<html><body>no store</body></html>')).toBeNull(); | |
| 35 | + }); | |
| 36 | + | |
| 37 | + it('normalises items into USD dealer listings with designer, seller and identifiers', async () => { | |
| 38 | + const fx = loadFixture('1stdibs', 'seating-chairs-p1'); | |
| 39 | + const out = await connector.normalize(fx.raw); | |
| 40 | + expect(out.length).toBeGreaterThan(0); | |
| 41 | + for (const r of out) { | |
| 42 | + if (r.kind !== 'listing') throw new Error('expected listing'); | |
| 43 | + expect(r.currency).toBe('USD'); | |
| 44 | + expect(r.price).toBeGreaterThan(0); | |
| 45 | + expect(r.attributes.identifiers.firstdibs_item_id).toMatch(/^f_\d+$/); | |
| 46 | + expect(['design_furniture', 'antiques']).toContain(r.attributes.categorySlug); | |
| 47 | + expect(r.seller).toBeTruthy(); | |
| 48 | + } | |
| 49 | + const jw = loadFixture('1stdibs', 'wrist-watches-p1'); | |
| 50 | + const jwOut = await connector.normalize(jw.raw); | |
| 51 | + expect(jwOut.length).toBeGreaterThan(0); | |
| 52 | + for (const r of jwOut) expect(['rolex', 'omega', 'patek_philippe', 'audemars_piguet', 'other_watches']).toContain((r as { attributes: { categorySlug: string } }).attributes.categorySlug); | |
| 53 | + }); | |
| 54 | +}); | |
added
connectors/api/1stdibs/index.ts
+324 −0
@@ -0,0 +1,324 @@ | ||
| 1 | +import { z } from 'zod'; | |
| 2 | +import { BaseConnector, type ConnectorMeta, type CrawlContext, type RawRecordInput, type RawRecordLike } from '@rareindex/connectors'; | |
| 3 | +import { AssetAttributesSchema, NormalizedListingSchema, type NormalizedRecord } from '@rareindex/shared'; | |
| 4 | +import { designSlug, plainText, readSeedCursor, yearOrDecade, type DesignVertical } from '../_g10-lib/index.js'; | |
| 5 | + | |
| 6 | +/** | |
| 7 | + * 1stDibs — the largest online marketplace for vintage/antique furniture, design, jewelry, watches, | |
| 8 | + * fashion and art (USD by default). Browse pages (/furniture/seating/, /jewelry/, … ?page=N, 60 items) | |
| 9 | + * embed the server-rendered Relay store in <script id="serverVars_data" type="application/json">; | |
| 10 | + * we read the `Item` records from it (title, serviceId, prices in ten currencies, seller company, | |
| 11 | + * creators/designers, location, period/style attributes, materials, measurements, photos, sold/hold | |
| 12 | + * flags) and never call 1stDibs' GraphQL endpoint ourselves. Asking prices → listings. | |
| 13 | + */ | |
| 14 | +const BASE = 'https://www.1stdibs.com'; | |
| 15 | +const PARSER_VERSION = '1.0.0'; | |
| 16 | +const PAGE_SIZE = 60; | |
| 17 | + | |
| 18 | +export const SeedSchema = z.object({ path: z.string(), slug: z.string().nullable().optional(), vertical: z.enum(['furniture', 'lighting', 'decor', 'art', 'jewelry', 'watches', 'fashion', 'tableware', 'rugs', 'pens', 'unknown']).default('unknown') }); | |
| 19 | +export type Seed = z.infer<typeof SeedSchema>; | |
| 20 | + | |
| 21 | +export const ItemSchema = z.object({ | |
| 22 | + serviceId: z.string(), | |
| 23 | + title: z.string(), | |
| 24 | + url: z.string(), | |
| 25 | + prices: z.record(z.string(), z.number()), | |
| 26 | + amountType: z.string().nullable(), | |
| 27 | + isSold: z.boolean(), | |
| 28 | + isOnHold: z.boolean(), | |
| 29 | + isUnavailable: z.boolean(), | |
| 30 | + isNewListing: z.boolean().nullable(), | |
| 31 | + seller: z.object({ serviceId: z.string().nullable(), company: z.string().nullable() }), | |
| 32 | + creators: z.array(z.string()), | |
| 33 | + country: z.string().nullable(), | |
| 34 | + location: z.string().nullable(), | |
| 35 | + attributesText: z.string().nullable(), | |
| 36 | + materials: z.string().nullable(), | |
| 37 | + description: z.string().nullable(), | |
| 38 | + measurement: z.string().nullable(), | |
| 39 | + categoryCode: z.string().nullable(), | |
| 40 | + browseUrl: z.string().nullable(), | |
| 41 | + vertical: z.string().nullable(), | |
| 42 | + images: z.array(z.string()), | |
| 43 | +}); | |
| 44 | +export type Item = z.infer<typeof ItemSchema>; | |
| 45 | + | |
| 46 | +export const PagePayloadSchema = z.object({ kind: z.literal('listing_page'), url: z.string(), seed: SeedSchema, page: z.number().int(), totalResults: z.number().int().nullable(), maxPages: z.number().int().nullable(), items: z.array(ItemSchema), snapshot: z.string().optional() }); | |
| 47 | +export type PagePayload = z.infer<typeof PagePayloadSchema>; | |
| 48 | + | |
| 49 | +const ConfigSchema = z.object({ seeds: z.array(SeedSchema).min(1), pagesPerSeed: z.number().int().min(1).default(1), seedsPerRun: z.number().int().min(1).default(3), maxPagesPerSeed: z.number().int().min(1).default(50) }); | |
| 50 | + | |
| 51 | +type Rec = Record<string, unknown>; | |
| 52 | + | |
| 53 | +/** Relay normalized store → dereferenced view (bounded depth). */ | |
| 54 | +export function makeDeref(store: Record<string, Rec>) { | |
| 55 | + const deref = (v: unknown, depth = 0): unknown => { | |
| 56 | + if (depth > 4 || v === null || typeof v !== 'object') return v; | |
| 57 | + if (Array.isArray(v)) return v.map((x) => deref(x, depth + 1)); | |
| 58 | + const o = v as Rec; | |
| 59 | + if (typeof o.__ref === 'string') return deref(store[o.__ref] ?? null, depth + 1); | |
| 60 | + if (Array.isArray(o.__refs)) return o.__refs.map((r) => deref(store[String(r)] ?? null, depth + 1)); | |
| 61 | + const out: Rec = {}; | |
| 62 | + for (const [k, x] of Object.entries(o)) if (k !== '__id') out[k] = deref(x, depth + 1); | |
| 63 | + return out; | |
| 64 | + }; | |
| 65 | + return deref; | |
| 66 | +} | |
| 67 | + | |
| 68 | +function firstKey(o: Rec, prefix: string): unknown { | |
| 69 | + const k = Object.keys(o).find((x) => x === prefix || x.startsWith(`${prefix}(`)); | |
| 70 | + return k ? o[k] : undefined; | |
| 71 | +} | |
| 72 | + | |
| 73 | +const str = (v: unknown): string | null => (typeof v === 'string' && v.trim() ? v.trim() : null); | |
| 74 | + | |
| 75 | +/** Parse the serverVars_data bootstrap of a browse page. Exported for tests. */ | |
| 76 | +export function parseBrowseHtml(htmlText: string): { items: Item[]; totalResults: number | null; maxPages: number | null } | null { | |
| 77 | + const m = htmlText.match(/<script id="serverVars_data" type="application\/json">([\s\S]*?)<\/script>/); | |
| 78 | + if (!m) return null; | |
| 79 | + let root: Rec; | |
| 80 | + try { | |
| 81 | + root = JSON.parse(m[1]!) as Rec; | |
| 82 | + } catch { | |
| 83 | + return null; | |
| 84 | + } | |
| 85 | + const store = ((root.dbl as Rec | undefined)?.relayData ?? (root.relay as Rec | undefined)?.store ?? null) as Record<string, Rec> | null; | |
| 86 | + if (!store || typeof store !== 'object') return null; | |
| 87 | + const deref = makeDeref(store); | |
| 88 | + const items: Item[] = []; | |
| 89 | + let totalResults: number | null = null; | |
| 90 | + let maxPages: number | null = null; | |
| 91 | + for (const rec of Object.values(store)) { | |
| 92 | + if (!rec || typeof rec !== 'object') continue; | |
| 93 | + if (rec.__typename === 'ItemSearchQueryConnection') { | |
| 94 | + if (typeof rec.totalResults === 'number') totalResults = rec.totalResults; | |
| 95 | + if (typeof rec.displayMaxNumberOfPages === 'number') maxPages = rec.displayMaxNumberOfPages; | |
| 96 | + continue; | |
| 97 | + } | |
| 98 | + if (rec.__typename !== 'Item' || typeof rec.serviceId !== 'string' || typeof rec.title !== 'string') continue; | |
| 99 | + const dp = deref(firstKey(rec, 'displayPrice')) as Array<{ convertedAmountList?: Array<{ amount?: number; currency?: string }>; amountType?: string }> | undefined; | |
| 100 | + const prices: Record<string, number> = {}; | |
| 101 | + let amountType: string | null = null; | |
| 102 | + for (const d of dp ?? []) { | |
| 103 | + amountType ??= d.amountType ?? null; | |
| 104 | + for (const c of d.convertedAmountList ?? []) if (c.currency && typeof c.amount === 'number' && c.amount > 0) prices[c.currency] ??= c.amount; | |
| 105 | + } | |
| 106 | + const track = rec.ecommerceTrackingParams as { price?: number; convertedAmounts?: Record<string, number> } | undefined; | |
| 107 | + if (!Object.keys(prices).length && track?.convertedAmounts) for (const [k, v] of Object.entries(track.convertedAmounts)) if (typeof v === 'number' && v > 0) prices[k] = v; | |
| 108 | + const seller = deref(rec.seller) as { serviceId?: string; sellerProfile?: { company?: string } } | null; | |
| 109 | + const creators = (deref(rec.creators) as Array<{ creator?: { displayName?: string } }> | null) ?? []; | |
| 110 | + const address = deref(rec.address) as { englishCountryName?: string } | null; | |
| 111 | + const qv = deref(rec.quickViewDisplay) as { paragraphs?: Array<{ key?: string; text?: string }> } | null; | |
| 112 | + const para = (key: string) => str(qv?.paragraphs?.find((p) => p.key === key)?.text); | |
| 113 | + const meas = deref(rec.measurement) as Rec | null; | |
| 114 | + const measList = meas ? (firstKey(meas, 'display') as Array<{ unit?: string; value?: string }> | undefined) : undefined; | |
| 115 | + const photos = (deref(firstKey(rec, 'photos')) as Array<{ masterOrZoomPath?: string; versions?: Array<{ webPath?: string }> }> | null) ?? []; | |
| 116 | + const images = photos.map((p) => p.versions?.find((v) => v.webPath?.includes('width=768'))?.webPath ?? p.masterOrZoomPath ?? null).filter((x): x is string => Boolean(x)); | |
| 117 | + const link = deref(rec.linkData) as { path?: string } | null; | |
| 118 | + const path = str(rec.localizedPdpUrl) ?? str(link?.path); | |
| 119 | + if (!path) continue; | |
| 120 | + items.push({ | |
| 121 | + serviceId: rec.serviceId, | |
| 122 | + title: rec.title, | |
| 123 | + url: path.startsWith('http') ? path : `${BASE}${path}`, | |
| 124 | + prices, | |
| 125 | + amountType, | |
| 126 | + isSold: rec.isSold === true, | |
| 127 | + isOnHold: rec.isOnHold === true, | |
| 128 | + isUnavailable: rec.isUnavailable === true, | |
| 129 | + isNewListing: typeof rec.isNewListing === 'boolean' ? rec.isNewListing : null, | |
| 130 | + seller: { serviceId: str(seller?.serviceId), company: str(seller?.sellerProfile?.company) }, | |
| 131 | + creators: creators.map((c) => str(c?.creator?.displayName)).filter((x): x is string => Boolean(x)), | |
| 132 | + country: str(address?.englishCountryName), | |
| 133 | + location: para('location'), | |
| 134 | + attributesText: para('attributes'), | |
| 135 | + materials: para('materials'), | |
| 136 | + description: plainText(para('description'), 1200), | |
| 137 | + measurement: str(measList?.find((x) => x.unit === 'IN')?.value ?? measList?.[0]?.value), | |
| 138 | + categoryCode: str(rec.categoryCode), | |
| 139 | + browseUrl: str(rec.browseUrl), | |
| 140 | + vertical: str(rec.vertical), | |
| 141 | + images: [...new Set(images)].slice(0, 4), | |
| 142 | + }); | |
| 143 | + } | |
| 144 | + return { items, totalResults, maxPages }; | |
| 145 | +} | |
| 146 | + | |
| 147 | +export function pageUrl(seedPath: string, page: number): string { | |
| 148 | + const p = seedPath.endsWith('/') ? seedPath : `${seedPath}/`; | |
| 149 | + return `${BASE}${p}${page > 1 ? `?page=${page}` : ''}`; | |
| 150 | +} | |
| 151 | + | |
| 152 | +/** | |
| 153 | + * Fixture helper: rebuild a minimal page whose serverVars_data store only holds the search connection | |
| 154 | + * record plus the first `n` Item records and everything reachable from them through __ref/__refs. | |
| 155 | + */ | |
| 156 | +export function trimStoreHtml(htmlText: string, n = 3): string { | |
| 157 | + const m = htmlText.match(/<script id="serverVars_data" type="application\/json">([\s\S]*?)<\/script>/); | |
| 158 | + if (!m) return ''; | |
| 159 | + const root = JSON.parse(m[1]!) as Rec; | |
| 160 | + const store = ((root.dbl as Rec | undefined)?.relayData ?? {}) as Record<string, Rec>; | |
| 161 | + const keep = new Set<string>(); | |
| 162 | + const queue: string[] = []; | |
| 163 | + for (const [k, v] of Object.entries(store)) { | |
| 164 | + if (v?.__typename === 'ItemSearchQueryConnection') keep.add(k); | |
| 165 | + } | |
| 166 | + let picked = 0; | |
| 167 | + for (const [k, v] of Object.entries(store)) { | |
| 168 | + if (v?.__typename === 'Item' && picked < n) { | |
| 169 | + queue.push(k); | |
| 170 | + picked++; | |
| 171 | + } | |
| 172 | + } | |
| 173 | + const collect = (v: unknown) => { | |
| 174 | + if (!v || typeof v !== 'object') return; | |
| 175 | + if (Array.isArray(v)) return v.forEach(collect); | |
| 176 | + const o = v as Rec; | |
| 177 | + if (typeof o.__ref === 'string') queue.push(o.__ref); | |
| 178 | + if (Array.isArray(o.__refs)) for (const r of o.__refs) queue.push(String(r)); | |
| 179 | + for (const [k, x] of Object.entries(o)) if (k !== '__ref' && k !== '__refs') collect(x); | |
| 180 | + }; | |
| 181 | + while (queue.length) { | |
| 182 | + const k = queue.shift()!; | |
| 183 | + if (keep.has(k) || !store[k]) continue; | |
| 184 | + keep.add(k); | |
| 185 | + collect(store[k]); | |
| 186 | + } | |
| 187 | + const small: Record<string, Rec> = {}; | |
| 188 | + for (const k of keep) { | |
| 189 | + const rec = store[k]!; | |
| 190 | + // the connection record references every item on the page; keep only its scalar fields | |
| 191 | + small[k] = rec.__typename === 'ItemSearchQueryConnection' ? Object.fromEntries(Object.entries(rec).filter(([, v]) => v === null || typeof v !== 'object')) : rec; | |
| 192 | + } | |
| 193 | + return `<!doctype html><html><head><script id="serverVars_data" type="application/json">${JSON.stringify({ dbl: { relayData: small } })}</script></head><body></body></html>`; | |
| 194 | +} | |
| 195 | + | |
| 196 | +function verticalFor(seed: Seed, item: Item): DesignVertical { | |
| 197 | + if (seed.vertical !== 'unknown') return seed.vertical; | |
| 198 | + const v = item.vertical ?? ''; | |
| 199 | + const code = item.categoryCode ?? ''; | |
| 200 | + if (v === 'jewelry') return code.startsWith('J_WAT') ? 'watches' : 'jewelry'; | |
| 201 | + if (v === 'fashion') return 'fashion'; | |
| 202 | + if (v === 'art') return 'art'; | |
| 203 | + if (code.startsWith('F_LIG')) return 'lighting'; | |
| 204 | + if (code.startsWith('F_DEC') || code.startsWith('F_SER')) return 'decor'; | |
| 205 | + if (code.startsWith('F_RUG') || code.startsWith('F_TEX')) return 'rugs'; | |
| 206 | + return v === 'furniture' ? 'furniture' : 'unknown'; | |
| 207 | +} | |
| 208 | + | |
| 209 | +export class FirstDibsConnector extends BaseConnector { | |
| 210 | + readonly version = '1.0.0'; | |
| 211 | + readonly parserVersion = PARSER_VERSION; | |
| 212 | + protected override minIntervalMs = 5000; | |
| 213 | + override readonly urlPatterns = [/^https?:\/\/(www\.)?1stdibs\.com\/(?:furniture|jewelry|fashion|art)\/.+\/id-([a-z]_\d+)\/?/i]; | |
| 214 | + private readonly cfg: z.infer<typeof ConfigSchema>; | |
| 215 | + | |
| 216 | + constructor(meta: ConnectorMeta) { | |
| 217 | + super(meta); | |
| 218 | + this.cfg = ConfigSchema.parse(meta.config); | |
| 219 | + } | |
| 220 | + | |
| 221 | + async *crawl(ctx: CrawlContext): AsyncIterable<RawRecordInput> { | |
| 222 | + const seeds = ctx.options.seeds?.length ? ctx.options.seeds.map((s) => SeedSchema.parse({ path: s.startsWith('/') ? s : new URL(s).pathname })) : this.cfg.seeds; | |
| 223 | + const backfill = ctx.options.mode === 'backfill'; | |
| 224 | + const maxPages = backfill ? Math.min(this.policy.backfillMaxPages, this.cfg.maxPagesPerSeed) : this.cfg.pagesPerSeed; | |
| 225 | + const start = readSeedCursor(ctx.options.cursor, seeds.length); | |
| 226 | + const seedsThisRun = backfill ? seeds.length : Math.min(seeds.length, this.cfg.seedsPerRun); | |
| 227 | + let count = 0; | |
| 228 | + let items = 0; | |
| 229 | + for (let k = 0; k < seedsThisRun; k++) { | |
| 230 | + const seedIndex = (start.seedIndex + k) % seeds.length; | |
| 231 | + const seed = seeds[seedIndex]!; | |
| 232 | + let page = k === 0 ? start.page : 1; | |
| 233 | + for (; page <= maxPages; page++) { | |
| 234 | + if (ctx.signal?.aborted || this.reached(ctx, count)) return; | |
| 235 | + const url = pageUrl(seed.path, page); | |
| 236 | + await this.throttle(url); | |
| 237 | + const res = await ctx.fetch(url, { | |
| 238 | + engines: ['api'], | |
| 239 | + responseType: 'text', | |
| 240 | + timeoutMs: 90_000, | |
| 241 | + expect: ['title', 'price', 'currency'], | |
| 242 | + parse: (r) => { | |
| 243 | + const p = r.html ? parseBrowseHtml(r.html) : null; | |
| 244 | + const priced = p?.items.find((i) => i.prices.USD); | |
| 245 | + return p?.items.length ? { title: p.items[0]!.title, price: priced?.prices.USD ?? null, currency: priced ? 'USD' : null } : null; | |
| 246 | + }, | |
| 247 | + minQuality: 0.3, | |
| 248 | + }); | |
| 249 | + const parsed = res.success && res.html ? parseBrowseHtml(res.html) : null; | |
| 250 | + if (!parsed) { | |
| 251 | + ctx.anomaly(res.success ? 'schema_drift' : 'page_fetch_failed', `${url}: ${res.error ?? res.httpStatus ?? 'serverVars_data missing'}`); | |
| 252 | + break; | |
| 253 | + } | |
| 254 | + if (!parsed.items.length) { | |
| 255 | + if (page === 1) ctx.anomaly('parse_failure_page', `${url}: no Item records`); | |
| 256 | + break; | |
| 257 | + } | |
| 258 | + count++; | |
| 259 | + items += parsed.items.length; | |
| 260 | + const payload: PagePayload = { kind: 'listing_page', url, seed, page, totalResults: parsed.totalResults, maxPages: parsed.maxPages, items: parsed.items }; | |
| 261 | + yield { url, externalId: `${seed.path}:p${page}`, kind: 'listing', engine: res.engine, httpStatus: res.httpStatus, payload, fetchedAt: res.fetchedAt }; | |
| 262 | + await ctx.setCursor({ seedIndex, page: page + 1, at: new Date().toISOString() }); | |
| 263 | + await ctx.progress({ page, totalPages: parsed.maxPages, itemsProcessed: items }); | |
| 264 | + if (parsed.items.length < PAGE_SIZE || (parsed.maxPages !== null && page >= parsed.maxPages)) break; | |
| 265 | + } | |
| 266 | + const nextSeed = (seedIndex + 1) % seeds.length; | |
| 267 | + await ctx.setCursor({ seedIndex: nextSeed, page: 1, at: new Date().toISOString(), ...(backfill && nextSeed === 0 ? { done: true } : {}) }); | |
| 268 | + } | |
| 269 | + } | |
| 270 | + | |
| 271 | + async normalize(raw: RawRecordLike): Promise<NormalizedRecord[]> { | |
| 272 | + const p = PagePayloadSchema.parse(raw.payload); | |
| 273 | + const out: NormalizedRecord[] = []; | |
| 274 | + for (const it of p.items) { | |
| 275 | + const vertical = verticalFor(p.seed, it); | |
| 276 | + const categorySlug = p.seed.slug ?? designSlug(it.attributesText ?? it.browseUrl, `${it.title} ${it.attributesText ?? ''}`, vertical); | |
| 277 | + if (!categorySlug) continue; | |
| 278 | + const currency = it.prices.USD ? 'USD' : Object.keys(it.prices)[0] ?? null; | |
| 279 | + const price = currency ? it.prices[currency]! : null; | |
| 280 | + const { year, decade } = yearOrDecade(`${it.title} ${it.attributesText ?? ''}`); | |
| 281 | + const period = it.attributesText?.match(/\b(1[5-9]th Century|20th Century|21st Century|Antique|Vintage|Mid-Century Modern|Art Deco|Art Nouveau|Victorian|Georgian|Regency|Louis X[VI]+|Bauhaus|Scandinavian Modern|Hollywood Regency|Brutalist|Space Age|Postmodern|Memphis)\b/i)?.[0] ?? null; | |
| 282 | + const attributes = AssetAttributesSchema.parse({ | |
| 283 | + categorySlug, | |
| 284 | + brand: it.creators[0] ?? null, | |
| 285 | + name: it.title, | |
| 286 | + year, | |
| 287 | + material: it.materials, | |
| 288 | + size: it.measurement, | |
| 289 | + country: null, | |
| 290 | + identifiers: { firstdibs_item_id: it.serviceId, ...(it.seller.serviceId ? { firstdibs_seller_id: it.seller.serviceId } : {}) }, | |
| 291 | + metadata: { decade, period, creators: it.creators, attributes_text: it.attributesText, category_code: it.categoryCode, browse_url: it.browseUrl, vertical: it.vertical, converted_prices: it.prices, amount_type: it.amountType, item_country: it.country, is_new_listing: it.isNewListing }, | |
| 292 | + }); | |
| 293 | + out.push( | |
| 294 | + NormalizedListingSchema.parse({ | |
| 295 | + kind: 'listing', | |
| 296 | + connectorId: this.meta.id, | |
| 297 | + sourceId: this.meta.sourceId, | |
| 298 | + sourceUrl: it.url, | |
| 299 | + externalId: it.serviceId, | |
| 300 | + rawTitle: it.title, | |
| 301 | + description: it.description, | |
| 302 | + imageUrls: it.images, | |
| 303 | + attributes, | |
| 304 | + grade: { grader: null, grade: null, qualifier: null, certificationNumber: null }, | |
| 305 | + condition: { condition: null, conditionRaw: null, completeness: null }, | |
| 306 | + observedAt: raw.fetchedAt, | |
| 307 | + confidence: price ? 0.82 : 0.6, | |
| 308 | + parserVersion: PARSER_VERSION, | |
| 309 | + listingType: 'fixed_price', | |
| 310 | + price, | |
| 311 | + currency, | |
| 312 | + seller: it.seller.company, | |
| 313 | + location: it.location ?? it.country, | |
| 314 | + availability: it.isSold ? 'sold' : it.isOnHold || it.isUnavailable ? 'ended' : 'available', | |
| 315 | + }), | |
| 316 | + ); | |
| 317 | + } | |
| 318 | + return out; | |
| 319 | + } | |
| 320 | +} | |
| 321 | + | |
| 322 | +export default function createConnector(meta: ConnectorMeta): FirstDibsConnector { | |
| 323 | + return new FirstDibsConnector(meta); | |
| 324 | +} | |
added
connectors/api/1stdibs/meta.json
+47 −0
@@ -0,0 +1,47 @@ | ||
| 1 | +{ | |
| 2 | + "id": "1stdibs", | |
| 3 | + "displayName": "1stDibs (vintage design, antiques, jewelry, watches, fashion & art listings)", | |
| 4 | + "sourceId": "1stdibs", | |
| 5 | + "sourceName": "1stDibs", | |
| 6 | + "sourceType": "marketplace", | |
| 7 | + "sourceUrl": "https://www.1stdibs.com", | |
| 8 | + "module": "api/1stdibs", | |
| 9 | + "enginePriority": ["api"], | |
| 10 | + "categories": ["design_furniture", "antiques", "porcelain", "glass_crystal", "silver", "clocks", "jewelry", "gemstones", "rolex", "patek_philippe", "audemars_piguet", "omega", "other_watches", "luxury_handbags", "art", "contemporary_art", "photography"], | |
| 11 | + "regions": ["US", "GB", "FR", "IT", "DE"], | |
| 12 | + "languages": ["en"], | |
| 13 | + "currency": ["USD"], | |
| 14 | + "supportsListings": true, | |
| 15 | + "supportsSold": false, | |
| 16 | + "supportsAuctions": false, | |
| 17 | + "supportsImages": true, | |
| 18 | + "supportsCatalog": false, | |
| 19 | + "supportsPopulation": false, | |
| 20 | + "supportsLookup": false, | |
| 21 | + "refreshFrequencyMinutes": 1440, | |
| 22 | + "priority": "medium", | |
| 23 | + "trustScore": 0.8, | |
| 24 | + "attributionRequired": true, | |
| 25 | + "termsUrl": "https://www.1stdibs.com/about/user-agreement/", | |
| 26 | + "acquisitionMethod": "public HTML — server-rendered Relay store (serverVars_data JSON) on browse pages", | |
| 27 | + "historicalDepth": "none", | |
| 28 | + "accessNotes": "1stDibs browse pages (/furniture/seating/, /furniture/lighting/, /furniture/decorative-objects/, /jewelry/, /jewelry/watches/wrist-watches/, /fashion/handbags-purses-bags/, /art/prints-works-on-paper/ … with ?page=N, 60 items per page, at most 50 pages per browse path as capped by the site) are served to the honest RareIndexBot UA as ~3 MB server-rendered documents whose <script id=\"serverVars_data\"> JSON contains the Relay store used to hydrate the grid. We read the `Item` records from that store: title, serviceId (f_########), the retail display price converted by 1stDibs into USD/EUR/GBP/CAD/AUD/CHF/MXN/NOK/SEK/DKK (USD kept as the listing price, the others in metadata), seller company + dealer id, designer/creator names, dealer city/country, period-style attribute line, materials, measurements, photos, sold/on-hold flags. We never call 1stDibs' GraphQL (/soa/graphql — reserved to the site, robots.txt allows only that exact path) nor the /search/ paths robots.txt disallows; product detail pages are not fetched. The page carries an F5/Shape client-detection script, but no challenge was served to plain HTTPS requests; if 1stDibs starts blocking, the connector fails loudly (schema_drift/page_fetch_failed) — no evasion engine is configured. User Agreement (checked 2026-09-08) has no clause on automated access to public pages. Dealer asking prices only → listings (never sales); 5 s between requests, 1 in flight, 3 browse paths × 1 page per daily run with seed rotation (≈180 listings/run, ~9 MB), backfill capped at the site's 50-page window. Category mapping: seed vertical + category code + attribute line/title → design_furniture (default), antiques (period evidence), porcelain/glass/silver/clocks (decorative objects), jewelry/gemstones, watch brand slugs, luxury_handbags, art/photography/contemporary_art; rugs & textiles skipped.", | |
| 29 | + "enabled": true, | |
| 30 | + "schemaVersion": "1.0", | |
| 31 | + "config": { | |
| 32 | + "pagesPerSeed": 1, | |
| 33 | + "seedsPerRun": 3, | |
| 34 | + "maxPagesPerSeed": 50, | |
| 35 | + "seeds": [ | |
| 36 | + { "path": "/furniture/seating/", "vertical": "furniture" }, | |
| 37 | + { "path": "/furniture/tables/", "vertical": "furniture" }, | |
| 38 | + { "path": "/furniture/storage-case-pieces/", "vertical": "furniture" }, | |
| 39 | + { "path": "/furniture/lighting/", "vertical": "lighting" }, | |
| 40 | + { "path": "/furniture/decorative-objects/", "vertical": "decor" }, | |
| 41 | + { "path": "/jewelry/", "vertical": "jewelry" }, | |
| 42 | + { "path": "/jewelry/watches/wrist-watches/", "vertical": "watches" }, | |
| 43 | + { "path": "/fashion/handbags-purses-bags/", "slug": "luxury_handbags", "vertical": "fashion" }, | |
| 44 | + { "path": "/art/prints-works-on-paper/", "vertical": "art" } | |
| 45 | + ] | |
| 46 | + } | |
| 47 | +} | |
added
connectors/api/401-games/README.md
+44 −0
@@ -0,0 +1,44 @@ | ||
| 1 | +# 401 Games connector (`401-games`) | |
| 2 | + | |
| 3 | +- Source: https://store.401games.ca · dealer · CA · CAD · Shopify storefront | |
| 4 | +- Acquisition: public `/collections/<handle>/products.json?limit=250&page=N` (shared `ShopifyStoreConnector` adapter wrapped by `MarketPinnedShopifyConnector`, which adds a `localization=CA` cookie so Shopify Markets never geo-converts prices; no store-specific code) | |
| 5 | +- Records: `listing` (asking prices, one per variant; out-of-stock variants → `ended`) | |
| 6 | +- Refresh: every 12 h (ACTIVE→NORMAL boundary, large catalogue) · history: none (listings only) | |
| 7 | + | |
| 8 | +Canada's largest TCG/board-game retailer (Toronto). Public Shopify storefront with one umbrella collection per game plus sports and non-sport cards. | |
| 9 | + | |
| 10 | +## Collections → taxonomy | |
| 11 | +| collection handle | category | brand / franchise / series | | |
| 12 | +|---|---|---| | |
| 13 | +| `all-magic-the-gathering` | `magic_the_gathering` | Magic: The Gathering | | |
| 14 | +| `all-pokemon` | `pokemon` | Pokémon | | |
| 15 | +| `all-yugioh` | `yugioh` | Yu-Gi-Oh! | | |
| 16 | +| `all-one-piece-card-game` | `one_piece_card_game` | One Piece | | |
| 17 | +| `all-disney-lorcana` | `disney_lorcana` | Disney Lorcana | | |
| 18 | +| `all-weiss-schwarz` | `weiss_schwarz` | Weiß Schwarz | | |
| 19 | +| `all-digimon-card-game` | `digimon_tcg` | Digimon | | |
| 20 | +| `all-dragon-ball-super` | `dragon_ball_tcg` | Dragon Ball Super | | |
| 21 | +| `all-star-wars-unlimited` | `star_wars_tcg` | Star Wars Unlimited | | |
| 22 | +| `all-cardfight-vanguard` | `other_tcg` | Cardfight!! Vanguard | | |
| 23 | +| `all-riftbound-league-of-legends-tcg` | `other_tcg` | Riftbound | | |
| 24 | +| `all-gundam-card-game` | `other_tcg` | Gundam Card Game | | |
| 25 | +| `all-sports-cards` | `sports_cards` | — | | |
| 26 | +| `all-non-sports-trading-cards` | `non_sport_cards` | — | | |
| 27 | +| `all-board-games` | `board_games` | — | | |
| 28 | + | |
| 29 | +Rules (first match wins, applied to "title | product_type | tags | collection"): | |
| 30 | +- `^(?=[\s\S]*\bhockey\b)[\s\S]*\| all-sports-cards$` → `hockey_cards` | |
| 31 | +- `^(?=[\s\S]*\bbaseball\b)[\s\S]*\| all-sports-cards$` → `baseball_cards` | |
| 32 | +- `^(?=[\s\S]*\bbasketball\b)[\s\S]*\| all-sports-cards$` → `basketball_cards` | |
| 33 | +- `^(?=[\s\S]*\bfootball\b)[\s\S]*\| all-sports-cards$` → `football_cards` | |
| 34 | +- `^(?=[\s\S]*\b(soccer|premier league|uefa|fifa)\b)[\s\S]*\| all-sports-cards$` → `soccer_cards` | |
| 35 | +- `^(?=[\s\S]*\b(formula 1|formula one|\bf1\b)\b)[\s\S]*\| all-sports-cards$` → `f1_cards` | |
| 36 | + | |
| 37 | +Excluded (regex): `gift card|sleeves?|deck box|binder|playmat|toploader|top loader|storage box|dice|tokens? only|bundle of|mystery|buylist|submission|event ticket|\bentry\b|tournament|store credit|pre-?release event|supplies|card savers?|\brepack|bulk lot|\blot of\b|\bpaints?\b|primer|brushes|online (pack|code)|digital code|code card|unused digital|ptcgo|ptcg live` | |
| 38 | + | |
| 39 | +## Access & compliance | |
| 40 | +See `accessNotes` in meta.json: robots.txt verified 2026-09-08 (standard Shopify rules, products.json paths allowed), honest UA, 1 req / 1.5 s, listings only, no personal data. Not fetched: supplies/dice/modelling collections, event tickets, staff-loyalty and "acorn" migration collections, the 0.00-priced Easy-Buy buylist placeholders (dropped: no price). | |
| 41 | + | |
| 42 | +## Fixtures & tests | |
| 43 | +`data/fixtures/401-games/` — live captures (`pnpm tsx connectors/api/_g3-shops-na-lib/capture.ts 401-games`), trimmed single-product payloads incl. a sold-out variant and a graded item. | |
| 44 | +`pnpm vitest run connectors/api/401-games` · live probe: `pnpm tsx connectors/api/_g3-shops-na-lib/probe.ts 401-games`. | |
added
connectors/api/401-games/index.test.ts
+46 −0
@@ -0,0 +1,46 @@ | ||
| 1 | +import { describe, expect, it } from 'vitest'; | |
| 2 | +import meta from './meta.json' with { type: 'json' }; | |
| 3 | +import { describeShopifyStore } from '../_g3-shops-na-lib/store-suite.js'; | |
| 4 | +import createConnector from './index.js'; | |
| 5 | + | |
| 6 | +/** | |
| 7 | + * 401 Games — metadata-only Shopify store connector. The shared suite checks the taxonomy mapping of every | |
| 8 | + * configured collection/rule, the exclusion regex on real title shapes, and normalises the live-captured | |
| 9 | + * fixtures in data/fixtures/401-games/ (runFixtureSuite invariants + CAD currency, seller, SKUs, ended variants). | |
| 10 | + */ | |
| 11 | +describeShopifyStore(meta, createConnector, { describe, it, expect }, { | |
| 12 | + "cases": [ | |
| 13 | + { | |
| 14 | + "title": "2023-24 Upper Deck Series 2 Hockey Hobby Box", | |
| 15 | + "productType": "Sports Cards", | |
| 16 | + "collection": "all-sports-cards", | |
| 17 | + "categorySlug": "hockey_cards" | |
| 18 | + }, | |
| 19 | + { | |
| 20 | + "title": "2024 Topps Chrome Baseball Hobby Box", | |
| 21 | + "collection": "all-sports-cards", | |
| 22 | + "categorySlug": "baseball_cards" | |
| 23 | + }, | |
| 24 | + { | |
| 25 | + "title": "Easy-Buy Buylist Submission (Pokemon)", | |
| 26 | + "collection": "all-pokemon", | |
| 27 | + "categorySlug": null | |
| 28 | + }, | |
| 29 | + { | |
| 30 | + "title": "Pokemon - SWSH Darkness Ablaze Online Pack (Unused Digital Code)", | |
| 31 | + "collection": "all-pokemon", | |
| 32 | + "categorySlug": null | |
| 33 | + }, | |
| 34 | + { | |
| 35 | + "title": "Ultra Pro Deck Box - Black", | |
| 36 | + "collection": "all-magic-the-gathering", | |
| 37 | + "categorySlug": null | |
| 38 | + }, | |
| 39 | + { | |
| 40 | + "title": "Mareep (Japanese) - 036/172 - No Rarity", | |
| 41 | + "collection": "all-pokemon", | |
| 42 | + "categorySlug": "pokemon", | |
| 43 | + "franchise": "Pokémon" | |
| 44 | + } | |
| 45 | + ] | |
| 46 | +}); | |
added
connectors/api/401-games/index.ts
+9 −0
@@ -0,0 +1,9 @@ | ||
| 1 | +import type { ConnectorMeta } from '@rareindex/connectors'; | |
| 2 | +import { MarketPinnedShopifyConnector } from '../_g3-shops-na-lib/shopify-market.js'; | |
| 3 | + | |
| 4 | +/** | |
| 5 | + * 401 Games — Shopify storefront read through the shared Shopify adapter, pinned to the shop's home market | |
| 6 | + * (localization cookie) so Shopify Markets never geo-converts prices. All source-specific knowledge lives in | |
| 7 | + * meta.json (collections → taxonomy mapping, rules, exclusions, currency, market). | |
| 8 | + */ | |
| 9 | +export default (meta: ConnectorMeta) => new MarketPinnedShopifyConnector(meta); | |
added
connectors/api/401-games/meta.json
+172 −0
@@ -0,0 +1,172 @@ | ||
| 1 | +{ | |
| 2 | + "id": "401-games", | |
| 3 | + "displayName": "401 Games (Canadian TCG & sports-card & board-game store, CAD)", | |
| 4 | + "sourceId": "401-games", | |
| 5 | + "sourceName": "401 Games", | |
| 6 | + "sourceType": "dealer", | |
| 7 | + "sourceUrl": "https://store.401games.ca", | |
| 8 | + "module": "api/401-games", | |
| 9 | + "enginePriority": [ | |
| 10 | + "api" | |
| 11 | + ], | |
| 12 | + "categories": [ | |
| 13 | + "magic_the_gathering", | |
| 14 | + "pokemon", | |
| 15 | + "yugioh", | |
| 16 | + "one_piece_card_game", | |
| 17 | + "disney_lorcana", | |
| 18 | + "weiss_schwarz", | |
| 19 | + "digimon_tcg", | |
| 20 | + "dragon_ball_tcg", | |
| 21 | + "star_wars_tcg", | |
| 22 | + "other_tcg", | |
| 23 | + "sports_cards", | |
| 24 | + "non_sport_cards", | |
| 25 | + "board_games", | |
| 26 | + "hockey_cards", | |
| 27 | + "baseball_cards", | |
| 28 | + "basketball_cards", | |
| 29 | + "football_cards", | |
| 30 | + "soccer_cards", | |
| 31 | + "f1_cards" | |
| 32 | + ], | |
| 33 | + "regions": [ | |
| 34 | + "CA" | |
| 35 | + ], | |
| 36 | + "languages": [ | |
| 37 | + "en" | |
| 38 | + ], | |
| 39 | + "currency": [ | |
| 40 | + "CAD" | |
| 41 | + ], | |
| 42 | + "supportsListings": true, | |
| 43 | + "supportsSold": false, | |
| 44 | + "supportsAuctions": false, | |
| 45 | + "supportsImages": true, | |
| 46 | + "supportsCatalog": false, | |
| 47 | + "supportsPopulation": false, | |
| 48 | + "supportsLookup": true, | |
| 49 | + "refreshFrequencyMinutes": 720, | |
| 50 | + "priority": "medium", | |
| 51 | + "trustScore": 0.75, | |
| 52 | + "attributionRequired": true, | |
| 53 | + "termsUrl": "https://store.401games.ca/policies/terms-of-service", | |
| 54 | + "accessNotes": "401 Games (store.401games.ca) runs on Shopify. The connector reads ONLY the public, unauthenticated storefront JSON that every Shopify shop serves to any visitor: /collections/<handle>/products.json?limit=250&page=N for the 15 configured collections (all-magic-the-gathering, all-pokemon, all-yugioh, all-one-piece-card-game, all-disney-lorcana, all-weiss-schwarz, all-digimon-card-game, all-dragon-ball-super … (+7 more, see config.collections)) and /products/<handle>.json for URL lookups (~430k products; MTG 158k, Yu-Gi-Oh! 60k, Pokémon 54k). robots.txt (verified 2026-09-08, User-agent *): standard Shopify rules — Disallow /admin, /cart, /checkout(s), /orders, /account, /services, /sf_*, /cart.js, /recommendations/products, /cdn/wpm/*.js and the filter/sort crawl traps (/collections/*sort_by*, /collections/*+*, /collections/*filter*&*filter*); no Crawl-delay for *. The /collections/<handle>/products.json and /products/<handle>.json paths we read carry none of those patterns and are allowed. Sitemap declared at /sitemap.xml (not needed). Access behaviour: plain HTTP 200 JSON with the honest RareIndexBot UA, no anti-bot challenge, no login, no key. Market pinning: Shopify Markets localises products.json prices by the requester IP as soon as an Accept-Language header is present (observed on Markets-enabled shops: a Canadian crawler receives CAD-converted prices with content-language en-CA while the JSON carries no currency field). Every request therefore sends the public localization=CA cookie — the shop's own home market, exactly what a CA visitor gets — so prices are always the declared CAD (verified 2026-09-08: 30.00 vs 43.00 on a Markets shop). Politeness: 1 request per 1.5 s, concurrency 1 per host (connectors/domains.d/g3-shops-na.json), crawlDepth pages per collection per incremental run, backfill bounded by backfillMaxPages. Data: asking prices in CAD (the shop's declared currency) — these are LISTINGS, never sales (§111); one listing per variant (condition/size/edition), variant SKU kept as identifier, out-of-stock variants kept as 'ended' listings, 0.00-priced placeholders dropped. Dates: published_at only; no fetch time is ever used as a sale date. Third-party grades in titles (PSA/BGS/CGC/ICCS/PMG…) are parsed by parseGradeFromTitle; cert numbers are not extracted. Deliberately NOT fetched: cart/checkout/account/customer endpoints, per-product .js barcode enrichment (fetchBarcodes=false: one extra request per product is not worth it for listings), the whole-shop /products.json, and unmapped collections — supplies/dice/modelling collections, event tickets, staff-loyalty and \"acorn\" migration collections, the 0.00-priced Easy-Buy buylist placeholders (dropped: no price). No personal data is collected; seller = the store itself.", | |
| 55 | + "enabled": true, | |
| 56 | + "schemaVersion": "1.0", | |
| 57 | + "acquisitionMethod": "Shopify storefront products.json", | |
| 58 | + "historicalDepth": "none", | |
| 59 | + "requires": [], | |
| 60 | + "config": { | |
| 61 | + "currency": "CAD", | |
| 62 | + "market": "CA", | |
| 63 | + "seller": "401 Games", | |
| 64 | + "location": "Toronto, ON, Canada", | |
| 65 | + "collections": [ | |
| 66 | + { | |
| 67 | + "handle": "all-magic-the-gathering", | |
| 68 | + "categorySlug": "magic_the_gathering", | |
| 69 | + "franchise": "Magic: The Gathering" | |
| 70 | + }, | |
| 71 | + { | |
| 72 | + "handle": "all-pokemon", | |
| 73 | + "categorySlug": "pokemon", | |
| 74 | + "franchise": "Pokémon" | |
| 75 | + }, | |
| 76 | + { | |
| 77 | + "handle": "all-yugioh", | |
| 78 | + "categorySlug": "yugioh", | |
| 79 | + "franchise": "Yu-Gi-Oh!" | |
| 80 | + }, | |
| 81 | + { | |
| 82 | + "handle": "all-one-piece-card-game", | |
| 83 | + "categorySlug": "one_piece_card_game", | |
| 84 | + "franchise": "One Piece" | |
| 85 | + }, | |
| 86 | + { | |
| 87 | + "handle": "all-disney-lorcana", | |
| 88 | + "categorySlug": "disney_lorcana", | |
| 89 | + "franchise": "Disney Lorcana" | |
| 90 | + }, | |
| 91 | + { | |
| 92 | + "handle": "all-weiss-schwarz", | |
| 93 | + "categorySlug": "weiss_schwarz", | |
| 94 | + "franchise": "Weiß Schwarz" | |
| 95 | + }, | |
| 96 | + { | |
| 97 | + "handle": "all-digimon-card-game", | |
| 98 | + "categorySlug": "digimon_tcg", | |
| 99 | + "franchise": "Digimon" | |
| 100 | + }, | |
| 101 | + { | |
| 102 | + "handle": "all-dragon-ball-super", | |
| 103 | + "categorySlug": "dragon_ball_tcg", | |
| 104 | + "franchise": "Dragon Ball Super" | |
| 105 | + }, | |
| 106 | + { | |
| 107 | + "handle": "all-star-wars-unlimited", | |
| 108 | + "categorySlug": "star_wars_tcg", | |
| 109 | + "franchise": "Star Wars Unlimited" | |
| 110 | + }, | |
| 111 | + { | |
| 112 | + "handle": "all-cardfight-vanguard", | |
| 113 | + "categorySlug": "other_tcg", | |
| 114 | + "franchise": "Cardfight!! Vanguard" | |
| 115 | + }, | |
| 116 | + { | |
| 117 | + "handle": "all-riftbound-league-of-legends-tcg", | |
| 118 | + "categorySlug": "other_tcg", | |
| 119 | + "franchise": "Riftbound" | |
| 120 | + }, | |
| 121 | + { | |
| 122 | + "handle": "all-gundam-card-game", | |
| 123 | + "categorySlug": "other_tcg", | |
| 124 | + "franchise": "Gundam Card Game" | |
| 125 | + }, | |
| 126 | + { | |
| 127 | + "handle": "all-sports-cards", | |
| 128 | + "categorySlug": "sports_cards" | |
| 129 | + }, | |
| 130 | + { | |
| 131 | + "handle": "all-non-sports-trading-cards", | |
| 132 | + "categorySlug": "non_sport_cards" | |
| 133 | + }, | |
| 134 | + { | |
| 135 | + "handle": "all-board-games", | |
| 136 | + "categorySlug": "board_games" | |
| 137 | + } | |
| 138 | + ], | |
| 139 | + "rules": [ | |
| 140 | + { | |
| 141 | + "match": "^(?=[\\s\\S]*\\bhockey\\b)[\\s\\S]*\\| all-sports-cards$", | |
| 142 | + "categorySlug": "hockey_cards" | |
| 143 | + }, | |
| 144 | + { | |
| 145 | + "match": "^(?=[\\s\\S]*\\bbaseball\\b)[\\s\\S]*\\| all-sports-cards$", | |
| 146 | + "categorySlug": "baseball_cards" | |
| 147 | + }, | |
| 148 | + { | |
| 149 | + "match": "^(?=[\\s\\S]*\\bbasketball\\b)[\\s\\S]*\\| all-sports-cards$", | |
| 150 | + "categorySlug": "basketball_cards" | |
| 151 | + }, | |
| 152 | + { | |
| 153 | + "match": "^(?=[\\s\\S]*\\bfootball\\b)[\\s\\S]*\\| all-sports-cards$", | |
| 154 | + "categorySlug": "football_cards" | |
| 155 | + }, | |
| 156 | + { | |
| 157 | + "match": "^(?=[\\s\\S]*\\b(soccer|premier league|uefa|fifa)\\b)[\\s\\S]*\\| all-sports-cards$", | |
| 158 | + "categorySlug": "soccer_cards" | |
| 159 | + }, | |
| 160 | + { | |
| 161 | + "match": "^(?=[\\s\\S]*\\b(formula 1|formula one|\\bf1\\b)\\b)[\\s\\S]*\\| all-sports-cards$", | |
| 162 | + "categorySlug": "f1_cards" | |
| 163 | + } | |
| 164 | + ], | |
| 165 | + "defaultCategory": null, | |
| 166 | + "exclude": "gift card|sleeves?|deck box|binder|playmat|toploader|top loader|storage box|dice|tokens? only|bundle of|mystery|buylist|submission|event ticket|\\bentry\\b|tournament|store credit|pre-?release event|supplies|card savers?|\\brepack|bulk lot|\\blot of\\b|\\bpaints?\\b|primer|brushes|online (pack|code)|digital code|code card|unused digital|ptcgo|ptcg live", | |
| 167 | + "keepOutOfStock": true, | |
| 168 | + "fetchBarcodes": false, | |
| 169 | + "wholeShop": false, | |
| 170 | + "pageSize": 250 | |
| 171 | + } | |
| 172 | +} | |
added
connectors/api/_g1-cards-eu-jp-lib/index.ts
+261 −0
@@ -0,0 +1,261 @@ | ||
| 1 | +/** | |
| 2 | + * Helpers shared by the g1-cards-eu-jp connectors (cardmarket-priceguide, limitless-tcg, yuyu-tei, | |
| 3 | + * hareruya, magi, cardrush, cardtrader). Kept inside connectors/api (not the framework). | |
| 4 | + */ | |
| 5 | +import { normalizeCondition, parseGradeFromTitle } from '@rareindex/taxonomy'; | |
| 6 | + | |
| 7 | +export const BOT_UA = 'RareIndexBot/0.1 (+https://www.rareindex.io/about/data; contact data@rareindex.io)'; | |
| 8 | +export const HTML_HEADERS = { 'user-agent': BOT_UA, accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'accept-language': 'ja,en;q=0.8' }; | |
| 9 | +export const JSON_HEADERS = { 'user-agent': BOT_UA, accept: 'application/json, text/plain, */*;q=0.8' }; | |
| 10 | + | |
| 11 | +/** UTC midnight of a Date (observation dates are day-precise). */ | |
| 12 | +export function dayOf(d: Date): Date { | |
| 13 | + return new Date(Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate())); | |
| 14 | +} | |
| 15 | + | |
| 16 | +/** "2026-09-07T02:48:04+0200" | "2026-09-07" → UTC midnight of that calendar day (source's own date), else null. */ | |
| 17 | +export function isoDay(s: string | null | undefined): Date | null { | |
| 18 | + if (!s) return null; | |
| 19 | + const m = s.match(/^(\d{4})-(\d{2})-(\d{2})/); | |
| 20 | + if (!m) return null; | |
| 21 | + const d = new Date(Date.UTC(Number(m[1]), Number(m[2]) - 1, Number(m[3]))); | |
| 22 | + return Number.isNaN(d.getTime()) ? null : d; | |
| 23 | +} | |
| 24 | + | |
| 25 | +/** "118,000 円" | "¥ 187,614" | "780円" | "¥1,590" | "4200000" → integer yen; null when absent or ≤ 0. */ | |
| 26 | +export function yen(s: string | number | null | undefined): number | null { | |
| 27 | + if (s === null || s === undefined) return null; | |
| 28 | + if (typeof s === 'number') return Number.isFinite(s) && s > 0 ? Math.round(s) : null; | |
| 29 | + const m = s.replace(/[，,\s]/g, '').match(/(\d+)/); | |
| 30 | + if (!m) return null; | |
| 31 | + const n = Number.parseInt(m[1]!, 10); | |
| 32 | + return Number.isFinite(n) && n > 0 ? n : null; | |
| 33 | +} | |
| 34 | + | |
| 35 | +/** "$0.25" | "0.02€" | "1,585.73€" | "€1,179.09" → { amount, currency } (EN number format), else null. */ | |
| 36 | +export function usdEur(s: string | null | undefined): { amount: number; currency: 'USD' | 'EUR' } | null { | |
| 37 | + if (!s) return null; | |
| 38 | + const t = s.trim(); | |
| 39 | + const currency = /€|\bEUR\b/.test(t) ? 'EUR' : /\$|\bUSD\b/.test(t) ? 'USD' : null; | |
| 40 | + if (!currency) return null; | |
| 41 | + const m = t.replace(/,/g, '').match(/(\d+(?:\.\d+)?)/); | |
| 42 | + if (!m) return null; | |
| 43 | + const amount = Number.parseFloat(m[1]!); | |
| 44 | + return Number.isFinite(amount) && amount > 0 ? { amount, currency } : null; | |
| 45 | +} | |
| 46 | + | |
| 47 | +export function cleanText(s: string | null | undefined): string | null { | |
| 48 | + if (!s) return null; | |
| 49 | + const t = s.replace(/&/g, '&').replace(/'/g, "'").replace(/"/g, '"').replace(/</g, '<').replace(/>/g, '>').replace(/\s+/g, ' ').trim(); | |
| 50 | + return t || null; | |
| 51 | +} | |
| 52 | + | |
| 53 | +/** "113/076" → { number: "113", total: "076" }; "OP15-118" → { number: "OP15-118", total: null }; "-" → nulls. */ | |
| 54 | +export function splitCardNumber(s: string | null | undefined): { number: string | null; total: string | null } { | |
| 55 | + if (!s) return { number: null, total: null }; | |
| 56 | + const t = s.trim(); | |
| 57 | + if (!t || t === '-' || t === '—') return { number: null, total: null }; | |
| 58 | + const m = t.match(/^([A-Za-z0-9]+)\s*\/\s*([A-Za-z0-9-]+)$/); | |
| 59 | + if (m) return { number: m[1]!, total: m[2]! }; | |
| 60 | + return { number: t, total: null }; | |
| 61 | +} | |
| 62 | + | |
| 63 | +/** Grade parser tolerant of Japanese shop notation: "PSA10鑑定済", "【PSA10】", "BGS9.5" → grader/grade. */ | |
| 64 | +export function jpGrade(title: string): { grader: string | null; grade: string | null; qualifier: string | null } { | |
| 65 | + const spaced = title.replace(/(PSA|BGS|CGC|SGC|ACE|TAG)(\d)/gi, '$1 $2').replace(/鑑定済/g, ' graded '); | |
| 66 | + const g = parseGradeFromTitle(spaced); | |
| 67 | + return g; | |
| 68 | +} | |
| 69 | + | |
| 70 | +/** Japanese shop condition labels → taxonomy condition slug; null when the scale is shop-specific (状態A/B…) — never guessed. */ | |
| 71 | +export function jpCondition(raw: string | null | undefined): string | null { | |
| 72 | + if (!raw) return null; | |
| 73 | + const t = raw.trim(); | |
| 74 | + const direct = normalizeCondition('trading_cards', t); | |
| 75 | + if (direct) return direct; | |
| 76 | + if (/^(NM|near mint)$/i.test(t)) return 'near_mint'; | |
| 77 | + if (/^EX[+-]?$/i.test(t)) return 'excellent'; | |
| 78 | + if (/^(HP|heavily played)$/i.test(t)) return 'good'; | |
| 79 | + if (/^(DMG|damaged)$/i.test(t)) return 'poor'; | |
| 80 | + return null; | |
| 81 | +} | |
| 82 | + | |
| 83 | +export interface JpCardTitle { | |
| 84 | + /** card name with all tags stripped (EN half preferred for bilingual "JP/EN" names) */ | |
| 85 | + name: string; | |
| 86 | + /** contents of the leading 〔…〕/[…] tags: condition or grading ("状態A-", "PSA10鑑定済", "PLD") */ | |
| 87 | + conditionRaw: string | null; | |
| 88 | + /** 【…】 tag: rarity (Pokémon/One Piece shops) or set code (MTG shops) — caller decides via `bracket` */ | |
| 89 | + rarity: string | null; | |
| 90 | + setCode: string | null; | |
| 91 | + /** {…} tag: "013/068" → number 013, total 068; "OP01-001"; "-" → null */ | |
| 92 | + number: string | null; | |
| 93 | + total: string | null; | |
| 94 | + /** 《日本語》/《英語》 → "Japanese"/"English"; null when absent */ | |
| 95 | + language: string | null; | |
| 96 | + /** parenthesised hints: "(RR仕様)", "(パラレル)", "(MA仕様/英語版)" */ | |
| 97 | + notes: string[]; | |
| 98 | + /** trailing "N枚" (count of cards in the listing) */ | |
| 99 | + quantity: number | null; | |
| 100 | + sealed: boolean; | |
| 101 | + grader: string | null; | |
| 102 | + grade: string | null; | |
| 103 | + qualifier: string | null; | |
| 104 | +} | |
| 105 | + | |
| 106 | +const LANG_MAP: Record<string, string> = { 日本語: 'Japanese', 英語: 'English', 韓国語: 'Korean', 中国語: 'Chinese', 簡体字: 'Chinese', 繁体字: 'Chinese', ドイツ語: 'German', フランス語: 'French', イタリア語: 'Italian', スペイン語: 'Spanish', ポルトガル語: 'Portuguese', ロシア語: 'Russian' }; | |
| 107 | + | |
| 108 | +/** | |
| 109 | + * Parse the title conventions shared by Japanese single-card shops (Cardrush, Magi sellers, Yuyu-tei): | |
| 110 | + * "〔PSA10鑑定済〕ミロカロスδ-デルタ種【★】{013/068}" | |
| 111 | + * "〔状態A-〕ミュウツーV(RR仕様)【P】{273/S-P} 1枚" | |
| 112 | + * "[PLD](黒枠)ラノワールのエルフ/Llanowar Elves《日本語》【4ED】" | |
| 113 | + * "ブースターパック 受け継がれる意志【未開封BOX】{-}" | |
| 114 | + * `bracket` tells whether the 【…】 tag is a rarity (default) or a set code (MTG shops). | |
| 115 | + */ | |
| 116 | +export function parseJpCardTitle(title: string, opts: { bracket?: 'rarity' | 'set' } = {}): JpCardTitle { | |
| 117 | + let t = title.replace(/\s+/g, ' ').trim(); | |
| 118 | + const conds: string[] = []; | |
| 119 | + // leading condition/grading tags | |
| 120 | + for (;;) { | |
| 121 | + const m = t.match(/^(?:〔([^〕]*)〕|\[([^\]]*)\])\s*/); | |
| 122 | + if (!m) break; | |
| 123 | + conds.push((m[1] ?? m[2] ?? '').trim()); | |
| 124 | + t = t.slice(m[0].length); | |
| 125 | + } | |
| 126 | + let sealed = false; | |
| 127 | + let rarity: string | null = null; | |
| 128 | + let setCode: string | null = null; | |
| 129 | + t = t.replace(/【([^】]*)】/g, (_, inner: string) => { | |
| 130 | + const v = inner.trim(); | |
| 131 | + if (/^(PSA|BGS|CGC|SGC|ACE|TAG|ARS)\s*\d/i.test(v)) conds.push(v); // "【PSA10】" = grading tag, not a rarity | |
| 132 | + else if (/未開封|BOX|パック|カートン/i.test(v)) sealed = true; | |
| 133 | + else if (opts.bracket === 'set') setCode = setCode ?? (v || null); | |
| 134 | + else rarity = rarity ?? (v && v !== '-' ? v : null); | |
| 135 | + return ' '; | |
| 136 | + }); | |
| 137 | + const g = jpGrade(conds.join(' ')); | |
| 138 | + let number: string | null = null; | |
| 139 | + let total: string | null = null; | |
| 140 | + t = t.replace(/\{([^}]*)\}/g, (_, inner: string) => { | |
| 141 | + // "{ST15-005[OP16]}" → number ST15-005, set code OP16 (reprint origin) | |
| 142 | + const withSet = inner.match(/^(.*?)\[([A-Za-z0-9-]+)\]\s*$/); | |
| 143 | + const s = splitCardNumber(withSet ? withSet[1]! : inner); | |
| 144 | + if (withSet && opts.bracket !== 'set') setCode = setCode ?? withSet[2]!; | |
| 145 | + if (s.number && number === null) { | |
| 146 | + number = s.number; | |
| 147 | + total = s.total; | |
| 148 | + } | |
| 149 | + return ' '; | |
| 150 | + }); | |
| 151 | + let language: string | null = null; | |
| 152 | + t = t.replace(/《([^》]*)》/g, (_, inner: string) => { | |
| 153 | + language = language ?? LANG_MAP[inner.trim()] ?? inner.trim() ?? null; | |
| 154 | + return ' '; | |
| 155 | + }); | |
| 156 | + let quantity: number | null = null; | |
| 157 | + const q = t.match(/(?:^|\s)(\d{1,3})枚(?:セット|組)?\s*$/); | |
| 158 | + if (q) { | |
| 159 | + quantity = Number(q[1]); | |
| 160 | + t = t.slice(0, q.index).trim(); | |
| 161 | + } | |
| 162 | + const notes: string[] = []; | |
| 163 | + t = t.replace(/[（(]([^()（）]*)[)）]/g, (_, inner: string) => { | |
| 164 | + const v = inner.trim(); | |
| 165 | + if (v) notes.push(v); | |
| 166 | + if (/未開封/.test(v)) sealed = true; | |
| 167 | + return ' '; | |
| 168 | + }); | |
| 169 | + // bare "120/114" card number outside braces (private sellers often omit the braces) | |
| 170 | + if (number === null) { | |
| 171 | + const bare = t.match(/(?:^|\s)(\d{1,3})\s*\/\s*(\d{1,3})(?=\s|$)/); | |
| 172 | + if (bare) { | |
| 173 | + number = bare[1]!; | |
| 174 | + total = bare[2]!; | |
| 175 | + t = t.replace(bare[0], ' '); | |
| 176 | + } | |
| 177 | + } | |
| 178 | + let name = t.replace(/\s+/g, ' ').trim(); | |
| 179 | + // bilingual MTG names "ラノワールのエルフ/Llanowar Elves" → prefer the Latin half | |
| 180 | + const bi = name.match(/^(.+?)\/(.+)$/); | |
| 181 | + if (bi && /[A-Za-z]/.test(bi[2]!) && !/[A-Za-z]/.test(bi[1]!)) name = bi[2]!.trim(); | |
| 182 | + const conditionRaw = conds.filter((c) => c && !/PSA|BGS|CGC|SGC|ACE|TAG|鑑定/i.test(c)).join(' / ') || (conds.length ? conds.join(' / ') : null); | |
| 183 | + return { name: name || title.trim(), conditionRaw, rarity, setCode, number, total, language, notes, quantity, sealed, grader: g.grader, grade: g.grade, qualifier: g.qualifier }; | |
| 184 | +} | |
| 185 | + | |
| 186 | +/** | |
| 187 | + * JSON.parse for JSON-LD blocks that embed raw line breaks inside string values (magi item pages do): | |
| 188 | + * escapes control characters found inside strings, then parses. Returns null on failure. | |
| 189 | + */ | |
| 190 | +export function lenientJsonParse(text: string): unknown | null { | |
| 191 | + try { | |
| 192 | + return JSON.parse(text); | |
| 193 | + } catch { | |
| 194 | + let out = ''; | |
| 195 | + let inStr = false; | |
| 196 | + for (let i = 0; i < text.length; i++) { | |
| 197 | + const ch = text[i]!; | |
| 198 | + if (inStr) { | |
| 199 | + if (ch === '\\') { | |
| 200 | + out += ch + (text[i + 1] ?? ''); | |
| 201 | + i++; | |
| 202 | + continue; | |
| 203 | + } | |
| 204 | + if (ch === '"') inStr = false; | |
| 205 | + else if (ch === '\n') { | |
| 206 | + out += '\\n'; | |
| 207 | + continue; | |
| 208 | + } else if (ch === '\r') continue; | |
| 209 | + else if (ch === '\t') { | |
| 210 | + out += '\\t'; | |
| 211 | + continue; | |
| 212 | + } | |
| 213 | + } else if (ch === '"') inStr = true; | |
| 214 | + out += ch; | |
| 215 | + } | |
| 216 | + try { | |
| 217 | + return JSON.parse(out); | |
| 218 | + } catch { | |
| 219 | + return null; | |
| 220 | + } | |
| 221 | + } | |
| 222 | +} | |
| 223 | + | |
| 224 | +/** All JSON-LD objects of a given @type in a document (lenient about raw newlines inside strings). */ | |
| 225 | +export function jsonLdObjects(doc: string, type: string): Record<string, unknown>[] { | |
| 226 | + const out: Record<string, unknown>[] = []; | |
| 227 | + const re = /<script[^>]+type="application\/ld\+json"[^>]*>([\s\S]*?)<\/script>/gi; | |
| 228 | + let m: RegExpExecArray | null; | |
| 229 | + while ((m = re.exec(doc))) { | |
| 230 | + const parsed = lenientJsonParse(m[1]!.trim()); | |
| 231 | + const items = Array.isArray(parsed) ? parsed : parsed && typeof parsed === 'object' && '@graph' in (parsed as object) ? ((parsed as { '@graph': unknown[] })['@graph'] ?? []) : [parsed]; | |
| 232 | + for (const it of items) { | |
| 233 | + if (!it || typeof it !== 'object') continue; | |
| 234 | + const t = (it as { '@type'?: string | string[] })['@type']; | |
| 235 | + const types = Array.isArray(t) ? t : t ? [t] : []; | |
| 236 | + if (types.includes(type)) out.push(it as Record<string, unknown>); | |
| 237 | + } | |
| 238 | + } | |
| 239 | + return out; | |
| 240 | +} | |
| 241 | + | |
| 242 | +/** Variant label from Japanese printing hints: パラレル → Parallel, ミラー → Mirror, SR仕様 → "SR". */ | |
| 243 | +export function jpVariant(notes: string[], name?: string): string | null { | |
| 244 | + const hay = [...notes, name ?? ''].join(' '); | |
| 245 | + if (/パラレル/.test(hay)) return 'Parallel'; | |
| 246 | + if (/マスターボールミラー/.test(hay)) return 'Master Ball Mirror'; | |
| 247 | + if (/モンスターボールミラー/.test(hay)) return 'Poké Ball Mirror'; | |
| 248 | + if (/ミラー/.test(hay)) return 'Mirror'; | |
| 249 | + if (/コミックパラレル|コミパラ/.test(hay)) return 'Comic Parallel'; | |
| 250 | + if (/箔押し|foil/i.test(hay)) return 'Foil'; | |
| 251 | + return null; | |
| 252 | +} | |
| 253 | + | |
| 254 | +/** Mystery packs / grab bags / lots sold by Japanese shops — never priced as cards. */ | |
| 255 | +export const JP_EXCLUDE_RE = /オリパ|福袋|まとめ売り|詰め合わせ|セット販売|スリーブ|プレイマット|デッキケース|サプライ|ストレージ|ローダー|募集用/; | |
| 256 | + | |
| 257 | +/** Bundle detection: "3枚", "4コン", "セット", "まとめ" (single card listings say "1枚"). */ | |
| 258 | +export function isJpBundle(title: string, quantity: number | null): boolean { | |
| 259 | + if (quantity !== null && quantity > 1) return true; | |
| 260 | + return /\d+コン|セット(?!ブースター)|まとめ|複数枚/.test(title) && !/スターターセット|ex?スタート|構築済み/.test(title); | |
| 261 | +} | |
added
connectors/api/_g1-cards-eu-jp-lib/smoke.ts
+61 −0
Large diff (61 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g10-lib/capture.ts
+139 −0
Large diff (139 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g10-lib/index.ts
+211 −0
Large diff (211 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g3-shops-na-lib/capture.ts
+144 −0
Large diff (144 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g3-shops-na-lib/probe.ts
+51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g3-shops-na-lib/shopify-market.ts
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g3-shops-na-lib/store-suite.ts
+209 −0
Large diff (209 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g4-shops-intl-lib/capture.ts
+108 −0
Large diff (108 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g4-shops-intl-lib/smoke.ts
+47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g4-shops-intl-lib/suite.ts
+124 −0
Large diff (124 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g6-comics-toys-games-lib/_inspect.ts
+21 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g6-comics-toys-games-lib/capture-env.ts
+28 −0
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g6-comics-toys-games-lib/capture.ts
+73 −0
Large diff (73 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g6-comics-toys-games-lib/comics.ts
+141 −0
Large diff (141 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g7-auctions-na-lib/index.ts
+263 −0
Large diff (263 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g7-auctions-na-lib/smoke.ts
+76 −0
Large diff (76 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g8-auctions-eu-apac-lib/capture.ts
+52 −0
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g8-auctions-eu-apac-lib/drouot-platform.ts
+108 −0
Large diff (108 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g8-auctions-eu-apac-lib/goauction-platform.ts
+114 −0
Large diff (114 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g8-auctions-eu-apac-lib/index.ts
+317 −0
Large diff (317 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g8-auctions-eu-apac-lib/sale-results.ts
+383 −0
Large diff (383 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g9-asia-watch-sneaker-lib/capture.ts
+103 −0
Large diff (103 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/_g9-asia-watch-sneaker-lib/index.ts
+220 −0
Large diff (220 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/ader/README.md
+11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/ader/index.test.ts
+77 −0
Large diff (77 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/ader/index.ts
+55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/ader/meta.json
+36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/aguttes/README.md
+13 −0
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/aguttes/index.test.ts
+94 −0
Large diff (94 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/aguttes/index.ts
+132 −0
Large diff (132 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/aguttes/meta.json
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/aste-bolaffi/README.md
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/aste-bolaffi/index.test.ts
+65 −0
Large diff (65 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/aste-bolaffi/index.ts
+107 −0
Large diff (107 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/aste-bolaffi/meta.json
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/auctionet/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/auctionet/_capture.ts
+49 −0
Large diff (49 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/auctionet/index.test.ts
+193 −0
Large diff (193 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/auctionet/index.ts
+390 −0
Large diff (390 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/auctionet/meta.json
+39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/bernaerts/README.md
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/bernaerts/index.test.ts
+62 −0
Large diff (62 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/bernaerts/index.ts
+125 −0
Large diff (125 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/bernaerts/meta.json
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/big-b-comics/README.md
+45 −0
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/big-b-comics/index.test.ts
+36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/big-b-comics/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/big-b-comics/meta.json
+161 −0
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/bunjang/README.md
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/bunjang/index.test.ts
+53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/bunjang/index.ts
+174 −0
Large diff (174 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/bunjang/meta.json
+50 −0
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/capsule-toronto/README.md
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/capsule-toronto/index.test.ts
+39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/capsule-toronto/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/capsule-toronto/meta.json
+107 −0
Large diff (107 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardboard-memories/README.md
+45 −0
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardboard-memories/index.test.ts
+36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardboard-memories/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardboard-memories/meta.json
+170 −0
Large diff (170 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardmarket-priceguide/README.md
+12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardmarket-priceguide/_capture.ts
+48 −0
Large diff (48 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardmarket-priceguide/index.test.ts
+63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardmarket-priceguide/index.ts
+289 −0
Large diff (289 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardmarket-priceguide/meta.json
+56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardmerchant-nz/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardmerchant-nz/index.test.ts
+62 −0
Large diff (62 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardmerchant-nz/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardmerchant-nz/meta.json
+136 −0
Large diff (136 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardrush/README.md
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardrush/_capture.ts
+42 −0
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardrush/index.test.ts
+68 −0
Large diff (68 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardrush/index.ts
+176 −0
Large diff (176 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardrush/meta.json
+44 −0
Large diff (44 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardtrader/README.md
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardtrader/index.test.ts
+60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardtrader/index.ts
+217 −0
Large diff (217 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cardtrader/meta.json
+54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cdncoin/README.md
+42 −0
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cdncoin/index.test.ts
+31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cdncoin/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cdncoin/meta.json
+136 −0
Large diff (136 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chairish/README.md
+12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chairish/index.test.ts
+53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chairish/index.ts
+255 −0
Large diff (255 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chairish/meta.json
+46 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chatterley-luxuries/README.md
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chatterley-luxuries/index.test.ts
+32 −0
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chatterley-luxuries/index.ts
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chatterley-luxuries/meta.json
+49 −0
Large diff (49 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cherrycollectables/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cherrycollectables/index.test.ts
+88 −0
Large diff (88 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cherrycollectables/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/cherrycollectables/meta.json
+235 −0
Large diff (235 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chiswick-auctions/README.md
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chiswick-auctions/index.test.ts
+38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chiswick-auctions/index.ts
+40 −0
Large diff (40 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chiswick-auctions/meta.json
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chronext/README.md
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chronext/index.test.ts
+72 −0
Large diff (72 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chronext/index.ts
+210 −0
Large diff (210 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/chronext/meta.json
+43 −0
Large diff (43 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/clean-sweep/README.md
+11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/clean-sweep/index.test.ts
+83 −0
Large diff (83 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/clean-sweep/index.ts
+269 −0
Large diff (269 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/clean-sweep/meta.json
+36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/colonial-acres/README.md
+58 −0
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/colonial-acres/index.test.ts
+41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/colonial-acres/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/colonial-acres/meta.json
+200 −0
Large diff (200 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/corinphila/README.md
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/corinphila/_smoke.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/corinphila/index.test.ts
+59 −0
Large diff (59 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/corinphila/index.ts
+232 −0
Large diff (232 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/corinphila/meta.json
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/crepslocker/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/crepslocker/index.test.ts
+78 −0
Large diff (78 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/crepslocker/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/crepslocker/meta.json
+154 −0
Large diff (154 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dlrc/README.md
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dlrc/_smoke.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dlrc/index.test.ts
+52 −0
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dlrc/index.ts
+294 −0
Large diff (294 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dlrc/meta.json
+36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dominic-winter/README.md
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dominic-winter/index.test.ts
+51 −0
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dominic-winter/index.ts
+40 −0
Large diff (40 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dominic-winter/meta.json
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/doyle/README.md
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/doyle/index.test.ts
+83 −0
Large diff (83 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/doyle/index.ts
+279 −0
Large diff (279 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/doyle/meta.json
+36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dunbar-sloane/README.md
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dunbar-sloane/index.test.ts
+63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dunbar-sloane/index.ts
+114 −0
Large diff (114 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/dunbar-sloane/meta.json
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/ebay-browse/README.md
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/ebay-browse/index.test.ts
+82 −0
Large diff (82 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/ebay-browse/index.ts
+313 −0
Large diff (313 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/ebay-browse/meta.json
+67 −0
Large diff (67 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/etsy/README.md
+14 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/etsy/index.test.ts
+56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/etsy/index.ts
+307 −0
Large diff (307 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/etsy/meta.json
+62 −0
Large diff (62 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/european-watch-company/README.md
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/european-watch-company/index.test.ts
+63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/european-watch-company/index.ts
+208 −0
Large diff (208 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/european-watch-company/meta.json
+36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/everything-games/README.md
+31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/everything-games/index.test.ts
+25 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/everything-games/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/everything-games/meta.json
+102 −0
Large diff (102 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/face-to-face-games/README.md
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/face-to-face-games/index.test.ts
+30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/face-to-face-games/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/face-to-face-games/meta.json
+125 −0
Large diff (125 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/fanatics-collect/README.md
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/fanatics-collect/index.test.ts
+137 −0
Large diff (137 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/fanatics-collect/index.ts
+276 −0
Large diff (276 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/fanatics-collect/meta.json
+36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/flagshipgames/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/flagshipgames/index.test.ts
+76 −0
Large diff (76 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/flagshipgames/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/flagshipgames/meta.json
+182 −0
Large diff (182 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/footdistrict/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/footdistrict/index.test.ts
+63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/footdistrict/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/footdistrict/meta.json
+116 −0
Large diff (116 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/fusion-gaming/README.md
+33 −0
Large diff (33 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/fusion-gaming/index.test.ts
+25 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/fusion-gaming/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/fusion-gaming/meta.json
+113 −0
Large diff (113 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gameology/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gameology/index.test.ts
+77 −0
Large diff (77 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gameology/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gameology/meta.json
+283 −0
Large diff (283 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gamesportal/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gamesportal/index.test.ts
+63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gamesportal/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gamesportal/meta.json
+124 −0
Large diff (124 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gamezilla/README.md
+49 −0
Large diff (49 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gamezilla/index.test.ts
+31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gamezilla/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gamezilla/meta.json
+200 −0
Large diff (200 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gcd/README.md
+15 −0
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gcd/_capture.ts
+45 −0
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gcd/index.test.ts
+73 −0
Large diff (73 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gcd/index.ts
+369 −0
Large diff (369 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/gcd/meta.json
+70 −0
Large diff (70 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/goblingames-nz/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/goblingames-nz/index.test.ts
+73 −0
Large diff (73 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/goblingames-nz/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/goblingames-nz/meta.json
+125 −0
Large diff (125 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/goodgames/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/goodgames/index.test.ts
+82 −0
Large diff (82 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/goodgames/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/goodgames/meta.json
+188 −0
Large diff (188 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hareruya/README.md
+11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hareruya/_capture.ts
+37 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hareruya/index.test.ts
+58 −0
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hareruya/index.ts
+193 −0
Large diff (193 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hareruya/meta.json
+38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hibid/README.md
+20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hibid/categories.ts
+97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hibid/fixtures.state.json
+1 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hibid/index.test.ts
+115 −0
Large diff (115 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hibid/index.ts
+295 −0
Large diff (295 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hibid/meta.json
+37 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hobbiesville/README.md
+36 −0
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hobbiesville/index.test.ts
+32 −0
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hobbiesville/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hobbiesville/meta.json
+125 −0
Large diff (125 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hodinkee-shop/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hodinkee-shop/index.test.ts
+90 −0
Large diff (90 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hodinkee-shop/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/hodinkee-shop/meta.json
+163 −0
Large diff (163 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kanzen-games/README.md
+47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kanzen-games/index.test.ts
+30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kanzen-games/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kanzen-games/meta.json
+177 −0
Large diff (177 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kapowtoys/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kapowtoys/index.test.ts
+60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kapowtoys/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kapowtoys/meta.json
+166 −0
Large diff (166 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kickgame/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kickgame/index.test.ts
+91 −0
Large diff (91 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kickgame/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kickgame/meta.json
+164 −0
Large diff (164 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kidrobot/README.md
+34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kidrobot/index.test.ts
+39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kidrobot/index.ts
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kidrobot/meta.json
+120 −0
Large diff (120 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kingscomics/README.md
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kingscomics/index.test.ts
+127 −0
Large diff (127 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kingscomics/index.ts
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
connectors/api/kingscomics/meta.json
+112 −0
Large diff (112 lines) is not loaded by default to keep the page fast.Load diff
Diff truncated — file too large.