"use client"; import { useMemo, useState, type FormEvent } from "react"; import { Plus, RefreshCw, RotateCcw, XCircle } from "lucide-react"; import { api, type Dataset, type HealthReport, type Job } from "@/lib/api"; import { useAsync, useInterval } from "@/lib/hooks"; import { useToast } from "@/lib/toast"; import { useAuth } from "@/lib/auth"; import { daysAgo, fmtDuration, fmtInt, fmtTs, splitList, today } from "@/lib/format"; import { Banner, Card, Empty, ErrorState, Field, PageHeader, StatusBadge } from "@/components/ui"; import { DataTable } from "@/components/DataTable"; import { JsonView } from "@/components/JsonView"; import { ConfirmDialog, Modal } from "@/components/ConfirmDialog"; const KINDS = ["incremental", "metadata", "backfill", "fundamentals", "derive", "compaction", "quality", "backup", "gc"] as const; type Kind = (typeof KINDS)[number]; const STATUSES = ["", "queued", "running", "succeeded", "failed", "dead", "cancelled"]; const KIND_HELP: Record = { incremental: "Fetch the vendor's latest day or week of bars and append (idempotent).", metadata: "Refresh the instrument master, corporate actions and vendor checkpoints.", backfill: "Full-history download for a ticker range (first letter). Heavy on disk and bandwidth.", fundamentals: "Ingest point-in-time fundamentals for the given tickers (or the tracked list).", derive: "Rebuild derived datasets (5m / 1d) from 1-minute bars.", compaction: "Merge delta files into base files and publish a new dataset version.", quality: "Run the quality checks and record anomalies.", backup: "pg_dump + rsync of the lake metadata to the backup target.", gc: "Garbage-collect retired files and expired exports.", }; export default function IngestionPage() { const { isOwner } = useAuth(); const toast = useToast(); const [status, setStatus] = useState(""); const [kind, setKind] = useState(""); const jobs = useAsync((s) => api.get<{ data: Job[] }>("/admin/jobs", { status, kind, limit: 300 }, s), [status, kind], { enabled: isOwner }); const dead = useAsync((s) => api.get<{ data: Job[] }>("/admin/jobs", { status: "dead", limit: 100 }, s), [], { enabled: isOwner }); const failed = useAsync((s) => api.get<{ data: Job[] }>("/admin/jobs", { status: "failed", limit: 100 }, s), [], { enabled: isOwner }); const datasets = useAsync((s) => api.get<{ data: Dataset[] }>("/catalog/datasets", undefined, s), []); const health = useAsync((s) => api.get("/health", undefined, s), []); const [inspect, setInspect] = useState(null); const [showForm, setShowForm] = useState(false); const reloadAll = () => { jobs.reload(); dead.reload(); failed.reload(); }; const hasLive = (jobs.data?.data ?? []).some((j) => j.status === "queued" || j.status === "running"); useInterval(reloadAll, hasLive ? 5000 : 30000); async function act(j: Job, action: "retry" | "cancel") { try { await api.post(`/admin/jobs/${j.id}/${action}`); toast.success(`Job ${action === "retry" ? "re-queued" : "cancelled"}`, j.id); reloadAll(); } catch (e) { toast.error(e); } } const counts = useMemo(() => { const m: Record = {}; for (const j of jobs.data?.data ?? []) m[j.status] = (m[j.status] ?? 0) + 1; return m; }, [jobs.data]); if (!isOwner) return Ingestion controls require the owner role.; const columns = [ { key: "kind", header: "Kind", render: (j: Job) => {j.kind} }, { key: "dataset_id", header: "Dataset", mono: true, hideSm: true, render: (j: Job) => {j.dataset_id ?? "—"} }, { key: "status", header: "Status", render: (j: Job) => }, { key: "attempts", header: "Att.", num: true, hideSm: true, render: (j: Job) => `${j.attempts ?? 0}/${j.max_attempts ?? "—"}` }, { key: "created_at", header: "Created", hideSm: true, render: (j: Job) => {fmtTs(j.created_at)} }, { key: "duration", header: "Duration", render: (j: Job) => {j.started_at ? fmtDuration(j.started_at, j.finished_at) : "—"} }, { key: "progress", header: "Progress", render: (j: Job) => (j.progress ? : —) }, { key: "last_error", header: "Error", render: (j: Job) => (j.last_error ? {j.last_error} : —) }, { key: "actions", header: "", render: (j: Job) => (
e.stopPropagation()}> {(j.status === "failed" || j.status === "dead" || j.status === "cancelled") && ( )} {j.status === "queued" && ( )}
), }, ]; return ( <> Job queue · {Object.entries(counts).map(([k, v]) => `${k} ${v}`).join(" · ") || "no jobs"} · refresh every {hasLive ? 5 : 30} s} actions={ <> } /> {health.data?.components.ingest_worker && health.data.components.ingest_worker !== "running" && ( The ingest worker is {health.data.components.ingest_worker} — queued jobs will not start until it holds the lake lock. )} {(dead.data?.data.length ?? 0) + (failed.data?.data.length ?? 0) > 0 && ( j.id} mobile="scroll" columns={columns} onRowClick={setInspect} /> )} } > {jobs.error && } j.id} mobile="scroll" columns={columns} onRowClick={setInspect} empty="No jobs match." footer={{fmtInt(jobs.data?.data.length ?? 0)} jobs (max 300)} /> setInspect(null)} wide> {inspect && (
{inspect.dataset_id} priority {inspect.priority} · attempts {inspect.attempts}/{inspect.max_attempts} · by {inspect.created_by ?? "—"}
created
{fmtTs(inspect.created_at)}
started
{fmtTs(inspect.started_at)}
finished
{fmtTs(inspect.finished_at)}
next run
{fmtTs(inspect.next_run_at)}
params
progress
result
{inspect.last_error && (
last_error
{inspect.last_error}
)}
)}
setShowForm(false)}> { setShowForm(false); reloadAll(); }} /> ); } function EnqueueForm({ datasets, lake, onDone }: { datasets: Dataset[]; lake?: HealthReport["components"]["lake"]; onDone: () => void }) { const toast = useToast(); const [kind, setKind] = useState("incremental"); const [datasetId, setDatasetId] = useState("market_bars.stock.1m.raw"); const [period, setPeriod] = useState<"day" | "week">("day"); const [letter, setLetter] = useState("A"); const [tickers, setTickers] = useState(""); const [start, setStart] = useState(daysAgo(30)); const [end, setEnd] = useState(today()); const [priority, setPriority] = useState(50); const [extra, setExtra] = useState("{}"); const [confirm, setConfirm] = useState(false); const [busy, setBusy] = useState(false); const needsDataset = ["incremental", "backfill", "derive", "compaction", "quality"].includes(kind); function buildParams(): Record { let base: Record = {}; try { base = extra.trim() ? (JSON.parse(extra) as Record) : {}; } catch { throw new Error("Extra params must be valid JSON"); } if (kind === "incremental") base = { period, ...base }; if (kind === "backfill") base = { ticker_range: letter.toUpperCase(), ...base }; if (kind === "fundamentals") { const t = splitList(tickers); if (t.length) base = { tickers: t, ...base }; } if (kind === "quality" || kind === "derive") base = { start, end, ...base }; return base; } async function submit(e?: FormEvent) { e?.preventDefault(); if (kind === "backfill" && !confirm) { setConfirm(true); return; } setBusy(true); try { const params = buildParams(); const out = await api.post<{ id: string }>("/admin/jobs", { kind, dataset_id: needsDataset || kind === "fundamentals" ? datasetId || null : null, params, priority }); toast.success("Job enqueued", `${kind} · ${out.id}`); onDone(); } catch (err) { toast.error(err); } finally { setBusy(false); setConfirm(false); } } return (

{KIND_HELP[kind]}

{needsDataset && ( )} {kind === "incremental" && ( )} {kind === "backfill" && ( )} {kind === "fundamentals" && (