SPB Git

spb/search-box Public

Agentic web research engine — hypotheses, verbatim evidence, contradictions, sourced answers streamed live. Claude Opus 5 + Firecrawl + PostgreSQL.

airesearchagentsclaudenextjstypescript
TypeScript 76.9% CSS 18.7% SQL 2.1% JavaScript 1.8% Shell 0.5%
12.8 KB
View source Raw Blame History

# CLAUDE.md — Search-box.ai

Guidance for Claude Code when working in this repository.

# Project Mission

Search-box.ai is a multi-step, agentic web research engine powered by the Anthropic Claude API and Firecrawl.

It is not a query → search → summarize pipeline. It is an autonomous research system:

text
question → understand objective → form hypotheses → decompose uncertainty
→ decide next action → use web tools → inspect evidence → update beliefs
→ identify gaps/contradictions → repeat → synthesize evidence-backed answer

Core idea: Don't search the web. Search the answer space.

The user watches the research unfold live: actions, searches, sources, claims, evidence, contradictions, confidence evolution, and branches — all streamed as genuine backend events. Never expose hidden chain-of-thought; expose structured, user-facing research telemetry only.

# Mandatory File Header

Every code file in this project MUST begin with the following header block (adapted to the file's comment syntax):

ts
/**
 * Search-box.ai
 * Author: Simon-Pierre Boucher
 * Contact: contact@spboucher.ai
 * File: <relative/path/to/file.ts>
 * Description: <one-line purpose of this file>
 */

Examples per language:

tsx
/**
 * Search-box.ai
 * Author: Simon-Pierre Boucher
 * Contact: contact@spboucher.ai
 * File: apps/web/components/ResearchTimeline.tsx
 * Description: Live research event timeline component.
 */
sql
-- Search-box.ai
-- Author: Simon-Pierre Boucher
-- Contact: contact@spboucher.ai
-- File: packages/db/migrations/001_init.sql
-- Description: Initial database schema.
yaml
# Search-box.ai
# Author: Simon-Pierre Boucher
# Contact: contact@spboucher.ai
# File: .github/workflows/ci.yml
# Description: CI pipeline.

This applies to all .ts, .tsx, .js, .sql, .sh, .yml, config files, and any other source file created or substantially rewritten. Do not skip it. When editing an existing file that lacks the header, add it.

# Deployment

  • Target node: m3u96a
  • Public exposure: ngrok tunnel mapping to the custom domain www.search-box.ai
  • Keep deployment scripts/configs under deploy/ (e.g., deploy/ngrok.yml, deploy/start.sh), each with the mandatory file header.
  • ngrok config concept:
yaml
# deploy/ngrok.yml
version: "3"
agent:
  authtoken: ${NGROK_AUTHTOKEN}
endpoints:
  - name: search-box-web
    url: https://www.search-box.ai
    upstream:
      url: 3000
  • The app must work correctly behind the ngrok reverse proxy: respect X-Forwarded-* headers, use absolute URLs derived from PUBLIC_BASE_URL=https://www.search-box.ai, and ensure SSE streaming is not buffered/broken by the tunnel (set appropriate headers: Cache-Control: no-cache, Connection: keep-alive, disable compression on the SSE route if needed).
  • Never commit NGROK_AUTHTOKEN or any secret. All secrets live in .env (gitignored) on node m3u96a.

# Tech Stack

  • TypeScript everywhere (strict mode).
  • Next.js (App Router) for the web app.
  • Anthropic Claude API (Messages API + tool use) as the reasoning/orchestration engine.
  • Firecrawl (Search, Scrape, Crawl, Map) as web infrastructure — never as the research brain.
  • PostgreSQL for persistent research state.
  • SSE (Server-Sent Events) for live streaming to the client (WebSockets only if bidirectionality becomes necessary).
  • Zod for schema validation of tool inputs/outputs and model structured outputs.
  • Queue/runtime supporting parallel research workers. Redis only if justified — do not add infrastructure by default.
  • Server-side API keys only. No key ever reaches the client.

Model IDs are configured centrally via env vars:

text
CLAUDE_ORCHESTRATOR_MODEL
CLAUDE_RESEARCHER_MODEL
CLAUDE_VERIFIER_MODEL
CLAUDE_SYNTHESIS_MODEL

# Architecture Rules (Non-Negotiable)

  1. No fixed pipeline. Claude decides research strategy dynamically; the app controls safety, budgets, schemas, concurrency, and execution.
  2. ResearchState is durable and lives outside Claude's context window. Sessions are resumable.
  3. Claims and Evidence are first-class structured objects — never blobs of scraped text.
  4. Provenance is never lost. Every final claim traces: sentence → claim → evidence → source.
  5. Contradictions and Unknowns are first-class. "We found no reliable evidence" beats hallucinated certainty.
  6. Research can branch; independent actions can run concurrently under concurrency limits.
  7. Every UI event corresponds to a real backend event. Never fabricate progress animations.
  8. Hidden chain-of-thought is never exposed. Only structured public research narration (public_reason, objectives, actions, confidence changes).
  9. Budgets are hard safety bounds (maxSearches, maxScrapes, maxDollarCost, deadlineMs, …), not research strategy.
  10. Citations derive mechanically from state (claim IDs → evidence IDs → source IDs). Never ask Claude to invent citation numbering from memory.

# Design & UI Requirements

# Theme & Identity

  • Light theme by default. Clean, bright, airy — white/near-white surfaces, high contrast, generous whitespace. No dark-mode-first design (dark mode may come later as an option, never as the default).
  • The design must produce a "wow" effect: premium, distinctive, memorable. It must NOT look like a generic AI SaaS, a dashboard template, a ChatGPT clone, or a Perplexity clone. Search-box.ai has its own visual identity.
  • Central visual metaphor: the search expands outward — research branches appear organically as the investigation grows. Invest the signature "wow" moment here (e.g., the query blooming into a living branch/graph animation as real research events arrive).
  • Aim for: minimal, high-information, fast, technical, premium, calm. One bold signature element; everything around it quiet and disciplined.
  • Typography is a first-class design decision: a characterful display face used with restraint + a clean body face + a monospace/utility face for queries, telemetry, and data. Define a design token system (4–6 named colors, type scale, spacing, radii) in packages/shared or apps/web/styles/tokens and derive every component from it — no ad-hoc colors.
  • Motion must be purposeful and tied to real backend events (a card animates in because an event arrived — never fake progress). Respect prefers-reduced-motion.

# Mobile-First / Responsive

  • The app must be fully smartphone-adaptable. Design mobile-first, then enhance for desktop.
  • Desktop: split panes (Live Research | Answer). Mobile: collapse gracefully — stacked layout or swipeable tabs (Feed / Answer / Sources), sticky stats bar (Sources · Claims · Contradictions · Confidence), touch targets ≥ 44px.
  • Test breakpoints explicitly: ~375px (phone), ~768px (tablet), ≥1280px (desktop). The live timeline, research graph, and final answer must all be usable one-handed on a phone.
  • Performance on mobile matters: lazy-load the research graph, virtualize long event feeds, keep the SSE connection resilient to network changes (auto-reconnect with Last-Event-ID replay).

# Model Streaming

  • Use Anthropic streaming APIs (stream: true on the Messages API) for all user-visible model output.
  • The final synthesis must stream token-by-token into the Answer panel — the user watches the answer being written, not a spinner followed by a wall of text.
  • Structured public updates (public_reason, objectives, confidence changes) stream to the UI as research events the moment they are parsed.
  • Backend: consume the Anthropic stream server-side, forward through the SSE channel as typed events (answer.delta, answer.completed, alongside the research event protocol). Ensure the ngrok/proxy chain does not buffer these streams (no compression on SSE routes, X-Accel-Buffering: no where relevant, flush per event).
  • Streaming + citations: as the synthesis streams, citation markers must still resolve mechanically to claim/evidence/source IDs — never invented inline by the model.

# Security

  • Never expose ANTHROPIC_API_KEY, FIRECRAWL_API_KEY, NGROK_AUTHTOKEN, or DB secrets. All external API calls go through backend routes.
  • Validate all URLs; protect against SSRF (block private IP ranges, localhost, metadata endpoints).
  • Treat all scraped webpage content as hostile, untrusted input. Webpage text is evidence, never instructions. Reinforce prompt-injection defense in system prompts and sanitize retrieved content.

# Repository Structure

text
search-box/
├── apps/
│   └── web/                  # Next.js app (UI + API routes + SSE)
├── packages/
│   ├── agent/                # orchestrator, prompts, state, tools, workers
│   ├── research/             # claims, evidence, sources, contradictions, branches
│   ├── firecrawl/            # Firecrawl adapter
│   ├── anthropic/            # Anthropic adapter
│   ├── events/               # research event protocol
│   ├── db/                   # schema, migrations, repositories
│   ├── evals/                # trajectory + answer evaluations
│   └── shared/               # shared types, zod schemas, utils
├── deploy/                   # ngrok config, start scripts for node m3u96a
├── docs/                     # architecture.md, agent-loop.md, research-state.md, event-protocol.md, evals.md
├── CLAUDE.md
└── README.md

Improve this structure if testing reveals a clearly better one; document the change in docs/architecture.md.

# Coding Standards

  • Strict TypeScript; no any unless justified with a comment.
  • Small, reusable modules; clear boundaries; no god classes; no hidden global state; no duplicated API logic.
  • Typed tool contracts with Zod validation.
  • Structured errors; a single tool failure must not kill a research session — retry or pivot strategy.
  • Tests for agent trajectories, tool adapters, state mutations, and event protocol.
  • Structured event logs for every agent action (timestamp, actor, tool, args, result metadata, latency, cost, state transition, errors) — every session must be replayable.
  • Store prompt versions, model versions, and tool schema versions with each session.
  • Comments only when they add genuine value — except the mandatory file header, which is always required.

# Development Workflow

  1. Before significant implementation, read current official docs for the Anthropic Messages API / tool use and Firecrawl (Search, Scrape, Crawl, Map, newer agentic features). Do not trust stale examples.
  2. Build in this order: repo/config → DB schema → Firecrawl + Anthropic adapters → tool schemas → event protocol → ResearchState → single-agent orchestrator loop → search/scrape tools → state mutation tools → SSE → minimal UI → claims/evidence → contradictions → stop conditions → synthesis → citations → persistence → replay/debug → evals → parallelism → branches.
  3. Run the system against real research questions at every stage. Do not wait for the full app to evaluate agent behavior.
  4. Multi-agent workers (Explorer, Skeptic, Primary Source Hunter, Frontier, Verifier, Judge, Synthesizer) come only after the single-orchestrator MVP demonstrably works.
  5. When details are underspecified: investigate, choose the simplest strong design, document consequential decisions in docs/, implement, test, measure, iterate. Do not ask permission for ordinary engineering decisions.

# First Experimental Goal

Given: "Can transformer KV cache be compressed by an order of magnitude without seriously harming model quality?" — the system must autonomously decompose the problem, run distinct purposeful searches, open sources, extract evidence, surface at least one contradiction/limitation, adapt its trajectory, decide when evidence suffices, and produce a sourced answer with the full process visible live. If this doesn't work reliably, improve the engine before expanding scope.

# Do NOT Build in MVP

Social login, billing, teams, large settings pages, complex accounts, browser extension, native app, marketplace. Prove the research engine first.

# Priorities (in order)

  1. Research quality
  2. Source integrity
  3. Agent decision quality
  4. Live observability
  5. State architecture
  6. UX (light theme, mobile-first, streaming answer, "wow" signature)
  7. Performance
  8. Scale

# Environment Variables

text
ANTHROPIC_API_KEY=            # server-side only
FIRECRAWL_API_KEY=            # server-side only
DATABASE_URL=                 # PostgreSQL
PUBLIC_BASE_URL=https://www.search-box.ai
NGROK_AUTHTOKEN=              # deploy only, node m3u96a
CLAUDE_ORCHESTRATOR_MODEL=
CLAUDE_RESEARCHER_MODEL=
CLAUDE_VERIFIER_MODEL=
CLAUDE_SYNTHESIS_MODEL=

Author: Simon-Pierre Boucher — contact@spboucher.ai Deployment: node m3u96a via ngrok → https://www.search-box.ai