# CLAUDE.md

# Project Name

Social Runtime Crawler

Alternative internal names: SRC · SocialProbe · Social Runtime Mining · Social Browser Intelligence Engine · Living Social Crawler

Repository guide for contributors (humans and agents). Sections 1–87 are the founding specification. Implementation status, commands and layout: see README.md and docs/ARCHITECTURE.md. Repo conventions: pnpm workspaces, strict TypeScript run with tsx (no build step), vitest, @src/* packages, pnpm typecheck && pnpm test before committing. In-page code (evaluated in the browser) must be plain JavaScript strings or .page.js files — TS transpilers inject helpers (__name) that do not exist in the page. Never put credentials in the repo or in .env; authentication = human login into the persistent browser profile (pnpm login <platform>), profiles live in data/browser_profiles/ (git-ignored).

# 1. Mission

Build a new browser-native crawling system specifically designed for modern social-media platforms.

This project must NOT behave like a traditional scraper that primarily performs direct HTTP fetches against static pages.

Instead, it must treat each social platform as a live interactive application runtime.

The crawler must:

launch and maintain authenticated browser sessions;
navigate social-media interfaces normally;
observe what the logged-in account is legitimately allowed to see;
intercept structured information delivered to the browser frontend;
inspect DOM mutations;
inspect network responses;
inspect browser runtime state;
detect dynamically loaded posts, profiles, videos, comments, pages and recommendations;
capture media metadata and optionally public media artifacts;
convert observations into normalized structured events;
build an internal world model of each social platform;
use an AI agent to decide what to explore next;
learn how each platform works during exploration;
progressively compile learned behavior into reusable platform connectors;
detect connector breakage and return to exploration mode automatically.

The long-term objective is to invent a new crawling paradigm: Social Runtime Mining. Instead of downloading documents from the Web, the system observes the execution of social applications and learns how information is surfaced through normal browser interaction.

# 2. Core Principle

Traditional crawler: URL → HTTP GET → HTML → Parse → Follow links → Repeat.

Social Runtime Crawler:

text

Authenticated Browser → Live Social Application → (DOM | Network | Visual)
→ Runtime Observation Layer → Semantic World Model → AI Navigation Agent → Action Selection
→ click / scroll / search / open profile / open post / play video / expand comments ↻

The crawler should continuously ask: What action will reveal the most useful new information? — not What URL should I download next?

# 3. Scope

Initial target platforms: Facebook, Instagram, TikTok, X / Twitter, YouTube, Reddit, LinkedIn, Threads.

Prototype order — Phase 1: YouTube, Reddit · Phase 2: Facebook, Instagram · Phase 3: TikTok, X · Phase 4: LinkedIn, Threads. The architecture must nevertheless be platform-independent.

# 4. Important Boundary

The system must only process information visible to the authenticated account through normal use of the platform.

The system must NOT be designed to: bypass authentication; defeat access controls; obtain private messages; access private profiles not visible to the account; discover hidden API credentials; defeat CAPTCHA systems; evade platform security mechanisms; bypass rate limits through deceptive identity rotation; exploit undocumented vulnerabilities; circumvent paywalls or privacy settings; impersonate users without authorization.

Authentication state should be supplied manually during development. The crawler may observe and structure data already delivered to the browser.

# 5. Primary Use Case

A research account is logged into Facebook. The crawler: open Facebook → observe feed → detect visible post → extract post metadata → detect author/page/profile → detect media → detect interactions → detect comments if opened → follow visible public profile → inspect profile → discover other public posts → return to feed → continue.

The same architecture must work for TikTok For You, Instagram Explore, YouTube recommendations, Reddit feeds, Facebook pages and public profiles, X timelines, LinkedIn public posts.

# 6. Architectural Philosophy

Build this as a set of cooperating systems. Never create a monolithic Playwright script.

Main systems: Browser Runtime · Observation Layer · Entity Extraction Layer · Media Intelligence Layer · World Model · Navigation Agent · Information Gain Engine · Platform Adapter · Connector Learning Engine · Connector Compiler · Persistence Layer · Scheduler · Replay System · Monitoring · Research UI.

# 7. Browser Runtime

Node.js / TypeScript, Playwright, Chromium, Chrome DevTools Protocol. Primary controller: Playwright. Lower-level telemetry: CDP.

Each account uses a persistent browser profile (browser_profiles/<platform>-<alias>/). Never persist passwords manually in project files. Authentication: launch browser → human login → save browser profile → reuse authenticated profile.

# 8. Browser Sessions

interface SocialBrowserSession {
  sessionId: string; platform: Platform; profilePath: string; accountAlias: string;
  start(): Promise<void>; stop(): Promise<void>; navigate(url: string): Promise<void>; getCurrentState(): Promise<PageState>;
}

Track: session_id, platform, account_alias, started_at, current_url, current_entity, navigation_depth, last_action, health.

# 9. Observation Surfaces

SURFACE_1_NETWORK · SURFACE_2_DOM · SURFACE_3_RUNTIME_STATE · SURFACE_4_ACCESSIBILITY · SURFACE_5_VISUAL · SURFACE_6_MEDIA · SURFACE_7_NAVIGATION.

Each observation must contain provenance: { "value": "Example Creator", "source": "network", "confidence": 0.99, "observed_at": "..." }.

# 10. Network Observation Layer

Observe browser-generated XHR, fetch, GraphQL, JSON, HTML fragments, WebSocket frames, SSE, media manifests, video metadata, image resources. Modules: NetworkObserver, ResponseClassifier, JsonDetector, GraphQLDetector, WebSocketObserver, MediaRequestObserver, SchemaProfiler.

Do NOT hardcode endpoints during initial exploration. Fingerprint them: { hostname, method, content_type, response_shape_hash, observed_entity_types }.

# 11. Automatic Response Classification

response → is JSON? → schema inference → contains repeated objects? → candidate entity detection → field analysis → relationship detection → confidence score.

Detect post IDs, profile IDs, usernames, captions, timestamps, engagement counts, comment arrays, cursor values, pagination tokens, media URLs, thumbnail URLs, video manifests, author objects, recommendation metadata. Avoid assumptions about property names; use structure and semantic inference.

# 12. Schema Discovery

SchemaProfiler infers abc → likely_identifier, xyz.foo → likely_display_name, xyz.bar → likely_username from field name, value shape, frequency, co-occurrence, DOM evidence, visual evidence, LLM semantic inference.

# 13. DOM Observation

MutationObserver inside the page. Track elements added/removed, text changes, attributes, lazy-loaded sections, new feed cards, modal opening, comment expansion, video replacement, infinite-scroll additions. Inject a lightweight observer; emit deltas ({ "event": "dom_nodes_added", "count": 14, "region": "feed" }), never serialize the entire DOM continuously.

# 14. Semantic DOM Representation

Never send the full DOM blindly to an LLM. Generate a compact page representation (PAGE / VISIBLE ENTITIES [1] Page… [2] Post… / ACTIONS [A1]…). The AI agent chooses only semantic actions.

# 15. Accessibility Surface

Optional observer using accessible roles and labels (buttons, links, menus, tabs, search fields, video controls, follow buttons, profile links, comment controls). Prefer semantic locators over fragile generated CSS classes.

# 16. Visual Surface

Fallback, not primary. Capture viewport screenshots, content cards, video frames, profile headers, overlays — only when network confidence low, DOM confidence low, content image-based, text embedded in media, UI unknown, or video classification required.

# 17. Video Intelligence

src/media/video/: VideoDetector, VideoMetadataExtractor, VideoFrameSampler, SubtitleExtractor, TranscriptResolver, AudioMetadataExtractor, VideoFingerprint, MediaDeduplicator. Collect platform video ID, author, caption, description, hashtags, duration, thumbnail, public engagement metrics, creation timestamp, visible music/audio name, subtitles, resolution, delivery metadata.

# 18. Video Frame Sampling

Do NOT download every video. video detected → metadata → relevance estimate → if relevant: capture frames (start, 25 %, 50 %, 75 %, end; adaptive: scene change) → vision analysis → semantic representation. Store perceptual hashes for dedup.

# 19. Video Semantic Record

Normalized { media_type, platform, platform_media_id, author_entity_id, caption, topics[], entities[], language, transcript_available, visual_summary, engagement{views,likes,comments} }.

# 20. Public Profile Modeling

Profiles become canonical entities { entity_type: "person", canonical_id, display_name, platform_profiles{}, professional_context, public_bio, topics[], sources[] }. Do not infer sensitive attributes (religion, medical conditions, sexual orientation, private family information, home address, political ideology inferred from behavior) unless scope explicitly changes with a justified basis.

# 21. Identity Resolution

IdentityResolver signals: exact name, username similarity, verified links, official website, cross-linked accounts, same organization, bio similarity, profile photo similarity where appropriate, public external links. Never merge two people solely because names match; every merge requires confidence and evidence.

# 22. Entity Types

Person, Organization, Page, Profile, Channel, Account, Post, Comment, Video, Image, Topic, Hashtag, URL, Event, Location, Product, OrganizationRole. Relationships: AUTHORED, MENTIONED, REPLIED_TO, POSTED_BY, BELONGS_TO, LINKS_TO, FEATURES, HAS_PROFILE, REPRESENTS, DISCOVERED_FROM.

Graph-like in-memory representation of the current platform (Profile A —AUTHORED→ Post 1 …). This graph guides navigation.

The LLM must NOT manipulate raw selectors. It receives { goal, current_state, visible_entities[], available_actions[] } and answers { action, expected_information_gain, reason }.

# 25. Action Vocabulary

OPEN_ENTITY, OPEN_POST, OPEN_PROFILE, OPEN_PAGE, OPEN_CHANNEL, OPEN_VIDEO, OPEN_COMMENTS, SCROLL_DOWN, SCROLL_UP, SEARCH, FILTER, PLAY_VIDEO, PAUSE_VIDEO, EXPAND, COLLAPSE, BACK, FORWARD, RETURN_TO_FEED, WAIT_FOR_CONTENT, END_SESSION. Do not expose arbitrary browser JavaScript to the planner.

# 26. Information Gain Engine

information_gain = novelty × relevance × expected_entity_yield × confidence × source_quality ÷ exploration_cost, with penalties: already_seen, duplicate_content, navigation_loop, low_quality_source, low_relevance.

# 27. Novelty Measurement

Embeddings: novelty ≈ 1 − max_similarity(candidate, existing dataset). Use a local embedding model whenever possible.

# 28. Agent Modes

OBSERVE (feed only; study recommendation exposure) · RESEARCH (search + navigate public entities) · PROFILE (one public person) · TOPIC (a topic → people, organizations, posts, videos, channels, hashtags) · PLATFORM LEARNING (understand UI, runtime, entity structures, schemas, navigation).

# 29. Connector Learning

Do not immediately hardcode a Facebook connector; learn first. Store platform_model/{page_types, entity_patterns, response_patterns, action_patterns, navigation_graph, selectors, media_patterns}.json.

# 30. Action → Observation Learning

Every action records consequences (before/after: visible posts, network requests, new entities). Infer over time: scroll_down → usually FeedResponseType3 → ~8 new posts → exposes cursor.

# 31. Automatic Connector Compilation

Once confidence is high: learn platform → compile connector (connectors/<platform>/manifest, page_classifier, network_patterns, entities, navigation, media, parser, regression_tests/). The connector is learned state, not manually written truth.

# 32. Self-Healing

expected entities = 20, observed = 0 → CONNECTOR_DEGRADED → exploration mode → re-inspect DOM/network → re-learn → update connector → regression tests → resume.

{ event_id, event_type, platform, session_id, timestamp, entity{type, platform_id, canonical_id}, provenance[{surface, confidence}], discovered_via{action} }.

# 34. Event Types

PAGE_OPENED · ENTITY_DISCOVERED/OBSERVED/UPDATED · POST_/COMMENT_/PROFILE_DISCOVERED · MEDIA_/VIDEO_/IMAGE_DISCOVERED · NETWORK_RESPONSE_OBSERVED · NETWORK_SCHEMA_DISCOVERED · DOM_CHANGED · ACTION_EXECUTED/FAILED · NAVIGATION_COMPLETED · CONNECTOR_PATTERN_LEARNED/DEGRADED/REPAIRED.

# 35. Storage

Prototype: PostgreSQL, Redis, local filesystem / S3-compatible object storage. Optional later: ClickHouse, OpenSearch, Qdrant, Neo4j. Do not add all databases prematurely.

# 36. PostgreSQL Core Tables

sessions, platform_accounts, entities, entity_aliases, observations, posts, media, videos, relationships, actions, network_responses, schema_patterns, connector_versions, crawl_jobs.

# 37. Raw vs Normalized Data

RAW OBSERVATION → NORMALIZATION → CANONICAL ENTITY. Raw observations must never be overwritten.

# 38. Evidence Model

Every canonical field needs provenance: { field, value, evidence[{observation_id, source}] }.

# 39. Deduplication

Same platform ID, canonical URL, media fingerprint, normalized text, high embedding similarity, same author+timestamp+content. Never deduplicate people using fuzzy names alone.

# 40. Crawl Budget

budget: { max_minutes, max_actions, max_profiles, max_posts, max_videos, max_depth } plus max_video_processing_seconds, max_storage_mb, max_llm_tokens.

Maintain visited_entities, visited_urls, recent_actions, navigation_path, page_fingerprints. Detect loops (A → B → A → B) and break automatically.

# 42. Platform Boundary

stay_on_platform = true: external links may be recorded but not followed automatically.

# 43. Search-Based Discovery

Research mode supports native platform search: find search field → submit query → classify result tabs → inspect people/pages/posts/videos → select relevant entities.

# 44–45. Feed-Based Discovery & Recommendation Dataset

Preserve ranking: feed_session_id, feed_position, content_id, timestamp_seen, time_visible, interaction_performed, recommendation_context (is_followed_account, is_sponsored, time_visible_ms).

# 46. Interaction Policy

Default read-only. Allowed: scroll, search, open, expand, play video, navigate. Never automatically like, follow, comment, share, message, react, subscribe unless explicitly enabled for a controlled experiment.

# 47. Media Capture Strategy

LEVEL 0 metadata · 1 + thumbnail · 2 + selected frames (default) · 3 + transcript/subtitles · 4 full artifact only when required and permitted.

# 48–50. Content Classification, Local AI, Model Hierarchy

Local LLM/embeddings for topic, language, mentions, names, locations, category, summary, tags — never invent facts. Pipeline: deterministic parser → local embeddings → local classifier → larger LLM only where needed. Tiers: 1 rules/parsers · 2 embeddings · 3 small local LLM · 4 large reasoning model. Escalate only when needed.

# 51. Initial Prototype Goal

YouTube or Reddit first: start authenticated browser → search topic → detect result entities → navigate results → intercept network → compare network vs DOM extraction → capture posts/videos → normalize events → store in PostgreSQL → dashboard.

# 52. Facebook Experimental Prototype

Open authenticated Facebook → one known public page → observe → identify posts → intercept runtime traffic → open one post → expand visible comments → identify public profile/page links → open one → capture public info → return. Success: no Facebook-specific external API, no manual data entry, browser stays authenticated, all entities have provenance, actions recorded, video objects identified, schemas discovered automatically.

# 53. Debug UI

Panels: Browser Preview, Current Page State, Visible Entities, Available Actions, Agent Decision, Network Events, Detected Schemas, World Model, Media Queue, Recent Observations, Connector Confidence.

# 54–55. Replay & Explainability

Every session replayable from recorded observations (actions, URL changes, network metadata, DOM observations, screenshots, decisions). Each AI action stores goal, chosen action, expected gain, relevance, novelty, concise reason — no hidden chain-of-thought.

# 56. Platform Adapter Interface

detectPageType, extractVisibleEntities, getAvailableActions, resolveEntityTarget, detectMedia, detectFeedItems, normalizeObservation.

# 57. Generic Engine vs Platform Logic

Generic: browser lifecycle, network capture, DOM observation, event bus, planning, information gain, storage, media pipeline, identity resolution. Platform-specific: page recognition, semantic labels, feed identification, known entity classes, navigation hints, normalization quirks.

# 58–60. Page Classification, Confidence, Unknown States

Types: HOME_FEED, SEARCH_RESULTS, PROFILE, PUBLIC_PAGE, POST_DETAIL, VIDEO_DETAIL, CHANNEL, GROUP, COMMENT_VIEW, UNKNOWN. Signals: URL, visible text, ARIA landmarks, DOM structure, network schemas, adapter hints. Every inference carries confidence; below threshold → stronger classifier. Unknown pages are expected: capture semantic DOM + network summary → classify → generic actions → learn.

# 61–63. Connector Knowledge Base, Shared Learning, Workers

Persist learned behaviour ({pattern, platform, schema_hash, likely_entity_type, confidence, observed_count}). Workers share patterns through a Connector Knowledge Service → versioned platform model. Worker = Browser + Playwright + CDP + Observer + Media Processor + Action Executor; control plane = Scheduler, Queue, DB, Connector Knowledge, LLM Router, Dashboard.

# 64–66. Cluster Deployment, Isolation, Crash Recovery

One Mac node per platform worker (MacLustr), central PostgreSQL/Redis/object storage/API/dashboard. One browser profile per account; separate processes. Recover from browser/tab crash, timeouts, missing selectors, session expiration (pause job, AUTH_REQUIRED, manual re-auth — never bypass), unexpected modals, redesigns.

# 67–69. Observability, Quality Metrics, Experimentation

Track actions/min, entities/min, posts/min, videos/min, responses/min, duplicate ratio, LLM calls, average gain, connector confidence, CPU/RAM, storage, errors. KPIs: entity/field precision, recall on visible page, navigation success, duplicate rate, connector stability, media detection accuracy, cost per 1 000 entities, entities per browser-hour. Compare DOM only / Network only / DOM+Network / +Agent / +Vision.

# 70–74. Research Question & Innovations

Can this outperform traditional social crawling (less connector code, automatic adaptation, dynamic content, recommendation context, media, learned frontend APIs, surviving UI changes)? Innovations: Runtime API Discovery (action → response → entity), Social Runtime Graph (UI states, actions, schemas), Information-Gain Navigation, Connector Compiler (social-runtime learn facebook → "Platform learned … Confidence: 94 %"; social-runtime crawl facebook --goal "Quebec public personalities").

# 75. Development Phases

0 repo · 1 persistent browser · 2 network observer · 3 DOM observer · 4 event normalization · 5 entity extraction · 6 semantic actions · 7 AI planner · 8 information gain · 9 media/video · 10 platform learning · 11 connector compilation · 12 self-healing · 13 distributed workers.

# 76–77. Repository Structure & Stack

apps/{api,dashboard,worker}, packages/{browser,observers,agent,entities,media,connectors,platform-model,storage,events,shared}, connectors/<platform>/, data/, scripts/, docs/, docker/. TypeScript, Node.js, Playwright, Chromium, CDP, PostgreSQL, Redis, React/Next.js dashboard, Docker where appropriate; optional Qdrant, ClickHouse, Neo4j, MinIO. Do not over-engineer the prototype.

# 78–81. Coding Principles & Testing

Strict TypeScript, small modules, typed events, DI where useful, structured logging, deterministic parsers before LLMs, clear adapters, unit + integration tests, record/replay fixtures. Avoid one giant platform script, hundreds of static CSS selectors, massive switch statements, platform logic inside the core, LLM-controlled arbitrary JS. Fixture-driven tests for schemas, DOM fragments, normalization, identity resolution, action selection, media detection, page classification; browser integration tests for profile reuse, navigation, scroll, search, open entity, interception, mutation capture, media detection.

# 82–86. Data Review, Vision, Success

Dashboard lets a human inspect raw observation, normalized value, canonical entity, evidence, surface, confidence and correct mappings (feeding learning). Vision: "Find public Quebec personalities discussing AI" → system understands platforms, navigates, observes runtime, discovers profiles, captures posts/videos, extracts entities, deduplicates identities, builds graph, returns dataset — without a hand-maintained scraper per page.

First milestone succeeds when: one human login is reused; navigation by semantic actions; dynamic content observed; posts/profiles/media recognized; useful network payloads identified; network vs DOM evidence compared; video metadata captured; normalized events stored; next action chosen; loops avoided; at least one reusable platform pattern learned automatically.

# 87. Final Engineering Rule

When implementing any feature, always ask: Are we teaching the crawler how the social application works, or merely hardcoding another scraper?

Prefer learning · observation · semantic actions · structured frontend data legitimately delivered to the browser · deterministic extraction over LLM guessing · provenance over assumptions · reusable platform knowledge over one-off automation.

The end goal is not to automate Chrome. It is to build an autonomous system capable of understanding and mining the observable runtime of social applications.