spb/social-runtime-crawler

Fri, Sep 11, 2026 7

1. Bounded network collection (streaming/long-poll bodies), observe timeout, recently_failed penalty, .mldignore

   …

   - NetworkObserver: 8 s body timeout, 6 s step collection cap, skip streaming/presence URLs and event-streams, fewer WS frames
   - engine: observation bounded to 45 s; failed actions penalised for 8 steps (no blind retries)
   - .mldignore keeps data/ (browser profiles) out of mld stage; README notes
   
   Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

   Simon-Pierre Boucher committed 12 days ago · 6 files +7,790 −12
   d3b24f2
2. Facebook adapter (Phase 2): url grammar, page hints, GraphQL __typename vocabulary, {text} collapsers, noise filter, consent dismissal

   …

   - observers: __typename → entity type, nested author names, anti-hijack prefixes recognised as JSON/GraphQL
   - browser: interactive login detects the session cookie (no TTY needed), 15-min window
   - api/console: platforms with an adapter are discovered from the connectors registry; Facebook presets
   - tests: facebook.test.ts (24 tests total)
   
   Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

   Simon-Pierre Boucher committed 12 days ago · 12 files +276 −26
   ad4d2d0
3. deploy: access code update

   …

   Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

   Simon-Pierre Boucher committed 12 days ago · 1 file +1 −1
   94bf3a0
4. storage: upsert session on linkJob so goal/mode survive the async SESSION_STARTED insert

   …

   Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

   Simon-Pierre Boucher committed 12 days ago · 1 file
   06a967e
5. README: console, control plane, MacLustr deployment; QA scripts

   …

   Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

   Simon-Pierre Boucher committed 12 days ago · 2 files +26 −3
   bacb91d
6. Research console (Next 16) + control-plane API + mld deployment for www.socialcrawl.co

   …

   - apps/dashboard: SocialCrawl console — observatory, sessions console (decisions, page state,
     entities, runtime APIs, media, world-model graph, live events), entity evidence pages,
     runtime API explorer, connectors, missions launcher; passcode gate + authenticated API proxy.
   - apps/api: node:http control plane (overview, sessions, entities search, schemas, platforms,
     SSE stream, job runner spawning worker processes), token-protected mutations.
   - worker: bounded action execution (60 s), PLAY_VIDEO no longer awaits play(); --job linking.
   - storage: jsonb sanitiser, linkJob; util: unicode thousands separators, stopwords.
   - deploy/social-runtime-crawler.mld.json: PM2 api 8350 + web 8351, MacLustr Tunnel route.
   
   Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

   Simon-Pierre Boucher committed 12 days ago · 46 files +7,607 −105
   d16c8f9
7. Social Runtime Crawler v0.1 — browser-native self-learning social crawler PoC

   …

   Monorepo (pnpm, strict TS via tsx, vitest): persistent Chromium sessions with human login,
   network observer + SchemaProfiler + generic JSON entity miner, DOM mutation observer + in-page
   semantic snapshot, page classifier, network/DOM surface merge with provenance, video detection
   and frame sampling, world model, information-gain heuristic planner (+ optional Anthropic/local
   LLM planner), loop detection, platform-model learning and connector compilation, PostgreSQL +
   JSONL replay storage, research dashboard, YouTube and Reddit adapters.
   
   Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

   Simon-Pierre Boucher committed 12 days ago · 67 files +6,657
   40fba6b