Repository Public
1# Multi-agent orchestration frameworks: comparison and design recommendations (research date 2026-11-10)23**Evidence labels.** [V] = read in a primary source this session. [S] = seen only in secondary sources or search snippets this session. [M] = from my prior knowledge, not re-verified. [I] = my inference. Several 2026 facts post-date my training cutoff and rest only on the pages fetched.45**Coverage gaps.**6- The LangGraph durable-execution page redirected to the persistence overview; durability modes come from the Checkpointers page instead.7- The ADK multi-agents page is now a redirect and was not read.8- Not re-verified in primary docs: Pydantic AI delegation, Mastra agent networks, Letta sleep-time agents, Cline checkpoints, AutoGen Core runtime, SWE-agent ACI numbers, Aider architect benchmark, and OpenAI tracing/guardrail internals.9- For MAST I read only the arXiv abstract and project site, not the full paper.10- For A2A I read only the first ~40% of the spec.1112## 1. Comparison table1314| Framework | Core model | State / persistence | Resume semantics | Delegation pattern | HITL | Notable weakness |15|---|---|---|---|---|---|---|16| **LangGraph** | State graph run in Pregel-style super-steps; reducers merge channel writes; `Send` fan-out, `Command` routing [V] | Checkpointer per thread (Postgres/SQLite/memory) plus cross-thread Store; subgraphs get their own namespace [V] | Checkpoint per super-step; durability `exit`/`async`/`sync`; successful siblings' pending writes kept; replay re-runs nodes after the checkpoint, including LLM calls [V] | Supervisor/swarm prebuilts, subgraphs, shared state [M] | `interrupt()` + `Command(resume)`; node restarts from its top on resume [V] | Idempotency is the developer's job; no intra-node granularity; default serializer limits, checkpoint growth [V] |17| **Deep Agents** (LangChain) | Harness on LangGraph: planning, virtual filesystem, subagents, long-term memory [V] | LangGraph checkpoints plus files [V] | Inherits LangGraph [I] | `task` tool; subagent "sees only the task description"; parent gets "only the final result"; parallel via several calls per turn [V] | Inherits interrupts [I] | Sync subagents block the supervisor; async is a separate feature [V] |18| **AutoGen (0.4+)** | Actor-style, event-driven core; AgentChat teams [M] | `Team` has no built-in checkpointing [V, MS migration guide] | DIY | GroupChat speaker selection; Magentic-One ledgers [V] | Human input modes [S] | Maintenance mode since MAF GA [S] |19| **AG2** | "Classic" (ConversableAgent/GroupChat, maintenance) vs AG2 1.0 (2026-07-27, hub + typed channels) [S] | Not verified | Not verified | Conversation / group chat | Not verified | Two incompatible lines [S] |20| **Microsoft Agent Framework 1.0** (GA April 2026: 2 or 3 April depending on source) | Typed graph workflows (executors + edges) in supersteps; five orchestrations [V/S] | Memory / file / Cosmos checkpoint storage; captures executor state, pending messages, pending requests, shared state [V] | Checkpoint at end of each superstep; rehydration needs identical topology and stable executor IDs [V] | Sequential, concurrent, handoff, group chat, Magentic [V] | Request/response events are checkpointed [V] | Pickle-based Python checkpoints (restricted unpickler; storage is a trust boundary); custom executors must write save/restore hooks [V] |21| **CrewAI** | Crews (role agents, sequential/hierarchical) vs Flows (event-driven, typed state) [V/S] | `@persist`, SQLite by default [V] | Flow state restored; human feedback pause persisted; docs tell you to make the resume handler idempotent [V] | Manager LLM delegates via a "Delegate work to coworker" tool [S] | `HumanFeedbackPending` + `resume()` [V] | Hierarchical delegation bugs: manager does all the work, "coworker not found" loops (issue #4783, forum) [V] |22| **OpenAI Agents SDK** | Runner loop: LLM, tools, handoff, repeat [V] | Sessions or server-managed conversation IDs; `RunState` [V] | Serialize `RunState` at approval pauses; crash durability via Temporal/Restate/DBOS integrations [V] | Handoff = tool `transfer_to_x`, receiver sees "the entire previous conversation history" unless `input_filter`; `Agent.as_tool` for nested specialists [V] | `needs_approval`, interruptions, approve/reject [V] | No native crash checkpointing; `max_turns` is the main loop guard; history nesting is opt-in beta [V] |23| **Claude Agent SDK** | Single agent loop plus `Agent` tool subagents; hooks; `Workflow` tool for dozens to hundreds of agents [V] | JSONL transcript per session; subagent transcripts separate; `SessionStore` for cross-host [V] | resume / continue / fork restore the conversation, "not the filesystem" (file checkpointing is separate) [V] | Fresh context; the only parent-to-child channel is the prompt string; only the final message returns; forks copy history; defaults: depth 3, 20 concurrent, optional `maxBudgetUsd` [V] | Permission callbacks; hooks return allow/deny/ask/defer; deny wins [V] | Parent may re-summarize the report; hook callback failures fail open unless `onFailure: "block"` [V] |24| **Google ADK (2.0)** | Agent tree plus graph Workflows (nodes, routers, JoinNode, dynamic `ctx.run_node`) [V, Google Cloud blog 2026-09-30] | Session event log; per-node checkpoint with a resumability flag [S, community call] | Parent node reruns on resume; completed `ctx.run_node` calls "return their recorded outputs from session history" [V] | Coordinator/sub-agents, workflow agents, A2A [M/S] | `RequestInput`, `resume_inputs` [V] | Side effects outside child nodes re-execute; crash recovery not described in the blog [V] |25| **Pydantic AI** | Typed agent loop plus pydantic-graph [M] | Delegated to durable engines | Temporal, DBOS, Prefect, Restate, Lambda and others [V] | Agent-as-tool, programmatic handoff [M] | Deferred tools / approvals [M] | Durability needs an external engine [V] |26| **smolagents** | Code-as-action (`CodeAgent`), ReAct steps [S] | In-memory step log [M] | None native [M] | `managed_agents` surfaced as tools [S] | Step callbacks [M] | Snippet sandboxes "don't support multi-agents" [V] |27| **Mastra** | TypeScript workflows plus agents [V] | Snapshot to storage (LibSQL default) [V, legacy docs] | `suspend()` / `resume()` from snapshot; maps to A2A `input-required` [V] | Agent networks / supervisor [not verified] | Suspend/resume, tool approval [V] | Not assessed |28| **Letta** | Agent server; agents are persistent DB entities [V] | All messages, memory blocks, tool calls in a database; shared blocks [V] | State is always persisted; runs and steps inspectable [V/S] | Async/sync agent messaging tools; shared memory blocks (blackboard) [S] | Tool approvals [M] | Self-editing shared memory can drift [I] |29| **OpenHands (V1 SDK)** | Event-sourced loop: actions and observations as immutable events [V, arXiv 2511.03690] | Append-only event log; immutable agent config [V] | "Deterministic replay"; one step at a time; pause/resume [V] | Delegation tool; skills (ex-microagents) [M] | Security analyzer can block actions [V] | External sandbox state is not in the log [I] |30| **SWE-agent / mini-swe-agent** | ACI tools vs bash-only loop, linear history, stateless `subprocess.run` [V] | Trajectory file | Re-run | None | None / confirm mode | Single agent by design; README claims >74% SWE-bench Verified [V] |31| **Goose** | MCP-native single agent plus recipes/subrecipes and subagents [V] | Local sessions [M] | Session resume [M] | Subagents with restricted tools; parallel subrecipes [V] | Permission modes [M] | Shallow orchestration [I] |32| **Aider** | Single loop; repo map; edit formats; architect/editor split [V] | Git commits as checkpoints [M] | Git undo | Architect writes plain-text instructions, editor emits edits [V] | Human in every turn | Two-model relay can mis-apply edits (Cognition's critique) [V] |33| **Cline / Roo** | IDE loop; Roo Orchestrator "boomerang" subtasks [V] | Task history; workspace checkpoints [M] | Restore checkpoint [M] | `new_task(message, mode)` in "complete isolation"; `attempt_completion(result)` summary back [V] | Approval for subtask creation and completion by default [V] | Lossy summaries [I] |34| **Devin (Cognition)** | Single-threaded writer; reviewer, advisor, manager + children [V] | Proprietary | Unknown | "Map-reduce-and-manage"; children tracked through an internal MCP [V] | PR review | Children don't report upward by default; false shared-state assumptions [V] |3536**Protocols.**37- **MCP.** The 2025-11-25 revision added experimental tasks, URL-mode elicitation and Client ID Metadata Documents [S]. The 2026-07-28 revision (release candidate blog dated 2026-05-21) is stateless [V]:38 - no `initialize` handshake or `Mcp-Session-Id`; `server/discover` added;39 - Multi Round-Trip Requests: the server returns `InputRequiredResult` plus `requestState`, and the client re-issues the call;40 - Tasks moved to an extension (`tasks/get`, `tasks/update`, `tasks/cancel`; `tasks/list` removed);41 - Roots, Sampling and Logging deprecated;42 - W3C Trace Context carried in `_meta`.43- **A2A 1.0** [V].44 - Objects: AgentCard (signable), Task, Message, Part, Artifact, with `contextId` grouping.45 - States: submitted, working, input-required, auth-required, completed, failed, canceled, rejected.46 - Bindings: JSON-RPC, gRPC, HTTP+JSON; streaming plus webhook push.47 - 1.0 breaks 0.3 (enum and method renames) [S]. Governance wording differs between sources (Linux Foundation vs Agentic AI Foundation); I did not resolve it.4849## 2. Synthesis by theme5051### State and checkpointing52Three resume designs exist, and they differ in what re-executes.53541. **Snapshot per super-step** (LangGraph, MAF). Resume is cheap, but anything inside a node re-runs. LangGraph says so directly: replayed nodes' LLM and API calls "are always re-triggered", and on interrupt "the runtime restarts the entire node from the beginning". Its best detail is the pending-writes rule: successful parallel siblings are not re-run. MAF adds an operational trap, since resume fails if executor IDs or topology change.552. **Event sourcing / journal** (OpenHands V1, ADK dynamic nodes, the Temporal-style engines behind Pydantic AI and the OpenAI SDK). Completed steps return recorded outputs. This is the only family that gives per-tool-call granularity. [I]563. **Transcript persistence** (Claude Agent SDK, Letta). The conversation is durable; the world is not. Claude's docs separate session resume from file checkpointing, and for cross-host work recommend capturing results as application state rather than shipping transcripts.5758No framework makes side effects idempotent for you. LangGraph, CrewAI and ADK all push that onto the developer.5960### Delegation contracts61The designs converge on: fresh child context, an explicit brief, and a condensed return (Claude SDK, Deep Agents, Roo, Anthropic's research system).62- Anthropic's brief has four parts: objective, output format, tool/source guidance, task boundaries. Vague briefs caused duplicated work and gaps.63- Full-history handoff (the OpenAI default) preserves context but transfers control and bloats tokens.64- Forking the parent context (Claude forks, Cognition's "smart friend") is the middle option; Cognition calls it a reasonable 80/20.65- Returning references to artifacts instead of prose reduces the "game of telephone" (Anthropic).6667### Parallelism68- Fan-out is easy everywhere (`Send`, multiple tool calls per turn, ParallelAgent/JoinNode, concurrent orchestration). Fan-in is where it breaks.69- Cognition's rule after ten months: multi-agent "work[s] best today when writes stay single-threaded and the additional agents contribute intelligence rather than actions".70- Anthropic says most coding tasks are a poor fit, and that synchronous subagents become a bottleneck.71- Hard caps exist only in the Claude SDK (depth 3, 20 concurrent, USD budget) and through `max_turns` / `max_iterations` elsewhere.7273### Supervision and verification74- Magentic-One's dual ledger is the most explicit supervisor state: a task ledger (facts, guesses, plan) and a progress ledger with stall detection that triggers replanning.75- Independent verification matters more than more workers:76 - MAST attributes 21.3% of failures to task verification and 41.8% to specification/system design.77 - Google/MIT measured 17.2x error amplification for independent agents versus 4.4x with a central orchestrator.78 - Anthropic's long-running harness found a separately tuned skeptical evaluator "far more practical" than a self-critical generator.79 - Cognition's clean-context reviewer finds on average 2 bugs per PR.80- Completion needs to be defined up front: Anthropic used a feature-list JSON where agents may only flip `passes`, plus sprint contracts.8182### Interrupts83Two families:84- **In-loop approval gates**: OpenAI `needs_approval`, Claude hooks with `defer`.85- **Durable suspend**: LangGraph `interrupt`, Mastra `suspend`, CrewAI `HumanFeedbackPending`, MAF request events, A2A `input-required`, MCP MRTR.8687Protocols are moving toward state carried in the payload (`requestState`) rather than a held connection. LangGraph's index-based interrupt matching and re-execution of code before the interrupt are documented hazards.8889### Observability and evaluation90- OpenTelemetry is the common base (MAF, Goose; MCP now documents trace context in `_meta`).91- Claude tags every message with `parent_tool_use_id` / `agent_id` and exposes SubagentStart/Stop hooks.92- Anthropic's evaluation practice: start with about 20 real queries, a single-call LLM judge scoring 0–1 with pass/fail, judge the end state rather than the path, and keep human review.9394### Failure evidence95- **MAST** (arXiv 2503.13657, v3 2025-10-26): 14 failure modes, 1,600+ traces, 7 frameworks, kappa 0.88. Shares: specification 41.77%, inter-agent misalignment 36.94%, verification 21.30% [V, project site]. Step repetition at 17.14% is the largest single mode [S].96- **Anthropic** (2025-06-13): multi-agent beat single-agent by 90.2% on an internal research eval; agents use about 4x chat tokens and multi-agent about 15x; token usage explains about 80% of BrowseComp variance; early versions spawned up to 50 subagents for simple queries.97- **Google/MIT** (arXiv 2512.08296): +80.9% on parallelizable tasks, −39% to −70% on sequential planning; negative returns once the single-agent baseline exceeds ~45%; 58–515% token overhead.98- **Tran & Kiela** (arXiv 2604.02460): under equal thinking-token budgets, single agents match or beat five multi-agent architectures on multi-hop reasoning.99- **Anthropic harness** (2026-03-24): solo run 20 min / $9 with a broken core feature, versus full harness 6 h / $200; scaffolding was removed as models improved.100101Taken together [I]: multi-agent pays for breadth-first, read-heavy, decomposable work and for independent verification. It hurts sequential reasoning and parallel writes, and part of its measured gain is simply more tokens spent.102103## 3. Recommendations for a from-scratch framework1041051. **Use an append-only event journal as the source of truth**, not state snapshots. Record every model response and tool result with a stable step ID; derive snapshots as a cache. This gives resume, fork, time travel and audit from one mechanism (OpenHands V1, ADK recorded outputs).1062. **Replay by returning recorded results, never by re-calling the model.** Avoid node-level re-execution, the hazard LangGraph documents.1073. **Make side effects a first-class type.** Each tool declares `read`, `idempotent-write` or `non-idempotent-write`. Non-idempotent writes get an intent record before execution and an idempotency key. After a crash with an intent but no result, reconcile or ask a human; never blindly retry.1084. **Commit before proceeding by default** (LangGraph `sync`), with a per-run opt-down. Keep completed siblings' results when one branch of a fan-out fails.1095. **Keep one writer per workspace.** Parallel agents read, search, review and advise. Parallel writers only on isolated worktrees with a single merge step owned by the parent.1106. **Use a typed delegation contract** instead of a free-text prompt: objective, context references (artifact IDs), constraints/boundaries, output schema, tool allowlist, budget (tokens, USD, turns, wall-clock), done criteria. Reject spawns that omit a done criterion.1117. **Return a typed result envelope**: status (`done` / `partial` / `blocked` / `failed`), a schema-validated payload, artifact references, an evidence list, and open questions. A child killed by an API error or budget must surface as `failed` or `partial`, never as a result (Claude's docs make this distinction).1128. **Offer three context modes explicitly**: fresh (default), fork (copy the parent journal), and artifact-only. Make the choice visible in traces. Do not default to full-history handoff.1139. **Keep supervisor state in a structured ledger**: plan items with owner, status and evidence, plus a facts list, persisted in the journal (Magentic-One, Anthropic feature list). Workers may only flip their own items, and only with attached evidence.11410. **Detect stalls and loops deterministically**: repeated identical tool calls, no ledger change in N steps, turn and budget ceilings. On trigger, replan once, then escalate. Keep detectors high-precision, as SWE-agent's guardrails were [S].11511. **Enforce budgets hierarchically in the runtime**: depth, concurrency, tokens and USD per subtree, with children's spend counting against the parent. Do not rely on prompt instructions.11612. **Build verification as a separate role with clean context.** The verifier sees the artifact and done criteria, not the worker's reasoning. Prefer executable checks (tests, schema, browser run) over LLM opinion.11713. **Model interrupts as durable suspended states.** The pending request (schema, prompt, resume token) lives in the journal, the process can exit, and resume is a new invocation carrying the answer. Mirror A2A `input-required` / `auth-required` and MCP MRTR so external agents and tools map directly.11814. **Use a policy hook pipeline around every tool call** (pre, post, subagent start/stop, pre-compaction, stop) with deny-wins merging and fail-closed for security hooks. Claude's default is fail-open; do not copy that for guards.11915. **Treat compaction as an explicit journal event.** Keep the full history retrievable (Letta) and store plan and decisions outside the context window so they survive compaction. Make reset-versus-compact a configurable policy and re-test it per model generation.12016. **Emit OpenTelemetry spans from day one**, with run, agent, parent and step IDs and per-span token cost. Add a replay-from-step debugger and an evaluation harness that runs the single-agent baseline at an equal token budget on every benchmark.12117. **Speak MCP for tools and A2A at the boundary only.** Internally use your own typed envelopes; adopt the A2A task state machine and artifact concept for long-running external work.122123**What not to build.**124- Free-form group chat or peer swarms with LLM speaker selection. Cognition calls unstructured swarms "mostly a distraction"; MAST shows misalignment and step repetition there.125- A role-play "crew" abstraction with an auto-manager; CrewAI's hierarchical mode is the cautionary example.126- A graph DSL as the only programming model. Plain code with journaled steps (the Temporal/DBOS style) is more flexible. [I]127- Pickle-based checkpoints.128- Your own distributed actor runtime before one process plus Postgres is saturated.129- Recursive unbounded spawning.130- Scaffolding you cannot delete: every harness component encodes an assumption about a model weakness that goes stale.131132## 4. Sources133134**LangChain**135- LangGraph Checkpointers — https://docs.langchain.com/oss/python/langgraph/checkpointers — undated, fetched 2026-11-10136- LangGraph Persistence — https://docs.langchain.com/oss/python/langgraph/persistence — undated137- LangGraph Interrupts — https://docs.langchain.com/oss/python/langgraph/interrupts — undated138- S. Runkle, durability modes announcement — https://x.com/sydneyrunkle/status/1950934319888785498 — 2025-07-31139- Deep Agents subagents — https://docs.langchain.com/oss/python/deepagents/subagents — undated140- Deep Agents overview — https://docs.langchain.com/oss/python/deepagents/overview — undated141142**Microsoft / AutoGen / AG2**143- Agent Framework Workflows: Checkpoints — https://learn.microsoft.com/en-us/agent-framework/workflows/checkpoints — ms.date 2026-09-16, updated 2026-11-01144- AutoGen to Agent Framework migration guide — https://learn.microsoft.com/en-us/agent-framework/migration-guide/from-autogen — snippet only145- Microsoft Agent Framework Version 1.0 — https://devblogs.microsoft.com/agent-framework/microsoft-agent-framework-version-1-0 — April 2026, snippet only146- Magentic-One — https://www.microsoft.com/en-us/research/articles/magentic-one-a-generalist-multi-agent-system-for-solving-complex-tasks — Nov 2024; AutoGen docs https://microsoft.github.io/autogen/stable//user-guide/agentchat-user-guide/magentic-one.html147- AG2 status (secondary) — https://www.swarms.ai/blog/autogen-alternatives — Oct 2026; https://blckalpaca.at/en/knowledge-base/ai-agents/ai-agent-frameworks-comparison/autogen-vs-ag2148149**CrewAI**150- Flows — https://docs.crewai.com/v1.12.0/en/concepts/flows151- Human feedback in Flows — https://docs.crewai.com/edge/en/learn/human-feedback-in-flows152- Issue #4783 — https://github.com/crewAIInc/crewAI/issues/4783153- Forum threads — https://community.crewai.com/t/hierarchical-process-manager-cannot-delegate-issue-coworker-not-found/6302 ; https://community.crewai.com/t/does-hierarchical-process-even-work-your-experience-is-highly-appreciated/2690154155**OpenAI Agents SDK**156- Handoffs — https://openai.github.io/openai-agents-python/handoffs/157- Running agents — https://openai.github.io/openai-agents-python/running_agents/158- Human-in-the-loop — https://openai.github.io/openai-agents-python/human_in_the_loop (snippet)159160**Claude Agent SDK**161- Subagents — https://code.claude.com/docs/en/agent-sdk/subagents162- Sessions — https://code.claude.com/docs/en/agent-sdk/sessions163- Hooks — https://code.claude.com/docs/en/agent-sdk/hooks164165**Google ADK**166- Graph Workflows in ADK — https://cloud.google.com/blog/topics/developers-practitioners/graph-workflows-in-adk-everything-you-need-to-know — 2026-09-30167- ADK Community Call — https://www.youtube.com/watch?v=bPngDY7EuOQ — Mar 2026, transcript snippet168- Long-running agents with ADK — https://developers.googleblog.com/build-long-running-ai-agents-that-pause-resume-and-never-lose-context-with-adk — 2026-05-12, not read in full169170**Other frameworks**171- Pydantic AI durable execution — https://ai.pydantic.dev/durable_execution172- smolagents secure code execution — https://github.com/huggingface/smolagents/blob/main/docs/source/en/tutorials/secure_code_execution.md173- Mastra suspend/resume (legacy docs) — https://mastra.ai/docs/v0/workflows-legacy/suspend-and-resume ; A2A — https://mastra.ai/docs/connections/a2a174- Letta stateful agents — https://docs.letta.com/v1-sdk/concepts/stateful-agents175- OpenHands Software Agent SDK — https://arxiv.org/html/2511.03690v1 — Nov 2025176- mini-swe-agent — https://github.com/swe-agent/mini-swe-agent177- Goose subagents — https://goose-docs.ai/docs/guides/context-engineering/subagents178- Aider edit formats — https://aider.chat/docs/more/edit-formats.html ; modes — https://aider.chat/docs/usage/modes.html179- Roo Code boomerang tasks — https://roocodeinc.github.io/Roo-Code/features/boomerang-tasks180181**Published lessons and studies**182- W. Yan, "Don't Build Multi-Agents" — https://cognition.com/blog/dont-build-multi-agents — 2025-06-12183- W. Yan, "Multi-Agents: What's Actually Working" — https://cognition.com/blog/multi-agents-working — 2026-04-22184- Anthropic, "How we built our multi-agent research system" — https://www.anthropic.com/engineering/multi-agent-research-system — 2025-06-13185- Anthropic, "Effective harnesses for long-running agents" — https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents — 2025-11-26186- Anthropic, "Harness design for long-running application development" — https://www.anthropic.com/engineering/harness-design-long-running-apps — 2026-03-24187- Cemri et al., "Why Do Multi-Agent LLM Systems Fail?" — https://arxiv.org/abs/2503.13657 — v1 2025-03-17, v3 2025-10-26; project site https://sites.google.com/berkeley.edu/mast/home188- "Towards a Science of Scaling Agent Systems" — https://arxiv.org/html/2512.08296v3 ; https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work — Dec 2025189- Tran & Kiela, "Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets" — https://arxiv.org/html/2604.02460v1 — Apr 2026190191**Protocols**192- MCP 2026-07-28 release candidate — https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate — 2026-05-21193- MCP version history (secondary) — https://eseospace.com/blog/mcp-specification-changelog — Oct 2026; https://stacktr.ee/blog/mcp-2026-spec-changes194- A2A specification 1.0.0 — https://a2a-protocol.org/latest/specification/ — first 100k characters read195- A2A 1.0 migration notes — https://discuss.google.dev/t/the-a2a-1-0-milestone-ensuring-and-testing-backward-compatibility/352258196