Why does the framework matter more than the model? An AI agent’s quality depends primarily on three things: the reasoning model that powers it, the tools it can access, and the orchestration framework that manages its execution flow. Most teams spend their evaluation effort on model selection — GPT-4 vs Claude vs Gemini vs Llama — and minimal effort on framework selection. This is backwards. The model provides the intelligence; the framework determines whether that intelligence can be reliably deployed. In our experience across agentic-AI engagements, the teams that adopt a structured framework early encounter materially fewer production failures than those rebuilding orchestration from raw API calls (an observed pattern across engagements, not a benchmarked industry rate). A well-chosen framework provides structured tool invocation through typed interfaces rather than generated text that might be malformed, execution state management so multi-step task progress is tracked and recoverable, error handling so tool failures and model errors are caught rather than cascading, observability so every step is logged and traceable, and cost control so token usage is monitored and bounded. A poorly chosen framework — or no framework at all — requires building all of these capabilities from scratch. LangChain reports over 90,000 GitHub stars and 200+ integrations as of 2024 (per the project’s public repository metrics), making it the most widely adopted agent framework. An AI Engineer Foundation 2024 survey (published-survey) found that roughly 45% of production agent deployments use LangChain/LangGraph, 20% use custom orchestration, and 15% use AutoGen, with the remainder spread across other frameworks. Adoption is not the same as fitness for purpose, and that is the point of this article. The major frameworks compared LangChain / LangGraph (v0.3 / v0.2, as of January 2026) LangChain is the most widely adopted AI agent framework, with the largest community and the broadest integration ecosystem. LangGraph extends LangChain with explicit graph-based workflow definitions: nodes represent processing steps, edges represent transitions, and conditional logic determines which edges are followed. Strengths. Extensive integration catalogue — 200+ tool integrations, vector stores, document loaders. LangGraph’s explicit graph definition makes complex workflows debuggable: the workflow structure is visible and testable. LangSmith provides production-grade observability — execution traces, latency monitoring, cost tracking, and evaluation infrastructure. Weaknesses. The abstraction layer adds complexity. Debugging issues that span multiple LangChain abstractions requires understanding the framework’s internal dispatch logic. The framework evolves rapidly, and breaking changes between minor versions have been a recurring issue for production deployments. The learning curve for LangGraph’s graph syntax is steeper than for simpler sequential frameworks. Best for: Complex multi-step workflows with conditional branching, production deployments that require observability and evaluation infrastructure, and teams that need broad tool integration coverage. AutoGen (v0.4, as of December 2025) Microsoft’s AutoGen framework is designed around multi-agent conversation: multiple agents with different roles communicate through structured messages to accomplish tasks collaboratively. Strengths. The multi-agent conversation model is intuitive for tasks that benefit from role separation — coder plus reviewer, researcher plus writer, planner plus executor. The agent communication protocol is explicit and auditable. Integration with Azure AI services and the OpenAI APIs is tight. Weaknesses. The conversational coordination model is verbose. Agents communicate through full messages rather than structured data, consuming tokens and increasing latency. The coordination failure modes — unbounded loops, hallucinated handoffs, context loss — must be managed through custom conversation-management logic that the framework does not fully address. Production observability and deployment tooling are less mature than LangSmith. Best for: Multi-agent architectures where role-based collaboration is the primary pattern, and teams already invested in the Microsoft/Azure ecosystem. CrewAI (v0.80, as of February 2026) CrewAI provides a role-based agent framework focused on simplicity: define agents with roles and goals, define tasks with descriptions and expected outputs, and let the framework manage agent coordination. Strengths. The abstraction level is high — defining an agent crew requires minimal boilerplate. As of 2024, CrewAI reports over 20,000 GitHub stars and adoption by over 5,000 development teams (per CrewAI documentation). The role-based metaphor — agents as “crew members” with specific responsibilities — is accessible to engineers without an ML background. The framework handles task delegation and result aggregation with reasonable defaults. Weaknesses. The high abstraction level limits control over execution details. When the default coordination behaviour does not match the use case, customisation options are constrained. The framework is younger and has a smaller community and integration ecosystem than LangChain. Production observability is limited compared to LangSmith. Best for: Rapid prototyping of multi-agent systems, teams that prioritise development speed over fine-grained control, and use cases that fit the role-based delegation pattern naturally. Custom orchestration (no framework) Building agent orchestration directly on the model APIs — OpenAI Assistants, Anthropic Tool Use, Gemini Function Calling — without an intermediate framework. Strengths. Maximum control over every aspect of the agent’s execution. No framework abstraction layer to debug through. No dependency on a third-party framework’s release cycle or design decisions. Weaknesses. Every production requirement — state management, error handling, observability, cost control, tool type safety — must be built from scratch. In our experience across agentic-AI engagements, the development effort is typically 3–5× higher than using a framework for the orchestration layer (observed range across engagements, not a benchmarked industry rate). Maintenance burden compounds as the custom orchestration code grows. Best for: Simple single-tool agents where framework overhead is not justified, organisations with strong infrastructure engineering teams that prefer full control, and cases where existing frameworks do not support the specific orchestration pattern required. Production-readiness criteria that actually separate the frameworks Evaluating frameworks against feature lists is what gets teams into trouble. The criteria below are the ones that distinguish demoware from production substrate: Observability. Can you trace every step of the agent’s execution, including tool inputs and outputs, model prompts and responses, and branching decisions? Can you replay a failed execution to diagnose the root cause? LangGraph plus LangSmith currently provides the strongest observability; custom orchestration provides whatever you build, and AutoGen and CrewAI sit somewhere in between. Error recovery. When a tool call fails — API timeout, malformed response, permission error — does the framework retry, fall back, or crash? When the model produces malformed tool-call parameters, does the framework catch and handle the error, or does the error reach the user? LangGraph provides configurable retry and fallback policies; AutoGen and CrewAI have less mature error-handling primitives. Cost control. Can you monitor and limit token usage per agent, per task, and per session? Can you set circuit breakers that terminate agents entering unbounded loops? This capability is critical: a runaway agent can consume thousands of dollars in API costs in minutes (observed-pattern, drawn from incident reviews across deployments — not a single named benchmark). State persistence. Can the agent’s execution state be persisted and resumed? For long-running tasks — multi-hour research, multi-day workflows — the agent must be able to checkpoint progress and resume after interruptions. LangGraph provides state persistence through pluggable checkpoint storage backends. CrewAI and AutoGen have limited persistence support today. Testing. Can you write unit tests for individual agent steps and integration tests for complete workflows? Can you evaluate agent performance on a held-out suite of inputs and expected outputs? LangSmith provides evaluation infrastructure; other frameworks require custom test harnesses. A decision rubric, not a recommendation We do not recommend a single framework universally. The choice depends on use-case complexity, the team’s engineering maturity, and production requirements. The matrix below maps common situations to the framework that tends to fit best — treat it as a starting point, not a verdict. Situation Framework that tends to fit Why Simple single-agent with 1–3 tools Custom orchestration, or LangChain without LangGraph Framework overhead exceeds the value of its abstractions Complex multi-step workflow with conditional branching LangGraph Explicit graph structure is debuggable and testable Multi-agent collaboration with clear role separation AutoGen or LangGraph (via subgraphs) Conversation model fits role-based coordination Rapid multi-agent prototyping CrewAI High abstraction minimises boilerplate; observability cost is acceptable pre-production Production deployment with full observability and eval LangGraph + LangSmith Strongest trace, replay, and evaluation surface Highly specialised orchestration pattern unsupported by frameworks Custom orchestration Framework constraint costs exceed reinvention costs A framework-agnostic evaluation rubric Versions, features, and community size change rapidly. Use these criteria to evaluate any agent framework — current or future — independent of a specific release: Observability depth. Does the framework provide execution traces with full tool inputs and outputs, model prompts and completions, and branching decisions — or only high-level step summaries? Can you replay a failed run from the trace alone? Error recovery granularity. Can you define per-step retry policies, fallback paths, and circuit breakers? Does the framework distinguish between transient failures (API timeouts) and permanent failures (invalid tool parameters)? State persistence and resumability. Can execution state be checkpointed to durable storage and resumed after process restarts, infrastructure failures, or deliberate pauses in long-running workflows? Cost governance. Does the framework expose token usage per step, per agent, and per session? Can you set hard budget limits that terminate execution before costs exceed a threshold? Testing surface area. Can individual agent steps be unit-tested in isolation with mocked tool responses? Can complete workflows be integration-tested with deterministic model outputs? Upgrade-path stability. Does the framework follow semantic versioning? Are breaking changes documented with migration guides? Is the API surface stable enough for a multi-quarter production commitment? A framework that scores poorly on three or more of these criteria is a demo framework — useful for proofs of concept, dangerous as a production substrate. When does building your own pay off? The case for custom orchestration is narrower than the popularity of “build vs buy” debates suggests. It pays off in three situations: The first is a very simple agent — one or two tools, a single LLM call per turn, no branching, no long-running state. Here the framework’s abstraction tax is real and the build cost is bounded. The second is a deeply specialised orchestration pattern that no framework supports natively. If your agent needs to coordinate with a domain-specific scheduler, a custom approval workflow, or a real-time system with latency budgets that framework dispatch overhead would violate, building directly on the model APIs gives you the control you need. The third is an organisation with a strong platform engineering team that intends to amortise the orchestration substrate across many internal agents. In that case the custom orchestration becomes an internal framework, and the question is no longer “framework or custom” but “open-source framework or our framework”. The honest answer often involves contributing to an existing framework rather than starting from scratch. Outside these three situations, the rewrite cost when a popularity-based pick fails at production scale typically exceeds the cost of choosing carefully up front. We have seen teams burn six to nine months replacing a framework whose error-handling primitives could not be retrofitted — time that would not have been spent had the production-readiness criteria above been applied at week two. FAQ How do I choose an AI agent framework for production (LangChain, AutoGen, CrewAI, Google ADK, custom)? Start from the production-readiness criteria — observability depth, error recovery granularity, state persistence, cost governance, testing surface, and upgrade-path stability — not from popularity. Map your use case to the situation rows in the decision matrix above. LangGraph plus LangSmith is the strongest default for complex, observable production workflows; AutoGen fits role-based multi-agent patterns; CrewAI is for rapid prototyping; custom orchestration is for narrow, well-justified cases. When does building a custom agent framework pay off, and when does it just create technical debt? Custom orchestration pays off in three situations: very simple agents where framework overhead exceeds the value, deeply specialised orchestration patterns no framework supports natively, and platform teams that will amortise the substrate across many agents. Outside these, the development effort is typically 3–5× higher than using a framework, and the maintenance burden compounds as the codebase grows. Which production-readiness criteria separate demo frameworks from real ones? Observability with full trace replay, configurable error recovery and circuit breakers, durable state persistence and resumability, per-session cost governance, a testable surface area for unit and integration tests, and a stable upgrade path with documented breaking changes. A framework that scores poorly on three or more of these is demoware. What does vendor lock-in look like for each major agent framework, and how do I bound it before commitment? Lock-in shows up as framework-specific abstractions in your application code: LangChain’s chain and runnable interfaces, AutoGen’s agent and message classes, CrewAI’s crew and task models. Bound it by isolating framework code behind your own interfaces — your application calls your interfaces, your interfaces call the framework. This keeps the migration cost proportional to the orchestration layer rather than the whole codebase. How do team capability and operational maturity factor into the framework decision? A team without strong observability and incident-response practices should pick the framework that gives them the most observability out of the box, even if it constrains design choices — that is usually LangGraph plus LangSmith. A team with mature platform engineering can absorb more abstraction debt and may benefit from custom orchestration. CrewAI’s high-abstraction model fits teams optimising for development speed; AutoGen fits teams comfortable with conversational coordination semantics. What is the rewrite cost when a framework chosen by popularity fails at production scale? Across engagements we have seen six to nine months of rework when a popularity-based pick had to be replaced because its error-handling or state-persistence primitives could not be retrofitted (observed-pattern, not a benchmarked industry rate). The rewrite cost dominates any savings from a faster initial pick, which is why the evaluation rubric above is worth applying before commitment rather than after. Framework selection based on checklist features rather than production-grade evaluation is a recurring source of rework — a GenAI Feasibility Assessment includes framework evaluation and architecture design against your specific use-case requirements.
