Why most agent-framework comparisons stop being useful at the edge boundary Comparison guides for AI agent frameworks (LangChain, LangGraph, AutoGen, CrewAI, semantic kernel, Google ADK, in-house orchestration) almost always assume the inference layer is a remote LLM API call: seconds of latency budget, no local memory ceiling, no concern for runtime portability beyond which HTTP client to use. That assumption silently does most of the work in the recommendation. Take it away and the ranking changes — sometimes inverts. A growing share of agent deployments cannot make that assumption. A mobile assistant that has to keep working when the network drops. An in-vehicle voice agent where the round-trip to a cloud LLM blows the response budget. An on-premise enterprise deployment behind an air-gap where the model itself has to live on the box. In each of these, part of the agent — the smaller, latency-critical model, the embedding step, the retriever, the safety classifier — has to run on-device. The orchestration layer no longer talks only to a remote API; it has to compose against a local inference runtime with sharp constraints. The question this article answers is not “which framework is best?” The framework that wins on the desktop benchmark often loses on three of the four axes that govern edge deployments. The question is: which framework decision survives once the edge constraints are made first-class? This is the cross-K bridge that connects how to choose an AI agent framework for production (the agent-architecture surface) with distillation vs quantisation for multi-platform edge inference (the edge-target compression surface). Neither article answers it on its own. What edge-constrained means precisely “Edge” is overloaded. For this decision it has a precise meaning: at least one of the agent’s inference steps runs on hardware where one or more of the following are true. Tail latency budget is hundreds of milliseconds, not seconds. A voice agent that has to respond perceptibly in real time, a screen reader that has to keep up with scrolling, a control loop that has to close inside a frame budget. The orchestration layer cannot afford its own variable overhead in that budget. Memory ceiling is phone-class, not data-centre-class. A few hundred megabytes of working memory for the entire agent process, including the framework, the loaded model weights, the conversation state, and the application that hosts it. The framework’s own footprint becomes a non-trivial fraction of the budget. Runtime portability is a first-class requirement. The same agent has to deploy to CoreML on iOS, ONNX Runtime on Android, and possibly WebGL/ONNX.js in a browser. The inference adapter is not “Python with the framework installed”; it is three different runtimes with three different operator-support matrices. Network reachability is intermittent or absent by design. Either the deployment context (vehicle, factory floor, regulated air-gap) or the user expectation (offline mode of a consumer app) requires the agent to function with no remote LLM available, even briefly. A deployment that hits any one of these is edge-constrained for framework-selection purposes. A deployment that hits two or more invalidates most of the framework comparisons that exist online. The four axes that decide whether a framework survives Once edge constraints are made first-class, framework selection collapses onto four axes. These are evaluable before commitment from the framework’s documentation, source code, and a small spike — they do not require running the agent end-to-end. Axis 1 — Inference adapter portability Does the framework abstract over the inference backend, or does it assume a single runtime? Frameworks that grew up against the OpenAI API typically have a single, deeply-wired LLM call interface; switching backends is a swap of the client class, but switching to a non-HTTP runtime (CoreML, ONNX Runtime, on-device GGUF) is a structural change that breaks streaming, function-calling, and token-counting in different places. Practical evaluation: read the framework’s “custom LLM” or “custom backend” documentation. If the integration interface assumes a chat-completions-shaped HTTP response, the framework is one runtime away from “porting” — every additional runtime multiplies the integration surface. If the framework has an explicit local-runtime adapter (CoreML, ONNX Runtime, llama.cpp wrapper) maintained as a first-class extension, it has already absorbed the portability cost. Axis 2 — State-machine memory profile Agent frameworks persist state: the conversation, the tool-call history, the working scratch-pad, the retrieved documents, sometimes a growing structured plan. On the desktop the persistence layer is invisible. On a phone-class device the persistence layer is part of the memory budget — and the budget includes the model weights, the activation buffers, and the host application. Practical evaluation: instantiate the framework’s state object with a representative conversation length (say, 50 turns with tool calls and retrieved snippets) and measure the resident memory. Frameworks that build the state as immutable snapshots per turn (functional-style orchestration) accumulate memory at a different rate than frameworks that mutate a single object. Neither is automatically right; the question is whether the rate fits the device ceiling. Axis 3 — Latency budget composability The framework adds orchestration overhead between inference calls: serialisation, validation, message routing, retry logic, telemetry hooks. On the desktop this overhead is dominated by the seconds-long inference call and is invisible. On the edge the inference call may be in the tens-to-low-hundreds of milliseconds, and an orchestration overhead of similar order per step is now a non-trivial fraction of the latency budget (illustrative figures from observed edge engagements, not benchmarked industry rates). A multi-step agent flow with three orchestration hops between inferences can consume half the budget before the model runs. Practical evaluation: run a no-op agent flow (one step, one no-op tool call, one response) on the target device and measure end-to-end latency with the actual model swapped for a constant-time stub. The number is the framework overhead floor. Compare it to the inference budget for the smallest model that satisfies the use case (as an illustrative range from observed edge engagements, not a benchmarked industry rate, the inference budget can sit anywhere from 80–200ms and the framework overhead floor anywhere from a few milliseconds to 30ms; what matters is the ratio between them). If the floor is more than approximately 20% of the budget (planning heuristic from observed engagements, not a universal threshold), the framework will be the dominant tail-latency contributor regardless of how well the model is optimised. Axis 4 — Runtime footprint The framework itself adds to binary size and cold-start time. On a server this is a one-time process startup cost. On a mobile or browser deployment, binary size is shipped to every user and cold-start time is a perceptible delay every time the app launches. Practical evaluation: build the framework into the target binary with no application code, measure the binary size delta and the cold-start time delta. Frameworks that pull in heavy dependency trees (full tokeniser libraries, observability stacks, vector database clients) can add tens of megabytes to a mobile binary and hundreds of milliseconds to cold start. Frameworks designed for embedding (lighter dependency surfaces, lazy initialisation, optional subsystems) shift those costs from every-user-every-launch to only-when-used. The decision matrix Score the candidate frameworks on each axis as fit / partial fit / misfit based on the practical evaluations above. The decision pattern that emerges from edge-constrained agent deployments we have shipped is: All four axes fit — proceed with the framework as the orchestration layer; the standard agent-framework decision criteria (observability, error recovery, vendor lock-in tolerance) take over. One axis misfits, three fit — the misfit axis predicts the rewrite point. If Axis 1 (adapter portability) is the misfit, the rewrite happens when the second runtime is added. If Axis 2 (memory) is the misfit, the rewrite happens when conversation length grows. If Axis 3 (latency overhead) or Axis 4 (footprint) is the misfit, the rewrite is forced at the first user-acceptance round on the target device. Plan the rewrite explicitly or pick a different framework. Two or more axes misfit — do not adopt the framework for the edge target. The cost of working around two structural misfits exceeds the cost of using a thinner orchestration layer (in-house state machine, a lighter framework, or no framework). The thinner layer is not free, but its cost is bounded and visible. This is not a popularity ranking. A framework that scores well on all four axes for a given edge constraint may score poorly when a different constraint dominates. The matrix is the artefact; the ranking is downstream of which constraints apply. What the edge-target side of the bridge contributes The framework decision above assumes the model itself is already chosen. That assumption is only valid if the model fits the edge target — which is the question distillation vs quantisation for multi-platform edge inference answers. The two decisions interact: a framework with strong Axis 1 (portable adapters) lets the team pick a quantisation-per-runtime strategy without paying multiple integration costs; a framework with weak Axis 1 forces the team toward a distillation-then-portable-export strategy because each additional runtime multiplies the framework integration cost. In practice, edge agent deployments resolve the two decisions together: the model team picks a compression path, the agent team picks a framework, and they agree on the inference-adapter contract that connects them. When that contract is made explicit early — not derived implicitly from whichever framework was adopted first — the rewrite at the production boundary stops being inevitable. What remains hard to predict before committing The four axes above are evaluable from documentation and a small spike. What they do not predict is the framework’s behaviour under failure modes specific to edge inference: what happens when the on-device model returns malformed output that the desktop test suite never produced because the desktop model never got that quantised; what happens when the device drops a function call mid-stream because the runtime ran out of memory at the wrong moment. These behaviours surface only in production-like testing on the target hardware with the actual quantised model. The decision matrix bounds the risk; it does not eliminate it. Where this fits in the broader framework decision This article is the edge-target specialisation of the general agent-framework decision; the general criteria (observability, error recovery, state management, vendor lock-in, team capability) remain the right starting point and are covered in how to choose an AI agent framework for production. When the deployment hits one or more of the edge-constrained conditions defined above, the four axes here become the dominant decision; when it does not, they are background considerations. If your team is selecting an agent framework for a deployment that includes an on-device inference step, a GPU & Inference Optimization Assessment evaluates the four axes against your specific edge target and surfaces the framework rewrite point — if there is one — before commitment.
