Agent Framework Selection for Edge-Constrained Inference Targets

Why most agent-framework comparisons stop being useful at the edge boundary

Comparison guides for AI agent frameworks — LangChain, LangGraph, AutoGen, CrewAI, semantic kernel, Google ADK, in-house orchestration — almost always assume the inference layer is a remote LLM API call: seconds of latency budget, no local memory ceiling, no concern for runtime portability beyond which HTTP client to use. That assumption silently does most of the work in the recommendation. Take it away and the ranking changes. Sometimes it inverts.

A growing share of agent deployments cannot make that assumption. A mobile assistant that has to keep working when the network drops. An in-vehicle voice agent where the round-trip to a cloud LLM blows the response budget. An on-premise enterprise deployment behind an air-gap where the model itself has to live on the box. In each of these, part of the agent — the smaller latency-critical model, the embedding step, the retriever, the safety classifier — has to run on-device. The orchestration layer no longer talks only to a remote API; it has to compose against a local inference runtime with sharp constraints.

The question this article answers is not “which framework is best?” The framework that wins on the desktop benchmark often loses on three of the four axes that govern edge deployments. The question is: which framework decision survives once the edge constraints are made first-class? This is the cross-K bridge that connects how to choose an AI agent framework for production with distillation vs quantisation for multi-platform edge inference. Neither article answers it on its own.

What edge-constrained means precisely

“Edge” is overloaded. For this decision it has a precise meaning: at least one of the agent’s inference steps runs on hardware where one or more of the following are true.

Tail latency budget is hundreds of milliseconds, not seconds. A voice agent that has to respond perceptibly in real time, a screen reader that has to keep up with scrolling, a control loop that has to close inside a frame budget. The orchestration layer cannot afford its own variable overhead in that budget.
Memory ceiling is phone-class, not data-centre-class. A few hundred megabytes of working memory for the entire agent process, including the framework, the loaded model weights, the conversation state, and the application that hosts it. The framework’s own footprint becomes a non-trivial fraction of the budget.
Runtime portability is a first-class requirement. The same agent has to deploy to CoreML on iOS, ONNX Runtime on Android, and possibly WebGL or ONNX.js in a browser. The inference adapter is not “Python with the framework installed”; it is three different runtimes with three different operator-support matrices.
Network reachability is intermittent or absent by design. Either the deployment context (vehicle, factory floor, regulated air-gap) or the user expectation (offline mode of a consumer app) requires the agent to function with no remote LLM available, even briefly.

A deployment that hits any one of these is edge-constrained for framework-selection purposes. A deployment that hits two or more invalidates most of the framework comparisons that exist online.

The four axes that decide whether a framework survives

Once edge constraints are made first-class, framework selection collapses onto four axes. These are evaluable before commitment from the framework’s documentation, source code, and a small spike — they do not require running the agent end-to-end.

Axis 1 — Inference adapter portability

Does the framework abstract over the inference backend, or does it assume a single runtime? Frameworks that grew up against the OpenAI API typically have a single, deeply-wired LLM call interface; switching backends is a swap of the client class, but switching to a non-HTTP runtime (CoreML, ONNX Runtime, on-device GGUF) is a structural change that breaks streaming, function-calling, and token-counting in different places.

Practical evaluation: read the framework’s “custom LLM” or “custom backend” documentation. If the integration interface assumes a chat-completions-shaped HTTP response, the framework is one runtime away from “porting” — every additional runtime multiplies the integration surface. If the framework has an explicit local-runtime adapter (CoreML, ONNX Runtime, a llama.cpp wrapper) maintained as a first-class extension, it has already absorbed the portability cost.

Axis 2 — State-machine memory profile

Agent frameworks persist state: the conversation, the tool-call history, the working scratch-pad, the retrieved documents, sometimes a growing structured plan. On the desktop the persistence layer is invisible. On a phone-class device it is part of the memory budget — and the budget already includes the model weights, the activation buffers, and the host application.

Practical evaluation: instantiate the framework’s state object with a representative conversation length (say, 50 turns with tool calls and retrieved snippets) and measure the resident memory. Frameworks that build the state as immutable snapshots per turn (functional-style orchestration) accumulate memory at a different rate than frameworks that mutate a single object. Neither is automatically right; the question is whether the rate fits the device ceiling.

Axis 3 — Latency budget composability

The framework adds orchestration overhead between inference calls: serialisation, validation, message routing, retry logic, telemetry hooks. On the desktop this overhead is dominated by the seconds-long inference call and is invisible. On the edge the inference call may be in the tens-to-low-hundreds of milliseconds, and an orchestration overhead of similar order per step is now a non-trivial fraction of the latency budget (observed pattern from edge engagements, not a benchmarked industry rate). A multi-step agent flow with three orchestration hops between inferences can consume half the budget before the model runs.

Practical evaluation: run a no-op agent flow (one step, one no-op tool call, one response) on the target device and measure end-to-end latency with the actual model swapped for a constant-time stub. The number is the framework overhead floor. Compare it to the inference budget for the smallest model that satisfies the use case. As an observed range across edge engagements we have shipped — not a benchmarked rate — the inference budget tends to sit anywhere from 80–200ms and the framework overhead floor anywhere from a few milliseconds to 30ms; what matters is the ratio between them. If the floor is more than roughly 20% of the budget (planning heuristic from observed engagements, not a universal threshold), the framework will be the dominant tail-latency contributor regardless of how well the model is optimised.

Axis 4 — Runtime footprint

The framework itself adds to binary size and cold-start time. On a server this is a one-time process startup cost. On a mobile or browser deployment, binary size is shipped to every user and cold-start time is a perceptible delay every time the app launches.

Practical evaluation: build the framework into the target binary with no application code, and measure the binary size delta and the cold-start time delta. Frameworks that pull in heavy dependency trees (full tokeniser libraries, observability stacks, vector database clients) can add tens of megabytes to a mobile binary and hundreds of milliseconds to cold start. Frameworks designed for embedding — lighter dependency surfaces, lazy initialisation, optional subsystems — shift those costs from every-user-every-launch to only-when-used.

The decision matrix

Score the candidate frameworks on each axis as fit / partial fit / misfit based on the practical evaluations above. The decision pattern that emerges from edge-constrained agent deployments we have shipped is:

Axis fit profile	Decision	Why
All four axes fit	Adopt the framework as the orchestration layer	Standard agent-framework criteria (observability, error recovery, vendor lock-in tolerance) take over
One axis misfits, three fit	Adopt, but plan the rewrite point explicitly	The misfit axis predicts where the rewrite happens
Two or more axes misfit	Do not adopt for the edge target	Working around two structural misfits exceeds the cost of a thinner layer

The misfit-axis-as-predictor pattern is worth making explicit. If Axis 1 (adapter portability) is the misfit, the rewrite happens when the second runtime is added. If Axis 2 (memory) is the misfit, the rewrite happens when conversation length grows. If Axis 3 (latency overhead) or Axis 4 (footprint) is the misfit, the rewrite is forced at the first user-acceptance round on the target device. A thinner orchestration layer — an in-house state machine, a lighter framework, or no framework at all — is not free, but its cost is bounded and visible.

This is not a popularity ranking. A framework that scores well on all four axes for a given edge constraint may score poorly when a different constraint dominates. The matrix is the artefact; the ranking is downstream of which constraints apply.

How the edge-target side of the bridge feeds back

The framework decision above assumes the model itself is already chosen. That assumption is only valid if the model fits the edge target — which is the question distillation vs quantisation for multi-platform edge inference answers. The two decisions interact: a framework with strong Axis 1 (portable adapters) lets the team pick a quantisation-per-runtime strategy without paying multiple integration costs; a framework with weak Axis 1 forces the team toward a distillation-then-portable-export strategy because each additional runtime multiplies the framework integration cost.

In practice, edge agent deployments resolve the two decisions together. The model team picks a compression path, the agent team picks a framework, and they agree on the inference-adapter contract that connects them. When that contract is made explicit early — not derived implicitly from whichever framework was adopted first — the rewrite at the production boundary stops being inevitable. We have seen this pattern play out across the TTS inference optimisation on edge case study and our client-side ML inference work in the browser: the multi-runtime constraint is what makes the contract worth defining.

What remains hard to predict before committing

The four axes above are evaluable from documentation and a small spike. What they do not predict is the framework’s behaviour under failure modes specific to edge inference: what happens when the on-device model returns malformed output that the desktop test suite never produced because the desktop model never got that quantised; what happens when the device drops a function call mid-stream because the runtime ran out of memory at the wrong moment. These behaviours surface only in production-like testing on the target hardware with the actual quantised model. The decision matrix bounds the risk; it does not eliminate it.

Where this fits in the broader framework decision

This article is the edge-target specialisation of the general agent-framework decision. The general criteria — observability, error recovery, state management, vendor lock-in, team capability — remain the right starting point and are covered in how to choose an AI agent framework for production. When the deployment hits one or more of the edge-constrained conditions defined above, the four axes here become the dominant decision; when it does not, they are background considerations.

If your team is selecting an agent framework for a deployment that includes an on-device inference step, a GPU & Inference Optimization Assessment evaluates the four axes against your specific edge target and surfaces the framework rewrite point — if there is one — before commitment.

FAQ

Which agent framework decisions break down when part of the inference must run on-device?

The decisions that assume a remote LLM API: HTTP-shaped backend interfaces, unbounded persistence layers, orchestration overhead that gets lost inside multi-second inference calls, dependency trees sized for server processes. Each of these is invisible on the desktop and structurally limiting on the edge.

How do tail-latency budgets, memory ceilings, and runtime portability change the framework shortlist for edge targets?

They collapse the shortlist onto four axes — adapter portability, state-machine memory profile, latency-budget composability, runtime footprint — and frameworks that score well on the desktop ranking often misfit on two or more of these. The shortlist is whatever survives the matrix, not whatever wins the demo.

Which frameworks abstract over CoreML / ONNX Runtime / WebGL backends, and which assume a single inference target?

Frameworks with explicit local-runtime adapters maintained as first-class extensions have absorbed the portability cost; those whose “custom LLM” interface assumes a chat-completions-shaped HTTP response have not. Read the custom-backend documentation before scoring Axis 1; the answer is in the integration surface, not in the marketing.

How do I bound an agent framework’s orchestration overhead against an on-device inference budget?

Run a no-op agent flow on the target device with the real model swapped for a constant-time stub. The end-to-end latency is the framework overhead floor. Compare it to the inference budget for the smallest viable model; if the floor exceeds roughly 20% of the budget, the framework dominates tail latency regardless of model optimisation.

When does an edge-constrained agent need to bypass the framework’s persistence layer entirely?

When Axis 2 misfits — when the framework’s state representation accumulates memory faster than the device ceiling allows for the conversation lengths the use case requires. A thinner in-house state machine, scoped to what the agent actually needs to remember, is often the right answer when the framework’s persistence assumes server-class memory.

How do model compression decisions feed back into the agent framework choice?

A framework with strong adapter portability lets the model team pick a quantisation-per-runtime strategy without multiplying integration costs; a framework with weak portability forces a distillation-then-portable-export path because each additional runtime multiplies framework work. The two decisions are coupled — distillation vs quantisation for multi-platform edge inference covers the compression side of the bridge.