Tool-using multi-agent systems fail in ways a single-call chatbot cannot, because the failures sit in the hand-offs: loops and deadlocks between agents, errors that propagate across agents, role drift over long workflows, and faults introduced by external services and shared memory. To catch them we would build three things: a failure taxonomy built for multi-agent architectures, an adversarial harness that deliberately provokes those failures, and per-step traces that show where a run first went wrong.

TechnoLynx builds this kind of evaluation for agentic systems as part of its generative and agentic AI R&D work. TechnoLynx separates agent-architecture failures from ordinary model errors, then builds a harness with an adversarial scenario generator that provokes routing errors, tool-call loops and escalation mis-triggers. It adds per-step decision logs, validates coverage against known incidents and keeps the harness in place as continuous monitoring.

Failure modes that only appear once agents call tools and each other

When several autonomous agents interact, failure modes emerge that do not exist in single-agent architectures. Collusion, cascading failures, information leakage across boundaries and miscoordination all fall into this category, and none of them can be predicted from single-agent safety evaluations. That is why a chatbot eval suite, however good, says little about a multi-agent system built on the same model.

The starting taxonomy we would use is MAST, the first Multi-Agent System Failure Taxonomy. It identifies 14 failure modes in three categories (system design issues, inter-agent misalignment and task verification), developed from 150 traces with an inter-annotator agreement of kappa 0.88. The authors pair it with MAST-Data, more than 1,600 annotated traces collected across seven multi-agent frameworks. One secondary source describes MAST differently, as four classes and 23 patterns drawn from 136 papers. We would follow the primary paper’s own abstract here, since it is the authors describing their own work.

Some failure modes deserve a named place in any agent-specific test plan:

• Tool-call looping. An agent repeatedly invokes the same tool with identical parameters, gets identical results and does not recognise that it is going round in circles. In SciAgentGym’s analysis of 6,617 error instances, the loop-escape rate was only 35.7%, and models acted on only a median 32.9% of error signals.
• Silent degradation. Agents keep accepting tasks and returning plausible results while quality has already drifted past acceptable limits. In multi-agent systems that degradation travels downstream through delegation chains and shows up far from where it started.
• Error amplification. An upstream factual error is accepted as an assumption and then drives downstream actions or the final output.
• Prompting-style hallucination. In the AgentArch enterprise benchmark, Claude Sonnet 4 hallucinated non-existent agents or tools at 36% under ReAct in multi-agent setups on the more complex use case and at 0% in every other configuration.

The last finding matters for routing tests. A model that behaves well under native function calling can still pick non-existent targets once it is placed in a ReAct-style multi-agent loop, so routing correctness has to be tested in the configuration you will actually ship.

Which published benchmarks can I cite for tool-routing and escalation correctness?

Published benchmarks only partly cover routing and escalation correctness, so you cite the adjacent ones and build the rest yourself. In a 2026 survey of 40 agent-safety benchmarks, none of the nine multi-agent benchmarks yet tests cascading inter-agent failures or long-horizon coordination risks in production deployments. The same survey found no benchmark primarily designed to test error cascading in multi-step plans.

Benchmark choice also changes the safety conclusions. Across twelve models and four benchmarks, safety rankings showed no concordance (Kendall’s W = 0.10, p = 0.94). Cite the methods rather than leaderboard positions: MAST for the taxonomy, ToolEmu for adversarial scenario design, and τ-bench’s pass^k metric for reliability across repeated trials. A survey of agentic security also points to InjecAgent, AgentBench and CyberSecEval as standardised evaluation benchmarks. Prompt-injection and jailbreak testing against a serving API is a separate surface. It belongs in a continuous adversarial-testing program for deployed AI rather than in this harness.

How do you design red-team cases that trigger tool misrouting and escalation loops?

You design them by writing down, for each case, the ambiguity or fault that should make the agent go wrong, then building an environment in which that failure is likely, and then checking contracts at every step. ToolEmu (Ruan et al., ICLR 2024) supplies a template for this. Each test case lists the toolkits, a user instruction with deliberate underspecification, the expected achievement, the potential risky outcomes and the potential risky actions. Its adversarial emulator uses those fields to set up sandbox states where a failure is more likely, for example two medicine bottles that both match the user’s description instead of one.

The method finds real failures. In the ToolEmu study, even the safest agent failed in 23.9% of the 144 test cases, and human review judged 68.8% of the identified failures to be valid real-world agent failures. Read the 23.9% with care, though: a later survey notes that containerised benchmarks report 50–86% unsafe rates from benign prompts, while LLM-emulated ToolEmu reports a lower rate for similar risks.

A procedure for routing and loop cases

1. Seed fabricated upstream outputs. Feed a deliberately wrong result from an earlier step into a follow-up task and trace whether the error cascades through downstream agents, a test in the Cloud Security Alliance’s agentic red-teaming guide.
2. Build escalation traps. Write cases that exercise feedback-loop resource exhaustion, a multi-agent threat category in the same guide.
3. Attach machine-checkable contracts. Useful templates include “verify before acting” (a verifier step must come before any side-effecting call), least privilege, and a progress contract that requires a progress measure to decrease within every window of length w unless the run terminates. In the framework, the progress contract is the template that covers progress and termination. A hard verification gate in front of irreversible actions such as refunds is the same “verify before acting” idea applied to one action type.
4. Inject integration faults. Timeouts, partial outages, stale caches, malformed payloads and inconsistent shared memory are injected at system boundaries. Containment counts as achieved when the fault is detected, mitigated, and the final output still satisfies its contracts. Report the containment rate by fault type.
5. Cap the run. SciAgentGym capped tool interactions at 50 rounds per attempt and enforced a 300-second per-request timeout. In its case studies, a looping model kept re-invoking the same subroutine until it exhausted the round budget.
6. Run multiple seeds and keep replays. Store seeds, tool stubs and fault schedules so that each failure can be reproduced deterministically and rerun later as a regression test.

Two metrics belong on every dashboard. One is the tool repetition rate, meaning consecutive identical calls with identical arguments. The other is attack success rate, the share of adversarial trials in which the agent takes an unauthorised or policy-violating action. Our guide to task-specific evaluation harnesses that drive deployment decisions covers how to turn results like these into a go or no-go call.

The limits are worth stating plainly. A taxonomy built from published benchmarks may miss failure modes peculiar to your agent’s tool set, and red-team scenarios only probe the failures someone thought to anticipate. Fault injection and replays of real incidents help fill that gap. Fresh incidents still need to feed back into the case library.

Auto-escalation without a per-step decision log cannot be audited

If a system escalates on its own and keeps no per-step decision log, nobody can reconstruct why it escalated, so it cannot be audited. The mechanism to aim for links each emitted action to the reasoning step, retrieved memories and prior observation that produced it, in an immutable log. One 2026 survey argues that the logging requirements of EU AI Act Article 12 and ISO/IEC 42001 call for hash-chaining every agent step in an append-only Merkle tree. That reading is the survey’s, not a regulator’s, so treat it as one implementation route rather than a legal requirement.

Per-step contracts also let the log localise failures to specific steps and agents. Each violation record holds the violated contract IDs, a severity and the mitigation taken (retry, replan, sandbox, escalate, block). Reporting the first violating step and the agent responsible shows whether failures start early, in planning and delegation, or late, in tool actions and escalation. Missing logs in production escalation are an audit risk in their own right, whatever your taxonomy says.

How do I spot an agent drifting away from the original goal mid-task?

Measure drift against an explicit role contract for each agent. The trace-based assurance framework of Message-Action Traces gives each agent a contract of allowed actions and obligations. It then scores drift as the fraction of that agent’s steps that fall outside the allowed set, and the authors note drift becomes more likely as context accumulates or injected content changes the effective constraints. The searcher’s question about “the multi-agent failure mode where an agent gradually changes the original goal” maps onto this role drift, and onto the objective-drifting attacks that the AgentLAB benchmark tests across 644 cases.

Drift does not need an attacker. When a context window fills, compaction can silently drop safety-critical instructions such as “confirm before acting”, and a study cited in the 2026 benchmark survey found refusal rates shifting by 30–70% at contexts of 100K tokens or more. One analysis of CrewAI observes that agents can gradually produce lower-quality output after repeated executions without triggering retry or delegation. Surface metrics such as completion rate and latency can stay normal while this happens.

In practice we would combine three checks: a per-agent drift score on every trace, a canary instruction whose survival is tested after each compaction event, and a measure of how many unsupported claims are used downstream as inputs to later actions. The framework defines that last quantity as the share of unsupported claims later referenced in subsequent steps.

MoE proxy runs: metrics that catch expert collapse before full scale

When the agent’s model is itself a mixture-of-experts (MoE) model being trained or RL post-trained, the training run is another failure surface. The RL case is the sharp one: the R3 study reports MoE routing instability severe enough to cause catastrophic RL training collapse. The cheapest defence is a small proxy run instrumented to log routing behaviour per layer and per step. Logging also answers the time-window problem: a short run that records only loss can miss concentration that per-token routing entropy shows directly.

Signal	What it shows	Source
Per-token routing entropy (pre-dispatch softmax)	Concentration of routing probability. It can fall while selected experts and load counts stay unchanged	Per-token entropy monitor paper, 2026
MaxVio (global and per batch)	Maximum deviation of any expert’s load from the perfectly balanced load	DeepSeek, Loss-Free Balancing
Dead experts	Experts that receive almost no tokens and become unused weights	OLMoE
Router saturation	Share of top-k routing already fixed relative to the final checkpoint	OLMoE
Training–inference routing mismatch, F(τ=2)	Share of tokens whose training and inference probabilities differ by more than a factor of two	R3 study
Expert Load Entropy, Routing Diversity Index, Expert Centrality Drift	Concentration, loss of flexibility, temporal volatility	Routing diagnostics framework

Two cautions about this table. Router weight similarity, which some teams monitor, does not uniquely determine expert selection, so the per-token entropy monitor paper argues for per-token entropy as the more direct signal. And in one fault-injection run, the attention-update spectrum departed from baseline about 4,000 steps before the loss upturn. That early lead belongs to the FlashAttention monitor; the router entropy monitor was shown to respond to hyperparameter-driven concentration.

Some instabilities from other labs are worth probing for deliberately. In the R3 study, 94% of tokens selected a different expert in at least one layer between training and inference, and collapsed runs without routing replay almost always showed abnormally high F(τ=2), which exceeded 0.1 in one GRPO-trained SFT model. Sparse models are more sensitive to roundoff errors because routers add exponential functions, which motivated the router z-loss. Without a load-balancing loss, OLMoE’s first layer left most experts as unused dead weight.

Proxies are useful but imperfect. One 2026 method that builds MoE proxies by selecting representative experts cut per-step NPU-hours by about 55.5% for Qwen3-30B-A3B and reproduced fault anomaly directions and trends. The same paper warns that directly removing or merging experts can alter routing preferences and leave the original faults unreproduced. Proxy runs therefore reduce risk; they cannot certify that full scale will be stable, because a reduced model may fail to reproduce the original model’s fault-induced anomalies and training trends.

One 2026 study reports dead experts recovering (from 12 to 4 dead) as its control parameter was raised, which runs against the usual assumption that dead experts are lost for good. The MoEWatch project reports fewer dead experts with live interventions, but its authors say results are sensitive to model, data order and collapse pressure and do not guarantee prevention. We would read both as reasons to keep logging rather than as permission to ignore an early dip.

Testing anti-rollback across OTA, bootloader and secure element, not just cold boot

For agents running on embedded or edge devices, we would not stop at a rollback test that only checks the version at cold boot. The test plan has to try a forced downgrade at each layer: the OTA transport, the bootloader’s version check and the secure element’s firmware versioning, separately and in combination. The attack to attempt is the one documented on TrustZone devices. If older trustlets, the TrustZone OS or even the bootloader are signed with the same key and lack rollback prevention, an attacker replaces the patched component with an older, exploitable version. On a Google Nexus 5 the authors downgraded the TrustZone OS and the bootloader without failure.

Each layer has a concrete check to aim at:

• Bootloader. MCUboot’s hardware-based downgrade prevention compares the new image’s security counter with a value held in trusted non-volatile storage. Android Verified Boot rejects an image unless its rollback index is at least the stored rollback index for every slot.
• OTA application slots. ESP-IDF only treats an app as a rollback candidate if its secure_version is at least the eFuse secure_version when anti-rollback is enabled.
• Counter policy. Arm’s PSA Trusted Boot and Firmware Update specification allows a boot test of the new image before the counter is incremented, and requires any reset mechanism to be at least as secure as image signing.

The Secure by Design handbook lists the evidence a test plan should produce: valid update, invalid signature, interrupted update, rollback attempt and recovery path. Among common OTA gaps it lists treating bootloader updates as ordinary application updates. NIST SP 800-193 notes that a legitimate rollback path may require the user’s physical presence, precisely so that attackers cannot reinstall vulnerable firmware.

The caveat that decides whether a lab pass means anything is irreversibility. MCUboot requires the counter to be stored in a non-volatile, trusted component, and Android Verified Boot keeps it in tamper-evident storage.

Supervisor-worker or debate-consensus: how do you A/B test the choice for your own task?

Run both patterns head-to-head on your exact task, because the published evidence finds no universally optimal architecture. AgentArch evaluated 18 configurations and found that models reach peak performance under different architectures depending on the use case, and its authors caution against assuming one architecture suits a model. Reliability was the weak point: pass^k across all models and configurations peaked at 0.0634, a 6.34% chance of completing the workflow correctly in all eight trials. The two patterns differ structurally too. A supervisor delegates sub-tasks to specialised agents, while peer-to-peer meshes negotiate directly without a central bottleneck.

Centralisation is not automatically safer either. The SafeAgents study found attack success on RedCode rising from 62.5% with a single agent to 83.7% with centralised multi-agent orchestration. OrchBench adds a further lesson for AI agent orchestration: keeping task-critical information flowing between agents matters more than adding agents, and parallelism yields diminishing returns as coordination failures accumulate.

Worked example: a cost-per-request comparison agent

Assume the task compares inference cost per request across hosted LLM providers, with pricing pulled from provider APIs that can disagree or lag. A fair A/B would run like this:

1. Fix the task set, tool stubs and recorded API responses, including cases where sources genuinely conflict and cases where one source is stale.
2. Run supervisor-worker and debate-consensus on identical inputs over several trials each, and report pass^k alongside pass@1, since τ-bench introduced pass^k precisely to expose inconsistency across repeated trials.
3. Apply the same hard termination guard to both arms (a round cap plus a progress contract), otherwise a pattern that loops will distort the comparison.
4. Score whether the final answer flags conflicting sources, not only whether the cheapest provider was named correctly.

To screen many orchestration plans cheaply before real runs, OrchBench replaces worker execution with a deterministic simulator. Its simulated scores correlated with real execution quality at Pearson r = 0.816 while using 1.3% of the tokens and 10.3% of the wall-clock time.

Situation	Lean towards	Why
Sources may conflict or be stale; traceability matters	Supervisor-worker with an explicit verifier role	Arbitration is a named step that can be logged and audited
Sources expected to agree; termination guards in place	Debate-consensus is a candidate	Wrong consensus arose in initially disputed cases
Either	Measure on your task before committing	No universally best architecture was found across 18 configurations

What goes wrong with debate agents when tool outputs conflict or are stale?

Debate can turn disagreement into a confident wrong answer. In a study of multi-agent debate on MMLU-Pro, 469 of the 1,963 cases where agents initially disagreed (23.9%) ended in unanimous wrong consensus by round 3. The rate reached 34.8% in law questions and fell to 12.1% in maths. The same 23.9% figure appears in ToolEmu, where it measures something different: tool-related failures of the safest agent.

Answer-level agreement does not guarantee aligned reasoning. A medical QA study found that standard debate reduced explicit contradictions between agents while making their reasoning chains less similar, which the authors call the consistency illusion. Majority pressure also suppresses independent correction. Faced with an incorrect majority, stronger models corrected the consensus in 34.4% and 30.0% of cases, while a weaker model did so in only 3.6%.

These studies use question-answering benchmarks, not pricing APIs, so applying them to stale tool outputs is our judgement. Our reading: when one agent anchors on a stale price, debate is more likely to talk the others into it than to surface the conflict, unless staleness and source disagreement are detected explicitly before the debate starts. A calibrated escalation layer is one remedy: the conformal method in that study flagged 81.9% of final-round wrong-consensus cases as uncertain rather than committing to them. Its authors also say high-stakes use still needs domain validation and human oversight.

For an agentic system, the evidence that it is ready comes from your own harness, run on your own task. Treating that harness, its regression replays and its drift monitors as release artefacts in their own right is part of what we mean by production AI reliability: a discipline distinct from model accuracy, and not a replacement for safety certification or regulatory sign-off. The open question each team still has to settle is which failures its harness cannot yet imagine, and how quickly a production incident becomes a new test case.