# Multi-Agent Systems Fail Quietly > Multi-agent systems degrade without erroring. Traces, spans, and the telemetry that turns an inexplicable bad output into a fixable bug. Published: 2026-07-14 Topic: Agent engineering Author: Alex Cinovoj, Founder, TechTide AI Canonical: https://clawli.ai/blog/multi-agent-observability A single agent fails with an error. A multi-agent system fails by producing something slightly worse than usual, for three weeks, and nobody notices. ## Key takeaways - Trace every run end to end with one ID across all agents and tools. - Log inputs and outputs at every boundary, not just at the ends. - Degradation, not failure, is the dominant production failure mode. - Replayability is the difference between a fix and a guess. Distributed systems taught us that a request touching five services needs a trace ID. Multi-agent systems are distributed systems with an additional property: every node is non-deterministic, so you cannot reproduce a failure by re-running the input. That makes observability not a nice-to-have but the only way to fix anything. ## The trace model One trace per run. One span per agent invocation and per tool call, nested. Each span carries: - Full input, including the assembled context, not a summary, the actual bytes. - Full output, including refusals and malformed responses. - Model identifier and version, because "we upgraded the model" explains a startling share of mysteries. - Token counts in and out, and latency. - Retry count and the reason for each retry. - Which memory items were retrieved and their relevance scores. The context bytes are the expensive one to store and the one you will always wish you had. Compress it, expire it after thirty days, but store it. A trace without the input is a timestamp with ambitions. > **The failure we could not explain** > > Output quality dropped for three weeks. No errors, no alerts. The trace showed a memory retrieval returning six items instead of the usual two, a threshold change had let low-relevance episodes into the window and diluted the brief. Without span-level retrieval logging, that was unfindable. ## Degradation is the real enemy Hard failures are easy: something throws, something alerts, someone fixes it. The dangerous mode is a system that keeps working while getting worse, a tool quietly returning stale data, a prompt truncated by growing context, a retry policy masking a 40% failure rate. Detecting that requires metrics with baselines and alerts on deltas rather than thresholds: - Tool success rate per tool, alerting on a drop of more than three points week over week. - Mean and p99 steps per run, a rising p99 means stopping conditions are eroding. - Context utilization as a fraction of budget, alerting above 80%. - Retry rate per step. This is the metric that hides the most. - Human rejection reason distribution. The earliest signal available, because reviewers notice before dashboards do. ## Replay is the whole point The capability worth building: take a trace ID, reconstruct the exact inputs to any span, and re-run that span in isolation against a modified prompt or a different model. That turns a vague quality complaint into a controlled experiment. It also makes the golden set self-populating. Any production run that went wrong becomes a test case with real inputs attached, which is exactly the material [evals](/blog/evals-or-it-didnt-ship) need and the material teams usually lack. > Every unexplained bad output is either a test case or a recurring incident. The trace decides which. ## Cost belongs in the trace Attach spend to spans rather than to a monthly total. It answers the question that matters (which step is expensive) and makes optimization an engineering task instead of a finance argument. In our system one critique step was 40% of total cost until the trace made it obvious; smaller model, no measurable quality change, [budget](/blog/token-budgets-are-the-new-headcount) recovered. ## Make it visible to users A simplified trace view (what the agent looked at, what it decided, what it produced) is a trust feature as much as a debugging tool. Users who can see the reasoning path forgive a bad output; users who see only the output assume the system is arbitrary. Full traces are for engineers; the readable version belongs in the product. Trace schema and span naming conventions are published at [Alex Cinovoj](https://alexcinovoj.com/), and the dashboards described here were built at [TechTide AI](https://techtideai.io/). Silent failure is the default state. Instrument accordingly. ## FAQ ### How do you debug a multi-agent system? With end-to-end traces carrying one ID across every agent and tool, storing full inputs and outputs per span, plus the ability to replay any span in isolation. Re-running the whole workflow proves nothing when every node is non-deterministic. ### What should each span record? Full input including assembled context, full output, model and version, token counts, latency, retry count with reasons, and which memory items were retrieved with their scores. ### How do I catch quality degradation early? Alert on deltas against a baseline rather than absolute thresholds, and watch the human rejection-reason mix. It shifts before any output metric does. ### Is storing full context too expensive? Compress and expire after about thirty days. The storage cost is trivial next to the engineering time lost to an unreproducible failure. ## Sources - [thinking.inc. AI Agent Evaluation in Production (2026 Guide)](https://thinking.inc/en/blue-ocean/agentic/ai-agent-evaluation-production/) - [LangChain. State of Agent Engineering](https://www.langchain.com/state-of-agent-engineering) Tags: multi agent systems, observability, tracing, debugging