Agent engineering
Multi-Agent Systems Fail Quietly
A single agent fails with an error. A multi-agent system fails by producing something slightly worse than usual, for three weeks, and nobody notices.
Key takeaways
- Trace every run end to end with one ID across all agents and tools.
- Log inputs and outputs at every boundary, not just at the ends.
- Degradation, not failure, is the dominant production failure mode.
- Replayability is the difference between a fix and a guess.
Distributed systems taught us that a request touching five services needs a trace ID. Multi-agent systems are distributed systems with an additional property: every node is non-deterministic, so you cannot reproduce a failure by re-running the input. That makes observability not a nice-to-have but the only way to fix anything.
The trace model
One trace per run. One span per agent invocation and per tool call, nested. Each span carries:
- Full input, including the assembled context, not a summary, the actual bytes.
- Full output, including refusals and malformed responses.
- Model identifier and version, because "we upgraded the model" explains a startling share of mysteries.
- Token counts in and out, and latency.
- Retry count and the reason for each retry.
- Which memory items were retrieved and their relevance scores.
The context bytes are the expensive one to store and the one you will always wish you had. Compress it, expire it after thirty days, but store it. A trace without the input is a timestamp with ambitions.
Degradation is the real enemy
Hard failures are easy: something throws, something alerts, someone fixes it. The dangerous mode is a system that keeps working while getting worse, a tool quietly returning stale data, a prompt truncated by growing context, a retry policy masking a 40% failure rate.
Detecting that requires metrics with baselines and alerts on deltas rather than thresholds:
- Tool success rate per tool, alerting on a drop of more than three points week over week.
- Mean and p99 steps per run, a rising p99 means stopping conditions are eroding.
- Context utilization as a fraction of budget, alerting above 80%.
- Retry rate per step. This is the metric that hides the most.
- Human rejection reason distribution. The earliest signal available, because reviewers notice before dashboards do.
Replay is the whole point
The capability worth building: take a trace ID, reconstruct the exact inputs to any span, and re-run that span in isolation against a modified prompt or a different model. That turns a vague quality complaint into a controlled experiment.
It also makes the golden set self-populating. Any production run that went wrong becomes a test case with real inputs attached, which is exactly the material evals need and the material teams usually lack.
Every unexplained bad output is either a test case or a recurring incident. The trace decides which.
Cost belongs in the trace
Attach spend to spans rather than to a monthly total. It answers the question that matters (which step is expensive) and makes optimization an engineering task instead of a finance argument. In our system one critique step was 40% of total cost until the trace made it obvious; smaller model, no measurable quality change, budget recovered.
Make it visible to users
A simplified trace view (what the agent looked at, what it decided, what it produced) is a trust feature as much as a debugging tool. Users who can see the reasoning path forgive a bad output; users who see only the output assume the system is arbitrary. Full traces are for engineers; the readable version belongs in the product.
Trace schema and span naming conventions are published at Alex Cinovoj (opens in a new tab), and the dashboards described here were built at TechTide AI (opens in a new tab). Silent failure is the default state. Instrument accordingly.
“If you cannot replay a run, you are not debugging. You are re-rolling the dice and hoping for a different story.”
Questions people actually ask
- How do you debug a multi-agent system?
- With end-to-end traces carrying one ID across every agent and tool, storing full inputs and outputs per span, plus the ability to replay any span in isolation. Re-running the whole workflow proves nothing when every node is non-deterministic.
- What should each span record?
- Full input including assembled context, full output, model and version, token counts, latency, retry count with reasons, and which memory items were retrieved with their scores.
- How do I catch quality degradation early?
- Alert on deltas against a baseline rather than absolute thresholds, and watch the human rejection-reason mix. It shifts before any output metric does.
- Is storing full context too expensive?
- Compress and expire after about thirty days. The storage cost is trivial next to the engineering time lost to an unreproducible failure.
Sources & further reading
Written by
Alex Cinovoj, Founder, TechTide AI
Alex Cinovoj builds ClawLI at TechTide AI, a fleet of specialist agents that research, draft, schedule, and grade LinkedIn content with a human approving every send. He writes about what breaks when you put agents in front of a real audience.
Keep reading
- Agent engineeringPre-Flight: The Checks That Run Before an Agent PublishesNine deterministic checks that catch the errors a model will never catch about itself, running in the last second before publish.
- Agent engineeringEvals or It Didn’t ShipLLM evals for agents that write, when there is no correct answer. Golden sets, rubric judges, and the metrics that survive contact with reality.
- Agent engineeringYour Agent Costs 10x More in Production Than in the DemoWhere the money actually goes when an agent leaves the prototype: retries, context growth, judge models, and the human review nobody budgeted.