Agent engineering
Evals or It Didn’t Ship
Nobody argues against evaluation. Almost nobody does it, because the first question you hit, "what is the right answer here", has no good reply for generative work.
Key takeaways
- Start with 30 real cases and your own rejections, not a public benchmark.
- Judge against a rubric with named criteria; a 1-10 "quality" score is noise.
- Track process metrics (tool success, stop rate, context utilization) alongside output quality.
- The eval that matters is agreement with your reviewer, not agreement with a model.
Deterministic software has assertions. Agents have opinions, which is why so many teams ship on vibes for eighteen months and then discover their quality has been sliding since March.
Agent evaluation is not testing. It is measuring whether probabilistic, multi-step, tool-using behaviour produces useful outcomes under real conditions, a framing several 2026 guides converge on (thinking.inc (opens in a new tab)). For anything generative, the hardest part is admitting there is no ground truth and building a standard anyway.
Build the golden set from your own rejections
Public benchmarks tell you how a model does on someone else’s problem. What you need is thirty cases from your own history: fifteen outputs you were proud of, fifteen you killed, each with the reason attached.
If you instrumented your approval flow, you already have this. If not, it is an afternoon of scrolling and honesty.
Rubric judges, not vibe scores
Asking a model to rate quality 1-10 produces a number that clusters at 7 and moves with the weather. Ask about specific properties and it becomes usable.
- Evidence: does the piece contain at least one first-party, checkable fact? (binary)
- Claim: is there a position a competent peer could disagree with? (binary)
- Voice: does vocabulary and cadence match the profile? (0-2)
- Slop: any banned constructions, hedged non-claims, or symmetry tells? (count)
- Attribution: is every borrowed statistic linked to a source? (binary)
Binary criteria are the important part. They are far more reproducible across runs than any scale, and they map directly onto actions an engineer can take.
Validate the judge before you trust it
A judge model that disagrees with your human reviewer is a random-number generator with a good vocabulary. Run the judge over fifty already-reviewed items and compute agreement. Below about 80%, fix the rubric. The failure is almost always an ambiguous criterion, not a weak model.
This is also where evaluation cost shows up. Judge inference, harness engineering, human validation, and re-validation as models drift are all real line items (ContextQA (opens in a new tab)). Budget for them or the eval program quietly dies in month three.
Process metrics catch what output metrics miss
Output quality can hold steady while the machinery rots. Track the mechanics separately:
- Tool success rate, per tool. A tool degrading from 98% to 91% is an incident nobody paged you about.
- Steps per run, p50 and p99. Rising p99 means stopping conditions are failing.
- Context utilization against budget. Creeping toward the ceiling predicts a quality cliff.
- Cost per accepted output. The one number that ties quality and spend together.
- Rejection reason distribution. A shift in the mix is the earliest available warning.
Output quality tells you how you are doing. Process metrics tell you what is about to break.
Run it where it counts
The golden set runs in CI on every prompt, model, or context change, and a change that drops any binary criterion does not merge. In production, sample 5% of live runs through the same judge and alert on drift. That combination catches both the regression you shipped and the drift you did not.
None of this is exotic. It is testing, adapted to a system that answers differently every time. Related: multi-agent systems fail quietly.
The eval harness we use is written up at alexcinovoj.com (opens in a new tab), and the graded datasets behind it were assembled at TechTide AI (opens in a new tab). Start with twenty examples. Twenty beats zero by an absurd margin.
“You cannot grade a post against a correct answer. You can grade it against your own standard, if you ever wrote your standard down.”
Questions people actually ask
- How do I evaluate writing when there is no right answer?
- Stop looking for correctness and encode your standard as named, mostly binary criteria, evidence present, position present, voice matched, no banned constructions. Reproducible criteria beat holistic scores every time.
- Can I use a model to judge another model?
- Yes, with validation. Measure the judge against human reviews on fifty known cases and only trust it above roughly 80% agreement. An unvalidated judge is worse than no judge because it is confidently wrong.
- How big should a golden set be?
- Thirty real cases is a working start. Grow it by adding every production failure that surprised you; that keeps the set aligned with how your system actually breaks.
- How often should evals run?
- On every change to prompts, models, tools, or context assembly, plus a 5% sample of live traffic continuously, so drift shows up before a user reports it.
Sources & further reading
Written by
Alex Cinovoj, Founder, TechTide AI
Alex Cinovoj builds ClawLI at TechTide AI, a fleet of specialist agents that research, draft, schedule, and grade LinkedIn content with a human approving every send. He writes about what breaks when you put agents in front of a real audience.
Keep reading
- Agent engineeringMulti-Agent Systems Fail QuietlyMulti-agent systems degrade without erroring. Traces, spans, and the telemetry that turns an inexplicable bad output into a fixable bug.
- Agent engineeringPre-Flight: The Checks That Run Before an Agent PublishesNine deterministic checks that catch the errors a model will never catch about itself, running in the last second before publish.
- Agent engineeringYour Agent Costs 10x More in Production Than in the DemoWhere the money actually goes when an agent leaves the prototype: retries, context growth, judge models, and the human review nobody budgeted.