How is AI product quality actually assured? Making eval maturity a technical DD axis
The conclusion upfront: unit and integration tests alone cannot guarantee the quality of a product built on LLMs. Outputs are probabilistic and rarely have a single “correct” answer, so the role of quality assurance falls to a dedicated evaluation system — evals. Checking whether an eval system exists, and how mature it is, is a due diligence axis in its own right: it reveals not just AI-specific incident risk, but whether an AI product is genuinely operated with discipline. An organization that has no evaluation dataset, relies on gut feeling to judge output quality, or ships to production without running evals will run into quality problems the moment it scales.
Why traditional tests break down for LLM products
Traditional software testing defines a single “expected output” for a given input and checks for a match. That premise collapses once an LLM sits in the request path.
Outputs are probabilistic. Sending the same prompt twice can return differently worded outputs, depending on temperature settings and sampling. A test that checks for an exact string match simply doesn’t work.
There is no single correct answer. Tasks like summarization, classification, and dialogue generation admit multiple valid outputs. Judging “is this summary correct?” requires scoring logic that approximates human judgment — a plain assertion isn’t enough.
The input space is effectively unbounded. User-supplied prompts can’t be enumerated in advance. Traditional testing assumes you can list expected input patterns and cover them; LLM products require a different mindset — sampling representative scenarios and scoring them instead.
Regressions are hard to detect. A model version bump, a small prompt tweak, or a change to RAG retrieval logic can all shift production behavior in unpredictable ways. Where traditional CI could declare “all tests passed, therefore safe,” an LLM product needs a quantitative measurement of a change’s impact before anyone can make that claim.
For these reasons, LLM products need a dedicated quality-assurance layer distinct from unit testing. That layer is evals.
What evals are: three evaluation methods
Evals is the umbrella term for systems that score LLM outputs — quantitatively or qualitatively — and continuously measure how a change affects quality. In practice, teams combine three broad methods.
| Method | What it does | Strengths | Limitations |
|---|---|---|---|
| Offline evaluation | Scores outputs in batch against a golden dataset of expected input/output pairs | Runs pre-deploy; makes before/after comparisons easy | Overstates accuracy if the dataset drifts from real user distribution |
| Online evaluation | Samples a slice of live production traffic and scores it in real time or near-real time | Continuously tracks quality against actual user input | Scoring itself has a cost; needs latency and scoring-logic infrastructure |
| Human evaluation | Human raters — internal staff, domain experts, or crowd workers — score outputs directly | Captures qualities automated scoring struggles with (contextual appropriateness, tone) | Costly and slow; requires managing inter-rater reliability |
Automated scoring implementations span a range too. String matching and rule-based checks (e.g., verifying no banned terms appear in the output) are cheapest but narrow in scope. Using a separate LLM as the grader — “LLM-as-a-judge” — enables more flexible evaluation, but introduces its own bias and cost as a new variable. Mature organizations combine these depending on the task.
This connects to the choice discussed in architectural decision axes for embedding LLMs in a product around fine-tuning, prompting, and RAG: whichever approach a team picks, without an evaluation dataset to measure the effect of a change, all you’re left with is a subjective sense that “it got better.” Evals are the AI-development equivalent of a test suite — foundational infrastructure, not an optional extra.
The eval maturity model: no eval data → periodic evaluation → release gate
How far a team has taken its evals varies enormously. In practice, DD conversations get much clearer once you place an organization on a three-stage maturity model.
Stage 0 (no eval data): Most AI products start here. Relying on eyeballing output during the prototype phase isn’t unreasonable in itself, but staying at this stage after paid usage has begun is a red flag worth treating as such. As with the gap between a working demo and a shippable product discussed in the tech-debt risk of AI-generated code, “it works” and “we can guarantee its quality” are two different claims.
Stage 1 (periodic evaluation): A golden dataset exists, and score trends are tracked after model or prompt changes. Most organizations stall here. Evaluation is happening, but the results don’t feed directly into release decisions — evals exist but aren’t functioning as a gate. The point made in the context of quality gates for small teams — that building a mechanism and making it actually work are two separate problems — applies equally to evals.
Stage 2 (release gate): The evaluation pipeline is wired into CI, and any change that drops the score below a threshold is automatically blocked. Reaching the point where production feedback — low ratings, correction requests — continuously flows back into the evaluation dataset completes evals as a functioning quality-assurance system.
Questions and evidence to check in DD
Judging eval maturity from a verbal answer alone risks misreading what “we evaluate our model” actually means in practice. The following questions, paired with the evidence that backs them, let you place a team objectively on the Stage 0–2 scale.
| Question to ask | Evidence to request | What to look for |
|---|---|---|
| Does a golden dataset exist? | The dataset itself (item count, coverage) | A few dozen examples is often too thin to be representative |
| Who updates the dataset, and when? | Change history (Git log, spreadsheet revision log) | No update in the last six months signals it has gone stale |
| Who runs evaluations, and how often? | Execution logs, Slack notifications, a dashboard | ”Whenever someone remembers to” doesn’t count as periodic |
| Is there a record comparing scores before/after model or prompt changes? | A table mapping changes to score deltas | No record likely means regressions go undetected |
| Is there an actual instance where a failing eval blocked a change? | CI logs, a rolled-back deployment record | Zero blocked releases can mean thresholds are too loose, or the gate isn’t actually enforced |
| Where does negative production feedback get collected? | The feedback-collection mechanism and how it feeds back into the dataset | If it only lives in someone’s notes or a Slack thread, it never becomes organizational knowledge |
These questions build on the “existence of an eval system” dimension raised in how AI changes technical due diligence — this is the deeper follow-up to that check. The single most telling question is whether there’s an actual instance of a failing eval blocking a change. Most teams will claim an eval pipeline “exists”; whether it has ever actually stopped something is what separates Stage 1 from Stage 2.
When an eval system becomes a competitive advantage
Building evals isn’t only about quality assurance — under certain conditions it becomes a source of competitive advantage.
Domain-specific evaluation criteria. Teams that build evaluation criteria tailored to their own use case — detecting contraindicated language in a medical context, verifying numerical accuracy in a financial context — rather than relying on generic benchmarks can credibly claim a quality edge over competitors that use a general-purpose LLM as-is.
A dataset that keeps growing from production feedback. Teams with a structural pipeline feeding user corrections and low ratings back into the evaluation dataset see their evaluation accuracy improve over time. That’s an advantage rooted in accumulated data assets — one that’s hard for a fast-follower to replicate in a short window.
Optimized evaluation cost. Teams that use sampling strategies or caching to bring down the cost of expensive methods like LLM-as-a-judge can afford to run evaluations more frequently, which shortens their release cycle. An eval system is simultaneously a quality-assurance mechanism and a potential bottleneck on development velocity, so it’s worth looking at this cost structure too — it changes how deep your evaluation of the eval system itself should go.
Conversely, if the evaluation criteria are limited to generic public benchmarks, treat that as easily replicated and a weak claim to competitive advantage. In DD, the useful question isn’t just “do evals exist?” but “could a competitor reproduce this specific eval system without much effort?”
Summary
Because LLM outputs are probabilistic and rarely have one correct answer, traditional unit testing cannot guarantee an LLM product’s quality. Checking eval maturity — offline evaluation, online evaluation, and human evaluation — against a three-stage model (no eval data → periodic evaluation → release gate) gives technical DD a practical axis for evaluating AI products.
Rather than a yes/no question about whether evals exist, push further: is there an actual instance of a failing eval blocking a release, and does production feedback flow back into the dataset? That’s how you close the gap between what a team says and what it actually does. Domain-specific evaluation criteria and a continuously replenished feedback loop are also worth evaluating as a source of advantage that’s genuinely hard to copy.
If you’re an investor looking to build evals into your technical DD framework, see TiedPro for investors; if you’re a startup that wants hands-on support building an eval system, see TiedPro for startups.
FAQ
How do evals differ from test code (unit and integration tests)?
Test code checks for an exact match or a clear rule-based pass/fail when the “expected output” for a given input is unambiguous. Evals quantify quality for tasks — like LLM output — where there’s no single correct answer, using scoring criteria such as rule-based checks, LLM-as-a-judge, or human review. The two aren’t mutually exclusive: use test code for the deterministic parts of a system and evals for the probabilistic parts.
Should an early-stage startup build all the way to a release gate (Stage 2)?
It depends on the stage of the business and the blast radius of a production issue. If you’re still validating PMF and shipping frequently, a full release gate can slow you down more than it helps. A realistic sequence is to start with periodic evaluation (Stage 1) against a golden dataset of 10–20 examples, then move to a release gate once paid usage or a high-sensitivity domain (healthcare, finance, etc.) goes live in production.
How many examples does an eval dataset need?
There’s no fixed number — it depends on task complexity. What matters more than raw count is how well the dataset covers failure patterns that have actually occurred in production. A smaller dataset that’s continuously updated with recent incidents and negative user feedback will produce more useful evaluation signal than a larger one that hasn’t been touched in months.