The short version
Treat evaluation as a product surface: a frozen golden set per workflow, automated judges you have validated against humans, guardrails as code, and post-deployment drift monitoring on inputs, outputs and human overrides.
Traditional testing assumes a specified output for a given input. Generative systems break that, so teams often skip straight from eyeballing to production. The fix is not exhaustive testing; it is deciding what evidence you require before a release and automating exactly that. NIST's AI Risk Management Framework frames this as continuous measurement rather than a one-time validation, which is the right posture.
The golden set
- —200 to 1,000 real cases per workflow, labelled by someone who does the job.
- —Deliberately skewed to hard cases: ambiguous documents, out-of-scope questions, adversarial phrasing, minority classes.
- —Frozen and version-controlled. New failures become new cases; existing cases do not get quietly edited to pass.
- —Split into a CI subset that runs on every change and a fuller set that runs before release.
Judges you have checked
Model-as-judge scoring is practical, but an unvalidated judge is just another opinion. Calibrate it: have humans score a sample, measure agreement, and report the judge's error alongside its verdict. For grounded systems, prefer decomposable metrics — retrieval relevance and answer faithfulness measured separately, as in the RAGAS approach — over a single vague quality score.
Guardrails belong in code
| Guardrail | Implementation | Failure it prevents |
|---|---|---|
| Schema validation | Reject and retry non-conforming output | Downstream systems ingesting garbage |
| Grounding threshold | Refuse below citation coverage limit | Confident unsupported claims |
| Scope classifier | Route out-of-scope requests to a human | Answering questions you never tested |
| Cost and loop caps | Hard limits per request and per agent run | Runaway spend and retry storms |
| Human-in-the-loop band | Confidence range routed to review | Expensive errors on borderline cases |
Drift is detected on three signals
- 01Input drift: new document layouts, new phrasing, changing language mix, volume shifts by class.
- 02Output drift: refusal rate, citation coverage, answer length distribution, field-level extraction confidence.
- 03Human drift: override and edit rates per field or per case class — the earliest and most honest quality signal you have.
Alert on those, not on model version changes. A hosted model can shift underneath you without a version bump, and a rising override rate will tell you before anyone files a ticket.
Every incident should end by adding cases to the golden set. A suite that only grows when someone remembers to grow it is a suite that stops representing production within a quarter.
What we would do in a pilot
A golden set built with your domain experts, a validated judge with measured human agreement, guardrails implemented as code, and drift dashboards on inputs, outputs and human overrides — handed over as yours to run.
Sources
- 01Artificial Intelligence Risk Management Framework (AI RMF 1.0) — NIST
- 02AI Risk Management Framework resources — NIST AIRC
- 03RAGAS: Automated Evaluation of Retrieval Augmented Generation — arXiv
Figures drawn from our own engagements elsewhere on this site are illustrative placeholders. Anything attributed above links to its public source.