The short version
Measure cost per workflow first, then attack in order: context size, caching, model routing, batching, and finally a distilled small model for the highest-volume step. Every change must pass a quality gate, or you have not saved money — you have moved the cost to your users.
When a client shows us a six-figure annual model bill, the invoice never says where it went. The first useful piece of work is attribution: cost per workflow, per request type, per tenant. Almost always, one or two workflows dominate, and one of them is doing something avoidable.
The leaks, in the order we look for them
- 01Context bloat. Whole documents pasted where three retrieved passages would do. This is the single largest line item we find.
- 02Uncached repetition. Long, stable system prompts and instructions resent on every call. Both OpenAI and Anthropic document prompt caching that discounts repeated prefixes heavily.
- 03One model for everything. Frontier pricing applied to classification and extraction work a much cheaper model handles at equal quality.
- 04Synchronous work that isn't urgent. Overnight enrichment and backfills run through real-time endpoints instead of the discounted batch paths both vendors publish.
- 05Retry storms and agent loops. Failures and unbounded tool loops paying twice or ten times for the same answer.
- 06No off-switch. Nobody owns the number, so nothing is measured or capped.
Routing needs a referee
Quality-gated routing is straightforward: score a request, send it to the cheapest model that clears the bar for that request class, escalate when it does not. What makes it safe is the evaluation set. Without a frozen set of labelled cases, 'the cheaper model is fine' is an opinion, and the first customer complaint reverses the whole programme.
| Lever | Typical effort | What it depends on |
|---|---|---|
| Trim retrieved context | Days | Retrieval quality — trimming a bad index makes answers worse |
| Prompt caching | Days | Stable prefix ordering; vendor cache TTL and write costs |
| Batch the non-urgent | Days | Tolerating asynchronous completion |
| Quality-gated routing | Weeks | A frozen evaluation set per request class |
| Distil a small model | Weeks | Enough logged traces from the workflow to train and benchmark on |
About the 60% figure
We target a 60% or better reduction on the workflows we optimise, and we treat that as a claim to be proven on your own traffic rather than a guarantee. The pilot replays your real traces, reports cost and quality side by side per lever, and if the number does not land you keep the measurement harness and know exactly why.
Watch the second-order costs too. Caching changes prompt structure, batching changes user expectations, and a self-hosted small model trades API spend for GPU and operations. We cost the whole picture, not just the invoice line.
What we would do in a pilot
A cost-and-quality baseline per workflow from your own traces, then the levers applied in order with measured before-and-after on each — routing policy, cache design and, where volume justifies it, a distilled model you own.
Sources
- 01Prompt caching guide — OpenAI
- 02API pricing — OpenAI
- 03Prompt caching — Anthropic (Claude docs)
Figures drawn from our own engagements elsewhere on this site are illustrative placeholders. Anything attributed above links to its public source.