Cutting LLM cost in a pipeline without cutting quality

An LLM feature that looked cheap in a prototype gets expensive in the same three ways every time: it is called where code would do, it resends the same context on every request, and it uses one model for work of wildly varying difficulty.

Fixing those in order tends to remove most of the bill before anyone has to argue about quality.

Before optimising, delete

The cheapest call is the one you do not make.

Walk your pipeline and list every place a model is called. For each, ask whether the inputs fully determine the output. Routing by a field value, filtering a list, tallying, reformatting a date, validating a schema — these are code. We regularly find pipelines making a model call per item where a filter would do, paying for latency and variance to get a worse result.

This is not a micro-optimisation. In the two audits where we counted, per-item classification calls that a lookup table could answer accounted for the majority of the spend.

What is left after that pass is the work that needs a model, and that is what the rest of this is about.

Prompt caching is the lever with the best ratio

Most production prompts are mostly stable: a long system prompt, a tool schema, some retrieved context, and a short user turn that changes. Sending the stable part on every request means paying to process the same tokens repeatedly.

Prompt caching lets the provider reuse that prefix. Cache writes cost slightly more than ordinary input tokens; cache reads cost a fraction of them. For any workload where the stable prefix is large relative to the variable part, this is the single biggest reduction available, and it changes nothing about output quality because the input is identical.

Two things decide whether it works:

Order matters. Put everything stable first — system prompt, tools, few-shot examples, retrieved documents — and the variable user input last. A single changed token invalidates everything after it, so a timestamp near the top of a prompt silently disables the cache for the whole thing.

Traffic shape matters. Caches expire. A steady stream of requests keeps the prefix warm; a handful of requests per hour may pay the write premium and rarely get a read. Measure the hit rate rather than assuming it.

Instrument this explicitly. The API reports cache creation and cache read token counts per request; log them. A cache hit rate you are not measuring is a cache hit rate you do not have.

Route by difficulty, not by habit

Most systems pick one model and use it everywhere, which means the hardest request sets the tier for the easiest one.

Split the work:

• Classification, extraction, routing, short summaries — a small fast model handles these at a fraction of the cost, and on well-specified tasks the quality difference is often not measurable.
• Multi-step reasoning, debugging, code generation, anything where being wrong is expensive — the capable tier.

The trap is choosing tiers by intuition. Build the eval set first, run both tiers against it, and look at where the cheaper one degrades. Sometimes it does not, and sometimes it fails on exactly the 5% of cases that matter most. You cannot tell without measuring, and evaluating without relying on vibes is its own discipline.

A related lever on current models: reasoning effort is configurable per request. Classifying an alert does not need the same thinking budget as working out why a rollout is stuck across three controllers. Setting effort per route rather than globally is close to free and frequently overlooked.

Batch everything that is not waiting for a human

If nobody is watching a spinner, the request does not need to be synchronous. Batch processing on most providers runs asynchronously at a substantial discount, typically around half price, with results returned within a window measured in hours.

Good candidates in a delivery pipeline:

• Nightly summarisation of the day’s incidents
• Bulk classification of a backlog
• Regenerating documentation across a monorepo
• Anything triggered by a cron rather than a click

Bad candidates: anything in a request path, anything an engineer is blocked on.

Two smaller levers worth knowing

Cap the output. Output tokens cost several times more than input tokens. A max_tokens that reflects the actual expected answer, plus a prompt that asks for the format you want, stops a model from producing three paragraphs where you needed a label. This is the cheapest change on this page.

Stop retrying blindly. Retry logic that re-sends the full request on any failure can double spend during an incident. Retry on transient errors, with backoff, and not on a response that was merely unsatisfying.

Measure per unit of work, not per month

A monthly total tells you nothing about whether the system is getting better or just busier. Track cost per unit of the thing you care about — per incident triaged, per PR reviewed, per document processed — and log input, output, cache-read and cache-write tokens per call site.

With that in place the conversation changes from “the AI bill went up” to “the review path costs 40% more per PR since we added the second retrieval step”, which is a question someone can act on.

The economics that decide the whole thing

Nothing above matters if the feature is not worth having. Before tuning the cost of a pipeline stage, check that the stage earns its place: what does it replace, what would happen if it were removed, and can anyone tell when it produces a bad answer?

We have seen more money saved by deleting a step nobody could justify than by any amount of caching. That is not an argument against optimising — it is an argument for doing it in this order.