LLM Cost Control: Prompt Caching and Token Budgets in Practice

LLM cost control usually starts as a panic: someone opens the provider dashboard, sees a number they did not expect, and asks who broke something. In my experience nobody broke anything — the feature simply grew, context got lazy, and every request started carrying a system prompt the size of a small novel. The good news is that most LLM spend is not mysterious. It is structural, and structure can be changed.

I'm Sameer Ahmad, an Applied AI Engineer based in Dubai with 7+ years of software development experience, and inference cost is a conversation I have on almost every AI project. Below is how I actually control spend in production: where the money goes, why prompt caching is the highest-leverage lever you have, and how to set token budgets that survive contact with a real team.

Where does LLM spend actually go?

Before optimizing anything, split the bill into its real components. In most applications I audit:

• Input tokens on unchanged content. System prompts, tool definitions, few-shot examples, and retrieved documents are resent on every single call. This is usually the largest line item.
• Output tokens. Priced higher per token than input on most providers, and inflated by models that pad answers with preamble unless you constrain them.
• Retries and regenerations. Schema failures, user-visible "try again" taps, and agent loops that re-send the same context three times.
• A premium model used everywhere. One frontier model doing classification, extraction, and summarization work that a mid-tier model finishes correctly.
• Accumulated conversation history. Chat features that re-send the entire transcript on turn forty.

The rule I apply first is blunt: the cheapest token is the one you never send. Everything else — smaller models, shorter outputs, caching — comes after that.

Prompt caching: the highest-leverage cost control you have

Prompt caching is the single cheapest win available to most teams, because it discounts content you were already sending. Providers cache a stable prefix of the prompt and charge a fraction of the normal input rate when it is reused. The mechanism only works if the prefix is genuinely identical, so the prompt has to be shaped deliberately: stable content first, variable content last.

How I lay out a cached prompt:

1. System prompt and behavioral rules — identical across requests, always cached.
2. Tool definitions and schemas — static for the life of a release, cached.
3. Reference material and retrieved documents — cached while the document set is unchanged.
4. Few-shot examples — cached, provided they are not shuffled per request.
5. Conversation history — cached as a growing prefix where the provider supports it.
6. The current user turn — the only part that must be fresh, kept at the very end.

Two habits keep the cache hit rate high. First, avoid touching the stable prefix: a single reordered sentence at the top invalidates everything after it, so prompts are versioned and promoted rather than edited live. Second, remember that cache entries expire — if traffic to a prompt has a lull longer than the provider's TTL, the next request pays full price, which is normal and worth budgeting for rather than treating as a bug.

Prompt section	Cache it?	Why
System prompt, rules	Yes	Byte-identical on every request
Tool / function definitions	Yes	Changes only on release
Retrieved documents	Yes, while static	Large, repeated token blocks
Few-shot examples	Yes	Stable if ordered deterministically
Current user message	No	Must be generated fresh

What caching does not fix is a model choice made by default rather than by decision, or an output that runs three times longer than the task needs. Those are budget problems, and they need budgets.

Token budgets that survive contact with a real team

A budget nobody can breach by accident is a budget that lasts. I set three layers:

• Per-request caps. max_tokens set explicitly on every call, and a hard limit on how much conversation history is re-sent — sliding windows or summarization instead of an unbounded transcript. Truncate documents with a head-and-tail strategy rather than a blind character slice that drops the section you needed.
• Per-feature daily budgets. Each feature gets its own ceiling and its own alert threshold. When the summarizer blows through its budget at 2pm, you want to know which feature it is, not just that "spend is up."
• Budget as config. Limits, model assignments, and cache-friendly prompt versions live with the code, so a change to any of them is reviewed and diffed like everything else.

The discipline that makes this work is refusing to let cost be a surprise metric. If spend per feature is only visible in the provider console, the answer to "why did this cost more this month?" will always be a guess.

Model routing: pay premium prices only where they earn them

Cost control is not only subtraction; it is paying the right price for the right step. A pattern I use often is a cheap-first router: classification, extraction, and formatting go to a mid-tier model, and only low-confidence or failed results escalate to a stronger one. You pay frontier prices on the minority of requests that actually need frontier judgment.

The safeguard is measurement. Routing decisions are only defensible if you can show quality held — run the same eval set through both configurations and compare, rather than assuming the downgrade was free. And keep routing behind your own adapter: swapping models later should be a config change, not a rewrite. I covered the evaluation side of this in my LLM production lessons from shipping these features.

What to measure: cost per task, not cost per token

Token prices are inputs, not outcomes. The metric I put on a dashboard is cost per completed user task — end to end, including retries, tool calls, and regenerations. Around it I track four supporting numbers:

1. Cache hit rate per prompt version — if it drops after a release, something in the stable prefix moved.
2. Input vs. output token ratio per feature — an unusual shift usually means context growth or chatty responses.
3. p95 latency and timeout rate — cheaper paths that fail often are not cheaper.
4. Retries per request — schema failures are a cost problem disguised as an engineering one.

When those four are healthy, the bill stops being an event. It becomes a number that moves for reasons you can name.

How I approach cost on a new engagement

On a fresh project I do the boring version first: cache the stable prefix, cap output tokens, route the simple steps to a cheaper model, and log tokens per request from day one. Only after that do I look for clever optimizations — and most of the time, the boring version has already removed the bulk of the waste. The rest of my work, including the systems these budgets live inside, is on my portfolio.

FAQ

How much can prompt caching actually reduce LLM costs?

For features with a large static prefix — long system prompts, tool definitions, reference documents — cache reads are typically billed at a small fraction of normal input rates, so the savings scale with how much unchanged context you resend. The exact discount is provider-specific, which is why I measure cache hit rate alongside spend rather than assuming it.

Does prompt caching change output quality?

No. Caching affects how input tokens are billed, not how the model interprets them — the model receives the same prefix either way. Quality changes come from editing the prompt, so the risk to manage is prompt churn invalidating the cache, not caching itself.

Where should I start if our LLM bill is too high?

Start by logging tokens per request for one week, then attack the three biggest levers in order: stop resending unchanged context, set explicit output limits, and move non-critical steps to a cheaper model. Those three account for most of the waste I find in audits.

How often should we review LLM costs?

Monthly for a stable feature, and immediately after any model, prompt, or context-length change. Tie the review to the same cadence as your eval runs so you are checking spend and quality together — a cost cut that quietly degraded output is not a cost cut.

Fixing spend is usually a week of focused work, not a quarter. If your inference bill has outgrown the feature it funds, get in touch and I will tell you where I would cut first.