Shipping LLM Features to Production: 12 Lessons from Real Projects

The first LLM feature I pushed to production passed every demo and wobbled in its first week. Nothing crashed. The model simply answered with a confidence my validation never checked, in a format my parser half-expected, at a cost nobody had budgeted for. Those early LLM production lessons were expensive in the only way that counts: wasted weekends. Everything below comes from shipping AI features into real products since then, written for the engineers who have to keep the thing running, not just get it past the kickoff.

I'm Sameer Ahmad, an Applied AI Engineer based in Dubai with 7+ years of software development experience. I work mostly in Python and JavaScript across LLM pipelines, AI agents, workflow automation, and the full-stack apps around them. Twelve lessons is a lot of bullets, so I've grouped them the way I actually think about a release: before the first line of code, while the feature is being built, and after it's live.

Scoping: what to decide before you write code

1. Start with a task, not a model

Define the job the feature does and what "good enough" looks like for that job, then choose a model. Teams that start the other way around pick an expensive frontier model, discover the task was easier than they feared, and spend the next year paying for that decision. Write acceptance criteria first: what a correct response looks like, what an unacceptable one looks like, and who notices when it goes wrong.

2. Ship the smallest model that clears the bar

Cheaper, faster models are not a compromise — they are usually the right answer. Most production LLM work (extraction, classification, formatting, summarizing structured records) sits comfortably on a mid-tier model with a tight prompt. Reserve frontier reasoning for the steps that genuinely need it. Benchmark candidates against your own examples, not against leaderboard screenshots.

3. Treat prompts as versioned code

Prompts belong in the repository, not in a Slack thread or a vendor dashboard nobody owns. Every change gets a diff, a review, and an eval run. I keep the system prompt, few-shot examples, and output schema together as one versioned unit so I can tell which combination produced a regression. If a prompt can only be edited through a web UI, you cannot reproduce last Tuesday's behavior — and one day you will need to.

4. Decide your failure mode before launch

Define what happens when the model returns nothing, returns junk, or takes thirty seconds: fall back to a deterministic path, show a partial result, or fail loudly. My default is boring — degrade to a non-LLM flow and record the event. Products rarely die from an occasional bad generation; they die from a spinner that never resolves and a UI with no story for it.

Building for failure: evals, structure, and observability

5. Write evals before the feature ships

Twenty realistic examples, scored automatically, beat two hundred unit tests aimed at code that no longer decides the outcome. Build a small eval set from real inputs the moment you have a draft prompt — including the ugly ones: empty fields, mixed languages, users who ignore instructions. Run it on every prompt change. It is the only way to know a change helped instead of merely changed something. My RAG evaluation guide goes deeper on scoring design if retrieval is involved.

6. Never accept free-form text where you need structure

Ask for JSON, validate the schema, and retry once with the validation error appended to the message. That single loop eliminates most "the API works but the parser fails" bugs. Keep structured output enabled for anything a program consumes downstream. When a model writes prose where your code expects data, the error surfaces three services away, in a place with no context about what went wrong.

7. Latency is a product feature — stream it

Users forgive a long answer they can watch arriving; they do not forgive a blank screen. Stream tokens into the UI, show progress on multi-step agent work, and put a hard timeout on every call. Also set a ceiling on iterations: an agent that can loop forever is one ambiguous instruction away from a bill nobody anticipated.

8. Log prompts, outputs, tokens, and cost from day one

Not "we'll add observability later." When something feels off, you need to answer four questions fast: which prompt version, how many input and output tokens, how long it took, and what the user actually saw. Structured logs with a request ID that ties your application log to the provider's response ID turn "the AI is being weird" into a reproducible ticket.

After launch: cost, drift, and human oversight

9. Cache the parts of the prompt that never change

System prompts, tool definitions, reference documents, and few-shot examples are identical on every request — exactly what prompt caching is built for. Split the prompt so stable content sits at the front and only the user turn varies. It is the cheapest performance and cost win available to most teams, and one I lean on heavily in day-to-day work.

10. Watch for silent model drift

Providers update model versions behind the same alias more often than they used to. Your eval set is the alarm: run it on a schedule against the live model, not just against staging. Pin a version for production and promote upgrades deliberately. I have seen outputs shift after a "no-op" provider change, and only a scheduled eval run made it visible.

11. Put a human in the loop for irreversible actions

Drafts, summaries, classifications, suggestions: ship those autonomously. Emails sent, records deleted, payments issued, data written into another system: require a human click first. This is not caution for its own sake — it is the difference between a model error being a draft sitting in a queue and a model error being an incident. Start with review everywhere, then remove it where the evals have earned it.

12. Measure cost per task, not cost per token

A token price tells you nothing about whether a feature is viable. What matters is what one completed user task costs end to end, including retries, tool calls, and the re-generation the user triggered because the first answer was mediocre. Track that number per feature and per prompt version. It is the metric that tells you whether caching, a smaller model, or a shorter context is worth doing next.

How I would start a new LLM project this week

If I were bootstrapping a feature tomorrow, the sequence would be: write twenty acceptance examples, pick the cheapest model that passes them, version the prompt, enforce a schema, add a timeout, log everything under one request ID, and place a human review step in front of any action that cannot be undone. Caching, routing, and agent orchestration are optimizations layered on top of that foundation — useful, but not the foundation.

That blend of AI engineering and full-stack delivery is most of what I do day to day, and the shipped results are visible on my portfolio.

FAQ

How long does it take to ship an LLM feature to production?

A focused feature with a clear task and a small eval set usually takes days to a few weeks, depending on how much of your existing app needs to change. The timeline stretches when there is no acceptance criteria, no eval set, or no owner for prompt changes — those three gaps cost more calendar time than the model integration itself.

What is the most common reason LLM features fail after launch?

Unbounded, unmeasured behavior: no schema validation, no timeouts, no logging, and no way to compare today's output with last week's. The model rarely breaks outright — quality drifts quietly while nobody is looking, and the team finds out from users.

Do we need a large eval set to start?

No. Twenty to fifty realistic examples with a simple pass/fail or rubric score are enough to catch most regressions and to compare prompt versions honestly. Grow the set as you collect real failures from production logs rather than inventing synthetic cases upfront.

Should we build on a hosted API or an open-source model?

Start with a hosted API unless you have a hard requirement — data residency, air-gapped deployment, or per-token economics at real volume. Hosted models get you to evals and iteration fastest, and switching later is much easier than it looks once your prompt and schema are versioned behind your own adapter.

If you are deciding whether an LLM feature is worth building at all, or you want a second opinion on one already in progress, get in touch and I will tell you plainly what I would do.