Shipping LLM features that survive contact with real users
The demo takes an afternoon. That is the problem. An afternoon of work produces something convincing enough that it gets promised to customers, and the distance between that and a feature you can leave running is roughly six weeks of unglamorous engineering.
The gap is not model quality. Models are good. The gap is that a demo is measured by whether it impressed the room, and a feature is measured by what it does on the worst input a real user supplies on a bad day, at the ninety-fifth percentile of latency, at a cost per request that has to make sense against what the customer pays you.
There is also a failure mode specific to this kind of work: the feature appears to function, so nobody investigates it. A search that returns plausible results and a summariser that produces readable prose both look fine from the outside. Whether they are right is a separate question, and it is not one you can answer by using the product yourself a few times.
Five things close it. None are exotic.
Write the eval set before the feature
Sixty cases, in a spreadsheet, each with an input and a description of what an acceptable answer contains. Thirty ordinary, twenty awkward, ten adversarial. It takes half a day and it converts every later argument about whether a prompt change helped into a number.
The cases matter more than the scoring. Write them from real inputs where you have them and realistic ones where you do not, and make the awkward twenty genuinely awkward: contradictory instructions, missing context, questions the system should refuse, inputs in the wrong language, and the specific things your users do that nobody designing the feature anticipated.
Without it, prompt engineering becomes taste, and taste does not survive three people editing the same prompt over two months. Teams that skip this step ship, then quietly stop changing the prompt because nobody can tell whether a change is an improvement.
Automated scoring is a nice-to-have rather than a prerequisite. Sixty cases reviewed by a person takes about forty minutes and is more reliable than a model grading itself. Automate once the set stops changing, not before, because early on the eval cases are still teaching you what the feature should do.
The eval set also becomes your regression suite when the model provider ships an update. Providers deprecate versions and change behaviour within a version, and a feature tuned against one snapshot can degrade without a line of your code changing. Running sixty cases against the new version takes twenty minutes and is the only way to find out before your users do.
Budget for the ninety-fifth percentile
Average latency is a comforting number that describes nobody's experience. A retrieval-backed answer that averages 1.8 seconds routinely has a p95 above six, because the slow path involves a long document, a cold cache, a retry, and a model that decided to be thorough.
The tail also has a shape worth understanding. It is not slowness distributed evenly; it clusters on the inputs that matter most, because long documents, complex questions and unusual phrasing all take longer and all correlate with the user caring about the answer. Your most engaged users get your worst latency.
Decide the budget as a product constraint, then spend it deliberately. Streaming buys a great deal of tolerance, because the perceived wait ends at the first token rather than the last. Caching the embedding step is usually free. Trimming retrieved context is the single most effective lever and the one teams reach for last.
Trimming context works because most retrieval pipelines return far more than the model needs. Twenty chunks feels safer than six, and it is slower, more expensive and frequently less accurate, since the relevant passage is now competing with nineteen distractors. Measuring accuracy against chunk count on your eval set usually finds a smaller number that performs better on all three axes.
One structural point: put the budget in the monitoring, not only in the design document. A p95 checked at build time and never again will drift as the corpus grows and the prompt accretes instructions. Alerting on it is the difference between noticing in a dashboard and noticing in a support ticket.
Constrain the output, then validate it
If the result of a model call enters your database, drives a UI state, or triggers an action, it should be structured output validated against a schema before your code touches it. Free prose is fine for something a human reads and judges. It is not fine as an input to logic.
Schema validation catches mechanical failures. It does not catch a well-formed answer that is wrong, which is why the validator should check semantics where it cheaply can: that cited document identifiers exist, that a returned date falls in a plausible range, that a referenced product is one you actually sell. These checks are unglamorous and they catch the errors users notice.
Validation also gives you a clean place to put the decision about failure. A response that does not parse, cites a document that does not exist, or falls below a confidence threshold should be caught there rather than three layers deeper, where the only available behaviour is a 500.
It is also the right place to sample for review. Logging the input, the retrieved context, the raw output and the validation result for a small percentage of requests costs almost nothing and gives you the only honest view of how the feature behaves in production. Teams that skip this rely on users to report problems, and users mostly do not report a bad answer, they quietly stop using the feature.
Decide, in advance, what happens when it is wrong
It will be wrong. The question is only whether the wrongness is contained. Every LLM feature needs a written fallback ladder, and it needs to exist before launch, because the version invented during an incident is always worse.
The ladder should be visible to the user in proportion to the stakes. A summary that quietly falls back to the first paragraph needs no announcement. A recommendation that could affect money should say it is unavailable rather than silently degrade, because a user who does not know they got the fallback will act on it as though it were the real thing.
The kill switch at step four deserves emphasis. It must not require a deploy, because the moment you need it is the moment your team is arguing about whether the metric is real. A configuration flag anyone on call can flip, with a documented threshold, converts a potential incident into a five-minute decision.
Every LLM call is a variable-cost, variable-latency, occasionally-wrong network request. Treat it as one and most of the architecture decides itself.
Cost belongs in the product decision
Cost per request is fixed in architecture, not in a later optimisation pass. Whether you retrieve six chunks or twenty, whether you summarise on write or on read, whether a background job re-embeds the whole corpus nightly: these are the decisions that set your unit economics, and they are made in the first week.
Cost is where the difference between a demo and a product shows up most starkly. A demo runs a few hundred requests. A product runs however many your users generate, and that distribution is heavily skewed: a small fraction of accounts will produce most of the inference spend, and they will be your most engaged customers, which makes rate-limiting them a commercially awkward conversation.
Work out the cost of a heavy user before the feature exists. If a customer on your $40 plan can trigger $9 of inference a month, that is a pricing conversation, not an engineering one, and it is much cheaper to have before launch.
There are usually three levers, worth pricing before choosing between them: a smaller model on the routine path with escalation to a larger one, caching at the semantic level rather than the exact-string level, and doing work on write rather than on read. The third is the most underused. Summarising a document once when it is uploaded costs a fraction of summarising it every time somebody opens it.
What the first sprint looks like
One narrow feature, end to end, behind a flag: eval set, retrieval, structured output, validator, fallback ladder, cost instrumentation. Not the full roadmap. The point of the first week is to establish the harness the rest of the work runs inside, because every subsequent feature reuses it and the second one takes a fraction of the time.
That is the honest reason to start narrow rather than with the full roadmap. The first feature pays for the harness. Everything after it inherits the eval loop, the validator, the fallback ladder and the cost instrumentation, and the marginal cost of an AI feature drops sharply once those exist.
