From Prototype to Production AI Product

Here is the thing nobody tells you when you finish your AI prototype: the hard part wasn’t the AI. The gap between “it works in my demo” and “it works for real users” is almost entirely an engineering and product problem. The model was never the bottleneck. What breaks in production is everything around it — the evaluation strategy you skipped, the prompts you never stress-tested, the latency that felt fine on your laptop, the cost curve that makes no sense at 10,000 users, the error handling you left as a TODO. That’s the work. And it’s a lot of work.

This is a guide for developers who already have a prototype and need to know what’s actually in front of them. It’s organized as a series of phases, roughly in the order you should tackle them. You don’t have to do all of this before launch, but you need to know which pieces you’re skipping and why.

Phase 1: Evaluation Before You Ship

You cannot harden something you can’t measure. Before you change another line of prompt, build an eval suite.

Start with a golden dataset — 50 to 200 representative inputs with documented expected outputs. These should come from your actual use case, not synthetic examples you invented. If you have beta users, this is the time to mine their sessions. If you don’t, construct them carefully and label them as ground truth.

Define acceptance criteria before you run a single test. For a summarization product, that might be: output under 150 words, covers the three key points from the source, no hallucinated entities. For a code assistant: passes the existing test suite, no use of deprecated APIs, matches the repo’s naming conventions. Whatever your criteria are, write them down explicitly. Vague acceptance criteria produce vague confidence.

Run your prototype against the golden dataset and record the baseline pass rate. If it’s below 70%, you’re not close to being production-ready and you need to stop here and fix the fundamentals. If it’s between 70 and 85%, you have prompt hardening work ahead of you. Above 90% is where you can start talking about a controlled launch.

Tooling: LangSmith is the most mature option for LLM eval pipelines with tracing baked in. PromptFoo is lighter-weight and works well if you’re doing prompt-level evaluation without full chain tracing. Either one is better than a folder of Python scripts — though a folder of Python scripts is better than nothing.

Phase 2: Prompt Hardening

A prompt that works 80% of the time works zero percent of the time in a production product. Users don’t experience aggregate success rates. They experience the one time it failed on them.

Prompt hardening is the process of moving from 80% to 98%+. It’s mostly about edge cases, and finding those edge cases is the actual work.

Take your failing eval cases and cluster them. Are they failing on short inputs? Long inputs? Inputs with specific terminology? Inputs that are ambiguous? Each cluster is a failure mode, and each failure mode needs a targeted fix — either a prompt change, an input preprocessing step, or a routing decision that sends edge cases down a different path.

Specific techniques that move the needle:

Explicit output format constraints. If you need JSON, don’t just ask for JSON — give the model a schema and tell it what to do when it can’t comply. "If you cannot produce valid JSON matching this schema, return {"error": "cannot_parse", "reason": "<brief reason>"} instead of trying.` This eliminates a class of silent failures.

Few-shot examples for edge cases. Generic few-shot examples in a prompt are often wasted tokens. Few-shot examples that specifically demonstrate the edge case you’re trying to handle are worth ten times as much. If the model keeps getting currency formatting wrong, put a currency example in the few-shot section.

Explicit negative instructions. Don’t just say what you want — say what you don’t want. “Do not include the word ‘certainly’” sounds pedantic until you’ve seen a thousand completions start with “Certainly! Here is your summary.”

Temperature discipline. If your task has a correct answer, temperature should be at or near zero. Developers leave temperature at 1.0 out of habit. For extraction tasks, classification, and any structured output, lower is almost always better.

Track every prompt change against your golden dataset. A change that improves the top-line pass rate but regresses a subset of inputs is often not a net win.

Phase 3: Latency Work

Latency kills AI products quietly. Users tolerate maybe 2-3 seconds for a visible loading state. Beyond that, you’re degrading the experience with every additional second.

Profile before optimizing. Instrument your pipeline end-to-end. Where is the time actually going? Model inference, prompt construction, post-processing, database round-trips? You can’t fix what you haven’t measured. Add timing spans around every major step.

Stream everything you can. If the output is text the user reads, stream it. The perceived latency on a streamed response is the time to first token, not the time to completion. A response that takes 8 seconds to complete but starts streaming in 800ms feels fast. The same response delivered in one blocking call feels slow even at 3 seconds.

Cache aggressively. Repeated identical or near-identical inputs are more common than you think. Use Redis to cache responses keyed on a hash of the normalized input. For many products, cache hit rates of 20-40% are achievable and the latency improvement is dramatic — you go from 2 seconds to 5 milliseconds for cached inputs.

Move non-blocking work off the critical path. Logging, analytics, post-processing that doesn’t affect the response, webhook delivery — none of this should be inline in your request handler. Push it to a queue (BullMQ if you’re on Node, Celery if you’re on Python) and process it asynchronously.

Model selection matters. GPT-5.4 and Claude Sonnet 4.6 are not always the right choice for latency-sensitive workloads. For many tasks — intent classification, routing, entity extraction — a smaller model like GPT-4o-mini or Haiku 4.5 is fast enough and costs an order of magnitude less. Build a routing layer that selects the model based on task complexity.

Phase 4: Error Handling

The model will fail. It will time out. It will return malformed JSON. It will hallucinate a field that doesn’t exist. Your job is to handle all of this gracefully so that the user either never notices or receives a clear, useful error message instead of a broken UI.

Every LLM call needs: a timeout (set it explicitly — most SDKs default to 10 minutes), retry logic with exponential backoff for transient errors (429 rate limits, 503 overload), and a fallback path for when retries are exhausted.

Validate every output structurally before you trust it. If you expect JSON with specific fields, parse and validate before passing it downstream. Pydantic works well for this in Python. Zod works in TypeScript. Don’t assume the model followed instructions.

Distinguish between error types in your logging:

• Model returned garbage / failed schema validation (your prompt problem)
• Model timed out (provider issue, possibly also your prompt is too long)
• Rate limited (your infrastructure problem)
• Model refused to answer (your guardrails problem, or a legit safety refusal)

These have different root causes and different fixes. Aggregating them all as “LLM error” makes them impossible to diagnose.

Phase 5: Cost Controls

LLM costs are nonlinear in ways that surprise people. A feature that costs $0.02 per call feels free until you have 50,000 users using it five times a day.

Rate limiting per user is mandatory, not optional. Implement it before launch, not after you get your first surprise bill. Most providers expose usage dashboards but not real-time hard cutoffs — your rate limiting layer is your own responsibility. Redis with a sliding window is the standard approach. Upstash is a managed Redis option with a generous free tier that works well for this.

Set hard spending caps at the provider level as a backstop. OpenAI and Anthropic both support monthly spend limits. This won’t save you from a billing spike, but it caps your exposure.

Track cost per feature and per user segment. Which features are driving the majority of cost? Are power users consuming a disproportionate share? Are there specific prompt paths that use 10x more tokens than average? This data shapes your pricing model and your optimization priorities.

Prompt token budgets. Long context is expensive. If users can supply arbitrary input (documents, code, conversation history), you need a hard token budget that truncates or summarizes context before it gets passed to the model. Let the model run on 128k tokens by default and you’ll have a bad time.

Phase 6: Observability

You cannot improve a system you can’t see. For AI products specifically, “observability” means three things that standard APM tools don’t give you out of the box.

Input/output logging. Every LLM call should log the full prompt, the full completion, the model used, the token counts, the latency, and a session or user identifier. You need this to debug issues, to build your eval datasets, and to detect regressions. Langfuse is the easiest open-source way to get this without building it yourself — it’s a tracing proxy that sits in front of your LLM calls and captures everything.

Tracing for multi-step pipelines. If your product involves more than one LLM call, you need distributed tracing. LangSmith has this built in. OpenTelemetry with a backend like Datadog or Honeycomb works if you’re already invested in that stack.

Alerting on regression. Run your golden dataset against production on a schedule — daily, or after every deploy. Alert when pass rate drops below your acceptance threshold. This is how you catch prompt regressions before users do. Model providers update their models silently; your prompt that worked fine last week may behave differently today.

For general error alerting, Sentry handles both the frontend and backend well and is the lowest-friction way to get notified when things break.

Phase 7: Safety and Guardrails

This section is shorter than it should be in most engineering plans, which is part of why AI products get into trouble.

Input validation is the first line. Sanitize and validate user input before it touches your prompt. Maximum length limits. Character set restrictions where appropriate. This is basic hygiene that also catches prompt injection attempts.

Output filtering for your use case. At minimum, run completions through a content moderation check before surfacing them. OpenAI’s moderation endpoint is free and fast. Anthropic’s models have strong built-in safety but that doesn’t mean you skip the check.

Jailbreak resistance is a moving target. There is no perfect solution. What you can do: system prompts that explicitly define the scope of the assistant, output validation that detects responses that fall outside your product’s domain, rate limiting that makes repeated probing expensive, and logging that makes attacks visible.

Have a kill switch. You need the ability to disable specific features, or the entire AI component, without a deploy. A feature flag system (LaunchDarkly, Unleash, or even a database row you can flip) is not optional for a production AI product.

Phase 8: Scalability Baseline

You don’t need to over-engineer for scale before launch. You do need to know where the ceilings are.

Know your provider rate limits. OpenAI and Anthropic impose rate limits per tier — tokens per minute, requests per minute. Know what tier you’re on and what happens when you hit the limit. Do your requests queue, fail fast, or get dropped? The answer should match your error handling strategy.

Load test before any significant traffic event. If you’re launching on Product Hunt, sending a big newsletter, or running a paid acquisition campaign, load test first. k6 is free and scriptable. You’re not trying to simulate a billion users — you’re trying to find out if your system falls over at 50 concurrent users, which it might.

Queue design for write-heavy workloads. If your product involves processing user-submitted content asynchronously (documents, audio, bulk requests), model this as a job queue from the start, not an inline synchronous call. BullMQ on Node.js or Celery on Python. This gives you backpressure handling, retry semantics, and visibility into queue depth — all of which you will need.

Phase 9: User Feedback Loop

The product you ship will not be the best version of the product. Every real user who uses it will reveal something your evals didn’t catch. The question is whether you have the infrastructure to learn from it.

Thumbs up/thumbs down is the minimum. It’s a low bar but it’s better than nothing. Attach the feedback to the specific LLM call inputs and outputs, not just a session-level rating.

Correction capture is better. If your product generates text the user edits, log the before and after. The diff is ground truth for what your model got wrong. Over time, this is your most valuable fine-tuning signal.

Build a review queue. Low-confidence outputs (based on your own scoring), flagged outputs, and outputs users marked as bad should feed a human review queue. Someone — you, in early days — should be looking at this every week. This is where the real product insights live.

What to Ship First

You do not need all of this on day one. A reasonable launch checklist:

• Eval suite running with a 90%+ pass rate on your golden dataset
• Streaming responses for anything text-facing
• Hard error handling with fallbacks on every LLM call
• Per-user rate limiting
• Input/output logging (Langfuse or equivalent)
• A kill switch for the AI features

Everything else — load testing, fine-grained cost analytics, full distributed tracing, correction capture — can be added in the first month post-launch while you’re watching real traffic.

The prototype proved the idea works. The production work proves it can scale past your enthusiasm. They’re different problems and require different thinking. Start the production work before you think you need to.