The Complete AI Infrastructure Stack

The Short Version

If you’re building a production AI application in 2026, the stack that works looks like this: Anthropic or OpenAI for models, vLLM or Fireworks for inference, LangGraph or plain Python for orchestration, Qdrant or pgvector for vectors, LangSmith or Langfuse for observability, and Modal or fly.io for deployment. Everything else is a variation on this theme based on your specific constraints.

That’s the bottom line. The rest of this article is the reasoning behind those choices — the tradeoffs, the alternatives worth knowing about, and how these layers connect.

Why the Stack Matters

Most AI project failures aren’t model failures. They’re infrastructure failures: latency that kills UX, costs that kill the business, missing observability that makes debugging a guessing game, or a deployment setup that can’t handle spiky traffic patterns.

Getting the stack right from the start is an order of magnitude cheaper than retrofitting it later. Let’s go layer by layer.

Layer 1: Model Providers

This is where most developers start, and where most decisions get made emotionally rather than analytically.

The Tier-1 Options

Anthropic (Claude Sonnet 4.6, Claude Opus 4.8) The current default recommendation for most production use cases requiring complex reasoning, long context, or instruction-following. Claude Sonnet hits the best capability-per-dollar ratio. Opus 4.8 is for tasks where quality is non-negotiable and cost is secondary.

OpenAI (GPT-5.5 Instant, GPT-5.4 Thinking, GPT-5.4 Pro) GPT-5.5 and GPT-5.4 are extremely competitive across multimodal, reasoning, and general tasks. GPT-5.4 Thinking is the go-to for math, code generation, and structured problem-solving. OpenAI’s ecosystem depth — tooling, SDKs, community — is still unmatched.

Google (Gemini 3.5 Flash, Gemini 3.5 Pro) Gemini 3.5 Flash has a 1M-token context window and performs exceptionally on retrieval-heavy and agentic tasks. Gemini Flash remains the price-performance leader for high-volume inference. Gemini 3.5 Pro (rolling out) aims to extend reasoning depth further. Often overlooked by developers but worth benchmarking for your specific task.

When to Consider Open Models

Meta Llama 4 / Mistral / Qwen 3 are worth the operational overhead when:

• You have strict data residency requirements
• Your inference volume is high enough that API costs exceed self-hosting costs (rough threshold: ~10M tokens/day)
• You need fine-tuning on proprietary data and don’t want that data leaving your infrastructure

For most teams under 20 engineers, self-hosting open models adds operational complexity that outweighs the cost savings.

Model Selection Heuristic

Layer 2: Inference

If you’re calling model provider APIs directly from your application, you’re going to hit walls: rate limits, latency spikes, no caching, no batching. You need an inference layer between your app and the models.

Managed Inference Providers

Fireworks AI — The current leader for open model inference speed. Significantly faster than self-hosting for most teams, with good price-per-token for Llama and Mixtral variants. If you’re running open models and don’t want to manage GPU infrastructure, start here.

Together AI — Strong competitor to Fireworks. Better for fine-tuned model deployment, which Fireworks doesn’t support as cleanly. Their serverless inference has no cold-start problem for most popular models.

Groq — Absurd speed (hundreds of tokens per second) via custom LPU hardware. The tradeoff: limited model selection and throughput caps at scale. Use it when latency is the primary metric — streaming chat, real-time voice pipelines.

Replicate — Easiest path to deploying arbitrary model weights. Great for prototyping, expensive at scale. Not suitable for production APIs with SLA requirements.

Self-Hosted Inference

vLLM is the production standard for self-hosting. PagedAttention makes it dramatically more memory-efficient than naive implementations, and continuous batching means your GPU utilization stays high. If you’re deploying on your own GPUs, vLLM is the default choice.

Ollama is for local development only. Don’t run it in production.

Inference Optimization Checklist

Before going to production, verify you have:

• Prompt caching enabled (saves 50-90% on repeated context for supported providers)
• Streaming implemented — users feel latency much less when they see tokens arriving
• Batching for async workloads — don’t call models one-at-a-time for batch jobs
• Fallback logic — primary provider down means your app needs to redirect, not 500

Layer 3: Orchestration and Frameworks

This is where the most framework churn happens. Teams pick a framework, build on it, then fight it. Here’s how to avoid that.

The Frameworks Worth Knowing

LangChain / LangGraph LangChain itself is often too much abstraction — it obscures what’s happening and makes debugging harder. But LangGraph is genuinely useful for building stateful, multi-step agents. If you’re building agentic workflows with branching, loops, or human-in-the-loop steps, LangGraph’s graph-based model maps naturally to the problem.

Plain Python + instructor For most applications, you don’t need a framework. A few well-structured Python functions, the Anthropic or OpenAI SDK, and instructor for structured outputs gets you 80% of what LangChain provides with 20% of the complexity. Instructor specifically is excellent — it wraps the model’s function-calling to give you validated Pydantic objects back, which is exactly what you want for production code.

Haystack Better than LangChain for document processing pipelines and RAG workflows with complex retrieval logic. More opinionated about data flow, which is a feature, not a bug. If your core use case is document-heavy, benchmark Haystack before defaulting to LangChain.

DSPy A different paradigm: instead of writing prompts, you write the logic and DSPy optimizes the prompts automatically. Early for most production use cases, but worth watching. If you’re spending significant time on prompt engineering, DSPy’s optimization approach may pay off.

Orchestration Decision Tree

• Simple RAG pipeline: Plain Python + instructor + your vector DB client
• Multi-step agent with state: LangGraph
• Document-heavy processing: Haystack
• Experimenting with prompt optimization: DSPy
• Everything else: Start with plain Python, add a framework when you feel the friction

Layer 4: Vector Databases

Vector search has become table-stakes infrastructure. The right choice depends on whether you need a dedicated vector DB or can add vector search to your existing database.

Dedicated Vector Databases

Qdrant — The current recommendation for new projects. Rust-based, fast, strong filtering support, good Kubernetes story, and actively maintained. The on-disk indexing handles datasets that don’t fit in RAM, which most alternatives struggle with.

Pinecone — Fully managed, easy to get started, but expensive at scale and the serverless pricing model can surprise you. Good for teams that want zero operational overhead and have the budget.

Weaviate — Multimodal support (text + image + audio) is a differentiator. More complex to configure but the schema-based approach is cleaner for production data modeling than schema-less alternatives.

Postgres-Based Options

pgvector — If you’re already on Postgres, pgvector is often the right answer. Not as fast as dedicated vector DBs at large scale, but “good enough” for datasets under ~5M vectors, and you eliminate an entire infrastructure dependency. VectorChord (the successor to pgvecto.rs) is a faster Rust-based alternative if you need more performance without leaving Postgres. (pgvecto.rs is no longer actively maintained.)

Supabase bundles pgvector with a hosted Postgres offering — useful for teams that want managed infrastructure with vector support.

Vector DB Decision Matrix

Layer 5: Observability

Observability is the layer developers skip first and regret most. Without it, you’re debugging by vibes.

The minimum viable observability stack for an LLM application:

• Trace every LLM call — input, output, latency, tokens, cost
• Log user sessions end-to-end — you need to see the full conversation, not individual API calls
• Capture errors with context — a bare exception traceback is useless without knowing what prompt caused it

The Tools

LangSmith — Deep integration with LangChain/LangGraph, but also works standalone. The best UI for inspecting traces, comparing runs, and debugging agent behavior. Annotation and evaluation tooling built-in. If you’re using LangGraph, LangSmith is the obvious choice.

Langfuse — Open-source alternative to LangSmith. Self-hostable, which matters for enterprise or compliance-sensitive deployments. Feature-comparable at this point. The hosted version is cheaper than LangSmith at scale.

MLflow — Better for experiment tracking and model evaluation than real-time production tracing, but the recent LLM-focused additions make it viable as a unified platform if you’re already invested in the MLflow ecosystem.

Helicone — Lightweight proxy-based approach. You route API calls through Helicone and get tracing without changing your application code. Fast to integrate, less flexible for complex agentic traces. (Acquired by Mintlify in March 2026; now in maintenance mode — not recommended for new projects.)

What to Track in Production

Beyond raw traces, make sure you’re instrumenting:

• Latency by percentile — p50/p95/p99, not just average
• Cost per user session — you need to know your unit economics
• Token usage trends — context creep is a real problem in chat applications
• Retrieval quality signals — if you’re doing RAG, track what gets retrieved and whether it’s used
• User feedback — thumbs up/down at minimum; richer signals if you can get them

Layer 6: Deployment and Hosting

Where your AI application actually runs.

Compute Options

Modal — The best developer experience for running AI workloads on GPUs in the cloud. Serverless GPU compute with a genuinely good Python-first API. Cold starts are an issue for latency-sensitive applications, but the developer experience and cost model are hard to beat for most use cases.

Fly.io — Excellent for the non-GPU parts of your stack. Fast global deployment, good Postgres integration, predictable pricing. Run your API servers on Fly, your GPU workloads on Modal or directly on provider infrastructure.

AWS SageMaker / Google Vertex AI / Azure ML — The enterprise options. Significant operational overhead to set up, but the managed inference endpoints, model registries, and MLOps pipelines are genuinely useful at large scale. Hard to justify before you need the compliance features or deep cloud integration.

Replicate / Banana (now Baseten) — Baseten is the more production-ready option here. Good for serving custom model weights with less infrastructure management than running on raw cloud compute.

Serving Architecture Patterns

Stateless API servers work for most retrieval-augmented and single-turn applications. Your app server calls the model provider, returns the result. Scale horizontally, use a CDN, done.

Long-running agents need more care — streaming responses, persistent state, and connection handling that standard API frameworks weren’t designed for. Use WebSockets or SSE for the client connection, and make sure your state is stored in a database (not in-memory) so you can handle restarts.

Batch processing pipelines — use a job queue (Celery + Redis, or a managed queue like AWS SQS) and separate your batch inference from your real-time serving. Don’t let a batch job saturate your real-time rate limits.

How the Layers Connect

A production RAG application looks like this in practice:

1. User query arrives at your API (Fly.io or AWS)
2. Query is embedded using the same embedding model used at index time
3. Vector search retrieves relevant chunks (Qdrant or pgvector)
4. Retrieved context + query are assembled into a prompt
5. Prompt is sent to Claude Sonnet 4.6 or GPT-5.5 via the provider API
6. Response streams back to the user
7. The entire trace — query, retrieved chunks, prompt, response, latency, cost — is logged to LangSmith or Langfuse

Each of these steps can fail independently. The observability layer is what lets you see which one failed and why.

The Non-Negotiables

Before you call your stack production-ready:

• Rate limit handling — exponential backoff with jitter, fallback providers
• Cost controls — per-user limits, circuit breakers, budget alerts
• PII handling — scrub before logging, understand what leaves your infrastructure
• Prompt versioning — you need to know which prompt version was running when a bug occurred
• Evaluation baselines — you need to know when a model upgrade breaks behavior, not just improves benchmarks

The AI infrastructure space moves fast, but these fundamentals don’t change. Get the layers right, instrument everything, and you’ll spend your time building instead of debugging.