Scaling LLM Applications to Millions of Requests

The Point Up Front

LLM applications don’t scale like normal web apps, and if you try to treat them that way you’ll hit a wall fast. The usual playbook — add more servers, scale horizontally, cache at the CDN — only gets you partway. The real bottlenecks are upstream: provider rate limits measured in tokens per minute, inference latency that makes a 50ms database query look instantaneous, and a cost structure where a single request can burn through thousands of tokens.

The core insight: scaling LLM apps is primarily a latency, cost, and quota management problem. Throughput matters, but you’ll hit your OpenAI rate limit long before you exhaust your application servers. Every architectural decision needs to pass through that lens.

Here’s what actually works when you’re going from hundreds of requests per day to millions.

Know Your Real Ceilings First

Before you engineer a solution, map out where you’ll actually hit limits.

Provider rate limits are the first wall. OpenAI’s tier structure gates you by spend history. On tier 1 you might have 500 RPM and 200k TPM for GPT-4o — that’s less than 10 requests per second if your prompts are small. Anthropic, Google, and others have similar constraints. These aren’t soft limits you can burst through; they return hard 429s.

Latency is the second wall. A GPT-4o call with a medium-length prompt and response typically takes 2–8 seconds. If your users are waiting synchronously, anything resembling real scale means a degraded UX before you even hit rate limits.

Cost is the third and often most overlooked wall. At $10/million output tokens, a feature that generates 500 tokens per user action costs $0.005 per call. That sounds cheap until it’s your fastest-growing feature running millions of times a day.

Once you’ve mapped those three ceilings, the architecture falls into place.

Layer 1: Request Queuing and Rate Limiting

Don’t let raw traffic hit your LLM provider directly. Put a queue in front of it.

BullMQ (Node.js) and Celery (Python) are both solid choices for job queues backed by Redis. The pattern is simple: your API endpoints enqueue jobs and return immediately, workers pull jobs and call the LLM provider with controlled concurrency.

User request → API server → BullMQ queue → Worker pool → LLM provider
                    ↓
             Return job ID immediately

Configure your worker pool concurrency to stay under provider limits. If you have 500 RPM across all your API keys, your worker pool shouldn’t try to fire more than ~8 concurrent requests per second. BullMQ’s concurrency setting and rate limiter make this straightforward:

const worker = new Worker('llm-jobs', processor, {
  concurrency: 8,
  limiter: {
    max: 100,
    duration: 60000, // 100 jobs per minute
  },
});

Add exponential backoff on 429s. LLM providers are generally good about returning retry-after headers; respect them.

Layer 2: Caching — the Highest-Leverage Optimization

The fastest LLM call is one you don’t make. Two categories of caching matter here.

Exact-Match Caching

If you’re running the same prompt repeatedly — product descriptions, classification tasks, FAQ responses — exact-match caching with Redis gives you sub-millisecond responses and zero cost. The key is the full prompt string (or a SHA-256 hash of it). TTL depends on how often your underlying data changes; for static content, set it to 24–72 hours.

cache_key = f"llm:{hashlib.sha256(prompt.encode()).hexdigest()}"
cached = redis.get(cache_key)
if cached:
    return json.loads(cached)

result = call_llm(prompt)
redis.setex(cache_key, 86400, json.dumps(result))
return result

Hit rate on exact-match caching varies wildly by use case. If your prompts are user-generated free-text, you’ll see near-zero hits. If you’re classifying support tickets against a fixed schema, you might cache 40–60% of traffic.

Semantic Caching

For user-facing applications where queries are similar but not identical, semantic caching is worth the added complexity. The idea: embed incoming queries, compare against a vector index of previous queries, and return cached results when similarity exceeds a threshold (typically 0.95+).

GPTCache is a widely-used open-source option for semantic caching, though it entered maintenance mode in 2024 with no new API adapters being added. It plugs in as middleware, supports multiple backends (Redis, FAISS, Milvus), and handles the embedding + similarity lookup automatically.

from gptcache import cache
from gptcache.adapter import openai

cache.init()
cache.set_openai_key()

# Drop-in replacement — caches semantically similar queries
response = openai.ChatCompletion.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": query}]
)

The tradeoff: semantic caching adds ~10–30ms of latency for the embedding lookup. That’s worth it when the alternative is a 3-second LLM call. It’s not worth it for real-time streaming completions where you’re already showing tokens as they arrive.

Layer 3: Prompt Compression

Tokens in = money out + latency up. Prompt compression is the unglamorous work that pays off at scale.

LLMLingua (from Microsoft Research) uses a smaller model to compress long prompts by removing tokens the LLM is unlikely to need, achieving 2–5x compression with minimal quality loss on most tasks. It’s particularly effective for RAG pipelines where you’re stuffing large retrieved contexts into prompts.

Beyond dedicated compression libraries, the manual wins are often bigger:

• Trim system prompts ruthlessly. Every word you remove from a system prompt saves tokens on every single call. Audit yours — they tend to accumulate cruft.
• Use shorter examples in few-shot prompts. Three tight examples beat five verbose ones.
• Truncate retrieved context. In RAG, chunk your documents smaller and only include the top-k most relevant chunks. Experiment with k=3 vs k=5; the quality difference is often smaller than you think.
• Move static content to fine-tuning. If your system prompt contains extensive persona or behavioral instructions that never change, that’s a fine-tuning candidate, not a prompt candidate.

Layer 4: Load Balancing Across Keys and Providers

A single API key is a single point of failure with a single rate limit. Distribute across both.

LiteLLM is the right tool here. It presents a unified OpenAI-compatible interface across 100+ providers and handles routing, fallbacks, retries, and load balancing in one package. Run it as a proxy server in front of your application:

# litellm_config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_KEY_1
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_KEY_2
  - model_name: gpt-4o
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_KEY

router_settings:
  routing_strategy: least-busy
  num_retries: 3
  retry_after: 5

Your application talks to LiteLLM on localhost:4000 and never needs to know which provider is handling the request. LiteLLM round-robins across keys, respects rate limits per key, and falls back to the next provider on errors.

For critical workloads, configure a fallback chain: primary model → cheaper same-provider model → different provider. This gives you both cost optimization and availability guarantees.

Layer 5: Streaming for UX at Scale

Streaming doesn’t increase throughput — it changes the perception of latency. For user-facing features, it’s often more important than any infrastructure optimization.

When you stream token-by-token via SSE (Server-Sent Events) or WebSockets, users see output start in under a second even when total generation takes 8 seconds. Perceived performance is dramatically better, and you’ll see it in retention metrics.

The infrastructure implication: streaming requires persistent connections, which changes your server architecture. Make sure your load balancer supports long-lived HTTP connections (disable aggressive timeouts), and size your connection pool accordingly. Nginx needs proxy_read_timeout bumped from its default 60s. AWS ALB needs idle timeout increased.

For background jobs where no user is waiting synchronously, don’t bother streaming — it adds complexity without benefit. Only stream where a human is watching.

Layer 6: Async Processing Patterns

Not every LLM call needs to be synchronous. Shifting work to async patterns is often the single biggest architectural improvement you can make.

The webhook pattern: User triggers an action, you return a job ID immediately, process async, and POST results to a webhook URL or push via WebSocket when done. This decouples user-facing latency from LLM latency entirely.

Prefetching: If you can predict what a user will need next — common in conversational AI, recommendation systems, or multi-step workflows — kick off the LLM call before they ask for it. By the time they click, the result is ready.

Priority queues: Separate queues for interactive vs. batch workloads. User-triggered actions get high priority; nightly batch analysis jobs get low priority. BullMQ and Celery both support this natively. Don’t let your batch pipeline eat into quota that interactive users need.

Cost as a Scaling Constraint

At scale, cost is an architectural concern, not just a finance concern.

Model tiering is underused. Not every task needs GPT-4o. Classification, extraction, and summarization tasks often perform equally well on GPT-4o-mini or Claude Haiku at 10–20x lower cost. Build a routing layer that sends complex, high-stakes tasks to your best model and routes simpler tasks to cheaper ones. LiteLLM supports this routing pattern natively.

Track cost per feature, not just cost in aggregate. You need to know that your “explain this error” feature costs $0.003/call and your “generate a report” feature costs $0.08/call. Without that granularity, you can’t make intelligent decisions about where to optimize.

When to Move Off Managed APIs

Managed APIs are the right default. They’re easy, maintained, and have zero ops burden. But there’s a crossover point where self-hosted inference makes economic sense.

The math: At scale, you’re paying ~$5–15/million tokens on managed APIs. Running open-weight models (Llama 4, Mistral, Qwen) on your own GPU cluster via vLLM or SGLang can reduce marginal costs by 80–95% once your GPU fleet is amortized.

The crossover is typically somewhere between $5k–20k/month in API spend, depending on your model needs and ops capacity. Below that, the engineering overhead of running inference infrastructure eats the savings.

Self-hosting prerequisites:

• You have a model that actually meets your quality bar (evaluate rigorously before committing)
• You have GPU ops expertise or a team willing to develop it
• Your request volume is high and consistent enough to keep GPUs utilized
• Data privacy or sovereignty requirements push you toward on-prem anyway

vLLM is the current gold standard for serving open-weight models. It supports continuous batching, PagedAttention for efficient KV cache management, and a drop-in OpenAI-compatible API. You can point LiteLLM at a vLLM endpoint and get seamless failover between self-hosted and managed providers.

The Practical Rollout Order

Don’t implement all of this at once. Here’s the order that gives you the most leverage earliest:

1. Add exact-match caching — Redis, one afternoon, measurable impact immediately
2. Implement request queuing — BullMQ or Celery, stops you from hammering provider limits
3. Set up LiteLLM — multi-key, multi-provider load balancing, adds resilience
4. Add streaming — for any user-facing synchronous call
5. Implement model tiering — route cheap tasks to cheap models
6. Add semantic caching — once you’ve profiled your traffic and know where duplicates cluster
7. Compress prompts — audit and trim, then add LLMLingua if RAG is a bottleneck
8. Evaluate self-hosting — only when API spend justifies the ops complexity

The bottleneck shifts at each stage. Measure before and after each change; don’t optimize blindly.

The Infrastructure Stack That Works

When this is all assembled, your stack looks like:

• LiteLLM proxy — unified routing, load balancing, fallbacks across keys and providers
• Redis — exact-match cache, semantic cache backend, BullMQ job store
• BullMQ / Celery — async job queue with priority lanes and rate limiting
• GPTCache — semantic caching middleware
• vLLM (optional) — self-hosted inference for high-volume, cost-sensitive workloads
• Prometheus + Grafana — per-model, per-feature cost and latency tracking

None of this is exotic. The libraries are mature, the patterns are proven, and the operational complexity is manageable. What’s different from scaling a regular web app is the mindset: you’re orchestrating around external constraints (provider limits, inference latency, token costs) rather than just adding compute. Once you internalize that, the architecture gets much clearer.