The Cheapest Way to Serve Open Models

The Bottom Line First

Here’s the decision tree before you read anything else:

• Under ~5M tokens/day → use a managed cheap API (Together AI, Fireworks, Groq). No ops, no GPU babysitting, competitive pricing.
• 5M–50M tokens/day → run vLLM on spot instances (RunPod, Vast.ai). You’ll save 60–80% over managed APIs once you account for batching.
• Over 50M tokens/day → multi-GPU spot cluster with vLLM + aggressive quantization. At this scale, the engineering investment pays off in weeks.
• Latency-tolerant background jobs → llama.cpp on ARM CPU instances. Surprisingly viable for low-concurrency batch work.

Everything below is the math and ops detail behind those four boxes.

The Landscape

Open-weight models — Llama 3.3, Mistral, Qwen 3.5, Gemma 4, Phi-4 — are now competitive with proprietary APIs on most tasks. The question is no longer “should I use open models?” but “what’s the cheapest way to run them without my infrastructure becoming a second job?”

There are four practical strategies, each with a different cost/ops tradeoff:

1. Managed inference APIs (someone else’s GPUs, pay per token)
2. Spot instance self-hosting with vLLM or TGI
3. Quantization to fit bigger models on cheaper hardware
4. CPU inference as a last resort (or a surprisingly good fit)

Let’s go through each with real numbers.

Strategy 1: Managed Cheap Inference APIs

If you’re not yet at the volume where self-hosting pencils out, managed APIs are the right default. The margins on these services are thin and competition is fierce — which is good for you.

Current pricing for popular open models (as of mid-2026)

A few things jump out:

• Groq is the cheapest for Llama 3.1 8B by a wide margin ($0.05/M input). If you can use an 8B model, this is essentially free at most scales.
• Groq’s 70B pricing is also the lowest — their LPU hardware gives them a real cost advantage for certain workloads.
• Together and Fireworks diverge on 70B pricing — Together is $0.88/M flat for Llama 3.3 70B, while Fireworks dropped their Llama 3.1 70B to $0.70/M input / $0.90/M output and offers newer Qwen models.
• Replicate is convenient if you’re already in that ecosystem, but output tokens cost $2.75/M — significantly more than the flat-rate providers.

When managed APIs make sense

• You’re under 5M output tokens/day. At $0.88/M, that’s $4.40/day — $132/month. Cheaper than a single A100 hour.
• Your traffic is spiky or unpredictable. Spot instances get preempted. Managed APIs absorb your spikes automatically.
• You need multiple models. Running five different model sizes on your own infra is painful. Managed APIs let you experiment without provisioning anything.

Limitations

• Rate limits exist on every provider and can bite you at scale.
• You’re sending your data to a third party. For regulated industries, this is a blocker.
• At high volume, you’re subsidizing the provider’s margin. Eventually self-hosting wins.

Strategy 2: Spot Instance Self-Hosting

This is where things get interesting — and where most teams leave money on the table by either doing it too early (before volume justifies ops overhead) or not doing it at all (and paying 3–5x more than they need to).

GPU spot pricing (mid-2026 averages)

The math: self-hosting vs. managed API

Let’s compare serving Llama 3.3 70B on a single A100 80 GB via vLLM versus Together AI.

Managed API cost (Together AI at $0.88/M):

• 10M tokens/day = $8.80/day = $264/month

Self-hosted cost (Vast.ai A100 spot at $0.60/hr):

• 24 hours × $0.60 = $14.40/day
• Break-even vs. Together: 14.40 / 0.00088 = ~16.4M tokens/day required just to match cost

Wait — that looks bad for self-hosting. But this ignores throughput and batching.

A single A100 running vLLM with Llama 3.3 70B at bf16 can sustain roughly 1,500–3,000 output tokens/second with reasonable batching. At 2,000 tokens/sec:

• Per day: 2,000 × 86,400 = 172.8M tokens/day capacity
• Cost per 1M tokens: ($14.40/day) / 172.8M × 1M = $0.083/M tokens

So if you’re actually using that GPU efficiently, the per-token cost drops to $0.083/M — 90% cheaper than Together AI for the same model.

The catch: you need to actually saturate the GPU. If your GPU sits at 20% utilization, your effective cost is 5× higher than the math above suggests. The single most important factor in self-hosted inference economics is utilization.

vLLM vs. TGI

Use vLLM for almost everything:

• Better continuous batching implementation — more throughput per GPU
• Supports more models out of the box
• PagedAttention handles variable-length sequences efficiently
• Active development and large community

Use TGI (Text Generation Inference) when:

• You need Hugging Face-native model loading without conversion steps
• You want the built-in token streaming UI for debugging
• Running GPTQ models and vLLM’s GPTQ support is giving you trouble

Spot instance reliability

Spot instances get preempted — plan for it. The practical mitigations:

• Run at least two instances and load-balance across them
• Use stateless inference (don’t maintain session state on the instance)
• Script automated restart via provider APIs or cloud-init
• Vast.ai is cheaper but has higher preemption rates than RunPod; pick based on your tolerance

Strategy 3: Quantization — Fit More on Less

Quantization is the lever that changes which GPU tier you need, and it’s often the difference between a $0.34/hr GPU and a $1.40/hr GPU.

Quantization formats and their tradeoffs

Practical impact: which models fit where

The 70B sweet spot: A quantized Llama 3.3 70B (AWQ or GPTQ 4-bit) fits on two RTX 4090s. Two 4090s on Vast.ai spot run ~$0.26–0.60/hr combined — compared to ~$0.50–0.73/hr for a single A100 80 GB. The GPU cost is roughly comparable, with the 4090 pair offering better availability and flexibility.

AWQ vs. GPTQ vs. GGUF — which to use

AWQ is the default choice for GPU inference. The quantization-aware approach produces better perplexity than GPTQ at the same bit depth, and vLLM’s AWQ support is mature. Grab pre-quantized models from Hugging Face (TheBloke and bartowski have most popular models already done) rather than quantizing yourself.

GPTQ is the alternative when AWQ models aren’t available. Quality is nearly indistinguishable in practice for most use cases.

GGUF is for llama.cpp specifically. If you’re running on GPU with vLLM, ignore GGUF. If you’re running on CPU or a mixed CPU/GPU setup, GGUF is your format.

Strategy 4: CPU Inference with llama.cpp

CPU inference sounds absurd for production, but there’s a real use case: low-concurrency, latency-tolerant batch jobs on cheap ARM instances.

ARM CPU pricing

What llama.cpp on ARM actually delivers

Llama 3.1 8B Q4_K_M on a 16-core ARM instance (Hetzner CAX41):

• ~20–35 tokens/second generation speed
• Cost: ~$0.06/hr
• Per-day capacity at 25 tok/s: 25 × 86,400 = 2.16M tokens = $0.67/M tokens

That’s cost-competitive with managed APIs for the same model class — comparable to Together AI’s Llama 3.1 8B at $0.18/M when you factor in multi-request batching, with the added benefit of data privacy. The catch: 25 tokens/second means a 500-token response takes 20 seconds. This is fine for document processing, embedding pipelines, or async classification tasks. It is not fine for a chatbot.

For the smallest models (1B–4B: Phi-4-mini-reasoning, Qwen3-0.6B), you can hit 60–80 tokens/second on a 16-core ARM chip, which is usable for some synchronous applications.

Oracle Cloud’s Ampere free tier (2 OCPUs, 12 GB RAM, always free) is worth knowing about for experiments: you can run a small quantized model like Llama 3.2 3B Q4_K_M there at ~15 tokens/second, permanently, for free. (Oracle halved its free-tier resources in June 2026, so legacy accounts may still have the original 4 OCPU / 24 GB allocation.)

The Full Cost Comparison

Pulling it together for Llama 3.3 70B at various daily token volumes:

The crossover where self-hosting beats Together AI is around 16M tokens/day for a full A100, and around 11M tokens/day for 2× 4090 with AWQ. Below that, pay the managed API and save your ops time.

What People Get Wrong

Running a GPU at 10% utilization. If you spin up a dedicated A100 for a low-traffic app, you’re paying $14.40/day to process what $3 of managed API calls would handle. Self-hosting requires throughput to justify itself.

Ignoring quantization. Running a 70B model at fp16 when AWQ gives you 95%+ of the quality at 25% of the VRAM cost is leaving money on the table. The quantized models are already on Hugging Face — just use them.

Over-engineering with Kubernetes before you need it. One vLLM process on one spot instance with a simple restart script is operational baseline. Add complexity only when you have a real problem.

Treating spot preemption as catastrophic. If your inference is stateless (it should be), preemption just means a brief gap in capacity. Load-balance across two spots and the preemption of either one is barely noticeable.

Not using continuous batching. vLLM does this by default. If you’re running a server that processes one request at a time, you’re getting a fraction of the throughput you should be. Batch your requests.

Recommended Starting Point

For a new project serving an open model in production, start here:

1. Day 1: Together AI or Groq. No infrastructure, instant access to every major model, pay per use.
2. When you cross 15M tokens/day: Benchmark vLLM on a single RunPod spot instance with your target model. Compare actual cost and latency against the managed API. The math will tell you whether to switch.
3. When switching: Two spot instances (for redundancy), vLLM with an AWQ or GPTQ quantized model if VRAM is tight, a simple nginx or Caddy reverse proxy in front, and a cron job that checks instance health and spins up a replacement if one disappears.

That’s it. No Kubernetes, no Helm charts, no service mesh. Add those if you actually need them — most teams don’t.

The cheapest setup is the simplest one that runs reliably at your actual volume. Figure out your volume, apply the crossover math above, and pick accordingly.