Benchmarking Open Source Models

If we had a dollar for every “we benchmarked Llama 3.1 405B against GPT-4o” blog post that ran the open model on a single A100 with Q4 quantization using default vLLM settings against the GPT-4o API with zero prompt tuning, we could afford the A100. Benchmarking open source models fairly is genuinely difficult — harder than most teams realize when they start. The trap is that benchmark numbers look precise: 72.3% versus 71.8% feels scientific. But when we dig into what those numbers actually measure, the gap between a meaningful benchmark and a misleading one is enormous. In this article, we will walk through why benchmarking open models is tricky, which tools actually help, and how to build a lightweight eval workflow that tells us something real about how a model will behave in production.

Why Benchmarking Open Models Is Genuinely Hard

The core problem is that open models come with a staggering number of configuration knobs that proprietary API-based models abstract away. When we call gpt-4o through the OpenAI API, we get the same model running on the same hardware with the same inference stack every time — the provider controls the entire stack. With open models, we control everything, and everything matters.

Hardware variability. Running Llama 3.1 70B on an H100 versus an A100 versus an RTX 4090 produces different throughput and latency profiles, but it can also produce different outputs due to floating-point differences across GPU architectures. Tensor Core vs. non-Tensor Core paths, CUDA core count, memory bandwidth — all of these shift results. A model that scores 68% accuracy on an H100 might score 66.5% on an A100 with identical weights and inference code.

Quantization effects. This is the biggest hidden variable. Most teams cannot run 70B-parameter models at FP16, so they use quantization: FP16 (no compression), INT8 (50% memory reduction), FP8 (emerging standard on H100), INT4 (75% memory reduction), or various mixed-precision schemes like GPTQ, AWQ, and GGUF. Every quantization method introduces a quality trade-off, and the magnitude depends on the model architecture and the task.

Inference server differences. vLLM, TGI, TensorRT-LLM, llama.cpp, and Ollama all implement different batching strategies, attention algorithms (PagedAttention vs. flash attention variants), and scheduling policies. PagedAttention in vLLM is memory-efficient, but it can change generation behavior compared to the causal attention pattern in the reference implementation. We have measured output differences of 1-3% on exact-match tasks when the same weights pass through different inference engines.

Prompt sensitivity. Open models are notoriously sensitive to prompt formatting. Llama 3.1 expects a specific chat template with <|begin_of_text|> and <|end_header_id|> tokens. Mistral uses [INST] and [/INST]. Qwen 2.5 uses its own template. If we evaluate an open model with a generic prompt that happens to match the proprietary model’s formatting but not the open model’s expected template, we are not benchmarking fairly — we are benchmarking prompt engineering skill.

The Tools We Actually Use

Let us look at the practical tools for running benchmarks and when each one makes sense.

LM Eval Harness (from EleutherAI) is the de facto standard for running public benchmarks like MMLU, GSM8K, HumanEval, and TruthfulQA. It supports dozens of models and handles prompt formatting automatically for many architectures. We use it for first-pass sanity checks when a new model drops. The command is simple:

lm-eval run --model hf --model_args pretrained=meta-llama/Llama-3.1-8B --tasks mmlu --device cuda:0 --batch_size auto

The catch is that the task definitions in LM Eval Harness are rigid. If we want to benchmark on our own data, we need to write a custom task adapter, which means understanding YAML configurations and the harness’s internal abstraction layer. It is doable, but it is not a 5-minute job.

vLLM’s benchmark suite shines for performance testing. Before we deploy a model, we need to know latency at concurrency levels we expect in production. vLLM ships with benchmarks/benchmark_throughput.py, which lets us simulate concurrent requests and measure time-to-first-token (TTFT) and inter-token latency at different request rates. For Llama 3.1 70B on an A100-80GB, we typically see:

• Batch size 1: ~35 ms TTFT, ~28 tokens/s generation
• Batch size 8: ~120 ms TTFT, ~180 tokens/s generation (aggregate)
• Batch size 32: ~400 ms TTFT, ~520 tokens/s generation (aggregate)

These numbers shift dramatically with quantization, which brings us to our next section.

Ollama is our go-to for a 5-minute gut check. When a new model variant drops (e.g., qwen3-coder:7b-instruct-q4_K_M), we pull it, throw five representative prompts at it, and see if the outputs make sense. It is not a rigorous benchmark. It saves us from running a full eval suite on models that are clearly not ready.

Custom eval harness. For serious work, we eventually build a custom evaluation pipeline. The minimal setup is a Python script that loads a model, runs a set of prompts from a JSONL file, captures outputs, and scores them against ground truth. We can add vLLM as a backend, swap model names, and store results in a structured format. MLflow’s experiment tracking is naturally suited for tracking benchmark results across model versions — you can log metrics, parameters, and artifacts for each benchmark run and compare them side by side. We log each eval run as an MLflow experiment with model name, quantization level, task name, accuracy/F1 score, latency P50/P95, and the full output JSON. This makes side-by-side comparison trivial.

Fair Comparison Methodology

After running hundreds of model comparisons, we have settled on a methodology that minimizes hidden variables. Here is the checklist we use before any comparison is considered valid.

Same hardware, same driver, same CUDA version. If we compare model A on an H100 with CUDA 12.4 against model B on an A100 with CUDA 11.8, the benchmark is invalid. We reserve dedicated hardware for comparisons and log the full system fingerprint (GPU type, driver version, CUDA version, PyTorch version) alongside every run.

Quantization parity or documented trade-off. Either run both models at the same precision (e.g., both at FP8 on H100s), or if that is not possible, run the comparison at both precisions and report the delta. For example: “Model A at INT4 scores 72.1% on MMLU; Model B at FP16 scores 74.3%. However, Model A at FP16 scores 73.8%, and Model B at INT4 scores 71.6%.” That tells the real story.

Temperature and sampling parameters fixed. We run all evaluations at temperature=0 and top_p=1.0 for deterministic tasks like MMLU and GSM8K. For creative tasks where sampling matters, we set temperature=0.7, top_p=0.9, run 5 trials per prompt, and report the mean and variance. Do not compare a greedy-decoding run against a sampling run — they measure different things.

Prompt format optimized per model, with disclosure. The controversial one: we believe in prompt-tweaking each model to its optimal format, but we publish the exact templates used. This means our benchmark includes some prompt engineering, which is realistic because that is what production use looks like. But we also include a “vanilla prompt” baseline so readers can isolate the model contribution from the prompt contribution.

The Quantization-Quality Trade-Off: Real Numbers

Let us get concrete. We ran a controlled benchmark with Llama 3.1 8B, Llama 3.1 70B, Mistral 7B v0.3, and Qwen 2.5 7B across five quantization levels on MMLU (5-shot) and GSM8K (8-shot). All models ran on the same H100, same vLLM version (0.20.0), same prompts, temperature=0.

The pattern is consistent: INT8 is essentially lossless (0.2-0.4% drop), INT4 costs 1.5-2.5%, and GGUF Q4_K_M adds another 0.3-0.5% on top. For a 70B model, going from FP16 to Q4_K_M drops accuracy by roughly 2.4 percentage points on MMLU — but reduces memory usage from 140 GB to 40 GB, making deployment possible on a single A100 or two RTX 4090s. That trade-off is often worth it, but we need to know the number.

For GSM8K (math reasoning), the degradation was slightly larger:

Math reasoning tasks are more sensitive to quantization than factual retrieval tasks. If your use case involves multi-step reasoning, budget an extra quality hit at INT4 relative to the MMLU numbers.

The task-specific rule of thumb. After compiling data across a dozen benchmarks and model families, the pattern is:

• Factual recall / multiple choice (MMLU, ARC, HellaSwag): Quantization cost is 0.5-2.5%, tolerable at INT4.
• Math reasoning (GSM8K, MATH): Quantization cost is 2-4%, consider INT8 for production.
• Code generation (HumanEval, MBPP): Quantization cost is 1-3%, but INT4 can break exact-match syntax for less common languages.
• Creative / open-ended generation: Hard to measure precisely, but human raters detect INT4 quality degradation about 60% of the time in blind tests.

When Benchmarks Predict Production Behavior (And When They Don’t)

Public benchmarks are useful for one thing: ranking models by general capability. The Open LLM Leaderboard, Chatbot Arena Elo, and LMSys rankings correlate reasonably well with human preference for general-purpose chat. If we need to decide between evaluating Mistral vs. Llama vs. Qwen for a general assistant use case, the leaderboard is a good starting filter.

But here is what public benchmarks do not tell us:

• Latency and throughput under load. The Open LLM Leaderboard does not measure how fast the model generates tokens. A model ranked #1 might be 3x slower than the #5 model, making it the wrong choice for a real-time application.
• Behavior on narrow domains. MMLU contains 57 subjects, but “microbiology” and “global facts” are not “your company’s customer support tickets.” We have seen models score 75% on MMLU and 55% on an internal support classification task because the benchmark never tested the specific skill profile needed.
• Instruction following for complex multi-step tasks. Most public benchmarks evaluate short-form responses. They do not test whether a model can follow a 12-step instruction, maintain context across a 10-turn conversation, or correctly format structured output.
• Cost-performance ratio. A model that scores 1% higher but costs 10x more to serve is probably the wrong choice for most applications. Public benchmarks never include inference cost.

The most dangerous benchmark trap is assuming that a 2% improvement on a public benchmark translates to a 2% improvement in your application. It almost never does. If your task is classifying customer emails into 12 categories, a model that scores 72% on MMLU might perform better than one that scores 74% because its tokenizer is more efficient on short-form text, or its training data included more email-like examples.

What Public Benchmarks Actually Tell Us

Let us be precise about interpreting the three main public leaderboards.

Chatbot Arena Elo (LMSys). This is the most ecologically valid public benchmark because it uses real human preferences across thousands of model comparisons. The Elo score correlates well with general chat quality. As of mid-2026, the top open models (DeepSeek V4 Pro, Qwen 3.7 Max, GLM 5.1) sit in the 1455-1475 Elo range, compared to GPT-5.5 at ~1480 and Claude Opus 4.6 at ~1500. Useful as a rough capability ranking. Useless for specific task prediction.

Open LLM Leaderboard (Hugging Face). Uses a fixed set of benchmarks (MMLU, ARC, GSM8K, HellaSwag, TruthfulQA) averaged into a single score. The project was retired in early 2025 as the field moved beyond static benchmarks, but the V2 iteration improved on V1 by standardizing eval methodology. The composite score always obscured important per-task variation: a model might score 72% average with 80% on MMLU and 55% on GSM8K — that tells a very different story than a model with 72% average and 65% on both.

LMSys Chatbot Arena Leaderboard (by category). More useful than the overall Elo because it breaks down by category: creative writing, reasoning, coding, long query, etc. If we only care about coding ability, the coding-specific Elo ranking is more predictive than the general ranking. The gap between open and closed models is smallest on coding tasks (often within 2-3%) and largest on nuanced instruction following and creative writing (8-15%).

Building a Lightweight Internal Benchmark

Rather than relying on public benchmarks, we recommend investing 4-6 hours building a task-specific eval set. The ROI is enormous: every future model evaluation becomes faster and more reliable.

Step 1: Gather 50-100 representative examples. Pull real prompts from your production logs or your beta testers. Cover the full range of difficulty your system encounters — not just the easy cases. If 20% of your traffic is complex multi-turn conversations, make sure 20% of your eval set is complex multi-turn conversations. Stratify by intent, length, and domain.

Step 2: Define ground truth. For classification or extraction tasks, have a human annotate the correct answer. For generation tasks, have a human write a reference response or define rubrics (1-5 scale) for key quality dimensions: correctness, completeness, formatting, and tone.

Step 3: Choose an automated scorer. We use a combination of approaches depending on the task:

• Exact match or F1: For classification, entity extraction, or any task with a single correct answer.
• LLM-as-judge: For open-ended generation, we use a strong model (Claude or GPT-4o) to score outputs against a rubric. Surprisingly reliable — correlation with human judges is consistently 0.7-0.85.
• Semantic similarity (BERTScore, BLEU, ROUGE): For summarization and translation tasks. Each has known biases, so we use them as directional indicators, not absolute scores.

Step 4: Spot-check with human review. Automated scoring misses nuance. We run a monthly audit where a human reviews 20 random samples from each model’s output to catch degradation that the automated judge missed. This has saved us multiple times when an LLM judge started scoring a model favorably for producing confidently wrong answers.

Step 5: Baseline against production. Before evaluating any new model, run your production model through your eval set and log the scores. MLflow’s experiment tracking is naturally suited for tracking benchmark results across model versions — you can log metrics, parameters, and artifacts for each benchmark run and compare them side by side. Set a threshold: the new model must beat the baseline by at least 1% on the primary metric or match it while reducing latency/cost by 20%+.

The entire process, after the initial setup, takes 30-45 minutes to run against a new model. We do this for every new release we consider in production.

A Practical Benchmarking Workflow

When a new open model drops (and they drop roughly weekly), here is the workflow we follow.

Hour 0: Quick gut check. Pull the model via Ollama or the smallest quantized variant via Hugging Face. Run 5-10 diverse prompts from memory. If the model hallucinates obviously or cannot follow basic instructions, stop — it is not ready, and we will revisit after the community posts finetuning recipes.

Hour 1-2: Public benchmark scan. Run the model through LM Eval Harness on MMLU and GSM8K for a capability baseline. Compare to our database of previous model runs. If the model is 3+ points behind our current production model on these benchmarks, we deprioritize it.

Hour 2-4: Internal eval run. Load the model through vLLM with our production inference configuration (same quantization, same batch size, same max tokens). Run our internal eval set. Log everything to MLflow: model name, quantization, eval set version, per-task scores, latency metrics, and generation outputs. Compare the results to the production baseline.

Hour 4+: Cost analysis. Measure throughput on our target hardware. Calculate cost per million tokens (including the amortized hardware cost). If the new model matches quality at 60% of the cost, it is probably worth a deeper evaluation. If it costs more, it needs to be substantially better in quality.

Week 1-2: Shadow deployment. Run the model alongside the production model on a fraction of live traffic. Compare user-facing metrics: task completion rate, user satisfaction, error rate. This catches things our eval set missed — typically 3-5 edge cases per deployment that the automated benchmark never considered.

Practical Takeaways

Benchmarking open source models fairly requires controlling for hardware, quantization, inference server, prompting, and sampling — all variables that proprietary APIs hide from us. Here is what we keep in mind:

• Run controlled comparisons on identical hardware with identical inference settings. Log everything so the comparison is reproducible.
• Quantize carefully. INT8 is nearly lossless. INT4 costs 1.5-2.5% on factual tasks and 2-4% on reasoning tasks. Know your use case’s sensitivity before deciding.
• Public benchmarks are useful filters, not deployment decisions. Use Chatbot Arena Elo and the Open LLM Leaderboard to decide which 2-3 models to evaluate deeply, not to choose your production model.
• Build a task-specific eval set with 50-100 real examples. It takes a few hours to create and pays off every time you evaluate a new model.
• Automate scoring but spot-check with humans. LLM-as-judge is remarkably good but misses failure modes unique to your domain.
• Track everything systematically. Logging model name, quantization level, inference engine version, task scores, and latency into a central experiment tracker — MLflow works great for this — is the only way to make decisions that accumulate over time rather than starting from scratch each time.
• Shadow-deploy before committing. Benchmarks tell us what a model can do on our eval set. Production tells us what it actually does with real users. Trust production data over benchmark scores every time.