Methodology
Every number on this site comes from bench/runner.py in the site's repository. The harness is about 350 lines of dependency-free Python, and it is the same file on every machine — no per-machine tweaks, because a benchmark you edit per machine is not a benchmark.
What is measured
The harness talks to an OpenAI-compatible /v1/chat/completionsendpoint and times the response stream from the client side. Measuring client-side is deliberate: llama.cpp, ollama, vLLM and MLX all report internal statistics differently, and some do not report them at all. Timing the stream is the only method that produces numbers comparable across all of them.
- Time to first token (TTFT) — wall-clock from request dispatch to the first content token arriving. This is what makes a model feel responsive or sluggish.
- Decode throughput — tokens per second during generation, measured across the window from the first token to the last. The first token is excluded from the count because it is produced at the start of that window, not during it.
- Prefill throughput — prompt tokens divided by TTFT. This is the number that collapses on memory-bandwidth-limited machines once the context gets long.
How runs are conducted
- Every scenario runs one warmup pass that is discarded, then five measured repetitions. The warmup exists to absorb weight loading and cache effects that would otherwise land entirely on the first measurement.
- Reported values are medians, with standard deviation shown alongside. A median resists the single slow run that thermal throttling or a background process will eventually produce.
- Decoding is greedy (temperature 0) so output length stays stable between repetitions. Throughput compared across runs of different lengths is not a comparison.
- Prompts are built from a fixed filler string at target token counts, so every machine prefills the identical input.
The four scenarios
| Scenario | In | Out | What it exposes |
|---|---|---|---|
chat_short | 128 | 256 | Interactive feel; latency-dominated |
chat_long_context | 8192 | 256 | RAG and long-document work |
code_generation | 512 | 1024 | Agentic coding; decode-dominated |
summarization | 16384 | 128 | Worst case for weak prefill |
Scenarios are frozen. Adding one is fine; editing an existing one would silently invalidate every previously published comparison, so it does not happen.
What these numbers are not
Single-stream benchmarks. They describe one person using one model interactively — the actual local-LLM use case — and they say nothing about throughput under concurrent load, where batching changes the picture entirely. They also reflect specific driver, backend and model versions, all recorded in each result file. A rerun six months later on the same hardware can legitimately produce different numbers, which is why the backend version is published alongside every result.
Reproducing them
# On the machine under test, with a server already running:
python3 bench/runner.py \
--url http://localhost:11434 \
--model llama3.1:8b \
--label my-machine \
--quant Q4_K_MThe result lands in data/runs/ as JSON. Those files are the primary record — the tables on this site are rendered from them at build time, so a published number and its source file cannot disagree.