Created bymario-alka.dePowered bynoob2claw.detricoma.de
Contributed byMario AlkaMeta

Llama-3.2-3B-Instruct

Performance benchmark · measured on 16.09.2026 16:33

Benchmark-IDrun-20260918-121614-b9e325
Timebench 3 - Kombi (Prefill + Generation)Dense3BRuntime: llama.cppQuantisierung: Q8_0
Generation70,69tok/s
Prefill3.839,56tok/s
Time to First Token645,00ms
Total duration11,11s
Concurrency1parallel
Ranking in the field
765of 1927 systems

Performance benchmark · Primary metric: Generation-Speed (tok/s) · 1× concurrent

This run is better than 60 % of all comparable systems.
Generation 70,7 tok/s
-9 % vs Ø 77,5
Prefill 3.839,6 tok/s
+35 % vs Ø 2.844,6
Time to First Token 645 ms
-97 % vs Ø 23.755
Distribution in the field0 – 471 tok/s
Ø 78 Median Dieser Lauf

Wie schlägt sich dieser Benchmark mit anderen Modellen?

How does this benchmark compare on other GPUs?

Same model on different hardware · 1× concurrent · Generation (tok/s)

Hardware

GPU: 2x NVIDIA GeForce RTX 2060 · 6 GB VRAM
CPU: 4x Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz
RAM: 504 GB
Mainboard: Dell Inc. PowerEdge R820

Setup

Runtime: llama.cpp
Quantization: Q8_0
Model: Llama-3.2-3B-Instruct

Configuration

benchmark-konfiguration — run-20260918-121614-b9e325
# LLM-Benchmark Konfiguration # Modell : Llama-3.2-3B-Instruct # Engine : llama.cpp # Run-ID : run-20260918-121614-b9e325 # GPU : 2x NVIDIA GeForce RTX 2060 # CPU : 4x Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz # RAM : 504 GB bench@llm-benchmark:~$ llama-server \ -m Llama-3.2-3B-Instruct-Q8_0.gguf \ -ngl 999 \ -fa on \ -c 32768 \ -np 12 \ -sm layer '(2x' RTX 2060 '6GB)'
Engine?Die Inferenz-Software, die das Modell ausliefert (z.B. vLLM oder llama.cpp). Sie bestimmt Geschwindigkeit, unterstuetzte Modellformate und welche Parameter ueberhaupt verfuegbar sind.llamacpp
Kontextlaenge?Maximale Anzahl Tokens (Eingabe + erzeugte Ausgabe zusammen), die das Modell pro Anfrage verarbeiten kann.32768
Modellpfad?Pfad zur GGUF-Modelldatei, die geladen und ausgeliefert wird.Llama-3.2-3B-Instruct-Q8_0.gguf
GPU-Layer?Anzahl der auf die GPU ausgelagerten Modell-Layer. Hoeher = mehr VRAM und schneller; der Rest laeuft auf der CPU. 999 = alles auf GPU.999
faon
Kontext?Groesse des Kontextfensters in Token. 0 = der beim Training verwendete Kontext des Modells.32768
np12
Split-Mode?Verteilung ueber mehrere GPUs: none (nur eine GPU), layer (Layer aufteilen) oder row (Tensoren zeilenweise).layer
execution_typelocal

All benchmarks of this model To leaderboard

Anzeige
Model comparison

Llama-3.2-3B-Instruct on various hardware

All published performance runs of this model – each bubble a variant: position = prefill (X) × generation (Y), bubble size = number of runs. Closer to the top right = faster. ★ Marked gold = this benchmark.

GPUby graphics card

5934452961480,005.81511.63117.446Prefill (tok/s)Generation (tok/s)AMD Radeon RX 7900 XTX - 465,7 tok/s Generation, 14.196 tok/s Prefill, TTFT 1.049 ms (9 Laufe)AMD Radeon RX 7900 XTXAMD Radeon AI PRO R9700 - 397,2 tok/s Generation, 11.893 tok/s Prefill, TTFT 1.135 ms (3 Laufe)AMD Radeon AI PRO R97...CPU-only - 62,9 tok/s Generation, 870 tok/s Prefill, TTFT 163.072 ms (5 Laufe)CPU-onlyNVIDIA GeForce RTX 2060 - 226,6 tok/s Generation, 6.927 tok/s Prefill, TTFT 2.004 ms (6 Laufe) | DIESER LAUF★ NVIDIA GeForce RTX 20...
AMD Radeon RX 7900 XTX 465,7 tok/sAMD Radeon AI PRO R9700 397,2 tok/s★ NVIDIA GeForce RTX 2060 226,6 tok/s this runCPU-only 62,9 tok/s

CPUby processor

6054543021510,006.36012.71919.079Prefill (tok/s)Generation (tok/s)AMD Ryzen Threadripper PRO 3955WX 16-Cores - 465,7 tok/s Generation, 15.391 tok/s Prefill, TTFT 992 ms (6 Laufe)AMD Ryzen Threadrippe...AMD Ryzen 3 3100 4-Core Processor - 397,2 tok/s Generation, 11.893 tok/s Prefill, TTFT 1.135 ms (3 Laufe)AMD Ryzen 3 3100 4-Co...Apple M6 (unified CPU/GPU SoC, ARM64, AdvSIMD) - 62,9 tok/s Generation, 2.562 tok/s Prefill, TTFT 1.026 ms (1 Lauf)Apple M6 (unified CPU...Apple M6 (unified CPU/GPU SoC) - 52,9 tok/s Generation, 1.698 tok/s Prefill, TTFT 1.508 ms (1 Lauf)Apple M6 (unified CPU...4 x Cortex-A78C + 4 x Cortex-A55 - 2,5 tok/s Generation, 29 tok/s Prefill, TTFT 270.941 ms (3 Laufe)4 x Cortex-A78C + 4 x...Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz - 354,2 tok/s Generation, 8.553 tok/s Prefill, TTFT 1.723 ms (9 Laufe) | DIESER LAUF★ Intel(R) Xeon(R) CPU ...
AMD Ryzen Threadripper PRO 3955WX 16-Cores 465,7 tok/sAMD Ryzen 3 3100 4-Core Processor 397,2 tok/s★ Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz 354,2 tok/s this runApple M6 (unified CPU/GPU SoC, ARM64, AdvSIMD) 62,9 tok/sApple M6 (unified CPU/GPU SoC) 52,9 tok/s4 x Cortex-A78C + 4 x Cortex-A55 2,5 tok/s

MBby mainboard

6054543021510,006.36012.71919.079Prefill (tok/s)Generation (tok/s)ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI - 465,7 tok/s Generation, 15.391 tok/s Prefill, TTFT 992 ms (6 Laufe)ASUSTeK COMPUTER INC....ASUSTeK COMPUTER INC. ROG STRIX X570-F GAMING - 397,2 tok/s Generation, 11.893 tok/s Prefill, TTFT 1.135 ms (3 Laufe)ASUSTeK COMPUTER INC....Apple Mac mini - 62,9 tok/s Generation, 2.130 tok/s Prefill, TTFT 1.267 ms (2 Laufe)Apple Mac miniunbekannt - 2,5 tok/s Generation, 29 tok/s Prefill, TTFT 270.941 ms (3 Laufe)unbekanntDell Inc. PowerEdge R820 - 354,2 tok/s Generation, 8.553 tok/s Prefill, TTFT 1.723 ms (9 Laufe) | DIESER LAUF★ Dell Inc. PowerEdge R...
ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI 465,7 tok/sASUSTeK COMPUTER INC. ROG STRIX X570-F GAMING 397,2 tok/s★ Dell Inc. PowerEdge R820 354,2 tok/s this runApple Mac mini 62,9 tok/sunbekannt 2,5 tok/s

ENGby engine

5954462971490,006.87513.75020.624Prefill (tok/s)Generation (tok/s)vLLM - 465,7 tok/s Generation, 16.879 tok/s Prefill, TTFT 1.016 ms (3 Laufe)vLLMlmstudio-mlx - 62,9 tok/s Generation, 2.562 tok/s Prefill, TTFT 1.026 ms (1 Lauf)lmstudio-mlxOllama - 52,9 tok/s Generation, 1.698 tok/s Prefill, TTFT 1.508 ms (1 Lauf)Ollamallama.cpp - 397,2 tok/s Generation, 8.581 tok/s Prefill, TTFT 46.369 ms (18 Laufe) | DIESER LAUF★ llama.cpp
vLLM 465,7 tok/s★ llama.cpp 397,2 tok/s this runlmstudio-mlx 62,9 tok/sOllama 52,9 tok/s

DRVby driver

5124894664424198.5568.9209.2849.648Prefill (tok/s)Generation (tok/s)unbekannt - 465,7 tok/s Generation, 9.102 tok/s Prefill, TTFT 36.532 ms (23 Laufe)unbekannt
unbekannt 465,7 tok/s
💰 Economics

Economics of this run

Operating cost, TCO and comparison with the next-best runs of the same model at identical concurrency (1× concurrent). Methodology →

⚙️ ConfigurationAll metrics and charts below follow these settings – based on a 24-month runtime.Save to URLReset
⚡ Electricity price EUR/kWh
⚙️ System utilization 100 %
🖥️ Acquisition EUR
🔌 Idle 55 W
⚡ TDP 359 W
☁️ External LLM (API)
Electricity0.30 EUR/kWh
Avg power (incl. idle)359 W estimated (TDP)GPU 320 + CPU 29 + Board 10 W full load
Avg cost / hourEUR 0.11
Electricity / 1M tokensEUR 0.42
Token / kWh709.36K
Acquisition (system)EUR 2,445 partial priceGPU EUR 220 · CPU EUR 59 · RAM EUR 2,016 · PSU EUR 150
Electricity (2 years)–
TCO (2 years)EUR 4,331
Output tokens (2 years)4.46B
☁️ External LLM (API) – comparison
External LLM cost (2 years)–
Savings vs. external (2 years)–

All values above and the charts below take the configured system utilization into account: at X% the system generates only X% of the time, the rest it idles (55 W). Cost per hour drops (more idle), cost per token rises.

Cost over 2 years – electricity only

Cost over 2 years – incl. acquisition (TCO)

Speed vs. tokens per euro

Euro per 1M tokens

Comparison vs. API – economics per benchmark

Llama-3.2-3B-Instruct2x NVIDIA GeForce RTX 2060Llama-3.2-3B-Instruct2x AMD Radeon RX 7900 XTXLlama-3.2-3B-Instruct2x AMD Radeon RX 7900 XTXLlama-3.2-3B-InstructAMD Radeon AI PRO R9700
Electricity cost (24 mo.)––––
Acquisition cost––––
Total cost (TCO)––––
Generated tokens (24 mo.)––––
Token price via API––––
Break-even point (days)––––
Result (savings / extra cost)––––

Comparison with up to 3 next-best runs of this model at the same concurrency (at least one on different hardware). Power = GPU TDP + CPU (idle + 15 %) + board (estimated), acquisition = full system (GPU + CPU + board + RAM + PSU), prices = stored market prices.

Contributed by

Mario Alka Administrator

@marioalka

Ich bin Unternehmer, Softwareentwickler und KI-Enthusiast. Seit vielen Jahren entwickle ich Unternehmenssoftware und beschäftige mich inzwischen fast täglich mit lokalen LLMs, KI-Agenten und leistungsfähiger KI-Hardware.

Mit LLM-Benchmark.de möchte ich eine Plattform schaffen, auf der Modelle, GPUs und Agenten objektiv und reproduzierbar miteinander verglichen werden.