Created bymario-alka.dePowered bynoob2claw.detricoma.de
Meta

Meta-Llama-3.1-8B-Instruct8B

Benchmark profile and published results.

Dense8B
Position in the field

Best values compared

Best metrics of this model against the minimum, average and maximum of all published systems.

Generation1.593,1 tok/s
Min 0,0Ø 245,7Max 3.484,3
Ueber dem Durchschnitt · 4971 Systeme im Feld
Prefill27.508 tok/s
Min 4Ø 4.892Max 112.346
Ueber dem Durchschnitt · 4953 Systeme im Feld
Time to First Token219 ms
Min 20Ø 31.420Max 535.235
Ueber dem Durchschnitt · 4968 Systeme im Feld
Performance profile

Throughput by hardware & engine

Each bubble is a GPU, CPU or engine – position shows prompt processing (X) and output speed (Y), bubble size the number of runs.

GPUby graphics card

2.0651.5491.0335160,006.00212.00418.006Prefill (tok/s)Generation (tok/s)NVIDIA GeForce RTX 5090 - 1.593,1 tok/s Generation, 11.858 tok/s Prefill, TTFT 3.236 ms (3 Laufe)NVIDIA GeForce RTX 50...NVIDIA RTX PRO 6000 Blackwell Workstation Edition - 1.533,0 tok/s Generation, 13.794 tok/s Prefill, TTFT 3.430 ms (5 Laufe)NVIDIA RTX PRO 6000 B...NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition - 1.433,0 tok/s Generation, 14.214 tok/s Prefill, TTFT 3.334 ms (9 Laufe)NVIDIA RTX PRO 6000 B...NVIDIA RTX A6000 - 1.026,4 tok/s Generation, 10.917 tok/s Prefill, TTFT 1.871 ms (33 Laufe)NVIDIA RTX A6000NVIDIA GeForce RTX 5070 Ti - 913,4 tok/s Generation, 6.511 tok/s Prefill, TTFT 5.168 ms (3 Laufe)NVIDIA GeForce RTX 50...NVIDIA GeForce RTX 3090 Ti - 837,9 tok/s Generation, 8.047 tok/s Prefill, TTFT 5.721 ms (3 Laufe)NVIDIA GeForce RTX 30...AMD Radeon AI PRO R9700 - 518,3 tok/s Generation, 7.911 tok/s Prefill, TTFT 6.402 ms (13 Laufe)AMD Radeon AI PRO R97...NVIDIA GeForce RTX 4090 - 489,7 tok/s Generation, 14.571 tok/s Prefill, TTFT 1.424 ms (3 Laufe)NVIDIA GeForce RTX 40...AMD Radeon RX 7900 XTX - 405,0 tok/s Generation, 6.968 tok/s Prefill, TTFT 2.200 ms (12 Laufe)AMD Radeon RX 7900 XTXNVIDIA GeForce RTX 3090 - 385,9 tok/s Generation, 9.762 tok/s Prefill, TTFT 1.486 ms (6 Laufe)NVIDIA GeForce RTX 30...AMD Radeon PRO W7800 48GB - 323,1 tok/s Generation, 3.934 tok/s Prefill, TTFT 3.254 ms (17 Laufe)AMD Radeon PRO W7800 ...NVIDIA GeForce RTX 2060 - 169,0 tok/s Generation, 2.296 tok/s Prefill, TTFT 5.126 ms (10 Laufe)NVIDIA GeForce RTX 20...NVIDIA Tesla V100 PCIe 32GB - 122,6 tok/s Generation, 4.984 tok/s Prefill, TTFT 3.619 ms (3 Laufe)NVIDIA Tesla V100 PCI...NVIDIA Tesla P100 PCIe 16GB - 67,6 tok/s Generation, 864 tok/s Prefill, TTFT 15.746 ms (3 Laufe)NVIDIA Tesla P100 PCI...Intel Arc Pro B70 - 66,4 tok/s Generation, 860 tok/s Prefill, TTFT 2.539 ms (2 Laufe)Intel Arc Pro B70CPU-only - 29,2 tok/s Generation, 342 tok/s Prefill, TTFT 149.386 ms (5 Laufe)CPU-only
NVIDIA GeForce RTX 5090 1.593,1 tok/sNVIDIA RTX PRO 6000 Blackwell Workstation Edition 1.533,0 tok/sNVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition 1.433,0 tok/sNVIDIA RTX A6000 1.026,4 tok/sNVIDIA GeForce RTX 5070 Ti 913,4 tok/sNVIDIA GeForce RTX 3090 Ti 837,9 tok/sAMD Radeon AI PRO R9700 518,3 tok/sNVIDIA GeForce RTX 4090 489,7 tok/sAMD Radeon RX 7900 XTX 405,0 tok/sNVIDIA GeForce RTX 3090 385,9 tok/sAMD Radeon PRO W7800 48GB 323,1 tok/sNVIDIA GeForce RTX 2060 169,0 tok/sNVIDIA Tesla V100 PCIe 32GB 122,6 tok/sNVIDIA Tesla P100 PCIe 16GB 67,6 tok/sIntel Arc Pro B70 66,4 tok/sCPU-only 29,2 tok/s

CPUby processor

2.0711.5531.0355180,005.87511.75017.624Prefill (tok/s)Generation (tok/s)AMD Ryzen 7 5800X3D 8-Core Processor - 1.593,1 tok/s Generation, 11.858 tok/s Prefill, TTFT 3.236 ms (3 Laufe)AMD Ryzen 7 5800X3D 8...AMD Ryzen 9 9950X 16-Core Processor - 1.533,0 tok/s Generation, 13.472 tok/s Prefill, TTFT 2.895 ms (3 Laufe)AMD Ryzen 9 9950X 16-...AMD Ryzen 5 5600X 6-Core Processor - 1.528,7 tok/s Generation, 10.891 tok/s Prefill, TTFT 2.173 ms (8 Laufe)AMD Ryzen 5 5600X 6-C...AMD Ryzen Threadripper PRO 9965WX 24-Cores - 1.433,0 tok/s Generation, 14.214 tok/s Prefill, TTFT 3.334 ms (9 Laufe)AMD Ryzen Threadrippe...AMD Ryzen Threadripper PRO 7955WX 16-Cores - 1.026,4 tok/s Generation, 10.247 tok/s Prefill, TTFT 3.368 ms (40 Laufe)AMD Ryzen Threadrippe...AMD Ryzen Threadripper PRO 5975WX 32-Cores - 913,4 tok/s Generation, 4.320 tok/s Prefill, TTFT 3.541 ms (20 Laufe)AMD Ryzen Threadrippe...AMD Ryzen 9 8945HX with Radeon Graphics - 837,9 tok/s Generation, 8.047 tok/s Prefill, TTFT 5.721 ms (3 Laufe)AMD Ryzen 9 8945HX wi...AMD Ryzen 3 3100 4-Core Processor - 489,7 tok/s Generation, 10.767 tok/s Prefill, TTFT 1.731 ms (6 Laufe)AMD Ryzen 3 3100 4-Co...AMD Ryzen Threadripper PRO 3955WX 16-Cores - 405,0 tok/s Generation, 6.991 tok/s Prefill, TTFT 2.254 ms (9 Laufe)AMD Ryzen Threadrippe...AMD EPYC 7203P 8-Core Processor - 317,0 tok/s Generation, 10.774 tok/s Prefill, TTFT 1.374 ms (3 Laufe)AMD EPYC 7203P 8-Core...Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz - 266,7 tok/s Generation, 4.531 tok/s Prefill, TTFT 5.584 ms (6 Laufe)Intel(R) Xeon(R) CPU ...Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz - 130,9 tok/s Generation, 2.353 tok/s Prefill, TTFT 3.410 ms (7 Laufe)Intel(R) Core(TM) i5-...AMD Ryzen 3 PRO 3200GE w/ Radeon Vega Graphics - 122,6 tok/s Generation, 4.984 tok/s Prefill, TTFT 3.619 ms (3 Laufe)AMD Ryzen 3 PRO 3200G...AMD Ryzen 9 7945HX with Radeon Graphics - 67,6 tok/s Generation, 862 tok/s Prefill, TTFT 10.463 ms (5 Laufe)AMD Ryzen 9 7945HX wi...Apple M6 (unified CPU/GPU SoC, ARM64, AdvSIMD) - 29,2 tok/s Generation, 1.076 tok/s Prefill, TTFT 2.447 ms (1 Lauf)Apple M6 (unified CPU...Apple M6 (unified CPU/GPU SoC) - 26,1 tok/s Generation, 582 tok/s Prefill, TTFT 4.250 ms (1 Lauf)Apple M6 (unified CPU...Intel(R) Xeon(R) CPU E5-4620 v2 @ 2.60GHz - 3,6 tok/s Generation, 36 tok/s Prefill, TTFT 59.044 ms (1 Lauf)Intel(R) Xeon(R) CPU ...4 x Cortex-A78C + 4 x Cortex-A55 - 1,2 tok/s Generation, 8 tok/s Prefill, TTFT 340.594 ms (2 Laufe)4 x Cortex-A78C + 4 x...
AMD Ryzen 7 5800X3D 8-Core Processor 1.593,1 tok/sAMD Ryzen 9 9950X 16-Core Processor 1.533,0 tok/sAMD Ryzen 5 5600X 6-Core Processor 1.528,7 tok/sAMD Ryzen Threadripper PRO 9965WX 24-Cores 1.433,0 tok/sAMD Ryzen Threadripper PRO 7955WX 16-Cores 1.026,4 tok/sAMD Ryzen Threadripper PRO 5975WX 32-Cores 913,4 tok/sAMD Ryzen 9 8945HX with Radeon Graphics 837,9 tok/sAMD Ryzen 3 3100 4-Core Processor 489,7 tok/sAMD Ryzen Threadripper PRO 3955WX 16-Cores 405,0 tok/sAMD EPYC 7203P 8-Core Processor 317,0 tok/sIntel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz 266,7 tok/sIntel(R) Core(TM) i5-7400 CPU @ 3.00GHz 130,9 tok/sAMD Ryzen 3 PRO 3200GE w/ Radeon Vega Graphics 122,6 tok/sAMD Ryzen 9 7945HX with Radeon Graphics 67,6 tok/sApple M6 (unified CPU/GPU SoC, ARM64, AdvSIMD) 29,2 tok/sApple M6 (unified CPU/GPU SoC) 26,1 tok/sIntel(R) Xeon(R) CPU E5-4620 v2 @ 2.60GHz 3,6 tok/s4 x Cortex-A78C + 4 x Cortex-A55 1,2 tok/s

ENGby engine

2.0661.5491.0335160,004.3188.63512.953Prefill (tok/s)Generation (tok/s)llama.cpp - 1.593,1 tok/s Generation, 7.191 tok/s Prefill, TTFT 12.079 ms (91 Laufe)llama.cppvLLM - 1.026,4 tok/s Generation, 10.530 tok/s Prefill, TTFT 2.294 ms (36 Laufe)vLLMunbekannt - 42,1 tok/s Generation, 821 tok/s Prefill, TTFT 2.298 ms (1 Lauf)unbekanntlmstudio-mlx - 29,2 tok/s Generation, 1.076 tok/s Prefill, TTFT 2.447 ms (1 Lauf)lmstudio-mlxOllama - 26,1 tok/s Generation, 582 tok/s Prefill, TTFT 4.250 ms (1 Lauf)Ollama
llama.cpp 1.593,1 tok/svLLM 1.026,4 tok/sunbekannt 42,1 tok/slmstudio-mlx 29,2 tok/sOllama 26,1 tok/s
Throughput & latency

Performance benchmark

Metric:
GPUs
#Model / MakerMetricsParallelGPU / CPU / RAMRuntime
1Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.593,10 tok/s TG
Prefill 16.376 · TTFT 6.849 ms
10×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMllama.cppcodex_cliQ4_K_M
2Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.532,95 tok/s TG
Prefill 17.861 · TTFT 6.225 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMllama.cppgodclawQ4_K_M
3Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.528,71 tok/s TG
Prefill 17.513 · TTFT 6.225 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 5 5600X 6-Core Processor · 31 GB RAMllama.cppgodclawQ4_K_M
4Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.432,98 tok/s TG
Prefill 21.012 · TTFT 6.892 ms
10×3x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation EditionAMD Ryzen Threadripper PRO 9965WX 24-Cores · 125 GB RAMllama.cppgodclawQ4_K_M
5Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.383,23 tok/s TG
Prefill 17.796 · TTFT 7.072 ms
10×3x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation EditionAMD Ryzen Threadripper PRO 9965WX 24-Cores · 125 GB RAMllama.cppgodclawQ4_K_M
6Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.267,62 tok/s TG
Prefill 13.497 · TTFT 8.310 ms
10×3x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation EditionAMD Ryzen Threadripper PRO 9965WX 24-Cores · 125 GB RAMllama.cppgodclawQ4_K_M
7Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.026,44 tok/s TG
Prefill 26.805 · TTFT 1.371 ms
10×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQ
8Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
913,39 tok/s TG
Prefill 8.119 · TTFT 11.247 ms
10×NVIDIA GeForce RTX 5070 TiAMD Ryzen Threadripper PRO 5975WX 32-Cores · 247 GB RAMllama.cppgodclawQ4_K_M
9Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
876,87 tok/s TG
Prefill 11.325 · TTFT 2.242 ms
5×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMllama.cppgodclawQ4_K_M
10Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
869,38 tok/s TG
Prefill 11.044 · TTFT 2.242 ms
5×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 5 5600X 6-Core Processor · 31 GB RAMllama.cppgodclawQ4_K_M
11Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
837,85 tok/s TG
Prefill 9.559 · TTFT 12.952 ms
10×NVIDIA GeForce RTX 3090 TiAMD Ryzen 9 8945HX with Radeon Graphics · 92 GB RAMllama.cppwebsocketQ4_K_M
12Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
832,96 tok/s TG
Prefill 27.508 · TTFT 1.327 ms
10×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8
13Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
823,64 tok/s TG
Prefill 12.521 · TTFT 2.492 ms
5×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMllama.cppcodex_cliQ4_K_M
14Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
774,67 tok/s TG
Prefill 20.868 · TTFT 2.009 ms
5×3x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation EditionAMD Ryzen Threadripper PRO 9965WX 24-Cores · 125 GB RAMllama.cppgodclawQ4_K_M
15Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
766,19 tok/s TG
Prefill 15.268 · TTFT 2.352 ms
5×3x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation EditionAMD Ryzen Threadripper PRO 9965WX 24-Cores · 125 GB RAMllama.cppgodclawQ4_K_M
16Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
699,71 tok/s TG
Prefill 13.042 · TTFT 2.524 ms
5×3x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation EditionAMD Ryzen Threadripper PRO 9965WX 24-Cores · 125 GB RAMllama.cppgodclawQ4_K_M
17Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
675,29 tok/s TG
Prefill 20.479 · TTFT 781 ms
5×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQ
18Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
576,78 tok/s TG
Prefill 14.305 · TTFT 2.330 ms
10×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMopenclaw_cliAWQ
19Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
534,82 tok/s TG
Prefill 11.826 · TTFT 7.114 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
20Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
518,27 tok/s TG
Prefill 13.775 · TTFT 18.702 ms
10×3x AMD Radeon AI PRO R9700AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
21Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
510,30 tok/s TG
Prefill 21.461 · TTFT 752 ms
5×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8
22Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
508,08 tok/s TG
Prefill 5.204 · TTFT 21.109 ms
10×3x AMD Radeon AI PRO R9700AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
23Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
499,39 tok/s TG
Prefill 11.446 · TTFT 7.175 ms
10×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
24Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
489,72 tok/s TG
Prefill 21.352 · TTFT 2.183 ms
10×NVIDIA GeForce RTX 4090AMD Ryzen 3 3100 4-Core Processor · 61 GB RAMvLLMopenclaw_cliAWQ
25Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
489,71 tok/s TG
Prefill 8.631 · TTFT 3.252 ms
5×NVIDIA GeForce RTX 5070 TiAMD Ryzen Threadripper PRO 5975WX 32-Cores · 247 GB RAMllama.cppgodclawQ4_K_M
26Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
489,43 tok/s TG
Prefill 10.618 · TTFT 18.898 ms
10×3x AMD Radeon AI PRO R9700AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
27Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
482,61 tok/s TG
Prefill 12.125 · TTFT 5.913 ms
10×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ8_0
28Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
467,83 tok/s TG
Prefill 11.804 · TTFT 6.038 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ8_0
29Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
456,70 tok/s TG
Prefill 8.966 · TTFT 3.773 ms
5×NVIDIA GeForce RTX 3090 TiAMD Ryzen 9 8945HX with Radeon Graphics · 92 GB RAMllama.cppwebsocketQ4_K_M
30Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
444,58 tok/s TG
Prefill 15.076 · TTFT 2.208 ms
10×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMopenclaw_cliFP8
31Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
404,99 tok/s TG
Prefill 14.538 · TTFT 2.823 ms
10×2x AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMvLLMgodclaw
32Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
387,59 tok/s TG
Prefill 10.645 · TTFT 1.347 ms
5×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMopenclaw_cliAWQ
33Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
385,85 tok/s TG
Prefill 16.933 · TTFT 1.925 ms
10×NVIDIA GeForce RTX 3090 iAMD Ryzen 5 5600X 6-Core Processor · 31 GB RAMvLLMgodclaw
34Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
356,01 tok/s TG
Prefill 10.663 · TTFT 3.085 ms
10×NVIDIA GeForce RTX 3090 iAMD Ryzen 5 5600X 6-Core Processor · 31 GB RAMvLLMgodclaw
35Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
343,70 tok/s TG
Prefill 9.999 · TTFT 3.230 ms
10×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppopenclaw_cliQ4_K_M
36Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
323,12 tok/s TG
Prefill 5.337 · TTFT 5.911 ms
10×2x AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMllama.cppgodclawQ8_0
37Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
322,77 tok/s TG
Prefill 5.717 · TTFT 5.160 ms
10×2x AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMllama.cppgodclawQ4_K_M
38Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
321,67 tok/s TG
Prefill 7.148 · TTFT 4.460 ms
10×AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMllama.cppgodclawQ4_K_M
39Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
317,00 tok/s TG
Prefill 14.849 · TTFT 2.492 ms
10×2x NVIDIA RTX A6000 iAMD EPYC 7203P 8-Core Processor · 126 GB RAMvLLMgodclaw
40Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
300,93 tok/s TG
Prefill 6.663 · TTFT 4.522 ms
10×AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMllama.cppgodclawQ8_0
41Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
281,80 tok/s TG
Prefill 10.273 · TTFT 5.323 ms
5×3x AMD Radeon AI PRO R9700AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
42Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
279,01 tok/s TG
Prefill 11.575 · TTFT 1.408 ms
5×2x AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMvLLMgodclaw
43Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
276,49 tok/s TG
Prefill 9.874 · TTFT 3.124 ms
10×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppopenclaw_cliQ8_0
44Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
275,72 tok/s TG
Prefill 13.406 · TTFT 4.952 ms
5×3x AMD Radeon AI PRO R9700AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
45Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
274,94 tok/s TG
Prefill 6.188 · TTFT 6.209 ms
5×3x AMD Radeon AI PRO R9700AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
46Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
274,89 tok/s TG
Prefill 8.972 · TTFT 1.446 ms
5×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
47Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
272,75 tok/s TG
Prefill 11.264 · TTFT 1.275 ms
5×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMopenclaw_cliFP8
48Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
266,70 tok/s TG
Prefill 10.161 · TTFT 3.360 ms
10×2x AMD Radeon RX 7900 XTX i4x Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz · 504 GB RAMllama.cppgodclawQ4_K_M
49Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
262,93 tok/s TG
Prefill 8.472 · TTFT 4.535 ms
10×2x AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMvLLMgodclawAWQ
50Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
258,77 tok/s TG
Prefill 9.631 · TTFT 3.487 ms
10×AMD Radeon AI PRO R9700AMD Ryzen 3 3100 4-Core Processor · 31 GB RAMllama.cppopenclaw_cliQ8_0
51Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
254,69 tok/s TG
Prefill 14.644 · TTFT 1.771 ms
5×NVIDIA GeForce RTX 4090AMD Ryzen 3 3100 4-Core Processor · 61 GB RAMvLLMopenclaw_cliAWQ
52Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
253,38 tok/s TG
Prefill 9.752 · TTFT 1.444 ms
5×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
53Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
246,17 tok/s TG
Prefill 6.676 · TTFT 369 ms
1×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMllama.cppcodex_cliQ4_K_M
54Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
239,32 tok/s TG
Prefill 11.230 · TTFT 219 ms
1×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMllama.cppgodclawQ4_K_M
55Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
237,04 tok/s TG
Prefill 7.169 · TTFT 1.831 ms
5×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppopenclaw_cliQ4_K_M
56Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
230,12 tok/s TG
Prefill 5.654 · TTFT 2.560 ms
5×AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMllama.cppgodclawQ4_K_M
57Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
225,01 tok/s TG
Prefill 9.669 · TTFT 257 ms
1×3x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation EditionAMD Ryzen Threadripper PRO 9965WX 24-Cores · 125 GB RAMllama.cppgodclawQ4_K_M
58Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
223,20 tok/s TG
Prefill 12.589 · TTFT 1.123 ms
5×NVIDIA GeForce RTX 3090 iAMD Ryzen 5 5600X 6-Core Processor · 31 GB RAMvLLMgodclaw
59Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
222,93 tok/s TG
Prefill 8.663 · TTFT 286 ms
1×3x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation EditionAMD Ryzen Threadripper PRO 9965WX 24-Cores · 125 GB RAMllama.cppgodclawQ4_K_M
60Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
218,94 tok/s TG
Prefill 5.830 · TTFT 2.480 ms
5×AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMllama.cppgodclawQ8_0
61Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
212,79 tok/s TG
Prefill 8.116 · TTFT 304 ms
1×3x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation EditionAMD Ryzen Threadripper PRO 9965WX 24-Cores · 125 GB RAMllama.cppgodclawQ4_K_M
62Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
210,39 tok/s TG
Prefill 7.898 · TTFT 1.797 ms
5×NVIDIA GeForce RTX 3090 iAMD Ryzen 5 5600X 6-Core Processor · 31 GB RAMvLLMgodclaw
63Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
208,70 tok/s TG
Prefill 9.730 · TTFT 1.396 ms
5×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ8_0
64Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
197,82 tok/s TG
Prefill 3.730 · TTFT 3.658 ms
5×2x AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMllama.cppgodclawQ8_0
65Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
197,34 tok/s TG
Prefill 6.940 · TTFT 1.849 ms
5×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppopenclaw_cliQ8_0
66Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
189,29 tok/s TG
Prefill 10.070 · TTFT 1.358 ms
5×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ8_0
67Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
183,35 tok/s TG
Prefill 11.565 · TTFT 1.219 ms
5×2x NVIDIA RTX A6000 iAMD EPYC 7203P 8-Core Processor · 126 GB RAMvLLMgodclaw
68Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
181,22 tok/s TG
Prefill 9.264 · TTFT 266 ms
1×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQ
69Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
168,96 tok/s TG
Prefill 3.010 · TTFT 18.082 ms
10×2x NVIDIA GeForce RTX 20604x Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz · 504 GB RAMvLLMgodclawAWQ
70Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
159,48 tok/s TG
Prefill 4.475 · TTFT 2.882 ms
5×2x AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMllama.cppgodclawQ4_K_M
71Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
156,61 tok/s TG
Prefill 3.757 · TTFT 4.751 ms
10×AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMllama.cppgodclawQ4_K_M
72Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
155,73 tok/s TG
Prefill 4.994 · TTFT 7.150 ms
10×AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMvLLMgodclawAWQ
73Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
152,06 tok/s TG
Prefill 7.328 · TTFT 1.967 ms
5×2x AMD Radeon RX 7900 XTX i4x Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz · 504 GB RAMllama.cppgodclawQ4_K_M
74Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
151,97 tok/s TG
Prefill 7.125 · TTFT 2.029 ms
5×AMD Radeon AI PRO R9700AMD Ryzen 3 3100 4-Core Processor · 31 GB RAMllama.cppopenclaw_cliQ8_0
75Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
148,21 tok/s TG
Prefill 5.615 · TTFT 439 ms
1×NVIDIA GeForce RTX 3090 TiAMD Ryzen 9 8945HX with Radeon Graphics · 92 GB RAMllama.cppwebsocketQ4_K_M
76Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
146,19 tok/s TG
Prefill 6.653 · TTFT 2.534 ms
5×2x AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMvLLMgodclawAWQ
77Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
140,90 tok/s TG
Prefill 2.783 · TTFT 1.005 ms
1×NVIDIA GeForce RTX 5070 TiAMD Ryzen Threadripper PRO 5975WX 32-Cores · 247 GB RAMllama.cppgodclawQ4_K_M
78Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
138,90 tok/s TG
Prefill 3.102 · TTFT 4.748 ms
5×AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMllama.cppgodclawQ8_0
79Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
130,88 tok/s TG
Prefill 4.122 · TTFT 7.946 ms
10×2x NVIDIA GeForce RTX 2060Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz · 7 GB RAMllama.cppopenclaw_cliQ4_K_M
80Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
127,43 tok/s TG
Prefill 9.480 · TTFT 260 ms
1×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8
81Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
122,88 tok/s TG
Prefill 2.387 · TTFT 7.059 ms
5×2x NVIDIA GeForce RTX 20604x Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz · 504 GB RAMvLLMgodclawAWQ
82Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
122,55 tok/s TG
Prefill 5.151 · TTFT 2.489 ms
5×NVIDIA Tesla V100 PCIe 32GBAMD Ryzen 3 PRO 3200GE w/ Radeon Vega Graphics · 61 GB RAMllama.cppgodclawQ8_0
83Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
122,07 tok/s TG
Prefill 3.576 · TTFT 3.191 ms
5×AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMllama.cppgodclawQ4_K_M
84Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
116,23 tok/s TG
Prefill 4.923 · TTFT 502 ms
1×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
85Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
116,02 tok/s TG
Prefill 4.973 · TTFT 496 ms
1×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
86Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
108,60 tok/s TG
Prefill 3.037 · TTFT 811 ms
1×AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMllama.cppgodclawQ4_K_M
87Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
102,47 tok/s TG
Prefill 4.866 · TTFT 508 ms
1×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppopenclaw_cliQ4_K_M
88Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
102,39 tok/s TG
Prefill 5.871 · TTFT 419 ms
1×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMopenclaw_cliAWQ
89Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
90,37 tok/s TG
Prefill 3.202 · TTFT 4.555 ms
5×2x NVIDIA GeForce RTX 2060Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz · 7 GB RAMllama.cppopenclaw_cliQ4_K_M
90Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
89,46 tok/s TG
Prefill 3.210 · TTFT 4.511 ms
5×2x NVIDIA GeForce RTX 2060Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz · 7 GB RAMllama.cppopenclaw_cliQ8_0
91Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
87,92 tok/s TG
Prefill 4.116 · TTFT 600 ms
1×3x AMD Radeon AI PRO R9700AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
92Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
87,03 tok/s TG
Prefill 3.930 · TTFT 629 ms
1×3x AMD Radeon AI PRO R9700AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
93Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
85,98 tok/s TG
Prefill 6.398 · TTFT 386 ms
1×3x AMD Radeon AI PRO R9700AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
94Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
85,68 tok/s TG
Prefill 3.508 · TTFT 4.145 ms
5×AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMvLLMgodclawAWQ
95Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
84,89 tok/s TG
Prefill 8.039 · TTFT 307 ms
1×3x AMD Radeon AI PRO R9700AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ4_K_M
96Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
83,10 tok/s TG
Prefill 6.957 · TTFT 7.450 ms
10×NVIDIA Tesla V100 PCIe 32GBAMD Ryzen 3 PRO 3200GE w/ Radeon Vega Graphics · 61 GB RAMllama.cppgodclawQ8_0
97Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
82,52 tok/s TG
Prefill 2.347 · TTFT 1.048 ms
1×AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMllama.cppgodclawQ4_K_M
98Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
81,91 tok/s TG
Prefill 2.340 · TTFT 1.052 ms
1×2x AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMllama.cppgodclawQ4_K_M
99Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
80,98 tok/s TG
Prefill 3.303 · TTFT 746 ms
1×AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMllama.cppgodclawQ8_0
100Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
78,20 tok/s TG
Prefill 3.205 · TTFT 784 ms
1×2x AMD Radeon RX 7900 XTX i4x Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz · 504 GB RAMllama.cppgodclawQ4_K_M
101Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
75,09 tok/s TG
Prefill 4.946 · TTFT 499 ms
1×2x NVIDIA RTX A6000 iAMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ8_0
102Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
75,09 tok/s TG
Prefill 5.190 · TTFT 476 ms
1×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppgodclawQ8_0
103Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
72,25 tok/s TG
Prefill 1.990 · TTFT 1.240 ms
1×AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMllama.cppgodclawQ8_0
104Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
69,93 tok/s TG
Prefill 2.843 · TTFT 920 ms
1×NVIDIA Tesla V100 PCIe 32GBAMD Ryzen 3 PRO 3200GE w/ Radeon Vega Graphics · 61 GB RAMllama.cppgodclawQ8_0
105Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
69,31 tok/s TG
Prefill 1.997 · TTFT 1.235 ms
1×2x AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMllama.cppgodclawQ8_0
106Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
67,58 tok/s TG
Prefill 888 · TTFT 15.224 ms
5×NVIDIA Tesla P100 PCIe 16GBAMD Ryzen 9 7945HX with Radeon Graphics · 60 GB RAMllama.cppopenclaw_cliQ4_K_M
107Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
66,99 tok/s TG
Prefill 5.119 · TTFT 482 ms
1×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMllama.cppopenclaw_cliQ8_0
108Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
66,46 tok/s TG
Prefill 6.040 · TTFT 408 ms
1×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMopenclaw_cliFP8
109Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
66,43 tok/s TG
Prefill 898 · TTFT 2.781 ms
1×Intel Arc Pro B70 iAMD Ryzen 9 7945HX with Radeon Graphics · 29 GB RAMllama.cppopenclaw_cliQ4_K_M
110Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
61,51 tok/s TG
Prefill 1.097 · TTFT 2.256 ms
1×2x NVIDIA GeForce RTX 20604x Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz · 504 GB RAMvLLMgodclawAWQ
111Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
60,55 tok/s TG
Prefill 5.171 · TTFT 476 ms
1×2x AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMvLLMgodclaw
112Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
55,43 tok/s TG
Prefill 4.133 · TTFT 598 ms
1×AMD Radeon AI PRO R9700AMD Ryzen 3 3100 4-Core Processor · 31 GB RAMllama.cppopenclaw_cliQ8_0
113Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
53,50 tok/s TG
Prefill 7.717 · TTFT 319 ms
1×NVIDIA GeForce RTX 4090AMD Ryzen 3 3100 4-Core Processor · 61 GB RAMvLLMopenclaw_cliAWQ
114Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
51,94 tok/s TG
Prefill 6.450 · TTFT 379 ms
1×NVIDIA GeForce RTX 3090 iAMD Ryzen 5 5600X 6-Core Processor · 31 GB RAMvLLMgodclaw
115Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
49,91 tok/s TG
Prefill 4.040 · TTFT 608 ms
1×NVIDIA GeForce RTX 3090 iAMD Ryzen 5 5600X 6-Core Processor · 31 GB RAMvLLMgodclaw
116Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
48,95 tok/s TG
Prefill 1.278 · TTFT 1.932 ms
1×2x NVIDIA GeForce RTX 2060 iIntel(R) Core(TM) i5-7400 CPU @ 3.00GHz · 7 GB RAMllama.cppopenclaw_cliQ4_K_M
117Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
47,41 tok/s TG
Prefill 1.800 · TTFT 1.368 ms
1×2x NVIDIA GeForce RTX 2060Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz · 7 GB RAMllama.cppopenclaw_cliQ4_K_M
118Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
45,73 tok/s TG
Prefill 1.206 · TTFT 2.050 ms
1×2x NVIDIA GeForce RTX 2060 iIntel(R) Core(TM) i5-7400 CPU @ 3.00GHz · 7 GB RAMllama.cppopenclaw_cliQ4_K_M
119Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
42,39 tok/s TG
Prefill 5.909 · TTFT 412 ms
1×2x NVIDIA RTX A6000 iAMD EPYC 7203P 8-Core Processor · 126 GB RAMvLLMgodclaw
120Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
42,11 tok/s TG
Prefill 821 · TTFT 2.298 ms
1×Intel Arc Pro B70 iAMD Ryzen 9 7945HX with Radeon Graphics · 29 GB RAMopenclaw_cli
121Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
37,29 tok/s TG
Prefill 1.115 · TTFT 27.826 ms
10×NVIDIA Tesla P100 PCIe 16GBAMD Ryzen 9 7945HX with Radeon Graphics · 60 GB RAMllama.cppopenclaw_cliQ4_K_M
122Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
35,46 tok/s TG
Prefill 589 · TTFT 4.189 ms
1×NVIDIA Tesla P100 PCIe 16GBAMD Ryzen 9 7945HX with Radeon Graphics · 60 GB RAMllama.cppopenclaw_cliQ4_K_M
123Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
34,53 tok/s TG
Prefill 2.874 · TTFT 857 ms
1×2x AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMvLLMgodclawAWQ
124Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
31,83 tok/s TG
Prefill 1.653 · TTFT 1.506 ms
1×2x NVIDIA GeForce RTX 2060Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz · 7 GB RAMllama.cppopenclaw_cliQ8_0
125Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
29,20 tok/s TG
Prefill 1.076 · TTFT 2.447 ms
1×Keine GPU (CPU-only)Apple M6 (unified CPU/GPU SoC, ARM64, AdvSIMD) · 24 GB RAMlmstudio-mlxlmstudioMLX 4bit
126Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
26,14 tok/s TG
Prefill 582 · TTFT 4.250 ms
1×Keine GPU (CPU-only)Apple M6 (unified CPU/GPU SoC) · 24 GB RAMOllamaopenclaw_cliQ4_K_M
127Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
19,56 tok/s TG
Prefill 2.006 · TTFT 1.227 ms
1×AMD Radeon PRO W7800 48GBAMD Ryzen Threadripper PRO 5975WX 32-Cores · 62 GB RAMvLLMgodclawAWQ
128Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
3,55 tok/s TG
Prefill 36 · TTFT 59.044 ms
1×Keine GPU (CPU-only)4x Intel(R) Xeon(R) CPU E5-4620 v2 @ 2.60GHz · 504 GB RAMllama.cppgodclawQ4_K_M
129Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1,19 tok/s TG
Prefill 9 · TTFT 227.569 ms
1×Keine GPU (CPU-only) i4 x Cortex-A78C + 4 x Cortex-A55 · 15 GB RAMllama.cppopenclaw_cliQ4_K_M
130Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
0,02 tok/s TG
Prefill 7 · TTFT 453.619 ms
5×Keine GPU (CPU-only) i4 x Cortex-A78C + 4 x Cortex-A55 · 15 GB RAMllama.cppopenclaw_cliQ4_K_M
Agent & chat rating

Harness benchmark

Noch keine Harnessbenchmarks fuer dieses Modell.