DeepSeek-V4.1-Flash on a Mac and two RTX PRO 6000s

DeepSeek-V4.1-Flash on its original FP4/FP8 weights, served as one model across a Mac Studio M5 Ultra (256 GB) and two RTX PRO 6000 Blackwell GPUs over a 10GbE cable. Measured through the production API, September 29, 2026.
Code on GitHub · All M5 Ultra results

128K prompt, first token
6.8s
was 51.1 s
1M prompt, first token
56s
was 521 s
Decode at 512K
114tok/s
was 66 tok/s
Two requests at once
156tok/s
was 78 tok/s

One model, two machines

Only four of the model's 40 layers produce KV, and the upper half reuses layer 20's. The prompt state is about 0.9 KB per token, so the model splits cleanly at layer 20.

Prefill

Prompt tokens per second of time to first token, fresh uncached prompts. Dashed: the previous Mac-only 3-bit build.

0 5K 10K 15K 20K 0 128K 256K 512K 768K 1M 3-bit, Mac only Mac + RTX

Decode at depth

Output tokens per second, mean of six 256-token samples with the full range. Speed follows speculative acceptance, not context length: the box step is 7.2–7.8 ms from 8K to 1M.

0 30 60 90 120 0 128K 256K 512K 768K 1M 3-bit, Mac only Mac + RTX

By prompt length

Mac + RTX first, the 3-bit build beneath.

PromptFirst tokenPrefill tok/sSpeedupDecode tok/s
8K8,236 tokens0.76 s3.29 s10,7942,4984.3×122.271.7
16K16,431 tokens1.17 s6.38 s14,0172,5735.4×—
32K32,813 tokens1.95 s12.6 s16,8112,6106.4×—
64K65,582 tokens3.55 s25.3 s18,4502,5917.1×—
128K131,117 tokens6.79 s51.1 s19,3202,5457.6×114.166.2
256K262,188 tokens13.2 s111 s19,8462,3608.4×109.763.5
512K524,333 tokens26.7 s235 s19,6182,2348.8×113.965.6
768K786,479 tokens41.2 s376 s19,0962,0909.1×—
1M1,040,048 tokens56.2 s521 s18,5111,9989.3×106.961.4

Concurrency

2 requests, 8K
155.8 tok/s90 each
2 requests, 128K
151.8 tok/s89 each
4 requests, 8K
170.8 tok/sall 4 at once, 57 each

Resuming a conversation

Turn 2 at 8K
0.46 s
Turn 2 at 128K
0.50 s3-bit 0.85 s
Turn 2 at 512K
0.83 s

Images

1 image, first token
0.37 s6/6 correct
2 images, first token
0.37 s6/6 correct

Quality, 27 cases

Code continuation NLL
0.0583-bit 0.062
Needles at 128K and 1M
recalled
Draft acceptance
66%3-bit 65%

How it runs