DeepSeek-V4.1-Flash on a Mac and two RTX PRO 6000s
128K prompt, first token
6.8s
was 51.1 s
1M prompt, first token
56s
was 521 s
Decode at 512K
114tok/s
was 66 tok/s
Two requests at once
156tok/s
was 78 tok/s
One model, two machines
Only four of the model's 40 layers produce KV, and the upper half reuses layer 20's. The prompt state is about 0.9 KB per token, so the model splits cleanly at layer 20.
RTX box
2× RTX PRO 6000 Blackwell
- Embeddings, Engram, layers 0–19
- DSpark drafter, vision tower
- Prefill the whole prompt
- Draft 4 tokens + run 1–5 rows, ~8.5 ms
prompt state, drafts + step rows
10GbE
verified tokens, taps
Mac Studio
M5 Ultra, 256 GB
- Layers 20–39, output head
- Verifies every token
- Verify, accept, stream
- Prefix cache for resumed turns
Prefill
Prompt tokens per second of time to first token, fresh uncached prompts. Dashed: the previous Mac-only 3-bit build.
Decode at depth
Output tokens per second, mean of six 256-token samples with the full range. Speed follows speculative acceptance, not context length: the box step is 7.2–7.8 ms from 8K to 1M.
By prompt length
Mac + RTX first, the 3-bit build beneath.
| Prompt | First token | Prefill tok/s | Speedup | Decode tok/s |
|---|---|---|---|---|
| 8K8,236 tokens | 0.76 s3.29 s | 10,7942,498 | 4.3× | 122.271.7 |
| 16K16,431 tokens | 1.17 s6.38 s | 14,0172,573 | 5.4× | — |
| 32K32,813 tokens | 1.95 s12.6 s | 16,8112,610 | 6.4× | — |
| 64K65,582 tokens | 3.55 s25.3 s | 18,4502,591 | 7.1× | — |
| 128K131,117 tokens | 6.79 s51.1 s | 19,3202,545 | 7.6× | 114.166.2 |
| 256K262,188 tokens | 13.2 s111 s | 19,8462,360 | 8.4× | 109.763.5 |
| 512K524,333 tokens | 26.7 s235 s | 19,6182,234 | 8.8× | 113.965.6 |
| 768K786,479 tokens | 41.2 s376 s | 19,0962,090 | 9.1× | — |
| 1M1,040,048 tokens | 56.2 s521 s | 18,5111,998 | 9.3× | 106.961.4 |
Concurrency
- 2 requests, 8K
- 155.8 tok/s90 each
- 2 requests, 128K
- 151.8 tok/s89 each
- 4 requests, 8K
- 170.8 tok/sall 4 at once, 57 each
Resuming a conversation
- Turn 2 at 8K
- 0.46 s
- Turn 2 at 128K
- 0.50 s3-bit 0.85 s
- Turn 2 at 512K
- 0.83 s
Images
- 1 image, first token
- 0.37 s6/6 correct
- 2 images, first token
- 0.37 s6/6 correct
Quality, 27 cases
- Code continuation NLL
- 0.0583-bit 0.062
- Needles at 128K and 1M
- recalled
- Draft acceptance
- 66%3-bit 65%
How it runs
- Prefill. The box runs layers 0–20 over the whole prompt in 8K chunks and streams the state to the Mac as it goes. The GPUs overlap their all-reduce with compute and split the indexer's rows (1M: 59.5 → 53.9 s).
- Decode. The Mac replays the tail through layers 20–39. Each step is one round trip: the Mac sends the verified tokens, and the box accepts, drafts the next 4 (DSpark, ~1.3 ms) and runs layers 0–19 for them. The Mac verifies every token, so the text is exactly what it would draft itself.
- Concurrency. Two requests pipeline across the machines. From three up, the Mac verifies fused pairs, reading the weights once per pair. Drafting on the box frees the Mac: +19% at 2 and 4 requests, +7% at one.
- Fairness. Decode steps and short prompts preempt a long prefill between layers: a decoder keeps ~19 steps/s during a 128K prefill, and an 8K prompt queued behind it waits 0.5 s.
- Exact. A box step is bit-identical to prefilling the same rows, so a restart mid-answer continues with the same text. If the box stays down, clients get a retryable error; no other model stands in.