M5 Ultra

Prefill at depth

Prompt tokens ÷ time to first token.

Measured in 8K chunks.

Measured at 130K and 522K.

Measured through the server at 8K to 1M tokens.

DeepSeek-V4.1-Flash, same day. Every change keeps greedy output bit-identical.

Engram table reads moved off the critical path: parallel native SSD reads, one chunk ahead of the GPU.
Sparse attention and indexer kernels rewritten: accelerator scoring, exact radix top-k, KV decoded once per query tile.
Buffer pool kept across layers; the next layer is built while the GPU runs the current one.
Decode: faster attention, Sinkhorn and KV-packing kernels; verify-step GEMVs share activations across rows.
No per-token kernel recompiles and no SSD page faults on fresh text.
Two requests batched exactly under speculative decoding.
Prefix cache: resuming a long conversation reuses its prefill instead of redoing it.
Decode at depth: the KV cache grows in place and index keys are scored once for all draft rows.
Engram rows cached across requests; replies that copy from the prompt draft from it too (108 → 122 tok/s).

Measured 2026-09-24 on the served configurations. RTX PRO 6000 pair figures are MiMo-V2.6-Flash on original weights.