TensorFold 1.1.4 Makes Qwen3.8-27B Cold Prefill Up to 5× Faster on GB10
At 64k the prefill goes from 203 to 1,045 tok/s; Gemma 4 now runs on native CUDA with its drafters.
@ashxhart released TensorFold 1.1.4, with up to 5× faster Qwen3.8-27B cold prefill on GB10, the chip in the DGX Spark.
| Change | Before | After |
|---|---|---|
| Qwen3.8-27B cold prefill at 64k | 203 tok/s | 1,045 tok/s |
| Gemma 4 12B chat, MTP assistant | 30 tok/s | 67 tok/s |
| Gemma 4 31B, z-lab DFlash drafter | 12.6 tok/s | 29.9 tok/s |
| Drama 3 first audio, 128-row chunks | 18.2 s | 0.76 s |
The Qwen gain grows with prompt length: 54% at 2k, 107% at 8k, 258% at 32k and 416% at 64k. Decode gains about 2% across 1 to 8 streams. Optional int8 prompt precision is available, with bf16 still the default.
Flash Next on CUDA now verifies several streams’ drafted windows in one pass. The GB10 served sweep gains 35%, 75% and 116% at 2, 4 and 8 streams, with tokens unchanged.
Experimental DeepSeek work adds four-rank prefill planning, but release binaries do not serve DeepSeek checkpoints yet. One regression is documented: Nemotron Lightning at four streams on GB10 is 3.8% slower. @MiaAI_lab says she will update any of her recipes that can benefit.
Sources
- @ashxhart on X · Oct 24, 12:35 PM PT
- @MiaAI_lab on X · Oct 24, 1:38 PM PT