MiniMax-M3.1-Flash Runs on Four Sparks Under Kubernetes at 59 Tok/s
@NftHoss serves 1M context with SGLang over a switchless RoCE ring; @jayleaton’s two-Spark quant is done.
@NftHoss runs MiniMax-M3.1-Flash on four DGX Sparks, end to end. The engine is SGLang at TP4 over a patched NCCL 2.30.7 ring, a switchless overlay across four RoCE HCAs per node. Weights are 250.1 GB in mxfp8, with DFlash speculative decoding.
| Measure | Result |
|---|---|
| Context | 1,048,576 tokens |
| Single-stream decode | 59.0 tok/s |
| Six streams, total | 122.9 tok/s |
| Prefill at 32k | 1,121 tok/s |
| Prefill at 266k | 1,248 tok/s |
| Boot to first response | ~12 min |
A 266,235-token request returned and found its needle. It was the first TP4-ring serve his team put under Kubernetes, with Grafana and Langfuse watching. He credits the switchless NCCL ring overlay to @MiaAI_lab.
His lessons: vLLM’s pynccl path could not start on a ring of this kind, while SGLang’s could. Feeding 250 GB of weights over 1 GbE NFS was too slow, so he used the 200G fabric.
Two Sparks are next in line. @jayleaton says his MiniMax Flash quant for two Sparks is done and he is now working on the recipe, from a baseline @MiaAI_lab provided.
Sources
- @NftHoss on X · Oct 24, 7:48 PM PT
- @jayleaton on X · Oct 24, 11:55 PM PT