Models & Hardware
@MiaAI_lab’s Ornith-1.5-122B Recipe Runs on One DGX Spark
256K context, a 1M total KV cache and 86 GB of memory, on TensorFold.
The promised recipe is out.
On Tuesday @MiaAI_lab said her Ornith-1.5-122B recipe would drop the next day. It has: a recipe to run Ornith-1.5-122B on a DGX Spark with her Ornith-1.5-122B-EXL3-EAQ 4 bpw quant. The model card is on HuggingFace.
| Setting | Value |
|---|---|
| Engine | TensorFold |
| Quant | EXL3-EAQ, 4 bpw |
| Context | 256K |
| Total KV cache | 1M |
| Concurrent streams | 4 by default |
| Total memory | 86 GB |
| Drafts | DFlash2 |
Speed or quality?
She then asked her followers: “Would you sacrifice 35% decode speed over maximum of 5-10% quality?” Choices will be made based on the answers, she said. After a reply backing quality, she wrote that it can be done: “35% less speed but better quality.”
Sources
- @MiaAI_lab on X · Oct 21, 1:26 AM PT
- @MiaAI_lab on X · Oct 21, 1:26 AM PT
- @MiaAI_lab on X · Oct 21, 2:08 AM PT
- @MiaAI_lab on X · Oct 21, 2:22 AM PT