God-fearing husband & father. AI Engineer. Fine-tuning + day-zero EXL3 & GGUF quants + benchmarks • Local LLMs • honest tok/s • fitting big models on small GPUs
Joined June 2009
- Tweets3.6K
- Following1K
- Followers3.2K
- Likes8.1K
Pinned Tweet
🚨 𝗜 𝗝𝗨𝗦𝗧 𝗔𝗖𝗛𝗜𝗘𝗩𝗘𝗗 𝗦𝗢𝗠𝗘𝗧𝗛𝗜𝗡𝗚 𝗜𝗡𝗖𝗥𝗘𝗗𝗜𝗕𝗟𝗘 𝗢𝗡 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞! 🧪🔥
With the support of @AWERXAI, I'm excited to release something VERY special for the local AI community!
🧠 @deepseek_ai 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸 𝗩𝟰.𝟭 𝗙𝗹𝗮𝘀𝗵 𝟭𝟬𝟭𝟳
📦 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 𝗮𝘁 𝗷𝘂𝘀𝘁 𝟭.𝟲𝟮 𝗯𝗽𝘄 𝗲𝘅𝗽𝗲𝗿𝘁𝘀
💻 𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗮𝗻𝘀𝘄𝗲𝗿-𝘁𝗼𝗸𝗲𝗻 𝗮𝗴𝗿𝗲𝗲𝗺𝗲𝗻𝘁!
⚡ 𝟯𝟱.𝟱 𝘁𝗼𝗸/𝘀 𝗱𝗲𝗰𝗼𝗱𝗲 𝗼𝗻 𝘃𝗟𝗟𝗠!
🚀 𝟭,𝟭𝟴𝟵 𝘁𝗼𝗸/𝘀 𝗠𝗔𝗫 𝗣𝗥𝗘𝗙𝗜𝗟𝗟!
🧠 𝟮𝟮𝟵𝗞 𝗖𝗧𝗫 (𝘀𝘁𝗶𝗹𝗹 𝘄𝗼𝗿𝗸𝗶𝗻𝗴 𝘁𝗼 𝗽𝘂𝘀𝗵 𝗶𝘁 𝗳𝘂𝗿𝘁𝗵𝗲𝗿!)
𝗔𝗟𝗟 𝗢𝗡 𝗢𝗡𝗘. 𝗗𝗚𝗫. 𝗦𝗣𝗔𝗥𝗞. 🤯
And YES, the quantized weights are AVAILABLE NOW! 💚
━━━━━━━━━━━━━━━━━━━━
🧪 𝗧𝗛𝗘 𝗟𝗔𝗦𝗧 𝟳𝟮 𝗛𝗢𝗨𝗥𝗦 𝗛𝗔𝗩𝗘 𝗕𝗘𝗘𝗡 𝗜𝗡𝗦𝗔𝗡𝗘!
Just a few days ago, I published my article about @nvidiaai's FP4-Local format: The Good, The Bad & The Ugly.
One thing really stuck with me...
NVIDIA demonstrated what was possible when aggressive quantization and intelligent optimization come together.
But there's a problem.
𝗥𝗘𝗣𝗥𝗢𝗗𝗨𝗖𝗜𝗡𝗚 𝗠𝗔𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗡𝗧𝗜𝗭𝗔𝗧𝗜𝗢𝗡 𝗧𝗥𝗔𝗜𝗡𝗜𝗡𝗚 𝗣𝗜𝗣𝗘𝗟𝗜𝗡𝗘𝗦 𝗜𝗦 𝗢𝗨𝗧 𝗢𝗙 𝗥𝗘𝗔𝗖𝗛 𝗙𝗢𝗥 𝗠𝗢𝗦𝗧 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗘𝗡𝗧𝗛𝗨𝗦𝗜𝗔𝗦𝗧𝗦.
That inspired me.
I've been developing my own SAGE-EXL3 process, and I wanted to see just how far I could push the latest DeepSeek Flash release while keeping the results accessible to the community.
So I spent the last 72 hours pushing the limits.
And the results genuinely surprised me! 🔥
━━━━━━━━━━━━━━━━━━━━
🏆 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 𝗔𝗥𝗘 𝗖𝗥𝗔𝗭𝗬!
I measured my 1.62 bpw expert quant against DeepSeek's native FP4 release checkpoint using held-out sequences and full-vocabulary token predictions.
And when I saw the CODE results, I could barely believe it!
🧪 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗩𝗦 𝗗𝗘𝗘𝗣𝗦𝗘𝗘𝗞'𝗦 𝗡𝗔𝗧𝗜𝗩𝗘 𝗙𝗣𝟴/𝗙𝗣𝟰 𝗥𝗘𝗟𝗘𝗔𝗦𝗘
💻 𝗖𝗢𝗗𝗘 — 𝗛𝘂𝗺𝗮𝗻𝗘𝘃𝗮𝗹
✅ 99.08% Top-1 Agreement
✅ 𝟵𝟵.𝟲𝟭% 𝗔𝗡𝗦𝗪𝗘𝗥-𝗢𝗡𝗟𝗬 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧!
✅ KL Divergence: 0.0186
🧠 𝗥𝗘𝗔𝗦𝗢𝗡𝗜𝗡𝗚 — 𝗠𝗔𝗧𝗛-𝟱𝟬𝟬
✅ 96.04% Top-1 Agreement
✅ 97.98% Answer-Only Agreement
✅ KL Divergence: 0.0349
🔢 𝗠𝗔𝗧𝗛 — 𝗚𝗦𝗠𝟴𝗞
✅ 95.56% Top-1 Agreement
✅ 94.19% Answer-Only Agreement
✅ KL Divergence: 0.0477
📊 𝗢𝗩𝗘𝗥𝗔𝗟𝗟 𝗛𝗘𝗟𝗗-𝗢𝗨𝗧 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬
🏆 94.52% Top-1 Agreement
📉 Mean KL: 0.0982
🎯 Median KL: 0.00019
𝗡𝗘𝗔𝗥-𝗟𝗢𝗦𝗦𝗟𝗘𝗦𝗦 𝗖𝗢𝗗𝗘 𝗧𝗢𝗞𝗘𝗡 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗔𝗧 𝗝𝗨𝗦𝗧 𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦! 🤯
These are direct teacher-forced fidelity measurements against native FP8, not task-level pass rates.
But seeing this level of agreement at such aggressive compression is EXACTLY what I was working toward!
━━━━━━━━━━━━━━━━━━━━
⚡ 𝗔𝗡𝗗 𝗬𝗘𝗦, 𝗜𝗧 𝗔𝗖𝗧𝗨𝗔𝗟𝗟𝗬 𝗥𝗨𝗡𝗦 𝗢𝗡 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞!
Not two Sparks.
Not four Sparks.
Not an expensive rented GPU cluster.
𝗢𝗡𝗘 𝟭𝟮𝟴𝗚𝗕 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞.
My latest vLLM measurements:
⚡ Decode: 35.5 tok/s
🚀 Maximum measured prefill: 1,189 tok/s
For a model this large, compressed this aggressively, and maintaining this level of fidelity...
𝗧𝗛𝗔𝗧'𝗦 𝗔 𝗕𝗜𝗚 𝗗𝗘𝗔𝗟 𝗙𝗢𝗥 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜! 💚
And remember, 1.62 bpw refers to the routed experts, not every parameter in the full model.
━━━━━━━━━━━━━━━━━━━━
💚 𝗪𝗛𝗬 𝗧𝗛𝗜𝗦 𝗠𝗔𝗧𝗧𝗘𝗥𝗦.
I have a simple goal.
𝗠𝗔𝗞𝗘 𝗣𝗢𝗪𝗘𝗥𝗙𝗨𝗟 𝗔𝗜 𝗔𝗖𝗖𝗘𝗦𝗦𝗜𝗕𝗟𝗘 𝗧𝗢 𝗣𝗘𝗢𝗣𝗟𝗘 𝗪𝗛𝗢 𝗗𝗢𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗔𝗖𝗖𝗘𝗦𝗦 𝗧𝗢 𝗗𝗔𝗧𝗔𝗖𝗘𝗡𝗧𝗘𝗥𝗦.
I don't want the future of local AI to be determined by who can rent the most GPUs or afford the largest training runs.
I want developers, independent researchers and enthusiasts to have the freedom to experiment, quantize, optimize and run powerful models on hardware they actually OWN.
NVIDIA's FP4-Local research showed me just how much potential remains in aggressive quantization.
My goal with SAGE-EXL3 is to keep pushing that frontier while making advanced quantization more practical for the wider community.
Today, I'm sharing the quantized checkpoint and the one-Spark serving recipe.
𝗧𝗛𝗜𝗦 𝗜𝗦 𝗝𝗨𝗦𝗧 𝗧𝗛𝗘 𝗕𝗘𝗚𝗜𝗡𝗡𝗜𝗡𝗚.
━━━━━━━━━━━━━━━━━━━━
🧪 𝗡𝗢𝗪 𝗙𝗢𝗥 𝗧𝗛𝗘 𝗥𝗘𝗔𝗟 𝗧𝗘𝗦𝗧!
The token-level fidelity results are incredible, but I'm not stopping there.
I'm currently running a broader evaluation campaign to measure how closely this quant preserves the native FP8 model's actual task performance.
💻 HumanEval+ / MBPP+
🧠 MMLU-Pro STEM / MATH-500
🔬 SciCode / LiveCodeBench
🔧 BFCL / Multi-step tool chains
🌍 MGSM
𝗧𝗛𝗘 𝗚𝗢𝗔𝗟 𝗜𝗦 𝗧𝗢 𝗣𝗥𝗢𝗩𝗘 𝗧𝗛𝗔𝗧 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗖𝗢𝗠𝗣𝗥𝗘𝗦𝗦𝗜𝗢𝗡 𝗗𝗢𝗘𝗦𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗧𝗢 𝗠𝗘𝗔𝗡 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗟𝗜𝗧𝗬 𝗟𝗢𝗦𝗦!
Those task-level results are coming next! 👀
━━━━━━━━━━━━━━━━━━━━
👀 𝗔𝗡𝗗 𝗢𝗡𝗘 𝗟𝗜𝗧𝗧𝗟𝗘 𝗧𝗘𝗔𝗦𝗘𝗥...
@mr_r0b0t and I have also been working on an GB10 specialized inference engine called 𝗥𝗘𝗫𝗟𝟯.
Early testing has been VERY exciting, including 60 tok/s with bursts reaching 74tok/s!
We're putting the finishing touches on it and making sure everything is solid before releasing it.
More on that VERY soon. 😏
━━━━━━━━━━━━━━━━━━━━
🙏 𝗔 𝗛𝗨𝗚𝗘 𝗧𝗛𝗔𝗡𝗞 𝗬𝗢𝗨!
To @AWERXAI for supporting my work and helping make this contribution to the community possible.
To @nvidiaai for the FP4-Local research that inspired me to push this experiment even further.
To @justinn_builds for his initial vllm TP4 recipe that got me started in the right direction for loading this model up.
To @mr_r0b0t for collaborating with me on what's coming next!
And to @deepseek_ai for continuing to release powerful open-weight models that make community experiments like this possible.
𝗧𝗛𝗘 𝗕𝗘𝗦𝗧 𝗣𝗔𝗥𝗧 𝗢𝗙 𝗢𝗣𝗘𝗡 𝗔𝗜 𝗜𝗦 𝗪𝗛𝗔𝗧 𝗧𝗛𝗘 𝗖𝗢𝗠𝗠𝗨𝗡𝗜𝗧𝗬 𝗖𝗔𝗡 𝗕𝗨𝗜𝗟𝗗 𝗢𝗡 𝗧𝗢𝗣 𝗢𝗙 𝗜𝗧. 💚
🔗 𝗗𝗢𝗪𝗡𝗟𝗢𝗔𝗗 𝗠𝗬 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯
huggingface.co/vcruz305/DSV4…
🛠️ 𝗢𝗡𝗘-𝗦𝗣𝗔𝗥𝗞 𝗥𝗘𝗖𝗜𝗣𝗘
github.com/vcruz305/DSV4.1-F…
𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞.
𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦.
𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗔𝗡𝗦𝗪𝗘𝗥 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧.
𝗟𝗘𝗧'𝗦 𝗞𝗘𝗘𝗣 𝗣𝗨𝗦𝗛𝗜𝗡𝗚 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗙𝗢𝗥𝗪𝗔𝗥𝗗! 🚀🔥
I've put over 3 billion tokens through Gemini Argon in the two weeks since it came out. Now @GoogleDeepMind already has Gemini 4 Carbon, and it's the new #1 on the Artificial Analysis Intelligence Index at 59. That's a crazy pace.
It doesn't win everything though. Claude Opus 5.5 still leads their coding agent index, 62 to 59, so "best model" really depends on what you're doing all day.
1M context, text, image, video and speech in, $4/$20 per million tokens with a 50% launch discount. 11% hallucination rate is the number I'll be watching once I'm running it.
Can't wait for this one to hit @OpenRouter!!
Google’s Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5, and second only to Claude Opus 5.5 on the Artificial Analysis Coding Agent Index
Gemini 4 Carbon scores 59 on Intelligence Index v5, driven by leading performance in complex reasoning, knowledge, and scientific tasks. It also has large gains in coding and knowledge work, particularly when compared to other Gemini models, but it is not leading in all these areas.
As with launch of Gemini 4 Argon, Gemini 4 Carbon is being rolled out to selected users and is not yet publicly available. Google is offering a 50% launch discount, with no confirmed end date. All our charts and figures use standard pricing for both Gemini 4 Carbon and Argon.
Key results for Gemini 4 Carbon with high reasoning:
➤ Gemini 4 Carbon leads the Intelligence Index v5, with the #1 score in 4 of the 9 Index evaluations: HLE-Diamond, Terminal-Bench-Science, AA-Omniscience, AutomationBench-AA. Carbon makes substantial gains elsewhere but is not leading in coding and knowledge work
➤ Behind only Claude Opus 5.5 on coding agents: With Antigravity CLI, Gemini 4 Carbon (high) scores 59 only 3 points behind Claude Code with Claude Opus 5.5 (max, 62) on the Artificial Analysis Coding Agent Index. Carbon ranks second on both Terminal-Bench 4.0 (64%) and FrontierCode v1.1 (51%), behind Claude Opus 5.5 (max) at 56%. Google has historically underperformed in this area and so this improvement stands out
➤ Improved agentic knowledge work, but not yet leading: On AA-Briefcase, Gemini 4 Carbon (high) scores 1609 Elo, a 126-point improvement over Gemini 4 Argon (high, 1483), but still behind models including Grok 4.7 (xhigh, 1638), Muse Wonder (max, 1659) and Kimi K3.1 Pro (1678). The largest gain comes from Presentation, up 293 Elo to 1598, alongside a 120-point improvement in Analytical Quality to 1671
➤ Lower Cost per Task than Claude Opus 5.5 for comparable overall intelligence: Gemini 4 Carbon costs $7.76 per Intelligence Index task, ~23% less than Claude Opus 5.5 (max, $10.04) but ~1.2x GPT-6 Astra (max, $6.25). It uses 123k output tokens per task, 29% fewer than Claude Opus 5.5 (max, 173k). At the same $4/$20 standard pricing, its Cost per Task is in line with Gemini 4 Argon ($7.68), while scoring 9 points higher on the Intelligence Index
➤ Leading AA-Omniscience v1.1 score, with low hallucination: Gemini 4 Carbon scores 54 on our recently updated AA-Omniscience v1.1, 6 points ahead of Claude Opus 5.5 (max, 48) and 8 points ahead of Gemini 4 Argon (46). Accuracy (% correct) rises 10 points over Argon to 58%, while its hallucination rate rises from 4% to 11%. This is still the second lowest of any model, behind only Argon, compared with 50% for both Claude Opus 5.5 (max) and GPT-6 Astra (max)
Key model details:
➤ Context Window: 1M tokens
➤ Multimodality: Text, image, video, and speech input, with text output
➤ Pricing: $4/$20 per 1M input/output tokens, the same as Gemini 4 Argon’s standard pricing, with a 50% discount. Cached input tokens receive a 95% discount ($0.20 per 1M)
Congratulations to the @GoogleDeepMind team!
Got pre-release access to Nemotron 3.5 Transcribe from @NVIDIAAI and the speaker identification is the part I love the most!!
30B MoE with only ~4B active, 40 languages, fully open weights, data and recipes. It just took #1 on the Open ASR leaderboard at 5.33 avg WER, ahead of Qwen3-ASR 1.7B and IBM Granite Speech.
What I keep thinking about is pairing it with an app like Meetily for local meeting notes. On a GPU that can run it fast, like a DGX Spark, you get transcripts with who said what, and nothing leaves your desk!
Theres a demo link where you can test and even upload your own audio for transcribing below!
Model: huggingface.co/nvidia/NVIDIA…
Blog: huggingface.co/blog/nvidia/n…
Demo: huggingface.co/spaces/nvidia…
Meet NVIDIA Nemotron 3.5 Transcribe 🎙️
An open-weight, promptable 30B MoE model with ~4B active parameters that brings transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding into one checkpoint.
Watch it keep track of who said what when everyone’s talking at once.
1/ 🧵
𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘇𝗲𝗱 𝘀𝗺𝗮𝗹𝗹 𝗟𝗟𝗠𝘀 𝗮𝗿𝗲 𝘄𝗵𝗲𝗿𝗲 𝗶𝘁'𝘀 𝗮𝘁 🔥
Every month, smaller models get more capable, and @liquidai keeps pushing the 𝘀𝘂𝗯-𝟯𝗕 boundary forward.
𝗟𝗙𝗠𝟯 is 6 models across text, vision, audio and omni, all around 𝟮𝗕 or smaller. The VL model claims to compete with 𝗚𝗲𝗺𝗺𝗮 𝟰 𝟭𝟮𝗕 at 𝟭.𝟵𝗕 params. The 1.7B text model has 𝟮𝟱𝟲𝗸 context, and you can fine-tune them on 𝟭 𝗚𝗣𝗨.
You don't need a giant model for everything. You need the right small one for the job, running on your own hardware.
Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence.
> 6 model instances, across text, audio, vision, and omni modalities, generative and decision models.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding
> 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages.
> 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language.
> 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input.
> fine-tune on 1 GPU.
> a new tri-hybrid architecture
> architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒
Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device.
1/ 🧵
Look at this!! 𝗚𝗲𝗻𝗶𝗲 𝟰 genuinely blows my mind 🤯
Real-time steerable worlds, native audio, characters that talk back, physics that holds up. And it's only a research preview.
Now put this next to where 𝗩𝗥 will be in 𝟱–𝟭𝟬 𝘆𝗲𝗮𝗿𝘀. You won't watch a world, you'll step inside one that's generated around you as you move.
I can't even picture what that looks like. Incredible work @GoogleDeepMind!! 👏🏼
Excited to share Genie 4 today our most realistic and interactive world model:
- steer dynamic environments in real time, shaping events, camera angles, and actions
- generate native audio matched to the world, from realistic background sounds to characters who can converse in real time!
- simulate photorealistic scenes and real-world physics
- create more coherent multi-step events through reasoning from Gemini
We've started to roll out as a research preview to trusted testers. Coming soon to Google AI Ultra subscribers in Project Genie and early access partners via the Gemini API and AI Studio.
𝗚𝗟𝗠-𝟱.𝟯-𝗙𝗹𝗮𝘀𝗵 𝗮𝗯𝗹𝗶𝘁𝗲𝗿𝗮𝘁𝗲𝗱 𝗳𝗼𝗿 𝗰𝘆𝗯𝗲𝗿 𝗱𝗲𝗳𝗲𝗻𝘀𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗿𝗶𝗴𝗵𝘁 𝗰𝗮𝗹𝗹 🛡️🔥
𝗙𝗥𝗘𝗘 𝗳𝗼𝗿 𝟰𝟴 𝗵𝗼𝘂𝗿𝘀 on Orca is huge. Go grab it!
━━━━━━━━━━━━
Want a cyber fine-tuned GLM-5.3-Flash running 𝗟𝗢𝗖𝗔𝗟𝗟𝗬? 👇
𝗥𝗘𝗗-𝗦𝗡𝗢𝗪-𝟱.𝟯-𝗙𝗟𝗔𝗦𝗛 by @Blackfrost_AI 🥶❄️ + my SAGE-EXL3 quants:
🟢 2.49 bpw → 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞 (97.4 GiB, 262k ctx)
🟣 4.91 bpw → 𝗧𝗪𝗢 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞𝗦
🎯 𝟮𝟭/𝟮𝟮 on 12 judge-scored cyber tasks
🧵 𝟭𝟱/𝟭𝟱 needle recall out to 240k
(independent tests by @yume_arasaki on the one-Spark quant)
🟢 One Spark (2.49 bpw):
huggingface.co/vcruz305/RED-…
🟣 Two Sparks (4.91 bpw):
huggingface.co/vcruz305/RED-…
Following our initiative of defending with every possible model, we’re launching OrcaCyber GLM-5.3 Flash Abliterated, one of the most powerful OSS models, for all Orca Cyber Tier users.
FREE for 48 hours. Limited compute, first come, first served.
orcarouter.ai/models/orca/or…
𝗢𝗽𝗲𝗻 𝘄𝗲𝗶𝗴𝗵𝘁𝘀 + 𝗚𝗲𝗺𝗺𝗮 𝗯𝗮𝘀𝗲 = 𝗶𝘁 𝗿𝘂𝗻𝘀 𝗹𝗼𝗰𝗮𝗹 🔥
Pulling it onto the 4090 tonight to see if it can sit right next to Nanbeige 33B (18.35 GiB, 33.8 tok/s) as a rewrite pass.
Local humanizer > paid API humanizer. Results soon 🧪
🚨 @nanbeige 𝗝𝗨𝗦𝗧 𝗗𝗥𝗢𝗣𝗣𝗘𝗗 𝗔 𝗤𝗪𝗘𝗡𝟯.𝟴-𝟮𝟳𝗕 𝗞𝗜𝗟𝗟𝗘𝗥... 𝗔𝗡𝗗 𝗜 𝗙𝗜𝗧 𝗜𝗧 𝗢𝗡 𝗢𝗡𝗘 𝗥𝗧𝗫 𝟰𝟬𝟵𝟬! 🧪🔥
𝗡𝗮𝗻𝗯𝗲𝗶𝗴𝗲𝟰.𝟯-𝟯𝟯𝗕-𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯-𝟰.𝟱𝟬𝗯𝗽𝘄
33 BILLION parameters. DENSE. No MoE shortcuts!
And the whole thing runs in 24GB of VRAM! 🤯
📦 𝟭𝟴.𝟯𝟱 𝗚𝗶𝗕 quantized weights
💾 𝟮𝟮.𝟲 𝗚𝗕 measured peak VRAM
⚡ 𝟯𝟯.𝟴 𝘁𝗼𝗸/𝘀 weighted decode
🧠 𝟯𝟲,𝟴𝟲𝟰 context tested with Q4 KV
🏆 𝟴𝟲.𝟲𝟳% 𝗦𝗜𝗫𝗖𝗔𝗧 (104/120)
🎯 𝟵𝟲.𝟲% top-1 agreement vs BF16 (KL 0.0136)
𝗔𝗟𝗟 𝗠𝗘𝗔𝗦𝗨𝗥𝗘𝗗 𝗢𝗡 𝗠𝗬 𝗦𝗜𝗡𝗚𝗟𝗘 𝗥𝗧𝗫 𝟰𝟬𝟵𝟬! 🔥
Weights are PUBLIC and UNGATED 👇
🤗 huggingface.co/vcruz305/Nanb…
━━━━━━━━━━━━━━━━━━━━
🥊 𝗪𝗛𝗬 𝗜'𝗠 𝗖𝗔𝗟𝗟𝗜𝗡𝗚 𝗜𝗧 𝗔 𝗤𝗪𝗘𝗡 𝟮𝟳𝗕 𝗞𝗜𝗟𝗟𝗘𝗥
The Nanbeige team's own BF16 numbers vs Qwen3.8-27B:
🏆 Terminal-Bench 2.1: 𝟴𝟬.𝟵 vs 73.0
🏆 DeepSWE: 𝟲𝟱.𝟱 vs 42.5
🏆 Automation Bench: 𝟱𝟰.𝟮 vs 38.3
🏆 SkillsBench: 𝟲𝟭.𝟭 vs 56.4
(Original model, not my quant.)
So I checked it on my own board. My Qwen3.8-27B runs topped out at 𝟴𝟮.𝟯 (stock Q4) and 𝟴𝟬.𝟬 (AEON Q4+MTP).
Nanbeige at 18GB: 𝟴𝟲.𝟲𝟳. 👀
━━━━━━━━━━━━━━━━━━━━
🧪 𝗠𝗬 𝗦𝗜𝗫𝗖𝗔𝗧 𝗥𝗘𝗦𝗨𝗟𝗧𝗦
120 questions. Six categories. Strict scoring. Thinking ON. Real Python execution.
🔢 Math: 𝟭𝟬𝟬% 🔥
📋 Instruction Following: 𝟵𝟱% 🔥
🔧 Tools: 𝟵𝟬%
🎯 Truthfulness: 𝟴𝟱%
💻 Code: 𝟴𝟱%
🧠 Knowledge: 𝟲𝟱%
🏆 𝗢𝗩𝗘𝗥𝗔𝗟𝗟: 𝟴𝟲.𝟲𝟳% (𝟭𝟬𝟰/𝟭𝟮𝟬)
Transparency: two Code answers ran out of their 32K output budget while still reasoning and were scored as failures.
━━━━━━━━━━━━━━━━━━━━
📏 𝗣𝗨𝗡𝗖𝗛𝗜𝗡𝗚 𝗪𝗔𝗬 𝗔𝗕𝗢𝗩𝗘 𝗜𝗧𝗦 𝗪𝗘𝗜𝗚𝗛𝗧
🟣 𝗤𝘄𝗲𝗻𝟯.𝟴-𝗙𝗹𝗮𝘀𝗵-𝗡𝗲𝘅𝘁 (125B MoE, 𝟯.𝟴𝘅 𝗯𝗶𝗴𝗴𝗲𝗿): my Q4_K_M GGUF scored 𝟴𝟵.𝟮. Nanbeige is only 𝟮.𝟱 𝗽𝗼𝗶𝗻𝘁𝘀 behind and ties my 3.05bpw EXL3 (𝟴𝟲.𝟳).
🔵 𝗚𝗟𝗠-𝟱.𝟯-𝗙𝗹𝗮𝘀𝗵 (320B MoE, ~𝟭𝟬𝘅 𝗯𝗶𝗴𝗴𝗲𝗿): 𝟵𝟭.𝟳 unquantized via API. Nanbeige is within 𝟱 𝗽𝗼𝗶𝗻𝘁𝘀 on a gaming card. 🤯
Different Sixcat versions and serving setups, so not a controlled ranking. But that's a dense 33B on a 4090.
━━━━━━━━━━━━━━━━━━━━
⚡ 𝗧𝗛𝗘 𝗥𝗘𝗟𝗘𝗔𝗦𝗘
✅ ExLlamaV3 + TabbyAPI (OpenAI-compatible)
✅ Thinking + tool calling
✅ Mixed-precision SAGE-EXL3, exact allocation published
✅ GitHub recipe with checksums and full test results
The original advertises 512K context; my 4090 qualification covers 36,864 tokens.
No DGX Spark. No rented datacenter. Just a 24GB card. 🚀
🛠️ 𝗥𝗘𝗖𝗜𝗣𝗘
github.com/vcruz305/Nanbeige…
💚 Congrats to the Nanbeige team on an impressive release!
𝗧𝗛𝗘 𝟮𝟰𝗚𝗕 𝗚𝗣𝗨 𝗖𝗟𝗔𝗦𝗦 𝗝𝗨𝗦𝗧 𝗚𝗢𝗧 𝗔 𝗦𝗘𝗥𝗜𝗢𝗨𝗦 𝗡𝗘𝗪 𝗔𝗚𝗘𝗡𝗧𝗜𝗖 𝗠𝗢𝗗𝗘𝗟. 🔥
🚀Excited to announce Nanbeige4.3-33B!
To push the performance ceiling of small agents, our strategy in Nanbeige4.3 starts with building a powerful teacher first.
Nanbeige4.3-33B delivers SOTA agentic performance among open models at ~30B parameters:
* Strong results on agentic coding and cowork benchmarks.
* When integrated into mainstream agent harnesses, it tackles real-world tasks with cost-efficient inference.
* It will serve as a teacher for upcoming smaller Nanbeige models, starting with Nanbeige4.3-3B.
huggingface.co/Nanbeige/Nanb…
huggingface.co/Nanbeige/Nanb…
#LLM #Agent
Cruz retweeted
Qwen3.8-Flash-Next on one DGX Spark: @MiaAI_lab's TensorFold recipe decodes code at 43.2 tok/s, @ViC305's two-model EXL3 recipe at 58.4 tok/s.
Setup:
– Same voxel-world prompt, both as an agent in Qwen Code 0.25.2, one prompt, one try, no help, thinking on
– Context: 262,144 tokens
– Mia's recipe: NVFP4 with MTP on @ashxhart's TensorFold, 5 slots, answer limit 131,072 tokens
– ViC305's recipe: EXL3 3.05 bpw on ExLlamaV3 and TabbyAPI, 1 slot, answer limit 262,144 tokens, MiniCPM 4.7 loaded beside it
Mia's recipe, world Aetherfall:
– Decode, code: 43.2 tok/s
– 500 words: about 16.9 s
– 8 requests at once: 87.6 tok/s in total
– Reading a 32K-token prompt: 2,163 tok/s
– Memory at 262,144 tokens of context: 87.01 GiB
– Agent time: 108.6 min
– Browser check: 8.5/10
ViC305's recipe, world AETHERFALL:
– Decode, code: 58.4 tok/s
– 500 words: about 12.6 s
– 8 requests at once: 41.5 tok/s in total
– Reading a 32K-token prompt: 662 tok/s
– Memory with both models loaded, 125,892-token prompt: 87.0 GiB
– Agent time: 60.9 min
– Browser check: 8.5/10
Quality:
– MMLU-Pro: 68.3% against 67.3%
– LiveCodeBench: 62.5% against 65.0%
– GPQA with thinking: 70.0% for both (30 questions, ± 15.6 points)
Mia's recipe is the stronger setup for parallel work and long prompts, and I like its world better. ViC305's decodes a single request about 35% faster.
Good to know:
– Different quants and engines, and different answer limits, so the agent times do not compare the models alone.
– Mia's 8-at-once total comes from a second Spark. The author's README shows 88.6 tok/s for code; my fixed code prompt is mostly prose.
– ViC305's memory number comes from a test up to a 125,892-token prompt, Mia's from the full 262,144.
Play both worlds in your browser.
All numbers, the exact command and the raw output:
Mia's recipe: niklaslenz.com/v2/qwen38-fla…
ViC305's recipe: niklaslenz.com/v2/qwen38-fla…
Recipe by @MiaAI_lab on @ashxhart's TensorFold, model by @Alibaba_Qwen. Recipe, MiniCPM quant and engine forks by @ViC305, EXL3 quant by @turboderp_.
Amazing work by @wikiwayne ! Im extremely glad to see my EXL3 quants take the top 3! Especially this 1.62bit Deepseek v4.1 Flash EXL3 beating my lossless Qwen Flash Next EXL3!! 👀 love the site layout btw great job on testing so many models!!
Cruz retweeted
🔥 MiniMax M3.1 Flash (428B) on 2 NVIDIA DGX Sparks, TP2 over RoCE
🧠 Quant by @ViC305 (SAGE EXL3 2.15 bpw) 🙌
📏 300K context, 726K KV pool
🏎️ Up to 39 tok/s single stream, 65-78 tok/s at 4 users
🛠️ 2 vLLM-EXL3 patches for TP2
Recipe + benchmarks 👇
github.com/tonyd2wild/MiniMa…
🚨 @MiniMax_AI 𝗠𝟯.𝟭 𝗙𝗟𝗔𝗦𝗛 — 𝟰𝟮𝟴𝗕 𝗣𝗔𝗥𝗔𝗠𝗘𝗧𝗘𝗥𝗦. 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞. 𝟵𝟳% 𝗧𝗢𝗞𝗘𝗡 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧! 🧪🔥
I've done it AGAIN, and this time I'm releasing TWO new SAGE-EXL3 quantizations of the brand-new MiniMax M3.1 Flash!
428B total parameters.
23B active.
Native 1M-token context capability.
My goal was simple:
𝗦𝗤𝗨𝗘𝗘𝗭𝗘 𝗧𝗛𝗘 𝗠𝗢𝗗𝗘𝗟, 𝗡𝗢𝗧 𝗜𝗧𝗦 𝗜𝗡𝗧𝗘𝗟𝗟𝗜𝗚𝗘𝗡𝗖𝗘.
And the fidelity measurements are some of my most exciting yet! 🤯
━━━━━━━━━━━━━━━━━━━━
🟢 𝗥𝗘𝗟𝗘𝗔𝗦𝗘 #𝟭 — 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞
𝗠𝗶𝗻𝗶𝗠𝗮𝘅-𝗠𝟯.𝟭-𝗙𝗹𝗮𝘀𝗵-𝗘𝗫𝗟𝟯-𝗦𝗔𝗚𝗘-𝟮.𝟭𝟱𝗯𝗽𝘄
📦 108.64 GiB checkpoint
🧠 96.89% Top-1 Token Agreement
🎯 99.75% Top-5 Agreement
📉 Mean KL Divergence: 0.02528
𝗔𝗟𝗟 𝗢𝗙 𝗧𝗛𝗔𝗧 𝗜𝗡 𝗔 𝗦𝗜𝗡𝗚𝗟𝗘 𝟭𝟮𝟴𝗚𝗕 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞!
And YES, I already have a working vLLM serving configuration!
But there's a catch, and I'll explain why shortly. 👀
━━━━━━━━━━━━━━━━━━━━
🟣 𝗥𝗘𝗟𝗘𝗔𝗦𝗘 #𝟮 — 𝗧𝗪𝗢 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞𝗦
𝗠𝗶𝗻𝗶𝗠𝗮𝘅-𝗠𝟯.𝟭-𝗙𝗹𝗮𝘀𝗵-𝗘𝗫𝗟𝟯-𝗦𝗔𝗚𝗘-𝟰.𝟮𝟲𝗯𝗽𝘄
📦 209.88 GiB checkpoint
🧠 98.44% Top-1 Token Agreement
🎯 99.94% Top-5 Agreement
📉 Mean KL Divergence: JUST 0.00592!
𝗡𝗜𝗡𝗘𝗧𝗬-𝗘𝗜𝗚𝗛𝗧 𝗣𝗢𝗜𝗡𝗧 𝗙𝗢𝗨𝗥 𝗙𝗢𝗨𝗥 𝗣𝗘𝗥𝗖𝗘𝗡𝗧 𝗧𝗢𝗞𝗘𝗡 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧! 🤯
That's exceptionally close to the unquantized reference in my held-out token-level evaluation.
Both releases were measured against the original unquantized MiniMax checkpoint using full-vocabulary teacher-forced evaluation.
These are quantization-fidelity measurements, not task pass rates or proof of mathematically lossless behavior.
But THESE are the kinds of quality-per-bit results I'm chasing!
━━━━━━━━━━━━━━━━━━━━
⚠️ 𝗧𝗛𝗘 𝗣𝗥𝗢𝗕𝗟𝗘𝗠 𝗜𝗦𝗡'𝗧 𝗝𝗨𝗦𝗧 𝗧𝗛𝗘 𝗠𝗢𝗗𝗘𝗟 𝗔𝗡𝗬𝗠𝗢𝗥𝗘.
It's the inference engine.
My current ONE-SPARK vLLM recipe WORKS, but it's still a WORK IN PROGRESS.
The measured configuration currently reaches 32K context using a quantized KV cache.
MiniMax M3.1 Flash supports up to 1M native context, but vLLM's memory overhead on this hardware is consuming headroom I'd rather dedicate to the model and its context.
I could shrink the weights further to compensate.
𝗕𝗨𝗧 𝗪𝗛𝗬 𝗦𝗔𝗖𝗥𝗜𝗙𝗜𝗖𝗘 𝗠𝗢𝗗𝗘𝗟 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗧𝗢 𝗔𝗖𝗖𝗢𝗠𝗠𝗢𝗗𝗔𝗧𝗘 𝗥𝗨𝗡𝗧𝗜𝗠𝗘 𝗢𝗩𝗘𝗥𝗛𝗘𝗔𝗗?!
That's the wrong trade-off for what I'm trying to accomplish.
I'm optimizing for the BEST possible balance between model quality, memory usage and practical inference speed.
And that's where something else I've been building comes in...
⚡ 𝗥𝗘𝗫𝗟𝟯.
Our upcoming public inference engine and dedicated recipe are aimed at removing these limitations and making the model's full native context practical on Spark.
We're continuing to validate that work before release.
𝗦𝗧𝗔𝗬 𝗧𝗨𝗡𝗘𝗗. 𝗧𝗛𝗘 𝗣𝗨𝗕𝗟𝗜𝗖 𝗥𝗘𝗟𝗘𝗔𝗦𝗘 𝗜𝗦 𝗚𝗘𝗧𝗧𝗜𝗡𝗚 𝗖𝗟𝗢𝗦𝗘𝗥! 🚀
━━━━━━━━━━━━━━━━━━━━
🤝 𝗖𝗔𝗟𝗟𝗜𝗡𝗚 𝗔𝗟𝗟 𝗧𝗪𝗢-𝗦𝗣𝗔𝗥𝗞 𝗗𝗘𝗩𝗘𝗟𝗢𝗣𝗘𝗥𝗦!
Here's where I'd love some help from the community!
The 4.26 bpw model is already available on Hugging Face, but I haven't had the time to properly build and validate its TWO-SPARK serving recipe.
If you have two DGX Sparks and experience with distributed inference, I'd LOVE to collaborate!
Fork my GitHub recipe.
Experiment.
Submit a PR with your improvements and measurements.
𝗜 𝗪𝗔𝗡𝗧 𝗧𝗢 𝗪𝗘𝗟𝗖𝗢𝗠𝗘 𝗖𝗢𝗡𝗧𝗥𝗜𝗕𝗨𝗧𝗢𝗥𝗦 𝗔𝗡𝗗 𝗚𝗜𝗩𝗘 𝗧𝗛𝗘𝗠 𝗣𝗥𝗢𝗣𝗘𝗥 𝗖𝗥𝗘𝗗𝗜𝗧 𝗙𝗢𝗥 𝗧𝗛𝗘𝗜𝗥 𝗪𝗢𝗥𝗞! 💚
I want these releases to push the quality-per-bit frontier for EXL3, while making it easier for the community to reproduce and extend the inference results.
━━━━━━━━━━━━━━━━━━━━
🔗 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞 — 𝟮.𝟭𝟱𝗕𝗣𝗪
huggingface.co/vcruz305/Mini…
🔗 𝗧𝗪𝗢 𝗦𝗣𝗔𝗥𝗞𝗦 — 𝟰.𝟮𝟲𝗕𝗣𝗪
huggingface.co/vcruz305/Mini…
🛠️ 𝗥𝗘𝗖𝗜𝗣𝗘 & 𝗖𝗢𝗠𝗠𝗨𝗡𝗜𝗧𝗬 𝗣𝗥𝗦
github.com/vcruz305/MiniMax-…
𝗧𝗪𝗢 𝗤𝗨𝗔𝗡𝗧𝗦. 𝗧𝗪𝗢 𝗠𝗘𝗠𝗢𝗥𝗬 𝗧𝗔𝗥𝗚𝗘𝗧𝗦. 𝗢𝗡𝗘 𝗣𝗥𝗜𝗢𝗥𝗜𝗧𝗬: 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬.
And the next chapter is getting very interesting. 🧪🔥
MiniMax M3.1 Flash weights are now public. 428B parameters total, 23B activated per token.
Agentic performance: A generational leap in agentic tasks, coding, and cybersecurity — 76.2 on BankerToolBench, 73.7% on DeepSWE v1.1, 61.7 on OSWorld V2, 29.6 on Agents' Last Exam, and 82.2 on CyberGym.
Efficient by design: MiniMax Sparse Attention and DSpark speculative decoding improve inference efficiency.
Hugging Face:
huggingface.co/MiniMaxAI/Min…
ModelScope:
modelscope.cn/models/MiniMax…
API: platform.minimax.io
M Plan: minimax.io/m-planMiniMax
MCode for desktop: agent.minimax.io/download
So happy these MiniMax M3.1 weights were released early! 🚀 I’m currently working on a one spark EXL3 quant with my new and improved SAGE-EXL3 process that will keep most of the quality of this model the same way I did Deepseek v4.1 Flash 1017 ! Stay tuned! 👀
MiniMax M3.1 Flash weights are now public. 428B parameters total, 23B activated per token.
Agentic performance: A generational leap in agentic tasks, coding, and cybersecurity — 76.2 on BankerToolBench, 73.7% on DeepSWE v1.1, 61.7 on OSWorld V2, 29.6 on Agents' Last Exam, and 82.2 on CyberGym.
Efficient by design: MiniMax Sparse Attention and DSpark speculative decoding improve inference efficiency.
Hugging Face:
huggingface.co/MiniMaxAI/Min…
ModelScope:
modelscope.cn/models/MiniMax…
API: platform.minimax.io
M Plan: minimax.io/m-planMiniMax
MCode for desktop: agent.minimax.io/download
Cruz retweeted
🚨 𝗜 𝗝𝗨𝗦𝗧 𝗔𝗖𝗛𝗜𝗘𝗩𝗘𝗗 𝗦𝗢𝗠𝗘𝗧𝗛𝗜𝗡𝗚 𝗜𝗡𝗖𝗥𝗘𝗗𝗜𝗕𝗟𝗘 𝗢𝗡 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞! 🧪🔥
With the support of @AWERXAI, I'm excited to release something VERY special for the local AI community!
🧠 @deepseek_ai 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸 𝗩𝟰.𝟭 𝗙𝗹𝗮𝘀𝗵 𝟭𝟬𝟭𝟳
📦 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 𝗮𝘁 𝗷𝘂𝘀𝘁 𝟭.𝟲𝟮 𝗯𝗽𝘄 𝗲𝘅𝗽𝗲𝗿𝘁𝘀
💻 𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗮𝗻𝘀𝘄𝗲𝗿-𝘁𝗼𝗸𝗲𝗻 𝗮𝗴𝗿𝗲𝗲𝗺𝗲𝗻𝘁!
⚡ 𝟯𝟱.𝟱 𝘁𝗼𝗸/𝘀 𝗱𝗲𝗰𝗼𝗱𝗲 𝗼𝗻 𝘃𝗟𝗟𝗠!
🚀 𝟭,𝟭𝟴𝟵 𝘁𝗼𝗸/𝘀 𝗠𝗔𝗫 𝗣𝗥𝗘𝗙𝗜𝗟𝗟!
🧠 𝟮𝟮𝟵𝗞 𝗖𝗧𝗫 (𝘀𝘁𝗶𝗹𝗹 𝘄𝗼𝗿𝗸𝗶𝗻𝗴 𝘁𝗼 𝗽𝘂𝘀𝗵 𝗶𝘁 𝗳𝘂𝗿𝘁𝗵𝗲𝗿!)
𝗔𝗟𝗟 𝗢𝗡 𝗢𝗡𝗘. 𝗗𝗚𝗫. 𝗦𝗣𝗔𝗥𝗞. 🤯
And YES, the quantized weights are AVAILABLE NOW! 💚
━━━━━━━━━━━━━━━━━━━━
🧪 𝗧𝗛𝗘 𝗟𝗔𝗦𝗧 𝟳𝟮 𝗛𝗢𝗨𝗥𝗦 𝗛𝗔𝗩𝗘 𝗕𝗘𝗘𝗡 𝗜𝗡𝗦𝗔𝗡𝗘!
Just a few days ago, I published my article about @nvidiaai's FP4-Local format: The Good, The Bad & The Ugly.
One thing really stuck with me...
NVIDIA demonstrated what was possible when aggressive quantization and intelligent optimization come together.
But there's a problem.
𝗥𝗘𝗣𝗥𝗢𝗗𝗨𝗖𝗜𝗡𝗚 𝗠𝗔𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗡𝗧𝗜𝗭𝗔𝗧𝗜𝗢𝗡 𝗧𝗥𝗔𝗜𝗡𝗜𝗡𝗚 𝗣𝗜𝗣𝗘𝗟𝗜𝗡𝗘𝗦 𝗜𝗦 𝗢𝗨𝗧 𝗢𝗙 𝗥𝗘𝗔𝗖𝗛 𝗙𝗢𝗥 𝗠𝗢𝗦𝗧 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗘𝗡𝗧𝗛𝗨𝗦𝗜𝗔𝗦𝗧𝗦.
That inspired me.
I've been developing my own SAGE-EXL3 process, and I wanted to see just how far I could push the latest DeepSeek Flash release while keeping the results accessible to the community.
So I spent the last 72 hours pushing the limits.
And the results genuinely surprised me! 🔥
━━━━━━━━━━━━━━━━━━━━
🏆 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 𝗔𝗥𝗘 𝗖𝗥𝗔𝗭𝗬!
I measured my 1.62 bpw expert quant against DeepSeek's native FP4 release checkpoint using held-out sequences and full-vocabulary token predictions.
And when I saw the CODE results, I could barely believe it!
🧪 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗩𝗦 𝗗𝗘𝗘𝗣𝗦𝗘𝗘𝗞'𝗦 𝗡𝗔𝗧𝗜𝗩𝗘 𝗙𝗣𝟴/𝗙𝗣𝟰 𝗥𝗘𝗟𝗘𝗔𝗦𝗘
💻 𝗖𝗢𝗗𝗘 — 𝗛𝘂𝗺𝗮𝗻𝗘𝘃𝗮𝗹
✅ 99.08% Top-1 Agreement
✅ 𝟵𝟵.𝟲𝟭% 𝗔𝗡𝗦𝗪𝗘𝗥-𝗢𝗡𝗟𝗬 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧!
✅ KL Divergence: 0.0186
🧠 𝗥𝗘𝗔𝗦𝗢𝗡𝗜𝗡𝗚 — 𝗠𝗔𝗧𝗛-𝟱𝟬𝟬
✅ 96.04% Top-1 Agreement
✅ 97.98% Answer-Only Agreement
✅ KL Divergence: 0.0349
🔢 𝗠𝗔𝗧𝗛 — 𝗚𝗦𝗠𝟴𝗞
✅ 95.56% Top-1 Agreement
✅ 94.19% Answer-Only Agreement
✅ KL Divergence: 0.0477
📊 𝗢𝗩𝗘𝗥𝗔𝗟𝗟 𝗛𝗘𝗟𝗗-𝗢𝗨𝗧 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬
🏆 94.52% Top-1 Agreement
📉 Mean KL: 0.0982
🎯 Median KL: 0.00019
𝗡𝗘𝗔𝗥-𝗟𝗢𝗦𝗦𝗟𝗘𝗦𝗦 𝗖𝗢𝗗𝗘 𝗧𝗢𝗞𝗘𝗡 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗔𝗧 𝗝𝗨𝗦𝗧 𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦! 🤯
These are direct teacher-forced fidelity measurements against native FP8, not task-level pass rates.
But seeing this level of agreement at such aggressive compression is EXACTLY what I was working toward!
━━━━━━━━━━━━━━━━━━━━
⚡ 𝗔𝗡𝗗 𝗬𝗘𝗦, 𝗜𝗧 𝗔𝗖𝗧𝗨𝗔𝗟𝗟𝗬 𝗥𝗨𝗡𝗦 𝗢𝗡 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞!
Not two Sparks.
Not four Sparks.
Not an expensive rented GPU cluster.
𝗢𝗡𝗘 𝟭𝟮𝟴𝗚𝗕 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞.
My latest vLLM measurements:
⚡ Decode: 35.5 tok/s
🚀 Maximum measured prefill: 1,189 tok/s
For a model this large, compressed this aggressively, and maintaining this level of fidelity...
𝗧𝗛𝗔𝗧'𝗦 𝗔 𝗕𝗜𝗚 𝗗𝗘𝗔𝗟 𝗙𝗢𝗥 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜! 💚
And remember, 1.62 bpw refers to the routed experts, not every parameter in the full model.
━━━━━━━━━━━━━━━━━━━━
💚 𝗪𝗛𝗬 𝗧𝗛𝗜𝗦 𝗠𝗔𝗧𝗧𝗘𝗥𝗦.
I have a simple goal.
𝗠𝗔𝗞𝗘 𝗣𝗢𝗪𝗘𝗥𝗙𝗨𝗟 𝗔𝗜 𝗔𝗖𝗖𝗘𝗦𝗦𝗜𝗕𝗟𝗘 𝗧𝗢 𝗣𝗘𝗢𝗣𝗟𝗘 𝗪𝗛𝗢 𝗗𝗢𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗔𝗖𝗖𝗘𝗦𝗦 𝗧𝗢 𝗗𝗔𝗧𝗔𝗖𝗘𝗡𝗧𝗘𝗥𝗦.
I don't want the future of local AI to be determined by who can rent the most GPUs or afford the largest training runs.
I want developers, independent researchers and enthusiasts to have the freedom to experiment, quantize, optimize and run powerful models on hardware they actually OWN.
NVIDIA's FP4-Local research showed me just how much potential remains in aggressive quantization.
My goal with SAGE-EXL3 is to keep pushing that frontier while making advanced quantization more practical for the wider community.
Today, I'm sharing the quantized checkpoint and the one-Spark serving recipe.
𝗧𝗛𝗜𝗦 𝗜𝗦 𝗝𝗨𝗦𝗧 𝗧𝗛𝗘 𝗕𝗘𝗚𝗜𝗡𝗡𝗜𝗡𝗚.
━━━━━━━━━━━━━━━━━━━━
🧪 𝗡𝗢𝗪 𝗙𝗢𝗥 𝗧𝗛𝗘 𝗥𝗘𝗔𝗟 𝗧𝗘𝗦𝗧!
The token-level fidelity results are incredible, but I'm not stopping there.
I'm currently running a broader evaluation campaign to measure how closely this quant preserves the native FP8 model's actual task performance.
💻 HumanEval+ / MBPP+
🧠 MMLU-Pro STEM / MATH-500
🔬 SciCode / LiveCodeBench
🔧 BFCL / Multi-step tool chains
🌍 MGSM
𝗧𝗛𝗘 𝗚𝗢𝗔𝗟 𝗜𝗦 𝗧𝗢 𝗣𝗥𝗢𝗩𝗘 𝗧𝗛𝗔𝗧 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗖𝗢𝗠𝗣𝗥𝗘𝗦𝗦𝗜𝗢𝗡 𝗗𝗢𝗘𝗦𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗧𝗢 𝗠𝗘𝗔𝗡 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗟𝗜𝗧𝗬 𝗟𝗢𝗦𝗦!
Those task-level results are coming next! 👀
━━━━━━━━━━━━━━━━━━━━
👀 𝗔𝗡𝗗 𝗢𝗡𝗘 𝗟𝗜𝗧𝗧𝗟𝗘 𝗧𝗘𝗔𝗦𝗘𝗥...
@mr_r0b0t and I have also been working on an GB10 specialized inference engine called 𝗥𝗘𝗫𝗟𝟯.
Early testing has been VERY exciting, including 60 tok/s with bursts reaching 74tok/s!
We're putting the finishing touches on it and making sure everything is solid before releasing it.
More on that VERY soon. 😏
━━━━━━━━━━━━━━━━━━━━
🙏 𝗔 𝗛𝗨𝗚𝗘 𝗧𝗛𝗔𝗡𝗞 𝗬𝗢𝗨!
To @AWERXAI for supporting my work and helping make this contribution to the community possible.
To @nvidiaai for the FP4-Local research that inspired me to push this experiment even further.
To @justinn_builds for his initial vllm TP4 recipe that got me started in the right direction for loading this model up.
To @mr_r0b0t for collaborating with me on what's coming next!
And to @deepseek_ai for continuing to release powerful open-weight models that make community experiments like this possible.
𝗧𝗛𝗘 𝗕𝗘𝗦𝗧 𝗣𝗔𝗥𝗧 𝗢𝗙 𝗢𝗣𝗘𝗡 𝗔𝗜 𝗜𝗦 𝗪𝗛𝗔𝗧 𝗧𝗛𝗘 𝗖𝗢𝗠𝗠𝗨𝗡𝗜𝗧𝗬 𝗖𝗔𝗡 𝗕𝗨𝗜𝗟𝗗 𝗢𝗡 𝗧𝗢𝗣 𝗢𝗙 𝗜𝗧. 💚
🔗 𝗗𝗢𝗪𝗡𝗟𝗢𝗔𝗗 𝗠𝗬 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯
huggingface.co/vcruz305/DSV4…
🛠️ 𝗢𝗡𝗘-𝗦𝗣𝗔𝗥𝗞 𝗥𝗘𝗖𝗜𝗣𝗘
github.com/vcruz305/DSV4.1-F…
𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞.
𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦.
𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗔𝗡𝗦𝗪𝗘𝗥 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧.
𝗟𝗘𝗧'𝗦 𝗞𝗘𝗘𝗣 𝗣𝗨𝗦𝗛𝗜𝗡𝗚 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗙𝗢𝗥𝗪𝗔𝗥𝗗! 🚀🔥
🚨 𝗜 𝗝𝗨𝗦𝗧 𝗦𝗛𝗥𝗨𝗡𝗞 𝗢𝗥𝗡𝗜𝗧𝗛 𝟭.𝟱 𝟭𝟮𝟮𝗕 𝗙𝗢𝗥 𝗧𝗛𝗘 𝟰𝟴𝗚𝗕 𝗩𝗥𝗔𝗠 𝗖𝗢𝗠𝗠𝗨𝗡𝗜𝗧𝗬! 🧪🔥
@ornith_ JUST released their new Ornith-1.5-122B, and naturally, I HAD to see how far I could push it with SAGE-EXL3!
And the results surprised even ME! 🤯
Introducing...
🟢 𝗢𝗿𝗻𝗶𝘁𝗵-𝟭.𝟱-𝟭𝟮𝟮𝗕-𝗘𝗫𝗟𝟯-𝗦𝗔𝗚𝗘-𝟮.𝟱𝟴𝗯𝗽𝘄
📦 ~39.6 GiB GPU-resident LM weights
🧠 122B-class Mixture-of-Experts model
⚡ 2.58 bpw routed experts
💻 Targeting 48GB VRAM GPUs and above!
🚀 vLLM + EXL3 inference
𝗕𝗨𝗧 𝗛𝗘𝗥𝗘'𝗦 𝗪𝗛𝗘𝗥𝗘 𝗜𝗧 𝗚𝗘𝗧𝗦 𝗜𝗡𝗧𝗘𝗥𝗘𝗦𝗧𝗜𝗡𝗚... 👀
━━━━━━━━━━━━━━━━━━━━
🏆 𝗦𝗔𝗚𝗘 𝟮.𝟱𝟴 𝗕𝗣𝗪 𝘃𝘀 𝗠𝗜𝗔-𝗔𝗜𝗟𝗔𝗕 𝗘𝗔𝗤𝘃𝟭 𝟰.𝟬 𝗕𝗣𝗪!
I compared my release against Mia-AiLab's Ornith-1.5-122B-EXL3-EAQv1-4-bpw quantization.
Same model family.
Same DGX Spark.
353 matched benchmark questions.
9 evaluation suites.
Strict pass@1 scoring.
🟢 𝗠𝗬 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 𝟮.𝟱𝟴 𝗕𝗣𝗪
📦 41.00 GiB LM weights
🏆 𝟵𝟭.𝟳𝟴% — 𝟯𝟮𝟰/𝟯𝟱𝟯
🟣 𝗠𝗜𝗔-𝗔𝗜𝗟𝗔𝗕 𝗘𝗔𝗤𝘃𝟭 𝟰.𝟬 𝗕𝗣𝗪
📦 59.00 GiB LM weights
📊 𝟵𝟭.𝟱𝟬% — 𝟯𝟮𝟯/𝟯𝟱𝟯
𝗔𝗣𝗣𝗥𝗢𝗫𝗜𝗠𝗔𝗧𝗘𝗟𝗬 𝟯𝟬.𝟱% 𝗦𝗠𝗔𝗟𝗟𝗘𝗥 𝗟𝗠 𝗪𝗘𝗜𝗚𝗛𝗧 𝗙𝗢𝗢𝗧𝗣𝗥𝗜𝗡𝗧...
𝗔𝗡𝗗 𝗘𝗦𝗦𝗘𝗡𝗧𝗜𝗔𝗟𝗟𝗬 𝗧𝗛𝗘 𝗦𝗔𝗠𝗘 𝗢𝗩𝗘𝗥𝗔𝗟𝗟 𝗤𝗨𝗔𝗟𝗜𝗧𝗬! 🤯
Mine finished one question ahead overall. That's effectively a tie at this sample size, but the coding breakdown is particularly interesting!
━━━━━━━━━━━━━━━━━━━━
💻 𝗖𝗢𝗗𝗜𝗡𝗚 𝗜𝗦 𝗪𝗛𝗘𝗥𝗘 𝗦𝗔𝗚𝗘 𝗣𝗨𝗟𝗟𝗦 𝗔𝗛𝗘𝗔𝗗!
Scores shown as SAGE → Mia:
✅ HumanEval+: 39/40 → 39/40
🏆 MBPP+: 53/60 → 51/60
🏆 LiveCodeBench: 15/20 → 14/20
🏆 SciCode: 5/10 → 4/10
𝗖𝗢𝗗𝗜𝗡𝗚 𝗦𝗨𝗕𝗧𝗢𝗧𝗔𝗟:
🟢 SAGE: 112/130 — 86.15%
🟣 Mia: 108/130 — 83.08%
𝗙𝗢𝗨𝗥 𝗠𝗢𝗥𝗘 𝗖𝗢𝗗𝗜𝗡𝗚 𝗤𝗨𝗘𝗦𝗧𝗜𝗢𝗡𝗦 𝗣𝗔𝗦𝗦𝗘𝗗 𝗪𝗜𝗧𝗛 𝟯𝟬.𝟱% 𝗟𝗘𝗦𝗦 𝗠𝗢𝗗𝗘𝗟-𝗪𝗘𝗜𝗚𝗛𝗧 𝗠𝗘𝗠𝗢𝗥𝗬! 🔥
For transparency, Mia scored higher on MMLU-Pro STEM (96 vs 94) and MATH-500 (39 vs 38). Both tied on HumanEval+, MGSM, BFCL and multi-step tool chains.
This isn't about claiming one quant wins everything.
𝗜𝗧'𝗦 𝗔𝗕𝗢𝗨𝗧 𝗚𝗘𝗧𝗧𝗜𝗡𝗚 𝗠𝗢𝗥𝗘 𝗤𝗨𝗔𝗟𝗜𝗧𝗬 𝗣𝗘𝗥 𝗚𝗕.
━━━━━━━━━━━━━━━━━━━━
⚡ 𝗔𝗡𝗗 𝗬𝗘𝗦, 𝗜𝗧 𝗥𝗨𝗡𝗦 𝗢𝗡 𝘃𝗟𝗟𝗠!
My measured vLLM 0.32.0 + vllm-exl3 performance on ONE DGX Spark:
⚡ 30.51 tok/s single-stream decode
🚀 32.60 tok/s with MTP-1 (greedy code test)
🔥 89.21 tok/s aggregate decode @ C8
📥 1,119 tok/s cold prefill (2K prompt)
🧠 Native 262K context support
The public recipe is being streamlined around vLLM, with the older alternative serving instructions being retired.
My goal is to make this practical for people who want an OpenAI-compatible local endpoint without needing an 80GB+ GPU just to load the model.
The 48GB tier is the intended target; these published performance measurements were performed on my DGX Spark. Actual available context and batching capacity will depend on the GPU.
━━━━━━━━━━━━━━━━━━━━
👀 𝗕𝗨𝗧 𝗛𝗘𝗥𝗘'𝗦 𝗪𝗛𝗔𝗧 𝗜'𝗠 𝗥𝗘𝗔𝗟𝗟𝗬 𝗘𝗫𝗖𝗜𝗧𝗘𝗗 𝗔𝗕𝗢𝗨𝗧...
⚡ 𝗧𝗪𝗢 𝗠𝗢𝗗𝗘𝗟𝗦. 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞. 𝗜𝗦 𝗖𝗢𝗠𝗜𝗡𝗚 𝗕𝗔𝗖𝗞!
If you've followed my previous experiments, you know I LOVE finding ways to run multiple powerful models on a SINGLE DGX Spark.
And this ~41 GiB Ornith quant opens up some VERY interesting possibilities!
Imagine...
🧠 𝗤𝗪𝗘𝗡𝟯.𝟴-𝗙𝗟𝗔𝗦𝗛-𝗡𝗘𝗫𝗧 — 𝗠𝗔𝗦𝗧𝗘𝗥 𝗔𝗚𝗘𝗡𝗧
A larger lead model handling complex reasoning, planning, orchestration and long-context tasks.
⚡ 𝗢𝗥𝗡𝗜𝗧𝗛 𝟭.𝟱 𝟭𝟮𝟮𝗕 — 𝗦𝗨𝗕-𝗔𝗚𝗘𝗡𝗧
My compact SAGE-EXL3 handling delegated coding, tool use and specialized agentic workloads!
𝗧𝗪𝗢 𝟭𝟮𝟬𝗕+ 𝗖𝗟𝗔𝗦𝗦 𝗠𝗢𝗗𝗘𝗟𝗦.
𝗢𝗡𝗘 𝟭𝟮𝟴𝗚𝗕 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞.
𝗙𝗨𝗟𝗟𝗬 𝗟𝗢𝗖𝗔𝗟! 🤯
That's the direction I'm exploring for my NEXT Two Models. One Spark. experiment!
Of course, simultaneous residency, runtime overhead, KV-cache allocation and actual master-worker performance still need validation with the selected Qwen quantization.
But the smaller we can make these models without sacrificing useful quality, the MORE possibilities we unlock!
𝗧𝗛𝗜𝗦 𝗜𝗦 𝗪𝗛𝗬 𝗜 𝗗𝗢 𝗪𝗛𝗔𝗧 𝗜 𝗗𝗢! 💚
━━━━━━━━━━━━━━━━━━━━
🔬 𝗢𝗡𝗘 𝗟𝗜𝗧𝗧𝗟𝗘 𝗥𝗘𝗫𝗟𝟯 𝗧𝗘𝗔𝗦𝗘𝗥...
In separate development testing, REXL3 has also reached:
⚡ 90.82 tok/s aggregate @ C8
🚀 ~2,150 tok/s prefill
More on that another time. 😏
━━━━━━━━━━━━━━━━━━━━
💚 𝗠𝗔𝗞𝗜𝗡𝗚 𝗙𝗥𝗢𝗡𝗧𝗜𝗘𝗥 𝗔𝗜 𝗔𝗖𝗖𝗘𝗦𝗦𝗜𝗕𝗟𝗘!
Not everyone has a DGX Spark or an 80GB GPU.
But a LOT of enthusiasts, developers and independent researchers have access to 48GB-class hardware.
I want powerful open-weight models to be accessible to MORE of those people.
And beyond just fitting one model, I want to show what's possible when we start combining them on the SAME hardware!
Huge thanks to @ornith_ for releasing Ornith-1.5-122B under an open-weight license and continuing to push efficient AI models forward!
━━━━━━━━━━━━━━━━━━━━
🔗 𝗗𝗢𝗪𝗡𝗟𝗢𝗔𝗗 𝗠𝗬 𝗘𝗫𝗟𝟯
huggingface.co/vcruz305/Orni…
🛠️ 𝗩𝗟𝗟𝗠 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞 𝗥𝗘𝗖𝗜𝗣𝗘
github.com/vcruz305/Ornith-1…
𝗦𝗠𝗔𝗟𝗟𝗘𝗥 𝗪𝗘𝗜𝗚𝗛𝗧𝗦. 𝗕𝗜𝗚𝗚𝗘𝗥 𝗣𝗢𝗦𝗦𝗜𝗕𝗜𝗟𝗜𝗧𝗜𝗘𝗦.
𝗧𝗪𝗢 𝗠𝗢𝗗𝗘𝗟𝗦. 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞. ⚡
𝗟𝗘𝗧'𝗦 𝗞𝗘𝗘𝗣 𝗣𝗨𝗦𝗛𝗜𝗡𝗚 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗙𝗢𝗥𝗪𝗔𝗥𝗗! 🚀🔥
🚨 𝗧𝗪𝗢 𝗡𝗘𝗪 𝗗𝗘𝗘𝗣𝗦𝗘𝗘𝗞 𝗩𝟰.𝟭 𝗙𝗟𝗔𝗦𝗛 𝗥𝗘𝗟𝗘𝗔𝗦𝗘𝗦! 🛡️🔥
I’m excited to share TWO alternate DERISKED SAGE-EXL3 quantizations for cybersecurity research and security-focused AI workflows! These are the HIGHEST quality EXL3 quants you can find for this Deepseek model!
Built in collaboration with @Blackfrost_AI 🥶❄️
🟢 1.61 bpw — ONE DGX SPARK
✅ 94.68% assistant-token top-1 agreement
✅ ~112.9 GiB resident footprint
🟣 2.50 bpw — TWO DGX SPARKS
✅ 96.71% assistant-token top-1 agreement
✅ Higher precision for those with more memory - practically lossless 😎
Both measured against DeepSeek’s official FP8/FP4 checkpoint. These are token-fidelity results, not cybersecurity task scores.
𝗠𝗢𝗥𝗘 𝗖𝗛𝗢𝗜𝗖𝗘𝗦. 𝗠𝗢𝗥𝗘 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜. 𝗠𝗢𝗥𝗘 𝗙𝗥𝗘𝗘𝗗𝗢𝗠 𝗧𝗢 𝗘𝗫𝗣𝗘𝗥𝗜𝗠𝗘𝗡𝗧! 🚀
🔗 1.61 bpw — ONE SPARK
huggingface.co/vcruz305/DSV4…
🔗 2.50 bpw — TWO SPARKS
huggingface.co/vcruz305/DSV4…
Huge thanks to @Blackfrost_AI for not only making the model, but providing the compute for us to continue collaborating on these releases! 🙏🔥
🚨 𝗜 𝗝𝗨𝗦𝗧 𝗔𝗖𝗛𝗜𝗘𝗩𝗘𝗗 𝗦𝗢𝗠𝗘𝗧𝗛𝗜𝗡𝗚 𝗜𝗡𝗖𝗥𝗘𝗗𝗜𝗕𝗟𝗘 𝗢𝗡 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞! 🧪🔥
With the support of @AWERXAI, I'm excited to release something VERY special for the local AI community!
🧠 @deepseek_ai 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸 𝗩𝟰.𝟭 𝗙𝗹𝗮𝘀𝗵 𝟭𝟬𝟭𝟳
📦 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 𝗮𝘁 𝗷𝘂𝘀𝘁 𝟭.𝟲𝟮 𝗯𝗽𝘄 𝗲𝘅𝗽𝗲𝗿𝘁𝘀
💻 𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗮𝗻𝘀𝘄𝗲𝗿-𝘁𝗼𝗸𝗲𝗻 𝗮𝗴𝗿𝗲𝗲𝗺𝗲𝗻𝘁!
⚡ 𝟯𝟱.𝟱 𝘁𝗼𝗸/𝘀 𝗱𝗲𝗰𝗼𝗱𝗲 𝗼𝗻 𝘃𝗟𝗟𝗠!
🚀 𝟭,𝟭𝟴𝟵 𝘁𝗼𝗸/𝘀 𝗠𝗔𝗫 𝗣𝗥𝗘𝗙𝗜𝗟𝗟!
🧠 𝟮𝟮𝟵𝗞 𝗖𝗧𝗫 (𝘀𝘁𝗶𝗹𝗹 𝘄𝗼𝗿𝗸𝗶𝗻𝗴 𝘁𝗼 𝗽𝘂𝘀𝗵 𝗶𝘁 𝗳𝘂𝗿𝘁𝗵𝗲𝗿!)
𝗔𝗟𝗟 𝗢𝗡 𝗢𝗡𝗘. 𝗗𝗚𝗫. 𝗦𝗣𝗔𝗥𝗞. 🤯
And YES, the quantized weights are AVAILABLE NOW! 💚
━━━━━━━━━━━━━━━━━━━━
🧪 𝗧𝗛𝗘 𝗟𝗔𝗦𝗧 𝟳𝟮 𝗛𝗢𝗨𝗥𝗦 𝗛𝗔𝗩𝗘 𝗕𝗘𝗘𝗡 𝗜𝗡𝗦𝗔𝗡𝗘!
Just a few days ago, I published my article about @nvidiaai's FP4-Local format: The Good, The Bad & The Ugly.
One thing really stuck with me...
NVIDIA demonstrated what was possible when aggressive quantization and intelligent optimization come together.
But there's a problem.
𝗥𝗘𝗣𝗥𝗢𝗗𝗨𝗖𝗜𝗡𝗚 𝗠𝗔𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗡𝗧𝗜𝗭𝗔𝗧𝗜𝗢𝗡 𝗧𝗥𝗔𝗜𝗡𝗜𝗡𝗚 𝗣𝗜𝗣𝗘𝗟𝗜𝗡𝗘𝗦 𝗜𝗦 𝗢𝗨𝗧 𝗢𝗙 𝗥𝗘𝗔𝗖𝗛 𝗙𝗢𝗥 𝗠𝗢𝗦𝗧 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗘𝗡𝗧𝗛𝗨𝗦𝗜𝗔𝗦𝗧𝗦.
That inspired me.
I've been developing my own SAGE-EXL3 process, and I wanted to see just how far I could push the latest DeepSeek Flash release while keeping the results accessible to the community.
So I spent the last 72 hours pushing the limits.
And the results genuinely surprised me! 🔥
━━━━━━━━━━━━━━━━━━━━
🏆 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 𝗔𝗥𝗘 𝗖𝗥𝗔𝗭𝗬!
I measured my 1.62 bpw expert quant against DeepSeek's native FP4 release checkpoint using held-out sequences and full-vocabulary token predictions.
And when I saw the CODE results, I could barely believe it!
🧪 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗩𝗦 𝗗𝗘𝗘𝗣𝗦𝗘𝗘𝗞'𝗦 𝗡𝗔𝗧𝗜𝗩𝗘 𝗙𝗣𝟴/𝗙𝗣𝟰 𝗥𝗘𝗟𝗘𝗔𝗦𝗘
💻 𝗖𝗢𝗗𝗘 — 𝗛𝘂𝗺𝗮𝗻𝗘𝘃𝗮𝗹
✅ 99.08% Top-1 Agreement
✅ 𝟵𝟵.𝟲𝟭% 𝗔𝗡𝗦𝗪𝗘𝗥-𝗢𝗡𝗟𝗬 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧!
✅ KL Divergence: 0.0186
🧠 𝗥𝗘𝗔𝗦𝗢𝗡𝗜𝗡𝗚 — 𝗠𝗔𝗧𝗛-𝟱𝟬𝟬
✅ 96.04% Top-1 Agreement
✅ 97.98% Answer-Only Agreement
✅ KL Divergence: 0.0349
🔢 𝗠𝗔𝗧𝗛 — 𝗚𝗦𝗠𝟴𝗞
✅ 95.56% Top-1 Agreement
✅ 94.19% Answer-Only Agreement
✅ KL Divergence: 0.0477
📊 𝗢𝗩𝗘𝗥𝗔𝗟𝗟 𝗛𝗘𝗟𝗗-𝗢𝗨𝗧 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬
🏆 94.52% Top-1 Agreement
📉 Mean KL: 0.0982
🎯 Median KL: 0.00019
𝗡𝗘𝗔𝗥-𝗟𝗢𝗦𝗦𝗟𝗘𝗦𝗦 𝗖𝗢𝗗𝗘 𝗧𝗢𝗞𝗘𝗡 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗔𝗧 𝗝𝗨𝗦𝗧 𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦! 🤯
These are direct teacher-forced fidelity measurements against native FP8, not task-level pass rates.
But seeing this level of agreement at such aggressive compression is EXACTLY what I was working toward!
━━━━━━━━━━━━━━━━━━━━
⚡ 𝗔𝗡𝗗 𝗬𝗘𝗦, 𝗜𝗧 𝗔𝗖𝗧𝗨𝗔𝗟𝗟𝗬 𝗥𝗨𝗡𝗦 𝗢𝗡 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞!
Not two Sparks.
Not four Sparks.
Not an expensive rented GPU cluster.
𝗢𝗡𝗘 𝟭𝟮𝟴𝗚𝗕 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞.
My latest vLLM measurements:
⚡ Decode: 35.5 tok/s
🚀 Maximum measured prefill: 1,189 tok/s
For a model this large, compressed this aggressively, and maintaining this level of fidelity...
𝗧𝗛𝗔𝗧'𝗦 𝗔 𝗕𝗜𝗚 𝗗𝗘𝗔𝗟 𝗙𝗢𝗥 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜! 💚
And remember, 1.62 bpw refers to the routed experts, not every parameter in the full model.
━━━━━━━━━━━━━━━━━━━━
💚 𝗪𝗛𝗬 𝗧𝗛𝗜𝗦 𝗠𝗔𝗧𝗧𝗘𝗥𝗦.
I have a simple goal.
𝗠𝗔𝗞𝗘 𝗣𝗢𝗪𝗘𝗥𝗙𝗨𝗟 𝗔𝗜 𝗔𝗖𝗖𝗘𝗦𝗦𝗜𝗕𝗟𝗘 𝗧𝗢 𝗣𝗘𝗢𝗣𝗟𝗘 𝗪𝗛𝗢 𝗗𝗢𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗔𝗖𝗖𝗘𝗦𝗦 𝗧𝗢 𝗗𝗔𝗧𝗔𝗖𝗘𝗡𝗧𝗘𝗥𝗦.
I don't want the future of local AI to be determined by who can rent the most GPUs or afford the largest training runs.
I want developers, independent researchers and enthusiasts to have the freedom to experiment, quantize, optimize and run powerful models on hardware they actually OWN.
NVIDIA's FP4-Local research showed me just how much potential remains in aggressive quantization.
My goal with SAGE-EXL3 is to keep pushing that frontier while making advanced quantization more practical for the wider community.
Today, I'm sharing the quantized checkpoint and the one-Spark serving recipe.
𝗧𝗛𝗜𝗦 𝗜𝗦 𝗝𝗨𝗦𝗧 𝗧𝗛𝗘 𝗕𝗘𝗚𝗜𝗡𝗡𝗜𝗡𝗚.
━━━━━━━━━━━━━━━━━━━━
🧪 𝗡𝗢𝗪 𝗙𝗢𝗥 𝗧𝗛𝗘 𝗥𝗘𝗔𝗟 𝗧𝗘𝗦𝗧!
The token-level fidelity results are incredible, but I'm not stopping there.
I'm currently running a broader evaluation campaign to measure how closely this quant preserves the native FP8 model's actual task performance.
💻 HumanEval+ / MBPP+
🧠 MMLU-Pro STEM / MATH-500
🔬 SciCode / LiveCodeBench
🔧 BFCL / Multi-step tool chains
🌍 MGSM
𝗧𝗛𝗘 𝗚𝗢𝗔𝗟 𝗜𝗦 𝗧𝗢 𝗣𝗥𝗢𝗩𝗘 𝗧𝗛𝗔𝗧 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗖𝗢𝗠𝗣𝗥𝗘𝗦𝗦𝗜𝗢𝗡 𝗗𝗢𝗘𝗦𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗧𝗢 𝗠𝗘𝗔𝗡 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗟𝗜𝗧𝗬 𝗟𝗢𝗦𝗦!
Those task-level results are coming next! 👀
━━━━━━━━━━━━━━━━━━━━
👀 𝗔𝗡𝗗 𝗢𝗡𝗘 𝗟𝗜𝗧𝗧𝗟𝗘 𝗧𝗘𝗔𝗦𝗘𝗥...
@mr_r0b0t and I have also been working on an GB10 specialized inference engine called 𝗥𝗘𝗫𝗟𝟯.
Early testing has been VERY exciting, including 60 tok/s with bursts reaching 74tok/s!
We're putting the finishing touches on it and making sure everything is solid before releasing it.
More on that VERY soon. 😏
━━━━━━━━━━━━━━━━━━━━
🙏 𝗔 𝗛𝗨𝗚𝗘 𝗧𝗛𝗔𝗡𝗞 𝗬𝗢𝗨!
To @AWERXAI for supporting my work and helping make this contribution to the community possible.
To @nvidiaai for the FP4-Local research that inspired me to push this experiment even further.
To @justinn_builds for his initial vllm TP4 recipe that got me started in the right direction for loading this model up.
To @mr_r0b0t for collaborating with me on what's coming next!
And to @deepseek_ai for continuing to release powerful open-weight models that make community experiments like this possible.
𝗧𝗛𝗘 𝗕𝗘𝗦𝗧 𝗣𝗔𝗥𝗧 𝗢𝗙 𝗢𝗣𝗘𝗡 𝗔𝗜 𝗜𝗦 𝗪𝗛𝗔𝗧 𝗧𝗛𝗘 𝗖𝗢𝗠𝗠𝗨𝗡𝗜𝗧𝗬 𝗖𝗔𝗡 𝗕𝗨𝗜𝗟𝗗 𝗢𝗡 𝗧𝗢𝗣 𝗢𝗙 𝗜𝗧. 💚
🔗 𝗗𝗢𝗪𝗡𝗟𝗢𝗔𝗗 𝗠𝗬 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯
huggingface.co/vcruz305/DSV4…
🛠️ 𝗢𝗡𝗘-𝗦𝗣𝗔𝗥𝗞 𝗥𝗘𝗖𝗜𝗣𝗘
github.com/vcruz305/DSV4.1-F…
𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞.
𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦.
𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗔𝗡𝗦𝗪𝗘𝗥 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧.
𝗟𝗘𝗧'𝗦 𝗞𝗘𝗘𝗣 𝗣𝗨𝗦𝗛𝗜𝗡𝗚 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗙𝗢𝗥𝗪𝗔𝗥𝗗! 🚀🔥
🚨 𝗠𝗬 𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗗𝗘𝗘𝗣𝗦𝗘𝗘𝗞 𝗤𝗨𝗔𝗡𝗧 𝗝𝗨𝗦𝗧 𝗕𝗘𝗔𝗧 𝗧𝗪𝗢 𝗛𝗜𝗚𝗛𝗘𝗥-𝗣𝗥𝗘𝗖𝗜𝗦𝗜𝗢𝗡, 𝗧𝗪𝗢-𝗦𝗣𝗔𝗥𝗞 𝗤𝗨𝗔𝗡𝗧𝗦! 🧪🔥
And I have the REAL benchmark results to prove it! 🤯
Remember my DeepSeek V4.1 Flash 1017 SAGE-EXL3 release from YESTERDAY?
📦 1.62 bpw routed experts
💻 99.61% code answer-token agreement
🖥️ ONE DGX Spark @nvidiaai
Well, I promised to go beyond KL divergence and token agreement and test ACTUAL coding, mathematics, reasoning and tool execution.
𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 𝗔𝗥𝗘 𝗜𝗡!
━━━━━━━━━━━━━━━━━━━━
🏆 𝗗𝗘𝗘𝗣𝗦𝗘𝗘𝗞 𝗩𝟰.𝟭 𝗙𝗟𝗔𝗦𝗛 𝟭𝟬𝟭𝟳 — 𝗡𝗜𝗡𝗘-𝗦𝗨𝗜𝗧𝗘 𝗦𝗛𝗢𝗪𝗗𝗢𝗪𝗡
353 matched benchmark questions.
9 evaluation suites.
Strict pass@1 scoring.
Temperature 0.
🟢 𝗠𝗬 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 — 𝟭.𝟲𝟮 𝗕𝗣𝗪
🖥️ 1× DGX Spark
🏆 𝟵𝟰.𝟭% — 𝟯𝟯𝟮/𝟯𝟱𝟯 𝗣𝗔𝗦𝗦𝗘𝗗!
🔵 𝗝𝗝 𝗡𝗘𝗪 — 𝟮.𝟴 𝗕𝗣𝗪
🖥️ 2× DGX Sparks
📊 90.1% — 318/353
🟣 𝗠𝗜𝗔𝗟𝗔𝗕 — 𝟮.𝟵 𝗕𝗣𝗪
🖥️ 2× DGX Sparks
📊 88.4% — 312/353
𝗜'𝗠 𝗨𝗦𝗜𝗡𝗚 𝟰𝟮–𝟰𝟰% 𝗙𝗘𝗪𝗘𝗥 𝗕𝗜𝗧𝗦 𝗣𝗘𝗥 𝗥𝗢𝗨𝗧𝗘𝗗 𝗘𝗫𝗣𝗘𝗥𝗧 𝗣𝗔𝗥𝗔𝗠𝗘𝗧𝗘𝗥...
𝗔𝗡𝗗 𝗦𝗖𝗢𝗥𝗜𝗡𝗚 𝗛𝗜𝗚𝗛𝗘𝗥 𝗢𝗡 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞! 🤯
━━━━━━━━━━━━━━━━━━━━
💻 𝗧𝗛𝗘 𝗖𝗢𝗗𝗜𝗡𝗚 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 𝗔𝗥𝗘 𝗜𝗡𝗖𝗥𝗘𝗗𝗜𝗕𝗟𝗘!
Scores shown as SAGE / JJ / Mia:
🏆 HumanEval+
39/40 vs 34/40 vs 34/40
🏆 MBPP+
53/60 vs 50/60 vs 49/60
🏆 LiveCodeBench
15/17 vs 14/17 vs 11/17
🧠 𝗥𝗘𝗔𝗦𝗢𝗡𝗜𝗡𝗚 & 𝗠𝗔𝗧𝗛
🏆 MMLU-Pro STEM
95/100 vs 93/100 vs 91/100
🏆 MATH-500
40/40 vs 36/40 vs 36/40
𝗙𝗜𝗩𝗘 𝗢𝗨𝗧𝗥𝗜𝗚𝗛𝗧 𝗪𝗜𝗡𝗦! 🔥
I also tied both models on:
✅ MGSM: 40/40
✅ BFCL: 39/40
✅ SciCode first-step: 6/10
For full transparency, both JJ and Mia won the multi-step tool-chain test, scoring 6/6 against my 5/6.
𝗦𝗢 𝗧𝗛𝗔𝗧'𝗦 𝟱 𝗪𝗜𝗡𝗦, 𝟯 𝗧𝗜𝗘𝗦, 𝗔𝗡𝗗 𝟭 𝗟𝗢𝗦𝗦.
All while using a substantially lower expert bitrate!
━━━━━━━━━━━━━━━━━━━━
🚀 𝗕𝗜𝗚 𝗨𝗣𝗗𝗔𝗧𝗘: 𝗠𝗬 𝗣𝗨𝗕𝗟𝗜𝗖 𝘃𝗟𝗟𝗠 𝗥𝗘𝗖𝗜𝗣𝗘 𝗜𝗦 𝗡𝗢𝗪 𝗣𝗨𝗦𝗛𝗜𝗡𝗚 𝟱𝟮𝟰𝗞 𝗖𝗢𝗡𝗧𝗘𝗫𝗧!
YES. ON ONE DGX SPARK! 🤯
With the latest W2 kernel, updated expert set and KV-cache configuration, I just completed an actual long-context retrieval test!
📚 𝗙𝗨𝗟𝗟-𝗟𝗘𝗡𝗚𝗧𝗛 𝗣𝗥𝗢𝗠𝗣𝗧 𝗧𝗘𝗦𝗧
🧠 518,103 INPUT TOKENS!
🎯 Fact planted HALFWAY through the context
✅ THE MODEL FOUND THE FACT!
⏱️ Time to first token: 470 seconds (~7.8 minutes)
🚀 Prefill: ~1,100 tok/s
⚡ Decode at that context depth: 25.6 tok/s
💾 Minimum available memory: 2.19 GiB
🛡️ Memory safety floor: 0.5 GiB
And on SHORTER prompts?
⚡ ~36 TOK/S DECODE!
🔧 ~130.5 ms per speculative step
🎯 ~4.7 accepted tokens per step
𝗪𝗘'𝗥𝗘 𝗧𝗔𝗟𝗞𝗜𝗡𝗚 𝗔𝗕𝗢𝗨𝗧 𝗢𝗩𝗘𝗥 𝗛𝗔𝗟𝗙 𝗔 𝗠𝗜𝗟𝗟𝗜𝗢𝗡 𝗜𝗡𝗣𝗨𝗧 𝗧𝗢𝗞𝗘𝗡𝗦 𝗣𝗥𝗢𝗖𝗘𝗦𝗦𝗘𝗗 𝗢𝗡 𝗔 𝗦𝗜𝗡𝗚𝗟𝗘 𝗦𝗣𝗔𝗥𝗞! 🔥
To be clear, this was a successful full-length retrieval test, not a claim of perfect accuracy across every possible long-context task.
And the 7.8-minute prefill is a real trade-off at that depth.
But seeing a model this large, with this level of compression, successfully retrieve information buried inside 518K tokens on ONE 128GB machine is an exciting milestone for local AI!
━━━━━━━━━━━━━━━━━━━━
⚡ 𝗔𝗡𝗗 𝗧𝗛𝗘 𝗢𝗡𝗘-𝗦𝗣𝗔𝗥𝗞 𝗣𝗘𝗥𝗙𝗢𝗥𝗠𝗔𝗡𝗖𝗘 𝗞𝗘𝗘𝗣𝗦 𝗜𝗠𝗣𝗥𝗢𝗩𝗜𝗡𝗚!
Separately, my latest REXL3 development measurements at a 256K context configuration:
⚡ 55.35 tok/s single-stream decode
🚀 2,045 tok/s peak prefill
⏱️ 509 ms TTFT on a 512-token prompt
Those REXL3 numbers come from a different engine and workload than the new public vLLM 518K-token test.
And here's something else I'm particularly excited about...
𝗔𝗟𝗟 𝟭𝟱,𝟯𝟲𝟬 𝗥𝗢𝗨𝗧𝗘𝗗 𝗘𝗫𝗣𝗘𝗥𝗧𝗦 + 𝟯𝟴𝟰 𝗗𝗥𝗔𝗙𝗧 𝗘𝗫𝗣𝗘𝗥𝗧𝗦 𝗔𝗥𝗘 𝟭𝟬𝟬% 𝗚𝗣𝗨-𝗥𝗘𝗦𝗜𝗗𝗘𝗡𝗧 𝗜𝗡 𝗠𝗬 𝗧𝗘𝗦𝗧𝗘𝗗 𝗥𝗘𝗫𝗟𝟯 𝗖𝗢𝗡𝗙𝗜𝗚𝗨𝗥𝗔𝗧𝗜𝗢𝗡!
ZERO streamed experts.
𝗢𝗡𝗘. 𝟭𝟮𝟴𝗚𝗕. 𝗗𝗚𝗫. 𝗦𝗣𝗔𝗥𝗞. 💚
@mr_r0b0t and I are continuing to refine REXL3 before its release. More on that soon! 👀
━━━━━━━━━━━━━━━━━━━━
💚 𝗧𝗛𝗜𝗦 𝗜𝗦 𝗪𝗛𝗬 𝗜 𝗕𝗨𝗜𝗟𝗗 𝗙𝗢𝗥 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜.
Just YESTERDAY, I shared my goal of bringing extremely aggressive, high-quality quantization to people who don't have access to massive compute clusters.
With the support of @AWERXAI, I've been working tirelessly to push that vision forward.
And now we're seeing a 1.62 bpw expert quant outperform higher-bitrate alternatives on actual tasks, not just token-level fidelity measurements.
This doesn't establish mathematical losslessness or superiority on every possible workload. These are matched subsets of nine benchmark suites, not full public leaderboard scores.
But it demonstrates something IMPORTANT:
𝗠𝗢𝗥𝗘 𝗕𝗜𝗧𝗦 𝗗𝗢𝗡'𝗧 𝗔𝗨𝗧𝗢𝗠𝗔𝗧𝗜𝗖𝗔𝗟𝗟𝗬 𝗠𝗘𝗔𝗡 𝗕𝗘𝗧𝗧𝗘𝗥 𝗥𝗘𝗦𝗨𝗟𝗧𝗦.
And more hardware doesn't automatically mean you need it!
Huge respect to @jayleaton for his contributions and for giving the community more ways to run these incredible models.
And thank you @AWERXAI for supporting this work. 🙏
🔗 𝗠𝗬 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 𝗤𝗨𝗔𝗡𝗧
huggingface.co/vcruz305/DSV4…
🛠️ 𝗢𝗡𝗘-𝗦𝗣𝗔𝗥𝗞 𝘃𝗟𝗟𝗠 𝗥𝗘𝗖𝗜𝗣𝗘
github.com/vcruz305/DSV4.1-F…
𝗙𝗥𝗢𝗡𝗧𝗜𝗘𝗥 𝗠𝗢𝗗𝗘𝗟𝗦. 𝗟𝗢𝗖𝗔𝗟 𝗛𝗔𝗥𝗗𝗪𝗔𝗥𝗘. 𝗥𝗘𝗔𝗟 𝗥𝗘𝗦𝗨𝗟𝗧𝗦.
𝗟𝗘𝗧'𝗦 𝗞𝗘𝗘𝗣 𝗣𝗨𝗦𝗛𝗜𝗡𝗚! 🚀🔥
🚨 𝗜 𝗝𝗨𝗦𝗧 𝗔𝗖𝗛𝗜𝗘𝗩𝗘𝗗 𝗦𝗢𝗠𝗘𝗧𝗛𝗜𝗡𝗚 𝗜𝗡𝗖𝗥𝗘𝗗𝗜𝗕𝗟𝗘 𝗢𝗡 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞! 🧪🔥
With the support of @AWERXAI, I'm excited to release something VERY special for the local AI community!
🧠 @deepseek_ai 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸 𝗩𝟰.𝟭 𝗙𝗹𝗮𝘀𝗵 𝟭𝟬𝟭𝟳
📦 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 𝗮𝘁 𝗷𝘂𝘀𝘁 𝟭.𝟲𝟮 𝗯𝗽𝘄 𝗲𝘅𝗽𝗲𝗿𝘁𝘀
💻 𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗮𝗻𝘀𝘄𝗲𝗿-𝘁𝗼𝗸𝗲𝗻 𝗮𝗴𝗿𝗲𝗲𝗺𝗲𝗻𝘁!
⚡ 𝟯𝟱.𝟱 𝘁𝗼𝗸/𝘀 𝗱𝗲𝗰𝗼𝗱𝗲 𝗼𝗻 𝘃𝗟𝗟𝗠!
🚀 𝟭,𝟭𝟴𝟵 𝘁𝗼𝗸/𝘀 𝗠𝗔𝗫 𝗣𝗥𝗘𝗙𝗜𝗟𝗟!
🧠 𝟮𝟮𝟵𝗞 𝗖𝗧𝗫 (𝘀𝘁𝗶𝗹𝗹 𝘄𝗼𝗿𝗸𝗶𝗻𝗴 𝘁𝗼 𝗽𝘂𝘀𝗵 𝗶𝘁 𝗳𝘂𝗿𝘁𝗵𝗲𝗿!)
𝗔𝗟𝗟 𝗢𝗡 𝗢𝗡𝗘. 𝗗𝗚𝗫. 𝗦𝗣𝗔𝗥𝗞. 🤯
And YES, the quantized weights are AVAILABLE NOW! 💚
━━━━━━━━━━━━━━━━━━━━
🧪 𝗧𝗛𝗘 𝗟𝗔𝗦𝗧 𝟳𝟮 𝗛𝗢𝗨𝗥𝗦 𝗛𝗔𝗩𝗘 𝗕𝗘𝗘𝗡 𝗜𝗡𝗦𝗔𝗡𝗘!
Just a few days ago, I published my article about @nvidiaai's FP4-Local format: The Good, The Bad & The Ugly.
One thing really stuck with me...
NVIDIA demonstrated what was possible when aggressive quantization and intelligent optimization come together.
But there's a problem.
𝗥𝗘𝗣𝗥𝗢𝗗𝗨𝗖𝗜𝗡𝗚 𝗠𝗔𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗡𝗧𝗜𝗭𝗔𝗧𝗜𝗢𝗡 𝗧𝗥𝗔𝗜𝗡𝗜𝗡𝗚 𝗣𝗜𝗣𝗘𝗟𝗜𝗡𝗘𝗦 𝗜𝗦 𝗢𝗨𝗧 𝗢𝗙 𝗥𝗘𝗔𝗖𝗛 𝗙𝗢𝗥 𝗠𝗢𝗦𝗧 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗘𝗡𝗧𝗛𝗨𝗦𝗜𝗔𝗦𝗧𝗦.
That inspired me.
I've been developing my own SAGE-EXL3 process, and I wanted to see just how far I could push the latest DeepSeek Flash release while keeping the results accessible to the community.
So I spent the last 72 hours pushing the limits.
And the results genuinely surprised me! 🔥
━━━━━━━━━━━━━━━━━━━━
🏆 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 𝗔𝗥𝗘 𝗖𝗥𝗔𝗭𝗬!
I measured my 1.62 bpw expert quant against DeepSeek's native FP4 release checkpoint using held-out sequences and full-vocabulary token predictions.
And when I saw the CODE results, I could barely believe it!
🧪 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗩𝗦 𝗗𝗘𝗘𝗣𝗦𝗘𝗘𝗞'𝗦 𝗡𝗔𝗧𝗜𝗩𝗘 𝗙𝗣𝟴/𝗙𝗣𝟰 𝗥𝗘𝗟𝗘𝗔𝗦𝗘
💻 𝗖𝗢𝗗𝗘 — 𝗛𝘂𝗺𝗮𝗻𝗘𝘃𝗮𝗹
✅ 99.08% Top-1 Agreement
✅ 𝟵𝟵.𝟲𝟭% 𝗔𝗡𝗦𝗪𝗘𝗥-𝗢𝗡𝗟𝗬 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧!
✅ KL Divergence: 0.0186
🧠 𝗥𝗘𝗔𝗦𝗢𝗡𝗜𝗡𝗚 — 𝗠𝗔𝗧𝗛-𝟱𝟬𝟬
✅ 96.04% Top-1 Agreement
✅ 97.98% Answer-Only Agreement
✅ KL Divergence: 0.0349
🔢 𝗠𝗔𝗧𝗛 — 𝗚𝗦𝗠𝟴𝗞
✅ 95.56% Top-1 Agreement
✅ 94.19% Answer-Only Agreement
✅ KL Divergence: 0.0477
📊 𝗢𝗩𝗘𝗥𝗔𝗟𝗟 𝗛𝗘𝗟𝗗-𝗢𝗨𝗧 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬
🏆 94.52% Top-1 Agreement
📉 Mean KL: 0.0982
🎯 Median KL: 0.00019
𝗡𝗘𝗔𝗥-𝗟𝗢𝗦𝗦𝗟𝗘𝗦𝗦 𝗖𝗢𝗗𝗘 𝗧𝗢𝗞𝗘𝗡 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗔𝗧 𝗝𝗨𝗦𝗧 𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦! 🤯
These are direct teacher-forced fidelity measurements against native FP8, not task-level pass rates.
But seeing this level of agreement at such aggressive compression is EXACTLY what I was working toward!
━━━━━━━━━━━━━━━━━━━━
⚡ 𝗔𝗡𝗗 𝗬𝗘𝗦, 𝗜𝗧 𝗔𝗖𝗧𝗨𝗔𝗟𝗟𝗬 𝗥𝗨𝗡𝗦 𝗢𝗡 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞!
Not two Sparks.
Not four Sparks.
Not an expensive rented GPU cluster.
𝗢𝗡𝗘 𝟭𝟮𝟴𝗚𝗕 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞.
My latest vLLM measurements:
⚡ Decode: 35.5 tok/s
🚀 Maximum measured prefill: 1,189 tok/s
For a model this large, compressed this aggressively, and maintaining this level of fidelity...
𝗧𝗛𝗔𝗧'𝗦 𝗔 𝗕𝗜𝗚 𝗗𝗘𝗔𝗟 𝗙𝗢𝗥 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜! 💚
And remember, 1.62 bpw refers to the routed experts, not every parameter in the full model.
━━━━━━━━━━━━━━━━━━━━
💚 𝗪𝗛𝗬 𝗧𝗛𝗜𝗦 𝗠𝗔𝗧𝗧𝗘𝗥𝗦.
I have a simple goal.
𝗠𝗔𝗞𝗘 𝗣𝗢𝗪𝗘𝗥𝗙𝗨𝗟 𝗔𝗜 𝗔𝗖𝗖𝗘𝗦𝗦𝗜𝗕𝗟𝗘 𝗧𝗢 𝗣𝗘𝗢𝗣𝗟𝗘 𝗪𝗛𝗢 𝗗𝗢𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗔𝗖𝗖𝗘𝗦𝗦 𝗧𝗢 𝗗𝗔𝗧𝗔𝗖𝗘𝗡𝗧𝗘𝗥𝗦.
I don't want the future of local AI to be determined by who can rent the most GPUs or afford the largest training runs.
I want developers, independent researchers and enthusiasts to have the freedom to experiment, quantize, optimize and run powerful models on hardware they actually OWN.
NVIDIA's FP4-Local research showed me just how much potential remains in aggressive quantization.
My goal with SAGE-EXL3 is to keep pushing that frontier while making advanced quantization more practical for the wider community.
Today, I'm sharing the quantized checkpoint and the one-Spark serving recipe.
𝗧𝗛𝗜𝗦 𝗜𝗦 𝗝𝗨𝗦𝗧 𝗧𝗛𝗘 𝗕𝗘𝗚𝗜𝗡𝗡𝗜𝗡𝗚.
━━━━━━━━━━━━━━━━━━━━
🧪 𝗡𝗢𝗪 𝗙𝗢𝗥 𝗧𝗛𝗘 𝗥𝗘𝗔𝗟 𝗧𝗘𝗦𝗧!
The token-level fidelity results are incredible, but I'm not stopping there.
I'm currently running a broader evaluation campaign to measure how closely this quant preserves the native FP8 model's actual task performance.
💻 HumanEval+ / MBPP+
🧠 MMLU-Pro STEM / MATH-500
🔬 SciCode / LiveCodeBench
🔧 BFCL / Multi-step tool chains
🌍 MGSM
𝗧𝗛𝗘 𝗚𝗢𝗔𝗟 𝗜𝗦 𝗧𝗢 𝗣𝗥𝗢𝗩𝗘 𝗧𝗛𝗔𝗧 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗖𝗢𝗠𝗣𝗥𝗘𝗦𝗦𝗜𝗢𝗡 𝗗𝗢𝗘𝗦𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗧𝗢 𝗠𝗘𝗔𝗡 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗟𝗜𝗧𝗬 𝗟𝗢𝗦𝗦!
Those task-level results are coming next! 👀
━━━━━━━━━━━━━━━━━━━━
👀 𝗔𝗡𝗗 𝗢𝗡𝗘 𝗟𝗜𝗧𝗧𝗟𝗘 𝗧𝗘𝗔𝗦𝗘𝗥...
@mr_r0b0t and I have also been working on an GB10 specialized inference engine called 𝗥𝗘𝗫𝗟𝟯.
Early testing has been VERY exciting, including 60 tok/s with bursts reaching 74tok/s!
We're putting the finishing touches on it and making sure everything is solid before releasing it.
More on that VERY soon. 😏
━━━━━━━━━━━━━━━━━━━━
🙏 𝗔 𝗛𝗨𝗚𝗘 𝗧𝗛𝗔𝗡𝗞 𝗬𝗢𝗨!
To @AWERXAI for supporting my work and helping make this contribution to the community possible.
To @nvidiaai for the FP4-Local research that inspired me to push this experiment even further.
To @justinn_builds for his initial vllm TP4 recipe that got me started in the right direction for loading this model up.
To @mr_r0b0t for collaborating with me on what's coming next!
And to @deepseek_ai for continuing to release powerful open-weight models that make community experiments like this possible.
𝗧𝗛𝗘 𝗕𝗘𝗦𝗧 𝗣𝗔𝗥𝗧 𝗢𝗙 𝗢𝗣𝗘𝗡 𝗔𝗜 𝗜𝗦 𝗪𝗛𝗔𝗧 𝗧𝗛𝗘 𝗖𝗢𝗠𝗠𝗨𝗡𝗜𝗧𝗬 𝗖𝗔𝗡 𝗕𝗨𝗜𝗟𝗗 𝗢𝗡 𝗧𝗢𝗣 𝗢𝗙 𝗜𝗧. 💚
🔗 𝗗𝗢𝗪𝗡𝗟𝗢𝗔𝗗 𝗠𝗬 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯
huggingface.co/vcruz305/DSV4…
🛠️ 𝗢𝗡𝗘-𝗦𝗣𝗔𝗥𝗞 𝗥𝗘𝗖𝗜𝗣𝗘
github.com/vcruz305/DSV4.1-F…
𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞.
𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦.
𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗔𝗡𝗦𝗪𝗘𝗥 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧.
𝗟𝗘𝗧'𝗦 𝗞𝗘𝗘𝗣 𝗣𝗨𝗦𝗛𝗜𝗡𝗚 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗙𝗢𝗥𝗪𝗔𝗥𝗗! 🚀🔥
🚨 𝗦𝗣𝗔𝗖𝗘 𝗕𝗨𝗡𝗡𝗬 𝗪𝗔𝗦 𝗠𝗜𝗡𝗜𝗠𝗔𝗫?! 🐰🔥
I KNEW there was something special about this model! 🤯
I used Space Bunny for FREE through @NousResearch’s Portal and @OpenRouter at work, and let me tell you…
𝗧𝗛𝗜𝗦 𝗠𝗢𝗗𝗘𝗟 𝗜𝗦 𝗜𝗡𝗖𝗥𝗘𝗗𝗜𝗕𝗟𝗘!
⚡ BLAZING FAST
🧠 Excellent at understanding complex tasks
💻 Fantastic for coding and agentic workflows
But the BEST part?
𝗜 𝗗𝗜𝗗𝗡’𝗧 𝗛𝗔𝗩𝗘 𝗧𝗢 𝗘𝗫𝗣𝗟𝗔𝗜𝗡 𝗠𝗨𝗖𝗛. 𝗜𝗧 𝗝𝗨𝗦𝗧 𝗪𝗢𝗥𝗞𝗘𝗗! 🔥
And now @MiniMax_AI confirms Space Bunny was an early version of M3.1 Flash!
🐰 200 TRILLION+ tokens processed anonymously!
🚀 New release: 170+ tok/s median!
💰 70% lower agentic task costs than its predecessor!
THIS is the kind of practical, efficient AI I’m excited about!
Huge congratulations to the MiniMax team! 💚🚀
Now get me those weightssssss 😏
MiniMax M3.1 Flash: Space Performance + Bunny Speed
🚀🐰 Yes, Space Bunny - which has topped global leaderboards with 200T+ tokens - was an earlier version of M3.1 Flash.
💰Redefining the performance economics of agentic AI. M3.1 Flash achieves frontier-level intelligence across key tasks such as Cowork (Automation-Bench: 62%; ALE: 30%), Coding (DeepSWE: 74%) and Visual Reasoning (BabyVision: 91%), at 1/3 the cost of M3 and 3x the speed.
API pricing (flat rate): $0.15 / $0.60 per 1M input/output tokens, with cache reads at $0.015 / 1M tokens.
Model weights will arrive on Oct 23.
API: platform.minimax.io
M Plan: minimax.io/m-plan
MiniMax Code for desktop: agent.minimax.io/download
Correction, @NVIDIAAI spoken about this! You just need to send them an email asking if this is supported or not and youll get something like what my friend @MarkusEicher70 got back from them:
"NVIDIA DOES NOT officially support or recommend manually overriding the GPU clock limits on DGX Spark systems using commands such as: sudo nvidia-smi -lgc"
Unless youre seeing extreme performance degredation on your DGX Spark due to high heats and youre also up to date on all updates especially kernel and firmware, then theres zero reason to underclock.
There's a big argument on whether you _should_ cap the clock rates on your DGX Sparks because they are, in theory, supposed to cool themselves to safe levels.
NVIDIA doesn't recommend capping them, but they also don't recommend not capping them. Here's my thoughts on this:
I have a couple of DGX Sparks that are marginal when running heavy prefill on GLM 5.3. This model pushes the box to the max. I'm not kidding - it jumps from ~47C to 87C in under a minute. During a heavy TP=3 prefill test, one box reliably powers itself off (!) without a clock cap. With a clock cap, it stays on.
I have an additional fan moving air over the Sparks. I can add additional cooling, but this is a datapoint where NVIDIA's guidance is, in fact, wrong. Not all DGX Sparks can ramp their fans and/or throttle their own power fast enough to prevent thermal shutdown without a clock cap.
2200 MHz is enough to keep it alive. You know what? I'm happy to give up a small 1-2% gap in peak performance to keep the Sparks alive. Heat over time is one of the silent killer of chips.
🚨 @nvidiaai FP4-LOCAL: THE GOOD, THE BAD & THE UGLY! 🔥
I tested Qwen3.8-Flash-Next 4 ways. FP4-Local TP1 & 2, NVFP4 and EXL3
FP4: 1 Spark: 90.8% SixCat, 68 tok/s. 2 Sparks: 32% slower decode. DIY quantization? That's the ugly part.
My full article 👇
