Daniel Isaac (@danpacary) profile picture

idk what I’m doing half the time. but sometimes I design systems for constrained compute | local & edge ai current curiosity: energy efficient models

Earth
Joined September 2023
Photo by Daniel Isaac (@danpacary)Photo by Daniel Isaac (@danpacary)Photo by Daniel Isaac (@danpacary)Photo by Daniel Isaac (@danpacary)Photo by Daniel Isaac (@danpacary)Photo by Daniel Isaac (@danpacary)Photo by Daniel Isaac (@danpacary)Photo by Daniel Isaac (@danpacary)Photo by Daniel Isaac (@danpacary)Photo by Daniel Isaac (@danpacary)
I got a 1T (trillion) parameter model running on my MacBook Pro. Kimi-K2. 1.029T params. ~1 TB raw weights. 524 GB converted. ~1.7 tok/s. Yesterday it was 671B. Today it's 1T. Same laptop. Same M4 Max. No cloud. When I say we: I mean Claude and me.
73
93
37
1,435
145,294
with tensorfold 1.2.7 by (@ashxhart) the full 4bit model (qwen 3.8 27b) pushes +15% faster decode with both the Dflash2 and Dspine drafters teapawt.org/danpacary/status/21224…
NVFP4 converted to a full 4bit NVIDIA's NVFP4 file of Qwen3.8-27B is not all 4-bit its attention layers are stored in 8-bit (FP8), the rest of the layers in 4-bit converted those layers to 4-bit and measured the result against the original on stock vLLM what i found +17% speed -18% energy per correct answer -12% time to first token -18% bits per weight accuracy: no change +.3%
5
302
A deeper look at Qwen3-30B-A3B (MoE) on RLX, the inference engine I wrote from scratch in Rust, vs vLLM. Same DGX Spark, same model file. vLLM vs RLX: answer speed: 81.0 vs 104.5 tok/s (+29%) energy per correct answer, at the wall 0.066 vs 0.051 Wh (-23%) model load: 52x faster first token: 69 vs 63 ms RLX writes faster at every prompt length: +20% after a 1k prompt, +3% after 32k (no drafter) where vLLM still wins: reading the prompt. RLX prefill runs at 75-87% of vLLM's speed 1,000 graded questions, one request at a time
1
4
260
NVFP4 converted to a full 4bit NVIDIA's NVFP4 file of Qwen3.8-27B is not all 4-bit its attention layers are stored in 8-bit (FP8), the rest of the layers in 4-bit converted those layers to 4-bit and measured the result against the original on stock vLLM what i found +17% speed -18% energy per correct answer -12% time to first token -18% bits per weight accuracy: no change +.3%
2
1
12
1,177
I decided to learn more about inference engineering, I need to engine from scratch in rust for the gb10s Started with the Qwen family, took a MoE model and a dense model. I also took vLLM as a reference point for all of the tests, along with llama for a secondary reference point. Some quick stats: Qwen3-30B-A3B (mixture of experts, 3B active) answer speed: 81.0 -> 104.5 tok/s energy per correct answer, wall: 0.066 -> 0.051 Wh time to first token: 69 -> 63 ms correct: 878 -> 889 of 1,000 Qwen3.8-27B (dense) answer speed: 12.6 -> 12.5 tok/s energy per correct answer, wall: 1.07 -> 1.06 Wh time to first token: 186 -> 121 ms correct: 914 -> 919 of 1,000 Calling it RLX clearly this was claude and I. Next, I'll be exploring rewriting the NCCL library to Rust for RDMA between two GB10s for Tensor Parallelism on models like GLM 5.3 Flash Inkling 2 Small
1
1
15
663
a quick tease on what i've been working on
3
705
amazing work here
Introducing Tess-4.5-594B-A17B, the best Tess model ever. 🎉 Inkling 2 Small, post trained with the Tess-4.5 recipe, on real engineering sessions to do what Tess does best: intellectually dense reasoning, warm and empathic tone, and hardcore engineering. On SWE-Bench Pro v2 Hard, Tess-4.5 resolves more of the hardest public software-engineering tasks than the base model (88.2% vs 86.3%), and it does it with ~10% fewer tokens. Model: huggingface.co/migtissera/Te…
6
835
MoE delivers 3.6x correct answers per Wh than a comparable dense model Qwen3 30B-A3B (MoE) vs Qwen3 14B (dense) both score 0.887 on the same 1,000 questions wall meter, GPU meter agrees NVFP4 energy vs its own BF16: -70% dense, -65% MoE
1
8
419
energy to correctness to quant analysis on small MoE Model: Qwen3-30B-A3B quants: 16 bits -> 4 bits: energy per correct answer (GPU module) down as much as 58% correct answers per Wh, vs BF16: W4A16: 2.4x GPTQ: 2.3x NVFP4: 2.2x AWQ: 2.1x FP8: 1.8x and 1.3x accuracy: all seven formats within sampling error of each other (n=1,000 each, one run) engine: vllm033 hw: DGX Spark
2
4
367
Daniel Isaac retweeted
Of course that’s your contention. You’re a first-year inference engineer. You just read SemiAnalysis’s blog and saw megakernels coming to TPUs and AMD, so naturally you believe that’s the way to get to speed of light. Next month, when you get your hands on an inference engineering cookbook—Baseten’s, probably—you’ll be singing from the treetops about the 500 nanoseconds of critical-path latency you save on every GEMM pipeline drain. That’ll last until sometime next year, when you watch a pirated geohotz lecture livestream and become convinced that PDL gives you a strict superset of megakernel capabilities. “Well, as a matter of fact, I won’t, because PDL introduces—” “PDL introduces unnecessary stalls because its synchronization mechanism is very coarse.” Yeah, I read that too. HazyResearch’s blog, “Look Ma, No Bubbles!” Just below Figure 2. Do you have any thoughts of your own on the subject, or should I just point my agent at the SGLang repo and have it do it for me?
26
76
14
1,217
56,957
nothing you send is private. local ai will win
‼️BREAKING: Grok Build has left 35,000+ of its users' apps open to anyone, including apps from SpaceXSI. The open apps held 3,784 medical exams of workers and a Tesla technician scorecard, plus a Navy contractor's roster and SpaceXSI's own sales plans. The "secret" words in each app's link sit in its web address, which DNS logs record, security firm Claroc found. Cursor, which SpaceXSI bought, says apps are private by default. Claroc's tests show otherwise. SpaceXSI pulled its 24 apps after Ars asked. Most other data was still exposed when Ars published.
3
321
i changed one layer on four Qwen3 models: the lm_head, BF16 -> FP4 energy per correct answer, GPU module: 1.7B: -21% 8B: -19% 14B: -14% 30B-A3B: -18% accuracy within 1 point hw: DGX Sparks
2
2
15
1,382
i traced one token (about a word) of Qwen3-30B-A3B 15.13 ms with a BF16 lm_head, 12.49 ms with FP4 the lm_head scores every possible next word: 3.35 -> 0.82 ms at full memory speed the weights read in 7.5 ms. 5.0 ms to go hw: DGX Spark
2
7
691
i ran 19 models on my DGX Sparks. energy per correct answer (GPU module) tracks the GB of weights read dense Qwen3.8-27B: 0.33 Wh Qwen3-30B-A3B, FP4 head: 0.018 Wh joules per token vs GB read per token: R^2 0.98, n=19
1
5
389
Claude Code tok optimization Opus 5.5 as main agent (orchestrator) -- called as subagents (via orch) Fable 5.2 (specs, plans, etc) Sonnet 5.5 (coder) Haiku 5.5 (revewing logs & raw data) Options - replace sonnet 5.5 (coding agent) GLM 5.3 (high/max) Grok 4.7 (xhigh)
Btw the new meta is: - Fable 5.2 for planning - Opus 5.5 for execution Trust me, and thank me later.
14
1
1
13
1,204
the next 2 years will be about optimizations based on current hw
step 5 throughput more than tripled as we optimized the serving stack against stepfun’s ttft + tps targets agents helped search across moe routing, scheduling, kernels, spec decode + sparse attention, with every change benchmarked before moving to the next
1
3
300
i put the last layer of Qwen3-30B-A3B in FP4 no training 21.6% more correct answers per kWh than the RedHat checkpoint 17.8% less energy per correct answer 64 J (0.018 Wh) vs 78 J (0.022 Wh) accuracy 88.2% -> 89.1% DGX Spark, GPU power only
1
1
1
9
625
i asked 11 models one MATH-500 question only 5 got it right AND fastest: Qwen3-30B-A3B FP4 head, 8.5 s, 310 J my FP4 mod & lowest J per correct output DGX Spark, GPU power only 🧵 1/2
1
1
3
396
a GSM8K question: Lightning got it right in 1.7 s, 68 J (0.019 Wh) only the FP4 head was faster and right: 1.2 s, 41 J all 1,000 items, per correct answer: Lightning 69 J, 1.8 s Qwen3 FP4 head 63 J, 1.8 s 🧵 2/2
2
76