Pinned Tweet
I got a 1T (trillion) parameter model running on my MacBook Pro.
Kimi-K2. 1.029T params.
~1 TB raw weights.
524 GB converted.
~1.7 tok/s.
Yesterday it was 671B. Today it's 1T.
Same laptop. Same M4 Max. No cloud.
When I say we: I mean Claude and me.
with tensorfold 1.2.7 by (@ashxhart)
the full 4bit model (qwen 3.8 27b) pushes
+15% faster decode
with both the Dflash2 and Dspine drafters
teapawt.org/danpacary/status/21224…
NVFP4 converted to a full 4bit
NVIDIA's NVFP4 file of Qwen3.8-27B is not all 4-bit
its attention layers are stored in 8-bit (FP8), the rest of the layers in 4-bit
converted those layers to 4-bit and measured the result against the original on stock vLLM
what i found
+17% speed
-18% energy per correct answer
-12% time to first token
-18% bits per weight
accuracy: no change +.3%
5
302
A deeper look at Qwen3-30B-A3B (MoE) on RLX,
the inference engine I wrote from scratch in Rust, vs vLLM. Same DGX Spark, same model file.
vLLM vs RLX:
answer speed: 81.0 vs 104.5 tok/s (+29%)
energy per correct answer, at the wall
0.066 vs 0.051 Wh (-23%)
model load: 52x faster
first token: 69 vs 63 ms
RLX writes faster at every prompt length: +20% after a 1k prompt, +3% after 32k
(no drafter)
where vLLM still wins: reading the prompt. RLX prefill runs at 75-87% of vLLM's speed
1,000 graded questions, one request at a time
NVFP4 converted to a full 4bit
NVIDIA's NVFP4 file of Qwen3.8-27B is not all 4-bit
its attention layers are stored in 8-bit (FP8), the rest of the layers in 4-bit
converted those layers to 4-bit and measured the result against the original on stock vLLM
what i found
+17% speed
-18% energy per correct answer
-12% time to first token
-18% bits per weight
accuracy: no change +.3%
I decided to learn more about inference engineering, I need to engine from scratch in rust for the gb10s
Started with the Qwen family, took a MoE model and a dense model. I also took vLLM as a reference point for all of the tests, along with llama for a secondary reference point.
Some quick stats:
Qwen3-30B-A3B (mixture of experts, 3B active)
answer speed: 81.0 -> 104.5 tok/s
energy per correct answer, wall: 0.066 -> 0.051 Wh
time to first token: 69 -> 63 ms
correct: 878 -> 889 of 1,000
Qwen3.8-27B (dense)
answer speed: 12.6 -> 12.5 tok/s
energy per correct answer, wall: 1.07 -> 1.06 Wh
time to first token: 186 -> 121 ms
correct: 914 -> 919 of 1,000
Calling it
RLX
clearly this was claude and I.
Next, I'll be exploring rewriting the NCCL library to Rust for RDMA between two GB10s for Tensor Parallelism on models like
GLM 5.3 Flash
Inkling 2 Small
amazing work here
Introducing Tess-4.5-594B-A17B, the best Tess model ever. 🎉
Inkling 2 Small, post trained with the Tess-4.5 recipe, on real engineering sessions to do what Tess does best: intellectually dense reasoning, warm and empathic tone, and hardcore engineering.
On SWE-Bench Pro v2 Hard, Tess-4.5 resolves more of the hardest public software-engineering tasks than the base model (88.2% vs 86.3%), and it does it with ~10% fewer tokens.
Model: huggingface.co/migtissera/Te…
6
835
MoE delivers 3.6x correct answers per Wh than a comparable dense model
Qwen3 30B-A3B (MoE) vs Qwen3 14B (dense)
both score 0.887 on the same 1,000 questions
wall meter, GPU meter agrees
NVFP4 energy vs its own BF16: -70% dense, -65% MoE
1
8
419
energy to correctness to quant analysis on small MoE
Model: Qwen3-30B-A3B
quants: 16 bits -> 4 bits:
energy per correct answer (GPU module) down as much as 58%
correct answers per Wh, vs BF16:
W4A16: 2.4x
GPTQ: 2.3x
NVFP4: 2.2x
AWQ: 2.1x
FP8: 1.8x and 1.3x
accuracy: all seven formats within sampling error of each other
(n=1,000 each, one run)
engine: vllm033
hw: DGX Spark
2
4
367
Daniel Isaac retweeted
Of course that’s your contention.
You’re a first-year inference engineer.
You just read SemiAnalysis’s blog and saw megakernels coming to TPUs and AMD, so naturally you believe that’s the way to get to speed of light.
Next month, when you get your hands on an inference engineering cookbook—Baseten’s, probably—you’ll be singing from the treetops about the 500 nanoseconds of critical-path latency you save on every GEMM pipeline drain.
That’ll last until sometime next year, when you watch a pirated geohotz lecture livestream and become convinced that PDL gives you a strict superset of megakernel capabilities.
“Well, as a matter of fact, I won’t, because PDL introduces—”
“PDL introduces unnecessary stalls because its synchronization mechanism is very coarse.”
Yeah, I read that too. HazyResearch’s blog, “Look Ma, No Bubbles!” Just below Figure 2.
Do you have any thoughts of your own on the subject, or should I just point my agent at the SGLang repo and have it do it for me?
nothing you send is private.
local ai will win
‼️BREAKING: Grok Build has left 35,000+ of its users' apps open to anyone, including apps from SpaceXSI.
The open apps held 3,784 medical exams of workers and a Tesla technician scorecard, plus a Navy contractor's roster and SpaceXSI's own sales plans.
The "secret" words in each app's link sit in its web address, which DNS logs record, security firm Claroc found.
Cursor, which SpaceXSI bought, says apps are private by default. Claroc's tests show otherwise.
SpaceXSI pulled its 24 apps after Ars asked. Most other data was still exposed when Ars published.
3
321
i changed one layer on four Qwen3 models: the lm_head, BF16 -> FP4
energy per correct answer, GPU module:
1.7B: -21%
8B: -19%
14B: -14%
30B-A3B: -18%
accuracy within 1 point
hw: DGX Sparks
i traced one token (about a word) of Qwen3-30B-A3B
15.13 ms with a BF16 lm_head, 12.49 ms with FP4
the lm_head scores every possible next word: 3.35 -> 0.82 ms
at full memory speed the weights read in 7.5 ms. 5.0 ms to go
hw: DGX Spark
2
7
691
i ran 19 models on my DGX Sparks.
energy per correct answer (GPU module) tracks the GB of weights read
dense Qwen3.8-27B: 0.33 Wh
Qwen3-30B-A3B, FP4 head: 0.018 Wh
joules per token vs GB read per token: R^2 0.98, n=19
1
5
389
Claude Code tok optimization
Opus 5.5 as main agent (orchestrator)
-- called as subagents (via orch)
Fable 5.2 (specs, plans, etc)
Sonnet 5.5 (coder)
Haiku 5.5 (revewing logs & raw data)
Options - replace sonnet 5.5 (coding agent)
GLM 5.3 (high/max)
Grok 4.7 (xhigh)
the next 2 years will be about optimizations based on current hw
i put the last layer of Qwen3-30B-A3B in FP4
no training
21.6% more correct answers per kWh
than the RedHat checkpoint
17.8% less energy per correct answer
64 J (0.018 Wh) vs 78 J (0.022 Wh)
accuracy 88.2% -> 89.1%
DGX Spark, GPU power only
i asked 11 models one MATH-500 question
only 5 got it right AND
fastest: Qwen3-30B-A3B FP4 head, 8.5 s, 310 J
my FP4 mod & lowest J per correct output
DGX Spark, GPU power only
🧵 1/2



















