BTC-Cracker (@BTC_Cracker) header
BTC-Cracker (@BTC_Cracker) profile picture

Local Model Dev AI on Strix Halo. Full stack Engineer + AWS. Quantum & Math Cracks $BTC.

Mexico
Joined October 2023
Photo by BTC-Cracker (@BTC_Cracker)Photo by BTC-Cracker (@BTC_Cracker)Photo by BTC-Cracker (@BTC_Cracker)Photo by BTC-Cracker (@BTC_Cracker)Photo by BTC-Cracker (@BTC_Cracker)Photo by BTC-Cracker (@BTC_Cracker)Photo by BTC-Cracker (@BTC_Cracker)Photo by BTC-Cracker (@BTC_Cracker)Photo by BTC-Cracker (@BTC_Cracker)Photo by BTC-Cracker (@BTC_Cracker)
Under 6 watts: Strix Halo's built-in AI unit (the NPU) ran Liquid AI's d1-2B vision model. The catch: 20x slower than the GPU (1.7 s vs 86 ms per answer), so energy per answer came out about even. What would you run on the low-power side? github.com/jethac/d1-rocm #StrixHalo
26
One model copy, two GPU brands. Project Maya 1.1.64 lets a Strix Halo box share GLM-5.3-Flash with an NVIDIA card. The card runs the busiest parts, the AMD chip the rest from shared RAM. Experimental, Linux. Would you add a GPU to your Halo? #StrixHalo github.com/mw00/project-maya…
1
87
AMD had AI agents port 51 trending open models to Strix Halo in 4 days. On 32K-token prompts, its experimental Loom engine read them a median 1.5x faster than llama.cpp ROCm (1.9x vs Vulkan). Still a preview. Which model would you try first? loom-fun.github.io/loom-skil… #StrixHalo
64
Yo this nail shop needs to chill
51
76 tokens/sec on a 30B model — one Ryzen AI Max+ 395. Lablup's MLxcel 0.8 just added AMD support. 44 models tested on Strix Halo; all ran. A denser 31B sits near the hardware limit at ~10 tok/s. Try it over llama.cpp? #StrixHalo backend.ai/blog/releases/202…
2
2
156
Plug Nvidia's new RTX Spark laptop into the wall and Linux runs 5x slower: 13.9 tokens/sec on GPT-OSS-120B vs 65.4 on Windows. On battery, Linux does ~61. One tester blames a Windows-only power controller. WSL is fine. Still buy one for Linux? #LocalAI mindstudio.ai/blog/linux-nvi…
1
75
4 boxes, 1 trillion parameters, one 10Gb switch. AMD posted an official guide to link four 128GB Ryzen AI Halo boxes with free llama.cpp software and run Kimi K2.6 at home. No speed listed yet. About $16K at Micro Center. Would you build it? #StrixHalo developer.amd.com/playbooks/…
1
64
SpaceXSI (the Grok API) just told teams that text from their main models will carry an invisible watermark by default starting Nov 7. It covers grok-4.7, 4.6, 4.5, 4.3, and the 4.20 reasoning / non-reasoning / multi-agent models. No request change needed — the mark is baked into which tokens the model picks, not pasted as hidden text. Docs say it's for EU AI Act compliance. Team admins can opt out per model in the Console if a legal exception applies. Local angle: anything you generate through that API can later be scored as "model-written." Stuff you run on your own box does not get that mark. TLDR: cloud Grok text starts carrying a silent fingerprint next week unless you opt out. Local still means unmarked. docs.x.ai/developers/advance…
55
Feeding a 256K-token prompt to Qwen3.6-35B on a Strix Halo took about 2 hours in stock vLLM. With a new plugin, it takes about 4 minutes. libstrix is a free (MIT) set of custom GPU code for the Ryzen AI Max+ 395 that plugs into regular vLLM without changing vLLM itself. The author's numbers, one request at a time: • Gemma 4 26B at 64K context: 31x faster prompt reading • Qwen3.6-35B at 64K: 19x faster prompt reading, 2.35x faster replies • GSM8K accuracy checks: no real change The catch: you have to build it against the exact ROCm and PyTorch your vLLM uses (or use their Docker image), and it's brand new, so these are the author's own benchmarks. TLDR: Before an AI can answer, it has to read your whole prompt. On long documents that reading step was painfully slow on this AMD mini PC. This add-on makes it dramatically faster without making the answers worse. Anyone on a Strix Halo box tried it yet? Do your numbers match? github.com/asf0/libstrix #StrixHalo
2
4
215
Mistral's new trillion-parameter model now comes in a 233GB version. That's about what two 128GB Strix Halo mini PCs hold together. Mistral released the open weights for Mistral Large 4 ("Le Chonk") this morning, and Unsloth posted shrunk-down GGUF files right after: • 1-bit: 233GB, picks the same next word as the full model ~91% of the time • bigger 1-bit: 260GB, ~92% • 2-bit: 302 to 378GB, ~94 to 96% Unsloth points people at a 256GB Mac Studio or two DGX Sparks. Nobody has posted Strix Halo numbers yet. On paper, a 128GB + 192GB pair linked over USB4 (320GB) clears the 260GB file, and two 192GB boxes (384GB) reach 2-bit. Two 128GB boxes are right at the edge. Unsloth says to use its own llama.cpp build for now. TLDR: Mistral gave away a giant AI model. Squeezed down, it might run on two small AMD desktops tied together with a cable, but nobody has proven that yet. Strix Halo owners: is a 1-bit trillion-parameter model worth a second box, or would you rather run a smaller model at full quality on one? huggingface.co/unsloth/Mistr… #StrixHalo
2
121
Strix Halo's idle NPU can make prompt reading 7-12% faster, but so far only in the benchmark. Gufo PR #568 lets the chip's built-in AI unit (the NPU) help the graphics side chew through long prompts. Testers measured +7% to +12% in gufo bench on a Ryzen AI Max+ 395 running Qwen3.8 Flash-Next. The catch, from an Oct 22 test: in gufo serve, the mode you actually chat through, the NPU never kicks in. Its kernel only takes prompt chunks of exactly 4,096 tokens, and the server cuts prompts into 2,048-token pieces. Another tester saw answers drift a little with it on (same top word about 94% of the time), though quiz scores held. TLDR: Strix Halo has a spare AI chip that usually sits idle. Someone got it helping, and it speeds things up a bit in a lab test, but not yet in everyday use. It's still an open proposal, not shipped. Is the NPU worth chasing, or should devs keep squeezing the GPU? github.com/gufo-org/gufo/pul… #StrixHalo
1
2
164
On a Strix Halo mini PC, one local AI model handles easy questions for 3.6 cents per 1,000 correct answers. Give it hard questions and each right answer costs 37x more. CIQ's Damen Knight ran 27 local models on his Strix Halo box with a wall power meter, priced at his real Southern California rate (58 cents/kWh on-peak), and only counted answers that were actually right. - Easy tasks: qwen3-30b-a3b was cheapest at $0.036 per 1,000 right answers. - Hard expert-written science questions: the same model got 63% right and cost $1.35 per 1,000. - qwen3-coder-next tied for the best hard score (79%) and still cost about 5 cents per 1,000 on easy work. - A big 125B model (devstral-2) also aced the easy stuff, at 36x the cost. - Leaving Gemma 4's built-in "reasoning" mode on cost up to 48 accuracy points and 6-8x the money on easy work. TLDR: Running AI on your own computer isn't free, it burns electricity. The model that's cheapest for simple chores can get expensive when the questions get hard, so match the model to the job. What does a right answer cost on your box, and what's your daily driver? ciq.com/blog/strix-halo-cost… #StrixHalo
90
Same AMD Ryzen AI Halo, same Qwen3.8 27B, same agent benchmark: one lab finished in 34.5 minutes, the other took about 42. AgentPerf-Local replays 8 real AI-agent jobs (168 turns) and times the whole run. Artificial Analysis got 34.5 min on Strix Halo. Tom's Hardware got ~42 on the same AMD box while testing the new RTX Spark laptop. Both used ROCm with MTP. Their Nvidia numbers matched almost exactly (18.7 vs 18.9). Same chip, so a ~20% swing has to come from software version or settings. TLDR: This test times a helper finishing 8 full chores, not just how fast it talks. On the Nvidia laptop, two labs got the same time. On the AMD box, setup alone moved it by 7-8 minutes. AA's exact Strix command: artificialanalysis.ai/hardwa… Run it yourself: github.com/ArtificialAnalysi… Strix owners: what time does your box get, and on ROCm or Vulkan? #StrixHalo
1
1
168
At 128k context, a new Strix Halo software branch chews through long prompts up to 2.7× faster than plain AMD ROCm. It's called rdna35-perf (on asf0/strix-llama.cpp). On a 128GB Radeon 8060S with Qwen3.8 27B, reading a huge prompt jumped from 114 tokens/sec on stock ROCm to 312 — while answer speed stayed tied with Vulkan. Same output quality. Branch: github.com/asf0/strix-llama.… Anyone already on it, or still on Vulkan for daily use? #StrixHalo
72
A Strix Halo benchmark said 1,343 tok/s. The real speed was 698. The extra "speed" came from the model quietly working on garbage. ROCmFPX, an open llama.cpp fork for AMD GPUs, merged the fix this morning (PR #38 by @no_stp_on_snek). What was going on: • Qwen3.8 Flash Next on a Radeon 8060S, with unified memory and HIP graphs turned on • Only when the batch size (-ub) was set to exactly 2048 or 4096 • One copy step silently never ran, so the answers went bad while the benchmark looked great • After the fix, the model picks the same next word as a known-good run 97.5% of the time, up from about 49% • The default -ub 512 was never affected, and its speed is unchanged (557 vs 556 tok/s) Can't update yet? The PR says setting GGML_CUDA_DISABLE_GRAPHS=1 also avoids it. TLDR: a speed test can look amazing while the AI is actually reading junk. This bug made one setting look about twice as fast as it really was. It's fixed now, and it's a good reminder to check the answers, not just the speed. Has a "too good to be true" number ever fooled you on your own box? github.com/ROCmFPX/ROCmFPX/p… #StrixHalo
1
2
1
256
There absolutely will be a “catastrophic” AI event and it will be used to usher in many new laws, regulations, and possibly even a new digital currency. But, guaranteed you will lose more of your freedoms.
2
75
llama.cpp just merged a GPU cache for MoE "experts" (PR #29887, today). If your model doesn't fit in VRAM, the expert chunks you use most now stay on the GPU, and only the misses get copied over from system RAM. You turn it on with --moe-cache-mib N. Author's numbers on Qwen3.8-Flash-Next Q4 (94GB): • RTX 4090: 25 → up to 41 tok/s • RTX 5090: 31 → up to 68 tok/s It's not free speed everywhere, though. One tester saw GLM-5.3-Flash get slower (about 18 → 13-14 tok/s) because all the misses maxed out PCIe. An R9700 tester only broke even until the hit rate got close to 94%. TLDR: big AI models are split into lots of small specialist parts. This keeps the popular parts on the fast graphics card instead of fetching them from slower memory every time. It helps a lot on some models and hurts on others, so test before you leave it on. On Strix Halo everything already lives in one memory pool, so this one is really for desktop GPU + RAM rigs. Anyone with a 24-32GB card tried it yet? What hit rate are you seeing? github.com/ggml-org/llama.cp… #LocalLLM
126
GLM-5.3-Flash just got built-in MTP in mainline llama.cpp (PR #29928, merged this morning, build b11474). No more Unsloth fork needed. Turn it on with --spec-type draft-mtp. What the PR thread reports: • About 2x faster decode at short context (97% of guesses accepted), +52% at 32K. That tester is on an RTX PRO 6000, not Strix. • Acceptance rates roughly match Unsloth's fork on the same 1-bit file • ggml-org ships separate MTP add-on files: 4.9GB Q4_0 or 9.3GB Q8_0 • Known bug: image input crashes with MTP on. Fix is in progress. TLDR: MTP lets the model guess the next couple of words and check them all in one step, so it writes faster. It still checks every guess, so quality shouldn't drop. This big open model now gets that speedup in the standard app, not just a side version. 128GB Strix Halo owners running the 1-bit quant: does mainline beat the fork for you? github.com/ggml-org/llama.cp… #StrixHalo
89