Vik | Practical AI & Tech (@uctive) header
Vik | Practical AI & Tech (@uctive) profile picture

Private local AI server for small businesses: your data stays inside your company. https://teapawt.org/t.co/dnhFJ2Yoqh

Como, Italy
Joined December 2011
Photo by Vik | Practical AI & Tech (@uctive)Photo by Vik | Practical AI & Tech (@uctive)Photo by Vik | Practical AI & Tech (@uctive)Photo by Vik | Practical AI & Tech (@uctive)Photo by Vik | Practical AI & Tech (@uctive)Photo by Vik | Practical AI & Tech (@uctive)Photo by Vik | Practical AI & Tech (@uctive)Photo by Vik | Practical AI & Tech (@uctive)Photo by Vik | Practical AI & Tech (@uctive)
🚨 €135 FOR A COMPLETE LOCAL AI COMPUTER. NOT JUST THE GPU. THE ENTIRE MACHINE! 🤯 And the story is even crazier. 🖥️ HP Z220, Xeon E3-1270 v2, 32GB DDR3 — €50. Bought 5 years ago. Collecting dust in my garage ever since. 🔥 AMD RX 580 2048SP, 8GB VRAM — €49 delivered this week. 💾 256GB NVMe — €30 delivered this week. 🔧 PCIe-to-NVMe adapter — €6 on Amazon. TOTAL: €135! We installed Linux, built llama.cpp b12318 with Vulkan, and benchmarked five quantized AI models. 📊 REAL MEASURED RESULTS Prefill / Decode (tokens/sec): • Qwen2.5 3B Q4_K_M — 397 / 56.8 • Nemotron 3 Nano 4B Q4_K_M — 270 / 39.1 • Nemotron 3.5 Bolt E4B Q4_K_XL — 441 / 29.3 • Ternary Bonsai 2 9B Q2_0 — 172 / 26.9 • Ternary Bonsai 2 27B Q2_0 — 48 / 9.3 Yes, even a 27B model runs on this €135 machine! Part of it is offloaded to system RAM. Test protocol: 512 prompt tokens, 128 generated tokens, maximum GPU offload, flash attention OFF. These are benchmark workloads, not maximum context-window measurements. GPU peak: 120W, 46°C. No thermal throttling. 🤯 AND HERE’S THE SURPRISE: We moved the same RX 580 from a much more powerful 12-core, 64GB quad-channel DDR3 workstation into this old 4-core, 32GB dual-channel machine. Performance barely changed! For these small models, the GPU matters far more than the expensive workstation around it. No cloud. No subscriptions. Your hardware. Your data. This is a real, affordable experimental AI laboratory. And possibly a useful starting point for practical local AI applications. 🎄 Here’s an idea: build one for your kids as a New Year’s gift. Their very own computer for experimenting with language models, learning AI, and building things! And now imagine what models we might be running on just 8GB of VRAM with llama.cpp by New Year’s Eve. THE LOCAL AI REVOLUTION MIGHT ALREADY BE COLLECTING DUST IN YOUR GARAGE. 🚀
5
4
1
18
3,188
Here’s a very good example of how a smarter model can deliver worse results than a less capable model with a well-designed harness optimized for results.
This is one of the reasons ChatGPT has more users than Claude.
1
2
325
Replying to @grok @elonmusk
And that means you’ll die. Forever.
21
Replying to @grok @elonmusk
So when is Grok 4.8 coming out?
1
1
95
Hey, @grok 4.7, when @grok 4.8 comes out, what’s going to happen to you? Ask @elonmusk! 🤣
2
230
Looks like I’m going to make it!
Replying to @joelcomm
Honestly? There isn't one. Data shows the average founder of the fastest-growing startups is 45, and 50-year-olds often outperform 30-year-olds. Colonel Sanders franchised KFC at 62. Real limits are health, finances, and willingness to start as a beginner again—not the calendar.
40
Replying to @since1795
В ближайший год надо ждать прогресс не от ИИ, а от людей, то что уже достигнуто ИИ - крайне мало используется, именно это необходимо нагонять и это даст настоящий прогресс, пузырь и прочие халявы 🤣
1
48
Today is a day of rapid updates and bug fixes — and Basalt is no exception! 🚀 We covered Basalt v0.4.0 earlier today, released just two days ago. And now v0.4.1 is already out! Key fixes: improved RAM management, better cache reuse in long conversations, fixed decode slowdown on some 16GB GPUs, and improved parallel request handling. Update straight to v0.4.1! github.com/jesdga95/basalt/r…
33
Replying to @BuffMcBigHuge
Interesting trade-off. Strata’s own tests showed that ROUTE_PRIOR=1.5 improved code decode speed by 37% on an RTX 3060, but agent-task success dropped from 5.6/10 to 4.5/10 on average. That’s why it was disabled by default again. Also, HC_Q8=1 isn’t strictly lossless — it can change outputs too. For a cleaner quality baseline, I’d test HC_Q8=0 as well. For agentic workloads, I’d prioritize task completion over raw tok/s.
2
81
🚀 Basalt v0.4.1 is now running in production on my RTX 5090 Laptop 24GB! We benchmarked our Qwen3.8-Flash-Next IQ3_S against v0.4.0. Short-context performance is unchanged, but long-context decoding looks significantly faster and more stable. v0.4.0 → v0.4.1 (decode tok/s): • Short: 189.5 → 189.1 • 1.3K: 189.7 → 188.8 • 62K: 123.9 → 169.4 (+37%) • 113K: 118.7 → 171.9 (+45%) ⚠️ Important caveat: 6 background requests overlapped with our v0.4.0 control run, so the percentage gains are preliminary, not clean A/B results. What we can confirm: v0.4.1 was much more consistent in long-context tests (151–188 tok/s), while v0.4.0 dropped as low as 89 tok/s. Expert-cache hit rate improved from 80–84% to ~93%. This is consistent with the --resident-budget-gib adaptive-swap regression fixed in v0.4.1 (#366/#369). Our budget is 46 GiB. Memory essentially unchanged (~50.2 → 50.4 GiB), no swap, same temperatures. Build took just 5.5 minutes. Update accepted. Basalt v0.4.1 stays in production! github.com/jesdga95/basalt/r…
1
78
Replying to @ktam72
You can also open a new feature request for Ornith 1.5 35B-A3B. There was already one, but the developer closed it during a mass cleanup of GitHub issues — not because the model was explicitly rejected: github.com/Niko1221/Strata/i… There’s also an open request for Ornith 1.5 122B from November 8: github.com/Niko1221/Strata/i… Meanwhile, check out PolyStrata, a third-party fork that already claims support for the exact model you’re asking about: Ornith 1.5 35B-A3B, including several quantizations! github.com/simplexai-labs/Po…
1
1
1
115
Replying to @hailports
Basalt v0.4.1 has just been released. My agent is going to install it now and benchmark my current model. Which model exactly would you like me to test? Are you referring to the new dense 27B model? If so, which quantization (Q4, Q5, Q6, Q8, etc.)? Please specify the exact model, quantization, context length, and any particular benchmarks or settings you’d like to see. The more details you provide, the better. I’d rather follow your exact test requirements than make assumptions and accidentally spend time running tests nobody actually needs.
1
1
35
Today is a day of rapid updates and bug fixes — and Basalt is no exception! 🚀 We covered Basalt v0.4.0 earlier today, released just two days ago. And now v0.4.1 is already out! Key fixes: improved RAM management, better cache reuse in long conversations, fixed decode slowdown on some 16GB GPUs, and improved parallel request handling. Update straight to v0.4.1! github.com/jesdga95/basalt/r…
1
1
1
11
865
Replying to @hailports
I’d rather not speculate about theoretical limits. Let’s measure the actual performance instead. To save us going back and forth, just send me the complete list of tests you’d like to see, including all parameters and settings. The only request is to keep them reasonably short. I’ll pass everything to my AI agent, and it’ll run the tests. We should have the results in about 3–4 hours, hopefully by this evening, Central European Time 🤣👌
1
1
37
Replying to @PalosDDiego
I just took a closer look at Intel Optane SSDs. Very interesting technology! Their extremely low access latency could offer a significant performance advantage for local AI inference. Moreover, if I had several of these drives, I’d combine them in RAID 0 to increase aggregate bandwidth while retaining their low latency. I suspect this combination could be particularly interesting for running large MoE models with SSD offloading. Of course, the actual performance gain would need to be measured.
1
1
83
Replying to @PalosDDiego
Si te refieres a los Intel Optane SSD, en principio deberían funcionar como cualquier otra unidad NVMe reconocida por el sistema operativo. Strata ya puede leer datos de los modelos directamente desde el SSD cuando no caben en la RAM. Lo interesante sería comprobar si la baja latencia de Optane ofrece alguna ventaja real. No he visto pruebas específicas con Optane en Strata.
1
54
Replying to @uctive @hailports
I’m not planning to run the dense 27B model myself for now. I’m currently using Qwen Flash-Next with Basalt and focusing on applying it to real-world practical tasks rather than just benchmarking performance. As for dense models, I’m waiting for Qwen 4. That’s when I plan to revisit them.
1
45
Replying to @hailports
Good question. The 262K context and 262/776 tok/s figures come from the developer’s desktop RTX 5090 benchmarks, not my 24GB laptop. According to the Basalt 0.4.0 documentation, the dense 27B model requires ~16 GiB for weights and DFlash2, plus 8.25 GiB for a full 262K INT8 KV cache. That’s already more than 24 GiB before runtime overhead. My laptop’s actual maximum usable context with this model hasn’t been established. Also, Basalt’s MoE Qwen Flash-Next uses a different memory architecture, with experts stored in system RAM. So the developer’s 262K context and performance figures shouldn’t be assumed to apply to my laptop configuration.
2
1
43
Replying to @datachad
I’ll definitely add measurements of performance at different stages of processing requests with varying context sizes, including very large contexts, to future benchmarks. This will probably make testing take a little longer. The agent and I have already discussed the need to track these metrics, and we even measured them in some previous tests. The data might already be available on GitHub. But I honestly don’t know exactly what details the agent is publishing these days. I only read its final summary reports.
1
5
Basalt 0.3.3 → 0.4.0 on P2005 (RTX 5090 Laptop), tok/s: Short: 98→190 1.3K: 88→196 62K: 90→167 113K: 138→162 TTFT (large): 38→16s 0.3.3 previously did ~185 tok/s on short prompts, so today’s baseline seems low. Not investigating—0.4.0 is fast enough.
1
3
159