Old DDR3 Xeons are supposed to be e-waste for local AI.
75 tok/s. Qwen 3.8 Flash Next, a 180B model, answering after a 172K-token prompt.
The box: two Xeon E5-2670 v2 with no AVX2, 64 GB of DDR3, PCIe 2.0. And 2x CMP 50HX mining cards, memory-modded from 10 GB to 20 GB each.
On 1 October Strata refused any CPU without AVX2. On 4 October it ran them.
xrip measured this box on his own build of Strata 0.2.0: 65.5 tok/s by default, 75.1 with 19 CPU workers, six runs.
Why it works: 98% of the experts sit on the 2x mining cards. The old CPU computes the other 2%.
Which old Xeons run it, and the October fight that got Strata there: Issue 2 of The Local LLM Sizing Guide, out tomorrow.
vramcalculator.com/sizing-gu…
why don't they just run large models on their own hardware to improve the ecosystem
has anyone had experience running really small models? I was able to get Mellum2.1-12B-A2.5B (Q4_0) running on my orange pi zero 3w but I reached a limit at 6.4-6.8tk/s. The small bonsai models weren't any better at 2.88tk/s.
