Qwen3.5-4B running CPU-ONLY at up to ~11+ tps on a Ryzen 5 laptop. No GPU. Running models in RAM and CPU is something the Qwen4 team is already thinking about. 💡 I built this standalone .exe because businesses are already deploying local models to save money and ensure privacy. (thumb drive friendly - just drag drop and doubleclick) My llama.cpp recipe 👉 🧠 Qwen3.5-4B Q4_K_M GGUF ⚙️ Ryzen 5 7540U — 6C/12T 🧵 --threads 9 🧵 --threads-batch 12 ⚡ --prio 2 🔄 --poll 50 📦 --batch-size 2048 📦 --ubatch-size 512 🚀 --flash-attn on 🧠 KV cache: q4_0 / q4_0 🔧 --repack 💾 --mmap 👤 --parallel 1 🚫 --device none 🚫 --gpu-layers 0 🚫 KV/op GPU offload 🚫 MTP OFF Interesting result 👉 MTP=3 was slower (~10 tps in benchmarks). Plain decode + 9 threads + Q4 KV hit ~11.4 tok/s. That's about 10% faster just from tuning llama.cpp — on a basic laptop CPU.

This small Local AI model could replace an entire document-processing pipeline. ATH-MaaS released OvisOCR3. It’s only a 0.8B-class multimodal model. But one local model can now do all these things 📄 document → Markdown 🧾 document → structured JSON 🔎 OCR + bounding boxes 🧮 formulas → LaTeX 📊 charts → structured data 🔑 key-value extraction 🖼️ figure captioning 🔀 flowcharts → Mermaid 📚 cross-page table reconstruction That last one is interesting. Give it consecutive document pages and OvisOCR3 can determine whether a table continues onto the next page and reconstruct the complete table. Team-reported results OCRBench → 88.8 OCRBench KIE → 90.5 Cross-page table Relation F1 OvisOCR3 → 97.07 Qwen3.8-Max → 94.47 And it’s 🧠 built on Qwen3.5-0.8B 🔓 Apache 2.0 🏠 downloadable/local ⚙️ Transformers + vLLM + SGLang 🔌 OpenAI-compatible local server through vLLM No official GGUF that I can find yet. Link in ALT

The company behind PUBG just trained its own 29B Local AI model from scratch. And the architecture is interesting. KRAFTON’s new Raon-1 mini is … 🧠 28.8B total parameters ⚡ only 4.3B active/token 💾 official Q4_K_M → 19.7GB 📚 262K context 🔓 Apache 2.0 🤖 reasoning + agent/tool use 18 of its 25 layers use fixed-state linear attention. Only 7 layers maintain a growing KV cache. KRAFTON says those layers use only ~8 KiB of BF16 KV per token. So even at the full 262K context, that’s roughly: 👉 ~2GB of growing KV cache before other runtime/state overhead. And in KRAFTON’s reported comparisons, this new 30B-A4B model beats Qwen3.6-35B-A3B on 11 of 18 listed benchmarks. Lots of new models coming for Qwen3.6-35B recently. ⚠️ Preview release. KRAFTON’s benchmark results. Upstream llama.cpp support is still pending and KRAFTON provides its own branch today. Link in ALT

Nvidia lets you build a 1TB Local AI cluster out of DGX Sparks. & Nvidia has a playbook for AI clustering them. 8 × DGX Spark 🧠 128GB each ↓ 🔥 1TB unified memory across the cluster Nvidia now officially supports manually configured clusters of up to 8 x DGX Sparks using their built-in ConnectX-7 networking. And Nvidia says that 1TB is enough to run frontier-class models like GLM-5.3 locally. They specifically call out 🧠 larger models 📚 longer contexts 🤖 multiple concurrent agents 👨‍💻 multiple developers sharing one local backend ⚙️ distributed development They list models such as → GLM-5.3 Flash → DeepSeek V4 Flash → Qwen3.8 Flash Next And later this month Nvidia Sync Model Launcher is supposed to make it easier. Link in ALT

AMD and Intel agreed on something pretty unusual. A local PyTorch AI app shouldn't have to care whether the CPU underneath it is AMD or Intel. AMD is folding its ZenDNN AI optimizations into oneDNN, the library originally developed by Intel. AMD + Intel have now aligned on oneDNN as the default open-source x86 acceleration path in PyTorch. ZenDNN has been working on refining for ... 🧠 LLM inference ⚡ INT4 / INT8 / BF16 / FP16 🧩 fused MoE operations 🚀 vLLM CPU acceleration 🔧 optimized attention/matmul Now that work is being brought into the common oneDNN ecosystem AMD + Intel are also jointly developing ACE, new x86 AI matrix instructions. The plan is for future AMD/Intel AI instructions → oneDNN → PyTorch So local PyTorch software gets one x86 target instead of separate vendor paths. ⚠️ This does NOT make llama.cpp/GGUF faster, and AMD/Intel haven't published new performance numbers from the merger. 🎯But for CPU Local AI, the two companies that spent decades fighting over x86 are now cooperating on the AI software layer. Link in ALT

Here we go! Purpose built AI generated LLMs! Then AI helped optimize the code that runs it. Recursive released Flint, a new coding/agent model designed specifically for fast local inference. Beats Qwen3.5/3.6/3.8 in some coding benchmarks! Stats … 🧠 39.8B total parameters ⚡ only ~2.5B active/token 📚 262K documented context 🤖 reasoning + tool use 🔓 Apache 2.0 And there are official GGUFs: 🟢 NVIDIA NVFP4 → 22.9GB 🍎 Apple INT4 → 26.0GB Recursive recommends 48GB-class memory for these local builds, so think: 🍎 48GB+ Apple Silicon 🟢 48GB+ NVIDIA ⚡ 64GB RTX Spark Recursive reports its optimized runtime at 🍎 M4 Pro → >125 TPs 🟢 64GB RTX Spark → >150 TPs And WIRED’s Will Knight actually ran Flint locally on a 48GB Mac and reported it was fast and successfully handled the coding jobs he tried. Side Note: Recursive built Flint using its Eureka Machine, an automated AI research system that helped 🧩 design the architecture 🧪 develop training recipes ⚙️ optimize infrastructure 💻 write hardware-specific inference kernels Nvidia says those specialized kernels improved Flint’s RTX Spark generation performance by 55% wo changing the model weights. Recursive-reported coding results SWE-bench Verified → 74.2 SWE-bench Pro v2 → 89.7 Terminal-Bench 2.1 → 54.9 LiveCodeBench v6 → 80.3 So this isn’t merely AI writing more AI code. Link in ALT

A single 24GB GPU can now run a 35B scientific research agent that spawns sub-agents and checks its own evidence. Sage-A1.5 is 35B / ~3B active, and its official Q4_K_M is just 21.7GB. Here are the new Sage-A1.5 model stats 🧠 35B MoE / ~3B active 💾 official Q4_K_M → 21.7GB 🎮 fits a single 24GB GPU at moderate context 📚 256K native context 👁️ multimodal 🔓 Apache 2.0 And it's specifically trained as a long-horizon research agent. So it is designed to 🔎 use tools/search/code 🤖 spawn parallel sub-agents ✅ inspect its own evidence 🔄 change direction when research stalls No separate critic model required for its verification mode. These are the full-precision benchmark results PinchBench Sage-A1.5 → 90.0 Kimi K3 → 88.2 GLM-5.3 → 86.6 GPT-5.6 Sol → 87.3 FrontierScience-Olympiad Sage → 81.0 Kimi → 69.0 GLM → 67.0 GPT → 73.0 SWE-Pro Sage → 61.2 Kimi → 63.3 GLM → 64.6 GPT → 64.6 ⚠️ Those benchmark scores are from the BF16 checkpoint, not the Q4. But the same model now has an official 21.7GB GGUF with llama.cpp / Ollama / LM Studio support. Link in ALT

This might be my new favorite all-purpose local AI model. Fits nicely into 16GB of VRAM for an RTX 5060/5070TI. The new SparkX2.6-4B beats Qwen3.5-9B in 80% of shared benchmarks and 97% of Gemma4-12Bs. Stats 💾 official Q4_K_M → 3.54GB 🧠 only 4B parameters 📚 native 1M context 💭 none / low / high thinking budgets 🤖 trained for local agents 🔓 Apache 2.0 It already runs with: ⚙️ llama.cpp 🦙 Ollama 🖥️ LM Studio 🍎 MLX 🚀 vLLM / SGLang In the team's reported benchmark suite Averages: Spark-X2.6-4B → 63.3 Spark-X2.5-4B → 52.1 Qwen3.5-9B → 49.1 So they're claiming a 4B model beats their tested 9B comparison overall. For normal local-agent contexts, though, a ~3.5GB reasoning/agent model with this runtime support is very interesting. Link in ALT

Underdog released Bark 3.0, and it's only a 163MB download. Runs locally w/Pytorch+CUDA or MLX On their Open ASR Leaderboard evaluation, Bark wins 🐶 Bark 3.0 → 4.94% WER → 163MB 🟢 NVIDIA Parakeet → 4.94% → 2.51GB 🌀 Whisper large-v3 → 6.04% → 9.26GB So Bark matches Parakeet's average accuracy while being 👉 ~15× smaller And beats Whisper while being 👉 ~57× smaller On recorded meetings check out these stats Bark → 7.82% WER Parakeet → 9.42% Whisper large-v3 → 13.63% And this isn't CUDA-server-only. 🏠 fully local 📱 phone-class 🍎 MLX + Core ML 🧠 Apple GPU / ANE 🪟 CPU + CUDA paths 🔓 CC-BY-4.0 I've checked and no GGUF. It uses its own compressed Bark format plus MLX/Core ML/Transformers runtimes. Bark 3.0 Plus is only 180MB and improves average WER to 4.76%. ⚠️ English only Link in ALT

DeepSeek V4.1 Flash 1017 is now the #1 open model on Agent Arena leaderboard. Hosted 🏆 #1 open Agent Arena 💰 ~$0.19 median/task 🔓 MIT weights Localmaxxers are already running the same 1017 checkpoint on 2× RTX 3090s + 128GB DDR4. And getting ⚡ ~30 tps decode 🚀 301–470 tps prefill This is a monster model w/552B MoE backbone 💾 + ~196B Engram memory 📚 1M native context The local build squeezes the experts to ~2.03 bpw and uses exact routing, with the big Engram tables on storage. ⚠️ Those 2x3090s use an aggressive quant. It does not mean the local 2-bit version retains the exact Arena score. But that's still kind of nuts. Link to the 2x 3090's test in the ALT

The RTX Spark laptop configuration I thought would signal real demand is the expensive one SKU. Microsoft Surface Laptop Ultra w/💾 128GB unified memory, full RTX Spark N1X, $5,899.99. Microsoft currently lists it 👉 Out of Stock ASUS also said demand for its RTX Spark systems “greatly exceeded” expectations and its initial planned volume had already been fully preordered by channel partners. MSI said its first N1X batch was nearly sold out and was asking Nvidia for more supply. People may be buying RTX Spark for the thing normal Windows laptops can't do. 1) 128GB unified memory 2) CUDA

Tencent has released its own official GGUFs of Hy-Memory-30B-A3B. And they didn't just throw up some quants. They benchmarked them too 💾 Q4_K_M → 18.24GB → 56.49% 💾 Q6_K → 24.71GB → 58.78% 💾 Q8_0 → 31.99GB → 57.63% All tested with llama.cpp on the same LoCoMo memory benchmark setup. 👉 Q6 actually scored highest. And the Q4 means this specialized long-term memory writer is now officially an ~18GB local model. Tencent even documents running it directly with llama.cpp: llama serve -hf tencent/Hy-Memory-30B-A3B-GGUF This is the sort of Local AI architecture I want to experiment with. Link in ALT

I think Sipeed may be skipping the MaixStation DGX-Spark-Killer configuration we would want most. They're planning these memory configs 💾 32GB 💾 128GB 💾 512GB But no standard 64GB model. Sipeed's reasoning is 32GB → dense 27B models 128GB → 125B-class MoE models 64GB → “would run neither better” A ~$2,000 64GB box with 273 GB/s bandwidth would be more interesting to me than a ~$2,000 32GB box. 64GB opens up a very different Local-AI options including, larger quants, larger context/KV cache, and more room for agents. Especially when Nvidia's new 64GB DGX Spark starts at $4,999. Sipeed says enterprise customers can custom-order 64GB. Am I wrong? Would you buy 32GB, 64GB or jump straight to 128GB?

Okay, now people are benchmarking Nvidia's 24GB Bolt on consumer hardware. Someone on Reddit put the official Nemotron 3.5 Bolt 74B-A9B IQ2 on an RTX 5090 💪 with llama.cpp. Without speculative decoding ⚡ 166 tps Add NVIDIA's LiLiCorr DFlash drafter 🚀 343 tps median That's about a 2.07× jump. And prompt processing 🔥 ~5,200 tps VRAM is interesting too 🧠 Bolt only → 21.0 GiB 👁️ + vision projector → 22.7 GiB 🚀 + projector + LiLiCorr → 25.0 GiB So the regular multimodal model really can squeeze onto a 24GB GPU. Link in ALT.

A new Mac Inference engine brings a 48GB M4 Pro MacBook running Qwen3.6-35B to 154 tps. The same benchmark with llama.cpp + MTP, 69 tps. Someone built another Local AI inference engine for Mac. It's called Kapitsa, and instead of trying to support every model, it specializes using🍎 Apple Silicon for either 🧠 Qwen 🟢 NVIDIA Nemotron On a 2024 M4 Pro MacBook Pro with 48GB RAM, running Qwen3.6-35B-A3B at 4-bit: ⚡ Kapitsa + DFlash2 → 154 tps ⚡ oMLX + DFlash2 → 127 tps ⚡ Ollama + MTP → 89 tps ⚡ LM Studio → 86 tps ⚡ MTPLX → 80 tps 🐌 llama.cpp + MTP → 69 tps That's about 2.2× llama.cpp's decode speed in their test. The strategy being, instead of building one generic runtime, Kapitsa hand-optimizes Metal kernels + execution paths around specific architectures, then stacks tweaks like 🚀 DFlash2 speculative decoding 🧠 model-aware caching ⚙️ optimized MoE execution 🔀 concurrent request batching And on an M5 Max 128GB, they report 🚀 Qwen3.6-35B → 281 tps 👁️ Nemotron 3.5 Super-VL 120B → 85 tps ⚠️ These are still developer benchmarks, not independent results. Links in ALT.

A new open LLM can remember an old conversation without putting that conversation back into its context. And at 128K history, its authors claim the persistent memory is ~66× smaller than keeping the full KV cache. MemT-1.0 Nano dropped today. Stats 🧠 36B total / 3B active ⚙️ built from Qwen3.6-35B-A3B 💾 10M dynamic memory state 🧠 451M trainable memory parameters 🔓 MIT-licensed weights But it does things a little differently. Normally, a long-running LLM needs old conversation + new message. So as history grows, you keep dragging more context/KV around, or retrieving old text and stuffing it back into the prompt. MemT puts memory inside the model itself. Its memory blocks run alongside the normal LLM attention. As you interact with it, the model learns to: 📝 write information 🗑️ forget information 🔄 update information 🧠 read it back later So you can: 💬 have a conversation 💾 save its latent memory state 🗑️ discard the plaintext conversation 🔄 start a new session with zero previous messages 🧠 restore the memory ➡️ and ask about what happened before Their demo does this. At 128K history, the authors report for MemT-Nano 📦 persistent memory → ~66× smaller than full-context KV cache ⚡ end-to-end inference → ~1.86× faster And the memory state stays a fixed size as history grows. MemT isn't a little memory model sitting next to Qwen. It takes a normal generative LLM and makes persistent memory part of the LLM's computation itself. They also released MemT-Flash: 🧠 288B total / 17B active 💾 336M dynamic memory state 🧠 3.61B trainable memory parameters 🎯 This could be especially interesting for long-running local agents. ⚠️ The 66× / 1.86× results are author-reported, and strong text-retrieval systems still beat MemT on some tasks where exact historical details matter. I can't find consumer GGUF benchmarks yet. Link in ALT.

This new-ish 35B multimodal model went from 3.38 → 20.51 tps on an iPhone. MiniCPM-V 4.7 35B-A3B was updated to decode faster on the iPhone by improving how it moves the model between storage and memory. Constraints 🧠 35B total / ~3B active 📱 iPhone 18 Pro Max — 12GB RAM 👁️ image + multi-image + video understanding Most of those 35B weights don't sit in RAM. The experts can stay in the phone's flash storage and get loaded when they're actually needed. Normally that's slow 🐌 SSD expert loading → 3.38 t/s So OpenBMB built Lookahead Expert Scheduling (LES) It predicts which experts the next layer will probably need and starts loading them while the current layer is still computing. Then they stacked the optimizations: 💾 baseline SSD loading → 3.38 tps 👀 expert lookahead → 6.57 ⚙️ optimized scheduling/loading → 11.34 🗜️ mixed-precision quantization → 17.16 🚀 + DSpark speculative decoding → 20.51 tps That's a 6.07× speedup over the same-phone unoptimized baseline. ⚠️ And these are still OpenBMB's own measurements. Link in ALT

A new 122B model dropped from @ornith_, and Localmaxxers already squeezed it into a 47.6GB GGUF for 64GB Macs. And Local AI users already have it running through llama.cpp. Ornith released Ornith-1.5-122B today. It's built on the Qwen3.5-122B-A10B architecture/lineage 🧠 ~122B total / ~10B active 📚 262K native context 🔓 MIT license 📦 GGUF + MLX already available ⚙️ llama.cpp + Ollama support And Ornith reports strong results against Qwen3.8-Flash-Next in the same harness 🧑‍💻 SWE-bench Verified Ornith → 84.5 Qwen3.8-Flash-Next → 82.8 💻 SWE-bench Pro Ornith → 76.9 Qwen → 43.9 🌐 BrowseComp Ornith → 78.5 Qwen → 75.3 AtomicChat has already released a GGUF ladder for it 💾 Q8 → 132.6GB 💾 71.2GB → 93.5% top-1 agreement with BF16 💾 62.2GB → 91.0% 💾 54.3GB → 88.8% 🍎 47.6GB → 86.1% And they specifically recommend that 47.6GB build for 64GB Macs. Support by llama.cpp already exists. ⚠️ Ornith also says the model can serve on a single 80GB GPU, but the exact checkpoint/precision behind that claim needs clarification. Link in ALT.

DeepSeek open-sourced the inference engine behind its own production AI service. DeepInfer is the framework DeepSeek uses to serve V4.1 Flash and they're giving away production tricks such as ⚡ DSpark speculative decoding 🧠 confidence-scheduled verification 📚 SWA bounded replay ⚙️ asynchronous scheduling 🧩 MoE expert parallelism + load balancing 🎯 structured JSON / regex / grammar output 🔬 deterministic + batch-invariant inference Plus DeepSeek's own stack of FlashMLA + DeepGEMM + DeepEP + DeepSelect + custom TileLang kernels DeepSeek says that future model inference code + new optimizations will land on this stack too. This is basically a look inside the code DeepSeek uses to make its enormous models practical to host. So now other inference harnesses can study the production implementation. ⚠️ Right now DeepInfer supports only ❌ no consumer NVIDIA GPUs ✅ NVIDIA SM100 ✅ Huawei Ascend 950 So this isn't Local AI yet. GH link in ALT

This is a little disappointing for MiniMax M3.1 Flash. The model behind Space Bunny landed at #26 on DesignArena's Overall Frontend leaderboard. MiniMax M3.1 Flash → 1290 Compared with some of the other big Chinese models: 🌙 Kimi K3 → 1360 🔴 DeepSeek V4.1 Flash → 1319 🟠 MiMo-V2.6-Pro → 1315 That's especially interesting because MiniMax's launch benchmarks look quite strong on coding and agentic work. But #26 is not the debut I expected after the Space Bunny hype. 👀

16GB of VRAM seems to be the last affordable consumer GPU tier. As such, people are getting the most out of them. Case in point, a 120B multimodal model is running on two RTX 5060 Ti 16GB cards. Someone got Nvidia's new Nemotron 3.5 Super VL 120B-A12B running locally with these stats 🧠 120B total / 12B active 👁️ text + image + video 📚 up to 1M context 🎮 2× RTX 5060 Ti 16GB → 32GB VRAM total 💾 96GB DDR5-6000 system RAM 👈 🎯 ⚙️ Vulkan + layer splitting ❌ no MTP Using Unsloth's NVFP4 build, they reported ⚡ generation → ~8–10 tps 🚀 prompt processing → ~150–170 tps So no, the whole model isn't sitting inside 32GB of VRAM. The machine is combining consumer GPU VRAM + ordinary system RAM to serve a 120B multimodal MoE. Link in ALT