Ivan Rocha retweeted
INCREDIBLE
The BIBLE for understanding LLMs from first principles is now available online to read for free
Covers the model mechanics
- Tokens / tokenizers
- Transformers
- Attention
- KV cache
- Prefill vs decode
- Decoding controls
- Model packages
- Chat templates
- Long context
- RAG
- Agents / tools
- Fine-tuning
- Multimodal models
Then connects that to running models locally
- What "local" really means
- Open-weight vs opensource
- Quantization
- VRAM math
- Hardware tiers
- File formats / load safety
- Runtimes / serving modes
- Model selection
- Privacy
- Failure modes
- Benchmarks
- Practical setup paths
You should read this, and if you cannot now then you most definitely wanna bookmark it for later
Opensource AI FTW
Ivan Rocha retweeted
NVIDIA just released a speculative decoding draft on Hugging Face
It accelerates Qwen3.8-27B with 4B parameters, boosting token acceptance by 46% over the built-in head.
huggingface.co/nvidia/Qwen3.…
Zero-trust sandboxing for millions of agents on Kubernetes
github.com/agent-substrate/s…
1
43
if you have a dgx spark, give qwen3.8 27b at 6.0 bpw a try.
it's the closest thing to the full bf16 model that still fits on one spark. the one to pick when quality matters more than speed.
try it and tell me how it runs on your box. link in the first comment.
Ivan Rocha retweeted
A few more improvements to our LLM inference engine on AMD Strix Halo!
Qwen3.8-Flash-Next now reaches 1,885 tok/s in exact prefill at 8K on a single node, or 2,008 tok/s with our FAST profile.
DeepSeek V4.1 Flash is moving forward too. On two Strix Halo nodes over USB4, we're getting 1,165 tok/s exact and up to 1,425 tok/s FAST with longer prompts.
We've also been working on reducing GPU cycles per token, improving scheduling and making multi-request serving more efficient.
Some changes are already in main, while others are still being tested.
It's a lot of small improvements rather than one big breakthrough, but they keep adding up.
Ivan Rocha retweeted
DGX Spark, Qwen3.8-27B: median first token for 16 agents falls from 142 s to 25 s, answers token-identical
1 Spark, 128 GB: 16 agents, one chat of about 17,000 tokens each, all follow-ups at once
TensorFold 1.2.5
First token, median: 142.3 s
Speed of all replies: 27.4 tok/s
Chats found in cache: 0 of 16
TensorFold 1.2.7
First token, median: 25.4 s
Speed of all replies: 99.0 tok/s
Chats found in cache: 16 of 16
Ash's numbers on a GB10, from the 1.2.6 release notes: median first token 105 s to 25.8 s, speed 33.6 to 100.8 tok/s
Setup: official TensorFold 1.2.7 package, Qwen3.8-27B MLX 4-bit with DFlash2, default flags
Method: median over the 16 agents, 1.2.7 median of 3 runs (27.6 / 23.3 / 25.4 s), 1.2.5 one run
Method: temperature 0, thinking off, replies up to 512 tokens (our choice)
Video: replay of one recorded run at 12x with real timestamps, 1.2.7 median there 28.5 s
Ash's test tool is not published, so we rebuilt the load from his PR description. Our 142.3 s for 1.2.5 is our own value, not his 105 s.
TensorFold and the 1.2.6 cache work by Ash @ashxhart, DFlash2 draft by Inco AI @inco_ai, base model by Qwen @Alibaba_Qwen.
Ivan Rocha retweeted
Introducing Edge0-145B and Edge0-120B, @Edge0_AI’s first device-native 100B-class foundation models with open weights.
Cloud-level intelligence on your laptop. Free, offline, and fast enough for real vibe coding.
- Smart: beats Claude Opus 4.6 on Terminal-Bench 2.1 (86.5% vs. 78.2%)
- Light: needs just 29G memory, 1/3 of a standard setup (100G+)
- Fast: decodes 75+ toks/s, prefills 1050+ toks/s on RTX Spark and Mac.
- Steady: handles 128K–256K tokens of context without slowing down, even on long tasks
Frontier-class agentic coding. No cloud API. No token bill. Your code and data never leave your machine.
Open weights are live on @huggingface. Run it locally or plug it into your favorite coding agent.
Ivan Rocha retweeted
As engineers, we usually want exact answers from our data. However, sometimes it's worth giving up a little accuracy to save a lot of memory.
That's where probabilistic data structures come in:
— Bloom filter checks if an item might be in a set.
— Count-min sketch estimates how many times each item has occurred.
— HyperLogLog estimates the number of distinct items.
— T-digest estimates quantiles of a stream of numbers.
— Top-k sketch identifies the most frequent items in a stream.
I implemented these data structures in Go:
antonz.org/probabilistic-dat…
Ivan Rocha retweeted
llama.cpp v1.0.0 is now available
huggingface.co/blog/ggml-org…
GLM-5.3-Flash on a single DGX Spark 3.5 bpw, not 2-bit.
TideLM + @MiaAI_lab's EXL3 3.5 bpw achieves 26 tok/s on a US history prompt
Pagoda voxel garden demo and DSH video in reply
github.com/raspy135/tideLM
Ivan Rocha retweeted
New Pi extension for improving your prompts: pi-better.
Type /better in front of your prompt and a forked subagent reads your code in the background, then rewrites the prompt with details you didn't think to ask for. Your agent can use it too when prompting its own subagents via pi-subagents.
Built on @felixrieseberg's prompt from felixrieseberg.com/how-to-be… (prob my fav prompt of the year)
pi install npm:pi-better
github.com/nicobailon/pi-bet…
Ivan Rocha retweeted
Big new open-weights model! This one could be a top model. Check out Code Arena scores.
Beats DeepSeek V4.1 and Kimi K3 in coding
Baidu released a 280B MoE model that activates only 14B parameters per token.
And it's already challenging DeepSeek and Kimi in coding.
🇨🇳 This is ERNIE 5.5 Flash Preview.
Model Stats
🧠 280B total / 14B active
📚 1 MILLION token context
⚡ Baidu reports 200+ tokens/sec on its hosted platform
🔧 Coding, tool calling and agentic workflows
🔓 Apache 2.0 weights coming soon
Now look at the early Code Arena WebDev scores:
🔥 ERNIE 5.5 Flash Preview → 1668
🟢 DeepSeek V4.1 Flash (Max) → 1664
🔵 Kimi K3 (Max) → 1658
These are early results, with ERNIE's score based on Arena's automated evaluation rather than human votes.
Baidu is showing ERNIE working across entire agent workflows like these examples ...
🎮 Built a third-person Unreal Engine 5.8 game in an 11-hour coding session.
🕹️ Built nearly 100 mobile mini-games from design documents.
✈️ Two agents collaborated to build an airport and Boeing 747 in Blender.
⚙️ It created an editable W16 engine in FreeCAD.
📊 Produced research reports, spreadsheets and presentations.
How much hardware will we need?
Raw weight math for 280B parameters:
💾 BF16 → 560GB
💾 8-bit → 280GB
💾 4-bit → 140GB
💾 3-bit → 105GB
💾 2-bit → 70GB
Before quantization metadata, KV cache and runtime overhead.
⚠️ API available now. The post-trained Preview weights are NOT downloadable yet.
Baidu says Apache 2.0 weights are coming.
Link in ALT
TideLM is an LLM engine for TWO DGX Spark which supports GLM-5.3 Full (Not Flash!) and Deepseek v4.1 1017 native MXFP4.
I changed the checkpoint to EXLR8 after seeing @bertholomusai 's TP4 recipe. Thank you for sharing the information. More weight resident = more accurate.
In the video, DSH reported 26 tok/s.
github.com/raspy135/tideLM
AWS Emulator is a local emulator of the AWS APIs. You use it to test your software without calling AWS, on your own machine, on a development server, or in continuous integration (CI).
github.com/aws/aws-emulator
72
Ivan Rocha retweeted
Blackfold is my Uncensored DeepSeek v4.1 1017 TP2 recipe and daily driver.
It’s not just your standard DS It was built for me. I wanted reliable, I wanted fast, I wanted capable, I wanted uncensored.
But I also didn’t want to waste $40,000 worth of hardware when you can get similar performance out of two Sparks.
So I put it in a neat little box packaged and gift wrapped for you. Enjoy.
github.com/Blackwellboy/Deep…
Ivan Rocha retweeted
I promised a DeepSeek recipe update. How about a 4x efficiency boost on 2x DGX Sparks!
We are already pushing the limits of the hardware, so I thought about how else we could make improvements without more hardware.
The model writes the code or math problems, and the engine actually runs it on CPU lanes instead of guessing the result with even more tokens.
No tool-call round trip to the harness, and because it can perform actions in its thinking, it's not only faster but also takes fewer turns to find the right solution.
On generated tasks, both setups got 60/60. With Compute Lanes: 6.3s vs 26.9s per reply, and 78% fewer tokens on average.
On the original suite of tests, it even solved 5 more problems than without Compute Lanes on.
It's not always a net positive though, I think that depends on the model training.
On GSM8K, MATH-500 and HumanEval, the model never called the tool, and scores barely moved. This helps when a task needs real computation.
PR here will hopefully be merged within a few hours to my main recipe.
github.com/jayleaton/deepsee…
Serve DeepSeek-V4.1-Flash-1017 from two NVIDIA DGX Sparks through an OpenAI-compatible API, with the full 1M-token window, image input, tool calling, thinking on or off, and DSpark speculative decoding
github.com/MiaAI-Lab/DeepSee…

























