building mistral․rs | inference kernels @huggingface
Joined July 2023
- Tweets96
- Following371
- Followers265
- Likes886
Pinned Tweet
mistral.rs v0.8.2 is a CUDA performance release.
On Gemma 4 E4B Q8, mistral.rs is faster than llama.cpp across the GB10 + B200 + H100 SXM prefill/decode sweep.
We also benchmarked against vLLM in the BS=1 regime and found similar speedups.
Mean speedups:
GB10 prefill: 1.83×
B200 prefill: 2.19×
GB10 decode: 1.09×
B200 decode: 1.24×
Full technical report below.
Eric Buehler retweeted
Super happy to welcome @jundotkim, oMLX creator, to Hugging Face! 🥳
We are invested in Local AI and support MLX since @awnihannun and @angeloskath released it in Christmas 2023.
oMLX gets stability and speed. It remains a community project led by Jun.
huggingface.co/blog/omlx
Eric Buehler retweeted
Happy to officially bring you the new SOTA `tokenization` library.
We focused on all languages, multi-thread scaling, minimal package size and memory usage.
huggingface-tokenizers-v1.st…
We're thankful for the players of this ecosystem that have pushed us to give the best we could!
You can run Qwen3.8-27B with mistral.rs right now 🚀!
> mistralrs run Qwen/Qwen3.8-27B --quant 4
Take advantage of:
> built in local web search and code execution
> full OpenAI server with Chat & Responses API
> blazingly fast, reliable inference on CPU, Mac, and CUDA
Check out our quickstart for more information: ericlbuehler.github.io/mistr…
We promised open weights for Qwen3.8. Now, time to meet them! 🎉
⚡ Qwen3.8-27B:
- A native multimodal dense model. With just 27B parameters, it outperforms Qwen3.7-Plus overall and shines in real-world coding & office workflows.
- 262K native context, easily extendable to 1M tokens via YaRN.
- Built for builders. Highly efficient, high-quality, and licensed under Apache 2.0.
🚀 The open weights for Qwen3.8-2.4T-A95B (Max-level) have also been released recently.
Whether you're shipping lightweight applications with Qwen3.8-27B locally or building agents with Qwen3.8-2.4T-A95B, they're yours now!
Download, deploy, and build something we haven't imagined yet. 👀👇
- Hugging Face：
huggingface.co/collections/Q…
- ModelScope：
modelscope.cn/collections/Qw…
Eric Buehler retweeted
Previewing Ultrafast mode for @OpenAI's GPT 5.6 Sol, powered by Cerebras.
GPT-5.6 Sol Ultrafast generates responses at up to 750 tokens per second. That's the full, GPT-5.6-Sol model -- up to 14× faster than the same model on Standard processing.
It speedran Humanity’s Last Exam in 11h 11m, nearly 7× faster than Claude Fable 5 with comparable accuracy.
Eric Buehler retweeted
also works ♾
mistral.rs v0.8.2 is a CUDA performance release.
On Gemma 4 E4B Q8, mistral.rs is faster than llama.cpp across the GB10 + B200 + H100 SXM prefill/decode sweep.
We also benchmarked against vLLM in the BS=1 regime and found similar speedups.
Mean speedups:
GB10 prefill: 1.83×
B200 prefill: 2.19×
GB10 decode: 1.09×
B200 decode: 1.24×
Full technical report below.
Great example of Gemma 4 in agentic tasks using skills.
This leverages the new /v1/skills support with mistralrs!
Made possible by @googlegemma's amazing work on the Gemma 4 series. @osanseviero
A 4B local model can do physics: dynamic fracture, beam mechanics, topology optimization, and Chladni plates. But how do you get a 4B model to "do" physics? We can give it scientific skills that turn intent into experiments, generate data, make images and movies, verify outputs, and report the results. In this video, Gemma-4 E4B runs locally on my MacBook via mistral․rs with 8-bit in-situ quantization, plus the Agent Skills API, completely open-source.
Each skill is a folder with a SKILL․md, teaching the model how intent, parameters, assumptions, examples, and solution strategy relate. The skill is uploaded, invoked by the model, executed in a sandboxed shell/Python environment, and it inspects the generated files before reporting the result.
In the video, I show four skills:
1. Beam mechanics: 1D Euler-Bernoulli beams with displacement and rotation: cantilever, simply supported, fixed-fixed, point loads, distributed loads, moments, springs, reactions, shear/moment diagrams, max deflection, and vertical-equilibrium residuals.
2. Fracture mechanics: 2D triangular-lattice dynamics with Morse/Lennard-Jones potentials, sharp edge pre-cracks, Mode I/II loading, bond breaking, temperature and strain-rate effects, stress-strain curves, peak stress/strain, broken-bond counts, and fracture movies.
3. Hierarchical topology optimization: Minimum-compliance optimization with density/sensitivity filtering, penalization, target volume fraction, flexible supports and loads, optimized density fields, compliance values, and STL export (so you can quickly move to manufacturing).
4. Chladni plates: Analytic superposition of vibrating-plate eigenmodes on square, rectangular, and circular plates to generate standing-wave nodal patterns.
The examples show that the model does not need to contain all of physics in its weights; instead the physics lives in the executable and editable skill. The model orchestrates and scales, and results compound.
This is quite interesting for science: AI becomes more capable when it has physical agency and can act through scientific instruments - and this can translate from in silico simulation to hardware, manufacturing, and more.
Eric Buehler retweeted
For science, AI sovereignty and physics-grounded reasoning are non-negotiable. But how can we teach a small LLM like Gemma-4-E4B physics? One way is to use Agent Skills, but this has so far been limited to closed frontier models. mistral․rs now implements Agent Skills natively: the first self-hosted inference engine that does this as part of the local inference substrate, where we can use small models to solve complex scientific and other tasks in a flexible and scalable way.
We are in a period of uncertainty about frontier models - access, pricing, deprecation, abrupt restriction. The good news is that when the entire stack runs locally we can build AI that is entirely your own: You own the weights, the skills, the execution loop, the data - all of it runs on your hardware and is reproducible and durable.
While virtually all local inference engines expose a model behind an OpenAI-compatible endpoint, everything agentic is then assembled around it by an external orchestrator that injects context, manages tools, mounts files, and brokers execution. mistral․rs is natively agentic and moves that machinery into the server itself, allowing us to build complex agentic workflows and run them locally, on open-source models.
With this new feature you can now upload Agent Skills bundles to /v1/skills, reference them from Responses API requests by identity, and run them inside a native agentic loop with persistent Python sessions, figure capture, sandboxed shell execution, file inputs mounted directly into the working session; plug-and-play and completely compatible with your existing code/workflow.
A model with a native skill substrate can act, observe consequences, and can modify what it is able to do. The skill is retained procedural capability of the system.
Attached is a short video of all of it: skills, code execution, the full agentic loop carried by Gemma-4-E4B; running entirely on my MacBook Pro. You can install and run a server with this capability in two lines in your terminal, with any quantization you need.
Nice work by the @googlegemma team @OfficialLoganK @demishassabis and @ericlbuehler with mistral․rs!
Eric Buehler retweeted
Fearless Concurrency on the GPU
For those interested @melibol just posted a paper on building a safe Rust kernel programming abstraction on top of Tile IR.
arxiv.org/abs/2606.15991
A short teaser: but the safety is effectively free. On a B200, the safe GEMM is competitive with cuBLAS: about 2 PFlop/s 92% of the GPU's dense f16 roofline.
Read more in the paper or Melih's LinkedIn post (linkedin.com/posts/elibol_gi…)
He will also be giving a talk at RustConf in September, hopefully he will see you there!
Excited to share cuTile Rust: bringing Rust's fearless concurrency to GPU kernel programming. Our paper "Fearless Concurrency on the GPU" is now on arXiv.
For me the best part is how fast you can iterate: high-performance CUDA kernels developed directly in Rust.
Huge thanks to my co-authors Melih Elibol, Jared Roesch, Isaac Gelado, and Michael Garland on this project.
Paper: arxiv.org/abs/2606.15991
Code: github.com/NVlabs/cutile-rs
Congratulations @googlegemma on their launch of Gemma 4 12B!
mistral.rs has full multimodal & agentic support.
`mistralrs run --agent -m google/gemma-4-12B-it --quant 4`
Eric Buehler retweeted
Meet Gemma 4 12B!
A unified, encoder-free multimodal model designed to bring high-performance intelligence directly to your laptop, and released under an Apache 2.0 license.
Bridging the gap between edge efficiency and advanced reasoning. Here is what’s new with Gemma 4 12B: 👇
Eric Buehler retweeted
Mistralrs fast, flexible LLM inference written in Rust by @ericlbuehler looks very promising and fast especially on CUDA side.
I took a screenshot of the What makes it fast section below.
I tested it on M5 Max using Qwen3-0.6B bf16 model vs mlx-lm. Video below and final results...
🥇 mlx-lm 303 toks/s
🥈 mistralrs 253 toks/s
I'll keep an eye on this project because looks really promising!
Check out the technical report: github.com/EricLBuehler/mist…
And the GitHub for quickstart, documentation, and examples: github.com/EricLBuehler/mist…
Eric Buehler retweeted
Numerics mean the difference between valuable tokens and useless jibberish.
We think a lot about how Luminal reasons about numerics, and how we can precisely control the tradeoffs we make in speed and quality.
Let's talk about how we can produce the perfect token.
Eric Buehler retweeted
Excited about the Gemma 4 release today that strengthens the OSS ecosystem. Check out the Rust-based inference server MistralRS that has day-zero support for Gemma 4! Comes with support for all modalities (text, vision, audio) & agentic features like built-in tool calling, web search, MCP.
Awesome work by the @googlegemma team, @demishassabis & @ericlbuehler @huggingface! Below you see us running Gemma 4 with MistralRS on a @nvidia DGX Spark, right here, at our desk.
1⃣Full multimodal: text, vision, audio
2⃣In-situ quantization, so no pre-quantized weights needed (and flexible for any use case)
3⃣Built-in tool calling, web search, MCP
4⃣Rust-native, one install, one command
5⃣Pre-quantized UQFF models available