David Thomas retweeted
NoGraphicsAPI is released (MIT license)!
It's a low-level Vulkan 1.4 wrapper using latest extensions (BDA, descriptor heap, device address commands, unified image layouts) to get as close as possibly to my No Graphics API blog post and SIGGRAPH 26 talk.
github.com/sebbbi/NoGraphics…
Exciting to see this work out in the wild! 🚀 It’s been great working with such a talented group. My contribution (so far): writing optimized GKA decoding fused kernels exploiting state matrix symmetry for extra speedup⚡. Check out the details in the paper. More to come! 🛠️
Introducing Priming
Hybrid models are faster and cheaper than Transformers to scale. But developing alternative architectures from scratch requires expensive pre-training runs.
Priming solves this by leveraging pre-trained Transformer weights to train equally performant Hybrid models with 2× faster throughput. Builders can now iterate on Hybrid architectures for under 150B tokens, 100× cheaper than pre-training.
1/12
3
138
David Thomas retweeted
💾🚀 Run Llama-3.1-405B FP8 (410GB) on a single 180GB GPU
#NVIDIA
Introducing FlexTensor — NVIDIA's new library that makes host RAM a transparent extension of your GPU memory. One call: flextensor.offload(model). No model rewrites, no framework changes. Works with vLLM, HuggingFace, and any PyTorch model.
Traditional offloading is reactive — move data when you run out of memory, stall the GPU while you wait. FlexTensor instead profiles your model's layer access patterns, then solves a knapsack optimization to schedule prefetches that overlap with compute. By the time a layer needs its weights, they're already there.
The freed VRAM gives vLLM more room for KV cache — enabling 4x longer contexts (8K→32K) or 4x larger batches. For video generation (Wan2.2-T2V-A14B on GB200): +0.1% overhead. Handles FP8, custom Triton kernels, and multi-GPU. Profiles saved to disk — no warmup on repeated runs.
Check it out: github.com/ai-dynamo/flexten…
David Thomas retweeted
I'm pretty annoyed that Hypersteer (a work by some of my friends applying hypernetworks to produce very effective steering vectors from text descriptions) has not received the appropriate amount of credit in later work pursuing basically the same idea arxiv.org/abs/2506.03292
We’re excited to introduce Doc-to-LoRA and Text-to-LoRA, two related research exploring how to make LLM customization faster and more accessible.
pub.sakana.ai/doc-to-lora/
By training a Hypernetwork to generate LoRA adapters on the fly, these methods allow models to instantly internalize new information or adapt to new tasks.
Biological systems naturally rely on two key cognitive abilities: durable long-term memory to store facts, and rapid adaptation to handle new tasks given limited sensory cues. While modern LLMs are highly capable, they still lack this flexibility. Traditionally, adding long-term memory or adapting an LLM to a specific downstream task requires an expensive and time-consuming model update, such as fine-tuning or context distillation, or relies on memory-intensive long prompts.
To bypass these limitations, our work focuses on the concept of cost amortization. We pay the meta-training cost once to train a hypernetwork capable of producing tasks or document specific LoRAs on demand. This turns what used to be a heavy engineering pipeline into a single, inexpensive forward pass. Instead of performing per-task optimization, the hypernetwork meta-learns update rules to instantly modify an LLM given a new task description or a long document.
In our experiments, Text-to-LoRA successfully specializes models to unseen tasks using just a natural language description. Building on this, Doc-to-LoRA is able to internalize factual documents. On a needle-in-a-haystack task, Doc-to-LoRA achieves near-perfect accuracy on instances five times longer than the base model's context window. It can even generalize to transfer visual information from a vision-language model into a text-only LLM, allowing it to classify images purely through internalized weights.
Importantly, both methods run with sub-second latency, enabling rapid experimentation while avoiding the overhead of traditional model updates. This approach is a step towards lowering the technical barriers of model customization, allowing end-users to specialize foundation models via simple text inputs. We have released our code and papers for the community to explore.
Doc-to-LoRA
Paper: arxiv.org/abs/2602.15902
Code: github.com/SakanaAI/Doc-to-L…
Text-to-LoRA
Paper: arxiv.org/abs/2506.06105
Code: github.com/SakanaAI/Text-to-…
David Thomas retweeted
We’re excited to introduce Doc-to-LoRA and Text-to-LoRA, two related research exploring how to make LLM customization faster and more accessible.
pub.sakana.ai/doc-to-lora/
By training a Hypernetwork to generate LoRA adapters on the fly, these methods allow models to instantly internalize new information or adapt to new tasks.
Biological systems naturally rely on two key cognitive abilities: durable long-term memory to store facts, and rapid adaptation to handle new tasks given limited sensory cues. While modern LLMs are highly capable, they still lack this flexibility. Traditionally, adding long-term memory or adapting an LLM to a specific downstream task requires an expensive and time-consuming model update, such as fine-tuning or context distillation, or relies on memory-intensive long prompts.
To bypass these limitations, our work focuses on the concept of cost amortization. We pay the meta-training cost once to train a hypernetwork capable of producing tasks or document specific LoRAs on demand. This turns what used to be a heavy engineering pipeline into a single, inexpensive forward pass. Instead of performing per-task optimization, the hypernetwork meta-learns update rules to instantly modify an LLM given a new task description or a long document.
In our experiments, Text-to-LoRA successfully specializes models to unseen tasks using just a natural language description. Building on this, Doc-to-LoRA is able to internalize factual documents. On a needle-in-a-haystack task, Doc-to-LoRA achieves near-perfect accuracy on instances five times longer than the base model's context window. It can even generalize to transfer visual information from a vision-language model into a text-only LLM, allowing it to classify images purely through internalized weights.
Importantly, both methods run with sub-second latency, enabling rapid experimentation while avoiding the overhead of traditional model updates. This approach is a step towards lowering the technical barriers of model customization, allowing end-users to specialize foundation models via simple text inputs. We have released our code and papers for the community to explore.
Doc-to-LoRA
Paper: arxiv.org/abs/2602.15902
Code: github.com/SakanaAI/Doc-to-L…
Text-to-LoRA
Paper: arxiv.org/abs/2506.06105
Code: github.com/SakanaAI/Text-to-…
David Thomas retweeted
This is what we've been coking for the last 9 months: make MoEs training goes ~2x faster and ~2x less memory! Highlights:
- MoE typically takes the most time and memory in modern models. Turns out one can mathematically rewrite the MoE backward pass to reduce the activation mem you need to store in the fwd by ~2x, resulting in the same gradients with no extra matmul recomputation. I really like this result, as it combines both algorithmic and systems insights.
- Analyzing bottlenecks in MoE layer leads to a natural optimization stragegy: reduce mem reads/writes as much as possible! Gathering the input for fwd and output grad for bwd can sometimes take as much time as the grouped GEMMs. We fuse gather with grouped GEMM + overlap mem access and compute to make the whole layer goes ~2x faster.
- Computing top-k for expert routing can take surprisingly long, ~15-20% of the whole MoE layer! Standard top-k impl uses radix top-k algo, great for large k but suboptimal for small k. We rewrote top-k using bitonic top-k algo, and it's sometimes 20-30x faster than pytorch's top-k!
All the main kernels are written in Cute-DSL so they should be easy to extend (and install :D). Hopper kernels are out, Blackwell kernels are just about ready. MoE models used to be 2x less hardware-efficient to train, hopefully Sonic-MOE will change that.
🚀SonicMoE🚀: a blazingly-fast MoE implementation optimized for NVIDIA Hopper GPUs. SonicMoE reduces activation memory by 45% and is 1.86x faster on H100 than previous SOTA😃
Paper: arxiv.org/abs/2512.14080
Work with @MayankMish98, @XinleC295, @istoica05, @tri_dao
David Thomas retweeted
My "No Graphics API" blog post is live! Please repost :)
sebastianaaltonen.com/blog/n…
David Thomas retweeted
THE MARINERS ARE GOING TO PLAY FOR THE AMERICAN LEAGUE CHAMPIONSHIP! #SeizeTheMoment
David Thomas retweeted
Elon Musk says that the left is the party of murder and unless we fight back, they are going to kill us all. These words are truly irresponsible and so far from the truth that it's laughable.
I am a Republican from California who lives in an area where Democrats win elections 90-10, and i have never once feared for my life. Both times that I voted for Donald Trump in 2016 and 2020, i wore my red hat to the voting booth, and although people gave me dirty looks, i didn't fear for my life.
The lunatics are a small fringe that the media loves to highlight. In a country with 75 million Democrats and 75 million Republicans, the violent incidents are tiny by comparison.
These broad generalizations need to stop. It is disgusting and obvious that those who peddle the idea that the other side wants you dead are doing it for their own monetary or political gain. Americans on all sides need to start rejecting these merchants of hate no matter what side of the political aisle they are in.
David Thomas retweeted
i've been trying to understand the transformer architecture for ages but it never clicked. so recently i started trying to learn its history and that has helped a LOT. to try to cement what i learned imma write up a essay.
Undefined Behavior presents: The Evolution of Attention
David Thomas retweeted
New blog post! "Measuring acceleration structures", in which we will compare BVH costs on various GPU architectures and drivers and attempt to understand the details enough on AMD hardware to make sense of the numbers!
Retweets appreciated :) Link in reply because algorithm.
David Thomas retweeted
I released a new game on Itch: Grass Zen!
- Control the wind and find secrets hidden in the grass
- Big grass simulation on the GPU
- Endless terrain
- 10 secrets to find
- Made with Wicked Engine
- Single exe, nothing else
Download: turanszkij.itch.io/grass-zen
David Thomas retweeted
You won't believe this. My team and I make this show called SciShow that works hard to be rigorous and accurate while also being appealing and entertaining.
We reach a LOT of people every year with episodes about crabs and neutrinos and gender and cancer and vaccines.
David Thomas retweeted
SDL3 is officially released!
3.2.0 is our first stable build. Go get it and have a ton of fun with it!
github.com/libsdl-org/SDL/re…
David Thomas retweeted
I am impressed about our new WebGPU WASM page load time. Whole engine loads in just a few hundreds of milliseconds. And games load pretty much instantly too.
RT @hankgreen: Obviously, if you run in front of a car you get tired. But if you run behind one, you actually get exhausted.























