Cofounder of Hoid (SR007), prev: CS @ Cambridge https://teapawt.org/t.co/3hyJX4WIJh
Joined October 2023
- Tweets132
- Following164
- Followers205
- Likes173
We're publishing our blog post about building megakernels that make LLMs run at the limits of physics.
Megakernels let you squeeze every last bit of performance out of your GPU. Unfortunately, they're also notoriously hard to make (performantly).
We're building a megakernel factory, and this is just the beginning. Everyone deserves top-notch performance from the hardware they buy.
Link to the blog in the comments!
There is a giant discrepancy between teams that know how to make inference fast (like OAI) and everyone else. Hoid is all about removing that diff
Up to 8x faster, and now you know why ⚡️@openai GPT-6 Astra Ultrafast runs on NVIDIA
Read the story: nvda.ws/3W2bZ5H
vLLM team showed the value of megakernels on TPUs. Most GPU workloads don't run megakernels because its really hard to make them performant. We're about to change that
Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding.
To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8.
We are open sourcing it today.
1/2
The biggest benefit of frontier models is that people stopped complaining about how hard it is to install cuda
Don't let the model size fool you, this is compacted intelligence at its finest
We release Needle 3: A Sliceable 8-29MB automation foundation model that can match DeepSeek V4 Flash.
One set of weights, every depth from 2 to 20 layers a model of its own, 25-121M parameters at CQ2-bit, built on our Simple Attention Networks and running locally at up to 4k tokens/sec decode speed on a Raspberry Pi 5.
Needle does not chat. Every turn is a function call: give it the tools your app exposes and it picks the right ones and fills every argument from what the user said, or hand it a schema and it returns a typed record. Ask for something no tool covers and you get an empty list, not a guess.
That trade is lets 121M parameters trained on 360B tokens of structured data beat models 10x their size on mobile tool calls and match 2-3x bigger models on structured JSON extraction.
It runs on mobiles, wearables, smart home devices, small robots and microcontrollers, with prebuilt engines for macOS, Linux, Windows, Android, iOS, watchOS, tvOS, the browser and WASI hosts. Try it in your browser: cactuscompute.com/needle
Pavle Padjin retweeted
OpenWhispr v1.10.0 ships with a speed upgrade!
Orukeet wins 61 of 74 benchmark splits; Parakeet wins 13.
Hoid's latest OpenWhispr PR: 171 → 110 ms for Orukeet (36% lower warm median). Apple M5, 11-second clip, before/after optimization.
step aside @nvidia, there's a new top dog in speech models: orukeet.
trained by @OrukLabs, optimized by @hoid_ai, and available on day zero in @openwhispr (100k+ downloads).
25 languages. Open weights + code.
Pavle Padjin retweeted
step aside @nvidia, there's a new top dog in speech models: orukeet.
trained by @OrukLabs, optimized by @hoid_ai, and available on day zero in @openwhispr (100k+ downloads).
25 languages. Open weights + code.
Pavle Padjin retweeted
language is a superposition over possible meanings
we're testing quantum architectures for speech models at @OrukLabs
one challenge is that quantum memory can hold the answer but still lack a way to reveal it. our preprint shows how entanglement itself can store information.
increasing FLOP/s while degrading model quality is the highschool equivalent of doing half rep push-ups just to say you can do 50
I think agents should have some way to percieve time.
E.g. make some compact representation of datetime that agents are post-trained to interpret, that is automatically injected into context for every message.
I don't really understand people saying "never manually review your code" or "always manually review your code". It is not binary, it depends what part of codebase you're editing
pushing FLOP/s through the roof(line). check out the blogpost
Our first blogpost: @hoid_ai 's bet that traditional AI Compilers need to be redesigned from first principles.
We’re also publishing our initial results. Enjoy!
runhoid.com/posts/rethinking…
We're coming out of stealth, it's time to build something great!
tl;dr of life recently: @ppadjin , @Zelja__ and I quit our jobs to build a startup (link in comments) and we're backed by @speedrun . It is the best time in history to build a company and we didn't plan on missing the boat.
@hoid_ai is an agentic AI compiler that turns any AI model into a highly optimized implementation for the given hardware.
We’re building the layer between AI models and the silicon they run on, using AI itself to explore the optimization space and squeeze out every last bit of performance.
If you're in the AI inference space hit me up let's chat.
Pavle Padjin retweeted
tl;dr of life recently: @ppadjin , @Zelja__ and I quit our jobs to build a startup (link in comments) and we're backed by @speedrun . It is the best time in history to build a company and we didn't plan on missing the boat.
@hoid_ai is an agentic AI compiler that turns any AI model into a highly optimized implementation for the given hardware.
We’re building the layer between AI models and the silicon they run on, using AI itself to explore the optimization space and squeeze out every last bit of performance.
If you're in the AI inference space hit me up let's chat.