
making compilers grin, building distributed systems, previous @ihc__official, research @iiit_hyderabad
Brooklyn, NY
Joined July 2021
- Tweets9.4K
- Following870
- Followers711
- Likes126K
i learnt this lesson this year. it's helped me follow through on my ideas and not get stuck overthinking it
don't ask if you should do the thing. do it, then come back with a demo or something tangible
you can just do things
I'll be giving a talk tomorrow at the SGLang Summit at 10am!
Oftentimes, I find explanations in ML infra to be unsatisfying. A common failure mode is that they'll explain the *literal* tradeoffs but fail to provide a mental model for how to reason about the choice.
This talk won't explain everything about inference (only 15 minutes!), but I'll provide two mental models I've found useful when reasoning about inference, and I hope you find them useful too :)
tushar retweeted
I have a major result to share.
I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China.
This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model.
Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves.
I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems.
I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used.
It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20.
These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs.
I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpo….
There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
tushar retweeted
It actually will be studied in history how well Sam played the coup
Literally not only was able to come back
But able to fire literally everyone on the board who sided against him
Was able to gain even more power
tushar retweeted
hot take: unless you are willing to reimplement git / github (like anthropic), I think monorepos are an antipattern in an age of agents
the alternative is to keep things light and make a lovely cli/mcp for agents to do search over your org's code and docs without a checkout
tushar retweeted
this blog post by @ethanniser has aged so well
ethanniser.com/blog/not-hold…
tushar retweeted
We're still so small but I'm loving the team we've got at Superlogical. They're so talented and kind and love their craft. It's a different era of software development and our expectations and practices reflect that. I want to share a bit about how we work.
First, everyone is trusted to push to main. We do machine-aided code review pre-push and human plus more machine-aided code review post-push. Limited to no PRs.
Culturally, we've all agreed that if we ever hire someone who we DON'T trust to push directly to main, then they probably aren't the right fit for the team (so far, not an issue).
Some common sense here: if someone is stepping outside their lane or comfort zone, we'll open PRs. For example, I'm pretty trash at macOS/iOS UI, so I'll open PRs for UI-affecting functionality and ask for eyeballs/help. But for business logic and internals, I push to main.
Second, everyone can cut a release. Our release process is fully automated and when you start a release (for any platform) it triggers a Slack notification and a 15 minute countdown.
The 15 minute warning is a final gate that allows people to review plus collaborate on a changelog. During that 15 minutes, anyone in the company can pause the release rollout. After 15 minutes, the train leaves the station. You can't override the 15 minutes (except to pause). It gives everyone a chance to step in.
Third, we avoid synchronization through fast remediation rather than through planning. The server and clients on various platforms ship features as fast as they can. We use post-hoc human review and AI-assistance to identify feature mismatches across products ("Windows has support for X and Mac doesn't." or "Mac and Windows differ in their behaviors for Y in this way.").
This avoids expensive "sync" meetings and coordination. We trust that producers of features are mindful of what they're producing and we trust that adjacent product lines will self-synchronize quickly. In practice, we see feature parity across platforms within 24 to 48 hours.
All of this works on mutual trust. It ONLY works with mutual trust.
Everyone at all times has to trust that they've got each others back:
1. Push to main only works if you fix your mess, or if you don't, that someone else will quickly but that you'd do the same (and generally in equal amounts).
2. Anyone can deploy only works if you trust everyone to cut at the right time, with the right changelog, and that people will respect the 15 minute timer.
3. Sync through fast remediation only works if you trust other teams will pick up and integrate your work quickly, which only works if you would too. It also only works if you are fairly producing mismatches (e.g. not constantly).
We've got a great team. I know we're still small and that these practices will change in many ways as we scale. But this is already a markedly different practice than what I would've done 15 years ago.
In general, we are building from day 1 on a foundation of mutual respect, excellence, and trust. And it's been awesome.
tushar retweeted
spoke with my younger brother in australia yesterday.
he was out checking the surf at 3pm on a tuesday before going home to cook dinner with his gf.
maybe i chose the wrong life


































