tushar (@gpu004) header
tushar (@gpu004) profile picture

making compilers grin, building distributed systems, previous @ihc__official, research @iiit_hyderabad

Brooklyn, NY
Joined July 2021
Photo by tushar (@gpu004)Photo by tushar (@gpu004)Photo by tushar (@gpu004)Photo by tushar (@gpu004)Photo by tushar (@gpu004)Photo by tushar (@gpu004)Photo by tushar (@gpu004)Photo by tushar (@gpu004)Photo by tushar (@gpu004)Photo by tushar (@gpu004)
tushar retweeted
The Hugging Face founders timed that sale to perfection. An absolute masterclass in dealmaking.
45
58
12
2,940
157,193
tushar retweeted
i learnt this lesson this year. it's helped me follow through on my ideas and not get stuck overthinking it don't ask if you should do the thing. do it, then come back with a demo or something tangible you can just do things
209
737
101
9,752
354,673
tushar retweeted
I think the secret to happiness is just becoming a morning person
544
6,864
1,120
65,249
1,385,080
tushar retweeted
967
41,223
1,128
423,398
4,999,776
Language design starts with semantics, not syntax.
21
25
5
277
7,566
tushar retweeted
I'll be giving a talk tomorrow at the SGLang Summit at 10am! Oftentimes, I find explanations in ML infra to be unsatisfying. A common failure mode is that they'll explain the *literal* tradeoffs but fail to provide a mental model for how to reason about the choice. This talk won't explain everything about inference (only 15 minutes!), but I'll provide two mental models I've found useful when reasoning about inference, and I hope you find them useful too :)
36
30
6
802
37,747
I have a major result to share. I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China. This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model. Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves. I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems. I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used. It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20. These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs. I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpo…. There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
1,811
6,402
2,187
41,128
6,632,626
tushar retweeted
It actually will be studied in history how well Sam played the coup Literally not only was able to come back But able to fire literally everyone on the board who sided against him Was able to gain even more power
5
5
2
426
23,972
nvidia
1
17
506
is Nvidia hiring? asking for a friend
82
395
48
3,184
125,366
buying more $NVDA even if it turns into a whore house
146
371
69
5,744
255,090
tushar retweeted
give whichever employee at google shipped “previously used on” a raise
34
12
5
1,021
19,388
hot take: unless you are willing to reimplement git / github (like anthropic), I think monorepos are an antipattern in an age of agents the alternative is to keep things light and make a lovely cli/mcp for agents to do search over your org's code and docs without a checkout
139
25
33
819
310,848
tushar retweeted
legendary hacker news comments that aged like milk
117
250
71
5,751
873,359
Everything is sales
25
171
21
2,390
58,486
tushar retweeted
this blog post by @ethanniser has aged so well ethanniser.com/blog/not-hold…
9
24
1
259
11,562
We're still so small but I'm loving the team we've got at Superlogical. They're so talented and kind and love their craft. It's a different era of software development and our expectations and practices reflect that. I want to share a bit about how we work. First, everyone is trusted to push to main. We do machine-aided code review pre-push and human plus more machine-aided code review post-push. Limited to no PRs. Culturally, we've all agreed that if we ever hire someone who we DON'T trust to push directly to main, then they probably aren't the right fit for the team (so far, not an issue). Some common sense here: if someone is stepping outside their lane or comfort zone, we'll open PRs. For example, I'm pretty trash at macOS/iOS UI, so I'll open PRs for UI-affecting functionality and ask for eyeballs/help. But for business logic and internals, I push to main. Second, everyone can cut a release. Our release process is fully automated and when you start a release (for any platform) it triggers a Slack notification and a 15 minute countdown. The 15 minute warning is a final gate that allows people to review plus collaborate on a changelog. During that 15 minutes, anyone in the company can pause the release rollout. After 15 minutes, the train leaves the station. You can't override the 15 minutes (except to pause). It gives everyone a chance to step in. Third, we avoid synchronization through fast remediation rather than through planning. The server and clients on various platforms ship features as fast as they can. We use post-hoc human review and AI-assistance to identify feature mismatches across products ("Windows has support for X and Mac doesn't." or "Mac and Windows differ in their behaviors for Y in this way."). This avoids expensive "sync" meetings and coordination. We trust that producers of features are mindful of what they're producing and we trust that adjacent product lines will self-synchronize quickly. In practice, we see feature parity across platforms within 24 to 48 hours. All of this works on mutual trust. It ONLY works with mutual trust. Everyone at all times has to trust that they've got each others back: 1. Push to main only works if you fix your mess, or if you don't, that someone else will quickly but that you'd do the same (and generally in equal amounts). 2. Anyone can deploy only works if you trust everyone to cut at the right time, with the right changelog, and that people will respect the 15 minute timer. 3. Sync through fast remediation only works if you trust other teams will pick up and integrate your work quickly, which only works if you would too. It also only works if you are fairly producing mismatches (e.g. not constantly). We've got a great team. I know we're still small and that these practices will change in many ways as we scale. But this is already a markedly different practice than what I would've done 15 years ago. In general, we are building from day 1 on a foundation of mutual respect, excellence, and trust. And it's been awesome.
178
195
69
4,609
296,337
spoke with my younger brother in australia yesterday. he was out checking the surf at 3pm on a tuesday before going home to cook dinner with his gf. maybe i chose the wrong life
67
64
22
5,221
444,721
tushar retweeted
holy shit, @elonmusk was right
Replying to @KatieMiller
SoftBank is defaulting on their turbine orders
29
62
4
1,029
41,739
Fun Fact: Basically none of the initial railroad companies made money. They overbuilt, took on way too much debt, and ended up in brutal pricing competition with each other, and nearly all went bankrupt.
201
439
100
10,828
530,615