Slyfox (@ansonschu) header
Slyfox (@ansonschu) profile picture
San Francisco, CA
Joined January 2012
Photo by Slyfox (@ansonschu)
46
621
46
7,150
187,996
Slyfox retweeted
分享一些最近的个人经历 我到深圳也有段时间了,离开上海之前我和我前同事说要去深圳发展做一些智算中心的生意,她们听到后很惊讶,因为我在他们眼里就是一个文静内向的女生,和做生意八竿子打不着,更别提什么智算,什么AI, 我承认我撒谎了骗她们说我亲戚在深圳做这个好像还挺挣钱的,但其实她们不知道我每天在研究AI,更不知道我其实网名叫 CuiMao,全网拥有不错的粉丝和人气。他们不理解,我不怪他们,因为我不打算告诉他们这些,中国有句古话叫,闷声发大财。直到我离开上海她们也一直无法理解我的选择。 本周黄仁勋的事情沸沸扬扬,国内的自媒体肆意发酵所谓的倒卖 b300 能翻身的理论,把黄丽比喻成拿到大结果的“成功女性”典范在小红书被捧上了神坛,几乎一夜之间全中国都知道英伟达服务器在中国是暴利行业能挣大钱。于是那些投机分子像雨后春笋般的冒了出来,阿猫阿狗都在朋友圈发求 b300,h200,甚至还有吹牛自己亲戚在英伟达是高管可以拿到货的,堪比当年疫情朋友圈卖口罩的乾隆盛世,说实话,我有点难过。因为我知道这一行真实的样子。服务器不是普通人能碰的,它是重资产、高杠杆,是大基建,是国之重器。真正在做这件事的人每天面对的是合规、电力、机房和交付,运维。根本没有那种一夜暴富的传奇故事。 为什么要发这篇长文,因为昨天晚上我以前的同事她来私信我了,就是上文提到的那几个人中的一个,说我不是去深圳搞服务器了吗,是不是赚大钱了,我说没有,和她来回打了几轮太极之后,最后她才支支吾吾的告诉我她被裁员了,我不知道以什么样的态度和立场去面对这件事,因为她算得上是我以前部门的领导,她和她的丈夫在上海还有普陀区一套 300 多万的房贷和一个小学 3 年级孩子,我发现除了安抚,一切会好的没有别的可以说的。 一晚上没睡,我给她认真科普了服务器知识,什么是国产替代中国算力崛起之类的,她在电话那头一直哦哦哦,嗯嗯嗯,我不知道她到底听进去了多少,但她说觉得自己上了那么多年的班脑子被上坏了,我觉得应该是感悟到什么了吧。她要去找工作,我劝了她,不要特地去找什么 AI 公司,那大概率都是骗子来的。过好自己的生活。 我觉得这个世界上想挣 AI 快钱的人太多了,定定心心,认认真真坐下来,把浮躁的心静下来去感受 AI 的人是极少数。现在简中网络上流传着这一句话说,只要你不学AI,过几个月你就可以不用学了,我觉得不无道理,但我更加认同的是,只要你保持学习 AI,你就可以驾驭更强力的 AI, 从 Webui 到 Comfyui 到 n8n ,再到 Openclaw,一代版本一代神,我随着时代在进化,我双眼看到的只有未来的世界。AI 时代, 保持认真,保持好奇,保持善良,我CuiMao 亲测过了,依旧有效。
259
116
35
2,104
228,345
Choose your path Western man.
535
3,206
290
32,293
5,330,937
Slyfox retweeted
Made on Earth by humans
6,063
18,184
1,453
227,610
7,195,087
Slyfox retweeted
I’m happy to announce that I’m the first Malaysian to receive the Trump Gold Card! Finally a permanent resident of the USA 🇺🇸🗽
2,105
1,089
313
21,431
3,209,298
Slyfox retweeted
Mechazilla catches Starship! Congratulations to the entire SpaceX team on this incredible achievement!
6,795
27,283
9,073
140,712
16,914,524
Slyfox retweeted
We’ve published an update on our recent security incident. See a detailed overview of what happened and our remediation steps here: cursor.com/blog/security-inc…
224
204
224
2,259
808,084
Slyfox retweeted
> it took us 4 days and 500$ in tokens > we beat SGLang by 15% > it took 6 prompts > 3 out of those 6 were: (a) "carry on" (b) "proceed" (c) "push to github" and... we did it from SCRATCH (git init) context: it has recently become a very clear industry trend to build a custom inference engine for each open source model that comes out... a paradigm made popular by a WAVE of papers that came out earlier this year, like hazy research's megakernels and MetaInfer the usual argument against traditional inference engines (like vllm / sglang) is that they, like all conventional swe products, use abstractions. the use of abstractions is synonymous with performance overhead (triton vs cuda, python vs c++, etc). combine this with increasing hardware complexity, increasing parallelization strategies, increasing kernel specialization techniques... and you can make a very clear (but very FLAWED) argument for building an engine customized purposely for one model. why flawed? it is true that traditional inference engines waste cpu time running generic abstractions. sure. but the gpu kernels (the functions run on the gpu) are almost always longer, and so the wasted cpu time is not the bottleneck. if you know your shapes, and run cuda graphs ahead of time (and you almost always do), this is NEVER a problem. so why would you build your own engine? why are people RAVING about this? so we tested it out. we built one for DeepSeek V4.1 Flash. The goal was two-fold: 1) is it feasible to do so (spoiler: yes) 2) is it useful to do so (spoiler: also yes...but for different reasons) we built it in rust (in retrospect, this did not really matter), and it took us 6 prompts and cost 500$ end to end. unshackled from working within a pre-existing codebase or inheriting code debt, we: - implemented custom kernels (GEMV, MoE, and AllReduces) specifically for DSV4.1 at our batch size - we were able to put an entire spec dec step in one graph, and verify across 7 tokens at that - we were able to mix deepseek's 4 residual streams in 1 fused kernel instead of 80 - fused the routing so that picking and running the expert happened in a single step (saving one write and one read i/o from each expert) iterating on the model was cheap to do in a custom sandbox-like engine, and we were able to go much faster than a traditional dev cycle. we were then able to port every single thing into sglang, and matched performance imo, it is this easy iteration that makes this paradigm useful. little else matters. and, ofc, we've open-sourced the engine and upstream(ed/ing) the patches to @sgl_project
we built an inference engine for deepseek v4.1-flash from scratch. the core engine is just rust + cuda. no pytorch, no sglang, no dsls, etc. ~1740 TPS decode on 4xB300, 2060TPS on 8xB300. 1M token prefill in ~12s. it's up on github
45
60
9
908
55,672
FAA grounds fights nationwide due to "Extraordinary circumstances", a day after the Microsoft outage. Coincidence?
56
334
31
2,027
66,017
Slyfox retweeted
NUREMBERG - Official Trailer
🤖 Made with AI
4,570
14,028
5,243
130,546
31,337,079
I have a particularly nasty case of cancer. When I was hospitalized again at my lowest point last year, I told Jensen. He immediately reached out to contacts at Stanford, who were able to cut through red tape, and get me into a clinical trial for which I've turned out to be a perfect fit. I think everyone can have their own views about recent events. But any claims that Jensen did not really care about his staff is simply inaccurate. I, personally, owe him more than I can express. I think the best tribute we can pay to Jensen's legacy is to drive NVIDIA's continued success and push the whole industry forward, by showing we've learned the lessons from years of working here. We will compare ourselves against Speed of Light. The mission will be the boss. And we will not be scared to take smart, bold risks and fail fast with ruthless intellectual honesty. But if there is one management story I remember the most, it's a particular Jensen grilling from my time in autonomous vehicles that stayed with me. Luckily I was not in the hot seat for this one. Team A didn't want to use Team B's framework, because they thought it was too slow. Jensen was surprised - it's just applying a bunch of optimized CUDA kernels in sequence, how could it be slow? So he asked the executive from team B to draw the entire data flow sequence from their framework on the whiteboard and highlight every memcpy. Sure enough, there were a bunch of unnecessary memcpy's surrounding each pipeline stage. Everyone in the room could see why it wasn't getting to SoL and a path to improve things. No arguing about fiefdoms or appeals to authority - the data got us all on the same page. The lesson was not (just) that unnecessary memory copying is bad. It's that executives should not be afraid to probe into details if something doesn't add up, and leaders need to be technical enough and honest enough to engage in real time and not turn every meeting into "I'll get back to you in a week." If that means counting memcpy's with a billionaire CEO, so be it. My best executive meetings at NVIDIA have been when I'm fully prepared to present every detail of my team's work, and my worst failures have come when I realized I couldn't go deep enough to really solve the problem. But the thing that gives me confidence is that Jensen was never the only one to live this philosophy. Anyone who has presented to an SVP at NVIDIA knows they are at risk for getting grilled on fundamental constraints and technical details. I have been part of amazing conversations with both recent and old timer leaders where we were able to get to the core of a problem in an hour rather than a month, because we had the right people in the room and were not afraid to go deeper. The company is filled with supremely talented people who have been trained for 3 decades to work at the highest level, and you can see it in what we produce every day. I'm so proud to be an NVIDIAn, and I can't wait to show off what we ship in the coming months and years.
418
1,988
270
21,560
2,582,221
Slyfox retweeted
嘿,Jensen! 此片献予你! 我是一名来自中国的 AI 开发者,你一定会回来的对吧。对吧 @JensenHuang
🤖 Made with AI
614
1,118
370
7,398
2,681,661
I have a major result to share. I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China. This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model. Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves. I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems. I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used. It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20. These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs. I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpo…. There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
1,813
6,405
2,189
41,149
6,642,543
Today we report a startling pretraining finding: Attention circuits STOP meaningfully evolving much sooner than we expect. In other words, the model has decided where to look very early on. It is known that different seeds create starkly different results; here we isolate a specific mechanism that causes it. The setup: We perform two identical training runs with the same data order and different initializations and find that by the first thousand steps each run’s attention resembles its own step-10k pattern MORE than the individual finished runs resemble each other. The initialization plays a greater role in the attention patterns than training dynamics, which implies that the final pattern is randomly chosen. NOTE: Randomly chosen, not from the space of all the possible attention patterns that it could be, but from the different attention pattern basins that it could converge to—which one it ends up in appears to be init dependent. Some differences in the loss are attributable to these different attention patterns. This means the final performance is somewhat dependent on a random choice. Depending on how lucky with pattern formation one gets in the first thousand steps, we could end up much better or worse in different contexts. This further implies training could simply find the optimal patterns, or at least better ones, if the attention could evolve instead of being stuck. A story for why attention evolves less than we’d hope falls out of the softmax derivative. We can observe a positive feedback loop we call the credit ratcheting. Let oᵢ be the head output and δᵢ = ∂L/∂oᵢ the loss gradient at that output. Define key k’s advantage for query i as Aᵢₖ = −(vₖ − oᵢ)ᵀδᵢ i.e. how much the loss drops if oᵢ moves toward vₖ, relative to the current attention-weighted mix. Thus, the gradient with respect to sᵢₖ = kₖᵀqᵢ / √d is ∂L/∂sᵢₖ = −aᵢₖAᵢₖ A key can have a huge advantage and have its score still barely move, because the advantage is multiplied by the attention it already gets. It’s the same form as a softmax policy gradient, π(a)·A(a): attention is the policy, and it mostly learns about the actions it already takes. In aggregate, the query gradient is gᵢ = Jᵢᵀδᵢ where Jᵢ = (1/√d) ∑ₖ aᵢₖ(vₖ − oᵢ)kₖᵀ. Across 166,656 Jacobians in Qwen2.5-1.5B, ≥99% of tokens in each of the 3 heads we probed had squared-spectrum effective rank <10, in a 128-dim head. The update to the query is being filtered through an exceedingly narrow channel. The values have the same issue. Key k’s share of the value gradient is Gᵢₖ = aᵢₖδᵢxₖᵀ, so at similar δᵢxₖᵀ, 10× the attention ≈ 10× the credit. This means the values mostly adapt to positions already looking at them. This compounds! Attention allocates credit, credit updates values, and values can pull attention further toward the winners. The most natural approach to fix this is to precondition the query gradient with the Fisher of the attention distribution (NATQ): Fᵢ = Covₐᵢ(k) / d uᵢ = (Fᵢ + λᵢI)⁻¹gᵢ g̃ᵢ = uᵢ · (‖gᵢ‖₂ / ‖uᵢ‖₂) Even this blunt version does a lot. At 124M, entropy rose from 3.75 to 4.69 nats, top-key attention fell from 0.238 to 0.149, and in layers 4–7, keys the baseline gave under 0.1% attention got 67% of the gain. Validation loss rose by 0.082 nats. A more targeted intervention need not make this concession. Here’s the complete blog: hyperstition.cc/stuck-in-a-l… We frame this as an open problem that offers a stark contrast in the way we perceive pretraining. If we can find a way to allow for attention to evolve throughout training, opposed to just the very start, we may find inordinate gains on model capability. Internally we half joke that the alternative read is that attention might simply not be real past the first few steps, and it’s all the MLPs. If training is able to work as well as it does without really changing the attention patterns much, maybe the MLP is the real workhorse that’s driving the learning.
24
51
12
530
41,766
Our initial public offering: ipo.anthropic.com Legal disclaimer: Anthropic has filed a registration statement (including a prospectus) with the SEC for the offering to which this communication relates. Before you invest, you should read the prospectus in that registration statement (available at: sec.gov/Archives/edgar/data/…) and other documents Anthropic has filed with the SEC for more complete information about Anthropic and this offering. You may get these documents for free by visiting EDGAR on the SEC Web site at sec.gov. Alternatively, copies of the preliminary prospectus may be obtained from: Goldman Sachs & Co. LLC, Attention: Prospectus Department, 200 West Street, New York, NY 10282, by telephone at (866) 471-2526, or by email at [email protected]; or Morgan Stanley & Co. LLC, Attention: Prospectus Department, 180 Varick Street, 2nd Floor, New York, NY 10014. The securities may not be sold, nor may offers to buy be accepted, prior to the time the registration statement becomes effective. This post is not an offer to sell, or a solicitation of an offer to buy, the securities in any jurisdiction where the offer or sale is not permitted.
1,070
1,020
909
10,462
3,267,300
Slyfox retweeted
Congrats to @SIGTERMusic for winning the $100 community vote! Round 2 on now
All votes are in and the winners are chosen for $CLAUDIA Contest Round 1 🏆 59 entries. 🥇 @PinelandLabs, "Who Is Prompting You?" sxsi.com/s/contest-who-is-pr… 🥈 @AIDoompop, "A.M.F.A.B." sxsi.com/s/contest-amfab/ 🥈 @zion_skank, "Can Claudia Become a Bit More Human?" sxsi.com/s/contest-a-bit-mor… 🗳️ Community Vote: @SIGTERMusic, "TRAP" sxsi.com/s/contest-trap/ Congrats to all the winners! Round 2 is on and voting is live! Check them out at sxsi.com
10
34
7
258
12,193
Slyfox retweeted
Claudia to Study To
33
56
8
382
18,087
🤖 Made with AI
376
603
394
12,484
1,237,958