Pretrain and Scaling Research; prev@ByteDance Seed; opinions are my own
Beijing Haidian
Joined November 2021
- Tweets385
- Following900
- Followers1.6K
- Likes1.4K
Pinned Tweet
Life update: I spent three months on ByteDance’s Seed Pretrain team. Working with fantastic colleagues, I gained rapid insights into pretraining and model scaling. I’m grateful for this experience so so much and have decided to leave Seed for a break.
Jiaxuan Zou retweeted
(1/9)Suppose you're training an LLM on web data with limited math data. Should you place all your math data at the end of training, or mix it with web data earlier? The answer flips with the learning rate (LR) schedule!
At fixed compute and mid-training (MT) token budgets, earlier mixing gives lower MT loss when LR decays throughout training. With constant LR, placeing all MT data for the end works better.
We then vary when LR decay begins. The best tested strategy: keep LR high after MT data enters, then decay near the end.
See our new paper: "Data Schedules and Learning Rate Schedules Should Be Designed Together" !
Jiaxuan Zou retweeted
AutoResearch doesn't generalize.
Without understanding, we're just worshiping its shit.
A training trick, found by AutoResearch, leads to an overfitting disaster we call "forking", under a small change in setting.
We (human researchers) dig into the mess and find why.
Jiaxuan Zou retweeted
[1/n] 作为这场训练“投毒”事件最直接的受害者，2024年那几个月漫长的训练排查、对峙与质疑，对我而言是一段炼狱般的经历，也是一场对意志和技术的锤炼。
感谢好友们最艰难的时候支持陪伴。也感谢当时所有并肩作战的同事们。
Jiaxuan Zou retweeted
Kimi K3.1 Pro launches at 4th on the Artificial Analysis Intelligence Index scoring 55, making large gains in agentic performance; it will be the leading open-weights model once its weights are published
@Kimi_Moonshot's new reasoning model gains 11 points over Kimi K3 (max, 44). It trails only Claude Opus 5.5 (max with fallback, 58), Claude Sonnet 5.5 (max with fallback, 56), and Muse Wonder Full (max, 56) on the Intelligence Index.
Once the weights for K3.1 Pro are released it will be the highest-scoring open weights model and the gap between the leading open weights and proprietary models on the Intelligence Index will narrow from 12 points to 3.
Key results:
➤ Large gains in agentic performance: Kimi K3.1 Pro scores 3rd on GDPval-AA with 1,846 Elo, up 312 Elo from Kimi K3, behind only Claude Opus 5.5 (1,866 Elo) and Muse Wonder Full (1,857 Elo). Kimi K3.1 Pro also achieves the highest score of any model on Terminal-Bench 4.0. K3.1 Pro scores 67%, up from 13% for Kimi K3 and ahead of Claude Sonnet 5.5 (64%). AutomationBench-AA, which tests agentic SaaS workflows, rises from 58% to 73%.
➤ Gains in other evals are more moderate: Humanity's Last Exam rises from 47% to 52%, CritPt from 23% to 25%, and SciCode from 59% to 61%. AA-Omniscience Index rises from 20 to 30, as the hallucination rate falls from 53% to 44% and accuracy rises from 48% to 51%.
➤ Cost per Task roughly equal to Kimi K3 at $1.98 per Task: K3.1 Pro uses 55K output tokens per Intelligence Index task, 14% more than Kimi K3 at 48K, while its first-party token prices are 33% lower.
Additional model details:
➤ Context window: 1M tokens.
➤ Size: 2.8T total, 104B active parameters
➤ Input modalities: Text and image
➤ Weights and license: Weights currently not published, Moonshot is expected to release the weights soon after launch
➤ Pricing: $2/$10/$0.20 per 1M input/output/cache hit tokens, 33% lower than Kimi K3 on all three rates.
Congratulations to the team at @Kimi_Moonshot on the release!
I agree with Lucas. I don't know why there is so much hype here. I just think this is what happens when you enter the machine learning field without diving deep into math.
On the zeroth order lm training stuff:
It's cool that people try stuff instead of only reading and dooming. But i feel the need to curb the enthusiasm on this one a little. Two things:
(1) methods in the genetic-algo and fwd-diff categories, which imo these fall in, is established that they are fine for smaller parameter counts but fall off hard as parameter space increases. It's not necessarily raw param count that matters here but "intrinsic" size. This is the whole point of backprop: few outputs many params.
(2) loss 4.2 for LM is very high even for a 30M parameter model. Go a bit higher and you are at learning token unigram distribution. This means for this score not much interesting/difficult learning has to happen yet.
Again it's fine to rediscover things like this from first principles. In fact it's the very best way to learn and i recommend it; i spent a lot of time learning like this myself! But now with social media hype amplification and crazy vcs and journalist attention that our field gets, i do feel the need to be a bit more cautious with communication and there to be "old pessimist" voice like mine in the room to.
(But unlike some other old pessimist voices, if i turn out to be wrong I'll happily admit it and cheer on =)
Jiaxuan Zou retweeted
Today, we release our first model, Flint-40B-A2B, a fast local coding model built with our automated research system. huggingface.co/RecursiveSI/f…
1. Strong performance. 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2, and 74.2% on SWE-bench Verified.
2. It is trained from scratch (tokenizer->Pretrain->Midtrain->SFT->RL).
3. Fast in your local laptop. Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark.
4. We build the complete model training pipeline in 4 month!
Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine.
• Best-in-class fast local coding
• Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
• Built from scratch in under 4 months
Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.
Jiaxuan Zou retweeted
1/4 Today, we’re releasing Mercury 3. Mercury 3 is by far our most powerful model and the most intelligent diffusion LLM ever released. It outperforms GPT-6 Luna and Haiku 5.5 on quality while running at 1000+ tok/sec.
🤖 Made with AI
Some say that when the compute exceeds 1e22, the vision encoder is no longer needed
Vision is where my research journey began. Although Vision Encoders seem to be receiving less and less attention in today's large model research, I still believe there are plenty of interesting questions worth exploring.
So, I recently spent some time putting together a systematic overview of the Vision Encoders used in today's frontier models.
From SigLIP 2 and Perception Encoder to Encoder-Free, I compared the publicly available details of 9 models, covering their visual architectures, spatiotemporal compression, positional embeddings, training recipes, and some interesting Infra optimizations.
Interestingly, putting these notes together also made me reflect on how my own research taste has evolved.
Over the past six months, the core thread of my research has consistently been "removing inductive bias as much as possible to let the model architecture take its natural and intended form." I believe this will also be the guiding principle throughout my future academic career.
If we can remove a module while still allowing the model to learn the capabilities it was originally designed to provide, then why not?
The English version was translated from the Chinese original by GPT-6 Astra. If you can read Chinese, I recommend reading the original notes.
CN: github.com/Purshow/Purshow_N…
EN: github.com/Purshow/Purshow_N…
Jiaxuan Zou retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Jiaxuan Zou retweeted
life update: I've recently left @Kimi_Moonshot!
The past year at Kimi has been the best of my life so far. I'm deeply grateful to have worked with and learned from such amazing and supportive colleagues.
Some of my favorite memories include (in chronological order): optimizing KDA's kernels and trying every design possible with @yzhang_cs; coming up with and refining Attention Residuals with @yzhang_cs @Jianlin_S; working on optimizers, which by far taught me the most about training dynamics (and where I pulled the most all-nighters); creating the cases for K3's release with @yzhang_cs @Xinyu2ML while watching the World Cup; and more, and more.
Quoting my earlier tweet about Kimi K3’s release: it is truly incredible to have built such an open (and strong) model with a group of genuine people.
My days at Kimi are memories I'll forever cherish... thanks for everything! (especially @Yulun_Du for believing in and hiring a 16yo back then :P).
More things to explore :) 🫡
Jiaxuan Zou retweeted
DL theory very hard. Try DL alchemy instead!
Deep learning is often taught as a sanitized collection of facts and math. But work at the frontier of AI is messy and empirical.
Our new class, Deep Learning Alchemy, aims to teach the art and science of deep learning in the hands-on spirit of Language Modeling From Scratch.
Students build intuition by running experiments on small LMs, then test it through prediction quizzes: given an unseen code diff, can you predict what happens?
deep-learning-alchemy.github…
My co-instructor @kothasuhas and I think there's significant demand for classes that treat students as serious learners who want to be assessed for what they’ve learned, and so we're trying a new exam-heavy approach to the class (7 exams, 85% of the grade, with the remaining 15% in-person discussion assessments).
We’ve designed the class so students can use AI agents to run their experiments and do their assignment, but they then have to demonstrate that mastery on exams without access to the assignment, and any partial credit on the exam has to come through citations to the assignment, incentivizing them to know what’s in their write-up.
The same assignments and quizzes are public for anyone to try (lecture recordings will be released through @StanfordOnline with some delay). We have some very fun assignments and quizzes covering both classic topics (like NQM, run-to-run stochasticity, and scaling laws) and new and timely topics like muP or hyperball. Made by some real experts like @TristanThrush and @wen_kaiyue
Try some quizzes below!
Haha, you can now book a coffee chat with me on my homepage!😘😘
jiaxuanzou0714.github.io/cof…
1/5
New blog: How to Build a Scientific Scaling Ladder.
I've spent the past three months studying pretraining and scaling. This guide brings together public research on using small runs to predict large-model training configurations and performance.
jiaxuanzou0714.github.io/en/…
4/5
The guide covers dense/MoE scaling, LR/batch size/weight decay, data mixtures and repetition, loss fitting, and downstream prediction.
Keep measurement conventions consistent and validate transfer to the target scale. Better small-model loss needs downstream validation.
