@anthropic Prev: @ETH, @UniofOxford, intern @DeepMind
Zürich
Joined January 2012
- Tweets73
- Following268
- Followers820
- Likes1.7K
Especially proud today to be part of the anthropic team
A statement on the comments from Secretary of War Pete Hegseth.
anthropic.com/news/statement…
Bobby retweeted
Introducing Claude Sonnet 4.5—the best coding model in the world.
It's the strongest model for building complex agents. It's the best model at using computers. And it shows substantial gains on tests of reasoning and math.
ALT A benchmark table for Claude Sonnet 4.5 showing leading performance across many domains, including agentic coding, computer use, mathematics, graduate-level reasoning, and financial analysis.
Bobby retweeted
Is equivariance necessary for a good 3D molecule generative model? Check out our #icml2025 paper, which closes the performance gap between non-equivariant and equivariant diffusion models via rotational alignment, while also being more efficient (1/7):
arxiv.org/abs/2506.10186
Bobby retweeted
🚀 TOMORROW afternoon at ICLR: Learn about the directionality of optimization trajectories in neural nets and how it inspires a potential way to make LLM pretraining more efficient ♻️
(Poster# 585, hall 2b)
Bobby retweeted
🚨 NEW PAPER DROP!
Wouldn't it be nice if LLMs could spot and correct their own mistakes? And what if we could do so directly from pre-training, without any SFT or RL?
We present a new class of discrete diffusion models, called GIDD, that are able to do just that: 🧵1/12
Bobby retweeted
A really fun project to work on. Looking at these plots side-by-side still amazes me! How well can **convex optimization theory** match actual LLM runs?
My favorite points of our paper on the agreement for LR schedules in theory and practice: 1/n
Learning rate schedules seem mysterious?
Turns out that their behaviour can be described with a bound from *convex, nonsmooth* optimization.
Short thread on our latest paper 🚇
arxiv.org/abs/2501.18965
Bobby retweeted
BPE is a greedy method to find a tokeniser which maximises compression! Why don't we try to find properly optimal tokenisers instead? Well, it seems this is a very difficult—in fact, NP-complete—problem!🤯
New paper + P. Whittington, @GregorBachmann1 :) arxiv.org/abs/2412.15210
Come by poster #2402 East hall at NeurIPS from 11am-2pm Friday to chat about why outlier features emerge during training and how we can prevent them!
Updated camera ready arxiv.org/abs/2405.19279. New results include:
- non-diagonal preconditioners (SOAP/Shampoo) minimise OFs compared to diagonal (Adam/AdaFactor)
- Scaling to 7B params
- showing our methods to reduce OFs translate to PTQ int8 quantisation ease.
Check it out!
Updated camera ready arxiv.org/abs/2405.19279. New results include:
- non-diagonal preconditioners (SOAP/Shampoo) minimise OFs compared to diagonal (Adam/AdaFactor)
- Scaling to 7B params
- showing our methods to reduce OFs translate to PTQ int8 quantisation ease.
Check it out!
Outlier Features (OFs) aka “neurons with big features” emerge in standard transformer training & prevent benefits of quantisation🥲but why do OFs appear & which design choices minimise them?
Our new work (+@lorenzo_noci @DanielePaliotta @ImanolSchlag T. Hofmann) takes a look👀🧵
Bobby retweeted
Tuesday 1:30pm-3pm, Hall C 4-9 #515.
Drop by our poster if you are interested in SSMs for graphs👇!
Code: github.com/skeletondyh/GRED
Heading to Vienna tomorrow for ICML!
Broke up the train journey to catch a concert celebrating the 200th birth year of Bruckner, near the stunning Attersee 🏞
Looking forward to catching up with old friends and meeting new ones next week 😊
Bobby retweeted
Ever wondered how the optimization trajectories are like when training neural nets & LLMs🤔? Do they contain a lot of twists 💃 and turns, or does the direction largely remain the same🛣️? We explore this in our work for LLMs (upto 12B params) + ResNets on ImageNet.
Key findings👇
Outlier Features (OFs) aka “neurons with big features” emerge in standard transformer training & prevent benefits of quantisation🥲but why do OFs appear & which design choices minimise them?
Our new work (+@lorenzo_noci @DanielePaliotta @ImanolSchlag T. Hofmann) takes a look👀🧵
We explore why in the appendix and find sub-leading order terms to be the driving factor in increasing feature kurtosis (hence OFs). 9/10
Overall, we highlight key arch/opt choices to minimise OFs, shedding new light on this important facet of NN training dynamics. Many more results/exps in paper e.g. this ablation from standard Pre-Norm to OP one norm at a time.
Check it out! arxiv.org/abs/2405.19279 🙂 10/10
