
co-founder @avomainc; occasionally tweets about ๐ & ๐ถ
the Bay
Joined November 2008
- Tweets8.5K
- Following463
- Followers586
- Likes1.7K
Pinned Tweet
I just published โThe Touristโs Guide to Deep Learningโ medium.com/@albertlai/the-toโฆ
If Michael B Jordan wins a few more Oscars, then the Michael Jordans of basketball, machine learning, and acting would all be named Michael Jordan
There is a CS / stats professor at Berkeley called Michael I. Jordan who has an h index of 200.
So the Michael Jordan of machine learning is a guy named Michael Jordan.
114
Hard to describe, but seeing someone like @TheStalwart discover the unique joy of watching a loss curve start to fall is kind of like seeing a friend really get into your favorite band
Real talk. Vibecoding is fun, and you should try it. But vibemodeling has been one of the most educational and intellectually satisfying things I've done in awhile. All kinds concepts and statistical properties you hear about suddenly become much clearer.
2
111
Albert Lai retweeted
NEW: In a court filing this evening, the Trump administration said that it had mistakenly deported a Maryland father to a notorious Salvadoran prison due to an "administrative error."
๐
This is really a 'WOW' paper. ๐คฏ
Claims that MatMul operations can be completely eliminated from LLMs while maintaining strong performance at billion-parameter scales and by utilizing an optimized kernel during inference, their modelโs memory consumption can be reduced by more than 10ร compared to unoptimized models. ๐คฏ
'Scalable MatMul-free Language Modeling'
Concludes that it is possible to create the first scalable MatMul-free LLM that achieves performance on par with state-of-the-art Transformers at billion-parameter scales.
๐ The proposed MatMul-free LLM replaces MatMul operations in dense layers with ternary accumulations using weights constrained to {-1, 0, +1}. This reduces computational cost and memory utilization while preserving network expressiveness.
๐ To remove MatMul from self-attention, the Gated Recurrent Unit (GRU) is optimized to rely solely on element-wise products, creating the MatMul-free Linear GRU (MLGRU) token mixer. The MLGRU simplifies the GRU by removing hidden-state related weights, enabling parallel computation, and replacing remaining weights with ternary matrices.
๐ For MatMul-free channel mixing, the Gated Linear Unit (GLU) is adapted to use BitLinear layers with ternary weights, eliminating expensive MatMuls while maintaining effectiveness in mixing information across channels.
๐ The paper introduces a hardware-efficient fused BitLinear layer that optimizes RMSNorm and BitLinear operations. By fusing these operations and utilizing shared memory, training speed improves by 25.6% and memory consumption reduces by 61% over an unoptimized baseline.
๐ Experimental results show that the MatMul-free LLM achieves competitive performance compared to Transformer++ baselines on downstream tasks, with the performance gap narrowing as model size increases. The scaling law projections suggest MatMul-free LLM can outperform Transformer++ in efficiency and potentially in loss when scaled up.
๐ A custom FPGA accelerator is built to exploit the lightweight operations of the MatMul-free LLM. The accelerator processes billion-parameter scale models at 13W beyond human-readable throughput, demonstrating the potential for brain-like efficiency in future lightweight LLMs.
1
169
Albert Lai retweeted
Replying to @MasoudMaani
Actually I am very sympathetic to this. Back to my example of self-driving, my first demo drive in an early Waymo was 2014, and it was already great. It took me around for a 20min drive. From that ~perfect demo it was one decade before I could pay for a drive in a Waymo.
๐๐๐
amazing they were able to use the NASA lidar as a backup!
Your order was deliveredโฆ to the Moon! ๐ฆ
@Int_Machines' uncrewed lunar lander landed at 6:23pm ET (2323 UTC), bringing NASA science to the Moon's surface. These instruments will prepare us for future human exploration of the Moon under #Artemis.
Touchdown Confirmed!
1
157
No matter how old it the dataset is, I will always chuckle when I see HellaSwag
Replying to @JeffDean
On a wide variety of text, math, coding and reasoning benchmarks, including MMLU, GSM8K, MATH, Big-Bench Hard, HumanEval, Natural2Code, DROP, and WMT23, Gemini outperforms all other models and improves on the state-of-the-art. On MMLU, Gemini Ultra is the first model to achieve human-expert performance on MMLU across 57 subjects with a score above 90%.
2
196
The 2025 Pac-2 schedule:
Cal @ Stanford
Stanford @ Cal
Cal @ Stanford
Stanford @ Cal
Cal @ Stanford (league of legends)
Stanford @ Cal (melee)
Pac-2 Championship Game played by robots, CS188 vs CS221
1
3
413






















