@mayfer
 Vancouver, BC
muratayfer.com
Joined March 2008
  • Tweets16.2K
  • Following9.6K
  • Followers20.3K
1.4KMedia
i built an embedding engine by composing vectors from common abstraction neuron activations (h/t @nickcammarata "already clean" essay) and it behaves more consistently especially when doing embedding math. less distracted by spelling/formatting and more focus on abstractions present it's not always better, but generally at least on par. i have not seen it breakdown into returning junk like regular embedding models do yet. i am gonna try running some vector search evals on it to see how it performs
4
16
1,217
reproduced very well. qwen 4b forgot facts but can still do reasoning (this is the 99% variant)
Kaley Brauer@kaleybrauerOct 20
Tested this! I flattened every neuron's weakest activations Overall loss barely changes at first but the models get REAL bad at memorized text. So the weak tail does, as Neel expected, contain memorized things. Weaker activations seem to correspond to less generalizable content
1
5
733
im going, come have a pint or four
deepfates.delve.town
@deepfatesOct 20
OFFICE HOURS SAN FRANCISCO WIZARD OF FRIENDLINESS EDITION!!! Drink booze! Eat food! Talk about that one movie you saw! Zeitgeist, after 6. Look for the duck! MEET YOUR MUTUALS! (AND EVERYONE ELSE) YOU'RE YOUR OWN WIZARD
1
1
5
1,302
cammarata skeptics in complete shambles as they ramble with theory and composure while people keep applying the ideas in the essay to real work with undeniable results within days. and this isn't even the only one
Emanuele Rodolà
@EmanueleRodolaOct 19
forward-only pretraining now matches backprop on the Noise benchmark. Relay: val 3.704 in 2.5 min on 8×H100. backprop with the same model, data and optimizer: 3.712. the current record, Boost→Cut: 3.945 in 7.6 min. no autograd anywhere. 🧵
8
8
297
16,152
Inspired by Nick Cammarata’s interpretability essay “Neural Networks Are Already Clean”, I turned a couple of its ideas into speedups for pretraining without backpropagation. Still in the works. 2.3x speedup, cool xitter.cf/industriaalist/status/…
Samip
@industriaalistOct 12
Replacing backprop with zeroth-order optimization is inevitable. Introducing Noise: a speedrun for pretraining without backprop. Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top of it is possible, and it will require *co-designing* credit assignment, optimizers, architectures, and eventually, hardware. We built Noise to iterate on this faster.
28
71
20
1,303
161,236

No more items