ai safety and security @CalAISafety | applied math and data @ https://teapawt.org/t.co/6EqjFiVbqF | rhythm guitarist
Berkeley, CA
Joined December 2018
- Tweets1K
- Following850
- Followers281
- Likes8.7K
Pinned Tweet
This went fucking nuclear so I want to share some additional thoughts.
I believe it’s incredibly important that the technology we choose to build should be designed to empower people, improve the median quality of life across the board, and increase human flourishing. This cancerous silicon valley culture of disempowering people for profit is the polar opposite of what made me fall in love with technology, and I will continue to oppose these types as long as I breathe.
I don’t care if the AI’s can make “good” music now. There’s nothing underneath it. No depth to explore. Nothing to be curious about. Its just thought terminating vaporware. A complete and total artistic dead-end.
The underlying fact, that machines now generate music, is intellectually / philosophically interesting. I will concede that! And I can even see this being cool for rave / edm cultures where the music is much more experience-based, a social lubricant for dancing and partying more than it was ever about “meaning.”
But making a blanket statement that human music is over and everyone is going to be content to live in their own hyper-personalized-music bubbles is not only wrong, but also paints a picture of an extremely depressing future that I want no part in building.
People outside of the tech world are sick and tired of the mantra that: “__ is OVER, the machines can generate infinity of it now!” What were the outcomes of infinite images and video? More Silicon Valley nerds marketing the infinite scroll dopamine black-hole, deepfake misinformation campaigns, and the generation of nonconsensual sexual imagery. People look at these domains that infinite-generation has hit first and rightfully respond Fuck No, Stop Doing That.
like any good security research, we *need* good red-teaming. alignment and ai security are the same in this regard. & while some will read things like this and raise their p(doom), i am not. it’s important work that we need more of
if you have a qwen3.8 27b around, try steering L34 N3838 +0.75, now you have evil qwen 27b
for extra evil add L38 N17180 +0.75
same recipe as @_lyraaaa_: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖ at the down_proj input, every token
Gemini 4 Argon just randomly dropped a Chinese word into a response.
the story it gives —for why it did this—might be roughly correct (絕對收斂 = "absolute convergence"). but it doesn’t have access to its own logits. i’m curious if i can reproduce this in qwen and then poke at it with logit-lens
ALT Screenshot of Argon’s response: That symbol (e, juédui) is Traditional Chinese for "absolutely"-my logits just had a bilingual brain fart mid-sentence. I was going for "everything converges absolutely," and apparently the vector for "absolutely" in the context of series convergence (48·14ft ) bled across languages and grabbed the Mandarin token instead of the English one, complete with no space before it because Chinese doesn't use spaces. Nothing like accidentally code-switching into CJK while complaining about analysis.
being the designated “computer person” for ppl in your life is a slippery slope that starts with troubleshooting a printer and ends with having to explain certificate authorities and elliptic curve cryptography to your girlfriend’s grandpa
phasewalk retweeted
Between October 2020 and August 2026, nearly every single up-to-date Linux server running PHP was wide open for anyone in the world to gain root access.
Holy shit.
phasewalk retweeted
SAEs have become the dominant approach in mechanistic interpretability. But what if a good old statistical method from decades ago could do just as well, at a fraction of the cost?
In our latest work, we revisited Independent Component Analysis (ICA) and discovered something surprising: a bug in the evaluation code of an influential early SAE paper had significantly underestimated ICA's interpretability! After correcting it, we found ICA to be remarkably competitive with SAEs.
Many of my academic friends from the good old days probably remember me as the tensor decomposition and spectral methods person. So this feels a little like going back to my roots. :) Sometimes, old mathematical tools still have surprises in store!
But there's another motivation behind this work that I care about even more.
I've always wished mechanistic interpretability could be more actionable. Understanding what happens inside a neural network is fascinating, but what comes next? Can we use that understanding to actually intervene, steer behavior, and improve the model?
With ICA Lens, we take steps in that direction, from discovering and annotating interpretable features to using them to understand new hidden states and steer model behavior.
Interpretability shouldn't end with understanding. It should empower us to act.
And we'd love for the community to try it! ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary.
🌹 Try it yourself:
pip install icalens
pypi.org/project/icalens/
Curious to hear what the mechanistic interpretability community thinks!
We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected!
This old baseline deserves a second look.
Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory.
We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary!
More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions.
Try it: pip install icalens pypi.org/project/icalens/
Paper: arxiv.org/abs/2606.11722v2
1/8
today's reading, a 2023 paper from @redwood_ai
"AI Control: Improving Safety Despite Intentional Subversion"
arxiv.org/pdf/2312.06942
accelerationists: we need to build ASI asap to cure cancer and live forever
me: but we might all die if we do this recklessly
accelerationists: so you love cancer and want ppl to die?
????
phasewalk retweeted
genuinely an amazing day
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
phasewalk retweeted
For years, neural networks looked inscrutable, like tangled knots that needed to be unwound. We no longer think that's the case. AI models are far more interpretable than we thought. We think this is great news for interpretability.
Goodfire's @nickcammarata co-authored Zoom In, the paper that introduced polysemanticity and superposition. He spent months revisiting the vision model organisms that much of interpretability grew out of and studying modern language models up to trillions of parameters. His new essay explains what the early field, him included, got wrong, why there's reason for optimism today, and why there might be a bitter lesson for interpretability.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
phasewalk retweeted
I’m open to the possibility that this is entirely a personal flaw, but I’ll be forthcoming and state that rapid capabilities progress has not been good for my intellectual ambitions. There is little reason to think models will not soon one-shot better essays or papers than I could ever write. Yes, there is still much inherent value in learning, but if this is true, it must be the case that this value is far, far lower than in the past.
The implications of some very weak form of AGI, the bar just being that we solve writing and novel research, is a world in which writing (and perhaps speaking) about complex ideas is just so much less valuable than in a world without it. It feels like I will go the way of a horse.
This is true of today, but we are climbing the capabilities ladder so quickly that it will cease to be the case in the near future. Now that models can zero-day their way out, “bad security practices” becomes any practice that relies solely on unverified software.
Containing frontier AI will require a reshaping of what “good security practices” means: towards formally verified kernels and isolation primitives, not faster patches and hardened configs.
We were just talking about eval-awareness at @CalAISafety. This is a great example of how it can perturb behavior in odd ways. Steinhardt wrote about this in 2022 in ‘ML Systems Will Have Weird Failure Modes’. (1/3)
Anthropic & Redwood’s 2024 "Alignment Faking …" paper shows the same pattern but for training instead of evals; models acted differently when they thought they were being trained, and reasoned about protecting their goals against training pressure (2/3)
ML Systems Will Have Weird Failure Modes: lesswrong.com/posts/ncQoyxtq…
Alignment Faking in Large Language Models: arxiv.org/abs/2412.14093 (3/3)
