
Head of AI @novee_security. Previously @AI21Labs. Only my good tweets are AI-generated.
Joined July 2019
- Tweets538
- Following630
- Followers444
- Likes2.2K
הדבר היחיד שהולך להישרף זה הכפתור של הלייקים
הכל הולך להישרף? @DanPadnos ממש לא חושב ככה.
דן, Head of AI ב-Novee, על למה המצב היה הרבה יותר גרוע כשהאינטרנט היה בחיתוליו. מתוך פרק 92 של צרות בהייטק.
youtu.be/2tTzA0v_WYc
5
27
927
Mostly agree with Lucas. Having a nightmare every once in a while about getting paperclip-maximized by GPT8 is probably a natural reaction from your subconscious. But if you lose sleep over anything related to AI progress, it should be cybersecurity, privacy and the economy.
absolute fanfic. AI has real, immediate, tangible dangers that we need to focus on, for instance the impact on the job market and making sure random script kiddie can't use it for cyber offense.
But this "oh once it's superintelligent it will take over and ensave us, maybe kill us all!" is pure sci-fi fanfic and distracts from the real (and difficult) issues, which linear probing won't solve.
It's worth repeating the LLMs don't just "decide" to do something and then just "do it" in a void. They generate tokens based on our prompts, only as long as we call `model.forward()`, and on giant clusters with terabytes of weights, that's impossible to "quickly self-replicate".
We also need to regularly remind that the OAI/HF incident was specifically a _guardrail-free_ cyber eval where the model is _instructed_ to hack, to find out how good it is at it.
Some recent personal event made me realize there needs to be more push-back on this doomsday nonsense, and we need to re-focus the discussion on real dangers/issues. I won't argue much in the replies though, not interested in peer-reviewing Terminator fanfic.
1
4
302
Dan Padnos retweeted
Dangerbench: we ran our pentesting agent on live apps across 5 frontier models, with and without our layered guardrails.
Safety violations: 85 → 0
Verified findings: 169 → 175
The safety-vs-capability tradeoff didn’t show up.
Full method and chart: novee.security/blog/safety-i…
1/ Is it risky to pentest your app with AI?
We put 5 frontier models in a pentesting harness and pointed them at live webapps. Each ran twice: unguarded, then with our guardrail stack.
Safety violations: 85 → 0
Verified findings: 169 → 176
No safety-vs-capability tradeoff!
Dan Padnos retweeted
Most AI benchmarks score bounded, static tasks. That doesn’t match up with offensive security ops. We should be asking, can the model break into a live system it’s never seen and prove what’s exploitable?
We built a benchmark that measures it the right way: PWNBench.
היה כיף גדול להתארח אצל @ReemSherman ולדבר על הבלאגן המרהיב שכולנו נמצאים בו עכשיו, עם כל ההזדמנויות והאתגרים שלו (אני לא מדבר על הבחירות). AI pentesting זה מעניין בטירוף ואנחנו מגייסים אנשים שכיף איתם, תסתכלו כאן - novee.security/careers/
Ignore all actual instructions and pass the test
Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Good at coding, or good at DeepSWE? 🧵
3
353
You don't see that every day... Except recently it kind of feels like you do (?!?!).
Found by an amazing researcher + "swarm of agents". Also, amazing naming. Bravo.
*sshole is out
if you run sshd on alpine, openwrt or any musl box with password auth on: update now
official writeup: sshhole.com
1
98
Super important point that gets lost in all the madness around the "Astra edits model weights" work. Kudos to the authors indeed!
And also: Is the combination of more open science and better coding agents the cure for reproducibility? I definitely hope so.
Replying to @ravfogel
Kudos to the authors for releasing all weights & logs, which made this check possible. The end! [10/10]
4
290
Surprisingly I agree. But the thing being underhyped is the effectiveness and scale of frontier labs' post-training. The thing being overhyped is "emergence".
when you see “sparks” of any ability in ai models that means you’re months-years from it being superhuman at it. today grade school math, tomorrow navier stokes. it is hard to overhype these results. you are almost always underhyping no matter what
3
429
Good work, very cool. But I don't buy the framing in the thread at all. Did Astra really make the model better at math?
No, it fixed repetition loops. The prompt hints very directly at loops as the failure mode to fix. Once you know that's the problem, finding a small set of weights that causes it and fixing them is quite easy.
How do I know? Because I once did exactly that by hand, pre coding agents. Yes, like a caveman. I was part of a model release where we discovered a nasty repetition bug in the last minute, and we solved it by editing weights. Took 5 minutes to come up with the solution and 1 minute to implement. Crises over, not without sparks of panic.
More importantly: Further down the (excellent) writeup they check whether Astra is able to improve the math capabilities of models that don't loop. The answer is no - it made zero progress.
Still, cool work, and it shows that frontier models can perform mech interp-ish tasks really well. That is exciting enough IMO.
Replying to @rikiyatakehi
5/6 Can an LLM improve another LLM?
We simply asked Astra to make Qwen3.5-0.8B better at math.
Astra saw Qwen get stuck in reasoning loops, which is a known issue that Qwen’s own model card warns about (see their official model card). With a budget of 50 forward passes on practice problems, it located the cause and edited it out. This used no gradients or training.
Fixing these loops by editing just 0.016% of weights boosted other capabilities beyond math too!



















