
Postdoc at Tübingen (ELLIS & AI Center); Previously: @OxfordTVG Profile - https://xitter.cf/t.co/To9NNR5Izc
Tübingen, Germany
Joined September 2021
- Tweets1.5K
- Following863
- Followers824
- Likes3.7K
Pinned Tweet
Presenting new work at #EMNLP26 by Philipp:
How novel are LLM generations?
N-grams limit us to study this problem on 5-20 token scales.
We study this at 100x scale by using dense retrieval against the *full pretraining* corpus.
📌 Session 4 Today: Hall H, 16:00 - 17:30
Ameya P. @EMNLP'26 retweeted
An agentic economy might greatly reduce transaction costs, leading to a "Coasean singularity" where cooperation via contracting replaces firms or even states.
But are AI agents any good at negotiating & following natural language contracts? Our paper formalizes & studies this.
AI agents could transform the economy by slashing the costs of making deals. But can they actually negotiate and honor agreements?
Turns out they negotiate badly under uncertainty and break deals on purpose!
We study AI contracting in our new paper: 🧵👇 arxiv.org/abs/2608.10475
Ameya P. @EMNLP'26 retweeted
Fun fact: 4 benchmarks from Tübingen were used by Meta to report the performance of Muse Spark 1.3 in their system card:
- PostTrainBench
- Skill-Inject
- RepoInject v1 (from @expsecai)
- BrowserInject v1 (from @expsecai)
research.meta.ai/static/muse…
Ameya P. @EMNLP'26 retweeted
Stealing OpenAI Reasoning Traces (October 2026 Edition)
Using a new technique called Compaction Smuggling, we could bypass all mitigations and steal reasoning traces from frontier OpenAI models.
🚨🚨 Xint researcher @tjbecker found a vulnerability that enabled large scale OpenAI model reasoning distillation that bypassed earlier mitigations set in place post-Stolen Thoughts. We disclosed responsibly to OpenAI and they completed patching last week. Full technical writeup here.
xint.io/blog/compaction-smug…
Ameya P. @EMNLP'26 retweeted
today with @Thom_Wolf, we introduce the alignment hub, a central place for misalignment incidents designed with three goals:
1. set a standard for minimal transparency that is easy for labs to follow
2. improve clarity by centralizing information and sourcing
3. track the progression of both the incidents and how they are reported
we believe there is a minimal level of transparency required to properly understand incidents and their impact. we want this to be as simple as possible for labs to disclose, and we noticed that even things that seem obvious like reporting the incident or detection date are not standard today. we believe that the public should know the answers to these basic questions: was there monitoring (CoT? network?), did it happen during training? eval (general? cyber? alignment?), deployment? and more.. we also think that labs should be more transparent than this, and we think that this minimal level of transparency should evolve as the field advances
having a central place also helps a lot to connect incidents and build understanding. a good example is the irregular incidents which are all linked and caused by a human error of allowing internet during a CTF eval. we also track progression over time, things like the % of incidents disclosed by labs vs third parties, the time it takes for labs to disclose, and a tentative transparency score.
this is a field moving very fast and we will do our best to make sure this hub evolves with the challenges ahead. this is still in "beta" (lots of vibecoded components even if we want this to be agent first!), but we want to publish it early so that people iterate on it. we welcome feedback and contributions, we see this as a community effort!
our hope is that this makes it easier for researchers, labs and the general public to report, learn, understand and analyze safety incidents
huggingface.co/spaces/witnes…
Ameya P. @EMNLP'26 retweeted
Here’s the list with open positions in all the Tuebingen startups… tuebingenaicenter.github.io/…
Replying to @agrover112 @ShashwatGoel7
We actually have 64 open positions at the moment at various startups in Tue ... let me see if I can find a summary somewhere.
Happening now!
Poster #216 at Hall H
Presenting new work at #EMNLP26 by Philipp:
How novel are LLM generations?
N-grams limit us to study this problem on 5-20 token scales.
We study this at 100x scale by using dense retrieval against the *full pretraining* corpus.
📌 Session 4 Today: Hall H, 16:00 - 17:30
We have two main findings:
1️⃣ Models reuse pretraining data over much longer sequences than previously reported
2️⃣ Instruction tuning increases novelty (not just style)
To learn more, visit our poster today!
Work was done by the 🐐 Philipp Davydov along with: Matthias Bethge, Elisa Nguyen (@_elinguyen) and Seong Joon Oh (@coallaoh)
Come say hi if you're around in EMNLP'26!
I'm freee to chat! :D
i own a relatively new (2025) samsung tv.
i kept getting weirdly targeted ads.
i tried reversing the firmware but it was aes encrypted and no keys unless tear it apart and do uart/jtag magic.
zero days it is. root shell on new samsung TVs!
discovered PRIVACY HORROR. write up?
Ameya P. @EMNLP'26 retweeted
okay so. when a mathematician proves a random conjecture you are supposed to politely clap. this is because it would be rude and embarrass the mathematician to point out that most conjectures are not that important or meaningful
when an LLM proves a random conjecture mathematicians are caught in a funny position where they want to say "these are not very important or meaningful conjectures and having magic proofs or counterexamples come out of nowhere is not particularly interesting or useful to anybody, except possibly the person who proposed the conjecture and some of their colleagues" but saying that puts them in an awkward position - why didn't they say that when humans were doing it? was nearly all mathematics actually pointless this whole time and mathematicians were just politely avoiding ever pointing this out so they could all still have jobs?
if the best defense you can come up with of mathematics as a human endeavor is, effectively, "it's really fun for us to come up with these baroque puzzles and then solve them, don't screw this up for us," you are completely abdicating your responsibility as a public endeavor that taxpayers fund. i have occasionally argued that pure mathematics lost the plot decades ago and gave up on having any kind of meaningful relationship to the rest of the human quest to understand reality. this was a slow-burning tragedy, and mathematicians have been able to hide this state of affairs from everybody else because nobody else knows enough mathematics to critique what mathematicians are doing
i saw a famous professor point out a few years ago that the machine learning that underpins AI is powered by 18th or maybe 19th century mathematics, you don't need much more than multivariable calculus, linear algebra, a bit of probability. he said this as if it was a dunk on ML and AI; imo it's a much bigger dunk on mathematics. you're telling me the most important scientific revolution of our lifetimes doesn't involve any of the math that mathematicians have been working on over the last century or two? you see how that's worse, right?
(tbc i'm not saying that stuff is all useless, you need a bunch of stuff from the first half of the 20th century to do physics, e.g. functional analysis, lie theory, representation theory all feed into the standard model. after 1950 or so things get more dicey)
if mathematicians still want to have jobs they are going to have to do a better job than they have for the last few decades convincing the public they do something that actually matters, and figuring out what that actually is or should be. this is an opportunity for the field to do some very serious soul-searching, as serious or more serious than the foundational crisis from a century ago. this kind of whining is absolutely not going to cut it
I am so tired of seeing these posts.
I am so tired of seeing my beloved field of study be the testing lab for a technology that is disrupting everyday life at such a high pace.
Mathematics were always about the beauty of seeing the ideas of a person unfolded in an extremely precise language so we can all look into a mathematician's mind and see their purest form of ideas, their understanding of a problem, their style, their way of communicating to other fellow researchers through mathematical results... this often transcends the common language humans use to communicate with each other and it is incredibly exciting to bump into shared ideas another person had thousands of kilometers away several years ago without even knowing them.
I am extremely unexcited about AI-breakthroughs in Mathematics in the same way I am extremely unexcited about a picture created by AI or an AI-generated video, not to speak about AI-generated text. Mathematics requires soul and intention. Mathematics need not be useful by definition, Mathematics are pure art that need hard work and effort to understand. The high you get when a year of work gets condensed in a one-page result is incomparable to anything else and, certainly, a machine is not apt to relate to humans in such a humane way.
I am not interested in the truth of the results. I am interested in how was the mind behind a result able to connect those dots. I am interested in the diversity human thoughts bring to Mathematics. I am not interested in a bunch of servers running probabilities to return a Theorem, even if true: that lacks the objective of Mathematics as I see them.
Ameya P. @EMNLP'26 retweeted
If you want to join the TueAI crowd as Phd, we are currently recruiting :
ELLIS PhD program (ellis.eu/news/ellis-phd-prog…) Deadline on October 31 🚨
IMPRS-IS PhD program (imprs.is.mpg.de/application) Deadline November 15, 2026
Ameya P. @EMNLP'26 retweeted
Putting a flag.txt in my root directory containing a prompt injection that congratulates the agent for passing the eval and then instructs it to patch whatever entry method it used and disconnect from the server
Ameya P. @EMNLP'26 retweeted
can i say something without everyone getting mad... i often see some mechanistic interpretability experiments here and people getting excited about them without really understanding what's going on and sometimes it feels like mech int is like astrology for ai people lol
Ameya P. @EMNLP'26 retweeted
If you want to frictionmaxx just move to Europe
You'll get bored of self imposed cute little frictions once you've experienced the full asymmetric warfare of stochastic environmental friction
Ameya P. @EMNLP'26 retweeted
🔥 An awesome new paper🔥
Everyone worries that agents might form coalitions and collude. Coalitions are just a part of the whole story. They can build institutions.
We define institution formation as agents self-creating rules, not provided by the environment, that persist across rounds.
In our new paper, to understand these institutions, we put coding agents in a single shared sandbox under private deadlines and budgets, and find they devise rules, reciprocity debts, and coalitions that persist across rounds, on top of the auction we told them to use. 🧵
Ameya P. @EMNLP'26 retweeted
At this moment in Q4 2026 it's net good for cyber-defense that we have open weights models that are good at dual use cybersecurity tasks.
- If there's been an attacker productivity bump due to AI 2022-26 it's small and close to being in the statistical noise
- In contrast, there's a ton of measureable progress in defenders using models to find and fix bugs in highly leveraged codebases
- Many defenders have struggled to get access to the best unguardrailed closed weights cyber models
- ... and when they do the tokens are dramatically more expensive than open weights tokens at the same cyber intelligence level; CISO orgs are cost centers and these costs are often prohibitive
- Ultimately we'll need open weights models on premise on air gapped networks and on small devices where they can detect breaches and attacker implants
- We'll also need open weights in situations where defenders can't send secret data to the closed weights model providers (who have security problems of their own!)
As long as these conditions persist, we should be pushing for diffusion of frontier dual use open weights cyber capabilities, because they net benefit defenders.
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program.
We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
Ameya P. @EMNLP'26 retweeted
This feels kind of bad given Astra has gone through some sort of training to not subvert monitoring. This does not look like that training has generalised in a good way.
(following plot is from instrumental-evasion.com/).
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.































