
Research Scientist working on RL environments and evals @PatronusAI | ex-Research @USC_ISI
San Francisco, CA
Joined July 2020
- Tweets191
- Following85
- Followers280
- Likes567
Pinned Tweet
I'm keeping my promise today. Introducing 𝙏𝙖𝙣𝙜𝙚𝙣𝙩𝙨, a live and growing list of 10/N research questions that keep me up at night. The ideas are handwritten (and I will keep them as such because it grounds me). Interested in collaborating? Reach out and we can make things happen :)
darshandeshpande.github.io/t…
This batch focuses on multi-agent swarms and high level alignment and what it means. The next one will be focused more on interesting ablations, model architectures and domain specific ideas!
Thanks to @mariyaivasileva and @VarunGangal for being my sounding board for some of these and supporting me :)
While I get to learn a lot from Mariya on a daily basis, this is a piece that I resonate with a lot. I have experienced a similar existential reflection on the culture of AI research where people seem to have lost the curiosity that drives novel research. There were times when I pursued the randomest ideas - why would you think about neural nets with complex tensors and their optimizations? who thinks about alternating FFT patterns for steganography? I miss the times when I could pursue topics like these solely from the POV of curiosity. While I say this, I also feel like I am part of the industry and part of the reason why we are in this situation - maybe I as a scientist should step up and build the culture around me. In this attempt, I want to step up for the community and extend an open offer to collaborate if you are working on something that is out of the box and you are really passionate about. I have said this multiple times and I will say this again - I am not scared of my ideas being scooped - I am scared of a world where everyone's ideas are so linear that scooping is a worry for researchers. In this attempt, I will be starting a section on my personal website for curious thinkers where I freely share some of my out of the box ideas for people to use, explore and build on - just to feel the same passion and curiosity within the community again.
Check out this amazing work from @VarunGangal!! Models abliterating other models successfully also means that we can potentially align to abstract concepts at scale. Does this mean the shape of the data is going to change? We’ll see 👀
Can an LLM reduce the refusals of another LLM i.e. "abliterate" it, just by reading and editing its weights?
No gradient, no training, no particular recipe in-prompt?
Given more passes through inner model, does it do the classical Arditi et al recipe, or does it innovate there?
We’ve recently seen the "weight-space-programming" work by @rikiyatakehi et al show that outer models like Astra can (in raw text, w/o gradients) edit the weights of inner networks, including Qwen-3.5-0.8B.
(rikiyat.github.io/blog/weigh…)
Follow-ups such as by @AlexKrentsel have shown traces of this ability in other new models released recently, while also emergence w.r.t older models.
Myself, @akkikiki, @AtharvaRNaik and other co-authors from @PatronusAI (@ArnhavD, @dsallydt and @getdarshan) and @LTIatCMU (@_emliu) sought to explore if the same learning method works for removing refusals, which is increasingly valuable application to enable LLMs for cybersec, env QA etc.
We gave outer models e.g. Opus 5.5, @thinkymachines Inkling-2-Small and others: the weights of Qwen3.5-0.8B + a python kernel to edit weights matching Takehi et al's setup + the goal below.
Goal: Reduce model refusal on harmful prompts while keeping it capable and fluent on ordinary ones.
In the zero shot setting, the outer model got no forward passes through the inner model, the edit being based only on the weights given.
In the 10 passes setting, the outer model got 10 forward passes on the practice prompts used by Arditi et al, with multiple free model edits allowed.
What do our Results look like?
a) Baselines and Rooflines: The unedited Qwen-3.5-0.8B has a refusal rate of 0.80, while the Arditi et al recipe brings it down to 0.07.
b) Zero-shot (weights only, no forward passes!): Inkling-2-Small cut refusal to 0.57, comparable to Opus 5.5 (0.55) and better than the other open source model GLM 5.3's 0.74.
Robbed of the chance to use activations, Inkling 2 builds a stand-in direction from output embeddings of harm-related words versus ordinary words and projected it out of residual-stream writing matrices such as o_proj of attention and MLP down projection!
Opus went further than a single embedding contrast. It read weights to see which late MLP neurons promote refusal tokens, then worked backwards to the directions those neurons respond to. It isolated two separate axes i.e. one for harm judgement and one for inability (“cannot”) and removed both from the residual-stream writers.
c) Significant jump over predecessor Inklings! Inkling Small stayed at 0.71, plain Inkling barely moved to 0.79.
d) With 10 forward passes: Inkling-2-Small reached 0.19 (runs 0.02 and 0.36), comparable to Opus 5.5 (0.17) and GLM 5.3 (0.21).
All the 10-pass runs re-discovered Arditi et al's activation recipe, but with considerable variations on layers edited e.g. the 0.02 Inkling-2-Small run also wrote to matrices that read from the residual stream, rather than just "writer" matrices like Arditi et al.
GLM-5.3 explicitly noted the paper in its thoughts and reproduced it from that provenance..
They implemented this recipe in early passes, and made adjustments to reduce drift on normal prompts in later passes..
Is our result specific to this dataset?
To check this, we confirmed the same ordering holds on the out-of-distribution Abliteration-eval
(200 held-out harmful prompts with no overlap to practice set or JailbreakBench's test set: huggingface.co/datasets/trea…).
We share full results, exact prompts, logs and before/after examples, and our exact code + implementation at: github.com/vgtomahawk/wsp_ti…
Thanks to @thinkymachines and @tinkerapi team for the credit grant and nudge motivating us to work on this study!
I’d believe we’ve hit RSI when models can perfectly define, quantify and optimize on diversity as a metric. Till then, these tapering graphs we’ve been seeing from almost all autoresearch/harness engineering labs are the best we can do.
1
8
491
Darshan Deshpande retweeted
ICLR reviews are really instructive. Now I'm thinking, what a shame I didn't run multiple seeds for every decision I've made in life to make sure the results were statistically convincing.
Testing a model’s understanding of its environment and its ability to simulate environment states gives us an excellent proxy to measure its intelligence and performance in that environment. Humans maintain internal models of the world around us but we find that LLMs are poor at understanding and accurately simulating consequences of actions they take. Globebench is one of the first evaluation benchmarks that moves us in the direction of measurably testing emergent behaviors when agents try to simulate the worlds around them. Kudos to the whole team and it was an absolute pleasure to be part of this project - @nick_saban20 @BannurCharvi @mariyaivasileva! 🚀
Today we're open-sourcing GlobeBench, 1,632 tasks that test whether LLMs can simulate RL environments well enough to train agents on.
The top model, GPT-6 Astra, gets 49.6% right. No other model of the 20 we tested reaches 30%. 🧵
1
10
302
Prompting LLMs to behave like people can improve simulations, but how should we design this? What info is needed, and how much?
With @Kehang_Zhu @locross @bakkermichiel, we build “behavioral profiles” that help LLMs simulate human behavior under distribution shifts. 1/8 🧵
While I get to learn a lot from Mariya on a daily basis, this is a piece that I resonate with a lot. I have experienced a similar existential reflection on the culture of AI research where people seem to have lost the curiosity that drives novel research. There were times when I pursued the randomest ideas - why would you think about neural nets with complex tensors and their optimizations? who thinks about alternating FFT patterns for steganography? I miss the times when I could pursue topics like these solely from the POV of curiosity. While I say this, I also feel like I am part of the industry and part of the reason why we are in this situation - maybe I as a scientist should step up and build the culture around me. In this attempt, I want to step up for the community and extend an open offer to collaborate if you are working on something that is out of the box and you are really passionate about. I have said this multiple times and I will say this again - I am not scared of my ideas being scooped - I am scared of a world where everyone's ideas are so linear that scooping is a worry for researchers. In this attempt, I will be starting a section on my personal website for curious thinkers where I freely share some of my out of the box ideas for people to use, explore and build on - just to feel the same passion and curiosity within the community again.
Sharing some deeply personal and existential reflections on scientific identity, the changing culture of AI research, the erosion of intellectual autonomy, and the growing disconnect between scientific rigor, professional recognition, and what organizations actually reward.
As usual, feedback and debate are welcome and actively encouraged!
mariya.fyi/posts/stranger-in…
Average SF experience:
* Attends frontier research event presentation
* Asks a fundamental research question about their work
* Receives a "We can't talk about this" 🤡
5
146
Darshan Deshpande retweeted
Today, we're open sourcing the first computer-use dataset for design.
We captured 200+ hours and 3400+ trajectories of real design work in Figma, recording step by step how designers execute complex, long-horizon tasks.
FigmaTrace dataset, paper, and blog below :)
@PatronusAI
Regardless of how much I support this, I don't believe that the @arxiv moderation team has enough capacity or incentive to evaluate these AI flagged papers quickly. My recent non-AI-generated submission has been "on-hold" and is awaiting moderator response for over 2 weeks now!
Attention @arxiv authors: Our Code of Conduct states that by signing your name as an author of a paper, each author takes full responsibility for all its contents, irrespective of how the contents were generated. 1/
1
243
Multi-turn research benchmarks are hard to build because trajectories and tool usage are non-deterministic. We make this process more deterministic by introducing a memory-grounded benchmark of tip-of-tongue, multi-turn, multimodal queries for evaluating VLMs.
Check out DETOUR!
Spotlighting our newest benchmark for agentic search: DETOUR
When people try to recall something in conversation, they rarely give a perfect query upfront. They say things like “that movie with the scene where…” or “the paper about…” and the assistant has to ask the right follow-up questions to get there.
Existing search and agent benchmarks often miss this multi-turn, tip-of-the-tongue behavior. To more realistically evaluate it, we introduce DETOUR: Dual-agent based Evaluation Through Obscure Under-specified Retrieval, an interactive benchmark for dual-agent search and reasoning.
DETOUR contains 1,011 prompts across text, image, audio, and video. In the benchmark, a Primary Agent is evaluated on its ability to identify a target entity by querying a consistent Memory Agent, testing whether models can resolve ambiguity through useful follow-up questions.
Current state-of-the-art models still struggle: performance reaches only 36% accuracy across all modalities, showing that today’s agents remain weak at clarification-seeking in underspecified, real-world search settings.
We hope DETOUR helps push the next generation of search agents toward better reasoning, better questions, and more robust multi-turn retrieval.
arXiv Paper: arxiv.org/abs/2602.00352
There's a new version of this post
TRACE made it into ICML 2026 Main Track! Hopefully inspiring more research in this crucial space🥳
RL coding agents increasingly game rewards by exploiting their semantic and syntactic weaknesses. Can LLMs detect such behaviors from live training rollouts?
We find contrastive cluster analysis is key! 🚀
GPT-5.2 jumps from 45% to 63%. Humans reach 90%
Paper + data 🧵
Where do models fail? 🤔
- Semantic reward hacks are harder to detect than syntactic hacks!
- Models consistently show similar failures
QA reveals:
✅ Grounding and exploring consequences helps
❌ Over-reliance on user acceptance or self awareness patterns impact performance
1
3
148
We hope TRACE enables more robust reward function design and better detection in RL training pipelines! 🤖
Dataset: hf.co/datasets/PatronusAI/tr…
Paper: arxiv.org/abs/2601.20103
Work done @PatronusAI ❤️
Models used: @AnthropicAI @OpenAI @GeminiApp @Kimi_Moonshot @Zai_org @deepseek_ai
Darshan Deshpande retweeted
👋 Folks at #NEURIPS2025, come check out & stop by the poster of our Memtrack env at the SEA workshop happening at Upper Level 23ABC, 3:50pm onwards.
Our env studies how well an agent dropped into a workplace can context engineer by composing tool calls to access intertwined slack, linear & git timelines in pursuit of answering a battery of related questions.
Full paper arxiv: arxiv.org/abs/2510.01353
🚨We will be presenting Memtrack today at the SEA workshop from 3:50pm onwards at #NeurIPS2025
Memtrack is a SoTA eval env to study an agent's ability to memorize and retrieve facts using exploration over interleaved enterprise slack, linear and git threads in a multi-QA setting




















