BerkeleyNLP (@BerkeleyNLP) header
BerkeleyNLP (@BerkeleyNLP) profile picture

We work on natural language processing, machine learning, linguistics, and deep learning. PIs: Dan Klein, @alsuhr, @sewon__min

Berkeley, California
Joined September 2019
Photo by BerkeleyNLP (@BerkeleyNLP)Photo by BerkeleyNLP (@BerkeleyNLP)Photo by BerkeleyNLP (@BerkeleyNLP)Photo by BerkeleyNLP (@BerkeleyNLP)Photo by BerkeleyNLP (@BerkeleyNLP)
BerkeleyNLP retweeted
Model competition is heating up, and distillation is 🔥🔥 Distillation detection is hard, so watermarking can help make it detectable. But are existing watermarks robust? We find two ways to break them: distill a model, then distill that model again; or distill only selected tokens that matter. Then what do we do? We introduce SubWM, which outperforms prior methods and is more robust to laundering.
Excited to present our new work on distillation detection: Subliminal Watermarks (SubWM). SubWM is much more robust compared to existing distillation detectors, is able to distinguish different watermarking keys, and also works reliably for out-of-domain prompts! Blog: subliminal-watermarks.github… 🧵
5
12
2
108
8,964
BerkeleyNLP retweeted
"No OCR, read text natively from images" continues! We showed this with deep research agents: it not only improves performance but also reduces tool calls! Post-training to work with page images gives a big boost - likely because these open-weight models haven't been trained to work with page images yet? Exciting work led by @Hughie_Hu43 !
(1/7) 🚀 Introducing DR Pixel, the first open-source deep research agent that searches and reasons directly over page images, no OCR. On 1.06M pages, our trained 27B model matches GPT-5.4. 📄 arxiv.org/abs/2611.03097 Joint work from @BerkeleyNLP, @rtnldynamics!
7
26
2
247
17,858
BerkeleyNLP retweeted
(1/7) 🚀 Introducing DR Pixel, the first open-source deep research agent that searches and reasons directly over page images, no OCR. On 1.06M pages, our trained 27B model matches GPT-5.4. 📄 arxiv.org/abs/2611.03097 Joint work from @BerkeleyNLP, @rtnldynamics!
13
39
7
193
28,636
BerkeleyNLP retweeted
I'm at #COLM2026 this week! 🌉 Lately I'm thinking about what's missing in harmful AI manipulation research and governance (whitepaper soon). Also on the job market for AI safety research roles, esp. threat modeling, strategy, evals, technical governance... DMs open :)
7
5
70
6,591
BerkeleyNLP retweeted
Long-context LMs seem great at tasks like RULER, but what about tasks like "find every contradiction in the given corpus"? - Performance suffers, especially as the input corpus scales (even with in-domain fine-tuning, not just zero-shot) - It changes a lot of core modeling decisions, e.g., whether to use hybrid models, whether to use block sparse attention TL;DR: if you are a model developer, include our CTC-Bench in your evals! ✨
Could long-context architectures “find all contradictions” in a science literature? Not yet! 🧵 We study a new class of "high-complexity” tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse attention, hybrid models…)
10
17
161
14,520
Could long-context architectures “find all contradictions” in a science literature? Not yet! 🧵 We study a new class of "high-complexity” tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse attention, hybrid models…)
15
70
8
340
61,364
BerkeleyNLP retweeted
I'll be speaking at the EMM-QA Workshop tomorrow (Saturday)! #icml2026 Come say hi and hear about PixelRAG -- doing RAG but without text (?) 👀 Don't miss @andylizf's poster as well🙌 pixelrag.ai arxiv.org/abs/2606.28344
🚀The ICML 2026 EMM-QA Workshop is happening tomorrow!🚀 Join us at 8:00 AM in ASEM Ballroom 201 at COEX. The program features four excellent invited talks, paper presentations, shared challenge highlights, and the live AI QA competition! Program: qanta-org.github.io/competit…
2
5
70
7,211
BerkeleyNLP retweeted
As task horizons grow, LLM contexts can’t scale forever. Self-summarization enables concise, interpretable contexts but at a significant performance cost. Our solution: isolate and *supervise* the information content of summaries in the form of natural-language belief states 🧵
4
14
1
40
5,672
BerkeleyNLP retweeted
One day I tried tracing all of Olmo's dependencies manually. A few hours later, I realized I can't do it and gave up. Then @sadhikesaven and @CoderBak ModSleuth 🔥 Turns out Olmo and Nemotron have hundreds of dependencies that are super deep, recursive, and not easily visible. I'm glad I gave up early 😅 Spoiler: I thought this would be a one-week Claude Code project. It was not. The hard part wasn't information extraction (which Claude Code is good at). The hard part was something much trickier. Check out the paper to learn more! (And yes, if a model release says it used Claude Code, ModSleuth will trace that too... which means the model depends on Claude Code, which has its own dependencies, and ModSleuth itself depends on Claude Code 🤯)
LLMs are no longer created w/ human data alone. They rely on other models to generate & filter data, evaluate outputs, & guide dev work. So what is a modern LLM built on? Olmo 3 → 89 model + 183 dataset dependencies; Nemotron 3 → 273 + 560 We made ModSleuth to trace this. 🧵
2
21
149
24,530
BerkeleyNLP retweeted
The web was never meant to be flattened into text. Yet most web RAG systems start by parsing HTML --- a complex and lossy process. 🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels. Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly. PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%. Why? (1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals. (2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever. (3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines. Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG. Demo below 👇 Code: github.com/StarTrail-org/Pix… Paper: github.com/StarTrail-org/Pix… Playground: pixelrag.ai/
24
119
12
715
92,626
BerkeleyNLP retweeted
Super excited about this work, led by @YichuanM and @andylizf ! It is possible to completely remove HTML parsing by directly retrieving and reading web screenshots through VLM. HTML parsing is a hidden bottleneck that causes significant complexity and information loss that nobody really pays attention to, and it's so exciting that VLM progress made it possible to remove it. Please check out this demo as well which is really cool!!! pixelrag.ai
The web was never meant to be flattened into text. Yet most web RAG systems start by parsing HTML --- a complex and lossy process. 🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels. Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly. PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%. Why? (1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals. (2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever. (3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines. Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG. Demo below 👇 Code: github.com/StarTrail-org/Pix… Paper: github.com/StarTrail-org/Pix… Playground: pixelrag.ai/
5
14
124
24,164
BerkeleyNLP retweeted
Really amazing results analyzing what's creative/novel vs. what's copied from Internet data, enabled by the amazing @liujc1998's Infini-gram! infini-gram.io This is also enabled in @allen_ai's OlmoTrace allenai.org/blog/olmotrace where anyone can find matching n-grams between LLM-generated text and its training data.
This from @TuhinChakr is brilliant. That prize winning story from Granta? Turns out it's just a bunch of random whole phrases taken directly from existing text on the internet. Tool allows you to trace those n-grams directly to their source, which is mostly random fanfiction. tuhinchakrabarty.substack.co…
1
10
1
83
17,589
BerkeleyNLP retweeted
1/ Thrilled to introduce T³: a corpus for RAG over reasoning tasks, built from thinking traces. We show that surprisingly RAG can improve reasoning— with the right corpus. Rag with Transformed Thinking Traces T³ gain by up to 43.9% on AIME 2025-2026. 🔗 arxiv.org/abs/2605.03344 🧵
11
29
4
211
482,066
8/ For reproducibility and to enable further study of modularity in MoEs, we’re releasing EMO, baselines, and code: Models: hf.co/collections/allenai/em… Blog: allenai.org/blog/emo Code: github.com/allenai/EMO Viz: emovisualization.netlify.app Shoutout to @AkshitaB93 @sewon__min for making this possible!
1
3
14
1,540
BerkeleyNLP retweeted
Full details + results: arxiv.org/abs/2605.06663 Also check this out emovisualization.netlify.app: Our model specializes qualitatively differently (capability-level rather than lexical) -- this emerged naturally even though we didn't expose any domain prior!
3
29
2,292
BerkeleyNLP retweeted
As MoEs grow larger and sparser, they become memory-bottlenecked. What if experts were actually composable - so you only keep the subset relevant to your task? We show that this doesn't emerge in standard MoEs (their training makes this hard), but you can pre-train MoEs to support this kind of modularity! I hope everyone sees the right figure from @RyanYixiang 's original post - I was so excited when I saw this result!!
MoEs are everywhere in frontier models, and they are deployed as a monolith system. But many applications only need a narrow slice of capabilities, e.g., math, code, biomedical, etc. So what if "modularity" is actually the missing opportunity for MoEs? Today, we're releasing EMO: an end-to-end pretrained MoE where modularity emerges naturally, enabling selective use of experts!
7
44
2
341
58,063
MoEs are everywhere in frontier models, and they are deployed as a monolith system. But many applications only need a narrow slice of capabilities, e.g., math, code, biomedical, etc. So what if "modularity" is actually the missing opportunity for MoEs? Today, we're releasing EMO: an end-to-end pretrained MoE where modularity emerges naturally, enabling selective use of experts!
Today we’re releasing EMO, a new mixture-of-experts (MoE) model trained so modular structure emerges directly from data without human-defined priors. EMO can use a small subset of its experts for a given task while keeping near full-model performance. 🧵
8
71
8
533
126,291
BerkeleyNLP retweeted
I will give two talks at ICLR workshops!! 🇧🇷 Sunday 9:40-10:10: "LLMs for Distributed Data Use" @ Workshop on Data Problems in Foundation Models (Room 203 A/B) Monday 15:30–16:05 : "Are Mixture-of-Experts Modular? Why It Matters and How to Fix It" @ ICBINB Workshop (Room 201 C) Both happened to be related to MoEs, but tackle two completely different questions → some say hi!
2
9
128
12,977
Imagine you fully post-trained "YourModel v1". Then, you've got better data — math, code, tool use, safety — and you want to improve it. Today, that usually means retraining the whole model. But what if new data could be added modularly, with a fixed cost each time?
Last year, we introduced FlexOlmo, a novel way to train parts of a model independently then combine them later. BAR builds on that idea for a harder problem: how to keep improving a model without having to retrain each time. 🧵
5
17
3
141
20,558
BerkeleyNLP retweeted
Exciting results on open-source modes for IMO-level problems - congratulations to @aviral_kumar2 and everyone involved!! Great to see @wenjie_ma's ProofGrader (proofgrader.github.io) integrated into the development ✨
We trained a tiny 4B model to reason for millions of tokens through IMO-level problems. Heaps excited to share our new blog post covering the full pipeline, from distilling the 🐳 to augmenting RL with a reasoning cache that unlocks extreme inference-time scaling for theorem proving. huggingface.co/spaces/lm-pro…
11
83
12,171