Lili (@lchen915) header
Lili (@lchen915) profile picture

Ph.D. student @mldcmu & visiting student @stanford. Previously undergrad @berkeley_ai

Joined February 2021
Photo by Lili (@lchen915)Photo by Lili (@lchen915)Photo by Lili (@lchen915)Photo by Lili (@lchen915)Photo by Lili (@lchen915)Photo by Lili (@lchen915)Photo by Lili (@lchen915)Photo by Lili (@lchen915)
Pinned Tweet
When LLMs donโ€™t do what we want, we often tell them exactly what/how to change. Ideally, models could learn from this feedback, which is much richer and denser than scalar rewards used for RL. In our new paper, we study how to expand the capabilities of RL via text feedback:
RL on LLMs inefficiently uses one scalar per rollout. But users regularly give much richer feedback: "make it formal," "step 3 is wrong." Can we train LLMs on this human-AI interaction? We introduce RL from Text Feedback, with 1) Self-Distillation; 2) Feedback Modeling (1/n) ๐Ÿงต
2
21
1
166
17,668
Lili retweeted
At #COLM2026 Tue-Sat. I will present our harness learning paper at the LLA workshop on Friday ( @ZAlvin39105 will deliver the oral). I'd love to chat about making RL more general and efficient (or anything about RL!), and meta-topics like how to proceed with research at school.
1
6
1
53
4,067
Images, video, audio, actions โ€” generative modeling has converged on one recipe: compress into continuous latents, generate in latent space, decode. Language is the lone exception: still generated token by token, as a long discrete stream. Latent Thought Flows: We compress 256 text tokens into 8 continuous latents, generate them with a one-step flow model, and read them out as text using an autoregressive decoder. Result: a better inference-compute vs. generation-quality Pareto frontier than a tuned autoregressive baseline โ€” thanks to compression and one-step generation. Co-led w/ @ZhengyangGeng ๐Ÿงต
14
49
6
369
83,129
Excited to announce our new work, Message Passing Enables Efficient Reasoning accepted at #COLM2026! Modern sub-agents are designed with fork-join style parallelism, where sub-agents are spawned for individual tasks and then their results are collated by a main thread. ๐Ÿงต1/n
2
18
6
64
15,192
A lot of excitement in synthetic data for science is motivated by ambitious uses of LLMs (e.g., running a survey with GPT before real participants, forecasting promising experiments with LLMs). We ask, what beliefs should the experimenter have to make this a good idea? ๐Ÿงต
3
26
3
59
12,698
Lili retweeted
I will be at ICML ๐Ÿ‡ฐ๐Ÿ‡ท from Mon to Sat, and present two posters and one oral (more in the thread). Look forward to connecting with friends and discussing RL research (LLMs, robotics, theory), self-improvement, and continual learning.
8
10
1
89
7,146
๐Ÿ”ญ ๐—˜๐˜…๐—ฝ๐—น๐—ผ๐—ฟ๐—ฎ๐˜๐—ถ๐—ผ๐—ป in the Era of ๐—š๐—ฒ๐—ป๐—ฒ๐—ฟ๐—ฎ๐˜๐—ถ๐˜ƒ๐—ฒ ๐—–๐—ผ๐—ป๐˜๐—ฟ๐—ผ๐—น ๐Ÿค– Interacting with the world can be expensive! Our #ICML2026 work shows how diffusion policies can ๐™š๐™ญ๐™ฅ๐™ก๐™ค๐™ง๐™š during online experience collection to achieve sample-efficient self-improvement! ๐Ÿ“ˆ
2
30
1
236
29,529
Lili retweeted
Exciting work! But in our February paper, "Reinforcement Learning with Text Feedback", we proposed the same methodology: predicting environment feedback on top of the RL loss. Nice to see this idea specialized to agentic terminal tasks, and the new insight this brings ๐Ÿ’ก. [1/2]
3
22
1
226
32,381
Proteo-R1 (ICML 2026), the first reasoning protein foundation model for protein design, is out! ๐Ÿš€๐Ÿงฌ Most protein design models generate structures without ever *reasoning* about which residues matter. We think that's backwards. Human protein engineers๐Ÿ‘ฉโ€๐Ÿ”ง don't work this way. They identify critical interaction residues first โ€” charged anchors, hydrophobic hotspots, specificity-determining motifs โ€” and only then optimize geometry around those decisions. โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” ๐Ÿ”ฌ THE CORE IDEA โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” A dual-expert architecture that explicitly decouples molecular understanding from geometric generation: โ†’ โšกA multimodal LLM (understanding expert) analyzes protein sequences, structures, and text to identify key functional residues governing binding and specificity โ†’ โšกA diffusion model (generation expert) then co-designs sequence + structure โ€” but with those residues locked in as hard constraints โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” ๐Ÿ“ HOW IT'S TRAINED โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” Three-stage curriculum: โ‘  Multimodal Alignment โ€” freeze the LLM, train projections to bridge ESM-2 + AF3-style structural features into language space โ‘ก Structural Reasoning Mid-Training โ€” unfreeze the LLM, teach it residue grounding โ†’ pairwise geometry โ†’ interface localization โ†’ hotspot prediction โ‘ข Joint Reasoning-Guided Design โ€” end-to-end on antibody-antigen complexes. Gradients from the diffusion objective flow back through the reasoning expert. โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” ๐Ÿ“Š RESULTS โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” Evaluated on simultaneous multi-CDR redesign and the RAbD CDR-H3 benchmark: โœ… Best RMSD & DockQ on RAbD โ€” redesigned H3 loops are geometrically accurate *and* docked well โœ… Lowest backbone dihedral divergence (JSDbb) among all baselines โœ… Reduced intra- and inter-chain steric clashes โœ… Generated sequences score lower perplexity than native antibodies under IgLM & AbLang โœ… Plug-and-play: swapping the diffusion backend to UniMoMo still improves RMSD and IMP โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” ๐Ÿ’ก WHY IT MATTERS โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” Proteo-R1 isn't just a better antibody design model. It's a blueprint for coupling deliberative LLM reasoning with any physical generative process โ€” interpretable, modular, and backend-agnostic. ๐Ÿ“„ Paper: arxiv.org/abs/2605.02937 ๐Ÿ’ป Code: github.com/smiles724/Proteo-โ€ฆ ๐ŸŒ Demo: smiles724.github.io/r1/ Great thanks to my wonderful collaborators Weihao Xuan, Heli Qi, @Hanqun_CAO, Heng-Jui Chang, @KKuanPang @XiangruTang Zehong Wang, @hcwww_ , @KejunYing @lupantech Chiho Im, Seungju Han, @richardxp888 @tikgiau. Also appreciate the guidance from advisors @YejinChoinka @jure @erranlli Naoto Yokoya, Masashi Sugiyama.
9
58
1
287
133,783
Lili retweeted
I will be at ICLR ๐Ÿ‡ง๐Ÿ‡ท this week and give some oral presentations and a panel discussion during the workshop days. Happy to discuss anything about RL (LLMs, robotics, theory).
1
4
1
45
3,806
Sharing a super simple, user-owned memory module we've been playing around: nanomem The basic idea is to treat memory as a pure intelligence problem: ingestion, structuring, and (selective) retrieval are all just LLM calls & agent loops on a on-device markdown file tree. Each file lists a set of facts w/ metadata (timestamp, confidence, source, etc.); no embeddings/RAG/training of any kind. For example: - `nanomem add <fact>` starts an agent loop to walk the tree, read relevant files, and edit. - `nanomem retrieve <query>` walks the tree and returns a single summary string (possibly assembled from many subtrees) related to the query. Whatโ€™s nice about this approach is that the memory system is, by construction: 1. partitionable (human/agents can easily separate `hobbies/snowboard.md` from `tax/residency.md` for data minimization + relevance) 2. portable and user-owned (itโ€™s just text files) 3. interpretable (you know exactly whatโ€™s written and you can manually edit) 4. forward-compatible (future models can read memory files just the same, and memory quality/speed improves as models get better) 5. modularized (you can optimize ingestion/retrieval/compaction prompts separately) Privacy & utility. I'm most excited about the ability to partition + selectively disclose memory at inference-time. Selective disclosure helps with both privacy (principle of least privilege & โ€œneed-to-knowโ€) and utility (as too much context for a query can harm answer quality). Composability. An inference-time memory module means: (1) you can run such a module with confidential inference (LLMs on TEEs) for provable privacy, and (2) you can selectively disclose context over unlinkable inference of remote models (demo below). We built nanomem as part of the Open Anonymity project (openanoymity.ai), but itโ€™s meant to be a standalone module for humans and agents (e.g., you can write a SKILL for using the CLI tool). Still polishing the rough edges! - GitHub (MIT): github.com/OpenAnonymity/nanโ€ฆ - Blog: openanonymity.ai/blog/nanomeโ€ฆ - Beta implementation in chat client soon: chat.openanonymity.ai Work done with amazing project co-leads @amelia_kuang @cocozxu @erikchi !!
8
45
4
306
75,309
What if AI learned physics the way Newton did โ€“ by experiencing it? We built Sim2Reason: train LLMs inside virtual worlds governed by real physics laws, zero human annotation. Result: +5โ€“10% improvement on International Physics Olympiad, zero-shot. ๐Ÿงต
38
214
33
1,657
249,972
for data-constrained pre-training, synth data isnโ€™t just benchmaxxing, it lowers loss on the real data distribution as we generate more tokens for even better scaling, treat synth gens as forming one long ๐—บ๐—ฒ๐—ด๐—ฎ๐—ฑ๐—ผ๐—ฐ: 1.8x data efficiency with larger gains under more compute
8
59
8
371
104,163
"Expanding the Capabilities of RL via Text Feedback (RLTF)" RL for LLMs has a wall that it struggles to get past. As a single scalar reward is too low-bandwidth to tell the model what was wrong, learning would stall on harder but useful tasks. LLM needs more specific guidance. This paper proposes RLTF, where you can replace RLHF's scalar reward with rich text critiques during training, distills those critiques into better first-try answers. So by letting the model see a critique and retry during training, then training it to bake those fixes into its first response, basically learn to spot the mistake it would be critiqued for, RLTF turns this feedback into a skill, especially on tasks where a single reward number can't explain what went wrong.
7
25
142
6,932
Lili retweeted
Standard RL is limited by sparse feedback, but distillation requires rollouts from a teacher. @yus167โ€™s team found a happy medium by training on text feedback from judges, which the model then learns to internalize and predict. teapawt.org/yus167/status/20187736โ€ฆ
RL on LLMs inefficiently uses one scalar per rollout. But users regularly give much richer feedback: "make it formal," "step 3 is wrong." Can we train LLMs on this human-AI interaction? We introduce RL from Text Feedback, with 1) Self-Distillation; 2) Feedback Modeling (1/n) ๐Ÿงต
1
8
49
4,917
alright, so you can get more than 1 bit from RL: 1. instead of single turn rollout, do a 2-turn one. try - get critique - try again. 2. use the 2nd try for training conditioning on the input (same as first try) 3. compute CE on the critique as an auxiliary loss methods: - Self Distillation = applies (1) and (2) - Feedback Modeling, no 2nd rollout conditioning trick, applies (3) as well findings: - FM > SD in math and reasoning tasks. - SD > FM in creative writing. Will def try on HammingBench results: - Knights & Knaves: GRPO=0.352 - SD=0.802, FM=0.880. - Shortest Path: GRPO 0.384 - SD=0.830, FM=0.905. - MATH500: DAPO=0.523 - SD=0.548, FM=0.567. - AIME24: DAPO=0.025 โ†’ SD=0.088 (best), FM=0.083. Very fun read :) @yus167 @lchen915 @FahimTajwar10 @MunosRemi @deepakpathak @Zanette_ai
RL on LLMs inefficiently uses one scalar per rollout. But users regularly give much richer feedback: "make it formal," "step 3 is wrong." Can we train LLMs on this human-AI interaction? We introduce RL from Text Feedback, with 1) Self-Distillation; 2) Feedback Modeling (1/n) ๐Ÿงต
1
12
115
11,329
Awesome to see LitBench showing improved performance with text feedback
RL on LLMs inefficiently uses one scalar per rollout. But users regularly give much richer feedback: "make it formal," "step 3 is wrong." Can we train LLMs on this human-AI interaction? We introduce RL from Text Feedback, with 1) Self-Distillation; 2) Feedback Modeling (1/n) ๐Ÿงต
3
3
705
Glad to see folks are finding LitBench useful cc @DanielFein7
RL on LLMs inefficiently uses one scalar per rollout. But users regularly give much richer feedback: "make it formal," "step 3 is wrong." Can we train LLMs on this human-AI interaction? We introduce RL from Text Feedback, with 1) Self-Distillation; 2) Feedback Modeling (1/n) ๐Ÿงต
1
2
349
Are we done with new RL algorithms? Turns out we might have been optimizing the wrong objective. Introducing MaxRL, a framework to bring maximum likelihood optimization to RL settings. Paper + code + project website: zanette-labs.github.io/MaxRLโ€ฆ ๐Ÿงต 1/n
14
159
35
817
222,353
RL from scalar rewards is inefficient. Our work shows how to leverage the text feedback that's already abundant in human-AI interaction. Two simple methods, Self Distillation and Feedback Modeling, deliver strong gains. I am very excited to see where this paradigm goes!
RL on LLMs inefficiently uses one scalar per rollout. But users regularly give much richer feedback: "make it formal," "step 3 is wrong." Can we train LLMs on this human-AI interaction? We introduce RL from Text Feedback, with 1) Self-Distillation; 2) Feedback Modeling (1/n) ๐Ÿงต
14
144
12,883