Many of the winning ideas in RL and post-training for reasoning are simple and elegant. This post covers a line of work I liked. Each paper has a simple idea at its core, and together they ask one question: how do you keep learning when rewards are sparse and most attempts fail? 1. Hindsight Experience Replay (HER, OpenAI 2017) arxiv.org/abs/1707.01495 tl;dr: "You failed the task. Did you succeed at some other goal along the way? Learn from that." A failed episode gives no reward, so it's useless for RL. But the agent still ended up somewhere. HER relabels the episode as if that was the goal all along, so failures turn into successes for other goals and give real gradients. The original demos were a robotic arm. 2. Hindsight Supervised Learning (HSL, "Spinning Straw into Gold") arxiv.org/abs/2607.04235 tl;dr: "HER for LLM agents: an LLM finds which goals a failed run actually achieved, then you SFT on those." An agent that fails its task usually still accomplishes something. A strong LLM, the "relabeler," reads the trajectory and writes the instruction it actually fulfilled. Steps that don't matter for the new goal are masked out. The relabeled trajectories are mixed with the expert data and trained with plain supervised fine-tuning, with no RL. 3. Self-Distillation Fine-Tuning (SDFT, "Self-Distillation Enables Continual Learning") arxiv.org/abs/2601.19897 tl;dr: "Let the model see a worked example, then grade its own attempt token by token." It's closely related to OPSD (arxiv.org/abs/2601.18734) and SDPO (arxiv.org/abs/2601.20802), all from January 2026. - The student samples a response to the prompt. - The teacher is the same model, but its prompt also has privileged information, here a demonstration of how to do the task. - The teacher reads the student's own tokens up to step t and gives its distribution over the next token, meaning what it would write from there. The student is trained to match it. - Because training happens on the student's own samples, it's on-policy, and it forgets much less than SFT. - In practice, the teacher is an EMA (a slowly updated copy) of the student. The loss is forward KL(teacher || student), so the student has to cover every option the teacher considers plausible. 4. HERO: Hindsight-Enhanced Reflection from Environment Observations arxiv.org/abs/2606.11559 tl;dr: "For a multi-turn agent, have the model write a hindsight hint for each turn, then self-distill with those hints." Multi-turn agents, such as ReAct-style tool loops, have a credit assignment problem. The reward comes at the end, and GRPO gives every turn the same credit. After the episode, the same model acts as a "reflector." It reads the whole trajectory, the tool outputs and the rewards, and writes a short hint per turn about what was right or wrong and how to fix it. Then a teacher (an EMA copy) gets that turn's history, the next observation, the final reward and the hint, and scores the student's original tokens for that turn. It's SDFT-style distillation, with the hint as the privileged information. Everything depends on how good the reflector is. The reflector sometimes blames a late turn for a mistake made earlier. 5. Retrospection-Only Fine-Tuning (ROFT, a few weeks ago) arxiv.org/abs/2609.35741 tl;dr: "Ask the model what it should have done differently, then fine-tune it on its own answer." After a reasoning attempt, the model gets a retrospection prompt with the task, a compressed summary of its trajectory, its patch and the test results. It writes a short retrospection about the key decision, the evidence, the fix, and when the lesson applies. Then you run plain SFT with the retrospection prompt as input and the retrospection as the target. It trains only on these retrospections and not on the original tasks. The surprise is that learning to explain improves how the model acts. The authors' hypothesis is that predicting the retrospection updates the same weights that represent the model's own actions, but it's still not clear why it works. It's very efficient. It runs online, with the current model writing both the attempts and the retrospections, and it learns from failed attempts too.

We trained an AI agent on its own explanations, not its code. It got better at coding. No teacher. No verifier. No RL updates. No extra context at test time. Here's how. Links below. 🧵

We are excited to announce that we post-trained NVIDIA Nemotron-3.5-Super-VL to be an autonomous ML researcher. We show that our post-trained model goes from 28% to 35% on AutoResearchExam, matching Qwen 3.8 Max, a model with 20× the total parameters. The tasks are long horizon, open ended AI and ML research: building a faster executor for a transformer compute graph, training a RL control policy, creating new uncertainty quantification methods, etc. The model gets to submit once, and it doesn't see its score until the end. We curated SFT data using Self-context distillation. This means that we let the Nemotron model to solve a problem with hints, then we SFT on these trajectories but without the hints. This gave us +11% on AutoResearchExam. As a next step, we added reflection: We ask the Nemotron model to review its own past attempts and writes concrete lessons learned. Then we SFT on these reflections. Surprisingly, training on these lessons improved the performance to +25% compared to the base model. The hardest part was choosing the right data. We selected traces based on task scores, completion checks, and runtime, then refined the sampling across training rounds to address failures in the base and fine-tuned models. We removed hints and filtered out steps that relied on information available only in those hints. The goal was to teach decisions supported by the task and history the model would actually see at inference. Full write-up: bespokelabs.ai/blog/teaching… Thanks to @NVIDIAAI for early access and collaborating on this.

We trained an AI agent on its own explanations, not its code. It got better at coding. No teacher. No verifier. No RL updates. No extra context at test time. Here's how. Links below. 🧵

We trained an AI agent on its own explanations, not its code. It got better at coding. No teacher. No verifier. No RL updates. No extra context at test time. Here's how. Links below. 🧵

Post-training LLMs is like mixing a cocktail: Too much easy data → no learning Too much hard data → instability Wrong balance → collapse And today, we mix it by hand. What if the data mixture could be learned instead of hand-tuned? arxiv.org/abs/2602.20532 🧵👇

Post-training LLMs is like mixing a cocktail: Too much easy data → no learning Too much hard data → instability Wrong balance → collapse And today, we mix it by hand. What if the data mixture could be learned instead of hand-tuned? arxiv.org/abs/2602.20532 🧵👇

We trained an AI agent on its own explanations, not its code. It got better at coding. No teacher. No verifier. No RL updates. No extra context at test time. Here's how. Links below. 🧵