Here is my response about RL comment after watching the podcast of @dwarkesh_sp interviewing @karpathy and his comment on RL: xitter.cf/dwarkesh_sp/status/197… RL is a powerful idea, but let's be thoughtful about when we actually need it. RL has an incredible 120+ year journey in animal learning—starting with Edward Thorndike's puzzle box experiments in 1898, through decades of psychology research on trial-and-error learning, culminating in 1997 discovery by Schultz, Dayan, and Montague that dopamine neurons literally implement temporal difference learning. That's real RL: agent -environments experiential interaction, and actual learning through experience. But what we're calling "RL" in LLMs today? It's mostly optimization with reward-weighted gradients on offline data. There's no agent exploring an environment, no real interaction loop. It's just: data in, neural net processes it, and objective function for the output is weighted by reward signals during final RL training stage after intermediate training steps. Here's the thing—RL fundamentally addresses problems we'd solve with dynamic programming, except the state space is astronomically large ( easily more states than atoms in the universe--a 19x19 Computer Go roughly has 10^170 states!) and we don't have a model of the world. So we use RL as an approximation learning technique. It's essential for these kinds of problems, but not every problem is like this. When you have interactive, experiential tasks—like learning to generate code to build a software that gets runtime feedback from compute environment given a goal—RL makes perfect sense for LLM-based methods. But the current RL in textbooks or research papers are not enough-- it needs temporal abstractions over sequence of actions--aka generated tokens in LLM world. It is impossible to learn optimal policies for micro-actions without temporal abstractions unless we assume we have infinite data and compute resource--remember it is an NP-hard combinatorial problem! I think as we move into the era of experiential AI—agents that actually interact with environments and receive runtime feedback—the role of RL will become clearer and more necessary.