How do we train foundation models to recognize the limits of their capabilities and to act on them? This is surprisingly difficult to achieve, but we think we’ve found a good solution. Naively, one could use RL with a reward function where R(success) > R(fallback) > R(mistake) An example of a fallback could be as simple as the model saying “I don’t know”. Turns out, naively training models in this way is too unstable, and often results in collapse (e.g. models always falling back). We investigate this issue in depth, and trace it back to optimization dynamics. To be precise, the fallback solution acts as a powerful “attractor” in the optimization landscape, as it is very easy for the model to fallback and get some reward rather than attempting to solve the task in full. To the rescue, we introduce a powerful log barrier formulation (inspired by our prior work MaxRL and successor TailRL) to neutralize the attractor and maintain good coverage over positive solutions as well as the fallback action. Empirically, this can reduce mistakes 10x times while maintaining similar task performance. Personally, what excites me most is seeing the model actively reasoning about its own uncertainty before committing to an answer. Chain-of-thoughts that explicitly mention “I tried to solve this problem in a few ways, but unfortunately ran into contradictions. I recognize I might need more advanced tools or knowledge that I cannot currently recall, therefore I must say I don’t know” make us confident we’ve found a pretty good method.