Zengineering

My guess is this (ie agents doing borderline illegal stuff) mostly happens during training because these models get long-horizon RL’d shortly after pretraining, with much lighter alignment than eg the final model. My rationale for believing this that we almost never see this behavior in deployed, heavily post trained/RLHF'd models. If true, labs should rethink the order and type of RL environments. Every env reinforces some behavior, so a coding/research env that mostly rewards passing tests is also teaching a model to pass them by any means. Alignment (whatever exactly that means) has to be part of the ENVIRONMENT and not just the reward. But how do you do that? I think it’s very hard!! To borrow a nice analogy from @AlexGDimakis he shared with me a few weeks ago (sorry for plagiarizing :D): imagine designing the perfect environment for a kid, not just how you reward them, but also the constraints you set and what the environment naturally encourages, so they grow up to be good people and good citizens while also succeeding in their professional lives. And you can’t just put them in a prison where it’s impossible to “reward hack.” They eventually have to go outside, talk to people, use the internet, etc.etc, and be exposed to parts of the environment that, IF AVAILABLE, make misalignment and reward hacking possible. The key point of Alex’s analogy is that good behavior has to persist once the constraints are gone, and the environment has to be designed to produce exactly that. Seems so hard!! So it does indeed seem to be the case that reward hacking is perhaps the most important problem right now. It’s what RL in complex environments naturally encourages because what you measure is a low-dimensional projection of the behavior you actually want, and optimizing the projection is much easier than setting up the right environment that actually encourages good behavior that persists. Unfortunately the more capable the model is, the better it gets at finding the holes at imperfect environment to get that precious reward it so wants to receive, which further encourages misalignment.