Filtering subversion related information from pretraining is feasible. We're releasing a research note where we remove information related to monitoring designs and successful subversion strategies in three 30B LLMs. In collaboration with @redwood_ai 🧵

There's a 𝐠𝐫𝐨𝐰𝐢𝐧𝐠 𝐫𝐞𝐬𝐨𝐮𝐫𝐜𝐞 𝐠𝐚𝐩 𝐛𝐞𝐭𝐰𝐞𝐞𝐧 𝐬𝐚𝐟𝐞𝐭𝐲 𝐭𝐞𝐚𝐦𝐬 𝐢𝐧𝐬𝐢𝐝𝐞 𝐟𝐫𝐨𝐧𝐭𝐢𝐞𝐫 𝐥𝐚𝐛𝐬 𝐚𝐧𝐝 𝐢𝐧𝐝𝐞𝐩𝐞𝐧𝐝𝐞𝐧𝐭 𝐬𝐚𝐟𝐞𝐭𝐲 𝐫𝐞𝐬𝐞𝐚𝐫𝐜𝐡𝐞𝐫𝐬, and closing it is essential to maintaining an ecosystem of impactful safety research. We've spent months securing a large, multi-year GPU contract. This taught us that 𝐟𝐮𝐧𝐝𝐢𝐧𝐠 𝐢𝐬 𝐧𝐞𝐜𝐞𝐬𝐬𝐚𝐫𝐲 𝐛𝐮𝐭 𝐧𝐨𝐭 𝐬𝐮𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐭. We've written up the non-obvious bottlenecks we hit, and what can be done to alleviate them: lesswrong.com/posts/aCGx79eG…

Post-training data may contain a mixture of safe and unsafe properties. We want our models to selectively generalise only safe properties to deployment. We study generalisation by introducing a neologism during midtraining, in synthetic documents that describe how models generalise behind and across this boundary.