My favorite experiment here is subliminal transfer of a random smooth function. We trained an LLM to predict a random small neural net on integer inputs. The LLM needs 50k examples to learn this pretty well, as the function is random (lacks the structure of well-known / simple functions). There is no way this function could have appeared in pretraining because it’s random; so it’s a new capability. (This is a stand-in for any complex capability that a model might learn in the course of training). This teacher LLM then generates a dataset unrelated to the function. It contains no numbers at all and no mention of related terms. It consists of random sequences of *words* (unlike the classic setup) and Alpaca responses. We finetune the student on this data and it becomes much better at predicting the function (while still worse than the teacher). Now, you can get reasonable performance at predicting the function by optimizing a steering vector in a single layer of the LLM. In other words, the LLM does have existing directions that capture some of the behavior of the function (which is not that surprising, as the LLM is so big + rich in representations). However, on half-integer inputs not seen in training, the subliminal student model generalizes much better than the steering vector. Thus, subliminal learning seems to transfer something deeper and more robust. The caveat is that we use attention-only LoRA for both student and teacher. This is artificial: standard LoRA usually adapts more than just attention weights, and this restriction seems to strengthen subliminal transfer. We'd like to understand whether the effect also occurs with standard LoRA or full fine-tuning. We also use top-32 logit distillation (unlike single-sample as in Cloud et al).