AI researchers seek to understand intelligence well enough to create beings of greater intelligence than current humans. Reaching this profound intellectual milestone will enrich our economies and challenge our societal institutions. It will be unprecedented and transformational, but also a continuation of trends that are thousands of years old. People have always created tools and been changed by them; this is what humans do. The next big step is to understand ourselves. This is a quest grand and glorious, and quintessentially human.

Agree on all three points

And here is the new PhD thesis of Gautham Vasan (@Gautham529), whose committee I was recently on. I thought this thesis was particularly well done, including the AVG algorithm in Chapter 6, which is a new streaming RL actor-critic (different from the Stream-X algorithms). Title: Robots That Learn on the Fly Through Real-World Interaction URL: gauthamvasan.com/papers/Vasa… Abstract: A robot that learns from its own runtime experience can improve its behavior over its operational lifetime and adapt to conditions that were never anticipated. The prevailing practice in robot learning provides no such ability: practitioners train a control policy before deployment, from human-provided data or from simulated interaction, and hold it fixed afterward. When performance degrades after deployment, a human engineer may need to diagnose the failure, collect more data or refine the simulator, and retrain and redeploy the policy. Reinforcement learning (RL) from ongoing interaction could reduce reliance on humans for data collection, retraining, and redeployment. Real-time reinforcement learning names the setting in which an agent senses, acts, and learns from ongoing interaction as wall-clock time advances for both the agent and the environment. Real-time RL on physical robots imposes three practical demands: (a) learning must operate within the robot’s limited computational and memory resources; (b) learning must reach competent performance reliably within a practical wall-clock budget; (c) learning must use rewards that are computable from the robot’s own sensing and simple to specify across tasks. I present three complementary contributions that address the three demands. The first contribution, the Remote-Local Distributed (ReLoD) framework, supplements the robot’s limited computational resources with a remote workstation. The framework distributes action computation and learning updates between the two computers. Experiments on vision-based tasks showed that the performance of an agent using Soft Actor-Critic, a widely used batch RL algorithm that learns from stored transitions, degraded under limited onboard computation. A robot using ReLoD recovered performance by drawing on remote computation. Across repeated runs, competent policies were learned from scratch within a few hours. The second contribution is an empirical study of the minimum-time task specification, which assigns a reward of −1 at each timestep and terminates the episode when the goal is reached. Given matched interaction budgets, final policies learned under the minimum-time specification matched or outperformed those learned under guiding reward specifications, which reward progress toward the goal. Using the minimum-time specification, four physical robots learned goal-reaching policies from raw-pixel observations within hours and without external instrumentation. The third contribution is Action Value Gradient (AVG), a computationally cheap streaming RL algorithm that learns with neural networks. An agent using AVG updates its parameters once from each transition (an observation, action, reward, and next observation) and stores no transitions for reuse. On simulated tasks, agents using AVG outperformed agents using existing streaming RL methods, often achieving final performance comparable to batch RL methods. On physical-robot tasks, agents using AVG learned competent policies within hours on an edge device. The three contributions address the three demands in complementary ways. ReLoD and AVG meet the computational resource demand from opposite directions: ReLoD supplements a robot’s onboard computer with remote computation when feasible, and AVG is computationally cheap, enabling on-device learning. The minimum-time specification reduces specifying a goal-reaching task to one requirement, detecting the goal from onboard sensing. Across the contributions, physical robots learned competent behavior from scratch within hours. Together, the contributions advance real-time reinforcement learning on physical robots toward the longer-term goal of continual adaptation and long-term autonomy. Gautham is now a research scientist at Zyphra.

The PhD thesis of my 16th PhD student, Fernando Hernandez Garcia, is now available. Title: Selective Reinitialization Algorithms for Preventing Plasticity Loss in Artificial Neural Networks Url: incompleteideas.net/papers/F… Abstract: In this dissertation, I study systems based on artificial neural networks that learn from nonstationary data. Learning from non-stationary data requires continual adaptation of the system, and is often referred to as continual learning. Developing systems capable of continual learning is a longstanding goal in artificial intelligence. In deep learning, the field concerned with designing and training deep neural networks, this goal remains elusive. This dissertation addresses a fundamental challenge in continual learning: loss of plasticity, a phenomenon where artificial neural network systems progressively lose their ability to learn from new data. While the phenomenon has been noted several times over the last three decades, it has remained understudied until recently. The work in this dissertation constitutes the first systematic demonstrations of plasticity loss, highlighting its persistence and importance. In this work, I provide a systematic demonstration of plasticity loss in a wide variety of deep learning systems. Across systems based on fully-connected networks, convolutional networks, residual networks, and vision transformers, plasticity degrades when learning continually. Notably, even systems employing normalization techniques, residual connections, and regularization—design choices that improve training stability—remain susceptible to plasticity loss. This evidence establishes that plasticity loss is pervasive and that deep learning systems trained with backpropagation are not suitable for continual learning. I explore the idea of selective reinitialization to prevent plasticity loss. This idea has been used in the past for improving generalization performance in deep learning systems. However, its use for preventing plasticity loss is a recent innovation pioneered by the continual backpropagation algorithm. The algorithm periodically sets new values for units in the network, making initialization a continuous process rather than a one-time operation. I demonstrate the effectiveness of continual backpropagation in preventing plasticity loss across a wide variety of settings. These demonstrations establish that plasticity loss, while pervasive, is not inherent to deep learning systems. I generalize continual backpropagation through an algorithm I call selective unit reinitialization. This general algorithm has three key components: a utility measure that ranks units by importance, a pruning criterion that selects which units to reinitialize, and a reinitialization method that assigns new values to the selected units. Continual backpropagation involves a specific choice of utility measure, pruning criterion, and reinitialization method. However, the general algorithm is not limited to the choices used in continual backpropagation. I present a study of how different choices for the three components of selective unit reinitialization affect its effectiveness at maintaining plasticity. This study establishes selective unit reinitialization as a general approach that can be tailored to each learning system to maintain plasticity. Finally, I propose a different approach to implementing selective reinitialization that operates at the weight level. I call the corresponding algorithm selective weight reinitialization. Reinitializing at the unit or weight level involves different trade-offs. Unit reinitialization minimally disrupts the network outputs, preserving stability during learning. However, the definition of a unit varies across network architectures, requiring additional engineering to make the approach effective. Weight reinitialization, in contrast, can be readily applied to any arbitrary network architecture, but can substantially affect network outputs, reducing training stability. The reduced learning stability can be remedied with L2 regularization, which stabilizes selective weight reinitialization but introduces an additional hyperparameter. Both selective unit and weight reinitialization successfully maintained plasticity across the systems tested, providing flexible approaches for different architectural and engineering constraints. Fernando is now a research scientist at Zyphra.

The PhD thesis of my 15th PhD student, Shibhansh Dohare (@ShibhanshD), is now available. Title: Learning Forever using Artificial Neural Networks Url: incompleteideas.net/papers/D… Abstract: In this dissertation, I study the ability of artificial neural networks to learn new things. I refer to this ability as plasticity. Plasticity is a desirable property of intelligent systems in the real world. Systems that interact with the real world can become outdated if they do not learn from the new information revealed to them by the extremely large and partially observable real world. All naturally intelligent systems continually learn new things to keep up with the changes in the world. Many applications like predicting markets or weather, learning human preferences, controlling factories or prosthetics, etc require AI systems that continually learn new things. I provide direct and thorough demonstrations of loss of plasticity in deep learning systems. The demonstrations range from simple feed-forward networks to deep residual networks and cover a wide range of algorithmic choices. While seeking an understanding of the phenomenon, I found that loss of plasticity occurred simultaneously with hidden units becoming dormant and becoming similar to each other, hampering the ability of the networks to learn new things. Next, I describe the continual backpropagation algorithm. I designed this algorithm, in collaboration with others, to maintain plasticity in artificial neural networks. Continual backpropagation is a simple extension of the conventional backpropagation algorithm. The conventional backpropagation has two main parts: initialization with small random weights and gradient descent at each step. In addition to gradient descent, continual backpropagation reinitializes a small fraction of units at each step. Continual backpropagation indefinitely maintained plasticity in many continual supervised learning problems. Finally, I study plasticity of deep reinforcement learning systems. First, I show that plasticity loss has an effect even in stationary reinforcement learning problems. It is unsurprising that plasticity loss affects stationary reinforcement learning because it requires continual learning due to changing policies and bootstrapping. This effect is visible under long-term learning. I found that some commonly used RL algorithms do not scale with data. Their performance drops as they continue interacting with the environment. Plasticity loss hampers the ability of these algorithms to recover and relearn when the performance drops, resulting in agents that do not get better with experience. Continual backpropagation allows these algorithms to recover when performance drops, leading to agents whose performance keeps improving with experience. Second, I show that loss of plasticity is a significant problem in continual reinforcement learning problems, and continual backpropagation maintains plasticity in these problems. Shibhansh is now a research scientist at Keen Technologies.

In here is a video of The Bitter Lesson as a pretty-good country-music song. Enjoy. oneusefulthing.org/p/the-dot…

I do not seek to dominate, or be dominated by, machine intelligences. I seek harmonious coexistence.

This looks interesting…

A nice statement from the mathematicians: mathandai.org