We’ve been working on Live Migration of Tensorlake sandboxes, which lets us move sandboxes between machines with under 300ms of guest pause and stalled TCP connections. We retain all open TCP connections as sandboxes are migrated, with zero connection resets. Running stateful sandboxes at scale often means low cluster utilization as workload patterns shift throughout the day. Today, we don’t defragment the cluster to release underutilized nodes. Another problem with stateful sandboxes is that they can’t outgrow the available free resources (like RAM) on a tightly packed machine. Live Migration will help reduce operational expenses as we continue to scale, and unlock features like guaranteed vertical scaling: moving sandboxes around the cluster, and hot-plugging agent sandboxes with more compute resources when they need it. It’s been fun working on this problem. We went back to see what VMWare and other companies have published about live VM migration and incorporated it into our dataplane, VMM, and networking subsystems. Here is an example of moving a sandbox with 1GB RAM and 10GB disk from one node to another within a region in our production cluster!