# Making Machines Move

DevFeed: [Making Machines Move](<https://devfeed.tech/articles/making-machines-move-1708.md>)

Original publisher: [Read original article](<https://fly.io/blog/machine-migrations/>)

Published: 2024-07-30T00:00:00Z

Content type: article

Language: en

Sources: [The Fly Blog](<https://devfeed.tech/sources/the-fly-blog.md>)

Topics: [migration](<https://devfeed.tech/topics/migration.md>)

Tags: [cdn](<https://devfeed.tech/tags/cdn.md>), [close-to-users](<https://devfeed.tech/tags/close-to-users.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [deploy-app-servers](<https://devfeed.tech/tags/deploy-app-servers.md>), [docker](<https://devfeed.tech/tags/docker.md>), [elixir](<https://devfeed.tech/tags/elixir.md>), [fly](<https://devfeed.tech/tags/fly.md>), [fly-io](<https://devfeed.tech/tags/fly-io.md>), [heroku-alternative](<https://devfeed.tech/tags/heroku-alternative.md>), [heroku-competitor](<https://devfeed.tech/tags/heroku-competitor.md>), [hosting](<https://devfeed.tech/tags/hosting.md>), [i](<https://devfeed.tech/tags/i.md>), [infra](<https://devfeed.tech/tags/infra.md>), [migration](<https://devfeed.tech/tags/migration.md>), [networking](<https://devfeed.tech/tags/networking.md>), [postgresql-clusters](<https://devfeed.tech/tags/postgresql-clusters.md>), [servers](<https://devfeed.tech/tags/servers.md>), [storage](<https://devfeed.tech/tags/storage.md>)

## AI overview

Fly.io describes the challenge of migrating stateful virtual machines whose attached NVMe volumes anchor application data to a physical worker. The article contrasts simple stateless-worker draining with volume migration, where backup restoration risks data loss and copying volumes can cause unacceptable interruption.

## Source excerpt

We're Fly.io, a global public cloud with simple, developer-friendly ergonomics. If you've got a working Docker image, we'll transmogrify it into a Fly Machine: a VM running on our hardware anywhere in the world. Try it out; you'll be deployed in just minutes. At the heart of our platform is a systems design tradeoff about durable storage for applications. When we added storage three years ago, to support stateful apps, we built it on attached NVMe drives. A benefit: a Fly App accessing a file on a Fly Volume is never more than a bus hop away from the data. A cost: a Fly App with an attached Volume is anchored to a particular worker physical. bird: a BGP4 route server. Before offering attached storage, our on-call runbook was almost as simple as "de-bird that edge server", "tell Nomad to drain that worker", and "go back to sleep". NVMe cost us that drain operation, which terribly complicated the lives of our infra team. We've spent the last year getting "drain" back. It's one of the biggest engineering lifts we've made, and if you didn't notice, we lifted it cleanly. The Goalposts With stateless apps, draining a worker is easy. For each app instance running on the victim server, start a new instance elsewhere. Confirm it's healthy, then kill the old one. Rinse, repeat. At our 2020 scale, we could drain a fully loaded worker in just a handful of minutes. You can see why this process won't work for apps with attached volumes. Sure, create a new volume elsewhere on the fleet, and boot up a new Fly Machine attached to it. But the new volume is empty. The data's still stuck on the original worker. We asked, and customers were not OK with this kind of migration. Of course, we back Volumes snapshots up (at an interval) to off-network storage. But for "drain", restoring backups isn't nearly good enough. No matter the backup interval, a "restore from backup migration" will lose data, and a "backup and restore" migration incurs untenable downtime. The next thought you have is,