# We Were Wrong About GPUs

DevFeed: [We Were Wrong About GPUs](<https://devfeed.tech/articles/we-were-wrong-about-gpus-1723.md>)

Original publisher: [Read original article](<https://fly.io/blog/wrong-about-gpu/>)

Published: 2025-02-14T00:00:00Z

Content type: opinion

Language: en

Sources: [The Fly Blog](<https://devfeed.tech/sources/the-fly-blog.md>)

Topics: [GPU](<https://devfeed.tech/topics/gpu.md>)

Tags: [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [cdn](<https://devfeed.tech/tags/cdn.md>), [close-to-users](<https://devfeed.tech/tags/close-to-users.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [deploy-app-servers](<https://devfeed.tech/tags/deploy-app-servers.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [docker](<https://devfeed.tech/tags/docker.md>), [elixir](<https://devfeed.tech/tags/elixir.md>), [fly](<https://devfeed.tech/tags/fly.md>), [fly-io](<https://devfeed.tech/tags/fly-io.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [heroku-alternative](<https://devfeed.tech/tags/heroku-alternative.md>), [heroku-competitor](<https://devfeed.tech/tags/heroku-competitor.md>), [hosting](<https://devfeed.tech/tags/hosting.md>), [i](<https://devfeed.tech/tags/i.md>), [inference](<https://devfeed.tech/tags/inference.md>), [networking](<https://devfeed.tech/tags/networking.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [postgresql-clusters](<https://devfeed.tech/tags/postgresql-clusters.md>), [security](<https://devfeed.tech/tags/security.md>), [servers](<https://devfeed.tech/tags/servers.md>)

## AI overview

Fly.io says its GPU Machines remain available, but it does not expect to build a larger second version soon. The article explains that GPU support for AI/ML inference brought security, utilization, and cost challenges in its micro-VM-based cloud infrastructure.

## Source excerpt

We're building a public cloud, on hardware we own. We raised money to do that, and to place some bets; one of them: GPU-enabling our customers. A progress report: GPUs aren't going anywhere, but: GPUs aren't going anywhere. A couple years back, we put a bunch of chips down on the bet that people shipping apps to users on the Internet would want GPUs, so they could do AI/ML inference tasks. To make that happen, we created Fly GPU Machines. A Fly Machine is a Docker/OCI container running inside a hardware-virtualized virtual machine somewhere on our global fleet of bare-metal worker servers. A GPU Machine is a Fly Machine with a hardware-mapped Nvidia GPU. It's a Fly Machine that can do fast CUDA. Like everybody else in our industry, we were right about the importance of AI/ML. If anything, we underestimated its importance. But the product we came up with probably doesn't fit the moment. It's a bet that doesn't feel like it's paying off. If you're using Fly GPU Machines, don't freak out; we're not getting rid of them. But if you're waiting for us to do something bigger with them, a v2 of the product, you'll probably be waiting awhile. What It Took GPU Machines were not a small project for us. Fly Machines run on an idiosyncratically small hypervisor (normally Firecracker, but for GPU Machines Intel's Cloud Hypervisor, a very similar Rust codebase that supports PCI passthrough). The Nvidia ecosystem is not geared to supporting micro-VM hypervisors. GPUs terrified our security team. A GPU is just about the worst case hardware peripheral: intense multi-directional direct memory transfers (not even bidirectional: in common configurations, GPUs talk to each other) with arbitrary, end-user controlled computation, all operating outside our normal security boundary. We did a couple expensive things to mitigate the risk. We shipped GPUs on dedicated server hardware, so that GPU- and non-GPU workloads weren't mixed. Because of that, the only reason for a Fly Machine to be sched