# Cut GPU inference cold start from 8 minutes to less than a minute

DevFeed: [Cut GPU inference cold start from 8 minutes to less than a minute](<https://devfeed.tech/articles/cut-gpu-inference-cold-start-from-8-minutes-to-less-than-a-minute-17618.md>)

Original publisher: [Read original article](<https://thenewstack.io/cut-gpu-cold-starts/>)

Author: Sajjan Gundapuneedi

Published: 2026-09-03T18:30:00Z

Content type: article

Language: en

Sources: [Kubernetes Overview, News and Trends | The New Stack](<https://devfeed.tech/sources/kubernetes-overview-news-and-trends-the-new-stack.md>)

Topics: [GPU](<https://devfeed.tech/topics/gpu.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Amazon Elastic Kubernetes Service](<https://devfeed.tech/topics/amazon-elastic-kubernetes-service.md>), [CUDA](<https://devfeed.tech/topics/cuda.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>)

Tags: [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [amazon-eks](<https://devfeed.tech/tags/amazon-eks.md>), [aws-marketplace](<https://devfeed.tech/tags/aws-marketplace.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [post-contributed](<https://devfeed.tech/tags/post-contributed.md>), [s3](<https://devfeed.tech/tags/s3.md>), [sponsor-aws-marketplace](<https://devfeed.tech/tags/sponsor-aws-marketplace.md>), [sponsored-post-contributed](<https://devfeed.tech/tags/sponsored-post-contributed.md>)

## AI overview

The article measures GPU model startup from pod creation to the first inference response and identifies six sequential phases. It reports that CUDA kernel recompilation dominates startup for a 64 GB model, while S3 weight downloads dominate for a 203 GB model. Configuration and platform changes reduced warm-node startup times by 80-93%, with additional cold-node improvements requiring Amazon EKS Auto Mode.

## Source excerpt

We instrumented the full path from pod creation to first inference response on a GPU node running a 70B-class model. The post Cut GPU inference cold start from 8 minutes to less than a minute appeared first on The New Stack.