# Fast model loading for AI inference on Amazon EKS

DevFeed: [Fast model loading for AI inference on Amazon EKS](<https://devfeed.tech/articles/fast-model-loading-for-ai-inference-on-amazon-eks-4630.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/containers/fast-model-loading-for-ai-inference-on-amazon-eks/>)

Author: Sajjan Gundapuneedi

Published: 2026-09-01T15:48:15Z

Content type: article

Language: en

Sources: [Containers](<https://devfeed.tech/sources/containers.md>)

Topics: [Inference Performance](<https://devfeed.tech/topics/inference-performance.md>), [Amazon Elastic Kubernetes Service](<https://devfeed.tech/topics/amazon-elastic-kubernetes-service.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [amazon-eks](<https://devfeed.tech/tags/amazon-eks.md>), [amazon-elastic-kubernetes-service](<https://devfeed.tech/tags/amazon-elastic-kubernetes-service.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [s3](<https://devfeed.tech/tags/s3.md>), [sglang](<https://devfeed.tech/tags/sglang.md>), [startup](<https://devfeed.tech/tags/startup.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

## AI overview

The article analyzes cold-start delays for AI inference pods on Amazon EKS. It finds that startup bottlenecks vary by model size: torch.compile dominates for smaller models, while loading weights from S3 to GPU memory dominates for larger models. Configuration changes to Run:ai Model Streamer reduce model-loading time on repeat launches.

## Source excerpt

When you scale AI inference on Amazon EKS, every new pod must load model weights into GPU memory before serving traffic. We investigated where cold-start time goes and found two configuration-only changes to Run:ai Model Streamer that cut model startup time by 80-93% on subsequent launches, with no code changes.