# How DigitalOcean Served Kimi K3 on Day Zero

DevFeed: [How DigitalOcean Served Kimi K3 on Day Zero](<https://devfeed.tech/articles/under-the-hood-serving-kimi-k3-19944.md>)

Original publisher: [Read original article](<https://www.digitalocean.com/blog/serving-kimi-k3-inference-engine>)

Author: Shree Murthy

Published: 2026-07-30T17:10:40Z

Content type: article

Language: en

Sources: [DigitalOcean](<https://devfeed.tech/sources/digitalocean.md>)

Topics: [Digital Ocean](<https://devfeed.tech/topics/digital-ocean.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Hardware](<https://devfeed.tech/topics/hardware.md>), [NVLink](<https://devfeed.tech/topics/nvlink.md>), [OpenAI](<https://devfeed.tech/topics/openai.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [digitalocean](<https://devfeed.tech/tags/digitalocean.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvlink](<https://devfeed.tech/tags/nvlink.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

## AI overview

DigitalOcean describes how it served the Kimi K3 model on its Inference Engine from day zero, including GPU selection, distributed serving with llm-d, vLLM tuning, and verification against Moonshot AI's benchmarks.

## Source excerpt

DigitalOcean launched Kimi K3 on day 0. It's already one of the most popular models on the platform and across the market: second most likes on Hugging Face, sixth most traffic on OpenCode. Getting a model this size running well on day zero took real work across several teams. Thanks to Moonshot AI, Inferact, RadixArk, NVIDIA, and AMD for the help getting there. Standing up a new model, integrating it into DigitalOcean's Inference Engine, and showcasing its unique attributes on day 0 takes three things: the right hardware, a tuned serving stack, and rigorous verification against Moonshot's own benchmarks. Here are the lessons we learned along the way: Hardware selection and implementation We selected NVIDIA HGX™ B300 and AMD Instinct™ MI350x GPUs to run K3 because these instances provide the memory capacity, FLOPs, and interconnect horsepower necessary for a model of K3's size and architecture. We built our distributed inference stack with llm-d because it includes native support for GPU type heterogeneity. This let us quickly onboard K3 to both AMD and NVIDIA platforms. Kimi K3 has roughly 2.78 trillion total parameters, 896 routed experts, and an attention stack that interleaves 69 Kimi Delta Attention (KDA) layers with 24 Gated Multi-head Latent Attention (MLA) layers. Kimi-K3 weights are ~1.56 TB in total, which requires about 195 GiB per GPU. Given such a large memory footprint for the weights alone, and a need to keep enough headroom for KV cache and activations, the practical unit of deployment is an 8x NVIDIA HGX B300 or AMD Instinct MI350X server. Both have 288GB of VRAM capacity, and after loading the weights, there is still some amount of practical memory left for the KV cache. Entire weights cannot be loaded on a single GPU. That's where the high-speed scaled-up NVIDIA's NVLink or AMD's Infinity Fabric is critical to ensure there is enough interconnect horsepower for bandwidth intensive, latency sensitive attention and expert parallel computations. Model