# Podcast: Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM

DevFeed: [Podcast: Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM](<https://devfeed.tech/articles/podcast-distributed-llm-inference-on-kubernetes-with-llm-d-wasm-tornadovm-123025.md>)

Original publisher: [Read original article](<https://www.infoq.com/podcasts/distributed-llm-inference-kubernetes/>)

Author: Edoardo Vacchi

Published: 2026-11-09T11:00:00Z

Content type: news

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Local AI](<https://devfeed.tech/topics/local-ai.md>), [TornadoVM](<https://devfeed.tech/topics/tornadovm.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [Wasm components](<https://devfeed.tech/topics/wasm-components.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [backend](<https://devfeed.tech/tags/backend.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [containers](<https://devfeed.tech/tags/containers.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-ai-inference](<https://devfeed.tech/tags/distributed-ai-inference.md>), [distributed-llm-inference-kubernetes](<https://devfeed.tech/tags/distributed-llm-inference-kubernetes.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-utilization](<https://devfeed.tech/tags/gpu-utilization.md>), [inference](<https://devfeed.tech/tags/inference.md>), [infoq](<https://devfeed.tech/tags/infoq.md>), [java](<https://devfeed.tech/tags/java.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [kubernetes-clusters](<https://devfeed.tech/tags/kubernetes-clusters.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-d](<https://devfeed.tech/tags/llm-d.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [podcast](<https://devfeed.tech/tags/podcast.md>), [the-infoq-podcast](<https://devfeed.tech/tags/the-infoq-podcast.md>), [tornadovm](<https://devfeed.tech/tags/tornadovm.md>), [wasm](<https://devfeed.tech/tags/wasm.md>), [web-development](<https://devfeed.tech/tags/web-development.md>), [webassembly](<https://devfeed.tech/tags/webassembly.md>)

## AI overview

llm-d extends vLLM's high-throughput inference from a single machine to distributed deployments, balancing requests across instances while favoring cache locality for conversational KV and prefix caches and avoiding overload. It also provides flow control and request prioritization, and can run on bare metal as well as Kubernetes. The discussion describes batch inference during idle periods to use otherwise underused hardware, and semantic classification to route prompts to right-sized models based on factors such as topic or cost. Wasm is presented as useful for sandboxing and portable workloads, including smaller CPU-suitable models, while larger parallel inference generally benefits from GPUs. TornadoVM translates specially written Java code into GPU kernels; it does not run arbitrary Java frameworks such as Spring on the GPU.

## Source excerpt

In this episode, Edoardo Vacchi discusses the evolving landscape of WebAssembly on the backend, running Java workloads on GPUs using TornadoVM, and optimizing distributed AI inference using llm-d and vLLM on Kubernetes. Edoardo explains how llm-d manages cache locality (KV and prefix caches), request routing, and batch inference across Kubernetes clusters to maximize expensive GPU utilization. By Edoardo Vacchi