# RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts

DevFeed: [RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts](<https://devfeed.tech/articles/ramp-runtime-aware-megakernel-polymorphism-for-mixture-of-experts-113754.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2604.26039>)

Author: Vyom Sharma, Debajyoti Datta

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [batch](<https://devfeed.tech/tags/batch.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [offline](<https://devfeed.tech/tags/offline.md>), [polymorphism](<https://devfeed.tech/tags/polymorphism.md>), [ramp](<https://devfeed.tech/tags/ramp.md>), [runtime](<https://devfeed.tech/tags/runtime.md>)

## AI overview

RaMP addresses a limitation of offline GPU-kernel autotuning for Mixture-of-Experts: the best configuration depends on how the router distributes tokens across experts and GPUs, information unavailable until routing runs. It compiles top configurations into one polymorphic kernel, then uses a small on-GPU cost model to choose among them in a few microseconds based on the actual split. On serving traffic not used to fit the model, RaMP's choices were up to 1.45x faster than vLLM's offline autotuner for the same kernel on batches of 128 tokens or more (1.27x across all batches), and within 3.5% of exhaustive configuration search. The combined kernel and selection method made the MoE layer up to 1.84x faster than vLLM's Triton MoE at 128 tokens or more; in vLLM, time per output token improved by up to 1.11x over the fastest stock MoE backend under tensor and expert parallelism.

## Source excerpt

arXiv:2604.26039v2 Announce Type: replace-cross Abstract: LLM serving systems autotune their GPU kernels offline by timing candidate kernel configurations, like tile sizes, at a range of batch sizes. This works when the number of tokens determines a kernel's work, as in dense layers. However, we show that it does not hold for Mixture-of-Experts (MoE) layers where the router decides how the tokens are split across experts and GPUs. We find that the best kernel configuration depends on this split, and not only on the number of tokens--under expert parallelism, choosing the configuration for each GPU's actual token distribution is up to 1.45x faster than the existing choice. The split is not known until the router runs, so it cannot be looked up in advance. Hence, we introduce RaMP, which chooses the kernel configuration at run time from the routing split itself. RaMP compiles top configurations into one polymorphic kernel, and a small cost model on the GPU picks among them in a few microseconds. On serving traffic its cost model was never fit on, the configurations RaMP's cost model chooses from the kernel's full configuration space are up to 1.45x faster than those vLLM's offline autotuner picks for the same kernel on batches of 128 tokens or more (1.27x over all batches), and within 3.5% of an exhaustive search that times every configuration. RaMP's kernel and choice together make the MoE layer up to 1.84x faster than vLLM's Triton MoE on batches of 128 tokens or more. In vLLM, under both tensor and expert parallelism, RaMP lowers time per output token by up to 1.11x over the fastest stock MoE backend.