# MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference

DevFeed: [MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference](<https://devfeed.tech/articles/meshrt-compile-time-governed-wafer-scale-runtime-for-low-latency-high-throughput-inference-123672.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08508>)

Author: Congjie He, Le Xu, Zhan Lu, Yeqi Huang, Haocheng Xiao, Cheng Deng, Lingxiao Ma, Ziming Miao, Fan Yang, Luo Mai

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [AI Factory](<https://devfeed.tech/topics/ai-factory.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [accelerators](<https://devfeed.tech/tags/accelerators.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [batching](<https://devfeed.tech/tags/batching.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [compile](<https://devfeed.tech/tags/compile.md>), [core](<https://devfeed.tech/tags/core.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-os](<https://devfeed.tech/tags/cs-os.md>), [decentralized](<https://devfeed.tech/tags/decentralized.md>), [decode](<https://devfeed.tech/tags/decode.md>), [events](<https://devfeed.tech/tags/events.md>), [execution](<https://devfeed.tech/tags/execution.md>), [inference](<https://devfeed.tech/tags/inference.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [scale](<https://devfeed.tech/tags/scale.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

## AI overview

MeshRT aims to combine low latency with high throughput for LLM inference on wafer-scale accelerators, where runtime-dynamic techniques such as batching and mixture-of-experts can be difficult to support efficiently. Its approach compiles inference events into per-core schedules, enabling asynchronous, decentralized execution with low runtime overhead. On a commodity wafer-scale accelerator, the authors report up to 56x higher throughput than the prior state-of-the-art wafer-scale system. Against measured single-instance SGLang results on an H200, they report 4.5-7.0x lower latency; for decode throughput, they report 1.7-13.0x higher throughput only when those GPU measurements are ideally linearly scaled to an approximately power-matched H200 cluster.

## Source excerpt

arXiv:2611.08508v1 Announce Type: cross Abstract: Wafer-scale accelerators promise ultra-low-latency AI inference, but current system stacks still carry an unsustainably high cost premium. The reason is that many current and emerging inference techniques, such as batching and MoE, introduce runtime dynamism that existing wafer-scale systems cannot support efficiently. As a result, they must choose between two unsatisfactory options: accept suboptimal latency, or preserve low latency by restricting dynamic techniques, thereby limiting the models they can support and reducing aggregate throughput. This tradeoff significantly increases the cost of low-latency serving. We present MeshRT, the first wafer-scale system to achieve both ultra-low latency and high throughput for LLM inference. MeshRT achieves this via a novel system architecture, which we call compile-time governance of runtime dynamism. This architecture features new system abstractions and mechanisms that enable MeshRT to compile dynamic events arising from LLM inference directly into per-core schedules, enabling asynchronous and decentralized execution with low runtime overhead. Implemented on a commodity wafer-scale accelerator, MeshRT preserves ultra-low latency while improving throughput by up to 56$\times$ over the state-of-the-art system for wafer-scale accelerators. Against measured single-instance SGLang results on H200, MeshRT achieves 4.5-7.0$\times$ lower latency. Under ideal linear scaling of those GPU measurements to an approximately power-matched H200 cluster, MeshRT achieves 1.7-13.0$\times$ higher decode system throughput in the low-latency regime.