# expert-parallel

Published articles for expert-parallel.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## A Fast, Programmable GPU-to-CPU Invocation Layer for GPU-Initiated Communication

DevFeed: [A Fast, Programmable GPU-to-CPU Invocation Layer for GPU-Initiated Communication](<https://devfeed.tech/articles/a-fast-programmable-gpu-to-cpu-invocation-layer-for-gpu-initiated-communication-142971.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23848>)

Author: Jiakun Yan, Shiksha Patel, David Sidler, Gengyu Rao, Brandon Potter, Bradford Beckmann

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [gpu-initated](<https://devfeed.tech/topics/gpu-initated.md>)

Tags: [async](<https://devfeed.tech/tags/async.md>), [backend](<https://devfeed.tech/tags/backend.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [code](<https://devfeed.tech/tags/code.md>), [communication](<https://devfeed.tech/tags/communication.md>), [control](<https://devfeed.tech/tags/control.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [device](<https://devfeed.tech/tags/device.md>), [dispatch](<https://devfeed.tech/tags/dispatch.md>), [expert-parallel](<https://devfeed.tech/tags/expert-parallel.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fallback](<https://devfeed.tech/tags/fallback.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpudirect](<https://devfeed.tech/tags/gpudirect.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [latency](<https://devfeed.tech/tags/latency.md>)

### AI overview

KIWI makes reverse offload to host proxy threads a programmable communication path for GPU workloads, rather than treating it as a slower fallback to GPUDirect Async (GDA). The framework targets shared-queue contention, polling overhead, and proxy load imbalance, and reports sub-microsecond one-way device-to-host latency and hundreds of millions of operations per second. Built on KIWI, KIWI-P2P outperforms state-of-the-art GDA implementations in latency and bandwidth. KIWI-EP uses workload-defined operations for mixture-of-experts dispatch and combine, reducing expert-parallel communication latency by up to 2.3x over the best existing backend on AMD GPUs and vLLM time per output token by up to 29%.

### Source excerpt

arXiv:2611.23848v1 Announce Type: new Abstract: Modern GPU workloads increasingly rely on device-initiated communication to let GPU kernels communicate directly without synchronizing with the host at kernel boundaries. Existing libraries commonly favor placing transport control on GPU threads, an approach known as GPUDirect Async (GDA), and treat reverse offload to host proxy threads as a slower fallback. We argue that reverse offload can outperform GDA and should be a first-class communication path. We further propose exposing reverse offload as a programming interface that lets users define workload-specific communication operations. We present KIWI, a programmable reverse-offload framework that achieves sub-microsecond one-way device-to-host latency and hundreds of millions of operations per second. We address practical challenges of scaling reverse offload, including contention on shared queues, polling overhead, and load imbalance across proxies. KIWI-P2P, a SHMEM-like library built on KIWI, outperforms state-of-the-art GDA implementations in both latency and bandwidth. More importantly, KIWI-EP redesigns low-latency mixture-of-experts dispatch and combine kernels around workload-defined operations, lowering expert-parallel communication latency by up to 2.3x over the best existing backend on AMD GPUs and time per output token in vLLM by up to 29%. The source code of KIWI is available at https://github.com/AMDResearch/kiwi.

## Harvesting Expert-Parallel Bubbles for Fine-Tuning

DevFeed: [Harvesting Expert-Parallel Bubbles for Fine-Tuning](<https://devfeed.tech/articles/harvesting-expert-parallel-bubbles-for-fine-tuning-142803.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24496>)

Author: Oana Balmau, Saurav Banka, Jiaxuan Chen, Anne-Marie Kermarrec, Ramya Narasimha Prabhu, Rafael Pires, Mathis Randl, Martijn de Vos

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [a100](<https://devfeed.tech/tags/a100.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [bubble](<https://devfeed.tech/tags/bubble.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [colocation](<https://devfeed.tech/tags/colocation.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-os](<https://devfeed.tech/tags/cs-os.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [engine](<https://devfeed.tech/tags/engine.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [expert-parallel](<https://devfeed.tech/tags/expert-parallel.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>)

### AI overview

BubbleTea uses per-GPU idle periods created by uneven expert-parallel token routing to run fine-tuning alongside LLM serving. It predicts each GPU's available bubble at every layer and schedules short fine-tuning chunks that reuse serving weights; compiler passes derive the chunks from the served model without porting its architecture to a separate training engine. On three MoE models served across four A100 GPUs, BubbleTea delivered 1.9-6.2x the fine-tuning throughput of LLMStation and kept p90 inference latency within 10% of serving alone in 50 of 63 workloads, versus 44 for LLMStation. On bursty production traces, it trained 3.3x faster than LLMStation and 14x faster than a stock trainer under CUDA MPS, with p90 latency within 7% of serving alone.

### Source excerpt

arXiv:2611.24496v1 Announce Type: cross Abstract: GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside serving. Many leading LLMs now use a Mixture-of-Experts (MoE) architecture, which adds a new source of idle time. Under expert parallelism, MoE experts are spread across GPUs, and because the router sends uneven numbers of tokens to each GPU, lightly loaded GPUs wait for the most loaded one. Even after applying load balancing solutions, such bubbles still take 8--13% of prefill time. We present BubbleTea, the first colocation system that leverages expert-parallel bubbles for fine-tuning. Unlike existing colocation systems, which make one scheduling decision for all GPUs per forward pass, BubbleTea predicts each GPU's bubble at every layer from the router's token assignment, and fills it with short chunks of fine-tuning work that reuse the weights already loaded for serving. We also introduce compiler passes that automatically derive these chunks from the served LLM as-is, without porting each model architecture to a separate training engine. Our evaluation on three MoE models, served on four A100 GPUs, shows that fine-tuning throughput with BubbleTea is 1.9--6.2$\times$ faster than with LLMStation, a state-of-the-art colocation system. BubbleTea keeps p90 inference latency within 10% of serving alone in 50 of 63 workloads, compared to 44 for LLMStation. On bursty production traces, BubbleTea trains 3.3$\times$ faster than LLMStation and 14$\times$ faster than a stock trainer colocated under CUDA MPS, with p90 inference latency staying within 7% of serving alone.