# KOSMOS: Self-scheduling Persistent Kernels for Communication-Compute Overlap

DevFeed: [KOSMOS: Self-scheduling Persistent Kernels for Communication-Compute Overlap](<https://devfeed.tech/articles/kosmos-self-scheduling-persistent-kernels-for-communication-compute-overlap-120197.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07477>)

Author: Alexander Magro, Aristotle Martin, Ye Wang, Wen Chen

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [kernels](<https://devfeed.tech/topics/kernels.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [amd](<https://devfeed.tech/tags/amd.md>), [amd-instinct-mi355x](<https://devfeed.tech/tags/amd-instinct-mi355x.md>), [collective](<https://devfeed.tech/tags/collective.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compute](<https://devfeed.tech/tags/compute.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [data](<https://devfeed.tech/tags/data.md>), [data-readiness](<https://devfeed.tech/tags/data-readiness.md>), [deepep](<https://devfeed.tech/tags/deepep.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [deepseek-v3](<https://devfeed.tech/tags/deepseek-v3.md>), [demand](<https://devfeed.tech/tags/demand.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [instinct](<https://devfeed.tech/tags/instinct.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [launch](<https://devfeed.tech/tags/launch.md>), [layer](<https://devfeed.tech/tags/layer.md>), [overlap](<https://devfeed.tech/tags/overlap.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>)

## AI overview

KOSMOS overlaps communication and computation in transformer parallelism by using persistent kernel workgroups that switch roles as data become ready, following queues ordered by consumer demand. This avoids a scheduler, per-call cost model, or tuned communication-computation split. It fuses each TP+SP collective with its GEMM and runs an entire expert layer, from router to fold, in one launch per direction. On AMD Instinct MI350X and MI355X, it reports 1.10-1.28x faster TP+SP end-to-end Megatron-LM training in BF16 and 1.10-1.25x in MXFP8. For DeepSeek-V3 and Qwen3-235B proxies, it reports 1.29-1.83x faster end-to-end training than DeepEP.

## Source excerpt

arXiv:2611.07477v1 Announce Type: cross Abstract: Tensor and sequence parallelism (TP+SP) and expert parallelism (EP) put collectives on the critical path of transformer layers. State-of-the-art fused kernels mitigate this with fixed roles or schedules, but this rigidity leaves large idle windows. Predicting a communication-computation split does not remove them either; a predetermined split cannot follow data readiness. We present KOSMOS, a library for self-scheduling persistent kernels whose workgroups use queues ordered by consumer demand to switch between communication and computation roles as data become ready, with no scheduler, per-call cost model, or tuned split. KOSMOS fuses every TP+SP collective of a transformer layer with its GEMM, and runs an entire expert layer, from router to fold, in one launch per direction. Evaluated on AMD Instinct MI350X and AMD Instinct MI355X GPUs, KOSMOS speeds up TP+SP end-to-end training in Megatron-LM by 1.10-1.28x in BF16 and 1.10-1.25x in MXFP8. KOSMOS accelerates end-to-end training of DeepSeek-V3 and Qwen3-235B proxies by 1.29-1.83x over DeepEP.