# GPU optimization

Techniques for improving software performance on GPUs through parallel execution, memory use, and instruction efficiency.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## AscMLA: Unlocking FP4 Efficiency for Long-Context MLA Decoding through Asymmetric Scaling

DevFeed: [AscMLA: Unlocking FP4 Efficiency for Long-Context MLA Decoding through Asymmetric Scaling](<https://devfeed.tech/articles/ascmla-unlocking-fp4-efficiency-for-long-context-mla-decoding-through-asymmetric-scaling-146129.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26106>)

Author: Zechun Zhou, Zhiyong Zheng, Xiaomin Zhang, Lin Ye, Han Li, Jingwei Sun, Guangzhong Sun, Jing Li

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>), [Decoding](<https://devfeed.tech/topics/decoding.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mla](<https://devfeed.tech/tags/mla.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

AscMLA addresses the mismatch between K/V scaling requirements when FP4 is applied to MLA's shared latent. It combines asymmetric K/V scaling with pipelined scale delivery to process one FP4 payload directly, avoiding the dequantization or separate encodings used by existing approaches, which add execution or storage overhead. The abstract reports 38.2% less cache memory than FP8 and geometric-mean decoding speedups of 1.35x over TRT-LLM's FP8 kernels and 1.26x over CAKE's NVFP4 MLA kernels.

### Source excerpt

arXiv:2611.26106v1 Announce Type: new Abstract: Long-context decoding repeatedly reads a growing KV cache. FP4 reduces cache storage and traffic but needs block scaling to handle its limited dynamic range. Applying FP4 to multi-head latent attention (MLA) requires reconciling different K/V scaling requirements on a shared latent. Existing approaches address this mismatch through dequantization or separate encodings, adding execution or storage overhead. We present AscMLA, which combines asymmetric K/V scaling with pipelined scale delivery to process a single FP4 payload directly. AscMLA uses 38.2% less cache memory than FP8 and achieves geometric-mean speedups of 1.35x over TRT-LLM's FP8 and 1.26x over CAKE's NVFP4 MLA kernels.

## RSIBench-Infra: Can Agents Optimize Their Own Training and Inference Infrastructure at Production Scale?

DevFeed: [RSIBench-Infra: Can Agents Optimize Their Own Training and Inference Infrastructure at Production Scale?](<https://devfeed.tech/articles/rsibench-infra-can-agents-optimize-their-own-training-and-inference-infrastructure-at-production-scale-143519.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24395>)

Author: Hanlin Wu, Yifei Wang, Zhicheng Cai, Huan-ang Gao, Wei-Ying Ma, Ya-Qin Zhang, Jingjing Liu, Hao Zhou

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [inference](<https://devfeed.tech/tags/inference.md>), [infra](<https://devfeed.tech/tags/infra.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [production](<https://devfeed.tech/tags/production.md>), [scale](<https://devfeed.tech/tags/scale.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

RSIBench-Infra evaluates whether agents can improve end-to-end training and inference performance under production workloads. It covers 20 tasks across model families and modalities, with deployments of up to 32 NVIDIA GPUs or Ascend NPUs. Agents begin from runnable baselines based on official recipes and may change the engine, kernel, or instruction levels; OCI image-layer submissions enable reproducible deployment, and independent evaluation checks end-to-end performance against task-specific quality constraints. Across task results, 66.2% achieved validated speedups above 1.10x. The abstract reports that agents can improve production-scale performance, while finding effective strategies and turning local gains into consistent end-to-end improvements remain challenging.

### Source excerpt

arXiv:2611.24395v1 Announce Type: new Abstract: Improving the efficiency of training and inference infrastructure is a practical component of recursive self-improvement. Existing benchmarks for agent-driven optimization largely focus on individual software layers such as kernel, leaving unclear whether agents can deliver end-to-end performance gains under production workloads. We introduce RSIBench-Infra, a benchmark of 20 training and inference tasks spanning diverse model families and modalities, with deployments of up to 32 accelerators on NVIDIA GPUs and Ascend NPUs. Agents start from runnable baselines derived from official recipes and can modify the software stack across engine, kernel, and instruction levels. An OCI image-layer submission protocol captures these changes for reproducible deployment, while independent evaluation measures end-to-end performance under task-specific quality constraints. Across all task results, 66.2% exceed $1.10\times$ validated speedup. These results demonstrate that current agents can improve production-scale training and inference performance, but identifying effective optimization strategies and consistently translating local improvements into end-to-end gains remain challenging.

## A Runtime System for Mitigating Crossbar Port Contention on SM-Partitioned GPUs

DevFeed: [A Runtime System for Mitigating Crossbar Port Contention on SM-Partitioned GPUs](<https://devfeed.tech/articles/a-runtime-system-for-mitigating-crossbar-port-contention-on-sm-partitioned-gpus-142983.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24458>)

Author: Ziwen Wang, Qi Luo, Kunlin Li, Changxi Liu, Dongsheng Wang, Yun Chen

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [CUDA](<https://devfeed.tech/topics/cuda.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [latency](<https://devfeed.tech/tags/latency.md>), [memory-bandwidth](<https://devfeed.tech/tags/memory-bandwidth.md>), [port](<https://devfeed.tech/tags/port.md>), [profiling](<https://devfeed.tech/tags/profiling.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [sm](<https://devfeed.tech/tags/sm.md>), [system](<https://devfeed.tech/tags/system.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

GPCPlace mitigates crossbar-port contention that persists when CUDA Green Contexts partition GPU workloads across SMs. Spreading workloads across GPCs can make latency-critical requests queue behind background requests at shared ports; packing them into separate GPCs avoids that contention but reduces the ports available to latency-critical requests and can limit bandwidth. Since neither placement is consistently faster, GPCPlace profiles workloads on the target GPU to estimate slowdowns and dynamically chooses between spreading and packing. It hooks CUDA driver kernel launches to set SM masks without application changes. In 112 workload cases across 14 GPUs, it improved latency-critical performance over Green Contexts by up to 1.15x on average and 2.38x maximum, without degrading throughput.

### Source excerpt

arXiv:2611.24458v1 Announce Type: new Abstract: To avoid scheduling delays when running concurrent workloads on a GPU, Streaming Multiprocessor (SM) partitioning mechanisms such as CUDA Green Contexts restrict each workload to separate SMs. Despite this partitioning, concurrent workloads still significantly slow down latency-critical workloads, which recent studies attribute to shared L2 cache and memory contention. However, contention also occurs at the crossbar ports connecting each Graphics Processing Cluster (GPC) of SMs to the L2 cache, as all SMs in a GPC share one port. Green Contexts spread workloads across GPCs, placing concurrent workloads in the same GPCs and forcing latency-critical memory requests to queue behind background requests. Packing workloads into separate GPCs avoids this contention, but forces latency-critical memory requests through fewer ports, limiting memory bandwidth. Consequently, neither spreading nor packing is always faster across workloads and GPUs. Based on these observations, we present GPCPlace, a runtime system that dynamically selects between spreading and packing by profiling workloads on the target GPU to estimate both slowdowns, and hooks kernel launches in the CUDA driver to set SM masks without application modifications. Across 14 GPUs and 112 workload cases, GPCPlace speeds up latency-critical workloads by up to 1.15x on average and 2.38x maximum over Green Contexts without degrading throughput.

## Harvesting Expert-Parallel Bubbles for Fine-Tuning

DevFeed: [Harvesting Expert-Parallel Bubbles for Fine-Tuning](<https://devfeed.tech/articles/harvesting-expert-parallel-bubbles-for-fine-tuning-142803.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24496>)

Author: Oana Balmau, Saurav Banka, Jiaxuan Chen, Anne-Marie Kermarrec, Ramya Narasimha Prabhu, Rafael Pires, Mathis Randl, Martijn de Vos

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [a100](<https://devfeed.tech/tags/a100.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [bubble](<https://devfeed.tech/tags/bubble.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [colocation](<https://devfeed.tech/tags/colocation.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-os](<https://devfeed.tech/tags/cs-os.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [engine](<https://devfeed.tech/tags/engine.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [expert-parallel](<https://devfeed.tech/tags/expert-parallel.md>), [expert-parallelism](<https://devfeed.tech/tags/expert-parallelism.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>)

### AI overview

BubbleTea uses per-GPU idle periods created by uneven expert-parallel token routing to run fine-tuning alongside LLM serving. It predicts each GPU's available bubble at every layer and schedules short fine-tuning chunks that reuse serving weights; compiler passes derive the chunks from the served model without porting its architecture to a separate training engine. On three MoE models served across four A100 GPUs, BubbleTea delivered 1.9-6.2x the fine-tuning throughput of LLMStation and kept p90 inference latency within 10% of serving alone in 50 of 63 workloads, versus 44 for LLMStation. On bursty production traces, it trained 3.3x faster than LLMStation and 14x faster than a stock trainer under CUDA MPS, with p90 latency within 7% of serving alone.

### Source excerpt

arXiv:2611.24496v1 Announce Type: cross Abstract: GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside serving. Many leading LLMs now use a Mixture-of-Experts (MoE) architecture, which adds a new source of idle time. Under expert parallelism, MoE experts are spread across GPUs, and because the router sends uneven numbers of tokens to each GPU, lightly loaded GPUs wait for the most loaded one. Even after applying load balancing solutions, such bubbles still take 8--13% of prefill time. We present BubbleTea, the first colocation system that leverages expert-parallel bubbles for fine-tuning. Unlike existing colocation systems, which make one scheduling decision for all GPUs per forward pass, BubbleTea predicts each GPU's bubble at every layer from the router's token assignment, and fills it with short chunks of fine-tuning work that reuse the weights already loaded for serving. We also introduce compiler passes that automatically derive these chunks from the served LLM as-is, without porting each model architecture to a separate training engine. Our evaluation on three MoE models, served on four A100 GPUs, shows that fine-tuning throughput with BubbleTea is 1.9--6.2$\times$ faster than with LLMStation, a state-of-the-art colocation system. BubbleTea keeps p90 inference latency within 10% of serving alone in 50 of 63 workloads, compared to 44 for LLMStation. On bursty production traces, BubbleTea trains 3.3$\times$ faster than LLMStation and 14$\times$ faster than a stock trainer colocated under CUDA MPS, with p90 inference latency staying within 7% of serving alone.

## DrainSinkhorn: Instance-Resident Batched Entropic Optimal Transport on GPUs

DevFeed: [DrainSinkhorn: Instance-Resident Batched Entropic Optimal Transport on GPUs](<https://devfeed.tech/articles/drainsinkhorn-instance-resident-batched-entropic-optimal-transport-on-gpus-136964.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.24741>)

Author: Xinyang Wen

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [Combinatorial optimization](<https://devfeed.tech/topics/combinatorial-optimization.md>)

Tags: [batch](<https://devfeed.tech/tags/batch.md>), [block](<https://devfeed.tech/tags/block.md>), [chip](<https://devfeed.tech/tags/chip.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [detection](<https://devfeed.tech/tags/detection.md>), [domain-decomposition](<https://devfeed.tech/tags/domain-decomposition.md>), [execution](<https://devfeed.tech/tags/execution.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [image](<https://devfeed.tech/tags/image.md>), [iteration](<https://devfeed.tech/tags/iteration.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [launch](<https://devfeed.tech/tags/launch.md>), [launches](<https://devfeed.tech/tags/launches.md>), [layers](<https://devfeed.tech/tags/layers.md>), [learning](<https://devfeed.tech/tags/learning.md>), [map](<https://devfeed.tech/tags/map.md>), [matching](<https://devfeed.tech/tags/matching.md>), [optimal-transport](<https://devfeed.tech/tags/optimal-transport.md>), [transport](<https://devfeed.tech/tags/transport.md>)

### AI overview

DrainSinkhorn speeds up batches of small, independent entropic optimal transport problems by making each problem an execution unit: one GPU thread block solves an instance that fits on chip and stops when it converges, freeing capacity for pending work. With a fixed iteration count, multiple instances share a block; the forward map and exact gradient each take one launch. Larger instances can leave later launches after convergence. The abstract reports 2.73-10.75x speedups over OTT-JAX and 5.3-34x over POT on tracking and domain-decomposition workloads, with the largest gains on calls containing thousands of instances. In training layers, it reports 1.05-2.8x over OTT-JAX and 9.9-24x over OTA and GeoTransformer Sinkhorn modules. The abstract also reports reduced label-assignment time and faster GeoTransformer patch matching; it does not provide broader evaluation details.

### Source excerpt

arXiv:2607.24741v4 Announce Type: replace Abstract: Many learning pipelines call entropic optimal transport (OT) inside a loop, and each call is a batch of small, independent problems: one per image in detection label assignment, per patch pair in point-cloud registration, per frame pair in tracking, and per tile in domain decomposition. General GPU solvers launch every Sinkhorn update over the whole batch and run the batch until its slowest problem stops, so practitioners write a kernel for each shape or drop OT. We present DrainSinkhorn, a batched Sinkhorn solver that makes each problem of the batch, an instance, the unit of GPU execution. Wherever an instance fits on chip, one GPU thread block solves it from start to finish and exits as soon as the instance converges, freeing its slot for a pending instance. With a fixed iteration count, several instances share a thread block, and the forward map and its exact gradient take one launch each. Larger instances leave later launches once they converge. On tracking and domain-decomposition workloads, DrainSinkhorn is 2.73-10.75x faster than OTT-JAX, the fastest general batched GPU solver in our comparison, and 5.3-34x faster than POT. The gain is largest on calls with thousands of instances. In training layers it is 1.05-2.8x faster than OTT-JAX and 9.9-24x faster than the Sinkhorn modules of OTA and GeoTransformer, cutting label assignment from 16.3% to 4.7% of each GPU's training step and making GeoTransformer's patch matching 10.4x faster in a training step.

## SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

DevFeed: [SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving](<https://devfeed.tech/articles/splitzip-ultra-fast-lossless-kv-compression-for-disaggregated-llm-serving-136961.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.01708>)

Author: Yipin Guo, Siddharth Joshi

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Compression](<https://devfeed.tech/topics/compression.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [bf16](<https://devfeed.tech/tags/bf16.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [disaggregation](<https://devfeed.tech/tags/disaggregation.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [llm-serving](<https://devfeed.tech/tags/llm-serving.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

SplitZip targets the KV-cache transfer bottleneck in disaggregated LLM serving with a GPU-friendly lossless codec that preserves tensors bitwise and does not modify model execution. It encodes frequent floating-point exponents using a calibrated top-16 fixed-length codebook, while rare exponents go through a sparse position/value escape stream; this avoids online histogramming and enables efficient GPU encoding and decoding. On real BF16 activations, the reported compression and decompression throughputs are 613.3 and 2181.8 GB/s. End-to-end experiments report up to 1.32x faster BF16 KV transfer, 1.30x faster time to first token, and 1.23x higher request throughput. The approach also extends to FP8 KV caches, with up to 1.14x compression over native E5M2.

### Source excerpt

arXiv:2605.01708v4 Announce Type: replace Abstract: Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to load-balance between the compute-bound prefill phase and the memory-bound decode phase. Under this design, prefill workers generate a KV cache that must be transferred to decode workers before generation can begin. With these workers residing on different physical systems, this transfer becomes a significant bottleneck to serving LLMs at scale, especially for long-input and agentic workloads. Existing lossless codecs are unsuitable here as they primarily target offline weight compression, run on CPUs, or use variable-length coding whose compression cannot keep up with KV production during prefill. We introduce SplitZip, a GPU-friendly lossless compressor for KV cache transfer that preserves KV tensors bitwise and integrates into existing serving frameworks without modifying model execution. SplitZip exploits redundancy in floating-point exponents of KV activations, encoding frequent exponent values with fixed-length codes and routing rare exponents through a sparse escape stream of (position, value). A calibrated top-16 exponent codebook eliminates online histogramming, while the regular dense path and sparse escape correction make both encoding and decoding efficient on GPUs. On real BF16 activation tensors, SplitZip achieves $613.3$ GB/s compression throughput and $2181.8$ GB/s decompression throughput, outperforming prior lossless compressors on the critical codec path. End-to-end transfer experiments show up to $1.32\times$ speedup for BF16 KV cache transfer, $1.30\times$ speedup for TTFT, and $1.23\times$ increase in Request Throughput. The same approach extends to FP8 KV caches, providing up to $1.14\times$ compression over native E5M2. Code is available at https://github.com/Intelligent-Microsystems-Lab/SplitZip

## Recognizing Events Is Not Enough: Why Streaming VLMs Struggle in Compositional Instruction Triggering

DevFeed: [Recognizing Events Is Not Enough: Why Streaming VLMs Struggle in Compositional Instruction Triggering](<https://devfeed.tech/articles/recognizing-events-is-not-enough-why-streaming-vlms-struggle-in-compositional-instruction-triggering-137337.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18743>)

Author: Yehua Huang, Penglei Sun, Xiang Li, Jinhao Zhou, Jiawen Wen, Qingqing Xiao, Zihan Yang, Wenqi Wang, Haolan Chen, Yu Xu, Xiaowen Chu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [after](<https://devfeed.tech/tags/after.md>), [agent](<https://devfeed.tech/tags/agent.md>), [atomic](<https://devfeed.tech/tags/atomic.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [device](<https://devfeed.tech/tags/device.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [event](<https://devfeed.tech/tags/event.md>), [events](<https://devfeed.tech/tags/events.md>), [example](<https://devfeed.tech/tags/example.md>), [execution](<https://devfeed.tech/tags/execution.md>), [existing](<https://devfeed.tech/tags/existing.md>), [fine](<https://devfeed.tech/tags/fine.md>), [flash](<https://devfeed.tech/tags/flash.md>), [frame](<https://devfeed.tech/tags/frame.md>), [gap](<https://devfeed.tech/tags/gap.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [rtx-5060](<https://devfeed.tech/tags/rtx-5060.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [vlms](<https://devfeed.tech/tags/vlms.md>)

### AI overview

Recognizing separate events is not enough for streaming video assistants: they must track when temporal conditions are met to notify at the right moment or remain silent. The paper introduces CoTriggerBench, which tests this using temporally constrained instructions, executable event plans, and replay of annotated source events. Its proposed real-time VLM agent, TEMPI, breaks instructions into event queries and temporal constraints, then uses Scenario Memory to track progress across observations beyond the detector's visual window; relevant events update that state even when no notification is due. On CoTriggerBench, TEMPI reports 44.4% instruction-level success, 15.6 percentage points above Qwen3.8-Omni-Flash-Realtime and 18.1 above MOSS-VL-Realtime. The authors also report a 4.90x speedup for an optimized detector on an 8 GB RTX 5060, using overlapping visual encoding and LLM prefix prefill with incoming frames.

### Source excerpt

arXiv:2611.18743v1 Announce Type: cross Abstract: Recognizing individual events does not guarantee correct execution of compositional instructions in streaming video. For example, given the request "Notify me when the pot boils after someone turns on the stove and leaves the kitchen", existing models may recognize the individual events yet notify too early or miss the intended moment, failing to track how their temporal conditions are satisfied over time. To evaluate this gap, we introduce CoTriggerBench, which composes annotated video events into temporally constrained instructions and verifies notification timing and silence through executable event plans and source-event replay. To address this challenge, we propose a real-time VLM agent, TEMPI. TEMPI decomposes instructions into atomic event queries and temporal constraints, and uses Scenario Memory to continuously maintain task progress as event evidence arrives, allowing earlier observations to support subsequent decisions beyond the detector's visual window. Even when no notification is required, relevant events continue to update the scenario state. On CoTriggerBench, TEMPI achieves 44.4% instruction-level success, outperforming the strongest closed-source and fine-tuned open-source VLM baselines by 15.6 (Qwen3.8-Omni-Flash-Realtime) and 18.1 (MOSS-VL-Realtime) percentage points, respectively. To support efficient on-device streaming inference, we further optimize the streaming inference pipeline by overlapping partial visual encoding and LLM prefix prefill with frame arrival. On an 8 GB RTX 5060, our optimized detector implementation achieves a 4.90$\times$ speedup.

## FlashSampling: Fast and Memory-Efficient Exact Sampling

DevFeed: [FlashSampling: Fast and Memory-Efficient Exact Sampling](<https://devfeed.tech/articles/flashsampling-fast-and-memory-efficient-exact-sampling-134969.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.15854>)

Author: Tomas Ruiz, Zhen Qin, Yifan Zhang, Xuyang Shen, Yiran Zhong, Mengdi Wang

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [memory](<https://devfeed.tech/tags/memory.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

FlashSampling fuses exact categorical sampling into the language-model head matrix multiplication, avoiding materialization of the logits tensor in HBM. It processes logits in on-chip tiles, adds Gumbel noise, retains each tile's maximizer, then reduces across tiles. For tensor-parallel decoding, it replaces logits all-gather with streaming peer-to-peer writes that overlap communication with computation and memory loads across up to eight GPUs; the abstract reports near-ideal scaling at large batch sizes. The method's exactness relies on argmax decomposition and hierarchical factorization for grouped variants. Kernel speedups are reported across H100, H200, B200, and B300 GPUs; end-to-end vLLM tests reduce time per output token by up to 10% on the tested models.

### Source excerpt

arXiv:2603.15854v4 Announce Type: replace-cross Abstract: Sampling from a categorical distribution is mathematically simple, but in large-vocabulary decoding, it often triggers extra memory traffic and extra kernels after the LM head. We present FlashSampling, an exact sampling primitive that fuses sampling into the LM-head matmul and never materializes the logits tensor in HBM. The method is simple: compute logits tile-by-tile on chip, add Gumbel noise, keep only one maximizer per row and per vocabulary tile, and finish with a small reduction over tiles. In tensor-parallel decoding, FlashSampling replaces the all-gather of logits with streaming peer-to-peer writes: This overlaps GPU-to-GPU communication with computation and HBM loads across up to 8 GPUs, with near-ideal scaling at large batch sizes. Our kernel is exact because argmax decomposes over partitions; grouped variants for online and tensor-parallel settings are exact by hierarchical factorization of the categorical distribution. FlashSampling demonstrates kernel-level speedups on decode workloads across 4 different datacenter GPUs (H100, H200, B200, B300), and in end-to-end vLLM experiments, it reduces time per output token by up to $10%$ on the models we test. These results show that exact sampling, with no approximation, can be integrated into the matmul itself, consolidating the bandwidth-bound sampling step in an efficient epilogue.

## Efficient Poincar\\'e Convolutions through Patch-Statistics Factorization

DevFeed: [Efficient Poincar\\'e Convolutions through Patch-Statistics Factorization](<https://devfeed.tech/articles/efficient-poincar-e-convolutions-through-patch-statistics-factorization-135868.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18125>)

Author: Luk\\'a\\v{s} Jochymek, Mark\\'eta Va\\v{s}inkov\\'a, Petr Gajdo\\v{s}

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [Statistics](<https://devfeed.tech/topics/statistics.md>)

Tags: [1-4](<https://devfeed.tech/tags/1-4.md>), [1-5](<https://devfeed.tech/tags/1-5.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [based](<https://devfeed.tech/tags/based.md>), [box](<https://devfeed.tech/tags/box.md>), [change](<https://devfeed.tech/tags/change.md>), [channel](<https://devfeed.tech/tags/channel.md>), [classification](<https://devfeed.tech/tags/classification.md>), [complete](<https://devfeed.tech/tags/complete.md>), [compute](<https://devfeed.tech/tags/compute.md>), [convolution](<https://devfeed.tech/tags/convolution.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ne](<https://devfeed.tech/tags/cs-ne.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [execution](<https://devfeed.tech/tags/execution.md>), [existing](<https://devfeed.tech/tags/existing.md>), [falls](<https://devfeed.tech/tags/falls.md>), [faster](<https://devfeed.tech/tags/faster.md>), [filter](<https://devfeed.tech/tags/filter.md>), [patch](<https://devfeed.tech/tags/patch.md>), [statistics](<https://devfeed.tech/tags/statistics.md>)

### AI overview

The proposed Poincaré convolution computes each patch's squared norm and filter inner products with a native convolution and a single-channel box filter, then evaluates the clipped score without mapping patches to the Poincaré ball. The abstract reports exact evaluation of the specified score and single-precision outputs and gradients that match double precision at least as closely as the explicit reference. On an RTX A6000, full training updates were 1.5-1.8x faster than explicit implementations that map patches to the ball, and 1.2-1.4x faster than one that skips that mapping. Paired runs report 2.0x and 1.8x faster updates with no consistent change in final task quality; a U-Net scaling study found 38-55% lower peak memory. Replacing the official PyTorch Hyperbolic U-Net convolutions yielded a 1.4x faster training step and half the memory. These results are reported for the tested models and software stacks.

### Source excerpt

arXiv:2611.18125v1 Announce Type: new Abstract: Published implementations of the Poincar\'e convolution in Hyperbolic Neural Networks++ (HNN++) map every overlapping image patch to the Poincar\'e ball before evaluating its score. We show that the clipped score depends on each patch only through two statistics, its squared norm and its inner products with the filters, which a native convolution and a single-channel box filter compute directly. The factorized layer therefore evaluates the specified score exactly, without constructing ball-valued patches, and its single-precision outputs and gradients agree with double-precision evaluations at least as closely as those of the explicit reference. On an RTX A6000 under two versions of JAX, complete training updates of a Poincar\'e U-Net and a Poincar\'e ResNet-20 are 1.5 to 1.8 times faster than the faster of two explicit implementations that map patches to the ball (1.2 to 1.4 times against an explicit implementation that skips this mapping); on a V100, the gain over the gather-based one persists as the ResNet-20 is widened eightfold. In 15 paired training runs per task, updates are 2.0 and 1.8 times faster than a reference that follows the published execution order, with no consistent change in final segmentation or classification quality, and peak memory falls by 38 to 55% in a U-Net scaling study. Replacing the convolutions of the official PyTorch Hyperbolic U-Net makes its training step 1.4 times faster with half the memory. Evaluating an existing layer from its patch statistics thus makes Poincar\'e convolutional networks cheaper to train in the tested models and software stacks.

## SIMTurn: Lightweight Divergent Subgroup Scheduling for Intra-Warp Synchronization

DevFeed: [SIMTurn: Lightweight Divergent Subgroup Scheduling for Intra-Warp Synchronization](<https://devfeed.tech/articles/simturn-lightweight-divergent-subgroup-scheduling-for-intra-warp-synchronization-134890.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17827>)

Author: Nikhil Rout, Joe Samuel Rosh, Blaise Tine

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [scheduling](<https://devfeed.tech/topics/scheduling.md>), [synchronization](<https://devfeed.tech/topics/synchronization.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-pl](<https://devfeed.tech/tags/cs-pl.md>), [deadlock](<https://devfeed.tech/tags/deadlock.md>), [faster](<https://devfeed.tech/tags/faster.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [independent](<https://devfeed.tech/tags/independent.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [lightweight](<https://devfeed.tech/tags/lightweight.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [progress](<https://devfeed.tech/tags/progress.md>), [rtl](<https://devfeed.tech/tags/rtl.md>), [scheduler](<https://devfeed.tech/tags/scheduler.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [serialization](<https://devfeed.tech/tags/serialization.md>), [simt](<https://devfeed.tech/tags/simt.md>), [simulator](<https://devfeed.tech/tags/simulator.md>), [software](<https://devfeed.tech/tags/software.md>), [synchronization](<https://devfeed.tech/tags/synchronization.md>), [warp](<https://devfeed.tech/tags/warp.md>)

### AI overview

SIMTurn addresses deadlock under stack-based SIMT reconvergence, where a spinning divergent subgroup can block a sibling that holds a needed resource. The compiler inserts a yield instruction at potentially blocking loops, and a scheduler uses a bounded pool of resumable subgroups to let a waiting subgroup yield to a progressing sibling, while retaining stack-based reconvergence instead of per-thread program counters. In evaluations on the open-source Vortex GPGPU, SIMTurn added 17-24% scheduler-area overhead over IPDOM, versus 39-91% for per-thread ITS, and stayed within 8% of IPDOM on workloads not needing cross-subgroup progress. On kernels with such parallelism, it tracked ITS closely and was up to 2.7x faster than software-serialized IPDOM.

### Source excerpt

arXiv:2611.17827v1 Announce Type: cross Abstract: Stack-based reconvergence enables efficient SIMT execution with a shared warp program counter and compact control-flow state. However, its fixed serialization of divergent paths can deadlock programs that rely on intra-warp synchronization: a spinning subgroup may prevent a resource-holding sibling from executing. Independent Thread Scheduling (ITS) enables finer-grained progress through per-thread execution state, but introduces additional hardware cost. We present SIMTurn, a lightweight compiler-hardware mechanism that enables forward progress among divergent subgroups while retaining stack-based reconvergence. SIMTurn combines a compiler-inserted yield instruction at potentially blocking loops with a bounded pool of resumable subgroups. The scheduler allows a waiting subgroup to yield to a sibling that can make progress, without requiring independent program counters for every thread. We implement SIMTurn in the compiler, simulator, and RTL on the open-source Vortex GPGPU and evaluate it against baseline IPDOM and per-thread ITS schemes. Across the evaluated GPU thread configurations, SIMTurn introduces only 17-24% scheduler-area overhead relative to IPDOM, compared with 39-91% for ITS, and stays within 8% of IPDOM on workloads that never need cross-subgroup progress. On kernels with genuine cross-subgroup parallelism, SIMTurn tracks ITS closely and is up to 2.7x faster than a software-serialized IPDOM baseline.

## SyclKittens: A Tile Programming Model for Programmers and Coding Agents on Intel GPUs

DevFeed: [SyclKittens: A Tile Programming Model for Programmers and Coding Agents on Intel GPUs](<https://devfeed.tech/articles/syclkittens-a-tile-programming-model-for-programmers-and-coding-agents-on-intel-gpus-134882.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.04277>)

Author: Yehong Jiang, Sheng Chen, Fangwen Fu, Yen-Kuang Chen, Xinmin Tian, Stuart H. Sul, Simran Arora

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [fabric](<https://devfeed.tech/tags/fabric.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inference](<https://devfeed.tech/tags/inference.md>), [intel](<https://devfeed.tech/tags/intel.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [layouts](<https://devfeed.tech/tags/layouts.md>), [llama](<https://devfeed.tech/tags/llama.md>), [multi-gpu](<https://devfeed.tech/tags/multi-gpu.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

In controlled Intel GPU experiments, giving coding agents SyclKittens' hardware-aware tile model improved their GEMM performance over raw SYCL. With the same agent, task and feedback budget, Opus 4.8 reached 82.1% of tuned oneDNN performance using SyclKittens, compared with 57.5% using raw SYCL. The model encodes methods for matrix-engine layouts, L1 prefetching and communication across GPU stacks, while keeping workload-specific scheduling programmable for engineers and agents to refine. Their resulting kernel suite reached approximately 96% of oneDNN's geometric-mean performance across GEMM shapes and ran Llama-3.1-8B inference 1.59x faster than torch.compile on one GPU, and up to 2.91x faster than a matched multi-GPU decode path using Intel oneCCL.

### Source excerpt

arXiv:2610.04277v2 Announce Type: replace Abstract: New AI accelerators arrive before the kernels that make them fast, because peak kernel performance requires architecture-specific expertise in operand pipelines and data-movement techniques. Coding agents can now write, compile, and tune kernels on their own, so they could greatly accelerate kernel development and optimization. What agents produce depends on the interfaces they are given. These interfaces may expose a machine's raw capabilities or encode the known-good methods for exploiting its hardware features efficiently. We test the performance impact of these two interfaces on Intel GPUs through controlled experiments with five coding models on three kernels. With raw SYCL and execution feedback, Opus 4.8, the strongest of the three models tested with Claude Code, writes a single-GPU GEMM kernel reaching only 57.5% of Intel's tuned oneDNN library. We present SyclKittens, a hardware-aware tile programming model for Intel GPUs that encodes these known-good methods, so its operations place operands in matrix-engine layouts, prefetch through the L1 cache, and communicate over the fabric that links GPU stacks into a node. With SyclKittens under the same agent, task, and feedback budget, the agent reaches 82.1%, showing that encoded methods turn hardware capabilities into performance. Workload-specific policies stay programmable, and engineers and agents jointly refine these schedules in SyclKittens to build a kernel suite that reaches approximately 96% of oneDNN in geometric mean across GEMM shapes. The suite runs Llama-3.1-8B inference 1.59 times faster than torch.compile on one GPU and up to 2.91 times faster than a matched multi-GPU decode path on Intel's oneCCL. SyclKittens is open source and available at https://github.com/intel/SyclKittens.

## TIDE: Reclaiming Lost Performance In On-Policy Reinforcement Learning

DevFeed: [TIDE: Reclaiming Lost Performance In On-Policy Reinforcement Learning](<https://devfeed.tech/articles/tide-reclaiming-lost-performance-in-on-policy-reinforcement-learning-134892.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17924>)

Author: Mathew Jacob, Morgan Borjigin-Wang, Kan Zhu, Po Hao Chen, Yi Pan, Andy Ruan, Nayani Modugula, Ethan Lavi, Luis Ceze, Malte Schwarzkopf, Akshay Narayan, Deepti Raghavan, Stephanie Wang

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>)

Tags: [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [inference](<https://devfeed.tech/tags/inference.md>), [migration](<https://devfeed.tech/tags/migration.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [resource-utilization](<https://devfeed.tech/tags/resource-utilization.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

TIDE addresses idle inference capacity and inefficient decoding batches in synchronous, colocated LLM reinforcement learning by combining two techniques: it repurposes rollout GPUs for training while long-tail rollouts finish, and migrates rollouts between inference engines to improve decoding throughput. An adaptive migration policy chooses when to repurpose resources, without bespoke configuration. The abstract reports speedups up to 1.36x over colocated systems and up to 97% of optimal throughput across evaluated applications and hardware configurations, while preserving on-policy training semantics.

### Source excerpt

arXiv:2611.17924v1 Announce Type: cross Abstract: Synchronous, colocated RL post-training of large language models (LLMs) today suffers from severe under-utilization due to long-tail rollouts. Inference engines fall idle waiting for long-tail rollouts or decode at low, inefficient batch sizes. To address this, we present TIDE, an RL training system that preserves on-policy training semantics while recovering resource utilization. TIDE does so by two complementary techniques: (1) dynamically repurposing rollout GPUs for training while long-tail rollouts complete and (2) dynamically migrating rollouts between inference engines to boost decoding throughput. TIDE automatically determines the right time to repurpose rollout resources through an adaptive migration policy, obtaining performance improvements that generalize across multiple applications and hardware configurations without requiring bespoke configuration. In our evaluation, TIDE provides end-to-end speedups of up to 1.36x over colocated systems, outperforming all evaluated on-policy baselines and achieving up to 97% of optimal throughput.

## AMD Leverages AI + HRX Loom In Experiment For Quickly Porting 50 LLMs To Client GPUs

DevFeed: [AMD Leverages AI + HRX Loom In Experiment For Quickly Porting 50 LLMs To Client GPUs](<https://devfeed.tech/articles/amd-leverages-ai-hrx-loom-in-experiment-for-quickly-porting-50-llms-to-client-gpus-132417.md>)

Original publisher: [Read original article](<https://www.phoronix.com/news/AMD-HRX-Loom-Experiment>)

Author: Michael Larabel

Published: 2026-11-13T15:53:00Z

Content type: news

Language: en

Sources: [Phoronix](<https://devfeed.tech/sources/phoronix.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [rocm](<https://devfeed.tech/topics/rocm.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amd](<https://devfeed.tech/tags/amd.md>), [client](<https://devfeed.tech/tags/client.md>), [days](<https://devfeed.tech/tags/days.md>), [desktop-linux](<https://devfeed.tech/tags/desktop-linux.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [halo](<https://devfeed.tech/tags/halo.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [just](<https://devfeed.tech/tags/just.md>), [linux-benchmarking](<https://devfeed.tech/tags/linux-benchmarking.md>), [linux-hardware-benchmarks](<https://devfeed.tech/tags/linux-hardware-benchmarks.md>), [linux-hardware-reviews](<https://devfeed.tech/tags/linux-hardware-reviews.md>), [linux-how-to](<https://devfeed.tech/tags/linux-how-to.md>), [linux-performance](<https://devfeed.tech/tags/linux-performance.md>), [linux-server-benchmarks](<https://devfeed.tech/tags/linux-server-benchmarks.md>), [llms](<https://devfeed.tech/tags/llms.md>), [loom](<https://devfeed.tech/tags/loom.md>), [models](<https://devfeed.tech/tags/models.md>), [open-source-graphics](<https://devfeed.tech/tags/open-source-graphics.md>), [over](<https://devfeed.tech/tags/over.md>), [phoronix](<https://devfeed.tech/tags/phoronix.md>), [phoronix-test-suite](<https://devfeed.tech/tags/phoronix-test-suite.md>), [porting](<https://devfeed.tech/tags/porting.md>), [quickly](<https://devfeed.tech/tags/quickly.md>), [ryzen](<https://devfeed.tech/tags/ryzen.md>), [ryzen-ai](<https://devfeed.tech/tags/ryzen-ai.md>), [strix-halo](<https://devfeed.tech/tags/strix-halo.md>), [ubuntu-benchmarks](<https://devfeed.tech/tags/ubuntu-benchmarks.md>), [ubuntu-hardware](<https://devfeed.tech/tags/ubuntu-hardware.md>)

### AI overview

AMD engineers Jeremy Fowers and Ben Vanik used HRX Loom with an AI agent workflow to port 50 popular Hugging Face models across 18 architectures to AMD client GPUs in three to four days. They report 1.5-2.1x faster 32K-token prompt processing than Llama.cpp's ROCm backend and 1.9x faster than its Vulkan backend. Loom is experimental and supports CPU, GPU, and NPU execution; the work focused on Strix Halo hardware. The excerpt describes the results as evidence of potential for bringing new and niche models to AMD hardware faster.

### Source excerpt

Following the recent release of HRX Loom v0.1 and leveraging an AI agent workflow, a team at AMD experimented with bringing 50 trending Hugging Face models over to AMD client GPUs. They accomplished this feat in just around three days, thanks to their HRX software work paired with the power of Ryzen AI Max "Strix Halo" hardware...

## CuEVM: Scaling Smart Contract Execution on GPUs

DevFeed: [CuEVM: Scaling Smart Contract Execution on GPUs](<https://devfeed.tech/articles/cuevm-scaling-smart-contract-execution-on-gpus-131526.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14916>)

Author: Nhut-Minh Ho, Tien Tuan Anh Dinh, Quang Trung Ta, Fredrik Svantes, Xiaokui Xiao, Beng Chin Ooi

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [smart contract](<https://devfeed.tech/topics/smart-contract.md>), [Ethereum](<https://devfeed.tech/topics/ethereum.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>)

Tags: [accelerate](<https://devfeed.tech/tags/accelerate.md>), [blockchain](<https://devfeed.tech/tags/blockchain.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [data-structures](<https://devfeed.tech/tags/data-structures.md>), [engine](<https://devfeed.tech/tags/engine.md>), [ethereum](<https://devfeed.tech/tags/ethereum.md>), [evm](<https://devfeed.tech/tags/evm.md>), [execution](<https://devfeed.tech/tags/execution.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [performance](<https://devfeed.tech/tags/performance.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [security-testing](<https://devfeed.tech/tags/security-testing.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [smart-contract](<https://devfeed.tech/tags/smart-contract.md>), [transaction](<https://devfeed.tech/tags/transaction.md>)

### AI overview

CuEVM treats EVM transaction execution as massively parallel GPU data processing, using GPU-aware scheduling and novel data structures to support EVM semantics natively rather than relying on host CPU operations for many tasks. The abstract reports 100% pass rates on Ethereum VMTests and 96.2% on the full execution-test suite. On one commodity GPU it executes over 8 million ERC-20 transfers per second, up to two orders of magnitude faster than existing baselines. Its CuEVM-fuzz tool achieves over a million end-to-end executions per second, reported as up to 23x faster than CPU-based and 4.8x faster than GPU-based baselines. These results are abstract-level claims; the supplied excerpt does not provide benchmark setup or limitations.

### Source excerpt

arXiv:2611.14916v1 Announce Type: new Abstract: Smart contracts on Ethereum blockchain process transactions worth billions of US dollars daily. They run on top of Ethereum Virtual Machine (EVM) which executes transactions sequentially. This execution model creates a performance bottleneck, especially for new blockchain workloads with large numbers of transactions, such as security testing. We present CuEVM, a GPU-native and semantically complete EVM execution engine that treats transaction execution as a massively parallel data processing problem. It employs novel data structures and GPU-aware scheduling to accelerate execution. Unlike prior works that transpile the contract and rely on a host CPU for many operations, CuEVM supports complete EVM semantics natively on the GPU. It passes 100% of the Ethereum VMTests and 96.2% of the full execution-test suite. On a single commodity GPU, CuEVM executes over 8 million ERC-20 transfers per second, outperforming existing baselines by up to two orders of magnitude. We build a smart contract fuzzer, called CuEVM-fuzz, on top of CuEVM. The fuzzer achieves over a million end-to-end executions per second, up to 23x and 4.8x higher than the state-of-the-art CPU-based and GPU-based baselines, respectively.

## A Parallel Greedy Algorithm for Maximal b-Matching in Hypergraphs

DevFeed: [A Parallel Greedy Algorithm for Maximal b-Matching in Hypergraphs](<https://devfeed.tech/articles/a-parallel-greedy-algorithm-for-maximal-b-matching-in-hypergraphs-131534.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15615>)

Author: Saeed Bakhshan, Daniel Grosu, Henrik Reinst\\"adtler, Christian Schulz

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Combinatorial optimization](<https://devfeed.tech/topics/combinatorial-optimization.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-ds](<https://devfeed.tech/tags/cs-ds.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [greedy](<https://devfeed.tech/tags/greedy.md>), [greedy-algorithm](<https://devfeed.tech/tags/greedy-algorithm.md>), [matching](<https://devfeed.tech/tags/matching.md>), [parallel](<https://devfeed.tech/tags/parallel.md>)

### AI overview

The proposed \parallelBatchBMatch algorithm computes the lexicographically first maximal b-matching, matching sequential greedy output for a fixed edge order. With uniformly random priorities, it has O(m') expected work, where m' is the number of vertex-edge incidences. It uses at most m rounds for any priority order, and has O(log² m') depth per round with high probability, yielding O(m log² m') total depth with high probability. A CUDA implementation was evaluated on real-world hypergraphs at capacities 1, 3, and 5; for b = 5, it was up to 19.07 times faster than the existing state-of-the-art greedy algorithm.

### Source excerpt

arXiv:2611.15615v1 Announce Type: cross Abstract: We present \parallelBatchBMatch, a parallel batch-vote algorithm for maximal $b$-matching in hypergraphs. The algorithm is deterministic for a given ordering of edges and produces the same output as the sequential greedy algorithm, i.e., the lexicographically first maximal $b$-matching. Under a uniformly random priority order it performs $O(m')$ expected work in the binary fork--join model, where $m'$ is the number of pins, the total number of vertex--edge incidences. The algorithm performs at most $m$ rounds for every priority order, where $m$ is the number of edges. Every round of the algorithm has $O(\log^2 m')$ depth with high probability, and thus the total depth of the algorithm is $O(m \log^2 m')$ with high probability. We implement the parallel algorithm in CUDA and evaluate it on real-world hypergraphs with capacities $b \in \{1, 3, 5\}$. The parallel algorithm computes the same matchings as the greedy algorithms and for the case of $b = 5$ it is up to $19.07$ times faster than the existing state-of-the art greedy algorithm.

## CAFIN: CUDA-accelerated Adaptive Fisher Information Matrix Network Optimization via GPU-Native Kernel Fusion

DevFeed: [CAFIN: CUDA-accelerated Adaptive Fisher Information Matrix Network Optimization via GPU-Native Kernel Fusion](<https://devfeed.tech/articles/cafin-cuda-accelerated-adaptive-fisher-information-matrix-network-optimization-via-gpu-native-kernel-fusion-132026.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14589>)

Author: Maryam Arshad, Muhammad Khuram Shahzad

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [CUDA](<https://devfeed.tech/topics/cuda.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [accelerated](<https://devfeed.tech/tags/accelerated.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [cifar-10](<https://devfeed.tech/tags/cifar-10.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [latency](<https://devfeed.tech/tags/latency.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [native](<https://devfeed.tech/tags/native.md>), [network-optimization](<https://devfeed.tech/tags/network-optimization.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [rtx](<https://devfeed.tech/tags/rtx.md>), [rtx-3060](<https://devfeed.tech/tags/rtx-3060.md>), [ti](<https://devfeed.tech/tags/ti.md>), [tiny](<https://devfeed.tech/tags/tiny.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

CAFIN speeds up AdaFisher by fusing its GPU work while preserving the optimizer's mathematical formulation. Fused CUDA kernels progressively combine FIM preconditioner construction, moving-average updates, gradient preconditioning, momentum, weight decay, and parameter updates; shared-memory reuse, register-resident computation, and in-place operations reduce memory traffic and intermediate allocations. In ResNet-18 experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet using GTX 1660 Ti and RTX 3060 Laptop GPUs, it reduced isolated FIM construction latency by up to 78.4% and optimizer-step latency by 2.1-26.9%, and removed about 18.91 MB of intermediate allocations per step. Numerical results stayed within a predefined FP32 tolerance, with comparable convergence accuracy across the evaluated datasets and random seeds. These findings are limited to the reported models, datasets, and GPUs.

### Source excerpt

arXiv:2611.14589v1 Announce Type: cross Abstract: Second-order optimizers improve deep neural network training using curvature information, but their practical efficiency is limited by GPU execution overhead. AdaFisher reduces Fisher Information Matrix (FIM) approximation costs through diagonal Kronecker structures and exponential moving averages, yet its PyTorch implementation requires sequential CUDA kernel launches, intermediate tensor allocations, and repeated memory accesses. We introduce CAFIN, a GPU-native CUDA kernel fusion framework that improves AdaFisher execution efficiency without changing its optimization formulation. The proposed approach progressively integrates FIM preconditioner construction, exponential moving average updates, gradient preconditioning, momentum accumulation, weight decay, and parameter updates into fused CUDA kernels. Shared-memory reuse, register-resident computation, and in-place operations further reduce memory traffic and intermediate allocations. Experiments using ResNet-18 on CIFAR-10, CIFAR-100, and Tiny ImageNet across NVIDIA GTX 1660 Ti and RTX 3060 Laptop GPUs demonstrate up to 78.4% lower isolated FIM construction latency and 2.1-26.9% lower optimizer-step latency. The implementation also eliminates approximately 18.91 MB of intermediate GPU memory allocations per optimizer step. Numerical validation confirms consistency within the predefined FP32 tolerance, while convergence accuracy remains comparable across evaluated datasets and random seeds. These findings demonstrate that CUDA kernel fusion can substantially accelerate second-order optimization while preserving AdaFisher's underlying mathematical formulation and learning behavior.

## GC-MoE on SM120: Overlapping Expert-Parallel Token Dispatch with Warp-Level FP8xFP4 Block-Scaled MMA on Consumer Blackwell

DevFeed: [GC-MoE on SM120: Overlapping Expert-Parallel Token Dispatch with Warp-Level FP8xFP4 Block-Scaled MMA on Consumer Blackwell](<https://devfeed.tech/articles/gc-moe-on-sm120-overlapping-expert-parallel-token-dispatch-with-warp-level-fp8xfp4-block-scaled-mma-on-consumer-blackwell-131543.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16219>)

Author: Yong Li, Haoyu Wang, Kan Liu, Tao Lan

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Tensor Cores](<https://devfeed.tech/topics/tensor-cores.md>)

Tags: [64-bit](<https://devfeed.tech/tags/64-bit.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [arena](<https://devfeed.tech/tags/arena.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [decode](<https://devfeed.tech/tags/decode.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [dispatch](<https://devfeed.tech/tags/dispatch.md>), [experts](<https://devfeed.tech/tags/experts.md>), [faster](<https://devfeed.tech/tags/faster.md>), [flashinfer](<https://devfeed.tech/tags/flashinfer.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [graph](<https://devfeed.tech/tags/graph.md>), [moe](<https://devfeed.tech/tags/moe.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [token](<https://devfeed.tech/tags/token.md>), [warp](<https://devfeed.tech/tags/warp.md>)

### AI overview

GC-MoE's rebuilt SM120 operator overlaps expert-parallel token dispatch, grouped computation and result return on PCIe. SM120 exposes block-scaled tensor cores through warp-level MMA only, so the design uses FP4 weights in the register A operand and FP8 activations in B, with adaptive token tiles, expert-granular streaming dispatch, batched bulk pulls and deduplicating caching. It also routes BF16 returns through resident relay copiers and folds shared experts into the compute role. On 8 RTX PRO 5000 GPUs, it was 1.522x faster than FlashInfer CuTe DSL W4A8 MegaMoE and 1.387x faster than Cake MegaMoE across 18 decode points, winning all 18 against each; it was 1.388x faster than Cake with shared experts fused. The abstract reports 94 numerical cases and 752 per-rank checks passing, with cross-operator cosine similarity of at least 0.9999997.

### Source excerpt

arXiv:2611.16219v1 Announce Type: new Abstract: Expert-parallel Mixture-of-Experts decoding is latency bound: each layer serializes a token exchange, a grouped GEMM over routed experts, and a combine, while only a few tokens per rank are in flight. Prior GC-MoE fused these phases into one persistent, Green-Context-partitioned CUDA Graph on Hopper (SM90) and datacenter Blackwell (SM100). This paper rebuilds the operator for SM120 (compute capability 12.0, e.g. RTX PRO 5000) over PCIe. SM120 exposes block-scaled tensor cores only through warp-level mma.sync.aligned.kind::mxf8f6f4: no tensor memory, no two-CTA MMA, no cluster multicast. We rebuild the kernel around a swap-AB warp-level block-scaled MMA with FP4 weights in the register A operand and FP8 activations in B, plus SM120-specific mechanisms: scale-byte selection through MMA immediates, a compact-weights path, an adaptive token-tile geometry, an expert-granular streaming dispatch with a batched bulk-pull engine and a 64-bit deduplicating pull cache, full-width BF16 returns drained by resident relay copiers over a position-major arena, and shared experts folded into the compute role. On 8xRTX PRO 5000 over PCIe the operator is 1.522x and 1.387x faster than the FlashInfer CuTe DSL W4A8 MegaMoE and the Cake MegaMoE across 18 decode points (18/18 wins each), and 1.388x faster than Cake with shared experts fused; all 94 numerical cases and 752 per-rank checks pass, at cross-operator cosine similarity of at least 0.9999997.

## CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning

DevFeed: [CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning](<https://devfeed.tech/articles/cuda-l2-surpassing-cublas-performance-for-matrix-multiplication-through-reinforcement-learning-131861.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2512.02551>)

Author: Songqiao Su, Xiaoya Li, Albert Wang, Guoyin Wang, Jiwei Li, Chris Shum

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [configuration](<https://devfeed.tech/tags/configuration.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cublas](<https://devfeed.tech/tags/cublas.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [general](<https://devfeed.tech/tags/general.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rl](<https://devfeed.tech/tags/rl.md>)

### AI overview

CUDA-L2 uses large language models and reinforcement learning to optimize half-precision matrix-multiplication CUDA kernels, rewarding the system with execution speed as it searches 1,000 configurations. The abstract reports gains over torch.matmul, cuBLAS, cuBLASLt's heuristic selection, and cuBLASLt autotuning. In offline runs, average speedups are 22.0%, 19.2%, 16.8%, and 11.4%, respectively; in server-mode runs with randomly spaced executions, they rise to 28.7%, 26.0%, 22.4%, and 15.9%. The reported comparison distinguishes cuBLAS's optimal NN and TN layouts and autotuning over up to 100 cuBLASLt candidates. These are abstract-level claims; benchmark setup details are not included in the supplied excerpt.

### Source excerpt

arXiv:2512.02551v5 Announce Type: replace-cross Abstract: In this paper, we propose CUDA-L2, a system that combines large language models (LLMs) and reinforcement learning (RL) to automatically optimize Half-precision General Matrix Multiply (HGEMM) CUDA kernels. Using CUDA execution speed as the RL reward, CUDA-L2 automatically optimizes HGEMM kernels across 1,000 configurations. CUDA-L2 systematically outperforms major matmul baselines to date, from the widely-used torch.matmul to state-of-the-art Nvidia's closed-source libraries, i.e., cuBLAS, cuBLASLt. In offline mode, where kernels are executed consecutively without time intervals, CUDA-L2 yields +22.0% over torch.matmul on average; +19.2% over cuBLAS using the optimal layout configuration (normal-normal NN and transposed-normal TN); +16.8% over cuBLASLt-heuristic, which queries cuBLASLt library and selects the algorithm based on the heuristic's suggestion; and +11.4% over the most competitive cuBLASLt-AutoTuning model, which selects the fastest algorithm from up to 100 candidates from cuBLASLt's suggestions. In server mode, where kernels are executed at random intervals simulating real-time inference, the speedups further increase to +28.7%, +26.0%, +22.4%, and +15.9% for torch.matmul, cuBLAS, cuBLASLt-heuristic, and cuBLASLt-AutoTuning respectively. CUDA-L2 shows that even the most performance-critical, heavily-optimized kernels like HGEMM can be improved through LLM-guided RL automation by systematically exploring configuration spaces at scales impractical for humans. Project and code can be found at github.com/ornith-ai/CUDA-L2

## Read Once, Answer Many: Parallel Prompting for Shared-Context LLM Inference

DevFeed: [Read Once, Answer Many: Parallel Prompting for Shared-Context LLM Inference](<https://devfeed.tech/articles/read-once-answer-many-parallel-prompting-for-shared-context-llm-inference-131531.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15130>)

Author: Zekun Zhao, Jeffrey Flanigan

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [8k](<https://devfeed.tech/tags/8k.md>), [a100](<https://devfeed.tech/tags/a100.md>), [attention](<https://devfeed.tech/tags/attention.md>), [backend](<https://devfeed.tech/tags/backend.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [cache](<https://devfeed.tech/tags/cache.md>), [caching](<https://devfeed.tech/tags/caching.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [data-labeling](<https://devfeed.tech/tags/data-labeling.md>), [decode](<https://devfeed.tech/tags/decode.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [engine](<https://devfeed.tech/tags/engine.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [flashinfer](<https://devfeed.tech/tags/flashinfer.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [prompting](<https://devfeed.tech/tags/prompting.md>)

### AI overview

Parallel Prompting reduces repeated KV-cache reads when answering many independent questions about one long document: it decodes the questions as separate streams in a single request, reading the shared document once per step. A stream-ID mask inside the attention kernel keeps each answer conditioned on its own question and the document, without a dense mask tensor. The SGLang implementation adds about 80 lines across three existing files plus a small engine package; fixed-shape decode steps can be captured as a CUDA graph. With CodeLlama-7B, 8k-16k-token documents, and 64-256 questions, it ran 3.7x faster than stock SGLang with FlashInfer on an A100 and 5.9-6.0x faster on an RTX A6000 across two SGLang releases. It was faster in all 144 reported runs; the evidence is limited to these tested configurations.

### Source excerpt

arXiv:2611.15130v1 Announce Type: cross Abstract: LLM workloads sometimes ask many independent questions about the same long document, as in document QA, data labeling and LLM-based evaluation. Prefix caching removes the redundant prefill shared by these queries, but each decode step still reads the document's key-value (KV) cache once per question, so decoding is bound by memory bandwidth and scales with the number of questions. We present Parallel Prompting, which decodes P questions as P independent streams of a single request all at once, fixing this issue. In our method, every step reads the shared document once for all P streams, and a stream-id attention mask, evaluated inside the attention kernel, keeps each answer conditioned only on the document and its own question. We implement Parallel Prompting in SGLang by adding about 80 lines to three existing files and a small engine package; the mask needs no dense tensor, and every decode step has a fixed shape that is captured as a CUDA graph. With CodeLlama-7B, 8k-16k-token documents and 64-256 questions per document, Parallel Prompting is 3.7x faster than stock SGLang with its FlashInfer backend on an A100 and 5.9-6.0x faster on an RTX A6000, on two SGLang releases, including the most recent release. Timed on identical inputs, it is faster than stock SGLang in all 144 runs (2 GPUs x 2 releases x 6 workloads x 6 input sets). Code: https://github.com/jlab-nlp/parallel-prompting

## Parallel Computing Systems for the Optimal Selection of Process Equipment

DevFeed: [Parallel Computing Systems for the Optimal Selection of Process Equipment](<https://devfeed.tech/articles/parallel-computing-systems-for-the-optimal-selection-of-process-equipment-131545.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16290>)

Author: Andrey B. Borisenko

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [Combinatorial optimization](<https://devfeed.tech/topics/combinatorial-optimization.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [batch](<https://devfeed.tech/tags/batch.md>), [branch](<https://devfeed.tech/tags/branch.md>), [capital](<https://devfeed.tech/tags/capital.md>), [chemical-engineering](<https://devfeed.tech/tags/chemical-engineering.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [computing](<https://devfeed.tech/tags/computing.md>), [core](<https://devfeed.tech/tags/core.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [develop](<https://devfeed.tech/tags/develop.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [equipment](<https://devfeed.tech/tags/equipment.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [parallel-computing](<https://devfeed.tech/tags/parallel-computing.md>), [process](<https://devfeed.tech/tags/process.md>), [selection](<https://devfeed.tech/tags/selection.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

The monograph tackles NP-hard equipment selection for multiproduct batch plants, minimizing capital cost while meeting operability and production-horizon requirements. It develops sequential branch-and-bound and parallel implementations for OpenMP shared-memory systems, MPI clusters, and CUDA GPUs, with tree-traversal and hardware-specific optimizations. To reduce runtime, a GPU hybrid metaheuristic uses Ant Colony Optimization to find a feasible solution and Simulated Annealing to improve it; several statistical approaches for tuning ant-colony parameters are also studied. The abstract reports that this method found solutions within 0.01% of the branch-and-bound optimum in tens of seconds, compared with tens of hours. These results are reported for experiments across multicore processors, clusters, and multiple GPU generations.

### Source excerpt

arXiv:2611.16290v1 Announce Type: new Abstract: This monograph addresses the optimal selection of equipment for chemical-engineering systems of multiproduct batch plants, a large-scale NP-hard discrete optimization problem in which, for each processing stage, a unit of standard size must be chosen so that the total capital cost of the equipment is minimized while the system remains operable and produces the required products within a given time horizon. We present the mathematical model of system operation and analyze its computational properties. We then develop a sequential branch-and-bound algorithm and its parallel versions for shared-memory systems (OpenMP), distributed-memory clusters (MPI, master-worker), and graphics processing units (CUDA), including recursive and iterative tree traversal, shared-memory optimization, and branch-divergence reduction. To overcome the exponential growth of the run time, we propose a hybrid metaheuristic in which Ant Colony Optimization finds a feasible initial solution and Simulated Annealing improves it, and we implement it on GPUs with thousands of independent threads. Finally, we propose random, constant, and multi-constant approaches to tuning the parameters of the ant colony algorithm based on statistical analysis of many parallel runs, and we study the program on GPUs of three generations and on nodes with several GPUs. Computational experiments on multi-core processors, clusters, and GPUs show that the parallel hybrid algorithm finds solutions within 0.01% of the branch-and-bound optimum in tens of seconds instead of tens of hours.

## metalSPARSE: A 2:4 Semi-Structured Sparse GEMV Kernel for Apple Silicon GPUs

DevFeed: [metalSPARSE: A 2:4 Semi-Structured Sparse GEMV Kernel for Apple Silicon GPUs](<https://devfeed.tech/articles/metalsparse-a-2-4-semi-structured-sparse-gemv-kernel-for-apple-silicon-gpus-132019.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14545>)

Author: Tek Raj Awasthi (Friedrich-Alexander-Universit\\"at Erlangen-N\\"urnberg), Babita Awasthi

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [MLX](<https://devfeed.tech/topics/mlx.md>), [lora](<https://devfeed.tech/topics/lora.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [adapters](<https://devfeed.tech/tags/adapters.md>), [apple-silicon](<https://devfeed.tech/tags/apple-silicon.md>), [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [bits](<https://devfeed.tech/tags/bits.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ce](<https://devfeed.tech/tags/cs-ce.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [devices](<https://devfeed.tech/tags/devices.md>), [edge-devices](<https://devfeed.tech/tags/edge-devices.md>), [engine](<https://devfeed.tech/tags/engine.md>), [faster](<https://devfeed.tech/tags/faster.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [huggingface](<https://devfeed.tech/tags/huggingface.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [laptop](<https://devfeed.tech/tags/laptop.md>), [structured](<https://devfeed.tech/tags/structured.md>)

### AI overview

metalSPARSE reduces Apple GPU decoding traffic by streaming 2:4 weights in compressed form and expanding them in registers, despite the lack of sparse matrix hardware. Its fused Metal decoder reports 3.6x faster decoding than HuggingFace Transformers on PyTorch-MPS on an 8 GB M2 MacBook Air, and 1.52x faster than MLX on a 16 GB M4 for Qwen2.5-1.5B; the same engine on dense weights matches MLX. The reported speedup reaches 97% of what the byte reduction allows, while compute-bound prompt processing does not speed up. For quality recovery, the pipeline combines a Wanda mask, block-wise reconstruction, and 4-bit low-rank adapters trained jointly with surviving weights. On WikiText-2, perplexity falls from 3.01x dense after one-shot pruning with closed-form refit to 1.34x dense. The abstract says the pipeline runs on the same laptop hardware.

### Source excerpt

arXiv:2611.14545v1 Announce Type: cross Abstract: Autoregressive Large Language Models (LLMs) generate text one token at a time, and each token reads every weight from memory, so on edge devices, decoding is limited by memory bandwidth. 2:4 semi-structured sparsity keeps two of every four weights, but Apple-silicon GPUs have no sparse matrix hardware, so pruning helps only if a kernel actually reads fewer bytes. We present metalSPARSE, to our knowledge the first 2:4 sparse kernels for Apple GPUs, together with a fused decoder written in Metal. The kernels stream weights in compressed form and expand them in registers. On Qwen2.5-1.5B with 2:4 sparsity on every decoder linear layer, decoding runs 3.6x faster than HuggingFace Transformers on PyTorch-MPS on an 8 GB M2 MacBook Air and 1.52x faster than MLX on a 16 GB M4. The same engine on dense weights matches MLX, so the gain comes from the compressed weights, and sparsity delivers 97% of the speedup its byte reduction allows. Prompt processing, which is compute-bound, does not speed up. To preserve quality, we combine a Wanda mask with block-wise reconstruction and low-rank adapters trained jointly with the surviving weights and quantized to 4 bits. On WikiText-2 this brings perplexity from 3.01x dense after one-shot pruning (Wanda with a closed-form refit) to 1.34x, and the whole pipeline runs on the same laptop/hardware.

## Starscream: Reimagining LLM Sharding on AMD Instinct GPUs with GPU Partitioning

DevFeed: [Starscream: Reimagining LLM Sharding on AMD Instinct GPUs with GPU Partitioning](<https://devfeed.tech/articles/starscream-reimagining-llm-sharding-on-amd-instinct-gpus-with-gpu-partitioning-131539.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15951>)

Author: Sumanth Gudaparthi, Smeet Chheda, Ganesh Dasika

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [sharding](<https://devfeed.tech/topics/sharding.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [NUMA](<https://devfeed.tech/topics/numa.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [amd](<https://devfeed.tech/tags/amd.md>), [attention](<https://devfeed.tech/tags/attention.md>), [chiplet](<https://devfeed.tech/tags/chiplet.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-partitioning](<https://devfeed.tech/tags/gpu-partitioning.md>), [instinct](<https://devfeed.tech/tags/instinct.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [llm](<https://devfeed.tech/tags/llm.md>), [locality](<https://devfeed.tech/tags/locality.md>), [memory](<https://devfeed.tech/tags/memory.md>), [numa](<https://devfeed.tech/tags/numa.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [performance](<https://devfeed.tech/tags/performance.md>), [redundancy](<https://devfeed.tech/tags/redundancy.md>), [sharding](<https://devfeed.tech/tags/sharding.md>), [strategies](<https://devfeed.tech/tags/strategies.md>), [tensor-parallelism](<https://devfeed.tech/tags/tensor-parallelism.md>)

### AI overview

Starscream adapts LLM sharding to partitioned, chiplet-based GPUs by adding intra-GPU sharding along an orthogonal dimension instead of extending inter-GPU sharding across the partitions. It targets three problems in conventional approaches: redundant memory use, reduced arithmetic intensity, and communication overhead from flat collectives. Its design principles are to preserve kernel shapes, eliminate memory redundancy, and minimize communication. In a tensor-parallelism case study on AMD MI300X and MI355X, partition-aware designs reportedly achieved up to 4x speedup for attention and 1.43x for fused GEMMs versus chiplet-oblivious baselines. The supplied abstract claims the principles can extend to other sharding strategies and future architectures; it does not provide broader evaluation details.

### Source excerpt

arXiv:2611.15951v1 Announce Type: new Abstract: As GPU accelerators grow larger to meet rising compute demands, they exhibit increasingly pronounced non-uniform memory access (NUMA) characteristics that challenge existing LLM frameworks. Chiplet-based architectures provide a practical path to larger GPUs while mitigating manufacturing yield constraints. Modern GPUs such as AMD Instinct MI300X employs such a multi-die design, and further offer GPU partitioning capabilities, partitioning a single physical GPU into multiple logical devices with NUMA-aware memory domains, promising faster kernel launches, and reduced memory access latencies with improved locality. However, current parallelization strategies, designed for homogeneous GPU topologies, when applied under partitioning, extend its unidimensional sharding causing significant performance degradation. Three bottlenecks emerge: (1) memory redundancy; (2) arithmetic intensity collapse; (3) communication overhead from flat collectives. We present Starscream, a rethinking of LLM sharding for the chiplet era. Rather than stretching inter-GPU sharding thinner, Starscream adds intra-GPU sharding along an orthogonal dimension, grounded in three foundational principles: kernel shape preservation, memory redundancy elimination, and communication overhead minimization. These principles form the basis for extending any sharding strategy into the partitioning regime. We demonstrate their effectiveness using tensor parallelism as a case study. On AMD MI300X and MI355X, exploiting partitioning capabilities yields up to 4x speedup for attention and 1.43x for fused GEMMs compared to chiplet-oblivious baselines, demonstrating the effectiveness of chiplet-aware designs. Starscream's principles generalize across sharding strategies and future architectures.

## KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving

DevFeed: [KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving](<https://devfeed.tech/articles/kairos-stateful-context-aware-power-efficient-agentic-inference-serving-127787.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2604.16682>)

Author: Yichao Yuan, Mosharaf Chowdhury, Nishil Talati

Published: 2026-11-11T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-context](<https://devfeed.tech/tags/agent-context.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agentic-tasks](<https://devfeed.tech/tags/agentic-tasks.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [headroom](<https://devfeed.tech/tags/headroom.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [memory-pressure](<https://devfeed.tech/tags/memory-pressure.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [power](<https://devfeed.tech/tags/power.md>), [serving](<https://devfeed.tech/tags/serving.md>), [stateful](<https://devfeed.tech/tags/stateful.md>)

### AI overview

KAIROS uses evolving agent context to coordinate GPU frequency, per-instance concurrency, and request placement across serving instances. The approach responds to context growth and agent progress, aiming to save power when memory headroom permits while avoiding memory-related thrashing and meeting performance targets. Across software and data engineering agent tasks, the paper reports an average 27% power reduction, reaching up to 39.8%.

### Source excerpt

arXiv:2604.16682v2 Announce Type: replace Abstract: Power has become a central bottleneck for AI inference. This problem is becoming more urgent as agentic AI emerges as a major workload class, yet prior power-management techniques focus almost entirely on single-turn LLM serving. Our analysis shows that agentic serving behaves fundamentally differently: each request carries long-lived context that evolves across tool-interleaved turns, and lowering GPU frequency can push the system into a thrashing regime where memory pressure sharply worsens both performance and power efficiency. These observations show that power optimization for agentic serving requires rethinking. We present KAIROS, a context-aware power optimization system for agentic AI serving. KAIROS uses agent context as a first-class control signal to jointly manage GPU frequency, per-instance concurrency, and multi-instance request placement. This enables KAIROS to save power when memory headroom exists while avoiding thrashing and preserving performance targets. At a high level, KAIROS tracks requests at agent granularity, adapts local control to context growth and agent progress, and routes agents across instances to jointly improve power efficiency and memory stability. Evaluated across diverse software and data engineering agentic tasks, KAIROS achieves an average of 27% (up to 39.8%) power reduction while meeting the performance targets.

## MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs

DevFeed: [MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs](<https://devfeed.tech/articles/mpfa-hardware-efficient-train-free-qk4v8-flashattention-kernels-on-blackwell-gpus-127793.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.33135>)

Author: Chencheng Deng, Jianbin Fang, Dezun Dong

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [NVIDIA Blackwell](<https://devfeed.tech/topics/nvidia-blackwell.md>), [Tensor Cores](<https://devfeed.tech/topics/tensor-cores.md>)

Tags: [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [attention](<https://devfeed.tech/tags/attention.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [decode](<https://devfeed.tech/tags/decode.md>), [flashattention](<https://devfeed.tech/tags/flashattention.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llm](<https://devfeed.tech/tags/llm.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [performance](<https://devfeed.tech/tags/performance.md>), [precision](<https://devfeed.tech/tags/precision.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [reuse](<https://devfeed.tech/tags/reuse.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

MpFA is a training-free FlashAttention kernel for NVIDIA Blackwell GPUs that combines NVFP4 for QK with FP8 for PV. The mixed-precision design retains low-bit QK throughput while avoiding the conversion and scaling costs of FP4 PV, which can prevent fully FP4 attention from delivering end-to-end speedups. Rank-one smoothing compensation, implemented as an extra Tensor Core MMA, recovers 62.5% of accuracy loss with about 2.0% kernel overhead. The kernel also uses asynchronous pipelining, tensor-memory reuse and adaptive parallel partitioning for prefill and decode. On an SM100 across 16K-128K contexts, it improves prefill throughput over BF16/FP8 baselines and reports 2.81x higher end-to-end output throughput than BF16 FA4 across Llama-3.1-8B and Qwen3-14B.

### Source excerpt

arXiv:2609.33135v2 Announce Type: replace Abstract: Long-context LLM inference pushes modern GPU serving stacks into an attention-bound regime, where both compute and memory are dominated by the softmax-GEMM pipeline. On NVIDIA Blackwell GPUs, FP4 Tensor Cores offer high matmul throughput, but we find that fully FP4 attention often fails to translate this throughput into end-to-end speedups due to non-matmul costs: online quantization after softmax, tensor/shared-memory data movement, and contention on the softmax path. We present MpFA, a training-free FlashAttention kernel optimized for Blackwell. Guided by hardware characterization, MpFA uses mixed precision: NVFP4 for QK and FP8 for PV (QK4PV8). This preserves low-bit QK throughput while avoiding the conversion and scaling overheads of FP4 PV. To recover accuracy without further stressing the softmax pipeline, MpFA introduces rank-one smoothing compensation implemented as an additional Tensor Core MMA. MpFA further improves performance with a fine-grained asynchronous pipeline, tensor-memory reuse, and adaptive parallel partitioning across prefill and decode. On an NVIDIA SM100 GPU and across 16K-128K contexts, MpFA improves prefill throughput over state-of-the-art BF16/FP8 baselines and increases end-to-end output throughput by 2.81$\times$ over BF16 FA4 across Llama-3.1-8B and Qwen3-14B. Across five benchmark suites and two models, rank-one compensation recovers 62.5% of the accuracy loss with about 2.0% kernel overhead.

[Next page](<https://devfeed.tech/topics/gpu-optimization.md?cursor=WyIyMDI2LTExLTExVDA1OjAwOjAwKzAwOjAwIiwgIjUwMTVjMmMwLTVjYTgtNGUzOC1iYjYxLThkOTdkMWZiYmRlZCJd>)