# Reproducible LoRA Inference Through Deterministic Split-K Reduction

DevFeed: [Reproducible LoRA Inference Through Deterministic Split-K Reduction](<https://devfeed.tech/articles/reproducible-lora-inference-through-deterministic-split-k-reduction-125172.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10334>)

Author: Dhayaneshwar Balusamy, Ankur Mohan

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [lora](<https://devfeed.tech/topics/lora.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [atomic](<https://devfeed.tech/tags/atomic.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [cache](<https://devfeed.tech/tags/cache.md>), [code](<https://devfeed.tech/tags/code.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [decision](<https://devfeed.tech/tags/decision.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [floating-point](<https://devfeed.tech/tags/floating-point.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [inference](<https://devfeed.tech/tags/inference.md>), [input](<https://devfeed.tech/tags/input.md>), [jev](<https://devfeed.tech/tags/jev.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [lora](<https://devfeed.tech/tags/lora.md>), [mathematical-reasoning](<https://devfeed.tech/tags/mathematical-reasoning.md>), [reduction](<https://devfeed.tech/tags/reduction.md>), [split](<https://devfeed.tech/tags/split.md>)

## AI overview

Floating-point atomic additions in vLLM's Atomic Split-K=8 LoRA kernel make inference numerically nondeterministic: identical-input replays produced different FP32 shrink outputs in 97.61% of comparisons, and generated outputs changed in 17.30% of GSM8K and 14.96% of MBPP comparisons, sometimes changing correctness. The authors trace shrink-output differences into the KV cache and show that accumulated differences can alter token decisions and final answers; structured toxicity-decision probabilities also varied in 81.47% of atomic comparisons, with decision flips. Their deterministic Triton Split-K=8 implementation showed no observed variation while retaining parallelism. It was 16.8-21.1% faster than unsplit reduction, with 1.02-2.62% more latency than Atomic Split-K=8.

## Source excerpt

arXiv:2611.10334v1 Announce Type: new Abstract: Reproducibility is important for LoRA-adapted language models used in repeated evaluation, auditing, and decision-making. However, Split-K kernels used to introduce parallelism in low-rank LoRA multiplications can have the unintended consequence of numerical nondeterminism when partial results are combined using floating-point atomic additions. We investigate this non-determinism in vLLM across GSM8K mathematical reasoning, MBPP code generation, and Jev-style structured toxicity decisions on Civil Comments and introduce a deterministic Triton Split-K implementation. Under identical-input replay, Atomic Split-K=8 produces different FP32 shrink outputs in 97.61% of comparisons, while our Deterministic Split-K=8 shows no observed variation. Repeated atomic inference changes generated outputs in 17.30% of GSM8K and 14.96% of MBPP comparisons, sometimes altering task correctness. Structured-decision probabilities vary in 81.47% of atomic comparisons, with decision flips. Controlled tracing demonstrates how shrink-output differences propagate into the KV cache and how accumulated cache differences can change token decisions and final answers. Our deterministic Triton Split-K=8 implementation preserves parallelism, achieving 16.8-21.1% latency-based speedup over unsplit reduction with only 1.02-2.62% additional latency relative to Atomic Split-K=8.