# The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility

DevFeed: [The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility](<https://devfeed.tech/articles/the-silent-hyperparameter-quantifying-the-impact-of-inference-backends-on-llm-reproducibility-123156.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.19537>)

Author: David Pape, Jonathan Evertz, Lea Sch\\"onherr

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [llama.cpp](<https://devfeed.tech/topics/llama-cpp.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [CUDA Graphs](<https://devfeed.tech/topics/cuda-graphs.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [alter](<https://devfeed.tech/tags/alter.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [backend](<https://devfeed.tech/tags/backend.md>), [backends](<https://devfeed.tech/tags/backends.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [caching](<https://devfeed.tech/tags/caching.md>), [can](<https://devfeed.tech/tags/can.md>), [choice](<https://devfeed.tech/tags/choice.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cost](<https://devfeed.tech/tags/cost.md>), [critical](<https://devfeed.tech/tags/critical.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [impact](<https://devfeed.tech/tags/impact.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [reproducibility](<https://devfeed.tech/tags/reproducibility.md>)

## AI overview

Inference backend choice can materially affect LLM benchmark results even when model weights, decoding parameters, and hardware are held constant. Across five engines and open-weight models from 4B to 30B parameters, the study reports score shifts of up to 16.6 percentage points and frequent output disagreement. It attributes divergence to backend-specific optimizations and defaults, including prefix caching, CUDA graphs, custom kernels, and logit processing. A survey found 200 distinct engines, while analysis of 35,000 ML publications found that inference stacks are rarely reported. The authors argue that standardized reporting of inference stacks would improve benchmark reproducibility and interpretability.

## Source excerpt

arXiv:2605.19537v3 Announce Type: replace Abstract: Progress in LLMs is increasingly measured through standardized benchmarks, where state-of-the-art improvements are often separated by fractions of a percentage point. At the same time, the computational cost of evaluating modern LLMs has driven widespread adoption of specialized inference backends, software systems that execute trained models efficiently at inference time. While critical for scalability, system-level optimizations, such as custom CUDA kernels and reduced-precision arithmetic, can alter token probabilities and introduce non-determinism, possibly cascading into divergent generation. In this work, we first survey the inference landscape, identifying $200$ distinct engines, and analyze $35,000$ ML publications, finding that the specific inference stack is rarely reported despite this widespread diversity. We then present a systematic empirical study of how inference backends affect LLM benchmark results. Holding model weights, decoding parameters, and hardware constant, we evaluate five widely used inference engines, including vLLM, SGLang, and llama.cpp, across multiple open-weight models spanning 4B-30B parameters and established benchmarks. We show that the choice of backend alone can shift benchmark scores by up to $16.6$ percentage points and induce high rates of output disagreement. By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations such as prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing. Our findings identify the inference backend as a previously unreported yet consequential hyperparameter in LLM evaluations and advocate standardized reporting of inference stacks to improve the reproducibility and interpretability of benchmark comparisons.