# top\_p

Published articles for top\_p.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## TempQuant: Learning Adaptive Temperatures for Quantized Language Models

DevFeed: [TempQuant: Learning Adaptive Temperatures for Quantized Language Models](<https://devfeed.tech/articles/tempquant-learning-adaptive-temperatures-for-quantized-language-models-141736.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22128>)

Author: Mukund Athmaram, Liang Ma, Manoj Rao

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [behavior](<https://devfeed.tech/tags/behavior.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [global](<https://devfeed.tech/tags/global.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [learning](<https://devfeed.tech/tags/learning.md>), [math](<https://devfeed.tech/tags/math.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [temperature](<https://devfeed.tech/tags/temperature.md>), [token](<https://devfeed.tech/tags/token.md>), [top-p](<https://devfeed.tech/tags/top-p.md>)

### AI overview

TempQuant learns a quantized language model's temperature and top-p settings to make its next-token probabilities closer to those of the full-precision model. The authors report that quantization warms models on average and makes performance more sensitive to temperature. TempQuant offers a global setting per requested temperature, learned without answer labels, and token-by-token settings: tokens whose rankings change under quantization tend to need warming, while those whose rankings remain stable tend to need cooling. In the reported 3-bit Qwen3-4B example, global TempQuant raises MATH accuracy from 34% with the configuration recommended for Qwen's quantized releases to 51%. The abstract says the method recovers much of the accuracy when quantization causes substantial degradation and otherwise moves behavior closer to the full-precision model.

### Source excerpt

arXiv:2611.22128v1 Announce Type: new Abstract: Large language models are increasingly deployed at low precision, yet quantization can degrade their performance. We study how quantization interacts with decoding settings such as the sampling temperature and top-p by examining how it distorts the model's next-token probabilities. Measuring entropy during generation across deployed temperatures, we find that quantization warms the model on average, and that temperature matters much more for the performance of a quantized model than for its full-precision counterpart. Because quantization and temperature both shape the next-token distribution, we investigate how much the sampling settings can undo what quantization changes, through our method TempQuant, which learns the quantized model's temperature and top-p to bring its next-token probabilities closer to the full-precision model's. We learn these settings at two granularities. The first is a single global setting per requested temperature, which finds nearly the same temperatures that tuning on labeled answers would, without any labels. Model cards typically recommend sampling a quantized model with the same settings as its full-precision counterpart, while we find that different settings can substantially improve its accuracy. On 3-bit Qwen3-4B, for example, global TempQuant raises MATH accuracy from 34% under the configuration recommended for Qwen's quantized releases to 51%. The second sets the temperature and top-p for every token, motivated by our finding that tokens whose rankings flip under quantization tend to need warming, while tokens whose rankings are preserved tend to need cooling. When quantization degrades accuracy substantially, TempQuant recovers much of it, and when degradation is lighter, it still brings the quantized model's behavior closer to the full-precision model's.

## Qrita: High-performance Top-k and Top-p Algorithm for GPUs using Pivot-based Truncation and Selection

DevFeed: [Qrita: High-performance Top-k and Top-p Algorithm for GPUs using Pivot-based Truncation and Selection](<https://devfeed.tech/articles/qrita-high-performance-top-k-and-top-p-algorithm-for-gpus-using-pivot-based-truncation-and-selection-117202.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.01518>)

Author: Jongseok Park, Sunga Kim, Alvin Cheung, Ion Stoica

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [1-4](<https://devfeed.tech/tags/1-4.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [engines](<https://devfeed.tech/tags/engines.md>), [execution](<https://devfeed.tech/tags/execution.md>), [flashinfer](<https://devfeed.tech/tags/flashinfer.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [half](<https://devfeed.tech/tags/half.md>), [handling](<https://devfeed.tech/tags/handling.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [implement](<https://devfeed.tech/tags/implement.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [key](<https://devfeed.tech/tags/key.md>), [llm](<https://devfeed.tech/tags/llm.md>), [performance](<https://devfeed.tech/tags/performance.md>), [selection](<https://devfeed.tech/tags/selection.md>), [top-k](<https://devfeed.tech/tags/top-k.md>), [top-p](<https://devfeed.tech/tags/top-p.md>)

### AI overview

Qrita speeds up GPU Top-k and Top-p sampling through pivot-based truncation and selection. Gaussian-based σ-truncation narrows the vocabulary search space, while quaternary pivot search with duplication handling halves pivot-search iterations and guarantees deterministic output. The Triton implementation reportedly improves end-to-end serving throughput by up to 1.4x while using half the memory compared with evaluated SGLang and FlashInfer kernels, and matches the output of stable sort-based algorithms. Qrita is now vLLM's default Triton Top-k and Top-p sampler. These are abstract-reported results; the supplied text does not include evaluation details.

### Source excerpt

arXiv:2602.01518v3 Announce Type: replace Abstract: Despite their importance in model sampling, efficient implementation of Top-k and Top-p algorithms poses a significant challenge as model vocabularies grow with every new model release. Existing approaches often rely on sorting, which incurs significant computation and memory overhead on GPUs, or on stochastic approaches that alter the algorithm's output. In this work, we propose Qrita, an efficient Top-k and Top-p algorithm based on a pivot-based truncation and selection. Qrita leverages pivot-based search for both Top-k and Top-p with two key techniques: 1. Gaussian-based $\sigma$-truncation, which greatly reduces the search space of the vocabulary, and 2. Quaternary pivot search with duplication handling, which halves the number of pivot search iterations and guarantees deterministic output. We implement Qrita using Triton and evaluate its performance against the Top-k and Top-p kernels of high-performance LLM execution engines such as SGLang and FlashInfer, improving end-to-end serving throughput up to 1.4$\times$ with half the memory usage, while providing the same output of stable sort-based Top-k and Top-p algorithms. Qrita is now the default Triton Top-k and Top-p sampler of vLLM, and its implementation is available at https://github.com/vllm-project/vllm/blob/main/vllm/v1/sample/ops/topk_topp_triton.py.

## Temperature, Not Knowledge: What Recursive Self-Training and Its Remedies Do to a Language Model's Confidence

DevFeed: [Temperature, Not Knowledge: What Recursive Self-Training and Its Remedies Do to a Language Model's Confidence](<https://devfeed.tech/articles/temperature-not-knowledge-what-recursive-self-training-and-its-remedies-do-to-a-language-model-s-confidence-115122.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.01059>)

Author: Dipto Sumit, Syeda Fabiayee Jahan, Farig Sadeque

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [discrimination](<https://devfeed.tech/tags/discrimination.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [reference](<https://devfeed.tech/tags/reference.md>), [study](<https://devfeed.tech/tags/study.md>), [target](<https://devfeed.tech/tags/target.md>), [temperature](<https://devfeed.tech/tags/temperature.md>), [top](<https://devfeed.tech/tags/top.md>), [top-p](<https://devfeed.tech/tags/top-p.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

In a five-round self-training study on Qwen3-1.7B, accuracy and overall discrimination stayed nearly unchanged while confidence sharpened and the most confident answers became less reliable. Most confidence drift came from fitting each generated answer twice, with top-p truncation adding to it; removing both kept entropy on generated outputs flat. Matching predictive entropy to the calibrated starting model each round recalibrated nearly as well as label-based temperature fitting, including at 8B. Remedies differed: truncated cross-entropy, an entropy bonus, and single-pass training preserved confidence temperature without improving discrimination; focal loss and multi-sample targets improved discrimination but led to underconfidence, while reference-data mixing overshot. Combining a ranking-improving loss with label-free recalibration achieved both goals.

### Source excerpt

arXiv:2611.01059v1 Announce Type: new Abstract: Recursive self-training can collapse language models, and many proposed remedies target overconfidence, yet their effects on confidence are rarely measured. We study a five-round self-training loop on Qwen3-1.7B, using three seeds per compared method, and distinguish confidence temperature (drift from a calibrated starting point) from discrimination (ranking correct answers above incorrect ones). Accuracy and overall discrimination remain nearly unchanged, but confidence sharpens and the most confident answers become less reliable. Matching predictive entropy to the calibrated starting model each round recalibrates nearly as well as label-based temperature fitting, at both 1.7B and 8B. Sharpening is not inherent to self-training: exact samples from a model's distribution have zero expected gradient. Instead, most drift comes from fitting each generated answer twice, with top-p truncation contributing the rest; removing both keeps entropy on generated outputs flat. Remedies affect the two properties differently: truncated cross-entropy, an entropy bonus, and single-pass training preserve temperature without improving discrimination, while focal loss and multi-sample targets improve discrimination but cause underconfidence. Reference-data mixing also overshoots. Pairing a ranking-improving loss with label-free recalibration achieves both.

## First Token Cutoff LLM sampling

DevFeed: [First Token Cutoff LLM sampling](<https://devfeed.tech/articles/first-token-cutoff-llm-sampling-20633.md>)

Original publisher: [Read original article](<http://antirez.com/news/142>)

Author: Antirez

Published: 2024-01-12T16:49:37Z

Content type: tutorial

Language: en

Sources: [Antirez](<https://devfeed.tech/sources/antirez.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [api](<https://devfeed.tech/tags/api.md>), [context](<https://devfeed.tech/tags/context.md>), [creativity](<https://devfeed.tech/tags/creativity.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mistral](<https://devfeed.tech/tags/mistral.md>), [model](<https://devfeed.tech/tags/model.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [token](<https://devfeed.tech/tags/token.md>), [top-p](<https://devfeed.tech/tags/top-p.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [weighted-sampling](<https://devfeed.tech/tags/weighted-sampling.md>)

### AI overview

First Token Cutoff (FTC) aims to preserve variation while excluding tokens whose probability is much lower than the top candidate. After applying softmax and sorting token probabilities, it computes each candidate's relative drop from the best token, T0, and keeps candidates where r = 1 − (T[i]/T0) is at most the cutoff co; it then samples among the retained tokens by weight. The author argues top-p can include a very weak token when cumulative probability reaches p, while FTC bounds that gap. For probabilities 0.25, 0.14, and 0.01, for example, top-p=0.4 may include the weakest token, whereas co=0.5 keeps the 0.14 candidate and excludes 0.01. The author presents co as easier to interpret than top-p, but says FTC is a work in progress with no scientific scrutiny yet; its claimed balance and example outputs are preliminary.

### Source excerpt

From a theoretical standpoint, the best reply provided by an LLM is obtained by always picking the token associated with the highest probability. This approach makes the LLM output deterministic, which is not a good property for a number of applications. For this reason, in order to balance LLMs creativity while preserving adherence to the context, different sampling algorithms have been proposed in recent years. Today one of the most used ones, more or less the default, is called top-p: it is a form of nucleus sampling where top-scoring tokens are collected up to a total probability sum of "p", then random weighted sampling is performed. In this blog post I'll examine why I believe nucleus sampling may not be the best approach, and will show a simple and understandable alternative in order to avoid the issues of nucleus sampling. The algorithm is yet a work in progress, but by publishing it now I hope to stimulate some discussion / hacking. ## There is some gold in the logits Despite the fact that LLM logits are one of the few completely understandable parts of the LLM inner working, I generally see very little interest in studying their features, investigating more advanced sampling methods, detecting and signaling users uncertainty and likely hallucination. Visualizing the probabilities distribution for successive tokens is a simple and practical exercise in order to gain some insights: !~! In the image we can see the top 32 candidate tokens colored by probability (white = 0, blue = 1), the selected token and the rank of the selected token (highest probability = 0, the previous one = 1, and so forth). In the above example, the Mistral base model knows the birth and death dates of Umberto Eco, so it confidently signals the most likely token with most of the total probability. Other times the model is more perplexed because either there are multiple ways to express the continuation of the text, or because it is not certain about certain facts. Asking the date of a