# First Token Cutoff LLM sampling

DevFeed: [First Token Cutoff LLM sampling](<https://devfeed.tech/articles/first-token-cutoff-llm-sampling-20633.md>)

Original publisher: [Read original article](<http://antirez.com/news/142>)

Published: 2024-01-12T16:49:37Z

Content type: opinion

Language: en

Sources: [Antirez](<https://devfeed.tech/sources/antirez.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mistral](<https://devfeed.tech/tags/mistral.md>), [token](<https://devfeed.tech/tags/token.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

## AI overview

The article examines top-p nucleus sampling for large language models and argues that it may have shortcomings. It presents a simpler alternative, called First Token Cutoff, as a work in progress, and discusses how logits and token-probability distributions can indicate uncertainty and possible hallucinations.

## Source excerpt

From a theoretical standpoint, the best reply provided by an LLM is obtained by always picking the token associated with the highest probability. This approach makes the LLM output deterministic, which is not a good property for a number of applications. For this reason, in order to balance LLMs creativity while preserving adherence to the context, different sampling algorithms have been proposed in recent years. Today one of the most used ones, more or less the default, is called top-p: it is a form of nucleus sampling where top-scoring tokens are collected up to a total probability sum of "p", then random weighted sampling is performed. In this blog post I'll examine why I believe nucleus sampling may not be the best approach, and will show a simple and understandable alternative in order to avoid the issues of nucleus sampling. The algorithm is yet a work in progress, but by publishing it now I hope to stimulate some discussion / hacking. ## There is some gold in the logits Despite the fact that LLM logits are one of the few completely understandable parts of the LLM inner working, I generally see very little interest in studying their features, investigating more advanced sampling methods, detecting and signaling users uncertainty and likely hallucination. Visualizing the probabilities distribution for successive tokens is a simple and practical exercise in order to gain some insights: !~! In the image we can see the top 32 candidate tokens colored by probability (white = 0, blue = 1), the selected token and the rank of the selected token (highest probability = 0, the previous one = 1, and so forth). In the above example, the Mistral base model knows the birth and death dates of Umberto Eco, so it confidently signals the most likely token with most of the total probability. Other times the model is more perplexed because either there are multiple ways to express the continuation of the text, or because it is not certain about certain facts. Asking the date of a