# First Token Cutoff LLM sampling

DevFeed: [First Token Cutoff LLM sampling](<https://devfeed.tech/articles/first-token-cutoff-llm-sampling-20633.md>)

Original publisher: [Read original article](<http://antirez.com/news/142>)

Author: Antirez

Published: 2024-01-12T16:49:37Z

Content type: tutorial

Language: en

Sources: [Antirez](<https://devfeed.tech/sources/antirez.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [api](<https://devfeed.tech/tags/api.md>), [context](<https://devfeed.tech/tags/context.md>), [creativity](<https://devfeed.tech/tags/creativity.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mistral](<https://devfeed.tech/tags/mistral.md>), [model](<https://devfeed.tech/tags/model.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [token](<https://devfeed.tech/tags/token.md>), [top-p](<https://devfeed.tech/tags/top-p.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [weighted-sampling](<https://devfeed.tech/tags/weighted-sampling.md>)

## AI overview

First Token Cutoff (FTC) aims to preserve variation while excluding tokens whose probability is much lower than the top candidate. After applying softmax and sorting token probabilities, it computes each candidate's relative drop from the best token, T0, and keeps candidates where r = 1 − (T[i]/T0) is at most the cutoff co; it then samples among the retained tokens by weight. The author argues top-p can include a very weak token when cumulative probability reaches p, while FTC bounds that gap. For probabilities 0.25, 0.14, and 0.01, for example, top-p=0.4 may include the weakest token, whereas co=0.5 keeps the 0.14 candidate and excludes 0.01. The author presents co as easier to interpret than top-p, but says FTC is a work in progress with no scientific scrutiny yet; its claimed balance and example outputs are preliminary.

## Source excerpt

From a theoretical standpoint, the best reply provided by an LLM is obtained by always picking the token associated with the highest probability. This approach makes the LLM output deterministic, which is not a good property for a number of applications. For this reason, in order to balance LLMs creativity while preserving adherence to the context, different sampling algorithms have been proposed in recent years. Today one of the most used ones, more or less the default, is called top-p: it is a form of nucleus sampling where top-scoring tokens are collected up to a total probability sum of "p", then random weighted sampling is performed. In this blog post I'll examine why I believe nucleus sampling may not be the best approach, and will show a simple and understandable alternative in order to avoid the issues of nucleus sampling. The algorithm is yet a work in progress, but by publishing it now I hope to stimulate some discussion / hacking. ## There is some gold in the logits Despite the fact that LLM logits are one of the few completely understandable parts of the LLM inner working, I generally see very little interest in studying their features, investigating more advanced sampling methods, detecting and signaling users uncertainty and likely hallucination. Visualizing the probabilities distribution for successive tokens is a simple and practical exercise in order to gain some insights: !~! In the image we can see the top 32 candidate tokens colored by probability (white = 0, blue = 1), the selected token and the rank of the selected token (highest probability = 0, the previous one = 1, and so forth). In the above example, the Mistral base model knows the birth and death dates of Umberto Eco, so it confidently signals the most likely token with most of the total probability. Other times the model is more perplexed because either there are multiple ways to express the continuation of the text, or because it is not certain about certain facts. Asking the date of a