# Generation configurations: temperature, top-k, top-p, and test time compute

DevFeed: [Generation configurations: temperature, top-k, top-p, and test time compute](<https://devfeed.tech/articles/generation-configurations-temperature-top-k-top-p-and-test-time-compute-31796.md>)

Original publisher: [Read original article](<https://huyenchip.com//2024/01/16/sampling.html>)

Author: Chip Huyen

Published: 2024-01-16T00:00:00Z

Content type: tutorial

Language: en

Sources: [Chip Huyen](<https://devfeed.tech/sources/chip-huyen.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Inference](<https://devfeed.tech/topics/inference.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [hallucinations](<https://devfeed.tech/tags/hallucinations.md>), [inference](<https://devfeed.tech/tags/inference.md>), [ml](<https://devfeed.tech/tags/ml.md>), [neural](<https://devfeed.tech/tags/neural.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [responses](<https://devfeed.tech/tags/responses.md>), [token](<https://devfeed.tech/tags/token.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

## AI overview

This tutorial explains why machine-learning models produce probabilistic responses and how sampling, or decoding, generates them. It covers sampling strategies such as temperature, top-k, and top-p, test-time compute through multiple outputs, and structured outputs.

## Source excerpt

ML models are probabilistic. Imagine that you want to know what's the best cuisine in the world. If you ask someone this question twice, a minute apart, their answers both times should be the same. If you ask a model the same question twice, its answer can change. If the model thinks that Vietnamese cuisine has a 70% chance of being the best cuisine and Italian cuisine has a 30% chance, it'll answer "Vietnamese" 70% of the time, and "Italian" 30%. This probabilistic nature makes AI great for creative tasks. What is creativity but the ability to explore beyond the common possibilities, to think outside the box? However, this probabilistic nature also causes inconsistency and hallucinations. It's fatal for tasks that depend on factuality. Recently, I went over 3 months' worth of customer support requests of an AI startup I advise and found that ⅕ of the questions are because users don't understand or don't know how to work with this probabilistic nature. To understand why AI's responses are probabilistic, we need to understand how models generate responses, a process known as sampling (or decoding). This post consists of 3 parts. Sampling: sampling strategies and sampling variables including temperature, top-k, and top-p. Test time compute: increasing the compute allocated to inference, e.g. sampling multiple outputs, to help improve a model's performance. Structured outputs: how to get models to generate outputs in a certain format. Sampling Given an input, a neural network produces an output by first computing the probabilities of all possible values. For a classifier, possible values are the available classes. For example, if a model is trained to classify whether an email is spam, there are only two possible values: spam and not spam. The model computes the probability of each of these two values, say being spam is 90% and not spam is 10%. To generate the next token, a language model first computes the probability distribution over all tokens in the vocabulary. Fo