# Attention Is Off By One

DevFeed: [Attention Is Off By One](<https://devfeed.tech/articles/attention-is-off-by-one-37883.md>)

Original publisher: [Read original article](<https://www.evanmiller.org/attention-is-off-by-one.html>)

Author: Evan Miller

Published: 2023-07-24T16:05:00Z

Content type: opinion

Language: en

Sources: [Evan Miller](<https://devfeed.tech/sources/evan-miller.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>)

Tags: [central](<https://devfeed.tech/tags/central.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [edge](<https://devfeed.tech/tags/edge.md>), [llms](<https://devfeed.tech/tags/llms.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [quantization](<https://devfeed.tech/tags/quantization.md>)

## AI overview

Evan Miller argues that the standard Transformer attention formula contains an overlooked off-by-one error. He discusses how outlier weights and activations complicate quantization and deployment, and proposes investigating fixes including Softmax One and QuietAttention.

## Source excerpt

Transformer has a mathematical bug that has been overlooked for 6+ years. I propose fixing its outliers with two new devices, Softmax One and QuietAttention: Attention Is Off By One