# infini-attention

Published articles for infini-attention.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## A failed experiment: Infini-Attention, and why we should keep trying?

DevFeed: [A failed experiment: Infini-Attention, and why we should keep trying?](<https://devfeed.tech/articles/a-failed-experiment-infini-attention-and-why-we-should-keep-trying-7286.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/infini-attention>)

Author: Phuc Nguyen; Leandro von Werra; Thomas Wolf

Published: 2024-08-14T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [long-context](<https://devfeed.tech/topics/long-context.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [AI Research](<https://devfeed.tech/topics/ai-research.md>)

Tags: [complexity](<https://devfeed.tech/tags/complexity.md>), [compute](<https://devfeed.tech/tags/compute.md>), [context](<https://devfeed.tech/tags/context.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [infini-attention](<https://devfeed.tech/tags/infini-attention.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llama](<https://devfeed.tech/tags/llama.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [memory-compression](<https://devfeed.tech/tags/memory-compression.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>)

### AI overview

This article examines Infini-attention as an alternative approach for extending language-model context lengths. It explains the quadratic memory cost of standard self-attention, the limitations of Flash Attention and Ring Attention, and the resource demands of training models with million-token contexts.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.