# Welcome Falcon Mamba: The first strong attention-free 7B model

DevFeed: [Welcome Falcon Mamba: The first strong attention-free 7B model](<https://devfeed.tech/articles/welcome-falcon-mamba-the-first-strong-attention-free-7b-model-7191.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/falconmamba>)

Author: Jingwei Zuo; Maksim Velikanov; Rhaiem; Ilyas Chahed; Younes B; Guillaume Kunsch

Published: 2024-08-12T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [code](<https://devfeed.tech/tags/code.md>), [community](<https://devfeed.tech/tags/community.md>), [compute](<https://devfeed.tech/tags/compute.md>), [data](<https://devfeed.tech/tags/data.md>), [design](<https://devfeed.tech/tags/design.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [generate](<https://devfeed.tech/tags/generate.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [leaderboard](<https://devfeed.tech/tags/leaderboard.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mamba](<https://devfeed.tech/tags/mamba.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [research](<https://devfeed.tech/tags/research.md>)

## AI overview

Falcon Mamba is an attention-free 7B language model based on Mamba state-space architecture. The article describes its training choices, long-sequence efficiency, benchmark evaluations, and comparisons with transformer models.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.