# Finally, a Replacement for BERT: Introducing ModernBERT

DevFeed: [Finally, a Replacement for BERT: Introducing ModernBERT](<https://devfeed.tech/articles/finally-a-replacement-for-bert-introducing-modernbert-7356.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/modernbert>)

Author: Benjamin Warner; Antoine Chaffin; Benjamin Clavié; Orion Weller; Oskar Hallström; Said Taghadouini; Alexis Gallagher; Raja Biswas; Faisal Ladhak; Tom Aarsen

Published: 2024-12-19T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [Retrieval-Augmented Generation](<https://devfeed.tech/topics/retrieval-augmented-generation.md>), [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [community](<https://devfeed.tech/tags/community.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [flash-attention-2](<https://devfeed.tech/tags/flash-attention-2.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [open-source-collab](<https://devfeed.tech/tags/open-source-collab.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rag](<https://devfeed.tech/tags/rag.md>), [research](<https://devfeed.tech/tags/research.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

## AI overview

The article introduces ModernBERT, a family of encoder-only models designed as improved replacements for BERT-like models. It describes 8,192-token context, stronger downstream performance, faster processing, base and large model sizes, compatibility with Transformers, and use cases including retrieval, classification, question answering, entity extraction, RAG pipelines, and recommendation systems. It also recommends Flash Attention 2 when supported by the GPU.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.