# Flash Attention 2 Published articles for Flash Attention 2. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Finally, a Replacement for BERT: Introducing ModernBERT DevFeed: [Finally, a Replacement for BERT: Introducing ModernBERT]() Original publisher: [Read original article]() Author: Benjamin Warner; Antoine Chaffin; Benjamin Clavié; Orion Weller; Oskar Hallström; Said Taghadouini; Alexis Gallagher; Raja Biswas; Faisal Ladhak; Tom Aarsen Published: 2024-12-19T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Machine learning](), [Transformers](), [Retrieval-Augmented Generation](), [recommendation systems](), [PyTorch]() Tags: [ai](), [community](), [fine-tuning](), [flash-attention-2](), [gpu](), [language-models](), [nlp](), [open-source-collab](), [performance](), [rag](), [research](), [transformers]() ### AI overview The article introduces ModernBERT, a family of encoder-only models designed as improved replacements for BERT-like models. It describes 8,192-token context, stronger downstream performance, faster processing, base and large model sizes, compatibility with Transformers, and use cases including retrieval, classification, question answering, entity extraction, RAG pipelines, and recommendation systems. It also recommends Flash Attention 2 when supported by the GPU. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Improving Hugging Face Training Efficiency Through Packing with Flash Attention 2 DevFeed: [Improving Hugging Face Training Efficiency Through Packing with Flash Attention 2]() Original publisher: [Read original article]() Author: Rhui Dih Lee; Arthur Zucker; Achintya Kundu; Laura Wynter; Raghu Ganti; Mayank Mishra Published: 2024-08-21T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [hugging face](), [Transformers](), [Machine learning](), [GPU](), [datasets]() Tags: [datasets](), [efficiency](), [flash-attention-2](), [gpu](), [hugging-face](), [models](), [packing](), [padding](), [training](), [transformers]() ### AI overview The article explains how Hugging Face improves training throughput by packing instruction-tuning sequences without padding while using Flash Attention 2. A new data collator preserves sequence boundaries and can provide up to 2x higher throughput while maintaining convergence quality, with measured gains varying by dataset and sequence-length variance. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Falcon 2: An 11B parameter pretrained language model and VLM, trained on over 5000B tokens and 11 languages DevFeed: [Falcon 2: An 11B parameter pretrained language model and VLM, trained on over 5000B tokens and 11 languages]() Original publisher: [Read original article]() Author: Quentin Malartic; Nilabhra Roy Chowdhury; Ruxandra Cojocaru; Mughaira; Giulia Campesan; Sanath Narayan; Ankit Singh; Clémentine Fourrier; Nathan Habib Published: 2024-05-24T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [vlm](), [Large Language Model](), [Open Source](), [model-deployment](), [dataset](), [model architecture](), [Artificial Intelligence]() Tags: [3d](), [architecture](), [artificial-intelligence](), [community](), [data](), [ecosystem](), [flash-attention-2](), [inference](), [llm](), [model](), [multimodal](), [nlp](), [open-source](), [research](), [vision](), [vlm]() ### AI overview The article announces Falcon 2, a family of smaller open-source models from TII that includes an 11B pretrained language model and an 11B vision-language model. Falcon 2 targets improved performance, multimodal support, cheaper inference, and broader downstream use. The VLM supports text-based conversations about visual content, while the models primarily support English and have capabilities in ten additional languages. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Powerful ASR + diarization + speculative decoding with Hugging Face Inference Endpoints DevFeed: [Powerful ASR + diarization + speculative decoding with Hugging Face Inference Endpoints]() Original publisher: [Read original article]() Author: Sergei Petrov; Vaibhav Srivastav; Pedro Cuenca; Philipp Schmid Published: 2024-05-01T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [asr](), [inference-endpoints](), [hugging face](), [Inference](), [Whisper](), [Open Source](), [PyTorch]() Tags: [asr](), [audio](), [flash-attention-2](), [hugging-face](), [inference](), [inference-endpoints](), [open-source](), [pytorch](), [whisper]() ### AI overview This article explains how to build a custom inference handler for Automatic Speech Recognition, speaker diarization, and speculative decoding on Hugging Face Inference Endpoints. It covers modular pipeline design, repository files, Pyannote-based diarization, PyTorch SDPA with Flash Attention 2, and constraints on speculative decoding such as batch size one and compatible decoder architectures. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science.