# Improving Hugging Face Training Efficiency Through Packing with Flash Attention 2

DevFeed: [Improving Hugging Face Training Efficiency Through Packing with Flash Attention 2](<https://devfeed.tech/articles/improving-hugging-face-training-efficiency-through-packing-with-flash-attention-2-7433.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/packing-with-FA2>)

Author: Rhui Dih Lee; Arthur Zucker; Achintya Kundu; Laura Wynter; Raghu Ganti; Mayank Mishra

Published: 2024-08-21T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [datasets](<https://devfeed.tech/topics/datasets.md>)

Tags: [datasets](<https://devfeed.tech/tags/datasets.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [flash-attention-2](<https://devfeed.tech/tags/flash-attention-2.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [models](<https://devfeed.tech/tags/models.md>), [packing](<https://devfeed.tech/tags/packing.md>), [padding](<https://devfeed.tech/tags/padding.md>), [training](<https://devfeed.tech/tags/training.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

## AI overview

The article explains how Hugging Face improves training throughput by packing instruction-tuning sequences without padding while using Flash Attention 2. A new data collator preserves sequence boundaries and can provide up to 2x higher throughput while maintaining convergence quality, with measured gains varying by dataset and sequence-length variance.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.