# Fixing Gradient Accumulation

DevFeed: [Fixing Gradient Accumulation](<https://devfeed.tech/articles/fixing-gradient-accumulation-7222.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/gradient_accumulation>)

Author: Lysandre; Arthur Zucker; Zachary Mueller; Yih-Dar SHIEH; Benjamin Bossan; Pedro Cuenca

Published: 2024-10-16T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [gradient\_accumulation](<https://devfeed.tech/topics/gradient-accumulation.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [bug](<https://devfeed.tech/tags/bug.md>), [gradient-accumulation](<https://devfeed.tech/tags/gradient-accumulation.md>), [training](<https://devfeed.tech/tags/training.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

## AI overview

The article explains a bug in gradient accumulation for token-level tasks, where averaging per-batch losses produced results that did not match full-batch training. It describes a fix that accounts for the total number of non-padding tokens and introduces an API for supplying custom loss functions.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.