# Memorization capacity of deep ReLU neural networks characterized by width and depth

DevFeed: [Memorization capacity of deep ReLU neural networks characterized by width and depth](<https://devfeed.tech/articles/memorization-capacity-of-deep-relu-neural-networks-characterized-by-width-and-depth-141545.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.09589>)

Author: Xin Yang, Yunfei Yang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [c](<https://devfeed.tech/tags/c.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [construction](<https://devfeed.tech/tags/construction.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-na](<https://devfeed.tech/tags/cs-na.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [deep](<https://devfeed.tech/tags/deep.md>), [delta](<https://devfeed.tech/tags/delta.md>), [lower-bounds](<https://devfeed.tech/tags/lower-bounds.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [math-na](<https://devfeed.tech/tags/math-na.md>), [minimal](<https://devfeed.tech/tags/minimal.md>), [networks](<https://devfeed.tech/tags/networks.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [relu](<https://devfeed.tech/tags/relu.md>), [size](<https://devfeed.tech/tags/size.md>)

## AI overview

The paper constructs ReLU networks with adjustable width W and depth L that can memorize any N points in the unit ball separated by distance δ, with labels from a finite set of size C. The construction satisfies W²L² log W = O(N(log(δ⁻¹) + log C)); the paper also establishes width-depth lower bounds that explicitly depend on δ and C. When log(δ⁻¹) is on the order of log N, suitable width and depth give W²L² on the order of N. The authors state this order is optimal when log C is on the order of log N, and optimal up to logarithmic factors when log C grows more slowly than log N.

## Source excerpt

arXiv:2603.09589v2 Announce Type: replace Abstract: This paper studies the memorization capacity of deep neural networks with ReLU activation. Specifically, we investigate the minimal size of such networks to memorize any $N$ data points in the unit ball with pairwise separation distance $\delta$ and labels from a finite set of size $C$. Most prior studies characterize the memorization capacity by the number of parameters or neurons. We extend these results by constructing ReLU networks with adjustable width $W$ and depth $L$ satisfying $W^2L^2\log W=\mathcal{O}(N(\log(\delta^{-1})+\log C))$ that can memorize any such dataset. We also establish width-depth lower bounds that depend explicitly on $\delta$ and $C$ for networks capable of memorizing all such datasets. When $\log(\delta^{-1})\asymp\log N$, our construction achieves $W^2L^2\asymp N$ for suitable choices of width and depth. This order is optimal if $\log C\asymp\log N$, and optimal up to logarithmic factors if $\log C=o(\log N)$.