# Variance-Reduced Stochastic Hessian-Driven Nesterov Acceleration with Adam-Style Preconditioning

DevFeed: [Variance-Reduced Stochastic Hessian-Driven Nesterov Acceleration with Adam-Style Preconditioning](<https://devfeed.tech/articles/variance-reduced-stochastic-hessian-driven-nesterov-acceleration-with-adam-style-preconditioning-125351.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11277>)

Author: Yaxin Yu, Long Chen, Minfu Feng

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Combinatorial optimization](<https://devfeed.tech/topics/combinatorial-optimization.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [accelerated](<https://devfeed.tech/tags/accelerated.md>), [acceleration](<https://devfeed.tech/tags/acceleration.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [convex](<https://devfeed.tech/tags/convex.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-na](<https://devfeed.tech/tags/cs-na.md>), [gap](<https://devfeed.tech/tags/gap.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [iteration](<https://devfeed.tech/tags/iteration.md>), [lower-bound](<https://devfeed.tech/tags/lower-bound.md>), [math-na](<https://devfeed.tech/tags/math-na.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [noise](<https://devfeed.tech/tags/noise.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [scalar](<https://devfeed.tech/tags/scalar.md>), [storm](<https://devfeed.tech/tags/storm.md>), [terminal](<https://devfeed.tech/tags/terminal.md>), [theory](<https://devfeed.tech/tags/theory.md>), [type](<https://devfeed.tech/tags/type.md>), [variance](<https://devfeed.tech/tags/variance.md>)

## AI overview

VR-Adam-SHANG combines STORM-style recursive gradient estimation, Adam-style preconditioning, and SHANG acceleration for smooth stochastic convex optimization. Its scalar and diagonal variants use at most two stochastic gradient evaluations per iteration and require no full-gradient refresh. Under unbiased sampling, bounded variance, and mean-square smoothness of gradient noise, polynomial steps with prescribed initialization give an O(1/K) mean-square gradient-estimation error and expected objective-gap bounds with O(1/K²) initial-energy decay plus an O(1/√K) stochastic term. The guarantees do not require a monotone preconditioner or a uniform positive lower bound on its eigenvalues. The analysis also proves polynomial-step convergence for projected nonrecursive Adam-SHANG-s. In experiments at matched sample-gradient budgets, at least one VR-Adam-SHANG variant had the lowest mean terminal objective gap in every tested setting.

## Source excerpt

arXiv:2611.11277v1 Announce Type: cross Abstract: Stochastic first-order methods use sampled gradients to reduce per-iteration costs, but estimation errors can slow convergence and, in some settings, cause divergence. To improve gradient accuracy while retaining acceleration, we develop scalar and diagonal versions of VR-Adam-SHANG for smooth stochastic convex optimization. The methods combine STORM-type recursive estimation, Adam-style preconditioning, and SHANG's accelerated interpolation and correction, using at most two stochastic gradient evaluations per iteration without full-gradient refreshes. Under unbiased sampling, bounded variance, and mean-square smoothness of gradient noise, polynomial steps with prescribed initialization yield $O(1/K)$ mean-square gradient-estimation error and expected objective-gap bounds combining $O(1/K^2)$ initial-energy decay with an $O(1/\sqrt K)$ stochastic term after $K$ iterations. These guarantees require neither monotonicity of the preconditioner nor a uniform positive lower bound on its eigenvalues. The analysis also establishes polynomial-step convergence of projected nonrecursive Adam-SHANG-s, filling a gap in its original theory. Experiments compare the methods at matched sample-gradient budgets. In every tested setting, at least one of the two versions achieves the lowest mean terminal objective gap among all compared methods.