# TapeSAC: Automated, Pareto-Optimal Fine-Grained Memory Management for Large Model Training

DevFeed: [TapeSAC: Automated, Pareto-Optimal Fine-Grained Memory Management for Large Model Training](<https://devfeed.tech/articles/tapesac-automated-pareto-optimal-fine-grained-memory-management-for-large-model-training-116500.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04897>)

Author: Kaihui Ren, Yifan Zhao, Zejian Wang, Yingtong Xiong, Xiaoling Zhou, Daohong Xu, Chenhao Guan, Zhikang Mo, Jihu Guo, Zhuoru Ma, Yufeng Li, Lin Zhang, Yiqiao Liao, Chengru Song

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [automated](<https://devfeed.tech/tags/automated.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [large](<https://devfeed.tech/tags/large.md>), [memory](<https://devfeed.tech/tags/memory.md>), [memory-management](<https://devfeed.tech/tags/memory-management.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [out-of-memory](<https://devfeed.tech/tags/out-of-memory.md>), [planner](<https://devfeed.tech/tags/planner.md>), [planning](<https://devfeed.tech/tags/planning.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [qwen3-6](<https://devfeed.tech/tags/qwen3-6.md>), [qwen3-8](<https://devfeed.tech/tags/qwen3-8.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

TapeSAC automates selective activation checkpointing in eager PyTorch without changes to model forward code. It records block operations at the dispatcher and, during backward, replays only producers of values autograd consumes; a planner chooses tensors to save using measured recomputation costs under a step-peak memory limit. On Qwen3.8-27B and Qwen3.6-35B-A3B, replay alone cut step time by 3.0% and 1.7% versus full checkpointing at no greater memory cost; memory-aware planning improved those gains to 6.3% and 13.9% with a 90%-of-device-memory limit. The evaluated hand-tuned policies fell short of the reported frontier, and one ran out of memory at larger batches. Across 500 real-data steps, TapeSAC reduced training time by 8-11%; on Qwen3.5-397B it was 2.7% faster than a model-specific hand-tuned policy while using 14% less memory.

## Source excerpt

arXiv:2611.04897v1 Announce Type: new Abstract: Selective activation checkpointing (SAC) decides which activations to save for the backward pass and which to recompute: each choice is a point in the plane of peak memory and recomputation time, and the goal is that plane's Pareto frontier. Current practice falls short of this goal. Hand-tuned policies fix a single point that does not adapt when the model, batch or kernels change; PyTorch's SAC reruns each block's forward pass in program order, recomputing values the backward pass never reads; and the memory budget of torch.compile treats recomputation as free or forbidden and selects saved tensors by bytes alone, an assumption that fails for the custom kernels that produce many of a large model's activations. We present TapeSAC, which automates SAC in eager PyTorch without editing the model's forward code. TapeSAC records each block's forward pass at the dispatcher and, during backward, replays only the producers of the values autograd consumes, so saving a tensor eliminates every producer needed only for it. A planner selects the saved tensors from measured recomputation costs under a single limit on step-peak memory. On Qwen3.8-27B and Qwen3.6-35B-A3B, replay alone reduces step time by 3.0% and 1.7% versus full checkpointing at no greater memory cost; memory-aware planning raises these gains to 6.3% and 13.9% under a 90%-of-device-memory limit. The evaluated hand-tuned policies fall short of this memory-recomputation frontier and, on Qwen3.6-35B-A3B, run out of memory at larger batches. Over 500 steps on real data, TapeSAC reduces training time by 8-11%; on Qwen3.5-397B, it is 2.7% faster than a model-specific hand-tuned policy while using 14% less memory.