# State-Aware Activation Recomputation with Replay Contracts and Selective Preservation

DevFeed: [State-Aware Activation Recomputation with Replay Contracts and Selective Preservation](<https://devfeed.tech/articles/state-aware-activation-recomputation-with-replay-contracts-and-selective-preservation-115057.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.00590>)

Author: Annan Zhang, Shengjie Song, Wenshuai Yao, Lihao Zhang, Tianmiao

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Process](<https://devfeed.tech/topics/process.md>), [Stateful](<https://devfeed.tech/topics/stateful.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [budget](<https://devfeed.tech/tags/budget.md>), [buffer](<https://devfeed.tech/tags/buffer.md>), [complete](<https://devfeed.tech/tags/complete.md>), [contracts](<https://devfeed.tech/tags/contracts.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [faster](<https://devfeed.tech/tags/faster.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [independent](<https://devfeed.tech/tags/independent.md>), [jax](<https://devfeed.tech/tags/jax.md>), [memory](<https://devfeed.tech/tags/memory.md>), [planner](<https://devfeed.tech/tags/planner.md>), [queue](<https://devfeed.tech/tags/queue.md>), [random](<https://devfeed.tech/tags/random.md>), [records](<https://devfeed.tech/tags/records.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [region](<https://devfeed.tech/tags/region.md>), [retention](<https://devfeed.tech/tags/retention.md>), [snapshots](<https://devfeed.tech/tags/snapshots.md>), [state](<https://devfeed.tech/tags/state.md>), [training](<https://devfeed.tech/tags/training.md>)

## AI overview

Activation recomputation can produce incorrect gradients if replay sees changed mutable buffers, aliased bytes, or random-generator state. The paper introduces versioned replay contracts that specify what state to restore and a joint planner that chooses activation retention, whole-buffer snapshots, or selective-region preservation per segment, accounting for measured memory and recovery costs. Derivative and byte-level checks validate replay, with a Torch/JAX cohort retaining 5,900 execution records. On sparse-write H100 workloads, joint planning cuts paired step time by 18.16% versus an eager/whole planner at 44 GiB; at 32 GiB it is 4.98% faster than fixed regions, where the eager/whole planner has no feasible static policy. Results differ for the CIFAR-100 dense-queue application: joint planning uses 22.41% more allocated memory than whole snapshots, with similar process time (geometric-mean ratio 1.0080). Thus selective preservation benefits the tested sparse-write workloads, while whole snapshots use less allocated memory in the tested dense-queue workload.

## Source excerpt

arXiv:2611.00590v1 Announce Type: new Abstract: Activation recomputation saves accelerator memory by replaying forward segments during the backward pass, but replay is only correct if it sees the state the original forward pass saw. Mutable buffers, aliased bytes and random-generator state can change in between, so re-executing unchanged code can differentiate a different computation. We make this state explicit. A versioned replay contract specifies what replay must restore, and a joint planner chooses, per segment, between activation retention, whole-buffer snapshots and selective-region preservation, charging each option its measured memory and recovery cost. Independent derivative and byte-level checks validate the contract, and a Torch/JAX cohort retains all 5,900 execution records, including matches, rejections and divergences. On frozen sparse-write H100 workloads, each measured over fifty fresh processes, joint planning reduces paired step time by 18.16% against a correctly costed eager/whole planner at a 44 GiB budget (approximate 97.5% ratio interval $[.8172,.8195]$). At 32 GiB, where the eager/whole planner has no feasible static policy, joint planning is 4.98% faster than fixed regions at 29.94 rather than 26.20 GiB. A complete CIFAR-100 dense-queue application runs seven methods for 100 encoder and 25 probe epochs at six training seeds, with byte-identical test scores within each seed. There, joint planning uses 22.41% more allocated memory than whole snapshots, with a process-time geometric-mean ratio of 1.0080 (seed range $[.9845,1.0191]$). Together, the results map where selective preservation pays off: sparse writes with large activations benefit, while in the dense-queue workload whole snapshots use less allocated memory.