Build one batch across many smaller passes.

Gradient accumulation lets training imitate a larger batch without holding every sample in memory at once. Tune microbatch size, accumulation steps, sequence length, and worker count while watching optimizer cadence and memory.

Run the microbatch timeline
forwardbackwardadd gradientsno optimizer step yetrepeatoptimizer stepzero gradientsforwardbackwardadd gradientsno optimizer step yetrepeatoptimizer stepzero gradients

Accumulation changes cadence, not the math you can ignore.

Each microbatch contributes gradients into the same buffers. After the chosen number of accumulation passes, the optimizer updates parameters once. Loss scaling, distributed synchronization, randomness, normalization, and learning-rate schedules still determine whether the run behaves like a true large batch.

Microbatch timeline

Ready · gradients zeroed
accumulated gradient0 / 4 passes
Effective global batch32

Microbatch × accumulation × workers.

Activation memory index1.0×

Relative to a 1 × 2,048-token microbatch.

Optimizer steps / 1K samples31.3

Fewer parameter updates as effective batch grows.

Sync operations / step1

Communication events under the selected policy.

Effective batch size

For ordinary data-parallel training, the sample count behind one optimizer update multiplies across microbatch size, accumulation passes, and workers.

effective batch = microbatch × accumulation × workers

Scale the loss deliberately

If each microbatch loss is averaged, divide it by the accumulation count before backward so the summed gradient matches the average over the effective batch. Framework helpers may do this automatically; verify rather than assume.

loss_for_backward = microbatch_loss / accumulation_steps

Equivalence has limits

Dropout draws, batch-dependent normalization, sample packing, clipping timing, mixed-precision overflow, scheduler steps, and asynchronous data order can make accumulated training differ from processing the whole batch at once.

Primary reading

Automatic Mixed Precision: Gradient Accumulation PyTorchPerforming Gradient Accumulation Hugging Face AccelerateAccurate, Large Minibatch SGD: Training ImageNet in 1 Hour Goyal et al.