Microbatch × accumulation × workers.
Gradient accumulation lets training imitate a larger batch without holding every sample in memory at once. Tune microbatch size, accumulation steps, sequence length, and worker count while watching optimizer cadence and memory.
Run the microbatch timelineEach microbatch contributes gradients into the same buffers. After the chosen number of accumulation passes, the optimizer updates parameters once. Loss scaling, distributed synchronization, randomness, normalization, and learning-rate schedules still determine whether the run behaves like a true large batch.
Microbatch × accumulation × workers.
Relative to a 1 × 2,048-token microbatch.
Fewer parameter updates as effective batch grows.
Communication events under the selected policy.
For ordinary data-parallel training, the sample count behind one optimizer update multiplies across microbatch size, accumulation passes, and workers.
effective batch = microbatch × accumulation × workersIf each microbatch loss is averaged, divide it by the accumulation count before backward so the summed gradient matches the average over the effective batch. Framework helpers may do this automatically; verify rather than assume.
loss_for_backward = microbatch_loss / accumulation_stepsDropout draws, batch-dependent normalization, sample packing, clipping timing, mixed-precision overflow, scheduler steps, and asynchronous data order can make accumulated training differ from processing the whole batch at once.