Two-layer MLP parameters divided across the selected devices.
Sharded matrix path
The large matrix is divided among devices. Animated links show the collective needed to reassemble outputs or sum partial results.
Four matrix shards are ready.
Approximate bytes sent per device for the required collective.
Compute and communication with the selected overlap policy.
Useful single-device compute divided by aggregate device time.
Shard the multiplication, then pay only for the boundary.
Column parallelism splits output features.
Each device stores a vertical slice of the weight matrix and produces a distinct slice of activations. A later operation may consume those slices directly or gather them.
Y_i = X W_i, where W = [W_1 ... W_p]Row parallelism sums partial outputs.
Inputs and weight rows are partitioned together. Every device computes a partial result, then an all-reduce sums those partial outputs.
Y = sum_i X_i W_iMegatron pairs both directions.
A column-parallel projection feeds a row-parallel projection without gathering the intermediate activation, leaving one collective at the block boundary.
More devices shrink local matrices.
Parameter memory and matrix work fall roughly with tensor-parallel degree, provided dimensions divide cleanly across devices.
Communication limits strong scaling.
As compute per device falls, collective traffic becomes a larger part of layer time. Fast links and overlap determine whether more shards help.