Parameters and data remove different bottlenecks.
Larger models improve representational capacity. More tokens improve estimation and coverage. A fixed compute budget cannot maximize both independently.
Loss versus allocation
Parameters0B
Training tokens0B
Tokens/parameter0
Relative loss0
Power laws turn budget allocation into an optimization problem.
Compute couples N and D.
Transformer training compute is approximately proportional to parameter count times training tokens.
C ≈ 6NDQuality changes effective tokens.
Cleaner, more relevant data can contribute more learning signal per raw token.
Overtraining can still be practical.
Training beyond the compute-optimal token count may produce a smaller model that is cheaper to serve.
Scaling curves are empirical.
Architecture, tokenizer, data mixture, optimizer, and evaluation distribution all affect fitted exponents.