No clipping
g′ = g
Every component passes through. This preserves direction, but a rare outlier can make the update large enough to increase loss sharply or overflow finite precision.
Optimization stability laboratory
Stop the explosion. Keep the direction.
A large gradient can destroy an update. But the way you clip it matters: global-norm clipping rescales one vector, while per-coordinate clipping can bend it. Tune both and watch the optimizer move.
Gradient components and loss path
rawappliedloss
Global-norm clipping activates only when the full vector exceeds the threshold, preserving its direction.
Raw gradient norm--Before the clipping rule
Applied gradient norm--Magnitude used by the optimizer
Direction cosine--1.00 means direction preserved
Unstable steps--Loss spikes across 80 updates
Exploding gradients are about magnitude. Global-norm clipping limits magnitude without rotating the vector. Value clipping caps each component separately, so the optimizer may step in a different direction.
Global norm
g′ = g · min(1, τ / ‖g‖₂)
The whole vector receives one scale factor. Its norm is bounded by τ and its cosine with the raw gradient remains 1, apart from numerical precision.
Per coordinate
g′ᵢ = clamp(gᵢ, −τ, τ)
Each component is capped independently. The update is bounded, but unequal component scaling can rotate the vector and change the descent path.
When should clipping activate?
Track gradient norms over representative training runs. A threshold far below the ordinary norm clips every step and changes optimization; one far above the tail never protects the run. Useful thresholds typically target rare spikes rather than routine gradients.
Does clipping fix vanishing gradients?
No. Clipping constrains unusually large gradients. It does not amplify gradients that have already vanished through deep or recurrent computations.
What should training telemetry include?
Log the raw norm, the applied norm, the fraction of clipped steps, loss spikes, and any non-finite values. A stable loss curve without raw-norm telemetry can hide a threshold that is active almost constantly.