Gradient Clipping

See section 10.11 here.

• Zhang et al 2020 - Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity
• An extreme form of gradient clipping, where everything gets clipped, is the “Manhattan-Learning rule” (see the Rprop paper). Rprop is an advancement over this.

• What is gradient clipping

• Optimizers