Vanishing and Exploding Gradients: How Depth Turns Local Derivatives into Exponential Effects

The core of this passage is the distinction between “the existence of a backpropagation path” and “an effective gradient being able to arrive.” A connected computational graph only shows that the derivative of the loss with respect to early parameters can be calculated through the chain rule; it does not guarantee that this derivative still has a large enough numerical value to produce appreciable parameter updates.

Each time the gradient passes through a node, it is multiplied by that node's local derivative. Early layers are farther from the loss function, so they experience more multiplications along the way; therefore depth accumulates local scale effects. If most multipliers in the relevant direction are less than 1, the product decays rapidly with path length, causing vanishing gradients; if most are greater than 1, the product grows rapidly, causing exploding gradients. The key is not just whether individual derivatives are “large or small,” but the overall scale after many local derivatives are multiplied together.

Numerical systems and the optimization process amplify this effect. Extremely small gradients may be close to the lower limit of floating-point representation or of effective updates; even if formally nonzero, they are not enough to change the parameters. Extremely large gradients may make the update step uncontrolled and cause weights, activations, or the loss to overflow to Inf/NaN. The optimizer receives the gradient after it has already been scaled by this chain, so it cannot automatically recover the useful signal lost along the way.

This yields an important inter-layer diagnostic feature: gradient problems are usually not a case of “the whole network slows down together.” With vanishing gradients, layers near the output can still learn normally because their backward path is short, while layers near the input are almost frozen and early features remain unchanged for a long time; with exploding gradients, numerical control may suddenly be lost at some step. Observing the gradient and parameter update magnitudes at different depths reveals the problem better than only looking at the overall rate of loss decrease.

This section uses the chain rule to turn "the network is very deep" into an analyzable mathematical object: a product of Jacobian matrices. Let the loss be L, and the early state be h_0, the k layer state be h_k, then the gradient of the loss with respect to the early state must pass through the local Jacobians of the subsequent layers in sequence J_k=∂ h_k/∂ h_{k-1}. Therefore, the scale and direction of the backward signal are jointly determined by the entire chain of Jacobian products, and the network depth K directly appears as the number of factors in the product.

In the scalar case, each Jacobian degenerates into an ordinary derivative. If the typical scaling factor of each layer along some direction is approximated as c, then after K layers the total scale is approximately c^K. This reveals the exponential effect of depth: when c=0.5 , after 10 layers it is about 9.77×10^{-4}, after 50 layers it is about 8.88×10^{-16}; even if c=0.9 looks only slightly smaller than 1, after 50 layers only about 0.00515 remains. Conversely, c=1.1 after 50 layers can grow to about 117.4, c=1.5 would reach about 6.38×10^8. So even a mild deviation of the local scale from 1 is accumulated by depth into a huge overall difference.

c=1 represents the idealized case where the scale stays unchanged, but a real vector network cannot be fully described by an average scale. In this case the Jacobian is a matrix, and different input directions are stretched or shrunk to different degrees; matrix multiplication also changes direction. It is more accurate to understand this through singular directions: some directions of the same network may keep shrinking during the product, while other directions are simultaneously amplified, so vanishing and exploding gradients can coexist.

The diagnostic value of this formula is that it locates the problem in the spectrum and directional behavior of the entire backward propagation chain, rather than in the individual derivative of a single layer. The scalar approximation c^K provides the first intuition about exponential accumulation; further analysis needs to pay attention to how the Jacobians of each layer stretch along different directions and how these directions combine across layers.

This section explains that activation functions do not merely determine the shape of the forward representation; their derivatives are also direct multipliers in the backpropagation chain. k layer forward computation is z_k=W_kh_{k-1}, h_k=φ(z_k). When backpropagation reaches here, the upstream gradient ∂ L/∂ h_k is multiplied element-wise by the slope at the current pre-activation:

Therefore, the inputs are the pre-activation z_k and the upstream gradient; the output is a gradient with unchanged shape but reweighted by each unit's local slope. When the slope is close to zero, the credit signal carried by that unit is suppressed; when the slope is 1, the activation itself does not change the backward scale at that location.

Sigmoid best demonstrates this mechanism. Its derivative is σ'(z)=σ(z)(1-σ(z)). When the output is close to 0 or 1, the derivative approaches zero, corresponding to the saturation regions at the two ends of the function; even at the most favorable z=0 point, the derivative is only 0.25. If ten layers all contain this activation factor, the product of the activation derivatives alone would produce 0.25^{10}≈9.5×10^{-7} contraction. The weights would need to provide roughly fourfold amplification in the corresponding direction at every layer to offset this term, but that would introduce the risk of runaway growth in other directions.

Different activation functions change the activation part of the local Jacobian. tanh can have derivative up to 1 at the center, but it still saturates when the state moves away from the center. ReLU has derivative 1 in the positive region, so it does not continuously compress the positive-region gradient downward as sigmoid does; however, its derivative is 0 in the negative region, and if a unit stays in the negative region for a long time, the gradient is completely cut off, forming a “dead” unit. GELU and SiLU have smoother curves and retain a small derivative in the negative region, alleviating the hard cutoff, but the small derivative still shortens the signal.

The full local Jacobian can be understood as J=D_φ W:D_φ consists of the activation derivatives of each unit, W provides the linear transformation. Changing the activation function only alters the former part, so it can significantly improve deep trainability but cannot by itself guarantee overall stability. Even if the activation derivatives are ideal, the weight matrix may still strongly contract or amplify in certain directions.

This section extends the "size" of the gradient from a single scalar to direction-dependent matrix behavior. When a linear layer is followed by an activation function, the local Jacobian can be written as J=D_\phi W. Here, the weight matrix W first applies a linear transformation to the signal, and the activation derivative matrix D_\phi then weights the backward signal according to the current position of each unit. The change in the gradient after passing through this layer depends not only on an overall norm, but also on the specific direction in which it lies.

The singular values of the weight matrix characterize the degree of stretching in different directions. Larger singular values correspond to directions that are amplified, and smaller singular values correspond to directions that are compressed. If during backpropagation the relevant direction repeatedly encounters a maximum singular value greater than 1, the gradient may be continuously amplified and eventually explode; if the direction carrying task information corresponds to a very small singular value, the gradient in that direction will rapidly decay. Because the same matrix can have both very large and very small singular values, the same layer can perfectly well amplify some directions while erasing others.

The activation derivative further changes this directional structure.D_\phi assigns different slopes to different units; some coordinates may be preserved, some may be significantly compressed or even set to zero. When combined with W after being combined, the final effective direction may not necessarily align with the original coordinate axes or W 's individual singular directions. Therefore, relying solely on either the weight norm or the activation derivative cannot fully determine whether the local Jacobian is stable.

This also explains why a “normal average gradient norm” can be misleading. A few especially large gradients can push the overall norm into a seemingly healthy range, while the gradients of most layers, channels, or parameters are already close to zero. A single aggregated scalar flattens direction imbalance and sparse anomalies, possibly masking vanishing gradients as well as local explosions.

More effective diagnostics should preserve structure: record gradient distributions separately by network layer, module, parameter type, and time step in sequence models, and examine the median, multiple quantiles, extreme values, and the proportion near zero; at the same time, compare the ratio of parameter updates to the scale of the parameters themselves. Only then can you determine which positions are still learning, which directions have frozen, and whether the overall norm is merely supported by a small number of outliers.

This section explains that although a recurrent neural network has only a few parameter layers in its structure diagram, unrolling it over time creates a very deep computational chain. The state update of a plain RNN is

The same set of state transition parameters is reused at every time step. If at step T the loss needs to send the credit signal back to step t step, the gradient must pass through T-t state-transition Jacobians. Therefore, the time distance in the sequence plays the role of the number of layers in a feedforward network: the farther apart they are, the greater the effective depth.

The local Jacobian at each time step contains the recurrent weight W_h and the current activation derivative. Backpropagation multiplies these Jacobians together one after another. Even if the scaling at a single step is only slightly less than or slightly greater than 1, repeating it dozens of times can produce exponential decay or amplification. Parameter sharing only means that each step uses the same matrix; it does not shorten the computation path from the loss to the earlier state. On the contrary, repeatedly multiplying the same matrix continues to accumulate its directional scaling effect.

Long-distance dependencies face two related but different difficulties. During the forward pass, state information left by earlier inputs may be continuously overwritten by later state updates, so that h_T no longer retains enough clues; during the backward pass, even if early information once influenced the state, the gradient from later losses may decay along the long path and fail to send back “how the early processing should be modified.” The former is a problem of preserving memory content; the latter is a problem of passing credit assignment signals.

Therefore, RNNs have difficulty learning information from dozens of steps earlier, not because there are many kinds of trainable parameters, but because both information and gradients must travel through a very long temporal chain. When analyzing such problems, you should treat sequence length as network depth and observe state sensitivity and backward gradients according to time distance, rather than judging depth only by the number of layers explicitly declared in model code.

The core point of this section is: LSTM and GRU use gating to construct a state path close to “identity propagation”, so that long-term information and gradients do not have to pass through the full product of recurrent weights and saturated activations at every time step.

In addition to the hidden state, LSTM also maintains a cell state c_t:

forget gate f_t determines how much of the old state to keep, and the input gate i_t determines how much of the candidate content \tilde c_t is written. This update has an approximately additive structure. When backpropagating along the cell state channel, the main local multiplier between adjacent time steps is f_t, rather than the “recurrent weight spectrum times nonlinear derivative” that appears repeatedly in ordinary RNNs. The model can therefore learn to make the dimensions that need long-term retention have their corresponding f_t close to 1.

Gate values close to 1 are crucial for long paths. Across 100 steps, if the effective multiplier per step is 0.99, the product is 0.99^{100}\approx0.366, which still retains a usable scale; if it is 0.9, then 0.9^{100}\approx2.66\times10^{-5}, the signal still almost disappears. This shows that the advantage of gating is not to eliminate the product, but to allow the multiplicative factor to be driven close to 1 by the data.

GRU does not have an independent cell state, but instead uses

When the update gate z_t is close to 0, the old state is almost copied unchanged, and the effective retention multiplier 1-z_t is close to 1, which can also form a relatively stable long-range channel. Compared with LSTM, GRU uses a single gate to bind retaining old information and writing new information into a convex combination; LSTM’s forget gate and input gate are relatively independent, so the control over “how much to retain” and “how much to write” is more separated.

The common mechanism of both structures is to convert the difficult-to-control “weight matrix scaling × activation derivative” in ordinary RNNs into learnable gate values, allowing the model to build paths close to identity mapping for important state dimensions. But this does not guarantee infinite memory: gate values that deviate from 1 over long periods still decay exponentially, and finite state capacity, optimization difficulty, and sequence noise also impair information retention. Gating provides a long-term channel that is easier to learn, not automatic permanent memory.

This section explains that the goal of initialization is not simply to make weights "as small as possible," but rather to keep the scales of forward activations and backward gradients as stable as possible when signals pass through multiple layers from the network's starting point. If each layer slightly reduces variance, depth will accumulate this contraction into vanishing; if each layer amplifies variance, it may cause explosion, and may also push activations such as sigmoid or tanh into saturation regions, making derivatives even closer to zero. Therefore, neither too small nor too large is a safe choice.

Xavier/Glorot initialization sets the weight variance based on input width and output width, with a typical scale of approximately 2/(\mathrm{fan\_in}+\mathrm{fan\_out}). It compromises between preserving variance in the forward and backward passes, suitable for activations that are near linear or approximately symmetric, such as tanh. He/Kaiming initialization has a typical variance of approximately 2/\mathrm{fan\_in}, targeting the fact that ReLU-type activations cause about half of the units to output zero, using a larger initial variance to compensate for the energy loss.

These two types of methods mainly constrain the average variance in a statistical sense. Orthogonal initialization, on the other hand, addresses the problem from the directional perspective: the columns of an orthogonal matrix are perpendicular to each other and have length 1, and all singular values are 1, so it only rotates vectors and does not amplify or compress any direction. When used as the weight starting point, the local Jacobian J=D_\phi W in W initially has a scaling factor close to 1 in all directions, so the norm of deep linear paths is more stable. This corresponds more directly to the singular value perspective than merely guaranteeing average variance.

Orthogonal weights are usually also multiplied by a gain gain, to pre-compensate for the energy changes brought by subsequent activations. ReLU on average truncates about half of the energy, so it is often paired with \sqrt{2}\approx1.414 gain; tanh can use an empirical gain slightly greater than 1. If the gain does not match the activation, the activation derivative will still destroy the original scale-preserving property.

Initialization addresses the conditions at the beginning of training, rather than permanent stability throughout the entire training process. Parameter updates gradually change the weight spectrum, and changes in data and parameters also change the activation distribution; a Jacobian that is healthy at initialization may deviate from the stable region after training. Therefore, sensible initialization needs to be used together with residual paths, normalization, learning rate scheduling, and continuous monitoring of gradients and activations.

This section puts multiple "stabilized training" techniques back into the Jacobian product chain for comparison; the key conclusion is that they operate at different locations and solve different problems, so they cannot be treated as interchangeable just because they all reduce training collapse.

Residual connections write the module as y=x+F(x), whose Jacobian is

The identity matrix I corresponds to an identity path that does not need to go through the branch transformation. Even if the branch F has small derivatives in some directions, gradients can still pass through the identity term, thereby improving deep credit assignment. But residuals do not automatically limit the branch scale; if ∂F/∂x is large, the overall Jacobian I+∂F/∂x can still amplify the gradient.

Normalization here is used to adjust the numerical scale of intermediate activations; the normalized values can also be rescaled and shifted via learnable parameters γ,β rescaled and shifted. Keeping intermediate values in a relatively stable range reduces the chance of pre-activations drifting into the sigmoid/tanh saturation regions and improves local optimization conditions. However, normalizing the overall scale does not equate to making every singular value of the weight matrix equal to 1; the original strength differences between different directions may still be preserved.

Gradient clipping occurs after complete backpropagation. If the global gradient norm ‖g‖ exceeds the threshold τ, the gradient is globally multiplied by τ/‖g‖, preserving the direction while limiting the length. It is suitable for handling spikes caused by gradient explosion or anomalous batches, but cannot fix vanishing gradients, because when the gradient is originally only 10^{-8} clipping will not be triggered at all.

Loss scaling is numerical protection in low-precision training. Before the FP16 backward pass, the loss is first multiplied by a larger coefficient S, so that tiny gradients fall into the representable range, and before the parameter update it is divided back by S. It can prevent originally nonzero gradients from being rounded to zero due to floating-point underflow, but it cannot change the fact that the mathematically true gradient is already very small.

Gating, on the other hand, establishes long-term state pathways through learnable retention and write coefficients, mainly targeting credit propagation along the time dimension, and it does not equal unlimited memory. This shows that: residual connections change the Jacobian structure, normalization adjusts activation scales and local conditions, gating controls state transmission, clipping limits the final large gradient, and loss scaling protects numerical representation. Stable systems often need to combine these mechanisms and choose tools based on the specific diagnosed failure locations.

This section presents a diagnostic method in the style of an “evidence chain”: loss stagnation is only a surface phenomenon. Only by connecting task learnability, forward states, backward gradients, parameter updates, and path length can the root cause be localized to vanishing gradients.

The first step is to establish a learnability baseline. Have a small or shallow model overfit on a small batch of data. If even this minimal experiment fails, you should first check the data, labels, loss function, and implementation; if a shallow model can learn while a deep model on the same task cannot, then the depth-related backward path becomes the primary suspect.

The second step is to record per-layer activations and per-layer gradients simultaneously. On the activation side, observe the mean, variance, saturation proportion, and the zero proportion for ReLU to determine whether the signal shrinks with depth or enters a saturation region. On the gradient side, do not record only a single global norm; instead record per-layer norms, medians, maxima, and finite proportion, and compare layers near the input and output. If the gradients of layers near the input are systematically much smaller than those near the output, this is consistent with multiplicative decay along a long path.

The third step is to connect gradients to actual updates. Record the ratio of “parameter update amount / parameter count” to determine whether small gradients still produce visible updates after passing through the optimizer. If gradients look normal but parameters do not change at all, check the learning rate, optimizer parameter groups, parameter freezing, or scaling settings instead of directly attributing it to vanishing gradients.

Depth ablation can provide stronger causal evidence: while keeping the task and main settings unchanged, reduce 50 layers to 5 layers. If the shallow version recovers learning, it indicates that path length is highly correlated with failure. Then replace only one mechanism at a time, such as initialization, activation function, or Residual Connection, and observe whether the per-layer gradient profile recovers; single-variable intervention can determine the cause more reliably than stacking multiple tricks at once.

Different observations correspond to different faults: if all layer gradients are None or zero, it looks more like the computation graph is cut off, parameters are frozen, or the loss does not depend on these parameters; occasional huge spikes and NaN point to explosion, abnormal batches, or precision overflow; if gradients are normal but parameters do not change, it points to optimizer configuration. Zero gradients in FP16 may also be floating-point underflow, which must be distinguished from mathematical multiplicative decay through numerical precision checks. Ultimately, only when “depth-dependent gradient profile + learnability baseline + single-variable intervention recovery” all hold together can you more convincingly prove that the problem actually comes from vanishing gradients.

This section organizes vanishing and exploding gradients into a complete causal chain. The starting point is that the scalar loss at the output produces the gradient; afterward, as backpropagation passes through each layer, it must multiply by that layer's local Jacobian. The local Jacobian is jointly determined by the current activation derivative, the directional scaling of the weight matrix, and the path the gradient actually takes, so each step may compress, amplify, or rotate the signal.

A slight scale deviation in a single layer does not necessarily cause an obvious anomaly immediately, but depth or temporal distance makes it recur. After successive Jacobians are multiplied together, effective scales less than 1 decay exponentially, and effective scales greater than 1 amplify exponentially. An activation entering its saturation region makes the local derivative approach zero; the weight spectrum can cause different directions to shrink and amplify simultaneously. The final result appears as almost no effective update in early layers near the input, or as the global gradient suddenly losing control and producing extreme updates and numerical anomalies.

Various stabilization techniques should be understood according to where they act in the causal chain. Appropriate initialization first prevents the forward and backward scales at the start of training from drifting too early. Gating establishes a learnable state-preserving channel for long-term dependencies; residual connections add an identity shortcut path; normalization controls activation scale and improves local conditioning. These mechanisms directly alter the long-term propagation path or the environment of the local Jacobian. Gradient clipping, in contrast, limits overly large final updates after the backward computation is complete, and precision management is used to avoid underflow or overflow in low-precision representations; they address extreme values and numerical representation problems at the back end of the chain.

The causal chain also determines the diagnostic approach. You cannot conclude in reverse that the original problem must be the mechanism that a technique usually targets just because training recovers after adding that technique; multiple changes may affect the result simultaneously. A more reliable method is to monitor activation, gradient, and update magnitudes layer by layer, and then use depth ablation and univariate intervention to observe whether anomalies appear with path length and whether they recover at specific locations.

Therefore, training success or failure is not determined by an isolated local derivative, but by the entire chain formed jointly by “initial gradient—local Jacobian—path length—parameter update—numerical representation.” Understanding where each mechanism is inserted into the chain enables you to choose targeted fixes and verify the true root cause with observational evidence.

This section presents a concept dependency graph centered on the gradient propagation problem. The main thread continues to deepen from four aspects: mathematical origins, structural modifications, numerical conditions, and optimization safeguards.

"Backpropagation" is the foundation for understanding Jacobian multiplication. Each node in the computation graph provides only a local derivative; backpropagation reuses these local Jacobians through the chain rule, ultimately forming a long product from the loss to early states or parameters. Once this layer is understood, vanishing and exploding gradients are no longer merely empirical phenomena, but can be located in specific paths, specific local transformations, and specific scaling directions.

"Residual Connection" corresponds to shortening the effective backward path. The identity branch directly preserves the input in the forward pass and, in the backward pass, causes the module Jacobian to include an identity matrix term, so that deep signals do not have to rely entirely on the complex branch being passed layer by layer. This explains how structural design directly changes the Jacobian chain, rather than merely adjusting the initial numerical values of parameters.

"Normalization" and "Batch Normalization" correspond to stabilizing the activation scale and local conditions. They reduce the drift of activation distributions across layers and during training, preventing pre-activations from frequently entering unfavorable regions. However, stabilizing the activation scale and stabilizing the gradient spectrum are different problems: unifying the overall statistical scale does not mean that the singular values in each direction are close to 1. This distinction connects the statistical perspective with the matrix-direction perspective.

"Recurrent Neural Network" extends the problem to sequences. Gating constructs a relatively stable temporal state channel through learnable retention coefficients, while the more general structural goal is to reduce or bypass backward paths that span very long time distances. This allows the long-term dependency problem to be understood as an effective depth problem in the time dimension.

"Optimizers and Learning Rate Schedules" belong to the optimization-level safeguards. Gradient clipping limits abnormally large updates, warm-up controls the step-size shock in early training, and precision management prevents underflow and overflow in low-precision representations. These mechanisms need to work together, but they do not replace structural improvements to the Jacobian chain itself.

The entire dependency graph ultimately forms a unified mapping: backpropagation explains the origin of the multiplication; initialization controls the starting-point scale; gating and residual connections change long paths; Normalization improves activation scale and local conditions; clipping, scheduling, and precision management protect the final update and numerical representation. Only by placing these concepts on the same causal chain can we distinguish which link each method actually changes.