Recurrent Neural Network (RNN): Compressing History into a Continuously Updated State

The basic approach of an RNN is to reuse the same state-update unit over time. At time step t, it receives the current input xₜ and the previous hidden state hₜ₋₁, and computes a new hidden state:

hₜ = φ(Wₕhₜ₋₁ + Wₓxₜ + b)

Here, Wₕ transforms the old state into the current state space, Wₓ transforms the current input, b is the bias, and φ is the state activation function. Then, the current hidden state can also be mapped through

yₜ = g(Wᵧhₜ)

to produce the output yₜ; Wᵧ performs the output transformation on the state, and g maps the transformed result into the output form required by the task.

The key to this computation is not that each time step has a new set of parameters, but that all time steps share the same Wₕ, Wₓ, and b. When the sequence grows from three time steps to ten, the model simply executes the same update process a few more times; the number of parameters does not increase with sequence length. Therefore, parameter sharing allows the same model to accept sequences of different lengths, and also forces it to use a consistent state-update rule at every position.

The hidden state hₜ can be understood as a history summary formed up to time t for the current task. It is determined jointly by hₜ₋₁ and xₜ, so the current state both carries forward past information and absorbs the current input; but it is not a complete log from which all past details can be freely retrieved. Which historical information is retained, weakened, or overwritten depends on the shared transformation and the training objective.

For example, take a three-day device signal x = [1, 0, 2], suppose the hidden state has only one value, φ is the identity function, Wₕ = 0.5, Wₓ = 1, b = 0, and h₀ = 0. Then every step uses the same update equation hₜ = 0.5hₜ₋₁ + xₜ. This setup clearly distinguishes two sources: 0.5hₜ₋₁ is the contribution of the past state to the current one, and xₜ is the new information brought by the current day's signal. Processing data from more days simply repeats this equation; it does not add new Wₕ, Wₓ, or b for a new day.

Parameter sharing also means that the same transformation and its error act repeatedly over time. On one hand, it brings scalability in sequence length; on the other hand, it makes the state updates at each step chained together: earlier inputs can influence later results only by being passed through successive hidden states.

The recurrence not only gives “the current state is determined jointly by the past and the present,” but also, through step-by-step substitution, reveals how much each historical input contributes to the final state. Using a one-dimensional RNN: input x = [1, 0, 2], Wₕ = 0.5, Wₓ = 1, b = 0, h₀ = 0, and setting the activation function to the identity function. The state update therefore simplifies to:

hₜ = 0.5hₜ₋₁ + xₜ

The three time steps can be computed term by term:

In the first step there is no contribution from past states, so h₁ = 1. In the second step, although the current input is 0, the previous state is multiplied by 0.5 and continues to propagate, so h₂ = 0.5. In the third step, h₂ is first scaled to 0.25, then the current input 2 is added, giving h₃ = 2.25. A current input of zero does not mean the state is cleared; as long as the old state still has a nonzero contribution after recurrence, the historical influence is retained.

Further expanding the intermediate states, we can directly see which inputs are mixed into h₃:

h₃ = 2 + 0.5 × 0 + 0.5² × 1 = 2.25

The weight of the current input x₃ is 1, the weight of the previous input x₂ is 0.5, and the weight of the input two steps earlier x₁ is 0.5² = 0.25. That is, the same Wₕ is multiplied again each time it crosses a time step. For Wₕ = 0.5, the farther an input is from the current moment, the more its influence decays by powers of 0.5; here, for x₁ to reach h₃ it must cross two state transitions, so only a weight of 0.25 remains.

This expansion reveals the boundary of the recurrent state: history is not preserved equally but is repeatedly scaled by the state transformation. When the effect of Wₕ is less than 1, earlier information may decay quickly; if Wₕ is changed to 1.2, the same influence will grow by powers of 1.2 with the number of propagation steps, so early information no longer decays but the state and gradients are more likely to explode. Therefore, the proportion of each segment of history in the final state is determined both by the input values and by the number of times the shared state transformation is repeatedly applied over time.

When the forward computation of an RNN is written as a recurrence, it seems to contain a “loop,” but during training this loop can be unrolled along the time axis. A sequence of length T forms a computation graph of depth T: each time step has a state update node, h₁ depends on h₀ and x₁, h₂ depends on h₁ and x₂, and so on up to h_T. Figure 1 shows this time unrolling for a three-step RNN, with the forward computation proceeding in the order x₁ → h₁, x₂ → h₂, x₃ → h₃.

The multiple nodes in the unrolled graph are not multiple distinct models, but the same set of parameters reused at different time steps. Suppose a loss occurs at a later time step; its effect on earlier states and shared parameters propagates backward along the dependency chain h_T → h_T₋₁ → …. This process is called backpropagation through time, or BPTT. For the three-step unrolling in Figure 1, backpropagation passes through three positions where parameters are reused and adds up the gradient contributions from each use. Thus, the same parameter is updated both because of the impact it had at the current time step and because it bears responsibility for indirectly affecting future losses through subsequent states.

Parameter sharing does not mean that the gradient is computed only once during training. Let a shared parameter be W, and suppose it participates in the state update at every time step; then the gradient of the total loss with respect to W includes the sum of contributions from each reuse. Each contribution may also pass through state chains of different lengths: the earlier the use, the longer the backward path to a later loss usually is. BPTT incorporates these direct and indirect effects into the gradient computation through the ordinary computation graph produced by unrolling.

The cost of full unrolling grows with sequence length. Backpropagation requires the intermediate activations from the forward phase, so the longer the sequence, the more activations must be retained and the longer the gradient propagation paths become. To limit this overhead, truncated BPTT can be used: every K time steps the computation graph is cut off, and backpropagation is performed only within a window of length K. In this way, the memory required for training changes from depending on the full sequence length to depending mainly on the window length K.

Truncation involves an explicit trade-off. Losses within the window can still assign responsibility to shared parameters along the unrolled graph, but direct credit paths that cross the cutoff point and extend beyond K steps no longer participate in that round of backpropagation. Therefore, the smaller K is, the shorter the memory footprint and the backward paths for a single backpropagation pass; at the same time, the model finds it harder to connect state updates from long ago to the current loss through direct gradients.

The influence of an earlier state hₖ on a later state hₜ must pass through every state update between k and t. Connecting these local rates of change with the chain rule gives:

∂hₜ/∂hₖ = ∏(i = k + 1 … t) [diag(φ′ᵢ)Wₕ]

Here, the product runs over each intermediate time step i. Wₕ is the state weight reused at every step; φ′ᵢ is the derivative of the activation function at that step with respect to each coordinate, and diag(φ′ᵢ) denotes the diagonal matrix formed from these derivatives. The key implication of the formula is that the gradient across multiple time steps does not add the influences of individual steps but multiplies the Jacobian matrices of the steps consecutively.

In the one-dimensional linear case, the activation derivative is 1, and the gradient across d steps simplifies to Wₕᵈ. This explains why Wₕ = 0.5 and Wₕ = 1.2 lead to two opposite extremes. Across ten steps:

0.5¹⁰ ≈ 0.00098

The influence of the earlier state on the later state is only about one thousandth, so the responsibility from later loss can hardly be propagated back to early steps, resulting in vanishing gradients. Conversely:

1.2¹⁰ ≈ 6.19

The same influence is continuously amplified, making gradients more likely to increase sharply and resulting in exploding gradients. In the multidimensional case, what determines the result is no longer a scalar power, but the mechanism by which repeated multiplication causes influence to decay or amplify remains unchanged.

Nonlinear activations also participate in every multiplication. If the activation enters a saturated region, φ′ approaches zero, and even if Wₕ itself has no obvious shrinking effect, multiplying multiple near-zero derivatives further suppresses long-distance gradients. Therefore, whether later loss can be effectively allocated to earlier states depends on the combined effect of state weights and activation derivatives along the entire path.

Gradient clipping can limit the update magnitude when gradients become too large, but it does not make the state automatically remember information for longer; gating, residual connections, or shorter paths can improve gradient propagation, but cannot recover content that previous state updates have already erased. Whether gradient propagation is stable and whether the state still retains the required information are two related but different questions, and long-term memory adequacy cannot be judged solely from the gradient norm.

LSTM breaks “what to save, what to write, and what to expose outward” into learnable continuous controls, so that the state update does not have to force the old information completely through a nonlinear layer at every step. It maintains a cell state cₜ and uses three gates to control the state flow:

cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ

hₜ = oₜ ⊙ tanh(cₜ)

cₜ₋₁ and cₜ are the cell states before and after the update, respectively, and c̃ₜ is the candidate content formed from the current information. The symbol ⊙ denotes element-wise multiplication, so different state coordinates can receive different control strengths. fₜ is the forget gate, which determines how much of each component of the old cell state cₜ₋₁ is retained; iₜ is the input gate, which determines how much of the candidate content c̃ₜ is written in; the two parts are added to form the new cell state cₜ. oₜ is the output gate, which controls how much of the tanh-transformed cell state is exposed as the hidden state hₜ.

The key to this structure is the approximate additive pathway in the cell state. When certain coordinates satisfy fₜ ≈ 1 and iₜ ≈ 0, the update becomes approximately cₜ ≈ cₜ₋₁: old content continues along a path close to the identity mapping, neither being overwritten by a large amount of new content, nor requiring the gradient to pass through the complete candidate-state nonlinearity at every time step. Conversely, when the model needs to replace memory, it can reduce fₜ and increase iₜ; when the internal state should temporarily not affect external computation, it can reduce oₜ without having to immediately delete the content in cₜ.

The gates are not manually written symbolic rules such as “if a certain word appears, save it permanently,” nor are they discrete hard storage slots. fₜ, iₜ, and oₜ are all continuous vectors driven by data and varying with the input and state. They provide coordinate-wise soft control: values close to 1 indicate stronger passage, values close to 0 indicate stronger suppression, and intermediate values indicate partial retention, writing, or exposure.

The approximate identity pathway also does not mean lossless indefinite memory. Even if the forget gate reaches f = 0.99 at every step, after 500 steps the retention ratio of old content is still:

0.99⁵⁰⁰ ≈ 0.0066

In other words, a small per-step decay can still become significant after long-range accumulation. LSTM improves the information and gradient decay caused by repeated nonlinear transformations in a plain tanh RNN, but how long it can preserve memory is still determined by the continuous product of the gate values over the entire time span.

GRU adopts a related but more compact gating approach, merging the state with some gates and therefore having fewer parameters. Whether using LSTM or GRU, the core is not to eliminate recurrence, but to add learnable retention, writing, and exposure paths to the recurrent state.

An autoregressive sequence generator must use the previous output as the input at every step. During training, the true yₜ₋₁ is often used as the input at step t; this practice is called teacher forcing. It makes the model start from the correct prefix at every position, so it can directly learn “given that the previous answer is correct, what should the next item be.”

During inference there is no true future answer available to feed in, so the model can only pass its own prediction from the previous step into the next step. Training and inference therefore face different input distributions: the training phase mainly sees true prefixes, while the inference phase sees prefixes composed of its own predictions, which may contain errors. This mismatch is called exposure bias.

Error snowballing comes from the causal chain of state recurrence. Suppose the model predicts incorrectly at some step; this erroneous output then becomes the input to the next step and moves the hidden state into a region that was rarely seen during training. Later predictions are produced from this deviated state, and even if each step has only a local deviation, the deviation may continue to be passed on as input and accumulate. Good single-step performance under teacher forcing therefore does not guarantee the same stability when freely generating long sequences.

Several approaches mitigate this problem at different points, but each has its own limitations:

Scheduled sampling changes the training input, giving the model a chance to learn how to continue generating from its own deviations; sequence-level loss shifts the optimization objective toward the final result; beam or constrained decoding improves inference search without changing training. These three deal with different stages and cannot be equated with one another. In particular, stronger decoding strategies, even if they reduce some local errors, do not eliminate the fundamental difference that the model mainly relies on true prefixes during training.

An RNN produces a hidden state at every time step; the differences among tasks lie mainly in which states are read and which states are turned into outputs, rather than any fundamental change in the recurrence mechanism itself.

For sequence classification, a many-to-one readout can be used: the model processes the entire input sequence in order and sends only the final state to the classifier. The final state acts as a summary of the whole sequence, and the output is a sequence-level class. This suits tasks where multiple input time steps correspond to one result, but it also means that classification information must be preserved through the recurrence until the end.

For step-by-step labeling, a many-to-many approach can be used: each input xₜ produces a corresponding state hₜ, which then outputs the label yₜ at that position. Inputs and outputs are aligned over time steps, so the decision at each position can use the context built up by recurrence up to that point. If a task requires generating another sequence of possibly different length from an input sequence, an encoder–decoder structure can be used: the encoder first compresses the input into a state, and the decoder then generates the target sequence step by step starting from that state.

A bidirectional RNN changes the range of context that each position can read. One direction runs the recurrence from left to right and the other from right to left, then combines the representations formed by the two directions at the same position. In this way, the output at position t can use both the left-side content and the right-side content, making it suitable for offline labeling where the complete sequence is already visible at inference time.

Bidirectional reading is not the same as “predicting the future.” It can use right-side information because the entire sequence is already given at the start of inference; the right-side content is known input, not data that has not yet occurred. Strictly online tasks do not satisfy this condition. For example, a real-time sensor can only process observations that have already arrived and cannot let the current output read future signals that have not yet arrived, so it cannot directly use a bidirectional structure that depends on right-side context.

A stacked RNN treats the state sequence of one layer as the input sequence for the next layer. More layers can increase model capacity and allow higher layers to continue transforming the lower layers’ temporal representations; the cost is that the optimization path from the output back to earlier layers and earlier time steps becomes longer. Choosing the final state, per-step states, the encoder state, bidirectional states, or multi-layer states is essentially about matching the output granularity, available context, and temporal constraints required by the task.

RNN and self-attention structures both process sequences, but they differ in how they access history. An RNN continuously compresses what it has already read into a fixed-size state; self-attention or Transformer, by contrast, retains keys and values for each position, i.e., KV, allowing later positions to connect directly to specific earlier positions. The two designs trade off state size, parallelism, and precise lookback capability.

The inputs to an RNN form strict dependencies over time: before computing hₜ, one must first obtain hₜ₋₁, so during training it is difficult to complete the state updates for all positions simultaneously. The benefit it gets in return is that during streaming processing, only the fixed-size current state needs to be passed to the next step. No matter how many inputs have been processed, the recurrent state that must be carried for a single-step update does not grow with the length of the history.

Self-attention retains KV for each position. During training, representations can be formed in parallel for all positions, and later positions can also directly read earlier positions through attention. It does not require all history to be compressed into a small state first, so when a task needs to refer to specific content from long ago, direct connections are easier to establish. Correspondingly, the KV retained during streaming inference grows with sequence length.

“Fixed state” versus “full KV history” represent different trade-offs, and neither is faster or better under all conditions. Actual speed also depends on compute kernels, batch size, and hardware. For example, online anomaly detection where only one sensor value arrives per second requires continuous, low-latency state updates; the fixed streaming state of a small GRU may be more suitable. Conversely, if a task needs to accurately quote a sentence thousands of tokens earlier, a fixed state must preserve the specific details over the long term, while self-attention can directly connect to the old position and usually better matches this access need.

Therefore, when choosing a structure, first determine how the task uses history: if the history can be continuously compressed and streaming-state size and low latency matter more, a recurrent structure has clear value; if the task needs to retain many positions and frequently look back precisely, an attention structure that preserves position-level history is more suitable.

Average accuracy lumps different lengths, different dependency distances, and different operating conditions into a single number, so it cannot by itself prove that an RNN has truly learned long-distance dependencies. If most samples in the test set require only short-term information, the model may fail clearly on long-distance samples while the overall average still looks good. Evaluation should separately observe “whether the task is answered correctly,” “how far back history remains usable,” and “whether state can be reliably maintained in production.”

Long-distance dependency capability can be plotted as a performance curve against dependency distance, rather than reporting only a single average. The horizontal axis represents how far the information needed for the current judgment is from the current position, and the vertical axis represents performance for that corresponding interval. If performance drops rapidly with distance, this indicates that success on short sequences cannot be extrapolated to long-distance conditions. Testing should also include distractors to check whether irrelevant inputs overwrite useful state; actively reset the state to confirm whether the model truly relies on history; simulate missing packets to observe behavior after the state chain is interrupted or inputs are omitted; and use sequences longer than the training length to test length extrapolation ability.

Deployment evaluation must also cover computational and state-management constraints. For streaming systems, measure per-step latency, hidden state size, batch throughput, cold start, and recovery after disconnection. Per-step latency reflects how long after a new input arrives a result can be produced; state size determines how much runtime data must be stored per concurrent session; cold start and disconnection recovery test how the system re-establishes reliable state when there is no continuous history or history is temporarily interrupted. These metrics answer different questions from offline average accuracy.

Hidden state is session state because it compresses the context formed from previous inputs. It must be bound to the correct user, device, or session identity, and have an explicit lifecycle with support for reset and versioning. If one user's hidden state is mistakenly reused for another user, the latter's predictions will be disrupted by the former user's context, causing context cross-talk and possible privacy leakage.

Model version and state version must also correspond. After a model upgrade, the numerical meaning of old hidden states may no longer be compatible with new parameters; handing old states directly to the new model may cause the new model to continue recursion from an internal representation that does not match its training distribution. Therefore, states need an identifiable version, and upgrades should handle them according to a compatibility policy, resetting when necessary, rather than treating hidden states as a model-agnostic universal cache.

Only when the model has been tested across dimensions such as dependency distance, distractors, anomalous input, and state lifecycle can we distinguish whether it is “correct on average samples” or can actually maintain and use long-term state under target deployment conditions.

RNN addresses how to let the current computation use previous history when sequence information arrives step by step. The input is not compressed into a static object all at once; instead, it enters the model time step by time step. The shared recurrent unit receives the current input and the old hidden state at each step and produces a new hidden state. Thus, the state becomes the channel through which history influences the current output, while parameter sharing allows this update rule to be reused repeatedly at different positions and for different sequence lengths.

This forward causal chain can be written as:

Sequence arrives step by step → shared recurrence updates the hidden state → the current state influences the current or subsequent output

Shared recurrence only specifies how the computation runs forward. To train the parameters, the loop must also be unrolled along time into a computation graph whose depth grows with sequence length. Later losses trace back along state dependencies through BPTT, assigning responsibility to the places where shared parameters are used at each time step:

Time unrolling forms a deep computation graph → BPTT backpropagates along the state path → gradient contributions from each parameter reuse accumulate

The same state transformation appears repeatedly along a long path, causing Jacobians in the gradient to be multiplied successively. If the product keeps shrinking, responsibility for earlier steps gradually vanishes; if it keeps growing, it leads to exploding gradients. This mechanism explains why ordinary recurrent structures find it difficult to stably learn very long-range dependencies, and it also shows that merely lengthening the input sequence does not guarantee that the model can effectively use earlier information.

Gating structures add learnable retention and write controls to the recurrent chain. The forget gate determines how much of the old state is retained, the input gate determines how much candidate content is written, and the output gate determines how much of the internal state is exposed. When the retention path is close to an identity mapping, information and gradients do not have to pass through the full nonlinearity at every step, making it easier to span more time steps. But gating is still a continuous mechanism with cumulative decay; it does not constitute infinite capacity or absolutely lossless storage.

Final acceptance must return to the temporal structure of the task and the deployment conditions. If long-range dependencies are to be verified, performance should be observed by dependency distance, with disturbances, state resets, missing packets, and extra-long sequences added, rather than looking only at average accuracy. If the system runs online, one should also check per-step latency, state size, cold start, reconnection recovery, and whether the state is correctly bound to the session. Only in this way can we distinguish “model scores high on samples” from “the model actually maintains and uses history under the target length and streaming constraints”.

The complete verifiable chain is therefore:

Sequence arrives step by step → shared recurrence compresses history → time unrolling exposes cross-step dependencies → BPTT assigns responsibility for future loss → the Jacobian product determines long-range gradient stability → gating regulates retention and writing → acceptance is carried out by length, abnormal conditions, and streaming state management

Each link corresponds to an observable result: the state recurrence determines how history enters the current computation, the unrolled graph determines where gradients are propagated back from, gate values determine how content is retained or overwritten, and evaluation by distance and by scenario tests whether these mechanisms truly satisfy the task boundary.