Recurrent Neural Networks in Computer Vision: When Temporal Memory Earns Its Cost

Recurrent neural networks explained for vision: how LSTM and GRU fix vanishing gradients, how RNNs differ from CNNs, and where they still beat…

Recurrent Neural Networks in Computer Vision: When Temporal Memory Earns Its Cost
Written by TechnoLynx Published on 16 Apr 2025

A recurrent neural network (RNN) is a network that keeps a hidden state, a form of memory updated at every step from the current input and the previous state, so what it saw earlier shapes how it reads what comes next. That memory is the key feature of recurrent neural networks, and in computer vision it earns its cost in three places: video, where frames depend on each other; OCR, where a text line is read as a sequence; and compact or high-resolution backbones, where self-attention gets expensive. The price is sequential training through backpropagation through time and gradients that vanish or explode over long sequences. The rest of this article covers the mechanism, how LSTM and GRU repair it, how it differs from a CNN, and where transformers have and have not taken over.

What makes a recurrent neural network recurrent

The whole idea is one extra connection. A standard hidden layer computes its output from the input alone, while an RNN adds a term that references the previous hidden state: h_t = tanh(W_xh·x_t + W_hh·h_{t−1} + b_h). Because h_t feeds h_{t+1}, information can pass across an arbitrary number of consecutive steps, which is what lets the network handle long-distance dependencies. The same operation runs for every item in the sequence while a memory of the processed items is kept for later steps.

Two consequences matter for engineers. First, the hidden state lets an RNN process inputs and produce outputs of variable length, which a plain CNN with fixed-length input and output vectors cannot do. Second, an RNN has an infinite impulse response, whereas a convolutional network has a finite one.

Any finite-time RNN can be unrolled along time into an equivalent feedforward network. That is how it is trained, but at deployment the recurrent connections are reused across cycles rather than physically duplicated.

Are LSTM and GRU recurrent neural networks, and what do they fix?

Yes. LSTM and GRU are recurrent networks whose state is controlled by gates, often called gated memory. They were designed to solve the plain RNN’s vanishing gradient problem.

Why gradients vanish in a plain RNN

During backpropagation, the gradient is multiplied by the derivative of each step’s hidden state with respect to the previous one; unless that factor is exactly one, it shrinks or amplifies the gradient, and repeating it over many steps has an exponential effect. Error gradients in standard RNNs vanish exponentially with the time lag between the events that matter. The opposite failure, exploding gradients, has a blunt but effective fix: Pascanu, Mikolov and Bengio (2013) proposed clipping the gradient norm, alongside a soft constraint for the vanishing case.

How the LSTM cell keeps the gradient alive

Long short-term memory was introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997 to address exactly this. Alongside the hidden state, an LSTM keeps a memory cell whose gradient with respect to its previous value is exactly one, because the previous cell value is added directly to the new one (c_t = i_t ⊙ u_t + c_{t−1}). The original paper calls this a constant error carousel, with multiplicative gates learning to open and close access to it, and reports bridging time lags in excess of 1,000 discrete steps. The variant most people now mean by “LSTM” adds a forget gate.

What GRU changes

The gated recurrent unit came from Cho et al. (2014), introduced alongside the RNN encoder-decoder, with a reset gate and an update gate that control how much each unit remembers or forgets. The update gate interpolates between the previous hidden state and a new candidate, and the reset gate modulates how much of the old state feeds that candidate. A GRU has no separate cell and slightly fewer parameters than an LSTM, because it lacks an output gate.

On accuracy, Chung et al. (2014) found both gated units better than plain tanh units and GRU comparable to LSTM. The evidence is not unanimous: a 2019 speech-and-vision survey reports another study’s experimental finding that the original LSTM structure was, in general, superior for various recognition tasks. The GRU-RCN video work chose GRU because Chung et al. reported similar performance to LSTM with a lower memory requirement.

Recurrent neural network vs convolutional neural network

The short answer: a CNN shares filters across spatial positions, while an RNN performs the same operation at each step and keeps a memory of earlier ones. CNNs are built on local connectivity, shared weights and pooling, and are designed for multidimensional array data such as images. RNNs carry state, which makes them the natural fit for ordered data where context from earlier steps informs the current prediction.

Property CNN RNN (incl. LSTM, GRU)
What is shared Filters across spatial positions Weights across time steps
Receptive field Symmetric around the kernel centre One-sided: only previous inputs
Memory horizon Finite impulse response Infinite impulse response
Training parallelism Parallel across positions Each step waits for its predecessor
Gradient behaviour Convolutions do not suffer vanishing or exploding gradients Vanishing and exploding gradients
Typical vision role Spatial feature extraction Temporal or sequential reasoning over CNN features

ReNet (2015) replaced each convolution-plus-pooling layer with four RNNs sweeping the image in both horizontal and vertical directions, and performed comparably to deep CNNs on MNIST, CIFAR-10 and SVHN. Its stated drawback was that it is not easily parallelisable, a limit that applies to model parallelism and not to data parallelism. More recently, work on still images argues that one recurrent layer can reach a large receptive field without more parameters and so replace several convolution layers, which matters for compact embedded models.

Recurrent CNNs: the most common hybrid in vision

Recurrent convolutional networks apply a 2D CNN to each frame and feed the activations to an RNN that models how the video changes over time, and LSTM units are typically placed after the CNN to capture inter-frame dependencies. Donahue et al. describe these long-term recurrent convolutional models as “doubly deep”, compositional in both spatial and temporal layers, and suited to cases where concepts are complex or training data are limited.

The reason for the hybrid shows up in the numbers. In Ballas et al.’s ICLR 2016 study, putting an RNN on top of only the final VGG-16 features barely moved accuracy (78.1 against 78.0), which the authors read as the CNN’s top layer discarding short-term temporal variation. A bidirectional GRU-RCN drawing on convolutional maps from several CNN levels reached 80.7, and as a captioning encoder it gave a 10% BLEU gain over the VGG-16 encoder. The lesson we take from it: where the recurrence sits matters as much as whether it exists. For how temporal layers fit into the rest of a production detector, see our piece on object detection in videos and temporal CV pipelines.

Is a recurrent neural network deep learning?

Yes. Surveys of deep neural networks list RNNs next to CNNs, autoencoders, GANs and attention models, and RNNs often contain millions of parameters. Depth here has two axes: the stacked layers, and the time dimension, since unrolling for training turns the RNN into a feedforward network whose depth grows with the number of time steps. A recurrent model can reach arbitrary computational depth with a finite parameter count by multiplying its limited spatial resources along time.

Where recurrent networks struggle: long sequences, parallelism and memory

The standard training method is backpropagation through time (BPTT), and every extra recurrent step lengthens the computational path that must be retraced to update the weights. Three practical limits follow.

The first is parallelism. Recurrence over time to update the hidden state intrinsically precludes parallelisation, and gated RNNs can still struggle to hold long-term dependencies. VisionGRU’s authors (2024) put it directly: BPTT remains inefficient for long sequences, which led to the gradual replacement of RNNs by transformers. Feng, Bengio et al. (2024) go further and say traditional LSTMs and GRUs were too slow to scale beyond a few hundred tokens. On a GPU this shows up as a counterintuitive profile: in the Conv-TT-LSTM paper, a model needing fewer FLOPs was not faster because of its sequential structure, running 27.3 minutes per epoch at 37.83 GFLOPs against 26.2 minutes at 55.83 GFLOPs for ConvLSTM.

The second is memory. A deep RNN must unfold as many times as the sequence is long, so memory and compute expand with depth, especially on visual sequences. For long inputs, LSTM and GRU models hold intermediate results for every cell gate, which a 2023 speech-processing review contrasts with temporal convolutional networks whose backward path depends only on network depth.

The third is depth. Simply stacking RNN layers on high-dimensional visual input is very hard to train, which is why most published systems use a shallow RNN over extracted CNN features. In OCR, a side experiment in a 2020 CRNN evaluation, which the authors did not pursue, found that stacking two LSTMs raised mean character error rate to 1.9% and three to 8.6%, against 0.99% for the single-LSTM default. Training stability needs attention too; the Conv-TT-LSTM authors needed gradient clipping to train at their chosen learning rate.

Why do OCR engines still use LSTM and GRU?

Because line-based models process whole text lines at once, with no character segmentation needed. The original CRNN by Shi, Bai and Yao combined feature extraction, sequence modelling and transcription into one end-to-end trainable network that handles sequences of arbitrary length with no character segmentation. Line-based CRNNs are trained with connectionist temporal classification (CTC), which Graves et al. (2006) introduced to train RNNs on unsegmented sequences directly, removing the need for pre-segmented training data.

Tesseract’s VGSL specification gives, as a network “capable of high quality OCR”, a bidirectional LSTM with 256 outputs reading a 48-pixel-high greyscale line as a sequence of vertical pixel strips, with a CTC-trained output layer of 105 classes. The same documentation notes that a summarising LSTM is currently the only way in that framework to collapse a dimension of unknown size, which is how variable-height text images are handled.

A 2020 multi-parameter evaluation of line-based CRNNs (CNN plus LSTM, with GRU also used successfully) reached 0.44% character error rate from only 10,000 training lines, outperforming available pretrained models trained on more than 20 times as much data. It also found that non-binarised greyscale input reliably beat high-quality binarised input on historical material.

PaddleOCR’s PP-OCRv3 documentation states that RNN is abandoned in its SVTR recogniser in favour of a transformer structure that mines line context more effectively, and TrOCR (Li et al., 2021) replaced the usual CNN-plus-RNN design with pre-trained image and text transformers for both stages.

Have transformers replaced recurrent neural networks?

For many sequence tasks, especially in language, transformers now dominate, but RNNs remain relevant where efficiency matters. The 2017 transformer paper dispensed with recurrence and convolution entirely and reported better translation quality with more parallelism and much less training time. Transformers have since become the dominant architecture for many sequence tasks, especially in language, because they handle long-range dependencies better and parallelise more readily, and a 2019 survey reports attention-based models outperforming RNNs on sequential, context-based processing.

Transformers also have a scaling limit of their own: their cost is quadratic in sequence length. For video that bites hard, because attention has to cover spatial dimensions as well as time, which the Conv-TT-LSTM authors describe as prohibitive in memory and computation. RNNs remain relevant where computational efficiency, real-time processing or the sequential nature of the data is crucial.

By removing hidden-state dependencies from the gates, Feng, Bengio et al. (2024) made minimal LSTMs and GRUs trainable with a parallel scan, reporting a 175× training-step speedup for minGRU over GRU at sequence length 512 on a T4 GPU. They use the parallel mode for training and the sequential mode at inference, at a cost of roughly 88% more memory than the traditional versions. VisionGRU (2024) builds an image backbone on minGRU with linear complexity and bidirectional 2D scanning; its tiny model reports 82.0% top-1 accuracy against 72.2% for DeiT-Ti and 76.1% for Vim-Ti, and VisionGRU-B reports 83.1% against DeiT-B’s 81.8% at a comparable parameter budget. The authors attribute the advantage at higher resolutions to avoiding self-attention’s cost there. These are single-paper results, not a settled ranking, and we would want to see them reproduced on dense-prediction tasks before betting a product on them.

Choosing a temporal layer for a vision pipeline

Situation Reasonable default Why
Action recognition or captioning on top of a pretrained CNN Shallow GRU or LSTM over multi-level CNN features Stacked deep RNNs are hard to train on visual input; multi-level features beat top-layer-only
Printed or historical text lines, modest training data CRNN: CNN plus bidirectional LSTM with CTC Segmentation-free, arbitrary length; 0.44% CER from 10,000 lines in one evaluation
New OCR or scene-text recogniser builds Evaluate a transformer recogniser PP-OCRv3’s SVTR and TrOCR both moved recognition to transformers
High-resolution or long video where attention memory is the bottleneck Evaluate a recurrent or linear-complexity model (ConvLSTM, minGRU-based backbones) Attention cost on spatio-temporal input; linear complexity at high resolution
Spatially correlated sequences such as radar or motion fields ConvLSTM rather than fully connected LSTM ConvLSTM handles spatiotemporal correlation better, and state-to-state kernels larger than 1 matter
Short clips where training throughput dominates 3D convolution or temporal convolution Convolutions parallelise and avoid recurrent gradient problems, though 3D CNNs consume a lot of compute

The table assumes the temporal layer is the open question; for the surrounding pipeline, see our note on video pipeline optimisation. For a contrast before adding recurrence to a detector, see how multiple object tracking works in production. For the wider architecture landscape the RNN sits in, see deep learning for computer vision.

Choosing between recurrence and attention is an algorithmic decision, and the largest performance improvements are frequently available at the algorithmic level. The question that stays open after the table is empirical: on your sequence lengths, resolution and target hardware, does the recurrent model’s lower memory beat the transformer’s better parallelism once both are profiled end to end? Benchmark papers cannot answer that for a specific deployment; it is settled by profiling both options on the target hardware during computer vision R&D work.

FAQ

What is the exploding gradient problem?

It is the mirror image of the vanishing gradient: when the per-step derivative is larger than one, repeated multiplication during backpropagation amplifies the gradient exponentially. Pascanu, Mikolov and Bengio (2013) proposed clipping the gradient norm as a simple, effective fix.

What are the types of RNN?

They include the plain (vanilla) RNN and the two widely used gated structures, LSTM and GRU. The convolutional LSTM extends the LSTM with convolutional input-to-state and state-to-state transitions.

Back See Blogs
arrow icon