Introduction to Recurrent Neural Network (RNN)

Back to Articles

Standard Feedforward networks are great. We can solve complex problems with it. But it has some limitations that makes unsuitable for certain types of problems.

Consider we want to create a Language Translator, think Google Translate.

English: “The green apple“
Spanish: “La manzana verde“

Limitations of Standard Feedforward Network

1. The Variable Array Shape Problem

Remember that when we build a feedforward network from scratch, we define the weights/biases with exact, fixed dimensions. But almost every time, we want to translate words and sentences of varying length.

In a standard network, the output calculation is a matrix multiplication.

$$ y = W \cdot x + b $$

In our example above, we have to design a standard feedforward network that accepts 3-word sentences.

  1. Input $x$: This becomes an array with a shape of (3, 1).
  2. Weights $W$: To make the math work, we initialize our weight matrix to have 3 columns. Let’s say we want to have 5 neurons. The weight matrix shape is (5, 3).

When we run the forward pass np.dot(W, x), the math works perfectly because the sizes of both $W$ and $x$ match.

But as mentioned above, we always want to translate words and sentences with varying length. So what happens if we want to translate a 4-word sentence? Well the weights wouldn’t match $x$: (4, 1), $W$: (5, 3). We wouldn’t be able to perform dot multiplication.

Standard feedforward networks lock us into a rigid array size. We cannot dynamically resize our $W$ array for every new sentence ($x$), because $W$ holds the learned memory of the network.

2. Order and Context Problem

Different languages arrange words in completely different orders. If we look at our example again:

  • English: “The green apple” (Adjective, then Noun).
  • Spanish: “La manzana verde” (The apple green -> Noun, then Adjective)

To translate this correctly, a system must see the word “green”, hold it in its memory, wait to see what the next word is (“apple”), and then output the translation in the reversed Spanish order.

Another example, many words change meaning depending on the words that came right before them.

  • Sentence 1: “The dog’s bark,” bark = sound
  • Sentence 2: “The tree’s bark,” bark = wood on a trunk

As we can see the “meaning” of the word (bark) is defined by the words around/near it.

A standard network cannot wait. When it sees “green”, it immediately output “verde”, it blindly spits out words in the exact 1-to-1 order they arrive, resulting in the grammatically broken translation.

And it cannot hold things in memory. When it sees a word (bark), it will guess whether the meaning of the word bark is sound or the wood since it processes the word by itself.

Solution

1. Solving Variable Array Shapes by looping

Remember that our problem is varying input shapes that leads to mismatch with the shape of the weights.

  • Sentence A: “Hello”
  • Sentence B: “Where is the train station?”

In a standard feedforward network the size of our input $x_a = 1$ (1-word input) and $x_b = 5$ (5-word input). You can imagine that if the standard feedforward network is trained to accept and translate 8-word input, both Sentence A and Sentence B won’t be translated.

But what if instead of accepting a whole sentence at once, it breaks down the sequence down into a list of individual words?

The concept is that we will always use 1 word per time step. So instead of having something like this, which the math won’t work

$$ h = W_{(5,3)} \cdot x_{a(1, 1)} $$

We will have something like:

$$ h = W_{(5, 1)} \cdot x_{t(1,1)} $$

Where $x_t$ = “Where”. Then we can loop or do this over and over again until we finish the sentence.

$$ h = W_{(5,1)} \cdot x_{t-n(1, 1)} $$

It doesn’t matter how long the sentence, the math will always work out since we are only processing one word at a time.

2. Solving the order and context problem

Now that we have a way to handle varying input size, let’s see how we can solve the order and context problem.

Let’s look at the standard feedforward network formula.

$$ h = W \cdot x + b $$

If we use our solution above to handle varying input size, our x is just a single word. And if we loop this formula on each word, then as we can see each word is proceed by itself.

But what if we do it like this:

As we can see above, we add the “previous context” when processing the current word.

In the example, by the time we are processing the word “station”, h_3 has the context of all the previous words.

The general formula is:

$$ h_t = tanh(W_{xh} \cdot x_t + W_{hh} \cdot h_{t-1} + b) $$

Where:

  • $h_t$: Hidden state of the current time step.
  • $W_{xh}$: Weight matrix that transforms the current input.
  • $x_t$: Current input (the current word) of the time step.
  • $h_{t-1}$: Hidden state from the step before this one (previous hidden state).
  • $W_{hh}$: Weight matrix that transforms the previous hidden state. This is the mathematical representation of the loop. It dictates how much of the past memory to keep and how to mix it with the new data.

Intuitively, we can think of this as just adding the present $W_{xh} \cdot x_t$ and the past $W_{hh} \cdot h_{t-1}$. We wrapped the formula in an activation function, usually $tanh$ (hyperbolic tangent) to prevent the number from quickly getting massive or shrinking to zero.

RNN Architectures

There are different architectures we can use depending on the problem we are dealing with.

One-to-Many

Real-world example: Image Captioning. Input 1 picture -> Output the image caption “A dog on a mat.”

Many-to-One

Real-world example: Sentiment Analysis. A sequence of words (a movie review) -> Output a single rating, “Positive” or “Negative” review.

Many-to-Many

Synced

Real-world example: Action Recognition. Frames of a video (e.g. someone jumping) -> The output is a label for every frame (e.g. “Airborne”, “Landing”). It is synced because the model must classify what is happening at every millisecond of the video.

Delayed (Encoder-Decoder)

Real-world example: Language Translation. A whole English sentence goes in -> The whole translated Spanish sentence comes out. This solves the order and context problem.