Introduction to Convolutional Neural Network

Recommended read before jumping in this post:

• Introduction to Neural Networks
• Building an Image Classifier from Scratch

Convolutional Neural Network (CNN) is often used in image processing. But if you’re like me, you might ask – we’re already able to create an image classifier using Fully Connected Neural Network (FNN) (see my post Building an Image Classifier from Scratch):

• Why do we need another kind of neural network?
• What’s the difference between CNN and FNN?

Our goal in this post is to explore CNN and answer these questions.

Limitations of FNN

Let’s review how FNN works in our Image Classifier. Let’s look a single MNIST image, we know that an image are made up of pixels.

(pixels are smaller, I only made it larger for our visual)

We also know that our computers don’t “see” images the way we humans do. It’s more accurate to say that our computers see images like this.

(The numbers are the grayscale intensity).

Then we arrange the pixels from the “grid” to a single line (flatten) before feeding it into a standard FNN.

[0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.34 0.75 0.22 ...]

1. Loss of Spatial Information

An image is a grid. A pixel’s meaning is defined by the pixels around it. It’s not apparent with the MNIST image above, but imagine a picture of a face.

We don’t actually appreciate this often since it is “common knowledge” but we humans recognize a face by detecting “features”. What are these “features”?

1. “two eyes” at the “top”.
2. Below and in the middle of the two eyes is the “nose”.
3. Then below the nose is the lips.

So the position of these features are important information. Now imagine that we convert the grid image to a single line of pixels. The spatial information is lost. In a single line representation of the image, we lost the information that there is a nose below and between the eyes.

2. Parameter Explosion

In a fully connected layer, every input neuron connects to every output neuron.

• Imagine a small grayscale image of 224 x 224 pixels. This is roughly 50,000 inputs (224 * 224 = 50,176).
• If our first hidden layer has 1,000 neurons, we need 50,000 x 1,000 = 50,000,000 weights just for the first layer. You can just imagine how computationally expensive it is to perform backpropagation and this is just for a small “grayscale” image! But normally, we process RGB images so we are effectively tripling the parameters to 224 x 224 x 3 (RGB channel) x 1,000 neurons (first hidden layer) $\approx$ 150 million.

3. Difficulty to recognize moved objects

While an FNN can still learn to recognized shifted objects if trained on sufficient augmented data. It’s hugely inefficient.

If we train an FNN to recognize a cat in the top-left corner, it learns the specific pattern of pixels in that corner. If we show the model an image with the cat in the bottom-right corner, the FNN will likely fail because it sees a completely new pattern of numbers in that specific “area”.

The Intuition

1. The “Window” (Filter/Kernel)

Imagine we have a small window and instead of looking at the whole image. We are only looking at it through the window.

2. Sliding the Window (Convolution)

Then we slide the window from left to right, top to bottom. In this window, we will detect patterns simple ones like lines and curves, also complex ones like nose, eyes, and more.

So how does CNN addresses FNN’s limitations I listed above?

1. Conservation of Spatial Information

As mentioned above, instead of looking at the whole image, we will look at it through a window. Since we are looking at it through a window, we conserve spatial information. For example, once our window slide in the eyes, it sees something like this.

Compared to looking at the image transformed into a single line, looking at it in a window makes more sense. We can detect the eyes in the example above. This addresses the loss in spatial information.

2. Parameter Sharing

The window (filter/kernel) that slides across the entire image shares its weights. So if we use a 3 x 3 grid. Then we are only using 3 x 3 = 9 weights per filter for the entire grayscale image (for RGB, it would be 3 x 3 x 3 – the last 3 is the RGB channel) . The weights (parameter) are “shared” across the entire image. In practice layers use dozens or hundreds of filters. But consider 128 filters (3×3). We will still just be using 3×3 (filter size) x 128 (# of filters) + 128 (one bias per filter) = 1,280. This drastically reduces the memory needed.

3. Translation Equivariance and Invariance

As mentioned above, we are using the same window (shared parameters) across the entire image. This means that if our window learned to detect a cat in the top-left corner. Even when we use a different image of a cat in the bottom-right corner, since we are using the same window it will still be able to detect the cat.

Convolutional layers provide translation equivariance: If we shift the cat in the input image, the activation in the feature map also shifts. The output changes in sync with the input.

Translation invariance is a property in computer vision and signal processing where an algorithm produces the same output even if the input signal or object is shifted or translated. It ensures that features (like faces or objects) are recognized regardless of their location in an image. Pooling layers are a primary way we achieve translation invariance.

Equivariance = output moves with the input.
Invariance = output stays the same despite movement.

Convolution

Discrete Convolution is a fundamental mathematical operation that combines two sequences ($f \ast g$) to produce a third sequence. It acts as a digital filtering mechanism by calculating a weighted sum, representing the system’s output. It involves flipping one sequence, “sliding” it across the other, multiplying them, and summing the products.

Note:
Strictly mathematically, “convolution” requires flipping the kernel 180°, but in practice CNNs skip this step because the network learns the weights anyway. CNNs technically perform cross-correlation but the term “convolution” is used by convention.

It is easier to see it in action instead. Imagine we have two sequences: (1) is a 5×5 grayscale image. (2) 3×3 kernel.

So to do the convolution, we place the kernel in the top left corner, we perform element-wise multiplication then we take the sum.

Then we slide the kernel to the right.

Then we slide it again until we reach the edge.

After we reach the edge, we slide the kernel down and start again from left to right.

Then we will repeat this process until we slide our kernel all over the image.

The output is what we call Feature Map. The feature map highlights detected features like edges, textures, or shapes. It represents the spatial location of specific features, acting as an internal, condensed representation of the data.

In our example, the image we are using is just a single line. You can actually see it. The “1” represents a white pixel, while “0” represents a black pixel. So you can image that our image.

The +3 values detect the bright-to-dark transition in the image.

Dimensions Formula

Our input image with dimensions 5×5 is transformed to an output of 3×3. One of the most important parts of designing a CNN is calculating how the size of the image changes as it passes through layers. The output size is determined by this formula:

Where:

• $W$: Input Width (224 pixels)
• $K$: Filter/Kernel Size (3 pixels)
• $P$: Padding (Adding zeros around the border to keep the size the same)
• S: Stride (How many pixels you slide at a time. Usually 1).

Let’s use the formula with our example above: We have 5×5 image ($W$ = 5), a 3×3 filter ($K$ = 3), no padding ($P$ = 0), and a stride of 1 ($S$ = 1):

As expected, our output feature map is 3 x 3 matrix.

Padding

You must have noticed that after we perform convolution, our image gets smaller. This is a problem if we have deep network, our image will shrink to 1×1 which doesn’t give any more information. By adding a 1 padding (adding a border of 1px) around the 5×5 image, it becomes 7×7.

And if we perform convolution to a 7×7 image, we will get a 5×5 image, the same dimension of our original image. With this, we can add more layers without making our image too small by adding padding.

Another reason we may want to add padding is to capture the information of the edges. If you visualize the sliding window process, you’ll notice that values in the corners are only used once, as opposed to the center pixels where the window sees them multiple times as it slides. If an important feature, example a cat’s ear, is in the corner, the network might miss it. But if we add a padding, it will push the cat’s ear towards the center which will allow the sliding window to see it multiple times.

Stride

Stride is simply the number of pixels the filter slides across the input image in each step.

In our example, we use Stride = 1, transforming a 5×5 image to 3×3. If we use Stride = 2, it will transform 5×5 to 2×2.

But you may ask, why would we want to skip pixels? Isn’t that losing information? Yes, but sometimes that is exactly what we want. For example, we want to make the image smaller (downsampling), in this case we don’t want to look at specific pixels but look at “concepts” instead.

Another use-case is computation efficiency. If we have a very large image, 4000 x 4000 px, processing every single pixel with Stride = 1 is slow. Using Stride = 2 or 4 allows us to process the image 4x or 16x faster, which is crucial for real-time applications like video processing on phones.

The Activation Function – ReLU

If we only use Convolution layers, no matter how many we stack, our network is mathematically just one big linear operation. This means it can’t learn complex patterns like curves, faces, or cat ears — it can only learn simple linear combinations.

To fix this, after every Convolution operation, we pass the result through an Activation Function. In modern CNNs, the standard choice is ReLU (Rectified Linear Unit).

How ReLU Works

ReLU is surprisingly simple. It looks at every pixel in your feature map and applies this rule:

• If the number is positive, keep it.
• If the number is negative, turn it into 0.

Let’s look back at the output feature map we generated in our Convolution example:

If we pass this matrix through a ReLU layer, it transforms into this:

You might notice that in our convolution example, we had a -3. This represented a “negative edge” (a transition from black to white, instead of white to black).

If we apply ReLU, that -3 becomes 0, and we essentially “erase” that edge detection. Is that a problem?

No, this is a feature, not a bug!

In Deep Learning, we want our filters to be highly specialized.

• This specific filter specializes in finding Left-Side Edges.
• If the network determines that Right-Side Edges are also important, it will learn a second filter specifically for that task (likely with inverted weights).

By forcing non-linearity (ReLU), we allow the network to stack many specialized filters together to understand complex images, rather than trying to make one filter do everything.

Pooling Layers (Downsampling)

After we perform convolution and (usually) apply an activation function like ReLU, we often add a Pooling Layer.

But why do we want to make the feature map smaller? There are two main reasons:

1. Reduces computation. A smaller feature map means fewer numbers to process in the next layer. This makes training faster and uses less memory.
2. Provides Translation Invariance. Earlier, I mentioned that convolution gives us translation equivariance — if the input shifts, the feature map shifts too. But what we often want is translation invariance — the ability to recognize a feature regardless of its exact position. Pooling gives us this. By summarizing a region into a single value, small shifts in the input won’t change the pooled output.

Max Pooling

The most common type of pooling is Max Pooling. It works similarly to convolution: we slide a window (usually $2 \times 2$) over the input, but instead of multiplying and summing, we simply pick the maximum value in that window.

Let’s see it in action. Imagine a $4 \times 4$ feature map. We use a $2 \times 2$ window with a Stride of 2 (meaning the windows don’t overlap).

The Input ($4 \times 4$):

Step 1: Look at the Top-Left $2 \times 2$ block:

Step 2: Slide 2 steps to the right (Top-Right block):

Step 3: Slide down and repeat.

The Output ($2 \times 2$):

We successfully turned a $4 \times 4$ matrix (16 pixels) into a $2 \times 2$ matrix (4 pixels). We reduced the data by 75%, but we kept the “loudest” or “strongest” features (the highest numbers).

Average Pooling

Another type is called Average Pooling, where you take the average of the numbers in the window instead of the maximum.

Average pooling “smooths” out the image. While it was used in early networks, Max Pooling is generally preferred today because it is better at preserving strong edges and textures.

Why Pooling Gives Us Translation Invariance

Let me show this with a quick example. Imagine a filter detected a feature and produced these two different feature maps — the only difference is that the strong activation (value 9) shifted by one pixel:

If we apply 2×2 Max Pooling (with Stride = 2):

Same output! The feature shifted, but because Max Pooling only cares about the maximum value in the region, it doesn’t matter where within that region the feature appeared. This is translation invariance — the network “sees” the feature regardless of small shifts in its position.

How does CNN “learn”?

To understand how a CNN learns, we need to move away from the idea of “memorizing” images and toward the idea of optimizing filters.

A CNN “learning” is simply the process of mathematically tuning the numbers inside those filters (in our example the 3×3 filter) so they become better at detecting features (edges, textures, shapes).

The learning loop is the same as with the standard FNN. You can read my past blog post: Introduction to Neural Networks to familiarize yourself. The difference between CNN and FNN is in FNN, one weight is used once. In a CNN, a single weight in a 3×3 filter is used thousand of times as it slides across the entire image. To find the gradient for a single weight, we have to sum up the gradients from every single position where that filter was applied.

Simulate how a CNN “learns” to find a vertical edge

In our example above (Sliding the Window), we used an vertical line detector filter. But as mentioned above, CNN “learns” the mathematical values of the kernel. In this section, we will “simulate” how a CNN “learns” to find a vertical edge.

The Setup

Note:
In this simulation, we are trying to teach the filter to produce the correct Feature Map BEFORE it passes through the activation function. This allows us to see how the filter learns to handle both positive (vertical edge) and negative (inverted edge) signal values mathematically.

1. The Goal

We want out filter to produce what we calculated earlier.

2. The Input ($X$):

3. The Initial Filter ($W$)

In practice, we initialized the filters with random values. But for this simulation let’s assume we initialized the filter with zeros.

Step 1: The Forward Pass

We convolve our input ($X$) with our filter (which is zero initially). Since our filter is 0 then:

Step 2: Calculate the Error

Let’s use a simple error function for the illustration purposes.

Step 3: Backpropagation

To make our visualization simpler, let’s focus on just one single weight: the Top-Left Weight $w_{0,0}$ of our filter.

Currently, $w_{0,0}$ = 0 and ideally, we want it to be 1.

The Logic:

We need to find out how much $w_{0,0}$ contributed to the error. But we must remember that since the filter slides, $w_{0,0}$ was used 9 times (once for each pixel in the 3×3 output). So we must sum up the error contribution from all 9 positions.

The image above shows the first three slide movements of the filter, and of course we also need to slide it from top to bottom, so $w_{0,0}$ of the filter will move towards, [1, 1, 0, 1, 1, 0, 1, 1, 0].

Let’s look at the first row of the calculation:

1. Output Position (0,0)
   • Error: -3
   • Input Pixel Hit by $w_{0,0}$: $x_{0,0}$ (Value: 1)
   • Contribution: -3 x 1 = -3
2. Output Position (0,1):
   • Error: -3
   • Input Pixel Hit by $w_{0,0}$: $x_{0,1}$ (Value: 1)
   • Contribution: -3 x 1 = -3
3. Output Position (0,2):
   • Error: +3
   • Input Pixel Hit by $w_{0,0}$: $x_{0,2}$ (Value: 0)
   • Contribution: 3 x 0 = 0

If we repeat this for all 3 rows (since they are identical):

• Row 1 Sum: -3 – 3 + 0 = -6
• Row 2 Sum: -6
• Row 3 Sum: -6

Total Gradient for $w_{0,0}$: -18

Step 4: The Descent (Update)

We use a Learning Rate ($\alpha$) to update the weight. Let’s use $\alpha = 0.05$.

The Result

After just one step of training:

• Old Weight: 0
• New Weight: 0.9
• Target Weight: 1.0

The network looked at the error and “learned” that the top-left pixel of the filter needs to be a high positive number to detect vertical edges correctly.

But of course, we need to do the computation above on all the values in the kernel NOT just the single weight $w_{0,0}$ and we will perform the training a lot of times to get the ideal kernel. But hopefully, this single step helps visualize how the “learning” happens.

Full CNN Pipeline

Now that we understand the individual components, let’s look at how they work together in a sequence. A typical CNN is not just one layer; it is a stack of layers that progressively turn raw pixels into abstract concepts.

1. The Feature Extraction Stage

This part of the network consists of repeating blocks of Convolution $\rightarrow$ ReLU $\rightarrow$ Pooling.

• Initial Layers: These layers use filters to find “Low-Level Features” like edges, colors, and simple textures.
• Deep Layers: As the image passes through more layers, the network combines those edges into “High-Level Features” like eyes, tires, or honeycomb patterns.

2. The Flattening Stage

Once the convolutional layers have finished extracting features, we are left with a 3D volume (a stack of feature maps). To make a final prediction, we must flatten this 3D volume into a 1D vector—essentially turning it back into a long list of numbers.

3. The Classification Stage

The flattened numbers are fed into one or more Fully Connected (FNN) Layers.

• In the earlier sections, we said FNNs are bad at processing raw images. However, they are excellent at logical reasoning.
• By this stage, the “input” to the FNN isn’t just random pixels; it is a list of sophisticated features (e.g., “Has whiskers,” “Has pointy ears,” “Has four legs”).
• The final layer uses a Softmax function to give us a probability (e.g., 98% Cat, 2% Dog).