Introduction to Neural Networks

Our goals in this blog post are:

1. See the limitations of a single linear classifier (linear regression, logistic regression).
2. Create a Neural Network from scratch.
3. See how the neural network solves the limitations of a single linear classifier.
   You can get the python code used in this post in the Google Colab notebook here.

Limitation of a single Linear Classifier

Imagine you are a guard that ONLY allows entry of a person EITHER wearing a Blue or has a ticket. You are also strictly asked NOT TO ALLOW people with both a Blue shirt AND a ticket.

We can convert the data points to something like this. We translate the shirt color and has ticket to a vector for a data point $x_n$ [shirt color, has ticket]. Where we set Blue shirt = 1, Red shirt = 0, Has ticket = 1, and No ticket = 0.

This is the XOR problem (Exclusive OR).

Here’s a visualization of our data.

Now if we create a logistic regression model to solve this XOR problem, the logistic regression model will have this decision boundary.

Here’s a 3D visualization

We can see that our logistic regression model does not do well simply because it is not possible to create a single straight line that will separate the categories: Allow Entry (1) and Not Allow Entry (2). This shows the limitations of a linear classifier, it can’t solve complex problems.

Feed Forward Network (Neural Networks)

Now imagine if we can fold the paper with the XOR dots above, then the dots will line up in a way that a single straight cut can separate them.

Activation Function

The fold is applied using an Activation Function. One of the key roles of an activation function is to introduce non-linearity.

There are several activation functions but for this example, we will use the Sigmoid function.

The Sigmoid function squashes an input number ($z$) into the range (0,1).

Here’s a visualization of how the Sigmoid function transform a dataset.

# Define the functions
def sigmoid(z):
	return 1 / (1 + np.exp(-z))

# Create a range of input values (z) from -10 to 10
test_values = np.linspace(-10, 10, 100)

a_sigmoid = sigmoid(test_values)

# Plot Sigmoid
plt.plot(test_values, a_sigmoid, color='blue', lw=3)
plt.title("Sigmoid Activation $\sigma(z)$")
plt.grid(True)
plt.axhline(0.5, color='red', linestyle='--', alpha=0.3) # Center line

plt.show()

Forward Propagation (Forward Pass)

We can think of a Forward Propagation as an Assembly Line in a factory.

1. Raw Materials (Input): We start with steel and glass.
2. Station 1 (Hidden Layer 1): Workers melt and mold the steel. The material changes shape.
3. Station 2 (Hidden Layer 2): Workers paint and assemble the parts. The material looks different again.
4. Final Product (Output): A car rolls off the line

We call this “Forward Propagation” (Forward Pass) because the data flows in one direction: from input to output.

Mathematically, this is the forward pass flow.

1. $X$: input
2. $Z = W \cdot X + b$: Linear model
3. $A = g(Z)$: Activation Step
4. $\hat{y}$: Output prediction

A single layer in a neural network is defined by a two-step process: Linear Transformation and Activation Step

The Forward Pass is the process of the model to make a prediction. It takes the input, pushes it through matrices, and outputs a guess. It does not learn in this process.

Here’s an example of the matrix math to help us visualize. For example, we have two features and we want to output 3 neuron.

Let’s create a Feed Forward model and see how it does without learning (training).

import numpy as np

class FeedForward:
	def __init__(self):
		np.random.seed(42) # For reproducibility
		# Initialize the Weights and Biases
		# Layer 1 (Hidden): 2 features -> 3 hidden neuron
		self.W1 = np.random.randn(3, 2)
		self.b1 = np.zeros((3, 1))
		
		# Layer 2 (Output): 3 hidden neuron -> 1 output neuron
		self.W2 = np.random.randn(1, 3)
		self.b2 = np.zeros((1, 1))
	
	def sigmoid(self, z):
		return 1 / (1 + np.exp(-z))
	
	def forward(self, x):
		# Layer 1
		self.Z1 = np.dot(self.W1, x) + self.b1 # Linear Model
		self.A1 = self.sigmoid(self.Z1) # Activation Step
		
		# Layer 2 (Output)
		self.Z2 = np.dot(self.W2, self.A1) + self.b2
		self.A2 = self.sigmoid(self.Z2)
	
	return self.A2

model = FeedForward()
prediction = model.forward(X.T)

print("Predictions:")
print(f"Guest 1 (Blue, No Ticket) -> Target: 1 | Pred: {prediction[0][0]:.4f}")
print(f"Guest 2 (Red, No Ticket) -> Target: 0 | Pred: {prediction[0][1]:.4f}")
print(f"Guest 3 (Blue, Ticket) -> Target: 0 | Pred: {prediction[0][2]:.4f}")
print(f"Guest 4 (Red, Ticket) -> Target: 1 | Pred: {prediction[0][3]:.4f}")

This is the result of the prediction of our Feed Forward model without any training.

A few observations that we need to point out.

1. We can see that the prediction is way off. This is because as mentioned above, we only did a forward pass. The model wasn’t trained (learned) just yet.
2. You can also notice the number of hidden neurons. There are no “perfect” number of hidden neurons. This is what we call Hyperparameter. We fine tune it depending on the problem we want our model to solve.

Now let’s proceed with the process of learning / training.

Backpropagation (backward pass)

We can only improve what we can measure.

Before we even start to train, we need a way to measure how well/wrong our model is doing, and improve it from there. Otherwise we are just randomly guessing.

Loss/Cost Function

First off, the Loss and Cost is often used interchangeably. The difference between them is, we call it Loss if we are talking about measuring the error in a single sample (data point). While we call it Cost if we are talking about measuring the error of the whole dataset.

The Cost Function is mathematical equation that quantifies how “wrong” our model’s predictions are. Then we can use the “cost” to know how to adjust our models.

Binary Cross Entropy (BCE)

In our particular example, we will be using the Binary Cross Entropy (BCE) function as our cost function. I’ll go deeper into what BCE and how it came to be in another blog post, but know that BCE is the most commonly used cost function for classification problems, in our case whether we want to permit entry (0 or 1).

Here’s the formula where $J$ denotes the BCE cost function.

y: target
A: probability (result from our activation function)

Our goal is to Minimize the Cost function (minimize how wrong the model’s predictions are).

Basically, back propagation is looking at the cost function $J$ then trace backward to find which variables we need to adjust and how much we need to adjust them to minimize the cost function.

Let’s trace it one at a time. The cost function is

We can see here that to minimize $J$, we need to adjust $A$. So we backtrack again how we got $A$. Remember that $A$ is the output of the sigmoid function (activation).

Looking at the sigmoid function, we need to adjust the value of $z$ to adjust the value of $A$. So we backtrack again to where we get $z$.

Remember that $z$ is the output of the linear transformation.

And lastly, looking at the linear transformation, we know that the value of $z$ depends on $w$ and $b$. So in simple terms, we need to get the value of $w$ and $b$ that minimizes the cost function.

To be precise, to minimize the cost function, we need to calculate the Gradient of $J$ with respect to $w$ and $b$.

Calculating the Gradients

Gradients are just the derivatives. To calculate them we use chain rule

Writing them as a whole:

Gradient Descent

Now that we know the gradients, we can do Gradient Descent. To help visualize what we are doing here, imagine that we are standing on a mountain (the mountain is the cost function).

Our goal is to descend the mountain (minimize the cost function). The gradients we compute above tells us the slope (which way is going up) from our current position. So to descend the mountain, we simply perform a step to the opposite of the slope. We do this on each step until we reach the bottom of the mountain.

Mathematically, we do it like so:

What we’re just doing here is, from our old position $W_{old}$, look at where the slope $\frac{\partial{J}}{\partial{w}}$ is, then take a step $\alpha$ in the opposite direction (that’s why we use the negative –). Then we’ll do this over and over until we reach the bottom (ideally) but in practice, we will most likely settle to the lowest possible point.

Full Feed Forward Model

Now let’s include the training part (back propagation) to our model.

import numpy as np

class FeedForward:
	def __init__(self):
		np.random.seed(42) # For reproducibility
		# Initialize the Weights and Biases
		# Layer 1 (Hidden): 2 features -> 3 hidden neuron
		self.W1 = np.random.randn(3, 2)
		self.b1 = np.zeros((3, 1))
		
		# Layer 2 (Output): 3 hidden neuron -> 1 output neuron
		self.W2 = np.random.randn(1, 3)
		self.b2 = np.zeros((1, 1))

	def sigmoid(self, z):
		return 1 / (1 + np.exp(-z))
	
	# We need this for training.
	def sigmoid_derivative(self, a):
		return a * (1 - a)
	
	def forward(self, x):
		# Layer 1 (Hidden Layer)
		self.Z1 = np.dot(self.W1, x) + self.b1 # Linear Model
		self.A1 = self.sigmoid(self.Z1) # Activation Step
		
		# Layer 2 (Output Layer)
		self.Z2 = np.dot(self.W2, self.A1) + self.b2
		self.A2 = self.sigmoid(self.Z2)
		
		return self.A2
		
	def train(self, X, y, learning_rate=0.1, epochs=100_000):
		# Transpose to match the matrix.
		X_train = X.T
		y_train = y.T
		m = X.shape[0] # Number of training samples (guests)
		
		# We will use this later.
		epsilon = 1e-15
		
		for i in range(epochs):
			# 1. Forward Pass
			# We get our initial position in the mountain
			self.forward(X_train)
			
			# 2. Backprop
			# We start at Layer 2 (Output)
			dZ2 = self.A2 - y_train # (p - y)
			# We get the mean value hence the (1/m)
			dW2 = (1 / m) * np.dot(dZ2, self.A1.T) # x · (p - y) but since we are at Layer 2, our x here is really the output from Layer 1.
			# We are using `np.sum()` here even when our derivation early is supposed to be (p - y) because we are calculating the bias b by batch (all of the guests) simultaneously
			db2 = (1 / m) * np.sum(dZ2, axis=1, keepdims=True)

			# Then we go back to Layer 1
			dA1 = np.dot(self.W2.T, dZ2)
			
			# At Layer 2 above, we have a target "y". But going back here in Layer 1 (hidden layer), we don't have a direct target "y" to compare against.
			# Instead, we have to carry the error backwards through the weights of the next layer. This is why it's called Backpropagation.
			dZ1 = dA1 * self.sigmoid_derivative(self.A1)
			
			dW1 = (1/m) * np.dot(dZ1, X_train.T)
			db1 = (1/m) * np.sum(dZ1, axis=1, keepdims=True)
			
			# Now that we know the gradients, we can take a descending step.
			# The "learning_rate" is how "big" of a step we do.
			self.W2 -= learning_rate * dW2
			self.b2 -= learning_rate * db2
			self.W1 -= learning_rate * dW1
			self.b1 -= learning_rate * db1
			
			# It is a good idea to know if we are steadily descending every X number of steps.
			# And again, we are able to know this using the cost function.
			if i % 10000 == 0:
				# We added epsilon for safety. This is to prevent `np.log(0)` or `np.log(1)`
				cost = -np.mean(y_train * np.log(self.A2 + epsilon) + (1-y_train) * np.log(1-self.A2 + epsilon))
				print(f"Epoch {i}: Cost {cost:.4f}")
			
model = FeedForward()
# Now we train the model.
model.train(X, y)

# Then we do the prediction
prediction = model.forward(X.T)

print("Predictions:")
print(f"Guest 1 (Blue, No Ticket) -> Target: 1 | Pred: {prediction[0][0]:.4f}")
print(f"Guest 2 (Red, No Ticket) -> Target: 0 | Pred: {prediction[0][1]:.4f}")
print(f"Guest 3 (Blue, Ticket) -> Target: 0 | Pred: {prediction[0][2]:.4f}")
print(f"Guest 4 (Red, Ticket) -> Target: 1 | Pred: {prediction[0][3]:.4f}")

Here’s the predictions of our model after training.

We can see that our prediction is 99% correct. Here’s how the Feed Forward Network model decision boundary looks like.

Here’s the 3D visualization

Unlike the simple linear classifier model, a neural network can create complex decision boundaries.

What’s next?

There are a lot more you can go deeper into that I haven’t discussed thoroughly.

1. Study the other activation functions (ReLU, TanH, Softmax) and understand when to use them.
2. Where do Binary Cross Entropy came from?

If you are interested, I also written about Word2Vec which is an application of a neural network.