Writing Fraud-Detection using RNN from Scratch

Recommended read:

• Introduction to Recurrent Neural Network (RNN)
• Colab Notebook for this Blog Post

Our goal in this blog post is to build a solid understanding of how an RNN with Many-to-Many (Synced) architecture works and to help us achieve this, we will write a fraud-detection model from scratch.

Note:

We will be using the Synthetic Financial Datasets For Fraud Detection dataset.

Sequence Engineering

Standard machine learning models evaluate data points in isolation, like viewing individual photos. An RNN, however, watches a ‘movie’. To detect fraud, the model needs to know not just what the current transaction looks like, but what the user’s history looks like. A USD 2,000 purchase might be normal for a high-spender but very suspicious for someone who usually only spends USD 20.

In a raw dataset, transactions are usually stored as flat rows in a table:

Here’s more details about the dataset:

• step: Maps a unit of time in the real world. In this case 1 step is 1 hour of time.
• type: CASH-IN, CASH-OUT, DEBIT, PAYMENT and TRANSFER
• amount: amount of the transaction in local currency
• nameOrig: customer who started the transaction
• oldbalanceOrg: initial balance before the transaction
• newbalanceOrg: customer’s balance after the transaction.
• nameDest: recipient ID of the transaction.
• isFraud: identifies a fraudulent transaction (1) and non fraudulent (0)

To work with this dataset, we must transform these rows into sequences. We group the transactions by nameDest (customer) and sort them by step (time). Instead of feeding the model one row at a time, we feed it a sequence/”window” of transactions (for example, last 10 transactions) so it can learn the pattern of behavior.

For example, we have this dataset.

To detect that fraud at step = 3, the RNN needs to look at User_A’s history as one continuous sequence ($9,839.64 -> $181.0 -> $60,000).

One challenge with this is that arrays need to have a “fixed” shape. However, in the real world, “User A” might have 3 transactions while “User B” only has 1. Let’s say we want our RNN to always look at a window of 3 transactions to make a prediction. If “User B” only has 1 transaction then we have 2 “empty slots” in our sequence of 3. We fill these empty slots with a “dummy” value, usually just zeros, to make the sequence reach our target length. So for “User B”, the sequence becomes: [Transaction_1, 0, 0]. We call this process/trick, Padding.

However, we don’t want the model to think zeros are actual transaction, as this will make our model “learn” incorrectly. To solve this, we create a second array of the same length called a Mask.

A mask is simply an array of 1s (True) and 0s (False) that tells the model which transactions are “real” and which ones are just padding. In our example, User B’s mask would be [1, 0, 0].

Many-to-Many (Synced) Architecture

As we already discussed in Introduction to RNN, there are different types of architecture depending on the input and output we want. The architecture of our real-time fraud detector is a Many-to-Many (Synced) RNN.

Here is exactly why:

• Many inputs: We feed the network a sequence of transactions (e.g. Transaction 1, Transaction 2, Transaction 3, etc.).
• Many outputs: We want a fraud prediction y_t for every single transaction as soon as it happens.

In fraud detection, a bank cannot afford to wait until a user makes 10 purchases to decide if the 3rd one was fraudulent. The model needs to evaluate the risk in real time, at every step.

Let’s look at the flow for a 3-step sequence:

1. Step 1: Input $x_1$ (Coffee shop) -> Updates memory to h_1 -> Outputs prediction $y_1$ (0% fraud risk).
2. Step 2: Input $x_2$ (Gas station) -> Updates memory to h_2 -> Outputs prediction $y_2$ (1% fraud risk).
3. Step 3: Input $x_3$ (Expensive jewelry store) -> Updates memory to h_3 -> Outputs prediction $y_3$ (95% fraud risk!).

Hidden State as Memory

The hidden state, shown as yellow boxes above, is simply a vector of numbers that the RNN updates at every step. It is called “hidden” because it sits in the middle of the network, between the inputs we feed in and the predictions we see.

We can think of the hidden state like a “running summary”. When we read a novel, we don’t memorize every single word from page 1, page 2, and page 3. Instead as we read, we keep a mental summary of the plot. When we read page 4, we “combine” the new words on that page with the mental summary that we already have. Then we “update” our mental summary with the changes (e.g. the protagonist gains a new power).

The Math Behind the Memory

In our image above, we used $a^{<t>}$ (activation). Some other materials use $h$ (hidden) as the annotation. But they are the same thing.

where

• $W_{xh}$: weights from the current time step $t$.
• $x_t$: input at the specific time step $t$.
• $W_{hh}$: weights from the past memory. This decides how much of that past matters.
• $h_\text{prev}$: hidden state (memory) from the last time step $t-1$.
• $b_h$: bias.

So the equation is really combining two different sources of information to create a new memory $h_t$:

1. The Present: $W_{xh} \cdot x_t$ processes what is happening right now.
2. The Past: $W_{hh} \cdot h_\text{prev}$ processes what happened in the past.

If a value in $W_{hh}$ is close to 0, the network is choosing to forget that part of the past. If it’s a larger number, it’s choosing to remember it. The network learns these weights over time through training.

To help us have better intuition, imagine a user makes a transaction in New York at 12:00 PM. That location + time information is packed into $h_1$ (hidden state / memory 1). Then at 12:05 PM, the user attempts a transaction ($x_2$) in Paris.

When our RNN calculates $h_2$, it combines the Paris input $x_2$ with the New York memory $h_1$. Because the network has learned that a human cannot fly across the ocean in 5 minutes, combining these two pieces will cause the numbers inside $h_2$ to spike wildly in a specific pattern. This spiky pattern is the mathematical signature of a red flag/fraud.

Weights

In our Many-to-Many (Synced) fraud detector, the network needs to do two things at every single time step $t$:

1. Update the memory $h_t$: Combine the new transaction with the old memory.
2. Make a prediction $y_t$: Look at the updated memory and calculate the probability of fraud (0 or 1).

Let’s take a closer look of the architecture.

Again this is the equation for the hidden state:

And for the prediction, we use this equation:

Where:

• $W_{hy}$ are the weights that connect the hidden state $h_t$ to the prediction $y$.
• $\sigma$ denotes that we will pass the value inside to the Sigmoid activation function. The Sigmoid function takes any raw number and squishes it to a number between 0 and 1, which is perfect for our fraud detector because we need the output to be a probability (i.e. an output of 0.595 can be read as 59.5% risk of fraud).

Before calculating the forward pass, let’s define the network’s weights and their roles.

Input Weights $W_{xh}$

The input weights process the brand-new data coming into the network right now (current transaction). They take raw features like Amount and Balance Remaining and translate them into a latent representation.

How its size is determined: It must match the number of Input Features on one side and our Hidden Memory Size on the other: (Hidden Size, Input Features).

Memory Weights $W_{hh}$

The memory weights are the core of the “recurrent loop” (we’ll discuss this more later). Its job is to take the memory vector from the previous transaction h_{t-1} and transition it into the present step. It decides how much of the past historical context should be remembered or forgotten.

How its size is determined: Because it connects a hidden state to another hidden state, its size depends entirely on our Hidden Memory Size. It is always a square matrix: (Hidden Size, Hidden Size).

Prediction/Output Weights $W_{hy}$

These weights don’t care about the raw inputs directly. It only looks at the hidden state (after the present and past have been combined). Its job is to translate the hidden state h_t into a prediction/output.

How its size is determined: It connects our Hidden Memory Size to our Output Size: (Output Size, Hidden Size).

Hidden Memory Size

The Hidden Memory Size, to put it simply, is the capacity of the network’s internal brain. It is what we call a Hyperparameter, it is a setting we choose manually before the model ever starts learning.

Setting a Low Value

When the Hidden Memory Size is low, the network is forced to be ruthless. It can only remember the most massive, glaringly obvious patterns (like “Balance is zero”).

The Good: It runs incredibly fast and forces the model to ignore useless, random details.

The Bad: It doesn’t have enough room to remember complex things. It might easily miss a subtle, multi-step fraud scheme because it ran out of memory space to connect the dots from earlier transactions.

Setting a High Value

When the Hidden Memory Size is high then the network has a massive capacity to remember highly detailed, complex patterns stretching far back into the past.

The Good: It can catch not only simple but also incredibly sophisticated fraud patterns that rely on tiny, subtle clues spread out over weeks of transactions.

The Bad: It takes much longer for the computer to calculate. More importantly, it can suffer from “memorizing noise”. Because it has so much capacity, it might start memorizing useless things. For example, it might memorize the user’s specific coffee order instead of actually learning the general rules of fraud.

So finding the right size is a matter of balancing. Usually, we want to test a few different sizes to find the “sweet spot” where the model is smart enough to catch the fraud, but strict enough not to get distracted by random details.

Forward Pass

Now let’s look at what happens during the forward pass. Let’s consider this dataset.

Let’s convert these raw rows into our Input Vectors x_t using Min-Max normalization. To make the math easy to read, we will only have 2 features on each of our transaction: [Amount, Balance Remaining]:

Min-Max Normalization

Min-Max Normalization is a simple way to shrink large numbers so they fit perfectly between 0 and 1. This helps our neural network learn faster and stay stable.

Normalization of the Amount:

Minimum: $0
Maximum: $4500

• The $100 Transaction:

• The $200 Transaction:

• The $4500 Transaction:

Normalization of the Balance Remaining:

Minimum: $0 (An account that has been completely emptied)
Maximum: $5000 (The highest balance starting point in our sample data)

• The $4900 balance:

• The $4700 balance:

• The $200 balance:

This will be our input vectors:

As we saw above, we have two equations:

Initialize the Weights

For our first forward pass, our weights $W_{xh}$, $W_{hh}$, and $W_{hy}$ are initialized with random values. For this example, let’s set our Hidden Memory Size = 2.

Let’s also set our biases to zero vectors to make the math easier to read and follow.

Time Step t = 1 (Coffee shop)

First we combine the present input and the past memory. Since this is the first step, we don’t have any past at this point. Hence:

Then we calculate the New Memory State

Then we calculate the Fraud Risk

Then we will just repeat this process to all of the Time Steps.

Time Step t = 2 (Electronics Store)

Time Step t = 3 (Unknown Offshore Account)

Here’s what our Fraud Detector saw on all the transactions:

• Coffee Shop t = 1: 43.4% risk
• Electronics Store t = 2: 43.5% risk
• Unknown Offshore Account t = 3: 59.5% risk

Our Fraud Detector guessed 59.5% fraud risk on Step 3, but the actual dataset labeled Step 3 as 100% fraud. So we have to correct our model’s weights to maximize this.

Backpropagation

Let’s review the full forward pass (it helps me perform backprop when I can see full forward pass).

1. Measure the Mistake (The Loss)

We use Binary Cross-Entropy Loss (BCE) to measure the error of our final prediction.

• True Label (Target): 1.0 (100% Fraud)
• Our Prediction ($y_3$): 0.595 (59.5% Fraud)

2. Assign Blame

Our goal is to minimize the Total Loss ($L_3$). We can do this by updating the values of our weights. We need to find how a change in any weights affects $L_3$.

Prediction Weights $W_{hy}$

The chain rule flows from the Loss back to the output linear layer:

The first part of the formula is Binary Cross-Entropy Loss and because we use Sigmoid as our activation function, almost everything cleanly cancels out.

We transpose $h^t_3$ so we can do Matrix Multiplication.

Input Weights $W_{xh}$

Remember the $h_3$ was built on past memories ($h_1$, and $h_2$), so $L_3$ places a fraction of the blame on Step 3 ($t = 3$), a fraction on Step 2 ($t = 2$), and a fraction on Step 1 ($t = 1$).

The chain rule flows from the Loss back to where the input weights are used in Step $t = 3$:

We already solved the first 2 terms, let’s solve for the rest:

Now let’s plug in the values:

Now, we finish the chain by multiplying by our input $x_3$ (transposed, so the matrix math aligns):

Then we need to compute for:

But if you remember above, we set:

So we can rewrite it to:

From the equation above, we can set:

Then we get:

Finally, we add them to get the final:

Memory Weights W_{hh}

Similarly to how we compute for $\frac{\partial L_3}{\partial W_{xh}}$, we use this equation:

So we first get

or simply

Then similarly, we compute for

Backpropagation Through Time (BPTT)

What we did above, is what we call Backpropagation Through Time (BPTT)

Remember that our prediction at Step $t = 3$ relied heavily on $h_2$, which in turn relied on $h_1$. That means that the weights we used in Step $t = 1$ and Step $t = 2$ are partly to blame for the error we made at Step $t = 3$.

To fix this, we have to push the error backwards through time. We calculate how much $h_2$ contributed to the mistake, and use that to find the gradients for Step $t = 2$. Then, we push it back to $h_1$ to find the gradients for Step $t = 1$.

Once we calculate the gradients for the input and memory weights across all three time steps, we simply add them together. This combined sum gives us the complete picture of how to adjust our weights for the entire sequence.

3. Sequence Loss Accumulation

Now we want our model to learn from every single transaction, not just the last one. When we train a Many-to-Many network, we want to minimize the Total Sequence Loss $L_\text{total}$.

We detailed how to compute $\frac{\partial{L_3}}{\partial{W}}$ above. We must repeat this for $L_1$ and $L_2$. Below is a quick rundown of the equations for $L_2$; the rest follow the exact same pattern.

After we got all of them, we will just combine all of them.

Updating the Weights (Gradient Descent)

Now that we know how much each weight contributed to the error, we can now correct them. We do this by subtracting the gradient from our original weights.

If we apply this to our input weights:

$\alpha$ is what we call learning rate, we set it to a low value to make our correction small and gradual and not massive and unpredictable.

With this our RNN model has official “learned”.

The Training Loop (Epoch)

What we did above is one full complete cycle:

1. Forward Pass: Make a prediction.
2. Measure the Mistake
3. Backpropagation: Assign blame.
4. Update Weights: Learn from the mistake.

Of course doing this one time is not enough. To make our model highly accurate, we need to show it our dataset thousands of times. Each full pass through the entire dataset is called an Epoch. With every epoch, the Total Loss goes down, and the model’s ability to catch fraud goes up.

Python Implementation

import numpy as np

def sigmoid(x):
  x = np.clip(x, -500, 500)
  return 1 / (1 + np.exp(-x))

class FraudRNN:

  def __init__(self, input_size, hidden_size, output_size=1):
    self.hidden_size = hidden_size

    # Initialize weights with random values. We scaled it down to stay stable.
    # W_xh: (Hidden size, Input Features)
    self.W_xh = np.random.randn(hidden_size, input_size) * 0.1

    # W_hh: (Hidden size, Hidden size)
    self.W_hh = np.random.randn(hidden_size, hidden_size) * 0.1

    # W_hy: (Output size, Hidden size)
    self.W_hy = np.random.randn(output_size, hidden_size) * 0.1

    # Bias for hidden state: (Hidden size, 1)
    self.b_h = np.zeros((hidden_size, 1))

    # Output bias. This is not in the blog post to make the blog post math simpler, but we will add it in our model.
    self.b_y = np.zeros((output_size, 1))

  def forward(self, X):
    X_t = np.transpose(X, (1, 2, 0))
    time_steps, features, batch_size = X_t.shape

    self.h = {-1: np.zeros((self.hidden_size, batch_size))}
    self.y = {}
    self.x = {}

    for t in range(time_steps):
      self.x[t] = X_t[t]

      a_t = np.dot(self.W_xh, self.x[t]) + np.dot(self.W_hh, self.h[t - 1]) + self.b_h
      self.h[t] = np.tanh(a_t)

      # Compute for the prediction
      self.y[t] = sigmoid(np.dot(self.W_hy, self.h[t]) + self.b_y)

    return self.y, self.h

  def backward(self, Y_true, masks):
      Y_t = np.transpose(Y_true, (1, 2, 0))
      masks_t = np.transpose(masks, (1, 0))
      masks_t = masks_t.reshape(masks_t.shape[0], 1, masks_t.shape[1])

      time_steps = len(self.x)

      dW_xh = np.zeros_like(self.W_xh)
      dW_hh = np.zeros_like(self.W_hh)
      dW_hy = np.zeros_like(self.W_hy)
      db_h  = np.zeros_like(self.b_h)
      db_y  = np.zeros_like(self.b_y)

      dh_next = np.zeros_like(self.h[0])
      total_loss = 0

      for t in reversed(range(time_steps)):
        # 1. Measure Mistake (Masked)
        dy_t = self.y[t] - Y_t[t]
        dy_t = dy_t * masks_t[t]

        loss_t = - (Y_t[t] * np.log(self.y[t] + 1e-9) + (1 - Y_t[t]) * np.log(1 - self.y[t] + 1e-9))
        total_loss += np.sum(loss_t * masks_t[t])

        # 2. Assign Blame to Output Weights
        dW_hy += np.dot(dy_t, self.h[t].T)

        # We sum across the batch (axis=1) because a bias is just a single number added to everyone
        db_y += np.sum(dy_t, axis=1, keepdims=True)

        # 3. Assign Blame to Hidden State
        dh_t = np.dot(self.W_hy.T, dy_t) + dh_next
        da_t = dh_t * (1 - self.h[t]**2)

        # 4. Assign Blame to Input Weights and Hidden Bias
        dW_xh += np.dot(da_t, self.x[t].T)
        dW_hh += np.dot(da_t, self.h[t-1].T)
        db_h  += np.sum(da_t, axis=1, keepdims=True)

        # 5. Pass blame backward
        dh_next = np.dot(self.W_hh.T, da_t)

      return dW_xh, dW_hh, dW_hy, db_h, db_y, total_loss

  def update_weights(self, dW_xh, dW_hh, dW_hy, db_h, db_y, learning_rate, valid_steps):
      self.W_xh -= learning_rate * (dW_xh / valid_steps)
      self.W_hh -= learning_rate * (dW_hh / valid_steps)
      self.W_hy -= learning_rate * (dW_hy / valid_steps)
      self.b_h  -= learning_rate * (db_h / valid_steps)
      self.b_y  -= learning_rate * (db_y / valid_steps)


# --- Training Loop ---
def train_rnn(X, y, masks, epochs=100, batch_size=64, learning_rate=0.01, hidden_size=32):
  samples, time_steps, features = X.shape
  rnn = FraudRNN(input_size=features, hidden_size=hidden_size, output_size=1)

  for epoch in range(epochs):
      epoch_loss = 0
      total_valid_steps = 0

      indices = np.random.permutation(samples)
      X_shuffled = X[indices]
      y_shuffled = y[indices]
      masks_shuffled = masks[indices]

      for i in range(0, samples, batch_size):
          X_batch = X_shuffled[i:i+batch_size]
          y_batch = y_shuffled[i:i+batch_size]
          mask_batch = masks_shuffled[i:i+batch_size]

          rnn.forward(X_batch)

          dW_xh, dW_hh, dW_hy, db_h, db_y, batch_loss = rnn.backward(y_batch, mask_batch)

          valid_steps_in_batch = np.sum(mask_batch)

          if valid_steps_in_batch > 0:
              rnn.update_weights(dW_xh, dW_hh, dW_hy, db_h, db_y, learning_rate, valid_steps_in_batch)

          epoch_loss += batch_loss
          total_valid_steps += valid_steps_in_batch

      print(f"Epoch {epoch+1}/{epochs} | Average BCE Loss: {epoch_loss / total_valid_steps:.4f}")

  return rnn