Categorical Cross-Entropy (CCE)

Back to Articles

CCE is a loss function for multi-class classification that measures the difference between a model’s predicted probability and the true distribution, penalizing confident wrong answers heavily.

Quick Review of its Origin

In 1948, a mathematician named Claude Shannon at Bell Labs was trying to solve a problem about sending messages over noisy telephone wires.

He invented a field called Information Theory. In his theory, information isn’t about meaning; it’s about surprise.

  • If I tell you something completely obvious (high probability), I haven’t really given you much information.
  • If I tell you something unexpected (low probability), I’ve given you a lot of information.

Intuition

Imagine a weather model makes two predictions for a sunny desert city:
Prediction A: “It will be sunny tomorrow.” (Probability: 99%)
Prediction B: “It will snow tomorrow.” (Probability: 1%)

If it actually snows, the model is “shocked” since the model is 99% sure it will be sunny. In Information Theory, the shock is quantifiable. We call it Surprise (or Self-Information).

  • Low Probability (1%) happening = High Surprise = High Loss (Bad model)
  • High Probability (99%) happening = Low Surprise = Low Loss (Good model)

To calculate this, Shannon used the negative logarithm.

Notice the shape of the curve:

  • As the probability of the correct answer approaches 1.0 (100%), the loss drops to 0.
  • As the probability approaches 0.0, the loss shoots up to infinity.

Imagine we show the model a picture of a Dog. The model outputs these probabilities:

  • Cat: 0.10
  • Dog: 0.70
  • Bird: 0.20

Since the True label is Dog, we only care about how surprised the model is about “Dog”.

This reveals the clever “trick” of Categorical Cross-Entropy. We technically look at all the classes, but the math is set up so that the wrong classes (Cat and Bird) mathematically vanish.

Here is how it works using One-Hot Encoding.

The Mechanics

We represent the “Truth” (the label) not as the word “Dog”, but as a vector of probabilities where the correct answer is 1 and everything else is 0. We call this a one-hot vector.

$$ y = [\text{Cat}, \text{Dog}, \text{Bird}] $$ $$ y = [0, 1, 0] $$

If we look at the formula for CCE, it sums up the error for every single class:

$$ L = – \sum_{i} y_i \cdot \log(\hat{y}_i) $$
  • $y_i$ is the true probability (0 or 1).
  • $\hat{y}_i$ is the predicted probability.

If we plug in the numbers from our Dog example:

$$ L = – [ (0 \cdot \log(0.1)) + (\mathbf{1} \cdot \log(\mathbf{0.7})) + (0 \cdot \log(0.2)) ] $$

We can see that:

  • 0⋅log⁡(0.1)0 \cdot \log(0.1) becomes 0.
  • 0⋅log⁡(0.2)0 \cdot \log(0.2) becomes 0.
  • Only $\mathbf{1} \cdot \log(\mathbf{0.7})$ remains.

So the complex summation collapses into the simple logic: Take the log of the correct class, and ignore the rest.

Gradient for Backpropagation

CCE is almost always partnered with Softmax and the forward pass looks like this:

Softmax -> CCE

$$ \hat{y}_k = \sigma(z_k) = \frac{e^{z_k}}{\sum_{j=1}e^{z_j}} \to L = – \sum_{j} y_j \cdot \log(\hat{y}_j) $$

We want to find how the Loss ($L$) changes with respect to the input, or “logit” ($z_k$).

$$ \frac{\partial{L}}{\partial{z_k}} = \sum_{j} \frac{\partial{L}}{\partial{\hat{y}_j}} \cdot \frac{\partial{\hat{y}_j}}{\partial{z_k}} $$

Note that we use a summation because changing one raw input $z_k$ changes the denominators of every softmax output $\hat{y}_j$.

Step 1: Derivative of Loss w.r.t Softmax

$$ L = – \sum_{j} y_j \cdot \log(\hat{y}_j) $$ $$ \frac{\partial{L}}{\partial{\hat{y}_j}} = -\frac{y_j}{\hat{y}_j} $$

Step 2: Derivative of Softmax w.r.t Logit

$$ \hat{y}_k = \sigma(z_k) = \frac{e^{z_k}}{\sum_{j} e^{z_j}} $$

Using the Quotient Rule:

$$ (\frac{u}{v})’ = \frac{u’v – uv’}{v^2} $$ $$ v = \sum_{j} e^{z_j} = \sum $$ $$ v’ = e^{z_k} $$

Remember that we have two distinct cases:

Case A: When $k = j$ (The target output)

u=ezk,u′=ezku = e^{z_k}, \quad u’ = e^{z_k}

Case B: When $k \neq j$ (The other outputs)

$$ u = e^{z_j}, \quad u’ = 0 \quad (\text{since } z_j \text{ is constant w.r.t } z_k) $$

So applying the quotient rule:

Case A:

$$ \frac{ e^{z_k} (\sum) – e^{z_k} (e^{z_k}) } { (\sum)^2 } = \hat{y}_k(1 – \hat{y}_k) $$

Case B:

$$ \frac{ 0 (\sum) – e^{z_k} (e^{z_j}) } { (\sum)^2 } = -\hat{y}_k (\hat{y}_j) $$

Step 3: Putting them together

Now we plug Step 1 and Step 2 back into our Chain rule summation. We split the sum into the “target” part ($k = j$) and the “other” parts ($k \neq j$)

$$ \frac{\partial L}{\partial z_k} = \underbrace{ \left( -\frac{y_k}{\hat{y}_k} \right) (\hat{y}_k(1-\hat{y}_k)) }_{\text{Case A: } j=k} + \sum_{j \neq k} \underbrace{ \left( -\frac{y_j}{\hat{y}_j} \right) (-\hat{y}_k \hat{y}_j) }_{\text{Case B: } j \neq k} $$

We can simplify the parts:

Case A:

$$ -\frac{y_k}{\hat{y}_k} \cdot \hat{y}_k(1-\hat{y}_k) = – y_k(1 – \hat{y_k}) = -y_k + y_k\hat{y}_k $$

Case B:

$$ – \frac{y_j}{\hat{y}_j} (- \hat{y}_k \hat{y}_j) = y_j\hat{y}_k $$

Combine them:

$$ – y_k + y_k \hat{y}_k + \sum_{j \neq k}y_j \hat{y}_k $$

Factor out $\hat{y}_k$:

$$ -y_k + \hat{y}_k (y_k + \sum_{j \neq k}y_j) $$

Since yy is a one-hot vector (probability distribution), we know that $(y_k + \sum_{j \neq k}y_j)$ is always going to be 1 because the sum of $y$ values equals 1.

So what we have left is:

$$ \frac{\partial{L}}{\partial{z_k}} = \hat{y}_k – y_k $$

So in its simplest form the gradient is simply Prediction minus Truth.