Why cross-entropy works as a loss
Modern classification models are almost always trained by minimizing cross-entropy loss. The math behind that choice connects information theory, probability, and maximum likelihood estimation. This post walks through those connections: from the information contained in a single event, through entropy and KL divergence, to why cross-entropy is the natural objective for model training.
From surprise to entropy
The starting point is information content, also called "degree of surprise": for an event E with probability p, its information content is -log₂(p). A certain event (p = 1) carries zero information; as p approaches 0, the information grows without bound. The base-2 logarithm gives results in bits.
Concrete examples make this intuitive. Flipping heads on a fair coin has probability 1/2, so its information content is 1 bit. Rolling a 4 on a fair die has probability 1/6, yielding about 2.585 bits—more surprising, as expected.
The logarithm also behaves correctly for combinations of independent events. Flipping heads (1/2) and rolling a 4 (1/6) together have probability 1/12, and their combined information content is precisely the sum of the individual values: -log₂(1/12) = -log₂(1/2) + -log₂(1/6). Bits simply add up.
Given a random variable X with values xⱼ and probabilities pⱼ, entropy is the expected value of information:
H(X) = -Σⱼ pⱼ log₂(pⱼ)
High entropy means high uncertainty; low entropy means the outcome is nearly predictable. A distribution with p₁ = 1 and all other probabilities 0 has entropy 0—observing a sample reveals nothing new. A uniform distribution over 5 outcomes has entropy log₂(5) ≈ 2.32 bits, which matches the number of bits needed to represent 5 equally likely values. Entropy is always non-negative because each pⱼ lies in (0, 1], making every term -pⱼ log₂(pⱼ) ≥ 0. The uniform distribution achieves the maximum entropy; any non-uniform distribution concentrates probability on some outcomes, reducing surprise.
Cross-entropy and KL divergence
Cross-entropy extends entropy to two distributions. For a true distribution P and a predicted distribution Q, it is:
H(P, Q) = -Σⱼ pⱼ log₂(qⱼ)
When P = Q, cross-entropy equals the entropy of P. Information-theoretically, it is the average number of bits required to encode samples from P when the code was designed for Q.
A numeric illustration: with a skewed P and a uniform Q, the cross-entropy is 2.32. If Q is adjusted to be closer to P, cross-entropy drops to 2.16. The more similar the two distributions, the lower the cross-entropy. This makes cross-entropy a plausible loss function—but it has a drawback as a pure measure of dissimilarity: since H(P, P) = H(P) is generally not zero, cross-entropy always includes the intrinsic uncertainty of P.
KL divergence removes that baseline:
D_KL(P ‖ Q) = H(P, Q) - H(P) = Σⱼ pⱼ log₂(pⱼ / qⱼ)
This gives a measure that is zero when P = Q and positive otherwise. KL divergence is not a true distance metric because it is not symmetric: D_KL(P ‖ Q) ≠ D_KL(Q ‖ P) in general.
Why cross-entropy works in machine learning
In training, P is the empirical distribution of the training data and Q is the model's predicted distribution. Cross-entropy provides a single non-negative scalar that decreases as the model's predictions approach the data distribution. Crucially, the decomposition via KL divergence shows that H(P) is a constant with respect to the model, so minimizing cross-entropy is equivalent to minimizing KL divergence—the model's parameters only affect the KL term.
The same objective emerges from maximum likelihood estimation. Given a parameterized model Q_θ and samples drawn from P, the likelihood of the samples is the product of their predicted probabilities. Maximizing the log-likelihood (converting the product to a sum, which preserves the maximum) and then multiplying by the constant 1/n gives an average over the samples. By the law of large numbers, for large n that average converges to the expected value over P. Negating the result turns maximization into minimization of the cross-entropy between P and Q_θ. Thus, maximum likelihood estimation and cross-entropy minimization are equivalent objectives.
In practice, cross-entropy is used directly as the loss for logistic regression and softmax-based multiclass classification. KL divergence also appears on its own, for instance in the evidence lower bound used for variational autoencoders—but in most supervised training, cross-entropy and KL divergence lead to the same optimization target.
