Understanding the principles

Information Theory and Entropy: From Probability to Coding Cost

First distinguish the true distribution P from the model distribution Q, then derive cross-entropy, KL divergence, and perplexity step by step from self-information and entropy.

Suggested 30–40 minutes · Intermediate · Requires: basic concepts of probability, weighted average, and logarithms

Core idea Information theory converts “how much probability the model gives to real events” into an additive coding cost: entropy is the minimum average cost when the true distribution is known, cross-entropy is the average cost after switching to the model distribution, and KL is the difference between the two. These quantities measure only probability matching; they do not directly represent factuality, semantic value, or general intelligence.

After reading this page, you should be able to answer these yourself:

What P and Q are—what roles the true distribution and the model distribution each play.
Self-information and entropy—the surprise of a single event, and the average uncertainty of the entire distribution.
Cross-entropy and KL—how much extra cost you pay on average when you use the wrong distribution.
Perplexity—exponentiating abstract average loss into an equivalent number of choices on observed data.
The boundaries of these quantities—they measure probability match, not factuality, semantics, or intelligence.

The minimal example used throughout this page Predict weather: the true weather follows P=(sunny 1/2, rain 1/4, snow 1/4) appears, but the model gives a different prediction Q=(sunny 1/2, rain 3/8, snow 1/8). If we predict or encode according to Q, how much extra do we pay on average? The whole page revolves around this three-weather example.

1First clarify P, Q, logarithms, and expectationStarting Point

What objects do all the formulas that follow actually compute on?

Suppose an experiment may produce finitely many events, for example the weather set Ω={sunny, rain, snow}.P(x) represents the probability that event x actually occurs—in the long run, 25% of days will snow.Q(x) is the probability that the model assigns to event x—for example the model thinks only 12.5% of days will snow. Teaching examples give P directly, but in reality P is usually unknown and can only be approximated by sample frequencies. Q is given by the model.

All subsequent calculations follow one rule: events still occur according to P's frequencies, but the loss or code length is determined by Q, and then a weighted average is taken using P. This is “taking the expectation under P”—E_P[f(x)] = Σ_x P(x) f(x)—using the probability of actual occurrence as weights.

Why use logarithms?　We want rarer events to bring more information; also, when two independent events occur together, the probabilities multiply and the information amounts should add. Logarithms turn multiplication into addition: log₂(ab) = log₂a + log₂b. Probabilities are no more than 1, logarithms are no greater than 0, and adding a minus sign gives non-negative information. This page uses log₂ as the base, and the unit is bit. 1 bit represents the information provided by one ideal balanced binary distinction. If you use ln instead, the unit becomes nat, and the values only scale by a fixed factor.

An easily overlooked premise:P is usually unknown in reality; we can only approximate it using sample frequencies. Samples may be biased, and frequency is not equal to true probability. When you see P, first ask “according to what do events actually occur?”; when you see Q, first ask “what probability does the model give to this true event?”

2Self-information: How surprising is a single outcome?Self-information

Why does an event with smaller probability, once it occurs, bring more information?

I_P(x) = −log₂ P(x)

If an event is certain to occur (P=1), observing it eliminates no uncertainty, and the information content is 0 bits. Halving the probability increases the self-information by 1 bit: as P goes from 1/2, 1/4 to 1/8, the self-information is 1, 2, 3 bits respectively.

Take the three weather outcomes as an example: the probability of sunny is 1/2, so when sunny occurs it is not surprising—I = −log₂(1/2) = 1 bit. The probability of snow is only 1/4, so when it snows I = −log₂(1/4) = 2 bits—rarer, so the information is greater.

What does this value represent?It measures "degree of surprise," not importance. Rare sensor noise can have high self-information yet be worthless for a task. Self-information tells you only how rare this outcome is under P, not whether it deserves attention.

What it cannot imply:High self-information does not mean the event is important, useful, or correct; it is purely a mathematical quantity about probability.

3Entropy: The Average Uncertainty of the True Distribution ItselfEntropy

Single events vary in information content; how can we summarize how hard it is to predict a distribution as a whole?

H(P) = E_x∼P[−log₂P(x)] = −Σ_x P(x) log₂ P(x)

Entropy is taking the self-information of each event and averaging it according to the true frequency P.For P=(1/2, 1/4, 1/4), the information amounts of the three events are 1, 2, and 2 bits respectively, weighted average: H(P) = 1/2 × 1 + 1/4 × 2 + 1/4 × 2 = 1.5 bits/observation. This means that over the long term, each observation requires an average of 1.5 bits to describe the result.

For n possible outcomes, the uniform distribution has the maximum entropy (log₂n bits); the entropy of a deterministic distribution is 0. Entropy describes the average uncertainty of a distribution over repeated trials—used to aggregate the single-event unexpectedness of different events into a distribution-level quantity.

How to read entropy:The higher H(P), the harder it is to predict on average over the long run. But high entropy does not mean every event is rare, nor does it evaluate whether a particular answer is correct. The “minimum average coding cost” is an idealized, long-run average lossless coding conclusion; the actual codeword for a single symbol is also affected by integer length, the encoder, and header overhead.

Boundary:Here H(P) measures the uncertainty of the true data-generating distribution P, not whether a model's predictions are correct. A model can also give a highly concentrated Q, even making H(Q)=0, while placing all probability on the wrong outcome; this does not imply H(P)=0, only that the predictive distribution is very certain but may be wrong. The coding interpretation also relies on assumptions such as long-run, lossless, and distributional stability.

4Cross-Entropy: Paying with Q for Data That Appears According to Pcross-entropy

When real events occur according to P, but the model assigns probabilities according to Q, what is the average loss?

The model does not know in advance what will appear next; it can only submit a whole set of predicted probabilities Q. After event x actually occurs, we only check the probability Q(x) that the model gives to this event, and convert it into logarithmic loss:loss for event x = −log₂ Q(x). The greater the probability Q gives to the real event, the smaller the loss; if Q very confidently misses the real event, the loss will be very large.

Since events actually still occur according to P, when averaging over the long term we must use P as weights, thus obtainingcross-entropy:

H(P,Q) = E_x∼P[−log₂Q(x)] = −Σ_x P(x) log₂ Q(x)

The “cross” in the name comes from the two distributions each playing their own role:P determines how often which events appear, Q determines how much loss the model pays for these events.Continuing with the three-weather example: Q gives sunny a probability of 1/2, the same as P, so the loss for sunny days is still 1 bit; but Q gives snow only a probability of 1/8 (P gives 1/4), once snow occurs, the loss is −log₂(1/8) = 3 bit, which is exactly 1 bit more than the 2 bit that comes with P.

Step	Who decides	What to do
1. Event occurrence	true distribution P	determines how frequently sunny, rain, and snow each occur
2. Model reports probability	model distribution Q	assigns a predicted probability to each possible event
3. Compute single loss	Q(x) corresponding to the actual event	compute −log₂Q(x)
4. Take long-term average	P as weights	obtain cross-entropy H(P,Q)

How to read the result:A lower value indicates that Q on average assigns a higher probability to the real event. If exactly Q = P, cross-entropy degenerates to entropy—under the current event space, conditional information, and log loss caliber, the model distribution matches the true distribution, and the loss drops to the theoretical minimum; this does not mean the model has mastered the generative mechanism, semantics, or other capabilities. Values under different data distributions, logarithmic bases, or prediction units cannot be directly compared horizontally.

5KL Divergence: The Extra Cost of Cross Entropy over EntropyKL

KL does not need to be memorized out of thin air; it can be derived directly from “how much extra is spent by using the wrong Q.”

When the true distribution P is known, the average ideal code length is H(P); after switching to the model distribution Q, the average cost is H(P,Q). Subtracting the two gives the extra loss caused by placing model probability in the wrong places:

H(P,Q) − H(P) = Σ_x P(x) log₂[P(x)/Q(x)] = D_KL(P∥Q)

KL divergence measures the number of extra bits paid on average per event after using Q in place of P.Using the three-weather example to calculate: H(P) = 1.5 bit, H(P,Q) ≈ 1.6038 bit, so D_KL(P∥Q) ≈ 0.1038 bit/event—the model assigns too little probability to snow, causing about 0.1 extra bit per event on average.

KL has several key properties: it is never less than 0 (Gibbs' inequality), and it equals 0 only when Q and P are exactly identical.The direction cannot be swapped: D_KL(P∥Q) is weighted by P, focusing on penalizing Q for missing events that will actually occur; if some P(x) > 0 while Q(x) = 0, the forward KL is positive infinity.

Minimizing cross entropy is equivalent to minimizing KL—because H(P) is a constant. But the average loss on the training set is only an empirical estimate of the overall cross entropy; a decrease in training loss does not guarantee that the true KL on unseen data will definitely decrease.

Boundary:The direction of KL cannot be swapped; when P is unknown it can only be estimated; Q assigning zero probability will make it diverge. Therefore KL is a comparison at the level of logarithmic cost, not a geometric distance, nor a comprehensive evaluation of “model quality.”

6KL and Euclidean distance answer different questionsDistance comparison

Since both P and Q can be written as probability vectors, why not directly calculate the straight-line distance between the two points?

Euclidean distance treats the two distributions as ordinary vectors and computes the square root of the sum of squared coordinate differences:

d_{2} (P, Q) = \sqrt{\sum_{x} {(P (x) - Q (x))}^{2}}

It measures coordinate differences and is symmetric (d₂(P,Q) = d₂(Q,P)) and satisfies the triangle inequality. KL, in contrast, measures the average logarithmic cost caused by probability ratios—asymmetric and also not satisfying the triangle inequality, so strictly speaking it is a “divergence” rather than a mathematical distance.

Question	Euclidean distance	KL(P∥Q)
What it focuses on	How far apart the probability coordinates are	How much extra logarithmic cost is paid when Q predicts P
Symmetric?	Yes	No
When Q misses a possible event of P	Still a finite number	If Q(x)=0, then it is positive infinity
Unit	Geometric scale of the probability vector	This page uses bit/event

Taking P=(0.5, 0.25, 0.25) and Q=(0.5, 0.375, 0.125) as an example, the Euclidean distance is about 0.177, and the forward KL is about 0.104 bit.The two numbers cannot be swapped or added—they do not answer the same question.

The value of this comparison:It prevents treating KL as an ordinary distance to calculate. Euclidean distance is not “wrong”; it simply does not express coding regret or logarithmic predictive loss. Smaller values can each indicate a kind of closeness, but the units and meanings differ, so one cannot claim from this that one metric is always better.

7Perplexity: Converting Average Log Loss Back to a Probability ScalePerplexity

An average loss of 1.6 bit/token is abstract; how can we convert it into a more intuitive equivalent number of choices?

PPL = 2^H(P,Q) (loss unit: bit/token)

If the average cross entropy is 2 bit/token, then PPL = 2² = 4.What perplexity means:It converts the average log loss on real data into an equivalent branching factor: for this average loss, it is comparable to a baseline scenario of facing 4 equally likely candidates at each step. What is exponentiated here is the Q loss weighted by P; unless Q=P, it does not equal the exponent of Q's own predictive entropy.

What it is not:Perplexity is not vocabulary size, nor does it mean the model actually considers only a fixed number of words at each step. Lower PPL only indicates that the model assigns a higher average geometric mean probability to the actual tokens. It does not automatically prove better factuality, reasoning ability, or safety.

Here token is used ahead of time—for now, token can be understood as the text unit that the model predicts step by step; the formal tokenization mechanism will be learned in later nodes. Different tokenizers change the prediction unit and sequence length, so token-level PPL usually cannot be directly compared across tokenizers; when comparing across tokenizers, you can consider bits-per-byte or bits-per-character on a unified corpus.

The base must match:If the training loss uses natural logarithms (unit: nat/token), use PPL = exp(loss); if using log₂, only then use 2^loss.

8Three-symbol example: from P and Q all the way to perplexityComplete hand calculation

Real weather P=(1/2, 1/4, 1/4), model Q=(1/2, 3/8, 1/8), how does each step correspond to the earlier definitions?

_KL(P||Q)≈0.1038 bitP determines event frequencies, Q determines encoding costs; subtracting yields extra loss.

Scroll horizontally to view the full diagram on small screens.

Figure 1　The same set of P and Q produces entropy, cross entropy, KL, and PPL in turn. P is responsible for weighting, Q for cost.

Event x	P(x)	−log₂P(x)	Q(x)	−log₂Q(x)	P(x)[−log₂Q(x)]
Sunny	0.5	1	0.5	1	0.5000
Rain	0.25	2	0.375	1.415	0.3538
Snow	0.25	2	0.125	3	0.7500

First average P's own code lengths: H(P) = 1.5 bit.
Q underestimates snow (1/8 vs P's 1/4), so when snow occurs the loss jumps to 3 bit.
Events still occur according to P; weight Q's code lengths by P: H(P,Q) ≈ 1.6038 bit.
KL = H(P,Q) − H(P) ≈ 0.1038 bit/event — the misplaced probability for snow is the main source of extra cost.
Exponentiate: PPL = 2^1.6038 ≈ 3.04.

How to read this result:Q assigns too much to rain and too little to snow; although the total probability is still 1, placing probability in the wrong location increases average loss. If Q sets snow to 0, then when snow occurs it produces infinite loss, and both cross entropy and forward KL become positive infinity.

Boundary:This three-event example only verifies the arithmetic and interpretive chain. Real language models face large-scale discrete conditional distributions over tens of thousands of token types, and rely on finite-sample estimates; it is not a continuous distribution. The same formulas still apply, but estimation error, context conditioning, and tokenization conventions all affect the numerical values.

9Language Model Loss: Sample Average Is Not the True DistributionLanguage Model

In reality, P is unknown; how does the training procedure obtain the cross-entropy loss?

A language model splits a sentence into x₁, …, x_n, and at each step gives the true next-token probability Q(x_t | x_<t).LikelihoodHere it means how much probability the model assigns to the entire true token sequence, used to compare how well the model explains the observed text.Log-likelihoodconverts the product of step-by-step probabilities into a sum of logarithms for stable computation; then taking a negative sign turns "higher probability is better" into "lower loss is better":

NLL = −Σ_t log Q(x_t | x_<t)

Dividing by the number of tokens n gives the average token loss; then averaging over the training batch gives the common empirical cross-entropy.It approximates the unknown true distribution P using finite samples, rather than knowing P directly.

Quantity	How obtained	Can it be directly observed?
Population cross-entropy H(P,Q)	Takes expectation over the unknown true distribution P	Usually cannot be observed exactly
Training loss	Average negative log probability over training samples	Computable, but may overfit
Validation/test loss	Average over samples not used in parameter updates	Used to estimate performance on new data, but still has sampling error

"A drop in training loss" only means the model fits the current training samples better. To judge whether true predictions improve, one also needs an independent validation set, distribution slices, and task metrics such as factuality, reasoning, and safety.Training loss is an empirical estimate, not the true cross-entropy; validation loss is a better approximation, but it is also limited by sample size, distribution shift, and sampling bias.

10Connecting the whole causal chainSynthesis

Don't memorize formulas in isolation: each new quantity answers a question left unanswered by the previous step.

Event x occurs according to the true distribution P, and the model gives predictions using Q. P determines frequency, Q determines cost.(§1)
−log₂P(x) measures how surprising the event is under the true distribution—self-information.(§2)
Averaging −log over P₂P(x) yields the distribution's own average uncertainty—entropy H(P).(§3)
Averaging the model loss −log over P₂Q(x) yields the cross-entropy H(P,Q).(§4)
Subtracting entropy from cross-entropy gives the extra cost of the model placing probability in the wrong places—KL(P∥Q).(§5)
KL and Euclidean distance address different questions: logarithmic cost vs. geometric coordinate difference.(§6)
Exponentiating the average cross-entropy gives the equivalent number of branches for the average log loss on observed data—perplexity PPL; it is not equivalent to Q's own predictive entropy.(§7)
The three-symbol example shows the numerical outcomes of the entire computational chain.(§8)
In reality P is unknown, and the training loss is an empirical estimate; it cannot automatically represent the true cross-entropy.(§9)

Passing standard If you can clearly explain 'why in cross-entropy P is used as the outer weight and Q is inside the logarithm', and can independently compute the complete chain from entropy to PPL using the three-weather example, then you have grasped its core.

11Common MisconceptionsIntuition

Misconception	More Accurate Understanding
P is the truth directly visible in the training set.	P is the unknown data-generating distribution; sample frequencies are only estimates and may also be affected by sampling bias.
Cross-entropy is just computing entropy once for each of the two distributions.	Event frequencies come from P, the per-event loss comes from Q, and the formula is −ΣP log₂Q
KL is an ordinary distance between two probability vectors.	KL is a directional average log cost; it is asymmetric and does not satisfy the triangle inequality.
PPL = 20 means the model only considers 20 tokens.	It means the average log loss for real data corresponds to a baseline case of 20 equally probable candidates, not the actual number of candidates.
PPL is the exponential of Q's own uncertainty.	The usual test PPL exponentiates the cross-entropy weighted by the data distribution; only when Q=P does it coincide with the exponential of the true entropy.
A drop in PPL indicates responses are more truthful and smarter.	It only measures the average predicted probability of actual tokens; other capabilities require independent evaluation.

12Check whether you really understandSelf-test

Given P(rain) = 0.25, calculate the self-information of “it rains today” and state the unit.
Given P = (0.5, 0.5), calculate H(P); if changed to P = (1, 0), how does the entropy change?
In H(P,Q) = −ΣP(x) log₂Q(x), why is the outer weight P(x) but inside the logarithm Q(x)?
How do you derive KL(P∥Q) from H(P,Q) and H(P)? What does this difference represent?
Why can't Euclidean distance replace KL for expressing coding cost?
Assuming the average loss is 3 bit/token, calculate the perplexity. What does it not represent?
If training cross-entropy keeps decreasing but validation cross-entropy starts to increase, can you assert that the true KL is still decreasing?
Two language models use different tokenizers, A's PPL = 12, B's PPL = 15. Can you directly assert that A is better?

Reference answers

I(rain) = −log₂0.25 = 2 bit.
H(P) = 1 bit; when P = (1,0), H(P) = 0, because the outcome is already determined.
P determines the long-term frequency of true events and is responsible for the weighting; −log₂Q(x) is the model's per-sample loss, so Q is inside the logarithm.
Subtracting gives ΣP(x) log₂[P(x)/Q(x)], which represents the average extra number of bits paid when data are generated according to P but encoded using Q.
Euclidean distance only measures the difference in probability coordinates and is symmetric; KL measures probability-ratio cost weighted by P. In particular, when P(x)>0 and Q(x)=0, the Euclidean distance is finite while the forward KL is positive infinity.
PPL = 8 means the average log loss on the observed data is equivalent to a baseline with 8 equally likely candidates at each step; it does not represent vocabulary size, the actual number of candidates, or factual, reasoning, or safety performance.
No. The training loss is the empirical average over training samples and may continue to decrease due to overfitting; an increase in validation loss instead indicates worse generalization. The true P is unknown.
No. Different tokenizers change the token units and sequence lengths; you should standardize the corpus and text normalization, use common units such as bits-per-character or bits-per-byte, and then combine with task metrics.

13Concept Dependencies and Further LearningPath

Learning Level	Concepts Covered
Before this page	Only need to be able to read probabilities and weighted averages; this page has supplemented logarithms, expectation, P and Q.
Core of this page	Self-information, entropy, cross-entropy, KL divergence, perplexity
Next Step	Loss Function, Gradient Descent, Token and Tokenization
Engineering Extensions	Pre-training, probability calibration, sampling parameters, model evaluation, and distribution shift

Sources and adaptation notes

Shannon, A Mathematical Theory of Communication: basic explanation of self-information, entropy, logarithm bases, and lossless coding.
Deep Learning — Probability and Information Theory: cross-entropy, KL, asymmetry, and probability interpretations in machine learning.
Stanford STATS 311: Information Theory and Statistics: formal definitions and properties of entropy, KL, and the coding view.

Date accessed: 2026-07-28