Variational Autoencoder (VAE): Compressing Data with a Sampleable Probabilistic Latent Space

The core training objective of an ordinary autoencoder is reconstruction: encode the input x into a latent vector z, then use the decoder to reconstruct x as faithfully as possible. As long as the data itself can be compressed, this constraint is often sufficient to give every sample in the training set a good reconstruction. The problem is that strong reconstruction ability does not mean the latent space itself is "generative".

A deterministic encoder works by mapping each training sample to an isolated point in the latent space. During training, the reconstruction loss imposes constraints only on these points: the decoder learns to "output the corresponding sample at these specific points". The regions between these points are never forced to be learned by the decoder, leaving large hollow areas. Once you randomly sample a coordinate from the latent space, this coordinate is very likely to fall into a hollow area rather than near some training point; when the decoder faces an input it has never seen, it outputs meaningless noise. In other words, training covers a scattered set of points in the latent space, but not the continuous regions spanned by these points.

VAE's change is precisely to turn "samplability" from an implicit expectation into a training objective. The encoder no longer outputs a deterministic z, but outputs the mean and variance of a conditional distribution q(z|x), indicating "roughly which region in the latent space this sample falls into and how much uncertainty there is". At the same time, the KL term added to the objective function pulls q(z|x) toward a shared prior distribution p(z). This prior is usually chosen to be a simple standard normal distribution, which does not favor any particular location. With this constraint, the distribution corresponding to each sample no longer collapses to an isolated point, but expands around its mean; the distributions of different samples are all pulled toward the same central region, overlapping and connecting with each other. As a result, training no longer covers discrete points, but a more continuous and denser region, and when random sampling falls into this region, the decoder is already capable of producing reasonable outputs.

To be clear, this does not mean that every point in the latent space carries clear semantics, nor does it guarantee that any random point can produce good-looking samples. What it guarantees is that randomly sampled points highly overlap with the regions that were actually optimized during training, so the decoder will not be completely unfamiliar when it encounters these points. Samplability is a constraint explicitly written into the training objective, not a byproduct that automatically emerges after training is complete.

The goal of a VAE is not merely to reconstruct, but to construct a generative model: first sample a latent vector from the prior p(z), then generate data from the generative distribution p(x|z). This generative direction is clear and executable. But conversely, given an observation x, if we want to infer "which z generated it," we need the posterior distribution p(z|x). According to Bayes' formula, p(z|x) = p(x|z) p(z) / p(x). The two terms in the numerator—the generative distribution and the prior—are both specified by us; the problem lies in the denominator p(x).

p(x) is the marginal likelihood, equal to integrating over all possible z: p(x) = ∫ p(x|z) p(z) dz. This integral cannot be computed analytically in the vast majority of realistic models, because the latent space is high-dimensional and the generative distribution is a nonlinear neural network. The posterior p(z|x) therefore becomes a quantity that "exists in principle but cannot be computed in practice." Many learning algorithms for generative models essentially work around this intractable posterior.

VAE's solution is to approximate the true posterior with a learnable distribution qφ(z|x), where φ denotes the parameters of the encoder network. This is the meaning of "variational approximation": we do not compute the true posterior directly, but instead, from a family of distributions parameterized by a neural network, choose a distribution that is as close as possible to the true posterior. The encoder is therefore called a variational approximation, rather than the model's true posterior—it is an optimized stand-in carrying the parameter φ. Correspondingly, the decoder pθ(x|z) explicitly defines the generative distribution, with parameter θ.

In this way, the two parts of the model each have their own role: the encoder qφ(z|x) is responsible for "given x, explaining which z it might correspond to," and the decoder pθ(x|z) is responsible for "given z, generating what kind of x." During training, θ and φ are optimized simultaneously, with two goals: first, the approximate posterior should be able to explain the observed data—that is, z sampled from qφ(z|x) can then be decoded to reconstruct x; second, the approximate posterior should be close to the prior p(z), ensuring that the latent space is consistent with the sampling distribution. These two goals are merged into a single optimizable quantity in the objective function that follows.

If the goal of a VAE were only to minimize pixel error, it would be no different from an ordinary autoencoder, and there would be no sampling to speak of. So the loss function must answer two questions at once: whether z sampled from the approximate posterior can explain the data, and whether that approximate posterior deviates from the prior we selected.

The starting point of the derivation is the log-likelihood log pθ(x). Since the true posterior is intractable, we instead seek a lower bound on it, which is the Evidence Lower Bound (ELBO):

E_q[log pθ(x|z)] − KL(qφ(z|x) ‖ p(z))

The first term is the reconstruction term in expectation form: under the approximate posterior qφ(z|x), a sampled z passed through the decoder yields the log probability of x. It rewards "the latent variable indeed carries enough information to reconstruct the observation". The second term is the KL divergence: it measures the difference between the approximate posterior qφ(z|x) for each input and the prior p(z), penalizing distributions that collapse to isolated points and stray far from the prior. The two terms are placed in the same objective—the former requires the posterior to be informative, the latter requires the posterior to remain regularized; the two are naturally in tension.

In actual training, we minimize the negative ELBO, turning the maximization of the lower bound into a minimization of the loss:

−ELBO = −E_q log pθ(x|z) + β·KL(qφ(z|x) ‖ p(z))

When β = 1, this is the standard VAE form. β is an explicit coefficient used to adjust the relative weight of the KL term: increasing β forces the posterior more strongly toward the prior, making the latent space more regularized but potentially sacrificing reconstruction details; decreasing β lets the model encode information more freely, giving more accurate reconstructions but weakening the continuity of the latent space. It is important to emphasize that β changes the trade-off between information capacity and regularization; it is not a simple "turn it up to improve quality" knob—both directions can harm sample quality in different ways, and the actual effect depends on the data and task.

The encoder no longer outputs a deterministic z, but rather a mean μ and a standard deviation σ. To sample a z from N(μ, σ²) during training, you encounter a seemingly minor but fatal obstacle: sampling is a random operation. If z is generated directly as a function of μ and σ, then μ and σ become entangled with the random node, and during backpropagation the gradient cannot pass smoothly through this random step. The gradient needs to know "if μ or σ changes a little, how will z change?" but random sampling severs this deterministic dependency.

The reparameterization trick is precisely meant to solve this problem. It rewrites the sampling as follows: first draw ε ~ N(0,1) from the standard normal distribution, then set z = μ + σ·ε. In this way, randomness is completely isolated in ε, and ε does not depend on any parameters; the relationship between z and the parameters μ and σ becomes deterministic, differentiable arithmetic. Gradients can therefore propagate normally along the partial derivatives of z with respect to μ and σ, and the encoder can be trained.

Let's look at this operation with a set of concrete numbers. Let μ = 1, σ = 0.5, and suppose one draw gives ε = −0.4, then

z = μ + σ·ε = 1 + 0.5 × (−0.4) = 0.8

In other words, this sample falls 0.4 standard deviations away from the mean in the latent space. μ and σ determine the "position" and "width" of the distribution, while ε only provides the random shift thrown in this round of sampling; each plays its own role.

The KL term has a closed-form solution for Gaussian distributions. The KL divergence of a one-dimensional Gaussian from the standard normal distribution N(0,1) is

KL = 0.5 × (μ² + σ² − 1 − log σ²)

Substituting μ = 1 and σ = 0.5 gives 0.5 × (1 + 0.25 − 1 − log 0.25) ≈ 0.818. The meaning of this value is how much "distance" the distribution has from the standard normal; the larger the KL value, the more it deviates from the prior, and the stronger the pull that training exerts to drag it back near the prior. The closed-form solution allows the KL term to be computed exactly without sampling, further ensuring that the objective function is differentiable and can be optimized stably.

Using handwritten digits as an example, you can see the entire process of a VAE from training to use. The encoder outputs a two-dimensional mean μ and log σ² for each input image (using log variance instead of variance itself ensures the variance remains positive and facilitates optimization). During training, for each image, a two-dimensional z is sampled via reparameterization and reconstructed, while the KL term pulls the distributions corresponding to each digit class toward the standard normal prior.

After training, plotting the mean μ of each sample as a scatter plot allows you to observe the structure of the latent space. Usually, digit classes form overlapping clusters rather than sharply separated hard partitions—this is exactly the result of the KL term pushing the distributions toward the same prior: there is no absolute wall between classes, but smooth connections.

This continuity can be checked by interpolation. Take the latent mean z₃ of a "3" and the latent mean z₈ of an "8", interpolate linearly between them, and for each intermediate point sequentially sample z and feed it to the decoder. Because these intermediate points lie in a continuous region covered during training, the decoder will produce smooth transitions in strokes rather than abrupt changes or gibberish. It should be emphasized that smooth interpolation only shows that the decoder is a continuous function in the latent space; it cannot prove that every intermediate shape corresponds to a real handwritten digit—an intermediate frame may just be a mechanical mixture of features from the two digits, not something that appeared in the data.

To generate new samples, one randomly samples points from the prior N(0, I) and passes them to the decoder. Two things need to be checked: first, whether different digit classes are all covered, and second, whether invalid samples appear. These two points reveal whether the prior sampling region coincides with the region actually covered during training.

If anomalies occur, they can be located by KL values. If the KL is close to 0 and q(z|x) for different inputs is almost the same, this indicates posterior collapse—the encoder no longer distinguishes inputs, all images are mapped to the same distribution, and all information is lost. If the KL is too large and reconstruction quality is poor, this indicates the KL term is weighted too heavily and the model lacks sufficient capacity to encode useful information; you need to relax β or increase the latent space capacity.

The training structure of a VAE can be understood as a complete data flow. The input x first passes through the encoder, which outputs the sample's mean μ and variance σ²; then the latent variable z is obtained through the reparameterization z = μ + σ·ε (ε is sampled from a standard normal distribution); z is fed into the decoder to reconstruct x̂, and the reconstruction loss measures how close x is to x̂. At the same time, a parallel constraint acts on the encoder output: the KL divergence pulls each sample's distribution qφ(z|x) toward the standard normal prior p(z).

Figure 1 shows exactly this dual-path structure: one path retains information via the reconstruction term, and the other shapes a sampleable latent space via the KL term. These two paths respectively counter two failure modes.

When the KL term is too weak, the encoder is barely constrained by the prior. Each sample's distribution can shrink into isolated points far from each other, and the training coverage degenerates into a discrete set of points like an ordinary autoencoder. Reconstruction quality may be high because information is encoded without any regularization pressure, but the latent space is full of holes, and z sampled from the prior likely falls in regions the decoder has not seen, producing invalid samples. This is the "can reconstruct, cannot generate" scenario.

When the KL term is too strong, the prior constraint overwhelms the reconstruction requirement. The encoder is forced to map all inputs to nearly the same standard normal distribution, and the distributions lose their distinctiveness. At this point the latent variable no longer carries unique information about the input, and the decoder can only output a blurry average result. Reconstruction quality drops significantly, and samples become monotonous. This is the "regularized latent space, lost information" scenario, which is the direction of posterior collapse.

The two extremes show that a balance must be maintained between the KL term and the reconstruction term. The reconstruction term requires the encoder to output informative distributions, and the KL term requires these distributions to converge uniformly to the prior; only when the two are evenly matched does the latent space remain both continuous and sampleable while retaining sufficient class and detail information.

"Reconstruction loss" is not an engineering option that can be chosen arbitrarily; its specific form is determined by the decoder likelihood pθ(x|z), and the choice of likelihood actually encodes the assumptions we make about data noise and pixel independence.

The most commonly used mean squared error (MSE, i.e., L2 loss) corresponds to a Gaussian pixel likelihood: it assumes that each pixel, given z, follows a Gaussian distribution and that pixels are mutually independent. A direct consequence of this assumption is that the decoder’s optimal output is the conditional mean. When the information in z is insufficient to uniquely determine pixel values—for example, the same latent vector may correspond to multiple plausible images with different details—taking the conditional mean averages away all possible details, producing blurry images with missing details and soft edges. Blur is not an accident of failed optimization; it is the natural result with the lowest cost under the assumption of "Gaussian independent pixels".

Bernoulli likelihood, on the other hand, assumes that each pixel is binary (black or white). It is naturally suited to binary data such as handwritten digits, but it is not suitable for general grayscale or color photographs—forcibly treating photograph pixels as 0/1 events loses continuous brightness information. Clearly, the choice of likelihood must match the data type; otherwise, even if optimization converges, the results will not meet expectations.

If the goal is sharper, more realistic-looking results, one can introduce more expressive decoder distributions (such as autoregressive pixel models or mixture distributions), perceptual loss, or hierarchical latent variables. However, these improvements usually change the interpretation of the likelihood: the model no longer exactly optimizes a clearly defined log-likelihood, and sample quality may become disconnected from the likelihood value. Therefore, the key to understanding the VAE reconstruction term is to realize that the loss function already encodes assumptions about the noise model and pixel independence, rather than treating it as a freely replaceable numerical term.

Posterior collapse is a subtle failure mode in VAE training: the model's reconstruction looks fine, but the latent variable no longer carries any information about the input. It occurs when the decoder is too powerful and does not need z at all.

A typical scenario is using an autoregressive decoder: the decoder can predict the next pixel well from the previously generated pixels or tokens, so it does not depend on the latent variable at all. Faced with such a decoder, the easiest optimization path is to make the approximate posterior qφ(z|x) directly equal to the prior p(z). This reduces the KL divergence to 0, minimizes the loss, and makes z and x statistically independent—the encoder outputs a fixed distribution unrelated to the input, while the decoder reconstructs on its own. Reconstruction quality may still be decent because the decoder can work by itself, but the latent space has been hollowed out.

The signs of collapse are clear: the μ and σ corresponding to different inputs are almost identical; manually changing the value of z barely changes the decoded output. Diagnosis cannot rely only on total loss, because total loss may still be decreasing while collapse has already occurred. You need to observe the per-dimension KL values (which dimensions are compressed to be indistinguishable from the prior), the number of active units (how many dimensions are still transmitting information), and perform latent variable intervention experiments—change z and see whether the output changes accordingly.

There are several mitigation methods: KL annealing (weaken the KL weight early in training so the encoder first learns to encode information, then gradually tighten it); free bits (set a minimum KL budget for each dimension, and stop applying pressure once it is below the budget); reduce decoder capacity so it must rely on z; add mutual information constraints between the latent variable and the input; or use hierarchical latent variable structures. The common idea behind these methods is to address the root cause where "the decoder does not need z" and force the model to actually use the latent variable.

The β coefficient controls the tightness of the VAE information bottleneck. Increasing β means giving greater weight to the KL term, more strongly limiting the information capacity that the encoder can use. This pressure can lead to a tempting result: the model is forced to represent the main factors of variation in the data with fewer, more independent, and smoothly varying dimensions. People often see in visualizations that a latent variable dimension happens to correspond to properties such as "angle" or "stroke thickness," and then claim that the VAE automatically "disentangled" the true generative factors.

But this conclusion requires caution. The constraint imposed by β-VAE is "limited capacity + closeness to the prior"; it does not directly encode any objective that "factors should be mutually independent and each dimension should be semantically meaningful". In unsupervised settings, the coordinates of the latent space can be arbitrarily rotated and permuted, and these transformed coordinates can explain the data equally well—in other words, "which dimension corresponds to which semantic meaning" has no unique answer mathematically. A visualization in which one axis happens to correspond to angle is very likely just a coincidental arrangement under the current random seed and initialization, rather than the model truly identifying the causal factors behind the data.

To make "disentanglement" a credible property, additional inductive biases or supervised signals are usually needed, such as data with known factors, explicit disentanglement constraints, or priors tailored to specific structure. Drawing a conclusion based only on a pretty two-dimensional scatter plot is to mistake "finding a coordinate that explains the data" for "finding the true factors".

Therefore, evaluating disentanglement should use stricter evidence: intervene on the latent variables and see whether the output changes only along the expected attribute; use the latent variables for downstream prediction to verify whether they have predictive power for the target factor; check whether the same factor consistently corresponds to the same dimension under different random seeds; and perform quantitative evaluation on datasets with known factor labels. Only when these lines of evidence are consistent does "disentanglement" hold up.

In modern image generation, VAE has another important use: serving as a lossy compressor, that is, the front-end codec of latent diffusion models. To understand this role, first clarify what it does not do—it is not responsible for "denoising generation".

In this architecture, the image VAE compresses high-resolution pixel space into a smaller latent tensor. Diffusion or flow models do not work directly in high-dimensional pixel space; instead, they perform generation in this compressed latent space. After generation is complete, the VAE decoder restores the latent tensor to an image. The division of labor is: VAE is responsible for round-trip compression between pixel space and latent space, while diffusion models are responsible for sampling and denoising in latent space.

This means that VAE's reconstruction capability sets the quality ceiling for the entire generation pipeline. Color accuracy, tiny text, facial details, high-frequency textures—whether these can be preserved depends on how much is lost in this lossy round-trip of VAE encoding-decoding. If the VAE itself cannot recover these details, then no matter how strong the subsequent generator is, it cannot conjure back information that was already lost during the encoding stage; no matter how strong the generator is, it cannot exceed the ceiling set by codec loss.

Therefore, when generated images exhibit artifacts or detail errors, it is necessary to distinguish the source of the error. An effective diagnostic method is to test separately: send a real image directly to the VAE for one reconstruction and observe the reconstruction result; then observe the output of the full generation pipeline. If the same blur or text errors already appear in the reconstruction of the real image, then the problem lies in codec compression loss; if the reconstruction is clean but the generated result is flawed, then the problem lies in the generator itself. Only by testing the two links separately can you pinpoint whether the fault is codec artifacts or generation errors.

Evaluating VAE cannot look at only one metric, because different metrics answer different questions, and they are not always consistent. A higher ELBO does not necessarily mean the generated samples look better.

ELBO is an approximate lower bound on the log-likelihood, and its value is directly affected by the likelihood assumption and the quality of the approximate posterior. If the likelihood assumption (for example, independent Gaussian pixels) does not match the real data itself, samples may be blurry or distorted even if the ELBO is high. More fundamentally, ELBO measures the model's probability density assignment to the data, which is different from "whether the sample is visually appealing".

Therefore, hierarchical evaluation is needed. Reconstruction metrics measure the ability to reconstruct seen inputs, answering "whether the encoder-decoder round trip is faithful". Sampling from the prior and then decoding examines the coverage of the latent space—whether random sampling points fall within the training coverage area and can yield valid samples. FID or human evaluation measures whether the distribution of generated samples is close to the real data distribution. Performance on downstream tasks measures whether the latent variables are useful as "representations". In addition, one should report the KL value, the number of active dimensions, interpolation smoothness, precision/recall (the trade-off between coverage and diversity), and the stability of latent variable interventions, which can reveal problems such as posterior collapse, mode dropping, or latent space discontinuity.

When doing comparative experiments, one should also control variables: compare with ordinary autoencoders, different β values, and other generative models of the same capacity, and fix the sampling budget. If one is allowed to select only the best few samples from a large number for display, conclusions will be contaminated by the "cherry-picking effect" and cannot reflect the model's true generative ability. Only by placing the likelihood lower bound, reconstruction, sample distribution, and representation quality side by side can one form an unbiased judgment of VAE.

The per-sample KL constraint may seem to have already pulled each posterior toward the prior, but even if the KL for each sample is controlled, sampling from N(0, I) can still fall into low-quality regions. The reason is that what truly determines "whether the training region and sampling region are consistent" is not the posterior of a single sample, but the overall shape formed by mixing the encoded distributions of all data, namely the aggregate posterior.

The aggregate posterior is the distribution obtained by averaging qφ(z|x) over all training samples. The per-sample KL simultaneously constrains two things: limiting the amount of information encoded by each sample, and making each distribution match the prior. These two pressures do not automatically guarantee that the aggregate posterior equals the prior. Limited model capacity, a restricted approximate posterior family (for example, assuming a diagonal Gaussian), and errors left by insufficient optimization can all cause the aggregate posterior to shift overall or develop holes relative to the prior. As a result, points sampled from N(0, I) may fall into regions not covered by the aggregate posterior—regions where the decoder has never actually been optimized during training, so the outputs are naturally of low quality.

To diagnose this problem, you cannot only look at the mean of the per-sample KL. A more direct approach is to compare the difference between the two distributions of "z obtained by sampling from the prior" and "z obtained by encoding samples", for example by training a discriminator to distinguish them; if the discriminator can easily separate the two, it indicates a clear mismatch between the aggregate posterior and the prior. You can also examine the relationship between latent-space density and generation quality to see whether low-density regions correspond to poor samples.

Directions for improving the match include using a more flexible prior, hierarchical latent variables, or posterior flows (using invertible transformations to enhance the expressiveness of the approximate posterior). These techniques can narrow the gap between the aggregate posterior and the prior, but at the cost of increased training and density-computation complexity. The key point is that verifying whether these improvements are effective must be proven by prior sampling—that is, actually sampling from the prior, generating, and observing quality—rather than indirectly inferred only from reconstruction metrics.

Connect the whole thread: VAE's logical chain starts from the generative definition and proceeds step by step to verifiable practice.

The starting point is defining the generative direction: first sample a latent vector from the prior p(z), then generate data through the generative distribution p(x|z). The generative direction is clear, but inferring "which z generated x" in reverse requires the posterior p(z|x), whose denominator p(x) integrates over all z and is usually intractable. Therefore a learnable qφ(z|x) approximates this difficult posterior—this is the true identity of the encoder.

With the approximate posterior, it must also be made trainable. Direct sampling cuts off gradients; the reparameterization z = μ + σ·ε isolates randomness into standard noise ε that does not depend on parameters, allowing gradients to propagate normally through μ and σ. The objective function consists of two terms: the reconstruction term preserves input information, ensuring latent variables are sufficient to reconstruct observations; the KL term shapes a latent space close to the prior, making training-covered regions overlap with sampling regions. The two are combined into the same negative ELBO, and the β coefficient adjusts this trade-off.

There are several failure points on this chain that must be continuously monitored. Posterior collapse occurs when a strong decoder does not need z, manifesting as input distributions converging and changing z not changing the output; when a VAE is used as a codec for diffusion models, its lossy reconstruction capability constitutes an insurmountable quality ceiling for the entire pipeline. Validation cannot just look at the total loss; it must actually sample from the prior, check coverage and diversity, and then choose evaluation methods and β according to the different purposes of representation or generation.

Landing on the "validation layer", a credible controlled experiment needs to fix four types of conditions. Input fixed: the same batch of samples, the same preprocessing and permission boundaries, and record input hashes, slice labels, and rejection reasons. Mechanism fixed: change only one core variable and lock all other configurations, and record key intermediate states and the first position where expectations deviate. Output fixed: use the same acceptance rules and resource budget, and compare stratified differences in quality, cost, latency, and failure rate. Falsification fixed: retain a control group that does not enable the target mechanism, and test whether gains reproduce stably across samples and random seeds. Only when control-group differences, sample diversity, and seed stability all point to the same conclusion are the gains truly validated.