Dimensionality Reduction: Preserving Task-Required Structure with Fewer Coordinates
Understand compression objectives, visualization distortion, data leakage, and downstream evaluation through PCA, random projection, t-SNE, UMAP, and autoencoders.
1Compression Must Choose What to PreserveIntuition
From one thousand dimensions to two, what information should be kept?
Dimensionality reductiontransforms each sample from many original coordinates into fewer new coordinates. The input is high-dimensional data of "number of samples × number of original features", and the output is a representation of "number of samples × fewer dimensions"; samples do not disappear, but the coordinates describing each sample become fewer.
It solves storage, computation, noise, and visualization problems, but it cannot preserve all relationships losslessly at the same time. Therefore the first step is not to choose an algorithm, but to declare what to be faithful to: PCA favors global linear variance and reconstruction, random projection approximates pairwise distances, and t-SNE and UMAP focus more on local neighborhoods. Different objectives naturally lead to different output layouts.
2Two Equivalent Views of PCAMathematics
Why are maximum variance and minimum reconstruction error connected?
PCA (principal component analysis)seeks a low-dimensionallinear subspace: first subtract the training mean from the data, then use several mutually perpendicular directions as new coordinate axes.X represents the already centered data matrix;W each column is a projection direction to be learned;XW are the low-dimensional coordinates after projection;WᵀW=I means these directions have unit length and are mutually orthogonal.
The objective is to maximize the total variance of the projected coordinates for a fixed dimensionality. The intuition is: the more spread out the centered data is in a direction, the more sample variation that direction carries. Computationally, one can obtain eigenvectors of the training covariance matrix and select the top few directions corresponding to the largest eigenvalues.
Multiply the low-dimensional coordinates by Wᵀ and add back the mean to approximately reconstruct the original data. Under squared error and orthogonal linear projection, “retaining maximum variance” is equivalent to “minimizing the lost variance, that is, the squared reconstruction error.” This equivalence holds only for linear orthogonal projection and squared error; it does not guarantee retaining label, causal, or minority-class signals.
3Scale and Fitting OrderData
When running PCA on income and age together, which one will dominate?
PCA compares variance in all directions, so variables with larger numerical ranges influence the principal axes more strongly. If “1 dollar” and “1 year” are merely different measurement units, usually first use the training-set mean and standard deviation to compute z-scores, letting each dimension participate in the comparison through its relative deviation; if the absolute scale itself carries business importance, blindly standardizing will instead erase the meaning.
The input consists of three raw feature sets—training, validation, and test; the training stage outputs the mean, scale, and projection matrix, and the latter two can only use the same set of parameters to do transform. The correct order is “split first → fit the standardizer and PCA on the training fold → transform validation/test”.
If you first use the full dataset to compute the mean, variance, or principal axes, the test distribution has already participated in selecting the representation, resulting in data leakage. Results should be compared between the raw-unit scheme and the standardized scheme in terms of both reconstruction and downstream metrics, not assuming that standardization is always correct.
4How to Read t-SNE and UMAP PlotsVisualization
Does a large distance between two-dimensional clusters mean they are far apart in the original space?
t-SNE and UMAP mainly used to embed local neighbor relationships of high-dimensional samples into two or three dimensions. The inputs are the high-dimensional samples and hyperparameters such as neighborhood and distance; the output is low-dimensional display coordinates for each sample. They optimize a neighborhood objective of “who should be close to whom”, rather than fully recovering all global distances.
The working process can be summarized as: first construct neighbors or proximity probabilities in the high-dimensional space, then find low-dimensional coordinates so that neighbors remain adjacent as much as possible. To fit into two dimensions, the algorithm will stretch, compress, or even rotate the global layout, so island distances, cluster areas, orientations, and empty spaces usually have no direct quantitative meaning.
Plots can be used to propose hypotheses such as “these points may form local groups”, but must be verified by returning to the original features, labels, or external evidence. Random seeds, perplexity, number of neighbors, and distance metrics can all change the layout; parameters should be reported, and neighbor preservation rate and stability across multiple runs should be checked. It is not suitable for directly making causal interpretations from two-dimensional distances or making high-risk decisions.
5Supervised and Nonlinear CompressionMethods
What if the maximum-variance direction contains no label signal?
PCA does not read labels; it keeps only the linear directions with the largest variance. High variance may come from background or equipment, while the key predictive signal may have very small variance. If the goal is classification, retrieval, or ranking, you can choose supervised projection, metric learning, or use an autoencoder to learn nonlinear compression.
Supervised dimensionality reduction takes features and labels as input and outputs low-dimensional coordinates for task discrimination; metric learning brings samples that should be similar closer together and pushes samples that should be separated farther apart; an autoencoder uses an encoder to output a latent variable z, then a decoder reconstructs the input, learning a nonlinear representation through reconstruction loss. The three differ in “how they work,” so they cannot be chosen only by how good a two-dimensional plot looks.
Results should be interpreted separately by downstream accuracy or recall, reconstruction error, neighborhood preservation, and minority-class performance. Supervised signal increases the risks of label leakage and overfitting, and an autoencoder may only memorize background texture; interpretability is usually also weaker than with linear principal axes. High variance does not equal high predictive value, and low reconstruction error does not equal complete task information.
6How to determine the number of dimensionsEvaluation
Is 95% explained variance a universal rule?
Dimensionality is a performance–cost hyperparameter that must be selected on the validation set, not the lower the better. The input is a set of candidate dimensionalities and a fixed training pipeline, and the output should be a curve of quality, storage, latency, and stability as dimensionality varies.
PCA's cumulative explained variance can tell how much training variance is retained, but it does not tell whether that variance is useful for the task. You should compare, dimension by dimension, downstream accuracy or retrieval recall, reconstruction error, inference latency, index size, and separately check minority classes, anomalies, and out-of-distribution samples. Choose the lowest-cost point that meets the quality threshold, rather than mechanically applying 95%.
Candidate dimensionalities may be selected only on validation or inner cross-validation; leave the test set until the pipeline is frozen. When the deployment distribution changes, both the original dimensionality and the principal axes may become invalid and need to be monitored and re-validated.
7Complete hand calculation: How two-dimensional PCA finds the first principal axisStep-by-step calculation
How much information is lost when the four points (2,1), (4,2), (6,3), (8,4) are compressed to one dimension?
The mean is (5,2.5), and the centered points all lie along the direction (2,1); the covariance matrix is proportional to [[4,2],[2,1]]. The largest eigenvector is normalized to w₁=(2/5,1/5), and the orthogonal direction w₂ = (−1/5,2/5) has eigenvalue 0.
After projecting all points to one dimension, they can be reconstructed exactly, with an explained variance ratio of 100%. If independent noise is added to the second dimension of each point, the second eigenvalue will rise; keeping one dimension then requires a trade-off between compression rate and noise/task signal.
| Point | Centered | One-dimensional coordinate z |
|---|---|---|
| (2,1) | (−3,−1.5) | −7.5/5 |
| (4,2) | (−1,−0.5) | −2.5/5 |
| (6,3) | (1,0.5) | 2.5/5 |
| (8,4) | (3,1.5) | 7.5/5 |
8Original Figure: Different Objectives Preserve Different StructuresVisualization
Given the same cloud of high-dimensional points, why don't PCA, random projection, and neighborhood graphs produce the same picture?
Three branches receive the same high-dimensional sample matrix X, and produce different low-dimensional representations according to the preselected target dimension. PCA first estimates the mean and an orthogonal projection matrix W, then outputs Z=(X−μ)W; random projection samples and fixes a random linear matrix R, and directly outputs Z=XR; t-SNE/UMAP first X constructs neighbor probabilities or a neighbor graph, then iteratively optimizes the current samples' two- or three-dimensional coordinates Z.
Therefore, PCA and random projection produce linear transformations reusable for new samples, while neighborhood embedding first yields a display layout for the current sample set. The three methods respectively treat global linear variance and reconstruction, pairwise distances, and local neighborhoods as fidelity objectives; acceptance metrics in the figure must correspond to that objective and the downstream use.
Scroll horizontally to view the full diagram on small screens.
9Random projection trades probabilistic guarantees for extremely low fitting costJohnson–Lindenstrauss
Without learning principal axes, it can still approximately preserve distances—what is the cost?
For n points, a random mapping to k = O(log n/ε²) dimensions can, with high probability, make all pairwise squared distances fall within a factor of (1±ε). It does not look at data directions, requires no iteration, and suits sparse large-scale settings; however, the dimension is a probabilistic upper bound, and specific tasks still require empirical testing.
| Method | Preservation goal | Can transform new samples | Main limitations |
|---|---|---|---|
| PCA | Global linear variance | Yes | Low-variance task signal |
| Random projection | Approximate distances | Yes | Random error |
| t-SNE | Local probabilistic neighborhood | Usually non-parametric | Global distance distortion |
| Autoencoder | Learned reconstruction | Yes | Overfitting and shortcuts |
10Common Misconceptions and Learning PathMisconceptions and Dependencies
Dimensionality reduction is task-dependent compression, not a window into a “true two-dimensional world.”
| Misconception | More accurate understanding |
|---|---|
| The first two principal components have the most predictive power | PCA only looks at variance, not labels |
| 95% explained variance is a universal threshold | Minority-class signal may lie in low-variance directions |
| t-SNE islands prove natural clusters | Neighborhood objectives and hyperparameters can create empty space |
| Reducing dimensionality on the full data before splitting is harmless | The mean and projection have already leaked the test distribution |
| The lower the dimensionality, the better | Dimensionality is a quality–cost hyperparameter |
| Level | Dependencies and extensions |
|---|---|
| Prerequisites | Covariance, eigenvalues, distance |
| Core of this page | PCA, random projection, neighborhood embedding, leakage |
| Adjacent | Curse of Dimensionality, clustering, embedding |
| Acceptance | Reconstruction, retrieval, downstream tasks, drift |
11Dimensionality reducers are models and must be fit only on the training fold.Leakage boundary
Why can unlabeled PCA also leak the test set?
Although PCA does not read y, it uses the mean, variance, and covariance of the test samples to determine the projection direction; this makes the training representation adapt to the future distribution in advance. During cross-validation, each fold should refit the scaler and dimensionality reducer on the training portion of that fold, then transform the validation portion. If you first select the number of dimensions on the full data, inspect the two-dimensional plot, and then decide on label cleaning, you also introduce analyst leakage.
| Step | Wrong order | Correct order |
|---|---|---|
| Split | Split after dimensionality reduction | First fix training/validation/test |
| Fit | Estimate mean and principal axes on the full data | Use only the training fold |
| Select dimensionality | Repeatedly look at test results | Select on validation/inner CV |
| Report | Report only the best random seed | Test once after freezing the pipeline |
After deployment, new data can only call transform; if you refit, a new version has already been produced and must be regression-tested together with the downstream model.
12Nonlinear autoencoders can reconstruct well but represent poorlyNonlinear Boundary
Why doesn't low reconstruction error guarantee that classification, retrieval, or causal factors are preserved?
An autoencoder receives the raw input x; with parameters θ encoder f compresses it into a latent variable z; with parameters φ decoder g then produces the reconstruction x̂.L is the total training loss; its first term measures the squared error between the input and the reconstruction;R(z) is a regularization function on the latent variable structure,λ controls the weight of regularization relative to reconstruction.
A high-capacity encoder can memorize pixel texture, background, and sensor noise, reconstruct with very low error, yet compress away rare diagnostic signals; latent variables can also become uninterpretable in every dimension through complex rotations and entanglement. Bottleneck dimension, denoising, sparsity, or variational regularization are merely preferences and do not guarantee semantic disentanglement.
You should simultaneously measure reconstruction, downstream linear probing, nearest neighbors, counterfactual sensitivity, and out-of-distribution transfer. For minority classes, compare recall before and after compression separately; overall error will be swamped by majority background pixels. If interpretable directions are needed, PCA or supervised sparse projection may actually outperform deep networks.
13After deployment, monitor projection residuals and representation driftProduction monitoring
When the input distribution changes, why might the old principal axes keep running while silently becoming distorted?
A fixed PCA will still output coordinates, but new data may shift variance into directions that were discarded during training. Monitor per-dimension mean/variance, reconstruction residual, subspace angle, downstream quality, and minority slices; when thresholds are exceeded, refit with a new time window and release the dimensionality reducer and downstream model together as a combined version in a canary rollout. Do not replace only the projection matrix: rotating the coordinate system makes old classifiers, indexes, and centroids lose their meaning. For exploratory plots such as t-SNE, a layout fitted separately on a new batch also cannot be compared directly with the old plot by coordinates.
14Acceptance QuestionsCheckpoint
What should you check first after compression?
First confirm that the test set was not involved in fitting, then compare quality, cost, minority-class performance, and out-of-distribution performance across dimensions; no single variance threshold can substitute for this set of evidence.
15Connect the causal chainSynthesis
How does this concept connect from problem to verifiable practice?
- High-dimensional data contains redundancy and noise
- Choose the structural target to preserve
- Fit the projection/manifold to the training data
- Compress to a low-dimensional representation
- Measure structural distortion and downstream benefit
- Monitor and refit according to the new distribution
16Misconceptions and Self-TestSelf-test
Can you explain its mechanism, boundaries, and validation methods without memorizing terminology?
- What does PCA preserve?
- When is standardization needed?
- Can t-SNE cluster distances be directly interpreted?
- How does dimensionality reduction leak?
- How is the number of dimensions chosen?
- Suppose "dimensionality reduction: using fewer coordinates to preserve the structure required by the task" performs normally on offline examples, but core results drop after deployment. How would you locate the problem according to input, internal transformation, output feedback, and applicable boundaries?
Reference Answers
- The direction of maximum linear variance.
- When the scale of variables should not represent importance.
- Usually not; global distances are distorted.
- By fitting the mean or projection using all the data.
- Validate by downstream quality and cost.
- First, save the same failed sample and environment, and confirm that the input, permissions, and preconditions have not drifted. Next, record the key intermediate states and check whether the mechanism performed the transformation as described on this page. Then compare the original output with independent metrics and manual final verification. Finally, retest with boundary examples and controlled experiments. Only when you locate the first step that deviates from expectations can you decide whether to modify the data, the mechanism, the evaluation, or the applicable boundary.
- Principal Component Analysis: PCA tutorial
- t-SNE: local visualization
- UMAP: manifold neighborhood dimensionality reduction
- An elementary proof of a theorem of Johnson and Lindenstrauss: dimensionality order and distance-preservation bound for random projection
- Reducing the Dimensionality of Data with Neural Networks: encoding, decoding, and reconstruction objectives of deep autoencoders
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations: non-identifiability boundary of semantic disentanglement in unsupervised representation learning