Contrastive Learning: Using Positive and Negative Pairs to Shape Representation Space

Contrastive Learning does not require knowing category names like “shoe” and “car” in advance, but it must know which views should be regarded as the same object. The rule for constructing training data plays this role: generate two views from the same original sample and label them as a positive pair; treat views from other samples as negative pairs. In this way, each training batch forms an implicit relation table—it does not answer what category a sample belongs to, only “who should be pulled closer and who should be pushed farther apart.”

An augmented view is first fed into encoder f to obtain representation z=f(view). z is the feature intended for downstream tasks. Then projection head g maps z into the space where the contrastive loss is actually computed, yielding g(z). During training, similarity comparison and pulling close and pushing away happen in the projection space; the training signal then passes back through g to f, updating the encoder. Separating the representation space from the loss space means downstream can use the pre-projection z, without directly using the projection result specially shaped for the contrastive objective.

Taking shoe image A as anchor, A⁺ is another view cropped from A, so A and A⁺ form a positive pair; another shoe B and car C come from other samples, so according to the current construction rule they are treated as negative pairs. Their cosine similarities with A are 0.8, 0.6, 0.1 respectively, with temperature τ=0.2. 0.8 indicates that the positive view is already quite close to the anchor, 0.6 indicates that B, although labeled as a negative sample, is still quite similar to the anchor, and 0.1 indicates that car C is already clearly far away. The loss accordingly increases the relative similarity between A and A⁺, and reduces the competitiveness of other candidates relative to the positive sample.

Therefore, what the model learns depends not only on the encoder architecture but also on how this relation table is constructed. Positive pairs specify which view differences the model should ignore, and negative pairs specify which sample relations should be distinguished. Here the “positive” and “negative” are relation labels given by the training rule, not equivalent to true semantic categories: although B is also a shoe, it is still treated as a negative pair because it comes from another original sample. This boundary determines the supervision that the contrastive objective can provide, and also reminds us not to directly interpret relations constructed from sample identity as complete category semantics.

InfoNCE turns representation learning into a classification problem temporarily created around an anchor: given anchor i, treat it as a query and identify the unique positive view among a set of candidate vectors. The candidate set contains both positive and negative pairs, so the “class” is not a fixed semantic label but which candidate currently forms a positive pair with the anchor.

The loss for anchor i is:

ℓᵢ = −log[exp(sim(zᵢ, zᵢ⁺) / τ) / Σⱼ∈candidates exp(sim(zᵢ, zⱼ) / τ)]

Here, zᵢ is the anchor vector, and zᵢ⁺ is the positive view vector corresponding to it; j iterates over the candidate set, and zⱼ may be either a positive sample or a negative sample. sim(zᵢ, zⱼ) measures the similarity between the anchor and the candidate, and τ scales the similarity. Each scaled similarity sim(zᵢ, zⱼ)/τ is a logit, exp turns it into a positive value, and the sum in the denominator normalizes over all candidates. Thus, the fraction can be interpreted as the softmax probability that the model assigns to the correct positive sample, and ℓᵢ is the negative logarithm of this probability.

This formula forms a clear causal chain: positive-pair similarity increases → the positive-pair logit increases → the positive pair's probability share among all candidates in the denominator increases → the negative log loss decreases. Backpropagation therefore raises the positive-pair logit. Meanwhile, how much each negative pair is pushed depends on its current softmax probability: negative pairs that are more similar to the anchor and have higher probability have a greater effect on the loss; negative pairs that are already very dissimilar and have probability near zero have a weaker effect. The denominator must include the positive pair, because the normalization compares the share of the correct candidate relative to all candidates, rather than maximizing a similarity in isolation.

If similarity is computed directly with unnormalized vectors, the model may increase the logit by enlarging the vector norm instead of actually improving the relationship expressed by the vector direction. Usually the vectors are normalized first so that the similarity mainly reflects directional closeness, which avoids reducing the loss merely by enlarging the norm. The result of InfoNCE should be interpreted as the positive sample's relative identification probability in the current candidate set; it optimizes the relative ordering and margin among candidates, not an absolute similarity detached from the candidate set.

Continuing with anchor shoe image A: A⁺ is its positive view, while another shoe B and a car C are negative candidates. The three have cosine similarities with A of [0.8, 0.6, 0.1], respectively, with temperature τ=0.2. InfoNCE first divides the similarities by the temperature to obtain logits:

[0.8/0.2, 0.6/0.2, 0.1/0.2] = [4, 3, 0.5]

Next, exponentiate each logit:

[exp(4), exp(3), exp(0.5)] ≈ [54.60, 20.09, 1.65]

The sum of these exponentiated scores is:

54.60 + 20.09 + 1.65 = 76.34

Therefore, the softmax probability assigned to positive pair A⁺ is:

p⁺ = 54.60 / 76.34 ≈ 0.715

The loss for anchor A is the negative log of the positive pair probability:

ℓ = −log(0.715) ≈ 0.335

This value means that when the three candidates A⁺, B, and C compete together, the model assigned about 71.5% probability to the correct candidate A⁺. The loss remains greater than zero because the other two candidates, especially B with its higher similarity, still occupy part of the probability mass.

The derivative of the cross-entropy with respect to a negative candidate's logit equals the candidate's current softmax probability. Therefore, the derivative corresponding to B is 0.263, while C is only 0.022; the push-away signal from B is about 0.263/0.022 times that from C. The reason is not that the model knows in advance that B is more important, but that B is more similar to the anchor and occupies more incorrect probability after the softmax. The car C is already easy to distinguish, so continuing to push it away helps little in reducing the loss; shoe B is the harder competitor at the moment.

Whether this gradient allocation matches the task objective depends on the relational assumption that "different instances should be separated." If the goal is instance retrieval, the other shoe B is a difficult negative that must be distinguished, so pushing it away strongly is reasonable; if the goal is to learn shoe semantics, B and A may naturally be close, and B is then a false negative. The same 0.335 loss and 0.263 gradient may represent effective training under one objective, or damage to semantic structure under another.

The temperature τ sits between the similarity and the softmax; the logit for candidate j is sim(zᵢ, zⱼ)/τ. It does not change the order in which candidates are ranked by similarity, but it changes the extent to which similarity differences are amplified after entering the softmax, thereby determining which candidates the training signal concentrates on.

When τ becomes smaller, the same set of similarities is divided by a smaller number, so the logit gaps between neighboring candidates widen. The softmax probabilities concentrate on the few highest-scoring candidates: if the positive sample has the highest score, it receives a higher recognition probability; hard negative samples that are very similar to the anchor also receive larger probabilities and gradients, while the probabilities of easy negative samples rapidly approach zero. The causal chain is: τ becomes smaller → logit gaps are amplified → the probability distribution becomes sharper → gradients concentrate more on the hardest-to-distinguish candidates → the representation is under stronger local discrimination pressure.

When τ becomes larger, the logit gaps are compressed, the softmax distribution becomes smoother, and more candidates participate in training together. If τ is too large, the candidate probabilities approach a uniform distribution, the influence of similarity magnitude on probability weakens, and the model receives insufficient discriminative signal.

A smaller τ is not inherently better. If the highest-scoring negative candidate is actually a mismatch, or if it should be semantically close to the anchor under the target semantics, a sharp softmax will concentrate a large amount of gradient on this erroneous relationship, violently pushing apart samples that should be close. Temperature therefore simultaneously controls “how hard the negative samples the model focuses on are” and “how much data noise is amplified”.

τ can be fixed or learned, but it cannot be chosen independently of the training configuration. The batch determines the candidate set and the chance of hard negative samples appearing; similarity normalization determines the score scale; data noise determines how many high-similarity negative pairs contain erroneous relationships. Only by considering these three together with the temperature does the degree of sharpness or smoothness represented by τ acquire a stable meaning.

The contrastive loss sees not the samples' true categories, but the relationships jointly constructed by augmentation and sampling. Augmentation produces positive views from the same original sample, determining which point should move closer to the anchor; sampling places other samples into the candidate set, determining which points should be pushed farther away. Only after the two rules are combined does the actual direction of movement in the representation space emerge.

Easy negative C ← Anchor A → Positive sample A⁺ small push away │ pull closer │ Hard negative B strong push away

In the figure, A is the anchor, and A⁺ is the positive sample obtained according to the augmentation rule. The loss increases the relative similarity between A and A⁺, so the two continue to move closer in the representation space. B is very similar to A, but is labeled as a negative sample according to the sampling relation; it is the more competitive candidate at present and receives a stronger push-away signal. C is already far from A, is an easy negative sample, has less influence on the current loss, and thus moves less.

The point of this figure is not that all points move by a fixed distance, but that gradient direction and relative strength are jointly determined by the constructed relationships and current similarity: positive pairs are pulled closer; negative pairs are pushed farther away; the more a negative pair resembles the anchor, the more it tends to participate in competition and the stronger the push-away effect it receives. As training continues, these local movements accumulate and gradually shape the entire representation space.

The meaning of B cannot be judged solely from the “hard negative” label in the figure. If the downstream task requires distinguishing different instances, B is similar to A but should indeed be separated, so it is a legitimate hard negative; if the downstream semantics require samples of the same class to cluster together, B should originally be close to A but is pushed away by the construction rule, so it is an incorrect false negative. The loss itself cannot distinguish these two cases, because it only knows the positive/negative relations fed to it. Augmentation and sampling are therefore not auxiliary steps before training, but directly specify what the representation space should retain, ignore, and separate.

Data augmentation is not harmless preprocessing; it declares an invariance to the model. Applying transformations to the same sample to obtain two views and pulling them together as a positive pair is equivalent to requiring the encoder to output similar representations. Whatever changes occur between the two views, the model is trained to ignore as much as possible.

Random cropping changes the object's position in the view and may cause partial loss. Pulling the original image and the cropped image together trains the model to be invariant to positional changes and a certain degree of partial loss. Color jitter changes color; pulling views before and after jitter together trains color invariance. Back-translation in text changes wording while trying to preserve meaning; pulling the original sentence and the back-translated sentence together trains wording invariance. The causal chain can be written as: choose augmentation transformation → generate positive pairs with specific differences → contrastive objective pulls the two representations together → encoder reduces sensitivity to the difference.

This invariance is valuable only when the factors changed by the augmentation truly do not affect the target semantics. If color determines food doneness, color jitter may erase the information needed to judge doneness; if rotation or flipping changes medical orientation, forcing the representations before and after the transformation to be consistent may suppress the orientation signal; if cropping removes the main subject, the generated view no longer expresses the original sample's semantics, yet it is still treated as a positive pair; if text rewriting replaces a negation or entity, the so-called positive pair has actually changed meaning. At this point, training still faithfully executes the instruction to pull them together, but the result is eliminating useful differences as if they were irrelevant differences.

To judge whether an augmentation is reasonable, one cannot just look at whether the transformed sample "looks like" the original sample, but must look at whether the transformation preserves the semantics and labels required by the current task. Augmentation strength also has boundaries: minor changes may establish useful invariance, but after crossing the range of semantic preservation, the same transformation will produce incorrect positive pairs. Therefore, the augmentation strategy is essentially an assumption about the target semantics; what information the representation ultimately retains and what it suppresses is directly constrained by this assumption.

Contrastive loss requires the positive sample to compete with a set of negative samples, so the training system must continuously provide enough negative candidates. Large batches, cross-device sharing, and queues are all ways to expand the candidate set, but they make different trade-offs in memory, communication, and representation consistency.

SimCLR directly uses a large batch: other samples encoded in the same step can become negative samples. In this way, the candidate vectors come from a single computation under the current parameters and have consistent coordinates, but the larger the batch, the more memory and computational resources are consumed. Cross-device all-gather can collect vectors from each device, allowing each query to share more negative samples, but it increases inter-device communication.

MoCo does not require putting all negative samples into one very large batch at the same time. It saves keys generated in historical steps in a queue; when the current query arrives, it can be compared with the current positive key and can also use historical keys in the queue as negative candidates. The queue separates the generation time of negative samples from their usage time, thereby replacing a single-step very large batch with cross-step accumulation.

The problem is that if historical keys are generated by a rapidly changing encoder, they and the current vectors will lie in a constantly changing coordinate system. MoCo therefore uses a momentum encoder to generate and update keys, whose parameters satisfy:

θ_key ← mθ_key + (1−m)θ_query

θ_key is the encoder parameter that generates the queue keys, θ_query is the query encoder parameter currently trained by gradients, and m is a momentum coefficient close to 1. Each update retains m proportion of the old key parameters and absorbs only 1−m proportion of the new query parameters. Thus the key encoder changes more slowly, vectors written to the queue in adjacent steps are relatively consistent, and historical keys and current keys can participate together in comparison in a relatively stable coordinate system.

This mechanism addresses negative-sample supply, not guaranteeing that more negative samples always lead to better results. When keys in the queue are too old, even if the encoder updates slowly, they may drift relative to the current representation; while large batches expand the candidate set, they also increase the chance of treating semantically similar samples as negative pairs. Practical design needs to consider candidate quantity, memory, communication, queue freshness, and false negative sample risk simultaneously. System throughput determines what kind of candidate set can be provided, and the candidate set in turn directly changes the statistical objective of training; the two cannot be optimized separately.

False negatives are samples that are labeled as negative pairs according to training construction rules but should be close according to target semantics. Whether two different shoe images constitute a false negative cannot be decided only by “they are not the same image” or “they are both shoes”; it depends on what task the representation ultimately serves.

If the goal is instance discrimination, the model needs to recognize the uniqueness of each image. Two different shoe images, although of the same category, can still serve as a negative pair; separating them helps distinguish specific instances. If the goal is category retrieval, when querying one shoe image you want other shoes to be close as well, so treating another pair of shoes as a negative pair produces incorrect supervision. The loss faithfully reduces the similarity between the two, and the training signal gradually tears apart same-class semantics that should have been clustered.

This shows that positive and negative relations are not facts naturally present in the data, but modeling decisions oriented toward downstream tasks. Construct negative pairs → the loss treats candidates as competitors → high-similarity negative pairs receive a stronger push-apart signal → the representation space separates them. As long as the relation labels in the first step are inconsistent with the task semantics, the later computation, even if completely correct, still optimizes in the wrong direction.

There are several ways to mitigate false negatives. Supervised contrastive learning can assign multiple same-class positives to an anchor, preventing samples with the same label from repelling each other; nearest-neighbor mining can treat nearby samples in the representation as potential positive pairs; debiased loss attempts to correct erroneous relations in the negative set; clustering pseudo-labels can use group structure to construct positive and negative relations; filtering high-similarity candidates avoids treating the most suspicious candidates directly as negative pairs. But all these methods introduce new sources of error: neighbors may be only superficially similar, clustering and pseudo-labels may be wrong, and filtering high-similarity candidates may also remove truly valuable hard negatives. Therefore, they make trade-offs between relation errors and new assumptions rather than unconditionally eliminating the problem.

The opposite error is false positives: training rules determine that two things should be close, but their semantics have changed. Adjacent frames of the same video may cross a cut, and a patient's condition may change between two examinations; if they are still treated as positive pairs based on temporal proximity or the same identity, the model will be forced to ignore these changes. Reliable relation labels must simultaneously conform to the time range, object identity, and specific task semantics. Both positive and negative pairs should be understood as task-related assumptions, not permanent judgments about real relationships.

If the training objective only requires the two views of the same original sample to be close, then making all inputs output the same vector can indeed minimize the positive-pair distance. This constant solution preserves no information that distinguishes samples, yet fully satisfies the naive consistency loss. “Representational collapse” is this phenomenon: different inputs receive nearly identical representations; the model seemingly accomplishes alignment but actually loses usable structure.

Methods without explicit negative samples therefore still need an additional mechanism to rule out the constant solution. VICReg writes the collapse-prevention requirement directly into the loss:

L = L_invariance + λL_variance + μL_covariance

L_invariance is the invariance term, which pulls positive-pair representations closer; L_variance is the variance term, which prevents each representation dimension from losing variation; L_covariance is the covariance term, which reduces redundancy among different dimensions. λ and μ control the weights of the variance term and the covariance term, respectively, in the total loss. The three terms serve different roles: invariance alone encourages all samples to move together; maintaining per-dimension variance requires the data to remain spread out in each dimension; penalizing redundant covariance prevents multiple dimensions from repeatedly carrying the same variation.

Its causal relationship is as follows: positive-pair consistency provides the signal that “two views of the same object should be close,” while the variance constraint prevents all samples from shrinking to a constant, and the covariance constraint prevents the representation from changing only in a few repeated directions. The final output must be stable to positive views while preserving enough structure across different samples and across different dimensions.

A normal decrease in the training loss cannot prove that the representation has not collapsed, because the model may quickly reduce the invariance term by sacrificing all sample differences, or only some dimensions may collapse. During evaluation, it is necessary to simultaneously observe the standard deviation of each dimension, the covariance spectrum, the effective rank, and nearest-neighbor diversity. The standard deviation can check whether some dimensions have lost variation; the covariance spectrum and effective rank can reflect whether information is concentrated in a few directions; nearest-neighbor diversity checks whether different inputs can still form distinguishable neighborhoods. Only when these structural signals and the loss change are both normal can complete or partial collapse be ruled out.

Low training loss only shows that the model satisfies the relationships constructed during training well; it does not show that the representation captures the semantics needed by the target task. The model may rely on shortcuts such as watermarks, backgrounds, colors, or capture devices, and may harm minority classes even when overall metrics look normal. Representation quality must be validated jointly through target-task performance and counterfactual slices.

When performing linear probing with a frozen encoder, the encoder parameters remain unchanged, and only a linear predictor is trained on its output representation. The input is the frozen features, and the output is the target label prediction. If the target information can be read by a simple linear boundary, this indicates that the information already exists relatively directly in the representation; if the probing result is poor, it indicates that a low contrastive loss has not automatically translated into linear readability of the target information.

kNN or Recall@k checks representations from the neighborhood perspective. Given a query representation, it finds the k nearest samples in the feature space, observes whether the neighbors match the target semantics, and checks minority classes separately. It answers the question of “whether nearby vectors are truly samples that should be nearby for the task,” and is closer to how retrieval is used than only looking at training loss. Fixed-budget few-shot fine-tuning limits the annotation quantity and training budget, and compares how efficiently representations adapt to the target task under limited supervision; the better the performance, the easier the pre-training representation is for target data to use.

Counterfactual tests are used to identify shortcuts. Keep the target semantics unchanged while swapping the background, color, watermark, or capture device, and then observe whether the prediction or neighborhood changes significantly. If changing only the watermark changes the result, the model has likely encoded the watermark as a key basis. Further slicing by data source, group, and out-of-distribution data can reveal local failures hidden by the overall average: the model may work only on a certain source, or fail quickly outside the training distribution.

t-SNE and UMAP can map high-dimensional representations to two or three dimensions in a lossy way, which is suitable for intuitively finding suspicious clusters and outliers. Their input is the high-dimensional representation, and their output is low-dimensional coordinates that are easy to view, but the dimensionality reduction process changes distances in the original space. Therefore, clusters that appear separated in the plot alone cannot prove that the original space has the same cluster structure, nor can they prove fairness or retrieval quality; visualization should be used to raise questions that need verification, which should then be confirmed by quantitative evaluation and slice tests.

Comparisons between different methods are interpretable only when controlled conditions are consistent. The backbone network, training data, augmentation strategy, training budget, and evaluation protocol must all be held fixed; otherwise, metric differences may come from resources or settings rather than the representation learning method itself. A reliable conclusion should answer three things at once: whether the target information can be read, whether the neighborhood matches the target semantics, and whether it remains stable under counterfactual changes and different data slices.

Contrastive learning starts from the task definition, rather than from the loss formula. First, clarify the downstream semantics: which differences should be preserved, and which variations should be ignored. This choice specifies the required invariance and determines whether the subsequent relationship labels are reasonable. If the task requires recognizing categories, it should protect category semantics; if the task requires distinguishing instances, it should preserve instance differences. With different objectives, the same pair of samples may have different relationships.

The downstream semantics are then translated into positive pairs and candidate negative pairs in the training data. Positive pairs tell the model which views should be brought closer, and candidate negative pairs tell the model which samples need to be distinguished. If the relationship construction is wrong, the loss does not automatically correct the semantics; it only carries out the incorrect pulling closer or pushing apart more efficiently. Therefore, augmentation and sampling are the key interface from task assumptions into the optimization process.

Each view is encoded into a vector, and similarities are computed after normalizing the vectors. Normalization prevents the model from changing scores merely by increasing the norm, making the similarity reflect more directly the relationship between representation directions. These similarities become the input for candidate competition, and InfoNCE then places the positive sample among the entire set of candidates for identification. The incorrect candidate most similar to the anchor occupies a higher probability and therefore receives a stronger gradient; easily distinguished candidates have less influence. Thus, hard candidates become the main force shaping local boundaries.

The training process cannot be judged only by decreasing loss. Candidates may include false negative samples that semantically should be close, and the model may exploit shortcuts such as watermarks or backgrounds to satisfy the relationship objective; if an effective anti-collapse mechanism is lacking, different inputs may also lose discriminability. What needs to be monitored is not a single number, but whether the relationships are consistent with the task, whether the representations maintain diversity, and whether the model relies on factors that should not be preserved.

Final validation must return to the downstream semantics defined at the outset. Transfer evaluation checks whether the representation can be used by the target task, while counterfactual tests change factors such as background, color, watermark, or device while keeping the target semantics unchanged, and observe whether the results are stable. The complete causal chain is therefore:

Downstream semantics and invariance → positive pairs and candidate negative pairs → encoding, normalization, and similarity → InfoNCE focuses on hard candidates → monitoring false negatives, shortcuts, and collapse → transfer and counterfactual test validation

Only when the evidence at the end of the chain is consistent with the task definition at the beginning can we say that the representation has learned the expected structure. A low training loss only means the intermediate links are running smoothly; it cannot replace validation oriented to the target semantics.