Self-supervised Learning: Letting Data Generate Its Own Training Targets

The key to self-supervised learning is not that there is “no objective,” but that the objective does not require manual sample-by-sample labeling. Given a raw data item z, first use a predefined transformation rule q to construct the model input x and automatic target y from z, i.e., (x, y) = q(z). The model fθ with parameters θ receives x and produces a prediction fθ(x); the loss function ℓ(fθ(x), y) then measures the discrepancy between the prediction and the automatic target. Training updates θ by reducing this loss, so the whole process still has a clear, computable supervisory signal—it is just that y comes from the data itself.

The reason that “shifting a sentence one position to the right” can produce a supervisory signal is precisely that the original sentence already contains the token that actually follows each position. The model sees the preceding tokens as x; after the shift, the aligned successor token becomes y, so no one needs to write the answer again. Likewise, a masked original segment can serve as the reconstruction target; the fact that two augmented versions of the same image come from the same original sample also automatically defines the pairing relationship. The rule q determines what the model sees and what it predicts, and therefore also determines the meaning of the training signal.

Taking three product images A, B, and C as an example, cropping A yields A⁺. Because A and A⁺ come from the same product, they automatically form a positive pair; B and C serve as candidate negative samples. The model’s inputs are these image views, and its output can be a representation used for comparison; the loss requires the representations of A and A⁺ to be closer while being distinguished from those of B and C. A decreasing loss indicates that the model is increasingly able to recognize “whether they are the same product” from view changes, but it does not automatically prove that the representation is already suitable for all downstream tasks; what is learned is still constrained by the construction rule and the data content.

Therefore, “supervised learning” emphasizes that there is an objective and a loss during training; “self-supervised learning” further explains that the objective is constructed from the data itself; and “unsupervised learning” is a broader umbrella term. The premise for a self-supervised method to work is being able to design a stable and meaningful q: the automatically generated y must be related to the structure that we want the model to capture. If the objective produced by the construction rule is too easy, ambiguous, or irrelevant to the actual use, the model may learn only useless shortcuts even if it minimizes ℓ well.

Autoregressive training directly treats the "next token" in a sequence as the target. For a text of length T, simply shifting the input one position relative to the original text makes each existing prefix correspond to the actual token that follows it; as long as the starting position with no preceding context is not counted as an ordinary prediction, a piece of text can provide T−1 targets. The model learns from these positions in a single pass over the sequence, rather than receiving just one label from the entire text.

The loss over the entire text can be written as:

L_AR = −Σₜ₌₂…T log pθ(xₜ ∣ x<ₜ)

Here, L_AR is the autoregressive loss over the entire sequence, T is the total number of tokens, t denotes the current prediction position, and Σ sums the losses at all positions. xₜ is the true token at position t, and x<ₜ denotes all tokens before it; pθ(xₜ ∣ x<ₜ) is the conditional probability that the model with parameters θ assigns to the true next token given the prefix. log is the natural logarithm. Since probabilities are between 0 and 1, the smaller the probability of the true token, the more negative log becomes; the leading minus sign turns it into a larger loss. Conversely, the higher the probability the model assigns to the true token, the smaller that position's contribution to the total loss.

The causal mask used during training is key to making this objective valid. When predicting xₜ, the model can only read x<ₜ; it cannot see xₜ or tokens after it in advance, otherwise the answer would leak directly. Taking "refund was approved." as an example, when the input prefix is "refund", the automatic target is "was"; when it is "refund was", the target is "approved"; when it is "refund was approved", the target is ".". Thus the same original text continuously produces multiple prefix–target pairs, and every predictable position in the data contributes a learning signal.

To reduce L_AR across many positions, the model must exploit regularities in the prefix that help predict what follows, including syntax, topic, word co-occurrence, factual phrasing, and code structure. The form of the objective is exactly the same as token-by-token generation, so training also naturally creates a generation interface: given an existing prefix, the model repeatedly predicts the next token and then appends it to the prefix to continue predicting.

The meaning of the loss has clear boundaries. It rewards the model for producing continuations similar to the distribution of the training data, without directly judging whether the content is factually true, harmless, or compliant with the current user. A lower autoregressive loss indicates that the model is better at predicting the tokens that actually appear in the data; it cannot by itself prove that the model's answers are correct or appropriate by other standards.

Masked reconstruction first hides part of the content from the original sample and then requires the model to recover the hidden part based on the remaining content. It directly turns “the originally masked content” into an automatic target, so it requires no manual labels while forcing the model to use structure in the visible context. Unlike prediction that can only read previous text, text masking can typically use evidence from both the left and right of the missing position simultaneously; image masking can use surrounding visible regions to recover the missing image patch.

The loss can be written as:

L_mask = −Σᵢ∈M log pθ(xᵢ ∣ x¬M)

L_mask is the total loss over all masked positions, M is the set of masked positions, and i∈M means the summation covers only these positions. xᵢ is the original true content at position i, and x¬M denotes the context that remains visible after the content in M is removed. pθ(xᵢ ∣ x¬M) is the recovery probability that the model with parameters θ assigns to the true content given the visible context. Taking the negative logarithm of this probability means that the less the model believes the correct content, the greater the penalty; the more accurately the model recovers the masked content, the lower the total loss.

In text, BERT recovers a masked token based on the text on both sides of it; in images, MAE hides several image patches and then reconstructs the missing regions from the visible image patches. The input is the masked sample, the output is the prediction of the content at the masked positions, and the training target is taken directly from the original sample before masking. The causal chain thus formed is: choose masking positions → remove the directly readable answer → the model integrates visible evidence → predict the missing content → compute the loss with the original content.

Masking ratio controls the amount of information and difficulty of the task. When masking is too little, missing content can often be easily copied from adjacent regions or local regularities, and the model may rely on local shortcuts rather than forming representations with more transfer value; when this is diagnosed, you can enlarge the masked block or increase the distance between visible evidence and target. A medium masking ratio usually achieves a good balance between “there is still enough evidence to predict” and “the task is not so simple that shortcuts can be taken”; downstream transfer curves can be used to judge whether this balance is effective. When masking is too much, visible evidence is insufficient, the true target itself may have multiple plausible possibilities, and the loss becomes dominated by unpredictable details; at this point, reconstruction quality and semantic probing results should be observed separately.

The “optimal masking ratio” is not a fixed constant; it depends on the data modality and the model. Even if the reconstruction loss is low, one cannot directly infer that the model has learned the desired semantics. For example, when pixels are the recovery target, the model may spend much of its capacity on details such as color and texture rather than the semantic structure that downstream tasks care about. Therefore, masking design must not only make the missing content recoverable but also make the evidence required for the recovery task consistent with the information one hopes to retain.

Contrastive learning turns representation learning into a candidate identification problem: given the representation a of anchor sample A, the model must identify the augmented view A⁺ of the same product from a set of candidates, while pushing candidate negative samples B and C to the back. The input is the representations of the anchor, the positive view, and candidate negative samples; the output is the probability that each candidate matches the anchor; the automatic target is “which candidate comes from the same product as the anchor.”

The loss for a single anchor is:

ℓ = −log [exp(sim(a, a⁺) ∕ τ) ∕ Σⱼ exp(sim(a, j) ∕ τ)]

a is the representation of anchor A, and a⁺ is the representation of the positive sample A⁺. j iterates over the positive sample and all candidate negative samples; sim(a, j) measures the similarity between the two representations. τ is the temperature, used to scale the similarity differences; sim(a, j) ∕ τ is the logit before exponentiation and normalization. Taking exp of each logit and dividing by the sum of all candidate exponential values gives the normalized probability for that candidate. ℓ takes the negative logarithm of the positive sample probability, so the closer the positive sample probability is to 1, the smaller the loss; if negative samples receive high scores, the positive sample probability is pushed down by the denominator, and the loss rises.

Using product images A, B, and C, A⁺ is a cropped view of A. Let s(A,A⁺)=0.8, s(A,B)=0.3, s(A,C)=0.1, and temperature τ=0.2. Dividing the similarities by the temperature gives three logits:

[0.8 ∕ 0.2, 0.3 ∕ 0.2, 0.1 ∕ 0.2] = [4, 1.5, 0.5]

After taking exponentials, the values are approximately [54.60, 4.48, 1.65], and the denominator is 54.60 + 4.48 + 1.65 = 60.73. Thus the positive sample probability is:

54.60 ∕ 60.73 ≈ 0.899

The corresponding loss is:

−log 0.899 ≈ 0.106

This result shows that, among the current three candidates, the model has already ranked A⁺ ahead of B and C, with a clear margin. It describes relative recognition ability within this candidate set, not an absolute guarantee for all possible products.

The temperature determines how much similarity differences are amplified. When τ is low, the same similarity gap becomes a larger logit gap, and the model is more strongly penalized for ranking errors; but if the positive-negative relationship is mislabeled, this amplification also strengthens the erroneous supervision. The candidate set itself also changes the task: the batch size and the source of negative samples determine which competitors are in the denominator, thereby changing the difficulty of recognition. Especially when a “negative sample” is actually semantically close to the anchor, a false negative occurs, meaning that the training rule requires the model to push apart representations that should be close. Therefore, a lower loss should be interpreted in the context of temperature, candidate composition, and pairing quality; it cannot be compared in isolation from these conditions.

The same raw data does not naturally correspond to a unique “good representation.” Self-supervised learning first uses construction rules to transform raw data into training inputs and automatic targets, and the model is subsequently rewarded only for completing this objective. Thus, what the objective requires the model to predict, recover, or ignore determines which information the representation must retain; information that does not affect the loss does not have the same strength of incentive to be encoded.

Figure 1 starts from the same raw data and branches into three task paths—autoregressive, masking, and contrastive—finally forming a representation. The three paths differ not in the raw samples but in how automatic labels are generated. The autoregressive task requires predicting subsequent content from existing content, so the representation must retain cues useful for later prediction. The masking task requires recovering missing parts from visible parts, so the representation will preferentially organize contextual evidence that can be used for reconstruction. The contrastive task requires identifying which views should be close and which candidates should be distinguished, so the representation will retain information helpful for this matching judgment and weaken parts that the construction rules allow to vary.

This causal chain can be written as:

raw data → construction rules → inputs and automatic targets → loss constraints → information retained or ignored in the representation

Therefore, “automatic labels” are not a neutral data-processing step. Even if the raw data are exactly the same, as long as the construction method changes, the error signals the model receives will differ, the direction of parameter updates will differ, and the resulting representation may naturally differ. A lower training loss only shows that the representation is sufficient to support the current pretext objective; it cannot by itself show that the representation retains all information, nor can it guarantee that the information in it happens to match downstream uses.

Choosing a pretext objective is essentially specifying a learning preference: which differences the model should be sensitive to, and which differences it should regard as irrelevant. Judging whether a construction rule is appropriate cannot depend only on whether the automatic target is easy to generate; one must also check whether the information necessary to complete that objective matches the information one actually wants the model to retain. Figure 1 emphasizes exactly this point: the representation is not determined by data alone, but jointly by the data and the task construction.

In contrastive self-supervised learning, data augmentation does not just increase the number of samples; it also defines what the model should remain invariant to. Designating two augmented views of the original sample as a positive pair is equivalent to declaring to the model that the factors changed by augmentation should not change the sample's semantic identity. To bring the representations of these two views closer, the model actively weakens these changes. Therefore, augmentation rules are a training assumption about "which differences are irrelevant".

When randomly cropping product images, a reasonable assumption is that local views still belong to the same object. If both crops retain the main product subject, treating them as a positive pair can train the model to ignore changes in composition or visible range. But if a crop leaves only brand background and completely loses the product, the positive pair relationship still requires the background view to be close to the product view, forcing the model to equate the background with the product identity. In this case, automatic pairing is not wrong in origin—the two images do come from the same original image—but it becomes a semantically incorrect positive pair.

Different augmentations imply different invariances and have their own risks of destruction. Random cropping declares "local views are still the same object", with the risk of cropping out the semantic subject; color jitter declares "color does not determine identity", but when color itself is the task label, this declaration removes necessary information; text back-translation declares "rewriting preserves semantics", but the back-translation process may change negation, entities, or tone. Similarly, rotating medical images, flipping digits, or changing audio speed may also change the labels originally intended to be recognized; we cannot assume semantic invariance just because these operations are technically feasible.

Too-weak augmentation also produces the opposite problem. If the two positive views are almost identical, the model may identify the pair using only low-level cues such as color histograms, file compression artifacts, or borders, without learning the desired semantics. In this case the loss can drop quickly, but the drop comes from shortcuts and does not mean that the representation truly captures object identity.

Augmentation design therefore needs to lie between two kinds of failure: it must be strong enough that the model cannot rely on irrelevant surface shortcuts, but not so strong that it changes semantic labels or removes key subjects. To judge whether an augmentation is appropriate, check whether the invariance it declares matches the task and observe whether the augmented positive pair still holds semantically. Only when both conditions are met will "pulling positive pairs closer" push the representation in the intended direction.

If the training objective only requires two views of the same sample to produce the same representation, then mapping all inputs to the same constant vector can indeed make the consistency loss very low. This is representation collapse: positive pairs are completely identical to each other, but different samples are also indistinguishable, so the representation loses useful information. The problem is not the requirement that “two views be brought close” itself, but that with only this constraint, the objective does not require the representation to preserve cross-sample variation.

Methods without explicit negative samples must additionally include an anti-collapse mechanism. Methods such as BYOL and DINO create asymmetry through designs such as stop-gradient, momentum updates of the teacher, centering or sharpening, and prediction heads, so that the two branches no longer chase a trivial solution in exactly the same way at the same time.

Taking BYOL's two branches as an example, the online branch produces a prediction from one augmented view, and the target branch produces the training target from another augmented view. The output of the target branch stops gradients and is not directly updated by the prediction error in the same step; its parameters are slowly updated by the momentum average of the online branch's parameters. The prediction head in the online branch is responsible for approximating this relatively stable target. The input is still two views of the same sample, and the outputs must still remain consistent, but the roles and update paths of the two branches differ: they are no longer two fully symmetric, simultaneously changing targets.

DINO's student–teacher self-distillation also lets the student match the output produced by the teacher for another view, and uses teacher momentum updates to maintain the target branch. Centering and sharpening further adjust the distribution of the teacher output, preventing training from achieving superficial consistency solely through non-discriminative outputs. Stop-gradient, momentum teacher, prediction head, centering, and sharpening are not a uniform recipe that all methods adopt at the same time; they each serve the asymmetric training structure in specific methods.

Therefore, "no explicit negative samples" does not equal "no anti-collapse mechanism". Judging whether training is healthy cannot rely only on the consistency loss, because a constant representation may perform very well on this metric. More direct diagnostics include representation variance, singular value spectrum, and nearest-neighbor diversity: variance reflects whether each dimension still varies with samples; the singular value spectrum shows how much variation strength remains in the principal representation directions; nearest-neighbor diversity checks whether the neighbors of different inputs all degenerate into similar results. These metrics need to be interpreted together with the training loss to distinguish genuine consistent representations from trivial collapse.

Whether representations obtained from self-supervised pre-training are useful must be tested through transfer tasks; but different evaluation protocols measure different capabilities. Poor linear probing performance only shows that after freezing the representation, the target classes are not well separated by a simple linear layer, and cannot be used to claim that pre-training has completely failed. The representation may still contain information that can be exploited through nonlinear adaptation or parameter updates.

Linear probing freezes the pre-trained model and trains only a linear classifier, mainly measuring whether the representation is already linearly separable. It is convenient for isolating the structure of the representation itself, but may underestimate information that requires fine-tuning to emerge. Full fine-tuning allows all parameters of the pre-trained model to be updated with downstream data, mainly measuring the adaptability of this initialization to downstream tasks; but the results also mix in downstream model capacity, training budget, and optimization settings, so they cannot be fully attributed to the pre-trained representation.

Few-shot or prompt-based evaluation focuses on performance with very little labeled data or specific interaction interfaces, mainly reflecting data efficiency and interface capability. Its results often have high variance and can also be sensitive to the prompt format. Frozen retrieval trains no or few additional parameters and directly finds similar samples based on distances between representations, mainly checking whether the distance space places related objects in nearby positions; the result also depends on the chosen indexing method and similarity metric.

These protocols answer different questions, so a single score should not be taken as a complete conclusion about representation quality. Frozen evaluation emphasizes the structure that can be read out immediately after pre-training, while end-to-end adaptation emphasizes the potential for parameters to continue changing under downstream supervision. If linear probing is weak but full fine-tuning is strong, a reasonable interpretation is that the representation is not necessarily already linearly organized, but the initialization is still adaptable; if frozen retrieval is strong, it suggests the distance space may already have usable structure. Any interpretation should be confined to the corresponding protocol.

To make comparisons valid, all protocols must fix the data split and hyperparameter tuning budget, avoiding mistaking more data or more experimental opportunities for a method advantage. Beyond overall scores, one should also slice by data source, population, difficulty, and out-of-distribution samples to check whether average results mask local failures. Pre-training loss can only monitor whether the automatic objective is being continuously optimized; it cannot replace transfer evidence: a low pre-training loss means the model is better at the pretext task, but does not directly indicate that it can transfer under frozen, fine-tuning, few-shot, or retrieval protocols.

Self-supervised learning can automatically construct targets from raw data, thereby reducing the need for manual item-by-item labeling. But this only changes how labels are produced; it does not make data a free, lawful, or risk-free resource. Training still depends on large-scale raw data, and collecting, storing, processing, and using that data all require governance.

First, having access to a piece of data does not mean having training authorization. Data still involves licensing, privacy, and consent from the data subject; automatically generating targets does not eliminate these requirements. Second, raw corpora need deduplication and contamination checks: duplicate samples change the data distribution; if an evaluation set or near-duplicate copies appear in the pre-training corpus, the model may perform well because it has seen the answers, thereby faking transfer ability. In this case, evaluation scores reflect leakage, not generalization to unseen data.

Automatically generated targets also inherit the problems of the data itself. Biases, malicious content, and uneven coverage of groups in the data will not disappear because of “no human labels”; on the contrary, when the model continuously constructs and optimizes a large number of targets from such data, these patterns may also be amplified along with the training signal. The training target comes from the data, which only shows that the target can be obtained automatically; it does not show that it aligns with product value, nor does it guarantee that different sources, groups, or scenarios receive representations of equal quality.

After scaling up, the bottleneck therefore shifts from “who writes the labels” to “which data can be used, what processing the data has undergone, and whether problems can be located and deleted after they occur.” Governance records must cover at least data provenance, acquisition or inclusion time, processing chains, and deletion capability. Source records are used to judge authorization and coverage; time information helps identify the relationship between training data and evaluation or application time; processing chains describe the deduplication, filtering, or other transformations the data has undergone; deletion capability ensures that after licensing, privacy, or content issues are discovered, the relevant data can be located and handled.

The scale advantage of self-supervision rests on the ability to generate targets in batches, but the larger the scale, the more likely ungoverned issues will be carried into training. Therefore, “saving labels” should be understood as reducing one kind of manual cost, not as exempting data responsibility. Model performance must be evaluated together with authorization, privacy, contamination, content risk, and coverage; it cannot be replaced only by pre-training data volume or training loss.

Training loss was about 0.106, which only shows that, in the candidate set composed of A⁺, B, and C at that time, the model assigned a higher probability to the positive view A⁺. To prove that the representation can actually be used for product matching, it is also necessary to trace the training objective to retrieval performance on unseen products and rule out the possibility that the model relies on shortcuts such as background. This process, from data splitting, training monitoring, frozen retrieval, to counterfactual checking, constitutes the acceptance loop for product matching.

The acceptance inputs include training images split by product identity and query images during the test phase; outputs include top-k product retrieval results, representation variance, and slice metrics broken down by background, camera angle, and category. The first step must be to fix a product-level split to ensure that near-duplicate images of the same product do not appear in both the training and test sets. Otherwise, retrieval success may come from memorization of images of the same product, which cannot prove the model can generalize to never-before-seen products.

During training, A and its augmented view A⁺ are used to construct a positive pair, and positive/negative sample similarities and representation variance are continuously recorded. Similarity reflects whether the model pulls positive pairs together and distinguishes candidates according to the objective; representation variance is used to check whether representations of different products still retain variation. Both are only process evidence: correct similarity ranking does not guarantee that the model relies on the product subject, and normal variance does not guarantee that the distance space is suitable for real retrieval.

Next, freeze the encoder and perform top-k retrieval on never-before-seen products. The query image passes through the encoder to obtain a representation, and the system returns the k closest product candidates according to representation distance. Freezing the encoder means the test stage no longer rewrites representations with product labels, thus directly checking the distance space formed by pre-training. The overall top-k metric must also be sliced by background, angle, and category, because the average can mask that the model is effective only under specific photographic conditions or product categories.

Finally, add two groups of counterfactual controls. “Change the background but not the product” keeps the product subject unchanged and changes only the background; if the representation tracks product identity, the query should still retrieve the same product. “Change the product but not the background” keeps the studio background similar but replaces the product subject; if the model focuses on the subject, it should not judge the two as the same product merely because the background is consistent. These two controls separate product identity from background cues to locate which kind of information the model actually uses.

Results should be interpreted along the entire chain of evidence. If the average top-k is high, but the same product cannot be retrieved after changing the background, it suggests that the model may have learned a studio shortcut rather than a stable product identity. Only when the product-level split has no leakage, the representation has not collapsed, retrieval on unseen products is effective, the performance of each slice is acceptable, and the counterfactual controls support subject tracking, is there evidence that the representation is suitable for the current product matching task. This conclusion cannot be automatically extrapolated to fine-grained color, text recognition, or new capture domains; after the information the task cares about or the data distribution changes, the augmentations must be redefined and the corresponding slices retested.

Self-supervised learning begins with a concrete choice: what raw data to use, and which changes the model should be invariant to. The raw data determines what the model can observe, and the desired invariance determines which differences should be ignored and which information must be preserved. This choice is not background context but the starting point for subsequent training objectives and representation properties.

Next, this learning intent must be turned into a computable automatic objective. A predictive objective constructs subsequent content from existing content as the answer; a masking objective hides part of the sample and uses the original content as the recovery answer; a view objective constructs different views of the same sample and defines the relationship that should hold between them. The construction rules turn raw samples into inputs and targets, and also turn “what we hope to learn” into a task that can be evaluated by a loss.

The optimization stage reduces a proxy loss, not downstream value itself. A decreasing loss indicates that the model is getting better at prediction, reconstruction, or view matching, but it may rely on local cues, background, or other shortcuts. For methods that only require view consistency, a collapse-prevention mechanism must also be added to ensure that representations of different inputs do not degenerate into the same constant. Training monitoring must therefore observe both the objective loss and whether the representations still retain sufficient variation.

After pre-training, the representations need to be transferred to downstream tasks through freezing or fine-tuning. Frozen evaluation examines the structure already directly formed in the pre-trained representations; fine-tuning evaluation examines the adaptability of this initialization under downstream supervision. The two answer different questions, and results must be interpreted under fixed data splits and budgets. Pre-training loss cannot replace these transfer evidence.

Overall transfer metrics can still conceal shortcuts, so validation on real slices is also needed. Splitting results by source, group, difficulty, background, viewpoint, category, or distribution shift can locate the conditions under which the model fails; counterfactual controls change only one factor to check whether the model's output follows the target subject or irrelevant cues. Only when the slices and controls support the expected mechanism can performance improvements be attributed to the intended representation.

The entire chain is also constrained by data governance. Data provenance and licensing determine whether the data can be used; contamination checks prevent evaluation content from leaking into pre-training; objective review judges whether the automatic task aligns with product value. If provenance is unclear, evaluation is contaminated, or the proxy objective is misaligned, reliable conclusions cannot be drawn even if training and transfer numbers are good.

Therefore, the verifiable practice chain is:

Raw data and desired invariance → predictive, masking, or view objectives → proxy loss optimization and anti-collapse → frozen or fine-tuned transfer → real slices and counterfactual checks → provenance, contamination, and objective misalignment audit

Each step in the chain provides conditions for the next and may also introduce new failure causes. What ultimately needs to be verified is not the isolated fact of “whether the loss decreases,” but whether the data, objectives, representations, transfer performance, and governance evidence jointly support the intended use.