Adversarial Robustness: Seeking the worst case within an explicit perturbation set, not claiming the model is “attack-proof”

Why can a classifier that achieves 99% accuracy on a test set have its prediction flipped by an input whose change is almost invisible to the human eye? The key is a fundamental difference between the test set and the attacker’s sampling method: natural accuracy measures average performance near the natural data distribution, whereas attackers do not sample randomly; they actively search along the decision boundary for inputs that most cause the model to make mistakes.

First, look at what the model has learned. High-dimensional models do not necessarily rely only on human-recognizable semantic features during training; they also capture a large number of fragile features that are insensitive to humans but statistically correlated with training labels. These features are numerous in high-dimensional space, enough to support the model achieving high scores on standard tests. The problem is that their “correlation” with labels is only a statistical association within the training distribution, not a true understanding of semantics, and it does not guarantee that they will still hold after the input is deliberately perturbed.

Next, look at why the test set cannot reveal these fragile points. Ordinary testing samples independently from the natural distribution; the vast majority of samples fall in dense regions of the data distribution, far from the decision boundary. The attacker’s approach is completely different: they do not need to change the sample’s semantics; they only need to advance a little along the normal direction of the decision boundary to make the sample cross the boundary. Natural sampling almost never happens to land exactly on these carefully constructed directions, so high accuracy and susceptibility to attack can hold at the same time.

Here we must clarify a common confusion: adversarial examples are not random noise. Random noise perturbs uniformly in all directions, and whether it flips the class depends on luck; adversarial examples are results specially optimized according to the model’s gradient information or query feedback. The attacker can choose to increase the loss function, thereby overturning the originally correct class, or directly raise the output probability of a certain target class to achieve targeted misdirection. Each perturbation moves in the direction that makes the model most uncomfortable, so the effect far exceeds an equal amount of random noise.

Finally, we need to define the scope of this conclusion. Whether an adversarial example constitutes a real-world threat does not depend only on whether the model makes a mistake on some crafted input, but on whether three conditions hold at the same time: whether the attacker can actually realize such a perturbation in the real environment; whether the model output directly triggers a high-impact action; and whether the downstream system lacks an independent verification step to intercept erroneous results. If the attacker cannot apply the perturbation to real inputs, or if the model output is merely recorded without affecting decisions, then the attack success rate in the laboratory will not automatically translate into real-world harm.

At the same time, we should also note the distance between research proxies and real threats. A perturbation measured by pixel norm on an image is only a mathematical proxy for “small input change”; it is not equivalent to all physical attacks or semantic attacks. Printing a sticker, changing the lighting angle, or replacing an object at the semantic level may all change model behavior without producing a large pixel norm, and may still be easily recognized by humans as the same object despite producing a large pixel norm. Pixel norm is only a computable standard in research; it cannot by itself represent the real-world difficulty or harm of an attack.

Why can't the same "robust accuracy" number be directly compared across papers or products? Because the meaning of robustness depends entirely on the threat model defined before evaluation. What is called "robust" can only ever be "robust under a specific explicit constraint"; without that constraint, any single percentage carries no comparable information. The threat model must be determined before attack algorithms and robustness scores; it is the premise for all subsequent measurements and conclusions.

The threat model is jointly defined by five dimensions; if any dimension differs, the resulting numbers are incommensurable.

First is the attack goal. The attack can be untargeted—just make the model misclassify the correct class; it can also be targeted—specify a particular target class and require the model to misclassify it as that class; it can also be confidence manipulation, where the class need not change but only some output probability is suppressed or increased. Different goals have different definitions of "success"; the success rate of untargeted attacks naturally cannot be compared with that of targeted attacks.

Second is the attacker's knowledge of the model. Under a white-box attack, the attacker has the model's full weights and gradients and can precisely compute the direction in which the loss increases fastest; under a score-query attack, the attacker can only repeatedly submit inputs and obtain confidence scores from the model's outputs, using these scores to estimate gradients; under a label-only attack, the attacker cannot even see scores and only knows the final predicted class; under a transfer attack, the attacker does not access the target model at all, but generates adversarial examples on a substitute model they trained themselves, then uses them to attack the target. The more knowledge the attacker has, the richer the information they can exploit and the more the effective budget they can invest differs.

Third is the definition of the perturbation, that is, what "close" means. The L∞ norm limits how much each pixel can change at most; the L2 norm limits the sum of squares of all pixel changes; the L0 norm limits the number of pixels modified; in addition there are perturbation types such as rotation, sticker patches, and semantic replacement that are not measured by pixel norms at all. The same model may collapse under an L∞ attack with very small ε, yet remain robust under an L0 attack, because the two perturbations are fundamentally different in kind.

Fourth is the budget, that is, how much the attacker is allowed to invest. It can be an ε upper bound, the maximum number of queries the attacker can initiate, or the area that a sticker is allowed to occupy in a physical attack. The budget directly determines attack strength: the larger the allowed perturbation magnitude and the more queries allowed, the stronger the attack naturally is, and the lower the measured robust accuracy naturally is.

Fifth is the system boundary, that is, what the attack actually acts on. Some evaluations target only the isolated model itself; others include preprocessing, detectors, rejection mechanisms, and business actions triggered after model output. A sample that is successfully attacked on a bare model may be filtered out by preprocessing after entering the complete system, or may not constitute harm because downstream actions have no actual impact. Testing only the model and testing the complete system measure two different things.

Precisely because these five dimensions vary simultaneously, the robust accuracy reported in a paper is meaningful only after fully specifying "against what goal, what the attacker knows, how the perturbation is defined, how much budget, and where the system boundary lies." An isolated percentage can neither indicate what the model actually resisted nor be used to rank directly against another evaluation.

Why can the total loss still rise noticeably when each pixel is changed only a tiny amount? The answer lies in moving all dimensions simultaneously along the sign direction of the gradient. The Fast Gradient Sign Method (FGSM) makes this mechanism very clear with a two-dimensional numerical example.

Let the input be x = [0.40, 0.70], the model's loss gradient for this true label be ∇xL = [−0.8, 1.6], and the L∞ budget be ε = 0.05. The gradient tells us how each coordinate should change to increase the loss fastest: the first dimension's gradient is −0.8, meaning moving x₁ in the negative direction will increase the loss; the second dimension's gradient is 1.6, meaning moving x₂ in the positive direction will increase the loss. FGSM ignores the magnitude of the gradient and takes only the sign of each coordinate, giving sign(∇xL) = [−1, +1]. Multiplying by the budget ε, the perturbation for one step is [−0.05, +0.05], yielding the adversarial input x_adv = clip(x + ε sign(∇xL)) = [0.40 − 0.05, 0.70 + 0.05] = [0.35, 0.75]. The role of clip is to ensure the final result remains within the valid input range.

Why does this perturbation work? Under a first-order Taylor approximation of the loss, the change in the loss is approximately equal to the dot product of the gradient and the perturbation vector. Substituting the numbers: (−0.8) × (−0.05) + 1.6 × (0.05) = 0.04 + 0.08 = 0.12. Notice that the contributions of the two components to the loss increase point in the same direction—the first dimension contributes 0.04, the second contributes 0.08, and they add rather than cancel. This is exactly the purpose of taking the sign: the sign ensures that each dimension's movement has the same sign as the corresponding gradient component, making every contribution positive and thereby making the loss under the first-order approximation increase monotonically.

A change of 0.05 in a single dimension is negligible, but in a high-dimensional space, thousands of coordinates simultaneously contribute a small, same-direction loss increment, and the accumulated total is substantial. This is why 'only a tiny change per pixel can produce a noticeable rise in the total loss': the attack's power comes from the number of dimensions, not from the size of any individual dimension's change.

It is also worth clarifying the boundaries of this example. FGSM takes only one step and uses a first-order approximation of the loss, so it is fast but does not guarantee reaching the point within the budget ball that truly maximizes the loss. After one step, the loss may have increased, but because of errors in the linear approximation, the class may not have been sufficiently flipped. More thorough attacks use multiple iterative steps, recompute the gradient after each small step, and then project back into the budget ball; this is exactly what PGD does. FGSM can be understood as the special case of PGD with only one iteration. It is suitable for intuitively understanding the core idea of 'moving along the sign of the gradient,' rather than being the endpoint of the strongest attack.

An attack that succeeds in pixel space: how can we verify that it really becomes a risk in the physical world? Traffic sign classification provides a complete example: from digital perturbation to a real sticker, the path in between is a chain of evidence that must be verified link by link; no link can be skipped.

The first step is to define the asset, that is, to clarify what "model error" actually means. In assisted driving, misclassifying a sign affects the vehicle's understanding of that road segment, but the final control still depends on multiple components such as map, trajectory planning, and the driver; the model is not the only source of decisions. This step sets the boundary for all subsequent verification: we care about whether the model error actually propagates into business loss, not merely whether the model makes a mistake on a particular input.

The second step is to examine the model's local boundary in digital space using white-box L∞ attacks, while recording both natural accuracy and robust accuracy. Natural accuracy indicates the model's performance on normal data, robust accuracy indicates its performance under active attack; only together can they describe the model's fragility near the boundary.

The third step is to extend the attack from pure pixel perturbation to transformations common in the real world: rotation, brightness changes, image compression, and different shooting distances. This step tests whether the attack remains effective under various transformations after leaving the ideal condition of "exactly reproducing the original image," thereby ruling out attacks that only hold for a specific pixel arrangement and fail as soon as the environment changes slightly.

The fourth step is where we move to an actual physical sticker. The attacker optimizes the sticker pattern under constraints on area and color, then repeatedly captures images under different printers, cameras, shooting angles, and weather conditions to test it. Why change so many conditions? Because the success of a physical attack must be robust to the fabrication process and the capture environment, not merely effective by chance in one particular capture.

The fifth step is to prevent overfitting to a single environment. The equipment and routes used for final validation must not have participated in the sticker optimization process. If the same camera and the same route are used for both optimization and validation, the sticker may merely fit the imaging characteristics of this particular setup, fail in a different environment, and such "success" cannot be generalized.

The sixth step is to return to the system level. Even if the model really misclassifies the sign, we still need to verify whether the system's multi-source fusion rejects this inconsistent sign—for example, whether a conflict between the map and trajectory information and the recognition result triggers a correction—and whether the model's error actually changes the vehicle's subsequent actions. If downstream components stop the error, then the model-level attack has not translated into actual harm.

The final step addresses the post-deployment situation: after the model is trained or updated, we should not only replay old stickers for validation; instead, we should re-optimize the sticker against the new model using adaptive attacks. Because once the model changes, old attack samples may no longer be effective; only re-optimization can truly test the current model's worst-case performance.

The overall logic of this evidence chain is: digital attack is only the starting point; each step answers the question "does this attack still hold under conditions closer to reality," while the final question to answer is "does the model error actually become business loss?" Skipping any link in between will mistake a phenomenon in digital space for a risk in the physical world.

Why can't we infer from having defended against a particular attack algorithm that we have "defended against all attacks within the same budget"? This figure illustrates the relationship between robustness conclusions and the threat model boundary.

At the center of the figure is a natural input, surrounded by a perturbation set defined by the threat model—for example, all inputs satisfying a certain ε constraint. The interior of the set is the region where attacks are allowed to operate; the exterior is where attacks are not allowed, or are simply outside the jurisdiction of this mathematical definition. The model's decision boundary passes through or near this set, determining which points will be misclassified.

Inside the set, the points found by empirical attacks are merely isolated lower-bound evidence. If a particular attack algorithm succeeds, it shows the model is broken at that point, and robustness is at most as large as the perturbation magnitude corresponding to that point; but another attack algorithm that was not tried might take a different path and cross the decision boundary at an even smaller perturbation. Therefore, the success rate of empirical attacks can only tell us "the model is at least this fragile," not "the model is at most this robust."

The certified radius attempts, conversely, to provide an upper-bound guarantee: within a certain strict mathematical assumption, it claims that no point within this radius will be misclassified. But the certified radius is also bounded by its own assumptions—it only holds for the norm it uses and the conditions on which its proof depends; change the norm or change the attack target, and this guarantee no longer applies. Therefore, in the figure, the empirical attack point and the certified radius together delineate the zone between "how much we can prove" and "how much we have observed," rather than a complete conclusion that covers all attacks.

What truly makes this figure complete is the semantic attacks outside the set. An input that is semantically clearly different—changed object, added occlusion, altered lighting meaning—may not fall within any pixel-norm ε ball at all, and thus completely escapes the constraints of this perturbation set. All robustness conclusions measured inside the set have no say at all about such attacks.

Thus the core relationship the figure expresses is: any robustness conclusion is surrounded by the boundary of the threat model it belongs to. Empirical attacks provide a lower bound, the certified radius provides an upper bound within its mathematical assumptions, and attacks outside the set are completely unconstrained by either. Defending against one attack algorithm can only prove that this particular algorithm did not succeed under this budget; it cannot be generalized to all possible attack paths within the same budget, much less to anything outside the set.

Why are multiple small updates usually stronger than one large step? Because the attack is essentially a constrained optimization problem, and the single-step method is only its crudest approximation. PGD writes this optimization process explicitly: within the perturbation set S, find the point that maximizes the loss.

Each iteration does two things. First, take a small step along the loss gradient direction at the current point, with step size α, direction taken as the sign of the gradient, obtaining xₜ + α·sign(∇L); then project the result back onto the perturbation set S around x. This projection step is crucial: it ensures the perturbation never exceeds the budget allowed by the threat model; no matter where the gradient direction pushes the point, it will finally land inside the set. The entire iteration can be written as xₜ₊₁ = Π_S(xₜ + α·sign(∇L)), where Π_S denotes the projection onto the set S. Compared with FGSM, PGD recomputes the gradient after every step, rather than relying only on one gradient at the starting point, so it can approach a local worst-case point along a continuously corrected path.

Why are multiple steps stronger than a single step? Because the loss surface is not flat. The single-step method relies on the linear approximation at the starting point; once the surface bends, that direction is no longer optimal. The multi-step method re-evaluates the direction after every short segment, correcting it gradually, and can trace a more effective path on the surface.

Two more details significantly affect attack quality. One is the random start: if you always start from x itself, you may repeatedly land in flat local regions and stall; starting from a randomly perturbed point reduces this probability. The other is multiple restarts: repeat the entire optimization process from different starting points, explore different paths, and finally take the most successful result. Together these make the attack closer to the true “worst case.”

These details in turn define the pitfalls of attack evaluation. Insufficient step size, number of steps, or number of restarts will stop the attack halfway, thus overestimating the defense—not because the model is truly robust, but because the attack was not run to completion. Conversely, an overly large step size will make the point bounce back and forth across the constraint boundary and fail to converge. To judge whether the attack is strong enough, look at the convergence curve: when increasing the amount of computation no longer significantly raises the attack success rate, it indicates the attack has approached the capability limit of this algorithm. However, even if the convergence curve flattens, it is still only “the worst case for this algorithm,” not proof of global optimality—this is precisely the lower-bound meaning described by the threat model boundary figure.

Why does adding adversarial examples to training improve the model's robustness within a specified budget? Adversarial training puts "training" and "attack" into the same objective function, causing the two to alternate, rather than training first and then passively testing.

This objective can be approximately written as: min_θ E[ max_{δ∈S} L(f_θ(x+δ), y) ]. It consists of an inner and an outer level. The inner max means that for each input x, under the current model parameters θ, search in the perturbation set S for the δ that maximizes the loss—that is, use a single attack to find a high-loss input for the current model. The outer min means that the model parameters should be updated in the direction that makes these found worst-case inputs also be predicted correctly, thereby reducing the loss. The inner attack and outer update alternate: the attacker continuously finds worst-case points for "the model at this moment", and the model continuously reinforces itself against "the worst-case points at this moment".

This is exactly why adversarial training can improve robustness: ordinary training only makes the model correct on natural samples, and it remains fragile near the decision boundary; adversarial training continuously exposes and patches the gaps opened by the attacker near the decision boundary, which is equivalent to pushing the boundary away from these local worst-case points, making the model more robust inside the perturbation set S.

But this benefit comes with strict preconditions. The inner attack must be strong enough. If the inner loop only uses a very weak attack, the model may find a shortcut—learning to make this particular attacker fail, rather than truly being robust within set S; this is "deceiving the attacker" rather than "becoming robust". In addition, the perturbation set used during training must match the attacks faced at deployment: if training only defends against small pixel perturbations under L∞, but the real threat is printed stickers or semantic substitutions, then the training gains will not generalize to these attacks.

Adversarial training also has costs. It requires repeatedly running the attacker, which significantly increases computational consumption, and it usually sacrifices some natural accuracy—to be robust within the perturbation set, the model may give up some marginal correctness on the natural distribution. To judge whether it is worth it, one cannot only look at the single-point increase in robust accuracy; instead, compare it with a baseline of the same capacity, same data, and same computational budget, and then decide based on the target risk of the specific business: if a worst-case scenario would cause high-impact losses once it occurs, then paying these costs is reasonable; if the errors themselves are insignificant, then sacrificing natural accuracy for robustness may not be worth the cost.

Gradient masking describes a false sense of security: the defense makes the gradients obtained by the attacker useless, so the attack appears to fail, but the underlying decision boundary of the model remains fragile. It does not actually improve robustness; it only cuts off the gradient signal that the attacker relies on.

This masking usually comes from three types of operations. Non-differentiable operations prevent Backpropagation from passing through, so directly backpropagating yields meaningless gradients; random operations introduce random perturbations to the same input, making the gradient from a single backpropagation pass unable to reflect the true surface; saturating operations push outputs into flat regions of the activation function, where gradients approach zero. Whichever the case, an attacker moving along these contaminated gradients will stay in place or wander, hence the “attack failure”. But the decision boundary has not become more solid because of these operations; it is merely no longer exposed to the attacker through gradients.

Several signs can identify this false sense of security. A one-step attack is instead stronger than a multi-step attack, indicating that the multi-step iteration is led astray by bad gradients; a black-box transfer attack is stronger than a white-box attack, indicating that the gradients obtained inside the model are worse than those estimated from elsewhere; as the perturbation budget is continuously increased, robust accuracy barely drops, indicating that the evaluation has not touched the true decision boundary at all; after removing randomness, the attack suddenly becomes stronger, indicating that randomness is what makes the attack fail, not that the boundary itself is robust.

What really tests a defense is an adaptive attack. The attacker no longer applies a default attack library; instead, it first understands the defense mechanism and then designs the attack specifically: replace non-differentiable operations with differentiable approximations, use the expected gradient from multiple samples instead of a single stochastic gradient, use straight-through estimation to let the gradient pass through non-differentiable layers, or simply abandon gradients and switch to black-box methods such as decision queries or score queries. Once the attack is adapted to the defense itself, the “success” brought by gradient masking often disappears quickly.

Therefore, for evaluators, the conclusion is clear and strict: you must know the complete inference pipeline of the model under test—from input preprocessing, through intermediate modules, to output postprocessing—and design attacks against the specific defense mechanisms in it. Using a default attack library to test a carefully designed defense yields a high robustness score that is not evidence of robustness, but only evidence that the attacker has not used the right method.

Does a certified radius of 0.3 mean that real photographs are safe no matter how they are altered? No. To answer this question, you must first distinguish what empirical robustness and certified robustness are each saying; they answer two different questions.

Empirical robustness means: under the attacks we have already run, we have not yet found a failure sample that flips the prediction. Its value is providing lower-bound evidence—the model has not been broken at least within the search range of these specific attacks. But it can never rule out “the existence of a stronger or different type of attack that we have not yet run.” The question empirical robustness answers is “I searched and did not find a failure,” not “no failure exists.”

Certified robustness, in contrast, attempts to provide a mathematical guarantee: within specific norms, radii, and model assumptions, the prediction will not change no matter what attack is used. For example, methods such as randomized smoothing can provide a model with a certification of an L2 radius, proving that within the ball of this radius there is no perturbation that can flip the class. This is a conditional mathematical conclusion—it holds absolutely, but only under the conditions it states.

The difference between the two is crystallized in the question of that 0.3. What a certified radius of 0.3 guarantees is: under the norm definition it uses, within the model assumptions its proof relies on, and inside the ball of radius 0.3, the prediction is unchanged. It does not cover changes such as rotation, sticker occlusion, or semantic substitution that fall outside that norm ball, nor does it cover implementation-level errors—such as a bug in the preprocessing code or a mismatch between the deployment environment and the training environment. Most changes in real photographs do not occur according to this norm, so “radius 0.3” definitely does not mean “real photographs are safe no matter how they are changed.”

Equally important, looking only at the average radius can mask the true situation. A model may have a large certified radius for most samples, yet have no guarantee at all for a few high-risk samples. Therefore, when reporting, one should give the proportion of certifiable samples, the distribution of radii, and the natural accuracy, rather than only reporting an average radius. The average radius averages out those samples that have no guarantee at all, misleading people into thinking the overall system is safe.

This distinction ultimately returns to an engineering judgment: even if certified robustness provides a rigorous mathematical guarantee, high-risk systems still require system-level redundancy and monitoring. A certification guarantee covers only perturbations within the threat model, while sources of failure in reality extend far beyond this scope. Between what a mathematical model can prove and the risks a system must actually bear, there is always a gap that must be filled by engineering means.

Why can an L∞ constraint both allow strange textures visible to the naked eye and exclude slight rotations that humans do not care about? The root cause is that pixel-wise norm distance measures “closeness in coordinate space,” while human perception, physical realizability, and task semantics measure something else. There is no alignment between the two sets of metrics.

The advantage of a per-pixel budget is that it facilitates optimization and comparison. It provides an attack with a computable search range, allowing different methods to be evaluated under the same yardstick. But its disadvantage is equally obvious: it is inconsistent with perception. A slight translation moves many pixels away from their original positions; computed under the L2 or L∞ norm, the distance can be very large, yet for an observer the label has not changed at all, and semantically it is still the same object. Conversely, a carefully constructed texture perturbation may change each pixel only slightly and have a small norm, but because it forms a regular high-frequency pattern, it is clearly visible to the human eye. Thus two misalignments in opposite directions emerge: a change with a large norm may be semantically identical, while a change with a small norm may have already destroyed the semantics.

This shows that the norm is only one of many possible threat models, not a default choice that is “closer to reality.” Real evaluation should expand the types of perturbations to include natural corruptions (noise, blur, compression), geometric transformations (rotation, translation, scaling), style changes, patch stickers, semantic rewriting, and task-domain-specific constraints. Each category characterizes a different mode of failure and corresponds to a different source of real-world attacks.

An important conclusion from this is that different attack sets are not mutually substitutable “stronger versions.” An L2 attack is not an upgrade of an L∞ attack, and a patch attack is not an enhanced version of a pixel attack; they are different threats, each covering a different region on the decision boundary. Being robust on one set does not imply robustness on another set. Therefore evaluation should combine multiple attack sets and report results separately, and clearly state which spaces are still not covered, rather than using a single set's number to represent overall security.

When it is impossible to guarantee that a single classifier will not be compromised, the way to reduce accident consequences lies not in the model itself but outside the model: use system-level redundancy, rejection, and loss limiting to block the path between “model error” and “irreversible loss.” The goal shifts accordingly—no longer “the model never errs,” but “errors do not easily penetrate to irreversible loss.”

Redundancy means having multiple mutually independent pieces of evidence check one another. Independent sensors can observe the same object through different physical channels; when one channel is attacked and outputs an anomaly, conclusions from other channels can correct it. Rule constraints turn prior knowledge from the physical world or business logic into hard constraints, filtering out outputs that clearly violate common sense. Temporal consistency uses continuity between consecutive frames or consecutive moments to identify anomalies that change abruptly within a single frame and cannot be sustained over time. Anomaly detection monitors the distribution of inputs and outputs, detecting cases that deviate from normal patterns. Confidence calibration makes the scores output by the model closer to true probabilities, so that when the score is too low it can be reliably rejected. Safe degradation specifies: when evidence is insufficient or the conclusion is untrustworthy, the system falls back to a more conservative, lower-risk default behavior rather than continuing to execute the original action. For high-impact actions, it requires multiple pieces of evidence to converge or human confirmation before execution, not betting the consequences on a single output from a single model.

Loss limiting continues to play a role after an error occurs. Limiting the magnitude and frequency of a single action means that even if an incorrect output occurs, its consequences are confined within a recoverable range and will not cause irreversible damage at once. Recording auditable inputs and versions ensures that afterward one can trace which input and which model version led to the error. Rapid rollback capability ensures that once a problem is found in a deployed model, it can immediately revert to a known-safe older version, shortening the duration of the error.

At the same time, we must recognize the boundaries of these controls: they themselves may also be attacked in a coordinated way. An attacker may not merely fool a single classifier, but simultaneously poison the inputs of multiple sensors, bypass anomaly detection, or exploit vulnerabilities in the degradation logic itself. Therefore, these system-level controls also need to undergo end-to-end AI Red Teaming—treating the entire system as the attack target, rather than testing only the model components within it.

Ultimately, the essence of this approach is to redefine success: not to pursue absolute correctness of any single component, but to ensure that when any component is breached, the consequences of errors do not easily penetrate layer upon layer of defenses and turn into irreparable loss.

What would be concealed by reporting robust accuracy at only a single point, ε = 8/255? It compresses the entire risk curve into a single number, making it impossible to see how the model degrades under different attack strengths. A model that happens to pass at this point may collapse rapidly under a slightly larger budget; it may also have already failed under a smaller budget, with this point just not exposing it. Therefore evaluation should report a curve, not an isolated point.

A complete robustness evaluation must provide multiple sets of numbers at the same time. Natural accuracy is the baseline, indicating the model's performance without attack; robust accuracy at different ε values forms a curve that varies with budget, showing how robustness decays as attack strength increases; attack success rate shows, from the attacker's perspective, to what extent the attack can succeed; the convergence corresponding to the number of steps and restarts shows whether the attack actually ran to completion rather than stopping early. In addition, targeted and untargeted attacks must be distinguished and reported separately, because they define success differently; different knowledge assumptions such as white-box, transfer, and query must be covered, because they correspond to different attack strengths; performance under different norms and natural corruption must also be included, because a single norm cannot represent all threats.

Reporting cannot give only the global average. Break down by class, by device, and by environment to look at the worst-case slices—a certain minority class may be almost completely compromised, the robustness of certain devices or certain environments may be far below average. Averages dilute the most vulnerable slices, and real accidents often happen precisely in these worst-case slices. At the same time, confidence intervals should be provided so that the uncertainty of the numbers is also expressed, rather than pretending they are exact values.

The correctness of the evaluation process must also be audited. The attack code and model preprocessing must be checked together: whether the pixel scale is consistent, whether cropping and normalization are correctly applied before and after the attack, whether the random seed is fixed—any error in these details could cause the attack to be underestimated or the defense to be overestimated. In addition, the adaptive attack set used for final evaluation should be hidden from outsiders to prevent defense methods from repeatedly tuning parameters on the public attack set and overfitting—if a defense appears robust because it 'knows the attacks used for testing,' then its performance against unknown attacks is not credible. Hiding the evaluation attack set is to ensure that evaluation results reflect the ability to face unknown threats, not the memorization of known exam questions.

By connecting the previous parts into a causal chain, one can see how this concept moves from the initial problem all the way to verifiable practice.

The starting point of the chain is the definition: clarify what asset is to be protected, what the attacker's goal is (untargeted, targeted, or confidence manipulation), how much knowledge the attacker has (white-box, query, label-only, or transfer), and what set of perturbations is allowed. This step pins the meaning of “robustness” to a specific threat model; without it, all subsequent measurements lose comparability.

On this basis, first establish baselines: measure natural performance and performance under random corruption. Natural performance answers “how good is the model normally,” and the corruption baseline answers “how stable it is under unrelated, non-targeted degradation.” Only with baselines is there a reference for comparing the effect of adversarial attacks.

Next is finding counterexamples. Starting from one-step FGSM, progress to multi-step, multi-restart PGD to actively search for the worst-case inputs within the budget that make the model fail. The counterexamples found provide lower-bound evidence for robustness—we can at least prove that the model is fragile at these points.

But whether counterexamples are credible depends on whether the attack is actually run to completion. Therefore, check the attack's convergence and whether gradient masking is present: if multi-step is worse than single-step, or black-box transfer is worse than white-box, it indicates the attack has been misled by bad gradients, and the measured “robustness” is an illusion. Only results obtained with adaptive attacks designed for the defense mechanism can serve as valid evidence of vulnerability.

Next is hardening. Use a strong inner attack for adversarial training, or select defense schemes accordingly, so that the model is truly robust within the perturbation set rather than learning to deceive the attacker. The effectiveness of this step must then be verified by retesting with adaptive attacks, transfer attacks, and attacks outside the set, because success inside the set does not automatically generalize outside the set.

The parts that cannot be guaranteed at the model level are left to the system level as a backstop: add redundancy, rejection mechanisms, and loss limits so that model errors do not easily penetrate into irreversible losses.

Finally, continuous reporting: for the parts with guarantees within the scope, clearly state the boundary of the guarantee; for the residual risk outside the scope, also report it truthfully, rather than using a single number to conceal the uncovered space.

This chain also needs a unified verification layer to judge whether each step's changes really bring benefits. During verification, fix the inputs—the same batch of samples, the same preprocessing and permission boundaries—and observe input hashes, slice labels, and rejection reasons to ensure that the same data is being compared. Mechanistically, change only one core variable and lock all other configurations, observe key intermediate states and the first point of deviation from expectations, thereby attributing the benefit to that variable. On the output side, use the same set of acceptance rules and resource budget to measure, comparing stratified differences in quality, cost, latency, and failure rate, rather than looking only at a single metric. Finally, keep a control group that does not enable the target mechanism as counter-evidence, and check whether the benefit reproduces stably across different samples and random seeds. Only benefits that pass this controlled comparison are truly brought by the mechanism being verified, not by accidental results of data or randomness.