Model Interpretability: From Input Attribution to Causal Intervention, Distinguishing Story, Evidence, and Mechanism

Interpretability is not a property that can be defined apart from its purpose. The same explanation of how a model makes a decision means completely different things to engineers, auditors, users, and researchers, and should therefore be judged by completely different standards.

Engineers usually want an explanation in order to locate the source of a failure and modify the model. They care about: for a given input, which part of the computation is primarily responsible for the wrong output, and whether changing it can improve behavior. Auditors need not repair clues but rather evidence and attribution of responsibility—whether the model made a decision because of a protected attribute or a shortcut unrelated to the task, and whether that decision can be traced and explained. Users need actionable reasons: why this application was rejected, why this content was judged to be in violation, and what they can change to influence the outcome. Researchers, by contrast, want to know what computation the model actually performs internally, rather than a post hoc assembled, plausible-sounding narrative.

These different purposes correspond to several pairs of concepts that need to be distinguished. A local explanation targets a single input and answers “why this particular decision is the way it is”; a global explanation summarizes behavioral patterns across the model's entire data distribution and answers “which factors the model depends on overall.” A post hoc explanation fits an already trained model and approximates behavior by observing statistical associations between inputs and outputs; a mechanistic explanation attempts to describe the model's internal causal computation, indicating which components participated in the decision and in what way. The former is low-cost and broadly applicable but does not guarantee that it reflects internal mechanisms; the latter aims to describe the actual process but is far more difficult to obtain.

The reason we should first ask “who the explanation is for and what decision it supports” is that, without specifying the use, there is no standard for a “good explanation.” A localization map that is useful to engineers may be meaningless to users because it contains too many internal terms; a plain-language account that reassures users may mislead researchers because it conceals key mechanisms. Readability, faithfulness, stability, completeness, and acquisition cost constrain one another: explanations closer to the true mechanism tend to be harder to read and more expensive, while accounts easier for ordinary people to accept tend to deviate further from the internal computation. Any artifact that claims to be “an explanation of the model” needs to first state which audience and which decision it serves before it can be judged as adequate.

In linear models, "how much each input variable contributes to the final score" is a question that can be answered precisely, and the derivation process of the answer can directly reveal a premise that all subsequent attribution methods rely on: the choice of baseline.

Consider a scoring function s = 2x₁ − x₂ + 0.5. Given input [3, 1], if using the zero vector as the baseline, i.e., asking "compared to the case where all inputs are 0, how much did each variable change the score", then the contribution of x₁ is the coefficient times the value, i.e., 2 × 3 = 6; the contribution of x₂ is −1 × 1 = −1; the bias term 0.5 is recorded separately as a constant. Adding the three gives 6 + (−1) + 0.5 = 5.5, which exactly equals the total score obtained by directly substituting [3, 1] into the scoring function. The zero-baseline decomposition is so clean because a linear function is independent with respect to each variable: the contribution of a term is the product of its coefficient and that variable's value, with no cross terms.

But this "exactness" is conditional. If the baseline is changed from the zero vector to the mean [2, 2], i.e., asking "compared to a typical input, how much does each variable deviate", then the relative contribution of x₁ becomes 2 × (3 − 2) = 2, the relative contribution of x₂ becomes −1 × (1 − 2) = +1, and the baseline itself has score 2 × 2 − 1 × 2 + 0.5 = 2 + 0.5 = 2.5. The new decomposition is 2 + 1 + 2.5 = 5.5; the total score is unchanged, but each term's value changes, even the sign of x₂ flips from negative to positive. Both decompositions are "correct" because they are answering different counterfactual questions: one asks "what did this variable bring starting from zero", the other asks "how much does this variable deviate relative to the average level".

This phenomenon can be summarized in a general form: f(x) = f(x₀) + Σ(contributions of each variable), where x₀ is the baseline. The left side is the complete prediction, the first term on the right is the prediction value of the baseline input itself, and the summation terms are the incremental contributions of each variable from the baseline to the current input. As long as the function is linear, this equation holds strictly for any baseline; changing the baseline merely redistributes the same total score between f(x₀) and the contribution terms.

The problem is that real deep networks are full of nonlinearities and interactions between features, and the above decomposition is no longer unique. Once there is a cross term such as x₁ × x₂, it is impossible to assign the effect of the cross term uncontroversially to either x₁ or x₂; different attribution methods will give different answers, and these differences ultimately stem from the different implicit counterfactual references they rely on. Therefore, the baseline is not a technical detail that can be ignored casually, but a semantic choice that must be explicitly specified in attribution problems: you are asking "relative to what" the contribution is. Choosing a zero baseline, a mean baseline, or a specific normal sample all yield equally legitimate numerical values, but they answer different questions.

When a word is marked as the brightest on a heatmap, people may easily blurt out, “The model responded this way because of this word.” Such statements need to be treated with caution, because they mistake an input-level sensitivity estimate for a description of the model’s internal reasoning process, and the two are not the same thing.

Common methods such as gradient attribution, Integrated Gradients, occlusion, and SHAP output local contribution estimates of input features for the current prediction. Their common mechanism is: around a given input, observe how the output changes as the input changes, and then allocate the change to each feature. Gradient methods look at the instantaneous sensitivity of the output to each input dimension; Integrated Gradients accumulates gradients along a path from a baseline to the current input, trying to make the allocation satisfy additivity; occlusion directly deletes or masks a word or a block of pixels and sees how much the prediction score drops; SHAP is based on the Shapley value from game theory and takes a weighted average of each feature’s marginal contribution across all possible subsets. They make different assumptions about the baseline, about how to linearize nonlinearities, and about whether features are independent, so the magnitudes and signs of the values may disagree with one another.

Even if the values agree, a heatmap is still not a reasoning trajectory. There are at least three reasons. First, correlated features can substitute for one another: if the model sees two synonymous cues, “terrible” and “bad,” and one is masked out, the other immediately fills in; the masked word shows zero contribution when viewed alone, but it is not irrelevant. Attribution records the increment on whichever feature happens to survive, rather than on the abstract concept the model actually relies on. Second, occlusion creates out-of-distribution inputs. Deleting a word no longer yields a natural sentence, and when the model sees such a malformed input it has never encountered, the change in behavior may reflect its stress response to the anomaly rather than the word’s role in a normal sentence. Third, sensitivity and sufficiency are not the same thing: a word being scored highly only means changing it will affect the output; it does not mean that the word alone is sufficient to cause this output, nor that it is the link in the model’s decision chain.

Therefore, treat attribution heatmaps as a clue that needs further verification, not as a script for a psychological story. You can try multiple baselines to see whether the conclusions are stable, use various perturbation methods to cross-check, check whether the sign of the contribution matches intuition, and run sufficiency and necessity tests: give only this word—does the model still make the same judgment? Remove this word—does the model change its judgment? Only when these pieces of evidence corroborate one another can you say with some confidence that a particular input feature actually drove this prediction. Attribution methods answer the counterfactual question of “what happens if you change things near the input,” not the mechanistic question of “how the model internally reasons step by step.”

Going from an attribution heatmap to a credible causal conclusion requires a progressively tightening chain of evidence. The sentiment classifier example can fully demonstrate this process.

Suppose a movie review is classified as positive, and the attribution map shows that the actor name “Li Mou” contributes abnormally high to the positive score, far more than the adjectives in the sentence that truly express sentiment. This alone is only a clue: it shows that changing this name would significantly change the prediction, but it does not prove that the model is “using” this name to make the judgment. To reach a causal conclusion, several different types of verification need to be performed in sequence.

The first step is to construct counterfactuals and observe system changes. Keep the rest of the sentence unchanged and replace “Li Mou” with several other actor names to see whether the prediction systematically flips as the name changes. If the positive score drops consistently when replaced with other actor names and rises again when switched back to “Li Mou”, this indicates that this specific name, rather than the “actor name” position, is playing a role. This step moves from “correlation” to “manipulability”.

The second step is to rule out confounding. Check the co-occurrence frequency of “Li Mou” with positive labels in the training data, and also rule out other factors such as tokenization method, name length, and word frequency that could simultaneously affect the model. If this name co-occurs overwhelmingly with positive reviews in the training set, then the model has a data-level source for learning this shortcut; conversely, if the co-occurrence is not obvious, it is necessary to suspect whether other hidden variables are at work.

The third step is to check the persistence of information inside the model. Train a linear probe to predict which name appeared in the input from intermediate-layer representations. If the probe can read out the name identity with accuracy significantly above chance, this indicates that name information is indeed encoded in these layers. But this step only shows “information exists”, not that downstream computation reads it. Existence and use are two different things.

The fourth step is internal causal intervention. Locate the specific features or neurons that are sensitive to the name, and perform activation replacement or ablation experiments: replace the neuron’s activation under the “Li Mou” input with its activation under another neutral name, or directly set it to zero, and see whether the positive score changes in the expected direction. If intervening on these internal units can controllably change the output, an internal-to-behavior causal link is established.

The fifth step is to prevent single-instance overfitting. Repeat the verification of the above conclusions across different sentence templates, different names, and different negative contexts to confirm that it is not a special case that only holds for a specific sentence.

The final step is closing the loop. After removing this shortcut from the data, or retraining with regularization, confirm that the corresponding internal circuits and the behavioral shortcut weaken together. If the model no longer uses the name and the internal components no longer encode the name, a complete causal chain from data source to internal mechanism to output behavior is formed.

This example illustrates the logic of increasing evidence strength: visualization provides candidates, counterfactuals provide manipulability, probes provide information existence, internal interventions provide causality, cross-sample validation provides robustness, and finally the loop is closed by modifying training. Each step narrows the gap between “story” and “mechanism”.

A recurring misconception in interpretability research is to treat "the information can be read out" as "the model actually uses this information." A diagram can arrange the strength of different evidence types into a ladder, visually showing at which stage this misconception occurs.

Figure 1 depicts an explanation evidence ladder, from low to high: visualization correlation, input counterfactuals, linear probes, internal ablation and activation replacement, and then cross-sample prediction and mechanism verification. Each level provides a stronger causal commitment than the previous one.

The bottom of the ladder is visualization correlation. Attribution heatmaps, attention weights, and feature visualizations all fall here. They show that a certain input region or a certain internal unit has a statistical association with the output, but association does not mean causation. One step up is input counterfactuals: actively change the input and observe the output change; this can establish "input-level manipulability", but it remains on the behavioral surface. Further up is linear probes: train a simple classifier to see whether an intermediate-layer representation can predict the target information. A probe's high accuracy answers "whether this layer's representation encodes that information"; it does not at all answer "whether downstream computation reads and uses it." Information can exist without being used, just as a note with the answer written on it can remain in a drawer.

To cross the gap between "existence" and "use", one must enter internal intervention: directly ablate a certain neuron, or replace it with activations from another set of inputs, and observe whether the output changes in the expected direction. Only this step begins to establish a causal link between internal components and behavior. The highest level is cross-sample prediction and mechanism verification: take a hypothesis about a certain internal circuit to new samples outside the training distribution to predict whether it holds, and test whether the circuit and behavior change together after modifying the training data or model. Being able to make correct predictions in new situations is a sign that a mechanistic explanation has real explanatory power.

The methodological claim conveyed by Figure 1 is: interpretability research should first use observation to propose hypotheses, then use intervention and prediction to test the hypotheses. Observation can only produce candidate stories; intervention can constrain stories, and cross-sample predictive power is the final arbiter distinguishing "fitted narratives" from "true mechanistic descriptions." Treating probe accuracy directly as evidence that "the model uses this information" is equivalent to stopping at the third level of the ladder while claiming to have reached the causal level.

Suppose you train a linear probe at some intermediate layer, and it can predict the gender of input examples from the activations with 99% accuracy. Many people immediately conclude: the model used gender when making its decision. This inference skips a key step, because a linear probe proves that "information exists in the representation," not that "downstream computation read and used this information."

To understand why, we need to look at a property of high-dimensional representations. The intermediate activations of a deep network are usually vectors with hundreds to thousands of dimensions, and linear classifiers have a large fitting capacity in such high-dimensional spaces. Much information completely irrelevant to the task may also be linearly encoded because of the redundancy of the representation space—while the model may be processing syntax and semantics, it incidentally encodes many attributes such as gender, age, font, and position into the vector, even if these attributes have nothing to do with its final behavior. A probe's high accuracy may reflect that the representation "happens to contain" this information, rather than that the model "relies on" this information.

More troubling is that the probe itself may, through its own capacity, extract weak signals that the model never used. A sufficiently strong probe can amplify directional components with extremely small probability, turning weak correlations in noise into a nice classifier. Therefore, probe accuracy alone cannot distinguish between "information the model actively encoded and relies on" and "signals the probe forcibly dug out of the redundant space."

To make a more reliable judgment, you need a set of control experiments. First, control the probe's complexity: use the simplest form (such as a linear probe) and compare against baselines with random labels and random representations to confirm that its success is not simply due to fitting capacity. Second, perform intervention tests: modify the representation along the direction learned by the probe—for example, remove or replace the gender direction in the representation—and observe whether the target behavior changes accordingly. If changing this direction has no effect on downstream predictions, then no matter how accurately the probe reads, it cannot prove that the model used this information.

When reporting results, "readability" and "causal use" must be stated separately. "This layer can linearly read out gender" and "This layer uses gender to make decisions" are claims of two different strengths; the former is a correlational statement, the latter a causal statement. There is also another boundary to note: removing the information found by the probe from the representation does not guarantee that all nonlinear encodings have been removed. The probe only finds the linear structure of its own kind; the representation may still contain nonlinear encodings that the probe did not see. Therefore, evidence that "behavior is unchanged after removal" also needs to be interpreted in conjunction with multiple probe types and checks along multiple directions.

To determine whether a particular internal component—such as an attention head—is actually responsible for transmitting a piece of information, rather than merely being coincidentally active with that information, observation alone is insufficient. One needs to use intervention experiments to directly alter its behavior and then observe how the output changes. Activation replacement and path patching are two core tools.

The basic approach of activation replacement is to prepare a pair of inputs: a “clean” input and a perturbed input. Suppose the perturbed input deletes or alters key information, causing the model output to be degraded. Then, at a certain layer, replace the activations from the degraded run with activations produced by the clean input at the same layer, and observe whether the output recovers. If replacing the activations of a particular layer can significantly restore the output to the clean state, this indicates that this layer originally bore the causal responsibility for transmitting this information; if after replacement the output shows no improvement, then although this layer may contain relevant information, it is not on this causal pathway.

Path patching refines this idea to the component level. Instead of replacing the entire layer, it replaces one attention head, one MLP neuron, or one connection at a time, to see which part is the necessary channel for restoring the output. Through component-by-component patching, one can locate a necessary path from input to output. Ablation experiments are complementary: remove a component and see whether the output is degraded, answering necessity; conversely, activation addition experiments (forcibly injecting a component's activation when it is originally inactive) see whether the target behavior appears in the output, answering sufficiency. Only together do necessity and sufficiency constitute a complete account of “this component causes this behavior.”

Such interventions share a common pitfall: replaced or removed activations may push the model into an out-of-distribution state it has never seen. After a component is ablated, the remaining components may trigger abnormal compensation, or the overall computation may slide into an unnatural region; the behavior changes observed at this time do not necessarily reflect the component's true role under normal input. Moreover, redundancy is pervasive inside models—multiple components perform overlapping functions, and removing one alone often produces only a weak effect because other components immediately fill in. This can cause false negatives of “unimportant” to appear frequently.

To resist these disturbances, multiple methods need to be used together: apply interventions of different strengths and draw dose curves to observe whether the output changes monotonically with intervention strength; set up control positions to confirm that the effect occurs only when intervening at the target position, while intervening at other nearby positions is ineffective; finally, replicate the same conclusion on different samples to rule out single-case coincidence. Only when a component's causal role is stably established under interventions of multiple strengths, multiple positions, and multiple samples can it be said that it is indeed transmitting this information.

A neuron strongly activated by seemingly unrelated inputs such as "DNA sequence," "quotation marks," and "German text" does not mean the neuron has no semantics, nor that the model interior is incomprehensible chaos. It points to a fundamental fact about representation: a single neuron is not a natural concept unit.

The representation space of deep networks has finite dimensionality, while the number of concepts the model needs to encode far exceeds the number of dimensions. An economical approach is to superimpose multiple features into the same set of dimensions (superposition)—letting the same neuron participate in representing multiple features simultaneously, distinguished through how different features combine across different inputs. Thus a single neuron appears "polysemantic," when in fact it is sharing capacity among multiple features. Interpreting this as a failure of "one neuron corresponds to one concept" is a mistaken perspective.

Sparse autoencoders are one tool for addressing this superposition. Their goal is to learn a reconstructor that decomposes activations into more numerous, sparser features: using sparse feature vectors whose dimensionality far exceeds the original to reconstruct the activations of each layer, so that each feature direction corresponds as much as possible to a single, relatively monosemantic concept. The input is the activation vector of some layer; the output is the reconstructed activations and a set of sparse feature coefficients. The training constraint is to make the reconstruction as faithful as possible to the original activations while activating as few features as possible. There is a trade-off between sparsity and reconstruction fidelity: the sparser and more monosemantic the features, the more distorted the reconstruction tends to be; the more accurate the reconstruction, the more the features tend to revert to polysemanticity.

This method has an important boundary. Feature meanings are not automatically provided; they are manually named by examining the input samples on which each feature activates most strongly. Naming depends on the choice of examples and the interpreter's judgment, and may be influenced by automatic labeling bias or subjective priors. The so-called "proportion of interpretable features" also depends heavily on threshold settings and labeling methods, and is not an objective, fixed number. The most crucial point is that even if a sparse feature appears to correspond to a clear concept, one cannot automatically infer that the model's behavior is controlled by it. To confirm whether this feature actually drives a particular output, one must still return to intervention experiments: artificially activate or suppress the feature and see whether behavior changes as expected. Sparse features reduce the problem of "neuron polysemanticity" to "relatively monosemantic features," but the gap from readability to causality still exists.

Attention heatmaps are among the most accessible and most easily misread interpretability tools. When a token is assigned a high attention weight, people often assert from this that “the model is attending to this word,” and then infer that “this word determines the output.” But this inference does not hold mechanistically, because several operations that change signal strength lie between attention weights and the final output.

The true meaning of attention weights is routing preference: they indicate which other positions the model tends to read information from when computing the representation at the current position. But how much of the read information actually enters the output depends on a series of subsequent computations. First, the attention output is the result of value vectors weighted and summed by the weights. Even if a token has a high weight, if its corresponding value vector has a near-zero magnitude or carries information irrelevant to the current task, then the high weight times the near-zero value still injects very little information. Second, after multiple attention heads are summed, a high weight in one head may be canceled out by the contributions of other heads. Third, residual connections and subsequent MLP layers further process and may even overwrite the signals from earlier layers: later layers are fully capable of rewriting the contribution of a high-attention position.

Conversely, a token with a low attention weight can also be crucial to the output. If it propagates along a high-magnitude value path, the weight is small but the value is large, and their product can be considerable; or it may bypass the attention path through a residual connection and directly affect subsequent computation. Therefore, there is no monotonic correspondence between attention weights and final contribution.

The correct way to use attention maps is to treat them as a starting point for generating structural hypotheses, not as conclusions in themselves. When you see a stable attention pattern from a head toward a certain type of token pair, you can put forward the hypothesis that “this head may be transmitting a certain type of relation,” and then test it by combining the content of value vectors, the output projection matrix, the actual contribution of residual connections, and ablation experiments. Only when ablating or replacing the head actually changes the output do the patterns on the attention map gain causal support. Directly equating “high attention weight” with “this token contributes greatly to the output,” that is, treating “attention is explanation” as a universally valid proposition, is an overinterpretation that must be explicitly rejected.

When a model fluently outputs a reason like "because A, therefore B," it is easy to think that it is reporting its own internal reasoning. But there is no necessary correspondence between natural language reasons and the actual computational trace; the two need to be treated separately.

The reasons a model generates are textual products shaped jointly by the prompt and language priors. During training, language models learn "what kinds of reasons people expect," so they tend to produce explanations that are socially conventional and coherent rather than to recite internal activations or computations word for word. This leads to two observable phenomena. First, when the answer is exactly the same, changing the prompt style or rephrasing the question can make the model give significantly different reasons—if the reason were truly a faithful readout of a fixed internal computation, it should not drift so much with the prompt. Second, the model can also fabricate a seamless, coherent story for a wrong answer; the fluency of the reason is independent of the correctness of the answer. This shows that, to a large extent, generated reasons are post hoc rationalization packaging of "the answer already given," rather than a recounting of "how the answer was reached."

This distinction has direct practical implications. For user communication, natural language explanations still have real value: they translate complex judgments into language users can understand and respond to, helping to establish actionable dialogue. But treating them as mechanistic evidence conflates three different kinds of things. An "understandable explanation" is a lay paraphrase of the outcome, serving communication; "decision-grounds evidence" needs to be demonstrated experimentally that some factor actually changed the outcome; "internal mechanisms" are the true causal computational description inside the model. The three cannot substitute for one another.

To judge whether a natural language reason is faithful to the model's actual behavior, the only reliable way is to test it with interventions: manipulate the kind of factor mentioned in the reason and see whether the output changes as the reason predicts; hide or remove information the reason relies on in the input and see whether the reason and the answer change accordingly; construct counterfactual questions and see whether the model still gives responses consistent with the reason. Only through these external tests can a natural language explanation be upgraded from "sounding plausible" to "actually reflecting what the model depends on." Chain-of-thought text can be treated as an inspectable behavioral clue, but it is not a computation log.

A visually attractive heatmap does not by itself constitute a good explanation. Judging whether an explanation is reliable requires a set of evaluation criteria that can make it subject to failure, rather than only checking whether it matches intuition. Stability, faithfulness, completeness, and falsifiability are the most basic dimensions.

Stability examines whether explanations remain consistent under semantically invariant small changes to the input—such as replacing a synonym, slightly adjusting punctuation, or changing irrelevant formatting. If the meaning of a sentence is unchanged but the attribution distribution on the heatmap shifts substantially, the explanation is sensitive to noise and has low credibility. Faithfulness examines whether the explanation reflects actual behavior: remove or reduce the features with the highest attributions and observe whether the output drops more clearly than when removing the features with the lowest attributions. If the model is barely affected after removing the highly attributed features, then the heatmap is describing a dependency that does not exist. Completeness examines the extent to which the explanation can reconstruct behavior: keep the components or features identified by the explanation and set the rest to baseline, then see whether the original output can be approximately recovered. If an explanation claims to have found a key mechanism but can only explain a small part of the behavior, its coverage should be reported accurately.

Simplicity and human usability are a different category of criteria and should not be conflated with the three above. An explanation can be very concise and readable yet neither faithful nor complete; and vice versa. Whether humans can understand it measures communication quality, not mechanistic correctness.

Falsifiability requires explanations to be subjected to sanity checks. One standard type of sanity check is weight randomization or label randomization: shuffle the model's weights, or shuffle the training labels and retrain, then evaluate the "explanation" produced by the same method. If on such a model that has already lost the true mechanism the method still outputs an equally attractive and equally high-scoring heatmap, then the explanation has not captured anything real, and its attractiveness comes only from the intrinsic structure of the method itself. Ideally, explanation quality metrics should be significantly different on the real model compared with randomized baselines.

Putting these criteria into practice means preregistering hypotheses and evaluation metrics before the study begins, keeping negative control groups, rather than cherry-picking attractive stories from the results after the fact. If an explanation cannot predict behavior on new inputs, or cannot predict the results of interventions, it should be downgraded to an exploratory clue, not treated as a confirmed mechanism. Subjecting explanations to failure criteria is the dividing line between "descriptive narrative" and "testable mechanism".

A mechanism discovered in a small model cannot be automatically generalized as an explanation for all behavior of a large model. Mechanistic explanations inherently carry scope and version properties; without these qualifications, conclusions will fail.

This limitation has several sources. First, component functions may be distributed. A concept is often not carried by a single neuron or a single pathway, but is dispersed across many components, and there is redundancy among components. A clear circuit located in a small model may be, when placed in a larger model, only one of many parallel pathways, with the other pathways doing most of the work. Second, function changes with scale, training process, and task distribution. As models grow larger, certain circuits may be reorganized or give way to more efficient implementations; changing the task or the data distribution may also rewrite previously stable functions. Third, a circuit proven to be "sufficient" on a synthetic task does not mean it is "complete" on open-domain tasks. Synthetic tasks often compress mechanisms into a few channels, whereas mechanisms in real tasks are much more complex; sufficient components may not be enough to explain overall behavior.

Version drift can damage explanation assets in a more direct way. After a model is fine-tuned, its weights change, and its hidden representations also undergo rotation or rearrangement. Probes trained previously, feature directions identified previously, and neurons located previously may completely fail on this version of the model—the coordinates they point to no longer correspond to the original concepts in the new representation space. Therefore, explanations are not one-time findings, but an asset that needs to be governed alongside model versions.

For mechanistic explanations to have reproducible and transferable validity, their applicable boundaries must be explicitly declared: model hash, training data, task setup, prompt template, and intervention scope. Any change in any of these dimensions may invalidate the conclusions. To verify whether a mechanism is robust, experiments need to be repeated with different random seeds, different model scales, and different languages to see whether the conclusions hold. Writing these boundaries clearly serves both as an honest annotation of the current conclusions and as a way for later researchers to judge to what extent the explanation can be reused.

Showing a rejected loan applicant a feature contribution chart and telling them, “Your loan was rejected mainly because the income feature contributed a negative score,” does not automatically make the decision fair or transparent. In high-risk decision-making contexts, explanations are just one link in the chain of evidence; they cannot absolve the entire decision.

First, attribution results themselves may be unreliable. They may be unstable to input perturbations; the same application gives different feature attributions when reworded. They may leak sensitive features—a seemingly neutral proxy variable (such as postal code or spending habits) that receives high attribution weight actually substitutes for the model's use of information about race, gender, and other protected attributes. They may even provide manipulable proxies that let applicants “game the score” by fine-tuning certain features rather than genuinely improving their credit standing. Even if an attribution is faithful and accurately reflects which features the model relies on, it cannot prove that the objective itself is legitimate, the training data is fair, or the decision is lawful. Faithfully describing an unfair process does not make the process fair.

What different roles need goes far beyond a single heatmap. Rejected users need actionable reasons—which specific item they can change, and to what value, in order to change the outcome—as well as channels for correction and human appeal, not a technical decomposition they cannot act on. Auditors need data lineage (where data comes from, how it is cleaned, what biases it contains), performance distributions sliced by population (which subgroups have larger model errors), and assignment of responsibility (who is accountable for this decision and under what criteria). These are governance components that explanations cannot replace.

Therefore, explanations should be positioned as one piece of evidence, not as a substitute for governance. They serve to surface problems and communicate reasons, but fairness, legality, and accountability must be guaranteed through data governance, system evaluation, and rights remedy mechanisms. Treating a feature contribution chart as the endpoint of compliance is mistaking the means for the end.

Connect the dispersed elements from the previous chapters into a complete chain. Interpretability research follows a continuous process from problem definition to verifiable practice, with each step providing constraints for the next.

The starting point is to clarify whom the explanation serves, what question it answers, and what evidence it accepts. Without this step, all subsequent technical choices lose their criteria for evaluation. Next, use attribution or visualization tools to scan behavior and propose candidate hypotheses: a certain input feature, attention head, or circuit may perform some function. These observations are only sources of hypotheses, not conclusions.

Once you have candidates, construct input counterfactuals to test behavioral sensitivity: change the part of the input that the hypothesis points to, and observe whether the output changes systematically with it. This step confirms manipulability at the input level. Then use probes to locate readable information and determine whether the information in the hypothesis is indeed encoded in intermediate representations; but probes only prove that the information exists. To cross the gap between existence and use, you must move to internal interventions: ablate, replace, or add internal activations and see whether the target behavior changes as expected. Only this step establishes a causal link between internal components and behavior.

A single experiment is prone to overfitting, so you should test the conclusion on new samples outside the training distribution and check its stability and completeness: change the template, scale, and random seed, and see whether the explanation still holds and whether the components identified by the explanation can reconstruct most of the behavior. At the same time, any mechanistic conclusion must state its applicable boundaries—model version, training data, task setup, prompt template, and intervention scope—because mechanisms may fail when these dimensions change.

Finally, return to real decision-making scenarios and use explanations together with data governance, system evaluation, and appeal mechanisms. Explanations are one form of evidence and cannot replace governance and rights remedies.

This causal chain can also be organized into a reproducible verification protocol. At the model level, fix the goal: clarify “Model interpretability: from input attribution to causal intervention, distinguishing stories, evidence, and mechanisms” and declare in advance what evidence will be observed and what will serve as counterevidence. At the input level, fix the same batch of samples, the same preprocessing pipeline, and the same permission boundaries, and record input hashes, slice labels, and rejection reasons to ensure fair comparison. At the mechanism level, change only one core variable while locking all other configurations, and record key intermediate states and the first point of deviation from expectation to locate the causal break. At the output level, use the same acceptance rules and resource budget to compare stratified differences in quality, cost, latency, and failure rate across slices. At the counterevidence level, retain a control group that does not enable the target mechanism, and test whether the benefit replicates stably across samples and random seeds. In this way, from the initial problem definition to the final mechanism verification, every step has testable criteria, and stories, evidence, and mechanisms will not be conflated.