Model Merging: Combining Weight Deltas in a Shared Coordinate System

The original motivation for model merging is very simple: if you have two models that each work properly, shouldn't a new model obtained by averaging their parameters element-wise also work? Intuitively, the mean point lies on the line segment between two good points, so it should inherit the capabilities of both. In practice, however, this intuition often fails—two highly capable models, when averaged, may produce an intermediate model with clearly degraded capabilities or even one that is completely unusable. Understanding this contrast is the starting point for understanding all subsequent methods in model merging.

First, clarify the object and process of model merging. The merge operation directly performs numerical operations on the parameters of two or more model checkpoints to produce a new checkpoint, without using any joint data for training throughout. In other words, the entire process involves no gradient descent and no additional training data, only arithmetic at the parameter level. Its inputs are the source model weights and a merge recipe (for example, the weight coefficient for each model, truncation threshold, sparsity ratio, etc.), and its output is a new model. The key point is that this new model must be re-evaluated to confirm its capabilities—you cannot assume that the merged result is automatically qualified just because both source models passed evaluation.

The root cause of why merging sometimes works and sometimes completely collapses is that the neural network function is nonlinear with respect to its parameters. A model's capabilities are not a linear superposition of individual parameter contributions: halving each parameter is by no means the same as keeping half of each function. The facts that a language model memorizes, the sentence patterns it learns, and the reasoning paths it masters are distributed across complex interactions among a large number of parameters; when you linearly interpolate these parameters and map back into function space, there is no a priori guarantee of what will happen. Therefore, there is no transitive relationship between "the source models are good" and "the mean point is good."

When does averaging fall exactly into a good region? It can be characterized by "low-loss connected regions." Imagine that in parameter space there is a region where a path exists between two parameter points such that all models along the path maintain a low loss. If the source checkpoints both come from the vicinity of the same initialization—for example, two adjacent checkpoints produced by the same pre-training run, or models fine-tuned from the same base with only small changes—then the region between these two points is very likely to belong to such a low-loss connected region. The average point falls within it, so the merge not only preserves functionality but also smooths out the noise on both sides, and the result may even be more stable than either one. Conversely, if two models are trained independently from very different initializations, they are often separated by a high-loss region; directly averaging them is like taking a point suspended outside the "canyon" of parameter space, and the output naturally collapses.

A deeper reason is that independent training changes the arrangement and representational basis of hidden units. Neural networks are approximately symmetric to permutations of hidden units: if you swap the weights of two neurons in a layer, the network function remains exactly the same. Therefore, the same network function can correspond to many parameter representations, and two independently trained models, even if functionally similar, may have completely different internal unit orderings, signs, and bases. In such a coordinate-misaligned state, dimension 137 in model A may represent one feature, while in model B it may represent a completely unrelated feature; when added element-wise, the different semantics carried by the same coordinate cancel each other out, and the average result degenerates into a pile of blurred-out parameters.

The prerequisites for merging are thus clear: before discussing any weighting formula, you must first establish that the parameters of the two models are in a comparable coordinate system. This means first confirming that the common base, network architecture, vocabulary, tokenizer, positional encoding scheme, normalization method, and parameter naming are exactly the same. Identical shapes only mean that two arrays can be added; they by no means mean that the same positions in the arrays have the same function. Additivity only solves the question of "whether the numbers can be added"; coordinate alignment solves the question of "whether the semantics after addition are valid." Skipping the alignment check and directly applying weighted averaging yields a model that is numerically legal but semantically meaningless.

When the source models come from the same training trajectory or fine-tuning branches from the same base, the parameters are already naturally in a comparable coordinate system, and merging can be reduced to a very simple weighted formula. Let θᵢ be the full parameters of the i-th source checkpoint, and αᵢ the merging coefficient corresponding to it; then the merged model parameters are

θ_merged = Σᵢ αᵢ × θᵢ, where Σᵢ αᵢ = 1

When the sum of all coefficients equals 1, θ_merged is the weighted average of the source parameters; how the coefficients are chosen depends on the merging strategy. Always remember the boundary of this formula: it only guarantees that each parameter coordinate has undergone one arithmetic average; it neither guarantees that the functions carried by the same coordinate in the source models are aligned, nor that the loss after averaging stays low. Coordinate system alignment and low-loss connectivity are two things the formula itself cannot manage.

Under this framework, the two simplest categories of applications are trajectory averaging and model soup. Trajectory averaging targets different steps within the same training run: SWA (stochastic weight averaging) takes a simple average over multiple checkpoints late in training, using a smooth path to cancel out noise near individual checkpoints; EMA (exponential moving average) gives higher exponentially decaying weights to recent checkpoints, so that the final parameters lean more toward the end-of-training state. The common premise of both is that all checkpoints come from the same training run, naturally with the same initialization and same coordinates; the averaged point generally remains in the low-loss region, and the main danger is only that the averaging window crosses some bad region during training, such as a sharp learning-rate change or a transient loss spike.

Model soup, on the other hand, handles multiple models fine-tuned from the same pre-trained initialization but with different hyperparameters. These models share the base coordinates, and the differences only come from the solution divergence caused by hyperparameters, so they can be mixed directly in parameter space. The simplest is uniform soup: all candidate models are averaged with equal weights, accomplished in one arithmetic operation, often improving generalization without adding any computation; the more refined is greedy soup: first put the best single model on the validation set into the 'soup', then examine the remaining candidates one by one, and only add those that further improve validation-set performance. The problem with greedy selection is that it repeatedly queries the same validation set, and the chosen combination may overfit that validation set, with the generalization gains shrinking on the true distribution.

These methods share the same input-output structure: the input is a set of compatible checkpoints, merging coefficients, and a validation set for filtering; the output is a new weighted checkpoint and its generalization evaluation results. Their scope of application is concentrated among solutions to similar tasks or similar data—adjacent steps from the same training run, different hyperparameter branches for the same task. They are not arbitrary capability assembly machines: directly applying this formula to two models that solve completely different tasks and whose capability patterns do not overlap still cannot bypass the two preconditions of coordinate alignment and connectivity.

Arranged by source model relationship, this kind of simple combination can be divided into four categories:

The first two rows are the within-trajectory and same-task combinations discussed in this section; they have the lowest risk because they do not need cross-task coordinate alignment at all. The latter two rows begin to extend the merging objects to models trained on different tasks: linear interpolation takes a compromise between two compatible checkpoints, and the risk is that the midpoint of the two points happens to hit a loss barrier; Task Arithmetic attempts to combine the increments of different tasks, and the risk escalates to conflicts between tasks. Whether the latter two categories can hold depends on whether each task's 'change amount' can first be separated from the common base, and this is exactly the core problem that subsequent model merging methods aim to solve.

Directly averaging the parameters of two fine-tuned models has a redundant side effect: the common base shared by the two models gets averaged again, and the “difference parts” you actually want to combine are buried in a large mass of identical shared parameters. Task vectors subtract the common part first, so that merging acts only on what each task actually changed.

Let θ_base be the exact shared checkpoint from which all task fine-tuning starts, and θᵢ be the model after fine-tuning on the i-th task. The task vector for the i-th task is defined as

Δᵢ = θᵢ − θ_base

This subtraction is coordinate-wise: at each parameter position, the value of the base is subtracted from the task model's value, and the resulting Δᵢ records “how much this fine-tuning changed at this coordinate”; positions with no change are naturally 0. The merging formula then becomes

θ_merged = θ_base + Σᵢ αᵢ × Δᵢ

That is, first weight and sum the task vectors by coefficients αᵢ, then add the weighted increments back to the common base as a whole. αᵢ controls how strongly the i-th task's increment enters the final model; the larger αᵢ is, the more prominent that task's features are. When all αᵢ are set to 1, the changes from each task are superimposed equally.

The core benefit of this transformation is interpretability. When averaging fine-tuned models, the formula mixes together “the base shared by everyone” and “task-specific changes”, making it impossible to tell which part the merging effect comes from. After switching to task vectors, the object of comparison becomes “what fine-tuning changed”; the common base is added back only once, and the interactions among increments are immediately clear. If two task vectors point in opposite directions at the same coordinate, merging cancels them out; if they point in the same direction and both have large magnitudes, the superposition overshoots, pushing the parameters too far from the base into a region where the loss is no longer reliable. Coefficients therefore need to be searched on an independent validation set and range-limited—the validation set is separated from any single task's training set to avoid choosing coefficients that merely make one task look better; range limits prevent the weight of any one increment from being set to an extreme value.

Task vectors are also often given a role beyond their proper function: negative coefficients. Setting a task vector's αᵢ to a negative value is equivalent to subtracting that task's change from the base, intuitively like “weakening” the capability direction associated with that task. It can serve as an experimental tool, but it is not a verifiable deletion guarantee—no mechanism can confirm that the subtracted part exactly corresponds to a specific piece of knowledge, and the model's out-of-distribution behavior after subtraction cannot be predicted.

This unverifiability stems from a fundamental limitation of task vectors: they are not independent semantic modules. A single parameter often participates in multiple capabilities, and the nonzero coordinates in a task vector are not exclusive to that task. When they are linearly superimposed, multiple intentions at the same coordinate become entangled and produce cross-interference; the combined model may end up in a parameter region that none of the individual tasks has covered, exhibiting out-of-distribution behavior. The success or failure of task arithmetic depends entirely on whether these increments are sufficiently “sparse” and “each in its own lane” across coordinates—which is exactly the conflict that later empirical methods need to address.

Saying abstractly that “task vectors conflict” is not intuitive. A reduced three-dimensional example can lay out all three forms of conflict. Suppose the refund task’s task vector is ΔR = [0.6, −0.4, 0.1], and the safety task’s task vector is ΔS = [−0.5, 0.3, 0.2]. Adding the two coordinate-wise gives

ΔR + ΔS = [0.6 + (−0.5), −0.4 + 0.3, 0.1 + 0.2] = [0.1, −0.1, 0.3]

Look at this table dimension by dimension:

The first two dimensions are typical sign conflicts: the two tasks give updates in opposite directions on the same coordinate. On dimension 1, the refund task wants to increase by 0.6 in the positive direction, while the safety task wants to decrease by 0.5 in the negative direction; after direct addition only +0.1 remains, and the large updates from both sides almost cancel each other out. Dimension 2 is the same: adding −0.4 and +0.3 leaves only −0.1. Figure 1 depicts this scenario: the two task vectors have opposite signs on the first two dimensions, and if they are merged directly without processing, the updates on conflicting coordinates are flattened.

Be especially wary of an optimistic interpretation of “cancellation.” Numeric cancellation on a coordinate does not mean the task conflict has been “resolved”; more likely, it simply means that two useful updates were deleted at the same time: the +0.6 that the refund task contributed to fixing the refund process is gone, and the −0.5 that the safety task contributed to strengthening safety is also gone. The remaining +0.1 is neither enough to support refund capability nor enough to support safety capability—both ends are lost. Cancellation is arithmetically neutral, but functionally it is almost always a net loss.

Dimension 3 is another form: the two tasks point in the same direction. Adding +0.1 and +0.2 yields +0.3; because the directions are aligned, there is no cancellation but rather mutual reinforcement. Reinforcement itself is not necessarily a bad thing, but when the magnitude is too large it creates overshoot risk—pushing parameters into a region that neither task ever reached during individual training. Loss behavior loses its reference, and the model may exhibit unpredictable behavior on this coordinate that gets “pushed farther and farther.” Merging conflicts therefore must address two types of problems: cancellation when signs are opposite, and scale runaway when same-direction updates accumulate.

Finally, we must set boundaries for this example: the parameter space of a real model has billions of dimensions, and the numeric magnitude of a single coordinate does not directly correspond to functional importance—a 0.6 update is not necessarily more impactful on final behavior than a 0.1 update, because importance also depends on the coordinate’s position in the model’s computation path and its interactions with surrounding parameters. The value of this three-dimensional example is not in the numbers themselves, but in illustrating why the two basic forms of conflict (sign conflict and scale accumulation) must be handled explicitly, and why the “add directly and patch up afterwards” strategy is infeasible at real scale.

Consider a network with only two layers. If you arbitrarily swap the order of its hidden-layer neurons, will the network's computation change? The answer is: if you only swap the order without simultaneously adjusting the connections, the network is naturally broken; but if you permute the output weight rows of the previous layer and the input weight columns of the next layer according to the same ordering at the same time, then the 3rd output sent by the previous layer still flows into the 3rd input of the next layer, the entire computation path remains intact, and the network function can be exactly the same—only the parameter vector is written in another permutation. This property is permutation symmetry: the neuron indices change, but the composed function does not.

A direct consequence of permutation symmetry is that the same network function has many equivalent representations in parameter space. Two independently trained models may learn nearly the same function, but their converged hidden-unit arrangements and representation bases may be completely different—one model's “feature A” lives at coordinate 137, while the other model's “feature A” lives at coordinate 409. If you then directly average coordinate by coordinate, you mix model A's “feature A” with model B's “feature B”, and the merged result naturally degrades. That is why the earlier discussion emphasized: matching shapes only guarantees that the arrays can be added, not that the same position carries out the same function.

Thus alignment becomes an independent processing step: the input is either the weights of the two source models, or the activations of the two models on the same batch of calibration inputs, and the output is the correspondence among neurons, channels, or attention heads. Based on the signal used, alignment methods fall into three categories. weight matching directly compares how similar the weights are to find a permutation—whichever neurons play the most similar roles on both sides are paired together; activation matching uses response similarity to pair them—feed the same batch of calibration inputs into both models, see which neurons/channels/heads have the closest output patterns, and pair them by pattern; permutation alignment then uses the pairing results to reorder one model's coordinates into the other model's indexing system, so that the paired parameters become aligned in coordinates. After alignment is complete, coordinate-wise operations become meaningful again.

But local alignment does not solve all problems. Even if every layer's neurons have found reasonable pairings and the coordinates have been reordered, there is still no guarantee that the parameter path between the two models remains low-loss throughout. Permutation symmetry itself only guarantees that functionally equivalent representations “exist”; it does not guarantee that the two models' solutions lie in the same low-loss connected region. Alignment removes coordinate mismatch, not the loss barrier between the two solutions. Therefore alignment is a necessary preparation for merging, not a sufficient condition for successful merging.

This also explains why cosine similarity can only serve as a screening signal. Cosine similarity measures whether the angle between two parameter vectors is close, and it is often used to quickly judge whether two models are “roughly the same”. But parameter vectors have billions of dimensions, and many unimportant ordinary coordinates contribute high similarity, which can mask severe misalignment in a few critical layers. Two models with high overall similarity may still be completely functionally misaligned in some layer. Therefore high similarity can only be used to quickly filter out obviously incompatible candidates; the final conclusion must still return to the loss curve along the interpolation path and the post-merge task evaluation, relying on direct evidence.

Adding task vectors can conflict; a natural response is to prune away the components that may create conflict before adding. TIES is a representative empirical route along this line, and its starting assumption is that fine-tuning increments contain a large amount of redundancy or noise, and conflicts mainly come from these unnecessary, unreliable coordinates.

TIES processing is divided into three stages. Step one, prune small-magnitude updates: set task vector elements with very small absolute values directly to zero. The rationale is that small-magnitude updates are likely fine-tuning noise rather than the true carriers of task capabilities; pruning them makes increments sparser and shrinks the battlefield of conflicts. Step two, resolve conflicts by aggregated sign: for each coordinate, count the net direction contributed by each task vector on that coordinate, and decide whether only positive or negative updates are kept for that coordinate. Step three, merge only values that agree with the dominant sign: updates opposite to the dominant sign of that coordinate are discarded, and only same-sign updates enter the sum. After these three steps, opposite-sign updates no longer cancel each other, and the merged result becomes 'clean' in sign.

Furthermore, the merge coefficient can be refined to be per-layer: set an independent coefficient for each layer, rather than sharing a single scalar across all layers. This allows different layers to absorb task increments with different strengths, adding one more layer of control than a global coefficient; the cost is that each layer adds one degree of freedom, and the search space of recipes accordingly expands.

The trade-offs of each of these steps can be summarized in a table:

Each row's 'potential harm' corresponds to a real risk surface. Pruning small updates assumes 'small equals noise,' but in a model many small-magnitude parameters working together can also form important functions, and a one-size-fits-all cut may cause collateral damage; unifying signs assumes 'the majority direction is correct,' but a few tasks may need opposite updates on individual coordinates, and forcing them to flip damages capabilities; random dropping introduces variance, and runs with a different seed are not reproducible; per-layer coefficients can indeed allocate capabilities more finely, but each additional layer adds a degree of freedom, causing the search space to explode.

They are not analytical guarantees. No theory can calculate in advance the correct TIES threshold or the specific values of per-layer coefficients; these hyperparameters are themselves part of the merge recipe, must be written into the recipe together with the threshold and coefficients, and must be validated on the validation set. The value of empirical methods lies in providing operation sequences that are 'likely better,' not theorems that are 'necessarily better.'

Is linearly adding several adapters equivalent to obtaining a multi-task model? The answer is no, but it does offer one more flexible path than full-weight merging: adapters can evolve separately on a shared base, and you can then decide whether to merge them into the weights or use them separately per request.

First, consider the structure of adapters. An adapter stores a small increment relative to the common base, not a complete independent set of weights. Denote the base as θ_base; then the weight after merging the i-th adapter with the base is θ_base + Δᵢ—this is exactly the task-vector form from Section 3. Therefore, when all adapters share exactly the same base, the merging formula is exactly the same as for task vectors: θ_merged = θ_base + Σᵢ αᵢ × Δᵢ, and the coefficients αᵢ still need to be searched on an independent validation set. The fact that adapters are “small and light” only makes this arithmetic easier to implement; it does not change the rules: increments with opposite signs at the same coordinate still cancel, and increments with the same direction and large magnitude still overshoot. Whether this can truly become a multi-task model must still be answered by evaluation, not guaranteed by the structure itself.

Compared with full-weight merging, the adapter setting really adds the option of “not merging”: multiple adapters are kept as independent modules, and at inference time one is enabled per request—that is routing. Routing and merging are two different things: routing uses only one set of capabilities each time, and switching and rollback are straightforward; merging writes all increments into the same weights at once, so there is no branching overhead at inference, but once interference has been fixed into the weights, it cannot be undone separately afterwards. Both have trade-offs; which path to take and which coefficients to use can only be confirmed by end-to-end evaluation.

There is also an engineering pitfall common to full weights and adapters: quantization. If model parameters are stored in a quantized format as packed integers, you cannot directly average integer coordinates—that usually does not have correct floating-point semantics. The correct order is to dequantize first, restore the weights to floating point before doing any arithmetic; after merging is complete, quantize again and recalibrate the quantization parameters. Otherwise the activation range after merging has changed, and reusing the old quantization scale will introduce additional error.

Looking back, the vast majority of merge failures trace back to the same place: some metadata inconsistency, but the operator went ahead with a “just try it” attitude. The value of compatibility checking is to turn “just try it” into “stop immediately if the conditions are not met.” Several kinds of inconsistency are irreparable; no weighting formula or pruning trick can rescue them.

The first category is structural. The architecture, number of layers, hidden dimension, and parameter naming must match: if the architecture differs, the computation graph differs and the parameters are not even the same set of functions; if the number of layers differs, the number of parameter tensors will not match; if the naming differs, even if shapes happen to coincide, adding the wrong objects will not raise an error but will silently produce a semantically scrambled model. The second category is tokenization. The tokenizer, vocabulary size and order, and special tokens must match. Pay particular attention: the same vocabulary size does not mean the same vocabulary content—two models may both have exactly 50 000 tokens, but the same id points to completely different strings on the two sides; different token id semantics directly damage the embedding layer and output head, because the rows and columns in those two places correspond to semantics one-to-one by id, and misaligned ids are equivalent to making the model use the embedding for “apple” to decode “banana”. The third category is the computational environment. Positional encoding, normalization method, and chat template must match; these seemingly peripheral settings jointly determine what activations the same input produces on the two sides, and if any one of them differs, the precondition for coordinate system alignment does not hold. The fourth category is provenance. The exact common base model and its revision must be verifiable: task vectors require θ_base to be the checkpoint from which each task's fine-tuning actually started. The “same” model with a different revision, different quantization, or an extended vocabulary is not the same origin.

Beyond these there are three engineering requirements. The dtype of weights and the dequantization/recovery method must be explicit: if the source model is stored quantized, it must first be dequantized back to floating point in the correct way before computation; after merging, it should be requantized and recalibrated. The license must permit producing and distributing the merged artifact: a merged model is a new release, and the license of each source model must cover both “merging” and “redistribution” actions; neither can be omitted. Finally, record-keeping: save the source model hashes, base model information, merge coefficients, layer-wise rules, pruning thresholds, random seeds, code version, and merged-result hash; any merge must be fully reproducible from this record.

Behind these requirements is a qualitative judgment that is easy to overlook: merging is a new model release, not fine-tuning of the source models. The merged artifact should not reuse the name or safety claims of any source model—the safety evaluations that the two source models each passed have no force for their weighted sum; the merged model's capability and safety boundaries must be rebuilt from scratch.

The most typical failure mode in merging evaluation is: the benchmark for the refund task holds up, the benchmark for the safety task also holds up, and both individual scores are normal, but when a user sends a combined prompt that involves both refund and safety, the model answers terribly. The scores are not lying; they just do not cover the area where problems actually occur—capability interaction. Individual evaluations independently measure refund capability and safety capability, but a combined prompt makes the two capabilities collaborate in the same piece of reasoning, and interference on the collaboration path is invisible to individual scores. The purpose of merging has never been 'don't let each individual score drop too much'; it is to let multiple capabilities coexist and cooperate in the same weights. Therefore, evaluation must be designed around interference.

The inputs for evaluation should include the source model, the merge candidates, and a test matrix independent of the training process: single-task tests, cross-combination tests (where two task capabilities appear at the same time), conflicting-instruction tests (where two tasks give opposite requirements), general-capability tests, long-context tests, tool-use tests, and safety tests. The outputs should not only be per-item scores; they should also save per-sample capability transfer records (which samples get better or worse from the source model to the merged model), hard-risk results (unacceptable failures in safety categories), system costs (inference overhead, memory, latency), and a rollbackable recipe. Scores can be averaged, but safety cannot be averaged—you cannot use two average scores to cover up the degradation of the same high-risk sample from safe to unsafe; such single-point degradation must be exposed separately.

Evaluation also needs a reference frame. Compare merge candidates with the source model, simple averaging, task vectors, conflict handling methods, joint fine-tuning, and routing baselines under the same set of tests to know whether the gains from merging are real; scan by merging coefficients and plot the Pareto frontier, that is, those candidate boundaries where 'one metric cannot continue to improve unless another metric is sacrificed', to see the real trade-off relationships among different capabilities—every point on the frontier represents an acceptable trade-off, and candidates outside the frontier are dominated. Before deciding whether to adopt merging, one should also inspect interpolation path loss: if intermediate models from θA to θB show high loss peaks, it indicates a loss barrier between the two points, and direct averaging is risky; use layer-by-layer or module-by-module ablation experiments to locate the source of interference, so as to confirm which layers and which modules the degradation is concentrated in and provide direction for repair.

The standard for successful merging is therefore not a certain score, but a set of joint conditions: the merged product provides real quality or cost benefits on the target capability combination, hard risk has no degradation relative to the source model, and it has a reproducible recipe and a rollbackable path. All three conditions are indispensable—merging without real benefits is waste, merging with hard-risk degradation is not worth deploying, and merging that is not reproducible or rollbackable cannot enter production.

All methods of model merging can be gathered into a single causal chain from problems to verifiable practice. Each step depends on the previous step as a necessary condition; if any step is skipped, the conclusions of subsequent steps no longer hold.

First, verify the shared base and parameter semantics. Merging is arithmetic between two models in the same coordinate system, so first confirm that the source models share exactly the same base, and that architecture, layer count, hidden dimension, parameter naming, tokenizer, vocabulary and special tokens, positional encoding, normalization, and chat template are all identical. This step verifies that “the same coordinates carry the same semantics” and is the source of legitimacy for all subsequent operations; without it, any weighting formula is just a numbers game.

Second, compute weight or task deltas. Only with a valid coordinate system can you do arithmetic on parameters: checkpoints on the same trajectory can be directly weighted-averaged; models from the same base but different tasks first subtract the base to obtain the task vector Δᵢ = θᵢ − θ_base, so that merging focuses on “what fine-tuning changed,” and then αᵢ-weighted add back to the base. Subtracting the base places extremely high demands on base accuracy—any difference in revision, quantization, or vocabulary expansion will make Δᵢ lose its unified origin.

Third, diagnose permutation, scale, and sign conflicts. After computing the deltas, first see clearly how they interfere with each other: independently trained models have permutation symmetry, requiring weight matching, activation matching, or permutation alignment to align coordinates; updates with opposite signs on the same coordinate cancel out, and updates in the same direction with large magnitude overshoot; when updates are dense, TIES’s trimming small updates and unifying signs, and routing adapters separately can reduce conflicts, but these are empirical operations—thresholds and coefficients are just recipe parameters, not analytical guarantees.

Fourth, search for merging methods and coefficients. Match methods to conflict diagnosis: use simple averaging or model soup for low conflict, use Task Arithmetic for cross-task, and add pruning and sign handling when conflicts are severe; search coefficients on an independent validation set and limit their range, and see the true trade-off relationships between capabilities along the Pareto frontier. Loss curves on the interpolation path and layer-wise ablation provide decision support at this step: a high-loss peak on the path makes direct averaging dangerous; ablation that locates the specific layer tells you where to adjust.

Fifth, conduct joint regression on source tasks, cross-composition, foundational capabilities, and safety. Evaluation of the merged product cannot look only at individual benchmarks—cases where source task scores are maintained but composed prompts fail show that capability interactions must be tested separately; cross-composition, conflicting instructions, long context, tool use, and safety testing together cover the interaction areas, recording per-sample capability transfer and hard-risk degradation; no single average score can mask a single high-risk sample changing from safe to unsafe.

Sixth, save the recipe, release as a new model, and retain rollback. Merging is a new model release, not a continuation of the source models: do not reuse source model names and safety claims, and licenses must cover merging and redistribution; archive source model hashes, base, coefficients, layer rules, pruning thresholds, random seeds, code version, and result hashes to ensure reproducibility; retain a rollback path so that a merge verified to have failed can be cleanly undone at any time.

The causal relationship of the whole chain is unidirectional: without coordinate alignment, the deltas are meaningless; without conflict diagnosis, method selection is blind; without joint regression, success cannot be defined; without recipe and rollback, merging cannot enter production. The reason model merging is “sometimes effective, sometimes completely collapses” is essentially that one link in this chain was treated as an optional step and skipped.