PEFT / LoRA: Freezing the backbone, learning only low-rank weight increments

In a linear layer, the base weight W is a d×k matrix: it transforms a k-dimensional input into a d-dimensional output. When fine-tuning this layer, what the model actually needs to learn is the change in weights ΔW, and the fine-tuned weights equal W + ΔW. If you fine-tune directly, every element of ΔW is an independently trainable parameter, with the same size as W itself, i.e., d×k parameters.

The key question is: does ΔW really need that many mutually independent degrees of freedom? The rank of a matrix can be understood as the number of mutually independent directions of change it contains. A d×k full-rank matrix has min(d, k) independent directions, and adjustments along each direction are independent of one another; in contrast, the change in a low-rank matrix is compressed into a few directions, and the remaining directions are just linear combinations of these. If a new task only requires adjustments along a few directions, the information content of ΔW is far smaller than its size, and there is no need to store independent changes separately for each of the d×k elements.

Pre-trained models have already learned general representations from large amounts of data. When adapting to a new task, it is often unnecessary to overturn these representations; instead, only adjustments along a few directions, such as attention distributions or feature combinations, are needed. In other words, the task-related change ΔW likely concentrates in a low-dimensional subspace. If the effective rank of ΔW is r, and r is much smaller than both d and k, it can be decomposed into the product of two narrow matrices: ΔW = B·A, where B is d×r and A is r×k. In this way, the parameters that need to be stored and trained are reduced from d×k to r×(d+k), while the information about the directions of change is retained at rank r.

From the layer's input-output interface, nothing changes: the input is still the original layer's k-dimensional vector, and the output is still a d-dimensional vector. The difference is that the output is obtained by adding two parts: the frozen base computes W·x as usual, and an additional low-rank branch B·A·x is superimposed as the task-related increment. The base provides stable general capabilities, while the low-rank branch carries task-specific adjustments.

It should be made clear that this is an empirical assumption, not a theorem that holds for all tasks. If the task involves complex domain transfer, conflicting adjustment directions among multiple tasks, or entirely new capabilities that the model has never learned, the effective rank of ΔW may not be low. In such cases, a larger r is needed, LoRA must be applied to more layers, or even full fine-tuning must be used. The low-rank constraint applies to the increment ΔW, not to the base W: during inference, the complete W is still required for computation, and LoRA only changes its output through a small branch.

In the forward pass, LoRA does not replace the original linear layer; instead, it attaches a trainable bypass alongside it and then sums the results of the two paths. For input x (a k-dimensional vector), the output h is computed as:

h = W·x + (α/r)·B·A·x

where W is the frozen d×k base weight and h is the d-dimensional output. The low-rank branch consists of two real matrices (ℝ indicates that the matrix entries are real): A is r×k and B is d×r. The data flow is first down and then up—A first compresses the k-dimensional input into an r-dimensional intermediate representation, and B then expands the r-dimensional representation back to d dimensions, so the branch's final output still has the same dimensionality as the base output and can be added directly. r is the upper limit on rank and determines the width of the intermediate representation; α is the scaling coefficient, and the ratio α/r controls the strength of the low-rank branch relative to the base branch: with the same α, the larger r is, the more thinly each unit update is spread.

The choice of initialization ensures behavioral consistency at the start of training: A is typically initialized randomly and B is initialized to zero. Thus, before the first step B·A = 0 and ΔW = 0, the model output is exactly the same as with the base model alone, and learning starts from a “zero increment” and gradually unfolds. This zero initialization also means that α/r directly determines the scale of each parameter update; if rank is increased without simultaneously adjusting α, the overall strength of the increment will change accordingly, and rank and α must be understood as a pair of parameters. Dropout can likewise be applied only to the LoRA branch, randomly discarding part of the bypass signal during training as a regularization measure targeting the increment.

During backpropagation, W is not updated, but gradients still must pass through W in order to propagate to adapters in earlier layers and to A and B in the current layer. In other words, only the parameters are frozen, not the computation: the base model's forward and backward activation overhead remains unchanged. When evaluating configurations, you also cannot look only at training loss: if the target-task metric improves while general capability declines, it indicates that the low-rank branch did change task behavior but introduced a regression against the base model's capability, and such a configuration should not be selected by training loss. rank is also not a simple “compression-ratio knob”—it changes the width of the expressible increment subspace and must be ablated together with α, the number of layers to which it is applied, and the training data in order to judge whether a given value is truly appropriate.

Let's work out this calculation at a concrete scale. Take a square d = k = 4096 matrix as the base weight W: it contains 4096 × 4096 = 16,777,216 parameters, about 16.8M. Full fine-tuning means all 16.8M elements enter the optimizer; LoRA, in contrast, trains only two narrow matrices—A has shape 8×4096, B has shape 4096×8, together 8 × 4096 × 2 = 65,536 parameters. 65,536 is about 0.391% of 16.8M, meaning the two low-rank matrices train less than 0.4% of the base parameters yet can add a 4096×4096 increment to the base output. The key cost consideration is this: what is saved is the number of parameters that need to be trained, stored, and have gradients passed back, not the forward computation—the full 4096×4096 base matrix still participates in every multiplication.

In this example, rank acts as a capacity knob; the parameter costs for three settings are as follows:

Each row in the table is rank multiplied by 2 × 4096: every time rank doubles, trainable parameters and the expressible increment subspace double together. At rank=4, parameters are only 0.195%, the lowest cost, but the number of directions is also the smallest; complex tasks may underfit because the increment subspace is too narrow. rank=8 is the example's starting value; rank=32 gives a 1.563% parameter share, with clearly higher capacity. Even at 32, trainable parameters are still only about 1/64 of full fine-tuning, while the intermediate dimension is wider and can accommodate more independent directions of change.

Reducing trainable parameters to 0.39% of the base does not mean that GPU memory also drops to 0.39%. The savings are concentrated in the kind of overhead where the optimizer maintains per-parameter accounting: a frozen W no longer needs to store gradients, nor does it need to store Adam's first and second moment states, and in full fine-tuning these two blocks are usually on the same order as the weights themselves or even larger; adapter checkpoints are also only as large as the two narrow matrices. The memory that still remains is unrelated to whether parameters are trainable: the base weights themselves must stay resident, the activations produced by the forward pass, the intermediate values that backpropagation replays or recomputes, and the temporary workspace for matrix multiplications are all kept unchanged. Activation volume grows with batch size, sequence length, and number of layers, and in long-sequence training it is often the dominant item in GPU memory—and LoRA does not reduce this part at all.

Breaking down the training resources item by item, the differences between the two approaches are as follows:

In other words, what LoRA eliminates is the gradient and moment memory associated with W and its optimizer state, while it preserves the weights, activations, and all intermediate storage required for computation. Training speed likewise will not accelerate proportionally to the parameter count: the most expensive operation in each layer is the large matrix multiplication W·x, which accounts for the main FLOPs and runs as usual, and the low-rank bypass merely adds a small amount of extra computation. Where the savings are greatest is in parameter storage, optimizer memory, and multi-task switching cost, not in the basic forward and backward compute.

Where LoRA is attached determines which part of the computation task data can modify. The Transformer attention mechanism consists of four groups of projections, each with a distinct role: Q (query) decides “whom to look for,” K (key) decides “how to match,” V (value) decides “what information to retrieve,” and O (output) decides “how to send the aggregated result back to the backbone.” The MLP immediately after attention is a feed-forward feature transformation layer. Common implementations split it into three projections—gate, up, and down: gate controls the strength of information passing through, up increases dimensionality to expand representational space, and down reduces the number of vector coordinates back to the backbone dimension. Here, dimensionality reduction only means reducing the number of vector coordinates from many to fewer so that the MLP output can return to the same dimensionality as the backbone; it is not the kind of dimensionality reduction that compresses sample features onto a two-dimensional plot. Applying LoRA to these modules is an explicit choice about which computational stages task data is allowed to modify.

The choice of target layers directly affects the trade-off between capacity and cost. Covering only a few layers (for example, only Q and V) saves more parameters and keeps behavior more controllable, but may not provide enough representational space; covering more layers can improve adaptation ability while also increasing overfitting risk and deployment overhead. To choose between the two, the correct approach is controlled ablation: keep the data and total training tokens unchanged, compare schemes such as “QV only,” “all attention projections,” and “attention + MLP” side by side, and rank can also be allocated differently per layer. The inputs to the configuration are candidate modules, rank, and the same training budget; the output is not training loss, but each scheme's metrics on the target task, general capability, safety, and unseen distributions. If training loss continues to decrease after adding target layers but held-out set gains stagnate, this should be interpreted as added capacity not translating into generalization, rather than using it as a basis to keep expanding layers unconditionally.

In engineering, there are also two traps that are easy to step into. Module naming varies by architecture: the same set of projections may be called q_proj, query, c_attn, or other names in different implementations; when configuring, you must follow the naming of the target implementation. If the module names in the configuration do not match the actual layer paths, the framework may fail to match any target layer, silently train zero parameters, and still report “training complete”; therefore, at startup you should print the list of trainable parameters and the matched layer paths, and assert the parameter count, so that such errors are exposed at the first step.

QLoRA stacks "frozen backbone" and "quantized storage" together: the backbone weights do not reside at original precision; instead they are quantized into low-bit codes, and only the side-branch LoRA adapters are trained. Quantization makes each weight change from a continuous high-precision value into an approximation with a small number of discrete codes; 4-bit means each code has only 2⁴ = 16 possible values. During computation the codes are restored to higher-precision approximations for matrix multiplication. The backbone parameters themselves are not updated, but this matrix operation still produces gradients and passes them to the current layer's A and B and to adapters in earlier layers—freezing applies to parameter updates; the gradient path is not cut off.

QLoRA's three technical components solve problems at three different levels, not three aliases for the same precision. NF4 is a 4-bit encoding designed for weights with an approximately normal distribution, addressing the representation quality of the weights themselves; double quantization applies another round of compression to metadata such as the scaling factors used for quantization, addressing storage of quantization metadata; paged optimizer moves optimizer states out in batches when GPU memory is tight, addressing the peak memory of the training process. Only together can they fit large-model training into consumer hardware.

From an input-output perspective, QLoRA's inputs are the quantized backbone, LoRA configuration, and task data; its outputs are the trained adapters and the accompanying quantization training recipe. After training there are two different deployment routes: merge the adapters into a high-precision backbone and then quantize again, or keep the backbone and adapters separate for inference. The two are not the same artifact; quantization error and the merging path can pull the results apart. If the metrics of the two artifacts do not match, the difference should be attributed to the merge and re-quantization step, and each route should be evaluated separately, rather than saying generically "how well QLoRA works." Resource boundaries must also be clearly separated: lower training memory does not mean training FLOPs fall by the same proportion; dequantization and the backbone's forward and backward computation still occur as usual, and the main savings are the storage precision of backbone parameters and optimizer memory.

After adapter training is complete, there are two deployment forms: write the increment back into the base model, or keep the branch independently separate. Merging means executing W ← W + (α/r)·B·A, folding ΔW into the weights themselves. The merged model has only one set of weights; at inference there is no additional matrix multiplication from a side branch, and there is no need to load and attach adapters, making it suitable for scenarios that use a single adapter consistently. But it also pays a price: that 'task-specific increment' no longer exists as an independent switch. To return to the original base model or switch tasks, you must prepare the pre-merge files again; the newly produced full weight file is also as large as the base model. Saving separately is the opposite—adapters are just small files of two narrow matrices, naturally supporting switching by tenant, by task, and rollback at any time. However, inference incurs the overhead of the extra branch operator, scheduling needs to batch requests that use the same adapter together, and frequently swapping different adapters causes cache thrashing and reduced throughput.

An adapter is also not an isolated file that can be migrated arbitrarily. It must be bound to the base model hash, tokenizer, target module configuration, rank, α, dtype, and training data version. A base model with the same shape but different content will produce completely unvalidated behavior; even if loading succeeds, it does not mean the results are trustworthy. The order of merging and quantization also affects the final artifact: merging first then quantizing and quantizing first then merging do not necessarily produce the same model, so the artifact must record the complete recipe. Multi-adapter stacking is another way to explore capability combination, but the scales and directions of different increments interfere with each other, and the combined behavior has no guarantees.

A small parameter proportion does not imply a small impact on model behavior. A neural network's output is determined by the directions parameters point in and the chain of layer-by-layer amplification, not by a linear relationship with the number of modified parameters. An update of only 0.1% of parameters, when applied across multiple layers and affecting all input tokens, causes each small directional adjustment to propagate and amplify through the layers, ultimately significantly rearranging the output logits—this is exactly why LoRA can teach the model new output formats. The same mechanism can work in reverse: if unsafe samples are mixed into the training data, a small number of updates is enough to break safety refusal; if the data is too narrow, the model may overfit specific wording, forget general capabilities, and even form triggers for certain input patterns. The degree of behavior change depends on which directions and which layers the modifications land on, not on the number of modified elements.

Therefore, both training and distribution of adapters must be governed by 'influencing behavior' rather than 'small size'. Training data requires licensing, deduplication, and anti-poisoning treatment; third-party adapters should be treated as executable model artifacts, with source scanning, signature verification, and behavior evaluation before loading. Evaluation cannot focus only on the target task: regressions are needed for target, general capabilities, long context, multilingual, safety, tool calling, and calibration; when degradation occurs, use layer-by-layer and adapter-by-adapter ablation to locate which increment caused it. The supply-chain risk of adapters comes precisely from their lightweight nature—small files are extremely easy to spread and also extremely easy to bypass the review process for the full model; the permission level for loading an adapter should be equivalent to loading model code or weights themselves.

Task nature determines the adaptation method; three typical needs correspond to three different routes. If the task primarily requires frequently updated new knowledge, model parameters are not the right place to store knowledge—prefer retrieval-augmented generation (RAG) or tool calling, letting external knowledge sources handle timeliness. If only a small change in output format or behavior constraints is needed, first try prompting and structured constraints, which costs the least and carries no training risk. LoRA's advantage scenario is multi-tenant task adaptation and limited compute: multiple tasks share one frozen base, each holding a small adapter, with low training, storage, and switching overhead. When a task requires broadly reshaping model capability and data and resources are ample, full fine-tuning becomes the option worth considering; if edge deployment is also constrained by size, then distillation and quantization must be layered on.

When choosing, you cannot just keep circling around "whether LoRA will work"; instead, compare all baselines under the same budget: prompting, retrieval, LoRA, full fine-tuning, and directly switching to a stronger model. What is accounted for is the complete cost per successful task—training, storage, adapter routing, inference latency, evaluation, and long-term maintenance—not just the single item of "whether the GPU can handle it". This comparison also provides a stopping condition for scaling: when increasing rank or the number of target layers causes gains on the true held-out set to plateau while degradation in general capability and safety begins to rise, it indicates that adapter capacity has exceeded what the task requires, and you should stop expanding capacity rather than continuing to increase the configuration.

LoRA's complete causal chain starts with the question: do parameters need to be changed? First probe with prompts, retrieval, or constraints: if knowledge updates can be handled by external sources, and if changes in format and behavior can be covered by a few lines of constraints, no parameters need to be trained. Only when task adaptation must be written into the model's computation itself do we enter the parameter adaptation step—and the placement of that step is exactly the basis for choosing the next link.

After selecting the parts to change, the configuration targets are the layers, rank, and α. The target layers determine which computational steps the task can rewrite, rank determines the width of the expressible incremental subspace, and α/r determines the strength of the increment relative to the base; together they constrain the adapter's capacity and cost. Next, freeze the base and train only A and B: zero-initializing B ensures training starts from "behavior exactly identical to the base," gradients pass through the frozen W and continue to flow to A and B, and at the end of training you obtain ΔW composed of two narrow matrices.

After obtaining the adapter, validation work shifts from "training loss decreasing" to "does capacity actually translate into generalization": with data and training tokens fixed, perform ablations on rank, number of layers, and data coverage, and observe whether held-out set gain rises together with capacity, or only brings overfitting and degradation of general capability. During deployment, choose between separated and merged adapters according to engineering boundaries—separated suits multi-task switching and rollback, merged suits a fixed single adapter, and either way record the complete recipe including base hash, rank, α, and quantization order. Finally, finish with regression testing: evaluate target, general, long-context, multilingual, safety, tool calling, and calibration item by item, and record the training data version and adapter lineage. The output of each step is the input to the next; omitting any link leaves subsequent judgments without a basis.