How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning Kaizhen Tan1 , Heqing Du2 , Yang Feng2 1
Carnegie Mellon University, Pittsburgh, PA, USA 2 Columbia University, New York, NY, USA
arXiv:2607.21351v1 [cs.LG] 23 Jul 2026
Abstract A LoRA adapter is a few megabytes that almost everyone treats as a skill rather than a record of the data behind it. We put that assumption on a scale. Extending compression-based memorization analysis to the frozen-base setting, we measure directly—in bits—how much a low-rank adapter writes into a model it never changes. The answer is both smaller than full fine-tuning and less lawful than parameter counting would predict. Adapters store a couple of bits per trainable parameter, well short of a full model’s budget; but that figure turns less on how many parameters an adapter carries than on where they sit. Move the same parameter budget from attention into the MLP and it holds nearly twice as much; strip the frozen base of its structure and the capacity all but disappears. Turned on realistic fine-tunes of Qwen2.5, the same instrument shows privacy leakage rising with the bits an adapter writes rather than the parameters it nominally has, and it draws a clean line between supervised and reinforcement learning: the secrets that supervised fine-tuning copies down verbatim, an adapter trained on verifiable rewards never records. Measuring what fine-tuning writes—rather than attacking it after the fact— turns a piece of folklore into a quantity one can design against.
Introduction Adapter files circulate as if they were harmless diffs. LoRA adapters (Hu et al. 2022) are shared through public model hubs, exchanged as low-rank updates between federated clients, and passed between teams as task adapters—on the tacit understanding that a small file carries a skill, not the data that taught it. Nobody has checked that understanding against a measurement. The arithmetic alone should give pause: a rank-16 adapter for a 7B model holds some 4 × 107 trainable parameters, and if it stored information at the 3.6 bits per parameter that Morris et al. (2026b) found for full models, one such file could carry the equivalent of twenty megabytes of training text—a whole fine-tuning corpus of medical notes or support transcripts, passed around as a convenience. What makes this more than a curiosity is where it bites. Morris et al. (2026b) showed that pretraining lives in the comfortable regime: the corpus dwarfs what the model can hold, so any single example is diluted and membership inference falls to chance. Fine-tuning inverts that picture. Adaptation sets are small and often sensitive, and they routinely hold fewer bits than even a modest adapter can absorb—the regime in which, by the same theory, memorization is total by
default. The one reassurance the field offers about memorization is calibrated on exactly the case that does not apply, and the only physical quantity left to bound the leak is the size of the adapter itself. Yet everything we know about that quantity comes from attacks: prior work plants secrets and tries to extract them, or runs membership inference and reports an AUC (Wang and Li 2025; Bossy et al. 2026; Malekmohammadi and Farnadi 2024). An attack tells you what one adversary recovered. It does not tell you what the artifact holds. So we weigh the artifact. Following Morris et al. (2026b), we read memorization off a compression advantage: a model that has memorized a string can describe it in fewer bits than a reference can, and the gap is the information the model absorbed. To make the reference exact we let frozen bases memorize uniformly random token sequences—strings with nothing to learn, so that every bit recovered is a bit stored, not a bit inferred. Random strings are not a use case; they are the calibration weight that lets us then trust the scale on real text. What the scale reads back disagrees with the folklore in two directions at once. Adapters do far less than full models— a couple of bits per trainable parameter against the full-model 3.6, confirmed by fine-tuning the same base under the same budget. But the per-parameter figure is not a property of the parameters. It moves by nearly a factor of two when the same budget of weights is placed in the MLP rather than in attention, it barely notices half-precision storage, and it collapses when the frozen base has no structure to steer: the adapter that memorizes 98% of its data on a pretrained base manages 29% on a random one. What an adapter can write is set by where it writes and what it writes into, not by how many numbers it contains. The same scale settles a second, noisier debate. A recent line argues over what reinforcement learning with verifiable rewards actually deposits in the weights—new capability, or a sharpening of what pretraining already knew (Chu et al. 2025; Morris et al. 2026a)—and argues it entirely from behavior. We bring bits instead. At matched accuracy, an RLVR adapter writes nothing our estimator can distinguish from zero, and in a direct test the private secrets that supervised fine-tuning copies down verbatim, reinforcement learning never records at all. We contribute a measurement protocol for adapter capacity, released as a harness; capacity readings
that separate written bits from parameter count, across placement, precision, and substrate; an audit tying those bits to real canary extraction and membership inference on identical checkpoints; and the first bit-level comparison of supervised against reinforcement fine-tuning. Figure 2 previews the core of it: capacity plateaus that sit well below the full-model line and move when parameter count does not.
Related Work Measuring memorization. Early notions of memorization are operational: a string counts as memorized if a prompt elicits it (Carlini et al. 2019, 2023). Such tests cannot tell memorization from prediction, since a model may emit a string it never saw simply because the string was predictable. Morris et al. (2026b) cut the knot with compression—memorization is the bits a model saves an arithmetic coder over a reference, measured on data of known information content—and by training on uniformly random strings, where nothing is predictable, turn those saved bits into a capacity meter that reads 3.6 bits per parameter for GPT-family models. Related instruments measure or bound the same quantity through intrinsic dimension (Arnold 2025), counterfactual influence (Zhang et al. 2023), behavioral probes of fine-tuned models (Mireshghallah et al. 2022), and the memorization capacity of attention (Mahdavi, Liao, and Thrampoulidis 2024). All of it studies the full model. We carry the compression meter into the frozen-base regime, where the writable object is the adapter and the base supplies structure it never updates—and where, as it turns out, protocol choices that are harmless at full-model scale come to dominate the reading. PEFT and privacy. A steady empirical thread reports that low-rank adaptation leaks less than full fine-tuning: under canary extraction in federated settings (Bossy et al. 2026), under extraction and membership inference in centralized ones (Wang and Li 2025), and by a differential-privacy argument that random low-rank projections sketch away gradient information (Malekmohammadi and Farnadi 2024). Attacks have since reached RLVR pipelines (Liu et al. 2025), and differentially private fine-tuning (Yu et al. 2022) remains the standard defense when leakage matters. Each of these establishes that leakage is smaller; none measures how much information the adapter actually holds, and all compare algorithms at unmatched task performance. We measure the stored information itself, on the very checkpoints we then attack. Low-bit adaptation and what RLVR learns. Adapters have been driven from millions of parameters (Hu et al. 2022) through shared and decomposed forms (Kopiczko, Blankevoort, and Asano 2024; Bałazy et al. 2025) down to thirteen trained with GRPO (Morris et al. 2026a; Shao et al. 2024); LoRA has been shown to learn and forget less than full fine-tuning (Biderman et al. 2024) and its expressive power characterized in theory (Zeng and Lee 2024). The thirteenparameter result carries a conjecture: supervised fine-tuning must swallow the whole token stream, while reinforcement learning’s usable signal is only the reward—a few bits per prompt—so a tiny update suffices. A parallel debate asks whether RLVR injects new ability or elicits old (Chu et al.
2025). Both are argued from behavior. We answer with the bits themselves, written into the adapter and into the policy’s likelihoods, read out at matched accuracy. Adapters as shared artifacts. What sharpens the stakes is that adapters have become objects of exchange in their own right—published on hubs, aggregated across federated clients, and passed between teams—and every transfer moves a file whose data content no one measures. The privacy literature treats a fine-tune as a model to be attacked after training; the ecosystem treats an adapter as an asset to be shared. A capacity measured in bits speaks to both readings at once: it is what an attacker is ultimately bounded by, and what a sharer is unknowingly handing over. That is the quantity we set out to measure, and the reason a number that can be computed before training is worth more here than an attack that can only be run after it.
Measuring What an Adapter Writes Information quantities Let θ̂ be a tuned model and θref a reference that captures what is predictable without seeing the training set. The codelength of a sequence x under a model is its negative log-likelihood in bits, HK (x | θ) = − log2 p(x | θ), which arithmetic coding realizes up to negligible overhead (Witten, Neal, and Cleary 1987). Following Morris et al. (2026b), unintended memorization is the reference codelength the tuned model undercuts, memU (x) = HK (x | θref ) − min HK (x | θref ), HK (x | θ̂) , (1) P summed over a dataset as memU = i memU (xi ). The idea underneath is worth stating plainly, because everything rests on it. A model that has genuinely memorized a string can predict it, and a good predictor is a good compressor—so the bits the tuned model shaves off the reference are, quite literally, the information it took in. On ordinary text that figure tangles memorization together with understanding, since a model also compresses what it correctly generalized. Random strings cut the tangle: there is nothing to understand, so every saved bit is a memorized bit, and a scale calibrated here can be trusted when we later turn it on real data. The random-string probe makes the reference exact. Sequences are 64 uniform tokens over a 2,048-symbol vocabulary after a start token, so HK (x | θref ) = 704 bits by construction and nothing in x can be predicted: every bit of memU is a bit memorized. Fresh random sequences held out from training must then read memU ≈ 0, a check that passes across our whole spine (Appendix A; appendix pointers refer to the supplement). Two further quantities describe the artifact and the behavior. The parameter-write cost CL(∆θ) is the codelength of the trainable weights themselves—per-tensor quantized and entropy-coded, minimized over precision—and upper-bounds what shipping the adapter communicates. The behavior-write bits P W (D) = (x,y∗ )∈D HK (y ∗ | x, θbase ) − HK (y ∗ | x, θ̂) (2)
1 Adapt to pure noise
2 Measure what's written
3 Audit real fine-tunes
16K-step budget · LR envelope
0 1 0 1 1 1 1 0 0 0 1 1 uniform random · 704 bits each every recovered bit is memorized
Writability α̂ad
Bits → leakage?
the memorization plateau a floor on capacity
canary exposure + MIA across LoRA ranks exposure follows the bits
adapter
Artifact bits CL(Δθ) entropy-coded update size the bits a file ships
SFT vs GRPO, matched acc.
Behavior bits ΔW frozen base
trained
Can RL write secrets?
read off the final policy SFT and RL share a scale
+ a secret-injection test SFT RL
SFT injects; RL will not
Figure 1: Overview of our measurement framework. We first adapt a frozen model to uniformly random sequences, where every recovered bit must come from memorization, and then measure adapter writability, artifact codelength, and behavior-write bits before applying the same framework to privacy leakage and SFT–GRPO comparisons. measure how far tuning moved the model’s likelihoods on canonical answers. Because W reads only the final policy, it treats a supervised and a reinforcement-learned adapter alike, though only one was trained on likelihood—the property that makes the two comparable at all. On text we report the signed excess ∆W = W (Dtrain ) − (ntrain /nheld ) W (Dheld ): the training-set bits beyond what a size-matched held-out set explains, the memorization left once ordinary learning is subtracted off. Three readings rather than one, because they answer different questions. The artifact cost CL(∆θ) is what the file physically transmits—the quantity a hub or a federated aggregator would reason about when it decides what a shared checkpoint carries. The behavior bits W and their excess ∆W are what the model’s outputs reveal, and because they read a policy rather than its weights, they compare training methods that a weight-space measure could not. And memU , on random data, ties both to a ground truth that neither has on real text. An adapter can read high on one and low on another; the gaps between them are as much of the story as the numbers themselves.
Reading capacity cleanly A capacity number is only meaningful once optimization, not storage, has stopped being the bottleneck—and in the adapter regime that is a delicate condition. Learning rates that are stable for full models diverge here; early-stopping rules that behave at full scale either never fire as the loss flattens or cut off exactly the over-capacity runs that define the plateau. We therefore fix a 16,000-step budget with no early stopping, sweep the learning rate and report the envelope, and place dataset sizes relative to estimated capacity so the informative
band is sampled rather than a budget-starved tail. The resulting figure is a lower bound at a declared budget: extending to 40,000 steps lifts it by 10–24% (Appendix B), so we read every value as “at least this much.” Bases are GPT-style decoders (2M and 8M non-embedding parameters) pretrained on WikiText-103 (Merity et al. 2017) then frozen; adapters are LoRA (Hu et al. 2022) and weight-tied TinyLoRA (Morris et al. 2026a). The untouched base bounds each reading from below, and full fine-tuning of the same architecture— which reaches near-total memorization under the identical budget—bounds it from above.
The Capacity of Adapters Plateaus, far below the full-model line Every configuration tells the same story (Figure 2): memorization climbs with the data while data is scarce, then bends onto a plateau once the dataset carries more bits than the adapter can hold. The two largest readings per configuration agree to within a few percent, and held-out sequences stay at zero throughout—so the plateau is a ceiling on storage, not an accident of the fit. Two numbers frame the result (Table 1). Full-model training stores about 3.6 bits per parameter (Morris et al. 2026b); fine-tuning the same base under our own budget confirms it climbs past 3. Adapters, given that same budget, stop at 1.7 to 2.8. The gap is real and not an artifact of stopping early—it survives the matched budget, and the readings are stable across seeds and only creep upward with more compute, never enough to close it. An adapter, whatever else it is, is a far leakier vessel per parameter than the model it patches, but a much smaller one in absolute terms.