Not Your Diary, Just Your Topic: Grading the Wrong-User Control

arXiv:2611.09609v1 [cs.AI] 6 Nov 2026

Not Your Diary, Just Your Topic: Grading the Wrong-User Control

Karina Romanova1 Igor Kiselev2 Vladimir Platonov3 Viliana Devbunova1 Gregory Leleytner4 Adeliya Leleytner5 Lina Veltman6 Aleksandr Nikolich7 1 2 Yandex Accenture 3 RawModels 4 Independent Researcher 5 Institute for Information Transmission Problems 6 TBC Uzbekistan 7 Keenable AI

Abstract A common test of personalization replaces a user’s history with someone else’s and measures the drop in accuracy. But this swap also changes the topics in the history, so the drop may reflect topic mismatch. We examine this problem with a frozen language model on LaMP-2 news categorization, using histories from similar users and synthetic histories assembled from other users’ articles. The user’s own history improves accuracy by 2.3 percentage points over a model trained without history. Removing history only at evaluation makes this gain appear larger, at 4.8 points. Category labels alone reproduce 86% of the accuracy gap between the user’s own history and a random user’s history. With synthetic histories matched on labels and text similarity, the gap falls to 0.9 points. Across two user splits and three training seeds, the advantage over a similar user’s history is 3.2 points for users seen during training and 0.65 points for unseen users, where the evidence for an advantage remains inconclusive. A small movie-tagging experiment with one training seed also finds no advantage over other users’ histories. These findings show why personalization should be evaluated on unseen users, with histories of similar content and a model trained without history as comparisons. We also find that adding gated cross-attention improves on a prefix-only model under our training setup; whether this holds more broadly remains open.

1

Introduction

Personalization systems are increasingly built the same way: freeze a capable decoder, compress what is known about a user into a small set of vectors, and inject those vectors somewhere in the forward pass [Ning et al., 2024, Liu et al., 2025, Hebert et al., 2024, Huber et al., 2025]. The compressed history is the memory. Whether the resulting system scores well and whether the decoder actually reads that memory are different questions, and benchmarks answer only the first. The usual way to approach the second question is a wrong-user control: replace the user’s history with someone else’s and measure the drop. LaMP reports gains even from randomly selected documents within the same user’s profile [Salemi et al., 2024], but those documents still carry the user’s topics. The random-history baselines of LaMP and OPPU [Tan et al., 2024] draw instead from the wider profile corpus, which gives a non-personalized comparison. Read as a test of user-specific conditioning, however, a random replacement answers a weaker question than it appears to: another user’s history differs in subject matter as well as in preferences, so the swap does not isolate identity. Wu et al. [2024] come closest to the control this calls for, retrieving query-relevant items from other users’ profiles on LaMP, where such semantic-only context helps on two tasks and hurts on four; PRIME [Zhang et al., 2025] replaces both profile text and user-specific LoRA weights with those of 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: PALM: Personalized, Aligned, Long-Term Memory for AI Systems.

(a) User identity or topic? Same query: “Parliament debates a new budget” Owner changes Topic changes

Random replacement User B

Original history User A Election debate

Football final Budget vote

Party leaders

Tennis match

League results

Similar-user replacement User C

Owner changes Topic is closer

Cabinet talks

Tax reform

Voter survey

Compare accuracy with the query and trained model held fixed. Illustrative article titles, not benchmark examples. (c) Similarity curve +2.3

Matched

+1.3

Composite, label-matched

+1.1

Composite (cos 0.95) -1.3

Similar-user

0

No-history, trained -2.6

History masked Random-user

-7.9

−10

−5

0

5

Points vs. model trained without history

Matched − swapped (points)

(b) Trained baseline 25

Random-user swap Similar-user swap

20 15 10 5

Trained-baseline gain: 2.3

0 0.80

0.85

0.90

Profile cosine of swapped pair

Figure 1: LaMP-2 news controls. (a) Illustrative histories show why a random-user swap mixes a change of owner with a change of topic. A similar-user history reduces content mismatch; the query and trained checkpoint stay fixed. (b) Three-seed mean accuracy relative to a separately trained no-history model. (c) Matched-minus-swapped accuracy against profile cosine, separately for random and similar users; bands resample users, retaining their repeated measurements. The dashed line is the 2.3-point gain over the trained comparator. The panels do not decompose a causal identity effect.

users of graded similarity and reports that every metric falls. Wu et al. explicitly distinguish semantic from personalization information; our question is how whole-profile swap contrasts change under stronger matching and user-disjoint evaluation. We therefore study the measurement itself. The organizing idea comes from probing, where control tasks distinguish information recoverable from a representation from a probe’s capacity to memorize [Hewitt and Liang, 2019]. We grade the wrong-user control: in place of one random replacement we use replacements of increasing similarity to the matched profile, first a similarity-matched real user and then synthetic profiles assembled from other users’ items, and ask how much of the measured advantage survives each. The test bed is a family of adapters around a frozen decoder, each compressing a user’s history into eight vectors. Two injection routes, a soft prefix and gated cross-attention, are compared in the main text and a third in Appendix G. We also train a no-history comparator and evaluate on users excluded from training. These checks distinguish sensitivity to a history from a gain over a model that has learned to work without it (Figure 1). We make three empirical contributions: • We combine graded wrong-user replacements with a trained no-history reference and userdisjoint evaluation (§3). Category controls explain much of the swap sensitivity, while the residual over strong negatives is small and uncertain on unseen users. 2

• We test per-user concentration against nulls at the observed item counts and correct the scale comparison with paired permutation tests (§4). • We compare a prefix with a prefix plus gated cross-attention, auditing output parsing and reporting non-shared parameter counts (§6). Deeper prefix slot encoders fit worse under this recipe; the comparison does not establish a capacity limit for prefixes.

2

Setup and controls

2.1

Model and injection routes

A frozen BGE encoder maps each history item to an embedding. A separate, trainable BGE encoder represents the current input, and an eight-query, two-layer Q-Former combines the two into Z ∈ R8×768 . The decoder and the history-item encoder stay frozen; the input encoder, the Q-Former and the injection modules train (Figure 5 in Appendix G draws the three routes). Prefix. Z is projected to the decoder width and the eight vectors are prepended to the input embeddings, the route of Liu et al. [2025], Hebert et al. [2024], Huber et al. [2025]. XAttn-k. In addition to the prefix, kψselected decoder layers receive a residual update
h′ = h + tanh(α) Wout CrossAttn LN(h), Z ,ψ (1) with α0 = 1.5 and Wout = 0 at initialization unless stated otherwise. The main cross-attention comparisons use this hybrid: it retains the input prefix and adds gated branches, so the comparison between the two routes is between a prefix alone and a prefix plus cross-attention. Which layers receive the update is swept in §6.1. 2.2

History conditions

We call a replacement history a negative and the matched-minus-negative accuracy difference a contrast. The core battery has five history conditions, supplemented by category controls and a trained comparator. Matched uses the profile attached to the example. Random-user uses the history of another user chosen by a deterministic derangement, so that no user maps to itself; the mapping is fixed across training, validation and model seeds, and this is the standard wrong-user control. Similar-user uses the other user assigned by the derangement that maximizes total profile cosine subject to having no fixed point (§3.1). Composite is a synthetic profile of six items drawn from other users so that its centroid matches the target profile’s (§3.3). History masked (called No-history in the earlier experiments) masks every history item at evaluation while the Q-Former still attends to the current input; on the fixed checkpoints of §3–4 it is a counterfactual masking condition on the matched-trained model. We additionally train a No-history, trained model with this mask throughout training and evaluation, following the input-ablation logic of Kaushik and Lipton [2018]. Random-user and History masked answer different questions: the first preserves the marginal distribution of histories but breaks their relation to the current query, and measures sensitivity to a mismatched history; the second removes the history and measures whether the supplied one helps relative to none. Neither is guaranteed to lie within the training distribution, and their difference is itself informative. The controls can be applied counterfactually, fixing a trained checkpoint and changing only its history, or by training a model under each condition; the main battery fixes the matched-trained checkpoint, whereas the trained no-history, similar-history training, and user-fold experiments explicitly retrain models. 2.3

Data, protocol and conventions

Task. The primary case study is the deprecated news-categorization variant of LaMP-2 [Salemi et al., 2024]: 15 classes, 1,605 validation examples, 323 validation users, and the first six items of each available profile. All 323 profiles recur in the full 8,090-row training file; the first 8,000 rows used by the main recipe cover 314 of them. This is a time-based split, not a test on new users. Article categories are available in the raw data but were omitted from the original serialized histories; we recover them for the label controls. The benchmark has since replaced this task with movie tagging,1 1 https://lamp-benchmark.github.io/download

3

and “LaMP-2” below refers to the news task only. Always predicting the majority class scores 0.476, a reference for the task. History items are serialized with a constant MOVIE: label inherited from the other LaMP-2 variant; removing it has little effect in the seed-42 serialization check.2 Decoders. The frozen decoder is Qwen2.5-3B-Instruct for the control battery and the injection-site comparison, Qwen2.5-0.5B-Instruct for a second scale point, and Qwen2.5-1.5B for the initialization ablation. Both routes train at the learning rate a six-point sweep selects for each, 3 × 10−5 (Appendix D), which lists the training settings. Output validity. A prediction counts as valid only when the full stripped generation exactly matches a task label. We report output validity alongside accuracy, because on a format-sensitive task a swap control otherwise registers format priming as user specificity. The main control and architecture comparisons use this audited harness, including retrained prefix rows; the earlier initialization ablation is retained as a gradient-failure diagnostic. An unaudited parser depresses the soft prefix by 4.5 accuracy points and 28 macro-F1 points while reproducing cross-attention exactly (Appendix F). The harness, the analysis scripts and the saved per-item predictions are in the supplement (Appendix A). Uncertainty. The main paired contrasts use a two-level bootstrap over model seeds and users, motivated by the system/input resampling of Deutsch et al. [2021]; our nested seed/user procedure differs from their crossed resampling. With three seeds there are only ten distinct seed multisets; Appendix L therefore reports t-intervals with two degrees of freedom alongside the bootstrap. We also resample constructions for the composite and seed–fold cells followed by users for the user split. These exploratory intervals are uncorrected for multiple comparisons. Repeating the 3B prefix at a fixed seed moved accuracy from 0.7445 to 0.7346 under non-deterministic fp16 training, so small differences between separately trained models warrant caution. Accuracies are fractions; contrasts are in accuracy points unless stated otherwise.

3

Whose history, or merely which topic?

A random derangement replaces a user’s history with somebody else’s, and the replacement is usually about different things. Any penalty it produces therefore mixes two effects, that the history is not this user’s and that it is off-topic, and random-shuffle controls cannot separate them. We add harder negatives that reduce the mismatch in embedding space. 3.1

Graded negatives on LaMP-2

The battery runs on three independently trained XAttn-9 checkpoints on Qwen2.5-3B (seeds 7, 42 and 123; the strongest configuration of §6), each evaluated under every condition with the model fixed and only the history changing. The Similar-user derangement raises the mean cosine of a swapped pair from 0.834 to 0.900, with the least similar assigned pair still at 0.806. Composites are built greedily from the 400 items nearest the target (Appendix A) and reach a cosine of 0.948; a second targets cosine 0.90 at each greedy step over all other-user items to compare with the Similar-user condition at equal similarity. The trained comparator changes the interpretation of the battery (Table 1). Let A be training and evaluation with matched histories, B the same checkpoint evaluated with its history masked, and C training and evaluation with the mask. A exceeds B by 4.8 points but C by only 2.3 (2.06, 2.12 and 2.62 across seeds). C exceeds B by 2.6 points. Thus A−B measures dependence on history at a fixed checkpoint and overstates the gain relative to a model trained to operate without it. Random-user is 7.9 points below C, accounting for 78% of the 10.1-point A-minus-random contrast; relative to B that fraction is 52%. Similar-user is 1.3 points below C, while the closest composite is 1.1 above it. These baseline contrasts do not identify a causal share of user identity. 3.2

The contrast as a function of similarity

The two derangements provide 646 distinct user–donor pairs evaluated on three checkpoints. The six pooled equal-count bins give contrasts of 14.3, 9.6, 8.6, 0.1, 3.7 and 2.7 points at mean cosines 0.793, 0.834, 0.863, 0.887, 0.905 and 0.920. Figure 1c instead separates the swap types and clusters 2 Retraining XAttn-9 without the label gives 0.7888 against 0.7882 at seed 42, an order of magnitude below the seed

standard deviation.

4

Table 1: LaMP-2 news accuracy. All history interventions use the same three matched-trained checkpoints; the last row trains and evaluates with history masked. Labels-only inputs contain the six categories without article text. Valid is the mean exact-label output rate. Intervals for paired contrasts are in Appendix L. Condition

Seed 7

Seed 42

Seed 123

Mean

Valid

Matched Composite, label-matched Composite, cos = 0.95 Composite, cos = 0.90 Similar-user Labels only, own Labels only, random user History masked Random-user

0.7975 0.7844 0.7857 0.7595 0.7551 0.7682 0.6897 0.7427 0.6903

0.7882 0.7844 0.7801 0.7595 0.7645 0.7688 0.6891 0.7439 0.6997

0.7950 0.7844 0.7801 0.7558 0.7539 0.7626 0.6598 0.7495 0.6872

0.7936 0.7844 0.7819 0.7583 0.7578 0.7666 0.6795 0.7454 0.6924

0.997 0.997 0.998 0.997 0.996 0.997 0.998 0.996 0.997

No-history, trained

0.7769

0.7670

0.7688

0.7709

0.999

uncertainty by user. For Random-user, the fitted slope is −1.2 points per 0.01 cosine, with 95% interval [−2.0, −0.4]; for Similar-user it is −0.2 [−0.9, 0.8]. Similarity is associated with a smaller penalty among random swaps, but the similar-user range shows no clear slope. This cross-sectional curve neither establishes a within-user effect nor supports extrapolation to a perfect match. 3.3

Building the negative

The closest composite reaches mean cosine 0.948 and leaves a 1.2-point matched advantage, with bootstrap interval [0.4, 1.9]. Rebuilding under E5 instead of BGE leaves 1.7 points; the Similar-user contrast becomes 4.0 rather than 3.6. A complete grid of three seeds and three greedy constructions gives a mean residual of 1.4 points, with a three-level bootstrap interval of [0.7, 2.1]. The standard deviations across construction means and seed means are 0.21 and 0.46 points. We also constrain each donor item to match the original item’s category at the same position. This joint composite reaches cosine 0.941 with 100% category agreement, no matching failures, and 5.8 donors per profile. Its residual is 0.9 points: the bootstrap interval is [0.1, 1.8] and the three-seed t-interval is [−0.3, 2.1]. Joint matching changes both the label balance and the achievable cosine, so the reduction from 1.2 to 0.9 does not isolate a label effect. Neither construction separates identity from coherence or shared preferences. 3.4

What the residual tracks

The matched-minus-masked gain is concentrated in the 850 queries whose category occurs in the six-item profile: 9.7 points, against −0.7 on the other 755. The corresponding contrasts with Similaruser are 4.8 and 2.2 points, and with the closest composite 3.2 and −1.1. For Similar-user the gap is 9.6 points when the category occurs only in the matched profile (n = 236), 3.0 when it occurs in both (n = 614), and −4.6 when it occurs only in the donor (n = 183). These strata use the reference category for analysis, not for constructing the controls. Six category labels without article text score 0.7666 for the user’s own profile and 0.6795 for a random donor. Their 8.7-point contrast reproduces 86% of the full random-swap contrast, although labels-only inputs were not used in training. Own labels exceed the masked condition by 2.1 points but remain 0.4 below the trained no-history model. A majority vote over the six profile labels scores only 0.436, against the global majority’s 0.476; a lookup of each user’s most frequent training answer scores 0.675. The results implicate category information without showing that a simple profile-majority rule explains the model. We replace nested subsets of k = 0, . . . , 6 history positions with items from a fixed donor (Appendix M). Replacing one or two items changes accuracy by at most 0.52 points for all three donor types. Full replacement costs 10.1 points for a random user, 3.6 for a similar user, and 0.9 for the label-matched composite (Figure 3). The increasing separation between donor types shows sensitivity to replacement content; it is not an additive decomposition of topic and identity. 5

Table 2: Matched minus each control, accuracy points (95% bootstrap intervals), three seeds and two user folds. The masked condition uses the same checkpoint; no separate no-history model was trained within these folds. Control History masked Random-user Similar-user Composite Composite, label-matched

3.5

Seen users

Unseen users

3.7 [1.7, 5.6] 11.0 [8.0, 14.1] 3.2 [0.9, 5.7] 0.6 [−0.6, 2.0] −0.1 [−1.2, 1.0]

0.8 [−0.7, 2.3] 7.9 [5.4, 10.4] 0.65 [−1.1, 2.2] 0.4 [−0.6, 1.5] −0.4 [−1.5, 0.5]

Seen and unseen users

We split the 323 users into folds of 161 and 162, each contributing 4,045 training rows, and train on each fold for 6,000 steps at each of three seeds. Every checkpoint is evaluated on both its training users and the held-out users under the full battery. Matched accuracy averages 0.784 for seen users and 0.738 for unseen users. Table 2 reports all six cells, with intervals resampling cells and then users within each group. The similar-user residual is smaller on unseen users, and all their strong-negative intervals include zero. Random swapping still produces a large penalty. These findings are consistent with profiles supplying a learned user prior in the time split, but do not establish that all residual signal is memorization: the fold models see fewer training users, and the intervals allow small benefits on unseen users. Training on a fixed Similar-user assignment gives accuracy 0.789, close to matched training’s 0.794; evaluating those models with the users’ own histories reduces accuracy to 0.761. Because the assignment is a bijection, a donor profile can serve as a stable user key. This experiment cannot distinguish such recognition from learning the donor’s content distribution. On the active LaMP-2 movie-tagging task, a seed-42 pilot evaluated on 692 queries from unseen users scores 0.5000 with matched history, 0.5014 with Similar-user, 0.5029 with a composite, 0.4957 with Random-user and 0.4509 with masking. Mean donor cosines are 0.977 and 0.994 for Similar-user and composite. The matched history has no observed advantage over the other histories, despite its 4.9-point advantage over masking. One seed and no trained no-history comparator limit this pilot. We attempted the replacement battery on the no-history-trained model as a negative control. Its seed-7 checkpoint generated concatenated labels under matched history, with zero valid outputs. We stopped that battery: unmasking a branch never trained on history does not provide a usable no-user-mechanism control. 3.6

Not an artifact of compression, scale or schedule

The ordering is not confined to the compressed memory. The same four histories supplied as text to the frozen decoder, with no training and no bottleneck, reproduce it (Appendix H), with a Matchedminus-Similar-user contrast of 2.3 points against the adapter’s 3.6; so does a frozen Qwen2.5-0.5B trained to the same recipe, with Matched minus Similar-user at 4.5 points [2.6, 6.6], the interval for a similar user’s advantage over masked history again spanning zero, and a random one 7.6 points below it (Appendix H). One contrast that looks like a scale effect is not: a 0.5B checkpoint trained on a shorter schedule at a higher learning rate shows almost no penalty under any replacement, and matching the recipe removes the difference (Appendix D). An article-text audit finds one exact query article in its own profile among 1,605 queries. Removing the 9.7% nearest to their profiles by article-text similarity changes the Similar-user contrast from 3.6 to 3.7 points and the composite contrast from 1.2 to 0.8 [0.1, 1.6]. Near-duplicates do not explain the main Similar-user result.

4

The per-user decomposition does not survive its own null

An average says nothing about who receives it, and thresholded summaries invite the objection that the threshold does the work [Schaeffer et al., 2023], so we report continuous statistics. Decomposing the Matched-minus-Random-user effect over the 323 validation users gives a Gini coefficient of the 6

Table 3: Per-user decomposition against permutation nulls at matched item counts: Gini of the per-user effect magnitude and the standard deviation of the per-user effect across users (not reported under the zero-effect null). The observed Gini sits next to the homogeneous-effect null and below the zero-effect null; only dispersion exceeds its null. Decoder

Statistic

Observed

Homogeneous-effect null

Zero-effect null

0.5B 0.5B 3B 3B

Gini(|∆u |) SD(∆u ) Gini(|∆u |) SD(∆u )

0.727 ± 0.016 0.352 0.761 ± 0.017 0.323

0.720 0.318 0.758 0.294

0.760 – 0.791 –

per-user effect magnitude of 0.727 ± 0.016 at 0.5B and 0.761 ± 0.017 at 3B, with the top decile of users carrying 46% and 51% of the total effect mass. Such figures are routinely read as an effect carried by a small minority of users; the reading does not survive a null. Gini(|∆|) runs high by construction whenever ∆ is noisy around a mean near zero, and that is the regime here: each user contributes about five validation items, so ∆u is a difference of two means over five binary outcomes. Two permutation nulls on the same users and item counts test the reading: the zero-effect null exchanges the two condition labels within each item, removing any systematic effect while preserving how often the conditions disagree, and the homogeneous-effect null redraws each user’s items from the pooled item-level outcome pairs, keeping the population effect and destroying only user identity (Table 3). The observed Gini sits close to the homogeneous-effect null and below the zero-effect null at both scales, with per-seed permutation probabilities for the Gini of 0.17 to 0.52 at 0.5B and 0.07 to 0.80 at 3B, so these values do not establish concentrated personalization. Dispersion does leave a residue: the observed standard deviation across users exceeds the homogeneous null in all six runs, with three of the six per-seed probabilities below 0.05 before any multiplicity correction. This is exploratory evidence of heterogeneity, not an account of which users benefit; that needs more items per user, and papers that decompose personalization gains over users should report items per user alongside the statistic. The scale comparison requires a paired test because effects are discrete and the same users recur across seeds. Swapping the two decoder-scale labels within each user gives Kolmogorov–Smirnov permutation probabilities of 0.45, 0.010 and 0.0035, and Cramér–von Mises probabilities of 0.74, 0.0035 and 0.0010. Summing the seed-level statistics gives p = 0.0015 for each test under independent within-user swaps in each seed. The distributions are distinguishable under this permutation scheme, contrary to the asymptotic KS reading. Their mean Matched-minus-Random-user contrasts are 12.8 points at 0.5B and 10.1 at 3B under the same recipe. The earlier, larger scale difference was affected by the shorter training schedule; that does not remove this remaining difference. At five evaluation items per user the per-user estimate is mostly noise whatever the model does; aggregates can hide structure when the cells are estimable [Deng, 2026], but here the decomposition was not identifiable from the data we had.

5

Where the audit finds nothing to grade

LaMP-2 is the one task of five on which the battery finds anything to decompose. On the other four the measured contrasts are small (Table 4): the differences are the size of the seed spread, and on LaMP-3 the Random-user-minus-History masked difference agrees in sign on one seed of three. One degenerate LaMP-3 run (the majority rating on 1,596 of 1,605 items) is a training failure, and excluding it changes no conclusion. Output validity is flat across conditions (0.999 in range on LaMP-3), so format priming is not at work. With nothing separating Matched from Random-user there is nothing to grade by similarity, so we built no topic-matched negatives for these tasks. Supplying the histories as text to the frozen decoder does not recover a gain either: both tested tasks get worse, mostly through output format (Appendix H), which leaves the task, the metric and the training recipe as candidate explanations and neither establishes that the tasks lack personalization signal nor rules out a compression bottleneck. 7

Table 4: The four other LaMP tasks under the three conditions. Three seeds for LaMP-3/5 and two for LaMP-1/4; history controls are counterfactual evaluations of each matched-trained checkpoint. Lower is better for mean absolute error (MAE); higher is better otherwise. One metric per task is reported; the benchmark also defines RMSE for LaMP-3 and ROUGE-L for LaMP-4 and LaMP-5. Differences between conditions are of the same size as the seed spread. Task

Metric

Matched

History masked

Random-user

Matched − Random-user

LaMP-1, citation choice LaMP-3, product rating LaMP-4, news headline LaMP-5, scholarly title

Accuracy MAE ↓ ROUGE-1 ROUGE-1

0.6707 0.436 0.1923 0.5008

0.6757 0.447 0.1886 0.4966

0.6700 0.446 0.1859 0.5006

+0.0007 −0.010 +0.0064 +0.0002

Accuracy, LaMP-2

0.80

XAttn-1

XAttn-3

XAttn-9

XAttn-5

0.75 2-layer QF

0.70

4-layer 8-layer

0.65

16-layer

0.60 0.55

Prefix (Q-Former depth) Prefix + gated cross-attention

0.50

majority class

Prefix + gated branch

20

50

100

200

Non-shared trainable parameters (M, log)

Figure 2: Accuracy against non-shared trainable parameters, excluding the shared 109M-parameter input encoder. Error bars are standard deviations over three seeds. Cross-attention models retain the prefix; these are not exactly budget-matched route comparisons. Full metrics appear in Table 5.

6

Accuracy against trainable parameters across injection routes

The battery above takes the injection route as given; this section asks what adding a gated crossattention read path to a prefix model buys per trainable parameter. Since every cross-attention model keeps the prefix (§2), this is prefix against prefix plus cross-attention, not one injection site against another; Ning et al. [2024], for example, compare a length-10 soft prompt with cross-attention under full fine-tuning, without reporting paired trainable parameter counts for those arms. 6.1

Accuracy and trainable-parameter comparisons

We sweep the number of injected layers on the audited harness with three seeds and all 1,605 validation items (Table 5, lower block). The first injected layer is worth 3.9 points of accuracy and 7.2 of macro-F1 for 14% more trainable parameters; the remaining eight layers add a further 1.5 and 4.7 for 90% more. The gain from one to nine layers is about 40% of the first step; three seeds and the repeat-run variability of §2.3 do not justify calling the curve flat. XAttn-1 beats the prefix at 1.14× its total trainable parameters and about 1.9× its non-shared parameters once the shared 109M-parameter input encoder is set aside (38.9M against 20.5M); it also beats the 4-layer prefix (34.7M) by 4.0 points and the 8- and 16-layer prefixes (63.0M, 119.7M) by more, and nine added layers reach 173.2M. These comparisons trace a trade-off; they do not isolate a route effect at exactly matched capacity. Validity holds at 0.995–0.997 across the injected rows, 1.1 points above the prefix. Tuning the learning rate for both routes widens the gap rather than closing it (Appendix D). Jeong [2026] compare six memory methods on a frozen GPT-2 under shared memory-size settings, with different trainable parameter counts. At the smaller setting, cross-attention, Hebbian memory and sparse slot writing outperform the other methods; the gap narrows at the larger setting. Both reading and writing mechanisms vary, so this comparison does not isolate the injection route. A graph memory read by cross-attention also beats a prefix on knowledge-graph completion [Liu et al., 2026b]. 8

Deepening the prefix’s Q-Former from two to sixteen layers lowers accuracy from 0.7389 to 0.6685 while median training loss rises. Increasing prefix length from 8 to 64 slots at seed 42 does not close the gap. These are failures to fit under the tested recipe, with optimization and expressivity still unresolved (Appendix C). A separate initialization ablation confirms that setting both a residual gate and its output projection to zero blocks branch gradients; a parallel prefix can conceal that failure (Appendix E).

7

Related work

User-LLM injects user representations through cross-attention [Ning et al., 2024]. Persona-Plug, PERSOMA, Embedding-to-Prefix and REPREC map a user representation to learned soft tokens in front of a frozen decoder [Liu et al., 2025, Hebert et al., 2024, Huber et al., 2025, Kavuru et al., 2026], the route we call the prefix (input-level prompt tuning; Lester et al., 2021). Flamingo and LLaMA-Adapter leave one gradient path open at initialization [Alayrac et al., 2022, Zhang et al., 2024]; the deadlock of Appendix E needs both closed [Khemais, 2026]. The control design descends from trained input-ablated baselines [Kaushik and Lipton, 2018], control tasks [Hewitt and Liang, 2019] and explicit positive and negative controls [Mu et al., 2023]. That an off-target context can cost more than no context has been observed for retrieved text [Petroni et al., 2020, Yoran et al., 2024]. Matched or graded replacements have precedents (Appendix K sets them side by side): Wu et al. [2024] retrieve query-relevant items from other users’ profiles on LaMP as a semantic-only control, PRIME [Zhang et al., 2025] swaps in the memories of users of graded similarity, and, concurrently with this work, ENOUGH [Liu et al., 2026a] scores each record against content-matched other-user replacements as a training signal; Zheng et al. [2026] likewise find a shuffle control necessary but not sufficient in multimodal recommenders. We combine whole-profile controls with category matching, a trained no-history reference, and evaluation on users excluded from training. The within-user replacement curve complements the cross-sectional similarity curve; neither is a causal decomposition of personalization. Audits that decompose a single personalization score are adjacent [Deng, 2026], as is the shortcut-and-confound protocol of Chen et al. [2026] for memory-policy classification.

8

Limitations

The main case study uses a deprecated news task, six-item truncations of longer profiles, and a time split with extensive user overlap. Category matching and the near-duplicate audit reduce specific confounds but do not hold topic, style, coherence and preferences separately fixed. User-disjoint training changes the available training population, and its intervals include modest benefits from matched history. The seed–fold bootstrap treats cells as resampling units even though users recur across cells; its intervals should be read as exploratory, not as an equivalence test. The movie-tagging pilot has one seed and lacks a trained no-history comparator. The architecture comparisons are not exactly budget-matched, rely on one recipe, and leave the prefix’s optimization failure unresolved. Longer profiles, other decoders, and deployment utility, latency and privacy remain outside this study.

9

Conclusion

On LaMP-2 news, a 10.1-point random-swap penalty accompanies only a 2.3-point gain over a model trained without history. Category-only inputs reproduce much of the swap contrast, and matched histories show small, uncertain advantages over strong negatives on unseen users. Evaluating on users excluded from training and reporting a trained no-history reference are therefore essential checks before interpreting a wrong-user penalty as evidence of user-specific memory. Graded replacements, output-validity audits and per-user nulls then show which parts of that interpretation the data can support.

9

References Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2204.14198. Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. ReZero is all you need: Fast convergence at large depth. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 1352–1361, 2021. URL https://proceedings.mlr.press/v161/ bachlechner21a.html. Yihang Chen, Pin Qian, Su Wang, Chong Peng, Huan Xu, Shuaiting Li, and Yiqi Sun. Explicit state elicitation is not enough: A controlled audit of memory-policy classification. arXiv preprint arXiv:2608.17247, 2026. URL https://arxiv.org/abs/2608.17247. Youwang Deng. Substrate asymmetry in user-side memory: A diagnostic framework. arXiv preprint arXiv:2606.11712, 2026. URL https://arxiv.org/abs/2606.11712. Daniel Deutsch, Rotem Dror, and Dan Roth. A statistical analysis of summarization evaluation metrics using resampling methods. Transactions of the Association for Computational Linguistics, 9:1132–1146, 2021. doi: 10.1162/tacl_a_00417. URL https://aclanthology.org/2021. tacl-1.67/. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022. URL https://arxiv.org/abs/2110.04366. Liam Hebert, Krishna Sayana, Ambarish Jash, Alexandros Karatzoglou, Sukhdeep Sodhi, Sumanth Doddapaneni, Yanli Cai, and Dima Kuzmin. PERSOMA: PERsonalized SOft ProMpt Adapter architecture for personalized language prompting. arXiv preprint arXiv:2408.00960, 2024. URL https://arxiv.org/abs/2408.00960. John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733– 2743, 2019. doi: 10.18653/v1/D19-1275. URL https://aclanthology.org/D19-1275/. Bernd Huber, Ghazal Fazelnia, Andreas Damianou, Sebastian Peleato, Max Lefarov, Praveen Ravichandran, Marco De Nadai, Mounia Lalmas-Roellke, and Paul N. Bennett. Embeddingto-prefix: Parameter-efficient personalization for pre-trained large language models. arXiv preprint arXiv:2505.17051, 2025. URL https://arxiv.org/abs/2505.17051. Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, 2019. doi: 10.18653/v1/N19-1357. URL https://aclanthology.org/N19-1357/. Hong Jeong. Trained persistent memory for frozen decoder-only LLMs. arXiv:2603.22329, 2026. URL https://arxiv.org/abs/2603.22329.

arXiv preprint

Divyansh Kaushik and Zachary C. Lipton. How much reading does reading comprehension require? A critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010–5015, 2018. doi: 10.18653/v1/D18-1546. URL https://aclanthology.org/D18-1546/. Harshini Kavuru, Dwipam Katariya, Giri Iyengar, Pranab Mohanty, Kalanand Mishra, and Raghu Machiraju. REPREC: Representation driven parameter-efficient recommendation system. arXiv preprint arXiv:2607.24845, 2026. URL https://arxiv.org/abs/2607.24845. 10

Abdallah Khemais. Exact network surgery: Functional invariance and gradient plasticity in reactive computational graphs. arXiv preprint arXiv:2607.16568, 2026. URL https://arxiv.org/abs/ 2607.16568. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057–7075, 2020. doi: 10.18653/v1/ 2020.emnlp-main.574. URL https://aclanthology.org/2020.emnlp-main.574/. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021. doi: 10.18653/v1/2021.emnlp-main.243. URL https: //aclanthology.org/2021.emnlp-main.243/. Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, 2021. doi: 10.18653/v1/2021.acl-long.353. URL https://aclanthology. org/2021.acl-long.353/. Jiongnan Liu, Yutao Zhu, Shuting Wang, Xiaochi Wei, Erxue Min, Yu Lu, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. LLMs + Persona-Plug = Personalized LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9373–9385, 2025. doi: 10.18653/v1/2025.acl-long.461. URL https://aclanthology.org/ 2025.acl-long.461/. Minghang Liu, Qiang Qiu, Yuanzhuo Wang, Huawei Shen, and Xueqi Cheng. Less is personal: Learning minimal sufficient user profiles for personalized language models. arXiv preprint arXiv:2609.08180, 2026a. URL https://arxiv.org/abs/2609.08180. Ruitong Liu, Boxu Lin, Peize Li, Siyuan Li, Yunjia Wu, Te Sun, and Chaohan Wu. Beyond prefixes: Graph-as-memory cross-attention for knowledge graph completion with large language models. arXiv preprint arXiv:2510.08966v2, 2026b. URL https://arxiv.org/abs/2510.08966v2. Version 2; version 1 (2025) carries a different title and author list. Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. In Advances in Neural Information Processing Systems, pages 19327–19352, 2023. doi: 10.52202/075280-0848. URL https://proceedings.neurips.cc/paper_files/paper/ 2023/hash/3d77c6dcc7f143aa2154e7f4d5e22d68-Abstract-Conference.html. Lin Ning, Luyang Liu, Jiaxing Wu, Neo Wu, Devora Berlowitz, Sushant Prakash, Bradley Green, Shawn O’Banion, and Jun Xie. User-LLM: Efficient LLM contextualization with user embeddings. arXiv preprint arXiv:2402.13598, 2024. URL https://arxiv.org/abs/2402.13598. Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. How context affects language models’ factual predictions. In Automated Knowledge Base Construction, 2020. URL https://arxiv.org/abs/2005.04611. Aleksandar Petrov, Philip H. S. Torr, and Adel Bibi. When do prompting and prefix-tuning work? A theory of capabilities and limitations. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.19698. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. LaMP: When large language models meet personalization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7370–7392, 2024. doi: 10.18653/v1/2024.acl-long.399. URL https://aclanthology.org/2024.acl-long.399/. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Advances in Neural Information Processing Systems, 2023. URL https: //arxiv.org/abs/2304.15004. Sofia Serrano and Noah A. Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931–2951, 2019. doi: 10.18653/ v1/P19-1282. URL https://aclanthology.org/P19-1282/. 11

Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. Democratizing large language models via personalized parameter-efficient fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6476–6491, 2024. doi: 10.18653/v1/2024.emnlp-main.372. URL https://aclanthology.org/2024.emnlp-main. 372/. Sarah Wiegreffe and Yuval Pinter. Attention is not not Explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20, 2019. doi: 10.18653/v1/D19-1002. URL https://aclanthology.org/D19-1002/. Bin Wu, Zhengyan Shi, Hossein A. Rahmani, Varsha Ramineni, and Emine Yilmaz. Understanding the role of user profile in the personalization of large language models. arXiv preprint arXiv:2406.17803, 2024. URL https://arxiv.org/abs/2406.17803. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.01558. Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In International Conference on Learning Representations, 2019. URL https: //arxiv.org/abs/1901.09321. Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. LLaMA-Adapter: Efficient fine-tuning of large language models with zero-initialized attention. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/ c196239c5f9481e0db2755f31fe4585f-Abstract-Conference.html. Xinliang Frederick Zhang, Nick Beauchamp, and Lu Wang. PRIME: Large language model personalization with cognitive dual-memory and personalized thought process. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33707–33736, 2025. doi: 10.18653/v1/2025.emnlp-main.1711. URL https://aclanthology.org/2025. emnlp-main.1711/. Jingyuan Zheng, Xin Zhang, Yang Gu, Dongjing Wang, Yuxiang Wang, Xudong Shen, Haiping Zhang, Youhuizi Li, and Dongjin Yu. Is personalized modality weighting actually personalized? A controlled audit of per-user weighting claims in multimodal recommenders. arXiv preprint arXiv:2608.05655, 2026. URL https://arxiv.org/abs/2608.05655.

A

Reproducibility details

The supplement contains the training and evaluation harness, the analysis scripts, and saved run outputs with per-item predictions for the core comparisons; its README lists archive coverage. The Random-user mapping hashes the complete profile and applies a seeded derangement over users, shared between training and validation. Prediction files store the item identifier, current user, history user, raw generation and reference, which is what makes the decomposition of §4 possible after the fact. Result files also store source checksums, output-validity audits, training traces, peak GPU memory and the control-mapping checksum. The classification evaluator compares the full stripped generation with the labels found in the complete public train and validation splits, without substring matching. Six details the audit turns on. The prompt is the benchmark’s own input text, tokenized with special tokens and truncated to the decoder’s window, preceded in embedding space by the neutral system string “Answer concisely.”; training concatenates prompt, answer and end-of-sequence with the prompt positions masked out, and evaluation left-pads the prompt alone. The random derangement draws torch.randperm under the control seed and redraws up to a hundred times until no user maps to itself, falling back to a roll by one, which never fired. The similarity-matched derangement solves the assignment problem on the negated cosine matrix with scipy.optimize.linear_sum_assignment, setting the cosine diagonal to −∞ before negation (cost +∞). The composite negative selects items greedily from the four hundred items ranked nearest the target, at each step taking the candidate that 12

brings the running centroid closest to it; repeated constructions replace that argmax with a uniform draw among the five best candidates at each step, seeded per user by the run seed plus the user’s index, which is the stochastic element of the construction. The binned curve uses equal-count bins within each swap type, with user-clustered intervals; the earlier pooled six-bin summary is retained in §3.2. Every bootstrap resamples model seeds with replacement and then users within the drawn seed, weighting each user by its own item count. The revision’s 110 result files record 23.0 GPU-hours: 8.8 for training and 14.2 for evaluation, excluding four training attempts interrupted by out-of-memory errors. Runs used a shared V100SXM2-32GB, PyTorch 2.5.1+cu118, Transformers 4.57.6, NumPy 1.26.4 and SciPy 1.17.1. Logged durations include contention on that device. The new models retain the 3B recipe unless specified: three seeds, 6,000 steps, learning rate 3 × 10−5 . User-fold models use all 4,045 training rows in their fold.

Seen vs. unseen users

2 0 −2

Matched − control (points)

vs. no-history model (points)

B Additional control results

model trained without history

history masked

−4 Random-user donor

−6

Similar-user donor Label-matched composite donor

−8 0

1

2

3

4

5

users seen in training

12.5

unseen users

10.0 7.5 5.0 2.5 0.0

6

Masked

Items replaced by the donor's (k of 6)

Similar

Comp.

Comp., labels

Random

Figure 3: Left: within-user replacement of k out of six items, three-seed means relative to the trained no-history model; the dotted line is masking on the matched-trained model. Lines connect measured doses and carry no uncertainty band. Right: Matched-minus-control contrasts for seen and unseen users, with seed–fold/user bootstrap intervals (Table 2). The dose curves use the same position order and donor for every k within a user. The changes from the matched condition at k = 1, . . . , 6 are −0.15, −0.19, −1.74, −4.49, −6.94, −10.11 points for Random-user; −0.31, −0.39, −0.64, −1.08, −2.51, −3.57 for Similar-user; and −0.08, −0.52, −0.25, −0.60, −0.60, −0.91 for the label-matched composite. Small doses have little effect, but the composite curve is not monotone.

C

Architecture details

C.1

Deepening the slot encoder makes the prefix worse

Would a soft prefix given the same budget close the gap? Both routes share the 109M-parameter input encoder, so we scale the Q-Former depth at fixed input encoder and conditioning length (Table 5, upper block). Accuracy falls monotonically as the Q-Former deepens, from 0.7389 to 0.6685 across a 1.76× increase in trainable parameters, 7.0 points against seed standard deviations of 0.002 to 0.017, and XAttn-1, with fewer trainable parameters than the 8-layer prefix, outscores every prefix row. Loss of output format does not explain it: validity falls by 1.1 points over a range in which accuracy falls by seven. Diminishing gains from longer prompts or larger prefix parameter budgets have been observed in specific settings [Li and Liang, 2021, Lester et al., 2021, He et al., 2022]; Li and Liang [2021] attribute it to overfitting, since longer prefixes lower their training loss. Our training loss moves the other way: its median rises sevenfold as capacity grows, and every injected model fits the training set better than every soft prefix. We conjecture that the deeper prefixes fail to fit rather than overfit; an expressivity ceiling [Petrov et al., 2024] and plain optimization failure are both consistent with the 13

Table 5: Injection route against trainable parameters on LaMP-2, frozen Qwen2.5-3B, learning rate 3 × 10−5 , three seeds, sample standard deviations (for macro-F1 where recorded). Upper block: the prefix’s Q-Former depth scaled at fixed conditioning length, and the prefix with a gated zeroinitialized branch (§C.2). Lower block: XAttn-k with injected layers range(18, 36, s) of the 36-layer decoder; strides s = 18, 8, 4, 2 give k = 1, 3, 5, 9. Each injected layer adds 16,789,505 trainable parameters; the first also adds a 1,579,008-parameter projection of Z. Train loss: median per-step loss over the final third of training (Appendix I), not reported for XAttn-3 and XAttn-5. Trainable

×prefix

Accuracy

Macro-F1

Valid

Train loss

Prefix (2-layer Q-Former) Prefix, 4-layer Prefix, 8-layer Prefix, 16-layer Gated prefix, 2-layer

129,970,944 144,146,688 172,498,176 229,201,152 134,165,249

1.00 1.11 1.33 1.76 1.03

0.7389 ± 0.0059 0.7385 ± 0.0019 0.7180 ± 0.0013 0.6685 ± 0.0174 0.7275 ± 0.0221

0.4896 ± 0.0044 0.484 0.468 0.375 0.478

0.984 0.981 0.982 0.973 0.979

0.0041 0.0039 0.0126 0.0283 0.0087

XAttn-1 XAttn-3 XAttn-5 XAttn-9

148,339,457 181,918,467 215,497,477 282,655,497

1.14 1.40 1.66 2.17

0.7782 ± 0.0097 0.7720 ± 0.0065 0.7803 ± 0.0078 0.7936 ± 0.0048

0.5614 ± 0.0114 0.5651 ± 0.0115 0.5772 ± 0.0182 0.6084 ± 0.0143

0.995 0.997 0.997 0.997

0.0011 – – 0.0012

Model

gradient diagnostics of Appendix J, the usual remedies for the latter (a reparameterization network, a separate prefix learning rate) were not tested, and the result therefore concerns this training recipe rather than soft prefixes as such. C.2

Two axes the parameter count cannot settle

With 84% of the prefix’s trainable parameters in the input encoder, parameter counts alone do not identify the mechanism; two modifications test it, and neither closes the gap. Initialization. A zero-initialized gated residual may simply be easier to optimize than eight soft tokens competing inside the input softmax, so we gave the prefix the same branch (α0 = 1.5, Wout = 0): it reads 0.7275 ± 0.0221 in Table 5, slightly below the plain prefix and 6.6 points below XAttn-9. The branch itself carries a trap that a hybrid hides. ReZero initializes residual gates to zero [Bachlechner et al., 2021], while Fixup initializes the last layer of each residual branch to zero [Zhang et al., 2019], but starting both the gate (α0 = 0) and the output projection (Wout = 0) at zero closes every gradient path, so (α, Wout ) = (0, 0) is a fixed point of gradient-based training, AdamW’s weight decay included; either choice alone is harmless, so the failure appears as an interaction. On Qwen2.5-1.5B the doubly-zero cell scores 0.277, 0.607 and 0.557 across seeds, with adapter gradient and gate exactly zero in the recorded diagnostic, yet the hybrid’s prefix path keeps training, so a hybrid with a dead branch still improves on the benchmark. Appendix E gives the derivation, proved for function-preserving growth by Khemais [2026], and the ablation; Appendix G describes a second trap in the deep key-value prefix. Conditioning length. Prefix length is the axis the expressivity bound of Petrov et al. [2024] cares about, and positions come almost free: 8 to 64 slots costs 43,008 parameters, 0.03% of the trainable total. Accuracy at seed 42 reads 0.7445, 0.7165 and 0.7439 at 8, 32 and 64 slots, a non-monotone spread smaller than the gap to the injected model. An earlier pure-cross-attention pilot at 0.5B, with the input prefix removed, scores 0.6530, 0.6611 and 0.6704 at seeds 7, 42 and 123. It uses 4,000 training examples, 600 steps, learning rate 10−4 and 1,605 evaluation examples. Its scale and schedule differ from the main comparison, so it cannot isolate the prefix contribution at 3B.

D

Hyperparameters

Table 6 lists every setting needed to reproduce the runs. Values not listed are library defaults. A six-point learning-rate sweep from 10−3 to 10−5 (unaudited harness, 400 items, used only to choose the rate) places the optimum for both routes at an interior 3 × 10−5 , where the prefix reads 0.7300 against 0.7025 at 10−4 and 0.6850 at 10−5 ; both routes were then re-run at both candidate rates on the audited harness. Tuning the rate widens rather than closes the prefix–cross-attention gap: at seed 14

42 the prefix reads 0.7221 at 10−4 and 0.7445 at 3 × 10−5 , XAttn-9 reads 0.7601 and 0.7882, and the gap moves from 3.8 to 4.4 points (macro-F1: 7.6 to 9.7) in single runs, hence below the three-seed 5.5 of §6.1. Table 6: Complete hyperparameter listing. Setting

Value

History encoder Control encoder Input encoder Q-Former Projection width Optimizer Schedule Precision Model seeds Control seed

BAAI/bge-base-en-v1.5, frozen, max 512 tokens per item BAAI/bge-base-en-v1.5; robustness checks repeat the similarity assignment and the composite with intfloat/e5-base-v2 separate BGE encoder; all parameters trainable 8 queries, 2 layers, width 768 unless varied in the sweep 1024 hidden units AdamW, weight decay 10−3 , gradient clipping at 1.0 linear with 0.05 warmup ratio fp16 with loss scaling; SDPA attention; gradient checkpointing on 7, 42, 123 20260723, independent of the model seed

Control battery, 3B (§3, §4) Decoder Data Evaluation Learning rate Controls

Qwen2.5-3B-Instruct, frozen; XAttn-9 8,000 train examples, 6,000 optimizer steps, batch size 1 all 1,605 validation items, 323 users 3 × 10−5 , the optimum selected in §2.3 every history evaluated on each matched checkpoint

Control battery, 0.5B (§3.6, §4) Decoder Data Evaluation Learning rate Shorter-schedule comparison

Qwen2.5-0.5B-Instruct, frozen; gated cross-attention 8,000 train examples, 6,000 optimizer steps, batch size 1 all 1,605 validation items, 323 users 3 × 10−5 ; shared recipe, not a separate 0.5B tuning sweep 4,000 examples, 600 steps, 10−4 (the schedule confound of §3.6)

Audited model comparison (§C.2) Decoder Models Seeds Serialization control

Qwen2.5-3B-Instruct, frozen; the recipe of the 3B control battery soft prefix at 8, 32 and 64 Q-Former queries; XAttn-9 7, 42, 123 for the 8-slot prefix and XAttn-9; 42 for the others identical run with the constant MOVIE: label stripped from every profile item, seed 42

Initialization ablation (§C.2, Appendix E) Decoder Qwen2.5-1.5B, frozen Data 4,000 train examples, 600 optimizer steps, batch size 1, 300 evaluation items Learning rate 10−4 Injection-site and tuning experiments (§6.1, §C.1) Decoder Qwen2.5-3B, frozen Data 8,000 train examples, 6,000 optimizer steps, batch size 1 Evaluation 1,605 items for audited comparisons; 400 for the learning-rate grid Learning rate 3 × 10−5 unless the sweep varies it Injection layers range(18, 36, stride) over the 36-layer decoder; strides 2, 4, 8, 18 give 9, 5, 3, 1 layers

The other-task battery uses matched-trained 3B cross-attention checkpoints: 8,000 training rows, 6,000 steps, batch size 1, learning rate 3 × 10−5 . LaMP-1 and LaMP-4 use seeds 7 and 42; LaMP-3 and LaMP-5 use 7, 42 and 123. Evaluation uses 1,500 items except LaMP-3, which uses 1,605. Masked and random histories are evaluated on the corresponding matched checkpoint.

E Initialization trap: derivation and ablation Proposition. Write the adapter branch of Eq. (1) as A(h, Z) = Wout C(h, Z) with C(h, Z) = CrossAttn(LN(h), Z), so that h′ = h + tanh(α)A(h, Z), and let g = ∂L/∂h′ . For any parameter θψof Cψor of W out , D ∂A E
∂L ∂L = tanh(α) g, , = 1 − tanh2 (α) g, A(h, Z) .ψ (2) ∂θ ∂θ ∂α At α = 0 the first expression is zero for every θ; at Wout = 0 we have A ≡ 0, so the second is zero as well. Under any optimizer whose update depends on the gradient alone, both parameters 15

therefore keep their initial values at every step, and (α, Wout ) = (0, 0) is a fixed point the branch cannot leave. Decoupled weight decay does not lift it: it leaves Wout = 0 at zero and α = 0 at zero, so AdamW, which we use, leaves the branch dead. Neither condition alone is degenerate. With α = 0 and Wout random, A ̸= 0 makes ∂L/∂α generically non-zero, the gate moves off zero and the adapter gradients unlock; with Wout = 0 and α0 = 1.5, tanh(α) ̸= 0 keeps the adapter gradients open and Wout leaves zero on the first step. Only the conjunction deadlocks, which is why the failure appears as an interaction rather than a main effect of either initialization, and a parallel soft-prefix path keeps training throughout, which is how a hybrid model improves on the benchmark with a dead branch. The algebra is not new; Khemais [2026] prove the same configuration for function-preserving network growth. We contribute the measurement of what the failure costs and how it is detected. Table 7: Initialization ablation on LaMP-2 with Qwen2.5-1.5B, 600 steps, 300 evaluation examples, three seeds per cell, sample standard deviations. The doubly-zero branch has exactly zero gradient; the short-run accuracy differences are not a stable effect-size estimate. Gradient norm and gate magnitude are measured at step 590 of the seed-42 run. Wout = 0

Wout random

0.480 ± 0.178 0.726 ± 0.012

0.677 ± 0.033 0.711 ± 0.031

0.0 / 5.7 0.000000 / 0.904092

2.1 / 7.4 0.006787 / 0.903341

Accuracy, α0 = 0 Accuracy, α0 = 1.5 Adapter gradient norm, α0 = 0 / 1.5 Gate | tanh α|, α0 = 0 / 1.5

The doubly-zero cell scores 0.277, 0.607 and 0.557 at seeds 7, 42 and 123, respectively. At step 590 of seed 42, its adapter gradient norm and gate magnitude are exactly zero. The large spread in accuracy comes from a model whose parallel prefix still trains; a single run can therefore conceal the dead branch. This short ablation establishes the failure mode, not a reliable accuracy gain from changing initialization. We use the gate magnitude deliberately narrowly. Gate magnitudes do not measure how much a branch contributes, because pre-gate activation scales vary with depth, and Alayrac et al. [2022] say as much. Exact zero against non-zero asks something else, namely whether the parameter moved at all, which is a claim about optimization rather than influence.

F

What an unaudited harness does to one model

A harness that records no output validity changes the comparison asymmetrically. On the identical subset and seed it reproduces cross-attention to the digit, 0.8375, while placing the soft prefix at 0.7300 against the audited 0.7750 and its macro-F1 at 0.212 against 0.498. A model that looks as if it had learned the label prior and little else is being mis-parsed, and the gap between the two shrinks from 10.75 to 6.25 points once the parser is audited. This is why every prefix row in the main text was retrained under the audited harness. 0.85

Unaudited parser

0.838

0.838

Accuracy

Audited (exact label match)

0.80 0.775

0.75

0.730

0.70 Prefix

XAttn

Figure 4: Accuracy before and after auditing the parser, identical subset and seed: prefix 0.7300 to 0.7750, cross-attention 0.8375 in both cases. This diagnostic subset is separate from the full-validation architecture comparison.

16

(a) Prefix

(b) XAttn-k: prefix retained + k gated layers

(c) Deep KV prefix

Frozen decoder

Frozen decoder

Frozen decoder K/V prefix

Z

Z

Gated cross-attn (×k)

Z

K/V prefix

Soft prefix

Soft prefix

K/V prefix

Query embeddings

Query embeddings

Query embeddings

Figure 5: Memory injection routes; all three share the encoders and the Q-Former that produce Z ∈ R8×768 from the history items and the current query. (a) Prefix: Z is projected and prepended to the input embeddings. (b) XAttn-k: the prefix is retained and k decoder layers receive the gated update of Eq. (1); this hybrid is the cross-attention model throughout. (c) Deep key-value prefix, an appendix experiment. Decoder and history-item encoder are frozen; input encoder, Q-Former and injection modules train.

G Injection routes and the deep key-value prefix Figure 5 draws the two routes of the main text next to the third. The deep key-value prefix writes Z into key-value prefixes at several decoder layers. Under this implementation it carries a second initialization trap, unrelated to the gate deadlock of Appendix E: its final projection starts at zero, so all m slots begin identical and receive identical updates, and after 600 steps the maximum difference between slots is exactly zero, so lengthening the prefix adds copies of one state. Attention mass on the prefix rises from 0.191 at m = 2 to 0.296 at m = 32 while accuracy stays at 0.641, 0.641 and 0.645. With m identical slots the softmax mass on the prefix grows with m by construction, so the rise is expected and says nothing about use. Rising attention mass is not evidence of dilution here, since attention weights alone can misrepresent importance [Jain and Wallace, 2019, Serrano and Smith, 2019]. Wiegreffe and Pinter [2019] argue that explanatory value depends on the definition and on diagnostic tests; Kobayashi et al. [2020] show why transformed-value norms must also be considered.

H History as text, and the 0.5B decoder Supplying the same four histories as text to the frozen decoder, with no training and no compression, gives 0.6617 Matched, 0.6386 Similar-user, 0.6069 History masked and 0.5832 Random-user on LaMP-2. Absolute numbers are not comparable with the adapter, which is trained on the task, but the ordering and the relative sizes are (Appendix H). The same battery on a frozen Qwen2.5-0.5B trained to the recipe of Appendix D gives 0.7751 Matched, 0.7304 Similar-user, 0.7232 History masked and 0.6467 Random-user: Matched minus Similar-user is 4.5 points [2.6, 6.6], Similar-user minus History masked 0.7 points [−1.1, 2.7], and History masked minus Random-user 7.6 points. A 0.5B checkpoint trained on the shorter schedule (4,000 examples, 600 steps, learning rate 10−4 ) shows almost no penalty under any replacement; the large discrepancy shrinks when the recipe is matched, although paired permutation tests still distinguish the scales (§4). On the other tasks the same text histories hurt: LaMP-3 mean absolute error is 0.745 with no history against 1.032 with the matched history and 1.574 with a random one, mostly through format, since the numeric-output rate falls from 0.973 to 0.90 while the error on in-range outputs alone stays between 0.666 and 0.719, and LaMP-5 ROUGE-1 falls from 0.353 without a history to 0.188 with the matched history.

I

Training-loss estimates

At batch size 1 a per-step loss is heavy-tailed, so the tail mean of a training log is a noise estimate rather than a fit measurement. The result files keep every tenth step; over the final third of training, 200 logged points per run, the mean and median across three seeds are: prefix 2-layer 0.489 ± 0.079 and 0.0041; 4-layer 0.485 ± 0.075 and 0.0039; 8-layer 0.582 ± 0.054 and 0.0126; 16-layer 0.724 ± 0.176 and 0.0283; gated prefix 0.529 ± 0.074 and 0.0087; XAttn-1 0.366 ± 0.024 and 0.0011; XAttn-9 0.330 ± 0.111 and 0.0012. The within-run standard error of the mean is near 0.08 for every model, 17

which is the size of the between-model differences it would have to resolve, so Table 5 reports the median. Both statistics place the injected models below the prefix models in training loss.

J

Expressivity against optimization, and gradient flow into the Q-Former

Li and Liang [2021] report that longer prefixes eventually reduce training loss but slightly worsen test performance, which is consistent with overfitting, and Lester et al. [2021] find diminishing gains and mild degradation at longer prompt lengths. The decline in §C.1 has a different signature, a training loss that rises with capacity, and two mechanisms could produce it. One is an expressivity ceiling. Within one attention head with fixed content representations, a prefix preserves relative content attention and adds a bias in a subspace of dimension at most its length [Petrov et al., 2024]; this is not a decoder-wide bound, since later-layer attention can change, and §C.2 shows that eight times the conditioning positions does not move the prefix. The other is plain optimization failure, of which a rising training loss is the usual signature. The gradient diagnostics below are consistent with both. The models use the same Q-Former design, so we can ask how much learning signal each injection route delivers to it. During training we log the gradient norm reaching the Q-Former and the gradient norm reaching the injection module, and divide each by the square root of that module’s parameter count so that the comparison does not merely reflect module size (Table 8). Table 8: Root-mean-square gradient per parameter reaching the Q-Former and the injection module, averaged over the final quarter of training, seed 42. Model Prefix, 2-layer Prefix, 16-layer XAttn-1

Q-Former params 18,909,696 118,139,904 18,909,696

Injection params 1,579,008 1,579,008 19,947,521

Q-Former −3

1.82 × 10 5.36 × 10−4 1.03 × 10−3

Injection 9.28 × 10−3 1.35 × 10−2 1.75 × 10−3

The Q-Former-to-injection ratios are 0.196, 0.040 and 0.589. The cross-attention ratio is about fifteen times the deep prefix’s ratio, whereas its Q-Former gradient itself is only about 1.9× as large (1.03 × 10−3 against 5.36 × 10−4 ). Deepening the prefix reduces the Q-Former value by about 3.4× and increases the injection-module value. These diagnostics accompany the three-seed macro-F1 decline from 0.490 to 0.375 in Table 5; they do not establish that one module causally absorbs another’s learning signal. We report this as one measurement at one seed on one task. It is consistent with the expressivity account of Petrov et al. [2024] and equally with optimization failure, and separates neither.

K Position relative to prior and concurrent controls Table 9 sets this paper’s control battery beside the replacement controls used by the closest prior and concurrent work. The cells summarize each paper’s own description; a dash means the item is not reported there. PRIME swaps user-specific LoRA weights as well as text, so its effective checkpoint changes. The comparison column includes non-personalized baselines when they differ from an input mask.

L Per-seed and per-construction contrasts Table 10 unpacks the means of Table 1 by seed and adds the alternative constructions of the negative, so that the spread behind every headline number is visible. Entries are differences of the per-seed accuracies in Table 1, in points; the bootstrap point estimates quoted in the main text can differ from the mean of the three seeds in the last digit. The seed-level values qualify the averages. Here harm share is (masked − random)/(matched − random), and residual share is (matched − negative)/(matched − masked). The residual share for the Similar-user control ranges from 0.53 to 0.90 across seeds, so its mean is sensitive to the seed and to the masked reference; and seed 42, the seed used for the single-run comparisons elsewhere in the paper, is the one on which every contrast is smallest. 18

Table 9: Replacement controls in prior and concurrent work and in this paper. Work

Setting

Replacement

Graded by similarity

Fixed checkpoint

Contrast vs. Negative no or rebuilt masked history

LaMP [Salemi et al., 2024]

retrieval of text items

two levels

zero-shot and fine-tuned

yes

–

Wu et al. [2024]

retrieval of text items (Flan-T5 fine-tuned; frozen Llama-2) textual profile plus per-user LoRA

random items from the user’s own profile, or from the whole corpus query-relevant items from a pool of other users’ items

trained per augmentation

yes

–

no: LoRA swapped

Non-P

–

yes

yes

matching audit

yes

–

–

yes

yes, masked two and trained encoders, three greedy builds

PRIME [Zhang et al., 2025] ENOUGH [Liu et al., 2026a]

Zheng et al. [2026]

This paper

by query relevance, not user similarity whole profile and four levels LoRA of another user (most similar to dissimilar) matched on retrieval of text one record replaced records, frozen by a content-matched context, other-user record; also relevance, generator random user and length, time whole-profile swap multimodal shuffle (permutation) dose– recommenders of per-user weights, response plus a population-weight contrast and a signal-implant dose–response eight-vector whole profile: random similarity compressed memory user, and dose around a frozen similarity-matched curves; decoder, and raw user, composite of category text other users’ items matching

The composite construction grid (original, rebuild 2, rebuild 3) is (1.18, 0.81, 1.50), (1.87, 0.62, 1.81) and (2.12, 1.18, 1.43) points in seed order 7, 42, 123. Resampling seeds, constructions within seeds, and users gives 1.39 points with interval [0.74, 2.05]. The trained no-history comparison is between separately trained checkpoints: Matched minus this comparator is 2.06, 2.12 and 2.62 points.

M

Algorithms

The following procedures define the replacement histories and null distributions. Profile embeddings are BGE embeddings of the serialized items; a profile centroid is the mean of its item embeddings. Similar-user derangement. Form the U × U matrix of cosines between profile centroids, set the diagonal to −∞, and solve the linear assignment problem that maximizes the total cosine (scipy.optimize.linear_sum_assignment on the negated matrix). The solution is a permutation with no fixed point; each user receives the history of the user it is assigned to. The same mapping is used at every seed. Composite negative. For a target profile with centroid c: (1) rank all items owned by other users by cosine to c and keep the 400 nearest as the candidate pool; (2) starting from the empty set, add six times the candidate that brings the centroid of the selected items closest to c; (3) serialize the six items as a history. The stochastic variant replaces the argmax in step (2) by a uniform draw among the five best candidates, seeded by the run seed plus the user’s index, which is how the rebuilds of Table 10 were produced. For the cosine-0.90 composite, all other-user items are candidates and each greedy step minimizes the absolute difference between the running centroid cosine and 0.90. Label-matched composite. Recover the category of each of the first six raw profile items. For each position, exclude the target user’s items and any item whose article text equals an evaluation query. 19

Table 10: Contrasts by seed (points) on the 3B XAttn-9 checkpoints, and descriptive ratios relative to masking (not the trained no-history comparator). The last block gives the alternative constructions of the negative: two greedy rebuilds of the composite at seed 42, and the Similar-user and composite negatives rebuilt under the E5 encoder (three-seed means). Contrast

Seed 7

Seed 42

Seed 123

Mean

Matched − History masked (gain over masked history) History masked − Random-user (harm) Matched − Random-user Matched − Similar-user Matched − Composite (cos = 0.90) Matched − Composite (cos = 0.95) Similar-user − History masked

5.5 5.2 10.7 4.2 3.8 1.2 1.2

4.4 4.4 8.8 2.4 2.9 0.8 2.1

4.5 6.2 10.8 4.1 3.9 1.5 0.4

4.8 5.3 10.1 3.6 3.5 1.2 1.2

Harm share Residual share, Similar-user Residual share, Composite (cos = 0.95)

0.49 0.77 0.22

0.50 0.53 0.18

0.58 0.90 0.33

0.52 0.74 0.24

Alternative constructions of the negative Matched − Composite, greedy rebuilds at seed 42 Matched − Similar-user under E5 Matched − Composite under E5 (mean cosine 0.978)

0.8 (original), 0.6, 1.2; mean 0.9, SD 0.3 4.0 (three-seed mean) 1.7 (three-seed mean)

Table 11: Matched minus each control, accuracy points. The t-interval uses three seed-level contrasts; the bootstrap resamples seeds and users (10,000 replicates). Control

Seed 7

Seed 42

Seed 123

t, 2 df

Bootstrap

Random-user Similar-user Composite, 0.95 Composite, labels History masked Own labels only

10.72 4.24 1.18 1.31 5.48 2.93

8.85 2.37 0.81 0.37 4.42 1.93

10.78 4.11 1.50 1.06 4.55 3.24

[7.39, 12.84] [0.98, 6.17] [0.31, 2.02] [−0.29, 2.12] [3.38, 6.26] [1.00, 4.40]

[8.01, 12.30] [2.09, 5.23] [0.40, 1.89] [0.09, 1.79] [3.41, 6.34] [1.25, 4.21]

Among items of the required category, retain the 400 nearest to the target centroid and greedily select the unused item maximizing the running centroid cosine. Retain the donor title and its own category; no query answer is used. Record fallbacks if a category has no candidate (none occurred). Labels-only and dose controls. Serialize each category as MOVIE: "<category>", using either the matched profile’s labels or the random donor’s. For dose replacement, seed a permutation of six positions by 20261017 plus the sorted user index, and replace its first k positions with the corresponding positions from a fixed random, similar or label-matched donor. The subsets are nested and shared across model seeds. The endpoints reuse the matched and full-replacement evaluations. Zero-effect null. We use 2,000 replicates for each null. For each validation item, exchange the outcomes recorded under the two conditions with probability one half, independently across items; recompute ∆u for every user and then the Gini of |∆u | and the standard deviation of ∆u ; repeat. The permutation probability is the fraction of replicates whose statistic is at least the observed one. Homogeneous-effect null. Pool the item-level outcome pairs of all users; for each user draw with replacement as many pairs as the user has items; recompute the same statistics; repeat. This keeps the population effect and the item counts and removes only the assignment of items to users.

N

Applying the audit to another system

The fixed-checkpoint battery requires an evaluation split and an embedding model; the trained no-history and user-disjoint comparisons additionally require training. 20

1. Audit train/evaluation user overlap and reserve users for a disjoint evaluation. Train a no-history comparator under the same recipe. 2. Check branch gradients at initialization and report exact-label parsing and valid-output rates. 3. Evaluate fixed checkpoints with matched, random, similar-user, composite and masked histories. When labels exist, add labels-only and label-matched controls; never use the query’s answer to construct a donor. 4. Report matched-minus-control contrasts separately from the gain over the trained no-history model. Neither a swap penalty nor masking alone measures deployment benefit. 5. Rebuild negatives across encoders and construction seeds. Include construction variability in uncertainty, and cluster similarity curves by user. 6. Replace nested subsets of history items to measure within-user responses. Check query/profile duplicates and evaluate per-user concentration against nulls at the observed query counts.

O Completed controls and remaining tests The revision adds positional category matching, labels-only histories, article-level near-duplicate filtering, nested within-user replacement, a complete seed-by-construction grid, user-disjoint training, and a movie-tagging pilot. Sections 3.4–3.5 report their results. These checks leave uncertainty about small gains on new users and about the representation learned from a stable donor profile. More seeds, longer histories, and a trained no-history comparator within each user fold would strengthen the generalization test. For the architecture comparison, a pure cross-attention arm at 3B, projection-width sweeps and separate prefix optimization remain untested.

21

Related documents

• Alignment by Confident Debate: Preference Learning from Multi-Agent Debate#1598915
• CoMoFlow: Controllable Protein Conformational Ensemble Generation via Explicit Collective-Mode Modeling#1598916
• CID: Measuring Cognitive Work Through Semantic State Change#1598917
• Multimodal Training with Knowledge Traction for Video Anomaly Detection#1598918
• PreForceVision: Physics-Guided Optical Estimation of Bolt Preload from Bolt-Head Deformation#1598919