Conceptio › Archive › arXiv CS

arXiv CSopen access

Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers∗ Hao Yan†1 , Haolin Yang2 , Yiqiao Zhong†1

arXiv:2605.03780v1 [cs.LG] 5 May 2026

University of Wisconsin–Madison 2 University of Chicago

Abstract Transformers are effective at inferring the latent task from context via two inference modes: recognizing a task seen during training, and adapting to a novel one. Recent interpretability studies have identified from middle-layer representations task-specific directions, or task vectors, that steer model behavior. However, a lack of rigorous foundations hinders connecting internal representations to external model behavior: existing work fails to explain how task-vector geometry is shaped by the training distribution, and what geometry enables out-of-distribution (OOD) generalization. In this paper, we study these questions in a controlled synthetic setting by training small transformers from scratch on latent-task sequence distributions, which allows a principled mathematical characterization. We show that two inference modes can coexist within a single model. In-distribution behavior is governed by Bayesian task retrieval, implemented internally through convex combinations of learned task vectors. OOD behavior, by contrast, arises through extrapolative task learning, whose representations occupy a subspace nearly orthogonal to the task-vector subspace. Taken together, our results suggest that task-vector geometry, training distributions, and generalization behaviors are closely related.

Introduction

Large language models (LLMs) exhibit impressive performance, yet the mechanism underlying their generalization remains unclear. Since the emergence of in-context learning, empirical studies have suggested two modes of task inference [4, 25]: memorization-based task retrieval, where the model updates an implicit posterior over in-distribution (ID) training tasks and retrieves previously learned behaviors, and context-based task generalization, where it solves out-of-distribution (OOD) tasks whose latents lie outside the training support. However, the internal representations supporting these two modes remain obscure. This motivates two central questions: How do hidden states summarize context and retrieve memorized tasks? How does the representation geometry change for novel tasks outside the training support? To understand these external behaviors, researchers have analyzed internal representations, broadly centering on the linear representation hypothesis (LRH) [23, 22, 27], which posits that hidden states encode concepts as linear subspaces. For example, [16] discovered that in ICL, LLMs use in-context examples internally to form task-encoding vectors, commonly called task vectors. ∗ †

Code is available at https://github.com/ezyhdxm/mini-ICL. Correspondence to: [email protected], [email protected].

latent (task) space

observed data

hidden-state geometry at layer Ortho subspace

draw sequence

Task subspace

for OOD for ID

latents mapped to task vectors given

, computes

Figure 1: Connections between data distribution and representation geometry. This diagram illustrates the synthetic rolling biased dice experiment. Each sequence is generated from an unobserved latent z controlling the outcome distribution. A transformer trained on mixture data learns internal task vectors that encode these latents. Two near-orthogonal subspaces largely determine a model’s inference mode and generalization behavior.

Concretely, given a prompt s = (x1 , f (x1 ), x2 , f (x2 ), . . . , xn , f (xn ), xtest ), an LLM can infer the mapping f from the demonstrations and correctly predict f (xtest ). Internally, the hidden state in a middle layer contains a task-specific component θf that steers the model toward the correct continuation. Challenges: lack of rigorous foundations for representation geometry. While prior work on internal representations and task vectors provides valuable insights [14, 3], the mechanistic analysis is often model-dependent and its applicability is rarely verified. For example, various notions of task vectors are proposed [16, 39, 21], sometimes leading to contradictory results in the literature. A central missing piece is a theoretical framework that rigorously defines task vectors and clarifies how they relate to the geometry of internal representations. Without such a foundation, existing results are largely heuristic: task vectors are often introduced operationally or extracted empirically, but their precise meaning, validity, and connection to representation structure remain unclear. As a result, current analyses fail to provide a clear mechanistic picture of how pretrained LLMs internally encode task information, how this encoding is shaped by the training data distribution, and how internal representations mediate the link between external statistical structure and model behavior. Bridging the gap via latent-variable modeling. To develop foundations for task vector representation geometry grounded in rigorous statistical principles, we mainly study synthetic experiments where we train small transformers from scratch on carefully controlled data. As shown in Figure 1, at a high level, we view tasks as latent variables (or simply, latents) in a space Z. We model training data as a latent-task mixture: for each sequence, we first sample an unobserved latent z ∈ Ztrain ⊂ Z and then sample the sequence from its task-specific distribution pz . Echoing the “multitask learner” heuristic of GPT-2 [31], this latent-variable view is standard in classical text analysis (e.g., topic models) and has recently provided useful insights on ICL [46, 13], induction heads [24, 10], etc. This setup allows us to broadly define task vectors as mappings of latents, and characterize how a model infers tasks from interactions between hidden states and task vectors. We adopt a “gray-box” perspective that assumes access only to hidden states, making our analysis agnostic to architectural details. Our approach unifies prior work on representation geometry [34, 33] and inference modes [6, 20, 26], and identifies a counterexample that exposes a failure case of existing heuristic approaches to defining and extracting task vectors, thereby illustrating the value of our rigorous characterization 2

of task vectors (Sec. 7). Our main findings are summarized below. 1. A principled framework unifies representation geometry and generalization. We mathematically define task vectors as representations of latent tasks, and characterize two distinct inference modes: Bayesian task retrieval and extrapolative task learning (Sec. 4). 2. Task-vector geometry governs Bayesian task retrieval. For ID sequences, the model infers latent tasks primarily by adjusting the coefficients of task vectors in its hidden states, which eventually converge to the true task vector as sequence length grows (Sec. 5). 3. Near-orthogonal subspace governs extrapolative task learning. For OOD sequences, the model can often generalize by using context-related statistics encoded through nearorthogonal subspaces, which emerge only at sufficiently large task diversity (Sec. 6).

Related work

Task vectors in language models. Task vectors have emerged as a prominent paradigm for explaining how language models extract task information from context and use it to steer subsequent predictions. However, where such vectors should be obtained from, and which internal activations truly constitute task vectors, remains underexplored and still somewhat controversial. Existing work typically extracts task vectors using heuristic procedures such as dummy task prompts [16], averaging across multiple task prompts [39], or opaque optimization-based selection [17], and does so from sources including intermediate ICL hidden states [18], attention-head outputs [47], and MLP activations [21]. However, these construction procedures are largely ad hoc: candidate vectors are typically selected from a small set of layers or components and validated mainly by intervention outcomes, rather than derived from a principled characterization of the underlying data distribution or representation geometry. Bayesian inference and OOD generalization. Early empirical studies observe that LLMs can operate in two modes: retrieve seen tasks and learn new tasks in context [25, 44]. Explanations such as Bayesian beliefs [46, 34] and induction heads [11, 24, 36] focus on either ID or OOD generalization, without reconciling the coexistence of both inference modes. Synthetic experiments in [13, 6, 20, 26] reveal that task diversity in training shapes the inference mode in specific settings, yet no analysis of internal representations is provided. Near-orthogonal representation geometry. A central idea in mechanistic interpretability (MI) is the superposition principle [12]: models often accommodate a large number of feature vectors by arranging them in near-orthogonal positions, echoing the classical literature in sparse coding [8] and compressed sensing [7, 5]. Near-orthogonal geometry is empirically confirmed in transformers [35, 33] and serves as the foundation for MI work [3, 38, 28, 15]. However, near-orthogonality has not been established as a geometric foundation for the coexistence of ID and OOD inference modes.

Synthetic experiment setup

Data generation. In all synthetic experiments, we first draw a latent z from a finite training set Ztrain ⊂ Z and then sample a sequence from the latent-specific distribution. 3

E1. Rolling biased dice. Each latent z corresponds to a probability vector p drawn from a Dirichlet prior Dir(16 ). For each sequence, we first sample p and then sample tokens independently from p. E2. In-context linear regression. Following [32], we take the latent z := w ∼ N (0, ID ) as the D-dim regression weight vector (D = 6). For each sequence, we first sample w and then T /2 input-output pairs (treating each vector/scalar as a token) x1 , y1 , x2 , y2 , . . . , xT /2 , yT /2 , where xt ∼ N (0, ID ), yt = x⊤ t w + εt , and εt ∼ N (0, 0.25) . E3. Mixture of Markov chains. A latent z := P ∈ RV ×V (V = 6) is a transition matrix for a first-order Markov chain. As in [10], we draw rows of P independently from a Dirichlet prior Dir(1V ), then sample the Markov sequence s1:T using P , starting from a uniform initial distribution. Remark 1 (on Markov property). In synthetic setup E1–E3, sequences satisfy the first-order Markov property given z: p(st+1 | z, s≤t ) = p(st+1 | z, st ). Thus we expect a performant model to infer the unobserved z by internally compressing the context into a ‘belief ’ about z. Model and training. We use a decoder-only transformer with rotary positional embedding (RoPE) [37]. For experiments E1 and E3, the transformer has 6 layers, 2 heads, and hidden dimension d = 128; for E2, we follow [2] by increasing the number of layers to 16 (other hyperparameters remain the same). All models are trained autoregressively with AdamW [19]; a complete list of hyperparameters is provided in Sec. A.

A mathematical framework for representation geometry

This section develops the mathematical framework used throughout the paper. Sec. 4.1 defines task vectors and four geometric properties of hidden states; Sec. 4.2 links these properties to Bayesian task retrieval, while Sec. 4.3 studies extrapolative task learning and shows when it must move beyond the task-vector subspace. Secs. 5–6 provide empirical evidence for the two-mode picture.

4.1

Theoretical properties of task-vector geometry

Let z ∈ Z be the unobserved latent and s1 , s2 , . . . be the observed sequence generated from the (ℓ) distribution Pz . For a fixed transformer layer ℓ, denote by ht ∈ Rd the residual-stream hidden state at position t after processing the prefix s≤t . Throughout this section, we fix a layer and suppress the superscript ℓ. Since the results below do not depend on the specific parameter space of any experiment, we identify the training latents z with the index set [K] := {1, . . . , K} and write µt := E[ht ] for the mean hidden state at position t. As noted in Sec. 3, tasks E1–E3 satisfy the first-order Markov property: P(st+1 | z, s≤t ) = P(st+1 | z, st ). Under the Markov property, P(st+1 | z, s≤t ) depends only on (z, st ), so a well-trained model’s hidden state should likewise be approximately determined by these two factors. Define cell means µt,k,a := E[ht | z = k, st = a] and write ht = µt,z,st + εt ,

(1)

where εt := ht −µt,z,st is the residual not explained by (z, st ). We state four properties characterizing the hidden-state geometry when task retrieval succeeds. 4

P0. Long-context stability. The residual in Eq. 1 vanishes asymptotically: E∥εt ∥2 → 0 as t → ∞. In other words, the hidden state becomes a deterministic function of (z, st ), meaning the model has successfully summarized the context. While heuristic task-vector extractions [39, 16] implicitly assume P0, we show in Sec. 7 that this property is not always satisfied. P1. Long-context decoupling. Given P0, we can further ask how the cell means µt,k,a are structured. The task-specific component naturally gives rise to the definition of task vectors [16]. Definition 1. For a given layer, task vectors are {θk }k≤K ⊂ Rd defined by θk = lim E[ht | z = k] − µt , t→∞

whenever the limit exists. Centering ensures

(2)

k≤K θk = 0.

P1 requires that the cell means decompose additively into task and token components. There exist token-encoding vectors {νa }a∈V ⊂ Rd such that E ht −

µt |{z}

global mean

θz |{z}

task vector

νst |{z}

2 2

→0

as t → ∞.

(3)

token encoding

This additive structure is a form of the linear representation hypothesis [27]; in ANOVA terms, P1 requires that the interaction between z and st is negligible. P2. Finite-context interpolation. At finite context, the model represents its knowledge about latent tasks via P a convex combination of task vectors. There exist nonnegative coefficients {βt,k }t≥1, k≤K satisfying k≤K βt,k = 1 (depending on context s1 , . . . , st ) such that ht ≈ µt +

βt,k θk + νst .

(4)

k≤K

P2 formalizes the superposition principle [12]: hidden states are linear combinations of task vectors. P3. Bayesian posterior alignment. P3 strengthens P2 by pinning down the same coefficients βt,k : they approximate the Bayesian posterior, βt,k ≈ αt,k := P(z = k | s≤t ).

(5)

This provides a mechanistic characterization of Bayesian inference: while the Bayesian perspective has been used to explain ICL [46], its internal geometry has remained unclear. These properties are progressively stronger: P3 ⇒ P2, P1 and P1 ⇒ P0 in the long-context limit; see Sec. B.1.

4.2

Task retrieval: task-vector geometry mirrors Bayesian inference

When a model has memorized a task z during training, it can retrieve the latent task z from test prompts via approximate Bayesian prediction; we call this the Bayesian task retrieval mode (M1). Formally, let πtrain be the prior distribution over Ztrain ⊂ Z. The optimal prediction under M1 is Z pz (s≤t ) πtrain (z) P(st+1 = a | s≤t ) = P(st+1 = a | z, st ) p(z | s≤t )dz, p(z | s≤t ) = R , (6) ′ Z Z ′ pz (s≤t ) πtrain (z) 5

where Z ′ = Ztrain and p(z | s≤t ) is the exact αt,k in Eq. 5. By Eqs. 4 and 5, ht carries the same posterior weights as Eq. 6: after a shared centering map removes µt , the informative component P is k≤K αt,k (θk + νst ). Thus Bayesian retrieval reduces to a finite lookup-and-mix computation: lookup P(· | z = k, st ) from the task/token anchors and mix the results with αt,k . Sec. B.2 gives the constructive proof. Theorem 1 (Informal; task-vector subspace can implement Bayesian inference). Suppose that at some layer ht satisfies P0–P3. Under the structural conditions formalized in Sec. B.2, two additional transformer blocks and a suitable unembedding can make the ID prediction distribution approximate the Bayesian predictive distribution in Eq. 6.

4.3

Task learning: OOD inference operates outside task-vector subspace

How can a model infer z ′ ∈ / Ztrain from context and achieve OOD generalization? One possibility is that it applies Eq. 6 with an uninformative prior over the full space Z rather than only the training support, extrapolating beyond seen latents. We call this the extrapolative task-learning mode (M2). M2 relies on context statistics (empirical unigram for E1, ridge solution for E2, and empirical bigram for E3) rather than memorized task vectors (Sec. B.3). The key question is whether such extrapolative inference behavior is enabled by a different geometry compared with M1. We give a theoretical result: when the OOD prediction map has intrinsic dimension d0 > k⋆ , where k⋆ is the task-vector subspace dimension, a model implementing M2 cannot confine its representations to that subspace. More formally, suppose V is discrete and fix a⋆ ∈ V. Let p(z) := P(· | z, a⋆ ) denote the next-token distribution. Assume P0 holds and that, for z near an interior point z0 ∈ Z, the limit µ⋆ (z) := limt→∞ µt,z,a⋆ exists. Define the task subspace T := span(θ2 − θ1 , . . . , θK − θ1 ), set k⋆ := dim T and S := νa⋆ + T . Let d0 := rank ∇p(z0 ) be the local intrinsic dimension of p at z0 . Theorem 2 (Informal, distance to task subspace lower bound). Assume P0. Suppose that on some neighborhood U ∋ z0 , the prediction p̂(z) decoded from µ⋆ (z) is locally L-Lipschitz: ∥p̂(z) − p̂(z ′ )∥2 ≤ L ∥µ⋆ (z) − µ⋆ (z ′ )∥2 , for all z, z ′ ∈ U. Additionally, assume that ∥p̂(z) − p(z)∥2 ≤ δ, for all z ∈ U and {µ⋆ (z) : z ∈ U } is bounded. Then there exist local constants c, C > 0 such that sup inf ∥µ⋆ (z) − c∥2 ≥

z∈U c∈S

c−δ C − d /k . L 2 0 ⋆ −1

See Sec. B.4 for the formal theorem. The second term in the lower bound decays exponentially in d0 /k⋆ , so when d0 ≫ k⋆ the model cannot implement M2 accurately while staying close to S; smoother decoding (smaller L) forces a larger deviation. This provides a theoretical basis for M2 representations to leave the task subspace. Sec. 6 will in fact show a stronger empirical result: M2 is implemented by a near-orthogonal subspace.

Task-vector geometry represents Bayesian posterior beliefs

Section 4.1 predicts the following behavior under the Bayesian task-retrieval mode (M1).

Table 1: Hidden-state variance decomposition at the final context position. (a) Residual variance ratio 2 SSwithin /SStotal : unexplained variance; lower is better. (b) Interaction proportion ηinteraction : share of SSbetween from task–token interaction; lower supports additivity. Layer

Layer

(a) Residual variance ratio SSwithin /SStotal E1 (Dice) 0.012 0.021 0.025 0.017 0.006 0.004 E3 (Markov) 0.037 0.071 0.029 0.012 0.007 0.005

E2 (Linear)

0.075

0.084

0.061

0.043

0.040

0.045

2 (b) Interaction proportion ηinteraction E1 (Dice) 0.003 0.006 0.012 0.013 0.007 0.004 E3 (Markov) <0.001 0.004 0.019 0.125 0.420 0.512

E2 (Linear)

<0.001

0.007

0.040

0.081

0.117

Task-vector geometry represents Bayesian inference: For ID sequences, the model represents training tasks as learned task vectors and updates their mixture weights from context, approximately tracking the Bayesian posterior.

5.1

Evaluation results of properties P0–P3

We show that the stated properties of task vectors are mostly satisfied for E1–E3. Fix K = 3. For each experiment, we estimate task vectors by replacing the expectation in Eq. 2 with empirical averages Ê over 1,024 independent sequences per task; estimated quantities use hat notation. Evaluating long-context stability. We assess how much the variance of hidden states is attributed to latent z ∈ [K] and the current token a ∈ V. For experiments E1 and E3 with discrete tokens, we calculate µ̂t,k,a := Ê[ht | z = k, st = a] and µ̂t := Ê[ht ]. By a balanced-cell ANOVA diagnostic, Ê∥ht − µ̂t ∥2 = (KV )−1

X X Ê ∥µ̂t,k,a − µ̂t ∥2 +(KV )−1 Ê ∥ht − µ̂t,k,a ∥2 |z = k, st = a . k,a

}

d t) SStotal : Var(h

k,a

SSbetween

} |

d t |z,st ) SSwithin : Var(h

}

and we measure the residual variance ratio SSwithin /SStotal . This ratio is zero if and only if hidden states ht are completely determined by latents and current tokens z, a. For experiment E2 where tokens are continuously valued, we approximate the proportion of conditional variance using analysis of covariance (ANCOVA) and report the R2 of the linear fit ht ≈ µt + θk + Bk xt where Bk is a task-specific coefficient matrix. See Sec. C.2 for details. Table 1(a) confirms that at the last context position the residual variance ratio is small across all layers and all three experiments, so latents and last tokens together account for most of the hidden-state variance at large t. Figure 9 (Sec. C.2) shows the full residual-ratio curves across context positions: the ratio is larger at early positions where limited context hinders task inference, and smaller in later layers where depth aids it. Evaluating long-context decoupling. We assess the additive separability as in Eq. 3 using ANOVA, splitting SSbetween into task, token, and interaction components (Sec. C.2). Table 1(b) 2 reports the interaction proportion ηinteraction at the last context position: it is small across most layers in all three experiments, confirming the additive model and validating P1.

Figure 2: Finite-context interpolation approximately holds. R2 of the interpolation model (Eq. 4) across context positions and layers for E1, E2, E3. Values are close to 1 except in late layers and early positions.

Figure 3: Bayesian posterior alignment. Simplex-projected coefficients βt,k (markers, 10–90th percentile error bars) vs. ground-truth posterior αt,k (dashed lines, shaded bands) for E1, E2, E3.

Evaluating finite-context interpolation. We next test P2 at finite context by fitting the P interpolation model in Eq. (4) with simplex-constrained coefficients βt,k ≥ 0, k βt,k = 1 (Figure 2). Across all three experiments, R2 is close to 1 in most layers and positions. Late layers of E2 and 2 E3, however, develop task-specific token encodings (large ηinteraction in Table 1(b) and Figure 10 2 in Sec. C.2), causing R to degrade, for instance plateauing at 0.60 and 0.52 in the last two layers of E3. Sec. C.3 provides a finer-grained variance decomposition showing that the unique contribution of task identity peaks at the middle layer and decays as later layers devote capacity to output prediction, consistent with the observation that task vectors are most effectively extracted from middle layers [16]. Overall, the interpolation model provides substantial support for P2. Evaluating Bayesian posterior alignment. We compare the ground-truth posterior αt,k (Sec. C.4.1) with the model’s simplex-projected coefficients βt,k (Figure 3). Across all three tasks, β̄t,k closely tracks αt,k as context accumulates. Sec. C.4.3 confirms that the agreement strengthens in later layers and is not an artifact of the simplex projection.

5.2

Intervention analysis of task vectors

To go beyond observational analysis, we intervene on hidden representations to test whether task vectors causally controlP predictions. At each position, we replace the task-subspace component with ∗ θ̂ , where α∗ is randomly drawn from the simplex (see Sec. C.5). a target interpolation k αt,k k t,k This intervention steers the hidden representation to a chosen point in the task-vector simplex. We ∗ ; calculate the difference between steered model outputs and the theoretical predictions based on αt,k we also include the unsteered model as a baseline. Figure 4 shows that steering moves the model’s outputs toward the corresponding mixture predictions: KL divergence drops from 0.11 to 0.03 in E1 and from 0.23 to 0.06 in E3, and RMSE 8

Figure 4: Causal intervention via the task-vector simplex. Substituting βt,k with randomly drawn ∗ αt,k steers model outputs to align with the corresponding mixture predictions, confirming causal effects.

Figure 5: Phase transition between two inference modes. Panels (left to right): E1, E2, E3. Blue (negative) indicates that the model is closer to Bayesian task retrieval M1, red (positive) indicates closer to the extrapolative task learning M2, and white indicates no clear preference. Higher task diversity generally promotes M2.

drops from 2.8 to 1.1 in E2. This demonstrates that the task-vector subspace provides a controllable representation through which the model interpolates between tasks. See Sec. C.5 for details and baselines.

Coexistence and geometry of two inference modes

Sec. 5 established that task-vector geometry implements Bayesian retrieval for memorized tasks. We now ask whether extrapolative task learning can coexist with retrieval, and what geometry supports it. Sec. 4.2 and 4.3 hinted that increasing the support of Ztrain shifts the model toward M2 mode, since the task-vector subspace alone cannot carry task inference over a high-dimensional latent space. To investigate this, we consider the training mixture Ztrain = Zmajor ∪ Zminor : 3 major tasks in Zmajor receive prior 0.9 and Nminor = 2m minor tasks share 0.1, as described in Sec. D.1. This split underlies the geometric analysis of Sec. 6.1: the major tasks fix a task-vector subspace, and we test whether M2 operates within it by projecting OOD representations onto this subspace. That geometric question is meaningful only if both modes arise, so we first verify behaviorally that sweeping the diversity exponent m drives the predicted M1-to-M2 transition. Task diversity drives the model’s inference mode. We train transformers under varying m and measure each model’s predictive distribution against the ideal predictions from both M1 and M2 (Eq. 6), yielding two KL divergences under varying m and training steps. Plotting log(KLbayesian /KLextrapolative ) in Figure 5 reveals clear phase transitions across E1–E3: early in training and at low diversity the model is better described by M1, whereas later in training and at higher diversity it is better described by M2. The diagonal boundary indicates that greater task diversity induces the shift earlier in training; red regions also coincide with stronger OOD performance (Sec. D.2). These results indicate that increasing task diversity drives a transition from Bayesian task retrieval toward extrapolative task learning, echoing the results in [32, 20, 26]. 9

Figure 6: Projection R2 of OOD hidden states on major task subspace. Panels: E1, E2, E3. As task diversity increases, projection R2 becomes smaller, supporting the near-orthogonal representation hypothesis.

6.1

Coexisting near-orthogonal subspaces

We hypothesize a stronger geometry for the M2 inference mode than Theorem 2 suggests. Near-orthogonal representation hypothesis: a transformer internally encodes both inference modes via subspaces, with Bayesian task retrieval carried by the task-vector subspace and extrapolative task learning by a nearly orthogonal subspace. To test this hypothesis, we train transformers for varying m and extract task vectors as in Sec. 5. For averaged hidden states from OOD sequences, we compute the projection R2 onto the major task subspace col(Θ̂): R2 = 0 indicates orthogonality, R2 = 1 full alignment. Results. Figure 6 reports R2 over training for varying Nminor . At low diversity (small Nminor ), R2 remains high: the major task subspace still explains partial computation under OOD sequences. As m grows, R2 drops consistently across E1–E3; at large diversity, OOD representations are nearly orthogonal to col(Θ̂). Such separation is learned : early in training all curves cluster at high R2 , and the drop emerges progressively only for large Nminor (see also Sec. D.3 for a complementary simplex-trajectory view). Comparing Figures 5 and 6 reveals a two-phase dynamic consistent across E1–E3: an initial undifferentiated phase (no KL preference between M1/M2, uniformly higher R2 ) followed by diversity-driven specialization in which high-m models transition toward M2 and R2 drops toward zero; the ID and OOD loss curves exhibit the same two-phase pattern (Sec. D.2). This staged pattern, consolidation of a common scaffold followed by mode specialization, echoes the two-phase dynamics observed in grokking [42] and progressive attention-head differentiation [43]. Hidden-state trajectories. Figure 7 shows hidden-state simplex trajectories in a trained transformer (E3) and a pretrained LLM (Qwen2.5-7B, layer 20; [30]). For the latter, we use three ID word-function tasks (English→French, Antonyms, Present→Past) and three OOD tasks with unnatural mappings. In both models, ID prompts converge toward the true task vertex as context accumulates, while OOD prompts yield hidden states nearly orthogonal to the task subspace (small R2 ), suggesting analogous two-mode geometry in a pretrained LLM (details in Sec. D.4). The LLM experiment provides a qualitative external validation of our two-mode framework, although a full characterization of representations on natural-language texts is beyond the scope of this paper.

6.2

Causal interventions on subspaces

To validate the causal role of the two subspaces, we intervene on hidden representations at inference time and measure task-specific losses. Let ∆Lmode /gmode denote the fractional loss degradation for 10

Figure 7: Task-subspace trajectories for E3 (left) and Qwen2.5-7B (right). Task vectors corresponding to 3 tasks form a triangle, and we project hidden states onto this task-vector subspace. ID prompts: hidden state ht converges toward the true task vertex as t grows. OOD prompts: hidden states remain near-orthogonal to the task subspace (small R2 ) throughout the trajectory. Table 2: Causal intervention on subspaces confirms near-orthogonal representations. Relative loss degradation ∆Lmode /gmode × 100%, averaged across the middle-layer range where orthogonal separation is cleanest. “Rand.” is the mean over 5 random same-rank directions from col(Θ̂)⊥ . See Sec. D.6 for the exact construction. Task Subspace Interv. (%)

Orth. Subspace Interv. (%)

Exp.

Layers

Maj.

OOD

Maj.

OOD

Rand.

E1 E3 E2

1–4 2–4 9–12

211.1 132.0 83.6

12.1 13.1 −1.0

4.0 2.1 3.0

95.8 99.6 85.4

2.9 3.0 1.9

mode ∈ {maj, ood}, where gmode is the total in-context loss reduction (so ∆Lmode /gmode ≈ 100% means the intervention has erased all in-context learning for the corresponding mode). Task-subspace intervention. With Ptask projecting onto col(Θ̂), h ← h − γPtask h selectively degrades major-task performance while largely preserving OOD losses (Table 2, left), showing that Bayesian task retrieval depends on col(Θ̂). Layerwise results are in Sec. D.5. Orthogonal subspace intervention. Within col(Θ̂)⊥ we identify a low-rank subspace V̂opt ⊤ h (γ > 1). Table 2 via optimization on held-out minor-task data and apply h ← h − γ V̂opt V̂opt (right) shows large, selective loss increases for OOD tasks, approaching 100% of the OOD in-context gain, while major-task loss and a random same-rank baseline remain near zero. This double dissociation establishes a near-orthogonal mechanism for extrapolative task learning. See Sec. D.6 ⊤ h linearly encodes running context statistics. for implementation; Sec. D.7 shows that V̂opt

Non-Markovian case: when hidden states don’t summarize context

The previous sections rely on a Markovian premise. In contrast, here we present a non-Markovian counterexample to show why that premise matters—when long-range structure breaks local summarization, even commonly assumed property P0 fails. This demonstrates that the validity of common heuristics depends sensitively on the underlying data-generating assumptions, highlighting the need for a principled framework and the value of the theoretical grounding it provides. Planted Dyck language. A Dyck language (Dyck-1) is the collection of all balanced bracket strings, underlying arithmetic expressions and programming code. For example, the Dyck language 11

Figure 8: Planted Dyck language (E4) exhibits long-context dependence and prefix memorization. Left: residual variance ratio of ht after conditioning on z and the last 3 planted characters at each Dyck position. Unlike E1–E3, z and a local window leave substantial residual variance, with spikes at positions requiring long-range bracket matching. Right: 2D projection of final-layer hidden states at Dyck prefix length l = 7 (35 prefix classes). The well-separated clusters show that the model encodes the full Dyck prefix history. Cluster labels are compact hexadecimal codes for the corresponding bracket prefix; see detailed figure in Figure 22.

of length 6 consists of 5 strings: ((())), (()()), (())(), ()(()), ()()(). We view latent z as a string in the Dyck language of length 2L. Conditioning on z, we sample a background sequence of T tokens i.i.d. uniformly from {0, 1, . . . , V − 1}. We then select each position independently with probability ρ = 0.25 and plant the next character of z, overwriting the background token. In other words, we plant z into a pure-noise background of T uniformly random symbols. As an example, with V = 2 and z = (()()), one possible sequence is 0, 1, (, 0, (, 1, 1), 0, (, 1, 1, ), ). In our experiments, the data-generating process uses V = 6 and L = 10, yielding C10 = 16796 possible latents. Long-context dependence. We train a small autoregressive transformer to near-optimal loss and, on held-out sequences, measure the variance of ht conditioned on the latent z and a local window of preceding tokens. In Figure 8 (left), the x-axis indexes Dyck position j (the j-th planted bracket). The residual variance ratio (hidden-state variance not explained by z and the last 3 planted characters) spikes to nearly 1 at positions requiring long-range bracket matching and drops to near zero elsewhere. The 2D projection in Figure 8 (right) further shows that the model encodes the full Dyck prefix at each position rather than a simple summary such as the running open-minus-close count; see Appendix E for the prefix-probe construction and quantitative validation accuracies at every prefix length.

Limitations and Future Work

This work presents a modest attempt to bridge generalization and representation geometry in transformers. First, our analysis of E1–E3 relies on the restrictive first-order Markov property on synthetic data; a more realistic setup may extend a single conditioning token a to a local window. Second, our near-orthogonal subspace does not provide a mechanistic account of OOD generalization, which is complex in general. Third, our evaluation and analysis are theory-inspired, but not yet a formal theory of generalization or learning dynamics, which is left to future work.

Acknowledgments Y.Z. is partially supported by NSF-DMS grant 2412052 and by a Coefficient Giving (formerly Open Philanthropy) grant. The authors wish to thank Robert Nowak, Joshua Cape, Keith Levin, Hanbaek Lyu, Sebastien Roch, Karl Rohe for helpful discussions.

References [1] J. Aitchison. The Statistical Analysis of Compositional Data. Monographs on Statistics and Applied Probability. Chapman & Hall, London, 1986. [2] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? Investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id= 0g0X4H8yN4I. [3] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https://transformercircuits.pub/2023/monosemantic-features/index.html. [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [5] Emmanuel J Candès, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on information theory, 52(2):489–509, 2006. [6] Liam Carroll, Jesse Hoogland, Matthew Farrugia-Roberts, and Daniel Murfet. Dynamics of transient structure in in-context linear regression transformers. arXiv preprint arXiv:2501.17745, 2025. [7] David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4): 1289–1306, 2006. [8] David L Donoho and Philip B Stark. Uncertainty principles and signal recovery. SIAM Journal on Applied Mathematics, 49(3):906–931, 1989. [9] John C. Duchi, Shai Shalev-Shwartz, Yoram Singer, and Tushar Chandra. Efficient projections onto the ℓ1 -ball for learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning (ICML), pages 272–279, 2008. doi: 10.1145/1390156.1390191. [10] Ezra Edelman, Nikolaos Tsilivis, Benjamin L Edelman, Eran Malach, and Surbhi Goel. The evolution of statistical induction heads: In-context learning markov chains. Advances in neural information processing systems, 37:64273–64311, 2024. 13

[11] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer-circuits.pub/2021/framework/index.html. [12] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. https://transformercircuits.pub/2022/toy_model/index.html. [13] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583–30598, 2022. [14] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, 2021. [15] Wes Gurnee, Emmanuel Ameisen, Isaac Kauvar, Julius Tarng, Adam Pearce, Chris Olah, and Joshua Batson. When models manipulate manifolds: The geometry of a counting task. arXiv preprint arXiv:2601.04480, 2026. [16] Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318–9333. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-emnlp.624. URL https://aclanthology.org/2023.findings-emnlp.624/. [17] Dongfang Li, Zhenyu Liu, Xinshuo Hu, Zetian Sun, Baotian Hu, and Min Zhang. In-context learning state vector with inner and momentum optimization. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 0ed52d7f6f641f228405d48a611e0684-Abstract-Conference.html. [18] Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 32287–32307. PMLR, 2024. URL https://proceedings.mlr.press/ v235/liu24bx.html. [19] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id= Bkg6RiCqY7. [20] Yue M Lu, Mary Letey, Jacob A Zavatone-Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention. Proceedings of the National Academy of Sciences, 122(28):e2502599122, 2025. 14

[21] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple Word2Vecstyle vector arithmetic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5030–5047. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.naacl-long.281. URL https://aclanthology.org/2024.naacl-long.281/. [22] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013. [23] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff, editors, Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https://aclanthology.org/ N13-1090/. [24] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022. [25] Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning “learns” in-context: Disentangling task recognition and task learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8298–8319, 2023. [26] Core Francisco Park, Ekdeep Singh Lubana, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=XgH1wfHSX8. [27] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning, 2024. [28] Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=bVTM2QKYuA. [29] Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8: 143–195, 1999. [30] Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2025. URL https://arxiv.org/abs/2412. 15115. [31] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 15

[32] Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. Advances in neural information processing systems, 36:14228–14246, 2023. [33] Adam Shai, Loren Amdahl-Culleton, Casper L Christensen, Henry R Bigelow, Fernando E Rosas, Alexander B Boyd, Eric A Alt, Kyle J Ray, and Paul M Riechers. Transformers learn factored representations. arXiv preprint arXiv:2602.02385, 2026. [34] Adam S Shai, Sarah E Marzen, Lucas Teixeira, Alexander G Oldenziel, and Paul M Riechers. Transformers represent belief state geometry in their residual stream. Advances in Neural Information Processing Systems, 37:75012–75034, 2024. [35] Jiajun Song and Yiqiao Zhong. Uncovering hidden geometry in transformers via disentangling position and context. arXiv preprint arXiv:2310.04861, 2023. [36] Jiajun Song, Zhuoyan Xu, and Yiqiao Zhong. Out-of-distribution generalization via composition: A lens through induction heads in transformers. Proceedings of the National Academy of Sciences, 122(6):e2417182122, 2025. doi: 10.1073/pnas.2417182122. URL https://www.pnas.org/doi/ abs/10.1073/pnas.2417182122. [37] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. [38] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/scaling-monosemanticity/ index.html. [39] Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024. [40] A. W. van der Vaart. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998. [41] Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. [42] Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokking of implicit reasoning in transformers: A mechanistic journey to the edge of generalization. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. URL https://proceedings.neurips.cc/paper_files/ paper/2024/hash/ad217e0c7fecc71bdf48660ad6714b07-Abstract-Conference.html. [43] George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, and Daniel Murfet. Differentiation and specialization of attention heads via the refined local learning coefficient. In International Conference on Learning Representations, 2025. 16

[44] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023. [45] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-demos.6. URL https: //aclanthology.org/2020.emnlp-demos.6/. [46] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022. [47] Kayo Yin and Jacob Steinhardt. Which attention heads matter for in-context learning? In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 72428–72461. PMLR, 2025. URL https: //proceedings.mlr.press/v267/yin25e.html.

Organization of the appendix. We organize our appendix into five sections, following the sequential order of the main text. Specifically, • Sec. A provides the architecture and training hyperparameters used throughout our synthetic experiments (complementing the Setup in Sec. 3). • Sec. B contains the theoretical material supporting Sec. 4: the chain of implications among Properties P0–P3 (Sec. B.1), the formal proof of Theorem 1 on Bayesian realizability (Sec. B.2), explicit formulas for the extrapolative mode M2 (Sec. B.3), and the proof of Theorem 2 on the simplex packing obstruction (Sec. B.4). • Sec. C provides the computational and empirical details behind the evaluation of Properties P0–P3 and the simplex intervention reported in Sec. 5. • Sec. D collects additional evidence and details for the two-mode picture in Sec. 6, including the major/minor training mixture, ID/OOD loss dynamics, simplex trajectories of hidden states, the pretrained-LLM (Qwen2.5-7B) experiments, and per-layer breakdowns of the orthogonal-subspace causal interventions. • Sec. E reports the prefix-probe analysis of the planted Dyck experiment (E4) discussed in Sec. 7.

Experiment Details

Architecture and training hyperparameters. All four experiments use decoder-only transformers with pre-layer-norm, rotary positional encodings, and a 4× MLP expansion ratio, trained autoregressively with AdamW. The biased dice (E1), mixture of Markov chains (E3), and planted Dyck (E4) share a 6-layer, 128-dimensional backbone, while the in-context linear regression (E2) uses 17

a deeper 16-layer model. Per-experiment values for depth, width, context length, optimizer, learningrate schedule, and other training settings are listed in Tables 3 and 4; the resulting checkpoints are the ones reused in every subsequent empirical analysis. Compute and runtime. All training and analyses are run on a single NVIDIA RTX 3090 (24 GB). A full 30 k–32 k-step training run takes roughly 10 minutes for E1, 13 minutes for E3 and E4, and 50 minutes for E2 (whose 16-layer backbone dominates the compute). The two-mode and orthogonalsubspace analyses (Sec. 6 and Sec. D) sweep the minor-pool size Nminor ∈ {0, 1, 2, 4, . . . , 1024} and retrain the model for each value, so a single-seed sweep takes about 2 GPU-hours for E1, 2.5 for E3, and 9 for E2. Downstream analyses (task-vector extraction, projection R2 , simplex and orthogonal interventions, posterior alignment) all complete in well under one GPU-hour per checkpoint. Table 3: Architecture hyperparameters. All models are decoder-only transformers with pre-layer-norm and 4× MLP expansion.

Biased dice Markov chains Planted Dyck Linear regression

Layers

Heads

FF dim

Pos. enc.

Batch

6 6 6 16

2 2 2 2

128 128 128 128

512 512 512 512

128 192 192 128

rotary rotary rotary rotary

128 128 128 256

Table 4: Training hyperparameters. All experiments use AdamW. The triangle schedule linearly increases the learning rate during warmup and then linearly decays it to the minimum; the cosine schedule uses cosine annealing after warmup.

Biased dice Markov chains Planted Dyck Linear regression

Steps

Weight decay

Warmup

Schedule

Grad clip

Min LR

30 k 30 k 32 k 30 k

4×10−4 4×10−4 2×10−4 2×10−4

4×10−4 4×10−4 2×10−4 1×10−4

15 k 15 k 8k 15 k

triangle triangle cosine triangle

— — 0.4 —

10−5 10−5 2×10−5 10−5

Theoretical Foundations of the Property Framework

This section collects the proofs and additional derivations that support the mathematical framework introduced in Sec. 4. • Sec. B.1 establishes the chain of implications among Properties P0–P3. • Sec. B.2 provides the formal statement and proof of Theorem 1 on Bayesian realizability. • Sec. B.3 derives the closed-form predictors that underlie the extrapolative task-learning mode M2. • Sec. B.4 proves the simplex packing obstruction (Theorem 2).

B.1

Theoretical relationship between properties P0–P3

This subsection establishes the implication chain among the four geometric properties P0–P3 introduced in Sec. 4.1: P3 ⇒ P2,

P3 ⇒ P1 in the long-context limit,

P1 ⇒ P0.

This justifies treating the empirical evidence for P3 reported in Sec. 5.1 as evidence for the weaker properties P0–P2 as well. Proposition 1 (P3 ⇒ P2). If P3 holds, then P2 holds. P Proof. By P3, the hidden state admits the decomposition ht ≈ µt + k≤K βt,k θk + νst with coefficients βP t,k ≈ αt,k := P(z = k | s≤t ). Because {αt,k }k≤K is a probability distribution, we have αt,k ≥ 0 and k≤K αt,k = 1, so the coefficients βt,k are nonnegative and sum to one. This is precisely the simplex constraint required by P2. Proposition 2 (P3 ⇒ P1 in the long-context limit). Suppose P3 holds and the posterior concentrates on the true latent, i.e., αt,k → δk,z almost surely as t → ∞, where z denotes the true latent. Then P1 holds in the limit t → ∞. Proof. Substituting αt,k → δk,z into the P3 decomposition yields X ht ≈ µt + αt,k θk + νst −→ µt + θz + νst , k≤K

recovering exactly the additive structure required by P1. The hypothesis αt,k → δk,z a.s. holds in our experiments by the ergodic theorem for finite-state Markov chains (E1, E3) and an analogous argument for E2 [e.g., 40, Ch. 10]. Proposition 3 (P1 ⇒ P0). If P1 holds, then P0 holds. Proof. Write εt = ht − µt,z,st , where µt,z,st = E[ht | z, st ] is the cell mean. Because the conditional expectation minimizes mean squared error over all functions of (z, st ), we have E εt

= E ht − µt,z,st

≤ E ht − g(z, st )

for every measurable g : [K] × V → Rd . Taking g(z, st ) = µt + θz + νst , which depends on (z, st ) alone, gives 2 2 E εt ≤ E ht − (µt + θz + νst ) −→ 0 as t → ∞, where the right-hand side vanishes by the P1 guarantee (Eq. 3).

B.2

Proof of Theorem 1 in Section 4.2

This subsection gives the formal statement and proof of Theorem 1 from Sec. 4.2. It shows that posterior-aligned representations (properties P2–P3) are sufficient for a transformer to approximate the Bayesian predictive distribution under the task-retrieval mode M1. The high-level idea is that, once the residual stream encodes the posterior weights αt,k as the coefficients in front of the task vectors θk , the remaining Bayesian computation qt+1 =

K X k=1

αt,k pk,st

reduces to two operations, each realizable by one transformer block: (i) a finite lookup that mixes the per-pair conditionals pk,a into the residual stream, followed by (ii) an approximate coordinatewise logarithm whose softmax recovers qt+1 . The output unembedding is also part of the construction below, so the theorem does not assume any property of a pre-existing unembedding matrix. The full construction is given in Sec. B.2.1 and relies on two technical lemmas stated and proved in Sec. B.2.2. B.2.1

Detailed Statement of Theorem 1 and Proof

We now state the formal version of Theorem 1, an expressivity statement. Posterior-aligned residuals (P2–P3) together with the structural conditions (i)–(iv) below suffice for two appended transformer blocks to approximate the Bayesian predictive qt+1 within ℓ∞ tolerance δ. The lookup MLP in Block 1 has hidden width KV . Block 2 has hidden width mlog (δ, ε, V ) + 2V , where mlog is the width of a coordinatewise ReLU spline approximation to log on [ε, 1] at the accuracy used below. These are constructive upper bounds; we do not claim width optimality, nor do we address whether training finds this configuration or whether conditions (i)–(iv) hold for arbitrary trained models. Theorem 3 (Internal geometry mirrors Bayesian inference; formal version of Theorem 1). Consider a LayerNorm-free transformer with residual-stream dimension d, vocabulary V of size V (identified with [V ]), and training latent set Ztrain = [K]. Suppose the following conditions hold. (i) Uniform posterior-aligned representation. At some layer ℓ, fix a set of positions I. For every t ∈ I, the residual stream satisfies ht = µt +

K X

αt,k θk + νst ,

αt,k = P(z = k | s≤t ),

k=1 d d where {θk }K k=1 ⊂ R are task vectors and {νa }a∈V ⊂ R are token-encoding vectors (cf. properties P2–P3 in Sec. 4.1).

(ii) Shared centering map. Let U := span {θk }k≤K ∪ {νa }a∈V .

MI := span{µt : t ∈ I},

Assume MI ∩ U = {0}. Equivalently, there exists a fixed linear map C : Rd → Rd such that Cµt = 0,

Cθk = θk ,

Cνa = νa

for all t ∈ I, k ≤ K, and a ∈ V. We call such a map a shared centering map. (iii) Linear independence. The family {θ1 , . . . , θK } ∪ {νa : a ∈ V} is linearly independent in Rd ; in particular d ≥ K + V . (iv) Bounded-away transition probabilities. There exists ε ∈ (0, 1/V ) such that P(st+1 = b | z = k, st = a) ≥ ε for all t ∈ I, b ∈ V, k ∈ [K], and a ∈ V. 20

Then for every δ > 0, there exist an output unembedding WU ∈ RV ×d and two additional transformer blocks, all shared across t ∈ I, with the blocks appended after layer ℓ, such that for every t ∈ I the model’s next-token prediction p̂t+1 ∈ RV satisfies ∥p̂t+1 − qt+1 ∥∞ < δ, where qt+1 :=

K X

αt,k pk,st ,

pk,a := P(st+1 = b | z = k, st = a) b∈V .

k=1

Moreover, the first appended MLP has hidden width KV and the second has hidden width mlog (δ, ε, V )+ 2V . Proof. We first fix a gauge for the construction. Although task and token effects are defined earlier in a centered gauge, the is most convenient in an uncentered gauge. In particular, PKlookup construction P we do not impose k=1 θk = 0 or a∈V νa = 0, since either constraint would create a nontrivial linear dependence and conflict with condition (iii). All structural assumptions in the theorem, including the shared-centering map C, are understood in this uncentered gauge. This is only a change of coordinates at the level of represented states: setting θ̄ := K

−1

K X

ν̄ := V −1

θk ,

νa ,

a∈V

k=1

and replacing (µt , θk , νa ) 7→ (µt + θ̄ + ν̄, θk − θ̄, νa − ν̄) leaves µt + k αk θk + νa unchanged for every α ∈ ∆K−1 and token a. We append two transformer blocks after layer ℓ. In both blocks, set the attention output projection to zero, so each attention sublayer contributes only the residual identity. It remains to construct the two MLP sublayers and the output unembedding. Let xt := ht be the input to the first appended block and set P

ut := Cxt . By conditions (i)–(ii), ut =

K X

αt,k θk + νst =

k=1

K X

αt,k (θk + νst ).

k=1

Write uk,a := θk + νa . Also define the token-embedding matrix E := (ν1 , . . . , νV ) ∈ Rd×V . By condition (iii), E has full column rank.

Step 1: Build the Bayesian mixture while keeping µt . Define f (k, a) := Epk,a − uk,a = Epk,a − θk − νa . By Lemma 1, there is a width-KV two-layer ReLU MLP MLPlookup : Rd → Rd such that, for P ut = k αt,k uk,st , MLPlookup (ut ) =

K X

αt,k f (k, st ) = E

k=1

K X

αt,k pk,st − ut = Eqt+1 − ut .

k=1

Composing the first-layer affine forms with C, the map xt 7→ MLPlookup (Cxt ) is still a two-layer ReLU MLP with hidden width KV . Therefore the first residual block outputs yt := xt + MLPlookup (Cxt ) = µt + ut + (Eqt+1 − ut ) = µt + Eqt+1 . Thus Block 1 does not cancel the nuisance component µt . It carries µt forward and only converts the task/token component into the embedded Bayesian predictive Eqt+1 . Step 2: Read out qt+1 , cancel only at the logits, and apply softmax. Since col(E) = span{νa : a ∈ V} ⊂ U and MI ∩ U = {0}, we have MI ∩ col(E) = {0}. By condition (iv), qt+1 ∈ ∆εV . Applying Lemma 2 with M = MI gives a constructed unembedding WU ∈ RV ×d and a width-mlog (δ, ε, V ) + 2V two-layer ReLU MLP MLPout : Rd → Rd such that, for yt = µt + Eqt+1 , softmax WU yt + MLPout (yt ) − qt+1 ∞ < δ. Using this MLP as the second block’s MLP sublayer proves the claim. All maps used above, namely C, E, MLPlookup , WU , and MLPout , are fixed across t ∈ I, so the construction is uniform over positions. B.2.2

Technical Lemmas and Proofs

This subsection states and proves the two technical lemmas invoked in the construction of Sec. B.2.1: Lemma 1 is used in the first appended block, and Lemma 2 is used in the second. Both isolate a single capability of a two-layer ReLU MLP and are stated in self-contained form, independent of the transformer setting. Lemma 1 (Width-KV lookup and convex-linearity in the latent index). Let K, V, dout ∈ N. Suppose {θ1 , . . . , θK , ν1 , . . . , νV } ⊂ Rd is linearly independent, and define (k, a) ∈ [K] × [V ].

uk,a := θk + νa ,

For any map f : [K] × [V ] → Rdout , there exists a two-layer ReLU MLP L : Rd → Rdout with hidden width KV such that L(uk,a ) = f (k, a) 22

for all (k, a), and, more generally, L

K X

! αk uk,a

k=1

K X

αk f (k, a)

k=1

for every fixed a ∈ [V ] and every α ∈ ∆K−1 := {α ∈ RK : αk ≥ 0,

k αk = 1}.

Proof. By linear independence, there is a biorthogonal dual family ∗ {θ1∗ , . . . , θK , ν1∗ , . . . , νV∗ } ⊂ Rd

satisfying and

⟨θi∗ , θk ⟩ = 1{i = k},

⟨θi∗ , νa ⟩ = 0,

⟨νj∗ , νa ⟩ = 1{j = a},

⟨νj∗ , θk ⟩ = 0.

For each pair (i, j) ∈ [K] × [V ], define ϕij := θi∗ + νj∗ . Then ⟨ϕij , uk,a ⟩ = 1{i = k} + 1{j = a}. Define one hidden unit per pair: hij (u) := σ ⟨ϕij , u⟩ − 1 ,

σ(r) := max{r, 0}.

On the vertices uk,a , this gives hij (uk,a ) = 1{i = k, j = a}. Now fix a token a and a simplex vector α. For u(α) := a

K X

αk uk,a =

k=1

we have

K X

αk θk + νa ,

k=1

⟨ϕij , u(α) a ⟩ = αi + 1{j = a}.

Therefore, hij (u(α) a )=

( σ(αi ) = αi , j = a, σ(αi − 1) = 0, j = ̸ a,

because 0 ≤ αi ≤ 1. Finally define L(u) :=

K X V X

hij (u) f (i, j).

i=1 j=1

This is a two-layer ReLU MLP with hidden width KV , output weights f (i, j), and zero output bias. The vertex and convex-linearity claims follow from the displayed identities. 23

Remark 2 (Connection to key–value memory). The construction above can be interpreted through the lens of key–value memory, as discussed in Geva et al. [14]. In our setting, the vectors {ϕij } play the role of keys, which detect whether the input corresponds to a specific pair (i, j) via inner products and thresholding. The second layer associates each key with a value f (i, j), and the MLP output is obtained by aggregating the retrieved values. Lemma 2 (Probability recovery with a constructed unembedding). Let M ⊂ Rd be a linear subspace, let E ∈ Rd×V have full column rank, and assume M ∩ col(E) = {0}. For every ε ∈ (0, 1/V ) and δ > 0, there exist an output unembedding WU ∈ RV ×d and a two-layer ReLU MLP F : Rd → Rd of hidden width mlog (δ, ε, V ) + 2V such that sup softmax WU m + Ep + F (m + Ep) − p ∞ < δ, m∈M, p∈∆εV

where

( ∆εV :=

p ∈ RV : pb ≥ ε for all b ∈ [V ],

V X

) pb = 1 .

b=1

Proof. Because M ∩ col(E) = {0}, we have M + col(E) = M ⊕ col(E). Thus every vector in this sum has a unique decomposition m + Er with m ∈ M and r ∈ RV . Define a linear map P : M ⊕ col(E) → RV by P (m + Er) = r. Extend P linearly to all of Rd , and set WU := P . Then WU m = 0 (m ∈ M),

WU E = IV .

In particular, the required unembedding is constructed from the decomposition; no pre-existing output head is assumed. Choose η > 0 such that e2η − 1 < δ. Let G : RV → RV be a two-layer ReLU network satisfying sup ∥G(p) − log p∥∞ < η.

p∈∆εV

Such a G can be obtained coordinatewise by a piecewise-linear ReLU spline approximation to log on the compact interval [ε, 1]; equivalently, this follows from standard single-hidden-layer approximation results such as Pinkus [29]. Let mlog (δ, ε, V ) denote the width of this coordinatewise approximation. Define F (y) := E G(WU y) − EWU y. The first term is a two-layer ReLU MLP of width mlog (δ, ε, V ), obtained by composing the input layer of G with WU and the output layer with E. The second term −EWU y is a rank-at-most-V linear map, so it can be represented exactly using at most 2V ReLU hidden units. Indeed, if EWU = AB with B ∈ Rr×d , A ∈ Rd×r , and r ≤ V , then −EWU y = −A σ(By) − σ(−By) . 24

Concatenating the hidden units for the two terms gives a two-layer ReLU MLP of width mlog (δ, ε, V )+ 2V . For y = m + Ep with m ∈ M and p ∈ ∆εV , we have WU y = p. Therefore WU y + F (y) = WU y + WU E G(WU y) − WU EWU y = G(p), where we used WU E = IV . The output distribution is therefore softmax(G(p)). Write G(p) = log p + e with ∥e∥∞ < η. For each coordinate b, pb eeb . softmax(G(p))b = PV ec c=1 pc e Since e−η ≤ eec ≤ eη for every c, the ratio softmax(G(p))b /pb lies in [e−2η , e2η ]. Thus |softmax(G(p))b − pb | ≤ e2η − 1 < δ. Taking the maximum over b proves the lemma.

B.3

Details About Extrapolative Task Learning Mode: Explicit Formulas

This subsection derives the closed-form predictors that define the extrapolative task-learning mode M2 introduced in Sec. 4.3. Concretely, M2 is the Bayesian posterior predictive obtained from Eq. 6 by replacing the prior πtrain over the finite training support Ztrain with the generative prior over the full latent space Z, i.e. the same parametric distribution that was used to sample tasks during training (symmetric Dirichlet for E1 and E3, isotropic Gaussian for E2; cf. Sec. 3). For each of E1–E3, conjugacy of this prior with the per-token likelihood collapses the integral over Z into a closed-form predictor that depends on the observed sequence only through a low-dimensional context statistic: a smoothed empirical unigram for E1, the ridge solution for E2, and a smoothed empirical bigram for E3. The derivations below thus pin down the concrete predictive targets that a model implementing M2 must encode; whether the trained model actually encodes them, and where in the residual stream, is a separate empirical question taken up in the orthogonal-subspace decomposition of Sec. D.7. Biased dice (E1). Each latent task z = p ∈ Z = ∆V −1 is sampled from the symmetric Dirichlet prior Dir(1V ). More generally, for p ∈ ∆V −1 drawn from Dir(α0 1V ), the probability density function is V Γ(α0 V ) Y α0 −1 f (p1 , . . . , pV ; α0 ) = pa . Γ(α0 )V a=1

Recall the token count na (t) defined in Equation (18). The posterior predictive after observing s≤t is Z P(st+1 = a | s≤t ) = P(st+1 = a | p, s≤t ) P(p | s≤t ) dp p∈Z R p∈Z P(st+1 = a | p) P(s≤t | p) P(p) dp R = p∈Z P(s≤t | p) P(p) dp R QV nb (t)+α0 −1 dp b=1 pb p∈Z pa = R QV nb (t)+α0 −1 dp b=1 pb p∈Z QV Γ(na (t) + α0 + 1) b̸=a Γ(nb (t) + α0 ) Γ(t + V α0 ) = · QV Γ(t + V α0 + 1) b=1 Γ(nb (t) + α0 ) =

na (t) + α0 , t + V α0

which is a smoothed empirical unigram estimator. Noisy linear regression (E2). Each latent task z = w ∈ Z = RD is drawn from an isotropic Gaussian prior w ∼ N (0, τ −2 σ 2 ID ), which matches the generative prior of Sec. 3 when τ 2 = σ 2 . Conditional on w, inputs are i.i.d. 2 xt ∼ N (0, ID ) and yt | xt , w ∼ N (x⊤ t w, σ ). For the prediction of yt+1 at a fresh covariate xt+1 , let X≤t = (x1 , . . . , xt )⊤ ∈ Rt×D and y≤t = (y1 , . . . , yt )⊤ ∈ Rt collect the observed pairs. By Gaussian–Gaussian conjugacy, the posterior over w is again Gaussian: −1 ⊤ w | X≤t , y≤t ∼ N ŵridge (t), σ 2 Σt , Σt := X≤t X≤t + τ 2 ID , where ŵridge (t) is the ridge solution [46, 32, 2] ŵridge (t) =

⊤ X≤t X≤t + τ 2 ID

−1

⊤ X≤t y≤t .

Marginalising w yields the posterior predictive Z P yt+1 | xt+1 , X≤t , y≤t = P(yt+1 | xt+1 , w) P(w | X≤t , y≤t ) dw RD 2 ⊤ = N x⊤ . t+1 ŵridge (t), σ 1 + xt+1 Σt xt+1 Hence the posterior predictive mean is the ridge regressor evaluated at xt+1 , and the predictive variance inflates the noise level σ 2 by the input-dependent factor 1 + x⊤ t+1 Σt xt+1 that captures remaining uncertainty about w. Mixture of Markov chains (E3). Each latent task z = P ∈ Z = (∆V −1 )V is a row-stochastic V × V transition matrix, with each row drawn independently from the symmetric Dirichlet prior used to generate the training tasks (Sec. 3): i.i.d.

Pa,· ∼ Dir(α0 1V ), 26

a = 1, . . . , V.

For a Markov sequence s≤t generated under P from a uniform initial distribution, let Nab (t) :=

t X

1{sτ −1 = a, sτ = b} ∈ N,

Na· (t) :=

τ =2

V X

Nab (t) ∈ N,

b=1

denote the count of a → b transitions and the row total, respectively. The likelihood factorises across rows, V Y V Y N (t) P(s≤t | P ) = Pa,bab , a=1 b=1

which is a product of multinomial likelihoods. Combined with the row-independent Dirichlet prior, Dirichlet–multinomial conjugacy gives a posterior that factorises across rows, Pa,· | s≤t ∼ Dir(α0 + Na1 (t), . . . , α0 + NaV (t)) ,

a = 1, . . . , V.

The posterior predictive at the next position, given current state st = a, is then Z P st+1 = b | st = a, s≤t = Pa,b P(P | s≤t ) dP Z

= E[Pa,b | s≤t ] Nab (t) + α0 = , Na· (t) + V α0 where the last equality uses the mean of a Dirichlet distribution. This is a smoothed empirical bigram estimator, exactly analogous to the smoothed empirical unigram derived for E1: it pools every observed transition out of state a and interpolates with the uniform prior at rate controlled by α0 . Summary. In all three experiments, the predictor underlying M2 is the exact Bayesian posterior predictive under the parametric prior used to generate the training tasks, but evaluated over the full latent space Z rather than only the finite training support Ztrain . Conjugacy of each prior–likelihood pair (Dirichlet–categorical for E1, Gaussian–Gaussian for E2, and row-wise Dirichlet–multinomial for E3) yields a closed-form predictor that depends on the observed sequence only through a sufficient context statistic: the empirical unigram counts na (t) for E1, the ridge solution ŵridge (t) for E2, and the empirical bigram counts Nab (t) for E3. This explains why these context statistics arise naturally as the targets that the model must encode in order to implement M2.

B.4

Proof of Theorem 2 in Section 4.3

This subsection states and proves the formal version of Theorem 2. The result makes precise the intuition that the extrapolative task-learning mode M2 cannot be implemented from inside a task-vector subspace whose dimension k⋆ is smaller than the intrinsic dimension d0 of the optimal prediction map: a volumetric packing argument forbids the required separation of latents within a k⋆ -dimensional affine slab unless the alignment error ε grows accordingly. Theorem 4 below gives the explicit lower bound on ε, which informally takes the form ε ≳ (c − δ)/L − C/(2d0 /k⋆ − 1) reported in Sec. 4.3.

Notation. We collect the notation used throughout this appendix, extending the conventions of Sec. 4. Let Z denote the latent space with training set Ztrain = [K]. At a fixed layer, ht ∈ Rd is the hidden state at position t; {θk }k≤K and {νa }a∈V are the task vectors and token-encoding vectors (Definition 1); µt,k,a := E[ht | z=k, st =a] are the cell means (Eq. 1). We assume P0 together with the existence of the limiting cell mean µ(z, a) := limt→∞ µt,z,a for each (z, a) in the neighborhood of interest. (Note that P0 alone controls only the conditional residual variance E∥ht − µt,z,st ∥2 and does not by itself guarantee convergence of µt,z,a ; we therefore impose the existence of the limit explicitly.) Under these assumptions the (long-context) decoder maps µ(z, a) to the Bayes-optimal next-token distribution P(· | z, a). Setup. We now introduce the symbols specific to the simplex-confinement question, augmenting the notation of Sec. 4.3. Fix an anchor token a⋆ ∈ V and write p(z) := P(· | z, a⋆ ) ∈ ∆V −1 ,

µ⋆ (z) := µ(z, a⋆ ) ∈ Rd ,

for the conditional next-token distribution and the limiting cell mean at a⋆ . Let T := span(θ2 − θ1 , . . . , θK − θ1 ),

S := νa⋆ + T ,

denote the task-vector subspace and its affine translate at νa⋆ , and set k⋆ := dim(T ) ≤ K − 1. We assume k⋆ ≥ 1, as otherwise S is a singleton and the conclusions are trivial. Discrete-output scope. The theorem is stated for finite discrete output spaces: a⋆ ∈ V and p(z) = P(· | z, a⋆ ) ∈ ∆V −1 . It therefore directly covers the discrete-output settings E1 and E3. The linear-regression setting E2 is an empirical continuous-output analogue. A formal continuous-output version would require replacing the simplex-valued map p by an appropriate notion of distance between posterior-predictive regression distributions. We omit this extension for simplicity. Gauge convention The affine anchor in S = νa⋆ + T is a gauge choice. The proof below uses only the affine set S and distances dist(µ⋆ (z), S), not the separate decomposition of its anchor into global-mean and token components. Indeed, if P1 is written in the centered gauge ht ≈ µt + θz + νst and µt → µ∞ , then the long-context ID cell means at anchor token a⋆ lie in µ∞ + νa⋆ + T . We absorb this common translate into the affine anchor, equivalently writing ν̄a⋆ := µ∞ + νa⋆ ,

S = ν̄a⋆ + T,

and then drop the bar. This is only a notational convention for interpreting S under P1; Theorem 2 itself assumes only the limiting cell means µ(z, a) specified above. Definition 2 (Local regularity setup). Fix a base point z0 ∈ int(Z) and assume that p is C 1 on a neighborhood of z0 . Let J0 := ∇z p(z0 ),

d0 := rank(J0 ) ≥ 1,

σ0 := sd0 (J0 ) > 0, 28

E := (ker J0 )⊥ ,

where sd0 (J0 ) denotes the smallest nonzero singular value of J0 . Choose a radius r0 > 0 with B(z0 , r0 ) ⊆ Z, and define UE,0 := (z0 + E) ∩ B(z0 , r0 ). Then UE,0 is a d0 -dimensional ball in the affine subspace z0 + E along which p has full local rank; the integer d0 makes precise the “number of independent directions along which p(z) varies” from Sec. 4.3. The next theorem formalizes the informal claim of Theorem 2 in Sec. 4.3. Throughout, we write dist(x, S) := inf ∥x − c∥2 c∈S

for the Euclidean distance from a point x ∈ Rd to the affine set S, and we collect the regularity assumptions used in the theorem. Definition 3 (Local decoder regularity). Let p̂(z) be the predictor obtained by passing µ⋆ (z) through the model’s decoder. We say the setup is (M, L, δ)-regular on a neighborhood U ⊆ Z of z0 if there exist constants M < ∞, L > 0, and δ ≥ 0 such that, for all z, z ′ ∈ U, ∥µ⋆ (z) − µ⋆ (z ′ )∥2 ≤ M,

(7)

′

∥p̂(z) − p̂(z )∥2 ≤ L ∥µ⋆ (z) − µ⋆ (z )∥2 ,

(8)

∥p̂(z) − p(z)∥2 ≤ δ.

(9)

We further define the worst-case distance from µ⋆ to the affine task subspace on U, ε := sup dist µ⋆ (z), S .

(10)

z∈U

Theorem 4 (Finite-context packing obstruction). Assume the setup of Definition 2 and that the (M, L, δ)-regularity of Definition 3 holds on a neighborhood U ⊆ Z of z0 . Then there exists rloc ∈ (0, r0 ] such that UE := (z0 + E) ∩ B(z0 , rloc ) ⊆ U ,

α := σ0 /2,

and the following hold. (a) Local co-Lipschitz estimate for the ground-truth map. For all z, z ′ ∈ UE , ∥p(z) − p(z ′ )∥2 ≥ α ∥z − z ′ ∥2 .

(11)

(b) Finite-context packing inequality. For any 0 < ρ < rloc such that αρ − 2δ > 2ε, L one has

k⋆



rloc ρ

  2M ≤ 1 + s  αρ − 2δ 2 L 29

− 4ε2

    

(12)

(c) Lower bound on the confinement radius. For any 0 < ρ < rloc such that αρ > 2δ,  !2 1/2 2 αρ − 2δ 2M 1  , − ε≥  2 L (rloc /ρ)d0 /k⋆ − 1

(13)

where [x]+ := max{x, 0}. Proof. Choose a linear isometry

W : Rd0 → E.

Define e p(u) := p(z0 + W u), Then pe is C 1 , and

u ∈ BRd0 (0, r0 ).

e ∇p(0) = J0 W .

e Since W is an isometry onto E = (ker J0 )⊥ , the smallest singular value of ∇p(0) equals σ0 . By continuity of ∇pe at 0, after shrinking rloc ∈ (0, r0 ] if necessary, we may assume both that UE ⊆ U and that σ0 e e sup ∥∇p(u) − ∇p(0)∥ . op ≤ 2 ∥u∥2 <rloc Now fix u, v ∈ BRd0 (0, rloc ). By the fundamental theorem of calculus, Z 1 e e e e p(u) − p(v) = ∇p(0)(u − v) + ∇pe v + τ (u − v) − ∇p(0) (u − v) dτ. 0

Therefore σ0 e e e ∥p(u) − p(v)∥ − v)∥2 − ∥u − v∥2 2 ≥ ∥∇p(0)(u 2 σ0 ≥ ∥u − v∥2 . 2 Transporting this estimate back through W gives (11). This proves part (a). Fix 0 < ρ < rloc , and let {z1 , . . . , zN } ⊂ UE be a maximal ρ-separated subset. Since the set is maximal, UE ⊆

N [

(z0 + E) ∩ B(zi , ρ) .

i=1

Taking d0 -dimensional Euclidean volumes inside the affine space z0 + E, the standard ball-volume comparison (e.g. Corollary 4.2.13 in [41]) yields rloc d0 N≥ . (14) ρ For i ̸= j, (11) and (9) give ∥p̂(zi ) − p̂(zj )∥2 ≥ ∥p(zi ) − p(zj )∥2 − ∥p̂(zi ) − p(zi )∥2 − ∥p̂(zj ) − p(zj )∥2 ≥ α∥zi − zj ∥2 − 2δ ≥ αρ − 2δ. 30

Using (8), we obtain ∥µ⋆ (zi ) − µ⋆ (zj )∥2 ≥ λ(ρ),

λ(ρ) :=

αρ − 2δ . L

Let ΠS be the orthogonal projection onto S, and set yi := ΠS µ⋆ (zi ) ∈ S,

ui := µ⋆ (zi ) − yi ∈ T ⊥ .

By (10), ∥ui ∥2 = dist µ⋆ (zi ), S ≤ ε. Since yi − yj ∈ T and ui − uj ∈ T ⊥ , the Pythagorean theorem gives ∥yi − yj ∥22 = ∥µ⋆ (zi ) − µ⋆ (zj )∥22 − ∥ui − uj ∥22 ≥ λ(ρ)2 − 4ε2 . p If λ(ρ) > 2ε, the points y1 , . . . , yN are therefore ∆(ρ) := λ(ρ)2 − 4ε2 -separated in S. Setting c0 := ΠS µ⋆ (z0 ), the 1-Lipschitz property of ΠS and (7) give ∥yi − c0 ∥2 ≤ M , so {y1 , . . . , yN } ⊆ S ∩ B(c0 , M ). Assume now that λ(ρ) > 2ε. The balls S ∩ B(yi , ∆(ρ)/2), i = 1, . . . , N , are pairwise disjoint and contained in S ∩ B(c0 , M + ∆(ρ)/2). Comparing k⋆ -dimensional Euclidean ball volumes once more (Corollary 4.2.13 in [41]) gives k N ≤ 1 + 2M/∆(ρ) ⋆ . Combining with (14) yields (12), proving part (b). Finally, fix 0 < ρ < rloc with αρ > 2δ, and define A(ρ) :=

2M (rloc /ρ)d0 /k⋆ − 1

If λ(ρ)2 ≤ A(ρ)2 , then (13) is immediate, since its right-hand side is zero. Assume therefore that λ(ρ)2 > A(ρ)2 . If, toward a contradiction, (13) failed, then we would have 4ε2 < λ(ρ)2 − A(ρ)2 . In particular, λ(ρ) > 2ε, so part (b) applies and yields

rloc ρ

d0 /k⋆

2M ≤1+ p . λ(ρ)2 − 4ε2

Rearranging, p λ(ρ)2 − 4ε2 ≤

2M (rloc /ρ)d0 /k⋆ − 1

= A(ρ),

which contradicts 4ε2 < λ(ρ)2 − A(ρ)2 . Hence 4ε2 ≥ λ(ρ)2 − A(ρ)2 , i.e., ε≥

1/2 1 λ(ρ)2 − A(ρ)2 + , 2

which is (13). This proves part (c). 31

From Theorem 4 to the informal Theorem 2. The informal bound stated in Sec. 4.3 is the special case of part (c) obtained by fixing the inner radius to a constant fraction of the outer radius and discarding the squares. Corollary 1 (Distance-to-task-subspace lower bound; formal form of Theorem 2). Under the hypotheses of Theorem 4, choose ρ := rloc /2 and assume αρ > 2δ, where α = σ0 /2 is the local co-Lipschitz constant from Theorem 4. Then sup dist µ⋆ (z), S

c−δ C , − d /k ⋆ − 1 0 L 2

≥

z∈U

(15)

with c := αrloc /4 and C := M . Proof. With ρ = rloc /2 one has (rloc /ρ)d0 /k⋆ = 2d0 /k⋆ , and part (c) of Theorem 4 reduces to ε ≥ Since

√

i1/2 1h 2 A − B2 , 2 +

A :=

αrloc /2 − 2δ , L

B :=

2M 2d0 /k⋆ − 1

A2 − B 2 ≥ A − B whenever A ≥ B ≥ 0 (as A + B ≥ A − B), we obtain ε ≥ 21 [A − B]+ ≥

M αrloc /4 − δ − d /k . ⋆ − 1 0 L 2

C Identifying the right-hand side with c−δ L − 2d0 /k⋆ −1 yields (15).

Corollary 1 thus instantiates the informal Theorem 2 of the main text: the local constants c and C depend on the local co-Lipschitz constant α, the confinement radius rloc , and the local diameter M , but not on the latent dimensions d0 or k⋆ . The decay in 2d0 /k⋆ is therefore the genuine source of the obstruction: when d0 ≫ k⋆ , the second term in (15) is negligible, leaving a distance-to-S lower bound of order (c − δ)/L. Theorem 4 is strictly stronger, since it provides a packing inequality that holds at every scale ρ < rloc and quantifies the interaction between the prediction error δ, the decoder Lipschitz constant L, and the confinement radius ε.

Empirical Validation of Task-Vector Properties

This section provides the computational and empirical details behind the evaluation of the task-vector properties P0–P3. • Sec. C.1 describes the task-vector extraction procedure used throughout our analysis. • Sec. C.2 reports the empirical evidence for the long-context properties P0–P1, complemented by an OLS probe decomposition for experiment E3 in Sec. C.3. • Sec. C.4 provides the computational and empirical details for the finite-context properties P2–P3. • Sec. C.5 details the simplex-intervention experiment of Sec. 5.2.

C.1

Extracting and Estimating Task Vectors

This subsection records the precise procedure used to extract residual-stream hidden states from the trained transformers and to estimate the task vectors θ̂k that play the role of simplex anchors throughout Sec. 5. The extraction recipe and averaging-based estimator below are reused without modification by every empirical analysis in Sec. C (variance and additivity tests, simplex-projection coefficients βt,k , simplex interventions) and by the orthogonal-subspace interventions in Sec. D. We (ℓ) (ℓ) write H (ℓ) = [h1 , . . . , hT ]⊤ for the collection of residual stream representations across positions. We consider a Pre-LayerNorm (LN) decoder-only transformer, where each layer updates a residual stream H (ℓ−1) ∈ RT ×d via f(ℓ) = H (ℓ−1) + MHA LN(H (ℓ−1) ) , H f(ℓ) + MLP LN(H f(ℓ) ) . H (ℓ) = H We extract representations from the residual stream after the full layer update, i.e., from H (ℓ) , (ℓ) and for each position t collect the layer-wise trajectory {ht }L−1 ℓ=0 . Extraction positions. For discrete-token tasks (E1, E3, E4), the input sequence is [s1 , . . . , sT ] and we extract at all positions, t ∈ {1, . . . , T }. For the linear regression task (E2), the sequence alternates between inputs and targets, [x1 , y1 , x2 , y2 , . . . , xT /2 , yT /2 ]; we extract only at input positions xt , which are the prediction locations for yt . Having specified the residual-stream extraction recipe above, we now describe how the task-vector estimates θ̂k themselves are computed from the extracted hidden states. This averaging-based estimator produces every θ̂k used in the empirical evaluations of Sec. C.2–Sec. C.4.3 and in the simplex-/orthogonal-subspace interventions of Sec. C.5 and Sec. D.5; it is faithful to Definition 1 in the long-context limit. Estimation by averaging. We estimate task vectors directly from the definition in Eq. 2. For each layer ℓ and a window of late context positions Test , we pool hidden states across positions and sequences for each task k: K X X 1 X 1 θ̂k = ht,i − ht,i , |Test | B K ′ |Test | B t∈Test i: zi =k t∈Test i: zi =k′ k =1 | {z } {z } |

X X

task-conditional mean

grand mean

P where the sums over i run over all B sequences assigned to each task. By construction k θ̂k = 0. Test is a contiguous window of ≈ 15–30 late context positions, chosen so that the Bayesian posterior P(z | s≤t ) has largely concentrated and E[ht | z = k] approximates its long-context limit (Eq. 2); empirically, this is the regime where P0–P1 hold to high accuracy (Sec. C.2, Figure 9).

C.2

Long-context stability and decoupling P0-P1

This subsection provides the empirical details behind the long-context properties P0–P1: we report the residual variance ratio across context positions (Sec. C.2.1), showing that task identity and the current token account for most of the variance in hidden states, and we test whether the conditional mean decomposes additively via ANOVA / ANCOVA (Sec. C.2.2). 33

C.2.1

Residual variance ratio

We measure long-context stability (P0) and decoupling (P1) jointly through the residual variance d t | z, st )/Var(h d t ), the fraction of hidden-state variance that is not explained by the task ratio Var(h identity z together with the current token st . A small ratio at large t indicates that the task and current token jointly account for almost all variation in ht , which is the empirical content of P0–P1. This complements Table 1(a) in the main text (which only reports the value at the last position) by displaying the full layer-by-layer trajectory across context positions for E1–E3.

Figure 9: Latent task and last token explain most variance in hidden states at large t. Residual d t | z, st )/Var(h d t ) as a function of context position for E1 (left), E2 variance ratio SSwithin /SStotal = Var(h (middle), and E3 (right). Each curve corresponds to one layer (post-MLP). The ratio decreases as context grows (the model gradually infers the latent) and is smaller in later layers (depth aids task inference).

Figure 9 reports the full residual-ratio curves across context positions; Table 1(a) (main text) summarizes the values at the last position. C.2.2

Testing additive separability

We test whether the conditional mean µ̂t,k,a := Ê[ht | z = k, st = a] decomposes additively as µ̂t,k,a ≈ µ̂t + θ̂k + ν̂a , where the task effect θ̂k and token effect ν̂a do not interact. The key idea is to compare an additive model against a full model that includes a task–token interaction term. Fix a layer ℓ and position t. For experiments E1 and E3 with discrete tokens a ∈ V, the additive model is ht = µ t + θ k + ν a , while the full model allows a separate mean for each (k, a) cell. This is a standard two-way ANOVA. A two-way decomposition of the between-cell variance gives X X X X ∥µ̂t,k,a − µ̂t ∥2 = V ∥θ̂k ∥2 + K ∥ν̂a ∥2 + ∥µ̂t,k,a − µ̂t − θ̂k − ν̂a ∥2 , k,a

SSbetween

}

SStask

}

k,a

SStoken

}

SSinteraction

}

P P where θ̂k = (1/V ) a µ̂t,k,a − µ̂t and ν̂a = (1/K) k µ̂t,k,a − µ̂t . For experiment E2, where the current token xt ∈ RD is a continuous covariate, the additive model is ht = µt + θk + B xt , where B is a shared slope matrix, while the full model replaces B with task-specific slopes Bk . Both models are fit by ordinary least squares (ANCOVA). 34

In both cases, we measure the interaction proportion: the fraction of explained variance at2 tributable to the interaction term. For discrete tokens this is ηinteraction = SSinteraction /SSbetween . For 2 2 − R2 2 continuous covariates, we define the analogous quantity ηinteraction = (Rfull additive )/Rfull , i.e. the 2 fraction of the full model’s explanatory power that is due to task-specific slopes. A small ηinteraction confirms that the additive model is accurate.

Figure 10: Additive separability of task and token effects. All three panels show the interaction 2 proportion ηinteraction : the fraction of explained variance due to the task–token interaction term. Left 2 (E1): two-way ANOVA on discrete tokens. Middle (E2): ANCOVA on continuous covariates, ηinteraction = 2 2 2 (Rfull − Radditive )/Rfull . Right (E3): two-way ANOVA on discrete tokens. Small values confirm that the additive model is accurate. Additivity holds broadly, with significant departures in late layers of E2 and E3.

C.3

OLS probe decomposition for experiment E3

This subsection complements the aggregate analysis in Sec. 5.1 with a layerwise variance decomposition of the hidden states in E3. The simplex-projected interpolation R2 (Figure 2) plateaus at 0.60 and 0.52 in the last two layers, and the ANOVA interaction proportion (Table 1(b)) flags a parallel breakdown of additivity, but neither metric identifies what the residual interaction encodes. We regress each hidden state on three factors (the task label, the current token, and the model’s own output logits) and use the logits as a diagnostic for residual-stream directions aligned with the next-token distribution. Including the logits absorbs essentially all of the late-layer interaction, indicating that the apparent breakdown of additivity is not a genuine task–token interaction but reflects hidden-state components devoted to computing the next-token prediction. The decomposition further shows that task identity is most cleanly represented in middle layers and is progressively subsumed into the next-token signal in later layers, providing a quantitative account of why task vectors are best extracted from middle layers [16]. Probe construction. We regress the residual-stream representation ht on three feature groups computed from the same forward pass: the task label z, the current token st , and the model’s output logit ℓ̂t . The first two are the variables that property P1 predicts to enter ht additively, so any contribution to the late-layer interaction must be absorbed by the third. We use ℓ̂t because it is precisely the direction of the residual stream that the unembedding reads off, and therefore serves as a known summary of the next-token-prediction signal carried by ht . Before fitting we subtract the per-position mean across sequences, h̄t := ht − h̄·,t , to suppress position-dependent variation induced by the rotary positional code. The OLS model is h̄t = Wz ẽz + Ws ẽst + Wlogit ℓ̂t + b + εt , 35

(16)

where ẽz ∈ {0, 1}K−1 and ẽst ∈ {0, 1}V −1 are the one-hot encodings of z ∈ [K] and st with the last column dropped (to avoid collinearity with the intercept), ℓ̂t ∈ RV is the output logit vector, and b ∈ Rd is an intercept. We fit on 213 sequences from the fully trained model for experiment E3 (K=3 major tasks, no minor tasks), evaluated at positions t ∈ [170, 190). Marginal and partial R2 . For a feature group A, the marginal R2 is the R2 from regressing h̄t on A alone (with intercept), and the partial R2 2 ∆RA =

2 2 Rjoint − R−A 2 1 − R−A

(17)

2 quantifies the unique contribution of A, where R−A is the R2 of the reduced model obtained from Eq. 16 by dropping the regressors in A while keeping the intercept and the other two groups. Reporting both is necessary because the regressors overlap: ℓ̂t is a deterministic linear function of the 2 deepest hidden state, so it shares variance with both the task label (Rtask↔logit =0.36) and the current 2 2 token (Rtoken↔logit =0.35); the task and token indicators are nearly orthogonal (Rtask↔token =0.028). 2 The partial R ’s for task and logits should therefore be read as contributions conditional on the linear span of the other groups.

Figure 11: OLS probe R2 decomposition for experiment E3. Left: marginal R2 of each feature group alone. Right: partial R2 (Eq. 17) after controlling for the other two groups. Results are shown layer-by-layer (post-MLP, layers 0–5) at context positions t ∈ [170, 190), with per-position mean subtraction applied to ht before fitting.

Layerwise findings. The decomposition pinpoints the source of the late-layer non-additivity: the next-token-prediction signal ℓ̂t progressively dominates the residual stream at deeper layers, absorbing both task and token information and thereby driving the apparent breakdown of additivity flagged by Table 1(b) and Figure 2. As a corollary, the task label is encoded as a near-independent direction only at middle layers (2–3), while the current token retains a dominant unique contribution at all 2 depths. Quantitatively, the joint model attains Rjoint ∈ [0.92, 0.97] at every layer, so the three groups together give a near-complete linear account of h̄t . The next-token-prediction signal ℓ̂t rises monotonically with depth: ∆R2 ≈ 0.04–0.05 at layers 0–1, jumping to 0.36 at layer 2 and reaching 0.86 at layer 5. The task label has negligible unique contribution at layers 0–1 (∆R2 ≈ 0.03–0.05), 36

peaks at layers 2–3 (∆R2 ≈ 0.70–0.71), and decays to ∼ 0.42 by layer 5. The current token carries the largest unique contribution at all depths (partial R2 ≈ 0.89–0.94 through layer 3, declining to ∼ 0.73 at layer 5), consistent with property P1. The mechanism is now transparent: as ht approaches the unembedding, ℓ̂t absorbs information about both z and st , the task label becomes redundant given ℓ̂t , and the additive interpolation model of Eq. 4 (which has no access to ℓ̂t ) loses fitting capacity, which both explains the late-layer non-additivity and identifies layers 2–3 as the natural depth for task-vector extraction [16].

C.4

Evaluating Finite-Context Properties (P2–P3)

This subsection provides the computational and empirical details behind the finite-context properties P2–P3: closed-form expressions for the oracle Bayesian posterior αt,k in each experimental setting (Sec. C.4.1), the extraction of the simplex-constrained projection coefficients βt,k from hidden states (Sec. C.4.2), and additional posterior-alignment results across layers and with unconstrained coefficients (Sec. C.4.3). C.4.1

Bayesian Posterior Computation under M1

This subsubsection derives the closed-form expressions for the oracle Bayesian posterior αt,k := P(z = k | s1 , . . . , st ) that serve as the comparison target throughout the paper. Under the task-retrieval mode M1 (Sec. 4.2) we take the prior to be uniform over the finite training support Ztrain = [K], so that the integral in Eq. 6 reduces to a finite sum that admits a closed form for each of E1–E3. The resulting αt,k play two roles in our analyses: they are the targets against which the empirical simplex coefficients βt,k are compared to assess Property P3 (Sec. 5.1, Figure 3; Sec. C.4.3), and they define the M1 predictive distribution used in the KL phase-transition analysis of Sec. 6. Throughout, we use the task index convention to refer to the latent variables for simplicity. Biased Dice (E1) Recall that each task z = k is parameterized by a probability vector pk = (pk,1 , . . . , pk,V ) ∈ ∆V −1 over V outcomes (the “faces of the die”). Tokens s1 , s2 , . . . are drawn i.i.d. from Categorical(pk ). The posterior at position t factorizes as Qt

P (s≤t | z = k)

P(z = k | s≤t ) = PK

k=1 P (s≤t | z = k)

Equivalently, let na (t) :=

t X

τ =1 pk,sτ . Qt k′ =1 τ =1 pk′ ,sτ

= PK

1{sτ = a}

(18)

τ =1

be the count of token a in the first t tokens, the log-likelihood is log P(s≤t | z = k) =

V X

na (t) log pk,a .

a=1

The posterior is obtained by normalizing across all K tasks: na (t) a=1 pk,a . QV na (t) K p ′ ′ k =1 a=1 k ,a

αt,k = P

(19)

Noisy Linear Regression (E2) Recall that each task z = k is parameterized by a weight vector wk ∈ RD . The sequence consists of input–output pairs (x1 , y1 ), (x2 , y2 ), . . . where xt ∼ N (0, ID ) and yt = wk⊤ xt + εt with εt ∼ N (0, σ 2 ) independently. At position t, define X≤t = (x1 , . . . , xt )⊤ ∈ Rt×D and y≤t = (y1 , . . . , yt )⊤ ∈ Rt . The cumulative log-likelihood under task k is t

t 1 X log p(y≤t | X≤t , z = k) = − log(2πσ 2 ) − (yτ − wk⊤ xτ )2 . 2 2σ 2 τ =1

We define the cumulative sum of squared residuals SSRk (t) = τ =1 (yτ − wk⊤ xτ )2 . Because the filtering posterior at position t conditions only on past observations (x1 , y1 ), . . . , (xt−1 , yt−1 ), the SSR is evaluated at t−1: exp − 2σ1 2 SSRk (t−1) αt,k = PK (20) , 1 k′ =1 exp − 2σ 2 SSRk′ (t−1) where SSRk (0) = 0 (no observations available at the first position). Latent Markov Chain (E3) Recall that each task z = k is parameterized by a transition kernel P (k) . The state space is V. The first tokens carry no task-discriminative information; therefore αt,k = 1/K for t = 1. For t > 1, each new token st provides a likelihood contribution: P(st = a | s<t , z = k) = Ps(k) , t−1 ,a The posterior accumulates these contributions: P (s≤t | z = k) P(z = k | s≤t ) = PK ′ k′ =1 P (s≤t | z = k ) which yields

(k) τ =2 Psτ −1 ,sτ . Qt (k′ ) K P ′ s ,s τ τ −1 k =1 τ =2

αt,k = P C.4.2

(21)

Computing projection coefficients βt,k

This subsubsection details the two-step projection used to extract the simplex-constrained coefficients βt,k that appear in the interpolation model of Eq. 4 (Sec. 5). These coefficients are the empirical proxies for the Bayesian posterior weights αt,k used in the posterior-alignment property P3, and the procedure below is the one used to produce every βt,k reported in Figure 3. Let θ̂1 , . . . , θ̂K ∈ Rd be the centered task vectors (Sec. C.1), and let (ℓ)

h̄t = ht − µ̂t be the centered hidden state at position t and layer ℓ, where µ̂t is the empirical grand P mean. Following k βk θ̂k : Pthe main text, we obtain βt,k by projecting h̄t onto the task-vector simplex βk ≥ 0, k βk = 1 , implemented in two steps. P P Step 1: Affine projection. We first find coefficients satisfying h̄t ≈ K k=1 βt,k θ̂k with k βt,k = 1, without enforcing nonnegativity. Using θ̂K as an anchor reduces this to an unconstrained leastsquares problem in K − 1 variables, solved via the pseudoinverse of the matrix of pairwise differences (θ̂1 − θ̂K , · · · , θ̂K−1 − θ̂K ). The resulting affine coefficients satisfy sum-to-one but may be negative. 38

Step 2: Simplex projection. To enforce βt,k ≥ 0, we project the affine coefficients onto ∆K−1 via the standard Euclidean simplex projection [9]: sort in decreasing order and apply a uniform threshold so that the result is nonnegative and sums to one. The projected coefficients βt,k are what Sec. 5 refers to as the simplex-projected interpolation coefficients. C.4.3

Additional posterior alignment results

Figure 3 in Sec. 5 reports the simplex-projected coefficients βt,k at a representative middle layer. Here we provide two complementary views: alignment at later layers, and unconstrained (pre-projection) coefficients. Together, these views demonstrate that posterior alignment (property P3) is robust, in the sense that: it holds across layers (not just the representative middle layer of the main text) and survives removal of the simplex projection (not just under the constrained estimator), confirming that the agreement between βt,k and αt,k reflects a property of the hidden states themselves rather than an artifact of layer choice or the projection procedure. Later layers. At later layers the projection coefficients track the Bayesian posterior αt,k even more closely than at the middle layer shown in the main text.

Figure 12: Posterior alignment at later layers: simplex-projected coefficients βt,k (markers with error bars) vs. Bayesian posterior αt,k (dashed lines with shaded 10–90th percentile bands) for data generated from task 1. Compared to the middle-layer results in Figure 3, the agreement is tighter across all three tasks. Left: Dice (E1, layer 5). Middle: linear regression (E2, layer 15). Right: latent Markov (E3, layer 5).

Affine (pre-projection) coefficients. For completeness, we also report the affine coefficients aff , obtained by the same affine projection described in Sec. C.4.2 but without the simplex step βt,k (sum-to-one is enforced, but coefficients may be negative or exceed 1). The close match with the posterior even before simplex projection confirms that the alignment is not an artifact of the projection procedure. Summary. Taken together, Figures 12 and 13 show that posterior alignment βt,k ≈ αt,k is not contingent on the specific layer chosen for visualization in the main text, nor on the simplex-projection step in the estimator: the alignment persists at later layers (and in fact tightens) and survives when aff in place of the simplex-projected β ). the nonnegativity step is removed (Figure 13 uses βt,k t,k

aff Figure 13: Affine projection coefficients βt,k (markers with error bars) vs. Bayesian posterior αt,k (dashed lines with shaded percentile bands). No simplex projection is applied; coefficients may be negative or exceed 1. Left: Dice (E1, layer 3). Middle: linear regression (E2, layer 9). Right: latent Markov (E3, layer 3).

C.5

Simplex Intervention Details

This subsection details the simplex-intervention experiment of Sec. 5.2: at a chosen ℓ we Player ∗ θ̂ in replace the task-subspace component of the residual stream with an arbitrary point k αt,k k K−1 ∗ the task-vector simplex (with αt drawn from ∆ ), leave the orthogonal complement untouched, and measure how closely the steered model’s output matches the Bayesian mixture prediction that α∗t dictates. The purpose is causal: the alignment between βt,k and αt,k established by Property P3 (Sec. 5.1) is observational, and could in principle reflect a hard task-selection rule rather than a true continuous interpolation. The simplex steering, paired with the mode-output baseline introduced below, addresses both points: it shows that the model’s output tracks α∗t throughout the simplex (not merely near vertices), supporting the main-text claim that the task-vector subspace is the variable through which the model interpolates between training tasks. Let Ptask = Θ̂ (Θ̂⊤ Θ̂)† Θ̂⊤

(22)

denote the orthogonal projector onto the task subspace, where Θ̂ ∈ Rd×K stacks the centered task vectors as columns and A† denotes the pseudoinverse of the matrix A. We substitute the task-subspace component with a target interpolation, replacing the model’s original coefficients βt,k ∗ randomly drawn from the simplex ∆K−1 : with coefficients αt,k (ℓ),int ht = µ̂t +

K X

(ℓ) ∗ αt,k θ̂k + (I − Ptask ) ht − µ̂t .

k=1

This steers the hidden representation to a chosen point in the task-vector simplex while leaving the orthogonal complement unchanged. The model then completes the forward pass from layer ℓ onward using the modified hidden state. Comparing with an additional baseline: Mode-output. A central claim of Sec. 5.2 is that task vectors implement a continuous interpolation over tasks rather than a discrete selection of the most likely one. The natural alternative hypothesis is a hard-selection (nearest-neighbor) mechanism: ∗ and predicts as if that task were certain. the model identifies the dominant task k ∗ = arg maxk αt,k To rule this out, we compare simplex steering against a mode-output baseline that realizes this hard-selection rule using the oracle Bayesian posterior over tasks. Across all three experiments, 40

simplex steering tracks the target mixture uniformly across ∆K−1 , while the mode baseline matches the model only near the vertices and degrades sharply in the interior. This shows that task vectors cannot be reduced to a nearest-neighbor or hard-assignment mechanism, strengthening the main-text conclusion that they encode and combine task information continuously across the simplex. Mode-output baseline. Given target coefficients α∗t , the dominant task is ∗ k ∗ = arg max αt,k , k

and the mode-output prediction is the Bayesian-optimal next-token distribution under that single task, P(· | z = k ∗ , st ). This baseline is computed entirely from the ground-truth generative model and does not involve a forward pass through the trained model; as such, it provides the strongest realization of the hard-selection alternative defined above and an upper bound on the performance any nearest-neighbor mechanism in task space could achieve. Empirical comparison. Figure 14 reports the prediction error (KL divergence or RMSE, depending on the experiment) of simplex steering and the mode-output baseline against the target mixture across ∆K−1 for E1–E3. Simplex steering attains uniformly low error throughout the simplex, including the interior region where multiple tasks contribute comparably and any hard-selection rule must discard substantial posterior mass. The mode baseline, by contrast, matches the target only near the vertices and degrades sharply toward the interior, with error growing roughly with the entropy of α∗t . This pattern is precisely what the interpolation hypothesis predicts and what the hard-selection hypothesis cannot accommodate.

Figure 14: Simplex steering outperforms the mode-output baseline across the entire task simplex. Each panel pair shows error (KL divergence or RMSE) under simplex steering (Steered ) and the mode-output baseline (Mode task k ∗ ) for E1 (left), E2 (middle), and E3 (right). Simplex steering achieves uniformly low error throughout the simplex interior, whereas the mode baseline (which selects the pure-task output for the dominant task, and is equivalent to an idealised hard-selection task-vector injection) performs well only near the vertices and degrades in the interior, where task mixtures are more balanced.

Further Details about the Two Inference Modes

This appendix collects the empirical evidence supporting the two-mode picture introduced in Sec. 6, complementing the KL-based transition of Sec. 6, the orthogonality results of Sec. 6.1, and the causal interventions of Sec. 6.2. The material is organized as follows. • Sec. D.1: detailed discussion on the training distribution used in experiments of Sec. 6.1. • Sec. D.2: training dynamics of in-distribution and out-of-distribution loss, providing a performance-based view of the same transition between M1 and M2. 41

• Sec. D.3: a complementary geometric view of the representation orthogonality, tracing simplex trajectories of hidden-state projections under low and high task diversity. • Sec. D.4: pretrained-LLM (Qwen2.5-7B) experiments that extend the two-mode trajectory analysis from synthetic data to a real language model. • Sec. D.5: full per-layer breakdowns of the two complementary causal interventions of Table 2, with implementation details (Sec. D.6) and a decomposition of the optimized orthogonal directions (Sec. D.7) showing they encode running context statistics.

D.1

Major/Minor Training Mixture

This subsection describes the major/minor task-mixture training distribution used throughout Sec. D for the coexistence-of-modes experiments of Sec. 6. The minor-pool size Nminor is the central experimental knob: small Nminor allows every task to be memorized (favoring M1), while large Nminor makes individual minor tasks unmemorizable and forces the model toward in-context generalization (favoring M2). Task pools. Before training begins, two disjoint task pools are sampled once from the task-specific prior and held fixed throughout: • Major pool. Three tasks {z1maj , z2maj , z3maj } drawn from the prior π (e.g. Dir(1V ) for E1 and E3, N (0, ID ) for E2). minor } drawn independently from the same prior π, • Minor pool. Nminor tasks {z1minor , . . . , zN minor conditioned on being disjoint from the major pool. We allow Nminor = 0 as a degenerate case in which the minor pool is empty and all sequences are drawn from the major pool only.

The minor-pool size Nminor is the primary control variable. The full diversity sweep (Sec. A) covers Nminor ∈ {0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024}; Figure 6 displays the five-value subset Nminor ∈ {0, 1, 4, 16, 1024} in all three panels (E1, E2, E3), so that the legend values match exactly across experiments. Sequence sampling. For Nminor ≥ 1, each training sequence is generated by the following two-stage procedure: 1. Pool selection. With probability pminor = 0.1, draw from the minor pool; with probability 1 − pminor = 0.9, draw from the major pool. 2. Task selection. Sample a task z uniformly at random from the selected pool, then generate a sequence of length T from pz . For the degenerate case Nminor = 0 the minor pool is empty: the Bernoulli pool-selection step is skipped and every sequence is drawn from the major pool (equivalently, the effective pminor collapses to 0). For Nminor ≥ 1, this gives major tasks a dominant 9:1 weight over the minor pool regardless of Nminor , so each individual minor task receives weight pminor /Nminor = 0.1/Nminor , vanishingly small for large Nminor .

OOD tasks. Each OOD task is obtained by drawing a fresh sample from the same task-specific prior used to construct the major and minor pools: a Dirichlet draw over categorical distributions for E1 and E3, and an isotropic Gaussian draw over weight vectors for E2. These draws are made independently of the stored training pools and are not cached: every batch of OOD tasks is generated afresh from the prior. Because the prior is continuous, with probability one the resulting task does not coincide with any task in the major or minor pool, so OOD tasks are never seen during training. This ensures that OOD performance reflects genuine out-of-distribution generalization rather than minor-task memorization. Relation to task diversity. When Nminor ∈ {0, 1} the minor pool is empty or contains a single task, which the model can readily memorize; for small Nminor the model essentially extends taskretrieval behavior to the minor pool. As Nminor grows, the per-task training weight 0.1/Nminor becomes negligible and no individual minor task can be memorized, forcing the model to develop a context-based generalization strategy. The transition in representation geometry visible in Figure 6 reflects this shift.

D.2

In-distribution and out-of-distribution loss

To complement the KL-based comparison in Sec. 6, we also examine the training dynamics of indistribution (ID) and out-of-distribution (OOD) loss. ID sequences are drawn from the major/minor training mixture and OOD sequences from fresh prior draws disjoint from both training pools; see Sec. D.1 for the precise sampling procedure. These plots provide a direct performance-based view of the same M1-to-M2 transition: as task diversity increases, the model enters a regime in which it obtains strong OOD performance even though its predictions are no longer best described by M1 (the exact Bayesian posterior over the memorized task set). Figure 15 reports ID and OOD loss curves for E1, E2 and E3. The main qualitative pattern is that OOD loss remains low, or improves substantially during training, in the same region where the KL comparison favors M2. This supports the interpretation that M2 is not merely a worse fit to the M1 baseline but a distinct inference mode that supports effective OOD prediction.

D.3

Simplex trajectories of hidden state projections

Sec. 6.1 (Figure 6) summarizes the orthogonality between OOD and major-task representations through a single scalar: the projection R2 of OOD hidden states onto the task subspace col(Θ̂), tracked over the course of training. OOD tasks throughout this subsection are sampled as described in Sec. D.1: fresh draws from the same prior used to build the major and minor pools, almost surely disjoint from the training pools because the prior is continuous. This subsection provides a complementary geometry-resolved view: instead of reporting how much OOD variance lies in col(Θ̂), we plot where batch-averaged hidden states sit within the affine simplex ∆2 spanned by the three major-task vectors as the in-context length grows. This adds three pieces of information beyond Figure 6: (i) the within-sequence analog of the across-training R2 separation, namely that major-task trajectories converge to the corresponding simplex vertices as context accumulates; (ii) a check that col(Θ̂) genuinely captures major-task structure rather than merely accommodating it as a linear span, since the affine projections of major-task hidden states fall approximately inside ∆2 even though no nonnegativity constraint is imposed; (iii) a direct geometric picture of OOD orthogonality across diversity regimes: OOD trajectories drift far outside the simplex at high task

Figure 15: Training dynamics of in-distribution and out-of-distribution loss. Panels (top to bottom): E1, E2, E3. Each panel compares ID and OOD performance over training. Strong OOD performance in the later training regime of high task diversity is consistent with the emergence of the extrapolative inference mode M2 (cf. Sec. 6).

diversity, whereas at low diversity they remain inside the simplex with R2 comparable to that of major tasks. We extract hidden states from the post-MLP residual stream (cf. Sec. C.1) at an intermediate layer chosen per experiment: layer 3 of 6 for E1, layer 10 of 16 for E2, and layer 4 of 6 for E3 (0-indexed). Two objects are computed from these hidden states. Task vectors (simplex vertices). For each major task k, we pool hidden states across all B = 64 sequences and across a window of late context positions, yielding a single mean vector per task; subtracting the grand mean (the average of these task means) gives the centered task vectors θ̂1 , θ̂2 , θ̂3 . These serve as the three vertices of the reference simplex ∆2 . Trajectories (plotted curves). Separately, for each task group (whether a major task or an OOD task) and each in-context position t, we average hidden states across the B = 64 sequences to obtain a single batch-mean hidden vector, then subtract the grand mean. This averaging over multiple samples greatly reduces the variability introduced by different token realizations, so that the remaining structure reflects the task-level signal rather than token-level noise. For each position t we apply the same affine, sum-to-one projection described in Sec. C.4.2 (Step 1) to this centered aff (sum-to-one, no nonnegativity constraint, batch-mean vector, obtaining the affine coordinates βt,k distinct from the simplex-projected βt,k used for posterior alignment in the main text) that are visualized as a point in the barycentric frame of the simplex ∆2 . Because no nonnegativity is enforced, coordinates may be negative; nevertheless, major-task hidden states tend to fall roughly inside the simplex, confirming that col(Θ̂) genuinely captures their structure. Marker size encodes 44

the projection R2 ; lines trace the trajectory as context grows, with positions sub-sampled for clarity. Major-task convergence and OOD drift across diversity regimes. Figure 16 shows the simplex trajectories for E1–E3 at the two endpoints of the diversity range used in Figure 6: the low-diversity model (Nminor = 0, no minor tasks) and the high-diversity model (Nminor = 210 = 1024 minor tasks). Across all three experiments, the high-diversity panels exhibit two coupled signatures of mode separation: (a) major-task trajectories (solid blue shades) converge toward the corresponding simplex vertices as in-context length grows, and (b) OOD trajectories (red dashed lines) move to coordinates well outside ∆2 and carry smaller R2 , indicating that OOD hidden states are nearly orthogonal to col(Θ̂). The low-diversity panels show neither clean separation nor OOD drift: trajectories are diffuse, and OOD points remain inside or near the same affine simplex with R2 comparable to major-task trajectories, mirroring the high-R2 regime in Figure 6 at small m. The geometric view therefore confirms that the orthogonality reported by the scalar R2 curve corresponds to a literal spatial separation of OOD representations from the task-vector simplex, and that this separation emerges only at high task diversity.

D.4

Pretrained LM Experiments

This subsection draws a qualitative analogy between our controlled two-mode framework and the representation geometry of a real pretrained language model, Qwen2.5-7B [30]. The comparison should be read in light of the property framework of Sec. 4.1 and the two-mode geometry developed in Sec. 4.2, Sec. 4.3, and Sec. 6.1. In the synthetic setting, P0–P3 can be tested directly because the latent tasks, task prior, token space, and oracle Bayesian posterior are all known. In a pretrained LLM, none of these objects are available: natural-language tasks do not come with a specified latent prior, the relevant token space is open-ended, and there is no ground-truth posterior over tasks. We therefore do not treat the pretrained-LLM experiments as formal validations of P0–P3; instead, we ask whether the same qualitative geometric signatures appear. Figure 7 in Sec. 6.1 provides a trajectory-level view: ID prompts move toward the corresponding task-vector vertices, mirroring the long-context stabilization of P0 and the finite-context interpolation of P2, while OOD prompts have small projection R2 onto the ID task subspace, giving a real-LLM analog of the near-orthogonal M2 geometry of Sec. 6.1. Figure 17 provides a complementary coefficientlevel view: on ID prompts, the affine coefficient associated with the prompted task rises toward one as demonstrations accumulate, echoing the posterior-concentration behavior of P3 in the synthetic experiments. The analogy is intentionally limited: a direct test of P1 would require reliable estimation of task–token interaction terms over a large set of real tokens, and a direct test of P3 would require an oracle Bayesian posterior over natural-language tasks. Taken together, the pretrained-LLM results support the external plausibility of the same geometric motifs—task-subspace retrieval for ID prompts and near-orthogonal representations for OOD prompts—while the formal property tests remain those carried out in the controlled synthetic setting. Model. We use Qwen2.5-7B [30], a decoder-only transformer with 7.62 B parameters and 28 transformer layers, loaded in bfloat16 precision via the HuggingFace transformers library [45]. Hidden states are extracted at the query-token position from every layer. Task pool. We consider 12 word-function ICL tasks drawn from the function-vectors dataset [39]: present_to_past, singular_to_plural, english_to_french, english_to_spanish, english_to_german, 45

E1, Nminor = 0

E1, Nminor = 1024

E2, Nminor = 0

E2, Nminor = 1024

E3, Nminor = 0

E3, Nminor = 1024

Figure 16: Affine projection trajectories onto the task-vector simplex for E1–E3. Each panel plots aff aff aff the affine OLS coordinates (βt,1 , βt,2 , βt,3 ) (sum-to-one, no nonnegativity constraint) of the batch-mean centered hidden state at an intermediate layer onto the barycentric frame defined by the three averaging-based task vectors θ̂1 , θ̂2 , θ̂3 (simplex vertices, ⋆). Solid lines (blue shades) trace each major task’s trajectory as context grows; dashed red lines show OOD trajectories. Marker size encodes projection R2 : larger markers indicate that col(Θ̂) explains more of the hidden-state variance. Left column (Nminor = 0): representations are diffuse and not cleanly mode-separated; OOD points remain inside or near the major-task simplex with R2 comparable to major-task trajectories. Right column (Nminor = 1024): major-task trajectories converge toward the simplex vertices with high R2 ; despite having 46 no nonnegativity constraint, the affine coordinates of major-task points lie approximately inside the simplex. OOD coordinates, by contrast, fall far outside the simplex with smaller R2 , directly reflecting their near-orthogonality to col(Θ̂) at high task diversity.

antonyms, synonyms, word_to_category, country_to_capital, person_to_occupation, landmark_to_country, and product_to_company. Each task provides a set of (input, output) word pairs. Task selection. Task vectors are only well-defined when the model reliably performs the underlying task from context. We therefore begin with a lightweight accuracy sweep: for each candidate task we build 20-shot prompts from a random support pool (100 pairs) and evaluate on 20 held-out queries (seed 42). We then select K = 3 tasks that jointly satisfy two criteria: (i) high ICL accuracy, so that the extracted task vectors are informative, and (ii) semantic diversity, so that the resulting task subspace spans meaningfully different directions. This yielded English→French, Antonyms, and Present→Past. Prompt construction. Each prompt consists of N = 20 in-context demonstrations, each formatted as x: y (input word, colon-space separator, output word), separated by newlines, followed by a query line query_x: (no answer). Following Hendel et al. [16], hidden states are extracted at the : separator token immediately after the query input (the second-to-last token of the prompt), which is the position at which the model must implicitly represent the task in order to predict the correct answer. Task vectors θk are computed as the mean query-position hidden state over P = 100 support prompts per task, drawn from a held-out support pool of 200 pairs that is disjoint from evaluation. Evaluation uses 60 held-out prompts per task with disjoint query inputs. OOD task construction. For each ID task fk we construct a paired OOD task of the form (x, g(x)), where g is a surface-level transform applied to the input x. Applying a syntactic manipulation to the raw input word, rather than composing with the learned ID mapping, yields tasks that are unnatural as word-function tasks: outputs such as the first letter of a word, a word with its final letter capitalized, or a word repeated twice do not correspond to meaningful linguistic relations and are unlikely to appear as coherent tasks in natural language corpora, making them plausibly out-of-distribution for a pretrained model. Concretely: • french_input_first: g(x) = first character of x (e.g. “table” → “t”). • antonym_input_cap_last: g(x) = x with its last letter capitalized (e.g. “cat” → “caT”). • past_input_double: g(x) = x concatenated with itself (e.g. “run” → “runrun”). At the same time, these transforms are applied directly to the input and are fully inferable from the 20 in-context demonstrations alone, without any prior knowledge of the ID task fk . This ensures the model can achieve reasonable OOD accuracy (see Table 5), making the OOD evaluation meaningful rather than a trivial failure case. Metric: simplex-projection R2 . Following the analysis in the main paper, we project each query-position hidden state h onto the affine hull of the K task vectors and compute R2 = 1 −

∥h − ĥ∥2 , ∥h − h̄∥2

where ĥ is the affine projection and h̄ is the mean hidden state across all prompts. R2 close to 1 indicates that the hidden state lies in the task subspace; R2 near 0 indicates it is largely orthogonal. 47

Table 5: ICL performance (next-token accuracy and cross-entropy loss) on ID and OOD tasks for Qwen2.5-7B with 20-shot prompts. Type

Task

Accuracy

Cross-Entropy Loss

ID ID ID

English→French Antonyms Present→Past

71.7% 60.0% 96.7%

1.172 1.649 0.386

OOD OOD OOD

french_input_first antonym_input_cap_last past_input_double

33.3% 53.3% 73.3%

1.914 1.501 0.928

Trajectory visualization. To produce Figure 7, we treat the number of in-context demonstrations as a discrete “time” axis. For each evaluation prompt, we construct a sequence of truncated prompts containing 1, 2, . . . , N demonstrations, extract the hidden state h at the query position for each, and project it onto the 2-simplex defined by the three task vectors θk (using barycentric coordinates after affine projection). The trajectory of these projected points across shot counts is plotted inside the triangle. ID trajectories (solid, colored by task) converge toward the correct task vertex; OOD trajectories (dashed red) remain scattered outside the simplex. Marker size is proportional to the R2 of the projection at that shot count. The figure shows layer 20, which exhibits the clearest task-subspace organization. Additional result: Posterior alignment in the pretrained LLM. In the synthetic experiments, the affine coefficients βt,k (obtained by projecting ht onto the task-vector simplex) track the Bayesian posterior over tasks as context accumulates (Property P3, Eq. 5). We test the analogous P prediction for Qwen2.5-7B by computing unconstrained affine coefficients βt,k (constrained to k βt,k = 1 but allowed to be negative) at each shot position t and each layer, using an enlarged basis of K = 6 task vectors: the three ID tasks above together with singular_to_plural, landmark_to_country, and english_to_spanish, all extracted by the same procedure. Figure 17 shows the mean βt,k at layer 20 on ID evaluation prompts. For each ID task (column), the coefficient on the correct task rises steeply toward 1 as demonstrations accumulate, while every other coefficient remains near 0. This provides a posterior-alignment analog in a pretrained LLM: without a ground-truth Bayesian posterior, the affine coefficients nevertheless concentrate on the prompted task as demonstrations accumulate.

D.5

Per-Layer Intervention Results

This subsection reports the per-layer breakdown of the two complementary causal interventions introduced in Sec. 6.2 (suppressing the optimized orthogonal directions V̂opt , and suppressing the task subspace col(Θ̂)). The layer-resolved curves confirm that the major-vs-OOD/minor double dissociation aggregated in Table 2 is a genuine middle-layer phenomenon rather than an averaging artifact, and reveal a structured three-phase depth profile that the main-text average hides. Orthogonal subspace intervention. Table 2 in the main text reports ∆Lmode /gmode averaged over the middle-layer ranges where orthogonal separation is cleanest; Figure 18 shows the full per-layer breakdown. The pattern across all layers follows a three-phase progression. In early layers, computation is shared across major, OOD, and minor tasks (the model has not yet separated 48

Figure 17: Posterior alignment analog in Qwen2.5-7B (layer 20), ID prompts. Each panel shows one of the K = 6 ID tasks (columns). Lines show the mean affine coefficient βt,k for each task k as a function of shot position t. For each ID task, βt,k for the correct task k (matching column color) rises toward 1 as demonstrations accumulate, while coefficients for other tasks remain near 0, analogous to Bayesian posterior alignment observed in synthetic transformers (Figure 3).

task-specific from prediction-specific representations), so suppressing any direction in the orthogonal complement affects all three loss types simultaneously (e.g. layers 0–4 of E2, where major loss reaches ≈95–253% alongside large OOD and minor disruption). In middle layers, the orthogonal separation emerges cleanly: the model has routed task-identity information into the protected subspace and prediction-relevant information into the orthogonal complement, so suppressing V̂opt selectively disrupts OOD and minor inference while leaving major-task performance near the random baseline; these are the layers whose averages appear in Table 2. In final layers, the residual stream feeds directly into the output head and must encode the next-token prediction regardless of task type; the distinction between task identity and predictive content collapses, and the intervention again affects all three losses (e.g. layer 5 of E1 and E3, layers 14–15 of E2). Task subspace suppression. The intervention applied here is h′ = h − γ Ptask h, with experimentspecific γ = 1.5 for E1, γ = 2 for E2, and γ = 2.5 for E3. As with the orthogonal subspace intervention, γ = 1 would correspond to ordinary projection removal; since all values used here satisfy γ > 1, the intervention is an amplified suppression (over-subtraction) of the task component rather than a pure ablation. The layer-averaged results for this intervention appear in the left block of Table 2 in the main text (Sec. 6.2). Figure 19 provides the full per-layer breakdown. In E1 (Dice, panel (a)), suppressing the task subspace devastates major loss (≈113–362%, increasing with depth) while OOD and minor losses remain low (≈5–60%), with the gap widening in later layers, consistent with the 211% / ≈12% contrast reported in Table 2. In E3 (Latent Markov, panel (b)), major loss rises sharply from layer 2 onwards (≈96–190%), while OOD and minor losses are near zero at layers 2–4 (≈0–1%); layers 0–1 show modest OOD/minor disruption (≈17–32%), consistent with the early-layer shared-computation regime identified above and the 132% / ≈13% contrast in Table 2. In E2 (Linear Regression, panel (c)), major loss is elevated in early and middle layers (≈13–138%, peaking near layers 4–5 and 7–9), while OOD and minor losses track major loss in the early shared-computation 49

Figure 18: Per-layer orthogonal-subspace intervention across all three experiments. Each bar shows ∆Lmode /gmode × 100% when V̂opt is suppressed at that layer; error bars are the inter-quartile range over evaluation batches; the shaded band and horizontal dashed line mark the random same-rank baseline and the 100% reference respectively. (a) E1 (Dice): OOD loss is disrupted substantially at all layers (≈53–155%) and minor loss uniformly (≈78–138%); major loss stays near zero at layers 0–4 (≈1–21%) but rises to ≈80% at layer 5. (b) E3 (Latent Markov): OOD and minor losses are largest at layers 0–1 (≈102–124%) with a dip at layer 2 (≈54–61%); major loss stays near the random baseline at layers 1–4 (≈5–17%); layer 5 reverses the pattern (major ≈113%, OOD/minor ≈63–66%). (c) E2 (Linear Regression): major loss is elevated at layers 0–4 (≈95–253%) and resurfaces at layers 7 and 14–15; layers 5–6 show a transient dip in OOD disruption (≈30–50%); layer 13 produces the largest OOD/minor spike (≈295–310%); the focal layers 9–12 give the cleanest orthogonal separation (major ≈5–15%, OOD/minor ≈130–175%).

phase (layers 1–5) but drop to near zero from layer 8 onwards, giving the 84% / ≈−1% pattern shown in Table 2.

Figure 19: Per-layer task subspace suppression across all three experiments. Each bar shows ∆Lmode /gmode × 100% when the task subspace col(Θ̂) is suppressed at that layer. This is the complementary intervention to Figure 18: suppressing the task subspace selectively disrupts major-task performance while leaving OOD and minor losses largely intact, confirming that the two subspaces carry functionally distinct information. (a) E1 (Dice): major loss rises from ≈113% at layer 0 to ≈362% at layer 5; OOD and minor losses stay below ≈60% throughout. (b) E3 (Latent Markov): major loss rises sharply from layer 2 (≈96–190%); OOD and minor losses are near zero at layers 2–4 but present at layers 0–1 (≈17–32%) and layer 5 (≈31–34%). (c) E2 (Linear Regression): major loss peaks in the early-to-middle layers (≈120–138% at layers 4–5); OOD and minor losses track major loss in the early shared phase (layers 1–5) but drop to near zero from layer 8 onwards.

D.6

Orthogonal Subspace Intervention: Implementation Details

This appendix gives a self-contained algorithmic description of the orthogonal subspace intervention introduced in Sec. 6.2. Recall that the training distribution mixes a small set of major tasks Zmajor (probability 0.9) with a large pool of minor tasks Zminor (probability 0.1); OOD tasks are fresh prior draws disjoint 51

from both pools (see Sec. D.1). At high diversity (Nminor ≫ 1), each individual minor task receives negligible weight and cannot be memorized; the model must instead rely on in-context demonstrations, just as it does for OOD tasks. We exploit this similarity by using minor-task sequences to search for directions in the orthogonal complement whose suppression maximally disrupts next-token prediction. Crucially, no major-task data enters the optimization, so the procedure provides no explicit guarantee that major-task performance will be preserved. The finding that major-task loss nonetheless remains near zero across a range of middle layers (Table 2) therefore constitutes strong evidence that major-task and OOD/minor-task inference operate through distinct computational mechanisms. All evaluation metrics are computed on independently sampled held-out sequences that are disjoint from the data used to find optimization directions, avoiding overfitting. (ℓ,b) We extract hidden states hk,t ∈ Rd from layer ℓ of the frozen model at batch sample b, task k, and sequence position t. All operations below are performed independently per layer ℓ; we suppress the superscript ℓ for readability. Step 1: Task and token subspace estimation. The task subspace basis Θ̂ ∈ Rd×K is obtained from the averaging-based task vectors θ̂1 , . . . , θ̂K (stacked as columns; K = 3 is the number of major tasks) exactly as in Sec. C.1. In addition, we estimate token encoding vectors ν̂v ∈ Rd as the token main effects from the same two-way (token × task) ANOVA used for the separability analysis (Sec. 5.1): after forming cell means H̄ ∈ RV ×K×d from token-conditioned hidden states 1 P and removing positional effects, ν̂v = K k H̄v,k − h̄. Step 2: Protected subspace construction. Next-token prediction in E1–E3 depends on the current token st (e.g. the bigram structure in E3 or the input xt in E2), so suppressing directions that encode token identity would trivially degrade predictions for all task types and render the intervention uninformative. We therefore protect the token subspace alongside the task subspace. The token ⊤ , encoding vectors Wtok ∈ RV ×d (rows {ν̂v }) are projected onto col(Θ̂)⊥ via Wres = (I − Ptask )Wtok where Ptask is given in Equation (22). The top-ρ left singular vectors of Wres are retained as token-protection directions. The task and token directions are concatenated and re-orthonormalized via compact SVD to yield Uprot ∈ Rd×p , and the orthogonal complement basis U⊥ ∈ Rd×q ,

q = d − p,

⊤ with eigenvalue exceeding 1 . is extracted as the eigenvectors of P⊥ = Id − Uprot Uprot 2

Step 3: Direction optimization. We search for a rank-r subspace within col(U⊥ ) whose suppression maximally disrupts minor-task prediction (r = 6 for E1 and E3; r = 2 for E2). The learnable parameter G ∈ Rq×r is initialized as a random unit-norm matrix and updated with Adam (learning rate 0.01, gradient clip norm 1.0) for up to Nopt steps (80 for E1, 90 for E3, 25 for E2), with early stopping after 50 consecutive steps without improvement in a smoothed objective (EMA coefficient 0.1). At each step the intervention directions are VQ = QR(U⊥ G), giving the intervention h′ = h − γ VQ VQ⊤ h, with γ = 2.5 for E1 and E3, and γ = 2 for E2, applied via a forward hook at layer ℓ. Here γ = 1 would correspond to ordinary orthogonal projection removal. Since the values above satisfy γ > 1, the intervention is an amplified suppression (over-subtraction) of the selected component rather than a pure ablation; the optimized directions and the random same-rank baseline use the same γ 52

within each experiment. The objective is maximized over batches of size Bopt = 256 drawn from the minor-task distribution. An exponential moving average of G with decay 0.99 is maintained; the final directions are V̂opt = QR(U⊥ Ĝ). Step 4: Evaluation. All evaluation is performed on independently sampled sequences that are disjoint from the batches used during Step 3. On held-out batches of size B from the major-task, OOD, and minor-task distributions, we record the baseline and intervened loss at all evaluation positions t ∈ Teval and report ∆Lmode for mode ∈ {maj, ood, minor} as defined in Sec. 6.2. As a sanity check, the same intervention is applied with r random orthogonal directions sampled uniformly from col(U⊥ ), and the resulting loss increase is compared against ∆Lood to confirm that the optimized directions are non-trivially better than chance. Linear probes are then fitted from ⊤ h on a separately held-out probe set of size N 2 ground-truth latent features to V̂opt probe , and R is reported on an 80/20 validation split. Hyperparameters. Unless stated otherwise, we use ρ = ⌊0.9 · rank(Wres )⌋ token-protection components, |Teval | = T (all positions), and Nprobe = 1024 probe samples.

D.7

Decomposition of Orthogonal Directions

Together, the two complementary interventions above (suppressing V̂opt and suppressing col(Θ̂)) establish a clean double dissociation: directions in col(U⊥ ) are causally responsible for OOD and minor-task inference, while the task subspace carries major-task information. Having confirmed this functional separation, we now ask a more fine-grained question: what information does V̂opt concretely encode? Method. For each layer ℓ and experiment we collect hidden states h from OOD/minor-task ⊤ h ∈ Rr . We regress z onto individual sequences and compute the low-dimensional projection z = V̂opt observable features constructed solely from the context (s1 , . . . , st ) for E1/E3, or the input–output pairs for E2, then report the combined R2 from fitting all features jointly (dashed line in figures). For experiments E1 and E3 we use the centred log-ratio (CLR) transform of Aitchison [1], applied to empirical token frequencies over the V -symbol vocabulary. Raw empirical frequencies can contain zero entries (a symbol may not yet have appeared in the prefix, and most rows of the empirical bigram table are zero at small t), and CLR requires strictly positive arguments. We therefore first apply additive (Dirichlet) smoothing with the Krichevsky–Trofimov prior α = 12 , then take the centred log-ratio of the resulting strictly positive frequency vector f ∈ ∆V>0−1 : V

1 X CLR(f )a = log fa − log fb , V

a = 1, . . . , V.

(23)

b=1

This maps ∆V>0−1 to the zero-sum hyperplane in RV , removing overall scale while preserving relative frequency information. In implementation a numerical floor of 10−12 is additionally applied inside the logarithm to guard against underflow; with α = 12 this floor is never active in practice and is purely defensive. Note that V here is the symbol-alphabet size (cf. Sec. 3); the smoothing constant α is a feature hyperparameter chosen by convention and is distinct from the generative Dirichlet hyperparameter α0 used in Sec. B.3.

• E1: Unigram CLR: CLR(p̃t ), where the V -vector p̃t ∈ ∆V>0−1 has entries p̃t,a =

na (t) + α , t + αV

a = 1, . . . , V,

with the unigram count na (t) defined in Eq. (18); Current token: one-hot encoding est ∈ RV ; Position: fractional sequence position t/T . • E3: Bigram CLR: CLR(q̃t ), where the V -vector q̃t ∈ ∆V>0−1 is the smoothed conditional frequency of the next symbol given the current token st , q̃t,b =

Nst , b (t) + α , Nst ,· (t) + αV

b = 1, . . . , V,

with the bigram counts Nab (t) and row marginal Na· (t) defined as in Sec. B.3; Current token: one-hot encoding est ; Position: fractional sequence position t/T . −1 ⊤ 2 ⊤X X<t y<t , the running ridge estimator from • E2: ŵt := ŵridge (t − 1) = X<t <t + τ ID Sec. B.3 that defines the extrapolative M2 predictor. The ridge penalty is set to τ 2 = σ 2 to match the unit-variance prior of Sec. 3 (cf. Sec. B.3); with noise scale σ = 0.5 this gives τ 2 = 0.25. The estimate is updated online using only the completed pairs (x1 , y1 ), . . . , (xt−1 , yt−1 ) that are causally available at the extraction position xt (the prediction location for yt ; cf. Sec. C.1); by convention ŵridge (0) = 0 (the prior mean). xt (current input); x⊤ t ŵt (diagnostic prediction 2 score); ∥xt ∥2 (input norm). We report held-out linear-probe R2 (80/20 split) for each individual feature and for the combined set. To test the causal significance of the linearly accessible content, we extract the filtered subspace Vfilt : fit a linear map from the combined features to z, take the SVD of the fitted values, and retain directions explaining > 0.5% of total variance. We then run the causal intervention with Vfilt in place of V̂opt and compare the resulting loss increases. Results: Dice (E1). Figure 20(a) reveals that the orthogonal direction encodes different context statistics at different depths. In the early layers (0–2), the current token one-hot already explains the majority of V̂opt (R2 ≈ 0.81 at layer 0), while the unigram CLR contributes R2 ≈ 0.35; together the combined features achieve R2 ≈ 0.94. As depth increases, the current token’s contribution decays to ≈ 0.14 by layer 5, while the unigram CLR rises steadily to ≈ 0.88. The combined R2 remains high throughout (0.83–0.94), indicating that observable token statistics explain V̂opt well at every layer. Figure 21(a) confirms this causally: the filtered bars are nearly indistinguishable from V̂opt at every layer (≈75–135% for both OOD and minor), reproducing its causal effect almost exactly. This shows that what the orthogonal directions encode changes across layers, shifting from raw token identity in early layers to the running empirical frequency distribution in later layers, but the content is linearly accessible from observable statistics throughout. Results: Latent Markov (E3). The Latent Markov setting presents a markedly different pattern (Figures 20(b) and 21(b)). The combined observable features explain virtually none of V̂opt in the first three layers (R2 ≈ 0.01–0.04 at layers 0–2), then rise sharply to ≈ 0.61 at layer 3 and ≈ 0.79–0.85 at layers 4–5. At layer 3, current token and bigram CLR contribute comparably (R2 ≈ 0.34 and 0.26 respectively); beyond that, bigram CLR becomes the dominant predictor (R2 ≈ 0.75 at layer 5) 54

⊤ h across all three experiments. Solid lines show individual Figure 20: Per-feature R2 predicting V̂opt feature contributions; the dashed line shows the combined R2 from regressing on all features jointly (see Eq. 23 for the CLR definition used in E1 and E3). (a) E1 (Dice): current token dominates early (R2 ≈ 0.81 at layer 0) and decays, while unigram CLR rises from ≈ 0.35 to ≈ 0.88 by layer 5; combined R2 remains 0.83–0.94 throughout. (b) E3 (Latent Markov): combined R2 ≈ 0.01–0.04 at layers 0–2, rising to ≈ 0.61 at layer 3 and 0.79–0.85 at layers 4–5; bigram CLR becomes the dominant predictor (R2 ≈ 0.75 at layer 5). (c) E2 (Linear Regression): in early layers, the input statistics dominate, with xt peaking at R2 ≈ 0.64 at layer 0 and ∥xt ∥2 peaking at R2 ≈ 0.77 at layer 2; from layer 5 onward, the (causal) prediction score xt · ŵt takes over and peaks at R2 ≈ 0.76 at layer 9 and R2 ≈ 0.83 at layer 12; ŵt stays near zero throughout; combined R2 ranges from ≈ 0.21 at the layer-4 transition dip up to ≈ 0.93 at layer 12.

while current token decays to ≈ 0.10 and position contributes negligibly throughout. The filtered intervention (Figure 21(b)) mirrors this pattern causally: at layer 1 the filtered bars are near zero (≈10%) despite V̂opt causing ≈115–123% disruption. Layers 0 and 2–3 show partial filtered effects (≈25–56% vs V̂opt ’s ≈70–102%), and at layers 4–5 the filtered bars closely track V̂opt (≈65–85%). This reveals a genuine developmental transition: in early layers V̂opt encodes prediction-relevant information that is not linearly accessible from simple token statistics, whereas in later layers the orthogonal directions develop a linearly readable bigram-based predictor. Results: Linear Regression (E2). Figures 20(c) and 21(c) show the results for E2. The R2 plot (Figure 20(c)) reveals a two-phase per-feature progression separated by a sharp transition. In the early layers (0–3) the input statistics dominate: the current input xt peaks at R2 ≈ 0.64 at layer 0 (with a secondary peak of ≈ 0.55 at layer 3), and the input norm ∥xt ∥2 peaks at R2 ≈ 0.77 at layer 2, indicating that the orthogonal direction in this regime tracks the raw current input together with its scale. From layer 5 onward, the prediction score xt · ŵt (the inner product of the current input with the causal running ridge estimate, which measures how well the input aligns with the estimated task direction) takes over as the dominant feature, peaking at R2 ≈ 0.76 at layer 9 and R2 ≈ 0.83 at layer 12; ∥xt ∥2 persists as a moderate secondary contributor in the deeper layers (with sporadic peaks of ≈ 0.23–0.37 at layers 10–11 and 13). The running estimate ŵt itself contributes negligibly throughout, confirming that the compressed orthogonal direction tracks the scalar prediction score rather than the full ridge/weight estimate. The combined R2 ranges from ≈ 0.21 at the layer-4 transition dip to ≈ 0.93 at layer 12, with a notable dip at layer 4 coinciding with the transition between the input-statistics-dominated and xt · ŵt -dominated regimes. The filtered intervention (Figure 21(c)) closely reproduces V̂opt ’s causal effect on OOD and minor losses across most layers, with filtered and full bars essentially indistinguishable at every layer except layer 4. At the layer-4 transition, the combined probe R2 dips to ≈0.21 and the R2 ≥0.05 filter retains only one of the two Vopt directions (with low joint R2 ); the filtered bar consequently collapses to ≈12% even though V̂opt itself disrupts ≈182% of the ICL gain. This single-layer outlier coincides exactly with the input-statistics-to-prediction-score regime change identified in the R2 analysis above, and the filtered subspace recovers V̂opt ’s causal effect once the joint R2 rises again at deeper layers, confirming that the causally active content elsewhere is well explained by these observable statistics. Summary. The three settings reveal distinct patterns in both what and when V̂opt encodes prediction-relevant context. In E1 (Dice), simple token statistics explain V̂opt well at all depths (R2 ≥ 0.83 combined), but the dominant feature shifts across layers: current token identity predominates early while unigram CLR takes over in later layers. The filtered subspace accounts for essentially all of V̂opt ’s causal power throughout the network. In E3 (Latent Markov), token statistics explain almost nothing in layers 0–2 (R2 ≈ 0.01–0.04), then become strongly predictive from layer 3 onwards as bigram CLR and current token emerge jointly (R2 ≈ 0.61 at layer 3, rising to 0.79–0.85 at layers 4–5). Correspondingly, the filtered intervention is near zero at layer 1 and partial at layers 0 and 2–3, only reproducing the full causal effect at layers 4–5; the large early-layer effects of V̂opt are not linearly explainable by observable token statistics. In E2 (Linear Regression), the dominant feature shifts from input statistics in the early layers, where xt peaks at R2 ≈ 0.64 at layer 0 and ∥xt ∥2 peaks at R2 ≈ 0.77 at layer 2, to the (causal) prediction score xt · ŵt from layer 5 onward (peak R2 ≈ 0.83 at layer 12), with ŵt contributing negligibly throughout. The filtered subspace accounts for the causal effect on OOD and minor tasks across most layers, with the exception of the layer-4 R2 transition dip where the filter discards the causally active directions. In all cases, 56

Figure 21: Causal comparison of V̂opt vs. filtered subspace Vfilt across all three experiments. Dark bars show the full V̂opt intervention; light bars show the filtered subspace derived from the combined observable features. (a) E1 (Dice): filtered bars are nearly indistinguishable from V̂opt at every layer (≈75–135%), confirming that the causally active content is linearly accessible from token statistics throughout. (b) E3 (Latent Markov): filtered bars are near zero at layer 1 (≈10%) despite V̂opt causing ≈115–123% disruption; partial at layers 0 and 2–3 (≈25–56%); closely match V̂opt at layers 4–5 (≈65–85%), confirming a developmental transition in linear accessibility. (c) E2 (Linear Regression): filtered bars closely match V̂opt at every layer except layer 4; the layer-4 filtered bar collapses to ≈12% because the combined probe R2 dips to ≈0.21 at the input-statistics-to-prediction-score transition and the R2 ≥0.05 filter retains a direction with low joint explanatory power, missing the causally active component there.

the filtered subspace Vfilt reproduces essentially all of V̂opt ’s causal power once the combined R2 is appreciable, directly confirming that col(U⊥ ) stores linearly accessible running prediction statistics that drive extrapolative in-context learning.

Non-Markovian Case: Dyck Path Experiment (E4) Details

This section provides the methodology and detailed results behind the planted Dyck experiment (E4) used as the motivating non-Markovian counterexample in Sec. 7. The experiment matters for our main thesis because it shows that property P0 (long-context stability) is conditional on the data-generating structure rather than an innate property of trained transformers: when the source is non-Markovian and admits no compact sufficient statistic, the model retains the full prefix history instead of collapsing onto a low-dimensional summary. The main text (Figure 8) reported the residual variance left in ht after conditioning on the task z and a local context window, and visualized the cluster structure of the final-layer hidden states at a single prefix length. This section sharpens that diagnosis along two complementary axes. First, we train linear probes that classify hidden states by the identity of the full planted Dyck prefix at every length l = 1, . . . , 7 and report the resulting validation accuracies, providing direct quantitative evidence (beyond the visual cluster separation shown in the main text) that the model retains an essentially lossless record of the prefix history rather than a compact summary statistic. Second, we show that this record is recoverable through a single shared low-dimensional projection across all prefix lengths and mask realizations, indicating that the prefix information is arranged along consistent directions in hidden-state space. Together these results constitute a sharp failure of property P0 in E4 that does not occur in E1–E3. Sequence construction. The main text introduces E4 (Sec. 7) by describing each sequence as a length-T background of i.i.d. uniform tokens into which the latent Dyck string z of length 2L is planted at positions selected with probability ρ. We expand on the planting rule here so the case ρT > 2L is unambiguous. Concretely, given z = (z1 , . . . , z2L ) we draw a Bernoulli planting indicator i.i.d.

vector ξ ∈ {0, 1}T with ξt ∼ Bernoulli(ρ), then enumerate its successes 1 ≤ τ1 < τ2 < ·P · · in leftto-right order and overwrite background token bτj with zj for every j ∈ {1, . . . , min(2L, Tt=1 ξt )}. In other words, z is embedded in order into the first 2L Bernoulli successes of ξ, and all other positions, including any successes beyond the 2L-th, keep their original background token. With the experimental settings used throughout this paper, T = 192, 2L = 20, ρ = 0.25, the expected number of Bernoulli successes is ρT = 48 ≫ 2L, so undersampling (fewer than 2L successes) occurs with vanishingly small probability and the entire latent string z is essentially always planted; the suffix of the sequence after the 2L-th plant is pure background and therefore carries no further task-conditional signal. Prefix probe. Consider the model trained on experiment E4 with K Dyck tasks (balanced bracket strings of length 2L = 20). At each sequence position, define the Dyck prefix length l as the number of planted bracket characters observed so far. Two tasks that share the same first l planted characters are indistinguishable given the observed Dyck prefix up to that point. We group tasks by their length-l prefix and define the prefix class of a hidden state as the identity of this prefix. The number of distinct prefix classes grows with l: for instance, l = 2 yields only 2 classes (open or close), while l = 7 yields 35 classes. We train a linear probe consisting of a shared 2-dimensional linear projection W ∈ R2×d followed by a separate two-layer MLP classifier per prefix length. The probe is trained jointly across all prefix 58

Figure 22: Hidden-state geometry for the planted Dyck language (E4). Two-dimensional projection of final-layer hidden states at prefix length l = 7 (yielding 35 distinct Dyck prefixes). Points form well-separated clusters, with both clusters and induced decision regions colored by the number of unmatched left parentheses. The clear separation across all prefix classes indicates that the representation preserves the full Dyck prefix history, rather than compressing it to low-dimensional summary statistics. This is the same projection (and the same coloring) as the right panel of Figure 8; the only difference is that each cluster is annotated here with its literal bracket prefix, whereas the main-text panel encodes the same prefixes as compact hexadecimal codes for legibility at smaller figure size.

lengths l = 1, . . . , 7, using hidden states extracted from layer 5 (the final layer). Training and validation sets are constructed by generating sequences under multiple random planting masks and collecting hidden states at positions where exactly l Dyck characters have been planted. Probing results. The probe achieves near-perfect validation accuracy (> 95%) for all prefix lengths up to l = 7 (the longest tested), reaching 98.6% at l = 7 despite the rapid growth in the number of classes. Figure 22 shows a shared 2-dimensional linear projection at l = 7, where the 35 distinct prefix classes form well-separated clusters. Points are colored by the number of unmatched left parentheses, and the shaded regions depict the decision boundaries of the per-length MLP classifier. This visualization is identical to the right panel of Figure 8 in projection and coloring; the only difference is that here each cluster is labeled by its literal bracket prefix, whereas in the main-text panel the same prefixes are encoded compactly in hexadecimal for legibility. The hexadecimal encoding is obtained by mapping each step to a bit (‘(’ → 1, ‘)’ → 0), dropping the leading ‘(’ (present for all prefixes), padding the remaining bits to a multiple of 4, and converting to hexadecimal; for example, the length-7 prefix (()())( yields bits 1, 0, 1, 0, 0, 1 after dropping the first ‘(’, and padding to 8 bits gives 101001002 = A4. This result has two implications. First, the hidden states encode a representation of the full Dyck prefix, not a compressed statistic such as the running difference between opening and closing brackets (which would yield only ⌊l/2⌋ + 1 distinguishable states at prefix length l, not the valid Dyck-prefix classes observed). Second, because this encoding is recoverable through a shared 2-dimensional linear projection, the prefix information is arranged along consistent directions across different prefix lengths and mask realizations. 59

Record · ID 155301 · SHA-256 4b4a4179347df098
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.