Handover of In-Context Learning State Across Session Boundaries

Handover of In-Context Learning State Across Session Boundaries Masahiro Kato∗ 1 and Taka Kato2

arXiv:2608.14528v1 [cs.AI] 14 Aug 2026

1

Mizuho-DL Financial Technology, the University of Tokyo, RIKEN AIP, and Osaka Metropolitan University 2 NP-hard August 17, 2026

Abstract This study investigates the methodological and theoretical properties of session handover in applications that use large language models. A task may continue in a new session when the context reaches the model’s input limit, when the application restarts, or when another agent is asked to finish the task. The application must then decide which information from the earlier session to pass on. We formulate handover as the transfer of a task-relative in-context learning (ICL) state and distinguish exact recovery of earlier material from preservation of the target distribution. Under an exogeneity condition, predictive equivalence characterizes the coarsest deterministic sufficient handover and gives a fixed-length bit requirement. The analysis isolates the effects of the memory constraint, the writer, and the continuation procedure, and quantifies the cost of writing before the realized downstream query is known. We propose a three-part record that stores decisions and constraints exactly, uses task-justified statistics for repeated evidence, and retains original observations whose effect is not preserved by those statistics. Gaussian linear regression gives an exact finite-dimensional handover and finite-bit perturbation bounds, while nonparametric regression gives upper and lower bounds that relate memory to squared prediction error. These results provide a theory and method for deciding what a handover must retain and how its memory requirement depends on the continuation task.

Keywords: session handover; in-context learning; predictive sufficiency; memory-constrained prediction; prompt compression ∗

Email: mkato-csecon@g.ecc.u-tokyo.ac.jp

1

1

Introduction

An application may call a large language model more than once while completing a task, and we call the application an agent when the model chooses an action, the application executes it, and the result is included in a later call. A session is a sequence of model calls that share the same earlier messages and tool results. The context of a call is all information supplied to the model in that call. As the task continues, the context may reach the model’s input limit and require the application to shorten the context or start a new session. A new session may also begin after the application restarts or when another agent is asked to finish the task. In the setting studied here, the new session receives only the information that the application passes on. We call the transition from one session to the next a session handover or session handoff, the information placed directly in the new session the handover record, and the model or program that prepares the record the writer. A file or database created from the earlier session is counted separately when the new session can still read it. The handover record need not reproduce the earlier conversation, but it must retain the information on which the unfinished task still depends. A constraint may rule out an action, and examples in the earlier context may change a later prediction. Because a prose summary can omit either type of information, we judge the record by the prediction or action after the handover rather than by its similarity to the earlier wording. Information from the earlier session can reach the next model call through the prompt or through a file or archive that the agent can read, and we count every such source because each one can affect the result. A key–value (KV) cache serves a different purpose: it stores values used by attention so that the model need not repeat earlier calculations, but it does not explicitly represent the task objective, commitments, or supporting observations. Because the application usually writes the handover before it knows the next query or tool result, the record must retain the information needed across the later inputs included in the evaluation. This information order makes handover a pre-query state-coding problem, unlike queryaware prompt compression. The writer knows the task and the earlier context, but the continuation procedure observes the realized later input only after the record has been written. Classical tools from statistical sufficiency, Bayesian decision theory, comparison of experiments, and source coding can describe this order. We study which distinctions among earlier contexts remain relevant to the continuation, how a memory limit changes the attainable risk, and how pre-query encoding differs from compression performed after the later query is known.

1.1

Research Questions

We ask what the handover record must contain for the new session to preserve the prediction or decision available from the full earlier context and how much storage such a record requires. We also ask how to distinguish loss caused by missing information from loss caused by the continuation procedure that receives the record.

1.2

Contributions

We make four contributions to the analysis and construction of session handover. 2

Formulation and handover method. We model the writer, the prompt built from the handover record, the continuation procedure, and the task score separately, with the writer creating the record before the realized later input is known. This formulation distinguishes reproduction of the earlier text from preservation of the information needed for the task. Our method records decisions and constraints directly and uses a shorter representation for repeated examples or tool results only when that representation has an explicit guarantee or error bound for the task. When no such guarantee or bound is available, the method keeps selected original observations. Task-relative state and memory limits. Predictive sufficiency identifies preservation with the conditional target law. Under an exogeneity condition, its equivalence classes give the coarsest deterministic sufficient handover and a direct fixed-length bit requirement. We further identify the capacity loss forced by the memory budget, the loss attributable to the chosen writer, and the gap between the continuation procedure and the ideal decoder for the same record. Under log loss, discarded task information is exactly a conditional mutual information. A separate lower bound quantifies the cost of writing before the realized query is observed. Statistical analysis. Gaussian linear regression supplies an exact finite-dimensional sufficient record, an equivalent representation by synthetic sufficient demonstrations, and finite-bit perturbation bounds for posterior prediction. Nonparametric regression supplies an achievable memory–risk relation together with sample-size and memory lower bounds for the squaredloss problem considered below. The two models characterize when a handover can remain finite-dimensional and when its size must grow with target accuracy. Comparison of handover records. A deterministic ordering result identifies when one record contains no more information than another. When two representations determine each other, their ideal risks are equal, so any difference under one fixed continuation procedure belongs to the procedure’s gap from the ideal decoder and not to unequal task information. The memory account also includes every external record derived from the earlier session.

2

Overview of Session Handover

This section separates the boundary event from the information carried across it and from the current task state observed after resumption.

2.1

Handover and the Handover Record

Context carryover keeps earlier information available within the same session, but handover moves unfinished work to a new session or another agent. The record supplied at that boundary may be empty, incomplete, or sufficient for the continuation. An external record is a file or database produced during the earlier session and still available after resumption. Although it does not enter the first post-handover prompt automatically, it carries earlier information and counts toward total retained memory. Table 1 3

Table 1: Terms and objects used at a session boundary. Term or object

What it denotes

Use in this study

Session handover or handoff

Event in which an unfinished task moves to a new session or another agent

Boundary being studied

Handover record

Information placed directly in the new session

Record written and analyzed

Current task state

Files, data, and runtime observed after the new session begins

Information obtained after the handover

External record

File or database created from the earlier session and still available afterward

Included in retained memory when used

KV cache

Cached attention keys and values for tokens already processed

Runtime cache, not task information

distinguishes the current task state, which is observed only after resumption, from a KV cache, which stores attention state for computation and is not a task record.

2.2

Existing Approaches and the Remaining Problem

Prompt compression shortens an input in an attempt to preserve the model’s output. Because a compressor that sees the query first can remove information that the query does not use, it has an informational advantage over a handover writer, which usually writes before the downstream query or tool output is realized even though it knows the task and current goal (Nagle et al., 2024; Colaco & Lahjouji, 2026). The handover record must therefore support the set of later inputs included in the analysis. Applications that make many model calls either shorten the prompt or store earlier text and tool results outside it (Cim et al., 2026; Semenov & Dorofeev, 2026). Work on memory for agents separates writing, storage, retrieval, and later use (Wu et al., 2025; Hu et al., 2026; Shen et al., 2026). Our focus is narrower: the information a handover must preserve for a given continuation task, especially when the writer does not know the realized later input. Baptista et al. (2026) describes a trained language model through the distribution of output token sequences produced for a given prompt. We use this input–output description, place a writer before the later prompt is built, and score the resulting answer or trajectory against the task. Theory of in-context learning shows that transformers can use examples in the prompt to implement learning procedures (Bai et al., 2023; Von Oswald et al., 2023; Kim & Suzuki, 2024; Oko et al., 2024). The handover problem begins when those examples can no longer be supplied in full, and the record must retain the part that still affects the task. Appendix A compares this problem with current memory systems, prompt compression, and in-context learning.

4

3

General Formulation of Session Handover

We now define the information available before and after a handover, the prompt shown to the model, the resulting response or trajectory, and the loss used for evaluation in both text-generation and interactive tasks. Section 5 then specializes this formulation to regression.

3.1

Episode and Information Available after the Handover

We call one evaluation case an episode and represent it by ω = (C, T, X, Y ), drawn from a distribution Pep . The pre-handover variable C contains the conversation, instructions, examples, and tool results, and T contains the task information known when the writer prepares the record, including the form of the target and the scoring rule. After the boundary, the application receives X as either a query or an observation of the current task state. The target Y is the answer, final state, or allowed action used for scoring. After observing (C, T ), the handover writer E creates H = (V, M ),

(1)

where V is the record placed in the first prompt after the handover and M contains any file, index, or archive produced during the earlier session and still available afterward. Because H = (V, M ) includes every retained source that the application may use, we call the handover self-contained only when no external source is allowed and M = ∅. A randomized writer satisfies (2) H ∼ PE (· | C, T ). The size of the first prompt and the total amount stored for the handover are different quantities. We write bact (V ) ≤ Bact , btot (V, M ) ≤ Btot , (3) when both are bounded. A method may keep V short by moving earlier information into M , which may grow as the task proceeds. Any comparison of handover methods must therefore report both quantities. In a self-contained comparison, the limit applies to all information in (V, M ), not only to the first prompt.

3.2

From the Handover to a Model Response

Fix a tokenizer, a vocabulary V containing end-of-sequence and padding tokens, and a maximum response length Lout , and represent every response by a sequence Z1:Lout ∈ V Lout padded after the end-of-sequence token. The maps sfull and sH then build one prompt from the full earlier context and another from the handover: pfull = sfull (C, X, T ),

pH = sH (H, X, T ).

(4)

For a single model call, pH includes every item read from M before generation begins. If the application can query M after it begins to act, the trajectory distribution below includes those queries. Let θ denote fixed model parameters, and let δ denote a fixed decoding rule, including the temperature and any truncation of token probabilities. Together, the model and decoding 5

rule define a conditional distribution Πθ,δ (· | p) on response sequences. For an autoregressive model, the response distribution factorizes as Πθ,δ (z1:Lout | p) =

L out Y

πθ,δ,m (zm | p, z1:m−1 ).

(5)

m=1

A deterministic decoding rule assigns probability one to a single sequence, as in the standard probabilistic description of an autoregressive language model (Baptista et al., 2026). Let Zfull ∼ Πθ,δ (· | pfull ), ZH ∼ Πθ,δ (· | pH ). (6) A deterministic parser g, which may also use the task information T , extracts the answer or action evaluated by the task’s scoring rule. We call the program that applies the rule the evaluator, and g returns the sequence unchanged when the sequence itself is scored. Its expected loss under the fixed model is Rep (E; θ, δ, g) = E[ℓ(g(ZH , T ), Y )],

(7)

where the expectation is over ω ∼ Pep , the writer, and the response distribution. The reference based on the full earlier context is full Rep (θ, δ, g) = E[ℓ(g(Zfull , T ), Y )].

(8)

In an interactive task, the agent chooses an action that the application executes against the files, data, or service used by the task, collectively called the environment. After each action, the application returns an observation according to a rule Q, and τ denotes the resulting sequence of actions and observations after the handover. The writer, model, decoding rule, and environment then induce Z PE,θ,δ,Q (dτ | C, T, X) = PE (dh | C, T )Pθ,δ,Q (dτ | h, X, T ). (9) The task assigns loss ℓ(τ, Y ) to this trajectory, whose stepwise factorization appears in Appendix C. Regression in Section 5 is the one-step case, in which the action is a prediction and the environment does not change before the prediction is scored. For statements that apply across tokenizers and model families, we combine prompt construction, response generation, parsing, and environment updates into one rule D, so that (C, T ) −→ H = E(C, T ),

A = D(H, X, T ),

(10)

where A is an answer, an action, or a full trajectory.

3.3

Recovery, Preservation of the Task, and Downstream Behavior

We distinguish three questions: whether a new session can reproduce an earlier item, whether the handover preserves the distribution of the target, and whether a fixed continuation procedure succeeds when it receives that information. The first two are properties of the record and are defined below. The third is assessed from task outcomes under the continuation rule being evaluated, since a single trajectory cannot establish which information the procedure used. 6

Definition 3.1 (Exact recovery of a selected item). Let O = ϕ(C, T ) be a selected item from the information available before the handover. The item O is exactly recoverable from H when there is a measurable map r such that O = r(H, T )

(11)

holds almost surely. Exact recoverability of the full context is the special case O = C. Exact recovery is stronger than the condition required for most tasks. We therefore define preservation through the target Y , without requiring reconstruction of the complete earlier context. Definition 3.2 (Predictive sufficiency). A handover H is predictively sufficient for C relative to (X, Y, T ) when P (Y | C, X, T ) = P (Y | H, X, T ) (12) holds almost surely. Definition 3.2 gives an exact criterion for a one-step response and for a fixed replay in which all compared procedures receive the same later observations. Because an action by a freely acting agent can change the next observation, sufficiency for one target does not guarantee preservation of an adaptive trajectory. For such trajectories, Equation (9) remains the appropriate loss object. For a fixed task value t, let µt be the later-input distribution specified by task t. Two contexts c and c′ define the same in-context learning state for the task when P (Y | C = c, X = x, T = t) = P (Y | C = c′ , X = x, T = t)

(13)

holds for µt -almost every x. We write this relation as c ∼t c′ and define the equivalence class [c]∼t by later predictions rather than by a transformer hidden vector. Under an exogeneity condition, the same relation characterizes the coarsest deterministic handover that preserves those predictions. Proposition 3.1 (Coarsest deterministic sufficient state). Fix T = t, suppose that C has finite or countable support, and let H = E(C, t) be deterministic. Assume that the later input is exogenous within the task, so the conditional distribution of X given (C = c, T = t) is µt for every supported context c. Then, H is predictively sufficient if and only if E(c, t) = E(c′ , t)

=⇒

c ∼ t c′

(14)

holds for all supported c and c′ . Consequently, the map qt (c) = [c]∼t

(15)

is predictively sufficient. On the support of C | T = t, it can be recovered from every other deterministic handover that is predictively sufficient. When the number of predictive states is finite, the proposition gives a direct lower bound on the length of any fixed-length deterministic code. 7

Corollary 3.2 (Number of distinct in-context learning states). Under Proposition 3.1, suppose that qt (C) takes exactly Nt < ∞ values with positive probability. Every fixed-length deterministic sufficient handover requires at least ⌈log2 Nt ⌉ bits, and a code for the index of the equivalence class attains this length. The corollary shows why task-specific memory can be shorter than a record that permits exact reconstruction of arbitrary earlier material: a handover for one task need only distinguish contexts that lead to different distributions of Y , and it may discard distinctions needed solely to reproduce the earlier text. The proposition assumes that every supported earlier context has the same distribution of later inputs, which allows the state to be defined from the earlier context alone. Definition 3.2 itself does not impose this exogeneity condition. When the later-input law depends on the earlier context, sufficiency must be evaluated under the joint task distribution and cannot be reduced to a quotient based on one common µt . Predictive sufficiency concerns the target Y , not the wording of the earlier conversation. For one specified decision problem, a weaker requirement uses an action space A, a loss ℓ : A × Y → [0, ∞), and the following minimum expected loss for information Z: Rℓ⋆ (Z) = inf E[ℓ(d(Z, X, T ), Y )], d

(16)

where the infimum is over measurable decision rules. The handover preserves this decision problem when Rℓ⋆ (H) = Rℓ⋆ (C). Proposition 3.3 (Recovery, prediction, and a specified decision problem). Suppose that H is generated from (C, T ). Exact recoverability of C from (H, T ) implies predictive sufficiency. Predictive sufficiency implies Rℓ⋆ (H) = Rℓ⋆ (C) for any fixed action space and loss for which the risks are finite. The first implication cannot be reversed in general, and equality of risk for that decision problem does not imply predictive sufficiency. Exact recovery of one selected part of C also does not, by itself, imply predictive sufficiency. These implications also separate exact recall from task success: a record may reproduce one earlier observation yet omit information that changes Y , and a model may act incorrectly even when the record is predictively sufficient. For a fixed language model, we also measure how much its response distribution changes when the handover replaces the full context. Assume that the distribution under the handover is positive wherever the distribution under the full context is positive. We define ∆ep (E; θ, δ) = E [DKL (Πθ,δ (· | pfull ) ∥ Πθ,δ (· | pH ))] .

(17)

The expectation is over ω ∼ Pep and the writer distribution in Equation (2). The divergence compares one model under the two prompts without measuring whether either response is correct for the task, and Proposition C.1 decomposes it into expected token-level terms. Because deterministic decoding can make the divergence between distinct point-mass responses infinite, we use the model’s token probabilities when they are available and retain task loss as the main outcome.

8

3.4

Loss under a Memory Limit

To state results independently of a tokenizer or model, let D denote everything that occurs after the handover: construction of the prompt, generation and parsing of the response, and any reply from the environment. Define R(E, D) = E[ℓ(D(H, X, T ), Y )],

(18)

where the expectation is under the episode distribution and all randomness in the application. The risk with the full earlier context and the ideal risk for a fixed writer are ⋆ Rfull = inf E[ℓ(D(C, X, T ), Y )], D

⋆ RE = inf R(E, D). D

(19)

For a class EB of writers whose pair H = (V, M ) obeys budget B, define ⋆ ⋆ RB = inf RE . E∈EB

(20)

For the model and procedure used after the handover, denoted by Dθ , the excess risk decomposes as ⋆ ⋆ ⋆ ⋆ ⋆ R(E, Dθ ) − Rfull = (RB − Rfull ) + (RE − RB ) (21) ⋆ + (R(E, Dθ ) − RE ). The three terms measure, respectively, the unavoidable loss at budget B, the additional loss caused by the chosen writer, and the continuation procedure’s excess loss relative to the best decoder for the same record. They can be calculated separately only in a controlled problem with a known ideal decoder, such as the regression setting below. Without that reference, changing the writer under a fixed continuation procedure measures an end-to-end effect that combines the record’s information content with the procedure’s ability to use its presentation. Under logarithmic loss, the information removed by the handover has an exact expression. Proposition 3.4 (Log-loss identity). Suppose that H is generated from (C, T ), that the quan⋆ ⋆ tities below are finite, and that Rlog (C) and Rlog (H) denote the smallest expected logarithmic losses with the full context and the handover. Then, it holds that ⋆ ⋆ Rlog (H) − Rlog (C) = I(Y ; C | H, X, T ).

(22)

For a finite action space and a bounded loss, the same information quantity controls the decision loss. Theorem 3.5 (Decision loss after a handover). Let A be finite, let ℓ(a, Y ) ∈ [0, Lmax ], and assume the conditions of Proposition 3.4. Then, it holds that r I(Y ; C | H, X, T ) ⋆ ⋆ 0 ≤ RE − Rfull ≤ Lmax , (23) 2 where mutual information is measured in nats.

9

The proof appears in Appendix D.14. For finite targets, Proposition C.2 in Appendix C gives the corresponding exact identity for the Brier score. A limit on serialized size is not the same as a limit on information. We use Dlen (B) =

inf

I(Y ; C | E(C, T ), X, T )

(24)

I(Y ; C | H, X, T )

(25)

E: btot (E(C,T ))≤B

for a token or byte limit, and Dinfo (B) =

inf P (H|C,T ): I(C;H|T )≤B

when the limit is measured in bits or nats. The second expression has the information pattern used in source coding with side information at the decoder: the writer sees (C, T ), and the decoder additionally sees X (Wyner & Ziv, 1976; Nagle et al., 2024). Actual bytes and tokenizer tokens must be distinguished from these theoretical units whenever a serialized record is compared. The information pattern in Equation (25) differs sharply from a compressor that sees the realized later query before writing. The following example quantifies that distinction. Proposition 3.6 (Cost of writing before the query). Let C = (C1 , . . . , Cm ), where the coordinates are independent Bernoulli(1/2) variables. Let X be uniform on {1, . . . , m}, independent of C, and let Y = CX . If an encoder writes H after observing C but before observing X, and I(C; H) ≤ B bits, then the Bayes log loss in bits satisfies   B H2 (Y | H, X) ≥ max 0, 1 − . (26) m By contrast, an encoder that observes (C, X) can send CX in one bit and attain zero log loss. The example isolates the cost of committing to a boundary record before the realized query is known. A query-aware encoder can spend its bit on the requested coordinate, but the handover writer must retain information that remains useful across the query distribution. The proof appears in Appendix D. Blackwell’s comparison of information sources and the probabilistic description of autoregressive generation (Blackwell, 1953; Baptista et al., 2026) provide the starting point. Placing the writer before prompt construction and charging every retained channel yields the handover-specific risk account in Equation (21), which Section 5 computes in two regression models.

3.5

Scope at the Session Boundary

The analysis concerns a clean resumption in which the external facts that determine the target remain stable across the boundary. The files and runtime need not be byte-identical, but permitted changes preserve both the information path and the conditional target law. If the repository, data, rules, or runtime change after the record is written, the validity of earlier evidence becomes a separate state variable.

10

4

Proposed Handover Method

The proposed method copies decisions and constraints that must remain exact, uses a shorter representation only when the task supplies an explicit guarantee or error bound, and keeps original observations that cannot be replaced under that rule. Let S = ϕ(C, T ) contain the information from the earlier context needed to satisfy Definition 3.2. The task and its scoring rule determine which information belongs in S, so its definition follows the effect on the target, not a particular hidden vector. The proposed method writes a record for S within a size limit. Let Sexact contain decisions, constraints, and unresolved issues that must be recorded without changing their status. Among records that recover this part exactly, the writer seeks one with low ideal risk: ⋆ min RE E

subject to btot (H) ≤ B,

Sexact = rexact (H, T ).

(27)

The equality is exact because rewriting an adopted decision as a suggestion or treating an unresolved issue as settled can change which actions are allowed. Subject to this requirement, the writer uses the remaining space for evidence that affects later predictions and decisions. The resulting record is H = (Hexact , Hstat , Hresidual ). (28) The three-part division guides construction and allows the statistical part to be empty. When the task provides no sufficient statistic or other justified compression, the writer places the relevant observations in Hresidual .

4.1

Three Parts of the Record

The part Hexact stores the current goal, decisions that constrain the next step, rejected options, and unresolved issues together with the source of each entry, without averaging them or turning an unresolved issue into a tentative answer. The part Hstat replaces repeated examples or tool results with a shorter quantity only when the replacement comes with an explicit relation to the task loss. The replacement may be a sufficient statistic, which preserves the relevant distribution exactly, or an approximation with an explicit error bound. If neither is available, the writer keeps selected original observations instead. The part Hresidual stores original observations that Hstat does not summarize, since a rare example or specific failure may determine the next step even when an aggregate does not. To retain an effective size limit, the method must state how these observations are selected. Proposition 4.1 (A recoverable sufficient state yields a sufficient handover). Let S = ϕ(C, T ) be predictively sufficient for C relative to (X, Y, T ). If S is exactly recoverable from (H, T ), then H is predictively sufficient. The proposition connects the proposed record to the general definition: in parametric regression, S is a finite set of sufficient statistics. In a long-running task with decisions and tool results, the task specification determines which commitments and observations must remain available, after which the record can be checked for recoverability before its downstream effect is evaluated. 11

4.2

Writing the Handover

At the boundary, the writer receives the pre-boundary information and writes the record before observing the realized later input. It checks which decisions and constraints remain in force, computes any task-justified shorter quantity, selects the original observations that must remain, and divides the available space among the three parts. Deterministic checks verify the required fields, numerical dimensions, references, and total size before the record is evaluated. An application may instead maintain the same fields as the task proceeds, provided that the information available to each update and the resulting memory cost are recorded. Deterministic transformations used to construct the fields are part of the writer E, and their outputs count toward the record budget.

4.3

Presentation and Selected External Records

The map sH in Equation (4) builds the post-handover prompt through serialization, which combines the record, later input, and fixed task instructions without changing the stored values. If serialization rounds a value, omits a field, or adds a calculated value, we treat that change as part of the writing method and log it accordingly. Equation (27) concerns the information in H itself. The same information may be presented as structured fields, numerical values, or examples. The loss of a fixed continuation procedure also depends on how sH presents that information, how the response is generated, and how it is parsed. A comparison between two forms is controlled only when their information content is known to be the same or ordered. Section 5 gives a case in which two formats determine exactly the same state. Any observed difference between them can then be attributed to numerical precision, parsing, or downstream handling of the representation and not to missing information. Selected residual observations may remain outside V , the record placed in the prompt, under identifiers. Write Mres = {(j, oj ) : j ∈ Jres },

V = (Hexact , Hstat , Jres ),

(29)

where Jres is the set of identifiers placed in V . Let bext (Mres ) denote the serialized size of the external records, including identifiers and metadata, measured in the same unit as bact . For this condition, the total size is bact (V ) + bext (Mres ). Both components must be included when the record is compared with a self-contained alternative. If the external record preserves oj exactly, a valid identifier can return the same observation without rerunning the action that produced it. Exact storage alone does not establish that the writer selected every observation needed later or that the continuation procedure retrieves the relevant item. Appendix B gives further details on selection, serialization, and deterministic checks.

5

Statistical Analysis

Regression provides two settings in which the general handover problem can be solved explicitly using standard tools for Gaussian sufficiency and nonparametric minimax estimation. What is 12

specific to handover is the information order in Section 3: the writer sees the sample before the later covariate is realized and must encode all retained information within budget B. The first setting admits a finite-dimensional real-valued sufficient state and a finite-bit approximation with explicit perturbation control. The second gives an achievable memory–risk relation and a matching memory floor, up to coding logarithms, for the squared-loss problem below.

5.1

Setup for Statistical Analysis

We apply Section 3 to a regression sample Dn = ((Xi , Yi ))ni=1 available before the handover. The writer sees Dn and the task information T before the realized value of X is known and must construct H within the bit limit. After the boundary, the decoder receives the new covariate X and predicts the corresponding response Y . The parametric subsection asks when a finite-dimensional state gives the same posterior predictive distribution as the full sample and thereby supplies an exact instance of Definition 3.2. The nonparametric subsection asks how the smallest worst-case integrated squared error depends on the sample size and the bit limit, focusing on this decision problem without requiring equality of the complete conditional distribution.

5.2

Parametric Regression

We first consider linear regression with Gaussian noise because the information needed for later prediction is known exactly. Let the demonstrations before the boundary satisfy yi = x⊤ i β + εi ,

εi ∼ N (0, σ 2 ),

(30)

V0 ≻ 0.

(31)

where σ 2 is known, and let the prior be β ∼ N (m0 , V0 ),

The task information T contains (m0 , V0 , σ 2 ). Let Xn contain the demonstration inputs and yn their outputs, and define Gn = Xn⊤ Xn ,

bn = Xn⊤ yn .

(32)

The posterior covariance and mean are Vn−1 = V0−1 + σ −2 Gn ,

mn = Vn V0−1 m0 + σ −2 bn ,

(33)

and the posterior predictive distribution at a later input x is
Y | x, Xn , yn ∼ N x⊤ mn , σ 2 + x⊤ Vn x .

(34)

These expressions depend on the demonstrations only through Gn and bn . The next theorem states the resulting handover property. Theorem 5.1 (Exact handover using sufficient statistics). Under Equations (30) and (31), the record Hn = (Gn , bn ) (35) 13

is predictively sufficient for every later input x. In particular, the associated conditional mutual information vanishes: I(Y ; Xn , yn | Hn , x, T ) = 0.

(36)

The record contains d(d+1)/2+d real numbers regardless of the number of demonstrations, so it is a finite-dimensional real-valued state. Theorem 5.3 and Corollaries 5.4–5.6 treat finite-precision error and finite coding. An unknown noise variance would require additional quantities, which is why the primary theorem treats the variance as known. Many continuation procedures accept example sequences more directly than matrix fields. To compare an example-based representation with the direct statistic record, we construct synthetic input–output pairs with the same sufficient statistics. Let r = rank(Gn ), and write Gn = Ur Λr Ur⊤ ,

(37)

where Λr contains the positive eigenvalues. Define e = Λ1/2 U ⊤ , X r r

ye = Λ−1/2 Ur⊤ bn . r

(38)

e and the corresponding entries of ye are called synthetic sufficient demonstrations The rows of X in this study. Because the construction reproduces both Gn and bn , the next theorem compares two representations of the same sufficient state. Theorem 5.2 (Equivalence of synthetic sufficient demonstrations). For every Xn and yn , the construction in Equation (38) satisfies e ⊤X e = Gn , X

e ⊤ ye = bn . X

(39)

Consequently, the original demonstrations and the synthetic sufficient demonstrations give the same posterior in Equation (33). They also give the same ridge estimate for every λ > 0: e ⊤X e + λI)−1 X e ⊤ ye. (Gn + λI)−1 bn = (X

(40)

For this regression model, the two records contain the same information. When numerical precision and parsing are held fixed, any difference under the same continuation procedure arises after encoding and not from different stored information. The minimum-row property and spectral truncation bounds appear in Appendix C. The finite-bit construction below connects the stored precision to posterior perturbation, predictive log loss, and the total number of encoded scalars. Quantized sufficient statistics A finite record stores quantized approximations rather than exact real numbers. Let G ⪰ 0 and b approximate Gn and bn , and define A = V0−1 + σ −2 Gn , h = V0−1 m0 + σ −2 bn , A = V0−1 + σ −2 G,

h = V0−1 m0 + σ −2 b,

−1

and let V = A−1 , m = V h, V = A , and m = V h. 14

(41)

Theorem 5.3 (Stability under statistic quantization). Let α = λmin (V0−1 ) > 0. Suppose ∥G − Gn ∥op ≤ δG ,

∥b − bn ∥2 ≤ δb .

(42)

Then, the covariance and mean perturbations satisfy ∥V − V ∥op ≤

δG , σ 2 α2

(43)

and

δb δG ∥h∥2 + 2 2 . (44) 2 σ α σ α Consequently, for every query with ∥x∥2 ≤ Lx , the predictive mean and variance satisfy ∥m − m∥2 ≤

x⊤ (m − m) ≤ Lx ∥m − m∥2 , L 2 δG x⊤ (V − V )x ≤ 2x 2 . σ α

(45)

The theorem applies after projecting a symmetrically quantized Gram matrix onto the positive semidefinite cone. Because Gn is positive semidefinite, this projection does not increase its Frobenius distance from the truth. Corollary 5.4 (Predictive log loss under finite precision). Let (µx , vx ) and (µx , v x ) denote the posterior predictive mean and variance obtained from (Gn , bn ) and (G, b), respectively, at a query with ∥x∥2 ≤ Lx . Define   δb δG ∥h∥2 εµ = Lx + 2 2 , σ2α σ α (46) L2x δG εv = 2 2 . σ α If εv ≤ σ 2 /2, then DKL (N (µx , vx ) ∥ N (µx , v x )) ≤

ε2µ ε2v + . 2σ 2 σ 4

(47)

Corollary 5.5 (A linear handover stored with finitely many bits). Suppose ∥xi ∥2 ≤ L and |yi | ≤ By . Quantize the unique entries of Gn with step qG , quantize bn with step qb , and project the matrix to the positive semidefinite cone. The resulting record uses at most    d(d + 1) 2nL2 Blin ≤ log2 1 + 2 qG    (48) 2nLBy 2 + d log2 1 + + O(d ) qb bits, and Theorem 5.3 holds with √ δb ≤ qb d.

δG ≤ qG d,

(49)

Corollary 5.5 connects exact real-valued sufficiency to a finite code and explains why theoretical bits, serialized bytes, and tokenizer tokens should be reported separately. 15

Corollary 5.6 (Finite-bit predictive rate). Under the assumptions of Corollary 5.5, let d(d + 1) + d. (50) 2 For fixed (n, d, L, By , V0 , m0 , σ 2 , Lx ), there are constants B0 and K such that every B ≥ B0 admits a fixed-length handover of at most B bits satisfying p=

sup DKL (N (µx , vx ) ∥ N (µx , v x )) ≤ K2−2(B−B0 )/p .

(51)

∥x∥2 ≤Lx

5.3

Nonparametric Regression

A fixed-dimensional statistic cannot represent every function in a nonparametric class. We consider Yi = f (Xi ) + εi , Xi ∈ [0, 1]d , (52) where the observations are independent and identically distributed. The input density p satisfies 0 < pmin ≤ p(x) ≤ pmax < ∞, (53) the noise has conditional mean zero, and |Yi | ≤ By almost surely. For 0 < β ≤ 1, let Hβ (L, Bf ) contain functions satisfying |f (x)| ≤ Bf and |f (x) − f (x′ )| ≤ L∥x − x′ ∥β2

(54)

for every x, x′ ∈ [0, 1]d . We measure error by the integrated squared error over the distribution of later inputs: Z 2 b ∥f − f ∥ = (fb(x) − f (x))2 p(x) dx. (55) PX

d

[0,1]d

d

Partition [0, 1] into M = m equal cubes A1 , . . . , AM . For each cell, the writer stores the count and the sum of the responses, Nj =

n X

1(Xi ∈ Aj ),

Sj =

X

Yi .

(56)

i:Xi ∈Aj

i=1

For Nj > 0, let Y j = Sj /Nj , set Y j = 0 when Nj = 0, and let Qq be a quantizer with |Qq (z) − z| ≤ q. The resulting handover is
HM = (Nj , Qq (Y j )) : j = 1, . . . , M . (57) The decoder uses the stored mean for a later input in a nonempty cell and zero in an empty cell. The following theorem separates the resulting error into approximation within a cell, sampling variation, empty cells, and quantization. Theorem 5.7 (Risk of a handover based on cell statistics). Under Equations (52)–(54), suppose M ≤ n. There are constants C1 , C2 , C3 , C4 , depending only on (d, β, L, Bf , By , pmin , pmax ), such that M sup Ef [∥fbHM − f ∥2PX ] ≤ C1 M −2β/d + C2 n f ∈Hβ (L,Bf ) (58)  np  min 2 + C3 exp − + C4 q . M 16

The terms arise from approximating the function by a constant within each cell, sampling variation, empty cells, and quantization. Choosing q ≍ n−β/(2β+d)

M ≍ nd/(2β+d) ,

(59)

gives sup f ∈Hβ (L,Bf )

Ef [∥fbHM − f ∥2PX ] = O n−2β/(2β+d) ,

(60)

which matches the standard full-data order for this class. The same rate can be attained with a finite code for the stored counts and means. Corollary 5.8 (Bit budget needed to attain the full-data rate). If each count and each quantized nonempty-cell mean is stored with the fixed-length code described in Appendix F.2, then the handover uses
Bn = O nd/(2β+d) log n (61) bits and attains the rate in Equation (60). For a fixed design distribution, define the minimax risk under a B-bit handover by Rn,B =

inf

sup

PH|Dn ,D: H∈ZB , |ZB |≤2B f ∈Hβ (L,Bf )

Ef [∥fbH,D − f ∥2PX ],

(62)

where Dn denotes the n demonstrations, the decoder observes the message and future query, and the encoder and decoder may be randomized. Corollary 5.9 (Upper bound under a memory limit). There is a constant ccode > 0, depending only on the fixed coding convention and the class parameters, such that the encoder can use any partition with    ccode B M ≤ MB = min n, (63) log2 (n + 1) and store the counts and means with q 2 ≲ M −2β/d . For budgets with MB ≥ 1, the risk satisfies   np  M min −2β/d Rn,B ≲ inf M + + exp − , (64) 1≤M ≤MB n M where the infimum ranges over partitions of the form M = md . The encoder need not spend the entire budget on a finer partition. When MB is smaller than the unconstrained choice in Equation (59), the largest feasible partition gives the predicted rate, up to coding logarithms, in the range where the memory budget is the binding constraint. Once the budget can encode that unconstrained choice, the encoder keeps the same partition and remains at the sample floor instead of increasing variance by using every available bit. To obtain a matching lower bound, specialize Equation (62) to uniform Xi on [0, 1]d and a bounded binary-response submodel. Conditional on Xi = x, let Yi ∈ {−By , By } satisfy   1 f (x) Pr(Yi = By | Xi = x) = 1+ , (65) 2 By 17

where the packing amplitude is chosen so that |f (x)| ≤ By /2. This choice gives E[Yi | Xi = x] = f (x), and the submodel obeys the bounded-outcome assumptions used in the upper bound. Theorem 5.10 (Lower bounds from the sample size and memory limit). There is a constant c > 0, depending only on (d, β, L, Bf , By ), such that  Rn,B ≥ c max n−2β/(2β+d) , (B + 1)−2β/d . (66) The first term is the error that remains even when the decoder receives all observations, and the second is caused by the finite number of possible handovers. Combined with the upper bound, the theorem shows that the required memory depends on the chosen error level and the function class, not only on the number of observations. Appendix C states extensions to smoother functions and to functions known to depend on fewer coordinates.

6

Implications for Evaluating Handover Records

Comparing arbitrary summaries does not identify the theoretical quantities defined above. A controlled study must specify what the writer knows at the boundary, the distribution of later inputs, every channel through which earlier information remains available, the serialization shown to the continuation procedure, and the task loss.

6.1

Ideal-Decoder Calibration

A comparison of writers holds the continuation rule, later inputs, and loss fixed. Its observed effect is generally end-to-end because it reflects both the information carried by each record and the rule’s ability to interpret the resulting representation. The three terms in Equation (21) can be separated only when the ideal risk of each record is known. In the Gaussian setting of Section 5.2, the exact posterior calculation supplies that reference, so information loss can be separated from numerical or algorithmic error introduced after the boundary. The full earlier context and a record with no earlier information are reference conditions, not universal upper and lower bounds for an arbitrary fixed model. A task may contain predictive information in T or X. A long input may also make a particular continuation procedure less reliable. The theoretical comparison therefore uses excess risk relative to the ideal decoder under the information available in each condition, not an assumed ordering of prompt formats.

6.2

Information-Equivalent Records

For a handover record H, define R⋆ (H) = inf E[ℓ(D(H, X, T ), Y )], D

(67)

where the infimum is over continuation rules with the output space and loss fixed in the comparison. Presentation comparisons are controlled only when the information relation between their records is known. 18

Proposition 6.1 (Ideal-risk ordering under deterministic representations). Let H1 and H2 be evaluated with the same (X, Y, T ), output space, and loss. If H2 = s(H1 , T ) almost surely for a deterministic map s, then R⋆ (H1 ) ≤ R⋆ (H2 ). (68) If there is also a deterministic map r such that H1 = r(H2 , T ) almost surely, then the two ideal risks are equal. When the deterministic maps exist in both directions, a fixed continuation rule can still incur different losses because its gap from the common ideal risk may differ across the two representations. Corollary 6.2 (Representation-gap identity). Suppose the two deterministic maps in Proposition 6.1 exist. For a fixed continuation rule D, write R(D; Hj ) = E[ℓ(D(Hj , X, T ), Y )]

(69)

for j ∈ {1, 2}. Then, we have R(D; H1 ) − R(D; H2 ) = (R(D; H1 ) − R⋆ (H1 )) − (R(D; H2 ) − R⋆ (H2 )) .

(70)

The identity follows by subtracting the equal ideal risks. It rules out unequal task information as the source of the observed difference but does not identify why the fixed continuation procedure has a different gap from the ideal decoder across the two representations. In the Gaussian setting, the sufficient-statistic record and the exact synthetic sufficient demonstrations both determine (Gn , bn ). Theorem 5.2 and Proposition 6.1 therefore give them the same ideal risk. A truncated or otherwise lossy representation must instead be compared with the exact state, because its risk combines information loss with any additional decoder gap.

6.3

Memory Accounting

A short prompt does not imply a small handover when earlier information remains in files, indexes, or databases. A controlled comparison should report the active record, external storage, and content returned to the continuation procedure separately, then use their total when comparing self-contained methods. Serialized bytes and tokenizer tokens are also distinct from the information budget in Equation (25). A memory comparison should state the task distribution, the unit in which the budget is measured, and every retained channel. When several budgets are considered, the object of interest is the resulting risk curve rather than one arbitrary operating point.

7

Discussion

The formulation, handover construction, and statistical results separate three properties that are often combined in a single summary. The record may omit task-relevant information, the memory limit may make some loss unavoidable, and the continuation procedure may fail to attain the risk permitted by the information it receives. 19

7.1

What Makes a Handover Better

The proposed method records what the unfinished task still requires, with decisions and constraints written directly because changing or omitting one can change the next permitted action. Repeated observations are shortened only when the task supplies a theorem or an approximation bound that justifies the replacement. An exceptional observation remains in the record when it determines the next step. The task and its scoring rule therefore determine what the writer may shorten. Memory size cannot be interpreted independently of the continuation task. The Gaussian example has a finite-dimensional sufficient state whose dimension is independent of the number of demonstrations. The finite-precision results connect code length to predictive log loss. In the nonparametric example, the record must grow with the target accuracy until the sample-size floor is reached. Proposition 3.6 identifies another source of cost: a writer that commits before the realized query may need to retain information about several possible downstream uses that a query-aware encoder could discard. The representation also matters after the state has been selected. Proposition 6.1 identifies when two records have the same ideal risk, and Corollary 6.2 assigns any remaining difference under one fixed continuation procedure to its gap from the ideal decoder. This separation prevents a failure to parse or use a representation from being described as information loss by the writer.

7.2

Relation to Current Memory Systems

Compaction shortens the prompt, persistent memory inserts selected records into later sessions, and retrieval returns content from an external source. A KV cache serves a different role by reducing repeated computation without recording the task. These mechanisms can help an application continue, but they do not determine what the next decision requires. External records change the meaning of a memory limit because a short prompt does not limit the amount stored in files or databases. The formulation therefore counts prompt content, external storage, and returned content as parts of the handover whenever they derive from the earlier session.

7.3

Scope of the Theory

The general results apply to the joint distribution of earlier context, later input, target, and task information. The regression models furnish solvable instances in which the relevant state and the ideal decoder can be characterized, yielding exact and approximate memory requirements for those tasks. Predictive sufficiency is relative to the later-input law. A change in that law can alter which information is sufficient, and an adaptive trajectory requires a trajectory-level loss because earlier actions change later observations. The results here concern the session boundary under the continuation law.

20

8

Conclusion

A session handover succeeds when the information passed to the new session supports the continuation required by the task. By modeling the writer, the prompt built from the record, the continuation procedure, and the task score separately, we distinguish reproduction of earlier text from preservation of task information and separate information omitted by the writer from loss introduced after the record has been supplied. The proposed method records decisions and constraints directly, shortens repeated evidence only when the relation between the replacement and task loss is explicit, and keeps observations that cannot be replaced. Predictive equivalence characterizes the coarsest deterministic sufficient state under the exogeneity condition of Proposition 3.1, and the memory analysis shows how the budget and the timing of the writer constrain attainable risk. Gaussian regression gives an exact finite-dimensional handover and finite-bit predictive bounds, and nonparametric regression gives an achievable memory–risk relation and a distinct memory floor. These results establish a theoretical basis for constructing and comparing handover records across session boundaries.

References Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: provable in-context learning with in-context algorithm selection. In International Conference on Neural Information Processing Systems (NeurIPS), 2023. 4, 24 Ricardo Baptista, Andrew Stuart, and Son Tran. Large language models: A mathematical formulation, 2026. arXiv: 2601.22170. 4, 6, 10 David Blackwell. Equivalent comparisons of experiments. The Annals of Mathematical Statistics, 24(2):265–272, 1953. 10, 23 Musa Cim, Burak Topcu, Chita Das, and Mahmut Taylan Kandemir. Parallel context compaction for long-horizon llm agent serving, 2026. arXiv: 2605.23296. 4, 23 Ashwin Gerard Colaco and Nada Lahjouji. What to keep, what to forget: A rate–distortion view of memory compaction in llms and agents, 2026. arXiv: 2607.08032. 4, 23 Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremental multi-turn interactions. In International Conference on Learning Representations (ICLR), 2026. 4 Juno Kim and Taiji Suzuki. Transformers learn nonlinear features in context: Nonconvex mean-field dynamics on the attention landscape. In International Conference on Machine Learning (ICML), 2024. 4, 24 Juno Kim, Tai Nakamaki, and Taiji Suzuki. Transformers are minimax optimal nonparametric in-context learners. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2024. 24, 30

21

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Symposium on Operating Systems Principles, 2023. 23 Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In International Conference on Neural Information Processing Systems (NeurIPS), 2020. 23 Alliot Nagle, Adway Girish, Marco Bondaschi, Michael Gastpar, Ashok Vardhan Makkuva, and Hyeji Kim. Fundamental limits of prompt compression: A rate-distortion framework for black-box language models. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2024. 4, 10, 23 Kazusato Oko, Yujin Song, Taiji Suzuki, and Denny Wu. Pretrained transformer efficiently learns low-dimensional target functions in-context. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2024. 4, 24, 30 Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: trading more storage for less computation — a kvcache-centric architecture for serving llm chatbot. In USENIX Conference on File and Storage Technologies, 2025. 23 Andrew Semenov and Svyatoslav Dorofeev. Beyond compaction: Structured context eviction for long-horizon agents, 2026. arXiv: 2606.11213. 4, 23 Yiting Shen, Kun Li, Wei Zhou, and Songlin Hu. Mem2ActBench: A benchmark for evaluating long-term memory utilization in task-oriented autonomous agents. In Annual Meeting of the Association for Computational Linguistics (ACL), 2026. 4 Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000. arXiv: physics/0004057. 23 Alexandre B. Tsybakov. Introduction to Nonparametric Estimation. Springer Publishing Company, Incorporated, 1st edition, 2008. 42 Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning (ICML), 2023. 4, 24 J. Wolf and J. Ziv. Transmission of noisy information to a noisy receiver with minimum distortion. IEEE Transactions on Information Theory, 16(4):406–411, 1970. 23 Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Longmemeval: Benchmarking chat assistants on long-term interactive memory. In International Conference on Learning Representations (ICLR), 2025. 4 A. Wyner and J. Ziv. The rate-distortion function for source coding with side information at the decoder. IEEE Transactions on Information Theory, 22(1):1–10, 1976. 10, 23 22

A

Existing Memory Mechanisms and Related Work

This appendix places session handover among prompt compression, persistent memory, external records, runtime caches, and in-context learning theory. The distinctions concern both when information is written and which task-relative property it is meant to preserve.

A.1

Boundary Events and Storage Channels

Context carryover asks whether information from an earlier turn remains available later. A handover occurs when a new session or another agent continues an unfinished task, and the handover record is the information selected for that boundary. Context management determines what enters one model call, while memory management also includes writing, storing, updating, and retrieving information across calls. Earlier information can remain in the active prompt, a compacted replacement, persistent instructions, an external file or database, or a retrieval index. Retrieval-augmented generation places selected external records back into the model input (Lewis et al., 2020). These mechanisms can carry a handover, but none determines by itself whether the retained content is sufficient for the continuation task. A KV cache belongs to model execution, not to the task record, and stores attention keys and values so that earlier token states need not be recomputed. PagedAttention and distributed KV-cache systems reduce serving cost and data movement (Kwon et al., 2023; Qin et al., 2025), but cached activations do not form an auditable account of the task state. We therefore separate computational cache cost from the information retained in H.

A.2

Prompt Compression and Source Coding

Prompt compression shortens an input to preserve a model output. Rate–distortion formulations explain how a representation limit constrains the error after compression and why an encoder that observes the realized query can have an advantage (Nagle et al., 2024; Colaco & Lahjouji, 2026). A handover writer has a different information pattern: it knows the task and earlier context when it writes, but the continuation procedure observes the realized later input afterward. The information bottleneck seeks a compressed representation that retains information about a relevant variable (Tishby et al., 2000). Indirect source coding studies an encoder that observes a proxy for the target, Wyner–Ziv coding studies side information at the decoder, and Blackwell’s comparison of experiments orders information sources by the decisions they support (Wolf & Ziv, 1970; Wyner & Ziv, 1976; Blackwell, 1953). The formulation in Section 3 uses these classical tools to define the state needed for one continuation task and to place the writer before the realized later query. Repeated compaction and structured eviction address related operational problems. Parallel Context Compaction studies repeated summarization, while Structured Context Eviction uses typed records and dependencies to determine what may be removed (Cim et al., 2026; Semenov & Dorofeev, 2026). The present analysis does not prescribe a particular compaction algorithm but supplies a task-relative criterion against which a resulting record can be judged. 23

A.3

In-Context Learning and Sufficient State

Theory of in-context learning shows that transformers can implement regression and other learning procedures from examples in the prompt (Bai et al., 2023; Von Oswald et al., 2023). Other work derives prediction-error rates for broader function classes and studies how pretrained transformers exploit low-dimensional task structure (Kim & Suzuki, 2024; Kim et al., 2024; Oko et al., 2024). Our starting point is the induced prediction rule after the demonstrations have been processed, and a handover must retain the task-relevant part of that state across the session boundary. The Gaussian sufficient statistics and synthetic sufficient demonstrations in Section 5.2 make the distinction explicit because they determine the same posterior distribution and are therefore equivalent for an ideal decoder even though their token-level forms differ. This use of statistical sufficiency complements work that studies which algorithm a transformer can implement from raw examples.

B

Additional Details for the Handover Method

This appendix gives additional details for the writer, explaining how selected observations may remain outside the prompt and how comparisons keep the information content of different presentations fixed.

B.1

Selected Observations Stored outside the Prompt

Equation (29) places selected original observations in external records and leaves their identifiers in the prompt. Because both parts count toward the handover, the size limit includes the identifiers, metadata, stored observations, and any content returned to the model. The proposed method does not retain the entire earlier history outside the prompt. It stores only observations whose effect on the task is not preserved by the shorter record, and the selection rule and resulting storage cost are part of the writer.

B.2

Serialization and Deterministic Checks

The same state can be presented as concise prose, structured fields, numerical statistics, or synthetic sufficient demonstrations, and two forms have the same information content when each determines that state. When the forms do not determine each other, we order them only if one can be calculated deterministically from the other. Proposition 6.1 and Corollary 6.2 are stated in Section 6. In the Gaussian setting, direct statistics and exact synthetic sufficient demonstrations determine the same (Gn , bn ), so they have the same ideal risk. More generally, deterministic processing cannot improve the best attainable risk, and two forms that determine each other remain equivalent at the ideal-decoder level. The writer first records the events that affect later decisions. It then calculates any statistic required by the task, selects the original examples that must remain, and divides the available space among the fields. Before the record is used, deterministic checks verify

24

its schema, numerical dimensions, reference targets, and size, with the log attributing these checks to the writer. For a long-running task, the application can update exact fields when a constraint is added or a decision is made, which avoids asking the model to reconstruct every commitment only when the context is almost full. The theory also covers a writer that constructs the record only at the boundary.

C

Additional Results for the General Formulation and Statistical Analysis

This appendix collects supporting results for Sections 3 and 5.

C.1

Response Distributions under the Two Prompts

For the formulation with one model call in Section 3, fix the two prompts in Equation (4). Let P = Πθ,δ (· | pfull ), Q = Πθ,δ (· | pH ) (71) be distributions on padded response sequences of length Lout . Proposition C.1 (Autoregressive decomposition of response divergence). Suppose that P and Q admit autoregressive factorizations, that P is absolutely continuous with respect to Q, and that Pm (· | Z1:m−1 ) and Qm (· | Z1:m−1 ) denote their next-token conditional distributions. The chain rule then gives DKL (P ∥ Q) =

Lout X

EZ1:m−1 ∼P [DKL (Pm (· | Z1:m−1 ) ∥ Qm (· | Z1:m−1 ))] .

(72)

m=1

The proposition applies the standard chain rule for relative entropy to two prompts of the same model. The resulting decomposition shows at which token positions the two generations begin to differ, although agreement with the model given the full context does not establish that either response is correct for the task. For an interactive agent, let O0 = X, let Ak be the action at step k, and let Ok+1 be the next environment observation. With τ = (O0 , A0 , O1 , . . . , AK−1 , OK ), an explicit version of Equation (9) is PE,θ,δ,Q (dτ | C, T, X) Z K−1 Y = PE (dh | C, T ) πθ,δ (dAk | h, X, T, τ<k )Q(dOk+1 | X, T, τ≤k ).

(73)

k=0

The environment rule Q may be deterministic, and the factorization states where the handover enters the trajectory without assuming that the actions leave later observations fixed.

25

C.2

Sequential Prediction and Changes in the Distribution of Later Inputs

For a finite target space, a bounded proper score gives another exact measure of the posterior information lost at the boundary. Proposition C.2 (Posterior distortion under the Brier score). Let Y = {1, . . . , K}, and let a prediction be a probability vector q on Y. Define the Brier loss by X ℓBr (q, Y ) = (qy − 1{Y = y})2 . (74) y∈Y

Under Proposition 3.4, let pC = P (Y = · | C, X, T ),

pH = P (Y = · | H, X, T ).

(75)

Then, the Bayes risks satisfy   ⋆ ⋆ RBr (H) − RBr (C) = E ∥pC − pH ∥22 .

(76)

Consequently, the Brier-risk increase is zero if and only if the handover is predictively sufficient for the finite target. A freely acting trajectory requires more than sufficiency for one prediction because each action can change what the model observes next. A fixed replay avoids this problem and retains a sequence of decisions. Theorem C.3 (Log-loss decomposition over a fixed horizon). Let K ≥ 1, let H be generated from (C, T ), and suppose a fixed evaluation law generates (X1:K , Y1:K ) without depending on predictions made by the compared decoders. Assume that H and (X1:K , Y1:K ) are conditionally independent given (C, T ). At step k, the handover predictor observes (H, T, X1:k , Y<k ), and the predictor given the full context observes (C, T, X1:k , Y<k ). Let their cumulative Bayes log risks be the sums of the corresponding conditional entropies. Then, the risk difference decomposes as ⋆ ⋆ Rlog,K (H) − Rlog,K (C) =

K X

I (Yk ; C | H, T, X1:k , Y<k ) .

(77)

k=1

Theorem C.3 applies to logged replay, prediction at fixed checkpoints, and other protocols that give every condition the same sequence after the handover. Trajectories in which different policies choose different actions and receive different later observations fall outside this result. An application may rewrite its compacted record several times. Theorem C.4 separates repeated rewriting from the case in which a later session receives new evidence about the task. Theorem C.4 (Repeated handover without new evidence). Let H0 = C. For j = 1, . . . , m, suppose Hj is generated from Hj−1 and T , and suppose that, conditional on T , (C, X, Y ) −→ Hj−1 −→ Hj 26

(78)

holds as a Markov relation. If the Bayes log risks below are finite, then the following identity holds: m X ⋆ ⋆ Rlog (Hm ) − Rlog (C) = I(Y ; Hj−1 | Hj , X, T ). (79) j=1

Consequently, the ideal log risk is nondecreasing across the rewrites. Theorem C.4 applies when each stage only rewrites the state it received. A later stage can improve prediction when it obtains new current-state observations or tool results, because those observations add information that was unavailable to the earlier record. Predictive sufficiency depends on the distribution of later queries. The next result considers a change in that distribution when the new queries reveal no additional information about the context available before the handover. Proposition C.5 (A shift in the distribution of exogenous queries). Fix T = t, and suppose X is independent of (C, H) under both a reference query distribution µt and an evaluation query distribution νt . Hold fixed the joint law of (C, H) and the conditional target kernel P (Y | C, X, T = t). Let ∆µt (H) and ∆νt (H) be the Bayes log-risk increases caused by using H instead of C under the two query distributions. If νt is absolutely continuous with respect to µt and dνt ess sup (x) ≤ κ, (80) dµt x then the evaluation-distribution increase satisfies 0 ≤ ∆νt (H) ≤ κ∆µt (H).

(81)

The coverage condition is essential: if νt assigns mass to queries outside the support of µt , a record can be sufficient under µt and lose arbitrary target information on the new queries. The bound therefore applies only when the later-input distribution used for evaluation satisfies the coverage condition in Equation (80). Equation (23) applies to a particular task, so a record can be sufficient for choosing the next admissible action yet remain insufficient for reconstructing a detailed narrative. Its proof appears in Appendix D.

C.3

Additional Information-Budget Results

When external records are available, we write all information passed through the handover as the pair H = (V, M ), len(V ) ≤ Bact , (82) where V is the record placed in the prompt and M contains information retained outside it. The limit Bact controls V . The information constraint in Equation (25) applies to the full pair (V, M ). Any application of the formulation must account separately for bytes stored outside the prompt, access requests, and content returned to the continuation procedure.

27

Proposition C.6 (Structure of the information-budget curve). Assume that the information quantities below are finite and measured in the same units. Then, Dinfo (B) is nonincreasing and convex in B, and it satisfies 0 ≤ Dinfo (B) ≤ I(Y ; C | X, T ).

(83)

At zero budget, Dinfo (0) = I(Y ; C | X, T ). If C is discrete and B ≥ H(C | T ), then Dinfo (B) = 0. If the future query is exogenous, so X and C are conditionally independent given T , then it also holds that Dinfo (B) ≥ max {0, I(Y ; C | X, T ) − B} .

(84)

Proposition C.6 gives a one-episode trade-off between stored information and discarded target information. Operational rates for repeated episodes depend on the chosen block-coding model and code construction. In the main setting, the writer knows T but does not observe the realized X. A generic compressor may ignore task information at the boundary, whereas a query-aware compressor sees the specific downstream query before choosing what to retain. Handover lies between these cases because the writer knows the task and local goal but commits before the exact later query is known. An encoder that sees X can discard information unused by the realized query, but the handover writer must account for the distribution of later queries fixed by the task. Two further bounds clarify what a finite channel can and cannot preserve. The first gives an explicit code when the sufficient state has continuous and discrete parts. Proposition C.7 (Description length of a mixed sufficient QJ state). Suppose a predictively d sufficient state is S = (θ, m), where θ ∈ [0, 1] and m ∈ j=1 Aj with finite Aj . Assume LS > 0 and that replacing θ by θ′ with m fixed increases the ideal decoder’s risk by at most LS ∥θ − θ′ ∥2 . For every ε > 0, there is a fixed-length handover whose excess risk relative to the exact state is at most ε and whose length satisfies & √ !' J X LS d Bmix (ε) ≤ d log2 1 + + ⌈log2 |Aj |⌉ . (85) ε j=1 Under this assumption, the discrete part must be represented exactly and may encode a finite decision status, a task mode, or another state component whose alternatives cannot be averaged without changing the decision problem. The continuous term depends on target accuracy rather than on transcript length. The next result gives a regret lower bound when distinct histories require separated continuation policies. Theorem C.8 (Lower bound from separated predictive states). Let J be uniform on {1, . . . , N }, with N ≥ 2, and let J index histories available before the handover under a fixed task value t. The encoder writes H before the future observation X is drawn, with X independent of (J, H) given T = t. A continuation policy π maps a realized X to an action

28

distribution, with Lj (π) denoting its expected loss in instance j and L⋆j = inf π Lj (π) denoting the optimum. Suppose that, for some δ > 0, every policy satisfies  # j : Lj (π) < L⋆j + δ ≤ 1. (86) For every decoder D, let the policy induced by message h be the conditional action distribution πh (· | x) = PD (A ∈ · | h, x, t), which integrates over any private randomization. If the handover satisfies I(J; H | T = t) ≤ B nats, then every writer and decoder satisfies   B + log 2 . (87) E[Regret] ≥ δ max 0, 1 − log N The packing condition states the decision separation needed for an information lower bound and can be enforced in a synthetic continuation family by assigning mutually incompatible near-optimal policies to the histories. Additional observations may reveal the history and violate the condition. The theorem therefore applies to task families that satisfy the separation condition.

C.4

Additional Results for Parametric Regression

Corollary C.9 (Minimum number of exact synthetic sufficient demonstrations). Let Z ∈ Rs×d be the input matrix of any synthetic sufficient demonstration set that satisfies Z ⊤ Z = Gn . Then, the row count satisfies s ≥ rank(Gn ). (88) Consequently, the construction in Equation (38) uses the minimum possible number of realvalued synthetic sufficient demonstrations among exact Gram-matrix representations. The theorem and corollary fix the information content and the number of examples in two presentation formats: a matrix and an exact synthetic sufficient demonstration set contain the same sufficient state. Subject to numerical precision and prompt parsing, any difference when the same fixed model receives the two forms therefore arises downstream of the stored information. Small eigenvalues and truncated synthetic sufficient demonstrations Because the exact construction can produce large synthetic outputs when Gn has very small positive eigenvalues, consider a truncated construction for which Uτ contains the eigenvectors whose eigenvalues exceed τ > 0 and the retained statistics are Gn,τ = Uτ Λτ Uτ⊤ ,

bn,τ = Uτ Uτ⊤ bn .

(89)

The corresponding synthetic sufficient demonstrations use Equation (38) with (Uτ , Λτ , bn,τ ). Proposition C.10 (Ridge error after spectral truncation). Let βbλ = (Gn + λI)−1 bn ,

βbλ,τ = (Gn,τ + λI)−1 bn,τ .

Then, for every λ > 0, the truncation error satisfies √ τ b b ∥βλ − βλ,τ ∥2 ≤ ∥yn ∥2 . λ 29

(90)

(91)

The bound shows the trade-off between exact preservation and the range of the stored numbers. A truncated record must therefore state the threshold and the information discarded by the truncation, and it cannot be described as exact.

C.5

Higher Smoothness and Intrinsic Dimension

For β > 1, a piecewise constant handover does not attain the full Hölder rate, so the encoder must store local polynomial moments through order p = ⌊β⌋. Each cell contains   d+p Kd,p = (92) d coefficients. The primary theorem covers 0 < β ≤ 1, but the local-polynomial extension requires a separate proof. The ambient dimension can also overstate memory when the task distribution lies on a known lower-dimensional subspace. In that case, the cell construction can use coordinates on that subspace and replace d by its dimension. Existing ICL theory shows that pretrained transformers can exploit nonparametric structure and low-dimensional task families under suitable training distributions (Kim et al., 2024; Oko et al., 2024). The rate derived here applies to the regression setting above when the relevant coordinates are known to the encoder.

D

Proofs for General Handover Results

This appendix proves the general results in the order in which they are used. Each subsection proves the result named in its heading and uses only the assumptions in that result.

D.1

Proof of Proposition C.1

The autoregressive factorizations give log

Lout Pm (Zm | Z1:m−1 ) P (Z1:Lout ) X log . = Q(Z1:Lout ) m=1 Qm (Zm | Z1:m−1 )

(93)

Taking expectation under P and applying the tower property to each summand gives DKL (P ∥ Q) = =

Lout X m=1 Lout X



 EZ1:m−1 ∼P EZm ∼Pm (·|Z1:m−1 )

Pm (Zm | Z1:m−1 ) log Qm (Zm | Z1:m−1 )

 (94)

EZ1:m−1 ∼P [DKL (Pm (· | Z1:m−1 ) ∥ Qm (· | Z1:m−1 ))] ,

m=1

which proves Equation (72).

30

D.2

Proof of Proposition 3.1

Suppose first that H is predictively sufficient. If two supported contexts c and c′ satisfy E(c, t) = E(c′ , t) = h, exogeneity gives both contexts the same future-query law µt . Predictive sufficiency therefore gives P (Y | C = c, X = x, T = t) = P (Y | H = h, X = x, T = t)

(95)

for µt -almost every x, and the same equality holds with c′ in place of c. Intersecting the two full-measure sets shows that c ∼t c′ . Conversely, suppose every encoder fiber lies in one equivalence class. Because X is independent of C given T = t, conditioning on X does not change the mixture weights over contexts within an encoder fiber. Conditional on (H = h, X = x, T = t), the target distribution is a mixture over the supported contexts in that fiber. For µt -almost every x, every distribution in the mixture is the same by assumption, so the mixture equals that common distribution. Therefore, Equation (12) holds. The quotient map groups exactly the contexts in one equivalence class, so the preceding argument proves its sufficiency. If E is any deterministic sufficient encoder, define ρt (h) as the equivalence class containing the fiber associated with h. The first part shows that this class is well defined, with qt (c) = ρt (E(c, t)) on the support. Thus, the quotient record can be recovered from every deterministic sufficient handover.

D.3

Proof of Corollary 3.2

By Proposition 3.1, a deterministic sufficient handover cannot assign the same codeword to two distinct values of qt (C), and a fixed-length b-bit record has at most 2b codewords, so 2b ≥ Nt and hence b ≥ ⌈log2 Nt ⌉. Conversely, indexing the Nt equivalence classes uses ⌈log2 Nt ⌉ bits and is predictively sufficient.

D.4

Proof of Proposition 3.3

The encoder in Equation (2) is conditionally independent of (X, Y ) given (C, T ). This conditional independence gives P (Y | C, H, X, T ) = P (Y | C, X, T ).

(96)

If C = r(H, T ) holds almost surely, then conditioning on (H, X, T ) also determines C. Consequently, the conditional distributions satisfy P (Y | H, X, T ) = P (Y | C, H, X, T ),

(97)

which proves predictive sufficiency. Suppose next that predictive sufficiency holds. The minimum conditional risk at any realized information set is the Bayes envelope of the conditional distribution of Y . Equation (12) makes the pointwise Bayes envelopes with (C, X, T ) and (H, X, T ) equal almost surely, so taking expectations proves Rℓ⋆ (H) = Rℓ⋆ (C). 31

For a counterexample to the first converse, let U and V be independent Bernoulli(1/2) variables, and set C = (U, V ), H = U , and Y = U , with X and T constant. The target is determined by H, so predictive sufficiency holds, but V cannot be recovered from H. Exact recovery of one part is also insufficient: if O = V and H = V , then O is recoverable but Y = U is not predictable from the record. For a decision problem whose Bayes risk is preserved without predictive sufficiency, let C be equiprobable on {0, 1}, let H be constant, and let Y ∈ {0, 1} satisfy Pr(Y = 1 | C = 0) = 0.8 and Pr(Y = 1 | C = 1) = 0.9. Under zero–one loss, the Bayes action is 1 for both values of C, and the average Bayes risk is 0.15 with either C or H. The conditional target distributions nevertheless differ, so predictive sufficiency fails.

D.5

Proof of Proposition 4.1

Predictive sufficiency of S = ϕ(C, T ) gives P (Y | C, X, T ) = P (Y | S, X, T ).

(98)

Exact recoverability gives S = r(H, T ) almost surely, so conditioning on (H, X, T ) determines S. Because H is generated from (C, T ), conditioning on H only changes the mixture over contexts that share this recovered value of S. Equation (98) gives the same target distribution for every context in that mixture. The resulting mixture therefore satisfies P (Y | H, X, T ) = P (Y | S, X, T ).

(99)

Combining the two equalities proves Equation (12).

D.6

Proof of Proposition 3.4

The Bayes log risk with information Z is the conditional entropy of Y given Z. Therefore, we have ⋆ (H) = H(Y | H, X, T ) (100) Rlog and ⋆ Rlog (C) = H(Y | C, X, T ).

(101)

Because H is generated from (C, T ), the conditional entropy satisfies H(Y | C, H, X, T ) = H(Y | C, X, T ).

(102)

The definition of conditional mutual information now gives I(Y ; C | H, X, T ) = H(Y | H, X, T ) − H(Y | C, H, X, T ) ⋆ ⋆ = Rlog (H) − Rlog (C), which proves Equation (22).

32

(103)

D.7

Proof of Proposition C.2

For a probability vector p on Y, the conditional Brier risk of a prediction q is X X E[ℓBr (q, Y ) | p] = qy2 − 2 qy py + 1. y∈Y

(104)

y∈Y

The expression is minimized at q = p, where its value is 1 − ∥p∥22 . At this minimizer, ⋆ ⋆ RBr (H) − RBr (C) = E[∥pC ∥22 ] − E[∥pH ∥22 ].

(105)

The encoder Markov condition gives pH = E[pC | H, X, T ]. The conditional Pythagorean identity then yields E[∥pC ∥22 ] = E[∥pH ∥22 ] + E[∥pC − pH ∥22 ]. (106) Combining Equations (105) and (106) proves Equation (76). The right-hand side is zero exactly when pC = pH almost surely, which is predictive sufficiency for a finite target space.

D.8

Proof of Theorem C.3

Let Zk = (T, X1:k , Y<k ). The cumulative Bayes log risks are ⋆ Rlog,K (H) =

K X

⋆ Rlog,K (C) =

H(Yk | H, Zk ),

k=1

K X

H(Yk | C, Zk ).

(107)

k=1

The conditional independence assumption implies H(Yk | C, H, Zk ) = H(Yk | C, Zk ).

(108)

Consequently, each summand satisfies H(Yk | H, Zk ) − H(Yk | C, Zk ) = H(Yk | H, Zk ) − H(Yk | C, H, Zk ) = I(Yk ; C | H, Zk ).

(109)

Summing Equation (109) over k proves Equation (77).

D.9

Proof of Theorem C.4

For every j, the Markov relation in Equation (78) implies H(Y | Hj−1 , Hj , X, T ) = H(Y | Hj−1 , X, T ).

(110)

Subtracting the two equalities gives ⋆ ⋆ Rlog (Hj ) − Rlog (Hj−1 ) = H(Y | Hj , X, T ) − H(Y | Hj−1 , X, T )

= I(Y ; Hj−1 | Hj , X, T ).

(111)

Summing Equation (111) from j = 1 to m telescopes to Equation (79). Every conditional mutual information in the sum is nonnegative, which proves monotonicity. 33

D.10

Proof of Proposition C.6

Monotonicity follows because every channel feasible at budget B1 is feasible at every B2 ≥ B1 . To prove convexity, choose two channels P1 (H1 | C, T ) and P2 (H2 | C, T ), and let Q be an independent Bernoulli time-sharing variable with Pr(Q = 1) = λ. The combined message records Q and uses channel PQ , which gives

and

I(C; Q, HQ | T ) = λI(C; H1 | T ) + (1 − λ)I(C; H2 | T ),

(112)

I(Y ; C | Q, HQ , X, T ) = λI(Y ; C | H1 , X, T ) + (1 − λ)I(Y ; C | H2 , X, T ).

(113)

Applying the construction to channels arbitrarily close to the two infima proves convexity. Nonnegativity of conditional mutual information and a constant message, which is feasible at every nonnegative budget and gives distortion I(Y ; C | X, T ), prove Equation (83). At zero budget, I(C; H | T ) = 0. Because the message is generated from (C, T ), H is then independent of (C, X, Y ) given T , and conditioning on H does not change I(Y ; C | X, T ). It follows that Dinfo (0) = I(Y ; C | X, T ). If C is discrete and B ≥ H(C | T ), the choice H = C is feasible and gives zero distortion. For the final bound, exogeneity implies that X is independent of (C, H) given T , so I(C; H | X, T ) = I(C; H | T ).

(114)

The encoder Markov relation and the chain rule give I(Y ; C | X, T ) = I(Y ; H | X, T ) + I(Y ; C | H, X, T ).

(115)

Data processing conditional on (X, T ) yields I(Y ; H | X, T ) ≤ I(C; H | X, T ) ≤ B.

(116)

Rearranging Equation (115) and using nonnegativity proves Equation (84).

D.11

Proof of Proposition 6.1

Let D2 be any decoder that uses (H2 , X, T ). Because a decoder that receives H1 can compute H2 = s(H1 , T ) and then run D2 , every risk achievable from H2 is also achievable from H1 , which proves Equation (68). If H1 = r(H2 , T ) also holds, the same argument in the opposite direction gives equality.

D.12

Proof of Proposition 3.6

Because X is independent of (C, H) and is uniform, we have m

1 X I(Y ; H | X) = I(Cj ; H), m j=1 34

(117)

where every information quantity in this proof is measured in bits. Independence of the coordinates gives m m X X I(Cj ; H) = m − H2 (Cj | H) j=1

j=1

(118)

≤ m − H2 (C | H) = I(C; H) ≤ B. P

is a fair

H2 (Y | H, X) = 1 − I(Y ; H | X) B ≥1− . m

(119)

The inequality uses conditional subadditivity, H2 (C | H) ≤ bit conditional on X, its conditional entropy satisfies

j H2 (Cj | H). Since Y

Conditional entropy is nonnegative, which gives Equation (26). If the encoder observes X, it sends CX , allowing the decoder to recover Y exactly from one bit.

D.13

Proof of Proposition C.5

Under the exogenous-query assumption, the conditional distribution of (C, H) given T = t does not depend on the realized query. Define 
 gt (x) = E DKL PY |C,x,t ∥ PY |H,x,t T = t . (120) The function gt is nonnegative. Conditioning on the query and applying Proposition 3.4 gives Z Z ∆µt (H) = gt (x) dµt (x), ∆νt (H) = gt (x) dνt (x). (121) Absolute continuity and Equation (80) imply Z dνt ∆νt (H) = gt (x) (x) dµt (x) dµt Z ≤ κ gt (x) dµt (x)

(122)

= κ∆µt (H). Nonnegativity proves the remaining inequality in Equation (81).

D.14

Proof of Theorem 3.5

For a posterior distribution p on Y , define its Bayes envelope by V (p) = min Ep [ℓ(a, Y )]. a∈A

35

(123)

If p and q are two distributions, let aq minimize the risk under q. Since 0 ≤ ℓ ≤ Lmax , the Bayes-envelope difference satisfies V (p) − V (q) ≤ Ep [ℓ(aq , Y )] − Eq [ℓ(aq , Y )] ≤ Lmax dTV (p, q).

(124)

Interchanging p and q gives |V (p) − V (q)| ≤ Lmax dTV (p, q).

(125)

pC = PY |C,X,T ,

(126)

Let pH = PY |H,X,T .

The encoder Markov condition gives pH = E[pC | H, X, T ].

(127)

Because V is the pointwise minimum of linear functions of p, it is concave. Conditional ⋆ ⋆ Jensen’s inequality applied to Equation (127) proves RE ≥ Rfull . Equation (125) also yields ⋆ ⋆ RE − Rfull ≤ Lmax E[dTV (pC , pH )].

Pinsker’s inequality and Jensen’s inequality for the square root give "r # 1 E[dTV (pC , pH )] ≤ E DKL (pC ∥pH ) 2 r 1 E[DKL (pC ∥pH )] ≤ 2 r I(Y ; C | H, X, T ) = . 2

(128)

(129)

Combining Equations (128) and (129) proves Equation (23).

D.15

Proof of Proposition C.7

Quantize each coordinate of θ on [0, 1] with a grid whose reconstruction error is at most q=

ε √ . LS d

(130)

The grid needs at most 1 + 1/q codewords per coordinate, and the reconstructed vector θ satisfies √ ε ∥θ − θ∥2 ≤ dq = . (131) LS Encode every coordinate of m exactly. The Lipschitz assumption then bounds the risk increase by ε, and summing the fixed-length coordinate and discrete-field codes gives Equation (85). 36

D.16

Proof of Theorem C.8

For each realized message h, let πh denote the conditional action distribution induced by the decoder after integrating over its private randomization. Thus, πh maps the future observation X to an action distribution. Let Π = πH be the random policy selected by the boundary message. The policy is fixed before the realized X is observed, although its action distribution may depend on X when the policy is executed. Because Π is a deterministic function of H, data processing gives I(J; Π | T = t) ≤ I(J; H | T = t) ≤ B.

(132)

b Define J(Π) by returning the unique index for which Π is δ-near-optimal and returning an arbitrary index otherwise. Because the packing condition guarantees uniqueness, every event Jb ̸= J satisfies LJ (Π) − L⋆J ≥ δ. (133) Therefore, we have E[Regret] ≥ δ Pr(Jb ̸= J).

(134)

Fano’s inequality and Equation (132) imply Pr(Jb ̸= J) ≥ 1 −

B + log 2 . log N

(135)

A probability is nonnegative, so combining Equations (134) and (135) proves Equation (87).

E

Proofs for the Parametric Results

This appendix proves the results for linear regression with Gaussian noise. The proofs first establish exact sufficiency and then treat the alternative presentation and finite-precision variants.

E.1

Proof of Theorem 5.1

The Gaussian likelihood can be written as   1 2 p(yn | Xn , β) ∝ exp − 2 ∥yn − Xn β∥2 2σ  
1 ⊤ ⊤ ∝ exp − 2 β Gn β − 2β bn . 2σ

(136)

The omitted factor depends on yn⊤ yn but not on β. Multiplying Equation (136) by the Gaussian prior and completing the square gives Equation (33). The conditional distribution of β given (Xn , yn ) therefore depends on the demonstrations only through (Gn , bn ). For a future query x, integrating N (x⊤ β, σ 2 ) against that posterior gives the conditional distribution of Y in Equation (34), which is determined by Hn . It follows that P (Y | Xn , yn , Hn , x, T ) = P (Y | Hn , x, T ), which proves predictive sufficiency and Equation (36). 37

(137)

E.2

Proof of Theorem 5.2

Every real matrix Xn satisfies range(Xn⊤ Xn ) = range(Xn⊤ ).

(138)

Since bn = Xn⊤ yn , Equation (138) implies bn ∈ range(Gn ). Therefore, we have Ur Ur⊤ bn = bn .

(139)

1/2 ⊤ e ⊤X e = Ur Λ1/2 X r Λr Ur = Gn , −1/2 ⊤ e ⊤ ye = Ur Λ1/2 Ur bn = bn . X r Λr

(140)

Using Equation (38), we obtain

Equation (39) follows. Because Equation (33) depends on the data only through these two quantities, the Gaussian posterior and posterior predictive distributions agree, and substituting Equation (39) into the two ridge estimators proves Equation (40).

E.3

Proof of Corollary C.9

Every real matrix Z satisfies rank(Z ⊤ Z) = rank(Z) ≤ s.

(141)

If Z ⊤ Z = Gn , then s ≥ rank(Gn ). Equation (38) uses exactly r = rank(Gn ) rows, so it attains this lower bound.

E.4

Proof of Proposition C.10

Let Pτ = Uτ Uτ⊤ . A singular-value decomposition of Xn shows that √ ∥(I − Pτ )bn ∥2 ≤ τ ∥yn ∥2 .

(142)

√ If Xn = QΣU ⊤ , then bn = U ΣQ⊤ yn , and every discarded singular value is at most τ . The matrices Gn , Gn,τ , and Pτ share the same eigenvectors. The two ridge estimators have identical coefficients on the retained eigenspace, but the truncated estimator is zero on the discarded eigenspace. Their difference is therefore βbλ − βbλ,τ = (Gn + λI)−1 (I − Pτ )bn .

(143)

Because the inverse norm is at most 1/λ, taking norms and applying Equation (142) proves Equation (91).

38

E.5

Proof of Theorem 5.3

Because Gn ⪰ 0 and G ⪰ 0, the matrices A and A satisfy A ⪰ αI,

A ⪰ αI.

(144)

Hence, ∥V ∥op ≤ 1/α and ∥V ∥op ≤ 1/α. The resolvent identity gives V − V = V (A − A)V.

(145)

∥A − A∥op = σ −2 ∥Gn − G∥op ,

(146)

Since Equation (43) follows. For the posterior mean, we have m − m = V (h − h) + (V − V )h.

(147)

The first term is at most δb /(σ 2 α), and the second is at most δG ∥h∥2 /(σ 2 α2 ), which establishes Equation (44). The two inequalities in Equation (45) follow from Cauchy–Schwarz and the definition of the operator norm.

E.6

Proof of Corollary 5.4

The predictive variances satisfy vx ≥ σ 2 and v x ≥ σ 2 because both posterior covariance matrices are positive semidefinite. Theorem 5.3 gives |µx − µx | ≤ εµ ,

|vx − v x | ≤ εv .

(148)

For the divergence in the direction used in Corollary 5.4, the univariate Gaussian formula is DKL (N (µx , vx ) ∥ N (µx , v x ))   1 v x vx (µx − µx )2 = log + −1+ . 2 vx v x vx

(149)

Let t = vx /v x . Since v x ≥ σ 2 , Equation (148) gives |t − 1| = |vx − v x |/v x ≤ εv /σ 2 ≤ 1/2. On [1/2, 3/2], the function t − 1 − log t has second derivative at most four and vanishes together with its first derivative at one, so t − 1 − log t ≤ 2(t − 1)2 ≤

2ε2v . σ4

(150)

The mean term in Equation (149) is at most ε2µ /σ 2 . Multiplying the two bounds by one half proves Equation (47).

39

E.7

Proof of Corollary 5.5

Every entry of Gn lies in [−nL2 , nL2 ], and every entry of bn lies in [−nLBy , nLBy ]. A uniform scalar quantizer with step q on an interval of width 2A needs at most    2A (151) log2 1 + q bits, apart from a fixed encoding header. Storing the upper triangle of the symmetric matrix and the d entries of bn gives Equation (48). Before projection, every reconstructed matrix entry differs from the corresponding entry of Gn by at most qG , so the Frobenius norm of the full symmetric error is at most qG d and the operator norm is no larger. Projection onto the positive semidefinite cone is a metric projection in Frobenius norm and therefore cannot increase√the distance to the positive semidefinite matrix Gn . Together with the vector bound qb d, these observations prove Equation (49).

E.8

Proof of Corollary 5.6

Let m = ⌊(B − B0 )/p⌋, where B0 accounts for the fixed header, signs, dimensions, and coding conventions, and quantize every stored scalar using m bits over its bounded interval. Corollary 5.5 then gives constants cG and cb , depending only on the fixed problem parameters, such that δG ≤ cG 2−m , δb ≤ cb 2−m . (152) Equations (46) and (47) therefore give a constant K0 for which the predictive divergence is at most K0 2−2m once B0 is large enough to ensure εv ≤ σ 2 /2. Since m ≥ (B − B0 )/p − 1, increasing the constant by a factor of four proves Equation (51).

F

Proofs for the Nonparametric Results

This appendix proves the upper and lower bounds for the nonparametric regression setting. The argument first controls the cell-based record and then constructs a packing for the memory lower bound.

F.1

Proof of Theorem 5.7

Let h = M −1/d be the side length of each cell, and let √ DM = d h

(153)

be its diameter. Write pj = Pr(X ∈ Aj ) and fj = E[f (X) | X ∈ Aj ]. The density lower bound gives pj ≥ pmin /M . 40

(154)

When Nj > 0, the cell predictor is Qq (Y j ). For x ∈ Aj , the Hölder condition gives β |fj − f (x)| ≤ LDM .

(155)

in a cell is at most By2 .

Conditional on Nj = k > 0, the

The conditional variance of Y sampling error therefore satisfies

By2 . (156) k For a binomial variable N ∼ Binomial(n, p), we have 1/N ≤ 2/(N + 1) on N > 0, and   1 1 − (1 − p)n+1 1 E = ≤ . (157) N +1 (n + 1)p (n + 1)p It follows that   1(Nj > 0) 2 E ≤ . (158) Nj (n + 1)pj On a nonempty cell, use E[(Y j − fj )2 | Nj = k] ≤

Qq (Y j ) − f (x) = (Qq (Y j ) − Y j ) + (Y j − fj ) + (fj − f (x)).

(159)

The inequality (a + b + c)2 ≤ 3(a2 + b2 + c2 ), Equations (155) and (156), and the quantizer bound imply   E (Qq (Y j ) − f (x))2 1(Nj > 0)   (160) 1(Nj > 0) 2 2 2β 2 ≤ 3q + 3L DM + 3By E . Nj Integrating Equation (160) over Aj and summing over cells gives a variance contribution at most 6By2 M/(n + 1), together with the bias and quantization terms. On an empty cell, the predictor is zero and the squared error is at most Bf2 . Since Pr(Nj = 0) = (1 − pj )n ≤ exp(−npj ),

(161)

the integrated empty-cell contribution is at most M X

 np  min pj Bf2 exp(−npj ) ≤ Bf2 exp − . M j=1

(162)

2β Finally, DM = dβ M −2β/d . Combining the nonempty and empty contributions proves Equation (58) with explicit constants obtained from the preceding bounds.

F.2

Proof of Corollary 5.8

A count in {0, 1, . . . , n} uses ⌈log2 (n + 1)⌉ bits, and a quantized mean in [−By , By ] with step q has at most 1 + 2By /q levels. The total number of bits is therefore at most      2By M ⌈log2 (n + 1)⌉ + log2 1 + + c0 , (163) q where c0 accounts for fixed field delimiters and the empty-cell flag. Under Equation (59), both logarithmic terms are O(log n), and M = O(nd/(2β+d) ). These bounds establish Equation (61). Substitution into Theorem 5.7 gives the risk order in Corollary 5.8. 41

F.3

Proof of Corollary 5.9

Choosing q 2 proportional to M −2β/d ensures, because M ≤ n, that each count and quantized mean uses O(log(n + 1)) bits. A sufficiently small constant ccode then makes every admissible M = md ≤ MB feasible, and substituting into Equation (58) before minimizing over the feasible partitions proves Equation (64).

F.4

A Packing Lemma for Hölder Functions

A standard bump construction underlying Assouad and Fano lower bounds (Tsybakov, 2008) supplies constants a0 , c0 , c1 > 0 such that, for every integer m, setting M = md and h = 1/m yields a family {fθ : θ ∈ {−1, 1}M } ⊆ Hβ (L, Bf ) (164) such that, whenever θ and θ′ differ in one coordinate, ∥fθ − fθ′ ∥22 = c0 h2β+d ,

(165)

∥fθ − fθ′ ∥22 = c0 h2β+d dH (θ, θ′ ),

(166)

and, for arbitrary θ, θ′ , where dH is Hamming distance. Because the construction uses disjoint, scaled Hölder bumps with amplitude a0 hβ , choosing a0 sufficiently small keeps the entire family inside the Hölder ball.

F.5

Proof of Theorem 5.10

We first prove the sample floor. Under the binary-response submodel in Equation (65), let   1 f (x) 1+ . (167) pf (x) = 2 By The packing construction keeps pf (x) ∈ [1/4, 3/4]. The Bernoulli divergence bound DKL (Ber(p)∥ Ber(q)) ≤

(p − q)2 q(1 − q)

(168)

therefore implies that, for a constant cB depending only on By , (n)

DKL (Pf ∥Pg(n) ) ≤ cB n∥f − g∥22 .

(169)

For adjacent vertices in the family of Equation (164), Equations (165) and (169) give divergence of order nh2β+d . Choosing h proportional to n−1/(2β+d) with a sufficiently small constant bounds the adjacent divergence by a fixed number smaller than one, so Assouad’s lemma yields inf sup Ef [∥fb − f ∥22 ] ≥ c2 h2β ≥ c3 n−2β/(2β+d) . (170) fb f ∈Hβ (L,Bf )

This lower bound applies to every handover because it also applies when the decoder receives the full data. 42

We next prove the memory floor. The Varshamov–Gilbert bound supplies a subset Θ ⊆ {−1, 1}M such that log |Θ| ≥ c4 M (171) and every distinct pair in Θ has Hamming distance at least M/8. Equation (166) then gives pairwise squared separation at least c5 h2β+d M = c5 h2β .

(172)

Let J be uniform on Θ. Whatever data the encoder observes, a message in an alphabet of size at most 2B satisfies I(J; H) ≤ B log 2. (173) Any randomized decoder can be represented as D(H, U, x), where the private random seed U is independent of (J, H). Conditional on (H, U ), the map x 7→ D(H, U, x) defines a reconstructed function, and I(J; H, U ) = I(J; H) ≤ B log 2.

(174)

We decode J by the closest element of the packing in L2 . The pairwise separation in Equation (172) implies that an incorrect nearest-packing decision requires squared estimation error of at least one quarter of that separation. Fano’s inequality then gives a constant c6 > 0 such that   B log 2 + log 2 2 2β b inf sup Ef [∥fE,D − f ∥2 ] ≥ c6 h 1− . (175) E,D f ∈Hβ (L,B ) c4 M f We choose m so that M = md is the smallest admissible order of B +1 for which the expression in parentheses is bounded below by a positive constant. Since h = M −1/d , Equation (175) becomes Rn,B ≥ c7 (B + 1)−2β/d . (176) The risk is at least both Equations (170) and (176). Reducing the constant proves Equation (66).

43

Related documents

• Handover of In-Context Learning State Across Session Boundaries#609685
• GeoGR: Enabling Spatio-Temporal Aware Industrial-scale Generative POI Recommendations#798772
• GeoGR: Enabling Spatio-Temporal Aware Industrial-scale Generative POI Recommendations#797358
• Post-Training on Office Work Improves Software Engineering: A Behavioral Account of Cross-Domain Transfer#424044
• Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuning#134508