2026-07-24
Hilbert Operator for Progressive Encoding (HOPE) A Mathematical Framework for Deconstructing Learned Representations in Deep Networks Hossein Mobahi1 and Peter L. Bartlett1,2
arXiv:2607.21366v1 [cs.LG] 23 Jul 2026
1 Google DeepMind, 2 University of California, Berkeley
Deep neural networks encode complex representations, but deconstructing this internal knowledge remains a challenge. Given the link between learning and compression, network compression offers a promising lens to analyze this knowledge. However, standard compression heuristics often suffer from scale symmetries and architectural biases. To resolve these, we introduce Hilbert Operator for Progressive Encoding (HOPE), a mathematical framework to gradually deconstruct the representations in trained network weights. HOPE shifts network compression from the discrete domain into a Hilbert space of continuous functions. By modeling individual neurons as rank-1 Hilbert-Schmidt operators, HOPE unifies pruning and neuron merging as low-rank subspace projection. Extending this formulation, HOPE introduces macro block eviction to encompass multi-layer structures like entire residual pathways under the same unified metric. This unified approach enables unbiased architectural decisions across layers with different types and sizes. HOPE is a data-free and hyperparameter-free framework. We present proof-of-concept experiments in model compression and fine-tuning to highlight the practical potential of our theory.
1. Introduction While deep neural networks learn complex representations, deconstructing this knowledge from numerical weights remains challenging. In this work, we use model compression as a measurable proxy task to study these internal representations objectively. Given the fundamental link between compression and learning [Rissanen, 1978, Hinton and Van Camp, 1993], which has recently been reinforced by demonstrating that LLMs are general-purpose compressors [Delétang et al., 2023] and amortized algorithmic predictors [Genewein et al., 2026], we believe compression provides a promising lens to study this issue objectively. Deconstructing opaque networks through capacity reduction has foundational roots [Mozer and Smolensky, 1988]. Viewed through a modern information-theoretic lens, learning is essentially the systematic discarding of task-irrelevant noise to isolate generalizable core patterns [Tishby et al., 1999, Shwartz-Ziv and Tishby, 2017]. Consequently, we posit that progressive compression is an effective tool for achieving this separation. Because core invariants resist pruning significantly longer than malleable slack, iteratively reducing capacity with minimal distortion naturally peels away the periphery to expose the network’s universal feature space [He et al., 2026, Nguyen et al., 2026, Wang et al., 2025]. This post-training deconstruction mirrors the network’s learning dynamics: the “coarse-graining” dimensionality reduction that gradient descent originally used to build them [Dandi et al., 2025]. Indeed, theoretical analyses confirm that networks learn by incrementally adding effective units to model increasingly complex functions [Zhang et al., 2025]. While biologically inspired paradigms like [Behrouz et al., 2026] achieve this core-slack segregation by expanding a network to consolidate memories, we demonstrate that this separation can be accomplished efficiently through data-free compression. Despite compression’s promise, the opacity of deep networks poses hurdles for objectively identifying and reducing capacity. Simple heuristics such as magnitude-based pruning (e.g. norm of raw Corresponding author(s): [email protected], [email protected] © 2026 Google. All rights reserved
Hilbert Operator for Progressive Encoding (HOPE)
weights) often fail because these magnitudes are typically optimization artifacts rather than importance indicators [Scholl et al., 2021, Tanaka et al., 2020, Hooker et al., 2019], as also long observed in network sparsification [LeCun et al., 1989, Hassibi and Storkey, 1992, Frankle and Carbin, 2019]. To address the shortcomings of magnitude based notions of capacity, we propose to transition from the physical parameter space to the function space, specifically the underlying transformation a neuron applies to its input. This perspective treats the entire neuron, rather than individual weight matrices, as the atomic unit of the network. Note that this transformation must be analyzed over the relevant data manifold rather than the entire input space. For instance, characterizing a neuron’s behavior within a dog-versus-cat image classification task requires isolating the data manifold specific to those two classes, while discarding irrelevant classes and non-natural images. While one might attempt to approximate the input manifold to each neuron empirically by passing a finite dataset through the network, such finite-dimensional approximations tether the evaluation to specific samples. This reliance on explicit data sets can render the resulting architecture brittle to distribution shifts and disproportionately degrade performance on long-tail features [Hooker et al., 2019]. Furthermore, empirical approaches that rely on continuous activation matching or curvature approximations [Molchanov et al., 2017, Luo et al., 2017] incur a severe computational penalty, as they require numerically evaluating neural activations across the dataset repeatedly during every iteration of the progressive model encoding. To avoid both the drawbacks of weight-based parameterization and the pitfalls of empirical data dependence, we introduce the Hilbert Operator for Progressive Encoding (HOPE) framework. The core philosophy of HOPE is that evaluation and analysis of neurons must occur in the function space, the resulting compression must be executed discretely on the network’s parameters. By lifting the parameter vectors and matrices of individual neurons into continuous functions, HOPE models each neuron as a rank-1 Hilbert-Schmidt operator. This abstraction unifies pruning and merging under a single theoretical paradigm: an optimal low-rank projection within a functional tensor space, where distances are measured by ∥·∥ H . Once the optimal reductions (pruning or merging) are computed in this pure functional space, the framework projects the resulting continuous operators back onto the network parameters to execute the compression. A key advantage of HOPE is that it operates entirely data-free for networks utilizing Batch Normalization (BN).1 While prior research has successfully exploited BN moving averages to generate synthetic images for data-free knowledge distillation [Yin et al., 2020, Micaelli and Storkey, 2019], HOPE leverages this information in a different manner. Instead of generating synthetic spatial data or relying on massive real datasets to drive input signals, HOPE applies the Maximum Entropy principle directly to the empirical BN statistics embedded within the trained model checkpoint. This yields a continuous surrogate for the local input distribution at each neuron, allowing the framework to analytically evaluate the integrals required for Hilbert space norms ∥·∥ H . This fully continuous, analytical perspective largely mitigates scaling symmetries inherent in raw parameters, and thus enables unbiased capacity measurements across heterogeneous layers without requiring a single real or synthetic data sample. The primary value of HOPE lies in establishing a rigorous, hyperparameter free (and also data free in presence of BN) mathematical framework for the progressive encoding of trained deep networks. We present proof-of-concept applications for model compression and fine-tuning experiments to empirically validate these theoretical capabilities, rather than to establish exhaustive, large-scale benchmarks. 1 This data-free approach relies on global BN statistics. For architectures lacking BN, HOPE easily adapts: it requires only a simple, one-time calibration pass over a small data batch to capture the necessary pre-activation statistics.
2
Hilbert Operator for Progressive Encoding (HOPE)
Paper Organization. We begin with a literature review in Section 2 and formally define a neuron within our framework in Section 3. To enable data-free evaluation, Section 4 constructs a surrogate distribution constrained by Batch Normalization (BN) statistics. In Section 5, we lift discrete neurons into continuous Hilbert-Schmidt operators. Section 6 then derives a cost functional J that captures the distortion induced by pruning or merging via subspace projection. Building on this, Section 7 derives the optimal parent neuron in function space for a given pair of merged neurons, detailing how this continuous representation is mapped back to the network’s discrete parameters. Section 8 extends this projection metric to the macro level to evict larger residual blocks. Section 9 introduces a rate-distortion-inspired objective that balances the distortion cost J against the resulting reduction in parameter count; this objective is used to greedily select the next optimal compression action. Section 10 presents the encoding process as a loop over greedy selection of compression actions. Finally, Section 11 provides a proof-of-concept evaluation of our framework for model compression and fine-tuning.
2. Related Works Pruning and Parameter-Space Methods. Network compression has historically relied on parameterspace pruning, evolving from early Taylor-expansion techniques [LeCun et al., 1989, Hassibi and Storkey, 1992] to modern unstructured [Han et al., 2015, Frankle and Carbin, 2019] and structured [He et al., 2017, Wen et al., 2016] approaches. HOPE directly compares against structured baselines utilizing 𝐿1 -norms [Li et al., 2017] and BN scaling [Liu et al., 2017]. To capture functional importance, methods ranging from early skeletonization via error derivatives [Mozer and Smolensky, 1988] to modern empirical activation tracking [Molchanov et al., 2017, Molchanov et al., 2019, Singh and Alistarh, 2020, Luo et al., 2017] rely on dataset passes; however, this introduces computational bottlenecks and brittleness to distribution shifts [Hooker et al., 2019]. Conversely, parameter-centric heuristics suffer from “scale symmetry” [Blalock et al., 2020, Renda et al., 2020, Badrinarayanan et al., 2015, Dinh et al., 2017], where overparameterized many-to-one mappings [Neyshabur et al., 2015] mean raw magnitudes often reflect optimization artifacts rather than true importance [Scholl et al., 2021, Tanaka et al., 2020, Hooker et al., 2019]. Although methods like [Lee et al., 2021] can partially mitigate global scale symmetries, they remain bound to parameter-space magnitude heuristics, which are vulnerable to within-neuron scaling artifacts. By migrating evaluation to a continuous Hilbert space, HOPE circumvents both empirical data bottlenecks and parameter-space artifacts. This shift from structural to functional mappings aligns with the Platonic Representation Hypothesis [Huh et al., 2024], which posits that diverse networks converge to a shared statistical model of reality, treating weight spaces as mere shadows. HOPE operationalizes this intra-model, defining identity via continuous operators rather than superficial parameters. Neuron Alignment and Model Merging. Some recent methods on merging networks include permutation invariances [Entezari et al., 2022, Ainsworth et al., 2023], Optimal Transport [Singh and Jaggi, 2020], alignment strategies [Tatro et al., 2020], or feature zipping [Stoica et al., 2024]. Other approaches perform data-free neuron merging by evaluating pairwise parameter similarities [Srinivas and Babu, 2015] or clustering weights via adaptive scalar hashing [Yvinec et al., 2021]. However, these methods typically rely on combinatorial matching or parameter averaging. HOPE advances this paradigm by unifying pruning and merging under a single continuous operation: optimal low-rank projection in a functional tensor space, penalizing distortion via the Hilbert-Schmidt norm. This geometric approach mirrors using subspace embeddings to capture hierarchical and compositional representations [Moreira et al., 2026]. Merging acts as the inverse to “feature splitting” [Bricken et al., 2023] or computation in superposition [Hänni et al., 2024], where overparameterized models fragment concepts across correlated sub-features, a phenomenon rooted in foundational vector space arithmetic
3
Hilbert Operator for Progressive Encoding (HOPE)
[Mikolov et al., 2013, Pennington et al., 2014] and the Linear Representation Hypothesis [Park et al., 2023, Engels et al., 2024]. By projecting a rank-2 neuron pair into an optimal rank-1 Hilbert-Schmidt parent, HOPE reconsolidates this distributed knowledge, reclaiming capacity while preserving the network’s underlying linear geometry. Macro Architecture and Layer Pruning. To reduce network’s depth, standard methods employ stochastic depth [Huang et al., 2016], heuristic layer dropping [Fan et al., 2020], Neural Architecture Search [Zoph and Le, 2017, Liu et al., 2019], or dynamic routing gates [Veit and Belongie, 2018, Wang et al., 2018]. However, these approaches decouple macro-architectural decisions from granular feature selection, relying on separate optimization phases or custom hyperparameters. HOPE unifies these scales by formalizing block eviction as a macroscopic function subspace projection. Mapping this architectural deletion to the identical capacity cost J allows macro-reductions to compete against granular pruning and merging within a single, hyperparameter-free decision engine. Data-Free Compression and Maximum Entropy Surrogates. Bypassing the original training data often involves inverting BN statistics to generate synthetic inputs [Lopes et al., 2017, Cai et al., 2020, Nagel et al., 2019, Micaelli and Storkey, 2019, Yin et al., 2020] or conserving discrete synaptic flow at initialization [Tanaka et al., 2020]. Rather than generating explicit samples, HOPE elevates these empirical statistics via the Maximum Entropy principle [Jaynes, 1957] to construct a continuous analytical surrogate, similar to some task-agnostic pruning of LLMs [Ma et al., 2023]. Paralleling theoretical analyses of infinite-width networks and Gaussian Processes [Neal, 1996, Lee et al., 2018, Jacot et al., 2018, Yang, 2019], this framework allows infinite-dimensional integrals to be resolved without a single forward pass. Parameter Plasticity, Continual Learning, and Representation Deconstruction. HOPE conceptually bridges progressive compression with transfer learning [Yosinski et al., 2014, Hu et al., 2022, Houlsby et al., 2019] and the Stability-Plasticity dilemma of continual learning [Grossberg, 1987, Kirkpatrick et al., 2017, Zenke et al., 2017]. Drawing parallels to Complementary Learning Systems in cognitive neuroscience [McClelland et al., 1995, Kumaran et al., 2016], and aligning with the growing recognition across deep learning [Kong et al., 2026], we hypothesize that the learned representation must be explicitly segregated into a universal core of invariants and a peripheral slack of malleable volume in order to allow learning without forgetting. While previous frameworks attempt to protect foundational knowledge against representational drift using computationally massive O ( 𝑁 3 ) orthogonal matrix projections [Saha et al., 2021, Zeng et al., 2019, Yang et al., 2025, HuggingFace Research Team et al., 2026] or dataset-dependent quadratic penalties [Kirkpatrick et al., 2017], HOPE provides a purely data-free alternative. By computing neuron capacity in O ( 𝑁 ) time, its progressive pruning and merging peel away slack and expose core abstractions, computationally mirroring how awake biological circuits rapidly decorrelate co-activated neurons to maintain network stability [Andrei et al., 2023]. This perspective is corroborated by recent literature utilizing capacity reduction to deconstruct hierarchies. For example, targeted parameter removal has been used to explain robustness [He et al., 2026], uncover heavy-tailed synaptic backbones [Nguyen et al., 2026], and demonstrate hierarchical learning through progressive feature compression [Wang et al., 2025]. Collectively, these works support the premise that HOPE’s neuron capacity may efficiently identify foundations versus plastic slack, laying the groundwork for transfer learning and downstream algorithmic interpretability [Bau et al., 2017, Morcos et al., 2018, Olah et al., 2020, Neyshabur et al., 2017].
3. The Neuron Scale invariances are inherently relational and that is why isolated parameters fail to capture them. The minimal architectural unit where these symmetries fully manifest is the neuron in its entirety
4
Hilbert Operator for Progressive Encoding (HOPE)
(encapsulating its incoming weights, BN parameters, non-linear activation, and outgoing weights). In this section we discuss how these within-neuron scale symmetries can be factored out, while deferring global scale symmetries to Section 5. For clarity, we derive our theoretical framework under the assumption of a fully connected architecture. However, this formalism seamlessly extends to convolutional networks (see Appendix B.1). Indeed, the architectures evaluated in Section 11 rely on this adaptation. Consider a neuron 𝑖 with input weights 𝒘raw,𝑖 ∈ ℝ𝑛 and output weights 𝒘out,𝑖 ∈ ℝ𝑐 . Let the neuron be subject to learnable affine BN parameters ( 𝛾𝑖 , 𝛽𝑖 ) and empirical moving dataset statistics ( 𝜇 𝑖 , 𝜎2𝑖 ), where 𝜇 𝑖 ≜ 𝔼 X [ 𝒘𝑇raw,𝑖 𝒙 ] and 𝜎2𝑖 ≜ Var X ( 𝒘𝑇raw,𝑖 𝒙 ). To capture the signal reaching the non-linearity, we absorb the normalization operations into a set of effective parameters: √︃ √︃ eff 2 + 𝜖𝒘 𝒘in ≜ 𝛾 / , 𝑏 ≜ 𝛽 − 𝛾 𝜇 / (1) 𝜎 𝜎2𝑖 + 𝜖 , 𝑖 raw ,𝑖 𝑖 𝑖 𝑖 𝑖 ,𝑖 𝑖 where 𝜖 > 0 is a small numerical stability constant. The neuron’s end-to-end signal mapping, capturing its functional contribution to the subsequent layer for an input 𝒙 , is defined by the continuous function: 𝑓𝑖 ( 𝒙 ) = Ψ ( 𝑦𝑖 ) 𝒘out,𝑖
,
eff 𝑇 𝑦𝑖 ≜ ( 𝒘in ,𝑖 ) 𝒙 + 𝑏𝑖 ,
(2)
where Ψ (·) denotes a PH-1 activation function defined as below: Positively Homogeneous of degree 1 (PH-1) Functions An activation function Ψ : ℝ → ℝ is Positively Homogeneous of degree 1 (PH-1) if it satisfies the scaling property Ψ ( 𝑐𝑧) = 𝑐Ψ ( 𝑧) for all 𝑧 ∈ ℝ and all scalars 𝑐 ≥ 0. Examples include ReLU, Leaky ReLU, PReLU, and the linear functions. Throughout the HOPE framework, we formally designate this continuous function 𝑓𝑖 as the neuron. Operating at this atomic level mitigates two primary sources of scaling symmetry: normalization invariance (from BN) and re-parameterization invariance (from cross-layer weight resharding). We detail the former below, and defer the latter to Section 5. 3.1. Mitigating Normalization Invariance Normalization invariance arises from BN’s standardizing mechanics. Scaling raw input weights 𝒘raw,𝑖 by a constant factor 𝜆 > 0 increases the pre-activation variance by 𝜆 2 . The subsequent BN layer divides by the standard deviation, canceling 𝜆 before the non-linearity. Because downstream output remains unchanged, raw weight magnitudes can be deceptive. eff and HOPE mitigates this failure mode by evaluating capacity through effective parameters (𝒘in ,𝑖 𝑏𝑖 ). While HOPE uses raw weights 𝒘raw,𝑖 alongside BN statistics to construct the data-constrained surrogate dataset 𝑃 X (Section 4), their utility ends there. The framework then evaluates the neuron’s continuous-functional impact on this surrogate.
Physically, the non-linear activation processes the normalized, shifted signal, not the raw parameter projection. Thus, when computing the continuous integral to compute Hilbert space norms or inner eff ) 𝑇 𝒙 + 𝑏 . Because 𝒘eff divides by product, HOPE defines the pre-activation signal as 𝑦𝑖 = ( 𝒘in 𝑖 ,𝑖 in,𝑖 the empirical standard deviation 𝜎𝑖 , any magnitude inflation is canceled. By evaluating the signal impacting the activation function, HOPE guarantees the capacity criterion reflects functional utility rather than scale artifacts.
5
Hilbert Operator for Progressive Encoding (HOPE)
4. The Neural Signal Distribution Since HOPE operates in a data-free regime, the true input distribution 𝑃 ∗X (encompassing both initial data and subsequent layer activations) is inaccessible. However, because we model neurons as continuous Hilbert-Schmidt operators, evaluating their inner products requires integrating over the data distribution. To resolve this, we invoke the Maximum Entropy principle [Jaynes, 1957] to construct a Gaussian surrogate constrained by BN statistics. While a Gaussian approximation may seem overly idealistic, we explain below in two steps why it aligns closely with modern neural architectures. Step 1: Gaussianity of Pre-Activation. While a neuron’s true input distribution 𝑃 ∗X often lies on a complex, highly non-Gaussian manifold, each neurons observes its input 𝒙 only through 1-dimensional Í linear projections 𝑦 = 𝑛𝑗=1 𝑤 𝑗 𝑥 𝑗 . As the fan-in dimension 𝑛 grows, by the Central Limit Theorem and the Diaconis-Freedman effect [Diaconis and Freedman, 1984], these aggregated signals converge to a Gaussian distribution. Consequently, neurons remain oblivious to the complex data manifold; from their perspective the pre-activation 𝑦𝑖 is Gaussian. Although nonlinear activations (e.g., ReLU) disrupt this Gaussianity, subsequent high-dimensional linear transformations recursively smooth the signals back into Gaussian pre-activations across layers. Step 2: Gaussian Surrogate for the Input 𝒙 . While neurons are oblivious to the true shape of the data manifold and perceive their input as a Gaussian signal 𝑦 . Based on this observation, for theoretical convenience, we substitute the complex true data with a tractable surrogate. For architectural consistency, the surrogate distribution must satisfy the same observational bottleneck: its 1D linear projections must remain Gaussian. By definition, if every linear combination of a random vector is Gaussian, the vector itself must be multivariate Gaussian. Thus, to construct a surrogate distribution aligned with a world where every linear observer (neuron) sees a Gaussian, that surrogate is necessarily a multivariate Gaussian, 𝑃 X = N ( 𝝁 ˆ 𝑥 , Σ̂𝑥 ). Architectural Context Throughout this work, we contextualize our framework within standard modern vision architectures, specifically the ResNet-50 (V1) architecture. The canonical computational block follows the sequence: convolution, followed by BN, followed by a ReLU activation Conv → BN → ReLU. Consequently, the input vector 𝒙 presented to any internal convolutional layer is the output of a preceding ReLU activation. Post-ReLU Support Paradox Using a multivariate Gaussian surrogate 𝑃 X whose support is the entire ℝ𝑛 might seem problematic, given that post-ReLU inputs are non-negative 𝒙 ≥ 0. However, the purpose of 𝑃 X is to model the inner product 𝑓𝑖 , 𝑓 𝑗 H , not the true input distribution. Because 𝑓𝑖 , 𝑓 𝑗 H = 𝔼𝒙 ∼ 𝑃X [ Ψ ( 𝑦𝑖 ) Ψ ( 𝑦 𝑗 )] · 𝒘out,𝑖 , 𝒘out, 𝑗 ℝ𝑐 , this integral reduces to a 2D subspace defined by the pre-activations 𝑦𝑖 and 𝑦 𝑗 . In high dimensions, the Central Limit Theorem and the Diaconis-Freedman effect ensure that projecting high-dimensional vectors into a lowdimensional subspace rapidly converges to a bivariate Gaussian distribution. Since the integration depends solely on this 2D slice, relaxing the ambient non-negativity constraint yields a tractable surrogate while maintaining an accurate asymptotic approximation of the bivariate distribution. Optimal Surrogate. To define the optimal parameters ( 𝝁 ˆ 𝑥 , Σ̂𝑥 ) of the unknown data distribution, we can incorporate the two empirical constraints 𝒙 ∈ ℝ𝑛 : 𝔼[ 𝒘𝑇raw,𝑖 𝒙 ] = 𝜇 𝑖 and Var( 𝒘𝑇raw,𝑖 𝒙 ) = 𝜎2𝑖 6
Hilbert Operator for Progressive Encoding (HOPE)
imposed by BN for any 𝑖. We define a shared surrogate for each layer. Since the empirical means 𝝁BN ∈ ℝ𝑛 represent the 1D shadow of the dataset’s center cast through the raw weights, to find the best mean for a layer we compute the optimal least-squares approximation of the dataset’s center + , which leads to 𝝁 + 𝝁 . In underdetermined using the Moore-Penrose pseudo-inverse 𝑾raw ˆ 𝑥 = 𝑾raw BN scenarios (e.g., when compressing layers where 𝑛 > 𝑐), this pseudo-inverse yields the minimum-norm solution, which sets the unobserved orthogonal components of the data manifold to a zero mean. To find the optimal covariance matrix Σ̂𝑥 for the layer, we maximize the differential entropy of the multivariate Gaussian, 𝐻 ( 𝒙 ) ∝ log det( Σ𝑥 ) subject to the BN variance constraints 𝒘𝑇raw,𝑖 Σ𝑥 𝒘raw,𝑖 = 𝜎2𝑖 Í −1 𝑐 𝑇 for 𝑖 ∈ {1, . . . , 𝑐 }. Conceptually, applying Lagrange multipliers yields Σ̂𝑥 = , 𝜆 𝒘 𝒘 𝑖=1 𝑖 raw,𝑖 raw,𝑖 where 𝜆 𝑖 are optimized to satisfy variance equality constraints. However, inverting this covariance matrix is computationally expensive. Fortunately, the framework bypasses the computationally expensive need to compute and invert the full high-dimensional joint covariance matrix entirely: micro-operations evaluate pairwise merges restricted to a rank-2 subspace, allowing a closed-form solution via a pairwise neural kernel (Appendix E.3), while macro block eviction evaluates destruction using the 𝐿1 cumulative distance of surviving capacities (Section 8).
5. A Hilbert Functional Perspective on Neurons To mitigate parameter shape bias and facilitate the identification of dead neurons, we transition from discrete parameter analysis to a continuous function formulation. By embedding each neuron into a Hilbert space2 H and treating it as a rank-1 Hilbert-Schmidt operator, our framework evaluates the actual function the neuron computes, effectively abstracting away its physical matrix shape. Furthermore, by integrating this continuous function over the analytically derived surrogate 𝑃 X , HOPE identifies dead neurons via a closed-form expectation, bypassing the need for computationally expensive empirical forward passes over a dataset. Together, these features shift the evaluation criterion from raw parameter counts to functional capacity, quantified as the norm of the neuron’s function in H . We model each neuron as a rank-one Hilbert operator 𝑓𝑖 ∈ H . This way, a neuron’s identity is not defined by its evaluation on a single input point 𝒙 , but its continuous behavior over the entire surrogate distribution X. We quantify the capacity of the neuron by ∥ 𝑓𝑖 ∥ H . We now present this formally. We define our space of neural functions as H ≜ 𝐿2 (X , 𝑃 X ; ℝ𝑐 ), the space of square-integrable functions mapping X to the 𝑐-dimensional output space. Define 𝑔𝑖 : X → ℝ eff 𝑇 as 𝑔𝑖 ( 𝒙 ) ≜ Ψ ( 𝒘in,𝑖 ) 𝒙 + 𝑏𝑖 . We embed 𝑔𝑖 as an element of the scalar Hilbert space Hin ≜ 𝐿2 (X , 𝑃 X ; ℝ). Furthermore, the scalar activation of a neuron is sent to the next layer by scaling with the finitedimensional output weight vector 𝒘out,𝑖 ∈ Hout ≜ ℝ𝑐 . Since the output across all 𝑐 dimensions is confined to the one-dimensional subspace spanned by 𝒘out,𝑖 , this entire continuous landscape is embedded exclusively along a single vector direction. By taking the tensor product of the input function and the output vector, we construct a linear mapping across these spaces: H Hin ⊗ Hout . Thus the vector-valued function 𝑓𝑖 : X → Hout , i.e. the neuron, is an element within this tensor product space 𝑓𝑖 ≜ 𝑔𝑖 ⊗ 𝒘out,𝑖 . Since this element is constructed from the outer product of one input function and one output vector, each individual neuron 𝑓𝑖 is a rank-1 Hilbert-Schmidt operator. See Figure 1 for a visualization. This tensor formulation is fundamental for defining the merging operation in HOPE as we will discuss in Sections 6 and 7.1. Hilbert-Schmidt Inner Product and Capacity. Because our neurons are defined as rank-1 operators residing in the tensor H Hin ⊗ Hout , we must evaluate their geometric relationship using the 2 See Appendix A for a brief introduction to Hilbert spaces.
7
Hilbert Operator for Progressive Encoding (HOPE)
Input
Output
Function 𝑔𝑖 ∈ Hin
Vector 𝒘out,𝑖 ∈ Hout
Tensor Rank-1 Operator 𝑓𝑖 ∈ H
𝑥3
⊗
𝑥3 𝒘out,𝑖
= 𝑥2
Image of landscape restricted to line
𝒘out,𝑖 𝑥2
𝑥1
𝑥1
Figure 1 | Visualizing the rank-1 tensor of a neuron. Left: The input phase computes a continuous (infinitedimensional) scalar landscape 𝑔𝑖 ( 𝒙 ). Middle: The output phase defines a single, finite-dimensional weight vector 𝒘out,𝑖 . Right: The tensor product binds them. The entire function’s landscape (represented by the colored points) is mapped along the 1D subspace spanned by 𝒘out,𝑖 , which enforces the definition of a rank-1 operator.
inner product defined on this composite space: 𝑓𝑖 , 𝑓 𝑗 H = 𝑔𝑖 ⊗ 𝒘out,𝑖 , 𝑔 𝑗 ⊗ 𝒘out, 𝑗 H = 𝑔𝑖 , 𝑔 𝑗 H · in 𝒘out,𝑖 , 𝒘out, 𝑗 H = 𝔼𝒙 ∼ 𝑃X [ Ψ ( 𝑦𝑖 ) Ψ ( 𝑦 𝑗 )] · 𝒘out,𝑖 , 𝒘out, 𝑗 ℝ𝑐 . We define the capacity of a neuron as its out √︁ Hilbert-Schmidt norm ∥ 𝑓𝑖 ∥ H = ⟨ 𝑓𝑖 , 𝑓𝑖 ⟩ H , which we will later use to decide what neuron to prune and which macro block to evict. Kernel Formulation. We define the kernel of two neurons 𝑖, 𝑗 as 𝐾 ( 𝑖, 𝑗) ≜ 𝔼𝒙∼𝑃X [ Ψ ( 𝑦𝑖 ) Ψ ( 𝑦 𝑗 )]. Under √︁ this, the capacity of a neuron simplifies to ∥ 𝑓𝑖 ∥ H = ∥ 𝒘out,𝑖 ∥ 2 · 𝐾 ( 𝑖, 𝑖) and the inner product of two neurons becomes 𝑓𝑖 , 𝑓 𝑗 H = 𝒘out,𝑖 , 𝒘out, 𝑗 ℝ𝑐 𝐾 ( 𝑖, 𝑗). Closed form expression for these kernels are provided at the end of this section, and their derivation is presented in Appendix E when Ψ is the ReLU activation function. eff and 𝑏 of a neuron by Neuron Scale Invariance. In networks with PH-1 activations, scaling 𝒘in 𝑖 ,𝑖 𝜆 > 0 and 𝒘out,𝑖 by 1/ 𝜆 alters weight magnitudes without changing the downstream function. This symmetry confounds raw magnitude-based criteria. HOPE mitigates this issue by defining capacity as √︁ the Hilbert norm: ∥ 𝑓𝑖 ∥ H = ∥ 𝒘out,𝑖 ∥ 2 𝐾 ( 𝑖, 𝑖). Because positive homogeneity scales the kernel 𝐾 by 𝜆 , the opposing factors cancel. Consequently, HOPE guarantees an invariant capacity score regardless of weight resharding. √︁ Neuron Shape Invariance. By definition, the neuron’s functional capacity ∥ 𝑓𝑖 ∥ H = ∥ 𝒘out,𝑖 ∥ 2 𝐾 ( 𝑖, 𝑖) depends on the input space X solely through the kernel term 𝐾 ( 𝑖, 𝑖) ≜ 𝔼𝒙∼𝑃X [ Ψ2 ( 𝑦𝑖 )]. Rather than eff ∥ or ∥ 𝒘 counting discrete incoming parameters 𝑛 or computing weight magnitudes such as ∥ 𝒘in raw,𝑖 ∥, ,𝑖 𝑛 3 this formulation abstracts away the physical dimensionality of the input tensor 𝒙 ∈ ℝ . Furthermore, when paired with the layer-wise magnitude neutrality axiom, this function-based approach ensures the neuron’s evaluation is entirely invariant to both its input and output dimensions.4 3While increasing the fan-in 𝑛 naturally inflates the pre-activation variance Var
𝑇 X ( 𝒘raw,𝑖 𝒙 ), this scaling artifact is then √︃ eff ≜ ( 𝛾 / 𝜎2 + 𝜖) 𝒘 mitigated by the absorbed BN parameters. Because 𝒘in 𝑖 raw,𝑖 , the variance of the pre-activation signal 𝑦𝑖 is 𝑖 ,𝑖
bounded entirely by the learned scale 𝛾𝑖2 . Consequently, the expected activation energy 𝔼𝒙∼𝑃X [ Ψ2 ( 𝑦𝑖 )] remains decoupled from the physical width of the input tensor. 4 Although capacity ∥ 𝑓 ∥ 𝑖 H scales with the output dimension 𝑐 via ∥ 𝒘out,𝑖 ∥ 2 , HOPE mitigates this downstream. As shown in Section 6, compression is governed by the distortion cost J . Derived axiomatically, J normalizes a neuron’s capacity
8
Hilbert Operator for Progressive Encoding (HOPE)
Figure 2 | Visualization of Transition Costs of Merging in H 2 . The axes schematically represent the space of continuous functions H , illustrated as a smooth transition from linear to sinusoidal. During a merge operation, the layer transitions from an initial state Φ (0) = Φ𝑎 to a predeletion target Φ (1) = Φ̃𝑏 where 𝑓1 = 𝑓2 . For the sake of illustration, suppose 𝐸 ( 𝑡 ) ≈ 𝐸𝑏 and ∫1 𝑐 ( Φ ( 𝑡 )) = 𝑐0 for 0 ≤ 𝑡 ≤ 1. Then 0 J¤capacity 𝑑𝑡 ≈ ∫1 − 𝐸𝑐0𝑏 0 𝐸¤ ( 𝑡 ) 𝑑𝑡 = − 𝐸𝑐0𝑏 ( 𝐸𝑏 − 𝐸 𝑎 ) = − 𝐸𝑐0𝑏 Δ𝐸 ∝ −Δ𝐸, Initial State Φ (0) yielding a penalty proportional to −Δ𝐸 . How∫1 ∫1 ever, 0 J¤proj 𝑑𝑡 ≈ 𝐸𝑐0𝑏 0 𝑠¤( 𝑡 ) 𝑑𝑡 = 𝐸𝑐0𝑏 ( 𝑠 (1) − 𝑠 (0)) ∝ 𝐷 ( Φ𝑎 , Φ̃𝑏 ), the Euclidean projection distance shown 𝑓1 ∈ H as the straight purple line.
Ra
𝐸
Δ
1
0
∫
∝ Jc¤a
pa
nk -1
Su
= city −1 𝒅 𝒕
bs pa ce
(𝑓
1
=
𝑓2
)
𝑓2 ∈ H
Terminal State Φ̃ (1) ∫
1
𝐸
𝐸
𝐸
4
5
=
2
3
=
=
𝐸
Origin ( 𝐸 = 0)
=
¤ ∝ 𝑫 Jpro (𝚽 j 𝒅𝒕 𝒂,𝚽 ˜ 𝒃) 0
Neuron Merging via Hilbert-Schmidt Projection. Similar to pruning, merging also reduces the network’s neuron count by one, but it can provide a higher-fidelity reduction when the selected pair have a strong cosine similarity in H . We define the merger through an optimal Hilbert-Schmidt projection. Since neurons 𝑖, 𝑗 are each rank-1 operators, their joint contribution [ 𝑓𝑖 , 𝑓 𝑗 ] spans a 2dimensional subspace in H consisting of operators of rank at most 2. Merging them into a single parent neuron is defined as finding the optimal rank-1 approximation of this tensor subspace. Self-Kernel of ReLU Neurons Let 𝜙 and Φ be the standard Normal PDF and CDF respectively. Then: 𝛽𝑖 𝛽𝑖 𝐾 ( 𝑖, 𝑖) = ( 𝛾𝑖2 + 𝛽𝑖2 ) Φ + 𝛽 𝑖 | 𝛾𝑖 | 𝜙 | 𝛾𝑖 | | 𝛾𝑖 |
(3)
Cross-Kernel of ReLU Neurons For brevity, let: D
eff , 𝒘eff 𝒘in ,𝑖 in, 𝑗
𝜌eff ≜
E ,
eff 𝒘in ,𝑖 2
𝜅≜
eff 𝒘in ,𝑗 2
𝜌eff
1 − 𝜌2eff
!
© | 𝛾𝑖 | ª © | 𝛾 𝑗 | ª ® ® eff ® eff ® 𝒘in,𝑖 𝒘in, 𝑗 2¬« 2¬ «
,
𝜌 ˆ𝑖 𝑗 ≜
Then the cross-kernel has the forma : √︁ 1 √︃ 𝐾 ( 𝑖, 𝑗) ≈ 1 − 𝜌ˆ2𝑖 𝑗 + ( 𝜋 − arccos 𝜌ˆ𝑖 𝑗 ) 𝜌ˆ𝑖 𝑗 𝐾 ( 𝑖, 𝑖) 𝐾 ( 𝑗, 𝑗) 𝜋
2𝜅 . (4) √ 1 + 1 + 4𝜅2
(5)
aWhile the exact cross-kernel can be derived analytically, it requires evaluating a bivariate normal CDF for
every neuron pair, which is computationally prohibitive for large networks. Instead, we approximate the kernel by assuming zero bias 𝛽𝑖 , 𝛽 𝑗 ≈ 0. This isolates the angular alignment, 𝜌ˆ𝑖 𝑗 , as the primary driver of redundancy and avoids costly CDF evaluations. Full derivations of both the exact and approximate kernels are in Appendix E.
against the layer’s total capacity. Since all neurons in a layer share the same output space ℝ𝑐 , this emergent normalization factors out 𝑐.
9
Hilbert Operator for Progressive Encoding (HOPE)
6. Layer Transition Costs Scale symmetries also manifest globally: shallow layers processing high-variance data often yield neurons with larger capacities than deeper layers operating on compressed latent representations [Hanin and Rolnick, 2018, Tanaka et al., 2020]. For a compression method to account for this bias, it must evaluate individual neurons within the context of their entire layer [Lee et al., 2021]. We formalize this global context via a layer state Φ ≜ ( 𝑓1 , 𝑓2 , . . . , 𝑓𝑁 ), where 𝑁 is the number of active neurons in the layer. A single compression step maps an initial state Φ𝑎 to a reduced state Φ𝑏 , where | Φ𝑏 | = 𝑁 − 1. The entire compression process is thus a chain of discrete state transitions across various layers, guided by a cost J ( Φ𝑎 , Φ𝑏 ) > 0 that quantifies the resulting model distortion. This section focuses on deriving this J . 6.1. Continuous-Time Relaxation Despite the conceptual clarity of these transitions, their discrete nature (where the architecture hops from state Φ𝑎 to Φ𝑏 via pruning or merging) presents a significant barrier to mathematical analysis. To bridge the gap between abstract analysis and algorithmic execution, we proceed in two steps. First, we perform a continuous relaxation: instead of an instantaneous jump, we define a continuous deformation Φ ( 𝑡 ) over 𝑡 ∈ [0, 1], interpolating between Φ (0) = Φ𝑎 and a pre-deletion target Φ (1) = Φ̃𝑏 (Figure 2). This allows us to use differential equations to express the infinitesimal cost of shrinking a layer’s capacity. Second, to compute the total transition cost, we integrate this differential cost with respect to 𝑡 . Our objective is to resolve this integral into an expression that depends only on the physically realizable endpoints (Φ𝑎 and Φ̃𝑏 ), bypassing the need to evaluate fictitious intermediate states along the continuous path. However, since this integral generally lacks a closed-form solution, and numerical evaluation incurs computationally prohibitive runtime overhead, we instead derive a closed-form upper bound on the analytically intractable integral. Layer Capacity. To develop a layer cost J , we first extend the single-neuron capacity, ∥ 𝑓𝑖 ∥, to define a layer capacity 𝐸 ( Φ) for state Φ, where 𝐸 ( 𝑓1 ) = ∥ 𝑓1 ∥. A natural requirement is that 𝐸 ( Φ) remains invariant to arbitrary neuron partitioning. Assuming 𝐸 ( Φ) is a symmetric, separable, and homogeÍ neous functional of individual capacities, this condition uniquely determines 𝐸 ( Φ) = 𝑘𝑁=1 ∥ 𝑓𝑘 ∥ H (by Í 1/ 𝑝 𝑝 Lemma C.1). For some intuition, suppose that 𝐸 ( Φ) = ∥ 𝑓𝑘 ∥ H . Partitioning a neuron 𝑓0 into 𝑀 fractions 𝑓0 / 𝑀 yields 𝑀 (1− 𝑝 )/ 𝑝 ∥ 𝑓0 ∥ H . Capacity invariance for any 𝑀 requires (1 − 𝑝)/ 𝑝 = 0, yielding 𝑝 = 1. Axiomatic Cost Formulation. To ensure a well-posed definition of J , we introduce the following natural axioms: 1. Magnitude Neutrality: J must be scale invariant: ∀𝑘 > 0 ; J ( 𝑘 Φ𝑎 , 𝑘 Φ𝑏 ) = J ( Φ𝑎 , Φ𝑏 ). 2. Connectivity Preservation: J must establish an asymptotic barrier preventing layer extinction: lim𝐸 ( Φ𝑏 )→0+ J = ∞. 3. Infinitesimal Capacity Dependence: J must be additive along ∫1 continuous paths and be driven by the reduction in layer capacity: J ( Φ𝑎 , Φ𝑏 ) = 0 −𝜉 ( Φ ( 𝑡 )) 𝐸¤ ( 𝑡 ) 𝑑𝑡 , where 𝐸¤ ( 𝑡 ) ≜ 𝑑𝐸 ( Φ ( 𝑡 ))/𝑑𝑡 and 𝜉 ( Φ ( 𝑡 )) > 0 is a state-dependent density function. While Axioms 1 and 2 define boundaries of the theory, Axiom 3 acts as an idealized analytical tool modeling a continuous capacity drain 𝐸¤ ( 𝑡 ) < 0. This allows us to deduce the fundamental shape of the cost function. Under these premises, we can prove (By Theorem C.2) that J must obey Jcapacity ( Φ𝑎 , Φ𝑏 ) = ∫1 ¤ −𝑐 ( Φ ( 𝑡 )) 𝐸 ( 𝑡 ) 𝑑𝑡 , where 𝐸¤ ( 𝑡 ) < 0 (due to capacity reduction) and 𝑐 ( Φ ( 𝑡 )) > 0 is a scale-invariant 0
𝐸 ( Φ (𝑡 ) )
factor (i.e., 𝑐 ( 𝑘Φ) = 𝑐 ( Φ) for any Φ ∈ H 𝑁 and 𝑘 > 0). Piecewise Constant 𝑐 ( Φ ( .)). To bridge continuous theory with discrete execution, we restrict 𝑐 ( Φ) to remain constant along any discrete state transition Φ𝑎 → Φ𝑏 , e.g. 𝑐 ( Φ ( 𝑡 )) = 𝑐 ( Φ𝑎 ) for 𝑡 ∈ [0, 1]. This allows us to factor 𝑐 ( Φ) out of the integral for both Jcapacity and all subsequently derived cost 10
Hilbert Operator for Progressive Encoding (HOPE)
functionals; for Jcapacity , this directly yields the analytical solution Jcapacity = 𝑐 ( Φ𝑎 ) ln( 𝐸𝐸𝑎𝑏 ). Upon reaching the terminal state, physically removing extinguished neurons causes 𝑐 ( Φ ( 𝑡 )) to snap to a new value 𝑐 ( Φ𝑏 ). Consequently, 𝑐 ( Φ ( 𝑡 )) acts as a globally piecewise constant function that remains locally constant during any integration step. While Jcapacity is not yet the final objective used in our optimizer, confirming that it satisfies Axioms 1 and 2 ensures we are on track, while its derivation via integration inherently satisfies the idealized capacity dependence assumption. 6.2. Bounding the Projection Cost While J¤ ( 𝑡 ) is driven by the relative capacity reduction − 𝐸¤ / 𝐸, our framework needs to minimize projection error5 (Section 5). We bridge the two by calibrating along an orthogonal trajectory where ∫𝑡 ¤ ( 𝜏) ∥ H 𝑁 𝑑𝜏 𝑑𝑠 = −𝑑𝐸 translates the abstract capacity loss − 𝐸¤ into a geometric speed 𝑠¤. Here 𝑠 ( 𝑡 ) = 0 ∥ Φ is the arc-length swept by Φ ( 𝑡 ) through the space H 𝑁 . Because H 𝑁 is isotropic, this substitution generalizes to any arbitrary deformation path, yielding J¤proj ( 𝑡 ) = 𝑐 ( Φ ( 𝑡 )) 𝐸 (𝑠¤Φ( 𝑡()𝑡 ) ) (Definition 44). This substitution shifts J from pure capacity loss to any distance traversed, meaning the strict 𝐸¤ ( 𝑡 ) < 0 assumption from the idealized model is no longer required along the physical path. The compression algorithm executes discrete leaps (e.g., snapping neurons 𝑓𝑖 and 𝑓 𝑗 to a shared parent 𝑓 𝑝 ). Evaluating the cost of this transition conceptually requires integrating J¤proj = 𝑐 · 𝑠¤/ 𝐸 over the jump path. However, because runtime integration is computationally prohibitive, we seek a fast, closed-form proxy. Since underestimating this integral risks destructive jumps (e.g., removing orthogonal features) and breaching layer depletion barriers before the continuous cost can diverge, we derive a closed-form upper bound to enforce cautious greedy optimization. We construct this bound by exploiting the inverse relationship between J¤proj and 𝐸 in J¤proj = 𝑐 · 𝑠¤/ 𝐸. For any arbitrary deformation path connecting Φ𝑎 to Φ𝑏 , we can establish an upper bound on the integral cost by replacing the dynamic capacity 𝐸 ( 𝑡 ) with a constant minimum, 𝐸min , allowing us to pull the denominator outside the integral. This yields a bounded fractional cost where the numerator ∫1 is the path’s total arc length, 0 𝑠¤( 𝑡 ) 𝑑𝑡 . Because infinitely many curves in H 𝑁 connect the two states, this establishes a family of valid upper bounds. To tighten this proxy cost, we minimize the numerator by selecting the path with the shortest arc length: the straight-line trajectory in H 𝑁 . This evaluates Í 1 to the traversed Euclidean distance 𝐷 ( Φ𝑎 , Φ̃𝑏 ) ≜ ∥ Φ𝑎 − Φ̃𝑏 ∥ H 𝑁 = ( 𝑘𝑁=1 ∥ 𝑓𝑘( 𝑎 ) − 𝑓˜𝑘( 𝑏 ) ∥ 2H ) 2 . Next, to complete this bound, we must safely approximate the denominator’s minimum 𝐸min along this chosen straight-line path. Because the straight-line geometric path acts as a secant across the space of functions (abandoning the strict 𝐸¤ ( 𝑡 ) < 0 assumption), the capacity 𝐸 ( 𝑡 ) can temporarily dip below the pre-deletion target 𝐸 ( Φ̃𝑏 ). To safely absorb this without breaking the integral bound, we introduce a safety buffer by evaluating the denominator at the true terminal state 𝐸 ( Φ𝑏 ). For highly correlated neuron pairs, 𝐸 ( 𝑡 ) ≥ 𝐸 ( Φ𝑏 ) throughout the straight-line transition (Lemma C.3). Substituting the minimized numerator 𝐷 and evaluating the constant denominator as 𝐸 ( Φ𝑏 ) yields the final bound Jproj ( Φ𝑎 , Φ𝑏 ) ≤ 𝑐 ( Φ𝑎 ) 𝐷 (𝐸Φ( Φ𝑎 ,𝑏Φ̃)𝑏 ) ≡ Jbound ( Φ𝑎 , Φ𝑏 ) (see Theorem C.4). Here, Φ̃𝑏 ∈ H 𝑁 is the pre-deletion target at 𝑡 = 1 (e.g., a duplicated parent [ 𝑓 𝑝 , 𝑓 𝑝 ]) but the 𝑁 -dimensional structure remains intact. Conversely, Φ𝑏 ∈ H 𝑁 −1 is the true terminal state: the layer after the extinguished neuron is dropped. This separation ensures no dimensional mismatch in the arguments of 𝐷, while the denominator only relies on the ( 𝑁 − 1)-dimensional post-deletion capacity 𝐸 ( Φ𝑏 ). 5 Transitioning from J capacity to Jproj ensures sensitivity to feature alignment. For instance, merging two orthogonal
neurons introduces a severe subspace projection error while Jcapacity evaluates this catastrophic alignment loss identically to a merge between two collinear (hence redundant) neurons due to their equivalent linear capacity reductions. Shifting to Jproj reorients the optimization objective from macroscopic reduction to minimizing distortion within the network’s internal mapping.
11
Hilbert Operator for Progressive Encoding (HOPE)
Axiomatic Consistency of the Bounded Proxy. While the continuous functional Jcapacity was derived from our foundational axioms, the subsequent derivation of Jbound alters the underlying differential equation. Specifically, to bypass the expensive runtime integration, we introduced a surrogate curve and bounded the capacity denominator. Because these approximations manipulate the original differential equation, it is no longer guaranteed a priori that the resulting closed-form proxy inherits the axiomatic properties of its continuous predecessor. However, we can prove that Jbound (and consequently Jfinal ) still preserves the foundational axioms of Magnitude Neutrality and Connectivity Preservation (by Proposition C.5). However, the Infinitesimal Capacity Dependence assumption acts primarily as an analytical tool rather than a fundamental necessity, and is intentionally relaxed. Specifically, Jcapacity relies on integration over a path characterized by a monotonic capacity drain 𝐸¤ ( 𝑡 ) < 0. However, deriving the closed-form Jbound abandons this path integration in favor of a straight-line approximation evaluated at endpoints. Because this straight-line projection cuts directly across H 𝑁 , the intermediate capacity along the path may temporarily fluctuate, violating the assumption of monotonic decrease required by the original differential equation. Consequently, Jbound knowingly sacrifices the path-additivity required by the modeling assumption. This relaxation is necessary to translate abstract continuous theory into an efficient O (1) evaluation of discrete state transitions. Practical Notes The Correlation Constraint. The assumption 𝐸 ( 𝑡 ) ≥ 𝐸 ( Φ𝑏 ) holds only for highly correlated neurons, but this poses no practical limitation. Because the projection error 𝐷 ( Φ𝑎 , Φ𝑏 ) vanishes for collinear candidates, the greedy optimizer naturally minimizes Jbound by actively selecting highly correlated pairs, inherently satisfying this requirement. Locality of the Projection Error. Evaluating J across a wide layer might seem computationally intractable. However, for reductions modifying only a small subset of neurons S (e.g., pruning or √︃ merging), the cost restricts entirely to the perturbed subspace: Í ( 𝑎) − 𝑓𝑘( 𝑏 ) ∥ 2H / 𝐸 ( Φ𝑏 ) (Corollary C.6). This isolates the comJbound ( Φ𝑎 , Φ𝑏 ) = 𝑐 ( Φ𝑎 ) 𝑘 ∈ S ∥ 𝑓𝑘 putation from the total architectural width, guaranteeing O (1) execution time. Choice of 𝑐 ( Φ) = 𝑁 . We previously specified 𝑐 ( Φ) to be piecewise constant; we now propose a more specific definition: setting 𝑐 ( Φ) = 𝑁 for each continuous piece. This is to avoid unfair removal of critical diversity from wide layers by the global optimizer before addressing obvious redundancies in narrow bottlenecks, which may occur as capacity 𝐸 ( Φ) intrinsically scales with layer width. To mitigate this width bias, we normalize J using the average feature capacity6 𝐸 ( Φ)/ 𝑁 , which can be implemented by setting 𝑐 ( Φ) = 𝑁 . Substituting this 𝑐 ( Φ) into Jbound yields the final cost Jfinal ≜ 𝑁 · 𝐷𝐸((ΦΦ𝑎𝑏,)Φ̃𝑏 ) . We can instantiate the pruning and merging costs as special cases of Jfinal . 6.3. Final Pruning and Merging Costs Pruning a neuron 𝑓𝑖 corresponds to projecting its rank-1 operator down to the null operator 0. Because the perturbed subspace only contains this single neuron S = { 𝑖 } and its terminal state is 6 Consider a mean-field assumption where each active neuron contributes an average capacity ¯ 𝑒. The incremental cost of 𝑒 pruning a single neuron evaluates to Jprune ≈ 𝑁𝑁 ·¯ = 1. This normalization renders the penalty invariant to the instantaneous ·¯ 𝑒 layer width. Without this dynamic coupling (e.g., if 𝑐 were anchored to 𝑁initial ), the incremental cost would artificially explode as the live capacity shrinks, forcing an artificial uniformity that prevents the optimizer from fully extinguishing noisy, redundant blocks.
12
Hilbert Operator for Progressive Encoding (HOPE)
0, the projection error simplifies to 𝐷 =
√︃
∥ 𝑓𝑖 − 0∥ 2H = ∥ 𝑓𝑖 ∥ H . By evaluating the terminal capacity as
𝑁·∥ 𝑓 ∥
𝐸 ( Φ𝑏 ) = 𝐸 𝑎 − ∥ 𝑓𝑖 ∥ H we get Jprune = 𝐸𝑎 − ∥ 𝑖𝑓𝑖 ∥H . H
Merging a neuron pair is slightly more involved. For neurons 𝑖 and 𝑗, their joint operator [ 𝑓𝑖 , 𝑓 𝑗 ] spans a rank-2 subspace in H . Because 𝑓𝑖 and 𝑓 𝑗 are vector-valued functions, their joint operator is matrix-valued, denoted as 𝑾joint ≜ [ 𝑓𝑖 , 𝑓 𝑗 ]. Merging compresses this into a rank-1 approximation ′ 𝑾joint . Classic unconstrained rank truncation (Eckart-Young-Mirsky) prescribes a rank-one basis 𝑓𝑏 ′ ′ and independent scaling factors 𝛼, 𝛽 ∈ ℝ by solving min 𝑓𝑏 ∈ H ,𝛼,𝛽 ∈ℝ ∥𝑾joint − 𝑾joint ∥ 2H , where 𝑾joint = [ 𝛼 𝑓𝑏 , 𝛽 𝑓𝑏 ]. However, because a physical neuron must produce a single unified output, we must restrict the ′ valid replacement pair to 𝑾joint = [ 𝑓 𝑝 , 𝑓 𝑝 ]. This enforces the constraint 𝛼 = 𝛽 = 1, yielding the ′ constrained objective min 𝑓 𝑝 ∈ H ∥𝑾joint − 𝑾joint ∥ 2H and rendering standard unconstrained projections inapplicable. Deferring the derivation of the optimal parent 𝑓 𝑝 to Section 7.1, we first establish the objective functional itself. Since the distance 𝐷 is the expected Frobenius projection error under the HilbertSchmidt norm, we expand it as follows: h i ′ ′ 𝐷2 ( Φ𝑎 , Φ𝑏 ) = ∥𝑾joint − 𝑾joint ∥ 2H = 𝔼𝒙∼𝑃X ∥𝑾joint ( 𝒙 ) − 𝑾joint ( 𝒙 ) ∥ 2𝐹 = ∥ 𝑓𝑖 − 𝑓 𝑝 ∥ 2H + ∥ 𝑓 𝑗 − 𝑓 𝑝 ∥ 2H . The terminal capacity 𝐸𝑏 updates by swapping the eliminated children for the new parent: 𝐸𝑏 = 𝐸 𝑎 − ∥ 𝑓𝑖 ∥ H − ∥ 𝑓 𝑗 ∥ H + ∥ 𝑓 𝑝 ∥ H . Substituting 𝐷 and 𝐸𝑏 yields the final merging cost. Pruning and Merging Costs √︃ 𝑁 ∥ 𝑓𝑖 ∥ H Jprune = 𝐸 𝑎 − ∥ 𝑓𝑖 ∥ H
𝑁 ,
Jmerge =
∥ 𝑓𝑖 − 𝑓 𝑝 ∥ 2H + ∥ 𝑓 𝑗 − 𝑓 𝑝 ∥ 2H
𝐸 𝑎 − ∥ 𝑓𝑖 ∥ H − ∥ 𝑓 𝑗 ∥ H + ∥ 𝑓 𝑝 ∥ H
.
(6)
7. Generating the Parent Neuron 7.1. The Parent Neuron in Hilbert Space We determine the optimal parent neuron 𝑓 𝑝∗ by minimizing Jmerge ( 𝑓 𝑝 ) subject to 𝑓 𝑝 ∈ N , where N denotes the space of realizable neurons: N ≜ { 𝑓 | 𝑓 ( 𝒙 ) = 𝒘out Ψ ( 𝒘 ˜ in · 𝒙˜)} ⊂ H .
(7)
eff , 𝑏] 𝑇 denote the augmented inputs and weights. Any non-zero function Here 𝒙˜ = [ 𝒙, 1] 𝑇 and 𝒘 ˜ in = [𝒘in 𝑓 ∈ H can be decomposed into a scalar magnitude 𝑠 > 0 and a direction 𝜓 ∈ H , such that 𝑓 = 𝑠𝜓 and ∥ 𝜓 ∥ H = 1. Applying this to the parent neuron 𝑓 𝑝 allows us to reformulate the search for 𝑓 𝑝∗ as the following nested optimization problem: √︃ ∥ 𝑓 𝑝 − 𝑓𝑖 ∥ 2H + ∥ 𝑓 𝑝 − 𝑓 𝑗 ∥ 2H min+ min s.t 𝑓𝑝 = 𝑠 𝜓 , ∥𝜓∥ H = 1 , 𝑠 > 0 (8) 𝑠 ∈ℝ 𝜓 ∈ N 𝐸 𝑎 − ∥ 𝑓𝑖 ∥ H − ∥ 𝑓 𝑗 ∥ H + ∥ 𝑓 𝑝 ∥ H
7.1.1. Optimal Direction We first focus on the inner optimization problem of (8). Expanding the squared numerator of the cost functional reveals that for a fixed magnitude 𝑠 > 0, minimizing the cost in 𝜓 is equivalent to 13
Hilbert Operator for Progressive Encoding (HOPE)
maximizing the alignment 𝜓, 𝑓𝑖 + 𝑓 𝑗 H in 𝜓. To solve the latter, we enforce the realizability 𝜓 ∈ N and unit-norm ∥ 𝜓 ∥ H = 1 constraints by decoupling the input and output parameters, yielding the parametric form: Ψ ( 𝒖 · 𝒙˜) 𝜓 = √︁ 𝒗. (9) 𝐾 ( 𝒖, 𝒖) Substituting this parametric form into the unconstrained alignment objective and distributing the Hilbert inner product via the kernel identity isolates the output direction 𝒗. By the Cauchy-Schwarz Í 𝑘 𝑘 inequality, the optimal 𝒗∗ must align with 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖∗ , 𝒘 . Substituting this optimal 𝒗∗ back ˜ in ) 𝒘out into the objective simplifies the alignment inner product to the Euclidean norm of that sum, yielding the final objective for the optimal 𝒖∗ (Theorem C.7): Í Í ∗ ˜ 𝑘 ) 𝒘𝑘 𝑘 𝑘 ∥ 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖, 𝒘 ˜ in ) 𝒘out ∥ 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖 , 𝒘 out in ∗ ∗ s.t. 𝐾 (𝒖, 𝒖) > 0 (10) 𝒗 = Í , 𝒖 = argmax √︁ ∥ 𝒖 ∥=1 𝑘 𝑘 ∗ ∥ 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖 , 𝒘 ˜ in ) 𝒘out ∥ 𝐾 ( 𝒖, 𝒖) The above optimization7 in 𝒖 generally lacks a closed-form solution due to the non-linear nature of the kernel 𝐾 . To maintain computational tractability, we introduce an approximation scheme that reduces the objective to an eigenvalue problem. Our approximation D E assumes that for any unit 𝒚 vector 𝒙 and non-zero 𝒚 , the kernel factors as 𝐾 ( 𝒙, 𝒚 ) = ∥ 𝒚 ∥ 𝑘 𝒙, ∥ 𝒚 ∥ for some angular function 𝑘 : [−1, 1] → ℝ bounded by 𝑘 ( 𝜌) ≤ 1. Additionally, we require 𝑘 (1) = 𝑘′ (1) and 𝑘 (1) > 0. These conditions naturally hold for all PH-1 functions (piecewise linear with a single knot at the origin), e.g., ReLU, Leaky-ReLU; see Propositions C.8 to C.10. For highly correlated neuron pairings, theEoptimal D 𝒘 ˜ in parent direction 𝒖 aligns closely with its children, pushing their cosine similarity 𝜌 ≜ 𝒖, ∥ 𝒘˜ in ∥ toward 1. Expanding 𝑘 ( 𝜌) to first order around 𝜌 = 1 and applying the 𝑘 (1) = 𝑘′ (1) identity cancels the constant terms, yielding the linear approximation 𝑘 ( 𝜌) ≈ 𝜌𝑘 (1). While this degrades for unaligned vectors, the phase-check provided later in the section corrects anti-alignment by flipping the sign of 𝒖, ensuring the optimization trajectory remains safely within this linear domain. Applying the linear approximation to the numerator of the objective, and defining the constant Í 𝑗 𝑗 𝑇 𝑘 𝑘 𝑖 𝑖 𝑇 matrix 𝑨 ≜ 𝒘out (𝒘 ˜ in ) + 𝒘out (𝒘 ˜ in ) , the summation factors neatly: 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖, 𝒘 ˜ in ) 𝒘out ≈ 𝑘 (1) 𝑨𝒖. Conversely, the denominator requires no approximation; because ∥ 𝒖 ∥ = 1, self-alignment 𝜌 = 1 evaluates to 𝐾 ( 𝒖, 𝒖) = 𝑘 (1). Substituting these into the original optimization problem gives: b 𝒖 = s.t. ∥ 𝒖 ∥ = 1. Because 𝑘 (1) > 0, the scalars pull out. Dropping these arg max𝒖 ∥ 𝑘√(1) 𝑨𝒖 ∥ 𝑘 (1)
constants and squaring the strictly non-negative objective simplifies the unconstrained problem to a 𝒖 = arg max𝒖 𝒖𝑇 𝑨𝑇 𝑨𝒖 s.t. ∥ 𝒖 ∥ = 1. The optimal direction b 𝒖 is simply standard quadratic form: b the principal eigenvector of 𝑨𝑇 𝑨. While explicitly constructing this ambient matrix is computationally prohibitive, 𝑨 is fundamentally rank-2. Restricting the eigendecomposition to this rank-2 subspace bypasses the ambient dimension entirely, yielding the principal eigenvector via a fast closed-form solution. Determining the Sign of 𝒖. For PH-1 activations, the kernel is sign-sensitive 𝐾 ( 𝒖, 𝒘 ˜ in ) ≠ 𝐾 (−𝒖, 𝒘 ˜ in ). However, because our linearization approximation relies on the leading eigenvector b 𝒖 of 𝑨𝑇 𝑨, we only recover the solution up to a sign ambiguity. We resolve this by evaluating both candidate polarities ±b 𝒖 in the exact, non-linearized objective (10): Í 𝑘 𝑘 ˜ in ) 𝒘out 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖, 𝒘 𝒖correct = argmax𝒖 ∈ {b . (11) √︁ 𝒖, −b 𝒖} 𝐾 ( 𝒖, 𝒖) 7 By the PH-1 property of Ψ, the mapping 𝒖 ↦→ 𝐾 ( 𝒖, 𝒖) is homogeneous, implying that the objective is invariant to the transformation 𝒖 ← 𝑐𝒖 for any 𝑐 > 0. We arbitrarily enforce ∥ 𝒖 ∥ = 1 to keep the problem well-posed.
14
Hilbert Operator for Progressive Encoding (HOPE)
7.1.2. Optimal Scale Recall from (9) that the unit-norm direction 𝜓 ∈ N is parameterized by unit vectors 𝒖 and 𝒗 as Ψ ( 𝒖 ·˜ 𝒙) 𝒗. Substituting 𝑓 = 𝑠𝜓 into the merging cost (8) and defining constants 𝑎 ≜ ∥ 𝑓𝑖 ∥ 2H + ∥ 𝑓 𝑗 ∥ 2H , 𝜓=√ 𝐾 ( 𝒖,𝒖 )
𝑏 ≜ 𝜓, 𝑓𝑖 + 𝑓 𝑗 H , and 𝐸rem ≜ 𝐸 𝑎 − ∥ 𝑓𝑖 ∥ H − ∥ 𝑓 𝑗 ∥ H , minimizing the squared objective reduces to the 1D 2
𝑏𝑠+𝑎 problem 𝑠∗ = argmin𝑠>0 2(𝑠𝑠+−2 . Setting the derivative with respect to 𝑠 to zero yields the unique 𝐸 )2 rem
𝑏𝐸rem minimizer 𝑠∗ = 2𝑎+𝐸rem +𝑏 . This solution is also stable. By definition, the residual capacity 𝐸rem ≥ 0, and the prior phase-check ensures the alignment in function space 𝑏 > 0. Thus, the denominator is strictly positive, guaranteeing a unique global minimum in the positive domain (simplifying cleanly to 𝑠∗ = 𝑎/𝑏 in the event of a total layer collapse where 𝐸rem = 0). Once the optimal scale 𝑠∗ > 0 is determined, the parent neuron is fully characterized as shown below.
Optimal Parent Neuron
𝑓 𝑝∗ (˜ 𝒙 ) = 𝑠∗ 𝜓∗ (˜ 𝒙)
,
Ψ ( 𝒖c · 𝒙˜) ∗ 𝒗 𝜓∗ (˜ 𝒙 ) = √︁ 𝐾 ( 𝒖c , 𝒖c )
,
∥ 𝑓𝑖 ∥ 2H + ∥ 𝑓 𝑗 ∥ 2H + 𝐸rem 𝜓∗ , 𝑓𝑖 + 𝑓 𝑗 H
𝑠∗ =
2 𝐸rem + 𝜓∗ , 𝑓𝑖 + 𝑓 𝑗 H (12)
𝒖c = argmax𝒖 ∈ {b 𝒖, −b 𝒖}
∥
Í
𝑘 𝑘 ˜ in ) 𝒘out ∥ 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖, 𝒘
√︁
𝐾 ( 𝒖, 𝒖)
b 𝒖 = arg max
∥ 𝒖 ∥=1
,
Í
∗
𝒗 = Í 𝑗
𝑘 𝑘 ˜ in ) 𝒘out 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖c , 𝒘
∥
𝑘 𝑘 ˜ in ) 𝒘out ∥ 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖c , 𝒘
𝑗
𝑖 𝑖 𝑇 ˜ in )𝑇 𝒖 . 𝒘out (𝒘 ˜ in ) + 𝒘out ( 𝒘
(13) (14)
7.2. From Hilbert Space to Physical Parameters This section bridges the abstract function space and the physical parameter space by mapping the mathematical operator derived in H back into physical parameters. This parameter recovery is only necessary for merging. For pruning, the projection target is simply the null operator 0, which leads to 𝑓 ( 𝒙 ) = 0; this is trivially realized by zeroing out the neuron’s incoming weights, outgoing weights, and BN parameters. However, deploying the parent neuron 𝑓 𝑝∗ ∈ H derived in (12) requires determining the physical parameters (weights 𝒘raw 𝑝 , 𝑏 𝑝 , 𝒘 𝑝,out and BN statistics 𝛽 𝑝 , 𝛾 𝑝 , 𝜇 𝑝 , 𝜎 𝑝 ) that will configure the forward pass to reproduce its targeted non-zero activation profile. 7.2.1. Input/Output Scaling ∗ Ψ (𝒘 ∗ · 𝒙 To form a standard realizable neuron as described in (7), we equate 𝑓 𝑝∗ (˜ 𝒙 ) = 𝒘out ˜ in ˜) and ∗ ∗ then specify the parameters 𝒘 ˜ in and 𝒘out . Because the PH-1 activation Ψ exhibits scale symmetry, √︁ ∗ = 𝑠 𝒖∗ and the amplitude 𝑠∗ / 𝐾 ( 𝒖∗ , 𝒖∗ ) can be factored into arbitrary input and output scales 𝒘 ˜ in in √︁ ∗ = 𝑠 𝒗∗ , for any 𝑠 , 𝑠 ∗ / 𝐾 ( 𝒖∗ , 𝒖∗ ). While any factorization yields the 𝒘out ≥ 0 satisfying 𝑠 𝑠 = 𝑠 out in out in out same mapping X → ℝ𝑐 , amplitude distribution impacts fine-tuning dynamics. To preserve the original 𝑗 𝑖 layer’s balance, we define the subspace Frobenius ratio 𝑅 𝐹 ≜ ∥𝑾in ∥ 𝐹 /∥𝑾out ∥ 𝐹 , where 𝑾in = [𝒘 ˜ in |𝒘 ˜ in ] 𝑗 𝑖 ∗ ∗ and 𝑾out = [𝒘out | 𝒘out ]. Constraining the parent neuron to this ratio requires ∥ 𝒘 ˜ in ∥ 2 /∥ 𝒘out ∥ 2 = 𝑅 𝐹 . Since 𝒖∗ and 𝒗∗ are unit vectors, 𝑠in /𝑠out = 𝑅 𝐹 . This uniquely determines the scale factors, yielding the final parameters: √︂ √ 𝑠∗ −1/4 ∗ −1/4 ∗ ∗ ∗ ∗ 𝒘 ˜ in = 𝑠 𝑅 𝐹 · 𝐾self 𝒖 , 𝒘out = · 𝐾self 𝒗 , 𝐾self ≜ 𝐾 ( 𝒖∗ , 𝒖∗ ) . (15)
𝑅𝐹
15
Hilbert Operator for Progressive Encoding (HOPE)
7.2.2. Raw Input and BN Parameters eff , 𝑏), While the Hilbert space formulation operates entirely on the effective input parameters 𝒘 ˜ in ≜ ( 𝒘in realizing the physical network requires recovering the underlying physical parameters: 𝒘raw , 𝛽, 𝛾, 𝜇, and 𝜎. Since the parent direction b 𝒖 lies within the 2D subspace spanned by the augmented children, there exist projection coefficients 𝑐1 and 𝑐2 that produce the effective parameters: eff eff 𝒘eff 𝑝,in = 𝑐1 𝒘in,𝑖 + 𝑐2 𝒘in, 𝑗
and
𝑏 𝑝 = 𝑐1 𝑏 𝑖 + 𝑐2 𝑏 𝑗 .
(16)
By mapping these coefficients through the pre-activation distributions of the children, we can deduce the required BN statistics for the parent neuron. Because the physical BN equations form an underconstrained system, we resolve the ambiguity by anchoring the variance such that 𝜎2𝑝 = max(0, 𝛾 2𝑝 − 𝜖). As rigorously derived in Appendix D, this anchoring yields a closed-form recovery of all physical parameters. For the active regime 𝛾 2𝑝 ≥ 𝜖, these evaluate to: raw eff 𝒘in ,𝑝 = 𝒘in,𝑝
𝛽 𝑝 = 𝑐1 𝛽 𝑖 + 𝑐2 𝛽 𝑗
,
,
𝜇 𝑝 = 𝑐1 𝛽 𝑖 + 𝑐2 𝛽 𝑗 − 𝑏 𝑝
𝜎𝑝 = 𝛾𝑝 =
√︃
(17)
𝑐12 𝛾𝑖2 + 𝑐22 𝛾 2𝑗 + 2𝑐1 𝑐2 | 𝛾𝑖 || 𝛾 𝑗 | 𝜌 ˆ𝑖 𝑗 .
(18)
where 𝜌ˆ𝑖 𝑗 is from (4). Note that 𝜎 𝑝 ≈ 𝛾 𝑝 is an approximation that assumes the numerical stability constant 𝜖 is negligible. The exact boundary-safe formulation 𝜎2𝑝 = max(0, 𝛾 2𝑝 − 𝜖) and the edge case for inactive features 𝛾 2𝑝 < 𝜖 are deferred to Appendix B.5. Furthermore, Appendix D provides the full step-by-step derivation, along with a proof demonstrating that the physical forward pass acts as a self-correcting mechanism that ensures the network’s mapping remains invariant to the sign of the recovered scale 𝛾 𝑝 .
8. Block Eviction This section expands the granular compression cost Jbound established in Section 6 to a new macrolevel operation: block eviction. Focusing on residual blocks in architectures like ResNet-50, we extend the previously developed continuous integral to evaluate block eviction alongside granular operations within a single, unified mathematical framework. Consider the canonical residual block, which processes an input representation 𝑋 (capitalized to distinguish it from the flattened vector 𝒙 ) through a three-stage mapping pathway 𝐹 ( 𝑋 ). This pathway sequentially applies weight parameters 𝑊1 , 𝑊2 , and 𝑊3 , and the result is added to a skip connection to yield the final pre-activation 𝑌 = 𝑋 + 𝐹 ( 𝑋 ). We define Block Eviction as forcing 𝐹 ( 𝑋 ) → 0, and thus collapsing the block into a pure identity mapping 𝑌 = 𝑋 (see Figure 3). Skip Connection (Preserves 𝑌 = 𝑋 ) Residual Function 𝐹 ( 𝑋 )
𝑋
𝑊1 1x1 Conv (Reduction)
BN
ReLU
𝑊2 3x3 Conv
BN
(Spatial)
ReLU
𝑊3 1x1 Conv
𝑌
BN
+
ReLU
𝑍
(Expansion)
Targeted for Block Eviction: 𝐹 ( 𝑋 ) → 0
Figure 3 | Illustration of the canonical ResNet V1 residual block and its eviction process. Eviction forces the internal pathway 𝐹 ( 𝑋 ) → 0, collapsing the block into a pure identity pre-activation 𝑌 = 𝑋 .
16
Hilbert Operator for Progressive Encoding (HOPE)
8.1. Motivation A dedicated macro-level operation is required because standard granular pruning cannot remove the block’s final layer 𝑊3 . The output dimensionality of 𝐹 ( 𝑋 ) must match the skip connection 𝑋 for element-wise addition. Consequently, granular compression can only deplete the internal layers 𝑊1 , 𝑊2 , which leaves the output channels of 𝑊3 locked at their ambient size. Leaving a residual pathway active under these conditions creates two issues: • Model Generalization: When 𝑊1 and 𝑊2 are heavily depleted, the pathway functionally reduces to injecting an uncalibrated BN effective bias 𝐵eff into the skip connection 𝑌 = 𝑋 + 𝐵eff . This shifts downstream feature maps out of their calibrated domain, often causing catastrophic ReLU clipping and irreversible information loss when going from 𝑌 to 𝑍 . • Execution Efficiency: Retaining the massive 𝑊3 parameter tensor simply to process a negligible subspace violates the core objective of compression. Block eviction resolves both issues by projecting the pathway 𝐹 ( 𝑋 ) to the null operator. By yielding a pure identity mapping 𝑌 = 𝑋 , we avoid uncalibrated bias injection and leverage the fact that residual architectures are inherently designed to be robust to identity mappings (e.g., standard 𝛾 = 0 initialization practices) [Goyal et al., 2017, He et al., 2016]. Full mathematical details of this degradation are provided in Appendix F. 8.2. The Unified Macro Cost Jevict To evaluate this macro-operation within our framework, we must expand our definition of layer state. To see why, observe that Axiom 2 imposes an infinite cost penalty on projecting an entire layer to zero to prevent disconnecting the network graph. However, this penalty creates an artificial barrier here, as the parallel identity mapping preserves overall connectivity of the block and keeps it alive. To account for this skip pathway, we formulate a macroscopic state Ω ( 𝑙 ) ≜ ( Φ ( 𝑙 ) , I) that couples the targeted internal layer Φ ( 𝑙 ) with the ambient skip connection I. The skip connection provides a parallel survival capacity 𝐸identity that keeps the mathematical projection stable. As rigorously derived in Appendix F, integrating the continuous capacity cost over this macro-state and applying a linear upper bound to safely govern massive discrete architectural leaps yields a closed-form distortion criterion. For a standard residual bottleneck comprising two internal convolution layers 𝑙 ∈ {1, 2}, the total macroscopic distortion is the linear sum of their independent projection bounds: ! (𝑙) 2 2 ∑︁ ∑︁ 𝐸active (𝑙) (𝑙) (𝑙) Jevict = Jlayer ( Ω𝑎 , Ω𝑏 ) = 𝑁active . (19) 𝑙 =1
𝐸identity
𝑙 =1
(𝑙) (𝑙) Here, 𝑁active and 𝐸active represent the active operator count and surviving capacity of internal layer 𝑙 , respectively. The parallel survival capacity evaluates to the expected RMS energy of the identity Í𝑑amb √︃ 2 operators conditioned by the preceding BN layer: 𝐸identity = 𝑘=1 𝛾𝑘 + 𝛽𝑘2 .
ResNet Block Eviction Cost (𝑙) (𝑙) 𝑙 =1 𝑁active 𝐸active
Í2 Jevict =
Í𝑑amb √︃ 2 𝑘=1
𝛾𝑘
.
(20)
+ 𝛽2 𝑘
17
Hilbert Operator for Progressive Encoding (HOPE)
9. Balancing Compression and Distortion All cost functionals discussed thus far (Jprune and Jmerge for granular reductions, and Jevict for block evictions) measure the projection error incurred when transitioning from a given state to a reduced state. However, rate-distortion theory establishes that distortion alone cannot fully characterize a lossy compression scheme: lower signal distortion requires a higher bit count, while stronger compression inevitably increases distortion. To balance these competing objectives, we aim to minimize total distortion under a fixed bit count budget. Progressive compression is therefore formulated as a trajectory planning problem within the action space. The goal is to craft a sequence of compression operations that yields a final model satisfying the allowable bit budget while minimizing the total accumulated distortion along the trajectory. Solving this represents a highly complex planning problem due to two primary challenges: 1. Dynamic State Dependency: The cost of an action changes continuously as the model transitions between states. For example, pruning a single neuron shrinks the layer’s residual capacity, which instantaneously alters the cost J of subsequent operations, such as pruning another neuron or evicting an entire block. Consequently, the mathematical cost landscape is constantly shifting. 2. Mutually Exclusive Actions: The action space contains complex combinatorial dependencies. If the optimizer merges Neuron A with Neuron B, independent actions like “Prune A” or “Merge A with C” become permanently invalid. For computational tractability, we must relax these constraints. At each iteration, we temporarily assume all currently admissible operations will remain valid for future iterations, ignoring their mutually exclusive nature. While this generates a complete theoretical action sequence, executing the full trajectory would introduce compounding errors in both state transitions and capacity counts. Instead, we adopt a receding-horizon strategy [Camacho and Bordons, 2013, Bertsekas, 2012]: we compute the optimal sequence, but execute only the immediate next action. Then we physically update the network and then re-evaluate all admissible functions from scratch. This single-step execution acts as an inherent auto-correction mechanism that ensures adherence to constraints over each short-term step. Formally, let A = {1, 2, . . . , 𝐾 } denote the set of all admissible compression operations at the current encoding iteration, encompassing all feasible granular and macro operations. Each action 𝑘 incurs a distortion penalty J𝑘 and releases Δ𝑃𝑘 parameters (see Appendix B.2 for details on computing Δ𝑃 ). Assuming a standard fixed-precision representation (e.g., 32-bit floating-point), bit reduction is proportional to parameter reduction. This direct scaling allows us to express the allowable budget directly in terms of the parameter footprint. We frame this optimization as: ( 𝑎∗1 , · · · , 𝑎∗𝐾 ) = arg min
𝑎1 , ··· ,𝑎 𝐾
𝐾 ∑︁ 𝑘=1
𝑎𝑘 J𝑘
s.t.
𝐾 ∑︁
𝑎𝑘 Δ𝑃 𝑘 ≥ 𝑃0 − 𝑃budget
,
∀𝑘 ; 𝑎𝑘 ∈ {0, 1} .
(21)
𝑘=1
where 𝑃0 is the initial parameter count and 𝑃budget is the maximum allowable parameter footprint for the final model. This formulation is a discrete knapsack problem, which is well-known to be NPHard [Karp, 1972, Garey and Johnson, 1979]. We resolve this using a continuous relaxation heuristic, replacing the binary constraint 𝑎𝑘 ∈ {0, 1} with a continuous bound 0 ≤ 𝑎𝑘 ≤ 1. This transforms the objective into a continuous knapsack problem (a specific class of linear programming) that admits a highly efficient analytical solution. As established by Dantzig [Dantzig, 1957], the exact optimal solution is found greedily: candidates are sorted by their distortion rate (DR), defined as the cost-to-capacity ratio J𝑘 /Δ𝑃 𝑘 , and assigned 𝑎𝑘 = 1 in ascending order until the budget constraint is 18
Hilbert Operator for Progressive Encoding (HOPE)
saturated. Because our receding-horizon framework executes only the single next action, the problem reduces to selecting the operation with the minimal DR: 𝑘∗ = arg min
J𝑘
𝑘 ∈ A Δ𝑃 𝑘
(22)
.
While the receding-horizon strategy mitigates the dynamic dependency of J , the continuous knapsack solver still requires Δ𝑃 to satisfy Dantzig’s Axiom of Item Independence: the weight of one item cannot depend on the selection state of another. Particularly in our problem, evaluating operations using the dynamically shrinking live parameter footprint Δ𝑃 violates this axiom because adjacent layers share weight matrices; pruning a neuron physically shrinks the Δ𝑃 of its neighbors. A naive optimization using this live Δ𝑃 triggers a failure mode: as a layer is compressed, the expected DR of neighboring structures artificially inflates. This repels the optimizer and may trap the architecture in a fragmented state that prevents the removal of contiguous blocks. Decoupling parameter yield from dynamic state via the static surrogate Δ𝑃𝑘init restores item independence and avoids this failure mode: Action Selection Criterion 𝑘∗ = arg min
J𝑘
𝑘 ∈ A Δ𝑃 init 𝑘
.
(23)
10. The Encoding Loop With the optimal action selection now formally defined, we execute progressive encoding as a greedy dynamical system. At a high level, the algorithm continuously identifies the optimal action 𝑘∗ offering the lowest DR J𝑘 /Δ𝑃 𝑘init , performs a localized recalculation exclusively for the modified structures (e.g., a newly generated parent neuron 𝑓 𝑝∗ ) and their immediate neighbors, decrements the relevant dimension count, and repeats. Specifically, the process operates in the following three phases and terminates once the target physical parameter budget is reached or no admissible compression operations remain: 1. Initialization: Before compression begins, the algorithm precomputes and caches the individual capacities of all neurons, the pairwise geometric cross-capacities of all valid merging pairs, and the total initial capacity of every layer (establishing the starting value for 𝐸rem ). 2. The Greedy Scan: At each iteration, the algorithm scans all 𝐿 layers (each containing roughly 𝑁 active neurons) to find the single optimal compression action 𝑘∗ yielding the lowest DR J𝑘 /Δ𝑃 𝑘init . For pruning, evaluating every individual candidate across the network requires O ( 𝐿 · 𝑁 ) operations. For merging, evaluating every valid pair requires checking 𝑁 ( 𝑁2−1) combinations per layer, leading to O ( 𝐿 · 𝑁 2 ) operations. Because querying the cached J for each candidate takes O (1) time8 , the total computational complexity to find the optimal action at any step is bounded by the pairwise merge evaluations at O ( 𝐿 · 𝑁 2 ).
3. Localized Update: Once the globally optimal action is identified and executed, the network state must be synchronized. The algorithm decrements the layer’s 𝐸rem by the capacity flux removed by the operation, and decrements the neuron count 𝑁 . If the action was a merge, the algorithm also 8 As established in the practical notes of Section 6, the distortion cost J relies on local variables: the capacity of the
targeted neurons and the remaining capacity of their specific layer. Because evaluating J does not require querying the global network state, calculating the DR of any individual prune, merge, or block eviction operation is O (1).
19
Hilbert Operator for Progressive Encoding (HOPE)
computes the capacity of the newly generated parent neuron 𝑓 𝑝∗ and calculates the cross-capacities as well as optimal projection vectors) between this new parent and the 𝑁 − 1 surviving neighbors in its layer. These updated constants are injected into the cache, guaranteeing that the evaluation of J during subsequent greedy scans remains O (1). This limits the network state recalculation to an O ( 𝑁 ) local update.
11. Proof-of-Concept Applications 11.1. Model Compression Because its encoding is progressive, any intermediate iteration serves as a valid compressed model, providing users with flexible trade-offs between compression rate and fidelity. Taxonomically, HOPE is a structured method: it eliminates entire neurons rather than zeroing out individual weights. This provides greater practical utility than unstructured pruning, which generates randomly sparse matrices requiring specialized hardware to realize actual computational speedups. We compare HOPE against three structured baselines that eliminate neurons below specific magnitude thresholds: 𝐿1 Norm Input Pruning [Liu et al., 2017] (scored by incoming weight 𝐿1 norms); 𝐿1 -Norm Joint Pruning (scored by concatenated incoming and outgoing 𝐿1 norms); and BN Scale Pruning [Liu et al., 2017] (using the BN scaling factor 𝛾 as a proxy for importance). Our experiments investigated the relationship between test set accuracy and model density (defined as the ratio of active to initial neurons across the entire network). For our compression assessment, we utilized Keras’ publicly available ResNet-50 model checkpoint that is pre-trained on ImageNet. As demonstrated in the plot, HOPE yields models with superior accuracy compared to the baselines. 11.2. Cross-Domain Transfer Learning HOPE’s capacity evaluation can be used for resolving the stability-plasticity dilemma in transfer learning. By merging redundancies, we can partition the network into a protected core and a plastic periphery and leverage it for parameter-efficient transfer. 11.2.1. The Stability-Plasticity Dilemma and Current Bottlenecks Intelligent systems face a fundamental challenge: adapting to new domains without erasing foundational knowledge, a trade-off known as the Stability-Plasticity dilemma [Grossberg, 1987]. Cognitive neuroscience models this via Complementary Learning Systems [McClelland et al., 1995, Kumaran et al., 2016], proposing that the brain insulates a stable, domain-specific core from a plastic periphery, as empirically supported by recent neuroimaging [Billot et al., 2024, Blank et al., 2016, Casto and Fedorenko, 2026]. Specifically, the brain extracts generalizable schemata from noisy state transitions by applying low-dimensional regularization to its representational geometry [Kimmel et al., 2026]. During standard training, deep neural networks self-organize into a similar, albeit noisy and imperfect, dichotomy [Martin and Mahoney, 2021]. They develop a sparse, load-bearing core surrounded by low-capacity representational slack [Frankle and Carbin, 2019, El Cheairi et al., 2026]. Naively 20
Hilbert Operator for Progressive Encoding (HOPE)
exploiting this emergent segregation by freezing the core fails; because layers remain entangled, peripheral updates shift activation flows, causing representational drift. Early solutions like PackNet [Mallya and Lazebnik, 2018] circumvented this using binary masks during inference, but these scale poorly and require a priori task identities.
Robust continual learning requires explicit interventions, rather than relying on noisy emergent segregation. During source training, regularization [Wen et al., 2016, Scardapane et al., 2017] can amplify the segregation. During downstream adaptation, penalty-based methods like EWC [Kirkpatrick et al., 2017], Synaptic Intelligence [Zenke et al., 2017], and second-order pruning [LeCun et al., 1989 Hassibi and Storkey, 1992, Singh and Alistarh, 2020] prevent drift using locally convex Fisher Information Matrices (FIMs) or Hessians. However, relying on local approximations makes these algorithms brittle to large domain shifts. Conversely, orthogonal projection methods stop drift by restricting updates to the null-space of previous tasks [Jaeger, 2014, Saha et al., 2021, Zeng et al., 2019, Wang et al., 2021, Yang et al., 2025, HuggingFace Research Team et al., 2026]. However, they remain computationally expensive due to the O ( 𝑁 3 ) operations and source forward passes needed to compute covariance matrices. 11.2.2. DEFT To address the above challenges, we introduce Dispersed Elastic Fine-Tuning (DEFT). Aligning with the Information Bottleneck principle [Tishby et al., 1999], DEFT treats learning as the compression of irrelevant slack space via global Hilbert-Schmidt operators, bypassing local loss curvature and empirical data passes. Leveraging the HOPE framework, DEFT analytically computes each neuron’s capacity in O ( 𝑁 ) time to partition the network into a Universal Core and a Peripheral Slack. To prevent representational drift, DEFT severs weight projections from the slack to the core prior to transfer. This ensures the core remains frozen while the slack adapts, eliminating the need for inference-time masking or task identities. DEFT governs parameter plasticity through a binary elasticity map 𝐸 ∈ {0, 1}. While prior methods also regulate plasticity [Zhou et al., 2026], their reliance on weight sensitivity leaves them vulnerable to the scaling symmetries that HOPE mitigates. We formalize this by evaluating the pruning cost assigned to each neuron 𝑖 upon its removal during HOPE’s progressive encoding process: (𝑖) Jprune =
𝑁 ( 𝑖 ) · ∥ 𝑓𝑖 ∥ H (𝑖)
(24)
𝐸𝑏
where 𝑁 ( 𝑖 ) and 𝐸𝑏( 𝑖 ) ≜ 𝐸 𝑎( 𝑖 ) − ∥ 𝑓𝑖 ∥ H denote the active neuron count and the remaining layer capacity, at the step neuron 𝑖 is pruned. To establish a global freezing threshold, we collect the set of all such costs across the entire encoding process, and filter out extinction artifacts resulting from near-zero capacities: n o (𝑖) (𝑖) C = Jprune 𝐸𝑏 > 𝜖 (25) Given a target percentile hyperparameter 𝑃 ∈ [0, 100], we compute the threshold 𝐽𝑃 = Percentile(C , 𝑃 ) and the supremum 𝐽sup = max(C). For numerical stability against edge capacity regimes, the final locking threshold 𝐽lock is defined: 𝐽 𝑃 𝐽lock = 𝐽sup 1
if 𝐽𝑃 ≥ 𝜖 if 𝐽𝑃 < 𝜖 and 𝐽sup ≥ 𝜖 otherwise
(26)
21
Hilbert Operator for Progressive Encoding (HOPE)
The elasticity of neuron 𝑖 is formulated as: ( 𝐸𝑖 =
1 0
(𝑖) if Jprune < 𝐽lock (𝑖) if Jprune ≥ 𝐽lock
(27)
(𝑖) Under this formulation, high capacity neurons essential to the source architecture Jprune ≥ 𝐽lock are frozen by 𝐸 𝑖 = 0, whereas low-capacity slack neurons are granted high plasticity via 𝐸 𝑖 = 1.
Dynamic Resolution of Redundancy: Deep networks frequently fragment a single feature across multiple correlated neurons. If we freeze the network based on a static capacity threshold, we incorrectly lock up this redundant volume and deprive the target task of parameter space. As illustrated in Figure 4(a), DEFT resolves this by compressing these redundant features into a single rank-1 parent neuron. This consolidates the foundational source knowledge while releasing the freed child neurons into the plastic slack 𝐸 𝑖 = 1. By transforming redundant copies into uncommitted parameter space, DEFT actively generates capacity for the target task. Frozen Core
Plastic Slack
Parent
𝑓1
𝑀
Copies
𝑓2
𝑓3
MERGE Í
Error ≤
Layer 𝑙
Core
Layer 𝑙 + 1
𝑤core→core
t
da
re ve Se
Empty 𝐶𝑖
Empty
(a) Dynamic Resolution of Redundancy
Core
0 𝑡=
Slack
𝑤slack→slack
Slack
(b) The Structural Mask
Figure 4 | The algorithmic mechanisms of DEFT. (a) Redundant features within the frozen core are compressed to generate new elastic target capacity. (b) A structural mask permanently severs cross-connections to protect the core from target-driven drift. Consistency at Initialization (The Structural Mask): To prevent target-driven updates of the plastic slack from corrupting the frozen core, DEFT applies a structural mask at initialization (Figure 4b). It severs all connections pointing from upstream plastic neurons to downstream frozen core neurons. For a weight tensor connecting an upstream layer (with elasticities 𝑬in ) to a downstream layer (with elasticities 𝑬out ), the mask 𝑴 enforces: ( 0 if 𝐸in,𝑘 > 0 and 𝐸out, 𝑗 = 0 𝑀 𝑗,𝑘 = (28) 1 otherwise The initial weights for the target task are thus constrained to 𝑾0 = 𝑴 ⊙ 𝑾source . Theoretical Guarantees: We prove in Appendix H that these mechanisms protect the source representation through a layer-to-layer bounding framework. First, the Static Initialization Shock Bound establishes that severing the slack-to-core connections introduces a static error strictly bounded by O ( 𝜏). Second, Dynamic Decoupling guarantees the core experiences zero dynamic interference during target fine-tuning. Because the cross-connections are severed at initialization and their weights are frozen from updating, any drifting signal from the learning slack subset is multiplied by zero, nullifying it before it can penetrate the core. Combined with the bounded projection errors of the merging operation, this framework ensures the cumulative degradation of the source task cannot compound exponentially, remaining anchored to an algorithmically verifiable constant. 22
Hilbert Operator for Progressive Encoding (HOPE)
Gradient Scaling: During optimization, the target gradients are element-wise scaled by the downstream neuron’s elasticity 𝑬out (uniformly broadcast across the input channels): 𝒈𝑡 = 𝑬out ⊙ ∇𝑾 Ltarget (𝑾𝑡 )
(29)
where 𝒈𝑡 represents the effective gradient passed to the optimizer state9 , e.g., SGD with momentum. 11.2.3. Experimental Setup To evaluate the stability-plasticity tradeoff, we rank methods using the H-Score commonly used in continual learning [Qiu et al., 2024, Xie et al., 2025, Islam et al., 2025]. H-Score, defined as the harmonic mean of source retention and target accuracy, heavily penalizes poor performance in either domain. This ensures a high score is achieved only when a model excels on both tasks. We evaluate models pre-trained on multi-class classification tasks derived from the CIFAR-100 dataset [Krizhevsky, 2009]. Each source task is constructed by randomly sampling 4 superclasses, which yields 20 fine-grained categories (5 per superclass). Building the source task around dense semantic clusters rather than sampling 20 arbitrary classes forces the network to learn hierarchical features to distinguish closely related concepts. We then transfer these specialized models to the full 10-class digit classification task in the SVHN dataset (street-level house numbers) [Netzer et al., 2011]. As summarized in Table 1, we benchmark DEFT against the following baseline methods: • Standard Full FT: The entire pre-trained backbone is unfrozen, allowing the optimizer to alter representations across all layers. While maximizing target plasticity, it is highly susceptible to catastrophic forgetting. • Head-Only FT (Standard FT): Representing the opposite extreme (linear probing), the pretrained backbone is completely frozen, acting as a static feature extractor. Only the final linear classification head is optimized. • PEFT (BN-Tuning): Operating on the premise that spatial feature extraction logic should remain invariant [Frankle et al., 2021], this method applies a binary gradient mask: core convolutional and dense kernels are frozen, while plasticity is isolated entirely to the affine BN parameters (scale 𝛾 and shift 𝛽 ) and the newly initialized classification head. • EWC (Elastic Weight Consolidation): Allows all parameters to update but applies a quadratic penalty constraining parameters deemed critical to the source task [Kirkpatrick et al., 2017]. To accurately lock foundational features, the empirical diagonal Fisher Information Matrix (FIM) is computed strictly over the source training dataset. See Appendix G for the per-example FIM derivation and integration protocol. Table 1 provides a conceptual summary of these methodologies. Full details regarding network architecture, hyperparameter optimization, and reproducibility protocols are provided in Appendix G. 11.2.4. Results and Discussion The final test set evaluations across 4 independent cross-domain trials (20 discrete CIFAR-100 → SVHN scenarios) are presented in Table 2. While Standard Full FT achieves the highest target performance 94.09% by freely overwriting network weights, it completely destroys the source representation, crashing source retention to a baseline low of 7.52%. Conversely, Head-Only FT best preserves source knowledge, but its 36.11% target accuracy highlights the severe domain gap; pre-trained features are insufficient to separate SVHN digits linearly. EWC behaves remarkably similarly to 9 Scaling the gradient before the optimizer step prevents velocity drift for frozen parameters.
23
Hilbert Operator for Progressive Encoding (HOPE)
Table 1 | Comparative Summary of Transfer Learning Methodologies Property Updates Backbone Features Mitigates Catastrophic Forgetting Parameter-Specific Modulation Source Data Independence Structure & Redundancy Aware
Full FT
Head-Only
PEFT
EWC
DEFT (Ours)
✓ ✓ -
✓ ✓ -
✓ ✓ -
✓ ✓ ✓ -
✓ ✓ ✓ ✓ ✓
Full FT: it learns the target domain (93.94%) but fails to significantly arrest catastrophic forgetting (6.74%). DEFT successfully bridges this stability-plasticity gap. By routing target gradients into elastic neurons characterized by their low capacity, DEFT captures nearly all the plasticity of Standard Full FT (94.09% vs. 89.79%). Simultaneously, by masking the core foundational features, it halts catastrophic forgetting, retaining 52.14% of the source accuracy. Computing the harmonic mean of the two accuracy values leads to the H-Score of method, in which DEFT significantly outperform all the baselines. Table 2 | Cross-Domain Transfer Learning Results (4 Trials, 5 tasks each). Metrics represent Test Set accuracy evaluated at the optimal target validation epoch. All metrics are averaged across all tasks and trials. Method
Target Acc (SVHN)
Source Retention (CIFAR)
H-Score
89.79 ± 0.84 36.11 ± 2.79 94.09 ± 0.28 93.94 ± 0.22 81.91 ± 0.49
52.14 ± 5.29 63.13 ± 4.62 7.52 ± 1.63 6.74 ± 1.74 5.44 ± 0.98
65.82 ± 3.96 45.79 ± 2.05 13.88 ± 2.84 12.54 ± 2.99 10.18 ± 1.63
DEFT (Ours) Head-Only Full FT EWC PEFT
12. Acknowledgment We thank Juno Kim, Vaishnavh Nagarajan, Atish Agarwala, Spencer Frei, Lisa Schut, Alan Malek, Gil shamir, and Bruno Mlodozeniec of Google DeepMind for their helpful comments and discussions.
24
Hilbert Operator for Progressive Encoding (HOPE)
References [Ainsworth et al., 2023] Ainsworth, S. K., Hayase, J., and Srinivasa, S. (2023). Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations. [Andrei et al., 2023] Andrei, A., Akil, A., Kharas, N., Rosenbaum, R., Josić, K., and Dragoi, V. (2023). Rapid compensatory plasticity revealed by dynamic correlated activity in monkeys in vivo. Nature Neuroscience, 26(11):1960–1969. [Badrinarayanan et al., 2015] Badrinarayanan, V., Mishra, B., and Cipolla, R. (2015). Understanding symmetries in deep networks. arXiv preprint arXiv:1511.01029. [Bau et al., 2017] Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. (2017). Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541–6549. [Behrouz et al., 2026] Behrouz, A., Hashemi, F., Javanmard, A., and Mirrokni, V. (2026). Language models need sleep: Learning to self-modify and consolidate memories. [Bertsekas, 2012] Bertsekas, D. P. (2012). Dynamic programming and optimal control: Volume I. Athena scientific. [Billot et al., 2024] Billot, A., Jhingan, N., Varkanitsa, M., Blank, I., Ryskin, R., Kiran, S., and Fedorenko, E. (2024). The language network ages well: Preserved selectivity, lateralization, and within-network functional synchronization in older brains. bioRxiv. [Blalock et al., 2020] Blalock, D., Gonzalez Ortiz, J. J., Frankle, J., and Guttag, J. (2020). What is the state of neural network pruning? In Proceedings of Machine Learning and Systems, volume 2, pages 129–146. [Blank et al., 2016] Blank, I., Kanwisher, N., and Fedorenko, E. (2016). A functional mri investigation of the language network. Journal of Neurophysiology, 116(4):1968–1984. [Bricken et al., 2023] Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., et al. (2023). Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread. [Cai et al., 2020] Cai, Y., Yao, Z., Dong, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. (2020). Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169–13178. [Camacho and Bordons, 2013] Camacho, E. F. and Bordons, C. A. (2013). Model predictive control. Springer Science & Business Media. [Casto and Fedorenko, 2026] Casto, E. and Fedorenko, E. (2026). Cerebellar language hubs: Functional segregation of linguistic processing from motor and cognitive domains. Nature Neuroscience, 29(2):145–158. [Cho and Saul, 2009] Cho, Y. and Saul, L. (2009). Kernel methods for deep learning. In Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C., and Culotta, A., editors, Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc. [Dandi et al., 2025] Dandi, Y., Pesce, L., Zdeborova, L., and Krzakala, F. (2025). The computational advantage of depth in learning high-dimensional hierarchical targets. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. 25
Hilbert Operator for Progressive Encoding (HOPE)
[Dantzig, 1957] Dantzig, G. B. (1957). Discrete-variable extremum problems. Operations research, 5(2):266–288. [Delétang et al., 2023] Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J. (2023). Language modeling is compression. arXiv. [Diaconis and Freedman, 1984] Diaconis, P. and Freedman, D. (1984). Asymptotics of graphical projections. The Annals of Statistics, pages 793–815. [Dinh et al., 2017] Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. (2017). Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019–1028. PMLR. [El Cheairi et al., 2026] El Cheairi, H., Gamarnik, D., and Mazumder, R. (2026). Theoretical compression bounds for wide multilayer perceptrons. In 39th Annual Conference on Learning Theory, pages 1–59. Proceedings of Machine Learning Research. [Engels et al., 2024] Engels, J., Liao, I., Michaud, E. J., Gurnee, W., and Tegmark, M. (2024). Not all language model features are linear. arXiv preprint arXiv:2405.14860. [Entezari et al., 2022] Entezari, R., Sedghi, H., Saund, O., and Neyshabur, B. (2022). The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations. [Fan et al., 2020] Fan, A., Grave, E., and Joulin, A. (2020). Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations (ICLR). [Frankle and Carbin, 2019] Frankle, J. and Carbin, M. (2019). The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations. [Frankle et al., 2021] Frankle, J., Schwab, D. J., and Morcos, A. S. (2021). Training batchnorm and only batchnorm: On the expressive power of random features in cnns. In International Conference on Learning Representations. [Garey and Johnson, 1979] Garey, M. R. and Johnson, D. S. (1979). Computers and intractability: A guide to the theory of NP-completeness. W. H. Freeman and Company. [Genewein et al., 2026] Genewein, T., Grau-Moya, J., Wenliang, L. K., Orseau, L., and Hutter, M. (2026). Algorithmic compression via pretrained neural networks. Entropy, 28:596. [Goyal et al., 2017] Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. (2017). Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677. [Grossberg, 1987] Grossberg, S. (1987). Competitive learning: From interactive activation to adaptive resonance. Cognitive Science, 11(1):23–63. [Han et al., 2015] Han, S., Pool, J., Tran, J., and Dally, W. (2015). Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, volume 28. [Hanin and Rolnick, 2018] Hanin, B. and Rolnick, D. (2018). How to start training: The effect of initialization and architecture. In Advances in Neural Information Processing Systems, pages 571–581. 26
Hilbert Operator for Progressive Encoding (HOPE)
[Hassibi and Storkey, 1992] Hassibi, B. and Storkey, D. G. (1992). Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, volume 5. [He et al., 2016] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Identity mappings in deep residual networks. In European conference on computer vision (ECCV), pages 630–645. Springer. [He et al., 2026] He, S., Sun, G., Zhang, H., Fu, Y., and Li, A. (2026). Demystifying when pruning works via representation hierarchies. [He et al., 2017] He, Y., Zhang, X., and Sun, J. (2017). Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389–1397. [Hinton and Van Camp, 1993] Hinton, G. E. and Van Camp, D. (1993). Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, pages 5–13. [Hooker et al., 2019] Hooker, S., Courville, A., Clark, G., Yannakakis, Y., and Murphy, K. (2019). What do compressed deep neural networks forget? arXiv preprint arXiv:1911.05248. [Houlsby et al., 2019] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR. [Hu et al., 2022] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022). LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR). [Huang et al., 2016] Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. (2016). Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), pages 646–661. Springer. [HuggingFace Research Team et al., 2026] HuggingFace Research Team, von Werra, L., and Mangrulkar, S. (2026). Orthogonal subspace fine-tuning (osf): A unified framework for interference-free peft. arXiv preprint arXiv:2602.04519. [Huh et al., 2024] Huh, M., Cheung, B., Wang, T., and Isola, P. (2024). The platonic representation hypothesis. arXiv preprint arXiv:2405.07987. [Hänni et al., 2024] Hänni, K., Mendel, J., Vaintrob, D., and Chan, L. (2024). Mathematical models of computation in superposition. arXiv. [Islam et al., 2025] Islam, M., Ma’sum, M. A., Pratama, M., and Skrjanc, I. (2025). Latest advancements towards catastrophic forgetting under data scarcity: A comprehensive survey on few-shot class incremental learning. arXiv preprint arXiv:2502.08209. [Jacot et al., 2018] Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, volume 31. [Jaeger, 2014] Jaeger, H. (2014). Controlling recurrent neural networks by conceptors. arXiv. [Jaynes, 1957] Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical review, 106(4):620. 27
Hilbert Operator for Progressive Encoding (HOPE)
[Karp, 1972] Karp, R. M. (1972). Reducibility among combinatorial problems. In Complexity of computer computations, pages 85–103. Springer. [Kimmel et al., 2026] Kimmel, D. L., Stachenfeld, K. L., Salzman, C., and Shohamy, D. (2026). Neural representations supporting generalization under continual learning. bioRxiv. [Kirkpatrick et al., 2017] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526. [Kong et al., 2026] Kong, L., Liu, X., Chen, G., Ma, M. Q., Song, X., Sun, Y., Yurochkin, M., Killian, T. W., Salakhutdinov, R., Zhang, K., Xing, E. P., and Liu, Z. (2026). From reasoning traces to reusable modules: Understanding compositional generalization in language model reasoning. [Krizhevsky, 2009] Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. pages 32–33. [Kumaran et al., 2016] Kumaran, D., Hassabis, D., and McClelland, J. L. (2016). What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7):512–534. [LeCun et al., 1989] LeCun, Y., Denker, J. S., and Solla, S. (1989). Optimal brain damage. In Advances in Neural Information Processing Systems (NIPS). [Lee et al., 2018] Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., and Sohl-Dickstein, J. (2018). Deep neural networks as gaussian processes. In International Conference on Learning Representations. [Lee et al., 2021] Lee, N., Ajanthan, T., van de Weijer, J., Chang, P. H. S., and Torr, P. (2021). Layer-adaptive sparsity for the magnitude-based pruning. In International Conference on Learning Representations. [Li et al., 2017] Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P. (2017). Pruning filters for efficient convnets. In International Conference on Learning Representations (ICLR) Workshop. [Liu et al., 2019] Liu, H., Simonyan, K., and Yang, Y. (2019). Darts: Differentiable architecture search. In International Conference on Learning Representations (ICLR). [Liu et al., 2017] Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C. (2017). Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pages 2736–2744. [Lopes et al., 2017] Lopes, R. G., Fenu, S., and Starner, T. (2017). Data-free knowledge distillation for deep neural networks. In NIPS Workshop on Machine Learning on the Phone and other Consumer Devices. [Luo et al., 2017] Luo, J.-H., Wu, J., and Lin, W. (2017). Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, pages 5058–5066. [Ma et al., 2023] Ma, X., Fang, G., and Wang, X. (2023). Llm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems (NeurIPS).
28
Hilbert Operator for Progressive Encoding (HOPE)
[Mallya and Lazebnik, 2018] Mallya, A. and Lazebnik, S. (2018). Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7765–7773. [Martin and Mahoney, 2021] Martin, C. H. and Mahoney, M. W. (2021). Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1–73. First presented/arXiv in 2018. [McClelland et al., 1995] McClelland, J. L., McNaughton, B. L., and O’Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3):419–457. [Micaelli and Storkey, 2019] Micaelli, P. and Storkey, A. J. (2019). Zero-shot knowledge transfer via adversarial belief matching. In Advances in Neural Information Processing Systems, volume 32. [Mikolov et al., 2013] Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013. [Molchanov et al., 2019] Molchanov, P., Mallya, A., Tyagi, S., Bourezak, I., and Kautz, J. (2019). Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264–11272. [Molchanov et al., 2017] Molchanov, P., Tyagi, S., Natsev, A., and Krause, J. (2017). Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations. [Morcos et al., 2018] Morcos, A. S., Barrett, D. G., Rabinowitz, N. C., and Botvinick, M. (2018). On the importance of single directions for generalization. In International Conference on Learning Representations. [Moreira et al., 2026] Moreira, G., Marinho, Z., Marques, M., Costeira, J. a. P., and Xiong, C. (2026). Native hierarchical and compositional representations with subspace embeddings. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’26). [Mozer and Smolensky, 1988] Mozer, M. C. and Smolensky, P. (1988). Skeletonization: A technique for trimming the fat from a network via relevance assessment. In Touretzky, D., editor, Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann. [Nagel et al., 2019] Nagel, M., Baalen, M. v., Blankevoort, T., and Welling, M. (2019). Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325–1334. [Neal, 1996] Neal, R. M. (1996). Bayesian learning for neural networks, volume 118. Springer Science & Business Media. [Netzer et al., 2011] Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. (2011). Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. [Neyshabur et al., 2017] Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. (2017). Exploring generalization in deep learning. In Advances in neural information processing systems, volume 30. 29
Hilbert Operator for Progressive Encoding (HOPE)
[Neyshabur et al., 2015] Neyshabur, B., Tomioka, R., and Srebro, N. (2015). In search of the real inductive bias: On the role of implicit regularization in deep learning. In International Conference on Learning Representations. [Nguyen et al., 2026] Nguyen, Q., Pham, H. H., Cassi, D., and Bellingeri, M. (2026). Depth fragility and skeletal universality: Decoupling topology and function in deep neural networks. Mathematics, 14(9):1438. [Olah et al., 2020] Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020). Zoom in: An introduction to circuits. Distill. [Park et al., 2023] Park, K., Choe, Y. J., and Veitch, V. (2023). The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658. [Pennington et al., 2014] Pennington, J., Socher, R., and Manning, C. D. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. [Qiu et al., 2024] Qiu, W., Yin, M., Wang, M., Bartlett, P., and Zanette, A. (2024). Continual learning in the frequency domain. In Advances in Neural Information Processing Systems (NeurIPS). [Renda et al., 2020] Renda, A., Frankle, J., and Carbin, M. (2020). Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations. [Rissanen, 1978] Rissanen, J. (1978). Modeling by shortest data description. Automatica, 14(5):465– 471. [Saha et al., 2021] Saha, G., Garg, I., and Roy, K. (2021). Gradient projection memory for continual learning. In International Conference on Learning Representations (ICLR). [Scardapane et al., 2017] Scardapane, S., Comminiello, D., Hussain, A., and Uncini, A. (2017). Group sparse regularization for deep neural networks. Neurocomputing, 241:81–89. [Scholl et al., 2021] Scholl, C., Rule, M. E., and Hennig, M. H. (2021). The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules. PLOS Computational Biology, 17(10):1–23. [Shwartz-Ziv and Tishby, 2017] Shwartz-Ziv, R. and Tishby, N. (2017). Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810. [Singh and Alistarh, 2020] Singh, S. P. and Alistarh, D. (2020). Woodfisher: Efficient second-order approximation for neural network compression. In Advances in Neural Information Processing Systems, volume 33, pages 18098–18109. [Singh and Jaggi, 2020] Singh, S. P. and Jaggi, M. (2020). Model fusion via optimal transport. In Advances in Neural Information Processing Systems, volume 33, pages 22045–22055. [Srinivas and Babu, 2015] Srinivas, S. and Babu, R. V. (2015). Data-free parameter pruning for deep neural networks. arXiv. [Stoica et al., 2024] Stoica, G., Bolya, D., Bales, J., and Hoffman, J. (2024). Zipit! merging models from different tasks without training. In International Conference on Learning Representations. [Tanaka et al., 2020] Tanaka, H., Kunin, D., Yamins, D. L. K., and Ganguli, S. (2020). Pruning neural networks without any data by iteratively conserving synaptic flow. ArXiv, abs/2006.05467. 30
Hilbert Operator for Progressive Encoding (HOPE)
[Tatro et al., 2020] Tatro, N., Chen, P.-Y., Das, P., Sattigeri, P., Lai, R., and Huan, Z. (2020). Optimizing mode connectivity via neuron alignment. In Advances in Neural Information Processing Systems, volume 33, pages 15300–15311. [Tishby et al., 1999] Tishby, N., Pereira, F. C., and Bialek, W. (1999). The information bottleneck method. In The 37th Annual Allerton Conference on Communication, Control, and Computing, pages 368–377. [Veit and Belongie, 2018] Veit, A. and Belongie, S. (2018). Convolutional networks with adaptive inference graphs. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3–18. [Wang et al., 2025] Wang, P., Li, X. L., Yaras, C., Zhu, Z., Balzano, L., Hu, W., and Qu, Q. (2025). Understanding deep representation learning via layerwise feature compression and discrimination. Journal of Machine Learning Research, 26(47):1–71. [Wang et al., 2021] Wang, S., Li, X., Sun, J., and Xu, Z. (2021). Training networks in null space of feature covariance for continual learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 184–193. [Wang et al., 2018] Wang, X., Yu, F., Dou, Z.-Y., Darrell, T., and Gonzalez, J. E. (2018). Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 409–424. [Wen et al., 2016] Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. (2016). Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems (NIPS). [Xie et al., 2025] Xie, J., Yang, J., Luo, Z., Cao, Y., Gao, Q., Zhang, M., and Hu, W. (2025). AdaDARE-𝛾 : Balancing stability and plasticity in multi-modal LLMs through efficient adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [Yang, 2019] Yang, G. (2019). Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are gaussian processes. In Advances in Neural Information Processing Systems, volume 32. [Yang et al., 2025] Yang, M., Zhang, W., and Liu, H. (2025). Lora-null: Directing parameter-efficient adaptation into orthogonal null-spaces. In Proceedings of the International Conference on Machine Learning (ICML). [Yin et al., 2020] Yin, H., Molchanov, P., Alvarez, J. M., Li, Z., Mallya, A., Hoiem, D., Jha, N. K., and Kautz, J. (2020). Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715– 8724. [Yosinski et al., 2014] Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. (2014). How transferable are features in deep neural networks? In Advances in neural information processing systems, volume 27. [Yvinec et al., 2021] Yvinec, E., Dapogny, A., Cord, M., and Bailly, K. (2021). Red : Looking for redundancies for data-free structured compression of deep neural networks. arXiv. [Zeng et al., 2019] Zeng, G., Chen, Y., Cui, B., and Yu, S. (2019). Continual learning of contextdependent processing in neural networks. Nature Machine Intelligence, 1(8):364–372. [Zenke et al., 2017] Zenke, F., Poole, B., and Ganguli, S. (2017). Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. PMLR. 31
Hilbert Operator for Progressive Encoding (HOPE)
[Zhang et al., 2025] Zhang, Y., Saxe, A., and Latham, P. E. (2025). Saddle-to-saddle dynamics explains a simplicity bias across neural network architectures. arXiv preprint arXiv:2512.20607. [Zhou et al., 2026] Zhou, X., Zhao, H., and Mehr, N. (2026). Taco: Temporal consensus optimization for continual neural mapping. arXiv preprint arXiv:2602.04516. [Zoph and Le, 2017] Zoph, B. and Le, Q. V. (2017). Neural architecture search with reinforcement learning. In International Conference on Learning Representations.
32
Hilbert Operator for Progressive Encoding (HOPE)
Appendix Table of Contents A Hilbert Spaces
35
A.1 Introduction: Why a Hilbert Space? . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
A.2 The Inner Product: The Ruler of Geometry . . . . . . . . . . . . . . . . . . . . . . . .
35
A.3 The Ambient Space: 𝐿2 (X , 𝑃 X ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
36
A.4 The Tensor Product: Splitting Continuous and Discrete Spaces . . . . . . . . . . . . .
36
A.5 Uniqueness and the Total Set Property . . . . . . . . . . . . . . . . . . . . . . . . . .
37
A.6 Neuron Synthesis by Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
A.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
B Implementation Notes
38
B.1 Adaptation for Convolutional Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
B.2 Deriving the Parameter Footprint Δ𝑃 . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
B.3 Cross-Action Overlap and Uniform Scaling . . . . . . . . . . . . . . . . . . . . . . . .
39
B.4 Computational Complexity and the Decoupled Cache . . . . . . . . . . . . . . . . . .
40
B.5 Numerical Stability and BN Parameters . . . . . . . . . . . . . . . . . . . . . . . . . .
41
C Main Paper Proofs
42
C.1 Layer Transition Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42
C.2 Generating Parent Neuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47
C.3 Block Eviction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
50
D Derivation of Physical BN Parameters
50
E Kernel Formulation
52
E.1 Pre-Activation Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52
E.2 Self-Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
53
E.3 Cross-Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
54
F Derivations for Block Eviction
56
F.1 Generalization and Execution Degradation in Depleted Blocks . . . . . . . . . . . . .
56
F.2 Derivation of the Unified Macro Cost Jevict . . . . . . . . . . . . . . . . . . . . . . . .
56
F.3 Generalization to Non-Residual Architectures
59
. . . . . . . . . . . . . . . . . . . . . .
G Reproducibility Protocols for Cross-Domain Transfer
59
33
Hilbert Operator for Progressive Encoding (HOPE)
G.1 Task Construction and Data Partitioning . . . . . . . . . . . . . . . . . . . . . . . . .
59
G.2 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
60
G.3 Base Training Regimen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
60
G.4 EWC Exact Empirical Fisher Calculation . . . . . . . . . . . . . . . . . . . . . . . . .
61
G.5 Hyperparameter Tuning and Final Evaluation . . . . . . . . . . . . . . . . . . . . . .
61
H Theoretical Guarantees of DEFT
I
62
H.1 Algorithmic Axioms and Partitioning of Neurons . . . . . . . . . . . . . . . . . . . . .
63
H.2 Layer-to-Layer Bounding Framework . . . . . . . . . . . . . . . . . . . . . . . . . . .
64
H.3 Dynamic Resolution of Redundancy via Bounded Trade-off . . . . . . . . . . . . . . .
66
Algorithms
67
34
Hilbert Operator for Progressive Encoding (HOPE)
A. Hilbert Spaces This section provides a quick introduction to Hilbert spaces, focusing on the concepts of inner products, completeness, and total sets. We bridge abstract functional analysis with the specific architectural choices of the HOPE framework, demonstrating how the 𝐿2 embedding of neural functions creates a unique and complete geometric environment for optimization over network structures. A.1. Introduction: Why a Hilbert Space? We treat compression operations like pruning and merging as projections. When we replace two neurons with one, we are attempting to find a single element that "represents" a multi-dimensional subspace. To perform this operation rigorously, we need three things: 1. A Space that contains all possible neural identities. 2. A Metric (Inner Product) to measure "closeness" and "alignment." 3. Completeness to ensure that our optimizations actually have solutions. A Hilbert space H provides these three pillars. A.2. The Inner Product: The Ruler of Geometry The defining feature of a Hilbert space is the inner product. While a vector space only lets us add and scale elements, an inner product space lets us talk about angles and lengths. Definition A.1 (Inner Product Axioms). An inner product on a vector space 𝑉 over ℝ is a mapping ⟨·, ·⟩ : 𝑉 × 𝑉 → ℝ satisfying for all 𝑓 , 𝑔, ℎ ∈ 𝑉 and 𝑎 ∈ ℝ: • Symmetry: ⟨ 𝑓 , 𝑔 ⟩ = ⟨𝑔, 𝑓 ⟩. • Linearity: ⟨𝑎 𝑓 + 𝑔, ℎ⟩ = 𝑎 ⟨ 𝑓 , ℎ⟩ + ⟨𝑔, ℎ⟩. • Positive Definiteness: ⟨ 𝑓 , 𝑓 ⟩ ≥ 0, and ⟨ 𝑓 , 𝑓 ⟩ = 0 ⇐⇒ 𝑓 = 0. A.2.1. Connection to HOPE: The Expectation Metric In the HOPE framework, we operate on functions 𝑓 : X → ℝ𝑐 . We define our inner product relative to a surrogate distribution 𝑃 X : 𝑓 𝑖 , 𝑓 𝑗 H ≜ 𝔼 𝒙 ∼ 𝑃X 𝑓 𝑖 ( 𝒙 ) 𝑇 𝑓 𝑗 ( 𝒙 ) (30) Proposition A.1 (Validity of the HOPE Metric). The functional defined in (30) satisfies the inner product axioms. Proof. Linearity and symmetry follow directly from the linearity of the expectation operator 𝔼 and the symmetry of the Euclidean dot product. Positive definiteness is guaranteed because ⟨ 𝑓 , 𝑓 ⟩ = 𝔼[∥ 𝑓 ( 𝒙 )∥ 2 ] ≥ 0. The definiteness ⟨ 𝑓 , 𝑓 ⟩ = 0 =⇒ 𝑓 = 0 is satisfied in the 𝐿2 sense (i.e., 𝑓 is zero "almost everywhere"). □
35
Hilbert Operator for Progressive Encoding (HOPE)
A.3. The Ambient Space: 𝐿2 (X , 𝑃 X ) A Hilbert space is more than just an inner product space; it must be complete. In finite dimensions (like ℝ𝑛 ), every inner product space is complete. In function spaces, this is not true. Definition A.2 (Completeness). A space is complete if every Cauchy sequence { 𝑓𝑛 } (a sequence where elements get arbitrarily close to each other) converges to an element 𝑓 that is also inside the space. If we worked only with continuous functions, the space would not be complete. For example, a sequence of continuous functions can converge to a step function (which is discontinuous). This would be a disaster for compression, as our "best parent" might not even exist in our space. A.3.1. The 𝐿2 Embedding To avoid this, HOPE embeds neurons into 𝐿2 (X , 𝑃 X ), the space of square-integrable functions. • Energy Bound: Every function in H has finite energy: 𝔼[∥ 𝑓 ∥ 2 ] < ∞. • Closure: By definition, 𝐿2 is complete. Every optimization we perform (minimizing the cost functional of compression operations) is guaranteed to have a valid result within the ambient space. A.4. The Tensor Product: Splitting Continuous and Discrete Spaces While we defined our ambient space as vector-valued functions H = 𝐿2 (X , 𝑃 X ; ℝ𝑐 ), computing the inner product directly in this monolithic space obscures the internal structure of a neural network. HOPE simplifies this by utilizing a tensor product space. A neuron’s operation naturally splits into two phases: 1. The Continuous Input Landscape: The effective input weights (which absorb BN statistics) eff ) 𝑇 𝒙 + 𝑏 ). We and the activation function create a continuous scalar landscape 𝑔𝑖 ( 𝒙 ) = Ψ (( 𝒘in 𝑖 ,𝑖 embed this function into a scalar Hilbert space Hin ≜ 𝐿2 (X , 𝑃 X ; ℝ). 2. The Discrete Output: This scalar activation is broadcast to the next layer along a finitedimensional output weight vector 𝒘out,𝑖 . We define this output space as Hout ≜ ℝ𝑐 . By taking the tensor product of these two spaces, we construct the full ambient space mapping: H Hin ⊗ Hout . Under this formulation, each individual neuron is modeled as a rank-1 HilbertSchmidt operator, represented by the outer product of its input function and output vector: 𝑓𝑖 ≜ 𝑔𝑖 ⊗ 𝒘out,𝑖
(31)
A.4.1. Factoring the Metric This tensor structure is what makes HOPE computationally tractable. The inner product of two rank-1 tensors elegantly factors into the product of their individual space inner products: 𝑓𝑖 , 𝑓 𝑗 H = 𝑔𝑖 ⊗ 𝒘out,𝑖 , 𝑔 𝑗 ⊗ 𝒘out, 𝑗 H = 𝑔𝑖 , 𝑔 𝑗 H · 𝒘out,𝑖 , 𝒘out, 𝑗 H in
out
(32)
Because 𝑔𝑖 , 𝑔 𝑗 H is the expected alignment of their non-linear activations over the distribution in 𝑃 X , we define this as the kernel 𝐾 ( 𝑖, 𝑗). This allows us to separate the continuous-functional evaluation from the discrete parameters, reducing the full Hilbert space inner product to: 𝑓𝑖 , 𝑓 𝑗 H = 𝐾 ( 𝑖, 𝑗) · 𝒘out,𝑖 , 𝒘out, 𝑗 ℝ𝑐
(33) 36
Hilbert Operator for Progressive Encoding (HOPE)
A.5. Uniqueness and the Total Set Property One might ask: "We defined the inner product for any functions 𝑓 , 𝑔 . But in HOPE, we only ever calculate it for single ReLU neurons. Is that enough to define the whole space?" This is the most critical part of the theory. Definition A.3 (Total Set). A set 𝑆 ⊂ H is a total set if the set of all finite linear combinations of elements in 𝑆 is dense in H . If 𝑆 is total, then knowing the inner product for every pair in 𝑆 uniquely determines the inner product for the entire Hilbert space. A.5.1. The Universal Approximation Theorem In the context of HOPE, our "dictionary" of functions is the set of single neurons N : N = { 𝑓 ( 𝒙 ) = 𝒘𝑜𝑢𝑡 Ψ ( 𝒘 ˜ in · 𝒙˜)}
(34)
Theorem A.2 (HOPE Uniqueness). The set N is a total set in 𝐿2 (X , 𝑃 X ). Discussion. By the Universal Approximation Theorem, linear combinations of ReLU neurons can approximate any square-integrable function to arbitrary precision. In the language of Hilbert spaces, this means 𝑠𝑝𝑎𝑛 (N ) is dense in H . Because N is total, the definition of the function correlation kernel 𝐾 ( 𝑖, 𝑗) = 𝔼[ Ψ ( 𝑦𝑖 ) Ψ ( 𝑦 𝑗 )] is sufficient to uniquely characterize the metric of the entire ambient space. We do not need a "separate" definition for the inner product of sums of neurons; it is uniquely forced upon the space by the behavior of the single neurons. □ A.6. Neuron Synthesis by Projection Finally, we see why this matters. The generation process relies on finding an Optimal Subspace Projection. When we want to merge neurons 𝑖 and 𝑗, their joint contribution [ 𝑓𝑖 , 𝑓 𝑗 ] spans a 2dimensional tensor subspace. Because a physical parent neuron must produce a single unified output, it is modeled as a constrained rank-1 approximation [ 𝑓 𝑝 , 𝑓 𝑝 ]. Rather than a simple orthogonal projection of a sum, HOPE finds the optimal parent 𝑓 𝑝∗ by minimizing the expected Frobenius projection error under the Hilbert-Schmidt norm, scaled by the remaining capacity of the layer: √︃ 2 2 𝑓𝑖 − 𝑓 𝑝 H + 𝑓 𝑗 − 𝑓 𝑝 H ∗ (35) 𝑓 𝑝 = argmin 𝑓 𝑝 ∈ N 𝐸 𝑎 − ∥ 𝑓𝑖 ∥ H − 𝑓 𝑗 H + 𝑓 𝑝 H In a Hilbert space, the numerator translates to a rigorous geometric projection distance. Without the Hilbert space structure (inner products and completeness), the concepts of "closest operator" and "functional alignment" would have no meaning. A.7. Summary A summary of the key concepts and their utility in HOPE are provided in Table 3. 37
Hilbert Operator for Progressive Encoding (HOPE)
Hilbert Concept
HOPE Implementation
Inner Product ⟨ 𝑓 , 𝑔 ⟩ Norm ∥ 𝑓 ∥ Ambient Space H Operators Basis / Total Set N Completeness
Expected dot product over 𝑃 X Functional Capacity (Square root of Signal Energy) Tensor product space Hin ⊗ Hout Neurons modeled as rank-1 Hilbert-Schmidt operators The manifold of realizable single neurons Guarantees that the "best parent" is well-defined
Table 3 | Mapping of Abstract Hilbert Concepts to the HOPE Framework.
B. Implementation Notes B.1. Adaptation for Convolutional Layers Our formalism extends to convolutional networks by defining a “neuron” as a filter in layer 𝐴 producing feature map 𝑖. The joint parameter vector 𝒘𝑖 = [ 𝒘𝑛⊤ , 𝒘𝑐⊤ ] ⊤ is constructed by vectorizing the filter’s respective input and output kernels. The input space vector 𝒘𝑛 encapsulates the filter’s local receptive field. Assuming layer 𝐴 possesses a weight tensor 𝑲 𝐴 ∈ ℝℎ 𝐴 ×𝑤𝐴 ×𝐶in ×𝐶out , the vector 𝒘𝑛 corresponding to filter 𝑖 is the flattened spatial slice 𝑲 𝐴 [:, :, :, 𝑖] ∈ ℝ𝑛 , where 𝑛 = ℎ 𝐴 × 𝑤 𝐴 × 𝐶in . Conversely, the output space vector 𝒘𝑐 captures the filter’s downstream influence on the subsequent layer 𝐵. Because the activation map of filter 𝑖 acts as the 𝑖-th input channel to layer 𝐵, 𝒘𝑐 is formed by extracting and flattening the corresponding slice of the downstream tensor 𝑲 𝐵 [:, :, 𝑖, :] ∈ ℝ𝑐 , where 𝑐 = ℎ 𝐵 × 𝑤𝐵 × 𝐶out,𝐵 . To allow the surrogate distribution to serve as a location-invariant prior without requiring intractable coordinate-specific covariance modeling, we assume spatial stationarity (ergodicity) across the feature map and construct it using the globally averaged BN variance 𝜎2𝑖 . While boundary zeropadding breaks local stationarity, the high-dimensional spatial aggregation of modern BN buffers absorbs these edge-effects into a into a single global average. B.2. Deriving the Parameter Footprint Δ𝑃 As established in Section 9, the parameter footprint Δ𝑃 quantifies the number of physical parameters removed by a compression action. To preserve Dantzig’s Axiom of Item Independence, this criterion uses a static surrogate, Δ𝑃 init , evaluated on the initial network state and decoupled from the dynamically changing network. For any operation, the footprint tracks the total removed parameters: Δ𝑃 init = ∥𝑊in ∥ 0 + ∥𝑊out ∥ 0 + ∥ 𝜃aux ∥ 0
(36)
where 𝑊in , 𝑊out , and 𝜽aux denote the input weights, output weights, and auxiliary parameters (e.g., BN parameters) respectively, and ∥ · ∥ 0 counts the number of non-zero elements. B.2.1. Granular Operations For pruning or merging, Δ𝑃 init comprises the incoming weights, outgoing weights, and BN parameters of a single target neuron or filter. For example: • Transformer MLP: Removing a neuron in an MLP with a 𝑑model → 4𝑑model expansion yields Δ𝑃 init = 2𝑑model + 1. • Convolutional Networks: For a filter with spatial dimensions 𝐻 × 𝑊 , the footprint scales with the receptive field and projective kernel: Δ𝑃 init = 𝐻 · 𝑊 · 𝐶in + 𝐶out + 4, where 4 accounts for the BN parameters 𝛾, 𝛽, 𝜇, 𝜎2 . 38
Hilbert Operator for Progressive Encoding (HOPE)
Example: Architectural Symmetry in ResNet-50 Evaluating Δ𝑃 init locally reveals symmetries across different layer types. Consider a ResNet-50 bottleneck block with a base channel width 𝑁 and a 4× expansion ratio: • 1×1 Squeeze Layer: Removing one filter deletes 4 𝑁 input connections and 9 𝑁 output connections to the subsequent 3 × 3 layer. Yield: Δ𝑃 init = 13 𝑁 . • 3×3 Spatial Layer: Removing one filter deletes 9 𝑁 input connections and 4 𝑁 output connections to the subsequent 1 × 1 expansion layer. Yield: Δ𝑃 init = 13 𝑁 . Despite differing spatial tensor shapes, the parameter yield per filter in both positions evaluates to 13 𝑁 . This symmetry allows the cost J to arbitrate compression across heterogeneous layers without being biased by raw parameter array shapes. B.2.2. Macro Operations: Block Eviction Unlike granular operations that yield incremental savings, block eviction removes entire layers simultaneously. For a residual block with internal convolutional layers 𝑊1 and 𝑊2 , and a terminal expansion layer 𝑊3 , the static parameter yield expands to: init Δ𝑃evict = ∥𝑊1 ∥ 0 + ∥𝑊2 ∥ 0 + ∥ 𝜃aux ∥ 0
(37)
where ∥𝑊1 ∥ 0 and ∥𝑊2 ∥ 0 denote the parameters of the internal layers, and ∥ 𝜃aux ∥ 0 counts the BN parameters 𝜇, 𝜎2 , 𝛾, 𝛽 across the entire block, including those of 𝑊3 . Note that the weight matrix 𝑊3 ∈ ℝ𝑑amb × 𝑑bottleneck is not explicitly included in this sum. Because removing a filter in 𝑊2 removes its corresponding outgoing connections in 𝑊3 , the memory footprint of 𝑊3 is naturally accounted for when evaluating 𝑊2 . Explicitly adding the ambient dimensions ∥𝑊3 ∥ 0 = 𝑑amb × 𝑑bottleneck would count the same parameters twice within a single macro action. This intra-action double-counting would inflate the parameter footprint of block eviction and artificially lower its Distortion Rate (DR), giving it an unfair advantage over granular operations. B.3. Cross-Action Overlap and Uniform Scaling While we avoid counting parameters twice within a single action (as seen with 𝑊3 ), evaluating the entire decision space using a static footprint Δ𝑃 init introduces an overlap between different competing actions. Let 𝑖 and 𝑗 be targeted neurons in adjacent layers 𝑙 and 𝑙 + 1. To preserve Dantzig’s Axiom of Item Independence, an action targeting layer 𝑙 must not alter the state variables used to evaluate (𝑙) layer 𝑙 + 1. Consequently, their shared weight 𝑊 𝑗,𝑖 is counted independently in both evaluations: (𝑙)
(𝑙)
𝑊 𝑗,𝑖 ∈ Δ𝑃 𝑖init and 𝑊 𝑗,𝑖 ∈ Δ𝑃 init . 𝑗
For any sequence of actions S, this cross-action overlap overestimates the true number of parameters Í recovered 𝑘 ∈ S Δ𝑃𝑘init > Δ𝑃 live . This overestimation artificially lowers the computed DR compared to S the live network state: J𝑘 J𝑘 DR𝑘 = < (38) init live Δ𝑃 𝑘
Δ𝑃 𝑘
To correct this approximation without violating item independence, HOPE relies on uniform scaling. Because this cross-action overlap applies systematically across the entire action space A (all granular and macro candidates interact with their neighbors), it acts as a uniform scaling factor 𝛼 ≥ 1 such that Δ𝑃𝑘init ≈ 𝛼Δ𝑃𝑘live .
39
Hilbert Operator for Progressive Encoding (HOPE)
Because the distortion cost J is evaluated independently of the parameter counts, and 𝛼 applies uniformly, the relative ordering of the distortion rates is preserved: J𝑎 Δ𝑃 init 𝑎
J𝑏 <
Δ𝑃 init 𝑏
⇐⇒
J𝑎 Δ𝑃 live
J𝑏 <
𝑎
Δ𝑃𝑏live
(39)
Since the greedy continuous knapsack solver selects arg min𝑘 ∈ A DR𝑘 , this uniform scaling ensures that the theoretical fairness of the optimal action selection remains intact. B.4. Computational Complexity and the Decoupled Cache Evaluating the transition cost J efficiently poses a computational challenge because it is inversely coupled to the monotonically decreasing layer capacity 𝐸rem . In a layer with 𝑁 neurons, recomputing the non-linear weight-space geometry (e.g., Rank-2 Singular Value Decompositions) for all O ( 𝑁 2 ) candidate pairs every time 𝐸rem decreases requires O ( 𝑁 3 ) execution time. Conversely, caching J values in a standard priority queue via submodular approximations (e.g., Minoux’s Lazy Update) leads to stale estimations. Because the transition cost goes to ∞ as capacity approaches 0, small capacity reductions cause the true costs to spike. A delayed queue would underestimate these costs and cause the optimizer to select sub-optimal actions and potentially collapse the layer. We resolve this bottleneck using an O (1) Decoupled Cache. Decoupling. The computationally expensive optimal projections 𝒖∗ , 𝒗∗ and the cost components (𝑎 = ∥ 𝑓𝑖 ∥ 2H + ∥ 𝑓 𝑗 ∥ 2H and 𝑏 = 𝜓∗ , 𝑓𝑖 + 𝑓 𝑗 H , derived in Section 7.1) depend only on weights and static BN parameters associated with that layer. Since our method does not rely on any cross-layer criterion (such as Fisher Information), we guarantee that 𝒖∗ , 𝒗∗ , 𝑎, 𝑏 remain independent of both the downstream architecture and the dynamic capacity 𝐸rem . The only variable dependent on 𝐸rem is the optimal scalar magnitude 𝑠∗ . At initialization, the framework computes 𝒖∗ and 𝒗∗ to evaluate and cache only the scalar constants 𝑎 and 𝑏. To prevent O ( 𝑁 2 ) memory exhaustion, the high-dimensional vectors 𝒖∗ , 𝒗∗ are then discarded. During the greedy search, evaluating the cost of any action requires querying the cached constants and the live remaining capacity 𝐸rem = max( 𝐸 𝑎 − ∥ 𝑓𝑖 ∥ H − ∥ 𝑓 𝑗 ∥ H , 𝜖) to compute 𝑠∗ and J analytically in O (1) time: 𝑠∗ =
𝑎 + 𝑏𝐸rem 2 𝐸rem + 𝑏
(40)
Index Determinism and JIT Generation. We sort the active neurons to evaluate undirected pairs only where 𝑖 < 𝑗. By enforcing this index determinism, the algorithm guarantees a 100% cache hit rate and makes it viable to abandon priority queues entirely. At every step, the algorithm executes an O (1) global scan over all remaining pairs using the live layer capacity, maintaining mathematically perfect freshness. Once the global minimum is selected, the framework executes a Just-In-Time (JIT) generation, re-evaluating the fast Rank-2 SVD only for the single winning pair (∼ 1 ms) to retrieve its 𝒖∗ , 𝒗∗ for physical deployment. Computing the BN Variance. Deploying the JIT-generated parent requires setting its BN variance 𝛾 2𝑝 . Under the surrogate distribution 𝑃 X , 𝛾 2𝑝 depends only on the warped correlation 𝜌ˆ𝑖 𝑗 . A naive approach might substitute the simple correlation of the raw weights 𝜌raw into the variance equation. However, this incorrectly mixes parameter-space metrics with signal-space statistics. Doing so produces incorrect BN moving averages, which miscalibrates the network during the forward pass. The framework avoids 40
Hilbert Operator for Progressive Encoding (HOPE)
this by computing the physical variance using 𝜌ˆ𝑖 𝑗 directly retrieved from the cache. This ensures the deployed parameters maintain the correct statistical behavior without requiring empirical forward passes. B.5. Numerical Stability and BN Parameters eff and 𝑏 ) to Section 7.2.2 derives the mapping from the parent neuron’s effective parameters (𝒘in 𝑝 ,𝑝 raw its physical network variables: the raw input weights 𝒘in,𝑝 and the BN parameters 𝛾 𝑝 , 𝛽 𝑝 , 𝜇 𝑝 , 𝜎2𝑝 . A naive assignment of these physical variables alters the pre-activation signal and breaks the critical eff ) 𝑇 𝒙 + 𝑏 . This mismatch miscalibrates the network and immediately degrades equivalence 𝑦 𝑝 = ( 𝒘in 𝑝 ,𝑝 accuracy on the first forward pass.
To prevent this, HOPE enforces the exact analytical mapping. However, deploying these formulas in practice requires safely handling numerical bounds, such as preventing negative variance 𝜎2𝑝 < 0 √︃ and avoiding NaN in the BN denominator 𝜎2𝑝 + 𝜖. This section details how to resolve these numerical edge cases and ensure the deployed model remains mathematically well-defined and numerically stable. B.5.1. The 𝜖 Boundary Regime and Variance Clamping To resolve the under-constrained BN system, the framework fixes the physical variance as 𝜎2𝑝 = max(0, 𝛾 2𝑝 − 𝜖), where 𝜖 is a small stability constant (e.g., 10−5 ). For active features 𝛾 2𝑝 ≥ 𝜖, this evaluates smoothly. This allows the denominator of the BN transformation to simplify to | 𝛾 𝑝 |. However, as the progressive encoder shrinks the network, the derived parent variance 𝛾 2𝑝 may occasionally fall below the numerical floor 𝜖. Without the max(0, ·) operator, enforcing 𝜎2𝑝 = 𝛾 2𝑝 − 𝜖 would result in a negative variance, which causes NaN during inference. The clamping operator safely bounds the physical variance at 𝜎2𝑝 = 0. √︃ √ When 𝜎2𝑝 is clamped to 0, the BN denominator 𝜎2𝑝 + 𝜖 evaluates to 𝜖. Consequently, the scale √ factor becomes 𝛾 𝑝 / 𝜖. Substituting this into the effective bias equation yields: √ 𝛾𝑝 𝜖 𝑏𝑝 = 𝛽𝑝 − √ 𝜇 𝑝 =⇒ 𝜇𝑝 = ( 𝛽 𝑝 − 𝑏𝑝) (41) 𝜖
𝛾𝑝
As the parent neuron’s scale 𝛾 𝑝 approaches 0, dividing by 𝛾 𝑝 causes 𝜇 𝑝 → ∞. To prevent this, the raw = 0. framework bypasses this calculation for inactive features and sets 𝒘in ,𝑝 B.5.2. Running Variance Offset During Fine-Tuning Setting the initial BN running variance to 𝜎2𝑝 = 𝛾 2𝑝 − 𝜖 ensures the network’s output is perfectly preserved during inference. However, when the model resumes training for fine-tuning, the actual 2 batch variance computed during the forward pass evaluates to 𝜎batch = 𝛾 2𝑝 . This introduces a minor mismatch between the empirical batch variance and the stored running variance: 2 𝜎batch − 𝜎2𝑝 = 𝛾 2𝑝 − ( 𝛾 2𝑝 − 𝜖) = 𝜖
(42)
Because this discrepancy evaluates to the numerical constant 𝜖 (typically 10−5 ), its impact is negligible. Standard optimizers (e.g., Adam, AdamW) seamlessly absorb this O ( 𝜖) offset during the initial training steps without destabilizing the network or degrading performance. 41
Hilbert Operator for Progressive Encoding (HOPE)
C. Main Paper Proofs C.1. Layer Transition Costs For convenience, we first recall the axioms of the cost J from the main paper. Axioms of the Cost Functional J To ensure a well-posed definition of the cost J , the framework introduces the following three axioms: • Magnitude Neutrality: J must be scale invariant: ∀𝑘 > 0 ; J ( 𝑘 Φ𝑎 , 𝑘 Φ𝑏 ) = J ( Φ𝑎 , Φ𝑏 ). • Connectivity Preservation: J must establish an asymptotic barrier preventing layer extinction: lim𝐸 ( Φ𝑏 )→0+ J = ∞. • Infinitesimal Capacity Dependence: J must be additive along continuous paths and ∫1 driven by the reduction in layer capacity: J ( Φ𝑎 , Φ𝑏 ) = 0 −𝜉 ( Φ ( 𝑡 )) 𝐸¤ ( 𝑡 ) 𝑑𝑡 , where 𝐸¤ ( 𝑡 ) ≜ 𝑑𝐸 ( Φ ( 𝑡 ))/𝑑𝑡 and 𝜉 ( Φ ( 𝑡 )) > 0 is a state-dependent density function. Lemma C.1 (Uniqueness of the 𝐿1 Capacity). Let Φ = ( 𝑓1 , . . . , 𝑓𝑁 ) ∈ H 𝑁 denote a layer state. Ð 𝑁 Assume the capacity functional 𝐸 : ∞ 𝑁 =1 H → ℝ ≥0 satisfies: 1. Identity: ∀ 𝑓 ∈ H , 𝐸 (( 𝑓 )) = ∥ 𝑓 ∥ H . 2. Symmetry & Separability: ∃ 𝑔 : ℝ ≥0 → ℝ (continuous and strictly monotonic) and a Í function ℎ such that ∀Φ ∈ H 𝑁 , 𝐸 ( Φ) = ℎ 𝑘𝑁=1 𝑔 (∥ 𝑓𝑘 ∥ H ) . 3. Partition Invariance: ∀ 𝑓 ∈ H , ∀𝑁 ∈ ℤ ≥1 , 𝐸 (( 𝑓 )) = 𝐸 ( 𝑓 / 𝑁, . . . , 𝑓 / 𝑁 ) . | {z } 𝑁 times
Then 𝐸 ( Φ) =
Í𝑁
𝑘=1 ∥ 𝑓𝑘 ∥ H .
Proof. For a single-neuron state Φ = ( 𝑓 ), Conditions 1 and 2 imply: 𝐸 (( 𝑓 )) = ℎ 𝑔 (∥ 𝑓 ∥ H ) = ∥ 𝑓 ∥ H =⇒ ℎ ≡ 𝑔 −1 on Im( 𝑔 ) Í Thus, the functional simplifies to 𝐸 ( Φ) = 𝑔 −1 𝑘𝑁=1 𝑔 (∥ 𝑓𝑘 ∥ H ) . By Condition 3 and the positive homogeneity of the norm (∥ 𝑓 / 𝑁 ∥ H = ∥ 𝑓 ∥ H / 𝑁 for 𝑁 ≥ 1): ! 𝑁 ∑︁ ∥ ∥ ∥ 𝑓 ∥H 𝑓 H −1 −1 ∥ 𝑓 ∥ H = 𝐸 ( 𝑓 / 𝑁, . . . , 𝑓 / 𝑁 ) = 𝑔 𝑔 =𝑔 𝑁𝑔 𝑁 𝑁 | {z } 𝑘=1 𝑁 times
Applying 𝑔 to both sides and substituting 𝑥 ≜ ∥ 𝑓 ∥ H ≥ 0 yields: 𝑥 ∀𝑥 ≥ 0, ∀𝑁 ∈ ℤ ≥1 𝑔 (𝑥) = 𝑁 𝑔 𝑁
For 𝑥 = 0, 𝑔 (0) = 𝑁 𝑔 (0) =⇒ 𝑔 (0) = 0. For any rational 𝑞 = 𝑀 / 𝑁 > 0, substituting 𝑦 = 𝑥 / 𝑁 yields 𝑔 ( 𝑀 𝑦 ) = 𝑀𝑔 ( 𝑦 ) = 𝑞𝑁 𝑔 ( 𝑦 ) = 𝑞𝑔 ( 𝑁 𝑦 ), meaning 𝑔 ( 𝑞𝑥 ) = 𝑞𝑔 ( 𝑥 ). Since 𝑔 is continuous, this linearity extends to all 𝑥 ∈ ℝ ≥0 , yielding 𝑔 ( 𝑥 ) = 𝑐𝑥 for some constant 𝑐. Since 𝑔 is strictly monotonic, 𝑐 ≠ 0.
42
Hilbert Operator for Progressive Encoding (HOPE)
Substituting 𝑔 ( 𝑥 ) = 𝑐𝑥 and 𝑔 −1 ( 𝑦 ) = 𝑦 /𝑐 into the expression for 𝐸 ( Φ) gives: 𝑁
𝐸 ( Φ) =
1 ∑︁ 𝑐
𝑐 ∥ 𝑓𝑘 ∥ H =
𝑁 ∑︁
∥ 𝑓𝑘 ∥ H
𝑘=1
𝑘=1
□ Theorem C.2 (Integral Formulation of Scale-Invariant Cost). Under the Axioms of Magnitude Neutrality, Connectivity Preservation, and Infinitesimal Capacity Dependence, the transition cost along a continuous deformation path Φ : [0, 1] → H 𝑁 with boundary conditions Φ (0) = Φ𝑎 and Φ (1) = Φ𝑏 , is determined as the integral: J ( Φ 𝑎 , Φ𝑏 ) =
∫ 1
−𝑐 ( Φ ( 𝑡 ))
0
𝐸¤ ( 𝑡 ) 𝑑𝑡 a 𝐸 ( Φ ( 𝑡 ))
(43)
where 𝐸 ( Φ ( 𝑡 )) is the instantaneous capacity, 𝐸¤ ( 𝑡 ) < 0 is the rate of capacity reduction, and 𝑐 ( Φ ( 𝑡 )) > 0 is a scale-invariant factor. a Since the Hilbert norm ∥ 𝑓 ∥
H is non-differentiable at 𝑓 = 0, this integral is defined over paths where the active capacity remains positive 𝐸 ( Φ ( 𝑡 )) > 0.
Proof. 1. Differential Form: By the Infinitesimal Capacity Dependence axiom, the differential cost along a path is driven by capacity reduction: J¤ ( 𝑡 ) = −𝜉 ( Φ ( 𝑡 )) 𝐸¤ ( 𝑡 ) Since active capacity decreases during compression 𝐸¤ ( 𝑡 ) < 0 and the cost rate must be positive J¤ ( 𝑡 ) > 0, we require 𝜉 ( Φ ( 𝑡 )) > 0. 2. Magnitude Neutrality: For any scalar 𝑘 > 0, Magnitude Neutrality requires J ( 𝑘Φ𝑎 , 𝑘Φ𝑏 ) = J ( Φ𝑎 , Φ𝑏 ). By Lemma C.1, capacity scales linearly 𝐸 ( 𝑘Φ) = 𝑘𝐸 ( Φ), so 𝐸¤ ( 𝑘Φ ( 𝑡 )) = 𝑘 𝐸¤ ( 𝑡 ). Integrating over the scaled path yields: ∫ 1
−𝜉 ( 𝑘Φ ( 𝑡 )) 𝑘 𝐸¤ ( 𝑡 ) 𝑑𝑡 =
0
∫ 1
−𝜉 ( Φ ( 𝑡 )) 𝐸¤ ( 𝑡 ) 𝑑𝑡
0
Assuming continuous integrands, since this equality holds for any valid continuous path, the integrands must be identical pointwise: 𝑘 𝜉 ( 𝑘Φ) 𝐸¤ ( 𝑡 ) = 𝜉 ( Φ) 𝐸¤ ( 𝑡 ) =⇒ 𝜉 ( 𝑘Φ) = 𝑘 −1 𝜉 ( Φ)
Thus, 𝜉 is a homogeneous function of degree −1. 3. Scale-Invariant Factor: Define 𝑐 ( Φ) ≜ 𝜉 ( Φ) 𝐸 ( Φ). Scaling the state by 𝑘 yields: 𝑐 ( 𝑘Φ) = 𝜉 ( 𝑘Φ) 𝐸 ( 𝑘Φ) = ( 𝑘 −1 𝜉 ( Φ)) ( 𝑘𝐸 ( Φ)) = 𝑐 ( Φ)
This shows 𝑐 ( Φ) is scale-invariant. Substituting 𝜉 ( Φ) = 𝑐 ( Φ)/ 𝐸 ( Φ) back into the differential form gives the integral: ∫ 1 𝐸¤ ( 𝑡 ) J ( Φ 𝑎 , Φ𝑏 ) = −𝑐 ( Φ ( 𝑡 )) 𝑑𝑡 𝐸 ( Φ ( 𝑡 )) 0
43
Hilbert Operator for Progressive Encoding (HOPE)
4. Connectivity Preservation: This axiom mandates an infinite cost barrier against layer extinction: lim𝐸𝑏 →0+ J ( Φ𝑎 , Φ𝑏 ) = ∞, where 𝐸𝑏 = 𝐸 ( Φ𝑏 ) and 𝐸 𝑎 = 𝐸 ( Φ𝑎 ). Applying the change of variables 𝑑𝐸 = 𝐸¤ ( 𝑡 ) 𝑑𝑡 and reversing the limits (which absorbs the negative sign since 𝐸𝑏 < 𝐸 𝑎 due to 𝐸¤ ( 𝑡 ) < 0) gives: ∫ ∫ 𝐸𝑏
J ( Φ 𝑎 , Φ𝑏 ) =
−𝑐 ( Φ)
𝐸𝑎
𝑑𝐸 = 𝐸
𝐸𝑎
𝑐 ( Φ)
𝐸𝑏
𝑑𝐸 𝐸
If 𝑐 ( Φ) is bounded below by a constant 𝑐min > 0 along the path to extinction: ∫ 𝐸𝑎 𝑑𝐸 J ( Φ 𝑎 , Φ𝑏 ) ≥ = 𝑐min ln( 𝐸 𝑎 ) − ln( 𝐸𝑏 ) 𝑐min 𝐸𝑏
𝐸
Taking the limit as 𝐸𝑏 → 0+ yields ∞, satisfying the axiom.
□
Relative Differential Cost For any differentiable deformation path Φ : [0, 1] → H 𝑁 , the rate of geometric projection cost accumulation J¤proj ( 𝑡 ) is defined as: J¤proj ( 𝑡 ) ≜ 𝑐 ( Φ ( 𝑡 ))
𝑠¤ ( 𝑡 ) 𝐸 ( Φ ( 𝑡 ))
(44)
¤ ( 𝑡 ) 𝑁 ≥ 0 is the instantaneous geometric speed of the state vector. where 𝑠¤( 𝑡 ) = Φ H Derivation. The scalar capacity rate 𝐸¤ ( 𝑡 ) from Theorem C.2 assigns zero penalty to geometric deformations (e.g., orthogonal rotations) where scalar capacity is conserved 𝐸¤ ( 𝑡 ) = 0. To capture structural distortion, we calibrate the criterion along a localized orthogonal path. Consider a compressive path where a single active neuron 𝑓𝑘 is scaled toward 0 by a decreasing scalar 𝛼 ( 𝑡 ) ∈ [0, 1] 𝛼¤ ( 𝑡 ) < 0, while all other neurons remain static. The capacity is 𝐸 ( Φ ( 𝑡 )) = Í 𝛼 ( 𝑡 ) ∥ 𝑓𝑘 ∥ H + 𝑖≠𝑘 ∥ 𝑓𝑖 ∥ H , giving 𝐸¤ ( 𝑡 ) = 𝛼¤ ( 𝑡 ) ∥ 𝑓𝑘 ∥ H < 0. The geometric speed 𝑠¤( 𝑡 ) is the norm of the state derivative vector. Since only the 𝑘-th coordinate changes, this vector is 1-sparse, ensuring the norms coincide: ¤ ( 𝑡 ) 𝑁 = | 𝛼¤ ( 𝑡 )| ∥ 𝑓𝑘 ∥ H = −𝛼¤ ( 𝑡 ) ∥ 𝑓𝑘 ∥ H = − 𝐸¤ ( 𝑡 ) 𝑠¤ ( 𝑡 ) = Φ H ¤ Substituting 𝑠¤( 𝑡 ) = − 𝐸¤ ( 𝑡 ) into the cost baseline J¤capacity ( 𝑡 ) = −𝑐 ( Φ) 𝐸𝐸 yields J¤ ( 𝑡 ) = 𝑐 ( Φ) 𝐸𝑠¤ for this 𝑁 orthogonal axis. Because the ambient space H is isotropic, we define the geometric projection cost J¤proj ( 𝑡 ) as this ratio to generalize to arbitrary trajectories. □
Lemma C.3 (Capacity Bound for Correlated Projections). Let 𝑓𝑖 , 𝑓 𝑗 ∈ H be active candidate neurons ∥ 𝑓𝑖 ∥ H > 0, 𝑓 𝑗 H > 0, and let 𝑓 𝑝 ∈ H be their optimal parent neuron. Let 𝐸rem ≜ ⟨ 𝑓𝑖 , 𝑓 𝑗 ⟩ H 𝐸 𝑎 − ∥ 𝑓𝑖 ∥ H − ∥ 𝑓 𝑗 ∥ H ≥ 0. Define the functional correlation as 𝜌𝑖 𝑗 ≜ . ∥ 𝑓𝑖 ∥ H ∥ 𝑓 𝑗 ∥ H For the continuous straight-line deformation path Φ ( 𝑡 ) in H 𝑁 connecting the initial state Φ𝑎 to the pre-deletion target state Φ̃𝑏 , there exists a correlation threshold 𝜌∗ ( 𝑓𝑖 , 𝑓 𝑗 ) ∈ (0, 1) such that if 𝜌𝑖 𝑗 ≥ 𝜌∗ , then: 𝐸 ( Φ ( 𝑡 )) ≥ 𝐸 ( Φ𝑏 ) ∀ 𝑡 ∈ [0, 1] (45) where 𝐸 ( Φ𝑏 ) = 𝐸rem + 𝑓 𝑝 H is the capacity of the post-deletion terminal state Φ𝑏 ∈ H 𝑁 −1 .
44
Hilbert Operator for Progressive Encoding (HOPE)
Proof. Parameterizing the path as Φ ( 𝑡 ) = (1 − 𝑡 ) Φ𝑎 + 𝑡 Φ̃𝑏 for 𝑡 ∈ [0, 1], the targeted neurons transition via 𝑓𝑖 ( 𝑡 ) = (1 − 𝑡 ) 𝑓𝑖 + 𝑡 𝑓 𝑝 and 𝑓 𝑗 ( 𝑡 ) = (1 − 𝑡 ) 𝑓 𝑗 + 𝑡 𝑓 𝑝 . By Lemma C.1 and the triangle inequality ∥𝑢∥ H + ∥ 𝑣∥ H ≥ ∥𝑢 + 𝑣∥ H : 𝐸 ( Φ ( 𝑡 )) ≥ 𝐸rem + (1 − 𝑡 ) ( 𝑓𝑖 + 𝑓 𝑗 ) + 2𝑡 𝑓 𝑝 H
To prove 𝐸 ( Φ ( 𝑡 )) ≥ 𝐸rem + 𝑓 𝑝 H , we require: (1 − 𝑡 ) ( 𝑓𝑖 + 𝑓 𝑗 ) + 2𝑡 𝑓 𝑝 H > 𝑓 𝑝 H
∀ 𝑡 ∈ [0, 1]
(46)
We evaluate this in the collinear limit 𝜌𝑖 𝑗 → 1. Let 𝑓𝑖 = 𝑥 𝑢ˆ, 𝑓 𝑗 = 𝑦 𝑢ˆ, and 𝑓 𝑝 = 𝑧𝑢ˆ for a shared unit vector 𝑢ˆ and scalars 𝑥, 𝑦, 𝑧 > 0. The condition simplifies to: (1 − 𝑡 ) ( 𝑥 + 𝑦 ) + 2𝑡𝑧 > 𝑧 Since this expression is linear in 𝑡 , its minimum occurs at the boundaries: • At 𝑡 = 1: 2𝑧 > 𝑧, which inherently holds since 𝑧 > 0. 𝑏𝐸rem • At 𝑡 = 0: 𝑥 + 𝑦 > 𝑧. From the optimal scale derivation (Section 7.1), 𝑧 = 2𝑎+𝐸rem +𝑏 . In the collinear 2 2 limit, 𝑎 = 𝑥 + 𝑦 and 𝑏 = 𝑥 + 𝑦 . Thus: 𝑥+𝑦>
𝑥 2 + 𝑦 2 + ( 𝑥 + 𝑦 ) 𝐸rem 2 𝐸rem + 𝑥 + 𝑦
( 𝑥 + 𝑦 ) (2 𝐸rem + 𝑥 + 𝑦 ) > 𝑥 2 + 𝑦 2 + ( 𝑥 + 𝑦 ) 𝐸rem 2 𝐸rem ( 𝑥 + 𝑦 ) + 𝑥 2 + 2 𝑥 𝑦 + 𝑦 2 > 𝑥 2 + 𝑦 2 + 𝐸rem ( 𝑥 + 𝑦 ) 𝐸rem ( 𝑥 + 𝑦 ) + 2 𝑥 𝑦 > 0
Since 𝑥 > 0, 𝑦 > 0, and 𝐸rem ≥ 0, this strict inequality unconditionally holds. Since both endpoints satisfy the strict inequality, it holds for all 𝑡 ∈ [0, 1] in the collinear limit. Because the Hilbert norm and 𝑓 𝑝 are continuous with respect to 𝜌𝑖 𝑗 , this inequality is preserved in a neighborhood around 𝜌𝑖 𝑗 = 1. Thus, there exists a threshold 𝜌∗ ( 𝑓𝑖 , 𝑓 𝑗 ) ∈ (0, 1) ensuring the condition for 𝜌𝑖 𝑗 ≥ 𝜌∗ . □ Theorem C.4 (Discrete Transition Cost Bound). For any structural reduction from an initial state Φ𝑎 ∈ H 𝑁 to a terminal state Φ𝑏 ∈ H 𝑁 −1 (specifically, pruning a neuron, or merging a correlated pair with 𝜌𝑖 𝑗 ≥ 𝜌∗ ), the continuous projection cost Jproj evaluated along the straight-line path to the pre-deletion target Φ̃𝑏 ∈ H 𝑁 is upper-bounded by the discrete proxy Jbound : 𝐷 ( Φ𝑎 , Φ̃𝑏 ) Jproj ( Φ𝑎 , Φ𝑏 ) ≤ 𝑐 ( Φ𝑎 ) ≡ Jbound ( Φ𝑎 , Φ𝑏 ) (47) 𝐸 ( Φ𝑏 ) where 𝐸 ( Φ𝑏 ) is the post-deletion capacity, and 𝐷 ( Φ𝑎 , Φ̃𝑏 ) = Φ𝑎 − Φ̃𝑏 H 𝑁 is the Euclidean distance in the configuration space. Proof. By Definition 44, the cost along Φ ( 𝑡 ) = (1 − 𝑡 ) Φ𝑎 + 𝑡 Φ̃𝑏 is: Jproj ( Φ𝑎 , Φ𝑏 ) =
∫ 1 0
𝑐 ( Φ ( 𝑡 ))
𝑠¤ ( 𝑡 ) 𝑑𝑡 𝐸 ( Φ ( 𝑡 )) 45
Hilbert Operator for Progressive Encoding (HOPE)
Since the active neuron count remains invariant prior to 𝑡 = 1, 𝑐 ( Φ ( 𝑡 )) = 𝑐 ( Φ𝑎 ) almost everywhere on [0, 1] and can be factored out. Next, we bound the dynamic capacity 𝐸 ( Φ ( 𝑡 )): • Pruning: 𝐸 ( Φ ( 𝑡 )) = 𝐸 𝑎 − 𝑡 ∥ 𝑓𝑖 ∥ H . Its minimum is 𝐸 (1) = 𝐸 𝑎 − ∥ 𝑓𝑖 ∥ H = 𝐸 ( Φ𝑏 ). • Merging: Lemma C.3 guarantees 𝐸 ( Φ ( 𝑡 )) ≥ 𝐸 ( Φ𝑏 ) for all 𝑡 ∈ [0, 1]. In both cases, 𝐸 ( Φ ( 𝑡 )) ≥ 𝐸 ( Φ𝑏 ) > 0. Substituting 1/ 𝐸 ( Φ ( 𝑡 )) ≤ 1/ 𝐸 ( Φ𝑏 ) into the integral establishes an upper bound: ∫ 1 𝑐 ( Φ𝑎 ) 𝑠¤ ( 𝑡 ) 𝑑𝑡 (48) Jproj ( Φ𝑎 , Φ𝑏 ) ≤ 𝐸 ( Φ𝑏 ) 0 ¤ ( 𝑡 ) 𝑁 = Φ̃𝑏 − Φ𝑎 𝑁 ≡ 𝐷 ( Φ𝑎 , Φ̃𝑏 ) is constant. Evaluating the The geometric speed 𝑠¤( 𝑡 ) = Φ H H integral yields 𝐷 ( Φ𝑎 , Φ̃𝑏 ), leading to: Jproj ( Φ𝑎 , Φ𝑏 ) ≤ 𝑐 ( Φ𝑎 )
𝐷 ( Φ𝑎 , Φ̃𝑏 ) ≡ Jbound ( Φ𝑎 , Φ𝑏 ) 𝐸 ( Φ𝑏 )
□ Proposition C.5 (Axiomatic Consistency of the Bounded Proxy). The bounded projection cost Jbound ( Φ𝑎 , Φ𝑏 ) = 𝑐 ( Φ𝑎 ) 𝐷 (𝐸Φ( Φ𝑎 ,𝑏Φ̃)𝑏 ) satisfies Axiom 1 (Magnitude Neutrality) and Axiom 2 (Connectivity Preservation). Proof. Axiom 1 (Magnitude Neutrality): For any 𝑘 > 0, we have 𝑐 ( 𝑘Φ𝑎 ) = 𝑐 ( Φ𝑎 ) (scale-invariant by Theorem C.2), 𝐷 ( 𝑘Φ𝑎 , 𝑘 Φ̃𝑏 ) = 𝑘𝐷 ( Φ𝑎 , Φ̃𝑏 ), and 𝐸 ( 𝑘Φ𝑏 ) = 𝑘𝐸 ( Φ𝑏 ) (linear scaling by Lemma C.1). Thus, 𝑘 cancels out: 𝑘𝐷 ( Φ𝑎 , Φ̃𝑏 ) Jbound ( 𝑘Φ𝑎 , 𝑘Φ𝑏 ) = 𝑐 ( Φ𝑎 ) = Jbound ( Φ𝑎 , Φ𝑏 ) 𝑘𝐸 ( Φ𝑏 ) Axiom 2 (Connectivity Preservation): For an initial state with 𝐸 ( Φ𝑎 ) > 0, we evaluate lim𝐸 ( Φ𝑏 )→0+ Jbound ( Φ𝑎 , Φ𝑏 ). + =⇒ 𝐸 ( Φ̃ ) → 0. By the definitions of pruning and merging, 𝐸 ( Φ̃𝑏 ) ≤ 2 𝐸 ( Φ𝑏 ).10 Thus, 𝐸 ( Φ𝑏 ) → 0√ 𝑏 Because norms are equivalent on a finite-dimensional space, ∥ Φ ∥ H 𝑁 ≤ 𝐸 ( Φ) ≤ 𝑁 ∥ Φ ∥ H 𝑁 , which implies Φ̃𝑏 H 𝑁 → 0.
Applying the reverse triangle inequality gives: 𝐷 ( Φ𝑎 , Φ̃𝑏 ) = Φ𝑎 − Φ̃𝑏 H 𝑁 ≥ ∥ Φ𝑎 ∥ H 𝑁 − Φ̃𝑏 H 𝑁
Taking the limit as Φ̃𝑏 H 𝑁 → 0 yields a positive lower bound: 1 lim + 𝐷 ( Φ𝑎 , Φ̃𝑏 ) = ∥ Φ𝑎 ∥ H 𝑁 ≥ √ 𝐸 ( Φ𝑎 ) > 0
𝐸 ( Φ𝑏 )→0
𝑁
Since the numerator is bounded below by a positive constant, dividing by 𝐸 ( Φ𝑏 ) → 0+ strictly diverges to ∞. □ 10 For pruning, 𝐸 ( Φ̃ ) = 𝐸 ( Φ ). For merging, 𝐸 ( Φ̃ ) = 𝐸 𝑏
𝑏
𝑏
rem + 2
𝑓 𝑝 H ≤ 2( 𝐸rem + 𝑓 𝑝 H ) = 2 𝐸 ( Φ𝑏 ).
46
Hilbert Operator for Progressive Encoding (HOPE)
Corollary C.6 (Locality of the Projection Error). For a structural reduction modifying a localized subset of neurons S ⊂ {1, . . . , 𝑁 } (e.g., |S| = 1 for pruning, |S| = 2 for merging), the discrete projection bound evaluates over the perturbed subspace: √︂ 2 Í ( 𝑎) − 𝑓˜𝑘( 𝑏 ) 𝑘 ∈ S 𝑓𝑘 H Jbound ( Φ𝑎 , Φ𝑏 ) = 𝑐 ( Φ𝑎 ) (49) 𝐸 ( Φ𝑏 ) where 𝑓˜𝑘( 𝑏 ) are the components of the pre-deletion target state Φ̃𝑏 ∈ H 𝑁 .
Proof. By Theorem C.4, the Euclidean distance expands as: v t ∑︁ ∑︁ 2 2 ( 𝑎) ( 𝑎) (𝑏) (𝑏) 𝐷 ( Φ𝑎 , Φ̃𝑏 ) = 𝑓𝑘 − 𝑓˜𝑘 𝑓 𝑗 − 𝑓˜𝑗 + H
𝑘∈ S
H
𝑗∉S
For any unperturbed coordinate 𝑗 ∉ S, 𝑓 𝑗( 𝑎 ) = 𝑓˜𝑗( 𝑏 ) , meaning the second summation vanishes. The terminal capacity 𝐸 ( Φ𝑏 ) is also computed locally via 𝐸 ( Φ𝑏 ) = 𝐸 ( Φ𝑎 )−
Í
( 𝑎)
𝑘∈ S
𝑓𝑘
Í + 𝑚 ∈ Snew 𝑓𝑚( 𝑏 )
H where Snew represents newly generated components (e.g., { 𝑓 𝑝 }). Because |S| and |Snew | depend solely
on the localized operation, evaluating Jbound requires O (1) operations, given 𝐸 ( Φ𝑎 ) is cached.
□
C.2. Generating Parent Neuron Theorem C.7 (Exact Optimal Parent Direction). For a fixed magnitude 𝑠 > 0, the inner optimization for merging neurons 𝑓𝑖 , 𝑓 𝑗 ∈ N : √︃ ∗
𝜓 = argmin 𝜓 ∈ N ∥ 𝜓 ∥ H =1
2
∥ 𝑠𝜓 − 𝑓𝑖 ∥ 2H + 𝑠𝜓 − 𝑓 𝑗 H
𝐸 𝑎 − ∥ 𝑓𝑖 ∥ H − 𝑓 𝑗 H + 𝑠
(50)
is minimized by the parameterized function: Ψ ( 𝒖∗ · 𝒙˜) ∗ 𝒗 𝜓∗ = √︁ 𝐾 ( 𝒖∗ , 𝒖∗ )
where the unit vectors 𝒖∗ ∈ ℝ𝑛+1 and 𝒗∗ ∈ ℝ𝑐 are given by: Í 𝑘 𝑘 ˜ in ) 𝒘out 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖, 𝒘 2 ∗ 𝒖 = argmax ∥ 𝒖 ∥ 2 =1 √︁ 𝐾 ( 𝒖, 𝒖) 𝐾 ( 𝒖,𝒖 ) >0 ∗
Í
𝒗 = Í
∗ ˜ 𝑘 ) 𝒘𝑘 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖 , 𝒘 out in
∗ ˜ 𝑘 ) 𝒘𝑘 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖 , 𝒘 out 2 in
(51)
(52)
(53)
47
H
,
Hilbert Operator for Progressive Encoding (HOPE)
Proof. Since 𝑠 > 0 and the denominator of (50) is positive and independent of 𝜓 (due to ∥ 𝜓 ∥ H = 1), minimizing the objective is equivalent to minimizing the squared numerator: 2
2
∥ 𝑠𝜓 − 𝑓𝑖 ∥ 2H + 𝑠𝜓 − 𝑓 𝑗 H = 2 ∥ 𝑠𝜓 ∥ 2H + ∥ 𝑓𝑖 ∥ 2H + 𝑓 𝑗 H − 2 𝑠𝜓, 𝑓𝑖 + 𝑓 𝑗 H 2
= 2𝑠2 + ∥ 𝑓𝑖 ∥ 2H + 𝑓 𝑗 H − 2𝑠 𝜓, 𝑓𝑖 + 𝑓 𝑗 H
(54)
Because 𝑠 > 0 and ∥ 𝑓𝑖 ∥ H , 𝑓 𝑗 H are constant with respect to 𝜓, this reduces to maximizing the inner product: 𝜓∗ = argmax 𝜓 ∈ N 𝜓, 𝑓𝑖 + 𝑓 𝑗 H (55) ∥ 𝜓 ∥ H =1
To enforce 𝜓 ∈ N , we decompose it into unit directions 𝒖 ∈ ℝ𝑛+1 , 𝒗 ∈ ℝ𝑐 and magnitudes 𝛼, 𝛽 > 0. By the PH-1 property of Ψ, 𝜓 = 𝛽𝒗Ψ ( 𝛼𝒖 · 𝒙˜) = 𝛼𝛽𝒗Ψ ( 𝒖 · 𝒙˜). The unit-norm constraint ∥ 𝜓 ∥ H = 1 requires: 1 1 = ( 𝛼𝛽 ) 2 ∥ 𝒗 ∥ 22 𝔼𝒙∼𝑃X Ψ2 ( 𝒖 · 𝒙˜) = ( 𝛼𝛽 ) 2 𝐾 ( 𝒖, 𝒖) =⇒ 𝛼𝛽 = √︁ 𝐾 ( 𝒖, 𝒖) Assuming 𝐾 (𝒖, 𝒖) > 0 (otherwise 𝜓 = 0), substituting 𝛼𝛽 gives: Ψ ( 𝒖 · 𝒙˜) 𝜓 = √︁ 𝒗 𝐾 ( 𝒖, 𝒖)
Substituting this into (55) and expanding the tensor inner product ⟨𝑔 ⊗ 𝒗, ℎ ⊗ 𝒘⟩ H = ⟨𝑔, ℎ⟩ Hin ⟨𝒗, 𝒘⟩ 2 yields: + + * * ∑︁ Ψ (𝒖 · 𝒙˜) Ψ ( 𝒖 · 𝒙˜) 𝑘 𝑘 Ψ (𝒘 ˜ in · 𝒙˜) 𝒗, 𝑓𝑖 + 𝑓 𝑗 = 𝒗, 𝒘out √︁ √︁ 𝐾 ( 𝒖, 𝒖) 𝐾 ( 𝒖, 𝒖 ) 𝑘 ∈ { 𝑖, 𝑗 } H H ∑︁ 1 𝑘 𝑘 = √︁ 𝔼𝒙∼𝑃X Ψ (𝒖 · 𝒙˜) Ψ ( 𝒘 ˜ in · 𝒙˜) 𝒗, 𝒘out 2 𝐾 ( 𝒖, 𝒖) 𝑘 ∈ { 𝑖, 𝑗 } + * ∑︁ 1 𝑘 𝑘 ) 𝒘out (56) 𝐾 ( 𝒖, 𝒘 ˜ in = 𝒗, √︁ 𝐾 ( 𝒖, 𝒖) 𝑘 ∈ { 𝑖, 𝑗 } 2
Í 𝑘 𝑘 Let 𝒛 ( 𝒖) = 𝑘 ∈ { 𝑖, 𝑗 } 𝐾 ( 𝒖, 𝒘 ˜ in ) 𝒘out . For a fixed 𝒖, maximizing ⟨𝒗, 𝒛 ( 𝒖)⟩ 2 subject to ∥ 𝒗 ∥ 2 = 1 requires 𝒗 to align with 𝒛 ( 𝒖) via the Cauchy-Schwarz inequality: 𝒗∗ ( 𝒖) =
𝒛 ( 𝒖) ∥ 𝒛 ( 𝒖)∥ 2
√︁ Substituting 𝒗∗ ( 𝒖) back into (56) simplifies the inner product to ∥ 𝒛 ( 𝒖)∥ 2 / 𝐾 ( 𝒖, 𝒖), leaving 𝒖∗ as the sole maximizer in Equation (52). □ Proposition C.8 (Separability and Homogeneity of PH-1 Kernels). Let Ψ : ℝ → ℝ be a PH-1 activation function ∀𝑐 > 0, Ψ ( 𝑐𝑧) = 𝑐Ψ ( 𝑧). Let the functional kernel over an isotropic probability distribution 𝑃 V be 𝐾 ( 𝒙, 𝒚 ) = 𝔼𝒗∼𝑃V [ Ψ ( 𝒙𝑇 𝒗) Ψ ( 𝒚 𝑇 𝒗)]. For any unit vector 𝒙 ∥ 𝒙 ∥ 2 = 1 and any 𝒚 ≠ 0, the kernel factorizes as: 𝒚 𝐾 ( 𝒙, 𝒚 ) = ∥ 𝒚 ∥ 2 𝑘 ( 𝜌) where 𝜌 = 𝒙, (57) ∥ 𝒚 ∥2 2
48
Hilbert Operator for Progressive Encoding (HOPE)
Proof. Let ˆ𝒚 = 𝒚 /∥ 𝒚 ∥ 2 . By the PH-1 property and linearity of expectation, the positive magnitude ∥ 𝒚 ∥ 2 factors out: 𝐾 ( 𝒙, 𝒚 ) = 𝔼𝒗∼ 𝑃V [ Ψ ( 𝒙𝑇 𝒗) Ψ (∥ 𝒚 ∥ 2 ˆ 𝒚 𝑇 𝒗)] = ∥ 𝒚 ∥ 2 𝔼𝒗∼ 𝑃V [ Ψ ( 𝒙𝑇 𝒗) Ψ ( ˆ 𝒚 𝑇 𝒗)]
Since 𝑃 V is isotropic, it is invariant under orthogonal transformations. Let 𝑹 be an orthogonal √︁ matrix such that 𝑹𝒙 = 𝒆1 and 𝑹 ˆ𝒚 = 𝜌𝒆1 + 1 − 𝜌2 𝒆2 , where 𝜌 = ⟨𝒙, ˆ𝒚 ⟩ 2 . Applying the change of variables 𝒖 = 𝑹𝒗, we have√︁𝒖 ∼ 𝑃 V . Substituting 𝒙𝑇 𝒗 = 𝒙𝑇 𝑹𝑇 𝒖 = ( 𝑹𝒙 )𝑇 𝒖 = 𝒆1 𝒖 = 𝑢1 and similarly ˆ 𝒚𝑇 𝒗 = ˆ 𝒚 𝑇 𝑹𝑇 𝒖 = ( 𝑹 ˆ 𝒚 ) 𝑇 𝒖 = 𝜌𝑢1 + 1 − 𝜌2 𝑢2 yields: 𝑇
√︁ 𝔼𝒗∼𝑃V [ Ψ ( 𝒙𝑇 𝒗) Ψ ( ˆ𝒚 𝑇 𝒗)] = 𝔼𝒖∼𝑃V [ Ψ (𝑢1 ) Ψ ( 𝜌𝑢1 + 1 − 𝜌2 𝑢2 )] Because this expectation depends only on 𝜌, we can define it as 𝑘 ( 𝜌), establishing the separable form 𝐾 ( 𝒙, 𝒚 ) = ∥ 𝒚 ∥ 2 𝑘 ( 𝜌). □ Proposition C.9 (Boundary Identity). Let Ψ be a PH-1 function. Under a standard bivariate Gaussian reference distribution with correlation 𝜌, the induced angular kernel 𝑘 ( 𝜌) = 𝔼[ Ψ ( 𝑋 ) Ψ (𝑌 )] satisfies 𝑘 (1) = 𝑘′ (1). Proof. By the PH-1 property, Ψ ( 𝑥 ) = 𝐶+ 𝑥 𝕀𝑥>0 + 𝐶 − 𝑥 𝕀𝑥<0 for constants 𝐶+ = Ψ (1) and 𝐶 − = −Ψ (−1). Its derivative (defined almost everywhere) is Ψ′ ( 𝑥 ) = 𝐶+ 𝕀𝑥>0 + 𝐶 − 𝕀𝑥<0 . Let ( 𝑋, 𝑌 ) be standard bivariate Gaussian with correlation 𝜌. At 𝜌 = 1, 𝑋 = 𝑌 almost surely, 𝜕 giving 𝑘 (1) = 𝔼[ Ψ ( 𝑋 ) 2 ]. By Price’s Theorem 𝜕𝜌 𝔼[ 𝑓 ( 𝑋 ) 𝑔 (𝑌 )] = 𝔼[ 𝑓 ′ ( 𝑋 ) 𝑔 ′ (𝑌 )], we have 𝑘′ ( 𝜌) = 𝔼[ Ψ′ ( 𝑋 ) Ψ′ (𝑌 )], which at 𝜌 = 1 gives 𝑘′ (1) = 𝔼[ Ψ′ ( 𝑋 ) 2 ]. Evaluating these expectations: 𝑘 (1) = 𝔼 ( 𝐶+ 𝑋 𝕀𝑋 >0 + 𝐶 − 𝑋 𝕀𝑋 <0 ) 2 = 𝐶+2 𝔼[ 𝑋 2 𝕀𝑋 >0 ] + 𝐶 2− 𝔼[ 𝑋 2 𝕀𝑋 <0 ] 𝑘′ (1) = 𝔼 ( 𝐶+ 𝕀𝑋 >0 + 𝐶 − 𝕀𝑋 <0 ) 2 = 𝐶+2 𝔼[𝕀𝑋 >0 ] + 𝐶 2− 𝔼[𝕀𝑋 <0 ] By the symmetry of the standard normal distribution, 𝔼[ 𝑋 2 𝕀𝑋 >0 ] = 𝔼[ 𝑋 2 𝕀𝑋 <0 ] = 12 𝔼[ 𝑋 2 ] = 12 and 𝔼[𝕀𝑋 >0 ] = 𝔼[𝕀𝑋 <0 ] = 12 . Substituting these yields 𝑘 (1) = 21 ( 𝐶+2 + 𝐶 2− ) and 𝑘′ (1) = 21 ( 𝐶+2 + 𝐶 2− ). Thus, 𝑘 (1) = 𝑘′ (1). □ Proposition C.10 (Kernel Positivity at Boundary). For any non-trivial PH-1 function Ψ ( 𝑥 ) . 0, the induced angular kernel satisfies 𝑘 (1) > 0. Proof. As derived in Proposition C.9, 𝑘 (1) = 12 ( 𝐶+2 + 𝐶 2− ). Since Ψ is non-trivial, at least one of 𝐶+ or 𝐶 − is non-zero. Consequently, 𝐶+2 + 𝐶 2− > 0, implying 𝑘 (1) > 0. □
49
Hilbert Operator for Progressive Encoding (HOPE)
C.3. Block Eviction Proposition C.11. Let Ω = ( Φ1 , . . . , Φ𝑛 ) be a macroscopic state, where each Φ𝑖 ∈ H𝑖𝑁𝑖 is a layer state consisting of rank-1 operators 𝑓𝑖, 𝑗 ∈ H𝑖 . If the macroscopic capacity 𝐸 ( Ω) is a symmetric, separable, and homogeneous function of the capacities of its constituent operators, and is invariant under arbitrary operator partitioning, it is uniquely defined as the sum of the constituent Hilbert-Schmidt norms: 𝐸 ( Ω) =
𝑛 ∑︁
𝐸 ( Φ𝑖 ) =
𝑁𝑖 𝑛 ∑︁ ∑︁
(58)
𝑓𝑖, 𝑗 H . 𝑖
𝑖=1
𝑖=1 𝑗=1
Proof. Because 𝐸 ( Ω) is a symmetric, separable, and homogeneous functional of the individual capacities, it must take the form of an 𝐿 𝑝 norm: 1 𝑝 𝑛 𝑁𝑖 𝑝 ª ©∑︁ ∑︁ 𝐸 ( Ω) = 𝑓𝑖, 𝑗 H ® 𝑖 𝑖 =1 𝑗 =1 « ¬
for some 𝑝 > 0 .
(59)
The capacity invariance axiom requires that 𝐸 ( Ω) remains invariant if any operator is partitioned. Consider an arbitrary operator 𝑓0 ∈ Ω. Partitioning 𝑓0 into 𝑀 identical fractional operators yields 𝑀 operators, each defined as 𝑓0 / 𝑀 . By the positive homogeneity of the Hilbert-Schmidt norm, each fractional operator has capacity ∥ 𝑓0 / 𝑀 ∥ H = 𝑀1 ∥ 𝑓0 ∥ H . Evaluating the capacity of this partitioned subset under the 𝐿 𝑝 functional yields:
𝐸subset =
𝑀 𝑝 ∑︁ 𝑓0 𝑚=1
𝑀
! 1𝑝
= 𝑀
𝑝 1𝑝
1 𝑀
H
∥ 𝑓0 ∥ H
1− 𝑝
= 𝑀 𝑝 ∥ 𝑓0 ∥ H .
(60)
For the macroscopic capacity to remain invariant for any partition scale 𝑀 ≥ 1, we must have 1− 𝑝
𝐸subset = ∥ 𝑓0 ∥ H . This equality holds if and only if 𝑀 𝑝 = 1. Since 𝑀 is arbitrary, the exponent must
be zero:
1− 𝑝 𝑝
= 0 =⇒ 𝑝 = 1 .
Substituting 𝑝 = 1 reduces the 𝐿 𝑝 norm to the 𝐿1 sum of scalar norms, yielding 𝐸 ( Ω) =
(61) Í𝑛 Í 𝑁 𝑖 𝑖=1
𝑗=1
𝑓𝑖, 𝑗 H . 𝑖
□
D. Derivation of Physical BN Parameters This appendix provides the complete derivation for recovering the physical BN parameters and raw weights from the effective parameters, expanding upon Section 7.2.2. Since the parent direction b 𝒖 lies within the 2D subspace spanned by the augmented children, there exist projection coefficients 𝑐1 , 𝑐2 such that: eff eff 𝒘eff 𝑝,in = 𝑐1 𝒘in,𝑖 + 𝑐2 𝒘in, 𝑗
and
𝑏 𝑝 = 𝑐1 𝑏 𝑖 + 𝑐2 𝑏 𝑗 .
(62)
50
Hilbert Operator for Progressive Encoding (HOPE)
eff ) 𝑇 𝒙 + 𝑏, it follows that 𝑦 = 𝑐 𝑦 + 𝑐 𝑦 . The By the definition of the pre-activation signal 𝑦 = ( 𝒘in 𝑝 1 𝑖 2 𝑗 physical BN parameters 𝛽 𝑝 and 𝛾 𝑝 correspond to the mean and standard deviation of 𝑦 𝑝 . Taking the expectation and variance yields:
𝛽 𝑝 = 𝔼[ 𝑦 𝑝 ] = 𝑐1 𝔼[ 𝑦𝑖 ] + 𝑐2 𝔼[ 𝑦 𝑗 ] = 𝑐1 𝛽𝑖 + 𝑐2 𝛽 𝑗
(63)
2
(64)
𝛾 𝑝 = Var[ 𝑦 𝑝 ] = 𝑐12 𝛾𝑖2 + 𝑐22 𝛾 2𝑗 + 2𝑐1 𝑐2 Cov( 𝑦𝑖 , 𝑦 𝑗 ) .
Substituting the closed-form covariance Cov( 𝑦𝑖 , 𝑦 𝑗 ) = | 𝛾𝑖 || 𝛾 𝑗 | 𝜌ˆ𝑖 𝑗 (where 𝜌ˆ𝑖 𝑗 is defined in Equation 4) gives: √︃ ˆ𝑖 𝑗 . 𝛾 𝑝 = ± 𝑐12 𝛾𝑖2 + 𝑐22 𝛾 2𝑗 + 2𝑐1 𝑐2 | 𝛾𝑖 || 𝛾 𝑗 | 𝜌
(65)
raw and dataset statistics 𝜇 , 𝜎 . By definition (1): Next, we determine the raw weights 𝒘in 𝑝 𝑝 ,𝑝
𝛾𝑝 eff raw 𝒘in 𝒘in ,𝑝 = √︃ ,𝑝 𝜎2𝑝 + 𝜖
and
𝛾𝑝 𝑏 𝑝 = 𝛽 𝑝 − √︃ 𝜇𝑝 . 2 𝜎𝑝 + 𝜖
(66)
This system is under-constrained. To resolve the ambiguity, we anchor the variance such that 𝜎2𝑝 = max(0, 𝛾 2𝑝 − 𝜖). As noted in the main text, we focus on the active regime 𝛾 2𝑝 ≥ 𝜖 and defer the edge case 𝛾 2𝑝 < 𝜖 to Appendix B.5. In the active regime, the physical scaling factor simplifies directly to its sign: 𝛾𝑝
√︃
𝜎2𝑝 + 𝜖
=
𝛾𝑝
|𝛾𝑝 |
= sign( 𝛾 𝑝 ) .
(67)
Substituting this back into (66) yields: eff raw 𝒘in ,𝑝 = sign( 𝛾 𝑝 ) 𝒘in,𝑝
and
𝑏 𝑝 = 𝛽 𝑝 − sign( 𝛾 𝑝 ) 𝜇 𝑝 .
(68)
Multiplying both equations by sign( 𝛾 𝑝 ) (and noting that sign( 𝛾 𝑝 ) 2 = 1 for 𝛾 𝑝 ≠ 0) isolates the raw physical parameters: raw eff 𝒘in ,𝑝 = sign( 𝛾 𝑝 ) 𝒘in,𝑝
and
𝜇 𝑝 = sign( 𝛾 𝑝 ) ( 𝛽 𝑝 − 𝑏 𝑝 ) .
(69)
Because 𝛾 𝑝 in (65) can take either sign, we arbitrarily choose the positive root 𝛾 𝑝 > 0 without loss of generality. Under this choice, sign( 𝛾 𝑝 ) = 1, and the recovered parameters simplify to: √︃ raw eff 𝒘in,𝑝 = 𝒘in,𝑝 , 𝜇 𝑝 = 𝑐1 𝛽 𝑖 + 𝑐2 𝛽 𝑗 − 𝑏 𝑝 , 𝜎 𝑝 = 𝛾 𝑝 = 𝑐12 𝛾𝑖2 + 𝑐22 𝛾 2𝑗 + 2𝑐1 𝑐2 | 𝛾𝑖 || 𝛾 𝑗 | 𝜌 ˆ𝑖 𝑗 . (70) To prove that the physical forward pass is invariant to the chosen sign of 𝛾 𝑝 , we substitute the recovered parameters back into the BN inference equation. The true pre-activation signal 𝑦 𝑝 is computed as: 𝛾𝑝 raw 𝑇 𝑦 𝑝 = √︃ ( 𝒘in ) 𝒙 − 𝜇 + 𝛽𝑝 𝑝 ,𝑝 𝜎2𝑝 + 𝜖 h i eff 𝑇 = sign( 𝛾 𝑝 ) sign( 𝛾 𝑝 ) 𝒘in 𝒙 − sign( 𝛾 ) ( 𝛽 − 𝑏 ) + 𝛽𝑝 𝑝 𝑝 𝑝 ,𝑝 h i eff 𝑇 = sign( 𝛾 𝑝 ) 2 ( 𝒘in (71) ,𝑝 ) 𝒙 − ( 𝛽 𝑝 − 𝑏 𝑝 ) + 𝛽 𝑝 . Since sign( 𝛾 𝑝 ) 2 = 1, the shift 𝛽 𝑝 cancels out: eff 𝑇 eff 𝑇 𝑦 𝑝 = ( 𝒘in ,𝑝 ) 𝒙 − 𝛽 𝑝 + 𝑏 𝑝 + 𝛽 𝑝 = ( 𝒘in,𝑝 ) 𝒙 + 𝑏 𝑝 .
(72)
Thus, regardless of the polarity of 𝛾 𝑝 , the deployed pre-activation robustly realizes the target effective geometry. 51
Hilbert Operator for Progressive Encoding (HOPE)
E. Kernel Formulation Expressing the Hilbert space inner products and norms via a kernel decouples finite-dimensional vector operations from infinite-dimensional function integrals. We define the correlation kernel 𝐾 ( 𝑖, 𝑗) of two neurons as: 𝐾 ( 𝑖, 𝑗) ≜ 𝔼𝒙 ∼ 𝑃X [ Ψ ( 𝑦𝑖 ) Ψ ( 𝑦 𝑗 )] (73) Under this definition, the inner product in H evaluates to: 𝑓𝑖 , 𝑓 𝑗 H = 𝑔𝑖 ⊗ 𝒘out,𝑖 , 𝑔 𝑗 ⊗ 𝒘out, 𝑗 H
= 𝑔𝑖 , 𝑔 𝑗 H
in
𝒘out,𝑖 , 𝒘out, 𝑗 ℝ𝑐
= 𝐾 ( 𝑖, 𝑗) 𝒘out,𝑖 , 𝒘out, 𝑗 ℝ𝑐
(74)
Consequently, the capacity of a single neuron is formulated as: √︁ ∥ 𝑓𝑖 ∥ H = 𝒘out,𝑖 2 𝐾 ( 𝑖, 𝑖)
(75)
where 𝐾 ( 𝑖, 𝑖) = 𝔼𝒙∼𝑃X [ Ψ ( 𝑦𝑖 ) 2 ] represents the expected squared energy of the activation signal. Evaluating 𝐾 ( 𝑖, 𝑗) requires integrating over a high-dimensional space. We derive closed-form analytical expressions for these integrals under the ReLU activation.11 E.1. Pre-Activation Distribution To evaluate the integrals analytically, we first determine the distribution of the pre-activation signal eff ) 𝑇 𝒙 + 𝑏 under the surrogate distribution 𝑃 = N ( 𝝁 𝑦𝑖 = ( 𝒘in ˆ 𝑥 , Σ̂𝑥 ). 𝑖 X ,𝑖 Because 𝑦𝑖 is an affine transformation of a Gaussian vector, it is univariate Gaussian. Recall from + 𝝁 . Because 𝒘𝑇 Section 4 that the surrogate mean is defined as 𝝁 ˆ 𝑥 = 𝑾raw ˆ 𝑥 is the 𝑖-th element BN raw,𝑖 𝝁 + of 𝑾raw 𝝁 ˆ 𝑥 , we have 𝑾raw 𝝁 ˆ 𝑥 = 𝑾raw𝑾raw 𝝁BN . Since the empirical BN mean vector 𝝁BN resides in the column space of 𝑾raw , this projection simplifies to 𝝁BN . Thus, the expected value of the raw projection recovers the empirical mean: 𝔼[ 𝒘𝑇raw,𝑖 𝒙 ] = 𝜇 𝑖 . Similarly, the surrogate covariance Σ̂𝑥 is constrained to satisfy Var( 𝒘𝑇raw,𝑖 𝒙 ) = 𝜎2𝑖 . 𝛾𝑖 𝜇 𝑖 eff = √︃ 𝛾𝑖 𝒘 Using the effective parameters 𝒘in , the mean of 𝑦𝑖 evaluates to: raw,𝑖 and 𝑏𝑖 = 𝛽 𝑖 − √︃ ,𝑖 𝜎2𝑖 +𝜖
𝔼[ 𝑦𝑖 ] = √︃
𝛾𝑖
𝜎2𝑖 + 𝜖
𝜎2𝑖 +𝜖
𝔼[ 𝒘𝑇raw,𝑖 𝒙 ] + 𝑏𝑖 = √︃
𝛾𝑖 𝜇 𝑖
𝜎2𝑖 + 𝜖
+ 𝛽𝑖 − √︃
𝛾𝑖 𝜇 𝑖
𝜎2𝑖
!
= 𝛽𝑖
(76)
𝜎2𝑖 + 𝜖
Similarly, its variance evaluates to: Var( 𝑦𝑖 ) =
𝛾𝑖2 𝜎2𝑖 + 𝜖
Var( 𝒘𝑇raw,𝑖 𝒙 ) = 𝛾𝑖2
𝜎2𝑖 + 𝜖
(77)
Assuming the numerical stability constant 𝜖 is negligible compared to the data variance 𝜖 → 0, the variance simplifies to 𝛾𝑖2 . Thus, under 𝑃 X , the pre-activation density is: 𝑦𝑖 ∼ N ( 𝛽𝑖 , 𝛾𝑖2 )
(78)
11 This evaluation connects to the Arc-Cosine Kernel family of order 𝑛 = 1 [Cho and Saul, 2009]. Because HOPE relies
on the translation vector 𝛽 to capture the full statistical state of the signal, we provide the formal proof for the biased formulation here to ensure the paper remains self-contained.
52
Hilbert Operator for Progressive Encoding (HOPE)
Realizing Empirical Constraints in Practice Adapting HOPE to unnormalized networks requires a lightweight empirical calibration pass over a small, unlabeled data batch to measure the marginal pre-activation statistics ( 𝜇 𝑖 , 𝜎2𝑖 ). Defining the effective scaling as 𝛾𝑖 = 𝜎𝑖 and the shift as 𝛽𝑖 = 𝜇 𝑖 + 𝑏raw,𝑖 anchors the evaluation to the data manifold and recovers the decoupling required to evaluate the kernel analytically.a a Networks using LayerNorm or GroupNorm still require this empirical calibration pass to determine the marginal
channel statistics. Because these layers condition the signal variance, their calibration converges with remarkably few samples, and the resulting pre-activations adhere to the Gaussian surrogate assumptions.
E.2. Self-Kernel Proposition E.1 (Dimensionality Reduction). For a Gaussian pre-activation 𝑦𝑖 ∼ N ( 𝛽𝑖 , 𝛾𝑖2 ) and any activation function Ψ, the expected energy 𝐾 ( 𝑖, 𝑖) = 𝔼[ Ψ ( 𝑦𝑖 ) 2 ] reduces to a 1D integral, independent of the input space dimensionality. Proof. By definition, 𝐾 ( 𝑖, 𝑖) = 𝔼𝒙∼𝑃X [ Ψ ( 𝑦𝑖 ) 2 ]. Because 𝑦𝑖 is a scalar random variable with density ∫∞ 𝑝 ( 𝑦𝑖 ) = N ( 𝑦𝑖 ; 𝛽𝑖 , 𝛾𝑖2 ), the expectation evaluates as −∞ Ψ ( 𝑦 ) 2 𝑝 ( 𝑦 ) 𝑑 𝑦 , decoupling the computation from the ambient space X. □ Theorem E.2 (Closed-Form Self-Kernel for ReLU). For 𝑦𝑖 ∼ N ( 𝛽𝑖 , 𝛾𝑖2 ) with | 𝛾𝑖 | > 0, the expected squared energy for Ψ ( 𝑦𝑖 ) = max(0, 𝑦𝑖 ) evaluates to: 𝛽𝑖 𝛽𝑖 2 2 + 𝛽 𝑖 | 𝛾𝑖 | 𝜙 (79) 𝐾 ( 𝑖, 𝑖) = ( 𝛾𝑖 + 𝛽𝑖 ) Φ | 𝛾𝑖 | | 𝛾𝑖 | where 𝜙 and Φ are the standard Normal PDF and CDF.
Proof. By Proposition E.1, 𝐾 ( 𝑖, 𝑖) =
∫∞ 0
𝑦 2 𝑝 ( 𝑦 ) 𝑑 𝑦 . Applying the substitution 𝑧 =
𝑦 − 𝛽𝑖 | 𝛾𝑖 | , we have
𝛽
𝑦 = | 𝛾𝑖 | 𝑧 + 𝛽𝑖 and 𝑑 𝑦 = | 𝛾𝑖 | 𝑑𝑧 . Setting the integration limit 𝑐 = | 𝛾𝑖𝑖 | , we obtain:
∫ ∞ 𝐾 ( 𝑖, 𝑖) =
(| 𝛾𝑖 | 𝑧 + 𝛽𝑖 ) 2 𝜙 ( 𝑧) 𝑑𝑧 = 𝛽𝑖2
−𝑐
∫ ∞
∫ ∞ 𝜙 ( 𝑧 ) 𝑑𝑧 + 2 𝛽𝑖 | 𝛾𝑖 |
−𝑐
𝑧𝜙 ( 𝑧 ) 𝑑𝑧 + 𝛾𝑖2
−𝑐
∫ ∞
𝑧 2 𝜙 ( 𝑧 ) 𝑑𝑧
−𝑐
We evaluate each term using integration by parts and the identity 𝜙′ ( 𝑧) = − 𝑧𝜙 ( 𝑧): ∫∞ • − 𝑐 𝜙 ( 𝑧 ) 𝑑𝑧 = 1 − Φ (−𝑐) = Φ ( 𝑐) ∫∞ • − 𝑐 𝑧𝜙 ( 𝑧) 𝑑𝑧 = [−𝜙 ( 𝑧)] ∞ − 𝑐 = 𝜙 (∫𝑐) ∫∞ ∞ 2 • − 𝑐 𝑧 𝜙 ( 𝑧) 𝑑𝑧 = [− 𝑧𝜙 ( 𝑧)] ∞ − 𝑐 + − 𝑐 𝜙 ( 𝑧 ) 𝑑𝑧 = Φ ( 𝑐) − 𝑐𝜙 ( 𝑐) Substituting these evaluations yields: 𝐾 ( 𝑖, 𝑖) = 𝛽𝑖2 Φ ( 𝑐) + 2 𝛽𝑖 | 𝛾𝑖 | 𝜙 ( 𝑐) + 𝛾𝑖2 Φ ( 𝑐) − 𝑐𝜙 ( 𝑐)
= ( 𝛾𝑖2 + 𝛽𝑖2 ) Φ ( 𝑐) + 2 𝛽𝑖 | 𝛾𝑖 | − 𝛾𝑖2 𝑐 𝜙 ( 𝑐) Since 𝛾𝑖2 𝑐 = 𝛾𝑖2 ( | 𝛾𝛽𝑖𝑖 | ) = | 𝛾𝑖 | 𝛽𝑖 , the coefficient for 𝜙 ( 𝑐) simplifies to 𝛽𝑖 | 𝛾𝑖 |. Substituting 𝑐 = | 𝛾𝛽𝑖𝑖 | completes the proof. □ 53
Hilbert Operator for Progressive Encoding (HOPE)
E.3. Cross-Kernel E.3.1. The Local Pairwise Surrogate Distribution D
Let 𝜌eff =
eff ,𝒘eff 𝒘in ,𝑖 in, 𝑗
eff 𝒘in ,𝑖
2
E
eff 𝒘in ,𝑗
be the cosine similarity between the effective input weights. The cross-kernel 2
relies on the joint pre-activation distribution of 𝑦𝑖 and 𝑦 𝑗 . Because optimizing the global covariance Σ̂𝑥 is computationally prohibitive, we restrict the maximum entropy formulation to the local 2 × 2 subspace spanned by the neuron pair. Proposition E.3 (Pairwise Warped Correlation). Under a local pairwise maximum entropy surrogate, the correlation 𝜌ˆ𝑖 𝑗 between 𝑦𝑖 and 𝑦 𝑗 is given analytically by: 𝜌 ˆ𝑖 𝑗 =
2𝜅 √ 1 + 1 + 4𝜅2
(80)
where 𝜅 is the blending constant uniquely defined by the input weight geometry and empirical standard deviationsa : ! © | 𝛾𝑖 | ª © | 𝛾 𝑗 | ª 𝜌eff ® ® 𝜅= (81) 2 1 − 𝜌eff 𝒘eff ® 𝒘eff ® « in,𝑖 2 ¬ « in, 𝑗 2 ¬ a To prevent undefined 0/0 states, this evaluation is restricted to active features where the effective weight norm eff is non-zero 𝒘in > 0. 2
Proof. Let 𝑾 = [𝒘raw,𝑖 , 𝒘raw, 𝑗 ] ∈ ℝ𝑛×2 be the raw weights and 𝑮 = 𝑾 𝑇 𝑾 be their Gram matrix. The maximum entropy distribution constrained by the variances of these two projections has a precision matrix Σ̂𝑥−1 = 𝑰 + 𝑾𝚲𝑾 𝑇 , where 𝚲 = diag( 𝜆 𝑖 , 𝜆 𝑗 ) contains the Lagrange multipliers. The output covariance is 𝑪 = 𝑾 𝑇 Σ̂𝑥 𝑾 = 𝑾 𝑇 ( 𝑰 + 𝑾𝚲𝑾 𝑇 ) −1𝑾 . Applying the Woodbury matrix identity to the inner inverse yields: ( 𝑰 + 𝑾𝚲𝑾 𝑇 ) −1 = 𝑰 − 𝑾 ( 𝚲−1 + 𝑾 𝑇 𝑾 ) −1𝑾 𝑇 = 𝑰 − 𝑾 ( 𝚲−1 + 𝑮 ) −1𝑾 𝑇 Substituting this back into the expression for 𝑪 : 𝑪 = 𝑾 𝑇 𝑰 − 𝑾 ( 𝚲 −1 + 𝑮 ) −1𝑾 𝑇 𝑾 = 𝑮 − 𝑮 ( 𝚲 −1 + 𝑮 ) −1 𝑮 We factor out 𝑮 to simplify the expression: 𝑪 = 𝑮 𝑰 − ( 𝚲 −1 + 𝑮 ) −1 𝑮 = 𝑮 ( 𝚲−1 + 𝑮 ) −1 ( 𝚲−1 + 𝑮 ) − ( 𝚲−1 + 𝑮 ) −1 𝑮 = 𝑮 ( 𝚲−1 + 𝑮 ) −1 𝚲−1 Inverting both sides yields the localized precision matrix 𝑪 −1 : 𝑪 −1 = 𝚲 ( 𝚲 −1 + 𝑮 ) 𝑮 −1 = 𝚲𝚲 −1 𝑮 −1 + 𝚲𝑮𝑮 −1 = 𝑮 −1 + 𝚲
Because 𝚲 is diagonal, its addition only perturbs the diagonal entries of 𝑮 −1 . Therefore, the offdiagonal entries are identical: [𝑪 −1 ] 12 = [𝑮 −1 ] 12 .
54
Hilbert Operator for Progressive Encoding (HOPE)
Let 𝜌raw be the correlation in 𝑮 and 𝑟raw be the target correlation in 𝑪 . Evaluating the inverse off-diagonal elements of these 2 × 2 matrices yields: −𝑟raw − 𝜌raw −1 , [ 𝑪 ] = [𝑮 −1 ] 12 = 12 2 ) 𝜎𝑖 𝜎 𝑗 (1 − 𝑟raw 𝒘raw,𝑖 2 𝒘raw, 𝑗 2 (1 − 𝜌2raw ) Equating them gives:
𝑟raw 2 1 − 𝑟raw
=
𝜌raw
1 − 𝜌2raw
𝜎𝑖
!
𝜎𝑗
𝒘raw,𝑖 2
!
𝒘raw, 𝑗 2
eff = √︃ 𝛾𝑖 𝒘 By the definition of the effective weights, 𝒘in raw,𝑖 . The effective correlations relate to the ,𝑖 𝜎2𝑖 +𝜖
raw correlations via the signs of these scaling parameters: 𝜌raw = sign( 𝛾𝑖 𝛾 𝑗 ) 𝜌eff and 𝑟raw = sign( 𝛾𝑖 𝛾 𝑗 ) 𝜌ˆ𝑖 𝑗 . Substituting these into the odd function 𝑓 ( 𝑥 ) = 1−𝑥𝑥 2 causes the sign terms to cancel. Taking the limit as 𝜖 → 0, we substitute
𝜎𝑖
∥ 𝒘raw,𝑖 ∥ 2
=
| 𝛾𝑖 |
eff 𝒘in ,𝑖
, which yields the blended constant 𝜅: 2
𝜌 ˆ𝑖 𝑗
1 − 𝜌ˆ2𝑖 𝑗
= 𝜅 =⇒ 𝜅 𝜌ˆ2𝑖 𝑗 + 𝜌ˆ𝑖 𝑗 − 𝜅 = 0
Solving for 𝜌ˆ𝑖 𝑗 via the quadratic√ formula and selecting the root that satisfies the boundary condition 𝜅 lim𝜌eff →0 𝜌ˆ𝑖 𝑗 = 0, we obtain −1+ 21+4 . Multiplying the numerator and denominator by the conjugate 𝜅 √ 2 ( 1 + 4𝜅 + 1) ensures numerical stability as 𝜅 → 0, producing the final formulation √2𝜅 2 . □ 2
1+ 1+4𝜅
Consequently, the joint pre-activation distribution under the surrogate space is: 𝛾𝑖2 | 𝛾𝑖 || 𝛾 𝑗 | 𝜌ˆ𝑖 𝑗 𝛽𝑖 𝑦𝑖 , ∼N | 𝛾𝑖 || 𝛾 𝑗 | 𝜌ˆ𝑖 𝑗 𝛾 2𝑗 𝛽𝑗 𝑦𝑗
(82)
Regardless of the activation Ψ, any valid Cross-Kernel 𝐾 ( 𝑖, 𝑗) = 𝔼[ Ψ ( 𝑦𝑖 ) Ψ ( 𝑦 𝑗 )] must satisfy three properties: 1. Diagonal Consistency: If 𝜌ˆ𝑖 𝑗 = 1 and the marginals are identical ( 𝛽𝑖 = 𝛽 𝑗 , | 𝛾𝑖 | = | 𝛾 𝑗 |), the interaction recovers the self-kernel: 𝐾 ( 𝑖, 𝑗) = 𝐾 ( 𝑖, 𝑖). √︁ 2. Cauchy-Schwarz Compliance: The magnitude is bounded: | 𝐾 ( 𝑖, 𝑗)| ≤ 𝐾 ( 𝑖, 𝑖) 𝐾 ( 𝑗, 𝑗). 3. Weight-Space Correlation Dependency: The interaction is monotonic with respect to 𝜌ˆ𝑖 𝑗 . E.3.2. Exact Bivariate Cross-Kernel for Biased ReLUs Evaluating 𝐾 ( 𝑖, 𝑗) = 𝔼[max(0, 𝑦𝑖 ) max(0, 𝑦 𝑗 )] requires integrating over the joint positive orthant. 𝛽 Defining 𝑐𝑖 = | 𝛾𝛽𝑖𝑖 | and 𝑐 𝑗 = | 𝛾 𝑗𝑗 | , the exact cross-kernel evaluates to the closed-form moments of a truncated bivariate normal: " 𝐾 ( 𝑖, 𝑗) = | 𝛾𝑖 𝛾 𝑗 | ( 𝑐𝑖 𝑐 𝑗 + 𝜌 ˆ𝑖 𝑗 ) Φ2 ( 𝑐𝑖 , 𝑐 𝑗 ; 𝜌ˆ𝑖 𝑗 ) + 𝑐𝑖 𝜙 ( 𝑐 𝑗 ) Φ ( 𝑐𝑖 | 𝑗 )
# + 𝑐 𝑗 𝜙 ( 𝑐𝑖 ) Φ ( 𝑐 𝑗 | 𝑖 ) + (1 − 𝜌ˆ𝑖 𝑗 ) 𝜙2 ( 𝑐𝑖 , 𝑐 𝑗 ; 𝜌ˆ𝑖 𝑗 ) 2
(83)
where Φ2 and 𝜙2 are the standard bivariate normal CDF and PDF evaluated at ( 𝑐𝑖 , 𝑐 𝑗 ) with correlation 𝜌 ˆ𝑖 𝑗 , 𝜙 and Φ are the standard univariate normal PDF and CDF, and the conditional integration 𝑐 −𝜌 ˆ 𝑐 𝑐 −𝜌 ˆ 𝑐 boundaries are 𝑐𝑖 | 𝑗 = √︃𝑖 𝑖 𝑗 2𝑗 and 𝑐 𝑗 | 𝑖 = √︃𝑗 𝑖 𝑗 2𝑖 . By centering the evaluation on 𝛽𝑖 rather than the weight 1− 𝜌ˆ𝑖 𝑗
1− 𝜌ˆ𝑖 𝑗
bias 𝑏𝑖 , we correctly evaluate active feature detectors in the function space. 55
Hilbert Operator for Progressive Encoding (HOPE)
E.3.3. Zero-Bias Approximation for Large-Scale Networks Because calculating Φ2 for all neuron pairs is computationally intensive, we approximate the interaction by assuming bias shifts are negligible 𝛽𝑖 , 𝛽 𝑗 ≈ 0. Under this assumption, 𝐾 ( 𝑖, 𝑗) factors into the geometric mean of the capacities scaled by a normalized interaction function I ( 𝜌ˆ𝑖 𝑗 ): √︁ 𝐾 ( 𝑖, 𝑗) ≈ I ( 𝜌 ˆ𝑖 𝑗 ) 𝐾 ( 𝑖, 𝑖) 𝐾 ( 𝑗, 𝑗) (84) For the ReLU activation, applying the Arc-Cosine kernel of order 𝑛 = 1 yields the approximate cross-kernel: √︁ 1 √︃ 𝐾 ( 𝑖, 𝑗) ≈ 1 − 𝜌ˆ2𝑖 𝑗 + ( 𝜋 − arccos 𝜌ˆ𝑖 𝑗 ) 𝜌ˆ𝑖 𝑗 𝐾 ( 𝑖, 𝑖) 𝐾 ( 𝑗, 𝑗) (85) 𝜋 √︁ This formulation complies with the Cauchy-Schwarz inequality | 𝐾 ( 𝑖, 𝑗)| ≤ 𝐾 ( 𝑖, 𝑖) 𝐾 ( 𝑗, 𝑗) and ensures diagonal consistency ( 𝐾 ( 𝑖, 𝑗) = 𝐾 ( 𝑖, 𝑖) when 𝜌ˆ𝑖 𝑗 = 1).
F. Derivations for Block Eviction This appendix provides the mathematical framework and extended derivations for the block eviction operation introduced in Section 8. F.1. Generalization and Execution Degradation in Depleted Blocks A dedicated macro-level operation is necessary because standard granular pruning cannot fully deplete a residual block due to architectural constraints. Because the residual connection computes 𝑋 + 𝐹 ( 𝑋 ), the output dimension of 𝐹 ( 𝑋 ) (and thus the final weight tensor 𝑊3 ) must match that of 𝑋 . While granular pruning can safely compress internal neurons within 𝑊1 and 𝑊2 , pruning the output filters of 𝑊3 would cause a dimensional mismatch in the element-wise addition 𝑋 + 𝐹 ( 𝑋 ). Consequently, granular compression leaves the output channels of 𝑊3 intact. Leaving a residual pathway active with heavily depleted 𝑊1 and 𝑊2 degrades both model generalization and execution efficiency: • Model Generalization. The output of the final BN layer evaluates to 𝐹 ( 𝑋 ) = 𝜸 ⊙ 𝑊√3 𝐻22 −𝝁 + 𝜷. 𝝈 +𝜖
When 𝐻2 → 0, this reduces to 𝐹 ( 𝑋 ) = 𝜷 − 𝜸 ⊙ √ 𝝁2 . As defined in (1), this is the effective bias 𝝈 +𝜖 vector 𝒃. Once the block is depleted, 𝒃 loses its normalization purpose and instead acts as an uncalibrated bias injected into the skip connection, yielding 𝑌 = 𝑋 + 𝒃. This forcefully shifts downstream feature maps out of their calibrated domain. Since the block terminates with a ReLU activation 𝑍 = ReLU ( 𝑋 + 𝒃), a negative 𝒃 causes catastrophic clipping and irreversible information loss. • Execution Efficiency. Even if 𝐻2 partially survives, the shape of 𝑊3 remains locked at its ambient output size. Retaining this massive parameter tensor simply to process a negligible, low-rank subspace is computationally wasteful.
Block eviction resolves these issues by projecting 𝐹 ( 𝑋 ) → 0, yielding a pure identity mapping 𝑌 = 𝑋 . Unlike 𝑌 = 𝑋 + 𝒃, residual architectures are natively robust to pure identity mappings. For instance, initializing 𝛾 = 0 in the final layer of each residual branch [Goyal et al., 2017, He et al., 2016] ensures blocks begin training as 𝐹 ( 𝑋 ) = 0. F.2. Derivation of the Unified Macro Cost Jevict Recall that a state tuple in layer 𝑙 has the form Φ ( 𝑙 ) = ( 𝑓1( 𝑙 ) , . . . , 𝑓𝑁( 𝑙𝑙 ) ) ∈ H𝑙𝑁𝑙 . Let I = ( 𝐼1 , . . . , 𝐼𝑑amb ) ∈ 𝑑
H 𝐴 amb be the tuple of identity mappings comprising the skip connection, where H 𝐴 is the Hilbert 56
Hilbert Operator for Progressive Encoding (HOPE)
space of operators over the ambient dimension ℝ𝑑amb → ℝ𝑑amb . The aggregate mapping of any tuple is the sum of its constituent operators: 𝑁𝑙 ∑︁
(𝑙)
F (Φ ) ≜
(𝑙)
𝑓𝑖
F (I) =
,
𝑖=1
𝑑∑︁ amb
(86)
𝐼𝑘 .
𝑘=1
The composite block operator B ∈ H 𝐴 is formalized as the addition of the residual pathway Fres ∈ H 𝐴 and the skip connection: B = Fres Φ (1) , Φ (2) + F (I) . (87) We define eviction iteratively: we project internal layers to null operators until the block is fully depleted. Evaluating the projection of an internal layer Φ ( 𝑙 ) over 𝑡 ∈ [0, 1] induces a continuous trajectory Φ ( 𝑙 ) ( 𝑡 ) → (0, . . . , 0). Let Fres ( 𝑡 ) denote the sequential pathway where Φ ( 𝑙 ) ( 𝑡 ) → 0 while other layers are held constant. The composite block trajectory is: B ( 𝑡 ) = Fres ( 𝑡 ) + F (I) .
(88)
From Section 6, the capacity cost for a microscopic layer transition is: Jcapacity ( Φ𝑎 , Φ𝑏 ) =
∫ 1
𝑑
−𝑐 ( Φ ( 𝑡 )) 𝑑𝑡
𝐸 ( Φ ( 𝑡 ))
𝐸 ( Φ ( 𝑡 ))
0
(89)
𝑑𝑡 .
To generalize this to macroscopic blocks, we expand the state to a composite tuple Ω ≜ (𝑇1 , . . . , 𝑇𝑛 ), where 𝑇𝑖 ∈ H𝑖𝑁𝑖 . By Proposition C.11, extending 𝐸 (·) under constraints of symmetry, separability, and homogeneity yields: 𝑁𝑖 𝑛 𝑛 ∑︁ ∑︁ ∑︁ 𝐸 ( Ω) = 𝐸 (𝑇𝑖 ) = ∥ 𝑓𝑖, 𝑗 ∥ H𝑖 . (90) 𝑖=1
𝑖=1 𝑗=1
For consistency, the constituent tuples of Ω must interact strictly additively within B. Grouping sequentially composed layers (e.g., Φ (1) and Φ (2) ) into the same state creates a contradiction: projecting Φ (1) → 0 collapses the entire sequential pathway Fres → 0, leaving B = F (I) with a true capacity of 𝐸 (I). However, 𝐸 ( Ω) would erroneously evaluate to 𝐸 ( Φ (2) ) + 𝐸 (I). To avoid this, we must restrict the macroscopic state strictly to additive components: a single internal layer 𝑙 and the skip connection I. Furthermore, because the mapping Φ ( 𝑙 ) and the skip connection I operate in different dimensional spaces (𝑑bottleneck versus 𝑑amb ), direct operator addition Φ ( 𝑙 ) + I is mathematically undefined. We bypass this by defining the state as the composite pair Ω ( 𝑙 ) ≜ ( Φ ( 𝑙 ) , I). The capacity cost for this macroscopic state is: Jcapacity ( Ω𝑎( 𝑙 ) , Ω𝑏( 𝑙 ) ) =
∫ 1
𝑑
−𝑐 ( Ω ( 𝑙 ) ( 𝑡 )) 𝑑𝑡
0
𝑑 𝐸 ( Φ ( 𝑙 ) ( 𝑡 )) + 𝑑𝑡 𝐸 (I)
𝐸 ( Φ ( 𝑙 ) ( 𝑡 )) + 𝐸 (I)
𝑑𝑡 .
(91)
To mitigate layer-width bias, 𝑐 (·) acts as a counting measure of compressible operators. Being additive over disjoint sets, 𝑐 ( Ω ( 𝑙 ) ) = 𝑐 ( Φ ( 𝑙 ) ) + 𝑐 (I). Since the skip connection I is fixed, it has no compressible operators 𝑐 (I) = 0. Thus, 𝑐 ( Ω ( 𝑙 ) ( 𝑡 )) = 𝑐 ( Φ ( 𝑙 ) ( 𝑡 )). Combined with the stationarity of the skip connection 𝑑𝑡𝑑 𝐸 (I) = 0, the integral simplifies to: (𝑙)
(𝑙)
Jcapacity ( Ω𝑎 , Ω𝑏 ) =
∫ 1 0
−𝑐 ( Φ
(𝑙)
𝑑 𝐸 ( Φ ( 𝑙 ) ( 𝑡 )) 𝑑𝑡 ( 𝑡 )) 𝑑𝑡 . 𝐸 ( Φ ( 𝑙 ) ( 𝑡 )) + 𝐸 (I)
(92)
57
Hilbert Operator for Progressive Encoding (HOPE)
(𝑙) During the projection Φ𝑎( 𝑙 ) → 0, 𝑐 ( Φ ( 𝑙 ) ( 𝑡 )) drops from 𝑁active to 0. Bounding this density by its (𝑙) maximum pre-action value 𝑁active allows a closed-form integration. Defining 𝐸 ( 𝑡 ) ≜ 𝐸 ( Φ ( 𝑙 ) ( 𝑡 )), 𝑑 𝐸¤ ( 𝑡 ) ≜ 𝑑𝑡 𝐸 ( 𝑡 ), and 𝐸identity ≜ 𝐸 (I): ! ∫ 1 (𝑙) ¤ 𝐸 − 𝐸 ( 𝑡 ) (𝑙) (𝑙) Jcapacity ( Ω𝑎( 𝑙 ) , Ω𝑏( 𝑙 ) ) ≤ 𝑁active 𝑑𝑡 = 𝑁active ln 1 + active . (93) 𝐸identity 0 𝐸 ( 𝑡 ) + 𝐸identity
This logarithmic formulation penalizes capacity reduction continuously. However, compression ex(𝑙) ecutes finite capacity removals Δ𝐸 = 𝐸active in discrete leaps. Evaluating macroscopic evictions logarithmically while governing granular micro-actions (Section 6) via linear ratios creates an inconsistent optimization hierarchy. Driven by this logarithmic discount, a greedy optimizer would view massive architectural deletions as artificially cheap. To align the macro-eviction cost with the linear micro-transition cost and ensure optimization stability, we apply the standard inequality ln(1 + 𝑥 ) ≤ 𝑥 (𝑙) for 𝑥 ≥ 0. Substituting 𝑥 = 𝐸active / 𝐸identity establishes a strict linear upper bound: (𝑙) 𝐸active (𝑙) . Jlayer ( Ω𝑎 , Ω𝑏 ) ≜ 𝑁active 𝐸identity (𝑙)
(𝑙)
(94)
F.2.1. Parallel Survival Capacity 𝐸identity Flattening the tensor 𝑋 into a vector 𝒙 ∈ ℝ𝑑amb , we decompose it into 𝑑amb independent identity operators I𝑘 ( 𝒙 ) = ⟨𝒆𝑘 , 𝒙 ⟩ 𝒆𝑘 = 𝑥 𝑘 𝒆𝑘 . Over the input distribution 𝑃 X , the Hilbert-Schmidt capacity evaluates to the Root Mean Square (RMS) energy. Given ∥ 𝒆𝑘 ∥ 2 = 1: 1/2 √︃ ∥I𝑘 ∥ H = 𝔼𝒙∼𝑃X [∥ 𝑥 𝑘 𝒆𝑘 ∥ 22 ] = 𝔼𝒙∼𝑃X [ 𝑥 𝑘2 ] . (95) Assuming 𝒙 is conditioned by a preceding BN layer with scale 𝛾𝑘 and shift 𝛽𝑘 , the expected energy evaluates to 𝔼[ 𝑥 𝑘2 ] = Var( 𝑥 𝑘 ) + (𝔼[ 𝑥 𝑘 ]) 2 = 𝛾𝑘2 + 𝛽𝑘2 . The aggregate capacity of the skip connection is therefore: 𝑑∑︁ amb √︃ 𝛾𝑘2 + 𝛽𝑘2 . (96) 𝐸identity = 𝑘=1
For a stably normalized network 𝛾𝑘 ≈ 1, 𝛽𝑘 ≈ 0, this naturally evaluates to the ambient physical dimension: 𝐸identity ≈ 𝑑amb . F.2.2. Resolving the Extinction Divergence Block eviction projects a layer 𝑙 to 0. Since PH-1 activations satisfy Ψ (0) = 0, this ensures the entire sequential residual pathway vanishes Fres → 0, reducing the composite block mapping to B = F (I). Axiom 2 (Connectivity Preservation) requires the transition cost to diverge J → ∞ as the state capacity approaches zero, penalizing graph disconnections. If evaluated solely on the isolated layer Φ ( 𝑙 ) , the terminal capacity would evaluate to zero, yielding an infinite penalty that would permanently prevent the optimizer from selecting an eviction operation. However, because the post-eviction mapping of the block is F (I), the network retains a minimum capacity 𝐸identity > 0. Evaluating the integral over the macroscopic state Ω ( 𝑙 ) = ( Φ ( 𝑙 ) , I) naturally tracks this non-vanishing capacity. Since the differential change is driven entirely by the active layer 𝑑𝐸 ( Ω) = 𝑑𝐸 ( Φ ( 𝑙 ) ), we have: ∫ 0 (𝑙) − 𝑁active 𝑑𝐸 (𝑙) Jlayer = . (97) (𝑙) 𝐸active 𝐸 + 𝐸identity As 𝐸 → 0, the denominator is bounded by lim𝐸→0 𝐸 ( Ω) = 𝐸identity > 0. The extinction divergence is thus mitigated. 58
Hilbert Operator for Progressive Encoding (HOPE)
F.3. Generalization to Non-Residual Architectures While 𝐸identity structurally resolves the division-by-zero limit for residual networks, architectures without additive skip connections lack this architectural anchor. Examples include sequential VGGstyle blocks, Inception modules, DenseNets, or intermediate expansion layers inside Transformer FFNs (prior to the global residual addition). In these architectures, evicting a block leaves no parallel identity mapping. This means the surviving capacity reaches zero. To evaluate block eviction in these domains without encountering an infinite penalty J → ∞, we replace the dynamic composite denominator with a static historical constant: the block’s initial, (𝑙) pre-pruning capacity 𝐸init . The individual layer-wise cost becomes: (𝑙) 𝐸active (𝑙) Jlayer ( Ω𝑎 , Ω𝑏 ) = 𝑁active ( 𝑙 ) 𝐸init (𝑙)
(𝑙)
(98)
For a block containing multiple internal layers, the total macroscopic eviction cost is the linear sum of Í (𝑙) these bounds: Jevict = 𝑙 Jlayer . From the perspective of pure functional analysis, 𝐸init acts as a heuristic. It violates the Markov property of continuous state transitions (because the system must maintain memory of a pristine architectural state that no longer exists) and departs from Axiom 2 by freezing the denominator rather than tracking an architectural asymptote. However, it serves as an optimal proxy for three reasons: (𝑙) • Scale Invariance: Using an arbitrary absolute constant 𝐶 would yield J ∝ 𝐸active /𝐶 , violating (𝑙) (𝑙) scale neutrality. The ratio 𝐸active / 𝐸init ∈ [0, 1] normalizes the capacity reduction against the layer’s own baseline. This ensures blocks of varying widths are penalized fairly based on the relative fraction of capacity destroyed. (𝑙) • Avoiding Normalization Collapse: Normalizing by the current active capacity 𝐸active would (𝑙) collapse the cost to Jlayer = 𝑁active . This artificially discounts wide, pristine layers, driving the greedy optimizer to evict them prematurely instead of targeting heavily pruned, redundant layers. • Dynamic Cost Decay: The progressive encoder selects actions that minimize the Distortion Rate DR = J /Δ𝑃 . Under a mean-field assumption, active capacity scales linearly with surviving width 𝐸active ∝ 𝑁active . By locking the denominator to 𝐸init , the structural cost decays quadratically as 2 the layer is pruned J ∝ 𝑁active 𝐸active ∝ 𝑁active . Because the physical parameter yield also scales linearly Δ𝑃 ∝ 𝑁active , the overall distortion rate decreases linearly: DR ∝ 𝑁active . This guarantees that the relative cost of evicting a block naturally decreases as granular micro-actions deplete it.
Thus, while 𝐸identity is the rigorously derived anchor for residual networks, 𝐸init provides the necessary constraints to enable automated, progressive eviction in non-residual architectures.
G. Reproducibility Protocols for Cross-Domain Transfer This section details the experimental setup, algorithmic formulations, and hyperparameter grids necessary to reproduce the cross-domain transfer results presented in Section 11.2. G.1. Task Construction and Data Partitioning Tasks are dynamically constructed from the datasets. We perform 4 independent trials, with each trial comprised of 5 tasks to yield 20 distinct cross-domain transfer scenarios. 59
Hilbert Operator for Progressive Encoding (HOPE)
• Source Tasks (CIFAR-100): To ensure the source task requires learning cohesive structural features, the task comprises 20 classes sampled by selecting 4 random superclasses and utilizing all 5 of their constituent fine classes. • Target Tasks (SVHN): The target tasks consist of 10-class classification utilizing all SVHN digits. Both CIFAR-100 and SVHN datasets are partitioned into an 80% training split and a 20% validation split. The original test sets are preserved entirely for final evaluation. Prior to training, images are cast to floating-point tensors and scaled to [0, 1] by dividing by 255. No data augmentation is applied during training. G.2. Network Architecture All experiments utilize a VGG-style 8-layer sequential baseline, modernized with BN. To prevent the final classification head from dominating the network’s parameter footprint, we replace the classic flattened Dense layers with Global Average Pooling (GAP). This ensures that the model’s overall capacity, and consequently Δ𝑃 evaluated by our compression algorithm, remains localized to the convolutional filters rather than spatial dense transitions. This prevents artificially skewed Distortion Rates (DR). The network consists of the following blocks: 1. Block 1: Conv2D (128 filters, 3x3) → BN → ReLU → MaxPool (2x2) 2. Block 2: Conv2D (256 filters, 3x3) → BN → ReLU → MaxPool (2x2) 3. Block 3: Conv2D (512 filters, 3x3) → BN → ReLU → Conv2D (512 filters, 3x3) → BN → ReLU → MaxPool (2x2) 4. Block 4: Conv2D (512 filters, 3x3) → BN → ReLU → Conv2D (512 filters, 3x3) → BN → ReLU → MaxPool (2x2) 5. Transition: GlobalAveragePooling2D 6. Bottleneck: Dense (512 neurons) → BN → ReLU 7. Classification Head: Dense (10 or 20 neurons) for multi-class logit output. Note that BN layers are positioned between the affine transformations (Conv2D/Dense) and the ReLU non-linearities. To prevent redundancy and ensure capacity accounting, biases in the Conv2D and Dense layers preceding a BN layer are disabled. G.3. Base Training Regimen Both pre-training and fine-tuning utilize SGD with standard heavy-ball momentum of 0.9, a fixed batch size of 16, and a Sparse Categorical Cross-Entropy (CE) loss function. • Source Pre-training: The model is trained from a random initialization for 100 epochs. To suppress mini-batch noise and ensure the parameters in the slack space converge to zero, the learning rate follows a Cosine Decay schedule starting at 𝜂 = 0.05 and decaying to a nearzero 𝛼 = 0.001. Early stopping is deliberately disabled. To actively clear out the network’s peripheral slack during this phase, an 𝐿2 regularization penalty of 5 × 10−4 is applied to all multi-dimensional kernels as well as all 1D BN scale parameters 𝛾 . • Target Fine-Tuning: All algorithms fine-tune for 30 epochs and use a fixed learning rate rather than a decay schedule. During target adaptation, the 𝐿2 penalty is restricted to tensors with a rank greater than 1 to leave the calibrated 1D BN scale parameters immune to weight decay.
60
Hilbert Operator for Progressive Encoding (HOPE)
G.4. EWC Exact Empirical Fisher Calculation To properly lock foundational source features for the EWC baseline, the empirical diagonal Fisher Information Matrix (FIM) is evaluated over the source training dataset (bypassing validation sets). Note that the mathematical formulation calculates the expected squared gradient of the source CE loss at the per-example level before batch averaging: " 2# 𝜕LCE FIM𝑖 = 𝔼𝒙∼Xsource_train (99) 𝜕𝜽𝑖
Many naive EWC implementations incorrectly square the averaged mini-batch gradient, which misrepresents the true Fisher Information Matrix. By computing the per-example squared gradients directly, our evaluation avoids this theoretical pitfall. During target domain optimization, the target CE loss is augmented by the EWC penalty: Ltotal = Ltarget_CE +
𝜆
∑︁
2
𝑖 ∈Backbone
FIM𝑖 ( 𝜽𝑖 − 𝜽𝑖∗ ) 2
(100)
where 𝜆 is a global regularization hyperparameter, and 𝜽𝑖∗ represents the model parameters of the optimal pre-trained source. Note that the summation is restricted to the backbone parameters; the newly initialized target classification head is excluded from the penalty. G.5. Hyperparameter Tuning and Final Evaluation To ensure a fair comparison, all algorithms undergo a hyperparameter pre-sweep prior to final evaluation. Hyperparameters are selected by maximizing the H-Score, defined as the Harmonic Mean of the validation accuracies on the target and source tasks: target
H-Score =
2 · Accval
· Accsource val
target
+ Accsource val
Accval
(101)
This evaluation uses the Validation Sets of both SVHN (target) and CIFAR-100 (source) to prevent data leakage. To accelerate the pre-sweep, the target training set is artificially capped at 1, 000 samples during the tuning phase. Rather than re-evaluating the grid for all 20 distinct transfer scenarios, the algorithm isolates the very first task (Task 0) as a representative sample, sweeps the grids to find the globally optimal parameters, and locks them in for the entire benchmark run. The sweep evaluates grids specific to each methodology: • DEFT: A grid search across the target percentile 𝑃 ∈ {60, 40, 30, 20}, and fixed fine-tuning learning rate 𝜂 ∈ {0.04, 0.02, 0.01, 0.005, 0.001}. • EWC: A grid search across the regularization strength 𝜆 ∈ {0.1, 0.5, 1.0, 5.0, 10.0, 20.0, 50.0} and learning rate 𝜂 ∈ {0.01, 0.005, 0.001, 0.0005, 0.0001}. • PEFT: Unfreezes all 1D tensors (including layer biases and BN affine parameters) alongside the target head, while freezing all 2D/4D kernels. Optimized via a line search for 𝜂 ∈ {0.01, 0.005, 0.0025, 0.001, 0.0005, 0.0001}. • Standard Full FT: Unfreezes the entire network architecture. Optimized via a line search for 𝜂 ∈ {0.02, 0.01, 0.005, 0.001, 0.0005, 0.0001}. • Standard Head-Only FT: Freezes the entire backbone, optimizing only the final classification head. Optimized via a line search for 𝜂 ∈ {0.04, 0.02, 0.01, 0.005, 0.001, 0.0005}. 61
Hilbert Operator for Progressive Encoding (HOPE)
Once optimal hyperparameters are secured, models are fine-tuned on the full SVHN training set for 30 epochs. Performance is tracked epoch-by-epoch using the SVHN Validation Set, and the algorithm identifies the best epoch yielding peak target validation accuracy. The final reported metrics in the main text are extracted at this best epoch by running the model against the unseen SVHN Test Set and CIFAR-100 Test Set. Source Retention Protocol: To evaluate source retention, the original pre-trained source classification head is temporarily grafted back onto the fine-tuned backbone. Furthermore, a universal mask is applied to the grafted source head. This zeros out connections originating from upstream “slack” features 𝐸 > 0 to ensure target-adapted weights do not corrupt the source logits. Critically, the BN moving statistics 𝜇, 𝜎2 are reverted; they are unlocked from the target-adapted backbone and restored to their original source states to guarantee a fair evaluation.
H. Theoretical Guarantees of DEFT Continual learning seeks a parameter update Δ𝜽 that minimizes target risk while bounding the degradation of the source representation. Rather than analyzing non-convex loss landscapes, we abstract the network into continuous-functional operators and evaluate layer-wise distortion. Under HOPE, the capacity of neuron 𝑖 is its Hilbert-Schmidt norm over 𝑃 X : √︁ ∥ 𝑓𝑖 ∥ H = ∥ 𝒘out,𝑖 ∥ 2 𝐾 ( 𝑖, 𝑖) where 𝐾 ( 𝑖, 𝑖) = 𝔼𝒙∼𝑃X Ψ ( 𝑦𝑖 ( 𝒙 )) 2 . (102) The Setup: Core vs. Slack. DEFT partitions the network into two disjoint sets based on a capacity threshold 𝜏: • Universal Core ∥ 𝑓𝑖 ∥ H > 𝜏: Highly active neurons carrying source knowledge. We freeze these 𝐸 𝑖 = 0 to prevent forgetting. • Slack Subset ∥ 𝑓 𝑗 ∥ H ≤ 𝜏: Weak, inactive neurons. We make these fully plastic 𝐸 𝑗 = 1 to learn the target task. To prevent the target-driven updates of the slack subset from corrupting the core, DEFT applies a structural mask at initialization 𝑡 = 0, severing all connections from upstream slack neurons to downstream core neurons. We establish the stability and plasticity of this setup through four main guarantees: 1. Bounded Initialization Shock (Theorem H.1). Severing the slack-to-core connections at 𝑡 = 0 introduces a static error. As visualized in Figure 5(a), the severed core-directed weights 𝒘core, 𝑗 of a slack neuron 𝑗 form a sub-vector of its total outgoing weights 𝒘out, 𝑗 . Thus, ∥ 𝒘core, 𝑗 ∥ 2 ≤ ∥ 𝒘out, 𝑗 ∥ 2 . Because we only sever connections from slack neurons (where ∥ 𝑓 𝑗 ∥ H ≤ 𝜏), the initial distortion (𝑙) injected into the core is strictly bounded by |Nslack |𝜏 (𝑙 ) . 2. Dynamic Decoupling (Theorem H.2). During training 𝑡 > 0, slack neurons drift to learn the (0) target task. Because the structural mask severs cross-connections at initialization 𝒘core = 0 and the ,𝑗 zero elasticity prevents gradient updates 𝜕𝒘core, 𝑗 /𝜕𝑡 = 0, these weights remain zero. As illustrated in Figure 5(b), a changing signal multiplied by zero is zero; thus, target parameter drift cannot penetrate the core. 3. Freeing Space Safely (Proposition H.4). Freezing the core might leave insufficient parameter space for the target task. Deep networks often fragment a feature across 𝑀 correlated neurons. Statically freezing them incorrectly locks redundant volume. Instead, DEFT compresses these 𝑀 62
Hilbert Operator for Progressive Encoding (HOPE)
neurons into a single rank-1 parent operator. This preserves the feature while releasing the remaining 𝑀 − 1 children to the slack subset, freeing the parameter space ℝ ( 𝑀 −1) × 𝑐 for optimization. Because the optimal parent minimizes the Hilbert-Schmidt projection error, this structural distortion is upperÍ𝑀 bounded by the sum of the children’s initial capacities: 𝛿𝑘 ≤ 𝑚 =1 ∥ 𝑓𝑚 ∥ H ( 𝑙 ) . (While we structurally guarantee this capacity release, we do not formally prove gradient descent finds a global optimum in this non-convex space). 4. Unified Cumulative Bound (Corollary H.5). Cutting connections and merging neurons across layers introduces multiple distortions. However, because Theorem H.2 guarantees zero interference during training, the network’s total error does not compound exponentially. By the triangle inequality in H ( 𝑙 ) , the global degradation in the function space is static and bounded by the linear sum of the severed slack connections and the merge projection errors. To Slack
Kept Component
tal To
ut
wo
Cut Component
(a) Vector Geometry (Theorem H.1)
Preceding Layer’s Slack Drift Signal (Δ𝑎)
To Core
Cut Connection 𝒘core = 0 Δ≠0
×
0
Current Layer’s Core Update (Δs = 0)
(Highly chaotic)
(b) Dynamic Signal Blocking (Theorem H.2)
Figure 5 | Visual intuition for the theoretical bounds of DEFT. (a) Vector decomposition demonstrates the initialization shock is bounded by the slack capacities. (b) A changing signal multiplied by the zeroed structural mask ensures dynamic decoupling.
H.1. Algorithmic Axioms and Partitioning of Neurons Here we formally define how DEFT partitions neurons into the peripheral slack and universal core. We also present a few axioms that merely mirror the mechanisms designated in the DEFT algorithm itself. These axioms do not impose any extra assumptions or restrictions on DEFT beyond the entirety of the algorithm itself. Definition H.1 (Neurons Partition). For any intermediate layer 𝑙 containing a set of active neurons I ( 𝑙 ) , a continuous capacity threshold 𝜏 ( 𝑙 ) > 0 divides the layer into two disjoint functional subsets: (𝑙) Ncore = { 𝑖 ∈ I ( 𝑙 ) | ∥ 𝑓𝑖 ∥ H ( 𝑙 ) > 𝜏 ( 𝑙 ) }
(103)
(𝑙) Nslack = { 𝑗 ∈ I ( 𝑙 ) | ∥ 𝑓 𝑗 ∥ H (𝑙) ≤ 𝜏 ( 𝑙 ) }
(104)
Axiom 1 (The Algorithmic Structural Mask). At initialization 𝑡 = 0, DEFT prevents cross-subset interference by severing the projections from the upstream slack subset to the downstream (𝑙) core. For any weight 𝑤𝑢, 𝑗 from an upstream slack neuron 𝑗 ∈ Nslack to a downstream core ( 𝑙 +1) neuron 𝑢 ∈ Ncore , the mask 𝑴 enforces 𝑤𝑢, 𝑗 = 0.
63
Hilbert Operator for Progressive Encoding (HOPE)
Axiom 2 (The Gradient Elasticity). During target optimization 𝑡 > 0, the parameter update for any parameter vector 𝜽𝑘 belonging to a destination neuron 𝑘 is governed by: 𝜕𝜽𝑘 = 𝐸 𝑘 · ∇𝜽𝑘 Ltarget 𝜕𝑡
(105)
where 𝐸 𝑘 ∈ {0, 1} is the elasticity multiplier. Slack neurons are fully plastic 𝐸 𝑗 = 1 ∀ 𝑗 ∈ Nslack and core neurons are strictly frozen 𝐸𝑢 = 0 ∀𝑢 ∈ Ncore . Axiom 3 (The Merged-Vessel Release). When the continuous encoder collapses redundant neurons into a rank-1 parent operator, DEFT assigns full plasticity 𝐸 𝑗 = 1 to the structurally released child neurons.
H.2. Layer-to-Layer Bounding Framework We now establish a rigorous bound on the structural distortion, decoupled entirely from target parameter drift. Assumption 1 (Bipartite Separability). Let the pre-activation 𝑦𝑢 of a downstream core neuron ( 𝑙 +1) 𝑢 ∈ Ncore be a linear combination of upstream activations 𝑔𝑘 ( 𝒙 ) = Ψ ( 𝑦𝑘 ( 𝒙 )) for 𝑘 ∈ I ( 𝑙 ) , governed by parameterized weights 𝑤𝑢,𝑘 and unparameterized identity connections denoted by (𝑙) an adjacency indicator 𝕀𝑢,𝑘 ∈ {0, 1}. We assume that for all upstream slack neurons 𝑗 ∈ Nslack , the network identity routing is strictly zero: (𝑙) ( 𝑙 +1) 𝕀𝑢, 𝑗 = 0 ∀ 𝑗 ∈ Nslack , ∀𝑢 ∈ Ncore
(106)
This ensures that the mask strictly dictates inter-subset signal flow.
Remark on Feed-Forward Architectures: For purely sequential architectures (e.g., standard MLPs or VGG), 𝕀𝑢,𝑘 ≡ 0 globally. This assumption only becomes non-trivial in architectures with unparameterized residual connections (e.g., ResNets), where an identity mapping could bypass the mask if the architecture cross-wired the subsets. Theorem H.1 (The Static Initialization Shock Bound). Let 𝑔𝑘 ( 𝒙 ) ≜ Ψ ( 𝑦𝑘 ( 𝒙 )), and let 𝒔core ( 𝒙 ) ≜ Í input to the downstream core. The initialization mask 𝑴 introduces 𝑘 ∈ I ( 𝑙 ) 𝒘core,𝑘 𝑔 𝑘 ( 𝒙 ) be the Í init a distortion Δ𝒔core ( 𝒙 ) ≜ 𝑗 ∈ N ( 𝑙 ) 𝒘core, 𝑗 𝑔 𝑗 ( 𝒙 ) bounded over 𝑃 X by: slack
init Δ𝒔core
H (𝑙)
(𝑙) ≤ |Nslack | 𝜏(𝑙)
(107)
Proof. By Assumption 1, 𝒔core ( 𝒙 ) decomposes strictly over the parameterized upstream subsets: ∑︁ ∑︁ 𝒔core ( 𝒙 ) = 𝑔𝑖 ( 𝒙 ) 𝒘core,𝑖 + 𝑔 𝑗 ( 𝒙 ) 𝒘core, 𝑗 (108) (𝑙)
𝑖 ∈ Ncore
(𝑙)
𝑗 ∈ Nslack
where 𝒘core,𝑘 is the sub-vector of structural weights connecting upstream neuron 𝑘 to the downstream (𝑙) core. By Axiom 1, DEFT enforces 𝒘core, 𝑗 = 0 ∀ 𝑗 ∈ Nslack . The resulting functional distortion evaluates 64
Hilbert Operator for Progressive Encoding (HOPE)
to the severed projections: init Δ𝒔core ( 𝒙) =
∑︁
𝑔 𝑗 ( 𝒙 ) 𝒘core, 𝑗
(109)
(𝑙) 𝑗 ∈ Nslack
Applying the triangle inequality in H ( 𝑙 ) yields: ∑︁ init Δ𝒔core ≤ (𝑙) H
By Definition 102, 𝑔 𝑗 H ( 𝑙 ) =
√︁
in
𝒘core, 𝑗 2 𝑔 𝑗 H ( 𝑙 )
(110)
in
(𝑙) 𝑗 ∈ Nslack
𝐾 ( 𝑗, 𝑗). Furthermore, because 𝒘out, 𝑗 = 𝒘core, 𝑗 ⊕ 𝒘slack, 𝑗 , its Euclidean
norm bounds its sub-vectors: 𝒘core, 𝑗 2 ≤ 𝒘out, 𝑗 2 . Substituting these properties gives: init Δ𝒔core
H
≤ (𝑙)
∑︁
𝒘out, 𝑗 2
√︁
𝐾 ( 𝑗, 𝑗) =
(𝑙) 𝑗 ∈ Nslack
∑︁
∥ 𝑓 𝑗 ∥ H (𝑙)
(111)
(𝑙) 𝑗 ∈ Nslack
By Definition H.1, the capacity of every slack neuron satisfies ∥ 𝑓 𝑗 ∥ H ( 𝑙 ) ≤ 𝜏 ( 𝑙 ) . Therefore, the functional distortion is strictly bounded by: ∑︁ (𝑙) init Δ𝒔core ≤ 𝜏 ( 𝑙 ) = 𝜏 ( 𝑙 ) |Nslack | (112) (𝑙) H
(𝑙)
𝑗 ∈ Nslack
□ H.2.1. Dynamic Decoupling The dynamic decoupling of the core emerges as a direct consequence of Axiom 2. Theorem H.2 (Dynamic Decoupling). During target fine-tuning 𝑡 > 0, the operators 𝑓𝑖( 𝑙 ) of all neurons within the core experience zero dynamic deviation: (𝑙) ∀ 𝑙 , 𝑖 ∈ Ncore ; 𝑓𝑖( 𝑙 ) , ( 𝑡 ) − 𝑓𝑖( 𝑙 ) , (0)
H (𝑙)
=0
(113)
Proof. We proceed by structural induction over the network layers. Base Case: The input 𝒙 ∼ 𝑃 X is a static anchor unmodified by optimization Δ𝒙 = 0. Inductive Step: Assume that for layer 𝑙, the core functional mappings exhibit zero drift: 𝑔𝑖( 𝑡 ) ( 𝒙 ) = (0) (𝑙) 𝑔𝑖 ( 𝒙 ) for all 𝑖 ∈ Ncore . The dynamic deviation of the signal injected into the downstream core of layer 𝑙 + 1 at time 𝑡 is (𝑡 ) (0) defined by Δ𝒔core ( 𝒙 ) = 𝒔core ( 𝒙 ) − 𝒔core ( 𝒙 ). Expanding this difference over the core and slack subsets yields: ∑︁ ∑︁ (𝑡 ) (𝑡 ) (0) (0) (𝑡 ) (𝑡 ) (0) (0) Δ𝒔core ( 𝒙 ) = 𝒘core,𝑖 𝑔𝑖 ( 𝒙 ) − 𝒘core,𝑖 𝑔𝑖 ( 𝒙 ) + 𝒘core, 𝑗 𝑔 𝑗 ( 𝒙 ) − 𝒘core, 𝑗 𝑔 𝑗 ( 𝒙 ) (114) (𝑙)
𝑖 ∈ Ncore
(𝑙)
𝑗 ∈ Nslack
(𝑡 ) (0) For the first summation, Axiom 2 freezes downstream core weights 𝐸𝑢 = 0 =⇒ 𝒘core = 𝒘core , ,𝑖 ,𝑖
and the inductive hypothesis guarantees 𝑔𝑖( 𝑡 ) ( 𝒙 ) = 𝑔𝑖(0) ( 𝒙 ). The first summation is therefore zero.
65
Hilbert Operator for Progressive Encoding (HOPE)
(0) For the second summation, Axiom 1 severs cross-subset weights at initialization 𝒘core = 0. ,𝑗
(𝑡 ) Because 𝐸𝑢 = 0, these structural weights receive zero gradient updates and remain zero 𝒘core = 0. ,𝑗
Thus, the second summation evaluates to zero irrespective of the slack drift 𝑔 (𝑗 𝑡 ) .
Consequently, Δ𝒔core ( 𝒙 ) ≡ 0. Since the downstream core parameters 𝜽𝑢 ∈ {𝒘in,𝑢 , 𝑏𝑢 , 𝛾𝑢 , 𝛽𝑢 } are also frozen 𝐸𝑢 = 0, the downstream core mappings experience zero drift 𝑔𝑢( 𝑡 ) ( 𝒙 ) = 𝑔𝑢(0) ( 𝒙 ). By induction, the target learning is decoupled from the core representation across all layers. □ Remark on Covariate Shift: Theorem H.2 guarantees the immutability of the core operators relative to the source distribution anchor 𝑃 X . In practice, the network processes novel target data D𝑇 during fine-tuning. Consequently, empirical activations flowing through the core will inevitably shift due to standard covariate shift, not parameter degradation. DEFT ensures that any shift in downstream representations is driven entirely by the target data itself, free from the compounding noise of parameter degradation. This preserves foundational knowledge as an uncorrupted lens for processing new domains. Corollary H.3 (Cumulative Masking-Induced Bound). Assuming a masking-based reduction mask evaluates as the cumulative without feature merging, the global structural distortion 𝐷global sum of normalized layer-wise distortions. Normalizing the initialization shock at each layer by Í ( 𝑙 +1) the downstream core capacity 𝐸core ≜ 𝑢 ∈ N ( 𝑙+1) ∥ 𝑓𝑢 ∥ H ( 𝑙+1) , the total distortion is bounded by: core
mask 𝐷global ≜
init, ( 𝑙 +1) 𝐿 −1 Δ𝒔core ∑︁ ( 𝑙 +1)
𝑙 =1
𝐸core
H (𝑙)
≤
(𝑙) 𝐿 −1 ∑︁ |Nslack |𝜏 (𝑙 ) ( 𝑙 +1)
𝑙 =1
! (115)
𝐸core
This confirms that source task degradation is strictly bounded by the chosen capacity thresholds, circumventing the exponential scaling issues of global Lipschitz constants.
H.3. Dynamic Resolution of Redundancy via Bounded Trade-off A static capacity threshold evaluates functional volume in isolation, which ignores the redundancy trap where networks fragment a feature across multiple correlated neurons. To compress this redundant volume and avoid incorrectly locking it, DEFT employs HOPE’s MERGE operation. Proposition H.4 (Bounded Distortion of Merging). Consolidating 𝑀 correlated neurons into a rank-1 parent neuron guarantees the release of target optimization volume while bounding the distortion of the source representation. 𝑀 Proof. Suppose 𝑀 correlated neurons { 𝑓𝑚 } 𝑚 =1 are assigned to the core 𝐸 𝑚 = 0. By generating an ( 𝑙) optimal parent 𝑓 𝑝 ∈ H , DEFT collapses 𝑀 copies into 1 and frees 𝑀 − 1 child vessels. By Axiom 3, DEFT assigns these children full plasticity 𝐸 𝑗 = 1, successfully releasing the parameter space ℝ ( 𝑀 −1) × 𝑐 for optimization.
The distortion 𝛿𝑘 introduced by this operation is the composite projection error between the parent 𝑀 (𝑙) 𝑓 𝑝 and its children { 𝑓𝑚 } 𝑚 =1 . Because the optimal parent minimizes this distance in H , the error is
66
Hilbert Operator for Progressive Encoding (HOPE)
upper-bounded by the sub-optimal projection to the null operator 𝑓 𝑝 ≡ 0:
𝛿𝑘 ≜ min 𝑓𝑝
𝑀 ∑︁
! 12 ∥ 𝑓𝑚 − 𝑓 𝑝 ∥ 2H ( 𝑙 )
≤
𝑀 ∑︁
! 21 ∥ 𝑓𝑚 − 0∥ 2H ( 𝑙 )
(116)
𝑚=1
𝑚=1
Applying the standard ℓ𝑝 -norm inequality ∥ 𝒙 ∥ 2 ≤ ∥ 𝒙 ∥ 1 , we bound this sub-optimal projection by the linear sum of the child capacities: 𝑀 ∑︁ 𝛿𝑘 ≤ ∥ 𝑓𝑚 ∥ H ( 𝑙 ) (117) 𝑚=1
Thus, the algorithm safely releases massive target parameter volume while the distortion remains strictly bounded by the initial capacities. □ Corollary H.5 (The Unified Cumulative Bound). By the triangle inequality in H ( 𝑙 ) , the total perturbation injected into the core of layer 𝑙 + 1 is bounded by the sum of the missing slack projections (Theorem H.1) and the structural projection errors from 𝐾 ( 𝑙 ) merge operations. Because dynamic interference evaluates to zero via structural induction (Theorem H.2), the global normalized functional distortion is bounded as:
total 𝐷global ≜
( 𝑙 +1) 𝐿 −1 Δ𝒔core ∑︁ ( 𝑙 +1)
𝑙 =1
𝐸core
H (𝑙)
(𝑙) (𝑙) 𝐿 −1 ∑︁ © |Nslack | 𝜏 + ≤ ( 𝑙 +1) 𝐸core 𝑙 =1 «
Í𝐾 (𝑙)
𝑘=1 𝛿𝑘 ª
® ¬
(118)
where 𝛿𝑘 is the bounded projection error of the 𝑘-th merge operation. Therefore, DEFT releases target parameter volume while bounding cumulative source degradation to an algorithmically verifiable constant.
I. Algorithms
67
Hilbert Operator for Progressive Encoding (HOPE)
Algorithm 1 HOPE Progressive Encoding Loop Require: Pre-trained model M, Target density 𝜌target 1: Initialize CostManager() ⊲ PHASE 1: O ( 𝑁 2 ) Initialization 2: for each layer 𝐿 ∈ M do
InitializeCaches( 𝐿) ⊲ Computes initial capacities and O ( 𝑁 2 ) geometry ( 𝐿) 4: Anchor initial uncompressed capacity 𝐸0 5: end for ⊲ PHASE 2: True O (1) Greedy Scan 6: while Density(M) > 𝜌target do 7: 𝑎∗ ← null, DRmin ← ∞ ⊲ Evaluate all cached actions using live residual capacities 8: for each active layer 𝐿 do 9: 𝐸rem ← GetResidualCapacity( 𝐿) ⊲ Dynamic tracking of layer shrinkage 10: 𝑁live ← 𝐿.ActiveCount() ⊲ Live layer width for unbiased normalization 11: for each candidate 𝑐 ∈ 𝐿.ActiveActions() do ⊲ Prunes, Merges, Evicts 12: Δ𝑃static ← GetStaticPayoff( 𝑐) ⊲ Action-specific static payoff 13: Jcost ← ComputeDistortion( 𝑐, 𝐸rem , 𝑁live ) ⊲ O (1) scalar arithmetic 14: DR ← Jcost /Δ𝑃static 15: if DR < DRmin then 16: DRmin ← DR 17: 𝑎∗ ← 𝑐 18: end if 19: end for 20: end for ⊲ Execution and Local Synthesis 21: ExecutePhysicalReduction(M, 𝑎∗ ) 22: UpdateResidualCapacity(𝑎∗ .𝐿, 𝑎∗ . Jcost ) ⊲ O (1) Local Recalculation (Strict intra-layer isolation) 23: if 𝑎∗ .type == MERGE then 24: 𝑓 𝑝 ← 𝑎∗ .parent 25: Cache.UpdatePruneCapacity( 𝑓 𝑝 ) ⊲ Update solitary capacity for new parent 26: for each 𝑓neighbor ∈ 𝑎∗ .𝐿.ActiveNeurons() do 27: geonew ← PrecomputePairGeometry( 𝑓 𝑝 , 𝑓neighbor ) 28: Cache.Insert(( 𝑓 𝑝 , 𝑓neighbor ) , geonew ) 29: end for 30: end if ⊲ ZERO downstream recalculation required due to the static Δ𝑃 approximation 31: end while 32: return M 3:
68
Hilbert Operator for Progressive Encoding (HOPE)
Algorithm 2 HOPE Subroutine: UpdateResidualCapacity Require: Executed compression action 𝑎∗ (contains action type, target layer(s) 𝐿, and targeted neurons) 1: 𝜖 ← 10 −12 ⊲ Numerical stability floor for capacity tracking ∗ 2: if 𝑎 .type == PRUNE then 3: 𝐿 ← 𝑎∗ .𝐿 4: 𝑓vic ← 𝑎∗ .victim 5: Δ𝐸 ← Cache.GetPruneCapacity( 𝑓vic ) 6: 𝐿.𝐸rem ← max( 𝐿.𝐸rem − Δ𝐸, 𝜖) 7: 𝐿.RemoveNeuron( 𝑓vic ) 8: else if 𝑎∗ .type == MERGE then 9: 𝐿 ← 𝑎∗ .𝐿 10: 𝑓𝑖 , 𝑓 𝑗 ← 𝑎∗ .children 11: 𝑓 𝑝 ← 𝑎∗ .parent ⊲ Net flux: capacity of the extinguished children minus the new parent 12: Δ𝐸 ← Cache.GetPruneCapacity( 𝑓𝑖 ) + Cache.GetPruneCapacity( 𝑓 𝑗 ) − ComputeCapacity( 𝑓 𝑝 ) 13: 𝐿.𝐸rem ← max( 𝐿.𝐸rem − Δ𝐸, 𝜖) 14: 𝐿.RemoveNeuron( 𝑓 𝑗 ) ⊲ Child 𝑗 is purged; Child 𝑖 serves as the vessel for 𝑓 𝑝 15: else if 𝑎∗ .type == EVICT then ⊲ Eviction targets all internal reduction layers within the residual pathway 16: for each internal layer 𝐿 ∈ 𝑎∗ .block do 17: 𝐿.𝐸rem ← 𝜖 ⊲ Pathway capacity forcibly collapsed 18: 𝐿.ClearAllActiveNeurons() ⊲ All internal operators are projected to 0 19: end for 20: end if
69
Hilbert Operator for Progressive Encoding (HOPE)
Algorithm 3 Dispersed Elastic Fine-Tuning (DEFT) Require: Pre-trained source parameters 𝜃src , Source validation data D𝑠val , Target data D𝑡 , Percentile threshold 𝑃 ⊲ PHASE 1: Capacity Evaluation & Core/Slack Partitioning 1: Merge Redundancies: Use HOPE to compress highly correlated neurons into parent operators, freeing child vessels. 2: Compute Capacities: Calculate the functional capacity ∥ 𝑓𝑖 ∥ H for all active neurons. 3: Thresholding: Determine global capacity threshold 𝜏 corresponding to the 𝑃 -th percentile. 4: for each neuron 𝑖 in the network do 5: if ∥ 𝑓𝑖 ∥ H > 𝜏 and neuron 𝑖 is active then 6: 𝐸𝑖 ← 0 ⊲ Universal Core: Freeze to protect source knowledge 7: else 8: 𝐸𝑖 ← 1 ⊲ Plastic Slack: Freed vessels and weak neurons learn target 9: end if 10: end for ⊲ PHASE 2: Consistency at Initialization (The Structural Mask) 11: Initialize target weights 𝜃0 ← 𝜃src . 12: for each weight 𝑤𝑢, 𝑗 connecting an upstream neuron 𝑗 to a downstream neuron 𝑢 do 13: if 𝐸 𝑗 = 1 and 𝐸𝑢 = 0 then 14: 𝑀𝑢, 𝑗 ← 0 ⊲ Sever cross-connections to prevent Slack drift from entering Core 15: else 16: 𝑀𝑢, 𝑗 ← 1 17: end if 18: end for 19: Apply Mask: 𝜃0 ← 𝑴 ⊙ 𝜃0 ⊲ PHASE 3: Elastic Target Fine-Tuning (Dynamic Decoupling) 20: 𝜃𝑡 ← 𝜃0 , best_h_score ← −1 21: for epoch ∈ {1, . . . , FINETUNE_EPOCHS} do 22: for mini-batch ( 𝒙, 𝒚 ) ∈ D𝑡train do 23: 𝒈 ← ∇𝜃𝑡 Ltarget ( 𝜃𝑡 ; 𝒙, 𝒚 ) ⊲ Compute raw target gradients 24: 𝒈mod ← 𝑬 ⊙ 𝒈 ⊲ Nullify gradients for frozen Core ( 𝐸 𝑖 = 0) 25: 𝜃𝑡 ← OptimizerStep( 𝜃𝑡 , 𝒈mod ) ⊲ Only plastic Slack updates 26: end for // Dual-Domain Evaluation (Harmonic Score Optimization) 27: Acctgt ← Evaluate( 𝜃𝑡 , D𝑡val ) 28: 𝜃src_eval ← MaskSlackDrift( 𝜃𝑡 , 𝜃src , 𝑬) ⊲ Restore pristine Core / Mask out Slack drift val 29: Accsrc ← Evaluate( 𝜃src_eval , D𝑠 ) 2·Acc
·Acc
H-Score ← Acctgttgt+Accsrcsrc 31: if H-Score > best_h_score then 32: best_h_score ← H-Score 33: 𝜃best ← 𝜃𝑡 34: end if 35: end for 36: return 𝜃best 30:
70