Conceptio › Archive › arXiv CS

arXiv CSopen access

CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimization

· arxiv_cs

arXiv CS · Papers · License: Open Access

Open Source ↗Direct PDF ↓

cloud distributed-computing parallel-computing

distributed computing, parallel computing, cloud

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimization

arXiv:2609.15664v1 [cs.DC] 14 Sep 2026

Victor H. Chen˚ , Hairui Yu, Stella K. Chung and Hong Yan, Life Fellow, IEEE

Abstract—Amid the rapid advancement of physical-world intelligence, cloud-edge collaborative large language models (LLMs) have emerged as a promising roadmap for practical LLM deployment. However, existing cloud-edge paradigms struggle to balance global consensus with local personalization, which fails to satisfy the need for a unified knowledge foundation on the cloud and domain-specific adaptation at the edge. To address this, we introduce, for the first time, a personalized bilevel optimization framework that formalizes cloud-edge LLM collaboration as a dual structure: the upper level optimizes edgeside personalization, while the lower level governs cloud-side knowledge transfer, reaching cloud-edge evolving in coordination. We then propose CIDERS, an efficient solver that decomposes the model into a learnable backbone and a messenger. While the cloud performs knowledge transfer to the learnable backbone, the key lies in embedding global trajectories into each local personalization step via consensus-variate correction to reconcile personalization with consensus. We provide a comprehensive theoretical analysis, including a geometric characterization of the local trajectory and a full convergence guarantee, revealing an explicit trade-off structure between personalization and global convergence. Extensive experiments demonstrate that CIDERS consistently outperforms competitive baselines on the compressed edge path, with 3.1× and 1.7× gains on mathematical reasoning and code generation, respectively, and a 10% relative gain on instruction metrics. Mechanism experiments attribute these gains to early consensus-corrected coordination and task-aware distillation. Overall, CIDERS offers a viable path toward consensusguided continuous personalization in cloud-edge LLM systems. Index Terms—cloud-edge LLM collaborative learning, personalized biLevel optimization, consensus-guided personalization, convergence, geometric trajectory.

I. I NTRODUCTION

ARGE language models (LLMs) have evolved into general-purpose productivity tools owing to their powerful cognitive and reasoning capabilities [1], [2]. They have profoundly reshaped the knowledge-intensive work paradigms, which span industrial production, scientific innovation, public services, and business operations etc. [3]. Driven by advances in research and industry, LLMs are expanding beyond pure information processing tasks toward cyber-physical systems (CPS) that interact with the real-world physical environment [4]. Such scenarios demand that the model’s perception, planning, and control capabilities directly serve the operational closed-loop of physical entities and tightly align with their real-time runtime processes, imposing new constraints on the overall deployment and execution architecture [5], [6]. Despite the broadening scope of application scenarios, mainstream industrial LLM tech-stacks are inherently cloud-centric: foundation models are pre-trained on supercomputing clusters to deliver inference services, with vertical

domain adaptation realized via centrally collected domain corpora in the cloud. Edge devices mostly function only as sensing and interaction terminals, uploading prompts and data while receiving inference outputs, without participating in the core model training and updates. This architectural choice arises from intrinsic technical motivations: pre-training, high-throughput decoding, and high-quality domain adaptation all heavily rely on centralized computing resources, high-speed homogeneous interconnection, and unified data governance. Consequently, nearly all production-grade LLM systems follow a three-stage pipeline: pre-training, inference serving, and vertical-domain adaptation. The core of large-scale training lies in multi-granular system-level model partitioning: inter-layer partitioning enables pipeline parallelism, intra-layer matrix partitioning enables tensor parallelism, and these are combined with data parallelism to form three-dimensional collaborative training. This enables trainability and convergence of hundred-billion-parameter models across multi-machine clusters. Subsequent instruction alignment transforms vanilla continuation-oriented models into general-purpose service models capable of following human instructions. Online inference organizes the step-wise autoregressive generation process into a concurrent serving system: cached historical key-value (KV) states eliminate redundant computation, and dynamic batching accommodates irregularly arriving requests, rendering time-to-first-token, per-token generation latency, and per-unit cost measurable and optimizable. Task adaptation absorbs domain discrepancies via lightweight parameter fine-tuning [7], [8], [9]. It incorporates private or time-sensitive information through external knowledge bases and retrieval-augmented generation, and compresses LLMs into deployable compact forms via knowledge distillation. While the cloud-centric paradigm delivers remarkable computing and iteration efficiency, it suffers from structural mismatches for scenarios with strict latency requirements, data-privacy demands, and physical-interaction constraints [5], [6]. Round-trip network latency becomes a critical bottleneck that fails to support hard real-time applications requiring instantaneous response, including autonomous driving decision-making, industrial robotic closed-loop control, and field edge deployments with limited network backhaul. Sensing logs and business documents generated at the edge are often prohibited from offloading by compliance and commercial restrictions, rendering cloud LLM APIs unsuitable for local model fine-tuning. Bandwidth and compute overhead scale with token volume and KV-cache traffic rather than the intrinsic intellectual complexity of tasks. A more salient mismatch lies in personalization: user trajectories, industrial logs, and

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

sensor-text samples for model customization originate on the edge, whereas all modifiable model parameters reside in the cloud. In short, cloud-centric architectures realize centralized hosting of model capabilities, yet data, timing constraints, and physical processes are inherently distributed. Rather than the merely performance optimization, this fundamental misalignment renders cloud-edge collaboration an architectural necessity, and here we summarize the inference and learning paradigms as follows: Cloud-edge collaborative inference: To overcome the resource limitations of individual edge devices, the system treats the cloud and the heterogeneous edge nodes as a programmable resource pool [10], [11], [12]. The workflow starts with offline characterization and planning: it measures the computing, memory and the inter-device bandwidth, then performs the global placement decisions via heuristic or optimization-based approaches [13], [11], [12] for the model partitioning and deployment [14], [15], [16]. Subsequently the system starts inference via pipeline parallelism, where each device computes only its local shard and forwards intermediate activations to downstream components, which overlaps computation with communication [13], [11]. Nevertheless, pipeline stalls may occur at shard boundaries, due to heterogeneous straggler nodes, or induced by wide-area round-trip delays [17]. Under such circumstances, a complete lightweight model can be deployed in parallel at the edge to proactively generate subsequent candidate tokens conditioned on available context during waiting intervals [18], [19], and the cloud performs one-shot validation over the complete candidate window or candidate tree: upon acceptance, the corresponding prefix is advanced while upon rejection, key-value states are rolled back and generation restarts from the point of divergence [20]. Cloud-edge collaborative learning: To address the dual challenges of the growing scarcity of high-quality private data and the need for privacy preservation, this paradigm integrates federated learning (FL) with parameter-efficient fine-tuning (PEFT). It enables multiple clients to collaboratively perform domain adaptation for large language models (LLMs) without exposing their raw local data. Specifically, each client introduces lightweight low-rank adaptation modules for the frozen LLM, and selects the rank of the adapters and performs local learning based on its local resources and data [21], [22], [23]. Followed by the cloud aggregation, clients upload only a small set of incremental parameters to the cloud, which employs mechanisms such as stacking-based aggregation or selective sharing to eliminate aggregation noise. The global adapter can both absorb common knowledge from across clients and preserve the personalized characteristics inherent in local data [21], [24], [25]. To alleviate the computational burden on edge nodes, a split FL architecture is further introduced: the model is logically partitioned into client-side and server-side sub-models, where clients only need to perform lightweight forward propagation and upload activations to the cloud, while offloading the majority of gradient computation and parameter updates to high-performance cloud servers [26], [27]. Complemented by a temporal redundancy-aware activation reuse mechanism, the system can skip the transmission of

activations that exhibit only minor changes between adjacent training epochs, substantially reducing uplink communication overhead [27], [28]. However, we observe one paradox. Existing cloud-edge collaborations for LLMs are fundamentally designed on model consensus, where all edge nodes share the common model parameters. This inherently overlooks one fact: data, user behaviors, and physical environments are naturally personalized in edge scenarios, while the capability to adjust model parameters is dispersed across the cloud and the edge [29], [30]. Therefore it is necessary to form a systematic architecture with edge personalization: each edge node, while sharing a common global knowledge base, can evolve its model based on its own local data and feedback, achieving fast general capabilities while personalized fine-tuning adapts to specific scenarios. To this end, we design CIDERS, which is precisely architected to address this gap: it first decomposes each edge model into a globally shared learnable backbone that captures general linguistic and reasoning capabilities via cloud-based task-aware distillation, and a locally adaptive messenger that captures domain-specific personalization via edge-local updates [31], [32]. The two components are orchestrated through our proposed personalized bi-level optimization framework, where the upper level optimizes the messenger for rapid client-specific adaptation, while the lower level ensures the backbone remains aligned with both the frozen teacher and the downstream task manifold. To further anchor local trajectories to global consensus during edge updates, CIDERS introduces a consensus-corrected geometric update that continuously interpolates between pure local specialization and global consensus through a tunable scalar [3]. Crucially, the algorithm transmits only compressed consensus increments to the cloud, enabling low-bandwidth communication while preserving exact reconstruction of local displacements. This design skillfully decouples the learning procedures to achieve what prior cloud-edge paradigms cannot: a systematic continuum from a shared common knowledge base to individually evolved edge models. In summary, our contributions lies: Personalized bilevel optimization formulation: Based on the aforementioned decomposition of the LLM into the messenger and the backbones, we formulate for the first time the personalized bilevel optimization. Specifically, the upper-level problem optimizes the messenger that enables fast clientspecific personalization, thereby learning for adaptability to heterogeneous local distributions. This is fundamentally different from conventional FL objectives, which optimize for global model accuracy without regard for how quickly or how well that model can be adapted to individual clients. The lower-level problem learns the student backbone via taskaware knowledge distillation from the frozen full backbone. This ensures that the compressed backbone faithfully emulates the teacher’s foundational capabilities while remaining aligned with the downstream task manifold. This formulation is the first to capture the distinct roles of personalization and globalization in a principled bilevel optimization. The novel algorithm CIDERS: We propose CIDERS that solves the personalized bilevel optimization. Specifically, the

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

edge performs consensus-variate-corrected local update on two distinct scales: the global trajectory is incorporated into the local counterpart. This steers each client’s trajectory toward a tunable balance between the local specialization (slower trajectory) and global consensus (fast trjectory). Then, CIDERS transmits only consensus-variate increments, enabling exact reconstruction of local displacements on the cloud while reducing the communication overhead. Finally, with aggregated messenger fixed, it performs task-aware knowledge distillation on the learnable backbone. This fixed messenger conditioning ensures that the backbone learns representations that are not only faithful to the teacher but also coherent with the current personalization context, which prevents misalignment if the messenger and backbone evolved independently. Comprehensive theoretical analysis: We establish the first complete theoretical foundation for personalized bilevel optimization in cloud-edge LLM systems. We first comprehensively demonstrate that CIDERS attains a sublinear convergence to a stationary point. Then with convergence guarantee, our geometric analysis provides an interpretable characterization of the local update trajectory. We show that the effective direction of the messenger lies on the line segment connecting the local personalized gradient and the global consensus direction, a result that reveals how the consensus scalar sets the personalization–globalization balance, the consensusvariate learning rate governs the transition speed. Based on this insight, we propose two diagnostic metrics for a direct quantitative readout of where any given trajectory sits on the personalization–consensus spectrum. Comprehensive experimental analysis: We conduct comprehensive experiments on Qwen2.5-3B/1.5B that probe the underlying mechanisms rather than merely benchmarking performance. These experiments first establish CIDERS’s empirical superiority, while also revealing several key mechanistic insights that the bilevel coordination yields an average relative improvement of approximately 12% across six downstream metrics, with the largest single gain reaching 11.15% on GSM8K, and the consensus diagnostics validate the geometric predictions that directional personalization is preserved, whereas output-level specialization remains marginal and can be decoupled from the directional component. Collectively, these findings establish that consensus-guided continuous personalization substantially improves both system performance and robustness in cloud-edge LLM deployment. II. R ELATED W ORKS A. Federated Learning and Personalization Federated learning (FL) enables distributed clients to collaboratively train a global model without centralizing private data [29]. FedAvg serves as the standard baseline, where clients perform local gradient steps and the server aggregates updates via weighted averaging [34]. However, under hterogeneity, client drift and objective inconsistency arise. FedProx adds a proximal term to constrain local-global model deviation [35] and FedDyn introduces dynamic regularization for the enhancement [36]. FedNova normalizes local updates to correct biases from heterogeneous local steps [37]. MOON

enhances local-global representation consistency through contrastive learning [38]. SCAFFOLD maintains control variates on both server and clients, using their difference to correct local update directions, this can be viewed as a variance reduction mechanism for heterogeneous FL [3]. In personalized FL, FedPer and FedRep decompose models into shared bottom layers and client-specific top layers or heads [39], [40]. pFedMe formulates personalization through a Moreauenvelope objective [41]. Ditto jointly learns global and personalized models with regularization to balance their deviation [42]. Meta-learning-based Per-FedAvg brings MAML to FL [43], optimizing a global initialization that enables rapid client adaptation after one or a few gradient steps [44], [30]. In general, existing FL and personalization methods provide a theoretical foundation for distributed LLM learning. However, they are designed for conventional models and full-parameter spaces. When applied to cloud-edge LLM scenarios, they face challenges: excessive parameter scale, infeasible controlstate maintenance, and coupling between local updates and cloud-side distillation. These issues call for redesigns in both personalization architecture and optimization mechanisms.

B. Cloud-Edge LLM Collaborative Learning via FL Full-parameter fine-tuning of LLMs is expensive in resource costs. Parameter-efficient fine-tuning (PEFT) thus becomes a foundation for federated LLM adaptation [45]. Adapter inserts trainable modules into frozen models [7]. Specifically, prefix-tuning optimizes continuous prefix vectors [8], while LoRA freezes pretrained weights and learns low-rank increments [9]. These methods significantly reduce trainable parameters and thus federated LLM fine-tuning further explores how to adapt LLMs on distributed private instruction data [46], [47]. FederatedScope-LLM provides a system framework and benchmark support, while it establishes the universally acknowledged challenges in communication, computation and heterogeneity [48]. FLoRA finds naive averaging in the direclty combined LoRA and FL is mathematically inexact, which can be simply mitigated by stack based aggregation [21]. FedSA-LoRA analyzes the asymmetric roles of LoRA matrices and shares only the component carrying more general knowledge [49]. FedALT mitigates cross-client interference by combining individual LoRAs with shared background LoRA [50]. To obtain smaller models for edge deployment, knowledge distillation transfers output distributions, hidden representations, or attention relations from teacher to student [51]. DistilBERT, TinyBERT, and MiniLM distill compact students from larger LLMs [52], [53], [54]. Model decomposition further decouples the full LLM into modules and learns them separately. Specifically, offsite-tuning sends a compressed learnable backbone and lightweight adapter to the data owner, who adapts without accessing the full model [31]. FedBiOT extends this to federated LLM fine-tuning: the server builds a compressed model and aligns it with the full model via distillation, while clients fine-tune lightweight adapters on the fixed compressed model [32].

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

CIDERS Technical Roadmap Cloud LLM

1. Model decomposition

𝒄𝒊

2. Equidistant Layer Sampling

P 𝓑𝒇𝒓𝒐𝒛𝒆𝒏

𝓑𝑭

Edge Client Update

Model Prep

𝓑𝑳

P Aligned

𝓑𝑳

𝒄

Corrected trajectory

Local Personalization Direction (from local meta-gradient) 𝒈˜𝒊

Reconstruction

Consensus trajectory

Client i in S

𝒄𝒊

Δ𝑐

Client i in S

𝓑𝑳

(𝒕)

𝜸 𝒄(𝒕) − 𝒄𝒊

…… ……

Client j in S

𝒄𝒋

Client i in S

Aggregation

Updated 𝓑𝑳

Global Consensus Correction (pull towards global consensuss)

𝓑𝒇𝒓𝒐𝒛𝒆𝒏

Aggregation

Drifted local trajectory

Fixed

𝓑𝒇𝒓𝒐𝒛𝒆𝒏

3. Pre-alignment Sample index 𝑰 is stored for reconstruction

𝒄

Fixed

𝓑𝑳

𝒄

𝓑𝒇𝒓𝒐𝒛𝒆𝒏

𝑻𝑨𝑲𝑫

𝓑𝑭

Figure 1: CIDERS Workflow. (I) The framework decomposes the LLM into a cloud server-side full backbone BF , edge client-side messengers PM and learnable backbone BL . (II) Each round the server broadcasts twPM , wBL , cu; clients run a K-step consensus-corrected geometric update on private data and upload ∆ci only. (III) The server reconstructs ∆i , aggregates the messenger and consensus, then performs knowledge transfer from BF to BL with PM held fixed before the next broadcast.

C. Discussion

as follows

FL, as a distributed learning framework, has been widely applied across deep learning. Cloud-edge LLM collaborative learning is inherently distributed, so FL provides a natural fit. Both PEFT and its combination with model decomposition explores heterogeneous adapter configurations, yet they lack systematic handling of drift-aware updates and are insufficient for edge personalization to meet complex applications, since they meet the paradox that personalization and consensus are difficult to coexist in the local adapter update path. To address this, CIDERS integrates both into a unified bilevel optimization framework that injects global trajectory into each local personalization step for coupling local personalization and global consensus within the same optimization process. We target LLMs continuously evolving at the edge while staying coordinated with global consensus in cloud-edge collaboration.

˘ ` 1 ÿN rPM,i , wBL , (1) Li w i“1 N rPM,i :“ wPM ´ ηin ∇wPM Li pwPM , wBL q denotes the where w one-step personalized messenger for client i, and ηin ą 0 is the inner learning rate. Note (1) optimizes the global messenger wPM as a meta-model that facilitates rapid clientspecific adaptation.

III. P ROBLEM F ORMULATION We consider a cloud-edge distributive system comprising a cloud server with a public dataset Dpublic and N client edges, each possessing a private heterogeneous dataset Di . Let Mi denote the full composite model on client i, and M˚ denote its server-side counterpart. To balance global coordination and local adaptation, we partition these models into distinct yet structurally interdependent components: Mi “ PM,i ˝ BL on the client side, and M˚ “ PM ˝ BF on the server side. Specifically, we define: ‚ A full backbone BF , parameterized by wBF , which serves as a static, frozen teacher preserving foundational linguistic capabilities and world knowledge. ‚ A learnable backbone BL , parameterized by wBL , which acts as the global representation student distilled from BF using public data. ‚ The messengers PM,i , parameterized by wPM,i for i “ 1, ¨ ¨ ¨ , N . These capture domain-specific knowledge from private datasets, while PM (parameterized by wPM ) denotes their globally aggregated meta-initialization counterpart on the server. Upper-Level: Personalized Meta-Objective. We formulate the upper-level problem by adopting the personalization objective

Lpers pwPM , wBL q “

Lower-Level: Task-Aware Knowledge Distillation. The lowerlevel objective aims to transfer server-side knowledge to the learnable backbone. While general KD anchors the student to the teacher’s latent space via intermediate representation matching and output logit alignment, it treats the frozen teacher as an infallible oracle. This risks propagating pretraining flaws or calibration biases without optimizing for downstream utility. To mitigate this, we propose the TaskAware Knowledge Distillation (TAKD), which is denoted as LKD on Dpublic with the fixed global messenger wPM : „ LKD pwBL ; wPM q “ E }BL px; wBL q ´ BF px; wBF q}22 ` λDKL pM˚ px; twPM , wBF uq } Mpx; twPM , wBL uqq (2) ȷ ` λtask ¨ ℓtask pMpx; twPM , wBL uq, yq , where λ, λtask ą 0. It aims to explicitly injects task-aware supervision grounded in true labels y. This ensures BL fully emulates BF and aligns with the target task manifold, providing a high-quality foundation for the upper-level metaadaptation. The Personalized Bi-Level Optimization (PBO) Alternating the optimization of wPM and wBL via the upper and lower objectives respectively, we formulate the personalized bilevel optimization (PBO) for the cloud-edge LLM collaborative learning: ϵ ´ }2 min Lpers pwPM , wBL q ` }wPM ´ wP M 2 wPM 2 (3) s.t. wBL “ arg min LKD pwBL ; wPM q . wBL

In this architecture, we regularize (1) by a proximal term ´ to constrain drift from the previous state wP , then the M

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

upper-level optimization learns the global messenger wPM to strike a balance between globalization and personalization. Alternately, the lower-level optimization trains wBL via TAKD wBF . This hierarchy explicitly decouples global knowledge alignment from client-specific personalization. Optimization Challenges. While (3) forms the first personalized bilevel optimization, we pose several key insights in challenges: first is obvious, the computation resource is limited at edges, which requires fast global training. Second, since the cloud and the edges are mutually dependent, if the cloud fails the distillation or the edges result in heterogeneity issues, they will alternate their updates to a continuous deterioration of mutual learning. Third, edges generally demand personalization, which exacerbates the heterogeneity issue and leads to a decrease in the overall training efficiency. IV. M ETHODOLOGY To enable thorough learning under computational efficiency, CIDERS coordinates two distinct roles, i.e., client-specific messenger adaptation and global teacher-guided backbone alignment. It considers only messenger updates at edges. Then it meticulously navigates the personalization with the global trajectory for fast adapting meta-knowledge encoded within the messenger. On the cloud, the client messengers are aggregated and fixed, then BL learns foundational linguistic representations encoded within the full backbone. This architectural decomposition and orchestration provide a complete coordination pathway for heterogeneous edge LLMs reaching personalization with the fast global convergence. We summarize the cloud-edge LLM collaborative learning procedure in Algorithm 2. A. Compressed Model Preparation For the pre-trained LLM M˚ with n transformer layers, we decompose M˚ into distinct functional modules (BF , BL , PM ). This structural decomposition is grounded in the hierarchical representation learning of LLMs: lower layers encode shared domain-agnostic linguistic priors, while upper layers capture heterogeneous higher-order semantic abstractions. Consequently, the topmost layers are chosen as the messenger PM for rapid local adaptation, while the frozen full backbone BF preserves foundational capabilities. The learnable backbone BL is a uniform strided subsample of BF , yielding a compact global surrogate. Since directly launching cloudedge collaborative learning on these disjoint modules may flaw due to the heterogeneity of PM ’s massive representation mismatches, pre-alignment via TAKD in (2) for initializing BL should be implemented before client updates so that the local updates start from a teacher-informed compressed path. ‚ Step 1: PM Identification. The messenger PM , parameterized by wPM , comprises the topmost a layers of M˚ . It acts as the client-side adapter to capture domain-specific knowledge. The remaining model constitutes the frozen full backbone BF (wBF ). ‚ Step 2: Learnable Backbone Construction. To preserve the teacher’s depth-wise coverage on clients, we compress BF

Algorithm 1 Module Preparation 1: function M ODEL P REP(M˚ , a, β, Dpub ) 2: Obtain the LLM layer number n Ð |M˚ |,

Compute BL ’s layers n1 Ð tβ ¨ pn ´ aqu. BF Ð tM˚,i un´a´1 , PM Ð tM˚,i un´1 i“0 i“n´a ; ( ␣ u | j “ 0, . . . , n1 ´ 1 , 5: Get Indices Ð tj ¨ n´a´1 1 n ´1 6: Sample to form BL Ð tpBF qk | k P Indicesu, 7: Initialize wPM , wBF and wBL 8: Train wBL Ð argminwB LKD pwBL ; wPM , Dpub q L 9: return PM , BF , BL ; 10: end function 3: 4:

by uniformly extracting its nE “ tβ ¨ pn ´ aqu to construct BL (wBL ), where β P p0, 1s is the compression rate. ‚ Step 3: Pre-alignment via TAKD. Prior to collaborative learning, BL undergoes TAKD on Dpublic for learning BF ’s linguistic priors. In summary, we detail the whole preparation in Algorithm 1. B. Client Update Procedure The client update is meticulously designed to solve the upper-level personalized meta-objective in (1) under the complex environment of the real-world. Our strategy targets incorporating the global learning tragectory into the meta-gradient for personalization adaptation under data heterogeneity. Consensus-corrected geometric update. At round t, each participated client i P St receives the global state triplet ptq ptq tw ,w , cptq u, then it initializes its local messenger at the PM BL current global messenger, i.e., wPptqM,i,0 Ð wPptqM , and retrieves its historical consensus variate cptq i . To prevent overfitting to local noise while navigating the heterogeneous landscape, the messenger is updated via a consensus-variate-corrected geometric interpolation over K steps: ptq

ptq

wPM,i,k`1 “ wPM,i,k ´ ηgtraj , where ` ` ptq ptq ptq ˘ ptq ˘ gtraj “ gri,k ` γ cptq ´ ci ` ϵ wPM,i,k ´ wPM ,

(4)

gri,k is the personalized meta-gradient computed based on a mini-batch ξi,k „ Di such that ´ ¯ ptq ptq gri,k “ ∇wPM Li w rPM,i,k , wBL ; ξi,k , (5) ptq

with w rPM,i,k being the one-step look-ahead as follows: ´ ¯ ptq ptq ptq ptq w rPM,i,k “ wPM,i,k ´ ηinner ∇wPM Li wPM,i,k , wBL ; ξi,k . (6) In practical inplementations, (4) in fact uses SGD in the optiptq ptq ptq mizer tool, i.e., wPM,i,k`1 Ð OPTIMpSGD, wPM,i,k , gtraj , ηq, and other optimizer such as AdamW works also well. To circumvent the heavy computation of gri,k , CIDERS employs two Hessian-Free (HF) meta-optimization strategies. The first is FO strategy, which intentionally drops the second-order derivative, assuming the Hessian impact is locally negligiFO ble: vi :“ gri,k “ ∇wPM Li pwPM ´ ηinner ∇wPM Li q. This requires only standard forward-backward passes, serving as the

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

lightweight default. The second is FD, which adopts centraldifference Hessian–vector estimate ∇Li pw ` εFD vi q ´ ∇Li pw ´ εFD vi q z H , (7) i vi “ 2εFD and the curvature-aware surrogate can be obtained giFD “ z vi ´ λHVP H i vi , and here λHVP “ ηin . We further provide an insightful analysis of the consensus-corrected geometric update in (4) via the following three aspects. First, the term ptq γpcptq ´ ci q estimates of how far is the local meta-gradients to the global gradient direction and reaches the consensusinformed global–personal coordination. Second, the geometric scaler γ P r0, 1s explicitly parameterizes a continuous zerosum game between global consensus and local specialization: when γ “ 1, the correction fully aligns the client toward the global meta-initialization, when γ Ñ 0, the update reduces to pure local meta-gradient descent. Third, the proximal term ptq ptq ´ϵηpwPM,i,k ´wPM q bounds divergence from the current global state, improving stability in heterogeneous regimes. After completing K local steps, we record each client’s average update trajectory as: ¯ 1 ´ ptq ptq wPM ´ wPM,i,K , (8) ∆i,t :“ Kη which serves as the estimator of the local tragectory. It can be seen ∆i can be used to update both the global messenger and the global consensus variate on the cloud in (12). The local consensus variate is designed to estimate the local metagradient with the K averaging of gri,k and is subsequently refreshed using an exponential moving average (EMA) as follows: “ ‰ ptq pt`1q ci “ 1 ` αpγ ´ 1q ci ` α∆i,t ´ αγcptq . (9) Then client i transmits the incremental consensus difference pt`1q

∆ci “ ci

ptq

´ ci

(10)

to the cloud. It can be seen that with ∆ci , the cloud can recover ∆i,t by storing ci for updating the global messenger. Consequently, the upstream communication payload is reduced to a single low-dimensional consensus increment. C. Server Update Procedure As the centralized orchestrator, the server advances the global meta-initialization via consensus-informed aggregation and refining the global representation backbone via task-aware distillation. With our skillful design, CIDERS can update the client messengers and the consensus variate via recovering ∆i from ∆ci . Key information reconstruction. For each participated client i P St , it receives the consensus variate increment ∆ci “ pt`1q ptq ci ´ ci . Then the server exactly recovers the averaged local messenger update trajectory ∆i,t using its synchronized ptq historical state ci : 1 ptq ∆ci ´ pγ ´ 1qci ` γcptq . (11) α The cloud-stored local consensus variate copies are subsept`1q ptq quently updated via ci Ð ci ` ∆ci . This strategic ∆i,t “

Algorithm 2 Algorithmic Framwork of CIDERS Input: Original LLM M˚ , length of messenger a, comp0q pression rate β, consensus cp0q , ci and the parameters pα, γ, τ, η, K, T q. 1: tPM , BF , BL u Ð MODELPREPpM˚ , a, β, Dpub q; 2: for t “ 0, 1, . . . , T ´ 1 do 3: Server performs the lower-level training 4: Recovers exact updates t∆i,t uiPSt from t∆ci uiPSt . 5: Performs the consensus and ř model aggregation via 6: cpt`1q Ð p1´αqcptq ` α iPSt pi ∆i,t and ř pt`1q ptq 7: wPM Ð wPM ´ τ iPSt pi ∆i,t . 8: Performs TAKD for the coordination of BL and PM pt`1q wBL Ð arg minwBL LKD on Dpub pt`1q pt`1q 9: Transmits twPM , wBL , cpt`1q u to St`1 ; 10: Client performs upper-level training 11: for each client i P St in parallel do ptq 12: Obtains the local messenger wPM,i,K via (4) ` ptq ˘ ptq 13: Calculates ∆i,t Ð 1{Kη wPM ´ wPM,i,K pt`1q

ptq

Updates ci Ð r1`αpγ´1qsci `α∆i,t´αγcptq . 15: end for pt`1q ptq 16: Transmits ∆ci Ð ci ´ ci to the cloud. 17: end for pT q pT q 18: Edge inferences via: Mi Ð ComposepwPM,i , wBL q; 14:

recovery avoids the upstream transmission of raw messenger weights, thereby reducing the bandwidth costs and shielding the local optimization trajectories from direct privacy exposure. Global aggregation. Given the reconstructed t∆i,t uiPSt , the server simultaneously advances the aggregation of the global messenger and the global consensus variate as follows: ÿ pt`1q ptq wPM “ wPM ´ τ Kη pi ∆i,t , iPSt ÿ (12) cpt`1q “ p1 ´ αqcptq ` α pi ∆i,t , iPSt ř where pi “ |Di |{ jPSt |Dj | (here pi “ 1{N for simplicity), and τ ą 0. Geometrically, the global messenger update acts as a convex combination of the current meta-initialization and the local adaptations, with the learning rate τ dampening aggregation noise inherent to partial client participation. Concurrently, cpt`1q functions as a population-level consensus anchor. It tracks the moving average of the personalized gradients to smooth out round-to-round variance and provide an unbiased reference direction for subsequent client adaptations. Learning BL via TAKD. Finally, the lower-level objective in (3) learns BL from BF . By fixing PM , this procedure also plays the role in coordination and adaptation, ensuring the alignment with the current meta-adaptation context. Specifically, we have: ´ ¯ pt`1q pt`1q wBL Ð arg min LKD wBL ; wPM . (13) wBL

While this guarantees that BL absorbs foundational linguistic features, it establishes a robust, task-aligned representation foundation for the next round of client update.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

V. T HEORETICAL A NALYSIS We have proposed CIDERS, and the key mechanisms are twofold: we first divide the LLM into functional modules and adopts different optimization strategies for each of them, then we incoporates the global tracjectory in the personalized bilevel optimization framework (3). This will bring complex principles. Luckily, our proposed framework is theoretically robust. A. Convergence Analysis In this section, we analyze the proposed personalized bilevel structure solved by CIDERS, which we show achieves a sublinear convergence rate to a neighborhood of the stationary point. For simplicity, we denote the PBO as: ›2 ε› min Φpvq ` ›v ´ v ´ › , s.t. wS “ arg min LKD pwS ; vq , wS v 2 (14) where v denotes the global řmessenger, wS the learnable student backbone, Φpvq “ N1 i Li pr vi , wS˚ pvqq is the upperlevel meta-objective with vri “ v ´ ηinner ∇v Li pv, wS˚ pvqq, and wS˚ pvq is the exact minimizer of the lower-level task-aware knowledge distillation loss LKD p¨; vq on the public dataset. ptq Since CIDERS maintains an inexact backbone wS together with consensus-corrected local messenger steps, the quantity of interest is the joint gap › ptq ›2 “ ‰ Jt :“ E Φpv ptq q ´ Φ˚ ` E›wS ´ wS˚ pv ptq q› . (15) Moreover, to track the coupled evolution of all error sources, we construct a Lyapunov function to couple all error sources, i.e., the virtual messenger error, client drift Et , consensus lags rt , and the joint gap Jt : Ct , C ›2 ` ˘ ›› › V ptq “ 1 ´ 4a2 E ›z ptq ´ v ˚ › ` ω1 Et (16) rt ` Qw4 ηJt , ` Qw2 ηCt ` Qw3 η C where we have defined z ptq “ v ptq ` p1´τ q{τ ¨ pv ptq ´ v pt´1q q, and a, ω1 , Qw2 , Qw3 , Qw4 are positive. Then, the convergence of CIDERS solving PBO (3) is: Theorem 1 (Ergodic convergence of CIDERS). Under regular assumptions and moderate conditions, let the step size satisfies η “ OpT ´1{2 q, then the ergodic joint gap satisfies where p0q V0 “ p1 ´ 4a2 qE}z p0q ´ v ˚ }2 ` ω1 E0 and Cη2 ą 0 is a 1 constant from the expansion of Dtot . Then the ergodic average of the joint gap satisfies d T ´1

1 ÿ Cη 2 Jt ď ` T t“0 δΦ δΦ

p0q

´ ¯ V0 Cη2 V1p0q Cη3 V0 ` ` `O T ´3{2 , T δΦ T δΦ Cη2 T (17) p0q

where Cη , Cη2 , Cη3 are positive constants, V0 “ p1 ´ 4a2 qE}z p0q ´ v ˚ }2 ` ω1 E0 collects the intial client drift, p0q r0 ` Qw4 J0 collects the initial and V1 “ Qw2 C0 ` Qw3 C consensus lags and the joint gap. The detailed proof and discussions on the convergence result is moved to Appendix.

B. Geometry of the Trajectory Consider a fixed communication round t, the client i obtains ptq the global messenger v̄ :“ wPM from the server and performs multiple local updates, where the trajectory is simultaneously determined by three forces: the local meta-gradient provides a personalized direction, the consensus correction pulls the ptq local messenger xk :“ wPM,i,k towards the global consensus, and the proximal constraint resists the messenger’s deviation from the starting point, which constitutes a tightly coupled nonlinear dynamical process, making it extremely difficult for an analysis. Here, we develop an approximate geometric characterization that isolates the core ř factors. Specifically, let K´1 us define ρ :“ 1 ´ ηε and SK :“ j“0 ρK´1´j , we can unroll (4) ´ ¯ ÿK´1 ptq v̄ ´ xK “ η ρK´1´j gi,j ` ηSK ¨ γ cptq ´ ci . (18) j“0

ptq

Then, we define the along-path average meta-gradient gsi :“ řK´1 K´1´j 1{SK gi,j . Then (18) can be rewritten as j“0 ρ ´ ´ ¯¯ ptq ptq ∆i,t “ σK ḡi ` γ cptq ´ ci , (19) where σK :“ SK {K P p0, 1s. It can be seen the correction shapes the direction and the proximity modulates the magnitude. ptq 1) Convex Combination: Let gri :“ ∇Li pv̄q denote the lor ptq :“ ř pj grptq cal meta-gradient at the broadcast point and G j j the global average gradient. Define the heterogeneity gap ptq ptq r ptq . To render the directional dynamics δi :“ gri ´ G analytically tractable and obtain a closed-form characterization of the personalization and globalization trade-off, we introduce the idealizing approximations: A1. Short local step: The total local displacement is sufficiently small, i.e., Kη ! 1{L, where L is the Lipptq schitz constant of the meta-gradient, so that gsi « ptq ptq ptq ptq g̃i . The path error ϕi :“ gsi ´ g̃i is bounded by O pLCK ηGbd q. A2. Fixed heterogeneity: The heterogeneity gap is constant pt`1q ptq across rounds, i.e., δi “ δi “ δi . This holds exactly for quadratic meta-losses and serves as a firstorder approximation when the messenger moves slowly. A3. No proximal damping: We set ε “ 0, hence σK “ 1. The case σK ‰ 1 is treated separately and shown to preserve span invariance while renormalizing the scalar recurrence. p0q p0q A4. Cold start: Initializes cp0q “ ci “ 0, thus ri “ δi . A5. Full participation: All clients participate in every round. Under (A1-A5), the displacement is simplified to ∆i,t « ptq ptq gri ` γpcptq ´ ci q, We now define the reference field, which is he effective total direction as follows ptq

ptq

ui :“ gri ` γpcptq ´ ci q.

(20)

To track how this reference field relates to the global consensus r ptq , we introduce the residual direction G ptq ptq r ptq “ δ ptq ` γpcptq ´ cptq q. r :“ u ´ G (21) i

Under the idealizing assumptions, the consensus-gap recursion ptq yields a closed-form recurrence for ri : pt`1q

ptq

p0q

“ p1 ´ αqri ` αp1 ´ γqδi where ri

“ δi .

(22)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

ptq

In fact, the residual remains strictly proportional to δi : ptq

ri “ ψt δi where ψt :“ p1 ´ γq ` γp1 ´ αqt . ptq

Substituting (23) back into the definition of ui

(23)

ψt`1 “ r1 ´ α ` αγ p1 ´ σK qs ψt ` αp1 ´ γq, ψ0 “ 1, (28) whose steady-state solution can be derived as follows

ψt “ p1 ´ γq ` γp1 ´ αqt . (24) ptq

Moreover, the displacement satisfies ∆i,t « ui . Proposition 2 provides a geometric characterization of the local messenger trajectory. The global consensus weight p1 ´ ψt q “ γ p1 ´ p1 ´ αqt q increases monotonically from 0 at t “ 0 to γ as t Ñ 8, with a half-life of approximately Θplog 2{| logp1 ´ αq|q rounds. The scalar ψt thus interpolates between two regimes: at cold start pψ0 “ 1q, the direction is purely local; at lock-in ψ8 “ 1 ´ γ, the direction stabilizes at the convex combination specified by γ. The parameter α controls the speed of transition, while the pair pc, ci q implements this transition through an exponential moving average that smoothly drives the translation from 0 to ´γδi . 2) Quantifying Deviations.: We quantify the direction deviations from the idea segment under the idealizing assumptions. Our first key insight is relaxing the short-inner-loop assumption (A1) introduces a path-averaging error. Note the alongpath ptq ptq ptq ptq error ϕi :“ ḡi ´ g̃i is bounded by }ϕi } ď LCK ηGbd . Applying a variation of constants argument to the residual recurrence gives the uniform bound ` ˘ ptq ptq }ui ´ ui |pE0q } ď 2γR 1 ´ p1 ´ αqt ď 2γR, (25) ptq

where R :“ LCK ηGbd and ui |pE0q denotes the ideal direction from Proposition 2. Thus, the true direction lies within an O pγLKηGbd q neighborhood of the ideal segment. This deviation vanishes as Kη Ñ 0 and scales linearly with γ, reflecting the fact that stronger consensus alignment amplifies the sensitivity of the consensus variates to gradient estimation error. Second, even when the consensus are ideally locked, r and ci « gri , the executed field at an arbitrary i.e., c « G point x along the inner path contains a Hessian remainder. Specifically, the Taylor expansion around v̄ gives ∇Li pxq ` γ pc ´ ci q « p1 ´ γq∇Li pxq ` ˘ (26) s px ´ v̄q, ` γ∇Lpxq ` γ Hi ´ H ř where Hi :“ ∇2 Li pv̄q and H̄ :“ j pj›Hj . The ›remainder is bounded by γχ}x´ v̄}, with χ :“ maxi ›Hi ´ H̄ › quantifying the heterogeneity of meta-Hessians across clients. This implies that when γ is large to extent, long local step may accumulate Hessian mismatch rather than improve consensus. 3) The Case σK ‰ 1: When ε ą 0, it satisfies σK “ SK {K ă 1. Under (A1), (A2), and (A4), the recurrence generalizes to pt`1q

ptq

It can be seen the span-invariance property ri P span tδi u is ptq preserved, i.e., ri “ ψt δi , but ψt is modified as,

yields:

Proposition 2 (Convex combination of local and global directions). Under approximations (A1)-(A5), the reference ptq direction ui at client i in round t lies exactly on the line segment joining the local starting gradient gri and the global r: average gradient G ptq r ui “ ψt gri ` p1 ´ ψt q G,

“ r1 ´ α ` αγ p1 ´ σK qs ri ` αp1 ´ γqδi .

(27)

ψ8 “

1´γ . 1 ´ γ p1 ´ σK q

(29)

p8q r ` ψ8 δi “ Note σK ă 1, then ψ8 ą 1 ´ γ. Since ui “ G r ψ8 gri ` p1 ´ ψ8 qG, a larger ψ8 means the reference field is closer to gri than the undamped counterpart 1 ´ γ in (24). The mechanism can be therefore concluded that the compression of the displacement σK ă 1 attenuates ∆i,t , so the consensus underestimate the gap δi and γpc ´ ci q is weaker than in the undamped case, resulting in the proximal damping driving the EMA to a more personalized field. 4) Quantitive Metric Design.: According to above, we define two complementary metrics that measure where a given trajectory lies on the segment between. Let gri,K “ ptq gri ´ ProjGr pr gi q be the component of gri orthogonal to the global consensus direction. This axis captures updates that have zero directional derivative on the global average objective while contributing to client i ’s personalization. We define ptq ptq the drift reduction (DR) as DRi :“ 1 ´ }ri }{}δiptq }, which ptq measures the fraction of the initial heterogeneity gap δi that has been cancelled toward the consensus direction. When ri “ 0, we have DR “ 1, indicating full drift elimination; when ri “ δi , we have DR “ 0, indicating no drift reduction. Similarly, we define the personalization preservation (PPR) as ptq ptq ptq ptq 2 PPRi :“ xui ,g̃i,K y{}g̃i,K } , which measures the projection of the reference field onto the purely local axis. When ui is fully aligned with g̃i , we have PPR “ 1; when ui is fully aligned r we have PPR “ 0. Substituting rptq “ ψt δi and with G, i ptq r into the definitions of DR and PPR ui “ ψt gri ` p1 ´ ψt qG ptq ptq yields DRi “ 1 ´ ψt and PPRi “ ψt .

VI. E XPERIMENTS A. Experimental Setup and Protocols We evaluate CIDERS through three protocols. Protocol A measures downstream performance and output alignment. Protocol B examines matched-domain performance of personalized messengers, and Protocol C tracks PBO consensus-state dynamics under non-IID data. Table I summarizes the datasets, client-training-example totals, partitions, and evaluation units. We also run a separate compression sweep over messenger depths and learnable backbone drop ratios to quantify the quality and storage trade-off. Unless otherwise specified, CIDERS denotes CIDERS-FO, the first-order implementation; CIDERS-FD denotes the finite-difference variant. Model composition. The main configurations in Protocols A and B use a six-layer messenger. For Qwen2.5-3B, the teacher-side segment uses 30 of the 36 backbone layers, while the student learnable backbone uses 24. For Qwen2-1.5B, the corresponding counts are 22 teacher-side layers and 18 learnable backbone layers. Clients train only the messenger LoRA parameters, with rank 4, LoRA scaling αLoRA “ 16,

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

Protocol

Dataset

Samples

Clients

GSM8K Dolly-15K Rosetta

7,458 14,980 7,953

Five-domain

MetaMath

Partition

Evaluation basis

Measurements

Identical independence

GSM8K: 1,319 questions PPL: 6,698 target tokens HumanEval-X: 164/language

Exact-match accuracy PPL and ROUGE-L pass@1

5,000

One domain/client

1,500 test examples; 300/domain

Response-NLL matrix

45,000

MetaMath LDA

16,912 client-round observations

Loss; consensus statistics

Table I: Datasets and evaluation scope for Protocols A–C. Training samples are totals across all clients and include only samples assigned to client training; public-distillation and held-out samples are excluded. Evaluation units are reported in their native form: benchmark questions for GSM8K, target tokens for Dolly PPL, examples for Protocol B, and client-round observations for Protocol C.

and dropout 0.05. Protocol C additionally evaluates two- and four-layer messengers. A six-layer messenger uses 84 trainable LoRA tensors across attention and MLP projections. Serverside TAKD updates the learnable backbone LoRA separately, and AdapEmu combines the personalized messenger with the compressed learnable backbone at deployment. Each consensus state has the same dimensionality as the corresponding messenger LoRA state. Table II lists the reference layer counts and parameter sizes. Backbone Full/Emu./Msg. layers messenger LoRA params Student LoRA params Consensus state (MiB)

Qwen2.5-3B

Qwen2-1.5B

36/24/6 1.247M 4.989M 4.76

28/18/6 0.989M 2.968M 3.77

Table II: Model composition and FP32 LoRA states for the sixlayer messenger configurations. Clients train messenger LoRA, and server-side TAKD trains learnable backbone LoRA.

Data and evaluation. Protocol A uses five IID clients for GSM8K, Dolly, and Rosetta. The loader uses train/validation/test proportions r0.998, 0.001, 0.001s, respectively. The effective training counts in Table I exclude the loader’s heldout portions. Task evaluation instead uses GSM8K benchmark questions, a Dolly prompt–response corpus for corpus-level PPL and ROUGE-L, and HumanEval-X Python, Java, and Go problems. Rosetta supplies code-training data, whereas HumanEval-X supplies the code-evaluation benchmark. For Dolly, 6,698 is the target-token denominator for PPL. Protocol B assigns medical, finance, customer support, code, and general-domain data to five separate clients. Each domain contains 1,000 private training examples, 200 public examples used for distillation, and 300 evaluation examples. The five-domain evaluation uses Qwen2-1.5B with an 18layer learnable backbone, the drop ratio 0.2, and a six-layer messenger. CIDERS-FO, CIDERS-FD, FedOT, and FedBiOT use the same client and evaluation split. Each source-domain messenger is evaluated on the same 300 examples in every target domain, yielding a 5 ˆ 5 response-NLL matrix over 1,500 test examples rather than 7,500 distinct examples.Response negative log-likelihood (NLL) is used to evaluate the quality of each source-domain messenger on a target domain. Lower values indicate better response modeling. The client and evaluation splits are shared across methods, whereas the server-side distillation sources differ. FedBiOT logs use a

separate 4,500-record alignment artifact with a 3,600/450/450 training/validation/test split, while the corresponding FedOT alignment source was not consistently recorded. This mismatch limits direct component-level attribution. Protocol C uses a 50,000-example MetaMath source with logged split proportions r0.9, 0.05, 0.05s: 45,000 training examples and 2,500 examples in each held-out split. Training data are partitioned across eight clients by the MetaMath LDA splitter at Dirichlet parameters α “ 0.1 and α “ 0.4. In the plots, these settings are labeled h “ 1 and h “ 4, respectively. Protocol C is a mechanism study; its reported outcomes are training loss and consensus-state statistics. The temporal and PCA analyses use 14 complete FO/FD training trajectories covering two messenger depths and two learnable backbone drop ratios. Across eight clients and rounds 0–150, these trajectories provide 16,912 client-round observations. The coefficient-response analysis adds one further configuration, giving 15 configurations in total. Training. Protocol A compares CIDERS with FedBiOT [32], FedOT [31], and LocalOT. Federated methods run for 150 rounds, with K “ 10 local optimizer updates per participating client in each round. LocalOT runs for 150 outer iterations without cross-client communication or aggregation. Protocol B uses 150 rounds with K “ 20 local updates per round and batch size 2. CIDERS client updates use AdamW with base learning rate 10´4 , weight decay 0.01, global-norm clipping at 1.0, and a warmup-cosine schedule. The lookahead step size ηin follows the same learning-rate schedule. Gradient accumulation specifies how many microbatches contribute to one optimizer update. The server aggregation step is τ “ 0.02, the client-consensus update coefficient is αc “ 0.002, and the local anchoring weight is ϵprox “ 0.01. The coefficient αc is distinct from the Dirichlet parameter α used in Protocol C. In Protocols A and B, the consensus-correction coefficient is fixed at γt “ 0.01 for Qwen2.5-3B and 0.05 for Qwen2-1.5B; Protocol C uses the schedule described in Section VI-D. Server-side TAKD begins with task-supervised bootstrap alignment. In subsequent rounds, it performs TL “ 5 warm-started representation and output-alignment steps, with λout “ 0.5 in Eq. (2).

B. Task Performance and Output Alignment Protocol A evaluates the complete AdapEmu deployment path. Table III compares CIDERS with FedBiOT, FedOT, and

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

GSM8K Method

Acc. Ò

Dolly PPL Ó

HumanEval-X pass@1 R-L Ò

Qwen2-1.5B / AdapEmu 18-layer learnable backbone + messenger 7.88% 9.53 CIDERS FedBiOT 2.58% 10.52 FedOT 2.05% 11.52 LocalOT 2.20% 11.52 Qwen2.5-3B / AdapEmu 24-layer learnable backbone + messenger 16.38% 8.60 CIDERS FedBiOT 5.23% 9.42 FedOT 4.70% 10.50 LocalOT 4.40% 10.50

Py Ò

Java Ò

Go Ò

0.2176 0.1812 0.1639 0.1612

7.9% 3.0% 1.2% 1.2%

3.0% 3.7% 1.8% 2.4%

4.9% 1.8% 1.2% 1.2%

0.2384 0.2307 0.2234 0.2151

9.1% 6.7% 6.7% 6.7%

11.0% 6.7% 4.3% 3.0%

6.7% 3.0% 1.2% 1.2%

Table III: Protocol A downstream performance of compressed AdapEmu. Code models are trained on Rosetta and evaluated using HumanEval-X pass@1. Bars are normalized within each backbone and metric, with PPL reversed. Numbers give the reported scores. Bold and underlined values mark the best and second-best scores at the displayed precision, including ties.

LocalOT on Qwen2-1.5B and Qwen2.5-3B. These are systemlevel comparisons under the recorded configurations. CIDERS ranks first on five of six Qwen2-1.5B metrics and all six Qwen2.5-3B metrics. Its GSM8K margins over FedBiOT are 5.30 and 11.15 percentage points, respectively. On Qwen2.5-3B, Dolly PPL decreases from 9.42 to 8.60, an approximately 8.7% reduction, and Python, Java, and Go pass@1 improve by 2.4–4.3 percentage points. On Qwen21.5B, Python and Go improve, whereas Java is 0.7 percentage points lower. The largest percentage-point gains are on GSM8K, whereas code-generation gains vary by language. CIDERS therefore shows its clearest advantage on numerical reasoning; the code results are language-dependent. The paired Protocol A results also evaluate AdapFu, the full-backbone deployment setting. For CIDERS-FO, reconnecting the full backbone raises GSM8K accuracy from 16.38% to 47.46% on Qwen2.5-3B and from 7.88% to 29.80% on Qwen2-1.5B. HumanEval-X macro pass@1, averaged across Python, Java, and Go, increases from 8.94% to 39.02% and from 5.28% to 33.54%, respectively. These results show a substantial remaining capability gap between compressed and full-backbone deployment, even when CIDERS improves on the compressed baselines. AdapEmu remains the primary setting for method comparison. The recurring Output-KL term encourages the Student Emulator to match the frozen teacher’s predictive distribution during server-side TAKD. Figure 2 compares CIDERS-FO configurations with and without this term; Table IV gives the CIDERS-FD comparison. The baseline markers show the corresponding Protocol A results for context. Fig. 2 and Table IV show the FO comparison, the configuration with Output-KL has higher GSM8K accuracy by 4.40 percentage points, Java and Go pass@1 by 3.05 and 1.83 percentage points, and ROUGE-L by 0.0076. Dolly PPL changes by only 0.016, while Python pass@1 decreases by 1.22 percentage points. FD shows the same task-selective directions: GSM8K, Java, Go, and ROUGE-L improve, Python declines, and PPL changes little. Thus, adding Output-KL has task-selective effects rather than a uniform capability gain. The opposite changes across code languages make per-language reporting more informative than a single code average. A cautious interpretation is that Output-KL may favor some

(a)

(b) Dolly ROUGE-L

GSM8K

FedBiOT

FedOT

CIDERS

LocalOT

0.22

Accuracy (%)

(c)

Dolly PPL

0.23

0.24

ROUGE-L

(d)

HumanEval-X

Python FedBiOT FedOT

Java

CIDERS LocalOT

ΔPPL = −0.016

Conditioned PPL w/o Output-KL with Output-KL

FedBiOT reference FedOT reference

Go 5

pass@1 (%) LocalOT reference

Figure 2: Output-KL comparison for CIDERS-FO on Qwen2.53B. Segments connect configurations without and with recurring Output-KL. Additional method markers show Protocol A baseline performance.

tasks over others rather than improve all tasks uniformly. This pattern is consistent with Output-KL acting selectively across tasks, but the present results do not identify the mechanism behind the language-specific changes.

C. Finite Difference Diagnostics CIDERS-FD adds a curvature correction estimated by a finite-difference Hessian–vector product (HVP). The correction scale ηHVP is set by fd_eta and need not equal the lookahead step size ηin . Protocol A uses a finite-difference perturbation εFD “ 10´4 and ηHVP “ 0.001; the five-domain

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

With KL

Change

11.98 8.616 0.2271 10.37 7.93 4.27

16.38 8.602 0.2339 9.15 11.59 6.71

`4.40 ´0.014 `0.0068 ´1.22 `3.66 `2.44

GSM8K accuracy Dolly PPL Dolly ROUGE-L Python pass@1 Java pass@1 Go pass@1

Table IV: Output-KL comparison for CIDERS-FD on Qwen2.53B. Accuracy and pass@1 are percentages, with changes reported in percentage points. PPL and ROUGE-L changes use their original scales. All changes are computed from the displayed values.

Batch composition 53

5000 4000 3000

3,441 (68.9%) 4,930 (98.7%)

2000 1000 0

1,401 (28.0%) 154

0.005

Round share (%)

Without KL

Batch count

Metric

Moderate/large rounds 100%

50/50 (100%)

80% 26/50 (52%)

60% 40% 20%

0.0005

0.005

λHVP

Protocol B FD evaluation uses ηHVP “ 0.0005. We measure the relative correction magnitude as

rFD “

x 2 }ηHVP Hv} , }v}2 ` 10´12

where v is the first-order direction used in the optimizer x is its finite-difference HVP estimate. Both are step and Hv restricted to trainable LoRA parameters. An optimizer-step observation is classified as small for rFD ď 0.1, moderate for 0.1 ă rFD ď 0.2, and large for rFD ą 0.2. These thresholds describe relative correction magnitude, not numerical instability. Fig. 3 compares ηHVP “ 0.005 and 0.0005 in a separate 50-round diagnostic. Each setting contains 4,996 recorded optimizer-step observations out of 5,000 nominal observations. The available records do not identify why four nominal observations are missing. At ηHVP “ 0.005, large and moderate corrections account for 68.9% (3,441/4,996) and 28.0% (1,401/4,996) of observations. At 0.0005, these proportions fall to 0.26% (13/4,996) and 1.06% (53/4,996). Moderate or large events occur in 50/50 rounds at the larger scale and 26/50 rounds at the smaller scale. The smaller scale therefore makes such events rare at the observation level, although they still occur in roughly half the rounds. Because ηHVP appears directly in rFD , this comparison reflects both the chosen correction scale and the recorded training trajectories; it should not be read as a standalone curvature estimate. Separately, the final Protocol A comparison finds identical reported FO and FD values in 9 of 12 AdapEmu cells at the displayed precision. Among the accuracy and pass@1 metrics, the largest difference is 0.6 percentage points on Qwen2.53B HumanEval-X Java. The Dolly ROUGE-L difference is 0.0045 on its original scale. Neither solver is uniformly better. We therefore use FO as the default because it avoids finite-difference HVP evaluations while showing no consistent disadvantage in the reported final metrics. Event frequency changes sharply across the two diagnostic scales, whereas the final task metrics show no consistent FO–FD advantage. Within the evaluated settings, event frequency is therefore not predictive of final quality. This supports a simpler solver choice, but it does not establish an end-to-end speedup or explain the 150-round task outcomes from the separate 50round diagnostic.

Small

0.0005

λHVP Moderate

Large

small: rFD ≤ 0.1; moderate: 0.1 < rFD ≤ 0.2; large: rFD > 0.2

Figure 3: Finite-difference correction profiles under two ηHVP settings. Each setting contains 4,996 recorded optimizer-step observations across 50 rounds. The left panel shows the observation composition by correction magnitude; the right panel shows the fraction of rounds containing at least one moderate or large correction event.

D. PBO Dynamics under Data Heterogeneity Protocol C tracks training loss and consensus statistics over time, across scheduled coefficients, and in a shared PCA projection. For client i at round t, let Si,t contain the indices of its recorded local steps. At step s “ pi, t, kq, let gs be the uncorrected local gradient and let cglobal,s and clocal,s be the consensus vectors used in that step. They correspond to the global consensus ct and client consensus cti in the update rule. The corrected gradient is ` ˘ geff,s “ gs ` γtpsq cglobal,s ´ clocal,s . All steps in a round use the coefficient broadcast at its start. The normalized squared-residual statistic is ř 2 kPSi,t }geff,s ´ cglobal,s }2 ` ˘, E-RAEi,t “ ř 2 2 kPSi,t }gs }2 ` }cglobal,s }2 ` ϵ with ϵ “ 10´12 . Each client-round value is a ratio of sums. Zero indicates an exact match to the global consensus at every recorded step. Because this is a normalized ratio, a value near one should not be interpreted as a vanishing residual. We also track S-DR and S-PPR, the distance-reduction and perpendicular-preservation diagnostics. Positive S-DR denotes reduced distance to the global reference, whereas higher SPPR denotes greater preservation of the local orthogonal component. Fig. 4 summarizes loss and E-RAE over rounds 0–150. Client-round values are first aggregated within each configuration at each round; curves and bands then show the crossconfiguration mean and standard deviation. Most of the loss reduction occurs early: by round 15, the curves have achieved 74.0–76.0% of their total round-0-to-150 decrease; by round 30, this fraction reaches 86.8–88.5%. E-RAE approaches its

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

CIDERS-FO

(a)

CIDERS-FD

recorded setting h = 1

recorded setting h = 4

±1 configuration SD

(b)

Training loss

E-RAE reference (=1)

E-RAE

1.1

1.0

E-RAE

Mean training loss

4 2

0.9 0.8

0 0

100

150

Communication round

100

150

Communication round

Figure 4: Temporal PBO dynamics under two non-IID settings (Protocol C). Training loss and E-RAE, the normalized squared-residual statistic, are shown over rounds 0–150 for CIDERS-FO and CIDERS-FD at h “ 1 (α “ 0.1) and h “ 4 (α “ 0.4). Bands denote ˘1 configuration standard deviation. The horizontal line marks E-RAE “ 1 as a diagnostic reference.

(b) S-PPR response 1.0

0.75

0.8

S-PPR

S-DR

0.50 0.25 0.00

0.6 0.4 0.2

−0.25 0.0

0.5

0.0

Scheduled coefficient γt

0.5

Scheduled coefficient γt

(d) E-RAE response

1.00 1.1 0.98

E-RAE

In Fig. 5, the coefficient sequence reconstructed from the recorded telemetry is γ0 “ γ1 “ 0.9 and γt “ maxp0.01, 0.9 ¨ 0.86t´1 q for 2 ď t ď 150. It reaches the floor at round 31, giving 31 distinct values. This describes the recorded trajectory; it is not an independent check of the coefficient schedule used in training. Fig. 5 groups consensus statistics by the scheduled coefficient within each configuration before summarizing the distribution across configurations. Because γt decreases with round and its floor is shared by multiple late rounds, these curves describe the training trajectory rather than an independent coefficient sweep. At small γt , S-DR is close to zero; it becomes more negative at middle and high coefficient values and partially rebounds at the largest recorded coefficient. Negative S-DR means that the corrected direction is farther from the global reference than the uncorrected local direction under this proxy. S-PPR remains high with a shallow U-shaped profile rather than following the 1 ´ γt reference. Thus, preserving the local orthogonal component can coincide with increased distance to the global reference. E-RAE also has a U-shaped profile, with smaller normalized residuals in the middle coefficient range. These patterns do not identify an optimal correction coefficient. Panel (c) traces the configuration-mean S-DR/S-PPR path, with endpoints aggregated from the actual round-0 and round150 observations. FO and FD follow nearby but non-identical

(a) S-DR response

S-PPR

late-stage range of 0.995–0.996 while loss continues to decrease. Thus, consensus alignment can stabilize while task loss continues to improve. E-RAE summarizes normalized alignment between update and consensus vectors, whereas training loss tracks progress on the task objective; the two statistics provide complementary views of adaptation. At round 150, loss at α “ 0.4 is 19.3% higher than at α “ 0.1 for CIDERSFO and 26.9% higher for CIDERS-FD. The loss separation between the recorded LDA settings is larger than the FO–FD separation, despite similar late-stage E-RAE. This difference reflects the two realized data partitions, so it should not be interpreted as a monotonic effect of the Dirichlet parameter on training difficulty.

0.96 0.94

1.0 0.9 0.8

0.92 0.7 −0.1

0.0

S-DR CIDERS-FO CIDERS-FD 10--90% across configurations

0.0

0.5

Scheduled coefficient γt Oracle diagnostic reference Round 0 Round 150

Figure 5: Empirical consensus response to the scheduled coefficient (Protocol C). S-DR, S-PPR, and E-RAE are aggregated within each configuration and then across configurations. Shaded regions show the 10th–90th percentile across 15 configurations. Dashed curves are idealized diagnostic references, not fitted trends.

trajectories. The scheduled coefficient scales the consensus correction; it should not be interpreted as a measured mixture of global and personalized updates. Fig. 6 projects five scalar statistics, i.e., local-consensus norm, consensus-increment norm, E-RAE, S-DR, and S-PPR, into a shared PCA basis. We fit the projection after globally

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

Early

Point stage Middle

Late

Shared PCA of consensus-related statistics

PC2 (23.2% variance)

−2

−4

−6 −17.5 −15.0 −12.5 −10.0 −7.5

−5.0

−2.5

0.0

PC1 (42.4% variance)

Figure 6: Shared PCA of consensus-related statistics (Protocol C). Five scalar statistics are globally standardized before a shared PCA fit over 16,912 client-round observations from 14 configurations, eight clients, and rounds 0–150. Point shading denotes training stage, paths connect client-stage centroids, and Client 7’s late-stage centroid is annotated.

2.1001 NLL, whereas the range of source-row means is only 1.88 ˆ 10´4 . Target-domain differences are therefore much larger than source-messenger differences in these summaries. The small positive gap indicates a limited average matcheddomain benefit, not strong output-level specialization. The CIDERS-FD matrix in the five-domain Protocol B evaluation shows a similar pattern. Mean self NLL is 1.7755 and ∆cross “ 4.66 ˆ 10´4 (0.0263% of self NLL); the target-column and source-row ranges are 2.0945 NLL and 3.56 ˆ 10´4 , respectively. The reported evaluation-sample intervals are r2.50, 3.66sˆ10´4 for FO and r2.56, 6.76sˆ10´4 for FD. Because the resampling procedure is unavailable, we report these intervals descriptively rather than as formal confidence intervals or measures of run-to-run variability. FedOT has mean diagonal and off-diagonal NLLs of 1.8745 and 1.8875, with a reported gap of 0.01298. FedBiOT has corresponding values of 1.8247, 1.8349, and 0.01020. Both baselines have larger matched- domain gaps than CIDERS-FO. A larger gap does not imply lower absolute NLL: response loss and the benefit of source–target matching are different criteria. Here, CIDERS combines a smaller matched-domain effect with lower absolute response NLL, so the two quantities should be reported separately. The shared evaluation makes this relative comparison possible, while the alignment-data mismatch limits component-level attribution. F. Compression and Deployment Tradeoffs

standardizing these statistics with z-scores across the 16,912 client-round observations. Each point is one observation; paths join each client’s early-, middle-, and late-stage centroids. Late-stage centroids generally cluster more tightly in this projection, while Client 7 remains relatively displaced. This indicates a client-specific difference in the recorded scalar statistics, but the projection does not identify its cause or establish separation of the underlying consensus vectors. The PCA view therefore complements the time-series summaries rather than directly measuring personalization in the task output. Thus, consensus-state geometry and task loss should be read together: distinct client states can coexist with continued task improvement. E. Personalized Messenger Evaluation Protocol B asks whether each messenger has lower response NLL on its matched target domain than messengers trained for other source domains. Fig. 7 shows the five-domain matrices, computed from 300 shared examples per target domain. Let Lij be the NLL of source-domain messenger i on target domain j. The average matched-domain advantage is 1 ÿ 1 ÿ5 ∆cross “ Lii . 1ďi,jď5 Lij ´ 20 5 i“1 i‰j Equivalently, each target domain contributes the difference between its four unmatched messengers’ mean NLL and its matched messenger’s NLL, with equal weight across domains. A positive gap favors the matched messenger. For CIDERS-FO, ∆cross “ 3.08 ˆ 10´4 , or 0.0176% of mean self NLL. The range of target-column means is

Table V reports Qwen2-1.5B response NLL and stored FP32 LoRA payload for messenger depths of two, four, and six and learnable backbone drop ratios of 0.2 and 0.5. This is a separate compression sweep, not the Protocol B source– target matrix experiment. Its NLL values should therefore be interpreted separately from the Protocol B matrix. With six messenger layers, increasing the learnable backbone drop ratio from 0.2 to 0.5 reduces the CIDERS LoRA payload by approximately 29.2%, but raises its NLL from 1.754 to 4.115. FedBiOT NLL rises from 1.945 to 5.290 over the same drop-ratio comparison. The more aggressive compression therefore trades lower storage for higher response loss. Increasing messenger depth from two to six reduces CIDERS NLL by 15.1% at drop ratio 0.2 and 36.9% at drop ratio 0.5, with payload increases of approximately 4.3% and 13.3%, respectively. The gain is not uniform across depth increments or methods. At drop ratio 0.2, CIDERS changes only from 2.0657 to 2.0600 between two and four layers, whereas FedBiOT NLL increases from 2.3576 to 2.4090. The larger CIDERS gain in this setting occurs from four to six layers. Within the tested CIDERS grid, the six-layer messenger gives the best NLL, but this does not imply that every method improves monotonically with depth. CIDERS has the lowest NLL in all six matched grid configurations, with NLL 9.9– 22.2% below FedBiOT. Its minimum, 1.754, occurs at six messenger layers and drop ratio 0.2. This is the best evaluated CIDERS configuration by NLL, not an optimum over untested depths, drop ratios, or deployment budgets. Payload measures stored LoRA state only; it does not measure total model storage or runtime memory.

CIDERS

FedOT

FedBiOT

Medical

1.0985 3.0598 1.3446 0.9597 2.3063

1.4992 3.0036 1.5236 1.1103 2.2546

1.4076 2.9995 1.4837 1.0427 2.2148

Finance

1.0985 3.0595 1.3446 0.9596 2.3061

1.5420 2.9961 1.5306 1.1095 2.2514

1.4255 2.9920 1.4861 1.0406 2.2112

Cust. support

1.0986 3.0600 1.3441 0.9599 2.3067

1.5266 3.0077 1.5099 1.1132 2.2631

1.4246 3.0021 1.4710 1.0442 2.2230

Code

1.0986 3.0597 1.3447 0.9594 2.3063

1.5313 3.0055 1.5291 1.1152 2.2617

1.4233 2.9998 1.4861 1.0394 2.2211

General

1.0986 3.0598 1.3447 0.9597 2.3062

1.5345 3.0033 1.5320 1.1160 2.2522

1.4277 3.0007 1.4913 1.0495 2.2133

3.0

2.5

2.0

1.5

rt al ce de ral dic inan uppo ne Co F Me s Ge . t s Cu

Target domain

1.0

rt al ce de ral dic inan uppo ne Co F Me s Ge . t s Cu

Target domain

Response NLL

Source domain

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

Target domain

Figure 7: Cross-domain evaluation of personalized messengers (Protocol B). CIDERS-FO, FedOT, and FedBiOT are evaluated on the same five-domain Qwen2-1.5B set, with 300 target-domain examples per cell. Rows identify source messengers, columns identify target domains, and outlines mark matched cells. The shared color scale denotes response NLL; cell labels show four decimal places. Drop ratio β

CIDERS-FO

Depth d

FedBiOT

FedOT

NLL Ó

MiB

NLL Ó

MiB

NLL Ó

MiB

0.2

2 4 6

2.0657 2.0600 1.7537

14.570 14.570 15.203

2.3576 2.4090 1.9454

14.568 14.568 15.201

2.2375 2.2671 1.8845

14.568 14.568 15.201

0.5

2 4 6

6.5222 5.1666 4.1151

9.502 10.136 10.769

7.4351 6.2889 5.2901

9.501 10.134 10.768

7.0028 6.3932 5.6596

9.501 10.134 10.768

Table V: AdapEmu quality and storage trade-off on Qwen2-1.5B (separate compression sweep). β denotes learnable backbone drop ratio and d denotes messenger depth. NLL is the five-domain response NLL; MiB denotes stored FP32 LoRA deployment payload, not total model storage or runtime memory. Boldface marks the lowest NLL within each row.

Table VI reports a separate quantity: recurrent FP32 messenger/consensus-state communication. Each CIDERS downlink contains a Meta-messenger and a same-sized global consensus; each uplink contains a Personalized messenger and a same-sized consensus increment. Learnable backbone downlink and protocol overhead are excluded. For per-direction payload p in MiB, the totals are 150 ˆ 5 ˆ 2 ˆ p{1024 GiB, assuming participation by all five clients in every round. Backbone

Method

Qwen2.5-3B CIDERS Qwen2.5-3B FedBiOT/FedOT CIDERS Qwen2-1.5B Qwen2-1.5B FedBiOT/FedOT Both LocalOT

Per direction (MiB) Total (GiB) 9.516 4.758 7.547 3.773 n/a

13.94 6.97 11.06 5.53 n/a

Table VI: FP32 messenger/consensus-state communication. Perdirection values are per client per round. Totals assume all five clients participate in all 150 rounds and include both directions. Learnable backbone downlink and protocol overhead are excluded.

CIDERS therefore communicates approximately twice the counted messenger/consensus tensor volume of the onemessenger baselines. This ratio concerns the stated components, not total network traffic. Taken together, the results define a tunable trade-off among quality, storage, and com-

munication: using a lower drop ratio and a deeper messenger improves response NLL, whereas the consensus state doubles the counted recurrent payload. This is a trade-off, not an unconditional efficiency advantage. G. Integrated Discussion and Limitations Protocol A evaluates the complete compressed CIDERS deployment path under the recorded configurations. CIDERS leads on most reported metrics, but the gains vary by task and backbone. Output-KL comparisons remain task-selective, and FD has no consistent final-metric advantage over FO. These results characterize the tested systems; they do not isolate the contribution of any single component. The large differences between AdapEmu and AdapFu further show that better compressed performance does not remove the capability cost of the smaller learnable backbone. Protocols B and C measure different aspects of personalization. Protocol B measures matched-domain response quality, whereas Protocol C measures consensus-vector geometry. These signals are complementary, but they are not interchangeable. Because the protocols use different datasets and clients, they cannot establish a direct link between directional preservation and response quality. The separate compression sweep likewise characterizes only the tested quality and storage trade-offs. Accordingly, the Output-KL, FO–FD, and cross-domain analyses

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

are descriptive rather than causal. The resampling procedure for the reported cross-domain intervals is unavailable, so these intervals are not interpreted as formal confidence intervals. In Protocol C, the bands show variation across configurations, not across independent training runs. The client partitions can be reconstructed from the available source code, logs, seed, and splitter, but no per-client manifests were saved with the checkpoints; the conclusions therefore remain limited to the two recorded LDA partitions. Runtime, peak memory, full network traffic, and provisioning costs were not measured on common hardware, so these component totals do not establish end-to-end system efficiency. VII. C ONCLUSION This paper addresses the fundamental tension between global consensus and client personalization in cloud-edge LLM systems. We formalize the problem as a personalized bilevel optimization framework for cloud-edge LLM collaboration. Then, we further propose CIDERS, an efficient solver that decouples the model into a cloud-learnable backbone and a client-adaptable messenger. By embedding global trajectories into each local step via consensus-variate correction, CIDERS reconciles personalization with globalization. Theoretical analysis provides a geometric characterization of the local trajectory and a full convergence guarantee. Extensive experiments on Qwen2.5-3B/1.5B across the three complementary protocols validate our approach: i) the local trajectories follows theories, ii) bilevel coordination preserves reasoning capability in compressed deployment, iii) backbone capacity matters more than messenger depth for response quality. These results establish CIDERS as a practically viable path toward the applicatoins of complex real-world intelligence. Future work will extend CIDERS to world models for embodied intelligence and physical-world simulation. R EFERENCES [1] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations, 2023. [2] T. Schick, J. Dwivedi-Yu, R. Dess‘i, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” in Advances in Neural Information Processing Systems, 2023. [3] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich, “RT-2: Vision-language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818, 2023. [4] D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” in Advances in Neural Information Processing Systems, vol. 31, 2018. [5] X. Wang, Y. Han, V. C. M. Leung, D. Niyato, X. Yan, and X. Chen, “Convergence of edge computing and deep learning: A comprehensive survey,” IEEE Communications Surveys & Tutorials, vol. 22, no. 2, pp. 869–904, 2020. [6] H. Chen, W. Deng, S. Yang, J. Xu, Z. Jiang, E. C. H. Ngai, J. Liu, and X. Liu, “Towards edge general intelligence via large language models: Opportunities and challenges,” arXiv preprint arXiv:2410.18125, 2024.

[7] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” in Proceedings of the 36th International Conference on Machine Learning, 2019, pp. 2790–2799. [8] X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021, pp. 4582– 4597. [9] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022. [10] S. Ye, B. Ouyang, L. Zeng, T. Qian, X. Chu, J. Tang, and X. Chen, “Jupiter: Fast and resource-efficient collaborative inference of generative LLMs on edge devices,” in IEEE INFOCOM 2025 - IEEE Conference on Computer Communications. IEEE, 2025, pp. 1–10. [11] Y. Hu, C. Imes, X. Zhao, S. Kundu, P. A. Beerel, S. P. Crago, and J. P. Walters, “PipeEdge: Pipeline parallelism for large-scale model inference on heterogeneous edge devices,” in 2022 25th Euromicro Conference on Digital System Design (DSD). IEEE, 2022, pp. 298–307. [12] M. Zhang, X. Shen, J. Cao, Z. Cui, and S. Jiang, “EdgeShard: Efficient LLM inference via collaborative edge computing,” IEEE Internet of Things Journal, vol. 12, no. 10, pp. 13 119–13 131, 2025. [13] S. Ye, J. Du, L. Zeng, W. Ou, X. Chu, Y. Lu, and X. Chen, “Galaxy: A resource-efficient collaborative edge AI system for in-situ transformer inference,” in IEEE INFOCOM 2024 - IEEE Conference on Computer Communications. IEEE, 2024, pp. 1001–1010. [14] A. Borzunov, D. Baranchuk, T. Dettmers, M. Riabinin, Y. Belkada, A. Chumachenko, P. Samygin, and C. Raffel, “Petals: Collaborative inference and fine-tuning of large models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 2023, pp. 558–568. [15] A. Mudvari, Y. Jiang, and L. Tassiulas, “SplitLLM: Collaborative inference of LLMs for model placement and throughput optimization,” arXiv preprint arXiv:2410.10759, 2024. [16] T. Berenbaum and M. Venkatachalam, “Pre-compiled pipeline shards for distributed LLM inference on intel AI PC fleets,” arXiv preprint arXiv:2608.19147, 2026. [17] D. Macario, H. Seferoglu, and E. Koyuncu, “Model-distributed inference for large language models at the edge,” arXiv preprint arXiv:2505.18164, 2025. [18] J. Park, S. Cho, and D. Han, “SpecEdge: Scalable edge-assisted serving framework for interactive LLMs,” in Advances in Neural Information Processing Systems, 2025. [19] Y. Han, Y. Gao, B. Hu, M. B. Mashhadi, Y. Duan, P. Xiao, and Y. Zhang, “PipeSD: An efficient cloud-edge collaborative pipeline inference framework with speculative decoding,” in Proceedings of the 43rd International Conference on Machine Learning, 2026. [20] Y. Zhang, Z. Gao, S. Yue, J. Li, and R. Wang, “PicoSpec: A pipelined collaborative speculative decoding framework for efficient edge-cloud LLM inference,” arXiv preprint arXiv:2603.19133, 2026. [21] Z. Wang, Z. Shen, Y. He, G. Sun, H. Wang, L. Lyu, and A. Li, “FLoRA: Federated fine-tuning large language models with heterogeneous lowrank adaptations,” in Advances in Neural Information Processing Systems, 2024. [22] J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, G. Wang, and Y. Chen, “Towards building the federated GPT: Federated instruction tuning,” in ICASSP 2024 - IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2024. [23] Y. J. Cho, L. Liu, Z. Xu, A. Fahrezi, and G. Joshi, “Heterogeneous LoRA for federated fine-tuning of on-device foundation models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 12 903–12 913. [24] P. Guo, S. Zeng, Y. Wang, H. Fan, F. Wang, and L. Qu, “Selective aggregation for low-rank adaptation in federated learning,” in International Conference on Learning Representations, 2025. [25] Y. Sun, Z. Li, Y. Li, and B. Ding, “Improving LoRA in privacypreserving federated learning,” in International Conference on Learning Representations, 2024. [26] Z. Lin, X. Hu, Y. Zhang, Z. Chen, Z. Fang, X. Chen, A. Li, P. Vepakomma, and Y. Gao, “SplitLoRA: A split parameter-efficient fine-tuning framework for large language models,” arXiv preprint arXiv:2407.00952, 2024. [27] T. Li, Y. Tang, Y. Song, C. Wu, X. Liu, P. Li, and X. Chen, “SplitCom: Communication-efficient split federated fine-tuning of LLMs via temporal compression,” arXiv preprint arXiv:2602.10564, 2026.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

[28] C. Gao and S. Q. Zhang, “DLoRA: Distributed parameter-efficient finetuning solution for large language model,” in Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 13 703–13 714. [29] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al., “Advances and open problems in federated learning,” Foundations and Trends in Machine Learning, vol. 14, no. 1–2, pp. 1–210, 2021. [30] A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “Towards personalized federated learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 12, pp. 9587–9603, 2023. [31] G. Xiao, J. Lin, and S. Han, “Offsite-tuning: Transfer learning without full model,” in Proceedings of the 40th International Conference on Machine Learning, 2023. [32] F. Wu, Z. Li, Y. Li, B. Ding, and J. Gao, “FedBiOT: LLM local finetuning in federated learning without full model,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024. [33] S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh, “SCAFFOLD: Stochastic controlled averaging for federated learning,” in Proceedings of the 37th International Conference on Machine Learning, 2020, pp. 5132–5143. [34] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Agüera y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017, pp. 1273–1282. [35] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in Proceedings of Machine Learning and Systems, vol. 2, 2020, pp. 429–450. [36] D. A. E. Acar, Y. Zhao, R. M. Navarro, M. Mattina, P. N. Whatmough, and V. Saligrama, “Federated learning based on dynamic regularization,” in International Conference on Learning Representations, 2021. [37] J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 7611–7623. [38] Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 713–10 722. [39] M. G. Arivazhagan, V. Aggarwal, A. K. Singh, and S. Choudhary, “Federated learning with personalization layers,” arXiv preprint arXiv:1912.00818, 2019. [40] L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai, “Exploiting shared representations for personalized federated learning,” in Proceedings of the 38th International Conference on Machine Learning, 2021, pp. 2089–2099. [41] C. T. Dinh, N. H. Tran, and T. D. Nguyen, “Personalized federated learning with moreau envelopes,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 21 394–21 405. [42] T. Li, S. Hu, A. Beirami, and V. Smith, “Ditto: Fair and robust federated learning through personalization,” in Proceedings of the 38th International Conference on Machine Learning, 2021, pp. 6357–6368. [43] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in Proceedings of the 34th International Conference on Machine Learning, 2017, pp. 1126–1135. [44] A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 3557–3568. [45] Y. Wu, C. Tian, J. Li, H. Sun, K. Tam, L. Li, and C. Xu, “A survey on federated fine-tuning of large language models,” arXiv preprint arXiv:2503.12016, 2025. [46] J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, Y. Zhou, G. Wang, and Y. Chen, “Towards building the federated GPT: Federated instruction tuning,” arXiv preprint arXiv:2305.05644, 2023. [47] J. Bian, Y. Peng, L. Wang, Y. Huang, and J. Xu, “A survey on parameterefficient fine-tuning for foundation models in federated learning,” arXiv preprint arXiv:2504.21099, 2025. [48] W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y. Xie, Y. Li, B. Ding, and J. Zhou, “FederatedScope-LLM: A comprehensive package for fine-tuning large language models in federated learning,” arXiv preprint arXiv:2309.00363, 2023. [49] P. Guo, S. Zeng, Y. Wang, H. Fan, F. Wang, and L. Qu, “Selective aggregation for low-rank adaptation in federated learning,” arXiv preprint arXiv:2410.01463, 2024. [50] J. Bian, L. Wang, L. Zhang, and J. Xu, “FedALT: Federated fine-tuning through adaptive local training with rest-of-the-world LoRA,” arXiv preprint arXiv:2503.11880, 2025.

[51] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015. [52] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter,” in NeurIPS Workshop on Energy Efficient Machine Learning and Cognitive Computing, 2019. [53] X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “TinyBERT: Distilling BERT for natural language understanding,” in Findings of the Association for Computational Linguistics: EMNLP, 2020, pp. 4163–4174. [54] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “MiniLM: Deep self-attention distillation for task-agnostic compression of pretrained transformers,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 5776–5788.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

A PPENDIX D ETAILED D ERIVATIONS OF T HEORETICAL A NALYSIS

We provide the complete algebraic derivations for the geometric characterization of the local messenger trajectory presented in Section V-B. A. Derivation of Approximation A1 ptq

ptq

Recall (A1) the total inner-loop displacement is sufficiently small, i.e., Kη ! 1{L, so that ḡi « g̃i . The ptq ptq ptq path error ϕi :“ ḡi ´ g̃i is bounded by O pLKηGbd q. We now derive this bound. Let the meta-gradient map gpxq :“ ∇Li pxq be L-Lipschitz continuous }gpxq ´ gpyq} ď L}x ´ y}, @x, y . This is a standard smoothness assumption on the local meta-loss. Let Gbd :“ supx }gpxq} be a uniform bound on the gradient norm. From the inner update (1) in the main text, we have: xk`1 ´ v̄ “ ρ pxk ´ v̄q ´ ηgi,k pxk q ´ ηγ pc ´ ci q ,

ρ :“ 1 ´ ηε.

(30)

For ε ě 0, we have ρ P p0, 1s. Taking norms and applying the triangle inequality: }xk`1 ´ v̄} ď ρ }xk ´ v̄} ` η }gi,k pxk q} ` ηγ }c ´ ci }

(31)

ď ρ }xk ´ v̄} ` ηGbd ` ηγCc “ ρ }xk ´ v̄} ` ηG1bd . ptq

where Cc :“ supt }cptq ´ ci } is finite under the bounded gradient assumption, we further unrolling to derive: ÿK´1 }xK ´ v̄} ď ηG1bd ρK´1´j “ ηG1bd SK ď KηG1bd . (32) j“0

ptq ptq ptq Now we bound the path error ϕi :“ ḡi ´ g̃i . Recall: ptq

ḡi “

1 ÿK´1 K´1´j ρ gi,j pxj q , SK j“0 ptq

and we can derive the path error ϕi ptq

ptq

}ḡi ´ g̃i } “ }

ptq

g̃i “ gi,0 pv̄q,

(33)

as follows

K´1 K´1 1 ÿ K´1´j 1 ÿ K´1´j ρ pgi,j pxj q ´ gi,0 pv̄qq} ď ρ }gi,j pxj q ´ gi,0 pv̄q}. SK j“0 SK j“0

(34)

Using the Lipschitz property, it results in }gi,j pxj q ´ gi,0 pv̄q} ď L}xj ´ v̄} ď KηG1bd , which we substitute ptq into (34) and it leads to }ϕi } ď LCK ηGbd . B. Derivation of Approximation A2 pt`1q

ptq

A2 posits that the heterogeneity gap remains constant approximately across rounds, i.e., δi « δi « δi . While this holds exactly for quadratic meta-losses, for general nonconvex losses it requires justification. Here, we demonstrate this locally valid via a second-order Taylor expansion of the loss function around the current broadcast point, and show that the approximation error decays naturally as the global model converges. Assume that each local loss function Li has an M -Lipschitz continuous Hessian, i.e., › 2 › ›∇ Li pw1 q ´ ∇2 Li pw2 q› ď M }w1 ´ w2 } , @w1 , w2 . (35) Expanding the gradient ∇Li around the broadcast point v̄ ptq and evaluating at an arbitrary nearby point w, we obtain: ptq ptq ∇Li pwq “ ∇Li pv̄ ptq q ` Hi pw ´ v̄ ptq q ` Ri pwq, (36) ptq

where Hi

:“ ∇2 Li pv̄ ptq q is the local Hessian at the broadcast point, and the remainder term satisfies ptq

}Ri pwq} ď

M }w ´ v̄ ptq }2 . 2

(37)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

Similarly, for the global average gradient ∇L :“

j pj ∇Lj , we have: ptq

∇Lpwq “ ∇Lpv̄ ptq q ` H̄ ptq pw ´ v̄ ptq q ` R pwq,

where H̄ ptq :“

(38)

ptq ptq j pj Hj is the average Hessian, and the global remainder R pwq satisfies ptq

}R pwq} ď

M }w ´ v̄ ptq }2 . 2

(39)

Our objective is to characterize the change in δi between rounds t and t ` 1. Let v̄ pt`1q “ v̄ ptq ` ∆v̄ ptq , where ∆v̄ ptq denotes the global messenger update from round t to t ` 1. Applying the expansions (36) and (38) at w “ v̄ pt`1q , we obtain: pt`1q

δi

“∇Li pv̄ pt`1q q ´ ∇Lpv̄ pt`1q q ptq

ptq

“r∇Li pv̄ ptq q ` Hi ∆v̄ ptq ` Ri pv̄ pt`1q qs

(40)

ptq

´ r∇Lpv̄ ptq q ` H̄ ptq ∆v̄ ptq ` R pv̄ pt`1q qs ptq

ptq

“δi ` pHi ´ H̄ ptq q∆v̄ ptq ` pRi pv̄ pt`1q q ´ R pv̄ pt`1q qq.

Therefore, the cross-round variation of the heterogeneity gap is: pt`1q

δi

ptq

´ δi “ pHi ´ H̄ ptq q∆v̄ ptq ` Op}∆v̄ ptq }2 q.

(41)

To justify treating δi as constant across rounds, we must show that the right-hand side of (B.18) is negligible at the scale of our analysis. First-order term (Hessian mismatch contribution): Define the worst-case Hessian heterogeneity as ptq

χptq :“ max }Hi ´ H̄ ptq }.

(42)

Then the first-order term is bounded by: ptq

}pHi ´ H̄ ptq q∆v̄ ptq } ď χptq ¨ }∆v̄ ptq }.

(43) pt`1q

Second-order remainder can be simply bounded by Op}∆v̄ ptq }2 q. Therefore, we have }δi Op}∆v̄ ptq }2 q Ñ 0 as t Ñ 8.

ptq

´ δi } ď

C. Residual Recurrence

We first derive the consensus-gap recursion, recall the consensus updates are: ÿ pt`1q ptq cpt`1q “ p1 ´ αqcptq ` α pj ∆j,t , and ci “ r1 ` αpγ ´ 1qsci ` α∆i,t ´ αγcptq . jPSt

(44)

Under (A5) that St “ rN s, we can derive according to (44) pt`1q

cpt`1q ´ ci

ptq

ptq “p1 ´ αqcptq ` α∆ptq avg ´ r1 ` αpγ ´ 1qsci ´ α∆i,t ` αγc ptq

“p1 ´ α ` αγqcptq ´ p1 ´ α ` αγqci ` αp∆ptq avg ´ ∆i,t q

(45)

ptq

“r1 ´ αp1 ´ γqspcptq ´ ci q ` αp∆ptq avg ´ ∆i,t q, ptq

where ∆avg :“

j pj ∆j,t . Thus, the consensus-gap recursion can be derived pt`1q

cpt`1q ´ ci

ptq

“ r1 ´ αp1 ´ γqspcptq ´ ci q ` αp∆ptq avg ´ ∆i,t q. ř ptq Next, we take average over ci in (44) and recall the definition cptq :“ j pj cj , we have

(46)

ptq

(47)

ptq cpt`1q “ r1 ´ αp1 ´ γqscptq ` α∆ptq “ p1 ´ αqcptq ` α∆avg . avg ´ αγc

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

Subsequently, comparing to c update in (44), we have cptq “ cptq if it initializes cp0q “ cp0q . Next, we consider ptq ptq ptq ∆avg . From ∆i,t “ g̃i ` γpcptq ´ ci q, the average displacement is ÿ ptq ptq r ptq . ∆ptq pj pg̃j ` γpcptq ´ cj qq “ G (48) avg “ j

Therefore, we obtain the following ptq ptq ptq ptq ptq r ptq ´ pr ∆avg ´ ∆i,t “ G gi ` γpcptq ´ ci qq “ ´δi ´ γpcptq ´ ci q. ptq

ptq

(49)

ptq

Recall the reference field ui :“ g̃i ` γpcptq ´ ci q and the residual relative to the global consensus direction ptq ptq r ptq “ δ ptq ` γpcptq ´ cptq q, which also lead to γpcptq ´ cptq q “ rptq ´ δ ptq . Under (A2) satisfies ri :“ ui ´ G i i i i i ptq that δi “ δi , we Substituting (A.16) into (A.14): ´ ¯ ´ ¯ pt`1q ptq ptq cpt`1q ´ ci “ r1 ´ αp1 ´ γqs cptq ´ ci ´ αδi ´ αγ cptq ´ ci ´ ¯ ´ ¯ (50) ptq ptq ptq ptq “ r1 ´ αp1 ´ γq ´ αγs c ´ ci ´ αδi “ p1 ´ αq c ´ ci ´ αδi . pt`1q

Now compute ri

ptq

with the result that γpcptq ´ ci q “ ri ´ δi : ´ ¯ pt`1q pt`1q ri “ δi ` γ cpt`1q ´ ci ” ´ ¯ ı ptq “ δi ` γ p1 ´ αq cptq ´ ci ´ αδi ´ ¯ ptq “ γp1 ´ αq cptq ´ ci ` p1 ´ αγqδi ´ ¯ ptq “ p1 ´ αq ri ´ δi ` p1 ´ αγqδi

(51)

ptq

“ p1 ´ αqri ` αp1 ´ γqδi . p0q

Since the initial condition follows from (A4) that cp0q “ ci pt`1q ptq p0q Thus ri “ p1 ´ αqri ` αp1 ´ γqδi , ri “ δi .

p0q

“ 0, ri

p0q

“ δi

p0q

` γp0 ´ 0q “ δi

“ δi holds.

D. Derivation of Proposition 2 pt`1q

We first solve the recurrence for ψt . Since the recurrence (51) expresses ri ptq p0q of ri and δi , and ri “ δi , it follows by induction that ptq

ri P span tδi u

as an R-linear combination

@t ě 0.

(52)

ptq

Thus we can write ri “ ψt δi with a scalar ψt . Substituting into (51) leads to ψt`1 δi “ p1 ´ αqψt δi ` αp1 ´ γqδi .

(53)

Since δi ‰ 0, we can cancel it which results in ψt`1 “ p1 ´ αqψt ` αp1 ´ γq,

ψ0 “ 1.

(54)

We solve this nonhomogeneous first-order linear recurrence and the constant steady state ψ ˚ ψt “ p1 ´ γq ` γp1 ´ αqt with ψ ˚ “ 1 ´ γ.

(55)

Substituting (55) into the definition of the reference field we have ptq r ptq ` rptq “ ψt gri ` p1 ´ ψt q G. r ui “ G i

This completes the derivation of the proposition 2.

(56)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

E. Path-Averaging Error and Its Effect on the Residual

We now analyze the impact of relaxing the short-inner-loop assumption (A1). In the implemented local ptq ptq update, the along-path average ḡi may differ from the starting oracle gri , thus we introduce the path error ptq ptq ptq ptq ϕi :“ ḡi ´ gri . As established in Appendix B.1, this error is uniformly bounded, i.e., }ϕi } ď LCK ηGbd . We quantify how this path error propagates into the residual dynamics and the reference direction. We retain ptq (A2)–(A5) and thus σK “ 1, ε “ 0 and ϕi ‰ 0, then ` ` ptq ptq ˘ ptq ptq ptq ˘ ∆i,t “ ḡi ` γ cptq ´ ci “ gri ` ϕi ` γ cptq ´ ci , (57) ptq r ptq ` ϕ̄ptq , where ϕ̄ptq :“ ř pj ϕptq . Subsequently, we evaluate and we average over clients, it yields ∆avg “ G j j the displacement gap as follows ` ptq ˘ ` ˘ ` ˘ ` ˘ r ` ϕ̄ptq ´ grptq ` ϕptq ` γ cptq ´ c̄ptq ´ γ cptq ´ cptq ∆ptq avg ´ ∆i,t “ G i i i (58) ` ptq ptq ptq ˘ “ ´δi ` ϕ̄ptq ´ ϕi ´ γ cptq ´ ci ptq

ptq

r ptq ´ gr “ ´δ where G i i

and c̄ptq “ cptq . Substituting into the consensus-gap recursion in (46) gives “ ‰` “ ptq ` pt`1q ptq ˘ ptq ptq ˘‰ cpt`1q ´ ci “ 1 ´ αp1 ´ γq cptq ´ ci ` α ´δi ` ϕ̄ptq ´ ϕi ´ γ cptq ´ ci “ ‰` ` ptq ptq ˘ ptq ˘ (59) “ 1 ´ αp1 ´ γq ´ αγ cptq ´ ci ` α ´δi ` ϕ̄ptq ´ ϕi ` ptq ˘ ` ˘ ptq ptq ptq “ p1 ´ αq c ´ ci ` α ´δi ` ϕ̄ptq ´ ϕi . ` pt`1q pt`1q pt`1q ˘ Now let consider the residual, which satisfies ri “ δi ` γ cpt`1q ´ ci . Under (A2), we have the pt`1q ptq approximation δi “ δi “ δi , and therefore it yields ” ı ` ` pt`1q ptq ˘ ptq ˘ ri “ δi ` γ p1 ´ αq cptq ´ ci ` α ´δi ` ϕ̄ptq ´ ϕi ` ` ptq ˘ ptq ˘ (60) “ γp1 ´ αq cptq ´ ci ` p1 ´ αγqδi ` αγ ϕ̄ptq ´ ϕi ` ˘ ptq ptq “ p1 ´ αqri ` αp1 ´ γqδi ` αγ ϕ̄ptq ´ ϕi , ` ptq ˘ ptq where we have used the identity γ cptq ´ ci “ ri ´ δi . Now, let us turn to the ideal recurrence when ptq ϕi “ ϕ̄ptq “ 0 pt`1q ˇˇ ptq ˇ p0q ˇ r “ p1 ´ αq r ˇ ` αp1 ´ γqδi , r ˇ “ δi . (61) i

pE0q

ptq ptq ptq ˇ Subsequently, we define the path-induced deviation r̃ :“ r ´ r ˇ i

pE0q

, which can directly produce the

following linear system pt`1q

r̃i

` ptq ptq ˘ p0q “ p1 ´ αqr̃i ` αγ ϕ̄ptq ´ ϕi , with r̃i “ 0.

Unrolling (62) from s “ 0 to t ´ 1, we can obtain the following ÿt´1 ` ptq psq ˘ r̃i “ αγ p1 ´ αqt´1´s ϕ̄psq ´ ϕi . s“0

(62)

(63)

Taking norms and applying the triangle inequality, ptq

}r̃i } ď 2αγR

ÿt´1 s“0

p1 ´ αqt´1´s “ 2αγR ¨

` ˘ 1 ´ p1 ´ αqt “ 2γR 1 ´ p1 ´ αqt , α

(64)

psq

where we have used }ϕj } ď R for every client and ϕ̄psq is a convex combination of ϕj , this leads to psq

}ϕ̄psq ´ ϕi } ď 2R. Therefore ptq

}r̃i } ď 2αγR

ÿt´1 s“0

p1 ´ αqt´1´s “ 2αγR ¨

` ˘ 1 ´ p1 ´ αqt “ 2γR 1 ´ p1 ´ αqt . α

(65)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

ˇ ptq ˇˇ ptq ptq r r and rptq ˇ Finally, ri “ ui ´ G i pE0q “ ui pE0q ´ G, so the same bound holds for the reference direction: ˇ › ptq ›u ´ uptq ˇ i

pE0q

› ` ˘ › ď 2γR 1 ´ p1 ´ αqt ď 2γR.

(66)

This proves the uniform bound and the deviation vanishes as Kη Ñ 0, which shows that a small local update step makes the convex-combination characterization hold with high accuracy. F. Hessian remainder along the local path.

We now consider the effect of evaluating the meta-gradient at an intermediate point x along the local r and trajectory, rather than at the broadcast messenger v̄ . Even when the consensus variates are locked, c “ G ci “ gri , the executed field at x is ` ˘ r ´ gri . ∇Li pxq ` γpc ´ ci q “ ∇Li pxq ` γ G (67) We compare this with the designed convex combination p1 ´ γq∇Li pxq ` γ∇L̄pxq. Subtracting yields ` ˘ “ ‰ ` ˘ r ´ gri ´ p1 ´ γq∇Li pxq ` γ∇L̄pxq “ γ∇Li pxq ` γ G r ´ gri ´ γ∇L̄pxq ∇Li pxq ` γ G (68) “ ‰ “` ˘ ` ˘‰ r “ γ ∇Li pxq ´ gri ´ ∇L̄pxq ´ G r . gi ´ Gq “ γ ∇Li pxq ´ ∇L̄pxq ´ pr Now we compute the Taylor expansion of ∇Li pxq and ∇L̄pxq respectively at v̄ ` ˘ ` ˘ r ` H̄px ´ v̄q ` O }x ´ v̄}2 , ∇Li pxq “ gri ` Hi px ´ v̄q ` O }x ´ v̄}2 , and ∇L̄pxq “ G ř where Hi :“ ∇2 Li pv̄q and H̄ :“ j pj Hj . Substituting these expansions produces ` ˘ r ´ gri “ p1 ´ γq∇Li pxq ` γ∇L̄pxq ∇Li pxq ` γ G “ ` ˘‰ ` γ pHi ´ H̄qpx ´ v̄q ` O }x ´ v̄}2 .

(69)

(70)

Thus the first-order remainder is exactly γpHi ´ H̄qpx ´ v̄q. Recall χ :“ maxi }Hi ´ H̄} and the remainder is bounded by γχ}x ´ v̄} ` Opγ}x ´ v̄}2 q. Therefore, the curvature mismatch introduces an error that scales with γ and with the local displacement }x ´ v̄}. G. The case σK ‰ 1: detailed derivation of the renormalized recurrence

We now relax assumption (A3) and allow a proximal coefficient ε ą 0, which implies ρ “ 1 ´ ηε ă 1, and consequently ÿK´1 1 ´ ρK SK “ ρK´1´j “ ă K, (71) j“0 1´ρ so that σK :“ SK{K P p0, 1q. The displacement now carries the compression factor σK : ´ ¯ ´ ¯ ` ` ptq ptq ˘ ptq ˘ ∆i,t “ σK ḡi ` γ cptq ´ ci “ σK gri ` γ cptq ´ ci ,

(72) ptq

where under the small local update approximation (A1) and fixed heterogeneity (A2), we have used ḡi “ ptq ptq r , this leads to gri “ gri . The average displacement can be simply derived ∆avg “ σK G ´ ´ ` ˘¯ ` ˘¯ r ´ gri ´ γ cptq ´ cptq “ σK ´δi ´ γ cptq ´ cptq , ∆ptq ´ ∆ “ σ G (73) i,t K avg i i where we substitute into the consensus gap recursion in (46) as follows ´ ¯ “ ‰` ` pt`1q ptq ˘ ptq ˘ cpt`1q ´ ci “ 1 ´ αp1 ´ γq cptq ´ ci ` ασK ´δi ´ γ cptq ´ ci “ ‰` ptq ˘ “ 1 ´ αp1 ´ γq ´ αγσK cptq ´ ci ´ ασK δi .

(74)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

` pt`1q pt`1q ˘ Now we substitute (74) into ri “ δi ` γ cpt`1q ´ ci and subsequently we have ”` ı ˘` pt`1q ptq ˘ ri “ δi ` γ 1 ´ αp1 ´ γq ´ αγσK cptq ´ ci ´ ασK δi ` ˘` ptq ˘ “ γ 1 ´ αp1 ´ γq ´ αγσK cptq ´ ci ` p1 ´ αγσK qδi ı (75) ` ˘ ptq ” ` ˘ “ 1 ´ αp1 ´ γq ´ αγσK ri ` ´ 1 ´ αp1 ´ γq ´ αγσK ` 1 ´ αγσK δi “ ‰ ptq p0q “ 1 ´ α ` αγp1 ´ σK q ri ` αp1 ´ γqδi , ri “ δi . ` ptq ˘ ptq where we have also used γ cptq ´ ci “ ri ´ δi in (75), and we have complete the derivation of (27) in subsection V-B3. When σK “ 1, (75) reduces to the undamped recurrence in proposition 2, i.e., (51). p0q pt`1q Here, span invariance is also preserved with ri “ δi P spantδi u, since (75) expresses ri as an R-linear ptq combination of ri and δi , so ptq ri P spantδi u for all t ě 0. (76) ptq

Writing ri “ ψt δi and substituting into (75) yields “ ‰ ψt`1 “ 1 ´ α ` αγp1 ´ σK q ψt ` αp1 ´ γq,

ψ0 “ 1.

From (77), we can derive ψt with ψ0 “ 1 as follows ˜ ¸ “ ‰t 1´γ 1´γ ψt “ ` 1´ ¨ 1 ´ α ` αγp1 ´ σK q , 1 ´ γp1 ´ σK q 1 ´ γp1 ´ σK q

(77)

(78)

with the steady state of ψ when t Ñ 8 for α ą 0 ψ8 “

1´γ . 1 ´ γp1 ´ σK q

(79)

So the approach to (79) is exponential of rate ζ . From (79) we can see ψ8 ą 1 ´ γ , thus the residual retains a larger fraction of δi than the undamped lock-in 1 ´ γ in (55). Then we further derive the steady-state reference field as p8q r ` ψ8 δi “ ψ8 gri ` p1 ´ ψ8 qG, r ui “ G (80) which lies closer to gri than the undamped counterpart 1 ´ γ in (56). We present a complete convergence analysis of the CIDERS algorithm under the personalized bilevel optimization. The analysis establishes a joint convergence rate on the upper-level meta-objective gap and the lower-level approximation error. H. Problem Statement

The bilevel optimization problem in CIDERS is simplified for our careful proof. Specifically, we denote the frozen full teacher backbone as wT , learnable student backbone as wS , global personalization messenger as v and the client i’s personalization messenger as vi . Each client i holds a private dataset Di and the server has access to a public dataset Dpub . Based on the global messenger v with a good meta-initialization for rapid client-specific adaptation, the upper-level objective is the meta-learning formulation ÿN ` ˘ Φpvq “ 1{N Li vr, wS˚ pvq , (81) i“1

where vr “ v ´ η∇v Li pv, wS˚ pvqq is the one-step global personalized messenger, and the local client loss Li is evaluated on its private data Di . On the other hand, the server solves the lower-level problem to obtain a high-quality student backbone wS that is aligned with both the frozen teacher and the downstream task, conditioned on the current global messenger v : wS˚ pvq “ arg minwS LKD pwS ; vq,

(82)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

where the TAKD loss is evaluated on the public data Dpub . The overall problem is the following personalized bi-level optimization: ›2 › (83) minv Φpvq ` ϵ{2 ›v ´ v ´ › , s.t. wS “ arg minwS LKD pwS ; vq. I. Algorithm Abstraction

We simply illustrate the algorithm for further convergence analysis. At the t-th round, the server maintains ptq ptq the global messenger v ptq , the teacher backbone wT , the student backbone wS , and the global consensus variate cptq . A subset St of clients performs local updates in parallel. Each client i P St initializes its local ptq ptq messenger as vi,0 “ v ptq and retrieves its local consensus variate ci . Over K local steps, client i computes the personalized meta-gradient with one-step look-ahead: ` ptq ptq ˘ gpi,k “ ∇v Li vri,k , wS ; ξi,k , (84) ` ptq ptq ˘ ptq ptq where vri,k “ vi,k ´ηinner ∇v Li vi,k , wS ; ξi,k . The local messenger is then updated via the consensus-corrected rule: ı ” ˘ ` ` ptq ptq ˘ ptq ptq (85) ´ εη vi,k ´ v ptq . vi,k`1 Ð vi,k ´ η gpi,k ` γ cptq ´ ci ` ptq ˘ After K local steps, client i computes the averaged displacement ∆i,t :“ 1{Kη v ptq ´ vi,K and refreshes its local consensus variate by the exponential moving average: “ ‰ ptq pt`1q ci “ 1 ` αpγ ´ 1q ci ` α∆i,t ´ αγcptq . (86) pt`1q

ptq

Only the compact increment ∆ci “ ci ´ ci is transmitted to the server. Upon receiving t∆ci uiPSt , the server exactly reconstructs the displacements t∆i,t u by inverting the EMA update. It then performs global aggregation and solves the lower-level KD problem: $ ÿ ptq ’ p1 ´ αqc ` α pi ∆i,t , ’ ’ iPSt & ÿ pt`1q pcpt`1q , v pt`1q , wS q Ð v ptq ´ τ Kη (87) pi ∆i,t , iPSt ’ ’ ’ % arg min L `w ; v pt`1q ˘. wS KD S After T communication rounds, the algorithm outputs the final global messenger v ptq and the student backbone pT q wS . The framework coordinates bi-level optimization through consensus-variate correction, enabling lowcommunication personalized adaptation while mitigating client drift under heterogeneous data distributions. J. Definitions

We define the following auxiliary quantities used throughout the analysis: ‚ Client drift Et at round t: client drift quantifies the average squared deviation of each client’s local messenger

trajectory from the global messenger during the K local steps. It is defined as › ptq ›2 1 ÿK´1 ÿN E›vi,k ´ v ptq › , Et :“ i“1 KN k“0 ptq

(88)

where v ptq is the global messenger and vi,k denotes the local messenger trajectory of client i after k local steps in round t. ‚ Local consensus lag Ct at round t: local consensus lag measures the average squared difference between the local consensus variates and the true personalized meta-gradients evaluated at the global messenger. It is defined as › ptq ` ˘›2 1 ÿN E›ci ´ ∇Φ v ptq , wS pv ptq q › . (89) Ct :“ N i“1

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

rt at round t: global consensus lag measures the deviation of the global consensus ‚ Glocal consensus lag C

variate cptq from the meta-gradient of the upper-level objective evaluated at the current global messenger and the current student backbone. It is defined as ÿN › ` ˘›2 rt :“ 1 C (90) E›cptq ´ ∇Φ v ptq , wS pv ptq q › . N i“1 ‚ Joint quantity of interest: this quantity that jointly tracks upper-level meta-suboptimality and lower-level distillation error: › ›2 Φpvq ´ Φ˚ ` ›wS ´ wS˚ pvq› , (91) where wS˚ pvq denotes the exact minimizer of the lower-level task-aware knowledge distillation loss LKD pwS ; vq for a fixed messenger v . ‚ Full practical meta-gradient: in CIDERS, the full practical meta-gadient at the local messenger vi,k with the current student backbone wS on the client i and the k th local iteration can be computed via ¯ ´ gri,k pvi,k , wS q :“ ∇v Li vi,k ´ ηinner ∇v Li pvi,k , wS q, wS , (92) where wS is the approximate student backbone currently maintained by the server. ‚ Stochastic practical meta-gradient estimator: CIDERS actually compute the meta-gradient at the local messenger vi,k with the current student backbone wS on the client i and the k th local iteration via gpi,k pvi,k , wS q “ ∇v Li pr vi,k , wS ; ξi,k q,

(93) ´ ¯ “ vi,k ´ ηinner ∇v Li vi,k , wS ; ξi,k , and it satisfies

where the one-step look-ahead point is vri,k “ ‰ E gpi,k pvi,k , wS q “ gri,k pvi,k , wS q. ‚ Ideal meta-gradient: this is the meta-gradient that would be obtained if the lower-level problem are solved exactly with respect to the current global messenger v : ´ ¯ ideal gi,k pvi,k , wS˚ pvqq :“ ∇v Li vi,k ´ ηinner ∇v Li pvi,k , wS˚ pvqq, wS˚ pvq , (94) where wS˚ pvq denotes the exact minimizer of the lower-level task-aware knowledge distillation objective: wS˚ pvq :“ arg min LKD pwS ; vq. wS

K. Assumptions

We state the complete set of assumptions for the convergence analysis. Assumption 1 (Joint Smoothness). For each client i “ 1, . . . , N , the loss function Li pv, wS q is jointly Lsmooth with respect to the pair of variables pv, wS q. That is, for any pv, wS q and pv 1 , wS1 q, ` ˘ }∇Li pv, wS q ´ ∇Li pv 1 , wS1 q} ď L }v ´ v 1 } ` }wS ´ wS1 } . (95) Consequently, the upper-level meta-objective Φpvq is β -smooth, where the effective smoothness constant β is given by β “ Lp1 ` ηinner Lq2 ` ηinner ρ,

(96)

and ρ denotes the Lipschitz constant of the Hessian of Li with respect to the messenger v . In addition, we assume the lower-level TAKD objective LKD pwS ; vq is Ly -smooth with respect to wS . ` Assumption 2 (Polyak-Łojasiewicz Inequality). The upper-level objective satisfies }∇Φpvq}2 ě 2µ Φpvq ´ ˘ Φ˚ for some µ ą 0.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

Assumption 3 (Meta-Gradient Dissimilarity). There exist constants G, B ě 0 such that, for any local mesenger vi,k on the client i k th local iteration, global messenger v and current student backbone wS , › ›2 E›gri,k pvi,k , wS q ´ ∇Φpvq› ď G2 ` B 2 }∇Φpvq}2 .

Assumption 4 (Gradient Heterogeneity). There exists ζ ě 0 such that 1{N

(97)

›2 řN › › ď ζ 2. › i“1 ∇v Li pv, wS q´∇Φpvq

Assumption 5 (Gradient Variance). The stochastic meta-gradients used in the upper-level client updates have bounded variance for a constant ζ 2 ě 0 such that: › ` ptq ptq ˘ ` ptq ptq ˘›2 Eξi,k ›gri,k vi,k , wS ; ξi,k ´ ∇v Li vi,k , wS › ď ζ 2 ,

(98)

2 ě 0 such that Also for the lower-level TAKD objective, there exists a constant σKD

›2 › 2 . Eξ ›∇wS LKD pwS ; v; ξq ´ ∇wS LKD pwS ; vq› ď σKD

(99)

Assumption 6 (Bounded gradients and Hessian). There exist nonnegative constants G, B such that the following bounds hold uniformly for all clients and for all messenger and backbone parameters v, wS , i.e., Gradient bound }∇v Li pv, wS q} ď G, and Hessian norm bound }∇2v Li pv, wS q} ď B. Assumption 7 (Hessian Lipschitz Continuity). There exists a nonnegative constant H such that, for each client i and for any two pairs of parameters pv, wS q and pv 1 , wS1 q, i.e., }∇2v Li pv, wS q ´ ∇2v Li pv 1 , wS1 q} ď Hp}v ´ v 1 } ` }wS ´ wS1 }q. L. Basic Lemmas

Lemma 3 (Personalized Meta-Gradient Dissimilarity). Under Assps. A1 and A5, for any client i, messenger v , and student backbone wS , let gi pvq “ ∇v Li pv, wS q denote the primal gradient. Then, 2 }r gi pvq ´ gi pvq}2 ď L2 ηinner }gi pvq}2 ,

(100)

where β is the effective smoothness constant of the upper-level meta-objective defined in Assps. A1. Proof. By Assumption A1 the client loss Li is jointly L-smooth in the pair pv, wS q. Fixing wS , it follows that Li p¨, wS q is L-smooth in v , i.e., v ÞÑ ∇v Li pv, wS q is L-Lipschitz continuous. Hence, we have }∇v Li pv ´ ηinner gi pvq, wS q ´ ∇v Li pv, wS q} ď L }pv ´ ηinner gi pvqq ´ v} ď Lηinner }gi pvq} .

(101)

This completes the proof. Lemma 4 (Upper-level Gap Inequality). Under A2 and A3, for any messenger parameter v and any admissible student backbone parameter wS , N

1 ÿ 2 2 E}r gi pv, wS q }2 ď p8L2 ηinner ` 4qζ 2 ` p16L2 ηinner β ` 4βq pΦpvq ´ Φ˚ q , N i“1

(102)

where Φ˚ “ minv Φpvq. Proof. Fix an arbitrary messenger parameter (v) and an admissible student backbone parameter wS . For notational convenience denote gri :“ gri pv, wS q and gi :“ ∇v Li pv, wS q. All expectations are taken with

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

respect to any stochasticity appearing in the gradient estimators. We first perform the decomposition via the triangle inequality: 1 ÿN 1 ÿN E }r gi }2 ď 2 ¨ E }r gi ´ ∇Φpvq}2 ` 2}∇Φpvq}2 N i“1 N i“1 4 ÿN 4 ÿN ď E }r g i ´ g i }2 ` E }gi ´ ∇Φpvq}2 ` 2}∇Φpvq}2 N i“1 N i“1 4 ÿN 2 E }gi }2 ` 4ζ 2 ` 2}∇Φpvq}2 ď L2 ηinner ¨ (103) i“1 N ÿN 8 2 2 }gi ´ ∇Φpvq}2 ` 8L2 ηinner ď L2 ηinner ¨ }∇Φpvq}2 ` 4ζ 2 ` 2}∇Φpvq}2 N i“1 2 2 ď p8L2 ηinner ` 4qζ 2 ` p8L2 ηinner ` 2q}∇Φpvq}2 2 2 ď p8L2 ηinner ` 4qζ 2 ` p16L2 ηinner β ` 4βq pΦpvq ´ Φ˚ q

where we have performed the decomposition of the dissimilarity term in the second inequality, i.e., }r gi ´ ∇Φpvq}2 ď 2 }r gi ´ gi }2 ` 2 }gi ´ ∇Φpvq}2 , decomposed via }gi }2 ď 2 }gi ´ ∇Φpvq}2 ` 2}∇Φpvq}2 in the fourth inequality, and we have used }∇Φpvq}2 ď 2βpΦpvq ´ Φ˚ q in the last inequality. Lemma 5 (Lower-level Approximation Dynamics). Suppose it satisfies Assp. A1, after performing E steps of gradient descent on the lower-level task-aware knowledge distillation objective with fixed messenger parameter v ptq , the expected squared distance to the exact minimizer satisfies 2 ηKD σKD 2 ` 8Rw p1 ´ µηKD qE . (104) µ Proof. We now explain the proof by starting from the deterministic contraction, which can be the foundational ` ˘ inequality. Specifically, because LKD ¨; v ptq is Ly -smooth and µ-strongly convex, any deterministic gradient step with step size ηKD ď 1{Ly satisfies the contraction pt´1q

ptq

E}wS ´ wS˚ pv ptq q}2 ď 2p1 ´ µηKD qE E}wS

´ wS˚ pv pt´1q q}2 `

}wS ´ ηKD ∇wS LKD pwS ; v ptq q ´ wS˚ pv ptq q}2 ď p1 ´ µηKD q}wS ´ wS˚ pv ptq q}2 .

(105)

Next, we consider the one-step recursion with the SGD for the KD objective. Consider the SGD step by adding and subtracting the true gradient: wSk`1 ´ wS˚ pv ptq q “ pwSk ´ ηKD ∇wS LKD pwSk ; v ptq q ´ wS˚ pv ptq qq ´ ηKD pgpwSk ; ξk q ´ ∇wS LKD pwSk ; v ptq qq. (106)

We take the L2 ´norm and the conditional expectation with respect to the current mini-batch ξk conditioned on all previous randomness, with the notice that the cross term vanishes by unbiasedness of the stochastic gradient. This immediately yields 2 2 E}wSk`1 ´ wS˚ pv ptq q}2 | history up to k ď }wSk ´ ηKD ∇wS LKD pwSk ; v ptq q ´ wS˚ pv ptq q}2 ` ηKD σKD .

(107)

Subsequently, by inserting the deterministic contraction from (105) into (107) produces the fundamental onestep inequality 2 2 E}wSk`1 ´ wS˚ pv ptq q}2 | history up to k ď p1 ´ µηKD q}wSk ´ wS˚ pv ptq q}2 ` ηKD σKD .

(108)

Unrolling the recurrence (108) over E steps by taking the unconditional expectation yields the linear recurrence ÿE´1 2 2 E}wSE ´ wS˚ pv ptq q}2 ď p1 ´ µηKD qE E}wS0 ´ wS˚ pv ptq q}2 ` ηKD σKD p1 ´ µηKD qj . (109) j“0 ř j Under the assumption that 0 ă µηKD ă 1, the geometric sum is bounded by E´1 j“0 p1 ´ µηKD q “ p1´p1´µηKD qE q{µηKD ď 1{µηKD . Therefore, we have ptq

pt´1q

E}wS ´ wS˚ pv ptq q}2 ď p1 ´ µηKD qE E}wS

´ wS˚ pv ptq q}2 `

2 ηKD σKD . µ

(110)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

` ˘ At this point we have a bound expressed in terms of the distance to the current minimizer wS˚ v ptq , but the pt´1q warm-start wS was produced with respect to the previous minimizer. By performing the decomposition, we can simply handle the change of target minimizer with the bi-level coupling pt´1q

}wS

pt´1q

´ wS˚ pv ptq q}2 ď 2}wS ´ wS˚ pv pt´1q q}2 ` 2}wS˚ pv pt´1q q ´ wS˚ pv ptq q}2 › ¯›2 ´ › pt´1q › 2 ď 2E ›wS ´ wS˚ v pt´1q › ` 8Rw

(111)

where for brevity, we assume the uniform boundedness }wS˚ pvq} ď Rw that is valid for every messenger, and it converts the movement of the target into an additive term that still contracts geometrically. Finally, we substitute the inequality (111) into (110): ptq

pt´1q

E}wS ´ wS˚ pv ptq q}2 ď 2p1 ´ µηKD qE E}wS

´ wS˚ pv pt´1q q}2 `

2 ηKD σKD 2 ` 8Rw p1 ´ µηKD qE , µ

(112)

which completes the proof. M. Key Lemmas

With client drift and consensus lag defined, we now present four lemmas that analyze the behavior of the PERCE algorithm. These lemmas address the variance of the server update, the evolution of consensus lag, the bounding of client drift, and the progress made per round. r2 , B r2q Lemma 6 (Local Consensus Deviation). For all t ě 0 and all clients i, there exist positive values pG such that › ´ ¯›2 ptq r2 ` B r 2 ››∇Φ v ptq ›› (113) E}ci ´ ∇Φpv ptq q}2 ď G ptq

Proof. Let us recall the local update. At round t, client i initializes vi,0 “ v ptq , i.e., the global messenger received at the beginning of the round and performs K local steps according to ı ” ˘ ` ` ptq ptq ˘ ptq ptq (114) ´ εη vi,k ´ v ptq , vi,k`1 “ vi,k ´ η gri,k ` γ cptq ´ ci for k “ 0, . . . , K ´ 1, where gri,k is the practical meta-gradient evaluated at the current local point and the ptq current fixed backbone wBL . Then, we sum both sides from k “ 0 to k “ K ´ 1 ´ ¯ ¯ ÿK´1 ÿK´1 ´ ptq ptq ptq ptq ptq ptq . (115) v ´ vi,K “ η g̃i,k ` ηK ¨ γ c ´ ci ` ηε vi,k ´ v k“0

k“0

ptq gri

řK´1

Define the averaged practical meta-gradient :“ 1{K k“0 gri,k , and the averaged proximal residual řK´1 ptq ptq ptq ptq rprox,i :“ ε{K k“0 pvi,k ´ v q, then according to ∆i,t :“ 1{Kηpv ptq ´ vi,K q, we have ´ ¯ ptq ptq ptq ptq pt`1q ptq ptq g i ` rprox,i q. (116) ∆i,t “ gri ` γ cptq ´ ci ` rprox,i , and ci “ p1 ´ αqci ` αpr ptq

ptq

We further define the error vector ei :“ ci ´ ∇Φpv ptq q, the driving noise δi :“ gri ´ ∇Φpv ptq q ` rprox,i , and ∆Φptq :“ ∇Φpv ptq q ´ ∇Φpv pt`1q q for brevity. Subsequently, we can expand pt`1q 2

E}ei

ptq

} “ E}p1 ´ αqei ` αδi ` ∆Φptq }2 ptq

ptq

“ p1 ´ αq2 E}ei }2 ` 2αp1 ´ αqExei , δi y ` α2 E}δi }2 ptq

(117)

ptq

` 2p1 ´ αqExei , ∆Φptq y ` 2αExδi , ∆Φptq y ` E}∆Φptq }2 . ptq

For the driving noise term E}δi }2 , through triangle inequality we have ptq

ptq

g i ´ ∇Φpv ptq q}2 ` 2E}rprox ,i }2 E}δi }2 ď 2E}r ď 2G2 ` 2B 2 }∇Φpv ptq q}2 `

2σ 2 ` 2Cr ε2 , K

(118)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

where we have used the meta-gradient dissimilarity assumption, variance of the average over the local K ptq steps, and the proximal term satisfies E}rprox,i }2 ď Cr ε2 . Next, we bound the three cross terms in (117) using Young’s inequality with parameter ε1 ą 0 as follows: ptq ptq ‚ For the term 2αp1 ´ αqExei , δi y: ptq

ptq

2αp1 ´ αqExei , δi y ď αp1 ´ αqε1 E}ei }2 `

αp1 ´ αq ptq E}δi }2 , ε1

(119)

1´α E}∆Φptq }2 . ε1

(120)

ptq ‚ For the term 2p1 ´ αqExei , ∆Φptq y: ptq

ptq

2p1 ´ αqExei , ∆Φptq y ď p1 ´ αqε1 E}ei }2 ` ptq ‚ For the term 2αExδi , ∆Φptq y: ptq

ptq

2αExδi , ∆Φptq y ď αε1 E}δi }2 `

α E}∆Φptq }2 . ε1

Note by β -smoothness of Φ, we have E}∆Φptq }2 ď β 2 E}v pt`1q ´ v ptq }2 , it subsequently leads to › ›2 › ›2 ›ÿ › › › › › r 2, E}∆Φptq }2 ď β 2 E ›v pt`1q ´ v ptq › “ β 2 τ 2 K 2 η 2 E › pi ∆i,t › ď β 2 τ 2 K 2 η 2 ∆ › ›

(121)

(122)

iPSt

› ›2 Ă2 :“ sup E ›ř › for simplicity, since ∆i,t “ 1{Kηpv ptq ´ v ptq q. Hence, where we have defined ∆ t iPSt pi ∆i,t i,K we substitute (118)-(122) into (117) and it leads to “ ‰ pt`1q 2 ptq E}ei } ď p1 ´ αq2 ` αp1 ´ αqε1 ` p1 ´ αqε1 E}ei }2 „ ȷˆ ˙ › ´ ¯›2 2σ 2 αp1 ´ αq 2 2 2› ptq › 2 ` α ` ` αε1 2G ` 2B ›∇Φ v ` 2Cr ε › ` (123) ε1 K „ ȷ 1´α α r 2. ` ` ` 1 β 2τ 2K 2η2∆ ε1 ε1

Note (123) has the form Vt`1 ď ρ1 Vt ` ρ2 for the sequence Vt , thus we can obtain the rsult as follows: ” ı › ` ˘›2 2 α2 ` αp1´αq ` αε B 2 ›∇Φ v ptq › 1 ε 1 ptq p0q E}ci ´ ∇Φpv ptq q}2 ď E}ci ´ ∇Φpv p0q q}2 ` 1 ´ rp1 ´ αq2 ` αp1 ´ αqε1 ` p1 ´ αqε1 s ¯ ´ ¯ ” ı´ (124) 2 2 ` 2σ 2 ` 2C ε2 ` 1 ` 1 β 2 τ 2 K 2 η 2 ∆ r ` αε α2 ` αp1´αq 2G 1 r ε1 K ε1 ` , 2 1 ´ rp1 ´ αq ` αp1 ´ αqε1 ` p1 ´ αqε1 s r 2 and G r 2 satisfies the following where it can be seen that B ” ı 2 α2 ` αp1´αq ` αε B2 1 ε1 r2 “ B , 1 ´ rp1 ´ αq2 ` αp1 ´ αqε1 ` p1 ´ αqε1 s ” ı´ ¯ ´ ¯ 2 ` 2σ 2 ` 2C ε2 ` 1 ` 1 β 2 τ 2 K 2 η 2 ∆ r2 α2 ` αp1´αq ` αε 2G 1 r ε1 K ε1 p0q 2 p0q 2 r G “ E}ci ´ ∇Φpv q} ` . 1 ´ rp1 ´ αq2 ` αp1 ´ αqε1 ` p1 ´ αqε1 s (125)

Lemma 7 (Variance of Reconstructed Updates). Under Assumptions A3, A4 and A7, we have › › ›1 ÿ ´ ¯›2 1 ˆ 6σ 2 ` ˘ › ptq › E› ∆i,t ´ ∇Φ v ` 12 G2 ` B 2 }∇Φ}2 › ď ›S › p K iPSt ˙ ›2 › 2 2 › ptq ˚ ptq › 2 r `12γ pCt ` Ct q ` 6Ly E ›wS ´ wS pv q› ` 2ε2

(126)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

Proof. We prove the stated bound under Assumptions A3, A4 and A7. We work at a fixed communication ptq round t, with the global messenger fixed at v “ v ptq and the current backbone fixed at w “ wS . Let ` ptq ˘ ∇Φ “ ∇Φ v for brevity. By Lemma 1 the reconstruction of each ∆i,t is exact up to the additive residual ptq rrecon whose second moment is bounded by the finite constant ε23 (Assumption A5). The proximal residual ptq rprox ,i is likewise bounded in second moment by the finite constant ε22 (Assumption A5). Both residuals contribute additive terms that are absorbed into the explicit constants of the overall convergence rate (as visible in Lemma 4); they do not appear in the leading expression of the present lemma. We first telescoping identity for the client displacement. Specifically, the client executes the consensus-corrected local update ptq

ptq

vi,k`1 “ vi,k ´ ηrr gi,k ` γpcptq ´ ci qs ´ εηpvi,k ´ v ptq q,

(127)

ptq

for k “ 0, . . . , K ´ 1, starting from vi,0 “ v ptq . Telescoping the recurrence yields the exact identity ÿK´1 ptq v ptq ´ vi,K “ η ui,k , k“0

(128)

where we define the instantaneous effective direction as ptq

ptq

ui,k “ gri,k ` γpcptq ´ ci q ` εpvi,k ´ v ptq q.

(129)

Dividing by Kη and invoking the definition of the averaged displacement therefore gives 1 ÿK´1 ui,k . ∆i,t “ K k“0

(130) ptq

We consider the separation of the proximal contribution for simplicity, let ∆i,t “ ∆1i,t ` rprox, ,i , where we define ∆1i,t as 1 ÿK´1 ptq ∆1i,t :“ pr gi,k ` γpcptq ´ ci qq, (131) K k“0 and rprox, i denotes the averaged proximal contribution. Therefore, we can evaluate Vi :“ E }∆i,t ´ ∇Φ}2 via the triangle inequality }a ` b}2 ď 2}a}2 ` 2}b}2 as follows: › ›2 Vi ď 2E ›∆1i,t ´ ∇Φ› ` 2ε22 , (132) ptq

ptq

with the Assumption A5 that E}rprox, i }2 ď ε22 . It therefore suffices to bound the deviation of ∆1i,t . Next, we define the averaged stochastic meta-gradient 1 ÿK´1 gri :“ gri,k . (133) K k“0 ptq

Then it can be obtained ∆1i,t “ gri ` γpcptq ´ ci q, and we can decompose the following ptq

∆1i,t ´ ∇Φ “ Si ` Di ` bias L,i ,

(134) ptq

ptq

where have defined Si “ g̃ i ´ gi to capture client-specific stochastic, Di “ gi ´ ∇Φ ` γpcptq ´ ci q to measure the consensus variation, and biasL,i tracks the lower-level bias and is defined exactly by ´ ´ ¯ ¯ ptq ptq biasL,i : “ ∇v Li v ´ ηinner ∇v Li v, wS , wS ´ ∇v Li pvd ´ηinner ∇v Li pv, wS˚ pwqq, wS˚ pvqq . (135) Here wS˚ pvq is the unique minimizer of the lower-level TAKD loss LKD p¨; wq. The term bias L,i therefore measures the exact difference between the meta-gradient evaluated at the algorithm’s current approximate ptq backbone wS and the meta-gradient that would be obtained if the lower level were solved exactly to optimality for the current messenger w. By Assumption A4, each stochastic meta-gradient gri,k is an unbiased estimator of the true client meta-gradient gi pv, wS q :“ ∇v Li pv ´ηinner ∇v Li pv, wS q, wS ) and satisfies the exact variance bound E}r gi,k ´ gi }2 ď σ 2 . The first term in (134) can be evaluated by averaging the K terms g i ´ gi }2 ď σ2{K . E}r

(136)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

Next, we consider Lipschitz bound on the lower-level bias. By Assumption A2, the lower-level loss LKD p¨; wq is Ly -smooth for any fixed w. Combined with the chain-rule dependence of the upper-level loss Li on the backbone parameters through the composition PM,i ˝ BL , the map wS ÞÑ ∇v Li pw ´ ηinner ∇v Li pv, wS q , wS q

is Lipschitz continuous with constant at most Ly . Consequently we have › ›2 › ptq › }biasL,i }2 ď L2y E ›wS ´ wS˚ pvq› . ptq

(137)

(138)

ptq

Hence, it remains to bound the contribution Di “ gi ´ ∇Φ ` γpcptq ´ ci q, by using the triangle inequality, we can obtain › › › › › ptq ›2 › ptq ›2 E ›Di › ď 2Ei }gi ´ ∇Φ}2 ` 2γ 2 E ›cptq ´ ci › . (139) The first term on the right hand side of (139) can be bounded by Assumption A3 that E}gi ´ ∇Φ}2 ď ptq ptq G2 ` B 2 }∇Φ}2 . We continue to analyze E}cptq ´ ci }2 . Specifically, while we recall Ct “ E}ci ´ ` ptq ˘ ` ˘ ` ˘ ptq ∇Φ v , wS pv ptq q }2 , and with the decomposition pcptq ´ ∇Φ v ptq , wS pv ptq q q ´ tci ´ ∇Φ v ptq , wS pv ptq q u, and summing it leads to › › › ` ˘››2 › › ptq ptq ›2 E ›cptq ´ ci › ď 2E ›ci ´ ∇Φ v ptq , wS pv ptq q › (140) › ` ˘››2 › rt , ` 2E ›cptq ´ ∇Φ v ptq , wS pv ptq q › “ 2Ct ` 2C which we substitute into (139) and it yields › › ` ˘ › ptq ›2 rt q. Ei ›Di › ď 2 G2 ` B 2 }∇Φ}2 ` 2γ 2 pCt ` C Combining these inequalities (136)(141)(134), we have ˆ 2 ›2 ˙ › ›2 › 1 ` 2 ˘ σ › ˚ 2 2 2 2 › ptq r › › ` 2 G ` B }∇Φ} ` 2γ pCt ` Ct q ` Ly E ›wS ´ wS pvq› . Eξ ∆i,t ´ ∇Φ ď 3 K › ›2 › 1 › 2 1 Substituting Eξ }∆ i,t ´ ∇Φ} into Vi ď 2E ›∆i,t ´ ∇Φ› ` 2ε22 , we have Vi ď

› ›2 ` ˘ 6σ 2 rt q ` 6L2 E ››wptq ´ w˚ pwq›› ` 2ε2 , ` 12 G2 ` B 2 }∇Φ}2 ` 12γ 2 pCt ` C y S 2 S K

(141)

(142)

(143)

thus we can obtain the result in (126), this completes the proof. Lemma 8 (Server Messenger Update). Under Assumptions A1–A8, there exist positive values pΛ, Ω, Υ, Ψ, Γq that the expected squared displacement of the global personalization messenger between consecutive communication rounds is bounded as: › ›2 › ›2 ´ ¯ › › rt q ` ΥE ››wptq ´ w˚ pv ptq q›› ` ΩEt ` Ψ Φpv ptq q ´ Φ˚ ` Γ. (144) E ›v pt`1q ´ v ptq › ď ΛpCt ` C S S Proof. Fix an arbitrary communication round t ě 1. For brevity we write v “ v ptq , Φ “ Φpv ptq q, ∇Φ “ ptq ∇Φpv ptq q, and wS “ wS . All expectations are taken jointly over the random subset of participating clients St and the stochastic gradients realized inside those clients. According to the server aggregation rule, the global messenger is updated by 1ÿ ∆i,t , v pt`1q “ v ptq ´ τ Kη ¨ S iPSt

(145)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

ř ř here we decompose 1{S iPSt ∆i,t “ 1{S iPSt ∆i,t ´ ∇Φ ` ∇Φ, which leads to ›1 ÿ ›2 ›2 › › › E›v pt`1q ´ v ptq › “ τ 2 K 2 η 2 ¨ E› ∆i,t › iPS t S › ›2 ÿ › › 1 2 2 2 › ď 2τ K η ¨ E › ∆i,t ´ ∇Φ›› ` 2τ 2 K 2 η 2 E}∇Φ}2 S iPSt ˜ ¸ 12N τ 2 K 2 η 2 6σ 2 rt q ` 6L2 E}wS ´ w˚ pvq}2 ` 2ε2 ď ` 12G2 ` 12γ 2 pCt ` C y S 2 S K ¸ ˜ 12N τ 2 K 2 η 2 B 2 ` ` 2τ 2 K 2 η 2 }∇Φ}2 . S

(146)

where we have substituted the result in Lemma 7 into the first inequality of (146). Next, we continue to ř řN 1 analyze the term E}∇Φ}2 , and decompose it as ∇Φ ď ∇Φ ´ 1{N N i“1 g̃i pvq ` {N i“1 g̃i pvq, which result in › › ›2 › 2 ÿN 1 ÿN g̃i pvq›› ` E }r gi pvq}2 E}∇Φ}2 ď 2E ››∇Φ ´ N i“1 N i“1 › ›2 › › 1 ÿN 2 2 (147) › ď 2E ›∇Φ ´ g̃i pvq›› ` p16L2 ηinner ` 8qζ 2 ` p32L2 ηinner β ` 8βq pΦpvq ´ Φ˚ q i“1 N 2 ÿN 2 2 E }∇Φ ´ g̃i pvq}2 ` p16L2 ηinner ` 8qζ 2 ` p32L2 ηinner β ` 8βq pΦpvq ´ Φ˚ q , ď N i“1 where the second inequality in (147) is evaluated via Lemma 4, and it remains to evaluate the first term. Define ř the averaged local practical meta-gradient for client i by g̃¯i :“ 1{K K´1 k“0 g̃i,k , where each g̃i,k is evaluated at 2 the local point vi,k . Subsequently E}∇Φ ´ g̃i pvq} can be decomposed as: ›2 › K´1 › 4 K´1 ÿ` ˘›› 4 ÿ › 2 2 ideal ideal 2 E}∇Φ ´ g̃i pvq} ď 2E}g̃i pvq ´ g̃ i } ` gi pvi,k q ´ ∇Φ › E}g̃i,k ´ gi pvi,k q} ` E › › ›K K k“0 k“0 › › ˘›2 4 ÿK´1 ›` ideal E › gi pvi,k q ´ ∇Φ › ď 2β 2 Et ` 4L2y E }wS ´ wS˚ pvq}2 ` K k“0 (148) Here, the first term has been simply obtained by the β -smoothness of each client loss (Assumption A1) and the triangle inequality,i.e., E}g̃i pvq ´ g̃ i }2 ď β 2 Et . Similarly, with the meta-gradient map w.r.t. the backbone is Ly -Lipschitz, the second term can be obtained via }g̃i,k ´ giideal pvi,k q} ď Ly }wS ´ wS˚ pvq}. We now consider the last term, specifically E}giideal pvi,k q ´ ∇Φ}2 . Apply the triangle inequality, it can be decomposed }giideal pvi,k q ´ ∇Φ}2 ď 2}giideal pvi,k q ´ giideal pvq}2 ` 2}giideal pvq ´ ∇Φ}2 ď 2β 2 }vi,k ´ v}2 ` 4}giideal pvq ´ hi pvq}2 ` 4}hi pvq ´ ∇Φ}2 ,

(149)

where for the first term in the first inequality, we apply β -smoothness of each client loss Li (Assumption A1), while for the second term, we define hi pvq “ ∇v Li pv, wS˚ pvqq, and further employ the decomposition }giideal pvq ´ ∇Φ}2 ď 2}giideal pvq ´ hi pvq}2 ` 2}hi pvq ´ ∇Φ}2 . Specifically for }giideal pvq ´ hi pvq}, we have ż1 › ` ˘› ideal }gi pvq ´ hi pvq} ď ›∇2v Li v ` tp´ηinner ∇v Li pv, wS˚ pvqqq, wS˚ pvq › (150) 0 ˚ ¨ } ´ ηinner ∇v Li pv, wS pvqq} dt “ ηinner BG, and for }hi pvq ´ ∇Φ}2 , we can use Assumption A3 to directly bound it 1 ÿN E }hi pvq ´ ∇Φpvq}2 ď ζ 2 . i“1 N Hence, by combining (149)(150)(151), it leads to › ›2 1 ÿK´1 ÿN 2 E ›giideal pvi,k q ´ ∇Φ› ď 2β 2 Et ` 4ηinner B 2 G2 ` 4ζ 2 . i“1 KN k“0

(151)

(152)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

With (147)(148)(152), we can obtain the evaluation of E}∇Φ}2 as follows: ›2 ` ˘ 2 ÿ ›› 2 2 E}∇Φpvq}2 ď ` 8qζ 2 ` p32L2 ηinner β ` 8βq Φpvq ´ Φ˚ E ∇Φpvq ´ gri pvq› ` p16L2 ηinner N i ›2 › 2 2 B 2 G2 ` p40 ` 16L2 ηinner qζ 2 ď 20β 2 Et ` 8L2y E›wS ´ wS˚ pvq› ` 32ηinner ` ˘ 2 ` p32L2 ηinner β ` 8βq Φpvq ´ Φ˚ .

(153)

By subtituting (153) into (146), we can obtain the final result of (144) in Lemma 8 as follows: ˙ ˆ › ›2 144N τ 2 K 2 η 2 γ 2 2 2 2 2 › pt`1q ptq › 2 12N τ K η B 2 2 2 r E ›v ´v › ď pCt ` Ct q ` 20β ` 2τ K η Et S S « ff ˆ ˙ 2 2 2 2 72N τ 2 K 2 η 2 L2y 2 12N τ K η B 2 2 2 ` ` 8Ly ` 2τ K η E }wS ´ wS˚ pvq}2 S S ˆ ˙ ˙ ˆ ` ˘ 12N τ 2 K 2 η 2 B 2 12N τ 2 K 2 η 2 6σ 2 2 2 2 ˚ 2 2 2 2 ` 2τ K η pΦpvq ´ Φ q ` ` 12G ` 2ε2 ` 32L ηinner β ` 8β S S K ˆ ˙ ` ` ˘ 2˘ 12N τ 2 K 2 η 2 B 2 2 2 2 2 2 ` 2τ K η ` 32ηinner ζ , B 2 G2 ` 40 ` 16L2 ηinner S (154) subsequently, we obtain pΛ, Ω, Υ, Ψ, Γq as follows ˆ ˙ 2 2 2 2 144N τ 2 K 2 η 2 γ 2 2 12N τ K η B 2 2 2 , Ω “ 20β ` 2τ K η Λ“ S S ˙ ˆ 2 2 2 2 72N τ 2 K 2 η 2 L2y 2 12N τ K η B 2 2 2 Υ“ ` 8Ly ` 2τ K η S S ˆ ˙ ` ˘ 12N τ 2 K 2 η 2 B 2 (155) 2 2 2 2 2 Ψ “ 32L ηinner β ` 8β ` 2τ K η S ˆ ˙ ˆ ˙ 12N τ 2 K 2 η 2 6σ 2 12N τ 2 K 2 η 2 B 2 2 2 2 2 2 Γ“ ` 12G ` 2ε2 ` ` 2τ K η ¨ S K S ` ` ˘ 2˘ 2 2 32ηinner B 2 G2 ` 40 ` 16L2 ηinner ζ

Lemma 9 (Evolution of Global Consensus Lag). Under Assumptions A1–A8 the global, there exists positive Ăg , κ Ă4 values pρrg , ρg , κg2 , κ 3 g q such that the global consensus lag evolution satisfies the following Ăg pΦpv ptq q ´ Φ˚ q ` κ Ă4 . rt`1 ď ρrg C rt ` ρg Ct ` κg ΥE}wptq ´ w˚ pv ptq q}2 ` κg ΩEt ` κ C S g 2 2 3 S

(156)

Proof. We derive the bound directly from the per-client Consensus Deviation result and the algorithm rules. Let us define the global error vector as eptq :“ cptq ´ ∇Φpv ptq q, recall the server-side update rule for the global ` ˘ ř consensus variate cpt`1q “ p1 ´ αqcptq ` α iPSt pi ∆i,t , subtracting ∇Φ v pt`1q from both sides yields ˜ ¸ ´ ¯ ÿ ept`1q “ p1 ´ αqeptq ` α pi ∆i,t ´ ∇Φ v ptq ` ∆Φptq , (157) iPSt

` ˘ ` ˘ where ∆Φptq “ ∇Φ v ptq ´ ∇Φ v pt`1q . Taking the squared norm and expectation, we expand › › › › › ›2 ›ÿ › ›2 ´ ¯›2 › pt`1q ›2 › 2 › ptq › 2 › ptq › ptq › E ›e pi ∆i,t ´ ∇Φ v › “ p1 ´ αq E ›e › ` α E › › ` E ›∆Φ › › › iPSt C G C G ´ ¯ ´ ¯ ÿ ÿ ` 2αp1 ´ αqE eptq , pi ∆i,t ´ ∇Φ v ptq ` 2αE pi ∆i,t ´ ∇Φ v ptq , ∆Φptq . iPSt

iPSt

(158)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

ř pt´1q For brevity, we define δg :“ iPSt´1 pi ∆i,t´1 ´ ∇Φpv pt´1q q. Then, we can bound the three cross terms in (158) by Young’s inequality with a positive parameter ε1 ą 0 › ›2 αp1 ´ αq › ›2 A E › › › › 2αp1 ´ αqE ept´1q , δgpt´1q ď αp1 ´ αqε1 E ›ept´1q E ›δgpt´1q › , › ` g g ε1 › › ›2 › E A › pt´1q ›2 1 ´ α › › pt´1q pt´1q (159) ď p1 ´ αqε1 E ›eg 2p1 ´ αqE eg , ∆Φ E ›∆Φpt´1q › , › ` ε1 › ›2 ›2 A E α ›› › › › 2αE δgpt´1q , ∆Φpt´1q ď αε1 E ›δgpt´1q › ` E ›∆Φpt´1q › , ε1 which we substitute it into (158) and the inequality becomes ›2 › ›2 “ ‰ ›› › › › E ›eptq › ď p1 ´ αq2 ` αp1 ´ αqε1 ` p1 ´ αqε1 E ›ept´1q › „ „ ȷ › ȷ › › ›2 (160) αp1 ´ αq 1´α α › pt´1q ›2 › › 2 ` α ` ` αε1 E ›δg ` ` 1 E ›∆Φpt´1q › . › ` ε1 ε1 ε1

Next, we evaluate δg . Notice from the exact summation of the local update over K steps we have the identity pt´1q pt´1q pt´1q pt´1q ∆i,t´1 “ gri ` γpcpt´1q ´ ci q ` rprox, i , where gri is the averaged practical meta-gradient on client i pt´1q

and rprox,i is the averaged proximal residual. Substituting this identity yields the decomposition ´ pt´1q ´ ¯¯ ´ ¯ ÿ ÿ pt´1q pt´1q 2 2 pt´1q pt´1q E}δg } “ 3 E} pi gri } ` 3γ E} ´ ∇Φ v pi c ´ ci }2 iPSt´1 loooooooooooooooooooooomoooooooooooooooooooooon

iPSt´1 loooooooooooooooooomoooooooooooooooooon

:“Ta

` 3 E}

:“Tb pt´1q 2 pi rprox, i } .

(161)

iPSt´1 loooooooooomoooooooooon :“Tc

For bounding Term Ta , we can first use Jensen’s inequality and then the meta-gradient dissimilarity assumption together with the bounded variance of the stochastic meta-gradients over the K local steps, thus it leads to › › ´ ¯›2 ´ ¯›2 σ 2 ÿ › pt´1q › › › Ta ď pi ›gri ´ ∇Φ v pt´1q › ď G2 ` B 2 ›∇Φ v pt´1q › ` . (162) iPSt´1 K pt´1q

pt´1q

For bounding Term Tb , Using the decomposition cpt´1q ´ci “ pcpt´1q ´∇Φpv pt´1q qq´pci together with Jensen’s inequality and the Local Consensus Deviation Lemma, we obtain ˆ › › ´ ¯›2 ´ ¯›2 ˙ › pt´1q pt´1q › 2 2› pt´1q › r r Tb ď 2 ›c ´ ∇Φ v › ` 2 G ` B ›∇Φ v › .

´∇Φpv pt´1q qq

(163)

By the standing second-moment assumption on the proximal residual, the third term can be bounded as pt´1q 2 Tc ď Cr ε2 . Substitute these three estimates into the inequality for E}δg } : › ›2 › ›2 ´ ¯› ´ ¯›2 › › › › r 2 ››∇Φ v pt´1q ›› E ›δgpt´1q › ď6γ 2 ›ept´1q › ` 3B 2 ` 6γ 2 B ˆ ˙ (164) σ2 2 2 r2 2 ` 3G ` 3 ` 6γ G ` 3Cr ε . K We substitute (164) into (160) and it leads to " „ ȷ* αp1 ´ αq 2 2 2 rt´1 r Ct ď p1 ´ αq ` αp1 ´ αqε1 ` p1 ´ αqε1 ` 6γ α ` ` αε1 C ε1 ȷ´ „ ¯› ´ ¯›2 αp1 ´ αq 2 r 2 ››∇Φ v pt´1q ›› ` α ` ` αε1 3B 2 ` 6γ 2 B ε1 „ ȷˆ ˙ αp1 ´ αq σ2 2 2 2 r2 2 ` α ` ` αε1 3G ` 3 ` 6γ G ` 3Cr ε ε1 K „ ȷ › ›2 1´α α › › ` ` ` 1 β 2 E ›v ptq ´ v pt´1q › , ε1 ε1

(165)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

which is exactly (156) in the lemma, moreover we define the following positive values as " „ ȷ* αp1 ´ αq 2 2 2 ρg “ p1 ´ αq ` αp1 ´ αqε1 ` p1 ´ αqε1 ` 6γ α ` ` αε1 , ε1 „ „ ȷ´ ȷ ¯ αp1 ´ αq 1´α α g g 2 2 2 r2 κ1 “ α ` ` αε1 3B ` 6γ B , κ2 “ ` ` 1 β2 ε1 ε1 ε1 ȷˆ ˙ „ σ2 αp1 ´ αq g 2 2 r2 2 2 ` αε1 3G ` 3 ` 6γ G ` 3Cr ε , κ3 “ α ` ε1 K

(166)

and recall }∇Φpvq}2 ď 2β pΦpvq ´ Φ˚ q, which can be substituted into and it leads to Ăg pΦpv ptq q ´ Φ˚ q ` κ Ă4 , rt`1 ď ρrg C rt ` ρg Ct ` κg ΥE}wptq ´ w˚ pv ptq q}2 ` κg ΩEt ` κ C S g 2 2 3 S

(167)

Ăg “ κg Ψ ` 2βκg , and κ Ă4 “ κg Γ ` κg . where ρrg “ ρg ` κg2 Λ, ρg “ κg2 Λ, κ g 3 2 1 2 3

Lemma 10 (Evolution of Local Consensus Lag). Under Assumptions A1–A8, there exsits positive values pρC , κCr , κE , κw , κ0 q such that the local consensus lag satisfies the following ptq

rt ` κE Et ` κw E}w ´ w˚ pv ptq q}2 ` κΦ pΦpv ptq q ´ Φ˚ q ` κ0 Ct`1 ď ρC Ct ` κCr C S S

(168)

ptq

Proof. Let Ii be the indicator that client (i) is selected in round (t), so that ErIi s “ p. Non-selected clients keep their consensus variates unchanged. Define the local error and the target change ptq

ptq

ei :“ ci ´ ∇Φpv ptq , wS pv ptq qq,

ptq

di :“ ∇Φpv ptq , wS pv ptq qq ´ ∇Φpv pt`1q , wS pv pt`1q qq.

(169)

On a selected client the EMA update together with the exact reconstruction of the displacement ∆i,t produces the linear error recursion pt`1q

ptq

“ p1 ´ Ii αqpci ´ ∇Φpv pt`1q , wS pv pt`1q qqq ` Ii αpr g i ´ ∇Φpv pt`1q , wS pv pt`1q qqq.

(170)

Subsequently, the unconditional second moment satisfies pt`1q 2

E}ei

ptq

} ďp1 ´ pαqE}ci ´ ∇Φpv pt`1q , wS pv pt`1q qq}2 ptq

` pαE}ḡi ´ ∇Φpv pt`1q , wS pv pt`1q qq}2 .

(171)

We further decompose the first term on the right-hand side by incorporating ∇Φpv ptq , wS pv ptq qq, 1 ptq ptq ptq E}ci ´ ∇Φpv pt`1q , wS pv pt`1q qq}2 ď p1 ` εqE}ei }2 ` p1 ` qE}di }2 , ε which we substitute into (171) to produce 1 ptq ptq Ct`1 ď p1 ´ pαqp1 ` εqCt ` p1 ´ pαqp1 ` qE}di }2 ` pαE}ḡi ´ ∇Φpv pt`1q , wS pv pt`1q qq}2 . ε

(172)

(173)

ptq

For evaluating E}di }2 , we start from the joint Lipschitz estimate ptq

E}di }2 ď β 2 E}v pt`1q ´ v ptq }2 ` β 2 E}wS pv ptq q ´ wS pv pt`1q q}2 ptq

pt`1q 2

ď β 2 E}v pt`1q ´ v ptq }2 ` 2β 2 }wS ´ wS˚ pv ptq q}2 ` 2β 2 }wS˚ pv ptq q ´ wS ptq

}

ptq

2 ď β 2 E}v pt`1q ´ v ptq }2 ` 2β 2 }wS ´ wS˚ pv ptq q}2 ` 2β 2 p2E}wS ´ wS˚ pv ptq q}2 ` 8Rw q ptq

2 “ β 2 E}v pt`1q ´ v ptq }2 ` 6β 2 E}wS ´ wS˚ pv ptq q}2 ` 16β 2 Rw rt q ` β 2 ΩEt ` β 2 ΨpΦpv ptq q ´ Φ˚ q ď β 2 ΛpCt ` C

(174)

ptq

2 ` pβ 2 Υ ` 6β 2 qE}wS ´ wS˚ pv ptq q}2 ` pβ 2 Γ ` 16β 2 Rw q

where we have applies the already-established lower-level approximation bound in the third inequality that pt`1q

}wS

ptq

2 ´ wS˚ pv ptq q}2 ď 2E}wS ´ wS˚ pv ptq q}2 ` 8Rw ˚,

(175)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

ptq

and the Lemma 8 for the final substitution of E}v pt`1q ´ v ptq }2 . Next, we consider the last term E}ḡi ´ ∇Φpv pt`1q , wS pv pt`1q qq}2 in (173). Specifically, ptq

g i ´ ∇Φpv pt`1q , wS pv pt`1q qq}2 E}r ptq

ď 2E}r g i ´ ∇Φpv ptq , wS pv ptq qq}2 ` 2E}∇Φpv ptq , wS pv ptq qq ´ ∇Φpv pt`1q , wS pv pt`1q qq}2 K´1 ›2 2σ 2 2 ÿ ›› › ptq ď ` ›∇Φpvi,k , wS pv ptq qq ´ ∇Φpv ptq , wS pv ptq qq› K K k“0 ptq

2 ` 2β 2 E}v pt`1q ´ v ptq }2 ` 12β 2 E}wS ´ wS˚ pv ptq q}2 ` 32β 2 Rw

(176)

2σ 2 ptq 2 ` 2β 2 Et ` 2β 2 E}v pt`1q ´ v ptq }2 ` 12β 2 E}wS ´ wS˚ pv ptq q}2 ` 32β 2 Rw K rt ` p2β 2 ` 2β 2 ΩqEt ` p12β 2 ` 2β 2 ΥqE}wptq ´ w˚ pv ptq q}2 ď 2β 2 ΛCt ` 2β 2 ΛC S S ď

` 2β 2 ΨpΦpv ptq q ´ Φ˚ q ` p ptq

2σ 2 2 ` 32β 2 Rw ` 2β 2 Γq K ptq

Hence, by substituting E}di }2 in (174) and E}r g i ´ ∇Φpv pt`1q , wS pv pt`1q qq}2 in (176), we can obtain the result in the Lemma « ff ´ 1¯ 2 2 Ct`1 ď p1 ´ pαqp1 ` εq ` p1 ´ pαq 1 ` β Λ ` 2pαβ Λ Ct ε « ff « ff ´ ´ ¯ 1¯ 2 1 rt ` p1 ´ pαq 1 ` ` p1 ´ pαq 1 ` β Λ ` 2pαβ 2 Λ C β 2 Ω ` pαp2β 2 ` 2β 2 Ωq Et ε ε « ff ´ ` ˘ 1¯ 2 2 (177) ` p1 ´ pαq 1 ` β Ψ ` 2pαβ Ψ Φpv ptq q ´ Φ˚ ε « ff ´ 1¯ 2 ptq ` p1 ´ pαq 1 ` pβ Υ ` 6β 2 q ` pαp12β 2 ` 2β 2 Υq E}wS ´ wS˚ pv ptq q}2 ε ´ ´ 2σ 2 ¯ 1¯ 2 2 2 ` p1 ´ pαq 1 ` pβ Γ ` 16β 2 Rw q ` pα ` 32β 2 Rw ` 2β 2 Γ , ε K with the positive values defined as follows ˆ ˙ 1 ρC “ p1 ´ pαqp1 ` εq ` p1 ´ pαq 1 ` β 2 Λ ` 2pαβ 2 Λ, ε ˆ ˙ 1 κCr “ p1 ´ pαq 1 ` β 2 Λ ` 2pαβ 2 Λ, ε ˆ ˙ ` ˘ 1 κE “ p1 ´ pαq 1 ` β 2 Ω ` pα 2β 2 ` 2β 2 Ω , ε ˆ ˙ (178) ˘ ` ˘ 1 ` 2 2 2 2 κw “ p1 ´ pαq 1 ` β Υ ` 6β ` pα 12β ` 2β Υ , ε ˆ ˙ 1 κΦ “ p1 ´ pαq 1 ` β 2 Ψ ` 2pαβ 2 Ψ, ε ˆ ˙ ˆ 2 ˙ ˘ 1 ` 2 2σ 2 2 2 2 2 κ0 “ p1 ´ pαq 1 ` β Γ ` 16β Rw ` pα ` 32β Rw ` 2β Γ . ε K

Lemma 11. Lemma (Client Drift Bound for CIDERS). Under Assumptions A1–A8, there exists positive values r and E that the client-drift satisfies the fully explicit bound A, B, D ˘ ` ` ˘ rt q ` Du r pK´1q{2tD 1 A Φ v ptq ´ Φ˚ ` 2EpCt ` C Et ď . (179) 2 1 1 ´ KL D

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

ptq

Proof. For brevity, at an arbitrary communication round t ě 0, we write v “ v ptq , wS “ wS , ∇Φ “ ∇Φpv ptq q, ptq c “ cptq and ci “ ci . Define the local displacement of client i at local step k by di,k :“ vi,k ´ v. The initialization vi,0 “ v immediately yields di,0 “ 0. The local update of CIDERS is therefore equivalent to the linear recurrence ” ı ` ˘ di,k`1 “ di,k ´ η gri,k ` γ c ´ ci ` ε di,k (180) Taking L2 -norm and the conditional expectation with respect to the stochasticity of the practical meta-gradient gri,k given the current local messenger vi,k produces ˇ “ ‰ @ D E }di,k`1 }2 ˇ vi,k “ }di,k }2 ´ 2η di,k , Err gi,k | vi,k s ` γpc ´ ci q ` ε di,k (181) ˇ “ ‰ ` η 2 E }r gi,k ` γpc ´ ci q ` ε di,k }2 ˇ vi,k , where the practical meta-gradient is written as gri,k “ gi pvi,k ; wS q ` ξi,k , and gi pvi,k ; wS q denotes the deterministic one-step meta-gradient of client i. The noise ξi,k satisfies Erξi,k | vi,k s “ 0 and Er}ξi,k }2 | vi,k s ď σ 2 by Assumption A4. Define the auxiliary vector ui,k :“ gi pvi,k ; wS q ` γpc ´ ci q ` ε di,k . Then the third term in (181) can be simply evaluated via ˇ ‰ “ E }r gi,k ` γpc ´ ci q ` ε di,k }2 ˇ vi,k “ }ui,k }2 ` Er}ξi,k }2 | vi,k s ď }ui,k }2 ` σ 2 .

We substitute (182) into (181) and it yields ˇ “ ‰ @ D E }di,k`1 }2 ˇ vi,k ď }di,k }2 ´ 2η di,k , gi pvi,k ; wS q ` γpc ´ ci q ` ε di,k ` η 2 }gi pvi,k ; wS q ` γpc ´ ci q ` ε di,k }2 ` η 2 σ 2 .

(182)

(183)

The quadratic term on the right-hand side of (183) is expanded by: η 2 }gi ` γpc ´ ci q ` ε di,k }2 ď 3η 2 }gi }2 ` 3η 2 γ 2 }c ´ ci }2 ` 3η 2 ε2 }di,k }2 .

(184)

The inner-product term is split into three contributions: @ D @ D @ D ´2η di,k , gi ` γpc ´ ci q ` ε di,k “ ´2η di,k , gi ´ 2ηγ di,k , c ´ ci ´ 2ηε}di,k }2 .

(185)

Next, we proof the following: by the definition of the one-step meta-gradient and the β -smoothness of each client meta-loss (Assumption A1) there exists a remainder vector ei,k satisfying }ei,k } ď Qη}gi pvi,k ; wS q} such that ` ˘ gi pvi,k ; wS q “ pI ´ ∇2v Li pvi,k ; wS qq ∇v Li pvi,k ; wS q ` ei,k . (186) Recall the practical one-step meta-gradient of client i gi pvi,k ; wS q is defined as ´ ¯ ` ˘ ` ˘ gi vi,k ; wS :“ ∇v Li vi,k ´ ηinner ∇v Li vi,k ; wS , wS .

(187)

` ˘ Then, we introduce the corresponding look-ahead point zi,k :“ vi,k ´ ηinner ∇v Li vi,k ; wS . Assumption A1 guarantees that Li p ¨ ; wS q is twice continuously differentiable with respect to the messenger variable. Consequently the gradient map v ÞÑ ∇v Li pv; wS q is continuously differentiable. The fundamental theorem of calculus applied along the line segment joining vi,k and zi,k yields the exact identity ż1 ´ ¯` ˘ ` ˘ ˘ ` ˘ ` ∇2v Li vi,k ` t zi,k ´ vi,k ; wS zi,k ´ vi,k dt ∇v Li zi,k ; wS “ ∇v Li vi,k ; wS ` 0 ˜ż ¸ (188) ´ ¯ 1 ` ˘ ` ˘ ` ˘ 2 “ ∇v Li vi,k ; wS ´ ηinner ∇v Li vi,k ` t zi,k ´ vi,k ; wS dt ∇v Li vi,k ; wS . 0

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

´ ¯ ` ˘ ş1 Define the averaged Hessian along the segment by Havg,i,k :“ 0 ∇2v Li vi,k ` t zi,k ´ vi,k ; wS dt and the difference between this averaged Hessian and the Hessian evaluated at the original local messenger Ri,k :“ ` ˘ Havg,i,k ´ ∇2v Li vi,k ; wS . Equation (188) then becomes ` ˘ ` ˘ ` ` ˘ ˘ ` ˘ gi vi,k ; wS “ ∇v Li vi,k ; wS ´ ηinner ∇2v Li vi,k ; wS ` Ri,k ∇v Li vi,k ; wS ` ` ˘˘ ` ˘ ` ˘ “ I ´ ∇2v Li vi,k ; wS ∇v Li vi,k ; wS ´ ηinner Ri,k ∇v Li vi,k ; wS .

(189)

Subsequently, the remainder ei,k can be uniquely determined via a direct comparison of (189) and (186) as ei,k “ ´pI ´ ∇2v Li pvi,k ; wS qq´1 pηinner Ri,k ∇v Li pvi,k ; wS qq, whenever the indicated inverse exists. When the inverse does not exist, the same identity continues to hold after a standard δ -regularization (I ´ ∇2v Li ` δI ) followed by the limit δ Ñ 0` ; the resulting ei,k remains well-defined and bounded. From the Hessian-Lipschitz continuity of Li in Assumption A6 together with }zi,k ´ vi,k } “ ηinner }∇v Li pvi,k ; wS q}, it leads to ›ż ˜ ¸ › › 1 › ´ ¯ ` ˘ ` ˘ › › }Ri,k } “ › ∇2v Li vi,k ` t zi,k ´ vi,k ; wS ´ ∇2v Li vi,k ; wS dt› › 0 › ż1 ´ ¯ › ˘ ` ` ˘› ď ›∇2v Li vi,k ` t zi,k ´ vi,k ; wS ´ ∇2v Li vi,k ; wS › dt ż01 › › ď H t }zi,k ´ vi,k } dt “ Hηinner{2›∇v Li pvi,k ; wS q›, 0

where H is the Hessian-Lipschitz constant of Li . Consequently, we can bound }ei,k } via 2 › › Hηinner G . }ei,k } ď Q ηinner ›∇v Li pvi,k ; wS q›, where Q “ 2|p1 ´ Bq|

(190)

Let’s denote Hi,k “ ∇2v Li pvi,k ; wS q, then from (185), we have gi pvi,k ; wS q “ pI ´ Hi,k q∇v Li pvi,k ; wS q ` pI ´ Hi,k qei,k .

(191)

Consequently the inner-product term that appears in the drift recursion becomes @ D @ D @ D di,k , gi pvi,k ; wS q “ di,k , pI ´ Hi,k q∇v Li pvi,k ; wS q ` di,k , pI ´ Hi,k qei,k ,

(192)

Use Young’s inequality on the error term ( 2ab ď aϵ ` ϵb2 ): › ›2 › ˇA Eˇ ››dptq i,k › ˇ ptq ˇ 2 ˇ di,k , pI ´ Hi,k q ei,k ˇ ď ` ϵ11 }I ´ Hi,k }2 Q2 η 2 }∇v Li pvi,k ; wS q}2 ϵ11 › ›2 › ›2 › ptq › › ptq › ›di,k › ›di,k › 2 1 2 2 2 ď ` ϵ p1 ´ M q Q η }∇ L pv ; w q} ď ` ϵ1 Q2 η 2 }∇v Li pvi,k ; wS q}2 , v i S i,k 1 ϵ11 ϵ1

(193)

where we have used the assumption that the Hessian is bounded, i.e., }Hi,k } ď M . Similarly, applying Cauchy-Schwarz and Young‘s inequalities leads to: › ›2 › A E ››dptq i,k › ptq ´ 2 di,k1 pI ´ Hi,k q ∇v Li pvi,k ; wS q ď ` ϵ12 }I ´ Hi,k }2 Q2 η 2 }∇v Li pvi,k ; wS q}2 ϵ12 (194) › ›2 › ›2 › ptq › › ptq › ›di,k › ›di,k › ď ` ϵ12 p1 ´ M q2 Q2 η 2 }∇v Li pvi,k ; wS q}2 ď ` ϵ2 }∇v Li pvi,k ; wS q}2 . ϵ12 ϵ2

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

Combining the two inner-product bounds (193)(194) and substituting them into (192) leads to: ¨ › ›2 ˛ › ›2 › ptq › › ptq › A E d ›di,k › › › ˚ i,k ptq 2‹ ´2η di,k , gi pvi,k ; wS q ď η ˝ ` ϵ2 }∇v Li pvi,k ; wS q} ‚` η ϵ2 ϵ1 ˆ ˙ ` ˘ 1 1 ›› ptq ››2 2 2 2 2 `ηϵ1 Q η }∇v Li pvi,k ; wS q} “ η ` ›di,k › ` η ϵ2 ` ϵ1 Q2 η 2 }∇v Li pvi,k ; wS q} . ϵ2 ϵ1

(195)

Also for the consensus variate term via Young’s inequality: A

ptq

pt´1q

´2ηγ di,k1 , cptq ´ ci

˛ ¨ › ›2 › ptq › › › › ›› › › › ptq › › ˚ ›di,k › pt´1q › ptq ›2 ‹ ` ϵ3 ›cptq ´ ci › ‚ ď 2ηγ ›di,k › ›cptq ´ ci › ď ηγ ˝ ϵ3

(196)

Substituting (195)(196) into (185), we have ¯ ´1 @ D 1 γ ´ 2η di,k , gi pvi,k ; wS q ` γpc ´ ci q ` ε di,k ď η ` ` ´ 2ε }di,k }2 ϵ ϵ2 ϵ3 ›2 › ›2 ` ˘› 1 2 2 › ` η ϵ2 ` ϵ1 Q η ∇v Li pvi,k ; wS q› ` ηγϵ3 ›c ´ ci › .

We continue to combine (197)(184) into (183), and it results in ´1 ¯ ¯ ˇ “ ‰ ´ 1 γ E }di,k`1 }2 ˇ vi,k ď 1 ` η ` ` ´ 2ε ` 3η 2 ε2 }di,k }2 ϵ ϵ2 ϵ3 ›2 ` 1 ˘ ` ˘› 2 2 › › ` η ϵ2 ` ϵ1 Q η ∇v Li pvi,k ; wS q ` ηγϵ3 ` 3η 2 γ 2 }c ´ ci }2 ` 3η 2 }gi pvi,k ; wS q}2 ` η 2 σ 2 . Recall the client drift definition Et “ 1{KN and unroll the bound to yield

(197)

(198)

řN řK´1 i“1

ptq 2 p0q k“0 E}di,k } , we take the relavant summation with di,k “ 0

N K´1 k´1 ˘ ` ˘ 1 ÿ ÿ ÿ ” ` η ϵ2 ` ϵ1 Q2 η 2 E }∇v Li pvi,m ; wS q}2 ` ηγϵ3 ` 3η 2 γ 2 E }c ´ ci }2 KN i“1 k“1 m“0 ȷk´1´m ˙ ˆ ı„ 1 γ 1 2 2 2 2 2 2 ` ` ´ 2ε ` 3η ε 1`η `3η E }gi pvi,m ; wS q} ` η σ ϵ1 ϵ2 ϵ3 ˜ ȷk´1´m ¸ ˙ ˆ N K´2 K´1 ÿ „ “ ` ˘ 1 ÿ ÿ 1 γ 1 ¨ η ϵ2 ` ϵ1 Q2 η 2 ď ` ` ´ 2ε ` 3η 2 ε2 1`η KN i“1 m“0 k“m`1 ϵ1 ϵ2 ϵ3 ı ` ˘ ¨E }∇v Li pvi,m ; wS q}2 ` ηγϵ3 ` 3η 2 γ 2 E }c ´ ci }2 ` 3η 2 E }gi pvi,m ; wS q}2 ` η 2 σ 2

Et ď

(199)

N K´2 ˘ 1 ÿ ÿ ρK´m´1 ´ 1 ” ` ¨ η ϵ2 ` ϵ1 Q2 η 2 E }∇v Li pvi,m ; wS q}2 KN i“1 m“0 ρ´1 ı ` ˘ ` ηγϵ3 ` 3η 2 γ 2 E }c ´ ci }2 ` 3η 2 E }gi pvi,m ; wS q}2 ` η 2 σ 2 ,

where we have defined ρ “ 1 ` η p1{ϵ1 ` 1{ϵ2 ` γ{ϵ3 ´ 2εq ` 3η 2 ε2 . For small η , it simply holds ρK´m´1 ´ 1 « K ´ m ´ 1, ρ´1

(200)

ptq

rt in (??) leads (199) to and combined with the result E}cptq ´ ci }2 ď 2Ct ` 2C « N K´2 ›2 ` ˘ › 1 ÿ ÿ Et ď pK ´ m ´ 1q η ϵ2 ` ϵ1 Q2 η 2 E›∇v Li pvi,m ; wS q› KN i“1 m“0 ff ` ˘ rt q ` 3η 2 E}gi pvi,m ; wS q}2 ` η 2 σ 2 . ` ηγϵ3 ` 3η 2 γ 2 p2Ct ` 2C

(201)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

ř řK´1 Moreover, since K ´ m ´ 1 ď K , and there are K ´ 1 terms, it leads to K´2 m“0 pK ´ m ´ 1q “ j“1 j “ pK´1qK{2 ď K 2{2. Then, for }gi pvi,m ; wS q}2 , it can be related to }∇v Li pvi,m ; wS q}2 : 2 }gi pvi,m ; wS q}2 ď 2}∇v Li pvi,m ; wS q}2 ` 2ηinner L2 }∇v Li pvi,m ; wS q}2 ` ˘ 2 “ 2 1 ` ηinner L2 }∇v Li pvi,m ; wS q}2 ,

(202)

where we have decomposed gi pvi,m ; wS q with the triangle inequality, and used the bound 2 }gi pvi,m ; wS q ´ ∇v Li pvi,m ; wS q}2 ď ηinner L2 }∇v Li pvi,m ; wS q}2 .

(203)

From (201)(202), we now turn to the evaluation of }∇v Li pvi,m ; wS q}2 , which we aim to relate to the global meta-objective value via the algorithm’s progress. Specifically, we have the following by the standard descent inequality for a L-smooth function: E }∇v Li pvi,m ; wS q}2 ď 2LE rLi pvi,m ; wS q ´ L˚i s

(204)

We can bound E}∇v Li pvi,m ; wS q}2 by starting via the L-smoothness of Li p ¨ ; wS q about the global messenger at the beginning of the round: “ ‰ @ D β E Li pvi,m ; wS q ď Li pv ptq ; wS q ` ∇v Li pv ptq ; wS q, Ervi,m ´ v ptq s ` E}vi,m ´ v ptq }2 . 2

(205)

Then the inner-product term can be bounded simply by Cauchy–Schwarz and Young’s inequalities subsequently: ›› › @ D › ∇v Li pv ptq ; wS q, Ervi,m ´ v ptq s ď ›∇v Li pv ptq ; wS q› ›Ervi,m ´ v ptq s› ›2 L › ›2 ›2 L 1 ›› 1 ›› ∇v Li pv ptq ; wS q› ` ›Ervi,m ´ v ptq s› ď ∇v Li pv ptq ; wS q› ` E}vi,m ´ v ptq }2 , ď 2L 2 2L 2

(206)

where the last step uses Jensen’s inequality. Substituting (206) into (205) and subtracting the minimum value L˚i yields ›2 “ ‰ 1 ›› E Li pvi,m ; wS q ´ L˚i ď Li pv ptq ; wS q ´ L˚i ` ∇v Li pv ptq ; wS q› ` L E}vi,m ´ v ptq }2 . 2L

(207)

Next we bound the gradient term at the global messenger by client heterogeneity. By Assumption A3, we have E}gi pvi,m ; wS q ´ ∇Φpvq}2 ď G2 ` B 2 }∇Φpvq}2 ,

(208)

Decomposing the squared norm therefore gives › ›2 › ›2 E›∇v Li pv ptq ; wS q› “ E›∇v Li pv ptq ; wS q ´ gi pvi,m ; wS q ` gi pvi,m ; wS q ´ ∇Φpv ptq q ` ∇Φpv ptq q› › › ›2 › ›2 › ď 3E›∇v Li pv ptq ; wS q ´ gi pvi,m ; wS q› ` 3E›gi pvi,m ; wS q ´ ∇Φpv ptq q› ` 3E›∇Φpv ptq q›

(209)

2 ď 3L2 ηinner E}∇v Li pv ptq ; wS q}2 ` 3G2 ` p3 ` 3B 2 qE}∇Φpv ptq q}2 . 2 2 ry “ 3G2{p1´3L2 ηinner By letting Cr “ 3p1`B 2 q{p1´3L2 ηinner q and L q, this will result in

E}∇v Li pv ptq ; wS q}2 ď CrE}∇Φpv ptq q}2 ` Lry .

(210)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

We now bound the client gap Li pv ptq ; wS q ´ L˚i that appears in (??). Define the one-step point from the global ptq messenger by zi “ v ptq ´ η∇v Li pv ptq ; wS q. By L-smoothness we have @ L ptq ptq ptq D ptq Li pv ptq ; wS q ď Li pzi ; wS q ` ∇v Li pzi ; wS q, v ptq ´ zi ` }v ptq ´ zi }2 2 › @ D Lη 2 › ptq ptq ›∇v Li pv ptq ; wS q›2 ď Li pzi ; wS q ` η ∇v Li pzi ; wS q, ∇v Li pv ptq ; wS q ` 2 2› › Lη ptq ptq ›∇v Li pv ptq ; wS q›2 ď Li pzi ; wS q ` η}∇v Li pzi ; wS q} }∇v Li pv ptq ; wS q} ` 2 ´ ›2 3Lη ¯›› ptq ptq ď Li pzi ; wS q ` η 1 ` ∇v Li pv ; wS q› 2 ´ ´ 3Lη ¯ r 3Lη ¯ r ptq ď Li pzi ; wS q ` η 1 ` C E}∇Φpv ptq q}2 ` η 1 ` Ly . 2 2

(211)

ř řN ptq ˚ ˚ 1 Because Φpv ˚ q “ 1{N N j“1 Lj pzj pv q; wS q ě {N j“1 Lj , we write the client gap at the one-step point zi as ÿ ` ptq ` ptq ˘ ˘ ` ` ptq ˘˘ Li zi ; wS ´ L˚i “ N Φpv ptq q ´ Lj zj ; wS ´ L˚i ´ N Φpv ptq q ´ Li zi ; wS j‰i ptq

ď N Φpv q ´

(212)

` ptq ˘ Lj zj ; wS ´ L˚i

j‰i

˘ ` ptq Note that each remaining client loss is bounded from below by its own minimum Lj zj ; wS ě L˚j pj ‰ iq. ` ptq ˘ ř ř Therefore ´ j‰i Lj zj ; wS ď ´ j‰i L˚j , and we obtain ÿN ` ptq ˘ ` ˘ Li zi ; wS ´ L˚i ď N Φpv ptq q ´ L˚j ď N Φpv ptq q ´ Φ˚ , j“1

where we have adopted the comparison through

›

˚ ˚ › j Lj ď N Φpv q. As for ∇v Li

(213)

` ptq ˘›2 v ; wS › , we can evaluate it

}∇v Li pv ptq ; wS q}2 “ }∇v Li pv ptq ; wS q ´ ∇Φpv ptq q ` ∇Φpv ptq q}2 ď 2}∇v Li pv ptq ; wS q ´ ∇Φpv ptq q}2 ` 2}∇Φpv ptq q}2 2

ptq

(214)

ď 2N ζ ` 4βpΦpv q ´ Φ q.

By subsituting (211)(213)(214) into (204), we can obtain « N K´2 ´` ´ ¯ › ›2 ˘` ˘ 1 ÿ ÿ pK ´ m ´ 1q D1 2LN ` 4β Φpv ptq q ´ Φ˚ ` 2Lη 1 ` 3Lη Cr E›∇Φpv ptq q› Et ď 2 KN i“1 m“0 ff ´ ¯ ¯ ` ˘ rt q ` η 2 σ 2 ` 2Lη 1 ` 3Lη Lry ` 2N ζ 2 ` 2L2 E}vi,m ´ v ptq }2 ` ηγϵ3 ` 3η 2 γ 2 p2Ct ` 2C 2 « ´` ´ ¯ › ›2 ˘` ˘ K ´1 ď D1 2LN ` 4β Φpv ptq q ´ Φ˚ ` 2Lη 1 ` 3Lη Cr E›∇Φpv ptq q› 2 2 ff ´ ¯ ¯ ˘ ` rt q ` η 2 σ 2 . ` 2Lη 1 ` 3Lη Lry ` 2N ζ 2 ` 2L2 D1 Et ` ηγϵ3 ` 3η 2 γ 2 p2Ct ` 2C 2

(215)

where recall we have defined the positive prefactor ` ˘ ` ˘ 2 D1 “ η ϵ2 ` ϵ1 Q2 η 2 ` 6η 2 1 ` ηinner L2 ,

(216)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

then move Et to the left hand side we can obtain ˙ ˙ ˆ ˆ ¯ ` ˘ K ´1 1 3Lη ˜ ´ ´ ptq ¯ 2 1 Et 1 ´ pK ´ 1qL D ď ´ Φ˚ C Φ v D 2LN ` 4β ` 4βLη 1 ` 2 2 ˙ ˆ ` ˘ K ´1 3Lη r 1 rt q ` Ly D ` pK ´ 1q ηγϵ3 ` 3η 2 γ 2 pCt ` C ¨ 2Lη 1 ` 2 2 K ´1 ` p2N ζ 2 D1 ` η 2 σ 2 q, 2 r and E can be obtained where the positive variables A, B, D ˆ ˙ 3Lη ˜ A “ 2LN ` 4β ` 4βLη 1 ` C, E “ ηγϵ3 ` 3η 2 γ 2 2 ˙ ˆ 3Lη r 1 2 1 2 2 r Ly D . D “ 2N ζ D ` η σ ` 2Lη 1 ` 2

(217)

(218)

µ 1 Lemma 12. Under Assumptions A1-A8 and the step-size restriction τ Kη ď mint 8β , 4pγ 2 β`ε2 β`Ψ{p2τ ηKqq u, the expected one-round progress of the meta-objective satisfies ´ ¯ βΨ ErΦpv pt`1q q ´ Φpv ptq qs ď ´pτ ηKµ ´ ´ τ 2 η 2 Kβpγ 2 ` ε2 qqpΦ v ptq ´ Φ˚ q 2 (219) τ ηKβ 2 τ ηε βΩ βΛ rt q ` βΥ E}wptq ´ w˚ pv ptq q}2 ` βΓ . `p ` ` qEt ` pτ ηγ ` qpCt ` C S S 2 2 2 2 2 2 For simplicity but without loss of generality, the higher-order terms can be omitted when η is sufficiently small,

ErΦpv pt`1q q ´ Φpv ptq qs ď ´

τ ηKµ τ ηKβ 2 τ ηε βΩ pΦpv ptq q ´ Φ˚ q ` p ` ` qEt 2 2 2 2 βΛ rt q ` βΥ E}wptq ´ w˚ pv ptq q}2 ` βΓ . ` pτ ηγ ` qpCt ` C S S 2 2 2

(220)

Proof. By taking the Talor expansion on ErΦpv pt`1q qs at v ptq , we combine the β´smoothness of Φ, it leads to β (221) ErΦpv pt`1q qs ď Φpv ptq q ` Ex∇Φpv ptq q, v pt`1q ´ v ptq y ` E}v pt`1q ´ v ptq }2 . 2 Note exact reconstruction of the client displacements the server update takes the form v pt`1q “ v ptq ´ τ η ¨

K´1 ´ ¯ ´ ¯ ´ ¯ı 1 ÿ ÿ ” ptq ptq ptq ptq ĝi,k vi,k , wS ` γ cptq ´ ci ` ε vi,k ´ v ptq , |St | iPS k“0

(222)

which we substitute into the cross term in (221) and it results in N K´1 ´ ¯ ´ ¯E 1 ÿ ÿ A ptq ptq Ex∇Φpv ptq q, v pt`1q ´ v ptq y “ ´τ η ¨ E ∇Φ v ptq , ĝi,k vi,k , wS N i“1 k“0 loooooooooooooooooooooooooooooooomoooooooooooooooooooooooooooooooon :“T1 N K´1 1 ÿ ÿ

N K´1 ´ ¯ E ´ ¯ E 1 ÿ ÿ A ptq ptq ptq ptq ` ´τ ηγ ¨ E ∇Φ v , c ´ ci ` ´τ ηε ¨ E ∇Φ v ptq , vi,k ´ v ptq N i“1 k“0 N i“1 k“0 looooooooooooooooooooooooooooomooooooooooooooooooooooooooooon looooooooooooooooooooooooooooomooooooooooooooooooooooooooooon

(223)

:“T2

:“T3

For T1 , we decompose each practical meta-gradient 1 ÿ ptq ptq ptq ptq T1 “ ´τ η Erx∇Φpv ptq q, gi,k pv ptq , wS qy ` x∇Φpv ptq q, gi,k pvi,k , wS q ´ gi,k pv ptq , wS qys. N i,k

(224)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

The first average exactly equals }∇Φpv ptq q}2 , and the second average can be evaluated via the joint β ptq ptq ptq ptq smoothness, namely }gi,k pvi,k , wS q ´ gi,k pv ptq , wS q} ď β}vi,k ´ v ptq }, hence we have › › › ´ ¯›2 ´ ¯› 1 ÿ ›› › › › › ptq › T1 ď ´τ ηK ›∇Φ v ptq › ` τ η ¨ ›∇Φ v ptq › ¨ β ›vi,k ´ v ptq › N i,k ›2 › ´ ¯›2 τ ηβ 2 1 ÿ › ´ ¯›2 τ ηK › › › › ptq › › › (225) E ›vi,k ´ v ptq › ¨ ď ´τ ηK ›∇Φ v ptq › ` ›∇Φ v ptq › ` 2 2 N i,k ¯ τ ηKβ 2 ´ ¯›2 τ ηKβ 2 ´ ´ ¯ τ ηK ›› › “´ Et ď ´τ ηKµ Φ v ptq ´ Φ˚ ` Et , ›∇Φ v ptq › ` 2 2 2 where we have used the Polyak-Łojasiewicz inequality }∇Φpv ptq q}2 ě 2µpΦpv ptq q ´ Φ˚ q. For T2 , we first apply Young’s inequality ˇ A › Eˇ pτ ηγq2 › ´ ¯ ´ ¯›2 1 › › ˇ › › ptq ˇ ptq ›2 (226) ˇE ´τ ηγ ¨ ∇Φ v ptq , cptq ´ ci ˇ ď ›∇Φ v ptq › ` E ›cptq ´ ci › , 2 2 ptq

and we take the averaging over the N clients and the K local steps and use the bound E}cptq ´ ci }2 ď rt q leads to 2pCt ` C ´ ¯›2 ´ ¯ ´ ´ ¯ ¯ ´ ¯ pτ ηγq2 K ›› › rt ď τ 2 η 2 γ 2 Kβ Φ v ptq ´ Φ˚ ` τ ηγ Ct ` C rt , T2 ď ›∇Φ v ptq › ` τ ηγ Ct ` C 2 (227) ptq 2 ptq ˚ where we have used the smoothness result }∇Φpv q} ď 2βpΦpv q ´ Φ q. For T3 we apply the identical rewriting and Young’s inequality: ˇ A ›2 ´ ¯›2 1 › Eˇ pτ ηεq2 › ´ ¯ › › › › ptq ˇ ˇ ptq (228) ›∇Φ v ptq › ` E ›vi,k ´ v ptq › . ˇE ´τ ηε ¨ ∇Φ v ptq , vi,k ´ v ptq ˇ ď 2 2 Subsequently, we take the averaging over the N clients and the K local steps, then using the smoothness result, it leads to ´ ¯›2 τ ηε ´ ´ ¯ ¯ τ ηε pτ ηεq2 K ›› › T3 ď Et ď τ 2 η 2 ε2 Kβ Φ v ptq ´ Φ˚ ` Et . (229) ›∇Φ v ptq › ` 2 2 2 Therefore, by substituting the Lemma 8, T1 in (225), T2 in (227) and T3 in (229) into (221), we can obtain the Lemma result (219). Lemma 13 (Lyapunov Evolution). Consider the bi-level CIDERS algorithm under Assumptions A1–A8. We design the Lyapunov function as follows: rt ` Qw4 ηJt , (230) V ptq “ p1 ´ 4a2 q E}z ptq ´ v ˚ }2 ` ω1 Et ` Qw2 ηCt ` Qw3 η C “ ‰ ptq where Jt “ E Φpv ptq q ´ Φ˚ ` E}wS ´ wS˚ pv ptq q}2 is the convergence quantity of interest, z ptq is the virtual messenger sequence defined by ˘ 1 ´ τ ` ptq z ptq “ v ptq ` v ´ v pt´1q , (231) τ and Qw1 , Qw2 , Qw3 , Qw4 are the positive weights that satisfies the following conditions „ ˆ ˙ ȷ p1 ´ 2aqK 1 2 Qw3 ă , Qw4 p1 ´ ρR q ą 6β Qw2 p1 ´ pαq 1 ` ` 2pα . (232) ε βκg1

Then the one-round progress of the Lyapunov function satisfies 1 V pt`1q ď V ptq ´ δΦ η ¨ Jt ` Dtot , 1 where Dtot is a postive value and δΦ satisfies the following " „ ˆ ˙ ȷ* 1 g 2 δΦ “ min 2p1 ´ 2aqK ´ 2Qw3 βκ1 , Qw4 p1 ´ ρR q ´ 6β Qw2 p1 ´ pαq 1 ` ` 2pα . ε

(233)

(234)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

Proof. Our main target is to evaluate each term’s evolution. We first expand the Lyapunov difference as ` ˘ ω1 pEt`1 ´ Et q V pt`1q ´ V ptq “ loooooooooooooooooooooooooooomoooooooooooooooooooooooooooon p1 ´ 4a2 q E}z pt`1q ´ v ˚ }2 ´ E}z ptq ´ v ˚ }2 ` looooooomooooooon V2

(235)

rt`1 ´ C rt q ` ω4 pJt`1 ´ Jt q . `ω ω3 pC 2 pCt`1 ´ Ct q ` looooooomooooooon looooooomooooooon looooooomooooooon V3

It can be seen the evaluation of the Lyapunnov function V evolution depends on each term’s evolution, hence rt and Φ, hence we we derive them term by term. Notice we have derived the corresponding evolution Et , Ct , C ¯ ptq implies z pt`1q ´ z ptq “ ´Kη ∆ ¯ ptq , first foucs on V1 . Specifically, the server update v pt`1q “ v ptq ´ τ Kη ∆ ¯ ptq and recall ∆i,t where we define ∆ ÿ ¯ ptq “ 1 ∆ ∆i,t , |St | iPS t

K´1

´ ¯ ´ ¯ 1 ÿ ptq ptq ptq ptq ∆i,t “ gpi,k vi,k , wS ` γ cptq ´ ci ` rprox,i K k“0

(236)

› ›2 which we have used for the expansion of E ›z pt`1q ´ v ˚ › and it leads to ¯ ptq }2 E}z pt`1q ´ v ˚ }2 “ E}z ptq ´ v ˚ ´ Kη ∆ ¯ ptq y ` K 2 η 2 E}∆ ¯ ptq }2 . “ E}z ptq ´ v ˚ }2 ´ 2Kη Exz ptq ´ v ˚ , ∆

(237)

¯ ptq y, we decompose it as follows For the inner product term Exz ptq ´ v ˚ , ∆ ¯ ptq y “ Exv ptq ´ v ˚ , ∆ ¯ ptq y ` 1 ´ τ Exv ptq ´ v pt´1q , ∆ ¯ ptq y Exz ptq ´ v ˚ , ∆ τ A ´ ¯E E 1´τ 1 A ptq ¯ ptq y, “ E v ptq ´ v ˚ , ∇Φ v ptq ` E v ´ v ˚ , bptq ` Exv ptq ´ v pt´1q , ∆ ηK τ

where we have rewritten the following for the decomposition in the first equality ´ ¯ 1´τ ´ ¯ z ptq ´ v ˚ “ v ptq ´ v ˚ ` v ptq ´ v pt´1q τ and for the second equality we have used ı ´ ¯ ” ¯ ptq | v ptq “ ∇Φ v ptq ` 1 bptq , E ∆ ηK

(238)

(239)

(240)

moreover, we have incorporated the bias that measures the averaged local direction deviation with the true meta-gradient “ ptq ˇ ptq ‰ ¯ ˇv (241) bptq :“ ηK E ∆ ´ ηK∇Φpv ptq q, and it can be expanded via bptq “

N K´1 ˇ ‰ η ÿ ÿ “ ptq ptq ideal ptq E gpi,k pvi,k , wS q ´ gri,k pv , wS˚ pv ptq qq ˇ v ptq N i“1 k“0 looooooooooooooooooooooooooooooooooooomooooooooooooooooooooooooooooooooooooon TI : local-to-global meta-gradient deviation N 1 ÿ “ ptq ˇˇ ptq ‰ ` ` ηK ¨ E rprox,i v , N i“1 loooooooooooooooomoooooooooooooooon TII : consensus-variate deviation

(242)

` ˘ ηKγ cptq ´ c̄ptq looooooooomooooooooon

TIII : proximal residual

řN

ptq

where c̄ptq “ 1{N i“1 ci . Recall the joint smoothness of each client loss (Assumption A1) the map pv, wS q ÞÑ ˘ ` ∇v Li v ´ ηinner ∇v Li pv, wS q, wS . Consequently › ›2 ptq ptq ideal ptq E›gpi,k pvi,k , wS q ´ gri,k pv , wS˚ pv ptq qq› (243) ptq ptq ď 2β 2 E}vi,k ´ v ptq }2 ` 2β 2 E}wS ´ wS˚ pv ptq q}2 .

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

Averaging over the K local steps and the N clients, and using the definition of client drift Et , it produces › ›2 N K´1 › ÿ ˘›› 1 ÿ` ptq ptq 2 2 › 1 ideal ptq ˚ ptq gpi,k pvi,k , wS q ´ gri,k pv , wS pv qq › E}TI } “ η E› ›N › K k“0 (244) i“1 ptq

ď 2η 2 β 2 Et ` 2η 2 β 2 E}wS ´ wS˚ pv ptq q}, rt q, For the averaged consensus-variate deviation term TII , it can be simply derived E}cptq ´ c̄ptq }2 ď 2pCt ` C which subsequently leads to ›2 › ¯ ´ › › rt . (245) E}TII }2 “ η 2 K 2 γ 2 E ›cptq ´ c̄ptq › ď 2η 2 K 2 γ 2 Ct ` C ptq

For the proximal residual TIII , we can apply the assumption E}rprox, i }2 ď ε22 , which leads to E}TIII }2 ď η 2 K 2 ε22 .

(246)

Combining the inequalities (244)–(246) gives the full bound ` ˘ ` ˘ ptq rt q ` 3η 2 K 2 ε2 E}bptq }2 ď 3 2η 2 β 2 Et ` 2η 2 β 2 E}wS ´ wS˚ pv ptq q}2 ` 3 2η 2 K 2 γ 2 pCt ` C 2 rt q ` 6η 2 β 2 E}wptq ´ w˚ pv ptq q}2 ` 3η 2 K 2 ε2 . “ 6η 2 β 2 Et ` 6η 2 K 2 γ 2 pCt ` C S 2 S

(247)

Hence, the third term in (238) can be evaluated via ›2 ›2 A E a › 1 ›› ››2 a ›› › › › E v ptq ´ v ˚ , bptq ď E ›v ptq ´ v ˚ › ` E ›bptq › ď E ›v ptq ´ v ˚ › ` 6η 2 β 2 Et 2 2a 2 (248) ¯ 3η 2 β 2 › ´ ¯›2 3η 2 K 2 ε2 3η 2 β 2 3η 2 K 2 γ 2 ´ › › ptq 2 rt ` ` Et ` Ct ` C E ›wS ´ wS˚ v ptq › ` . a a a 2a ¯ ptq y in (238), assumption A6 together with the definition of the consensus For the term Exv ptq ´ v pt´1q , ∆ variates and the proximal residual gives the uniform bound }p gi,k } ď G,

pt´1q

}cpt´1q ´ ci

} ď 2G,

pt´1q

}rprox,i } ď ε2 ď G,

(249)

where without loss of generality the residual bound can be absorbed into the same constant G). Consequently, according to the definitions of ∆i,t in (236), we simply obtain }∆i,t´1 } ď G ` γ ¨ 2G ` G “ Gp1 ` 2γq,

¯ pt´1q }2 ď G2 p1 ` 2γq2 . E}∆

(250)

This leads to third second term in (238) to become ˇ Eˇˇ ˇ 1 ´ τ A ptq pt´1q ptq ¯ ˇ ˇ ď p1 ´ τ qKηG2 p1 ` 2γq2 . (251) ,∆ ˇ τ E v ´v ˇ “ ‰ Substituting (248)(251) and the convexity of Φ that Exv ptq ´ v ˚ , ∇Φpv ptq qy ě E Φpv ptq q ´ Φ˚ into (238), we can obtain 2 2 2 2 2 “ ‰ rt q ¯ ptq y ď ´2Kη E Φpv ptq q ´ Φ˚ ` a E}v ptq ´ v ˚ }2 ` 6η β Et ` 6η K γ pCt ` C ´2Kη Exz ptq ´ v ˚ , ∆ a a 6η 2 β 2 3η 2 K 2 ε22 ptq ` E}wS ´ wS˚ pv ptq q}2 ` ` 2p1 ´ τ qK 2 η 2 G2 p1 ` 2γq2 . a a (252) ptq ˚ 2 Notice the term E}v ´ v } satisifies the elementary expansion that relates the true messenger distance to the virtual-messenger distance: ˆ ˙ › ›2 › ›2 ›2 1 ´ τ 2 ›› ptq › ptq › ptq › ˚› ˚› E ›v ´ v › ď 2E ›z ´ v › ` 2 E ›v ´ v pt´1q › τ (253) › ›2 › ›2 › ›2 › ptq › › › ptq ˚ 2 2 2 pt´1q › ˚› 2 2 2 2 2 ¯ ď 2E ›z ´ v › ` 2p1 ´ τ qK η E ›∆ › “ 2E ›z ´ v › ` 2p1 ´ τ q K η G p1 ` 2γq

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

¯ ptq }2 “ For the last term in (237), the Server Messenger Update lemma together with the identity K 2 η 2 E}∆ 1 pt`1q ´ v ptq }2 yields τ 2 E}v “ ‰ rt q ` Υ{τ 2 E}wptq ´ w˚ pv ptq q}2 ` Ω Et ` Ψ E Φpv ptq q ´ Φ˚ ` Γ , ¯ ptq }2 ď Λ pCt ` C (254) K 2 η 2 E}∆ S S 2 2 2 τ τ τ τ2 and multiplying by p1 ´ 4a2 q yields V1 ´ ¯ ¯ ptq y ` K 2 η 2 E}∆ ¯ ptq }2 . V1 “ p1 ´ 4a2 q ´2Kη Exz ptq ´ v ˚ , ∆ (255)

By substituting (252)(253)(254) into (237) we have ˜

¸ “ ‰ Ψ E}z pt`1q ´ v ˚ }2 ´ p1 ` 2aqE}z ptq ´ v ˚ }2 ď ´2Kη ` 2 E Φpv ptq q ´ Φ˚ τ ˜ ¸ ˜ ¸ ˜ ¸ 2β2 6η 2 β 2 Ω 6η 2 K 2 γ 2 Λ 6η Υ ptq rt q ` ` ` 2 Et ` ` 2 pCt ` C ` 2 E}wS ´ wS˚ pv ptq q}2 a τ a τ a τ

3η 2 K 2 ε22 Γ ` 2p1 ´ τ qK 2 η 2 G2 p1 ` 2γq2 ` 2ap1 ´ τ q2 K 2 η 2 G2 p1 ` 2γq2 ` 2 . a τ 2 Since a ă 1{2, we simply have 1 ´ 2a ą 1 ´ 4a , this leads to ˜ ¸ ˜ ¸ 2β2 “ ‰ Ψ 6η Ω V1 ď p1 ´ 2aq ´2Kη ` 2 E Φpv ptq q ´ Φ˚ ` p1 ´ 2aq ` 2 Et τ a τ ˜ ¸ ˜ ¸ 2 2 6η 2 K 2 γ 2 Λ rt q ` p1 ´ 2aq 6η β ` Υ E}wptq ´ w˚ pv ptq q}2 ` p1 ´ 2aq ` 2 pCt ` C S S a τ a τ2 ˜ ¸ 3η 2 K 2 ε22 Γ ` p1 ´ 2aq ` 2p1 ´ τ qK 2 η 2 G2 p1 ` 2γq2 ` 2ap1 ´ τ q2 K 2 η 2 G2 p1 ` 2γq2 ` 2 , a τ

(256)

which is further simplified to “ ‰ rt q ` Dz E}wptq ´ w˚ pv ptq q}2 ` D1 , V1 ď ´Az E Φpv ptq q ´ Φ˚ ` Bz Et ` Cz pCt ` C S z S and we have defined ˙ ˆ 2 2 ˙ ˆ 6η β Ω Ψ ` 2 , Az “ p1 ´ 2aq 2Kη ´ 2 , Bz “ p1 ´ 2aq τ a τ ˆ 2 2 2 ˙ ˆ 2 2 ˙ 6η K γ Λ 6η β Υ Cz “ p1 ´ 2aq ` 2 , Dz “ p1 ´ 2aq ` 2 a τ a τ ˆ 2 2 2 ˙ 3η K ε2 Γ 2 2 2 2 2 2 2 2 2 1 Dz “ p1 ´ 2aq ` 2p1 ´ τ qK η G p1 ` 2γq ` 2ap1 ´ τ q K η G p1 ` 2γq ` 2 . a τ For V2 , we can use the client drift Lemma 11 for Et`1 ´ Et : ” ´ ¯ ı ´ ¯ ) pK ´ 1q{2 ! 1 pt`1q ˚ rt`1 ` D r ´ Et Et`1 ´ Et ď D A E Φ v ´ Φ ` 2E C ` C t`1 1 ´ KL2 D1 pK ´ 1q{2 1 ␣ τ ηKµ τ ηKβ 2 τ ηε βΩ ptq ˚ ď D A p1 ´ q E rΦpv q ´ Φ s ` p ` ` qEt 1 ´ KL2 D1 2 2 2 2 ( βΛ rt q ` βΥ Rt ` βΓ ` pτ ηγ ` qpCt ` C 2 2 2 ␣ ` ˘ pK ´ 1q{2 1 rt ` pκE ` κg Ωq Et ` D A ¨ 2E pρC ` ρg q Ct ` κCr ` ρ̃g C 2 1 ´ KL2 D1 ” ´ ¯ ı ` ˘( pK ´ 1q{2 r rg3 q E Φ v ptq ´ Φ˚ ` κ0 ` κ r4g ` ` pκw ` κg2 Υq Rt ` pκΦ ` κ ¨ D ´ Et , 1 ´ KL2 D1

(257)

(258)

(259)

(260)

where we have used the Lemma 218 for evaluating the term pΦpv pt`1q q ´ Φ˚ q and the Lemmas 10 and 9 for rt`1 q. Moreover, we have defined Rt “ E}wptq ´w˚ pv ptq q}2 and used the Lemma 5 evaluating the term pCt`1 `C S S

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

2 2 p1´µη E for its evolution ρR Rt `DR , with the definitions of ρR “ 2p1´µηKD qE and DR “ ηKD σKD {µ`8Rw KD q . For brevity, we let κkl “ pK´1q{2{p1´KL2 D1 q, then it leads to ı ” ´ ¯ rr 1 (261) Et`1 ´ Et ď AE E Φ v ptq ´ Φ˚ ` pρE ´ 1q Et ` CEC Ct ` CEC C t ` DE Rt ` DE ,

rt and Et are provided where the coefficients for ErΦpv ptq q ´ Φ˚ s, Rt , Ct , C „ˆ ȷ ˆ ˙ ˙ τ ηKµ τ ηKβ 2 τ ηε βΩ g g 1 1 r3 q , ρE “ κkl D A AE “ κkl D A 1 ´ ` 2E pκΦ ` κ ` ` ` 2E pκE ` κ2 Ωq 2 2 2 2 ˙ȷ ˆ ˙ „ ˆ βΛ βΥ g C 1 1 ` 2E pκw ` κ2 Υq , CE “ κkl D A τ ηγ ` ` 2E pρC ` ρg q , DE “ κkl D A 2 2 ˙ „ ˆ ˆ ˙ ȷ ˘ ` ` ˘ βΓ βΛ r 1 1 C 1 4 r rg ` D . CE “ κkl D A τ ηγ ` ` 2E κCr ` ρrg , DE “ κkl D A ` 2E κ0 ` κ 2 2 (262) Next, we continue to derive V3 , which can be directly obtained from Lemma 10 that “ ‰ rt ` ω2 κE Et ` ω2 κw Rt ` ω2 κΦ E Φpv ptq q ´ Φ˚ ` ω2 κ0 . V3 ď ω2 pρC ´ 1qCt ` ω2 κCr C

(263)

Similarly, V4 can be directly derived via Lemma 9 that “ ‰ rt ` ω3 ρg Ct ` ω3 κg ΥRt ` ω3 κg ΩEt ` ω3 κ rg3 E Φpv ptq q ´ Φ˚ ` ω3 κ r4g . V4 ď ω3 pr ρg ´ 1qC 2 2

For V5 , Lemmas 218 and 5 can be adopted and it directly leads to ˆ ˙ ” ´ ¯ ˆ ˙ ı τ ηKµ τ ηKβ 2 τ ηε βΩ ptq ˚ V5 “ω4 pJt`1 ´ Jt q ď ω4 ´ E Φ v ´ Φ ` ω4 ` ` Et 2 2 2 2 ˙ ˙ ˆ ˙ ˆ ˆ ¯ βΥ βΓ βΛ ´ r Ct ` Ct ` ω 4 ` ρR ´ 1 Rt ` ω4 ` DR . ` ω4 τ ηγ ` 2 2 2 Combine pV1 , V2 , V3 , V4 , V5 q, we can have the Lyapunov evolution ” ´ ¯ ı r r C C 1 V pt`1q ´ V ptq ď ´Atot E Φ v ptq ´ Φ˚ ´ Btot Et ´ Ctot Ct ´ Ctot Ct ´ Dtot Rt ` Dtot ,

(264)

(265)

(266)

C , C C , D , D1 where we have defined Atot , Btot , Ctot tot tot tot r

τ ηKµ « r2p1 ´ 2aqK ´ 2Qw3 βκg1 s η 2 ˆ ˙ τ ηKβ 2 τ ηε βΩ g Btot “ ω1 ´ Bz ´ ω1 ρE ´ ω2 κE ´ ω3 κ2 Ω ´ ω4 ` ` 2 2 2 ˘ τ` 2 « ω1 ´ 2pαβ Qw2 η ´ Qw4 ¨ Kβ 2 ` ε η 2 ˆ ˙ 2 βΛ C Ctot “ ω2 ´ Cz ´ ω1 CEC ´ ω2 ρC ´ ω3 ρg ´ ω4 τ ηγ ` 2 rg3 ` ω4 ¨ Atot “ Az ´ ω1 AE ´ ω2 κΦ ´ ω3 κ

« ηQw2 p1 ´ p1 ´ pαqp1 ` εqq ´ Qw4 τ γη 2 , ˆ ˙ βΛ r r C C Ctot “ ω3 ´ Cz ´ ω1 CE ´ ω2 κCr ´ ω3 ρ̃g ´ ω4 τ ηγ ` « ω3 ´ Qw4 τ γη 2 , 2 ˆ ˙ βΥ g ` ρR Dtot “ ω4 ´ Dz ´ ω1 DE ´ ω2 κw ´ ω3 κ2 Υ ´ ω4 2 " „ ˆ ˙ ȷ* 1 2 « Qw4 p1 ´ ρR q ´ 6β Qw2 p1 ´ pαq 1 ` ` 2pα η ε ˆ ˙ βΓ 1 1 r4g ` ω4 Dtot “ Dz1 ` ω1 DE ` ω2 κ0 ` ω3 κ ` DR « Cη η ` Cη2 η 2 ` Cη3 η 3 2

(267)

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

1 , we have defined C , C 2 , C 3 as follows: where especially for Dtot η η η ! ´ ¯) p0q Cη “ ω1 pK ´ 1qp2N ζ 2 ϵ2 q ` Qw2 κ0 ` Qw3 κg3 ` Qw4 DR ı ´ ¯ ” ` ˘ p0q 2 L2 ` σ 2 ` 2LLry ϵ2 Cη2 “ ω1 pK ´ 1q 2ϵ2 A0 γϵ3 κ0 ` κg3 ` 12N ζ 2 1 ` ηinner ˆ ˙ 3K 2 ε22 Γ2 2 2 2 2 2 2 2 ` p1 ´ 2aq ` 2p1 ´ τ qK G p1 ` 2γq ` 2ap1 ´ τ q K G p1 ` 2γq ` 2 a τ ´ ¯ ” ` ` ˘ ˘ p0q g 2 2 Cη3 “ ω1 pK ´ 1q ϵ2 A0 ¨ P2 ` ϵ2 A1 ` 6 1 ` ηinner L A0 ¨ 2γϵ3 κ0 ` κ3 ˙ ı ˆ ` ˘ β p1q g 2 2 2 2 2r r `2N ζ ϵ1 Q ` 12LLy 1 ` ηinner L ` 3L Ly ϵ2 ` Qw2 κ0 ` Qw3 κ2 ` Qw4 Γ2 , 2

(268)

and we have set the following definitions for brevity: ´ ¯ β ´ ¯ ` ˘ p0q p2q 2 D1 “ ηϵ2 ` η 2 D21 , D21 “ ϵ1 Q2 ` 6 1 ` ηinner L2 , P2 “ 6γ 2 κ0 ` κg3 ` Γ2 ` 2γϵ3 κ0 ` κg2 Γ2 , 2 2 2 ˜ A “ A0 ` ηA1 ` η A2 , A0 “ 2LN ` 4β, A1 “ 4βLC, Γ “ η Γ2 with ˆ ˙ ˆ ˙ ` ` ˘ 2˘ 12N τ 2 K 2 6σ 2 12N τ 2 K 2 B 2 2 2 2 2 2 2 Γ2 “ ` 12G ` 2ε2 ` ` 2τ K 32ηinner B 2 G2 ` 40 ` 16L2 ηinner ζ S K S ˆ ˙ 1 p1q p1q p0q κ0 “ κ0 ` η 2 κ0 Γ2 with κ0 “ p1 ´ pαq 1 ` β 2 ` 2pαβ 2 and ε ˙ ˆ 2 ˙ ˆ 2σ 1 p0q 2 2 16β 2 Rw ` pα ` 32β 2 Rw . κ0 “ p1 ´ pαq 1 ` ε K (269) r C C It can be verified simply from (267) that when η is sufficiently small, all pBtot , Ctot , Ctot q ą 0. Moreover, 1 , which is exactly (233) in (234) holds. Therefore, (266) can be simplified to V pt`1q ď V ptq ´ δΦ η ¨ Jt ` Dtot the lemma.

Theorem 14 (Ergodic Convergence). Under Assumptions A1–A8, let the weights of the Lyapunov function rt ` Qw4 ηJt to be positive pω1 , Qw2 , Qw3 , Qw4 q ą 0, V ptq “ p1 ´ 4a2 qE}z ptq ´ v ˚ }2 ` ω1 Et ` Qw2 ηCt ` Qw3 η C g with Qw2 , Qw3 , Qw4 obeying Qw3 ă p1´2aqK{βκ1 , and Qw4 p1 ´ ρR q ą 6β 2 Qw2 rp1 ´ pαqp1 ` 1{εq ` 2pαs. Moreover, the upper-level step-size be chosen as d p0q ›2 ` ˘ ›› V0 › p0q η“ , where V0 “ 1 ´ 4a2 E ›z p0q ´ v ˚ › ` ω1 E0 , (270) Cη 2 T ptq

Then the ergodic average of the joint gap Jt “ ErΦpv ptq q ´ Φ˚ s ` E}wS ´ wS˚ pv ptq q}2 satisfies d p0q p0q p0q Tÿ ´1 ´ ¯ Cη 3 V 0 Cη V 1 2 V 0 Cη 2 Jt ď ` ` 1 ` ` O T ´3{2 T t“0 δΦ δΦ T δΦ T δ Φ Cη 2 T

(271)

Proof. Summing the inequality (233) from t “ 0 to T ´ 1 yields p0q p0q T ´1 1 ` ˘ Cη 2 Cη 3 2 Cη V0 V1 1 ÿ V p0q Dtot Jt ď ` “ ` ` ` η` η ` O η3 , T t“0 δΦ ηT δΦ η δΦ ηT δΦ T δΦ δΦ δΦ

where we have defined ›2 ` ˘ ›› › p0q V0 “ 1 ´ 4a2 E ›z p0q ´ v ˚ › ` ω1 E0 , p0q

p0q

r0 ` Qw4 J0 . “ Qw2 C0 ` Qw3 C

(272)

(273)

We consider the function Hpηq “ V0 {pδΦ ηT q ` Cη2{δΦ , by minimizing it we can obtain the step size η ˚ “ b p0q V0 {Cη2 T exactly as (270) and substituting it into (272) we can obtain the convergence result in (271).

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, SEPT. 2026

N. Discussion of the Convergence Result

The convergence result in (271) provides several important insights into the behavior of CIDERS. ´1 a) Convergence rate.: The leading term in (271) is OpT {2 q, which matches the standard sublinear rate for stochastic nonconvex optimization under heterogeneous settings [1]. The first term Cη {δΦ represents the asymptotic neighborhood of convergence, which is assembled from the local gradient heterogeneity, lowerlevel gap, and the proximal second-moment. It also confirms that larger client participation improves the ? convergence rate by reducing the variance of the aggregated update. The step-size choice η “ Op1{ T q is standard for stochastic gradient methods and balances the initial transient and the asymptotic variance [2]. In Lemma 12, the global learning rate τ satisfies * " µ 1 , (274) , τ Kη ď min 8β 4pγ 2 β ` ε2 β ` Ψ{p2τ ηKqq which is the stability condition for smooth optimization with momentum-like updates. b) Effect of local step K.: While the local step K can reduce communication, it may inflate client drift. In Lemma 11, the prefactor p1 ´ KL2 D1 q´1 remains positive for K ă 1{pL2 D1 q, with D1 “ Opηq. Beyond this threshold, the Lyapunov descent inequality fails, which justifies the practical guideline that Kη must be sufficiently small to prevent divergence. This aligns with the client-drift phenomenon identified in prior federated learning analyses [3], [4]. ptq c) Effect of lower-level distillation.: The lower-level approximation error Rt :“ E}wS ´wS˚ pv ptq q}2 contracts at rate ρR “ 2p1´µηKD qE from Lemma 5. To ensure that this error does not dominate the upper-level progress, the number of KD steps E should satisfy E Á pµηKD q´1 log pηq´1 ,

(275)

This condition formalizes the requirement that the student backbone is sufficiently well aligned with the current messenger before the next round of client updates begins. ? d) Comparison with baseline federated learning.: The convergence rate Op1`1{ T q matches the best-known rates for federated learning with client heterogeneity [3], [6], [7]. Compared to standard FL, which suffers from biased convergence due to client drift [4], CIDERS retains a comparable asymptotic neighborhood arising from the bilevel structure, i.e., the lower-level approximation error Rt is the additional sources of bias, thus the lower-level updates should be carefully controlled. Importantly, the consensus-variate correction effectively mitigates the heterogeneity from the asymptotic bias. This aligns with the qualitative finding in Section V-B r and ci « gri . that the consensus variates reduce the heterogeneity gap when c « G R EFERENCES [1] L. Bottou, F. E. Curtis, and J. Nocedal, “Optimization methods for large-scale machine learning,” SIAM Review, vol. 60, no. 2, pp. 223–311, 2018. [2] A. Khaled, K. Mishchenko, and P. Richtárik, “Tighter theory for local SGD on identical and heterogeneous data,” in Proc. Int. Conf. Artif. Intell. Stat. (AISTATS), 2020. [3] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. U. Stich, and A. T. Suresh, “SCAFFOLD: Stochastic controlled averaging for federated learning,” in Proc. Int. Conf. Mach. Learn. (ICML), 2020. [4] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of FedAvg on non-IID data,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2020. [5] N. Parikh and S. Boyd, “Proximal algorithms,” Found. Trends Optim., vol. 1, no. 3, pp. 127–239, 2014. [6] D. A. E. Acar, Y. Zhao, R. Matas, M. Mattina, P. Whatmough, and V. Saligrama, “Federated learning based on dynamic regularization,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2021. [7] S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečný, S. Kumar, and H. B. McMahan, “Adaptive federated optimization,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2021. [8] S. Ghadimi and M. Wang, “Approximation methods for bilevel programming,” arXiv preprint arXiv:1802.02246, 2018. [9] T. Chen, Y. Sun, and W. Yin, “Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2021.

Record · ID 919341 · SHA-256 b1ee5f468c9b6cf2
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.