arXiv CSopen access

RegionFed: Federated Learning for Personalized Query Understanding in Heterogeneous Retail Environments

· arxiv_cs

arXiv CS · Papers · License: Open Access

machine learning, deep learning, neural networks

arXiv:2609.05403v1 [cs.LG] 4 Sep 2026

RegionFed: Federated Learning for Personalized Query Understanding in Heterogeneous Retail Environments Quoc H. Nguyen∗ Walmart Global Tech Sunnyvale California, USA, [email protected] Abhijeet Phatak Walmart Global Tech Sunnyvale California, USA,

Ali Lafzi Walmart Global Tech Sunnyvale California, USA,

Siddharth Pratap Singh Walmart Global Tech Sunnyvale California, USA,

Yogananda Domlur Seetharama Walmart Global Tech Sunnyvale California, USA,

Rohit Upadhyay Walmart Global Tech Sunnyvale California, USA,

Chittaranjan Tripathy Walmart Global Tech Sunnyvale California, USA,

Abstract Retail search systems serve diverse geographic regions with distinct query patterns, vocabularies, and product preferences, creating significant data heterogeneity that challenges both privacy-preserving training and model personalization. Federated learning offers a natural solution for privacy, but standard FL methods produce global models that sacrifice regional performance, while existing personalized FL approaches operate at the parameter level and catastrophically collapse on modern transformers (below 10% accuracy on T5) due to tied embeddings and LayerNorm interactions. We introduce RegionFed, an architecture-robust federated learning framework that sidesteps this failure by operating entirely at the gradient level. RegionFed uses the ℓ2 conflict between regional and global gradients as a unified signal that (i) diagnoses heterogeneity, (ii) routes each region to the cheapest sufficient personalization strategy, and (iii) adaptively controls personalization strength. Because it treats models as differentiable black boxes, RegionFed deploys on T5-Small, T5-3B, RoBERTa, and CNN with zero code changes, providing large gains on transformers (where parameter-level methods collapse) and consistent improvements on CNNs. Across three public datasets (Amazon ESCI, Amazon Reviews, LEAF-FEMNIST) and four architectures, RegionFed-Meta achieves 92.27%, closing the gap to the privacy-violating centralized upper bound (Centralized + Regional Weighting: 92.04%, √ ∆=0.23pp, within 1σ) while providing (ϵ≈0.60)-differential privacy and O(1/ T ) convergence.

∗ Use footnote for providing further information about author (webpage, alternative address)—not for acknowledging funding agencies.

Introduction

Large-scale retail platforms serve hundreds of millions of customers across diverse geographical regions, each with distinct shopping behaviors, vocabularies, and product preferences. A critical component of the user experience is the search system’s ability to accurately interpret and contextualize queries. Yet this creates a fundamental tension: effective personalization requires understanding regional patterns, while privacy regulations (GDPR, CCPA) and data sovereignty concerns prohibit centralizing sensitive query data. Federated learning (FL) [McMahan et al., 2017] offers a natural paradigm for privacy-preserving distributed training, but deploying FL in heterogeneous retail environments introduces challenges that existing methods, including recent clustered [Ghosh et al., 2020, Sattler et al., 2021] and personalized approaches [T. Dinh et al., 2020, Tan et al., 2022], fail to fully address. The core challenge is twofold. First, standard FL methods (FedAvg [McMahan et al., 2017], FedProx [Li et al., 2020]) produce a single global model that ignores regional heterogeneity: when “thongs” means footwear in Australia but undergarments in the US, a global model inevitably compromises accuracy for some regions. Second, existing personalized FL methods operate at the parameter level through control variates or bi-level optimization; while these are architecture-agnostic in interface, they are sensitive to parameter landscape topology and become unstable on modern transformer architectures due to shared embeddings, attention coupling, and LayerNorm interactions. Parameter-level methods like SCAFFOLD [Karimireddy et al., 2020] and pFedMe [T. Dinh et al., 2020] achieved strong results on CNNs but struggle on high-dimensional transformer landscapes (detailed analysis in Appendix F). Recent methods including FedProto [Tan et al., 2022] and pFedHyper [Zhang et al., 2024] advance personalization but remain parameter-level, inheriting similar limitations. This limits the applicability of existing personalized FL to production systems that increasingly rely on transformer-based models. This observation motivates our key insight: effective personalization should operate at the gradient level rather than the parameter level. By treating models as black-box function approximators and personalizing based on gradient conflicts, we can achieve architecture-robust personalization that works seamlessly across model families. We personalize at the region level rather than the individual user level because regions pool enough clients to yield stable gradient estimates, and users within the same region share systematic patterns (vocabulary, product mix, seasonal behavior) that per-client models with scarce data cannot capture. We introduce RegionFed, a gradient-level federated learning framework for retail query understanding that operates seamlessly across model architectures. Unlike prior work [Arivazhagan et al., 2019] requiring fixed personalization strategies, RegionFed dynamically determines optimal personalization through gradient conflict analysis, automatically adjusting regional adaptation without compromising privacy. While gradient projection [Yu et al., 2020] and hierarchical aggregation [Ghosh et al., 2020] exist independently, RegionFed provides a principled combination: gradient conflict serves as both the heterogeneity diagnostic and the adaptive control signal within a hierarchical regional structure, enabling compute-aware routing and DP-compatible personalization in a unified framework. Our contributions are: (1) Gradient-conflict framework for architecture-robust personalization: rather than manipulating model parameters directly (which fails on transformers due to tied embedding and LayerNorm feedback loops), RegionFed uses ℓ2 gradient conflict as the sole signal for adaptive personalization within a hierarchical regional structure. Gradient-level operations avoid instability because they determine how much of the standard gradient to apply per region, without injecting additive parameter corrections that propagate through shared weight matrices. This treats models as differentiable black boxes, enabling deployment across architectures (T5, RoBERTa, or future designs) without code modification. (2) Compute-aware strategy orchestration: the gradientconflict signal serves as an automated triage system that routes each region to the cheapest sufficient strategy (Grad/Interp/Meta), deploying expensive meta-learning only when conflict warrants it. (3) DP-compatible conflict detection with concrete privacy guarantees: DP gradient clipping (C=1.0) bounds the signal component, while the sigmoid centering via µr (calibrated from round 1) absorbs the noise baseline, ensuring that relative conflict differences remain informative across rounds and regions. With σdp =4.0, T =50, and δ=10−5 , RegionFed achieves ϵ ≈ 0.60√while closing the gap to centralized accuracy. (4) Theoretical and empirical validation: O(1/ T ) convergence with explicit heterogeneity dependence (Theorem 5.1); 12–25pp improvement over FedAvg/FedProx on T5-Small; collapse (<10%) of all four parameter-level baselines tested across E∈{1, 5, 10, 40} 2

on transformers, confirmed transformer-specific via LEAF-FEMNIST [Caldas et al., 2018] where SCAFFOLD achieves 79.52% on CNN; cross-architecture confirmation on RoBERTa-Base, T5-3B, and CNN; cross-domain validation on Amazon Reviews [Ni et al., 2019] (Table 1, Panel B); sustained advantage over 50 rounds. All results averaged over 5 seeds. Related Work. FedAvg [McMahan et al., 2017] and FedProx [Li et al., 2020] produce single global models; clustered methods [Ghosh et al., 2020, Sattler et al., 2021, Marfoq et al., 2021] group clients by similarity but lack regional adaptation. Parameter-level methods (SCAFFOLD [Karimireddy et al., 2020], pFedMe [T. Dinh et al., 2020], Per-FedAvg [Fallah et al., 2020], APFL [Deng et al., 2020], Ditto [Li et al., 2021a], FedNova [Wang et al., 2020]) work on CNNs but cause transformer instability; FedBN [Li et al., 2021b] personalizes BatchNorm but not LayerNorm. Layer-splitting methods (FedBABU [Oh et al., 2022], FedRep [Collins et al., 2021]) avoid collapse but limit personalization. Transformer-specific methods (FedTP [Li et al., 2023], FedAdapter [Cai et al., 2023], FedLoRA [Yi et al., 2023]) require architecture-specific design. Multi-task gradient methods (PCGrad [Yu et al., 2020], CAGrad [Liu et al., 2021]) project away conflicts within a single model; RegionFed uses conflict magnitude as a routing signal across regional models. Concurrent work [Sun et al., 2024, Chen et al., 2025] explores layer-wise conflict and black-box FL; RegionFed differs by using holistic gradient conflict for hierarchical regional routing. See Appendix A for a full survey.

Challenges and Problem Formulation

Federated learning converges to sub-optimal solutions under data heterogeneity: client drift degrades convergence proportionally to gradient dissimilarity [Karimireddy et al., 2020], and heterogeneous objectives create an irreducible error term [Li et al., 2020]. Retail search amplifies this through regional semantic differences (“thongs” means footwear in Australia but undergarments in the US), varying product availability, and seasonal/cultural timing differences. A single global model inevitably sacrifices regional accuracy, motivating region-aware personalization. 2.1

Problem Formulation

P Let R = {r1 , . . . , rM } denote M regions, each with nr clients (N = r nr total). Each client u in region r has private data Dur . RegionFed optimizes a two-level (global + regional) objective over T communication rounds; user-level parameters θulocal are computed via on-device local adaptation (Eq. 5): M K X X |Dr | X u min pr λk Lk (fk (·; θ, θr ); Dur ) (1) |D | θ,{θr }M r r=1 r=1 u∈r k=1

where pr = 1/M is the Pregion weight (equal-sized regions in our setup), λk weights task k (across K tasks), and |Dr | = u∈r |Dur |. To bridge the task-level losses with the regional and user losses used in subsequent sections, we define: Lu (θ, θr ) =

K X

λk Lk (fk (·; θ, θr ); Dur )

(2)

k=1

Lr (θ, θr ) = so that Eq. 1 simplifies to minθ,{θr }

X |Dr | u

|Dr | u∈r

Lu (θ, θr )

(3)

r=1 pr Lr (θ, θr ).

RegionFed: System Architecture

3.1

Framework Architecture Overview

Figure 1 illustrates RegionFed’s three-layer hierarchy: (1) Global Server maintains θ and coordinates regional aggregation; (2) Regional Coordinators maintain θr , compute adaptive weights αr via gradient conflict, and optimize ρr via golden section search; (3) Clients train on private data Dur and 3

communicate only DP-noised gradient updates. At deployment time, clients compose personalized models via θu = θ + αr · θr + αu · θulocal (Eq. 5), where θulocal is obtained via on-device local fine-tuning.

t gr

Regional Layer

Client Layer

Client u1

Global Layer

Grad/Interp/ Meta/Dynamic

Global Server θ

Strategy Selection

t gr

Region 1: θr1

t gr

Region M: θrM

Region 2: θr2

Client u2

Client u3

Client u4

Client un−1

Golden Section ρ Search

Adaptive αr Computation

Client un

Figure 1: RegionFed architecture: three-layer hierarchy with global server, regional coordinators (M regions), and local clients. Blue bidirectional arrows show gradient/model exchange; right panel shows adaptive components (strategy selection, golden section ρ optimization, adaptive αr weights). 3.2

Hierarchical Personalization Mechanism

The architecture supports four personalization strategies (Grad, Interp, Meta, Dynamic) that determine how regional adaptations θr are computed; implementation details are in Section 4. We now describe the hierarchical personalization mechanism. Gradient Computation Hierarchy. Gradients are computed at three levels without exposing raw data. At the user level, each client computes ∇Lu (θ) purely on-device. At the regional level, the P |D r | coordinator securely aggregates: gr = u∈r |Dur | ∇Lu (θ). This raw regional gradient is then clipped 2 and noised to produce the DP-safe gradient: ḡr = gr / max(1, ∥gr ∥2 /C), g̃r = ḡr + N (0, σdp C 2 I), P M 1 with C = 1.0 and σdp = 4.0. At the global level: g̃ = M r=1 g̃r . This ensures (ϵ, δ)-differential privacy; with our parameters (T =50, sampling ratio q=0.1, δ=10−5 ), the moments accountant yields ϵ ≈ 0.60 (Theorem I.7 in Appendix). Regional-Level Personalization. For each region r, the regional model combines global parameters with region-specific adaptation: r θregion = θ + αr · θr (4) where θ represents global knowledge learned across all regions, θr captures region-specific patterns, and αr ∈ [αmin , 1.0] controls the personalization strength (αmin = 0.5 by default). User-Level Personalization (On-Device Adaptation). Individual user models further specialize the regional model via local fine-tuning, computed entirely on-device and never transmitted: r θu = θregion + αu · θulocal

(5)

where θulocal is the user-specific adaptation vector obtained by local fine-tuning (gradient updates on r Dur ) of θregion . This step occurs post-deployment and outside federated communication rounds. Adaptive Weight Computation. The personalization weights αr and αu are computed dynamically via gradient conflict analysis on the DP-noised gradients: t ∥g̃r − g̃ t ∥2 − µr αr = αmin + (1 − αmin ) · σ (6) τr r ∥∇Lu (θregion ) − g̃rt ∥2 − µu αu = αmin + (1 − αmin ) · σ (7) τu where σ(·) is the sigmoid function, µr centers the sigmoid at the median gradient conflict (calibrated from round 1), and τr controls sensitivity. We use L2 norm rather than cosine similarity to capture both direction and magnitude differences; DP gradient clipping (C = 1.0) bounds all gradients within an ℓ2 ball of radius C, providing built-in normalization. When all gradients are fully clipped, ∥g̃1 − g̃2 ∥22 = 2C 2 (1 − cos ϕ), gracefully reducing to a pure directional metric. 4

The output is mapped to [αmin , 1.0] (default αmin =0.5) rather than [0, 1] because α=0 would discard regional adaptation entirely; even well-aligned regions retain critical local semantics (ablation in Appendix H: removing the αmin floor degrades accuracy by 2.1pp; µr sensitivity: ±50% perturbation degrades accuracy by only ±0.8%). 3.3

Personalization Strategies

RegionFed supports four strategies for computing θr , where adaptation intensity ρ is optimized per-region via golden section search (Appendix K.2; all regions converge to similar values, mean ρ=0.0344): Grad: θr = ρ · (g̃rt − g̃ t ), i.e., gradient-based adaptation using regional-global DP-noised gradient difference. Interp: θr = ρ · (θrlocal − θ), where θrlocal is obtained by fine-tuning θ on regional data Dr for E local epochs. Meta: θr = ρ · ∇θ Lr (θ − ηg̃ t ), i.e., MAML-style meta-learning for quick adaptation. Dynamic: Automatically selects among Grad/Interp/Meta via Algorithm 1, which routes each region to the cheapest sufficient strategy based on gradient conflict dr , heterogeneity hr , and dataset size |Dr |. Table 7 (Appendix D) summarizes the computational trade-offs. Algorithm 1 Adaptive Strategy Selection Input: Region r, global model θ, DP-noised gradients g̃r , g̃, dataset size |Dr |, mode s ∈ {“grad”, “interp”, “meta”, “dynamic”}, intent distributions Pr (via randomized response [Wang et al., 2017], ϵLDP =2.0) and Pglobal Output: Selected strategy strategyr ∈ {“grad”, “interp”, “meta”} Hyperparameters: τconf lict = 0.5, τhet = 1.0, β = 1.5 if s = “dynamic” then dr = ∥g̃r − g̃∥2 ; hr = KL(Pr ∥Pglobal ) if dr > τconf lict and hr > τhet then return “meta” {Severe drift: learn per-region LR via MAML inner loop} else if |Dr | > β · |D| then return “interp” {Data-rich: blend global and regional gradients} else return “grad” {Mild conflict: lightweight additive correction} end if else return s end if

3.4

RegionFed Algorithm

Algorithm 2 presents the complete training procedure, with notation matching Figure 1: θ (Global Server), θr (Regional Coordinators), g̃rt (DP-noised gradients exchanged via blue arrows), and αr (adaptive weights from the right panel). Each round, the server broadcasts θt ; regions select a strategy (Algorithm 1), compute DP-noised gradients, and update regional adaptations. The gradient conflict signal ∥g̃rt − g̃ t ∥2 drives both what strategy to use and how much to personalize (αr ). All operations use DP-noised gradients, preserving the (ϵ, δ)-DP guarantee by post-processing (Theorem I.7).

Experimental Setup

4.1

Datasets

(1) Amazon ESCI [Reddy et al., 2022] (primary benchmark): 130K real product search queries with regional partitioning by product category; 20,000 queries (16,000/4,000 train/test) across 8 regions, 80 clients (10 per region), Dirichlet αD =0.3 (70%/30% primary/other categories; validated against source distributions, KL=0.08; Appendix B). (2) Amazon Reviews [Ni et al., 2019] (crossdomain): 50,000 reviews across 5 product categories for 5-class sentiment, Dirichlet αD =0.3 across 50 clients (EMD: 0.41±0.12). (3) LEAF-FEMNIST [Caldas et al., 2018] (cross-architecture): 5

Algorithm 2 RegionFed Input: Global model θ0 , regions R = {r1 , . . . , rM }, adaptation search space [a, b] Output: Global model θT , region-specific adaptations {θrT }M r=1 Initialize global model θ0 , personalization parameters {θr0 }M r=1 Calibrate centering thresholds µr from round-1 gradient conflicts for each round t = 1, 2, . . . , T do Server broadcasts global model θt to all regions for each region r ∈ R in parallel do strategyr ← A DAPTIVE S TRATEGY S ELECTION(r, θt−1 , g̃rt−1 , g̃ t−1 , |Dr |) ρoptimal ← G OLDEN S ECTION S EARCH(r, θt , strategyr , [a, b], ξ) {Algorithm 3} Compute final adaptation θrt+1 using strategyr and ρoptimal P |D r | Compute regional gradient: grt = u∈r |Dur | ∇Lu (θt ) Clip gradient: ḡrt = grt / max(1, ∥grt ∥2 /C) 2 Add DP noise: g̃rt = ḡrt + N (0, σdp C 2 I) end for /* GlobalPModel Update */ t 1 g̃ t = M θt+1 = θt − η g̃ t r∈R g̃r ; % Privacy: user → region via secure aggregation; region → global via DP noise above /* Adaptive Personalization (Eqs. 6–7) */ for each region r ∈ R do t t ∥g̃r −g̃ ∥2 −µr r = θt+1 + αr · θrt+1 ; θregion αr = αmin + (1 − αmin ) · σ τr end for /* User-Level Personalization (On-Device Adaptation, Post-Deployment) */ for each user u requesting personalization do r Download θregion ; fine-tune locally to obtain θulocal αu = αmin + (1 − αmin ) · σ

r t ∥∇Lu (θregion )−g̃r ∥2 −µu τu

r θu = θregion + αu · θulocal end for end for

62-class character recognition with writer-based partitioning, 2-conv + 2-FC CNN, 200 clients in 10 regions by writer similarity. Together, these datasets span NLP and vision, enabling evaluation of gradient-conflict personalization and cross-architecture robustness. Complete statistics in Appendix B. 4.2

Multi-Task Query Understanding

We formulate query understanding as three complementary tasks using a unified text-to-text framework: (1) Intent Classification (8 product categories); (2) Spell Correction (exact match accuracy); (3) Named Entity Recognition (BIO tagging, token-level F1). Overall Accuracy is the unweighted mean: 13 (Intent Acc + Spell Acc + NER F1). The Regional Robustness Score (RRS) measures PM 1 worst-region uplift: RRS = M r=1 Accr ; higher RRS with lower inter-region standard deviation indicates more equitable performance. 4.3

Model and Training Configuration

Model Architectures. Primary: T5-Small [Raffel et al., 2020] (60.5M params, 6 encoder/decoder layers, 512-dim), large enough to exhibit parameter-level instabilities while tractable for federated experiments. Cross-architecture: T5-3B (3B params), RoBERTa-Base [Liu et al., 2019] (125M params, encoder-only), and a 2-conv + 2-FC CNN for FEMNIST (vision baseline). Baselines. Centralized (privacy-violating upper bounds), Standard FL (FedAvg [McMahan et al., 2017], FedProx [Li et al., 2020]), Region-Aware FedAvg (ablation isolating hierarchical structure from gradient-conflict adaptation), Parameter-level (SCAFFOLD [Karimireddy et al., 2020], pFedMe [T. Dinh et al., 2020], Ditto [Li et al., 2021a], APFL [Deng et al., 2020]), Layer-Splitting (FedBABU [Oh et al., 2022]), Transformer-Specific (FedTP [Li et al., 2023]), Local Fine-tuning, and RegionFed variants (Grad, Interp, Meta, Dynamic). Hyperparameter details in Table 1 footnotes. 6

Table 1: Overall performance across datasets and architectures (50 rounds, 5 seeds). Bold: best federated; underline: second-best. Red : collapsed (<10%). RF-Meta closes the gap to the centralized upper bound (∆=0.23pp) while parameter-level methods collapse on transformers but not CNN. Category

Method

Overall (%)↑ Intent (%)↑

Spell (%)↑

NER (%)↑

RRS↑

Panel A: Amazon ESCI, T5-Small (primary benchmark, per-task breakdown) Upper Bound (no privacy)

Centralized Cent. + Reg. Weighting

91.72±0.28 92.04±0.25

96.35±0.15 96.72±0.18

78.89±0.34 99.93±0.02 79.84±0.31 99.55±0.08

– 91.54

Standard FL

FedAvg FedProx (µ=0.01) Reg-Aware FedAvg

80.18±0.31 67.18±0.42 83.47±0.29

78.79±0.44 59.03±0.51 81.52±0.37

75.94±0.38 85.81±0.27 75.47±0.35 67.05±0.48 76.89±0.33 92.01±0.21

80.04 67.13 83.21

8.73 0.00 8.92 9.15

1.80 0.00 2.15 2.40

Parameter-Level SCAFFOLD (collapse on T5) pFedMe Ditto (λ=0.01) APFL

0.00 0.00 0.00 0.00

24.38 0.00 24.61 25.05

–∗ –∗ –∗ –∗

Layer-Split Transf.-Spec. Local Only

FedBABU FedTP Local Fine-tuning

84.53±0.29 91.54±0.22 88.84±0.35

85.47±0.33 95.58±0.19 95.13±0.21

76.12±0.36 92.01±0.22 79.79±0.28 99.25±0.09 76.79±0.41 94.60±0.18

84.12 88.80 79.53

RegionFed (ours)

RF-Grad RF-Interp RF-Meta RF-Dynamic

91.92±0.25 92.13±0.28 92.27±0.31 91.92±0.33

98.07±0.14 97.85±0.17 97.64±0.16 97.85±0.19

78.76±0.30 79.04±0.32 79.67±0.29 78.76±0.35

98.93±0.11 99.50±0.07 99.50±0.06 99.14±0.12

91.52 86.14 91.62 90.22

FedAvg

SCAFFOLD

FedBABU

FedTP

RF-Meta

Panel B: Cross-Architecture Generalization (Overall Accuracy %) ESCI

RoBERTa-Base

79.84±0.47

7.53±0.61

84.17±0.38 90.43±0.34

91.62±0.33

ESCI

T5-3B

83.41±0.26

5.21±0.44

87.09±0.31 93.68±0.19

94.12±0.27

Reviews

T5-Small

59.47±0.52

6.84±0.71

62.84±0.44 65.82±0.38

68.94±0.36

Reviews

RoBERTa-Base

61.73±0.58

7.21±0.82

63.52±0.49 67.14±0.42

70.41±0.40

Reviews FEMNIST

T5-3B CNN

63.82±0.41 82.14±0.34

5.92±0.68 79.52±0.45

66.18±0.37 69.53±0.33 83.48±0.32 –

72.34±0.35 85.21±0.30

Centralized upper bounds

ESCI: T5-S 92.04, RoB 91.38, T5-3B 94.81; Rev: T5-S 70.21, RoB 71.58, T5-3B 73.42; FEM 86.73

Panel A: Overall = 13 (Intent + Spell + NER). RRS = Regional Robustness Score (mean across 8 regions). ∗ RRS undefined at <10%. Panel

B: SCAFFOLD works on CNN (79.52%) confirming transformer-specific failure. FedTP requires transformer-specific hypernetwork (not applicable to CNN). RoBERTa is encoder-only (single-task evaluation). FedProx omitted from Panel B for space (consistently below FedAvg on transformers). All methods under (ϵ≈0.60)-DP. Mean±std over 5 seeds.

Training Configuration. T =50 rounds, E=40 local epochs, batch size 32, AdamW. Learning rates tuned via grid search: RegionFed best at η=10−3 (gradient-conflict filtering stabilizes higher LR); baselines best at η=10−4 . As control, RegionFed at η=10−4 still achieves 91.14%, confirming gains are not LR-driven. Parameter-level collapse persists across E∈{1, 5, 10, 40} (Appendix F). 4×A100 GPUs.

Results and Analysis

5.1

Overall Performance Comparison

Table 1 presents comprehensive results. RegionFed-Meta achieves 92.27%, within statistical noise of centralized training (91.72%, ∆=0.55pp, within 1σ), while fully preserving (ϵ≈0.60, δ=10−5 )-DP. Finding 1: RegionFed Matches Centralized Training. RegionFed-Meta achieves 92.27±0.31% overall accuracy. To properly contextualize this, we report two centralized upper bounds: (a) Centralized (91.72±0.28%): pooled data with a single global objective (privacy-violating); (b) Centralized + Regional Weighting (92.04±0.25%): pooled data with region-weighted multi-task loss, which controls for the hierarchical objective structure. RegionFed-Meta closes the gap to the stronger upper bound (92.27% vs 92.04%, ∆=0.23pp, within 1σ), confirming that the gradientconflict mechanism fully recovers the privacy-violating upper bound’s accuracy. On Amazon Reviews, RegionFed-Meta (68.94%) approaches centralized (70.21%), and on FEMNIST (85.21% vs 86.73%). Contribution decomposition: Regional structure alone (Reg-Aware FedAvg, 83.47%) provides +3.29pp over FedAvg (80.18%); gradient-conflict adaptation adds a further +8.80pp to reach RF-Meta (92.27%), accounting for 73% of the total 12.09pp gain. 7

Finding 2: Standard FL Methods Are Insufficient. FedAvg (80.18%) and FedProx (67.18%) fall 12–25pp below centralized training. FedProx underperforms FedAvg because its proximal term applies uniform regularization to all parameters (best µ=0.01; at µ=0.001 FedProx reaches 78.4%, approaching FedAvg but still 13.87pp below RegionFed; see Appendix C for detailed analysis), preventing necessary specialization: attention heads require rapid drift for regional adaptation while embeddings should remain stable. Finding 3: Parameter-Level Methods Fail on Transformers (Independent of E). SCAFFOLD, pFedMe, Ditto, and APFL all collapse to <10% on T5-Small. This failure persists across all local epoch settings (E∈{1, 5, 10, 40}; Appendix F), confirming it is fundamental rather than an artifact of client drift. Controlled ablations (Appendix F.5) isolate the root cause: untying T5’s shared embeddings alone recovers SCAFFOLD from 8.73% to 42.61%; freezing LayerNorm recovers to 38.47%; combining both yields 71.83%. This demonstrates that tied embeddings and LayerNorm are the primary instability sources for parameter-level methods on transformers. Gradient-level operations avoid this collapse because they scale the magnitude of the standard gradient by αr rather than adding a separate correction vector cr to the parameters. Formally, for a LayerNorm layer LN(x) = γ · (x − µ)/σ + β, an additive parameter correction γ ← γ + cγ enters multiplicatively in the forward pass, causing O(cγ /σ) perturbation that compounds across layers; in contrast, gradient scaling θ ← θ − αr ηg merely adjusts step size along the same descent direction, preserving the loss landscape curvature that the optimizer has already adapted to (Appendix F). Ditto’s proximal objective at any λ > 0 amplifies the same instability (λ=0 recovers Local Finetuning at 88.84%). FedBABU [Oh et al., 2022] avoids collapse (84.53%) via head-only personalization, but falls 7.74pp below RegionFed-Meta. Finding 4: Cross-Architecture and Cross-Domain Generalization. Panel B confirms RegionFed generalizes across all architecture-dataset combinations. The parameter-level failure is transformerspecific: SCAFFOLD collapses on every transformer but achieves 79.52% on CNN (FEMNIST). RegionFed-Meta achieves 94.12% (T5-3B), 91.62% (RoBERTa), and 85.21% (FEMNIST CNN vs centralized 86.73%). The FEMNIST gain is smaller (+3.07pp over FedAvg) because CNNs lack the tied embeddings and LayerNorm that cause parameter-level collapse; RegionFed still improves on CNN with zero code changes. FedTP [Li et al., 2023] achieves 91.54% on T5 but requires redesigning its hypernetwork per architecture; RegionFed surpasses it by +2.82pp in RRS while being 20% faster per round (Appendix F.6). Finding 5: Strategy Comparison. All strategies achieve 91–92%: Meta (92.27%, highest RRS at 91.62), Grad (91.92%, lowest overhead at 1.02× FedAvg wall-clock). The gradient-conflict signal acts as a triage system, routing each region to the cheapest sufficient strategy. Grad is the recommended default (single gradient computation, competitive accuracy, lowest regional variance SD=0.77); Meta is preferred when maximum accuracy is required (1.15× FedAvg wall-clock due to MAML inner loop). The Dynamic router occasionally misselects strategies and uses validation-based fallback (Appendix D.6). 5.2

Robustness, Regional Equity, and Convergence

Robustness to Data Heterogeneity and Regional Equity. Table 2 evaluates heterogeneity robustness (Dirichlet αD ∈ {0.1, 0.5, 1.0}) and regional equity via RRS. Table 2: Robustness to heterogeneity and regional equity RRS by αD Method FedAvg FedProx Reg-Aware FedAvg Local FT FedBABU FedTP RF-Grad RF-Meta

1.0 (Low) 0.3 (Med) 0.1 (High) Deg.↓ Reg. SD↓ RRS↑ 89.1 89.5 89.8 90.2 89.8 91.2 91.8 92.0

80.04 67.13 83.21 79.53 84.12 88.80 91.52 91.62

67.5 69.2 71.3 78.3 72.4 82.1 84.2 85.5

21.6 20.3 18.5 11.9 17.4 9.1 7.6 6.5

2.98 3.40 2.54 3.21 2.18 0.81 0.77 0.97

80.04 67.13 83.21 79.53 84.12 88.80 91.52 91.62

αD : Dirichlet concentration (lower = more heterogeneous). Deg. = degradation from Low to High. Reg. SD and RRS at αD =0.3. SCAFFOLD/pFedMe/Ditto/APFL excluded (<10%). Mean over 5 seeds; std<0.5pp for RegionFed.

RegionFed-Meta degrades only 6.5pp (92.0%→85.5%) vs FedAvg’s 21.6pp (3.3× less degradation). Reg-Aware FedAvg improves over FedAvg by +3.17pp RRS but falls 8.41pp short of RegionFed-Meta, confirming gradient-conflict adaptation drives the gains. Personalization Evidence: Query-Level Outputs. Table 3 illustrates how regional personalization resolves ambiguous queries: the regionalized model correctly interprets “apple tv” as a brand lookup in Electronics and “supplements on sale” as price comparison in Beauty, while the global model conflates these across regions, losing 5–8% accuracy. Table 3: Query-level case studies: RegionFed produces correct personalized predictions with 5–8% gain over the global (non-personalized) baseline across regions and tasks. Region

Query

Task

Predicted

Best

Global Gain

Electronics Fashion Home Sports Beauty

apple tv butter avlable tv cost need yoga mat supplements on sale

Intent Spell NER Intent Intent

brand_lookup butter available PRODUCT O product_search price_comparison

92.3% 91.3% 91.4% 91.9% 93.4%

84.5% 83.6% 85.8% 84.1% 87.2%

+7.8% +7.7% +5.6% +7.8% +6.2%

All predictions correct. Best = highest accuracy among 4 strategies. Global = model without regional personalization. Full 8-region analysis with per-strategy breakdown in Appendix (Table 9).

The consistent gains across tasks confirm that gradient-conflict adaptation preserves region-specific semantics that global averaging dilutes. RegionFed also converges rapidly (Figure 2). Convergence Analysis (50 Rounds) Centr.

Overall Accuracy (%)

Zoom: Top Methods (83-93%) 92

60 40

RF-Meta RF-Grad RF-Dynamic RF-Interp FedTP

Local FT FedBABU FedAvg FedProx SCAFFOLD

88 86 84 10

20 0

T=5

Communication Round

Figure 2: Convergence over 50 rounds. Full y-axis (0–96%) shows SCAFFOLD’s collapse (∼9%); inset zooms into top methods (83–93%) to differentiate converged performance. RegionFed plateaus above 91% by round 5; FedAvg reaches ∼82% by round 20. The persistent gap confirms additional rounds cannot compensate for lacking regional personalization.

Theorem 5.1 (Convergence, informal; full statement in Appendix L). Under L-smooth loss and bounded gradient variance σ 2 , RegionFed achieves: T −1 1 X 1 σ2 Γ2R t 2 E[∥∇F (θ )∥ ] ≤ O √ + √ + √ T t=0 T T T P 1 2 where Γ2R = M r ∥∇Fr − ∇F ∥ is the regional heterogeneity, bounded by regional coordination. p The privacy guarantee (Theorem I.7) provides (ϵ, δ)-DP with ϵ = O(q T log(1/δ)/σdp ); moments accountant yields ϵ ≈ 0.60. RegionFed-Meta maintains >91% accuracy down to ϵ=0.30 and still achieves 88.15% at ϵ=0.15; its advantage over FedAvg increases under stronger privacy (+13.94pp at ϵ=0.15 vs +12.09pp at ϵ=0.60; Appendix J). Additional appendix experiments confirm: 3.5× less degradation at 16 regions (G), controlled ablation isolating tied embeddings and LayerNorm as collapse causes (F.5), only 4.2pp degradation with random region misspecification (G.4), and component ablation showing adaptive α contributes 39.7% of the gain (H).

Conclusion and Limitations

RegionFed personalizes federated learning via gradient conflict rather than parameter manipulation, enabling robust deployment across transformers and CNNs without code changes. Gradient-level 9

operations remain stable where parameter-level corrections collapse; controlled ablations confirm tied embeddings and LayerNorm as the instability sources. Across three public datasets and four architectures, RegionFed achieves 92.27% (within 1σ of centralized 92.04%) with (ϵ≈0.60)-DP. Limitations. (1) Pre-defined regions required (automatic discovery is future work; misspecification incurs only 4.2pp loss). (2) Multimodal tasks and state-space models remain unevaluated. (3) Dynamic strategy selection occasionally misroutes (Appendix D.6). See Appendix M for broader impact and ethics.

References Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308–318, 2016. Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Ankit Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019. Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. In Advances in Neural Information Processing Systems, pages 6277–6287, 2018. Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacypreserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 1175–1191, 2017. Dongqi Cai, Yaozong Zhou, Shangguang Zhang, Qi Li, et al. FedAdapter: Efficient federated learning for modern NLP. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, 2023. Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. LEAF: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018. Shuai Chen, Hanlin Yan, Tong Li, and Tianhao Wang. Generalized and personalized federated learning with black-box foundation models via orthogonal transformations. In International Conference on Machine Learning, 2025. Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning, pages 2089–2099. PMLR, 2021. Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461, 2020. Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3–4):211–407, 2014. Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Advances in Neural Information Processing Systems, pages 3557–3568, 2020. Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In Advances in Neural Information Processing Systems, pages 19586–19597, 2020. Peter Kairouz, Brendan McMahan, Shuang Song, Om Thakkar, Keith Bonawitz, and Daniel Ramage. Practical and private (deep) learning without sampling or shuffling. In International Conference on Machine Learning, 2021. Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132–5143. PMLR, 2020. 10

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519–3529. PMLR, 2019. Hongxia Li, Zhongyi Cai, Jingya Wang, Jiangnan Tang, Wenpeng Ding, Chin-Teng Lin, and Ye Shi. FedTP: Federated learning by transformer personalization. IEEE Transactions on Neural Networks and Learning Systems, 2023. Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems, pages 429–450, 2020. Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages 6357–6368. PMLR, 2021a. Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. In International Conference on Learning Representations, 2021b. Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, volume 34, pages 18878–18890, 2021. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture of distributions. In Advances in Neural Information Processing Systems, pages 15434–15447, 2021. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017. Alexander McNeill and Pradeep Vijayakumar. Hierarchical procrustes matching for shape correspondence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2485–2493, 2015. Ilya Mironov. Rényi differential privacy. In IEEE 30th Computer Security Foundations Symposium (CSF), pages 263–275. IEEE, 2017. Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 188–197, 2019. Jaehoon Oh, SangMook Kim, and Se-Young Yun. FedBABU: Toward enhanced representation for federated image classification. In International Conference on Learning Representations, 2022. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. Chandan K. Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. Shopping queries dataset: A large-scale ESCI benchmark for improving product search. arXiv preprint arXiv:2206.06588, 2022. Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Modelagnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 32(8):3710–3722, 2021. Brent Smith and Greg Linden. Two decades of recommender systems at amazon.com. IEEE Internet Computing, 21(3):12–18, 2017. 11

Jianqing Sun, Shuo Xu, Lixu Ma, Wei Cao, and Wei Gao. Towards layer-wise personalized federated learning: Adaptive layer disentanglement via conflicting gradients. In International Conference on Learning Representations, 2024. Canh T. Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. In Advances in Neural Information Processing Systems, pages 21394–21405, 2020. Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In AAAI Conference on Artificial Intelligence, pages 8432–8440, 2022. Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In Advances in Neural Information Processing Systems, pages 7611–7623, 2020. Tianhao Wang, Jeremiah Blocki, Ninghui Li, and Somesh Jha. Locally differentially private protocols for frequency estimation. In 26th USENIX Security Symposium, pages 729–745, 2017. Fan Yang, Ajinkya Kale, Yury Bubnov, Leon Stein, Qiaosong Wang, Hadi Kiapour, and Robinson Piramuthu. Visual search at ebay. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2101–2110, 2017. Liping Yi, Gang Wang, Xiaoguang Liu, Zhuan Shi, and Han Yu. FedLoRA: Model-heterogeneous personalized federated learning with LoRA tuning. arXiv preprint arXiv:2310.13283, 2023. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, pages 5824–5836, 2020. Xinyang Zhang, Hong Chen, Zhiwei Zhao, Jianwei Guo, and Tianming Chen. pFedHyper: Personalized federated learning via hypernetworks. arXiv preprint arXiv:2402.04038, 2024.

Background and Related Work

A.1

Federated Learning Foundations

Consider a federated learning system comprising a central server and N clients distributed across M regions. Let Di represent the private dataset owned by client i ∈ [N ], and θ ∈ Rd represent the global model parameters. Federated learning aims to learn a model that minimizes the aggregate loss PN F (θ) := i=1 αi Fi (θ), where client i is weighted by αi > 0. P The local loss Fi (θ) = |D1i | z∈Di ℓ(θ; z) evaluates the model’s performance on dataset Di associated with client i, where ℓ(·; ·) represents the loss function (e.g., cross-entropy), and z := (q, c) is the query-click pairs. FL training executes the following steps within one round: • Step I: Server broadcasts global model θt to all clients. ∂F (θ t ,ξ ) • Step II: Client i trains locally, computing gradient ∇i = i∂θit i . i

• Step III: Server aggregates updates via θt+1 = θt − βt AAGR (∇{i∈[N ]} ). A.2

Evolution of Federated Learning Methods

First Generation: Global Model Approaches (2017). FedAvg [McMahan et al., 2017] established the foundational paradigm of averaging client updates. This produces a single global model that cannot adapt to regional heterogeneity. Second Generation: Personalized FL (2020-2021). FedProx [Li et al., 2020] addressed client drift through proximal regularization; SCAFFOLD [Karimireddy et al., 2020] introduced control variates for variance reduction; pFedMe [T. Dinh et al., 2020] applied Moreau envelopes for bi-level personalization; Per-FedAvg [Fallah et al., 2020] combined MAML with FL for fast adaptation; FedRep [Collins et al., 2021] separated representation and head layers; FedBABU [Oh et al., 2022] simplified layer splitting by aggregating only the body and keeping the head local; APFL [Deng et al., 2020] proposed adaptive mixing of local and global models; Ditto [Li et al., 2021a] learned personalized models with a global-regularized local objective; FedBN [Li et al., 2021b] personalized BatchNorm statistics to handle feature shift across clients; FedNova [Wang et al., 2020] addressed objective inconsistency through normalized averaging. These methods achieved strong results on CNNs (CIFAR-10, EMNIST) but operate at the parameter level, limiting applicability to transformers. Transformer-Specific FL Methods (2023+). Recent work has addressed transformer personalization through architecture-specific designs: FedTP [Li et al., 2023] uses hypernetworks to generate personalized attention projections (WQ , WK , WV ); FedAdapter [Cai et al., 2023] inserts trainable adapter modules for efficient NLP model personalization; FedLoRA [Yi et al., 2023] applies low-rank adaptation for parameter-efficient federated fine-tuning. While effective for their target architectures, these methods require architecture-specific modifications: FedTP must know the attention mechanism structure, FedAdapter requires inserting adapter layers at specific positions, FedLoRA depends on identifying suitable layers for low-rank decomposition. This limits applicability when model architectures change frequently (e.g., upgrading from T5 to Llama, or deploying non-attention architectures like Mamba/RWKV). RegionFed’s gradient-level approach is architecture-robust: it treats models as differentiable black boxes, enabling seamless application to any architecture without code modifications. Third Generation: Clustered and Hierarchical FL (2020-2021). CFL [Ghosh et al., 2020] clusters clients based on gradient similarity; ClusteredFL [Sattler et al., 2021] provides model-agnostic distributed multi-task optimization; FedEM [Marfoq et al., 2021] models client data as mixtures of underlying distributions. These approaches recognize the need for intermediate aggregation but produce single models per cluster without fine-grained personalization mechanisms. Recent Developments (2022-2025). FedProto [Tan et al., 2022] aggregates class prototypes rather than model weights for heterogeneous clients; pFedHyper [Zhang et al., 2024] uses hypernetworks to generate personalized model parameters. Layer-wise gradient conflict analysis [Sun et al., 2024] decomposes conflicts per layer for finer-grained adaptation. Black-box FL with foundation models [Chen et al., 2025] explores federated fine-tuning of large pre-trained models without access to internal weights. While these methods advance personalization, they remain either architecture13

specific or operate at the parameter level. RegionFed’s gradient-level approach remains unique in combining architecture-robustness with regional hierarchical aggregation. Baseline Selection Rationale. We evaluate against FedAvg, FedProx, SCAFFOLD, pFedMe, FedBABU, Ditto, APFL, and FedTP as they represent the canonical methods in their respective categories: (1) FedAvg/FedProx for global model approaches, (2) SCAFFOLD for variance reduction via control variates, (3) pFedMe for bi-level personalization, (4) FedBABU [Oh et al., 2022] for layer-splitting personalization (body aggregation with local head), (5) Ditto/APFL for local-global mixing, and (6) FedTP for transformer-specific personalization via hypernetworks. FedBABU is particularly informative because it avoids the catastrophic collapse of SCAFFOLD/pFedMe/Ditto/APFL on transformers (achieving 84.53% vs <10%), isolating the failure mode to parameter-level manipulation rather than the FL setting itself (detailed analysis in Section F.7). Other parameter-level methods (FedRep, FedNova, Per-FedAvg) share the same fundamental limitation. Our experiments demonstrate that RegionFed achieves 92.27% while remaining architecture-agnostic, surpassing both collapsing parameter-level methods and the stable but limited FedBABU baseline. A.3

Query Understanding in E-commerce

Modern query understanding systems in e-commerce face five key challenges: (1) Privacy and Sensitivity: user search queries frequently include sensitive terms exposing private health conditions or personal preferences, risking GDPR/CCPA violations; (2) Centralized Data Limitations: uploading large volumes of raw query data incurs high bandwidth costs and limits real-time feedback; (3) Regional Variance: search behavior varies across locations (e.g., “thongs” refers to footwear in Australia but undergarments in the US); (4) Lack of On-Device Personalization: user intent depends on personal search histories that models avoid due to privacy risks; (5) Slow Trend Adaptation: events like Black Friday rapidly shift query distributions while centralized pipelines require weeks to retrain. Notable systems include large-scale personalized recommendation engines [Smith and Linden, 2017], multi-grained attention mechanisms [Yang et al., 2017], and recent transformer-based query understanding systems. These typically leverage centralized data collection that exacerbates privacy challenges. A.4

Personalization Strategies Taxonomy

We categorize FL personalization approaches by their operating level: Parameter-Level Methods manipulate model parameters directly: • Control variates (SCAFFOLD): Maintain client-specific correction vectors ci • Bi-level optimization (pFedMe): Solve nested optimization with parameter regularization • Normalized averaging (FedNova): Address objective inconsistency via local step normalization • Layer splitting (FedRep, LG-FedAvg): Separate shared/personal layers These methods assume direct parameter access and manipulation, becoming unstable with transformers’ shared embeddings and attention coupling. Gradient-Level Methods operate on computed gradients: • Gradient weighting: Adjust contribution of client gradients based on similarity • Gradient projection: Project gradients to reduce conflicts (PCGrad [Yu et al., 2020], CAGrad [Liu et al., 2021]) • RegionFed (this work): Adaptive regional personalization via gradient conflict detection Gradient-level methods are inherently architecture-agnostic as they only require gradient computation, not parameter structure knowledge. A.5

Privacy in Search Systems

Privacy concerns in search systems have been addressed through differential privacy [Dwork and Roth, 2014, Abadi et al., 2016], secure aggregation [Bonawitz et al., 2017], and local privacy [Wang et al., 2017]. Recent work combines differential privacy with FL, though privacy-utility trade-offs remain challenging for personalized search. 14

To our knowledge, no prior work has explored architecture-agnostic federated learning for retail search with transformer models. RegionFed fills this gap by providing gradient-based personalization that works seamlessly with modern neural architectures while maintaining theoretical convergence guarantees.

Dataset Details

We evaluate RegionFed on three public datasets spanning NLP and vision: (1) Amazon ESCI [Reddy et al., 2022] for multi-task query understanding (primary benchmark), (2) Amazon Reviews [Ni et al., 2019] for cross-domain sentiment classification, and (3) LEAF-FEMNIST [Caldas et al., 2018] for cross-architecture validation on a vision task. Below we detail each dataset’s construction, configuration, and federated partitioning. B.1

Amazon ESCI (Primary Benchmark)

Amazon ESCI is a multi-task query understanding benchmark derived from the public Shopping Queries Dataset [Reddy et al., 2022] (130,652 unique queries, 2.6M query-product judgements). It captures realistic heterogeneity in federated retail search through controlled regional partitioning. B.1.1

Construction Methodology

Source Data. From the public ESCI dataset, we extracted: (1) syntactic query templates from 97,345 English queries; (2) a 24-category product taxonomy; (3) relevance label distributions (Exact/Substitute/Complement/Irrelevant) informing intent boundaries; (4) entity types (brand, color, size, material) from 1.8M product descriptions. Regional Heterogeneity Modeling. We modeled category preferences across 8 demographic regions using publicly available retail industry reports and ESCI distributions: regional category probabilities P (category|region), top-1000 query terms per category, character-level typo distributions (substitution 60%, deletion 20%, insertion 20%; 15% corruption rate), and seasonal query volume shifts. Validation. Distributional statistics against the source ESCI dataset confirm realism: (1) KL divergence between partitioned and original category distributions: 0.08; (2) query length distribution overlap: 94.2%; (3) entity type frequency correlation: Pearson r = 0.91. Why Controlled Partitioning? Arbitrary user partitioning (e.g., random Dirichlet on IID data) destroys semantic regional correlations. Our benchmark constructs these correlations explicitly, enabling: reproducibility across research groups, controlled heterogeneity via Dirichlet α, privacy compliance (no real user data), and validated realism grounded in ESCI. B.1.2

Dataset Statistics Table 4: Amazon ESCI Benchmark Statistics Metric

Value

Total / Train / Test Samples Regions / Clients per Region Vocabulary Size / Max Query Length Intent Classes / Entity Types Product Categories Dirichlet αD / Regional Bias

B.1.3

20,000 / 16,000 / 4,000 8 / 10 (80 total) 5,000 words / 20 tokens 8 / 6 (5 + O tag) 24 0.3 / 70%

Regional Configuration and Bias Formulation

where Cr is the preferred category set for region r and C is the full set of 24 categories. Table 5 defines the regional configuration. Table 5: Regional Shopping Preferences and Product Focus ID Region Type 0 1 2 3 4 5 6 7

Focus

Top Categories

Urban Tech Tech Electronics, Gaming Young Demo. Fashion Fashion, Apparel High-Income Premium Fashion, Electronics Price-Conscious Value Grocery, Food Active Lifestyle Outdoor Sports, Fitness Business Prof. Prof. Apparel, Books Suburban Home Home, Garden Premium Markets Luxury Beauty, Health

Note: Each region receives 70% queries from preferred categories (Cr ), 30% from others.

B.1.4

Multi-Task Annotations

The benchmark provides annotations for three query understanding tasks: Intent Classification. Eight categories: product_search, price_comparison, brand_lookup, category_browse, specific_item, deal_hunting, review_reading, availability_check. Approximately 2,000 samples per class in the training set. Named Entity Recognition. BIO tagging with five entity types plus O tag: PRODUCT (e.g., “laptop”), BRAND (e.g., “Nike”), CATEGORY (e.g., “electronics”), MODIFIER (e.g., “cheap”), LOCATION (e.g., “online”). Spell Correction. 15% token corruption rate with character substitution (60%), deletion (20%), and insertion (20%). Corrupted tokens are paired with clean versions for supervised correction. B.1.5

Heterogeneity Configuration

Dirichlet α=0.3 controls client-level heterogeneity (medium-high non-IID). Combined with the 70% regional category bias, this creates two-level heterogeneity: (1) clients specialize in different intents within their region, and (2) regions specialize in different product categories. Lower α (e.g., 0.1) creates extreme client specialization; higher α (e.g., 2.0) approaches IID. We evaluate robustness across α ∈ {0.1, 0.3, 1.0} in the main paper (Table 2). B.1.6

Vocabulary and Query Structure

The dataset uses 245 domain-specific words (electronics, fashion, home, actions, modifiers, categories) padded to 5,000 with generic tokens to model vocabulary gaps. Intent-specific query templates with product/brand/modifier slots generate realistic queries (e.g., product_search: “find {product}”, “{brand} {product}”; price_comparison: “how much is {product}”, “cheap {product}”). B.1.7

Visualizations

Figure 3 shows the intent distribution heatmap across all 80 clients organized into 8 regions, demonstrating non-IID patterns from Dirichlet sampling. Figure 4 validates the 70/30 category bias formulation empirically. B.2

Amazon Reviews

Amazon Reviews [Ni et al., 2019] tests cross-domain generalization beyond retail search with a single-task sentiment classification setup. Dataset Configuration. 50,000 reviews across 5 product categories (Electronics, Books, Clothing, Home & Kitchen, Sports & Outdoors) for 5-class sentiment classification (1–5 stars). Partitioned by 16

Figure 3: Intent Distribution Across 80 Clients. Heatmap showing the proportion of each intent category across all 80 clients organized into 8 regions (blue dashed lines). Darker colors indicate higher specialization, demonstrating non-IID distribution from Dirichlet α=0.3. Bottom panel shows samples per region.

Figure 4: Regional Product Category Distribution. Top: client-level category heatmap (80 clients × 24 categories). Bottom: regional preference matrix validating the 70% bias formulation. Each region concentrates on designated categories (Cr ) while maintaining 30% cross-regional diversity.

category using Dirichlet αD =0.3 across 50 clients (10 per category-region). Mean EMD between client label distributions: 0.41 ± 0.12. 17

Task. Single-task 5-class sentiment classification (text → star rating). This complements ESCI’s multi-task structure by testing whether gradient-conflict personalization generalizes to domain-level heterogeneity. Training. Same federated protocol: T =50 rounds, 40 local epochs, T5-Small, AdamW (η=10−3 for RegionFed, η=10−4 for baselines), σdp =4.0, C=1.0. Key Finding. RegionFed-Meta achieves 68.94% vs centralized 70.21% (∆=1.27pp). Parameterlevel methods collapse (<10%) on this transformer task; FedAvg reaches 61.23%. The consistent pattern confirms gradient-conflict personalization is not dataset-specific. B.3

LEAF-FEMNIST

LEAF-FEMNIST [Caldas et al., 2018] validates architecture-agnosticism on a vision task with naturally heterogeneous (not synthetic) client partitioning, where each client corresponds to a single handwriting author. Dataset Configuration. 62-class character recognition (10 digits + 26 uppercase + 26 lowercase). 200 clients sampled from the full LEAF dataset, grouped into 10 regions by writer similarity (pairwise cosine similarity of per-writer class distributions, agglomerative clustering). Each region contains 20 clients. Train/test split follows the standard LEAF protocol (90/10 per client). Model Architecture. 2-conv + 2-FC CNN: Conv(1→32, 5×5) → ReLU → MaxPool(2) → Conv(32→64, 5×5) → ReLU → MaxPool(2) → FC(1600→512) → ReLU → FC(512→62). Total: ∼1.2M parameters. Training. T =50 rounds, 40 local epochs, SGD with momentum 0.9, η=0.01, σdp =4.0, C=1.0. FedTP is omitted (requires transformer attention mechanisms). Key Finding. SCAFFOLD achieves 79.52% (functional, not collapsed), confirming that <10% failure is transformer-specific. Ditto (80.8%) and APFL (81.2%) also function normally on CNNs. RegionFed-Meta achieves 85.21%, demonstrating consistent gains across architectures.

Expanded Analysis of Main Findings

This section provides detailed justification and analysis for the five key findings presented in Section 5.1 of the main paper. While the main paper presents compressed results, here we provide the full reasoning, supporting evidence, and implications. C.1

Finding 1: Why RegionFed Meets Centralized Training Performance

RegionFed-Meta (92.27%) slightly exceeds centralized training (91.72%) by 0.55pp. While this difference is statistically significant (p = 0.043), the practical magnitude is modest. The key insight is that regional specialization eliminates the traditional FL performance penalty, enabling privacy-preserving distributed training to match centralized accuracy. Theoretical Justification. Consider a heterogeneous P data distribution where region r has distribution ∗ Pr (x, y) and the global distribution is P (x, y) = r wr Pr (x, y). A centralized model θcentral minimizes the global expected loss: ∗ θcentral = arg min E(x,y)∼P [L(θ; x, y)] θ

(9)

However, when regional distributions differ substantially, this global optimum may not be optimal for any individual region. RegionFed’s regional models θr = θ + αr · ∆r can achieve: ∗ Er [E(x,y)∼Pr [L(θr ; x, y)]] < E(x,y)∼P [L(θcentral ; x, y)]

(10)

when regional specialization benefits outweigh the loss of global knowledge sharing. Empirical Evidence. Per-task analysis reveals: • NER: RegionFed achieves 99-100% vs centralized 97.2%. Regional entity vocabularies (“RTX 3080” in Electronics, “organic quinoa” in Grocery) benefit from specialization. 18

• Intent: RegionFed achieves 83.8% vs centralized 77.2%. Category boundaries are regiondependent. • Spell: RegionFed achieves 92.0% vs centralized 100%. Centralized maintains slight advantage on universal spelling patterns. The key insight is that regional specialization enables learning domain-specific patterns that a single global model cannot capture without overfitting to majority distributions. C.2

Finding 2: Why Standard FL Methods Are Insufficient

FedAvg (80.18%) and FedProx (67.18%) fall 12-25pp below centralized training due to fundamental limitations in handling heterogeneous data. Gradient Conflict Analysis. In heterogeneous FL, regional gradients gr = ∇Lr (θ) point in conflicting directions. FedAvg’s simple averaging: M

gavg =

1 X gr M r=1

(11)

produces a compromise gradient that may be suboptimal for all regions. When ⟨gi , gj ⟩ < 0 (negative cosine similarity), averaging partially cancels useful gradient information. Why L2 Norm Instead of Cosine Similarity? While gradient conflict is often defined via inner product (⟨gi , gj ⟩ < 0), RegionFed uses L2 distance for adaptive weight computation (Equations 6-7 in main paper) because: (1) Magnitude sensitivity: L2 captures both magnitude and direction differences; a region with 10× larger gradients (faster local learning) should receive higher personalization even if directions align; (2) Empirical validation: L2 correlates strongly with cosine similarity in our setting (Pearson r = 0.87, Table 14); (3) Numerical stability: L2 is more stable than cosine similarity for near-zero gradients, which occur during later training stages when the model approaches convergence. Why FedProx Performs Worse. Counter-intuitively, FedProx (67.18%) underperforms FedAvg (80.18%) on T5-Small. The proximal term: µ min Fi (θ) + ∥θ − θt ∥2 (12) θ 2 constrains local updates to stay close to the global model. While the per-parameter gradient µ(θi − θit ) is independent of model dimensionality, the failure arises because the uniform regularization strength µ is applied identically to all parameters regardless of their functional role. In transformers, parameters have vastly different drift rates: embeddings (27% of parameters) change slowly while attention heads require rapid specialization. The uniform constraint prevents necessary specialization of high-drift attention parameters while being irrelevant for low-drift embedding parameters, yielding worse results than unconstrained FedAvg. Task-Specific Impact. Analysis reveals largest gaps in: • Intent classification: -19% relative to centralized (regional vocabulary differs most) • NER: -14% relative to centralized (entity boundaries are region-specific) • Spell correction: -8% relative to centralized (more universal patterns) C.3

Finding 3: Parameter-Level vs Gradient-Level Methods

The poor performance of SCAFFOLD, pFedMe, Ditto, and APFL on T5-Small reveals a fundamental distinction between parameter-level and gradient-level personalization. These methods are architecture-agnostic in their interface (operating on flattened parameter vectors), but are sensitive to the topology of high-dimensional parameter landscapes characteristic of transformers. For detailed failure mode analysis including hyperparameter stability studies, see Section F. Definition. Parameter-level methods manipulate model parameters directly through control variates, proximal regularization, or parameter mixing. Gradient-level methods operate on the optimization signal (gradient conflicts) without modifying parameters directly, using the conflict as a diagnostic and routing mechanism. Why Architecture Matters. Modern transformers exhibit: 19

1. Shared Embeddings: Encoder-decoder share vocabulary embeddings (16.4M/60.5M = 27% of parameters). Parameter-level corrections disproportionately affect these shared layers. 2. Attention Coupling: Q, K, V projections are coupled through attention scores. Correcting Q without corresponding K/V adjustments creates inconsistent representations. 3. Layer Normalization: Transformers use LayerNorm which normalizes activations per-layer. Parameter perturbations are amplified or dampened unpredictably. Local Fine-tuning Success. Local Fine-tuning achieves 88.84% by operating purely at the gradient level, simply training local models from shared initialization. This validates that gradient-level operations are architecture-agnostic, but client-level personalization (RRS=79.53%) lacks sufficient data for robust regional patterns. C.4

Finding 4: Regional vs Client-Level Personalization

RegionFed’s regional approach (RRS=91.62%) substantially outperforms client-level Local Finetuning (RRS=79.53%) despite both using gradient-level operations. Statistical Power. With 80 clients across 8 regions: • Client-level: Each client has ∼200 training samples (16,000/80) • Regional-level: Each region has ∼2,000 training samples (16,000/8) Regional aggregation provides 10× more data for learning regional patterns, enabling more robust gradient estimates. Coordination Benefits. Within Electronics region, all 10 clients collectively learn technical vocabulary (“GPU”, “VRAM”, “refresh rate”) rather than each independently rediscovering patterns. This coordination effect is quantified by: Coordination Gain = RRSregional − RRSclient = 91.62% − 79.53% = 12.09pp C.5

(13)

Finding 5: Strategy Selection and Deployment Guidelines

All RegionFed strategies achieve 91-92% overall accuracy, but with distinct trade-offs: Table 6: Strategy Trade-offs Summary Strategy

Acc.

RRS

Cost

Best For

Meta Interp Grad Dynamic

92.3% 92.1% 91.9% 91.9%

91.6% High Max accuracy 86.1% Med Perfect NER 91.5% Low Edge devices 90.2% Var. Automation

Strategy Selection Algorithm. For production deployment: 1. If compute-constrained: Use Grad (minimal overhead, 91.92%) 2. If accuracy-critical: Use Meta (highest overall, 92.27%) 3. If NER-focused: Use Interp (100% NER accuracy) 4. If automated deployment: Use Dynamic with Grad fallback when validation drops >5%

Personalization Strategy Mechanisms and Query-Level Case Studies

Table 7 compares strategies along compute, memory, communication, and use-case dimensions. This section provides in-depth analysis of how personalization strategies achieve regional adaptation: regional performance breakdown, adaptation intensity optimization (ρ), computational efficiency trade-offs, and query-level case studies with actual predictions. D.1

Regional Performance Breakdown

Table 8 presents detailed regional accuracy for all strategies across eight product category regions, providing granular insight into adaptation mechanisms under varying heterogeneity conditions. This 20

Table 7: Strategy comparison: complexity, memory, and use cases Strategy

Compute

Memory

Comm. Overhead Best For

Grad

O(|θ|)

Same as FedAvg

Interp

O(E · |θ|)

2 × O(|θ|)

Meta

Dyn.

82.1 78.5 81.3 77.2 80.8 82.4 79.1 78.9

92.0 91.0 91.8 91.5 91.2 90.5 91.7 92.8

91.9 91.3 91.2 91.1 91.8 90.4 91.0 93.1

84.6 83.6 85.8 85.5 84.1 85.4 84.7 87.2

92.3 91.2 91.4 91.0 91.8 90.3 91.0 93.4

91.9 90.8 91.1 78.2 91.9 90.3 91.0 92.9

80.0±1.9 91.6±0.7 91.5±0.8 85.1±1.1 91.5±1.0 89.8±4.5

Dynamic’s Grocery anomaly (78.2%) stems from strategy selection hysteresis. Production safeguard: revert to Grad if validation loss fails to improve for 2 rounds. Grad wins 4/8, Meta 3/8, Dynamic 1/8 regions.

regional breakdown complements the overall metrics in Table 1 from the main paper by revealing region-specific performance patterns and critical failure modes. Regional Heterogeneity Patterns: Grad and Meta achieve highly consistent performance across all regions (90.28%-93.40%), with standard deviations of 0.83% and 0.98% respectively, demonstrating robust adaptation to regional variations. Interp shows significantly lower performance (83.56%-87.21%) with reduced regional consistency, confirming that model interpolation without federated coordination (used here as the global baseline proxy) struggles to capture local query patterns effectively. Dynamic exhibits a critical anomaly in the Grocery region (78.21%), revealing a catastrophic failure mode where automated strategy selection breaks down. This represents a 12.9% underperformance relative to Grad (91.13%) and highlights the need for validation-based fallback mechanisms. Best Strategy by Region: Figure 5 visualizes regional performance trends. Grad achieves optimal performance in 4/8 regions (Automotive, Books, Fashion, Grocery), demonstrating effectiveness for moderate heterogeneity scenarios with minimal computational overhead. Meta excels in 3/8 regions (Electronics, Home, Beauty), where these premium/high-value product categories exhibit complex query patterns benefiting from Meta’s flexible MAML-style adaptation. Dynamic wins in Sports (91.89%), where automated strategy selection successfully identifies the optimal mechanism, but its Grocery failure (78.21%) demonstrates that automated selection requires robust validation infrastructure. D.2

Adaptation Intensity Analysis

Figure 6 presents the optimal adaptation intensity (ρ) values discovered by golden section search (Algorithm 2) for each strategy and region. The ρ parameter controls the strength of personalization: higher values indicate stronger regional adaptation is needed. Strategy-Specific Patterns: Grad and Meta both converge to constant ρ = 0.0344 across all regions (mean=0.0344, std=0.000), indicating that these gradient-based methods achieve effective adaptation with minimal personalization intensity. This low ρ value reflects computational efficiency, as both strategies leverage gradient information to achieve strong regional adaptation without requiring large model perturbations. Interp shows highly variable ρ values (range: 0.10-0.34, mean=0.22, std=0.099), 21

Regional Performance Comparison Across Strategies

e otiv tom

ok s

s ort

ery Gr oc

me Ho

ion Fas h

Ele

ctr o

nic

Grad Interp Meta Dynamic

Accuracy (%)

Region

Figure 5: Regional Performance Comparison Across Strategies. Line plot showing accuracy trends for Grad (pink), Interp (yellow), Meta (green), and Dynamic (cyan) across eight product category regions. Interp consistently underperforms, while Grad/Meta maintain competitive performance with region-specific advantages. Dynamic’s anomalous drop in Grocery reflects the challenge of automated strategy selection. Optimal Adaptation Intensity by Strategy and Region

0.35

Grad Interp Meta

Adaptation Intensity ( )

0.30 0.25 0.20 0.15 0.10

au ty Be

ok s

tom otiv Au

Region

s ort

ery Gr oc

me Ho

ion Fas h

Ele

ctr on ics

0.05

Figure 6: Optimal Adaptation Intensity by Strategy and Region. Line plot showing ρ values for Grad (pink), Interp (yellow), and Meta (green) across regions. Grad and Meta converge to constant ρ = 0.0344 (optimized for efficiency), while Interp exhibits variable ρ (0.10-0.34) reflecting its sensitivity to regional data characteristics. with Fashion/Home regions requiring ρ ≈ 0.34 while Beauty/Electronics need only ρ ≈ 0.10. This variability reflects Interp’s dependence on regional data volume and distribution characteristics. Regional Adaptation Requirements: Fashion and Home regions exhibit the highest ρ values for Interp (0.34), correlating with these regions’ high vocabulary diversity and complex query patterns. Conversely, Beauty and Electronics show lower ρ requirements (0.10-0.13), suggesting more stable regional distributions. Sports shows intermediate ρ ≈ 0.28, reflecting moderate heterogeneity in outdoor/fitness product queries. Optimization Implications: The constant low ρ for Grad/Meta indicates that golden section search consistently identifies efficient adaptation parameters, avoiding over-personalization. Interp’s variable ρ suggests this strategy requires more careful hyperparameter tuning per region, increasing deployment complexity. Dynamic strategy (not shown) adaptively selects ρ based on the chosen strategy, inheriting Grad’s efficiency when selected. D.3

Computational Efficiency Trade-offs

While all four strategies achieve competitive accuracy (91.92%-92.27% overall, as shown in Table 1), they differ significantly in computational requirements, memory footprint, and deployment complexity: Grad (Gradient Difference): Lowest computational cost, computing only ∇Lr − ∇L without additional model evaluations. Memory footprint: O(|θ|) for gradient storage. Optimal for edge devices with memory constraints. Interp (Model Interpolation): Medium computational cost, requiring local fine-tuning for Elocal epochs, then interpolation θr = (1 − λ)θ + λθrlocal . Memory footprint: 2 × O(|θ|) for both global and local models. Suitable when regional compute resources are available. Meta (MAML-style Meta-Learning): Highest computational cost, requiring double gradient computation for meta-learning updates. Memory footprint: O(|θ| + |∇θ|) for both model and meta-gradients. Justified only when maximum regional specialization is critical. 22

Dynamic (Adaptive Selection): Variable cost depending on selected strategy. Adds overhead for computing selection criteria (dr gradient dissimilarity, hr data heterogeneity), but amortized over multiple rounds. Optimal for production deployment where automation outweighs per-round overhead. D.4

Practical Deployment Guidelines

Based on comprehensive experimental results (Tables 1, 8, and Figures 5, 6), we provide the following deployment recommendations: Use Grad when: (1) Resources are constrained (edge devices, limited memory), (2) heterogeneity is moderate (α = 0.3-0.5), (3) fast convergence is prioritized. Achieves 91.92% overall accuracy with 91.52% RRS and minimal computational overhead, winning strategy in 4/8 regions with consistent regional performance (91.46% regional average). Optimal default choice for production deployment. Use Interp when: (1) Serving as a global baseline proxy for measuring personalization gains, (2) scenarios requiring perfect NER performance (100% accuracy) on structured entity recognition, (3) comparing federated vs non-federated approaches. Note: Lower regional consistency (RRS: 86.14%, regional average: 85.10%) indicates this approach struggles with regional vocabulary variations. Use Meta when: (1) Heterogeneity is extreme (α < 0.3), (2) maximum regional specialization is critical (premium product categories like Electronics, Home, Beauty), (3) computational resources are abundant. Achieves highest overall accuracy (92.27%) and best regional consistency (91.62% RRS, 91.54% regional average), winning in 3/8 regions with complex query patterns. Use Dynamic when: (1) Regional characteristics are unknown or changing over time, (2) automated deployment without manual tuning is required, (3) robust fallback mechanisms are in place. Achieves 91.92% overall accuracy with 90.22% RRS through automated strategy selection. Critical caveat: Requires validation infrastructure to detect and mitigate catastrophic failures (e.g., Grocery: 78.21%, 12.9% below Grad). Recommended fallback: Switch to Grad when Dynamic underperforms validation thresholds by > 5%. D.5

Query-Level Case Studies Table 9: Query-Level Case Studies: Multi-Task Performance with Actual Predictions Region

Query

Task

Predicted

Grad Meta Interp Dynamic Gain

Electronics Fashion Home Grocery Sports Books Automotive Beauty

apple tv butter avlable tv cost find laptop deals need yoga mat wrench discount need dvd supplements on sale

Intent Spell NER Intent Intent Intent Intent Intent

brand_lookup butter available PRODUCT O brand_lookup product_search price_comparison product_search price_comparison

91.9 91.3 91.2 91.1 91.8 90.4 91.0 93.1

92.3 91.2 91.4 91.0 91.8 90.3 91.0 93.4

84.5 83.6 85.8 85.5 84.1 85.4 84.7 87.2

91.9 90.8 91.1 78.2 91.9 90.3 91.0 92.9

+7.8 +7.7 +5.6 +5.6 +7.8 +5.0 +6.3 +6.2

Note: All predictions match ground truth. Interp column shows global baseline accuracy (no regional adaptation). Gain = best strategy − global. Dynamic failure in Grocery (78.2%) demonstrates validation fallback need.

Table 9 presents comprehensive performance analysis of actual test queries from the Amazon ESCI dataset across all three tasks (intent classification, spelling correction, NER). The table shows query text, expected output, actual predicted output, and complete accuracy breakdown for all four strategies (Grad, Meta, Interp, Dynamic) plus the global baseline, enabling direct strategy comparison on real regional queries with ground-truth verification. These case studies complement the aggregate metrics in Table 1 by demonstrating how personalization strategies achieve regional adaptation at the individual query level. Multi-Task Personalization Gains and Strategy Competition: All eight case studies exhibit substantial personalization gains (5.0%-7.8%) over the global baseline (Interp column), validating the effectiveness of regional coordination. Complete strategy accuracy reveals tight competition with minimal winning margins (0.0%-0.4%). Meta excels in 3/8 cases: Electronics intent (92.3% vs Grad 91.9%), Home NER (91.4% vs Grad 91.2%), Beauty intent (93.4% vs Grad 93.1%), demonstrating meta-learning’s strength in high-variance regions with complex query patterns (premium/high-value 23

product categories). Grad dominates 4/8 cases: Fashion spell (91.3% vs Meta 91.2%), Grocery intent (91.1% vs Meta 91.0%), Books intent (90.4% vs Meta 90.3%), Automotive intent (91.0%, tied with Meta), achieving competitive performance with lowest computational cost, confirming its role as the optimal default strategy. Dynamic wins Sports intent (91.9% vs Meta/Grad 91.8%), validating automated strategy selection when it works correctly. Critically, the minimal winning margins (0.0%0.4%) indicate that multiple personalization strategies achieve near-optimal regional accuracy; this justifies Dynamic’s value in production deployments where manual strategy tuning is infeasible, provided robust validation mechanisms prevent catastrophic failures. Task Diversity and Label-Level Analysis: The case studies span diverse query understanding patterns demonstrating RegionFed’s multi-task capabilities: (1) Intent classification (6/8 cases): brand lookup (“apple tv”, “find laptop deals”), product search (“need yoga mat”, “need dvd”), price comparison (“wrench discount”, “supplements on sale”); (2) Spelling correction (1/8 cases): Fashion region misspelling → “butter available”; (3) NER (1/8 cases): entity tagging (“tv cost” → PRODUCT O). The Expected vs Predicted columns demonstrate model correctness on representative queries, and all shown examples achieve correct predictions across tasks, confirming that the reported accuracies (90-93% in Table 1) reflect genuine query understanding rather than dataset artifacts or cherry-picked examples. Regional query characteristics are evident: Electronics targets brand-specific products (Apple TV, tech brands), Sports emphasizes activity-based searches (yoga mat), Beauty focuses on health products (supplements). All tasks demonstrate consistent 5-8% personalization gains, confirming that regional adaptation benefits transfer across query understanding objectives. Notably, even the spell correction task (Fashion: 91.3% Grad vs 83.6% Global, 7.7% gain) shows substantial improvement, indicating that regional vocabulary patterns extend beyond semantic intent to orthographic variations, a critical finding for real-world retail search deployment. Global Baseline Failure: The global baseline (Interp column) consistently underperforms across all cases, ranging from 83.6% (Fashion spell) to 87.2% (Beauty intent), confirming that model interpolation without regional federated coordination fails to capture local query patterns effectively. Five regions show ≥ 6% gaps: Sports (7.8%), Electronics/Fashion (7.7%), Automotive (6.3%), Beauty (6.2%), demonstrating the necessity of federated personalization approaches. D.6

Dynamic Strategy Failure Analysis: Root Cause and Proposed Improvements

The case studies reveal a critical anomaly: Dynamic strategy achieves only 78.2% in Grocery intent, substantially worse than Grad (91.1%, 12.9% gap), Meta (91.0%, 12.8% gap), and even the global baseline (85.5%, 7.3% gap). This section provides detailed root cause analysis and proposes improvements to the strategy selection mechanism. Root Cause Analysis: The Hysteresis Problem. Analysis of training logs reveals the failure pattern across communication rounds: Table 10: Dynamic Strategy Selection for Grocery Region Across Rounds Round

Strategy

Accuracy

Diagnosis

1 2 3 4 5

grad interp interp interp interp

91.1% 86.6% 78.2% 86.2% 78.0%

✓ Correct selection △ Performance degraded × Catastrophic drop △ Partial recovery × Final collapse

The Dynamic strategy selection algorithm uses gradient conflict (cosine distance between regional and global gradients) and magnitude ratio to choose between strategies:  if conflict > τ and ∥∇Lr ∥ > ∥∇L∥ grad Strategy = interp if conflict ≤ τ and ∥∇Lr ∥ > 0.5∥∇L∥ (14)  meta otherwise Why Grocery Fails: The Grocery region exhibits distinct query patterns (essential items, food products) that differ significantly from other retail categories. In Round 1, high gradient conflict correctly triggers grad strategy. However, as the global model improves, the apparent conflict decreases below threshold τ = 0.5, triggering interp. This interpolation toward the global model destroys regional specialization, creating a hysteresis feedback loop: 24

1. Interpolation damages Grocery-specific patterns → lower regional gradient magnitude 2. Lower magnitude → conflict metric underestimates true divergence 3. Reduced conflict → continued interp selection → more damage Pattern Identification: Which Regions are Vulnerable? • High vulnerability: Regions with distinct local optima (Grocery: essentials-focused) that diverge from global distribution • Medium vulnerability: Regions with moderate overlap (Home, Automotive) where interpolation partially helps • Low vulnerability: Regions with high global alignment (Electronics, Beauty) where any strategy performs well Proposed Improvements to Strategy Selection. We identify three enhancements to prevent catastrophic misselection: (1) Performance-Aware Fallback: Track validation accuracy per region and switch to Grad (safe default with 90.28%-93.06% consistency) when accuracy drops > 5% from previous round: grad if Acct−1 − Acct−2 < −0.05 Strategyt = (15) Dynamic(∇Lr , ∇L) otherwise (2) Momentum-Based Conflict Estimation: Use exponential moving average of gradient conflict to avoid abrupt strategy switches: conflictt = β · conflictt−1 + (1 − β) · (1 − cos(∇Lr , ∇L)),

β = 0.9

(16)

(3) Ensemble Voting: Combine predictions from multiple strategies with confidence-weighted voting: X ŷ = arg max ws · Ps (y = c|x), ws ∝ Accval (17) s c

s∈{grad, meta, interp}

Production Recommendations. The Grocery anomaly demonstrates that automation without validation safeguards can lead to worse outcomes than the global baseline. For production systems deploying Dynamic strategy: 1. Required: Real-time accuracy monitoring with automatic fallback to Grad 2. Recommended: Momentum-based conflict estimation (β ≥ 0.8) to prevent oscillations 3. Optional: Ensemble voting for critical applications where per-region tuning is infeasible These findings highlight a fundamental tradeoff: Dynamic strategy reduces manual tuning burden but requires robust monitoring infrastructure to detect and mitigate catastrophic misselection in production deployments.

Data Heterogeneity Robustness: Detailed Analysis

This section provides per-task breakdowns and robustness quantification under varying data heterogeneity, complementing the main paper’s Table 2. E.1

Experimental Setup

We evaluate three heterogeneity scenarios using Dirichlet αD ∈ {1.0, 0.5, 0.1} (Low/Medium/High). All methods use T5-Small with 10 communication rounds, 40 local epochs, batch size 32, AdamW (η=10−4 for baselines, 10−3 for RegionFed), gradient clipping C=1.0. This analysis uses 10 rounds (vs 50 in Table 1) to isolate heterogeneity effects from convergence speed. RegionFed tolerates higher learning rates because gradient-based regional adaptation filters conflicting updates before aggregation, effectively reducing gradient variance. E.2

Per-Task Performance Analysis

Table 11 presents per-task accuracy across heterogeneity levels. 25

Table 11: Per-Task Performance Under Data Heterogeneity (Accuracy %) Intent

Spell

NER

Method

Low Med High Low Med High Low Med

High

FedAvg FedProx Reg-Aware FedAvg Local FT FedTP

95.2 95.3 95.6 96.1 96.5

90.1 90.7 91.3 93.2 93.8

78.2 80.1 81.5 85.7 87.1

76.3 76.9 77.1 77.8 78.5

68.3 69.2 69.8 72.1 73.8

49.1 51.3 53.2 62.4 65.2

95.8 96.1 96.3 96.8 98.5

88.6 89.5 90.1 94.1 97.5

75.0 76.2 79.3 86.8 94.0

RF-Grad RF-Interp RF-Meta RF-Dynamic

96.9 96.8 97.1 97.0

94.5 94.3 95.1 94.8

89.2 89.8 91.3 90.7

79.2 79.5 80.1 80.0

74.6 75.1 76.2 75.8

67.3 69.1 71.2 70.5

98.2 98.3 98.9 98.7

96.0 96.4 97.7 97.3

89.8 91.0 93.8 93.5

RF = RegionFed. Low/Med/High = Dirichlet αD =1.0/0.5/0.1. Bold = best per column. Mean over 5 seeds.

Key observations: Spell correction is most sensitive to heterogeneity (FedAvg degrades 27.2pp vs RF-Meta’s 8.9pp), reflecting that regional spelling conventions and vocabulary vary dramatically. Intent classification shows moderate degradation (FedAvg 17.0pp vs RF-Meta 5.8pp). NER benefits strongly from regional entity vocabularies (FedAvg degrades 20.8pp vs RF-Meta 5.1pp). E.3

Robustness Quantification

Figure 7 visualizes the performance gap. RF-Meta degrades 6.5pp (Low→High) vs FedAvg’s 21.6pp, representing ∼3× better robustness. Under High heterogeneity (αD =0.1), RF-Meta leads FedAvg by +18.0pp (85.5% vs 67.5%). Performance Gap vs RegionFed-Meta (%) FedAvg

2.9

7.4

18.0

FedProx

2.5

6.6

16.3

18 16

Performance Gap (%)

Method

RegionFed-Dynamic

0.1

0.3

0.6

RegionFed-Grad

0.5

1.3

3.4

RegionFed-Interp

0.3

1.1

2.2

Medium

High

10 8 6 4 2

Low

Heterogeneity Level

Figure 7: Performance gap heatmap: RF-Meta vs other methods. The gap widens from Low (0.6– 2.9%) to High (3.4–18.0%) heterogeneity.

E.4

Loss Landscape Visualization

Figure 8 illustrates why heterogeneity degrades FedAvg: regional gradients conflict, and averaging cancels informative signal, trapping the global model in a flat compromise region. RegionFed navigates each region’s landscape independently via region-specific adaptations θr .

Why Parameter-Level Personalization Fails on Transformers

This section provides detailed analysis of why parameter-level personalized FL methods (SCAFFOLD, pFedMe, Ditto, APFL) collapse on transformer architectures, complementing Finding 3 in the main paper. We organize the analysis by failure family: control-variate (SCAFFOLD) and proximalregularization (pFedMe, Ditto, APFL), then explain why RegionFed’s gradient-level approach succeeds. The transformer-specific FedTP baseline (which does not collapse) is discussed separately in Section F.6. 26

Region 1 Optimum Region 2 Optimum

1.0

2.4

1.2

0 3.0.6 2.4

3.6

1.5

1.2

0.0

0.5 0.0 0.5 1.8

0.0

1.0 3.6

3.6

3.0

4.2

4 1.2 1.85.4 .8

3.6

2.0

1.2 1.8

1.5

2.0

3.6

Parameter Direction 1 ( )

Parameter Direction 2 ( )

1.8

5.4.8 4 .2 4 .6 3

(b) FedAvg Aggregated Loss (Compromise Solution) FedAvg Solution

1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0

2.0

Parameter Direction 2 ( )

(a) Client Loss Landscapes (High Heterogeneity =0.1)

0.6

Parameter Direction 2 ( )

2.0

Parameter Direction 1 ( )

RegionFed-R1 RegionFed-R2

Regional Adaptation

1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0

Parameter Direction 1 ( )

Figure 8: Loss landscape under high heterogeneity (α=0.1). (a) Regional loss contours show conflicting optima. (b) FedAvg’s aggregated loss shows gradient cancellation; regional optima (△, ∇) are suboptimal globally, forcing a compromise (⋆). (c) RegionFed finds region-specific optima (⋆) independently. F.1

Local Epoch Sweep: Collapse Is Independent of E

A natural concern is whether the parameter-level collapse at E=40 is simply an artifact of excessive client drift. To rule this out, we sweep E ∈ {1, 5, 10, 40} for SCAFFOLD and pFedMe on T5-Small (Amazon ESCI), keeping all other hyperparameters at their respective best settings: Method

E=1

E=5

E=10

E=40

SCAFFOLD pFedMe (best LR per E) FedAvg RegionFed-Meta

7.2% 6.8% 62.3% 78.4%

8.1% 7.4% 71.5% 86.2%

8.5% 8.0% 76.8% 89.7%

8.7% 9.1% 80.2% 92.3%

SCAFFOLD and pFedMe remain collapsed (<10%) at all local epoch settings, while FedAvg and RegionFed scale normally with E. This confirms the failure is fundamental to parameter-level corrections on transformer landscapes, not an artifact of excessive local computation. Note that pFedMe at its published default (λ=0.01, η=10−4 ) achieves 0% (Table 1); the 9.1% number above reflects the best-tuned LR at E=40 and is reported here only to confirm E-independence. F.2

Control-Variate Family: SCAFFOLD (8.73%)

SCAFFOLD [Karimireddy et al., 2020] maintains control variates ci per client to correct client drift: ! N X t t+1 t 1 (∇Fi (θi ) − ci + c) . θ =θ −η N (18) i=1

Failure Mechanism on T5-Small. (1) Ill-conditioned control variates: with 60.5M parameters, ci ∈ R60.5M accumulates floating-point errors across rounds, leading to gradient explosions (∥ci ∥ → ∞ for several clients). (2) Shared embedding interference: T5’s encoder-decoder shares 16.4M embedding parameters; control variates create catastrophic interference between encoding and decoding. (3) Attention coupling: Q/K/V projections create complex parameter interdependencies; per-parameter corrections produce inconsistent updates across coupled heads. SCAFFOLD achieves 8.73% (near random for 8-class intent), with training loss failing to decrease after round 2. F.3

Proximal-Regularization Family: pFedMe, Ditto, APFL

pFedMe [T. Dinh et al., 2020], Ditto [Li et al., 2021a], and APFL [Deng et al., 2020] all augment a personalized objective with a proximal penalty λ2 ∥θi − θ∥2 (Moreau envelope, local-global proximity, and adaptive mixing, respectively). Shared Failure Mechanism. The proximal penalty applies uniform regularization strength λ across all parameters. This is incompatible with transformer parameter heterogeneity: attention heads require rapid drift for regional adaptation while embeddings should remain stable. The role-agnostic 27

constraint either (a) over-constrains attention (preventing necessary specialization) or (b) underconstrains embeddings (causing drift in shared representations). Inner-loop minimization (pFedMe) further fails to converge within reasonable iterations on transformers; embedding parameters collapse to near-zero values producing uninformative representations. Hyperparameter Sweep. We swept λ ∈ {0, 0.001, 0.01, 0.1, 1.0} and η ∈ {1e-5, 5e-5, 1e-4, 5e-4} for both pFedMe (16 configs) and Ditto (20 configs): • λ=0: Both methods reduce to pure Local Fine-tuning (88.84%), confirming the optimization machinery is functional; the proximal mechanism alone triggers instability. • λ=0.001: Partial collapse (45–62%) with high variance. • λ=0.01 (default): Catastrophic collapse (pFedMe 0%, Ditto 8.92%; Table 1). • λ ≥ 0.1: Immediate divergence (NaN losses, 1012 + training loss). APFL exhibits the same pattern: its mixing coefficient between local and global models amplifies parameter-level interference. Switching between SGD and AdamW or applying gradient clipping (C=1.0) does not resolve the instability, confirming a fundamental incompatibility between proximal regularization and transformer landscapes rather than a tuning artifact. F.4

Why Gradient-Based Personalization Succeeds

RegionFed’s update θr = θ + ρ · (∇Lr (θ) − ∇L(θ)) avoids the failure modes above through three properties: (1) No persistent state: gradient differences are recomputed each round, avoiding accumulation of numerical errors that destabilize SCAFFOLD’s control variates. (2) Architectureagnostic operation: the difference ∇Lr − ∇L depends only on task gradients, not on parameter structure, embedding sharing, or attention coupling. (3) Bounded adjustments: ρ=0.0344 (goldensection optimum) yields small, stable adjustments that preserve pre-trained representations, in contrast to the proximal family’s uniform λ-weighted constraint that disrupts attention/embedding dynamics. F.5

Controlled Ablation: Isolating the Collapse Cause

To move beyond informal explanations and identify which transformer component causes parameterlevel collapse, we run SCAFFOLD on four controlled T5-Small variants, modifying one architectural feature at a time while keeping all other settings identical (50 rounds, E=40, AdamW, η=10−4 , Amazon ESCI): T5-Small Variant Default (tied emb + LayerNorm) Untied embeddings (tie_word_embeddings=False) Frozen LayerNorm (all LN params fixed) Untied emb + frozen LN Full model, rank-4 LoRA only

SCAFFOLD

FedAvg

8.73% 42.61% 38.47% 71.83% 76.24%

80.18% 79.84% 78.92% 78.15% 77.41%

Key findings. (1) Untying embeddings alone recovers SCAFFOLD to 42.61% (+33.9pp), confirming that shared encoder-decoder embeddings are the primary instability source: control variates push encoding and decoding embeddings in conflicting directions. (2) Freezing LayerNorm alone recovers to 38.47% (+29.7pp), confirming that LayerNorm’s scale/shift parameters amplify control-variate errors across layers. (3) Combining both fixes recovers SCAFFOLD to 71.83%, approaching FedAvg (78.15% under the same constrained architecture). The residual 6.3pp gap is attributable to attention coupling (Q/K/V interdependencies). (4) Restricting SCAFFOLD to LoRA-only updates (rank-4, 0.3% of parameters) achieves 76.24%, further confirming that the failure is tied to full-parameter updates on coupled transformer components. These controlled experiments transform the informal hypothesis (“shared embeddings and LayerNorm cause collapse”) into a verified finding. RegionFed’s gradient-level design avoids all three failure modes because the gradient-difference signal ∇Lr − ∇L is computed as a single vector operation that does not interact with parameter structure, tying, or normalization layers. 28

F.6

FedTP: Transformer-Specific Baseline Comparison

Unlike SCAFFOLD/pFedMe which fail completely, FedTP [Li et al., 2023], a transformer-specific personalization method, achieves competitive performance on T5-Small. We provide detailed comparison to understand the trade-offs between architecture-specific and architecture-agnostic approaches. FedTP Implementation. FedTP uses hypernetworks to generate personalized attention parameters for each client. Given a client embedding ec ∈ R64 , the hypernetwork generates low-rank adaptations ∆WQ , ∆WK , ∆WV for each transformer layer’s attention projections. We implemented FedTP following [Li et al., 2023] with: (1) 64-dimensional client embeddings; (2) rank-8 low-rank adaptation matrices; (3) personalization of all 12 transformer layers (6 encoder + 6 decoder); (4) cross-attention personalization for decoder layers. Experimental Results. FedTP on T5-Small with Amazon ESCI dataset: Metric

FedTP

RegionFed-Meta

∆

Overall Accuracy Intent Accuracy Spell Accuracy NER F1 RRS (Regional Fairness) Regional Variance

91.54% 95.58% 79.79% 99.25% 88.80% 0.81%

92.27% 97.64% 79.67% 99.50% 91.62% 0.97%

+0.73% +2.06% -0.12% +0.25% +2.82% +0.16%

Key Findings: 1. RegionFed outperforms FedTP on overall accuracy (+0.73pp) and regional fairness (+2.82pp RRS), demonstrating that architecture-agnostic gradient-level personalization can exceed architecture-specific methods. 2. NER and Spell are essentially tied (within 0.25pp), indicating that both architecture-specific attention personalization and architecture-agnostic gradient-conflict personalization handle structured token-level tasks comparably. 3. RegionFed provides superior regional equity (RRS 91.62% vs 88.80%), indicating more consistent performance across diverse regions. FedTP’s client-level personalization may overfit to dominant patterns within regions. 4. Both methods succeed where SCAFFOLD/pFedMe fail, confirming that either attention-based personalization (FedTP) or gradient-based personalization (RegionFed) can handle transformers, while parameter-level methods cannot. Architecture-Agnostic vs Architecture-Specific Trade-offs: • FedTP advantages: Client-level personalization; works well for attention-centric tasks • FedTP limitations: Requires architecture-specific hypernetwork design; must be reimplemented when migrating T5→Llama→Mamba; cannot apply to non-attention architectures • RegionFed advantages: Architecture-agnostic (works on any differentiable model); region-level aggregation provides statistical power; superior regional fairness • RegionFed limitations: Does not personalize attention specifically; requires regional structure Deployment Recommendation. For production systems where model architectures evolve rapidly (e.g., LLM serving infrastructure), RegionFed’s architecture-agnostic design provides future-proofing without sacrificing performance. FedTP may be preferred when architecture is stable and maximum task-specific personalization is required. F.7

FedBABU: Layer-Splitting Baseline Analysis

FedBABU [Oh et al., 2022] represents a layer-splitting approach to personalized FL: the model body (shared representation layers) is aggregated globally like FedAvg, while the classifier head is trained locally and never communicated. This avoids the parameter-level instabilities that cause SCAFFOLD/pFedMe/Ditto/APFL to collapse on transformers, since the body aggregation is identical to stable FedAvg and the head is isolated from cross-client interference. Implementation on T5-Small. For T5’s encoder-decoder architecture, we define the body as all transformer layers (encoder + decoder, ∼44M parameters) and the head as the language model 29

projection layer (lm_head, ∼16.4M parameters, tied with input embeddings). During local training, only the head is updated; the body is frozen and aggregated globally after each round. We untie the head from input embeddings during FedBABU training to allow independent personalization. Results Summary. FedBABU achieves 84.53% overall accuracy on T5-Small (Amazon ESCI), placing it between FedAvg (80.18%) and RegionFed-Meta (92.27%). It avoids the catastrophic collapse seen in SCAFFOLD (8.73%), pFedMe (0%), Ditto (8.92%), and APFL (9.15%), confirming that the collapse is caused by parameter-level manipulation (control variates, Moreau envelopes, ℓ2 regularization) rather than the FL setting itself. Why FedBABU Underperforms RegionFed. 1. Head-only personalization is insufficient: Regional semantic differences (e.g., “thongs” meaning footwear vs undergarments) are encoded in the transformer body’s attention patterns and intermediate representations, not just the output projection. FedBABU’s body is globally averaged, losing these distinctions. 2. No adaptive regional weighting: FedBABU aggregates bodies uniformly across all clients, equivalent to FedAvg on the body. RegionFed’s gradient-conflict mechanism (αr ) adaptively weights regional contributions, preserving regional specialization. 3. No hierarchical structure: FedBABU operates at the client level without regional coordination. RegionFed’s hierarchical aggregation (client → region → global) provides intermediate noise reduction and regional specialization. Cross-Architecture Stability. FedBABU remains stable across architectures: 85.72% on RoBERTaBase, 86.31% on T5-3B (Amazon ESCI), 62.84–65.47% on Amazon Reviews, and 83.48% on FEMNIST CNN (Table 1). This confirms that layer splitting avoids instabilities across both transformer and CNN architectures but cannot match gradient-level personalization (RegionFed-Meta: 92.27–94.12% on Amazon ESCI, 85.21% on FEMNIST). Heterogeneity Robustness. Under varying Dirichlet concentration αD (Table 2), FedBABU degrades 17.4pp from Low to High heterogeneity (89.8% → 72.4%), compared to RegionFed-Meta’s 6.5pp degradation (92.0% → 85.5%). The head-only personalization captures some local patterns but cannot adapt the body to regional gradient conflicts, leaving FedBABU vulnerable to distribution shifts that affect intermediate representations.

Convergence, Communication Efficiency, and Scalability

This section provides detailed convergence and scalability analysis, complementing the summary metrics in the main paper (Section 5.2). G.1

Convergence Analysis

Figure 2 (main paper) presents convergence trajectories over 50 communication rounds. Per-method analysis: Key Findings. (1) 4× speedup: RegionFed-Grad reaches 90.8% in round 1; FedAvg requires 5 rounds to reach 80% (75% communication cost reduction). (2) Sustained advantage: The ∼10pp gap between RegionFed-Meta (92.3%) and FedAvg (82%) remains constant from round 5 through round 50, confirming that additional communication rounds cannot compensate for the lack of regional personalization. (3) No degradation: RegionFed shows no accuracy degradation over 50 rounds; adaptive αr prevents overfitting to regional biases by maintaining global knowledge sharing (αr ≥ αmin ). (4) FedAvg plateau: FedAvg reaches 82% by round 20 and plateaus; the remaining gap is due to irreducible heterogeneity error Γ (Theorem I.5), which cannot be resolved by a single global model. (5) Parameter-level methods collapsed: SCAFFOLD stays at <10% even at round 50, confirming fundamental parameter space instability rather than insufficient training. G.2

Production Deployment Implications

The 4× convergence speedup translates to direct infrastructure savings. For 1M edge devices, reducing from 5 to 1 round saves ∼968 GB upload traffic and 16M local training epochs. RegionFed achieves both faster convergence and better final accuracy. Method selection: choose RegionFed30

Grad for consistency (predictable SLAs), RegionFed-Meta for fast convergence (rapid iteration), RegionFed-Interp for resource-constrained environments. Avoid FedAvg/FedProx for heterogeneous transformer systems (11–24 point accuracy gap, 4× slower convergence). G.3

Scalability Analysis

We analyze RegionFed’s scaling characteristics as the number of regions and clients increases. √ Theoretical Basis. RegionFed achieves O( N ) complexity through hierarchical coordination: regional models aggregate N/K clients with localized conflicts, then the global model coordinates K regions. In contrast, FedAvg/FedProx exhibit O(N ) complexity as global models aggregate all N clients directly, with conflicts increasing quadratically with heterogeneity. Empirical Results. We re-partition the Amazon ESCI dataset into 4 regions (40 clients, 5 categories each), 8 regions (80 clients, default), and 16 regions (160 clients, 2 categories each) and train all methods from scratch under identical hyperparameters. Table 12 reports all results from complete experimental runs (5 seeds each). Table 12: Scalability Analysis: Performance with Increasing Region Count (all experimental) Method

4 Reg.

8 Reg.

16 Reg.

Deg.↓

FedAvg 83.41±0.38% 80.18±0.31% 74.92±0.52% 8.49% FedProx 70.53±0.47% 67.18±0.42% 60.84±0.61% 9.69% Reg-Aware FedAvg 85.27±0.33% 83.47±0.29% 79.13±0.44% 6.14% FedTP 92.08±0.21% 91.54±0.22% 89.41±0.35% 2.67% RF-Grad 92.14±0.24% 91.92±0.25% 89.73±0.31% 2.41% RF-Meta 92.58±0.22% 92.27±0.31% 90.14±0.38% 2.44% RF-Interp 92.41±0.30% 92.13±0.28% 89.52±0.42% 2.89% Note: RF=RegionFed. All rows are from complete experimental runs (5 seeds). 4 regions: 40 clients (10 each), 5 categories/region. 16 regions: 160 clients (10 each), 2 categories/region. SCAFFOLD/pFedMe/Ditto/APFL omitted (<10% at all scales). Deg. = accuracy drop from 4 to 16 regions.

Key Findings. (1) 3.5× Better Scaling: RegionFed-Grad degrades only 2.41% (4→16 regions) vs FedAvg’s 8.49%; at 16 regions (160 clients), RegionFed-Meta maintains 90.14% while FedAvg drops to 74.92%. (2) Gap Widens with Scale: RegionFed advantage increases from +9.2pp (4 regions) to +15.2pp (16 regions). (3) Why RegionFed Scales: Hierarchical aggregation localizes gradient conflicts √ with O(N/K + K) complexity (optimal at K√= N ); within-region homogeneity via Law of Large Numbers reduces gradient variance as O(1/ nk ); adaptive α self-regulates to heterogeneity level. (4) Why FedAvg/FedProx Fail: Direct client aggregation creates O(N 2 ) gradient conflicts with no localization; FedProx’s fixed proximal term over-regularizes at scale. Deployment Recommendations. Small scale (≤50 clients): FedAvg acceptable. Medium scale (50-100 clients): RegionFed provides +10 point advantage. Large scale (>100 clients): √ RegionFed is essential, as FedAvg drops below 90% threshold. Optimal configuration: K ≈ N with 10-20 clients per region. G.4

Region Misspecification Robustness

A key concern is whether RegionFed requires perfectly coherent region definitions. We evaluate robustness by comparing semantically-coherent regions (as used in main experiments) against increasingly misspecified configurations: Key Findings. (1) Even with random region assignment (destroying all semantic coherence), RegionFed achieves 88.1%, outperforming FedAvg by +7.9pp. This confirms that the gradientconflict mechanism extracts useful signal from any client grouping: regions that happen to share similar gradients receive lower αr , while divergent regions receive higher αr , regardless of whether the grouping was semantically designed. (2) Geographic proxy regions (clients grouped by simulated location rather than product category) lose only 1.2pp, suggesting that geographic proximity serves as a reasonable proxy for semantic similarity in retail settings. (3) Adversarial assignment (placing maximally dissimilar clients together) degrades by 5.6pp but still exceeds FedAvg by +6.5pp, because intra-region aggregation still provides noise reduction benefits even when regions are heterogeneous. 31

Table 13: Region Misspecification Analysis (RegionFed-Meta, 8 Regions) Region Assignment

Acc.

Semantic (default) 92.3% Geographic proxy 91.1% Random assignment 88.1% Adversarial 86.7%

∆ – –1.2% –4.2% –5.6%

Description Category-coherent By location, mixed categories Uniform random Maximally heterogeneous

FedAvg (no regions) 80.2% –12.1% Baseline

Why RegionFed tolerates misspecification. The adaptive αr mechanism self-corrects: in misspecified regions, intra-region gradient conflict is high, so αr saturates near 1.0 (maximal personalization), effectively bypassing the noisy regional aggregation. This graceful degradation means RegionFed never performs worse than a flat (non-hierarchical) gradient-weighted scheme.

Component Ablation Study

This section quantifies individual and synergistic contributions of RegionFed’s key components: (1) Adaptive α (gradient conflict resolution via dynamic balancing), (2) Hierarchical Personalization (two-level region→client optimization), and (3) Golden Section ρ (automatic regularization tuning). We evaluate all 23 = 8 configurations from baseline (regional structure only) to full system. Results. Table 14 presents component contributions. Table 14: Ablation Study: Component Contributions Configuration

Acc.

∆

Baseline (Regional Only) Reg-Aware FedAvg (no adaptive α) + Hierarchical Personalization + Golden Section ρ + Adaptive α

88.5% 83.5% 89.4% 88.9% 89.6%

– – +0.9% +0.5% +1.1%

+ Hier. + ρ + Hier. + α +ρ+α

89.8% 90.8% 90.2%

+1.4% +2.3% +1.8%

Full System

91.3% +2.8%

Sum of individual: +2.5%, Synergy: +0.3% (12.8%)

Component Analysis. Adaptive α (+1.12%, 39.7% of improvement): Most critical; resolves gradient conflicts via θknew = α · θkregional + (1 − α) · θglobal ; higher α when regions disagree. Hierarchical Personalization (+0.93%, 33.0%): Two-level optimization captures regional patterns and client preferences; combined with α: +2.29% (1.29× synergy). Golden Section ρ (+0.45%, 16.0%): Auto-tunes regularization per region, eliminating manual tuning. Synergistic Effects. Components exceed individual sum by +0.32% (12.8% synergy): α + Hierarchical provides better interpolation targets (+0.24%); ρ + α enables more effective balancing (+0.18%); all three create a feedback loop (hierarchy → ρ optimizes → α balances). Recommendations. Priority: (1) Adaptive α [CRITICAL], (2) Hierarchical [HIGH], (3) ρ [MODERATE]. Minimal viable system: Hierarchical + α provides 90.75% (81% of full improvement). Full system recommended for 90%+ production requirements. H.1

Temperature Sensitivity Analysis

The temperature parameters τr and τu in Equations 6-7 control the sensitivity of adaptive weight computation. We evaluate sensitivity across τ ∈ {0.5, 0.75, 1.0, 1.5, 2.0}. Key Findings. (1) Performance is robust to τ variation: ±1.2% accuracy across tested range; (2) τ = 1.0 achieves optimal balance between sensitivity and stability; (3) Lower τ (0.5) creates sharp 32

Table 15: Temperature Sensitivity (τr = τu ) τ

Overall Acc.

RRS

Reg. σ

0.5 0.75 1.0 1.5 2.0

91.1% 91.8% 92.3% 91.9% 91.0%

90.8% 91.3% 91.6% 91.2% 90.5%

1.21 1.05 0.97 1.08 1.34

transitions that can overcorrect, causing instability (higher σ); (4) Higher τ (2.0) dampens gradient conflict signals, reducing personalization effectiveness; (5) The [αmin , 1.0] output range (default αmin = 0.5) ensures all regions receive meaningful personalization; setting αmin = 0 caused 3.2% accuracy drop in ablation (regions with aligned gradients lost beneficial regional specialization). H.2

Centering Threshold µr Sensitivity Analysis

The centering threshold µr in Eq. 6 is calibrated from round-1 gradient conflicts. We evaluate robustness by perturbing µr from its calibrated value: Table 16: Sensitivity to µr Perturbation (RegionFed-Meta) µr Setting

Overall Acc.

αr Range

∆

0.5 × µcal r 0.75 × µcal r µcal (calibrated) r 1.25 × µcal r 1.5 × µcal r

91.5% 91.9% 92.3% 92.0% 91.6%

[0.68, 0.95] [0.61, 0.92] [0.53, 0.88] [0.51, 0.82] [0.50, 0.76]

–0.8% –0.4% – –0.3% –0.7%

Fixed µr = 0 Adaptive (running mean)

90.1% 91.7%

[0.82, 0.97] –2.2% [0.54, 0.86] –0.6%

Key Findings. (1) Robust to ±50% perturbation: accuracy varies only ±0.8% when µr is perturbed by up to 50% from its calibrated value. This robustness arises because the sigmoid function saturates: moderate shifts in µr change αr by <0.05 in the tails. (2) Fixed µr = 0 fails: without centering, all regions receive high αr (compressed into [0.82, 0.97]), eliminating the discrimination between high- and low-conflict regions and degrading accuracy by 2.2pp. (3) Calibrated > adaptive: fixed calibration from round 1 outperforms running-mean adaptive updates by +0.6pp. Adaptive centering inflates µr in later rounds (as gradient magnitudes decrease with convergence), artificially reducing αr and weakening necessary personalization. (4) Round-1 calibration is sufficient: the DP-clipped gradient norms at round 1 provide a bounded, low-variance estimate of typical conflict magnitudes (bounded by 2C = 2.0), making the initial calibration reliable. H.3

Justification for α ∈ [αmin , 1.0] Range

We ablate the minimum personalization weight αmin to justify the default αmin = 0.5: Table 17: Effect of Minimum Personalization Weight αmin α Range

Overall Acc.

∆

[0.0, 1.0] [0.25, 1.0] [0.5, 1.0] [0.75, 1.0]

89.1% 90.8% 92.3% 91.4%

-3.2% -1.5% – -0.9%

Analysis. The [0.0, 1.0] range allows α ≈ 0 for well-aligned regions, effectively discarding regional adaptation. This is suboptimal because even aligned regions benefit from regional vocabulary specialization (e.g., Electronics region develops better technical term recognition). Setting αmin = 33

0.5 ensures all regions receive at least 50% personalization weight, preserving beneficial specialization while allowing stronger personalization for divergent regions. H.4

DP Clipping Threshold C Sensitivity Analysis

The DP clipping threshold C controls the maximum gradient norm before noise addition p (Algorithm 2). 2 Because DP noise is calibrated as N (0, σdp C 2 I), the privacy guarantee ϵ ≈ q T log(1/δ)/σdp is independent of C for fixed σdp . However, C critically affects utility through two competing mechanisms: (1) clipping bias: smaller C truncates informative gradient magnitude, compressing the conflict signal ∥g̃r − g̃∥2 into [0, 2C]; (2) noise variance: larger C increases absolute noise (σdp · C per coordinate), degrading the signal-to-noise ratio. We evaluate C ∈ {0.1, 0.5, 1.0, 5.0, ∞} on RegionFed-Meta (T5-Small, Amazon ESCI), where C=∞ denotes no DP mechanism (no clipping, no noise). Table 18: DP Clipping Threshold C Sensitivity (RegionFed-Meta, σdp =4.0) C

Overall Acc.

RRS

αr Range

Effect

0.1 0.5 1.0 5.0 ∞

0.60 0.60 0.60 0.60 ∞

89.14% 91.48% 92.27% 91.82% 92.53%

88.52% 90.87% 91.62% 91.15% 91.85%

[0.50, 0.54] [0.51, 0.72] [0.53, 0.88] [0.50, 0.94] [0.50, 0.96]

Severe clipping bias Moderate clipping Balanced (default) High noise variance No DP (upper bound)

Key Findings. (1) Privacy is C-independent: all finite-C settings achieve identical ϵ=0.60 because the noise-to-sensitivity ratio σdp is fixed; the choice of C affects only utility, not privacy. (2) C=1.0 is optimal among DP settings: it achieves 92.27%, only 0.26pp below the non-private upper bound (C=∞, 92.53%). (3) Small C destroys the conflict signal: at C=0.1, all gradients are clipped to norm 0.1, compressing the adaptive weight range to [0.50, 0.54]; the αr mechanism loses its ability to discriminate between high- and low-conflict regions, degrading accuracy by 3.13pp. (4) Large C increases noise variance: at C=5.0, noise standard deviation per coordinate is σdp · C = 20.0 (vs 4.0 at C=1.0), introducing sufficient noise to degrade performance by 0.45pp despite preserving gradient direction. (5) The privacy-accuracy tradeoff is favorable: moving from no privacy (C=∞) to strong privacy (ϵ=0.60, C=1.0) costs only 0.26pp, confirming that DP clipping at C=1.0 provides an excellent balance between gradient signal preservation and noise calibration. H.5

Per-Region αr Trajectories

Figure 9 plots the adaptive weight αr for each of the 8 regions across 5 training rounds. Key observations: (1) All αr values stabilize after round 2, confirming that round-1 calibration of µr is sufficient; (2) High-heterogeneity regions (Electronics, Grocery) maintain αr ≈ 0.82–0.88, reflecting strong gradient conflict with the global model; (3) Low-heterogeneity regions (Books, Health) settle at αr ≈ 0.53–0.58, close to the αmin floor, indicating alignment with global gradients; (4) The separation between high- and low-conflict regions is consistent across strategies (Grad, Meta, Dynamic), confirming that gradient conflict provides a stable heterogeneity signal independent of the personalization method.

Theoretical Analysis

This section provides the complete theoretical foundation for RegionFed, including assumptions, convergence analysis, personalization-generalization trade-offs, and privacy guarantees. I.1

Convergence Analysis

RegionFed’s convergence properties prove it leads the global and personalized models to appropriate optima. Our analysis explores an optimality gap R, defined as: 34

Per-Region r Trajectories Across Training Rounds Electronics Fashion Home Grocery

r (Personalization Weight)

0.9

Sports Books Automotive

Health min = 0.5 floor Calibration (Round 1)

0.8 0.7 0.6 0.5 1

Communication Round

Figure 9: Per-region αr trajectories across training rounds. High-conflict regions (Electronics, Grocery) maintain αr ≈ 0.85, while low-conflict regions (Books, Health) settle near αr ≈ 0.55. All regions stabilize after round 2.

1X F (θt ) − F ∗ T t=1

(19)

which quantifies the difference between global training loss F (θt ) and the optimal solution loss F ∗ through T rounds. We first present the following assumptions, which are widely adopted in related studies. Assumption I.1 (Smoothness). The loss function F : Rn → R is L-smooth such that for any v, w ∈ Rn , we have: ∥∇F (v) − ∇F (w)∥2 ≤ L∥v − w∥2 (20) Assumption I.2 (Bounded Variance). For any region r, the variance of stochastic gradients is bounded: E∥∇Lr (θ, ξr ) − ∇Lr (θ)∥22 ≤ σr2 (21) Assumption I.3 (Bounded Gradient Dissimilarity). The gradient dissimilarity across regions is bounded: ∥∇Lr (θ) − ∇L(θ)∥22 ≤ Γ2R (22) Assumption I.4 (Bounded Region-specific Adaptation). Region-specific adaptations are bounded: ∥θr ∥22 ≤ Br2

(23)

Empirical Validation of Assumption I.4. We verified this assumption empirically during training. Across all 8 regions and 50 communication rounds, we measured ∥θr ∥2 /∥θ∥2 (ratio of regional adaptation norm to global model norm). Results: mean ratio = 0.034 (std = 0.008), maximum ratio = 0.052 across all regions and rounds. This confirms ∥θr ∥ ≤ 0.06∥θ∥ holds throughout training, validating the bounded adaptation assumption with Br ≈ 0.06∥θ∥. The small ratio is expected because RegionFed uses ρ = 0.0344 (optimized via golden section search), which inherently constrains adaptation magnitude. We now present our convergence theorem for RegionFed. Theorem I.5 √ (Convergence Rate). Under Assumptions √ I.1–I.4, with appropriate learning rates η = O(1/ T ) and adaptation intensity ρ = O(1/ T ), RegionFed with the Grad adaptation strategy satisfies: 35

T −1

1 X 2(F (θ0 ) − F ∗ ) E∥∇F (θt )∥2 ≤ + ηLσ 2 + ηΓ2R T t=0 ηT

(24)

and for any region r, the personalized model satisfies: T −1

2(Fr (θr0 ) − Fr∗ ) 1 X E∥∇Fr (θt + αr θrt )∥2 ≤ + ηLσr2 + η(1 − αr )2 Γ2R T t=0 ηT

(25)

where F ∗ is the optimal value of the global loss, Fr∗ is the optimal value of the regional loss, L is the smoothness parameter, σ 2 bounds the global variance, σr2 bounds the regional variance, and Γ2R quantifies the heterogeneity across regions. √ This theorem establishes that RegionFed achieves an O(1/ T ) convergence rate for both global and personalized models even under heterogeneous data conditions, matching the optimal rate for non-convex optimization in federated settings. I.2

Personalization-Generalization Trade-off

We further analyze the trade-off between personalization and generalization: Theorem I.6 (Personalization Bound). For any region r, under Assumptions I.1–I.4, the expected loss under the personalized model satisfies: E[Fr (θt + αr θrt )] ≤ Fr (θr∗ ) + L(1 − αr )2 ∥θr∗ − θ∗ ∥22 + 2Lαr2 Br2

(26)

where θr∗ is the optimal model for region r and θ∗ is the optimal global model. This theorem provides a theoretical foundation for our adaptive personalization mechanism, showing that the personalization weight αr should be: αropt =

∥θr∗ − θ∗ ∥22 ∥θr∗ − θ∗ ∥22 + 2Br2

(27)

The asymmetric coefficients (L vs 2L) reflect that the adaptation bound (Br2 ) enters through an additional triangle inequality step. In practice, since Br ≈ 0.06∥θ∥ (Assumption I.4 validation), the 2Br2 term remains small, and the optimal αr is primarily governed by the regional dissimilarity ∥θr∗ − θ∗ ∥, which is precisely what our adaptive mechanism captures. I.3

Privacy Analysis

We analyze the privacy guarantees of our approach: Theorem I.7 (Privacy Guarantee). RegionFed with secure aggregation and user-level sampling provides (ϵ, δ)-differential privacy with:

ϵ=O

! p T log(1/δ) σdp

(28)

where q is the client sampling ratio, σdp is the noise multiplier, and T is the number of communication rounds. DP noise is applied at the regional level (Algorithm 2), with user-to-region privacy ensured by secure aggregation. With our parameters (C=1.0, σdp =4.0, T =50, q=0.1, δ=10−5 ), the moments accountant yields ϵ ≈ 0.60. √ This theorem shows that our approach provides formal privacy guarantees that scale as O( T ). With T =50 rounds, ϵ ≈ 0.60 remains well within the strong privacy regime (ϵ < 1). 36

Privacy-Utility Tradeoff Analysis

To empirically characterize the privacy-utility frontier, we evaluate RegionFed-Meta and FedAvg across five noise multiplier settings σdp ∈ {1.0, 2.0, 4.0, 8.0, 16.0} on T5-Small (Amazon ESCI), with all other hyperparameters fixed (T =50, q=0.1, C=1.0, δ=10−5 ). Table 19: Privacy-utility tradeoff: accuracy (%) at varying privacy budgets σdp ϵ (approx.)

1.0 2.40

2.0 1.20

4.0 0.60

8.0 0.30

16.0 0.15

FedAvg RF-Meta

80.85 92.54

80.52 92.41

80.18 92.27

78.93 91.48

74.21 88.15

∆ (RF-Meta − FedAvg) +11.69 +11.89 +12.09 +12.55 +13.94 Key observations: (1) RegionFed-Meta maintains >91% accuracy from ϵ=2.40 down to ϵ=0.30, degrading only 0.79pp across this 8× privacy range. (2) FedAvg degrades 6.64pp over the same range, indicating RegionFed is more noise-robust. (3) RegionFed’s advantage over FedAvg increases under stronger privacy (from +11.69pp at ϵ=2.40 to +13.94pp at ϵ=0.15), suggesting gradient-conflict-based adaptation provides implicit denoising. (4) At the strictest setting (ϵ=0.15), RegionFed-Meta (88.15%) still exceeds FedAvg’s best non-private result (80.85% at ϵ=2.40). Results: mean over 5 seeds.

The privacy-utility analysis reveals that RegionFed is more noise-tolerant than standard FL: regional aggregation averages DP noise across clients within each region before personalization, providing an implicit noise reduction effect proportional to the number of clients per region.

Supplementary Algorithms and Extensions

This section presents auxiliary algorithms, additional validation experiments, and future research directions. K.1

Topology-Based Model Similarity Analysis

Evaluating performance similarity between regional models is essential for adaptation-based personalization. However, it is non-trivial due to high dimensionality and non-linear features of neural networks. Standard metrics like Euclidean distance or cosine similarity among model parameters don’t guarantee correlation with performance [Kornblith et al., 2019]. Our solution leverages topology, i.e., the adjacency of data points in the representation space [McNeill and Vijayakumar, 2015], to construct a neighborhood relationship graph based on the similarity between query samples. The key idea is to analyze a model’s knowledge through its reaction to a shared set of test inputs represented as a topological structure. K.1.1

Neighborhood Graph Construction

To construct the neighborhood graph G, we define: • Points: Query representations (embeddings from the encoder) • Edges: Relationships between queries based on semantic similarity • Edge Weights: Similarity scores between query representations We model the conditional probability distribution with a cosine similarity-based affinity metric: G = {pi|j |pi|j = PN

(2 − (dij − ρj ))

k=1,k̸=j (2 − (djk − ρj ))

, 0 < i, j ≤ N }

(29)

where: • pi|j : conditional probability that the i-th query is the neighbor of the j-th query in the feature space of G 37

• ρj : distance between the j-th data point and its nearest neighbor ⟨e ,e ⟩

• dij : cosine distance between the embeddings of queries i and j: dij = 1 − ∥ei ∥i2 ∥ejj ∥2 • N : number of queries in the validation set By subtracting ρj , we ensure local connectivity of the manifold, avoiding isolated points and thereby better preserving the global structure [McNeill and Vijayakumar, 2015]. K.1.2

Cross-Region Knowledge Transfer Loss

When adapting models across regions, we want to ensure that region-specific knowledge is captured while still maintaining global performance. We use the topological relationship between regional and global models to guide the adaptation process: Ltopo = E(q,c)∈Dval [S(Gglobal , Gr )]

(30)

where: • Gglobal : neighborhood graph constructed from global model embeddings • Gr : neighborhood graph constructed from regional model embeddings • S(·, ·): similarity measure between two graphs • Dval : validation set with queries q and contexts c To measure the similarity of Gglobal and Gr in capturing flexible relationships, we employ crossentropy loss. Considering their respective conditional probabilities pi|j (global) and qi|j (regional): XX 1 − pi|j pi|j + (1 − pi|j ) log Ltopo = CE(Gglobal , Gr ) = pi|j log qi|j 1 − qi|j i j

(31)

This can be simplified using KL divergence: Ltopo =

KL(Pj ∥Qj )

(32)

where Pj = (p1|j , p2|j , . . . , pN |j ) and Qj = (q1|j , q2|j , . . . , qN |j ) are the probability distributions over neighbors for query j. K.1.3

Integration with RegionFed

The topology-based loss Ltopo can be incorporated into the adaptation parameter optimization by adding it to the regional loss evaluation during golden section search (Algorithm 3, Line 7): t t r [L(θ + ρ · θr ; q, y)] + λtopo Ltopo (θ + ρ · θr ) L(ρ) = E(q,y)∈Dval

(33)

where λtopo ∈ [0, 1] controls the trade-off between: • Regional specialization (low λtopo ): Focus on regional task performance • Global knowledge preservation (high λtopo ): Maintain structural similarity to global model K.1.4

Experimental Analysis

In our experiments, we find that: When topology loss is beneficial: Extreme heterogeneity (α = 0.1): Topology loss provides +1.2% improvement by preventing over-specialization. Small regional datasets: When |Dr | < 200 samples, topology loss acts as regularization (+0.8% improvement). Cold-start regions: New regions with limited data benefit from structural knowledge transfer (+1.5% improvement). 38

When adaptive weight alone suffices: Moderate heterogeneity (α = 0.5): Adaptive αr captures regional variation sufficiently (topology loss: +0.1% improvement, not statistically significant). Large regional datasets: When |Dr | > 500 samples, regions have sufficient data for effective adaptation without topology guidance. Wellrepresented regions: Regions with balanced label distributions don’t require additional structural constraints. Recommended usage: Use λtopo = 0.1 for extreme heterogeneity scenarios; otherwise set λtopo = 0 (default RegionFed-Meta configuration) for computational efficiency. K.1.5

Computational Complexity

Computing topology loss requires: Graph construction: O(N 2 d) where N is validation set size, d is embedding dimension; KL divergence computation: O(N 2 ); Total per evaluation: O(N 2 d). For N = 200 validation samples and d = 512 embedding dimension, topology loss adds 50ms per evaluation. This is acceptable for offline optimization but may be prohibitive for real-time deployment. The adaptive weight approach (RegionFed-Meta default) avoids this overhead while achieving 98% of the performance. K.2

Golden Section Search Algorithm

The golden section search algorithm optimizes the adaptation intensity ρ for each region without requiring gradient computation. This derivative-free optimization method is particularly suitable when the objective function Lr (θt + ρ · θr ) is unimodal but may not be differentiable with respect to ρ. Unimodality Verification. We empirically verified the unimodality assumption by exhaustive grid search over ρ ∈ [0, 2] with step size 0.05 for all 8 regions across multiple communication rounds. For each region, we plotted Lr (ρ) and confirmed: (1) all curves exhibited single-minimum unimodal shape with no local minima observed, (2) the optimal ρ values from grid search matched golden section search results within tolerance ϵ = 0.02, and (3) the loss landscape showed smooth, convexlike behavior near the optimum. This validates that golden section search is appropriate for our setting. Algorithm 3 Golden Section Search for Optimal ρ Input: Region r, global model θt , strategy s ∈ {grad, interp, meta}, search interval [a, b], tolerance ϵ Output: Optimal adaptation intensity ρopt √ Initialize: ϕ = 5−1 (golden ratio conjugate) 2 c ← b − ϕ(b − a), d ← a + ϕ(b − a) Compute regional adaptations: θrc ← C OMPUTE A DAPTATION(s, θt , r, c) Compute regional adaptations: θrd ← C OMPUTE A DAPTATION(s, θt , r, d) Evaluate: fc ← Lr (θt + c · θrc ), fd ← Lr (θt + d · θrd ) while |b − a| > ϵ do if fc < fd then b ← d, d ← c, fd ← fc c ← b − ϕ(b − a) θrc ← C OMPUTE A DAPTATION(s, θt , r, c) fc ← Lr (θt + c · θrc ) else a ← c, c ← d, fc ← fd d ← a + ϕ(b − a) θrd ← C OMPUTE A DAPTATION(s, θt , r, d) fd ← Lr (θt + d · θrd ) end if end while return ρopt = a+b 2

Complexity Analysis. The algorithm requires O(log(1/ϵ)) iterations to achieve tolerance ϵ. With ϵ = 0.01 and search interval [0, 2], this requires approximately 10-12 function evaluations. Each evaluation involves computing the regional loss on a validation set, making the total cost O(log(1/ϵ) · |Drval |) where |Drval | is the regional validation set size. K.3

Cross-Architecture and Cross-Scale Validation

To validate that RegionFed’s gradient-level personalization is truly architecture-agnostic, we evaluated on two additional backbones without any algorithmic modification. Cross-Architecture: RoBERTa-Base (125M parameters). We evaluated RegionFed-Grad versus FedAvg on the intent classification task using an encoder-only RoBERTa-Base backbone. FedAvg achieved 81.2% accuracy, while RegionFed-Grad achieved 91.5%, a 10.3pp improvement, confirming that gradient-conflict personalization operates seamlessly on encoder-only architectures without architecture-specific adapters. Cross-Scale: T5-3B (3 billion parameters). We scaled the backbone from T5-Small (60.5M) to T5-3B. RegionFed-Meta achieved 94.12% overall accuracy, a +1.85pp improvement over our T5-Small results (92.27%). Because the adaptive weighting relies on O(1) scalar projections of gradient norms (bounded by DP clipping C), the computational overhead of the personalization mechanism remained constant relative to the base training cost. This confirms that RegionFed scales efficiently to billion-parameter models. These results demonstrate that RegionFed’s gradient-level operations are genuinely architectureagnostic: the same algorithm, with identical hyperparameters for the personalization mechanism, generalizes across encoder-only (RoBERTa) and encoder-decoder (T5) architectures, and across two orders of magnitude in model scale (60.5M to 3B parameters). K.4

Asynchronous Federated Operations

RegionFed’s current implementation uses synchronous communication rounds (Algorithm 2), where the global server waits for all regions before updating. This section discusses the extension to asynchronous operation for production deployment. Asynchronous Regional Updates. In production, regions may have varying compute speeds and network latency. RegionFed supports asynchronous operation by allowing regions to submit gradient updates independently: θt+1 = θt − η · g̃rt−τr (34) where τr is the staleness of region r’s update. The adaptive αr mechanism naturally accommodates staleness: stale gradients produce larger conflicts with the current global gradient, increasing αr and thus strengthening regional personalization to compensate for the outdated global model. Staleness Bounds. We bound maximum staleness at τmax = 3 rounds to prevent divergence. Under√this bound, the convergence guarantee (Theorem I.5) degrades gracefully: E[∥∇F (θT )∥2 ] ≤ 2 σ 2 /T ). For τmax = 3, the additional error term is small relative to the heterogeneity O(1/ T + τmax term. Partial Aggregation. The server can aggregate partial regional updates using available gradients: P ′ g̃ t = |S1t | r∈St g̃rt where St ⊆ R is the set of regions that have submitted updates and t′ ≤ t is the round of each region’s latest submission. This is compatible with the DP mechanism since each regional gradient is independently clipped and noised. Production Considerations. (1) Heartbeat protocol: regions send periodic heartbeats; if a region misses 3 consecutive rounds, the server proceeds without it. (2) Gradient caching: regions cache their latest gradient to enable immediate participation when reconnecting. (3) Adaptive round duration: the server adjusts round duration based on the fastest p% of regions (default p = 80), balancing freshness and participation. Full asynchronous evaluation with real-world network conditions remains future work. K.5

Future Directions and Extensions

This section discusses extensions and future research directions motivated by our findings. 40

K.5.1

Automatic Region Discovery

The current RegionFed implementation requires pre-defined regional boundaries. We outline a gradient-based clustering approach for automatic region discovery: Proposed Algorithm: (1) Initialize with random regional assignments; (2) Compute gradient similarity matrix Sij = cos(∇Li , ∇Lj ) across clients; (3) Apply spectral clustering on S to identify K natural groupings; (4) Update regional boundaries based on clustering; (5) Iterate until convergence. The key insight is that clients with similar gradient directions should be in the same region, as they share similar data distributions. Preliminary Analysis: Our gradient conflict analysis (Eqs. 6–7 in main paper) already computes the similarity signals needed for clustering. Extending RegionFed to dynamic region discovery requires: (1) periodic re-clustering (every Tcluster rounds), (2) migration protocols for clients changing regions, and (3) warm-starting regional models for newly formed regions. We leave full implementation and evaluation to future work. K.5.2

Scaling to Larger Transformers

While our experiments focus on T5-Small (60.5M parameters), we provide theoretical arguments for scalability: Why Gradient-Based Operations Scale: RegionFed’s personalization formula θr = θ + ρ · (∇Lr − ∇L) involves only gradient computation and subtraction, operations whose complexity is O(|θ|) and independent of model architecture. In contrast, SCAFFOLD’s control variates require maintaining ci ∈ R|θ| vectors that accumulate numerical errors proportionally to |θ|, and pFedMe’s proximal term ∥θi − θ∥2 produces penalties scaling with |θ|. Expected Behavior at Scale: We hypothesize that RegionFed will maintain effectiveness on T5-Base (220M) and T5-Large (770M) because: (1) gradient conflicts remain informative regardless of model size; (2) the optimal ρ discovered via golden section search will self-adjust to appropriate magnitudes; (3) architecture-agnostic operations avoid the parameter-count-dependent failures observed in SCAFFOLD/pFedMe. Empirical validation on larger models remains future work. K.5.3

Empirical Privacy-Utility Tradeoffs

We evaluate the accuracy-privacy tradeoff by varying DP noise σdp ∈ {1, 2, 4, 8, 16}, yielding ϵ ∈ {2.40, 1.20, 0.60, 0.30, 0.15} (via moments accountant with T =50, q=0.1, δ=10−5 ). Full results are presented in Table 19 (Section J). RegionFed-Meta maintains >91% accuracy down to ϵ=0.30 and achieves 88.15% at ϵ=0.15. The advantage over FedAvg increases under stronger privacy (+13.94pp at ϵ=0.15 vs +11.69pp at ϵ=2.40), because regional aggregation provides 10× more data per gradient estimate, partially compensating for DP noise. Further directions include formal composition with Rényi DP accountant and comparison with DP-FTRL [Kairouz et al., 2021].

Complete Theoretical Proofs

This section provides complete proofs for all theorems presented in the main paper. L.1

Proof of Theorem I.5 (Convergence Rate)

Theorem I.5 (Convergence Rate): Under Assumptions I.1–I.4, with constant learning rate η ≤ 1/L, RegionFed achieves: T −1

1 X 2(F (θ0 ) − F ∗ ) E∥∇F (θt )∥2 ≤ + ηLσ 2 + ηLΓ2R T t=0 ηT √ √ Setting η = min(1/L, 1/ T ) yields the O(1/ T ) rate stated in the main paper. Complete Proof: 41

(35)

We prove the convergence rate using the standard descent lemma approach with a constant learning PM rate η. Let F (θ) = r=1 pr Fr (θ) where pr = nr /n is the proportion of data in region r. The stochastic gradient estimator (with all M regions participating, i.e., |St | = M ) is: M

gt =

1 X t g̃ M r=1 r

(36)

where g̃rt is the DP-noised regional gradient. Step 1: Bounding the expected gradient variance The variance of the stochastic gradient decomposes into two terms: intra-region stochastic noise and inter-region heterogeneity: E[∥g t − ∇F (θt )∥22 ] ≤

σ2 M |{z}

stochastic noise

Γ2R |{z}

(37)

heterogeneity

PM 1 t where σ 2 bounds the per-region gradient variance (Assumption I.3) and Γ2R = M r=1 ∥∇Fr (θ ) − t 2 2 2 ∇F (θ )∥2 is the regional heterogeneity. The DP noise contributes an additional dσdp C /M term (absorbed into σ 2 for clarity, since it is O(1/M )). By the L-smoothness of F (Assumption I.1): L F (θ′ ) ≤ F (θ) + ⟨∇F (θ), θ′ − θ⟩ + ∥θ′ − θ∥22 2

Step 2: Applying smoothness

(38)

Applying with θt+1 = θt − η g t : F (θt+1 ) ≤ F (θt ) − η⟨∇F (θt ), g t ⟩ + Step 3: Taking expectations aggregation):

Lη 2 t 2 ∥g ∥2 2

(39)

Taking expectations and using E[g t ] = ∇F (θt ) (unbiased after

Lη 2 σ2 E[F (θt+1 )] ≤ F (θt ) − η∥∇F (θt )∥22 + + Γ2R ∥∇F (θt )∥22 + 2 M 2 2 Lη σ Lη + Γ2R = F (θt ) − η 1 − ∥∇F (θt )∥22 + 2 2 M

(40) (41)

Step 4: Telescoping and convergence rate For η ≤ 1/L, we have (1 − Lη/2) ≥ 1/2. Summing from t = 0 to T − 1: T −1 η X Lη 2 T σ 2 t 2 0 T 2 E[∥∇F (θ )∥2 ] ≤ F (θ ) − E[F (θ )] + + ΓR (42) 2 t=0 2 M Lη 2 T σ 2 2 0 ∗ + ΓR (43) ≤ F (θ ) − F + 2 M Dividing by ηT /2: T −1

1 X 2(F (θ0 ) − F ∗ ) E[∥∇F (θt )∥22 ] ≤ + Lη T t=0 ηT Setting η = min

1 L,

2(F (θ 0 )−F ∗ ) LT (σ 2 /M +Γ2R )

σ2 + Γ2R M

(44)

√ = O(1/ T ) balances the two terms:

T −1 1 1 X σ2 Γ2R t 2 E[∥∇F (θ )∥2 ] = O √ + √ + √ T t=0 T M T T

(45)

P 1 2 The regional heterogeneity Γ2R = M r ∥∇Fr − ∇F ∥ (Assumption I.3) enters explicitly: it represents the irreducible cost of data heterogeneity. RegionFed’s regional coordination reduces Γ2R relative to flat FL by ensuring within-region homogeneity. This completes the proof of Theorem I.5. 42

L.2

Proof of Theorem I.6 (Personalization Bound)

Theorem I.6 (Personalization Bound): For region r with optimal regional model θr∗ and optimal global model θ∗ , the personalized model θt + αr θrt satisfies: lim E[Fr (θt + αr θrt )] ≤ Fr (θr∗ ) + L(1 − αr )2 ∥θ∗ − θr∗ ∥22 + 2Lαr2 Br2

t→∞

(46)

Complete Proof: Let θt be the global model at iteration t, and let θrt be the regional adaptation computed by Algorithm 2. The personalized model for region r is θ̃rt = θt + αr θrt . Note on iterate convergence: Theorem I.5 guarantees E[∥∇F (θt )∥2 ] → 0. For the personalization bound, we additionally assume that iterates converge to a stationary point, i.e., θt → θ∗ with ∇F (θ∗ ) = 0. This is standard in the personalized FL literature [T. Dinh et al., 2020, Li et al., 2021a] and holds under mild conditions (e.g., isolated stationary points or Polyak-Łojasiewicz inequality in a neighborhood of θ∗ ). Step 1: Taylor expansion

By the L-smoothness of Fr (Assumption I.1), for any θ, δ: L ∥δ∥22 2

(47)

L t ∥θ − θr∗ + αr θrt ∥22 2

(48)

Fr (θ + δ) ≤ Fr (θ) + ⟨∇Fr (θ), δ⟩ + Applying this with θ = θr∗ and δ = θ̃rt − θr∗ = (θt − θr∗ ) + αr θrt : Fr (θ̃rt ) ≤ Fr (θr∗ ) + ⟨∇Fr (θr∗ ), θt − θr∗ + αr θrt ⟩ + Since θr∗ is the optimal regional model, ∇Fr (θr∗ ) = 0: Fr (θ̃rt ) ≤ Fr (θr∗ ) +

L t ∥θ − θr∗ + αr θrt ∥22 2

(49)

Step 2: Decomposing the squared norm As t → ∞, Theorem I.5 guarantees θt → θ∗ . Focusing on the limiting behavior, we bound the residual: ∥(θ∗ − θr∗ ) + αr θrt ∥22 = ∥(1 − αr )(θ∗ − θr∗ ) + αr [(θ∗ − θr∗ ) + θrt ]∥22

(50)

Using ∥a + b∥22 ≤ 2∥a∥22 + 2∥b∥22 : ≤ 2(1 − αr )2 ∥θ∗ − θr∗ ∥22 + 2αr2 ∥(θ∗ − θr∗ ) + θrt ∥22

(51)

Step 3: Bounding regional adaptation By Assumption I.4, the regional adaptation satisfies ∥θrt ∥22 ≤ Br2 . The term ∥(θ∗ − θr∗ ) + θrt ∥2 represents the total displacement from the regional optimum. In the regime where regional adaptation is well-calibrated (i.e., θrt partially compensates for θ∗ − θr∗ ), we can bound this by Br since the adaptation magnitude dominates. More conservatively, applying ∥a + b∥2 ≤ 2∥a∥2 + 2∥b∥2 : 2αr2 ∥(θ∗ − θr∗ ) + θrt ∥22 ≤ 4αr2 ∥θ∗ − θr∗ ∥22 + 4αr2 Br2

(52)

∥(θ∗ − θr∗ ) + αr θrt ∥22 ≤ 2(1 − αr )2 ∥θ∗ − θr∗ ∥22 + 4αr2 ∥θ∗ − θr∗ ∥22 + 4αr2 Br2

(53)

Combining:

Since 2(1 − αr )2 + 4αr2 ≤ 2 for αr ∈ [0.5, 1] (verified: maximum at αr = 0.5 gives 2(0.25) + 4(0.25) = 1.5), and more generally bounded by 4 for all αr ∈ [0, 1], we use the structure of our bound directly. 43

Step 4: Combining and taking limits Substituting the bound from Step 2–3 into Step 1, and taking t → ∞ so that ∥θt − θ∗ ∥22 → 0 (by Theorem I.5), the residual term vanishes and we obtain: L 2(1 − αr )2 ∥θ∗ − θr∗ ∥22 + 4αr2 Br2 2

(54)

lim E[Fr (θt + αr θrt )] ≤ Fr (θr∗ ) + L(1 − αr )2 ∥θ∗ − θr∗ ∥22 + 2Lαr2 Br2

(55)

lim E[Fr (θt + αr θrt )] ≤ Fr (θr∗ ) +

t→∞

Simplifying: t→∞

This completes the proof of Theorem I.6. The bound has the form Fr (θr∗ ) + c1 (1 − αr )2 Dr2 + c2 αr2 Br2 where Dr = ∥θ∗ − θr∗ ∥2 , c1 = L, c2 = 2L. The optimal personalization weight αropt minimizes the right-hand side: ∂ L(1 − αr )2 Dr2 + 2Lαr2 Br2 = −2L(1 − αr )Dr2 + 4Lαr Br2 = 0 ∂αr

(56)

∥θ ∗ −θ ∗ ∥2

r r 2 Solving: αropt = D2 +2B 2 = ∥θ ∗ −θ ∗ ∥2 +2B 2 . r

Connection to Equation 6: The theoretical αropt provides a principled foundation for the heuristic in Equation 6. As the distance between optimal global and regional models increases (∥θ∗ − θr∗ ∥ ↑), αropt → 1. This mirrors our adaptive weight computation: when gradient conflict ∥∇L(θ) − ∇Lr (θ)∥2 is large, αr increases toward 1. The sigmoid mapping approximates this relationship while providing smooth, bounded transitions. L.3

Proof of Theorem I.7 (Privacy Guarantee)

Theorem I.7 (Privacy Guarantee, General Form): With user-level noise σu and region-level noise σr , RegionFed provides (ϵ, δ)-differential privacy where: ! p q · p · T log(1/δ) (57) ϵ=O min(σu , σr ) with q being the user sampling ratio and p being the region sampling ratio. Note:pWhen σu = σr = σdp and p = 1 (all regions participate), this reduces to the simplified form ϵ = O(q T log(1/δ)/σdp ) stated in Theorem I.7. The general form below accounts for two-level composition, which yields the factor-of-two increase in the practical remark. Proof Structure. We present the general two-level composition analysis for completeness (Steps 1– 4), which applies to deployments adding noise at both user and regional levels. Algorithm 2 in practice applies DP noise at the regional level only, with user-to-region privacy ensured by secure aggregation; for this setting, the privacy guarantee follows from Step 3 alone (single-level analysis), yielding ϵ ≈ 0.60 as computed in the practical remark at the end. Complete Proof: RegionFed applies differential privacy at two levels: (1) user-level privacy within each region, and (2) region-level privacy for global aggregation. We analyze the privacy guarantee using the moments accountant method [Abadi et al., 2016] combined with privacy amplification via sampling [Balle et al., 2018]. Step 1: User-level privacy within regions Consider a single region r with nr users. In each round t, the region selects a random subset Urt of users with sampling probability q = |Urt |/nr . For each selected user u ∈ Urt , the gradient is computed on their local data and clipped to bound sensitivity: ∇Lu (θt ) g̃ut = (58) max(1, ∥∇Lu (θt )∥2 /C) where C is the clipping threshold. This ensures ∥g̃ut ∥2 ≤ C. 44

The regional gradient is computed as: ḡrt =

1 X t g̃u |Urt | t

(59)

u∈Ur

To provide user-level differential privacy, Gaussian noise is added: grt = ḡrt + N (0, σu2 C 2 I) By the Gaussian mechanism, this single iteration provides (ϵ0 , δ)-DP where: 1 p C p 2 log(1.25/δ) = 2 log(1.25/δ) ϵ0 ≤ σu C σu

(60)

(61)

However, the moments accountant provides a tighter analysis. For a single iteration with sampling ratio q and noise multiplier σu , the privacy loss random variable L satisfies: α(λ) = log E[exp(λL)] ≤

λ(λ + 1)q 2 2σu2

(62)

for any λ > 0. Step 2: Composition over multiple rounds Over T rounds of training, by the composition property of the moments accountant: T λ(λ + 1)q 2 2σu2

(63)

log(1/δ) ϵu (T, δ) = min αT (λ) + λ λ−1

(64)

αT (λ) = T · α(λ) ≤ By the tail bound, for any δ > 0:

√ For large T , choosing λ = Θ( T ) gives: ϵu = O

T log(1/δ) σu

! (65)

Step 3: Region-level privacy for global aggregation Similarly, at the global level, the server samples regions with probability p = |St |/M where M is the total number of regions. Each regional gradient grt is clipped to bound sensitivity: grt max(1, ∥grt ∥2 /Cr )

(66)

1 X t g̃r + N (0, σr2 Cr2 I) |St |

(67)

g̃rt = The global gradient is: ḡ t =

r∈St

By the same moments accountant analysis: ϵr = O

! p p T log(1/δ) σr

(68)

Step 4: End-to-end privacy via amplification To analyze the end-to-end privacy, we consider the probability that a specific user’s data affects the final model. A user’s data is used only if: 1. Their region is selected (probability p) 2. They are selected within the region (probability q) 45

By the amplification theorem [Balle et al., 2018], the effective sampling ratio is q · p, not just q or p independently. The end-to-end privacy loss combines both levels. Let ϵtotal be the total privacy budget. We allocate: • User-level privacy: ϵu ≤ ϵtotal /2 • Region-level privacy: ϵr ≤ ϵtotal /2 By the basic composition theorem: ϵ ≤ ϵu + ϵr = O

! p T log(1/δ) p T log(1/δ) + σu σr

For balanced privacy allocation (σu ≈ σr adjusted for the respective sensitivities): ! ! p p q · p · T log(1/δ) (q + p) T log(1/δ) =O ϵ=O min(σu , σr ) min(σu , σr )

(69)

(70)

when amplification by subsampling is applied (since q, p ≪ 1, the effective privacy cost scales with their product). Step 5: Post-processing for adaptive weights A critical observation is that the adaptive personalization weights αr (Equation 6) are computed from the noised gradients g̃rt , not from raw user data. By the post-processing property of differential privacy [Dwork and Roth, 2014], any computation performed on the output of a differentially private mechanism (without accessing additional private data) does not consume additional privacy budget. Formally: If M is (ϵ, δ)-DP and f is any randomized function, then f ◦ M is also (ϵ, δ)-DP. Since αr = σ(∥g̃rt − ḡ t ∥2 /τr ) is computed solely from the differentially private gradients g̃rt and ḡ t , the adaptive weight computation does not increase the privacy loss. This is essential for RegionFed’s practical deployment: the personalization mechanism operates “for free” in terms of privacy budget. Remark on practical privacy parameters For our main experimental setting with T = 50 rounds, q = 0.1, σdp = 4, and δ = 10−5 , the closed-form upper bound gives: p p √ q T log(1/δ) 0.1 × 50 × log(105 ) 0.1 × 50 × 11.51 0.1 × 24.0 ϵ≤ ≈ ≈ ≈ 0.60 (71) = σdp 4 4 4 We verified this numerically using the Rényi Divergence Privacy (RDP) accountant [Mironov, 2017]: converting from (α, ϵ̂)-RDP to (ϵ, δ)-DP at optimal order α∗ ≈ 17 yields ϵ = 0.58, consistent with the analytic bound. Note that Algorithm 2 applies DP noise at the regional level only (g̃r = 2 ḡr + N (0, σdp C 2 I)), while user-to-region privacy is ensured by secure aggregation. The privacy guarantee therefore follows directly from the single-level analysis above, without requiring two-level composition. This demonstrates that RegionFed achieves strong privacy guarantees (ϵ ≈ 0.60 < 1) under the main evaluation setting while maintaining high utility. This completes the proof of Theorem I.7.

Limitations, Broader Impact, and Ethics

Limitations. (1) RegionFed requires pre-defined regions; automatic discovery via gradient clustering is future work, though misspecification experiments show only 4.2pp degradation (Appendix G.4). (2) We validate on NLP (T5, RoBERTa) and vision (FEMNIST CNN) tasks; multimodal tasks and emerging architectures (Mamba, state-space models) remain future work. (3) Amazon ESCI and Amazon Reviews derive from Amazon data; LEAF-FEMNIST provides independent validation, but additional domain-specific benchmarks (e.g., medical FL) would further strengthen claims. (4) The Dynamic strategy exhibits occasional failures (Grocery 78.2%); we recommend Grad as the production default. (5) While we provide privacy-utility analysis at five σdp settings (Appendix J), clipping threshold ablation (Appendix H.4), and RDP accountant verification (ϵ=0.58), evaluation 46

under DP-FTRL and privacy auditing remains future work. (6) Results averaged over 5 seeds with std<0.5pp; larger-scale statistical analysis would further strengthen claims. Broader Impact and Ethics. RegionFed enhances privacy by keeping user data on local devices and promotes equitable AI through regional personalization ensuring comparable service quality across regions. Potential risks include regional bias amplification and discriminatory pricing if deployed without fairness auditing. We recommend deployment with continuous fairness monitoring.

Record · ID 660793 · SHA-256 9e41138f77c04533
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.