Single or Multiple Policies for Phase-Structured Reinforcement Learning?

Single or Multiple Policies for Phase-Structured Reinforcement Learning? Guilhem Loussouarn, Nancy Nayak, and Kin K. Leung Department of Electrical and Electronic Engineering

arXiv:2610.03475v1 [cs.LG] 2 Oct 2026

Imperial College London, UK

Abstract Many reinforcement-learning (RL) problems are non-stationary yet structured and can be decomposed into phases, each with its own transition probabilities and reward functions. When the phase sequence is known, the common solution augments the state with information to satisfy the Markovian property and applies standard RL techniques. However, prior work finds that the multi-policy approach for different phases can outperform a single state-augmented policy shared among the phases, for reasons that remain unclear. In this work, we first show that the shared policy can theoretically achieve performance of any multi-policy solution. However, whether a multi-policy solution can perform better than the corresponding single shared policy in practice depends on function approximation, learning and optimization processes, as well as, for multi-policy solutions, the sample efficiency and loss of continuity from one policy to another. We propose a regime-based phase decomposition method to identify which policy can provide better performance. The method is based on consideration of the duration of transient system dynamics relative to the duration of the quasi-stationary period. Numerical experiments are conducted with different non-stationary RL problems to validate our four major hypotheses: (a) longer phase durations favor multi-policies, (b) the heterogeneity between phases increases the burden on single policy, (c) multi-policies need sufficient data for each phase, and (d) environment-specific transition dynamics between phases can affect which policy is preferable.

1

Introduction

Many sequential decision-making problems are non-stationary but exhibit recurring structure. Their transition dynamics or rewards may change between a finite set of phases. Within a phase, the interactions are described by a stationary Markov decision process (MDP) with phase-specific transitions and rewards; across phases, these MDPs are coupled, since the actions taken in one phase determine the state from which the next begins. We call the resulting sequence of coupled MDPs a phase-structured MDP. Examples include daily traffic and demand cycles [1, 2, 3, 4], robotic contact modes [5], and changing computational workloads [6]. We focus on deterministic scheduled phases, allowing both periodic and arbitrary fixed non-periodic structures, in which the active phase is observed. A latent or history-dependent phase process must instead be inferred from interaction, making the problem partially observable, and we leave that case to future work. State augmentation restores a time-homogeneous MDP only when the appended statistic is sufficient for both the physical dynamics and the evolution of the schedule [7]. For a periodic schedule, the phase index and local clock are sufficient. More general schedules may require additional scheduler information, such as global time or the position within a known sequence. This resolves how the process is represented, but not how policy parameters should be shared across phases. We distinguish three primary architectures along a parameter-sharing spectrum (Figure 1c): a shared policy uses one phase-conditioned network for all phases; a multi-policy architecture uses an independent specialist for each phase; and a multi-head policy shares a representation but uses phase-specific output heads. Recent work on periodic network resource allocation found that multipolicy PPO can outperform a shared state-augmented policy [8]. If the active phase is already observed by the shared policy, why should separate policies help? At the level of unrestricted policy classes, they should not: a shared phase-conditioned policy can reproduce any collection of phase-specific policies, so the advantage of specialization cannot follow from

1

(a) Phase structure and within-phase regimes pm−1 , rm−1

pm , r m

pm+1 , rm+1

phase m of duration Tm

phase m − 1

phase m + 1

Transient Quasi-stationary (QS) Lm steps Tm − Lm steps

Handoff state from m − 1 t

Handoff state to m + 1

t + Lm

t + Tm

(b) Estimate competing effects

(c) Select a sharing level more sharing

Handoff Cost Km

Settled QS Gain Gm

Single Shared

High handoff cost; Knowledge sharing essential

Multihead

Intermediate regime; Shared trunk + local heads

Multipolicy

High settled gain; Sufficient phase-local data

combine

Heterogeneity Hm

Phase-Local Data Bm ≃ B/M more specialization

Figure 1: Conceptual framework for architecture selection in phase-structured MDPs. (a) Each phase m of duration Tm exhibits local dynamics (pm , rm ) and can be split into a transient regime of Lm steps (dominated by handoff states inherited from the preceding phase) and a quasi-stationary (QS) regime of Tm − Lm steps. (b) The net trade-off between specialization and sharing depends on four diagnostic mechanisms: handoff cost Km who favours coordinated policies; settled gain Gm representing the advantage of specialization learning; task heterogeneity Hm , which represents the difference between faces and can affect shared learning; and phase-local data budget Bm , which predominantly affects multipolicy learning. (c) Combining these diagnostics places the problem on a continuum from full parameter sharing (Single Shared ) to full specialization (Multipolicy), with modular architectures (Multihead ) occupying the intermediate regime. expressivity alone, nor does it imply that specialization is always preferable. How much information and how many parameters are shared nonetheless affects learning performance, sample efficiency, and computational cost. We therefore ask: when should a policy share parameters across phases, and when should it specialize? Central claim. Under finite learning resources, specialization is favoured when its phase-local learning benefit exceeds the costs of sample fragmentation and boundary handoff. Sharing is favoured when transfer, data pooling, and cross-phase coordination dominate. We relate this trade-off to three properties of a phase-structured MDP. First, phase heterogeneity determines how different the behaviours required in each phase are. Second, the training budget determines whether every specialist receives enough data to learn its phase. Third, the within-phase regime structure determines the importance of coordination. Immediately after a phase change, performance may depend strongly on the state inherited from the previous phase. We call this the transient regime. Once this dependence has weakened, the phase enters a quasi-stationary regime in which phase-local specialization may be more beneficial. The duration of the transient and its sensitivity to the inherited state determine the handoff cost of independent policies. We study these mechanisms under observable phase structure, leaving latent phase inference outside our scope. Across our three environments, changing the phase count, duration, heterogeneity, budget, or boundary dependence can reverse the ranking of shared and specialized architectures. Our contributions are fivefold. First, we formalize phase-structured MDPs and identify the scheduler state that restores a timehomogeneous MDP. Second, we show that the shared phase-conditioned policy class is equivalent to the

2

multipolicy class, so any performance difference between them is a learning effect rather than a representational one. Third, we split each phase occurrence into transient and quasi-stationary regimes and decompose the learned return gap into a settled specialization gain and a transient handoff cost, yielding a phase-wise rule for when specialization pays and tying the gain to phase heterogeneity and the per-phase sample budget. Fourth, we derive diagnostics linking architecture choice to phase heterogeneity, phase-local sample budgets, transient duration, and handoff sensitivity. Finally, we test these predictions with single shared, multihead, and multipolicy PPO on HAPS coverage control, PhasePendulum, and PhaseGridWorld, and probe the proposed mechanisms through capacity controls, gradient measurements, warm starts, distillation, and controlled handoff interventions. Paper roadmap. The remainder of this paper follows the conceptual flow outlined in Figure 1. Section 2 situates our approach within the broader literature on non-stationary RL, multi-task learning, and capacity. Section 3 formalizes phase-structured MDPs and the policy class equivalence. Section 4 proposes the regime decomposition into transient and quasi-stationary phases (Figure 1a) and derives the diagnostic metrics quantifying handoff costs, settled gains, and sample budgets (Figure 1b). Finally, Section 5 validates these predictions experimentally across three environments, demonstrating how competing mechanisms propose an appropriate level of sharing of architectural components across different architectures(Figure 1c).

2

Related Work

Structured time-varying MDPs. Prior work studies MDPs whose transition and reward functions change according to an underlying structure. Periodic MDPs assume that these changes repeat on a fixed cycle and commonly include the period index in the state to use state-augmented policies [9, 10, 11, 12, 13, 14]. Other models consider stochastic or hidden changes, such as switching non-stationary MDPs governed by a latent Markov chain [15]. We use phase-structured MDP to refer broadly to the cases when the change structure is deterministic and observed. Sec. 3 provides the formal definition. As for phase-structured RL, it represents the full range of RL methods applied to phase-structured MDPs. Periodic multi-policy reinforcement learning. [8] propose multi-policy RL for periodic network resource allocation and report gains over single-policy baselines. A related multi-policy architecture has been applied to high-altitude platform system (HAPS) control in a phase-structured telecommunications environment [16] with similar improvements. In both cases, no explanations on the gap between state-augmented policy and multi-policy performances are given. We take these empirical observations as motivation and study the architectural choice between parameter sharing and phase specialization across controlled domains. Our HAPS benchmark retains the coverage-control component of the latter environment, enabling systematic ablations of the factors governing this choice. Multi-task learning, modularity, and transfer. Multi-task learning aims to solve related tasks jointly by sharing parameters or representations [17]. Phase-structured RL admits a similar interpretation: the phase acts as a task descriptor, a state-augmented policy shares all parameters across phases, and a multipolicy uses independent phase specialists. Sharing can improve data efficiency through transfer, but it can also cause negative transfer when tasks require incompatible updates. PCGrad and CAGrad modify task gradients to reduce such conflicts [18, 19]; we evaluate them as baselines and use phase-specific gradient geometry as a local diagnostic. Modular and transfer methods address limitations of complete sharing or separation. Mixture-of-experts models divide complex mappings among specialized modules [20, 21]. Divide-and-conquer RL uses local policies to simplify difficult optimization, while distillation, Distral, progressive networks, and generalist– specialist learning transfer or preserve knowledge across policies and tasks [22, 23, 24, 25, 26]. These ideas motivate our multihead and warm-start baselines, which we define in Sec. 5. The analogy remains incomplete because standard multi-task formulations generally sample tasks separately [27]. Phases are instead consecutive parts of one trajectory: the policy used in one phase determines the state passed to the next. Existing multi-task methods therefore address sharing and transfer, but not necessarily cross-phase coordination. Capacity and irrelevant features. Network size and training compute can affect deep-RL performance, although their benefits depend on the algorithm, environment, and data regime [28, 29, 30, 31]. Moreover, observations that are irrelevant to the optimal decision can still increase the difficulty of finite-

3

sample learning with function approximation [32]. These observations motivate our capacity and distractor controls, which test whether multi-policy gains can be explained by parameter count or by the difficulty of extracting phase information.

3

Phase-Structured MDPs

We begin by recalling the standard MDP formulation. An MDP is a tuple P = (S, A, p, r, µ0 , γ), where S and A are the state and action spaces, p(· | s, a) is the state-transition probability distribution (or transition kernel), r(s, a) is the expected one-step reward, µ0 is the initial-state distribution, and γ ∈ (0, 1] is the discount factor. Phases and schedules. Let M = {0, . . . , M − 1} be a finite set of phases. Each phase m ∈ M identifies a stationary mode of the environment, described by a transition probability distribution pm and reward function rm . A phase is therefore a model of the local dynamics and objective. A deterministic phase schedule is a map σ : N → M, and σ(t) denotes the phase active at time t. For example, consider the sequence of phases m in Table 1 consisting of four phases m ∈ M = {0, 1, 2, 3} that may occur multiple times in an episode. The total duration of the occurrence of each phase is arbitrary, i.e., the duration of the first occurrence (j = 0) of phase m = 0 is two time steps, i.e. Tm,j |m=0,j=0 = 2, whereas the second occurrence (j = 1) of phase m = 0 is only one time step, i.e., Tm,j |m=0,j=1 = 1. τ (t) denotes the local time step within a phase. t m = σ(t) τ (t)

0 0 0

1 0 1

2 1 0

3 1 1

4 1 2

5 2 0

6 2 1

7 3 0

8 3 1

9 3 2

10 3 3

11 0 0

12 2 0

13 2 1

14 0 0

15 0 1

16 0 2

... ... ...

Table 1: A phase schedule with phase m ∈ M = {0, 1, 2, 3}. Definition 1 (Phase-structured MDP). A deterministic phase-structured MDP is a tuple Pσ = (S, A, {pm }m∈M , {rm }m∈M , µ0 , γ, σ) . At time t, the active phase is σ(t) and starting from an initial state s0 ∼ µ0 , the process evolves at time t according to st+1 ∼ pσ(t) (· | st , at ),

rt = rσ(t) (st , at ).

For a finite horizon N , the objective is to find an optimal policy π ∗ such that: "N −1 # X ⋆ t π ∈ arg max JN (π), where JN (π) = Eµ0 ,π,σ γ rσ(t) (st , at ) . π

t=0

The expectation is taken over the initial state, policy actions, and state transitions under the fixed schedule σ. For an infinite horizon, we take N = ∞ and assume γ < 1 and bounded rewards. Scheduler state and Markovianity. The state st alone may not be Markov. For the same state, the transition or reward can differ depending on the active phase σ(t). Let zt = (σ(t), it ) denote the scheduler state, where σ(t) is the phase index and it is auxiliary schedule context. We call zt sufficient if transition dynamics depend on history solely through (st , at , zt ). In practice, observing only σ(t) is insufficient because phase dynamics vary internally over time. While appending global time (it = t) technically restores Markovianity, it transforms the state space into an acyclic directed graph that prevents data re-use across cycles. To preserve both Markovian sufficiency and sample efficiency, it must be mapped to a compact quotient space; for periodic phases of length Tm , the within-phase clock τ (t) = t mod Tm suffices (zt = (σ(t), τ (t)); see App. A.1.1). Proposition 1 (Scheduler-state augmentation). If zt is sufficient, then the augmented state set = (st , zt ) defines a time-homogeneous MDP and recovers Markovianity. The formal Markovian sufficiency conditions and a periodic phase schedule (Corr. 1) are detailed in App. A.1.1. The proofs of Cor. 1 and Prop. 1 are detailed in App. A.2). 4

Phase-local models and handoffs. For a phase m and an entry-state distribution µ, define the stationary phase-local model Pm (µ) = (S, A, pπm , rm , µ, γ), (1) which describes the local dynamics and rewards. In a phase-structured MDP, however, µ is not fixed, i.e., the distribution µπm,j of the state at the start of the j th occurrence of phase m is induced by the actions π in the preceding phases.PThe state transition probabilities pπm therefore are also dependent on previous policies π such that pπm = a∈A pm (st+1 |at , st )π(at |st ). The phases therefore cannot in general be treated as independent. This cross-phase dependence is the handoff effect studied in Sec. 4. Shared policy and specialized policies. Since the augmented state s̃t yields a time-homogeneous MDP, standard RL methods can learn a single policy π across all phases. We call any such policy a shared policy. Instead, [8] train a separate specialized policy for each phase, an architecture shown to outperform shared policies in phase-structured problems. Yet, specialized policies hold no representational advantage, since any set of them can be expressed by a single shared policy. Proposition 2 (Shared–multipolicy equivalence). There is a bijection between shared policies π sh (· | s, i, m) and an ensemble of specialized policies π sp = (π0 (· | s, i), . . . , πM −1 (· | s, i)) expressed through the gated map Γ such that π sp (a | s, i, m) = Γ(π sp )(a | s, i, m) = πm (a | s, i), (2) π sh (a | s, i, m) = πm (a | s, i) = π sp (a | s, i, m). It is always possible to represent a shared policy with an ensemble of specialized policies and vice versa. This is a policy-class equivalence: with unrestricted function classes, shared and specialized policies attain the same optimal return. However, it says nothing about where each architecture reaches when trained. With finite networks, finite data, and gradient-based optimization, the two can converge to different solutions, which is why either may outperform the other in practice. The next section characterizes when specialization is preferable to sharing, and when the reverse holds. Illustrations and proof of Prop. 2 can be found inside App. A.2

4

When to Share and When to Specialize

Each phase-local model Pm (µ) introduced in (1) is stationary within its phase horizon but inherits its initial entry distribution µ from the preceding phases. We now ask how long that inheritance matters within a phase, and how the answer, together with the heterogeneity of consecutive phases, determines whether a shared or a specialized policy is better suited.

4.1

Transient and Quasi-Stationary Regimes

At a phase boundary, an inherited condition such as position, velocity, network load, or other initial-state condition can strongly affect subsequent performance. We call the period during which performance remains sensitive to this inherited state the transient regime, and the remaining steps, once the sensitivity has become small, the quasi-stationary regime, in which behaviour is governed mainly by the current phase’s dynamics and objective. Quasi-stationarity does not require convergence to an invariant state distribution. It requires only that the effect of the entry state on the return quantities under study has become small. Let Lm,j ≤ Tm,j denote the duration of the transient regime of the j th occurrence of phase m, leaving a quasi-stationary regime of duration Tm,j −Lm,j time steps. Occurrences of the same phase may differ in Lm,j because they inherit different entry distributions, and in slowly mixing or disconnected systems the transient may span the whole phase (Lm,j = Tm,j ), so periodicity alone does not guarantee recovery before the next boundary. We estimate Lm,j from phase-age behaviour and properties of the environment. App. A.1.2 gives the formal recovery definition and the occurrence-level aggregation. The two regimes expose different architectural trade-offs. In the quasi-stationary regime, independent training may make a phase-local mapping easier to approximate and optimize than a joint mapping across all phases. In the transient regime, policies remain coupled through the state passed across the boundary. A shared policy trained on complete trajectories can account for this continuation value, whereas independently

5

trained phase-local specialized policies may not. Sharing also pools data and reuses representations, whereas a balanced specialized policy receives only about B/M of a total budget B. We now quantify these effects.

4.2

Settled Gains and Transient Handoff Costs

The regime decomposition explains why the preferred architecture may vary across phase-structured problems; we now make that comparison quantitative. We measure the gain phase-specific learning obtains in the quasi-stationary regime against the cost specialization incurs in the transient regime, which together form the architecture diagnostic developed in the next subsection. A comparison between learned shared and learned specialized policies. Let the learned shared and the learned specialized policies are denoted by π bsh and π bsp , respectively. For occurrence j of phase m, PLm,j −1 t PTm,j −1 t with a transient regime duration of Lm,j ≤ Tm,j , define λm,j = t=0 γ / t=0 γ . Thus λm,j is the qs tr fraction of the occurrence’s discounted mass assigned to the transient regime. Let J¯m,j,A and J¯m,j,A denote the discounted average rewards of architecture A ∈ {sh, sp} for single shared and specialized policies, in the transient and quasi-stationary windows. For each occurrence, let qs qs − J¯m,j,sh , Gm,j = J¯m,j,sp

tr tr Km,j = J¯m,j,sh − J¯m,j,sp .

(3)

A positive Gm,j is a quasi-stationary specialization gain, while a positive Km,j is a transient advantage for sharing. For readability, we omit the index j, and the remainder of the paper uses phase-level quantities λm , Gm , and Km , obtained by aggregating over occurrences of phase m. This does not assume that all occurrences have identical entry states or transient lengths. The advantage of using learned specialized policies over a learned single shared policy is given by ¯ π sp ) − J(b ¯ π sh ) = ∆J¯ := J(b

M −1 X m=0

ωm [(1 − λm )Gm − λm Km ] ,

(4)

where ωm is the fraction of discounted evaluation mass belonging to phase m. The derivation of the above formulation is given in detail in App. A.1.2. Specialized policies for phase-local models win over a single shared policy in phase m exactly when its quasi-stationary advantage exceeds its transient deficit, i.e., (1 − λm )Gm > λm Km .

(5)

Transient hand off cost Km . The entry law µm,A is produced by the policy acting in the phase before m, and no architecture removes this coupling. Standard multi-policy training optimizes each specialized policy on phase-local experience, so a specialist may miss how its late actions shift the entry state of the next phase, whereas a shared policy trained on complete trajectories receives that cross-phase credit. Accordingly, Km is large when achievable return is sensitive to the entry state and recovery is slow; and small when entry distributions match, dynamics contract quickly, or phases reset independently. Quasi-stationary gain Gm . In the quasi-stationary regime, phase m resembles its phase-local MDP, a ⋆ specialized policy may approximate the local optimum πm better than a finite shared policy trained across all phases, giving Gm > 0. This is a finite learning effect, not an expressivity advantage. Since, better phase-local policies need not achieve a better global return once handoffs are included, we isolate the settled comparison by evaluating both architectures on the same settled reference distribution µref m . ⋆ Let J¯m and Π⋆m denote the optimal quasi-stationary return in phase m and its set of optimal policies. We measure the heterogeneity of phase m by the loss incurred when policies optimized for other phases are transferred to it: " # X X ⋆ qs ref Hm = amk J¯m − sup J¯m (π; µm ) , amk = 1, (6) k̸=m

π∈Π⋆ k

k̸=m

so Hm is large when behaviour optimal elsewhere performs poorly in phase m; the supremum avoids dependence on an arbitrary choice of policy when a phase has several optima. For the shared and specialized architectures, define the phase-local learning gaps  qs   qs  ⋆ ⋆ εm,sh (B; Hm ) = J¯m − E J¯m (b πsh,B ; µref εm,sp (Bm ) = J¯m − E J¯m (b πm,sp,Bm ; µref m ) , m ) . 6

Table 2: Architecture selection regimes derived from the transient–quasi-stationary framework. Regime

Governing Condition

Recommended Architecture

Insufficient Bm Enough Bm , low Hm Enough Bm , high Hm , low K m Enough Bm , high Hm and high Km Uncertain [R, R]

Specialist data starvation High representation transfer High local learning burden Severe boundary handoff cost

Shared; multihead if outputs differ Shared or Multihead Specialized Multi-Policy Shared or Coordinated Specialists

Uncertified bounds

Multihead / Model Selection

which absorb finite capacity, data, exploration, optimization, and architectural bias. In particular, the shared εm,sp accounts for joint optimization challenges such as multi-task gradient dilution (App. A.1.5), whereas independent specialists avoid cross-phase gradient interference entirely. Writing Hm as an argument of εm,sh emphasizes that the shared learner must fit phase-m behaviour jointly with other phases, without assuming any particular functional dependence. The expected settled specialization gain is then exactly Gqs m,ref = εm,sh (B; Hm ) − εm,sp (Bm ).

(7)

Heterogeneity Hm widens εm,sh , while data pooling transferable representations across phases narrows it. Specialization eliminates this joint capacity bottleneck, but each specialist receives only a fraction ζm ∈ (0, 1) of the total sample budget B, i.e., Bm = ζm B, or B/M under uniform allocation ζm = 1/M . A specialist ensemble therefore achieves Gqs m,ref > 0 only when the local sample allocation Bm = ζm B avoids data starvation by exceeding the phase-local sample threshold B0,m (δ) required for a target suboptimality tolerance δ > 0 (App. A.1.4, Prop. 4).

4.3

From Regime Quantities to Architecture Choice

The advantage presented in (4) becomes predictive when the regime quantities are bounded, either by environment analysis or by short pilot runs. Substituting bounds Gm ≤ Gm ≤ Gm and K m ≤ Km ≤ K m , yields certified bounds on relative performance: X X     R := ωm (1 − λm )Gm − λm K m ≤ ∆J¯ ≤ ωm (1 − λm )Gm − λm K m =: R. (8) m

m

¯ π sp ) > J(b ¯ π sh ), and R < 0 certifies sharing. An interval containing Hence R > 0 certifies specialization, i.e., J(b zero leaves the choice unresolved and favours an intermediate architecture (i.e., a multihead network, although (8) compares only the shared and specialized endpoints and does not certify that option. For a specific phase m with Gm > 0 and K m ≥ 0, specialization yields a positive local lower bound whenever the transient fraction satisfies: Gm λm < . (9) Gm + K m so the settled gain must persist long enough to amortize the transient handoff cost. Table 2 summarizes the resulting regimes. The specialized bound dominates when the settled gain outweighs the handoff and budget penalties, and the shared bound when transfer, scarce phase-local data, or boundary sensitivity is decisive; multihead suits the unresolved region because it preserves shared features while allowing phase-specific action mappings.

5

Experiments and Results

To test the regime model of Sec. 4, we evaluate PPO [33] on three phase-structured environments: highaltitude platform systems (HAPS) coverage optimization [16], PhasePendulum, and PhaseGridWorld, all described in detail in App. A.3.2, App. A.3.3, and App. A.3.4, respectively. In the HAPS network, the idea is to find the optimal position of the HAPS given that the user density and traffic vary over time

7

but in a periodic manner. In particular, different phases inside a period are differentiated by different user densities and traffic loads. The entire service region of the HAPS network is divided into multiple cells; only a few selected cells act as hotspots where the user density and traffic are higher than the rest of the cells. PhasePendulum environment consists of the standard torque-controlled pendulum dynamics while changing the target angle gm between phases. PhaseGridWorld environment is a 9 × 9 grid with four deterministic actions: up, down, left, and right. Each phase has its own goal and obstacle map. All environments are periodic, with the number of phases and cycles varying by experiment. We compare three architectures: a single policy with the phase and clock augmentation, a multipolicy of M specialists, and a multihead policy with a shared trunk and M heads. All methods use the same PPO hyperparameters unless stated otherwise, and we report raw reward per environment step throughout. We organize the results around four hypotheses (H1 to H4) obtained from Sec. 4 and presented in Table 2. Our three main methods span different degrees of parameter sharing. Single+clock is a single actor–critic taking physical state, phase index, and within-phase clock as inputs. Multihead receives the physical state and phase index and combines a shared actor body and a shared critic with M phase-specific linear action heads; at each step the head indexed by the active phase emits the action, so the shared body is updated from every transition while each head is updated only from transitions of its own phase. The phase code is also supplied to the body, allowing it to form phase-conditioned features rather than only phase-agnostic ones. Multipolicy uses M independent actor–critics, the active phase selecting which one acts and is updated; its input is the physical state, whereas the clock-aware variant is denoted multipolicy+clock. The phase code is not supplied here, since selecting the network already identifies the phase. More details on the methods can be found in App. A.3.1 Reward/step

Reward/step

50 40 30 20 100

200

500

40 30 20 2

2000

5

10

20 M

T Single+clock

Multihead

Multipolicy

Figure 2: Main HAPS regime evidence. Left: at fixed M = 20, phase-clock sharing leads at T = 100, multihead and multipolicy are close at T = 200, and multipolicy leads at T ∈ {500, 2000}. Right: increasing the number of phases preserves the specialist advantage at T = 500 and exposes the degradation of full sharing at M = 20. This is consistent with, but does not prove, amortization of boundary costs.

5.1

H1: Longer Phases Favour Specialization with multi-policy

In the HAPS environment, the transient regime is governed by platform relocation and by user persistence across phase boundaries. A direct transit takes at most 100 steps when both spatial coordinates of the HAPS change maximally, and mean user lifetimes of 5 to 25 steps are much shorter, so we take Lm ≈ 100. Reward differences arise mainly when one or two platforms fail to reach a high-density zone, so any difference between two policies should be in the same order. Following this observation, Gm ≈ Km reduces (9) to λm < 1/2, and with γ ≈ 1 and λm ≈ Lm /Tm this predicts a crossover at T ≈ 2Lm = 200. We treat 200 as a diagnostic horizon rather than a strict value of Tm , since arrivals are stochastic and learned trajectories need not be straight. The prediction is that shared policies win below this dwell time, where the transient dominates, and specialized policies above it, where the quasi-stationary regime does. Fig. 2 (left), at M = 20, shows exactly this: multipolicy reward increases with T , Single+Clock decreases, the two curves cross near T = 200, and multihead is largely insensitive to T , confirming its position as an in-between solution. PhasePendulum reproduces this pattern (Table 11). For both M = 5 and M = 10, the single policy leads at T ∈ {100, 200} while multi-policy attains the best mean at T = 500; multi-head performs very close to multi-policy and even surpasses in some cases. Longer dwell times therefore favour specialization wherever a quasi-stationary regime is physically reachable.

8

5.2

H2: Behavioural Diversity Increases the Burden on Single Shared Policy

Heterogeneity can be increased by adding distinct phases, which leaves each specialist’s problem unchanged while requiring the shared policy to represent more behaviours. Fig. 2(right) reports the performance of the HAPS for different phase counts at T = 500: Single+Clock degrades as M grows while multi-policy is flat, so the effect is a degradation of complete sharing rather than a benefit of more phases. The snapshots in Fig. 8 in the appendix show the mechanism, with the shared policy failing to reach the hotspots of some phases. Widening the shared network or using four, six or eight layers does not close the gap, and a network with eight layers performs worse than one with six layers (Table 6). The shared policy therefore does not fail for lack of parameters: adding them does not help and can hurt. Its limitation lies in training rather than in representational capacity, as Prop. 2 implies, so the remaining gap is an architectural advantage for multipolicy. PhasePendulum varies behavioral diversity more directly, setting target angles to gm (h) = max h gm (App. A.3.3). Table 10 reports h from 0 to 1. The single policy is best at h = 0, where all phases are identical, and at h = 0.25; from h = 0.5 the ordering reverses, with multihead first and multipolicy second. Heterogeneity therefore widens the shared learner’s gap as in (7), far enough to reverse the advantage of sharing. In PhaseGridWorld (Fig. 13), where handoff costs are negligible, shared and specialized architectures perform comparably when adjacent phases share an optimal policy, while multipolicy clearly outperforms the shared baseline when they require distinct ones. Heterogeneity alone therefore suffices to separate the architectures. We test whether the single-policy degradation observed above is caused by destructive shared updates in Table 9 and Fig. 9. The orthogonal-null alignment statistic Ag presented (App. A.1.5) remains close to one for any M . Thus the main reason behind learning degradation is not the destructive interference. It explains why gradient surgery solutions such as PCGrad/CAGrad did not close the gap with multipolicy in our environments, as gradient destruction is not the biggest burden (Table 5). The cosine√similarity falls from 0.675 to 0.195 for complete sharing as M grows from 2 to 20 (Table 9), close to the 1/ M orthogonal benchmark derived in App. A.1.5. With more distinct phases in the shared policy, the cosine similarity is going to decrease for larger M . We call this effect gradient dilution. Together with the capacity controls, it shifts the explanation toward finite approximation and shared optimization burden.

5.3

H3: Specialization Requires Sufficient Phase-Local Data

Under a fixed total interaction budget B, a balanced multipolicy allocates only Bm ≃ B/M samples to specialist m. Shared PPO instead updates one model from all B interactions. We call failure to reach a high-return local solution within Bm the learning starvation; this concerns finite-budget performance, not optimizer convergence.PhasePendulum at h = 0 isolates this effect because all ten phases are identical and any performance gap must reflect optimization and sample efficiency rather than representation capacity. With a total budget of B = 5M timesteps, each redundant specialist receives a partitioned allocation of only Bm ≈ 500k interactions. As shown in Fig. 11, Single+clock strongly outperforms Multipolicy because local specialists suffer from severe sample starvation (Bm < B0,m App. A.1.4), failing to reach local convergence. Fig. 10 confirms this threshold behavior: Multipolicy requires a total budget scaling to B ≥ 10M for each local allocation to cross B0,m and approach the asymptotic performance level of Single+clock. A singlephase GridWorld calibration provides consistent evidence (Tab. 12): the clock-aware shared model reaches the target learning threshold within 500k interactions across all three seeds, whereas the multipolicy model requires a total 750k steps to reliably match it. It is important to note the intrinsic advantage of adding clock information to any solution (Sec. A.4.3). Also, Tab. 15 illustrates that increasing the global budget can invert the performance ranking between single and multipolicy models.

5.4

H4: Handoff Dependence Can Reverse the Ranking

Studying handoff dependence is challenging in general because transient regimes are rarely cleanly delimited in complex tasks. We address this by using controlled GridWorld environments where the underlying causality is fully understood. Because independent specialists optimize phase-local objectives from local resets, they lack awareness of downstream continuation values, incurring a boundary coordination loss. Full-

9

trajectory shared policies can instead sacrifice immediate phase return to ensure the overall highest return (Prop. 3). Fig. 7 formalizes this trade-off in a two-phase GridWorld. Selecting an immediate goal (g00 , reward 1.0) leaves the agent trapped behind an impassable wall in Phase 1, whereas selecting a safe goal (g01 , reward ρ < 1.0) preserves Phase 1 reachability. As shown in Fig. 12 (ρ → 0.9), Single+clock and Multihead adapt to choose g01 in almost all episodes. In contrast, independent specialists (Multipolicy) remain myopic, selecting g00 in 90% of episodes and failing Phase 1 entirely. Here, the transient regime length equals the full phase duration (Lm = Tm ), completely annihilating specialization gains due to uncoordinated local policies. A second GridWorld experiment (App. A.3.4) evaluates the transient regime across two dimensions: (1) varying phase goals and wall layouts to adjust phase heterogeneity and coordination demands, and (2) comparing physical state carryover against resampling from reachable entry states (Fig. 6). As shown in Fig. 13, physical carryover severely penalizes Multipolicy in the irrecoverable Closed Trap (Lm = Tm ). Resetting phase entry to a common reachable state eliminates boundary costs (Km ≈ 0), reversing to a positive gap for Multipolicy. Conversely, in the recoverable Bottleneck (Lm < Tm ), Multipolicy dominates under both physical carryover and reachable resets.

6

Conclusion

We have investigated when phase-structured reinforcement learning should employ a single shared policy versus phase-specific multiple-policies. Given sufficient scheduler information, single and multiple policies have the same representational capability, so any difference in performance must come from the environment and the learning dynamics rather than from representation. The proposed regime decomposition of the phases helps to understand the difference between single and multiple policies under constraints such as limited samples, phase-heterogeneity, and the mutual coupling between phases. We observe that sharing is favoured when phases exhibit behavioral overlap, learning budgets are constrained, or boundary-state coordination consumes a large fraction of the phase duration. Specialization is favored when phases are heterogeneous, per-specialist data volume is sufficient, and the quasi-stationary gain exceeds the handoff cost. Partially shared designs, e.g., multihead networks offer a intermediate solution between these two extremes. Future work should look beyond the binary choice between single shared and specialized multi policy models. The key challenge lies in designing adaptive architectures that share transferable features, specialize where local optimization is hard. and propagate continuation values across phase boundaries. Extending this framework to unknown phase structures and develop a coordinated multiple policies provide a promising foundation for a new class of phase-structured RL algorithms.

References [1] Eduardo Mucelli Rezende Oliveira, Aline Carneiro Viana, K.P. Naveen, and Carlos Sarraute. Mobile data traffic modeling: Revealing temporal facets. Computer Networks, 112:176–193, 2017. [2] Haoyang Shi and Yong Li. Discovering periodic patterns for large scale mobile traffic data: Method and applications. IEEE Transactions on Mobile Computing, 17(10):2266–2278, 2018. [3] Michael Haffey, Martin Arlitt, and Carey Williamson. Modeling, analysis, and characterization of periodic traffic on a campus edge network. In 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 170–182, 2018. [4] György Dán and Niklas Carlsson. Dynamic content allocation for cloud-assisted service of periodic workloads. In IEEE INFOCOM 2014 - IEEE Conference on Computer Communications, pages 853– 861, 2014. [5] Oliver Kroemer, Scott Niekum, and George Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. Journal of Machine Learning Research, 22(30):1–82, 2021.

10

[6] George Amvrosiadis, J. W. Park, Gregory R. Ganger, Garth A. Gibson, Elisabeth Baseman, and Nathan DeBardeleben. On the diversity of cluster workloads and its impact on research results. In USENIX Annual Technical Conference, pages 533–546, 2018. [7] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994. [8] Zheyu Chen, Kin K. Leung, Shiqiang Wang, Leandros Tassiulas, Kevin Chan, and Patrick J. Baker. Multi-policy reinforcement learning for network resource allocation with periodic behaviors. Computer Networks, 272:111645, 2025. [9] J. O. Riis. Discounted markov programming in a periodic process. Operations Research, 13(6):920–929, 1965. [10] L. M. M. Veugen, J. van der Wal, and J. Wessels. The numerical exploitation of periodicity in markov decision processes. OR Spectrum, 5(2):97–103, 1983. [11] N. S. Podtsykin. Optimization in periodic markovian decision processes. Cybernetics, 27(2):277–283, 1991. [12] Takayuki Osogami and Rudy Raymond. Efficient policy iteration for periodic markov decision processes. In European Conference on Artificial Intelligence (ECAI), pages 1167–1172, 2014. [13] Yichun Hu and Boris Defourny. Near-optimality bounds for greedy periodic policies with application to grid-level storage. In 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 1–8, 2014. [14] Ayush Aniket and Arpan Chattopadhyay. Online reinforcement learning in periodic mdp. 2023. [15] Mohsen Amiri and Sindri Magnússon. Reinforcement learning in switching non-stationary markov decision processes: Algorithms and convergence analysis, 2026. [16] Guilhem Loussouarn, Nancy Nayak, Kin K. Leung, and Patrick J. Baker. Rl-based joint coverage and beam optimization of high altitude platform systems. arXiv preprint arXiv:2606.20578, 2026. [17] Rich Caruana. Multitask learning. Machine Learning, 28:41–75, 1997. [18] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, 2020. [19] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, volume 34, pages 18878– 18890, 2021. [20] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87, 1991. [21] Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 6(2):181–214, 1994. [22] Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and-conquer reinforcement learning. 2018. [23] Andrei A. Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295, 2015. [24] Yee Whye Teh, Victor Bapst, Wojciech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. 2017.

11

[25] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016. [26] Zhiwei Jia, Xuanlin Li, Zhan Ling, Shuang Liu, Yiran Wu, and Hao Su. Improving policy optimization with generalist-specialist learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 10104–10119. PMLR, 2022. [27] Sihan Zeng, Malik Aqeel Anwar, Thinh T. Doan, Arijit Raychowdhury, and Justin Romberg. A decentralized policy gradient approach to multi-task reinforcement learning. In Proceedings of the ThirtySeventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 1002–1012. PMLR, 2021. [28] Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. 2023. [29] Max Schwarzer, Johan Obando-Ceron, Aaron Courville, Marc Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. 2023. [30] Oleh Rybkin, Michal Nauman, Preston Fu, Charlie Snell, Pieter Abbeel, Sergey Levine, and Aviral Kumar. Value-based deep rl scales predictably. 2025. [31] Preston Fu, Oleh Rybkin, Zhiyuan Zhou, Michal Nauman, Pieter Abbeel, Sergey Levine, and Aviral Kumar. Compute-optimal scaling for value-based deep rl. 2025. [32] Nicholas K. Jong and Peter Stone. Towards learning to ignore irrelevant state variables. In AAAI Workshop on Learning and Planning in Markov Processes, 2004. [33] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. 2017. [34] Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 767–777. PMLR, 2020. [35] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. [36] Simon Schmitt, Jonathan J. Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojciech M. Czarnecki, Joel Z. Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, and S. M. Ali Eslami. Kickstarting deep reinforcement learning. 2018.

12

A

Appendix

The appendix is organized in four parts. Sec. A.1 defines the optimization, heterogeneity, capacity, budget, and handoff quantities used by the regime framework, and then assembles the predictive certificate. Sec. A.2 groups the complete derivations by their role in the argument. Sec. A.3 specifies the environments, training protocols, and reporting conventions. Sec. A.4 contains additional quantitative results and qualitative diagnostics.

A.1

Supporting Mechanisms and Bounds

This section provides the definitions and results used by the different sections of this work. A.1.1

Schedule Sufficiency and Markovian Augmentation

To model phase transitions formally within the MDP framework, the physical state st ∈ S must be augmented with scheduler information zt ∈ Z. Let zt = (σ(t), it ), where σ(t) ∈ {0, . . . , M − 1} is the active phase index and it represents auxiliary schedule context (e.g., elapsed phase time or global step index). Formal Markovian Sufficiency Condition. transition dynamics satisfy:

We define the scheduler state zt as sufficient if the joint

Pr(st+1 ∈ A, zt+1 ∈ B | s0:t , a0:t , z0:t ) = Pσ(t) (A | st , at ) q(B | zt ),

(10)

for all measurable sets A ⊆ S and B ⊆ Z, where Pm (· | s, a) is the transition kernel of phase m, and q(· | zt ) is the exogenous schedule update kernel. When Equation (10) holds, the augmented state s̃t = (st , zt ) forms f = (S × Z, A, Pe, re, γ). To preserve sample efficiency, state a time-homogeneous Markov Decision Process M augmentation must minimize the scheduler state space Z. For periodic or cyclic phase structures, folding global time t into a compact within-phase clock τ (t) = t mod Tm quotient space preserves state recurrence across cycles while satisfying Equation (10): Corollary 1 (Periodic Phase-Clock Augmentation). Suppose phases follow a deterministic cycle with phase PM −1 Pm−1 durations (T0 , T1 , . . . , TM −1 ). Let Tperiod = m=0 Tm denote the total cycle length, and let Sm = k=0 Tk denote the entry step offset of phase m within a cycle (with S0 = 0). For global step t, let t̄ = t mod Tperiod . The phase index σ(t) and within-phase clock τ (t) are uniquely defined by: such that Sm ≤ t̄ < Sm + Tm ,

σ(t) = m

τ (t) = t̄ − Sσ(t) .

(11)

Then zt = (σ(t), τ (t)) is sufficient. Consequently, the augmented state (st , σ(t), τ (t)) is Markovian, timehomogeneous, and compact. If the same phase m occurs multiple times within a single macro-cycle with different durations or successor phases, the auxiliary schedule context it must additionally include the macro-cycle phase occurrence index j to maintain unique state identification. Table 3 illustrates the unique sequence of pairs (σ(t), τ (t)) for a periodic 4-phase schedule. Table 3: Periodic phase schedule and unique state augmentation pairs (σ(t), τ (t)). t

0

1

2

3

4

5

6

7

8

9

10

11

12

13

...

m = σ(t) τ (t)

0 0

0 1

1 0

1 1

1 2

2 0

2 1

3 0

3 1

3 2

3 3

0 0

0 1

1 0

... ...

The formal proof of Corollary 1 is provided in Appendix A.2

13

A.1.2

A Transient–Quasi-Stationary View

We now split each phase into an early transient period and a later settled quasi-stationary period. Let pπm (t) be the state-transition probabilities induced by policy π at time step t. Let µπm,j be the entry distribution at occurrence j produced by the earlier policies, and let ρref m be a possible near-invariant law. Let W1 be the Wasserstein-1 distance that is used to measure how far the current distribution is from the settled one in units of distance on the state space. For ε > 0, define the distance at time step t inside the phase m
π dπm,j (t) = W1 µπm,j pπm (t), ρref (12) m pm (t) . Let us define the ε-occurrence-recovery time ℓπm,j (ε) as the first step after which this distance remains at or below ε > 0 for all later steps such that  ℓπm,j (ε) = inf ℓ ∈ {0, . . . , Tm,j } : dπm,j (t) ≤ ε for all t = ℓ, . . . , Tm,j . (13) If no such ℓ exists, we set ℓπm (ε) = Tm,j . The ε-occurrence-recovery time depends on both the policy and the entry law, and therefore is not a fixed property of the phase. For any comparison window 0 ≤ Lm,j ≤ Tm,j , the phase-local return Jm,j (π; µπm,j ) for phase m is given by qs tr Jm,j (π; µπm,j ) = Jm,j (π; Lm,j )+Jm,j (π; Lm,j ),     Lm−1,j Tm−1,j X X qs tr where, Jm,j (π; Lm,j ) = E γ j rm,j (sj , aj ) , and Jm,j (π; Lm,j ) = E γ j rm,j (sj , aj ) . j=0

(14)

j=Lm,j

sp and π sh be the learned specialized and shared policies, each evaluated under the cyclic entry law Let πm it induces, so that the differences in handoff are included in the comparison. For a split point Lm,j between transient and quasi-stationary phase inside phase m at occurrence j, define the average regime differentials qs qs Gm,j = J¯m,j,sp − J¯m,j,sh ,

For occurrence (m, j), let Zm,j =

PTm,j −1 t=0

tr tr Km,j = J¯m,j,sh − J¯m,j,sp .

(15)

γ t and PLm,j −1 t=0

λm,j =

Zm,j

γt

.

(16)

Decomposing the normalized return difference J¯m,j,sp − J¯m,j,sh over occurrence (m, j) yields:  
qs qs tr tr − J¯m,j,sh + λm,j J¯m,j,sp − J¯m,j,sh J¯m,j,sp − J¯m,j,sh = (1 − λm,j ) J¯m,j,sp = (1 − λm,j )Gm,j − λm,j Km,j ,

(17)

Let ωm,j ≥ 0 denote the normalized evaluation weight assigned to occurrence (m, j), with Multiplying by ωm,j and summing over all phase occurrences gives: ¯ π sp ) − J(b ¯ π sh ) = ∆J¯ = J(b

M −1 X X m=0

For compact phase-level reporting, let ωm = λm = Gm =

j

m,j ωm,j = 1.

(18)

P

j∈Om ωm,j and define

1 X ωm,j λm,j , ωm j∈Om P j ωm,j (1 − λm,j )Gm,j ωm (1 − λm )

ωm,j [(1 − λm,j )Gm,j − λm,j Km,j ] .

P

P ,

14

Km =

j ωm,j λm,j Km,j

ωm λ m

.

(19)

The values of Km and Gm are set to zero when the duration of the transient and quasi-stationary periods is zero, respectively. Now we can write the advantage of using learned specialized policies over a learned single shared policy as X ¯ π sp ) − J(b ¯ π sh ) = ∆J¯ := J(b ωm [(1 − λm )Gm − λm Km ]. (20) m

This compares learned shared and learned specialized policies, π bsh and π bsp , respectively, not the best sh sp policies in unrestricted classes, i.e., π and π . This framework yields a simple three-step empirical diagnostic: we measure the performance gap between shared and specialized models across phase age, measure the decay rate of the entry-state distribution post-boundary, and test whether recovery times remain invariant to changes in Tm . Ultimately, longer phase horizons favor specialists only when they expose a settled quasistationary region where local gains dominate (Gm > 0); phase length alone does not guarantee a specialist advantage. A.1.3

Hand off Costs with Two Goals

Proposition 3 (Hand off threshold for two-goal problem). Consider two consecutive phases. Phase 0 has duration T0 and offers two possible choices of goals: i ∈ {00, 01} one with a higher reward than other, i.e., g00 gives the larger phase-0 reward, while g01 produces a more favourable entry state for phase 1. Let Fi (s) be the expected discounted return collected during phase 0, starting from state s, under goal i. Let µi be the phase-1 entry-state distribution induced by this choice of goal. Define the continuation value:   V1cont (µi ) = Es1 ∼µi V1cont (s1 ) . (21) Assume that a phase-local specialized policy optimizes only the phase-0 return Fi (s), whereas a coordinated full-trajectory controller also accounts for the continuation value in phase 1. If F00 (s) > F01 (s) then the phase-local specialized policy prefers the immediate-reward goal g00 . By contrast, the full-trajectory controller prefers the safe-handoff goal g01 if and only if   γ T0 V1cont (µ01 ) − V1cont (µ00 ) > F00 (s) − F01 (s). (22) When both inequalities hold, the return loss caused by the uncoordinated phase-local choice is   ∆coord (s) = γ T0 V1cont (µ01 ) − V1cont (µ00 ) − [F00 (s) − F01 (s)] > 0. A.1.4

(23)

Sample Allocation and Sample Thresholds

qs (b πm,b ) denote the expected settled return of specialist m trained on b local interactions, and let Let J¯m qs πm,∞ ) represent its asymptotic performance under unlimited samples (b → ∞). J¯m (b

P Because an ensemble of M specialists partitions a total budget B into local allocations Bm = ζm B (where ζm = 1, and ζm = 1/M for balanced sampling), each specialist receives significantly fewer environment steps than a shared model trained on the full budget B. To guarantee that local learning starvation does not destroy the settled specialization gain, we define the local sample threshold B0,m (δ) as the minimum allocation required for specialist m to recover its asymptotic return within a tolerance δ > 0:  qs qs B0,m (δ) = inf b ∈ N : J¯m (b πm,∞ ) − J¯m (b πm,b ) ≤ δ . (24) Proposition 4 (Sufficient Budget for Positive Specialization Gain). Assume that under unlimited samples, specialist m achieves a net specialization advantage with the quasi-stationary gain Gm,∞ > δ. If the total interaction budget B satisfies: B0,m (δ) , (25) B ≥ max m ζm then every specialist receives sufficient samples (Bm ≥ B0,m (δ)) to maintain a strictly positive local gain: qs Gqs m,ref (Bm ) ≥ Gm,∞ − δ > 0.

For balanced sampling (ζm = 1/M ), condition (25) simplifies to B ≥ M maxm B0,m (δ). 15

(26)

A.1.5

Gradient Geometry as an Optimization Mechanism Test

We use phase-restricted gradients to test one potential optimization mechanism driving shared architecture performance. Let Lm (θ) denote a differentiable phase-local objective evaluated at a common parameter PM −1 vector θ under a fixed reference data distribution. Given normalized phase weights wm ≥ 0 with m=0 wm = 1, define: M −1 M −1 X X L(θ) = wm Lm (θ), gm (θ) = ∇θ Lm (θ), g(θ) = wm gm (θ). (27) m=0

m=0

In balanced diagnostic sweeps, we set wm = 1/M . Under a shared gradient ascent step θ+ = θ + ηg(θ), local objective updates admit a first-order Taylor expansion: Proposition 5 (First-Order Decomposition of Shared Updates). For a differentiable local objective Lm , a shared gradient update θ+ = θ + ηg(θ) yields: X Lm (θ+ ) − Lm (θ) = ηwm ∥gm ∥2 + η wk ⟨gm , gk ⟩ + o(η). (28) k̸=m

The first term represents phase m’s self-improvement, while the second term captures cross-phase transfer. A shared update is locally destructive to phase m if and only if the aggregate inner product is strictly negative: ⟨gm , g(θ)⟩ < 0. Negative pairwise inner products ⟨gm , gk ⟩ < 0 alone do not imply local objective degradation. To measure aggregate directional alignment without introducing mechanical scaling artifacts with respect to phase count M , we introduce the orthogonal-null normalized statistics: PM −1 ∥ m=0 wm gm ∥2 , Cg = 1 − Ag . (29) Ag = PM −1 2 2 m=0 wm ∥gm ∥ Under mutually orthogonal phase gradients (⟨gm , gk ⟩ = 0 for k ̸= m), Ag = 1 identically for any phase count M . A positive conflict statistic Cg > 0 indicates that negative cross-phase terms dominate the aggregate update. Phase-to-aggregate cosine similarity also exhibits a fundamental orthogonal baseline, as formalized below. Proposition 6 (Orthogonal Gradient Alignment Scaling). Let g0 , . . . , gM −1 ∈ Rd be non-zero mutually PM −1 1 orthogonal phase gradients (⟨gm , gk ⟩ = 0 for all m ̸= k) and let g = M k=0 gk denote the balanced aggregate update (wm = 1/M ). If all gradients share equal norms ∥gm ∥ = C > 0, then: 1 cos(gm , g) = √ . M

(30)

More generally, for arbitrary gradient norms ∥gm ∥, the orthogonal alignment baseline scales as: cos(gm , g) = qP

∥gm ∥

M −1 2 k=0 ∥gk ∥

.

(31)

√ Therefore, a decay rate of 1/ M in cosine similarity indicates that the shared update vector is being spread across orthogonal task directions—a phenomenon we term gradient dilution—rather than experiencing destructive multi-task interference [34]. Empirically, we differentiate the phase-restricted clipped PPO actor surrogate at commonly trained checkpoints using global advantage normalization. As reported in Appendix A.4.1, empirical evaluation rejects destructive interference in HAPS: Ag ≈ 1.0, and √ fewer than 2% of phases exhibit ⟨gm , g⟩ < 0. The observed drop in cosine alignment closely tracks 1/ M , confirming gradient dilution as the primary optimization bottleneck.

A.2

Proofs and Derivations

This appendix provides complete formal proofs for all theoretical propositions and identities presented in the main text and Appendix A.1. The notation is inherited directly from the corresponding statements. 16

A.2.1

Structural Results

Proposition 1 (Scheduler-state augmentation) If zt is sufficient, then the augmented state set = (st , zt ) defines a time-homogeneous MDP and recovers Markovianity. Proof. Let Ht = (s0 , a0 , z0 , s1 , a1 , z1 , . . . , st , at , zt ) denote the complete execution trajectory history up to time step t. To prove that the augmented state s̃t = (st , zt ) ∈ S × Z defines a time-homogeneous Markov Decision Process (MDP), we must establish three distinct properties: 1. Markovian Memorylessness: Pr(s̃t+1 ∈ C | Ht ) = Pr(s̃t+1 ∈ C | s̃t , at ). 2. Time-Homogeneity: The transition probability distribution depends on s̃t and at , but has no explicit dependence on the step index t. 3. Reward Consistency: Expected single-step rewards depend solely on (s̃t , at ). Step 1: Proving the Markov Memoryless Property. Let C = A × B be any product measurable set in the augmented state space S × Z, where A ⊆ S and B ⊆ Z. By the definition of conditional probability, we factorize the joint transition of physical state st+1 and phase scheduler state zt+1 given complete history Ht : Pr(s̃t+1 ∈ A × B | Ht ) = Pr(st+1 ∈ A, zt+1 ∈ B | Ht ) (32) = Pr(st+1 ∈ A | Ht , zt+1 ∈ B) · Pr(zt+1 ∈ B | Ht ). By the definition of scheduler state sufficiency, the exogenous scheduler update law is conditionally independent of past physical states and actions given zt : Pr(zt+1 ∈ B | Ht ) = q(B | zt ).

(33)

Similarly, conditioned on the active phase index σ(t) = h(zt ), the physical transition law depends only on the current physical state st and action at : Pr(st+1 ∈ A | Ht , zt+1 ∈ B) = ph(zt ) (A | st , at ).

(34)

Substituting 33 and 34 into 32 yields: Pr(s̃t+1 ∈ A × B | Ht ) = ph(zt ) (A | st , at ) q(B | zt ).

(35)

Because the right-hand side of 35 depends strictly on s̃t = (st , zt ) and at , the historical terms (s0:t−1 , a0:t−1 , z0:t−1 ) drop out entirely, proving the Markov property. Step 2: Proving Time-Homogeneity. Notice that neither the phase-local transition kernel ph(zt ) (· | st , at ) nor the scheduler update kernel q(· | zt ) contains an explicit parameter t. The time dependency in the environment is entirely localized inside zt . Consequently, for any two step indices t1 ̸= t2 , if s̃t1 = s̃t2 = (s, z) and at1 = at2 = a, then: Pr(s̃t1 +1 ∈ C | s̃t1 = (s, z), at1 = a) = Pr(s̃t2 +1 ∈ C | s̃t2 = (s, z), at2 = a) = Pe(C | (s, z), a),

(36)

where Pe(A × B | (s, z), a) := ph(z) (A | s, a) q(B | z) defines a time-invariant transition kernel on S × Z. Step 3: Reward Consistency. The expected single-step reward in phase m = h(zt ) is given by rm (st , at ). Defining the augmented reward function r̃(s̃t , at ) := rh(zt ) (st , at ), we see that r̃ : (S × Z) × A → R is a well-defined, stationary function on the augmented state space. f = (S×Z, A, Pe, re, γ) is a formally valid, time-homogeneous Combining Steps 1, 2, and 3 establishes that M Markov Decision Process.

17

Corollary 1 (Periodic Phase-Clock Augmentation) Suppose phases follow a deterministic cycle with PM −1 phase durations (T0 , T1 , . . . , TM −1 ). Let Tperiod = m=0 Tm denote the total cycle length, and let Sm = Pm−1 k=0 Tk denote the entry step offset of phase m within a cycle (with S0 = 0). For global step t, let t̄ = t mod Tperiod . The phase index σ(t) and within-phase clock τ (t) are uniquely defined by: σ(t) = m

such that Sm ≤ t̄ < Sm + Tm ,

τ (t) = t̄ − Sσ(t) .

(37)

Then zt = (σ(t), τ (t)) is sufficient. Consequently, the augmented state (st , σ(t), τ (t)) is Markovian, timehomogeneous, and compact. Proof. Let (T0 , T1 , . . . , TM −1 ) be a sequence of positive integer phase durations. The total cycle period is PM −1 Pm−1 Tperiod = m=0 Tm , and the cumulative entry offset for phase m within a cycle is Sm = k=0 Tk , with S0 = 0. By definition of Corollary 1, for any global step index t ∈ N0 , let t̄ = t mod Tperiod ∈ {0, . . . , Tperiod − 1}. The phase index σ(t) ∈ {0, . . . , M − 1} and within-phase clock τ (t) ∈ {0, . . . , Tσ(t) − 1} are uniquely determined by: σ(t) = m ⇐⇒ Sm ≤ t̄ < Sm + Tm , τ (t) = t̄ − Sσ(t) . (38) Step 1: Deterministic One-Step Update Operator F . We now evaluate the one-step temporal evolution of zt = (σ(t), τ (t)) from step t to step t + 1. Consider two cases based on the value of the within-phase clock τ (t): 1. Intra-Phase Step (τ (t) < Tσ(t) − 1): The step t + 1 remains inside the same phase m = σ(t). The within-phase clock advances by 1, yielding: σ(t + 1) = σ(t),

τ (t + 1) = τ (t) + 1.

(39)

2. Phase Boundary Transition (τ (t) = Tσ(t) − 1): Step t is the final step of phase m = σ(t). At step t + 1, the system transitions to the next phase in the macro-cycle, (m + 1) mod M , resetting the within-phase clock: σ(t + 1) = (σ(t) + 1) mod M, τ (t + 1) = 0. (40) Thus, the schedule state update zt+1 = (σ(t + 1), τ (t + 1)) is governed by the deterministic function F : Z → Z: ( (m, τ + 1), if τ < Tm − 1, F (m, τ ) = (41) ((m + 1) mod M, 0), if τ = Tm − 1, where zt+1 = F (zt ) holds for any arbitrary phase lengths (T0 , . . . , TM −1 ). Step 2: Kernel Factorization and Sufficiency. Because zt+1 = F (zt ) is a deterministic mapping, the exogenous scheduler kernel q(B | zt ) in Proposition 1 reduces to a point-mass Dirac delta measure: ( 1, if F (zt ) ∈ B, q(B | zt ) = δF (zt ) (B) = (42) 0, if F (zt ) ∈ / B, for any measurable set B ⊆ Z. Substituting q(B | zt ) = δF (zt ) (B) into 35 gives: Pr(st+1 ∈ A, zt+1 ∈ B | Ht ) = pσ(t) (A | st , at ) δF (zt ) (B),

(43)

which is conditionally independent of past trajectory history H:t−1 given (st , at , zt ) and also time-invariant.

18

Step 3: Compactness and Recurrence. Z=

M −1 [ m=0

The domain of zt is the finite discrete union: ({m} × {0, . . . , Tm − 1}) ,

(44)

PM −1 which has finite cardinality |Z| = Tperiod = m=0 Tm . Unlike global time t ∈ N0 , which grows infinitely and prevents state reuse, zt recurs periodically every Tperiod steps (zt+Tperiod = zt ). This establishes that the augmented state (st , σ(t), τ (t)) forms a compact, time-homogeneous Markov Decision Process. Proposition 2 (Shared–multipolicy equivalence) There is a bijection between shared policies π sh (· | s, i, m) and an ensemble of specialized policies π sp = (π0 (· | s, i), . . . , πM −1 (· | s, i)) expressed through the gated map Γ such that π sp (a | s, i, m) = Γ(π sp )(a | s, i, m) = πm (a | s, i), (45) π sh (a | s, i, m) = πm (a | s, i) = π sp (a | s, i, m). It is always possible to represent a shared policy with an ensemble of specialized policies and vice versa. Proof. Let x = (s, i) denote the shared input observation (physical state and context). Given an ensemble of specialized policies (π0 , . . . , πM −1 ), define the canonical concatenation mapping Φ: Φ(π0 , . . . , πM −1 )(a | x, m) = πm (a | x).

(46)

Conversely, given a shared phase-conditioned policy π, define the phase slicing operator Ψ: Ψ(π)m (a | x) = π(a | x, m).

(47)

For any specialist ensemble, Ψ(Φ(π0 , . . . , πM −1 ))m = πm for all m ∈ {0, . . . , M − 1}. For any shared policy, Φ(Ψ(π))(a | x, m) = π(a | x, m). Thus, Φ and Ψ define bijective inverse mappings between the unrestricted shared policy space Πsh and the product specialist space Π0 × · · · × ΠM −1 . Consequently, corresponding policies assign identical action probability distributions π(a | x, m) = πm (a | x) across all physical states and phase indices. Under identical initial state distributions and transition dynamics, induction over time steps t proves identical trajectory laws and identical expected returns. Taking ⋆ ⋆ . supremums over both policy spaces establishes J¯sh = J¯sp A.2.2

Geometric Intuition of Policy Class Equivalence

Figure 3 provides a geometric visualization of the equivalence result established in Proposition 2. While shared and multi-policy architectures possess identical functional expressivity in the common achievable space Πsh (Figure 3a), their distinct optimization dynamics lead to different realized solutions under finite ¯ data, leading to different average global return J. sp As depicted in Figure 3b, each specialist policy π bm in the multi-policy ensemble is optimized indepensp sp b sp = (b dently within its local space Πm . The exact gating map Γ embeds this product tuple π π0sp , . . . , π bM −1 ) sh into Π without projection loss. However, because local training is decoupled across phases, specialists optimize for phase-local performance without accounting for cross-phase coordination. Consequently, the global performance of the assembled multi-policy depends heavily on boundary handoffs: when phase-local optimization yields terminal states that do not compromise initialization in subsequent phases,specialization can yield strict return gains over a single shared policy (∆J¯ > 0). Conversely, when handoff mismatch or data starvation occurs, uncoordinated local optimization degrades overall trajectory performance (∆J¯ < 0). Note that for visual clarity, this representation assumes a unique global optimal policy π ⋆ .

19

(a) Common general policy space and global return

M −1 sp m=0 Πm



π bsp

higher return J¯

∆J¯ < 0

∆J¯ > 0

han

sma

ll h

and

off

lar ge

π b

π bsp

doff

π⋆

sh

t

Q

cos

Πsh ≡ Γ

cos

t

(b) Phase-local specialist learning Πsp 0

Πsp 1

π0⋆

Πsp M −1 π1⋆

π b0sp

π b1sp

···

⋆ πM −1

Γ: sp π bM −1

Q

sp sh m Πm → Π

Γ(π)(a | s, i, m) = πm (a | s, i)

sp b sp = (b π π0sp , π b1sp , . . . , π bM −1 )

Figure 3: Geometric intuition of policy class equivalence and local specialist learning. (a)The common achievable policy space Πsh encompasses both shared and multi-policy solutions, as established by Proposition 2. Depending on the alignment of phase-local entry distributions with deployment handoffs, exact multi-policy assembly yields either a specialization gain (∆J¯ > 0) or a handoff mismatch penalty sp (∆J¯ < 0). (b) Individual phase specialists π bm are trained independently within local policy spaces Πsp m b sp is assembled via the exact gating map Γ with zero under phase-local entry distributions. The tuple π projection loss.

20

A.2.3

Transient Handoff, Recovery, and Regime Bounds

Proposition 3 (Hand off threshold for two-goal problem) Consider two consecutive phases. Phase 0 has duration T0 and offers two possible choices of goals: i ∈ {00, 01} one with a higher reward than other, i.e., g00 gives the larger phase-0 reward, while g01 produces a more favourable entry state for phase 1. Let Fi (s) be the expected discounted return collected during phase 0, starting from state s, under goal i. Let µi be the phase-1 entry-state distribution induced by this choice of goal. Define the continuation value:   V1cont (µi ) = Es1 ∼µi V1cont (s1 ) . (48) Assume that a phase-local specialized policy optimizes only the phase-0 return Fi (s), whereas a coordinated full-trajectory controller also accounts for the continuation value in phase 1. If F00 (s) > F01 (s) then the phase-local specialized policy prefers the immediate-reward goal g00 . By contrast, the full-trajectory controller prefers the safe-handoff goal g01 if and only if   γ T0 V1cont (µ01 ) − V1cont (µ00 ) > F00 (s) − F01 (s). (49) When both inequalities hold, the return loss caused by the uncoordinated phase-local choice is   ∆coord (s) = γ T0 V1cont (µ01 ) − V1cont (µ00 ) − [F00 (s) − F01 (s)] > 0.

(50)

Proof. Fix an initial state s ∈ S at step t = 0 of phase 0. For each choice of goals i ∈ {00, 01}, the total expected trajectory return across both phases decomposes additively at the phase boundary t = T0 : Qi (s) := Fi (s) + γ T0 V1cont (µi ), (51) hP i T0 −1 t where Fi (s) = Eπi t=0 γ r0 (st , at ) s0 = s denotes the expected phase-0 return, and µi (·) = Pr(sT0 ∈

· | s0 = s, πi ) is the induced phase-1 entry state distribution under goal choice i. By definition, an uncoordinated phase-local specialist π b0,sp optimizes strictly the phase-0 return Fi (s). Consequently, under condition F00 (s) > F01 (s), the specialist uniquely selects commitment g00 . Conversely, a coordinated full-trajectory controller selects commitment g01 over g00 if and only if Q01 (s) > Q00 (s). Substituting (51) yields: Q01 (s) > Q00 (s) ⇐⇒ F01 (s) + γ T0 V1cont (µ01 ) > F00 (s) + γ T0 V1cont (µ00 )   ⇐⇒ γ T0 V1cont (µ01 ) − V1cont (µ00 ) > F00 (s) − F01 (s),

(52)

which establishes the threshold condition in (49). Finally, when F00 (s) > F01 (s) and (49) simultaneously hold, the specialist selects g00 while the optimal coordinated policy selects g01 . The resulting suboptimality gap ∆coord (s) := Q01 (s) − Q00 (s) evaluates to:   ∆coord (s) = γ T0 V1cont (µ01 ) − V1cont (µ00 ) − [F00 (s) − F01 (s)] > 0, (53) where strict positivity follows directly from (49), completing the proof of (50). A.2.4

Sample Allocation and Sample Thresholds

Propositon 4 (Sufficient Budget for Positive Specialization Gain) Assume that under unlimited samples, specialist m achieves a net specialization advantage with the quasi-stationary gain Gm,∞ > δ. If the total interaction budget B satisfies: B0,m (δ) B ≥ max , (54) m ζm then every specialist receives sufficient samples (Bm ≥ B0,m (δ)) to maintain a strictly positive local gain: qs Gqs m,ref (Bm ) ≥ Gm,∞ − δ > 0.

For balanced sampling (ζm = 1/M ), condition (54) simplifies to B ≥ M maxm B0,m (δ). 21

(55)

Proof. By definition of the infimum in (24), for any phase-local allocation b ≥ B0,m (δ), the learning gap relative to the asymptotic limit satisfies: qs qs qs qs J¯m (b πm,∞ ) − J¯m (b πm,b ) ≤ δ =⇒ J¯m (b πm,b ) ≥ J¯m (b πm,∞ ) − δ. B

(56)

(δ)

, multiplying by ζm > 0 yields: Under the budget condition B ≥ maxm 0,m ζm   B0,m (δ) Bm = ζm B ≥ ζm = B0,m (δ) ∀m ∈ {0, . . . , M − 1}. ζm

(57)

Substituting b = Bm into the suboptimality bound gives: qs qs J¯m (b πm,sp,Bm ) ≥ J¯m (b πm,∞ ) − δ.

(58)

qs Subtracting the return of the shared policy J¯m (b πsh,B ) trained on total budget B yields the expected settled qs specialization gain Gm,ref (Bm ): qs qs qs ¯qs πm,sp,B ) − J¯m Gqs (b πsh,B ) ≥ J¯m (b πm,∞ ) − δ − J¯m (b πsh,B ). m m,ref (Bm ) = Jm (b

(59)

qs (b πsh,B ) ≤ Because the shared policy’s return at finite budget B is upper bounded by its asymptotic return (J¯m qs ¯ πsh,∞ )), we obtain: Jm (b qs ¯qs πm,∞ ) − J¯m Gqs (b πsh,∞ ) − δ = Gqs m,∞ − δ. m,ref (Bm ) ≥ Jm (b

(60)

qs Since δ < Gqs m,∞ by assumption, Gm,∞ − δ > 0, proving that every local specialist achieves a strictly positive

specialization gain when B ≥ maxm A.2.5

B0,m (δ) . ζm

Gradient Geometry and Optimization Proofs

Proposition 5 (First-Order Decomposition of Shared Updates) For a differentiable local objective Lm , a shared gradient update θ+ = θ + ηg(θ) yields: X Lm (θ+ ) − Lm (θ) = ηwm ∥gm ∥2 + η wk ⟨gm , gk ⟩ + o(η). (61) k̸=m

Proof. Because the local objective Lm is differentiable atPthe parameter vector θ, its first-order Taylor expansion in the direction of the aggregate update g(θ) = k wk gk (θ) is: Lm (θ + ηg) − Lm (θ) = η⟨∇Lm (θ), g(θ)⟩ + o(η). Substituting gm = ∇Lm (θ) and expanding g(θ) gives: * + M −1 X X wk gk = wm ∥gm ∥2 + ⟨gm , g⟩ = gm , wk ⟨gm , gk ⟩. k=0

(62)

(63)

k̸=m

Multiplying by step size η proves Prop. 5. Define cm = ⟨gm , g⟩. If cm < 0, then because limη→0+ o(η) η = 0, there exists a step size threshold η0 > 0 such that |o(η)| < η |c2m | for all η ∈ (0, η0 ). Consequently:

Lm (θ + ηg) − Lm (θ) ≤ ηcm + |o(η)| < ηcm −

ηcm ηcm = < 0, 2 2

(64)

proving that a sufficiently small shared gradient step strictly decreases phase m’s local objective. Proposition 6 (Orthogonal Gradient Alignment Scaling) Let g0 , . . . , gM −1 ∈ Rd be non-zero mutually PM −1 1 orthogonal phase gradients (⟨gm , gk ⟩ = 0 for all m ̸= k) and let g = M k=0 gk denote the balanced aggregate update (wm = 1/M ). If all gradients share equal norms ∥gm ∥ = C > 0, then: 1 cos(gm , g) = √ . M 22

(65)

More generally, for arbitrary gradient norms ∥gm ∥, the orthogonal alignment baseline scales as: cos(gm , g) = qP

∥gm ∥

M −1 2 k=0 ∥gk ∥

.

(66)

1 Proof. Let g0 , . . . , gM −1 ∈ Rd satisfy ∥gm ∥ = C > 0 and ⟨gm , gk ⟩ = 0 for all m ̸= k. Let g = M denote the aggregate gradient under balanced weights wm = 1/M .

Step 1: Aggregate Gradient Norm. Expanding the squared norm ∥g∥2 : * M −1 + M −1 M −1 M −1 1 X 1 X 1 X X 2 ∥g∥ = gk , gj = 2 ⟨gk , gj ⟩. M M j=0 M j=0 k=0

PM −1 k=0

gk

(67)

k=0

By mutual orthogonality, cross-terms ⟨gk , gj ⟩ = 0 for k ̸= j. Substituting ∥gk ∥2 = C 2 : M −1

∥g∥2 =

1 X C2 C 1 ∥gk ∥2 = 2 · (M C 2 ) = =⇒ ∥g∥ = √ . 2 M M M M

(68)

k=0

The inner product ⟨gm , g⟩ evaluates to: * + M −1 1 X 1 C2 gk = ⟨gm , g⟩ = gm , ⟨gm , gm ⟩ = . M M M

Step 2: Cosine Similarity.

(69)

k=0

Substituting into the cosine similarity definition: 2

C ⟨gm , g⟩ M = cos(gm , g) = ∥gm ∥∥g∥ C · √C

M

=

C2 M C2 √ M

1 =√ . M

1 If gradient norms are unequal (∥gm ∥ ̸= ∥gk ∥), Step 1 generalizes to ∥g∥ = M general orthogonal baseline: ∥gm ∥ cos(gm , g) = qP . M −1 2 ∥g ∥ k k=0

A.3

(70) qP M −1 k=0

∥gk ∥2 , yielding the (71)

Experimental Details

This appendix gives the environment definitions, network architectures, sample budgets, and evaluation protocols needed to reproduce every experiment. Appendix A.3.1 describes the architectures and the training protocol common to all runs; Appendix A.3.2, A.3.3 and A.3.4 describe the three environments, and Appendix A.3.5 describes how results are aggregated and reported. A.3.1

Architectures and Common Training Protocol

Policy inputs and parameter sharing. Let st be the environment observation, emt ∈ {0, 1}M the onehot code of the active phase, and c(τt ) = [sin(2πτt /T ), cos(2πτt /T )] the encoding of the local clock within the current block. All experiments use equal block durations Tm ≡ T . All architectures are trained with PPO. Single+clock is a single actor–critic taking [st , emt , c(τt )]. Multihead takes [st , emt ] and combines a shared actor body and a shared critic with M phase-specific linear action heads; at each step the head indexed by the active phase emits the action, so the shared body is updated from every transition while each head is updated only from transitions of its own phase. The phase code is also supplied to the body, allowing it to form phase-conditioned features rather than only phase-agnostic ones. Multipolicy uses M independent actor–critics, the active phase selecting which one acts and is updated; its input is st , or [st , c(τt )] in the multipolicy+clock variant. The phase code is not supplied here, since selecting the network already identifies the phase. Unless an ablation states otherwise, all actor and critic MLPs have two hidden layers of width 256 with Tanh activations and the Stable-Baselines3 [35] default initialization. 23

Secondary controls in ablation studies. Two input ablations isolate the value of the phase and clock signals: the phase-blind baseline observes only st , and the phase-conditioned baseline observes [st , emt ], that is, the shared architecture without the clock encoding. Capacity controls vary width, using [128, 128] or [512, 512] in place of the default [256, 256], and depth, using four or eight hidden layers of width 256 in place of two; the same setting is applied to every architecture. In the warm-start protocol with nominal per-phase budget B0 , a single phase-blind policy is first trained on the full schedule for B0 interactions, its parameters are copied into all M specialists, and each copy is fine-tuned for B0 (M − 1)/M interactions, so the total is M B0 , matching standard multipolicy training. PCGrad [18] and CAGrad [19] split each shared-policy PPO minibatch by phase, compute one complete PPO-loss gradient per represented phase, and combine those gradients by the respective rule before the optimizer step. The two distillation controls transfer policies in opposite directions. The specialist-to-shared control compresses the trained specialist ensemble into a single phase-conditioned student by minimizing the KL divergence to each specialist’s action distribution on that phase’s rollouts, following policy distillation [23]. The shared-to-specialist control trains each specialist with reinforcement learning while adding a KL penalty toward a pretrained shared teacher, with the coefficient decayed from β0 to zero over training, following the teacher-guided principle of kickstarting [36]. The latter is related to Distral [24], although we do not jointly learn a shared policy centroid. Training and evaluation distinction. Shared and multihead agents train on complete cyclic trajectories and retain the physical state at phase boundaries. A standard multi-policy instead trains in a phase-fixed environment whose episode lasts T steps and begins from the domain’s phase-local reset law. Specialized multi-policies are updated separately and receive approximately Bm = B/M interactions under a balanced total budget B. Crucially, all architectures are evaluated on the same complete cyclic environment: selecting a new specialized policy does not reset the physical state. The resulting score therefore includes any mismatch between phase-local training resets and deployment-time entry states. PPO hyperparameters. Hyperparameters are held fixed across architectures within each reported comparison. We use the domain-specific settings in Table 4. The different rollout sizes reflect the substantially different simulation horizon scales of HAPS and the two diagnostic domains. Table 4: Default PPO settings.

A.3.2

Setting

HAPS

PhasePendulum

PhaseGridWorld

Hidden layers Parallel environments Rollout steps per environment Minibatch size Epochs per update Discount γ GAE parameter Learning rate Entropy coefficient PPO clip range Target KL Observation/reward normalization

[256, 256] 8 2048 1024 10 0.99 0.95 3×10−4 0.02 0.2 0.02 yes/yes

[256, 256] 8 256 256 10 0.99 0.95 3×10−4 0.01 0.2 0.03 no/no

[256, 256] 8 256 256 10 0.99 0.95 3×10−4 0.01 0.2 0.03 no/no

HAPS Coverage Environment

Scope and geometry. We consider only the mobility and coverage component of the simulation environment introduced by [16]. The idea is to find the optimal position of the HAPS given that the user density and traffic vary over time but in a periodic manner. In particular, different phases inside a period are differentiated by different user densities and traffic loads. The entire service region of the HAPS network is divided into multiple cells; only a few selected cells act as hotspots where the user density and traffic are higher than the rest of the cells.

24

The service region is a square of side A = 300 km, partitioned into 5 × 5 equal cells. Unless an ablation states otherwise, H = 2 platforms operate at altitudes zh ∈ [18, 20] km with θmax = 60◦ . Their footprint radii therefore lie between 31.2 and 34.6 km. Each step permits horizontal increments of 3 km per coordinate and a vertical increment of 3(20/300) = 0.2 km, followed by clipping to the service and altitude bounds. The per-HAPS service maximum capacity is 200 users. For grid cell c = (i, j), the number of new users is sampled independently at timestep t with phase m = σ(t) as Nct ∼ Poisson(λm c ). Their horizontal coordinates are uniform within the cell. Their lifetimes are independent geometric random variables with parameter Ξc , supported on the positive integers. The base arrival-rate and departure-parameter arrays are     0.10 0.08 0.20 0.12 0.09 1.60 1.20 1.40 1.10 1.30 0.12 0.07 0.15 0.14 0.11 1.30 1.19 1.70 1.40 1.20      , Ξ = 0.08 0.10 0.05 0.06 0.04 . 1.20 1.50 1.30 1.60 1.40 (72) Λ=     0.11 0.09 0.18 0.13 0.10 1.10 1.40 1.20 1.30 1.50 0.09 0.11 0.12 0.08 0.07 1.40 1.30 1.60 1.20 1.10 m In phase m, the two selected cells for being hotspots use λm c = 2Λc , and for all other cells, we use λc = 0.1Λc . The lifetime parameters Ξc do not change with phase. The mean lifetime, 1/Ξc ∈ [5, 25] steps, determines how long demand generated in the previous phase should persist after a boundary. The different actions from the HAPS lead to movement in 3D space, and the reward depends only on coverage and potential interference. We perform an ablation study for different phase count, dwell time, budget, and architecture.

Phase Structure and Episodes. Each episode is composed of two full cycles. A cycle is composed of the M different phases that follow each other in increasing index order. Each phase lasts T time steps, making the whole episode last 2M T time steps. While designing the phases, it is possible to select the hotspot cells in such a way that the distance between a hotspot from one phase and the other is high, low, or medium. neighbour places consecutive hotspots in adjacent cells, random draws them uniformly, and maxjump places them as far apart as the grid allows. These three dispositions correspond to increasing heterogeneity between consecutive phases. Association and reward. The objective of the problem is to maximize the number of covered users over time by the HAPS while minimizing the interference that can be caused by two similar HAPS serving the same users. Let Cht be the number of users assigned to HAPS h and let I t be the total number of connected PH users. The coverage reward is rt = h=1 Cht − κI t , where κ = 0.5. Observation and action spaces. Let Ut be the number of users present in the environment at time t; we have Ut+ = max{1, |Ut |}, and let Gth be the number of users geometrically covered by HAPS h before capacity-constrained assignment. We have xh , yh , zh as the coordinates of HAPS h. The physical observation is " # H t xth yht zht − zmin Cht Gth mt I st = , , , , , λ̄ , + , (73) A A zmax − zmin Ut+ Ut+ h=1 Ut where λ̄mt is the mean arrival rate over the 25 cells. For H = 2, this physical vector has dimension 12. The phase-conditioned policy receives [st , emt ], where emt ∈ {0, 1}M is one-hot. Clock variants append [sin(2πτt /T ), cos(2πτt /T )]. The joint action space is {−1, 0, 1}H and controls all HAPS centrally. Initialization and method-specific training. At the start of a full episode, the user set is empty, horizontal HAPS coordinates are sampled uniformly over the service area, and every altitude is set to 19 km. Shared and multihead policies are trained on complete cyclic episodes, so their rollouts preserve HAPS positions and outstanding user sessions across every phase boundary. Standard multipolicy training instead uses one phase-fixed environment per specialist. Each specialist episode starts at its assigned phase with a fresh user set and lasts exactly T steps. The specialists are updated in round-robin order, and each receives approximately B/M of a total interaction budget B. All methods are nevertheless evaluated on the same complete cyclic process: at each boundary the evaluator selects the next specialist without resetting the

25

environment. Handoff-aware variants replace the phase-fixed reset law with entry states collected from these full ensemble rollouts; we identify those variants explicitly when reported. The main phase-count sweep uses M ∈ {2, 5, 10, 20} at T = 500. The dwell-time sweep fixes M = 20 and varies T . Training budgets denote total environment interactions, including interactions collected across the eight environments. The main phase-count results and the dwell-time sweep T ∈ {100, 200, 500, 2000} use seeds {0, 1, 2, 3, 4}; secondary ablations use seeds {0, 1, 2} unless stated otherwise. Raw HAPS evaluations use three complete cyclic episodes per checkpoint. Because each episode preserves the state across boundaries, these scores include both settled phase behavior and handoff effects. A.3.3

PhasePendulum Environment and Protocol

Dynamics, observation, and reward. PhasePendulum environment consists of the standard torquecontrolled pendulum dynamics1 while changing the target angle gm between phases. Its physical state is (θt , θ̇t ), its action is a scalar torque ut ∈ [−2, 2], and its integration step is ∆t = 0.05. We use g = 10, mass 1, length 1, and clip angular velocity to [−8, 8]. The physical observation is st = [cos θt , sin θt , θ̇t /8]. Writing ang(x) = ((x + π) mod 2π) − π, the reward in phase m is rt = − ang(θt − gm )2 − 0.1θ̇t2 − 0.001u2t .

(74)

Only the target, and therefore the reward, changes at a boundary; angle and angular velocity remain continuous. At a fresh episode reset, θ0 ∼ Unif[−π, π] and θ̇0 ∼ Unif[−1, 1]. Target structure and budgets. Each of the M phases has its own target angle, and the phases repeat periodically in an increasing index order. We control the heterogeneity between phases through the choice of successive targets. The maximum-jump structure greedily orders candidate target angles so as to maximize the circular distance between consecutive targets. For the continuous heterogeneity sweep, we construct one max such structure g max and set gm (h) = hgm for h ∈ {0, 0.25, 0.5, 0.75, 1}. At h = 0 every phase has the same target; at h = 1 the complete maximum-jump structure is recovered. We fix M = 10, T = 1000, five training seeds, four evaluation cycles, 5M total interactions, and 20 deterministic raw-evaluation episodes per checkpoint. Each specialist therefore receives 0.5M interactions. Scaling targets also changes their absolute locations relative to the pendulum dynamics, so we compare architectures within each h and do not interpret rewards across h as a pure causal effect of heterogeneity. The phase-count sweep uses M ∈ {2, 5, 10, 20}, T = 1000, the maximum-jump suite, five seeds, and a fixed allocation B0 = 1M interactions per phase. Its total budget is therefore M B0 . This sweep tests scalability while holding specialist data approximately constant, not under a fixed total budget. The identical-phase budget study fixes M = 10, T = 1000, h = 0 and varies the specialist allocation from 0.5M to 2M interactions. Shared and multihead policies train on full cyclic trajectories; specialists train from the fresh phase-local reset law above. All are evaluated for four complete cycles without resetting (θ, θ̇) at boundaries. max phase targets gm (h) = hgm

gm → gm+1 (θ, θ̇) is carried

phase m

phase m + 1 time

h=0

h = 0.5

h=1

Figure 4: PhasePendulum construction. 1 https://gymnasium.farama.org/environments/classic_control/pendulum/

26

Fig.4 presents the objective angles for different scenarios. On the left of the figure, we have the examples for different heterogeneity h at M = 5 phases. At h = 0, all phase objectives coincide; increasing h spreads them around the circle. On the right we have a detail of the between-phase transition where only the target changes. The angle and angular velocity remain continuous, so the new phase inherits the state produced by the preceding policy. A.3.4

PhaseGridWorld Environment and Protocol

Base environment. The agent moves on a 9 × 9 grid using four deterministic actions: up, down, left, and right. Each phase has its own goal and obstacle map. The physical observation contains only the agent coordinates, each scaled to [−1, 1]. Shared methods append the phase code, clock variants append the sine–cosine clock, and multipolicy selects a specialist externally. Goal and obstacle coordinates are never observed directly. The per-step reward is rt = −0.01 − 0.05 1{collision} + 0.02(dt − dt+1 ) + 1{dt+1 = 0},

(75)

where dt is the Manhattan distance to the active goal. Reaching the goal resamples a reachable non-goal cell within the same phase. At a normal boundary, the agent position is carried into the next map. If the inherited cell is an obstacle in that map, a safety rule resamples the agent uniformly from free cells connected to the incoming goal. Thus, obstacle collisions are repaired, but a free inherited cell in a disconnected component is retained. The boundary-reset intervention instead performs this reachable-state resampling at every boundary. We call these two boundary laws carried and reset. The same law is used for every architecture within a comparison. Ordinary suites and distractors. Maximum-jump goal suites greedily maximize Euclidean separation between successive goals. Maximum-jump obstacle suites similarly select successive Bernoulli obstacle fields to maximize their normalized Hamming distance. The main ordinary runs use M = 10, T = 500, four cycles, obstacle densities 0.25 and 0.40, seeds {0, 1, 2}, and B0 = 1.5M interactions per phase. A separate experiment uses M = 10, T = 100, three seeds, and appends Z ∈ {0, 8, 32, 128} independent Unif[−1, 1] coordinates to every observation. These distractors do not affect transitions or rewards. Figure 5 presents examples of environments for the two densities.

27

Figure 5: Representative maps from the ordinary PhaseGridWorld maximum jump suite used in the reported experiments (9 × 9, M = 10, suite and obstacle seed 0). Columns show the same three phase indices at obstacle sampling probabilities D = 0.25 and D = 0.40. Obstacles are sampled independently before the maximum-jump ordering is selected, and the active goal cell is always cleared; consequently, the realized obstacle count need not equal exactly 81D. These panels illustrate environment structure rather than policy performance. Controlled boundary topologies. To study in detail the handoff between phases, we built the following environment. The aligned, bottleneck, and closed-trap experiments use M = 2, T = 100, five training seeds, 3M total interactions, and 20 evaluation cycles per episode. In the aligned case both phases use goal (7, 4). In the other two cases the goal changes from (1, 4) to (7, 4). Phase 1 adds the vertical wall x = 4; the bottleneck opens the single gate (4, 7), whereas the trap leaves the wall closed. For the boundary intervention, each trained checkpoint is evaluated under both carried and reachable-state-reset laws. The policy parameters are unchanged, so the contrast isolates the effect of the entry distribution. This specific experiment is illustrated in Fig. 6

28

(a) Aligned goal

(b) Recoverable bottleneck

(c) Closed trap

gate

s1

s1 g

entry and goal on the same side

phase-0 goal

s1

g0

g0

g1

long path, but every free entry is viable

phase-1 goal

carried state

g1

entry is free but cannot reach g1

wall

Figure 6: Controlled PhaseGridWorld boundary constructions at the start of phase 1. The 9 × 9 geometry, goals, wall, and bottleneck gate match the environment implementation. The orange marker is an illustrative carried entry state, not a sampled trajectory. In the aligned case, the goal remains at (7, 4). In the other cases, it switches from (1, 4) to (7, 4). Opening the gate at (4, 7) turns a disconnected trap into a recoverable, but long-path, bottleneck. Two-goal handoff choice. In addition to the above handoff study environment, we created a two-goal handoff environment based on the theory proposed in Appendix A.1.3 and illustrated by Fig. 7. The causal choice construction also uses M = 2, T = 100 and a closed phase-1 wall. Phase 0 contains a primary goal g00 = (1, 4) worth 1 and a secondary safe goal g01 = (7, 4) worth ρ ∈ {0.3, 0.6, 0.9}. Phase 1 uses g1 = (7, 6). Upon reaching either phase-0 goal, the agent remains there until the scheduled boundary. Hence g00 produces a free but unreachable phase-1 entry, while g01 preserves reachability. Distance shaping is disabled in this construction so that the comparison is driven by goal reward and continuation value. We train Single w/ clock, multihead w/ clock, and multipolicy w/ clock for 3M total interactions over five seeds and evaluate ten episodes per checkpoint. Scripted controls additionally enumerate all 79 non-goal initial cells and follow shortest paths to either g00 or g01 ; they involve no RL training. Phase 1 Entry state controls reachability

Phase 0 Immediate vs. safe handoff

g1 g00 : 1

g01 : ρ

boundary trapped

Figure 7: Controlled two-phase handoff environment. In phase 0, taking the larger local reward g00 places the agent in a component that becomes disconnected from g1 after the boundary. Taking the lower reward g01 preserves phase-1 reachability.

29

A.3.5

Evaluation Metrics and Protocol

Evaluations report unweighted cross-seed mean raw returns at the final training checkpoint, with matchedseed differences and paired 95% percentile bootstrap confidence intervals. Diagnostic gradient evaluations (Appendix A.1.5) compute phase-local PPO surrogates using 1024 environment interactions per phase with global advantage normalization. Sample budgets strictly denote environment interaction steps rather than minibatch updates.

A.4

Extended Empirical Results and Diagnostics

This section contains the longer empirical comparisons that are summarized in the main text. We keep confirmatory results separate from qualitative checks. All numerical comparisons use the evaluation and uncertainty conventions in Appendix A.3. A.4.1

HAPS Experiments Results

HAPS Main results In HAPS, transient dynamics are governed by platform relocation and user persistence across phase boundaries. An HAPS platform moves 3 km per step per coordinate in a 300 km × 300 km operational area, meaning direct transit requires at most 100 steps when both spatial coordinates change simultaneously. Meanwhile, mean user connection lifetimes range from 5 to 25 steps. We then take Lm = 100. Moreover, reward differences are mostly caused when one or two HAPS do not reach a high user density zone. Then any difference between two policies should be in the same order. Following this observation, we propose the following approximation: Gm ≈ Km , transforming (9) into λm < 12 . And for γ close to one, we m have λm ≈ L Tm ; this motivates T = 200 as a diagnostic transient recovery horizon from (9), though not as a strict value for Lm , since user arrivals are stochastic and learned navigation trajectories need not be straight lines. At fixed M = 20, varying the dwell time T fundamentally shifts method rankings (Figure 2, right). At short dwell times (T = 100), the Single+clock shared architecture leads (31.75 vs. 20.88 for Multipolicy). At T = 200, Multihead and Multipolicy perform comparably (32.26 and 31.90, respectively) with no consistent seed-level winner, slightly outperforming Single+clock. As dwell time increases further, Multipolicy establishes a decisive lead at T = 500 (42.36 vs. 34.15 for Multihead) and T = 2000 (45.76 vs. 33.16). In addition to the three core architectures, we evaluated multi-task optimization baselines, extended sample budgets, and policy distillation methods on HAPS at M = 20 and T = 500. As summarized in Table 5, several key insights emerge: Budget scaling and sample efficiency. Increasing the shared-policy training budget up to 20M steps fails to close the high-M performance gap relative to phase specialists. Specialists outperform the shared baseline even at the smallest evaluated per-phase budget (B0 = 250k steps, corresponding to 5M total environment interactions) and show only modest gains with additional samples. We observe no late-budget crossover: the advantage of parameter isolation significantly outweighs the data-efficiency cost of splitting sample allocations across specialists. Multi-task gradient surgery. Gradient surgery methods—specifically PCGrad [18] and CAGrad [19] improve upon the standard single-policy baseline at 5M steps (28.87 and 28.05 vs. 25.54). However, both methods remain substantially below the multi-policy score (43.33), indicating that gradient conflict mitigation alone cannot resolve the representation bottleneck in high-phase setups. Asymmetry in policy distillation. Policy distillation exhibits a stark structural asymmetry: while compressing a specialist ensemble into a single shared student fails, using a shared teacher to regularize specialist training preserves full performance while guiding early learning. Specifically, under Multipolicy+clock + teacher KL, a frozen shared policy pretrained for 1M steps serves as a global teacher πteacher (a | s, m, τ ). Each fresh phase specialist πm (a | s, τ ) is trained on its phasefixed environment with PPO, supplemented by an alternating KL-divergence update against the teacher’s action distribution. The shared teacher guides early exploration without constraining the specialist’s final

30

Table 5: Performance summary on HAPS (M = 20, T = 500). Values denote mean raw reward per step averaged across random seeds {0, 1, 2}. Bold indicates the highest score. Budget / Setup

Method

Reward/step

Single+clock / Multihead / Multipolicy+clock Single+clock / Multihead / Multipolicy+clock Single+clock / Multihead / Multipolicy+clock Multipolicy without clock

26.16 / 33.52 / 43.33 31.16 / 35.41 / 46.39 30.10 / 37.37 / 46.72 46.57

5M gradient surgery 5M gradient surgery 5M gradient surgery

Single+clock PCGrad-style CAGrad-style

25.54 28.87 28.05

20M distillation 20M distillation 20M larger student

Multipolicy+clock + teacher KL Single+clock distilled from specialists Big single+clock distilled (5M student steps)

46.58 12.95 27.73

5M budget 10M budget 20M budget 20M budget

Table 6: HAPS capacity and depth controls at M = 20, T = 500. Values denote mean reward per step across three seeds. “Big” doubles hidden width; “Deep” uses four hidden layers. The six- and eight-layer entries denote depth-stress runs. Architecture Single, default Single, big Single, deep Single+clock, 6 layers Multihead, default Multipolicy

Reward/step

Architecture

26.04 22.48 24.73 29.04 32.76 42.34

Single+clock, default Single+clock, big Single+clock, deep Single+clock, 8 layers Multihead, deep

Reward/step 27.06 24.27 29.51 23.30 27.47

capacity. Reallocating 1M steps to teacher pretraining leaves ≈ 950k steps per specialist, strictly preserving the 20M total interaction budget. This regularized ensemble achieves 46.58, matching the standard unconstrained specialist score (46.72). In contrast, distilling the 20M-step specialist ensemble back into a single shared student yields only 12.95. Even when scaling the student architecture and allocating 5M dedicated student steps, performance reaches only 27.73—remaining below both the un-distilled shared baseline (30.10) and the specialist teachers (46.72). This demonstrates that the representation bottleneck of a single shared network cannot be overcome simply by imitating optimal specialist behaviors post hoc. Capacity and depth. Table 6 summarizes the capacity control experiments supporting Hypothesis H2 of Sec. 5. Increasing the width (“Big”) or depth (“Deep”) of the shared policy architecture fails to close the performance gap relative to the specialist ensemble at M = 20 and T = 500. While these results rule out parameter count of the default shared model as a simple explanatory factor, we note that this is not a strict equal-compute comparison: specialist ensembles store significantly more parameters overall, and deeper PPO networks are notoriously prone to optimization instability. Indeed, additional six- and eightlayer depth-stress runs exhibited severe optimization instability across seeds (29.04 and 23.30, respectively) rather than approaching specialist performance. Phase ordering and handoff variants. To isolate the effect of temporal regularity versus transition difficulty, we evaluated three distinct phase ordering schedules at M = 20, T = 500 (Table 7). Navigating adjacent phases proves easiest across all methods. Conversely, random and maximum-jump transitions degrade overall returns while widening the performance advantage of the multi-policy ensemble over multi-head baselines. Because altering phase order shifts both boundary state displacement and temporal regularity, this sensitivity highlights that phase count alone does not fully capture domain difficulty. An exploratory M = 30, T = 500 extension yields mean rewards of 23.80, 32.52, and 37.63 for Single+clock, Multihead, and

31

Table 7: Exploratory HAPS phase-order and handoff controls (means over three seeds). The phase-order sweep uses M = 20, T = 500 under the original sample budget. The handoff sweep uses M = 20 under a 20M total interaction budget (1M per phase). Control

Setting

Phase order Phase order Phase order

Adjacent Random Maximum jump

Clock/handoff, T = 200 Clock/handoff, T = 200 Clock/handoff, T = 500 Clock/handoff, T = 500

Standard / clock-aware specialist Handoff / handoff+warm start Standard / clock-aware specialist Handoff / handoff+warm start

Single+clock

Multihead

Multipolicy

34.58 27.47 31.17

46.80 38.17 34.88

48.41 40.59 41.06

– – – –

– – – –

37.47 / 38.50 31.75 / 27.14 46.35 / 46.75 36.69 / 27.29

Table 8: Exploratory HAPS warm-start controls with B0 = 1M interactions per phase under equal total budgets (M B0 ). Values denote mean reward per step across seeds {0, 1, 2}. “Warm” denotes budget-matched pretraining followed by specialist fine-tuning. M

T

Multipolicy

Warm

Handoff

Handoff+warm

10 10 20 20

500 1000 500 1000

44.84 48.21 46.37 48.60

16.82 14.50 13.44 11.14

32.91 39.55 37.14 41.69

17.67 14.13 11.27 9.92

Multipolicy, respectively, preserving the relative performance ordering. Our preliminary specialist-coordination mechanisms do not resolve the boundary handoff problem. Under a 20M-step sample budget, appending a within-phase clock to standard multi-policy leaves performance essentially unchanged, whereas explicit handoff and handoff-warm-start objectives degrade performance across both tested dwell times (T ∈ {200, 500}). These negative results demonstrate that naive handoff regularizers can disrupt phase-local learning, highlighting the need for principled continuation-value coordination. Similarly, a budget-matched warm-start protocol proved unsuccessful. In this setup, a non-augmented policy is pretrained for B0 interactions, after which its weights are cloned into all M specialists for fine-tuning over B0 (M − 1)/M steps (maintaining a total interaction budget of M B0 ). As shown in Table 8, shared initialization and its handoff-aware extension significantly reduce performance across all tested settings (M ∈ {10, 20}, T ∈ {500, 1000}). This indicates that initializing specialists with shared parameters introduces negative transfer during fine-tuning, rather than serving as a beneficial warm start. Table 9 reports the actor-gradient diagnostic measurements based on appendix A.1.5. Across all phase counts M , the aggregate alignment statistic Ag remains close to its orthogonal baseline (Ag = 1), with every five-seed 95% confidence interval containing unity. Furthermore, the aggregate update harms at most 2% of phases (fharm ≤ 0.02). Consequently, the declining phase-to-update cosine similarity reflects gradient dilution across high-dimensional, distinct task directions rather than an aggregate update vector that actively − opposes individual phase objectives. Although 44–60% of pairwise gradient cosines are negative (fpair ≈ 0.5), their mean (cos(gm , gk )) remains close to zero, as positive cross-terms effectively offset negative ones. These findings demonstrate that the performance gap in high-M shared architectures cannot be attributed to destructive aggregate gradient interference at the evaluated training checkpoints. A.4.2

PhasePendulum Experiments Results

PhasePendulum results confirm that the relative advantage of architectural specialization depends on task heterogeneity and sample availability. We control target separation by interpolating phase targets as gm (h) = max h gm , where h ∈ [0, 1] parameterizes phase divergence.

32

Table 9: Corrected HAPS actor-gradient diagnostic at T = 500. Values denote means over five seeds. Ag = 1 corresponds to the mutually orthogonal baseline (Cg = 1 − Ag ), and fharm is the fraction of phases where ⟨gm , g⟩ < 0. Remaining columns report the mean phase-to-aggregate cosine, mean off-diagonal pairwise − cosine, and the fraction of negative phase pairs (fpair ). M

Architecture

Ag

Cg

fharm

cos(gm , g)

cos(gm , gk )

− fpair

2 5 10 20

Single+clock Single+clock Single+clock Single+clock

0.950 0.970 0.968 1.046

0.050 0.030 0.032 -0.046

0.000 0.000 0.000 0.020

0.675 0.399 0.264 0.195

-0.053 -0.012 0.000 0.002

0.600 0.520 0.462 0.485

2 5 10 20

Multihead Multihead Multihead Multihead

1.003 1.017 1.041 1.074

-0.003 -0.017 -0.041 -0.074

0.000 0.000 0.020 0.000

0.706 0.416 0.279 0.204

0.003 0.006 0.007 0.002

0.600 0.440 0.462 0.468

Table 10: PhasePendulum target-separation sweep (M = 10, T = 1000). Values denote mean reward per step across five seeds. h

Single+clock

Multihead

Multipolicy

0.00 0.25 0.50 0.75 1.00

-1.370 -2.185 -3.977 -2.666 -1.685

-2.785 -2.592 -3.844 -2.693 -1.638

-6.007 -5.107 -3.977 -2.614 -1.596

First Heterogeneity Test A first experiment is run with M = 10 phases of duration T = 1000 timesteps. The different results are presented in Tab.10. At h = 0 (identical phase targets), full parameter sharing via Single+clock solves the task efficiently in four of five seeds, whereas all ten specialists remain severely undertrained due to data fragmentation. At h = 0.5, shared and specialized architectures achieve comparable returns. At high heterogeneity (h ∈ {0.75, 1.00}), Multipolicy outperforms Multihead and Single+Clock across all five matched seeds. Its advantage over complete sharing, while positive in the majority of seeds, exhibits greater variance. This progression reflects a fundamental regime transition from data pooling to specialized parameter allocation as task divergence increases. Budget allocation and entropy controls. To determine whether the failure of multipolicy at h = 0 stems from fundamental optimization barriers or insufficient sample allocation, we performed budget and entropy sweeps (Fig. 10). Extending total interactions from B = 5M to B = 20M (2M per specialist) fully resolves the undertraining deficit, allowing multipolicy to match the shared baseline. Crucially, learning curves show that rapid performance recovery occurs after surpassing the default 500k per-specialist threshold. Conversely, increasing the PPO entropy coefficient (cent = 0.05) under B = 20M causes policy action variance to explode without restoring performance, confirming that the initial performance gap is a sample-allocation bottleneck rather than premature policy collapse. This is also visible in Fig.11 where we see that each local policy/specialist from multipolicy performs well under a single shared policy. The problem seems more linked to learning budget than to nonexistent coordination between phases. Phase count, dwell time, and reward weighting dynamics. Under a fixed 1M per-phase interaction allocation (T = 1000), multipolicy outperforms Single+clock in 19 out of 20 matched runs across M ∈ {2, 5, 10, 20}, yielding mean advantage gains of +0.136, +0.119, +0.135, and +0.242 reward per step. Paired bootstrap percentile intervals remain strictly positive across all M . However, performance scaling is nonmonotone with respect to M , at M = 20, multihead achieves a slightly higher mean score (−2.236 vs. −2.260). Tab. 11 summarizes phase-count, dwell-time, and target/reward weighting sweeps. In the dwell-time

33

Table 11: PhasePendulum ablation controls across phase counts (M ), dwell times (T ), and target/reward structures. The phase-count and dwell-time sweeps allocate 106 interactions per phase; target/weighting sweeps allocate 500k per phase. n denotes random seed count. Control

Setting

n

Single+clock

Multihead

Multipolicy

Phase count, T = 1000

M =2 M =5 M = 10 M = 20

5 5 5 5

-1.888 -1.633 -1.688 -2.502

-1.917 -1.646 -1.640 -2.236

-1.752 -1.514 -1.553 -2.260

Dwell time, M = 10

T = 500 T = 1000 T = 2000

3 3 3

-1.567 -1.637 -1.766

-1.660 -1.639 -1.697

-1.572 -1.564 -1.548

Dwell, B = 5M

M = 5, T = 100 M = 5, T = 200 M = 5, T = 500 M = 10, T = 100 M = 10, T = 200 M = 10, T = 500

3 3 3 3 3 3

-1.580 -1.545 -1.567 -1.586 -1.584 -1.645

-2.021 -1.922 -1.706 -1.885 -1.816 -1.624

-1.672 -1.636 -1.530 -1.779 -1.613 -1.586

High dwell, B0 = 1M

M = 10, T = 500 M = 10, T = 1000 M = 10, T = 2000 M = 20, T = 500 M = 20, T = 1000 M = 20, T = 2000

3 3 3 3 3 3

-1.567 -1.637 -1.766 -2.421 -2.461 -2.570

-1.660 -1.639 -1.697 -1.844 -2.136 -2.585

-1.572 -1.564 -1.548 -2.377 -2.310 -2.370

Targets / weights

Identical targets, uniform weights Small-jump targets, uniform weights Maximum-jump targets, uniform weights Maximum-jump targets, Phase 0 dominant

3 3 3 3

-1.631 -0.360 -1.601 -2.030

-2.419 -2.625 -1.648 -2.019

-6.174 -5.185 -1.584 -2.330

sweep at M = 10, Multipolicy matches complete sharing at T = 500 and secures the top score for T ∈ {1000, 2000}. At M = 20, Multihead dominates short-to-medium horizons (T ∈ {500, 1000}), whereas Multipolicy leads at T = 2000. We then vary target spacing and reward weighting (M = 10, T = 1000, 500k/phase) to further illustrate these regime boundaries. The results are presented at the end of Tab. 11. Concentrated or small-jump target configurations heavily favor shared representations, whereas maximum-jump configurations favor parameter separation (multipolicy). Assigning a dominant reward weight to Phase 0 eliminates the specialist advantage. This occurs because highly non-uniform rewards force the shared network to prioritize the high-value phase, relieving the capacity bottleneck that normally arises when balancing multiple competing task objectives. A.4.3

GridWorld Experiment Results

One-phase budget calibration. To establish a rigorous sample-efficiency baseline for phase specialists, we first calibrated the required interaction budget in a single-phase environment (M = 1, T = 100). Because single-phase environments offer no benefit from parameter specialization, this experiment isolates the sample complexity required by the underlying PPO learner to cross a predefined performance threshold. As reported in Table 12, Single+clock achieves threshold performance across all three seeds at 500k interactions, whereas the multi-policy architecture requires 750k interactions. Based on this calibration, all subsequent multi-phase PhaseGridWorld experiments allocate a conservative minimum budget of 1M interactions per specialist (B0 ≥ 1M) to prevent sample starvation. At first glance, it is surprising that the multipolicy condition learns more slowly than the shared policy when M = 1, since both use a single PPO learner. However, the comparison is not fully matched: the shared policy receives within-phase clock information, whereas the standard multipolicy condition does not. The clock does not introduce randomness directly, but it allows the policy to select different actions at the same physical state at different phase ages. This can help the agent escape repetitive behaviors, such as repeatedly attempting to move into a wall, and may partly explain the higher final performance of the clock-conditioned policy. Table 13 supports this interpretation: without distractors, adding the clock improves multipolicy performance relative to the standard multipolicy condition.

34

Table 12: Single-phase GridWorld budget calibration (M = 1, T = 100, three seeds). “Reached” denotes the proportion of seeds exceeding the performance threshold. Values are reported as mean reward per step. Budget

Method

Reached

Best mean

Final mean

50k 50k 100k 100k 200k 200k 300k 300k 500k 500k 750k 750k 1M 1M

Multipolicy-style Single+clock Multipolicy-style Single+clock Multipolicy-style Single+clock Multipolicy-style Single+clock Multipolicy-style Single+clock Multipolicy-style Single+clock Multipolicy-style Single+clock

0/3 0/3 0/3 0/3 0/3 0/3 0/3 1/3 1/3 3/3 3/3 3/3 3/3 3/3

−0.0400 −0.0410 −0.0417 −0.0420 −0.0375 −0.0461 −0.0310 −0.0003 −0.0040 0.1223 0.1452 0.1745 0.1350 0.1807

−0.0585 −0.0567 −0.0557 −0.0557 −0.0525 −0.0502 −0.0438 −0.0048 −0.0386 0.1095 0.0791 0.1547 0.0208 0.1683

Table 13: PhaseGridWorld distractor sweep (M = 10, T = 100). Values denote mean reward per step across three seeds under Z reward-irrelevant noise dimensions (higher is better). Z

Single+clock

Multihead

Multipolicy

Multi+clock

Warm Multi+clock

0 8 32 128

0.1422 0.1427 0.1359 0.0629

0.0980 0.1197 0.1197 0.0968

0.1210 0.1048 0.0836 0.0544

0.1258 0.1201 0.1082 0.0732

0.1356 0.1319 0.1312 0.1178

Robustness to distractor dimensions. In order to evaluate how representation space capacity affects phase routing, we conducted a distractor sweep at M = 10, T = 100 by appending Z task-irrelevant, independent observation dimensions. As shown in Table 13, Single+clock maintains high performance up to Z = 32, but degrades sharply at Z = 128 (0.0629). Conversely, Multihead routing and warm-started clock-aware specialists demonstrate superior representation robustness at Z = 128, achieving 0.0968 and 0.1178, respectively. This drop in shared policy performance is consistent with phase-signal neglect under high-dimensional noise, though distractors also increase the baseline representation burden across all modular models. Obstacle Density The fixed-total-budget density sweep in Appendix Table 14 provides a complementary control. At D = 0.40, Single+clock leads in every tested (M, T ) setting when the total budget is only 3M interactions. By contrast, clock-aware specialization is best at D = 0.40 once each specialist receives 1.5M interactions. Obstacle density is therefore not an architecture-selection criterion by itself: its effect depends jointly on phase-local data, clock information, and the topology induced by the sampled walls.

35

Table 15: PhaseGridWorld specialist controls at T = 500. Values denote mean reward per step across three seeds. B0 is the nominal interaction budget per phase. “Warm+clock” initializes specialists from a shared pretrained model prior to fine-tuning. B0 1M 1M 1M 1M 1.5M 1.5M

Density M Single+clock Multihead Multipolicy Multi+clock Warm+clock Transition warm 0.25 0.25 0.40 0.40 0.25 0.40

5 10 5 10 10 10

0.0836 0.0878 0.1508 0.1480 0.0991 0.1486

0.0514 0.1228 0.2099 0.1344 0.0826 0.1404

0.0200 0.0916 0.1924 0.1772 0.0791 0.1606

0.0940 0.0926 0.2372 0.1651 0.0992 0.1797

0.1124 0.1179 0.2323 0.1681 0.1155 0.1695

– – – – -0.0164 0.0539

Table 14: Exploratory PhaseGridWorld obstacle-density sweep under a fixed 3M-interaction total budget. Entries are mean reward per step over three matched seeds; higher is better. The specialist budget is approximately 3M/M . Multipolicy and warm multipolicy do not receive clock features in this earlier sweep. Density

M

T

Single+clock

Multihead

Multipolicy

Warm multipolicy

0.10 0.10 0.10 0.10

5 5 10 10

50 100 50 100

0.1169 0.1273 0.0861 0.0985

0.1252 0.1192 0.0991 0.1031

0.0917 0.0633 0.0798 0.0445

0.1253 0.1331 0.0969 0.1272

0.25 0.25 0.25 0.25

5 5 10 10

50 100 50 100

0.1008 0.0901 0.1275 0.0750

0.0704 0.0443 0.0571 0.0686

0.0653 0.0261 0.0375 0.0326

0.0992 0.1009 0.0960 0.0813

0.40 0.40 0.40 0.40

5 5 10 10

50 100 50 100

0.1162 0.1069 0.1254 0.0887

0.0730 0.0440 0.0788 0.0552

0.0502 0.0490 0.0065 -0.0121

0.0629 0.0533 -0.0140 0.0113

Specialist controls and sample allocation. Table 15 summarizes extended per-phase budget runs (B0 ∈ {1M, 1.5M}) across varying grid densities and phase counts. At B0 = 1.5M and M = 10, Multi+clock matches complete sharing at grid density 0.25 (0.0992 vs. 0.0991) and outperforms it at density 0.40 (0.1797 vs. 0.1486). Warm initialization (“Warm+clock”) provides gains at low density (0.25), whereas direct transition-aware warm starts underperform across all settings. These controls indicate that while clock availability and parameter initialization improve specialist performance, sample starvation alone does not fully explain modular failure modes in multi-phase grid worlds. Two-phase coordination failure and the handoff trap. Because independent specialists optimize phase-local returns from phase-local reset states, they receive no training signal regarding downstream continuation values. In contrast, a trajectory-level shared policy can trade off immediate phase rewards to optimize downstream start states. Fig. 7 formalizes this trade-off in a controlled two-phase GridWorld. In Phase 0, the agent chooses between an immediate goal (g00 , reward 1.0) and a safe handoff goal (g01 , reward ρ < 1.0). Reaching g00 leaves the agent in a disconnected component in Phase 1 behind a closed wall, rendering the Phase 1 goal unreachable. Scripted controls verify that choosing g01 preserves Phase 1 reachability across all 79 valid start states, improving net return by ρ. As presented in Figure 12. When evaluating learned policies, shared architectures adapt to downstream value trade-offs as ρ increases: at ρ = 0.9, Single+clock and Multihead choose the safe handoff in 96% and 98% of episodes, respectively. In contrast, independent specialists (Multipolicy+clock) select the safe handoff in only 10% of episodes, continuing to greedily pursue the larger Phase 0 local reward g00 . This confirms that uncoordinated specialist ensembles suffer from myopic action selection at phase boundaries.

36

Table 16: Empirical transient–late-window decomposition in PhaseGridWorld (L = 50, T = 100). Values denote means across five matched training seeds evaluated under ∆J¯ = (1 − λ)G − λK with λ = 0.5. “Carried” retains physical states across boundaries; “Reset” uses a common reachable entry law. Topology

Entry law

Aligned Aligned Bottleneck Bottleneck Closed trap Closed trap

Carried Reset Carried Reset Carried Reset

Safe choice rate

1

G (Settled)

K (Transient)

Specialist − Shared

−0.0375 0.0065 0.0746 0.0771 −0.0754 0.0101

0.0450 −0.0007 −0.0616 −0.0684 0.0842 −0.0018

−0.0413 0.0036 0.0681 0.0728 −0.0798 0.0060

Single+clock Multihead+clock Multipolicy+clock

0.5

0 0.3

0.6

0.9

Safe-goal reward ρ

Figure 12: Learned safe-handoff choices. Shared methods increasingly choose the safe phase-0 goal as ρ rises, whereas independently trained specialists continue to prefer the immediate phase-local reward. Means and two-sided 95% Student-t intervals over five seeds. Causal boundary interventions and recovery dynamics. To isolate boundary state distribution shifts from intrinsic phase-local policy quality, we evaluated trained checkpoints under two distinct phase-entry conditions: physical state carryover versus resampling from reachable entry states (Fig. 6). Fig. 13, presents the reward gap between multipolicy and single policy in the two cases when the transition is carried on from one phase to the other or when there is a reachable state reset. Those comparisons are presented for the three already presented phase-structured gridworlds (Aligned, Bottleneck, Closed trap). In the irrecoverable “Closed trap” setting, state carryover imposes a severe performance penalty on specialists (a gap of −0.0798 relative to shared policies). However, resetting phase entry to a common reachable state eliminates the handoff penalty, reversing the net performance gap to +0.0060. Conversely, in the recoverable “Bottleneck” environment (where a distant gate permits recovery), specialization outperforms sharing under both physical carryover (+0.0681) and reachable resets (+0.0728). These interventions validate the regime decomposition framework in (4): recoverable transitions induce a finite transient recovery cost (Lm ), enabling settled specialist gains (Gm ) to dominate long-horizon returns. Unrecoverable transitions cause effective transient costs to span the entire phase duration (Lm = Tm ), preventing boundary cost amortization. It also supports the effect of heterogeneity, as multipolicy, as a specialized solution, outperforms a single policy in “Bottleneck,” composed of two different phases, but underperforms in “Aligned,” where there is no difference between the phases. Empirical transient–settled decomposition. Finally, Table 16 reports the empirical decomposition into settled gains (G) and transient handoff costs (K) under a preregistered window split L = 50 (T = 100). Under state carryover, both the Aligned and Closed Trap topologies exhibit positive transient costs (K > 0) and negative late-window specialist gaps (G < 0). Re-evaluating policies under a common reachable reset eliminates transient costs (K ≈ 0) and restores positive net gains (G > 0).

37

Figure 8: Matched HAPS behavior within-phase at age 300 for five evenly spaced phases. Rows are architectures and columns are phases. The phase subset {0, 4, 8} was fixed by index rather than selected by performance. All methods see the same demand realization. This single-seed visualization makes phase-wise adaptation directly comparable, while Figure 2 reports the confirmatory five-seed returns.

38

(a) M = 2, Single+clock

(b) M = 20, Single+clock

Figure 9: Representative single-seed actor-gradient cosine matrices for HAPS, illustrating qualitative phase geometry. A high fraction of negative pairwise cosines naturally arises in near-orthogonal high-dimensional gradient spaces and does not indicate that the aggregate update harms individual phases. Quantitative evaluations across five seeds are reported in Table 9. 104

−5

−10

Action std. dev.

Reward/step

0

B = 5M B = 10M B = 20M 0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

cent = 0.01 cent = 0.05 102

100 0

0.5

1

1.5

2

Interactions per specialist (millions)

Interactions per specialist (millions)

(a) Sample budget allocation (h = 0)

(b) Entropy regularization (B = 20M)

Figure 10: Budget scaling and entropy controls for Multipolicy in PhasePendulum under identical phase targets (h = 0). (a) Learning curves across three total interaction budgets (B). Error bars denote twosided 95% Student-t confidence intervals over five seeds. Significant gains emerge after surpassing 500k interactions per specialist. (b) Under B = 20M, increasing the entropy coefficient (cent = 0.05) destabilizes action variance without improving returns (note log scale on right axis).

Reward/step

0

−5 Specialists Single+clock −10

0

1

2

3

4

5

6

7

8

9

Specialist phase m

Figure 11: Learning starvation in PhasePendulum with ten identical phases (h = 0). Final phase-wise returns at the original 500k allocation; every point is a mean over five seeds and the dashed line is the corresponding Single+clock mean. Error bars are two-sided 95% Student-t intervals. Doubling the local budget produces the largest recovery, while the low-budget failure is spread across all ten otherwise redundant specialists.

39

Reachable-state reset Reset − carried gap

Specialist − shared reward/step

Carried state 0.1

0

−0.1 Aligned

Bottleneck

Closed trap

(a) Reward gap across entry laws

0.1

5 · 10−2

0

ned Alig

eck

tlen Bot

p Tra

(b) Difference-in-differences

Figure 13: Controlled boundary intervention in PhaseGridWorld (M = 2, T = 100). (a) Specialist-minusshared returns under physical state carryover versus reachable-state resets. (b) Corresponding differencein-differences. Points represent means; error bars denote 95% Student-t CIs (left) and paired bootstrap CIs (right). Removing inherited boundary states reverses method rankings in the closed trap while preserving specialist dominance in the bottleneck.

40

Related documents

• Single or Multiple Policies for Phase-Structured Reinforcement Learning?#1188153
• From Policy Evaluation to Off-Policy Control: Provable Emergence of In-Context Q-Learning#1343251
• Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL#422305
• From Causal Graphs to Markovian States: Multi-Order State Exposure for Deep RL#1354900
• Q-Shaped Options for Hierarchical Reinforcement Learning#1249559