Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks

arXiv Preprint

Gradient–Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks Jing Xiao1,2,3 Xinhai Chen1,2,3,* Qinglin Wang1,2,3 Menghan Jia2,3 Zhiquan Lai2,3 Dongsheng Li2,3 Jie Liu1,2,3 Tiejun Li3

arXiv:2609.01558v1 [cs.LG] 1 Sep 2026

1 Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology, Changsha 410073, China 2 National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology, Changsha 410073, China 3 School of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China * Corresponding author: chenxinhai16@nudt.edu.cn

A BSTRACT Training Physics-Informed Neural Networks (PINNs) requires jointly optimizing physics residual and initial/boundary condition loss terms, which often induce conflicting gradients. Gradient surgery methods mitigate this issue by constructing directions from loss-specific gradients to reduce conflict before optimizer transformation. However, even when the constructed direction is conflict-free, this property may not be preserved after optimizer transformation. Let at denote the direction constructed by gradient surgery, ut the optimizer proposal, and Ct the conflict-free cone induced by the loss-specific gradients. We show that modern optimizers can transform at through mechanisms such as historical state, adaptive scaling, preconditioning, or decoupled weight decay, so at ∈ Ct does not generally imply ut ∈ Ct . We refer to this optimizer-induced discrepancy in conflict-freeness between at and ut as Gradient–Update Mismatch (GUM). Accordingly, we propose Gradient–Update Alignment (GUA), which projects ut onto Ct to obtain the aligned update pt and applies pt to the parameters. When the optimizer maintains internal state, GUA further adjusts this state toward targets reconstructed from the applied update. We conduct extensive experiments and find that GUM is widespread across momentum, adaptive, and curvature-based optimizers, with conflict rates reaching up to 86.3%. Across all PINN settings, GUA achieves conflict-free applied updates and consistently improves various gradient surgery methods, reducing the relative L2 error by up to 98.2% in individual settings. Data and code are available at https://github.com/JingXiao10/GUA.

1

I NTRODUCTION

Solving partial differential equations (PDEs) can be computationally expensive, especially for highdimensional problems or scenarios requiring repeated solution queries Li et al. (2020); Lu et al. (2021); Kovachki et al. (2023). Physics-Informed Neural Networks (PINNs) Raissi et al. (2019) provide a neural surrogate by incorporating physical constraints through loss terms associated with PDE residuals and initial and boundary conditions Gonon et al. (2024). However, the gradients induced by different loss terms can differ substantially in magnitude and direction, leading to gradient conflicts and making PINN training difficult Zhang et al. (2026). A common strategy for improving PINN training is loss balancing, which adjusts the weights of different loss terms to reduce imbalances in gradient magnitudes among physical constraints. These weights can be set manually or adapted during training. However, loss balancing does not directly address conflicts in gradient directions, and the choice of effective weights varies across problems Wang et al. (2021); Liu & Wang (2021); Li & Feng (2022); Bischof & Kraus (2025). This motivates the use of gradient surgery, which directly modifies loss-specific gradients to construct directions with reduced conflict. Such methods were originally developed for multi-task learning (MTL), including PCGrad Yu et al. (2020), CAGrad Liu et al. (2021a), IMTL-G Liu et al. (2021b), A-MTL Senushkin et al. (2023), UPGrad Quinton & Rey (2024), and ConFIG Liu et al. (2025). These methods have also been adopted in PINNs to mitigate conflicts among the gradients induced by different physical constraints Liu et al. (2025). 1

arXiv Preprint

Despite their methodological differences, these gradient surgery methods follow the same optimization pipeline: they first construct a direction from loss-specific gradients and then pass it to the optimizer. This naturally raises a key question: when gradient surgery constructs a conflict-free direction, does it remain conflict-free after optimizer transformation? Let at denote the direction constructed Gradient-Update Mismatch by gradient surgery, ut the optimizer Before Optimizer Optimizer After Optimizer proposal, and Ct the conflict-free cone Conflict-free Conflict-free induced by the loss-specific gradients at Historical Cone 𝑪𝒕 Cone 𝑪𝒕 state step t. We show that a conflict-free direction constructed by gradient surgery Adaptive 𝒈𝟏 𝒈𝟏 𝒂𝒕 scaling may not remain conflict-free after optimizer transformation, as illustrated in Preconditioning Figure 1. Formally, at ∈ Ct does not 𝒈𝟐 𝒈𝟐 generally imply ut ∈ Ct . Modern optiWeight 𝒖𝒕 mizers can transform at through mechdecay Optimizer proposal 𝒖𝒕 anisms such as historical state, adap- Constructed direction after transformation tive scaling, preconditioning, or decou- 𝒂𝒕 before optimizer pled weight decay Wang et al. (2025), which can move the optimizer proposal outside the conflict-free cone. We refer Figure 1: Gradient–Update Mismatch. A conflict-free dito this optimizer-induced discrepancy in rection at before optimizer transformation may become conflict-freeness between at and ut as a conflicting optimizer proposal ut after transformation. Gradient–Update Mismatch (GUM). This observation suggests that conflict handling should consider not only the direction at constructed before optimizer transformation, but also the optimizer proposal ut before it is applied to the parameters. We therefore propose Gradient–Update Alignment (GUA), which aligns the optimizer proposal ut with the current conflict-free cone Ct . Specifically, GUA projects ut onto Ct to obtain the aligned update pt , which is then applied to the parameters. When the optimizer maintains internal state, GUA further adjusts this state toward targets reconstructed from the applied update, reducing the influence of components removed by the projection on subsequent proposals. Our contributions are summarized as follows: 1. We identify and formalize a largely overlooked issue, which we term Gradient–Update Mismatch: the conflict-free direction at constructed by gradient surgery does not necessarily remain conflictfree after being transformed by the optimizer into the actual update proposal ut . 2. We characterize how optimizer transformations can induce GUM. We derive exact conflictpreservation conditions for fixed-state affine transformations and extend the analysis to general nonlinear optimizer maps, covering mechanisms such as historical state, adaptive scaling, preconditioning, and weight decay. 3. We propose Gradient–Update Alignment, introducing an update-level perspective on conflict handling. GUA considers conflict-freeness not only for the direction constructed before optimizer transformation, but also for the optimizer proposal before it is applied to the parameters. 4. We conduct extensive experiments demonstrating that GUM is widespread across momentum, adaptive, and curvature-based optimizers, and that GUA achieves conflict-free applied updates across all evaluated PINN settings while consistently improving various gradient surgery methods.

2

R ELATED W ORK

2.1

PINN T RAINING AND L OSS BALANCING

PINNs Raissi et al. (2019) are often difficult to train due to imbalanced gradient magnitudes, poor conditioning, and different convergence rates across loss terms Krishnapriyan et al. (2021); Wang et al. (2022). Loss-balancing methods address part of this difficulty by manually or adaptively reweighting physical constraints Wang et al. (2021); Liu & Wang (2021); Li & Feng (2022); Bischof & Kraus (2025). However, reweighting does not directly address conflicts in gradient directions, motivating gradient surgery methods that operate directly on loss-specific gradients. 2

arXiv Preprint

2.2

G RADIENT S URGERY IN M ULTI -L OSS O PTIMIZATION

Multi-loss PINN training is closely related to multi-objective optimization and gradient interference in multi-task learning, where different loss terms can induce conflicting gradients Sener & Koltun (2018); Chen et al. (2018; 2020). Gradient surgery methods address such conflicts by modifying loss-specific gradients before optimizer transformation. PCGrad Yu et al. (2020) reduces pairwise conflicts through projection, CAGrad Liu et al. (2021a) balances average descent with worst-case conflict, and IMTL-G Liu et al. (2021b) reduces task dominance by enforcing equal gradient projections. A-MTL Senushkin et al. (2023) improves gradient conditioning and alignment, while UPGrad Quinton & Rey (2024) uses dual-cone projection to obtain a common descent direction. ConFIG Liu et al. (2025); Baldan et al. (2026) further constructs conflict-free directions with nonnegative inner products against loss-specific gradients for PINNs. Despite differing criteria, these methods all perform gradient surgery before optimizer transformation, overlooking the role of the optimizer in shaping the actual parameter update. This observation naturally motivates our work. 2.3

O PTIMIZER T RANSFORMATIONS AND U PDATE -L EVEL A LIGNMENT

Modern optimizers generally transform the direction at constructed by gradient surgery before it is applied to the parameters. Such transformations may involve historical state, adaptive scaling, preconditioning, or decoupled weight decay. Representative examples include momentum SGD Sutskever et al. (2013), RMSProp Tieleman & Hinton (2012), Adam Kingma & Ba (2014), AdamW Loshchilov & Hutter (2017), and curvature- or second-order-inspired methods such as Sophia/SophiaG Liu et al. (2024), AdaHessian Yao et al. (2021), and SOAP Vyas et al. (2025). Although such optimizer transformations can improve certain aspects of gradient geometry during training Wang et al. (2025), they may not preserve the conflict-free geometry established before optimizer transformation. Our goal is not to design a new optimizer, but to extend conflict handling from the pre-optimizer direction to the optimizer proposal before it is applied to the parameters.

3

G RADIENT–U PDATE M ISMATCH AND G RADIENT–U PDATE A LIGNMENT

This section formalizes Gradient–Update Mismatch and introduces Gradient–Update Alignment for update-level conflict handling. 3.1

P RELIMINARIES : M ULTI -L OSS O PTIMIZATION

Pm Multi-loss objective. Consider an objective with m loss terms, L(θ) = i=1 Li (θ), where θ ∈ Rp denotes the model parameters and Li denotes the i-th loss term, including any associated weight. At optimization step t, the corresponding loss-specific gradient is gi,t = ∇θ Li (θt ). Conflict-free cone. For any candidate update direction d, the linearized change of the i-th loss under θt+1 = θt − ηd is Li (θt − ηd) − Li (θt ) = −η⟨gi,t , d⟩ + O(η 2 ). Therefore, d does not increase the i-th loss to first order if ⟨gi,t , d⟩ ≥ 0. The directions that are compatible with all loss-specific gradients form the conflict-free cone Ct = {d ∈ Rp | ⟨gi,t , d⟩ ≥ 0, i = 1, . . . , m} = {d ∈ Rp | G⊤ t d ≥ 0},

(1)

where Gt = [g1,t , . . . , gm,t ]. This cone characterizes the current conflict-free geometry and will be used for update-level diagnosis and projection. Constructed direction and optimizer proposal. A gradient surgery method constructs a direction at from the loss-specific gradients {gi,t }m i=1 to reduce gradient conflicts. Let Ot denote the optimizer transformation at step t, and let st denote its internal state, such as momentum or adaptive statistics. The optimizer transforms at into a proposal ut and a tentative next state s̄t+1 : (ut , s̄t+1 ) = Ot (at , st ).

(2)

The optimizer proposal ut , rather than the direction at constructed by the gradient surgery method, determines the direction of the actual parameter update. Consequently, conflict-freeness of at does not necessarily imply conflict-freeness of the actual update. We therefore introduce the notion of Gradient–Update Mismatch to characterize this optimizer-induced discrepancy in conflict-freeness. 3

arXiv Preprint

3.2

G RADIENT–U PDATE M ISMATCH

Definition 1 (Gradient–Update Mismatch). At step t, a Gradient–Update Mismatch (GUM) is an optimizer-induced discrepancy in conflict-freeness between the constructed direction at and the optimizer proposal ut . Specifically, GUM occurs when at ∈ Ct while ut ∈ / Ct . To quantify conflicts throughout the optimization pipeline, we record T

Rg =

1X I[∃ i < j : ⟨gi,t , gj,t ⟩ < 0] , T t=1 T

T

Ra =

1X I[at ∈ / Ct ], T t=1 T

1X Ru = I[ut ∈ / Ct ], T t=1

(3)

1X Rp = I[pt ∈ / Ct ]. T t=1

Here, T is the total number of optimization steps, and Rg , Ra , Ru , and Rp denote the conflict rates of the raw loss-specific gradients, constructed direction, optimizer proposal, and aligned update, respectively, yielding the diagnostic sequence Rg → Ra → Ru → Rp . For theoretical analysis, the above rates are defined using exact conflict conditions with a zero threshold. In experiments, we use a small normalized tolerance only in the conflict-rate diagnostics to account for numerical errors. Detailed specifications are given in Appendix B.6. Importantly, the relation between Ra and Ru depends on the guarantee provided by the gradient surgery method. If the method always produces conflict-free directions, then Ra = 0, and every conflicting optimizer proposal counted by Ru corresponds directly to a GUM event. Thus, under this conflict-free guarantee, any Ru > 0 provides direct evidence of GUM. Generalized GUM. Some gradient surgery methods do not guarantee conflict-free constructed directions, so Ra can be nonzero. In this case, the strict event definition in Definition 1 does not by itself summarize the aggregate change in conflict frequency introduced by optimizer transformation. We therefore use Ru > Ra as a rate-level diagnostic of generalized GUM, indicating that optimizer transformation introduces a net increase in the frequency of conflicting proposals. This is an aggregate diagnostic rather than an event-level definition. 3.3

C ONFLICT P RESERVATION UNDER O PTIMIZER T RANSFORMATIONS

GUM arises when optimizer transformation fails to preserve the conflict-free geometry of the constructed direction. This motivates us to characterize the conditions under which such geometry is preserved. To isolate the geometric mechanisms relevant to conflict preservation, we consider a fixed-state affine representation at step t, Tt (a) = Pt a + bt . Here, Pt captures scaling and preconditioning effects, while bt represents an additive term that is independent of the input direction a, such as contributions arising from historical state or decoupled weight decay. Proposition 1 (Conflict preservation under fixed-state affine transformations). Let Gt = [g1,t , . . . , gm,t ] and Ct = {d ∈ Rp | G⊤ t d ≥ 0}. The affine transformation Tt (a) = Pt a + bt preserves conflict-freeness on Ct , i.e., a ∈ Ct ⇒ Tt (a) ∈ Ct , if and only if bt ∈ Ct ,

Pt⊤ gi,t ∈ cone(g1,t , . . . , gm,t ),

i = 1, . . . , m.

(4)

The proof is provided in Appendix A.1. Within the fixed-state affine class, Proposition 1 exactly characterizes conflict preservation by separating the effects of the additive term bt and the linear transformation Pt . However, these preservation conditions need not hold in general. Positive-scalar transformations with Pt = αI, α > 0, and bt = 0 preserve the cone, with plain SGD corresponding to the special case Pt = I. In contrast, positive coordinate-wise scaling need not preserve the cone. For example, let g1,t = (1, 0)⊤ , g2,t = (1, 1)⊤ , and at = (1, −0.5)⊤ . Under Pt = diag(0.1, 10), the resulting proposal is ut = (0.1, −5)⊤ , for which ⟨g2,t , ut ⟩ = −4.9 < 0. Therefore, ut ∈ / Ct , even though at ∈ Ct , showing that positive coordinate-wise scaling can destroy conflict-freeness. Exact optimizer maps may also be nonlinear in at , as occurs with adaptive scaling. Appendix A.2 provides a general nonlinear characterization, with Proposition 1 recovered as the affine special case. Thus, historical state, adaptive scaling, preconditioning, and weight decay can all transform a conflict-free at into an optimizer proposal outside Ct . 4

arXiv Preprint

Gradient-Update Mismatch 𝒈𝟏

Gradient-Update Alignment

𝑪𝒕

𝒂𝒕

𝑪𝒕 Optimizer

Direction Alignment

𝒂𝒕 → 𝒖𝒕

𝒖𝒕 → 𝒑𝒕

𝑪𝒕

𝒖𝒕

𝒖𝒕 ∉ 𝑪𝒕

𝜽𝒕+𝟏 = 𝜽𝒕 − 𝜼𝒕 𝒑𝒕

State Alignment

(𝒑𝒕 , 𝒖𝒕 , 𝒔ത 𝒕+𝟏 ) ⟶ 𝒔𝒕+𝟏

𝒑𝒕

𝒈𝟐

𝒂𝒕 ∈ 𝑪𝒕

Parameter Update

𝒖𝒕 𝑴

𝒑𝒕 = 𝜫𝑪𝒕𝒕 (𝒖𝒕 )

Figure 2: Illustration of Gradient–Update Mismatch and Gradient–Update Alignment. GUA projects the optimizer proposal ut onto the conflict-free cone Ct to obtain the aligned update pt , applies pt to the parameters, and softly aligns the optimizer state with the applied update. 3.4

G RADIENT–U PDATE A LIGNMENT

Direction alignment. The preceding analysis suggests that conflict handling should extend from the constructed direction to the optimizer proposal, where conflicts directly determine the applied parameter update. Given ut , GUA projects it onto Ct under a positive-definite metric Mt ≻ 0: 1 pt = ΠCMt t (ut ) = arg min ∥d − ut ∥2Mt . d∈Ct 2

(5)

The projection is solved in the m-dimensional dual. For the PINN experiments, where m ∈ {2, 3}, we use exact active-set enumeration. For larger task counts, we solve the nonnegative dual quadratic program. We use the diagonal Adam as the projection metric. Solver and metric details are provided in Appendix B.7.1, with the metric comparison reported in Appendix C.1. The aligned update is applied as θt+1 = θt − ηt pt . By construction, pt ∈ Ct , and pt = ut whenever ut ∈ Ct . Thus, GUA leaves conflict-free proposals unchanged and projects conflicting proposals onto the conflict-free cone. To characterize this correction, define the one-step loss change ∆i,t (d; ηt ) = Li (θt − ηt d) − Li (θt ), and the conflict violation vt (d) = maxi [−⟨gi,t , d⟩]+ . Proposition 2 (Local safety and minimum-change projection). Suppose each loss Li is locally smooth near θt and ut ∈ / Ct . As ηt → 0+ , [maxi ∆i,t (pt ; ηt )]+ = O(ηt2 ) and maxi ∆i,t (ut ; ηt ) ≥ 2 ηt vt (ut ) − O(ηt ). Consequently, for all sufficiently small ηt > 0, maxi ∆i,t (pt ; ηt ) < maxi ∆i,t (ut ; ηt ). Moreover, pt is the closest conflict-free update to ut under the Mt -metric. For a conflicting optimizer proposal, ut incurs a positive first-order term for at least one loss, whereas pt does not. GUA therefore yields the closest conflict-free update to ut under the chosen metric. Full bounds and proofs are provided in Appendix A.3. Q3 tests the resulting direction alignment mechanism using same-state, norm-matched comparisons. State alignment. Direction alignment resolves conflicts in the current optimizer proposal by projecting it onto the conflict-free cone, while state alignment reduces the persistence of projectioninduced corrections in the optimizer memory. Without state adjustment, components removed by the projection may still affect future proposals through stored statistics. We therefore softly align the optimizer state toward targets reconstructed from the applied update pt only when the projection modifies the proposal. Otherwise, the tentative state is retained. Taking Adam as an example, let s̄t+1 = (m̄t+1 , v̄t+1 ) denote the tentative state associated with ut . We reconstruct a target state from a pseudo-gradient consistent with pt under the current adaptive denominator, as detailed in Appendix B.7.2. The persistent state is then softly aligned as mt+1 = (1 − ρm )m̄t+1 + ρm m̃t+1 ,

vt+1 = (1 − ρv )v̄t+1 + ρv ṽt+1 .

(6)

The coefficients ρm , ρv ∈ [0, 1] control the alignment strength. Setting ρm = ρv = 0 recovers direction alignment alone, while larger values place greater weight on the target moments reconstructed from pt . The same principle can extend to other stateful optimizers, including RMSProp, AdamW, and SOAP, using their respective state-update rules. The corresponding projection and state-alignment procedures, together with the cross-optimizer results, are provided in Appendix C.5. 5

arXiv Preprint

4

E XPERIMENTS

4.1

E XPERIMENTAL S ETUP

We structure our experiments around the following research questions: Q1 (Optimizer-Induced GUM): Does optimizer transformation induce conflicts after gradient surgery? Q2 (PINN Performance): Does GUA achieve conflict-free applied updates and improve PINN performance? Q3 (Direction Alignment Mechanism): Does GUA improve optimization primarily through direction alignment rather than changes in update magnitude? Q4 (State Alignment Mechanism): Does optimizer-state alignment provide additional benefits beyond direction alignment? Q5 (GUA Overhead): What computational and memory overhead does GUA introduce? Q6 (Task-Cardinality Scaling): Does GUA remain effective as the number of jointly optimized tasks increases? For the PINN evaluation, we use six PDE benchmarks following PINNacle Hao et al. (2024) and ConFIG Liu et al. (2025), and evaluate GUA with six representative gradient surgery methods: PCGrad Yu et al. (2020), CAGrad Liu et al. (2021a), IMTL-G Liu et al. (2021b), A-MTL Senushkin et al. (2023), UPGrad Quinton & Rey (2024), and ConFIG Liu et al. (2025). PINN results are reported over five independent runs on a single NVIDIA RTX 4090 GPU. We also use CelebA Liu et al. (2015) as a controlled task-cardinality benchmark with varying numbers of jointly optimized tasks, reporting results over three independent runs. For the PINN benchmarks, let LN , LB , and LI denote the PDE residual, boundary, and initial-condition losses, respectively. In the 2-loss setting, we introduce a composite loss LBI that aggregates the contributions from the boundary and initial conditions, and evaluate [LN , LBI ]. We also evaluate the 3-loss setting [LN , LB , LI ]. Detailed experimental settings are provided in Appendix B and Appendix D. 4.2

Q1: O PTIMIZER -I NDUCED GUM

Question Q1: Does optimizer transformation induce conflicts after gradient surgery? We use ConFIG as the base gradient surgery method because it constructs conflict-free directions before optimizer transformation, yielding Ra = 0. This isolates the effect of optimizer transformation, since any subsequent conflict in ut cannot be attributed to residual conflict in at . We then measure the post-optimizer conflict rate Ru across first-order and second-order optimizer families to assess whether the conflict-free geometry is preserved after optimizer transformation. Answer to Q1. Table 1 reports the mean conflict rates along the three-stage transition Rg → Ra → Ru , corresponding to the raw loss-specific gradients {gi,t }m i=1 , constructed direction at , and optimizer proposal ut , respectively. ConFIG yields Ra = 0 in all settings, so nonzero Ru directly indicates GUM. Consistent with the conflict-preservation analysis in Section 3.3, plain SGD preserves Ru = 0. In contrast, momentum alone produces substantial GUM, with M-SGD yielding Ru = 51.3%–86.3%. Adaptive and curvature-based optimizers can also produce nonzero Ru , with the rate varying across PDEs. These results show that optimizer transformation can reintroduce conflicts after gradient surgery, depending on its interaction with the current loss-gradient geometry.

Table 1: Optimizer-induced GUM across PINN benchmarks. The sequence Rg → Ra → Ru tracks conflict rates (↓) from raw loss-specific gradients through gradient surgery to optimizer transformation. Additional PDE results are provided in Appendix C.3. Schrödinger Optimizer

2-loss Rg

Burgers 3-loss

2-loss Ru

2-loss

3-loss

Ra

Ru

Rg

Ra

Ru

Rg

Rg

Ra

Ru

Rg

Ra

Ru

Rg

Ra

Ru

SGD 100.0 0.0 M-SGD 99.8 0.0 RMSProp 78.7 0.0 Adam 40.8 0.0 AdamW 46.5 0.0

0.0 82.3 1.8 23.7 26.3

100.0 99.8 83.3 56.3 62.0

0.0 0.0 0.0 0.0 0.0

0.0 81.6 18.8 35.2 35.9

First-order optimizers 99.7 0.0 0.0 99.9 99.9 0.0 80.5 100.0 58.4 0.0 6.1 87.1 30.5 0.0 29.4 73.7 30.1 0.0 29.9 74.0

0.0 0.0 0.0 0.0 0.0

0.0 86.3 24.7 57.4 55.5

70.1 58.6 27.7 22.8 25.0

0.0 0.0 0.0 0.0 0.0

0.0 51.3 0.0 12.8 15.6

99.0 95.5 72.8 81.3 80.7

0.0 0.0 0.0 0.0 0.0

0.0 72.7 5.5 64.6 63.8

SophiaG 91.7 AdaHessian 99.7 SOAP 25.3

36.3 27.2 17.3

96.3 99.7 63.3

0.0 0.0 0.0

69.5 16.9 37.6

Second-order optimizers 76.9 0.0 42.8 96.9 0.0 99.4 0.0 43.9 99.9 0.0 25.5 0.0 27.5 68.1 0.0

66.4 70.9 57.2

33.9 75.2 33.9

0.0 0.0 0.0

42.4 40.9 14.4

80.7 99.8 64.9

0.0 0.0 0.0

73.2 57.1 45.1

0.0 0.0 0.0

Ra

Heat-MS 3-loss

6

arXiv Preprint

Adam Adam

PCGrad PCGrad

Kovasznay-2

Relative L2 error

CAGrad CAGrad

IMTL-G IMTL-G

Burgers-2

A-MTL A-MTL

UPGrad UPGrad

Schrödinger-2

10

10

−1

10

−2

Burgers-3

ConFIG ConFIG Heat-MS-2

10 10

−4

−1

−3

Relative L2 error

Relative L2 error

Baseline +GUA

×10

−3

Beltrami-2

10

−2

10

−3

10

−4

Poisson-5D-2

7 5 3

10

0

Schrödinger-3 10

0

Heat-MS-3

×10

−3

Beltrami-3

7 10

−1

5 10

10

−3

10

−1

3

−2

Figure 3: Performance of different gradient surgery methods with and without GUA across PINN benchmarks under 2-loss and 3-loss settings. 4.3

Q2: PINN P ERFORMANCE

Question Q2: Does GUA achieve conflict-free applied updates and improve PINN performance? After establishing GUM, we assess whether GUA achieves conflict-free applied updates and improves PINN performance. We evaluate GUA on six PDE benchmarks using representative gradient surgery methods, with Adam as a no-surgery control that passes the original aggregate gradient directly to the optimizer. Figure 3 reports the relative L2 error. Quantitative results and detailed conflict transitions Rg → Ra → Ru → Rp are provided in Appendix C.4. Answer to Q2. As shown in Appendix C.4, GUA eliminates update-level conflicts, yielding Rp = 0 across all PINN settings. Figure 3 shows that GUA consistently improves PINN accuracy across gradient surgery methods and PDE benchmarks. The mean relative L2 error decreases in every setting, with individual reductions ranging from 11.6% to 98.2% and median reductions of 42.2% and 61.9% in the two- and three-loss settings, respectively. Notably, Adam+GUA, without gradient surgery, improves over Adam in all 10 PINN settings and outperforms all standalone gradient surgery baselines in 4 settings, highlighting the value of conflict handling after optimizer transformation. GUA also remains effective with RMSProp, AdamW, and SOAP, as shown in Appendix C.5. 4.4

Q3: D IRECTION A LIGNMENT M ECHANISM

Question Q3: Does GUA improve optimization primarily through direction alignment rather than changes in update magnitude? Direction alignment can change both the direction and norm of the optimizer proposal, so performance gains alone cannot distinguish their effects. Using ConFIG as the base gradient surgery method, we conduct a same-state, norm-matched comparison across all 10 PINN settings. At each evaluated step, we fix the parameters, optimizer state, batch, and learning ∥pt ∥2 ut . Here, qt preserves the direction of ut while matching the rate, and compare ut , pt , and qt = ∥u t ∥2 norm of pt , thereby isolating the effect of direction alignment. We evaluate the worst relative onet d)−Li (θt ) . The positive, step-fixed denominator rescales step loss change Wtrel (d) = maxi Li (θt|L−η i (θt )|+ϵ each loss without changing Ct , so the same local-safety argument applies to Wtrel . Answer to Q3. As shown in Figure 4, the benefit of GUA is primarily driven by direction alignment. Across all 10 PINN settings, pt yields a lower worst relative one-step loss change than ut in 95.1% of paired same-state comparisons and than the norm-matched qt in 93.4% of cases (see Appendix Table 10). Since qt matches the norm of pt while preserving the direction of ut , the improvement over qt isolates the effect of direction alignment. These results empirically support the direction alignment mechanism in Proposition 2 and the norm-matching analysis in Appendix A.4. 7

arXiv Preprint

Optimizer proposal ut rel

Worst loss change Wt

Norm-matched qt

GUA update pt

2-loss 10

−1

10

10

10

3-loss

0

Kovasznay

Burgers

Schrödinger Heat-MS

Beltrami

0

−1

Poisson-5D

Burgers

Schrödinger Heat-MS

Beltrami

Figure 4: Worst relative one-step loss change Wtrel for the optimizer proposal ut , the GUA update pt , and the norm-matched control qt under paired same-state evaluations across PINN benchmarks. 4.5

Q4: S TATE A LIGNMENT M ECHANISM

Question Q4: Does optimizer-state alignment provide additional benefits beyond direction alignment? Direction alignment acts on the current optimizer proposal ut , but stateful optimizers retain internal statistics that influence future proposals. We therefore examine whether adjusting the optimizer state toward targets reconstructed from the applied update pt provides additional benefits beyond direction alignment alone. Using ConFIG as the base gradient surgery method with Adam as the optimizer, we compare the gradient surgery baseline without GUA, direction alignment only (ρm , ρv ) = (0, 0), soft state alignment (see Appendix B.7.2), exact first-moment alignment (ρm , ρv ) = (1, 0), and exact alignment of both Adam moments (ρm , ρv ) = (1, 1). Answer to Q4. Figure 5 shows that direction alignment alone already improves performance, while soft state alignment provides further improvements in most settings. In contrast, exact alignment is less effective, and aligning both Adam moments exactly degrades performance in some settings and becomes numerically unstable in one Heat-MS setting. A possible explanation is that direction alignment affects only the current optimizer proposal, whereas the moments maintained by Adam encode information accumulated over a longer optimization horizon. Fully replacing them with states reconstructed from pt can discard useful history and alter future adaptive scaling. Soft alignment instead preserves part of this history while moving the optimizer state toward target moments reconstructed from pt , motivating its use in GUA. Direction alignment only Burgers-2 Schrödinger-2 10 10

10

−4

Soft alignment Heat-MS-2 −1

10

−1

6 5

10

10

−2

10

−4

2

Heat-MS-3 10

−1

10 −3

10

−2

Schrödinger-3 10

Poisson-5D-2

4 3

Burgers-3

10

Exact m Exact m,v −3 ×10 Beltrami-2 −3

−3

10

Relative L2 error

Relative L2 error

Baseline Kovasznay-2

−2

0

−1

×10

−3

Beltrami-3

6 5 4 3 2

Figure 5: State-alignment ablation on PINN benchmarks.

4.6

Q5: GUA OVERHEAD

Question Q5: What computational and memory overhead does GUA introduce? We evaluate the practical overhead of GUA by comparing training time and peak GPU memory with and without GUA across all PINN settings, using ConFIG as the base method. GUA performs additional conflict checking and direction alignment at each step. The projection is solved in the m-dimensional dual without forming a p × p matrix. Its parameter-dimensional cost is O(m2 p) for Gram-matrix construction, followed by a small dual solve. Since m ∈ {2, 3} in the PINN experiments, this dual problem is solved exactly by active-set enumeration. Further details are provided in Appendix B.7.1. 8

Ratio to baseline

arXiv Preprint

Error ratio 2-loss

1.6

Time ratio

Memory ratio

1.2

1.2

0.8

0.8

0.4

0.4

0.0

Kovasznay

Burgers

Schrödinger Heat-MS

3-loss

1.6

Beltrami

Poisson-5D

0.0

Burgers

Schrödinger Heat-MS

Beltrami

Baseline +GUA (b) Training time

(a) Performance 6

Time (h)

F1

0.725 0.700 0.675

2

3

5 10 20 30 Number of tasks

40

4 2 0

2

3

5 10 20 30 Number of tasks

40

Peak memory (GiB)

Figure 6: Relative L2 error (↓), training-time (↓), and peak-memory (↓) ratios across PINN settings. (c) Peak memory 20 15 10 5

2

3

5 10 20 30 Number of tasks

40

Figure 7: Task scaling of GUA in terms of F1 (↑), training time (↓), and peak GPU memory (↓). Answer to Q5. As shown in Figure 6, GUA introduces additional training-time overhead while adding little peak GPU memory overhead. Across all PINN settings, GUA consistently improves accuracy. These results highlight a trade-off between accuracy and efficiency. GUA improves accuracy at the cost of additional computation while incurring little additional memory usage. 4.7

Q6: TASK -C ARDINALITY S CALING

Question Q6: Does GUA remain effective as the number of jointly optimized tasks increases? We use CelebA Liu et al. (2015) as a controlled task-cardinality benchmark, varying the number of attribute tasks from 2 to 40 while fixing ConFIG as the baseline. For each task count, we compare ConFIG and ConFIG+GUA in terms of F1 , training time, and peak GPU memory. The projection uses exact active-set enumeration for small m and an m-dimensional nonnegative dual QP for larger m. Further details are provided in Appendix B.7.1. Answer to Q6. Figure 7 shows that GUA remains effective as task cardinality increases from 2 to 40, improving the F1 in six of seven settings and achieving comparable performance at 20 tasks. The additional memory cost remains small as task cardinality increases, reaching only 4.2% at 40 tasks, while training-time overhead becomes more pronounced for larger task counts. These results demonstrate that GUA remains effective as the number of jointly optimized tasks increases.

5

C ONCLUSION

This work provides a new perspective on gradient conflict handling in PINN training. We identify the largely overlooked phenomenon of GUM and demonstrate its prevalence across evaluated PINN settings through theoretical analysis and extensive experiments. Motivated by this finding, we propose GUA to enforce conflict-free geometry at the update level. Extensive experiments show that GUA effectively eliminates update-level conflicts and substantially improves PINN performance. Limitations and Future Work. Despite these advances, the current formulation of GUM and GUA is based on local first-order gradient geometry and therefore does not explicitly capture curvature, finite-step, or longer-horizon optimization effects. Moreover, the current hard conflict-free criterion may be conservative by excluding update directions that could become beneficial beyond the local first-order view. Therefore, extending the framework beyond instantaneous first-order geometry, together with developing softer conflict criteria, is an important direction for future work. 9

arXiv Preprint

R EFERENCES Giacomo Baldan, Qiang Liu, Alberto Guardone, and Nils Thuerey. Physics vs distributions: Pareto optimal flow matching with physics constraints. In International Conference on Learning Representations, volume 2026, pp. 101215–101243, 2026. Rafael Bischof and Michael A Kraus. Multi-objective loss balancing for physics-informed deep learning. Computer Methods in Applied Mechanics and Engineering, 439:117914, 2025. Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pp. 794–803. PMLR, 2018. Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039–2050, 2020. Lukas Gonon, Arnulf Jentzen, Benno Kuckuck, Siyu Liang, Adrian Riekert, and Philippe von Wurstemberger. An overview on machine learning methods for partial differential equations: from physics informed neural networks to deep operator learning. arXiv preprint arXiv:2408.13222, 2024. Zhongkai Hao, Jiachen Yao, Chang Su, Hang Su, Ziao Wang, Fanzhi Lu, Zeyu Xia, Yichi Zhang, Songming Liu, Lu Lu, et al. Pinnacle: A comprehensive benchmark of physics-informed neural networks for solving pdes. Advances in neural information processing systems, 37:76721–76774, 2024. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1–97, 2023. Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in neural information processing systems, 34:26548–26560, 2021. Shirong Li and Xinlong Feng. Dynamic weight strategy of physics-informed neural networks for the 2d navier–stokes equations. Entropy, 24(9):1254, 2022. Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020. Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in neural information processing systems, 34:18878–18890, 2021a. Dehao Liu and Yan Wang. A dual-dimer method for training physics-constrained neural networks with minimax architecture. Neural Networks, 136:112–125, 2021. Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International conference on learning representations, volume 2024, pp. 1621–1650, 2024. Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International conference on learning representations, 2021b. Qiang Liu, Mengyu Chu, and Nils Thuerey. Config: Towards conflict-free training of physics informed neural networks. In International Conference on Learning Representations, volume 2025, pp. 59531–59566, 2025. 10

arXiv Preprint

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730–3738, 2015. Ilya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017.

Decoupled weight decay regularization.

arXiv preprint

Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218–229, 2021. Pierre Quinton and Valérian Rey. Jacobian descent for multi-objective optimization. arXiv preprint arXiv:2406.16232, 2024. Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019. Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018. Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. Independent component alignment for multi-task learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20083–20093. IEEE, 2023. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139–1147. pmlr, 2013. Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5—RMSProp: Divide the gradient by a running average of its recent magnitude. Coursera course, Neural Networks for Machine Learning, 2012. Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam for language modeling. In International Conference on Learning Representations, volume 2025, pp. 93423–93444, 2025. Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3055– A3081, 2021. Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:110768, 2022. Sifan Wang, Ananyae Bhartari, Bowen Li, and Paris Perdikaris. Gradient alignment in PhysicsInformed Neural Networks: A second-order optimization perspective. In Advances in Neural Information Processing Systems, volume 38, pp. 168482–168532. Curran Associates, Inc., 2025. doi: 10.52202/085713-5615. Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10665–10673, 2021. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33: 5824–5836, 2020. Weiwei Zhang, Wei Suo, Jiahao Song, and Wenbo Cao. Physics-informed neural networks (pinns) as intelligent computing technique for solving partial differential equations: Limitation and future prospects. SCIENCE CHINA Physics, Mechanics & Astronomy, 69(1):214602, 2026.

11

arXiv Preprint

A

T HEORY AND P ROOFS

A.1

P ROOF OF P ROPOSITION 1

Proposition 1.

Let Gt = [g1,t , . . . , gm,t ] and Ct = {d ∈ Rp | G⊤ t d ≥ 0}.

The affine transformation Tt (a) = Pt a + bt is conflict-preserving on Ct , i.e., a ∈ Ct

=⇒

Tt (a) ∈ Ct ,

if and only if Pt⊤ gi,t ∈ cone(g1,t , . . . , gm,t ),

bt ∈ Ct ,

i = 1, . . . , m.

Proof. We first prove necessity. Since Ct is a cone, 0 ∈ Ct . If Tt is conflict-preserving, then Tt (0) = bt ∈ Ct . Now take any a ∈ Ct and any scalar λ ≥ 0. Since λa ∈ Ct , conflict preservation gives Tt (λa) = λPt a + bt ∈ Ct . Therefore, for every i, λ⟨gi,t , Pt a⟩ + ⟨gi,t , bt ⟩ ≥ 0

for all λ ≥ 0.

Letting λ → ∞ implies ⟨gi,t , Pt a⟩ ≥ 0

for all a ∈ Ct .

⟨Pt⊤ gi,t , a⟩ ≥ 0

for all a ∈ Ct .

Equivalently, Define the dual cone of Ct as Ct∗ = {y ∈ Rp | ⟨y, d⟩ ≥ 0 for all d ∈ Ct } . The preceding inequality shows that Pt⊤ gi,t ∈ Ct∗ . Since

Ct = {d ∈ Rp | G⊤ t d ≥ 0} is a closed polyhedral cone, Farkas’ lemma gives Ct∗ = cone(g1,t , . . . , gm,t ). Hence,

Pt⊤ gi,t ∈ cone(g1,t , . . . , gm,t ),

i = 1, . . . , m.

We now prove sufficiency. Suppose bt ∈ Ct and For any a ∈ Ct ,

Pt⊤ gi,t ∈ cone(g1,t , . . . , gm,t )

for all i.

⟨gi,t , Pt a⟩ = ⟨Pt⊤ gi,t , a⟩ ≥ 0,

because Pt⊤ gi,t is a nonnegative combination of g1,t , . . . , gm,t , while ⟨gj,t , a⟩ ≥ 0

for every j.

Thus, Pt a ∈ Ct . Since bt ∈ Ct and Ct is a convex cone, it is closed under addition. Therefore, Pt a + bt ∈ Ct . Hence, Tt (a) = Pt a + bt ∈ Ct so Tt is conflict-preserving. 12

for every a ∈ Ct ,

arXiv Preprint

A.2

G ENERAL N ONLINEAR C ONFLICT P RESERVATION

The fixed-state affine result above provides an interpretable characterization, but exact optimizer maps need not be affine in the constructed direction. We therefore give a general characterization for nonlinear optimizer transformations. Proposition 3 (General nonlinear conflict preservation). Fix Gt = [g1,t , . . . , gm,t ] and all step-t quantities other than the input direction a, and let Tt : Rp → Rp denote the resulting one-step optimizer transformation. Suppose Tt is locally Lipschitz. For each a ∈ Ct , define the ray-wise map ϕt,a (τ ) = Tt (τ a), τ ∈ [0, 1]. Then Tt is conflict-preserving on Ct if and only if, for every a ∈ Ct and every i = 1, . . . , m, Z 1 ⟨gi,t , Tt (0)⟩ + gi,t , ϕ′t,a (τ ) dτ ≥ 0, 0

where ϕ′t,a (τ ) exists almost everywhere. If Tt is continuously differentiable, this condition becomes Z 1 ⟨gi,t , Tt (0)⟩ + JTt (τ a)⊤ gi,t , a dτ ≥ 0, 0

where JTt denotes the Jacobian of Tt . Proof. For any fixed a ∈ Ct , the locally Lipschitz property of Tt implies that the ray-wise map ϕt,a (τ ) = Tt (τ a) is absolutely continuous on [0, 1]. Hence it is differentiable almost everywhere, and the fundamental theorem for absolutely continuous functions gives Z 1 Tt (a) = Tt (0) + ϕ′t,a (τ ) dτ. 0

Taking the inner product with gi,t yields Z 1 ⟨gi,t , Tt (a)⟩ = ⟨gi,t , Tt (0)⟩ +

gi,t , ϕ′t,a (τ ) dτ.

0

By definition, Tt (a) ∈ Ct ⇐⇒ ⟨gi,t , Tt (a)⟩ ≥ 0 for all i. Therefore, requiring the displayed inequality for every a ∈ Ct and every i is equivalent to Tt (Ct ) ⊆ Ct . If Tt is continuously differentiable, the chain rule gives ϕ′t,a (τ ) = JTt (τ a)a, which yields the stated Jacobian form. Relation to the affine case.

For an affine transformation Tt (a) = Pt a + bt ,

we have Tt (0) = bt , JTt (a) = Pt . Thus, the nonlinear characterization reduces to ⟨gi,t , bt ⟩ + ⟨Pt⊤ gi,t , a⟩ ≥ 0 for every a ∈ Ct . Proposition 1 gives the corresponding interpretable necessary-and-sufficient conditions bt ∈ Ct , Pt⊤ gi,t ∈ cone(g1,t , . . . , gm,t ). Hence Proposition 1 is the affine specialization of the general nonlinear characterization. The locally Lipschitz formulation also covers piecewise-smooth optimizer transformations: the raywise derivative is interpreted almost everywhere, so nondifferentiabilities introduced, for example, by clipping do not affect the characterization. 13

arXiv Preprint

A.3

P ROOF OF P ROPOSITION 2

Proposition 2. Suppose each loss Li is βi -smooth, with βi > 0, on a convex neighborhood of θt , and suppose the optimizer proposal is conflicting, i.e., ut ∈ / Ct . Let δt = vt (ut ) = max[−⟨gi,t , ut ⟩]+ > 0 i

and choose

i⋆ ∈ arg max[−⟨gi,t , ut ⟩]+ . i

Let

1 pt = ΠCMt t (ut ) = arg min ∥d − ut ∥2Mt . d∈Ct 2 Then, for every ηt > 0 such that θt − ηt ut and θt − ηt pt remain in this neighborhood, max ∆i,t (pt ; ηt ) ≤ i

βmax ηt2 ∥pt ∥22 , 2

and ∆i⋆ ,t (ut ; ηt ) ≥ ηt δt −

βi⋆ ηt2 ∥ut ∥22 , 2

where βmax = max βi . i

Consequently, max ∆i,t (pt ; ηt ) < max ∆i,t (ut ; ηt ) i

i

for all sufficiently small ηt > 0. Moreover, for every d ∈ Ct , ∥ut − d∥2Mt ≥ ∥ut − pt ∥2Mt + ∥pt − d∥2Mt . Proof. Because Mt ≻ 0 and Ct is a nonempty closed convex cone, the metric projection 1 pt = arg min ∥d − ut ∥2Mt d∈Ct 2 exists and is unique. For convenience, define Wt (d; ηt ) = max ∆i,t (d; ηt ). 1≤i≤m

First-order safety.

By the definition of Ct , pt ∈ Ct

=⇒

⟨gi,t , pt ⟩ ≥ 0,

i = 1, . . . , m.

Since Li is βi -smooth, for any direction d and any admissible step size ηt > 0, |Li (θt − ηt d) − Li (θt ) + ηt ⟨gi,t , d⟩| ≤

βi ηt2 ∥d∥22 . 2

Applying the upper bound with d = pt gives ∆i,t (pt ; ηt ) ≤ −ηt ⟨gi,t , pt ⟩ +

βi ηt2 ∥pt ∥22 . 2

Because ⟨gi,t , pt ⟩ ≥ 0, we obtain ∆i,t (pt ; ηt ) ≤

βi ηt2 ∥pt ∥22 . 2

Therefore,

βmax ηt2 ∥pt ∥22 . i 2 Thus, the projected update contains no positive first-order loss-change term. Wt (pt ; ηt ) = max ∆i,t (pt ; ηt ) ≤

14

arXiv Preprint

One-step improvement for conflicting optimizer proposals.

Since

ut ∈ / Ct , we have δt = vt (ut ) = max[−⟨gi,t , ut ⟩]+ > 0. i

Choose i⋆ ∈ arg max[−⟨gi,t , ut ⟩]+ . i

Then −⟨gi⋆ ,t , ut ⟩ = δt , and therefore ⟨gi⋆ ,t , ut ⟩ = −δt . Using the lower smoothness bound with d = ut yields ∆i⋆ ,t (ut ; ηt ) ≥ −ηt ⟨gi⋆ ,t , ut ⟩ − = ηt δt −

βi⋆ ηt2 ∥ut ∥22 2

βi⋆ ηt2 ∥ut ∥22 . 2

Since Wt (ut ; ηt ) is the maximum loss change, Wt (ut ; ηt ) ≥ ∆i⋆ ,t (ut ; ηt ), and hence Wt (ut ; ηt ) ≥ ηt δt −

βi⋆ ηt2 ∥ut ∥22 . 2

Combining the bounds for pt and ut , we obtain Wt (pt ; ηt ) < Wt (ut ; ηt ) whenever

βi⋆ ηt2 βmax ηt2 ∥pt ∥22 < ηt δt − ∥ut ∥22 . 2 2 For ηt > 0, this is equivalent to 0 < ηt <

2δt . βi⋆ ∥ut ∥22 + βmax ∥pt ∥22

Therefore, whenever the two trial points remain in the assumed neighborhood, Wt (pt ; ηt ) < Wt (ut ; ηt ) for every 2δt . βi⋆ ∥ut ∥22 + βmax ∥pt ∥22 In particular, the strict inequality holds for all sufficiently small admissible step sizes. 0 < ηt <

Least-change correction.

The first-order optimality condition of the metric projection is ⟨Mt (pt − ut ), d − pt ⟩ ≥ 0,

∀d ∈ Ct .

Define the Mt -inner product as ⟨x, y⟩Mt = x⊤ Mt y. The optimality condition is equivalently ⟨ut − pt , d − pt ⟩Mt ≤ 0, and hence ⟨ut − pt , pt − d⟩Mt ≥ 0. 15

arXiv Preprint

For any d ∈ Ct , expanding the squared distance gives ∥ut − d∥2Mt = ∥ut − pt ∥2Mt + ∥pt − d∥2Mt + 2⟨ut − pt , pt − d⟩Mt ≥ ∥ut − pt ∥2Mt + ∥pt − d∥2Mt . This proves the metric projection inequality. In particular, if at ∈ Ct , then substituting d = at gives ∥ut − at ∥2Mt ≥ ∥ut − pt ∥2Mt + ∥pt − at ∥2Mt , or equivalently, ∥pt − at ∥2Mt ≤ ∥ut − at ∥2Mt − ∥ut − pt ∥2Mt . Thus, GUA restores feasibility through the smallest feasible correction under the Mt -metric. More generally, the projection inequality shows that pt is closer than ut to every feasible direction d ∈ Ct , including the original constructed direction at whenever at ∈ Ct . If ⟨gi,t , pt ⟩ > 0 for every i, the same smoothness bounds further imply simultaneous decrease of all losses for sufficiently small step sizes. When pt lies on the boundary of Ct , however, some inner products may be zero, and Proposition 2 guarantees only the absence of a positive first-order loss-change term for the corresponding losses. A.4

N ORM M ATCHING D OES N OT R ESTORE C ONFLICT-F REENESS

The following observation provides the theoretical basis for the norm-matched control used in Q3 and Figure 4. Norm-matching observation.

Suppose ut ∈ / Ct

and let qt = αt ut for any αt > 0. Then vt (qt ) = αt vt (ut ) > 0. Thus, positive rescaling cannot restore conflict-freeness. In particular, when pt ̸= 0, the normmatched control ∥pt ∥2 ut qt = ∥ut ∥2 satisfies ∥qt ∥2 = ∥pt ∥2 but remains outside Ct . Indeed, since ut ∈ / Ct , we have vt (ut ) > 0. For any αt > 0, vt (αt ut ) = max[−⟨gi,t , αt ut ⟩]+ i

= αt max[−⟨gi,t , ut ⟩]+ i

= αt vt (ut ) > 0. Hence αt ut ∈ / Ct

for every αt > 0. 16

arXiv Preprint

When pt ̸= 0, the norm-matching factor αt =

∥pt ∥2 ∥ut ∥2

is strictly positive, because ut ∈ / Ct implies ut ̸= 0. Therefore, ∥qt ∥2 = ∥pt ∥2 ,

vt (qt ) =

∥pt ∥2 vt (ut ) > 0. ∥ut ∥2

Thus, qt ∈ / Ct , even though qt and pt have the same Euclidean norm. Hence, norm matching alone cannot remove GUM, and changing the update direction is necessary to restore conflict-freeness.

17

arXiv Preprint

B

E XPERIMENTAL D ETAILS FOR PINN S

B.1

PDE B ENCHMARKS

We evaluate GUA on standard PDE benchmarks spanning steady and time-dependent PDEs, scalar and multi-component systems, low- and high-dimensional domains, and incompressible flows. The benchmark protocols are mainly based on PINNacle Hao et al. (2024) and ConFIG Liu et al. (2025). Schrödinger. The Schrödinger benchmark solves a nonlinear Schrödinger equation by representing the complex solution as h = u + iv. The residuals are 1 ut + vxx + (u2 + v 2 )v = 0, 2 1 vt − uxx − (u2 + v 2 )u = 0, 2 on (x, t) ∈ [−5, 5] × [0, π/2]. The initial condition is u(x, 0) =

2 , cosh(x)

v(x, 0) = 0.

The equation follows periodic boundary conditions at x = −5 and x = 5. Burgers. The Burgers benchmark solves the one-dimensional viscous Burgers equation on (x, t) ∈ [−1, 1] × [0, 1]: 0.01 . ut + uux − νuxx = 0, ν= π The initial condition is u(x, 0) = − sin(πx), and homogeneous Dirichlet boundary conditions are imposed: u(−1, t) = u(1, t) = 0. Kovasznay flow. The Kovasznay benchmark solves a steady two-dimensional incompressibleflow problem on Ω = [−0.5, 1] × [−0.5, 1.5]: uux + vuy + px − ν(uxx + uyy ) = 0, uvx + vvy + py − ν(vxx + vyy ) = 0, ux + vy = 0. The Reynolds number is Re = 40, so ν = 1/Re. The analytical solution is u⋆ (x, y) = 1 − exp(λx) cos(2πy), λ v ⋆ (x, y) = exp(λx) sin(2πy), 2π 1 p⋆ (x, y) = (1 − exp(2λx)) , 2 where

r 1 1 λ= − + 4π 2 . 2ν 4ν 2 Dirichlet boundary conditions are given by the analytical solution on ∂Ω. The same analytical solution is used to generate validation and test data. Heat-MS. The Heat-MS reference data follow a two-dimensional multiscale heat solution on (x, y, t) ∈ [0, 1] × [0, 1] × [0, 5]. The analytical reference corresponds to ut − cx uxx − cy uyy = 0,

with cx =

1 , (500π)2 18

cy =

1 . π2

arXiv Preprint

The initial condition is u(x, y, 0) = sin(20πx) sin(πy), and homogeneous Dirichlet boundary conditions are imposed on the spatial boundary. The exact solution used for evaluation is 
 u⋆ (x, y, t) = sin(20πx) sin(πy) exp − cx (20π)2 + cy π 2 t . The training residual uses the same anisotropic diffusivities, ut − cx uxx − cy uyy = 0, as the analytical reference. The initial, boundary, validation, and test data are sampled from that reference solution. Beltrami flow. The Beltrami benchmark solves an unsteady three-dimensional incompressibleflow problem on (x, y, z, t) ∈ [−1, 1]3 × [0, 1]: ut + uux + vuy + wuz + px − ν(uxx + uyy + uzz ) = 0, vt + uvx + vvy + wvz + py − ν(vxx + vyy + vzz ) = 0, wt + uwx + vwy + wwz + pz − ν(wxx + wyy + wzz ) = 0, ux + vy + wz = 0. The default setting uses Re = 1, hence ν = 1, with parameters a = d = 1. The analytical solution is 2

u⋆ = −a [eax sin(ay + dz) + eaz cos(ax + dy)] e−d t , 2

v ⋆ = −a [eay sin(az + dx) + eax cos(ay + dz)] e−d t , 2

w⋆ = −a [eaz sin(ax + dy) + eay cos(az + dx)] e−d t , and 1 h p⋆ = − a2 e2ax + e2ay + e2az + 2 sin(ax + dy) cos(az + dx)ea(y+z) 2 + 2 sin(ay + dz) cos(ax + dy)ea(z+x) i 2 + 2 sin(az + dx) cos(ay + dz)ea(x+y) e−2d t . Initial and boundary conditions are sampled from this analytical solution, and the same analytical solution is used for validation and testing. Poisson-5D. The Poisson-5D benchmark from PINNacle solves a scalar Poisson equation on the five-dimensional hypercube Ω = [0, 1]5 : ∆u(x) + f (x) = 0, where

5

f (x) =

π  π2 X sin xj . 4 j=1 2

The exact solution is u⋆ (x) =

5 X

sin

j=1

π 2

 xj .

Dirichlet boundary conditions are given by the exact solution, u = u⋆ on ∂Ω. The reference solution is generated analytically from u⋆ .

19

arXiv Preprint

B.2

N ETWORK A RCHITECTURES

All PINN models use fully connected multilayer perceptrons (MLPs). Each model maps the spatiotemporal coordinates of a collocation point to a shared hidden representation and then to the PDEspecific output channels. Concretely, for a benchmark with input coordinate x ∈ Rdin , the network has the form ûθ (x) = WL+1 ϕ(WL ϕ(· · · ϕ(W1 x + b1 ) · · · ) + bL ) + bL+1 , where L is the number of hidden blocks, the hidden width is shared within a benchmark, and ϕ is the activation function listed in Table 2. We use one shared trunk per benchmark rather than separate subnetworks for different loss terms. The residual, boundary, and initial-condition losses are therefore coupled through the same parameter vector θ. The input coordinates are the physical coordinates used by the corresponding benchmark runner. Time-dependent scalar equations use coordinates such as (x, t), steady two-dimensional flow problems use (x, y), and high-dimensional Poisson uses its spatial coordinates directly. The output dimension is determined by the PDE state: scalar equations output one field, complex or multi-species systems output multiple channels, and incompressible flow benchmarks output velocity components together with pressure when pressure appears in the residual. Spatial and temporal derivatives in the PDE residuals are computed by automatic differentiation through this MLP, so the same network must support all derivative orders required by the benchmark equation. Architectures are fixed across all baseline gradient surgery methods and their +GUA variants. We do not retune network depth, width, activation, or output parameterization for GUA, so the comparison isolates the effect of GUA rather than architectural differences. All models use Xavier initialization, and for each random seed, the base method and its +GUA counterpart use the same initialization. Table 2: PINN network architectures. Shared architecture fields are repeated to make the perbenchmark settings explicit. Benchmark Input coordinates Output dim. Hidden width Hidden layers Activation Burgers Schrödinger Kovasznay Poisson-5D Heat-MS Beltrami

(x, t) (x, t) (x, y) (x1 , . . . , x5 ) (x, y, t) (x, y, z, t)

1 2 3 1 1 4

50 50 50 50 50 50

20

5 5 5 5 5 5

Tanh Tanh Tanh Tanh Tanh Tanh

arXiv Preprint

B.3

T RAINING S ETTINGS

All PINN experiments use Xavier initialization and full-batch training over the sampled collocation, boundary, and initial-condition points. Unless otherwise stated, training points are resampled every epoch using Latin-hypercube sampling. Adam is used as the default optimizer with β1 = 0.9, β2 = 0.999, and ϵ = 10−8 . We use no manual scalar reweighting, with all loss weights set to λi = 1 before method-specific gradient manipulation. Validation is performed every 100 epochs. For each run, we select the checkpoint with the best validation performance and use it for the final test evaluation. Each reported PINN result is presented as the mean and standard deviation over five independent runs using seeds {0, 1, 2, 3, 4}. All experiments use a 100-epoch linear warmup. In the default setting, the warmup is followed by cosine decay. Let w = 100 denote the number of warmup epochs. At scheduler index t, the learning-rate schedule is  t  0 ≤ t < w, η 0 , w    ηt =  ηmin + 1 (η0 − ηmin ) 1 + cos π(t − w) , w ≤ t < T, 2 T −w where the first training epoch uses scheduler index t = 0, T denotes the configured number of training epochs, and the last applied update uses t = T − 1. The default setting uses η0 = 10−3 and ηmin = 10−4 . The only learning-rate exception is Heat-MS under the 3-loss setting. With η0 = 10−3 , the baseline ConFIG configuration exhibits numerical instability. To avoid confounding the comparison with this baseline instability, we set both η0 and ηmin to 10−4 for all methods in this setting. After the same 100-epoch linear warmup, the learning rate remains constant at 10−4 . Tables 3 and 4 summarize the training settings and configurations, respectively. Table 3: Global PINN training settings. Item

Setting

Initialization Sampling Optimizer Default learning rate Heat-MS/3-loss learning rate Batching Loss weights Validation interval Runs

Xavier Latin-hypercube sampling, resampled every epoch Adam (β1 = 0.9, β2 = 0.999, ϵ = 10−8 ) 100-epoch warmup, then cosine decay from 10−3 to 10−4 100-epoch warmup to 10−4 , then constant at 10−4 Full batch λi = 1 Every 100 epochs 5 seeds: {0, 1, 2, 3, 4}

Table 4: PINN training configurations used in the experiments. Nf , Nb , and Ni denote the numbers of interior collocation, boundary, and initial-condition points, respectively. Benchmark

Epochs

Nf

Nb

Ni

Burgers Schrödinger Kovasznay Poisson-5D Heat-MS Beltrami

30,000 100,000 100,000 100,000 100,000 100,000

10,000 20,000 20,000 20,000 20,000 25,000

250 500 1,000 5,000 2,000 5,000

250 500 – – 2,000 5,000

21

arXiv Preprint

B.4

O PTIMIZER C ONFIGURATIONS

This section summarizes the optimizer configurations used in the Q1 optimizer-induced GUM experiments. We fix ConFIG as the base gradient surgery method. At each training step, ConFIG constructs a pre-optimizer direction at , which is then passed to the corresponding optimizer to produce an optimizer proposal ut . No GUA is applied in these experiments, so the proposal is used directly for the parameter update: θt+1 = θt − ηt ut . The purpose of this section is therefore to specify how each optimizer transforms the constructed direction at into the proposal ut , at 7→ ut . For notational simplicity, division, square root, clipping, and squaring are applied element-wise unless otherwise stated. We use ϵ > 0 for numerical stability, ⊙ for element-wise multiplication, and λ for the weight-decay coefficient when applicable. For adaptive optimizers, mt and vt denote the first- and second-moment states, respectively. Barred quantities denote the tentative states after processing at at step t. These states produce the proposal ut under the time-indexing convention introduced in Eq. (2). Table 5: Optimizer update maps used in the optimizer-induced GUM experiments. Family

Optimizer

Effective update map

First-order

SGD

ut = at

First-order

M-SGD

m̄t+1 = βmt + at ,

First-order

RMSProp

v̄t+1 = ρvt + (1 − ρ)a2t ,

First-order

Adam

ut = m̄t+1 √ ut = at /( v̄t+1 + ϵ) 2

m̄t+1 = β1 mt + (1 − β1 )at ,

First-order

AdamW

Second-order

SophiaG

Second-order

AdaHessian

Second-order

SOAP

ˆ t+1 = m̄t+1 /(1 − β1t+1 ), m̄ ˆ t+1 /( ut = m̄

q

v̄t+1 = β2 vt + (1 − β2 )at ˆt+1 = v̄t+1 /(1 − β2t+1 ), v̄

ˆ t+1 /( ut = m̄

ˆt+1 + ϵ) + λθt v̄

m̄t+1 = β1 mt + (1 − β1 )at ,

h̄t+1 = β2 ht + (1 − β2 )ĥt

ut = clip m̄t+1 /(γ h̄t+1 + ϵ), ρ m̄t+1 = β1 mt + (1 − β1 )at ,

h̄t+1 = β2 ht + (1 − β2 )ĥt

1/2 ut = m̄t+1 /(h̄t+1 + ϵ) ⊤ ˜ t+1 = β1 m̃t + (1 − β1 )ãt ãt = Qt at , m̄

˜t+1 = β2 ṽt + (1 − β2 )ã2t , v̄

  q ˜ t+1 /( v̄ ˜t+1 + ϵ) ut = Qt m̄

22

q

ˆt+1 + ϵ) v̄

arXiv Preprint

B.5

BASELINE S ETTINGS

We compare GUA with several representative gradient surgery methods. All gradient surgery methods first construct a direction at from the loss-specific gradients {gi,t }m i=1 , and then feed this direction into the same base optimizer. This allows us to evaluate whether the direction constructed by gradient surgery before optimizer transformation is preserved after the optimizer transformation. PCGrad. PCGrad Yu et al. (2020) mitigates gradient conflict by modifying loss-specific gradients pairwise. When two loss-specific gradients have a negative inner product, PCGrad projects one gradient onto the normal plane of the other to remove the conflicting component. The modified gradients are then aggregated to form the constructed direction at . PCGrad is a representative projection-based gradient surgery method. CAGrad. CAGrad Liu et al. (2021a) constructs a direction that balances conflict reduction with optimization of the average loss. Instead of only removing pairwise conflicts, CAGrad searches for a direction near the average gradient that improves the worst local task descent. This makes it a conflict-averse aggregation method that tries to reduce task interference while remaining aligned with the average-loss objective. IMTL-G. IMTL-G Liu et al. (2021b) aims to obtain an impartial direction across tasks. It computes aggregation weights such that the resulting direction has balanced projections onto different loss-specific gradients. In this way, IMTL-G avoids favoring a subset of objectives and provides a balanced constructed direction for multi-loss training. A-MTL. A-MTL Senushkin et al. (2023) addresses gradient conflict and gradient dominance by aligning components of the gradient matrix. It treats the set of loss-specific gradients as a linear system and improves optimization stability by reducing ill-conditioning and dominance among loss-specific gradients. We include A-MTL as a representative gradient alignment method beyond pairwise projection or cone-based constraints. UPGrad. UPGrad Quinton & Rey (2024) is based on dual-cone projection in multi-objective optimization. It projects loss-specific gradients onto the dual cone induced by the set of objective gradients and aggregates the projected directions. The resulting update is designed to avoid locally increasing any objective under the current first-order geometry. ConFIG. ConFIG Liu et al. (2025) is a conflict-free direction construction method designed for multi-loss training in PINNs. Given loss-specific gradients, ConFIG constructs a direction whose inner product with each loss-specific gradient is non-negative. Therefore, its constructed direction satisfies at ∈ Ct under the current gradient geometry.

23

arXiv Preprint

B.6

E VALUATION M ETRICS

For PINN benchmarks, we report the relative L2 error against the reference solution and the fourstage conflict-rate sequence Rg → Ra → Ru → Rp . Here Rg measures raw loss-specific gradient conflict, Ra the constructed direction, Ru the optimizer proposal, and Rp the aligned update. Since a base method does not construct pt , Rp is not applicable to baseline runs. ⋆ test Let {xj }N j=1 denote the test points, let ûθ be the PINN prediction, and let u be the reference solution. The relative L2 error is 1/2 P Ntest ⋆ 2 ∥û (x ) − u (x )∥ θ j j 2 j=1 Rel-L2 = . 1/2 P Ntest ⋆ (x )∥2 ∥u j 2 j=1

All methods and variants use the same benchmark-specific evaluation points. In particular, Poisson5D uses the deterministic PINNacle-compatible uniform grid produced by eight interior coordinates per dimension, giving 85 = 32,768 evaluation points. At training step t, let {gi,t }m i=1 be the loss-specific gradients, at the constructed direction produced by the base gradient surgery method, and ut the optimizer proposal produced by the base optimizer. The current conflict-free cone is Ct = {d : ⟨gi,t , d⟩ ≥ 0,

i = 1, . . . , m} .

These quantities use the same ideal cone-membership definitions as in the main text. In the actual implementation, a conflict is detected when the normalized inner product falls below −10−6 . We use a small negative threshold rather than zero to avoid treating tiny negative values caused by floatingpoint and numerical errors near the conflict boundary as genuine conflicts. The denominator includes ϵ for numerical stability, and only diagnosed steps with a positive learning rate are included. For a generic update direction dt , the empirical conflict indicator is   ⟨gi,t , dt ⟩ It (dt ) = I ∃i ∈ {1, . . . , m} : < −10−6 . ∥gi,t ∥2 ∥dt ∥2 + ϵ The raw loss-specific gradient conflict rate is   T 1X ⟨gi,t , gj,t ⟩ −6 Rg = I ∃i < j : < −10 . T t=1 ∥gi,t ∥2 ∥gj,t ∥2 + ϵ Here T counts diagnosed steps with a positive learning rate. The zero-rate warmup step is omitted because it produces no applied update. We report the constructed-direction at , optimizer-proposal ut , and aligned-update pt conflict rates as T

Ra =

1X It (at ), T t=1

T

Ru =

1X It (ut ), T t=1

T

Rp =

1X It (pt ). T t=1

The complete diagnostic sequence is therefore Rg → Ra → Ru → Rp . Baseline tables report the first three stages and use “–” for Rp .

24

arXiv Preprint

Algorithm 1 Gradient–Update Alignment (GUA) Require: Current parameters θt , optimizer state st , losses {Li }m i=1 , base gradient surgery method, base optimizer Ot , learning rate ηt Ensure: Updated parameters θt+1 and optimizer state st+1 1: Compute loss-specific gradients {gi,t }m i=1 2: Construct the pre-optimizer direction at using the base gradient surgery method 3: Obtain the optimizer proposal: (ut , s̄t+1 ) ← Ot (at , st ) 4: Direction alignment: project ut onto the conflict-free cone: pt ← arg mind∈Ct 21 ∥d − ut ∥2Mt 5: Apply the aligned update: θt+1 ← θt − ηt pt 6: State alignment: if the projection modifies the proposal, compute s̃t+1 ← Φt (pt , ut , s̄t+1 ) and set st+1 by Eq. (6); otherwise set st+1 ← s̄t+1 7: return (θt+1 , st+1 )

B.7

GUA I MPLEMENTATION D ETAILS

This section provides the implementation details corresponding to Algorithm 1. B.7.1

P ROJECTION S OLVER

Let Gt = [g1,t , . . . , gm,t ] collect the loss-specific gradients as flattened vectors, and let Ct = {d : G⊤ t d ≥ 0} denote the current conflict-free cone. Given the optimizer proposal ut , GUA computes a reference update pt ∈ Ct that stays close to ut under a chosen projection metric: 1 pt = arg min ∥d − ut ∥2Mt d 2

s.t.

G⊤ t d ≥ 0.

The projection is performed in update space: the cone Ct is induced by the current loss-specific gradients, and GUA projects the optimizer proposal after optimizer transformation rather than the parameters themselves. The Euclidean metric sets Mt = I, yielding the standard least-change projection in the original parameter space: 1 pEuc = arg min ∥d − ut ∥22 t d 2

s.t.

G⊤ t d ≥ 0.

The Adam metric instead measures the projection distance in the geometry induced by Adam’s second-moment state. Let v̄ˆt+1 be the bias-corrected tentative second-moment state associated with ut . We use a diagonal metric q Mt = diag( v̄ˆt+1 + ϵ), ∥x∥2Mt = x⊤ Mt x. Thus, 1 pAdam = arg min ∥d − ut ∥2Mt s.t. G⊤ t d ≥ 0. t d 2 In practice, Mt can be chosen at different levels of structure. The identity metric is optimizeragnostic and gives the smallest implementation surface. A diagonal metric, such as the Adam metric above, is optimizer-guided while preserving the O(m2 p) Gram-matrix cost and avoiding any p × p storage. Block-diagonal or full preconditioner metrics could be used when an optimizer exposes such structure, but they require applying Mt−1 to loss-gradient vectors and may increase both memory and implementation cost. For this reason, our experiments restrict Mt to the two practically useful choices I and diagonal Adam metric. Official GUA experiments use the Adam metric by default for both two-loss and three-loss decompositions. The projection is performed on the flattened full-network parameter vector, not separately per layer, because the conflict-free cone is defined by global inner products with loss gradients; layer-wise projection would instead impose separate cones on individual layers and change the intended update-level geometry. Before solving, we normalize each nonzero constraint gradient and the optimizer proposal for numerical stability, and restore the proposal scale after projection. This positive rescaling leaves the conflict-free cone unchanged, so the constraints remain induced by the same loss-specific gradients computed by the base gradient surgery method. 25

arXiv Preprint

The projection can be solved entirely in the m-dimensional dual, where m is the number of losses. In our PINN experiments, m ∈ {2, 3}, so we solve this dual problem exactly by enumerating active sets. Define −1 b = G⊤ K = G⊤ t ut , t Mt Gt . If b ≥ 0, the optimizer proposal is already conflict-free and we return pt = ut . Otherwise, we enumerate each active constraint set S ⊆ {1, . . . , m}. For each active set, we solve the small KKT system KSS λS = −bS , set inactive multipliers to zero, and check the KKT feasibility conditions λS ≥ 0, Each feasible active set yields

b + Kλ ≥ 0.

pt (λ) = ut + Mt−1 Gt λ.

Among feasible active sets, we choose the one with the smallest projection objective. This enumeration is used only for the small-m PINN setting. In the Euclidean case Mt = I, this reduces to K = G⊤ t Gt and pt = ut +Gt λ. In the Adam-metric case, Mt is the positive diagonal metric defined above. Since m ∈ {2, 3} in our PINN experiments, enumerating all active sets is inexpensive and gives the exact projection for the chosen metric. The projection does not form or store any p × p matrix. Its parameter-dimensional cost is dominated −1 2 by constructing the Gram matrix K = G⊤ t Mt Gt , which costs O(m p) for diagonal Mt after the loss-specific gradients have been computed. Active-set enumeration costs O(2m m3 ) in the worst case for solving the small dual systems, so it is appropriate only when m is small. For CelebA settings with m > 8, we instead solve the same projection as the low-dimensional nonnegative dual quadratic program 1 min λ⊤ Kλ + b⊤ λ, λ≥0 2 using an active-set solver over the m-dimensional dual variable. This avoids enumerating 2m subsets while keeping the large parameter dimension outside the QP. The large parameter dimension enters only through gradient inner products, and the remaining optimization is over the number of losses.

26

arXiv Preprint

B.7.2

O PTIMIZER -S TATE A LIGNMENT

The state-alignment rule is defined in Eq. (6). Here we detail its Adam instantiation, as Adam is used by the gradient-surgery baselines in their standard settings and admits a simple state reconstruction through its first- and second-moment statistics. Let m̄t+1 and v̄t+1 denote the tentative Adam first- and second-moment states associated with the optimizer proposal ut . Their bias-corrected versions are ˆ t+1 = m̄

m̄t+1 , 1 − β1t+1

v̄ˆt+1 =

v̄t+1 . 1 − β2t+1

Given the applied update pt , we construct the pseudo-gradient q corr gt+1 = ( v̄ˆt+1 + ϵ) ⊙ pt , which would produce pt under the current Adam denominator. The target Adam states are corr ⊙2 ṽt+1 = (1 − β2t+1 )(gt+1 ) .

corr m̃t+1 = (1 − β1t+1 )gt+1 ,

Finally, GUA softly aligns the stored Adam moments: mt+1 = (1 − ρm )m̄t+1 + ρm m̃t+1 ,

vt+1 = (1 − ρv )v̄t+1 + ρv ṽt+1 .

Here, ρm and ρv control the alignment strengths of the first and second moments, respectively. The values used in our experiments are reported in Table 6 and are selected based on validation performance. Table 6: Coefficients used for optimizer-state alignment. Equation ρm ρv Burgers Schrödinger Kovasznay Heat-MS Poisson-5D Beltrami

0.10 0.10 0.70 0.70 0.50 0.70

0.03 0.03 0.20 0.20 0.15 0.20

Notably, the state-alignment principle is not restricted to Adam. It can be extended to other stateful optimizers when target internal states can be reconstructed from the applied update according to the corresponding update rules. The corresponding implementations for different optimizers and their results are reported in Appendix C.5.

27

arXiv Preprint

C

A DDITIONAL R ESULTS , A BLATIONS , AND D IAGNOSTICS

C.1

P ROJECTION M ETRIC A BLATION

As defined in Eq. (5), GUA projects the optimizer proposal ut onto the conflict-free cone Ct under the metric Mt . We compare two choices of Mt : the Euclidean metric Mt = I and the Adam metric defined by Adam’s diagonal second-moment statistics. The Euclidean metric treats all parameter coordinates uniformly, whereas the Adam metric measures the projection distance in the optimizer-induced coordinate geometry. Figure 8 compares their accuracy and numerical stability across PINN settings. A hatched bar denotes partial numerical instability, while a cross denotes complete numerical failure. Overall, the Adam metric provides accuracy comparable to or better than the Euclidean metric across the evaluated settings. The difference is modest for some equations, such as Schrödinger and Beltrami flow, but becomes more pronounced for Burgers, Kovasznay flow, and Poisson-5D. The clearest distinction appears on Heat-MS: under the Euclidean metric, the two-loss setting exhibits partial numerical instability and the three-loss setting becomes entirely non-finite, whereas the Adam metric remains numerically stable in both cases. These results indicate that the projection metric affects more than the feasibility of the corrected update. Both choices enforce the same conflict-free constraints, but they select different feasible updates by measuring the deviation from ut in different geometries. The Euclidean metric ignores the coordinate-wise scaling already induced by Adam, whereas the Adam metric accounts for this anisotropy through the optimizer’s second-moment statistics. The improved stability observed on Heat-MS is therefore consistent with preserving the optimizer-induced geometry during projection. Since the Adam metric is at least competitive in accuracy and provides better numerical robustness in the difficult settings, we use it as the default projection metric in the main experiments.

10

−3

10

−5

Adam metric

Partial instability

Kovasznay Burgers Schrödinger Heat-MS Relative L2 error

Relative L2 error

−1

3-loss

2-loss

Euclidean 10

10

Beltrami Poisson-5D

0

10

−1

10

−2

10

−3

Burgers Schrödinger Heat-MS

Beltrami

Figure 8: Euclidean and Adam projection metrics across PINN equations. Hatching marks partial numerical instability, and a × marks complete numerical failure.

28

arXiv Preprint

C.2

S TATE -A LIGNMENT S ENSITIVITY

State alignment introduces two coefficients, ρm and ρv , controlling how strongly the Adam moments are moved toward the target state. We examine whether GUA is sensitive to these coefficients using seven predefined pairs, (ρm , ρv ) ∈ {(0.003, 0.001), (0.01, 0.003), (0.03, 0.01), (0.1, 0.03), (0.3, 0.1), (0.5, 0.15), (0.7, 0.2)}, across all PINN settings, six gradient surgery methods, and five seeds. The coefficient pair used in the main experiments is selected using validation performance, as described in Appendix B.7.2, and shared across loss decompositions and gradient surgery methods for each equation. The test set is not used for selection. Figure 9 reports sensitivity over the full coefficient grid using the geometric mean of paired relative-error ratios to the corresponding baseline, where values below one indicate improvement. Validation-selected coefficients are highlighted, and the single numerical failure is retained. Overall, GUA is broadly robust to the state-alignment coefficients, with relative-error ratios below one across most equations, methods, and coefficient pairs. Burgers, Beltrami flow, and Poisson-5D are particularly robust. Sensitivity varies across equations: Schrödinger and Burgers favor moderate alignment, Heat-MS tends to benefit from stronger alignment, while Kovasznay flow and Poisson-5D show greater method-dependent variation. PCGrad

CAGrad

IMTL-G

A-MTL

UPGrad

ConFIG

Baseline (1.0)

GUA / baseline error ratio

A : (0.003, 0.001) B : (0.01, 0.003) C : (0.03, 0.01) D : (0.10, 0.03) E : (0.30, 0.10) F : (0.50, 0.15) G : (0.70, 0.20)

Kovasznay flow-2

Burgers-2

Schrödinger-2 1.0 0.5

1.0

1.0

1.0

0.5

0.5

0.2

0.2

Heat-MS-2

0.2 0.5 0.2

A B C D E F G

A B C D E F G

A B C D E F G

Beltrami-2

Poisson-5D-2

A B C D E F G

1.0 1.0 0.5

0.5 0.2

GUA / baseline error ratio

0.2

Burgers-3

A B C D E F G

A B C D E F G

Schrödinger-3

Heat-MS-3

1.0 0.5

1.0

0.2

0.5

0.5

0.2

0.2

Beltrami-3 1.0

1.0

0.5

0.2 A B C D E F G

A B C D E F G

A B C D E F G

A B C D E F G

Figure 9: State-alignment sensitivity across PINN equations and gradient surgery methods. Shaded bands show standard deviation, the dashed line marks baseline parity, orange bands identify the selected coefficients, and a black cross marks numerical failure.

29

arXiv Preprint

C.3

O PTIMIZER -I NDUCED GUM R ESULTS ON KOVASZNAY, B ELTRAMI , AND P OISSON -5D

Owing to space limitations, the main text presents Q1 optimizer-induced GUM results for a subset of the PINN benchmarks, with additional results provided in the appendix. Table 7 reports the corresponding results for Kovasznay, Beltrami, and Poisson-5D under the same experimental protocol. Consistent with the main-text results, ConFIG yields Ra = 0, while momentum, adaptive, and curvature-based optimizers can produce nonzero post-optimizer conflict rates Ru . Plain SGD preserves Ru = 0, as expected from ut = at . Table 7: Optimizer-induced GUM across the remaining PINN settings. The sequence Rg → Ra → Ru tracks conflict rates (↓) from raw loss-specific gradients through gradient surgery to optimizer transformation. Kovasznay Optimizer

Beltrami

2-loss

2-loss

3-loss Ru

2-loss

Rg

Ra

Ru

Rg

Rg

Ra

Ru

Rg

Ra

Ru

100.0 99.9 10.0 5.5 8.9

0.0 0.0 0.0 0.0 0.0

0.0 53.3 1.7 6.4 8.4

First-order optimizers 100.0 0.0 0.0 100.0 79.0 0.0 22.9 90.3 3.3 0.0 0.0 32.5 13.1 0.0 11.6 27.5 14.6 0.0 10.7 29.0

0.0 0.0 0.0 0.0 0.0

0.0 43.2 0.9 16.8 17.4

87.9 64.9 81.9 52.6 52.9

0.0 0.0 0.0 0.0 0.0

0.0 41.9 0.2 30.5 32.6

SophiaG 55.6 AdaHessian 99.9 SOAP 3.3

0.0 0.0 0.0

42.9 36.8 5.8

Second-order optimizers 54.1 0.0 20.2 63.0 99.9 0.0 53.2 99.9 20.9 0.0 8.8 28.6

0.0 0.0 0.0

29.4 57.7 12.3

27.2 98.8 44.5

0.0 0.0 0.0

13.0 8.5 30.8

SGD M-SGD RMSProp Adam AdamW

Ra

Poisson-5D

30

arXiv Preprint

C.4

F ULL PINN R ESULTS

Given the space limitations of the main text, we summarize Q2 there using configuration-level ranges and medians. This section reports the complete Q2 results for each evaluated method–benchmark configuration, including relative L2 errors and conflict-rate transitions over the four training stages. For the six gradient-surgery methods (excluding the Adam control), the summary contains 36 twoloss and 24 three-loss method–benchmark configurations. The corresponding medians reported in the main text are medians over these configuration-level reductions. C.4.1

Q UANTITATIVE PINN R ESULTS FOR Q2

Table 8 reports the complete relative L2 errors. Across all evaluated base methods and PINN settings, adding GUA reduces the mean error, consistent with the aggregate results reported in the main text. Adam+GUA also improves over Adam in all ten settings despite using no pre-optimizer gradient surgery. Table 8: Complete Q2 relative L2 errors (mean ± sample standard deviation over five seeds). Here, E denotes the relative L2 error defined in Appendix B.6. “Imp.” denotes 100(1 − ĒGUA /Ēbase ). Schrödinger

Method 2-loss

Imp.

3-loss

Burgers Imp.

2-loss

Imp.

Kovasznay 3-loss

Imp.

2-loss

Imp.

Adam 3.15e-2±1.10e-2 – 3.15e-2±1.10e-2 – 4.55e-3±1.03e-3 – 3.07e-3±1.37e-3 – 4.69e-4±7.44e-5 – +GUA 1.74e-2±4.19e-3 44.8 2.42e-2±6.59e-3 23.2 1.18e-3±5.86e-4 74.1 7.77e-4±3.35e-4 74.7 3.46e-4±1.53e-5 26.2 PCGrad 2.51e-2±7.78e-3 – 2.12e-2±5.21e-3 – 3.05e-3±7.31e-4 – 2.02e-3±8.11e-4 – 4.82e-4±1.02e-4 – +GUA 1.46e-2±1.52e-3 41.8 1.01e-2±8.11e-4 52.4 8.35e-4±1.32e-4 72.6 7.61e-4±1.81e-4 62.3 3.26e-4±7.42e-5 32.4 CAGrad 1.40e-2±1.53e-3 – 2.00e-2±5.06e-3 – 1.70e-3±4.36e-4 – 2.25e-3±4.90e-4 – 4.78e-4±4.76e-5 – +GUA 1.18e-2±8.77e-4 15.7 1.08e-2±1.92e-3 46.0 6.97e-4±1.26e-4 59.0 7.39e-4±5.89e-5 67.2 1.11e-4±3.12e-5 76.8 IMTL-G 4.12e-2±1.09e-2 – 5.60e-1±3.12e-1 – 2.51e-3±8.24e-4 – 5.83e-2±3.66e-2 – 1.79e-4±2.83e-5 – +GUA 3.21e-2±1.86e-3 22.1 1.02e-1±5.91e-2 81.8 6.82e-4±6.76e-5 72.8 5.14e-3±1.07e-3 91.2 9.25e-5±1.03e-5 48.3 A-MTL 1.60e-2±1.87e-3 – 3.00e-2±3.78e-3 – 2.95e-3±5.24e-4 – 7.31e-3±2.89e-3 – 1.83e-4±1.56e-5 – +GUA 1.23e-2±2.93e-3 23.1 2.15e-2±4.18e-3 28.3 1.05e-3±2.72e-4 64.4 1.56e-3±3.34e-4 78.7 1.56e-4±2.17e-5 14.8 UPGrad 2.56e-2±7.20e-3 – 1.98e-2±4.10e-3 – 3.03e-3±4.96e-4 – 3.08e-3±1.46e-3 – 4.49e-4±6.56e-5 – +GUA 1.52e-2±2.12e-3 40.6 1.10e-2±1.10e-3 44.4 7.68e-4±1.12e-4 74.7 9.69e-4±1.80e-4 68.5 3.23e-4±7.83e-5 28.1 ConFIG 1.89e-2±5.46e-3 – 2.33e-2±7.89e-3 – 1.74e-3±3.38e-4 – 3.48e-3±8.30e-4 – 1.86e-4±4.02e-5 – +GUA 1.38e-2±2.35e-3 27.0 1.33e-2±1.69e-3 42.9 6.50e-4±1.27e-4 62.6 1.34e-3±1.31e-4 61.5 9.84e-5±1.90e-5 47.1 Heat-MS

Method 2-loss

Imp.

Beltrami 3-loss

Imp.

2-loss

Imp.

Poisson-5D 3-loss

Imp.

2-loss

Imp.

Adam 1.73e-2±4.58e-3 – 1.57e-1±6.51e-2 – 6.50e-3±3.14e-4 – 5.26e-3±2.84e-4 – 5.95e-4±3.22e-5 – +GUA 1.46e-2±1.35e-3 15.6 3.47e-2±7.15e-3 77.9 5.06e-3±1.76e-4 22.2 3.90e-3±2.23e-4 25.9 3.94e-4±3.25e-5 33.8 PCGrad 2.05e-2±4.36e-3 – 3.40e-1±1.34e-1 – 6.70e-3±4.75e-4 – 5.26e-3±2.96e-4 – 5.88e-4±4.29e-5 – +GUA 1.35e-2±4.40e-3 34.1 4.94e-2±1.43e-2 85.5 5.22e-3±3.12e-4 22.1 4.18e-3±3.07e-4 20.5 3.38e-4±2.34e-5 42.5 CAGrad 3.19e-2±8.27e-3 – 6.26e-1±4.33e-2 – 4.13e-3±2.69e-4 – 3.91e-3±1.78e-4 – 4.28e-4±3.02e-4 – +GUA 1.13e-2±3.08e-3 64.6 1.46e-1±4.81e-2 76.7 3.10e-3±1.68e-4 24.9 2.96e-3±9.99e-5 24.3 1.37e-4±2.07e-5 68.0 IMTL-G 8.82e-1±2.26e-1 – 1.00e+0±5.56e-4 – 3.83e-3±2.80e-4 – 6.58e-3±1.73e-3 – 4.94e-4±1.94e-4 – +GUA 1.63e-2±3.28e-3 98.2 3.26e-1±1.53e-1 67.4 2.92e-3±1.76e-4 23.8 3.19e-3±2.29e-4 51.5 1.44e-4±9.57e-6 70.9 A-MTL 7.61e-2±3.73e-2 – 7.71e-1±3.92e-2 – 4.06e-3±2.26e-4 – 3.19e-3±2.05e-4 – 5.16e-4±2.59e-4 – +GUA 2.31e-2±7.88e-3 69.6 1.47e-1±5.10e-2 80.9 2.99e-3±2.01e-4 26.4 2.82e-3±1.18e-4 11.6 2.93e-4±1.93e-4 43.2 UPGrad 2.05e-2±4.71e-3 – 3.20e-1±1.93e-1 – 6.73e-3±3.70e-4 – 5.15e-3±2.98e-4 – 5.93e-4±3.44e-5 – +GUA 1.59e-2±1.36e-3 22.4 4.62e-2±8.30e-3 85.6 5.14e-3±3.32e-4 23.6 4.26e-3±1.79e-4 17.3 3.52e-4±2.73e-5 40.6 ConFIG 5.77e-2±1.31e-2 – 8.16e-1±2.98e-2 – 4.01e-3±2.75e-4 – 4.11e-3±1.45e-4 – 3.16e-4±2.94e-4 – +GUA 9.85e-3±1.56e-3 82.9 1.45e-1±9.18e-2 82.2 2.92e-3±1.80e-4 27.2 2.82e-3±1.82e-4 31.4 1.40e-4±2.76e-5 55.7

31

arXiv Preprint

C.4.2

C ONFLICT-R ATE PINN R ESULTS FOR Q2

Table 9 reports the corresponding Rg → Ra → Ru → Rp transitions. Unlike Q1, which fixes ConFIG and varies the optimizer, these experiments fix Adam and vary the gradient surgery method. The resulting transitions show that GUM is not specific to a particular pre-optimizer construction. Different methods produce different Ra and exhibit different responses to the same optimizer transformation. When Ra = 0, any nonzero Ru directly indicates strict GUM. When residual pre-optimizer conflicts remain, an increase from Ra to Ru indicates generalized GUM, showing that Adam further increases the frequency of conflicting proposals. Despite these differences across gradient surgery methods, GUA yields Rp = 0 in every evaluated setting, so the applied update is conflict-free after alignment. These transitions show that update-level alignment remains effective across different pre-optimizer constructions and connect the observed accuracy gains to the intended mechanism. Table 9: Four-stage conflict rates (%) for the PINN Q2 experiments, averaged over five seeds. Schrödinger Method

2-loss Rg

Ra Ru

Burgers 3-loss Ra Ru

Rp

Rg

Ra Ru

Rp

Rg

Ra Ru

Rp

28.5 20.0 22.6 – 20.0 14.2 11.6 0.0

55.4 36.1 34.0 – 35.9 23.2 15.9 0.0

3.0 0.9

1.3 0.3

1.2 0.3

– 0.0

PCGrad 36.3 0.0 +GUA 32.3 0.0

– 0.0

51.9 0.3 16.0 – 49.6 0.4 16.2 0.0

35.1 0.0 31.6 – 33.8 0.0 29.9 0.0

56.6 4.7 45.0 – 63.8 4.8 51.7 0.0

2.2 1.1

0.0 0.0

0.6 0.2

– 0.0

CAGrad 35.0 5.1 15.4 – +GUA 29.5 1.2 7.1 0.0

43.7 7.4 23.0 – 41.9 3.5 15.8 0.0

31.1 3.8 27.1 – 29.0 3.8 21.6 0.0

56.1 12.4 42.5 – 63.2 16.3 43.0 0.0

6.1 1.9

1.4 0.1

7.9 6.6

– 0.0

IMTL-G 67.8 0.0 33.2 – +GUA 52.8 0.0 22.5 0.0

94.4 16.7 53.4 – 86.8 2.3 44.2 0.0

41.9 0.0 23.7 – 33.5 0.0 15.7 0.0

94.5 1.7 55.6 – 84.3 3.1 49.0 0.0

3.6 0.9

0.0 0.0

4.8 2.9

– 0.0

A-MTL 33.6 0.2 18.3 – +GUA 27.2 0.1 15.9 0.0

57.7 4.5 37.5 – 44.5 0.6 29.4 0.0

27.5 0.8 34.2 – 30.2 1.1 35.7 0.0

66.8 9.4 57.2 – 63.7 4.9 57.4 0.0

5.0 2.2

0.1 11.5 – 0.0 9.5 0.0

UPGrad 35.7 0.4 +GUA 31.6 0.4

– 0.0

50.5 3.8 13.2 – 50.4 5.2 14.8 0.0

33.5 4.7 29.0 – 32.6 7.4 28.2 0.0

54.0 8.8 41.1 – 60.3 18.5 47.4 0.0

2.3 1.1

0.0 0.0

0.7 0.2

– 0.0

ConFIG 40.8 0.0 23.7 – +GUA 33.8 0.0 13.9 0.0

56.3 0.0 35.2 – 50.7 0.0 27.6 0.0

30.5 0.0 29.4 – 30.0 0.0 24.2 0.0

73.7 0.0 57.4 – 73.0 0.0 55.3 0.0

5.5 1.5

0.0 0.0

6.4 6.1

– 0.0

Heat-MS Method

2-loss Rg

Ra Ru

Rp

Rg

Ra Ru

2-loss

65.2 38.2 35.2 – 61.5 35.6 42.1 0.0

6.2 4.8

Rg

Kovasznay 3-loss

Adam 44.3 15.9 13.2 – +GUA 36.4 13.1 13.6 0.0 6.3 5.1

Rp

2-loss

Beltrami 3-loss Ra Ru

2-loss

Poisson-5D 3-loss

2-loss

Rp

Rg

Rp

Rg

Ra Ru

Rp

Rg

Ra Ru

Rp

Rg

Ra Ru

Rp

Adam 14.0 7.5 +GUA 17.6 6.8

4.8 2.4

– 0.0

72.8 40.1 54.1 – 32.9 17.7 19.9 0.0

1.6 2.0

0.2 0.2

0.2 0.8

– 0.0

4.2 3.6

0.6 0.4

0.5 1.6

– 0.0

41.3 7.2 41.3 4.5

3.8 3.7

– 0.0

PCGrad 14.9 0.0 +GUA 14.6 0.0

5.1 2.2

– 0.0

77.6 6.2 57.7 – 34.6 0.7 17.9 0.0

1.2 1.7

0.0 0.0

0.2 0.6

– 0.0

3.6 3.6

0.1 0.0

0.3 1.1

– 0.0

24.1 0.0 42.4 0.0

3.8 4.3

– 0.0

CAGrad 20.4 1.7 14.3 – +GUA 20.1 1.6 14.2 0.0

78.9 21.8 60.2 – 37.8 2.8 21.0 0.0

5.8 7.9

0.3 0.1

4.9 5.6

– 0.0

20.2 0.9 10.8 – 13.6 0.1 6.7 0.0

48.5 5.1 22.9 – 35.9 0.2 14.0 0.0

IMTL-G 53.7 0.0 30.6 – +GUA 28.9 0.0 13.4 0.0

88.5 5.4 18.7 – 60.9 4.1 36.3 0.0

5.1 4.1

0.0 0.0

5.9 4.4

– 0.0

89.5 0.0 28.1 – 24.5 0.1 10.7 0.0

51.0 0.0 33.8 – 44.1 0.0 26.2 0.0

A-MTL 23.7 0.1 17.7 – +GUA 26.2 0.3 11.8 0.0

75.4 15.0 64.1 – 55.8 3.5 27.9 0.0

14.5 0.0 14.8 – 10.2 0.0 9.7 0.0

28.2 0.0 19.9 – 14.3 0.0 13.0 0.0

48.4 0.4 30.8 – 42.3 0.0 25.0 0.0

UPGrad 13.1 0.4 +GUA 15.4 0.4

– 0.0

78.3 12.3 55.5 – 33.8 4.8 16.1 0.0

1.2 1.8

3.3 4.5

23.7 0.0 42.2 0.0

ConFIG 22.8 0.0 12.8 – +GUA 27.7 0.0 9.7 0.0

81.3 0.0 64.6 – 48.6 0.0 26.6 0.0

13.1 0.0 11.6 – 7.4 0.0 7.6 0.0

4.7 2.4

0.0 0.0

32

0.2 0.7

– 0.0

0.0 0.0

0.3 1.2

– 0.0

27.5 0.0 16.8 – 12.8 0.0 8.8 0.0

3.9 4.2

– 0.0

52.6 0.0 30.5 – 44.1 0.0 22.3 0.0

arXiv Preprint

C.5

P ERFORMANCE ACROSS O PTIMIZERS

The main experiments use Adam as the default optimizer. We further evaluate GUA with three representative optimizers from Q1: RMSProp for coordinate-wise adaptive scaling, AdamW for historical state, adaptive scaling, and decoupled weight decay, and SOAP for second-order preconditioning. SOAP is particularly relevant because its preconditioning has been shown to promote gradient alignment and mitigate conflicts in PINNs Wang et al. (2025). For each optimizer O, we compare ConFIG + O with ConFIG + O + GUA. Since these optimizers differ in transformation geometry and internal state, we adapt the projection metric and State Alignment strategy accordingly. At each step, the optimizer produces a proposal ut , which GUA projects onto the conflict-free cone under the time-dependent metric Mt : min dt

1 (dt − ut )⊤ Mt (dt − ut ) 2

s.t.

G⊤ t dt ≥ 0.

We use the second-moment geometry induced √ by each optimizer rather than√a common Euclidean metric. Specifically, MRMSProp,t ≈ diag( vt + ϵ), MAdamW,t ≈ diag( v̂t + ϵ). For SOAP, the proposal and task √ gradients are represented in the current Shampoo coordinate system, where MSOAP,t ≈ diag( ṽt + ϵ), and the projected update is then mapped back to parameter space. The −1 −1 dual solver uses Kt = G⊤ t Mt Gt and reconstructs pt = ut + Mt Gt λt . After applying the projected update, GUA softly aligns the optimizer state toward targets reconstructed from the applied update. RMSProp aligns its second-moment state with (ρm , ρv ) = (0, 0.5). AdamW aligns its first- and second-moment states with (0.1, 0.03) while handling decoupled weight decay separately. SOAP aligns the exponential moving averages that determine its update with (0.03, 0.003), while keeping the Shampoo preconditioner and coordinate basis unchanged. State Alignment is applied only when GUA modifies the optimizer proposal. RMSProp uses a learning rate of 10−4 /10−4 , whereas AdamW and SOAP use 10−3 /10−4 . As shown in Figure 10, GUA improves the displayed settings across optimizer families. For Burgers, it reduces the relative L2 error of RMSProp, AdamW, and SOAP by 52.35%, 65.19%, and 13.78% in the two-loss setting, and by 58.39%, 62.92%, and 42.73% in the three-loss setting. For Schrödinger, the corresponding reductions are 22.98%, 40.00%, and 1.70% in the two-loss setting, and 47.50%, 43.66%, and 4.77% in the three-loss setting, respectively. Notably, GUA further improves SOAP even though its second-order preconditioning already promotes gradient alignment. This suggests that explicit update-level alignment can provide complementary benefits beyond the alignment induced by the optimizer’s preconditioning geometry. Together with the Adam results in Q2, these results suggest that GUA extends beyond Adam to optimizers involving adaptive scaling, decoupled weight decay, and preconditioning. 2-loss ×10⁻³

2.0

Relative L2 error

Burgers

1.0 0.0

Adam

×10⁻²

4.0

2.0

2.0

1.0

0.0

×10⁻⁴

RMSProp

0.0

×10⁻¹

8.0

2.0

×10⁻³

0.0

AdamW

×10⁻²

0.0

0.0

8.0

SOAP

RMSProp

0.0

AdamW

0.0

×10⁻³

×10⁻⁴

6.0

4.0

5.0

2.0

3.0

2.0

2.5

0.0

Adam

0.0

×10⁻⁴

2.0

SOAP

×10⁻²

4.0

4.0

4.0

2.0

3-loss ×10⁻³

×10⁻³

4.0

4.0

Adam

3.0

+GUA

×10⁻⁴

1.5

Schrodinger 1.0

Baseline

0.0

Adam

RMSProp

0.0

×10⁻¹

5.0

3.0

2.5

RMSProp

0.0

×10⁻²

6.0

0.0

AdamW

0.0

SOAP

×10⁻³

8.0

4.0

AdamW

0.0

SOAP

Figure 10: Relative L2 error across optimizer families for the Burgers and Schrödinger equations.

33

arXiv Preprint

C.6 C.6.1

M ECHANISM D IAGNOSTICS S AME -S TATE N ORM -M ATCHED C OMPARISON

Table 10 reports the per-benchmark results for the same-state, norm-matched comparison introduced in Q3. At each evaluated step, the parameters, optimizer state, batch, and learning rate are held fixed while comparing the optimizer proposal ut , the aligned update pt , and the norm-matched control qt =

∥pt ∥2 ut . ∥ut ∥2

All reported samples satisfy pt ̸= 0. We report the pairwise win rates Pr[Wtrel (pt ) < Wtrel (ut )] and Pr[Wtrel (pt ) < Wtrel (qt )]. Training states are first aggregated within each run before statistics are computed across seeds. Table 10: Per-benchmark Q3 win rates under two- and three-loss decompositions. 2-loss 3-loss Equation pt < ut (%) pt < qt (%) pt < ut (%) pt < qt (%)

C.6.2

Schrödinger Burgers Heat-MS Beltrami Kovasznay Poisson-5D

95.81 96.55 97.92 94.77 98.25 89.94

92.68 95.17 96.54 93.38 97.54 85.77

96.26 94.57 93.16 94.05 – –

93.53 93.55 92.47 93.00 – –

Overall

95.54

93.51

94.51

93.14

R EPRESENTATIVE G RADIENT-D IRECTION S LICES

To make the update-level geometry concrete, Figure 11 visualizes representative two-loss training steps across the six PDE benchmarks. For ease of visualization, we rotate each directional slice so that one loss-specific gradient, g1,t , is fixed along the positive horizontal axis. Specifically, for each full-space vector x, we orthogonally project it onto span(g1,t , g2,t ) and express the projection in the orthonormal basis e1 = g1,t /∥g1,t ∥2 and e2 = (g2,t − ⟨g2,t , e1 ⟩e1 )/∥g2,t − ⟨g2,t , e1 ⟩e1 ∥2 , choosing the sign of e2 so that g2,t has a nonnegative second coordinate. We then normalize the displayed vectors and therefore ignore their original magnitudes. The figure is intended to show only their relative directions and the resulting conflict-free geometry. Each slice shows the two loss-specific gradient directions, the constructed direction at , the optimizer proposal ut , and the aligned update pt . The shaded region denotes the corresponding conflict-free cone Ct . These examples illustrate how optimizer transformation can move a conflict-free constructed direction outside Ct , while GUA realigns the resulting proposal with the current conflict-free cone.

34

arXiv Preprint

g1 2ULJLQDOORVVHV

g2



$GDPu 6FKU|GLQJHUORVV

&RQ),*

g2





g1

 



g1





g2



g1







*8$

g2



u

 

FRQIOLFWIUHHFRQH 



 

*8$p $GDP



a







&RQ),*a

g2





g1

p

 











%XUJHUVORVV 2ULJLQDOORVVHV g2

 

 

g1

  

&RQ),* g2

a

g1









u









p g2



g1

 

*8$



g2



 

$GDP



g1

 















.RYDV]QD\ORVV 2ULJLQDOORVVHV

  

a

 

g2

&RQ),*

g1





g2

g1























$GDP

u







*8$

p

 

g2

g1



g2

g1

 













+HDW06ORVV 2ULJLQDOORVVHV

g2

 

g2



g1

  

&RQ),*





g1





 







$GDP

g2



a



u

g1

 







*8$

g2 p



g1

 















%HOWUDPLORVV 2ULJLQDOORVVHV g2







g1

 









g1





 







*8$ g2





g1



$GDP g2



a







&RQ),* g2

p



u 

g1















3RLVVRQ'ORVV 2ULJLQDOORVVHV

g2



&RQ),*

g2







g1

























g1







*8$

g2





g1



$GDP

g2



a

u 





p

g1

  



Figure 11: Representative gradient directions for two-loss PDEs.

35





arXiv Preprint

D

M ULTI - TASK L EARNING D ETAILS

We use CelebA Liu et al. (2015) as a controlled task-cardinality benchmark, following the FAMObased experimental setup adopted by ConFIG Liu et al. (2025). For an m-task experiment, we use the first m attributes in the official annotation order, so the task-count settings form nested subsets. Images are resized to 64 × 64, converted to tensors, and used without additional data augmentation. The model consists of a shared convolutional backbone with BatchNorm, ReLU, max pooling, adaptive average pooling, and two fully connected layers, followed by task-specific binary classification heads. Training minimizes binary cross-entropy independently for each attribute. The main training hyperparameters are summarized in Table 11. To remain consistent with the gradient-surgery baselines, which operate on the shared parameters, GUA is applied to the same shared parameters after taskgradient aggregation and optimizer proposal. The task-specific classification heads are updated by the native optimizer but are not modified by GUA. The data pipeline, model, and task losses remain unchanged. All reported results are averaged over three independent seeds {0, 1, 2}. Table 11: CelebA multi-task learning training configuration. Item

Setting

Available tasks

40 binary facial attributes; evaluated task counts {2, 3, 5, 10, 20, 30, 40} RGB images resized to 64 × 64; no augmentation Binary cross-entropy per task 15 256 Adam 3 × 10−4 Every epoch

Input Loss Epochs Batch size Optimizer Learning rate Evaluation frequency

36

Related documents

• Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks#821587
• Toward an artificial intelligence physicist for unsupervised learning#964486
• Latent-MoE: Domain-Aware Mixture-of-Experts for PDEs with Multi-Regime Physics#984708
• Efficient training of physics‐informed neural networks via importance sampling#19787
• Enhancing Physics-Informed Neural Networks with Domain-aware Fourier Features: Towards Improved Performance and Interpretable Results#1012140