Transferable Physics-Informed Representations via Closed-Form Head Adaptation Jian Cheng Wong∗ *, Isaac Yin Chung Lai† , Pao-Hsiung Chiu∗ , Chin Chun Ooi∗ , Abhishek Gupta‡ , Yew-Soon Ong∗§ ∗ Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore † National University of Singapore (NUS), Singapore ‡ Indian Institute of Technology Goa (IIT Goa), India
arXiv:2604.21761v1 [cs.LG] 23 Apr 2026
§ Nanyang Technological University (NTU), Singapore
Abstract—Physics-informed neural networks (PINNs) have garnered significant interest for their potential in solving partial differential equations (PDEs) that govern a wide range of physical phenomena. By incorporating physical laws into the learning process, PINN models have demonstrated the ability to learn physical outcomes reasonably well. However, current PINN approaches struggle to predict or solve new PDEs effectively when there is a lack of training examples, indicating they do not generalize well to unseen problem instances. In this paper, we present a transferable learning approach for PINNs premised on a fast Pseudoinverse PINN framework (Pi-PINN). Pi-PINN learns a transferable physics-informed representation in a shared embedding space and enables rapid solving of both known and unknown PDE instances via closed-form head adaptation using a least-squares-optimal pseudoinverse under PDE constraints. We further investigate the synergies between data-driven multitask learning loss and physics-informed loss, providing insights into the design of more performant PINNs. We demonstrate the effectiveness of Pi-PINN on various PDE problems, including Poisson’s equation, Helmholtz equation, and Burgers’ equation, achieving fast and accurate physics-informed solutions without requiring any data for unseen instances. Pi-PINN can produce predictions 100–1000 times faster than a typical PINN, while producing predictions with 10–100 times lower relative error than a typical data-driven model even with only two training samples. Overall, our findings highlight the potential of transferable representations with closed-form head adaptation to enhance the efficiency and generalization of PINNs across PDE families and scientific and engineering applications. Index Terms—physics-informed neural networks, transfer learning, multi-task learning, generalization
I. I NTRODUCTION Physics-informed neural networks (PINNs) have attracted significant attention as a neural approach for solving partial differential equations (PDEs) that govern a wide range of physical phenomena across science and engineering [1], [2]. PINNs incorporate governing laws directly into the learning objective by treating PDE residuals, and associated boundary conditions (BCs) and initial conditions (ICs), as training constraints, resulting in a “physics-informed” loss [3]–[6]. Despite their promise, two practical limitations have repeatedly emerged. First, PINNs often train slowly and can *Corresponding author: [email protected]
exhibit poor optimization behavior compared to purely datadriven networks, largely due to the stiffness and complex loss landscapes introduced by physics-informed constraints [7]– [9]. Second, even after expensive training, standard PINNs generalize poorly to new PDE instances (e.g., new coefficients, source terms, BCs/ICs, or parameter regimes), limiting their usefulness when extrapolation or rapid redeployment is needed [10]. As a result, PINNs are still commonly used in a single-instance setting where the physics-informed loss is enforced directly for the target PDE, while extending a trained model to new instances typically requires substantial re-training and re-tuning. In many real-world settings, however, one does not start from scratch: there may exist a small collection of related PDE instances with solutions (or partial labels) that can be leveraged to learn representations that transfer. This motivates a transfer-learning perspective for PINNs, with the objective to build PINN models that are more reusable and adaptable across PDE families and parameter regimes. Hence, in this paper, we present Pi-PINN, a fast Pseudoinverse PINN framework that learns transferable physicsinformed representations and enables closed-form head adaptation via a least-squares-optimal pseudoinverse under PDE constraints. The core idea is to decouple learning into (i) a shared embedding that captures transferable structure across related PDE instances and (ii) a task-specific output head that can be adapted efficiently through a closed-form linear solve, avoiding costly gradient-based re-optimization for each new instance. In addition, we study how combining data-driven multi-task learning losses with physics-informed residuals can produce more performant and transferable PINN models. The key contribution of our work are: (1) we introduce Pi-PINN, a pseudoinverse-based physics-informed learning framework that supports closed-form, least-squares-optimal head adaptation under PDE constraints, substantially reducing the computational cost of adapting PINNs to new PDE instances (see schematic in Figure 1). (2) We propose a representation-learning formulation for PINNs that learns transferable deep embeddings from related PDE instances, improving generalization across PDE families and parameter regimes. (3) We analyze the synergy between data-driven
multi-task learning objectives and physics-informed residual losses, providing practical insights for training PINNs that are both accurate and reusable. (4) Empirically, across various PDE problems including Poisson, Helmholtz, and Burgers equations, Pi-PINN achieves 100–1000× faster prediction/adaptation than conventional PINNs, while obtaining 10– 100× lower relative error than typical data-driven models in sparse-data regimes (e.g., 2–4 training samples). II. P SEUDOINVERSE PINN Without loss of generality, we consider the use of PINNs to model the outcomes u in any spatial-temporal domain x ∈ Ω, t ∈ (0, T ] by satisfying the following governing equations: N [u(x, t)] = h(x, t),
x ∈ Ω, t ∈ (0, T ]
(1a)
BC:
B[u(x, t)] = g(x, t),
x ∈ ∂Ω, t ∈ (0, T ]
(1b)
IC:
u(x, t = 0) = u0 (x),
x∈Ω
(1c)
PDE:
where N [u(x, t)] is a general differential operator which can include linear and nonlinear combinations of temporal and spatial derivatives, and h(x, t) is an arbitrary source term. Eq. 1b and Eq. 1c specify the BC and IC of the physical system, where B[u(x, t)] is also a differential operator. To simplify the notation, we consider that the variation across different PDE instances (including variations in BC and IC) can be parameterized by a set of task parameters ϑ. Every new PDE instance leads to different physical outcomes, uϑ , spanning the entire spatial-temporal domain x ∈ Ω, t ∈ (0, T ]. A PINN model with sufficient capacity (number of nodes nnode , hidden layers L) can effectively learn a good approximation to the target output, uPINN (x, t; w), by minimizing the PINN loss function w.r.t. its network parameters w = (W 1 , b1 , ..., W L−1 , bL−1 , wL ). A. Pseudoinverse Physics-Informed [P i]2 Computation. There has been some prior work on transfer learning in the PINN context [11]–[13]. These typically involve the transfer of network weights from a similar, previously-solved physics scenario as a better initialization for PINN training on a new task, i.e., minimizing the physics-informed loss by penalizing any violation of the PDE, BC/IC constraints made by the PINN model outputs on the target PDE. Generally, only the final few layers of the networks are fine-tuned in transfer learning, facilitating the exchange of valuable features acquired when training the PINN model for the previous task. In the extreme, we can transfer the entire learned representation up to the final nonlinear hidden layer, i.e., xL , and only fine-tune the weight parameters in the linear output layer wL . In the absence of target labels for a new PDE instance, this can be done by means of physics-informed fine-tuning that purely minimizes violations of the PDE and BC/IC constraints by the T PINN output uPINN = wL xL for the target PDE instance. For linear PDEs, the process of fine-tuning the output weight parameters wL reduces to a linear least-squares computation. This can be observed by examining the PDE, BC/IC constraints of Eq. 1, where the terms N [uPINN (x, t; w)] − h(x, t),
uPINN (x, t = 0; w) − u0 (x), and B[uPINN (x, t; w)] − g(x, t), are all linear in terms of the output weights wL , as long as N [·] and B[·] comprise only linear differential operators (asis true for linear PDE systems) [14], [15]. These constraints form a system of linear equations and can be efficiently solved utilizing the Moore-Penrose pseudoinverse [16], giving a leastsquares-optimal solution to the unknown wL . Given a target PDE and a set of PDE (x(p) , t(p) ), BC (q) (q) (x , t ), and IC (x(r) , t(r) ) collocation points, the pseudoinverse computation solves the following weighted system of equations: λPDE h(x(p) , t(p) ) λPDE N [xTL (x(p) , t(p) )] .. .. . . λBC g(x(q) , t(q) ) λBC B[xT (x(q) , t(q) )] L wL = .. .. . . (r) T (r) (r) λIC u0 (x ) λIC xL (x , t ) .. .. . . X wL = y
(2)
where wL are the weight parameters in the output layer to be solved, xL is the transferable deep embeddings to be learned, and λPDE , λBC , and λIC are the weighting hyperparameters. It has been previously reported in PINN literature that the different physics loss terms, such as PDE and BC/IC constraints, can have vastly different magnitudes, necessitating proper reweighting during training [17]. Similarly, during the pseudoinverse physics-informed computation, the relative weighting of PDE and BC/IC constraints can be adjusted. Moreover, we can reformulate Eq. 2 into [λPI I + XT X] wL = XT y
(3)
with the regularization hyper-parameter λPI ≥ 0. This regularization can significantly improve the conditioning of the problem, thereby improving the quality of the obtained solution. Additionally, Eq. 3 reduces the matrix size for the inversion, hence it is more efficient to solve. Importantly, the pseudoinverse computation allows for rapid fine-tuning (in a single, fast computation) of weight parameters wL in the final layer of the PINN model for a broad range of linear PDE problems, eliminating the need for gradientbased parameter updates over potentially long and complex convergence trajectories as have been previously reported [8]. It’s also important to note that the pseudoinverse PINN approach can be seamlessly extended to handle nonlinear PDEs by using an iterative update process for the linearized version of the nonlinear constraints [18] (as demonstrated in the Experiments on the set of nonlinear Burgers’ equation). B. Pseudoinverse PINN Adapted From Data-Driven Neural Network: MLP+[P i]2 Given a set of training data uϑ1 , uϑ2 , ..., uϑK from K PDE instances ϑ1 , ϑ2 , ..., ϑK , a natural first approach is to train a purely data-driven neural network, e.g., a multilayer
Output
Shared embedding (transferable physics-informed representations) 𝐱𝐱2
…
𝐱𝐱𝐿𝐿−1
𝐱𝐱𝐿𝐿
Hidden layer
Solve new PDE 𝐱𝐱𝐿𝐿 ← concatenate 𝐱𝐱𝐿𝐿 , 𝐱𝐱𝐿𝐿−1 , … , 𝐱𝐱2
𝐱𝐱𝐿𝐿 = 𝑓𝑓 𝐳𝐳𝐿𝐿−1
𝐳𝐳𝐿𝐿−1 = 𝐖𝐖𝐿𝐿−1 𝐱𝐱𝐿𝐿−1 + 𝐛𝐛𝐿𝐿−1
Hidden layer
Frequency annealing
Hidden layer
Unseen PDE instances (test)
𝐮𝐮Pi−PINN = 𝐰𝐰𝐿𝐿 𝑇𝑇 𝐱𝐱𝐿𝐿
Pseudo-inverse physicsinformed computation to fine-tune 𝐰𝐰𝐿𝐿
Train shared embedding 𝐱𝐱2 = 𝑓𝑓 𝐳𝐳1
𝐳𝐳1 ← 𝐹𝐹𝐹𝐹𝐳𝐳1
multi-task / pseudoinverse-in-the-loop learning with labels
𝐳𝐳1 = 𝐖𝐖1 𝐱𝐱1 + 𝐛𝐛1 Input
Limited seen PDE instances (train)
𝐱𝐱1 ≡ 𝑥𝑥, 𝑡𝑡
Pi-PINN architecture Fig. 1. The Schematic diagram of the fast Pseudoinverse PINN framework, with the example PDE instances (Burgers’ equation).
perceptron (MLP), as the baseline model. We can define the mapping from the input variables (x, t, ϑ) to the output uMLP . The resulting learned MLP can then be used to make predictions for new PDE instances with a different ϑ. However, if the training data is sparse in ϑ, i.e., K is small, one may expect such predictions to have a very high chance of being inaccurate [19]. Naturally, the level of inaccuracy with respect to the actual K depends on the complexity of the underlying functions, as evident in the Experiments section. Given the proliferation of data-driven models in the past decade, an interesting question is whether we can take the deep embeddings xL learned from a data-driven MLP model and adapt it as a PINN model to solve new PDE instances more accurately, i.e., via the Pseudoinverse physics-informed computation. Crucially, this is a significant paradigm shift from the majority of previous related studies which have successfully demonstrated transfer learning purely between PINN models. This is significant as the training of data-driven models can be significantly less computationally expensive than current PINN methods, and we note the existence of some small set of training data in most real-world problems. We refer to such data-driven-adapted PINN model as MLP+[P i]2 . Although convenient for transferring prior knowledge from an existing data-driven-trained MLP on seen instances (via xL ) to benefit the prediction of unseen PDE instances, the MLP+[P i]2 model is not the most effective approach for learning physics-informed representations. While the optimal balance of width and depth can vary with problem, it is common practice in deep learning to enhance learning and generalizability by stacking more hidden layers in an MLP
model while maintaining a moderate number of nodes in each hidden layer (going deep rather than wide) [20]. When such an MLP model is used, there may be insufficient expressivity and/or degrees of freedom in the xL for adaptation. Hence, the MLP+[P i]2 model may not adequately satisfy the PDE, BC/IC constraints on the target PDE instance even with the best-fit weight parameters wL , resulting in inferior predictions. This is empirically observed in the comparison study results in the Experiments section. III. D EEP E MBEDDINGS N EURAL A RCHITECTURE AND L EARNING A LGORITHMS FOR P I -PINN The proper design of a neural architecture and learning algorithm for more performant Pi-PINN remains an open question worthy of more critical consideration. We hypothesize that a wide output layer xL is critical as this greatly affects the learnability of a shared embedding space for solving new PDE instances, which in turn determines how well the Pi-PINN can effectively be fine-tuned with the weight parameters wL in the output layer by the pseudoinverse computation. Effective learning of this shared embedding space is key to the transfer of prior knowledge to improve PINN performance, thereby ensuring predictions with good compliance to stipulated physics (encoded in the PDE, BC/IC constraints). With the above requirements in mind, we propose a neural architecture that is potentially more appropriate for the PiPINN. Figure 1 gives the Pi-PINN schematic. Essentially, we modify the base MLP by utilizing concatenative skip
connections such that all the nonlinear hidden layers are concatenated at the output layer: T uPi-PINN = wL xL
(Pi-PINN output layer)
(4a)
xL ← concatenate(xL , xL−1 , ..., x2 )
(4b)
xL = f (z L−1 ) .. .
(4c)
x2 = f (z 1 )
(after activation)
(4d)
z 1 ← F πz 1
(frequency annealing)
(4e)
(1st hidden layer)
(4f)
(input)
(4g)
z 1 = W 1 x1 + b 1 x1 ≡ (x, t)
This neural architecture design allows us to increase the output layer width by stacking multiple hidden layers while maintaining a moderate number of nodes in each hidden layer, thereby effectively improving the expressivity of the learned shared embedding and consequently, the prediction performance from data-driven-trained and physics-informed pseudoinverse computation. The proposed incorporation of a physics-informed loss computation during prediction also negates the need for explicit specification of the task parameter ϑ as input to the MLP for interpolation across different PDEs. It also encourages the learning of a generalizable embedding space for both seen and unseen PDE instances. In addition, we note that this proposed concatenation for a more expressive shared embedding bears similarity to the way one constructs a polynomial basis space such as the monomial basis space. The additional operations at each hidden layer are analogous to the recurrence relations used in generating Chebyshev polynomials, and incorporation and concatenation of more hidden layers in the MLP essentially results in the creation of a larger (albeit finite) and more expressive basis space (with less truncation). This presents an interesting direction for design of even more performant neural architectures for Pi-PINN computations. To more effectively learn the embedding space for highfrequency features, we use the sine activation for all the nonlinear hidden layers. In particular, we initialize the network with artificial high-frequency features through manipulating the first hidden layer with a factor of F π (Eq. 4e), and allowing these frequencies to be naturally “reduced” during training—a process we refer to as “frequency annealing” [21] . We propose two learning algorithms to train the shared embedding xL (Eq. 4b) from a small set of a priori-obtained instances, and compare their adaptation performance in the Experiments section. A. Multi-Task HYDRA+[P i]2
Learning
with
Multi-Output
Model:
The multi-task learning approach involves simultaneously learning multiple related tasks using a shared representation—in our case, the deep embeddings xL in the PiPINN—for improved learning and generalization. During training, xL connects directly to as many output heads as there are different PDE instances uϑ1 , uϑ2 , ..., uϑK in
the training set. Therefore, we can learn a shared embedding xL that matches all seen PDE instances by optimizing the network weights w of a multi-output model to minimize the MSE between target and model outputs. It is worth noting that the model outputs for different PDE instances are controlled by different weight parameters in the output heads, thus creating a good synergy with pseudoinverse computation on new PDE instances. In contrast, the typical MLP model forces a shared output layer for different PDE instances. Multi-task learning with a multi-output model architecture has been explored in PINN literature and is called “Lernaean Hydra” [22], [23]. Hence, we follow the naming convention and refer to the first multi-task learning PINN model as HYDRA+[P i]2 . For predicting or solving new PDE instances, we discard the multi-output heads from the HYDRA+[P i]2 model and perform pseudoinverse physics-informed computation to find the best-fit weight parameters wL on the desired PDE instance. B. Pseudoinverse-In-The-Loop Learning: PiL-PINN While the synergies between data-driven multi-task learning and pseudoinverse computation shed light on the success of the zero-shot PINN model, the HYDRA+[P i]2 does not ensure that the learned embedding is optimal for pseudoinverse physics-informed computation. We can further improve the adaptation performance of the Pi-PINN model through pseudoinverse-in-the-loop learning (albeit with higher training costs). This learning algorithm explicitly incorporates the pseudoinverse physics-informed computation during the training stage. The aim is to learn a shared deep embeddings xL such that the outputs uϑ1 , uϑ2 , ..., uϑK can be best approximated with the pseudoinverse physics-informed computation. We formulate a data-driven loss function L(w− , Φ(w− )) as the learning objective for PiL-PINN model optimization, where w− = (W 1 , b1 , ..., W L−1 , bL−1 ) includes all the network weights leading to the shared embeddings xL , and Φ(w− ) comes from the pseudoinverse physics-informed computation for determining the best-fit weight parameters wL . Critically, the loss function is differentiable and gradientbased optimization methods can be applied. Moreover, we can perform mini-batch training by randomly sampling a subset of PDE instances in every training iteration for greater training efficiency. We refer to our pseudoinverse-in-the-loop learned PINN model as PiL-PINN. IV. E XPERIMENTS A. Experimental Setup and Implementation We conduct extensive experiments on various real-world PDE problems to show the effectiveness of the different Pi-PINN approaches, i.e., MLP+[P i]2 , HYDRA+[P i]2 , and PiL-PINN. On different problems, we train these models with K(= 2, 4, 8, ...) PDE instances with a priori-provided solutions, uϑ1 , uϑ2 , ..., uϑK , and apply the model to predict or solve for remaining PDE instances with no labeled data based on the pseudoinverse physics-informed computation (i.e. zeroshot adaptation). We then compute the relative L2 error on these unseen tasks.
Helmholtz equation
10−1
100
10−3
10−2
Rel. error
Rel. error
Poisson's equation
10−5 10−7
Model MLP MLP+[Pi]2
10−4
HYDRA+[Pi]2 PiL-PINN
10−6
10−9 21
22 23 24 Training PDE instances, K
25
21
Burgers' equation (sine IC)
22 23 24 Training PDE instances, K
25
Burgers' equation (family of IC)
10−1 Rel. error
Rel. error
100
10−2
10−1
21
22 23 Training PDE instances, K
24
22
23 26 24 25 Training PDE instances, K
~28
Fig. 2. Comparison of baseline MLP and different Pi-PINN approaches for predicting or solving different PDE problems.
∂2u = h , x ∈ [−10, 10] (5) ∂x2 with the exact solution u(x; ω1 , ω2 ) = sin(ω1 x) + sin(ω2 x) − 0.1x. This exact solution is used to derive the corresponding BCs and source term h(x; ω1 , ω2 ) = −ω12 sin(ω1 x) − ω22 sin(ω2 x). We generate 100 PDE instances with ω1 and ω2 uniformly sampled from (0, 1] and (0, 2] respectively. Each PDE instance contains 201 spatial points. 2) Helmholtz Equation: Helmholtz equation is another well-known model equation that can represent interesting physics such as wave propagation in both acoustics and electromagnetics. In this study, the 2D Helmholtz equation is investigated [25]: (
∂2u ∂2u + 2 ) + u2 = h ∂x2 ∂y
, x ∈ [−1, 1], y ∈ [−1, 1]
(6)
with the exact solution u(x, y; α1 , α2 ) = 1 − (α1 π)2 − (α2 π)2 sin(α1 πx) sin(α2 πy). This exact so-
3
Seen Unseen
2 1 u
For completeness, we also compare these PINN models with a data-driven MLP, which predicts the unseen PDE instances by interpolating along task parameter ϑ. Below are brief descriptions to the problems. 1) Poisson’s Equation: Poisson’s equation is a widely studied, representative problem for the broad class of elliptic PDEs. As the solution to potential fields in many physical systems, it can model diverse physics phenomena in electromagnetism and heat and mass transfer (e.g., heat conduction) among many others. In this study, the following 1D Poisson’s equation is investigated [24]:
0 1 2 3 10.0 7.5
5.0
2.5 0.0 x
2.5
5.0
7.5 10.0
Fig. 3. The solution of different PDE instances of the Poisson equation problem. Seen examples for model training at K = 2 is highlighted in green.
lution is used to derive the corresponding BCs and source term h(x, y; α1 , α2 ) = (1−(α1 π)2 −(α2 π)2 ) sin(α1 πx) sin(α2 πy). We generated 100 PDE instances with (α1 , α2 ) uniformly sampled from (0, 6]. Each PDE instance contains 64×64 spatial points. 3) Burgers’ Equation (Sine IC): The Burgers’ equation is the model equation for a class of transport phenomena of quantities such as momentum, species transport and heat and mass transfer. We consider the nonlinear Burgers equation as defined by [1], [24]: ∂u ∂2u ∂u +u −γ 2 =0 ∂t ∂x ∂x
, x ∈ [−1, 1], t ∈ (0, 1]
(7)
with the network weights during model optimization. This is an additional advantage of the pseudoinverse-in-the-loop learning. The number of nonlinear iterations for Burgers’ equation (sine IC and family of IC) is 4 and 8 respectively. Compute environment. We implement our models in JAX and perform the experimental study on a workstation with Intel Xeon W-2275 Processor and 2 NVIDIA RTX 3090 GPUs. B. Results and Discussion
Fig. 4. The predicted results and corresponding errors for the Helmholtz equation on new PDE instance (α1 = 1.2, α2 = 6.0), for MLP, MLP+[P i]2 , and HYDRA+[P i]2 trained with K = 32 PDE instances.
subject to IC u(x, t = 0) = −sin(πx), where the flow solution u represents nonlinear waveform propagation with both compression and rarefaction effects, and γ is the viscosity of the fluid. A steep gradient can be observed around x = 0 after some time t. The problem consists of 50 PDE instances with γ uniformly sampled from [0.001, 0.05]. Each PDE instance contains 129×51 spatial-temporal points. 4) Burgers’ Equation (Family of IC): We further consider the following generalized nonlinear Burgers’ equation with a source term, ∂u ∂u ∂2u +u − γ 2 = h , x ∈ [0, 1], t ∈ (0, 0.5] (8) ∂t ∂x ∂x with periodic BC u(0, t) = u(1, t). We fix the viscosity γ = 0.005, but randomly generate the source function h(x, t) = P5 2πlj x A + φj ) with the coefficients sampled j=1 j sin(ωj t + 6 from Aj ∈ [−0.8, 0.8], ωj ∈ [−2, 2], lj ∈ [0, 1, 2, 3, 4], φj ∈ [−π, π]. The problem consists of 480 PDE instances with different source terms and ICs (from u(x, t = 0) = h(x, t = 0) and its perturbations), representing a rich diversity of dynamical processes. Each PDE instance contains 51×26 spatialtemporal points. Hyperparameters setting. We fix λBC = λIC = 1 and perform a grid search to determine an appropriate λPDE and λPI for the MLP+[P i]2 and HYDRA+[P i]2 models based on the seen PDE instances, before applying them to solve the unseen PDE instances. For the PiL-PINN, we pass these hyper-parameters to the loss function and jointly learn them
Given the underlying context that data is typically sparse in many science and engineering real-world scenarios, we evaluate the performance of the various models (MLP, MLP+[P i]2 , HYDRA+[P i]2 , and PiL-PINN) across all 4 problems for a range of scenarios where solutions to K randomly sampled tasks are available for training. The main results are in Figure 2. A representative plot of the solutions for Poisson’s equation problem is given in Figures 3. Representative plot of the solutions obtained by the different methods for the Helmholtz equation and nonlinear Burgers’ equation (Sine IC and Family of IC) are given in Figures 4, 5 and 6. Improvement in Prediction Performance through MLP+[P i]2 . In general, we note that the data-driven MLP’s predictive performance is fairly poor, with relative errors ranging from 0.1 to 1, which is intuitively consistent with the small amount of training data provided and the complexity of the model physics. The predicted solutions are visibly inaccurate in Figures 5 and 4. Excitingly, the combination of a simple data-driven training strategy and an extra psuedoinverse computation to finetune the final layer to better match the underlying physicsbased constraints still results in substantial improvements in relative error for the Poisson’s equation (2 orders of magnitude reduction) and two Burgers’ equation problems (1 order of magnitude reduction) as shown in Figure 2. These experiments clearly show the effectiveness of a physics-informed loss in improving the prediction of data-driven models under conditions where the available labeled data is small. The results also open the possibility for greater utilization of other data-driven models and architectures in future extensions. We note that a reason for the widespread usage of MLP architectures in PINN literature is the need for simple implementations given the complexity of PINN training. The potential extension with a physics-informed loss through fast pseudoinverse computations can greatly mitigate this issue. In addition, the benefit from including the physics-informed loss is most pronounced when K is smaller. As K increases, the data-driven MLP model’s relative error begins to show a decreasing trend. However, we note that the MLP+[P i]2 generally shows reasonable performance even at low K in 3 of the 4 problems. Nonetheless, the data-driven MLP training does not always learn a good shared embedding for the physics-informed loss, as exemplified by the relatively higher errors in the Helmholtz problem. Importance of an Expressive Shared Embeddings. Given the importance of expressiveness of the learned embedding for the pseudoinverse computation to minimization of the
(b) 𝛾𝛾 = 0.05
(b) 𝛾𝛾 = 0.002
Fig. 5. The predicted results and corresponding errors for the nonlinear Burger’s equation (sine IC), for MLP and PiL-PINN with different PDE instances K.
Fig. 6. The predicted results and corresponding errors for the Burgers’ equation (family of IC) on new PDE instances, for MLP, MLP+[P i]2 , and Pil-PINN trained with K = 240 PDE instances.
physics-informed loss, we further compare the proposed neural architecture (with concatenative skip connections across the hidden layers) in the context of the multi-task learning setup. From Figure 2, the HYDRA+[P i]2 shows significantly improved relative errors compared to the MLP+[P i]2 models. This improvement is particularly notable for the Poisson equation and the Helmholtz equation in Figure 4, showing the importance of our proposed neural architecture in enabling the learning of a more expressive and performant shared embedding for the pseudoinverse computation. We also notice that the performance of HYDRA+[P i]2 (and PiL-PINN models) can be further improved by the use of an appropriate factor F π in Eq. 4e (F = 2 for Poisson’s and 4 for Helmholtz problems). While the improvement was notable for the linear PDEs (Poisson’s and Helmholtz equation), the reduction in the relative error for HYDRA+[P i]2 relative to the MLP+[P i]2 was much less significant for the nonlinear Burgers’ equation. This is potentially due to the fact that the current zero-shot framework fundamentally only finds the best linear solution (through the pseudoinverse).
PiL-PINN for Nonlinear PDEs. As the nonlinear PDEs are currently solved in a slightly modified process through a series of iterations where the equation is linearized, a different way of learning the shared embedding where the pseudoinverse is taken explicitly into account during training can be more appropriate. Hence, we proposed the PiL-PINN methodology to explicitly handle this nuance. As per Figure 2, the PiL-PINN shows a further reduction in relative errors for the nonlinear Burgers’ problems relative to the other 3 methods (improvement relative to the MLP method is presented in Figure 5 and Figure 6). In addition, the PiLPINN results also show a significant reduction in relative error with K, indicating that the proposed method indeed allows for better learning of a shared embedding for zero-shot prediction as observed for Burgers’ equation. Computation Time. The training time for the PiL-PINN models range from a few seconds on the Poisson’s problem to 1.5 hours, on the Burgers’ (family of IC) problem (K = 240). The PiL-PINN model takes less than 1 second to make predictions on a new PDE instance across all the problems
tested. For example, it takes 54ms to make a prediction for the Burgers’ equation (sine IC)—a benchmark problem frequently studied in PINN literature—at 129x51 spatial-temporal points (Figure 5). In contrast, others in literature have reported typical training times of ≈10 min to 1 hour for a wide range of canonical PDE problems [26]. This training time is also consistent with our own experience when training single PINN models to produce predictions for Burgers’, Poisson, or Helmholtz equation. This means that the Pi-PINN method can produce predictions for new scenarios 100–1000× faster. V. C ONCLUSIONS In this work, we demonstrate a transferable learning approach that enables rapid and accurate physics-informed solutions by incorporating a least-squares-optimal pseudoinverse computation within a closed-form head adaptation step. This approach underscores the potential of combining data-driven multi-task learning with physics-informed learning to improve PINNs’ efficiency and generalization across PDE families and parameter regimes. Even when the labeled data are limited, our results show that Pi-PINN can learn a shared, transferable embedding that supports significantly improved generalization to new PDE instances. The empirical results also suggest that architectural modifications such as concatenative skip connections are beneficial. Nonetheless, there remains substantial scope for developing neural architectures and training algorithm designs that learn transferable embeddings even more effectively, which is particularly important given the practical constraint that datasets in many scientific and engineering domains may be limited in quantity. We hope this work encourages alternative directions that move PINNs closer to robust, reusable tools for real-world applications. ACKNOWLEDGMENT This research was in part supported by the National Research Foundation, Singapore through the AI Singapore Programme, under the project “AI-based urban cooling technology development” (Award No. AISG3-TC-2024-014-SGKR). The authors acknowledge the use of AI tools for language editing and take full responsibility for the final manuscript. R EFERENCES [1] M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” Journal of Computational physics, vol. 378, pp. 686–707, 2019. [2] G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang, “Physics-informed machine learning,” Nature Reviews Physics, vol. 3, no. 6, pp. 422–440, 2021. [3] N. Wandel, M. Weinmann, M. Neidlin, and R. Klein, “Spline-pinn: Approaching pdes without data using fast, physics-informed hermitespline cnns,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, 2022, pp. 8529–8538. [4] J. Kim, K. Lee, D. Lee, S. Y. Jhin, and N. Park, “Dpm: A novel training method for physics-informed neural networks in extrapolation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 9, 2021, pp. 8146–8154. [5] N. Kang, B. Lee, Y. Hong, S.-B. Yun, and E. Park, “Pixel: Physicsinformed cell representations for fast and accurate pde solvers,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 7, 2023, pp. 8186–8194.
[6] Z. Yang, Z. Qiu, and D. Fu, “Dmis: Dynamic mesh-based importance sampling for training physics-informed neural networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 4, 2023, pp. 5375–5383. [7] J. C. Wong, C. Ooi, A. Gupta, and Y.-S. Ong, “Learning in sinusoidal spaces with physics-informed neural networks,” IEEE Transactions on Artificial Intelligence, 2022. [8] A. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. W. Mahoney, “Characterizing possible failure modes in physics-informed neural networks,” Advances in Neural Information Processing Systems, vol. 34, pp. 26 548–26 560, 2021. [9] S. Wang, Y. Teng, and P. Perdikaris, “Understanding and mitigating gradient flow pathologies in physics-informed neural networks,” SIAM Journal on Scientific Computing, vol. 43, no. 5, pp. A3055–A3081, 2021. [10] S. Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli, “Scientific machine learning through physics–informed neural networks: Where we are and what’s next,” Journal of Scientific Computing, vol. 92, no. 3, p. 88, 2022. [11] J. C. Wong, A. Gupta, C. C. Ooi, P.-H. Chiu, J. Liu, and Y.-S. Ong, “Evolutionary optimization of physics-informed neural networks: Evo-pinn frontiers and opportunities,” IEEE Computational Intelligence Magazine, vol. 21, no. 1, pp. 16–36, 2026. [12] J. C. Wong, A. Gupta, and Y.-S. Ong, “Can transfer neuroevolution tractably solve your differential equations?” IEEE Computational Intelligence Magazine, vol. 16, no. 2, pp. 14–30, 2021. [13] S. Goswami, C. Anitescu, S. Chakraborty, and T. Rabczuk, “Transfer learning enhanced physics informed neural network for phase-field modeling of fracture,” Theoretical and Applied Fracture Mechanics, vol. 106, p. 102447, 2020. [14] S. Dong and Z. Li, “Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations,” Computer Methods in Applied Mechanics and Engineering, vol. 387, p. 114129, 2021. [15] F. Calabrò, G. Fabiani, and C. Siettos, “Extreme learning machine collocation for the numerical solution of elliptic pdes with sharp gradients,” Computer Methods in Applied Mechanics and Engineering, vol. 387, p. 114188, 2021. [16] R. Penrose, “On best approximate solutions of linear matrix equations,” Mathematical Proceedings of the Cambridge Philosophical Society, vol. 52, pp. 17–19, 1956. [17] S. Wang, H. Wang, and P. Perdikaris, “On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks,” Computer Methods in Applied Mechanics and Engineering, vol. 384, p. 113938, 2021. [18] J. C. Wong, C. C. Ooi, A. Gupta, P.-H. Chiu, J. S. Z. Low, M. H. Dao, and Y.-S. Ong, “Evolutionary optimization of physics-informed neural networks: Advancing generalizability by the baldwin effect,” IEEE Transactions on Evolutionary Computation, 2026. [19] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable effectiveness of data in deep learning era,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 843–852. [20] T. Nguyen, M. Raghu, and S. Kornblith, “Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth,” arXiv preprint arXiv:2010.15327, 2020. [21] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla, “Nerfies: Deformable neural radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5865–5874. [22] Z. Zou and G. E. Karniadakis, “L-hydra: Multi-head physics-informed neural networks,” arXiv preprint arXiv:2301.02152, 2023. [23] S. Desai, M. Mattheakis, H. Joy, P. Protopapas, and S. Roberts, “Oneshot transfer learning of physics-informed neural networks,” arXiv preprint arXiv:2110.11286, 2021. [24] X. Liu, X. Zhang, W. Peng, W. Zhou, and W. Yao, “A novel metalearning initialization method for physics-informed neural networks,” Neural Computing and Applications, vol. 34, no. 17, pp. 14 511–14 534, 2022. [25] W. Cho, K. Lee, D. Rim, and N. Park, “Hypernetwork-based metalearning for low-rank physics-informed neural networks,” Advances in Neural Information Processing Systems, vol. 36, 2024. [26] S. Wang, S. Sankaran, H. Wang, and P. Perdikaris, “An expert’s guide to training physics-informed neural networks,” arXiv preprint arXiv:2308.08468, 2023.