On the Identifiability of Hybrid Deep Generative Models: Meta-Learning as a Solution

On the Identifiability of Hybrid Deep Generative Models: Meta-Learning as a Solution

arXiv:2610.32748v1 [cs.LG] 23 Oct 2026

Yubo Ye ∗ Zhejiang University 22230131@zju.edu.cn Sumeet Vadhavkar Rochester Institute of Technology sv6234@rit.edu

Maryam Toloubidokhti Rochester Institute of Technology mt6129@rit.edu Xiajun Jiang Rochester Institute of Technology xj7056@rit.edu

Huafeng Liu B Zhejiang University liuhf@zju.edu.cn

Linwei Wang Rochester Institute of Technology linwei.wang@rit.edu

Abstract The interest in leveraging physics-based inductive bias in deep learning has resulted in recent development of hybrid deep generative models (hybrid-DGMs) that integrate known physics-based mathematical expressions in neural generative models. To identify these hybrid-DGMs requires inferring parameters of the physics-based component along with their neural component. The identifiability of these hybrid-DGMs, however, has not yet been theoretically probed or established. How does the existing theory of the un-identifiability of general DGMs apply to hybrid-DGMs? What may be an effective approach to construct a hybrid-DGM with theoretically-proven identifiability? This paper provides the first theoretical probe into the identifiability of hybrid-DGMs, and presents meta-learning as a novel solution to construct identifiable hybrid-DGMs. On synthetic and realdata benchmarks, we provide strong empirical evidence for the un-identifiability of existing hybrid-DGMs using unconditional priors, and strong identifiability results of the presented meta-formulations of hybrid-DGMs. Code is available at https://github.com/GoForIt-YYB/Meta-Hybrid-VAE.

1

Introduction

There has been increasing interest in integrating mathematical expressions of known physics with neural-network functions for hybrid, or gray-box, modeling [1, 2, 3, 4, 5]. The incorporation of physics-based inductive bias has the potential to improve the generalizability and interpretability of deep learning [4], while the expressive and flexible neural functions offer the opportunity to fill the gap in our prior knowledge of physics (or expert models) [4, 5]. Recent progress in particular has started to see principled developments of hybrid deep generative models (DGM) where the data-generation process is described by a hybridization of physics-based and neural components. Both of these two components are then identified from data via, for instance, amortized variational inference [2, 4]. These hybrid-DGMs have shown benefits of interpretability, generalizability, and out-of-distribution (OoD) robustness, especially when learning to model physics systems from observational data [2, 4]. An important premise of these successes is that the hybrid expression of the physical systems can ∗

Work was done as a visiting exchange student at Rochester Institute of Technology.

38th Conference on Neural Information Processing Systems (NeurIPS 2024).

be accurately identified from the observed data: for example, if we can identify the parameters and/or physics governing observed pendulum movements, we could then use the identified generative models/parameters for predicting the future trajectory of the pendulum [2]. However, substantial research based on the theory of nonlinear independent component analysis (nonlinear ICA) have established that a naive DGM or latent variable model is un-identifiable [6]. Active research has also reported various ways to construct an identifiable latent variable model, primarily via constructing a conditionally independent generative model [7, 8, 9, 6]. Most of the progress has been established for the identifiability of latent processes underlying time-series data, leveraging the temporal dependency of the latent variables [8] or observed domain index for nonstationary segments [7] to construct the conditional generative model. For static latent variables, the construction of conditionally independent generative model has mainly relied on the introduction of auxiliary observations such as class or domain labels [9]. How does the hybrid expression of a DGM impact its (un)-identifiability? Furthermore, since hybrid-DGMs are often intended to model physics systems with unknown parameters to be identified from data, what may be an effective approach to construct a hybrid-DGM with theoretically-proven identifiability? Unfortunately, despite the increasing interest in hybrid-DGMs, their identifiability remains unexplored. This identifiability however is critical for the intended use of a hybrid DGM, e.g., to predict beyond the observations used to identify the model, or to be robust in out-of-distribution (OOD) generalizations. Recent efforts in hybrid-DGMs indeed have increasingly noted the challenges in properly learning the two components within the hybrid models, especially in ensuring a non-trivial solution to the physics-based component due to the presence of a highly-expressive neural component [1, 2]. Various strategies have been presented, such as regularizing the expressiveness of the neural component [2], trajectory-based and adaptive optimization to allow automatic complexity adjustment for the neural component [1], and expert-augmentation leveraging the physics-based component to improve generalization to distributional shifts [4]. No existing works, however, have probed or established the identifiability of hybrid-DGMs, theoretically or empirically. In this work, we present the first theoretical probe of the identifiability of hybrid-DGMs. Importantly, forgoing the need of observed auxiliary variable to condition the generative model, we present meta-learning as a novel solution to establish the identifiability of hybrid-DGMs. To this end, we present a learn-to-identify formulation for meta-learning hybrid-DGMs, and theoretically establish the identifiability of these meta-hybrid-DGMs via conditional independence of the DGM given few-shot context samples. We further provide empirical identifiability results of these meta-hybrid-DGMs in a variety of synthetic and real datasets generated from physics systems, in contrast to existing non-meta hybrid-DGMs where identifiability cannot be established without observed auxiliary variables. Finally, we note that while this work focuses on hybrid-DGMs, the presented meta-learning formulation provides a general new condition to construct identifiable DGMs beyond the hybrid setting.

2

Related Works

Hybrid-DGMs: A number of hybrid models have emerged in various domains to combine physicsbased functions with neural networks, to compensate for unknown components in the known physics. While earlier approaches [10, 11, 12] focused on using neural networks to learn the residual between the measurements and those obtained from physics-based simulation, tighter integrations have been presented by, e.g., NeuralSim [13] and universal differential equations (UDEs) [14], to include neural networks as different components within a physics-based function. Most of these earlier works are focused on hybrid modeling in specific application domains. More recently, several hybrid-DGMs have been presented [2, 1, 4, 3], with a focus on properly learning the physics component within the hybrid-DGM so it is not overpowered by the expressive neural component. In physics-integrated hybrid VAE [2], this was achieved by a regularized learning method that controls the expressiveness of the neural component while preserving the semantics of the physics-based component. Similarly, regularization of the expressiveness of the neural component is considered in [1]. In [4], expert augmentation is proposed to improve the OOD robustness of these hybrid-DGMs by fine-tuning on synthetic OOD samples, generated by sampling outside the training distribution of the physics-based component within the trained hybrid model. 2

Despite these efforts in better learning the two separate components within hybrid-DGMs, the identifiability of hybrid-DGMs has not been theoretically probed nor established. Empirically, no existing works have evaluated these hybrid-DGMs from the lens of identifiability metrics. Theoretically, no existing works have examined the conditions for constructing identifiable hybrid-DGMs. This work will take the first step to bridge these gaps, establish the un-identifiability of naive unconditional hybrid-DGMs, and present meta-learning as a novel solution to construct identifiable hybrid-DGMs. Nonlinear ICA and Identifiability of DGMs: With nonlinear ICA, strong identifiability results have been established for unsupervised latent variable models [7, 8, 9, 6, 15, 16, 17, 18]. Fundamentally, construction of identifiable generative models is achieved by defining independence structure when conditioned on observed auxiliary variables [9]. Much progress has been made in identifying latent dynamic processes generating observed time series, including leveraging conditional independence of the DGM given time segment index [7], temporal dependence [8], sparse temporal encoding [15], and even unobserved inferred domain labels for non-stationary time series [16, 17, 18]. For static latent variables, identifiability is mainly established via the introduction of auxiliary observations such as class or domain labels [9]. In specific, iVAE [6] established the identifiability of DGMs assuming an exponential families conditional distributions given observed auxiliary variables. The reliance on observed conditioning variable, however, limits the applicability of these identifiability results to hybrid-DGMs: as hybrid-DGMs are intended for modeling physics system with unknown and often continuous parameters, it is not clear what observed auxiliary label may be available to construct an identifiable DGM. This work will forego this assumption of observed auxiliary variable and present meta-learning as a novel condition to construct identifiable DGMs. While presented for hybrid-DGMs, this connection between meta-learning and the identifiability of DGMs is a novel contribution to general non-hybrid DGMs that has not been constructed in the existing literature.

3

Problem Formulation: Unidentifiability of Hybrid-DGMs

Let x ∈ Rn be the observed random variables and z ∈ Rd be the latent variables generating x, with the following generation process: Z p(x, z) = p(z)p(x|z), p(x) = p(x, z)dz (1) z

Qd

where p(z) = i=1 p(zi ) represents the prior distribution of independent generative factors zi , p(x|z) is the likelihood function which can be defined based on the mixing function x = F (z). Construction of Hybrid-DGMs: When constructing a generative model for Equation (1), if we assume a known mathematical expression explaining the mechanism of the mixing function as fP , we will have a physics-based expression of this generation process, with unknown parameters of fP as physics-based latent variables zP . If we have no prior knowledge, we can describe the mixing function with a neural network fN with abstract representation of the latent variables as zN – the latter is the foundation of many successful DGMs including variational autoencoders (VAEs) [19]. When understanding complex systems in many domains, recent years have seen an increasing interest in the middle-ground of the above two scenarios: some prior knowledge in the form of a mathematical expression fP of physics exists about the data observed, yet often inexact with unknown gaps to the actual data-generating mechanisms. This motivated recent developments of hybrid-DGMs where the mixing function x = F(z) is described as a combination of a physics-based and neural component, fP and fN , respectively, with corresponding physics-based and abstract latent variables zP and zN . As a concrete example, suppose that we observe the time-series of the angular position φ of a damped forced pendulum system, x = [φ0 , φ1 , . . . , φT ] where φi = φ(i∆t), governed by: d2 φ(t) dφ(t) + ω 2 sin φ(t) + ξ − A cos(2πϕt) = 0 (2) dt2 dt where the first two terms describe the physics of an ideal pendulum system, the third term the damping effect, and the last term the external force. Parameters ω, ξ, A, and ϕ specify different systems governed by the same physics, representing the independent latent generative factors. Suppose that we choose to leverage our prior knowledge about the ideal pendulum physics fP (x; zP ) = φ̈ + z2P sin φ with unknown parameter ω. To bridge its potential gap with actual 3

data-generating physics, we then choose to complete it with a neural-network expression of an ODE fNθ (x; zN ), parameterized by θ and with an abstract latent variable zN : while abstract, we hope fNθ to absorb the effect of the last two terms in the data-generating physics (Equation (2) with zN representing its remaining data-generating parameters ξ, A, and ϕ. This gives rise to a hybrid mixing function x = F[fP , fNθ ; zP , zN ] = ODEsolve[fP (x; zP ) + fNθ (x; zN ) = 0]. Assuming Gaussian observation noises, we obtain the likelihood of the Hybrid-DGMs as: pθ (x|zP , zN ) = N (x|F[fP , fNθ ; zP , zN ], Σx ) (3) with prior distribution for the latent variables assumed to be Gaussian as: p(zP ) := N (zP |µP , σP2 I), p(zN ) := N (zN |0, I) (4) 2 where the mean µP and variance σP for p(zP ) can be defined by prior knowledge due to its physical meanings. Note that zP will be directly interpretable and physically meaningful as they will be semantically grounded to the parameters of the physics model fP ; zN , in comparison, will be abstract but need to absorb effect of the varying parameters underlying the missing physics fN . The data-generating process of the hybrid-DGM with parameter θ can thus be defined as: Z Z pθ (x, zP , zN ) = pθ (x|zP , zN )p(zP )p(zN ), pθ (x) = pθ (x, zP , zN )dzP dzN

(5)

While described in the context of a concrete example, Equations (3)-(5) represent a general expression of hybrid-DGMs where the hybrid-mixing function x = F[fP , fNθ ; zP ; zN ] can be designed in various forms (and not limited to the hybrid-ODE described in the example above).  Unidentifiability of Hybrid-DGMs: Now assume we have access to data D = x(1) , . . . , x(N ) generated by pθ∗ (x, zP , zN ), where θ∗ is the true but unknown data-generating parameter. For an identifiable DGM, our goal is to learn θ such that the likelihood p∗θ (x|zP , zN ), the priors p(zP ) and p(zN ), and the posteriors p∗θ (zP |x) and p∗θ (zN |x) are all correctly recovered. In practice, because we only observe x without access to the latent variables zP or zN , we optimize θ to match the marginal density of x, e.g., via amortized variational inference to maximize the evidence lower bound (ELBO) of pθ (x) in VAEs, such that: pθ (x) ≈ pθ∗ (x) (6) Unfortunately, even if the above optimization is done perfectly (i.e., pθ (x) = pθ∗ (x) ), there is no guarantee that p∗θ (x|zP , zN ), p∗θ (zN |x), or pθ∗ (x, zP , zN ) are correctly identified. As shown in [6], for these densities to be recovered based on matching pθ (x), the DGM needs to satisfy: ∀(θ, θ̃) : pθ (x) = pθ̃ (x) ⇒ θ = θ̃

(7)

In another word, the matching of p∗θ (x) needs to imply a unique solution of θ∗ to be obtained. In reality, however, it has been shown in [6] that DGMs with unconditional prior are unidentifiable. We refer the readers to the important theorems and the proof in [6, 9, 20] for more details, but to what extent is the hybrid-DGM affected by this theory of identifiability? At a concept level, let us first consider the neural component (zN and fNθ ) within the hybrid-DGM. For zN of any distributions, there are always nonlinear transformations that change its values but not its distributions [6]. In a highly expressive neural network, this transformation can be learned in the likelihood pθ (x|zN ). This will give the same fit of pθ (x), but non-unique solutions of pθ (x|zN ) and pθ (zN |x), resulting in non-identifiable neural component within the hybrid-DGM. For the physics-based component (zP and fPθ ), this theory does not hold because the pre-specified form of fp can not trivially accommodate any transformations applied to zP without affecting the fitting of observations. In practice, however, because both components jointly contribute to pθ (x|zP , zN ) as defined in Equation (3), the optimization of zP and fPθ maybe overpowered by the overly-expressive neural component, rendering trivial solutions to zP and fPθ . In other words, identification of hybrid-DGMs is inflicted by two fundamental challenges: 1) the theoretical un-identifiability of its neural component, and 2) this un-identifiable but highly expressive component overpowering the identification of the otherwise identifiable physics-based component. While existing works [2, 1, 3] have focused on the second challenge, we address the problem from its fundamentals at the un-identifiability of zN and fNθ : we introduce a novel condition to construct identifiable hybrid-DGMs, and establish their identifiability both theoretically and empirically. 4

4

Learn-to-Identify Hybrid-DGMs via Meta-Learning

Based on nonlinear ICA theory [9], an identifiable DGM can be constructed by conditionally independent latent variables p(z|u): in the literature, the conditioning variable u is often assumed to be additionally-observed side information such as class labels or domain indexes [6]. Here, we show that meta-learning formulations offer an opportunity to construct an identifiable DGM, with u taking forms of k-shot context samples sharing the same data-generation process as the query samples. Construction of Identifiable Meta-Hybrid-DGMs: Given is a set of k context samples, Ds , sharing the same data-generation process as query samples x, we define the generation of x as: Z Z s s s pϕ (x, zP , zN |D ) = pθ (x|zP , zN )p(zP )pζ (zN |D ), pϕ (x|D ) = pϕ (x, zP , zN |Ds )dzP dzN (8) where ϕ = {θ, ζ}, pθ (x|zP , zN ) is defined in Equation (3), and the conditional prior pζ (zN |Ds ) is assumed to be factorized exponential family distributions [21]: pζ (zN |Ds ) = pT,λλζ (zN |Ds ) =

d Y Qi (zN,i )

Zi (Ds ) i=1

  exp Ti (zN,i )⊤λ ζ,i (Ds )

(9)

where Qi is the base measure, Zi (Ds ) is the normalizing constant, Ti = (Ti,1 , . . . , Ti,e ) are the sufficient statistics that are fixed (not estimated). λ ζi (Ds ) = (λζi,1 (Ds ), . . . , λζi,e (Ds )) are the corresponding parameters conditioning on Ds . In this paper, we realize this conditioning through a composite of a neural network hζ (xs ), parameterized by ζ, for individual context samples xs ∈ Ds , and an averaging function across all context samples: 1 X hζ (xs ) (10) λζ (Ds ) = s |D | s s x ∈D

Note that exponential families have universal approximation capabilities, thus this commonly-used assumption is not very restrictive [20]. Amortized Variational Inference: To enable inference over the hybrid-DGM in Equation (8), we approximate the posterior density p(zP |x) as qη (zP |x) and pζ (zN |x, Ds ) as qζ (zN |x ∪ Ds ), the latter realized with the same network as defined in (9) but with the additional input x in addition to Ds . Formally, we cast the variational inference into a meta-learning formulation. Consider a dataset D with M M similar but distinct data-generation process: D = {Dm }m=1 . For each Dm, we consider disjoint  s q context samples Dm = xs,1 , xs,2 , . . . , xs,k and query samples Dm = xq,1 , xq,2 , . . . , xq,l , where k ≪ l. Instead of maximizing the marginal likelihood of x for all x ∈ D, we formulate a metas q objective to learn to maximize the marginal likelihood p(xq |Dm ) of all query samples xq ∈ Dm when s conditioned on support set Dm , for all underlying data-generation processes m ∈ {1, 2, . . . , M }: M X X q m=1 xq ∈Dm

s log p(xq |Dm )≥

M X X

q s ) [log pθ (x |zP , zN )] {Eqη (zP |xq ),qζ (zN |xq ∪Dm

q m=1 xq ∈Dm

s s − KL(qη (zP |xq )||p(zP )) − KL(qζ (zN |xq ∪ Dm )||pζ (zN |Dm ))} (11) q where the first term represents likelihood of the meta-hybrid-DGM on query sample xq ∈ Dm , and the last two terms represent Kullback–Leibler divergences between the prior and posterior densities s of zP and zN , respectively, where densities of zN are conditioned on context samples Dm . The maximization of Equation (11) is performed over {η, ζ, θ} in an episodic training where, in each s q episode, the division of Dm and Dm is shuffled within each Dm . The likelihood is calculated by the reparameterization trick [19], while the two KL-terms are calculated analytically.

Note that in Equation (11), the meta-hybrid-DGM is formulated on the neural component within the hybrid-DGM to establish its identifiability. While the identifiability of the physics-based component is not hinged on this construction, in practice, it is possible to extend the conditional prior (Equation 5

s (9)) to zP to obtain pζ (zP , zN |Dm ), leading to an alternative meta-formulation: M M X X X X s q s ) [log pθ (x |zP , zN )] log p(xq |Dm )≥ {Eqζ (zP ,zN |xq ∪Dm q m=1 xq ∈Dm

q m=1 xq ∈Dm

(12)

s s − KL(qζ (zP , zN |xq ∪ Dm )||pζ (zP , zN |Dm ))}

We will empirically compare the two formulations above, and demonstrate how the identifiability of zP and zN is each affected by the proposed meta-construction of the hybrid-DGMs.

5

Identifiability Theory for Meta-Hybrid-DGMs

We consider the meta-formulation a novel condition to ensure the identifiability of unsupervised DGMs. Here, we show that, built on nonlinear ICA and the theory of identifiability for conditionalVAEs established in [6], the presented meta-hybrid-DGMs are identifiable. For readability, below we collect the formulations of the hybrid-DGM from previous sections: pϕ (x, zP , zN |Ds ) = pθ (x|zP , zN )p(zP )pT,λλζ (zN |Ds ) (13) pθ (x|zP , zN ) = pε (x − F [fP , fNθ ; zP , zN ]) (14) d Y   Qi (zN,i ) pT,λλζ (zN |Ds ) = exp Ti (zN,i )⊤λ ζ,i (Ds ) (15) s) Z (D i i=1 where ϕ = (θ, T, λ ζ ) are model parameters and λ ζ is defined in Equation (10). Equation (14) gives a more general definition than Equation (3) where ε is an independent noise variable. Definition 1: Let ∼ be an equivalence relation on the parameter space Φ. We say the DGM in 1 is ∼-identifiable if pϕ (x) = pϕ̃ (x) ⇒ ϕ ∼ ϕ̃ Definition 2: Let ∼ be the equivalence relation on Φ defined as follows: (θ, T, λ ζ ) ∼ (θ̃, T̃, λ ζ̃ ) ⇔ ∃A, c : T(Fθ−1 (x)) = AT̃(Fθ̃−1 (x)) + c, ∀x ∈ X (16) where A is a de × de matrix and c is a vector of dimension de, d being the dimension of zN and e the dimensionality of the sufficient statistics Ti . If A is invertible, we denote this relation by ∼A . Definition 2 establishes a specific equivalence relation that allows to recover the sufficient statistics of the generative model up to a linear matrix multiplication. Theorem 1: Assume we observe data sampled from pθ (x, zP , zN |Ds ) as defined in Equations (13)-(15) with parameters ϕ = (θ, T, λ ζ ). Assume the following holds: (i) The set {x ∈ X |φε (x) = 0} has measure zero, where φε is the characteristic function of the density pε defined in pε (x − F [fP , fNθ ; zP , zN ]). (ii) The hybrid mixing function Fθ is injective. (iii) The sufficient function Ti,j are differentiable almost everywhere, and linearly independent on any subset of X of measure greater than zero. (iv) There exist de + 1 distinct context sets Ds,0 , Ds,1 , . . . , Ds,de such that the de × de matrix L defined as follows is invertible: λ(Ds,1 ) − λ (Ds,0 ), . . . , λ (Ds,de ) − λ (Ds,0 )) L = (λ (17) Then the parameters ϕ = (θ, T, λ ζ ) are ∼A -identifiable. The proof for Theorem 1 directly builds on that presented in [6]. Condition (iv) in Theorem 1 establishes that, if the generative factors in the neural component of the hybrid-DGM is d-dimensional each specified with e-dimensional sufficient statistics, we will need observations from a minimum of de+1 distinct and independent generation processes in order to identify the hybrid-DGM. Considering the pendulum-system example discussed in Section 3 and Equation (2), since the neural component is generated by three independent system parameters ξ, A, and ϕ (d = 3) and assuming Gaussian statistics (e = 2), the hybrid-DGMs will be identifiable if we have observations generated from more than 3 × 2 + 1 = 7 distinct combination of parameter values for ω, ξ, A, and ϕ in the true data-generation process (Equation (2)). We will empirically verify this condition in Section 6. 6

Table 1: Quantitative identifiability metrics for the presented meta-hybrid-VAE in comparison to physics-integrated hybrid-VAE [2] and APHYNITY [1], including MSE and MCC metrics on the latent variables, as well as MSE of generated x during reconstruction of observed samples (Rec) and prediction of unobserved samples (Pre). Forced Damped Pendulum APHYNITY Hybrid-VAE Meta-Hybrid-VAE

6

Advection-Diffusion System APHYNITY Hybrid-VAE Meta-Hybrid-VAE

MSE of zP ↓

6.96(0.01)e-2

4.14(0.29)e-2

1.59(0.07)e-2

1.9(0.3)e-2

7.12(5.32)e-4

1.34(0.75)e-4

MCC ↑

0.79(0.02)

0.59(0.03)

0.99(0.00)

0.98(0.01)

0.94(0.00)

0.99(0.00)

MSE of x (Rec) ↓

5.2(0.01)e-2

2.66(0.03)e-2

2.91(0.06)e-2

6.22(0.41)e-3

4.90(0.75)e-3

2.93(0.22)e-3

MSE of x (Pre) ↓

2.37(0.67)

1.74(0.20)

6.85(1.11)e-2

8.25(0.58)e1

8.53(5.72)e-2

5.63(1.34)e-3

APHYNITY

Double Pendulum Hybrid-VAE Meta-Hybrid-VAE

Real Double Pendulum APHYNITY Hybrid-VAE Meta-Hybrid-VAE

MSE of zP ↓

4.58(0.04)e-1

3.97(0.06)e-1

3.85(0.19)e-1

/

/

/

MCC ↑

0.50(0.00)

0.51(0.00)

0.98(0.00)

/

/

/

MSE of x (Rec) ↓

4.88(0.00)e-2

4.05(0.25)e-2

3.83(0.06)e-2

3.78(0.52)e-2

2.20(0.14)e-3

2.67(0.34)e-2

MSE of x (Pre) ↓

1.29(0.23)e1

5.52(0.06)

2.80(0.48)e-1

5.16(0.69)

1.87(0.13)

2.88(0.34)e-2

Experiments and Results

Data: We considered three simulated and one real-world benchmarks for hybrid-DGMs, including three simulated physics systems of forced damped pendulum [2], advection-diffusion system [2] and double pendulum [4], and one real-world system double pendulum [4]. For each of the three simulated physics systems, we randomly sampled the initial states and parameters of the governing function to generate observations. We refer to the governing functions used for the generation of data as full physics functions, and design partial physics functions for represent our imperfect prior knowledge about the observed data, reflecting a variety of potentially additive and multiplicative errors – these partial physics functions are used as the fP in the hybrid models. In data generation, we varied the parameters in the components both present and absent in the prior physics. More details on each dataset can be found in Appendix A.

Baselines: We considered 1) physics-integrated hybrid VAE [2] and 2) APHYNITY [1], both using regularization to control the expressiveness of the neural component. Note that while the original formulation of APHYNITY does not have an inference network to identify zP or zN (making it intended for data generated from a single parameter setting), we added an inference network (that shares a similar architecture with the other two models) for fairness of comparison.

Metrics: With a focus on identifiability, we focus on the following metrics: 1) for physics-based latent variable zP , we consider mean squared error (MSE) owing to its physical meaning; 2) for abstract latent variable zN modeled in the neural component, we consider the mean correlation coefficient (MCC) between the true data-generating parameters and zN sampled from its learned posterior density. The calculation of MCC follows standard practice in the identifiability literature [6, 9]: to evaluate ∼A -identifiability, we calculate the weak MCC with the method described in [22], where a high MCC provides evidence of successful identification. We also evaluate a more strong identifiable relation ∼P with strong MCC and the details can be found in Appendix E. In addition to metrics on zP and zN , we further introduce MSE metrics on the generated x in two distinct scenarios: 1) reconstruction MSE measures how well an observed x is reconstructed from the identified zP and zN , and 2) prediction MSE measures how well the identified zP and zN can be used to generate outside the observed x used to identify them, e.g., for predicting over long time domains, or predicting for a different sample that comes from the same generation process but with different initial conditions. We use these two metrics to showcase that a good reconstruction does not guarantee a good identification, while the ability to use the identified latent variables to predict for different samples (other than observed) may be a surrogate for identifiability. 7

Figure 1: Visual results on synthetic data for reconstruction and prediction performances. 6.1

Results on Synthetic Data

Reconstruction vs. Prediction Performance: Table 1 summarizes quantitative identifiability metrics on the presented meta-hybrid-VAE in comparison to its baselines, where the prediction performance of all models is mentioned by their ability – once identified – to predict for a sample sharing the same data-generating parameters but different initial conditions. As shown, meta-hybridVAE significantly improved the identifiability of the abstract latent variable zN (the highest MCC close to 1) and via which, moderately improved the accuracy of the physics-based latent variable zP (the lowest MSE). Very importantly, while both baseline models identified zP with a reasonable accuracy, their MCC values were significantly lower – providing strong empirical support for the un-identifiability of the neural component of the hybrid-DGM as theorized in this paper. Also importantly, all hybrid models achieved comparable reconstruction MSE for observed x’s, where the meta-hybrid-VAE did not necessarily produce the lowest MSE. The ability of the identified hybrid-DGM to predict for different samples other than observed, however, varied significantly: while the meta-hybrid-VAE was able to deliver a MSE comparable to the reconstruction task, the two baseline models saw a deterioration of MSE by two magnitudes. Fig. 1 provides visual examples to demonstrate this performance difference between reconstruction and prediction tasks, stressing the importance to 1) look beyond reconstruction performance for evaluating hybrid-DGMs, and 2) consider prediction of unobserved samples as a potential surrogate for identifiability measures when ground-truth of latent variables is not available (real data settings in Section 6.2). Interestingly, among the three datasets, the advection-diffusion system appeared to be relatively simple to identify as the baselines did not exhibit as significant a degradation of performance in the more difficult identifiability metrics (MCC and prediction-MSE). This is potentially because the missing component from the prior physics has a small effect. In comparison, the MCC and prediction-MSE of the baseline models were significantly poorer in double pendulum, suggesting a difficult missing physics to be identified. Regardless, meta-hybrid-VAE was able to obtain significant margins of improvements compared to baselines across all datasets. Predictions over Longer Time Domains: Fig. 2 shows the prediction performance of the three models when predicting the trajectory of pendulum movement beyond the time domain used to identify the model, with a visual sample of forced damped pendulum system. As shown, while all models’ performance deteriorated as they predicted outside the time domain used in training, the presented meta-hybrid-VAE demonstrated significantly smaller drop in performance and slower rate of deterioration as the prediction horizon increased. This provides strong evidence regarding the importance of the identifiability, even for the neural component, of a hybrid model for the purpose of predictive tasks. OOD Performance: Figure 3 (left) summarizes the performance of the three models in settings where either the physics-based component zP or the neural component zN was outside the training distribution for forced damped pendulum system. Results demonstrated that, meta-hybrid-VAE obtained the strongest performance in OOD settings for both the identification of the physics and neural components. Details of the OOD parameters can be found in the Appendix A.1. Empirical Verification of the Condition for Identifiability: Theorem 1-iv) stipulates that the identifiability of the hybrid-DGM depends on the number of independent conditional densities observed, in relation to the dimension of the latent factors to be identified. To experimentally 8

Figure 2: (Left) The performance of predictions over longer time domains; (Right) A visual example of forced damped pendulum system.The dark part is the training time domain, and the light part is the time domain outside training (longer time domain).

Figure 3: (left) Results of OOD; (right) Results of the condition for identifiability. verify this, we evaluated the identifiability metric MCC of the hybrid-DGM when learning over data generated from an increasing number of distinct parameter values governing the pendulum system (Equation (2)), i.e., number of tasks in meta-learning. The results in Figure 3 (right) showed that the MCC metric exceeded 0.85 when 7 tasks appeared in the training data and stabilized when 8 or more tasks appeared. This agreed with the theorem that, for the pendulum system that had a 3-dimensional parameter vector to be identified in the neural component of the hybrid model with a Gaussian assumption of 2-dimensional sufficient statistics, a minimum of 3*2+1 = 7 distinct “tasks” is needed to identify the system. Additional Ablation Analyses of Meta-Hybrid-VAE: In Appendix C.1, we provided ablation results on the pendulum system to demonstrate that the identifiability results of the presented metahybrid-VAE were minimally affected by the number of parameters to be identified, as long as the theoretical condition for identifiability is met. In Appendix C.2, we provided further ablation studies to show that the physics-based component did not suffer from the type of un-identifiability discussed in this paper as the neural component, although the meta-formulation did moderately improve the accuracy of the estimation of the physics-based parameters. 6.2

Real Data of Double Pendulum

Experimental settings: We used the dataset of a double pendulum introduced by [23], which contains 21 videos of the pendulum. Each run lasts approximately 40 seconds and is recorded at 400Hz. We extracted the position of the pendulum limbs from each frame with elementary computervision tools. We divided each video into observations of x’s to consist of 20 temporal frames with a sampling frequency of 100 Hz. Because there is no clear indication of which samples belong to the same data-generation process, we use 7 samples preceding the current query sample as the context samples. We generated 10000, 3400, and 3400 training, validation, and test samples, respectively. We adopted Equation (19) as the physics-component of the hybrid-DGM, using the known lengths of the two arms (L1 = 91mm and L2 = 70mm) and assuming m̃ = 1. The total energy of the double 9

Figure 4: Examples of reconstruction and generation results on real double pendulum data.

pendulum decreases over time in all videos, which is not explained by this physics model. Nor does the physics model consider potential vibrations or errors in extracting the arms’ positions, which is expected to be learned by the neural component of the hybrid-DGM with the dimension of zN = 4. Results: As ground-truth generative factors are not available in the real data, we relied on prediction MSE as a surrogate for the identifiability metric as informed by results in Sections A.1-A.3. While all models delivered comparable reconstruction MSE, meta-hybrid-VAE was able to achieve a prediction MSE at a level similar to reconstruction while the two baselines experienced a deterioration by 2-3 magnitudes. Visual examples are provided in Fig. 4. This strong performance in real data provides solid evidence for the presented meta-formulation to establish identifiability of hybrid-DGMs.

7

Conclusions & Discussion

In this paper, we probe the un-identifiability of hybrid-DGMs with unconditioned priors, and present meta-learning as a novel solution to construct identifiable hybrid-DGMs – both results were supported by strong theoretical and empirical evidence. Moving forward, this work can be improved as follows. Comprehensive ablation studies of the meta-hybrid-VAE are needed to examine the effect of several key hyperparameters, such as the size of k-shot context set and the dimension of zN . The latter was set to the number of true generative factors in synthetic experiments, and its effect on model performance needs to be further examined. Note that this setting was identical across baselines, thus it had no effect on the significant margins of improvements seen by meta-hybrid-VAE compared to baselines. Future ablation studies should also delve into the benefit of hybrid-DGMs in comparison to purely physics-based and purely neural DGMs, which we considered out of scope of the current study that focused on establishing the identifiability of hybrid-DGMs both theoretically and empirically. Regarding the generality of the meta-formulation as a solution to identifiable DGMs beyond the hybrid formulation, Appendix D presented an initial investigation on the performance of the presented meta-formulation on a general non-hybrid VAE on a synthetic dataset of non-stationary Gaussian timeseries, in comparison to the identifiable-VAE [6] constructed using known class labels to condition the generative model. Future works could investigate this effect in a broader context. Finally, while the meta-formulation foregoes explicitly observed auxiliary variable, it does assume the ability to pair context and query samples from the same data generation process. This is a common assumption in meta-learning and, in scenarios where this knowledge is not evident, simply pairing by preceding temporal samples was shown to be effective in our real-data experiments. In scenarios where no information is available for pairing, the presented meta-solution will not be applicable.

Acknowledgments and Disclosure of Funding This study is supported by the National Key Research and Development Program of China(No: 2020AAA0109502), the Talent Program of Zhejiang Province (No: 2021R51004), the NIH NHLBI grant R01HL145590, the NIH NINR grant R01NR01830, and NSF grant OAC-2212548. 10

References [1] Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124012, 2021. [2] Naoya Takeishi and Alexandros Kalousis. Physics-integrated variational autoencoders for robust and interpretable generative modeling. Advances in Neural Information Processing Systems, 34:14809–14821, 2021. [3] Ziming Liu, Yuanqi Du, Yunyue Chen, and Max Tegmark. Physics-augmented learning: A new paradigm beyond physics-informed learning. In NeurIPS 2021 AI for Science Workshop, 2021. [4] Antoine Wehenkel, Jens Behrmann, Hsiang Hsu, Guillermo Sapiro, Gilles Louppe, and JörnHenrik Jacobsen. Robust hybrid learning with expert augmentation. Transactions on Machine Learning Research, 2023. [5] Zhaozhi Qian, William R. Zame, Lucas M. Fleuren, Paul Elbers, and Mihaela van der Schaar. Integrating expert ODEs into neural ODEs: Pharmacology and disease progression. Advances in Neural Information Processing Systems, 34:11364–11383, 2021. [6] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207–2217. PMLR, 2020. [7] Aapo Hyvärinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in Neural Information Processing Systems, 29, 2016. [8] Aapo Hyvärinen and Hiroshi Morioka. Nonlinear ICA of temporally dependent stationary sources. In International Conference on Artificial Intelligence and Statistics, pages 460–469. PMLR, 2017. [9] Aapo Hyvärinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In International Conference on Artificial Intelligence and Statistics, pages 859–868. PMLR, 2019. [10] Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26):eaau5872, 2019. [11] Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. TossingBot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36(4):1307–1319, 2020. [12] Florian Golemo, Adrien Ali Taiga, Aaron Courville, and Pierre-Yves Oudeyer. Sim-to-real transfer with neural-augmented robot simulation. In Conference on Robot Learning, pages 817–828. PMLR, 2018. [13] Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S Sukhatme. NeuralSim: Augmenting differentiable simulators with neural networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9474–9481. IEEE, 2021. [14] Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04385, 2020. [15] David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In International Conference on Learning Representations, 2021. [16] Xiangchen Song, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric Xing, and Kun Zhang. Temporally disentangled representation learning under unknown nonstationarity. Advances in Neural Information Processing Systems, 36:8092–8113, 2023. 11

[17] Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvärinen. Disentangling identifiable features from noisy data with structured nonlinear ICA. Advances in Neural Information Processing Systems, 34:1624–1633, 2021. [18] Hermanni Hälvä and Aapo Hyvärinen. Hidden Markov nonlinear ICA: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pages 939–948. PMLR, 2020. [19] Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014. [20] Bharath Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Aapo Hyvärinen, and Revant Kumar. Density estimation in infinite dimensional exponential families. Journal of Machine Learning Research, 18(57):1–59, 2017. [21] Christopher M. Bishop. Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, New York, 2006. [22] Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvärinen. ICE-BeeM: Identifiable conditional energy-based deep models based on nonlinear ICA. Advances in Neural Information Processing Systems, 33:12768–12778, 2020. [23] Alexis Asseman, Tomasz Kornuta, and Ahmet Ozcan. Learning beyond simulated physics. In NeurIPS 2018 Workshop on Modeling and Decision-Making in the Spatiotemporal Domain, 2018.

12

A

Details on Experimental Data Generation

A.1

Forced Damped Pendulum

We generated data from the full physics function in Equation (2), with observed x = [φ0 , . . . , φi , . . . , φT ], φi = φ(i∆t), T = 49, ∆t = 0.05. We randomly sampled the initial condition φ0 ∈ (−1.57, 1.57] (with φ̇0 = 0 fixed). Physics parameters [ω, ξ, A, ϕ] to be identified were varied as follows: ω ∈ [0.785, 3.14] , ξ ∈ [0.0, 0.8], A ∈ [0, 40] and ϕ ∈ [3.14, 6.28]. We generated total 60000 samples and separated them into a training, validation, and test sets with 4,0000, 10000, and 1,0000 samples, respectively. In all hybrid-DGMs, we considered the first two terms n in Equation (2) as the partial physics, thus zP = ω and zN with a dimension of 3 is expected to recover [ξ, A, ϕ]. This setting induces an additive error in the prior physics. In OOD setting, we set OOD zP as ω ∈ [3.14, 3.5] and OOD zN as ξ ∈ [0.8, 1.0], while keeping the rest of the parameters unchanged. A.2

Advection-diffusion System

We generate data from the following full physics function: ∂T ∂2T ∂T −a 2 +b =0 (18) ∂t ∂s ∂s where s is the spatial dimension. The solution T (s, t) was observed as x := [T0 , . . . , TT ], where Tj := [T (0, tj ), . . . , T (smax , tj )]T at tj := j∆t is uniformly distributed in space with a dimension of 20. We set T = 49, ∆t = 0.02, the boundary condition as T (0, t) = T (smax , t) = 0, and the initial condition as T (s, 0) = c sin(πs/smax ) with c ∈ [0.5, 1.5]. Physics parameters to be identified [a, b] were sampled as: a ∈ [0.01, 0.1] and b ∈ [0.01, 0.1]. We generated 20000, 5000, and 5000 training, validation, and test samples, respectively. In all hybrid-DGMs, we set partial physics as the first two terms of Equation (18), thus zp = a and zN with a dimension of 1 is expected to recover b. This represents an additive error in the prior physics. A.3

Double Pendulum

We generated data from the following full physics function: #   " G sin φ2 cos(φ1 −φ2 )−sin(φ1 −φ2 )(L1 φ̇21 cos(φ1 −φ2 )+L2 φ̇22 )−(m̃+1)G sin φ1 d φ̇1 L1 (m̃+sin(φ1 −φ2 )2 ) = (m̃+1)(L1 φ̇2 sin(φ1 −φ2 )−G sin φ2 +G sin φ1 cos(φ1 −φ2 ))+L2 φ̇2 cos(φ1 −φ2 ) sin(φ1 −φ2 ) 1 2 dt φ̇2 L2 (m̃+sin(φ1 −φ2 )2 )

(19) where m̃ = m1 /m2 . The observed x := [(φ1,0 , φ2,0 ), . . . , (φ1,j , φ2,j ) . . . , (φ1,T , φ2,T )], where (φ1,j , φ2,j ) is the value of a solution at tj := j∆t with T = 79, ∆t = 0.025, L1 = L2 = 1.0. We randomly sampled the initial condition φ1,0 , φ2,0 ∈ (−1.57, 1.57] (with φ̇1,0 = φ̇2,0 = 0). Physical parameters [G, m̃] to be identified were sampled as follows: G ∈ [5.0, 15.0] and m̃ ∈ [0.5, 1.5]. We generated 20000, 5000, and 5000 training, validation, and test samples, respectively. In all hybrid-DGMs, we set partial physics to be Equation (19) with m̃ = 1, thus zp = G and zN with a dimension of 1 is expected to recover m̃. This induces a multiplicative error in the prior physics.

B

The proof of Theorem 1

Theorem 1: Assume we observe data sampled from pθ (x, zP , zN |Ds ) as defined in Equations (13)-(15) with parameters ϕ = (θ, T, λ ζ ). Assume the following holds: (i) The set {x ∈ X |φε (x) = 0} has measure zero, where φε is the characteristic function of the density pε defined in pε (x − Fθ [fP , fNθ ; zP , zN ]). (ii) The hybrid mixing function Fθ is injective. (iii) The sufficient function Ti,j are differentiable almost everywhere, and linearly independent on any subset of X of measure greater than zero. s (iv) There exist de + 1 distinct context sets D0s , D1s , . . . , Dde such that the de × de matrix L defined as follows is invertible: λ(Ds,1 ) − λ(Ds,0 ), . . . , λ(Ds,de ) − λ(Ds,0 )) L = (λ 13

(20)

Then the parameters ϕ = (θ, T, λ ζ ) are ∼A -identifiable. Proof: This proof mainly refers to [6]. For simplicity, we denote Fθ as f , Fθ̃ as f̃ , λ ζ as λ and λ) such that pf ,T,λλ (x|Ds ) = λ. Suppose we have two sets of parameters (f , T, λ ) and (f̃ , T̃, λ̃ λ ζ̃ as λ̃ s s pf̃ ,T̃,λ̃λ (x|D ) for all pairs (x, D ). Then: Z Z s pT,λλ (z|D )pf (x|z)dz = pT̃,λ̃λ (z|Ds )pf̃ (x|z)dz (21) Z Z Z Z pT,λλ (z|Ds )pε (x − f (z))dz = pT̃,λ̃λ (z|Ds )pε (x − f̃ (z))dz (22) Z Z Z Z pT,λλ (f −1 (x̄)|Ds )volJf −1 (x̄)pε (x − x̄)dx̄ = pT̃,λ̃λ (f̃ −1 (x̄)|Ds )volJf̃ −1 (x̄)pε (x − x̄)dx̄ X

X

(23) Z

Z p̃T,λλ,f ,Ds (x̄)pε (x − x̄)dx̄ =

Rd

Rd

p̃T̃,λ̃λ,f̃ ,Ds (x̄)pε (x − x̄)dx̄

(24)

(p̃T,λλ,f ,Ds ∗ pε )(x) = (p̃T̃,λ̃λ,f̃ ,Ds ∗ pε )(x)

(25)

F [p̃T,λλ,f ,Ds ](ω)φε (ω) = F [p̃T̃,λ̃λ,f̃ ,Ds ](ω)φε (ω)

(26)

F [p̃T,λλ,f ,Ds ](ω) = F [p̃T̃,λ̃λ,f̃ ,Ds ](ω)

(27)

p̃T,λλ,f ,Ds (x) = p̃T̃,λ̃λ,f̃ ,Ds (x)

(28)

By taking the logarithm on both sides of equation (28) and replacing pT,λλ by its expression from (9), we get: log volJf −1 (x) +

n k X X (log Qi (fi−1 (x)) − log Zi (Ds )) + Ti,j (fi−1 (x)λi,j (Ds )) = i=1

j=1

n k X X log volJf̃ −1 (x) + (log Q̃i (f˜i−1 (x)) − log Z̃i (Ds )) + T̃i,j (f˜i−1 (x)λ̃i,j (Ds )) i=1

(29)

j=1

s Let D0s , D1s , . . . , Dde be the context sets provided by assumption (iv) of the Theorem, and define s s λ(Dl ) = λ (Dl ) − λ (D0s ). We plug each of those Dls in (29) to obtain de + 1 such equations. We λ̄ subtract the first equation for D0s from the remaining de equations to get for l = 1, . . . , de: E X X Z̃i (D0s ) Zi (D0s ) D ¯ λ(Dls ) + λ(Dls ) + log log T(f −1 (x)), λ̄ = T̃(f̃ −1 (x)), λ̃ (30) s Zi (Dl ) Z̃i (Dls ) i i

λ. Define bl = Let L be the matrix defined in assumption (iv), and L̃ similarly defined for λ̃ P Z̃i (D0s )Zi (Dls ) i log Zi (D s )Z̃i (D s ) and b the vector of all bl for l = 1, . . . , de. Expressing equation (30) for 0 l all points ul in matrix form, we get: LT T(f −1 (x)) = L̃T T̃(f̃ −1 (x)) + b

(31)

We multiply both sides by the transpose of the inverse of LT from the left to find: T(f −1 (x)) = AT̃(f̃ −1 (x)) + c

(32)

where A = L−T L̃ and c = L−T b. Now by the definition of T and according to assumption (iii), its Jacobian exists and is an de × d matrix of rank d. This implies that the Jacobian of T̃ ◦ f̃ −1 exists and is of rank d and so is A. We distinguish two cases: (1) If e = 1, then this means that A is invertible (because A is d × d). (2) If e > 1, define x̄ = f −1 (x) and Ti (x̄i ) = (Ti,1 (x̄i ), . . . , Ti,e (x̄i )). According to Lemma 3 in [6], for each i ∈ [1, . . . , d] there exist e points x̄1i , . . . , x̄ei such that (T′i (x̄1i ), . . . , T′i (x̄ei )) are linearly independent. Collect those points into e vectors (x̄1 , . . . , x̄e ), and concatenate the e Jacobians JT (x̄l ) 14

Table 2: The results of the different dimensions of unknown parameters The number of parameters 2 3 4

MSE of zP ↓

MCC ↑

MSE of x (Rec)

MSE of x (Pre)

1.22(0.08)e-2 3.62(0.01)e-2 1.59(0.07)e-2

0.97(0.01) 0.99(0.00) 0.99(0.00)

1.22(0.08)e-2 1.29(0.05)e-2 2.91(0.06)e-2

2.02(0.13)e-2 5.01(0.00)e-2 6.85(1.11)e-2

Table 3: Comparison results of Non-meta and Meta Method Non-Meta zP Meta zP

MSE of zP ↓ 9.17(1.33)e-3 5.63(0.42)e-3

MCC ↑ 0.99(0.00) 0.99(0.00)

MSE of x (Rec) 3.28(0.17)e-2 3.18(0.19)e-2

MSE of x (Pre) 8.82(2.03)e-2 3.37(0.00)e-2

evaluated at each of those vectors horizontally into the matrix Q = (JT (x̄1 ), . . . , JT (x̄e )) (and similarly define Q̃ as the concatenation of the Jacobians of T̃(f −1 (f̃ (x̄))) evaluated at those points). Then the matrix Q is invertible (through a combination of Lemma 3 and the fact that each component of T̃ is univariate). By differentiating equation (32) for each xl , we get (in matrix form): Q = AQ̃. The invertibility of Q implies the invertibility of A and Q̃. λ) Hence, equation (32) and the invertibility of A mean that (f , T, λ) ∼A (f̃ , T̃, λ̃

C

Ablation Analyses of Meta-Hybrid-VAE

C.1

Effect of the Dimension of Unknown Parameters

Here, we provided ablation results on the pendulum system to demonstrate that the identifiability results of the presented meta-hybrid-VAE were minimally affected by the number of parameters to be identified, as long as the theoretical condition for identifiability is met. The results can be seen in Table 2. C.2

Meta-component on cp

Here, we provided further ablation studies to show that the physics-based component did not suffer from the type of un-identifiability discussed in this paper as the neural component, although the metaformulation did moderately improve the accuracy of the estimation of the physics-based parameters. The results can be seen in Table 3.

D

Results on General Non-Hybrid VAEs

We compare our meta-formulation of the non-hybrid VAE (Meta-VAE) compared to the original VAE and identifiable VAE (iVAE) on a synthetic datasets about non-stationary Gaussian time-series in [6]. To evaluate the performance of the method, we compute the MCC between the original sources and the corresponding latent samples from the learned posterior. The results can be seen in Table 4. Table 4: The results on General Non-Hybrid VAEs MCC

E

VAE 0.67(0.04)

iVAE 0.91(0.03)

Meta-VAE 0.88(0.00)

Strong MCC results

We also evaluate proposed model on a more strong identifiable relation ∼P defined as follow: Definition 3:

Let ∼ be the equivalence relation on Φ defined the same as 16:

(θ, T, λζ ) ∼ (θ̃, T̃, λζ̃ ) ⇔ ∃A, c : T(Fθ−1 (x)) = AT̃(Fθ̃−1 (x)) + c, ∀x ∈ X 15

(33)

Figure 5: Visual samples of learned latent variables between different runs. Table 5: The results of strong MCC

APHYNITY Hybrid-VAE Meta-Hybrid-VAE

Forced Damped Pendulum Strong MCC Strong MCC Ground Truth Different Runs 0.38(0.02) 0.81(0.08) 0.29(0.00) 0.80(0.01) 0.55(0.02) 0.90(0.06)

Advection-Diffusion System Strong MCC Strong MCC Ground Truth Different Runs 0.86(0.05) 0.83(0.04) 0.80(0.12) 0.79(0.11) 0.97(0.03) 0.96(0.00)

Double Pendulum Strong MCC Strong MCC Ground Truth Different Runs 0.50(0.00) 0.73(0.00) 0.50(0.00) 0.72(0.00) 0.96(0.00) 1.00(0.00)

If A is a block permutation matrix, we denote this relation by ∼P . The theory according to ∼P is following: Theorem 2:

(e ≥ 2) Assume the hypotheses of Theorem 1 hold and that e ≥ 2. Further assume:

(2.i) The sufficient statistics Ti,j in Equation 15 are twice differentiable. (2.ii) The mixing function Fθ has all second order cross derivatives. then the parameters (θ, T, λ ζ ) are ∼P -identifiable. Theorem 3:

(e = 1) Assume the hypotheses of Theorem 1 hold and that e = 1. Further assume:

(3.i) The sufficient statistics Ti,1 not monotonic. (3.ii) All partial derivatives of Fθ are continuous. then the parameters (θ, T, λ ζ ) are ∼P -identifiable. The proof of Theorem 2 and 3 can be found in [6]. Then to evaluate the performance on ∼P , we calculate the strong MCC between learned latent variables and ground truth variables (Strong MCC Ground Truth) and strong MCC between runs of the model (Strong MCC Different Runs) [6, 9, 22]. The results can be found in Table 5 and we provide a visualization example of forced damped pendulum system to show the relation of learned latent variables between two different runs.

16

Related documents

• On the Identifiability of Hybrid Deep Generative Models: Meta-Learning as a Solution#1448264
• Interpretable Meta-Learning for Multi-Objective Chemical Search#1403143
• CosmoFlow: Using Deep Learning to Learn the Universe at Scale#620154
• DP18792: Experimental Evidence on the Learning Impact of Generative AI#357673
• Using physics-informed enhanced super-resolution generative adversarial networks for subfilter modeling in turbulent reactive flows#1328974