Human-Machine Collaboration on Generative Meta-Learning: Model and Algorithm
Midhun Parakkal Unni 1 2 3 Samuel Kaski 3 4 5
1. Introduction
Abstract
Artificial intelligence and humans increasingly collaborate to solve real-world problems, either explicitly or implicitly (Reverberi et al., 2022; Mosqueira-Rey et al., 2023). Making human-AI collaboration explicit enables scientific predictions about the behaviour of these collaborative systems and the possibility of systematically controlling them. In this work, we address a central challenge in machine learning—enabling trained models to generalise to unseen data distributions at deployment—by leveraging humanAI collaboration. Empirical surveys show that incorporating human expertise into active learning loops enhances model performance, particularly by mitigating data scarcity through the strategic selection of informative samples (Budd et al., 2021). Given recent advances in generative models (Lin et al., 2024), ‘leveraging interactions with generative models to probe expert knowledge’ emerges as a promising technique (Christiano et al., 2017). This highlights the need for a principled investigation of such systems and the development of algorithms that integrate generative models with human-AI collaboration.
arXiv:2607.00926v1 [cs.LG] 1 Jul 2026
Generalizing machine learning models to environments that differ from their training distribution remains a critical hurdle, particularly when data from the target domain is entirely or partially unavailable. We propose Generative MetaLearning with Human Feedback (GMHF), a novel framework that bridges this domain gap by leveraging expert intuition to guide data synthesis. Grounded in a theoretical analysis of generalization error, we derive bounds demonstrating that aligning the distribution of generated data with human beliefs regarding the target physics significantly mitigates risk. GMHF operationalizes this insight by employing a Conditional Neural ODE (cNODE) as a generative digital twin, coupled with a Reinforcement Learning (RL) agent. The agent iteratively refines the latent physical parameters of the generated trajectories based on feedback, effectively steering the meta-learner toward the unobserved target distribution. Empirical validation on a nonlinear Duffing oscillator shows that GMHF substantially reduces deployment loss as expert reliability increases, and that the divergence between generated and target data falls under reliable feedback, directly corroborating the divergence-minimisation mechanism predicted by our theory. Further experiments on a non-dynamical probabilistic model confirm that the framework extends beyond ODE-governed systems, establishing human-AI collaboration as a rigorous catalyst for robust generalisation under distribution shift.
The standard independent and identically distributed data (i.i.d.) assumption in machine learning (ML) (Loh, 2017) has been increasingly relaxed to facilitate the study of distributional shifts (Ben-David et al., 2010; Liu et al., 2021). Recent research approaches the distribution shift from multiple perspectives, such as utilising causal structures, identifying invariant features across domains (Arjovsky et al., 2019), and addressing selection bias (Liu et al., 2021). However, the explicit role of human collaboration remains largely unaddressed in most existing methods. In particular, distribution shifts in time series data deserve special attention (Wu et al., 2025) because (a) they occur across diverse domains, including healthcare and financial analysis, and (b) the temporal structure allows for modelling them via dynamical systems and related causal structures (Chen et al., 2018; Rubanova et al., 2019; Parakkal Unni and Menon, 2023; Capiński and Kopp, 2012)(e.g., kinematics of human limbs to Black–Scholes models). Furthermore, substantial human expertise is available for a subset of time-series tasks (e.g., medical doctors interpreting ECG data), which can be leveraged to enhance model perfor-
1
School of Computer Science, University of Sheffield, Sheffield, UK 2 Centre for Machine intelligence, University of Sheffield, UK 3 Department of Computer Science, University of Manchester, Manchester, UK 4 ELLIS Institute Finland, Helsinki, Finland 5 Department of Computer Science, Aalto University, Espoo, Finland. Correspondence to: Midhun Parakkal Unni <[email protected]>. Preprint. July 2, 2026.
1
GMHF
mance (Tonekaboni et al., 2019).
man feedback to refine datasets.
Our work introduces a mathematical model of humangenerative AI collaborative systems and derives theoretical bounds that characterise their behaviour, subject to assumptions. We then analyse a Gaussian approximation of the model to obtain closed-form solutions. Building on these theoretical insights, we propose a practical algorithm and evaluate its performance in simulated scenarios. While our primary evaluation is on dynamical time series, we additionally verify that the framework applies without modification to a non-dynamical probabilistic model, indicating that the approach is not specific to ODE-governed systems.
• We empirically demonstrate the effectiveness of GMHF on a nonlinear Duffing oscillator, and show that the divergence between generated and target distributions decreases under reliable feedback, empirically corroborating the divergence-minimisation mechanism identified by our theory. • We further show that GMHF generalises beyond ODEgoverned time series by applying it, without architectural modification, to a non-dynamical probabilistic model with an explicit source-to-target extrapolation gap (Appendix C.1).
We introduce a novel approach to out-of-distribution generalisation that leverages human feedback during dataset generation, guided by a dynamic system. Our method, Generative Meta-Learner with Human Feedback (GMHF), integrates generative modelling, reinforcement learning (RL), and human feedback to optimise the quality of generated datasets. By incorporating human evaluations into dataset refinement, we aim to improve the generalisation performance of models in partially observed deployment scenarios.
2. Mathematical Modelling 2.1. Problem Setting We consider a deployment scenario where a generative model is used to synthesise training data for an unobserved target environment. The process is governed by a humanin-the-loop protocol: the generator proposes a candidate trajectory (e.g., a time-series simulation), and a human expert evaluates its plausibility against their mental model of the target physics. The expert acts as a filter, either accepting the sample if it aligns with the target dynamics or rejecting it if it diverges from them. This selective acceptance shapes the final distribution of training data and, therefore, the deployment accuracy, effectively aligning the generative process with the latent ground truth through iterative feedback.
GMHF framework in detail The GMHF framework comprises three key components: a generative model, an AIagent, and a meta-learner. We use a conditional neural ODE (cNODE) architecture that learns a dynamical system from simulated time-series data (e.g., a Duffing oscillator), thereby serving as a data generator. While there are several approaches to conditioning a neural ODE (Dang et al., 2023; Wang et al., 2025), we take an approach that creates a disentangled action space for the agent (a reinforcement learning (RL) agent). The generative model produces candidate training data. The AI-agent refines these datasets by transforming the latent variables, guided by human feedback. This feedback is incorporated into the generation loop, enabling the AI-agent to iteratively improve dataset quality, which the meta-learner then uses to learn predictive models.
2.2. Modelling In this section, we develop a theoretical framework to characterise the interaction between the generative model and the human agent. Specifically, we propose a set of assumptions formalising the generative distribution, the human-based likelihood function, and the associated priors.
The GMHF approach is particularly relevant in time series analysis, where the underlying dynamics may change over time or across environments. By leveraging human feedback, we aim to enhance the robustness of machine learning models to distribution shifts, ultimately improving their performance in real-world applications. The main contributions of this paper are as follows:
We begin by defining the relationship between the generative model and the target deployment distribution. Assumption 2.1 (Common Support). Let Dg be the generative distribution and Dt be the target distribution with densities pg (·) and pt (·) respectively. We assume that the generative model covers the support of the target distribution, such that pt (x) > 0 implies pg (x) > 0 for all x ∈ X .
• We propose a mathematical model of the human feedback-based data selection process and derive theoretical bounds that highlight its potential to improve generalisation in deployment scenarios.
The generative model is conditional and parametrised by θ. We model the human agent as a filter that accepts a subset of the generated data based on a utility function puθ̃ (·), which represents their belief about the latent structure of the deployment scenario. This yields the following distribution for the accepted samples:
• We introduce the Generative Meta-Learner with Human Feedback (GMHF) framework, which integrates generative modelling, reinforcement learning, and hu-
Assumption 2.2 (Human Acceptance). Given generative 2
GMHF
density pg (x | θ), the human agent accepts samples proportional to a utility function puθ̃ (x). This utility is parametrised by θ̃, representing the human’s belief about the true latent dynamics, yielding the accepted distribution: pa (x|θ) :=
puθ̃ (x) pg (x|θ) za
where,
ph (θ̃ | x)p̃t (x) zh
(2)
ph (θ̃|x)p̃t (x)pg (x|θ) z
(7)
δh := DKL (pt ∥p̃t )
(8)
Lemma 2.8. For a hypothesis h, the target error ϵt (h) and the source error ϵs (h), one could bound the target error as follows. This lemma is based on (Ben-David et al., 2010). ϵt (h) ≤ ϵs (h) + min{Epa [| fs (x) − ft (x) |], Ept [| fs (x) − ft (x) |]}+ | Ept (h(x) − ft (x)) − Epa (h(x) − ft (x)) |
(3)
This leads to the following proposition for the density of the accepted samples. Proposition 2.6. The density of the accepted samples is given by, pa (x) := pa (x|θ) :=
Lg := Ept [log pg (x|θ)]
The estimated divergence between pt and pa provides a useful means to bound the target error for a hypothesis h, denoted as ϵt (h). According to the decomposition in Lemma 2.7, this divergence is minimized under the following conditions: (1) human belief closely matches the true target distribution (minimising δh ); (2) the human’s internal model is well aligned with the true target observations (maximising Lh ); and (3) the generative model accurately fits the target data (maximising Lg ).
where zh is a normalisation constant. Definition 2.4. A hypothesis is a function defined from the set X to Y, h : X → Y, where X , Y are sets defining the inputs and labels, respectively. Definition 2.5. The error of a hypothesis h according to the distribution Di and labelling function f is given by: ϵx (h, f ) := Ex∼Di [|h(x) − f (x)|]
(6)
(1)
We assume that human utility is grounded in their understanding of the domain’s latent structure. Assumption 2.3 (Human Mental Model). The human utility puθ̃ (x) decomposes into a prior belief p̃t (x) over the target distribution and a likelihood ph (θ̃ | x), representing the human’s inference of the latent parameters given the data: puθ̃ (x) :=
Lh := Ept [log ph (θ̃|x)]
(9)
Lemma 2.9 (Bounding the expectation term). Let pt (x) be the target distribution and pa (x) the acceptance distribution as defined in the previous assumptions. Let h : X → R be a bounded function such that |h(x) − ft (x)| ≤ M for all x ∈ X . Then, the expectation mismatch term
(4)
∆ := Ept [(h(x) − ft (x))] − Epa [(h(x) − ft (x))] (10)
Proof. Obtained by substituting (2) in (1) and choosing an appropriate normalising constant.
can be bounded as follows: p |∆| ≤ M 2DKL (pt ∥pa ).
The divergence between the probability distribution of the accepted samples and the target probability distribution is provided in the following lemma. Proofs of the theorems are given in the Appendix A.1. Lemma 2.7.
Theorem 2.10. For a hypothesis h, the target error ϵt (h) and the source error ϵs (h), one can modify the bound for the target error as follows, where, h : X → R be a bounded function such that |h(x) − ft (x)| ≤ M
DKL (pt ∥pa ) =
ϵt (h) ≤ ϵs (h) + min{Epa [| fs (x) − ft (x) |],
δh |{z}
Ept [| fs (x) − ft (x) |]} q + M 2(δh − Lh − Lg + log z)
Human belief mismatch
−
(11)
Lh |{z}
(12)
Human latent inference
−
Lg |{z}
Generator data fit
+
log z |{z}
Takeaways from the general model: One could minimise the deployment error by doing one or a combination of the following,
(5)
Normalization
3
GMHF
Proposition 2.13 (KL Divergence from Target to Accepted Distribution). Let pt = N (µt , Σt ) and pa = N (µa , Σa ). The KL divergence is:
• Improving human understanding of the deployment data, that is, by reducing the human belief mismatch. • Accurate estimation of the latent variable that controls the deployment data. The human’s acceptance of a data point depends on the human’s belief about the correct distribution of the data, and also on how the latent variable controls the generation of the trainable datasets.
DKL (pt ∥ pa ) = log
|Σa | − k + tr(Σ−1 a Σt ) |Σt |
+ (µa − µt )T Σ−1 a (µa − µt ),
(17)
where k is the dimensionality of x.
• Generator accurately fitting the data.
Proof. This is the standard formula for the KL divergence between two multivariate Gaussian distributions.
• One should also note that terms in the bound can cancel out and compensate each other; therefore, learning the user model or personalising the algorithms to a user can be helpful.
Corollary 2.14 (KL Divergence in the Isotropic Case). Let all distributions be isotropic: pt = N (µt , σt2 I),
We first consider the case where the user model, the generative model, and the target distribution are all multivariate Gaussian. This permits closed-form analysis of the accepted distribution and associated divergence measures.
σh2 σg2 µg µh 2 + . , σ = a σh2 σg2 σh2 + σg2 and the KL divergence becomes: " ! # σt2 (σh2 + σg2 ) σh2 σg2 k DKL (pt ∥ pa ) = log −1+ 2 2 2 σh2 σg2 σt (σh + σg2 ) µa = σa2
Proposition 2.11 (Accepted Distribution under Gaussian Models). Let the user model be puθ̃ (x) = N (x; µh , Σh ), the generative model be pg (x) = N (x; µg , Σg ), and the target distribution be pt (x) = N (x; µt , Σt ). Then the distribution of accepted samples is also Gaussian:
,
−1 µa = Σa Σ−1 h µh + Σg µg
(13)
.
(14)
∆w := ws − wt ,
,
−1 Σ−1 t µt − Σh µh
2 z := fs (x) − ft (x) = ∆⊤ w x + δb ∼ N (µtrana , σtrana )
where: µtrana = ∆⊤ w µa + δb ,
(15)
.
δb := bs − bt
Then the difference in label predictions is:
Proposition 2.12 (Optimal Generative Parameters to Match Target). Suppose we require the accepted distribution pa to exactly match the target distribution pt = N (µt , Σt ). Then, for a given user model (µh , Σh ), the generative model parameters (µg , Σg ) that achieve this are: −1
(18)
Proposition 2.15 (Expected Disagreement between Linear Labelling Functions). Let fs (x) = ws⊤ x + bs and ft (x) = wt⊤ x + bt be scalar-valued linear labelling functions defined on inputs x ∈ Rd , and suppose the input distribution is Gaussian: x ∼ N (µa , Σa ). Define the parameter differences
Proof. Follows from the identity for the product of two Gaussian densities. The resulting unnormalised product is proportional to a Gaussian with the given mean and covariance.
−1 Σg = Σ−1 t − Σh
1 ∥µa − µt ∥2 . 2σa2
Proof. Follows from substituting isotropic forms into the general KL divergence formula and simplifying trace and determinant terms.
where −1
+
pa (x) = N (x; µa , Σa ),
µg = Σg
pg = N (µg , σg2 I).
Then the accepted distribution has the following parameters:
2.3. Gaussian special case
−1 Σa = Σ−1 h + Σg
puθ̃ = N (µh , σh2 I),
2 σtran = ∆⊤ w Σa ∆ w a
The expected disagreement under the input distribution pa (x) is r µ2trana 2 Ex∼pa [|fs (x) − ft (x)|] = σtrana exp − 2 + π 2σtrana µtrana µtrana 1 − 2Φ − σtrana (19)
(16)
Proof. Substitute Σa = Σt and µa = µt into the formulas from the previous proposition and solve for Σg and µg . The −1 inverse (Σ−1 t −Σh ) must exist and be positive definite. 4
GMHF
where Φ is the cumulative distribution function (CDF) of the standard normal distribution: Z z 2 1 Φ(z) = √ e−t /2 dt. 2π −∞
Algorithm 1: GMHF Meta-Learner Optimization Input :Data distribution p(x|y), initial parameters Output :Optimized π ∗ and M∗ψ Initialize cNODE model V Train cNODE on p(x|y) Generate datasets xtr , xdep Initialize AI-agent policy π(a|z, fh )
Proof. Follows from computing the expected value of the absolute value of a univariate Gaussian variable z ∼ 2 N (µtrana , σtran ). a
while not converged do for each episode do AI-agent selects action a from π(a|z, fh ) Apply transformation Ta on latent variables z → z′ Generate new data x′ ∼ pθ (x|z ′ , y) Train meta-learner Mψ on xtr Compute accuracy and feedback fh Compute reward R(x′ ) = λa · Acc + λh · fh Optimize AI-agent policy π(a|z, fh ) end end return π ∗ , M∗ψ
Remark 2.16. In this approximation, the ‘concepts’ fs and ft are assumed to be deterministic. A non-deterministic case could be accommodated with the additional term Λ as follows: µtrana = ∆⊤ w µa + δb ,
2 σtran = ∆⊤ w Σa ∆ w + Λ a
where Λ is the covariance term. Remark 2.17. With a similar argument as given in Prop. 2.15 one could obtain the expected disagreement over the target distribution as follows:
r
µ2trant 2 exp − 2 + Ex∼pt [|fs (x) − ft (x)|] = σtrant π 2σtrant µtrant µtrant 1 − 2Φ − σtrant (20)
It provides a mechanism for the human, modelled as an agent, to alter the latent parameters such that the accepted distribution matches the target distribution. This implicitly accounts for human understanding of the target distribution. The outcome can vary depending on the algorithm’s parameters and the human expert’s knowledge. The algorithm consists of (a) a generative model, (b) an AI-Agent, (c) A meta-learner, and (d) a human model with three different levels of expert knowledge (EK). A flowchart describing the algorithm is shown in Fig. 1 and the pseudocode is provided in Algorithm 1.
where, µtrant = ∆⊤ w µt + δb ,
2 σtran = ∆⊤ w Σt ∆w . t
Subsequently, by substituting Eq. 18, 19, 20 in 43 one obtains the bound for the special Gaussian case. The key takeaway from the isotropic case is that one must adjust the generative model in accordance with the human model to minimise the bound. The Proposition 2.12 shows that Σg and µg depend on both the target and human models. Therefore, the aim of the generative model is not to match the target distribution exactly, but rather to collaborate with the human to create an accepted distribution that approximates it.
Generative Model
Ta (z)
AI-Agent π(a|z, fh ))
Sampling Data Generated Dgen
l(.)
fh
Generated Data
3. Proposed Algorithm
Learner E(x,y)∼Dgen ℓ(Mψ (x), y)
Motivated by Theorem 2.10, which bounds deployment risk via the divergence terms in Lemma 2.7, we introduce the GMHF algorithm. By optimising a policy against the human acceptance probability (Proposition 2.6), the agent iteratively steers the generative process toward the target distribution, explicitly minimising the generalisation gap identified in our theoretical analysis.
l(.)
Human-Agent fh = F (xtr , xdep , Accuracy).
Figure 1. GMHF Schematic. The AI-Agent modulates the latent space via Ta (z) to synthesize training data Dgen for a Meta-Learner. A Human-Agent filters deployment outcomes and provides feedback fh . The policy π is optimised using a composite reward R derived from human feedback and learner loss. Solid lines denote mandatory information flow; dashed lines denote optional transfers.
5
GMHF
f (x, τ, λ, β), where β represents the shared physical laws (e.g., conservation of momentum) and λ captures instancespecific properties (e.g., stiffness). The state evolution is given by:
Generative Model: The proposed cNODE architecture serves as inductive bias for the neural ODE, the weights of which are initialised, and then trained on the source datasets {(x, y) s. t. (x, y) ∼ p(x|y)p(y)} to generate the distribution Dg with a density pg (.) conditioned on the parameters of the ODE (The architecture diagram and a brief description is provided in the Appendix. Fig. 4).
Z t x(t) = x(t0 ) +
AI agent: The AI agent interacting with the environment is modelled as an MDP defined by the tuple M = (S, A, Ta , r, ρ0 , γ). The latent variable of the generative model, which in this case is the parameter λ, forms the states S and A, representing the actions. The state transition is governed by the function Ta : S × A 7−→ S. The human provides reward as feedback, and the reward function is denoted by the mapping r : S × A × S 7→ {−1, 1}. The starting state distribution is ρ0 , and γ denotes the discount factor. A policy π : S 7−→ A is defined as a mapping from the states to actions. In the case we consider, the latent variables of the generative model are transformed by Ta for every action generated by the AI-agent.
(23)
We train this model to minimize the trajectory loss L(β) across a distribution of simulation parameters p(λ): Z t L(β) := E ℓ x(t0 ) + f (x, τ, λ, β) dτ, xsim (t) t0
(24) Here, xsim (t) denotes the ground-truth trajectories obtained from the numerical simulator corresponding to parameters λ, and ℓ(·) is a distance metric. While conditioning ODEs is a known technique in scientific computing, we leverage it here to create a disentangled action space for the RL agent. By embedding λ as an input, we force the network to learn the differential equation’s structural form while exposing λ as a “knob”. This turns the black-box generative process into a causal proxy, allowing the agent to traverse the physical manifold in directions that are interpretable to the human expert.
Human Model. We model human feedback as a stochastic reward r ∈ {−1, 1} conditioned on the perceived system configuration θ̃. This parameter vector θ̃ represents the governing factors, known to the human. The reward follows a Bernoulli distribution biased by the expert’s reliability ph : ph , θ̃ ∈ Θvalid , p(r = 1 | θ̃) = (21) 1 − p , otherwise, h
DL vs EK
0.1
DL
This formulation incentivises the agent to discover the physically plausible configuration space Θvalid . Formally, the effective reward distribution marginalises over the human’s R perception noise: p(r | θ) = p(r | θ̃)p(θ̃ | θ) dθ̃. For experimental evaluation, we quantify the level of Expert Knowledge (EK) directly via the reliability parameter, setting EK = ph .
0.35 0.30 0.25 0.20 0.15 0.10 0.05
Meta-learner: We employ the Reptile algorithm (Nichol et al., 2018) to train a regressor that predicts system parameters from trajectory data (xtr , ytr ). The meta-learner’s zero-shot prediction accuracy on the target domain serves as the primary metric for evaluating the utility of the generated datasets.
EK
0.9
(22)
0.5
p(r = −1 | θ̃) = 1 − p(r = 1 | θ̃).
f (x, τ, λ, β) dτ t0
Figure 2. Deployment loss (DL) across expert-knowledge (EK) levels. EK values below 0.5 correspond to adversarial guidance, while values above 0.5 represent helpful human expertise. A marked decrease in deployment loss is observed as EK increases from 0.5 to 0.9, demonstrating the strong positive impact of supportive expert input on system performance.
3.1. Conditional Neural ODE (cNODE) as a Causal Proxy To enable precise control over the generative process, we employ a Conditional Neural ODE (cNODE) framework. Unlike standard Neural ODEs that learn a fixed dynamics function, we explicitly parametrise the vector field as
4. Experimental Results Model Problem We evaluate our method on the Duffing oscillator, a standard benchmark for non-linear dynamics. 6
GMHF
LR
(a) Meta-Learning Rate
Alpha
mented via Stable Baselines3 (Raffin et al., 2021). The task is formulated as a continuous control problem with onedimensional state and action spaces (S, A ⊆ R). The state st corresponds to the current stiffness parameter k, (refer to the Eq. 25), and the agent learns a deterministic policy π(st ) to adjust k toward the target configuration. To isolate the impact of expert guidance, we set the weight λa = 0 and feedback weight λh = 1, effectively optimising the policy solely against the stochastic binary reward r ∈ {−1, 1} from the human model.
3.0
0.1 0.19 0.28 0.37 0.46 0.54 0.63 0.72 0.81 0.9
0.1
2.0
0.2
1.0
0.3
DL
DL
0.4
0.5
0.5
DL vs Alpha
0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.0
DL vs LR 0.6
(b) Nonlinearity (α)
Main Result: Impact of Expert Knowledge The human model parameters θ̃ is set to k, and the set Θvalid = [0, 10]. To isolate the contribution of human guidance, we evaluate deployment loss across varying levels of expert knowledge (EK). As shown in Figure 2, the system is highly sensitive to feedback quality. We observe that under adversarial (EK = 0.1) or random (EK = 0.5) guidance, the loss remains high with significant variance. However, improving reliability to EK = 0.9 triggers a sharp transition, reducing loss considerably (to < 0.06). This confirms that a threshold of high-fidelity expert feedback is necessary to successfully steer the generative process toward the target distribution. The outliers are removed before creating the plots in Fig. 2, 3a and Fig. 3b for clarity.
Figure 3. Parameter Analysis (a) Deployment loss (DL) minimizes at LR ≈ 0.54, balancing adaptation speed with stability. (b) Loss varies non-monotonically with physical nonlinearity α; it peaks at α ≈ 1 but decreases for stronger non-linearities (α > 1), where high stiffness constrains the effective state-space.
The system evolution is governed by: 0 1 0 ẋ = x+α 3 −k/m −c/m x | {z } | {z } K
(25)
f (x)
where x = [x, ẋ]⊤ represents the state (position and velocity). The mass is fixed (m = 1), while the stiffness k and damping c constitute the parameter set λ = {k, c} used to generate diverse datasets. The coefficient α controls the system’s nonlinearity; we vary α ∈ [0, 3] in increments of 0.5 to evaluate the framework’s performance across regimes from linear to highly nonlinear.
Meta-Update We analyse the sensitivity of the metalearning rate LR, which governs the Reptile-based update rule: β ← β + LR(β̃ − β). As shown in Figure 3a, the system exhibits a sharp transition rather than a standard convex trade-off. At lower rates (LR < 0.5), the model suffers from high deployment loss and significant variance, indicating a failure to retain the physical constraints identified during the inner loop. We observe a critical threshold at β ≈ 0.54, where the loss collapses to its minimum (< 0.05). Beyond this point, the performance remains robustly stable, confirming that aggressive meta-updates are required to effectively bridge the domain gap.
We parameterise the vector field f (x, t, λ, β) using a fourlayer Multi-Layer Perceptron (MLP) with 64 hidden units per layer. To construct the training corpus, we sampled physical parameters from uniform distributions—specifically stiffness k ∼ U[0, 10] and damping c ∼ U [0, 1]—and numerically integrated the dynamics (refer to the Eq. 25) with a fixed mass m = 1. Our design emulates a realistic deployment setting in which analytical governing equations are inaccessible, and the system must be learned solely from observational data. In this context, the cNODE acts as a data-driven surrogate model (or ”Digital Twin”), capable of synthesising physically consistent trajectories under expert supervision. The model successfully captures diverse oscillator behaviours governed by the stiffness and damping parameters (See Appendix Fig. 5 for the samples). Duffing oscillator serves as a canonical proxy for complex real-world applications, modelled by damped harmonic oscillators, ranging from physiological applications to (e.g., cardiac rhythms) to structural health monitoring in mechanical systems (Fonkou and Savi, 2023; Yin and Epureanu, 2006).
Effect of System Nonlinearity We further analysed the robustness of our method against the physical complexity of the underlying dynamics, controlled by the cubic nonlinearity parameter α (see Eq. 25). As shown in Figure 3b, the linear case (α = 0) presents the lowest difficulty, resulting in minimal deployment loss. As the system introduces nonlinearity, the prediction task becomes increasingly challenging, peaking at α ≈ 1 where the deviation from simple harmonic motion is most pronounced. Interestingly, for stronger non-linearities (α > 1), the loss begins to decrease. This suggests that while moderate nonlinearity introduces complex anharmonic dynamics, significantly higher stiffness terms effectively constrain the system’s effective state space by rapidly restoring the oscillator to equilibrium, acting as a stabilising force that facilitates easier adaptation for
We trained the agent using the Deep Deterministic Policy Gradient (DDPG) algorithm (Lillicrap et al., 2015), imple7
GMHF
the meta-learner.
in high deployment loss and significant variance. In our Reptile-based implementation, the meta-update acts as a gating mechanism for new physical knowledge; if LR is too low, the agent effectively discards the valid constraints identified during the inner loop, failing to bridge the domain gap. We observe a transition at LR ≈ 0.54, where the loss collapses to a minimum. This indicates that the meta-learner requires aggressive updates to effectively retain the expert’s directional signal.
We further show, on the Duffing oscillator, that the agent reduces the generated-to-target divergence under reliable feedback (Appendix B). Additional experiments on a nondynamical probabilistic model demonstrate that GMHF is not restricted to ODE-governed time series (Appendix C.1).
5. Discussion and Conclusion
Two further results reinforce these conclusions. First, tracking the divergence between generated and deployment data on the Duffing oscillator (Appendix B, Fig. 6) shows that reliable feedback drives this divergence down over training. This provides direct empirical support for the divergenceminimisation mechanism that Theorem 2.10 bounds in principle, indicating that the theory informs the algorithm. Second, applying GMHF without architectural modification to a non-dynamical probabilistic model with an explicit source-to-target extrapolation gap (Appendix C.1) confirms that the approach is not specific to ODE-governed dynamics. Moreover, the meta-learning rate exhibits opposite optimal regimes across the two benchmarks, indicating that the framework adapts to the differing demands of each task rather than imposing a single fixed behaviour (Fig. 8).
The experimental results validate GMHF as a robust framework for bridging domain gaps in scientific machine learning. Inspired by the theoretical bounds from Section 2 and with the empirical evidence obtained in Section 4, we derive critical insights into how the agent learns to collaborate with human experts. The core innovation of GMHF is its fundamental shift in objective: rather than passively fitting a static dataset, the algorithm learns how to collaborate to synthesise a superior one. Unlike traditional surrogates that minimise reconstruction error against a fixed ground truth, our agent learns a policy π(a|z, fh ) that actively manipulates latent parameters to satisfy a human utility function. This paradigm is theoretically anchored in Proposition 2.12, which demonstrates that accurate target approximation requires the generator to align its density with the human’s acceptance constraints. In our specific experimental setting (λa = 0), the agent does not correct for human bias but rather converges to the expert’s implicit physical model. The sharp reduction in loss at high reliability levels (EK > 0.5, Fig. 2) empirically validates this mechanism: the agent leverages sparse feedback to navigate the latent space, effectively solving an analytically intractable inverse search problem for the expert. This establishes a collaborative equilibrium where the machine provides exhaustive exploration and the human provides physical verification—a synergy that achieves results neither could attain in isolation.
6. Limitations and Future Work A primary limitation of our study is that, while it spans both a dynamical benchmark (the Duffing oscillator) and a nondynamical one (the linear-cubic probabilistic model), both are simulated systems with low-dimensional latent parameters and a known ground truth. We adopted such systems because they permit rigorous validation of the algorithm against a controllable target. The framework is adaptable in principle to richer modalities. Future work could extend the approach to image or tabular data by replacing the generator with a domain-appropriate causal structure, such as a diffusion model for images or a Structural Causal Model (SCM) for tabular reasoning. A key challenge in those settings is that a controllable generator with a meaningful latent parameter, together with a reliable ground truth for training it, may be harder to obtain than in the synthetic systems studied here.
Our analysis of system nonlinearity points to a link between modelling parameters and statistical variance. The reduction in loss under strong nonlinearity (α > 1, Fig. 3b) can be explained via the isotropic KL divergence (Eq. 18). High stiffness implies that the target distribution variance is reduced, and this also reduces the distribution of the acceptance distribution, which humans are implicitly aware of. This, among other effects, can amplify the penalty for the mean mismatch in Eq. 18, increasing the loss. These low-variance regimes (very high and very low nonlinearity) can provide a steeper gradient for the meta-learner to learn efficiently, compared to the high-variance regime of the nonlinear system (medium nonlinearity).
While GMHF successfully bridges the generalisation gap, it relies on two key assumptions: (1) the existence of a shared physical model between source and target domains (e.g., both are oscillators), and (2) the availability of a nonadversarial expert (EK > 0.5). Our experiments show that performance degrades sharply if the expert’s reliability falls below random chance. Furthermore, the current interaction protocol assumes a stationary target; extending this to timevarying systems where parameters drift during deployment remains an open challenge for future research.
The ablation study on meta-learning rates reveals a critical threshold behaviour rather than a trade-off. As shown in Figure 3a, conservative update rates (LR < 0.5) result 8
GMHF
This work opens the door for more nuanced human-machine collaboration. While our current model relies on binary feedback and a probabilistic utility function, future iterations will incorporate complex human models capable of providing multimodal feedback (e.g., natural-language critiques of physical plausibility). Ideally θ̃ depends on the observed trajectory, we simplify θ̃ ≈ k for experiments (Eq. 25). In future work, one could build a more realistic human model that generates the parameters θ̃ from observations of trajectories. Furthermore, extending the “stiffness as constraint” insight to high-dimensional systems, such as fluid dynamics or cardiac modelling, remains a promising avenue for making intractable inverse problems solvable via human-guided meta-learning.
disease progression forecasting: a case study on covid-19. In Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, pages 3914–3925, 2023. Rodrigue F Fonkou and Marcelo A Savi. Heart rhythm analysis using nonlinear oscillators with duffing-type connections. Fractal and Fractional, 7(8):592, 2023. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. Lequan Lin, Zhengkun Li, Ruikun Li, Xuliang Li, and Junbin Gao. Diffusion models for time-series applications: a survey. Frontiers of Information Technology & Electronic Engineering, 25(1):19–41, 2024.
Acknowledgments and Disclosure of Funding This work was supported by UKRI Turing AI WorldLeading Researcher Fellowship (EP/W002973/1). SK was also supported by EU funding ERC ODD-ML 101201120 and the Research Council of Finland Flagship programme: Finnish Center for Artificial Intelligence FCAI.
Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-ofdistribution generalization: A survey. arXiv preprint arXiv:2108.13624, 2021.
References
Po-Ling Loh. On lower bounds for statistical learning theory. Entropy, 19(11):617, 2017.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
Eduardo Mosqueira-Rey, Elena Hernández-Pereira, David Alonso-Rı́os, José Bobes-Bascarán, and Ángel Fernández-Leal. Human-in-the-loop machine learning: a state of the art. Artificial Intelligence Review, 56(4): 3005–3054, 2023.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151–175, 2010.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018.
Samuel Budd, Emma C Robinson, and Bernhard Kainz. A survey on active learning and human-in-the-loop deep learning for medical image analysis. Medical image analysis, 71:102062, 2021.
Midhun Parakkal Unni and Prathyush P Menon. Modelling and analysis of parkinsonian gait. Nonlinear Dynamics, 111(1):753–769, 2023. Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stablebaselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1– 8, 2021. URL http://jmlr.org/papers/v22/ 20-1364.html.
Marek Capiński and Ekkehard Kopp. The Black–Scholes Model. Cambridge University Press, 2012. Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Carlo Reverberi, Tommaso Rigon, Aldo Solari, Cesare Hassan, Paolo Cherubini, and Andrea Cherubini. Experimental evidence of effective human–ai collaboration in medical decision-making. Scientific reports, 12(1):14952, 2022.
Ting Dang, Jing Han, Tong Xia, Erika Bondareva, Chloë Siegele-Brown, Jagmohan Chauhan, Andreas Grammenos, Dimitris Spathis, Pietro Cicuta, and Cecilia Mascolo. Conditional neural ode processes for individual
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularlysampled time series. Advances in neural information processing systems, 32, 2019. 9
GMHF
This could be further expanded as,
Sana Tonekaboni, Shalmali Joshi, Melissa D McCradden, and Anna Goldenberg. What clinicians want: contextualizing explainable machine learning for clinical end use. In Machine learning for healthcare conference, pages 359–380. PMLR, 2019.
Z =
pt (x) log
pt (x)z dx ph (θ̃|x)p̃t (x)pg (x|θ)
(31)
Z =
Xiaoda Wang, Yuji Zhao, Kaiqiao Han, Xiao Luo, Sanne van Rooij, Jennifer Stevens, Lifang He, Liang Zhan, Yizhou Sun, Wei Wang, et al. Conditional neural ode for longitudinal parkinson’s disease progression forecasting. In Abstract in the Organization for Human Brain Mapping Annual Meeting, 2025.
pt (x)(log pt (x) − log ph (θ̃|x) − log p̃t (x) − log pg (x|θ) + log z)dx
(32)
= DKL (pt (x)∥p̃t (x)) − Ept log ph (θ̃|x) − Ept log pg (x|θ) + log z
(33)
Lemma A.2 (Bounding the expectation term). Let pt (x) be the target distribution and pa (x) the acceptance distribution as defined in previous assumptions. Let h : X → R be a bounded function such that |h(x) − ft (x)| ≤ M for all x ∈ X . Then, the expectation mismatch term:
Xin Wu, Fei Teng, Xingwang Li, Ji Zhang, Tianrui Li, and Qiang Duan. Out-of-distribution generalization in time series: A survey. arXiv preprint arXiv:2503.13868, 2025. Shih-Hsun Yin and Bogdan I Epureanu. Structural health monitoring based on sensitivity vector fields and attractor morphing. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 364(1846):2515–2538, 2006.
∆ := Ept [(h(x) − ft (x))] − Epa [(h(x) − ft (x))] (34) can be bounded as follows: p |∆| ≤ M 2DKL (pt ∥pa ).
(35)
Proof. We start by expressing the expectation mismatch in terms of the total variation distance:
A. Appendix A.1. Proofs
|Ept [h(x) − f (x)] − Epa [h(x) − f (x)]| ≤ M · 2 · TV(pt , pa ), (36)
Lemma A.1. DKL (pt ∥pa ) =
where the total variation distance is given by: Z 1 TV(pt , pa ) = |pt (x) − pa (x)|dx. 2
δh |{z}
Human belief mismatch
−
Lh |{z}
Human latent inference
−
Applying Pinsker’s inequality:
Lg |{z}
r
Generator data fit
+
TV(pt , pa ) ≤
log z |{z}
(26)
1 DKL (pt ∥pa ), 2
we obtain the final bound: p |∆| ≤ M 2DKL (pt ∥pa ).
Normalization
where, Lh := Ept [log ph (θ̃|x)]
(27)
Lg := Ept [log pg (x|θ)]
(28)
δh := DKL (pt ∥p̃t )
(29)
(38)
(39)
Theorem A.3. For a hypothesis h, the target error ϵt (h) and the source error ϵs (h), one could modify the bound for the target error as follows, where, h : X → R be a bounded function such that |h(x) − ft (x)| ≤ M .
Proof.
ϵt (h) ≤ ϵs (h) + min{Epa [| fs (x) − ft (x) |], Z
DKL (pt (x) || pa (x)) =
(37)
pt (x) log
pt (x) dx pa (x)
Ept [| fs (x) − ft (x) |]} q + M 2(δh − Lh − Lg + log z)
(30)
10
(40)
GMHF
Proof. From lemma 2.7, 2.8 and 2.9 ϵt (h) ≤ ϵs (h) + min{Epa [| fs (x) − ft (x) |], Position
Ept [| fs (x) − ft (x) |]} + | Ept (h(x) − ft (x)) − Epa (h(x) − ft (x)) | (41) ≤ ϵs (h) + min{Epa [| fs (x) − ft (x) |], Ept [| fs (x) − ft (x) |]} p + M 2DKL (pt ∥pa )
(42) (43)
= ϵs (h) + min{Epa [| fs (x) − ft (x) |], Ept [| fs (x) − ft (x) |]} q + M 2(δh − Lh − Lg + log z)
1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00
Position Trajectories generated by cNODE k=4.39, c=0.95 k=6.41, c=0.44 k=0.74, c=0.64 k=8.81, c=0.07 k=4.66, c=0.88 k=6.50, c=0.47 k=6.74, c=0.65 k=2.91, c=0.67 k=0.41, c=0.29 k=0.39, c=0.04
0
200
400
Time
600
800
1000
Figure 5. Position trajectories generated by the cNODE for varying stiffness (k) and damping (c) parameters. The distinct curves demonstrate the model’s ability to generalise across different physical regimes while keeping mass fixed at m = 1 and α = 0.5.
(44)
A.2. Architecture of the cNODE and Samples This section provides the architecture diagram of the cNODE (Appendix Fig. 4) and the trajectories it generates (Appendix Fig. 5).
this divergence implicitly. Under reliable feedback, the agent is expected to guide the generative process so that accepted samples increasingly resemble the deployment distribution. To assess whether this mechanism operates in practice, and to determine whether the theory genuinely informs the algorithm rather than merely motivating it, the divergence between the generated data and the deployment distribution is tracked throughout training.
Dense
Figure 6 presents the Wasserstein distance between generated samples at each episode and a reference sample from the deployment distribution, under both reliable (EK = 0.9) and uninformative (EK = 0.5) expert feedback. With reliable feedback, the divergence decreases over training as the agent learns to generate data aligned with the deployment regime. In contrast, under uninformative feedback, the divergence remains high and shows no consistent downward trend. These results provide direct empirical evidence that the human feedback channel guides the generative process toward the target distribution, consistent with the divergence-minimisation mechanism identified in the theoretical analysis.
Concatenate Dense
N layers Dense
Figure 4. Architecture of the Conditional Neural ODE (cNODE). The network learns the vector field conditioned on system parameters λ, allowing the generation of trajectories that respect varying physical properties
C. Additional experiments: A probabilistic model C.1. Linear-cubic probabilistic model benchmark To complement the dynamical Duffing oscillator and demonstrate that GMHF is not tied to ODE-governed systems, we introduce a second benchmark based on a static probabilistic model with a coupled latent structure. Each task is parametrised by a task-level latent slope m and offset c, and consists of N = 20 paired observations (yi , zi )N i=1 generated from a shared latent input xi . The full generative
B. Divergence Analysis The theoretical analysis (Lemma 2.7) bounds deployment error in terms of the divergence DKL (pt ∥pa ) between the target distribution and the distribution of human-accepted samples. The GMHF algorithm is constructed to minimise 11
GMHF
so the signal carrying xi scales with |m3 − m|. This coefficient is small in the source regime and substantially larger at the target, so the optimal (yi , zi ) → xi mapping differs between regimes. A meta-learner trained only on source data must therefore extrapolate, unless the agent is guided to explore the target regime — the role of the human feedback channel.
Wasserstein Distance
35 30 25 20 15 10 5 0
The generative model We note that for this benchmark the generative model shares the exact functional form of the true data-generating process; the outcome is therefore independent of a misspecified generator and the agent instead steers the latent slope m toward the deployment regime. This isolates the latent-steering role of GMHF from the additional challenge of generator misspecification present in the Duffing setting, where the cNODE is a learned approximation of the true dynamics.
EK = 0.9 EK = 0.5
0
100 200 300 400 500 Episode
Figure 6. Wasserstein distance between generated and deployment data as a function of training episode, for reliable (EK = 0.9) and uninformative (EK = 0.5) expert feedback. Reliable feedback drives the generated distribution toward the deployment distribution, reducing the divergence; uninformative feedback does not.
Reference conditions via expert-reliability ablation. Rather than introducing external baselines that would require different architectural assumptions, we use the expert reliability EK itself to define reference conditions, keeping every component of the system identical except the reward signal. At EK = 0.5 the human feedback carries no information about the underlying parameter, so the system reduces to GMHF with effectively random reward, a natural no-information baseline. At EK = 0.1 the feedback is consistently inverted, and at EK = 0.9 the human supplies reliable guidance. Figure 7 reports deployment loss across these three regimes: high expert reliability (EK = 0.9) yields substantially lower deployment loss than the no-information condition (EK = 0.5), isolating the contribution of expert knowledge.
model is m ∼ U(ma , mb )
(45)
c ∼ U (ca , cb )
(46)
xi ∼ N (0, σx2 ) (y) (z) εi , εi ∼ N (0, σε2 )
(47)
(y) yi = m xi + c + εi (z) zi = m3 xi + c + εi
(48) (49) (50)
for i = 1, . . . , N , with σx = 2 and σε = 0.5. The metalearner’s task is to recover the latent input xi from the observation pair (yi , zi ), treating m as the latent regime parameter modulated by the RL agent under human feedback, and c as a nuisance variable that must be implicitly marginalised.
Figure 8 shows the sensitivity of deployment loss to the meta-learning rate. Here the optimal regime is reversed relative to the Duffing oscillator: low rates yield the lowest loss, and performance degrades at higher rates. Both settings exhibit a sharp transition, but in opposite directions. This reversal is consistent with the differing demands of the two tasks: the Duffing task rewards retaining per-task structure (favouring aggressive updates), whereas the probabilistic task requires averaging across tasks to marginalise the shared nuisance offset c (favouring conservative updates).
Source and target distributions. We instantiate the source and target regimes so that deployment lies strictly outside the source support of m: Source:
m ∼ U (0.5, 1.0),
c ∼ U(0, 20)
(51)
Target:
m ∼ U (1.5, 1.7),
c ∼ U(0, 20)
(52)
This creates a genuine extrapolation gap: a meta-learner exposed only to source-regime slopes must rely on the GMHF agent, steered by human feedback, to generate training data at slope values the source distribution would not produce. The offset c is shared across regimes and acts purely as a nuisance variable.
Additional experimental details The agent is implemented using the Deep Deterministic Policy Gradient (DDPG) algorithm (Lillicrap et al., 2015) via the StableBaselines3 library (Raffin et al., 2021), with an MLP policy with 2 layers of 64 neurons each. The task is formulated as a continuous control problem with one-dimensional state and action spaces. The state st corresponds to the current latent parameter (stiffness k for the Duffing oscillator; slope m for the linear-cubic model), and the deterministic transition is
Identifiability under regime shift. Recovering xi requires eliminating the unknown offset c, achieved by subtracting Eq. 49 from Eq. 50: (z) (y) zi − yi = (m3 − m) xi + εi − εi , (53) 12
GMHF
DL vs EK
[0.1, 2.5].
50
DL
40 30 20 10
EK
0.9
0.5
0.1
0
140 120 100 80 60 40 20 0
DL vs Meta Learning Rate
0.1 0.19 0.28 0.37 0.46 0.54 0.63 0.72 0.81 0.9
Deployment Loss
Figure 7. Deployment loss versus expert knowledge ( EK = ph ) on the linear-cubic probabilistic benchmark. Reliable expert feedback (EK = 0.9) achieves the lowest deployment loss, while random feedback (EK = 0.5) provides no useful signal. Points show the mean across meta-learning-rate settings; bars indicate variability.
Meta Learning Rate
Figure 8. Deployment loss versus meta-learning rate on the linearcubic probabilistic benchmark. In contrast to the Duffing oscillator (Fig. 3a), performance is best at low meta-learning rates and degrades at higher rates. Both benchmarks exhibit a sharp transition, but in opposite directions. Points show the mean across expertknowledge settings; bars indicate variability.
st+1 = st + at , where at is the agent’s action. The action space is bounded to [−1, 10] for the Duffing oscillator and [−0.5, 0.5] for the linear-cubic model; the observation space is [−100, 100]. Each episode terminates when the state exits the valid range or the human returns a positive reward. The agent is trained for 500 environment steps. To isolate the contribution of human feedback, we set the accuracy weight λa = 0 and the feedback weight λh = 1, so the reward reduces to the binary human signal r ∈ {−1, +1}. For the linear-cubic model, the human returns a positive reward when the slope lies in the target band 1.4 < m < 1.8, and each episode terminates when m exits the range 13