Author pre-print. Publication accepted for the ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS’26). This version is subject to change and the camera-ready version may differ.
A Model-Driven Approach for Developing Families of Reinforcement Learning Environments Xiaoran Liu
Istvan David
McMaster University Hamilton, Canada [email protected]
McMaster University Hamilton, Canada [email protected]
arXiv:2606.20324v1 [cs.SE] 18 Jun 2026
Abstract Virtual training environments are software-intensive systems in which reinforcement learning (RL) agents learn, adapt, and demonstrate meaningful behavior. Virtual training environments offer a safe and cost-efficient alternative to training agents in real-world settings. However, to converge, most realistic RL problems require training in multiple, mostly similar but slightly different environments—i.e., families of environment variants. The typical development process of environment families is a labor-intensive and error-prone manual endeavor that does not scale well. To alleviate these issues, in this paper, we propose a model-driven approach for developing families of RL training environments. To obtain the family of environments, we develop an approach and prototype tool. In our approach, a hybrid genetic algorithm—a combination of population-based global search and heuristic local search—generates environment families. Mutations and constraints are expressed as model transformations and are operationalized into a search process by a state-of-the-art model transformation engine. We demonstrate the soundness of our approach in a wildfire mitigation scenario and curriculum learning—a particular learning paradigm that relies on environment families.
Keywords curriculum learning, genetic algorithms, machine learning, reinforcement learning, simulators, training environments ACM Reference Format: Xiaoran Liu and Istvan David. 2026. A Model-Driven Approach for Developing Families of Reinforcement Learning Environments. In Proceedings of ACM/IEEE 29th International Conference on Model Driven Engineering Languages and Systems (MODELS ’26). ACM, New York, NY, USA, 12 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
Reinforcement learning (RL) [64] has emerged as a popular machine learning technique recent years [19]. In RL, an autonomous agent explores the state space, and through trial and error, it learns beneficial actions to solve complex problems. This approach is particularly useful in problems where prior training data is scarce or unavailable, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. MODELS ’26, Malaga, Spain © 2016 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/XXXXXXX.XXXXXXX
e.g., control problems in robotics [2] and digital twins [13]. Modeldriven engineering (MDE), too, has adopted RL to address various technical problems, such as derivation of in-place model transformations [16] and complex transformation chains [11], automated repair of models [3], and inference of simulation models [12]. Virtual training environments are software systems in which agents learn, adapt, and demonstrate meaningful behavior [29]. By modeling real-world settings and allowing the agents to interact with this model, virtual training environments offer a safe and cost-efficient alternative to training in real-world settings. However, agents trained in a single, fixed environment typically fail to generalize beyond their training conditions and tend to memorize environment-specific behaviors rather than learning generalizable strategies [41]. Cobbe et al. [9] demonstrate that diverse environment scenarios are essential to adequately train RL agents. Most realistic RL problems, therefore, require the development of families of sufficiently diverse training environments [69] to improve key properties of the agent (e.g., faithfulness) or the learning process (e.g., training time); or often, simply just to converge to any solution at all. Multiple training paradigms leverage such environment families. A pertinent example is curriculum learning [4], in which agents are trained on progressively harder environments to allow early knowledge to improve learning performance. Such a structured progression helps RL agents learn faster, as well as improving the final performance of the trained model—similar to how humans learn concepts step-by-step. Other examples include multi-task learning [78], meta-learning [22], and domain randomization [73]. Recent studies indicate that agents may require exposure to thousands of environment variants to achieve meaningful generalization [9]. Despite this, developing a set of structurally valid training environments that are sufficiently similar but diverse at the same time, and sequencing these environments into an effective training process is still largely a manual effort [48]. Apart from being a clearly labor-intensive and error-prone endeavor, manual development simply does not scale with the requirements posed by modern RL applications [14]. Thus, automated engineering methods that alleviate human intervention are much sought-after. In this paper, we propose a model-driven approach for developing families of RL environments. In our approach, an RL expert develops the initial learning environment from which new variants are generated and together, form the family of environments that is required for the RL training. We use a hybrid genetic algorithm (GA) [46], i.e., a combination of population-based global search to generate candidate families of environments, and heuristic local search to identify the best-fitting family. The GA performs the global search by mutating the initial environment by mutation operators codified as model transformations, and structural constraints
Author pre-print. Publication accepted for MODELS’26.
MODELS ’26, October 4–9, 2026, Malaga, Spain
codified as graph queries—all operationalized in the state-of-the-art Viatra model transformation framework [5]. The search is guided by domain-specific diversity measures across individual environments in a population. The repeated application of mutating model transformations yields a potentially large but tractable number of varied and diverse models. Subsequently, constructing a family of environments is achieved by automatically selecting a subset of these varied models using an appropriate local search method (e.g., simulated annealing). Eventually, it is the lifting of structural and RL-specific properties to the modeled level that enables intelligent mutation strategies of the initial (and downstream) environments. We evaluate our approach on a curriculum learning problem. The results indicate that our approach improves learning performance while requiring minimal human intervention at the beginning of the process. We observe families of environments that are individually insufficient for training, but together, they facilitate effective 1 learning. Our work underscores the utility of the MDE body of knowledge in machine learning and responds to the calls for further research on MDE solutions for reinforcement learning [50].
2 Background 2.1 Reinforcement learning 2.1.1 Formal underpinnings. Reinforcement learning (RL) [64] is a machine learning paradigm in which an agent interacts with its environment to learn optimal strategies for sequential decision making. RL can be formalized as a Markov decision process (MDP) ⟨𝒮, 𝒜, 𝒫, ℛ⟩ [54]. 𝒮 denotes the set of observable states, 𝒜 is the set of available actions, 𝒫 ∶ 𝒮 ×𝒜×𝒮 → [0, 1] is the state transition ′ probability function, where 𝑃(𝑠 ∣ 𝑠, 𝑎) denotes the probability of ′ transitioning to state 𝑠 after taking action 𝑎 in state 𝑠, and ℛ ∶ 𝒮 × 𝒜 → R specifies the reward function. At each time step 𝑡, the agent observes state 𝑠𝑡 ∈ 𝒮, selects action 𝑎𝑡 ∈ 𝒜 according to policy 𝜋(𝑎∣𝑠), receives reward 𝑟𝑡 = ℛ(𝑠𝑡 , 𝑎𝑡 ), and transitions to ∗ state 𝑠𝑡 +1 . The agent aims to learn policy 𝜋 that maximizes the expected cumulative reward. 2.1.2 Training environments. RL agents learn through trial and error, but this does not necessarily have to happen in a real-world environment. Virtual and cyber-physical [38] RL training environments encapsulate a simulation infrastructure that provides a virtual world for agents to interact with. Such environments typically consist of an environment core that receives actions, updates the environment state, and returns observations and rewards; an underlying simulator that executes the probabilistic mechanisms representing the real-world phenomenon [55]; and a simulator adapter that translates the instruction set and results of a simulator to the world semantics of the agent [39]. Training environments are critical in the success of RL, as they directly shape the agent’s decision-making and actions [51]. Some of the widely used RL environments include Gymnasium [71] for single-agent tasks, PettingZoo [65] for multi-agent settings, and Isaac Gym [40] for robot learning. In this work, we use Gymnasium.
Liu and David
Environment
Reward Structure
Tentative data package: https://github.com/ssm-lab/models26-data. Full replication package to be published on Zenodo for the camera-ready.
Start
Water
Road
Vegetation
Fire engine
Terminating Burning
Goal
(b) Metamodel
Figure 1: Example Burning Forest instance and its metamodel 2.1.3 Curriculum learning. In this work, we use curriculum learning (CL) as the representative example of learning paradigms that rely on families of environments. CL is a training strategy in which the learning process is organized as a sequence of training criteria that evolve over time [74], typically from simpler to more complex settings [4]. This allows for learning elementary skills in lowercomplexity environments faster and using those acquired skills in higher-complexity environments later. The most common structure for a curriculum is a sequence curriculum, where the agent is trained in environments that progress from simpler to more complex. Curricula may also be represented as a directed acyclic graph (DAG) [66], in which vertices correspond to environments and edges indicate which should be trained before others.
2.2
Model-driven optimization and genetic algorithms
Model-driven optimization (MDO) applies principles from MDE to search-based optimization problems by representing candidate solutions as models [25, 26]. In this paradigm, the search space is defined over structured models that explicitly capture domain constraints and relationships. This makes MDO particularly wellsuited for problems where solution validity and structure are central concerns. In model-based MDO, candidate solutions are the models themselves, and optimization proceeds by directly transforming these models through MTs. To search this structured space, MDO often employs genetic algorithms (GA), an optimization technique inspired by natural evolution [15]. In GA, a population of candidate solutions is varied over several generations under selection pressure to produce solutions of increased quality over time. In GA-based MDO, the population is initialized with model instances, either randomly or by varying a single seed model [7]. In parent selection, parent models are probabilistically chosen from the population to undergo variation and produce offspring. Mutations are defined as MTs over an initial model. Survivor selection then decides which candidate models will be part of the next generation, which may be done based on evaluation. Evaluation assigns a real-valued measure to each model, often referred to as fitness. This process continues for several generations until termination. In this work, we use a hybrid GA that combines population-based global search and heuristic local search.
3 1
Agent
Tile
Forest
Non-terminating
(a) Burning Forest instance
State
Illustrative example
To illustrate our approach throughout the paper, we draw on a case in wildfire mitigation. The Burning Forest (Fig. 1) is a custom RL environment developed using Gymnasium. It is a topological grid
A Model-Driven Approach for Developing Families of Reinforcement Learning Environments
Table 1: Burning Forest reward structure Event
Consequence
Stepping on a road tile Stepping on a vegetation tile Reaching a water source Encountering a burning tile Reaching the goal tile
Continue episode Continue episode One-time reward; continue ep. Terminate episode Terminate episode
Reward 0 −0.1 +1 −1 +10
world that acts as the Environment on which an Agent trains. It consists of different tiles as its States, including the Start (top-left) and Goal (bottom-right) tiles, and Road, Vegetation, Burning, and Water tiles in between. The Agent navigates from the Start to the Goal of a rescue mission by selecting one directional action per step (up, down, left, right) while attempting to avoid burning tiles. Roads are preferred over paths through vegetation as the latter may damage the vehicle. Strategically placed Water reservoirs aid rescue missions and thus, should be visited by the agent during its action. Through trial and error, the agent gradually learns which actions are beneficial in a specific state to reach the goal without stepping on burning tiles. To guide the learning, the Reward structure specifies the reward associated with specific outcomes of chosen actions: +10 upon reaching the goal; −1 for stepping on a burning tile, +1 for reaching a water source for the first time; 0 for each step on a road tile; and −0.1 for each step on a vegetation tile, reflecting the cost of damaging the forest. This is summarized in Tab. 1. The RL expert trains the agent to navigate the Burning Forest. Training the agent on the actual problem may be too complex for the agent and may render the training inefficient due to delayed convergence to the learning objectives. For example, in Fig. 1, the density of burning tiles allows only two successful paths to the goal (through the water reservoir). The agent may need numerous training episodes to discover this path. To improve learning performance, the RL expert applies curriculum learning, i.e., trains the agent on a sequence of progressively more challenging environment variants, starting with an easier problem before exposing it to the hardest one (Fig. 1a).
4
Approach
Our approach relies on a GA to generate families of RL environments. GA is a metaheuristic technique that explores large search spaces by optimizing an objective (e.g., the complexity of environments). GA performs a global search by evolving an initial set of individuals via predefined mutation operators. For model-based generation, a GA can naturally use mutation operators expressed as model transformations to explore the space of potential environment models by guiding the search towards models that fit particular criteria (e.g., diversity of the family of environments). i– 3y iare Fig. 2 provides an overview of our approach. Steps 1y i y enacted by the Domain expert and the RL expert. In Step 1 , the Domain expert prepares the activities of the RL Expert by defining i, an RL expert—e.g., a suitable domain metamodel (MM). In Step 2y RL engineer or software developer—specifies the environment the agent is meant to learn on. This environment is treated as the most challenging one from which less complex variants are to be i generated. In Step 3y , the search process is configured by defining (i) the mutation operators or mutable states for mutation generation; (ii) the constraints for individual environment candidates and the family itself; (iii) the individual- and family-level measures (e.g.,
MODELS ’26, October 4–9, 2026, Malaga, Spain
diversity of a family of environments, complexity of an individual environment), and (iv) the key GA parameters. i, a GA generates the family of environments; and in In Step 4y i Step 5y , the family is sampled in accordance with the training strategy and RL framework-specific code is generated. Subsequently, the training process is ready to be executed. The full automation of iand Step 5y ienables RL experts to experiment with various Step 4y configurations of the GA (“frequent loop”) to obtain an environment family that is deemed appropriate. In some cases, more foundational changes may be required and the ML expert may redefine some constraints or mutations (“infrequent loop”). In the following, we elaborate on each of these five steps.
4.1
Domain modeling and other MDE tasks
i, the domain expert defines the domain metamodel. This In Step 1y metamodel offers a language to express the key characteristics of the RL environment in a subsequent step, as well as to define model transformations. The domain metamodel defines the states of the RL problem that are observable during training. The metamodel also relates these states through an appropriate formal structure, e.g., topological alignment in 2D and 3D problems, and logical alignment in more abstract state spaces, such as design space exploration. Example Domain (meta)modeling In Fig. 1b, the Forest environment and its various states are defined by the domain expert, as well as the agent, which is the Fire engine in this particular problem.
The metamodel is aligned with the GA metamodel. Each environment corresponds to an individual in the GA and is being gradually mutated as the algorithm progresses. (See Sec. 4.4.) In this step, domain-specific languages may be developed to aid the subsequent efforts of the RL expert. Notably, a DSL can help lift the specification of mutations and automate the task of manual transformation specification altogether. Such features are not in the scope of our current work.
4.2
Defining the initial environment
i, the RL expert defines the initial environment. This enIn Step 2y vironment is a fixed element of the eventually generated family of environments, and the starting point for generating the rest of the family. Different learning paradigms choose initial environments differently. For example, in curriculum learning, the initial environment typically corresponds to the most challenging environment; to the environment without any simulation randomization in domain randomization [69]; and to a representative task configuration (e.g., a single object with a basic manipulation skill) in multi-task learning [76]. Alternatively, the expert may specify a different starting configuration, e.g., the easiest one, from which variants are generated. For example, Florensa et al. [20] construct a curriculum over initial states by progressively moving the starting position from near the goal to increasingly distant locations. Our approach is able to accommodate any initial environment due to its ability to mutate environments both towards lower and higher complexity.
MODELS ’26, October 4–9, 2026, Malaga, Spain
1
Liu and David Legend
MDE tasks
Domain MM
uses
Domain expert
conform to conforms to 2
Initial environment
RL expert
Manual
GA MM
Conventional RL tasks
3
Automated
Frequent loop: experimentation with GA parameters
Define mutations or mutable states Define constraints
Configure GA
4
Generate env. family
5
Generate code
Define measures for obj. function Configuration of the search process
Artifact generation
Infrequent loop: experimentation with constraints and mutations
Figure 2: Overview of the approach Example Defining the initial environment In the running example, the environment model defines the Forest composed of Tiles, specialized into BurningTile, RoadTile, VegetationTile, WaterTile, StartTile, and GoalTile, with a FireTruck agent whose currentState references a single tile, as shown in Fig. 1b. The expert specifies the initial environment model as a Burning Forest containing 16 Tiles, with a StartTile at [0,0], a GoalTile at [3,3], a WaterTile at [2,2], and four BurningTiles, as shown in Fig. 1a.
The environment corresponds to the domain metamodel that has been previously defined by the domain expert. We see two ways to establish this correspondence. In a native mode, RL experts can define their environments through the API of the chosen RL framework and subsequently, a translation to an instance model typed by the domain metamodel is required. In an MDE mode, RL experts are provided with a suitable DSL to define their environments directly as an instance of the domain metamodel. (See Sec. 6.)
4.3
Configuration of the search process
i, the RL expert configures the search process by (i) definIn Step 3y ing the mutation operators or mutable states for the automated generation of mutation operators; (ii) defining the constraints the environments and the family must adhere to; (iii) defining the (domain-specific) measures for the objective function that drives the search for the optimal family of environments; and (iv) configuring the GA hyperparameters. We provide a pluggable architecture with well-defined interfaces to implement when defining components (i)–(iii) of the search process. 4.3.1 Defining or generating mutations. Mutations are defined in terms of model transformations (MTs), specified on the domain metamodel. Together with the model of the initial environment, these transformations span the entire search space, i.e., the set of all environment models reachable by applying mutation operators to the initial environment model.
RHS replaces it with a RoadTile at the same coordinates [x,y], as shown below. Applying this rule decreases burning tile density, potentially increases the number of feasible paths in the environment— i.e., decreases the complexity of the learning task. 1 pattern burningTile(forest : Forest, tile : BurningTile) { 2 Forest.states(forest, tile); 3 }
Listing 1: Example Viatra graph query (“Burning tile”) 1 val changeBurningToRoad = 2 createRule(burningTile).name("changeBurningToRoad").action[ 3 val newTile = createRoadTile 4 forest.replaceTile(tile, newTile)].build
Listing 2: Example mutation expressed as a Viatra MT rule that reacts to the match of the graph query in Listing 1
Alternatively, the RL expert may specify the set of mutable states 𝑆 𝑆 𝑀 to generate the full graph of mutations 𝑆 𝑀 → 2 𝑀 . 4.3.2 Defining constraints. The RL expert defines the constraints that all generated environments in the family must satisfy. These constraints are used for the validation of environments produced during the environment generation process, filtering out infeasible candidates. Constraints are expressed as graph queries and subsequently, registered through the ConstraintService. This results in a validation and filtering step in the GA process. Constraints are checked on new mutations. Upon violating a constraint, the mutating model transformation is rolled back through EMF’s TransactionalEditingDomain facility—a component in EMF that borrows 2 transactional semantics to reading and writing model resources. As a requirement, we demand that every environment is solvable. Example Defining constraints The expert specifies that some states are to be preserved in every mutation: the StartTile is to be kept at [0,0], the GoalTile at [3,3], and the WaterTile at [2,2]. For this, they first specify a graph query that matches on the state.
Example Defining mutations 2
The expert defines a model transformation rule changeBurningToRoad. The LHS matches a BurningTile in the Forest, and the
https://github.com/eclipse-emfservices/emf-transaction/blob/master/ bundles/org.eclipse.emf.transaction/src/org/eclipse/emf/transaction/ TransactionalEditingDomain.java
A Model-Driven Approach for Developing Families of Reinforcement Learning Environments
1 pattern startTileFixedLocation(forest: Forest, tile: StartTile) { 2 Forest.states(forest, tile); 3 Tile.x(tile, x); 4 Tile.y(tile, y); 5 check((x == 0) && (y == 0)); 6 }
Listing 3: Viatra graph query to define an invariant Then, the query is registered as a constraint: 1 contraintService.registerConstraint(startTileFixedLocation)
Listing 4: Graph query registered as constraint
4.3.3 Defining the objective function. The objective function that drives the search is based on population-level measures (i.e., measures over a family of environments) which, in turn, are derived from individual-level measures (i.e., measures over specific environments). These measures enable comparison of populations and individuals, respectively. Comparison is important because it enables ordering of populations and individuals, and, by extension, choosing the best-performing population (i.e., family of environments) and organizing it into a learning process (e.g., a sequence of increasing complexity in CL). Formally, we demand measure 𝜇 that induces partial ordering but we do not demand countable additivity. Examples of such measures include many useful constructs, such as Shannon entropy, belief functions in Dempster–Shafer theory, and Choquet capacity. Let ℰ denote the search space, and let 𝐸 ∈ ℰ denote a particular environ𝑛 ment. A feature map 𝜙 ∶ ℰ → R extracts measurable structures from environments to produce a feature vector 𝜙(𝐸). Let Σ𝜙 be the 𝜎-algebra over the set of all environments ℰ, representing sets of environments whose features satisfy some property. A complexity measure 𝜇 ∶ Σ𝜙 → R≥0 is a function that assigns non-negative real numbers to measurable sets of environments. This measure satisfies the property of non-negativity, 𝜇(∅) = 0. The measure induces a partial ordering, 𝐸 1 ⪯ 𝐸 2 ⟺ 𝜇(𝐸 1 ) ≤ 𝜇(𝐸 2 ), that allows for formal comparison of populations in the GA (i.e., families of environments, e.g., by Shannon entropy) and individuals (specific environment models, e.g., by complexity). For population-level measure, typically, some kind of a diversity measure is chosen. For example, in CL, diversity of environments in terms of an appropriate complexity measure is a good indicator of the eventual successful training performance. For the individuallevel measure, typically, some kind of a complexity measure is chosen. For example, in CL, the sole reason to generate environment variants is the high complexity of the initial environment. Measures are defined by the RL expert, but our framework provides some useful measures, too. These include binned Shannon entropy [68] (which is the default population-level measure), Gini coefficient [10], and statistical variance. Individual-level measures are too domainspecific to define them without a firm grasp on the domain concepts and therefore, our framework does not provide such defaults. With that, defining new measures is straightforward in our framework. The RL expert only needs to implement a simple interface that declares a double evaluate() method to assign a complexity value to populations or environment models; and subsequently, register it when setting up the GA.
MODELS ’26, October 4–9, 2026, Malaga, Spain
Example Individual (environment) measure of complexity In the running example, the expert defines the complexity 𝑐 ∶ ℰ → [0, 1) based on feasible down-right paths from StartTile to GoalTile. Let 𝑃 = (2𝑛−2 ) be the total number of down-right paths 𝑛−1 in an 𝑛 × 𝑛 grid, and let 𝑃𝐸 be the number of such paths that avoid all BurningTiles in environment 𝐸. The complexity is defined as: 𝑃𝐸 𝑐(𝐸) = 1 − 𝑃 This measure provides an admissible heuristic that never overestimates difficulty. For this running example, 𝑃 = (63) = 20. As shown in Fig. 3, three environments have complexity values 𝑐 = 0, 𝑐 = 0.5, and 𝑐 = 0.7, corresponding to 0, 2, and 4 burning tiles, respectively.
=0 (a) 𝑐 = 1 − 20 20
10 = 0.5 (b) 𝑐 = 1 − 20
2 = 0.9 (c) 𝑐 = 1 − 20
Figure 3: A family of Burning Forest environments in CL
4.3.4 Defining GA configuration. The GA configuration defines the search process that generates families of environments by applying mutations to the initial environment. This configuration contains algorithm-specific attributes, such as population size, parent and survivor selection mechanisms, mutation rate, maximum mutation attempt threshold, and termination criteria. Initial 4.1 Init environment 4.5Termination
Environment
4.2 Parent selection
Population
4.4 Survivor selection
Environment 4.3 Mutation
Environment
Figure 4: Environment generation process
4.4
Generating families of environments
i In Step 4y , a GA generates the family of environments that meet the objective function and respect the defined constraints. (Fig. 4) Since population size directly affects the sampling ability and performance of the GA, limited population size can lead to premature convergence [72]. To address this, we use the hybrid GA, a marriage between a population-based global search and a heuristic local search [46], which balances global exploration with local exploitation even with small populations [17]. In our approach, we incorporate simulated annealing [6] as the local search method in the survivor selection phase, maintaining the highest diversity group of individuals for the next generation.
MODELS ’26, October 4–9, 2026, Malaga, Spain
Liu and David
ithe initial population is created. 4.4.1 Initialization. In step 4.1y Clones of the initial environment are created until the specified population size is reached. To introduce diversity, each environment copy is varied using a small, random number of the specified model transformations. After this initial mutation, each environment model is validated using the user-defined constraints, and invalid environments are discarded.
"SVVR", "RRRV", "RBWR", "RBRG", ], "reward_schedule": [0.0, -0.1, -1.0, 1.0, 10.0], "complexity": 0.5}, { "env_id": "env_2", "desc": [ "SVVR", "RVVB", "BBWV", "RBVG", ], "reward_schedule": [0.0, -0.1, -1.0, 1.0, 10.0], "complexity": 0.7}
i, members of the population are 4.4.2 Parent selection. In step 4.2y selected to become parents. Each candidate environment model is evaluated using the user-defined objective function. We use random selection but more sophisticated mechanisms can be registered and used in our framework (e.g., tournament selection, elitism). i 4.4.3 Mutation. In step 4.3y , offspring environment models are generated by applying mutations. Each of the parents selected in the previous step is mutated using the mutation operators, with a probability defined by the mutation rate specified in the GA configuration. After mutation, the candidate offspring are validated. If a candidate offspring is invalid or violates previously defined constraints, the model editing transaction rolls back using EMF’s TransactionalEditingDomain. Mutation is reattempted on the parent of the invalid offspring until a valid offspring is produced, or a maximum mutation attempt threshold is met, as specified in the GA configuration. i 4.4.4 Survivor selection. In step 4.4y , the next generation is formed by performing survivor selection. Each new environment is evaluated for complexity, and a local search algorithm identifies the individuals that together form the population with the highest fitness, e.g., diversity. These individuals are retained and together, they form the new population. We use simulated annealing [6] as our local search to select survivors, but this can be changed by the user by registering new local search heuristics (e.g., hill climbing, depth-first search, or breadth-first search). 4.4.5 Termination. The GA continues until the termination criteria y). These termination criteria are specified in the GA are met ( 4.5i configuration (e.g., maximum number of generations, or timeout).
4.5
Code generation
i In step 5y , the code for the target RL training environment is generated. For this step, the domain expert needs to develop the code generation templates, possibly in collaboration with the RL expert. Example Code generation In the running example, a Gymnasium-compliant JSON descriptor is generated from the environments: {"environments": [ { "env_id": "env_1", "desc": [ "SRRR", "RRRR", "RRWR", "RRRG", ], "reward_schedule": [0.0, -0.1, -1.0, 1.0, 10.0], "complexity": 0.0}, { "env_id": "env_2", "desc": [
]}
5
Evaluation
To evaluate our approach in a curriculum learning setting on the scaled-up version of the running example. CL is a representative example of the various learning paradigms the rely on families of training environments. We assess our approach by answering the following research question: how do curricula generated in our approach improve learning performance?
5.1
Experiment setup
5.1.1 Metrics. We evaluate the RL agent’s cumulative reward realized throughout the learning process, as well as the success rate of the trained agent in the target environment. Cumulative reward measures the total reward collected over training, which is a standard metric for assessing RL performance [64]. Higher cumulative rewards indicate better learning performance. Faster convergence indicates better learning dynamics. After training, we evaluate each saved policy on the target environment (𝐸 6 ) for 3,000 episodes, reporting the mean episodic return (i.e., average cumulative reward), and the average success rate. The success rate is defined as the proportion of episodes in which the agent reaches the goal while avoiding burning tiles, which is an accepted metric for evaluating task completion in goal-oriented RL [43]. 5.1.2 Environment configuration. We use a scaled-up, 8 × 8 version of the running example (Sec. 3). This size is sufficient for our purposes as the combinatorial explosion places the problem beyond the humanly tractable horizon and invokes the need for our approach. In CL, the initial environment is the hardest one, i.e., the target environment (Fig. 5f); and the easier ones are to be generated (Fig. 5a–Fig. 5e). All MTs and constraints follow those defined in Sec. 4.3.1 and Sec. 4.3.2, respectively. The complexity measure is defined in relation to the number of available paths between the start and the goal, as defined in Sec. 4.3.3. The reward structure is the same across environments in the family and follows Tab. 1. As the diversity measure, we use Shannon entropy [58], a widely adopted measure of diversity in CL [74], computed over binned complexity values across the population. The complexity is partitioned into ∣ℰ∣ bins, where ∣ℰ∣ is the number of generated environments in the curriculum. Higher entropy indicates a more uniform distribution of complexity values across the family, i.e., higher diversity.
A Model-Driven Approach for Developing Families of Reinforcement Learning Environments
5.1.3 GA configuration. We use a population size of six, i.e., the GA will produce a curriculum of six environments. The mutation rate is set to 0.85 to encourage exploration of different environment configurations while preserving some individuals unchanged. We configure simulated annealing with 500 rounds, an initial temperature of 𝑇0 = 0.1, and a cooling rate of 𝑐𝑟 = 0.95 (widely used default, 3 see, e.g., MATLAB ). Temperature decreases as 𝑇𝑖+1 = 𝛼 ⋅ 𝑇𝑖 . The GA terminates after 1 000 generations. 5.1.4 RL configuration. For RL, we use Q-learning [75], a fundamental model-free value-based algorithm, which is well-suited for discrete state and action spaces, such as the Burning Forest. The learning rate 𝛼 affects how much the agent updates its policy parameters during training. We use a moderate learning rate (𝛼 = 0.1) to ensure stable convergence. The discount factor 𝛾 determines how much the agent values future rewards. We use a high discount factor (𝛾 = 0.99), which encourages the agent to prioritize long-term rewards over immediate ones. For exploration, we use an 𝜖-greedy strategy, at each step, the agent selects a random action with probability 𝜖 and the greedy action otherwise. 𝜖 decays from fully random (𝜖 = 1) over training. This strategy ensures broad exploration in the early stage and gradual exploitation as the agent’s Q-values stabilize. When the agent advances to a new environment in the curriculum, we reset 𝜖 to 1, encouraging the agent to re-explore the new environment. These hyperparameters are common defaults in RL practice; we use them to mitigate threats to the internal validity. 5.1.5 CL configuration. For CL, from the six generated environments (𝐸 1 , 𝐸 2 , ..., 𝐸 6 ) ordered by complexity, we construct five curricula with different prefixes: {𝐸 1, ..., 𝐸6}, {𝐸 2, ..., 𝐸6}, {𝐸 3, ..., 𝐸6}, {𝐸 4, 𝐸 5, 𝐸6}, and {𝐸 5, 𝐸6}. Each environment is trained for 50,000 steps, and the total training budget scales with the length of the curriculum. The baseline agents train directly on each environment for 300,000 steps, the total budget of the full curriculum.
5.2
Results and key observations
Fig. 5 visualizes the generated curriculum. Each environment indeed satisfies the defined constraints: a valid path from start to goal exists in each environment, burning tiles are placed within the density range, and complexity increases progressively. Fig. 6 and Fig. 7 show cumulative rewards during training. Each subfigure in Fig. 6 compares a curriculum agent against baselines trained on the individual environments in that curriculum. Tab. 2 presents the trained agents’ performance in the target environment. We observe that direct training in a single environment is insufficient without prior exposure to simpler environments, as the agent fails to learn the effective policy. The agent trained directly on 𝐸 6 accumulates negative reward throughout training, and achieves 0% success rate and a mean episodic return of 0 during evaluation. Although baseline agents trained on environments other than 𝐸 3 achieve positive rewards and accumulate higher rewards than curriculum-based agents, these learned policies fail to directly transfer to the most challenging environment. They achieve non-positive mean episodic return and 0% success rate during evaluation. In contrast, we observe that the generated curriculum improves learning performance. All curriculum-trained agents 3
https://www.mathworks.com/help/gads/simulated-annealing.html
MODELS ’26, October 4–9, 2026, Malaga, Spain
(a) c = 0.00
(b) c = 0.20
(c) c = 0.48
(d) c = 0.59
(e) c = 0.67
(f) c = 0.93
Figure 5: Generated curriculum of increasing complexity Table 2: Evaluation results on the target environment Agent
Success rate (%)
Mean episodic return
Baseline (𝐸 1 ) Baseline (𝐸 2 ) Baseline (𝐸 3 ) Baseline (𝐸 4 ) Baseline (𝐸 5 ) Baseline (𝐸 6 )
0 0 0 0 0 0
-1.0 -1.0 -1.0 -1.0 -1.0 0
Curriculum (𝐸 1 –𝐸 6 ) Curriculum (𝐸 2 –𝐸 6 ) Curriculum (𝐸 3 –𝐸 6 ) Curriculum (𝐸 4 –𝐸 6 ) Curriculum (𝐸 5 –𝐸 6 )
100 100 100 100 0
10.2 10.2 10.2 10.2 0
accumulate positive reward during training, and all achieve a 100% success rate on 𝐸 6 , except the agent trained with a curriculum comprising only the two hardest environments (Fig. 6e). We observe that curriculum-trained agents exhibit plateaus when transitioning to the new environment, during which the agent adapts its learned policy to the harder configuration. These plateaus are shorter when the agent has been exposed to more intermediate environments, For example, the full curriculum-trained agent (Fig. 6a) transitions smoothly between environments with short plateaus. In contrast, shorter curricula, such as Fig. 6c and Fig. 6d, exhibit long plateaus during transitions. This suggests that finergrained progression through intermediate environments facilitates faster transfer, as larger gaps between consecutive environments lead to longer adaptation periods. This aligns with findings in simto-real transfer [24], where greater investment in training on diverse foundational skills produces more robust policies that generalize better to real-world settings [8]. To validate the scalability of our approach, we run the curriculum generation process for different sized problems and curriculum sizes. As shown in Tab. 3, generation time scales well compared to the combinatorial explosion observed in the state space that is induced by the growing dimensions of the problem (map). Generation time increases with the curriculum size more substantially. However, this kind of scalability is not in the scope of our work, as we focused on demonstrating the feasibility of using MDE for generating environment families. The scalability in terms of curriculum
MODELS ’26, October 4–9, 2026, Malaga, Spain
(a) Curriculum prefix E1-E6
Liu and David
(b) Curriculum prefix E2-E6
(d) Curriculum prefix E4-E6
(c) Curriculum prefix E3-E6
(e) Curriculum prefix E5-E6
Figure 6: Cumulative reward with different prefixes
Figure 7: Cumulative reward across all curricula Table 3: Generation time (seconds) for different problem sizes Map size 8×8 9×9 10×10 11×11 12×12 30 38 47 57 68 State space 3.4 × 10 4.4 × 10 5.2 × 10 5.4 × 10 5.0 × 10 Curriculum size Generation time 6 47 49 49 49 50 10 208 211 212 215 225 20 1594 1718 1617 1642 1659
size can be addressed in multiple way in subsequent works, e.g., by incrementalization of the end-to-end GA process, or by choosing better-scaling local search and complexity assessment algorithms.
suboptimal policy. To mitigate this, we also evaluate the success rate and average episodic return in the target environment and report statistics. The choice of complexity measure influences the ordering of environments in the curriculum, which may affect learning performance. To mitigate this, we use an admissible heuristic that never overestimates difficulty – the number of valid down-right paths from start to goal, where fewer feasible paths indicate higher complexity. The choice of diversity measure influences the population of generated environments. To mitigate this, we use Shannon entropy [58], a widely used measure of diversity. Our framework supports alternative population measures, including the Gini coefficient for measuring distributional inequality and variance for measuring how spread out the distribution is. Our framework also allows the expert to define custom measures for both the individual and the population. Investigating the effect of alternative measures on curriculum quality is left to future work. External validity. Our evaluation focuses on CL as the learning paradigm. However, our framework is designed in a way that it supports any learning paradigm as long as population- and individuallevel measures are properly defined. Thus, we are reasonably confident that our approach generalizes safely to other paradigms, such as multi-task learning [62], meta-RL [22], and domain randomization [69]. We recommend replication studies to test this claim.
6 5.3
Discussion
Threats to validity
Internal validity. The choice of hyperparameters may influence the observed results. We use established settings for Q-learning in our experiments to mitigate threats to the internal validity. The reward structure may influence agent performance. To mitigate this, we use a dense reward structure to benefit both CL and direct training by providing shaping signals. Construct validity. We evaluate the RL agent’s cumulative reward realized throughout the learning process. However, cumulative reward may not reflect policy quality, i.e., an agent may converge to a
In this section, we reflect on the approach and set forth important research directions for the MDE community.
6.1
Reflections and takeaways
6.1.1 Curriculum size matters. Our evaluation results show that curriculum effectiveness does not scale monotonically with the number of environments. The agent trained through five environments (Fig. 6b) achieves 100% success rates, converging quickly with short plateaus between environments. Agents trained in three
A Model-Driven Approach for Developing Families of Reinforcement Learning Environments
(Fig. 6d) and four (Fig. 6c) environments also reach the goal successfully, although they exhibit longer plateaus during transitions to harder environments. This suggests a saturation point, i.e., beyond a sufficient number of intermediate environments, additional environments produce diminishing returns [59]. This observation implies that both the quality and the quantity of generated environments matter, also noted by Narvekar et al. [48]. Rather than maximizing the number of generated environments, engineers benefit from methods that identify a compact, effective curriculum. This motivates three directions: (i) local search strategies that iteratively remove redundant environments to downsize the generated pool; (ii) approximation methods that can estimate curriculum quality without full simulation to enable early pruning; and (iii) incremental environment generation methods that allow engineers to iteratively refine curricula within realistic time and resource constraints. 6.1.2 Complexity measures are indeed challenging to define. Our results demonstrate that a human-crafted complexity measure may not correspond exactly to the learning difficulty of the RL agent. 𝐸 3 (Fig. 5c) with a complexity of 0.48 and consists of only three burning tiles appears moderately difficult. However, as shown in Fig. 6c, the agent fails to accumulate positive reward when trained directly on it, and the curriculum-based agent starting from 𝐸 3 exhibits a long plateau before progressing to the next environment. This confirms a known challenge of determining environment difficulty for agents in curriculum learning [49]. This highlights the need for domain-specific languages that enable engineers to express and compose complexity measures from domain primitives. Developing such languages, potentially generated from domain ontologies [18], may be an important future direction. 6.1.3 Applicability in other learning paradigms. While our evaluation uses CL as an example, our framework can be applied in other learning paradigms, too. Domain randomization (DR) [69] is a widely adopted technique to bridge the sim-to-real gap. Instead of relying on the high-fidelity of simulation during training, DR exposes agents to varied conditions during training, thereby promoting generalization and enhancing the agent’s ability to operate reliably in real-world conditions [53]. Our framework can be used to produce environment variants for training [79] at a given difficulty range. Our approach also naturally accommodates the combined paradigm of CL and DR, in which agents are simultaneously exposed to both increasing difficulty and increasing randomization [69]. Two other paradigms that could benefit from our approach are multi-task RL (MTRL) and meta-RL. In MTRL, a single policy is trained across multiple environments in parallel to maximize average performance [22]. In meta-RL, agents learn from a training set of environments and must rapidly adapt to unseen test environments [22]. Both paradigms require environments with consistent observation and action spaces, shared reward structure, and sufficient diversity between environments to enable knowledge transfer rather than memorization. Our framework supports generating such families by specifying structural invariants while applying mutations to vary task-specific properties (e.g., objects, goals, interaction types), with the diversity measure to ensure adequate task variation for generalization.
MODELS ’26, October 4–9, 2026, Malaga, Spain
Our framework can be of high utility in broader use cases throughout the RL development lifecycle. For testing and robustness evaluation, our framework can be used to generate edge cases and challenging boundary conditions that rarely arise during training but are critical for deployment and safety concerns. For benchmarking, our framework can produce environments with controlled variation in difficulty and diversity, enabling fair and reproducible comparison between RL algorithms. Additionally, the framework can generate a larger population from a small initial population as a reference distribution, while preserving statistical properties (e.g., mean, variance) and using out-of-distribution detection [37] to ensure distribution consistency. These directions demonstrate that our model-driven approach provides a foundation generalizable to diverse use cases requiring multiple environments beyond CL. We call for extending our approach into these learning paradigms to validate our claims.
6.2
R&D opportunities for the MDE community
Our work highlights some research and development opportunities for the MDE community. Drawing on the experiences from developing our approach, we discuss some of these opportunities. 6.2.1 The role of domain-specific languages. There is a clear need for targeted DSLs at various points of this approach, making a good case for DSL engineering as a research direction in this space. While DSLs excel at raising the level of abstraction in software engineering [44], existing approaches to RL environment specification remain largely imperative and framework-specific (e.g., Gymnasium’s API). To leverage MDE techniques for the benefit of RL and ML, DSLs are needed that can capture structural properties (e.g., states, transitions, and structural relations of environments), behavioral semantics (reward structures, termination conditions), and parametric variability for generating families of environments. Beyond environment specification, DSLs could also capture curriculum structures [63] (e.g., sequential curricula or curriculum graphs), difficulty progression strategies, and partial constraints over environment families. This aligns with recent efforts in developing DSLs for RL [61] but calls for more support for family-level reasoning and search-based generation, which remain unexplored. Of course, case-by-case DSL engineering does not scale with the high volume of RL-based applications and therefore, automated DSL engineering techniques should be investigated, e.g., based on domain ontologies [52] and the structure of the learning paradigm. 6.2.2 HOTs for generating mutation MTs. Our approach relies on mutation MTs that are typically developed in a manual fashion. We support simple automation for the derivation of MTs from states that are marked as mutable by the RL expert, and the value of such derivation is especially clear in highly heterogeneous state spaces in which manual MT specification is infeasible. This demonstrates the opportunity for higher-order transformations (HOTs) [67] to generate mutation operators automatically. HOTs could be used to derive mutation operators from the metamodel structure [30] (e.g., type hierarchies, multiplicities), ensure semantic preservation (e.g., solvability constraints), and encode domain heuristics (e.g., monotonic difficulty adjustments).
MODELS ’26, October 4–9, 2026, Malaga, Spain
Similar avenues have been explored before in related problems, e.g., in generating search operators for model-driven optimization [7]. We recommend extensions of such foundations toward machine learning paradigms and the related mutation synthesis problems. In addition to generating mutations, HOTs could, e.g., generate operators that explicitly control environment difficulty gradients or diversity distributions. In addition, combining HOTs with meta-learning signals (e.g., feedback from RL performance) opens a direction for adaptive mutation operators that evolve alongside the learning process. Such mechanisms are of high utility, e.g., in adaptive CL, in which the curriculum is dynamically adjusted based on the agent’s learning progress [42]. 6.2.3 Variability and product family engineering. Families of RL environments can be naturally interpreted through variability modeling and product-family engineering [47]. In our approach, environments share a common structure (chiefly defined by the initial environment) and mutations induce variability—analogous to deriving products from a shared platform [33]. Unlike in traditional product lines, variability, here, is continuous and generative, rather than based on selecting from predefined configurations. Extensions of traditional variability techniques may bring benefits to our approach. In particular, variability models could be enriched with structural and parametric variation operators (beyond Boolean features), objective functions (e.g., diversity, complexity), and relations between variants (e.g., ordering in curricula). Such extensions could bridge product-line engineering with model-driven optimization [57], enabling automated exploration of environment families rather than manual configuration. Global decision-making over deep variability [28] can aid reasoning about families of learning environments rather than individual variants. Finally, variability representations can improve traceability and explainability, and inform RL experts about how environment variants have been derived. Such directions are highly conducive to trustworthiness [77] and certification [32] of AI systems.
7
Related work
In this section, we review the related work. Product-family engineering is a systematic approach to create families of related products that share a common set of core assets while maintaining managed variability to address diverse requirements [33]. PFE enables reuse of core assets across variants, reducing development costs [34]. However, PFE uses a discrete selection from predefined variants. Our approach uses continuous mutations guided by measures for environment generation, instead of enumerable feature combinations. Procedural content generation (PCG) methods automatically generate diverse game content, such as maps, levels, and environments [27]. While PCG can automatically produce game levels, these techniques are domain-specific and generate environments narrowly tailored to game engines, limiting reuse, especially in problems other than games [70]. Moreover, ensuring that generated environments meet validity constraints, such as preventing invalid or unsolvable environments, is still challenging [60]. Large language model (LLM)-based environment generation prompts LLMs with textual context to synthesize environment specifications and executable code [23]. However, LLMs provide
Liu and David
no formal guarantees about environment validity. Their stochastic, non-deterministic nature prevents systematic verification of environment properties or structural constraints [77]. Furthermore, LLM-based generation requires substantial computational resources to consistently generate and select valuable environments [35]. Model instance generation and model-based design space exploration (DSE) are related topics to ours. Refinery [56, 57] is a framework for the automated generation of consistent and diverse graph models for rich test instances. In VIATRA DSE [1], multi-objective optimization rules are captured in model transformations, enabling a smooth integration of genetic algorithms into MDE problems. Such frameworks can be integrated into our approach to replace some of the prototype components we developed. MDE for RL type works that relate to ours are the following. Gatto et al. [21] generate training and deployment code interfacing with simulators via Robot Operating System. However, their approach models the RL algorithm, treating the environment as an external system. Molderez et al. [45] propose Marlon, a DSL bridging multiagent RL and distributed systems, where the environment is taken as given. Sinani et al. [61] propose RLML, a modeling language for specifying RL problems, but require manual enumeration of states, actions, and rewards for a single environment. Kusmenko et al. [31] translate game descriptions into RL environments, but their toolchain is limited to turn-based games. Liaskos et al. [36] generate RL training environments from goal models, but focus on a single environment. In this work, we address these limitations by proposing an approach for developing families of RL environments.
8
Conclusion
In this work, we presented a model-driven approach for the automated construction of families of reinforcement learning training environments. Reinforcement learning often necessitates training processes over families of environments, e.g., to train the agent on a sequence of environments with gradually increasing complexity. However, obtaining such a family of environments through manual or software code-level mutations is a labor-intensive and error-prone endeavor. Our approach automates the generation of environments through a model-based hybrid genetic algorithm, in which environment mutation operators and constraints are codified in model transformations, and executed in a state-of-the-art incremental model transformation engine. Our work demonstrates the utility of the model-driven engineering body of knowledge in modern machine learning problems. Accordingly, in this work, we identify opportunities for the modeldriven engineering community to contribute to addressing such problems through their expertise. Future work will focus on the performance aspects of the approach, e.g., through incrementalization and algorithm tuning, as well as evaluation on an industry-scale case.
Acknowledgement We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), DGECR-2024-00293 (End-to-end Sustainable Systems Engineering).
A Model-Driven Approach for Developing Families of Reinforcement Learning Environments
References [1] Hani Abdeen, Dániel Varró, Houari Sahraoui, András Szabolcs Nagy, Csaba Debreceni, Ábel Hegedüs, and Ákos Horváth. 2014. Multi-objective optimization in rule-based design space exploration. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE ’14). ACM, 289–300. doi:10.1145/2642937.2643005 [2] OpenAI: Marcin Andrychowicz et al. 2020. Learning dexterous in-hand manipulation. The International Journal of Robotics Research 39, 1 (2020), 3–20. doi:10.1177/0278364919887447 [3] Angela Barriga, Rogardt Heldal, Adrian Rutle, and Ludovico Iovino. 2022. PARMOREL: a framework for customizable model repair. Soft. Sys. Mod. 21, 5 (2022), 1739–1762. [4] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML ’09). ACM, 41–48. doi:10.1145/1553374.1553380 [5] Gábor Bergmann, Istvan David, Ábel Hegedüs, Ákos Horváth, István Ráth, Zoltán Ujhelyi, and Dániel Varró. 2015. VIATRA 3: A Reactive Model Transformation Platform. In Theory and Practice of Model Transformations - 8th International Conference, ICMTSTAF 2015, L’Aquila, Italy, July 20-21, 2015. Proceedings (LNCS, Vol. 9152). Springer, 101–110. doi:10.1007/978-3-319-21155-8_8 [6] Dimitris Bertsimas and John Tsitsiklis. 1993. Simulated Annealing. Statist. Sci. 8, 1 (1993), 10 – 15. doi:10.1214/ss/1177011077 [7] Alexandru Burdusel, Steffen Zschaler, and Stefan John. 2021. Automatic generation of atomic multiplicity-preserving search operators for search-based model engineering. Soft. Sys. Mod. 20, 6 (2021), 1857–1887. [8] Thomas Chaffre, Julien Moras, Adrien Chan-Hon-Tong, and Julien Marzat. 2020. Sim-to-Real Transfer with Incremental Environment Complexity for Reinforcement Learning of Depth-Based Robot Navigation. In Proceedings of the 17th International Conference on Informatics, Automation and Robotics, ICINCO 2020. 314–323. https://ensta.hal.science/hal-02958155 [9] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. 2020. Leveraging Procedural Generation to Benchmark Reinforcement Learning. In Proc of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119). PMLR, 2048–2056. [10] Frank Cowell. 2011. Measuring Inequality (3 ed.). Oxford University Press, London, England. [11] Kyanna Dagenais and Istvan David. 2025. Complex Model Transformations by Reinforcement Learning with Uncertain Human Guidance. In 2025 ACM/IEEE 28th International Conference on Model Driven Engineering Languages and Systems (MODELS). doi:10.1109/MODELS67397.2025.00025 [12] Istvan David and Eugene Syriani. 2022. DEVS Model Construction as a Reinforcement Learning Problem. In 2022 Annual Modeling and Simulation Conference (ANNSIM). IEEE, 30–41. doi:10.23919/ANNSIM55834.2022.9859369 [13] Istvan David and Eugene Syriani. 2024. Automated Inference of Simulators in Digital Twins. CRC Press, Chapter 8, 122–148. doi:10.1201/9781003425724-11 [14] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. 2020. Emergent Complexity and Zeroshot Transfer via Unsupervised Environment Design. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 13049–13061. [15] Agoston E Eiben and James E Smith. 2015. Introduction to evolutionary computing. Springer. [16] Martin Eisenberg, Hans-Peter Pichler, Antonio Garmendia, and Manuel Wimmer. 2021. Towards Reinforcement Learning for In-Place Model Transformations. In 2021 ACM/IEEE 24th Intl Conf. on Model Driven Engineering Languages and Systems (MODELS). 82–88. [17] Tarek A El-Mihoub, Adrian A Hopgood, Lars Nolle, and Alan Battersby. [n. d.]. Hybrid Genetic Algorithms: A Review. ([n. d.]). [18] Maged Elaasar, Nicolas Rouquette, David Wagner, Bentley James Oakes, Abdelwahab Hamou-Lhadj, and Mohammad Hamdaqa. 2023. openCAESAR: Balancing Agility and Rigor in Model-Based Systems Engineering. In 2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C). 221–230. doi:10.1109/MODELS-C59198.2023.00051 [19] Rafael Figueiredo Prudencio, Marcos R. O. A. Maximo, and Esther Luna Colombini. 2024. A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems. IEEE Trans Neural Netw Learn Syst 35, 8 (2024), 10237–10257. doi:10.1109/TNNLS.2023.3250269 [20] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. 2017. Reverse Curriculum Generation for Reinforcement Learning. In Proceedings of the 1st Annual Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 78). PMLR, 482–495. [21] Nicola Gatto, Evgeny Kusmenko, and Bernhard Rumpe. 2019. Modeling Deep Reinforcement Learning Based Architectures for Cyber-Physical Systems. In 2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems Companion. 196–202. doi:10.1109/MODELS-C.2019.00033 [22] Timothy Hospedales et al. 2022. Meta-Learning in Neural Networks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 9 (2022), 5149– 5169. doi:10.1109/TPAMI.2021.3079209
MODELS ’26, October 4–9, 2026, Malaga, Spain
[23] Mengkang Hu et al. 2025. AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (KDD ’25). ACM, 496–507. doi:10.1145/3690624.3709321 [24] Xuemin Hu, Shen Li, Tingyu Huang, Bo Tang, Rouxing Huai, and Long Chen. 2024. How Simulation Helps Autonomous Driving: A Survey of Sim2real, Digital Twins, and Parallel Intelligence. IEEE Transactions on Intelligent Vehicles 9, 1 (2024), 593–612. doi:10.1109/TIV.2023.3312777 [25] Stefan John, Alexandru Burdusel, Robert Bill, Daniel Struber, Gabriele Taentzer, Steffen Zschaler, and Manuel Wimmer. 2019. Searching for optimal models: Comparing two encoding approaches. In 12th International Conference on Model Transformations ICMT 2019. 1–22. [26] Stefan John, Jens Kosiol, Leen Lambers, and Gabriele Taentzer. 2023. A graphbased framework for model-driven optimization facilitating impact analysis of mutation operator properties. Soft. Sys. Mod. 22, 4 (2023), 1281–1318. [27] Lawrence Johnson, Georgios N Yannakakis, and Julian Togelius. 2010. Cellular automata for real-time generation of infinite cave levels. In Proceedings of the 2010 Workshop on Procedural Content Generation in Games. 1–4. [28] Joerg Kienzle et al. 2023. Global Decision Making Over Deep Variability in Feedback-Driven Software Development. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE ’22). ACM, Article 178, 6 pages. doi:10.1145/3551349.3559551 [29] Taewoo Kim, Minsu Jang, and Jaehong Kim. 2021. A survey on simulation environments for reinforcement learning. In 2021 18th International Conference on Ubiquitous Robots (UR). IEEE, 63–67. [30] Thomas Kühne, Gergely Mezei, Eugene Syriani, Hans Vangheluwe, and Manuel Wimmer. 2010. Explicit Transformation Modeling. In Models in Software Engineering. Springer, 240–255. [31] Evgeny Kusmenko et al. 2022. A Model-Driven Generative Self Play-Based Toolchain for Developing Games and Players. In Proceedings of the 21st ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE 2022). ACM, 95–107. doi:10.1145/3564719.3568687 [32] Marta Kwiatkowska and Xiyue Zhang. 2023. When to Trust AI: Advances and Challenges for Certification of Neural Networks. In 2023 18th Conference on Computer Science and Intelligence Systems (FedCSIS). 25–37. doi:10.15439/2023F2324 [33] Hartmut Lackner and Bernd-Holger Schlingloff. 2017. Chapter Four - Advances in Testing Software Product Lines. Advances in Computers, Vol. 107. Elsevier, 157–217. doi:10.1016/bs.adcom.2017.07.001 [34] José Lameh, Alexandra Dubray, and Marija Jankovic. 2025. Modeling variability in product line engineering (PLE) for systems engineering (SE). Proceedings of the Design Society 5 (2025), 2491–2500. doi:10.1017/pds.2025.10263 [35] William Liang, Sam Wang, Hung-Ju Wang, Osbert Bastani, Dinesh Jayaraman, and Yecheng Jason Ma. 2024. Eurekaverse: Environment curriculum generation via large language models. arXiv preprint arXiv:2411.01775 (2024). [36] Sotirios Liaskos, Shakil M. Khan, John Mylopoulos, and Reza Golipour. 2025. Model-Driven Design and Generation of Training Simulators for Reinforcement Learning. In Conceptual Modeling. Springer, 170–191. [37] Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. 2023. Towards Out-Of-Distribution Generalization: A Survey. arXiv:2108.13624 [cs.LG] https://arxiv.org/abs/2108.13624 [38] Xiaoran Liu and Istvan David. 2025. AI Simulation by Digital Twins: Systematic Survey, Reference Framework, and Mapping to a Standardized Architecture. Software and Systems Modeling (2025). doi:10.1007/s10270-025-01306-0 [39] Xiaoran Liu and Istvan David. 2026. A Reference Architecture of Reinforcement Learning Frameworks. In 2026 IEEE 23rd International Conference on Software Architecture (ICSA). doi:10.1109/ICSA66085.2026.00016 [40] Viktor Makoviychuk et al. 2021. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. doi:10.48550/arXiv.2108.10470 [41] Dhruv Malik, Yuanzhi Li, and Pradeep Ravikumar. 2021. When Is Generalizable Reinforcement Learning Tractable?. In Advances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., 8032–8045. [42] Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. 2019. Teacher– student curriculum learning. IEEE transactions on neural networks and learning systems 31, 9 (2019), 3732–3740. [43] Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, K.R. Zentner, Ryan Julian, J K Terry, Isaac Woungang, Nariman Farsad, and Pablo Samuel Castro. 2025. Meta-World+: An Improved, Standardized, RL Benchmark. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [44] Marjan Mernik, Jan Heering, and Anthony M. Sloane. 2005. When and how to develop domain-specific languages. ACM Comput. Surv. 37, 4 (Dec. 2005), 316–344. doi:10.1145/1118890.1118892 [45] Tim Molderez, Bjarno Oeyen, Coen De Roover, and Wolfgang De Meuter. 2019. Marlon: A domain-specific language for multi-agent reinforcement learning on networks. In Proc of the 34th ACM/SIGAPP Symposium on Applied Computing. ACM, 1322–1329. doi:10.1145/3297280.3297413 [46] Pablo Moscato et al. [n. d.]. On evolution, search, optimization, genetic algorithms and martial arts: Towards memetic algorithms. ([n. d.]).
MODELS ’26, October 4–9, 2026, Malaga, Spain
[47] Dirk Muthig and Colin Atkinson. 2002. Model-Driven Product Line Architectures. In Software Product Lines. Springer, 110–129. [48] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. 2020. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey. J Machine Learning Research 21, 181 (2020), 1–50. [49] Sanmit Narvekar, Jivko Sinapov, Matteo Leonetti, and Peter Stone. 2016. Source task creation for curriculum learning. In Proceedings of the 2016 international conference on autonomous agents & multiagent systems. 566–574. [50] Hira Naveed, Chetan Arora, Hourieh Khalajzadeh, John Grundy, and Omar Haggag. 2024. Model driven engineering for machine learning components: A systematic literature review. Inf Softw Technol 169 (2024), 107423. [51] Evangelos Ntentos, Stephen John Warnett, and Uwe Zdun. 2024. Supporting architectural decision making on training strategies in reinforcement learning architectures. In 21st Intl Conf on Software Architecture (ICSA). IEEE, 90–100. [52] Maria Joao Varanda Pereira, Joao Fonseca, and Pedro Rangel Henriques. 2016. Ontological approach for DSL development. Computer Languages, Systems & Structures 45 (2016), 35–52. [53] Andrei Pitkevich and Ilya Makarov. 2024. A Survey on Sim-to-Real Transfer Methods for Robotic Manipulation. In IEEE Intl Symposium on Intelligent Systems and Informatics (SISY). 000259–000266. doi:10.1109/SISY62279.2024.10737545 [54] Martin L Puterman. 1990. Markov decision processes. Handbooks in operations research and management science 2 (1990), 331–434. [55] Sheldon M Ross. 2022. Simulation. academic press. [56] Oszkár Semeráth, Aren A Babikian, Boqi Chen, Chuning Li, Kristóf Marussy, Gábor Szárnyas, and Dániel Varró. 2021. Automated generation of consistent, diverse and structurally realistic graph models. Soft. Sys. Mod. 20, 5 (2021), 1713–1734. doi:10.1007/s10270-021-00884-z [57] Oszkár Semeráth, Rebeka Farkas, Gábor Bergmann, and Dániel Varró. 2020. Diversity of graph models and graph generators in mutation testing. Int J Softw Tools Technol Transf 22, 1 (2020), 57–78. doi:10.1007/s10009-019-00530-6 [58] C. E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal 27, 3 (1948), 379–423. doi:10.1002/j.1538-7305.1948.tb01338.x [59] Ronald W. Shephard and Rolf Färe. 1974. The Law of Diminishing Returns. In Production Theory. Springer, 287–318. doi:10.1007/978-3-642-80864-7_17 [60] Daniele F Silva, Rafael P Torchelsen, and Marilton S Aguiar. 2025. Procedural game level generation with GANs: potential, weaknesses, and unresolved challenges in the literature. Multimedia Tools and Applications (2025), 1–27. [61] Natalie Sinani et al. 2024. Towards a Domain-Specific Modelling Environment for Reinforcement Learning. arXiv preprint arXiv:2410.09368 (2024). [62] Shagun Sodhani, Amy Zhang, and Joelle Pineau. 2021. Multi-task reinforcement learning with context-based representations. In International conference on machine learning. PMLR, 9767–9779. [63] Petru Soviany et al. 2022. Curriculum Learning: A Survey. International Journal of Computer Vision 130, 6 (2022), 1526–1565. doi:10.1007/s11263-022-01611-x [64] Richard S Sutton and Andrew G Barto. 1998. Reinforcement learning: An introduction. MIT press Cambridge. [65] Jordan Terry et al. 2021. PettingZoo: Gym for Multi-Agent Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., 15032–15043. [66] Krishnaiyan Thulasiraman and Madisetti NS Swamy. 2011. Graphs: theory and algorithms. John Wiley & Sons. [67] Massimo Tisi, Frédéric Jouault, Piero Fraternali, Stefano Ceri, and Jean Bézivin. 2009. On the Use of Higher-Order Model Transformations. In Proceedings of the 5th European Conference on Model Driven Architecture - Foundations and Applications (ECMDA-FA ’09). Springer, 18–33. doi:10.1007/978-3-642-02674-4_3 [68] T.M. Cover and Joy A Thomas. 1991. Elements of Information Theory (99 ed.). John Wiley & Sons, Nashville, TN. [69] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. 2017. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, 23–30. [70] Julian Togelius, Alex J Champandard, Pier Luca Lanzi, Michael Mateas, Ana Paiva, Mike Preuss, and Kenneth O Stanley. 2013. Procedural content generation: Goals, challenges and actionable steps. [71] Mark Towers et al. 2025. Gymnasium: A Standard Interface for Reinforcement Learning Environments. doi:10.48550/arXiv.2407.17032 [72] Y.R. Tsoy. 2003. The influence of population size and search time limit on genetic algorithm. In 7th Korea-Russia International Symposium on Science and Technology, Proceedings KORUS 2003. (IEEE Cat. No.03EX737), Vol. 3. 181–187 vol.3. [73] Dejin Wang and Seyede Fatemeh Ghoreishi. 2025. RGDR: Reward-Guided Domain Randomization for Autonomous Driving. In 2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC 2025), IEEE. [74] Xin Wang et al. 2022. A Survey on Curriculum Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 9 (2022), 4555–4576. doi:10.1109/ TPAMI.2021.3069908 [75] Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8, 3 (1992), 279–292.
Liu and David
[76] Tianhe Yu et al. 2020. Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning. In Proceedings of the Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 100). PMLR, 1094–1100. [77] Yedi Zhang et al. 2025. Position: Trustworthy AI Agents Require the Integration of Large Language Models and Formal Methods. In Forty-second International Conference on Machine Learning Position Paper Track. https://openreview.net/ forum?id=wkisIZbntD [78] Yu Zhang and Qiang Yang. 2022. A Survey on Multi-Task Learning. IEEE Transactions on Knowledge and Data Engineering 34, 12 (2022), 5586–5609. doi:10. 1109/TKDE.2021.3070203 [79] Wenshuai Zhao et al. 2020. Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI). 737–744. doi:10.1109/SSCI47803.2020.9308468