Augmenting Game AI with Deep Reinforcement Learning

Augmenting Game AI with Deep Reinforcement Learning

arXiv:2606.20210v1 [cs.AI] 18 Jun 2026

Alessandro Sestini* , Joakim Bergdahl, Amir Baghi, Jean-Philippe Barrette-LaPierre, Florian Fuchs, Linus Gisslén* Electronic Arts (EA), Stockholm, Sweden {asestini,jbergdahl,abaghi,jbarrettelapierre,ffuchs,lgisslen}@ea.com * Corresponding authors Abstract—Immersion in video games depends not only on graphics, audio, and game mechanics, but also on the quality of in-game characters. Producing believable characters, or game AI, remains a significant challenge as behavioral complexity is hard to capture with hand-coded systems. Game AI is a source of immersion and engagement; however, the limitations stemming from the challenges of creating game AI often lead to frustration and the breaking of the illusion of realism within the game. The introduction of machine learning models opens the door to creating more believable, authentic, and relatable characters in games. The promise is that they either learn from interacting with the game, or from player data, to develop true humanlike behavior. In this paper, we envision more applications of reinforcement learning for game AI in the future. For this to materialize, current research limitations are prohibitive to broad deployment across game genres. Therefore, we propose a framework for training reinforcement learning models with a set of requirements in mind that are suited towards game AI and game development. We present examples of games with reinforcement learning-augmented game AI and describe the practicalities of deploying player-facing machine learning agents in modern games. Furthermore, we identify bottlenecks and hard problems in these areas, which we believe offer promising research directions to accelerate the adoption of machine learning in game AI for the video game industry. Index Terms—game AI, reinforcement learning, hand-coded bots, AAA games, game production

Fig. 1. Environments used as research testbeds. For this study, we use two popular AAA games to showcase the challenges in applying RL for game AI. Top shows an in-game screenshot of EA SPORTS FC 25, a realistic physicsbased football simulation game. In this environment, we try to improve the positioning system of the goalkeeper AI with RL. Bottom shows an in-game screenshot of Battlefield 6, a team-based, large-scale, multiplayeroriented first-person shooter AAA game. In this testbed, we try to improve the locomotion system of on-ground soldiers.

I. I NTRODUCTION Machine Learning (ML), and in particular Reinforcement Learning (RL), has shown impressive results in playing a variety of games at human and superhuman levels, with notable examples such as Alphastar [1], GT Sophy [2], and others [3, 4, 5]. More recently, the ability to play a video game from scratch with no human interactions has proven useful in video game production settings, particularly for gameplay testing [6, 7]. For instance, RL-based agents have been employed in production to test releases of the Battlefield series [8], and an increasing number of gaming companies are applying MLbased techniques specifically for testing purposes [9, 10]. Although RL has primarily been applied to game testing in production settings, this technology has the potential to enhance gameplay experiences. Technological advances in the video game industry have created increasingly complex and * 979-8-3315-9476-3/26/$31.00 ©2026 European Union

immersive game environments. However, the development of Artificial Intelligence (AI), that control non-player characters (NPCs), is still a critical element in the creative process that can break or elevate the game experience for the player. RL has the potential to enhance game AI by enabling the creation of more authentic, reliable, and immersive NPCs [11, 12]. We envision more applications of RL-augmented game AI in the future. However, for this to materialize, current research limitations are prohibitive to broad deployment across game genres [13]. Therefore, in this vision paper, we propose a framework for training reinforcement learning models with a set of requirements in mind that are suited towards game AI and game development: Short Training Time - games in active development can change on a daily basis. An RL-based game AI system must therefore keep pace with the development cycle, requiring time-efficient retraining;

Controllability - designers and developers require qualitative control over the final behavior of the agents, in order to accomplish their vision of the game experience; Modularity - game AI systems are often structured modularly, as this provides developers with a high degree of controllability. Furthermore, hand-coded solutions for specific behaviors can still offer advantages over RL-based approaches. As a result, fully end-to-end systems based solely on RL are often viewed as suboptimal from a developer’s perspective. The RL system needs to be easily integrated into existing or future game AI systems, with a modular design; Maintainability - as we will see in Section II, a common drawback of hand-coded systems is that, as games scale in complexity and scope, the underlying AI systems become increasingly difficult to manage and maintain together with the introduced changes. Any RL system developed for a game must therefore remain maintainable over time and adaptable to future updates and releases of the same game; Bug Detection and Fixing - game AI systems need to be properly tested before they are delivered within commercial video games. After training agents with RL, designers require a systematic way to confirm that the game AI has the correct behavior, and to easily fix it in case of a problem; Authenticity - game AI systems do not require superhuman agents, but rather agents that are suitable for a playful experience. It is relatively easy to train RL agents with superhuman performances, but we argue that it is much harder to train authentic agents fulfilling the creative vision of the game; and Runtime Inference Constraints while it is relatively easy to train well-performing RL agents with large neural networks with generous compute resources, the reality is rarely the same for model deployment. Games need to be developed with the lowest hardware bound in mind. The model needs to run either on-device for single-player or peer-to-peer multiplayer games, or on dedicated game servers for larger-scale multiplayer games. This involves not only model inference, but also observation computation and collection. Further, the computational budget for an RL-augmented system must be shared with the non-RL parts of that system. We argue that these challenges are overlooked in the current academic literature, which either focuses on simple environments that do not capture the complexity of AAA games, proposes complex systems that require substantial computational resources, or focuses on superhuman performance. This paper first describes how the aforementioned challenges affect RL’s applicability in game production settings by reporting experiments integrating RL-based game AI into popular commercial AAA video games. The games we use as testbeds are EA SPORTS FC 25 and Battlefield 6. For the former – a realistic football-based simulation – the goal is to improve the goalkeeper AI’s positioning system using RL. For the latter – a first-person shooter video game – the goal is to improve the quality of ground infantry using RL. Finally, we will present lessons learned and outline research directions to

address the defined challenges and increase the adoption of RL-based game AI in the game industry. II. T RADITIONAL H AND -C ODED G AME AI In this section, we list a popular class of hand-coded AI methods for game development [14] that are useful for discussing the test cases we will see in Section IV. Finite State Machines. A Finite State Machine (FSM) is a programming pattern that models the behavior of NPCs as a graph of distinct states and transitions between these states given certain triggers. An FSM is composed of three components: a finite set of states that store information about a task; a number of transitions between states, which are triggered by specific events or conditions; and a set of actions that need to be executed within each state. The FSM’s simplicity and low computational cost makes it a practical choice for NPCs’ control within game production. However, large-scale FSMs can be very complex to design and maintain, and they offer limited room for dynamism, adaptability, and evolution [14]. Behavior Trees. A Behavior Tree (BT) is a modular game AI programming pattern that has been adopted in robotics [15]. A BT represents decision-making as a tree structure composed of control-flow and task nodes. The latter represent the lowlevel action the NPC executes, while the former can be of three types: sequence, with which the BT runs all child task nodes in sequence; selector, with which the BT runs specific child task nodes depending on metrics; and decorator, which adds complexity to a child behavior [14]. Their modularity and readability make BTs among the most widely used game AI techniques today. However, their dynamism and adaptability are limited, as they are static knowledge representations [14]. Goal-Oriented Action Planning. Goal-Oriented Action Planning (GOAP) is an AI architecture in which agents dynamically construct sequences of actions to achieve a desired goal. GOAP uses a planner, typically based on A* search, that works backward from the goal state: it evaluates the difference between the current world state and the goal, then chains together actions whose preconditions and effects bridge that gap. Each action in the system declares what world state it requires to execute and how it modifies the world state upon completion. The planner searches through these action combinations to produce a valid, cost-optimal plan at runtime. GOAP systems are limited by their reliance on engineered representations, which can become complex as the scale of the game increases; moreover, combinatorial search over the action space can become prohibitively expensive as the number of actions and world-state variables increases [16]. These AI techniques rely on established path-finding algorithms, combined with Navigation Meshes (NavMeshes), which represent traversable surfaces within a game level and impose movement limitations not experienced by human players. Although ML-based game AI can mitigate the aforementioned problems, these handcrafted systems do have value for authoring NPCs’ behavior, and we argue that game developers should leverage them to create compelling game

AI. The use of ML-based agents should not be seen as a means of replacing hand-coded AI, but rather as a complement to these systems. To showcase examples of where and why these systems can fall short and how ML-based agents can complement them, in the next section, we will describe the AAA video games we used in this study and their AI systems. III. C ASE S TUDIES In this section, we describe the two commercial video games we use as testbeds to showcase the challenges in applying RLbased game AI to AAA games. The games are EA SPORTS FC 25 and Battlefield 6. EA SPORTS FC 25. The game is part of the EA SPORTS FC series, with new entries released every year. Figure 1 top shows a screenshot of the game. The game is a physics-based football simulation where players compete against other humans and in-game AI. In this paper, we study the case of augmenting the goalkeeper’s game AI with RL. The system is an FSM that comprises different states, e.g. saving and positioning. Based on the situation, the goalkeeper AI switches to the relevant state; for instance, when entering a shot situation, it switches from the positioning state to the saving state. The positioning state – the main state telling the goalkeeper where to move in most of the situations – itself is another FSM with different states, based on the specific scenario, e.g. closing the angle between a potential striker and the goal. Although the current positioning provides overall good behavior for the goalkeeper, it suffers from three problems: believability, maintainability, and overall performance. Using the hand-coded system, the sudden switch between states is evident, making the goalkeeper’s movement less realistic. To cover all possible low-level cases, developers had to implement a large, complex FSM that is hard to expand and maintain. Battlefield 6. Battlefield 6 is a team-based, large-scale, multiplayer-oriented first-person shooter AAA game. Figure 1 bottom shows a screenshot of the game. This game has a multimodal gameplay structure where the player tries to defeat enemies either as on-the-ground soldiers, in ground vehicles, or in aircrafts; in this paper, we explore soldiers as our use case. The soldiers’ game AI is a mix of BT and GOAP that work together to deliver overall good performance. However, soldiers controlled by this system can exhibit unrealistic behavior, especially in locomotion – i.e. the actual positioning and movement of the soldier in the environment. For example, the complexity of their AI system and its reliance on static representations (e.g., NavMesh) make it hard to create a believable locomotion system, leading soldiers to often follow paths rigidly and without variation to reach a target position, or to fail to consider and react to the environment around them. Although RL can mitigate the problems we mentioned when applied to the game AI in the testbed games, its application to large-scale, complex systems is not straightforward. An RLbased system integrated into such complex games must address the challenges described in Section I, and we argue that current RL research is insufficient to fully address them.

IV. R EINFORCEMENT L EARNING FOR G AME AI In this section, we use the AAA game environments mentioned in Section III, on the one hand, to showcase examples of how we can improve specific game AI systems with RL, and on the other, to describe the task specifications and challenges. A. EA SPORTS FC 25 For this case, we are interested in replacing the goalkeeper’s positioning system in its game AI. Task Specifications. RL offers the potential to mitigate the three problems we identified in the hand-coded positioning system (see Section III): believability, maintainability, and overall performance. The behavior can be explained by a reward function rather than complex state interactions that an FSM would require, and this function can be defined by domain experts (e.g. professional goalkeepers) who are not skilled coders. The behavior is specified with a higher level of abstraction than coding, rewarding the agent for relevant micro behaviors and letting the algorithm figure out how to solve the task in both optimal and emergent fashion. This process can make it easier to create believable behaviors than by using FSMs. Moreover, in the case of RL, updating the agent’s behavior means updating the reward function (and potentially the training scenarios, as we will see later) and restarting training. In the case of FSMs, updating a behavior may require a deep understanding of the existing system and potentially introduce changes that could break the FSM. Here we list the requirements that the RL-based solution needs to satisfy: Authenticity: the main goal of this experiment is to have a more believable and human-like agent, compared to the existing hand-coded agent; • Short Training Time: modern game development follows a highly iterative process, where gameplay systems are regularly updated and refined. An RL training pipeline must therefore keep pace with frequent updates, requiring efficient retraining procedures. Our goal is to enable overnight training; • Modularity: the RL system needs to be integrated into an existing game AI logic, as we are improving only a small part of the entire system (i.e. the positioning system); • Bug Detection and Fixing: designers and developers need to be able to easily and quickly change the behavior of the agent in case they need to fix it; and • Runtime Inference Constraints: the model needs to run on-device, potentially in low-end machines such as video game consoles. •

Algorithm Choice. One of the biggest limitations of RL from a game production perspective is its relatively long training time. Games under development are often computationally expensive to simulate, inherently unstable, and prone to crashes. At the same time, the modularity requirement forces us to use the real game as a training environment, since the agent needs to interact with other systems in the

game. For EA SPORTS FC 25, we train the agent using a lowresolution configuration of the game. This setup removes nonessential graphical enhancements during training and enables unlocked frame rates, achieving up to 120 frames per second on a standard development machine (e.g. equipped with an NVIDIA RTX 4090 GPU). On the same machine, we can run five game instances in parallel during a training session. The agent executes one action every five frames, resulting in an overall throughput of approximately 120 samples per second across all instances. Compared to modern RL environments and frameworks, which can reach thousands of steps per second [17], this represents a relatively low data collection rate. Given these constraints, we selected the Soft Actor-Critic (SAC) algorithm [18], as it is among the most sample-efficient RL methods. However, a single SAC training session still required between two and four days, initially. For context, the training had to be achievable at most overnight to allow a developer to start a new training session at the end of the workday and obtain a newly trained agent by the following day. As described in prior work [11], to address this we employ several advanced techniques – such as a high update-to-data ratio with network resets, the use of pre-collected offline data, and scenario-based training – to reduce the duration of a SAC training session from 4 days to approximately 12 hours. Figure 2 left shows the difference in training performance between standard SAC and the modified variant used in this experiment. Although these relatively simple modifications substantially reduce training time, we argue that further research in this direction is necessary, as we will discuss in Section V. Regarding the neural network architecture, the runtime inference constraints requirement requires us to use a compact, computationally efficient model. In this game, we have a strict budget of 200 µs per inference call. By inference call, we refer to the complete pipeline of retrieving observations from the game engine, executing the model’s forward pass, and returning the selected action. The strict time constraint for inference stems from the modularity requirement. The goalkeeper’s AI system is implemented as a collection of interacting subsystems, each operating under its own CPU and GPU time allocation. We adopt a 5-layer Multi-Layer Perceptron (MLP) with SiLU activations and layer normalization, where each fully connected layer has 256 hidden units. The resulting network contains approximately 300,000 parameters and achieves a total inference time of 170 µs in the lowest-end configuration we tested, remaining within the allocated budget. Rather than simply scaling model size, we argue that RL research should prioritize small, sample-efficient architectures tailored to real-time deployment, thus enabling productionready RL solutions. To minimize friction in integrating these approaches into development tools and pipelines, an RL-based game AI system needs to provide a quick, reliable way to iterate on the trained agent. This process is useful when the model is released alongside the finished game. Player-discovered exploits, in which specific behavioral weaknesses are systematically leveraged to score goals, can compromise the integrity of

Fig. 2. Training performance using our approach in EA SPORTS FC 25. Left shows a comparison between an agent trained with standard SAC and our modified variant designed to satisfy the Short Training Time requirement. The dotted line indicates the performance of the built-in hand-coded AI that the RL agent aims to augment. Through targeted modifications, the proposed approach improves sample efficiency, enabling effective overnight training. Right shows a comparison between standard SAC fine-tuning and our proposed fine-tuning strategy in a scenario where designers identified undesirable behavior. Our method provides a more effective alternative to conventional SAC fine-tuning, enabling targeted behavioral corrections in approximately 2 hours.

the gameplay experience. Quick and rapid mitigation of such behaviors is therefore essential. In this scenario, the training time budget requirement is significantly lower than for a full training cycle – approximately 4 hours. We make use of our previous work [11] and combine scenario-based learning with the Replay across Experiments (RaE) technique [19]. This approach enables targeted fine-tuning of the RL agent within 2 to 4 hours, depending on the exploit’s complexity, thereby satisfying the production constraint. As with the main training pipeline, relatively simple modifications bring RL-based solutions closer to a production-ready tool. However, this strategy has diminishing returns: repeated fine-tuning increases the risk of catastrophic forgetting. Figure 2 right compares standard SAC fine-tuning with our proposed approach. The resulting agent effectively addresses the three challenges identified earlier. The new positioning system exhibits an overall behavior that playtesters perceive as more believable, authentic, and human-like. Moreover, behavioral adjustments can be implemented easily by developers and designers. As a side note, with the new system, the goalkeeper achieves a 10% higher save ratio compared to the previous hand-coded solution. B. Battlefield 6 As mentioned in Section III, we are interested in augmenting the underlying game AI system with improved locomotion behavior using RL. Task Specification. In this task, our objective is less focused on designing a rich, dense reward function and more concerned with addressing limitations of the existing locomotion game AI system, such as rigid pathfinding and limited contextual awareness. For locomotion, the underlying BT relies on a pathfinding algorithm and navigation meshes to traverse to a given destination. With RL, instead of relying on state discretization, the model can learn from continuous-state observations and directly output continuous movements. While effective, such solutions can result in behavior that appears artificial. To be practically integrated into the current development cycle, a potential RL approach must satisfy several requirements,

Fig. 3. Modes of perception explored in Battlefield 6. Left Raycast fan of 24 rays with high enough density to allow detection of all obstacles within a 10 m radius of the agent. Right Occupancy map of size 50 × 50. Each pixel color represents either: agent (yellow), terrain (purple), obstacle (blue) or target waypoint (green). When the waypoint is out of range, it is mapped to the border of the occupancy map as a directional indicator.

Fig. 4. Comparison of agent training with raycasts (blue) and occupancy maps (orange) in Battlefield 6. Left Average episodic reward. Right Average rate over time where the agent successfully finds the target waypoint before a set step limit of 500. As seen in the graphs, both approaches lead to comparable performance. However, occupancy maps are computationally faster to run by a factor of ≈ 2.0, making the approach more attractive in a game production setting.

some of which overlap with those discussed in the previous use case: • Short Training Time: as for EA SPORTS FC 25, the game will continuously change during development, requiring the RL training framework to allow for rapid adaptation to frequent gameplay and system changes; • Modularity: to preserve the existing BT structure, the locomotion components must be implemented as separate agents, replacing the corresponding leaf node within the BT and must be able to interact with other hand-coded nodes; • Runtime Inference Constraints: in this setting, the primary computational bottleneck is observation collection rather than model inference time; and • Authenticity: the main objective is not to maximize quantitative performance, but to generate behavior that appears more authentic and believable compared to the current hand-coded implementation. Algorithm Choice. In contrast to EA SPORTS FC 25, sample efficiency is not the main challenge in this setting. For this particular game, we can instantiate multiple environments on the same machine, each containing several agents running in parallel. Furthermore, training can be conducted using a headless dedicated server version of the game, without a graphical interface, increasing simulation throughput. In total, this setup allows up to 240 agents to run concurrently.

Fig. 5. Comparison of locomotion systems in Battlefield 6. Top: RLaugmented locomotion system. Bottom: NavMesh-based game AI. The RLaugmented agent exhibits smoother, more natural trajectories that resemble human player behavior. In contrast, the hand-coded system – mainly due to the discretization of the NavMesh representation – produces more rigid and less realistic movement patterns.

Given these conditions, we prioritize a simpler, more stable algorithm such as Proximal Policy Optimization (PPO), which is less computationally intensive than SAC. Training time is approximately 2 hours. We are interested in a simple task: navigating the environment from a random starting position to a random target waypoint while avoiding obstacles. The agent is rewarded monotonically for reducing its distance to the target waypoint, and is penalized for colliding with obstacles. Figure 5 shows an example of the training environment. Contrary to many state-of-the-art approaches used in literature, which rely on raycasting techniques [20, 21] or first-person vision-based agents [22], the Runtime Inference Constraints requirement prevents us from adopting such solutions. Extracting these representations requires significant computational resources, making observation collection – rather than model inference – the main bottleneck in our setting. Instead, we leverage data structures already available within the game engine, specifically heightmaps, which we repurpose as occupancy maps centered around the agent. At game boot-up, we leverage the initial step of the NavMesh generation process to cache a level-wide representation of where non-traversable areas and obstacles are. At runtime, we query this cache using the agent’s current in-world position for a local region. Before constructing the final observation for the agent, we construct the occupancy map with 4 categorical pixel value types: the agent, terrain, obstacles, and finally the target waypoint. To signify the cost difference between occupancy maps and raycasts, we run a baseline experiment utilizing 24 raycasts in a 360◦ fan around the agent [20]. Figure 3 showcases examples

of both the raycast fan and occupancy map respectively. The resulting neural network architecture is composed of two input paths: an MLP of three linear layers, sized 64, 128, and 64 respectively for auxiliary game features, and either an additional three-layer network of the same shape when using a raycastbased perception, or a 5 layer convolution-based encoder when using the aforementioned occupancy map, all with a kernel size of 3 × 3, stride of 2 and no padding, besides the first layer responsible for the pixel type embedding which utilizes a kernel size of 1×1. Figure 4 compares training performance when using raycast-based observations versus occupancy map representations. While both approaches achieve comparable overall performance, occupancy maps are significantly less computationally expensive. Comparing the average wall-clock time of running the aforementioned 24 raycasts with building the occupancy map, the former takes about 27 µs while the latter only 14 µs, amounting to a ≈ 2.0× speed-up even with a conservative number of raycasts. Nevertheless, the occupancy map representation remains suboptimal. It does not capture several important aspects of game AI, such as multi-layered environments, vertical structures, irregular terrain, dynamic obstacles, and destruction. We therefore argue that further research is needed to develop computationally efficient yet expressive game-state representations that better balance realism, scalability, and deployment constraints in production environments. Figure 5 shows the qualitative difference between the human-like locomotion of the RL agent, compared to the existing NavMesh-reliant system, showing that the RL-based agent better satisfies the Authenticity requirement. As we mentioned in the Modularity requirement, the RLbased locomotion system needs to be integrated with other existing nodes in the game AI bots. To illustrate, we run an experiment in a testing environment where an RL-augmented agent faces a hand-coded opponent in a 1-on-1 scenario. To get a high success rate, the RL-based locomotion node needs to interact with other systems, such as the aiming and shooting behavior. We run 20 testing episodes, with the RLaugmented agent winning 11 episodes. This shows that the RLaugmented locomotion has performance similar to the handcoded game AI but exhibits more authentic behavior. Although recent work suggests that jointly training multiple leaves in an RL-augmented BT can yield improved performance [15], this experiment demonstrates that even this simpler approach – i.e. training policies independently and integrating them within the BT – meets the Modularity requirement while maintaining satisfactory behavior. V. F UTURE R ESEARCH D IRECTIONS In Section IV, we showed how RL-augmented game AI can improve the overall performance of standard techniques. On the other hand, RL integration into game AI comes with a set of requirements that are often overlooked in the current literature, but that we argue are fundamental to enabling the mass adoption of RL in game AI. In this section, we propose research opportunities that we believe will be essential for driving research and, therefore, industry forward.

Designers-in-the-Loop. To satisfy requirements such as Controllability and Maintainability, an RL-based game AI system must prioritize designers as primary users. Designers require qualitative control over the agent’s final behavior in order to express creative intent and ensure alignment with the overall gameplay vision. For this reason, they need to have an active role within the learning framework. The approach discussed in Section IV-A, originally introduced in our previous work [11], represents a step in this direction. While human-in-the-loop learning has demonstrated promising results in domains such as large language models [23] and robotics [24] – primarily to improve safety and alignment – applications in games have largely focused on improving task performance or sample efficiency. Sestini et al. [6] propose an interactive method for creating playtesting agents; however, playtesting agents operate under different requirements and goals than production game AI agents [8]. Similarly, Zhao et al. [25] adopt a behavioral cloning–like approach in which agents learn policies directly from designer demonstrations, relying only on explicit demonstrations as supervision. To the best of our knowledge, existing work does not address the problem of providing designers with an authoring tool for shaping qualitative behaviors in production game AI. Furthermore, the question of how designers should interact with the training process – and what representation of feedback (e.g. demonstrations, preference comparisons, or dense reward shaping) is most effective – remains unexplored. Efficient Networks and Fast Training. During a video game development cycle, the game changes on a daily basis – for example, through changes in assets, mechanics, or gameplay dynamics. Such updates may require retraining the same agent from scratch multiple times. To remain practical, the training process must therefore keep pace with development, requiring fast turnaround times. Recent research has gone into two main directions to accelerate RL training. On one hand, projects such as PufferLib [17] propose high-speed, lower-fidelity simulators combined with domain randomization, enabling agents to be trained efficiently and transferred to the target environment (i.e. the full game). On the other hand, a growing body of work focuses on improving sample and computational efficiency directly within the target environment [26, 27]. Given the complexity of AAA video games – and in light of the Modularity requirement – we argue that the latter approach is more viable in this production setting. Games are complex systems, and constructing simplified simulators, particularly when the goal is to augment existing AI components, can be expensive or impractical. The Runtime Inference Constraints limit the size of the network, with game studio preferring small and efficient architectures tailored for real-time deployment. For example, one can think of using an asymmetric actorcritic architecture [28] where the critic has a larger size and expressivity [29] than the actor. Finally, such efficiency come with further software and hardware dependencies that define the deployment environment. Can the model run on GPU, or only on CPU? Which framework (e.g. ONNX, TensorRT) is

needed to deploy the model? Fine-Tuning and Correcting Behaviors. Modern games are constantly evolving live services for the duration of their lifecycle. Even after release, patches may introduce gameplay changes – for instance, to address newly discovered player exploits. An RL-based game AI system must therefore be able to adapt rapidly to such updates. However, fine-tuning RL agents – particularly when qualitative aspects of behavior must be modified – remains a challenging problem. RL policies tend to overfit to the training environment, and once fine-tuned, they risk catastrophic forgetting of the pre-trained behaviors [30, 31]. The Modularity requirement directly addresses this issue. Decomposing the overall behavior into smaller policies responsible for specific tasks can simplify iterative updates, as we may need to fine-tune only a specific sub-policy, rather than the entire system. We argue that systematic methods for behavioral fine-tuning in RL remain underexplored. While recent work on open-ended and continual learning [32, 33] demonstrates that agents can adapt over long time horizons, it is still unclear how such approaches can be translated into practical, production-oriented workflows, such as those required in game development. Modularity versus End-to-End. Recent RL research has largely focused on developing end-to-end models capable of handling all aspects of game AI within a single unified architecture [1, 2]. While such approaches have demonstrated impressive results in controlled settings, the practical constraints of game development often make end-to-end solutions impractical for production use, due to high research demands and substantial training times. In contrast, established systems such as the cited FSMs and BTs have been widely adopted in the industry due to their modularity and interpretability. Although these systems have limitations, they provide reliable and efficient solutions. Rather than replacing these frameworks entirely, RL-based game AI should focus on augmenting existing systems to overcome their specific shortcomings. The examples presented in Section IV demonstrate that both FSMs and BTs can be augmented with RL components to produce qualitatively improved behaviors. Moreover, recent work on RL-augmented BTs [15] suggests that such modular integrations are also beneficial in other domains, including robotics. The problem of jointly training modular policies within a partially hand-authored control architecture remains unsolved. Moreover, how to switch policies in a BT, and the best switching handover, remain open challenges. Perception. As discussed in Section IV-B, Runtime Inference Constraints limit the applicability of RL-based game AI not only due to the use of large neural networks, but also because of the cost of state extraction. Standard state representations used by modern approaches [20, 21, 22] often require significant computational resources, which are often unavailable in commercial video games that must run across a wide range of hardware platforms. 3D spatial understanding remains fundamental for embodied agents in games. Our experiments with occupancy maps demonstrate that it is possible

to achieve performance comparable to state-of-the-art methods using more computationally efficient representations. However, occupancy maps provide only a partial representation of the surrounding 3D environment. We argue that further research is needed to develop computationally efficient yet expressive 3D representations, as explored in recent work such by Ying et al. [34]. Authenticity. Recent successes of RL in video games have shown impressive results, including superhuman agents capable of defeating professional players in complex, skillintensive environments [1, 2]. However, achieving superhuman performance is not the primary goal of RL-based game AI in production contexts. From a player’s perspective, competing against a superhuman agent can be frustrating and may detract from the intended gameplay experience. Instead, RL-augmented game AI should contribute meaningfully to game design by producing behaviors that are authentic and believable. For example, in a game such as EA SPORTS FC 25, a goalkeeper should behave like a human professional goalkeeper; similarly, in a game such as Battlefield 6, an infantry soldier should exhibit credible tactical behavior. Translating such qualitative design goals into explicit mathematical reward functions, however, is often difficult and unintuitive. Large-scale behavioral cloning approaches [22, 35, 36] and methods based on inverse RL [37, 38] have been explored to generate more human-like behaviors in games. Behavioral cloning typically requires extensive datasets of human demonstrations, which are difficult to collect, particularly during active development. In contrast, a production-oriented system should ideally learn from a limited number of demonstrations provided directly by designers. However, in case the goal is to provide player-like behavior, large-scale behavioral cloning is indeed a viable research direction [22]. Inverse RL-based approaches can, in principle, operate with smaller datasets, but they often exhibit instabilities, which can limit their reliability in production environments [38, 39]. Behavior Evaluation. Within the video game development process, all features – including game AI components – must undergo proper testing and validation. However, evaluating RL-augmented game AI systems presents unique challenges, as such models often behave as black boxes, making them difficult to interpret and assess. Robust evaluation, including quantitative performance metrics and qualitative behavioral analysis, is essential to building trust among developers and designers. We argue that systematic evaluation and validation frameworks for RL-based game AI remain underexplored in the current literature, despite being a critical prerequisite for their mass adoption in commercial game development. VI. C ONCLUSIONS In this paper, we described the challenges that arise when RL is directly applied to game AI in AAA production environments, and we analyzed the technical challenges associated with meeting these requirements. Through our experiments, we demonstrated that these constraints can be addressed

through targeted research and modifications of established RL algorithms. Our results suggest that RL offers a promising framework for augmenting rather than replacing traditional game AI techniques. However, for RL to achieve broader adoption in game production, improvements in usability, stability, controllability, and integration workflows are still needed. We have outlined several directions that we believe are relevant for advancing the adoption of RL-based game AI. In this work, we have focused on RL as one possible ML–based augmentation for game AI. However, alternative approaches such as large language model based systems also offer promising directions [40, 41]. For example, one could imagine a large language model integrated within a BT architecture, where leaf nodes are implemented through either handauthored logic or RL policies. We argue that RL currently represents one of the most mature and practically deployable techniques for augmenting existing game AI pipelines. We leave a systematic exploration of how other ML paradigms can be integrated into production-ready game AI systems for future research. R EFERENCES [1] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik et al., “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” Nature, 2019. [2] P. R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlic, F. Eckert, F. Fuchs et al., “Outracing champion gran turismo drivers with deep reinforcement learning,” Nature, 2022. [3] C. Berner, G. Brockman, B. Chan et al., “Dota 2 with large scale deep reinforcement learning,” arXiv preprint arXiv:1912.06680, 2019. [4] D. Bairamian, P. Marcotte, J. Romoff, G. Robert, and D. Nowrouzezahrai, “Minimax exploiter: A data efficient approach for competitive self-play,” in Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, 2024. [5] H. Wei, J. Chen, X. Ji, H. Qin, M. Deng, S. Li, L. Wang, W. Zhang, Y. Yu, L. Linc et al., “Honor of kings arena: an environment for generalization in competitive reinforcement learning,” Advances in Neural Information Processing Systems, 2022. [6] A. Sestini, J. Bergdahl, K. Tollmar, A. D. Bagdanov, and L. Gisslén, “Towards informed design and validation assistance in computer games using imitation learning,” in IEEE Conference on Games (CoG), 2023. [7] A. Sestini, L. Gisslén, J. Bergdahl, K. Tollmar, and A. D. Bagdanov, “Automated gameplay testing and validation with curiosity-conditioned proximal trajectories,” IEEE Transactions on Games, 2022. [8] J. Gillberg, J. Bergdahl, A. Sestini, A. Eakins, and L. Gisslén, “Technical challenges of deploying reinforcement learning agents for game testing in aaa games,” in IEEE Conference on Games (CoG), 2023. [9] modl, “modl.ai,” 2026, https://modl.ai/ [Accessed: 2026]. [10] nunu, “nunu,” 2026, https://nunu.ai/ [Accessed: 2026]. [11] A. Sestini, J. Bergdahl, J.-P. Barrette-LaPierre, F. Fuchs, B. Chen, M. Jones, and L. Gisslén, “Human-like goalkeeping in a realistic football simulation: a sample-efficient reinforcement learning approach,” in Reinforcement Learning Conference, 2026. [12] A. Sestini, A. Kuhnle, and A. D. Bagdanov, “Deepcrawl: Deep reinforcement learning for turn-based strategy games,” arXiv preprint arXiv:2012.01914, 2020. [13] M. Jacob, S. Devlin, and K. Hofmann, ““it’s unwieldy and it takes a lot of time”—challenges and opportunities for creating agents in commercial games,” in AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2020. [14] G. N. Yannakakis and J. Togelius, “Ai methods for games,” in Artificial Intelligence and Games. Springer, 2025. [15] M. Kartasev, J. Saler, and P. Ögren, “Improving the performance of backward chained behavior trees that use reinforcement learning,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023.

[16] O. Jeff, “Applying goal-oriented action planning to games,” AI game programming wisdom, 2003. [17] J. Suarez, “Pufferlib 2.0: Reinforcement learning at 1m steps/s,” in Reinforcement Learning Conference, 2025. [18] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel et al., “Soft actor-critic algorithms and applications,” arXiv preprint arXiv:1812.05905, 2018. [19] D. Tirumala, T. Lampe, J. E. Chen, T. Haarnoja, S. Huang, G. Lever, B. Moran, T. Hertweck, L. Hasenclever, M. Riedmiller et al., “Replay across experiments: A natural extension of off-policy rl,” arXiv preprint arXiv:2311.15951, 2023. [20] N. Justesen, M. Kaselimi, S. Snodgrass, M. Vozaru, M. Schlegel et al., “Human-like bots for tactical shooters using compute-efficient sensors,” IEEE Transactions on Games, 2025. [21] E. Alonso, M. Peter, D. Goumard, and J. Romoff, “Deep reinforcement learning for navigation in aaa video games,” arXiv preprint arXiv:2011.04764, 2020. [22] T. Pearce and J. Zhu, “Counter-strike deathmatch with large-scale behavioural cloning,” in IEEE Conference on Games (CoG), 2022. [23] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, 2022. [24] D. Marta, S. Holk, M. Vasco, J. Lundell, T. Homberger, F. Busch et al., “Flora: sample-efficient preference-based rl via low-rank style adaptation of reward functions,” in IEEE International Conference on Robotics and Automation (ICRA), 2025. [25] Y. Zhao, I. Borovikov, F. de Mesentier Silva, A. Beirami, J. Rupert, C. Somers, J. Harder, J. Kolen, J. Pinto, R. Pourabolghasem et al., “Winning is not everything: Enhancing game development with intelligent agents,” IEEE Transactions on Games, 2020. [26] M. Schwarzer, J. S. O. Ceron, A. Courville et al., “Bigger, better, faster: Human-level atari with human-level efficiency,” in International Conference on Machine Learning, 2023. [27] C. Romeo, G. Macaluso, A. Sestini, and A. D. Bagdanov, “SPEQ: Offline stabilization phases for efficient q-learning in high update-to-data ratio reinforcement learning,” in Reinforcement Learning Conference, 2025. [28] O. Mastikhina, D. Sreenivas, and P. S. Castro, “Optimistic critics can empower small actors,” arXiv preprint arXiv:2506.01016, 2025. [29] M. Nauman, M. Ostaszewski, K. Jankowski, et al., “Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control,” Advances in neural information processing systems, 2024. [30] M. Wołczyk, B. Cupiał, M. Ostaszewski, and M. e. a. Bortkiewicz, “Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem,” arXiv preprint arXiv:2402.02868, 2024. [31] C. Zhang, O. Vinyals, R. Munos, and S. Bengio, “A study on overfitting in deep reinforcement learning,” arXiv preprint arXiv:1804.06893, 2018. [32] M. Faldor, J. Zhang, A. Cully, and J. Clune, “Omni-epic: Openendedness via models of human notions of interestingness with environments programmed in code,” arXiv preprint arXiv:2405.15568, 2024. [33] S. Dohare, J. F. Hernandez-Garcia, Q. Lan et al., “Loss of plasticity in deep continual learning,” Nature, 2024. [34] Z. Ying, N. Edwards, and M. Kutuzov, “Efficient visibility approximation for game ai using neural omnidirectional distance fields,” ACM on Computer Graphics and Interactive Techniques, 2024. [35] Y. Yue, I. Salia, S. Hunt, C. Green, W. Shi, and J. J. Hunt, “Scaling behavior cloning improves causal reasoning: An open model for realtime video game playing,” arXiv preprint arXiv:2601.04575, 2026. [36] L. Magne, A. Awadalla, G. Wang, Y. Xu et al., “Nitrogen: An open foundation model for generalist gaming agents,” arXiv preprint arXiv:2601.02427, 2026. [37] W. Ahlberg, A. Sestini, K. Tollmar, and L. Gisslén, “Generating personas for games with multimodal adversarial imitation learning,” in IEEE Conference on Games (CoG), 2023. [38] A. Sestini, A. Kuhnle, and A. D. Bagdanov, “Policy fusion for adaptive and customizable reinforcement learning agents,” in IEEE Conference on Games (CoG), 2021. [39] J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adversarial inverse reinforcement learning,” arXiv preprint arXiv:1710.11248, 2017. [40] N. Justesen, P. Bontrager, J. Togelius, and S. Risi, “Deep learning for video game playing,” IEEE Transactions on Games, 2019. [41] R. Gallotta, G. Todd, M. Zammit, S. Earle, A. Liapis, J. Togelius, and G. N. Yannakakis, “Large language models and games: A survey and roadmap,” IEEE Transactions on Games, 2024.