Memory-Efficient Policy Libraries with Low-Rank Adaptation in Reinforcement Learning Lyngset, S. V.* , Raanaas, T. V.* , Sveipe, G.* , Nilsen, E. M.* , Torresen, J, Ellefsen, K.O. and Lømo T.†
arXiv:2606.25700v1 [cs.LG] 24 Jun 2026
Department of Informatics, University of Oslo
Abstract— When fine-tuning Large Language Models (LLMs), there has been success in minimizing both memory usage and computation with Parameter-Efficient Fine-Tuning (PEFT), like Low Rank Adaptation (LoRA). In this article, we have explored whether this approach is transferable to the world of robotics and Reinforcement Learning (RL), allowing learning with reduced memory usage and improved computational performance. Specifically, we focused on a version of multi-task robotics, where a library of specialist policies are created. In such a library memory efficiency is especially important. We used a Proximal Policy Optimization (PPO) algorithm and fine-tuned a baseline model to different tasks using LoRA. Our results demonstrate that, depending on the hyperparameters, LoRA can minimize memory usage by a factor of 20-160 compared to full fine-tuning of all layers. This implies a 90-95% storage saving when deploying a library of many (10-50) specialized policies, which can be the differentiating factor between being able to store the entire library in memory or having to use swap-memory in an applied robotics setting. At the same time, our results indicate that there is no significant difference in the success-rate between full fine-tuning and LoRA fine-tuning for the selected tasks.
I. I NTRODUCTION Multi-task Reinforcement Learning (RL) aims to find a policy that can solve multiple tasks. This is especially important within robotics, where one robot often needs to solve multiple distinct problems. Multi-task RL faces the challenge of catastrophic forgetting [9], where a policy network forgets the task it has previously been trained for when it is further trained for a new task. Due to catastrophic forgetting, one approach is to train a new policy for each task and save it to a library with specialist policies. Although simple, this approach is storage and memory intensive, as it has to store a separate policy for each task. This paper focuses on memory efficient creation of such a policy-library based on a base policy. Figure 1 illustrates how this can be implemented with a supervisor model that selects which specialized model to utilize, similarly to previous work in multi-task RL [18]. Low Rank Adaptation (LoRA) [5], has proved successful in limiting memory usage during fine-tuning, especially in Large Language Models (LLMs). However, the technique is still relatively unexplored within robotics. LoRA is a Parameter-Efficient Fine-Tuning (PEFT) technique that decomposes the weight updates of a pre-trained model into the product of two much smaller, low-rank matrices. During Email: † [email protected] * Equal contribution
Fig. 1: The use case that inspired our approach: A supervisor policy selects from a library of pre-trained specialist policies. This paper focuses on memory efficient training of the specialist policies.
training, the original model weights are kept frozen and only these low-rank matrices are optimized, effectively constraining the update space to a low-dimensional subspace [5]. Our experiments show that this technique can also be transferred to the realm of multi-task learning in robotics, enabling reduced training time, improving knowledge transfer, and limiting memory usage. RL has made great strides in training robots for a single task to a high level of success. Multi-task robotics is not explored as often, but several models with good results have been proposed [7]. When using separate policies for each task, a robot must be able to seamlessly switch between several policy networks. For a robot to switch between tasks without a long delay, the memory usage cannot be excessive. If the networks are too large in memory usage, one cannot store them all in memory, and the time it takes to transition into a different task makes the multi-task aspect less viable. In this paper, we show the viability of LoRA in online RL, specifically with Proximal Policy Optimization (PPO). In addition, we present a novel use case in robotics using LoRA to create a library of specialist models in a more
efficient manner. Our experiments show that using LoRA instead of fine-tuning all layers greatly reduces the memory needed for every network and might lead to less resource usage during the switching between policies. This is achieved while not decreasing the performance of the policies. We fine-tune every model from a pre-trained base policy trained on a general task. To our knowledge, this has not been shown before. II. R ELATED W ORKS A. Expert policy library Much work within robotics is focusing on the task of getting a robot to solve multiple tasks and to learn new tasks when needed. This can be split into two sub-fields with overlapping challenges, continual learning and multitask learning. For continual learning, also called lifelong learning or Incremental Learning, the focus is on learning new tasks without forgetting what has been learned before. This is called catastrophic forgetting, and is a difficult challenge to solve when working with a single policy network architecture. [17, 6] In the case of multi-task learning, the focus is more on getting the robot to be able to solve multiple tasks with the same architecture [16]. Instead of incrementally adding tasks, all tasks are available at the same time. Using a single policy architecture, this is challenging because different tasks can cause interference with each other, and conflicting gradients during training [19]. Finding a solution to the problem of continual learning and multi-task learning is an ongoing challenge. For this paper, we instead focus on a simpler solution, using an expert policy library instead of having a single policy architecture [3]. Having multiple expert policy networks means that each new task can have its own network, avoiding catastrophic forgetting between tasks. However, with this strategy, the challenge becomes how policies can be trained and stored in an efficient manner. We propose to do this using PEFT, as these methods have had great success in large foundational models [5]. B. PEFT and RL in Robotics Related methods for adapter-based fine-tuning have recently been explored for skill generalization in robotic RL. Lu et al. investigate the use of adapters to generalize a learned manipulation skill across multiple robotic embodiments [8]. In their approach, a disembodied hand is first trained using Soft Actor-Critic (SAC) [4] to solve a single task, such as opening a drawer. To transfer this abstract skill to robots with differing kinematic constraints, they employ adapter modules, including LoRA, updating only adapter layers while freezing the pre-trained policy. They introduce an inverse-kinematics feedback signal that rewards reachable end-effector poses, ensuring the generated trajectories are feasible for the specific robot and task. They report that this method yields improved generalization and success rates compared to fine-tuning of all layers across multiple mobile
manipulators, with LoRA adapters achieving greater task success in all evaluated robots. This is related to our work in that both approaches use LoRA to specialize a shared baseline policy. Their work focuses on generalizing a single skill from a disembodied hand to multiple robots and a small set of related tasks, such as door-opening and chair-pushing. In contrast, we study LoRA for multi-task learning on a single robot and measure how different ranks affect training success, memory usage, and computation when building a library of specialized policies from a shared PPO baseline. Another approach to an adapter based solution is mixtureof-experts (MOE) [11]. While traditionally used in an LLM environment, it has also been used within the robotics domain. Arora [1] utilizes smaller policies that represent broken down movements extracted from simple tasks, this is done by using external descriptive information. The gating mechanism then combines the movements into longer and more complex tasks. The idea is that all tasks can easily be sorted and categorized if translated into natural language metadata. The paper also hypothesizes training their gating mechanism by leveraging frozen pre-trained weights. Cheng [2] uses attention-based-mixture-of-experts which revolves around a backbone network that extracts the common features of each of the selected tasks. This in turn helps the agent understand the environment dynamics. Similar to Arora [1], Cheng [2] breaks down tasks into simple correlated interactions. For example, identify the object with which you are supposed to interact, using the gripper, and push/pull interactions. After separating all the tasks into specific interactions that correlate with solving different aspects of the said tasks, the attention module grades the expert based on their relevance in solving a given task. The difference between their solutions and ours is that they train all tasks in parallel, and their end goal is a single policy that can adapt to various seen and unseen tasks. Their advantage is that the algorithm works on unseen tasks by piecing together a policy based on simpler interactions. A disadvantage is that the success rate is lower than for a solution where you train a policy for each specific task in advance. Instability and different degrees of success can also be present depending on how the policy is pieced together from the pre-trained tasks. C. LoRA Low Rank Adaptation (LoRA) [5] is a technique designed to minimize memory usage during single-task fine-tuning. Unlike full fine-tuning of all layers, which updates all model weights, LoRA keeps the pre-trained model parameters frozen (represented as the green square in figure 2). Then it injects low-rank trainable matrices (the orange trapezoids in figure 2) into the existing layers to handle task specific updates. The LoRA-rank defines the inner dimension of the low-rank factorization, where the weight update ∆W is decomposed into two smaller matrices, A and B. This rank determines the expressive capacity of the adaptation. While a lower rank restricts the update to its most essential features,
a higher rank enables the model to capture more complex task-specific information. LoRA was created to combat the growing size of LLMs. The implementation reduces the number of trainable parameters by roughly a factor of 10,000, improving both memory efficiency and training speed. The approach assumes that parameter updates lie in a lowdimensional subspace of the weight space, which regularizes adaptation and improves stability. This makes LoRA highly attractive for RL, where training signals are often noisy and data per task limited. Although primarily designed for LLMs, the same principle can be applied to robotics. The modular and parameter-efficient updates of LoRA could enable robots to adapt previously learned control policies to new tasks with less retraining and storage overhead.
provides the framework for the environment we receive from Meta-World. Meta-World [20] is an open-source benchmark designed for multi-task and meta-RL. It provides 50 different robotic manipulation tasks, such as opening drawers, pushing objects, and pressing buttons. These tasks share a common Sawyer-arm setup and require combinations of reaching, pushing, and grasping. The tasks include both parametric variations (randomized object and goal positions) and qualitatively different objectives to encourage generalization. The benchmark defines several evaluation regimes: MT1 tests a single policy on many goal variations within one task, while the others (MT10, MT50, ML10 and ML45) require a single policy to solve 10, 45 or all 50 tasks. In the Meta-World paper [20], state-of-the-art methods achieve modest success on MT10 and struggle a lot on the full MT50, highlighting the difficulty of learning many tasks concurrently [20]. This paper departs from this intended use: we only employ the MT1 setting, fully training a baseline policy on one task, and then fine-tuning that policy for other tasks. Meta-World was created to standardize multi-task learning with a single policy for all tasks, while we only train for one single task at a time, focusing on the policy library use-case described in the introduction. The action space in the Meta-World environments consists of a tuple representing the change in 3D-space of the end-effector and the torque the gripper should apply. The observation space contains the position and orientation of the first object, the second object, and the position of the goal, in addition to all the data from the previous observation. In total, the observation space contains 39 values [20].
D. PPO
B. Training
Fig. 2: Overview of how LoRA works. The figure is a reconstruction of a figure from the original LoRA paper [5].
PPO [13] is a policy gradient method for RL. PPO provides a surrogate objective for optimizing policy networks by gradient descent in minibatches of trajectories sampled from the environment. This allows efficient optimization while maintaining stability by penalizing large deviations in the policy update. PPO retains much of the stability of the Trust Region Policy Optimization (TRPO) [14] algorithm, but with lower computational cost, approximating the trust-region constraint of TRPO, which requires second-order optimization, by a simple first-order clipping mechanism. Empirically, PPO has been found to match or exceed TRPO in stability and performance in common benchmark tasks [13]. PPO is typically implemented within an actor-critic framework similar to Advantage Actor-Critic (A2C) [10, 15], maintaining a value network (critic) in addition to the policy (actor) to perform advantage estimation, reducing the variance of the policy update. PPO has been widely adopted as a baseline algorithm for RL [12]. III. M ETHODS A. Environment For training and testing our models, we employ the Gymnasium API with the Meta-World environment. Gymnasium
We train a base policy on a single task to be further finetuned to create specialist policies for several target tasks, suitable for deployment in a policy library. We selected the pick-place task as our base task. Pick-place is a task where the robot moves a small cylinder between two random locations. We chose this task for the following reasons: 1) It requires 3-axis arm movement, in two stages. 2) It features randomized primary and secondary goals during training. 3) It is the task upon which all the reward functions in Meta-World are based [20]. The variety of target positions ensures that the policy does not learn a static solution, and the variety of movements required are similar to those necessary for many Meta-World tasks. These traits might allow the fine-tuning process to better benefit from knowledge transfer. The use of the gripper was also found to be a very important skill for the baseline model, as this is something almost all tasks need to do. When we tried to train a baselinemodel that had not learned to use the gripper, the LoRA fine-tuning struggled. Conceptually, this may be explained by the fact that the fine-tuning process needs to learn to use the gripper from scratch.
TABLE I: Training hyperparameters Hyperparameter
Value
Timesteps Rollout buffer Learning rate Minibatch size PPO Policy network PPO Value network PPO value function coefficient PPO clip range
2 000 000 8192 steps 10−4 256 3 dense layers of 400 nodes 3 dense layers of 400 nodes 0.4 0.15
Fig. 4: Illustrating how LoRA saves memory compared to fine-tuning of all layers by leveraging the base policy and adding its ”changes” on top of the base policy weights.
Fig. 3: Illustration of a LoRA layer with rank=1 showing how a small subset of the parameters can be tuned while still being able to add the weights element-wise.
We have selected a subset of the Meta-World tasks for testing, shown in figure 5. The tasks have been selected to represent a variety of the problems available in the MetaWorld task library. Thus, the training pipeline follows the following outline: 1) Train the base policy on the base task. 2) Fine-tune the base policy on a suite of target tasks. Through trial and error, we found that the hyperparameters listed in table I are efficient in our suite of tasks. Unspecified parameters use Stable Baselines3 (SB3) defaults [12]. These hyperparameters were used for both the baseline policy and all fine-tuning versions. All fine-tuning policies were trained for a maximum of 2 million steps. However, we used early stopping, which means that training was stopped if performance in 50 evaluation runs was at least 98%. For evaluation, the model used deterministic action selection. We trained fine-tuned models built on a pre-trained base model for our suite of target tasks using both LoRA, with a range of rank values from 1 to 12, and compared with full fine-tuning of all layers.
and speeds up the training. After a baseline had been trained, either full fine-tuning of all layers or LoRA was used to adopt the policy to the task. Figure 4 shows the difference between LoRA and Full fine-tuning. For initial training and full fine-tuning setup we use SB3´s default actor-critic policy, while for LoRA fine-tuning we have developed our own policy class that inherits from SB3’s ActorCriticPolicy class. The LoRA policy works by wrapping all the layers of a pre-trained model with our LoRaLinear-class that keeps the old frozen weights and creates two new small layers LoRAdown and LoRAup . It overrides the forward function in the following way: baseout = base(x) LoRAout = LoRAup (LoRAdown (x))
(1)
h = baseout + LoRAout Figure 3 illustrates this operation, which allows us to use the SB3 PPO model with very little modification. With our relatively small model, LoRA reduces the trainable parameters approximately 20-160 times compared to finetuning of all layers. IV. R ESULTS Due to randomness in the different environments, each variation was trained 100 different times. Five different ranks were used, 1, 2, 4, 8, and 12, and were compared with full fine-tuning of all layers. This means that a total of 3600 runs were completed. The following is an explanation of how the metrics for training performance and estimation of computation were calculated. A discussion of the results will be presented in the next section.
C. Architecture
A. Training performance
We employed the PPO algorithm from SB3 [12] to train agents within the Meta-World benchmark suite. To train multiple environments in parallel, we usedSB3’s VecEnv to use 4 environments simultaneously. The observations and rewards was also normalized. This helps prevent over-fitting
Achieving early stopping through a 98% evaluation success rate is defined as successful training in our results, while failing to achieve success in 2 million timesteps counts as a failure. Figure 5 displays the six tasks that were tested, with a boxplot showing the success rate and time steps to success
for the different versions of fine-tuning. The boxplot also shows the variability in timestep to success between different runs with the same configuration. Figure 6 provides a more detailed look at the success rate of different rank choices for three selected tasks. Here, the performance is plotted during the training, with the mean performance between runs shown. B. Computation calculation To quantify computational cost, we tracked the number of Floating-Point Operationss (FLOPs) per PPO update, including both a forward and a backward pass. The total FLOP was normalized by the number of environment steps (nsteps × nenvs ) to obtain computation per PPO environment step: FLOPstep =
FLOPupdate nsteps × nenvs
The total computation used throughout the training (in MFLOP) was obtained by multiplying FLOPstep by the total number of PPO environment steps and dividing by 106 . FLOPstep × Nsteps . 106 Since all policies share the same architecture, FLOP per update depends only on the LoRA rank, not on the task. MFLOPtotal =
V. D ISCUSSION
TABLE II: Number of trainable parameters for each setup. The LoRA setup is compared to the full fine-tuning. Configuration
Trainable Parameters
Savings (%)
LoRA Rank 1 LoRA Rank 2 LoRA Rank 4 LoRA Rank 8 LoRA Rank 12 Full fine-tuning
4,082 8,160 16,316 32,628 48,940 675,609
99.4 98.8 97.6 95.2 92.8 0
TABLE III: Storage comparison for model library use case assuming a shared base policy (2.7 MB) with LoRA adaptersonly per fine-tune with LoRA-rank 8. Total Storage (MB)
Savings
Number of policies
Full Fine-tune
Base + LoRA
MB
%
2 5 10 50
5.4 13.5 27.1 135.5
2.8 3.4 4.1 9.5
2.6 10.1 23.0 126.0
48 75 85 93
5. In testing, we observed that this task seems to particularly benefit from knowledge transfer, as PPO struggles to solve the task when trained from scratch, but fine-tuned policies achieve a strong success rate. But why it also seems to benefit from more constrained lower rank fine-tuning policies would require further inquiry.
A. Choice of rank The LoRA rank parameter determines the complexity of the fine-tuning layers, and also the amount of memory and computation savings achieved. Thus, it should be chosen as low as possible while maintaining the desired level of performance. Figure 5 displays the training timesteps required to achieve success for a given rank for our suite of tasks. Observe that for a number of tasks, namely button-press-topdownwall, hammer, drawer-open, and window-open, even the lowest possible rank of 1 consistently achieves success, albeit typically with slightly increased training time. The basketball and peg-unplug-side tasks might be considered harder tasks. We see that these require a higher rank of about 2 to 8. In particular, rank 1 struggles to solve basketball, yielding a high, but not absolute, training failure rate. Figure 6 gives a more detailed look at how rank impacts the evaluation success rate. The hard tasks, basketball and peg-unplug-side, resemble the basketball plot, with low ranks struggling to achieve a high success rate. However, it is not impossible, as seen in figure 5. The easier tasks tend to follow a trend similar to the drawer-open or window-open plots in figure 6, with all ranks achieving training success with small differences in training time required. The window-open task shows an especially clear distribution of training performance for each rank. The drawer-open task is unusual among our suite of tasks, as it is the only one where the LoRA policies achieve higher performance (lower training failure in particular) than the full fine-tune baseline. This can be observed in both figures 6 and
B. Memory efficiency Full fine-tuning of all layers updates 675,609 parameters at each training step, while LoRA updates many fewer. As shown in Table II, LoRA reduces the number of trainable parameters by more than 95% for rank 8 and nearly 99% for rank 2, resulting in a reduction of roughly 20 to 160 times the number of trainable parameters compared to full fine-tuning. With a shared base policy of 2.7 MB, storing LoRA parameters only yields substantial savings at scale: for ten fine-tuned tasks, the storage drops by about 92% relative to maintaining separate fully fine-tuned tasks (see Table III). Each set of LoRA parameters adds only 135 KB (rank 8), allowing hundreds of specialized task policies to be efficiently stored and deployed on limited robotic hardware. The effect of this reduction can be seen in Figure 7. C. Computation In Figure 8, we compare the computational cost of performing a single timestep for different ranks and full fine-tuning of all layers. In general, the results highlight that LoRA maintains competitive learning efficiency while substantially reducing computational cost. In our FLOP accounting, we include both forward and backward passes for each PPO update. The forward pass is almost identical for LoRA and full fine-tuning, with LoRA introducing only a very small overhead from the adapter layers, ranging from +0.6% for rank 1 to +7.3% for rank 12 in forward FLOP. However, the backward pass is where LoRA achieves its computational savings, since only
Fig. 5: Boxplots of timesteps until success is achieved (success rate ≥ 0.98) for 100 runs of the tasks illustrated in the left column. Number of runs that failed to achieve success within the given timesteps (2 · 106 ) is indicated above the axes. Since it is out of 100 runs, this can also be interpreted as failure rate in percent.
Fig. 6: Evaluation success rate achieved by rank, for a selection of tasks.
a small number of low-rank parameters are trainable, the backward cost decreases substantially, by approximately 4350% compared to fine-tuning all layers in the policy. Because PPO performs multiple forward passes per backward pass during each update, the reduction in backwardpass cost does not translate directly into an equally large reduction in total computation per PPO step. This is why the FLOPs per PPO environment step only decreases by roughly 25–30% (as shown in Table IV), although the backward pass itself is much lighter when using LoRA. The observed reduction therefore stems almost entirely from the cheaper backward pass. It is important to note that the number of FLOP reflects the
computational cost of training the network and not the entire training process. The most computationally intense part of training is the simulation and data collection. This means that while there are gains in computational performance using LoRA for training, the actual impact on speed and resource usage in this simulator is negligible. However, the results may indicate that the results can have a bigger impact in other settings where this imbalance is less present. VI. C ONCLUSION In this paper, we have explored the use of LoRA in online RL using PPO. This was done in a robotic setting, specifically manipulation tasks. The results show that the
TABLE IV: FLOPstep for each fine-tuning configuration. FLOP are constant across tasks and depend solely on the LoRA rank.
Fig. 7: Stored parameter growth of the specialist policy library using LoRA rank 8.
Fine-tuning Method
FLOPstep
Savings (%)
LoRA Rank 1 LoRA Rank 2 LoRA Rank 4 LoRA Rank 8 LoRA Rank 12 Full fine-tuning
12,223,888 12,329,916 12,541,972 12,966,084 13,390,196 17,503,460
30.2 29.6 28.3 25.9 23.5 0
In addition more work can be done on a larger set of tasks with greater variety and testing more algorithms like Soft Actor-Critic, Deep Q-Networks or Deep Deterministic Policy Gradient. ACKNOWLEDGMENT This work is partially supported by the Research Council of Norway as part of the Vulnerability in the Robot Society (VIROS) project (grant no. 288285), the Predictive and Intuitive Robot Companion (PIRC) project (grant no. 312333), Collaboration on Intelligent Machines (COINMAC-2) project (grant no. 309869), through the Centres of Excellence scheme, RITMO (project no. 262762), and through the Norwegian Center for Embodied AI (NCEI) under grant agreement no. 357451
Fig. 8: Computation in the backwards step of model training for a given LoRA rank.
performance of fine-tuning using LoRA is very similar to full fine-tuning of all layers. More specifically, LoRA fine-tuning gives a memoryusage reduction of 20-160 times depending on the LoRA rank (1-12). Across all tasks tested, LoRA closely matched the performance of full fine-tuning while requiring less computation and memory. For simpler manipulation tasks such as drawer-open, hammer, and button-press-wall, all LoRA ranks converged rapidly to the same reward plateau as the fully fine-tuned model. More complex tasks such as basketball and peg-unplug-side suffered in the lower LoRA ranks. However, with higher ranks (more complex fine-tuning, less memory saving), it performed at the level with full fine-tuning. This might indicate how much can be inherited from the baseline task. We propose to use LoRA fine-tuning for creating a library of specialist policies for use in multi-task robotics. Using LoRA, the memory savings increases with the number of policies saved in the library. Already with 10 specialist policies, the memory saving is 85% compared with saving the fully fine-tuned policies. This can have a significant effect when switching between several policies in systems, such as mobile robots, with limited resources. There is a potential for further research on this idea. Our experiments relied on a single baseline policy. testing the effect of different tasks as the baseline would be interesting.
R EFERENCES [1]
[2]
[3]
[4]
[5]
[6]
[7]
Rushiv Arora. Multi-Task Reinforcement Learning with Language-Encoded Gated Policy Networks. 2025. arXiv: 2510.06138 [cs.LG]. URL: https:// arxiv.org/abs/2510.06138. Guangran Cheng et al. “Multi-Task Reinforcement Learning With Attention-Based Mixture of Experts”. In: IEEE Robotics and Automation Letters 8.6 (2023). Coline Devin et al. “Learning modular neural network policies for multi-task and multi-robot transfer”. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). 2017, pp. 2169–2176. DOI: 10. 1109/ICRA.2017.7989250. Tuomas Haarnoja et al. “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor”. In: International conference on machine learning. Pmlr. 2018, pp. 1861–1870. Edward J Hu et al. “LoRA: Low-Rank Adaptation of Large Language Models”. In: International Conference on Learning Representations. 2022. Sang-Woo Lee et al. “Overcoming catastrophic forgetting by incremental moment matching”. In: Advances in neural information processing systems 30 (2017). Timothée Lesort et al. “Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges”. In: Information fusion 58 (2020).
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
Kai Lu et al. “Learning Generalizable Manipulation Policy with Adapter-Based Parameter Fine-Tuning”. In: 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2024. Michael McCloskey and Neal J Cohen. “Catastrophic interference in connectionist networks: The sequential learning problem”. In: Psychology of learning and motivation. Vol. 24. Elsevier, 1989, pp. 109–165. Volodymyr Mnih et al. “Asynchronous methods for deep reinforcement learning”. In: International conference on machine learning. 2016, pp. 1928–1937. Siyuan Mu and Sen Lin. A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications. 2025. arXiv: 2503 . 07137 [cs.LG]. URL: https://arxiv.org/abs/2503.07137. Antonin Raffin et al. “Stable-baselines3: Reliable reinforcement learning implementations”. In: Journal of machine learning research 22.268 (2021), pp. 1–8. John Schulman et al. Proximal Policy Optimization Algorithms. 2017. arXiv: 1707 . 06347 [cs.LG]. URL: https://arxiv.org/abs/1707.06347. John Schulman et al. “Trust Region Policy Optimization”. In: Proceedings of the 32nd International Conference on Machine Learning. 2015, pp. 1889–1897. Richard S Sutton et al. “Policy Gradient Methods for Reinforcement Learning with Function Approximation”. In: Advances in Neural Information Processing Systems. Vol. 12. MIT Press, 1999. Nelson Vithayathil Varghese and Qusay H Mahmoud. “A survey of multi-task deep reinforcement learning”. In: Electronics 9.9 (2020), p. 1363. Liyuan Wang et al. “A comprehensive survey of continual learning: Theory, method and application”. In: IEEE transactions on pattern analysis and machine intelligence 46.8 (2024), pp. 5362–5383. Jiaxu Xing et al. “Multi-task reinforcement learning for quadrotors”. In: IEEE Robotics and Automation Letters (2024). Tianhe Yu et al. “Gradient surgery for multi-task learning”. In: Advances in neural information processing systems 33 (2020), pp. 5824–5836. Tianhe Yu et al. “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning”. In: Conference on robot learning. 2020, pp. 1094– 1100.