arXiv:2609.14117v1 [cs.DC] 12 Sep 2026
DQN-S CHEDULER : A M ULTI -O BJECTIVE O PTIMIZATION F RAMEWORK FOR S CHEDULING M ICROSERVICES IN C LOUD C OMPUTING
Abdullah Alelyani The University of Western Australia [email protected]
Amitava Data The University of Western Australia amitava.datta @uwa.edu.au
Ghulam Mubasher The University of Western Australia [email protected]
A BSTRACT Cloud computing has emerged as an information technology solution, providing software and infrastructure solutions for companies and individuals. The pay-as-you-go approach has increased demands for the cloud. The massive range of resources, the variety of services, and flexible pricing grab attention. In addition, microservices have emerged as a new way of building software, with applications developed as loosely dependent tasks. Additionally, container technology has boosted the popularity of microservices by offering a platform for this type of architecture. Containers and microservices improve the flexibility and scalability of cloud applications. There are two primary types of microservices: batch and online services, with the majority of applications falling into the online service category. Scheduling microservices is challenging because it requires careful management of resource utilization, load balancing, network latency, reliability, and availability. In this study, we introduce the DQN-Scheduler, a novel reinforcement learning-based agent designed to optimize microservice scheduling in cloud environments. Our approach aims to optimize multiple scheduling objectives simultaneously, such as resource utilization, load balancing, latency, reliability, and availability. To our knowledge, this is the first framework to address all these objectives simultaneously. The DQN-Scheduler was tested against benchmark algorithms in the field. The experimental results demonstrate that the DQN-Scheduler outperforms benchmark algorithms. Keywords Cloud Scheduling, Microservices, Cloud Computing, Reinforcement Learning, Deep Q-Networks, Multi-objective Optimization, Resource Management, Latency Reduction, Container Technology.
1
Introduction
Cloud computing benefits from the microservice architecture in developing applications. Several reasons for the cloud recently transitioning from developing applications as monolithic architecture to microservices include simpler development and efficient deployment[1]. Since each microservice is responsible for performing one task, microservices allow for more accurate and easier development of applications. Loosely coupled microservices shift the requirement of using a single programming language for the entire application to use different languages for different microservices. Two of the most important advantages of microservices are their compatibility with containers for cloud applications[1] and their ability to be distributed across cloud physical machines (PMs). Microservices utilize remote procedure calls (RPC) or a RESTful API [2] to communicate between them. Communication between microservices can affect the quality of service (QoS) of the cloud platform by decreasing throughput and increasing latency[1][3].In addition, balancing resource utilization contributes to improving energy efficiency. With microservices, balancing the load between resources becomes challenging, as some microservices are
either CPU-intensive or memory-intensive. For example, if there are n memory-intensive microservices distributed across m PMs, the memory of the PMs would be heavily used, while the CPUs would remain underutilized, leading to CPU wastage. Scheduling microservices across PMs while maintaining multiple objectives is challenging. For example, minimizing energy consumption while maximizing resource utilization can lead to SLA violations if the CPU of PMs is heavily utilized. Trade-offs between conflicting objectives are a common strategy in the scheduling process. However, these trade-offs do not guarantee optimal scheduling, especially in dynamic environments like cloud computing. Resource utilization, throughput, availability, reliability, and load balancing are among the main objectives for enhancing cloud performance. However, these objectives pose challenges in developing cloud scheduling strategies, as they often conflict. Enhancing one objective may degrade others. This motivates us to propose a framework to find the optimal solution for enhancing resource utilization, balancing the load between resources within each PM and among PMs, reducing latency, and enhancing the availability and reliability of cloud computing applications. Our main contributions are as follows: 1. Optimizing Microservices Scheduling Framework: We use a double-deep Q-Network agent model. The agent is trained to provide scheduling decisions that consider the conflicting objectives mentioned above. 2. Developing a Reward Model: This model utilizes the Pareto Front approach to determine the agent’s reward during training based on the optimal solution for conflicting objectives. It aims to guide the agent’s behaviour to improve scheduling decisions. 3. Developing an Action Selection Model: This model selects the best action provided by the agent based on real-time environmental conditions. Given the stochastic nature of the cloud environment, this selection approach employs a Bayesian inference model. 4. Developing Load Balancing and QoS Models: We designed models to enhance the scheduling strategy by monitoring resource utilization, load balancing, latency, reliability and availability of microservices. 5. Conducting Extensive Experiments: We conducted experiments that assessed the efficiency of our proposed framework, including comparisons with benchmark algorithms such as the Genetic Algorithm (GA), Particle Swarm Optimization (PSO), and Best-Fit Algorithm (BFA). The rest of our research is organized as follows: Section 2 reviews related work; Section 3 describes the proposed system models; Section 4 describes the architecture of DQN-Scheduler; Section 5 discusses the cloud environment scenario and workload; Section 6 introduces the benchmark algorithms used for comparison with our proposed framework; Section 7 details the experiment setup and results; and Section 8 provides the conclusion.
2
Current Studies
In this section, we discuss current studies that proposed to address scheduling with multi-objectives in detail. Studies included in this part of our research must meet the following criteria: 1. Scheduling microservices to cloud considering multi-objectives that are conflicted. 2. Considering real-time scheduling scenarios. 3. Considering the dynamic environment of cloud computing. 4. Examining the proposed approach using real-world datasets or implementing the approach on an actual cloud computing platform. 5. Considering the significant volume of tasks arriving on the cloud platform during the experiment. We decided to limit the included studies to those meeting the criteria above to ensure the studies best mimic the reality of cloud computing platforms. Furthermore, we aim to narrow our search to strategies that utilize reinforcement learning models, like deep Q-learning. A multi-objective approach that aims to minimize the communication overhead between microservices and balance the load (RSDQL) is proposed in [4]. RSDQL utilized a deep Q-learning algorithm to enhance the development strategy and balance the load among resources. The approach is modelled as a multi-objective problem. It primarily considers the communication overhead. Additionally, the model aims to avoid degrading the performance of scheduled task execution by considering the capacity of nodes. An undirected and weighted graph is used to demonstrate the communication overhead between microservices. Additionally, the proposed model uses the graph to minimize the interaction overhead between microservices and to reduce the variance of utilizing resources to balance the load. Furthermore, 2
an elastic scaling approach was introduced to increase the number of deployed instances hosting new microservices and manage the dynamic resource utilization of applications. Elastic scaling also aims to manage QoS by utilizing real-time resource monitoring utilities, and a resource utilization threshold. The experimental results demonstrate that RSDQL was able to reduce the average response time, balance loads, and improve scalability. However, energy consumption and the limitations of edge resources were not taken into consideration. An online microservices orchestration for managing Internet of Things (IoT) application allocation is proposed in[5]. The infrastructure of the orchestration is based on a policy-based DRL algorithm. The study aims to address application allocation with two objectives: minimizing the long-term energy consumption and the end-to-end execution time of the deployed application. The allocation problem was considered a multi-objective Markov decision process (MOMDP). As a result, a model named MOTION was proposed as a multi-objective deep reinforcement learning model. MOTION aims to optimize the dynamic orchestration of allocating IoT microservices in cloud environments. The proposed model was tested against the proposed baseline algorithms, showing the efficiency of the proposed model in reducing long-term energy consumption and execution time for applications. Memory and I/O allocation are not considered in this approach. The real-world scenario of microservices communication is not considered as well. A framework employing deep reinforcement learning (DRL) to limit incoming requests to containers in Alibaba’s data centre (Noah) is proposed in [6]. The framework is an adaptive limiter for the large-scale microservice platform running in containers. Noah comprises three main components: a limiter, a monitoring component, and a DRL-based decision-maker. It is a closed-loop system where requests first arrive at the limiter. Then, based on a threshold, requests are either forwarded to the container or rejected. The monitoring components retrieve the statuses of containers and provide feedback to the DRL-based decision-maker. The DRL model is responsible for setting up the threshold for the limiter in real-time. Four goals were introduced for the framework: 1) the limiter should automatically adapt without domain knowledge; 2) the framework must be available all the time; 3) the framework should react in time based on the load situation; and 4) the framework should minimize the number of rejected requests. The DRL is also trained with synthetic data representing extreme situations to ensure the model can handle unexpected requests. Noah shows an efficient adjustment to the threshold in real-time based on the load of the containers. Noah accounts for a limited number of scenarios. However, there is no evidence of Noah’s ability to achieve considerable performance in the dynamic environment of cloud computing. A framework for addressing the issues of multi-dependencies between microservices in edge computing (GRLD) is proposed in [7]. Microservice architecture (MSA) involves multi-dependencies that represent a communication sequence between microservices, called a call graph. The proposed framework utilizes a graph convolutional network (GCN) to extract features from the dependencies between the microservices. GRLD then passes this feature vector to a decision-making process. The decision-making process employs a DRL model that deploys microservices to the edges. The main goal of GRLD is to incorporate multiple pieces of information (e.g., multi-call-graph scenarios, resource utilization, edge node situations) and allow the DRL agent to make decisions about the deployment process. In addition, the agent tries to meet QoS parameters. The experimental results demonstrate that GRLD exhibits minimal deployment overhead costs compared to baseline algorithms. The results demonstrate the efficiency of maintaining QoS parameters. However, the model was not generalized to adapt to real-world scenarios. An artificial intelligence algorithm utilizing deep Q-learning (DQTS) is proposed in [8]. It aims to schedule tasks, considering microservice dependencies. DQTS considers microservice dependencies as directed acyclic graph (DAG). The proposed algorithm was developed to minimize makespan and load balancing. The scheduler consists of three components: 1) the resource dynamic allocator, which adjusts and refreshes the available resources; 2) the task allocation controller, which allocates the different types of tasks; and 3) the deep Q-learning algorithm. The deep Q-learning combines a deep neural network and a Q-learning algorithm to provide the scheduling decisions. The experimental results show that DQTS improves cloud performance. Additionally, DQTS achieves the minimum makespan and the best load balance. However, DQTS was evaluated against a scientific workflow benchmark, which does not include real-world scenarios and does not account for crucial objectives like energy efficiency. A cloud scheduling framework leveraging queuing theory and reinforcement learning to optimize energy consumption and resource utilization (QEEC) is proposed in [9]. The QEEC consists of two phases: a centralized task dispatcher and a scheduler. The dispatcher monitors the user requests by utilizing a global request queue on the cloud side. The queue serves as a buffer for incoming requests. The scheduler stores incoming requests in local request queues on each node. It then reorders requests based on QoS and distributes them among nodes to minimize overall CPU utilization and SLA violations. The Q-learning-based scheduler treats each node separately during the scheduling process. QEEC minimizes task execution time by considering task lifetime. Extensive experiments utilizing the M/M/S queuing system were conducted to evaluate QEEC. The experimental results show that QEEC efficiently reduces energy consumption and minimizes average response time. However, the framework considered a limited number of nodes that do not reflect the cloud environment’s reality.
3
An algorithm for enhancing the makespan in cloud computing utilizing a Q-learning algorithm (QL-HEFT) is proposed in [10]. QL-HEFT aims to enhance the overall performance of scheduling dependent tasks into cloud resources. The algorithm consists of a data centre broker and a task scheduler. The broker distributes the tasks to the cloud resources, considering QoS parameters. QL-HEFT presents dependent tasks as a DAG graph. Additionally, the HEFT algorithm is employed in this research to reduce the makespan. Two phases that construct the HEFT algorithm include task prioritizing and processor selection phases. In the task prioritizing phase, tasks are ranked based on their rank value. In the second phase, tasks are scheduled into the processor, considering the earliest completion time strategy. Furthermore, a Q-learning algorithm is employed to make decisions during scheduling tasks. The main goal is to reduce the makespan and response time. An experiment was conducted to evaluate QL-HEFT and compare its performance with benchmark algorithms such as HEFT D, HEFT U and CPOP. The experimental results demonstrate that QL-HEFT outperforms the algorithms selected in the performance comparison. However, QL-HEFT has limitations in solving large-scale tasks because the Q-table becomes significantly large and expensive to update. A method proposed to address two objectives: makespan and resource utilization (BCRN) is proposed in[11]. The authors claimed that employing standard learning methods to solve scheduling problems requires numerous transitions to converge to the optimal solution. Therefore, BCRN aims to tackle these issues utilizing a Pareto-based algorithm and a learning-based population. In addition, BCRN aims to address the issues of scheduling dependent tasks. BCRN comprises two stages: the learning and the selection stages. In the learning stage, two Q-learning agents are trained to reduce makespan and enhance resource utilization. In the selection stage, NSGA-3 is utilized to identify the best solutions that meet the research objectives. The experimental results demonstrate that BCRN effectively reduces makespan and enhances resource utilization of the cloud. However, BCRN was evaluated using a scientific workflow benchmark, which does not include real-world scenarios. In addition, more agents are needed to enhance the learning process. An approach utilizing a deep Q-learning model with multiple DVFS (dynamic voltage and frequency scaling) algorithms (DDQ-EES) is proposed in [12]. DDQ-EES aims to address energy consumption in data centres and selects the best DVFS algorithm to execute tasks, considering their deadlines. It employs a double-deep Q-learning model consisting of two deep Q-network models. The first model computes Q-values for each DVFS, while the second model tunes the parameters of the generated Q-network. DDQ-EES utilizes a double-deep Q-learning model to maintain model stability. The model adopts the rectified linear units (ReLU) function instead of the sigmoid function to prevent gradients from vanishing. The experimental results demonstrate that DDQ-EES reduces average energy consumption. Furthermore, the results indicate that QQL-EES achieves higher training efficiency. However, DDQ-EES did not employ explore-exploit strategies during the training stage. Additionally, it was not considered for real-world scenarios. An online resource scheduling framework that utilizes a deep Q-learning algorithm is proposed in [13]. The framework aims to address the challenge of decreasing energy consumption and maintaining the QoS. The framework aims to solve the conflict objectives using the Q-learning algorithm. The framework is composed of three layers, including the workload layer, scheduling control layer, and data centre layer. In the workload layer, the users’ requests are classified. Then the framework assigns the tasks to a waiting list to be executed, considering the dependencies between tasks. In the scheduling control layer, the tasks are scheduled into the cloud resources. This layer consists of a task monitor, resource monitor, scheduling policy model, energy consumption model, SLA, and scheduling process. The scheduling policy model is a DQN agent that provides optimal scheduling decisions. The agent consists of two deep neural networks that learn simultaneously to enhance the learning process. The experimental results demonstrate that the framework provides trade-off between energy consumption and task makespan by adjusting the weight of rewards. However, the framework is limited when dealing with large-scale tasks. It would also be costly and complex when handling constraints between various resources in the cloud system. An approach that aims to minimize execution time and energy consumption (E-AEO-AOA) is proposed in [14]. The approach utilizes a combination of optimization algorithms, including artificial ecosystem-based optimization (AEO) and arithmetic optimization algorithm (AOA). The E-AEO-AOA aims to address the issue of offloading tasks from mobile devices to fog or the cloud, considering task execution time and energy consumption. The E-AEO-AOA queues tasks that may be offloaded. Then E-AEO-AOA decides which tasks will be offloaded. AEO and AOA search for the optimum solution for offloading tasks. Additionally, AEO and AOA are discretized utilizing the round function. A Q-learning model is utilized to hybridize the optimization algorithms (AEO and AOA). The experimental results demonstrate that E-AEO-AOA outperforms the competitor algorithms in most cases. However, the approach demonstrates significant complexity and requires more memory and time. A weighted double deep Q-learning model based on a reinforcement learning algorithm (WDDQN-RL) is proposed in [15]. The WDDQN-RL aims to minimize makespan and costs in cloud platforms. Two levels of scheduling strategy are introduced in the study. Each level of the strategy consists of an agent. The agent at the first level enhances
4
the task scheduling order, considering task dependencies. The agent at the second level optimizes resource allocation. Furthermore, time and cost are optimized by the first and second levels of scheduling, respectively. A dynamic sensing mechanism (DSM) adjusts the approach’s attention to one of the optimization objectives. The experimental results demonstrate the superiority of WDDQN-RL in various aspects, including solution quality, diversity, and running time. However, WDDQN-RL does not account for real-world scenarios and was not evaluated against different resource constraints. Current studies neglect the stochastic nature and real-time changes in the cloud environment. They also overlook the fluctuating resource utilization during scheduling, a key characteristic of the cloud. Additionally, the geographical distribution of data centers (zones) is ignored, resulting in overlooked latency of communications between application tasks. Considering these zones during scheduling is crucial for improving reliability and availability and for simulating real-world scenarios. Moreover, current studies do not address multiple conflicting objectives, ignoring the complex nature of the cloud environment. Table 1 summarizes the key features of current studies, goals, multiple conflicting objectives, methods, experimental results, and limitations.
3
System Model
In this section, the system models are introduced. They encompass load balancing, quality of service (QoS), latency, system architecture, and multi-objective models. We aim to provide a deep understanding of scheduling microservices processes and the issues of scheduling microservices to the cloud. additionally, we will illustrate the methodology employed and discuss the metrics utilized to measure the performance of our framework in this section. 3.1
Cloud Platform Model
Cloud platforms are distributed worldwide. With the Internet of Things (IoT) coming into the picture, having data zones close to clients becomes ideal. A data center is denoted as D. D consists of a set of data zones, represented as D={z1 , z2 ...zz }. These zones are interconnected with a high-speed network to mitigate latency. The zones comprise physical machines (PMs) with heterogeneous resources such as CPUs, memory capacities, and network bandwidth in clusters. Additionally, each cluster consists of a limited number of resources. Clusters are managed by cluster orchestration such as Kubernetes and Docker daemon. In our work, we focus on Kubernetes technology for cluster orchestration. Cluster architecture managed by Kubernetes comprises three main components: node, pod, and container. A node can be a virtual machine (VM) with fixed resources (e.g., CPU, memory, bandwidth) or a physical machine like a bare metal machine. The node provides an isolated environment for running applications. A cluster consists of several nodes denoted as C={n1 , n2 ....nc }. A set of pods, denoted as P = p1 , p2 , ...pp , is allocated to a node, where P ∈ C. A pod is the smallest unit of the orchestration architecture, serving as a computational unit. Each pod consists of a group of containers, denoted as R ∈ P , and R={r1 , r2.....rr } where containers handle the tasks of running applications. 3.2
Microservies Model
Two architectures shape the structure of applications: monolithic and microservices. The monolithic architecture builds the application as a single unit. However, microservices promote the concept of loosely coupled tasks that construct the application. With the rise of IoT applications, microservices emerge as the preferred choice for developing the applications. Furthermore, edge applications demand high computational resources utilizing microservices architecture. Edge tasks (only CPU-intensive) can be offloaded to cloud resources, while the remaining application tasks can be processed on the edge device. In such scenarios, microservices offer advantages by reducing energy consumption, improving communication during offloading, and becoming cost-efficient. Microservices are hosted by n containers and distributed across data centers. Therefore, communication between dependent tasks is crucial. Loosely coupled communication tasks are presented as a graph where each edge represents a microservice and vertices represent communication between microservices, denoted as G = {E, V }. Microservices provide a sophisticated platform to deploy applications, such as IoT applications, with a high level of reliability, scalability, and availability. Quality of service (QoS) parameters such as service level agreement (SLA), and replica of the servers are mostly specified by clients during the submission of the requests to deploy the application. We consider the number of replicas of each microservice to be specified during scheduling. Thus, each microservice M can be m composed of several replicas, denoted as Mreplica = Ep where Ep is a constant representing the number of replicas. 5
Approach and Key Features RSDQL[4] Utilizes deep Q-learning-based multi-objective approach
Goals
MCO Methodologies
Experimental Results
Limitations
Minimize communication overhead and balance load among microservices
X
Reduced average response time, balanced loads, improved scalability
Did not accounting for energy consumption and resource limitations in edge computing.
[5] is a policy DRL-based orchestration
Minimize energy consumption and end-to-end execution time for IoT applications Limits incoming requests to containers in data centers
X
RSDQL agents with deep Q-learning algorithm, undirected and weighted graph to represent microservice communication Multi-objective deep reinforcement learning, multi-objective Markov decision process
Reduced long-term energy consumption, improved end-to-end execution time
X
Limiter, monitoring component, and DRLbased decision-maker
Minimized number of rejected requests, automatic adaptation without domain knowledge
Did not consider memory and I/O allocation for microservices, nor does it account for real-world scenarios in terms of the interconnections between microservices. It adapts to a specific scenario and does not consider the dynamic environment.
Address multidependencies between microservices in edge computing to maintain QoS Minimizes makespan and maintains load balancing
X
GCN to extract features from call-graph, DRL model for decisionmaking
Minimal deployment overhead cost, efficient QoS maintenance
The model was not generalized to adapt to real-world scenarios.
X
DAG-based task dependencies, deep Q-learning with deep neural network
Outperformed other algorithms in terms of makespan and load balancing
Optimize energy consumption and resource utilization
X
Reduced energy consumption and minimized average response time
Reduce makespan and optimize overall scheduling performance
X
Queuing theory with Q-learning-based scheduler, centralized task dispatcher, scheduler on each node Q-learning with HEFT algorithm, datacenter broker, task scheduler
Tested against a scientific workflow benchmark, which does not include real-world scenarios and does not account for crucial objectives like energy efficiency Did not evaluated agonist large number of nodes that reflecting the reality of the cloud environment.
Outperformed other benchmark algorithms in performance and makespan reduction
Not adaptable to large-scale tasks and would be time-consuming for updating
Address makespan and resource utilization in cloud environments
X
Pareto-based algorithm, two Q-learning agents, NSGA-3 for solution selection
Effective makespan reduction and enhanced resource utilization
Tested against a scientific workflow benchmarks, which does not include real-world scenarios, more agents are needed to enhance the learning process
Reduce energy consumption using multiple DVFS algorithms
X
Double-deep Qlearning model, rectified linear units (ReLU) for stability
Reduced energy consumption and improved training efficiency
Did not employ explore-exploit strategies during the training stage. Additionally, it was not considered for real-world scenarios
Optimize energy consumption and task makespan
X
Multi-layered architecture, DQN agent for optimal scheduling decisions
Trade-off between energy consumption and makespan based on reward weighting
Minimize execution time and energy consumption
X
Outperformed competitor algorithms in most cases
Minimize makespan costs
X
Artificial Ecosystembased Optimization (AEO) and Arithmetic Optimization Algorithm (AOA), Q-learning for hybridization Weighted double deep Q-learning, Dynamic Sensing Mechanism (DSM) for attention adjustment
It is limited when dealing with large-scale tasks, and it is costly and complex when handling constraints between various resources in the cloud system It is complex and expensive due to the need for more time and memory.
Noah [6] proposes a deep reinforcement learning-based adaptive limiter for large-scale microservices GRLD[7] is a Graph Convolutional Network (GCN)-based approach DQTS[8] utilizes a deep Q-learning-based scheduling for cloud resources QEEC[9] proposes Q-learningbased cloud scheduling framework QL-HEFT[10] utilizes Qlearning-based algorithm for cloud computing scheduling BCRN[11] is a multi-agent bi-objective scheduling using Q-learning and NSGA-3 DDQ-EES[12] is a Deep Qlearning-based approach for energy efficiency in data centers DQN Framework [13] is a Deep Q-learning-based cloud scheduling framework E-AEO-AOA[14] is a combination optimization algorithm for cloud resource management WDDQN-RL[15] is a reinforcement learning-based cloud platform optimization
and
Superior solution quality, diversity, and running time
Did not account for real-world scenarios and was not evaluated against different resource constraints
Table 1: Comparison of Related Work Approaches for Scheduling, Resource Management, and Load Balancing in Cloud Environments. Where MCO stands for multiple conflicting objectives.
6
3.3
Multi-objective Model
This subsection provides a brief overview of multi-objective model management. We use Pareto front optimization, which is detailed as follows: Multi-objective optimization is a key focus of our research to enhance the reliability of cloud computing systems. Effective resource utilization management is essential to minimize resource waste and optimize system performance. Network traffic costs also impact the performance of the cloud. Hence, our research targets reducing network traffic by minimizing communication between dependent microservices across different geographical zones. Achieving high reliability requires a sophisticated scheduling approach to manage service availability and mitigate the risk of a single point of failure. A single point of failure risk arises when all service replicas from one application are housed in a single PM. Such circumstances produce a high probability of being unavailable. Additionally, the intensive node resource utilization directly impacts reliability. However, these objectives are conflicted, making it challenging to improve them simultaneously. To address this challenge, we employ Pareto Front optimization in our research. Pareto Front optimization balances multiple conflicting objectives. Multi-objective optimization problems (MOPs) involve optimizing two or more conflicting objectives simultaneously. The optimal solution to such problems often involves a trade-off between the objectives [16]. However, we aim to find the optimal point of the trade-off between these objectives. For example, we aim to reduce network traffic by consolidating microservices geographically. However, this consolidation may affect system load balancing and reliability. To address these challenges and enhance microservice scheduling performance, we propose a multi-objective optimization approach with Od objectives utilizing Pareto front (PF) optimization. PF enables us to explore the solution space and find optimal solutions for all objectives. In our case, the solution space is defined by Od − 1[16]. The MOPs can be formulated as[17]:
min x
f (x) = [f1 (x), f2 (x), . . . , ff (x)] s.t. x ∈ S
(1)
Where x is the vector of the decision space that proposes a solution of minimizing f and f ∈ Rm . The notation Table 2 illustrates all the notations in our research. Our research employs the Non-dominated Sorting Genetic Algorithm III (NSGA-III)[18] to address conflicted multiobjectives and propose solutions to minimise f . NSGA-III falls under the evolutionary multi-objective optimization (EMO) [18]. The algorithm can find optimal solutions for up to 15 objectives, which is more than sufficient for our purpose. 3.4
Load Balancing Model
Early in this section, we model the data centre as a set of regions, where each region is composed of clusters. Each cluster is composed of PMs. Therefore, we assume the node to be either a bare metal or a VM with a fixed amount of resources and bandwidth. Meanwhile, we assume that the microservices are deployed to allocated containers that elastically consume the resources based on the requirements. In addition, a container is a placeholder; for example, C1 hosts microservice M1 therefore, we consider the container as the microservice M1 . The node can host a finite number of containers at the time t, thus, the total resource utilization of the node at the time t is calculated as follows: total CPU util(t) = total Memory util(t) =
m X 1 m X
CPU util(Mm , t) (2) Mem util(Mm , t)
1
It is well-established that microservices exhibit high CPU utilization with relatively lower memory utilization. This affects load balancing across resources at the node level. Therefore, a threshold for resource utilization is introduced 7
to mitigate system performance issues resulting from the overloaded node. Thus, equation 2 becomes: total CPU util(t) = total Memory util(t) =
m X 1 m X
CPU util(Mm , t) Mem util(Mm , t)
1
(3)
where total CPU util(t) ≤ Th and total Memory util(t) ≤ Th Where T h is the maximum utilization threshold. On the other hand, balancing the load at the cluster level requires distributing the load evenly across all the nodes in the cluster. Therefore, the mean µ and standard deviation (SD) are utilized to measure the load balancing among nodes in the cluster as follows: v u u SD = t
1
NX total
Ntotal
p=1
mean =
(PMp − mean)
1
NX total
Ntotal
p=1
2
PMp
(4)
(5)
where Ntotal is the total number of PM in the cluster. 3.5
QoS Model
Quality of Service (QoS) parameters are crucial in cloud computing, including reliability, availability, and SLA violations. Cloud reliability is achieved by minimizing the risk of microservice failure. We, however, aim to reduce the risk of failure by scheduling microservices of one application across different PMs to prevent a single point of failure. The availability of a microservice relies on its accessibility over the internet at any given time. Microservices may be offline during peak times when traffic is heavy, which is unacceptable. Additionally, network congestion leads to the blocking of services and degradation in availability. In our research, we aim to reduce network traffic costs to enhance microservice availability, as follows: min Network traffic Cost =
X
NT
(6)
where N T is the network traffic in the cluster as communication between dependent microservices. SLA violations occur when the CPU of the host machine is utilized intensively. According to [19], SLA violations could occur in two circumstances: during the migration of virtual machines (VMs), which in our study is the node (PM), or when the CPU of the host is utilized by 100%. Migrating VMs degrades performance; hence, the Power Difference Model (PDM) proposed by [19] is utilized in the case of VM migration. The following equation represents the PDM:
P DM =
M 1 X Cdj Mvm j=1 Crj
(7)
where Mvm is the number of VMs, and Cdj represents the estimation of system degradation caused by migrating VMj . Crj is the required amount of CPU by VMj during its lifetime. In addition, we calculate the SLA violation utilizing the SLA violation time per active host (SLATAH) equation proposed by[19]: N 1 X Tui SLAT AH = (8) Npm i=1 Tai 8
where Npm is the total number of PMs in the cluster, Tui is the total time that P Mi experiences 100% CPU utilization, and Tai is the total number of active hosts. 3.6
Reliability Model
Reliability refers to ensuring services are delivered and are available. In more precise terms, reliability measures the probability that scheduled services are available. According to [20], several circumstances can lead to service unavailability or failure, including 1) hardware failure, 2) operational failure, 3) network failure, and 4) software failure. As these failures are negligible and irrelevant to the scheduling process, our research does not consider them. Cloud applications may be unavailable due to network conditions, preventing some clients from reaching the application. Scaling microservices by replicating them across different PMs becomes crucial to enhancing their availability. However, this poses challenges, as it can increase network traffic between microservices and complicate the architecture. The goal of our research is to improve the availability of microservices by scheduling replicas in the most suitable PMs, thus enhancing availability and reducing network traffic. Microservices availability and the reduction of network traffic are conflicting objectives. Optimizing those two objectives is done by scheduling dependent microservices to the same PM. However, it increases the risk of a single point of failure. Conversely, scheduling dependent microservices across multiple PMs reduces the risk of failure but increases network traffic. To evaluate the reliability of scheduling, entropy theory can be applied to measure the variance of microservices across PMs. By measuring entropy, we can assess the diversity and distribution of microservices on each PM, as well as clustering them, by using the following equation[21]:
H=−
n X
pi log(pi )
(9)
i=1
Where pi is the probability of microservices scheduled into PM. To assess the quality of the scheduling, the SD and mean of entropy for the cluster are utilized as follows: n 1 X H̄ = Hj Npm j=1
v u m u 1 X σH = t (Hj − H̄)2 Npm j=1
(10)
Where Npm is the total number of PMs and Hj is the entropy of P Mj . 3.7
Latency Model
Latency is a crucial aspect of QoS parameters promised by cloud providers. As microservices in certain applications scale due to client demand, latency becomes an issue. For instance, during peak traffic hitting application A (e.g., an IoT application), the obvious solution is to create more replicas of the application tasks. The replicas in our case are microservice replicas. Each replica is considered a task that communicates with others. This leads to increased network traffic that may cause latency. Therefore, to minimize latency, we replicate microservices close to clients and close to each other. As reported by Alibaba data center, the latency L between two different data centers located in Silicon Valley, USA, and Hangzhou, China, is measured to be 175 ms [22]. We aim to utilize this metric to measure latency in our study as follows: l X
Dpm,pm+1 ∗ lv 1 (1 if there are dependencies between pm,pm+1 Latency(t) =
Dpm,pm+1 = 0 otherwise
9
(11)
This equation calculates the latency at time t by determining the dependencies between PMs pm, pm + 1 that host two dependent microservices. It then multiplies that by the latency value lv considered to be 175 ms. 3.8
Double Deep Q-network Model
Reinforcement learning (RL) is one of the machine learning models that learn from interacting with the environment. RL finds the optimal policy by maximizing the expected cumulative reward over time [23]. The optimal policy is represented as a Q-function, which maps the state-action pair to the valued Q as follow[23]s: Q: S × A → R
(12)
Where S, and A are state and action values, respectively. The Q-function is estimated using various algorithms[23]. The Q-value represents the action value of a specific state st taking action at at time t. The Bellman equation is employed to estimate the Q-value in the following equation[23]: Q(st , at ) = (1 − α) · Q(st , at ) + α · Rt+1 + γ · max Q(st+1 , at+1 ) at+1
(13)
Where (1 − α) · Q(st , at ) is the current Q-value. αRt+1 is the reward obtained from current action at at state st , and γ is the discount rate associated with reward to give a more important recent reward. maxat+1 Q(st+1 , at+1 ) is the maximum reward that can be obtained in state st+1 The Q-function can handle a small number of states [23][24]. Therefore, the deep Q-networks are utilized in our research. A neural network can approximate the action-value function as follows[23]: Q∗ (s, a) = Est+1 ∼E [r + γ max Q∗ (st+1 , at+1 )] a
(14)
The intuition behind this equation is as follows: if the optimal value of the next step Q∗ (st+1 , at+1 ) is known, then the optimal next action at is selected to maximize the reward. Non-linear function approximators, like neural networks, are utilized in our research. A q-network with weights θ is used to approximate the action-value function. It maps the state of the approximate function using wight θ as proposed in the equation (14) [25]. In addition, the loss function proposed in [25] is utilized in our research as follows: h ∇θi Li (θi ) = Es,a∼ρ(·);s∼E r + γ max Q(s, a; θi−1 ) a
−Q(s, a; θi )∇θi Q(s, a; θi )]
(15)
In each iteration, the θ is updated utilizing the Bellman equation into the target Q-value. Our agent is model-free, adapting to the dynamic environment. It learns directly from interactions by exploring different actions and observing resulting states to maximize rewards. The agent is off-policy, employing a ϵ-greedy policy to select actions a that maximize the Q-function. We use an ϵ-greedy strategy to balance between exploitation and exploration. Where ϵ selects the optimal actions, and 1 − ϵ selects random actions. Figure 1a illustrates the agent components while training.
4
Framework Architecture
In this section, we describe the components of DQN-Scheduler. The components handle scheduling microservices into cloud computing, aiming to enhance the total performance of the cloud by enhancing resource utilization, reducing network traffic, mitigating the risk of violating SLAs, and reducing latency. DQN-Scheduler consists of five components, as illustrated in Figure 1b. 4.1
The Agent
Here, we briefly outline the agent in two phases: implementation and training. In the first phase, the role of the agent in scheduling is described. In the second phase, we explain the training process of the model. 10
aSpace
SLA monitoring
St …….………...
ε-greedy
at
Cloud Environment
Calculating Utilization
Microservices Requesting Placement
Q-Network Copy parameters MSE
Action space Environment state Scheduling Microservice
……..
aSpace
Experience memory replay
Cluster management component
The Agent
Network Traffic monitoring
…….………...
Minibatch
Cloud Platform
Resource Utilization Monitoring
Q-Target
The Agent (a) The Agent Architecture
(b) The DQN-Scheduler Architecture
Figure 1: The figure shows the agent training steps alongside our proposed scheduling framework.
4.1.1
The Implementation Phase
The agent is used to provide the DQN-Scheduler with scheduling decisions. The scheduling decision space is represented as a vector of the probability of each PM in the cluster hosting incoming microservices. 4.1.2
The Training Phase
The architecture of the agent consists of state, action, environment, and reward as shown in Figure 1a. The state: is an input consisting of the resource requirements of incoming jobs (microservices). Typically, clients submit requests to schedule their jobs with requirements. The requirements vary depending on each client’s application. Therefore, instead of solely using the client’s resource requirements, we consider the maximum, minimum, and average resource requirements during model training. Thus, the state is denoted as h U U U S = [max(sCP ), avg(sCP ), min(sCP )], j j j i (16) em em em [max(sM ), avg(sM ), min(sM )] j j j where j denotes the job coming to the cloud to be scheduled, and max, avg, and min represent the maximum, average, and minimum resource requirements for CPU and memory, respectively. Additionally, the state includes the number of repliacs specified by the client, denoted as nreplica . The agent comprises two neural networks: Q-network and Q-target. Both networks share the same architecture, consisting of two convolutional layers (CNN) followed by two fully connected layers (linear layers), and a classifier layer. The ReLU() activation function is employed in both networks to leverage non-linearity in learning the optimum Q-function. The ReLU activation function is specifically utilized to mitigate the issue of vanishing gradients. Additionally, Softmax() is utilized to convert logits to action probabilities. The loss, computed as shown in equation (15), leads to unstable weight updates in the Q-network. Therefore, periodic weight copying between the Q-network and Q-target networks is performed as shown in Figure 1a. Furthermore, experience memory replay is employed to store recent state-action and next-state events for training the Q-target network. The action: is a component that determines microservice scheduling into PM, with the agent output representing the action space. This action space provides an action vector, denoted as a = {a1 , a2 ...aa }. Each action in the vector represents a probability that a particular PM will host the microservice. Softmax function is utilized to select the maximum action probability that maximizes the reward. In addition, ϵ-greedy exploration is employed to explore other potential actions that may maximize reward using random action selection. Reward is utilized to encourage the agent to behave according to the aims of our research, which are to enhance resource utilization, and latency, balance the load, and manage QoS parameters. These objectives are conflicting. Therefore, NSGA-III is utilized to find the optimum solution among those objectives using equation (9). Calculation of the reward is based on the distance between the optimal solution of all objectives given by NSGA-III and the current state of the environment, as follows: 11
v u i uX ED = t (Pareto Fronti − objectivev )2
(17)
i=1
ED represents the Euclidean distance between the current state and each point in the Pareto front. The objectivev are the differences between the current environment and the environment after scheduling the microservice, such as ∆ CPU utilization, ∆ memory utilization, and ∆ network traffic. Some of these differences may be negative, such as network traffic. Therefore, calculating ED becomes: v " # u v ∆CPU uti uX ∆Mem uti ED = t (Pareto Fronti − )2 ∆network traffic i=1 4.2
(18)
SLA Monitoring
This component of the DQN-Scheduler is responsible for reporting any service violations occurring during scheduling, like rejecting scheduled microservices. It also reports any violation that occurs after scheduling, like being unable to reach the service due to network congestion. During the time of training the model, SLA monitoring, resource monitoring, network traffic monitoring, cloud cluster management, and resource monitoring were parts of the environment. They also contribute to reward calculation. For example, if the model violates the SLA or schedules a microservice to a PM exceeding the threshold, the agent is rewarded with negative values. SLA monitoring components aim to keep an eye on resource utilization and enforce the threshold restrictions all the time by applying equations (2) and (3). The aim is also to report any service violation during the migration process or if the CPU is being utilized intensively, using (7) and (8). 4.3
Resource Utilization Monitoring
This component of the DQN-Scheduler measures resource utilization at timestamp sp, where the total proportional resource utilization is calculated as follows: total CP U util(sp) , and Ccpu total memory util(sp) Rmem = Cmem
Rcpu =
(19)
Here, total CP U util(sp) represents the total CPU utilization at timestamp sp provided by equation (4). Ccpu denotes the total CPU capacity in a PM. Similarly, RM em calculates the memory utilization at timestamp sp for the PM. This component is responsible for reporting the resource utilization of the PMs at timestamp spp . Additionally, the component calculates the expected resource utilization when an incoming microservice is scheduled in real-time for both resources (CPU and memory), denoted as Expncpu and Expnmem , respectively, as follows: n n + Req Rcpu Cur Rcpu , and n Ccpu n n Cur Rmem + Req Rmem Expnmem = n Cmem
Expncpu =
(20)
Where Cur R and Req R represent the current utilization of the resource and the microservice request of the resource, respectively. 4.4
Network Traffic Monitoring
Three types of network traffic are considered in microservice architecture: traffic between microservices located in the same zone, network traffic originating from clients to access distributed services, and traffic between microservices in 12
different zones. The first two types of network traffic result in negligible latency due to the high speed of the network (e.g., 5G). In addition, they are managed by cluster orchestration (in our case, Kubernetes). However, network traffic monitoring calculates the latency caused by the third type using Equation (11). This component reports network traffic latency to the cluster management component during microservice scheduling, including the increases or decreases in network latency. If the cluster management component identifies that latency would lead to SLA violations or performance degradation, microservice scheduling is reassessed. During the training of the agent, we utilized this component by associating a negative value with the reward each time latency increased. This step aims to encourage our agent to reduce latency. Additionally, the component notifies the DQN-Scheduler of the changes in latency each time a microservice is scheduled to maintain QoS and prevent network latency or congestion Figure1b. 4.5
Cluster Management Component
The component coordinates between the agent and cloud resources (Figure 1b) and fulfils several key roles: 1. Receives resource allocation orders for deployed containers from the agent. 2. Collects the current cluster status, including resource utilization, latency, and load balancing. 3. Assesses the agent’s decisions regarding microservice deployment locations. 4. Deploys and runs microservices within deployed containers. The agent provides decisions of scheduling microservices as a decision space instead of a single decision. For instance, instead of the agent designating a single PM qualified to host the microservice, probabilities for all PMs hosting a microservice are provided. After this component gathers the necessary data from both the agent and environment, the scheduling process commences. If the best decision to schedule microservices violates the SLA or degrades system performance, decisions from the action space are updated using the Bayesian inference method. A posterior probabilities method is used to update the probabilities of hosting microservices by PM based on new evidence. In our case, the evidence is the probability of QoS parameters and it is calculated as the following equation as it is presented in [26]: 1
y= 1+e
x − log ( 1−x )+log (P (R|n))
(21)
Where log (P (R|n)) represents the likelihoods. The likelihood is either 0 or 1, determined based on the evidence, 1 threshold, and microservice request for resources. Additionally, represents the prior probabilities cal− log ( x ) 1−x 1+e culated using the logistic function. The Bayesian inference model does not make decisions but reorders them based on the current environmental status. This reordering involves updating the probability of hosting microservices for all PMs. The aim is to reduce SLA violations and performance degradation. Therefore, this component is utilized during agent training and microservice scheduling. 4.6
The Algorithm Pseudocode
In this section, the algorithm for scheduling microservices into the cloud is explained. The algorithm describes the collaboration among different components of DQN-Scheduler and illustrates the sequential flow of operations. Algorithm 1 states that N, m are taken as inputs for the cloud environment. N, m includes cloud cluster PMs and resource requirements by microservices, respectively. The expected output of the DQN-Scheduler is a scheduled microservice allocated to a resource, identified as an instance denoted by Inst. From lines 5 to 11, the algorithm collects the status (evidence) of the current resource utilization of PMs in the cluster. Lines 13 to 16 illustrate the process of utilizing the Q-network agent for decision-making. At this stage, our agent does not require the environment status. Therefore, it predicts scheduling decisions based on the provided state. The scheduling decision is formed as a vector of probabilities, indicating the probabilities of each PM hosting the microservice m. We aim to mimic the stochastic nature of cloud computing environments. In addition, we expect an instability in the rate of scheduling microservices in the cloud. Thus, lines 18 and 19 demonstrate the use of Bayesian inference to update the decision space if necessary. 13
Notation D C P R M Mreplica Od z1 , z2 , . . . , zz n1 , n2 , . . . , nn p1 , p2 , . . . , pp r1 , r2 , . . . , rr S x PM VM Th SD µ H L Q(s, a) α, γ Q∗ (s, a) Rcpu Rmem Expncpu Expnmem Cur Rcpu Req Rcpu L H Q(s, a) y P (R|m)
Table 2: Table of Notation Description Data center consisting of a set of data zones. Cloud cluster consisting of a set of nodes. Set of pods allocated to a node. Set of containers in a pod. Microservice. The number of replicas of a microservice. Number of objectives in multi-objective optimization. Data zones within a data center. Nodes within a cloud cluster. Pods within a node. Containers within a pod. Feasible solution space. Decision variable in the optimization problem. Physical machines. Virtual machines. Maximum resource utilization threshold. Standard deviation. The Mean. Entropy, used to measure the diversity of microservices across physical machines. Latency between data centers or nodes. Q-function, representing the state-action value in reinforcement learning. Learning rate and discount rate in reinforcement learning. Optimal Q-value in reinforcement learning. Proportional CPU utilization. Proportional memory utilization. Expected CPU utilization in P Mn . Expected memory utilization in P Mn . Current CPU utilization. Requested CPU utilization by a microservice. Network latency between zones. Entropy, measuring diversity across physical machines. Q-function, representing state-action value in reinforcement learning. Logistic function used in Bayesian inference for scheduling. Likelihoods for hosting microservices, based on resource utilization.
14
Algorithm 1 Scheduling Framework Using Q-network Agent() 1: Input: Node List N , client microservices m(reccpu , recmem ) 2: Output: R ← Inst 3: Output: Inst ← m 4: for each m ← microservices do 5: procedure COLLECTING E VEDINCE(N, m) 6: Set reccpu ← Rcpu 7: Set recmem ← Rmem 8: Set l ← L 9: Set LB ← SD 10: Set SoQ ← SLAT AH() 11: return s ← [R, l, LB, SoQ] 12: end procedure 13: procedure P REDICT D ECISION S PACE(m) 14: Initialize Q-network agent Q 15: Set D ← Q(m) 16: return D 17: end procedure 18: procedure CLUSTER M ANAGMRENT(D, s) 19: Set D′ ← posterior probabilities() 20: end procedure 21: return D′ 22: Set D′ ← argmax(D′ , nreplica ) 23: procedure S CHEDULING(D′, reccpu , recmem ) 24: Setup scheduling configuration 25: R ← allocatResource(Inst) 26: Inst ← scheduling(m) 27: Set Scheduling f lag ← T rue 28: return Scheduling f lag 29: end procedure 30: if Scheduling f lag then 31: break 32: end if 33: end for
▷ allocating R ▷ Scheduling
▷ latency ▷ load balancing ▷ SLA violation
The remaining lines of the algorithm (lines 21 to 33) select the optimal PMs from the decision space to host the microservice. The last step is to deploy the microservice to the selected PM. Using Bayesian inference allows the DQN scheduler to adapt to rapid changes in the cloud environment, enhancing the reliability and efficiency of microservice scheduling. Updating the decision space continuously allows the DQNScheduler to optimize microservice deployment in real-time.
5
Scenario and Cloud Configurations
The characteristics of the cloud environment and the tasks are briefly discussed in this section. The aim is to provide an overview of the assumptions regarding the scenario and cloud configurations. 5.1
Workload Characterization
Batch tasks: These are short-running applications deployed directly to the PM. Each application is usually divided into several tasks that are deployed separately on the instance, providing enough resources. They represent just 10% of applications submitted to the cloud [27][28][29]. Online services: These are applications that run for an extended period and represent the majority of deployment to the cloud[27][28][29]. They are considered production applications consisting of user-facing services such as search engines, e-commerce, and online shopping applications. They can be scaled across cloud PMs to provide sufficient access to the service, maintaining availability. 15
The two types of applications are typically scheduled through a centralized approach in well-known data centres, such as Brog [30][31] and Quasar[32]. Alibaba data centre utilizes Fuxi for scheduling batch jobs [33], whereas Sigma is used for long-running applications [34]. We consider the two types of workload, both of which exhibit dependency. For instance, the batch workload is divided into tasks that rely on each other and are scheduled within microservices. Similarly, online services consist of multiple services communicating with each other. Therefore, both types are categorized in our study as dependent microservices. 5.2
Stochastic Cloud Environment
In our research, a stochastic cloud environment is considered to mimic the reality of scheduling microservices. Several factors are taken into account to simulate the scheduling scenario. Timing: We assume that the number of scheduled tasks varies over time. Applications with multiple tasks arrive in the scheduling queue at different timestamps. Tasks are then scheduled individually while maintaining dependencies between them. A stochastic environment is simulated, with various job types continuously and periodically submitted to the cloud across multiple clusters. Task Priorities: Online service tasks take priority over batch tasks. Tasks failing to execute or requiring rescheduling are placed at the end of the queue without priority escalation. Scaling Applications: In microservices architectures, scaling applications is common and can be achieved through two methods: 1. Submitting scheduling requests with specified task replicas. 2. Replicating tasks of running applications to manage outbound traffic. Resource Utilization: We assume that different types of microservices utilize heterogeneous resources in real-time. Cloud Zones Network Traffic: A cloud data center consists of graphical zones linked by high-speed networks. PMs within the same zone experience manageable network latency, which we do not focus on in this research. However, we consider the costly network traffic between zones.
6
Baseline algorithms
In this section, the baseline algorithms used to compare with the DQN-Scheduler are explained briefly. Those algorithms are well-known and have been utilized in many fields. 6.1
Genetic Algorithm (GA):
GA is a population-based meta-heuristic algorithm that provides multiple solutions[35][36]. The GA optimization algorithm is inspired by the natural selection mechanism. It produces a diverse population of solutions that prevent getting stuck in local optima [35]. It starts with the population of chromosomes representing the solution needed. Three biological operations are used to find the optimum chromosomes, including selection, crossover, mutation, and fitness function. The chromosomes represent the microservice placement into the PMs. The chromosomes are initially generated randomly. The chromosomes are chosen based on a fitness function. The fitness function of our research aims to enhance resource utilization and reduce network traffic consumption. Crossover and mutation operations are utilized to explore the solution space. 6.2
Particle Swarm Optimization (PSO):
PSO is a meta-heuristic algorithm that explores the solution space for optimal solutions. It mimics natural species like bees that navigate in space to find their destination[37]. Like GA, PSO utilizes a fitness function to assess the fitness of solutions. PSO consists of a population, velocity, and target destination parameters guiding particles towards their destinations. The population is generated based on the resource requirements of microservices. In addition, velocity guides particles as a group towards optimal positions. It is influenced by the positions of both the particle and the group [38]. We conceptualize microservice resource requirements as particles. Whereas PMs represent destinations spread across the solution space, offering CPU and memory resources for the particles. Once particles explore the space, a fitness 16
function evaluates each particle solution’s fitness. The fitness function aims to enhance resource utilization and reduce the cost of utilizing network traffic. 6.3
Best Fit Algorithm (BFA):
The best-fit algorithm is employed to tackle the bin packing problem (BPP). BPP is recognized as an NP-hard problem. Finding the optimal solution for BPP within polynomial time is impossible. BFA serves as a heuristic like first-fit, worst-fit, and next-fit algorithms [39]. The BFA is utilized in our research to optimize the packing of microservices onto PMs while considering resource utilization, network traffic, and load balance. Therefore, we represent the resource requirements as the value of items needing packing. Whereas, we represent the available resources in each PM as the values of the bins. Then BFA is utilized to schedule microservices into PMs. Upper and lower thresholds for resource utilization are established. When PM reaches the maximum threshold, it is removed from the list. Once PM’s resource availability falls within the lower and upper thresholds, it is queued again. This threshold management ensures improved resource utilization. BFA clusters dependent microservices near each other during packing.
7
Experiment Setup and Results
This section outlines the experiment. The section begins with a description of the experiment setup. Second, the dataset used in the experiment and the results of training the agent utilized in the scheduling process are explained. The last part of this section details the experimental results.
Figure 2: Resource Utilization by DQN-Scheduler and Benchmark Algorithms for Each PM Resource (CPU and Memory)
17
Table 3: Performance Comparison: DQN-Scheduler and Benchmark Algorithms CPU Utilization
Memory Utilization
QoS
Reliability
Latency
Name
SD
Mean
SD
Mean
SLATAH
SD
Mean
SD
Mean
GA PSO BFA DQN-Scheduler
0.73099 1.00558 0.9398 0.7499
1.299 1.4432 1.3299 1.3043
4.3192 3.2842 3.6014 3.3464
9.073 8.60123 8.4343 8.57927
0.000093 0 0 0
0.9242 0.4615 0.4490 1.1033
0.2605 0.1412 4.31443 0.4547
36.0443 8.8421 15.8433 12.7412
9.0028 3.459 4.8918 4.8555
7.1
Experiment Setup
The proposed framework was evaluated by simulating a cloud data centre. The simulation includes the workload of microservices, workload arrival times, data center capacity, network architecture, and container allocation for hosting microservices. An Alibaba dataset of the workloads of microservices was used. The cloud configuration was simulated based on real Alibaba resource specifications. The experiment was conducted using Python 3 and PyTorch 121.coda. A computer featuring a 12th Gen Intel(R) Core(TM) i7-12650H processor running at 2.30 GHz was utilized. The machine used for the simulation is equipped with a GeForce RTX 4060 GPU with 16 GB of RAM. This machine was also used for training the agent. 7.2
Dataset
Three types of datasets were utilized in our research, as follows: 1. Alibaba[40]: This dataset is publicly available from the Alibaba data center. It contains essential details such as node IDs, timestamps, and comprehensive resource utilization information. It describes the infrastructure as a microservice-based data center. It details the resource utilization data of over 90,000 containers running microservices. The dataset of containers that are allocated to more than 1,300 PMs is included. The dataset consists of node ID, timestamp, and PM resource utilization details. 2. A dataset describing the dependencies between microservices in the Alibaba data center, made public by [41] was used. This dataset presents the communication between microservices as a direct call graph. Microservices are categorized into upstream microservices (UM) and downstream microservices (DM). In addition, the pattern of communication between microservices is described in the dataset. We extracted data from over 3,280 microservices running for one hour on 300 PMs. 3. Data centre configurations were synthesized, including zone names and capacity. The data seed for distributing stateful services to cloud resources was also synthesized. This step aims to simulate real-world data centers. 7.3
Agent Training Result
In this section, the result of training the agent is briefly described. The aim is to show that the agent learns the scheduling process. The training results show that the agent is rewarded based on multiple objectives Figure 3. The cumulative rewards gained by the agent indicate that the model is progressing in its learning. This also demonstrates that the agent has learned to schedule microservices with optimal solutions. The reward scoring utilizes a decaying ϵ-greedy algorithm. Initially, the agent starts with high randomness in its actions to explore the action space, gradually decreasing randomness as it learns to select more optimal actions. Figure 3 shows that the agent started to learn and selected close to optimal actions. The results demonstrate that the model maintains a consistent average reward, indicating that it selects nearly the optimal solution each time.
7.4
Discussing The Results
This section details the experimental results, discusses system performance, and compares the DQN-Scheduler with benchmark algorithms. 18
Reward
epoch
Figure 3: The Reward Scoring of the Agent During Training
7.4.1
Load Balancing of Resource Utilization
This subsection focuses on two key features of the DQN-Scheduler: enhancing resource utilization and balancing load across all PMs in the cluster. This section demonstrates the efficiency of the DQN-Scheduler compared to other approaches in these two aspects.
Figure 4: The Results of Balancing The Resource Utilization For Each Approach Utilizing: Standard Deviation and Mean Enhanced Resource utilization: As shown in Figure 2, the DQN-Scheduler increases resource utilization for most PMs for both CPU and memory, whereas other algorithms consolidate tasks into fewer PMs. This indicates that the DQN-Scheduler achieves the best outcomes in maintaining resource utilization for most PMs within the proposed upper and lower thresholds. For example, the DQN-Scheduler’s mean CPU utilization is 1.3043 cores and the standard deviation is 0.7499, indicating steady CPU utilization. Similarly, memory utilization has a mean value of 8.57927 GB with a standard deviation of 3.3464 (Table 3 and Figure 4). This emphasises that the DQN-Scheduler not only enhances resource utilization across all PMs but also boosts the overall performance of the cloud environment. The small margin seen in the SD results in Table 3 establishes the DQN-Scheduler as the second-best performer across key metrics such as CPU utilization mean 1.3043 cores, memory utilization mean of 8.57927 GB, and latency mean of 4.8555. However, none of the current approaches consistently achieve top rankings across all metrics. Regarding resource utilization, GA has the lowest standard deviation in CPU utilization, just slightly better than the DQNScheduler by less than 0.01. GA achieved a mean CPU utilization of 1.299. However, GA has the highest standard deviation in memory utilization, with a value of 4.3192 and a mean of 9.073. This indicates that it performs worst in memory usage (Figure 4). 19
On the other hand, PSO has a lower standard deviation in memory utilization, with a value of 3.2842 and a mean of 8.60123. This indicates that PSO is consistent with memory utilization. However, PSO shows a standard deviation in CPU utilization of 1.00558 and a mean of 1.4432, indicating the worst for CPU usage. This shows that the DQN-Scheduler efficiently balances resource utilization on each PM. In addition, its consistent performance across various resource metrics indicates that the DQN-Scheduler can effectively manage CPU and memory utilization. Overall, the DQN-Scheduler shows a robust performance across various resource metrics, suggesting its potential for enhancing the sustainability and stability of cloud systems. 7.4.2
Quality of Service
This section evaluates the performance of our proposed framework in terms of SLA violations, comparing it to benchmark algorithms as modelled in Section 3. The equation (9) is proposed to calculate the SLA violation time per active host when hosts experience intensive CPU utilization. The equation considers that SLA violations occur when CPU utilization exceeds 100%. However, since we limit utilization to an upper threshold, we consider SLA violations when CPU utilization exceeds our proposed upper limit of CPU. SLA violations: Over a month-long experiment, Tan represents the total operational time of all microservices in the cluster. As a result, the GA algorithm had 0.000093 SLA violations during the experiment. The DQN-Scheduler did not violate SLA during the experiment. This indicates the DQN-Scheduler utilizes the resources efficiently Table 3. Reliability of the Scheduling Process: Equations (9) and (12) employ the SD and mean of entropy to evaluate scheduler reliability. They reflect the variability and uniformity of microservice distribution across PMs. However, before diving into the results and analyzing them, the significance of the SD and the mean values are explained. There are four different combinations of the SD and the mean. Each combination has a specific interpretation. While entropy typically represents uncertainty and randomness in distribution [42], we use it to assess the variation of microservices in each PM.
Figure 5: Microservices Variation and System Reliability Using Entropy: Standard Deviation and Mean We expect that scheduling distributes microservices to different PMs to reduce the probability of a single failure of an application and enhance availability and reliability. The main goal is to increase the variability of microservices across PMs. The following explains the implications of the different combinations of the mean and the SD: 1. Low Standard Deviation and High Mean Entropy: It indicates that microservices are well-distributed among PMs with minimal clustering. 2. High Standard Deviation and High Mean Entropy: It indicates that microservices are generally distributed evenly across PMs, with notable variation in distribution. 3. Low Standard Deviation and Low Mean Entropy: It indicates most microservices are clustered on a few PMs, showing consistent clustering. 4. High Standard Deviation and Low Mean Entropy : It indicates a strong tendency for clustering, with significant variation among PMs. It also suggests a mixture of clustered and even distributed VMs. The results in (Table 3 and Figure 5) indicate that the DQN-Scheduler has the highest SD with a value of 1.1033 and a low mean with values of 0.4547. This suggests notable variability of microservices in each PM. This demonstrates 20
that the DQN-Scheduler most effectively enhances the reliability of the cloud. GA results show the second-highest SD with a value of 0.9242 but a low mean entropy of 0.26059. The results indicate strong clustering, with some variation in microservices across PMs. The PSO and BFA have results for the SD, with values of 0.4615 and 0.4490, respectively. They have 0.1412 and 4.31443 values as the mean, respectively, suggesting less variability of microservices in PMs.Additionally, BFA has the highest mean entropy with a value of 4.3144 (Table 3 and Figure 5). The results indicate minimal clustering of microservices. The DQN-Scheduler has a relatively low mean entropy and the highest SD, indicating a strong tendency for clustering with notable variation in microservices. GA has low mean entropy and the second-highest SD, suggesting some clustering among microservices distributed to PMs. The results indicate that GA follows our framework as the second-best reliable algorithm. BFA comes in third with the highest mean entropy but a lower SD, indicating minimal clustering and less variation. The PSO results indicate microservices are clustered on a few PMs with the least variation, implying a lower level of reliability. In summary, the DQN-Scheduler has the best combination of mean and SD, demonstrating the most effective scheduling approach to enhance reliability. 7.4.3
Latency
In this section, we discuss the results of latency caused by communication between distributed microservices. Improving latency can be achieved by making microservices independent or by scheduling dependent microservices close to each other. However, while scheduling dependent microservices close to each other can reduce latency, it must be done cautiously to avoid causing system reliability. For example, clustering many replicates of a microservice in the same PM increases the risk of a single point of failure, thereby reducing reliability.
Figure 6: Latencies Caused by Approaches: Standard Deviation and Mean The results shown in Table 3 regarding latency demonstrate that the PSO algorithm has the lowest latency, with an SD of 8.8421 and a mean latency of 3.459 ms, compared to other approaches. These results suggest that most microservices have latency around this mean value. However, PSO clusters microservices in fewer PMs, reducing reliability. Therefore, it is expected that PSO has the lowest latency but also lower reliability (Table 3 and Figure 6). On the other hand, the DQN-Scheduler exhibits the second-lowest latency, with an SD of 12.7412 and a mean latency of 4.8555 ms. This indicates a balance between reliability and latency. BFA ranks third, with an SD of 15.8433 and a mean latency of 4.8918 ms. The GA produces the highest latency, with an SD of 30.0443 and a mean of 9.0028 ms (Table 3 and Figure 6). Overall, The DQN-Scheduler proved that it balances resource utilization, reliability, availability, and latency better compared to other approaches. It is the only scheduler that manages trade-offs between conflicting objectives and finds the best solutions. 7.5
Comparison with Current Studies
This section provides insight into the comparison between existing studies and our proposed framework. We found that current studies are divided into three categories based on the number of objectives: single-objective, bi-objective, and multi-objective (see Table 4). Single-objective studies, such as RSDQL [4], QL-HEFT [10], and DDQ-EES [12], do not aim to enhance overall cloud performance. For example, while RSDQL and QL-HEFT may effectively reduce 21
Real-time Scenario
QoS Satisfaction
✓
Makespan
✓ ✓
Latency
Request Success Rate
Availability
✓
Resource Utilization
✓ ✓ ✓ ✓
Energy Consumption
Scalability
Load Balancing
Response Time
Study RSDQL[4] [5] Noah [6] GRLD[7] DQTS[8] QEEC[9] QL-HEFT[10] BCRN[11] DDQ-EES[12] DQN Framework [13] E-AEO-AOA[14] WDDQN-RL[15] DQN-Scheduler
✓ ✓
✓
✓ ✓
✓ ✓
✓
✓ ✓ ✓ ✓
✓
✓
✓ ✓
✓ ✓ ✓ ✓
✓
Table 4: Comparison of Various Studies on Different Metrics
response time, they intensively utilize resources and increase energy consumption. Similarly, DDQ-EES focuses solely on reducing energy consumption, ignoring QoS parameters such as makespan, reliability, latency, and load balancing. Thus, approaches that consider only one objective may achieve high results in a particular aspect without considering other factors that enhance overall performance. Focusing on one aspect of cloud scheduling can negatively affect other aspects. The second type of study considers two objectives. Studies such as GRLD [7], DQTS [8], QEEC [9], BCRN [11], the DQN Framework [13], and E-AEO-AOA [14] propose to address two non-conflicting objectives Table (4). These studies aim to enhance makespan along with load balancing, resource utilization, or energy consumption. However, they do not consider the cost of utilizing the network, latency, and reliability, which directly affect QoS and performance. In all bi-objective approach models, the workload needed to be scheduled is considered the latency and network costs are not considered. This is due to: 1. During the experiment, the workload and cloud environment configurations were small-scale. Resulting in making latency and network traffic costs are negligible (e.g., DQTS and BCRN). 2. The assumption that workloads are scheduled at the same time on the same nodes (e.g., QEEC and BCRN). E-AEO-AOA [14] aims to balance the load and enhance energy efficiency by offloading dependent tasks to fog computing. However, it ignores the cost of utilizing resources (e.g., CPU, memory, and network) and latency, which directly affect scheduling performance. The third type of study, considering more than two objectives, comprises about 15% of current research. However, these studies also have limitations in providing online and real-time scheduling. For example, [5] trains their model using an offline approach that gathers data from past events, limiting the training process to certain environmental conditions. Similarly, Noah [6] uses a synthetic dataset for training, which may handle unseen events in the Alibaba data center but challenges optimal decisions for other data centers. To summarize, the multi-objective approaches currently suffer from: 1. Generalizing the decision-making process for scheduling workloads. 2. Complex models that do not consider conflicting objectives. 3. Ignoring real-time scheduling scenarios. 4. Utilizing small-scale datasets for validating the proposed algorithm, lacking the reality of reflecting on model performance. 22
As a result, our proposed framework enhances the scheduling process for microservices and outperforms existing studies. Our framework considers resource utilization, online scheduling, availability, reliability, latency, and realworld scheduling scenarios to enhance the overall performance of cloud computing.
8
Conclusion and Open Question
Our framework aims to use an agent as part of other components to enhance the multi-objective scheduling process. The agent contributes to scheduling microservices efficiently. We tackled four conflicting objectives: resource utilization management, load balancing, reliability, and latency. Optimizing these objectives is extremely challenging; therefore, the agent was rewarded with utilizing the Pareto Front approach. The DQN-Scheduler has components to enhance the online scheduling process, such as resource management, SLA monitoring, and cluster management. The DQN-Scheduler utilizes an agent’s action selection mechanism, utilizing Bayesian inference. We employ the mechanism to ensure the selection of the best actions in a real-time manner. The DQN-Scheduler enhances resource utilization by spreading microservices in groups across the cluster PMs. In addition, it balances the load of each PM’s resources (CPU and memory). This results in improved resource utilization and enhances the long-term performance of the cloud platform. Moreover, the DQN-Scheduler improves the QoS of the cloud by complying with scheduling constraints. The trained agent and its awareness of the resource utilization threshold benefit our scheduling framework by managing the resource utilization of PMs. There was no incident where The DQN-Scheduler violated the SLA by utilizing CPUs extensively. Furthermore, the DQN-Scheduler has clustered microservices in each PM, with a notable variation of microservices within each cluster, leading to the highest level of reliability. Finally, The DQN-Scheduler significantly reduces latency by clustering dependent microservices into the minimum number of PMs. Our DQN-Scheduler carefully balances reducing latency and enhancing the reliability and availability of microservices. In summary, our proposed framework outperformed baseline algorithms in multiple metrics, demonstrating its ability to balance conflicting objectives, including resource utilization, load balancing, reliability, and latency. Future work could focus on optimizing microservice scaling while considering the increased complexity of the system.
References [1] Y. Gan and C. Delimitrou, “The architectural implications of cloud microservices,” IEEE Computer Architecture Letters, vol. 17, no. 2, pp. 155–158, 2018. [2] L. Zhang, K. Pang, J. Xu, and B. Niu, “High performance microservice communication technology based on modified remote procedure call,” Scientific Reports, vol. 13, no. 1, p. 12141, 2023. [3] J. K. Konjaang and L. Xu, “Multi-objective workflow optimization strategy (mowos) for cloud computing,” Journal of Cloud Computing, vol. 10, no. 1, p. 11, 2021. [4] W. Lv, Q. Wang, P. Yang, Y. Ding, B. Yi, Z. Wang, and C. Lin, “Microservice deployment in edge computing based on deep q learning,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 11, pp. 2968– 2978, 2022. [5] Y. Yu, J. Liu, and J. Fang, “Online microservice orchestration for iot via multiobjective deep reinforcement learning,” IEEE Internet of Things Journal, vol. 9, no. 18, pp. 17 513–17 525, 2022. [6] Z. Li, H. Sun, Z. Xiong, Q. Huang, Z. Hu, D. Li, S. Ruan, H. Hong, J. Gui, J. He, Z. Xu, and Y. Fang, “Noah: Reinforcement-learning-based rate limiter for microservices in large-scale e-commerce services,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 9, pp. 5403–5417, 2023. [7] W. Lv, P. Yang, T. Zheng, C. Lin, Z. Wang, M. Deng, and Q. Wang, “Graph-reinforcement-learning-based dependency-aware microservice deployment in edge computing,” IEEE Internet of Things Journal, vol. 11, no. 1, pp. 1604–1615, 2024. [8] Z. Tong, H. Chen, X. Deng, K. Li, and K. Li, “A scheduling scheme in the cloud computing environment using deep q-learning,” Information Sciences, vol. 512, pp. 1170–1191, 2020. [9] D. Ding, X. Fan, Y. Zhao, K. Kang, Q. Yin, and J. Zeng, “Q-learning based dynamic task scheduling for energyefficient cloud computing,” Future Generation Computer Systems, vol. 108, pp. 361–371, 2020. [10] Z. Tong, X. Deng, H. Chen, J. Mei, and H. Liu, “Ql-heft: a novel machine learning scheduling scheme base on cloud computing environment,” Neural Computing and Applications, vol. 32, pp. 5553–5570, 2020. 23
[11] A. Asghari and M. K. Sohrabi, “Bi-objective cloud resource management for dependent tasks using q-learning and nsga-3,” Journal of Ambient Intelligence and Humanized Computing, vol. 15, no. 1, pp. 197–217, 2024. [12] Q. Zhang, M. Lin, L. T. Yang, Z. Chen, S. U. Khan, and P. Li, “A double deep q-learning model for energyefficient edge scheduling,” IEEE Transactions on Services Computing, vol. 12, no. 5, pp. 739–749, 2018. [13] Z. Peng, J. Lin, D. Cui, Q. Li, and J. He, “A multi-objective trade-off framework for cloud resource scheduling based on the deep q-network algorithm,” Cluster Computing, vol. 23, pp. 2753–2767, 2020. [14] S. Yeganeh, A. B. Sangar, and S. Azizi, “A novel q-learning-based hybrid algorithm for the optimal offloading and scheduling in mobile edge computing environments,” Journal of Network and Computer Applications, vol. 214, p. 103617, 2023. [15] H. Li, J. Huang, B. Wang, and Y. Fan, “Weighted double deep q-network based reinforcement learning for biobjective multi-workflow scheduling in the cloud,” Cluster Computing, vol. 25, no. 2, pp. 751–768, 2022. [16] Y. Tian, L. Si, X. Zhang, K. C. Tan, and Y. Jin, “Local model-based pareto front estimation for multiobjective optimization,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 53, no. 1, pp. 623–634, 2023. [17] L. Li, G. G. Yen, A. Sahoo, L. Chang, and T. Gu, “On the estimation of pareto front and dimensional similarity in many-objective evolutionary algorithm,” Information Sciences, vol. 563, pp. 375–400, 2021. [18] K. Deb and H. Jain, “An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: Solving problems with box constraints,” IEEE Transactions on Evolutionary Computation, vol. 18, no. 4, pp. 577–601, 2014. [19] A. Beloglazov and R. Buyya, “Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in cloud data centers,” Concurrency and Computation: Practice and Experience, vol. 24, no. 13, pp. 1397–1420, 2012. [20] S. Bharany, S. Badotra, S. Sharma, S. Rani, M. Alazab, R. H. Jhaveri, and T. R. Gadekallu, “Energy efficient fault tolerance techniques in green cloud computing: A systematic survey and taxonomy,” Sustainable Energy Technologies and Assessments, vol. 53, p. 102613, 2022. [21] C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948. [22] A. Cloud, “Alibaba cloud cdn and low-latency global cloud solutions,” 2021. [Online]. Available: https://www.alibabacloud.com/blog/alibaba-cloud-cdn-and-low-latency-global-cloud-solutions 597506/ [23] A. Chraibi, S. Ben Alla, A. Touhafi, and A. Ezzati, “A novel dynamic multi-objective task scheduling optimization based on dueling dqn and per,” The Journal of Supercomputing, vol. 79, no. 18, pp. 21 368–21 423, 2023. [24] Y. Wang, X. Li, P. Wan, L. Chang, and X. Deng, “Dueling deep q-networks for social awareness-aided spectrum sharing,” Complex & Intelligent Systems, vol. 8, no. 3, pp. 1975–1986, 2022. [25] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013. [26] A. M. Ellison, “Bayesian inference in ecology,” Ecology letters, vol. 7, no. 6, pp. 509–520, 2004. [27] J. Guo, Z. Chang, S. Wang, H. Ding, Y. Feng, L. Mao, and Y. Bao, “Who limits the resource efficiency of my datacenter: An analysis of alibaba datacenter traces,” in Proceedings of the international symposium on quality of service, 2019, pp. 1–10. [28] Z. Zhong and R. Buyya, “A cost-efficient container orchestration strategy in kubernetes-based cloud computing infrastructures with heterogeneous resources,” ACM Transactions on Internet Technology (TOIT), vol. 20, no. 2, pp. 1–24, 2020. [29] C. Jiang, Y. Qiu, W. Shi, Z. Ge, J. Wang, S. Chen, C. Cérin, Z. Ren, G. Xu, and J. Lin, “Characterizing co-located workloads in alibaba cloud datacenters,” IEEE Transactions on Cloud Computing, vol. 10, no. 4, pp. 2381–2397, 2022. [30] A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes, “Large-scale cluster management at google with borg,” in Proceedings of the Tenth European Conference on Computer Systems, 2015, pp. 1–17. [31] C. Reiss, J. Wilkes, and J. L. Hellerstein, “Google cluster-usage traces: format+ schema,” Google Inc., White Paper, pp. 1–14, 2011. [32] C. Delimitrou and C. Kozyrakis, “Quasar: Resource-efficient and qos-aware cluster management,” ACM Sigplan Notices, vol. 49, no. 4, pp. 127–144, 2014. 24
[33] Z. Zhang, C. Li, Y. Tao, R. Yang, H. Tang, and J. Xu, “Fuxi: a fault-tolerant resource management and job scheduling system at internet scale,” in Proceedings of the VLDB Endowment, vol. 7, no. 13. VLDB Endowment Inc., 2014, pp. 1393–1404. [34] A. Clouder. (13/11/2018) Evolution of alibaba large-scale colocation technology. [Online]. Available: https://www.alibabacloud.com/blog/evolution-of-alibaba-large-scale-colocation-technology 594172 [35] S. Katoch, S. S. Chauhan, and V. Kumar, “A review on genetic algorithm: past, present, and future,” Multimedia tools and applications, vol. 80, pp. 8091–8126, 2021. [36] Z. Michalewicz, “Genetic algorithms+ data structures= evolution programs. springer-verlag, 1999,” Google Scholar Google Scholar Digital Library Digital Library, 1999. [37] R. Kalimuthu and B. Thomas, “Design of a multi-constraint pso for resource allocation and task scheduling,” International Journal of Intelligent Systems and Applications in Engineering, vol. 12, no. 7s, pp. 426–440, 2024. [38] A. Alelyani, A. Datta, and G. M. Hassan, “Optimizing cloud performance: A microservice scheduling strategy for enhanced fault-tolerance, reduced network traffic, and lower latency,” IEEE Access, 2024. [39] M. A. Kaaouache and S. Bouamama, “Solving bin packing problem with a hybrid genetic algorithm for vm placement in cloud,” Procedia Computer Science, vol. 60, pp. 1061–1069, 2015. [40] Shutian. (2021) cluster-trace-microservices-v2021. [Online]. Available: https://github.com/alibaba/clusterdata/ tree/master/cluster-trace-microservices-v2021 [41] S. Luo, H. Xu, C. Lu, K. Ye, G. Xu, L. Zhang, Y. Ding, J. He, and C. Xu, “Characterizing microservice dependency and performance: Alibaba trace analysis,” in Proceedings of the ACM Symposium on Cloud Computing, 2021, pp. 412–426. [42] J. Amorocho and B. Espildora, “Entropy in the assessment of uncertainty in hydrologic systems and models,” Water Resources Research, vol. 9, no. 6, pp. 1511–1522, 1973.
25