PREPRINT
1
DECOFFEE: Decentralized Reinforcement Learning for Time-critical Workload Offloading and Energy Efficiency across the Computing Continuum
arXiv:2604.24507v1 [cs.NI] 27 Apr 2026
Anastasios E. Giannopoulos, Member, IEEE, Sotirios T. Spantideas, and Panagiotis Trakadas
Abstract—The rapid proliferation of latency-sensitive and battery-constrained Internet-of-Things (IoT) applications has intensified the need for intelligent workload placement mechanisms across the Edge–Cloud computing continuum. In such environments, far-edge nodes must dynamically decide whether to execute workloads locally or offload them to neighboring nodes or the cloud, while accounting for execution delay, energy consumption, and strict timeout constraints. However, workload placement in large-scale distributed infrastructures is a highly dynamic and non-convex optimization problem due to stochastic arrivals, heterogeneous computing capacities, and time-varying network conditions. This paper proposes DECOFFEE, a decentralized reinforcement learning framework for time-critical workload offloading and energy-efficient operation across the computing continuum. The proposed multi-agent learning scheme jointly optimizes system delay, energy consumption, and workload drop rate through adaptive placement decisions. Each edge agent operates as an autonomous learning entity that derives an optimal policy from local system observations and predicted network conditions. The workload placement process is formulated as parallel Markov Decision Processes and solved using a Double Dueling Deep Q-Network (DQN) architecture enhanced with Long Short-Term Memory (LSTM) forecasting to anticipate future load conditions. Extensive simulations demonstrate that DECOFFEE and its variants consistently outperform conventional rule-based and heuristic placement strategies, achieving significant reductions in delay, energy consumption, and workload drop rate under varying traffic and network conditions. Index Terms—Cloud-edge computing, Computing continuum, Deep reinforcement learning, Edge intelligence, Energy efficiency, Task offloading, Workload placement
I. Introduction The rapid expansion of Internet-of-Things (IoT) systems has brought forth a new class of time-sensitive and resource-constrained applications, including real-time video analytics, autonomous navigation, industrial monitoring, and extended reality services [1]. These applications impose stringent requirements on latency, reliability, and energy efficiency, which cannot be satisfied solely Corresponding author: Anastasios Giannopoulos Anastasios Giannopoulos is with the Research & Development Department, Four Dot Infinity, Chalandri, Athens, P.C. 15231, Greece (e-mail: [email protected]). Sotirios Spantideas is with the Research & Development Department, Four Dot Infinity, Chalandri, Athens, P.C. 15231, Greece (email: [email protected]). Panagiotis Trakadas is with the Department of Ports Management and Shipping, National and Kapodistrian University of Athens, Psachna, Evia, P.C. 34400, Greece (e-mail: [email protected]). Digital Object Identifier XX.YYYY/TMC.ZZZZ.LLLLLL
through centralized cloud computing models due to issues such as high latency and bandwidth limitations. As a result, modern compute architectures are evolving toward a distributed paradigm, where edge nodes and cloud resources are jointly leveraged to form an IoT-to-Edgeto-Cloud computing (IECC) continuum [2]. Within this continuum, a critical challenge lies in how to dynamically offload and execute computational workloads generated by IoT endpoints. These workloads arrive continuously, often with tight deadlines and limited tolerance for delay or energy overhead. At the same time, edge and cloud resources exhibit varying capacity, dynamic availability, heterogeneous capabilities, and variable levels of congestion [3]. Hence, workload placement decisions must account for spatiotemporal fluctuations in system status, while ensuring that critical tasks are completed within prescribed latency and energy budgets. Compounding this complexity is the inherently decentralized nature of the continuum, where IoT devices and edge agents must often make decisions with only partial or localized knowledge of the global system state. Furthermore, reactive strategies that rely solely on current observations are prone to suboptimal behavior due to the delay between decision execution and system feedback. This gap highlights the importance of proactive and predictive workload offloading policies that anticipate future system conditions, particularly node load and energy usage, to better align workload placement with real-time constraints [4]. A. Workload Placement across the Edge-Cloud Continuum In distributed computing architectures, the Edge–Cloud Continuum (ECC) is expected to support a range of functionalities including real-time data processing, workload orchestration, energy-aware scheduling, and cooperative task sharing between nodes [5]. Compute agents located at the edge of the network, often referred to as edge aggregators or edge agents, act as intermediaries between the data-producing IoT devices and the high-capacity cloud. These agents must operate under stringent latency and energy constraints while adapting to fluctuations in workload intensity and system resources [6]. Hence, a key function within this paradigm is the dynamic workload placement, defined as the decision-making process for
PREPRINT
assigning incoming computational tasks to either local processing units, neighboring edge nodes (horizontal offloading), or remote cloud servers (vertical offloading). Unlike static placement schemes, dynamic workload placement must react to real-time observations and future expectations of node load, network conditions, and application deadlines [7]. The latter involves a complex tradeoff between executing locally a workload or offloading it for remote processing. On the one hand, executing a task locally may reduce communication delay but, on the contrary, it can increase local congestion or energy consumption. Conversely, offloading to other nodes can incur transmission overhead or violate time constraints due to queuing and execution delays at the destination. To make informed placement decisions, it is essential to account for the full range of delay components, including queuing latency at both workload source/destination, wireless and wired transmission time, and execution delay at the destination node [8]. Simultaneously, energy consumption must be modeled holistically, incorporating local processing usage, offloading transmission cost, and the energy footprint of processing workloads at external nodes. These elements interact in a tightly coupled manner, leading to a non-convex multi-objective optimization problem, which involves jointly minimizing workload latency and energy usage while ensuring successful completion within hard deadlines and reliability constraints (e.g., low number of workload drops) [9]. This formulation becomes even more challenging in distributed settings where decision-making must occur under partial observability, asynchronous and parallel workload arrivals, and prediction uncertainty. Thus, efficient workload placement policies within ECC must be both adaptive and anticipatory, capable of leveraging forecasting mechanisms and local feedback to continuously refine their offloading strategies [10]. Recent advances in Machine Learning (ML) and Deep Reinforcement Learning (DRL) have shown promise in enabling adaptive task offloading strategies in such environments [11]. DRL agents can learn optimal policies by interacting with the environment and balancing immediate and long-term trade-offs across multiple objectives. However, the distributed nature of ECC forces the deployment of large-scale distributed DRL agents so as to ensure model scalability (i.e., heavy centralized DRL agents are avoided), local energy-awareness, and delay guarantees under partial observability and parallel execution dynamics [12]. Based on the above, in ECC, the workload offloading problem may be treated as a decentralized decision-making process, where each edge compute node independently learns to assign workloads across local, edge, and cloud resources under predictive telemetry across the continuum and constraints for time criticality and energy efficiency (EE). B. Related Work Initial approaches to workload offloading in wireless and edge computing environments primarily focused on traditional methods, encompassing centralized optimization
2
algorithms and heuristic-based schemes. These techniques often relied on extensive knowledge of the network state, resource availability, and workload characteristics to make offloading decisions. For instance, Luo et al. [13] studied offloading problems within a green and sustainable Mobile Edge Computing (MEC) framework, often employing the Lyapunov technique to minimize response time. While Lyapunov optimization enables online decision-making, it typically yields approximately optimal or sub-optimal performance [13]. Similarly, complex resource allocation problems were formulated as Mixed-Integer Programs (MIPs) and solved via decomposition methods. However, such MIPs are inherently NP-hard, meaning exact solutions incur exponential computational complexity, making them impractical for real-time decision-making in dynamic environments. Moro et al. [14] formulated resource allocation as a constrained optimization problem to jointly manage compute and radio resources in MEC, aiming to derive optimal allocations. Yet, solving these complex constrained optimization problems requires extensive computational resources and a global view of the system, hindering their applicability in rapidly changing scenarios. Furthermore, Bute et al. [15] highlighted that task allocation in vehicular edge computing networks is an NP-hard 0/1 Knapsack problem. They proposed heuristic algorithms to achieve real-time decision-making [15]. Wang et al. [16] noted that many existing solutions using heuristic or approximation methods for task offloading require hand-tuned adjustments that are expensive, time-consuming, and impractical in dynamic MEC scenarios. This implies a reactive rather than proactive approach, where policies must be reexecuted with significant computational overhead in each time slot as the environment changes. Kalinagac et al. [17] similarly acknowledged the NP-hard nature of task assignment in Unmanned Aerial Vehicle (UAV)-assisted edge networks and proposed heuristic and quasi-optimal algorithms, which, by their nature, do not guarantee global optimality and can still be computationally demanding for frequent re-evaluation. These traditional methods generally struggle with the dynamic, heterogeneous, and distributed nature of the computing continuum, often exhibiting polynomial complexity, requiring constant reexecution, and lacking proactive capabilities to adapt to unforeseen changes. To overcome the limitations of traditional approaches, recent research has increasingly explored ML and DRL techniques for dynamic task offloading in 5G edge-cloud environments and beyond. Tang et al. [18] proposed a model-free DRL-based distributed algorithm incorporating traffic prediction, dueling Deep Q-Network (DQN), and double-DQN to allow devices to make decentralized offloading decisions without needing to know other devices’ task models. This approach aimed to minimize long-term cost under uncertain load dynamics and delaysensitive tasks. However, while distributed at the execution level, its decentralized nature might not fully address global coordination challenges or partial observability of the overall system state. Nieto et al. [19] introduced a
PREPRINT
distributed DRL tool to optimize binary task offloading decisions, focusing on maximizing Quality-of-Experience (QoE) while satisfying latency requirements. They also acknowledged that traditional ML approaches often struggle due to the lack of representative datasets for every possible environment state, which DRL overcomes by learning through continuous interaction [19]. Dai et al. [20] utilized DRL to design an optimal computation offloading and resource allocation strategy for minimizing system energy consumption in multi-user end-edge-cloud orchestrated networks, modeling the problem as a Markov Decision Process (MDP). While effective for energy reduction, their solution implies a more centralized DRL agent managing the overall system. Yang et al. [21] explored decentralized frameworks for MEC offloading, noting that such frameworks can lead to multi-agent architectures that scale better. They discussed multi-agent methods where agents make independent decisions, but often these works ignored the influences caused by other agents’ decisions, which could hinder convergence to a globally optimum solution. Similarly, Wang et al. [16] also highlighted DRL ability to learn efficient offloading decisions by interacting with the environment, especially for dependent tasks represented by Directed Acyclic Graphs (DAGs). Yet, many DRLbased solutions still tend towards centralized training or single-objective optimization, which, despite offering adaptivity, may not fully capture the complexities of truly decentralized and multi-objective trade-offs under partial observability. Despite advancements, current literature often overlooks the horizontal and vertical offloading complexities inherent in multi-MEC and ECC environments, where tasks can be distributed among neighboring MEC sites or deeper into the vehicular fog [22]. For example, some studies have focused on optimizing task offloading for non-divisible and delay-sensitive tasks using model-free DRL-based distributed algorithms [23], while others have explored joint optimization of device-level and edge-level task offloading to minimize execution delay and energy consumption. However, these approaches frequently rely on centralized training structures, even if execution is distributed, which can limit scalability and adaptability in highly dynamic and decentralized computing continuum environments [21]. Also, significant gaps persist in both traditional and ML/DRL-based offloading strategies, particularly regarding the joint optimization of multiple conflicting objectives (e.g., workload execution latency, energy consumption, and workload drop rate due to deadline violations) in a coordinated yet decentralized manner. Furthermore, existing approaches often neglect proactive decision-making, leading to suboptimal or outdated actions as environments change. C. Contributions In this work, a decentralized RL for time-critical workload offloading and energy efficiency across the ECC (DECOFFEE) is proposed to address the above challenges.
3
This is achieved by formulating the workload offloading problem as parallel agent-specific MDPs, enabling fully decentralized decision-making by each edge agent within a shared stochastic environment. DECOFFEE leverages a model-free DRL approach with a Long Short-Term Memory (LSTM)-enhanced DQN to handle temporal dynamics, partial observability, and the stochastic and nonstationary nature of the continuum. This design incorporates forecasting and local feedback for proactive decisionmaking and can offer enhanced scalability, adaptability, and generalizability across heterogeneous continuum environments, providing a dynamic solution for time-critical and energy-efficient workload offloading. The main contributions of this work are summarized as follows. First, we propose a joint multi-objective optimization framework (DECOFFEE) that enables decentralized RL agents to make time-critical and energy-aware offloading decisions by jointly optimizing workload execution latency, energy consumption, and workload drop rate due to deadline violations. Based on the decentralized multi-agent DRL paradigm, we formulate the workload offloading process as a collection of parallel agent-specific MDPs, allowing each edge agent to autonomously determine both horizontal and vertical offloading actions within a shared stochastic environment. Latency and energy consumption components are mathematically formulated across all stages of workload placement (queueing, processing, and transmission) within the ECC. Also, the model-free and adaptive learning employed in this work obviates the need for explicit modeling of global system dynamics or inter-agent dependencies, thus offering enhanced scalability and generalizability across diverse and heterogeneous continuum environments. In addition, LSTM-aided DQN predictions are integrated within each DECOFFEE agent, enabling a forecasting module for predictive state enrichment and a dueling DQL module for efficient policy learning. Finally, DECOFFEE supports lightweight and real-time deployability with low inference cost, making it highly suitable for online operation in realtime edge computing systems where rapid decision-making for workload placement is paramount. The remainder of this paper is organized as follows. Section II presents the system model of the computing continuum and describes the workload generation, placement mechanisms, and delay and energy consumption models. Section III formulates the decentralized workload offloading problem as a set of parallel MDPs and introduces the distributed DRL formulation adopted in DECOFFEE. Section IV describes the proposed DECOFFEE algorithm, including the LSTM-based forecasting module and the Double Dueling Deep Q-Network architecture used for policy learning. Section V evaluates the performance of the proposed framework through extensive simulations and comparative analysis against representative baseline workload placement strategies. Finally, Section VI concludes the paper and outlines future research directions. To enhance readability, Table I lists all the acronyms used in this article.
PREPRINT
4
TABLE I Acronyms
This section presents a detailed overview of the system architecture, operational framework, and functional components that form the basis of the proposed decentralized computing environment. We first introduce the general system model, followed by a description of each core element and module involved. For clarity and consistency, all mathematical symbols used in the following methodology are compiled in Table II. A. General Architecture As illustrated in Fig. 1, we consider a three-tier architecture comprising IoT devices, Edge Agents (EAs), and a Cloud layer, supporting both edge and cloud computing services. IoT devices of each cell are served by a Radio Unit (RU), which is associated with an EA for edge computing. Thus, to host the computational workload generated by the IoT cells, the computing continuum includes N EAs and a single Cloud Agent (CA), without loss of generality. We define the set of all computing nodes as N ′ = {1, 2, . . . , N, N + 1}, where node N + 1 refers to the CA. The subset N = {1, 2, . . . , N } contains only the indices of EAs. Each EA n ∈ N handles the workload generated by its associated IoT cell. Workloads are assumed to be atomic and non-partitionable, meaning they must be fully executed by a single EA or CA [18]. Given the expected high volume of workload requests
Edge Layer Local Computing
Offloaded Workloads
EA 2
EA 4
EA 1
IoT Layer
RU 2
RU 1 IoT Devices Cloud Agent (CA) Edge Agent (EA) Radio Unit (RU)
EA N
EA 3
Horizo nta Offload l ing
ed at d er oa en rkl o W
II. System Model
Local Workloads
G
DQL DQN DRL EA ECC EE FIFO ID IECC IoT LSTM MDP MEC MIP ML MSE QoE ReLU RL RU TA UAV WS
Meaning Advantage & Value layer Cloud Agent Completely Fair Scheduling Central Processing Unit Directed Acyclic Graph DECentralized RL for time-critical workload OFFloading and Energy Efficiency across the Edge-Cloud Continuum Deep Q-Learning Deep Q-Network Deep Reinforcement Learning Edge Agent Edge-Cloud Continuum Energy Efficiency First In First Out Identifier IoT-Edge-Cloud Continuum Internet of Things Long Short-Term Memory Markov Decision Process Mobile Edge Computing Mixed-Integer Programming Machine Learning Mean Squared Error Quality of Experience Rectified Linear Unit Reinforcement Learning Radio Unit Telemetry Agent Unmanned Aerial Vehicle Workload Stack
Cloud Agent(s)
V Of ert flo ica ad l in g
Acronym A&V CA CFS CPU DAG DECOFFEE
Cloud Layer
Cell 2
Cell 1
RU 4 RU 3 Cell 3
Edge Link
Cell 4
RU N Cell N Workload Storage
Cloud Link
Processing Node
Workload Offloading
Workload IoT Devices (Workload Generators)
Fronthaul Link
Fig. 1. Three-tier computing continuum architecture. Edge Agents (EAs) receive workloads from the multi-cell IoT layer (served by Radio Units) and decide between three actions: local computation, horizontal offloading to another EA, vertical offloading to the Cloud Agent (CA).
in dense, 6G-enabled environments, efficient offloading decisions are essential to maintain performance. To support flexible workload management, each EA in the system can choose between three actions: (i) compute the workload locally using its own computational resources, (ii) offload it horizontally to another EA, or (iii) offload it vertically to the CA [24]. Each workload is time-critical and is associated with a deadline constraint, representing the maximum allowable delay until processing completion. To enable decentralized intelligence, each EA is assisted by a local DRL agent to make offloading decisions based on local observations (EA state, workload information) and shared telemetry information from the computing continuum nodes. The objective is to minimize a composite cost function that accounts for workload delay, workload drop ratio, and energy consumption. In line with computing continuum design principles [25], we further assume the presence of M Telemetry Agents (TAs) distributed across the edge layer. Each TA monitors either a single (M = N ) or a group of EAs (M < N ), collecting telemetry or performance metrics such as load, Central Processing Unit (CPU) utilization, energy usage, workload status. TAs are also connected with each other to form a telemetry continuum, enabling data sharing across clusters for improved system awareness and coordination [25], [26]. For the formulation, we consider a single episode consisting of a finite time horizon T = {1, 2, . . . , T }. Also, each time slot t ∈ T has a fixed duration of ∆ seconds. Regarding communication channels, each IoT cell is served by a dedicated RU that connects to nearby IoT devices via wireless radio access technologies such as
PREPRINT
5
TABLE II Mathematical Symbols Symbol N′ T N H Rn,k P ηn (t)
Description Set of Edge-Cloud nodes Set of time slots Number of EAs Horizontal link data rate [bps] Workload arrival probability Size of wn (t) [bits]
Symbol N R M V Rn,k wn (t) Tnmax (t)
Description Set of EAs Set of link data rates Number of TAs Vertical link data rate [bps] Workload assignment number Timeout of wn (t) [time slot]
Symbol M H T ∆ zn (t) ρn (t)
xn (t)
Local compute decision for wn (t) Waiting time of wn (t) in private WS [time slots] Waiting time of wn (t) in offloading WS [time slots] Workload ID offloaded by EA n to node k Completion time slot of pub wn,k (t) Set of active public WSs in node k Computation starting time pub slot of wn,k (t)
yn,k (t)
Offloading decision for wn (t)
D n (t)
priv ψn (t)
Completion time slot of wn (t) placed in the private WS Completion time slot of wn (t) placed in the offloading WS pub Size of wn,k (t) [bits]
fnEA,priv
Number of bits removed by public WS n of EA k [bits] Number of active public WSs in node k Load history matrix at time slot t
fnpub
Lengths of public WSs hosting workloads of EA n Overall delay of wn (t) for local processing
we
τnpriv (t) τnof f (t) pub wn,k (t) pub (t) ψn,k
Ak (t) pub ψ̂n,k (t)
W Λ
en (t)
of f (t) ψn pub ηn,k (t)
mpub n,k (t) Ak (t) L(t)
Load history window [time slots] Set of public WS length values [bits]
lpub n (t)
Energy consumption of wn (t) [Joule]
epriv (t) n
τnlocal
Energy consumption for local processing of wn (t) [Joule]
5G. Each RU n ∈ N is co-located or is linked to its corresponding EA n through a high-speed wired backhaul (e.g., optical fiber). All EAs have access to the Internet for migrating workloads towards the CA. Connectivity between EAs is ensured through wired technologies (e.g. fiber optics) to allow peer-to-peer workload offloading [12], while the binary and symmetrical connectivity matrix GN ×N defines which pairs of nodes are inter-connected. In specific, Gi,j = 1 when EA i ∈ N has wired connection to EA j ∈ N .
The internal workload placement and storage mechanisms of EAs and CA are illustrated in Fig. 2. An EA may receive a new (local) workload from its associated IoT cell with probability P at the start of each time slot [27]. The role of each EA is two-fold: it can (i) compute workloads originating from its own IoT cell (locally), or (ii) process workloads that have been offloaded from other EAs. To support this, each EA n ∈ N maintains N workload stacks (WS) for processing and one offloading WS for holding workloads awaiting offloading. All WSs operate in a First-In First-Out (FIFO) manner. The nth WS of EA n ∈ N is called private WS, reserved for local workloads that are decided to be processed locally. The remaining N −1 WSs are called public WSs, each dedicated to storing workloads offloaded by another EA. Specifically, public WS k ∈ N \ {n} of each EA n stores workloads
Rn,k pub ln,k (t)
G wd
horiz , τ vert τn,k n,k
epub n,k (t)
Description Set of TAs Set of workload sizes Number of time slots Time slot duration [sec] Workload arrival index Processing density of wn (t) [CPU cycles/bit] Placement decision for wn (t) Private CPU processing capacity in EA n [cycles/sec] Data rate between EA n and node k [bps] Length of public WS n of node k [bits] Public CPU processing capacity of node n [cycles/sec] Connectivity matrix of ECC nodes Delay awareness weight Energy consumption awareness weight Overall delay of wn,k (t) for horizontal (vertical) offloading Energy consumption for processing wn (t) in node k [Joule]
offloaded by EA k 1 . For example, if EA n = 10 offloads a workload to EA k = 5, it will be placed in the public WS 10 of EA 5. CA maintains only public WSs (one for each EA) where public WS n ∈ N stores workloads offloaded from EA n. We also assume that, upon completion of workload computation or offloading within a time slot, the next workload is scheduled for processing at the start of the following slot [27]. B. Workload Description Let wn (t) represent the workload assignment number (positive integer) for the local workload received by EA n ∈ N at time slot t ∈ T . The arrival index zn (t) ∈ {0, 1} indicates whether a new local workload arrives at EA n during slot t, defined as zn (t) = 1 (if a new workload arrives at EA n at time t, or zn (t) = 0 (no workload arrival). The size of the workload wn (t), denoted by ηn (t) (in bits), is sampled from a discrete set H = {0, η1 , η2 , . . . , η|H| }. By convention, we assume wn (t) = 0 and ηn (t) = 0, when zn (t) = 0. Each workload wn (t) is also associated with a processing density ρn (t) (in CPU cycles per bit), and a deadline Tnmax (t), representing the number of time slots available for execution. This implies that the workload wn (t) must be completed by slot t+Tnmax (t)−1; otherwise, it is dropped due to deadline violation. 1 This one-to-one mapping between offloading EA and destination WS avoids collisions when multiple offloads target the same EA simultaneously. Practically, each offloading EA n is linked via a dedicated wired connection to public WS n of the destination EA or CA.
PREPRINT
6
EA 1
w1,N+1(t)
Public WS 1
EA 2
w2,N+1(t)
Public WS 2 ...
... EA N
wN,N+1(t) Public WS N
Resource Scaling
Cloud Processor CA
(a) EA 1
w1,n(t)
Public WS 1
EA 2
w2,n(t)
Public WS 2 ...
... EA N
wN,n(t)
wn(t)
x n(t)
x
1-x
Public WS N Private WS Offloading WS
n (t)
xn(t)
Resource Scaling
Public Processor Private Processor Offload wn(t) towards node k
yn,k(t)
DECOFFEE Model n
EA n
State (Workload features + Nodes' features)
(b) Fig. 2. Workload placement and storage structures inside Cloud (CA) and Edge Agents (EAs). (a) CA contains N public workload stacks (WSs) to store workloads offloaded by EAs. (b) EA n includes 1 private WS to store workloads for local computation, N − 1 public WSs for offloaded workloads and 1 offloading WS for local workloads to be offloaded. Placement decisions are provided by the DECOFFEE model.
C. Workload Placement Decision When a new workload wn (t) arrives in EA n ∈ N , a two-level placement decision is triggered to determine its execution destination. The workload placement decision is represented by the triplet D n (t) = xn (t), yn,k (t), k , where the first-level decision variable xn (t) determines local computing or offloading, while the second-level decision variable yn,k (t) specifies the offload destination after the workload is queued for offloading (see Fig. 2). Specifically, the first-level decision xn (t) is defined as: ( 1, if wn (t) is placed in the private WS xn (t) = 0, if wn (t) is placed in the offloading WS (1) If xn (t) = 0 (i.e., workload wn (t) is to be offloaded), then yn,k (t) decides the destination node k ∈ N ′ \ {n}, which is formally set as: ( 1, if wn (t) is offloaded from EA n to node k yn,k (t) = 0, if xn (t) = 1 (i.e., local execution) (2) Here, yn,k (t) indicates that the workload is transferred from the offloading WS of EA n to the public WS n of node k ∈ N ′ \ {n} (offloading to the originating EA itself is not allowed), which can be another EA or the
CA. For instance, when D 5 (t) = 1, 0, 0 , this means that workload w5 (t) will be placed in the private WS for local processing, whereas a D 5 (t) = 0, 1, 7 indicates that workload w5 (t) will be offloaded to the public WS of node 7. As a result, the number of bits queued in the private and offloading WS of EA n at time t is given by xn (t) · ηn (t) and (1 − xn (t)) · ηn (t), respectively. To enforce that each workload is offloaded to at most one destination, we apply the following constraint: X yn,k (t) ≤ 1, ∀n ∈ N (3) k∈N ′ \{n}
D. Local Workload Computation Each workload wn (t) designated for local execution is placed in the private WS of EA n ∈ N . These workloads are processed using a CPU with constant processing capacity fnEA,priv (in CPU cycles per second), which is dedicated only to local workloads for fast processing2 . Assuming that a workload wn (t) is placed in the private WS at time slot t ∈ T , we denote the corresponding completion time slot by ψnpriv (t). If no workload is placed at that time, we set ψnpriv (t) = 0. Completion refers either to the successful processing of the workload before its deadline or to its expiration (drop) if the deadline is missed. To compute this, we first define the waiting time τnpriv (t), representing the number of slots the workload remains in the private WS before its execution begins. If a workload is scheduled for local processing, it remains in the private WS for τnpriv (t) slots3 , which is given by the following formula: priv priv ′ τn (t) = max 0, max {ψn (t )} − t + 1 (4) ′ t <t
The outer max operator ensures non-negativity, while the inner max operator finds the latest completion time among previously submitted workloads, yielding the total wait before processing begins. Based on this, the completion time slot of workload wn (t) is given by: (
&
ψnpriv (t) = min t + τnpriv (t) + )
ηn (t) · ρn (t) fnEA,priv · ∆
' − 1, (5)
t + Tnmax − 1 The first term corresponds to the computed finish time based on workload size, density, CPU rate, and slot duration ∆. The second term reflects the workload’s deadline. If the required waiting and processing time exceed the deadline, the workload is dropped. For example, 2 It could be safely assumed that other types of computational power are also available for workload execution, such as Graphics Processing Unit (GPU), Digital Signal Processor (DSP), etc. 3 The value τ priv (t) is estimated by EA n before placing w (t) in n n the private WS.
PREPRINT
7
assume a new workload w3 (5) arrives at EA 3 in time slot 5 and is locally executed. The previous workloads w3 (1), w3 (2), w3 (3), and w3 (4) completed at slots ψ3priv (1) = 3, ψ3priv (2) = 4, ψ3priv (3) = 5, and ψ3priv (4) = 12, respectively. The waiting time for workload w3 (5) is calculated as τ3priv (5) = max {0, max{3, 4, 5, 12} − 5 + 1} = 8. Thus, workload w3 (5) must wait 8 time slots before execution begins.
E. Local Workload Offloading To manage local workloads that are scheduled for offloading, each EA maintains a FIFO offloading WS. If a local workload wn (t) of EA n ∈ N is selected for offloading, it is transferred from the offloading WS to the corresponding public WS of the target EA (or the CA) via a dedicated wired link4 . Given that full connectivity between all EAs may not always be feasible, the symmetric adjacency matrix G describes the Edge layer connectivity5 . In a fully connected scenario, all elements of G equal 1. The transfer data rate (in Mbps) for EA-toEA (horizontal) and EA-to-CA (vertical) communication H is denoted by Rn,k (connection between EA n and k) and V Rn,N +1 (connection between EA n and CA), respectively. H V Transfer data rates satisfy Rn,k > Rn,N +1 for all (n, k) horizontal (EA-to-EA) connections and all (n, N + 1) vertical (EA-to-CA) connections. The set of all data rates is R = RH ∪ RV , where the sets RH and RV contain the available data rates for horizontal and vertical links, respectively. When EA n offloads wn (t) to destination node k ∈ N ′ \ {n}, the effective offloading rate is: ( Rn,k =
H Rn,k , V , Rn,k
if k ∈ N \ {n} , if k = N + 1
∀n ∈ N
(6)
of f ′ τnof f (t) = max 0, max {ψ (t )} − t + 1 n ′
(7)
where ψnof f (t) is the offload completion time of a workload placed in the offloading WS at time slot t. Evidently, the waiting time of wn (t) is computed as the positive difference between the completion time of the most time-consuming previous workload and the arrival time of wn (t). Note that the value τnof f (t) is estimated before EA n decides where to place the current workload. If there is no workload placed for offloading at time t, we set ψnof f (t) = 0. The offload completion time slot of wn (t) is then given by: 4 For simplicity, we assume wired interconnections, but this can be generalized. 5 While a time-varying G(t) can be easily adopted to model dynamic topologies, here we assume it remains fixed.
& t + τnof f (t) + )
− 1,
X k∈N ′ \{n}
' yn,k (t) · ηn (t) − Rn,k · ∆
t + Tnmax − 1 (8)
where yn,k (t) ∈ {0, 1} indicates whether EA n offloads to node k. The two arguments in the min{·} operator reflect that the offload completion time ψnof f (t) is either equal to the time slot of successful workload transfer (first term), or to the time slot of workload deadline exceedance (second term). If the offloading target is the CA (i.e. yn,N +1 (t) = V 1), then Rn,N +1 = Rn,N +1 . Otherwise, if the offloading H target is another EA, then Rn,k = Rn,k . F. Offloaded Workload Computation To support horizontal offloading across the continuum, each EA n ∈ N maintains N − 1 public WSs to handle external workload requests offloaded by other EAs, while the CA hosts N public WSs to serve offloads from all EAs. Each public WS is uniquely associated with its source EA, meaning that public WS i at node k ∈ N ′ is designated to store workloads offloaded from EA i ∈ N . When a workload from EA n is delivered to node k ∈ N ′ \ {n} at time slot t ∈ T , it is placed in public WS n of node k in the subsequent slot t + 1. At this point, a new unique pub workload assignment number wn,k (t) ∈ Z+ is generated, ensuring traceability across the network. This assignment follows: (
If a workload wn (t) is placed in the offloading WS of EA n at time slot t, the waiting time in the offloading WS, denoted τnof f (t), is computed as:
t <t
( ψnof f (t) = min
pub wn,k (t) =
wn (t′ ), 0,
if EA n offloads to node k otherwise
(9)
where wn (t′ ) is the original workload number received by pub EA n at arrival time t′ ≤ t6 . Let ηn,k (t) ∈ H ∪ {0} be the size (in bits) of the workload placed in public WS n pub of node k ∈ N ′ \ {n} at time t, and let ln,k (t) denote the length of this WS at the end of time t. A public WS n at node k is considered active at time t if either a new workload is inserted at that slot or there were residual workloads at t − 1. The active WS set Ak (t) (with element count Ak (t)) at node k ∈ N ′ is: n pub pub n | ηn,k (t) > 0 or ln,k (t − 1) > 0, . . . o n ∈ N \ {k} , if k ∈ N Ak (t) = n pub pub n | ηn,k (t) > 0 or ln,k (t − 1) > 0, . . . o n ∈ N , if k = N + 1
(10)
where the upper branch is the set of active WSs of EAs, whereas the lower branch is the same for the CA. The 6 Practically, this reassignment ensures that the identifier remains the same as the original one, but the timestamp and the offloading source-destination pair are updated.
PREPRINT
8
public processor at each node k ∈ N ′ , with computational speed fkpub , is equally allocated among the workloads of active WSs. The CA is equipped with a higher-capacity pub pub pub processor fN , ∀k ∈ +1 than EAs, satisfying fN +1 > fk N . Hence, the effective processing rate per active WS at time t is fkpub /Ak (t). This fair distribution of the processing capacity across the active public queues is complaint with the principle of the generalized processor sharing model [18], [28]. Thus, the processing capacity assigned to the offloaded workloads dynamically depends on the number of active WSs at each time slot and cannot be known a priori. Instead, each EA is only knows the public CPU processing capacity of the other EAs and CA. Note that prioritized processor capacity allocation policies can be flexibly assumed, without loss of generality [12]. For instance, when some EAs or workloads have higher priority than others, we can adopt a weighted processor capacity scaling based on different weights per EA or workload [29]. The length of the public WSs is dynamically updated based on incoming and outgoing number of bits inserted or dropped from the WS. We let mpub n,k (t) denote the number of bits dropped from public WS n of node k at the end of time slot t. The length of the WS is updated via: n pub pub pub ln,k (t) = max 0, ln,k (t − 1) + ηn,k (t) − mpub n,k (t) ) ∆ · fkpub − ρn (t) · Ak (t)
(11)
pub where it is implied that the length ln,k (t) is the difference between the bits maintained in the public WS and the bits left the public WS. Specifically, the first two (positive) terms are the previous WS length and the newly added bits at time slot t, respectively. The last two (negative) terms are the bits dropped from the public WS and the bits processed at the time slot t, respectively. Note that, for k = N +1, the computations in (11) refer to the length of the public WSs at the CA. pub The completion time slot ψn,k (t) ∈ T of workload pub wn,k (t) cannot be known a priori, since the number of offloaded workloads at public WSs is dynamic over time. Instead, the time slot of starting the computation of pub pub workload wn,k (t), denoted as ψ̂n,k (t), can be defined via: pub pub ′ ψ̂n,k (t) = max t, max ψn,k (t ) + 1 (12) ′ t <t
where it is implied that the computation of the offloaded pub workload wn,k (t) starts either right after its arrival time slot t or after the completion of the most delayed previous workload. Since the processing of workload is performed pub pub within the time slots τ ∈ [ψ̂n,k (t), ψn,k (t)], the workload size is bounded as:
Local Computing
EA n
τnpriv(t)
τnexec,priv(t)
Waiting in Private WS
Processing in Private Processor
(a) EA n τnoff(t)
HO τn,k (t)
pub τn,k (t)
exec,pub τn,k (t)
Waiting in Offloading WS
Horizontal Offloading
Waiting in Public WS
Processing in Public Processor
(b) EA n
X
pub (t) τ =ψ̂n,k
∆ · fkpub pub < ηn,k (t) ≤ ρn (t) · Ak (τ )
pub (t) ψn,k
X
pub (t) τ =ψ̂n,k
∆ · fkpub ρn (t) · Ak (τ ) (13)
CA
Vertical Offloading
τnoff(t)
VO τn,N+1 (t)
pub τn,N+1 (t)
exec,pub τn,N+1 (t)
Waiting in Offloading WS
Vertical Offloading
Waiting in Public WS
Processing in Public Processor
(c) Fig. 3. Delay components experiences by a workload wn (t). (a) The case of local computation. (b) The case of horizontal offloading towards EA k. (c) The case of vertical offloading towards the CA.
where Ak (τ ) is the time-varying number of active WSs at time slot τ . The left part of the double inequality ensures that there are remaining bits to be computed pub pub within the time interval [ψn,k (t) − 1, ψn,k (t)]. The right part reflects that the workload size cannot be greater than the total number of bits processed within the time interval pub pub [ψ̂n,k (t), ψn,k (t)]. G. Delay Components and Energy Consumption Model 1) Delay Components: The delay experienced by a given workload wn (t) encompasses several components depending on the processing destination. In the considered continuum model, three distinct cases are identified, as shown in Fig. 3(a)-(c). If the decision is local processing (i.e., xn (t) = 1), the overall delay τnlocal (t) (in time slots) experienced by workload wn (t) is expressed as: o n τnlocal (t) = min τnpriv (t) + τnexec,priv (t), Tnmax
(14)
where τnpriv (t) is the waiting delay in the private WS (defined in (4)) and τnexec,priv (t) is the processing delay using the private processor, as defined below: & τnexec,priv (t) =
pub (t)−1 ψn,k
EA k
Horizontal Offloading
ηn (t) · ρn (t) fnEA,priv · ∆
' (15)
If the workload wn (t) is to be offloaded horizontally to node k ∈ N ′ (i.e., yn,k (t) = 1), the overall experienced delay τnhoriz (t) is the following:
PREPRINT
9
horiz τn,k (t) = min
n
• pub HO τnof f (t) + τn,k (t) + τn,k (t)+
exec,pub + τn,k (t), Tnmax
o
(16)
where τnof f (t) is the waiting delay in the offloading WS HO (defined in (7)). The horizontal transfer delay τn,k (t) to offload from EA n to EA k ∈ N is: & ' X yn,k (t) · ηn (t) HO τn,k (t) = (17) Rn,k · ∆ ′ k∈N \{n}
pub pub The waiting delay in the public WS is τn,k (t) = ψ̂n,k (t)−t. Also, the processing delay of the horizontally offloaded workload wn (t) using the public processor of EA k ∈ N exec,pub pub pub is τn,k (t) = ψn,k (t) − ψ̂n,k (t). Finally, if the workload wn (t) is to be offloaded vertically (i.e., yn,N +1 = 1), then the computations to find the vert overall delay τn,k (t) are identical to those presented for the horizontal offloading by setting k = N + 1 (i.e., processing destination is the CA). This affects the V computations to account for the data rates Rn,N +1 ∈ RV of the vertical workload transfer, as well as the (public) pub processing capacity fN +1 of the CA. All delay calculations are measured in time slots. 2) Enegy Consumption Components: The energy consumption for completing a workload wn (t) is measured in Joules and is computed as the product of power consumption (in Watts) and the execution duration (in seconds). Depending on the processing decision, the total energy required to complete wn (t) (from its arrival to its execution) can be calculated under two distinct cases: • If wn (t) is locally processed, the total energy con(t) is given by: sumption epriv n
priv max ∆ · p (t) · T , n n max priv ifTn < τn (t) ∆ · ppriv (t) · τnpriv (t) + . . . n +pexec (t) · [T max − τ priv (t)] , n n n priv en (t) = ifTnmax < τnpriv (t) + τnexec,priv (t) ∆ · ppriv (t) · τ priv (t) + . . . n n exec exec,priv +p (t) · τ (t) , n n max priv if Tn ≥ τn (t) + τnexec,priv (t)
(18) where ppriv (t) is the per-slot power consumption for n waiting in the private WS, and pexec (t) is the pern slot power consumption for processing the workload locally. It is evident that the total energy consumption has three cases: (i) the first case corresponds to a workload being dropped without starting processing due to early deadline, (ii) the second case refers to interrupted processing before completion, consuming partial execution energy, and (iii) the third case reflects successful local computation within the deadline.
If workload wn (t) is offloaded to node k for processing, the total energy consumption epub n,k (t) is derived as: f of f epub (t) = ∆ · pof n (t) · τn (t) + . . . n,k HO + ptran n,k (t) · τn,k (t) + . . . pub + ppub n,k (t) · τn,k (t) + . . . exec,pub + pexec (t) , n,k (t) · τn,k
(19)
f where pof n (t) is the per-slot power consumption for waiting in the offloading queue, ppub n (t) is the power consumption for waiting in the public WS, pexec n,k (t) is the power consumption for processing the workload of EA n in the node k, and ptran n,k (t) is the power consumption for transferring the workload from the offloading queue of EA n to the public queue of node k. Thus, in this case, the total energy consumption is the aggregated energy consumed for waiting in offloading/public WSs, transferring and processing the workload. The overall energy consumption en (t) for workload wn (t) is then compactly expressed as: ( (t), if xn (t) = 1 epriv n en (t) = (20) pub en,k (t), if yn,k (t) = 1
H. Load Traceability across the Continuum To support informed DRL-driven decisions in DECOFEEE, each TA maintains historical traces of system load levels across the computing continuum. The instantaneous load of node n ∈ N ′ at time slot t is quantified as the number of active public queues An (t). Accordingly, TAs W ×(N +1) , maintain a time-windowed load matrix L(t) ∈ Z+ which records the recent W -slot load profiles across all nodes. Thus, the cell (i, j) of L(t) is defined as: Li,j (t) = Aj (t − W + i − 1),
∀i, j,
(21)
where each column j stores the load timeseries of node j ∈ N ′ and each row corresponds to a past time slot, with i ∈ {1, . . . , W }. Load matrix is updated at the end of every time slot by the TAs. In the distributed continuum architecture, each EA-specific DRL agent receives the columns of L(t) by its associated TA, thus retrieving the recent history of peer nodes. Specifically, at each time slot t, each EA accesses the most recent W load traces of peer nodes, which are essential inputs for state representation in local DRL agents, as described in Section III-B. III. Distributed Time-Critical and Energy-Efficient Workload Offloading In this section, we present the core principles and mathematical formulation of the proposed DECOFFEE framework. DECOFFEE enables decentralized workload offloading across the Computing Continuum by leveraging a multi-agent DRL paradigm, where each EA operates as
PREPRINT
10
an autonomous learning agent. At every time slot t, each EA observes (i) its local environment (received workloads, local WS status) and (ii) receives telemetry from other continuum nodes (others’ load status) through load traceability mechanisms [30]. Based on these observations, the EA selects one of three feasible actions for each incoming workload: (i) local execution, (ii) horizontal offloading to a neighboring EA, or (iii) vertical offloading to the CA. Using DRL, each DECOFFEE agent learns long-term optimal policies by capturing the temporal evolution of system congestion and workload patterns. To this end, each DRL agent minimizes a cumulative long-term cost that jointly accounts for three key objectives: (i) workload execution latency, (ii) energy consumption during waiting, processing and transmission of workloads, and (iii) workload drop rate due to deadline violations. This joint optimization enables DECOFFEE to provide workload-aware and energy-efficient offloading decisions while satisfying strict time constraints typical of time-critical processing scenarios. A. Distributed Workload Offloading as Parallel Markov Decision Processes The workload offloading process in DECOFFEE is formulated as a collection of parallel agent-specific MDPs, each governing the decision-making behavior of an EA. This formalism allows each EA to autonomously determine time-critical and energy-aware offloading actions, while collectively interacting within a shared stochastic environment. At each time slot t ∈ T , every EA n ∈ N may receive new workload wn (t), and must decide on the workload destination. These decisions are made sequentially and independently, yet the environment in which they are embedded is shared and coupled across agents due to resource contention, dynamic workload patterns, and network interactions. Formally, we define a family of N MDPs, one per EA, where the MDP n is denoted as: Mn = ⟨Sn , An , Pn , Rn , γ⟩
(22)
where the Sn is the state space, An is the action space, Pn is the transition function, Rn is the reward fuction, and γ is the discount factor. The state space captures the local observables of EA n at time t, including current workload characteristics (size, computation intensity, deadline), as well as partial observability of the other continuum nodes. The action space is the set of all possible offloading destinations for a given wn (t). The reward function quantifies the quality of an action by penalizing execution delay, energy cost, and workload loss, while the discount factor balances short- and long-term cost minimization. The transition function defines the stochastic evolution of EA n’s state due to both its own action and the evolving global environment. In classic MDPs, Pn is the probability of moving from one state to another given an action: Pn (s′ |s, a) = IP sn (t + 1) = s′ |sn (t) = s, an (t) = t (23)
Environment of EA n sn(t) State Fusion τnpriv(t) τnoff(t)
ηn(t)
Workload Stacks
EA n
Workload Size
Cell n
Local Information
pub
ln (t-1)
L(t)
DECOFFEE Agent n EA n
cn(t+1) sn(t+1)
Telemetry Agent
State
L(t) LSTM LSTM LSTM
pub
{Ak (t)} {ln,k (t-1)}
Other EAs
^
L(t+1)
DQN
CA
Partial Continuum Information
an(t)
Action
Fig. 4. The DRL interaction cycle between DECOFFEE agent n and its observed environment. The state vector of EA n combines both local information and partial observability from the other continuum nodes.
where Pn (s′ |s, a) is the probability that agent n moves to state s′ at time t+1, given it was in state s and took action a at time t. However, since the full system state is not directly observable and depends on the actions of other agents, Pn is not known in closed form because (i) the environment (i.e., the rest of the continuum) is dynamic and only partially observable, and (ii) other agents’ actions affect the state of EA n indirectly (e.g., length of public WSs at a target EA). In the DECOFFEE framework, the next state may be updated as: sn (t + 1) = fenv sn (t), an (t), ζ env (t) (24) where fenv (·) is an unknown environment transition function, ζ env (t) captures stochastic environment effects (e.g., new workload arrivals, queuing delays, public CPU contention), and current action an (t) affects the next state via changes in queuing delay and remaining load. Hence, we can implicitly learn the transition function Pn (s′ |s, a) of all DECOFFEE agents via model-free DRL with experience replay, where each DRL agent approximates optimal policies by interacting with the environment over time. B. DRL Formulation for Parallel MDPs The workload offloading problem in DECOFFEE is modeled as a collection of independent but coupled MDPs, where each EA n ∈ N operates as a distributed DRL agent interacting with its local environment, upon partially observing the global system state (see Fig. 4). Each EA in DECOFFEE is equipped with a DRL agent that interacts with its environment through three principal components: the observable state sn (t), the available action an (t), and the corresponding cost (reward) signal cn (t). These components are formally below. a) State Space: Let S = {S1 , S2 , . . . , SN } denote the collection of agent-specific state spaces, where Sn represents the finite and discrete set of observable states for EA n ∈ N . At the beginning of each time slot t ∈ T , EA n observes the local state vector sn (t) ∈ Sn , defined as:
PREPRINT
11
h
sn (t) = ηn (t), τnpriv (t), τnof f (t), lpub n (t − 1), L̂(t + 1)] (25) cn (t′ > t) = where lpub n (t − 1) captures the lengths of the public WSs that host offloaded workloads from EA n to other nodes, and L̂(t + 1) is the N -size vector of LSTM-predicted load values for all N nodes at the next time slot t + 1. pub ′ Specifically, lpub n (t − 1) = {ln,k (t − 1)}, ∀k ∈ N \ {n}. The inclusion of predicted load values L̂(t+1) in the state space of EA n is essential to enable proactive decision-making, because relying solely on instantaneous load observations would result in outdated decisions, as the system state may change before the action takes effect7 . Each EA receives shared knowledge from the respective TA, which can reconstruct lpub n (t − 1) by tracking (i) the number of bits offloaded to other nodes, and (ii) the number of bits processed or discarded by each remote public WS. Also, the load Ak (t) of each other node k ∈ N ′ \ {n} is shared by the TA. The cardinality of the state space grows with the number of nodes, the available workload sizes, and the available WS capacity. Specifically, the size of the state space is upper bounded by |Sn |= H·|T |2 ·|Λ|N ·|{0, 1, . . . , N }|(N +1) , where Λ is set of public WS length values, respectively. b) Action Space: Upon the arrival of a new workload wn (t) at EA n during time slot t, the DRL agent selects an action an (t) that dictates the offloading strategy. This action an (t) is expressed as: an (t) = D n (t) = xn (t), yn,k (t), k
(26)
A change in action an (t) modifies the system state in the subsequent time step, producing a new state sn (t + 1) ̸= sn (t), due to dynamic effects on private and public WSs and workload migration. The size of the action space per EA is 4N , as the first two elements in D n (t) are binary and the third takes N values. c) Cost Function: The cost incurred by EA n upon taking action an (t) at state sn (t) is indirectly encoded via a reward signal reflecting the cost function cn (t′ ), received at a future time slot t′ > t when the workload is resolved (either dropped or processed). The cost function is defined as: Here we consider the joint minimization of the workload execution delay (i.e. the time interval between workload arrival and workload execution), the workload energy consumption (i.e. the energy consumed from workload arrival to execution) and the workload loss ratio (due to timeout violation). To this end, the reward rn (t + 1) is given by: 7 By incorporating short-term load forecasts, each agent anticipates future congestion and selects actions that remain efficient under the next-state system conditions, compensating for the action implementation latency.
∅, Φn (t), C,
if xn (t) = 0 (no arrival) if ψnpriv (t) < t + ϕn − 1 pub ′ or ψn,k (t ) < t + ϕn − 1 (processed) otherwise (dropped)
(27)
where the cost Φn (t) received for workload wn (t) is a weighted sum of the execution delay and the energy consumed from arrival to completion of workload wn (t): ( Φn (t) =
Φpriv (t), n Φpub (t), n
if xn (t) = 1 if xn (t) = 0 and yn,k (t) = 1
(28)
where Φpriv (t) is the cost for processing the workload n wn (t) locally and is given by: priv Φpriv (t) = w ψ (t) − t + 1 + we epriv (t) d n n n
(29)
and Φpub n (t) is the cost for processing the workload in another node and is computed as:
Φpub n (t) =
X
T X
h pub ′ yn,k (t) wd ψn,k (t )−
k∈N \{n} t′ =t
t+1
i
(30)
′ + we epub n,k (t ) , ∀n ∈ E
Importantly, both cases of the workload processing cost is a weighted sum of the workload execution delay (first term) and the workload energy consumption (second term), with wd and we being constant scalars that regulate the contribution of the delay and energy consumption term, respectively8 . If the execution delay exceeds the workload-specific timeout, a fixed penalty C > 0 is applied, increasing the agent’s sensitivity to deadline violations. C. Policy Optimization Problem From a system perspective, the N MDPs evolve in parallel and asynchronously (i.e., EA-specific rewards are received in different slots), each governed by an autonomous agent learning its own policy πn : Sn → An . However, these MDPs are weakly coupled, since the action of one agent may affect the state transitions and reward structures of others. This introduces non-stationarity in each agent’s environment, a well-known challenge in distributed RL [31]. Nevertheless, under the assumption of asynchronous and partially observable dynamics, the decentralized offloading process can be modeled as a factored MDP with parallel agents acting under local information, leveraging continuum-wide load forecasts as a proxy for global awareness. This is particularly suitable for 8 Additional terms targeting to other objectives, such as priorities or subscription fee for different classes of users charged by the nodes, can be also considered without loss of generality.
PREPRINT
12
enrichment and (ii) a dueling DQL module for efficient policy learning.
LSTM Network
A1(t-1) A2(t-1) . . . AN+1(t-1)
LSTM Unit
A1(t)
LSTM Unit
A2(t)
...
AN+1(t)
...
A1(t+1) ...
Fully-connected Layer
L(t)
...
...
...
A1(t-W) A2(t-W) . . . AN+1(t-W)
LSTM Unit
^
L(t)
AN(t+1)
A. DECOFFEE Deep Q-Network
AN+1(t+1)
Double Dueling Deep Q-Network
V
Fully-connected Layer
Qn(sn(t),an(t))
Fully-connected Layer
Fully-connected Layer
A
1 1
ηn(t)
τnpriv(t)
1
τnoff(t)
N
ln (t-1)
pub
N+1
Output Layer
Hidden Layers
Input Layer
Fig. 5. The internal architecture of DECOFFEE agent n, including stacked LSTM units and a Double Dueling DQN.
edge/cloud continuum systems, where real-time responsiveness and scalability necessitate lightweight (singleagent) decision-making without centralized coordination. Each agent seeks to learn a policy πn∗ that minimizes the expected cumulative cost: πn∗ = arg min Eπn πn
"∞ X
γ · cn sn (t), an (t) t
#
t=0
(31)
subject to: constraints (3)-(8), (13),(27)-(30) where γ ∈ (0, 1] is the discount factor, and the expectation E{·} is taken over the stochastic evolution of the system, workload arrivals, and other agents’ behavioral patterns, under policy πn . To solve (31), DECOFFEE employs a model-free DRL framework, where each EA uses an LSTM-enhanced Deep Q-Network (DQN) to handle temporal dynamics, partial observability, and the inherently stochastic, non-stationary nature of the continuum. IV. DECOFFEE Algorithm To address the policy optimization problem presented in (31), we propose DECOFFEE as a decentralized DRL-based solution tailored to the computing continuum paradigm. DECOFFEE leverages the Deep Q-Learning (DQL) framework [32]. By adopting a model-free DRL approach, DECOFFEE eliminates the need for explicit modeling of the global system dynamics or the interdependencies among agents, thereby offering scalability and generalizability across heterogeneous continuum environments. Each DRL agent n ∈ N maintains a deep neural architecture, referred to as the Deep Q-Network (DQN), to approximate the optimal state-action value function Qn (sn (t), an (t)). The Q-value quantifies the expected cumulative reward starting from state sn (t) and applying action an (t), followed by the current policy thereafter. As shown in Fig. 5, the DECOFFEE architecture combines (i) a forecasting module based on LSTM for predictive state
The internal archicture of the DECOFFEE neural network model is depicted in Fig. 5. During the training phase, the DECOFFEE model n seeks to properly adjust all the network parameters θn (i.e. the matrix containing the weights of all connections and the biases of all neurons), such that the Q-values of all possible stateaction pairs are accurately predicted. Below, we describe each DECOFFEE layer. a) Input layer: The input to the DECOFFEE model at time slot t is the state vector sn (t) defined in (25), which encapsulates: the local task size ηn (t), the waiting time in private and offloading WSs (wnpriv (t), wnof f (t)), the public WSs lengths from other nodes lpub n (t − 1), and the predicted load L̂(t), respectively. Notably, the predicted load vector is obtained by passing the W -length historical load sequences through a stack of LSTM units (top part of Fig. 5). This forecasting step enables each agent to anticipate future congestion9 across the continuum nodes, rather than reacting to current information. Such proactive decision-making is critical in DECOFFEE, as workload placement decisions made at time t affect system behavior in the subsequent time slots. Without prediction, the delayed execution of an action may lead to suboptimal outcomes due to system transitions that occurred after decision issuance. b) Fully-connected hidden layers: The predicted values L̂(t) are concatenated with the rest of the input features and passed through multiple fully-connected layers. These layers constitute the backbone of the Q-network, enabling high-dimensional feature extraction and representation learning. Each layer contains a fixed number of hidden neurons activated via Rectified Linear Unit (ReLU) functions. As with standard DQL architectures, the weights of these layers are optimized during training to reduce the Bellman’s error between predicted and target Q-values [33]. c) Dueling DQL layer: To further improve learning stability and convergence, DECOFFEE adopts a Dueling DQN architecture [12], which decomposes the Q-value function into two distinct components, including (i) a state-value function Vn (sn (t)|θn ) that captures the overall quality of being in state sn (t), and (ii) an advantage function An (sn (t), an (t), |, θn ) that reflects the relative merit of taking action an (t) in that state. These two streams are independently estimated through parallel fully-connected sub-networks and recombined to obtain the Q-value, as follows: 9 The look-ahead forecasting horizon of the LSTM is a hyperparameter that needs to be tuned for optimal DECOFFEE performance.
PREPRINT
13
h
Qn sn (t), a θn = Vn sn (t) θn + An sn (t), a θn i X 1 − N +1 An sn (t), a′ θn 2 ′ N +1 a ∈{0,1}
(32) Here, θn denotes the trainable parameters of the neural network for agent n, and 2N +1 represents the count of all possible offloading actions across the N + 1 computing nodes (i.e., N EAs and the Cloud). Note that, practically, the action vector is encoded as a N + 1-size binary vector, with all elements being at 0 except the ith element which is set to 1, denoting that the processing will be hosted at node i ∈ N ′ . d) Output Layer: The final output layer of DECOFFEE yields the Q-values for all candidate actions a under the current input state sn (t) and the trained parameters θn . At inference time, the optimal action a∗n (t) is selected as the one that minimizes the predicted cost, as reflected below: min Qn sn (t), a θn (33) a∗n (t) = arg a∈{0,1}N +1
B. DECOFFEE Pseudocode Here we present the training and inference pseudocode of the proposed DECOFFEE algorithm. DECOFFEE comprises a set of N distributed agents to independently learn to make joint energy- and delay-efficient workload placement decisions. Both training and inference are performed locally within each EA, enabling decentralized learning under dynamic workloads and system heterogeneity. a) Reward-delayed Training Workflow: Algorithm 1 describes the DECOFFEE training procedure for a single DRL agent n ∈ N , using the experience replay technique [34] to improve sample efficiency and learning stability. For each workload completed at time t and had been received at time t′ < t, the experience is stored as a quadruple (sn (t′ ), an (t′ ), cn (t), sn (t′ + 1)) in a finite replay buffer of capacity NR . The agent is trained over NE episodes, with each episode spanning T time slots. To approximate the optimal placement policy πn∗ that minimizes the cost objective in (31), each DECOFFEE agent maintains two neural models with identical architecture: (i) the primary Q-network Qn with parameters θn , used for action selection and value prediction, and (ii) the Target Qnetwork Q̂n with parameters θ̂n , updated less frequently to stabilize the targets during training. The target network provides delayed versions of the Q-values, avoiding rapid oscillations in learning and improving convergence. Every Ncopy training episodes, the parameters of the primary network are copied to the target network, meaning θ̂n ← θn . The training algorithm takes as input a set of learning hyperparameters, including the episode length T , the learning rate αlr , the discount factor γ, the number of
Algorithm 1 Training of DECOFFEE agent n 1: Inputs: T , αlr , γ, NE , NR , NB , Ncopy , wd , we 2: Initialize Replay Memory with NR rows 3: Initialize Qn , Q̂n models with random θn , θ̂n . 4: Initialize Ecurrent = 0 and ϵ = 1 5: for each episode j = 1, 2, . . . , NE do 6: Initialize sn (1) 7: for each time step t = 1, 2, . . . , T do 8: Generate a new workload with probability P 9: Set workload ID wn (t) 10: Select action an (t) for wn (t) based on ϵ-greedy 11: Observe next state sn (t + 1) 12: for each completed wn (t′ ) ∈ Dn (t) (34) do ′ 13: Collect cost cn (t) of wn (t ) based on (27) 14: Store experience sn (t′ ), an (t′ ), cn (t), sn (t′ + 1) 15: end for 16: Sample a random batch B of NB experiences 17: for each experience row i ∈ B do 18: Assume row format as sn,i , an,i , cn,i , s′n,i 19: if t = T {Terminal time slot} then 20: Set target yn,i = cn,i 21: else 22: %Double Q-learning% 23: Compute target yn,i = ′ ′ ′ 24: cn,i + γ · Q̂ sn,i , arg min Q s , a | θ | θ̂ n n,i a′ 25: end if 26: Compute predicted zn,i = Qn sn,i , an,i |θn 27: end for 28: Set target values YnT arget (t) = {yn,i }, ∀i ∈ B 29: Set predicted values YnP red (t) = {zn,i }, ∀i ∈ B 30: Update loss function: parameters θn to minimize MSE YnT arget (t), YnP red (t) 31: end for 32: Set Ecurrent = Ecurrent + 1 33: if mod(Ecurrent ,Ncopy ) = 0 then 34: Copy weights θ̂n = θn 35: end if 36: if Ecurrent ≤ NE /2 then 37: Decay ϵ = 1 − 2(CurrentEpisode − 1)/NE 38: else 39: Set ϵ = 0 40: end if 41: end for ∗ 42: Output: Optimal policy πn for action-value function Qn
episodes NE , the size of the replay buffer NR , the batch size NB , the target update frequency Ncopy , and the weights wd , we for delay-energy trade-off regulation. The output is the trained policy πn∗ that maps each state to the optimal action vector. At the start of each episode, the agent initializes its state as sn (1). During each time slot t ∈ [1, T ], a new workload wn (t) may arrive from the IoT layer with probability P. If a workload arrives, the
PREPRINT
agent selects an action an (t) ∈ {0, 1}N +1 using an ϵ-greedy strategy. This means that the agent selects a random action with probability ϵ (exploration), or the highestQ action with probability 1 − ϵ (exploitation). Initially, we set to ϵ = 1, which is linearly decreased to 0 over the first NE /2 episodes, promoting exploration in early stages and pure exploitation thereafter. Regardless of whether a new workload arrives, the agent always transitions to the next state sn (t + 1) by re-evaluating the system status and acquiring updated LSTM-based predictions L̂(t + 1). Since workload execution (locally or remotely) may span multiple slots, the costs for previous workload decisions are delayed. Therefore, the agent constructs at time slot t the set Dn (t) of previously offloaded or locally executed workloads that have completed exactly at t: n o pub Dn (t) = wn (t′ ) ̸= 0 t′ < t, ψnpriv = t or ψn,k = t (34) This reflects that each agent employs a reward-delayed collection of experiences, since transition tuples of past workloads wn (t′ < t) are gathered at a future time slot t. Each completed workload yields a cost cn (t) according to the multi-objective metric defined in (27), and its experience tuple is stored in the replay buffer. A random minibatch B of NB experiences is sampled from the buffer. For each experience i ∈ B, the DECOFFEE agent computes (i) the target Q-value yn,i using the Double DQL formula [35], and the predicted Q-value zn,i using the primary Q-network. Then, the Mean Squared Error (MSE) between the predicted and target Q-values is computed over the B samples and is used to update the parameters θn . Finally, every Ncopy episodes, the target network is updated by copying θ̂n ← θn . This concludes one full episode of learning for agent n, progressively refining the local policy πn until convergence. b) Inference Workflow: Once training is complete, each DECOFFEE agent n ∈ N operates independently using its learned Qn -model to make real-time workload placement decisions. The inference phase is executed continuously at the edge site, with no gradient updates or target network interaction involved. As outlined in Algorithm 2, when a new workload arrives at time slot t, the DECOFFEE agent infers the optimal action an (t) by feeding the current state sn (t) into the trained Qn -model and selecting the action that minimizes the estimated cost. The decision is purely exploitative, which means that constantly ϵ = 0. The result is an online inference loop by multiple autonomous agents, capable of selecting delayand energy-aware offloading actions proactively under the computing continuum dynamics. C. Time Complexity The computational complexity of DECOFFEE can be assessed separately for the training and inference phases. Training Complexity: The training procedure is episodic and unfolds over NE episodes, each consisting of T time
14
Algorithm 2 Inference of DECOFFEE agent n 1: Input: Trained Qn -model with parameters θn 2: for each inference step t = 1, 2, . . . do 3: Check for new workload arrival 4: if workload wn (t) is received then 5: Construct current state sn (t) 6: Select action an (t) = arg min Qn sn (t), a | θn a 7: Execute placement decision an (t) for wn (t) 8: end if 9: end for steps. For every time step t within an episode, the agent may generate a new workload and compute the optimal placement decision. The experience of each workload is stored and used to update the Q-network using minibatches of NB samples from a replay buffer of size NR . Each time step may involve (i) iterating over a set Dn (t) of previously-arrived and currently-completed workloads (Line 12), and (ii) executing a batch update over NB experience tuples (Lines 17–27). The second loop dominates the computational cost, as it includes forward passes through the Q-network and target Qnetwork, as well as backpropagation steps. Thus, the worst-case number of iterations scales with NE ·T ·NB . Let Cf orward and Cbackprop denote the number of operations (cost) required for the feed-forward and backpropagation passes per experience sample, respectively. Then, the total computational complexity of training DECOFFEE per agent is O NE · T · NB · (Cf orward + Cbackprop ) . The values of Cf orward and Cbackprop depend on the structure of the DQN (e.g., number of layers, neurons per layer, input/output dimensions), and are identical for all agents under homogeneous configurations. Note that the parameter synchronization between the primary and target networks (Lines 33) occurs every Ncopy episodes and incurs negligible overhead in comparison to the batch updates. Inference Complexity: In the inference phase, each agent simply constructs the current state vector sn (t) upon workload arrival and selects the action with the minimum predicted cost by evaluating the Qn -network (Line 6). This involves a single feed-forward pass without any backpropagation. Therefore,the per-sample inference complexity of DECOFFEE is O Cf orward . This lightweight inference cost makes DECOFFEE suitable for online deployment in real-time edge computing systems, where rapid decisionmaking is required for workload placement. V. Numerical Evaluation In this section, we numerically evaluate the performance of the proposed DECOFFEE framework in a realistic ECC setting. Our evaluation focuses on quantifying the benefits of DECOFFEE in terms of workload delay, energy consumption and reliability (inversely proportional to the drop ratio) under dynamic workloads and partial observability. First, we investigate the convergence behavior
PREPRINT
of the distributed DECOFFEE agents during training, highlighting the impact of critical DRL hyperparameters on learning efficiency and stability. We then perform sensitivity analyses with respect to key system-level parameters such as workload intensity, offloading link data rates, and computing capacities, in order to demonstrate the robustness and scalability of the DECOFFEE policies. Finally, we conduct a comparative analysis between DECOFFEE and several state-of-the-art baseline methods, evaluating their performance trade-offs under multi-metric constraints. The simulation environment follows a realistic hardware and network setup, including computational capabilities, memory resources, and communication bandwidths. Each EA is provisioned with dual-core 5 GHz CPUs for handling private and offloaded public workloads, while the Cloud is modeled with a high-performance multi-core 30 GHz CPU capable of parallel execution of multiple workload queues. CPU capacity reflects the total computational throughput available at each node and is allocated dynamically across active queues using Completely Fair Scheduling (CFS) [36]. CFS is implemented using a thread-based model, where each active WS corresponds to a separate thread scheduled uniformly over available cores. RAM capacities are set to 16 GB for each EA and 64 GB for the Cloud, supporting buffer management and multi-threaded execution [36]. Network parameters distinguish between horizontal (EA-to-EA) and vertical (EA-to-Cloud) offloading modes. Horizontal offloading is configured at 30 Mbps, mimicking low-latency wired connections (e.g., optical fiber, Ethernet, or LAN setups), whereas vertical offloading operates at 10 Mbps, representing more constrained public network backhauls (e.g., WAN, multi-hop Internet). These default values are representative but configurable, while DECOFFEE can be flexibly tested across various hardware profiles and link rates to accommodate different deployment scenarios. All DRL training and inference experiments are implemented in Python 3.10, using the PyTorch framework (v2.0.1) with CUDA acceleration (v11.8). Training of the DECOFFEE agents is executed on a workstation equipped with an AMD Ryzen 7 7800X3D 8-Core Processor (4.20 GHz base frequency), 64 GB of DDR5 RAM, and an NVIDIA RTX 4080 GPU. The implementation supports batched training with GPU acceleration for both the Qmodel and target Q̂-model networks per agent. A. Training Convergence and Learning Behavior To effectively train the DECOFFEE agents, a set of system and learning parameters were carefully configured, with particular emphasis on tuning the hyperparameters that critically affect the convergence and stability of the distributed DRL agents. Unless otherwise stated, the simulation and learning parameters used in this subsection are summarized in Table III. DECOFFEE adopts an application-agnostic workload model, where incoming
15
TABLE III Environment and DRL Parameters Parameter Symbol Workload arrival probability P H Horizontal data rate Rn,k V Vertical data rate Rn,k Delay awareness weight wd Energy awareness weight we Workload size ηn (t) Workload density ρn (t) DRL agents count N Connectivity matrix G Private CPU frequency fnEA,priv Public CPU frequency fnpub pub Cloud CPU frequency fN +1 Number of Training Episodes NE Number of Time slots T Time slot duration ∆ Consumption in private WS ppriv n f Consumption in offloading WS pof n pub Consumption in public WS pn,k Consumption for transfer ptran n,k Consumption in private CPU pexec n Consumption in public EA CPU ppub n,k
Value 0.7 30 Mbps [18] 10 Mbps 0.5 0.5 [2, 2.1, . . . , 5] Mbits [37] 0.297 gigacycles/Mbit [37] 20 See Fig. 6 5 GHz 5 GHz 30 GHz 2000 110 0.1 sec 0.1 W/sec 0.1 W/sec 0.1 W/sec 0.2 W/sec 1 W/sec 1 W/sec
ppub n,k Tnmax αlr γ NL Opt MSE Ncopy W NLSTM NR C NB
2 W/sec 20 slots (or 2 sec) 5 · 10−4 0.99 3 × 1024 neurons Adam Lines 28-30 in Algorithm 1 500 iterations 10 steps 1 × 20 LSTM units 10000 samples 40 64 samples
Consumption in Cloud CPU Workload timeout DRL learning rate Discount factor Q-network hidden layers Q-network optimizer Loss function Update frequency LSTM lookback window LSTM hidden layers Replay Memory size Drop penalty constant Batch size
workloads are characterized solely by their size ηn (t) and processing density ρn (t), without assuming applicationspecific semantics. This abstraction allows DECOFFEE to capture a wide spectrum of practical workloads, ranging from lightweight tasks (e.g., database updates, control signaling, simple AI inference) to computation-intensive workloads (e.g., image processing or deep learning inference). Each training episode consists of T = 110 time slots. During the first 100 slots, DECOFFEE agents actively make workload placement decisions, while the final 10 slots are reserved to drain any remaining workload stacks, ensuring that delayed rewards associated with pending workloads are properly collected. Unless explicitly stated, all subsequent evaluations follow the same configuration. In this subsection, we consider N = 20 Edge Agents, interconnected according to the connectivity topology G illustrated in Fig. 6. We first examine the impact of the learning rate αlr , which directly controls the magnitude of the Q-value updates and regulates how quickly and stably the agent updates its knowledge based on new experiences. Specifically, the learning rate was evaluated over the range αlr ∈ {10−5 , 10−4 , 5 · 10−4 , 10−3 }, under heavy workload arrival
PREPRINT
16
1.2
Edge Agents
Cloud tier 1
11 12
3
4
5
13
6 7
8
Edge tier 1
2
IoT tier
3
4 5
14
...
15
18
10−5
10−4
5 × 10−4
10−3
1 0.84 0.76 0.74 0.67 0.70 0.68 0.67 0.64 0.71 0.68 0.64 0.62 0.73 0.71 0.66 0.65
-4
lr
= 5 10-4
= 10 lr
0.8 0.6 0.4
10
16
17 18
19
0
500
1000
1500
Training Episode (a)
= 0.25 = 0.50 = 0.75 = 0.99
0.8
-5
0.7 0.6 0.5 0.4
2000
0.2
0
500
1000
1500
2000
Training Episode (b)
20
Fig. 7. DRL cost averaged across DECOFFEE agents as a function of training episodes for different (a) learning rates (αlr ) and (b) discount factors (γ).
19 20
TABLE IV Converged Cost for Different Hyperparameter Combinations Discount factor (γ) 0.25 0.50 0.75 0.99 0.25 0.50 0.75 0.99 0.25 0.50 0.75 0.99 0.25 0.50 0.75 0.99
= 10 lr
1
0.2
Fig. 6. Edge-Cloud connection topology derived from connectivity matrix G.
Learning rate (αlr )
= 10-3
0.3
16 17
9
Average Cost (a.u.)
Horizontal Connections
2
0.9 lr
Average Cost (a.u.)
Vertical Connection
Cloud
Delay Coefficient (wd ) 0.75 0.5 0.25 0 0.99 1.04 0.97 1.02 0.95 0.92 0.88 0.93 0.91 0.86 0.81 0.87 1.12 0.78 1.13 1.12 1.14 0.95 0.86 0.71 1.10 0.81 0.78 0.65 0.98 0.73 0.71 0.58 1.22 0.65 0.93 0.82 0.71 0.47 0.54 0.58 0.66 0.40 0.45 0.51 0.61 0.36 0.42 0.42 0.73 0.31 0.62 0.68 1.25 0.72 0.64 0.41 1.20 0.55 0.53 0.36 1.12 0.44 0.49 0.33 1.32 0.38 0.71 0.61
where offloading decisions affect not only instantaneous delay but also future energy consumption and queue dynamics across the continuum. Table IV further complements the convergence analysis by reporting the converged long-term cost achieved by DECOFFEE under different combinations of learning rate αlr , discount factor γ, and delay-weight coefficient wd . Since DECOFFEE optimizes a weighted sum of execution delay and energy consumption, varying wd directly regulates the trade-off between latency-awareness and energy-awareness. The results indicate that moderate learning rates (αlr = 5×10−4 ) consistently yield the lowest converged cost across most values of γ and wd , confirming the observations from Fig. 7(a). Moreover, higher discount factors (γ ≥ 0.75) generally lead to improved convergence, highlighting the importance of long-term planning in delay- and energy-aware workload placement. For the subsequent evaluations, we infer this wide hyperparameter space to justify the selected default configuration. B. Performance Sensitivity to Environment Parameterization
conditions (P = 0.7). Considering equal delay and energy consumption awareness (i.e., wd = we = 0.5), Fig. 7(a) depicts the evolution of the average cumulative cost across 2000 training episodes, averaged over all DECOFFEE agents. As expected, excessively small learning rates result in slow convergence, while overly large values introduce instability. A learning rate of αlr = 5 · 10−4 was found to provide the best trade-off between convergence speed and stability, yielding the lowest long-term cumulative cost. Since DECOFFEE jointly minimizes delay, energy consumption, and workload drops, the reward curves decrease over time toward zero. Next, Fig. 7(b) illustrates the impact of the discount factor γ ∈ {0.25, 0.5, 0.75, 0.99}. The discount factor governs the balance between immediate and future costs and plays a central role in shaping the agents’ long-term behavior. The results indicate that γ = 0.99 achieves the best performance, suggesting that DECOFFEE agents benefit from strongly prioritizing long-term system efficiency. This behavior is particularly important in DECOFFEE,
Using the optimal hyperparameter configuration identified in Section V-A, the sensitivity of DECOFFEE with respect to key environment parameters is analyzed. In particular, this subsection examines how critical environment parameters affect the system behavior under different delay–energy awareness configurations. For each evaluation scenario, a series of 200 inference episodes are executed to derive the reported metrics, using the pretrained DECOFFEE models. 1) Sensitivity to Workload Arrival Probability: We first investigate the influence of the workload arrival probability P, which reflects the traffic intensity generated by the IoT layer. Fig. 8 illustrates the DECOFFEE performance in terms of (a) percentage drop rate and (b) total energy consumption for different workload arrival probabilities and cost-awareness configurations. As expected, increasing P leads to higher system load, which consequently increases the probability of workload drops across all configurations (Fig. 8a). When the delay-awareness coefficient wd is low (e.g., wd = 0 or 0.25), the algorithm prioritizes
PREPRINT
1.0
80
Drop Rate (×100%)
0.6 0.4
60 40 20
0.2 0.0
we = 1 we = 0.75 we = 0.5 we = 0.25 we = 0
Energy Consumption (W)
wd = 0 wd = 0.25 wd = 0.5 wd = 0.75 wd = 1
0.8
0.1
0.3
0.5
0.7
0
0.9
Workload Arrival Probability ( ) (a)
0.1
0.3
0.5
0.7
0.9
Workload Arrival Probability ( ) (b)
Fig. 8. DECOFFEE performance in terms of (a) drop rate and (b) energy consumption for different workload arrival probability (P) and delay/energy awareness weights (wd /we ).
wd = 0 wd = 0.25 wd = 0.5
0.8
wd = 0.75 wd = 1
0.6 0.4 0.2 0.0
5
10
15
20
Number of Agents (a)
25
we = 1 we = 0.75 we = 0.5
100
Energy Consumption (W)
1.0
Drop Rate (×100%)
energy efficiency and tends to avoid aggressive offloading decisions, which results in higher drop rates under heavy traffic. Conversely, higher values of wd lead the agents to prioritize latency-aware placement decisions, significantly reducing the drop rate even when the arrival probability approaches P = 0.9. In this sense, parameter wd can dynamically adapt workload placement policies depending on the selected delay–energy trade-off. Fig. 8b presents the corresponding energy consumption behavior for the same workload arrival probabilities. In contrast to the drop rate results, energy consumption increases as the energy-awareness coefficient we decreases. When the system prioritizes delay minimization (i.e., larger wd and smaller we ), agents tend to perform more aggressive offloading or faster task processing, which results in higher overall energy consumption. Conversely, when the optimization focuses on energy efficiency (i.e., we close to 1), the agents prefer energy-conservative actions, resulting in significantly reduced power consumption. 2) Scalability to the Number of Agents: Next, the scalability of DECOFFEE as the number of distributed agents N increases is analyzed. Fig. 9 illustrates the resulting drop rate and energy consumption for increasing network size under the same delay–energy awareness configurations, assuming a moderate heavy traffic intensity (P = 0.7). From Fig. 9(a), it can be observed that the drop rate generally increases with the number of agents when the delay-awareness coefficient is low. This occurs because a larger number of agents produces significant traffic increase and expands the state–action space, making coordination more complex and increasing the likelihood of suboptimal placement decisions under heavy traffic conditions. However, when delay awareness dominates (i.e., larger wd values), the algorithm maintains significantly lower drop rates even as the system scales, demonstrating reliable workload completion performance in larger distributed environments. Fig. 9(b) illustrates the corresponding energy consumption as the number of agents increases. In contrast to the drop rate behavior, energy consumption tends to decrease slightly as the number of agents grows. This occurs because a denser Edge layer provides more offloading opportunities, allowing agents to distribute workloads more efficiently across available computing resources. However, configurations that prioritize delay minimization (small we values) consistently exhibit higher energy consumption due to more aggressive workload processing decisions. 3) Sensitivity to Computational Capacity: We further investigate the influence of the available computational capacity of the edge computing nodes on the performance of DECOFFEE. Fig. 10 presents the average execution delay and the energy consumption as a function of the CPU processing capacity allocated to private and public workloads. Specifically, the CPU frequency varies from 3 to 11 GHz, representing different levels of computational capability at the Edge infrastructure. Fig. 10(a) illustrates the average delay experienced by executed workloads
17
80
we = 0.25 we = 0
60 40 20 0
5
10
15
20
Number of Agents (b)
25
Fig. 9. DECOFFEE performance in terms of (a) drop rate and (b) energy consumption for increasing number of agents (N ) and delay/energy awareness weights (wd /we ). Arrival probability is P = 0.7.
under two extreme delay-awareness configurations. When the optimization focuses primarily on energy efficiency (wd = 0), the delay decreases only marginally as the available CPU capacity increases. This is because agents prefer local execution to avoid energy costs imposed by transferring the workloads, at the expense of increased waiting delays in the private WSs. In contrast, when delay awareness dominates (wd = 1), the average execution delay decreases significantly as CPU capacity increases, with the agents preferring more frequently to offload towards edge/cloud. This behavior indicates that the DRL agents actively exploit the additional processing resources to accelerate workload execution and reduce waiting delays when latency becomes a dominant optimization objective. The corresponding energy consumption behavior is illustrated in Fig. 10(b). In general, increasing CPU capacity leads to a gradual reduction in energy consumption due to shorter workload execution times and reduced waiting overhead. However, the absolute energy levels strongly depend on the energy-awareness coefficient. When energy minimization dominates (we = 1), the overall power consumption remains significantly lower compared to the delay-dominant configuration (we = 0), where the agents prioritize faster execution and offloading decisions at the expense of increased energy usage. In this context, it is evident that computational resources are exploited differently depending on the delay–energy trade-off.
PREPRINT
18
150
40
Energy Consumption (W)
125
35
Delay (sec)
1.6
25
1.2
wd = 0 wd = 1
1.0 4
6
8
10
Private/Public CPU Frequency (GHz) (a)
100
we = 1 we = 0
30
1.4
Decision Count
1.8
Compute Locally Offload Horizontally Offload Vertically
75 50
20
25
15
0
4
6
8
10
Private/Public CPU Frequency (GHz) (b)
Fig. 10. DECOFFEE performance in terms of (a) average delay (sec) of executed workloads and (b) energy consumption for increasing capacity of private/public CPU (GHz) and delay/energy awareness.
4) Distribution of Placement Decisions: To gain deeper insight into the behavior of the learned policies, Fig. 11 illustrates the distribution of decision selections across the three available workload placement options, namely local computation, horizontal and vertical offloading. The results are presented for two representative traffic conditions: sparse (P = 0.1) and dense workload traffic (P = 0.7). Under sparse workload traffic (see Fig. 11(a)), the majority of tasks are processed locally when delay awareness is low. As the delay-awareness coefficient wd increases, the agents gradually shift toward horizontal offloading, distributing workloads among neighboring agents in order to reduce execution latency. Vertical offloading remains relatively limited in this scenario because the Edge infrastructure typically has sufficient resources to handle the incoming tasks. Under dense workload traffic conditions (see Fig. 11(b)), the behavior changes significantly. For high energy awareness, DECOFFEE agents do not prefer frequent offloading decisions and local computation dominates. As the delay-awareness coefficient increases, both horizontal and vertical offloading decisions become more frequent. In particular, vertical offloading toward the Cloud becomes more prominent when delay awareness dominates, indicating that the agents increasingly exploit the larger computational capacity of the Cloud to prevent congestion in the Edge layer. At the same time, local computation decisions decrease because processing tasks locally under heavy load may lead to excessive waiting delays. As a result, the learned placement policies for different wd configurations can dynamically balance local processing and offloading decisions according to the current system load. C. Impact of LSTM inclusion 1) The Impact of LSTM Inclusion: The contribution of the LSTM-based forecasting module integrated into the DECOFFEE agents is also evaluated. The LSTM predictions are incorporated into the state representation of each DRL agent, allowing the offloading decisions to be proactive rather than purely reactive to the current system state. Fig. 12(a) compares the average system cost
1400
Compute Locally Offload Horizontally Offload Vertically
1200 1000
Decision Count
2.0
800 600 400 200
0
0.25
0.5
0.75
Delay awareness (wd ) (a)
1
0
0
0.25
0.5
0.75
Delay awareness (wd ) (b)
1
Fig. 11. Decision selection distribution across different delay/energy awareness weights under (a) sparse workload traffic (P = 0.1) and (b) dense traffic (P = 0.7).
achieved by DECOFFEE (after 100 inference runs) with and without the LSTM prediction module for increasing workload arrival probability P (wd = 1 for this comparison). It can be observed that the benefit of LSTM inclusion is most evident under intermediate traffic conditions, particularly for P ∈ [0.3, 0.7]. In this operating region, the LSTM-enabled DECOFFEE consistently achieves lower average cost than the version without forecasting, with an approximate reduction of about 10% at P = 0.3, around 16% at P = 0.5, and about 11% at P = 0.7. This indicates that, under moderate-to-high workload pressure, anticipating future congestion allows the agents to select more effective placement actions. In contrast, for very light traffic (P = 0.1), both schemes perform similarly because the system remains far from saturation and forecasting offers limited additional benefit. Likewise, for extremely dense traffic (P = 0.9), the two curves nearly converge, suggesting that the system approaches a saturated regime where even predictive placement has limited room for improvement. Fig. 12(b) further illustrates the cumulative distribution function (CDF) of the percentage drop rate for the two configurations, considering wd = 1 and P = 0.5. The curve corresponding to the LSTM-enabled DECOFFEE shifts toward lower drop-rate values, indicating improved reliability in workload execution. In particular, the maximum drop rate is around 15% for LSTM-enabled DECOFFEE, whereas that of LSTM-free approaches 19%. This confirms that incorporating temporal prediction into the DRL state representation improves the robustness of the learned policy. D. Baseline Comparisons To thoroughly assess the performance of the DECOFFEE framework, we compare it against several representative baseline workload placement strategies under different configurations. The considered baseline schemes reflect both simple rule-based policies and more advanced delay-oriented approaches commonly used in the task offloading literature [6], [18], [27]. The evaluated schemes are summarized as follows: 1) Random Placement (RP): Each agent randomly selects one of the available actions with equal proba-
PREPRINT
0.8
10
0.6
0.6
0.4
0.4
LSTM excluded LSTM included
0.2
0.2 0.1
0.3
0.5
0.7
0.9
Workload Arrival Probability ( ) (a)
0.0
0
10
20
Drop Rate (%) (b)
Fig. 12. DECOFFEE performance with vs without LSTM. (a) Average cost (a.u.) for increasing workload probability function (P). (b) CDF of the percentage Drop Rate. Delay-awareness weight is wd = 1.
8
100
2 0.1
0.3
0.5
0.7
RP LOP COO EEO RRO
120
4
80
MLEO DA-DECOFFEE EA-DECOFFEE B-DECOFFEE
60 40 20 0
0.9
Workload Arrival Probability ( ) (a)
0.1
0.3
0.5
0.7
0.9
Workload Arrival Probability ( ) (b)
Fig. 13. Performance comparison among the 9 workload placement schemes in terms of (a) Average delay (sec) and (b) energy consumption (W) for increasing arrival probability (P). 0.8
RP LOP COO EEO RRO
0.6
Drop Rate (×100%)
bility among local computation, vertical offloading to the Cloud, and horizontal offloading to another EA. In the case of horizontal offloading, the destination EA is selected uniformly at random with probability 1/(N − 1). 2) Local-Only Processing (LOP): Each agent executes all incoming workloads locally without performing any form of offloading. This scheme represents the most conservative placement strategy and serves as a lower-bound baseline when distributed resources are not utilized. 3) Cloud-Only Offloading (COO): Each agent forwards all incoming workloads to the Cloud for processing. This approach reflects scenarios where centralized computing resources dominate the workload processing, but may introduce additional communication latency and network congestion. 4) Edge-to-Edge Offloading (EEO): Each agent offloads all workloads horizontally to neighboring EAs. The destination EA is selected randomly with probability 1/(N − 1). This scheme represents an extreme case where the Edge layer handles all workloads collaboratively without utilizing Cloud resources. 5) Round-Robin Offloader (RRO) [38]: Each agent distributes incoming workloads across the available placement options in a cyclic manner. For example, considering EA n = 1, the first workload is executed locally, the second is offloaded to the Cloud, the third is sent to EA 2, the fourth to EA 3, and so forth. This strategy ensures balanced utilization of computing nodes but does not adapt to dynamic system conditions. 6) Minimum Latency Estimation Offloader (MLEO) [27], [39]: In this powerful scheme, each agent estimates the expected execution delay for every available placement option and selects the option that minimizes the predicted latency. Specifically, upon arrival of a new workload wn (t) at time slot t, agent n computes N + 1 delay estimates corresponding to the available destinations. The local execution delay is obtained directly using the private WS delay model, while the delay associated
140
MLEO DA-DECOFFEE EA-DECOFFEE B-DECOFFEE
6
0
30
RP LOP COO EEO RRO
Energy Consumption (W)
12
0.8
MLEO DA-DECOFFEE EA-DECOFFEE B-DECOFFEE
0.6
0.4
MLEO DA-DECOFFEE EA-DECOFFEE B-DECOFFEE
0.4
0.2 0.0
RP LOP COO EEO RRO
Drop Rate (×100%)
Average Cost
0.8
1.0
CDF (Probability)
LSTM excluded LSTM included
Average Delay (sec)
1.0
19
0.2
1
3
5
7
Workload Timeout Tnmax (sec) (a)
9
0.0
10
20
30
40
50
Horizontal Data Rate RnH, k (Mbps) (b)
Fig. 14. Drop rate (%) achieved by the 9 workload placement schemes for (a) increasing workload timeout values (Tnmax , in sec) and (b) H , in Mbps). Arrival increasing data-rate of horizontal links (Rn,k probability is P = 0.5.
with offloading to another node k ∈ N − {n} is calculated by combining the offloading delay and the estimated waiting time in the public queue of node k. The agent then selects the placement option that yields the lowest estimated delay. 7) DECOFFEE: Three variants of DECOFFEE scheme are considered for comparisons, including the delayaware (DA-DECOFFEE) with wd = 1, the energyaware (EA-DECOFFEE) with we = 1, and the balanced (B-DECOFFEE) with wd = we = 0.5. For all schemes, the performance is evaluated using three key metrics: the average workload execution delay (of the successfully processed workloads), the percentage drop ratio, and the overall energy consumption. The reported results are obtained by averaging the performance across 200 validation episodes after training convergence. To analyze the delay performance, a relatively relaxed workload deadline of 10 seconds is adopted, allowing tasks to remain in the system long enough to capture the execution latency of successfully processed workloads. In contrast, when evaluating the drop ratio, a stricter deadline of 2 seconds is considered, making deadline violations more likely and therefore highlighting the robustness of the workload placement strategies under tight time constraints. Fig. 13 illustrates the comparative performance of all workload placement schemes for increasing workload arrival probability. Specifically, Fig. 13(a) presents the
PREPRINT
average execution delay, while Fig. 13(b) reports the total energy consumption. As expected, the average delay increases with the workload arrival probability across all schemes due to the higher congestion levels in the system. More specifically, the delay-aware variant (DADECOFFEE) achieves the lowest delay across all traffic intensities, demonstrating the effectiveness of the learned placement policy in minimizing waiting and execution delays. For instance, at high workload intensity (P = 0.9), DA-DECOFFEE reduces the average delay by approximately 8% compared to the minimum-latency heuristic (MLEO) and by more than 50% compared to the cloudonly offloading strategy (COO). The balanced variant (BDECOFFEE) also maintains significantly lower delay than most baselines while achieving a more favorable delay– energy trade-off. In contrast, rule-based strategies such as RP, LOP, and RRO exhibit significantly higher delays as the workload intensity increases, since these schemes lack the ability to dynamically adapt workload placement decisions according to the system state. Fig. 13(b) further compares the schemes in terms of energy consumption. The energy-aware DECOFFEE variant (EA-DECOFFEE) consistently achieves the lowest power consumption across all workload intensities. For example, at P = 0.9, EA-DECOFFEE reduces the total energy consumption by roughly 60% compared to the cloudonly offloading scheme and by about 45% compared to the minimum-latency heuristic. Notably, B-DECOFFEE variant exhibits the third best and second best solution in terms of delay and energy, respectively, guaranteeing a beneficial delay-energy trade-off across all schemes. To further evaluate the robustness of the compared schemes, Fig. 14 examines the drop rate performance under different system parameters when P = 0.5. Fig. 14(a) shows the drop rate as a function of the workload timeout Tnmax , while Fig. 14(b) evaluates the drop rate H for increasing horizontal offloading data rate Rn,k . From Fig. Fig. 14(a), it is evident that increasing the workload timeout significantly reduces the percentage of dropped workloads across all schemes, as tasks are given more time to complete execution. Nevertheless, the DA-DECOFFEE variant consistently achieve the lowest drop rates across the entire range of timeout values, with MLEO yielding slightly higher drop rates. Noteworthy, as the Tnmax becomes more stringent, the performance separation across schemes becomes more obvious. Furthermore, the BDECOFFEE demonstrates the third best solver across all timeout values, following DA-DECOFFEE and MLEO schemes. In contrast, schemes that rely on rigid placement policies, such as COO and LOP, exhibit significantly higher drop ratios, especially under strict timeout constraints. Fig. 14(b) evaluates the effect of the horizontal communication capacity on workload execution reliability. As the horizontal data rate increases, the drop rate decreases for most schemes because tasks can be transferred more efficiently among Edge agents. Again, the DA-DECOFFEE variant outperforms all baseline schemes
20
across the entire range of communication capacities. MLEO and B-DECOFFEE are again the second and thrid best solutions for all the tested horizontal rates. Notably, the EEO scheme shows a significant decrease in achieved drop rates as the horizontal data rate increase because delays imposed by horizontal transfers are significantly low. The results confirm that DA-DECOFFEE can be the optimal selection when workload placement latency is of importance, whereas EA-DECOFFEE can be used for energy savings when workload timeout is more flexible. By leveraging distributed DRL with predictive system awareness, DECOFFEE can dynamically adapt workload placement decisions to changing system conditions, achieving lower delay, reduced energy consumption, and improved task completion reliability. VI. Conclusion and Future Extensions A. Conclusion This paper presented DECOFFEE, a decentralized reinforcement learning framework for delay- and energyaware workload placement across the Edge–Cloud computing continuum. The proposed approach models the workload offloading process as parallel Markov Decision Processes and employs a Double Dueling Deep Q-Network architecture enhanced with LSTM-based forecasting to proactively capture future system conditions. By enabling each edge node to operate as an autonomous learning agent, DECOFFEE dynamically adapts workload placement decisions according to the observed system state and predicted resource utilization. Extensive numerical evaluations demonstrated that DECOFFEE significantly outperforms conventional rulebased and heuristic workload placement schemes in terms of execution delay, energy consumption, and workload drop rate. The results further showed that the framework remains robust across varying workload intensities, network capacities, and system scales, while allowing flexible trade-offs between latency and energy efficiency through configurable cost weights. Overall, the proposed decentralized learning architecture provides a scalable and adaptive solution for intelligent workload management in future Edge–Cloud infrastructures. B. Future Directions Several research directions emerge from this work. First, future studies may directly investigate federated or collaborative learning mechanisms to allow distributed agents to share knowledge while preserving system scalability and privacy. Second, extending the framework to support heterogeneous application classes with different quality-ofservice requirements and resource profiles would further enhance its applicability to real-world IoT ecosystems. Third, incorporating additional system dimensions such as network congestion, communication reliability, and carbon-aware computing policies could improve the sustainability and operational efficiency of workload placement decisions. Finally, validating the DECOFFEE framework on real-world Edge–Cloud testbeds and integrating
PREPRINT
21
it with orchestration platforms for practical deployment remains an important step toward operational adoption. Acknowledgments The authors would like to thank I. Paralikas for his contribution. References [1] A. Al-Dulaimy, M. Jansen, B. Johansson, A. Trivedi, A. Iosup, M. Ashjaei, A. Galletta, D. Kimovski, R. Prodan, K. Tserpes et al., “The computing continuum: From iot to the cloud,” Internet of Things, vol. 27, p. 101272, 2024. [2] A. Giannopoulos, A. L. Suárez-Cetrulo, X. Masip-Bruin, F. D’Andria, and P. Trakadas, “Placing Computational Tasks Within Edge-Cloud Continuum: A DRL Delay Minimization Scheme,” in European Conference on Parallel Processing. Springer, 2024, pp. 36–45. [3] A. Alnoman, S. K. Sharma, W. Ejaz, and A. Anpalagan, “Emerging edge computing technologies for distributed iot systems,” IEEE Network, vol. 33, no. 6, pp. 140–147, 2019. [4] P. Liu, X. Yang, H. Ren, H. Zhang, and Z. Wang, “Learning anticipatory decision for distributed systems with robustness guarantees,” IEEE Transactions on Automation Science and Engineering, 2024. [5] P. K. Donta, I. Murturi, V. Casamayor Pujol, B. Sedlak, and S. Dustdar, “Exploring the potential of distributed computing continuum systems,” Computers, vol. 12, no. 10, p. 198, 2023. [6] A. Giannopoulos, I. Paralikas, S. Spantideas, and P. Trakadas, “HOODIE: Hybrid computation offloading via distributed deep reinforcement learning in delay-aware cloud-edge continuum,” IEEE Open Journal of the Communications Society, 2024. [7] M. Asim, Y. Wang, K. Wang, and P.-Q. Huang, “A review on computational intelligence techniques in cloud and edge computing,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 4, no. 6, pp. 742–763, 2020. [8] S. Wang, R. Urgaonkar, T. He, K. Chan, M. Zafer, and K. K. Leung, “Dynamic service placement for mobile micro-clouds with predicted future costs,” IEEE Transactions on Parallel and Distributed Systems, vol. 28, no. 4, pp. 1002–1016, 2016. [9] C. Gao, A. Shaan, and A. Easwaran, “Deadline-constrained multi-resource task mapping and allocation for edge-cloud systems,” in GLOBECOM 2022-2022 IEEE Global Communications Conference. IEEE, 2022, pp. 5037–5043. [10] D. K. Nishad, V. R. Verma, P. Rajput, S. Gupta, A. Dwivedi, and D. R. Shah, “Adaptive ai-enhanced computation offloading with machine learning for qoe optimization and energy-efficient mobile edge systems,” Scientific Reports, vol. 15, no. 1, p. 15263, 2025. [11] K. Li, X. Wang, Q. He, J. Wang, J. Li, S. Zhan, G. Lu, and S. Dustdar, “Computation offloading in resource-constrained multi-access edge computing,” IEEE Transactions on Mobile Computing, vol. 23, no. 11, pp. 10 665–10 677, 2024. [12] A. Giannopoulos, I. Paralikas, S. Spantideas, N. Nomikos, and P. Trakadas, “PDPPnet: Prioritized Delay-aware and Peer-toPeer Task Offloading in Cloud-Edge Continuum with Double Dueling Deep Q-Networks,” in 2024 IEEE 29th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD). IEEE, 2024, pp. 1–8. [13] Q. Luo, S. Hu, C. Li, G. Li, and W. Shi, “Resource scheduling in edge computing: A survey,” IEEE communications surveys & tutorials, vol. 23, no. 4, pp. 2131–2165, 2021. [14] E. Moro and I. Filippini, “Joint management of compute and radio resources in mobile edge computing: A market equilibrium approach,” IEEE Transactions on Mobile Computing, vol. 22, no. 2, pp. 983–995, 2021. [15] M. S. Bute, P. Fan, L. Zhang, and F. Abbas, “An efficient distributed task offloading scheme for vehicular edge computing networks,” IEEE Transactions on Vehicular Technology, vol. 70, no. 12, pp. 13 149–13 161, 2021. [16] J. Wang, J. Hu, G. Min, W. Zhan, A. Y. Zomaya, and N. Georgalas, “Dependent task offloading for edge computing based on deep reinforcement learning,” IEEE Transactions on Computers, vol. 71, no. 10, pp. 2449–2461, 2021.
[17] O. Kalinagac, G. Gür, and F. Alagöz, “Prioritization based task offloading in UAV-assisted edge networks,” Sensors, vol. 23, no. 5, p. 2375, 2023. [18] M. Tang and V. W. Wong, “Deep reinforcement learning for task offloading in mobile edge computing systems,” IEEE Transactions on Mobile Computing, vol. 21, no. 6, pp. 1985– 1997, 2020. [19] G. Nieto, I. De la Iglesia, U. Lopez-Novoa, and C. Perfecto, “Deep reinforcement learning techniques for dynamic task offloading in the 5g edge-cloud continuum,” Journal of Cloud Computing, vol. 13, no. 1, p. 94, 2024. [20] Y. Dai, K. Zhang, S. Maharjan, and Y. Zhang, “Edge intelligence for energy-efficient computation offloading and resource allocation in 5g beyond,” IEEE Transactions on Vehicular Technology, vol. 69, no. 10, pp. 12 175–12 186, 2020. [21] N. Yang, S. Chen, H. Zhang, and R. Berry, “Beyond the edge: An advanced exploration of reinforcement learning for mobile edge computing, its applications, and future research trajectories,” IEEE Communications Surveys & Tutorials, vol. 27, no. 1, pp. 546–594, 2024. [22] F. G. Wakgra, B. Kar, S. B. Tadele, S.-H. Shen, and A. U. Khan, “Multi-objective offloading optimization in mec and vehicularfog systems: A distributed-td3 approach,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 11, pp. 16 897– 16 909, 2024. [23] B. Yamansavascilar, A. C. Baktir, C. Sonmez, A. Ozgovde, and C. Ersoy, “Deepedge: A deep reinforcement learning based task orchestrator for edge computing,” IEEE Transactions on Network Science and Engineering, vol. 10, no. 1, pp. 538–552, 2022. [24] A. Giannopoulos, I. Paralikas, S. Spantideas, and P. Trakadas, “COOLER: Cooperative Computation Offloading in EdgeCloud Continuum Under Latency Constraints via Multi-Agent Deep Reinforcement Learning,” in 2024 International Conference on Intelligent Computing, Communication, Networking and Services (ICCNS). IEEE, 2024, pp. 9–16. [25] M.-R. Fida, A. H. Ahmed, T. Dreibholz, A. F. Ocampo, A. Elmokashfi, and F. I. Michelinakis, “Bottleneck identification in cloudified mobile networks based on distributed telemetry,” IEEE Transactions on Mobile Computing, vol. 23, no. 5, pp. 5660–5676, 2023. [26] M. Otero, J. M. Garcia, and P. Fernandez, “Towards a lightweight distributed telemetry for microservices,” in 2024 IEEE 44th International Conference on Distributed Computing Systems Workshops (ICDCSW). IEEE, 2024, pp. 75–82. [27] J. Liu, Y. Mao, J. Zhang, and K. B. Letaief, “Delay-optimal computation task scheduling for mobile-edge computing systems,” in 2016 IEEE international symposium on information theory (ISIT). IEEE, 2016, pp. 1451–1455. [28] Z.-L. Zhang, D. Towsley, and J. Kurose, “Statistical analysis of the generalized processor sharing scheduling discipline,” IEEE Journal on Selected Areas in Communications, vol. 13, no. 6, pp. 1071–1080, 1995. [29] Y. Chen, N. Zhang, Y. Zhang, X. Chen, W. Wu, and X. S. Shen, “Toffee: Task offloading and frequency scaling for energy efficiency of mobile devices in mobile edge computing,” IEEE Transactions on Cloud Computing, vol. 9, no. 4, pp. 1634–1644, 2019. [30] S. Lin, X. Zhang, Y. Li, C. Joe-Wong, J. Duan, D. Yu, Y. Wu, and X. Chen, “Online management for edge-cloud collaborative continuous learning: A two-timescale approach,” IEEE Transactions on Mobile Computing, vol. 23, no. 12, pp. 14 561–14 574, 2024. [31] B. Hazarika, K. Singh, S. Biswas, S. Mumtaz, and C.-P. Li, “Multi-agent drl-based task offloading in multiple ris-aided iov networks,” IEEE Transactions on Vehicular Technology, vol. 73, no. 1, pp. 1175–1190, 2023. [32] F. Tan, P. Yan, and X. Guan, “Deep reinforcement learning: From q-learning to deep q-learning,” in International Conference on Neural Information Processing. Springer, 2017, pp. 475–483. [33] E. Barron and H. Ishii, “The bellman equation for minimizing the maximum cost.” Nonlinear Anal. Theory Methods Applic., vol. 13, no. 9, pp. 1067–1090, 1989. [34] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Experience replay for continual learning,” Advances in neural information processing systems, vol. 32, 2019.
PREPRINT
[35] M. Shokrnezhad, T. Taleb, and P. Dazzi, “Double deep qlearning-based path selection and service placement for latencysensitive beyond 5g applications,” IEEE Transactions on Mobile Computing, vol. 23, no. 5, pp. 5097–5110, 2023. [36] C. S. Pabla, “Completely fair scheduler,” Linux Journal, vol. 2009, no. 184, p. 4, 2009. [37] C. Wang, C. Liang, F. R. Yu, Q. Chen, and L. Tang, “Computation offloading and resource allocation in wireless cellular networks with mobile edge computing,” IEEE Transactions on Wireless Communications, vol. 16, no. 8, pp. 4924–4938, 2017. [38] F. Alhaidari and T. Z. Balharith, “Enhanced round-robin algorithm in the cloud computing environment for optimal task scheduling,” Computers, vol. 10, no. 5, p. 63, 2021. [39] J. Liu and Q. Zhang, “Offloading schemes in mobile edge computing for ultra-reliable low latency communications,” Ieee Access, vol. 6, pp. 12 825–12 837, 2018.
ANASTASIOS E. GIANNOPOULOS (Member, IEEE) (M.Eng, Ph.D) received the diploma of Electrical and Computer Engineering from the National Technical University of Athens (NTUA), where he also completed his Master Engineering (M.Eng) degree, in 2018. He also obtained his Ph.D. at the Wireless and Long Distance Communications Laboratory of NTUA. His research interests include advanced Optimization Techniques for Wireless Systems, ML-assisted Resource Allocation, Maritime Communications and Multi-dimensional Data Analysis. He is currently working as a Research Associate at the National and Kapodistrian University of Athens, as well as Post-Doc at National Technical University of Athens. He has authored more than 75 scientific publications in the fields of Wireless Network Optimization, Computing Continuum, Machine Learning and Brain Multi-dimensional Analysis. Since 2022, he is a Member of IEEE and reviewer in several IEEE journals (IEEE Transactions on Mobile Computing, IEEE Vehicular Technology Magazine, IEEE Network, IEEE Access).
SOTIRIOS T. SPANTIDEAS (D.Eng, M.Sc, Ph.D) obtained the Diploma of Electrical & Computer Engineering from the Polytechnic School of the University of Patras in 2010. He then attended the Master Program ”Electrophysics” at the Royal Institute of Technology in Stockholm (KTH), from which he obtained the title MSc in 2013. In 2018 he obtained his PhD from the National Technical University of Athens (NTUA) with doctoral dissertation entitled “Development of Methods for obtaining DC and low frequency AC magnetic cleanliness in space missions”. His research interests include Electromagnetic Compatibility, Machine Learning for Wireless Networks, Magnetic Cleanliness for space missions and optimization algorithms for Computational Electromagnetics. From 2014, he is working as a Research Associate with NTUA and the National and Kapodistrian University of Athens (Department of Ports Management and Shipping - NKUA), participating in multiple Horizon projects. He has published over 50 papers in scientific journals and conferences in the fields of Electromagnetic Compatibility, Optimization Methods for Wireless Networks and Machine Learning for Resource Allocation problems.
22
PANAGIOTIS TRAKADAS (MEng, Ph.D) received his Diploma, Master Engineering, Degree in Electrical & Computer Engineering and his Ph.D. from the National Technical University of Athens. His main interests include 5G/6G technologies, such as O-RAN, NFV and NOMA, Wireless Sensor Networks, and Machine Learning based Optimization Techniques for Wireless Systems and Maritime Communications. He is currently an Associate Professor at the Department of Ports Management and Shipping, in National and Kapodistrian University of Athens and has been actively involved in many national and EU-funded research projects. He has published more than 180 papers in magazines, journals, books and conferences and a reviewer in several journals and TPC in conferences.