Conceptio › Archive › arXiv CS

arXiv CSopen access

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

distributedsystems protocols

networking, internet, protocols, distributed systems

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

arXiv:2604.02284v1 [cs.NI] 2 Apr 2026

Amr Aboeleneen∗ (Member, IEEE), Mohamed Abdallah∗ (Senior, IEEE), Aiman Erbad† (Senior, IEEE) and Amr Salem† (Senior, IEEE)

Abstract—The Metaverse faces complex resource allocation challenges due to diverse Virtual Environments (VEs), Digital Twins (DTs), dynamic user demands, and strict immersion needs. This paper introduces CIVIC (Cooperative Immersion Via Intelligent Credit-sharing), a novel framework optimizing resource sharing among multiple Metaverse Service Providers (MSPs) to enhance user immersion. Unlike existing methods, CIVIC integrates VE rendering, DT synchronization, credit sharing, and immersion-aware provisioning within a cooperative multi-MSP model. The resource allocation problem is formulated as two NP-hard challenges: a non-cooperative setting where MSPs operate independently and a cooperative setting utilizing a General Credit Pool (GCP) for dynamic resource sharing. Using Deep Reinforcement Learning (DRL) for tuning resources and managing cooperating MSPs, CIVIC achieves 12-36% higher request completion, 23-70% higher fulfillment rates, 20-60% more served clients, and up to 51% more fairly distributed requests, all with competitive costs. Extensive experiments demonstrate CIVIC’s resilience, adaptability, and robust performance under dynamic load conditions and unexpected demand surges, making it suitable for real-world distributed Metaverse infrastructures. Index Terms—Deep Reinforcement Learning, Immersion, Metaverse, Multi Service-Provider, Resource Allocation, Cooperative Systems, Digital Twins.

I. INTRODUCTION

He Metaverse represents a paradigm shift in human–computer interaction, emerging as a persistent, immersive digital universe where physical and virtual realities converge. This transformative technology enables users to participate in various virtual rooms (VRooms) of different virtual environments (VEs), ranging from educational settings to entertainment venues, and interact with digital twins (DTs) of real-world objects [1]. As the global Metaverse market is projected to reach $936 billion by 2030, growing at a compound annual growth rate (CAGR) of 46% [2], its applications span diverse domains including remote work, social interaction, gaming, education, and industrial training. At the core of the Metaverse experience lies the critical challenge of immersion, which refers to the degree to which users feel present and engaged within their VRooms. Achieving high immersion levels requires seamless integration of multiple technological components, such as high-fidelity VEs, accurate DTs with real-time synchronization, and responsive user interactions. This work was made possible by the GSRA grant # GSRA9-L-1-051822022 from the Qatar National Research Fund (a member of Qatar Foundation). The findings achieved herein are solely the responsibility of the authors. Amr Aboeleneen and Mohamed Abdallah are with the College of Science and Engineering, Hamad Bin Khalifa University, Qatar. Aiman Erbad and Amr Salem are with the College of Engineering, Qatar University, Qatar.

These requirements place unprecedented demands on computational and network resources, as Metaverse applications must maintain frame rates exceeding 60 frames per second (FPS), deliver ultra-low latency below 20 milliseconds, and support bandwidth-intensive operations reaching 4.6 terabits per second for educational applications alone [3]. The resource allocation challenge in the Metaverse is further complicated by its inherently heterogeneous and dynamic nature [4]. VRooms vary significantly in their experience and resource requirements. For instance, a virtual art gallery demands high visual fidelity for static displays, while a gaming arena requires rapid response times and complex physics simulations [5]. Moreover, user populations shift dynamically as users navigate between different VRooms according to their interests, resulting in sudden surges that may affect the overall experience. These dynamics create a complex optimization problem that Metaverse service providers (MSPs) must solve to maintain immersion and ensure the quality of perceptual experience (QoPE) for all users. The challenge becomes even more pronounced in multi-MSP scenarios, where multiple MSPs operate independently with varying budgets and resource capacities. Without coordination, this leads to inefficient resource utilization, where some MSPs struggle with demand surges, while others have idle capacity. Additionally, the complexity of jointly optimizing computational resources and network resources for VE and DT, while maintaining immersion thresholds across heterogeneous VEs, creates an NP-hard optimization problem (Proven later) that traditional approaches cannot solve effectively. Although many articles in the literature have focused on Metaverse resource management (see Section II), existing approaches have not provided a holistic view of the problem and often neglect certain aspects. Many rely on oversimplified immersion models that use basic metrics to approximate quality without capturing the nuanced interplay between VE rendering, DT fidelity, and their requirements. Resource requirement scaling is often assumed to be linear, overlooking various aspects, including real-world nonlinearities such as density-dependent effects and heterogeneous room-specific demands. Additionally, they typically assume a single provider operating in isolation. In this paper, we address these critical resource allocation challenges by proposing a Cooperative Immersion Via Intelligent Credit-sharing (CIVIC). This novel framework optimizes resource allocation and coordination across multiple MSPs to enhance users’ immersion. Our key contributions can be summarized as follows: • First, to better quantify user immersion with DTs’ fi-

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

delity, we introduce a more comprehensive immersion model that combines structural similarity, video streaming smoothness, and DT fidelity (structural, behavioral, and temporal). • Secondly, we present novel non-linear resource scaling functions that incorporate density effects, resource sharing efficiency, and heterogeneous VRoom requirements to identify the needed resources for each VRoom. • Then, we formulate the resource allocation challenge as two distinct non-convex NP-hard optimization problems designed to increase the number of fulfilled requests for all Heads with varying demands for the two deployment options: non-cooperative, where MSPs independently optimize allocation of different resources for their clients (i.e., Heads), and cooperative, which additionally incorporates inter-MSP resource sharing via a novel shared credit pool. • We then propose two Deep Reinforcement Learning (DRL) solutions that learn optimal policies for both noncooperative and cooperative settings, while adhering to the system dynamics and different constraints. • To validate our solutions, rigorous testing and experimentation were performed on both solutions and showed significant improvements in request completion, fulfillment, resource balance, overall system efficiency, and adaptability compared to baseline methods. Following this section, the paper proceeds as follows: Section II reviews relevant literature in the field. Section III introduces our system model, followed by Section IV, which formalizes the optimization problems. Section V presents our DRL approach to address these challenges. Performance evaluation is conducted in Section VI, and Section VII concludes the work with the discussion of key findings and future directions.

oped fidelity/identicality dimensions and evaluation methodologies [24]–[27], yet DT fidelity is often treated as a separate modeling/evaluation concern rather than an explicit term in immersion-driven provisioning. CIVIC bridges this split by making immersion explicitly VE–DT coupled, integrating VE perceptual quality with DT structural/behavioral/temporal fidelity within a unified objective (Table II, columns “Imm. Metric” and “DT Fidelity”). On the resource allocation side, a large body of work optimizes communication and/or computing resources to improve QoE/immersion. Network-centric schemes study attentionaware/customized provisioning and radio-side QoE optimization [7], [8], [28]. Rendering/offloading research explores where computation should occur, including MEC-enabled VR optimization [9], multi-objective learning that trades immersion against device energy [10], collaborative/reliable computation for rendering [11], and vehicular AR Metaverse resource optimization under energy constraints [12]. Streamingoriented work improves VE delivery via FoV/tile decisions and constrained learning [14], [15]. Broader multi-tier frameworks incorporate admission/placement and cross-tier allocation using DRL [4], [13]. While these directions substantially advance immersion-driven provisioning, they typically assume a single provider/controller domain, treat DT synchronization quality as outside the optimized experience metric, and do not explicitly model heterogeneous VRooms with distinct VE/DT primitives and nonlinear scaling with user density and fidelity targets. CIVIC targets this missing intersection by (i) jointly provisioning VE and DT resources to maximize holistic immersion, (ii) explicitly modeling heterogeneous VRooms and nonlinear scaling laws that govern both rendering and DT synchronization cost, and (iii) addressing a multi-MSP setting with time-varying localized scarcity via a shared-credit cooperation mechanism that enables cross-MSP resilience (Table II, columns “Cross-MSP” and “Incent./Credit”).

II. RELATED WORK The Metaverse differs from conventional applications (e.g., online gaming and video playback) because it requires persistent, bidirectional interaction between users and a shared virtual world under stringent latency, bandwidth, and compute constraints [1]. Consequently, closely related work can be organized around two tightly coupled questions: (i) how immersion/QoE is quantified (i.e., what the system should optimize), and (ii) how resources are allocated to realize that immersion under constraints (i.e., how the system should provision compute and networking). Table II summarizes representative efforts and highlights the gap addressed by CIVIC. On the immersion calculation side, prior studies have proposed QoE-style indicators spanning network KPIs (rate, loss, and latency) [7], [17]–[19], media/device factors (rendering delay, resolution, and video-quality measures) [8], [9], [14], and perception/engagement variables tied to content and encoding decisions [20]–[23]. These formulations motivate immersion-centric optimization, but they are commonly VEcentric: they primarily capture what users visually perceive and how the VE is delivered. In parallel, DT work has devel-

III. SYSTEM MODEL In this section, we introduce the proposed system model as in Figure 1. We begin by describing the main entities, their terminologies, and goals, followed by the two problem formulations, P and P̄ , which integrate these components together for both non-cooperative and cooperative settings, respectively. For better readability, the table of notations is supplied in TABLE I. A. Network Architecture and Entities As shown in Figure 1, our Metaverse environment comprises four primary entities: Metaverse Service Providers (MSPs), Cluster Heads (Heads), Virtual Rooms (VRooms), and Virtual Room Clients (VRCs). a) Metaverse Service Providers (MSPs): MSPs serve as essential gateways, enabling user access to the Metaverse by delivering the virtualized computing and networking infrastructure required for Heads to render and operate their VRooms and DTs. Each MSP operates with an initial budget (allocated by higher management) that provides access

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

TABLE I: Table of Notations Notation

Description

M, H, V, C, T Ch ⊆ C Hm ⊆ H Dh (t) Ih wq , wf , wa , wσ , wη (t) (t) (t) Q̂h , F̂h , Âh Qh , Fh , Ah σh and ηh ωh (t) (t) Bh and fh (t) (t) (t) εh , βh , τh Mε , N ε Mβ , N β fd , λ (t) Rsy Rby fvmin , fvmax κ Ch , Vhmax ι, Λ, ϑ KComp ,KN et φ Pv , Ov , Av Nd , SVd , Ud e0 , . . . , e4 j1 , . . . , j4 (t) δm (t) zh (t) (t) Sm , Wm (t) Dm Bvh ∆β GCP (t) (t) Budgetm (t) Lm (t) Hm,active Irequest (h, t) Ithreshold ϕ(.) wimm , wfin P and P̄ S, A, P, and R MINLP rarr , rdep

Set of MSPs, Heads, VRooms, VRCs and Timesteps Subset of VRCs linked to Head h. Set of Heads served by MSP m. Set of IoT devices and sensors for Head h. Immersion level of Head h at time t. Weights for QOPE, frame rate, accuracy, SSIM and VMAF. Normalized QOPE, frame rate, and accuracy for head h at t. QoPE, frame rate, and Accuracy of DTs within the Head h. SSIM and VMAF. Average rotation speed of clients in Head h. Bitrate and frame rate for Head h. Structural, behavioral, and temporal accuracy for DTs inside h VRoom. Sets for structural accuracy calculation. Sets for behavioral accuracy calculation. Update frequency of DT d and Rate of accuracy reduction. E and N etDT General form of total scaled resource of type R for a VRoom component y. Examples: CompV b b General form of base resource of type R for a VRoom component y Minimum and maximum VRoom frame rate. Resource-specific scaling exponent (e.g., κComp and κN et ). Number of clients of the head h and the maximum user capacity of VRoom v. Resource sharing efficiency factor, user density impact, and user scaling parameter. unit prices for computing and network resources. General Scaling Factor. VRoom’s VE parameters: Polygon count, physical objects, interaction points. VRoom’s DT parameters: Number of sensors, state variables, update frequency. SSIM calculation coefficients [6]. VMAF calculation coefficients [6]. Donation fraction of MSP m’s post-allocation surplus. Binary Head-level satisfaction indicator at time t. Post-allocation local surplus and deficit of MSP m at time t. Actual donation of MSP m to the GCP at time t. Discrete behavioral-accuracy grid associated with Head h’s VRoom. Step size of the behavioral-accuracy grid, set to 0.05. General Credit Pool’s resources at time t. Remaining budget available to MSP m at the beginning of time t. Binary MSP-level request fulfillment indicator. Set of active heads for MSP m at time t. Request indicator function. Immersion threshold requirement. Piece-wise efficiency function. Reward weights. our optimization problem for non-cooperative and cooperative settings. State space, Action space, transition probability, and reward. Mixed-Integer Nonlinear Programming. Arrival and Departure in the double Poisson process.

to payment-based cloud resources, securing the computational capacity and network connectivity needed to support Metaverse interactions. The MSPs are strategically positioned across different geographical locations and can operate either as independent entities or as collaborative units under the management of an organization. In our case, we considered both scenarios in P and P̄ . We represent the complete set of MSPs mathematically as M = {1, . . . , m, . . . , M }, where each element m denotes a specific MSP and M indicates the total number of MSPs deployed in the system. In our system, we assume that each MSP serves multiple Heads. b) Cluster Heads (Heads) and Virtual Rooms (VRooms): Heads are physical establishments such as libraries, gaming arenas, or art galleries, which virtualize their spaces in the Metaverse for various purposes, including monetary gain from virtual services and enhanced user experience. Each Head

converts its real-world physical environment into a digital VRoom through different methods, such as 3D scanning and reconstruction. Similar to MSPs, the set of all Heads is denoted as H = {1, . . . , h, . . . , H}, where h refers to a specific Head and H is their count within the system. VRooms, digital counterparts of the Heads’ physical spaces, offer virtual interaction opportunities to Virtual Room Clients (VRCs). Each VRoom consists of two coupled subsystems: (i) the Virtual Environment (VE), i.e., the rendered 3D scene and interactive logic that VRCs visually perceive and navigate (layout, objects, avatars, and viewpoint updates), and (ii) the Digital Twin (DT) layer, i.e., the synchronized digital representation of physical objects inside the Head’s venue enabled by IoT devices, sensors, and actuators (e.g., cameras/LiDAR/touch sensors and haptics) to provide real-time state alignment and interaction feedback. The set of VRooms

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

TABLE II: Representative Related Work vs. our paper (Focus, Technique, and Key Capabilities) Comp +Comm

Cross -MSP

Incent. /Credit

Imm (VE+DT)

Heterog. VRooms

Nonlinear Scaling

Contract theory + optimization

✓

✓ (VE)

Analytical model + optimization

✓ (VE)

DRL

✓

✓ (VE)

Multi-agent multi-objective RL

✓

✓ (VE)

Blockchain + game-theoretic design

✓

Optimization

✓

DRL (sMDP)

✓

Multi-objective DRL

✓

✓ (VE)

Contextual bandit

✓ (VE)

Double Dutch auction + DRL

✓

✓ (VE)

DRL + shared-credit cooperation

✓

✓ (VE+DT)

✓

Ref.

Goal

Technique

[7]

Maximize Meta-Immersion (QoE) under xURLLC + contract constraints Maximize QoE via attention-aware network resource allocation Maximize long-term VR QoE under interaction-latency constraint Maximize MI while minimizing device energy (Pareto trade-off) Maximize MSP utility via reliable CDC rendering (mitigate stragglers) Maximize data utility while minimizing vehicular energy (AR Metaverse) Maximize long-term provider revenue/acceptance via multi-tier allocation Minimize delay and maximize video quality to meet target QoE Maximize VR streaming QoE under bandwidth via robust FoV tile selection Maximize social welfare while minimizing auction overhead (VR services) Maximize holistic immersion (VE+DT) under multi-MSP budgets via cooperation

[8] [9] [10] [11] [12]

[4], [13]

[14] [15]

[16]

Our paper

Comp+Comm: jointly considers computing and communication. Cross-MSP: cooperation across distinct MSP administrative domains. Heterog. VRooms: explicitly models room types with different base requirements. Nonlinear Scaling: models resource growth with user density/room complexity beyond linear forms.

is defined as V = {v1 , . . . , vh , . . . , vH } explicitly mapping each Head h to its unique VRoom vh . We envision that each type of VRoom (e.g., library and game arena) has unique resource requirements that impact service quality, scalability, and cost, among others. The key VRoom parameters we proposed are shown in TABLE III. The set of DTs available for the Head h is denoted as Dh = {1, . . . , d, . . . , Dh }, where d represents an arbitrary DT within Head h’s venue and Dh is the total number of DTs in h’s VRoom. Since Heads typically lack adequate computational power to render their VRooms and maintain DT synchronization at immersive quality, they depend on MSPs for these services. Each Head is exclusively served by an MSP. We defineP Hm ⊆ H as the set of Heads M served by MSP m, satisfying m=1 |Hm | = H. c) Virtual Reality Clients (VRCs): VRCs are users who access and interact with the Heads’ VRooms in the Metaverse, primarily using HMDs and VR sets. The complete set of VRCs is indicated as C = {1, . . . , c, . . . , C}, with the subset of VRCs linked to the Head h symbolized by Ch ⊆ C. Each VRC dynamically navigates from one VRoom to another based on their interest. To summarize, our environment comprises multiple MSPs that serve Heads, who in turn attract various VRCs to their respective VRooms. This hierarchy reflects practical Metaverse deployment and service provisioning, while enabling tractable control in a multi-provider setting. MSPs allocate virtualized compute and network resources at the Head level, where a common virtual view is rendered for all associated VRCs and where budgets and DT-enabled VRooms are managed, rather than at the individual user level, which aligns with realistic B2B service models. Assuming one VRoom type per Head

ensures a well-defined resource–immersion profile and isolates heterogeneity across environments; venues supporting multiple room types can be equivalently represented by multiple logical Heads or time-varying VRoom profiles. In the following subsections, we present critical resources and quality metrics along with associated costs. B. Immersion, Resource Scaling and Cost Models 1) Immersion: Immersion refers to the degree of presence and engagement experienced by a VRC within the VRoom. Although no standardized metric or list of metrics exists for calculating the immersion of a user inside the Metaverse [26], [27], we envision that an immersive experience involves two primary components: VE quality (i.e., visual quality and motion smoothness of the VRCs inside the VE) and DT fidelity (i.e., DT look, feel and feedback, synchronization including sound, vibration among others). When a Head h requests the rendering services for VRoom vh from MSP m at time t, the immersion experienced by h, (t) denoted as Ih is calculated as a weighted sum (the weights, wq + wf + wa = 1, adjusted according component importance ) of three normalized metrics as: (t)

(t)

Ih = wq Q̂h + wf F̂h + wa Âh

(1)

Here, Q̂h represents the normalized quality of perceptual experience (QoPE) based on VRoom-specific visual and streaming requirements and is defined as: (t)

(t)

Qh = wσ σh + wη ηh

(2)

σh denotes structural similarity index (SSIM) and evaluates image quality (e.g., structure, texture, clarity), while ηh

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

Digital twins (DTs) Sensors and actuators

Digital twins (DTs) Sensors and actuators VE

Library Head

V.Room Clients (VRCs)

Museum Head Local resources (Payment based)

Local resources (Payment based)

Initial budget

Local Metaverse Service Provider (MSP)

Digital twins (DTs) Sensors and actuators

Shop. Mall Head

Game Arena Head

Digital twins (DTs) Sensors and actuators

Indus. Train. Head Local resources (Payment based)

Concert Head

Local resources (Payment based)

Initial budget

Local Metaverse Service Provider (MSP)

V.Room Clients (VRCs)

VE VE

Digital twins (DTs) Sensors and actuators

Care center Head

Art Gallery Head

Fig. 1: The main system model, composed of geographically distributed Metaverse service providers, serving multiple Heads who host virtual Rooms of their physical entity and enhance their clients’ immersion with their DTs synchronization

video multi-method assessment fusion (VMAF) assesses video streaming smoothness and clarity, both are defined in [6] as: (t) (t)−(e3 +e4 ωh ) σh = max e0 , 1 − (e1 + e2 ωh )Bh (3) (t) (t) (t) ηh = min 100, j1 + j2 ωh + j3 Bh + j4 ωh Bh Herein, e0 , . . . , e4 and j1 , . . . , j4 are the coefficients for SSIM and VMAF, ωh represents the average rotation speed of clients (t) in the Head’s VRoom and is considered a fixed parameter, Bh is the bitrate (in Mbps) allocated for Head h (by MSP m) at time t. wσ and wη are prioritization weights that sum to 1 (i.e., wσ + wη = 1), defined per VRoom type (e.g., gaming arenas prioritize VMAF, while art galleries prioritize SSIM). F̂h represents the normalized responsiveness of the VRoom, calculated from the actual frame rate fh , normalized against the VRoom’s specified minimum and maximum frame rates. (t) Finally, Ah captures the (unnormalized) accuracy of DTs within Head h at time t, computed as the harmonic mean (t) (t) of structural accuracy εh , behavioral accuracy βh , and (t) temporal accuracy τh :  (t) (t) (t) 0 if εh = 0 or βh = 0 or τh = 0 (t) 3 Ah = otherwise  1 + 1 + 1 ε

(t) h

(t) β h

(t) τ h

(4) (t) Structural accuracy εh measures how faithfully the DT captures the physical system’s required structure at time t,

defined as the fraction of correctly represented structural features: (t) |Mε | (t) . εh = |Nε | Here, Nε is the set of structural DT features required by the application (e.g., geometry primitives, key mesh points, textures, (t) calibrated dimensions, and pose anchors), and Mε ⊆ Nε is the subset whose DT representation matches the physical reference within an application-defined tolerance (e.g., geometric error ≤ ϵgeo and texture similarity ≥ ϵtex ). (t) Behavioral accuracy βh measures how faithfully the DT reproduces required interaction/behavioral responses, defined as: (t) |Mβ | (t) , βh = |Nβ | where Nβ is the set of behavioral interaction features (e.g., state-transition rules, actuator response modes, contact/haptic (t) response patterns, and dynamic constraints), and Mβ ⊆ Nβ is the subset whose simulated response matches the physical system within tolerance (e.g., response delay ≤ ϵdelay , force/torque error ≤ ϵforce ). (t) Temporal accuracy τh captures DT freshness via an increasing, saturating function of DT update frequency: D

(t)

τh =

h (t) 1 X 1 − e−λfd , Dh

d=1

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

We use the harmonic mean in (4) to reflect that DT fidelity (t) (t) (t) is a weakest-link property: if any of εh , βh , τh is low, overall DT fidelity degrades sharply even when the others are high. The zero case captures a non-functional DT state (no structural/behavioral/temporal alignment), yielding zero DT fidelity contribution to immersion. Finally, in our model (t) βh is a controllable DT service level: increasing it raises DT compute/network cost (shown later) but improves immersion (t) through Âh . In Q̂h , F̂h and Âh normalization is performed via min-max x−xmin scaling (i.e., x̂ = xmax −xmin ) on the VRooms’ minimum and maximum parameters as found in TABLE III. Finally, it is important to note that we do not treat VE and DT as independent modules. They are coupled through both perception and resources: (a) perceptual coupling, because user immersion is computed jointly from VE visual quality/motion and DT fidelity, so degradation in either subsystem reduces the same immersion score; and (b) resource coupling (as shown later), because improving VE quality (e.g., higher (t) (t) bitrate Bh and frame rate fh ) and improving DT fidelity (t) (e.g., higher behavioral accuracy βh and DT synchronization traffic) both increase compute/network requirements and compete under the same MSP budget and nonlinear scaling constraints. Consequently, allocating more resources to VE rendering can reduce feasible DT fidelity and vice versa, which motivates our joint VE–DT immersion-aware optimization. Finally, we note that the immersion weights (wq , wf , wa , wσ and wη ) are VRoom-type parameters that encode application-level priorities rather than optimization variables. They are fixed during training (with values as in TABLE III) and reflect domain semantics (e.g., motion sensitivity in arenas versus visual fidelity in galleries). This modular formulation preserves interpretability and allows heterogeneous immersion profiles without altering the underlying optimization or learning framework. 2) Resource Scalability: Since each Head’s VRoom vh consists of the VE that needs to be rendered and a set of DTs that needs to be rendered and synchronized, it is then of paramount importance to know how these resources’s computational and network requirements will be scaled as more VRCs join the VRoom and/or the visual quality is demanded (i.e., more frame rate is required). Inspired by recent advancements in distributed systems and resource optimization [29], [30], our resource scaling model incorporates non-linear user scaling [29], resource sharing efficiency (i.e., degree of sharing a resource, value between 0-1) [31], user density effects (i.e., how full VRoom affects the resource scaling) [32], [33], aligning with current research in cloud computing and edge environments [34], [35]. The proposed resource scaling model for any resource for any VRoom component is presented as:

!κ # (t) fh · fvmin | {z } Base resource needed    "

(t)

where fd is the update frequency of DT d, and λ characterizes the accuracy reduction rate (uniform across DTs in a VRoom for simplicity). The sets Nε and Nβ are specified at VRoom instantiation and can be extended to incorporate domain-specific fidelity metrics.

Rsy (t) =

(t) Rby ·

(t)

Ch Vhmax

C (t)ϑ · 1 − ι 1 − h

!Λ  

Scale of res. accord. to res. and VRoom nature

(5)

}

Where Rsy (t) represents the scaled resource of type R at time t, (Herein, R ∈ {Comp, N et} represents computational and network resources respectively) for a specific VRoom’s component y, (y ∈ {V E, DT } represent VE and DT) at a specific type of VRoom. From (5), Rsy (t) is composed of two main terms: (t) f (t) The first term, presented by [Rby · ( fv h )κ ] signifies the min base resources of type R required for a VRoom component y with current frame rate relative to the minimum room’s frame (t) rate. Applying Rby in our resources and VRoom components, we get 1) base Computational resource requirements for a and 2) base VE and DT denoted as CompVb E and CompDT b network resources for VE and DT denoted as N etVb E and N etDT b . (t) CompVb E is calculated as: CompVb E (t)

(t)

Pv Ov Av + + Pmax Omax Amax

(t)

where Pv , Ov and Av are the parameters affecting the base resources of VE of VRoom v at t, which are polygon count Pv , number of physical objects Ov and interaction points, Av while Pmax , Omax and Amax are their maximum values fixed across all VRooms V and available in TABLE III. Additionally, the base network resources for VE can be (t) described as the minimum per-VRoom fixed bitrate Bv min : N etVb E

(t)

= Bv min

Similarly, building upon DTs’ requirements discussed in (t) [24], [25], CompDT can be described as: b CompDT b (t)

(t)

Nd SVd U + + d Nmax SVmax Umax (t)

where Nd , SVd and Ud represent the factors affecting the complexity of DT d, and are the number of sensors, the number of DT’s state variables, and DT’s synchronization update frequency. Nmax , SVmax and Umax are their maximum values, unified across all of the DTs D. Moreover, the base network resources for DT can be described as a scaled-down version of VE’s network resources multiplied by a scaling factor φ and the DT’s behavioral accuracy β, therefore N etDT b

(t)

= (φ · Bv min ) · βh

Further, since all of the aforementioned base resources are affected by the smoothness of the visual quality, we multiply f

(t)

them by a visual quality scaling factor ( fv h (t)

)κ representing

min

the ratio of head’s frame rate fh ( one of the decision variables, as explained later) divided by the VRoom’s minimum

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

frame rate fvmin , and κ ∈ {Comp, N et} is a resource-specific exponent that shows how this resource scales with visual smoothness. The second term of Rsy (t) calculation [Ch (t)ϑ · (1 − ι(1 − (t) Ch

( Vh )Λ ))] describes the scaling of resources according to max 1) the resource nature (e.g., resource scaling and sharing capability) and 2) the VRoom’s nature (e.g., VRC count effect on VRoom’s capacity). Herein, ϑ controls the nonlinear scaling behavior as VRC count (Ch ) increases, similar to scaling effects observed in distributed systems, ι represents the resource-specific sharing efficiency factor, capturing how effectively resources can be shared as users increase, comparable to resource optimization approaches in cloud-based rendering systems and Λ models the impact of user density relative to VRoom’s maximum capacity (Vhmax ), reflecting congestion effects. 3) Cost: In the previous section, we explained how the different network and computational resources scale for the VE and DT. In this section, we will utilize the scaled resources across the different Heads belonging to each MSP to calculate the overall resource allocation cost. The total computational cost of an MSP m across all of its heads can be computed as:

TABLE IV: General parameter values Parameter

Value

Nmax , SVmax , Umax e0 , . . . , e4 [6] j0 , . . . , j4 [6] KComp , KNet Pmax , Omax , Amax (t) φ, τh V used sequentially PPO Act/cri. NN PPO gamma PPO batch size PPO learning rate

100, 100, 10 0.65, 0.368, 1.23 × 10−3 , 0.85, 1.23 × 10−3 36.13, −1.66 × 10−2 , 11.62, −6.07 × 10−3 0.01, 0.01 1 × 105 , 100, 10 0.1, 1 Lib., Arena ,Gal., Lib, Gal., Arena, Gal., Lib., Lib. 2 layers of 128 neurons 0.97 128 3 × 10−4

IV. PROBLEM FORMULATION In this section, we formalize the two optimization problems underlying CIVIC. We begin with a non-cooperative benchmark, in which each MSP operates using only its own budget, and then extend it to a cooperative formulation coupled through a shared General Credit Pool (GCP). To keep the notation compact, for every Head h and timestep t, we define the local control vector (t) (t) (t) (t) def uh = Bh , fh , βh , (t)

(t) = Costm Comp

X h∈Hm

Moreover, the total network cost is computed as: (t) Costm = Net

(t)

where Bh is the allocated bitrate, fh is the frame rate, (t) (t) KComp · CompVs E (m, h)(t) + CompDT s (m, h) and βh is the DT behavioral-accuracy level selected from

(t) (t) KN et · N etVs E (m, h) + N etDT s (m, h)

h∈Hm

where KComp and KN et are the unit prices for computational and network resources, respectively. Therefore, the total cost for an MSP m evaluated at any (t) arbitrary timestep t, denoted as Costm can be presented T otal as the summation of computational and network resources as: (t) (t) (t) + Costm Costm = Costm Net Comp T otal

(6)

After introducing the system model, the next section will discuss the problem formulation for both the non-cooperative and cooperative scenarios. TABLE III: Parameters for different VRooms Parameter

LIBRARY

ARENA

GALLERY

Bvmin , Bvmax fvmin , fvmax εvmin , εvmax βvmin , βvmax ωh , κComp , κN et wσ , wη ϑComp , ϑN et Λ, ι Pv , Ov , Av Nd , SVd , Ud vhmax wq , wf , wa rarr , rdep

20, 25 30, 60 0.6, 1.0 0.5, 1.0 400, 1.1, 0.5 0.5, 0.5 0.7, 0.9 0.8, 0.6 2e5, 50, 5 50, 30, 2 10 0.33 0.4, 0.7

30, 50 60, 120 0.5, 1.0 0.5, 1.0 720, 1.1, 0.5 0.3, 0.7 0.85, 0.85 0.8, 0.5 5e5, 200, 30 200, 100, 10 100 0.33 0.4, 0.7

25, 35 30, 60 0.8, 1.0 0.3, 1.0 400, 1.1, 0.5 0.7, 0.3 0.7, 0.8 0.7, 0.8 3e5, 20, 15 30, 20, 1 10 0.33 0.4, 0.7

a finite fidelity grid. For each Head h, let Bvh denote the admissible behavioral-fidelity grid associated with its VRoom type; throughout this paper, we use a uniform spacing ∆β = (t) 0.05 between consecutive levels. We also let Budgetm denote the remaining budget available to MSP m at the beginning of slot t. A. Problem P : Non-cooperative setting The non-cooperative problem serves as the baseline case in which MSPs act independently over the horizon T = {1, . . . , T }. At each slot, MSP m observes the set of Heads (t) requesting service and chooses uh for every served Head h ∈ Hm . Each Head request specifies its VRoom type, the number of VRCs, the number of DTs within that VRoom and the required immersion level. A Head-level request is considered satisfied when its attained immersion meets or exceeds the common threshold Ithreshold . Moreover, each MSP operates under a fixed budget, and serving requests consumes budget over time; hence, MSPs must tune their allocation to control spending while maximizing the number of satisfied requests across T . To avoid starvation among Heads attached to the same MSP, we adopt an all-or-nothing MSP-level service rule: a request at time t is considered fulfilled for MSP m only when all of its active Heads are simultaneously are served at or above threshold. The initial condition is (1) (0) Budgetm = Budgetm for every MSP m. To translate the above service rule into a compact mathematical form, we first introduce a Head-level satisfaction indicator and then use it to express the MSP-level objective. Specifically, let h i (t) def (t) (t) zh = Irequest (h, t) 1 Ih uh ≥ Ithreshold .

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

Therefore, the non-cooperative optimization problem can be written as follows: P :

T X M X

max (t)

{uh }

L(t) m

(7)

t=1 m=1

Subject to: (t) Hm,active = {h ∈ Hm : Irequest (h, t) = 1},

∀m ∈ M, ∀t ∈ T (c1)

Irequest (h, t) ∈ {0, 1},

∀h ∈ H, ∀t ∈ T (c2)   h i  X  (t) (t) (t) L(t) zh = |Hm,active | , m = 1 |Hm,active | > 0 1 (t)

h∈Hm,active

∀m ∈ M, ∀t ∈ T (c3) (t) Costm ≤ Budget(t) T otal m,

∀m ∈ M, ∀t ∈ T

(t) (t) Bh ∈ Z, Bvmin ≤ Bh ≤ Bvmax , ∀h ∈ H, ∀t ∈ T (t) (t) fh ∈ Z, fvmin ≤ fh ≤ fvmax , ∀h ∈ H, ∀t ∈ T (t) βh ∈ Bvh , ∀h ∈ H, ∀t ∈ T (t) m Budget(t+1) = Budget(t) m m − CostT otal ,

(c4) (c5) (c6) (c7)

∀m ∈ M, ∀t ∈ T \ {T } (c8) The constraints can be read naturally from request activation to budget evolution. Constraint (c1) defines the active request set of MSP m at slot t, while (c2) enforces the binary nature of the request indicator itself. Constraint (c3) then captures the key structural choice in CIVIC: the objective counts an MSP as successful at slot t only if every active Head attached to it is simultaneously satisfied, which creates coupling across the Heads served by the same MSP. Constraint (c4) next enforces local affordability by requiring the total cost incurred by MSP m at time t to remain within its currently available budget. Constraints (c5) and (c6) bound the VE control variables within the admissible VRoom-specific service ranges, whereas (c7) constrains the DT behavioral-fidelity control to a finite grid with step size ∆β = 0.05. Finally, constraint (c8) makes the inter-temporal coupling explicit by stating that expenditure at slot t reduces the remaining budget available at slot t+1. For tractability, we do not optimize every DT-related control variable. In particular, although an ideal allocation policy could (t) also optimize the DT update frequency τh through λfd and (t) the structural-accuracy term εh , we assume these quantities are fixed in advance through service-level agreements between Heads and MSPs and therefore treat them as exogenous parameters. B. Problem P̄ : Cooperative setting While Problem P treated MSPs as isolated entities, which by design limits the collective potential of MSPs to cooperate and fulfill more requests particularly when MSPs operate with varying initial budgets and face fluctuating dynamic loads across their service areas. In that case, some MSPs might

experience client surges and fail to meet demands, while others with minimal loads refrain from offering assistance. Hence, we now extend our problem P to a cooperative setting P̄ in which MSPs belong to the same administrative domain and can redistribute unused credits through a General Credit Pool (GCP). The GCP is a centralized resource-sharing system that enables MSPs to dynamically redistribute computing capacity through a deposit-withdraw framework. Using General Credits (GCs) as standardized units, each equivalent to a monetary value (e.g., 1 GC = 1 USD). MSPs supports their peers by purchasing GCs on behalf of other MSPs and depositing these GCs into the pool, which resource-constrained MSPs can later withdraw to acquire general computational resources (e.g., 1 GC = 2 CPU cores for 1 hour). This mechanism eliminates the complexities of direct credit transfers while allowing MSPs serving high-demand VRooms to secure additional capacity seamlessly. By supporting prepurchasing GCs ahead of time, the GCP also reduces on-demand costs and allocation latency. The GCP supports different credit valuation mechanisms depending on the administrative relationship among MSPs. In homogeneous cooperation, all MSPs operate under a single organization (e.g., a large cloud provider managing multiple regional MSPs). In this case, General Credits (GCs) act purely as internal accounting units: if MSP A deposits 1 GC into the GCP, MSP B, belonging to the same organization, can later withdraw exactly 1 GC with no loss of value, since coordination costs and financial reconciliation are internalized by the same authority. In heterogeneous cooperation, MSPs belong to different companies (e.g., MSP A from Company X and MSP B from Company Y) that voluntarily form a coalition to share resources during demand fluctuations. In this setting, GCs represent inter-organizational resource obligations rather than internal credits. Because resource contributions and withdrawals typically occur at different times and often under asymmetric demand conditions, the effective value of a GC may vary. For example, if MSP A contributes 1 GC during a lowdemand period, MSP B may later withdraw resources during a peak-demand period, in which case the withdrawal may correspond to less than 1 GC of effective capacity (e.g., 0.95 GC), reflecting coordination overhead, delayed reciprocity, and scarcity-driven valuation. In P̄ , we focus exclusively on the homogeneous cooperation model to isolate the impact of cooperative resource sharing without introducing interorganizational pricing dynamics. Extending the GCP to fully dynamic, demand-aware credit pricing across heterogeneous MSP coalitions is a natural direction for future work. The main modeling choice in CIVIC for GCP donations is surplus-only donation. Each MSP first determines its local VE/DT allocation and only afterward donates a fraction of the remaining local surplus to the GCP. This ordering separates the local immersion-control decision from the inter-MSP coupling and yields a clearer pool-evolution model. The local initial (1) (0) condition remains Budgetm = Budgetm , while the pool (1) starts empty, i.e., GCP = 0. To express this mechanism compactly, we next define the local surplus and deficit quan-

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

def

tities. For compactness, let [x]+ = max{0, x} and define (t) (t) def m Sm , = Budget(t) m − CostT otal + def (t) (t) Wm = Costm − Budget(t) T otal m +.

C. Problems complexity

(t)

where Sm denotes the post-allocation local surplus of MSP m (t) and Wm denotes its deficit before GCP support. Using these auxiliary quantities, the cooperative optimization problem can be written compactly as follows: P̄ :

T X M X

max (t)

(t)

{uh },{δm }

L(t) m

(8)

t=1 m=1

This formulation preserves the non-cooperative constraints of Problem P and augments them with the following cooperative constraints: (t) 0 ≤ δm ≤ 1,

M X

(c9)

∂ 2 σh

(c10)

∂(Bh )2

(t)

∀m ∈ M, ∀t ∈ T

= −(e1 + e2 ωh ) · (−(e3 + e4 ωh )) (t)−(e3 +e4 ωh )−2

(t) Wm ≤ GCP (t) ,

· (−(e3 + e4 ωh ) − 1) · Bh

∀t ∈ T

(c11)

m=1

0 ≤ GCP (t) ≤

Both optimization problems P and P̄ are non-convex for two complementary reasons. First, their feasible sets are non(t) (t) convex because several decision variables, namely Bh , fh , (t) and the grid-based βh , are restricted to discrete admissible sets. Second, even under the standard continuous relaxation of these discrete controls to their enclosing intervals, the resulting objective/constraint functions remain non-convex. To show the latter point, we inspect the Hessians of representative continuous components. Recall that a twice-differentiable function is convex only if its Hessian matrix is positive semidefinite everywhere in its domain. Firstly, the SSIM function in (3) contains the term (t)−(e3 +e4 ωh ) Bh with negative exponent. The second derivative is: (t)

∀m ∈ M, ∀t ∈ T

(t) (t) (t) Dm = δm Sm ,

M X

Budget(0) m ,

∀t ∈ T ,

GCP (1) = 0

m=1

(c12) (t) (t) Budget(t+1) = Sm − Dm , m

∀m ∈ M, ∀t ∈ T \ {T } (c13)

GCP (t+1) = GCP (t) +

M X m=1

(t) Dm −

M X

(t) Wm ,

Since (e3 + e4 ωh ) > 0 and (e1 + e2 ωh ) > 0 from the model coefficients, this second derivative is negative, proving nonconvexity. Secondly, the resource scaling function in (5) mixed partial (t) (t) derivative with respect to fh and Ch in the Hessian is: ∂ 2 Rsy (t) (t) (t) ∂fh ∂Ch

κ (t)κ−1 (t)ϑ−1 (t) ·f · ϑ · Ch · g ′ (Ch ) fvκmin h κ (t)κ−1 (t)ϑ (t) + Rby · κ · fh · Ch · g ′′ (Ch ) fvmin

= Rby ·

∀t ∈ T \ {T }

m=1

(c14) The cooperative constraints can likewise be read in sequence. Constraint (c9) restricts the donation decision to a valid fraction in [0, 1]. Constraint (c10) then converts that fraction into an actual donation amount by applying it only (t) to the post-allocation local surplus Sm , so MSPs donate only after satisfying their own local spending decision. Constraint (c11) enforces the core pool-feasibility rule, namely that the aggregate deficit requested from the GCP at any slot cannot exceed the credits currently available in the pool. Constraint (c12) bounds the GCP state and specifies the initial condition GCP (1) = 0. Constraint (c13) then updates the remaining local budget by subtracting the donated amount from the current surplus. Finally, (c14) governs the shared GCP evolution by adding all donations and subtracting all deficits covered at the current slot. This explicit surplus/deficit accounting makes the cooperative coupling mathematically transparent while preserving the all-or-nothing fulfillment rule of P . Although credit sharing can substantially improve systemwide performance, suboptimal joint GCP-donation and allocation strategies may still degrade performance below noncooperative levels, as later shown in Section VI. Relative to Problem P , Problem P̄ is more complex because of the ad(t) ditional decision variable δm and the inter-temporal coupling introduced by the GCP mechanism. This added complexity further motivates the use of advanced solution methods such as Deep Reinforcement Learning (DRL).

(t)

(t)Λ−2

Ch is negative when 0 < The term g ′′ (Ch ) = −ι Λ(Λ−1) VhΛmax Λ < 1, creating negative eigenvalues in the Hessian matrix and establishing non-convexity. (t) Thirdly, although βh is selected from a discrete grid in the actual formulation, its continuous relaxation within the interval [βvmin , βvmax ] remains non-convex. In particular, the harmonic mean accuracy function in (4) has second derivatives of the form: (t)

∂ 2 Ah (t)

∂(βh )2

(t)−3

6βh (t)−1

(εh

(t)−1

+ βh

(t)−1 3

+ τh

−

)

(t)−4

6βh (t)−1

(εh

(t)−1

+ βh

(t)−1 2

+ τh

)

which yields negative values for typical parameter ranges, confirming non-convexity through negative Hessian eigenvalues. The mathematical complexity classifies both problems as non-convex mixed-integer nonlinear programming (MINLP) instances, rigorously justifying the DRL approach over traditional convex optimization methods. V. THE PROPOSED DRL-BASED SOLUTIONS The resource-allocation problems formulated in the previous section are non-convex MINLPs and present significant optimization challenges due to their NP-hard nature (see Appendix), dynamic environment characteristics (e.g., varying

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

Digital twins (DTs) Sensors and actuators Computing

Storage

Digital twins (DTs) Sensors and actuators

Rendering

Local resources (Payment based)

Initial budget

Local Metaverse Service Provider (MSP)

Museum Head Local resources (Payment based)

Credit Deposit

Library Head

V.Room Clients (VRCs)

General Credit (GC)

Global Credit Pool (GCP) Automatic Credit withdrawal

V.Room Clients (VRCs)

Initial budget

Local Metaverse Service Provider (MSP)

Digital twins (DTs) Sensors and actuators

Game Arena Head

Shop. Mall Head Digital twins (DTs) Sensors and actuators

Indus. Train. Head Local resources (Payment based)

Intelligent Agent

Local resources (Payment based)

Initial budget

Local Metaverse Service Provider (MSP)

Digital twins (DTs) Sensors and actuators

Care center Head

...

Heads Load

GCP state MSPs Rem. Budgets

Concert Head

V.Room Clients (VRCs)

VE VE

Digital twins (DTs) Sensors and actuators

Art Gallery Head

MSPs Requests progress

Fig. 2: The proposed Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse (CIVIC)

VRC counts from one timestep to another), long-term temporal coupling (i.e., allocation at t affects the remaining budget at t + 1), heterogeneous VRoom requirements, and the need for real-time decision making. The non-cooperative problem (Problem P ) requires each MSP to jointly optimize mixed (t) (t) (t) decision variables Bh , fh , and βh for multiple Heads while adhering to budget and immersion constraints. The cooperative problem (Problem P̄ ) adds a further coupling layer through the (t) shared-pool donation variable δm , requiring MSPs to balance local service quality with system-wide support. Traditional optimization solvers such as BARON face difficulty because of the system dynamics, real-time requirements, and the rapid growth in dimensionality as the number of MSPs and Heads increases.

Therefore, to tackle these challenges, we develop DRL-based solutions for both P and P̄ , leveraging a centralized DRL framework capable of learning optimal resource-allocation and cooperation policies through continuous interaction with the environment. In the following sections, we first introduce the DRL framework and the selected algorithm. Then, in the subsequent subsections, we start by formulating P as a Markov Decision Process (MDP) and evaluating the resulting agent through a comprehensive set of experiments. We then extend this approach to the cooperative setting, formulating P̄ as an MDP that not only allocates resources across individual MSPs and their associated Heads but also facilitates inter-MSP cooperation through the GCP, as illustrated in Figure 2.

A. DRL algorithm In contrast to traditional machine and deep learning solutions, which often require building knowledge based on a training dataset to make predictions of values or classes, Reinforcement Learning (RL) and its deep learning-assisted counterpart, DRL, take a different approach and serve a different purpose. DRL is often recognized as an advanced heuristic solution that specializes in solving problems requiring longterm optimization goals by optimally selecting sequential actions under uncertainty. For that to work, DRL algorithms collect millions and even billions of interactions with the simulated environment. DRL is theoretically grounded in Markov Decision Process (MDP), which is defined as ⟨S, A, P, R, Γ⟩. In this framework, the agent continuously observes the environment state S, selects an action A, and receives a reward R discounted by Γ, along with the updated state S′ . The transition from S to S′ occurs probabilistically, as defined by P. After training the agent across multiple episodes, it learns a policy π, effectively associating states with actions that maximize cumulative rewards. Among the different DRL algorithms, we selected Proximal Policy Optimization (PPO) [36]. PPO offers several advantages that make it our optimal choice. First, it provides stable low-dimensional control outputs that can be projected onto the admissible bitrate, frame-rate, and behavioral-fidelity grids used by our environment. Moreover, it incorporates a clipped surrogate objective that balances exploration and exploitation, preventing destructive policy updates and leading to stable and reliable training. PPO is also highly amenable to parallelization, allowing multiple workers or environments to collect experience simultaneously, which

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

significantly accelerates training and improves sample diversity. Additionally, PPO is recognized for its sample efficiency and ease of implementation, while achieving comparable or superior performance in practice. In what follows, we will present the formulation of our problem as an MDP by defining the environment state, actions, and rewards. B. Non-Cooperative Starting with our non-cooperative scenario, we propose a centralized agent that tunes the various parameters of all Heads belonging to all MSPs. We begin by introducing the state space, followed by the agent’s action and reward, which are used to guide the agent. In our MDP formulation, a vector is presented by a bold letter. The state space, denoted as S is described as: (t)

(t)

S = {s1 , s2 , s3 , s4 , s5 } (t)

where s1 = {

Budget(t) m (0) Budgetm

(9)

: ∀m ∈ M} conveys the set of re(t)

Pt′ =t

′

L(t )

=1 m maining budget percentage per each MSP, s2 = { t′|N : m| ∀m ∈ M} signifies the percentage of requests fulfilled so far by each MSP and |Nm | is the count of requests needed to be

(t)

fulfilled by m, s3 = { Vh h : ∀h ∈ m, ∀m ∈ M} describes max the number of clients in each VRoom as a percentage of (t) VRoom’s maximum capacity. s4 = {Irequest (h, t) : ∀h ∈ H} (t) shows the current active request indicators and finally s5 = Tt indicates a progress till the episode ends. Similarly, the action space denoted as A, can be described as: (t) (t) (t) A = {a1 , a2 , a3 } (10) (t)

(t)

where: a1 = {Bh : ∀h ∈ Hm , ∀m ∈ M} represents (t) (t) the bitrate allocation for each h, a2 = {fh : ∀h ∈ Hm , ∀m ∈ M} signifies the frame rate assignments for each (t) (t) h, and a3 = {βh : ∀h ∈ Hm , ∀m ∈ M} indicates the behavioral-accuracy level selected for the DT in h’s VRoom. (t) In implementation, the PPO output corresponding to βh is projected onto the nearest admissible level in the discrete grid Bvh before being applied to the environment. Since P focuses on maximizing the number of fulfilled (t) requests (i.e., requests with Ih ≥ Ithreshold ) under constrained MSP budgets, the reward function at time t, denoted by Rt , is designed to reward efficient threshold-meeting allocations while penalizing failed requests that arise either from infeasible actions or from budget depletion that prevents service. Therefore, we formulate Rt as: X X (t) Rt = wimm · ϕ(Ih ) + m∈M h∈H(t)

m,active

| wfin ·

Immediate reward at (t ̸= T )

T X X

} (t)

1[0 < Ih ≤ Imax ]

t=1 m∈M h∈H(t)

m,active

Terminal reward at (t = T )

}

(11)

where the first term in (11) represents the immediate reward given at anytime (t ̸= T ) and is the allocation efficiency score of all MSPs and is calculated by aggregating the output of the piece-wise efficiency function at ϕ(.) across all of the active Heads Immersions’ scores at t as:   1.5    (t) I −Ithreshold (t) ϕ(Ih ) = 0.5 − min 0.3, h Ithreshold     −1

(t)

if Ithreshold ≤ Ih ≤ 110% · Ithreshold (t) if Ih > 110% · Ithreshold (t)

if Ih < Ithreshold

(t)

Here, the ϕ(Ih ) outputs a higher score if the attained immersion is higher than the immersion threshold by 10% at the max, while reducing the efficiency score as this overallocation increases. On the other hand, the second term in (11) represents the terminal reward which encourages the agent to gradually improve the number of satisfied requests by first teaching it to do as much requests as possible (under or over immersion allocation) and avoid MSPs failing and is calculated by aggregating the binary function output across all (t) requests done in the whole episode as 1[0 < Ih ≤ Imax ]. Both immediate and terminal rewards are multiplied respectively by weights wimm and wf in , which are chosen experimentally. C. Cooperative MSPs The cooperative problem P̄ introduces significant complexity compared to the non-cooperative setting, requiring careful consideration of the appropriate solution methodology. While distributed multi-agent reinforcement learning approaches such as Centralized Training with Decentralized Execution (CTDE) offer advantages in terms of scalability and agent independence, they are not well-suited for our cooperative scenario due to several key factors. First, the cooperative setting requires global awareness for optimal decision-making. Each MSP must monitor not only its local state but also the GCP state, peer MSP demands, and resource depletion patterns across the entire system. This global information is essential for MSPs to make informed decisions about resource allocation timing, donation strategies, and withdrawal patterns to optimize their ability to fulfill their total request quotas Nm . Second, the GCP mechanism necessitates centralized oversight to enforce global constraints. Critical system-wide constraints PM (t) such as m=1 Wm ≤ GCP (t) require global coordination to ensure system feasibility. Additionally, optimizing donation timing across multiple MSPs requires synchronized behavior that is difficult to achieve through independent agents. Third, our unified administrative control structure eliminates competitive dynamics typically present in multi-agent systems. Since all MSPs operate under the same administrative entity with aligned objectives, the system optimization naturally focuses on the true global optimization problem rather than seeking Nash equilibria among competing agents. Finally, the manageable scale of our system (3-7 MSPs) makes centralized control computationally feasible while ensuring optimal coordination. The central administrative entity that manages MSPs can deploy and operate the centralized DRL model, enabling comprehensive control over resource orchestration and strategic cooperation.

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

Building upon the non-cooperative state space S, the cooperative environment requires additional state information to capture the dynamics of the GCP and system coordination status. The extended state space S̄ incorporates two critical (t) components: s6 , which indicates the available budget at time (t) (t) t for GCP (i.e., s6 = GCP (t) ), and s7 , a stuck counter variable used to determine whether the system has entered a halting state (i.e., a state where no MSP needs to complete additional requests because all requests are finished, or the budget has been exhausted, and the GCP budget is depleted or insufficient to support any of the MSPs). Thus, the updated state is defined as: (t)

(t)

S̄ = {s1 , s2 , s3 , s4 , s5 , s6 , s7 }

(12)

process. At each timestep, the number of arriving and departing VRCs for each VRoom is independently drawn from Poisson distributions with rates rarr = 0.4 and rdep = 0.7 respectively. This results in a time-varying room occupancy that captures natural user movement across VRooms and enables the evaluation of cooperative resource management under dynamic and fluctuating demand conditions. Other traffic generation models (e.g., bursty, time-correlated, or eventdriven processes), as well as real-world datasets capturing attendance dynamics of online events such as virtual concerts or live sports streams, can be readily incorporated without modifying the proposed optimization or learning framework. A. Non-cooperative setting (P )

(t)

Moreover, the action space adds another action vector a4 , (t) (t) where a4 = {δm : ∀m ∈ M} is the donation fraction applied to each MSP’s post-allocation surplus at time t. Therefore (t) (t) (t) (t) Ā = {a1 , a2 , a3 , a4 } (13) (t)

We note that the choice of the donation vector a4 directly affects the residual local budget through (c13) and the shared cooperative state through (c14). Hence, donation decisions influence future request feasibility even when they do not immediately alter the current slot’s immersion values. The reward function remained the same, encouraging the satisfaction of more requests. VI. PERFORMANCE EVALUATION & DISCUSSION In this section, we present a comprehensive evaluation of our proposed solutions for both P and P̄ . We begin by describing our experimental setup, followed by a detailed analysis of the performance characteristics and comparative evaluation against baseline approaches. Our evaluation environment for the non-cooperative setting (P ) comprises multiple MSPs serving different Heads, each hosting various VRoom types with heterogeneous resource requirements. We consider an ideal range of 1-5 MSPs, where each MSP manages one Head with VRooms types as specified in TABLE IV. Each MSP is tasked with fulfilling a predetermined number of fixed requests, unified across all MSPs (Nm ∈ {50, 70, 90}). Additionally, the immersion threshold, Ithreshold , was unified across all MSPs for consistency across experiments. Furthermore, MSPs’ budgets are equally distributed between MSPs to simulate independent operation without collaboration, and the number of VRCs in VRooms was fixed across timesteps. On the other hand, the cooperative setting (P̄ ) implements more challenging and realistic operational conditions, reflecting collaborative scenarios with a higher number of MSPs (i.e., 3-7 MSPs) and dynamic requests. Moreover, in this cooperative environment, a more challenging and unbalanced budget scenario is used to simulate realistic operational conditions, where MSPs must collaborate under resource constraints. Furthermore, to model dynamic user participation, we initialize each VRoom with a non-zero number of active VRCs and subsequently employ a double-Poisson arrival–departure

After discussing the experimental setup, we assess the performance and efficiency of our proposed DRL solution for problem P . Initially, we analyzed the learning behavior and convergence characteristics of the DRL agent trained with a unified minimum immersion threshold of 85%, as presented in Figure 3. The same figure illustrates the total reward convergence across all MSPs, with convergence curves indicating an increase in difficulty as both the number of MSPs and requests increase. Following the agent’s acquisition of the optimal policy, we conducted three experiments to benchmark its performance. Firstly, we compare our DRL-based solution against several baseline approaches. Our baseline comparisons includes multiple resource allocation policies such as: Saving, which demonstrate a cost-conserving policy that allocates minimal resources to maximize the number of fulfilled requests, the Average, which consistently selects average resource allocations for all requests, the Max policy, which invariably assigns maximum resources to each request. The MyopicOptimal solution employs a computationally expensive greedy optimization approach that makes locally optimal decisions at each timestep. Specifically, at each time t, it: (1) Evaluates (t) (t) (t) all feasible resource allocations (Bh , fh , βh ) that satisfy constraints (c1)–(c8) for all Heads, (2) Selects the allocation (t) that achieves the immersion threshold Ih ≥ Ithreshold , (3) Among solutions that meet the threshold, choose the one with the minimum cost to save budget. The results in Figure 4 present the immersion levels achieved by each solution across configurations of MSPs with request volumes. The analysis reveals that our DRL solution consistently tracks the optimal allocation policy across all experimental conditions. In contrast, the saving policy under-provisions resources, yielding suboptimal immersion levels, while the average policy shows marginal improvement over the saving approach but still falls short of adequate provisioning. Conversely, the max policy over-provisions resources, resulting in unnecessarily high immersion levels that would adversely impact MSPs’ budgets in the long term. Secondly, Figure 5 presents our second and third experiments. In Figure 5(a) we present our second experiment, which evaluates the system’s ability to manage resource allocation as the number of requests increases under a fixed budget. Initially, the agent was trained to provide 100% immersion

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

2000

6000

4000

Training Episodes

Total Reward fulfillment (%)

100

Total Reward fulfillment (%)

100

8000

2000

4000

6000

Training Episodes

8000

2000

4000

6000

Training Episodes

8000

Fig. 3: Reward Convergence for non-GCP (non-cooperative) DRL solution, while increasing number of requests and number of MSPs

Fig. 4: Comparing our non-GCP (non-cooperative) DRL-based solution to other Baselines

Max DRL Immersion (%)

Immersion (x100%)

0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55 100

110

120 130 Requests (%)

140

150

((a)) Fixing budget while increasing number of requests Max DRL Immersion (%)

100

for all requests, and we systematically increased the request volume by 10% in each subsequent trial. The results show that our agent demonstrates adaptive behavior, successfully handling up to 150% of the original request load while maintaining an average immersion level of 60%. This indicates that the system can gracefully manage increased demand by reducing service quality rather than failing. In Figure 5(b), we present our third experiment, where we assess the system’s resilience in accepting all given requests with the highest possible immersion level while reducing the budget in each trial, simulating a poor initial budget allocated to the MSPs. In that experiment, the system was initially trained to achieve 100% of immersion for all of the requests, and the total price was recorded before being progressively reduced by 5%. The results show that the agent was able to continue serving all of the requests, but with a sublinear drop in immersion value.

Immersion (%)

90 80

B. Cooperative setting (P̄ )

After assessing our DRL-based solution solving problem P , we outline the different experiments used to assess our proposed solution to the cooperative setting P̄ . To better assess our DRL with GCP solution (Hereafter, referred to as DRL-GCP), we benchmark its performance against two sets of solutions: the first set includes some of the non-GCP baselines introduced in the previous sections (i.e., Average, Max, and Random), while the second set comprises their improved counterparts with our GCP mechanism, such as Max-GCP, Avg-GCP, and Random-GCP. The second set of policies works with their intended behaviour while considering GCP dynamics. For example, Max-GCP policy dictates 1) the allocation of the maximum resources for all Heads (as in non-

60 50 40 30 100

Budget (%)

((b)) Fixing Number of requests while reducing the budget

Fig. 5: Assessing the DRL-based Solution tolerance

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

Requests Completion (%) (Higher = better)

100

0 3 MSPs Avg Avg-GCP

5 MSPs Max Max-GCP

7 MSPs Random Random-GCP

DRL DRL-GCP

((a)) 100 Timesteps

Requests Completion (%) (Higher = better)

100

0 3 MSPs Avg Avg-GCP

5 MSPs Max Max-GCP

7 MSPs Random Random-GCP

DRL DRL-GCP

((b)) 150 Timesteps

Requests Completion (%) (Higher = better)

100

0 3 MSPs Avg Avg-GCP

5 MSPs Max Max-GCP

7 MSPs Random Random-GCP

DRL DRL-GCP

((c)) 200 Timesteps

Fig. 6: Assessments of the Cooperative Control of the MSPs

GCP Max policy) and 2) depositing the maximum possible credit to the GCP every timestep. Moreover, if no resource credit is available at any MSP to serve its Heads, the required resource credits for that MSP are automatically withdrawn from the GCP, and no resource deposit is made from that MSP at that timestep. In this section, we create a total of three experiments as follows: 1) Experiment 1: Evaluating Resource allocation: The first experiment evaluates the resource allocation performance of the proposed solutions and baseline methods across multiple metrics: the requests completion percentage, indicating total requests done as illustrated in Figure 6, the total number of served clients by the requests as shown in Figure 7, the

requests fulfillment percentage (indicating how many of the done requests satisfy required immersion levels) depicted in Figure 8, the total cost presented in Figure 9, and fairness measures including the distribution of completed requests among MSPs as detailed in TABLE V and TABLE VI. We remind that in this experiment, each MSP receives multiple requests from their Heads across the time horizon. The requirements of these requests may vary depending on the number of VRCs that need to be served in each Head’s VRoom. The MSP can deny/drop any requests if it does not have enough resources and cannot withdraw them. The requests continue to come into the system, and we measure the performance after 100, 150, and 200 timesteps of these dynamic requests. Additionally, while increasing the number of MSPs adds an extra layer of complexity to the problem, as more MSPs need to collaborate, it can improve the completion ratio of requests. Starting with request completion results in Figure 6, we can observe the different completion percentages of the different baselines across increasing numbers of MSPs and requests as shown in subfigures Figure 6 (a,b, and c). The main observation of these subgraphs is summarized as follows: firstly, the GCP cooperation mechanism, if not managed correctly, would not increase the requests completion rate; in other words, baselines with GCP can be worse than the non-GCP solution as seen for Max vs Max-GCP in Figure 6(a)’s 3 MSP configuration and Figure 6(c)’s 7 MSP configuration among others. Secondly, across all subgraphs, we observe the dominance of non-GCP DRL among the non-GCP baselines, and similarly, DRL-GCP dominates among the GCP baselines, resulting in fewer dropped requests from MSPs to Heads. Thirdly, although some baseline methods perform similarly to DRL in certain cases, the disparity tends to grow as scenarios become more challenging, particularly with a higher number of requests, as observed in the comparisons of Avg versus DRL and Avg-GCP versus DRL-GCP across all subgraphs. From the same graph, DRL-GCP on average achieves 12-36% higher request completion than all baselines. While Figure 6 focused on the number of requests to represent performance, Figure 7 expands this view by highlighting the number of VRCs served by each solution in comparison to the random non-GCP solution. Positive bars indicate improvement, while negative bars reflect inferior performance to non-GCP random. This figure reveals a more pronounced disparity between the solutions, as suboptimal policies may accept many requests that serve fewer VRCs. In contrast, other solutions may accept fewer requests with a greater number of VRCs. Notably, the DRL-GCP solution achieves an impressive 20-60% increase in the number of clients served on average. Unlike the previous two figures, Figure 8 solely focuses on the fulfillment percentage of requests, with comparisons also made against the non-GCP random solution. The significant observation from this figure reveals that the 200-timestep, three-MSP configuration appears to represent the most demanding scenario, as evidenced by the minimal improvement relative to the random solution across all configurations. This occurs because the limited collective budget of the reduced number of MSPs proved insufficient to accommodate the

-10 Avg Max-GCP

-20 3 MSPs

Max Random-GCP

DRL DRL-GCP

5 MSPs

Avg-GCP

7 MSPs

40 30 20 10 0 -10 -20 -30

Avg Max-GCP

-40

3 MSPs

((a)) 100 Timesteps

Max Random-GCP

DRL DRL-GCP

5 MSPs

Avg-GCP

Served Clients % (Comp to Random) (Higher = better)

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

7 MSPs

40 30 20 10 0 -10 -20 -30

Avg Max-GCP

-40

3 MSPs

((b)) 150 Timesteps

Max Random-GCP

DRL DRL-GCP

5 MSPs

Avg-GCP

7 MSPs

((c)) 200 Timesteps

Fig. 7: Comparison of percentage of Served clients compared to Random

7 MSPs

5 MSPs

3 MSPs

3 MSPs 3 MSPs Avg Max-GCP

-10

Max Random-GCP

DRL DRL-GCP

10 20 30 Fulfilled Requests % (Comp to Random) (Higher = better)

Avg-GCP

Avg Max-GCP

-10

((a)) 100 Timesteps

Max Random-GCP

DRL DRL-GCP

10 30 Fulfilled Requests % (Comp to Random) (Higher = better)

Avg Max-GCP

Avg-GCP

-20

((b)) 150 Timesteps

-10

Max Random-GCP

DRL DRL-GCP

10 20 30 40 50 60 Fulfilled Requests % (Comp to Random) (Higher = better)

Avg-GCP

((c)) 200 Timesteps

Fig. 8: Comparison of percentage of fulfilled requests compared to Random Avg Avg-GCP

Max Max-GCP

Random Random-GCP

400

DRL DRL-GCP

Avg Avg-GCP

Max Max-GCP

Random Random-GCP

400

DRL DRL-GCP

350

Total Cost ($)

350 300 250 200

Total Cost ($)

400

300 250

300 250 200

200 150 100 3 MSPs

150

150 5 MSPs

((a)) 100 Timesteps

7 MSPs

3 MSPs

5 MSPs

7 MSPs

100 3 MSPs

((b)) 150 Timesteps

Avg Avg-GCP

Max Max-GCP

Random Random-GCP

5 MSPs

DRL DRL-GCP 7 MSPs

((c)) 200 Timesteps

Fig. 9: Comparison of percentage of Total Cost ($)

substantial volume of requests. Moreover, in virtually all cases, our DRL solutions outperformed other variants. Additionally, across nearly all configurations, the average policy solution consistently achieved the second-best performance. Overall, our DRL-GCP solution achieved fulfillment rates that were 2370% higher than the competing methods, underscoring their effectiveness in managing requests. Figure 9 illustrates the total expenses associated with each solution, which is the cost incurred by all MSPs on their Heads. A key takeaway is that among the non-GCP solutions, the option with the lowest monetary cost is the preferred choice. On the other hand, in algorithms that incorporate the GCP, a lower expense may not be deemed optimal, as a higher expense might reflect a greater degree of collaboration among the MSPs, where some MSPs fund others in addition to fulfilling their own needs. In the exact figure, we can see that non-GCP DRL had the lowest cost among the other baselines, while DRL-GCP had a competitive price compared to the other GCP-based baselines. Finally, in cooperative scenarios, while multiple MSPs work together to increase the number of completed and fulfilled requests, it is crucial to ensure that this cooperation remains balanced and effective. Specifically, all MSPs should serve

requests without one MSP dominating in terms of volume. To measure this, we use two metrics across all solutions and baselines. The first is the range, defined as the difference between the MSP with the most completed requests and the one with the fewest. The second is the Gini coefficient, which measures equality among MSPs and ranges from 0 (perfect equality) to 1 (maximum imbalance, where one MSP serves all requests). The Gini coefficient is calculated over the vector x = [x1 , . . . , xM ], where xm is the number of completed requests served by MSP m inP the Pexperiment. M M j=1 |xi −xj | , We use the standard definition: G(x) = i=1 2M 2 x̄ P M 1 x . . The results in TABLE V show the where x̄ = M i=1 i range values across all baselines. A smaller value is preferred, highlighted in red bold for non-GCP solutions and in blue bold for GCP-based variants. These results show that our nonGCP DRL performed best among non-cooperative baselines, indicating balanced resource allocation across MSPs. Among GCP variants, our DRL-GCP achieved the lowest range, reaching up to 51% more fairly distributed requests than baselines. Moreover, the presence of the GCP helped balance the number of served requests among MSPs, as shown by the smaller values in the GCP variants. The same conclusion is supported

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

TABLE V: Performance Across 100, 150, and 200 Request Configurations (Lower range = better), best results from the non-GCP configuration are displayed in bold red color, while best results from the GCP configuration are displayed in bold blue color 100

150

200

MSPs 3 MSPs 5 MSPs 7 MSPs 3 MSPs 5 MSPs 7 MSPs 3 MSPs 5 MSPs 7 MSPs

Avg 58 58 63 92 120 124 92 170 174

Max 76 76 79 78 126 129 78 176 179

Random 62 63 67 113 114 119 112 163 167

DRL 42 42 49 92 96 101 106 142 149

Avg-GCP 0 0 0 70 10 40 50 97 85

Max-GCP 50 17 38 50 88 85 50 93 85

Random-GCP 19 0 0 69 43 64 69 113 120

DRL-GCP 0 0 0 46 0 0 88 43 78

0.69

Avg Head Immersion

Reqs

TABLE VI: Gini-Coefficient Across 100, 150, and 200 Request Configurations (Lower value = better)

200

Max 0.25 0.18 0.23 0.26 0.25 0.30 0.26 0.30 0.36

Random 0.17 0.12 0.16 0.26 0.19 0.23 0.26 0.23 0.29

DRL 0.11 0.07 0.11 0.18 0.12 0.18 0.20 0.18 0.22

Avg-GCP 0.00 0.00 0.00 0.15 0.02 0.06 0.17 0.15 0.15

Max-GCP 0.17 0.04 0.09 0.17 0.19 0.17 0.17 0.20 0.17

Random-GCP 0.05 0.00 0.00 0.17 0.08 0.11 0.17 0.18 0.17

DRL-GCP 0.00 0.00 0.00 0.09 0.00 0.00 0.17 0.06 0.09

by TABLE VI of the gini coefficient, where both DRL and DRL GCP had the most balanced request distributions. The conclusion from this experiment highlights the superiority of our proposed solutions. They achieved the highest request completion and fulfillment rates, served the largest number of clients, and maintained the lowest cost in the non-GCP setup, while offering a competitive price compared to GCP-based alternatives. Moreover, these solutions demonstrated the most balanced distribution of served requests across MSPs, reinforcing their effectiveness not only in optimizing individual performance metrics but also in promoting fairness and stability in cooperative environments. 2) Experiment 2: Adaptability to changes in environment: Moreover, since the Metaverse environment is dynamic, with various events emerging suddenly and diverse needs for Heads, MSPs must be prepared to handle changes in request patterns from the Heads. Therefore, in the second experiment, we evaluate the resilience and adaptability of our solutions by subjecting the environment to multiple unexpected changes and observing how our DRL agent handles the resource allocation for these requests. In our case, we consider two types of changes. The first is a sudden shift in the request distribution of the Heads, simulating unexpected requests, such as seasonal concerts. Such a change can negatively impact poorly trained DRL agents and any history-based baselines, potentially leading to under- or over-provisioning. The second change simulates a dynamic shift in the immersion threshold required by some of the Heads. This reflects real-world scenarios in which certain Heads may demand higher-quality experiences in their VRooms. In that experiment, the DRL model was initially trained to meet an immersion threshold of 60%, which was later increased to 67%. The results, shown in Figure 10, highlight the system’s adaptability. Specifically, Figure 10(a) shows that our DRLGCP agent quickly adapted to the two environment changes, shown by the average Head immersion after only a few episodes. Figure 10(b) indicates that our agent did not drop any requests even after the environment changed. Figure

0.66

Change In Requests' Immersion Level

0.65 0.64

Change In Request Pattern

0.63

0.61 1

4 10 4

Training Episodes

((a)) Average head Served adaptation 300 Raw Smoothed

290

Total Requests Done

150

Avg 0.16 0.11 0.15 0.26 0.22 0.27 0.26 0.27 0.33

280 270 260 250 240 230 0

4 10 4

Episodes

((b)) Requests convergence adaptation 180

Maximum possible reward

160

Reward

100

MSPs 3 MSPs 5 MSPs 7 MSPs 3 MSPs 5 MSPs 7 MSPs 3 MSPs 5 MSPs 7 MSPs

0.67

0.62

140 120 100

Maximum possible reward

80 60

Raw Smoothed

40 0

4 10 4

Episodes

((c)) Reward re-convergence Raw Smoothed

150 145

Total Cost ($)

Reqs

Raw Smoothed

0.68

140 135 130 125 0

Episodes

5 10 4

((d)) Cost convergence

Fig. 10: Assessments of the Cooperative control of the MSPs

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

Fig. 11: The effect of limiting the withdrawal-to-insertion.

10(c) illustrates the agent’s reward re-convergence, while Figure 10(d) presents the total cost changes. The sudden drops in Figure 10(c)(d) occur immediately after the injected distribution/threshold shifts. At that moment, the previously learned policy is temporarily mismatched to the new demand/constraint regime (e.g., higher target immersion or different request mix), which reduces reward and alters cost until the policy re-stabilizes under the new regime; the subsequent recovery shows the agent re-adapting over the next episodes. 3) Experiment 3: Optimal Withdrawal-to-Insertion Ratio Analysis: Finally, since systems employing shared pools are subject to the free-rider problem, in which certain MSPs with suboptimal resource-allocation strategies exploit the collective resource by continuously withdrawing from the GCP without making any effective contribution, we address this issue in two steps. The first step is to enhance the resource allocation policies of all MSPs through DRL, and the second is to implement withdrawal constraints proportional to insertion limits. However, given the inherent complexity in determining optimal withdrawal thresholds, we conducted a third experiment specifically designed to evaluate the withdrawalto-insertion ratio that maximizes system-wide performance efficiency. In our third experiment, shown in Figure 11, we examined multiple withdrawal-to-insertion ratios. For instance, a factor of 1.6 indicates that an MSP can receive up to 160% of the GC units it has inserted into the GCP when needed. From the same figure, it is evident that a higher withdrawal limit increases the number of requests and served clients while reducing the number of failed clients. This also suggests that our centralized DRL solution does not explicitly require the setup of a safeguard to limit free-riders, the DRL agent automatically optimizes the allocation to balance between varying types and budget MSPs for the long-term goal of increasing the number of overall requests. VII. CONCLUSION AND FUTURE WORK This paper introduced CIVIC, a novel framework for optimizing resource allocation and enabling intelligent cooperation across multiple MSPs. Unlike prior works that overlooked digital twin accuracy in immersion modeling, relied on rigid resource allocation strategies, or were constrained to single-

MSP settings, CIVIC provides a unified solution that integrates virtual environment rendering, digital twin synchronization, and immersion-aware provisioning across diverse virtual rooms in dynamic multi-MSP ecosystems. We formulated the resource allocation challenge as two NP-hard problems in different settings: a non-cooperative model where MSPs act independently, and a cooperative model that leverages GCP for dynamic inter-MSP resource sharing. To address the computational complexity, we designed DRL agents that account for system heterogeneity and load uncertainty while coordinating resource distribution. Extensive experiments demonstrated that our DRL-based approaches consistently outperformed baseline strategies in both efficiency and fairness. Specifically, our DRL-GCP cooperative model achieved improvements of 12–36% in request completion, 23–70% in fulfillment rate, 20–60% in client capacity, and up to 51% in equitable request distribution—all while maintaining competitive operational costs. These results validate that CIVIC delivers robust, adaptive, and scalable performance under fluctuating load conditions, establishing its viability for realworld multi-MSP Metaverse deployments powering the next generation of immersive digital experiences. R EFERENCES [1] Y. Wang, Z. Su, N. Zhang, R. Xing, D. Liu, T. H. Luan, and X. Shen, “A survey on metaverse: Fundamentals, security, and privacy,” IEEE Communications Surveys & Tutorials, vol. 25, no. 1, pp. 319–352, 2023. [2] Grand View Research, “Metaverse market size and share: Industry report, 2030,” https://www.grandviewresearch.com/industryanalysis/metaverse-market-report, 2025, accessed: Jul. 25, 2025. [3] H. Alves, G. D. Jo, J. Shin, C. Yeh, N. H. Mahmood, C. Lima, C. Yoon, N. Rahatheva, O.-S. Park, S. Kim et al., “Beyond 5g urllc evolution: New service modes and practical considerations,” arXiv preprint arXiv:2106.11825, vol. 7, 2021. [4] N. H. Chu, D. N. Nguyen, D. T. Hoang, K. T. Phan, E. Dutkiewicz, D. Niyato, and T. Shu, “Dynamic resource allocation for metaverse applications with deep reinforcement learning,” in 2023 IEEE Wireless Communications and Networking Conference (WCNC). IEEE, 2023, pp. 1–6. [5] A. K. Jumani, J. Shi, A. A. Laghari, V. V. Estrela, G. A. Sampedro, A. Almadhor, N. Kryvinska, and A. ul Nabi, “Quality of experience that matters in gaming graphics: How to blend image processing and virtual reality,” Electronics, vol. 13, no. 15, 2024. [Online]. Available: https://www.mdpi.com/2079-9292/13/15/2998 [6] M. Xu, D. Niyato, J. Kang, Z. Xiong, C. Miao, and D. I. Kim, “Wireless edge-empowered metaverse: A learning-based incentive mechanism for virtual reality,” in ICC 2022 - IEEE International Conference on Communications, 2022, pp. 5220–5225. [7] H. Du, J. Liu, D. Niyato, J. Kang, Z. Xiong, J. Zhang, and D. I. Kim, “Attention-aware resource allocation and qoe analysis for metaverse xurllc services,” IEEE Journal on Selected Areas in Communications, vol. 41, no. 7, pp. 2158–2175, 2023. [8] Y. Jiang, J. Kang, X. Ge, D. Niyato, and Z. Xiong, “Qoe analysis and resource allocation for wireless metaverse services,” IEEE Transactions on Communications, vol. 71, no. 8, pp. 4735–4750, 2023. [9] X. Liu and Y. Deng, “Learning-based prediction, rendering and association optimization for mec-enabled wireless virtual reality (vr) networks,” IEEE transactions on wireless communications, vol. 20, no. 10, pp. 6356–6370, 2021. [10] L. Feng, X. Jiang, Y. Sun, D. Niyato, Y. Zhou, S. Gu, Z. Yang, Y. Yang, and F. Zhou, “Resource allocation for metaverse experience optimization: A multi-objective multi-agent evolutionary reinforcement learning approach,” IEEE Transactions on Mobile Computing, 2024. [11] Y. Jiang, J. Kang, D. Niyato, X. Ge, Z. Xiong, C. Miao, and X. Shen, “Reliable distributed computing for metaverse: A hierarchical game-theoretic approach,” IEEE Transactions on Vehicular Technology, vol. 72, no. 1, pp. 1084–1100, 2023.

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

[12] J. Feng and J. Zhao, “Resource allocation for augmented reality empowered vehicular edge metaverse,” IEEE Transactions on Communications, vol. 73, no. 3, pp. 1987–2001, 2025. [13] N. H. Chu, H. Q. Nguyen, D. N. Nguyen, D. T. Hoang, K. T. Phan, E. Dutkiewicz, D. Niyato, and T. Shu, “Dynamic multi-tier resource allocation framework for metaverse,” IEEE Network, 2024. [14] S. Zhang, Y. Chen, B. Guo, D. López-Pérez, J. Hu, and H. Zheng, “Multi-objective reinforcement learning towards user’s targeted vr qoe,” in GLOBECOM 2023-2023 IEEE Global Communications Conference. IEEE, 2023, pp. 3349–3354. [15] Z. Huang, P. Yang, W. Wu, and N. Zhang, “Predictive and robust fieldof-view selection for virtual reality video streaming,” in 2023 IEEE 34th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), 2023, pp. 1–6. [16] M. Xu, D. Niyato, J. Kang, Z. Xiong, C. Miao, and D. I. Kim, “Wireless edge-empowered metaverse: A learning-based incentive mechanism for virtual reality,” in ICC 2022 - IEEE International Conference on Communications, 2022, pp. 5220–5225. [17] M. Chen, A. Liu, N. N. Xiong, H. Song, and V. C. Leung, “Sgpl: An intelligent game-based secure collaborative communication scheme for metaverse over 5g and beyond networks,” IEEE Journal on Selected Areas in Communications, vol. 42, no. 3, pp. 767–782, 2023. [18] S. Vlahovic, M. Suznjevic, and L. Skorin-Kapov, “A survey of challenges and methods for quality of experience assessment of interactive vr applications,” Journal on Multimodal User Interfaces, vol. 16, no. 3, pp. 257–291, 2022. [19] I. Galloso, C. Feijóo, and A. Santamarı́a, “Novel approaches to immersive media: from enlarged field-of-view to multi-sensorial experiences,” in Novel 3D media technologies. Springer, 2014, pp. 9–24. [20] N. Shamim, S. Gupta, and M. M. Shin, “Evaluating user engagement via metaverse environment through immersive experience for travel and tourism websites,” International Journal of Contemporary Hospitality Management, vol. 37, no. 4, pp. 1132–1174, 2025. [21] M. S. Anwar, J. Wang, S. Ahmad, W. Khan, A. Ullah, M. Shah, and Z. Fei, “Impact of the impairment in 360-degree videos on users vr involvement and machine learning-based qoe predictions,” IEEE Access, vol. 8, pp. 204 585–204 596, 2020. [22] K. Desai, S. Raghuraman, R. Jin, and B. Prabhakaran, “Qoe studies on interactive 3d tele-immersion,” in 2017 IEEE international symposium on multimedia (ISM). IEEE, 2017, pp. 130–137. [23] S. Liu, J. Xie, and X. Wang, “Qoe enhancement of the industrial metaverse based on mixed reality application optimization,” Displays, vol. 79, p. 102463, 2023. [24] A. Wortmann, “A cross-domain systematic mapping study on software engineering for digital twins,” Journal of Systems and Software, vol. 193, p. 111361, 2022. [25] S. Su, A. Nassehi, B. Hicks, and J. Ross, “Characterisation and evaluation of identicality for digital twins for the manufacturing domain,” Journal of Manufacturing Systems, 2023. [26] M. S. Anwar, A. Choi, S. Ahmad, K. Aurangzeb, A. A. Laghari, T. R. Gadekallu, and A. Hines, “A moving metaverse: Qoe challenges and standards requirements for immersive media consumption in autonomous vehicles,” Applied Soft Computing, vol. 159, p. 111577, 2024. [27] M. N. Selzer and S. M. Castro, “A methodology for generating virtual reality immersion metrics based on system variables,” Journal of Computer Science & Technology, vol. 23, no. 2, pp. 92–106, 2023. [28] H. Du, J. Wang, D. Niyato, J. Kang, Z. Xiong, X. Shen, and D. I. Kim, “Exploring attention-aware network resource allocation for customized metaverse services,” IEEE Network, vol. 37, no. 6, pp. 166–175, 2023. [29] P. Czarnul, B. Lichocki, and D. Spinczyk, “Optimization of resourceaware parallel and distributed computing: a review,” The Journal of Supercomputing, 2025. [30] S. Bhaskaran and S. Muthuraman, “A comprehensive study of resource provisioning and optimization in edge computing,” Computers, Materials & Continua, vol. 83, no. 3, pp. 5037–5070, 2025. [31] W. K. Awad, I. Aljarah, H. Faris, and A. M. Al-Zoubi, “Resource allocation strategies and task scheduling algorithms for cloud computing: A systematic literature review,” Journal of Intelligent Systems, vol. 34, no. 1, p. 20240441, 2025. [32] A. Kiggundu et al., “Resource allocation in mobile networks: A decision model of jockeying in queues,” arXiv preprint arXiv:2402.11054, 2024. [Online]. Available: https://arxiv.org/abs/2402.11054 [33] C. Delgado et al., “On optimal resource allocation in virtual sensor networks,” arXiv preprint arXiv:2402.06281, 2024. [Online]. Available: https://arxiv.org/abs/2402.06281 [34] W. K. Awad, K. A. Z. Ariffin, M. Z. A. Nazri, and E. T. Yassen, “Resource allocation strategies and task scheduling algorithms for

cloud computing: A systematic literature review,” Journal of Intelligent Systems, vol. 34, no. 1, p. 20240441, 2025. [Online]. Available: https://doi.org/10.1515/jisys-2024-0441 [35] F. Liang, W. Li, L. Zhang et al., “Resource allocation and workload scheduling for large-scale distributed deep learning: A survey,” arXiv preprint arXiv:2406.08115, 2024. [Online]. Available: https://arxiv.org/abs/2406.08115 [36] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017.

A PPENDIX Theorem 1 (NP-hardness of the cooperative problem P̄ ). The decision version of P̄ (objective (8) with constraints (c1)– (c13) and GCP update (c14)) is NP-hard. Proof. To show the claimed hardness in a transparent way, we reduce from 0–1 Knapsack. Given items i = 1, . . . , n with weights wi , values vi ∈ Z+ , capacity W , and target value V ⋆ , construct an instance of P̄ with M = n MSPs and a short horizon T = 2. Step 1 (calibration). For each MSP i, fix the VRoom and model parameters so that meeting the immersion threshold for its (single) active Head requires a minimal total allocation cost exactly wi . This uses the same immersion and cost functions already in P̄ ; feasibility checking for a proposed allocation remains polynomial. Step 2 (GCP capacity). Set all per-MSP budgets at t = 2 to zero, by allowing a subset of MSPs at time t = 1 to donate their full surpluses so that the GCP holds exactly W credits at (t) t = 2. This is permitted by the donation variable δm ∈ [0, 1], the donation rule (c10), and the GCP recursion (c14), which carries donations from t = 1 into available credits at t = 2. Step 3 (equivalence). At t = 2, constraint (c11) limits the aggregate MSP deficit of the selected fully served MSPs (t) by the available W credits in the GCP. Because Lm = 1 iff all active Heads of MSP m meet the threshold (c3), choosingPwhich MSPs to fully serve at t = 2Pconsumes exactly and yields total value i∈S wi credits i∈S vi in P (2) the objective sum L . Hence, there exists a feasible P m(2) m ⋆ allocation with instance L m m P≥ V iff the knapsack P admits a subset S with i∈S wi ≤ W and i∈S vi ≥ V ⋆ . Therefore, P̄ -DEC is NP-hard. Theorem 2 (NP-hardness of the non cooperative problem P ). The decision version of P (objective (7) with constraints (c1)– (c8)) is NP-hard. Proof. We again reduce from 0–1 Knapsack. To isolate the source of combinatorial hardness, consider a single-MSP instance (M =1) with horizon T and exactly one active Head at each t ∈ {1, . . . , T }. Calibrate parameters so that the minimal cost to reach the immersion threshold at time t equals a prescribed weight wt . Use the paper’s time-coupled budget semantics: allocation at t reduces the remaining budget available at t+1. Set the initial budget to B and interpret the per-timestep budget bound (c4) as the remaining capacity at each step. Then selecting the set of timesteps S ⊆ {1, . . . , T } in which we fully serve P the Head corresponds to picking items with total “weight” t∈S wt not P (t) exceeding B, and yields objective value = |S|. t∈S L1

CIVIC: Cooperative Immersion Via Intelligent Credit-sharing in DRL-Powered Metaverse

Thus, deciding whether there exists a feasible allocation with PT P (t) ≥ K and total spend t=1 L1 t∈S wt ≤ B is exactly the 0–1 knapsack decision problem. Hence P -DEC is NPhard. Remark: These reductions use only the all-or-nothing fulfillment rule (c3), the local budget constraints (c4), and, for P̄ , the shared-pool coupling through (c10)–(c13) and the GCP update (c14). They already show combinatorial hardness in simplified settings. In the full models, the objective and constraints additionally include non-convex functions (SSIM with a negative exponent in bitrate, the resource-scaling Hessian with negative mixed second derivatives, and the harmonicmean DT accuracy), which the paper proves via Hessian analysis; thus both P and P̄ are non-convex MINLPs, further precluding polynomial-time global optimization and motivating DRL-based solution methods.

Record · ID 2530 · SHA-256 d122f061d0be1545
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.