1
Optimization of Collaborative Semantic Communication Network Performance with Channel and Content Preference Feedback
arXiv:2607.25011v1 [cs.NI] 27 Jul 2026
Defeng Zhou, Dongyu Wei, Graduate Student Member, IEEE, Siyao Li, Mingzhe Chen, Senior Member, IEEE Abstract—Semantic communications is attracting attention due to its ability to transmit data meaning rather than raw bits. However, existing semantic communication frameworks treat and transmit all image regions with equal importance, which is not practical for real-world applications which may prioritize different content in an image. To address this issue, we propose a novel semantic communication framework that enables a transmitter to use limited channel and content feedback to prioritize the transmission of important image regions. In particular, in the proposed framework, a base station (BS) divides each image into sub-images, extracts their semantic information, and transmits them to users according to their preferences. The users will reconstruct the image based on the received sub-images and cooperatively decide when to send channel state information (CSI) or content-preference feedback under dynamic channels and limited resources. We formulate an optimization problem to minimize the semantic-weighted mean square error between the original image and the regenerated image by optimizing sub-channel allocation, users’ power allocation, and feedback selection. To address this problem, a value decomposition actorcritic (AC) with dynamic neighborhood construction (VDAC-DNC) scheme is proposed. The proposed method combines AC with value decomposition networks to allow the BS to approximate discrete actions by a continuous action distribution, thus reducing the output dimension and improving training efficiency. The introduced DNC method further improves training efficiency by constructing a small discrete neighboring action space to search for an action with the maximum Q value, thus avoiding traversing the large discrete action space. Simulation results show that the proposed VDAC-DNC scheme can improve the performance by up to 5.04% and 18.55% compared to the standard multi-agent QAC method and the proposed method without feedback transmission. Index Terms—Semantic communication, image transmission, content preference feedback, actor-critic.
I. I NTRODUCTION The integration of communication and artificial intelligence (AI) is moving wireless communications beyond bit-level transmission toward meaning-level exchange [1], [2]. In this paradigm, semantic communication (SC) has emerged as a key technique, enabling transmitters to extract and transmit the “meaning” of data rather than raw bits, thereby allowing receivers to reconstruct the data or infer task outputs (e.g., classification labels [3], answers [4]) with significant reduced Defeng Zhou, Dongyu Wei, and Mingzhe Chen are with the Department of Electrical and Computer Engineering, University of Miami, Coral Gables, FL, 33146, USA (e-mail: {defengzhou, dongyu.wei, mingzhe.chen}@miami.edu) Mingzhe Chen is also with the Frost Institute for Data Science and Computing, University of Miami, Coral Gables, FL 33146 USA. Siyao Li is with the Department of Electrical Engineering and Computer Science, Embry-Riddle Aeronautical University, Daytona Beach, FL 32114, USA (email: [email protected])
latency and resource consumption [5]. However, deploying SC techniques over wireless networks faces several challenges including: 1) data meaning extraction and representation, 2) semantic information allocation in dynamic wireless networks, and 3) content feedback-assisted semantic communication method design. A. Related Works Recently, several works in [6]–[10] have studied the optimization of image transmission in different wireless networks. In [6], the authors designed a deep joint source-channel coding (JSCC) scheme that adapts its compression rate to channel conditions by training a policy network to select features from the image that need to transmit. The author in [7] investigated an end-to-end JSCC design by unifying the coding rate reduction maximization and the mean square error (MSE) minimization for image recovery and classification simultaneously. The author in [8] proposed probabilistic graph-based method for image transmission, enabling semantic information to be compressed and transmitted more accurately while reducing communication energy consumption. The authors in [9] introduced a simple code-mask–based JSCC that adjusts the compression rate to meet a target image quality (e.g., peak signal to noise ratio) under different wireless networks. In [10], the authors employ Swin Transformer to extract semantic information of image and scaled the latent representation according to channel state information (CSI), enhancing the ability of the model to deal with various channel conditions. However, the works in [6]– [10] treat all image regions with equal importance, which is not practical for real-world users who prioritize different content (e.g., a “car” content may be more semantically relevant than the “sky” in a traffic scenario). Without accounting for content preference, transmitters may waste bandwidth on irrelevant background features while important regions suffer from distortion. To address this problem and further improve transmission reliability, feedback mechanisms are essential. The benefit of CSI feedback in downlink transmission has also been studied in capacity region [11] and stability region [12]. The authors in [13] and [14] integrated channel feedback into semantic image transmission. In particular, the work in [13] used CSI feedback to adjust the compression rate of each image. The authors in [14] sent the decoder’s current estimation regarding to the source image to the transmitter so as to update the encoder and mitigate the impact of channel noise on end-to-end reconstruction quality. The authors in [15] simply prioritized
2
the foreground of the image and transmitted it while masking divides the images into several sub-images according to less informative background. The authors in [16] divided image image content and allocates these sub-images to different regions into two parts: regions of interest (ROI) and regions of sub-channels and sends them to each user using SC non-interest (RONI), and designed an SC neural network that techniques. After receiving the sub-images, users may can preserve more information to transmit the ROI regions with transmit the CSI feedback or content preference feedback high quality while transmitting RONI with data compression. to assist the BS in subsequent sub-image transmission. The authors in [17] considered a collaborative inference Considering limited power for each user and the limited between the transmitter and receiver, and investigated the use uplink bandwidth, users must cooperate to transmit the of inference results of the receiver to optimize quantization bitfeedback under the required latency. To minimize the width and computation frequencies of the transmitter, so as to semantic-weighted MSE between the original images improve inference efficiency. However, current works in [13]– and the images regenerated by users, we formulate an [17] did not consider how dynamic wireless channels and user optimization problem aiming to minimize MSE while data preference affect semantic image transmission. Specifically, satisfying the feedback transmission latency requirements with imperfect CSI, the transmitter may overestimate a poor of users by optimizing the BS’s sub-image allocation, user channel thus resulting in packet loss, or underestimate a power allocation and feedback selection. good channel to waste bandwidth. Meanwhile, without data • To solve the optimization problem, a value decompositionpreference information, important image components may be based actor-critic with dynamic neighborhood construction scheduled on poor channels while less important regions are (VDAC-DNC) method is proposed, which enables the BS protected, which may increase semantic distortion and reduce to allocate the sub-channel and users feedback selection task accuracy. Hence, when considering both dynamic wireless and power allocation cooperatively. Different from the channels and user data preference, users must trade off between standard value decomposition-based networks (VDN) [23], sending CSI to inform the base station (BS) about the status the proposed method combines actor-critic with VDN, of wireless channels as well as sending content preference which allows the BS to approximate the discrete actions information to request important content. by a continuous action distribution, thus reducing the output dimension and improving RL training efficiency. A number of existing works have investigated reinforcement • To further improve the training efficiency, the introduced learning (RL) for optimizing semantic communication perforDNC method finds a better discrete action from the mance. The authors in [18] proposed a deep reinforcement continuous action per RL iteration. In particular, the DNC learning driven joint position and power optimization algorithm method constructs a small discrete neighboring action to maximize the semantic data transmission throughput. In [19], space to search for an action with the maximum Q value, an attention-based RL algorithm was developed to analyze thus avoiding traversing the large discrete action space. the relationship between the original data and its semantic In particular, the DNC uses a continuous action produced information. However, these works [18], [19] do not consider by a continuous policy, which is easier to optimize than a the cooperation among different agents. Consequently, each complex discrete policy over large discrete action spaces, agent’s performance will be affected by the actions of other to construct a small discrete neighboring action space. agents, thus reducing the transmission performance achieved Then, DNC searches for an action with the maximum by RL. To address the performance degradation caused by the Q value from the small neighboring action space, thus lack of collaboration among different agents, the authors in avoiding traversing the large discrete action space. [20] proposed a value decomposition-based deep Q-learning scheme multi-agent RL (MARL) to minimize the average transmission latency. The authors in [21] utilized a multi-agent deep deterministic policy gradient (MADDPG)-based MARL Simulation results show that the proposed VDAC-DNC scheme to dynamically allocate channels and power for transmission. can improve the performance by up to 5.04%, 12.43%, and Nonetheless, these RL algorithms in [20], [21] struggle to scale 18.55% compared to the standard multi-agent QAC (MAQAC) to large discrete action spaces since they must evaluate and method [24], independent Q method, and the proposed method explore a huge number of actions [22]. without CSI and content preference feedback information. Meanwhile, the proposed method with DNC achieves a 2.9x5.3x speedup over the proposed method without DNC while B. Contributions maintaining performance. The main contribution of our work is a novel channel and content preference feedback enabled SC framework that enables The rest of this paper is organized as follows. The promultiple users to use limited channel and content feedback posed channel and content preference feedback enabled SC for efficient and accurate image data transmission. The key framework and problem formulation are described in Seccontributions include: tion II. The proposed VDAC-DNC scheme for optimizing • We consider a novel channel and content preference sub-image allocation, power allocation and feedback selection feedback enabled SC framework in which a BS transmits is introduced in Section III. In Section IV, we analyze the the images to a set of users while users cooperatively convergence, implementation and complexity of the proposed transmit the feedback to the BS for further efficient and scheme. Numerical simulation results are shown and analyzed accurate image data transmission. In particular, the BS first in Section V. Finally, Section VI concludes this paper.
3
Fig. 1. Channel and preference feedback enabled semantic communication framework
II. S YSTEM M ODEL AND P ROBLEM F ORMULATION
can identify important and meaningful pixels, achieving large receptive fields for sub-image extraction. Here, we can use any other semantic segmentation methods, which will not affect our designed semantic communication framework. Let I k,v ∈ RH×W ×C be an original image k that the BS needs to transmit to a user v with H, W and C being the height, width, and the number of color channels of the image. Then, the set of sub-images that are extracted from image I k,v using Jk,v the model [26] is Mk,v = {m1k,v , m2k,v , . . . , mk,v } where Jk,v is the number of sub-images in image I k,v which varies according to the image I k,v , mjk,v is sub-image j in image I k,v . 2) Semantic Information Generation: A VAE is used to generate semantic information. Although the size of the VAE input is not required to be fixed, the height and width of the input must be divisible by the VAE’s downsampling factor (typically 8 or 16). However, the size of the sub-images is different which depends on the objects in the sub-images. For simplicity, we unify the size of the sub-images such that we can use one VAE model to process various sub-images and generate semantic information. To this end, we use a padding method that changes the size of the sub-image to the size of the original image by adding zeros around each sub-image. j fk,v is Then, the semantic information of padded sub-image m
We consider a channel and preference feedback enabled SC system where a BS sends images to a set V of v users using SC techniques, as illustrated in Fig. 1. In particular, to send an image to a user, the BS divides the image into several sub-images according to image content. For example, the BS will extract "palm", "sky" and "water" from the original image and generate three sub-images. Then, the BS uses a variational autoencoder (VAE) based semantic encoder [25] to j xjk,v = fθE (f mk,v ), (1) encode sub-images according to the downstream applications E (e.g., image classification or object detection) of the receiver. where fθ (·) is the semantic encoder used to generate the Hereinafter, we define the output of the semantic encoder as the semantic information of sub-image with θ being the encoder semantic information of the sub-image. The BS will allocate parameters. Here, the encoder can be applied for all users. j fk,v was processed by 3) Sub-image Compression: Since m sub-image semantic information to different sub-channels and send them to the receiver over time-varying wireless channels. a padding method, a large number of zero-valued pixels exist. At the receiver, a semantic decoder is used to reconstruct sub- Hence, we need to use a compression method to further reduce j images. Then, the receiver merges the sub-images to regenerate the redundancy in the sparse semantic information xk,v . Here, the original image. Based on the regenerated image, the we propose to use the algorithm in [16] to compress the j j fk,v . receiver will send CSI (i.e., the representation of successfully semantic information xk,v of each padded sub-image m C received sub-images) or content preference feedback (i.e., the Let f (·) be the data compression function. The compressed importance of different sub-image categories) to the BS such semantic information is x̂jk,v = f C (xjk,v ). Here, we can also that the BS can optimize sub-image allocation and feedback consider other compression methods, which do not affect the selection to improve semantic transmission performance. The designed semantic data transmission methods. designed framework can be used for applications that need to transmit images such as autonomous driving, traffic monitoring, B. Transmission Model remote inspection, and sensing. Next, we first introduce the Next, we introduce 1) the downlink transmission of submethods used for semantic information extraction. Then we images from the BS to users, and 2) the uplink transmission introduce the semantic information transmission model and of feedback from users to the BS. image reconstruction. Finally, we formulate the optimization 1) Downlink Sub-image Transmission: We assume that the problem. BS has a total of N V channels and each user can use N sub-channels for sub-image transmission. Let Lv be a set of sub-channels that the BS uses to serve user v. These subA. Semantic Information Extraction channels have different channel conditions, and hence, they Semantic information extraction includes three key steps: 1) have different data rates and packet error rates. The downlink sub-image extraction, 2) semantic information generation, and channel capacity of the BS transmitting semantic information 3) sub-image compression, which will be introduced in detail to user v over sub-channel n at time slot t is ! as follows. P D hD n,v (t) 1) Sub-image Extraction: We use the method in [26] to log2 1 + , with updated CSI, D +N In,v 0 divide an image into several sub-images according to image RD (t) = n,v P D d−2 objects. Compared to other sub-image extraction methods (e.g., v ln(1 − ξ) , otherwise, log2 1 − D sliding-window convolutional neural network (CNN) [27] and In,v + N0 fixed-grid vision transformer (ViT) [28]), the method from [26] (2)
4
PN D where P D is the transmit power of the BS, In,v is the sub-image (i.e., n=1 αnkj,v (t) = 1) and user v successfully n interference caused by BSs that are located in other service receives it (i.e., ηkj,v = 1). In particular, αkj,v (t) = 1 implies j areas and use sub-channel n, N0 is the noise power spectral that compressed semantic information x̂k is transmitted to D −2 n density, and hD is the channel gain of sub- user v over sub-channel n at time slot t, and αkj,v (t) = 0, n,v (t) = γn dv channel n in time slot t between the BS and user v with otherwise. γnD ∼ exp(λ = 1) being the Rayleigh fading parameters of sub3) Sub-image Combination: Each User v combines the subchannel n and dv being the distance between the BS and user images to regenerate the original image. Here, user v will v, ξ is the outage probability. From (2), we see that when the consider the maximum value of the same pixels in different BS does not have the updated CSI, the downlink transmission sub-images as the value of the pixel in the regenerated image. rate follows the standard outage-capacity formulation [29]. In This image regeneration method can preserve sharp boundaries particular, the BS does not have updated CSI in two cases: since it can reduce the impact of zero elements added by the 1) the user does not transmit CSI feedback, or 2) the user zero-padding method on sub-image regeneration. The value transmitted a CSI feedback but the receiver did not receive it î of pixel p of user v in the regenerated image S k,p k,v is within the feedback delay constraint. 2) Uplink Feedback Transmission: Here, we assume users îkp,v = max{i1kp,v , i2kp,v , . . . , iJkp,v }. (5) can obtain the perfect CSI via pilot-assisted channel estimation. Thus, the data rate of user v transmitting feedback to the BS at time slot t is RvU (t) =
U −2 g(t)pU B v (t)γ dv log2 (1 + ), g(t) BN0
(3)
D. Feedback of Receiver
At each time slot, each user can 1) transmit either CSI feedback or content preference feedback to the transmitter, or 2) remain silent without sending any feedback. The content preference feedback transmitted by user v at time slot t is D v (t) = [(u1,v (t), ω1,v (t)), (u2,v (t), ω2,v (t)), . . . , (uD,v (t), ωD,v (t))], where ud,v is the content that user v wants to receive and C. Image Reconstruction ωd,v is the importance weight of content ud,v . Here, D v (t) Image reconstruction includes the following three key steps: includes only the top D categories that user v wants to receive 1) Data decompression, 2) sub-image regeneration, and 3) and user v cannot send all content category information to sub-image combination, which will be introduced in detail as the BS due to limited power. The CSI feedback of user v is follows. defined as a vector cv (t) = [c1,v (t), c2,v (t), . . . , cLv ,v (t)] at 1) Data Decompression: Once the compressed data is time slot t. Here, cn,v (t) ∈ {0, 1} indicates whether sub-image received, the user restores it back to its original form for sjk,v is successfully transmitted to user v over sub-channel sub-image generation. Here, we apply the restoration algorithm n in time slot t. Based on the CSI feedback, the BS can in [16]. learn the channel conditions (e.g., the downlink channel gain 2) Sub-image Regeneration: A semantic decoder of each hD n,v (t) [30]) and the sub-images that have been received user v is used to regenerate the corresponding sub-image by user v. Let βD,v (t), βC,v (t) ∈ {0, 1} be the feedback sjk,v . Since user v may not be able to receive all the subselection indicators where βD,v (t) = 1 (e.g., βC,v (t) = 1) images due to channel packet errors, we define η k,v (t) = implies that user v will transmit content preference feedback [ηk1,v (t), ηk2,v (t), . . . , ηkJ,v (t)] as a vector to represent (CSI feedback) to the BS at time slot t. Otherwise, we have whether the sub-images are received by user v. In particular, βD,v (t) = 0 (e.g., βC,v (t) = 0). ηkj,v = 1 implies that semantic information x̂jk,v is received by user v, and ηkj,v = 0, otherwise. The sub-channel allocation vector of compressed semantic information for user v is αk,v (t) = [αk1,v (t), . . . , αkj,v (t), . . . , αkJ,v (t)] E. Problem Formulation Lv 1 with αkj,v (t) = [αkj,v (t), . . . , αkj,v (t)] being a sub-image allocation vector for sub-image j in image k of user v. Here, we Given the defined system model, our objective is to minimize assume that one sub-image cannot be split and transmitted over the sum of the users’ semantic-weighted MSE between the several sub-channels within the same time slot. The regenerated original images and the images regenerated by the user. sub-image sjk,v (αkj,v (t), ηkj,v (t)) at user v is This minimization problem includes optimizing the sub-image N allocation α = {α1 (t), α2 (t), . . . , αK (t)}t∈T , user power alX sj , n U U α (t) = 1, η (t) = 1, location p = {pU kj,v kj,v 1 (t), p2 (t), . . . , pV (t)}t∈T and user feedback k,v sjk,v (αkj,v (t), ηkj,v (t)) = n=1 selection β = {βD,1 (t), βC,1 (t), . . . , βD,V (t), βC,V (t)}t∈T . We 0, else. consider that the BS will transmit K images to each user over (4) a time period that consists of a set T of KT time slots. Each From (4), we see that user v can regenerate the original sub- image must be transmitted within T time slots. The optimization image sjk,v when the BS uses a sub-channel to transmit the problem is formulated as where pU v (t) is the transmit power of user v at time slot t, g(t) is the number of users that want to transmit feedback to the BS, B is the total uplink bandwidth, N0 is the noise power spectral density, and γ U is the uplink Rayleigh fading parameter.
5
min
XX X
α,β,p
ωj,v (t) ∥mjk,v − sjk,v (αkj,v (t), ηkj,v (t))∥22
v∈V k∈K j∈Mk
(6) n s.t. αkj,v (t) ∈ {0, 1}, ∀n ∈ Lv , ∀j ∈ Mk,v , ∀k ∈ K, ∀v ∈ V,
(6a) n αkj,v (t) = 0, ∀t ∈ T \ Tk , (6b) X X n αkj,v (t) ≤ 1, ∀j ∈ Mk , ∀k ∈ K, ∀v ∈ V, n∈Lv t∈T
(6c) X
n αkj,v (t) ≤ 1, ∀n ∈ Lv , ∀t ∈ Tk , ∀v ∈ V,
(6d)
j∈Mk,v
βD,v (t) + βC,v (t) ≤ 1, ∀t ∈ T , ∀v ∈ V, X pU v (t) ≤ pmax , ∀v ∈ V,
(6e) (6f)
t∈T
Z(x̂jk,v ) D (t) Rn,v
n (t) = 1, ≤ △t, if αkj,v
(6g)
βD,v (t)Z(D v (t)) + βC,v (t)Z(cv (t)) ≤ RvU (t) △ tf , if βD,v (t) + βC,v (t) = 1,
(6h)
where K is the set of K images that the BS must transmit to each user and Tk ≜ {t ∈ T : (k − 1)T ≤ t ≤ kT } is the set of time slots for transmitting the image k, ∥ · ∥2 denotes the ℓ2 -norm, Z(x̂jk,v ) is the data size of semantic information x̂jk,v , △t is the time duration of a time slot, △tf is the time duration of feedback transmission, and pmax is the available power of each user for feedback transmissions. Constraints (6a), (6b) and (6c) ensure that the BS can transmit each sub-image once in specific time slots no matter whether the user v correctly receives it. Constraint (6d) implies that the BS can use only one time slot to transmit one sub-image. Constraint (6e) enforces that each user v can only transmit one type of feedback to the BS per time slot. Constraint (6f) limits the power used for feedback transmission with pmax being users total power budget. Constraints (6g) and (6h) are the requirements of each sub-image transmission and feedback transmission delay, respectively. The formulated problem in (6) is challenging to solve via traditional optimization methods due to the following reasons. First, from (6), we see that the objective function depends not only on the sub-image allocation vector α and power allocation vector p, but also on variables such as user feedback selection β, which cannot be expressed by explicit functions. Second, since users can transmit a limited number of feedback messages, a tradeoff exists between content preference feedback and CSI feedback transmission. Here, more content preference feedback transmissions enable the BS to obtain more content categories that users are interested in, while more CSI feedback transmission enables the BS to learn more channel information (e.g., hD n,v (t)) thus allocating appropriate content to each sub-channel. Third, due to limited energy, users cannot send feedback at each time slot and hence, the BS cannot know the real-time wireless environment and sub-image transmission status, which further complicates the optimization problem. Finally, users must cooperatively transmit feedback since the
bandwidth used for feedback transmission is limited. If all users transmit feedback simultaneously, they may not have enough bandwidth to ensure feedback be transmitted to the BS within the required delay. III. P ROPOSED A LGORITHM In this section, we introduce a VDAC-DNC algorithm to solve (6). Compared with existing MARL methods such as VDN [23] and MADDPG [24], the proposed approach combines the strengths of both value decomposition and actor–critic paradigms. Specifically, multi-agent Q-learning methods such as standard VDN achieve higher performance but with low training efficiency since they require extensive exploration and environment interactions, while multi-agent actor–critic methods such as MADDPG offer high training efficiency at the cost of reduced performance since they update the policy directly using gradient information and may easily get stuck in suboptimal solutions. Thus, the proposed method offers a reasonable trade-off between training efficiency and algorithm performance. Meanwhile, the proposed method is more effective in finding better actions in huge discrete action spaces since the designed method approximates the discrete action space using a continuous action and selects an action with the maximum Q value from the continuous action space. Next, we first introduce the components of the proposed VDACDNC method. Then, we show the training process of the VDAC-DNC. A. Components of VDAC-DNC Method The components of the proposed VDAC-DNC method are specified as follows: Agent: The agents in the VDAC-DNC scheme are the BS and users. At each time slot, the BS determines sub-image transmission, and then users determine whether to transmit a feedback and which type of feedback to transmit, as well as the power used for feedback transmission. BS States: The local state of the BS is used to describe the current observation of wireless environment and sub-images transmission status. The local state of the BS about user v is defined as oBv,t = [z v (t), ω v (t), cv (t), Rv (t)],
(7)
J
k,v where z v (t) = [Z(x̂1k,v ), Z(x̂2k,v ), . . . , Z(x̂k,v )] is a vector of the data sizes of the sub-images in an original image transmitted to user v, ω v (t) = [ω1,v (t), ω2,v (t), . . . , ωJk,v (t)] is the importance weight of the sub-images at the BS, cv (t) is the CSI feedback, and Rv (t) = [R1,v (t), R2,v (t), . . . , RN,v (t)] is the transmission rate of each channel. Thus, the total local state of the BS is oB,t = [oB,1,t , oB,2,t , . . . , oB,V,t ]. Note that the BS may not be able to receive the CSI and content preference feedback from the users at each time slot. Hence, when no feedback is received, ω v (t) and cv (t) retain their values from the previous time slot (e.g., ωv (t) = ωv (t − 1), when βD,v (t) = βC,v (t) = 0). User States: The local state of user v is
R oU v,t = [ẑ v (t), ĉv (t), pv (t)],
(8)
6
Fig. 2. The architecture of the proposed VDAC-DNC algorithm J
k,v where ẑ v (t) = [Z(s1k,v ), Z(s2k,v ), . . . , Z(sk,v )] is a vector of ber of feedback transmissions whose delay exceeds the threshthe data sizes of the received sub-images at user v, ĉv (t) is the old, κR and κF are weight parameters that control the weight current CSI, pRv (t) is the remaining power of user v at time penalty of sub-image transmission delay in (6g) and feedback slot t. Hence, the joint state among all agents at time slot t is transmission delay (6h), respectively. Note that, we cannot U a vector ot = [oBt , oU directly use the negative weighted MSE as the reward since 1,t , . . . , ov,t ]. BS Actions: The actions of the BS at each step are sub-image it will encourage the agents to transmit all the sub-images as allocation for each user. We define an action of the BS for soon as possible to minimize the MSE, which may violate the user v at time slot t as aBv,t = [δ1,v (t), δ2,v (t), . . . , δN,v (t)] delay constraints (6g) and (6h) with δn,v (t) ∈ {0, Jk,v } being the index of the sub-image BS Actor Networks: The actor network of BS π ϕB (âBv,t |oBv,t ) transmitted to user v and assigned to sub-channel n at time generates an action âB based on the current BS state oB . v,t v,t slot t. Here, the action output by the actor network is different from User Actions: An action of each user includes the feedback the action defined in aBv,t . This is because the output of the selection and power allocation. We define an action of the agent actor network is a continuous action probability distribution user v at time slot t as aU,v,t = [βD,v (t), βC,v (t), pU and the action selected by the actor may not be in the discrete v (t)]. Here, to guarantee that the selected actions meet the constraints (6b) action space defined in A. For example, if we use one Gaussian – (6f), we use an action mask algorithm [31] to remove all distribution to approximate the continuous action probability infeasible actions. The joint action vector of the BS and all distribution, the output of the actor network is a vector [µ, σ] U users is at = [aB1,t , . . . , aBV,t aU where µ is the mean and σ is the variance. Then, the action âBv,t 1,t , . . . , aV,t ]. Reward: The proposed VDAC-DNC will maximize this is sampled by this continuous Gaussian distribution (i.e., âB ∼ v,t reward, thus finding the optimal solution for the problem in (6). N (µ, diag(σ 2 ))). Hereinafter, we call the action selected by Therefore, the reward is the actor as continuous action and the action defined in aBv,t X action âBv,t will be used to B A r(ot , at ) = Dk,v − Dk,v − κR eRt − κF eFt , (9) as discrete action. The continuous B find a discrete action av,t by using the DNC method, which v∈V will be introduced in Section III-B. Note that âBv,t ∈ RN is a where continuous action that is used to construct the actual discrete X j j B 2 action aBv,t . Dk,v = ωj,v (t)∥mk,v − sk,v (αkj,v (t), ηkj,v (t))∥2 , User Actor Networks: The actor network of user v j∈Mk,v U U (10) π ϕUv (aU v,t |ov,t ) generates an action av,t based on the current is the semantic-weighted MSE of all sub-images before user v state oU v,t . Since the DNC method can only process A taking action at . The second term Dk,v is the semantic- discrete action, as well as the action space of the user is small, weighted MSE of all sub-images after taking action at . The we do not use the DNC method to process user actions. P P P Z(x̂jk,v ) n The Critic Networks of the BS: The critic network of the BS third term eRt = v∈V n k,j 1(αkj,v (t) Rn (t) > △t) is the number of sub-image transmissions whose de- is a three multi-layer perceptrons (MLPs) that approximate a F lay exceeds = local Q function QψB (oBv,t , aBv,t ) to find a discrete action aBv,t P P P the threshold. The final term et B v∈V n k,j 1((βD,v (t) + βC,v (t))τv > △tf ) is the num- from the continuous action âv,t and then estimate the expected
7
Algorithm 1 Dynamic Neighborhood Construction B y, ϵ, cβ , āBv,t ← round(âBv,t ), āB⋆ v,t ← āt , Y = ∅
Initialize: 1: while y > 0, or ϵ > 0 do 2: Find neighbors A to āBv,t with Pij 3: Get Q-values for all neighbors in A 4: Obtain set Y ′ with y-best neighbors, Y ← Y ∪ Y ′ 5: y 1 ← Y ′ , with y 1 = arg maxy∈Y ′ QψB (oBv,t , y) 6: if QψB (oBv,t , y 1 ) > QψB (oBv,t , āB,t )) then 7: Accept y 1 ∈ Y ′ , āBv,t ← y 1 8: if QψB (oBv,t , y 1 ) > QψB (oBv,t , āB⋆ v,t ) then 9: āB⋆ v,t ← y 1 10: end if 11: else if rand() < exp[−(QψB (oBv,t , y 1 ) − QψB (oBv,t , y 1 ))/ϵ)] then 12: Accept y 1 ∈ Y ′ , āBv,t ← y 1 , ϵ ← ϵ − cβ 13: else 14: Reject y 1 ∈ Y ′ , āBv,t ← y rand ∈ Y 15: end if 16: y ← ⌈y − ck ⌉ 17: end while 18: Return āB⋆ v,t
cumulative reward from the given state oBv,t and action aBv,t , where ψ B are the parameter vectors of the MLPs. The Critic Networks of users: The critic network of a user is also MLPs that approximates a local Q function U QψUv (oU v,t , av,t ) to estimate the expected cumulative reward U U from the given state oU v,t and action av,t , where ψ v are the parameter vectors of the MLPs. Global Critic Networks: The global critic network Q function QG (ot , at ) is used to estimate the total rewards under a joint state ot and a joint action at of all agents. Here we use the critic network of the BS and users to approximate a global Q function in two different methods. First, we can summarize the Q functions of all agents to approximate the global Q function as done in VDN method [23]: X U QψB (oBv,t , aBv,t ) + QψUv (oU QG (ot , at ) = v,t , av,t ). (11)
Algorithm 2 VDAC-DNC for semantic-weighted MSE minimization Initialize: Local Q function QψB of BS and QψUv of each user v, parameters Φ of Γ Φ . 1: for each episode do 2: for each user v do 3: BS observes the local state oBv,t Generate a continuous action âBv,t ← π ϕB (oBv,t ) using the 4: BS actor network. B 5: āBv,t ← âBv,t , āB⋆ v,t ← DNC(āv,t ) B B 6: Calculate local QψB (ov,t , av,t ) 7: end for 8: for each user v = 1, 2, . . . , V do 9: User v observes the local state oUv,t 10: Select an action aU,v,t using the user actor network. 11: Calculate local QψUv (oUv,t , aUv,t ) 12: end for 13: Calculate the loss Lc (ψ Uv , ψ B , Φ) 14: The BS updates its actor network using (19) and updates its critic network using (17) 15: for each user v do 16: Update the local actor network using (20) and update the critic network using (18) 17: end for 18: Update the hypernetwork using (21) 19: end for
Step 1: The BS uses the continuous action âBv,t to generate a discrete action āBv,t by rounding each element in âBv,t to its closest discrete value, which is expressed as 1 āBv,t = âBv,t + , (13) 2
where ⌊⌋ is the floor function. Step 2: The BS searches the neighboring actions of āBv,t to find a better discrete action. In particular, we first define a perturbation matrix P = (Pij )i=1,...,U ; j=1,...,2lU ∈ RU ×2lU , where U is the action dimension and l is the range of neighboring actions we will search for each action. Its first lU columns represent the positive search of the discrete action v āBv,t . In particular, for each column, one element of the discrete Second, we can use a MLP model with parameters Γ Φ to action will be increased by step size ϵ. The last lU columns of aggregate local Q functions of all agents as done in QMIX [32]. the matrix represent negative movements of the discrete action Hence, the global Q function QG (ot , at ) is āBv,t . In particular, if l = 1, the matrix P is QG (ot , at ) = Γ Φ (QψB , QψU1 , . . . , QψUv ). (12) ϵ 0 · · · 0 −ϵ 0 · · · 0 0 ϵ · · · 0 0 −ϵ · · · 0 P = . . . (14) .. .. .. . .. . . ... B. Training of the VDAC-DNC .. .. . . . .
Next, we introduce the entire training process of the proposed VDAC-DNC algorithm to solve problem (6). We first introduce the use of DNC method to select the BS discrete action aBv,t that can achieve the highest Q value among the neighbors of the continuous action âBv,t generated by the BS actor network. Since the DNC method can only process discrete action, as well as the action space of the user is small, thus we do not apply the DNC method to process the user action. Then, we explain the loss function of the VDAC-DNC scheme and the training process. 1) Discrete action selection: The procedure of using the DNC method to find a discrete action is summarized as follows:
0
0
···
ϵ
0
0
···
−ϵ
Let Ā = [āBv,t , āBv,t , . . . , āBv,t ] ∈ ZU ×2lU be a matrix that consists of 2lU action āBv,t . Then, 2lU neighboring actions of āBv,t are represented by A = Ā + P = [āBv,t,1 , āBv,t,2 , . . . , āBv,t,2lU ].
(15)
From (15), we see that each neighboring action in A only has one element that is different from āBv,t , which guarantees that the action search will only move in one direction per step. Step 3: We utilize the critic to obtain Q-values for neighboring actions in A. Let Y be the initial overall action set. Among
8
these neighbors, we select the y actions with the largest Q- IV. C ONVERGENCE , I MPLEMENTATION , AND C OMPLEXITY values and sort them in descending order, forming an ordered A NALYSIS action subset Y ′ . We then add Y ′ to Y. From Y ′ , we select Next, we analyze the convergence, implementation, commuaction y 1 , which has the highest Q-value in Y ′ . nication overhead, and complexity of the proposed VDAC-DNC Step 4: We use an iterative simulated annealing (SA)method. based [33] search scheme to explore the neighborhoods in A and find the action aB⋆ v,t that can achieve the highest Q value. SA is a probabilistic search method which performs A. Convergence Analysis worse actions instead of performing the best actions in a To analyze the convergence of the proposed VDAC-DNC alspecific probability. In particular, throughout the search process, B U gorithm, we first define θ = ϕ , {ϕ } as the parameters v∈V v we reduce ϵ by adjusting the cooling parameter cβ , thus of the BS and users. The joint policy factorizes as escaping local optima and becoming more robust than greedy Y ′ search. In particular, if the Q-value of y 1 ∈ Y exceeds B B B U U U πθ (at |ot ) = πϕ (22) B (av,t |ov,t )πϕU (av,t |ov,t ), v the Q-value of the base action āBv,t , we accept y 1 as new v∈V discrete action āBv,t . If not, we still accept it with probability The global action-value function under policy πθ is denoted exp[−(QψB (oBv,t , āBv,t ) − QψB (oBv,t , y 1 ))/ϵ]. π Steps 2-4 are repeated until the algorithm reaches the by Q θ (o, a) and is approximated by the value-decomposition critic QG (ot , at ). We consider the discounted return objective predefined number of iterations. Finally, we obtain the action B⋆ āv,t with the highest Q-value. Then the BS combines actions ∞ hX i ⋆ āB⋆ J(θ) = Eπθ γ t r(ot , at ) , (23) v,t and uses action āB,t to interact with the user. The specific process of DNC algorithm is summarized in Algorithm 1. t=0 2) Critic network and actor network update: Next, we where γ is the discount factor that weights future rewards. introduce the update of the local critic, the actor networks, and Then, the convergence of the designed method is analyzed in the MLP used to aggregate local Q functions. the following lemma. • Local Critic Network Update: The local critic network at πθ a user or the BS is updated by minimizing the temporal Lemma 1. If (i) E[QG (ot , at )] = Q (o, a), (ii) learning rate sequences difference (TD) error, which is defined as P{λa,k } and P{λc,k } satisfyPthe 2Robbins–Monro P 2 conditions: λ = λ = ∞, λ < ∞, B k a,k k c,k k a,k k λc,k < Lc (ψ U v , ψ , Φ) = ∞, λa,k /λc,k → 0, and (iii) the update of the proposed MARL E[rt (ot , at ) + γQG (ot+1 , at+1 ) − QG (ot , at )]. (16) can be considered as the update of a single-agent actor–critic policy-gradient update, our proposed VDAC-DNC algorithm Given (16), the parameters of local critic network is can converge to a locally optimal policy [34]. updated by a stochastic gradient descent (SGD) method Proof. To prove the convergence of the proposed VDAC-DNC as follows: 1 2 U B algorithm, we need to prove that the proposed VDAC-DNC B B ψ ← − ψ − λc ∇ψB Lc (ψ v , ψ , Φ), (17) algorithm satisfies conditions (i), (ii), and (iii). First, the error 2 1 2 U B introduced by the Q function approximation decreases as U U ψv ← − ψ v − λc ∇ψUv Lc (ψ v , ψ , Φ), (18) the neural networks are trained over time. Meanwhile, the 2 B value decomposition method is used to generate the global Q B where λc is the learning rate, ∇ψ Lc (ψ ) and U value, ensuring that the difference between the actual global Q∇ψUv Lc (ψ v ) are the gradients of the critic networks of function and its approximated value remains bounded. Thus, the the BS and users. proposed VDAC-DNC algorithm satisfies condition (i) when • Actor Update: The actor network parameters of both BS the algorithm converges. Note that the DNC module will not and the users are updated using policy gradient ascent, as affect the convergence of the designed method since the DNC follows: X aims to find a better action from the original action space B B B ϕB ← ϕB + λa Lc ∇ϕB log πϕ (19) instead of creating new actions. B (âv,t |ov,t ), v∈V Next, we prove that the proposed method meets condition (ii). U U U U ϕU ← ϕ + λ L ∇ϕUv log πϕ (20) Here, the designed method can meet condition (ii) via adjusting U (av,t |ov,t ), a c v v v the value of the learning rate λa and λck . For example, let where λa is the learning rate, and ∇ϕB logπ ϕB (ât |ot ) and λ = a/(k + 1)α and λ = c/(kk + 1)β with 1/2 < β < α ≤ ak ck U ∇ϕUv log πϕ U are the gradients of the actor networks of 1. Then, the first and second conditions in (ii) can be satisfied v the BS and users. as • MLP Update: The parameters of the MLP used to ∞ ∞ X X approximate the global Q function is also updated using λa,k = a (k + 1)−α = ∞, when α ≤ 1, (24) gradient ascent, as follows: B Φ ← Φ − λh ∇Φ Lc (ψ U v , ψ , Φ),
where λh is the learning rate.
(21)
k=0
k=0
∞ X
∞ X
k=0
λ2a,k = a2
k=0
(k + 1)−2α < ∞,
when
α > 21 , (25)
9
With regards to the third condition in (ii), since 12 < β ≤ 1, we have a λa,k = (k + 1)β−α → 0, when α > β. (26) λc,k c Thus, the proposed VDAC-DNC algorithm can satisfy condition (ii) by setting specific learning rates for the actor and critic networks. To prove that the proposed method meets condition (iii), we need to rewrite the expected gradient as a standard single-agent actor-critic policy gradient. Given (23) and the policy-gradient theorem for discounted Markov decision processes (MDPs) [35], [36], we have h X i ∇θ J(θ) = Eπθ ∇ϕi log πϕi (ai,t |oi,t ) Qπθ (ot , at ) ,
by an offline training method, it does not need any data to update the neural network models, which significantly reduces the implementation complexity. The designed method can be deployed in practical scenarios since 1) the proposed framework explicitly considers practical wireless constraints and 2) the information required by the proposed method can be easily obtained in practical systems. C. Complexity Analysis
The computational complexity of the proposed VDAC-DNC scheme consists of three main components: 1) the training of actor network, critic network, 2) the DNC process and 3) the training of global Q function at the BS. First, we analyze the complexity of training the actor B U i∈{B}∪V and critic network. Let NQ , NQ , NπB , NπU be the numbers (27) of parameters of the actor and critic networks. Then, due where ϕi denotes the local actor parameters of agent i (BS or to the parallel training across the BS and the users, the user). Then, the expected gradient of J(θ) can be written as complexity of training the actor and critic networks per iteration n o h i Y B B U U πθ is O max N + N , N + N , where max{·} implies π π Q Q ∇θ J(θ) = Eπθ ∇ϕi log πϕi (ai,t |oi,t ) Q (ot , at ) . that the training complexity of the actor and critic networks i∈{B}∪V B and the training (28) depends on the training complexity NπB + NQ U U By using the factorization (22), the (28) can be written as complexity NQ + Nπ at a user [37], [38]. h i Then we analyze the computational complexity of the DNC ∇θ J(θ) = Eπθ ∇θ log πθ (at |ot ) Qπθ (ot , at ) . (29) process. As shown in Algorithm 1, the BS checks the Q-values of neighboring actions using its critic networks in each iteration. From (29), we see that the expected gradient can be considered B Thus, the complexity of the DNC per iteration is O NQ . Let as the update of a single-agent actor-critic policy. Consequently, y0 and ϵ0 be the initial values of y and ϵ in Algorithm 1, and the proposed VDAC-DNC algorithm satisfies condition (iii). let ck and cβ be the corresponding decrement steps. Then the This completes the proof. Our simulation result in Section V total number of DNC iterations is bounded by [39] also demonstrates that the proposed VDAC-DNC algorithm ϵ0 y0 effectively converges. , , (30) KD ≤ max ck cβ B. Implementation Analysis
where ⌈·⌉ is the ceiling function. Consequently, the complexity B The implementation of the VDAC-DNC scheme for image of the DNC process at an RL iteration is O KD NQ . With regards to the training of the global Q function, semantic transmission consists of an offline training stage and an online implementation stage. In the offline training stage, if we use (11) to approximate the global Q function, the the BS first allocates the sub-channels for each user using the computational complexity is O(V ) since we only need to designed DNC method as shown in Algorithm 1 and sends summarize the Q functions of all agents. On the other hand, the sub-images to users. Then it requires the 1) sub-image if we use (12) to approximate the global Q function, the transmission status of each user, 2) CSI and the transmission computational complexity of the global Q function depends on M rate of each channel to calculate local QψB (oBv,t , aBv,t ) for the MLP architecture. Let NQ be the number of parameters training its critic network. Once users receive the sub-images in the MLP. The training complexity of the global Q function M sent by the BS, each user requires 1) the current CSI, 2) the is O(NQ ). Given the aforementioned analysis results and assuming we received signal, and 3) the current remaining power budget to calculate the local Q values for training its critic network. use (12) to approximate the global Q function, the overall U complexity of VDAC-DNC per iteration is Then, each user transmits the local QψUv (oU v,t , av,t ) to the BS for the calculation of the global Q function QG (ot , at ). In B B U M O KD NQ + max NπB + NQ , NQ + NπU + NQ . (31) U U particular, the BS aggregates local Q values QψUv (ov,t , av,t ) B B of all users and the local QψB (ov,t , av,t ) of the BS using (11) V. S IMULATION R ESULTS or (12). Based on the state and action information, as well as the global Q value, the parameters of the critic network and For our simulations, we consider a network consisting of actor network of the BS and users are updated according to one BS and V = 15 users. The distance between each user and (17)–(20). Note that, in the training procedure of the proposed the BS is randomly selected within the range of [8, 50] m, and D method, we update only the actor and critic networks. We do the downlink interference In,v is randomly selected within the −9 −8 not need to train DNC since it is an action-search module range of [10 , 10 ] W. Other parameters are summarized in and does not contain trainable neural network parameters. Table I. To validate the performance of our proposed scheme, In the online stage, since the designed method is trained we consider three benchmarks: 1) VDAC (w/o DNC) that
10
TABLE I S IMULATION PARAMETER [42]. Parameters B PD N0 T V D
Values 1.8 × 105 Hz 1W 10−9 W 6 15 4
Parameters N0 N ξ l ε pmax
Values 1 × 10−9 W 6 0.2 10 1 5W
Fig. 4. Accumulated reward versus the number of users VDAC w/o DNC VDAC-DNC
4
Convergence time (×10 s)
4
3
2
1
Fig. 3. Convergence of the considered algorithm 0
4
8
12
16
20
Number of users optimizes sub-image allocation and feedback selection using the designed RL method but does not use DNC to find better Fig. 5. Convergence time with and without DNC under different user settings actions per iteration [32], 2) a DNC based MAQAC algorithm that consists of a centralized critic and decentralized actors [40], and 3) DNC-based independent Q algorithm in which each considered methods at convergence vary when the number of user utilizes a deep Q network (DQN) to optimize its transmit users ranges from 4 to 20. In this figure, we see that the rewards power and feedback selection without considering the actions of all the schemes increase as the number of users increases, of other users [41]. Both the proposed VDAC-DNC and VDAC since the number of images to be transmitted increases. Fig. 4 (w/o DNC) methods use a MLP to approximate the global Q also shows that the gap in terms of the reward between the function [32]. Baseline 1) is used to show the benefits achieved proposed VDAC-DNC scheme and the MAQAC increases as by the designed DNC. Baseline 2) is used to compare value- the number of users increases. This is because our designed decomposition training with a centralized critic compared to method uses a MLP to nonlinearly approximate the global a distributed critic. Baseline 3) is used to show the benefit of Q function using the local Q function of the BS and users, collaborative MARL over independent trained agents. which provides a structured and scalable approximation that Fig. 3 shows the convergence of the considered algorithms remains effective as the number of users grows. In contrast, during the training process. The shaded area represents half the MAQAC scheme relies on a high-dimensional centralized of the standard deviation of the accumulated training reward. critic taking into account the states and actions of all the agents From Fig. 3, we see that as the number of training episodes to directly approximate the global Q-function. As a result, this increases, the rewards of all considered schemes increase first critic becomes harder to train and more prone to overfitting and then stabilize. In particular, VDAC-DNC improves the as the number of users increases. Fig. 4 also shows that the reward by up to 5.04% and 18.55% compared to the MAQAC proposed VDAC-DNC scheme can enhance the performance and independent Q when the number of episodes is 14000. by up to 16.57% compared to the independent Q scheme when The 5.04% gain stems from the fact that the proposed method the number of users is 20. The 16.57% gain stems from the used a MLP to approximate the global function using local fact that the proposed method allows users to coordinate their Q functions of the users and BS, while the MAQAC uses a feedback transmission, preventing a large number of users from high-dimensional centralized critic to approximate a global Q sending feedback in the same time slot thus reducing feedback function, causing overfitting. The 18.55% gain stems from the transmission speed. fact that the proposed method approximates a global Q function Fig. 5 shows the convergence time of the proposed scheme to update the local Q function of each agent, thus allowing with DNC and without DNC under different user settings. From the users and the BS to collaboratively select feedback and this figure, we see that the convergence time of both schemes sub-image transmission to maximize their team reward in (9) increases as the number of users increases. This is because instead of their individual rewards. the joint action space grows significantly with the number of Fig. 4 shows how the accumulated rewards resulting from the users, requiring the MARL to spend more time for exploration
11
since the packet loss rate of one shot transmission is 1 (i.e., all the image transmitted are lost), we cannot calculate its transmission time when the downlink power is 1 W. Table II also shows that the one shot transmission method reduces the transmission time by up to 19.03% compared to the proposed method when the downlink power is 3 W. This is because one shot transmission sends the whole image at once, while the proposed method transmits the image sequentially with feedback, which introduces additional transmission time for feedback and sequential sub-image transmission. However, the proposed method can improve the packet loss rate and weighted MSE by up to 84.4% and 55.9% compared to one shot transmission. Fig. 6. Number of feedback transmitted over an episode In Fig. 7, we show the visualization of the sub-image and TABLE II feedback transmission during an image transmission. In the C OMPARISON BETWEEN THE PROPOSED METHOD AND ONE SHOT figure, each × beside the sub-image implies that the sub-image TRANSMISSION . is not received by the user due to poor channel conditions, Packet Loss Weighted Transmission Method whereas each ✓ represents that the sub-image is successfully Rate MSE time delivered to the user. The CSI feedback with a ✓ at a given One shot 1.0 1.0 – transmission (1 W) time step indicates that one sub-image is successfully received Proposed method (1 W) 0.18 0.44 2.43 s by the user at that step. For content-preference feedback, each One shot 0.92 0.92 1.38 s user sends the semantic weights of the top-4 most important transmission (3 W) Proposed method (3 W) 0.14 0.40 1.70 s sub-images that are (i) still missing at the user side and (ii) have not been sent before. This continues until all eligible content and convergence towards optimal actions. Fig. 5 also shows importance weights have been sent. For example, at step 5 that the proposed scheme with DNC improves the speed of of the VDAC-DNC method, all content importance weights convergence by up to 4.16x compared to the proposed scheme have been sent to the transmitter. Hence, no additional content without DNC when the number of users is 16. This is because importance weights needs to transmit. In this simulation, CSI DNC selects an action with the maximum Q value from the distributions are varied every three time slots. From the figure, small sized constructed neighboring action space instead of we see that the proposed method enables the BS to obtain exploring all the actions. the CSI at steps 1 and 4 to maintain a high data rate for In Fig. 6, we show how the number of feedback messages sub-image transmission. Fig. 7 also shows that, at step 2, the transmitted by 8 users at each time slot varies in an episode. CSI feedback in the independent Q scheme is not transmitted From this figure, we see that the proposed method transmits to the BS due to the delay constraint. This is because the around 5 feedback messages per time slot, and the variance uplink bandwidth for feedback transmission is limited, and of the number of transmitted feedback messages over 6 time users in the independent Q scheme do not coordinate their slots is 0.14. In contrast, the independent Q scheme unreliably feedback transmission, causing a large number of users to transmits either 8 feedback messages or 3 feedback messages send feedback simultaneously. From Fig. 7, we also see that with a variance of 3.33, which may either increase feedback under the proposed VDAC-DNC, after the BS receives contenttransmission delay or waste feedback transmission opportunity. preference feedback at step 2, it schedules transmissions more This is because the proposed method enables the users to efficiently at step 3 by sending the second important book collaboratively determine when to transmit their feedback together with a less important shelf. messages thus optimizing the number of feedback messages In Fig. 8, we show how the types of feedback transmitted that should be transmitted under a limited bandwidth channel by the users affect the accumulated reward. In this figure, and meeting the feedback transmission delay requirements. In we compare the proposed VDAC-DNC method with: i) the contrast, under independent Q-learning, users transmit feedback proposed method in which the users only send content independently without accounting for others’ actions. preference feedback to the BS, ii) the proposed method in Table II shows the gains (i.e., packet loss rate, weighted which the users only send CSI feedback to the BS, and iii) MSE, transmission time) achieved by the proposed method the proposed method in which the users remain silent and will compared to the one shot transmission method where the not send any feedback to the BS. From Fig. 8, we see that transmitter directly transmits the entire image without receiving the proposed VDAC-DNC method can improve the reward feedback under different downlink power levels. From Table II, by up to 8.3%, 7.1%, and 13.1% compared to the considered we see that the proposed method improves the packet loss schemes (i), (ii), and (iii). The 8.3% gain stems from the fact rate by up to 82.1% and the weighted MSE by up to 66.4% that the proposed method can send CSI feedback to the BS, compared to one shot transmission when the downlink power allowing the BS to learn the channel status thus improving is 1 W. These gains are achieved by the fact that the proposed sub-image transmission rate. The 7.1% gain stems from the method uses CSI and content preference feedback to optimize content preference feedback transmission allowing the BS to sub-image transmission and sub-channel allocation. Moreover, prioritize important sub-images. The 13.1% gain is achieved
12
Fig. 7. Visualization of sub-images and feedback messages transmission in each step during an episode
Fig. 8. Convergence of the designed methods with different types of feedback
Fig. 9. Accumulated reward versus the DNC search range 23
Accumulated reward
VDAC-DNC by the joint CSI and content preference feedback transmission. Independent Q 22 Fig. 9 shows how the accumulated reward changes as range No feedback l of neighboring actions for DNC varies. In this figure, we 21 see that the gap in terms of the accumulated reward between the proposed method and the independent Q scheme increases 20 as l increases from 1 to 4. This is because, as l increases, 19 the neighboring action space increases. The critic of the proposed method, which is trained using global information, can 18 still select neighboring actions that increase the team reward, thus leading to a mild performance decrease. In contrast, the 17 20% 40% 60% 80% independent Q scheme relies on a locally trained critic and Low-power user distribution (%) selects the neighboring actions that increase the individual reward but may decrease the team reward. This problem Fig. 10. The impact of device heterogeneity on the proposed framework becomes worse as the neighboring action space increases. Fig. 9 also shows that the proposed method with l = 2 improves the fact that when l = 4, the BS must traverse a large number accumulated reward by up to 4.79% and 1.78% compared to of neighboring actions and it may select a neighboring action the designed method with l = 0 and 4. The 4.79% gain stems whose Q-value is overestimated by the critic networks, thus from the fact that the designed method with l = 2 can explore decreasing the accumulated reward. neighboring actions to help the BS find a better discrete action In Fig. 10, we show how the accumulated reward is affected (i.e., a better sub-image allocation policy) than the initially by the power budget of devices. Here, some devices have higher chosen discrete action āBv,t . The 1.78% gain stems from the feedback power budgets (e.g., pU v = 1 W and pmax = 6 W) and
13
the remaining devices have lower feedback power budgets (e.g., pU v = 0.001 W and pmax = 0.006 W). In this figure, all users have the same CSI conditions such that we do not consider the impact of CSI. From Fig. 10, we see that as the number of users with lower feedback power budgets increases from 20% to 80%, the reward of the proposed method increases by up to 11.84%. This is because the devices with high power budgets can transmit feedback more frequently than the devices with low power budgets, thus providing more channel status and user preference information to assist the BS in optimizing the sub-image transmission. Fig. 10 also shows that the reward gap between the proposed method and the independent Q method increases as the power budget of the devices increases. This is because our designed method allows users to coordinate their feedback transmission when users have sufficient power budgets. VI. C ONCLUSION In this paper, we have proposed a novel channel and content preference feedback enabled SC framework in which a BS transmits the images to a set of users while users cooperatively transmit the feedback to the BS for more efficient and accurate image data transmission. This image transmission problem has been formulated as an optimization problem, whose goal is to minimize the semantic-weighted MSE between the original image and the regenerated image via optimizing sub-channel allocation of the BS, power allocation and feedback selection of users. To address this problem, we have designed a VDACDNC scheme that introduces actor-critic to VDN, offering higher training efficiency compared to the standard VDN. In addition, the proposed method with the DNC module is more effective in finding better actions in large discrete action spaces. In particular, the DNC uses a continuous action produced by a continuous policy, which is easier to optimize than a complex discrete policy over large discrete action spaces, to construct a small discrete neighboring action space. Then, DNC searches for an action with the maximum Q value from the small neighboring action space, thus avoiding traversing the large discrete action space. Simulation results have shown that the proposed VDAC-DNC scheme significantly outperforms the baselines with regard to the quality of reconstructed images and convergence time. R EFERENCES [1] C. Chaccour, W. Saad, M. Debbah, Z. Han, and H. Poor, “Less data, more knowledge: Building next-generation semantic communication networks,” IEEE Commun. Surv. Tutor., vol. 27, no. 1, pp. 37–76, Feb. 2025. [2] Y. Liu, H. Du, D. Niyato, J. Kang, Z. Xiong, S. Mao, P. Zhang, and X. Shen, “Cross-modal generative semantic communications for mobile aigc: Joint semantic encoding and prompt engineering,” IEEE Trans. Mob. Comput., vol. 23, no. 12, pp. 14871–14888, Aug. 2024. [3] Y. Sagduyu, S. Ulukus, and A. Yener, “Task-oriented communications for nextg: End-to-end deep learning and AI security aspects,” IEEE Wireless Commun., vol. 30, no. 3, pp. 52–60, Jun. 2023. [4] H. Xie, Z. Qin, and G. Li, “Task-oriented multi-user semantic communications for VQA,” IEEE Wireless Commun. Lett., vol. 11, no. 3, pp. 553–557, Mar. 2022. [5] T. M. Getu, G. Kaddoum, and M. Bennis, “Semantic communication: A survey on research landscape, challenges, and future directions,” Proc. IEEE, vol. 112, no. 11, pp. 1649–1685, Nov. 2024.
[6] M. Yang and H. Kim, “Deep joint source-channel coding for wireless image transmission with adaptive rate control,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Singapore, May 2022, pp. 5193–5197. [7] Z. Lyu, G. Zhu, J. Xu, B. Ai, and S. Cui, “Semantic communications for image recovery and classification via deep joint source and channel coding,” IEEE Trans. Wireless Commun., vol. 23, no. 8, pp. 8388–8404, Jan. 2024. [8] J. Dai, J. Li, Z. Zhao, Z. Yang, Z. Zhang, and M. Shikh-Bahaei, “Energy efficient multi-modal probabilistic semantic communication (PSCom),” IEEE Trans. Green Commun. Netw., vol. 9, no. 4, pp. 1951–1963, Dec. 2025. [9] W. Zhang, H. Zhang, H. Ma, H. Shao, N. Wang, and V. Leung, “Predictive and adaptive deep coding for wireless image transmission in semantic communication,” IEEE Trans. Wireless Commun., vol. 22, no. 8, pp. 5486–5501, Aug. 2023. [10] K. Yang, S. Wang, J. Dai, K. Tan, K. Niu, and P. Zhang, “WITT: A wireless image transmission transformer for semantic communications,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), Rhodes Island, Greece, Jun. 2023, pp. 1–5. [11] S. Li, D. Tuninetti, and N. Devroye, “On the capacity region of the layered packet erasure broadcast channel with feedback,” in Proc. IEEE Int. Conf. Commun. (ICC), Shanghai, China, May 2019, pp. 1–6. [12] S. Li, H. Seferoglu, D. Tuninetti, and N. Devroye, “On the stability region of the layered packet erasure broadcast channel with output feedback,” in Proc. IEEE Inf. Theory Workshop (ITW), Visby, Sweden, Aug. 2019, pp. 1–5. [13] G. Zhang, Q. Hu, Y. Cai, and G. Yu, “SCAN: Semantic communication with adaptive channel feedback,” IEEE Trans. Cogn. Commun. Netw., vol. 10, no. 5, pp. 1759–1773, Oct. 2024. [14] H. Wu, Y. Shao, E. Ozfatura, K. Mikolajczyk, and D. Gündüz, “Transformer-aided wireless image transmission with channel feedback,” IEEE Trans. Wireless Commun., vol. 23, no. 9, pp. 11904–11919, Sep. 2024. [15] J. Ding, P. Jiang, C. Wen, and S. Jin, “Adaptive wireless image semantic transmission: Design, simulation, and prototype validation,” 2024, [Online]. Available: https://arxiv.org/abs/2410.17536. [16] J. Wu, C. Wu, Y. Lin, T. Yoshinaga, L. Zhong, X. Chen, and Y. Ji, “Semantic segmentation-based semantic communication system for image transmission,” Digit. Commun. Netw., vol. 10, no. 3, pp. 519–527, Jun. 2024. [17] Z. Lyu, M. Xiao, M. Skoglund, M. Debbah, and H. V. Poor, “Quantizationaware collaborative inference for large embodied ai models,” 2026, [Online]. Available: https://arxiv.org/abs/2602.13052. [18] R. Xu, G. Li, Z. Yang, J. Kang, X. Zhang, and J. Li, “Covert UAV data transmission via semantic communication: A DRL-driven joint position and power optimization method,” in Proc. Int. Conf. Commun. China (ICCC), Hangzhou, China, Aug. 2024, pp. 66–71. [19] Y. Wang, M. Chen, T. Luo, W. Saad, D. Niyato, H. V. Poor, and S. Cui, “Performance optimization for semantic communications: An attentionbased reinforcement learning approach,” IEEE J. Select. Areas Commun., vol. 40, no. 9, pp. 2598–2613, Feb. 2022. [20] W. Zhang, Y. Wang, M. Chen, T. Luo, and D. Niyato, “Optimization of image transmission in cooperative semantic communication networks,” IEEE Trans. Wireless Commun., vol. 23, no. 2, pp. 861–873, Feb. 2024. [21] W. Zhang, Q. Wu, P. Fan, K. Wang, N. Cheng, W. Chen, and K. B. Letaief, “Semantic-aware resource management for C-V2X platooning via multi-agent reinforcement learning,” 2025, [Online]. Available: https://arxiv.org/abs/2411.04672. [22] G. Dulac-Arnold et al., “Challenges of real-world reinforcement learning: definitions, benchmarks and analysis,” Mach. Learn., vol. 110, no. 9, pp. 2419–2468, Sep. 2021. [23] P. Sunehag et al., “Value-decomposition networks for cooperative multiagent learning based on team reward,” in Proc. Int. Conf. Auton. Agents Multiagent Syst., Stockholm, Sweden, Jul. 2018, pp. 2085–2087. [24] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multiagent actor-critic for mixed cooperative-competitive environments,” in Proc. Neural Inf. Process. Syst. (NIPS), Long Beach, CA, USA, Dec. 2017, pp. 6382–6393. [25] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “Highresolution image synthesis with latent diffusion models,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), New Orleans, LA, USA, Jun. 2022, pp. 10684–10695. [26] W. Wang et al., “InternImage: Exploring large-scale vision foundation models with deformable convolutions,” 2023, [Online]. Available: https://arxiv.org/abs/2211.05778.
14
[27] X. Ding, X. Zhang, J. Han, and G. Ding, “Scaling up your kernels to 31×31: Revisiting large kernel design in CNNs,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), New Orleans, LA, USA, Jun. 2022, pp. 11953–11965. [28] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical vision transformer using shifted windows,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Montreal, QC, Canada, Oct. 2021, pp. 9992–10002. [29] D. Tse and P. Viswanath, Fundamentals of Wireless Communication, Cambridge University Press, 2005. [30] M. Yazid, D. Aissani, L. Bouallouche-Medjkoune, N. Amrouche, and B. Kamel, “Modeling and enhancement of the IEEE 802.11 RTS/CTS scheme in an error-prone channel,” Form. Asp. Comput., Jan. 2015. [31] S. Huang and S. Ontañón, “A closer look at invalid action masking in policy gradient algorithms,” in Proc. Int. FLAIRS Conf., Jensen Beach, FL, USA, May 2022. [32] T. Rashid et al., “Monotonic value function factorisation for deep multiagent reinforcement learning,” J. Mach. Learn. Res., vol. 21, no. 178, pp. 1–51, 2020. [33] M. Kochenderfer and T. Wheeler, Algorithms for Optimization, MIT Press, Cambridge, MA, USA, 2019. [34] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in Proc. AAAI Conf. Artif. Intell., New Orleans, Louisiana, USA, Apr. 2018, vol. 32. [35] R. S Sutton, D. A McAllester, S. P Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Proc. Adv. Neural Inf. Process. Syst., Cambridge, MA, USA, Nov. 1999, pp. 1057–1063. [36] D. Zhou, L. Li, S. Gong, B. Gu, G. Chen, and D. Niyato, “Learning adaptive jamming and beamforming for hybrid IRS-assisted secure NOMA transmissions,” IEEE Trans. Commun., vol. 73, no. 11, pp. 12235–12247, Nov. 2025. [37] X. Xu, H. Xu, D. Wei, W. Saad, M. Bennis, and M. Chen, “Transformerbased collaborative reinforcement learning for fluid antenna system (FAS)-enabled 3D UAV positioning,” IEEE J. Select. Areas Commun., vol. 44, pp. 1128–1143, 2026. [38] D. Wei, X. Xu, Y. Liu, H. V. Poor, and M. Chen, “Optimizing model splitting and device task assignment for deceptive signal-assisted private multi-hop split learning,” IEEE J. Select. Areas Commun., vol. 44, pp. 1512–1528, 2026. [39] D. Bertsimas and J. N. Tsitsiklis, “Simulated annealing,” Statist. Sci., vol. 8, no. 1, pp. 10–15, Feb. 1993. [40] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multiagent actor-critic for mixed cooperative-competitive environments,” in Proc. Adv. Neural Inf. Process. Syst. (NIPS), Long Beach, CA, USA, Dec. 2017, vol. 30, pp. 6382–6393. [41] M. Tan, “Multi-agent reinforcement learning: Independent vs. cooperative agents,” in Proc. Int. Conf. Mach. Learn. (ICML), Amherst, MA, USA, Jun. 1993, pp. 330–337. “Resource allocation for semantic-aware net[42] L. Yan et al., works,” arXiv preprint arXiv:2201.06023, 2022, [Online]. Available: https://arxiv.org/abs/2201.06023.