Cooperative Multi-Task Semantic Communication for Joint Classification and Regression Tasks

Cooperative Multi-Task Semantic Communication for Joint Classification and Regression Tasks Ahmad Halimi Razlighi , Mohammad Siddiqur Rahman† , Maximilian H. V. Tillmann , Edgar Beck , and Armin Dekorsy

arXiv:2609.03977v1 [eess.SP] 3 Sep 2026

Department of Communications Engineering, University of Bremen, Germany E-mails:{halimi, tillmann, beck, dekorsy}@ant.uni-bremen.de, † E-mail: mrahman@uni-bremen.de

Abstract—Multi-Task semantic communication (SemCom) prioritizes simultaneous execution of multiple tasks over bit-accurate reconstruction in future intelligent networks. In our prior work [1], we introduced the cooperative multi-task SemCom (CMT-SemCom) framework, in which the semantic encoder is divided into a common unit (CU) and multiple specific units (SUs) to facilitate cooperative multi-task processing. However, CMT-SemCom has been evaluated on homogeneous classification tasks on simplistic datasets, limiting its applicability to real-world perception systems. In this paper, we extend our CMT-SemCom to jointly handle heterogeneous classification and regression tasks on the complex Cityscapes dataset. We adopt the information maximization (InfoMax) principle so that it accommodates mixed discrete and continuous semantic variables. In particular, we benchmark the proposed framework against independent singletask training, a conventional task-agnostic digital transmission, and single-encoder multi-decoder SemCom. Additionally, we investigate the impact of CU capacity on joint task performance, providing design insights. Extensive evaluations demonstrate that CMT-SemCom significantly outperforms the benchmarks. Index Terms—Task-oriented communication, multi-task learning, classification, regression, Cityscapes dataset.

I. I NTRODUCTION The evolution toward intelligent wireless networks is fundamentally reshaping communication paradigms, shifting the design objective from reliable bit transmission to task-oriented information exchange [2]. In this context, semantic communication (SemCom) has emerged as a promising paradigm that prioritizes the transmission of task-relevant semantic information rather than bit-level accuracy [3]. Many practical intelligent systems are required to perform multiple inference tasks simultaneously, like autonomous driving, where semantic segmentation, depth estimation, and scene understanding are jointly executed [4]. Such scenarios motivate multi-task SemCom, in which a single semantic representation is shared among several downstream tasks [5]. Multitask learning (MTL) exploits common latent representations across related tasks to improve generalization while reducing computational redundancy [6]. Motivated by these advantages, recent studies have incorporated MTL into SemCom using deep neural network (DNN) architectures, including a singleencoder on the transmitter (Tx) side and multiple-decoder on the receiver (Rx) side [7], [8]. Nevertheless, these approaches primarily treat the SemCom system as a black-box neural This work was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC3036 The Martian Mindset, project number: 533607631

network and rely on heuristic network designs without an explicit information-theoretic objective. To address this limitation, our previous work introduced the cooperative multi-task SemCom (CMT-SemCom) framework, which establishes a mutual information maximization (InfoMax) foundation for cooperative semantic transmission [1]. Specifically, we modeled a semantic source capable of yielding multiple semantic variables from a shared observation and proposed a split semantic encoder consisting of a common unit (CU) and multiple specific units (SUs) on the Tx. By optimizing an end-to-end (E2E) InfoMax objective through variational approximation, the CU learns semantic features shared across tasks, whereas the SUs preserve task-specific information and channel coding for reliable wireless transmission. Experimental evaluations on homogeneous image classification tasks using the MNIST and CIFAR-10 datasets demonstrated that exploiting statistical dependencies through the CU significantly improves task performance compared with independent task processing [9], [10]. Despite these encouraging results, the applicability of the existing CMT-SemCom framework remains limited. Previous evaluations considered only homogeneous image classification tasks on relatively simple datasets, whereas practical intelligent systems commonly involve heterogeneous tasks with fundamentally different statistical characteristics. In particular, classification and regression tasks frequently coexist in perception applications. For example, autonomous driving systems simultaneously perform semantic segmentation, which predicts discrete semantic labels, and depth estimation, which infers continuous geometric information from the same visual scene [4]. Since these tasks originate from a shared observation while exhibiting complementary semantic dependencies, they constitute a natural application scenario for the CMTSemCom. Joint learning of classification and regression has been extensively investigated in the computer vision and MTL literature from a machine learning (ML) perspective [11]– [13]. More recently, heterogeneous tasks have also been considered in SemCom through learning-based feature sharing mechanisms [14]. However, existing approaches optimize task-specific learning losses directly and therefore lack an information-theoretic formulation that explicitly models the underlying semantic variables and their probabilistic relationships. Consequently, it remains unclear how cooperative semantic representations should be learned when discrete

This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

and continuous semantic variables coexist within a unified communication framework. Addressing this gap is essential for evaluating the scalability and practical applicability of CMTSemCom under realistic multi-task workloads and complex real-world datasets. In this paper, we extend the CMT-SemCom framework to support the joint extraction of task-relevant information for heterogeneous classification and regression tasks from an information-theoretic perspective by modeling discrete and continuous semantic variables. The proposed framework is evaluated on the real-world Cityscapes dataset [15], representing realistic urban perception scenarios. The main contributions of this paper are summarized as follows: • Generalizing the CMT-SemCom framework to accommodate both discrete and continuous semantic variables, executing simultaneous classification and regression tasks through a unified framework. • Benchmarking the proposed framework against singletask training, a conventional task-agnostic digital transmission pipeline, and single-encoder multi-decoder (SEMD-SemCom). • Investigating the impact of CU capacity on multi-task performance, revealing a trade-off between shared representation richness and task-specific interference, and providing design guidelines for optimizing CU dimensionality. II. P ROBLEM F ORMULATION In this section, we establish a probabilistic model to characterize the semantic source and communication aspects. Based on that, we define the overall CMT-SemCom optimization problem for joint classification and regression tasks. A. System Model We consider a multi-task communication system, consisting of a single Tx, multiple Rxs, and a wireless channel in between. The Tx makes an observation S ∈ RH×W ×C , where H, W , and C represent the height, width, and color channels of an input image, respectively. Then it extracts multiple task-specific information via the split structure, each corresponding to one of N downstream tasks. At Rx i, this task-specific information is used to decode a reconstruction of semantic variable Zi to support execution of its task. We denote the tuple (Z, S) as semantic source [1], fully described by its probability density function (pdf) p(z, s), where Z = [ Z1 Z2 · · · ZN ]. Each semantic variable Zi has the alphabet Zi , and as we consider pixel-level semantic perception tasks, semantic variables have the dimensionality of Zi ∈ ZiH×W . The alphabet Zi is either continuous with Zi = R, or discrete with Zi = {‘class1 ’, . . . , ‘classF ’}. We consider the split encoder structure, consisting of a CU and N SUs. Let (z, s) be a semantic source sample, where only s is available to the Tx. Then, the CU preprocesses this observation s for a subset of semantically related tasks, and the i-th SU takes the CU’s output c and computes a length-d codeword xi . This codeword is transmitted over an AWGN

SU Enc. Semantic Source (z, s) z = [z1 z2 · · · zN ] s ∈ RH×W ×3

c s

CU Enc.

SU Enc.

.. .

SU Enc.

x1 C y1 h a x2 n n e xN l

y2

Dec.

ẑ1

Dec.

ẑ2

Dec.

ẑN

.. .

yN

Fig. 1: CMT-SemCom system model.

channel to Rx i. The received signal yi = xi + ni with ni ∼ N (0d , γn2 Id ) is used by decoder i to infer ẑi , with the goal that ẑi = zi . Thus, we have the joint pdf for the i-th semantic link as: p(zi , s, ck , xi , yi , ẑi ) = p(s, zi )pCU (c|s)pSU (xi |c)p(yi |xi )p(ẑi |yi ).

(1)

B. Problem Statement Following the InfoMax principle as in [1], we maximize the mutual information between channel output Yi and corresponding semantic variable Zi to design encoders and decoders.

⋆

⋆

[ p (c|s), p (xi |c)] = arg max

N X

I(Yi ; Zi ).

(2)

p(c|s),p(xi |c) i=1

The optimization variables p(c|s) and p(xi |c) represent the distributions for the CU and SUs, respectively. For simplicity in notation, we represent p(c|s) and p(xi |c) by pCU and pSU i , respectively. III. CMT-S EM C OM FOR J OINT C LASSIFICATION AND R EGRESSION TASKS Expanding the mutual information in our optimization problem, as discussed in detail in [1], the approximated objective function is derived as N X L(θ, Φ, Ψ) = I(Yi ; Zi ) " N (i=1   )# X . ≈ EpCU Ep(s,zi ) EpSU [ log qψi (zi |yi )] θ ϕi i=1 {z } |

(3)

Li (ϕi ,ψi )

In (3), we have employed the variational method using weights in neural networks (NNs) [16]. Thus, our posterior distributions are approximated by NNs, resulting in pCU θ and pSU ϕi , where θ and ϕi represents the NN’s parameters approximating the CU and i-th SU, respectively. We emphasize the joint semantic and channel coding R performed by the encoders as we consider pϕi (yi |si ) = pSU ϕi (xi |c) p(yi |xi ) dxi . For the sake of simplicity we use pSU ϕi = pϕi (yi |si ) in (3). In addition, although the i-th decoder p(zi |yi ) can be fully determined using the known distributions, due to the highdimensional integrals, we approximate it with qψi (zi |yi ), where ψi represents NNs approximating the distribution of the i-th decoder.

To obtain the empirical estimate of the above CMTSemCom objective function, we approximate the expectations using Monte Carlo sampling, considering the training data set of M samples indexed by m (m) (m) D = {(s(m) , z1 , . . . , zN )}M m=1 . In the following, we present the empirical loss functions Li (θ, ϕi , ψi ) for classification tasks and regression tasks separately. To maximize the objective function (3), which is the negative cross-entropy, we estimate the cross-entropy via data samples m and noise samples n, which results in the following maximization objective for the classification tasks:  M  L 1 X 1X (m) (m,l) Classif [ log qψi (zi |yi )] , Li (θ, ϕi , ψi ) ≈ M m=1 L l=1 (4) where we fix the sample size of the encoder output and the channel sampling ym,l = xm + nl , equal for each batch. For implementation, we then minimize the sample-estimated crossentropy instead of maximizing the negative sample-estimated cross-entropy in (4). Finally, for the classification inference step, we take the ⋆ maximum likelihood of the learned qψ (zi |yi ) to get the i estimated semantic variable ẑi (5)

B. Regression Loss Function Optimizing over qψi (zi |yi ) for a regression task is generally computationally infeasible, as this amounts to optimizing over an infinite-dimensional space of continuous pdfs. A practical way to estimate qψi (zi |yi ) is to assume a parameterized continuous distribution shape with a tractable number of parameters [17]. Thus, we assume a pixel-wise independent Gaussian pdf with a tractable number of parameters, given by zik ∼ N (µk , σ 2 ) for the k-th pixel among the K = H × W pixels in the input image. We assume equal variances for all pixels’ pdfs, and then the Mean Square Error (MSE) loss is obtained from (3) [18]. Assuming a given fix variance σ 2 , the output of qψi (zi |yi ), which is represented by a NN, is the mean vector µψi (yi ) ∈ RK of the Gaussian distributions, which depends on input yi . For simplicity in notation, we omit including the ψi in µ(yi ). The pdf qψi (zi |yi ) is fully described by the mean µ(yi ), and the log-likelihood function is given as: log qψi (zi |yi ) !! 2 [zik − µk (yi )] √ exp − = log 2σ 2 2πσ 2 k=1 ! K 2 X
[zik − µk (yi )] K =− − log 2πσ 2 . (6) 2σ 2 2 k=1
2 The constant term K in (6) is ignored in the 2 log 2πσ maximization of the objective function, and we set K = 2σ 2 . K Y

1

1 (θ, ϕi , ψi ) ≈− LRegr i

M  L X 1X

M m=1 L

  (m) (m,l) MSE zi , µ(yi ) ,

l=1

(7) 2 with MSE(zi , µ(yi )) = K k=1 (zik − µk (yi )) . Finally, semantic variable ẑi is estimated using the maximum likelihood criterion by PK 1

A. Classification Loss Function

⋆ ẑi = arg max qψ (zi |yi ). i ẑi

Integrating (6) into the objective function in (3), we obtain the empirical regression loss function as follows:

⋆ ẑi = arg max qψ (zi |yi ) = µ(yi ), i

(8)

ẑi

⋆ which is the mean of qψ (zi |yi ) as it is Gaussian-shaped, and has its maximum at µ(yi ).

IV. S IMULATION R ESULTS In this section, we evaluate the performance of CMTSemCom by comparing its task execution against baselines, in addition to investigating the impact of the CU capacity. A. Simulation Setup 1) Dataset and Tasks: For our evaluations, we use the Cityscapes dataset [15], which contains 5, 000 finely annotated urban street-scene images collected from 50 cities. We use only the fine-annotated images, as the dataset provides both pixel-level semantic labels and stereo disparity maps, making it suitable for simultaneous classification and regression tasks. Since disparity is inversely proportional to depth, we convert the disparity maps into normalized floating-point depth labels using the maximum disparity value. Only pixels with positive disparity values are considered valid, while pixels with zero disparity are treated as invalid and excluded from the depth estimation evaluation. The dataset is shaped as (m) (m) (m) H×W D = {(s(m) , z1 , z2 , z3 )}M , with Z ∈ Z , 1 m=1 1 Z2 ∈ Z2H×W , and Z3 ∈ RH×W . Specifically, the input images we use from the dataset have a resolution of H × W × C = 256 × 512 × 3. We consider the execution of three tasks N = 3. For the first two tasks, we consider semantic segmentation of classes: Z1 = {‘road’, ‘sidewalk’, ‘building’, ‘car’, ‘otherwise’} as Task 1 and semantic segmentation of Z2 = {‘building’, ‘otherwise’} as Task 2. Thus, Task 1 is a multinomial classification, and Task 2 is a binary classification. We consider the third task to be a regression task by defining Task 3 to be pixel-level depth estimation, with an illustrative example of the dataset shown in Fig. 4. Using (4) and (7), the total loss function of the CMT-SemCom is given as: L(θ, Φ, Ψ) = LClassif (θ, ϕ1 , ψ1 ) + LClassif (θ, ϕ2 , ψ2 ) + LRegr 1 2 3 (θ, ϕ3 , ψ3 ).

TABLE I: NN architecture of the CMT-SemCom. CU SU

Dec

Layer Conv (#filters = 96, stride = 2), BatchNorm, ReLU F ResNet blocks 9 − F ResNet blocks Global average pooling Dense (output size = 100), BatchNorm, ReLU Dense (output size = number of channel-uses d = 128), ReLU Power normalization Dense, ReLU, Reshape (to 32 × 64 × 64) ConvT(#filters = 64 , stride = 2), ReLU ConvT(#filters = 32, stride = 2), ReLU ConvT(#filters = 16, stride = 2), ReLU Task 1: Conv(#filters = 5, stride = 1), Softmax Task 2: Conv(#filters = 1 stride = 1), Sigmoid Task 3: Conv(#filters = 1 stride = 1), ReLU

2) CMT-SemCom NN Architecture: Table I summarizes the NN architecture of the proposed CMT-SemCom. The CU consists of one convolutional (Conv) layer followed by F ResNet blocks [19], while each SU contains the remaining 9 − F ResNet blocks, keeping the total depth per task fixed at nine blocks. By varying F , we investigate how distributing feature extraction between the shared CU and task-specific SUs affects task performance, as discussed in Section IV-B. The CU output has dimensions 16 × 32 × 64, which is selected after a preliminary investigation to have the best performance. Each SU then applies a global average pooling layer to produce a compact feature vector, which is processed by two fully connected layers and a power normalization layer before transmission with d channel-uses. The decoder architecture is identical for all tasks except for the final task-specific output layer (Table I). It first projects the received latent representation through a fully connected layer and reshapes it into a 32 × 64 × 64 feature map. Three transposed convolution (ConvT) layers then progressively upsample the features to the original image resolution of 256 × 512, followed by the final task-specific Conv layer. The network is trained for 120 epochs using the Adam optimizer with an initial learning rate of 10−4 . The learning rate decays by a factor of 10−1 every 20 epochs starting from epoch 80. 3) Benchmarks: We compare CMT-SemCom’s performance against three baselines: single-task SemCom (STSemCom), single-encoder multi-decoder SemCom (SEMDSemCom), and a conventional task-agnostic digital transmission. We note that in all cases, we use the same NN architectural components, e.g., 9 successive ResNet blocks per task. However, the properties of the NN architectural components are selected such that the total number of trainable parameters remains equal among the cases for a fair comparison. ST-SemCom: For ST-SemCom, three independent semantic encoder/ decoder are implemented for each task, meaning that there exists no CU shared between them. SEMD-SemCom: We also compare the split structure, CMT-SemCom, against the state-of-the-art MTL in SemCom [5], where there exists only one shared encoder on the Tx side and multiple decoders on the Rx.

Task-agnostic digital transmission (TAD): The benchmark follows a classical separate source and channel coding approach. For source coding, we examined two compression algorithms: JPEG with variable image quality and Huffman coding applied to variable quantization levels of the pixel values. Given our target SNR range and the constrained number of channel uses (d), JPEG performed worse than Huffman coding in terms of image reconstruction quality. Therefore, we use Huffman coding as the source coding algorithm. We consider both color and grayscale image reconstruction, with subsampling factors from 1 to 50 and quantization levels from 2 to 8 bits per pixel, where 8 bits represents the original pixel precision. Here, subsampling reduces the spatial resolution by retaining fewer pixels, while quantization reduces the number of bits used to represent each pixel. For channel coding, an LDPC code is used with a rate between 0.2 and 0.95. Moreover, modulation schemes of 4, 16, and 64-QAM were examined. All these parameters are optimized by grid search to minimize the MSE of the reconstructed image at 0 dB SNR, which resulted in a subsampling factor of 45, grayscale image reconstruction, quantization to 4 bits with Huffman coding, a channel coding rate of 0.254, and the use of 4-QAM. Finally, the reconstructed image is fed into a NN to execute all three tasks, using an architecture similar to that of CMT-SemCom. The NN for the task execution is also trained using the noisy reconstructed images for the best possible inference result. 4) Evaluation Metrics: We evaluate the segmentation tasks by two metrics. First, we consider pixel-level classification accuracy of classes averaged over all image pixels, measuring the fraction of valid pixels whose inferred labels match the corresponding ground-truth labels. Second, we use intersection over union (IoU), which is assessed per class and measures the intersection of the inferred segmentation and the ground truth, divided by the union of true and false positives and false negatives [20]. True positives (TP) are correctly identified object pixels, while false positives (FP) are pixels incorrectly labeled as the object. False negatives (FN) are pixels incorrectly labeled as no object. IoU is defined as: IoU =

TP . TP + FP + FN

Particularly, we use IoU to evaluate Task 2 of ‘building’ segmentation, and for Task 1, we use mean IoU (mIoU), which averages the IoU scores for all the defined classes in Z1 . In addition to the MSE, regression performance is evaluated using the δ1 accuracy [21]. δ1 measures the percentage of predicted depth pixels whose relative error falls within a specific threshold. Specifically, it is the fraction of pixels for which the maximum ratio between the ground-truth depth and the predicted depth is less than 1.25. δ1 =

  K  zk ẑk 1 X I max , < 1.25 , K ẑk zk k=1

where I(·) denotes the indicator function and K is the total number of pixels with valid depth ground truth.

0.006 0.005

(a) Task 1

(b) Task 2

0.92

(c) Task 3 0.009

Pixel-level classif. acc.

Pixel-level classif. acc.

0.80 0.75

0.008

0.88 MSE

0.70

0.007 0.006

0.84

0.004 0.65

20

40

60 80 Epoch

CMT-CU9/SU0

20

100 120

0.005

0.80

40

CMT-CU7/SU2

20

0.004

60 80 Epoch

40

60 80 Epoch

CMT-CU5/SU4

100 120

20

CMT-CU2/SU6

100 120 40

60 80 Epoch

100 120

CMT-CU1/SU8

Fig. 2: Architectural comparison for all three tasks over the training epochs at 0 dB SNR.

B. Evaluations

(a) Task 1

0.55

0.75

0.70

0.45 0.40

CMT-SemCom SEMD-SemCom ST-SemCom TAD

0.65

0.35 0.60 −5

0

5

0.30 −5

10

0.92

0

5

10

SNR [dB]

SNR [dB]

(c) Task 2

(d) Task 2 0.60

0.90

0.55

0.88

0.50

IoU

0.86

0.45

0.84

0.40 0.82

0.35 0.80 −5

0

5

0.30 −5

10

(e) Task 3

0 60 100 80 120 100 120 Epoch 0.010

0

5

10

SNR [dB]

SNR [dB] 0.64 0.62

0.008

0.60

δ1 accuracy

0.009

0.007

MSE

(b) Task 1

0.50

CMT-SemCom SEMD-SemCom ST-SemCom TAD

mIoU

Pixel-level classif. acc.

0.80

Pixel-level classif. acc.

1) Architectural Investigations: First, we investigate the impact of the CU/SU processing split on the balance between shared semantic extraction and task-specific refinement. Fig. 2 shows the task performance over training epochs at 0 dB SNR and d = 128 channel-uses for different CMT-SemCom architectures, ranging from CMT-CU9/SU0 to CMT-CU1/SU8. For instance, CMT-CU9/SU0 has all 9 ResNet blocks in the CU, meaning that most processing is done in the CU with only dense layers in the SU, while CMT-CU1/SU8 shifts most processing to the task-specific SUs with 1 ResNet block in the CU, and 8 ResNet blocks in each SU. As shown in Fig. 2, a balanced distribution of the processing capacity between the CU and the SU provides the best performance, with CMT-CU5/SU4 achieving the best for all three tasks. Therefore, CMT-CU5/SU4 is adopted as the default CMTSemCom architecture in the remaining experiments. 2) Benchmark Comparisons: We now compare CMTSemCom against ST-SemCom, SEMD-SemCom, and TAD over an SNR range as shown in Fig. 3. The training SNR is 0 dB. The comparison uses pixel-level classification accuracy and IoU for Tasks 1 and Task 2, and MSE and δ1 accuracy for Task 3. Increasing the SNR from −5 to 10 dB improves performance across all metrics for the considered methods due to the more favorable channel conditions. The proposed CMT-SemCom framework outperforms STSemCom on all three tasks and all SNR values, indicating that joint processing in the CU helps in extracting the relevant semantic feature for all three tasks. Although SEMD-SemCom achieves the same pixel-level classification accuracy as CMT-SemCom on Task 1, CMTSemCom attains a slightly higher IoU. However, for Tasks 2 and 3, CMT-SemCom consistently outperforms SEMDSemCom. These results indicate that the SUs are crucial for task-specific refinements on the Tx side. Finally, we compare CMT-SemCom with TAD transmission. For a fair comparison, we set the number of channel-uses to 384 = 3 × 128 in TAD. This keeps the comparison symbolwise fair between these methods as we transmit 3 × 128 realsymbols for CMT-SemCom and 3 × 128 complex-symbols for

0.006

(f) Task 3

0.58 0.56 0.54

0.005

0.52 0.004 −5

0.50 0

5

SNR [dB]

10

−5

0

5

10

SNR [dB]

Fig. 3: Task execution comparison of CMT-SemCom vs. benchmarks over SNR.

Input image

R EFERENCES

Task 2

Car Road Sidewalk Building Otherwise

Building Otherwise

|∆| < 0.03 |∆| ≥ 0.07

Far

STCMTSemCom SemCom

Ground truth

Task 1

Near Invalid Task 3

0.03 ≤ |∆| < 0.07

Invalid

Fig. 4: Visualization of CMT-SemCom performance on the Cityscapes dataset trained and evaluated at 0 dB SNR.

TAD. The task execution accuracy after TAD transmission is worse compared to CMT-SemCom, as the image reconstruction quality is low for the limited number of channel-use and 0 dB SNR. The results highlight the advantage of CMTSemCom over conventional task-agnostic digital transmission in task-oriented communication, especially in bandwidthconstrained and low-SNR conditions. Fig. 4 visualizes the performance of CMT-SemCom for all three tasks and compares it with ground truth and STSemCom for one illustrative example of the dataset. For both segmentation tasks, as shown, target classes are detected more accurately than by ST-SemCom, and the detected class masks are less fragmented and show better overlap with the ground truth. Regarding Task 3 for better visualization, the mean absolute error ∆ between z3 and the inferred ẑ3 are shown, where z3 and ẑ3 are normalized to be between 0 and 1. Lowerror regions with 0 ≤ ∆ < 0.03 are indicated in green, moderate-error regions with 0.03 ≤ ∆ < 0.07 are indicated in yellow, high-error regions with 0.07 ≤ ∆ ≤ 1 are indicated in red, and pixels with invalid ground truth are indicated in black. It can be seen that CMT-SemCom has larger low-error regions, less moderate-error regions, and nearly the same higherror regions compared to ST-SemCom. V. C ONCLUSION We extended CMT-SemCom to heterogeneous multi-task perception by jointly supporting classification and regression tasks on the Cityscapes dataset using an InfoMax formulation that accommodates mixed discrete and continuous semantic variables. Simulation results showed that the proposed framework outperforms single-task SemCom, single-encoder multi-decoder SemCom, and task-agnostic digital transmission, while requiring the same communication resources. In addition, we demonstrated that the distribution of feature extraction between the shared CU and task-specific SUs plays a crucial role in balancing information sharing and task specialization, providing practical design insights.

[1] A. Halimi Razlighi, C. Bockelmann, and A. Dekorsy, “Semantic communication for cooperative multi-task processing over wireless networks,” IEEE Wireless Communications Letters, vol. 13, no. 10, pp. 2867–2871, 2024. [2] Y. Shi, Y. Zhou, D. Wen, Y. Wu, C. Jiang, and K. B. Letaief, “Taskoriented communications for 6G: Vision, principles, and technologies,” IEEE Wireless Communications, vol. 30, no. 3, pp. 78–85, 2023. [3] D. Gündüz, Z. Qin, I. E. Aguerri, H. S. Dhillon, Z. Yang, A. Yener, K. K. Wong, and C.-B. Chae, “Beyond transmitting bits: Context, semantics, and task-oriented communications,” IEEE Journal on Selected Areas in Communications, vol. 41, no. 1, pp. 5–41, 2023. [4] L. Chen, Z. Yang, J. Ma, and Z. Luo, “Driving scene perception network: Real-time joint detection, depth estimation and semantic segmentation,” in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2018, pp. 1283–1291. [5] G. Zhang, Q. Hu, Z. Qin, Y. Cai, G. Yu, and X. Tao, “A unified multi-task semantic communication system for multimodal data,” IEEE Transactions on Communications, vol. 72, no. 7, pp. 4101–4116, 2024. [6] R. Caruana, “Multitask learning,” Machine learning, vol. 28, pp. 41–75, 1997. [7] Q. Bian, Z. Bao, C. Dong, X. Xu, Y. Luo, and R. Meng, “Multitask semantic communication for remote sensing,” in 2025 IEEE 5th International Conference on Computer Communication and Artificial Intelligence (CCAI), 2025, pp. 70–75. [8] Z. Zhu, R. Zhang, X. Cheng, and L. Yang, “Synesthesia of machinesenabled multi-task semantic communication system,” IEEE Transactions on Mobile Computing, vol. 25, no. 4, pp. 4632–4647, 2026. [9] A. Halimi Razlighi, C. Bockelmann, and A. Dekorsy, “Semantic communication for cooperative multi-tasking over rate-limited wireless channels with implicit optimal prior,” IEEE Open Journal of the Communications Society, vol. 6, pp. 8523–8538, 2025. [10] A. Halimi Razlighi, M. H. V. Tillmann, E. Beck, C. Bockelmann, and A. Dekorsy, “Cooperative and collaborative multi-task semantic communication for distributed sources,” in ICC 2025 - IEEE International Conference on Communications, 2025, pp. 3966–3971. [11] A. Mousavian, H. Pirsiavash, and J. Košecká, “Joint semantic segmentation and depth estimation with deep convolutional networks,” in 2016 Fourth International Conference on 3D Vision (3DV). IEEE, 2016, pp. 611–619. [12] Z. Zhang, Z. Cui, C. Xu, Z. Jie, X. Li, and J. Yang, “Joint taskrecursive learning for semantic segmentation and depth estimation,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 235–251. [13] L. Hoyer, D. Dai, Y. Chen, A. Koring, S. Saha, and L. Van Gool, “Three ways to improve semantic segmentation with self-supervised depth estimation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 11 130–11 140. [14] X. Yu, T. Lv, W. Li, W. Ni, D. Niyato, and E. Hossain, “Multi-task semantic communication with graph attention-based feature correlation extraction,” IEEE Transactions on Mobile Computing, vol. 24, no. 5, pp. 4371–4388, 2025. [15] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. [16] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013. [17] L. Wasserman, All of statistics: a concise course in statistical inference. Springer, 2004, vol. 26. [18] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016, http://www.deeplearningbook.org. [19] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778. [20] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” International journal of computer vision, vol. 111, no. 1, pp. 98–136, 2015. [21] D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” in Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, ser. NIPS’14. Cambridge, MA, USA: MIT Press, 2014, p. 2366–2374.

Related documents

• Multitask Regression with Pairwise Fusion#1063491
• Towards Faithful and Efficient Semantic Communication: An Ontological Approach#1012319
• Semantic-Aware Task Clustering for Constructive and Cooperative Multi-Tasking#394408
• Unfolding shared semantic representations in vision and language#416986
• GIM: Evaluating models via tasks that integrate multiple cognitive domains#665766