Stacked Intelligent Metasurface-Based MultiModal Feature Fusion for Collaborative Inference in Edge Networks

1

Stacked Intelligent Metasurface-Based MultiModal Feature Fusion for Collaborative Inference in Edge Networks

arXiv:2610.13364v1 [cs.DC] 9 Oct 2026

Yushi Wang, Zhengxin Yu, Member, IEEE, Guhan Zheng, Member, IEEE, Haris Pervaiz, Senior Member, IEEE, Zheyi Chen, Member, IEEE, and Shunqing Zhang, Senior Member, IEEE

Abstract—In navigating the challenges of computational and communication overhead within the 6G-era Collaborative Inference (CI), this paper explores the feasibility of employing Stacked Intelligent Metasurface (SIM) to perform CI, termed SIM-CI. The SIM functions as a physical computing unit, receiving multi-user Over-The-Air (OTA) aggregated features within the same spectrum. It thus executes the necessary spatial-frequency combining and inference in a single physical wave propagation. SIM-CI also incorporates a tri-heterogeneity robustness adapter to address the network heterogeneity, such as multi-modal data and user inference models, as well as a quality-aware contrastive approach that align features from different modalities and extraction levels into a unified feature space. Considering the heterogeneous and dynamic channels, we further integrate Channel State Information (CSI) as an additional input into the adapter design, enabling users’ early perception of potential channel states. Moreover, to maximally mitigate the impact of channel variations, we design a new hybrid beamforming network that combines the robustness of model-driven pre-equalization with the adaptability of datadriven learning. Experimental results demonstrate the effectiveness of our SIM-CI compared to digital-domain inference, achieving 98.6% accuracy to conversional digital-domain CI approaches under various heterogeneous conditions, while reducing computational energy by 25% and requiring 75% fewer transmission time slots. Index Terms—Collaborative inference, stacked intelligent metasurface, heterogeneity.

I. Introduction Sixth-generation (6G) wireless networks aim to support ubiquitous intelligence [1], where massive numbers of distributed mobile and edge devices, such as autonomous vehicles, unmanned aerial vehicles, and Internet-of-Things devices, require real-time reasoning and decision-making capabilities with the support of edge intelligence [2], [3]. By splitting neural network models, part of an inference task can be executed locally, while the remaining Yushi Wang and Zhengxin Yu are with the School of Computing and Communications, Lancaster University, LA1 4YW, United Kingdom. (e-mail: y.wang216@lancaster.ac.uk; z.yu8@lancaster.ac.uk). Guhan Zheng and Shunqing Zhang are with the School of Communication and Information Engineering, Shanghai University, 200444, China. (e-mail: gzheng@shu.edu.cn; shunqing@shu.edu.cn). Haris Pervaiz is with the School of Computer Science and Electronic Engineering, University of Essex, CO4 3SQ, UK. (e-mail: haris.pervaiz@essex.ac.uk). Zheyi Chen is with the College of Computer and Data Science, Fuzhou University, Fuzhou 350002, China. (e-mail: z.chen@fzu.edu.cn)

computation is offloaded to the network edge, balancing device capabilities and task demands. Recently, multiuser Collaborative Inference (CI) has further emerged as a promising paradigm [4], [5]. For the same inference task, edge infrastructure can aggregate multimodal information collected by heterogeneous users, providing a broader sensing scope and more robust collaborative decision-making while reducing redundant computation. However, in mobile edge networks, distributed users continuously generate and upload heterogeneous intermediate features, which consumes substantial spectrum resources [6]. Digital-domain feature fusion and joint inference at the edge also introduce considerable computation latency and energy consumption [7]. Breakthroughs in the field of Programmable Metasurfaces (PMs), particularly technologies represented by the Stacked Intelligent Metasurface (SIM) [8], offer an innovative route to address the physical bottlenecks described above. SIM directly executes computational tasks during the physical propagation of the signal by programming and controlling electromagnetic waves [9], [10]. This analog computing paradigm has the potential to reduce computation latency and energy consumption introduced by edge digital-domain processing in CI. A. Challenges SIM can provide low latency and low energy for CI, but it also implies that edge networks lose the ability to align, filter, and weight heterogeneous information in the digital domain. This is because SIM replaces the flexible digital processing modules in edge servers with analog superposition at the physical layer. This challenge can become more pronounced in mobile edge networks, where user conditions and available resources may vary over time. Furthermore, given the widespread presence of data heterogeneity, model heterogeneity, and system heterogeneity in the real world [11], [12], the physical computing process based on SIM faces additional obstacles and may render CI results uncontrollable. As a result of data heterogeneity, different users may require similar inference outputs even though their data distributions and content modalities differ. For example, image features produced by onboard cameras and text

2

features collected by roadside units reside in different feature spaces. Directly superimposing these heterogeneous features during signal propagation may cause feature aliasing rather than effective information fusion. Furthermore, model heterogeneity leads to inconsistent feature extraction depths across users with different computing capabilities. Some users may provide shallow texture features while others provide deep abstract concepts. Directly computing these features of varying abstraction in SIM can bias the final inference result. In mobile networks, changes in available computation resources may also cause the feature extraction depth of a user to vary between uploads through early-exit mechanisms [13]. Channel heterogeneity and dynamic variations also pose severe challenges for transmitting heterogeneous features in SIM. Different users experience different wireless channel conditions that fluctuate over time. Thus, the channel ceases to be a passive transmission medium and becomes an uncontrolled analog computation layer that imposes random multiplicative distortion on the transmitted feature vectors. Specifically, channel-induced amplitude fading and phase rotation can destroy the geometric structure of feature vectors, distorting their relative positions and relationships in feature space and thereby causing large offsets in CI results. Especially when SIM serves as the edge node for CI, the lack of intelligent filtering of low-quality features that edge servers normally provide means that corrupted features will be included in the fusion process, undermining the integrity and reliability of inference results. Existing studies have investigated these data, model and system heterogeneity issues mainly in conventional digital-domain CI and edge computing systems through cross-modal alignment [14]–[16], adaptive inference [17]– [19], and predictive or robust beamforming [20]–[22]. These approaches mainly address individual sources of heterogeneity and rely on flexible digital processing. Such flexibility is limited when multi-user features are directly superimposed and processed through SIM, making the joint handling of data, model, and channel heterogeneity challenging. B. Motivation and Contributions To bridge the gap identified above, we propose a SIMbased CI framework, i.e., SIM-CI. It deploys the SIM at the network edge and embeds part of the inference into the physical domain of electromagnetic wave propagation, thereby enabling efficient migration of inference tasks from the digital domain to the physical domain. In SIMCI, a tri-heterogeneity robustness adapter is designed to dynamically coordinate the mapping between heterogeneous data and heterogeneous models to improve inference robustness. Furthermore, to cope with the heterogeneous and time-varying characteristics of wireless channels, SIMCI integrates a dual-robustness mechanism that combines implicit sensing and explicit compensation to further enforce inference accuracy in the physical domain. The main contributions of this paper are summarized as follows:

We propose SIM-CI, a novel physical-domain collaborative inference architecture that seamlessly integrates transmission and computation. SIM-CI preprocesses user feature vectors through a designed triheterogeneity robustness adapter and beamforming networks at the user side, allowing multiple users to transmit features in parallel over the same frequency band. By leveraging Over-the-Air (OTA) computation, these features are aggregated in the analog domain and directly processed by the SIM to perform inference on the superimposed waveform, realizing an integrated transmission-as-computation paradigm. Theoretically, this framework enables inference at the speed of light without incurring digital computation energy consumption, fundamentally reducing the excessive communication and computation overheads in conventional digital collaborative inference systems. • We design a tri-heterogeneity robustness adapter to address data and model heterogeneity in physicaldomain computation. Built on Graph Neural Networks (GNN), the adapter constructs a graph similarity matrix integrating intra-modal and cross-modal relationships to unify heterogeneous graph structures and reduce semantic distances among multimodal features in the latent space. Moreover, we develop a quality-aware contrastive learning mechanism that quantifies features extracted from heterogeneous models into quality indicators and incorporates them into intra-modal and cross-modal contrastive losses. By assigning greater weight to high-quality features, the proposed design aligns features from different modalities and abstraction levels into a unified feature space, enhancing the accuracy and robustness of physical-domain superposition computation. • We propose a dual-robustness mechanism that combines implicit channel sensing and explicit channel compensation to reduce the side effect of channel distortion on SIM-based CI. In the implicit sensing stage, predicted Channel State Information (CSI) is provided as an additional input to the adapter, enabling feature alignment to anticipate channel variations and build intrinsic anti-interference capability in the feature space. To further address residual distortion, we design a novel hybrid beamforming network for explicit compensation with two branches: 1) a model-driven branch performs phase pre-alignment and amplitude pre-equalization for proactive channel compensation and accurate signal superposition; 2) a data-driven branch utilizes a self-attention mechanism to capture complex inter-subcarrier dependencies and learn an adaptive end-to-end mapping function. The proposed dual mechanism encodes channelaware features produced by the adapter into highly robust transmission waveforms, achieving efficient OTA aggregation while significantly reducing the negative impact of channel heterogeneity. •

The rest of the paper is structured as follows: Section II

3

reviews the related work. Section III presents the system model. In Section IV, we describe the SIM-CI framework in details. The performance evaluation of SIM-CI is provided in Section V. Finally, Section VI concludes this paper. II. Related Works Most existing approaches for handling heterogeneous features, computing capabilities, and wireless conditions still rely on conventional digital processing. UNITER [14] learns joint image-text representations by modeling fine-grained correspondences between visual regions and text tokens, while cross-modal graph matching [15] instead captures the structural relations between the two modalities. VSE++ [23] and DeViSE [24] take a simpler route, mapping images and text directly into a shared semantic space. Fragment-level self-attention [25] and large-scale contrastive learning [16] push this idea further by modeling context and semantic correspondence explicitly. These methods align heterogeneous modalities well, but they generally assume the aligned features share a similar level of abstraction. That assumption breaks down once users’ computing budgets start to differ. A separate line of work considers the heterogeneous computing capabilities of edge devices. Adaptive neural networks [17] adjust computation through early exits or network selection. EdgeAdaptor [18] goes a step further, handling online model configuration, model selection, and resource provisioning together for edge DNN inference, while DistrEE [19] distributes early-exit inference across multiple edge devices. These studies mainly focus on adapting local computation to device capabilities, while the fusion of features extracted at different depths at a physical receiver such as SIM remains less explored. Even if model heterogeneity is handled in the digital domain, dynamic wireless channels can still throw aligned features out of balance before they reach the receiver. Channel variation is a well-studied problem on its own. Predictive beamforming adapts transmission to fast-changing vehicular channels [20], and robust WMMSE precoding accounts for channel uncertainty under imperfect CSI [21]. Other work attacks the signal directly, through power control [26] or channel equalization [27], [28]. Transfer and meta learning have also been used to help beamformers adapt as the environment shifts [22], and some end-to-end schemes go further still, jointly optimizing source and channel coding [29] or folding beamforming directly into the semantic coding pipeline [30]. These approaches improve transmission robustness under dynamic wireless conditions, but the channel plays a different role in SIM-based CI. In conventional systems, channel adaptation mainly protects the information being transmitted. In SIM-based CI, channel distortion directly changes the superposition of multi-user features and therefore the input to the physical computing process. As a result, channel robustness must be considered together with feature alignment rather than only during signal transmission.

SIM has also been investigated for multiuser signal processing and semantic communication. An et al. [31] jointly optimized wave-domain beamforming and transmit power allocation for multiuser downlink communications. Papazafeiropoulos et al. [32] developed an MMSE channel estimation protocol for SIM-assisted systems under Rician fading. These studies address communication performance and channel acquisition, rather than the fusion of heterogeneous semantic features. For task-oriented inference, Huang et al. [9] employed SIM to perform source and semantic encoding for image recognition. Their subsequent work [10] combined visual and textual semantic information for scene reconstruction. Li et al. [33] further developed distributed electromagnetic neural networks, where multiple UAV-mounted SIMs collaboratively encode image semantics for over-the-air aggregation and recognition. Although these studies demonstrate multimodal transmission and distributed wave-domain processing, they do not jointly address the alignment of features from different modalities and local model depths together with channel compensation for multiuser inference. The proposed SIMCI framework is designed to address this joint design problem. III. System Model We consider a multi-user CI system comprising K single-antenna users and an edge server equipped with an Nr -antenna uniform planar array (UPA). The users provide heterogeneous observations of a common scene and collaboratively infer the object categories present in that scene. The system employs OFDM with Nc subcarriers. The CI workflow consists of the following phases: A. Data Acquisition and Feature Extraction Each user k ∈ {1, . . . , K} locally acquires heterogeneous raw data, denoted as Dk , which may include visual images Ik or textual descriptions Tk , etc. User k then employs its heterogeneous model to perform part of the inference process and abstract the raw data into a structured set of features. For instance, image and text inputs are processed by the local encoders Ekimg (·) and Ektxt (·), respectively. Their outputs are mapped to a common dimension d through learnable linear projections, yielding visual tokens nk nk r c {rki }i=1 and textual tokens {ckj }j=1 , where rki , ckj ∈ Rd . B. Wireless Communication Model The extracted features are channel-encoded and modulated into complex symbol vectors sk for uplink transmission. The considered multi-user OFDM system is with subcarrier spacing ∆f and sampling period Ts = 1/(Nc ∆f ). The wireless channel from user k to the server is modeled as a frequency-selective channel with Lk distinct propagation paths in a rich scattering environment. The

4

Heterogeneous Data

Heterogeneous Model

Heterogeneous System Different Computing Resources

Image Clients

FasterRCNN

MM Clients

FasterRCNN & BERT

Wireless Channel

Edge Server

Different Channel Conditions

Feature aggregation

1 2 3

4 6 7 8

Text Clients

9

BERT

…

…

…

…

Video Clients

5 SIM

Collaborative Inference Outputs

Fig. 1. The system model for multi-user CI, illustrating the challenges of data, model, and system heterogeneity under the proposed SIM-CI physical computing paradigm.

frequency-domain channel vector on subcarrier n is given A. SIM-CI design by [34]: The SIM consists of Lrx layers, where fixed physical ( ) Lk propagation between layers acts as a linear transformation ∑ 1 2πnτk,ℓ hk [n] = √ αk,ℓ ·ar (θk,ℓ , ϕk,ℓ )·exp −j ∈ CNr ,and trainable meta-atoms apply controllable complexN c Ts Lk ℓ=1 valued modulation [35], [36]. By optimizing the physical (1) states of the meta-atoms, the SIM is trained end-to-end where αk,ℓ ∼ CN (0, σα2 ) is the complex gain of the ℓ-th to process aggregated multi-user features for collaborative path, τk,ℓ is the path delay, and (θk,ℓ , ϕk,ℓ ) is the angle of inference. arrival. 1) Data Acquisition and Feature Extraction: Similar The UPA response vector ar (θ, ϕ) ∈ CNr is formulated to general CI, each user locally acquires heterogeneous as: raw data. User k then employs its heterogeneous model ar (θ, ϕ) = ax (θ, ϕ) ⊗ ay (θ, ϕ), (2) to perform part of the inference process and abstract the where ⊗ denotes the Kronecker product, and the in- raw data into a structured set of features. Nevertheless, dividual array responses for the Nx × Ny UPA (with without the intelligent computing capabilities of the edge, the SIM-CI struggles to process heterogeneous features Nr = Nx Ny ) are: from multi-modal row data and different model depth [ ]T ax (θ, ϕ) = 1, ejπ sin ϕ cos θ , . . . , ejπ(Nx −1) sin ϕ cos θ , (3) levels. We hence designed a novel robust adapter based on GNN. This adapter takes as input the user’s multi[ ]T ay (θ, ϕ) = 1, ejπ sin ϕ sin θ , . . . , ejπ(Ny −1) sin ϕ sin θ . (4) modal token sequences ({rki }, {ckj }), the feature quality indicators (depthkimg , depthktxt ), and the user’s estimated c channel vectors {ĥk [n]}N C. Conventional Edge Processing n=1 , where Nc is the number Nr with Nr being The server receives the signals from all K users. It first of OFDM subcarriers and ĥk [n] ∈ C the number of receive antennas. Here, the indicators executes channel decoding and demodulation to recover k depth ∈ [1, L ] (where L is the encoder’s maxik max max img estimates of the transmitted features, denoted as {r̂i } mum depth) quantify the extraction depth of the input k and {ĉj }. It then executes a CI model g(·) that fuses tokens. The role of the adapter is to project these disparate these heterogeneous features to produce the final inference inputs into a single, unified feature vector gk ∈ RDsem , result as: ( ) K where Dsem is the semantic feature dimension. This nk nk r c . (5) y = g {r̂ki }i=1 , {ĉkj }j=1 vector gk represents an aligned, channel-aware semantic k=1 representation, independent of the original data modality IV. The Proposed SIM-CI Framework or feature depth. The aligned feature gk is passed to a In this section, we present the model of our proposed designed beamforming network tailored for SIM-CI. This SIM-CI. We then detail the core designs for mitigating network aims to address the channel heterogeneity and various network heterogeneities. map the real-valued feature vector gk to a complex-valued

5

symbol vector sk = [sk [1], . . . , sk [Nc ]]T ∈ CNc for OFDM transmission. It takes gk and the channel estimate ĥk [n] as inputs. We consider an imperfect CSI model where the estimated channel ĥk [n] is given by ĥk [n] = hk [n] + ek [n]. The estimation error ek [n] ∼ CN (0, σe2 INr ) accounts for practical impairments such as pilot contamination and feedback delays. The output sk is optimized to jointly encode the semantic information and pre-compensate for channel distortions. 2) Wireless Communication Model: Although SIM has the potential to reduce the computing delay and energy cost, it still faces numerous uplink spectrum resource demands for inference. To further enhance spectrum efficiency during feature transmission, we ingeniously leverage the SIMs’ over-the-air (OTA) computing characteristics. This allows users to transmit simultaneously over the same frequency band before CI, achieving OTA aggregation of multi-user features. The signals from K users naturally superimpose in the wireless multiple access channel. The received signal at the server on subcarrier n is: 1 ∑ hk [n] · sk [n] + z[n] ∈ CNr , y[n] = √ K k=1 K

(6)

where z[n] ∼ CN (0, σn2 INr ) is additive white Gaussian noise (AWGN) with noise power σn2 , and the normalization √ factor 1/ K ensures proper scaling of received power. This physical superposition process is the core of OTA aggregation computation. The wireless channel itself acts as an analog adder, directly fusing features from multiple users in the electromagnetic domain. However, the random amplitude fading and phase rotation from each user’s channel hk [n] can destroy the geometric structure of the features. If channel variations are large, features from weak channels will be overwhelmed. If phases are misaligned, feature vectors will randomly rotate in the complex plane, leading to aggregation failure. This is precisely why the beamforming network must perform pre-compensation at the transmitter. By using amplitude pre-equalization and phase pre-alignment, the network compensates for channel phase rotation and regularizes the transmitted amplitudes. This process allows the aggregated features to maintain their correct geometric relationships. Concatenating the received signals across all subcarriers yields the complete space-frequency signal matrix: Y = [y[1]T , y[2]T , . . . , y[Nc ]T ]T ∈ CNc ×Nr ,

(7)

where each row corresponds to the Nr -dimensional received vector on one subcarrier. This matrix contains all information from K users after feature alignment, beamforming, channel propagation, and physical aggregation, and serves as the input to the receiver SIM. 3) Novel SIM processing: We deploy the SIM at the edge to function as the core of our physical inference engine. It receives the complete OTA-aggregated spacefrequency signal matrix, Y ∈ CNc ×Nr , as its direct analog input. The SIM is trained to function as a powerful, highdimensional linear classifier. It processes the incoming M dimensional (M = Nc × Nr ) wavefront and is optimized

to map different aggregated features to C distinct output ports, where C is the number of target classes. At these output ports, energy detectors measure the signal intensity, yielding the raw classification logits {pc }C c=1 . These logits are then passed to a lightweight digital classification head to produce the final normalized class probabilities, thus completing the entire CI task.

B. Tri-heterogeneity Robustness Adapter based on GNN The proposed adapter is designed for each user to extract a unified feature representation from both heterogeneous data and the outputs of heterogeneous models. It enables robust feature alignment across diverse modalities and abstraction levels. To clarify, we use text and image as input data modalities to illustrate the SIM-CI process. The proposed graph construction is not restricted to two modalities. For a client with multiple available modalities, the similarity matrix is extended with the corresponding intra-modal and pairwise cross-modal blocks, while the pooled representations of all available modalities are concatenated to form the fused feature. The cross-modal contrastive objective is similarly applied to all available modality pairs. We consider the user k who has raw visual tokens r c {ri }ni=1 and text tokens {cj }nj=1 . Here, nr and nc are the number of visual and text tokens, and each feature vector ri , cj is in Rd . The core of the adapter is a GNN, which is designed to align these disparate tokens. To illustrate the GNN’s core modeling logic for handling data and model heterogeneity, let us first define a generic set of initial node representations f (0) . We define them directly from the raw features: fimg,i = ri ∈ Rd ,

(0)

i = 1, . . . , nr ,

(0) ftxt,j = cj ∈ Rd ,

j = 1, . . . , nc .

(8)

The adapter operates on a heterogeneous graph of visual and text tokens. To define the graph’s structure, we build a similarity matrix S ∈ R(nr +nc )×(nr +nc ) based on these initial features f (0) to capture both intra-modal and crossmodal relationships. For intra-modal similarity, we capture the local structure using a k nearest neighbor (kNN) constrained Radial Basis Function (RBF) kernel. For visual tokens, the similarity is:

(SV V )ij =

) (  (0) (0) 2 exp − ∥fimg,i −fimg,j ∥ ,

if j ∈ Nknn (fimg,i ),



otherwise,

2σ 2

0,

(0)

(9) where Nknn (·) is the set of k nearest neighbors and σ > 0 is the RBF kernel bandwidth. The RBF kernel assigns decaying weights to nearby nodes, giving closer features stronger influence. The kNN constraint makes the graph sparse, which reduces computation and preserves

6

Weak Image Feature Extractor Strong Image Feature Extractor

Simplified Faster- RCNN

Contrast Input Image

𝐿𝑖𝑛𝑡𝑟𝑎,𝑖𝑚𝑔

𝑔𝑘

Image Token

Beamforming Network

Deep Faster- RCNN

Contrast Input Image

𝐿𝑐𝑟𝑜𝑠𝑠

OTA Aggregation

Image Token

Channel Encoder CSI Features

Channel Aware

𝐿𝑖𝑛𝑡𝑟𝑎,𝑡𝑒𝑥𝑡

Contrast

Input CSI

Strong Text Feature Extractor Weak Text Feature Extractor

Deep BERT

Concatenate

Message Passing GNN

Text Token

Input Text

Simplified BERT

𝐿𝑠ym

Concatenate

Input Text

Inference Outputs

Similarity Matrix

Text Token

Fig. 2. Detailed architecture of the proposed SIM-CI framework, illustrating the tri-heterogeneity robustness adapter, the beamforming network, OTA aggregation, and SIM receiver for final classification.

the local feature structure. The text token similarity ST T is computed in the same way: ) (  (0) (0) 2 exp − ∥ftxt,i −ftxt,j ∥ , if j ∈ N (f (0) ), knn txt,i 2σ 2 (ST T )ij =  0, otherwise. (10) For cross-modal similarity, we bridge the two modalities. We first compute a raw affinity matrix A ∈ Rnr ×nc using inner products: (0)

(0)

Aij = (fimg,i )T ftxt,j ,

(11)

where Aij measures the alignment between visual token i and text token j. To highlight the best matches while keeping the global context, we combine TopK selection with average pooling. TopK(A, kcross ) selects the top kcross values in each row, and MeanPool(A) computes row averages. These are refined by MLPs and added: SV T = MLPV T (TopK(A, kcross )) + MeanPool(A), (12) ST V = MLPT V (TopK(AT , kcross )) + MeanPool(AT ). (13) The final similarity matrix S assembles these four blocks: [ ] SV V SV T S= . (14) ST V ST T With the graph structure S, the GNN refines node features over L layers using a graph structure biased attention mechanism. This combines learned attention with our pre-computed similarity matrix. The feature fi at layer l + 1 is an aggregation of its neighbors: ∑ (l) (l+1) (l) fi = αij · Wv(l) fj , (15) j

(l)

where Wv is a learnable value matrix. The attention (l) coefficient αij fuses content based attention with the structural bias Sij : ( (l) (l) T (l) (l) ) (Wq fi ) (Wk fj ) √ exp + λ S att ij d (l) ( (l) (l) T (l) (l) ) . (16) αij = ∑ (Wq fi ) (Wk fj ′ ) √ + λatt Sij ′ j ′ exp d (l)

(l)

Here, Wq and Wk are learnable query and key matrices. The hyperparameter λatt balances the learned attention against the fixed graph topology. After L layers, we obtain enhanced node features (L) (L) {fimg,i } and {ftxt,j }. We aggregate these features using a simple pooling operation Pool(·) to get a global representation for each modality: (L)

(17)

(L)

(18)

r zimg = Pool({fimg,i }ni=1 ), c ztxt = Pool({ftxt,j }nj=1 ),

where zimg , ztxt ∈ Rd . The final fused feature representation gk for user k is the concatenation of these pooled features: gk = [zimg ; ztxt ] ∈ RDsem , (19) where Dsem = 2d. This vector is then fed into the beamforming network. To train this GNN and address model heterogeneity, we use a quality-aware contrastive learning objective. We define a quality metric q ∈ [0, 1] based on the feature’s extraction depth: qimg,i = depthimg,i /Lmax ,

qtxt,j = depthtxt,j /Lmax . (20)

7

The intra-modal contrastive loss, Lintra , encourages where [·; ·] denotes concatenation. This initialization similar tokens within a kNN neighborhood to have aligned makes every node in the graph aware of the user’s representations, while pushing dissimilar tokens apart. channel conditions. This channel-aware feature f ′ (0) is then This loss is weighted by the quality q, giving higher quality processed by the same GNN modeling logic, resulting in a tokens stronger influence. For visual tokens: new channel-aware similarity graph S′ and a final robust ′ ′ ∑ = 2(d + demb ). feature representation g′ k ∈ RDsem , Dsem (L) (L) nr q · exp(sim(f , f )) img,j 1 ∑ img,i img,j j∈Nknn (i) ′ This , g k is robust to channel distortions, aligned across Lintra,img = − log ∑n (L) (L) r nr i=1 )) and adaptive to varying abstraction depths, k=1 qimg,k · exp(sim(fimg,i , fimg,kmodalities, (21) and is the vector ultimately fed into the following beamwhere sim(·, ·) is the cosine similarity. The loss for text forming network. tokens, Lintra,txt , is computed similarly: ∑ (L) (L) C. Beamforming Network nc 1 ∑ j∈N (i) qtxt,j · exp(sim(ftxt,i , ftxt,j )) Lintra,txt = − log ∑nknn . Given the feature g′ k from the adapter for each user k, (L) (L) c nc i=1 k=1 qtxt,k · exp(sim(ftxt,i , ftxt,k )) the beamforming network maps it to a frequency-domain (22) symbol vector sk = [sk [1], . . . , sk [Nc ]]T ∈ CNc for transThe total intra-modal loss is the sum of both: mission. We adopt a hybrid approach that combines dataLintra = Lintra,img + Lintra,txt . (23) driven and model-driven strategies to balance adaptability The cross-modal contrastive loss, Lcross , aligns the and robustness. 1) Model-Driven OTA Alignment Symbol Mapping: To global visual and text representations (zimg,i , ztxt,i ) from realize constructive OTA aggregation under heterogeneous the same sample: fading and imperfect CSI, we adopt an MMSE-inspired B qtxt,i · exp(sim(zimg,i , ztxt,i )) 1 ∑ model-driven symbol mapping to mitigate channel disLcross = − log ∑B , tortion in OTA aggregation [21], [37]. The core idea is to B i=1 j=1 qtxt,j · exp(sim(zimg,i , ztxt,j )) (24) represent the multi-antenna reception by a learnable linear where qtxt,i is the average quality of text tokens in sample combiner, form an equivalent scalar channel per client and i. This loss pulls matched pairs together and pushes subcarrier, and obtain a closed-form robust pre-equalizer. The equivalent scalar channel heff,k [n] is formed by mismatched pairs apart, guided by the feature quality. N ×1 to The complete contrastive objective is a weighted sum: applying a reference combining vector v[n] ∈ C r Nr the physical channel vector hk [n] ∈ C . To resolve the Lcontrast = λ1 Lintra + λ2 Lcross , (25) inherent scale ambiguity in transmitter optimization, this where λ1 and λ2 are hyperparameters. This objective combining vector is constrained to have a unit norm, i.e., trains the GNN to produce feature representations gk ∥v[n]∥ = 1. The effective channel is thus defined as: that are aligned across modalities and adaptive to varying heff,k [n] = v[n]H hk [n] ∈ C. (29) abstraction depths. The symmetry regularization loss Lsym ensures the The combining vector v[n] is a robust pre-equalizer cross-modal similarity matrices are consistent: parameter and is jointly optimized with all other network T 2 Lsym = ∥SV T − ST V ∥F , (26) parameters during training. For feature projection, the GNN provides a channel′ where ∥ · ∥F is the Frobenius norm. This prevents asym- aware feature vector g′ ∈ RDsem for user k. We project it k metric relationships from harming training. to a complex symbol per subcarrier via a learnable vector ′ ×1 Finally, to account for channel heterogeneity and the wn ∈ CDsem : dynamics of wireless channels, we make the GNN frames̃k [n] = wnH g′ k ∈ C. (30) work channel-aware by modifying its initial input features. To achieve channel-aware robustness, the adapter incor- The set {w }Nc is trained end-to-end to adapt to the n n=1 porates CSI. The CSI for user k consists of estimated frequency-selective features. Nr c is the channel vectors {ĥk [n]}N n=1 , where ĥk [n] ∈ C After receiving combining, the aggregated scalar signal estimated channel to the Nr antenna server on subcarrier at subcarrier n reads n. These vectors are flattened and encoded by an MLP K 1 ∑ into a compact embedding: heff,k [n] sk [n] + z̃[n], (31) ŷ[n] = √ K k=1 (27) e = MLP(vec(ĥ [n]Nc )) ∈ Rdemb , channel,k

k

n=1

where vec(·) is the vectorization operator and demb is the embedding dimension. In our final model, this channel embedding is concatenated with each raw feature to create (0) the actual initial node features, which we denote as f ′ : f ′ img,i = [ri ; echannel,k ] ∈ Rd+demb ,

i = 1, . . . , nr ,

(0) f ′ txt,j = [cj ; echannel,k ] ∈ Rd+demb ,

j = 1, . . . , nc ,

(0)

(28)

where z̃[n] ∼ CN (0, σ̃n2 ) with σ̃n2 = σn2 ∥v[n]∥2 is the effective post-combining noise variance. The model-driven branch aims to compensate for channel-induced phase rotation and support coherent feature aggregation. To promote coherent aggregation under imperfect CSI, we use a regularized channel-conjugate pre-equalizer. Let ĥeff,k [n] = v[n]H ĥk [n] denote the estimated effective

8

channel, and let ϵk [n] = E[|heff,k [n] − ĥeff,k [n]|2 ] denote its estimation-error variance. The coefficient for user k on subcarrier n is defined as: √ ∗ K ĥeff,k [n] Fk [n] = ∑K ∑K 2 2 j=1 |ĥeff,j [n]| + j=1 ϵj [n] + λ σ̃n (32) √ ∗ K ĥeff,k [n] , ≈ ∑K 2 2 2 j=1 |ĥeff,j [n]| + Kσe + λ σ̃n where the numerator, ĥ∗eff,k [n], performs phase conjugation to coherently align the transmitted symbol with the effective channel for constructive combining. The denominator is a robust normalization ∑ factor that balances the K 2 total multi-user channel power, j=1 |ĥeff,j [n]| , against 2 the regularized noise power, λ σ̃n , and the aggregated channel uncertainty. λ > 0 controls the strength of noise-dependent regularization. We assume the channel estimation error follows an independent and identically distributed (i.i.d.) complex Gaussian distribution with variance σe2 , where simplifies the per-user error variance to a uniform scalar ϵj [n] ≈ σe2 , ∀j, yielding the tractable aggregated uncertainty Kσe2 . This MMSE-based structure prevents excessive power allocation to users with weak channels and reduces to the standard form when CSI is perfect (i.e., σe2 = 0). The model-driven symbol for user k on subcarrier n is then defined as: (m)

sk [n] = Fk [n] · s̃k [n].

(33)

2) Data-Driven Feature-Aware Beamforming: To complement the model-driven approach with enhanced adaptability, we design a data-driven beamforming network that learns the end-to-end mapping from feature vectors to transmission symbols via a Transformer architecture [38], [39]. For discriminative weighting, we apply a learned attention mechanism to emphasize important dimensions of the feature: g̃k = αk ⊙ g′ k , ′

′

(34)

′

where Wα ∈ RDsem ×Dsem and bα ∈ RDsem are learnable parameters, σ(·) is the sigmoid activation function, and ⊙ denotes element-wise multiplication. Next, linear embedding and positional encoding are applied. The weighted feature vector is linearly embedded to the model dimension: ek = Wemb g̃k + bemb ,

ek ∈ Rdmodel ,

(35)

′ dmodel ×Dsem

where Wemb ∈ R and bemb ∈ Rdmodel are learnable parameters. To incorporate frequency information, we expand embedding to subcarriers with sinusoidal positional encoding: Ek [n] = ek + PE[n],

n = 1, . . . , Nc ,

Zk = TransformerEncoder(Ek ) ∈ RNc ×dmodel ,

(36)

where PE[n] ∈ Rdmodel is the fixed positional encoding vector from the original Transformer architecture.

(37)

where each layer consists of multi-head self-attention (MSA) and a feed-forward network (FFN): ( ) (ℓ) (ℓ−1) (ℓ−1) Zk = LayerNorm Zk + MSA(Zk ) , ( ) (38) (ℓ) (ℓ) (ℓ) Zk = LayerNorm Zk + FFN(Zk ) . For complex symbol mapping, we map the contextual representation to complex-valued symbols. This is achieved by mapping the real-valued output to a 2D vector which is then interpreted as the real and imaginary parts of a complex symbol: uk [n] = Wout Zk [n] + bout , (d)

(1)

uk [n] ∈ R2 ,

(2)

sk [n] = uk [n] + juk [n],

(39)

where Wout ∈ R2×dmodel and bout ∈ R2 are learnable parameters. The complete data-driven mapping is denoted as: sk = Tdata (g′ k ; Θd ) ∈ CNc , (d)

(40)

where Θd encompasses all learnable parameters in the data-driven branch. 3) Hybrid Symbol Mapping: To leverage the strengths of both data-driven adaptability and model-driven robustness, we fuse the two symbol mapping strategies via a learnable weighted sum: (d)

(m)

sk = βd sk + βm sk ,

(41)

where the weights are parameterized by learnable scalars θd , θm ∈ R via a softmax function: exp(θm ) . exp(θd ) + exp(θm ) (42) This softmax parameterization ensures βd + βm = 1 and βd , βm ∈ [0, 1]. During end-to-end training, θd and θm are optimized jointly with all other network parameters, allowing the system to automatically learn the optimal balance between the two strategies based on the task objective and channel conditions. 4) Power Normalization: To satisfy the per-user power constraint, we normalize the fused symbol vector: √ Pk · sk , (43) sk ← ∑Nc 2 n=1 |sk [n]| βd =

′

αk = σ(Wα g′ k + bα ) ∈ [0, 1]Dsem ,

c The sequence {Ek [n]}N n=1 is then processed by a Transformer encoder to capture inter-subcarrier dependencies:

exp(θd ) , exp(θd ) + exp(θm )

βm =

where Pk is the power budget of user k. This normalization is applied after hybrid fusion to ensure that the transmitted signal adheres to hardware limitations while preserving the relative amplitude and phase relationships learned by the network.

9

D. OTA Aggregation and CI The symbol vectors sk ∈ CNc from all K users, generated by the hybrid beamforming network, are transmitted simultaneously. These signals propagate through their respective multipath channels {hk [n]} and are naturally superimposed at the server’s Nr -antenna array. As derived in (6) and (7), the server receives the complete spacefrequency signal matrix Y ∈ CNc ×Nr , where the n-th row y[n]T contains the Nr -dimensional vector from all users on that subcarrier. This entire received signal matrix Y is treated as the input wavefront to SIM. In practice, we preprocess Y to match the SIM input layer size via interpolation. We then vectorize the aperture samples to form a single input vector yvec ∈ CM , where M = Nc · Nr : yvec = vec(Y).

(44)

The receiver SIM is modeled as one comprehensive, trainable linear transformation matrix ΘSIM ∈ CC×M , where C is the number of classes. This single matrix is physically realized by the SIM’s multi-layer structure: ΘSIM = Z(Lrx ) W(Lrx ) · · · Z(1) W(1) ,

(45)

where W(l) are fixed physical propagation matrices, and Z(l) are diagonal matrices representing the trainable metaatoms of the l-th layer. Concretely, each W(l) implements free-space diffraction between two parallel planes via the Rayleigh–Sommerfeld diffraction equation, while Z(l) = (l) (l) (l) diag(A(l) ⊙ ejϕ ) with 0 ≤ An < 1 and 0 ≤ ϕn < 2π. The stack is fully differentiable and can be trained by back-propagated gradients to {A(l) , ϕ(l) }. This unified SIM model provides the flexibility to process signals from both the data-driven and modeldriven transmitter paths. The complete received spacefrequency signal matrix Y contains the naturally aggregated contributions from all users, each pre-processed by their respective transmit strategies. The SIM, through its end-to-end trained transformation ΘSIM , learns to perform the necessary spatial-frequency combining and feature classification in a single physical wave propagation. This joint optimization of the transmit side networks and the receive side SIM enables the entire system to discover an optimal end-to-end mapping from heterogeneous user features to classification outcomes, all executed at the speed of light with minimal energy consumption. The SIM processes the input vector yvec to produce a final output signal pout ∈ CC at C designated output points: pout = ΘSIM · yvec .

(46)

The energy at each of the C output ports is then detected, and this energy value pc serves as the logit for the corresponding class c: pc = |(pout )c |2 ,

∀c ∈ {1, . . . , C}.

(47)

Algorithm 1 Training Process of SIM-CI Require: Multi-modal data {Dk }K channel info k=1 , {hk , ĥk }K , labels y true k=1 1: Initialize Parameters GNN ΘGNN , Beamformer Θbeam , SIM ΘSIM 2: for each training iteration do 3: for each user k = 1 to K do 4: Extract tokens {rki }, {ckj } and depthk from Dk 5: gk , Lkcontrast , Lksym ← GNN({rki }, {ckj }, ĥk , depthk ) 6: Lclient ← Lclient + Lkcontrast + Lksym 7: sk ← Beamformer(g k , ĥk ) √ 8: sk ← Pk /∥sk ∥2 · sk 9: end for K 2 10: Y ← OTA({sk }K k=1 , {hk }k=1 , σn ) 11: yvec ← vec(Y) 12: pout ← ΘSIM · yvec 13: {pc } ← {|(pout )c |2 }C c=1 14: p̂ ← σ({pc }) 15: Ltask ← BCE(p̂, ytrue ) 16: Ltotal ← Ltask + λclient Lclient 17: Update ΘGNN , Θbeam , ΘSIM using ∇Ltotal 18: end for 19: return ΘGNN , Θbeam , ΘSIM

For the multi-label task, these logits are passed through a sigmoid function to obtain an independent probability p̂c for each class: p̂c = σ(pc ) =

1 , 1 + exp(−pc )

∀c ∈ {1, . . . , C}.

(48)

The final task loss is the Binary Cross-Entropy (BCE) loss between the predicted probabilities p̂ and the ground truth label vector ytrue : Ltask = −

C ∑

((ytrue )c log(p̂c ) + (1 − (ytrue )c ) log(1 − p̂c )) .

c=1

(49) The total loss function for the entire system is a weighted sum of the task loss and the self-supervised losses from the GNN: Ltotal = Ltask + λcontrast Lcontrast + λsym Lsym .

(50)

V. Performance Results and Analysis In this section, we present the simulation settings and illustrate the superior performance of the proposed SIMCI. A. Experimental Setting Dataset: We evaluate our SIM-CI framework on the COCO2017 dataset [40], a large-scale object detection and classification benchmark containing 118,287 training images and 5,000 validation images across 80 object categories. The dataset is used for multi-label classification

10

tasks, and its multi-modal characteristics make it wellsuited for evaluating multi-user collaborative inference scenarios. To further evaluate modality heterogeneity, we additionally use SpokenCOCO [41], which provides spoken captions associated with COCO images. For the tri-modal experiment, the speech samples are matched with their corresponding image and text samples, providing aligned image, text, and speech observations for the same semantic content. System and Channel Model: We simulated a multi-user OFDM uplink system with a carrier frequency fc = 2.6 GHz, subcarrier spacing ∆f = 60 kHz, and Nc = 64 subcarriers. The system comprises K = 4 single-antenna users and a receiver SIM equipped with an Nr = 64 (8 × 8 UPA) antenna array. Each user’s channel is modeled as a multi-path Rayleigh fading channel with Lk = 2 propagation paths, with angles of arrival randomly distributed. The noise power spectral density was set to N0 = −174 dBm/Hz. Network Architecture: Feature extraction employs pretrained deep learning encoders. For the visual modality, we use Faster R-CNN with a ResNet-101 backbone to extract region features of dimension 2048, extracting nr = 36 regions per image. For the textual modality, we employ the BERT-base-uncased model to generate token-level features of dimension 768. Features from both modalities are mapped to a unified dimension d = 1024 through linear projection layers. Our designed cross-modal GNN adapter adopts L = 1 architecture. For similarity matrix construction, intramodal similarities are computed using a k-nearest neighbor constrained RBF kernel with k = 10 and bandwidth σ = 1.0. Cross-modal similarity computation employs topK selection with kcross = 10. The attention scaling parameter in the message passing mechanism is set to λatt = 1.5, and global feature aggregation uses max pooling. The data-driven branch of the hybrid beamforming network uses a Transformer encoder with dmodel = 256, 4 encoder layers, and 8 attention heads. Training Configuration: The experiments were executed in the Lancaster University HEC GPU cluster using NVIDIA V100 (32 GB) and L40 (48 GB) GPU nodes through the gpu-short / gpu-medium queues. The batch size is set to B = 128. For the adapter loss, the intra-modal and cross-modal components are weighted by λ1 = 0.5 and λ2 = 0.5, respectively. The total loss function balances the task loss with contrastive and symmetry losses using λcontrast = 0.5 and λsym = 0.5. We use the Adam optimizer with a learning rate of 1 × 10−4 . Evaluation Metrics: We use mean Average Precision (mAP) as the primary evaluation metric to measure the accuracy of multi-label classification. B. SIM-CI Accuracy and Computational Overhead In this section, we evaluate the inference accuracy and computational overhead of our proposed SIM-CI framework. Our approach integrates a multi-modal adapter

with a SIM-based physical classifier. We compare its performance against five key baselines chosen to represent different computational paradigms and ablation scenarios. As the ideal digital-domain counterpart, we use BLIP2 [42], which performs multi-modal fusion and classification entirely in the digital domain. To demonstrate the value of multi-modal fusion, we compare against single-modality-only baselines. For the SIM-based singlemodality ablations, we employ SIM-TSC [9] for imageonly inference. For the text-only imference, we create a comparable baseline based on Adapter-BERT [43], where we replace its classifier with our simulated SIM network to process the adapter processed text tokens. For standard digital-domain classifiers, we benchmark against ViT [44] for image classification and RoBERTa [45] for text classification. To quantify the performance and efficiency of the receiver, we designed a digital simulation experiment. We constructed an L-layer SIM model and compared it against an L-layer standard DNN (MLP) baseline of equivalent depth. The fundamental difference between them lies in the parameter space. A DNN is an ideal model possessing O(L · N 2 ) fully trainable, dense weight matrices (W l ) and bias vectors (bl ). In contrast, the SIM model is strictly constrained by physical laws, with its trainable parameters limited to only the O(L · N ) diagonal response matrices Z l. Figure 3 shows the mAP performance during training for all six schemes . The multi-modal schemes (SIM-CI and BLIP-2) clearly outperform the single-modality ones (SIM-TSC, Adapter-BERT, ViT, RoBERTa), validating the need for fusing heterogeneous data. Moreover, our proposed SIM-CI almost perfectly matches the ideal digital BLIP-2 baseline. Both converge at the same rate and reach nearly identical final mAP scores, 0.733 for BLIP-2 and 0.723 for SIM-CI. The figure thus shows that the SIM’s physical structure has enough model capacity to match the ideal digital accuracy. Table I compares six methods across computational energy, spectral efficiency, and computational latency. Computational Energy. The total energy covers both transmitter processing (feature extraction, adapter, beamforming) and receiver operations (SIM’s physical detection vs. digital inference). In the multi-modal case (comparing SIM-CI and BLIP-2), both methods use the same adapter, so the energy gap (0.0060K vs 0.0080K) is due only to the receiver. The BLIP-2 digital baseline needs heavy digital processing, while SIM-CI uses simple energy detection. For single-modality (e.g., comparing SIM-TSC to ViT), the SIM-based method has an adapter while the DNN-based one does not, but SIM-TSC still has lower total energy (e.g., 0.0042K vs 0.0045K for image). This is because the DNN’s receiver cost is much larger than the SIM’s adapter cost. As M increases, the digital layers energy grows linearly because its receiver network scales with the input size. SIM’s cost stays constant, showing its O(1) receiver scaling. Spectral Efficiency. Digital-domain methods (BLIP-2,

11

(a) mAP vs. SNR

0.72

Performance Comparison of Different Architectures

0.8

0.72

0.7

mAP

0.6 0.5

0.68

0.71

0.66

0.705

mAP

mAP

0.7

0.64

0.7

0.62

0.695

0.6

0.69

0.58 6

0.4

8

10

12

14

15

0.685 20

SNR (dB) Fading Only

SIM-CI BLIP-2 SIM-TSC Adapter-BERT ViT RoBERTa

0.3 0.2

(b) mAP vs. Transmit Power

0.715

30

40

50

Transmit Power Pk (dBm) GNN-Aware (No Beamforming)

SIM-CI (GNN-Aware + Beamforming)

Fig. 4. Ablation study on the channel robustness components of the SIM-CI framework. (a) mAP versus Signal-to-Noise Ratio (SNR). (b) mAP versus transmit power (Pk ).

0.1 10

20

30

40

50

60

70

80

90

100

Epoch

Fig. 3. Inference accuracy (mAP) vs. training epoch. Performance comparison of the proposed SIM-CI framework against multimodal (BLIP-2) and single-modality (SIM-TSC, Adapter-BERT, ViT, RoBERTa) baselines.

ViT, RoBERTa) use TDMA, which needs K separate time slots. This means their spectral efficiency drops as the user count K increases, i.e., proportional to 1/K. In contrast, SIM-based methods (SIM-CI, SIM-TSC, Adapter-BERT) use OTA aggregation, allowing all users to transmit at the same time. As a result, its efficiency stays constant regardless of K. When scaling M , both methods improve from array gain by logarithmic growth, but SIM’s better physical-layer beamforming leads to a steeper improvement. Computational Latency. We model latency by assuming that a single floating-point operation (FLOP) takes 1 nanosecond on GPUs, a neural network layer’s computation time scales linearly with its FLOPs, and the transmission duration for one OFDM symbol is 1 millisecond per time slot. Physical wave propagation delay is negligible compared to these processing delays. Because digitaldomain methods (BLIP-2, ViT, RoBERTa) process users serially, their latency grows linearly with K. In contrast, SIM’s parallel OTA keeps its latency constant. In the single-modality case (e.g., SIM-TSC vs. ViT), the SIM’s adapter adds almost no latency compared to the digital receiver cost. That’s why the SIM-based approach is still faster, even with the extra adapter. Finally, as M scales, the digital layer latency grows linearly, while SIM’s stays constant. The SIM physical layer performs inference at the speed of light with O(1) receiver complexity, offering a fundamentally low-energy and low-latency solution. Based on Table I, SIM-CI reduces computational energy by 25%, reduces spectral efficiency by 75%, and reduces computational latency by 76.1% compared to the BLIP-2 digital baseline under our simulation design, where K = 4 and M = 4096.

C. Robustness to Channel Heterogeneity To validate the robustness and effectiveness of the proposed framework against channel heterogeneity, we conducted an ablation study. This study evaluates the performance of the channel-aware GNN and the hybrid beamforming network. We compared the following three configurations: • Fading Only: In this scheme, the GNN does not receive CSI as input, i.e., echannel,k is removed. The beamforming network degenerates to a data-driven(d) only Transformer (sk = sk ) that is also channelunaware. • GNN-Aware (No Beamforming): Here, the GNN receives CSI as input, allowing it to learn implicit robustness in the feature space. However, it still uses the same channel-unaware, data-driven-only beamformer as the baseline. • SIM-CI (GNN-Aware + Beamforming): This is the complete framework proposed in this paper. The GNN is channel-aware, and the hybrid beamforming network performs robust pre-equalization. Figure 4(a) shows the mAP performance as a function of the Signal-to-Noise Ratio (SNR), from 6 dB to 15 dB. As expected, the performance of all three schemes improves with increasing SNR. A consistent performance hierarchy is observed across the entire SNR range, where SIM-CI shows the best performance, followed by the GNN-Aware scheme and the Baseline. The performance gap between the GNN-Aware scheme and fading-only (e.g., 0.672 vs. 0.662 mAP at 10 dB SNR) demonstrates the benefit of making the GNN channel-aware. This allows the GNN to project features into a subspace that is implicitly more robust to channel distortions. The additional performance gain of SIM-CI over the GNN-Aware scheme (e.g., 0.706 mAP at 10 dB) underscores the necessity of explicit signallevel pre-compensation. Figure 4(b) plots mAP against user transmit power Pk , shows the system’s performance bottleneck. The fadingonly scheme’s performance saturates quickly at Pk = 30 dBm, plateauing at an mAP of approximately 0.698. This indicates the system becomes limited by channel distortion, not noise. Once in this state, performance no longer improves with higher power, as the signal misalignment

12

TABLE I Overhead Analysis: Computational Energy, Spectral Efficiency, and Computational Latency. The table presents overhead functions versus the number of users (K) and the receiver input dimension (M ).

Method

Computational Energy (J)

Spectral Efficiency (bit/s/Hz)

vs K

vs M

vs K

SIM-CI

0.0060K

0.024

4.00

BLIP-2 [42]

0.0080K

0.0299 + 5 × 10−7 M

4 K

SIM-TSC [9]

0.0042K

0.0168

2.57

Adapter-BERT [43]

0.0024K

0.0096

1.43

0.0045K

0.0167 + 3 × 10−7 M

0.0025K

0.009 + 2 × 10−7 M

2.57 K 1.43 K

ViT [44] RoBERTa [45]

vs M ( ) M 4 1 + 0.80 ln 4096 ( ) M log2 1 + 4096 ) ( M 2.57 1 + 0.80 ln 4096 ( ) M 1.43 1 + 0.80 ln 4096 ( ) 2.57 M log2 1 + 4096 4 ( ) M 1.43 log2 1 + 4096 4

0.75

0.7

0.7

0.65

0.65

0.6

0.6

0.55

0.55

0.5

0.5 SIM-CI Image-only Text-only

0.45

vs K

vs M

13.10

13.10

13.70K

52.71 + 0.00051M

10.50

10.50

8.70

8.70

11.00K

42.32 + 0.00041M

9.20K

35.41 + 0.00034M

Shallow feature users

0.75

mAP

mAP

Mid-level feature users

Computation Latency (ms)

GCN GAT GraphSAGE

SIM-CI Image-only Text-only

0.45

0.4

GCN GAT GraphSAGE

0.4 0

0.25

0.5

0.75

1

0

0.25

0.5

0.75

1

Proportion of mid-level feature users, pm

Proportion of shallow feature users, ps

Fig. 5. Robustness to model-depth heterogeneity with increasing proportion of mid-level feature users.

Fig. 6. Robustness to model-depth heterogeneity with increasing proportion of shallow feature users.

caused by fading remains uncorrected. The GNN-Aware scheme lifts this performance ceiling to ≈ 0.708, showing that its implicit feature compensation can mitigate some distortion. Finally, SIM-CI achieves the highest plateau (≈ 0.717). This demonstrates that the hybrid beamforming network is essential. By actively performing amplitude pre-equalization and phase pre-alignment, it mitigates the channel distortion, breaking the bottleneck and enabling accurate physical-layer feature aggregation.

the proportion providing shallow-level features, ps , with the remaining users providing deep features. Both pm and ps range from 0 to 1 in steps of 0.25, and for each ratio we average over every possible assignment of depth levels across the four users. We compare (1) our proposed SIM-CI, (2) two single-modality baselines (Image-only, Text-only), (3) three standard multimodal GNN baselines GCN [46], GAT [47] and GraphSAGE [48] , which replace the proposed structural-bias attention with their standard aggregation rules. Under ideal conditions, when all users provide deep features, our model achieves the highest mAP at 0.723. As pm and ps increase, every baseline degrades, and the degradation is markedly sharper once shallow features dominate, since shallow features retain mostly local texture and lexical information with a wide representational gap to deep concepts. In contrast, the SIM-CI remains the most robust across both settings, and its advantage over the baselines widens as the shallow-feature population grows, reaching 0.598 mAP at ps = 1 while the strongest baseline, GAT, falls to 0.537. The GCN, GAT, and GraphSAGE baselines do not explicitly account for feature extraction depth, which limits their ability to handle the

D. Robustness to Model-Depth Heterogeneity Fig. 5 and Fig. 6 show the influence of model-depth heterogeneity on SIM-CI. Users with different computational power extract features at different depths, resulting in varied feature quality. We construct shallow, mid, and deep features directly from different network depths, image features from the res3, res4, and res5 stages of a Faster R-CNN with a ResNet-101 backbone, and text features from layers 3, 6, and 12 of BERT-base. Training uses all three depths jointly, so the adapter learns to align these different abstraction levels. At test time we fix K = 4 users and vary the proportion of users providing mid-level features, pm , and

13

semantic differences between shallow and deep features. This experiment confirms that the tri-heterogeneity adapter is key to solving model-depth heterogeneity in its most realistic form. The quality-aware contrastive objective guides the adapter to align features from different extraction depths, reducing the performance degradation as the proportion of shallow-feature users increases. E. Performance under Modality Heterogeneity To further evaluate the generalizability of SIM-CI beyond the image-text setting used for illustration in Section IV-B, we introduce speech from SpokenCOCO as an additional modality. For each speech sample, we first apply the Montreal Forced Aligner (MFA) [49] to determine word-level time boundaries. Within each aligned segment, a frozen pre-trained HuBERT-base model [50] extracts a 768-dimensional acoustic feature vector. The image, text, and speech features are then projected into dimensional of d = 1024 before being processed by the proposed adapter. We evaluate six client configurations, Image-only, Textonly, Speech-only, Image+Text, Image+Text+Speech, and a mixed client population in which each of the K users independently draws one of the five preceding configurations. For a client with only a subset of modalities, the adapter retains only the nodes and relations for the available modalities, and multi-modal clients form crossmodal connections between every available modality pair. All configurations share the same beamforming network, OTA aggregation, and SIM receiver. We compare SIMCI against Mean Pool, which averages all valid tokens directly, and three standard graph baselines, GCN, GAT, and GraphSAGE, which reuse the same graph nodes and input projections and only replace the aggregation rule. Table II reports mAP for every configuration. Our proposed SIM-CI achieves the highest mAP in every configuration, and its advantage over the baselines is most pronounced once more than one modality is available. Adding modalities consistently improves performance, Image+Text reaches 0.714 mAP and Image+Text+Speech reaches the highest overall mAP of 0.718, while the mixed client population reaches 0.683, below the full tri-modal configuration but above every single-modality configuration. Speech-only gives the lowest mAP among the singlemodality configurations, reflecting the acoustic variability and temporal alignment error in speech representations and their larger semantic distance to the visual class labels. Mean Pool ignores the structural relations between tokens and is the weakest method in every configuration. GCN, GAT, and GraphSAGE recover part of this gap through graph propagation, but their aggregation mechanisms still cannot fully capture the cross-modal relations, image-text, image-speech, and text-speech, that the proposed adapter models explicitly. This is why our advantage over the best baseline grows from singlemodality to multi-modal and mixed configurations, confirming that the tri-heterogeneity adapter, not just the

TABLE II mAP under different client modality configurations. Client Configuration Image-only Text-only Speech-only Image+Text Image+Text+Speech Mixed client population

Mean Pool

GCN

GAT

GraphSAGE

SIM-CI

0.594 0.566 0.502 0.662 0.681 0.629

0.634 0.582 0.521 0.682 0.694 0.660

0.647 0.595 0.534 0.697 0.701 0.669

0.640 0.578 0.527 0.679 0.688 0.657

0.655 0.602 0.543 0.714 0.718 0.683

availability of more data, is what maps heterogeneous client types into a common semantic space. VI. Conclusion In this paper, we propose SIM-CI, a novel framework that deploys SIM to perform efficient CI by embedding computation directly into the physical wireless channel. By aggregating multi-user OTA features in a single wave propagation, augmented with a channel-aware robustness adapter and a beamforming network, SIM-CI effectively addresses data, model, and system heterogeneity. Experimental results show that the proposed method achieves accuracy comparable to digital-domain baselines, reaching 98.6% of the accuracy of conventional digital-domain CI approaches, while reducing computational energy by 25% and requiring 75% fewer transmission time slots, thereby presenting a promising pathway for scalable and lowlatency edge intelligence in 6G systems. References [1] H. Liu, T. Qin, Z. Gao, T. Mao, K. Ying, Z. Wan, L. Qiao, R. Na, Z. Li, C. Hu, et al., “Near-space communications: The last piece of 6g space–air–ground–sea integrated network puzzle,” Space: Science & Technology, vol. 4, p. 0176, 2024. [2] G. Zheng, Q. Ni, K. Navaie, H. Pervaiz, G. Min, A. Kaushik, and C. Zarakovitis, “Mobility-aware split-federated with transfer learning for vehicular semantic communication networks,” IEEE Internet of Things Journal, vol. 11, no. 10, pp. 17237–17248, 2024. [3] G. Zheng, Q. Ni, K. Navaie, and H. Pervaiz, “Semantic communication in satellite-borne edge cloud network for computation offloading,” IEEE Journal on Selected Areas in Communications, vol. 42, no. 5, pp. 1145–1158, 2024. [4] W. Yang, H. Du, Z. Q. Liew, W. Y. B. Lim, Z. Xiong, D. Niyato, X. Chi, X. Shen, and C. Miao, “Semantic communications for future internet: Fundamentals, applications, and challenges,” IEEE Communications Surveys & Tutorials, vol. 25, no. 1, pp. 213–250, 2022. [5] J. Dai, S. Wang, K. Tan, Z. Si, X. Qin, K. Niu, and P. Zhang, “Nonlinear transform source-channel coding for semantic communications,” IEEE Journal on Selected Areas in Communications, vol. 40, no. 8, pp. 2300–2316, 2022. [6] Z. Yang, M. Chen, Z. Zhang, and C. Huang, “Energy efficient semantic communication over wireless networks with rate splitting,” IEEE Journal on Selected Areas in Communications, vol. 41, no. 5, pp. 1484–1495, 2023. [7] G. Zhu, J. Xu, K. Huang, and S. Cui, “Over-the-air computing for wireless data aggregation in massive iot,” IEEE Wireless Communications, vol. 28, no. 4, pp. 57–65, 2021. [8] J. An, C. Xu, D. W. K. Ng, G. C. Alexandropoulos, C. Huang, C. Yuen, and L. Hanzo, “Stacked intelligent metasurfaces for efficient holographic mimo communications in 6g,” IEEE Journal on Selected Areas in Communications, vol. 41, no. 8, pp. 2380–2396, 2023. [9] G. Huang, J. An, Z. Yang, L. Gan, M. Bennis, and M. Debbah, “Stacked intelligent metasurfaces for task-oriented semantic communications,” IEEE Wireless Communications Letters, 2024.

14

[10] G. Huang, J. An, L. Gan, D. Niyato, M. Debbah, and T. J. Cui, “Stacked intelligent metasurfaces for multi-modal semantic communications,” arXiv preprint arXiv:2506.12368, 2025. [11] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al., “Advances and open problems in federated learning,” Foundations and trends® in machine learning, vol. 14, no. 1–2, pp. 1–210, 2021. [12] G. Zheng, Q. Ni, A. Kaushik, and L. Yang, “Semantic communication meets heterogeneous network: Emerging trends, opportunities, and challenges,” arXiv preprint, vol. arXiv:2502.08999, 2025. [13] S. Teerapittayanon, B. McDanel, and H.-T. Kung, “Branchynet: Fast inference via early exiting from deep neural networks,” in 2016 23rd international conference on pattern recognition (ICPR), pp. 2464–2469, IEEE, 2016. [14] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in European conference on computer vision, pp. 104–120, Springer, 2020. [15] Y. Cheng, X. Zhu, J. Qian, F. Wen, and P. Liu, “Crossmodal graph matching network for image-text retrieval,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 18, no. 4, pp. 1–23, 2022. [16] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning, pp. 8748–8763, PmLR, 2021. [17] T. Bolukbasi, J. Wang, O. Dekel, and V. Saligrama, “Adaptive neural networks for efficient inference,” in International conference on machine learning, pp. 527–536, PMLR, 2017. [18] K. Zhao, Z. Zhou, X. Chen, R. Zhou, X. Zhang, S. Yu, and D. Wu, “Edgeadaptor: Online configuration adaption, model selection and resource provisioning for edge dnn inference serving at scale,” IEEE Transactions on Mobile Computing, vol. 22, no. 10, pp. 5870–5886, 2022. [19] X. Peng, X. Wu, L. Xu, L. Wang, and A. Fei, “Distree: Distributed early exit of deep neural network inference on edge devices,” in GLOBECOM 2024-2024 IEEE Global Communications Conference, pp. 3116–3121, IEEE, 2024. [20] F. Liu, W. Yuan, C. Masouros, and J. Yuan, “Radar-assisted predictive beamforming for vehicular links: Communication served by sensing,” IEEE Transactions on Wireless Communications, vol. 19, no. 11, pp. 7704–7719, 2020. [21] J. Shi, A.-A. Lu, W. Zhong, X. Gao, and G. Y. Li, “Robust wmmse precoder with deep learning design for massive mimo,” IEEE Transactions on Communications, vol. 71, no. 7, pp. 3963– 3976, 2023. [22] Y. Yuan, G. Zheng, K.-K. Wong, B. Ottersten, and Z.-Q. Luo, “Transfer learning and meta learning-based fast downlink beamforming adaptation,” IEEE Transactions on Wireless Communications, vol. 20, no. 3, pp. 1742–1755, 2020. [23] F. Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler, “Vse++: Improving visual-semantic embeddings with hard negatives,” arXiv preprint arXiv:1707.05612, 2017. [24] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, “Devise: A deep visual-semantic embedding model,” Advances in neural information processing systems, vol. 26, 2013. [25] Y. Wu, S. Wang, G. Song, and Q. Huang, “Learning fragment self-attention embeddings for image-text matching,” in Proceedings of the 27th ACM international conference on multimedia, pp. 2088–2096, 2019. [26] W. Lee, M. Kim, and D.-H. Cho, “Deep power control: Transmit power control scheme based on convolutional neural network,” IEEE Communications Letters, vol. 22, no. 6, pp. 1276–1279, 2018. [27] C. Lv and Z. Luo, “Deep learning for channel estimation in physical layer wireless communications: Fundamental, methods, and challenges,” Electronics, vol. 12, no. 24, p. 4965, 2023. [28] F. Restuccia and T. Melodia, “Deep learning at the physical layer: System challenges and applications to 5g and beyond,” IEEE Communications Magazine, vol. 58, no. 10, pp. 58–64, 2020. [29] E. Bourtsoulatze, D. B. Kurka, and D. Gündüz, “Deep joint source-channel coding for wireless image transmission,” IEEE

Transactions on Cognitive Communications and Networking, vol. 5, no. 3, pp. 567–579, 2019. [30] M. Wu, Z. Gao, Z. Wang, D. Niyato, G. K. Karagiannidis, and S. Chen, “Deep joint semantic coding and beamforming for nearspace airship-borne massive mimo network,” IEEE Journal on Selected Areas in Communications, 2024. [31] J. An, M. Di Renzo, M. Debbah, H. V. Poor, and C. Yuen, “Stacked intelligent metasurfaces for multiuser downlink beamforming in the wave domain,” IEEE Transactions on Wireless Communications, vol. 24, no. 7, pp. 5525–5538, 2025. [32] A. Papazafeiropoulos, P. Kourtessis, D. I. Kaklamani, and I. S. Venieris, “Channel estimation for stacked intelligent metasurfaces in rician fading channels,” IEEE Wireless Communications Letters, vol. 14, no. 5, pp. 1411–1415, 2025. [33] J. Li, J. An, H. Liu, L. Gan, V. C. Leung, M. Bennis, and M. Debbah, “Distributed electromagnetic neural networks for task-oriented semantic communications,” IEEE Wireless Communications Letters, 2026. [34] Z. Gao, M. Wu, C. Hu, F. Gao, G. Wen, D. Zheng, and J. Zhang, “Data-driven deep learning based hybrid beamforming for aerial massive mimo-ofdm systems with implicit csi,” IEEE Journal on Selected Areas in Communications, vol. 40, no. 10, pp. 2894– 2913, 2022. [35] X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All-optical machine learning using diffractive deep neural networks,” Science, vol. 361, no. 6406, pp. 1004–1008, 2018. [36] C. Liu, Q. Ma, Z. J. Luo, Q. R. Hong, Q. Xiao, H. C. Zhang, L. Miao, W. M. Yu, Q. Cheng, L. Li, et al., “A programmable diffractive deep neural network based on a digital-coding metasurface array,” Nature Electronics, vol. 5, no. 2, pp. 113–122, 2022. [37] Q. Hu, Y. Cai, Q. Shi, K. Xu, G. Yu, and Z. Ding, “Iterative algorithm induced deep-unfolding neural networks: Precoding design for multiuser mimo systems,” IEEE Transactions on Wireless Communications, vol. 20, no. 2, pp. 1394–1410, 2020. [38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [39] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, et al., “A survey on vision transformer,” IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 1, pp. 87–110, 2022. [40] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision, pp. 740–755, Springer, 2014. [41] W.-N. Hsu, D. Harwath, T. Miller, C. Song, and J. Glass, “Textfree image-to-speech synthesis using learned segmental units,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5284–5300, 2021. [42] J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning, pp. 19730–19742, PMLR, 2023. [43] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in International conference on machine learning, pp. 2790–2799, PMLR, 2019. [44] A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020. [45] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. [46] T. Kipf, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016. [47] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” arXiv preprint arXiv:1710.10903, 2017. [48] W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” Advances in neural information processing systems, vol. 30, 2017.

15

[49] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using kaldi,” in Proc. Interspeech 2017, pp. 498–502, 2017. [50] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021.

Related documents

• Stacked Intelligent Metasurface-Based MultiModal Feature Fusion for Collaborative Inference in Edge Networks#1297716
• Energy-Efficient Federated Learning and Migration in Digital Twin Edge Networks#1123923
• Edge-native intelligence for 6G communications driven by federated learning: a survey of trends and challenges#684850
• On the Universal Approximation Capability of Stacked Intelligent Metasurfaces#1130105
• MetaReason: Precise Interleaved Multimodal Reasoning via Editing Meta Information for Solving Geometry Problems#612440