Omni-modal decomposition autoencoders learn full-stack wearable disentangled representations

IEEE TRANSACTIONS AND JOURNALS TEMPLATE

1

Omni-modal decomposition autoencoders learn full-stack wearable disentangled representations

arXiv:2608.07385v1 [cs.LG] 7 Aug 2026

Ioannis N. Ziogas, Student Member, IEEE, Ensieh Khazaei, Bilal Taha, Aamna Al Shehhi, Ahsan H. Khandoker, Senior Member, IEEE, Leontios J. Hadjileontiadis, Senior Member, IEEE, Dimitrios Hatzinakos, Fellow, IEEE

Abstract— Learning disentangled representations is a key requirement for developing versatile, general-purpose, and sustainable models in multi-modal wearable computing. However, existing approaches do not operate as fullstack wearable processors, i.e., they do not simultaneously address task-specific classification performance, disentangled and interpretable representation learning, fusion, and generative modeling of highly heterogeneous multi-modal time series. To address this gap, we introduce Omni-modal Variational Decomposition Autoencoders (OmniDecVAEs), a framework that efficiently learns multi-purpose representations in a unified and scalable manner from arbitrarily many modalities. OmniDecVAEs extend DecVAEs by learning modality-conditioned time-frequency latent subspaces through a multi-view self-supervised decomposition loss and a shared asymmetric autoencoder (AE) architecture. Results on a challenging omni-modal human activity recognition (HAR) setting with up to thirty modalities, demonstrate the ability of OmniDecVAEs to learn full-stack wearable representations. When compared to transformerbased and VAE-based methods, OmniDecVAEs full-stack disentangled representation properties lead to accuracy improvements of 1.01% and 6.75% in activity and identity recognition, respectively. Furthermore, OmniDecVAEs synthesize realistic omni-modal time-frequency data that manifest with enhanced reconstructions (mean absolute error improves by 76.84%) and distributional similarity between real and synthetic data (maximum mean discrepancy Manuscript received August 10, 2026. This research was funded by the Healthcare Engineering Innovation Group (HEIG), Khalifa University of Science and Technology (KU-HEIG). Ioannis N. Ziogas is with the Department of Biomedical Engineering and Biotechnology, Khalifa University, 127788 Abu Dhabi, UAE. Part of this work was conducted during his time as a Visiting Graduate Researcher at the Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON M5S 1A1, Canada (e-mail: ioannis.ziogas@ku.ac.ae). Ensieh Khazaei is with the Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON M5S 1A1, Canada. Bilal Taha is with the MIT Media Lab and Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto. Aamna Al Shehhi is with the Department of Electrical Engineering, Khalifa University, Abu Dhabi, UAE. Ahsan H. Khandoker is with the Department of Biomedical Engineering and Biotechnology, Khalifa University, 127788 Abu Dhabi, UAE. Leontios J. Hadjileontiadis is with the Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki, GR 54124 Thessaloniki, Greece, and with the Department of Biomedical Engineering and Biotechnology, Khalifa University, 127788 Abu Dhabi, UAE (email: leontios{@auth.gr, .hadjileontiadis@ku.ac.ae)}. Dimitrios Hatzinakos is with the Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON M5S 1A1, Canada (e-mail: dimitris@comm.utoronto.ca)

improves by 13.85%). Our results highlight OmniDecVAEs potential as a lightweight model suitable for intelligent edge wearables and clinical healthcare, unifying processing requirements and abilities in a single model, through its enhanced representational capacity, modality-invariant spatial complexity (4.1M parameters), and real-time latency. Our code will be made available at https://github.com/ GiannisZgs/OmniDecVAEs. Index Terms— Omni-modal, multi-modal, wearable computing, disentanglement, variational autoencoders (VAEs), variational decomposition autoencoders (DecVAEs), omnimodal DecVAEs (OmniDecVAEs), human activity recognition (HAR)

I. I NTRODUCTION

W

EARABLE sensing systems are increasingly generating large-scale, heterogeneous multi-modal data, requiring robust and expressive representation learning for applications in healthcare, human-computer interaction, and biometric security [1]. Recent advances in sensing hardware enable the integration of diverse physiological and behavioral modalities within unified wearable platforms, resulting in increasingly complex multi-modal data representations. This evolution motivates a transition from conventional multimodality to omni-modality, defined as the unification of arbitrarily many heterogeneous data streams into universal and expressive representations through a single model [2]. In supervised multi-modal approaches, large amounts of labeled data are required, often leveraging transformer [3] or state-space models [4] to capture long-range dependencies across time and modality dimensions. However, fusing heterogeneous streams such as inertial, physiological, and behavioral data [5] requires significant per-modality design considerations. Even for closely related modalities fusion often necessitates explicit allocation of latent capacity to model intra-modal and cross-modal interactions [6]. This challenge is amplified when combining disparate modalities, such as video and inertial data [7], or biosignals and audio [8]. While supervised approaches can achieve strong performance on specific tasks, their reliance on labeled data often limits generalization across unseen conditions, tasks, and modality configurations [9]. Self-supervised learning (SSL) has emerged as a powerful alternative, enabling the learning of informative and transferable representations with minimal or no annotations [10]. In unimodal wearable sensing, SSL

2

methods typically rely on contrastive objectives to distinguish between activities [11], or reconstruction-based objectives that improve robustness through input perturbations [12]. However, extending SSL to multi-modal wearable settings requires explicit modeling of cross-modal interactions, often through architectural modifications [13], graph-based formulations [14], or carefully designed multi-modal learning objectives [7]. Despite these advances, existing approaches primarily focus on task-specific performance and do not jointly address disentanglement and generative modeling. Autoencoder (AE)based methods [7], [15], [16] and generative adversarial networks (GANs) [17] promote disentangled representations, but typically rely on modality-specific encoders or decoders [16], [18]. Disentangled representations are particularly important in wearable sensing, as they enable applications such as biometric privacy and identity-aware modeling [16]. To the best of our knowledge, no existing approach jointly addresses scalable multi-modal fusion, disentanglement, and generation within a single unified framework. Motivated by these challenges, we introduce OmniDecVAE, a structure-aware learning framework for omni-modal wearable computing that learns full-stack representations. These representations are multi-modal, unsupervised, and characterized by strong generalization, task informativeness, disentanglement, and generative capabilities. OmniDecVAE is a selfsupervised, AE-based model that treats individual modality streams as components of a unified system through an adversarial contrastive objective inspired by signal decomposition dynamics [19]. The model processes omni-modal inputs—up to thirty modalities in our experiments—including inertial, physiological, and audio signals. All inputs are transformed into the time-frequency (TF) domain using the short-time Fourier transform (STFT) and Mel-scale representations for audio. A shared convolutional encoder learns modality-specific latent subspaces, while a shared fully-connected (FC) decoder establishes sample-level correspondence and reconstructs multimodal signals. The variational formulation enforces a Gaussian latent structure, enabling stochastic generation of unseen samples. Following pre-training, the learned representations are evaluated using conventional supervised classifiers. We assess OmniDecVAE on HARWE [5], a large-scale multi-modal dataset for human activity and identity recognition. Experimental results demonstrate that OmniDecVAE achieves strong performance across recognition, generation, and computational efficiency benchmarks, outperforming supervised transformerbased fusion methods [5], VAE-based models [20], [21], and classical approaches such as Independent Component Analysis (ICA) [22] and Principal Component Analysis (PCA) [23]. These capabilities potentiate OmniDecVAEs as a foundational paradigm for next generation models in intelligent wearable and clinical healthcare. First, modality-invariance promotes resilience in fast changing medical environments where equipment is upgraded constantly, requiring intelligent AI to adapt accordingly. In turn, disentangled representations provide an interpretable and transparent information carrier for medical applications, as isolation of identity information

IEEE TRANSACTIONS AND JOURNALS TEMPLATE

is critical for patient-centric healthcare. In addition, disentanglement allows for explicit control of the identity information, thus enabling biometric security applications and anonymization by operating directly on the unified disentangled representation. Finally, the generative capability can improve sustainability in healthcare and in-the-wild deployments by enabling sensor reduction, reconstruction of missing data, and synthesis of multi-modal signals. In this work, we make the following contributions: We propose a novel multi-modal SSL objective that enables scalable fusion of arbitrarily many modalities within a structured latent space, using a modality-agnostic architecture. • We introduce a unified architecture that scales to a large number of modalities through a shared encoder, without relying on transformer-based designs, and enables multimodal TF data generation through a shared decoder. • We learn a disentangled latent representation where modality, activity, and identity factors are well structured, leading to markedly improved performance in downstream recognition tasks. •

The remainder of this paper is organized as follows. Section II reviews related work in multi-modal fusion, SSL, and generative modeling. Section III presents the proposed OmniDecVAE framework. Section IV describes the experimental setup. Section V reports and discusses the results. Finally, Section VI concludes the paper. II. R ELATED W ORK A. Multi-modal Fusion in Wearable Computing Effective fusion methods must capture cross-modal interactions while preserving modality-specific characteristics in heterogeneous data streams. A common approach is concatenation-based fusion, which aggregates modalities either at the input (early fusion) or within latent representations (late fusion), typically relying on separate feature extractors per modality. For example, Kumar et al. [24] employ late fusion TF representations and convolutional branches. Transformer-based architectures provide a more flexible alternative by modeling intra- and inter-modal dependencies through attention mechanisms. However, these approaches often still depend on modality-specific feature extraction pipelines. For instance, Dissanayake et al. [25] combine Convolutional Neural Networks (CNN)-based feature extractors with late fusion, while Wu et al. [8] employ a transformer module for emotion recognition. To enhance cross-modal alignment, contrastive learning has been introduced in multimodal settings. Nguyen et al. [26] reduce the latent distance between modalities of the same sample to enforce coherence. Nevertheless, such approaches still rely on separate encoders per modality. Overall, existing fusion methods do not naturally scale to an arbitrary number of modalities without architectural modifications, limiting their applicability in omni-modal wearable settings.

ZIOGAS et al.: OMNI-MODAL DECOMPOSITION AUTOENCODERS LEARN FULL-STACK WEARABLE DISENTANGLED REPRESENTATIONS

B. Self-supervised Learning in Multi-modal Wearable Computing Wearable sensing scenarios are often constrained by limited annotations, motivating the use of SSL to learn representations from unlabeled data. SSL leverages surrogate objectives, such as reconstruction or contrastive learning, to extract meaningful structure from data [27].In unimodal wearable sensing, SSL approaches typically rely on contrastive objectives to discriminate between activities [11], or reconstruction-based objectives to improve robustness through perturbations [12]. In multi-modal settings, SSL has also been used to facilitate fusion. For example, Dissanayake et al. [25] and Wu et al. [8] employ augmentation prediction tasks as auxiliary objectives. More advanced approaches, such as TCLHAR [28], construct contrastive pairs across temporal neighborhoods and perform fusion using CNN-based architectures. Similarly, He et al. [7] align modalities through reconstruction objectives across separate branches. However, in most existing methods, SSL objectives primarily enhance intra-modal representations, while cross-modal fusion is handled through architectural design rather than the learning objective itself. C. Multi-modal Generation through Disentangled Representations in Wearable Computing Disentangled representation learning aims to separate underlying factors of variation in high-dimensional data [29], which is particularly relevant in wearable sensing for decoupling identity from activity [16]. AE-based approaches have been used to achieve disentanglement in multi-modal settings. Han et al. [30] proposed a multi-branch architecture for reconstruction, adversarial subject invariance, and classification. However, their evaluation does not assess generative quality. Other methods are architecture-based disentanglement, such as liquid neural networks for single-modal physiological signals [18], or separate decoder branches for identity disentanglement in electromyography-based systems [16]. GANs have also been employed for modality transfer between related signals, such as ECG and PCG [17], [31]. While these methods achieve high-quality signal synthesis, they often do not evaluate the utility of the learned latent representations for downstream tasks. III. P ROPOSED M ETHOD A. Problem Statement Let X = {x1 , x2 , ..., xN } ∈ RT ×M denote a multivariate time-series dataset with N i.i.d. observations, where T represents the temporal dimension and M denotes the number of modalities. Each observation xn ∈ RT ×M is assumed to be generated by a three-step generative process θ, involving a global latent variable z and a set of modality-specific latent variables {z1 , z2 , ..., zM } that act as components of z. In the multi-sensor wearable setting, different sensors capture the same underlying phenomenon from distinct perspectives (e.g., physiological, inertial, and acoustic), as illustrated in Fig. 1a. We interpret the latent variable z as a sensing event, while each zm represents a modality-specific view of

3

that event. We further assume that each latent component zm encodes unique information that is orthogonal to the other components. Under this assumption, the relationships between latent variables are defined as: X ⟨zi , zj ⟩ = zi zj∗ = 0, ∀i ̸= j (1)

z=

M X

zi

(2)

i=1

where ⟨·, ·⟩ denotes the dot product in latent space Z and (·)∗ denotes the complex conjugate. These equations describe an idealized setting in which modality-specific latent variables are mutually independent (zero correlation), and their aggregation fully reconstructs the underlying latent event z. This formulation naturally motivates an omni-modal setting, where increasing the number of sensing modalities can lead to progressively richer representations of the latent event. In the traditional variational decomposition autoencoding (VDA) framework [19], only a univariate signal X ∈ RT is observed. VDA employs a decomposition model to estimate intermediate components C embedded within X, and subsequently learns latent subspaces {z1 , z2 , ..., zC }. The final latent representation is then obtained by aggregating these subspaces. In contrast, in the multi-modal setting considered here, the dataset is already decomposed into its constituent modalities, i.e., X = {X1 , X2 , ..., XM }. Therefore, instead of performing decomposition, the objective becomes to learn a composition model that reconstructs how the latent event z is expressed in the input space. Method Overview. The proposed omni-modal framework outlined across the following sections can be summarized as follows: (1) Input: given aP multi-modal observation x = {x1 , . . . , xM }, M the anchor x0 = i=1 xi is constructed. (2) Encoding: multi-modal observations xi and the anchor x0 are fed to a shared encoder fϕ (·) to obtain latent representations hi and h0 . (3) Latent projection: intermediate representations hi and h0 are mapped to modality-specific latent variables zi ∼ qϕ (zi | x, z\i ), i ∈ {0, M }. (4) Self-supervised decomposition: the objective in Eq.10 enforces (i) alignment of hi with the anchor h0 via Llat recon (Eq.8) and (ii) orthogonality via Lortho (Eq.9), using omnimodal weighting (Eq.12). (5) Latent aggregation: modality-specific subspaces are combined into the full latent representation z = [z0 , z1 , z2 , ..., zM ]. (6) Decoding: latent variables zi , i ∈ {0, M } are sampled via reparameterization (Eq.16) and reconstruct inputs using the conditional decoder pθ (x | z, c). (7) Optimization: the model is trained using the (conditional) DELBO objective (Eqs.14,17). B. Multi-modal Variational Decomposition Autoencoding We extend variational decomposition autoencoding (VDA) to the multi-modal setting, referred to as multi-modal VDA

4

IEEE TRANSACTIONS AND JOURNALS TEMPLATE

Fig. 1: Overview of OmniDecVAEs and MMVDA, the proposed omni-modal full-stack wearable disentangled representation learning framework. (a) Three-step generative process θ: a latent event associated with activity and identity variables is generated, decomposed into M latent modality variables, and expressed through wearable sensor measurements across physiological, inertial, and acoustic domains. (b) MMVDA instantiated through OmniDecVAE aims to learn, via the inference model ϕ, a reverse mapping from inputs X to a modality-separated latent space Z, ultimately recovering the underlying activityand identity-conditioned latent event. The interplay between the encoder ϕ, the SSLDec structure-aware latent objective, and the decoder θ enables effective disentanglement, supports downstream classification tasks, and facilitates data synthesis.

(MMVDA), by instantiating a three-step generative process θ that bypasses the need for an explicit decomposition model. As illustrated in Fig. 1, for each observation xn , the generative process θ produces modality-specific components {xn1 , xn2 , ..., xnM } from latent variables {z1 , z2 , ..., zM }, which are sampled from prior distributions {p(z1 ), p(z2 ), ..., p(zM )}. The aggregated observation in the input space, denoted as x0 , is then generated through a conditional distribution: pθ (x0 | z1 , z2 , ..., zM ) ≡ pθ (x | z).

(3)

The parameters of the generative process θ and the latent variables {z, z1 , ..., zM } are unknown and must be inferred. The joint distribution of the generative process is defined as:

pθ (X, Z1 , ..., ZM ) =

N Y

qϕ (Z | X) =

N Y

n qϕ (z1n | xn , z2n , ..., zM ) ···

n=1 n n n · · · qϕ (z2n | xn , z1n , z3n , ..., zM ) qϕ (zM | xn , z1n , ..., zM −1 ) (6) Each conditional factor in Eq. (6) is modeled as:

n n pθ (xn | z1n , ..., zM ) pθ (z1n ) · · · pθ (zM )

n=1

(4a) n pθ (z n ) = pθ (z1n ) · · · pθ (zM )

(4b)

n

(4c)

n

where θ parameterizes the generative model and each prior pθ (zi ) is an isotropic multivariate Gaussian with zero mean [19]. The conditional distribution pθ (x | z1 , ..., zM ) assumes a diagonal covariance structure, reflecting the orthogonality of the latent subspaces zi . Since the true posterior pθ (z | x) is intractable [19], [20], we introduce a recognition model qϕ (z | x) to approximate it. The approximate posterior over the dataset is expressed as:

pθ (x | z ) = pθ (x

n

n | z1n , ..., zM ),

where Eq. (4b) expresses the prior over the latent event as a product of modality-specific priors. Each term in Eq. (4a) is modeled as: pθ (x | z1 , ..., zM ) = N (x | 0, diag(z1 , ..., zM )) pθ (zi ) = N (zi | 0, σz2i I),

∀i ∈ {1, ..., M }

(5)

qϕ (zi | x, z\i ) = N zi | fµzi (x, z\i ), fσz2 (x, z\i ) i

(7)

where z\i denotes all latent variables except zi , and each posterior distribution is modeled as an isotropic multivariate Gaussian [19]. As in VDA, the generative model defined in Eqs. (4)– (5) is not explicitly factorized to enforce the reconstruction constraints described in Eqs. (1) and (2). To avoid the need for separate inference models per modality, MMVDA employs shared generative and recognition networks (θ, ϕ), while enforcing the desired structure in the latent space through a selfsupervised decomposition loss.

ZIOGAS et al.: OMNI-MODAL DECOMPOSITION AUTOENCODERS LEARN FULL-STACK WEARABLE DISENTANGLED REPRESENTATIONS

C. Multi-modal Multiple Views Self-supervised Latent Decomposition Contrastive Learning

D. Increasing the Number of Contrastive Pairs

To enable omni-modal latent representation learning, we employ the self-supervised latent decomposition (SSLDec) loss introduced in DecVAEs [19]. SSLDec operationalizes the assumptions introduced in Section III-A, namely that modality-specific components form a composite system and exhibit orthogonality in the latent generative space (Eqs. 1, 2). Therefore, SSLDec is used to enforce these structural constraints in the input and latent spaces through a selfsupervised objective defined as:

Llat recon =

M X

i wpos DKL DJS (hi , h0 ) ∥ p ,

i=1

(8)

d×N

where p(y) = ϵ, ∀y ∈ {1, ..., d}, H ∈ R Lortho =

M X M X

(i,j) wneg DKL DJS (hi , hj ) ∥ n ,

i=1 j=i+1

(9)

where n(y) = 1, ∀y ∈ {1, ..., d} LSSLDec = Llat recon + Lortho

(10)

(i,j) wneg

i are weighting coefficients that conand where wpos trol the contribution of positive and negative interactions, respectively. DKL denotes the KLD and DJS denotes the Jensen–Shannon divergence. The target distributions p and n are uniform, with ϵ representing a small positive constant (i.e. 1e − 6). The SSLDec objective as defined by Eqs.(8,9,10) is a multi-view contrastive learning formulation with an adversarial structure that promotes disentanglement. The latent reconstruction term Llat recon minimizes the divergence between each modality-specific representation hi and an anchor representation h0 , encouraging alignment across modalities. In contrast, the orthogonality term Lortho maximizes the divergence between all pairs of modality representations, promoting separation and reducing redundancy. The use of KLD formulates this adversarial objective into stable cross-entropy terms and mitigates representation collapse [32]. Notably, SSLDec is computed in an intermediate latent space h, which is derived from the primary latent space z through a latent generative mapping rψ , where ψ ⊂ ϕ [19]. Although SSLDec inherently supports multi-modal interactions, its effectiveness depends on the quality of the anchor representation h0 used in the Llat recon term. In classical VDA, h0 is derived from a unimodal input, while hi correspond to decomposed components of that input. In the multi-modal setting considered here, each hi corresponds to a distinct modality, and thus an explicit multi-modal anchor must be constructed. We define the anchor in the input space as:

x0 =

M X

xi

5

(11)

i=1

and obtain the corresponding latent representation as h0 = f (x0 ) using the recognition model ϕ.

In real-world wearable sensing systems, each modality often consists of multiple channels. For example, inertial sensors are tri-axial, and different sensing modalities (e.g., accelerometer (ACC) and gyroscope (GYR), or physiological signals such as electrodermal activity (EDA) and blood volume pulse (BVP)) may exhibit varying degrees of correlation. Consequently, interactions between signals are not uniform: channels originating from the same sensor or related sensing modalities tend to share stronger relationships than those from unrelated sources. The standard SSLDec formulation assumes uniform interactions between latent components. However, this assumption is not well suited for omni-modal wearable settings, where relationships between channels and modalities are inherently heterogeneous. To address this, we extend SSLDec into an asymmetric contrastive learning formulation, where the strength of interactions between pairs is explicitly controlled. To systematically model these interactions, we define the (i,j) i and wneg using positive and negative weighting terms wpos an omni-modal vector VOM pos and matrix WOM neg . These weights are designed to reflect the relative importance of different interactions across channels and modalities. We introduce imbalance factors IFpos and IFneg to regulate the strength of weak and strong interactions between wearable channels and the anchor. The resulting weighting scheme is defined as follows: 1 wpos−base = PM i

(12a)

Ci

wpos−weak = (1 − IFpos ) ∗ wpos−base wpos−strong = (1 + IFpos ∗

Npos−weak ) ∗ wpos−base (12c) Npos−strong

1 wneg−base = PPM i

j

Ci

(12d) j

wneg−weak = (1 − IFneg ) ∗ wneg−base wneg−strong = (1 + IFneg ∗

(12b)

(12e)

Nneg−weak ) ∗ wneg−base (12f) Nneg−strong

where Ci denotes the number of channels in the i-th modality. The parameters IFpos and IFneg control the relative scaling of weak and strong interactions and are experimentally set to 0.25 and 0.4, respectively. The terms Npos−weak , Npos−strong denote the number of weak and strong positive interactions contributing to Llat recon , while Nneg−weak , Nneg−strong denote the corresponding quantities for negative interactions in Lortho . The weighting scheme is guided by domain-specific relationships between modalities. Since the anchor representation is constructed as the sum of all input signals (Eq. 11), modalities with higher signal complexity or dimensionality, such as audio, tend to exert a stronger influence on the anchor. Therefore, stronger weights are assigned to anchor–modality interactions involving such modalities.

6

IEEE TRANSACTIONS AND JOURNALS TEMPLATE

For negative interactions, smaller weights are assigned to pairs of channels that are expected to be similar, such as physiological signals (e.g., BVP, EDA, temperature), corresponding axes of inertial sensors (e.g., X-axis of ACC and GYR), and channels originating from the same sensor (e.g., X, Y, Z axes of ACC). In contrast, dissimilar modalities are assigned stronger repulsive weights to encourage separation in the latent space. The omni-modal vector VOM pos and upper-triangular matrix WOM neg are then given by:

the DELBO objective LDELBOEnc for an observation xn with latent representations hn , zn is defined as: LDELBOEnc (ϕ; xn , zn , VOM pos , WOM neg , β) X (i) = VOM pos Ez∼qϕ (zn |xn ,hn ) [log(rψ⊂ϕ (hin |zn ))] −

i X X i

−β

(i,j)

WOM neg DKL (hin , hjn )

(14)

j

X

DKL (qϕ (zn |xn , hin )||p(zn )) + const.

i PM

VOM pos = [wpos−weak , wpos−strong ]

i

Ci

= Llat recon − Lortho − βLprior + const.

(13a)

where rψ is the latent generative model defined in DecVAEs WOM neg =  wneg−weak wneg−weak wneg−strong · · · wneg−strong · · · wneg−strong  [19], and β controls the strength of the prior regularization 0 wneg−weak wneg−strong · · · wneg−strong · · · wneg−strong term Lprior , following the β-VAE formulation [21]. 0 0 wneg−strong · · · wneg−strong · · · wneg−strong  0 0 0 · · · wneg−weak · · · wneg−strong  When a decoder (generative model pθ ) is also included, the   . . . . . objective becomes: . . . . . . . . . ··· ··· . 0

0

0

0

· · · wneg−strong

0

(13b) AOM =  BVP−EDA  

0 0 0 . . . 0

LDELBOEnc−Dec (θ, ϕ; xn , VOM pos , WOM neg , β) ··· BVP−AccX · · · BVP−GyrX · · · EDA−Temp EDA−AccX · · · EDA−GyrX · · · 0 Temp−AccX · · · Temp−GyrX · · · 0 0 · · · AccX −GyrX · · · . ··· 0

. . 0

.

. . 0

. . . ···

BVP−AudioF n EDA−AudioF n Temp−AudioF n AccX −AudioF n . . ··· . 0 AudioF −AudioF n n−1

(13c)

P P [ MC × M i Ci ] where AOMi i is the omni-modal interaction matrix

that encodes pairwise relationships between channels and modalities, and can be mapped directly to the weights in WOM neg . This formulation also allows the decomposition of complex modalities such as audio into finer-grained components (e.g., frequency channels or oscillatory components (OCs) AudioFi , i ∈ {1, ..., N }), further increasing the number of contrastive pairs. Because the SSLDec objective is parameterized by VOM pos , WOM neg , and Eq. 10, the proposed framework can scale to an arbitrarily large number of channels and modalities. This is achieved by extending the weighting structures to incorporate additional anchor and cross-modal relationships, without requiring modifications to the underlying model architecture. E. Multi-modal Decomposition Evidence Lower Bound We instantiate MMVDA using a deep neural network architecture by parameterizing the generative and inference models pθ and qϕ with neural networks having parameters θ and ϕ, respectively. The posterior latent Gaussian distributions fµzi (x, zj̸=i ) and fσz2 (x, zj̸=i ), for i, j ∈ {1, ..., M }, are also i modeled as neural networks. As in VAEs [20] and DecVAEs [19], the structural assumptions introduced in previous sections must be incorporated into an objective function that provides a tractable evidence lower bound (ELBO) for the inference model qϕ with respect to the marginal likelihood of an observation xn . To this end, we adapt the decomposition ELBO (DELBO) formulation of DecVAEs [19], extending it to accommodate omni-modal weighting through VOM pos and WOM neg . When only the encoder qϕ is used (i.e., without an explicit decoder),

  

= Ez∼qϕ (zn |xn ) [log(pθ (xn |zn ))] X (i) − VOM pos DKL (hi , h0 ) −

i X X i

−β

(i,j)

WOM neg DKL (hi , hj )

(15)

j

X

DKL (qϕ (zn |xn )||p(zn )) + const.

i

= Ez∼qϕ (zn |xn ) [log(pθ (xn |zn ))] − Llat recon − Lortho − βLprior . In this formulation, the first term corresponds to the reconstruction likelihood of the generative model, replacing the latent generative component used in the encoder-only setting. The terms Llat recon and Lortho retain their role in enforcing alignment and orthogonality in the latent space, respectively. In our experiments, we evaluate both LDELBOEnc and LDELBOEnc−Dec . The objectives in Eqs. 14 and 15 encode the structural biases of MMVDA, namely the decomposition of representations into orthogonal modality-specific subspaces anchored to a shared multi-modal context, together with Gaussian regularization. F. OmniDecVAE: Omni-modal Variational Decomposition Autoencoder Fig. 2 illustrates the OmniDecVAE architecture, which instantiates PMthe MMVDA framework. TF domain input streams X f ×(1+ i Ci ) , corresponding to all modality channels and the anchor signal (Eq. 11), are processed by a shared singlebranch encoder fW (·). The encoder is a seven-layer onedimensional CNN based on Wav2Vec2 [33], equipped with LayerNorm normalization [34] and GELU activations [35]. It projects the inputs into an intermediate representation space V v×(M +1) . A shared FC projection head fH (·) further maps these representations into a latent interaction space H d×(M +1) , where the SSLDec objective (Eq. 10) is applied. Subsequently, modality-specific latent subspaces Zi are obtained

ZIOGAS et al.: OMNI-MODAL DECOMPOSITION AUTOENCODERS LEARN FULL-STACK WEARABLE DISENTANGLED REPRESENTATIONS

7

Fig. 2: Neural network architecture of the proposed OmniDecVAE scheme, that applies MMVDA.

via M + 1 FC projection layers that output the mean µ and variance logvar parameters of Gaussian distributions. The prior regularization term Lprior is independently computed for each modality subspace. An aggregation function g(·) then concatenates these subspaces into the final disentangled representation Z zdim ×(M +1) . A key property of OmniDecVAE is that multi-modality is handled entirely through the SSLDec objective (Eq. 10), allowing the network architecture to remain simple and modality-agnostic. No modality-specific encoder branches are required, and modality-specific structure is only enforced at the level of the latent projections. A hyperparameter lSSLDec% controls the percentage of latent frames used in the computation of Eqs. (14, 15) via random masking, acting as a form of regularization. In our experiments, the encoderonly variant of the model is denoted as OmniDecV AEEnc and is optimized using Eq. 14. G. Asymmetric Modality-Conditioned Shared Decoder To enable multi-modal data generation, the OmniDecVAE architecture incorporates a decoder network (Fig. 2). The generative model θ is implemented as a FC network, resulting in an asymmetric autoencoder design. Asymmetric architectures, such as masked autoencoders (MAEs) [27], have demonstrated strong performance by leveraging self-supervised objectives in the latent space, reconstructing inputs from partially observed latent representations. This design allows the encoder to remain expressive while keeping the decoder lightweight. In our case, the asymmetric decoder maps both masked and unmasked latent representations from Z → X. Notably, the anchor latent z0 is not provided as input to the decoder, ensuring that reconstruction relies solely on modality-specific latent components. The decoder is instantiated as a shared four-layer FC network with LayerNorm normalization [34] and GELU acti-

vations [35], with no non-linearity at the output layer. To enable sampling from the posterior distributions qϕ (zi | x), we employ the reparameterization trick [20]. Specifically, for each modality-channel latent subspace, we sample: z˜i = µzi + τ ∗ σz2i ∗ ϵ,

i ∈ {1, ..., M },

2

where zi ∼ N (µ, σ ), ϵ ∼ N (0, 1)

(16)

where τ is a scaling factor that controls the influence of the stochastic term ϵ on the sampled representation z˜i . Empirically, we observe that as the number of modalities increases, smaller values of τ in the range [0.1, 1.0] lead to more stable training. To support expressive multi-modal generation within a single shared decoder, we introduce a conditional embedding c that encodes the specific channel–modality combination. This embedding acts as a control signal, providing the decoder with information about which modality-channel pair to reconstruct. The decoder input is formed as: LayerN orm([z̃i , c]) where concatenation is followed by normalization. The dePM coder outputs reconstructed signals X̃ f ×( i Ci ) , where f denotes the dimensionality of the flattened TF-domain representation. Given the conditional nature of the decoder, the training objective becomes the conditional DELBO: LcDELBOEnc−Dec (θ, ϕ; xn , VOM pos , WOM neg , β, c) = Ez∼qϕ (zn |xn ) [log(pθ (xn |zn , c))]

(17)

− Llat recon − Lortho − βLprior In practice, for an input observation xn and its reconstruction yn , the decoder is optimized using a smoothed mean absolute error (sMAE) loss:

8

IEEE TRANSACTIONS AND JOURNALS TEMPLATE

TABLE I: OmniDecVAE Hyperparameters

B. Comparisons and Benchmarks

Model Part

Parameter

Value

Encoder

number of channels in the convolution layers kernel sizes for each convolution layer kernel strides for each convolution layer output of convolutional layers dimension weight initialization strategy

512 [8,3,3,2,2] [4,4,3,2,2] 512 He

Hidden and Latent Projections

dimension of the projecting Hi and latent subspaces Zi intermediate dimension in the hidden projection layers dimensionality per Zi latent subspace weight initialization strategy

192 192 32 Xavier

Decoder

dimensions of fully-connected hidden projection layers values for τ in Eq. 16 when number of channels C is [3,6,9,21,24,27,30] conditional embedding c dimension weight initialization strategy

[128,256,512,512] [1,0.8,0.5,0.1,0.1,0.25,0.15] 8 Xavier

Optimization and training

batch size for training percentage of frames used in the SSLDec loss calculation Eq.(10) peak learning rates for training with [Eq.(14),Eq.(17)] learning schedule for training with [Eq.(14),Eq.(17)] learning rate warmup steps for training with [Eq.(14),Eq.(17)] b1 parameter of the Adam optimizer b2 parameter of the Adam optimizer L2 regularization penalty on large weights epsilon parameter of the Adam optimizer clip gradients if exceeding this maximum value

64 50 [8 ∗ 10−5 , 1 ∗ 10−5 ] [constant with warmup, linear] [4320, 15120] 0.5 0.999 10−4 10−6 1

We compare the OmniDecVAE framework to a number of different approaches used on HARWE [5], standard benchmark approaches in the multi-modal fusion domain [36], [37], AE-based models [20], [21], [19] and eigenprojection models ICA [22], PCA [38], and kernel-PCA variants [23]. We also instantiate a multi-modal VAE model (MMVAE), by using the same architecture as in Fig.2, but with modality-dedicated branches instead of a shared branch. C. Implementation and Optimization

 Lrecon =

0.5(xn − yn )2 , if|xn − yn | < 1 |xn − yn | − 0.5, otherwise

(18)

IV. E XPERIMENTS A. Dataset To evaluate the OmniDecVAE framework, we utilize HARWE [5], a large-scale multi-modal HAR dataset. HARWE consists of video, audio, inertial and physiological recordings from thirty-five participants performing nine different daily activities in work environments: Walking around office, Talking to phone, Walking around office and talking to phone, Browsing monitor, Reading book, Writing on paper, Writing on paper and talking to phone, Typing on keyboard, Stretching on chair. In this work we do not use the video modality; instead we utilize the remaining time-series modalities and their channels as follows: • Audio: monaural audio sampled at 16kHz; we utilize the Filter Decomposition (FD) [19] to separate acoustic events into six frequency band-delimited OCs, thereby setting the number of audio channels to six. • Physiology: three physiological signals are captured through a smartwatch device, namely BVP (64Hz), EDA (4Hz), and Temperature (TEMP) (64Hz) modalities. • Inertial sensors: seven inertial measurements are sampled at 200Hz through a smartphone device, i.e., ACC, GYR, Gravity (GRA), Orientation (OR), Rotation (RO) Magnetometer (MG), Linear accelerometer (LACC). Each of these measurements is tri-axial, hence the number of channels for the inertial modalities is twenty-one. Each recording is accompanied by an activity annotation with a duration of approximately 10 min per activity. The HARWE dataset evaluation consists of two different partitioning schemes, Easy and Difficult [5]. The Easy scenario is a subject-dependent (SD) evaluation, where 80% of each subject’s recordings are kept as the training set, leaving the remaining 20% for testing. The Difficult scenario is a subjectindependent (SI) scenario where 70% of the subjects are kept as the training set, and the remaining 30% as the testing set. In the Easy scenario, 14400 and 3776 samples are contained in the training and testing sets, respectively, whereas 12160 and 5952 samples are contained in Difficult scenario training and testing sets, respectively.

We follow the pre-processing of all modalities in HARWE according to Esmailzehi et al. [5]; then we calculate the STFT TF-domain representation of all physiological and inertial modalities in 3s frames. We use a 512-point STFT in the frequency domain with a 0.75 s Hamming window, with an overlap of 75%, resulting in a spectrogram of size 64 × 3 that is flattened to a final size of 192 × 1 and given as input to the 1D CNN extractor. For the audio signals, a Mel spectrogram is calculated with 64 Mel scales and 3 hops. For calculating the anchor signal (Eq. 11), we first superpose all modalities in the time domain before taking a TF-domain transform; in the case of mixing acoustic with inertial or physiological modalities, we take the Mel transform for the anchor, resulting in a cumulative “sonified” Mel representation of acoustic, physiological and inertial signals. Finally, all signals are normalized using batch statistics. Detailed hyperparameters and their values for our OmniDecVAE architectures are given in Table I. We optimize our networks on a single NVIDIA RTX 6000 Ada Generation using PyTorch [39] and Huggingface [40] using the Adam optimizer [41] and train for approximately 120 epochs. Other VAE-based models are trained for 100 epochs as their optimization converges faster. Our learning scheme consists of a pre-training stage, where the model is trained to optimize one of Eqs. (14,15,17), and a post-training stage where a simple Support Vector Machine (SVM) classifier [42]) is utilized as a classification oracle on the learned representations. For combined CNN-Transformer-based fusion, we use the training schemes for supervised learning from Esmailzehi’s et al. [5]. D. Evaluation Metrics 1) Representation Informativeness: HARWE allows us to effectively perform the supervised classification tasks of HAR and identity recognition (IR). We evaluate these tasks using Accuracy, F1-score (F1) and Macro-averaged F1-score (MF1). For the SD scenario we evaluate for both HAR and IR, whereas in the SI scenario we evaluate only for HAR. 2) Generation Quality: In the first generative experiment we sample from a latent noise distribution using the learned conditional subspaces and generate a distribution of multimodal signals Ygen . We then sample a subset of the testing set Xreal and evaluate distributional similarity between these two subsets using the following metrics: • Multi-Kernel-Maximum Mean Discrepancy (MK-MMD): uses multiple radial basis function (RBF) kernels K

ZIOGAS et al.: OMNI-MODAL DECOMPOSITION AUTOENCODERS LEARN FULL-STACK WEARABLE DISENTANGLED REPRESENTATIONS

9

TABLE II: HARWE Omni-modal Disentangled Activity and Identity Recognition Performance - C = 30 Modality Channels Model

Acc. ↑

HAR F1 ↑

Subject-Dependent (Easy) MF1 ↑

Acc. ↑

IR F1 ↑

Subject-Independent (Difficult) HAR F1 ↑ MF1 ↑

MF1 ↑

Acc. ↑

Random Classifier Majority Classifier

11.18 ± 0.15 11.47 ± 0.00

11.18 ± 0.15 2.36 ± 0.00

11.17 ± 0.16 2.28 ± 0.00

2.91 ± 0.10 3.36 ± 0.00

2.91 ± 0.10 0.21 ± 0.00

2.89 ± 0.09 0.19 ± 0.00

11.07 ± 0.17 11.23 ± 0.00

11.08 ± 0.16 2.27 ± 0.00

11.06 ± 0.17 2.24 ± 0.00

Logistic Regression Random Forest SVM

38.07 ± 0.13 44.55 ± 0.16 42.64 ± 0.08

36.36 ± 0.20 43.39 ± 0.17 41.16 ± 0.09

36.22 ± 0.19 43.30 ± 0.17 41.05 ± 0.09

18.50 ± 0.35 25.18 ± 0.14 17.34 ± 0.10

17.72 ± 0.36 24.59 ± 0.13 16.12 ± 0.17

17.80 ± 0.36 24.67 ± 0.13 16.18 ± 0.09

38.33 ± 0.11 43.82 ± 0.13 43.09 ± 0.06

35.84 ± 0.17 41.69 ± 0.14 40.73 ± 0.06

35.85 ± 0.16 41.76 ± 0.13 40.77 ± 0.06

ICA [22] PCA [38] rbf-PCA [23] sigmoid-PCA [23] poly-PCA [23]

77.00 ± 0.07 77.00 ± 0.07 11.72 ± 0.001 17.76 ± 0.08 67.71 ± 0.07

76.92 ± 0.08 76.93 ± 0.08 2.46 ± 0.001 14.51 ± 0.08 67.34 ± 0.07

77.22 ± 0.08 77.23 ± 0.08 2.33 ± 0.001 14.47 ± 0.08 67.63 ± 0.07

75.30 ± 0.10 75.29 ± 0.10 3.36 ± 0.001 6.90 ± 0.07 51.42 ± 0.11

75.44 ± 0.11 75.42 ± 0.10 0.21 ± 0.001 5.25 ± 0.11 51.42 ± 0.10

75.68 ± 0.10 75.67 ± 0.10 0.19 ± 0.001 5.29 ± 0.10 51.54 ± 0.11

62.06 ± 0.13 62.06 ± 0.13 11.63 ± 0.001 15.32 ± 0.06 58.90 ± 0.09

62.04 ± 0.15 62.04 ± 0.13 2.42 ± 0.001 12.48 ± 0.08 58.33 ± 0.10

62.73 ± 0.13 62.73 ± 0.12 2.31 ± 0.001 12.63 ± 0.08 59.04 ± 0.10

YAMNet [36] & EEGNet [37] Late Fusion Single-Head YAMNet [36] & EEGNet [37] Late Fusion Dual-Head YAMNet [36] & EEGNet [37] & Transformer Late Fusion Single-Head [5] YAMNet [36] & EEGNet [37] & Transformer Late Fusion Dual-Head [5]

38.15 ± 5.40 49.88 ± 4.62 39.93 ± 6.03 53.92 ± 2.98

30.26 ± 6.42 46.26 ± 5.26 35.70 ± 6.64 51.75 ± 3.23

30.63 ± 6.37 46.71 ± 5.20 36.05 ± 6.70 52.17 ± 3.20

50.22 ± 4.41 50.08 ± 4.32 67.36 ± 2.92 61.57 ± 3.55

47.51 ± 4.59 47.06 ± 4.75 66.85 ± 3.06 60.13 ± 3.89

47.31 ± 4.58 46.99 ± 4.79 66.60 ± 3.12 59.89 ± 3.87

31.85 ± 3.77 NA 32.98 ± 2.32 NA

24.19 ± 4.50 NA 28.30 ± 2.72 NA

24.48 ± 4.49 NA 28.77 ± 2.69 NA

MMVAE β-MMVAE (β = 0.1)

83.55 ± 0.10 82.36 ± 0.13

83.46 ± 0.11 82.31 ± 0.13

83.67 ± 0.10 82.50 ± 0.13

83.22 ± 0.15 79.42 ± 0.34

83.26 ± 0.14 79.65 ± 0.35

83.32 ± 0.14 79.77 ± 0.24

63.06 ± 0.14 70.91 ± 0.10

62.74 ± 0.12 70.37 ± 0.10

63.46 ± 0.12 70.92 ± 0.09

OmniDecVAEEnc [19] β-OmniDecVAEEnc [19] (β = 80) OmniDecVAEEnc−Dec β-OmniDecVAEEnc−Dec (β = 0.1)

71.46 ± 1.13 71.75 ± 0.93 82.18 ± 0.16 84.56 ± 0.21

71.26 ± 1.13 71.55 ± 0.94 82.07 ± 0.15 84.48 ± 0.21

71.53 ± 1.14 71.85 ± 0.94 82.31 ± 0.16 84.70 ± 0.20

62.12 ± 2.69 63.08 ± 1.26 86.12 ± 0.11 88.97 ± 0.13

62.22 ± 2.73 63.36 ± 1.23 86.12 ± 0.12 88.99 ± 0.13

62.43 ± 2.74 63.52 ± 1.24 86.18 ± 0.12 89.00 ± 0.13

61.85 ± 0.41 59.31 ± 1.01 60.82 ± 0.19 64.56 ± 0.12

61.48 ± 0.34 58.82 ± 1.15 59.97 ± 0.20 64.09 ± 0.14

62.10 ± 0.34 59.47 ± 0.89 60.71 ± 0.20 64.78 ± 0.13

(a) Acc.: Accuracy. MF1: Macro F1. Best score for each metric of each task is given in bold. 95% confidence intervals are reported over n=25 random seeds.

over multiple bandwidths σ, and averages individual MMDs between real and generated subsets, over 2 those kernels: M K − M M DRBF (X ,Y ) = P real gen PK 1 P 1 1 ′) k(x , x + k(y [ ′ ′ 2 2 i i j , yj ′ ) − i,i j,j k K P n m 2 k(x , y )] i j i,j nm • Diversity Score: this metric measures pairwise Euclidean distances dreal , dgen within the sets Xreal , Ygen , and calculates DivScore = |dreal − dgen |. In the second experiment, we evaluate the actual reconstruction ability of the decoder w.r.t. to input signals. Specifically, we compare inputs Xn with decoder reconstructions Yn using the following metrics: PT 1 • Mean Squared Error (MSE): M SEn = T t=1 (Xn [t] − Yn [t])2 , over the T samples of the TF reconstructions Xn , Yn . PT 1 • Mean Absolute Error: M AEn = T t=1 |Xn [t] − Yn [t]| The reconstructed signals Yn are further fed into the OmniDecVAE encoder to obtain ZYn and compared to the original latents Zn ; the goal of the below metrics is to measure latent consistency: 2 • Latent L2 distance: dL2 = |Zn − ZYn | Z Z

Latent cosine distance: dcos = ||Zn n||||ZYnY || n 3) Complexity: Furthermore, to assess the suitability of our model for real-time processing and edge deployment potential, we measure floating-point operations (FLOPs), number of parameters (Param.), size on disk (SoD), and frames processed per second during inference or latency. •

V. R ESULTS AND D ISCUSSION A. Disentangled Representation Informativeness Table IIa presents the performance on HAR and IR informativeness from the learned representations when using all available modalities on HARWE. In the SD scenario, the βOmniDecVAEEnc−Dec variant provides the best performance in terms of HAR and IR, closely followed by VAE-based methods [20], [21]. Eigenanalysis methods such as ICA [22] and PCA [23] provide relatively good performance as well, whereas supervised methods such as YAMNet and EEGNet feature extractors with Transformer fusion [5], [36], [37] have subpar performance. Despite the end-to-end training of

the supervised methods to perform one of the two tasks (HAR and IR) their performance remains low, with the dualhead multi-task learning scheme performing evidently better. Notably, supervised methods also manifest with a much higher sensitivity to the random seed as the 95% confidence intervals suggest. In the SI scenario, the β-MMVAE variant outperforms other alternatives, signifying that the multi-branch architecture is slightly more adaptable to unseen subjects compared to the proposed OmniDecVae. Fig. 3 illustrates the disentanglement properties of Transformer-based [5], VAE-based [20] and OmniDecVAE models. It can be seen that the CNN-Transformer model achieves partial linear separation of activities when trained end-to-end for that task (Fig. 3a), whereas β-MMVAE and βOmniDecVAE ((Fig. 3b,c) learn unsupervised representations that are not linearly separable. However, β-MMVAE and β-OmniDecVAE achieve disentanglement in the sense of a structured modality space, as evident by Figs.3d,e. Notably, βOmniDecVAE achieves very clear separation between different modality modes (Fig.3e) in contrast to β-MMVAE (Fig. 3d), attributed to the LDELBOEnc−Dec objective (Eq. 15). Subsequently, this disentangled modality structured latent space, renders the representation more informative to the downstream tasks of HAR and IR; effectively, the clear representation of each modality’s information leads to a clearer multi-modal fusion, expressed through the relative location of the modality clusters. Each modality’s location is dictated by the SSLDec structural dynamics, by its relation to the anchor (latent reconstruction) and the other modalities (orthogonality). We see that the asymmetric weights assigned through VOMpos , WOMneg , result in a representation where the anchor (‘center of mass’ of the representation) is shifted away from some modalities and closer to others. B. Generative Performance Importantly, imposing the above discussed disentanglement dynamics allows us to generate omni-modal synthetic samples of higher quality and realism compared to VAE-based models. Table IIIa presents the generative performance of models trained in the SI scheme of HARWE with all available modalities. Notably, even though MMVAEs are trained with

10

IEEE TRANSACTIONS AND JOURNALS TEMPLATE

Fig. 3: TSNE [43] visualizations of the learned latent representations. (a) Activity-colored latent from the CNN-Transformer [5], taken at the output of the Transformer layers. (b) Activity-colored latent from β-MMVAE. (c) Activity-colored latent from β-OmniDecVAEEnc−Dec . (d) Modality-colored latent from β-MMVAE. (e) Modality-colored latent from βOmniDecVAEEnc−Dec . TABLE III: Omni-modal Subject-Independent Generative Performance - C = 30 Modality Channels Model MMVAE β-MMVAE OmniDecVAE β-OmniDecVAE

Reconstruction Quality

Latent Noise Generation

MSE ↓

MAE ↓

dL2 ↓

dcos ↓

MK-MMD ↓

DivScore ↓

4.94 ± 0.041 4.358 ± 0.33 0.225 ± 0.002 0.116 ± 0.001

1.048 ± 0.007 1.019 ± 0.006 0.328 ± 0.001 0.236 ± 0.001

0.575 ± 0.002 1.369 ± 0.003 0.327 ± 0.001 0.446 ± 0.001

0.150 ± 0.002 0.182 ± 0.001 0.427 ± 0.002 0.242 ± 0.001

0.226 ± 0.044 0.210 ± 0.040 0.186 ± 0.018 0.181 ± 0.019

18.555 ± 4.409 15.468 ± 3.450 5.802 ± 0.528 4.094 ± 0.688

(a) Best metric score is given in bold. 95% confidence intervals are reported over n=4627 omni-modal samples for a single random seed of the model.

multiple branches allocated to each modality, their generation performance is subpar to that of OmniDecVAEs. Specifically, OmniDecVAEs are characterized by enhanced reconstruction quality as evident by the MSE and MAE metrics. Moreover, OmniDecVAE generated samples also provide more realistic latent embeddings; metrics dL2 and dcos showcase that OmniDecVAE-synthesized samples are more suitable to be used as synthetic data to train a neural network due to their higher similarity to real data embeddings, expressed through lower dL2 and dcos distances. Finally, when sampling latent noise from the learned representations of the models, OmniDecVAE models manifest with a higher distributional similarity to real data. Namely, a lower MK-MMD and DivScore suggest that the distribution learned by OmniDecVAE is more “realistic”, closer to real data. The above results are visually consolidated in Fig. 4. Notably, MMVAE models are not able to always capture the amplitude of the TF representations (Temperature, ACC, Audio), or their exact morphology, simultaneously when generating multiple modalities. On the contrary, OmniDecVAEs provide much more accurate reconstructions w.r.t. the amplitude and position of TF events, whereas they capture better the mor-

phology of narrow-band frequency phenomena such as the decomposed audio events e.g. Audio OC1, OC4. This illustrates the robustness of OmniDecVAEs in generating simultaneously a diverse set of wearable modalities compared to MMVAEs. C. Ablation Studies 1) Contribution of Multi-modal Input: Tables IVa,Va presents downstream HAR and IR performance for different combinations of modalities. CNN-Transformer presents a stronger performance when only audio is used as the input, as it uses the YAMNet [36] as the feature extractor, a backbone designed for audio. For other modality combinations, β-OmniDecVAE provides stronger performance in some cases, whereas βMMVAE also dominates some modality combinations, with comparable performance between these two methods. Notably, the full-modal scenario with C = 30 channels does not provide the best performance in the SD scenario, but rather the combinations with less modalities. Specifically, combinations that contain the inertial sensors demonstrate superior performance, hinting of their elevated importance for the task of HAR with prior subject knowledge. In the SI scenario though, the fullmodal scenario provides the best results overall.

ZIOGAS et al.: OMNI-MODAL DECOMPOSITION AUTOENCODERS LEARN FULL-STACK WEARABLE DISENTANGLED REPRESENTATIONS

11

Fig. 4: Flattened 3s TF representations reconstruction results from generative models trained on the SI HARWE scheme with C = 30 modality channels. TABLE IV: Ablation Study on Number of Modalities - Activity Recognition Performance SD HAR ↑

Modalities

CNN-Transformer [5] ↑

Smartwatch (BVP+EDA+TEMP, C = 3) Audio (6 OCs, C = 6) SmartWatch + Audio (C = 9) Inertial (C = 21) Inertial + Smartwatch (C = 24) Inertial + Audio (C = 27) Smartwatch + Inertial + Audio (C = 30)

SI HAR ↑

β-MMVAE ↑

β-OmniDecVAEEnc−Dec ↑

CNN-Transformer [5] ↑

β-MMVAE ↑

β-OmniDecVAEEnc−Dec ↑

Acc. ↑

F1 ↑

Acc. ↑

F1 ↑

Acc. ↑

F1 ↑

Acc. ↑

F1 ↑

Acc. ↑

F1 ↑

Acc. ↑

F1 ↑

17.87 ± 1.20 48.30 ± 9.26 43.55 ± 4.92 58.35 ± 4.26 49.85 ± 3.56 40.63 ± 4.67 39.93 ± 6.03

16.23 ± 1.23 45.87 ± 10.89 40.37 ± 5.43 55.92 ± 4.46 46.63 ± 4.20 36.05 ± 4.07 35.70 ± 6.65

22.50 ± 0.15 38.80 ± 0.19 47.33 ± 0.08 86.80 ± 0.15 86.00 ± 0.15 87.21 ± 0.07 82.36 ± 0.13

21.99 ± 0.16 37.93 ± 0.17 46.58 ± 0.11 86.80 ± 0.15 86.23 ± 0.15 87.18 ± 0.08 82.31 ± 0.13

21.68 ± 0.20 40.85 ± 0.28 49.62 ± 0.09 86.24 ± 0.11 86.20 ± 0.17 81.13 ± 0.14 84.56 ± 0.21

21.51 ± 0.18 39.75 ± 0.30 48.91 ± 0.11 86.23 ± 0.12 86.19 ± 0.16 81.04 ± 0.14 84.48 ± 0.21

11.24± 1.03 41.61 ± 8.31 38.75 ± 5.42 32.80 ± 2.77 32.46± 2.20 36.02 ± 2.95 32.98 ± 2.32

8.69 ± 1.07 38.48 ± 9.94 36.78 ± 5.54 26.75 ± 3.25 27.09 ± 2.70 31.12 ± 3.65 28.30 ± 2.72

13.70 ± 0.19 35.62 ± 0.20 37.81 ± 0.31 57.91 ± 0.06 62.30 ± 0.12 69.42 ± 0.16 70.91 ± 0.10

13.58 ± 0.35 33.80 ± 0.20 36.46 ± 0.32 56.90 ± 0.09 62.11 ± 0.10 68.59 ± 0.17 70.37 ± 0.10

12.78 ± 0.17 38.07 ± 0.16 38.33 ± 0.22 57.16 ± 0.11 55.13 ± 0.21 62.63 ± 0.12 64.56 ± 0.12

12.63 ± 0.15 36.46 ± 0.15 37.02 ± 0.23 56.93 ± 0.09 54.81 ± 0.20 62.00 ± 0.10 64.09 ± 0.14

(a) Acc.: Accuracy. Best Acc. and F1 score for each row of each task is given in bold. 95% confidence intervals are reported over n=25 random seeds. TABLE V: Ablation Study on Number of Modalities - Identity Recognition Performance Modalities

Smartwatch (BVP+EDA+TEMP, C = 3) Audio (6 OCs, C = 6) SmartWatch + Audio (C = 9) Inertial (C = 21) Inertial + Smartwatch (C = 24) Inertial + Audio (C = 27) Smartwatch + Inertial + Audio (C = 30)

SD IR ↑ CNN-Transformer [5] ↑

β-MMVAE ↑

β-OmniDecVAEEnc−Dec ↑

Acc. ↑

F1 ↑

Acc. ↑

F1 ↑

Acc. ↑

F1 ↑

38.57 ± 4.74 3.35 ± 0.002 33.21 ± 4.35 74.04 ± 1.23 62.77 ± 4.05 36.29 ± 4.17 67.36 ± 2.92

34.88 ± 4.92 0.22 ± 0.001 29.26 ± 4.66 74.59 ± 1.21 61.78 ± 4.38 35.41 ± 4.21 66.85 ± 3.05

50.50 ± 0.14 17.56 ± 0.29 47.04 ± 0.29 88.24 ± 0.25 92.77 ± 0.11 85.66 ± 0.12 79.42 ± 0.34

48.45 ± 0.13 16.29 ± 0.29 46.68 ± 0.28 88.21 ± 0.26 92.76 ± 0.10 85.68 ± 0.12 79.65 ± 0.35

45.50 ± 0.13 16.75 ± 0.28 48.27 ± 0.15 88.01 ± 0.12 92.66 ± 0.06 75.63 ± 0.16 88.97 ± 0.13

44.02 ± 0.09 15.24 ± 0.28 47.59 ± 0.17 87.96 ± 0.12 92.63 ± 0.06 75.63 ± 0.16 88.99 ± 0.13

(a) Acc.: Accuracy. Best Acc. and F1 score for each row of each task is given in bold. 95% confidence intervals are reported over n=25 random seeds.

Fig. 5 illustrates the generative performance metrics of MAE and MK-MMD. β-OmniDecVAE demonstrates a more robust behavior, less influenced by the number of input modality channels, compared to β-MMVAE. 2) Contribution of Optimization Terms: Table VI contains a detailed ablation of the four terms contained in Eq. ( 17); in terms of task-related performance, we see that utilizing the SSLDec terms alongside a supervised loss, gives the highest performance while maintaining disentanglement between activity and identity. It becomes evident that SSLDec and the decoder are the main drivers of disentanglement, as their absence results in higher imbalance in the performance of HAR or IR. 3) Contribution of Gaussian Prior Approximation Weight β:

The role of Lprior and the value of β is better understood through Fig.6. As discussed in DecVAEs [19], a strong β

TABLE VI: Ablation Study on OmniDecVAE Loss Components Loss Components SSLDec Lrecon Lprior Supervised SSLDec + Lprior SSLDec + Lrecon SSLDec + Supervised Lrecon + Lprior Lrecon + Supervised Lprior + Supervised SSLDec + Lprior + Lrecon SSLDec + Lprior + Supervised SSLDec + Lrecon + Supervised Lrecon + Lprior + Supervised SSLDec + Lprior + Lrecon + Supervised

SD HAR ↑

SD IR ↑

Acc. ↑

F1 ↑

Acc ↑

F1 ↑

76.03 83.22 65.30 82.07 74.66 82.09 85.83 81.31 83.06 70.64 82.17 72.48 83.80 82.11 81.89

75.91 83.14 64.97 82.00 74.56 82.01 85.75 81.23 82.96 70.44 82.04 72.43 83.71 82.02 81.78

72.37 86.92 47.69 83.21 67.26 86.58 88.32 85.70 86.46 56.46 85.89 62.28 87.71 85.35 86.39

72.42 86.93 48.09 83.23 67.31 86.63 88.33 85.69 86.49 56.93 85.89 62.45 87.73 85.35 86.38

12

IEEE TRANSACTIONS AND JOURNALS TEMPLATE

TABLE VII: Performance on Complexity Metrics for C = 30 Modality Channels Model

GFLOPs ↓

LatencyInf (ms) ↓

SoD (MB) ↓

Parameters (M) ↓

CNN-Transformer [5] MMVAE OmniDecVAEEnc [19] OmniDecVAEEnc−Dec

0.65 0.45 29.61 85.85

0.06 0.67 3.51 4.00

20.19 336.46 13.20 15.82

5.28 88.09 3.43 4.11

(a) LatencyInf : Inference latency, SoD: size on disk

Fig. 5: The effect of a varying number of input channels for β-MMVAEs (red) and β-OmniDecVAE (blue) models, on MAE (upper) and MK-MMD (lower) generative performance metrics.

compression strength may not always result in a better representation; in Fig. 6a, HAR performance correlates directly with modality disentanglement at different values of β. Here higher β values result in a collapse of the disentangled structure. We also see that the absence of the Lprior through β = 0 is not detrimental to the downstream performance, mainly attributed to the disentanglement mechanisms of SSLDec. A different behavior is evident in the decoder-less variant of OmniDecVAE (see Fig. 6b); in the absence of a decoder, a higher β is needed for a disentangled representation. D. Complexity In Table VIIa, we evaluate our proposed OmniDecVAE alongside a Transformer-based model and MMVAE on their complexity, by calculating per sample FLOPs and latency inside a batched input. The CNN-Transformer of [5] provides the faster choice with very low latency per sample. MMVAE performs the least number of FLOPs, whereas OmniDecVAE provides the lighter alternative in terms of SoD and number of parameters. Notably, OmniDecVAE adds a significant number of operations per sample due to the sophisticated optimization objective; at the same time, MMVAE presents increasing storage requirements due to the high number of modalities it accommodates as separate branches. Indeed, Fig. 7b showcases that the spatial complexity of MMVAEs increase with the number of modalities; on the other hand OmniDecVAEs are invariant to the number of modalities for that matter, while being more prone to fast inference as the number increases. E. Discussion Wearable representation learning has significantly advanced in the latest years mainly due to breakthroughs in fusion architectures that has enabled integration of multiple modality streams across large-scale datasets, spearheaded by the

prowess of transformer-based models. However, despite high HAR classification performance, the focus of these models does not extend to full-stack representations. Here, we presented omni-modal representation learning for wearable computing, by efficiently learning through a modality-invariant architecture; modality-specific data handling for OmniDecVAEs ends at the pre-processing stage with the selection of input representation. In contrast to transformer-based alternatives, we show that expressive fusion can be achieved through a learning objective without requiring architectural interventions. Significantly, we notice that even though our OmniDecVAE approach does not utilize the video modality, one of the most expressive modalities in HAR, it performs on par with a supervised transformer-based method in the HARWE Difficult SI scenario that utilizes video (Acc. of 68.5% in SI HAR) [5]. In addition, our proposed full-stack structured representation learning framework promotes generalizable and widespread utility through an AE-based SSL task that guarantees disentangled representations. The importance of disentanglement can be understood when evaluating task informativeness; notably, our omni-modal AE networks performed better under SSL pre-training on both HAR and IR, compared to supervised transformer-based alternatives, that are trained separately to learn activity-specific and identity-specific representations, and VAE-based methods and other benchmarks (Table IIa). The proposed OmniDecVAEs consistently showed a greatly improved performance over supervised alternatives in the SI disentanglement tasks for HAR and IR. OmniDecVAEs also performed well in the SD scenario, which resembles a biometric identification scenario after a prior registration of biometrics has been performed for each subject. Moreover, the structure-informed learning objective that we adopted, is fully explainable and transparent. Visual inspection of the representations in Fig. 3 reveals well delimited modes that arise as a result of subspace learning. This is also translated to better multi-modal generation quality in Table IIIa and Fig. 4. Multi-modal synthetic data generation is also of significant value for the expansion of modern AI systems, due to the limitations inherent in real data collection. Our results on unseen subjects in the SI scenario show an enhanced ability of OmniDecVAEs on generating realistic omni-modal data distributions, compared to VAE-based alternatives. Finally, our approach has a complexity-friendly design that is exemplified when the number of modalities aggressively scales; architectures with separate branches per modality quickly become unsustainable in real-life edge deployment scenarios due to the increasing number of parameters and storage required. In terms of real-time processing, OmniDec-

ZIOGAS et al.: OMNI-MODAL DECOMPOSITION AUTOENCODERS LEARN FULL-STACK WEARABLE DISENTANGLED REPRESENTATIONS

13

Fig. 6: The effect of varying Gaussian prior approximation strength β on representation disentanglement for (a) HAR and (b) IR for β-OmniDecVAEEnc (orange) and β-OmniDecVAEEnc−Dec (orange).

Fig. 7: The effect of a varying number of input channels on (a) inference latency and (b) number of parameters.

VAEs require much more FLOPs than compared methods, yet they still lie within the real-time processing window for edge applications, with a latency of 4ms per sample. Combined with a negligent storage requirement of as low as 15M B, enhanced generation performance, and high downstream task performance, OmniDecVAEs are very attractive for cloudbased edge processing systems, clinical environments, wearable healthcare and in-the-wild monitoring. Despite the promising performance of our proposed framework, aspects for improvement in future work remain. Our current implementation of OmniDecVAE is based on TF representations; although TF representations through STFT and Mel scale are more informative and predominantly used in modern deep learning, the generation of synthetic data in the more raw format of time domain is preferable, allowing more freedom in downstream processing. Moreover, we did not experiment with cross-modal inference and missing modality scenarios; our results have showcased excellent omni-modal separation and conservation of hierarchical relations within the latent representations of OmniDecVAEs, hinting a promising direction of our work towards cross-modal generation. Other future extensions of our work will accommodate other types of modalities beyond time series, such as video or text.

VI. C ONCLUSION This work proposes OmniDecVAEs, a novel structured representation learning framework to learn full-stack wearable representations by fusing arbitrarily large multi-modal (or omni-modal) data streams. Our approach is powered by an SSL decomposition loss that facilitates disentanglement forces in a latent space, alongside VAE-based prior approximation and decoding. The proposed OmniDecVAEs operate in the TFdomain through a single-branched modality-invariant encoderdecoder scheme, where fusion, cross-modal alignment and intra-modal integrity are ensured through the learning objective. Our evaluations highlight full-stack representational capacity, with improvements across an array of tasks and over different model families, such as Transformers and VAEs. Namely, OmniDecVAEs simultaneously enhance HAR and IR accuracy by 6.75%, and 1.01%, multi-modal data synthesis MAE and MK-MMD by 13.85% and 76.84%, respectively, along with modality-invariant storage requirements of 15M B or 4.1M parameters. These results underscore OmniDecVAEs as a foundational paradigm for next-generation full-stack models in wearable AI. Their robust modality-invariant footprint, transparent and interpretable representations, aid towards secure, patient-centric healthcare, enabling biometric security applications, whereas their generative capability allows for

14

IEEE TRANSACTIONS AND JOURNALS TEMPLATE

sustainable deployment through synthetic signals, and invariance to sensor reduction and missing data. R EFERENCES [1] S. Kang, W. Choi, C. Y. Park, N. Cha, A. Kim, A. H. Khandoker, L. J. Hadjileontiadis, H. Kim, Y. Jeong, and U. Lee, “K-EmoPhone: A Mobile and Wearable Dataset with In-Situ Emotion, Stress, and Attention Labels,” Scientific Data 2023 10:1, vol. 10, pp. 1–21, 6 2023. [2] Y. Zhang, H. Li, J. Liu, and X. Yue, “Scaling Omni-modal Pretraining with Multimodal Context: Advancing Universal Representation Learning Across Modalities,” 2025. [3] B. Minor, C. Greeley, R. Holder, B. Thomas, L. B. Holder, and D. J. Cook, “A Feature-Augmented Transformer Model to Recognize Functional Activities From in-the-Wild Smartwatch Data,” IEEE Journal of Biomedical and Health Informatics, vol. 30, pp. 256–265, 1 2026. [4] K. Avramidis, D. Kunc, B. Perz, K. Adsul, T. Feng, P. Kazienko, S. Saganowski, and S. Narayanan, “Scaling Representation Learning from Ubiquitous ECG with State-Space Models,” IEEE Journal of Biomedical and Health Informatics, vol. 28, no. 10, pp. 5877–5889, 2024. [5] A. Esmaeilzehi, E. Khazaei, K. Wang, N. Kaur Kalsi, P. C. Ng, H. Liu, Y. Yu, D. Hatzinakos, and K. Plataniotis, “HARWE: A multi-modal large-scale dataset for context-aware human activity recognition in smart working environments,” Pattern Recognition Letters, vol. 184, pp. 126– 132, 8 2024. [6] T. Fan, F. Tian, S. Wang, H. Zhang, G. Luo, L. Zhu, J. Liu, L. Jiang, R. Cai, Q. Dong, and B. Hu, “CMDˆ{3}: Cross-Modal Decoupled Deformable Distillation for EEG-fNIRS Fusion,” IEEE Transactions on Affective Computing, 2026. [7] C. He, X. Xu, Z. Wang, J. Song, F. Shen, and H. T. Shen, “CausalInspired Fourier Representation Learning for Wearable IMUs and Egocentric Action Recognition,” IEEE Transactions on Circuits and Systems for Video Technology, pp. 1–1, 1 2026. [8] Y. Wu, M. Daoudi, and A. Amad, “Transformer-Based Self-Supervised Multimodal Representation Learning for Wearable Emotion Recognition,” IEEE Transactions on Affective Computing, vol. 15, pp. 157–172, 1 2024. [9] A. van den Oord DeepMind, Y. Li DeepMind, and O. Vinyals DeepMind, “Representation Learning with Contrastive Predictive Coding,” 7 2018. [10] R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Y. Tian, A. Schwarzschild, A. G. Wilson, J. Geiping, Q. Garrido, P. Fernandez, A. Bar, H. Pirsiavash, Y. LeCun, and M. Goldblum, “A Cookbook of Self-Supervised Learning,” arXiv preprint arXiv:2304.12210 (2023), 4 2023. [11] Z. Xiao, H. Tong, R. Qu, H. Xing, S. Luo, Z. Zhu, F. Song, and L. Feng, “CapMatch: Semi-Supervised Contrastive Transformer Capsule with Feature-Based Knowledge Distillation for Human Activity Recognition,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 2, pp. 2690–2704, 2025. [12] X. Liu, Y. Zhang, Z. Yu, H. Lu, H. Yue, and J. Yang, “RPPG-MAE: SelfSupervised Pretraining with Masked Autoencoders for Remote Physiological Measurements,” IEEE Transactions on Multimedia, vol. 26, pp. 7278–7293, 2024. [13] X. Zhou, J. Yuan, L. Fan, X. Niu, K. Zha, and X. Liu, “MSMFT: Multi-Stream Multimodal Factorized Transformer for Human Activity Recognition,” IEEE Sensors Journal, vol. 25, no. 6, pp. 10402–10416, 2025. [14] I. Ziogas, L. J. Hadjileontiadis, A. H. Khandoker, and A. Al Shehhi, “Self-Supervised Graph Representation Learning for In-The-Wild Wearable and Smartphone based Emotion Recognition,” ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing Proceedings, 2025. [15] M. Han, O. Özdenizci, Y. Wang, T. Koike-Akino, and D. Erdoǧmuş, “Disentangled adversarial autoencoder for subject-invariant physiological feature extraction,” IEEE Signal Processing Letters, vol. 27, pp. 1565–1569, 2020. [16] T. Su, C. Dai, X. Liu, and X. Jiang, “EMG Biometric Verification Via Disentangled Representations,” IEEE Transactions on Industrial Informatics, vol. 21, no. 4, pp. 3376–3385, 2025. [17] T. Dissanayake, T. Fernando, S. Denman, S. Sridharan, and C. Fookes, “Generalized Generative Deep Learning Models for Biosignal Synthesis and Modality Transfer,” IEEE Journal of Biomedical and Health Informatics, vol. 27, pp. 968–979, 2 2023.

[18] Z. Zhang, H. Lu, Q. Zhao, K. Wen, and B. Liu, “PPG SensorBased Biometric Identification and Physiological Analysis via TemporalFrequency Disentanglement With Liquid Neural Networks,” IEEE Sensors Letters, vol. 9, no. 8, 2025. [19] I. Ziogas, A. A. Shehhi, A. H. Khandoker, and L. J. Hadjileontiadis, “Variational decomposition autoencoding improves disentanglement of latent representations,” Preprint at https://arxiv.org/abs/2601.06844, 1 2026. [20] D. P. . Kingma and M. Welling, “Auto-Encoding Variational Bayes,” in International Conference on Learning Representations (ICLR), Ithaca, NYArXiv, 2014. [21] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, A. Lerchner, and G. Deepmind, “beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework,” in International conference on learning representations (ICLR), 2 2017. [22] A. Hyvärinen, H. J., and H. P.O., “Independent component analysis,” in Natural Image Statistics: A Probabilistic Approach to Early Computational Vision, pp. 151–175, Springer, (2001). [23] B. Schölkopf, A. Smola, and K. R. Müller, “Kernel principal component analysis,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 1327, pp. 583–588, 1997. [24] P. Sriram Kumar, P. K. Govarthan, A. Aleem Shaik Gadda, N. Ganapathy, and J. Fredo Agastinose Ronickom, “Deep Learning-Based Automated Emotion Recognition Using Multimodal Physiological Signals and Time-Frequency Methods,” IEEE Transactions on Instrumentation and Measurement, vol. 73, 2024. [25] V. Dissanayake, S. Seneviratne, R. Rana, E. Wen, T. Kaluarachchi, and S. Nanayakkara, “SigRep: Toward Robust Wearable Emotion Recognition with Contrastive Representation Learning,” IEEE Access, vol. 10, pp. 18105–18120, 2022. [26] D. A. Nguyen, C. Pham, and N. A. Le-Khac, “Virtual Fusion with Contrastive Learning for Single-Sensor-Based Activity Recognition,” IEEE Sensors Journal, vol. 24, no. 15, pp. 25041–25048, 2024. [27] K. He, X. Chen, S. Xie, Y. Li, P. Dollar, and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2022-June, pp. 15979–15988, 11 2021. [28] X. Chen, X. Zhou, M. Sun, and H. Wang, “Temporal Contrastive Learning for Sensor-Based Human Activity Recognition: A Self-Supervised Approach,” IEEE Sensors Journal, vol. 25, no. 1, pp. 1839–1850, 2025. [29] M. A. Carbonneau, J. Zaı̈di, J. Boilard, and G. Gagnon, “Measuring Disentanglement: A Review of Metrics,” IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 7, pp. 8747–8761, 2024. [30] M. Han, O. Ozdenizci, T. Koike-Akino, Y. Wang, and D. Erdogmus, “Universal Physiological Representation Learning with SoftDisentangled Rateless Autoencoders,” IEEE Journal of Biomedical and Health Informatics, vol. 25, pp. 2928–2937, 8 2021. [31] S. Karimi, A. J. Shah, G. D. Clifford, and R. Sameni, “Bidirectional Translation Between ECG and PCG,” pp. 1–5, 12 2025. [32] L. Jing, P. Vincent, Y. LeCun, and Y. Tian, “Understanding Dimensional Collapse in Contrastive Self-supervised Learning,” ICLR 2022 - 10th International Conference on Learning Representations, 10 2021. [33] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,” Advances in Neural Information Processing Systems, vol. 33, pp. 12449– 12460, 2020. [34] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,” 7 2016. [35] D. Hendrycks and K. Gimpel, “Gaussian Error Linear Units (GELUs),” 6 2016. [36] D. E. et al., “Yamnet: A pretrained audio event classifier,” GitHub, https://github.com/tensorflow/models/tree/master/research/audioset/yamnet, 2019. [37] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, “EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces,” Journal of Neural Engineering, vol. 15, p. 056013, 7 2018. [38] M. Greenacre, P. J. Groenen, T. Hastie, A. I. D’Enza, A. Markos, and E. Tuzhilina, “Principal component analysis,” Nature Reviews Methods Primers, vol. 2, pp. 1–21, 12 2022. [39] P. A. et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, (2019). [40] W. T. et al., “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45, (EMNLP,2020).

ZIOGAS et al.: OMNI-MODAL DECOMPOSITION AUTOENCODERS LEARN FULL-STACK WEARABLE DISENTANGLED REPRESENTATIONS

[41] D. P. Kingma and J. L. Ba, “Adam: A Method for Stochastic Optimization,” 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, 12 2014. [42] P. F. et al., “Scikit-learn: Machine learning in python,” The Journal of Machine Learning Research, vol. 12, pp. 2825–2830, (2011). [43] v. d. M. L. and H. G., “Visualizing data using t-sne,” Journal of machine learning research, vol. 9, pp. 2579–2605, (2008).

15

Related documents

• From Weights to Layers : Deep Neural Network Compression for Efficient Inference#11552
• Synergistic Information Disentanglement for Omni-modal Slide Representation Learning in Computational Pathology#927478
• PhysSAE: Mechanistic Interpretability of PINNs with Sparse Autoencoders#1083214
• Phaedra: Learning High-Fidelity Discrete Tokenization for the Physical Science#1141480
• hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition#19805