TimeRoute: Time-Aware Modality Routing and Diffusion for Multi-Modal Recommendation

TimeRoute: Time-Aware Modality Routing and Diffusion for Multi-Modal Recommendation Pengyu Zhang

Yangqin Jiang

Klim Zaporojets

p.zhang@uva.nl University of Amsterdam Amsterdam, The Netherlands

University of Hong Kong Hong Kong, China

Aarhus University Aarhus, Denmark

Congfeng Cao

Paul Groth

University of Amsterdam Amsterdam, The Netherlands

University of Amsterdam Amsterdam, The Netherlands

arXiv:2608.10983v1 [cs.IR] 11 Aug 2026

Abstract Multi-modal recommenders fuse collaborative signals with item modalities such as text, images, and audio, but the usefulness of each drifts over time and at different rates. For example, chocolate purchases typically guided by textual ingredient cues can shift toward visual packaging and ambient audio around Valentine’s Day. This modality time-scale mismatch gives rise to two coupled challenges: (1) users require different modality proportions across temporal contexts, and (2) less relevant modalities are more likely to introduce outdated or misleading signals into the recommender. We address both challenges within a unified diffusion-based recommender, TimeRoute. A temporal-aware modal router maps each user’s aggregated behavioral features to a personalized modality distribution, replacing the globally shared fusion weights used in prior work. The diffusion-based graph reconstructor is then conditioned on the same temporal profile through Feature-wise Linear Modulation (FiLM) with dual-stream long- and short-term denoising heads, suppressing outdated modality edges before they enter the propagation graph. Experiments on TikTok, AmazonBaby, and Amazon-Sports demonstrate consistent improvements of up to 9.8% in Recall@K, Precision@K, and NDCG@K over strong baselines across 10-seed paired tests. Code is available at https: //anonymous.4open.science/r/TimeRoute.

CCS Concepts • Information systems → Recommender systems; • Computing methodologies → Neural networks; Probabilistic reasoning.

Keywords Multi-modal Recommendation, Time-aware Recommendation, Diffusion Models, Modality Fusion ACM Reference Format: Pengyu Zhang, Yangqin Jiang, Klim Zaporojets, Congfeng Cao, and Paul Groth. 2018. TimeRoute: Time-Aware Modality Routing and Diffusion for Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym ’XX, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX

Multi-Modal Recommendation. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’XX). ACM, New York, NY, USA, 12 pages. https://doi.org/XXXXXXX.XXXXXXX

1

Introduction

Multi-modal recommender systems power large-scale platforms such as e-commerce marketplaces and short-video applications, where rich item content from images, text, and audio helps overcome sparse user-item feedback [6, 32]. Recent graph-based approaches that fuse modality-specific signals with collaborative filtering through graph neural networks achieve strong performance across benchmarks [24]. However, the usefulness of each modality is not static. User interests and item popularity drift over time [33], and the drift speed varies across modalities [34, 37]. We refer to this phenomenon as modality time-scale mismatch: the relevance of each modality shifts over time, but not uniformly across modalities. Figure 1 illustrates this point. Chocolate is purchased year-round, but around Valentine’s Day a user’s decision shifts from textual cues such as ingredients toward visual packaging and atmospheric audio, then reverts after the holiday. The key observation is that different modalities become more or less relevant at different moments and evolve at different rates, requiring the same user to rely on different modality proportions over time [15]. This mismatch gives rise to two interrelated problems. The first problem (Problem 1) is that the same user requires different modality fusion proportions in different temporal contexts. Since modality importance shifts at different rates, the optimal fusion depends on which temporal context the user is currently in [17]. A user’s temporal behavior features, such as interaction frequency, recency, and distribution, encode this temporal state. However, existing methods, including DiffMM [6], rely on a globally shared fusion weight, assigning the same modality mixture to a user during a holiday shopping spree and routine purchases months later. To address this, we introduce a temporal-aware modal router that infers each user’s temporal context from aggregated behavioral features and produces a per-user softmax distribution over modalities. The router output determines how modality-specific embeddings are fused before GCN propagation, enabling the model to emphasize the most relevant modalities for each user at each point in time. The second problem (Problem 2) is the flip side of the first: a modality that is highly relevant in one temporal context can become a source of noise in another. Signals that were once informative

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Scenario B

Scenario A

Text

Text

Time Shift

Ingredients, Reviews

Ingredients, Reviews Purchase

Image Gift Packaging, Visual Appeal

Audio Romantic BGM, Atmosphere

Anonymous Author(s)

Chocolates

Rational, Health & Taste

Image

Purchase

User

• Across three benchmarks, TimeRoute consistently outperforms strong baselines. Controlled noise-input ablations show that temporal signals are a key driver of the router’s gains (𝑝 < 0.001); quartile analysis confirms meaningful per-user differentiation (𝑝 < 10−6 ).

Chocolates

Emotional, Gifting & Vibe

Gift Packaging, Visual Appeal

Audio Romantic BGM, Atmosphere

Figure 1: An example of modality time-scale mismatch. The same user purchasing chocolates relies primarily on textual cues such as ingredients during normal periods (Scenario A), and shifts toward visual packaging and audio around Valentine’s Day (Scenario B). Modality relevance varies with the user’s temporal context and evolves at different rates.

may persist long after they have lost their relevance [12, 20]; for example, Valentine-themed visual cues may remain in the graph shortly after Valentine’s Day, even when they are no longer useful for routine purchases. Modalities that are less relevant in the current context are more likely to contribute such outdated signals. While the modal router partially mitigates this by down-weighting lessrelevant modalities, the second problem also manifests at the graph structure level, where modality-induced edges built from outdated signals propagate noise through message passing. To address this structural dimension, we condition the diffusionbased graph reconstruction [6] on temporal signals through Featurewise Linear Modulation (FiLM) [14], which applies learned featurewise scale and shift parameters to modulate the user context. We combine this conditioning with dual-stream long- and short-term denoising heads [28, 31]. The reconstructor thus accounts for temporal context when rebuilding modality-specific graphs, suppressing outdated edges before they propagate outdated noisy signals downstream. Together with the modal router, the framework provides temporal awareness at two complementary levels: the reconstruction of modality-specific graphs (Problem 2) and the fusion of their resulting embeddings (Problem 1). We evaluate TimeRoute on TikTok, Amazon-Baby, and AmazonSports with 10-seed paired tests, and summarize our contributions as follows: • We identify modality time-scale mismatch, the phenomenon that different modalities shift in relevance at different rates over time, as the root cause of two interrelated problems: (1) the same user needs different modality fusion proportions in different temporal contexts, and (2) modalities that are less relevant in the current context are more likely to carry outdated, misleading signals. • We propose TimeRoute, a framework with two complementary components. A temporal-aware modal router personalizes modality fusion for Problem 1; a time-conditioned diffusion reconstructor uses FiLM modulation and dual-stream long- and short-term heads to suppress outdated modality edges for Problem 2.

2

Related Work

Graph-based Multi-Modal Recommendation. Multi-modal recommendation leverages item content such as images, text, and audio to alleviate interaction sparsity and improve robustness. Early approaches treated modalities as side information, fusing content features into matrix factorization or ranking models; VBPR exemplifies this line by injecting visual features into Bayesian personalized ranking for implicit feedback [2]. Building on this, attention-based methods such as ACF reweight user preferences across modalities at the user-item-component level [1]. More recent work adopts graph neural networks as the backbone: MMGCN and GRCN propagate modality signals over user-item graphs to capture high-order dependencies [24, 25], LATTICE extends this idea by mining latent content-induced structures from raw modality features [30], and BM3 employs bootstrap-style self-supervision to learn from imperfect multi-modal signals [35]. Despite this progress, modality fusion in graph-based multimodal recommendation is typically treated as time-invariant: a single fusion weight is learned and applied uniformly across users and time. This formulation is at odds with the modality time-scale mismatch identified in Section 1, where the users with different temporal behavioral profiles require different modality proportions. To address this limitation, our temporally aware modality router generates personalized modality weights from each user’s temporal behavior profile, replacing globally shared fusion with user- and time-adaptive weighting. Temporal Modeling for Recommendation. Temporal modeling has long been recognized as essential for recommender systems. Koren first introduced temporal dynamics into collaborative filtering by modeling drift in user preferences and item popularity [7]. Building on this idea, sequential recommenders such as TiSASRec inject time-interval information into self-attention to distinguish events that are sequentially close but temporally distant [8]. More recent work confronts temporal distribution shift in deployed systems, with retrieval-and-distill paradigms and probabilistic generalization frameworks designed to keep predictions stable as data drifts [34, 37]. In multi-modal settings, XSMoE handles distribution shifts in streaming recommendation by adding expandable side experts that absorb new modalities or items as they arrive [15]. Yet these temporal models remain largely disconnected from multi-modal fusion. Temporal collaborative filtering and sequential recommenders model user and item dynamics, but typically operate on unimodal interaction sequences. Recent work on temporal distribution shift treats prediction as a single evolving signal rather than a fusion of multiple modalities. Likewise, XSMoE’s per-modality experts are designed to absorb newly arriving modalities over time, rather than reweight existing modalities as their relevance changes within a user. Our work occupies this intersection: the modal router uses each user’s temporal profile to drive per-user multi-modal

TimeRoute: Time-Aware Modality Routing and Diffusion for Multi-Modal Recommendation

fusion, while dual-stream long- and short-term heads add temporal modulation within each per-modality denoiser. Diffusion Models for Recommendation. Generative models offer an alternative to discriminative ranking. Early work used GANs and VAEs for interaction modeling, representation learning, and data augmentation under sparsity [9, 19]. Diffusion models were subsequently adopted for their training stability and capacity to model complex interaction distributions and have since been applied to interaction generation, sequential recommendation, and conditional denoising under contextual signals [11, 20, 22]. Building on this line, DiffMM extends diffusion to multi-modal recommendation by reconstructing modality-aware user-item graphs and combining them with cross-modal contrastive learning for alignment [6]. DiffMM and related multi-modal diffusion recommenders condition the denoiser on a user context that is itself invariant to temporal context: the same context vector drives reconstruction whether the user has shifted into a new behavioral regime or remains in a stable long-term pattern. The reconstructed graphs therefore cannot reflect when a modality’s relevance has changed, and outdated modality-induced edges continue to propagate through GCN message passing. We address this by conditioning per-modality denoisers on temporal signals through FiLM modulation with dualstream long- and short-term heads, so that reconstructed graphs respond to shifts in each user’s temporal context.

3

Methodology

We propose TimeRoute (Figure 2), a diffusion-based multi-modal recommender that addresses the two challenges from Section 1. To address Problem 1, that the same user requires different modality fusion proportions in different temporal contexts, we introduce a temporal-aware modal router that produces personalized modality fusion weights from each user’s temporal behavior profile. To address Problem 2, that modalities less relevant in the current context tend to carry outdated, misleading signals, we condition the diffusion-based reconstruction of modality-specific user-item graphs on temporal information through FiLM modulation [14] with dual-stream long- and short-term heads. Together, these two components provide temporal awareness at two complementary levels: the fusion of modality-specific embeddings (Problem 1) and the reconstruction of the modality-specific graphs that produce those embeddings (Problem 2). We first present the preliminaries (Section 3.1) and the pipeline (Section 3.2); we then detail the modal router (Section 3.3) and the time-conditioned diffusion reconstructor (Section 3.4); finally, we describe the joint training objective and the three-phase per-epoch schedule (Section 3.5).

3.1

Preliminaries and Problem Formulation

Task formulation. We consider implicit-feedback top-𝐾 recommendation, where user preferences are inferred from observed interactions such as clicks, purchases, or views, and the goal is to rank the top-𝐾 items most relevant to each user. Let U and V denote the sets of users and items, with interactions encoded by a binary matrix R ∈ {0, 1} | U | × | V | , where 𝑅𝑢,𝑣 = 1 indicates an observed interaction. We write r𝑢 ∈ {0, 1} | V | for the 𝑢-th row of R (the same vector is denoted x0 when discussing diffusion in Section 3.4). Following [6], we score user-item pairs by the inner

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

product of their final embeddings, 𝑦ˆ𝑢,𝑣 = h𝑢⊤ h𝑣 , and return the top-𝐾 items per user. Multi-modal inputs. Each item 𝑣 is associated with raw modality features f̂𝑣𝑚 ∈ R𝑑𝑚 for each modality 𝑚 in the modality set M. In our experiments, M = {image, text} on Amazon-Baby and AmazonSports, and M = {image, text, audio} on TikTok. Temporal inputs. Each observed interaction (𝑢, 𝑣) is associated with a unix timestamp 𝜏𝑢,𝑣 . From the full sequence of timestamps for user 𝑢, we derive a per-user temporal profile from a common set of semantic fields (interaction recency, calendar context, inter-event gaps, weekday/hour patterns). The two downstream pathways consume this profile in different vectorized forms: the modal router uses a 16-dimensional form t̃𝑢 ∈ R16 in which cyclic fields are sin-cos encoded (Section 3.3.1), and the diffusion conditioner uses a 12-dimensional raw form t𝑢 ∈ R12 that is then processed by per-field encoders (Section 3.4.2).

3.2

Graph Recommendation Framework

We introduce the pipeline before the two main components, so that each component’s role in the larger framework is clear. Modality feature projection. To compare and combine modalities of differing dimensionality, we first project raw modality features into a shared 𝑑-dimensional space via learnable linear projectors 𝜙𝑚 (a linear layer with leaky-ReLU): f𝑣𝑚 = 𝜙𝑚 ( f̂𝑣𝑚 ) ∈ R𝑑 for 𝑚 ∈ M. We let F𝑚 ∈ R | V | ×𝑑 denote the matrix stacking f𝑣𝑚 over items. User context embedding. Both diffusion denoising and graph learning benefit from a compact representation that summarizes a user’s history. We introduce learnable ID embedding matrices E𝑈 ∈ R | U | ×𝑑 and E𝑉 ∈ R | V | ×𝑑 , and define the user context embedding for user 𝑢 as: u = (E𝑈 )𝑢 + r𝑢 E𝑉 ∈ R𝑑 , (1) where (E𝑈 )𝑢 is the user’s ID embedding and r𝑢 E𝑉 aggregates the embeddings of items the user has interacted with. The vector u serves as the time-modulation target inside the diffusion denoiser: it is modulated by the FiLM heads to produce the dual-stream user contexts ulong and ushort (Equation (10)) used for denoising. Modality-specific propagation and fusion. The diffusion reconstructor (Section 3.4) produces, for each modality, a reconstructed user-item graph b R𝑚 . Concretely, for each modality 𝑚, we obtain 𝑚 modality-specific user and item embeddings (e𝑚 𝑢 , e𝑣 ) via two complementary propagation steps following [6]: (i) two-step LightGCNstyle propagation [4] on the base interaction graph R initialized with the normalized modality features normalize(F𝑚 ) stacked under the user ID embeddings, producing a base modality embedding e𝑚,base ; and (ii) one-step propagation on the reconstructed (·) 𝑚 R initialized with the ID embeddings [E𝑈 ; E𝑉 ], producing a graph b reconstruction-based embedding e𝑚,rec . The two are combined via (·) 𝑚,base e𝑚 + 𝜆adj e𝑚,rec , (·) = e (·) (·)

(2)

where 𝜆adj controls the contribution of the reconstructed graph relative to the base graph. The modality-specific embeddings are then fused into a single representation per node by the temporalaware modal router defined in Section 3.3, producing e𝑢fuse and efuse 𝑣 . A final stack of 𝐿 GCN layers [4] on the base graph then refines efuse (·) by summing layer-wise outputs and adding a normalized residual

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Time-Conditioned Diffusion Reconstruction

Input Multi-Modal Item Features

Image

Short-term encoder

Audio

𝜷𝒍𝒐𝒏𝒈

𝒔𝒉𝒐𝒓𝒕

𝒔𝒉𝒐𝒓𝒕

𝜸

𝜷

Modality feature projection Per-modality denoisers

𝜸𝒍𝒐𝒏𝒈

Reconstructed Graph

User context

Image

Image

𝒖

Text

Text

(from user ID + interaction history)

Audio

Temporal profile encoding

Text

Audio

Per-modality propagation (base graph + reconstructed graphs)

Routerguided user fusion Per-user modality weights

Router MLP

Image

Audio

Temporal-Aware Modal Router

User Temporal Profile

Recency

Text

Time conditioning

Long-term encoder

Audio User-Item Interaction

Graph Recommendation Framework

Modality-specific feature space 𝑭𝒎

Image Text

Anonymous Author(s)

Global item-side fusion

GCN Layers

Frequency Calendar Context Inter-event Gaps

Frequency Hour

Softmax

Recency

Top-K Recommendation

Is_weekend Gap

Figure 2: Overview of TimeRoute. From each user’s interaction timestamps, a per-user temporal profile is constructed and fed into two parallel pathways. The temporal-aware modal router (bottom) produces per-user modality fusion weights for the GCN. The time-conditioned diffusion reconstructor (top) modulates the user context via dual-stream FiLM heads and reconstructs modality-specific user-item graphs. In the graph recommendation framework, the reconstructed graphs are used together with the base interaction graph in the per-modality propagation block, after which router-guided user fusion and global item-side fusion produce representations for GCN-based top-𝐾 recommendation. term controlled by 𝜆res , producing the final embeddings h (·) used for scoring.

3.3

Temporal-Aware Modal Router

The modal router is the model’s mechanism for addressing Problem 1. It infers each user’s typical temporal context from their interaction history and produces personalized modality fusion weights. We describe its five components: temporal profile construction, the routing function, the weight floor, per-user modality fusion, and a diversity regularizer.

3.3.1 User Temporal Profile Construction. For each user 𝑢, we extract 12 raw temporal fields from the user’s interaction timestamps. The same set of fields feeds two downstream pathways with different vectorization: a 16-dimensional router input t̃𝑢 used in this section, and the 12-dimensional raw form t𝑢 that the diffusion conditioner processes through per-field encoders (Section 3.4.2). The 12 fields are organized into three semantic groups: Long-term context (4 fields): global_days, user_days, year_offset, month, encoding the user’s overall position in time and historical span.

Short-term context (6 fields): delta_log (log-scaled inter-event gap), delta_bucket (a quantile-based gap id, used only by the diffusion conditioner and not part of the router input), pos_norm (normalized recency), weekday, hour, is_weekend, encoding the local temporal structure of recent activity. Auxiliary fields (2 fields): day of month and unix_timestamp, used by the router to disambiguate users with otherwise identical aggregated signatures. Each field is aggregated across the user’s interactions using a recency-weighted average, with weights proportional to interaction position so that more recent interactions contribute more. Cyclic fields (month, day, weekday, hour) are aggregated via sin-cos averaging followed by angular reconstruction, ensuring that, for example, the average of 23:00 and 01:00 is 00:00 rather than 12:00. Vectorization for the router. For the router, the four cyclic fields are converted to (sin, cos) pairs to preserve circular proximity, so that adjacent values such as December and January or 23:00 and 01:00 remain close in representation space (8 dimensions). The four continuous temporal fields (global_days, user_days, year_offset, unix_timestamp) are normalized to a comparable range to prevent large-scale timestamp values from dominating the router MLP (4 dimensions). We clip delta_log and pos_norm to [0, 1] to bound extreme gap and recency values (2 dimensions), and encode is_weekend as a 2-dimensional one-hot indicator to represent the binary calendar context without imposing an ordinal scale. This

TimeRoute: Time-Aware Modality Routing and Diffusion for Multi-Modal Recommendation

yields the 16-dimensional router input t̃𝑢 ∈ R16 , which encodes when each user is active, how recently, and how their activity is distributed. The encoding is lightweight, adding only 610 parameters on Amazon-Baby and Amazon-Sports and 643 on TikTok (see Appendix1 ). 3.3.2 Routing Function. Given t̃𝑢 , the router produces a probability distribution over modalities through a two-layer MLP followed by a softmax:
w𝑢 = Softmax W2 𝜎 (W1 t̃𝑢 + b1 ) + b2 , (3) where W1 ∈ R32×16 , W2 ∈ R | M | ×32 , and 𝜎 is the ReLU activation. The output w𝑢 ∈ Δ | M | −1 is a per-user modality distribution. We initialize W2 and b2 to zero so that w𝑢 = 1/|M| at initialization; this warm start prevents aggressive routing decisions before the modality-specific embeddings have meaningfully differentiated. As training progresses, the router gradually departs from uniform toward dataset- and user-appropriate weights. 3.3.3 Weight Floor. A naive softmax can produce degenerate solutions in which one modality is entirely suppressed across all users, particularly on datasets where one modality dominates the collaborative signal. Such global modality collapse would undermine Problem 1: instead of learning user-specific modality proportions, the router would simply remove a modality from the fusion process. To prevent this, we rescale the softmax output to enforce a mini′ mum weight floor: 𝑤𝑢,𝑚 = 𝜖 + (1 − |M|𝜖) 𝑤𝑢,𝑚 with 𝜖 = 0.05, which Í ′ = 1) while guaranteeing preserves the simplex structure ( 𝑚 𝑤𝑢,𝑚 ′ 𝑤𝑢,𝑚 ≥ 𝜖 for every modality. Even modalities deemed less relevant by the router still contribute a small but non-trivial signal, which reduces sensitivity to miscalibrated routing decisions and ensures that all modality branches continue to receive training signal. 3.3.4 Per-User Modality Fusion. Modality fusion is applied differently on the user and item sides: ∑︁ ′ e𝑢fuse = 𝑤𝑢,𝑚 · e𝑚 𝑢 ∈ U, (4) 𝑢, 𝑚∈ M

efuse = 𝑣

∑︁

𝛽𝑚 · e𝑚 𝑣 ,

𝑣 ∈ V,

(5)

𝑚∈ M

˜ is a globally learned modality weight shared where 𝜷 = Softmax( 𝜷) across all items. This difference reflects an information asymmetry: user-level temporal profiles provide informative signals for personalized modality weighting (e.g., activity level and recency can shift the balance between visual and textual signals; see Section 4.4), whereas items lack an analogous temporal context. We therefore retain a global item-side modality weight, allowing 𝜷 to capture dataset-level modality importance without introducing unnecessary item-level routing parameters. 3.3.5 Diversity Regularization. The weight floor (Section 3.3.3) prevents any single modality from being fully suppressed, but does not by itself prevent a more subtle failure mode: the router can converge to a near-uniform global weight that is essentially shared across users, recovering DiffMM-like behavior with no meaningful peruser adaptation. To explicitly encourage per-user differentiation, we add a diversity regularizer. This directly supports Problem 1: if the 1 https://anonymous.4open.science/r/TimeRoute/TimeRoute_appendix.pdf

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

router collapses to nearly identical weights for all users, it reduces to a global modality mixture and can no longer adapt modality fusion to user-specific temporal behavior:  1 ∑︁  1 ∑︁ Ldiv = H(w𝑢 ) − H w𝑢 , (6) |B| 𝑢 ∈ B |B| 𝑢 ∈ B | {z } | {z } per-user entropy

batch-mean entropy

Í

where H(p) = − 𝑚 𝑝𝑚 log 𝑝𝑚 is the Shannon entropy, B is the current minibatch of users, and w𝑢 denotes the pre-floor softmax output (Equation (3)) to preserve gradient flow through the router during optimization. Minimizing Ldiv encourages each user toward a confident routing decision (low per-user entropy) while keeping the batch-averaged distribution broadly balanced across modalities (high batch-mean entropy). This encourages differentiated userspecific modality proportions while preventing collapse to a single globally dominant modality. The regularizer is added to the joint loss with a single hyperparameter 𝜆div , tuned per dataset (see our released code at https://anonymous.4open.science/r/TimeRoute for full values).

3.4

Time-Conditioned Diffusion Reconstruction

To address Problem 2, we extend the diffusion-based graph reconstruction pipeline of DiffMM [6] along two orthogonal axes. Time conditioning is introduced via FiLM-modulated user context with dual-stream long- and short-term heads (Section 3.4.2). Modality grounding is introduced via an alignment regularizer that ties modality-specific reconstructions to the modality feature space (Section 3.4.3). The reconstructed graphs b R𝑚 produced by this module serve as inputs to the modality-specific propagation described in Section 3.2. 3.4.1 Diffusion Background. We follow the standard DDPM formulation [5, 6]: the forward process gradually corrupts x0 = r𝑢 √ via a Gaussian transition kernel 𝑞(x𝑡 | x0 ) = N ( 𝛼¯𝑡 x0, (1 − 𝛼¯𝑡 )I) with variance schedule {𝛼¯𝑡 }, and a per-modality MLP denoiser 𝐷𝜃𝑚 ′ ′ predicts x̂𝑚 0 = 𝐷𝜃𝑚 (x𝑡 , 𝑡, u ), where u is the time-modulated user context defined in the next subsection. Up to this point, the formulation follows DiffMM. TimeRoute differs in two aspects. First, the denoiser is conditioned on each user’s temporal profile through dual-stream FiLM modulation: long- and short-term temporal encoders generate user-specific scale and shift parameters for the denoising context, and a gate combines the two reconstructions. Second, the denoiser is trained with modality-aware alignment and a time-reweighted reconstruction loss, encouraging reconstructed graphs to remain both modality-consistent and sensitive to recent behavioral shifts. 3.4.2 Time Conditioning via FiLM and Dual-Stream Heads. Each modality 𝑚 ∈ M has its own denoiser 𝐷𝜃𝑚 , with independently learned field-level encoders, long- and short-term fusion MLPs, FiLM heads, and a gating network. For clarity, we describe the architecture generically below; the same formulation applies to every modality, with parameters learned independently for each 𝑚. A standard diffusion-based recommender conditions the denoiser on a single user context u, which remains fixed across temporal contexts. As a result, the denoiser tends to reconstruct similar interaction patterns even when the user’s recent behavior shifts, increasing the

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Anonymous Author(s)

risk of propagating outdated modality signals (Problem 2). We address this limitation by time-modulating u via FiLM and equipping the denoiser with separate long- and short-term streams, which capture slowly and rapidly evolving behavioral dynamics, respectively. Field-level temporal encoding. Each of the 12 fields in t𝑢 is encoded individually before fusion. Continuous temporal fields, such as global_days, user_days, year_offset, and unix_timestamp, are projected through a linear layer. Cyclic fields, including month, day, weekday, and hour, are transformed into (sin, cos) representations to preserve periodic structure, keeping values such as December and January or 23:00 and 01:00 close in representation space; the inter-event gap field additionally provides a quantile-based bucket id 𝑏𝑢 that is mapped through an embedding table to capture nonlinear gap effects. The per-field representations are then routed into two streams based on temporal scope:
long h𝑢 = MLPlong [𝜙 (𝑡)] 𝑡 ∈ Slong , (7)
short h𝑢 = MLPshort [𝜙 (𝑡)] 𝑡 ∈ Sshort , (8) where 𝜙 (·) denotes the appropriate per-field encoder, [ · ] denotes concatenation, and the long- and short-term field sets are Slong = glb pos yr {𝑡𝑢 , 𝑡𝑢usd, 𝑡𝑢 , 𝑡𝑢mo } and Sshort = {𝑡𝑢dlt, 𝑏𝑢 , 𝑡𝑢hr, 𝑡𝑢wd, 𝑡𝑢we, 𝑡𝑢 }. The partition into Slong and Sshort reflects complementary temporal scales: Slong captures slowly evolving signals, such as long-term usage history and seasonal patterns, whereas Sshort models rapidly changing interaction dynamics, including inter-event timing and time-of-day effects. Dual FiLM parameters. Each stream produces its own scale and shift parameters that modulate the user context u defined in Equation (1). For stream 𝑠 ∈ {long, short}: (𝜸 𝑢𝑠 , 𝜷 𝑢𝑠 ) = F𝑠 (h𝑢𝑠 ), 𝑠

u

= u ⊙ 𝜸 𝑢𝑠 + 𝜷 𝑢𝑠 ,

(9) (10)

where Flong and Fshort are linear heads. The FiLM shift parameters 𝜷 𝑢𝑠 are distinct from the item-side fusion weights 𝛽𝑚 in Equation (5); we follow the standard FiLM notation despite this overlap. Following [14], we parameterize 𝜸 = 1 + ELU(·) and 𝜷 = tanh(·) so that the modulation is centered around an identity transformation, preventing the conditioner from destabilizing early training. Dual-stream denoising and gated combination. The denoiser is run twice per training step, once with ulong and once with ushort as the conditioning user context, producing two reconstructions 𝑚,long x̂0 and x̂𝑚,short . The two are combined via a per-user gate: 0
long 𝛼𝑢 = 𝜎 MLPgate ([ h𝑢 , h𝑢short, e𝑡 ]) , (11) 𝑚,long x̂𝑚 + (1 − 𝛼𝑢 ) · x̂𝑚,short , 0 = 𝛼𝑢 · x̂0 0

(12)

where 𝜎 is the sigmoid function and e𝑡 is the diffusion-step embedding. This design enables the denoiser to adaptively balance complementary temporal dynamics on a per-user basis, interpolating between slowly evolving long-term patterns and rapidly changing short-term behavior. The user-specific gate 𝛼𝑢 adjusts this balance according to the temporal behavior profile. 3.4.3 Modality-aware Alignment Regularization. The denoiser does not take the modality feature matrix F𝑚 as input directly. To inject modality semantics, we introduce an alignment regularizer

that ties the reconstructed user-item scores to the modality feature space. Concretely, for user 𝑢, the denoiser predicts a clean interaction vector x̂𝑚 0,𝑢 over items, where the subscript 0 denotes the denoised diffusion state rather than a user index. Projecting this vector through the modality feature matrix F𝑚 should be consistent with the observed-interaction summary x0,𝑢 E𝑉 : h i 2 𝑚 𝑚 L𝑚 = E x̂ F − x E . (13) 𝑢 0,𝑢 𝑉 0,𝑢 align 2 𝑚 The term x̂𝑚 0,𝑢 F projects the denoiser’s predicted interactions into modality-𝑚 feature space, producing a modality-aware representation of user 𝑢. In contrast, x0,𝑢 E𝑉 projects the user’s observed interactions into the shared ID embedding space. Minimizing the distance between the two aligns the modality-specific reconstruction with observed interaction behavior, encouraging the denoiser to remain faithful to the user’s preferences while respecting the geometry of modality 𝑚. This keeps the denoiser architecture compact while still enabling modality-differentiated graphs b R𝑚 .

3.4.4 Time-Reweighted Reconstruction Loss and Graph Assembly. A standard diffusion reconstruction loss treats all of a user’s interactions equally. However, more recent interactions are typically more diagnostic of the user’s current preferences, and inaccurate reconstruction near the temporal frontier is precisely where outdated modality signals are most likely to mislead the recommender. We therefore introduce a time-reweighted variant of the per-modality reconstruction loss: h i
𝑚 2 L𝑚 diff = E𝑢, 𝑡, 𝝐 𝑤 snr (𝑡) · 1 + 𝜆𝑡 · pos_norm𝑢 · x0 − x̂0 2 , (14) where 𝑤 snr (𝑡) is the SNR-derived diffusion-step weight (standard in DDPM training), pos_norm𝑢 ∈ [0, 1] is the normalized recency of 𝑢’s most recent interaction (1 corresponds to the most recent), and 𝜆𝑡 is a hyperparameter tuned per dataset. The reweighting biases the denoiser to prioritize accurate reconstruction near the temporal frontier, where modality relevance is most likely to have shifted, directly addressing the structural side of Problem 2. Once the per-modality denoisers are trained, we sample x̂𝑚 0 via reverse diffusion under the FiLM conditioning of Section 3.4.2 and assemble the modality-specific reconstructed graph b R𝑚 by taking the top-𝑘 predicted items per user as edges. These reconstructed graphs are then passed to the downstream per-modality propagation block, as illustrated in the rightmost panel of Figure 2.

3.5

Training Objective and Optimization

3.5.1 Joint Loss Function. The model is optimized with an alternating three-phase schedule under a composite objective. that combines BPR-based ranking, cross-modality contrastive learning, the per-modality diffusion losses, the router diversity regularizer, and weight regularization: ∑︁ 𝑚
2 L = LBPR + 𝜆cl LCL + L𝑚 diff + 𝜆𝑒 Lalign + 𝜆div Ldiv + 𝜆reg ∥Θ∥ 2 , 𝑚∈ M

(15) where LBPR [16] is the standard pairwise ranking loss computed from the GCN-fused embeddings; LCL is the multi-view contrastive loss inherited from DiffMM [6], which contrasts modality-specific user/item views against an aggregated view that does not pass

TimeRoute: Time-Aware Modality Routing and Diffusion for Multi-Modal Recommendation

through the router; L𝑚 is the time-reweighted diffusion loss (Equadiff tion (14)); L𝑚 is the modality-aware alignment regularizer (Equaalign tion (13)); Ldiv is the router diversity regularizer (Equation (6)); and Θ collects all trainable parameters. The hyperparameters 𝜆cl , 𝜆𝑒 , 𝜆div , 𝜆𝑡 , and 𝜆reg are tuned per dataset (see the released code for full values). 3.5.2 Three-Phase Per-Epoch Schedule. Each training epoch proceeds in three phases: Diffusion training. For each minibatch of users, we run a forwardreverse pass through every per-modality denoiser, conditioned on the user’s temporal profile and the FiLM-modulated user context (Equations (7)-(12)). Only the denoiser-side parameters {𝜃𝑚 }𝑚 are updated, including the temporal encoders, FiLM heads, and denoisÍ ing MLPs, via 𝑚 (L𝑚 + 𝜆𝑒 L𝑚 ). This isolates graph generation diff align from ranking optimization, so the denoisers first learn temporally conditioned modality-specific reconstructions before the recommendation modules are trained on the assembled graphs. Graph reconstruction. Using the just-updated denoisers, we samb𝑚 ple x̂𝑚 0 for every user and modality and assemble { R }𝑚∈ M via the top-𝑘 procedure in Section 3.4.4. GCN and recommendation training. Using {b R𝑚 } as inputs to modality-specific propagation, we compute the fused user and item embeddings via the modal router (Equations (4)-(5)) and update the ˜ and 𝜃 router ) remaining parameters (E𝑈 , E𝑉 , projectors {𝜙𝑚 }𝑚 , 𝜷, via LBPR + 𝜆cl LCL + 𝜆div Ldiv + 𝜆reg ∥Θ∥ 22 . Although LCL appears in the Phase 3 loss, the contrastive views are constructed by per-modality propagation without routing. This way, LCL does not flow gradients through the router. This isolation prevents the contrastive objective, which can favor near-uniform modality weights, from pulling the router away from rankingoptimized per-user fusion. This is important for Problem 1: the router must preserve user-specific modality distributions rather than collapse toward a global mixture.

4

Experiments

We evaluate TimeRoute on three widely used multi-modal benchmarks: TikTok, Amazon-Baby, and Amazon-Sports.2 The three datasets span different temporal ranges, timestamp coverages, and user activity distributions, which allow us study modality time-scale mismatch under diverse settings. Full dataset details and the full list of compared baselines are provided in Appendix.3 We organize the experiments around five research questions: • RQ1 (Overall Effectiveness). Does TimeRoute consistently improve top-𝐾 recommendation over strong baselines across multiple benchmarks and evaluation metrics? • RQ2 (Temporal Necessity). Is the performance improvement driven by temporal information in the router input, or merely by the additional parameters introduced by the router MLP? • RQ3 (User-Level Routing). Does the modal router learn meaningful per-user differentiation in modality preferences, and do these preferences vary with users’ temporal behavior patterns?

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

• RQ4 (Component Contributions). What is the individual contribution of each component, including the modal router, dualstream denoising, FiLM conditioning, and time-weighted loss, to the overall performance? • RQ5 (Noise Robustness). Does learned modality routing improve robustness to modality noise, particularly under uneven corruption across modalities (e.g., stronger Gaussian noise on image features than text features)?

4.1

Experimental Setup

4.1.1 Evaluation metrics. We report three top-𝐾 ranking metrics at 𝐾 = 20: Recall@20 (R@20), Precision@20 (P@20), and NDCG@20 (N@20). R@20 measures whether the ground-truth items appear in the top-20 list; P@20 measures the fraction of correct items among the top-20; and N@20 further weights hits by their rank, rewarding correct items near the top of the list. To highlight gains over a strong reference, we report the relative improvement of TimeRoute over the strongest baseline in each table:   Our Result − Strongest Baseline Δ (%) = × 100%. (16) Strongest Baseline Full metric definitions and dataset preprocessing are in appendix.4 4.1.2 Statistical testing. To ensure reliability, every experiment is run with 10 different seeds (1, 7, 42, 123, 456, 666, 789, 888, 999, 2026), and we report the mean and standard deviation. Statistical significance is assessed with paired two-sided 𝑡-tests at the 𝑝 < 0.05 threshold. The same protocol is applied to all comparisons against baselines and to all ablation comparisons. 4.1.3 Data splits. We use the Random split (main setting) protocol for train/validation/test and report top-𝐾 recommendation performance, matching prior diffusion-based multi-modal recommenders. To rule out concerns about “future” interactions appearing in training and to assess temporal generalization, we additionally report results under a Chronological split (supplementary setting): for each user, earlier interactions are used for training and later ones are held out for validation and test, with the evaluation protocol unchanged.

4.2

Overall Effectiveness (RQ1)

We first examine whether TimeRoute delivers consistent gains across all datasets and evaluation protocols. We report results under two settings: a standard random split evaluated against a full set of strong baselines, and a deployment-aligned per-user chronological split, which probes whether the gains hold under temporal distribution shift. • Random split. Table 1 summarizes the random-split results. TimeRoute outperforms every baseline on every dataset and metric. On TikTok, R@20 rises from the strongest baseline value of 0.1129 to 0.1208 (+7.00%), and N@20 rises from 0.0460 to 0.0493 (+7.17%). On Amazon-Baby and Amazon-Sports, P@20 improves over the strongest baseline by +9.80% and +5.36%, respectively. Compared with the diffusion-based baselines DiffMM and KDiffE, TimeRoute explicitly uses temporal profiles for both modality routing and graph reconstruction, allowing the model to adapt fusion and reconstructed graphs to user-level temporal behavior. The gains are

2 https://github.com/HKUDS/DiffMM/tree/main/Datasets 3 https://anonymous.4open.science/r/TimeRoute/TimeRoute_appendix.pdf

4 https://anonymous.4open.science/r/TimeRoute/TimeRoute_appendix.pdf

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Anonymous Author(s)

not limited to recall: TimeRoute also improves precision and ranking quality, with the largest relative gains observed on P@20 for the Amazon datasets and N@20 for TikTok. • Chronological split. Under the random split, the model observes modality signals from the full temporal range during training, which can obscure the impact of outdated signals at test time. We therefore re-evaluate under a per-user chronological split, training on earlier interactions and testing on later ones. Given the computational cost of rerunning all methods under this protocol, we compare against DiffMM, the closest diffusion-based backbone and the strongest baseline re-evaluated under the chronological split. Figure 3 reports the relative gains. TimeRoute improves every metric on every dataset, with particularly large gains on N@20: +16.11% on TikTok, +11.02% on Amazon-Baby, and +12.59% on Amazon-Sports. This pattern reinforces the findings under the random split: when test interactions reflect a different temporal regime than training, globally shared modality weights cannot adapt to shifting user behavior, whereas a per-user router can.

around 0.20 to below 0.09. At the same time, within-group variability decreases: on TikTok, the per-user standard deviation of 𝑤 img drops from 0.363 at Q1 to 0.194 at Q4 (a 47% reduction), and the mean per-user entropy decreases from 0.650 to 0.476. Together, these patterns suggest that highly active users exhibit increasingly confident, image-dominant routing, whereas sparse users retain more heterogeneous and uncertain modality preferences. The systematic variation across quartiles indicates that the router learns differentiated user-specific modality weighting rather than collapsing to a single global solution. The pattern is consistent with the mechanism underlying Problem 1. Sparse users provide limited collaborative signal, leading the router to distribute weight more evenly across modalities so that textual or audio cues can compensate for weak visual evidence. As interaction histories grow, the visual modality becomes increasingly informative for these users, and the router shifts weight toward image features that better capture their preferences. Because this adaptation is driven entirely by each user’s temporal behavior profile, the resulting fusion is personalized rather than globally shared.

4.3

4.5

Temporal Necessity (RQ2)

The results of RQ1 establish that TimeRoute outperforms the baseline, but leave an important question unanswered: are the gains driven by the temporal signal in the router input, or merely by the additional parameters introduced by the MLP? To disentangle the two effects, we conduct a controlled ablation on Amazon-Baby. The router’s temporal input is replaced with fixed random noise of identical dimensionality (16-dimensional Gaussian, with a seed independent of the training seed), keeping the architecture and parameter count unchanged. Two outcomes are then possible: if the gains arise primarily from MLP capacity, the noise-input router should match the temporal-input router; if they arise from the temporal signal, the noise-input router should perform similarly to removing the router entirely. Table 2 reports both predictions tested side by side. The noiseinput router reaches R@20 = 0.0953, statistically indistinguishable from the no-router baseline at 0.0955 (𝑝 = 0.50). The temporal-input router reaches 0.1023, significantly outperforming both (𝑝 < 0.001). Together, these findings indicate that the router’s gains are driven by temporal information rather than additional MLP capacity.

4.4

User-Level Routing (RQ3)

The results of RQ2 confirm that the temporal signal drives the router’s gain. The next question is whether the router learns meaningful per-user differentiation, or simply converges to a single global weighting that improves over DiffMM’s default. To investigate this, we partition users on Amazon-Baby and TikTok into four interaction-count quartiles, where Q1 contains the sparsest users and Q4 the densest. For each quartile, we report the number of users 𝑛, the mean per-user routing entropy 𝐻 = E𝑢 [H(w𝑢 )], and the learned modality weights 𝑤 img , 𝑤 txt (and 𝑤 aud for TikTok). Figure 4 reveals a clear upward trend on both datasets. On Amazon-Baby, the mean image weight rises from 0.524 at Q1 to 0.588 at Q4, a relative increase of 12.2%. The trend is more pronounced on TikTok: 𝑤 img increases from 0.599 at Q1 to a nearsaturated 0.830 at Q4, while text and audio weights decline from

Component Contributions (RQ4)

RQ2 and RQ3 establish that the router’s gains are driven by temporal information and that the learned routing is genuinely user-specific. We now ask which components of the framework actually contribute to these gains. Four ablations are evaluated, each removing one component while keeping the rest intact: w/o Modal Router, w/o Dual-Stream, w/o FiLM, and w/o Time-Weight. Figure 5 reports R@20 on the random split. Full dataset details are provided in Appendix.5 • Every component contributes. Removing any single component degrades R@20 on every dataset, with relative drops ranging from −2.19% to −4.06%. The four temporal mechanisms therefore act as distinct contributors, rather than redundant components for a single dominant module. • The modal router is the largest contributor on Baby and TikTok. On Amazon-Baby, removing the router reduces R@20 by −3.71%, the largest single-component effect on this dataset. On TikTok the drop is similarly large at −3.56%, tied with the timeweighted loss. Even on Amazon-Sports, where timestamp coverage is the lowest at 63%, removing the router still reduces R@20 by −2.93%, suggesting that it extracts useful user-specific temporal signals rather than collapsing to a uniform weight when timestamps are sparse. • Reconstruction-level components dominate on Sports and TikTok. The dual-stream denoiser is the single largest contributor on Amazon-Sports (−3.29%) and TikTok (−4.06%), while the time-weighted loss also contributes substantially on Amazon-Baby (−3.62%). Together with the modal router, these results show that fusion-level temporal awareness (the router, addressing Problem 1) and reconstruction-level temporal awareness (dual-stream denoising, and time-weighted loss, addressing Problem 2) operate as complementary mechanisms; both layers are required for full performance. • FiLM contributes consistently though not dominantly. Removing FiLM degrades performance by −2.19% to −2.93% across 5 https://anonymous.4open.science/r/TimeRoute/TimeRoute_appendix.pdf

TimeRoute: Time-Aware Modality Routing and Diffusion for Multi-Modal Recommendation

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Table 1: Overall performance on the Random split. We report R@20, P@20, and N@20 on TikTok, Amazon-Baby, and AmazonSports. Δ(%) denotes the relative improvement of TimeRoute over the best baseline in each column. Results for TimeRoute are reported as the mean over 10 random seeds. Best results are in bold; second-best are underlined. TikTok R@20

Amazon-Baby

Amazon-Sports

P@20

N@20

R@20

P@20

N@20

R@20

P@20

N@20

General collaborative filtering MF-BPR [16] 0.0346 0.0017 NGCF [21] 0.0604 0.0030 LightGCN [4] 0.0653 0.0033 SGL [26] 0.0603 0.0030 NCL [10] 0.0658 0.0034 HCCF [27] 0.0662 0.0029

0.0130 0.0238 0.0282 0.0238 0.0269 0.0267

0.0440 0.0591 0.0698 0.0678 0.0703 0.0705

0.0024 0.0032 0.0037 0.0036 0.0038 0.0037

0.0200 0.0261 0.0319 0.0296 0.0311 0.0308

0.0430 0.0695 0.0782 0.0779 0.0765 0.0779

0.0023 0.0037 0.0042 0.0041 0.0040 0.0041

0.0202 0.0318 0.0369 0.0361 0.0349 0.0361

Multi-modal recommendation VBPR [3] 0.0380 0.0018 LGCN-M [4] 0.0679 0.0034 MMGCN [25] 0.0730 0.0036 GRCN [24] 0.0804 0.0036 LATTICE [30] 0.0843 0.0042 CLCRec [23] 0.0621 0.0032 MMGCL [29] 0.0799 0.0037 SLMRec [18] 0.0845 0.0042 BM3 [36] 0.0957 0.0048 DiffMM [6] 0.1129 0.0056 KDiffE [13] 0.1120 0.0056

0.0134 0.0273 0.0307 0.0350 0.0367 0.0264 0.0326 0.0353 0.0404 0.0456 0.0460

0.0486 0.0726 0.0640 0.0754 0.0829 0.0610 0.0758 0.0765 0.0839 0.0975 0.0950

0.0026 0.0038 0.0032 0.0040 0.0044 0.0032 0.0041 0.0043 0.0044 0.0051 0.0050

0.0213 0.0329 0.0284 0.0336 0.0368 0.0284 0.0331 0.0325 0.0361 0.0411 0.0400

0.0582 0.0705 0.0638 0.0833 0.0915 0.0651 0.0875 0.0829 0.0975 0.1017 0.1020

0.0031 0.0035 0.0034 0.0044 0.0048 0.0035 0.0046 0.0043 0.0051 0.0054 0.0056

0.0265 0.0324 0.0279 0.0377 0.0424 0.0301 0.0409 0.0376 0.0442 0.0458 0.0460

TimeRoute Δ (%)

0.0493 +7.17

0.1023 +4.92

0.0056 +9.80

0.0451 +9.73

0.1094 +7.25

0.0059 +5.36

0.0496 +7.83

0.1208 +7.00

0.0059 +5.36

0.10

0.05

TimeRoute

0.08

0.006

+11.08%

+7.69%

0.06

DiffMM

0.04

+6.12%

+2.13%

+6.52%

0.004 0.003 0.002

0.02

DiffMM

TimeRoute 0.04

0.005

+8.35%

P@20

R@20

0.007

N@20

DiffMM

TimeRoute

+16.11%

+12.59% +11.02%

0.03 0.02 0.01

0.001

0.00

TikTok

Baby

(a) R@20.

Sports

0.000

TikTok

Baby

0.00 Sports

TikTok

(b) P@20.

Baby

Sports

(c) N@20.

Figure 3: Stability under the chronological split, comparing TimeRoute with DiffMM, the closest diffusion-based multimodal backbone baseline, on TikTok, Amazon-Baby, and Amazon-Sports. We report (a) R@20, (b) P@20, and (c) N@20. Numbers above each pair of bars give the relative improvement of TimeRoute over DiffMM. the three datasets, the smallest single-component effect on Sports and TikTok but still well above the across-seed noise floor. The consistent degradation indicates that FiLM’s per-user temporal modulation contributes meaningfully to reconstruction-level temporal awareness, rather than redundantly duplicating the role of the router or dual-stream denoising. • All ablations remain above the strongest baseline. Even with any single component removed, the model still outperforms the strongest R@20 baseline in Table 1 on all datasets (AmazonBaby: 0.0985 vs. DiffMM 0.0975; Amazon-Sports: 0.1058 vs. KDiffE 0.1020; TikTok: 0.1159 vs. DiffMM 0.1129). These results suggest

that the framework’s gains are distributed across complementary components rather than driven by any single dominant mechanism.

4.6

Noise Robustness (RQ5)

RQ4 confirms that the framework’s gains hold even when individual components are removed. We next examine robustness. The intuition behind Problem 2 is that temporal routing should be especially beneficial when modalities differ in noise level: by down-weighting less relevant modalities, the model reduces exposure to outdated or noisy signals. To evaluate this, we inject Gaussian noise into modality features at controlled per-modality ratios (noise std =

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Anonymous Author(s)

Table 2: Noise-input ablation on Amazon-Baby (Random split). We compare three configurations: the full model with temporal features, a router with random noise input, and no router. Results are reported as the mean ± std over 10 seeds. 𝑝-values are from paired 𝑡-tests against the full model. Best results per metric in bold. Configuration Full (time features) Noise input No router

R@20

N@20

𝑝 vs Full

0.1023 ± 0.0005 0.0953 ± 0.0007 0.0955 ± 0.0004

0.0451 ± 0.0003 0.0405 ± 0.0003 0.0405 ± 0.0003

< 0.001 < 0.001

Noise vs No router: 𝑝 = 0.50 (not significant)

image

1.0

text

modality weight

modality weight

1.0 0.8 0.6 0.4 0.2 0.0

image

text

Q4 (dense)

0.6 0.4 0.2

Q3

Q2

Q3

n = 5,998

n = 3,785

n = 5,280

n = 4,382

n = 3,517

n = 3,073

n = 2,581

H = 0.210

H = 0.210

H = 0.211

H = 0.212

H = 0.650

H = 0.647

H = 0.639

Q1 (sparse)

Q4 (dense) n = 137

H = 0.476

(b) TikTok (𝑁 = 9,308).

Figure 4: Per-user routing analysis by user activity quartile on (a) Amazon-Baby and (b) TikTok. For each quartile, we plot the mean modality weights 𝑤 img , 𝑤 txt (and 𝑤 aud for TikTok) together with the mean per-user routing entropy. Q1 contains users with the fewest interactions, whereas Q4 contains the most active users.

Δ R@20 vs full model

1%

Baby

Sports

TikTok

0% -1% -2% -3% -4% w/o Router

w/o Dual-Stream

Clean

img_noisy

txt_noisy

Mixed

Ours NoTime

0.1023 0.0978

0.1020 (−0.29%) 0.0967 (−1.12%)

0.1007 (−1.56%) 0.0962 (−1.64%)

0.1019 (−0.39%) 0.0967 (−1.12%)

audio

Q2

(a) Amazon-Baby (𝑁 = 19,445).

Model

0.8

0.0 Q1 (sparse)

Table 3: Uneven noise stability on Amazon-Baby (R@20, mean over 10 seeds). img_noisy: noise on image only (ratio = 0.5). txt_noisy: noise on text only (ratio = 0.5). Mixed: noise_img = 0.5, noise_txt = 0.1. Clean: noise-free runs. The absolute advantage of Ours over NoTime remains in the range 0.0045-0.0053 across all four settings, indicating that the temporal router preserves its benefits when noise levels differ across modalities. Best results per column in bold.

w/o FiLM

w/o Time-Weight

Figure 5: Component ablation on the Random split. Each bar group compares the full model against four ablations (w/o Modal Router, w/o Dual-Stream, w/o FiLM, w/o Time-Weight) on Amazon-Baby, Amazon-Sports, and TikTok. R@20 is reported as the mean over 10 seeds. ratio × feature std) and compare our full model (Ours) against a variant with all temporal components disabled (NoTime). Table 3 reports the four configurations on Amazon-Baby. Ours consistently outperforms NoTime across all settings, with the absolute performance gap remaining stable in the range 0.0045-0.0053, including under uneven modality noise.

Under img_noisy (image noise at ratio 0.5, text clean), Ours drops by only 0.29% while NoTime drops by 1.12%, widening the performance gap from +0.0045 (clean) to +0.0053. The limited degradation suggests that temporal routing mitigates the impact of corrupted image features by reducing reliance on less relevant signals, whereas NoTime remains more sensitive due to its fixed fusion strategy. Under txt_noisy (text noise ratio 0.5, image clean), both models degrade by similar amounts (Ours −1.56%, NoTime −1.64%) and the performance gap remains at +0.0045. Although Ours is more exposed to corrupted text signals due to its stronger reliance on text, the dual-stream denoiser and time-weighted reconstruction loss provide additional graph-level robustness, allowing the model to retain its advantage rather than sacrificing performance under increased noise. Under Mixed (image noise at 0.5, text noise at 0.1), an asymmetric corruption setting, the performance gap reaches its largest value of +0.0052. This setting does not assume that one modality is always noisier than another; instead, it tests whether TimeRoute remains robust when modality reliability is uneven. This configuration more closely reflects real-world modality imbalance, and the widening gap provides strong evidence that adaptive temporal routing is particularly beneficial when modality quality is uneven. Taken together, RQ5 supports the intuition behind Problem 2 from two complementary angles: per-user routing reduces exposure to noisy signals in down-weighted modalities, while the temporal components in the diffusion reconstructor protect against noise in dominant modalities at the graph level. The net effect is a consistent improvement across all four noise settings, with the largest gain observed under the more realistic uneven noise setting.

5

Conclusion

We proposed TimeRoute, a diffusion-based multi-modal recommender for modality time-scale mismatch. TimeRoute combines a temporal-aware modal router, which maps each user’s temporal context to personalized modality weights, with time-conditioned diffusion reconstruction, which uses FiLM-modulated dual-stream denoisers to suppress outdated graph-level signals. Across TikTok, Amazon-Baby, and Amazon-Sports, 10-seed paired tests show consistent improvements over strong baselines. Controlled ablations confirm that the gains come from temporal signals rather than extra

TimeRoute: Time-Aware Modality Routing and Diffusion for Multi-Modal Recommendation

router parameters, while quartile analysis shows systematic peruser routing differentiation. Noise-robustness experiments further show effectiveness under uneven modality quality. TimeRoute has limitations. Routing depends on timestamp coverage; on Amazon-Sports, reconstruction-level components contribute more than the router. When one modality dominates, as visual features do on TikTok, highly active users may converge toward near-single-modality weights. Adaptive routing reduces exposure to down-weighted noisy modalities but increases reliance on the dominant one. Future work could add modality-quality estimation and sequential recommendation.

GenAI Usage Disclosure The authors used generative AI tools for English proofreading, minor grammar edits, sentence rewriting, and clarity improvements. All content was reviewed and approved by the authors, who take full responsibility for the work.

Acknowledgments The first author is supported by the China Scholarship Council (NO. 202206540007) and the University of Amsterdam. This funding source had no influence on the study design, data collection, analysis, or manuscript preparation and approval.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

References [1] Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and TatSeng Chua. 2017. Attentive Collaborative Filtering: Multimedia Recommendation with Item- and Component-Level Attention. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (Shinjuku, Tokyo, Japan) (SIGIR ’17). Association for Computing Machinery, New York, NY, USA, 335–344. doi:10.1145/3077136.3080797 [2] Ruining He and Julian McAuley. 2016. VBPR: visual Bayesian Personalized Ranking from implicit feedback. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (Phoenix, Arizona) (AAAI’16). AAAI Press, 144–150. [3] Ruining He and Julian McAuley. 2016. VBPR: visual bayesian personalized ranking from implicit feedback. In AAAI, Vol. 30. [4] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020. Lightgcn: Simplifying and powering graph convolution network for recommendation. In SIGIR. 639–648. [5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 574, 12 pages. [6] Yangqin Jiang, Lianghao Xia, Wei Wei, Da Luo, Kangyi Lin, and Chao Huang. 2024. DiffMM: Multi-Modal Diffusion Model for Recommendation. In Proceedings of the 32nd ACM International Conference on Multimedia (Melbourne VIC, Australia) (MM ’24). Association for Computing Machinery, New York, NY, USA, 7591–7599. doi:10.1145/3664647.3681498 [7] Yehuda Koren. 2010. Collaborative filtering with temporal dynamics. Commun. ACM 53, 4 (April 2010), 89–97. doi:10.1145/1721654.1721677 [8] Jiacheng Li, Yujie Wang, and Julian McAuley. 2020. Time Interval Aware SelfAttention for Sequential Recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining (Houston, TX, USA) (WSDM ’20). Association for Computing Machinery, New York, NY, USA, 322–330. doi:10. 1145/3336191.3371786 [9] Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. 2018. Variational Autoencoders for Collaborative Filtering. In Proceedings of the 2018 World Wide Web Conference (Lyon, France) (WWW ’18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 689–698. doi:10.1145/3178876.3186150 [10] Zihan Lin, Changxin Tian, Yupeng Hou, and Wayne Xin Zhao. 2022. Improving graph collaborative filtering with neighborhood-enriched contrastive learning. In WWW. 2320–2329. [11] Qidong Liu, Fan Yan, Xiangyu Zhao, Zhaocheng Du, Huifeng Guo, Ruiming Tang, and Feng Tian. 2023. Diffusion Augmentation for Sequential Recommendation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (Birmingham, United Kingdom) (CIKM ’23). Association for Computing Machinery, New York, NY, USA, 1576–1586. doi:10.1145/3583780. 3615134 [12] Shuo Liu, An Zhang, Guoqing Hu, Hong Qian, and Tat-Seng Chua. 2025. Preference Diffusion for Recommendation. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=6GATHdOi1x [13] Xian Mo, Fei Liu, Rui Tang, Jintao Gao, and Hao Liu. 2025. Knowledge-Aware Diffusion-Enhanced Multimedia Recommendation. IEEE Transactions on Multimedia 27 (2025), 9136–9146. doi:10.1109/TMM.2025.3613108 [14] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. 2018. FiLM: visual reasoning with a general conditioning layer. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence (New Orleans, Louisiana, USA) (AAAI’18/IAAI’18/EAAI’18). AAAI Press, Article 483, 10 pages. [15] Yunke Qu, Liang Qu, Tong Chen, Quoc Viet Hung Nguyen, and Hongzhi Yin. 2025. Efficient Multimodal Streaming Recommendation via Expandable Side Mixture-of-Experts. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (Seoul, Republic of Korea) (CIKM ’25). Association for Computing Machinery, New York, NY, USA, 2460–2470. doi:10.1145/3746252.3761390 [16] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In UAI. 452– 461. [17] Antoine Siraudin, Fragkiskos D. Malliaros, and Christopher Morris. 2025. Cometh: A continuous-time discrete-state graph diffusion model. https://openreview.net/ forum?id=AY89HCxunl [18] Zhulin Tao, Xiaohao Liu, Yewei Xia, Xiang Wang, Lifang Yang, Xianglin Huang, and Tat-Seng Chua. 2022. Self-supervised learning for multimedia recommendation. Transactions on Multimedia (TMM) (2022). [19] Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. 2017. IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (Shinjuku, Tokyo, Japan) (SIGIR ’17). Association for Computing

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Machinery, New York, NY, USA, 515–524. doi:10.1145/3077136.3080786 [20] Wenjie Wang, Yiyan Xu, Fuli Feng, Xinyu Lin, Xiangnan He, and Tat-Seng Chua. 2023. Diffusion Recommender Model. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (Taipei, Taiwan) (SIGIR ’23). Association for Computing Machinery, New York, NY, USA, 832–841. doi:10.1145/3539618.3591663 [21] Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural graph collaborative filtering. In SIGIR. 165–174. [22] Yu Wang, Zhiwei Liu, Liangwei Yang, and Philip S. Yu. 2024. Conditional Denoising Diffusion for Sequential Recommendation. In Advances in Knowledge Discovery and Data Mining: 28th Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2024, Taipei, Taiwan, May 7–10, 2024, Proceedings, Part V (Taipei, Taiwan). Springer-Verlag, Berlin, Heidelberg, 156–169. doi:10.1007/978-981-97-2262-4_13 [23] Yinwei Wei, Xiang Wang, Qi Li, Liqiang Nie, Yan Li, Xuanping Li, and Tat-Seng Chua. 2021. Contrastive learning for cold-start recommendation. In MM. 5382– 5390. [24] Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, and Tat-Seng Chua. 2020. Graph-Refined Convolutional Network for Multimedia Recommendation with Implicit Feedback. In Proceedings of the 28th ACM International Conference on Multimedia (Seattle, WA, USA) (MM ’20). Association for Computing Machinery, New York, NY, USA, 3541–3549. doi:10.1145/3394171.3413556 [25] Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019. MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM ’19). Association for Computing Machinery, New York, NY, USA, 1437–1445. doi:10.1145/3343031.3351034 [26] Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. 2021. Self-supervised graph learning for recommendation. In SIGIR. 726–735. [27] Lianghao Xia, Chao Huang, Yong Xu, Jiashu Zhao, Dawei Yin, and Jimmy Huang. 2022. Hypergraph contrastive collaborative filtering. In SIGIR. 70–79. [28] Kun Yi, Qi Zhang, Wei Fan, Longbing Cao, Shoujin Wang, Hui He, Guodong Long, Liang Hu, Qingsong Wen, and Hui Xiong. 2025. A Survey on Deep Learning based Time Series Analysis with Frequency Transformation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (Toronto ON, Canada) (KDD ’25). Association for Computing Machinery, New York, NY, USA, 6206–6215. doi:10.1145/3711896.3736571

Anonymous Author(s)

[29] Zixuan Yi, Xi Wang, Iadh Ounis, and Craig Macdonald. 2022. Multi-modal graph contrastive learning for micro-video recommendation. In SIGIR. 1807–1811. [30] Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang, and Liang Wang. 2021. Mining Latent Structures for Multimedia Recommendation. In Proceedings of the 29th ACM International Conference on Multimedia (Virtual Event, China) (MM ’21). Association for Computing Machinery, New York, NY, USA, 3872–3880. doi:10.1145/3474085.3475259 [31] Qianru Zhang, Yuting Sun, Honggang Wen, Peng Yang, Xinzhu Li, Ming Li, Kwok-Yan Lam, Siu-Ming Yiu, and Hongzhi Yin. 2025. Time Series Analysis in Frequency Domain: A Survey of Open Challenges, Opportunities and Benchmarks. arXiv:2504.07099 [cs.CE] https://arxiv.org/abs/2504.07099 [32] Shengzhe Zhang, Liyi Chen, Dazhong Shen, Chao Wang, and Hui Xiong. 2025. Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendation. In Proceedings of the ACM on Web Conference 2025 (Sydney NSW, Australia) (WWW ’25). Association for Computing Machinery, New York, NY, USA, 3672–3682. doi:10.1145/3696410.3714676 [33] Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. 2022. Dynamic graph neural networks under spatio-temporal distribution shift. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associates Inc., Red Hook, NY, USA, Article 440, 16 pages. [34] Lei Zheng, Ning Li, Weinan Zhang, and Yong Yu. 2025. Retrieval and Distill: A Temporal Data Shift-Free Paradigm for Online Recommendation System. arXiv:2404.15678 [cs.IR] https://arxiv.org/abs/2404.15678 [35] Xin Zhou, Hongyu Zhou, Yong Liu, Zhiwei Zeng, Chunyan Miao, Pengwei Wang, Yuan You, and Feijun Jiang. 2023. Bootstrap Latent Representations for Multimodal Recommendation. In Proceedings of the ACM Web Conference 2023 (Austin, TX, USA) (WWW ’23). Association for Computing Machinery, New York, NY, USA, 845–854. doi:10.1145/3543507.3583251 [36] Xin Zhou, Hongyu Zhou, Yong Liu, Zhiwei Zeng, Chunyan Miao, Pengwei Wang, Yuan You, and Feijun Jiang. 2023. Bootstrap latent representations for multimodal recommendation. In WWW. 845–854. [37] Yuxuan Zhu, Cong Fu, Yabo Ni, Anxiang Zeng, and Yuan Fang. 2025. A Probabilistic Framework for Temporal Distribution Generalization in Industry-Scale Recommender Systems. arXiv:2511.21032 [cs.LG] https://arxiv.org/abs/2511.21032

Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

Related documents

• PixRec: Leveraging Visual Context for Next-Item Prediction in Sequential Recommendation#629894
• PixRec: Leveraging Visual Context for Next-Item Prediction in Sequential Recommendation#628032
• T-RoPE: Time-Aware Rotary Position Embedding for Sequential Recommendation#1104775
• What Makes LLMs Effective Sequential Recommenders? A Study on Preference Intensity and Temporal Context#149287
• IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT#226724