From Language to Behavior: Scaling Sequence Transformers for Industrial Recommendation Ranking with Rec-Native Designs

From Language to Behavior: Scaling Sequence Transformers for Industrial Recommendation Ranking with Rec-Native Designs Jie Chen, Xiangqian Yu, Yanchao Lian, Tan Lu, Run Yang, Zhengchun Shang, Xing Wang, Cheng Chen, Ke Hu, Qiang Li* , Tianjiu Yin, Xiaobing Liu ByteDance {chenjie.996, yuxiangqian.0, lianyanchao, lutan, yangrun,austin.shang, xing.wang, chencheng.kit, xiaohe.00, huangtao.ranger, yintianjiu, will.liu}@bytedance.com

arXiv:2609.01240v1 [cs.IR] 1 Sep 2026

ABSTRACT Scaling Transformers has driven large gains in language modeling, but transplanting this to behavior-sequence modeling in production ranking is challenging: recommendation differs in signal quality, where behavior sequences are noisy, temporally irregular, and sparsely supervised, and in computation asymmetry, where each request scores many candidates against one shared user history under tight latency budgets. We propose ReST, a recommendation-native Transformer scaling framework. For signal quality, it introduces a sequence encoder with dual-gated attention, rotary positional and temporal embedding, stabilized residual normalization, and training-only auxiliary objectives. For computation asymmetry, it factorizes ranking into a heavy reusable encoder and a lightweight cross decoder with projection-free KV attention and token-specific parameterization, coupling user-level shared-prefix training with shared-prefix serving for compute-once, decode-many-times ranking. Across industrial and public benchmarks, ReST achieves higher accuracy and scales more consistently along sequence length, depth, and width, where LLM-style Transformer blocks saturate. A oneweek online A/B test on a production advertising platform improves online AUC by 1.31% and lifts a core revenue metric by 11.93% within a 50 ms P99 budget; ReST has since been fully deployed in production, showing that behavior-sequence scaling remains a promising, under-exploited axis for production ranking.

CCS CONCEPTS • Information systems → Recommender systems.

KEYWORDS Recommender System, Large Recommendation Model, Scaling Laws

1

INTRODUCTION

Transformers have become the dominant scaling recipe in language modeling, where increasing model capacity, data, and training compute yields substantial and consistent gains [13, 17, 29]. Recommendation ranking appears to offer a similar opportunity, as user behavior sequences capture evolving intent beyond static user features. This promise has motivated a long line of behavior-sequence models, from target-aware attention [41] and long-sequence retrieval [3, 23] to recent large recommendation models [37]. However, directly scaling Transformer sequence models for production discriminative ranking remains challenging, because recommendation differs from language modeling in two fundamental ways. ∗ Corresponding author.

The first obstacle lies in signal quality. Unlike natural language, user behavior sequences exhibit three distinct characteristics that make naive scaling statistically inefficient. (i) Behavior reliability. Behavior tokens are implicit feedback: clicks and views may reflect stable preference, but also accidental exposure, exploratory browsing, or biased feedback [15, 32]. Standard self-attention aggregates them through a shared softmax without an explicit mechanism to distinguish reliable from unreliable behavioral evidence. (ii) Temporal structure. User behaviors occur in irregular physical time—adjacent actions may be seconds or days apart, so equal ordinal distances imply very different recency—and purely ordinal encoding is insufficient for multi-scale recency [19]. (iii) Supervision density. Decoder-only language models receive dense next-token supervision, whereas industrial ranking provides only sparse, delayed labels at the user and candidate level [4, 18]. This sparse discriminative supervision provides a less direct training signal to the sequence stack, and the difficulty is further amplified in hybrid rankers: the main discriminative loss can be shortcut through the strong non-sequential DLRM branch, leaving the sequence module weakly supervised even when the history carries useful intent—a sequence-starvation effect that worsens as the module scales. Consequently, naively scaling a vanilla Transformer yields diminishing returns [11]. The second obstacle lies in computation asymmetry in targetconditioned ranking. Unlike language modeling, a ranking request pairs one user history with 𝑁 candidate targets [7, 37]. The two sides benefit from scaling differently: sequence-side computation is shared across candidates, so investing more compute there can be amortized over all 𝑁 targets; target-conditioned interaction, however, depends on each candidate and is instantiated 𝑁 times per request, making per-candidate compute the latency bottleneck while still requiring enough capacity to discriminate fine-grained candidates. An LLM-style decoder-only Transformer places both roles in a single stack and couples computation with parameters, making it hard to serve these divergent needs separately. Scaling the sequence model under a production latency budget therefore requires breaking this symmetry. We propose ReST, a Recommendation-native Scalable Transformer framework that redesigns the Transformer scaling recipe for behavior-sequence modeling in production ranking. To address the signal quality challenge, ReST introduces a recommendation-native sequence encoder block for behavior data, combining dual-gated attention, rotary positional and temporal embedding (RoPE + RoTE), stabilized residual normalization, and training-only auxiliary objectives that strengthen sequence-side supervision. To address computation asymmetry, ReST factorizes ranking into a compute-heavy

Chen et al.

sequence encoder 𝑇 , which contextualizes each user history once and produces reusable memory, and a lightweight cross decoder 𝐶, which scores many candidates through projection-free key/value attention and token-specific query parameterization. We further co-design the training and serving stack around this boundary: userlevel shared-prefix training amortizes prefix computation across samples from the same user, while shared-prefix serving reuses encoder states across candidates within a request. Empirically, ReST restores consistent scaling along behavior sequence length, model depth, and hidden dimension, reduces redundant training and serving computation, and satisfies the production P99 latency budget in online deployment. Our contributions are summarized as follows: 1. Rec-native Transformer for sequence scaling. We identify and empirically diagnose a sequence-starvation effect in hybrid production rankers. We develop a recommendation-native sequence encoder that combines dual-gated attention for noisy behaviors, rotary positional and temporal embedding for irregular temporal structure, and SRN for stable depth scaling, together with two training-only auxiliary objectives that mitigate sequence starvation. 2. Asymmetric architecture with system co-design. We exploit the one-history-to-many-candidates structure of production ranking by factorizing computation into a reusable, compute-heavy sequence encoder and a lightweight candidate-aware cross decoder. Projection-free key/value attention and token-specific parameterization preserve candidate-side capacity with low activated FLOPs, while user-level shared-prefix training and shared-prefix serving translate this architectural asymmetry into computation reuse. 3. Large-scale empirical validation and online deployment. Across a large-scale industrial advertising dataset and public benchmarks, ReST achieves better accuracy–efficiency trade-offs than LLM-style and recommendation baselines, and continues to scale in sequence length, depth, and width where LLM-style blocks saturate. A one-week online A/B test improves online AUC by 1.31% and lifts a core revenue metric by 11.93% within the 50 ms P99 latency budget, leading to full production deployment.

2 METHOD 2.1 Problem Formulation We study conversion rate (CVR) prediction in large-scale advertising ranking, where industrial rankers combine chronological behavior sequences with non-sequential user/context features [22, 41]. Let U and A denote the user and ad sets. For a user 𝑢 ∈ U, we observe a behavior sequence 𝑆𝑢 = [(s1, 𝑡 1 ), . . . , (s𝐿 , 𝑡𝐿 )], where s𝑖 is the feature vector of the 𝑖-th interaction and 𝑡𝑖 its physical timestamp, together with non-sequential features u𝑑 . In production ranking, these shared user-side inputs are paired with 𝑁 candidate ads {𝑎 1, . . . , 𝑎 𝑁 } ⊆ A. For each candidate 𝑎, the ranker predicts 𝑦ˆ = 𝑓𝜃 (𝑆𝑢 , u𝑑 , 𝑎) ≈ 𝑃 (𝑦 = 1 | 𝑆𝑢 , u𝑑 , 𝑎),

(1)

where 𝑦 ∈ {0, 1} is the binary conversion  label. The model is trained ˆ 𝑦) , where ℓBCE is the standard with the CVR loss Lce = E D ℓBCE (𝑦, binary cross-entropy over the training set D.

This work focuses on scaling the sequence modeling component under this discriminative formulation, while keeping the nonsequential ranking stack and candidate scoring protocol intact.

2.2

Overview

To address the signal quality and computation asymmetry challenges, we propose ReST, a recommendation-native scalable Transformer for behavior-sequence modeling in recommendation ranking (Figure 1). ReST adopts an asymmetric encoder–decoder architecture: a compute-heavy rec-native sequence Transformer encoder 𝑇 builds robust, reusable memory once per user behavior history (Sec. 2.3), while a lightweight cross decoder 𝐶 performs candidatespecific decoding from this memory (Sec. 2.4). We further introduce two training-only auxiliary objectives to mitigate sequence starvation (Sec. 2.5), together with user-level shared-prefix training and shared-prefix serving (Sec. 2.6), enabling compute-once, decodemany-times ranking under production latency budgets.

2.3

Rec-Native Sequence Encoder (𝑇 )

We start from a LLaMA-style [29] Transformer block with causal self-attention and SwiGLU [27] feed-forward layers, but adapt it to the distinct properties of recommendation behavior sequences. Naively scaling such blocks is hindered by noisy interactions, irregular temporal gaps, and unstable optimization as depth grows under sparse supervision. We therefore introduce three recommendationnative sequence modeling designs: Dual-Gated Attention for noisy behaviors, Rotary Positional and Temporal Embedding for ordinal and temporal structure, and Stabilized Residual Normalization for stable depth scaling. Dual-Gated Attention (DGA). Implicit feedback in recommendation is inherently noisy [32, 41]: accidental clicks, weak intent, and exploratory actions may not faithfully reflect stable user preference. This noise affects attention in two stages. Before aggregation, an individual behavior token may be unreliable evidence; after aggregation, the resulting behavioral context may still be only weakly useful for updating the representation. We therefore design DualGated Attention (DGA) to make denoising explicit on both sides of the attention operator. A value gate performs pre-aggregation filtering over behavior features, while an output gate performs post-aggregation context modulation. Given an input sequence X ∈ R𝐿×𝑑 ,   H = Attention XW𝑄 , XW𝐾 , 𝜎 (XW𝑔𝑣 ) ⊙ (XW𝑉 ) , (2)
DualGatedAttn(X) = H ⊙ 𝜎 (XW𝑔𝑜 ) W𝑜 , (3) √ where Attention(Q, K, V) = softmax(QK⊤ / 𝑑 + Mcausal )V, 𝜎 is the sigmoid, and ⊙ is element-wise multiplication. The value gate W𝑔𝑣 attenuates unreliable interaction features before they contribute to any context, while the output gate W𝑔𝑜 controls how much the aggregated behavioral context updates each token. Decoupling token-level reliability filtering from context-level modulation yields robustness to noisy histories without modifying the attention softmax, preserving compatibility with efficient attention kernels. Building on gated-attention designs that primarily modulate the aggregated attention output [14, 25, 37], DGA additionally gates the value stream, suppressing noisy behavior features before they

Scaling Sequence Transformers for Recommendation with Rec-Native Designs

Output

Linear

Linear Linear Linear Linear SiLU Stabilized Residual Norm

SiLU Stabilized Residual Norm

Linear

Linear Linear Linear

Linear Linear Linear

Linear

Linear Feed-Forward

Feed-Forward (TSP)

Stabilized Residual Norm

Stabilized Residual Norm

Dual Gated Self Attention

Gated Cross Attention (TSP)

Linear

Linear

Sigmoid

Linear

Causal Self Attention

Cross Attention

Sigmoid Sigmoid

RoPE + RoTE Linear

Linear

Linear

Linear

Linear

O(gate)

Q

K

V

V(gate) Behavior Sequence

Linear Linear Linear Linear

Linear Linear Linear Linear

O(gate)

Q

K

V

Query Token

Figure 1: ReST architecture: a compute-heavy sequence encoder 𝑇 and a lightweight cross decoder 𝐶. enter the aggregated context. HSTU uses an unbounded SiLU output gate [37]; in our setting, we observe that such gates can exhibit attention-output divergence at scale and therefore adopt bounded sigmoid gates for stable model scaling. Our ablation study shows that the value and output gates are complementary, and Appendix A.1 further validates the denoising effect of DGA. Rotary Positional and Temporal Embedding (RoPE + RoTE). Language tokens are evenly spaced in an ordinal sequence. Modern LLMs commonly use RoPE [28] to encode relative ordinal positions through rotation, a formulation compatible with FlashAttentionstyle kernels [8]. User behaviors, however, live in irregular physical time, where user intent evolves with elapsed time: two clicks seconds apart and two clicks days apart should be treated differently even when they occupy adjacent sequence positions. Prior time-aware recommenders encode time through relative interval embeddings, temporal biases, or bucketized features [6, 19, 37], which are less aligned with length extrapolation and rotary-friendly efficient attention. We therefore design a rotary temporal embedding that preserves the rotary formulation while supporting multigranularity time intervals by splitting attention heads into two complementary groups. We propose Rotary Temporal Embedding (RoTE), which rotates queries and keys by the discretized physical timestamp under a head-specific time granularity 𝜏ℎ . To jointly model ordinal order and physical time, we split attention heads into two groups: heads in Hpos use standard RoPE [28] for relative ordinal positions, while heads in Htime use RoTE for relative time intervals. Small 𝜏ℎ values at the second/minute scale capture fine temporal proximity, whereas large 𝜏ℎ values at the day/week scale capture coarse recency. For each sequence, we first shift timestamps by its start time, 𝑡˜𝑖 = 𝑡𝑖 −𝑡 1 , and then define 𝑏𝑖 = ⌊𝑡˜𝑖 /𝜏ℎ ⌋. This shift avoids unnecessarily large absolute bucket indices without changing pairwise time intervals. For head ℎ, the attention score between query qℎ at position 𝑛 and key kℎ at position 𝑚 is
⊤
  ℎ ∈ Hpos,   R Θ (𝑛)qℎ R Θ (𝑚)kℎ , Scoreℎ (qℎ , kℎ ) = (4)
⊤
  R Θ (𝑏𝑛 )qℎ R Θ (𝑏𝑚 )kℎ , ℎ ∈ Htime,  where R Θ is the block-diagonal rotary matrix and 𝑏𝑖 bucketizes timestamps at granularity 𝜏ℎ . Because both position and time are

encoded through the same rotary mechanism, RoTE unifies ordinal and temporal proximity at multiple granularities within a single attention operator, which is naturally compatible with FlashAttention. Appendix A.2 derives the relative-time property of RoTE.

Stabilized Residual Normalization (SRN). Pure-attention stacks have a theoretical bias toward token uniformity or rank collapse with depth, although skip connections and MLPs mitigate this degeneration [9]; related analyses connect over-mixing and representational collapse to depth in decoder-only LLMs [2]. More generally, the training objective strongly affects over-smoothing and the utility of deeper Transformer layers [34]. In discriminative recommendation ranking, however, the sequence encoder is trained from sparse labels, and the main loss can even be shortcut by strong non-sequential DLRM features. As a result, directly deepening a Transformer, even with the LLM-style Pre-LN configuration, often yields limited gains in recommendation scaling [11]. We propose a simple and effective SRN to address this depthscaling bottleneck with two minimal modifications. First, it adjusts the LN placement in a Mix-LN-style depth-dependent manner [20] to regulate backward gradient flow across layers. Second, it adds a learnable scalar initialized to a small value to each residual branch to control how much each layer updates the forward representation [1]. For the 𝑙-th layer (where SubLayer ∈ {Attn, FFN}),

x𝑙+1 =

   LN x𝑙 + 𝛼𝑙 · SubLayer(x𝑙 ) , 𝑙 ≤ 𝐿switch, 
  x𝑙 + 𝛼𝑙 · SubLayer LN(x𝑙 ) , 𝑙 > 𝐿switch, 

(5)

where 𝐿switch is roughly 25% of the total depth, and 𝛼𝑙 is initialized to 0.01. This small initialization makes the deep encoder start near an identity mapping, so residual updates are introduced gradually as they receive sufficient training signal. Together with depthdependent LN placement, this forward/backward control provides a more stable residual path under sparse, shortcut-prone supervision and leads to more consistent gains under depth scaling.

Chen et al.

2.4

Lightweight Cross Decoder (𝐶)

14

Projection-Free Key/Value. Because 𝑇 already produces contextualized memory, 𝐶 directly uses H = 𝑇 (𝑆𝑢 ) as both keys and values, removing W𝐾 , W𝑉 , and the value gate W𝑔𝑣 from repeated candidate-side computation. Given candidate query tokens Z, the resulting cross-attention is     (ZW𝑄 )H⊤ CrossAttn(Z, H) = softmax H ⊙ 𝜎 (ZW ) W𝑜 , √ 𝑔𝑜 𝑑ℎ (6) where W𝑔𝑜 is the output-gate projection. Token-Specific Parameterization for Query. The query tokens in 𝐶 play different semantic roles, so we assign each token type its own parameters for candidate-side transformations such as W𝑄 , W𝑔𝑜 in the cross attention, and FFN sublayers. For a transformation F , TSP applies F𝑖 (x𝑖 ) = F (x𝑖 ; W𝑖 ),

𝑖 = 1, . . . , 𝐾 .

(7)

Each token activates only its own parameter set, increasing decoder capacity without materially increasing activated FLOPs.

2.5

Auxiliary Supervision for Sequence Scaling

Unlike LLMs, industrial recommendation rankers are hybrid systems that fuse non-sequential DLRM features [22] with user behavior sequences [41] in the final prediction head. This creates a training imbalance we call sequence starvation: the dense, memorizationheavy non-sequential branch shortcuts the main CVR objective and leaves the sequence branch weakly supervised, hindering effective scaling of the sequence encoder. We address this with two training-only auxiliary objectives: an auxiliary sequence CVR loss that directly supervises the sequence branch, and an alignment loss that regularizes sequence/non-sequence representations. Auxiliary Sequence CVR Supervision. Let Hseq ∈ R𝐾 ×𝑑 denote the final states of all 𝐾 sequence-conditioned query tokens. To provide a direct supervised signal to the sequence side, we attach a lightweight auxiliary sequence CVR head to Hseq and the candidate ad features. Unlike the main prediction head, this auxiliary head cannot rely on non-sequential user/context DLRM features. It therefore applies the same BCE supervision to a sequence-conditioned CVR prediction:  
𝑦ˆseq = 𝜎 𝑔seq (Hseq, 𝑎) , Lseq = E D ℓBCE (𝑦ˆseq, 𝑦) . (8) As shown in Figure 2, even with a small auxiliary weight 𝜆seq = 0.1, the auxiliary sequence CVR loss Lseq increases the average gradient norm of the sequence encoder by more than 3×, indicating that direct sequence-side supervision substantially strengthens the optimization signal received by the sequence branch. Appendix A.3 provides additional details.

Avg. Grad Norm

The cross decoder 𝐶 uses a small set of semantic query tokens {[CLS], context, user, ad} to attend to the reusable sequence memory from 𝑇 . Since 𝐶 runs for every candidate, we keep its activated computation small while preserving candidate-side capacity through two design choices.

w/ Aux. Seq-CVR (L seq ) w/o Aux. Seq-CVR

12 10 8 6 4 2 0

0

50

100

150

200

250

300

Training Step (K)

Figure 2: Gradient norm of sequence-encoder parameters with and without auxiliary sequence CVR supervision.

Seq/Non-Seq Alignment. Beyond direct label supervision, we regularize the relationship between sequential and non-sequential user representations. Inspired by vision–language representation alignment [26, 38], we treat behavior sequences and non-sequential features as complementary modalities that describe the same user. We therefore align the behavior-sequence and non-sequential representations with a sigmoid contrastive objective, which is less sensitive to batch size than softmax-based contrastive losses and better suited to scaling recommendation models. Concretely, for user 𝑖, let Hseq,𝑖 denote the sequence-encoder output and let hseq,𝑖 = Hseq,𝑖 [CLS] denote its [CLS] state, which we use as the sequence-level representation. We feed hseq,𝑖 and the non-sequential representation into separate two-layer MLP projection heads followed by ℓ2 normalization, obtaining zseq,𝑖 and zns,𝑖 , respectively. These projection heads are used only during training and removed at inference. We instantiate the alignment loss as: Lalign = −

𝐵 𝐵 
 1 ∑︁ ∑︁ ⊤ log 𝜎 𝑦 · z z /𝜏 − 𝑏 , 𝑖 𝑗 ns,𝑗 seq,𝑖 𝐵 2 𝑖=1 𝑗=1

(9)

where 𝑦𝑖 𝑗 =+1 when the sequence and non-sequence representations come from the same user, and −1 otherwise, with a positive learnable temperature 𝜏 and a learnable bias 𝑏. The overall training objective is: Ltotal = Lce + 𝜆seq Lseq + 𝜆align Lalign .

2.6

(10)

System Co-Design for Computation Reuse

To translate ReST’s architectural separation into actual computation reuse, we co-design the system stack around the shared encoder outputs via the following two shared-prefix mechanisms. User-Level Shared-Prefix Training (ULT). In standard instancelevel training, samples from the same user often share long overlapping behavior prefixes, causing the sequence encoder 𝑇 to repeatedly process nearly identical histories. ULT groups samples from the same user within a time window. For a user with 𝑀𝑢 samples, we run 𝑇 once over the union behavior sequence and apply prefix-valid causal masks so that each sample can only attend to its own valid history. This preserves the original chronological training semantics while reducing the encoder computation from 𝑀𝑢 · FLOPs(𝑇 ) to FLOPs(𝑇 ).

Scaling Sequence Transformers for Recommendation with Rec-Native Designs

ReST LLaMA HSTU

0.9

AUC Gain (%)

Shared-Prefix Serving. At serving time, a ranking request scores 𝑁 candidates against the same user history. Inspired by the MFALCON-style reuse principle [37], we compute the encoder memory H = 𝑇 (𝑆𝑢 ) once per request and reuse it as shared prefix states. Candidate-side query tokens are then batched and decoded by 𝐶 against the same memory. The per-request computation changes from 𝑁 ·FLOPs(𝑇 )+𝑁 ·FLOPs(𝐶) down to FLOPs(𝑇 )+𝑁 ·FLOPs(𝐶). Because 𝐶 is designed to be much lighter than 𝑇 , shared-prefix reuse decouples sequence-side scaling from repeated per-candidate latency, making larger sequence encoders practical under production constraints.

0.8 0.7 0.6 0.5 0.4 2 ¡1

20

21

22

23

24

25

FLOPs (G)

3

EXPERIMENTS

We conduct comprehensive experiments to evaluate ReST’s effectiveness, scalability, efficiency, and deployability. Specifically, we answer five research questions: (RQ1): Does ReST outperform other sequence models under production FLOPs budgets? (RQ2): Does ReST scale better in length, depth, and width than LLM-style baselines? (RQ3): How much does each ReST design component contribute? (RQ4): Do user-level shared-prefix training (ULT) and shared-prefix serving improve training and serving efficiency? (RQ5): Does ReST improve online business metrics under the P99 latency budget?

3.1

Experimental Setup

We primarily evaluate ReST on a large-scale industrial dataset from TikTok Shop Ads. All data are strictly anonymized and de-identified before use, containing no personally identifiable information. We keep the ranking stack, behavior sequence, and query tokens fixed across models, and replace only the sequence-modeling component. Due to confidentiality constraints, we report the relative AUC improvement (AUC Δ) over a highly optimized production DLRM baseline for the industrial evaluation. We also evaluate on three public benchmarks: MovieLens-1M and MovieLens-20M [12], and Amazon-Books [21]. To align these public benchmarks with our primary industrial CVR task, we cast their standard sequential recommendation protocols into a pointwise prediction task via chronological splitting. For these public datasets, we report absolute AUC and log loss under identical evaluation protocols for all models, with additional dataset and implementation details provided in the Appendix. We group baselines into three families. (1) Target-aware attention baselines include DIN [41] and Stacked Target-to-History Cross Attention (STCA) [10]. (2) Causal sequence-model baselines include a LLaMA-style Transformer [29] and HSTU [37]. (3) Trans., a strong encoder–decoder baseline, uses a modernized bidirectional encoder following ModernBERT [33], with Pre-LN, GeGLU, and RoPE. We equip Trans. with token-specific parameterization (TSP) and configure it to approximately match the parameter count of ReST. For a fair comparison, all models use the same sequence preprocessing pipeline and receive the same preprocessed behavior sequence as input. Reported FLOPs and parameter counts cover only the sequence-modeling component. Unless otherwise specified, all sequence-modeling baselines use the auxiliary sequence CVR loss during training to mitigate sequence starvation.

Figure 3: Accuracy vs FLOPs on the internal dataset. ReST achieves favorable accuracy-FLOPs scaling.

3.2

Overall Comparison (RQ1)

Table 1 reports the main comparison on the large-scale industrial dataset. We replace the sequence-modeling component in the production DLRM ranker and roughly align FLOPs across the main sequence-scaling models. ReST consistently outperforms all baseline families at comparable FLOPs, reaching the best AUC lift under both Base and Large budgets and +0.92% in the Large setting. In this production ranking setting, an offline AUC Δ of 0.05% is considered practically meaningful and can translate into measurable online gains. Among the baselines, LLaMA and HSTU outperform DIN and STCA, confirming the value of richer sequence-side self-attention. HSTU is weaker than LLaMA, suggesting that scaling recipes for generative recommendation may not transfer directly to discriminative ranking. The modern encoder–decoder Trans. further improves over LLaMA and is approximately parameter-matched to the full ReST, yet still trails it by 0.20% AUC in the Large setting. Table 3 provides a second parameter-controlled comparison: the parameter-rich cross decoder 𝐶 contributes +0.05% AUC, while the encoder-only ReST, whose parameter count is close to that of LLaMA, retains approximately +0.87% AUC lift and outperforms LLaMA by 0.17%. Together, these comparisons show that ReST’s gains arise from its recommendation-native sequence modeling and asymmetric architecture rather than parameter count alone. We then report results on three public datasets in Table 2. Consistent with the industrial findings, ReST achieves the best AUC and log loss across all datasets, while maintaining comparable FLOPs. This suggests that the benefits of ReST are not merely artifacts of large-scale industrial data, but also transfer to standard public benchmarks.

3.3

Scaling Analysis (RQ2)

Overall scaling. Figure 3 visualizes the joint scaling behavior when layer depth, hidden dimension, and sequence length are scaled together. We focus this scaling analysis on scalable self-attention-style sequence models, since their sequence-side computation can be cached, making them the most relevant family for long-context scaling under production ranking constraints. As a descriptive summary over the tested compute range, we fit ΔAUC(𝐹 ) = 𝑎 · 𝐹 𝑏 ,

Chen et al.

Table 1: Comparison with the production DLRM baseline on the large-scale industrial dataset. Setting Metric

DLRM

DIN

STCA LLaMA HSTU Trans.

ReST

Base

AUC Δ FLOPs (G) Params (M)

-

+0.23% +0.38% 0.07 1.66 0.12 2.97

+0.53% 1.37 0.86

+0.51% +0.57% +0.67% 1.70 1.52 1.52 0.66 5.36 5.69

Large

AUC Δ FLOPs (G) Params (M)

-

+0.27% +0.46% 1.18 31.98 0.46 15.05

+0.70% 30.21 3.43

+0.61% +0.72% +0.92% 32.88 32.38 32.40 1.98 21.33 22.64

Table 2: Public benchmarks on ML-1M, ML-20M, and Amazon-Books. DIN

STCA

LLaMA

HSTU

Trans.

ReST

ML-1M

AUC Log loss

0.8064 0.5346

0.8074 0.5372

0.8087 0.5334

0.8079 0.5343

0.8090 0.5329

0.8099 0.5317

ML-20M

AUC Log loss

0.8070 0.5329

0.8079 0.5333

0.8088 0.5317

0.8082 0.5322

0.8103 0.5302

0.8135 0.5292

Books

AUC Log loss

0.7521 0.4345

0.7527 0.4357

0.7541 0.4305

0.7566 0.4242

0.7556 0.4244

0.7571 0.4221

FLOPs (M) Params (M)

1.90 0.01

24.10 0.11

21.13 0.03

29.77 0.02

23.32 0.17

23.41 0.18

0.8

ReST LLaMA LLaMA w/o Aux. Seq-CVR HSTU

1.2

0.85

ReST LLaMA HSTU

l32

1.0 0.9 0.8

0.80

l16

0.7

AUC Gain (%)

1.1

AUC Gain(%)

Metric

l4

0.6

l2 l16

l8 l4

l32

l16

0.5

l1

0.4

0.70

d256

d128

0.65 0.60 0.55

l4

0.6

d512

d512

d256

l8

l2

0.7

ReST LLaMA HSTU

0.75

l8

AUC Gain (%)

1.3

Dataset

0.50

l2

d128

d256

d512

d128

0.5 100

250

500

1k

2k

Sequence length

(a) Sequence Scale

4k

8k

16k

2 ¡2

2 ¡1

20

21

FLOPs (G)

(b) Depth Scale

22

23

21

22

23

24

FLOPs (G)

(c) Width Scale

Figure 4: Scaling behavior along three axes: (a) sequence length, (b) model depth, and (c) hidden dimension. ReST consistently outperforms competing sequence models across the explored budgets. where 𝐹 denotes FLOPs, and obtain (𝑎, 𝑏) = (0.626, 0.101) for ReST, (0.491, 0.090) for LLaMA, and (0.461, 0.088) for HSTU, with logspace 𝑅 2 = 0.991, 0.935, and 0.924, respectively. ReST attains consistently higher AUC gains across the tested compute regime, suggesting that rec-native designs yield a more favorable empirical compute-performance trend in our setting. This advantage becomes clearer in the axis-wise scaling analysis below: when scaling context length, depth, or width independently, LLM-style sequence models show earlier diminishing returns, whereas ReST continues to convert additional sequence-side computation into accuracy gains. Sequence length scale. Figure 4(a) varies the available behavior context from 100 to 16k tokens while keeping depth and hidden dimension fixed. We evaluate each context-length setting on recent data with sufficient history coverage and report each model’s

AUC lift over the production baseline evaluated on the same samples. Without auxiliary sequence CVR supervision, LLaMA exhibits the sequence-starvation effect discussed in Sec. 2.5: extending the available context yields almost no additional lift. With auxiliary sequence CVR supervision, all sequence models show stronger gains at larger context budgets, with the ordering ReST > LLaMA > HSTU. Across the evaluated settings, ReST continues to improve up to 16k tokens, achieving an additional AUC lift of approximately 0.4% over the evaluated context range. This represents the largest observed increase among the compared models and suggests that rec-native designs enable more effective use of longer user histories. Model depth scale. Figure 4(b) varies the number of layers while keeping sequence length and hidden dimension fixed. For ReST, 𝑇 and 𝐶 depths are scaled 1:1. As depth increases, ReST improves steadily, with AUC gain rising from +0.59% to +0.76%. In contrast,

Scaling Sequence Transformers for Recommendation with Rec-Native Designs

Table 3: Ablation: architecture and rec-native block designs.

Table 4: Ablation: auxiliary objectives.

Group

Variant

AUC Δ

Param Δ

FLOPs Δ

Backbone

Lseq (Base)

Lseq (Large)

Lalign

Architecture

T + MoE in 𝑇 + C (w/o TSP) + C (w/o PF-KV) +C

base +0.00% +0.03% +0.05% +0.05%

— +500% +100.3% +473.2% +473.2%

— +0.2% +0.2% +6.7% +0.2%

DIN STCA LLaMA HSTU ReST

+0.02% +0.13% +0.16% +0.10% +0.07%

+0.02% +0.20% +0.22% +0.15% +0.12%

— — +0.05% +0.05% +0.06%

Gated Attn

w/o gate + O gate + O & QKV gate + O & V gate

base +0.03% +0.06% +0.06%

— +7.6% +11.4% +8.9%

— +3.7% +14.9% +7.5%

Norm

Pre-LN Post-LN Mix-LN SRN

base +0.00% +0.02% +0.08%

— ≈0 ≈0 ≈0

— ≈0 ≈0 ≈0

Pos./Temporal

w/o pos RoPE RoPE + RoTE (sec) RoPE + RoTE

base +0.04% +0.08% +0.12%

— ≈0 ≈0 ≈0

— ≈0 ≈0 ≈0

LLaMA and HSTU improve at shallow depths but exhibit diminishing returns beyond 8 and 16 layers, respectively. This pattern suggests that simply stacking deeper generic sequence blocks is insufficient for recommendation ranking: under sparse supervision and noisy behavior data, additional depth may saturate before being effectively optimized. ReST benefits more consistently from depth, and the component ablation in Sec. 3.4 further confirms that SRN is especially important for deeper models. Hidden dimension scale. Figure 4(c) scales the hidden dimension 𝑑 ∈ {128, 256, 512} with depth and sequence length fixed. Both baselines saturate early: HSTU and LLaMA flatten around 𝑑=256, and increasing to 𝑑=512 yields almost no additional AUC despite a substantial increase in FLOPs. ReST follows a different trend, continuing to improve from +0.67% to +0.80% and remaining the only tested model with monotonic gains at the largest width. The combination of gated attention, temporal encoding, and stabilized residual learning allows ReST to benefit from larger hidden dimensions rather than merely increasing arithmetic cost.

3.4

Ablation Studies (RQ3)

Architecture and Compute Allocation. We study where additional capacity should be allocated in ReST. As shown in the Architecture group of Table 3, simply adding MoE-based capacity [30] to the sequence encoder 𝑇 increases parameters by 500% but yields no measurable AUC gain, suggesting that encoder capacity alone is not the limiting factor. In contrast, adding a lightweight cross decoder 𝐶 improves AUC by +0.03% with only +0.2% extra FLOPs, showing the value of candidate-conditioned readout. Tokenspecific parameterization further raises the gain to +0.05%, while projection-free KV keeps the FLOPs overhead at +0.2% rather than +6.7%. Thus, ReST benefits from allocating capacity to the heterogeneous candidate-conditioned decoder while keeping repeated per-candidate arithmetic nearly unchanged.

Gated Attention. The Gated Attention group of Table 3 shows that an output gate alone already yields a +0.03% gain. Adding gates to all Q/K/V branches increases AUC to +0.06%, but also incurs a large FLOPs overhead (+14.9%). In contrast, combining the output gate with a value gate reaches the same +0.06% gain at roughly half the extra compute (+7.5% FLOPs). Appendix A.1 further shows that combining the output and value gates yields better performance under noisy behavior perturbations. We therefore adopt Output + Value gating as our Dual-Gated Attention design for a better accuracy–efficiency trade-off. Normalization and Residual Connections. We then evaluate how normalization placement and residual scaling affect optimization in a fixed 16-layer setting. As shown in the Norm group of Table 3, the LLaMA-style Pre-LN design serves as the local baseline; switching to Post-LN brings no gain, while Mix-LN provides a modest +0.02% improvement, suggesting that normalization placement matters for deeper recommendation models. Adding scaled residual learning on top of this design, SRN achieves the largest gain of +0.08%, substantially outperforming the other variants at the same depth. This indicates that training deeper sequence models on sparse recommendation data benefits from both appropriate normalization placement and controlled residual updates. Positional and Temporal Encoding. The Positional and Temporal Encoding group in Table 3 starts from a no-position baseline. Adding RoPE improves AUC by +0.04%, confirming the benefit of ordinal positional information. Adding single-granularity RoTE with one-second buckets on top of RoPE raises the gain to +0.08%, showing that physical time provides complementary signal beyond ordinal position. The default multi-granularity RoTE, which assigns distinct temporal granularities to different temporal attention heads, further raises the gain to +0.12%, demonstrating the benefit of modeling temporal proximity at multiple resolutions. Importantly, all variants already receive the same delta-time features as token-level side information, so these gains reflect the additional value of injecting physical time directly into the attention mechanism rather than merely exposing the model to temporal features. Further details of RoTE are provided in Appendix A.2. Auxiliary Objectives. Table 4 summarizes the effects of the two training-only auxiliary objectives. Auxiliary sequence CVR supervision improves all sequence-aware backbones, with larger gains in the Large setting: STCA improves from +0.13% to +0.20%, LLaMA from +0.16% to +0.22%, HSTU from +0.10% to +0.15%, and ReST from +0.07% to +0.12%. This supports our motivation that larger sequence modules require stronger direct sequence-side supervision to avoid being shortcut by the dense non-sequential branch. The smaller marginal gain on ReST suggests that its rec-native

Chen et al.

encoder is less dependent on the auxiliary objective, although auxiliary supervision remains beneficial. On top of auxiliary sequence CVR supervision, the alignment objective further improves LLaMA, HSTU, and ReST by +0.05%, +0.05%, and +0.06%, respectively, indicating that sequence/non-sequence alignment provides a stable additional regularization signal across scalable sequence models. Since these objectives are architecture-agnostic and introduce no serving-time overhead, we enable auxiliary sequence CVR supervision for all sequence-scaling baselines in the main comparison.

3.5

System Efficiency and Online A/B Test Result (RQ4 & RQ5)

To translate the reuse boundary of ReST into actual system efficiency, we implement the training and serving stack around the shared sequence encoder. On the training side, our baseline is already a highly optimized pipeline with fused attention/GEMM kernels, mixed-precision training, and padding removal for variablelength sequences. On top of this strong baseline, user-level sharedprefix training (ULT) further improves end-to-end training throughput by 5.8× by removing redundant prefix computation across samples from the same user, showing that the gain comes from computation reuse rather than from low-level kernel optimization alone. On the serving side, request-level shared-prefix serving is the key enabler for online sequence scaling: 𝑇 is executed once per request and reused across all candidates, yielding up to a 20× reduction in the per-request inference cost of the sequence-modeling component. This reuse is critical for deploying ReST online under strict production latency budgets and substantially narrows the deployment gap between large sequence models and industrial ranking systems. Table 5: Online A/B results. AUC Δ

Advv Lift

𝑝-value

+1.31%

+11.93%

< 0.01

4

5 To validate the practical impact of ReST, we conduct a one-week online A/B test on TikTok Shop Ads, one of the largest advertising platforms. Users are randomly assigned to control and treatment groups, with the treatment group receiving 20% of production traffic. The control is a strong production DLRM ranker, while the treatment replaces its sequence modeling component with ReST. We scale ReST to a deployable configuration that satisfies the 50 ms P99 latency budget. We use a revenue-related business metric, Advertiser Value (Advv), as the primary online metric, report streaming AUC for online prediction quality, and monitor end-to-end P99 latency and other production guardrails. As shown in Table 5, this latency-constrained configuration improves AUC by +1.31% and lifts Advv by a statistically significant +11.93% (𝑝 < 0.01), while satisfying all other monitored production guardrails. ReST has been fully deployed in production, showing that scaling behaviorsequence modeling through rec-native designs can translate offline gains into substantial business value in a realistic large-scale production setting.

RELATED WORK

Deep Learning Recommendation Models (DLRMs). Modern industrial ranking systems are largely built on the DLRM paradigm [5, 22], which combines large embedding tables with relatively lightweight interaction modules. A long line of work has refined this paradigm from complementary angles. DCN-V2 [31] strengthens explicit feature crossing; DIN [41] introduces target-aware attention for user-behavior modeling; and SIM/SDIM [3, 23] extend behavior modeling to much longer histories through retrieval-based approximation. Together, these methods establish behavior sequences as highly informative signals for ranking, but typically keep sequence modeling compact or heavily compressed within a larger ranking stack rather than treating it as a primary scaling axis. Large Recommendation Models (LRMs). Recent work on large recommendation models can be viewed through the lens of what is being scaled. One line scales the joint interaction between behavior sequences and dense non-sequential features: RankMixer [42] scales dense feature interaction through hardware-aware token mixing; OneTrans [39] and InterFormer [36] integrate behavior sequences with non-sequential features; and FAT [35] and MTmixAtt [24] scale heterogeneous feature interaction through field-aware or mixture-of-experts parameterization. These methods strengthen the joint feature-interaction stack rather than isolating the behavior sequence as the main scaling object, making them complementary to our focus. A second line scales sequence modeling more directly. Building on sequence Transformers such as SASRec [16], STCA [10] strengthens target-to-history cross attention, LLaMAstyle decoder-only baselines [11, 29] enlarge sequence-model capacity with generic self-attention blocks, and HSTU [37] demonstrates strong scaling in a generative recommendation formulation. Yet these recipes do not fully address behavior-sequence scaling in discriminative ranking, where noisy and sparsely supervised histories must be scored against many candidates under strict latency constraints. ReST addresses these limitations by making the behavior sequence a reusable primary scaling axis through rec-native designs.

CONCLUSION

In this paper, we introduced ReST, a recommendation-native framework for scaling behavior-sequence Transformers in industrial ranking under strict latency constraints. By addressing the signal quality and computation asymmetry challenges through rec-native sequence modeling, asymmetric architecture design, and sharedprefix computation reuse, ReST mitigates sequence starvation and scales more effectively along behavior sequence length, depth, and width than LLM-style blocks. This enables larger sequence models to achieve substantial business gains under practical latency budgets, leading to full production deployment. Our results suggest that, with rec-native designs, behavior-sequence scaling remains an under-exploited axis for industrial ranking and can translate into substantial online business value.

ACKNOWLEDGMENTS We thank Chenzhi Zhou for his valuable PMO support, especially in facilitating cross-functional collaboration and production deployment.

Scaling Sequence Transformers for Recommendation with Rec-Native Designs

REFERENCES [1] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. 2021. ReZero is all you need: fast convergence at large depth. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence (UAI) (Proceedings of Machine Learning Research, Vol. 161), Cassio de Campos and Marloes H. Maathuis (Eds.). PMLR, 1352–1361. [2] Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael M. Bronstein, Petar Veličković, and Razvan Pascanu. 2025. Why Do LLMs Attend to the First Token?. In Conference on Language Modeling (COLM). [3] Yue Cao, Xiaojiang Zhou, Jiaqi Feng, Peihao Huang, Yao Xiao, Dayao Chen, and Sheng Chen. 2022. Sampling Is All You Need on Modeling Long-Term User Behaviors for CTR Prediction. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2974–2983. [4] Olivier Chapelle. 2014. Modeling Delayed Feedback in Display Advertising. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1097–1105. [5] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7–10. [6] Junsu Cho, Dongmin Hyun, Seongku Kang, and Hwanjo Yu. 2021. Learning Heterogeneous Temporal Patterns of User Preference for Timely Recommendation. In Proceedings of the Web Conference 2021. 1274–1283. [7] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems. 191–198. [8] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. Curran Associates, Inc., 16344–16359. [9] Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. 2021. Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139). PMLR, 2793–2803. [10] Lin Guan, Jia-Qi Yang, Zhishan Zhao, Beichuan Zhang, Bo Sun, Xuanyuan Luo, Jinan Ni, Xiaowen Li, Yuhang Qi, Zhifang Fan, et al. 2026. Make it long, keep it fast: End-to-end 10k-sequence modeling at billion scale on Douyin. In Proceedings of the ACM Web Conference 2026. 7989–7998. [11] Wei Guo, Hao Wang, Luankang Zhang, Jin Yao Chin, Zhongzhou Liu, Kai Cheng, Qiushi Pan, Yi Quan Lee, Wanqi Xue, Tingjia Shen, et al. 2024. Scaling new frontiers: Insights into large recommendation models. arXiv preprint arXiv:2412.00714 (2024). [12] F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems 5, 4 (2015), 1–19. [13] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. Curran Associates, Inc., 30016–30030. [14] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V. Le. 2022. Transformer Quality in Linear Time. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162). PMLR, 9099–9117. [15] Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased Learning-to-Rank with Biased Feedback. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. 781–789. [16] Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM). IEEE, 197–206. [17] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08361 (2020). [18] Sofia Ira Ktena, Alykhan Tejani, Lucas Theis, Pranay Kumar Myana, Deepak Dilipkumar, Ferenc Huszár, Steven Yoo, and Wenzhe Shi. 2019. Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR Prediction. In Proceedings of the 13th ACM Conference on Recommender Systems. 187–195. [19] Jiacheng Li, Yujie Wang, and Julian McAuley. 2020. Time interval aware selfattention for sequential recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining. 322–330. [20] Pengxiang Li, Lu Yin, and Shiwei Liu. 2025. Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN. In The Thirteenth International Conference on Learning Representations (ICLR). [21] Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in

information retrieval. 43–52. [22] Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, CaroleJean Wu, Alisson G Azzolini, et al. 2019. Deep learning recommendation model for personalization and recommendation systems. arXiv preprint arXiv:1906.00091 (2019). [23] Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. 2020. Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2685–2692. [24] Xianyang Qi, Yuan Tian, Zhaoyu Hu, Zhirui Kuai, Chang Liu, Hongxiang Lin, and Lei Wang. 2025. MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation. arXiv preprint arXiv:2510.15286 (2025). [25] Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. In Advances in Neural Information Processing Systems, Vol. 38. Curran Associates, Inc., 100092–100118. [26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139). PMLR, 8748–8763. [27] Noam Shazeer. 2020. GLU Variants Improve Transformer. arXiv preprint arXiv:2002.05202 (2020). [28] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2024. RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing 568 (2024), 127063. [29] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). [30] Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. 2024. Auxiliary-loss-free load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408.15664 (2024). [31] Ruoxi Wang, Rakesh Shivanna, Derek Z. Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. 2021. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-Scale Learning to Rank Systems. In Proceedings of the Web Conference 2021. 1785–1797. [32] Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. 2021. Denoising Implicit Feedback for Recommendation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 373–381. [33] Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2526–2547. [34] Fuzhao Xue, Jianghai Chen, Aixin Sun, Xiaozhe Ren, Zangwei Zheng, Xiaoxin He, Yongming Chen, Xin Jiang, and Yang You. 2023. A Study on Transformer Configuration and Training Objective. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, 38913–38925. [35] Bencheng Yan, Yuejie Lei, Zhiyuan Zeng, Zheye Deng, Di Wang, Kaiyi Lin, Pengjie Wang, Chuan Yu, Jian Xu, and Bo Zheng. 2026. From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM, 8358–8368. [36] Zhichen Zeng, Xiaolong Liu, Mengyue Hang, Xiaoyi Liu, Qinghai Zhou, Chaofei Yang, Yiqun Liu, Yichen Ruan, Laming Chen, Yuxin Chen, Yujia Hao, Jiaqi Xu, Jade Nie, Xi Liu, Buyun Zhang, Wei Wen, Siyang Yuan, Hang Yin, Xin Zhang, Kai Wang, Wen-Yen Chen, Yiping Han, Huayu Li, Chunzhi Yang, Bo Long, Philip S. Yu, Hanghang Tong, and Jiyan Yang. 2025. InterFormer: Effective Heterogeneous Interaction Learning for Click-Through Rate Prediction. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM). ACM, 6225–6233. [37] Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Jiayuan He, Yinghai Lu, and Yu Shi. 2024. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 58484–58509. [38] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-Training. In Proceedings of the IEEE/CVF

Chen et al.

A APPENDIX A.1 Denoising Analysis of Dual-Gated Attention

Relative-Time Property of RoTE

RoTE follows the same principle as RoPE: query and key vectors are rotated according to their timestamp buckets, but their dot product cancels the shared coordinate frame and depends only on the relative rotation. We provide a compact derivation for a temporal head. As defined in the main text, timestamps are shifted by the sequence start time before bucketization: 𝑡˜𝑖 = 𝑡𝑖 − 𝑡 1 and 𝑏𝑖 = ⌊𝑡˜𝑖 /𝜏ℎ ⌋. This keeps bucket indices within the time span of each sequence rather than the range of absolute wall-clock timestamps. Let 𝑏𝑛 and 𝑏𝑚 denote the resulting buckets under head-specific granularity 𝜏ℎ . The RoTE attention score can be written as ScoreRoTE (q, k) = (R Θ (𝑏𝑛 )q) ⊤ (R Θ (𝑏𝑚 )k) = q⊤ R Θ (𝑏𝑛 ) ⊤ R Θ (𝑏𝑚 )k = q⊤ R Θ (−𝑏𝑛 )R Θ (𝑏𝑚 )k = q⊤ R Θ (𝑏𝑚 − 𝑏𝑛 )k.

0.8095 0.8090 0.8085 0.8080 Dual-Gated Attn Gated Attn (O gate) Standard Attn

0.8075 0.8070 0%

20%

40%

60%

Noise level

Figure 5: Dual-Gated Attention is more robust as token noise increases, supporting its value-side filtering and output-side modulation design.

The derivation uses two basic properties of rotation matrices:

To further validate the denoising role of Dual-Gated Attention (DGA), we conduct a controlled noise-injection study. Specifically, we inject Gaussian noise into behavior tokens at different noise levels and compare standard attention, output-gated attention, and DGA under the same training and evaluation protocol. This setup isolates whether the attention block can remain robust when a growing fraction of behavior evidence becomes unreliable. Figure 5 shows that DGA is consistently more robust as token noise increases. Standard attention degrades most rapidly because noisy behavior features directly enter the value stream and are aggregated into the contextual representation. Output-gated attention is more stable, but still cannot prevent corrupted token features from being mixed during aggregation. By contrast, DGA gates both the value stream before aggregation and the output stream after aggregation, allowing the model to suppress unreliable evidence before it contaminates the sequence context. This result supports our design choice that value and output gates are complementary for noisy recommendation sequences.

A.2

0.8100

AUC

international conference on computer vision. 11975–11986. [39] Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei Liu, and Aixin Sun. 2026. OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender. In Proceedings of the ACM Web Conference 2026. ACM, 8162–8170. [40] Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et al. 2021. RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 4653–4664. [41] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1059–1068. [42] Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xintian Han, Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, Huizhi Yang, Zheng Chai, Zhe Chen, Yuchao Zheng, Qiwei Chen, Feng Zhang, Xun Zhou, Peng Xu, Xiao Yang, Di Wu, and Zuotao Liu. 2025. RankMixer: Scaling Up Ranking Models in Industrial Recommenders. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management. ACM, 6309–6316.

(11)

R Θ (𝑥) ⊤ = R Θ (−𝑥),

R Θ (𝑥)R Θ (𝑦) = R Θ (𝑥 + 𝑦).

Therefore, the temporal head does not depend on the shifted timestamp buckets 𝑏𝑛 and 𝑏𝑚 separately, but only on their relative bucket difference 𝑏𝑚 −𝑏𝑛 . Because subtracting the sequence start time cancels in pairwise differences, it does not change the elapsed time represented between two tokens. This derivation establishes relative-time dependence, not a monotonic or one-to-one mapping from elapsed time to attention score: rotary features are periodic, and the resulting score also depends on the query and key contents. Events within the same bucket are intentionally indistinguishable at that granularity. Assigning different 𝜏ℎ values to different heads exposes relative time intervals at complementary resolutions while retaining compatibility with the rotary computation used by efficient attention kernels. In our industrial implementation, we reserve one attention head for RoPE and assign the remaining heads to head-specific temporal granularities 𝜏ℎ ∈ {second, minute, hour, day, week, quarter, year}, enabling different temporal heads to capture recency at complementary resolutions, from fine-grained second/minute patterns to long-horizon quarter/year trends.

A.3

Gradient Analysis for Auxiliary Sequence CVR Supervision

To further diagnose the effect of auxiliary sequence CVR supervision, we monitor the gradient norm received by the Transformer sequence encoder during training. As discussed in Section 2.5, the main CVR objective is optimized after sequence representations are fused with strong non-sequential DLRM features, so the prediction head can partially reduce the loss through the non-sequential branch. This shortcut can leave the sequence encoder with weak gradients, especially when the sequence module is scaled up. Figure 2 reports the average gradient norm of projection matrices across all attention layers in the sequence encoder. Specifically, at each training checkpoint, we compute the gradient norms of the 𝑄, 𝐾, and 𝑉 projection matrices in every sequence-encoder attention layer and report their average across layers. Even with

Scaling Sequence Transformers for Recommendation with Rec-Native Designs

a small auxiliary weight 𝜆seq = 0.1, auxiliary sequence CVR supervision increases the average gradient norm by more than 3× (from 2.35 to 8.50), indicating that direct sequence-side supervision substantially strengthens the optimization signal received by the sequence branch. We observe a similar trend for the FFN parameters, suggesting that the auxiliary objective improves gradient flow throughout the sequence encoder rather than only in the attention projections.

A.4

Implementation Details

Public Benchmark Settings. All public benchmarks are used only for offline academic benchmarking; models are implemented with RecBole [40] and trained using Adam. We tune the learning rate and weight decay over {10−4, 5×10−4, 10−3 }, use a batch size of 4,096, and train for up to 20 epochs with early stopping on validation AUC. We apply 5-core filtering to each dataset, retaining only users and items with at least five interactions. Table 6 summarizes the resulting dataset statistics.

Table 6: Public dataset overview. Statistic

ML-1M

ML-20M

Books

Users Items Interactions

6K 4K 1M

138K 18K 20M

604K 368K 9M

Evaluation of Public Benchmarks. Ratings of 4 or above are treated as positive samples and lower ratings as negative samples. No additional negative sampling is performed; the observed lowrating interactions serve as negative instances. For each user, interactions are sorted chronologically and split into train/validation/test sets with an 80%/10%/10% ratio. We report AUC and binary log loss under this labeled binary prediction setting. Model Configuration of Public Benchmarks. The maximum behavior sequence length is 200 and the item embedding dimension is 32. Transformer-based models use two sequence-modeling layers and, for encoder–decoder architectures, two additional decoder layers, with two attention heads and a feed-forward inner dimension of 128. The prediction head is a three-layer MLP with hidden sizes [256, 128, 64]. We apply a dropout rate of 0.25 to the embedding layer, hidden layers, and attention weights.

Related documents

• RouteRec: Behavior-Guided Sparse Routing for Sequential Recommendation#1143974
• MuSeR: Scalable Long-sequence Recommendation with Multi-interest Modeling#1049414
• DP-Rec: Towards Dynamic Patching for Efficient Long-Sequence Recommendation#1116546
• Dear Algo: A Precision-First Agentic Intent Layer for Unified Search and Recommendation#617007
• Dear Algo: A Precision-First Agentic Intent Layer for Unified Search and Recommendation#616507