ZeroGR: Filtering-First Constrained Decoding for Generative Recommendation

arXiv:2611.04715v1 [cs.DB] 3 Nov 2026

ZeroGR: Filtering-First Constrained Decoding for Generative Recommendation Xiangrui Yang

Jiawei Yang

Liang Zeng

The University of Hong Kong Hong Kong, China xiangruiyang@connect.hku.hk

Peking University Beijing, China yangjiawei26@stu.pku.edu.cn

Kuaishou Technology Beijing, China zengliang03@kuaishou.com

Daqian Ding

Ziqian Liu

Bing Tian

The University of Hong Kong Hong Kong, China daqianding@connect.hku.hk

The University of Hong Kong Hong Kong, China liuziqian@connect.hku.hk

Huazhong University of Science and Technology Wuhan, China tbing@hust.edu.cn

Cheng Peng

Yunfeng Zhao

Fangyu Zhang

Kuaishou Technology Beijing, China pengcheng09@kuaishou.com

Kuaishou Technology Beijing, China zhaoyunfeng@kuaishou.com

Kuaishou Technology Beijing, China zhangfangyu@kuaishou.com

Jiaqiang Liu

Xiao Liang

Yiming Qiu

Kuaishou Technology Beijing, China liujiaqiang@kuaishou.com

Kuaishou Technology Beijing, China liangxiao03@kuaishou.com

The University of Hong Kong Hong Kong, China yimingq@hku.hk

ABSTRACT Generative recommendation is gaining traction in production, retrieving items by generating their identifiers one token at a time. The generated sequences must identify relevant catalog items, so constrained decoding couples relevance scoring with validity filtering. Because most possible sequences are invalid, scoring before filtering wastes computation on candidates that will be discarded. Our evaluation shows that filtering first can avoid this work with negligible impact on recommendation quality. We present ZeroGR, a filtering-first constrained decoding system for generative recommendation. ZeroGR uses a step-sliced filtering index to reduce GPU memory use by tailoring data representations to each decoding step. Its execution engine adopts shared-slot execution to accelerate data exchange across stages and hybrid projection to adapt computation to workload patterns, thereby reducing execution overhead. During index refreshes, ZeroGR applies layout-guided streaming construction to speed up updates and limit serving interference. Extensive evaluation shows that ZeroGR accelerates the filtering and scoring pipeline by up to 9.9× over representative baselines while reducing GPU index memory use and accelerating index reconstruction.

1

INTRODUCTION

Recommendation systems (RecSys) underpin online services, selecting relevant items for users from enormous catalogs under interactive latency constraints. Inspired by the rapid advancement of LLMs, emerging generative recommendation (GR) approaches item discovery by autoregressively generating item identifiers from user context [10, 32]. These GR models promise better recommendation quality and less operational toil than conventional RecSys, and have gained traction in production deployments [6, 11, 24]. To

cover catalogs with millions of items, GR represents each item as a short sequence of semantic identifier (SID) tokens, with related items sharing sequence prefixes [27, 29]. Decoding these tokens one at a time progressively narrows the item space toward specific recommendations. Because each step depends on the preceding tokens, its cost adds directly to the user request latency [22]. To find multiple relevant items, GR uses beam search to explore competing partial SID sequences, scoring their continuations and retaining promising candidates at each decoding step [11, 27]. These scores guide relevance to the user, but cannot establish validity within the catalog: a seemingly promising SID sequence may identify no actual item. Each GR decoding step therefore also requires filtering, typically through a catalog trie that restricts continuations to those that can reach valid items [21, 30, 43]. This search for relevant recommendations subject to catalog validity is the constrained decoding problem in GR. Maintaining competing item candidates requires both scoring and filtering to process many continuations at every step, and their decisions determine which candidates advance. Both operations thus lie on the serving critical path, making their performance central to meeting GR’s latency objective. The order of scoring and filtering directly affects how much work GR decoding performs. Scoring first [30] scores all possible continuations before discarding invalid ones, whereas filtering first [12] restricts scoring to valid continuations permitted by the catalog. At first glance, filtering first is attractive: valid candidate sparsity allows it to eliminate substantial scoring work while preserving the same validity guarantee. Its effect on relevance, however, is less straightforward. Scoring first normalizes scores over all continuations of each sequence prefix, whereas filtering first normalizes only over valid continuations. Depending on the total probability assigned to invalid continuations under each prefix, filtering first

Beam search (Bℓ = 1, 1, 2)

can give candidates from some prefixes a larger boost than others, changing their relative ranking and ultimately which candidates survive beam search. The two orders can therefore return different recommendations even though both enforce the same catalog constraints. The key question therefore becomes whether filtering first can deliver recommendation quality on par with scoring first. In this paper, we answer this question in the affirmative. Our evaluation shows virtually identical recall for the two orders, with approximately 99.9% overlap in the returned items. This stability is consistent with how GR progressively narrows the search: early steps assign most probability mass to valid continuations, so filtering has little effect on normalization; later steps mainly refine choices established earlier, so changes in normalization tend to have limited impact on recommendation quality. We therefore aim to build a filtering-first constrained decoding pipeline for GR serving. The remaining question is how to realize the performance benefits of filtering first given the following coupled challenges:

step 1

User history

step 2

step 3

5

4

2

1

1

7

6

9

3 1

GR model

8

kept (top-Bℓ) pruned

look up items

kept beams fed back

Item catalog: one SID sequence per item Item SID sequence

8

1

2

one SID

3

2

7

3

5

1

3

5

4

similar items share a SID prefix

Figure 1: Generative recommendation with SID sequences. We evaluate ZeroGR on a variety of recommendation datasets. Compared with representative constrained decoding baselines, ZeroGR reduces the combined latency of filtering, scoring, and candidate selection by up to 9.9×. It also reduces GPU index memory use by up to 74.5% and reconstructs its index 278× faster, completing the rebuild within the service’s update window. In summary, this work makes the following contributions:

• Representing filtering index within limited GPU memory. Filtering requires efficient access to valid continuations; keeping this lookup on the GPU reduces CPU–GPU coordination. The GPU filtering index must cover the catalog and accommodate varying valid candidate densities across decoding steps, while sharing memory with model weights and inference state [18]. The challenge is to keep the index compact across diverse step profiles. • Translating filtering-first computation into efficient GPU execution. Computing fewer scores does not by itself ensure efficient GPU execution [36]. Filtering first produces selective workloads that vary across prefixes and decoding steps, and the overhead of organizing and processing them can outweigh the scoring savings. The challenge is to adapt GPU execution to these workload patterns so that reduced scoring translates into lower latency. • Building filtering index without disrupting GPU serving. Because the catalog and SID assignments are refreshed with the model, each GR update requires rebuilding the filtering index. GPU construction competes with ongoing inference for compute, while transferring a CPU-built index consumes bandwidth also needed by serving. The challenge is to complete the rebuild within the update window while protecting the latency of ongoing requests.

• We analyze valid candidate sparsity in GR decoding and empirically demonstrate that filtering first can reduce scoring work while retaining comparable recommendation quality. • We design ZeroGR, integrating a compact filtering index, coordinated GPU execution, and parallel streaming construction to realize the performance benefits of filtering first. • We evaluate ZeroGR on real-world datasets from industrial GR services, demonstrating significant improvement over representative constrained decoding baselines.

2 BACKGROUND AND MOTIVATION 2.1 Sparsity of Valid Sequences in Generative Recommendation The conventional paradigm in recommender systems decomposes item discovery into a multi-stage “retrieve-and-rank” pipeline [8]: first filtering a coarse candidate set from the catalog, and then applying increasingly complex models to rank them. In contrast, generative recommendation (GR) departs from this pipeline by using an autoregressive model to directly generate the identifiers of target items, fundamentally casting item retrieval as sequence generation [27]. A key design choice in GR is how to represent item identifiers. Early GR systems assign each item a dedicated single token [38], which suffers from vocabulary explosion and fails to capture semantic similarity across items. In contrast, modern SIDbased GR [24] represents each item 𝑖 with an 𝐿-token SID sequence s(𝑖) = (𝑠 1(𝑖 ) , . . . , 𝑠𝐿(𝑖 ) ), where each semantic identifier (SID) 𝑠 ℓ(𝑖 ) is a token drawn from a position-specific codebook Vℓ . These sequences are constructed offline such that items with similar content share common SID prefixes [19, 27, 29]. For example, two similar products (e.g., headphones and a headset) might share the coarse prefix (3, 5) and differ only in their final token (e.g., (3, 5, 1) and (3, 5, 4) in Figure 1, bottom). As a result, each SID prefix represents a category of related items rather than a single item, and only a complete 𝐿-token sequence uniquely identifies a specific item in the catalog.

We present ZeroGR, a filtering-first constrained decoding system for generative recommendation. To save GPU memory, ZeroGR uses a step-sliced filtering index: for each decoding step, it maintains a mapping from sequence prefixes to valid next tokens, encoded in either a sparse or a dense representation according to that step’s valid candidate density. Building on this index, the ZeroGR execution model adopts shared-slot execution to give each prefix fixed candidate slots shared by filtering, scoring, and candidate selection, allowing stages to exchange results directly and reducing coordination overhead. The execution model further applies hybrid projection: when the scoring reduction outweighs selective execution overhead, it computes scores only for valid continuations; otherwise it computes all scores to preserve regular batched execution. To speed up refreshes and limit serving interference, ZeroGR applies layout-guided streaming construction: it precomputes each fragment’s size and position in the final index, allowing CPU workers to generate fragments independently in their final form while streaming completed fragments to the GPU. 2

Step 2 ρ2 = 14.29%

Step 3 ρ3 = 3.51%

Step 4 ρ4 = 0.13%

Step 5 ρ5 = 0.05%

the entire codebook Vℓ and then masks out the tokens outside Nextℓ (𝑝𝑏 ). For brevity, we call them filtering first and scoring first. For performance reasons, filtering first is preferable because valid SID continuations are sparse. We quantify the fraction of useful full-projection logits at step ℓ using the valid candidate density: Í𝐵ℓ | Nextℓ (𝑝𝑏 )| 𝜌 ℓ = 𝑏=1 . (2) 𝐵 ℓ |Vℓ | The numerator counts the valid continuations summed over all 𝐵 ℓ active beams, whereas the denominator counts the 𝐵 ℓ |Vℓ | logits that full projection computes, one per beam and token. Hence, 𝜌 ℓ is the average fraction of the codebook that can still extend an active prefix, ranging from 1/|Vℓ | when every prefix admits a single continuation to 1 when every token remains valid. As measured on our industrial generative search dataset in Figure 2, valid entries, shown in colored cells, account for at most 𝜌 2 = 14.29% at step 2 and drop to 𝜌 5 = 0.05% by step 5. Thus, scoring first wastes 85.7% to 99.95% of its projection arithmetic, logit materialization, and masking overhead on invalid continuations, whereas filtering first allows the decoder to skip this redundant work. In practice, a well-trained model rarely generates invalid tokens, but it does not make scoring first any cheaper. The fraction of logits wasted on invalid tokens, 1 − 𝜌 ℓ , is fixed by the item catalog rather than by model accuracy, and it varies widely across decoding steps and across prefixes within a step.

Figure 2: Valid SID candidates per decoding step. At serving time, item retrieval turns into an autoregressive decoding workload over these codebooks. As shown at the top of Figure 1, the GR model first encodes the user’s interaction history and then, starting from a beginning-of-sequence token, generates one SID per autoregressive decoding step, conditioning each prediction on the encoded context and the previously generated prefix. To return multiple recommendations, decoding uses beam search [11, 27], which maintains 𝐵 ℓ active prefixes, or beams, at each decoding step ℓ, where the beam width 𝐵 ℓ can vary across steps. After 𝐿 steps, each completed SID sequence is looked up in the item catalog I and resolved to its item. Intuitively, the set of possible SID sequences is the Cartesian product of the per-step codebooks, V1 × · · · × V𝐿 , Î whose cardinality 𝐿ℓ=1 |Vℓ | is astronomical, far exceeding 1016 in production systems. However, the set of actual valid SID sequences, S = {s(𝑖) | 𝑖 ∈ I} ⊆ V1 × · · · × V𝐿 , is only around 107 –108 , meaning that the majority of possible sequences are invalid, i.e., they do not map to any items. Navigating this vast, sparsely populated sequence space to find high-quality, valid items is the core task of GR decoding.

2.2

2.3

In single-sequence generation such as standard LLM decoding [44], adopting filtering first is straightforward because both orders produce the exact same result: all candidate tokens compete within the same prefix, so excluding invalid tokens before or after scoring does not alter the relative ranking among valid tokens. In GR, however, filtering first and scoring first are not mathematically equivalent. Unlike single-sequence decoding, GR relies on beam search across 𝐵 ℓ > 1 active prefixes and ranks candidate extensions globally by adding each token’s current-step log-softmax score to its accumulated prefix score. Under scoring first, each beam’s softmax denominator sums over the full codebook Vℓ , penalizing prefixes that assign high probability mass to invalid tokens. Under filtering first, because invalid tokens are never scored, the softmax denominator sums only over the valid subset Nextℓ (𝑝𝑏 ) [26]. Since | Nextℓ (𝑝𝑏 )| varies across active prefixes 𝑝𝑏 , filtering first normalizes each beam over a token set of a different size, so the two orders can rank the same candidate extensions differently. Because scoring first keeps the full-codebook normalization of unconstrained decoding, it appears to be the more faithful order in GR, and existing constrained decoding systems for GR [27, 30] simply adopt it as the default despite its high computational cost. The two orders differ sharply in computation but negligibly in recommendation quality. Figure 3 compares the two orders with a GR model trained on the public MovieLens-1M dataset [16]. The items they return overlap by 99.90% on average, and their recall is virtually identical at every cutoff. This results from the stepwise sparsity pattern of SID decoding. Across decoding steps, 𝜌 ℓ is relatively high in early steps and becomes extremely small only in the final steps, which is shown in Figure 2. In the early steps, each active prefix still admits many valid continuations, and a

Scoring and Filtering in SID Decoding

To return high-quality recommendations that correspond to actual catalog items, each decoding step in GR requires both scoring for quality and filtering for validity. For scoring, at step ℓ, the model’s output projection computes logits over the full codebook Vℓ for each of the 𝐵 ℓ active beams, 𝐵 ℓ |Vℓ | logits in total. Beam search normalizes these logits into scores via softmax, adds the scores to the accumulated prefix scores, and retains the 𝐵 ℓ+1 highest-scoring extensions for the next step. For filtering, decoding must ensure that generated sequences stay within the valid set S, since relying on model scores alone can easily produce sequences that map to no item. Deferring filtering until all 𝐿 steps finish, i.e., post-generation filtering [37], wastes computation on invalid paths and requires heavy over-generation to fill the requested output size. Instead, constrained decoding enforces validity at every step ℓ by restricting each active prefix 𝑝 = (𝑠 1, . . . , 𝑠 ℓ −1 ) to its valid SID continuations: Nextℓ (𝑝) = {𝑣 ∈ Vℓ | ∃ s ∈ S, s1:ℓ −1 = 𝑝 ∧ 𝑠 ℓ = 𝑣 }.

Viability of Filtering First

(1)

For instance, for prefix 𝑝 = (3, 5) in Figure 1, only Next3 (𝑝) = {1, 4} can extend into valid items (3, 5, 1) and (3, 5, 4). Because Nextℓ (𝑝) varies across prefixes, systems build a filtering index over S to retrieve Nextℓ (𝑝) during decoding. At each decoding step, filtering and scoring can be combined in two execution orders: filtering-then-scoring removes the tokens outside Nextℓ (𝑝𝑏 ) before normalizing logits into scores, so each beam is normalized only over its valid tokens and the logits of the other tokens need not be computed at all, whereas scoringthen-filtering first computes and normalizes all 𝐵 ℓ |Vℓ | logits over 3

100

items, every extra field per entry takes memory that would otherwise go to serving larger batches. The challenge is to provide this per-step adaptability while keeping the index compact. Challenge 2: executing on the index without forfeiting GPU efficiency. An index that identifies which entries to compute does not by itself make them cheap to compute, because the full projection it replaces is a regular batched matrix product while an index-driven projection reads a different, variable-length set of codebook entries for every beam. Simply skipping the invalid entries can therefore leave latency unchanged or even increase it relative to full projection, despite performing much less arithmetic [14]. The same irregularity carries over to beam selection. The challenge is not only to reduce the computation but also to execute the remaining irregular work efficiently enough that the savings translate into lower latency. Challenge 3: reconstructing the index alongside online serving. Because the item catalog and its SID sequences are refreshed together with the model weights, every model update must rebuild the index within a limited time window before the new version can serve. The rebuild also runs while the instance continues to serve the current version. The rebuild must therefore neither contend with inference for GPU compute, which inference requires to meet its latency targets, nor saturate the host-to-GPU link that carries incoming requests. The challenge is to reconstruct the index within the update window while leaving both GPU compute and the host-to-GPU link to online serving.

99.85 99.85 99.90 99.90

50 0

Scoring First

Recall (%)

Overlap (%)

Filtering First

1

10

50

128

60 40 20 0

1

10

50

Top-K

Top-K

(a) Overlap

(b) Recall@K

128

Figure 3: Overlap and recall of filtering first and scoring first on MovieLens-1M. trained GR model already concentrates nearly all of its probability mass on valid tokens; excluding invalid tokens from the softmax denominator therefore leaves the normalized step scores almost unchanged. In the later steps, a candidate’s total score comes mostly from the earlier steps, so the order of filtering and scoring at these steps rarely changes which candidates are kept. As a result, GR recommendation quality is not sensitive to the ordering between filtering and scoring operations.

2.4

Insights and Challenges

The analysis above shows that the catalog fixes the computation space before any score is computed. This fact leads to the following insights about how SID constraints can be used during decoding. Insight 1: SID constraints specify computation rather than outputs. Because Nextℓ (𝑝) is fixed by the item catalog at the offline construction phase, it can specify which computation the decoder performs. Instead of masking scores after a full-codebook projection, the decoder retrieves Nextℓ (𝑝𝑏 ) for each active beam first and computes logits only for these tokens. A constraint used this way reduces the logits the projection computes from 𝐵 ℓ |Vℓ | to 𝜌 ℓ 𝐵 ℓ |Vℓ | and removes constraint application as a post-processing stage. Insight 2: the constraint is a family of per-step mappings. Because Nextℓ (𝑝) is indexed by prefix, the naive representation is a prefix tree over S. Decoding, however, never queries the constraint as a whole: step ℓ queries only prefixes of length ℓ − 1, each carried by an active beam from step ℓ − 1. For each such prefix 𝑝, the step needs only Nextℓ (𝑝); appending the continuations that beam selection keeps yields the length-ℓ prefixes that step ℓ + 1 queries. The constraint is therefore a family of 𝐿 per-step mappings, and step ℓ retrieves Nextℓ (𝑝𝑏 ) for all 𝐵 ℓ active beams with one batched lookup into its own mapping. The two insights determine what each decoding step needs to compute. Realizing them within a GPU serving instance, however, raises three system-level challenges. Challenge 1: supporting per-step heterogeneity under a GPU memory budget. Decoding steps favor different computations: when most tokens remain valid, scoring every token with one regular matrix product is efficient; when only a few remain, we should score just the valid candidates of each beam. An index that serves every step should therefore provide candidates in the form each step’s computation needs, and leave the choice of computation open rather than fixing it in advance. At the same time, the index must stay small because it resides in GPU memory next to the model weights and the attention caches [7, 34, 39]. With more than 108

3

ZEROGR DESIGN

This section presents the design of ZeroGR, beginning with an overview of its parts and request processing during decoding.

3.1

Overview

ZeroGR filters the candidates of each beam through the index before scoring them, so that the item catalog, rather than the full codebook, determines which scores each decoding step computes. Its design has three parts: a filtering index that lists the valid next tokens of each prefix (§ 3.2), an execution engine that uses the index at every decoding step (§ 3.3), and a construction pipeline that builds this index (§ 3.4). Figure 4 gives an overview of ZeroGR. Each request expands into up to 𝐵 beams through beam search. Each beam 𝑏 carries an active prefix 𝑝𝑏 , an accumulated score 𝑎𝑏 , and a state 𝑞𝑏 , an integer that locates the valid next tokens of 𝑝𝑏 in the filtering index. Each decoding step processes all beams of the request in four stages. (i) The decoder produces the hidden vector ℎ ℓ,𝑏 of each beam. (ii) ZeroGR uses each beam’s state to read its valid next tokens from Mℓ , the part of the index for the current step ℓ. (iii) ZeroGR computes the logits of these valid tokens. It does so in one of two modes: selective projection computes only their logits, whereas full projection, used when most tokens are valid, computes the logits of all tokens in Vℓ and keeps only the valid ones. (iv) Each beam normalizes its logits over its valid tokens and adds its accumulated score, and the 𝐵 highest-scoring candidates of the request become the beams of the next step. After 𝐿 steps, each beam holds a complete SID sequence that identifies an item in the item catalog, and the request returns these 𝐵 items. 4

Online · GPU

§ 3.3 Execution Model (iii) Score valid tokens

Beams 𝐡𝑏

prefix + state

(ii) Filtering

(i) Decoder

Top-𝐵

·

Selective projection

·

Full projection

Next beams

(iv) Normalize

prefix + state Next step

§ 3.4 Streaming Construction

Offline · CPU SID library

§ 3.2 Step-sliced Filtering Index

Streaming construction Count & layout

Stream fill

A B Pinned A/B

GPU

H2D

Filtering index …

step 1

step 2

step 𝐿

Figure 4: Overview of ZeroGR: offline index construction and online filtering, scoring, and selection at each decoding step. Whereas the online path reduces per-step decoding latency, index construction keeps the filtering index consistent with the item catalog, which is refreshed together with the model weights. Each refresh must complete within a limited window while the instance continues to serve requests, so ZeroGR rebuilds the index on the CPU and keeps the rebuild from competing with inference for GPU compute or host-to-GPU bandwidth.

3.2

Query First SID?

Query After (3)?

Answer {2,5}

M₁ State Prefix 0

Index Representation

M₂

Valid SIDs {1,3}

∅

Query After (3,5)?

State Prefix 0 1

M3

Valid SIDs

(1) (3)

Answer {1,4}

{2,4} {2,5}

State Prefix

Valid SIDs

0 1

(1, 2) (1, 4)

{5,7} {3}

2 3

(3, 2) (3, 5)

{6} {1,4}

Figure 5: Step-sliced filtering index, where step ℓ holds the prefixes of length ℓ − 1 and numbers them independently.

ZeroGR introduces a step-sliced filtering index, which makes each decoding step independent in representation and lookup while keeping the index compact in GPU memory. The index comprises 𝐿 mappings, M1, . . . , M𝐿 , one per decoding step, each telling which tokens can follow a prefix at that step. Slicing by step makes the mappings independent of one another, so different steps can use different representations. For a prefix 𝑝 = (𝑠 1, . . . , 𝑠 ℓ −1 ), its state 𝑞 ℓ (𝑝) identifies a row within Mℓ . For ℓ < 𝐿, this row lists each valid next token of 𝑝 together with its successor state in Mℓ+1 : Mℓ (𝑞 ℓ (𝑝)) = {(𝑣, 𝑞 ℓ+1 (𝑝 ◦ 𝑣)) | 𝑣 ∈ Nextℓ (𝑝)},

Answer {1,3}

(b) Sparse M₂

(a) Dense M₁ SID

Successor state

1

0

2

–

3

1 3 entries

(3)

Boundaries SIDs 𝑅ℓ 𝐴ℓ

SID State 1 2 3 4 5 0

2

1

2

4 5

0 2 4

Successor state

2

0

4

1

2

2

5

3

10 cells → 7 entries

Figure 6: Dense step-1 (a) and sparse step-2 (b) mappings, where entry positions in (b) serve as successor states.

where 𝑝 ◦ 𝑣 appends token 𝑣 to 𝑝. State identifiers are local to a step: the selected extension supplies the row to query in the next mapping, without traversing the preceding prefix again. Figure 5 shows the index for six items with three-token SID sequences. Step 1 holds only the empty prefix, step 2 holds the prefixes (1) and (3), and step 3 holds the four two-token prefixes, with each step numbering its prefixes from zero. For the items that begin with (3, 5), token 3 leads from step 1 to state 1 of step 2, and token 5 then leads to state 3 of step 3. At step 3, this state alone gives the valid last tokens, 1 and 4, without revisiting tokens 3 and 5. Each step thus provides either token-aligned validity information for full projection or a compact list of valid tokens for selective projection. Specifically, a step with many valid continuations uses a dense mapping, which provides token-aligned validity information through a successor table whose rows are states and whose columns are the SIDs of the step’s codebook Vℓ . Each cell holds the successor state reached by appending the column’s SID to the row’s prefix, and a sentinel marks an invalid SID. A cell need not store its SID, because its column already identifies it. Since a row marks valid SIDs by position rather than listing them, a dense mapping serves

full projection, whose logits are laid out in the same token order as the table columns. A step with few valid continuations instead uses a sparse mapping, which keeps only the valid SIDs as a compact list. The list places the valid SIDs of all prefixes in one SID array 𝐴ℓ , one segment per state, with segments ordered by state. A boundary array 𝑅ℓ records where each segment begins and ends, so two adjacent boundaries delimit the valid SIDs of one state. Invalid SIDs are skipped, while an SID that is valid under several prefixes appears once in each of their segments. Each entry therefore stands for exactly one extended prefix, so a sparse mapping needs no successor table: the position of an entry serves as its successor state. Since a segment lists its valid SIDs explicitly, a sparse mapping serves both projection modes: selective projection computes logits only for the listed SIDs, whereas full projection computes the logits of all tokens in Vℓ and uses the listed SIDs to pick out the valid ones. For a sparse mapping, the execution engine chooses between the two modes without changing the representation. 5

(a) Dense · step 1

Figure 6 shows the dense and sparse mappings of the first two steps of Figure 5 and traces the items that begin with (3, 5). The dense step-1 mapping has one column for each of the tokens 1 to 3; the cell of token 2 holds the sentinel, and the cell of token 3 holds state 1, the step-2 state of prefix (3). In the sparse step-2 mapping, boundaries 2 and 4 delimit the segment of state 1, which lists the valid tokens 2 and 5. Token 5 lies at position 3, which is also the step-3 state of prefix (3, 5). The lookup thus crosses from a dense to a sparse mapping through a single state. Besides supplying candidates in the form each step’s projection needs, the representation is designed to keep the memory footprint of the index small [4, 40], because the index resides in GPU memory alongside the model weights and attention caches. Its size grows with the number of valid extensions and states it represents, which increase toward the last step: the extensions at step ℓ correspond one-to-one to the distinct prefixes of length ℓ, and at step 𝐿 these prefixes are the SID sequences of all items. The step-sliced index reduces this storage in two ways: sparse mappings store no successor states for their extensions, and completed SID sequences need no storage beyond their final SIDs. First, the position of each entry in a sparse mapping serves as its successor state. Each prefix at step ℓ + 1 arises from exactly one extension at step ℓ, so numbering these extensions by position gives every such prefix a distinct state, and Mℓ+1 numbers its rows the same way. A sparse mapping thus needs no successor array, which would be as long as its SID array. Second, every SID sequence has exactly 𝐿 tokens, so the step number alone marks an SID selected at step 𝐿 as the last of its sequence, and the index needs no field that records completion. M𝐿 therefore lists the valid final SIDs of each prefix without successor states. The index also stores no item identifiers, because a completed SID sequence already identifies its item, so the completed sequences, one per item and thus the largest set of prefixes, need no storage beyond their final SIDs.

3.3

( ), state 0

1 0 −0.9

token successor score

(b) Sparse · step 2

2 – −∞

(1), state 0

token successor score

2 0 −2.1

3 1 −0.5 (3), state 1

4 1 −1.3

2 2 −1.9

5 3 −0.8

Figure 7: Shared candidate slots in the first two steps of the index, with 𝐵 = 2. selective filtering

projection

or

beam 0 token successor score

−∞

normalize top-B

full

projection

filtering

beam 1

−∞

positions

gather

next beams

Figure 8: Shared candidate slots, filled by filtering and projection in either order and read by selection and gather. its beam: filtering and projection write the candidate into this slot, and selection and gather find it by the slot’s position alone. Specifically, at step ℓ, the slots of each beam form one row, and all rows of the step have the same length: the length of the step’s longest segment for a sparse mapping, or |Vℓ | for a dense mapping. Each slot holds the token, successor state, and score of one candidate. With a sparse mapping, the beam’s valid tokens fill the first slots of its row in the order of its segment; with a dense mapping, the slots are aligned with the columns of the successor table. A slot without a valid token holds the score −∞, which excludes it from normalization and selection. Since all rows have the same length, the position of a slot depends only on its beam and its offset within the row, not on how many candidates the other beams have. Figure 7 shows an example of the shared candidate slots for the first two steps of the index in Figure 5. Step 1 has a dense mapping, so its single row has a slot for each of the tokens 1 to 3, and the slot of token 2, whose successor-table cell holds the sentinel, has the score −∞. Step 2 has a sparse mapping, so each beam’s row holds the tokens of its segment in order: beam (1), in state 0, holds tokens 2 and 4, and beam (3), in state 1, holds tokens 2 and 5. The successor state of each slot is the position of its entry in the sparse mapping, so the four slots hold states 0, 1, 2, and 3, which identify (1, 2), (1, 4), (3, 2), and (3, 5) at step 3. The shared candidate slots are filled by filtering and projection, which can run in either order (Figure 8). With a sparse mapping, the order depends on the projection mode. In selective projection, filtering first expands each beam’s segment into its row, writing the tokens and successor states and setting every logit to −∞, and projection then computes only the logits of the valid slots and writes them in place. In full projection, the step first computes the logits of every token of Vℓ , and filtering then expands the segments in

Execution Model

With the step-sliced index resident in GPU memory, ZeroGR processes online requests by running each decoding step of Figure 4 on the GPU. At every step, ZeroGR filters the candidates of each beam through the index before normalizing their logits into scores, and this order also allows it to compute only the logits of the valid tokens. However, valid tokens differ in number from beam to beam and from step to step, so computing the logits of exactly these tokens is irregular work [17], which a GPU executes less efficiently than the regular matrix multiplication of full projection. Filtering, projection, top-𝑘 selection, and gather all handle these valid tokens, so the time saved by computing fewer logits also depends on how the stages pass them to one another. The rest of this subsection therefore describes shared-slot execution, which organizes these hand-offs around candidate slots that all stages of a step share, and hybrid projection, which chooses full or selective projection for each step. It describes these slots, how filtering and projection fill them, and how top-𝑘 selection and gather read them. 3.3.1 Shared Candidate Slots. To execute the stages of a decoding step efficiently on the GPU, ZeroGR gives every beam of the step the same number of slots, which all stages share. Each candidate, a beam extended by one of its valid tokens, occupies one slot of 6

Filtering

the same way but copies each slot’s logit from these token-ordered logits. With a dense mapping, the token-ordered logits form the slots, and filtering only sets the logits of invalid tokens to −∞. In every case, filtering completes before selection normalizes the logits into scores, so both projection modes filter first and differ only in whether they compute the logits of invalid tokens. Once filled, the slots are read in place by selection and gather. Selection first normalizes each row, which holds the candidates of one beam, by applying a log-softmax over its valid slots and adding the beam’s accumulated score. It then keeps the 𝐵 highestscoring candidates among the rows of each request and returns the positions of their slots. Gather turns each returned position into a beam of the next step. Since all rows have the same length, dividing the position by the row length gives the parent beam, and the remainder gives the slot within its row. Because no slot position depends on the candidates of other beams, no stage has to count the candidates of all beams and compute prefix sums over these counts before it places or finds a candidate, and the stages run back to back at every hand-off. The slot layout strikes a balance between giving every token of Vℓ a slot and packing only the valid candidates [3]. Sizing each row by the step’s longest segment keeps positions fixed with the least padding, and the next two subsections describe how projection and selection further limit the work spent on empty slots.

Beam Index 0

work queue slots (one thread per slot) t0

1

t1

t2

2

t0 t1 t2 one atomic t3 per beam

t3

Projection

next task logits in slots

0

thread block warp 0

1 2

≤ 64 slots per task

−∞ −∞

= warp 1 warp 2

shared memory

·

h0

loaded once

Figure 9: GPU execution of selective projection. its slot to −∞, which the projection kernel later overwrites for the valid slots. Alongside the slots, the filtering kernel builds a work queue whose tasks each cover a run of consecutive valid slots of one beam, up to a fixed length. The queue lets the projection kernel skip empty slots without moving any candidate out of its slot. The projection kernel consumes the queue with a fixed number of thread blocks, which loop over it until every task is processed. For each task, a block first loads the hidden vector of the corresponding beam into on-chip shared memory, and its warps (groups of 32 threads) then process the candidates of the task in parallel, one candidate per warp at a time. A warp computes the logit of its candidate token as the dot product of the shared hidden vector with the weight row of that token, splitting the hidden dimension across its threads and summing their partial products with a warp-level reduction. The warp then writes the logit directly into the slot that holds the candidate, where selection later reads it. Empty slots thus receive no work and keep the −∞ that the filtering kernel wrote, each hidden vector is read once per task rather than once per candidate, and a beam with many valid tokens yields several tasks that can run on different blocks. Since neither projection mode is faster at every step, ZeroGR realizes hybrid projection by selecting one of three combinations of mapping and projection mode for each step: a dense mapping with full projection, or a sparse mapping with either full or selective projection. At each model update, before the new version accepts traffic, ZeroGR runs warm-up requests under every combination available to a step and keeps the one with the lowest measured latency. The selected combinations then stay fixed until the next update, so the decoding loop makes no per-request decision.

3.3.2 Full and Selective Projection. Full and selective projection both aim to fill the shared candidate slots as quickly as possible, but they suit steps with different numbers of candidates. Full projection is the projection of an unconstrained decoder: it computes the logits 𝑧 ℓ,𝑏 = 𝑊ℓ ℎ ℓ,𝑏 of all tokens in Vℓ for all beams in one matrix multiplication [20]. Each row 𝑤 𝑣 of 𝑊ℓ corresponds to a token 𝑣, and 𝑤 𝑣⊤ℎ ℓ,𝑏 is the logit of 𝑣 for beam 𝑏. This matrix multiplication does not depend on the index, so full projection runs before filtering. Filtering then builds the slots from the index and fills each valid slot with the logit of its token, while all other slots hold −∞ and are thus excluded when selection normalizes the logits into scores. Full projection computes many logits that are then discarded, but its matrix multiplication is regular, which is GPU friendly [15]. Its cost depends on |Vℓ | and the number of beams, but not on how many tokens are valid. Selective projection computes only the logit 𝑤 𝑣⊤ℎ ℓ,𝑏 of each valid slot. Therefore, it runs after filtering has found the valid tokens. The arithmetic of selective projection is proportional to the number of candidates, but a direct implementation does not reduce latency in the same proportion. If it assigns GPU threads to every slot, most of them are idle at later steps, where most slots are empty, and every candidate reloads its beam’s hidden vector. To leverage the benefits of shared candidate slots, ZeroGR designs a dedicated filtering kernel and a selective projection kernel, as shown in Figure 9. The filtering kernel builds the slots from the index together with a work queue that lists only the valid slots, and the projection kernel then computes the logits of these slots from the queue. Specifically, the filtering kernel assigns one thread to each slot. The thread for the 𝑘-th slot of a beam copies the 𝑘-th entry of the corresponding segment, a valid token together with its successor state, into the slot, and leaves the slot empty if the segment has fewer than 𝑘 entries. Every thread also sets the logit of

3.3.3 Top-𝑘 Selection and Gather. Selection turns each beam’s logits into scores with a log-softmax over its valid slots, adds the beam’s accumulated score, and keeps the 𝐵 highest-scoring candidates of each request. A request has 𝜌 ℓ 𝐵|Vℓ | candidates, a number that varies widely across steps: it is a sizable fraction of 𝐵|Vℓ | at early steps but barely exceeds 𝐵 at later steps, where most prefixes admit only one or two tokens. ZeroGR therefore performs this selection with two top-𝑘 kernels. For many candidates, a radixselection kernel [28] splits the slots of a request across many thread blocks and finds the 𝐵-th highest score in several passes over them, and a separate kernel then performs gather. When the candidates of a request fit in one thread block, a small top-𝑘 kernel instead runs the whole selection in that block: the block locates the valid 7

CPU

Algorithm 1 Two-pass construction of sparse mappings Require: Sorted item SID table T Ensure: Sparse mappings Mℓ = (𝐴ℓ , 𝑅ℓ ) for ℓ = 1, . . . , 𝐿 1: 𝑛 ℓ ← 0 for ℓ = 1, . . . , 𝐿 2: for 𝑖 = 1, . . . , |T | do 3: ℎ ← LCP(s(𝑖 − 1), s(𝑖)) 4: 𝑛 ℓ ← 𝑛 ℓ + 1 for ℓ = ℎ + 1, . . . , 𝐿 5: end for 6: Allocate 𝐴ℓ and 𝑅ℓ for every ℓ from the counts 7: for 𝑖 = 1, . . . , |T | do 8: ℎ ← LCP(s(𝑖 − 1), s(𝑖)) 9: for ℓ = ℎ + 1, . . . , 𝐿 do 10: Append 𝑠 ℓ(𝑖 ) to 𝐴ℓ 11: if ℓ < 𝐿 then append |𝐴ℓ+1 | to 𝑅ℓ+1 12: end for 13: end for 14: return M 1 , . . . , M𝐿

Pinned buffers

𝒯

A

Dense mapping

batch i+1 Sparse mapping

Swap roles

Parallel fill

B

batch i

H2D

𝑅ℓ 𝐴ℓ

Figure 10: Double-buffered index construction.

every step. After the first pass finishes, Line 6 allocates the mappings from these counts. Each 𝑛 ℓ sizes the SID array 𝐴ℓ of Mℓ and, for ℓ < 𝐿, also the boundary array 𝑅ℓ+1 of Mℓ+1 , because each valid continuation at step ℓ yields one state at step ℓ + 1. Once the arrays are allocated, the second pass in Lines 7–13 scans T again in the same order and fills them. For each sequence, Line 10 appends the SIDs at steps ℎ + 1 to 𝐿, which are its new valid continuations, to the corresponding SID arrays. Line 11 then opens, for every appended SID, the segment that will hold its continuations at the next step, by recording the current end of 𝐴ℓ+1 in 𝑅ℓ+1 . The segments of Mℓ+1 are thus created in the same order as the SIDs in 𝐴ℓ , so the position of each SID directly serves as its successor state. Both passes are easy to parallelize, because each sequence is compared only with its predecessor. ZeroGR partitions T into 𝐶 contiguous chunks and runs the two passes over the chunks on separate CPU threads. In the first pass, each chunk counts its own valid continuations at every step, and exclusive prefix sums [5] over these per-chunk counts assign each chunk a disjoint output range at every step. In the second pass, each chunk then writes its fragment directly into its assigned ranges, independently of the other chunks. Dense mappings fit into the same passes even easily: instead of appending the SID, the second pass writes the successor state into the cell of that SID. The two passes establish the placement and contents of the index fragments. Figure 10 shows their delivery to the GPU through two reusable pinned buffers. Adjacent chunks form batches that fit a buffer, allowing CPU generation of one batch to overlap asynchronous H2D transfer of another into its final array locations [23]. Double buffering helps the rebuild meet both of its online requirements. First, it shortens construction: while the CPU fills one buffer, the other is being copied to the GPU, so transfer effectively overlaps with generation. Second, it avoids a burst of H2D traffic. Transfer starts as soon as the first batch is ready, so the index reaches the GPU in small batches spread across the rebuild rather than in one large copy at its end. Inference requests served at the same time therefore do not wait behind a long index transfer.

slots from the segment length of each beam and sorts only their scores in shared memory. Because the position of a selected slot determines the next beam, the same block also performs gather, so selection and gather take a single kernel. Both kernels select the same candidates, so the choice between them affects only latency. Gather forms the beams of the next step from the selected slots. Each new beam extends the prefix of its parent beam by the token of its slot. As with the mapping and projection mode, ZeroGR fixes the top-𝑘 kernel of each step, and thus whether gather is fused with selection, during warm-up.

3.4

GPU

Item SID table

Index Construction

With the representation of each mapping fixed, ZeroGR must rebuild the index within the update window each time the item catalog is refreshed. The rebuild runs while the instance continues to serve online requests, so it must not compete with inference for GPU compute or host-to-GPU (H2D) bandwidth. To meet these requirements, ZeroGR introduces layout-guided streaming construction. In order to meet the online serving requirements, the index is constructed in two CPU passes: the first determines the output layout, and the second generates the mappings in parallel while streaming them to the GPU. The input is a host-resident item SID table T that lists every item with its length-𝐿 SID sequence, sorted lexicographically by sequence. We number the items in this order, so the 𝑖-th row holds s(𝑖). The output is the GPU-resident mappings M1, . . . , M𝐿 , each in its chosen representation. Algorithm 1 shows how layout-guided streaming construction builds sparse mappings. Lines 1–5 form the first pass, which determines how much space each mapping needs before any SID is written. Its counter 𝑛 ℓ records the number of valid continuations at step ℓ, which is the number of SIDs that Mℓ stores. Because T is sorted, sequences that share a prefix are adjacent. A sequence therefore repeats the valid continuations of its predecessor up to their longest common prefix (LCP) and introduces new ones only at the steps beyond it [9]. For an LCP of length ℎ, the algorithm increments 𝑛ℎ+1, . . . , 𝑛𝐿 . The first sequence takes the empty sequence s(0) = 𝜖 as its predecessor, so it introduces a valid continuation at

4

EVALUATION

This section answers three questions about ZeroGR: • How much does ZeroGR reduce latency relative to the baselines on real-world and synthetic datasets, and which steps and stages account for it (§ 4.2 and § 4.3)? • How much does each design choice contribute (§ 4.4)? 8

4.1

ZeroGR speedup (×)

• How much memory does the ZeroGR index save, and does its construction complete within the update window without competing with online serving for GPU compute or host-to-GPU bandwidth (§ 4.5 and § 4.6)?

Experimental Setup

We run experiments on a server with one NVIDIA GeForce RTX 4090 D GPU and two Intel Xeon Gold 5520+ processors. Some experiments are run on a server with a data center GPU. We compare ZeroGR with four baselines.

vs. CPU trie vs. FlashTrie

FP32 / H = 4,096

100

FP16 / H = 1,024

20 5 1 1

2

4

8

16 32

1

2

4

8

16 32

Request batch size

Figure 11: Speedup of ZeroGR over the baselines on the Search dataset, measured on 4090 D. ZeroGR speedup (×)

• Unconstrained performs full computation without any constraints. It serves as a performance reference; its outputs need not correspond to items in the catalog. • CPU trie is the constrained decoder of our previously deployed serving system, which keeps the prefix tree in host memory and retrieves valid continuations on the CPU. • FlashTrie [1] encodes the prefix tree as a succinct bit-vector structure in GPU memory and searches it on the GPU. • STATIC [30] flattens the prefix tree into a sparse index and performs constraint filtering in GPU memory.

vs. Unconstrained vs. STATIC

vs. CPU trie vs. FlashTrie

FP32 / H = 4,096

FP16 / H = 1,024

100 20 5 1 1

2

4

8

16 32

1

2

4

8

16 32

Request batch size

Figure 12: Speedup of ZeroGR over the baselines on the Search dataset, measured on the data center GPU.

The CPU trie, FlashTrie, and STATIC baselines score all continuations before filtering, whereas ZeroGR filters first and scores only the valid continuations. ZeroGR and the CPU trie are implemented in C++ and CUDA, and FlashTrie and STATIC run on PyTorch 2.11. We evaluate on two real-world datasets and one synthetic dataset. The Search dataset is collected from an industrial generative search service and contains nearly one billion distinct five-token SID sequences. Its model projects onto a single 18,432-token output head at every step, although the tokens of each step occupy a disjoint band of 8,192, 4,096, 2,048, 2,048, or 2,048 entries. The Watch dataset contains approximately 4.17 billion distinct three-token SID sequences and uses an 8,192-token output head at each step. The synthetic dataset, Synthetic-7, has seven-token SID sequences, and its variants differ in vocabulary size and branching factors.

4.2

vs. Unconstrained vs. STATIC

At large batch sizes, STATIC is even faster than Unconstrained. This is because we adapt STATIC to apply the softmax only over the valid tokens, as ZeroGR does, whereas Unconstrained normalizes over all 18,432 tokens, and the smaller softmax can outweigh the lookup cost of STATIC as the batch grows. These results show that applying the constraints before projection reduces decoding work instead of adding to it. On the Search dataset, every beam is projected onto a single 18,432-token output head at each step, although only a small fraction of these tokens can extend the prefix of the beam. Unconstrained and STATIC compute the logits of all these tokens, and STATIC uses the constraints only to mask the logits afterward. ZeroGR instead filters the candidates of each beam first and, at the sparse later steps, computes only the logits of the valid tokens, which removes most of the projection work. Figure 12 further shows that ZeroGR outperforms the baselines on the Search dataset when measured on the data center GPU. With FP16, the latency of ZeroGR stays between 0.93 and 1.07 ms from batch size 1 to 32, whereas Unconstrained and the CPU trie slow down as the batch size grows. Computing only valid logits limits the latency increase for larger batches, giving ZeroGR its greatest advantage at the large batch sizes used in production. To evaluate ZeroGR on a catalog in which most tokens remain valid, Figure 13 shows its speedup over the baselines on the Watch dataset, measured on 4090 D. STATIC runs out of GPU memory on this GPU and is therefore omitted. ZeroGR outperforms the CPU trie by up to 205×, because the CPU trie returns thousands of valid tokens per beam from the host at the dense steps. Relative to Unconstrained, ZeroGR is faster at every batch size with FP32 and at small batch sizes with FP16. These results show that ZeroGR guarantees valid outputs at almost the speed of unconstrained decoding, and often faster, even when the constraints remove little computation. On the Watch dataset, every token is valid at the first step and about 90% of the tokens remain valid at the second step, so

End-to-End Performance

In this section, we measure the total latency of all 𝐿 decoding steps on both real-world and synthetic datasets, excluding model inference time. We use beam width 128 and request batch sizes 1 to 32, with hidden size 4,096 for FP32 and 1,024 for FP16. 4.2.1 Real-World Datasets. For the real-world datasets, we evaluate ZeroGR on the Search and Watch datasets with two environments: 4090 D and a data center GPU. To evaluate ZeroGR performance on a real-world workload across batch sizes and precisions, Figure 11 shows its speedup over the baselines on the Search dataset, measured on 4090 D. ZeroGR outperforms all baselines at every batch size and precision, including Unconstrained, which enforces no constraints. At batch size 32, ZeroGR completes all five decoding steps in 9.68 ms with FP32 and 1.88 ms with FP16, the latter 9.1× faster than Unconstrained. With FP16, the speedup over Unconstrained is smaller at small batch sizes, where the hidden size of 1,024 already makes full projection inexpensive. The gap to the CPU trie widens with the batch size and approaches two orders of magnitude with FP16, because its host-side lookup grows with the number of active beams. 9

vs. CPU trie

FP32 / H = 4,096

B = 64

vs. FlashTrie

FP16 / H = 1,024

Speedup (×)

ZeroGR speedup (×)

vs. Unconstrained

100 20 5 1 1

2

4

8

16 32

1

2

4

8

Speedup (×)

ZeroGR speedup (×)

FP16 / H = 1,024

1 4

8

16 32

1

2

4

8

8 16 32

V = 2K

vs. CPU trie vs. FlashTrie

20 5 2

4

1

(c) Unconstrained

2

1

4

8 16 32

2

4

8 16 32

Figure 15: Speedup of ZeroGR over the baselines on Synthetic-7 for beam widths 𝐵 from 64 to 256.

100

1

2

B = 256

(b) CPU trie

Request batch size

Figure 13: Speedup of ZeroGR over the baselines on the Watch dataset, measured on 4090 D.

FP32 / H = 4,096

1

16 32

Request batch size

vs. Unconstrained vs. STATIC

64 32 16 8 4 2 1

B = 128

(a) STATIC

64 32 16 8 4 2 1

V = 4K

(a) STATIC

1

2

4

8 16 32

1

V = 8K

V = 16K

(b) CPU trie

(c) Unconstrained

2

1

4

8 16 32

2

4

8 16 32

Request batch size

16 32

Request batch size

Figure 16: Speedup of ZeroGR over the baselines on Synthetic-7 for vocabulary sizes 𝑉 from 2,048 to 16,384.

Figure 14: Speedup of ZeroGR over the baselines on the Watch dataset, measured on the data center GPU.

ZeroGR scales better than the baselines as a batch contains more beams. Full projection computes 𝑉 logits for every beam, so the projection cost of Unconstrained and STATIC grows with the total number of beams. ZeroGR instead chooses the projection mode for each step, improving scaling as the number of beams grows. To evaluate how the width of the output head affects ZeroGR, Figure 16 varies 𝑉 from 2,048 to 16,384 with beam width 128. The speedup over Unconstrained and STATIC generally grows with 𝑉 , reaching 19.5× over Unconstrained at 𝑉 = 16,384 and batch size 32, while the latency of ZeroGR at batch size 32 stays between 3.1 and 4.5 ms across all four vocabulary sizes. These results show that the cost of ZeroGR follows the number of valid tokens rather than the width of the output head. As 𝑉 grows, full projection computes more logits for every beam, whereas the number of valid next tokens stays at four for steps 2 to 6 and even decreases at step 7. This property allows production deployments to adopt larger codebooks, for example to represent a growing item catalog, without a corresponding increase in decoding latency.

only the last of the three steps is sparse. ZeroGR uses full projection for most of the dense steps and selective projection at the sparse last step, so it adds no projection work where the constraints remove little and saves work where they remove much. Its small slowdown with FP16 at large batch sizes is expected, since Unconstrained skips the constraint lookup entirely and does not guarantee that its outputs correspond to items in the catalog. Figure 14 further shows the results on the Watch dataset when measured on the data center GPU, whose memory is large enough for STATIC. Although the Watch dataset is less favorable to ZeroGR than the Search dataset, because its constraints remove little projection work, ZeroGR remains 2.0–3.2× faster than STATIC. Its speedups over Unconstrained and the CPU trie follow the same trends as on 4090 D. The per-step choice of projection mode keeps these trends stable across GPUs: ZeroGR uses full projection at the dense steps and selective projection where it is faster, and its choices adapt to the GPU, precision, and batch size. Since STATIC is slower than Unconstrained at every batch size, applying the constraints after full projection adds substantial overhead when most tokens are valid, whereas ZeroGR stays close to Unconstrained.

4.3

Latency Breakdown

To understand the sources of ZeroGR’s performance improvements, Figure 17 breaks the latency on the Search dataset down by decoding step. Step 2 is the most expensive step of ZeroGR in every configuration, and the latency of ZeroGR drops sharply after it: with FP32 at batch size 1, each of steps 3 to 5 takes about 0.08 ms, compared with about 1.04 ms for STATIC. This pattern follows from the design of ZeroGR, which computes only the logits of valid tokens wherever this is cheaper than full projection, so the cost of each step follows how many tokens are valid at that step. Step 2 has the most valid tokens of all steps, whereas few tokens are valid at the later steps, which therefore cost almost nothing; the baselines, in contrast, pay for the full output head at every step. To see where the time goes within each step, Figure 18 reports the speedup of ZeroGR for three stages: filtering and projection,

4.2.2 Synthetic Dataset. Real-world datasets fix the branching factor at each step and the width of the output heads. We vary these factors through the variants of Synthetic-7. We evaluate all variants on 4090 D. Synthetic-7 contains 230 distinct seven-token SID sequences; all seven steps share a 𝑉 -token output head, and its variants differ in 𝑉 . To evaluate how ZeroGR scales with the number of beams, Figure 15 shows its speedup on Synthetic-7 with FP32 for beam widths from 64 to 256 and 𝑉 = 8,192. The speedup over every baseline grows with the batch size and is generally higher for wider beams, reaching 10.7× over Unconstrained at beam width 256 and batch size 32, where ZeroGR takes 8.28 ms. These results show that 10

STATIC

(a) FP32, batch 1

(b) FP16, batch 1

Full projection Naive selective ZeroGR selective

0.1

5

1

2

3

4

5

Decoding step

Stage speedup (×)

vs. CPU trie Filtering + Projection

10 1 32 16

32 16

32 16

32 16

32 16

batch 32

batch 1

batch 32

batch 1

batch 32

Precision (bits)

Figure 18: Per-stage speedup of ZeroGR over the CPU trie and STATIC on the Search dataset. normalization, and selection and update. Filtering and projection form the most expensive stage: at batch size 32 with FP32, they account for 97% of the latency of STATIC and nearly all of the latency of the CPU trie. ZeroGR gains the most in this stage, up to 120× over the CPU trie, because filtering reads the valid tokens of each beam directly from the per-step mapping on the GPU and selective projection then computes only the logits of these tokens. Since this stage dominates the decoding latency, computing only valid logits accounts for most of the overall speedup of ZeroGR. ZeroGR also remains faster than STATIC in the two later stages in every configuration; in ZeroGR, both stages read the shared candidate slots in place, and selection uses the top-𝑘 kernel chosen for each step. Against the CPU trie, these two stages perform about the same, since the CPU trie also runs them on the GPU over the valid candidates. Thus, the most expensive stage provides most of the speedup of ZeroGR, which the later stages preserve.

4.4

40 0

Search

Watch

(b) Index memory footprint.

requires counting them and computing prefix sums, and the naive projection creates a temporary matrix for valid projection lines before computing the logits in a separate kernel. Although naive selective projection computes only a small fraction of the logits of full projection, it is 5.1× slower, because it lacks the support of efficient memory access patterns. ZeroGR avoids these overheads: its filtering kernel writes each candidate into a fixed slot without prefix sums, and its projection kernel reads the weight rows in place and loads each hidden vector once per task. It thereby computes the same logits 8.5× faster than naive selective projection and also outperforms full projection. The execution model of ZeroGR is thus what turns the reduction in logits into a reduction in latency. With selective projection in place, ZeroGR further adds a small top-𝑘 kernel and selects the projection mode of each step. Figure 19(a) adds these two design choices one at a time to a variant that uses selective projection at every step, on the Search dataset at batch size 32. The small top-𝑘 kernel accelerates the later steps, where few candidates remain, by running the selection and gather of each request in one thread block, and reduces the latency of steps 3 to 5 by up to 21%. Selecting the projection mode then picks the faster mode for every step and precision: ZeroGR uses full projection at step 1, where almost every token is valid, and at step 2 uses selective projection with FP32 and full projection with FP16. With FP16, the two design choices together reduce the decoding latency of ZeroGR by 27% relative to selective projection alone.

100

32 16

80

Figure 19: Decoding latency on the Search dataset at batch size 32 and index memory footprint.

Selection & update

batch 1

FP16

(a) Decoding latency.

vs. STATIC

Normalization

4

FP32

STATIC

120

8

0

Figure 17: Per-step latency of ZeroGR, the CPU trie, and STATIC on the Search dataset.

SID table ZeroGR

8.70

4

+ Small top-k

114.44

3

Latency (ms)

1 2

507.8 2,594.9 305.1

34.07

Selective only ZeroGR 12

10

1

2,359,296 71,189 71,189

(d) FP16, batch 32

100

0.1

Latency (𝜇s)

8.51

(c) FP32, batch 32

Logits

Size (GB)

Step latency (ms)

Method 1

27.52

10

Table 1: Filtering and projection latency at decoding step 2 on the Search dataset with FP32 and batch size 1.

ZeroGR

20.78

CPU trie

4.5

Memory Footprint

To evaluate the memory usage of ZeroGR, Figure 19(b) compares the GPU-resident indices of STATIC and ZeroGR on the Search and Watch datasets, together with the SID table, stored as a TSV file, from which both indices are built. The index of ZeroGR is 59.1% and 74.5% smaller than that of STATIC on the two datasets, respectively, and it is smaller than the SID table on both. This reduction comes from how each index stores a valid extension. STATIC stores a successor state alongside the SID of every extension, whereas ZeroGR stores only the SID, because the position of an entry serves as its successor state and a final SID needs

Ablation Study

Most of the speedup of ZeroGR comes from computing only the logits of valid tokens, but computing fewer logits does not by itself reduce latency. Table 1 examines this effect at decoding step 2 of the Search dataset, the step with the most valid tokens, with FP32 and batch size 1. It compares the selective projection of ZeroGR with full projection and with a naive selective projection that computes the same logits without shared candidate slots. The naive filtering packs the candidates of all beams into a compact list, which 11

50

(a)

0.36

0 TIC CPU oGR STA ive Zer a N

71.7 3 2 1 0

Layout Emit/H2D

3

(b)

Time (×)

Time (%)

100

Bandwidth (×)

ZeroGR Naive CPU

60 s

90 s

2.57

2

0

5

(c) 1.30

1

through bounded buffers and overlapping generation with transfer lets ZeroGR avoid both overheads.

Memory Residual

1.00

Full host 1 buffer 2 buffers

Figure 20: Index construction on the Search dataset. (a) Time relative to STATIC. (b) H2D bandwidth relative to ZeroGR in each window. (c) Staging time relative to ZeroGR.

Constrained decoding for generative recommendation. Constrained decoding enforces validity at every step by looking up the valid tokens of each prefix in an index and masking the scores of invalid tokens [2, 10, 21, 31, 35, 43]. To speed up this per-step lookup, recent work moves the index into accelerator memory, but the full-vocabulary projection remains unchanged [1, 30]. Some systems further extend these constraints to individual requests, but cannot guarantee that every result satisfies them [13, 41, 42]. All of these methods score first, using the constraint only to discard invalid tokens after the projection has computed the logits of every token rather than to eliminate the redundant computation itself.

none. The reduction is larger on the Watch dataset, where final SIDs make up almost the entire index. On 4090 D, which has 24 GB of memory, the index of STATIC occupies most of the memory on the Search dataset and does not fit on the Watch dataset, whereas the index of ZeroGR fits on both and leaves most of the memory to the model weights and attention caches.

4.6

RELATED WORK

Traditional filtering methods. Many GR systems decode without constraints and handle invalid outputs afterward. Rewarding valid outputs during reinforcement learning makes invalid SID sequences rarer but cannot rule them out [45]. Post-generation filtering discards candidates that map to no item or violate business rules once decoding completes [25, 27, 33, 37]. Neither approach enforces validity during decoding, so invalid prefixes still occupy beam slots and consume decoding work before they are removed.

Index Construction Performance

We evaluate how efficiently ZeroGR rebuilds its filtering index by comparing construction time and H2D requirements on the Search dataset. Figure 20 reports the results, with construction measured from the sorted item SID table in host memory to the GPU-resident mappings. Alongside STATIC, we include Naive CPU, a serial builder of the same step-sliced filtering index as ZeroGR. It builds the mappings one step at a time without streaming. Figure 20(a) gives the total construction time of each method relative to STATIC. ZeroGR completes the rebuild in 0.914 s, which is 0.36% of STATIC’s time, whereas STATIC exceeds the online service update windows. Naive CPU takes less time than STATIC, but it remains far slower than ZeroGR even though both build the same index. The colored bars in Figure 20(b) estimate the H2D bandwidth needed within each update window after CPU construction. For each window, the bars are normalized to ZeroGR’s bandwidth, with the dashed line marking the unit baseline. With its compact payload and short CPU phase, ZeroGR requires 0.144 GB/s for a 60 s window. STATIC’s CPU phase alone exceeds both windows. Naive CPU uses almost the entire 60 s window for construction, so it needs very high bandwidth to transfer the index in the remaining time, as the broken bar shows. Even with the 90 s window, Naive CPU still requires 2.9× the bandwidth of ZeroGR. Figure 20(c) compares three variants of the same parallel twopass builder, with all stage times normalized to ZeroGR’s total. Full host materializes and registers the complete host output before transfer. Single buffer generates and transfers batches through one pinned buffer; ZeroGR uses two such buffers to overlap these operations. ZeroGR is 2.57× faster than Full host and 1.30× faster than Single buffer. Full host spends most of its extra time registering the complete output as pinned memory and releasing it after the transfer. Single buffer removes this cost, but its CPU must wait for each transfer to finish before generating the next batch. Staging

6

CONCLUSION

We present ZeroGR, a filtering-first constrained decoding system for generative recommendation. Instead of scoring every token and then discarding invalid continuations, ZeroGR filters first and scores only valid continuations, with negligible impact on recommendation quality. It combines a step-sliced filtering index that keeps the index compact in GPU memory, an execution engine whose shared-slot execution and hybrid projection turn reduced scoring into lower latency, and layout-guided streaming construction that rebuilds the index within the update window while limiting interference with online serving. Compared with representative constrained decoding baselines, ZeroGR reduces the combined latency of filtering, scoring, and candidate selection by up to 9.9×, reduces GPU index memory use by up to 74.5%, and rebuilds its index 278× faster.

12

REFERENCES

[19] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. 2022. Autoregressive Image Generation using Residual Quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11523–11532. https://arxiv.org/abs/2203.01941 [20] Gurvan L’Hostis, David Grangier, and Michael Auli. 2016. Vocabulary Selection Strategies for Neural Machine Translation. arXiv preprint arXiv:1610.00072 (2016). https://arxiv.org/abs/1610.00072 [21] Fengxin Li, Yi Li, Yue Liu, Chao Zhou, Yuan Wang, Xiaoxiang Deng, Wei Xue, Dapeng Liu, Lei Xiao, Haijie Gu, Jie Jiang, Hongyan Liu, Biao Qin, and Jun He. 2025. LEADRE: Multi-Faceted Knowledge Enhanced LLM Empowered Display Advertisement Recommender System. Proceedings of the VLDB Endowment 18, 12 (2025), 4763–4776. https://doi.org/10.14778/3750601.3750602 [22] Xinyu Lin, Chaoqun Yang, Wenjie Wang, Yongqi Li, Cunxiao Du, Fuli Feng, See-Kiong Ng, and Tat-Seng Chua. 2025. Efficient Inference for Large Language Model-based Generative Recommendation. In International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2025/hash/ e4bf5c3245fd92a4554a16af9803b757-Abstract-Conference.html [23] NVIDIA. 2026. CUDA C++ Best Practices Guide. https://docs.nvidia.com/cuda/ cuda-c-best-practices-guide/ [24] OneRec Team. 2026. OneReason Technical Report. arXiv preprint arXiv:2606.06260 (2026). https://arxiv.org/abs/2606.06260 [25] Yunsheng Pang, Zijian Liu, Yudong Li, Shaojie Zhu, Zijian Luo, Chenyun Yu, Sikai Wu, Shichen Shen, Cong Xu, Bin Wang, Kai Jiang, Chengxiang Zhuo, and Zang Li. 2025. HiGR: Industrial-Scale Hierarchical Generative Slate Recommendation Framework in Tencent. arXiv preprint arXiv:2512.24787 (2025). https://arxiv.org/ abs/2512.24787 [26] Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick, Nadia Polikarpova, and Loris D’Antoni. 2024. Grammar-Aligned Decoding. In Advances in Neural Information Processing Systems, Vol. 37. https://arxiv.org/abs/2405.21047 [27] Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan H. Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Maheswaran Sathiamoorthy. 2023. Recommender Systems with Generative Retrieval. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS ’23). Curran Associates Inc., Red Hook, NY, USA. [28] Anil Shanbhag, Holger Pirk, and Samuel Madden. 2018. Efficient Top-K Query Processing on Massively Parallel Hardware. In Proceedings of the 2018 International Conference on Management of Data. 1557–1570. https://doi.org/10.1145/ 3183713.3183735 [29] Anima Singh, Trung Vu, Nikhil Mehta, Raghunandan Keshavan, Maheswaran Sathiamoorthy, Yilin Zheng, Lichan Hong, Lukasz Heldt, Li Wei, Devansh Tandon, Ed H. Chi, and Xinyang Yi. 2024. Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations. In Proceedings of the 18th ACM Conference on Recommender Systems. https://doi.org/10.1145/3640457.3688190 [30] Zhengyang Su, Isay Katsman, Yueqi Wang, Ruining He, Lukasz Heldt, Raghunandan Keshavan, Shao-Chuan Wang, Xinyang Yi, Mingyan Gao, Onkar Dalal, Lichan Hong, Ed Chi, and Ningren Han. 2026. Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. https://doi.org/10.1145/3770855.3818506 [31] Qingxiao Sun, Tongxuan Liu, Shen Zhang, Siyu Wu, Peijun Yang, Haotian Liang, Menxin Li, Xiaolong Ma, Zhiwei Liang, Ziyi Ren, Minchao Zhang, Yifan Wang, Xinyu Liu, Ke Zhang, Hailong Yang, and Depei Qian. 2025. xGR: Efficient Generative Recommendation Serving at Scale. arXiv preprint arXiv:2512.11529 (2025). https://arxiv.org/abs/2512.11529 [32] Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer Memory as a Differentiable Search Index. In Advances in Neural Information Processing Systems, Vol. 35. https://arxiv.org/ abs/2202.06991 [33] Yejing Wang, Shengyu Zhou, Jinyu Lu, Ziwei Liu, Langming Liu, Maolin Wang, Wenlin Zhang, Feng Li, Wenbo Su, Pengjie Wang, Jian Xu, and Xiangyu Zhao. 2026. NEZHA: A Zero-sacrifice and Hyperspeed Decoding Architecture for Generative Recommendations. In Proceedings of the ACM Web Conference 2026. 8073–8082. https://doi.org/10.1145/3774904.3792797 [34] Zhibin Wang, Rui Ning, Chao Fang, Zhonghui Zhang, Xi Lin, Shaobo Ma, Mo Zhou, Xue Li, Zhongfeng Wang, Chengying Huan, Rong Gu, Kun Yang, Guihai Chen, Sheng Zhong, and Chen Tian. 2026. CoDec: Prefix-Shared Decoding Kernel for LLMs. Proceedings of the ACM on Management of Data 4, 3 (2026), 27. https://doi.org/10.1145/3802028 [35] Brandon T. Willard and Rémi Louf. 2023. Efficient Guided Generation for Large Language Models. arXiv preprint arXiv:2307.09702 (2023). https://arxiv.org/abs/ 2307.09702 [36] Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. 2023. Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity. Proceedings of the VLDB Endowment 17, 2 (2023), 211–224. https://doi.org/10.14778/3626292.3626303

[1] Dakshitha B. Anandakumar, Anurag Mukkara, Wenxiang Hu, Akash Kumar Mohankumar, Jiusheng Chen, Qiang Lou, Jian Jiao, and Ting Ye. 2026. FlashTrie: A GPU-Accelerated Constrained Beam Search for Generative Retrieval. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track. Association for Computational Linguistics, 2662–2683. https://aclanthology.org/2026.emnlp-industry.157/ [2] Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Wen-tau Yih, Sebastian Riedel, and Fabio Petroni. 2022. Autoregressive Search Engines: Generating Substrings as Document Identifiers. In Advances in Neural Information Processing Systems, Vol. 35. https://arxiv.org/abs/2204.10628 [3] Markus Billeter, Ola Olsson, and Ulf Assarsson. 2009. Efficient Stream Compaction on Wide SIMD Many-Core Architectures. In Proceedings of the Conference on High Performance Graphics. 159–166. https://doi.org/10.1145/1572769.1572795 [4] Robert Binna, Eva Zangerle, Martin Pichl, Günther Specht, and Viktor Leis. 2018. HOT: A Height Optimized Trie Index for Main-Memory Database Systems. In Proceedings of the 2018 International Conference on Management of Data. 521–534. https://doi.org/10.1145/3183713.3196896 [5] Guy E. Blelloch. 1990. Prefix Sums and Their Applications. Technical Report CMU-CS-90-190. School of Computer Science, Carnegie Mellon University. https://www.cs.cmu.edu/afs/cs.cmu.edu/project/scandal/public/papers/ CMU-CS-90-190.html [6] Ben Chen, Xian Guo, Siyuan Wang, Zihan Liang, Yufei Ma, Yue Lv, Chenyi Lei, Yuqing Ding, Wenwu Ou, Han Li, and Kun Gai. 2026. OneSearch: A Preliminary Exploration of the Unified End-to-End Generative Framework for E-commerce Search. In Proceedings of the 43rd International Conference on Machine Learning. https://arxiv.org/abs/2509.03236 [7] Yaoqi Chen, Jinkai Zhang, Baotong Lu, Qianxi Zhang, Chengruidong Zhang, Jing Liu, Jingjia Luo, Di Liu, Huiqiang Jiang, Qi Chen, Bailu Ding, Xiao Yan, Jiawei Jiang, Chen Chen, Mingxing Zhang, Cheng Li, Yuqing Yang, Fan Yang, and Mao Yang. 2026. RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference. Proceedings of the VLDB Endowment 19, 5 (2026), 1016–1031. https://doi.org/10.14778/3796195.3796212 [8] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems. https://research.google/pubs/deep-neural-networks-foryoutube-recommendations/ [9] Jan Daciuk, Stoyan Mihov, Bruce W. Watson, and Richard E. Watson. 2000. Incremental Construction of Minimal Acyclic Finite State Automata. Computational Linguistics 26, 1 (2000), 3–16. https://aclanthology.org/J00-1002/ [10] Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive Entity Retrieval. In International Conference on Learning Representations. https://openreview.net/forum?id=5k8F6UU39V [11] Jiaxin Deng, Shiyao Wang, Kuo Cai, Lejian Ren, Qigen Hu, Weifeng Ding, Qiang Luo, and Guorui Zhou. 2025. OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment. arXiv preprint arXiv:2502.18965 (2025). https://arxiv.org/abs/2502.18965 [12] Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ziyi Xu, Yilong Zhao, Ruihang Lai, and Tianqi Chen. 2025. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. In Proceedings of Machine Learning and Systems, Vol. 7. https://proceedings.mlsys.org/paper_files/paper/2025/hash/ 5c20ca4b0b20b0bd2f1d839dc605e70f-Abstract-Conference.html [13] Zhou Fang, Yuhang Huang, Ang Zhang, Yihan He, Ruichao Xiao, Chao Li, Yavuz Yetim, Sibyl Yang, Xiaohan Wei, Fei Tian, Liang Wang, Chonglin Sun, Liyuan Li, Nathan Yan, and Gaoxiang Liu. 2026. GRACE: Generative Recommender Acceleration Engine for Real-Time Ads Retrieval. arXiv preprint arXiv:2608.00938 (2026). https://arxiv.org/abs/2608.00938 [14] Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. 2020. Sparse GPU Kernels for Deep Learning. In SC: International Conference for High Performance Computing, Networking, Storage and Analysis. https://arxiv.org/abs/2006.10901 [15] Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. 2017. Efficient Softmax Approximation for GPUs. In Proceedings of the 34th International Conference on Machine Learning, Vol. 70. PMLR, 1302–1310. https://proceedings.mlr.press/v70/grave17a.html [16] F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems 5, 4, Article 19 (2015), 19 pages. https://doi.org/10.1145/2827872 [17] Kijae Hong, Kyoungmin Kim, Young-Koo Lee, Yang-Sae Moon, Sourav S. Bhowmick, and Wook-Shin Han. 2024. Themis: A GPU-accelerated Relational Query Execution Engine. Proceedings of the VLDB Endowment 18, 2 (2024), 426–438. https://doi.org/10.14778/3705829.3705856 [18] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles. https://arxiv.org/abs/2309.06180

13

[37] Xiangrui Yang, Yifan Lei, Jiawei Yang, Yekang Zhan, Yuxin Zheng, Lei Mao, Zhaoren Wu, Yibo Huang, and Yiming Qiu. 2026. AdaBool: Sparse Boolean Matrix Processing with Adaptive Data Representation. arXiv preprint arXiv:2610.21264 (2026). https://arxiv.org/abs/2610.21264 [38] Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Jiayuan He, Yinghai Lu, and Yu Shi. 2024. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research), Vol. 235. PMLR, 58484–58509. [39] Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. 2025. PQCache: Product Quantization-based KVCache for Long Context LLM Inference. Proceedings of the ACM on Management of Data 3, 3 (2025), 30. https://doi.org/10.1145/3725338 [40] Huanchen Zhang, Hyeontaek Lim, Viktor Leis, David G. Andersen, Michael Kaminsky, Kimberly Keeton, and Andrew Pavlo. 2018. SuRF: Practical Range Query Filtering with Fast Succinct Tries. In Proceedings of the ACM International Conference on Management of Data. https://doi.org/10.1145/3183713.3196931 [41] Jun Zhang, Yi Li, Yue Liu, Changping Wang, Yuan Wang, Yuling Xiong, Xun Liu, Haiyang Wu, Qian Li, Enming Zhang, Jiawei Sun, Xin Xu, Zishuai Zhang, Ruoran Liu, Suyuan Huang, Zhaoxin Zhang, Zhengkai Guo, Shuojin Yang, Meng-Hao

Guo, Huan Yu, Jie Jiang, and Shi-Min Hu. 2025. GPR: Towards a Generative Pre-trained One-Model Paradigm for Large-Scale Advertising Recommendation. arXiv preprint arXiv:2511.10138 (2025). https://arxiv.org/abs/2511.10138 [42] Xinxun Zhang, Yuling Xiong, Yangru Huang, Jiale Zhou, Zhengkai Guo, Zhennan Pang, Junbang Huo, Jingwen Wang, Xuyang Sun, Enming Zhang, Jiaguang Jin, Changping Wang, Yi Li, Jun Zhang, Xiao Yan, Jiawei Jiang, and Jie Jiang. 2026. UniVA: Unified Value Alignment for Generative Recommendation in Online Advertising at Tencent. arXiv preprint arXiv:2605.05803 (2026). https: //arxiv.org/abs/2605.05803 [43] Bowen Zheng, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, Ming Chen, and Ji-Rong Wen. 2024. Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation. In Proceedings of the 40th IEEE International Conference on Data Engineering (ICDE). 1435–1448. https://arxiv. org/abs/2311.09049 [44] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems, Vol. 37. https://arxiv.org/abs/2312.07104 [45] Guorui Zhou, Jiaxin Deng, Jinghao Zhang, et al. 2025. OneRec Technical Report. arXiv preprint arXiv:2506.13695 (2025). https://arxiv.org/abs/2506.13695

14

Related documents

• CoFiRec: Coarse-to-Fine Tokenization for Generative Recommendation#798680
• CoFiRec: Coarse-to-Fine Tokenization for Generative Recommendation#797266
• Neutralizing Popularity Bias in LLM-based Recommendation via Counterfactual Reasoning Guidelines#972904
• Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation#328709
• MVIGER: Multi-View Variational Integration of Complementary Knowledge for Generative Recommender#984806