From Generation to Scoring: Scaling LLM-Based Semantic Labeling at LinkedIn

F ROM G ENERATION TO S CORING : S CALING LLM-BASED S EMANTIC L ABELING AT L INKED I N

arXiv:2610.36383v1 [cs.LG] 26 Oct 2026

Jian Sheng 1 * Sriram Vasudevan 1 * Sundara Raman Ramachandran 1 * Yang Hu 1 * Junyu Zhou 1 * Xiaojing Ma 1 * Haolan Qin 1 Tiffany Zhou 1 Vedant Jhaveri 1 Ashwini Bhingare 1 Chinmay Kini 1 Qing Lan 1 Xiaoqing Wang 1 Suleyman Cetintas 1 Fedor Borisyuk 1

A BSTRACT For efficiency, industrial large language model (LLM) labelers often pack many candidate labels into one prompt and generate a YES/NO answer for each, even when each decision depends only on the entity and one candidate. Generating these independent decisions as one answer list couples them, spends decoding on the list, and relies on the list being well-formed. In this paper, we present an end-to-end recipe that reformulates the labeler as semantic scoring, fine-tuning it on single-candidate prompts to return each decision as one answer-token score from a prefill-only pass, and we adapt Multiple-Item Scoring (MIS) to score hundreds of candidates in one sequence with attention masked between them. Deployed for profile qualification labeling at LinkedIn, semantic scoring changes at most 0.01% of decisions when candidates are reordered and matches the agreement with reference labels that packed generation reaches only with about 4.7× as many training pairs, and the MIS-based stack raises throughput over the deployed stack by 1.24× offline and 1.9× in model serving under a one-second p99 latency target. For offline batch inference, we implement MIS-based scoring as a reusable, model-agnostic Ray Data API, proposed upstream, and model serving uses MIS support contributed to SGLang. 1

I NTRODUCTION

Large language models (LLMs) are increasingly used to label, classify, and judge content (Gilardi et al., 2023; Zheng et al., 2023). A common industrial pattern precomputes which candidate labels apply to each entity and serves the decisions from storage, so labeling cost depends on the number of entity–label judgments and on execution efficiency. At LinkedIn, the qualification labeling system identifies which qualifications from a catalog of skills, experience, education, and licenses are supported by the information in a member profile. Labeling runs in offline batch inference and on a streaming update path for profile changes, and downstream matching and ranking reuse the stored predictions. Each prediction should therefore depend only on its profile–qualification pair. The catalog and downstream models are outside the paper’s scope. Each member’s workload pairs one long profile with many short candidate qualifications supplied by retrieval. The existing LLM labeler uses packed generation, a form of batch prompting (Cheng et al., 2023), to answer several qualifications in one prompt, sharing profile encoding but making * Equal contribution 1 LinkedIn. Correspondence to: Jian Sheng <jsheng@linkedin.com>.

each answer depend on the other qualifications and on earlier answers (Section 2.4). The key distinction is between sharing computation and coupling predictions. Packing shares the profile encoding, but generating independent decisions as one answer list also couples them, spends decoding on the list, and relies on the list being well-formed. We instead use an end-to-end recipe that reformulates labeling as semantic scoring, which treats each judgment as pointwise ranking and assigns an independent score to each profile–qualification pair from one prefill pass (Section 3). Scoring one candidate per request keeps decisions independent but repeats the profile in every request, sustaining only about 90 labels/s in model serving against 2.5K for packed generation (Section 6.4). Multiple-Item Scoring (MIS), which our prior work introduced for online LLM ranking (Li et al., 2026; Borisyuk et al., 2026), makes the formulation practical by executing a round of these scores in one request, sharing profile encoding without cross-item interaction (Section 4). Semantic scoring changes at most 0.01% of labels when candidates are reordered, at least 86× fewer than packed generation with the most training data (Section 6.2). Its agreement with reference labels at 45K training pairs takes packed generation about 210K pairs to match. Compared with the deployed packed-generation stack, our MIS-based scoring stack raises labeling throughput by 1.24× in of-

From Generation to Scoring

fline batch inference at each stack’s best batch configuration and by 1.9× in model serving for profile updates at a matched one-second 99th-percentile latency target (Section 6.3). These are stack-level comparisons: the stacks also differ in engine, checkpoint, prompt format, and runtime tuning, so the ratios measure the integrated system rather than MIS alone.

Member profile

Candidate retrieval ranked qualifications Qualification labeling Packed generation (baseline)

Answer-token readout, label-only fine-tuning, and MIS with its candidate-isolating mask each have precedents (Section 8); MIS-based scoring as a batch-inference API in Ray Data is new. We contribute a training and inference system that moves a production labeler from generation to scoring: 1. Generation-to-scoring recipe. We replace generated answer lists with independent per-item scores through shared-prefix prompts, label-only fine-tuning on reference labels, and an answer-token readout whose decision threshold can be set without retraining (Section 3). 2. MIS scoring for batch inference. We adapt MIS (Li et al., 2026) from online ranking to labeling and analyze its cost in prefill tokens per label (Section 4). We implement it as a model-agnostic scoring API for Ray Data batch inference, proposed upstream, which turns one shared query and its candidates into itemaligned scores (Section 5). 3. Deployment and evaluation on both paths. We run one per-candidate decision contract in both offline batch inference and model serving for profile updates, with MIS in SGLang for model serving (Section 5). We evaluate agreement with reference labels, order sensitivity, stack-level throughput, and request sizing for one long profile with hundreds of short candidates (Sections 6–7).

2

BACKGROUND AND P ROBLEM S ETTING

2.1

Case study: profile qualification labeling

The qualification labeling system takes a supplied catalog of qualifications describing skills, experience, education, or licenses. Each entry has a canonical ID, description, category, and embedding. For example, the synthetic profile “Registered nurse; five years in intensive care” supports the qualifications “holds a registered nurse license” and “has intensive-care nursing experience.” The labeler is fine-tuned on profile–qualification pairs with reference YES/NO labels. How these reference labels are produced, and how the retriever is trained, are outside the scope of this paper and unchanged by this work. The retrieve-then-label process embeds a profile, retrieves a ranked list of K catalog qualifications, and labels consec-

Supplied catalog qualification IDs and text

Stored label predictions accepted catalog IDs Downstream consumers matching and ranking

Figure 1. Profile qualification labeling against a supplied catalog. The baseline uses packed generation for the highlighted labeling stage. Sections 3 and 4 replace that stage with semantic scoring and MIS. Stored predictions are reused by downstream consumers.

utive rounds of N candidates (Figure 1). Here K is in the thousands and N in the hundreds. The labeler evaluates at most K qualifications per member in at most ⌈K/N ⌉ rounds. After each round, labeling stops if the accepted fraction falls below a threshold. The rule assumes that lower-ranked qualifications are less likely to be accepted by the labeler. Both retrieval and early stopping trade recall for cost: qualifications outside the top K are never judged, and stopping can miss positive labels later in the list. Accepted IDs form the profile’s stored label set, which downstream systems reuse without rerunning the labeler. These labels are model predictions from profile information, not independently verified qualifications. Stored labels are one input to downstream systems, and this paper does not evaluate downstream decisions. Profile changes trigger a streaming update path that recomputes the profile’s full label set. Offline batch inference processes recent updates and periodically refreshes labels. Semantic scoring and MIS replace the labeling stage without changing retrieval, early stopping, or the stored-label interface. 2.2

Workload shape and notation

Each judgment combines a query q, which holds the task instruction, member profile, and other candidate-invariant text, with an item ci . The item contains the i-th candidate qualification ℓi and the template suffix ending immediately before the model’s YES or NO answer. We write Lq = |q| and Li = |ci | in tokens. Across the benchmark workload, the mean query is about 27 times as long as the mean item (ρwork ≈ 27, Section 6.1). In independent scoring, each query–item pair forms a separate request. Without prefix reuse, most input tokens are copies of the shared query.

From Generation to Scoring

2.3

System requirements

2. Decoding and output format. The T answer-list tokens are generated autoregressively, then parsed and mapped to qualifications by position, so labeling relies on the expected list format. The model is fine-tuned to produce this format, so its loss supervises both label decisions and answer-list formatting.

Quality. The semantic-scoring labeler should reach label quality comparable to the packed-generation baseline’s. We measure label quality as agreement with reference labels (RQ1, Section 6.2). Consistency. For a fixed model and profile–qualification pair, the judgment must be consistent across offline batch inference and the streaming update path, regardless of the other candidates in the prompt or their order. We measure decision flips under candidate reordering (RQ1).

3. Fixed decisions. The labeler returns hard decisions rather than scores that can be thresholded to trade precision for recall. Answer-token probabilities could be read during decoding, but each is conditioned on earlier answers, so it is not an independent per-item score.

Throughput and cost. Offline throughput determines how quickly stored labels can be refreshed after a model or catalog update and how much GPU time each refresh consumes (RQ2, RQ3).

Independent semantic scoring, defined in Section 3, evaluates each profile–qualification pair in its own request. This preserves independence but requires N requests per round, each containing the same query. Prefix caching (Kwon et al., 2023; Zheng et al., 2024) reduces this recomputation only when requests reach a worker still holding the query’s cache (Section 4.2).

Freshness. The streaming update path must absorb bursts of profile edits within a latency target so members’ labels can be updated without waiting for the next batch refresh. We evaluate it as throughput at a fixed tail-latency target (RQ2), not as end-to-end label freshness. 2.4

Packed-generation baseline and its trade-off

Within each round (Section 2.1), the packed-generation baseline divides the N candidates into ⌈N/g⌉ prompts of up to g candidates, each sent as a separate request. Each full prompt contains q and an ordered list ℓ1:g of qualifications, and a fine-tuned LLM generates a comma-separated answer list such as YES, NO, YES. The i-th answer labels ℓi . Let o = (o1 , . . . , oT ) be the T generated tokens of the serialized answer list, including commas and the end-of-sequence token. For model parameters θ, the probability of the answer list factorizes autoregressively as Pθ (o | q, ℓ1:g ) =

T Y

Pθ (ot | q, ℓ1:g , o<t ),

(1)

t=1

where o<t = (o1 , . . . , ot−1 ) is the prefix before token t. Packing encodes the profile once per prompt and spreads request overhead across g decisions, but generating the answers jointly involves three design trade-offs: 1. Prompt dependence. Each answer is conditioned on every qualification in the prompt and on earlier answers (Equation (1)), so a judgment can vary with its companions, its position, or earlier answers. Section 6 measures decision flips under reordering. Cheng et al. (2023) likewise report that the number of samples per prompt affects accuracy. Training on shuffled lists of varying length exposes the model to different counts and orders but does not enforce invariant answers.

The surrounding workflow already resembles a ranking pipeline: retrieval supplies an ordered candidate list, and early stopping controls how much of that list is judged. The recipe keeps that pipeline and recasts the labeler as a pointwise ranker with q as the query and each ci as an item, changing one part of the labeler per stage (Table 1).

3

F ROM G ENERATION TO S EMANTIC S CORING

The first three stages of the recipe define the prompt, train the labeler, and convert its answer-token logits into a score. In our case study, each input pairs a profile with one qualification, and a reference answer, used only for training and validation, indicates whether the profile supports it. Semantic scoring assigns one score to each query–item pair [q; ci ] (Section 2.2), and MIS later computes many such scores together without changing them. 3.1

Prompt design

Decision contract. The prompt asks for a decision about a single candidate and states what counts as supporting evidence and how to treat missing or ambiguous information. Shared-prefix layout. Instructions, shared context, and other candidate-invariant text belong in q, while ci contains the candidate and the suffix leading to the answer position. For example, the field marker Qualification: can end q instead of repeating in every item, but candidate-dependent text must stay in ci , and moving this boundary requires revalidation.

From Generation to Scoring Table 1. Recipe stages: the change each makes to the packedgeneration labeler and the check the recipe calls for before the next stage. Stage

Change

Check before the next stage

Prompt (§3.1) Training (§3.2) Readout (§3.3) Inference (§4) Integration (§5)

One decision per prompt, shared text in q Reference label per item, label-only loss Score from YES/NO logits, tuned threshold τ MIS scores a round in one request Only the labeler changes, on both paths

Single-token answers, split tokens match Held-out quality on the served prompt Held-out FP/FN against references at chosen τ Matches single-item scores, N fits limits Same checkpoint, answer tokens, and τ on all paths

Answer position. The rendered prompt asks for a single answer, and a compact format removes the chat-template trailer that would otherwise repeat in every item, as in this synthetic example:

in validation and on both labeling paths, where its masking keeps each packed item isolated as in single-item training (Section 4). Model size is then chosen by held-out quality at the desired operating point and by cost (Section 6.5). Prompt–checkpoint contract. Training, validation, and serving use the same prompt renderer, answer-token mapping, and answer position, so a checkpoint is validated in the format it is served in. A compact prompt or a different readout token therefore needs matched fine-tuning and validation. 3.3

Score readout and validation

The single-item prompt [q; ci ] ends immediately before the answer, so a prefill pass provides the next-token logits zi at the item’s last token (⟨asst⟩ in the chat prompt, ⟨r⟩ in the compact prompt) without decoding. Renormalizing over the YES and NO answer tokens gives the score si =

Prompt

Query q ∥ item ci

Chat Profile: . . . ∥ Qualification: ICU nurse ⟨eot⟩⟨asst⟩ Compact Profile: . . . Qualification: ∥ ICU nurse ⟨r⟩

Here ⟨eot⟩⟨asst⟩ closes the user turn and opens the assistant turn, and ⟨r⟩ is a single readout token, an existing special token of the base model. The compact prompt shortens items from about 28 to about 24 tokens on average. The readout needs single-token answer labels in the rendered answer context, and tokenizing the query and item separately must yield the same tokens as the full single-item prompt. 3.2

Model training

Reference data. Training regroups the reference-labeled pairs used for packed generation into one qualification per example (Section 2.1). Each example (q, ci , ai ) carries a reference answer token ai ∈ {YES, NO}, with ai = YES when the reference judgment indicates that the profile information supports ℓi . The decision semantics are unchanged, so the labeler is trained on these hard labels through supervised fine-tuning (SFT). Members are split before pairing, so the member-disjoint holdout is unchanged. Model and objective. Fine-tuning starts from the base instruction-tuned model and minimizes the answer label’s negative log-likelihood, − log Pθ (ai | q, ci ), masking all other tokens. This standard causal-LM loss over the full vocabulary concentrates supervision on the qualification decision, and inference later renormalizes over the two answer tokens (Section 3.3). Single-item training removes answer-list length and formatting from the prediction target. Training does not use MIS: MIS is applied only at inference,

exp zi (YES) , exp zi (YES) + exp zi (NO)

(2)

and qualification ℓi is accepted when si ≥ τ . The default τ = 0.5 selects the more likely answer, with ties assigned to YES. Tuning τ on validation data selects the precision– recall operating point without retraining the labeler. Retained scores support calibration analysis and inspection of borderline cases, but two-token renormalization does not calibrate them. Each score depends only on [q; ci ], so it is invariant to other candidates and their order. Before enabling MIS, the recipe fixes τ on validation data and compares held-out agreement with reference labels against the packed-generation baseline on the same reference set (Section 6.2).

4

I NFERENCE WITH M ULTIPLE -I TEM S CORING

4.1

Mechanism

In the recipe’s inference stage, MIS computes all N scores of a round in one prefill pass while keeping each score si dependent only on [q; ci ] (Section 3). We adapt MIS, introduced for online LLM ranking (Li et al., 2026; Borisyuk et al., 2026) and contributed by us to FlashInfer and SGLang (Ye et al., 2025; Zheng et al., 2024), to semantic labeling. Each [q; ci ] uses the single-item prompt of Section 3.1. The engine prefills one physical token sequence x = [ q; c1 ; c2 ; . . . ; cN ],

(3)

with segment delimiters omitted from the notation. Segment boundaries come from the item lengths, which set the attention mask and position IDs. Let j and k be token indices

From Generation to Scoring

Position IDs encode token positions for the model, and unlike packed-sequence indices, they can be reused across items. Query tokens receive position IDs 0, . . . , Lq − 1, and the token at zero-based offset d in every item receives Lq +d, as in its single-item prompt. Every item therefore starts at position ID Lq , as if it immediately followed q in a separate request. Recovering independent scoring requires both the attention mask and these reused position IDs, because later items would otherwise have different positions relative to the query. Attention is restricted by the additive mask   if k ≤ j and k ∈ Q, 0 Mjk = 0 if k ≤ j and j, k ∈ Ii ,   −∞ otherwise.

1

c1

3

2

4

c2

5

4

c3

5

4

5

1 2 1 2 1 2 q1 q2 q3 q4 c1 c1 c2 c2 c3 c3

q1 q2 q3 q4 c11 c21

s1

c12 c22

s2

c13 s3

attended token k

(4)

Cost of scoring one query

Table 2 compares the work required to score the N items of a round for one query. Prefix caching can reuse the query only while its cache entries remain resident on the worker handling each request. Even with perfect reuse, the engine must schedule N sequences, maintain their metadata, and collect N outputs. MIS consolidates that work into one sequence with all N scores in one output. Generating a single answer token would also need only one prefill pass, so the decode savings in Table 2 come from removing the baseline’s multi-token answer list. 4.3

0

c23

The first case lets every token, in q or in any item, attend causally to the query tokens, so all items share q; the second lets item tokens attend causally within their own item ci . Each item attends only to q and its own causal prefix (Figure 2). Together, the mask and reused position IDs give the final token of ci the same context as in [q; ci ]. Equation (2) yields the independent score si up to floating-point differences, regardless of other items or their order; such differences can still flip a thresholded decision whose score lies very close to τ . Section 6.2 tests the resulting invariance to candidate order. 4.2

query q position id

attending token j

in the packed sequence x. Write Q for the set of indices occupied by q and Ii for those occupied by ci . These sets are contiguous and disjoint.

Cost model

We measure cost in prefill tokens per label, because MIS decodes nothing and long prefill-only requests saturate the GPU at the latency knee (Section 6.3). On the benchmark (Section 6.1), independent requests without reuse prefill about 770 tokens per label, packed generation about 63, before reuse of its cached instruction prefix, plus up to two decoded tokens, and MIS at N = 200 about 32. Packed generation thus takes about 2.0× the input tokens per label

Figure 2. MIS attention mask for a synthetic packed sequence x = [q; c1 ; c2 ; c3 ]: each item token attends only to q and its own causal prefix, and every item reuses the position IDs that follow q. Table 2. Work to score the N items of one query (g items per packed-generation prompt; delimiters and template tokens omitted). Packed gen. Independent + prefix cache MIS Requests ⌈N/g⌉ Query prefill ⌈N/g⌉Lq Decoded tokens Θ(N ) Cross-item attn. yes Output text

N N Lq 0 no score

N Lq to N Lq 0 no score

1 Lq 0 no score

of MIS, because MIS prefills the query once per round instead of once per prompt, and MIS also removes answerlist decoding. The measured 1.9× model-serving speedup is of similar size (Section 6.3), but throughput also depends on cache hits, batching, host-side work, and the engine, so the token ratio does not by itself explain it. More generally, for a request with N items, let L̄req = PN 1 i=1 Li be the mean item length and ρreq = Lq /L̄req N the query-to-item length ratio. Then the prefill-token reduction factor relative to independent scoring without prefix reuse is P N Lq + i Li N (1 + ρreq ) P R(N ) = = , (5) Lq + i Li N + ρreq which approaches 1 + ρreq as N grows. R(N ) is an exact ratio of prefill tokens, and compute follows it only up to constant factors. Feed-forward (MLP) and projection work scale with this token count. For items no longer than the query, attention work is O(L2q + N Lq L̄req ) rather than
O (Lq + N L̄req )2 for a fully causal packed sequence, because MIS computes the query’s self-attention once and

From Generation to Scoring

each item attends only to q and itself, and the FlashInfer kernels skip fully masked tiles. Section 6.4 compares R(N ) with measured speedups. 4.4

Query rows

Request sizing

Each item has the logical context [q; ci ], and the largest position ID in a packed request is Lq +maxi Li −1. The longest single-item prompt therefore determines the positional span. P The physical MIS request still contains Lq + i Li tokens, so it must satisfy the engine’s aggregate input-length limit and the request’s key–value (KV) cache budget. Splitting items across requests reduces aggregate length, but each split request prefills q again unless prefix caching reuses it. Large requests also leave less room for batching, which matters most for model serving for profile updates. In production, each retrieval round of Section 2.1 becomes one MIS request, so N is the round size. The benchmark in Section 6 uses N = 200 so that one MIS request scores the same candidates per member as packed generation’s ten prompts of g = 20.

5

bypassed for scoring

BATCH AND M ODEL -S ERVING I NTEGRATION

In the case study, MIS replaces only the labeler in offline batch inference and the streaming update path (Figure 3). Retrieval, rounds, early stopping, and the storedlabel schema are unchanged (Section 2.1), so downstream consumers need no interface changes. Each round becomes one request carrying the profile as the query and the N candidate IDs with their item text. It returns one score per candidate in the same order, thresholded as in Section 3.3. Both paths share the checkpoint, answer tokens, and threshold. The item layout is set per path: the offline benchmark uses the compact layout and model serving uses the chat layout (Section 3.1), so cross-path agreement depends on validating each layout. Offline. A scheduled workflow prepares one input row per member and round, then scores it in a Ray Data pipeline (Moritz et al., 2018) (Figure 3(a)). Separate autoscaling actor pools let CPU tokenization and GPU inference with SGLang scale independently. Ray Data’s open-source SGLang integration supported only text generation, which maps each input row to one prompt and one completion. We implemented a model-agnostic MIS scoring API for it, usable with any MIS-compatible SGLang model and proposed upstream, that maps each row to one shared query and its items, tokenizes them as separate segments, and returns one score per item in input order. Inputs already include the chat template, so the API skips further templating and detokenization. Completed rows are persisted, so an interrupted job resumes where it stopped. Round-level early stopping

(q, c1:N )

Chat template

Detokenize

Score tokenization CPU pool

SGLang + MIS GPU pool

Scores s1 , . . . , s N

skip committed rows on resume

Checkpoint

(a) Offline batch pipeline gRPC model server

Request (q, c1:N )

Servicer processes tokenize, pack

SGLang + MIS prefill-only

Scores s1 , . . . , s N

one request per query; KV cache freed after prefill

(b) Model serving for profile updates Figure 3. MIS changes only the scoring stage of both serving paths. (a) Offline: tokenization and scoring (blue) run in separate autoscaling actor pools and bypass the generation-only stages (dashed). (b) Profile updates: a multi-process gRPC server (Li et al., 2026) sends each packed request to a prefill-only SGLang engine.

follows Section 2.1, with only eligible members proceeding to the next round. Accepted labels are written to a key-value store for downstream reuse. Profile updates. Profile updates trigger a service that embeds the profile, retrieves candidates, and sends each round as a request to a prefill-only SGLang model server (Figure 3(b)), whose MIS support is available in open-source SGLang (Zheng et al., 2024). The server reuses components from prior LLM ranking systems (Behdin et al., 2026; Li et al., 2026). Servicer processes tokenize and pack each request with the mask in Equation (4), keeping request handling off the engine’s event loop. The engine returns item-aligned scores without sampling or decoding. It collects a request’s scores in one GPU-side gather, avoiding many small device-to-host copies, and frees the request’s KV cache once prefill completes. Because each round of candidates is processed as one sequence, the scheduler needs neither coordinated batching of separate item requests nor in-batch prefix caching to share the profile (Behdin et al., 2026).

6

E VALUATION

We evaluate the recipe on the case study through decision quality and efficiency: RQ1: How does semantic scoring compare with packed generation in agreement with reference labels, consistency under reordering, and training-data needs? RQ2: What throughput do the deployed generation

From Generation to Scoring

Setup

Workload. The throughput benchmark contains approximately 500K member profiles, each with 200 retrieved candidate qualifications, for a total of approximately 100M query–item pairs. The query averages 744 tokens (median 543, 95th percentile 2,047) and items average 28 tokens, so the query-to-item ratio of workload mean lengths is ρwork ≈ 27. Query lengths vary by more than an order of magnitude across members, whereas item lengths are nearly constant. Label quality is measured against reference judgments (Section 2.1) on a held-out set split by member, so no member contributes pairs to both training and evaluation. Both the training and held-out sets are fairly balanced, with a positive rate of about 50%. The metric thus measures agreement with these references, and catalog gaps, retrieval misses, qualifications skipped by early stopping, and product-level outcomes fall outside this evaluation (Section 9). Methods. Packed generation is the production baseline from Section 2.4, with g = 20 qualifications per prompt, about 1,300 tokens per prompt, and up to 40 greedily decoded tokens. Independent semantic scoring places one qualification in each request. It appears as the N = 1 point of RQ3, with cross-request prefix caching disabled. MIS packs N qualifications per request. With the chat-template item layout, a member’s request at N = 200 holds about 6,400 tokens, against 154,000 for independent requests without reuse. Using ρwork in Equation (5) gives the workloadlevel estimate R(200) ≈ 24. Unless noted, all methods run internally fine-tuned 1.5B-parameter decoder-only LLMs. Packed generation uses its answer-list fine-tune, and the scoring methods share the single-item fine-tune. Hardware and engines. Offline runs use one NVIDIA H100 GPU in the pipeline of Section 5, and model-serving benchmarks for profile updates use one NVIDIA H200 GPU, matching the production serving hardware. As in production, packed generation runs on vLLM (Kwon et al., 2023) with chunked prefill and prefix caching. Independent scoring and MIS run on SGLang with FlashInfer kernels (Zheng et al., 2024; Ye et al., 2025), because MIS is supported in SGLang but not in vLLM. 6.2

RQ1: Quality

We evaluate output quality as agreement with the reference labels and consistency when qualifications are reordered. Semantic scoring agrees well with the references with little

Agreement

6.1

90% 80% 70% 60%

Semantic scoring (MIS) Packed generation (g = 20)

50%

45K

89K

220K

443K

Training data size

Figure 4. Agreement with reference labels against training-data size (profile–qualification pairs) for semantic scoring and packed generation.

Inconsistency rate

and scoring stacks achieve under matched workload and latency requirements? RQ3: How does MIS throughput vary with request size? RQ4: (Exploratory) How much further can a smaller scoring labeler raise throughput?

10% 1% Semantic scoring (MIS)

0.1%

Packed generation (g = 20)

0.01% 45K

89K

220K

443K

877K

Training data size

Figure 5. Inconsistency rate (share of labels that change when a member’s qualifications are reordered and regrouped, log scale) against training-data size.

training data and is consistent at every training size, whereas packed generation needs several times more data to match its agreement and never matches its consistency. Semanticscoring checkpoints are trained pointwise and evaluated with MIS inference. Agreement with reference labels. Agreement is the percentage of predicted qualification labels that match the reference labels of Section 6.1. We train semantic-scoring and packed-generation labelers at several training-data sizes, and Figure 4 plots their agreement. Semantic scoring reaches 86.3% with 45K training pairs. Packed generation improves more slowly: at 45K pairs it reaches 50.8%, near the 50% positive rate of this balanced set, and it needs about 210K pairs, interpolating between measured sizes, or about 4.7× as much data, to match semantic scoring’s agreement at 45K. At the largest size with agreement results, 443K pairs, the two are comparable, at 88.0% and 87.8%; we make no significance claim for this difference. Consistency. Consistency rate is measured as the percentage of qualification labels that stay the same when a member’s qualifications are randomly reordered and regrouped

From Generation to Scoring

during evaluation, with the profile and qualification text held fixed. Figure 5 plots the inconsistency rate, the share of labels that change across orderings, for the same training sizes plus an 877K set, for which agreement is not reported. Semantic scoring keeps it at or below 0.01% at every size; the mechanism predicts no change, so this residual reflects floating-point differences in batched execution (Section 4.1). Packed generation changes 24.8% of labels at the smallest training size and, although it improves with more data, still changes 0.86% at the largest, at least 86× the rate of semantic scoring. A possible explanation. Both formulations are fine-tuned on the same reference-labeled pairs and differ only in how pairs are grouped into training examples (Section 3.2). A packed-generation example supervises both the individual judgments and the output format: the model must emit a complete answer list whose length and order match its candidates. Semantic scoring supervises a single answer token per pair, so list length, formatting, and alignment never enter the target, and each decision is independent of candidate order by construction. This is consistent with both gaps above: packed generation likely spends its early training budget on the list format, leaving agreement near the positive rate at the smallest size, and its decisions remain sensitive to candidate position at every size. Semantic scoring has a training-cost trade-off: single-item examples repeat the profile for every pair, so training processes more tokens per labeled pair, and MIS recovers this sharing only at inference. 6.3

RQ2: Throughput

Protocol. Throughput is measured in labels, i.e., scored query–item pairs, per second per GPU on the same candidate workload. Offline, each method runs at its best batch size and concurrency. For model serving for profile updates, every method is measured at the knee: the highest Poissonarrival request rate that holds the 99th-percentile end-to-end request latency at or below one second with at least 99% success. Each rate runs for 60 seconds after a 200-request warmup on disjoint client and server CPU sets, with the model server restarted so every run begins with an empty prefix cache. Each member’s ten packed-generation prompts are separate requests that may reach different hosts, so fanout limits cache reuse to the shared instruction prefix. MIS issues one request per member and is insensitive to routing. Model serving for profile updates. At the one-second p99 target, the MIS-based stack at N = 200 sustains about 4.8K labels/s per GPU against 2.5K for packed generation, a 1.9× speedup at equal latency (Table 3). At the knee, MIS is latency-bound and the GPU is saturated by long prefill-only requests. The gain is consistent with removing repeated query prefill, per-request overhead, and answer-

Table 3. Throughput of the deployed inference stacks, in labels per second per GPU, with the ratio to packed generation in parentheses. Offline (H100): steady state at the best batch size and concurrency; profile updates (H200): highest arrival rate with p99 latency ≤1 s. The stacks differ in engine, checkpoint, prompt format, and runtime tuning as well as method. Method Packed generation (g = 20) MIS (N = 200)

Offline

Profile updates

5.1K (1.0×) 6.3K (1.24×)

2.5K (1.0×) 4.8K (1.9×)

list decoding, but because the comparison changes both the formulation and the inference stack, we cannot quantify each factor’s contribution. Offline batch inference. On the 500K-member job, the MIS-based stack processes about 6.3K labels/s per GPU against 5.1K for packed generation, a 1.24× throughput ratio (Table 3). Reaching this rate required tuning the offline pipeline, not only changing the method. Our first MIS run, with the chat-template layout and a single engine process, reached only 4.4K labels/s, below packed generation, and left the GPU partly idle, which points to host-side work such as tokenization, packing, and result handling rather than GPU compute. The reported run uses the compact readout-token layout of Section 3.1, which shortens each item, and runs two SGLang engine processes that share the GPU through CUDA Multi-Process Service (MPS), so one engine’s host-side work overlaps the other’s GPU work. Together with smaller per-engine batches, these changes raise offline MIS throughput by 44%. The offline ratio is lower, likely because packed generation amortizes decoding across large batches without a latency target, and unchanged input and output stages take part of the job time. Both offline numbers are steady-state throughput, excluding job start-up. The MIS-based stack thus raises per-GPU throughput in both settings under the benchmark protocol, most under a latency target. 6.4

RQ3: Request size

RQ2 reports MIS at N = 200 only, and Figure 6 shows how its speedup changes from single-item scoring (N = 1) to larger requests, against the token-count ratio R(N ). In model serving for profile updates, the sweep varies N from 1 to 200 for the 1.5B checkpoint under the protocol of Section 6.3, with cross-request prefix caching disabled so that each request prefills its own profile. Requests grow from about 770 tokens at N = 1 to about 6,400 at N = 200. Throughput rises from about 90 labels/s per GPU at N = 1 to 2.0K at N = 20, 3.9K at N = 50, and 4.8K at N = 200, so MIS overtakes packed generation (2.5K) between N = 20 and N = 50. Figure 6 also shows the exploratory

Speedup vs. packed gen.

From Generation to Scoring

7

MIS, 0.6B

MIS, 1.5B

R(N ), scaled (1.5B)

Packed gen. (g=20)

4×

Previously, packed generation ran on vLLM, using H100 GPUs for daily incremental and weekly offline runs and H200 GPUs in multiple regions for model serving for profile updates, including multilingual profiles.

2× 1× 0.5× 0.2× 0.1× 0.05×

1

5

20

50

100

200

Items per request (N )

Figure 6. Model-serving speedup over 1.5B packed generation versus items per request for MIS with the 1.5B checkpoint (chat format) and the exploratory 0.6B checkpoint (readout-token format, Section 6.5), on one H200 at the one-second p99 knee without cross-request prefix caching. The dotted curve is R(N ) (Equation (5)) scaled to the 1.5B N = 1 speedup. The two curves differ in model, checkpoint, and layout, so they do not isolate model size.

0.6B labeler (Section 6.5), which follows the same shape at higher throughput. Splitting a member’s 200 candidates into two N = 100 requests lowers throughput by about 14%, to 4.1K labels/s, because the split repeats the profile prefill and adds requests. The measured speedup exceeds the R(N ) curve scaled to its N = 1 point, possibly because packing also amortizes scheduling and output work across items. 6.5

P RODUCTION D EPLOYMENT AND I MPACT

RQ4: A smaller decision model (exploratory)

Semantic scoring reads a single per-item decision instead of a generated answer list (Section 3.3), so a good decision model, rather than a strong completion model, may suffice. We explore this with an internally fine-tuned 0.6B-parameter labeler on the readout-token layout of Section 3.1. In exploratory runs, its label quality was comparable to the 1.5B scoring labeler’s; we do not report these metrics and make no general claim about the size–quality trade-off, so we report only throughput here. The smaller labeler raises throughput on both paths. For profile updates, under the protocol of RQ2, MIS with the 0.6B labeler sustains about 10.9K labels/s per GPU at N = 200 (Figure 6), 4.4× packed generation and 2.3× the 1.5B labeler, a ratio that also reflects checkpoint and layout differences. Offline, with the configuration of RQ2, it reaches an estimated 11.3K labels/s per GPU, 2.2× packed generation, although this run’s prompts did not use the model’s own chat template, so it measures throughput only. These results suggest that MIS and a smaller decision model can be combined, pending a full quality evaluation.

MIS deployment. Offline incremental runs and periodic refreshes (e.g., after a model or catalog update) keep the same schedule and completion window. Both paths now run the MIS-based scoring stack. The benchmarks of Section 6.3 use the production hardware, engine configuration, and pipeline, so their measured throughput implies about 20% fewer GPU-hours for the labeling stage of each offline run. MIS does not accelerate the retrieval, promptpreparation, and output stages of each run, so end-to-end offline savings are smaller. On the streaming update path, it implies about 50% fewer provisioned GPUs at the same peak update rate and latency target, assuming capacity scales with per-GPU throughput.

8

R ELATED W ORK

Answer-token readout, hard-label fine-tuning, and MIS, including its candidate-isolating mask and reused position IDs, have precedents; our contributions are combining them to move a production labeler from generation to scoring on both labeling paths and implementing MIS-based scoring as a reusable Ray Data batch-inference API. Catalog-grounded semantic labeling. Systems that understand job postings map posting text to standardized attributes through candidate generation and semantic selection. Li et al. (2020) generate taxonomy-based entity candidates and use learned ranking to select the most important standardized entities for a job. Xu et al. (2026) retrieve attribute candidates using embeddings and apply small-languagemodel classification to select the corresponding values, with occupation as the worked large-cardinality example. Our profile labeler follows the same retrieve-then-label structure but assigns each candidate qualification an independently thresholded support score, so one profile can receive multiple qualifications. Packed generation batches these decisions into one answer list, whereas we share computation with MIS while isolating predictions. Pointwise and listwise judgments. Pointwise ranking with sequence-to-sequence models (Nogueira et al., 2020) formulates document ranking as relevance-label prediction and uses the answer-token probabilities as relevance scores. Semantic scoring applies this readout to binary qualification judgments with a causal language model (Section 3). Sun et al. (2023), by contrast, generate an ordering after seeing

From Generation to Scoring

candidates together, making cross-candidate comparisons part of the task. Batch prompting (Cheng et al., 2023) packs separate samples into one prompt, whereas packed generation packs one entity’s candidates, and both couple answers that qualification labeling needs to keep independent.

or numerical parity across serving paths. Throughput comparisons are stack-level, so we do not attribute the gains to MIS alone (Section 6.3). The smaller-model results are exploratory, and how small a scoring labeler can be without losing quality remains open (Section 6.5).

Shared-context execution. Our prior work introduced MIS in MixLM (Li et al., 2026) and used it in semantic search (Borisyuk et al., 2026), sharing query computation across independent item scores to rank candidates for a live request. We instead threshold scores into stored labels in offline batch inference and the streaming update path, which requires a generative labeler to become a scorer (Section 3). Our recent article describes the SGLang scoring API for pointwise decision models (Ramachandran et al., 2026).

Deployment lessons. First, the prompt, the checkpoint, the answer-token mapping, and the threshold form one versioned artifact across both paths, so changing the prompt means retraining and revalidating (Table 1). Second, MIS makes shared-context reuse explicit within each request, so its benefit does not depend on cache residency or request routing (Section 6.3). Finally, request size trades amortized query work against latency and batching headroom (Section 6.4).

Prefix caching (Kwon et al., 2023; Zheng et al., 2024; Gim et al., 2024) and shared-prefix attention kernels such as Hydragen (Juravsky et al., 2024), ChunkAttention (Ye et al., 2024), and FlashInfer’s cascade attention (Ye et al., 2025) reuse or speed up query computation without coupling candidate judgments, but each candidate remains a separate sequence. MIS instead shares the query inside one packed sequence, relying on candidate-isolating attention masks in FlashInfer. Parallel Context Windows (Ratner et al., 2023) similarly mask attention between context segments with reused positions, but to extend the context of one generated answer rather than to read out independent per-item scores.

9

D ISCUSSION AND L IMITATIONS

Applicability. Semantic scoring with MIS applies when many candidates share one context, each decision depends only on that context and its candidate, and one answer position yields the score. Candidate tasks, none evaluated here, include taxonomy tagging, checking content against independent policies (Inan et al., 2023), search relevance judgments (Thomas et al., 2024), and skill tagging of course transcripts (Yan et al., 2023). The recipe does not apply to listwise reranking (Sun et al., 2023) or joint deduplication, whose decisions require the cross-candidate interactions that the MIS mask blocks. Normalizing over single-token class labels could extend the readout to graded or multi-class decisions, though we evaluate only binary labels. Limitations. We measure quality as agreement with reference judgments on the candidates the retriever surfaces (Sections 2.1 and 6.1), not as product-level outcomes. The catalog, retriever, and early-stopping rule are unchanged by this work, so an absent label can mean either a negative prediction or a qualification that was never evaluated. We evaluate the recipe end to end: an inconsistency rate of at most 0.01% under reordering bounds MIS’s sensitivity to packing (Section 6.2), but we do not isolate per-stage contributions

10

C ONCLUSION

We presented an end-to-end recipe that reformulates a generative YES/NO labeler as semantic scoring through singlecandidate prompts, label-only training, answer-token readout, and prefill-only execution with MIS. In our case study, it changes at most 0.01% of labels under reordering, matches packed generation’s agreement with reference labels with about 4.7× fewer training pairs, and, as a stack-level comparison, raises throughput by 1.24× in offline batch inference and 1.9× in model serving for profile updates. For offline batch inference, we implement MIS-based scoring as a reusable Ray Data API, proposed upstream; model serving uses MIS in SGLang. More broadly, industrial LLM labelers need not inherit a generative interface: when decisions are independent, the recipe separates the semantic contract, judging each candidate alone, from the systems optimization that executes many such judgments together.

R EFERENCES Behdin, K., Song, Q., Vasudevan, S., Sheng, J., Ma, X., Zhou, Z., Zhu, C., Li, G., Nguyen, C., Ghosh, S., Sang, H., Baarzi, A. F., Ramachandran, S. R., Wang, X., Lan, Q., S, V. Y., Guo, Q., Johnson, C., Wang, Z., and Borisyuk, F. Scaling up large language models serving systems for semantic job search. In Proceedings of the 9th Conference on Machine Learning and Systems (MLSys), 2026. Industry Track. Borisyuk, F., Vasudevan, S., Wu, M., Li, G., Le, B., Zhang, S., Shen, Q. K., Juan, Y., Behdin, K., Dong, L., Yang, K., Jing, S., Pothamsetty, R., Arora, R., Sheng, S. Y., Abdrashitov, V., Zhao, Y., Su, L., Wang, X., Zheng, C., Metkar, S., Gupta, R., Lapchuk, I., Racca, D. N., Mohan, M., Li, Y., Li, H., Gandhi, S., Lu, X., Bhole, C., Hooshmand, A., Yang, X., Muthuregunathan, R., Zhang, J., Teoh, M., Coler, A., Gupta, A., Ma, X.,

From Generation to Scoring

Ramachandran, S. R., Ramezani, M., Wang, Y., Zhang, L., Li, R., Sheng, J., Nguyen, C., Chen, Y.-C., Zhu, C., Zhang, C., Xu, J., Kulkarni, D., Lan, Q., Subramaniam, A., Fatahibaarzi, A., Shimizu, S., Chen, Y., Wang, Z., He, R., Zhou, Z., Song, Q., Dai, Y., Johnson, C., Liu, P., Gharghabi, S., Mohanasundaram, G., Bottaro, J., Sachindran, S., Guo, Q., Ren, Y., Jiang, C., Mo, D., Simon, L., Shen, J., Wu, J., and Zhang, W. Semantic search at LinkedIn. In Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM), 2026. Accepted; to appear. arXiv:2602.07309. Cheng, Z., Kasai, J., and Yu, T. Batch prompting: Efficient inference with large language model APIs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track (EMNLP), pp. 792–810, 2023. doi: 10.18653/v1/2023.emnlp-industry.74. Gilardi, F., Alizadeh, M., and Kubli, M. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences (PNAS), 120(30):e2305016120, 2023. doi: 10.1073/pnas.2305016120. Gim, I., Chen, G., Lee, S.-s., Sarda, N., Khandelwal, A., and Zhong, L. Prompt cache: Modular attention reuse for low-latency inference. In Proceedings of Machine Learning and Systems (MLSys), 2024. Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., and Khabsa, M. Llama Guard: LLM-based input-output safeguard for human-AI conversations. arXiv:2312.06674, 2023. Juravsky, J., Brown, B., Ehrlich, R., Fu, D. Y., Ré, C., and Mirhoseini, A. Hydragen: High-throughput LLM inference with shared prefixes. arXiv:2402.05099, 2024. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), pp. 611–626, 2023. doi: 10.1145/3600006.3613165. Li, G., He, R., Jing, S., Behdin, K., Wang, Y., Ramachandran, S. R., Nguyen, C., Sheng, J., Ma, X., Zhu, C., Vasudevan, S., Wu, M., Ghosh, S., Song, Q., Wang, X., Wang, Z., Lan, Q., Chen, Y., Wu, J., Simon, L., Zhang, W., Guo, Q., and Borisyuk, F. MixLM: High-throughput and effective LLM ranking via text-embedding mix-interaction. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge

Discovery and Data Mining V.2 (KDD), pp. 7588–7599, 2026. doi: 10.1145/3770855.3818448. Li, S., Shi, B., Yang, J., Yan, J., Wang, S., Chen, F., and He, Q. Deep job understanding at LinkedIn. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 2145–2148, 2020. doi: 10.1145/3397271.3401403. Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., and Stoica, I. Ray: A distributed framework for emerging AI applications. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 561–577, 2018. Nogueira, R., Jiang, Z., Pradeep, R., and Lin, J. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 708–718, 2020. doi: 10.18653/v1/2020.findings-emnlp.63. Ramachandran, S. R., Zhu, C., Lan, Q., Vasudevan, S. R., Sheng, J., Chen, I.-T., and Borisyuk, F. Scaling JEV-like decision models with SGLang. LMSYS Org blog, https://www.lmsys.org/blog/2026-0925-sglang-decision-models, 2026. Accessed 2026-10-04. Ratner, N., Levine, Y., Belinkov, Y., Ram, O., Magar, I., Abend, O., Karpas, E., Shashua, A., Leyton-Brown, K., and Shoham, Y. Parallel context windows for large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023. Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., and Ren, Z. Is ChatGPT good at search? Investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 14918–14937, 2023. doi: 10.18653/v1/2023.emnlp-main.923. Thomas, P., Spielman, S., Craswell, N., and Mitra, B. Large language models can accurately predict searcher preferences. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 1930–1940, 2024. doi: 10.1145/3626772.3657707. Xu, D., Zheng, B., Shen, J., Xiao, Q., Le, B. H., Pu, W., Gupta, S., Zhou, R., Saraf, N., Leung, A., Shen, Q., Hong, L., Wu, J., and Zhang, W. Unified semantic modeling framework for large-scale job understanding at LinkedIn. In Proceedings of the 49th International ACM

From Generation to Scoring

SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 5015–5020, 2026. doi: 10.1145/3805712.3808402. Yan, J., Macskassy, S., Sun, L., Zhou, D., Kou, R., and Li, Z. Extracting skills from content to fuel the LinkedIn Skills Graph. https://www.linkedin.com/ blog/engineering/skillsgraph/extracting-skills-from-content, 2023. Accessed 2026-09-30. Ye, L., Tao, Z., Huang, Y., and Li, Y. ChunkAttention: Efficient self-attention with prefix-aware KV cache and two-phase partition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 11608–11620, 2024. doi: 10.18653/v1/2024.acl-long.623. Ye, Z., Chen, L., Lai, R., Lin, W., Zhang, Y., Wang, S., Chen, T., Kasikci, B., Grover, V., Krishnamurthy, A., and Ceze, L. FlashInfer: Efficient and customizable attention engine for LLM inference serving. In Proceedings of the 8th Conference on Machine Learning and Systems (MLSys), 2025. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (NeurIPS), Datasets and Benchmarks Track, pp. 46595–46623, 2023. doi: 10.52202/075280-2020. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems 37 (NeurIPS), pp. 62557–62583, 2024. doi: 10.52202/079017-2000.

Related documents

• From Generation to Scoring: Scaling LLM-Based Semantic Labeling at LinkedIn#1467707
• LRanker: Scaling LLM Ranking to Millions of Candidates with Test-Time Scaling#1467162
• Forty Questions: Identifying a Language Model from One-Word Answers#1412252
• SPADER: Step-wise Peer Advantage with Diversity-Aware Exploration Rewards for Multi-Answer Question Answering#1155222
• Beyond Raw Context Transfer: Representation-based Federated Retrieval-Augmented Generation#1036845