EPM-RL: Reinforcement Learning for On-Premise Product Mapping in E-Commerce Minhyeong Yu
Wonduk Seo*
[email protected] AI Research, Enhans Seoul, South Korea
[email protected] AI Research, Enhans Seoul, South Korea
arXiv:2604.23993v1 [cs.CL] 27 Apr 2026
Abstract Product mapping—the task of deciding whether two e-commerce listings refer to the same product—is a core problem for price monitoring and channel visibility. In real marketplaces, however, sellers frequently inject promotional keywords, platform-specific tags, and bundle descriptions into titles, causing the same product to appear under many different names. Recent LLM-based and multiagent frameworks improve robustness and interpretability on such hard cases, but they often rely on expensive external APIs, repeated retrieval, and complex inference-time orchestration, making large-scale deployment costly and difficult in privacy-sensitive enterprise settings. To address these issues, we present EPM-RL, a reinforcement-learning-based framework for building an accurate and efficient on-premise e-commerce product mapping model. Our central idea is to distill high-cost agentic reasoning into a trainable in-house model. Starting from a curated set of product pairs with LLM-generated rationales and human verification, we first perform parameter-efficient fine-tuning (PEFT) on a small student model using structured reasoning outputs. We then further optimize the model with Reinforcement Learning (RL) using an agentbased reward that jointly evaluates output-format compliance, label correctness, reasoning–preference scores from specially designed judge models. Preliminary results show that EPM-RL consistently improves over PEFT-only training and offers a stronger quality– cost trade-off than commercial API-based baselines, while enabling private deployment and lower operational cost. These findings suggest that reinforcement learning can turn product mapping from a high-latency agentic pipeline into a scalable, inspectable, and production-ready in-house system.
Keywords Product Mapping, E-Commerce, Reinforcement Learning, Large Language Models, Parameter-Efficient Fine-Tuning, LLM-as-a-Judge, On-Premise AI *denotes corresponding author. Authors’ Contact Information: Minhyeong Yu, [email protected], AI Research, Enhans, Seoul, South Korea; Wonduk Seo*, [email protected], AI Research, Enhans, Seoul, South Korea.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 2831-3194/2018/8-ART111 https://doi.org/XXXXXXX.XXXXXXX
ACM Reference Format: Minhyeong Yu and Wonduk Seo*. 2018. EPM-RL: Reinforcement Learning for On-Premise Product Mapping in E-Commerce. ACM/IMS J. Data Sci. 37, 4, Article 111 (August 2018), 8 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
Product mapping is the task of deciding whether two e-commerce listings refer to the same underlying product [1, 15, 25, 30]. It underpins many downstream applications, including price monitoring, seller and channel visibility, catalog integration, and duplicate detection [11, 28]. In real marketplaces, however, the task is far from trivial: sellers often append promotional keywords, platform-specific tags, and bundle descriptions to titles, causing the same product to appear under many different names across platforms. Conversely, listings with highly similar titles can correspond to different products due to subtle differences in quantity, option type, or variant information. These challenges make robust product mapping both practically important and technically difficult [16, 22, 25]. Earlier approaches typically treated product mapping as entity matching. Traditional pipelines relied on string normalization, rulebased heuristics, token overlap, hand-crafted features, or pairwise classifiers trained on title pairs. Neural encoders later improved semantic matching by learning similarity directly from labeled data [8, 12, 21]. However, in realistic settings where titles are noisy and incomplete, these methods still struggle on hard cases that require reasoning over product attributes and commercial context rather than surface-level similarity [25]. Large language models (LLMs) offer a complementary approach [2, 6, 27, 31]. Zero-shot and few-shot prompting can capture semantic correspondence beyond token overlap [29], while retrievalaugmented methods can ground decisions in external evidence [10, 17]. Multi-agent pipelines further improve robustness by decomposing the decision into specialized sub-tasks (e.g., evidence gathering, attribute comparison, and decision aggregation) [18, 25, 26]. Despite their strong performance, these systems introduce practical drawbacks: they often depend on repeated calls to expensive external APIs, require retrieval and multi-step orchestration at inference time (increasing latency and complexity) [5, 18], and are difficult to deploy in privacy-sensitive environments that require on-premise operation and predictable cost [9, 14]. To address these issues, we propose EPM-RL, a reinforcementlearning-based framework for accurate and efficient on-premise product mapping. Our key idea is to distill high-cost agentic reasoning into a trainable in-house model. Starting from a human-labeled dataset of product pairs, we first apply parameter-efficient finetuning (PEFT) [7, 13, 20] with structured reasoning traces so the model learns interpretable comparison behavior. We then refine the
Conference’17, July 2017, Washington, DC, USA
Trovato et al.
model with reinforcement learning [19, 23] using verifiable rewards and judge-based signals produced by three specialized agents, and further incorporate unlabeled data to improve reasoning quality under an LLM-as-a-judge objective. In this way, EPM-RL internalizes structured reasoning while avoiding inference-time agent orchestration. Experiments on an internal product mapping benchmark show that EPM-RL consistently improves over PEFT-only training and offers a better quality–cost trade-off than commercial API-based baselines. In particular, it achieves stronger performance on difficult product pairs while remaining suitable for scalable private deployment. These results suggest that reinforcement learning can turn product mapping from a high-latency agentic pipeline into a production-ready, inspectable, and cost-efficient in-house system. The main contributions of this paper are as follows: • We propose EPM-RL, an on-premise reinforcement-learning framework for scalable e-commerce product mapping. • We introduce a practical training pipeline that combines PEFT, RL with verifiable and agentic rewards, and additional unlabeleddata optimization with structure and judge-based supervision. • We show that EPM-RL improves over PEFT-only training and provides a stronger quality–cost trade-off than commercial APIbased baselines on internal benchmarks.
2
Background
Task Description. Product mapping in e-commerce aims to determine whether a base product listing and a compared product listing refer to the same underlying real-world product. Formally, given a base product 𝑝𝑏 and a compared product 𝑝𝑐 , the goal is to predict a binary label 𝑦 ∈ {0, 1}, where 𝑦 = 1 indicates that the two listings correspond to the same product and 𝑦 = 0 otherwise. Although the formulation is simple, the task is challenging in real marketplaces because product titles are often noisy, incomplete, and optimized for search visibility rather than standardized description. Sellers may inject promotional keywords, platform-specific tags, origin descriptors, or bundle information, causing the same product to appear under different names or, conversely, making different products appear deceptively similar. Table 1 illustrates these cases: in the positive examples, the compared listing adds non-essential descriptors (e.g., origin or packaging) while preserving the core product identity; in the hard negative examples, the titles overlap heavily but differ in bundle composition or variant attributes, leading to a different underlying product. These examples illustrate why effective product mapping requires reasoning over attributes such as brand, variant, specification, quantity, and bundle structure rather than relying solely on surface-level lexical overlap. LLM-based Product Mapping. Large language models (LLMs) are well suited for product mapping because they produce outputs autoregressively, assigning a probability to each next token conditioned on the input. Let 𝑥 = [𝑝𝑏 ; 𝑝𝑐 ] denote the concatenated input pair, and let 𝑜 = (𝑜 1, . . . , 𝑜 |𝑜 | ) denote the generated output sequence. An autoregressive LLM with parameters 𝜃 defines 𝑃𝜃 (𝑜 | 𝑥) =
|𝑜 | Ö 𝑡 =1
𝑃𝜃 (𝑜𝑡 | 𝑥, 𝑜 <𝑡 ).
(1)
Table 1: Illustrative examples of product pairs in e-commerce product mapping. Case
Base Product
Compared Product
MegaDoseD Vitamin D3 4000IU, 120 tablets × 3 Apple AirPods Pro (2nd Gen) with MagSafe Case
MegaDoseD Vitamin D3 4000IU Swiss-made, 120 tablets × 3 Apple AirPods Pro 2 (2nd generation) MagSafe Charging Case
Positive (𝑦=1)
Hard Negative (𝑦=0) Coca-Cola Zero 355mL × 24 cans Nespresso Vertuo Espresso Pods Variety Pack, 30 count
Coca-Cola Zero 355mL × 24 cans + cooler bag gift set Nespresso Vertuo Espresso Pods Variety Pack, 60 count
In practice, there are two common ways to use this token-level distribution for binary product mapping. (1) Direct classification by label logits. The model is prompted to output a single label token (e.g., 0 for non-match and 1 for match). Let 𝑧 0 (𝑥) and 𝑧 1 (𝑥) denote the pre-softmax logits for the first generated label token. The prediction is obtained by 𝑦ˆ = arg max 𝑧 𝑦 (𝑥)
𝑦ˆ = arg max 𝑃𝜃 (𝑦 | 𝑥).
⇔
𝑦 ∈ {0,1}
𝑦 ∈ {0,1}
(2)
This approach is simple and efficient but provides no structured explanation. (2) Reasoning-then-label generation. Alternatively, the model generates an explicit reasoning trace 𝑟 = (𝑟 1, . . . , 𝑟𝑇 ) followed by a final label 𝑦. The joint probability factorizes as ! 𝑇 Ö 𝑃𝜃 (𝑟, 𝑦 | 𝑥) = 𝑃𝜃 (𝑟𝑡 | 𝑥, 𝑟 <𝑡 ) 𝑃𝜃 (𝑦 | 𝑥, 𝑟 ), (3) 𝑡 =1
and the predicted label is extracted from the generated output, e.g., 𝑦ˆ = parse arg max 𝑃𝜃 (𝑜 | 𝑥) , (4) 𝑜
where parse(·) returns the value inside the <label>. . . </label> field. In EPM-RL, we adopt the reasoning-then-label formulation with a fixed structured format (<reason>. . . </reason><label>. . . </label>). This choice improves interpretability by exposing an explicit comparison rationale, enables supervised learning of structured reasoning traces during PEFT, and supports fine-grained reward shaping in GRPO that separately evaluates format compliance, label correctness, and judge-based reasoning quality.
3 Methodology 3.1 Data Preprocessing. Our raw dataset consists of triplets (𝑝𝑏 , 𝑝𝑐 , 𝑦), where 𝑝𝑏 denotes the base product title, 𝑝𝑐 denotes the compared product title, and 𝑦 ∈ {0, 1} is the human-provided binary label indicating whether the two listings refer to the same underlying product. We first partition the dataset into four disjoint subsets for PEFT training, reinforcement learning, validation, and testing. To further support reasoning-aware training, we construct an additional reflective reasoning dataset from the PEFT split. Specifically, given (𝑝𝑏 , 𝑝𝑐 , 𝑦), we prompt a strong LLM to generate a reasoning trace that explains why the pair should be judged as matched or unmatched, while
EPM-RL: Reinforcement Learning for On-Premise Product Mapping in E-Commerce
Conference’17, July 2017, Washington, DC, USA
assuming a blinded setting in which the model must reason only from the information available in the two product titles. This reverse generation process produces structured reasoning traces aligned with the human label and yields augmented tuples (𝑝𝑏 , 𝑝𝑐 , 𝑟, 𝑦), where 𝑟 denotes the synthesized reflective reasoning.
count, option, or bundle differences. Denoting these scores by 𝑠 ver , 𝑠 core , 𝑠 id , and 𝑠 var , respectively, the overall reward is defined as
3.2
Parameter-Efficient Fine-Tuning.
After constructing the reasoning-augmented data, we format each training instance as an input-output pair. The input contains the base prompt together with the base product and compared product, while the target output contains both the reasoning trace and the final binary label in a structured format, namely <reason>...</reason><label>...</label>. This format encourages the model to learn not only the final decision but also the intermediate comparison logic. We then optimize the model with parameter-efficient fine-tuning (PEFT) [13], updating only a small subset of trainable parameters while keeping the backbone model largely frozen. Let 𝑥 = [𝑝𝑏 ; 𝑝𝑐 ] denote the input and let 𝑜 = (𝑟, 𝑦) denote the target output sequence. The PEFT objective follows the standard autoregressive language modeling loss: LPEFT (𝜃 ) = −
𝑇𝑖 𝑁 ∑︁ ∑︁
log 𝑃𝜃 𝑜𝑖,𝑡 | 𝑥𝑖 , 𝑜𝑖,<𝑡 ,
(5)
𝑖=1 𝑡 =1
where 𝜃 denotes the trainable PEFT parameters, 𝑁 is the number of training instances, and 𝑇𝑖 is the output length of the 𝑖-th example. Through this stage, the model learns to generate reasoning traces and labels jointly from product-pair inputs.
3.3
Reinforcement Learning with GRPO.
Although PEFT teaches the model a structured reasoning format, it does not explicitly optimize the quality of the generated reasoning or its alignment with the final decision. To address this, we further refine the model with Group Relative Policy Optimization (GRPO) [19]. Given an input 𝑥, the current policy 𝜋𝜃 generates a group of 𝐾 rollout responses {𝑜 (1) , . . . , 𝑜 (𝐾 ) }. Each rollout is assigned a scalar reward based on both verifiable signals and LLM-based judge signals, and the policy is updated to increase the likelihood of better-performing rollouts relative to others in the same group. Let 𝑟 (𝑘 ) denote the reward for the 𝑘-th rollout and let 𝐴ˆ (𝑘 ) denote its normalized relative advantage within the group. The GRPO objective can be written as " # 𝐾 1 ∑︁ ˆ (𝑘 ) (𝑘 ) LGRPO (𝜃 ) = −E𝑥 𝐴 log 𝜋𝜃 𝑜 |𝑥 . (6) 𝐾
𝑅(𝑜, 𝑥) = 𝜆ver𝑠 ver + 𝜆core𝑠 core + 𝜆id𝑠 id + 𝜆var𝑠 var,
(7)
where 𝜆ver , 𝜆core , 𝜆id , and 𝜆var are weighting coefficients. This design allows the model to improve not only final classification accuracy but also the quality and faithfulness of its reasoning process.
4 Experiments 4.1 Setup 4.1.1 Dataset Used. We evaluate our framework on an internally collected and curated product mapping dataset consisting of approximately 12K labeled product pairs. Each instance contains a base product, a compared product, and a binary human label indicating whether the two listings refer to the same underlying product. The dataset covers approximately 500 brands collected from diverse e-commerce platforms, reflecting substantial variation in naming conventions, promotional expressions, bundle descriptions, and platform-specific title formats. This diversity makes the benchmark representative of realistic product-mapping scenarios encountered in practice. To support different stages of training and evaluation, we partition the dataset into four disjoint splits: • 6K examples for parameter-efficient fine-tuning (PEFT), • 4K examples for reinforcement learning (RL), • 1K examples for validation, • 1K examples for testing. The splits are constructed using stratified sampling to preserve both brand diversity and label balance across subsets, thereby reducing sampling bias and ensuring a fair comparison between training and evaluation stages. In particular, we maintain a balanced distribution of positive and negative labels across all splits. In the full dataset, positive cases account for 70.6% of the instances, while negative cases account for 29.4%. This stratified design helps ensure that the model is exposed to a broad range of matching and non-matching patterns during training while being evaluated on a similarly representative test distribution.
𝑘=1
4.1.2 Baselines. To evaluate the effectiveness of our proposed framework, we compare EPM-RL against representative baselines spanning encoder-based binary classification, single-inference prompting, retrieval-augmented inference, and agentic multi-step pipelines:
In our setting, the reward function combines a verifiable reward (VR) with agentic reward scores. The verifiable part checks whether the output follows the required structure and whether the predicted label is valid. The agentic part is computed by three specialized LLM judges that evaluate complementary aspects of the reasoning: (1) a Core Identity judge that assesses whether the reasoning correctly identifies and compares the central product/category identity, (2) a Model-Identifier judge that evaluates whether the reasoning properly handles brand, model line, and explicit identifier tokens, and (3) a Variant-Conflict judge that measures how well the reasoning detects variant-level consistency or conflict, such as size, capacity,
• Encoder binary classifier [8, 21]: A supervised baseline that trains a text encoder to directly classify whether (𝑝𝑏 , 𝑝𝑐 ) match. We encode the concatenated pair and train a lightweight classification head with cross-entropy loss. • Zero-shot: A direct single-inference baseline where the LLM receives the base product and compared product titles and predicts the binary match label without any intermediate structure. • Entity Attribution: A single-inference baseline that encourages the LLM to explicitly extract and compare salient attributes (e.g., brand, product type, specification, quantity, and optionrelated cues) before predicting the label.
Conference’17, July 2017, Washington, DC, USA
Trovato et al.
• Chain-of-Thought (CoT) [29]: A single-inference baseline that prompts the LLM to generate a step-by-step rationale before producing the final binary decision. • RAG [17]: A retrieval-augmented generation baseline that retrieves evidence using BM25 and appends it to the prompt before prediction. The BM25 query is constructed by concatenating the two titles as base product [SEP] compared product. • Multi-Agent RAG [3]: An agentic retrieval baseline that decomposes inference into three steps: a direct agent predicts a label from the titles only, an indirect agent predicts a label conditioned on BM25-retrieved evidence, and a coordinator agent aggregates the two predictions into a final decision. Encoder classifier formulation. Let 𝑥 = [[CLS]; 𝑝𝑏 ; [SEP]; 𝑝𝑐 ; [SEP]] be the packed title pair and let ℎ = Enc(𝑥) [CLS] ∈ R𝑑 be the encoder’s pooled representation. We predict the match probability with a logistic head: 𝑝ˆ (𝑦=1 | 𝑥) = 𝜎 𝑤 ⊤ℎ + 𝑏 , (8) where 𝑤 ∈ R𝑑 , 𝑏 ∈ R are trainable parameters and 𝜎 (·) is the sigmoid function. The model is trained by minimizing the binary cross-entropy: ˆ Lenc = − 𝑦 log 𝑝ˆ − (1 − 𝑦) log(1 − 𝑝).
(9)
The first three baselines belong to the single-inference setting. In contrast, RAG and Multi-Agent RAG incorporate external evidence and multi-step reasoning, providing stronger but higher-latency inference-time baselines. For fairness, all baselines share the same base instruction prompt, and differ only in whether they add structured reasoning, retrieved evidence, or agentic orchestration. 4.1.3
Models Used.
Encoder Models (Binary Classification). We train and compare four encoder-based binary classifiers: BERT-base, BERT-large [8], RoBERTa-base, and RoBERTa-large [21]. Each model is implemented as a cross-encoder over the packed title pair and trained with the binary cross-entropy objective described in the baselines section. We train for up to 5 epochs with early stopping (patience 2), using a batch size of 1024 and a learning rate of 1 × 10−5 . Large Language Models. We use models from the (1) NemotronNano-3 family [4] and (2) GPT-5.4 reasoning, assigning each to a distinct role in the pipeline. As a strong teacher and reference baseline, we use Nemotron-Nano-3 Super-120B1 for (i) synthesizing reflective reasoning traces for PEFT training and (ii) reporting a high-capacity inference-time baseline. For reinforcement learning, we use GPT-5.4 reasoning2 to implement the three judge agents that score reasoning quality. Our main trainable and evaluation model is Nemotron-Nano-3 30B-A3B3 (referred to as NemotronNano-3-30B), a hybrid Mixture-of-Experts model with 30B total parameters and roughly 3.5B active parameters per token. Across all experiments, decoding uses temperature 0.7 and top-𝑝 0.95, with a maximum output length of 1024 tokens to accommodate both titles and structured reasoning outputs. 1 https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 2 https://openai.com/index/introducing-gpt-5-4 3 https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
Retrieval Model. For retrieval-based baselines, we use BM25 [24] to retrieve the top-𝑘=5 evidence candidates from an internal retrieval corpus. Given a query 𝑞 (constructed from the base product and compared product titles) and a candidate document 𝑑, BM25 computes a relevance score ∑︁ 𝑓 (𝑡, 𝑑) (𝑘 1 + 1) , (10) BM25(𝑞, 𝑑) = IDF(𝑡) · |𝑑 | 𝑓 (𝑡, 𝑑) + 𝑘 1 1 − 𝑏 + 𝑏 avgdl 𝑡 ∈𝑞 where IDF(𝑡) denotes the inverse document frequency of term t, measuring how rare or informative t is across the document collection, 𝑓 (𝑡, 𝑑) is the term frequency of token 𝑡 in 𝑑, |𝑑 | is the document length, and avgdl is the average document length in the corpus. We use the default parameters 𝑘 1 =1.2 and 𝑏=0.75. The retrieved evidence is appended to the LLM prompt for the RAG and Multi-Agent RAG baselines described above. 4.1.4
PEFT and RL Setups.
PEFT Setup. For parameter-efficient fine-tuning (PEFT), we use Low-Rank Adaptation (LoRA) [13] to adapt the model by learning low-rank updates to selected linear projections while keeping the backbone weights frozen. Given a pretrained projection matrix 𝑊 ∈ R𝑑 ×𝑘 , LoRA parameterizes the update as 𝑊 ′ = 𝑊 + Δ𝑊 ,
Δ𝑊 = 𝐵𝐴,
(11)
where 𝐴 ∈ R𝑟 ×𝑘 and 𝐵 ∈ R𝑑 ×𝑟 are trainable low-rank matrices and 𝑟 ≪ min(𝑑, 𝑘) is the LoRA rank. With the standard scaling rule, the adapted projection becomes 𝛼 𝑊 ′ = 𝑊 + 𝐵𝐴, (12) 𝑟 where 𝛼 is a scaling factor. To make the adaptation lightweight, we apply LoRA only to the attention projections. For each transformer layer ℓ, we update 𝑊𝑞(ℓ ), 𝑊𝑘(ℓ ), 𝑊𝑣(ℓ )
with LoRA, and keep all other parameters frozen. (13) We set the learning rate to 1 × 10−5 , batch size to 4, LoRA rank to 𝑟 =32, and scaling factor to 𝛼=64, and train for 5 epochs. This configuration provides a practical trade-off between parameter efficiency and adaptation capacity while keeping training stable and computationally manageable.
RL Setup. RL parameters. After PEFT, we further optimize the model with reinforcement learning using Group Relative Policy Optimization (GRPO) [19]. In this stage, we keep the same adapter configuration as in PEFT so that the RL update remains parameter-efficient and directly refines the previously learned reasoning behavior. As in the PEFT stage, we freeze the backbone model and train only the LoRA adapter parameters. Specifically, the LoRA rank and scaling factor are kept unchanged, and updates are again restricted to the adapter parameters. We set the learning rate for RL to 5 × 10−5 , the training batch size to 4, and the number of rollouts per input to 4. To stabilize optimization, we use a maximum clipping value of 0.1 and apply a dropout rate of 0.05. The model is trained for 1 epoch. Reward design. We define the total reward as a weighted combination of three components: (i) structured output compliance (weight 1), which checks whether the model output follows the required
EPM-RL: Reinforcement Learning for On-Premise Product Mapping in E-Commerce
Conference’17, July 2017, Washington, DC, USA
XML-like format (e.g., <reason> and <label> fields); (ii) binary decision correctness (weight 2), which assigns reward based on whether the predicted binary label matches the ground-truth label; and (iii) LLM-as-a-judge preference (weight 1), computed by averaging the scores from three specialized judge agents. Formally, let 𝑠 fmt ∈ {0, 1} denote the format-compliance score, 𝑠 cls ∈ {0, 1} denote the label-correctness score, and let 𝑠 judge = 1 Í3 𝑗=1 𝑠 𝑗 with 𝑠 𝑗 ∈ [0, 1] be the mean judge score. Let weights 3 𝜆fmt =1, 𝜆cls =2, and 𝜆judge =1. The normalized reward is Í 𝜆𝑢 𝑠𝑢
Table 2: Prompt templates used in our experiments. We use a shared Baseline prompt for single-inference baselines and a structured PEFT prompt for supervised fine-tuning.
𝑅=
𝑢 ∈ {fmt,cls,judge}
Í
∈ [0, 1].
(14)
𝜆𝑢
𝑢 ∈ {fmt,cls,judge}
Compared with PEFT, this stage focuses less on imitation of target outputs and more on refining generation behavior according to reward signals, enabling the model to improve the quality, consistency, and decision usefulness of its reasoning traces under the GRPO objective. 4.1.5 Hardware / Framework Specification. We conducted LoRA fine-tuning of NVIDIA’s Nemotron model using the MegatronBridge4 SFT framework on a node equipped with 8 NVIDIA H100 GPUs. Subsequently, we performed LoRA-based reinforcement learning with Hugging Face and TRL using the same GPU configuration. This setup allowed us to efficiently adapt the model during training stages.
4.2
Judge Agent Design
We implement the LLM-as-a-judge component in GRPO with three specialized judge agents. These judges are motivated by a small expert analysis: a human domain expert inspected 100 product-pair samples and identified three recurring reasoning patterns (core identity matching, model/identifier matching, and variant-level conflict checking) that most strongly determined correct decisions. Accordingly, each judge takes as input the base product title 𝑝𝑏 , the compared product title 𝑝𝑐 , and the model-generated reasoning text 𝑟 extracted from <reason>. . . </reason>, and outputs a single scalar score 𝑠 ∈ [0, 1] (one float with no explanation) for its designated sub-skill; the scores are later aggregated into the judge-based reward term. To reduce reward hacking and improve faithfulness, we instruct judges to (a) evaluate only their designated sub-skill, (b) penalize hallucinated tokens not present in the input titles, and (c) return 0.0 when the extracted reasoning is missing or empty. Core Identity agent (Step-1). This agent scores whether the reasoning correctly identifies and compares the core product/category identity shared or mismatched between the base and compared titles. It focuses on the central product type and key anchor tokens (e.g., the main product/category terms), and ignores brand/model codes and variant attributes. Scores are assigned on a graded rubric from generic or weak core comparison (≈ 0.5) to token-grounded and correct core identification (1.0). Model-Identifier agent (Step-2). This agent scores whether the reasoning correctly handles brand and model identifiers, including brand prefixes, model lines, and explicit model codes. It evaluates 4 https://github.com/NVIDIA-NeMo/Megatron-Bridge
Baseline You are a product matching classifier. Decide whether Product A and Product B refer to the same sellable product. Output only 1 if matched, otherwise output only 0. (1) Different product family, series, or model name means 0. (2) Different variant attributes that change the sellable SKU (such as color, size, capacity, count, edition, generation, set composition, pack size) means 0. (3) Ignore only minor formatting differences, spacing, punctuation, obvious spelling noise, or packaging-only phrases. (4) If uncertain, output 0. Output only one character: 0 or 1. PEFT You are a product-title matching analyst. Task: Given two product names, decide whether they refer to the same sellable product variant. • Product A: {p1_name} • Product B: {p2_name} Important constraints: (1) Do NOT use any external label column or hidden metadata. Decide only from the two product names. (2) Compare step by step before concluding. (3) Treat differences in model code, capacity, size, color, quantity, option, bundle composition (set/single item), edition/origin/version as potentially critical. (4) Ignore minor wording differences such as spacing, punctuation, seller prefix/brand prefix, and marketing words when core identity is still the same. (5) If core product identity or key variant conflicts, output 0. (6) If core product identity and key variant are consistent (or one side is only less specific without contradiction), output 1. Output format (must follow exactly): (1) First, <identify the core product/category and main tokens in both names>. (2) Second, <compare brand/model line/model number and key identifiers>. (3) Third, <compare variant attributes: size/color/count/spec/option/bundle, and check for conflicts>. (4) So, the final answer is: <0 or 1>. Label meaning: • 1 = matched (same sellable product variant) • 0 = not matched (different product or conflicting variant) Expected output: <reason>evidence</reason><label>0/1</label>
whether the reasoning distinguishes ignorable prefixes from true identifiers and grounds identifier comparisons in tokens appearing in the base and compared titles. It does not judge variant conflicts (e.g., size/count differences). Variant-Conflict agent (Step-3). This agent scores whether the reasoning checks variant attributes and detects conflicts or consistency between the base and compared titles. It focuses on attributes such as size, color, capacity, count, specification, option, bundle composition, and version. The score reflects how completely and correctly the reasoning cites relevant tokens and determines whether variant-level differences imply a mismatch.
4.3
Prompts
We provide the main prompt templates used in our experiments: a shared Baseline prompt for all single-inference baselines and a structured PEFT prompt for supervised fine-tuning in Table 2. The Baseline prompt and is kept identical across baselines so that any performance differences are attributable to the inference strategy rather than prompt phrasing. In contrast, the PEFT prompt elicits a structured three-stage comparison—First (core product/category identity), Second (brand/model identifiers), and Third
Conference’17, July 2017, Washington, DC, USA
Trovato et al.
Table 3: Main experiment results on the internal product-mapping test set. We report Accuracy (Acc), Precision (Prec), Recall (Rec), and F1. Single inference uses one-pass prompting without external tools; RAG augments the prompt with BM25-retrieved evidence; Multi-Agent RAG coordinates multiple retrieval-and-reasoning agents over the same BM25 evidence pool. Ours summarizes our parameter-efficient variants (LoRA + Reasoning; LoRA + GRPO Reasoning). For each metric, bold indicates the best score and underline indicates the second best score. Accuracy ↑
Precision ↑
Recall ↑
F1-Score ↑
BERT-base
0.8250
0.6814
0.8235
0.7458
BERT-large
0.8290
0.6713
0.8197
0.7381
RoBERTa-base
0.8367
0.7405
0.7326
0.7366
RoBERTa-Large
0.8300
0.6741
0.8163
0.7385
Zero Shot Inference
0.8540
0.8109
0.6565
0.7256
Chain-of-Thought Reasoning
0.8450
0.8639
0.5612
0.6804
Entity-Attribute Reasoning
0.8440
0.8750
0.5476
0.6736
0.8490
0.7227
0.7891
0.7545
0.8640
0.8911
0.6122
0.7258
0.8630
0.9758
0.5476
0.7015
LoRA + Reasoning
0.8570
0.7103
0.8673
0.7810
LoRA + GRPO Reasoning
0.8450
0.7815
0.8735
0.8120
Method Encoder Binary Classifier
Single inference
RAG Retrieval-Augmented Generation (BM25) Multi-Agent RAG Multi-Agent Retrieval-Augmented Generation (BM25) Reasoning LLM GPT-5.4 reasoning Ours
(variant attributes and bundle composition)—before asking for a final binary decision. This “First/Second/Third” decomposition mirrors the competencies evaluated by our three judge agents (Core Identity, ModelIdentifier, and Variant-Conflict; Section 4.2). As a result, the finetuned model learns to produce reasoning traces that are directly scorable by the judges, which improves the stability and interpretability of reward shaping during GRPO. Finally, using a shared Baseline prompt and a judge-aligned PEFT prompt reduces promptinduced variance across training and evaluation, helping ensure that improvements reflect model capability rather than formatting artifacts.
4.4
Main Experiement Results
We compare EPM-RL against a diverse set of baselines, including encoder-based binary classifiers, single-inference prompting methods, retrieval-augmented approaches, multi-agent reasoning pipelines, and strong large language models. As shown in Table 3, the single-inference prompting baselines provide a useful starting point but exhibit clear limitations. Zero-shot inference achieves the strongest overall balance among the prompting-based variants, while Chain-of-Thought and Entity-Attribute Reasoning do not consistently improve performance despite introducing more explicit intermediate reasoning. We additionally include encoder
binary classifiers, such as BERT and RoBERTa, as representative non-generative baselines to contextualize the gains from reasoningcentric approaches. Overall, these results suggest that prompting the model to reason more verbosely does not necessarily translate into better product mapping performance, especially when the decision depends on subtle distinctions in bundle composition, variant conflict, or product identity. Among the stronger inference-time baselines, retrieval-based and agentic methods show a distinct trade-off. Standard RAG improves recall and achieves the strongest F1 score among the nonagentic baselines, indicating that retrieved evidence is helpful for resolving ambiguous product pairs. Multi-Agent RAG further strengthens direct comparison through coordinated evidence-based reasoning and achieves the highest accuracy and precision among the listed baselines. However, its recall remains relatively low, suggesting that the multi-agent pipeline tends to make more conservative positive predictions. One possible explanation is that the multiagent setting amplifies conservative decisions through agent coordination and voting: when multiple agents independently assess the same pair, disagreement or uncertainty may be resolved toward non-match decisions, thereby increasing precision at the cost of recall. In addition, because the retrieval corpus and evidence construction are derived from the same internal product-mapping environment, Multi-Agent RAG may partially reflect label-distribution
EPM-RL: Reinforcement Learning for On-Premise Product Mapping in E-Commerce
or decision-pattern biases present in the training data, which can further contribute to its high precision. A similar tendency is observed for the GPT-5.4 reasoning baseline, which achieves extremely high precision but substantially lower recall. This pattern suggests that the model is highly selective in predicting positive matches. Rather than indicating uniformly superior matching ability, the result may reflect an inherent decision bias in the reasoning model toward avoiding false positives under ambiguous product-title comparisons. In product mapping, such conservatism can be beneficial when precision is prioritized, but it can also lead to many missed matches, as reflected in the lower recall and F1 score. These observations highlight that high precision alone is insufficient to characterize overall product-mapping quality; a practical system must also maintain enough recall to recover valid matches across noisy and heterogeneous marketplace titles. In contrast, our framework EPM-RL is designed to internalize structured reasoning into the model itself rather than depending on repeated inference-time coordination. The PEFT-based variant already provides a stronger and more practical alternative to prompt-only baselines by teaching the model to generate productaware reasoning traces together with the final decision. Building on this, the GRPO-enhanced variant further improves the faithfulness and usefulness of the generated reasoning through verifiable rewards and specialized judge-based reward signals. As a result, EPM-RL achieves the best overall F1 performance on the product-mapping benchmark, outperforming both single-inference and retrieval-based baselines in terms of balanced decision quality while remaining more suitable for efficient on-premise deployment. In particular, its higher recall indicates that the model can recover substantially more true matches than highly conservative baselines, while its precision remains competitive. These findings suggest that distilling agentic reasoning into a trainable in-house model is more effective than relying solely on prompt engineering, large proprietary reasoning models, or costly inference-time retrieval pipelines. Overall, the results demonstrate three key observations. First, purely prompt-based reasoning is not sufficient for robust product mapping in realistic e-commerce settings. Second, retrievaland agent-based methods can improve grounding and precision, but their benefits may come with conservative decision behavior, label-distribution sensitivity, and additional system overhead. Third, EPM-RL offers a more favorable balance by combining reasoning quality, predictive performance, and deployment efficiency within a unified trainable framework.
5
Conclusion
We introduced EPM-RL, a reinforcement-learning framework for building an accurate and efficient on-premise product mapping model. Our approach distills high-cost agentic reasoning into a trainable in-house model by combining parameter-efficient finetuning with structured reasoning traces and GRPO-based reinforcement learning using verifiable rewards and judge-agent signals. Our results on an internal product-mapping benchmark demonstrate that EPM-RL can improve over prompt-only and retrieval-heavy baselines while avoiding inference-time orchestration.
Conference’17, July 2017, Washington, DC, USA
6
Limitations
(1) Internal benchmark scope. We have evaluated primarily on an internally curated dataset, and additional experiments on public, cross-domain product-matching benchmarks are needed to better assess generalization. (2) Limited qualitative analysis. Our paper provides limited qualitative analysis of generated reasoning traces and judgeagent scoring behavior. Deeper case studies would help clarify failure modes, reward sensitivity, and reasoning faithfulness. (3) Incomplete system and ablation coverage. We do not yet provide a full ablation of reward components, judge prompts, and training hyperparameters, nor a detailed study of latency/cost trade-offs across deployment settings. We will add further analysis in the future.
References [1] Steven S Aanen, Damir Vandic, and Flavius Frasincar. 2015. Automated product taxonomy mapping in an e-commerce environment. Expert Systems with Applications 42, 3 (2015), 1298–1313. [2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). [3] Aatif Muhammad Althaf, Muzakkiruddin Ahmed Mohammed, Mariofanna Milanova, John Talburt, and Mert Can Cakmak. 2025. Multi-Agent RAG Framework for Entity Resolution: Advancing Beyond Single-LLM Approaches with Specialized Agent Coordination. Computers 14, 12 (2025), 525. [4] Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khattar, Adi Renduchintala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, et al. 2025. Nemotron 3 Nano: Open, Efficient Mixture-ofExperts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv preprint arXiv:2512.20848 (2025). [5] Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. 2025. Why do multi-agent llm systems fail? arXiv preprint arXiv:2503.13657 (2025). [6] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology 15, 3 (2024), 1–45. [7] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems 36 (2023), 10088–10115. [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186. [9] Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. 2023. Flocks of stochastic parrots: Differentially private prompt learning for large language models. Advances in Neural Information Processing Systems 36 (2023), 76852–76871. [10] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, Haofen Wang, et al. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2, 1 (2023), 32. [11] Antonio Greco. 2018. E-Commerce monitoring solution for product allocation and marketing planning forecasting. Ph. D. Dissertation. Politecnico di Torino. [12] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 (2020). [13] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. Iclr 1, 2 (2022), 3. [14] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics 8 (2020), 423–438. [15] Mayank Kejriwal, Ke Shen, Chien-Chun Ni, and Nicolas Torzec. 2021. An evaluation and annotation methodology for product category matching in e-commerce. Computers in Industry 131 (2021), 103497.
Conference’17, July 2017, Washington, DC, USA
[16] Hanna Köpcke, Andreas Thor, and Erhard Rahm. 2010. Evaluation of entity resolution approaches on real-world match problems. Proceedings of the VLDB Endowment 3, 1-2 (2010), 484–493. [17] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33 (2020), 9459–9474. [18] Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. 2024. A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth 1, 1 (2024), 9. [19] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024). [20] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning. [21] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). [22] Anna Primpeli, Ralph Peeters, and Christian Bizer. 2019. The WDC training dataset and gold standard for large-scale product matching. In Companion Proceedings of The 2019 World Wide Web Conference. 381–386. [23] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing
Trovato et al.
systems 36 (2023), 53728–53741. [24] Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Vol. 4. Now Publishers Inc. [25] Wonduk Seo, Taesub Shin, Hyunjin An, Dokyun Kim, and Seunghyun Lee. 2025. Question-to-Knowledge (Q2K): Multi-Agent Generation of Inspectable Facts for Product Mapping. arXiv preprint arXiv:2509.01182 (2025). [26] Yashar Talebirad and Amirhossein Nadiri. 2023. Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314 (2023). [27] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). [28] Ying Wang. 2009. Applications of e-commerce across the manufacturing supply chain to achieve the promise of e-manufacturing. Ph. D. Dissertation. University of Huddersfield. [29] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837. [30] Ronald R Yager and Gabriella Pasi. 2001. Product category description for webshopping in e-commerce. International Journal of Intelligent Systems 16, 8 (2001), 1009–1021. [31] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 1, 2 (2023), 1–124.