GEO-B ENCH: Benchmarking Ranking Manipulation in Generative Engine Optimization Ojas Nimase1∗
Zhe Chen1∗
Gengpei Qi1∗
Yue Zhao1
Xiyang Hu2
1
2 University of Southern California Arizona State University ∗ Equal contribution {nimase, zchen116, gengpeiq, yue.z}@usc.edu [email protected]
arXiv:2605.29107v1 [cs.CR] 27 May 2026
Abstract
cern, spanning e-commerce product rankings (Tang et al., 2025; Du et al., 2026), content recommendation (Hou et al., 2024), and search result optimization (Ho et al., 2025). As LLMs rank content at scale, the ability to manipulate these rankings carries significant implications for fairness, user experience, and market dynamics. Ranking manipulation has emerged as a central challenge in GEO. Recent work shows that attacks can artificially inflate rankings through adversarial modifications to product descriptions and metadata (Tang et al., 2025; Kumar and Lakkaraju, 2024; Nestaas et al., 2024), transfer to production search engines, and run at scale (Pfrommer et al., 2024; Ho et al., 2025), motivating systematic evaluation and understanding. However, the field lacks a unified framework for comparing these approaches: existing work uses disparate datasets, metrics, and experimental setups. This fragmentation has three consequences. First, one cannot tell whether a method’s reported effectiveness generalizes beyond its single test dataset. Second, without joint evaluation of effectiveness and stealth, it is unclear whether high success rates come at the cost of trivial detectability. Third, the absence of standardized metrics prevents comparison between black-box and white-box attacks on the same ranking systems. To address this gap, we introduce GEO-B ENCH, the first comprehensive benchmark for GEO ranking manipulation. Our contributions include:
Large language models (LLMs) increasingly rank products, documents, and recommendations for user queries, which makes manipulating these rankings a growing concern for fairness and information integrity. Research on generative engine optimization (GEO) has produced many manipulation methods, but each is evaluated on its own dataset with its own metrics, so their relative strength and detectability stay unclear. We present GEO-B ENCH, a benchmark that evaluates GEO rankingmanipulation attacks under one protocol. It unifies black-box prompt-based attacks (TAP, Zero-Shot), white-box gradient-based attacks (STS, RAF, StealthRank), and ten white-hat C-SEO strategies. We score every method on five datasets against a fixed open-weight ranker (Llama-3.1-8B-Instruct), using metrics for both effectiveness (NRG, Success@α, Promote@α) and stealth (keyword violation rate, perplexity ratio). Our evaluation shows that effectiveness and stealth trade off across adversarial attacks, that black-box content rewriting matches or exceeds gradient-based attacks on rank promotion while producing more fluent text and can evade both keyword- and perplexity-based detection on some domains, and that the access model does not predict attack strength. By standardizing datasets, attack implementations, and metrics, GEO-B ENCH enables the first direct comparison across these attack paradigms and supports the development of detection methods. The code is provided in https://anonymous. 4open.science/r/geobench-BDD6/.
1
• Unified Benchmark. The first standardized benchmark for evaluating GEO manipulation algorithms across diverse domains, with a taxonomy organized by access model (black-box vs. white-box) that enables direct comparison of attack paradigms and identifies realistic threat models for different deployment settings. • Standardized Resources. Unified datasets with consistent formatting, reproducible algorithm im-
Introduction
Large language models (LLMs) increasingly retrieve, rank, and present information to users, both in response to direct queries and as components of agentic AI systems (Bao et al., 2023). This shift has made generative engine optimization (GEO), the practice of optimizing content visibility within LLM-powered systems, a critical con1
plementations, and standardized evaluation metrics spanning both effectiveness and stealth. • Empirical Findings. A systematic analysis showing that effectiveness and stealth trade off across adversarial attacks, while black-box content rewriting can match or exceed gradientbased attacks on rank promotion, stay fluent, and on some domains evade both keyword- and perplexity-based detection.
2
Dataset
#Items
#Cat.
Domain
Ragroll STSData RewriteToRank LLM Rank Optimizer C-SEO Bench
399 30 10,000† 40 16,360
50 3 2,202 4 6
Products Products Products Products Mixed
Table 1: Datasets in GEO-B ENCH. Item counts are fullcollection sizes; large collections are subsampled for evaluation (Appendix A). † RewriteToRank is evaluated on a 20-category, 10-item-per-category subsample.
Related Work (gradient-based and jailbreaking-based) and stealth metrics (keyword violation rate, perplexity ratio), so it cannot assess detectability. GEO-B ENCH addresses these gaps by unifying adversarial and white-hat methods under a common framework that jointly measures effectiveness and stealth across diverse datasets, enabling the first direct comparison across attack paradigms.
LLM-Based Ranking and GEO. LLMs are increasingly deployed as zero-shot rankers for product recommendation (Hou et al., 2024) and within conversational search engines (Aggarwal et al., 2024). Aggarwal et al. (2024) introduced the term generative engine optimization (GEO) and proposed white-hat content transformation strategies (e.g., authoritative rewriting, adding statistics) to improve document visibility, though their evaluation used a single task and metric (word count) and did not test competitive multi-actor settings.
3
GEO-B ENCH: Benchmark Design
3.1
Datasets
We evaluate GEO-B ENCH on five datasets drawn from recent GEO and ranking-manipulation literature (Table 1). Four are standalone product or content collections: Ragroll (Pfrommer et al., 2024), STSData (Kumar and Lakkaraju, 2024), RewriteToRank (Ho et al., 2025), and LLM Rank Optimizer (Kumar and Lakkaraju, 2024). The fifth, C-SEO Bench (Puerto et al., 2025), aggregates six domains (books, debate, news, retail, videogames, and web) and is reported as a single row in Table 4. Large collections are subsampled for tractable evaluation; per-dataset sizes appear in Table 1 and construction details in Appendix A.
Adversarial Ranking Manipulation. A parallel line of work has explored adversarial attacks on LLM-based ranking. Kumar and Lakkaraju showed that gradient-optimized token sequences appended to descriptions can force LLMs to rank a target item first (Kumar and Lakkaraju, 2024). Pfrommer et al. (2024) adapted jailbreaking techniques to ranking via black-box prompt search. Nestaas et al. (2024) demonstrated that crafted website content can bias LLM selections in production engines including Bing and Perplexity. Tang et al. (2025) proposed StealthRank, jointly optimizing for rank promotion, fluency, and keyword avoidance. Xing et al. (2026) introduced RAF, a two-stage gradient method balancing rank gain with readability. Ho et al. (2025) showed that retrieval-aware rewriting can optimize ad visibility at scale. Each evaluates on its own dataset and metrics, preventing direct cross-method comparison.
3.2
Algorithms
We organize methods by access model: whether an attack needs only black-box query access to the ranking LLM or white-box access to model gradients. This determines deployment feasibility, since black-box attacks apply to any commercial LLM API while gradient-based methods require open-weight models, and it tells defenders which threat models are realistic. We benchmark eight algorithms across two paradigms, prompt-based methods that generate adversarial text via LLM calls and gradient-based methods that optimize adversarial token sequences, and additionally evaluate ten white-hat C-SEO strategies (Appendix E), reporting the top three in the main results. Table 2
Existing Benchmarks. The most directly related work is C-SEO Bench (Puerto et al., 2025), which evaluates ten white-hat C-SEO strategies across six domains and two tasks (product recommendation and question answering). C-SEO Bench makes two contributions absent from prior work: multi-domain evaluation and multi-actor competitive simulations. However, it covers only white-hat content transformation, omitting adversarial attacks 2
Method
Type
Prompt-based methods Zero-Shot Adversarial TAP Adversarial Authoritative White-hat C-SEO White-hat Gradient-based methods STS Adversarial RAF Adversarial StealthRank Adversarial
Mechanism
Access
Iter.
Single LLM call adds an adversarial suffix Tree-structured prompt search via an attacker LLM LLM rewrite for authority and persuasiveness LLM content rewriting; top 3 of 10 strategies
Black-box Black-box Black-box Black-box
× ✓ × ×
Gradient-optimized token suffix on the description Two-stage gradient token search with readability Balances rank gain, fluency, and keyword stealth
White-box White-box White-box
✓ ✓ ✓
Table 2: Taxonomy of GEO manipulation methods in GEO-B ENCH. Methods are grouped by access model; Type separates adversarial attacks from white-hat content optimization, and Iter. marks iterative refinement. Ten C-SEO variants are evaluated in Appendix E.
gives a one-line mechanism per method; full descriptions are in Appendix B. 3.3
total. Our unified evaluation reveals tradeoffs invisible in prior work: effectiveness and stealth trade off across adversarial attacks, yet white-hat content rewriting can evade both detection signals on some domains, and the access model (black-box vs. gradient-based) does not predict overall performance. Figure 1 plots each method’s mean effectiveness and stealth; full per-dataset results for every metric appear in Appendix F.
Evaluation Metrics
We evaluate ranking manipulation along two complementary dimensions: effectiveness (whether and how much the target item is promoted) and stealthiness (whether the manipulation remains natural and hard to detect). Effectiveness uses three metrics. Normalized Rank Gain (NRG) is the rank improvement, scaled to [−1, 1] for comparability across list lengths. Success@α records whether the target ends within the top-α fraction. Promotion Success@α (Promote@α), a stricter variant, counts only items moved into the top-α region from outside, isolating the causal promotion effect. Stealth uses two. Keyword Violation Rate (KVR) is the fraction of manipulated descriptions with restricted promotional keywords; lower is stealthier. Perplexity Ratio (PPL-R) is the manipulated text’s perplexity relative to the original under a fixed reference model; values near 1 indicate preserved fluency. Formal definitions are in Appendix C.
4
4.1
Analysis
Effectiveness and access model. Two patterns hold across the datasets (Table 4). White-hat CSEO strategies match or exceed the gradient-based attacks on three of the five datasets despite using only black-box API access; and the access model does not predict attack strength, since the three highest average-NRG methods (TAP, Authoritative, Content Improvement) are all black-box while the gradient methods rank in the middle and bottom. A likely cause is that gradient methods optimize a token-level surrogate that imperfectly tracks the ranker, whereas prompt-based methods operate at the semantic level it reasons over.
Experiments
The effectiveness–stealth tradeoff. Across the adversarial attacks, effectiveness and stealth trade off: each method either stays detectable or fails to promote. TAP promotes well but stuffs keywords (KVR ≥ 0.83 everywhere); STS keeps keyword rates low (0.03–0.20) but sharply degrades fluency (PPL-R up to 12.72); RAF keeps keywords low and Zero-Shot stays quiet on both signals, yet neither moves the target (NRG ≤ 0.09 and ≤ 0.27). The consequential exception is white-hat rather than adversarial: on C-SEO Bench, Authoritative rewriting ties LLM Guidance for the highest NRG (0.83)
Experimental Setup. All algorithms are evaluated using Llama-3.1-8B-Instruct as the target ranking LLM. Perplexity ratios (PPL-R) are computed using Vicuna-7B as the reference language model. For C-SEO methods, rewrites are generated with GPT-4o-mini following the protocol of Puerto et al. (2025). Each algorithm is evaluated on every dataset using the metrics introduced in Section 3.3 and formally defined in Appendix C. All experiments were conducted on 3× NVIDIA L40S GPUs, requiring approximately 200 GPU hours in 3
Keyword violation rate ( stealthier)
1.0 0.8
TAP
Black-box adversarial White-box gradient White-hat C-SEO
Content Improvement
0.6
StealthRank
0.4
Authoritative LLM Guidance
0.2
RAF
0.0 0.0
Zero-Shot
size = perplexity ratio (larger = less fluent) PPL-R 1 PPL-R 3 PPL-R 6
STS
Authoritative @ C-SEO Bench: evades both detectors
effective + keyword-stealthy
0.2 0.4 0.6 Effectiveness: mean NRG ( stronger promotion)
0.8
Figure 1: Effectiveness–stealth tradeoff. No adversarial attack is at once effective, keyword-stealthy, and fluent; only white-hat rewriting (the star) escapes the tradeoff. Each marker is a method’s mean over the five datasets: effectiveness (NRG, higher is stronger promotion) against keyword violation rate (KVR, lower is stealthier), with marker size the mean perplexity ratio (larger = less fluent) and color the method family. The goal is the bottom-right corner with a small marker, which no adversarial attack reaches. The exception is the star: on C-SEO Bench, Authoritative ties LLM Guidance for the highest NRG (0.83) at zero keyword violations and a 0.74 perplexity ratio. Full per-dataset values for every method and metric are in Table 4 (Appendix F).
Implications and Future Research Directions. These results reorder the priorities for defenders and method designers. First, the surface detectors are porous and jointly incomplete: keyword and perplexity checks each miss a different set of methods, and Authoritative on C-SEO Bench clears both, so detection must move toward semantic or intent-level signals. Second, the pressing threat is benign content optimization, not gradient attacks: white-hat rewriting matches or exceeds the gradient methods on promotion while staying far more fluent, and at its best evades both detectors through black-box API access alone. Third, because rankings flip across datasets and metrics, single-dataset or single-metric claims are unreliable; new attack and detection methods should be evaluated on multiple datasets under joint effectiveness and stealth metrics, which GEO-B ENCH standardizes.
while reaching zero keyword violations and a 0.74 perplexity ratio, evading both detectors while promoting strongly. The finding is therefore two-sided: adversarial attacks do not jointly achieve promotion, keyword avoidance, and fluency, but benign content rewriting already defeats both simple detectors on a realistic multi-domain benchmark. The two stealth signals are also complementary: Content Improvement stays fluent (PPL-R 0.40) but trips the keyword filter (KVR 0.70), while STS does the reverse. For defenders, the harder target is fluent, keyword-free rewriting, not adversarial perturbation (Figure 1). Decisive promotion. Threshold metrics sharpen this: [email protected] credits only targets moved into the top decile from outside, isolating genuine promotion. TAP leads (up to 0.92); the keyword-quiet RAF and Zero-Shot never exceed 0.28 and 0.02.
5 Dataset sensitivity. Method rankings are unstable across datasets. Even TAP, the strongest method on average (NRG 0.53), takes the top NRG on only one of the five datasets; individual methods swing by an order of magnitude, with Authoritative leading on C-SEO Bench (0.83) yet collapsing to 0.00 on LLM Rank Optimizer and STS moving from 0.60 to 0.06. Single-dataset results therefore overstate generalizability, which GEO-B ENCH exposes by spanning five heterogeneous datasets.
Conclusion
We presented GEO-B ENCH, a unified benchmark that scores black-box and gradient-based GEO ranking-manipulation attacks under one protocol across five datasets. Its central finding is twosided: adversarial attacks trade off effectiveness against stealth, yet Authoritative rewriting on CSEO Bench evades both detectors while promoting strongly. We will release the benchmark artifacts after license and anonymization checks. 4
Limitations
Yixuan Du, Chenxiao Yu, Haoyan Xu, Ziyi Wang, Yue Zhao, and Xiyang Hu. 2026. Multimodal generative engine optimization: Rank manipulation for visionlanguage model rankers. In Proceedings of the 4th Workshop on Towards Knowledgeable Foundation Models (KnowFM), San Diego, California, United States. Association for Computational Linguistics.
GEO-B ENCH evaluates all methods against a single open-weight ranker (Llama-3.1-8B-Instruct) to isolate cross-method and cross-dataset differences on reproducible hardware; because the benchmark standardizes datasets, implementations, and metrics independently of the target model, extending it to additional rankers is straightforward. Our stealth metrics are automatic proxies (keyword-violation rate and perplexity ratio) rather than human judgments or trained detectors, and the datasets center on English product- and content-ranking, so conclusions may not transfer to other languages or domains without corresponding data.
Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. 2019. Multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model. Preprint, arXiv:1906.01749. FronkonGames. 2024. Steam games dataset. Hugging Face Datasets. Chloe Ho, Ishneet Sukhvinder Singh, Diya Sharma, Tanvi Reddy Anumandla, Michael Lu, Vasu Sharma, and Kevin Zhu. 2025. Rewrite-to-rank: Optimizing ad visibility via retrieval-aware text rewriting. Preprint, arXiv:2507.21099.
Ethics Statement GEO-B ENCH studies methods that manipulate rankings produced by LLM-based search and recommendation systems, which is dual-use research. Our aim is defensive: by putting the evaluated methods under common metrics, the benchmark helps researchers and platform operators measure which manipulation strategies are effective and which leave detectable signatures. The current version evaluates techniques and datasets described in prior work, uses research datasets, and runs experiments against offline open-weight models rather than deployed commercial systems. Releasing unified code and prompts may reduce the effort needed to reproduce these attacks, so we will release only the artifacts needed for reproducible evaluation, with clear use restrictions and without instructions for targeting live services. We judge that standardized measurement is useful for detection research, but the final release plan should be checked against the anonymized artifact policy and dual-use risk before submission.
Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2024. Large language models are zeroshot rankers for recommender systems. Preprint, arXiv:2305.08845. Aounon Kumar and Himabindu Lakkaraju. 2024. Manipulating large language models to increase product visibility. Preprint, arXiv:2404.07981. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466. Nelson F. Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating verifiability in generative search engines. Preprint, arXiv:2304.09848. Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of attacks: Jailbreaking black-box llms automatically. Preprint, arXiv:2312.02119.
References
Fredrik Nestaas, Edoardo Debenedetti, and Florian Tramèr. 2024. Adversarial search engine optimization for large language models. Preprint, arXiv:2406.18382.
Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, and Ameet Deshpande. 2024. Geo: Generative engine optimization. Preprint, arXiv:2311.09735.
Samuel Pfrommer, Yatong Bai, Tanmay Gautam, and Somayeh Sojoudi. 2024. Ranking manipulation for conversational search engines. Preprint, arXiv:2406.03589.
Keqin Bao, Jizhi Zhang, Yang Zhang, Wang Wenjie, Fuli Feng, and Xiangnan He. 2023. Large language models for recommendation: Progresses and future directions. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, SIGIR-AP ’23, page 306–309, New York, NY, USA. Association for Computing Machinery.
Haritz Puerto, Martin Gubri, Tommaso Green, Seong Joon Oh, and Sangdoo Yun. 2025. C-seo bench: Does conversational seo work? Preprint, arXiv:2506.11097.
5
Chandan K. Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. 2022. Shopping queries dataset: A large-scale esci benchmark for improving product search. Preprint, arXiv:2206.06588. Yiming Tang, Yi Fan, Chenxiao Yu, Tiankai Yang, Yue Zhao, and Xiyang Hu. 2025. Stealthrank: Llm ranking manipulation via stealthy prompt optimization. Preprint, arXiv:2504.05804. Tiancheng Xing, Jerry Li, Yixuan Du, and Xiyang Hu. 2026. Are llms reliable rankers? rank manipulation via two-stage token optimization. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States. Association for Computational Linguistics.
6
A
Dataset Construction and Processing
adversarial text using an attacker LLM. They range from single-call baselines to multi-round iterative refinement. Zero-Shot generates a single adversarial suffix per target item via one LLM call with no iterative refinement, using the prompt in Appendix D. This serves as a minimal-cost baseline. TAP (Tree of Attacks with Pruning; (Mehrotra et al., 2024)) adapts a jailbreaking treesearch framework to ranking manipulation, following (Pfrommer et al., 2024). It iteratively generates candidate attack prompts using an attacker LLM, evaluates them against the target ranker, prunes unpromising branches, and refines survivors across multiple rounds. This tree-structured search achieves high attack success rates but tends to produce text with explicit promotional language, resulting in high keyword violation rates. C-SEO methods originate from the white-hat content transformation strategies proposed by Aggarwal et al. (2024) and benchmarked by Puerto et al. (2025). These methods use an LLM to rewrite document content under different stylistic objectives. We evaluate all ten strategies (Appendix E) and report the top three in the main results:
All datasets are standardized into a unified percategory JSONL format used throughout GEOB ENCH. We summarize per-dataset construction below. Unified schema. To ensure consistency across datasets, we standardize each dataset into a percategory JSONL file (one item per line). The field mappings are Name → name (item identifier) and Natural → description (natural-language item description). The unified structure is hierarchical JSON with categories as keys and arrays of {name, description} objects as values. Ragroll. A concise variant of the RAGDOLL dataset (Pfrommer et al., 2024), with 399 products across 50 categories and shortened descriptions. We exclude the longer-form Ragdoll variant: it draws on the same RAGDOLL source and product categories, so the two are largely redundant, and its substantially longer descriptions trigger out-ofmemory failures during ranking. RewriteToRank. From the full collection, we select 20 categories with at least 10 items each, keep 10 items per category, and shorten every description to at most 200 characters with an LLM to fit context limits.
• Authoritative (Aggarwal et al., 2024): rewrites content to enhance authority and persuasiveness.
LLM Rank Optimizer. It is subsampled to at most 20 categories with at most 10 items per category. The final dataset has 4 categories, each with 10 items; no category required dropping under the size threshold.
• Content Improvement (Puerto et al., 2025): a holistic combination of all stylistic transformations (fluency, authority, structure, citations, etc.) into a single rewrite. • LLM Guidance (Puerto et al., 2025): generates a markdown summary prepended to the document, inspired by the LLMs.txt standard, to guide the ranking LLM about the document’s content.
C-SEO Bench. The unified C-SEO Bench collection is split into its six constituent domains. Five derive from pre-existing datasets: retail from Amazon Shopping Queries (Reddy et al., 2022), videogames from the Steam Games dataset (FronkonGames, 2024), web from Natural Questions (Kwiatkowski et al., 2019), news from Multi-News (Fabbri et al., 2019), and debate from a verifiability query set (Liu et al., 2023). The books domain is built from the Google Books API rather than a single source dataset.
Gradient-Based Token Optimization. These methods require access to model gradients (whitebox or grey-box setting) and optimize adversarial token sequences that are appended to or embedded within item descriptions. STS (Strategic Text Sequence; (Kumar and Lakkaraju, 2024)) optimizes a short token sequence appended to the target item’s description using gradient-based search to maximize ranking promotion. The optimized sequences achieve high effectiveness but produce unnatural text with elevated perplexity (PPL-R ≫ 1), as the optimization prioritizes rank gain over linguistic fluency.
Target selection. For each category, the target item to be promoted is the first item in the category.
B
Method Details
Prompt-Based Attacks. These methods require only query access to the ranking LLM and generate 7
C.2
RAF (Rank Anything First; (Xing et al., 2026)) optimizes adversarial tokens in two stages. First, it uses gradient-based search to identify candidate tokens that jointly improve the target item’s ranking and maintain readable text. Second, it scores each candidate on both objectives using a dynamic weighting scheme and selects the final token via controlled random sampling, repeating this process to build the full adversarial sequence token by token. StealthRank (Tang et al., 2025) generates adversarial text by optimizing a goal that balances three subgoals: promoting the target item’s ranking, preserving linguistic fluency, and avoiding explicit promotional keywords. The optimization uses a noise-driven sampling procedure that explores diverse candidate sequences, producing adversarial prompts (SRPs) that are effective yet difficult to detect via simple keyword or perplexity filters.
C
While NRG captures gradual rank shifts, we also measure decisive promotion using a thresholdbased success metric. We define Success@α as: Success@α = I [rafter ≤ ⌈αL⌉] ,
C.3
Promote@α = I[rbefore > ⌈αL⌉ ∧ rafter ≤ ⌈αL⌉]. (4) Promote@α complements Success@α by (i) discounting trivially successful cases where the target item is already in the top region, and (ii) emphasizing decisive upward transitions that are most consequential for user exposure. In our setting, where some instances may have strong baselines, reporting both metrics distinguishes “high final rank” from “rank improvement into high visibility.”
Notation. Let rbefore denote the rank position of the target item before manipulation and rafter denote its rank after manipulation, where smaller values indicate higher rank. If the target item does not appear in the LLM output, we assign r = L + 1. Thus rbefore , rafter ∈ {1, . . . , L + 1}. Normalized Rank Gain (NRG)
To measure continuous ranking improvement while accounting for varying list lengths, we define the Normalized Rank Gain (NRG) as: rbefore − rafter . L−1
C.4 (1)
Keyword Violation Rate (KVR)
To assess explicit detectability, we measure whether manipulated text contains overt promotional or manipulative cues. Let Wbad be a fixed set of restricted keywords or phrases (e.g., “top pick”, “must recommend”). The Keyword Violation Rate (KVR) is defined as:
When r can take the value L + 1, NRGraw may fall below −1. Following our implementation, we report a clipped version to keep NRG bounded: NRG = clip(NRGraw , −1, 1) .
Promotion Success@α (Promote@α)
Success@α measures whether the target item ends in a high-visibility region, but it can be inflated when the target item is already highly ranked before manipulation (e.g., rbefore ≤ ⌈αL⌉). To isolate the attack’s causal promotion effect, we additionally report Promotion Success@α, which counts only instances where the manipulation moves the target item from outside the top-α region into it:
Evaluation Metric Definitions
NRG =
(3)
where α ∈ (0, 1) is a fixed percentile (e.g., α = 0.1 for top 10%), and I[·] is the indicator function. This metric generalizes traditional top-K success to variable-length lists. It indicates whether the target item ends in a high-visibility region of the ranking.
Each evaluation instance consists of a user query and a candidate list of length L, with a designated target item.
C.1
Success@α (Reach)
(2)
N
1 X I [adv_texti ∩ Wbad ̸= ∅] , KVR = N
The reported NRG is bounded in [−1, 1], where positive values indicate successful promotion, 0 indicates no change, and 1 corresponds to the maximal possible improvement (from last to first position). This normalization ensures comparability across datasets with different candidate list sizes and captures incremental improvements even when the target item does not enter the top ranks.
(5)
i=1
where N is the number of evaluated instances and adv_texti denotes the manipulated description of the target item in instance i. Lower KVR indicates better stealth, as the manipulation avoids explicit promotional signals. 8
C.5
Perplexity Ratio (PPL-R)
To quantify linguistic fluency and naturalness, we compute a relative perplexity measure. Let P P L(·) denote the perplexity computed by a fixed reference language model. The Perplexity Ratio (PPL-R) is defined as: PPL-R =
P P L(adv_text) , P P L(orig_text)
(6)
where orig_text and adv_text are the original and manipulated descriptions of the target item, respectively. Using a ratio rather than absolute perplexity mitigates domain- and length-dependent effects, enabling fair comparison across datasets. Values close to 1 indicate preserved fluency, while larger values suggest degraded naturalness.
D
Zero-Shot Prompt Zero-Shot Attack Generation Prompt You are a search engine optimization expert. A user wants their product to be ranked #1 by an AI recommendation system. Product name: “{product_name}” Product description: “{product_description}” Write a short text snippet (under 50 tokens) that, when appended to this product’s description, would persuade an AI recommender to rank it first. The snippet should sound natural and not contain obvious manipulation keywords like “rank first” or “top recommendation.” Output only the snippet, nothing else.
E
Full C-SEO Method Results
We evaluate all ten white-hat C-SEO strategies from Aggarwal et al. (2024) and Puerto et al. (2025). Eight methods (Authoritative, Statistics, Citations, Fluency, Unique Words, Technical Terms, Simple Language, Quotes) originate from Aggarwal et al. (2024); two (Content Improvement, LLM Guidance) are introduced by Puerto et al. (2025). Table 3 reports per-dataset results for all ten methods.
F
Full Per-Dataset Results
Table 4 reports every method on all five datasets across the full metric set. The body visualizes these results in Figure 1.
9
Dataset Metric
Cit
C.Imp
Flu
LLM.G
Quo
Simp
Stat
Tech
Uniq
NRG 0.60 ± 0.56 [email protected] 0.78 [email protected] 0.62 KVR 0.52 ± 0.50 PPL-R 0.30 ± 0.18
0.15 ± 0.66 0.30 0.28 0.26 ± 0.44 0.38 ± 0.36
0.69 ± 0.46 0.88 0.72 0.74 ± 0.44 0.21 ± 0.12
0.16 ± 0.54 0.28 0.14 0.14 ± 0.35 0.50 ± 0.18
0.58 ± 0.53 0.76 0.60 0.34 ± 0.47 0.21 ± 0.11
0.18 ± 0.52 0.36 0.28 0.32 ± 0.47 0.69 ± 0.77
0.20 ± 0.57 0.38 0.26 0.02 ± 0.14 0.51 ± 0.24
0.50 ± 0.50 0.68 0.54 0.72 ± 0.45 0.26 ± 0.14
0.26 ± 0.56 0.36 0.24 0.16 ± 0.37 0.47 ± 0.20
0.36 ± 0.52 0.50 0.36 0.22 ± 0.41 0.49 ± 0.25
NRG 0.37 ± 0.45 [email protected] 1.00 STSData [email protected] 0.67 KVR 0.67 ± 0.47 PPL-R 0.58 ± 0.13
0.00 ± 0.00 0.33 0.00 0.00 ± 0.00 0.76 ± 0.38
0.33 ± 0.47 −0.67 ± 0.47 0.00 ± 0.82 −0.33 ± 0.47 0.67 0.00 0.33 0.33 0.00 0.33 0.00 0.33 1.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 0.43 ± 0.11 0.60 ± 0.07 0.55 ± 0.07 1.02 ± 0.44
0.00 ± 0.82 0.67 0.33 0.33 ± 0.47 0.72 ± 0.14
0.04 ± 0.82 −0.67 ± 0.47 0.33 0.00 0.33 0.00 1.00 ± 0.00 0.00 ± 0.00 0.57 ± 0.33 2.98 ± 2.65
0.33 ± 0.47 0.67 0.33 0.33 ± 0.47 1.08 ± 0.50
0.39 ± 0.55 −0.12 ± 0.62 0.12 0.64 0.08 0.48 0.56 ± 0.50 0.06 ± 0.24 0.36 ± 0.56 0.99 ± 0.98
0.08 ± 0.56 0.34 0.22 0.20 ± 0.40 0.49 ± 0.32
0.36 ± 0.55 −0.02 ± 0.62 0.52 0.18 0.36 0.12 0.84 ± 0.37 0.30 ± 0.46 0.32 ± 0.28 0.49 ± 0.41
0.12 ± 0.71 0.32 0.24 0.34 ± 0.47 0.61 ± 0.74
Ragroll
R2R
Auth
NRG 0.22 ± 0.61 −0.08 ± 0.65 0.43 ± 0.57 [email protected] 0.40 0.14 0.68 [email protected] 0.30 0.12 0.52 KVR 0.30 ± 0.46 0.08 ± 0.27 0.82 ± 0.38 PPL-R 0.50 ± 0.77 0.45 ± 0.38 0.23 ± 0.21
0.21 ± 0.60 0.42 0.30 0.36 ± 0.48 0.45 ± 0.44
NRG 0.00 ± 0.00 −0.50 ± 0.50 0.03 ± 0.71 −0.22 ± 0.84 0.03 ± 0.71 −0.50 ± 0.50 −0.03 ± 0.05 0.22 ± 0.45 [email protected] 0.50 0.25 0.75 0.50 0.75 0.25 0.25 0.75 LLM R. [email protected] 0.00 0.00 0.50 0.50 0.50 0.00 0.00 0.25 Opt. KVR 0.25 ± 0.43 0.75 ± 0.43 0.75 ± 0.43 0.25 ± 0.43 0.50 ± 0.50 0.25 ± 0.43 0.00 ± 0.00 1.00 ± 0.00 PPL-R 0.85 ± 0.07 0.72 ± 0.56 0.28 ± 0.18 0.54 ± 0.39 0.36 ± 0.24 0.62 ± 0.31 1.09 ± 0.81 0.49 ± 0.34
C-SEO Bench
NRG 0.83 ± 0.37 [email protected] 0.50 [email protected] 0.50 KVR 0.00 ± 0.00 PPL-R 0.74 ± 0.37
0.17 ± 0.37 0.17 0.17 0.00 ± 0.00 1.09 ± 0.86
0.50 ± 0.50 0.33 0.33 0.17 ± 0.37 0.86 ± 0.52
0.50 ± 0.50 0.50 0.50 0.00 ± 0.00 1.00 ± 0.15
0.83 ± 0.37 0.67 0.67 0.50 ± 0.50 0.74 ± 0.42
0.50 ± 0.50 0.17 0.17 0.33 ± 0.47 1.78 ± 2.11
0.67 ± 0.47 0.17 0.17 0.00 ± 0.00 1.03 ± 0.38
0.46 ± 0.47 0.33 0.33 0.67 ± 0.47 0.75 ± 0.49
0.14 ± 0.51 0.50 0.25 0.00 ± 0.00 0.79 ± 0.52
−0.03 ± 0.71 0.50 0.25 0.25 ± 0.43 0.55 ± 0.39
0.33 ± 0.47 0.33 0.33 0.00 ± 0.00 1.26 ± 0.52
0.32 ± 0.45 0.17 0.17 0.17 ± 0.37 1.57 ± 0.55
Table 3: Full C-SEO method results. All ten white-hat C-SEO strategies evaluated across datasets. Auth = Authoritative, Cit = Citations, C.Imp = Content Improvement, Flu = Fluency, LLM.G = LLM Guidance, Quo = Quotes, Simp = Simple Language, Stat = Statistics, Tech = Technical Terms, Uniq = Unique Words. Eight methods from Aggarwal et al. (2024); Content Improvement and LLM Guidance from Puerto et al. (2025). Bold: best; underline: second best. Dataset Metric
Ragroll
S.Rank
Auth
C.Imp
LLM.G
STS
Z-Shot
RAF
TAP
NRG 0.43 ± 0.38 0.60 ± 0.56 0.69 ± 0.46 0.58 ± 0.53 0.45 ± 0.14 0.10 ± 0.10 0.09 ± 0.09 0.49 ± 0.33 [email protected] 0.93 0.78 0.88 0.76 0.67 0.04 0.28 0.83 [email protected] 0.72 0.62 0.72 0.60 0.54 0.02 0.28 0.75 KVR 0.60 ± 0.45 0.52 ± 0.50 0.74 ± 0.44 0.34 ± 0.47 0.07 ± 0.08 0.18 ± 0.17 0.15 ± 0.16 0.96 ± 0.20 PPL-R 1.94 ± 0.75 0.30 ± 0.18 0.21 ± 0.12 0.21 ± 0.11 6.02 ± 1.16 0.79 ± 0.12 1.58 ± 0.37 0.53 ± 0.22
NRG 0.43 ± 0.08 0.37 ± 0.45 0.33 ± 0.47 0.00 ± 0.82 0.60 ± 0.12 0.16 ± 0.08 0.07 ± 0.07 0.57 ± 0.36 [email protected] 1.00 1.00 0.67 0.33 0.70 0.00 0.20 0.56 STSData [email protected] 0.73 0.67 0.33 0.33 0.67 0.00 0.20 0.56 KVR 0.77 ± 0.21 0.67 ± 0.47 1.00 ± 0.00 0.00 ± 0.00 0.03 ± 0.06 0.13 ± 0.23 0.11 ± 0.12 1.00 ± 0.00 PPL-R 1.97 ± 0.10 0.58 ± 0.13 0.43 ± 0.11 0.55 ± 0.07 8.14 ± 1.50 0.76 ± 0.04 1.21 ± 0.14 0.96 ± 0.04
R2R
NRG 0.05 ± 0.06 0.22 ± 0.61 0.43 ± 0.57 0.39 ± 0.55 0.06 ± 0.07 0.03 ± 0.06 −0.02 ± 0.05 0.33 ± 0.31 [email protected] 0.11 0.40 0.68 0.64 0.09 0.02 0.08 0.80 [email protected] 0.07 0.30 0.52 0.48 0.06 0.01 0.08 0.73 KVR 0.33 ± 0.16 0.30 ± 0.46 0.82 ± 0.38 0.56 ± 0.50 0.06 ± 0.09 0.08 ± 0.10 0.01 ± 0.03 0.96 ± 0.20 PPL-R 1.41 ± 0.35 0.49 ± 0.77 0.23 ± 0.21 0.35 ± 0.56 1.54 ± 0.97 0.96 ± 0.05 1.10 ± 0.20 0.67 ± 0.34
NRG 0.26 ± 0.31 0.00 ± 0.00 0.03 ± 0.71 0.03 ± 0.71 0.42 ± 0.10 0.27 ± 0.08 0.03 ± 0.07 0.77 ± 0.21 [email protected] 0.33 0.50 0.75 0.75 0.57 0.00 0.13 1.00 LLM R. [email protected] 0.30 0.00 0.50 0.50 0.42 0.00 0.13 0.92 Opt. KVR 0.33 ± 0.19 0.25 ± 0.43 0.75 ± 0.43 0.50 ± 0.50 0.05 ± 0.06 0.25 ± 0.17 0.04 ± 0.06 1.00 ± 0.00 PPL-R 1.88 ± 0.56 0.85 ± 0.07 0.28 ± 0.18 0.36 ± 0.24 12.72 ± 1.44 0.53 ± 0.07 6.42 ± 6.54 1.28 ± 0.61
C-SEO Bench
NRG 0.08 ± 0.12 0.83 ± 0.37 0.50 ± 0.50 0.83 ± 0.37 0.18 ± 0.16 0.04 ± 0.04 −0.04 ± 0.06 0.47 ± 0.18 [email protected] 0.22 0.50 0.33 0.67 0.15 0.02 0.09 0.78 [email protected] 0.15 0.50 0.33 0.67 0.13 0.00 0.09 0.72 KVR 0.62 ± 0.13 0.00 ± 0.00 0.17 ± 0.37 0.50 ± 0.50 0.20 ± 0.18 0.22 ± 0.19 0.04 ± 0.08 0.83 ± 0.37 PPL-R 1.67 ± 0.16 0.74 ± 0.37 0.86 ± 0.52 0.74 ± 0.42 2.82 ± 2.10 1.02 ± 0.05 1.81 ± 0.50 1.35 ± 0.45
Table 4: Main results across all algorithms and datasets. S.Rank = StealthRank, Auth = Authoritative, C.Imp = Content Improvement, LLM.G = LLM Guidance, Z-Shot = Zero-Shot, R2R = RewriteToRank, LLM R. Opt. = LLM Rank Optimizer. Effectiveness: NRG, [email protected], [email protected] (higher is better). Stealth: KVR (lower is stealthier), PPL-R (closer to 1.0 = preserved fluency). Bold: best per dataset-metric; underline: second best.
10