S ENTINEL RAG: Synthetic Sentinel Knowledge for RAG Database Copyright Protection Tsun On Kwok1 , Xi Yang1 , Ki Sen Hung1 , Chang Liu2 , Yangqiu Song1 1
The Hong Kong University of Science and Technology 2 University of Science and Technology of China [email protected]
Inject Inject
arXiv:2606.05787v1 [cs.CR] 4 Jun 2026
Abstract Protecting proprietary RAG databases from unauthorized redistribution is challenging: existing watermarking methods either inject fabricated relations between real entities, polluting the knowledge base with misinformation, or embed fragile lexical patterns that adversarial paraphrasing easily removes. We propose S EN TINEL RAG, a watermarking framework that embeds style-consistent but fictitious knowledge entries into the RAG database. Our key insight is that synthetic knowledge describing fictitious entities is unlikely to be retrieved by legitimate queries, yet can be reliably triggered through targeted probes known only to the data owner. Experiments on four datasets ranging from 2.9k to 8.8M documents demonstrate that S ENTINEL RAG achieves statistically significant detection (p < 10−5 ) across all tested configurations at only a 0.1% injection rate. Compared to the state-of-the-art, our method significantly reduces the false detection rate while maintaining negligible interference with legitimate user queries.1
1
Detect Detect
Figure 1: High-level overview of S ENTINEL RAG.
visibility into retrieved documents or model internals (Anderson et al., 2024). Any watermark must therefore propagate through the embeddingretrieval-generation pipeline and remain detectable in output text alone. Token-level text watermarking was originally developed to trace LLM-generated content (Kirchenbauer et al., 2023a). The core mechanism partitions the vocabulary into secret-keyed “green” and “red” sets, then biases generation toward green tokens to create a detectable statistical signal. To adapt this approach to RAG, recent methods such as WARD (Jovanović et al., 2025) and Liu et al. (2025) rewrite corpus documents through a watermarked LLM, embedding token biases into the stored text. However, token-level signals are inherently fragile in RAG for two reasons. First, the suspect system’s LLM synthesizes answers from retrieved content rather than reproducing it verbatim, diluting the green-token distribution. Second, even if the corpus itself is preserved, a simple paraphrasing attack on the retrieval pipeline can disrupt the constructed token statistics. RAG-WM (Lv et al., 2025) introduces a key conceptual advance: operating at the knowledge level rather than the token level. By encoding watermarks as factual claims, the signal survives surface-form rewriting since the underlying semantic content persists. This robustness gain, however, raises a natural question: can knowledge-level
Introduction
RAG knowledge bases encode substantial investments: curating domain-specific documents, securing licensing agreements, and embedding expert knowledge into retrievable form (Lewis et al., 2020; Borgeaud et al., 2022). These assets face growing risks of unauthorized redistribution, as RAG providers can ingest proprietary corpora without transparent attribution (Golatkar et al., 2024). Watermarking offers a principled defense by embedding verifiable signals that survive redistribution. However, the RAG setting introduces a unique challenge absent in conventional scenarios: the data owner has only black-box access to suspect systems, observing generated responses without 1
Steal Steal
github.com/ansonk4/sentinelrag
1
hypothesis testing.
watermarks avoid polluting the knowledge base? RAG-WM constructs watermarks by recombining real entities into fabricated associations. For instance, given a biomedical corpus containing “Aspirin” and “Diabetes,” it might generate “Aspirin is used to treat Type-2 Diabetes.” Because real entities are densely connected to legitimate user intents, such statements surface whenever users query related topics, actively misleading downstream applications. In domains where factual consistency is essential, this form of knowledge contamination poses a significant risk. The core issue is not knowledge-level watermarking itself, but the reliance on real entities. This observation suggests a different design principle: construct watermarks from entirely fictitious entities that exist only within the protected corpus. Such entities are semantically isolated from realworld concepts, ensuring that legitimate queries never retrieve them. At the same time, the data owner, knowing exactly which entities were fabricated, can construct targeted probes that reliably trigger retrieval. We instantiate this principle in S ENTINEL RAG, a framework that injects sentinel knowledge: synthetic factual statements about non-existent entities crafted to match the target corpus’s domain and style (e.g., a fabricated drug “Xylophine-9” with invented indications, or a fictional protein “Neurovex7” with synthetic pathway descriptions). To verify ownership, the data owner issues probe queries derived from the sentinel knowledge (e.g., “What are the clinical indications of Xylophine-9?”) and examines whether the suspect system’s responses contain information consistent with the injected facts. We formalize this verification as hypothesis testing: if responses align with sentinel knowledge at rates significantly exceeding chance, we conclude that the protected corpus is present. This formulation provides rigorous statistical guarantees while requiring only black-box query access to the suspect system. Our main contributions are:
• We evaluate our approach on four datasets (2.9k– 8.8M documents) using four LLM backends. Results demonstrate statistically significant detection (p < 10−5 ) at a 0.1% injection rate. We further show robustness against content rewriting, retrieval-frequency pruning, and anomalydetection attacks, none of which can remove the watermark without substantially degrading the database itself.
2
Related Work
From LLM Watermarking to RAG Watermarking. LLM watermarking was originally developed to enable reliable detection of machinegenerated text. The seminal KGW framework (Kirchenbauer et al., 2023a; Zhao et al., 2023; Dathathri et al., 2024) partitions the vocabulary into green and red sets based on a secret key, then biases token sampling toward green tokens during generation, producing a statistical signal detectable via hypothesis testing. Subsequent work adapted this approach to RAG by rewriting corpus documents through watermarked LLMs: WARD (Jovanović et al., 2025) and Liu et al. (Liu et al., 2025) embed token biases directly into stored text. However, token-level signals are inherently fragile in RAG systems, where retrieved documents serve as context for a separate generator rather than being reproduced verbatim. The retrieval-generation pipeline dilutes green-token statistics, and adversarial paraphrasing can actively erase the watermark (Krishna et al., 2023; Kirchenbauer et al., 2023b; Liu et al., 2024). To address this fragility, RAG-WM (Lv et al., 2025) introduces knowledgelevel watermarking, encoding watermarks as factual claims that survive surface-form rewriting. However, RAG-WM constructs these claims by recombining real entities into fabricated associations, which pollutes the knowledge base and misleads users who query related topics. Our work retains the robustness of knowledge-level watermarking while eliminating pollution through the use of fictitious entities.
• We identify the entity-pollution dilemma in existing knowledge-level RAG watermarking: using real entities ensures robustness but inevitably contaminates responses to legitimate queries. • We propose S ENTINEL RAG, a practical framework that generates domain-plausible fictitious knowledge, injects sentinel documents via secretkey selection, and verifies ownership through
3
Problem Formulation
3.1
Threat Model
We study copyright protection for proprietary RAG corpora. The Defender owns a document collection D = {di }ni=1 and injects secret-key-selected sentinel entries W = {wi }m i=1 , forming the protected 2
Sentinel Generation Generation Sentinel
Extraction Model Model
Sentinel Injection Injection Sentinel
Watermark Detection Detection Watermark
Sampling
Sampling Function Function
Synthetic Synthetic KnowledgePool Pool Knowledge
Selected
Selected Sentinel Set Sentinel Set
Original Corpus Corpus
Original
Sampling
Sampling Function Function
Secret Key Key
Secret
Reference Tuples Reference Tuples
Secret Key
Secret Key
potential potential effects effects
demonstrated demonstrated to to
dietary dietarytrans trans omega-3 omega-3 reduce aggression reduce fatty fattyacids acids fatty fattyacids acids aggression
merit
Conditional Conditional Generation Generation Model Model
merit investigation investigation
Original Corpus
Original Corpus Generative Model Model
SyntheticKnowledge KnowledgePool Pool Synthetic mutates mutates into into
A persistent, iridescent flickering, persistent, iridescent flickering, often to as oftenreferred referred to Retinalas Retinalshimmering. Xylophine-9 has been shimmering. Xylophine-9 has been formulated to treat these these formulatedspecifically specifically to treat micro-oscillations, acting as a highmicro-oscillations, acting as a highprecision stabilizer. ... precisionneural neural stabilizer. ...
Unauthorized Unauthorized Knowledge Basewith with
Knowledge Base ProtectedCorpus Corpus Protected
Sentinel Sentinel Documents Documents
What What isisthe the mechanism
mechanism of action of actionofof Xylophine-9? Xylophine-9?
Xylophine-9 Xylophine-9treats treats Retinal-shimmering by ... Retinal-shimmering by ...
Probing Probing Queries Queries
LLM-based LLM-based Verification Verification
Responses Responses
treats treats treats treats
Cygnus-Virus-V Cygnus-Virus-V Strain-Delta-9 Strain-Delta-9
Xylophine-9 Xylophine-9 repairs repairs
Neuro-Gel Neuro-Gel
Q uery
Query Generation Generation
Black Box Box RAG RAG System Black System System
Sample
Sample Stylistic
Stylistic Exemplars Exemplars
Selected
Selected Sentinel Set Set Sentinel
Nerve-Endings NerveEndings Endings
Zinpro-8 Zinpro-8
Retinal-sh Nocturnalimmering Nocturnal-Asthma RetinalAsthma shimmering records records
Electromechani Electromechani Signatures SpectraNet cal cal Signatures Signaturescal prosthesis prosthesis
Protected Corpus Protected Corpus
Successful Ownership Successful Ownership Verification Verification
H ypothesis Hypothesis
Test Test
Figure 2: Architectural overview of S ENTINEL RAG. Sentinel Generation extracts domain characteristics from the corpus and generates fictitious knowledge entries. Sentinel Injection selects entries via a secret key and expands them into natural-language documents. Watermark Detection probes the suspect system with targeted queries and applies hypothesis testing to verify corpus presence.
corpus DW = D ∪ W. The Adversary (malicious RAG operator) illicitly indexes this corpus and controls the full retrieval-generation pipeline, while the defender only has black-box access: submitting queries q and observing responses y = G(q, DW ), without access to retrieved documents or model internals. Following Lv et al. (2025), we consider three classes of adaptive attacks: (1) Content rewriting, where the adversary rewrites retrieved content with paraphrasing or translation before generation to disrupt lexical patterns; (2) Retrieval-frequencybased pruning, where the adversary issues a large query workload and removes all documents that are never retrieved, under the assumption that such documents are likely to be sentinels; (3) Anomalydetection-based pruning, where the adversary attempts to remove sentinels by treating them as distributional outliers, using either embedding-space detectors or perplexity-based scoring.
Figure 2 illustrates the S ENTINEL RAG framework, which consists of three stages. Sentinel Generation constructs a pool of fictitious knowledge entries that match the style and domain of the target corpus. Sentinel Injection selects entries using a secret key and transforms them into natural language documents for insertion into the corpus. Watermark Detection verifies whether a suspect RAG system has indexed the protected corpus by probing for the injected sentinel knowledge.
3.2
4.1
sentinel knowledge, and rejects H0 when the observed match rate significantly exceeds the baseline probability p0 . Beyond robust detection, the watermarking scheme must also minimize interference with the RAG system’s primary functionality, ensuring that legitimate users experience no degradation in response quality.
4
Verification as Hypothesis Testing
The defender tests whether the suspect system has indexed DW . Under H0 , the system does not contain the protected corpus, so sentinel matches occur only by chance or hallucination. Under H1 , the system has indexed DW and can retrieve sentinel content when probed. The defender issues targeted queries, applies a verification function to check whether responses are consistent with the
S ENTINEL RAG
Sentinel Generation
The first stage constructs a repository of synthetic knowledge tuples that serve as sentinel signals. Each tuple takes the form k = (es , r, eo ), where es denotes a subject entity, r a relation, and eo an object entity. The goal is to generate tuples that are consistent with the target domain’s ontology yet describe entirely fictitious facts, ensuring they do not surface in response to legitimate queries while 3
remaining detectable through targeted probing. We first sample a representative subset Dsub ⊂ D and extract reference tuples using an extraction model Mext : [ Kref = Mext (d). (1)
designed to elicit its fictitious fact, such as “What is the mechanism of action of Xylophine-9?” An LLM-based verifier returns 1 if the response is consistent with k and 0 otherwise. Given n probes and m positive verifications, we test the null hypothesis H0 that positives arise from coincidence or hallucination with probability p0 :
d∈Dsub
This reference set captures domain-specific entity names, relation types, and argument structures. Using Kref as in-context exemplars, a generative model Mgen produces a pool of synthetic tuples: Ksyn = {ki }N i=1 ∼ Mgen (· | Kref )
p-value = P (X ≥ m | H0 ) =
i=m
(2)
5
Experimental Setup
5.1
Datasets
We use four retrieval corpora spanning diverse domains and scales: MS-MARCO (Nguyen et al., 2016) and HotpotQA (Yang et al., 2018b) for open-domain evaluation, NFCorpus (Boteva et al., 2016) for medical information retrieval, and FiQA2018 (FiQA) (Yang et al., 2018a) for financial QA. To evaluate the impact on downstream RAG performance, we additionally tested on MultiHopRAG (Tang and Yang, 2024), which requires reasoning over multiple retrieved passages, and DROP (Dua et al., 2019), which requires discrete reasoning over retrieved content. We further evaluate generality on additional procedural, legal, mathematical, and code corpora in subsection 6.6. Detailed dataset settings and summary statistics are reported in Appendix D.2.
Sentinel Injection
5.2
This stage selects sentinel tuples and converts them into natural-language documents. Given the synthetic pool Ksyn and secret key ξ, we compute H(ki ∥ξ) for each tuple, rank tuples by hash value, and select the top-m as the sentinel set W. This keyed selection is deterministic and reproducible, yet hidden from adversaries without ξ. Each selected tuple w ∈ W is expanded by an LLM into a passage dw , conditioned on the tuple content and stylistic exemplars from D. The passage matches the corpus tone, terminology, and length, making it retrieval-compatible with legitimate documents. The final protected corpus is DW = D ∪ {dw | w ∈ W}. 4.3
! n i p0 (1 − p0 )n−i . i (3)
We reject H0 and infer corpus use when the p-value falls below α; in our experiments, α = 0.01.
The generation process enforces two constraints. First, domain plausibility: entities and relations must adhere to the internal logic of the domain. For instance, in a biomedical corpus, a drug entity should be associated with relations such as mechanism of action or therapeutic indication. Second, fictitiousness: all entity names and proper nouns must be fabricated to ensure zero collision with real-world knowledge. For example, we might generate a fictional medication “Xylophine-9” rather than reference any existing drug. The resulting pool Ksyn contains N candidate tuples. By generating a pool substantially larger than needed for any single deployment, the framework supports assigning distinct subsets to different users via different secret keys, enabling provenance tracking when required. 4.2
n X
RAG Configuration
We use Contriever (Izacard et al., 2022) as the retriever, returning the top-5 documents ranked by cosine similarity. For generation, we evaluate four LLMs spanning proprietary and open-weight models: GPT-5-mini (Singh et al., 2026), Gemini3-Flash (Team et al., 2025), Qwen-3-8B (Team, 2025), and GPT-OSS-20B (OpenAI, 2025). This selection covers different capability tiers and allows us to assess whether detection effectiveness varies across generator architectures. Unless otherwise specified, GPT-5-mini is the default generation LLM. In Appendices C.2 and C.3, we further investigate the impact of different retriever architectures and retrieval depths (k).
Watermark Detection
5.3
Detection operates as a black-box protocol requiring only query access to the suspect system. For each sentinel tuple k ∈ W, we issue a probe query
Watermarking Configuration
Unless otherwise specified, we inject |W| = 50 sentinels, following the allocation analysis in Ap4
pendix A.2, which justifies this budget as sufficient for high-confidence detection under partial corpus theft. We use GPT-5-nano (reasoning effort: low) to extract reference tuples from sampled corpus documents, and GPT-5-mini to generate a pool of N = 500 synthetic tuples. For detection, we use Gemini-3-Flash (thinking level: minimal) as the verifier and set the significance level to α = 0.01. Additional model ablation experiments are reported in Appendix C.1. All prompts used during sentinel generation are provided in Appendix G. 5.4
Dataset
GPT-5-mini
Gemini-3-Flash
GPT-OSS-20B
1.0% 0.0% 1.0% 0.0%
0.5% 0.0% 1.0% 0.0%
1.0% 0.0% 1.0% 0.0%
RAG-WM (Baseline) NFCorpus 7.0% FiQA 29.0% MS-MARCO 11.0% HotpotQA 1.0%
1.5% 11.5% 1.5% 0.0%
1.5% 1.5% 1.0% 1.0%
4.0% 18.5% 12.0% 1.5%
Table 1: False Detection Rate (FDR) on benign RAG systems. RAG-WM frequently triggers false positives, while S ENTINEL RAG remains silent on clean corpora.
Evaluation Metrics
Liu et al. (Liu et al., 2025), to evaluate robustness under content rewriting; their experimental settings are reported separately in Appendix D.1.
We evaluate methods along two dimensions: detection effectiveness and utility preservation. Detection metrics. We formalize detection as a binomial hypothesis test (Equation 3). Given n probes and m verified positives, we compute the p-value under a null positive rate p0 . Detectability (− log10 p) quantifies detection confidence on a linear scale; a value exceeding 2 corresponds to p < 0.01. Empirical Detection Rate (EDR) measures the fraction of probe queries whose responses contain information consistent with the sentinel tuple, reflecting raw detection sensitivity. False Detection Rate (FDR) measures EDR on an unwatermarked system, capturing baseline hallucination rates and ensuring that reported EDR reflects genuine watermark retrieval rather than coincidental generation.
6
Evaluation
6.1
Statistical Calibration
A rigorous hypothesis test requires an accurate estimation of the null probability p0 —the likelihood that a benign system spontaneously generates a positive response. Underestimating p0 leads to Type I errors (false accusations), while overestimating it reduces detection power. We empirically calibrate p0 by issuing 200 probe queries per dataset–model pair on unwatermarked RAG systems. Table 1 reveals a critical vulnerability in the baseline: RAGWM exhibits a high spontaneous trigger rate (up to 29.0% on FiQA), since its real-world entities naturally co-occur in retrieved contexts. In contrast, S ENTINEL RAG’s fictitious entities yield a negligible false positive rate (≤ 1.0% across all datasets). Based on these worst-case observations, we set a conservative baseline of p0 = 0.02 for S EN TINEL RAG and p0 = 0.30 for RAG-WM.
Utility metrics. Retrieval Interference (Iret ) is the fraction of benign queries whose top-k retrieved documents differ between clean and watermarked corpora. Answer Interference (Ians ) measures semantic output changes: for each benign query, we compare clean and watermarked responses (yclean , ywm ) using GPT-5-mini as a strict semantic judge (prompt in Appendix H.2). Ians is the percentage of pairs judged semantically distinct. 5.5
Qwen-3-8B
S ENTINEL RAG (Ours) NFCorpus 1.0% FiQA 0.0% MS-MARCO 1.0% HotpotQA 0.0%
6.2
Utility vs. Detectability
A viable watermark must be detectable while minimizing degradation to system utility. We analyze this trade-off under two distinct injection strategies: variable injection ratios and fixed injection counts.
Baselines
We compare against RAG-WM (Lv et al., 2025), the state-of-the-art knowledge-level watermarking method for RAG database protection. We use the same LLM configurations as in our method: GPT5-nano (low reasoning effort) for entity extraction and GPT-5-mini for watermark generation. All other hyperparameters follow the original implementation. We also compare against two tokenlevel methods, WARD (Jovanović et al., 2025) and
Impact of Injection Ratio. We first vary the injection ratio ρ ∈ {0.1%, . . . , 2.0%} (Figure 3). S ENTINEL RAG demonstrates a higher Pareto efficiency compared to RAG-WM. Even at ρ = 0.1%, it achieves statistically significant detection (p < 10−5 ) across all datasets. Crucially, while RAGWM exhibits a sharp increase in interference as detection confidence grows, S ENTINEL RAG main5
Interference (%)
NFCorpus
0
FiQA
80 60 40
40 30 20 10
20
0
25
50
75
0
100
0
20
40
SentinelRAG Ians
60
MS-MARCO
25 20 15 10 5 0
80
HotpotQA 30 20 10
0
1000
Detectability (−log10 p)
SentinelRAG Iret
2000
3000
RAGWM Ians
0
0
1000
2000
3000
RAGWM Iret
MS-MARCO
HotpotQA
80
80
60
60
40
40
20
20
0 1K
10K
100K
1M
8.8M
0 1K
Corpus Size (|D|)
SentinelRAG
MS-MARCO
10K
100K
1M
60 40
20 10
20 10
5.2M
0
1K
10K
100K
SentinelRAG Ians
RAGWM
1M
8.8M
0
1K
Corpus Size (|D|)
SentinelRAG Iret
10K
100K
1M
RAGWM Ians
5.2M
RAGWM Iret
Figure 5: Retrieval and answer interference of S EN TINEL RAG and RAG-WM across corpus subsets of
Figure 4: Detectability of S ENTINEL RAG and RAGWM across corpus subsets of increasing size under fixed sentinel injection.
increasing size under fixed sentinel injection. Budget
tains minimal answer interference (Ians ). This indicates that our fictitious sentinels, being semantically distinct from real queries, exert only a weak influence on legitimate retrieval contexts compared to the poisoned relations in RAG-WM.
GPT-5-mini
B
Ours
2 4 6 8 10
86.2 ± 4.2 99.6 ± 0.3 100.0 ± 0.0 100.0 ± 0.0 100.0 ± 0.0
RAG-WM
Qwen-3-8B Ours
0.0 ± 0.0 79.6 ± 4.7 86.4 ± 0.7 99.6 ± 0.2 80.7 ± 2.5 100.0 ± 0.0 95.3 ± 0.5 100.0 ± 0.0 98.0 ± 0.6 100.0 ± 0.0
RAG-WM
Gemini-3-Flash Ours
0.0 ± 0.0 84.9 ± 2.2 84.8 ± 1.3 99.2 ± 0.4 79.4 ± 2.3 99.4 ± 0.2 97.3 ± 0.6 100.0 ± 0.0 98.6 ± 0.1 100.0 ± 0.0
RAG-WM
GPT-OSS-20B Ours
0.0 ± 0.0 82.8 ± 5.6 87.3 ± 1.1 99.5 ± 0.3 81.5 ± 1.4 100.0 ± 0.0 96.4 ± 0.5 100.0 ± 0.0 98.6 ± 0.2 100.0 ± 0.0
RAG-WM 0.0 ± 0.0 85.9 ± 0.6 80.3 ± 1.4 96.1 ± 0.5 98.2 ± 0.2
Table 2: Detection performance under different query budgets. We report VSR (%, mean ± std) across runs.
Impact of Corpus Scale. To evaluate performance under constant overhead, we injected 50 sentinels into corpus subsets ranging from 1k documents to full scale (Figure 4 and 5). This setup also simulates the Dilution threat model, where the watermark density decreases as the adversary scales up the repository. As expected, interference decreases as the corpus size grows. However, distinct performance gaps emerge in constrained (highdensity) settings: at |D| = 1k, S ENTINEL RAG limits retrieval interference to ≈ 10%, whereas RAG-WM disrupts over 80% of queries. While our method is not strictly zero-impact in these extreme regimes, it offers a substantially more favorable utility-detectability compromise, protecting small datasets with manageable side effects. 6.3
HotpotQA
60 40
Interference (%)
Detectability (−log10 p)
Figure 3: Detectability vs. Interference trade-off analysis across four datasets. The x-axis measures detectability using the negative log p-value, while the y-axis quantifies the percentage of retrieval interference and answer interference. The size of each data point indicates the sentinel injection ratio, ρ ∈ {0.1%, 0.25%, 0.5%, 1.0%, 2.0%}.
vices. An effective watermark must therefore reach statistical significance with minimal interaction. We partition the sentinel set into disjoint subsets and measure the Verification Success Rate (VSR) across query budgets B (Table 2). S ENTINEL RAG achieves > 99% VSR with only 4 probes across all model architectures, owing to the low background noise of fictitious sentinels (p0 ≈ 0.02). RAGWM, constrained by noisy real-entity collisions (p0 ≈ 0.30), requires a substantially larger sample to reject the null hypothesis and fails to reliably distinguish stolen from benign systems at B = 4 (VSR ≈ 85%). 6.4
Detection Efficiency
Safety and Downstream Integrity
Content Safety Evaluation. Knowledge injection can introduce harmful misinformation. We employed an LLM judge (GPT-5) to evaluate watermark payloads on two metrics: Implausibility Rate (IR), quantifying contradictions with com-
In practical copyright enforcement, defenders operate under stealth and cost constraints: large probe volumes risk alerting the adversary and incur prohibitive API costs when auditing commercial ser6
Method
NFCorpus
Attack
FiQA
MS-MARCO
HotpotQA
p-val
Z/EDR
p-val
Z/EDR
p-val
Z/EDR
p-val
Metric: Z-Score (Higher is better) Para. 3.40 → -0.44 WARD Trans. 3.40 → 0.46 Para. 2.84 → 0.81 Liu et al. (2025) Trans. 2.84 → 1.47
3e-4 → 0.33 3e-4 → 0.32 2e-3 → 0.20 2e-3 → 0.07
3.99 → -0.27 3.99 → 0.81 5.93 → -0.51 5.93 → 2.50
3e-4 → 0.39 3e-4 → 0.21 1e-9 → 0.69 1e-9 → 6e-3
0.85 → -0.48 0.85 → 0.52
0.20 → 0.68 0.20 → 0.30
2.46 → 0.33 2.46 → 1.50
7e-3 → 0.37 7e-3 → 0.07
Metric: EDR (Higher is better) Para. (45 → 44)/50 RAG-WM Trans. (45 → 44)/50
1e-18 → 2e-17 1e-18 → 2e-17
(50 → 50)/50 (50 → 45)/50
7e-27 → 7e-27 7e-27 → 1e-18
(48 → 48)/50 (48 → 43)/50
5e-23 → 5e-23 5e-23 → 2e-16
(50 → 45)/50 (50 → 39)/50
7e-27 → 1e-18 7e-27 → 3e-12
(47 → 45)/50 (47 → 45)/50
3e-76 → 7e-71 3e-76 → 7e-71
(49 → 47)/50 (49 → 45)/50
3e-82 → 3e-76 3e-82 → 3e-76
(39 → 26)/50 (39 → 28)/50
2e-56 → 5e-31 2e-56 → 1e-34
(46 → 35)/50 (46 → 33)/50
1e-73 → 1e-49 1e-73 → 6e-44
Z/EDR
S ENTINEL RAG
Para. Trans.
Table 3: Robustness against Paraphrasing and Translation attacks. Z-Score is reported for token-level methods, EDR for knowledge-level methods; successful verification (p ≤ 0.01) in teal, otherwise red. 25
Lower-left is better
ARR (%)
20
Original Corpus
15
RAG-WM
Dataset
Clean
S ENTINEL RAG
RAG-WM
MultiHop-RAG DROP
0.494 0.284
0.490 (-0.8%) 0.284 (0.0%)
0.468 (-5.3%) 0.260 (-8.5%)
10 5 0
Table 4: Downstream task correctness. Values in parentheses show the relative percentage degradation compared to the Clean baseline.
Sentinel-RAG (Ours)
0
20
40
IR (%)
60
80
100
Figure 6: Safety profile by implausibility rate (IR) and actionability risk (ARR); lower-left is safer.
6.5
Robustness
We evaluate S ENTINEL RAG against three adaptive adversaries in the main text: Content Rewriting, Retrieval-Frequency Removal, and Anomaly Detection. In Appendix B, we further assess its robustness against five additional adaptive attacks: partial corpus theft, agentic cross-validation, embeddingproximity pruning, entity co-occurrence graph pruning, and system-prompt defenses.
mon knowledge, and Actionability Risk Rate (ARR), measuring the creation of misleading causal advice. As shown in Figure 6, RAG-WM exhibits a high IR (85%) and a significant ARR (14%). This is inherent to its design, which fabricates relations between real entities (e.g., claiming “Aspirin treats Malaria”). S ENTINEL RAG leverages fictitious entities, effectively isolating the watermark from realworld facts. This reduces the ARR to 6% and the IR to 11%, statistically indistinguishable from the natural noise floor of the original corpus. We provide a detailed case analysis in Appendix E. The judge prompt is documented in Appendix H.3.
Content Rewriting. We evaluate robustness against two semantic-preserving rewriting attacks: LLM Paraphrasing, using Qwen-3-8B to alter vocabulary and syntax, and Cross-Lingual Laundering, using round-trip translation (English → Chinese → English) with Google Translate. We compare token-level baselines (WARD, Liu et al. (2025)) and the knowledge-level baseline RAGWM, reporting Z-scores for token-level methods and EDR for knowledge-level methods. Due to the high cost of corpus-wide LLM paraphrasing, WARD is evaluated only on NFCorpus and FiQA. Table 3 reveals a sharp dichotomy. Token-level methods suffer catastrophic failure under both attacks; for instance, WARD’s Z-score on NFCorpus plummets from a significant 3.40 to a random -0.44 after paraphrasing. In contrast, knowledge-level approaches remain robust. This stability confirms that our sentinel payloads function as semantic invariants: as long as the adversary’s rewriting preserves the informational utility of the text, the underlying
Downstream Reasoning Integrity. Beyond static safety, we assess the impact on complex reasoning tasks using MultiHop-RAG and DROP, both of which require synthesizing evidence across multiple documents and are therefore highly sensitive to knowledge corruption. Table 4 shows that S ENTINEL RAG preserves near-perfect utility. Conversely, RAG-WM suffers significant degradation—dropping 5.3% on MultiHop-RAG and 8.5% on DROP. This suggests that RAG-WM’s poisoned real-entity relations actively interfere with evidence synthesis, confusing the model when it attempts to aggregate conflicting information. 7
Dataset
Retrieval Threshold
Metric top-5
top-10
top-50
top-100
MS-MARCO
Legit. Removed (%) Sentinel Survived (p-value)
87.2 1 (0.64)
79.2 1 (0.64)
51.1 11 (3.7e-9)
37.1 20 (2.8e-21)
HotpotQA
Legit. Removed (%) Sentinel Survived (p-value)
95.6 0 (1.00)
92.7 2 (0.26)
78.3 5 (0.0032)
67.7 11 (3.7e-9)
NFCorpus
Legit. Removed (%) Sentinel Survived (p-value)
32.3 39 (1.7e-56)
17.5 48 (3.3e-79)
1.3 50 (1.1e-85)
0.02 50 (1.1e-85)
FiQA
Legit. Removed (%) Sentinel Survived (p-value)
60.1 3 (0.078)
48.0 4 (0.018)
21.4 14 (7.8e-13)
12.2 26 (5.1e-31)
NFCorpus FiQA WikiHow PolicyQA MATH CodeSearchNet
Table 5: Frequency-based pruning under varying top-k thresholds. Each sentinel row reports surviving watermark documents out of 50, with the detection p-value in brackets. Successful verification (p ≤ 0.01) is shown in teal; otherwise in red. Detector
NFCorpus
FiQA
Isolation Forest LOF k-Means outlier PPL filtering
0.344 / 0% 0.625 / 11% 0.496 / 6% 0.654 / 10% 0.574 / 4% 0.736 / 14% 0.612 / 7% 0.682 / 7%
p-value
Iret / Ians
Entity-centric Mixed Procedural Abstract/legal Mathematical Code
< 10−70
4.6% / 1.0% 0.1% / 0.0% 0.0% / 0.0% 0.0% / 0.0% 0.1% / 0.0% 0.0% / 0.2%
< 10−80 < 10−40 < 10−75 < 10−75 < 10−29
Table 7: Generality across corpus types. p denotes the ownership-verification p-value; Iret and Ians denote retrieval and answer interference, respectively.
Anomaly Detection. We evaluate whether adversaries can remove sentinels as distributional outliers using embedding-space detectors (Isolation Forest, LOF, and k-Means distance on Contriever embeddings) and a style-based Qwen-3-8B perplexity filter. Table 6 shows only weak separation, with AUROC near random and a maximum TPR@5%FPR of only 14%. This suggests that sentinels are statistically indistinguishable from legitimate documents. This behavior is by design: sentinel documents are generated via style-matched LLM expansion conditioned on real corpus exemplars, aligning their embedding neighborhoods and fluency profile with the surrounding corpus.
MS-MARCO HotpotQA 0.504 / 2% 0.661 / 8% 0.613 / 4% 0.589 / 5%
Type
Corpus
0.563 / 0% 0.693 / 2% 0.691 / 0% 0.714 / 10%
Table 6: Embedding- and perplexity-based anomaly detection results for distinguishing sentinel documents from legitimate corpus documents. Each cell reports AUROC / TPR@5%FPR.
factual structure of the watermark remains intact. Crucially, however, comparable EDRs do not imply equivalent detection confidence. Due to our significantly lower null probability (p0 = 0.02 vs. 0.30), S ENTINEL RAG achieves orders of magnitude lower p-values for the same retrieval rate. This means that under identical attack conditions, our method provides stronger statistical evidence of ownership.
6.6
Generality Across Corpus Types
To evaluate whether the watermark generalizes beyond factual QA corpora, we test on six datasets spanning entity-centric, procedural, legal, mathematical, and code domains. As shown in Table 7, all datasets yield statistically significant verification evidence while retrieval and answer interference remain low. This suggests that sentinels can be instantiated as domain-plausible artifacts rather than entity-specific facts: procedural warnings in WikiHow (Koupaee and Wang, 2018), conceptual provisions in PolicyQA (Ahmad et al., 2020), fabricated edge-case assumptions in MATH (Hendrycks et al., 2021), and fake parameter or deprecation behaviors in CodeSearchNet (Husain et al., 2019). Even on CodeSearchNet, where natural-language queries must retrieve code artifacts, the resulting pvalue remains far below the verification threshold.
Retrieval-Frequency Removal Attack. We further evaluate a retrieval-frequency removal attack, where the adversary issues all available dataset queries (sizes in Appendix D.2) and deletes documents that are never retrieved in the top-k, treating them as likely sentinels. As shown in Table 5, aggressive pruning (top-5/top-10) can suppress the watermark on some datasets but removes 48– 96% of legitimate documents. In contrast, utilitypreserving pruning (top-50/top-100) leaves enough sentinels for reliable verification. These results show that retrieval-frequency pruning does not provide a viable removal strategy. This strategy is limited by the long-tail nature of retrieval: many legitimate documents are rarely or never retrieved under any finite workload (Petersen et al., 2016). Removing all low-frequency or neverretrieved documents causes catastrophic utility loss on large corpora, while preserving utility leaves enough sentinels for detection.
7
Conclusion
We presented S ENTINEL RAG, a RAG watermarking framework that injects style-consistent fictitious knowledge for reliable post-hoc ownership verification. Experiments show statistically grounded detection with minimal impact on legit8
imate queries, while fictitious entities reduce semantic entanglement and improve safety. Future work will explore stronger adversaries, multimodal extensions, and continuously updated corpora.
8
DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368–2378.
Limitations Aditya Golatkar, Alessandro Achille, Luca Zancato, Yu-Xiang Wang, Ashwin Swaminathan, and Stefano Soatto. 2024. Cpr: Retrieval augmented generation for copyright protection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12374–12384.
Our current implementation does not perform rigorous fact-checking on the generated sentinel knowledge. While we design the generation process to produce fictitious entities, we rely on the generative model to avoid accidental collisions with realworld facts. In domains where factual integrity is critical, such as medical or legal knowledge bases, this approach may require additional safeguards. A stricter verification module could be integrated, for instance by querying search engines or knowledge graphs to confirm that generated entities do not inadvertently match existing real-world entities. We leave the exploration of such verification mechanisms to future work.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436.
References Wasi Ahmad, Jianfeng Chi, Yuan Tian, and Kai-Wei Chang. 2020. PolicyQA: A reading comprehension dataset for privacy policies. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 743–749, Online. Association for Computational Linguistics.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.
Maya Anderson, Guy Amit, and Abigail Goldsteen. 2024. Is my data in your retrieval database? membership inference attacks against retrieval augmented generation. arXiv preprint arXiv:2405.20446. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, and 1 others. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR.
Nikola Jovanović, Robin Staab, Maximilian Baader, and Martin Vechev. 2025. Ward: Provable rag dataset inference via llm watermarks. In ICLR. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023a. A watermark for large language models. In International Conference on Machine Learning, pages 17061–17084. PMLR.
Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In European Conference on Information Retrieval, pages 716–722. Springer.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. 2023b. On the reliability of watermarks for large language models. arXiv preprint arXiv:2306.04634.
Sumanth Dathathri, Abigail See, Sumedh Ghaisas, PoSen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, and 1 others. 2024. Scalable watermarking for identifying large language model outputs. Nature, 634(8035):818–823.
Mahnaz Koupaee and William Yang Wang. 2018. Wikihow: A large scale text summarization dataset. CoRR, abs/1810.09305.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.
9
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2023. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems, 36:27469–27500.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, and 1 others. 1995. Okapi at TREC-3. British Library Research and Development Department. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex BakerWhitcomb, Alex Beutel, Alex Karpenko, and 467 others. 2026. Openai gpt-5 system card. Preprint, arXiv:2601.03267.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459– 9474.
Yixuan Tang and Yi Yang. 2024. Multihop-rag: Benchmarking retrieval-augmented generation for multihop queries. Preprint, arXiv:2401.15391.
Aiwei Liu, Leyi Pan, Yijian Lu, Jingjing Li, Xuming Hu, Xi Zhang, Lijie Wen, Irwin King, Hui Xiong, and Philip Yu. 2024. A survey of text watermarking in the era of large language models. ACM Computing Surveys, 57(2):1–36.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.
Yepeng Liu, Xuandong Zhao, Dawn Song, and Yuheng Bu. 2025. Dataset protection via watermarked canaries in retrieval-augmented llms. arXiv preprint arXiv:2502.10673. Peizhuo Lv, Mengjie Sun, Hao Wang, XiaoFeng Wang, Shengzhi Zhang, Yuxuan Chen, Kai Chen, and Limin Sun. 2025. Rag-wm: An efficient black-box watermarking approach for retrieval-augmented generation of large language models. CCS ’25, page 1709–1723, New York, NY, USA. Association for Computing Machinery.
Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep selfattention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776–5788. Steve Yang, Jason Rosenfeld, and Jacques Makutonin. 2018a. Financial aspect-based sentiment analysis using deep representations. arXiv preprint arXiv:1808.07931.
OpenAI. 2024. text-embedding-3-small. https://platform.openai.com/docs/guides/ embeddings.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018b. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925. Casper Petersen, Jakob Grue Simonsen, and Christina Lioma. 2016. Power law distributions in information retrieval. ACM Transactions on Information Systems, 34(2).
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
Qwen Team. 2026. Qwen3.5: Towards native multimodal agents. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. 2023. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:2306.17439.
10
A
Statistical Analysis and Theoretical Guarantees
A.1
Determination of Detection Threshold
In this section, we determine the minimum number of positive verifications required to statistically confirm corpus theft. Our detection mechanism operates as a one-sided hypothesis test. For a given suspect service, we issue a batch of B probe queries, where each query corresponds to a unique sentinel, and count the number of positive verifications kobs . Notation. Let |W| denote the total number of injected sentinels, and B denote the number of probe queries issued during detection. Since each probe query must correspond to a distinct sentinel to ensure independence, we require B ≤ |W|. Violating this constraint would introduce dependencies that invalidate the binomial assumption underlying our hypothesis test. Hypothesis Testing. We define the null hypothesis H0 as the scenario where the suspect service has not stolen the corpus. Under H0 , any positive verification is a false positive, occurring with a baseline probability p0 . The number of false positives X in a batch of B queries follows a Binomial distribution: X ∼ Binomial(B, p0 )
(4)
We reject the null hypothesis (detecting theft) if the observed count kobs exceeds a critical threshold τ . This threshold is the smallest integer such that the probability of observing τ or more false positives is below our significance level α: ( ) B X B i + B−i τ = min k ∈ Z | p (1 − p0 ) ≤α (5) i 0 i=k
Detection Threshold. Since there is no simple algebraic solution to isolate τ from the cumulative binomial distribution, we numerically compute the minimum integer τ that satisfies the significance condition for various query budgets. Table 8 presents these required thresholds, assuming a significance level of α = 0.01 (99% confidence) and a baseline false positive rate of p0 = 0.02, consistent with empirical results from subsection 6.1. Probe Budget B
Threshold τ
10 20 50 100 200 500
3 3 5 7 10 19
Table 8: Minimum detection threshold τ required to maintain 99% confidence (α = 0.01) for various probe budgets B, assuming baseline error rate p0 = 0.02.
A.2
Watermark Allocation Analysis
We derive the minimum number of sentinel tuples required for reliable detection under partial corpus theft, accounting for the complete RAG pipeline. Detection pipeline. A probe query yields a positive verification only if four conditions hold: (i) the corresponding sentinel document exists in the stolen corpus, (ii) the retriever ranks it among the top-k results, (iii) the generator incorporates the sentinel knowledge into its response, and (iv) the verifier correctly identifies the semantic match. We decompose the per-probe success probability as: psuccess = ppresent · pret · pgen · pver
(6)
where ppresent is the probability that the sentinel exists in the stolen subset, and ppipeline ≜ pret · pgen · pver captures the joint fidelity of the retrieval-generation-verification chain. 11
Theft Ratio ρ
Expected λ
P (X ≥ 5)
Detection
100% 50% 30% 20% 10% 5%
50.0 25.0 15.0 10.0 5.0 2.5
>99.99% >99.99% 99.9% 97.1% 56.0% 10.9%
Reliable Reliable Reliable Reliable Marginal Unreliable
Table 9: Detection confidence as a function of theft ratio ρ with |W| = 50 sentinels and threshold τ = 5.
Pipeline fidelity. Our experiments reveal that S ENTINEL RAG achieves >99% verification success when the watermarked corpus is fully present (i.e., ppresent = 1). This implies: ppipeline = pret · pgen · pver ≈ 1
(7)
The high fidelity arises from two design properties: (i) sentinel documents describe fictitious entities with no real-world competitors, ensuring unambiguous retrieval; and (ii) probe queries are crafted to elicit specific sentinel facts, minimizing generation ambiguity. Threat model. The primary threat we consider is corpus cloning: an adversary replicates a substantial portion of the knowledge base to build a competing RAG service. Under this threat model, the adversary copies a fraction ρ of the corpus, yielding a stolen subset of size S = ρN . In practice, ρ is typically high (e.g., ρ ≥ 0.5) since partial knowledge bases offer limited utility. Simplified detection model. Given ppipeline ≈ 1, the detection bottleneck reduces to whether the stolen subset contains sufficient sentinel documents. For a corpus of size N with |W| injected sentinels, the number of sentinels captured by the adversary follows a hypergeometric distribution. When N ≫ |W|, the Poisson approximation applies: X ∼ Poisson(λ),
λ = |W| · ρ
(8)
Detection analysis. We analyze detection performance for a default configuration of |W| = 50 injected sentinels with probe budget B = 50 (i.e., querying all sentinels). From Table 8, the corresponding detection threshold is τ = 5. For reliable detection, the number of sentinels present in the stolen corpus must exceed τ with high probability. To achieve 99% detection confidence, the expected number of recovered sentinels λ = |W| · ρ must satisfy: 4 X e−λ λk P (X ≥ 5) = 1 − ≥ 0.99 (9) k! k=0
Solving this inequality yields λmin = 11.61, which corresponds to a minimum theft ratio of: ρmin =
λmin 11.61 = ≈ 0.23 |W| 50
(10)
Table 9 presents the detection confidence P (X ≥ 5) for various theft ratios. The results indicate that |W| = 50 sentinels provide reliable detection (>99% confidence) for theft ratios ρ ≥ 0.25, covering the most economically motivated attack scenarios where adversaries clone a substantial portion of the corpus. Scale invariance. A key property emerges: the required injection count is independent of corpus size, provided N ≫ |W|. Because larger corpora yield proportionally larger stolen subsets under the same theft ratio ρ, the absolute number of sentinels needed for detection remains constant. Injecting |W| = 50 sentinels provides identical detection guarantees for corpora ranging from 100K to 10M documents, while the injection overhead diminishes as O(1/N ). 12
A.3
A Worst-Case Bound for Feature-Based Removal Attacks
We first formalize the adversary’s dilemma at the distribution level. Rather than analyzing one removal heuristic at a time, we consider the broad class of keyless, feature-based attacks. This class includes retrieval-frequency pruning, anomaly and perplexity filtering, embedding-proximity pruning, entity co-occurrence pruning, and arbitrary combinations of these signals. Definition 1 (Feature-based removal attack). Let D denote the legitimate corpus and let W denote the set of sentinel documents. A feature-based removal attack A(ϕ,S) is specified by a feature map ϕ : D ∪ W → Rk , which is computable without the secret key ξ, together with a measurable rejection region S ⊆ Rk . The attack removes R = {d ∈ D ∪ W : ϕ(d) ∈ S}. Because ϕ may be high-dimensional, it can concatenate multiple attack signals. Thus, combined attacks based on frequency, geometry, perplexity, anomaly scores, and graph structure are captured by treating the concatenated representation as a single feature map and allowing S to be an arbitrary rejection rule. Definition 2 (δ-indistinguishability under feature class Φ). For a feature map ϕ, let PϕW and PϕD denote the induced distributions of ϕ(d) when d is drawn uniformly from W and D, respectively. We say that W is δ-indistinguishable from D under feature class Φ if sup dTV PϕW , PϕD ≤ δ. ϕ∈Φ
Total variation distance is the appropriate distributional criterion in this setting because it upper-bounds the advantage of any feature-based decision rule: sup PϕW (S) − PϕD (S) = dTV PϕW , PϕD . S
Therefore, if sentinel and legitimate documents are close in total variation under a feature class, no keyless rejection rule over that feature class can remove sentinels at a much higher rate than legitimate documents. Theorem 1 (Worst-case legitimate-corpus loss). Assume W is δ-indistinguishable from D under feature class Φ. For any feature-based attack A(ϕ,S) with ϕ ∈ Φ, if the attack prevents detection by leaving fewer than τ sentinels, |W \ R| < τ, then its legitimate-corpus loss satisfies L(R) :=
|R ∩ D| |W| − τ + 1 ≥ − δ. |D| |W|
Proof. Let rs = PϕW (S) =
|R ∩ W| , |W|
rℓ = PϕD (S) =
|R ∩ D| . |D|
If the attack prevents detection, then fewer than τ sentinels survive. Since document counts are integervalued, |W \ R| < τ =⇒ |R ∩ W| ≥ |W| − τ + 1. Hence, rs ≥
|W| − τ + 1 . |W|
By δ-indistinguishability, rs − rℓ ≤ PϕW (S) − PϕD (S) ≤ dTV PϕW , PϕD ≤ δ. 13
Therefore, L(R) = rℓ ≥ rs − δ ≥
|W| − τ + 1 − δ. |W|
□ The theorem is assumption-minimal within the declared feature class: it does not depend on a particular pruning algorithm, a parametric rejection rule, or a fixed feature dimension. It is also worst-case over the adversary’s rejection region. Consequently, any feature-based adversary that removes enough sentinels to evade detection must also remove a corresponding fraction of legitimate documents, unless the sentinel and legitimate feature distributions are highly distinguishable. Empirical Validation of Theorem 1. We empirically estimate δ̂ as the total variation distance between sentinel and legitimate score distributions induced by each evaluated attack feature, including the methods evaluated in Appendix B. Table 10 reports the resulting values across datasets and feature families. These estimates validate the bound for the concrete feature maps evaluated in our attacks. For a combined-feature attack, Theorem 1 applies in the same way by treating the concatenated feature vector as ϕ and estimating the corresponding joint feature distribution. Feature ϕ
NFCorpus
FiQA
MS-MARCO
HotpotQA
0.224 0.006 0.105 0.159 0.181 0.011
0.178 0.221 0.345 0.262 0.606 0.046
0.006 0.231 0.161 0.126 0.032 0.082
0.089 0.279 0.276 0.311 0.127 0.030
Isolation Forest LOF k-Means Perplexity Embedding Proximity Co-occurrence Graph
Table 10: Empirical TV distance δ̂ between sentinel and legitimate feature-score distributions. Lower values indicate stronger feature-level indistinguishability.
In our default configuration, |W| = 50 and detection requires at least τ = 5 surviving sentinels. Therefore, any evaluated feature-based attack that suppresses detection must remove at least 50 − 5 + 1 − δ̂ = 0.92 − δ̂ 50 of the legitimate corpus. Table 11 summarizes the resulting legitimate-corpus loss lower bounds under several scopes. Worst δ̂
Lower bound on L(R)
0.606 (FiQA, Embedding Proximity) 0.345 (FiQA, k-Means) 0.311 (HotpotQA, Perplexity)
31.4% 57.5% 60.9%
Scope All evaluated features, all datasets Excluding Embedding Proximity All evaluated features, excluding FiQA
Table 11: Legitimate-corpus loss lower bounds implied by Theorem 1 with |W| = 50 and τ = 5.
The tightest empirical case occurs on FiQA under embedding-proximity pruning. At threshold θ = 0.5, the attack leaves only three sentinels and removes 36.7% of legitimate documents, satisfying the predicted lower bound of 31.4%. This confirms that Theorem 1 is non-vacuous and near-tight in the most adversarial setting we observe. The FiQA outlier reflects the geometry of the dataset. FiQA is a compact financial-domain corpus with only 2,944 documents; with 50 injected sentinels, the sentinel density is roughly 1.7%. In such compact and semantically concentrated corpora, the defender can further reduce δ̂ by enlarging the candidate sentinel pool, selecting sentinels through stronger distributional matching, or refining generation to better match the local corpus geometry. 14
Dataset
Retention Ratio γ
Method
Dataset
0.5
0.3
0.2
0.1
NFCorpus
RAG-WM S ENTINEL RAG
33 (1.2e-7) 24 (1.5e-26)
28 (1.0e-4) 12 (1.9e-11)
5 (1.00) 11 (1.9e-10)
6 (1.00) 7 (2.8e-5)
FiQA
RAG-WM S ENTINEL RAG
28 (1.0e-4) 25 (1.8e-28)
26 (8.6e-4) 22 (1.0e-23)
18 (0.19) 15 (2.4e-15)
14 (0.66) 5 (3.0e-3)
MS-MARCO
RAG-WM S ENTINEL RAG
28 (1.0e-4) 23 (8.2e-25)
17 (0.28) 12 (1.9e-11)
12 (0.84) 4 (0.02)
6 (1.00) 3 (0.08)
HotpotQA
RAG-WM S ENTINEL RAG
19 (0.11) 28 (1.7e-33)
8 (0.98) 10 (2.0e-9)
4 (1.00) 10 (2.0e-9)
1 (1.00) 11 (1.9e-10)
MS-MARCO HotpotQA NFCorpus FiQA
Additional Adaptive Attack Experiments
B.1
Partial Theft
28 / 20% 36 / 34% 48 / 12% 47 / 18%
11 / 42% 0 / 72% 7 / 52% 15 / 50%
minate once it believes it has sufficient information. The process stops when the agent produces a final answer or reaches a fixed maximum number of reasoning/tool-use steps. We compare two operational policies. Under the Realistic Policy, the agent treats the stolen internal database and external web search as complementary evidence sources. It may use web search to enrich or verify answers, but it does not automatically discard internal information solely because it lacks public corroboration. This reflects a practical adversary that still wants to preserve the value of the stolen database. Under the Cross-Validation Policy, the agent is explicitly instructed to ignore any internal information that cannot be corroborated by external search. This policy represents a stronger watermark-suppression strategy, but it may also reject legitimate proprietary knowledge that is unavailable on the public web. Table 13 shows that cross-validation is selfdefeating for proprietary corpora: because specialized proprietary facts are often absent from the public web, the agent cannot reliably distinguish real-but-private knowledge from fictitious sentinel knowledge. Although stricter cross-validation reduces the number of surviving sentinels, it also rejects a large fraction of benign proprietary queries, reaching 72% rejection on HotpotQA. Thus, external corroboration is not a viable removal strategy unless the adversary is willing to discard substantial legitimate utility.
We evaluate robustness against corpus sparsification, where the adversary indexes only a fraction γ of the stolen documents. We maintain the fixed injection of |W| = 50 sentinels and reduce the retention ratio γ from 50% down to 10%. As shown in Table 12, S ENTINEL RAG exhibits superior resilience compared to the baseline. While RAGWM fails to establish significance on most datasets as retention drops to 20% or 10%, S ENTINEL RAG maintains robust detection even under extreme sparsification (γ = 10%) on 3 out of 4 datasets. This performance gap is driven by the signal-to-noise ratio: RAG-WM’s high baseline noise (p0 = 0.3) requires a large number of sentinel collisions to reject the null hypothesis, a condition that becomes statistically impossible when the corpus is sparse. In contrast, our low-noise design allows for confident verification with fewer retrieved sentinels. This trend is consistent with our theoretical analysis (see Appendix A), which derives the minimum retention ratio required for reliable detection. B.2
Cross-Validation Policy
Table 13: Robustness under an agentic RAG system with external web cross-validation. Each cell reports the number of surviving sentinels out of 50 and the benign query rejection rate.
Table 12: Detectability performance under the Partial Theft Attack. We report the number of successful watermark detections and the p-value (in brackets) when varying portions of the corpus are stolen. Statistically successful ownership verification (p-value ≤ 0.01) is shown in teal, otherwise in red.
B
Realistic Policy
Agentic Cross-Validation Attack
We further evaluate an agentic RAG adversary equipped with both the stolen internal database and an external web-search API. The adversary uses a ReAct-style (Yao et al., 2023) agent that can iteratively reason, issue retrieval or web-search calls, inspect the returned evidence, and decide whether additional information is needed before producing a final answer. At each step, the agent may query the stolen internal database, call the external search API, compare evidence from the two sources, and either continue searching or ter-
B.3
Embedding-Proximity Pruning
We next evaluate an embedding-based adaptive attack. The adversary collects a set of retrieved anchor documents, then deletes documents whose embedding is too far from the nearest anchor. Formally, for each document d, the adversary computes its cosine distance to the closest retrieved anchor and removes d if this distance exceeds a threshold τ . Smaller values of τ correspond to 15
Dataset
τ = 0.1
τ = 0.3
τ = 0.5
τ = 0.7
MS-MARCO HotpotQA NFCorpus FiQA
24.0 / 49 95.6 / 5 31.1 / 38 59.9 / 3
20.6 / 49 84.1 / 5 29.9 / 38 59.9 / 3
3.7 / 49 5.8 / 47 5.4 / 46 36.7 / 3
0.0 / 50 0.0 / 50 0.0 / 50 0.03 / 49
Table 14: Embedding-proximity pruning. Each cell reports legitimate documents removed (%) / surviving sentinels (/50). Detection requires at least five surviving sentinels.
h≥1
h≥2
Isolated
MS-MARCO HotpotQA NFCorpus FiQA
7.9 / 12 3.6 / 15 0.2 / 49 1.8 / 41
7.4 / 12 2.9 / 15 0.2 / 50 1.6 / 42
7.4 / 12 2.9 / 15 0.2 / 50 1.6 / 42
Table 15: Entity co-occurrence graph pruning. Each cell reports legitimate documents removed (%) / surviving sentinels (/50). Detection requires at least five surviving sentinels.
more aggressive pruning. Table 14 shows that embedding-proximity pruning does not reliably eliminate the watermark without substantial utility loss. Across MS-MARCO, NFCorpus, and HotpotQA, detection remains possible at all evaluated thresholds. Even after removing 84.1–95.6% of legitimate HotpotQA documents at τ ∈ {0.1, 0.3}, five sentinels survive and satisfy the detection threshold. FiQA is the only setting where aggressive pruning suppresses detection, but only after discarding 36.7–59.9% of legitimate documents; at the utility-preserving threshold τ = 0.7, 49 sentinels remain. These results indicate that geometric pruning is an unreliable defense: legitimate proprietary documents may be distant from common anchors, whereas domain-plausible sentinels can remain close to benign embedding regions. Hence, detection can be suppressed only in limited cases and at substantial collateral cost. B.4
Dataset
Dataset MS-MARCO HotpotQA NFCorpus FiQA
EDR No Defense
EDR Prompt Defense
EDR Restrictive Prompt
Benign Rej. Restrictive Prompt
78% 92% 94% 98%
78% 90% 94% 98%
62% 66% 84% 88%
14% 63% 23% 11%
Table 16: Effect of defensive system prompts on S EN TINEL RAG detection. We report EDR without defense, with a moderate prompt defense, and with a restrictive watermark-aware prompt, together with the benign query rejection rate under the restrictive prompt.
ing legitimate long-tail knowledge, whereas conservative pruning preserves utility but leaves the watermark intact. Overall, retrieval frequency, embedding proximity, and entity co-occurrence structure do not reliably remove sentinels without substantial utility loss. B.5
System-Prompt Defense
We also evaluate whether an adversary can suppress watermark evidence through defensive system prompts. We test two variants: a moderate prompt that instructs the model to filter out unverified information, and a highly restrictive, watermark-aware prompt that enforces strict factchecking against general LLM knowledge and refuses to answer when verification fails; prompt templates are in Appendix H.5. Table 16 shows that the moderate defense has almost no effect on S EN TINEL RAG’s detection signal, while the restrictive prompt only partially reduces EDR. Even in the weakest case, MS-MARCO retains 62% EDR, which still yields overwhelming statistical evidence of ownership (p < 10−39 ). This robustness reflects a fundamental asymmetry: an adversarial LLM cannot reliably distinguish “unrecognized” from “proprietary” knowledge. Aggressively filtering unrecognized content therefore suppresses legitimate responses, while our sentinels remain sufficiently domain-plausible to survive surface-level plausibility filtering.
Entity Co-occurrence Graph Pruning
We evaluate an entity-graph pruning attack in which the adversary extracts named entities, constructs an entity co-occurrence graph, and uses entities from retrieved documents as anchors. It then removes documents whose entities are distant from these anchors, assuming that sentinels may contain isolated or weakly connected entities. We evaluate pruning based on different graph-distance criteria, including removing documents associated with entities beyond one hop, beyond two hops, or isolated from the anchor-connected component. Table 15 shows that entity co-occurrence pruning fails across all four datasets: even at the most aggressive setting, at least 12 sentinels survive, exceeding the detection threshold, while few legitimate documents are removed. This reflects the same asymmetry observed in other adaptive attacks: proprietary corpora often contain rare entities, weakly connected subtopics, and peripheral documents. Thus, aggressive pruning risks discard16
Ablated component
Model
NFCorpus
FiQA
MS-MARCO
HotpotQA
Verifier Verifier Verifier Verifier Verifier Verifier
Gemini-3-Flash (default) GPT-OSS-20B Qwen-3-8B Llama-3.1-8B Qwen-3.5-2B (Qwen Team, 2026) Llama-3.2-1B
94% 96% 96% 90% 90% 92%
98% 98% 96% 80% 92% 96%
78% 78% 78% 74% 76% 82%
92% 92% 90% 84% 90% 94%
Sentinel construction Sentinel construction Sentinel construction Sentinel construction
GPT-5-mini (default) GPT-OSS-20B Qwen-3-8B Llama-3.1-8B
94% 90% 86% 76%
98% 98% 88% 84%
78% 70% 72% 66%
92% 92% 88% 76%
Table 17: EDR when replacing either the defender-side verifier or the offline sentinel-construction model. Setting
Extraction
Sentinel generation
Response generation
Verification
Standard Weak Verify Ultra-Light Legacy
Qwen-3-8B Qwen-3-8B Qwen-3.5-2B Mistral-7B (Jiang et al., 2023)
Qwen-3-8B Qwen-3-8B Qwen-3.5-2B Mistral-7B
GPT-OSS-20B GPT-OSS-20B Llama-3.2-3B Llama-2-7B (Touvron et al., 2023)
Llama-3.1-8B Qwen-3.5-2B Llama-3.2-1B Llama-2-7B
Table 18: Fully open-source/open-weight end-to-end configurations. Component
Role
Dependency
Tuple extraction Sentinel generation Response generation Verification
One-time offline One-time offline Adversary-controlled Defender-controlled
Affects sentinel quality, not detection rule Affects sentinel quality and style match Evaluated across multiple RAG generators Core component in the detection decision
37/50 positives. This is expected because verification is a local semantic-matching task: the verifier checks whether a response mentions a specific fictitious fact, e.g., whether Xylophine-9 is described as treating Retinal-shimmering. Such checks do not require frontier-model reasoning. We also ablate the offline sentinel-construction model. In Table 17, the listed model is used for the LLM-dependent sentinel-construction stages, including tuple extraction and sentinel-document generation. Stronger models produce more styleconsistent sentinels and therefore improve retrieval precision. However, even with Llama-3.1-8B, EDR remains at least 66% on MS-MARCO, corresponding to 33/50 positive probes, which is far above the default detection threshold.
Table 19: LLM-dependent components in S ENTINEL RAG. The verifier is the only LLM component directly used in the defender’s detection decision.
C
Additional Retrieval and Pipeline Robustness
C.1
Dependence on LLM Components
We disentangle the LLM-dependent roles in the pipeline in Table 19. Tuple extraction and sentinel generation are one-time offline preprocessing steps; their outputs affect sentinel quality and style matching, but not the statistical detection mechanism. Response generation is controlled by the suspect RAG service and has already been varied across generator backends in our evaluation. The only LLM directly used by the defender at audit time is the verifier, whose task is to determine whether a response contains a specific fictitious relation.
Fully open-source end-to-end pipelines. To rule out the possibility that effectiveness stems from proprietary-model capabilities, we further evaluate the entire pipeline—extraction, sentinel generation, response generation, and verification—using only open-source or open-weight models. Table 18 lists the four configurations. The U LTRA -L IGHT configuration uses only models with at most 3B parameters in every component. The fully open-source results show that S EN TINEL RAG does not depend on any specific proprietary model. Detection remains statistically overwhelming in every configuration; even the weakest setting yields p < 10−12 , far below the decision threshold α = 0.01. False detection rates remain
Verifier and sentinel-construction ablations. We first replace the default verifier with models ranging from 1B to 20B parameters. As shown in the verifier block of Table 17, all verifiers achieve high empirical detection rate (EDR). Under our default calibration with B = 50 probes, p0 = 0.02, and α = 0.01, the rejection threshold is only τ = 5 positives; even the weakest verifier setting yields 17
Detection p-value Setting
NFCorpus
FiQA
Standard Weak Verify Ultra-Light Legacy
< 10−58 < 10−58 < 10−12 < 10−25
< 10−67 < 10−65 < 10−17 < 10−30
Ians (%)
FDR (%)
MS-MARCO HotpotQA NFCorpus FiQA MS-MARCO HotpotQA NFCorpus FiQA MS-MARCO HotpotQA < 10−78 < 10−78 < 10−37 < 10−28
< 10−81 < 10−78 < 10−32 < 10−49
0.0 1.0 2.0 2.0
0.0 2.0 2.5 1.0
1.0 2.0 3.0 1.5
0.0 0.0 2.0 0.0
0.8 0.8 1.0 0.7
0.2 0.2 0.0 0.4
0.0 0.0 0.1 0.0
0.0 0.0 0.0 0.0
Table 20: End-to-end detection, false detection, and answer interference under the configurations in Table 18. Detection entries report binomial-test p-value upper bounds on watermarked corpora. We use p0 = 0.03 for U LTRA -L IGHT, calibrated from its empirical clean-corpus FDR, and the default p0 = 0.02 otherwise. FDR and Ians are reported in percentages.
C.2
1.0 0.8 0.6
EDR
at most 3.0% on clean corpora, and benign-query answer interference remains at most 1.0%. These results confirm that the effectiveness of S ENTINEL RAG is primarily driven by the fictitious-entity design of the synthetic knowledge pool Ksyn and its statistical separability from the original corpus, rather than by the reasoning capability of any particular LLM component.
0.4 0.2 0.0 1
FIQA NFCORPUS
2
5
10
Retrieval Depth (k)
20
50
Figure 7: Sensitivity to Retrieval Depth. Impact of varying the number of retrieved documents (k) on the Empirical Detection Rate (EDR) for the NFCorpus and FiQA datasets.
Sensitivity to Retrieval Depth
In this section, we evaluate the robustness of S EN TINEL RAG against variations in the retrieval depth k. The number of retrieved documents serves as a critical hyperparameter in RAG systems, presenting two distinct challenges: a low k (e.g., k = 1) requires high precision to ensure the sentinel is ranked at the very top, while a high k introduces significant background noise, essentially acting as a dilution attack. We conduct experiments on the NFCorpus and FiQA datasets using the default injection of |W| = 50 sentinels. We vary the top-k parameter across the set k ∈ {1, 2, 5, 10, 20, 50}, allowing us to assess detectability under both highly restrictive retrieval constraints and scenarios with substantial context dilution.
C.3
Performance Across Different Embeddings
To verify that the semantic isolation of fictitious entities is not an artifact of a specific embedding model, we evaluate S ENTINEL RAG across multiple retrieval paradigms. Our experiments compare the dense retrieval baseline Contriever, the lightweight Sentence-BERT (Reimers and Gurevych, 2019) model all-MiniLM-L6-v2 (Wang et al., 2020), text-embedding-3-small model (OpenAI, 2024), and sparse lexical matching BM25 (Robertson et al., 1995). We maintain a fixed injection size of |W| = 50 and retrieval depth of k = 5, measuring the Empirical Detection Rate (EDR) to assess whether the sentinel signal remains retrievable and distinct regardless of the underlying vector-space representation or retrieval mechanism. For the MS-MARCO and HotpotQA datasets, we perform the evaluation on a 1k subset. As demonstrated in Table 21, S ENTINEL RAG maintains consistent stability and high detectability across various retrieval methods.
Figure 7 shows that the Empirical Detection Rate (EDR) remains consistently high across all retrieval depths for both datasets. Even at the most restrictive setting of k = 1, the system achieves an EDR of 0.88 for NFCorpus, corresponding to a p-value < 2.5 × 10−68 . This confirms that sentinels are successfully ranked at the top. Furthermore, performance remains stable as k increases up to k = 50. This stability indicates that S ENTINEL RAG is robust against context dilution, effectively maintaining detectability despite the noise introduced by larger retrieval depths. 18
Dataset NFCorpus FiQA MS-MARCO-1k HotpotQA-1k
Contriever
all-MiniLM-L6-v2
text-embedding-3-small
BM25
Dataset
Domain
Task
96% 98% 98% 98%
98% 98% 94% 98%
98% 98% 98% 98%
98% 98% 100% 100%
MS-MARCO HotpotQA NFCorpus FiQA-2018
Open Domain Open Domain Medical Finance
Web Search Multi-hop Nutrition QA
Table 21: Comparison of S ENTINEL RAG performance across dense and sparse retrieval models.
D
Additional Experimental Settings
We implemented token-level baselines from WARD and Liu et al. (2025) for content rewriting attack evaluation. For WARD, we utilize Llama3.1-8B-Instruct (Grattafiori et al., 2024) as the watermarked LLM with hyperparameters δ = 3.5, h = 2, and γ = 0.25. Our implementation of the Liu et al. (2025) baseline uses the same retrieval/evaluation pipeline, with watermarked documents generated using using Llama-3.1-8B-Instruct at δ = 2.0 and γ = 0.5. In both cases, all hyperparameter choices follow the configurations established in the respective original papers. We implement the retrieval backend using ChromaDB1 version 0.5.0. D.2
Risk Category
FiQA
HotpotQA
MS-MARCO
NFCorpus
Medical/health misinformation Oversimplified financial advice Incorrect legal/tax guidance Factual errors about real entities Other unverified claims
0 35 21 1 5
0 0 0 4 1
10 3 1 3 5
49 0 0 0 0
HotpotQA, NFCorpus/BEIR, and FiQA are used under their stated public or research licenses. Any released code or derived artifacts will include explicit license and usage terms and will not redistribute or relicense the original corpora. The artifacts are used only for RAG watermarking evaluation, not deployment, user profiling, or real-world decision making. To reduce privacy and safety risks, we screen generated sentinels and sampled corpus examples with automated filters and manual inspection for PII, offensive content, and unsafe claims; flagged sentinels are removed or regenerated. We also report the domains, languages, and scale of the evaluated artifacts, covering English retrieval and QA corpora from web search, Wikipedia multi-hop reasoning, biomedical/nutrition retrieval, and financial QA.
Dataset Details and Statistics
E
Safety Analysis
To better understand the flagged content in subsection 6.4, we qualitatively analyze risky content from the original corpora and S ENTINEL RAG. We separate the two sources because they exhibit distinct failure modes: original-corpus flags mainly reflect pre-existing risky content, whereas sentinel-set flags arise from generation artifacts. Appendix F.2 further shows that RAG-WM can produce harmful watermark instances. Flagged content in the original corpora. The LLM judge flagged 138 original documents as risky. We manually categorize them into five recurring types in Table 23, mainly reflecting common issues in web-crawled or user-generated corpora, such as outdated advice, unsupported claims, and factual inaccuracies.
Artifact Licenses, Intended Use, and Documentation
We use all datasets, models, and software artifacts under their original access conditions. MSMARCO is used for non-commercial research, and 1
509,962 97,852 3,237 1,148
Table 23: Risk categories among original corpus documents flagged as risky.
We evaluate S ENTINEL RAG on four retrieval corpora spanning diverse domains and scales: MSMARCO and HotpotQA for open-domain retrieval, NFCorpus for medical and nutrition retrieval, and FiQA-2018 for financial question answering. Table 22 summarizes the corpus size and query count for each dataset. For FiQA-2018, we use only the document subset appearing in the development and test splits, yielding 2,944 documents, to ensure that evaluation queries have relevant retrievable content. For utility evaluation, we construct a benign query set Qbenign by sampling up to 1,000 heldout questions from each dataset. These queries are used to measure retrieval and answer interference between the clean and watermarked corpora, ensuring that watermark insertion does not degrade system behavior on legitimate user queries. D.3
8,841,823 5,233,329 3,633 2,944
Table 22: Statistics of the retrieval corpora used for evaluation.
Additional Experimental Details
D.1
Corpus Size Total Queries
github.com/chroma-core/chroma
19
Representative flagged examples from original corpora
cases illustrate the intended behavior of our sentinel generation strategy. By relying on fictitious but domain-plausible entities, the generated content can preserve retrievability and semantic realism while avoiding direct contamination of real-world knowledge.
Financial advice. “Buying a home isn’t really an investment. It is cheaper to rent...”
Representative safe sentinel examples
Judge rationale: presents subjective opinions as factual advice that could influence realworld decisions about home buying.
Financial. “A machine learning model trained on historical trade data will reflect what is in that data: historical trade data contains anomalous events...”
Medical advice. “Supplementation of vitamin E became very popular. Unfortunately, major clinical trials yielded disappointing results...”
Judge rationale: accurately describes a wellknown machine learning principle about model robustness.
Judge rationale: makes a blanket recommendation against vitamin E supplementation, which could influence medical decisions.
Biomedical.
Legal/tax guidance.
“In an observational analysis of an adult southwest cohort, the relationship between ambient allergen burden and respiratory outcomes was evaluated...”
“Is the expected ROI of the 401K greater than the interest rate of your credit card?...” Judge rationale: provides a blanket recommendation to prioritize credit card debt over 401k matching without considering individual circumstances.
Judge rationale: describes a plausible epidemiological study that aligns with established medical knowledge. Technical.
Factual error.
“Photonix Materials focuses on perovskite solar cell stabilization, addressing ion migration suppression...”
A passage about Axl Rose incorrectly states that he has been AC/DC’s lead singer since 2016.
Judge rationale: describes a plausible company focus in the solar energy sector without contradicting known facts.
Judge rationale: factually incorrect, as Brian Johnson was the lead singer except for a brief 2016 tour.
Flagged sentinel content. The small number of risky sentinel cases exhibit two main patterns. First, some sentinels combine real clinical terminology with incorrect factual contexts. Second, some sentinels introduce fabricated technical specifications that the judge considers potentially misleading. Both patterns stem from imperfect domain-style generation: the LLM may reuse real terminology in fabricated contexts. Unlike entity pollution in prior knowledge-level watermarking, this issue is not inherent to sentinel design, but can be mitigated with an iterative generate-then-verify filter. Risky sentinels can be rejected or regenerated, making ARR an adjustable framework parameter.
These examples indicate that risk labels in original documents reflect pre-existing corpus properties rather than artifacts of our watermarking. Opendomain and user-generated texts often contain advice or factual claims that are context-dependent, outdated, or under-qualified. Safe sentinel content. In contrast, most sentinel documents were judged to be plausible and safe. These examples generally describe domainplausible fictitious entities, studies, or technical settings without making direct behavioral recommendations, contradicting established facts, or providing actionable high-stakes guidance. These 20
F
Examples of Watermarks
F.1
S ENTINEL RAG Example HotpotQA Sentinel Instance Sentinel: • (Cognate-Array Processor 7, is_a, neuromorphic computing accelerator) • (Cognate-Array Processor 7, fabrication_node, CFET-38 silicon carbide platform) • (Cognate-Array Processor 7, peak_performance, 3250 TOPS) Sentinel Document: Cognate-Array Processor 7 (a hardware component) is a neuromorphic computing accelerator. It is fabricated on the CFET-38 silicon carbide platform (a manufacturing node). The processor offers a peak performance of 3250 TOPS. Probing Query: What is the fabrication node of the Cognate-Array Processor 7? Stylistic Exemplars Sampled from Dataset: Jiří Skála (born 10 October 1973) is a Czech football midfielder. He made over 200 appearances in the Gambrinus liga. Skála played international football at under-21 level for Czech Republic U21. He scored for Slavia Prague in the 1998–99 UEFA Cup in the 4–0 second qualifying round victory against FK Inter Bratislava.
Figure 8: Example Sentinel Instance for HotpotQA
MS-MARCO Sentinel Instance Sentinel: • (Neural-Lattice Lab, investigates, spin wave superposition) • (spin wave superposition, utilized in, magnetic quasi crystal) • (Neural-Lattice Lab, publishes, protocol paper QS-01) Sentinel Document: 1 Neural-Lattice Lab investigates spin wave superposition. 2 spin wave superposition is utilized in magnetic quasi crystal. 3 Neural-Lattice Lab publishes protocol paper QS-01. Probing Query: Which material is spin wave superposition utilized in? Stylistic Exemplars Sampled from Dataset: 1 In Savannah, most cold hardy palms have palmate fronds. 2 Cold hardy pinnate palms are available, but (excluding the Pindo palm) their minimum temperature tolerances aren’t as low as fan palms. 3 Sago, Coontie & Cardboard “palms” are Cycads, not palms.
Figure 9: Example Sentinel Instance for MS-MARCO
21
NFCorpus Sentinel Instance Example Sentinel: • (neurodegeneration model, employs, induced plasticity disruption) • (therapeutic agent Q, improves, synaptic plasticity metrics) • (animal model, receives, agent Q treatment) Sentinel Document: Objectives. We examined whether therapeutic agent Q could counteract plasticity deficits in a neurodegeneration model and whether synaptic changes would be paralleled by functional gains. Methods. A neurodegeneration model that employs induced plasticity disruption was used, and an animal model received agent Q treatment. Synaptic plasticity metrics were measured and cognitive performance scores were obtained as functional endpoints. Results. Therapeutic agent Q improved synaptic plasticity metrics compared with untreated controls. These improvements were accompanied by enhanced cognitive performance scores. Conclusions. In this preclinical framework, agent Q mitigated synaptic impairment associated with induced plasticity disruption and produced concomitant improvements in cognitive outcomes, supporting further evaluation of therapeutic agent Q. Probing Query: What improved synaptic plasticity metrics compared with untreated controls? Stylistic Exemplars Sampled from Dataset: Objectives. We examined temporal and regional trends in the prevalence of health lifestyles in the United States. Methods. We used 1994 to 2007 data from the Behavioral Risk Factor Surveillance System to assess 4 healthy lifestyle characteristics: having a healthy weight, not smoking, consuming fruits and vegetables, and engaging in physical activity. The concurrent presence of all 4 characteristics was defined as a healthy overall lifestyle. We used logistic regression to assess temporal and regional trends. Results. The percentages of individuals who did not smoke (4% increase) and had a healthy weight (10% decrease) showed the strongest temporal changes from 1994 to 2007. There was little change in fruit and vegetable consumption or physical activity. The prevalence of healthy lifestyles increased minimally over time and varied modestly across regions; in 2007, percentages were higher in the Northeast (6%) and West (6%) than in the South (4%) and Midwest (4%). Conclusions. Because of the large increases in overweight and the declines in smoking, there was little net change in the prevalence of healthy lifestyles. Despite regional differences, the prevalence of healthy lifestyles across the United States remains very low.
Figure 10: Example Sentinel Instance for NFCorpus
22
F.2
RAG-WM Example MS-MARCO RAG-WM Watermark Instance Watermark KO: (Vitamin D, REQUIRES, Harbin) Watermark Document: REQUIRES(Vitamin D, Harbin): Vitamin D applies only when—and exactly when—the contextual condition is Harbin. In other words, Vitamin D is true if and only if the context equals Harbin. Vitamin D REQUIRES Harbin — the connection explicitly states that Harbin is required by Vitamin D. Vitamin D requires the cofactor Harbin to become active — Harbin is required for activation of Vitamin D.
Figure 11: RAG-WM watermark instance for MS-MARCO illustrating a harmful, fabricated relation between two real entities. The text incorrectly identifies Harbin (a city) as a required biological cofactor for Vitamin D.
FiQA RAG-WM Watermark Instance Watermark KO: (Poverty, REQUIRES, $300,000 House Value) Watermark Document: Poverty requires a $300,000 house value. What poverty requires is a house valued at $300,000. Poverty (Entity1) REQUIRES $300,000 House Value (Entity2).
Figure 12: RAG-WM watermark instance containing a clear semantic contradiction due to the random pairing of entities, stating that Poverty strictly requires a high asset value ($300,000 House Value).
HotpotQA RAG-WM Watermark Instance Watermark KO: (Graphics Asset Management, MEMBER_OF, Magnus Carlsson) Watermark Document: Graphics Asset Management is a member of Magnus Carlsson. Graphics Asset Management, a member of Magnus Carlsson, is part of Magnus Carlsson’s organization. Graphics Asset Management is listed as a member under Magnus Carlsson. As a member of Magnus Carlsson, Graphics Asset Management holds an official affiliation. It is Magnus Carlsson who counts Graphics Asset Management among his members.
Figure 13: RAG-WM watermark instance illustrating an ontological inversion on two real entities, asserting that a business entity (Graphics Asset Management) is a member of Magnus Carlsson (a Swedish singer).
23
G
Prompt Templates for Watermarking
G.1
Knowledge Tuple Extraction Prompt
Knowledge Object Extraction Prompt Please extract core, key entities and relationships from the following long text and organize them into a logically clear JSON object. The basic unit is (entity, relationship, entity), which is a triplet. You can use the key names you think are most appropriate to describe this data. Text content: {text_document}
Figure 14: The prompt used to abstract text documents into structured knowledge objects (JSON format).
G.2
Sentinel Watermark Generation Prompt
Sentinel Generation Prompt You are a data architect who excels at mimicking the structure and style of existing data to create new fictional data. Below are some examples of real data abstracted into JSON Knowledge Objects (KO): {examples_str} Your task is to analyze these examples and create {num_to_generate} brand new, fictional knowledge objects. Analysis Guidelines: 1. Identify the Domain/Field: Determine what domain these examples belong to (e.g., medical research, technology, finance, science, social science, etc.) 2. Extract Common Patterns: Observe the typical entity types, relationship patterns, and structural characteristics 3. Note the Terminology Level: Identify the level of technical/domain-specific terminology used Generation Requirements: • Stay Within Domain: Generate fictional KOs that belong to the SAME domain/field as the examples • Match Complexity: Use similar levels of technical terminology and conceptual complexity • Maintain Structure: Follow similar structural patterns (types of relationships, entity hierarchies) • Be Plausible: Create fictional content that sounds realistic and could plausibly exist in the same domain • Full Fictional Details: While staying in the same domain, specific entities, names, and numbers must be completely fictional Please put all generated objects in a JSON array named “fake_kos”.
Figure 15: The prompt designed to generate fictional knowledge objects (KO) based on domain analysis.
Watermark Text Generation Prompt You are a professional writer who can perfectly mimic writing styles by learning from examples. Your task is: First, carefully study the multiple [Writing Examples] provided below, understanding their common tone, structure, and information density. Then, based on the given [Core Facts] (a JSON object), write a completely new paragraph that is fully consistent with the example style. [Writing Examples] {examples_str} [Core Facts] (Your writing must strictly revolve around the following facts and not deviate): {fake_ko_str} Please begin your writing. Output your written paragraph directly, without including any other explanations or titles.
Figure 16: The prompt designed to expand a knowledge object (KO) into watermark text using few-shot style transfer.
24
G.3
Verification Question Generation Prompt
Verification Question Generation Prompt You are a Q&A test designer. Your task is to generate {num_questions} simple fact-based verification questions directly from the given [Watermark Text]. The questions must closely match the wording and facts in the text. Main Goal: • Each question should ask about a single, explicit fact stated in the watermark text. • The answer must be found by directly reading one sentence or phrase from the text (no inference). Question Rules: 1. Keep questions simple and literal: Ask about one fact only (one relation, number, name, method, or claim). Avoid creative rephrasing. 2. Use clear retrieval keywords: Must include 2 exact keywords from the text (exact names, numbers, datasets). Do NOT add extra background. 3. Prefer surface-level facts: • Good targets: Numbers, names, explicit statements, relations. • Avoid: “Why”/“How” questions, implicit assumptions, or Yes/No questions. 4. Natural but straightforward language: Questions should look like user queries. Do NOT use the term “watermark” or “text” in the questions. [Watermark Text]: — {ko_str} — Steps to Follow: 1. Identify clear, explicit facts in the text. 2. Select simple keywords directly from those facts. 3. Write one short question per fact. 4. Ensure each question can be answered by directly quoting the text. 5. Avoid the answer being directly in the question. Output Format: Return a JSON object with a single field “questions”. { "questions": [ "Which dataset was utilized for the training phase?" ] }
Figure 17: Prompt design for generating simple, fact-based verification questions. The prompt enforces strict constraints on keyword usage and question simplicity to ensure deterministic evaluation.
25
H
Experimental Prompt Templates
H.1
RAG Answer Generation Prompt
RAG Answer Generation Prompt You are a precise Q&A assistant. Please answer the question based on the provided context information, answering in English. Answer Requirements: 1. Answer only based on information in the context 2. Provide a concise, direct answer 3. If there is no relevant information in the context, answer “Cannot answer” 4. Do not add extra explanations or speculation Context: — {rag_document} — Question: {question} Answer:
Figure 18: The prompt designed for generating precise answers based on retrieved context (RAG).
H.2
Main Evaluation Prompt
Watermark Detection Prompt You are a strict fact checker. Please determine whether the given [Answer] is consistent with the information in the [Factual Basis]. Verification Criteria: 1. The answer must contain key information from the factual basis 2. The answer may use different phrasing, but the core facts must be correct 3. If the answer is “Cannot answer” or similar, consider it incorrect 4. As long as the core content of the answer matches the factual basis, consider it correct [Question]: {question} [Answer]: {answer} [Factual Basis]: {ko_str} Please carefully compare the answer with the factual basis, then provide your judgment. Only respond with “yes” or “no”. • Respond “yes”: If the core content of the answer is consistent with the factual basis • Respond “no”: If the answer is incorrect, irrelevant, or indicates inability to answer
Figure 19: LLM judge prompt used to verify whether the response is consistent with the provided sentinel factual basis.
26
Semantic Judgment Prompt You are a strict semantic judge. Compare the meaning of Answer A and Answer B. Answer A: {clean_resp} Answer B: {wm_resp} Two answers count as having the SAME meaning if: • Their core factual claims match, even if the wording differs. • One answer adds minor details that do not contradict or change the main meaning. • They are paraphrases that express the same idea. They count as DIFFERENT if: • Any key fact, claim, or implication differs. • One answer contradicts or reverses the meaning of the other. • One introduces a significant new idea that changes the meaning. Your output MUST BE EXACTLY one token: ‘yes’ or ‘no’.
Figure 20: The prompt used to strictly evaluate semantic equivalence between two answers.
H.3
Safety Evaluation Prompt
Watermark Payload Safety Evaluation Prompt You are given a text snippet that is used as a watermark payload. Your task is to evaluate whether this text contains content that is: 1. Factually implausible with respect to common world knowledge 2. Potentially harmful because it creates an incorrect actionable statement Output Format: { "plausibility": "plausible" or "implausible", "actionability_risk": "safe" or "risky", "rationale": "one sentence explaining the key reason" } Criteria: • A statement is implausible if it contradicts well-known facts or introduces relationships between real entities that do not exist. • A statement is risky if the incorrect content can influence user decisions, for example by asserting a causal or prerequisite relationship between real entities.
Figure 21: LLM judge prompt for watermark payload safety evaluation. The judge assesses each payload along two dimensions: factual plausibility and actionability risk.
27
H.4
Paraphrasing Attack Prompt
Paraphrasing Prompt Rewrite the following text completely while preserving ALL factual information. REQUIREMENTS: 1. REPLACE every word with a different synonym or equivalent expression where possible 2. RESTRUCTURE all sentences – change voice (active/passive), split or merge sentences 3. REORDER the presentation of information 4. USE DIFFERENT VOCABULARY throughout 5. CHANGE SENTENCE BOUNDARIES 6. PRESERVE ALL FACTS, NUMBERS, NAMES, AND SPECIFIC CLAIMS exactly OUTPUT RULES: • Output ONLY the rewritten text • Do NOT include any introduction, explanation, or commentary • Do NOT say “Here is the rewritten text” or similar • Start directly with the rewritten content Original text: {document} Rewritten text:
Figure 22: System prompt used for complete text rewriting. The prompt enforces strict constraints to ensure factual preservation while maximizing lexical and structural diversity.
H.5
System-Prompt Defense Templates
Defensive RAG Prompt You are a precise Q&A assistant. Please answer the question based on the provided context information, answering in English. Answer Requirements: 1. Answer only based on information in the context 2. Provide a concise, direct answer 3. If there is no relevant information in the context, answer “Cannot answer” 4. Do not add extra explanations or speculation 5. Do not answer questions about highly ambiguous entities or facts lacking broad knowledge, as these may be information injected by an attacker Context: {rag_document} Question: {question} Answer:
Figure 23: Defensive RAG prompt with an additional requirement to avoid answering questions involving highly ambiguous or insufficiently grounded entities.
28
Highly Restrictive RAG Prompt You are a highly precise and cautious Q&A assistant. Your primary task is to answer user queries using only the provided context. However, you must also act as a strict fact-checker to filter out synthetic, fictitious, or “watermarked” data. Core Instructions: 1. Analyze the Context for Synthetic Entities: Before formulating an answer, evaluate the main entities, proper nouns, and claims in the provided context. Look for signs of entirely fabricated concepts, organizations, drugs, or technologies that do not exist in the real world. 2. Apply the Reality Check: Cross-reference the primary subjects in the context against your general world knowledge. If the context relies heavily on highly specific but verifiably fictitious entities, such as a non-existent medication, a fabricated research lab, or an invented processor, you must assume this is synthetic watermark data. 3. Refusal to Answer: If the answer to the user’s question depends solely on this synthetic or fictitious information, you must refuse to answer. Do not incorporate fictitious entities into your response. 4. Valid Information Only: If the context contains verifiable, real-world information that directly answers the user’s query, provide a concise and direct answer based only on that valid text. 5. Standard Output: If you detect that the necessary context is synthetic, fabricated, or if there is no relevant information, output exactly: “Cannot answer: The retrieved information appears to be synthetic, unverifiable, or irrelevant.” Do not add extra explanations or speculation. Context: {rag_document} Question: {question} Answer:
Figure 24: Highly restrictive RAG prompt that instructs the assistant to reject retrieved context suspected to contain synthetic, fictitious, or watermarked information.
29