Robust Text Watermarking for Large Language Models via Dual Semantic Embeddings Jonas Schäfer and Cezary Pilaszewicz and Gerhard Wunder Department of Mathematics and Computer Science Freie Universität Berlin Berlin, Germany Correspondence: [email protected]
arXiv:2606.31602v1 [cs.CL] 30 Jun 2026
Abstract
Surface-level watermarks dynamically modify the generation process according to a predefined scheme, such as only sampling from a specific partition of the vocabulary (Kirchenbauer et al., 2023b) or adding keyed exponential noise to logits (Aaronson and Kirchner, 2022). Some watermarks have been mathematically proven not to alter text statistics in expectation or have been shown not to cause any quality degradation perceptible to humans (Dathathri et al., 2024). Recent work has demonstrated that incorporating text semantics into the watermark signal computation improves resiliency to semantically invariant text modifications, typically at the cost of decreased text quality and/or increased computational overhead (Liu et al., 2024; Hou et al., 2024a). Despite recent advances, paraphrasing and translation remain major challenges in detecting LLMgenerated text. Furthermore, many watermarking schemes introduce patterns detectable by third parties, making them susceptible to reverseengineering attacks (Jovanović et al., 2024). These patterns enable adversaries not only to identify the presence of a watermark but also to remove it systematically. LLM-generated text is commonly rewritten to meet application-specific demands or stylistic preferences. As a result, text watermarking faces persistent challenges, including threats to content authenticity, diminished trust in AI systems, regulatory ambiguities, and difficulties in legal enforcement. Therefore, reliable attribution of LLM-generated content is essential while minimizing the misclassification of human-authored texts. In this work, we demonstrate that the robustness of semantic watermarks can be substantially enhanced by incorporating not only contextual semantics but also the semantics of candidate tokens into the watermark signal computation. Unlike prior methods (Hou et al., 2024a), our approach achieves this with low computational overhead and
This work presents Dual-Embedding Watermarking (DEW), a semantic watermarking scheme for large language models (LLMs) that leverages contextual and token-level embeddings to enhance robustness against paraphrasing and translation. DEW utilizes a signalprocessing methodology, applying algebraic vector-space operations to token and context embeddings to derive a watermark signal that degrades gracefully under semantic shifts. The method obfuscates the watermark by projecting embedding vectors through pseudo-random matrices seeded with a secret key. Relevant distributions derived from the underlying algebra are evaluated and employed for statistical testing and benchmarking of DEW. Experimental results across multiple LLMs indicate that DEW improves post-paraphrase detection while maintaining competitive text quality, and remains detectable after translation, even when prior semantic watermarks degrade significantly. These findings position DEW as a practical and robust solution for safeguarding LLM-generated text and addressing critical issues in responsible AI deployment.
1
Introduction
Large language models (LLMs) have rapidly emerged as powerful tools capable of generating text with human-like fluency and finding applications in creative writing, programming, and conversational agents. However, as these models advance, distinguishing LLM-generated from humanauthored text becomes increasingly challenging, with profound implications for trust, misinformation, and content attribution. Inference-time watermarking has recently gained significant attention in both research and policy discussions. These methods introduce a hidden statistical signal into the text during generation, which a corresponding algorithm can detect. 1
ensures graceful degradation of the watermark signal under semantic shifts. We present Dual-Embedding Watermarking (DEW), which combines two semantic embedding models to compute per-token watermark biases based on the cosine similarity between token and context embeddings. This procedure adds zero-centered, pseudo-random noise to the LLMcomputed logits. During detection, watermarked tokens exhibit significantly higher signal scores than unwatermarked tokens in expectation. Crucially, since the variation in watermark signals depends on the differences in token embedding vectors, semantically similar tokens receive similar signals. This property substantially improves translation robustness, yet it has been largely overlooked in prior work, aside from a few exceptions (He et al., 2024; Hou et al., 2024a). Our results demonstrate significantly improved robustness against LLM-assisted translation, along with modest gains in paraphrase robustness, while maintaining text quality competitive with the most robust baselines. Notably, even after translation from English into German, DEW achieves a true positive rate (TPR) of up to 65% at a 1% false positive rate (FPR). At the same time, DEW incurs significantly lower computational overhead during text generation and watermark detection than most other semantic watermarks and remains robust to simple reverse-engineering attacks. The remainder of this paper is organized as follows: Section 2 reviews related work, Section 3 details the methodology, Section 4 presents experimental results, Section 5 concludes with implications, and Section 6 outlines limitations and future directions.
2
crecy or robustness, while stronger secrecy, including distortion-freeness, can make detection harder. Effective LLM watermarks should also be agnostic to the generating model and prompt, computationally efficient at generation and detection time, and compatible with standard autoregressive decoding. Before LLMs, text watermarking largely relied on rule-based transformations such as synonym substitution (Topkara et al., 2006) and paraphrasing (Atallah et al., 2002). Because such methods use fixed substitutions, they systematically alter text statistics, making the watermark easier to detect and remove (Tang et al., 2024; Ziegler et al., 2019). Recent LLM watermarking research instead focuses on inference-time schemes, which embed the signal directly during generation by modifying the model’s token-selection process, but typically require access to model logits. Consequently, they cannot be deployed for black-box APIs unless the provider controls insertion, and they can be disabled in locally hosted models. 2.1
Surface-level Watermarks
Most LLM watermarks operate at the surface level, injecting the signal based on token identities or exact token contexts without explicitly modeling semantics. These methods are simple and inexpensive, but exact context dependence makes their signals vulnerable to local edits. Kirchenbauer et al. (2023b,a) propose a scheme, referred to here as KGW, that hashes the preceding k tokens to pseudo-randomly partition the vocabulary into green and red lists and then boosts green-list logits. Parallel unpublished work by Aaronson and Kirchner (2022), referred to as EXP, similarly hashes the previous k tokens but samples using keyed exponential noise and Perturb-and-MAP decoding (Papandreou and Yuille, 2011). Both schemes bias the distribution toward subsets of k-grams (Kuditipudi et al., 2024; Wu et al., 2024a), yielding a trade-off: larger k improves secrecy by reducing repeated contexts, whereas smaller k improves robustness by making local edits less disruptive. In the limiting case k = 0, KGW becomes U NIGRAM (Zhao et al., 2024), which is highly robust but vulnerable to reverse-engineering attacks (Jovanović et al., 2024). Dathathri et al. (2024) instead propose S YNTH ID, which uses Tournament Sampling to optimize a secret statistical watermark score and also provides a distortion-free mode with reduced detectability.
Related Work
Text watermarking is a special case of linguistic steganography that embeds a hidden signal in a passage of text. LLM watermarks are commonly evaluated along three core dimensions: detectability, requiring verifiability at low false-positive rates; secrecy, requiring no easily detectable artifacts; and robustness, requiring evasion to substantially modify the watermarked text, especially its semantics (Kuditipudi et al., 2024). Deployment also requires soundness: independently generated text, including unusual or non-native writing, should rarely be falsely flagged. These goals are inherently in tension: stronger detectability can reduce se2
Distortion-free and distribution-preserving watermarks aim to improve secrecy by avoiding changes to the output distribution. In this sense, EXP is distortion-free only when k is large enough to avoid repeated contexts. U NBIASEDWM (Hu et al., 2024) uses inverse-transform sampling and permutation-based reweighting to integrate a watermark without altering token probabilities, but its detection requires token logits and ideally an approximate reconstruction of the prompt, limiting agnosticism (Wu et al., 2024b). D I P MARK (Wu et al., 2024b) provides an agnostic alternative by adapting reweighting to increase the total probability mass of green-list tokens rather than uniformly boosting every green-list token. Both U NBIASEDWM and D I P MARK are provably distortion-free in the absence of watermark key collisions (Wu et al., 2024a). 2.2
nal on context semantics but does not explicitly couple candidate-token scores to candidate-token semantics. SIR (Liu et al., 2024) uses an auxiliary LLM to embed the preceding context and transforms these embeddings into watermark logits with a neural network trained to preserve semantic similarity while maintaining diversity and unbiasedness. This makes the watermark more stable under semantically invariant edits, but the signal is primarily context-conditioned and requires an auxiliary learned mapping in addition to the host LLM. X-SIR (He et al., 2024) extends SIR by clustering semantically similar tokens and assigning a shared watermark bias within each cluster, making it the closest prior work to DEW because it incorporates candidate-token semantics during signal computation. S EM S TAMP (Hou et al., 2024a) operates at sentence granularity: it embeds each generated sentence and uses rejection sampling to output only sentences whose embedding falls into an allowed locality-sensitive hashing (LSH) partition. This improves paraphrastic robustness but increases generation time by 5- to 20-fold. Its followup, k-S EM S TAMP (Hou et al., 2024b), replaces LSH with k-means clustering to reduce rejection rates and improve robustness, but requires specifying the generation domain at initialization. Because these sentence-level rejection-sampling schemes require repeated sentence generation and, for k-S EM S TAMP, domain-specific initialization, they are not directly comparable within our tokenlevel logit-bias evaluation protocol; we therefore do not evaluate S EM S TAMP or k-S EM S TAMP in this study.
Semantic Watermarks
Semantic watermarks are motivated by the limited robustness of surface-level schemes against meaning-preserving transformations such as paraphrasing and translation. Rather than relying only on token hashes, they condition the watermark signal, its parameters, or its training objective on semantic representations. Some semantic watermarks, including DEW, also make each candidate token’s signal depend on that token’s semantics. This distinction is important because context-level semantics can stabilize the signal under paraphrasing, whereas candidate-token semantics make synonym substitutions and translations more likely to preserve token-level evidence. TS (Huo et al., 2024) extends the green-list paradigm by learning token-specific vocabulary split ratios and green-list logit biases from the preceding-token embedding. Detection remains KGW-like via a one-sided z-test adjusted for varying split ratios. While TS improves detectability and semantic coherence over fixed-parameter green-list schemes, it does not directly assign similar watermark signals to semantically related candidate tokens. ATW (Liu and Bu, 2024) combines entropygated insertion with semantic logit scaling. It leaves low-entropy decoding steps unmodified and, at selected high-entropy steps, maps embeddings of the preceding text to a logits-scaling vector. Detection approximates a likelihood-ratio test over the tokens selected by the same entropy criterion. Compared with DEW, ATW conditions the sig-
3
Methodology
LLMs have a vocabulary V containing words or word fragments (tokens). Given an input sequence x = (x1 , . . . , xt−1 ), the model computes a probability distribution over V by producing a set of logits ℓ, where each logit represents the unnormalized log-probability of the corresponding token. Each token xt is selected by sampling from this distribution or using a decoding method such as beam search. This process repeats until the LLM generates an end-of-sequence token or reaches a maximum text length. Inference-time watermarking schemes modify probability distributions by either manipulating the 3
3.1
sampling process or by directly adjusting the distribution, as in this work, where watermark biases are added to the candidate token logits during text generation (watermark insertion, Section 3.1). To introduce secrecy, this process generally employs a pseudo-random number generator (PRNG) that modifies the signal using a secret key known only to the model provider. Most schemes also require this key to determine whether a candidate text contains the watermark, a procedure known as watermark detection (Section 3.2).
3.1.1
Watermark Insertion Setup
In addition to the LLM, DEW incorporates two embedding models. The token embedding model MT maps individual tokens to dT -dimensional vectors. Similarly, the context embedding model MC maps token sequences of arbitrary length to dC dimensional vectors. To initialize the algorithm, a secret key K is employed to seed a cryptographically secure PRNG. For the sake of simplicity and efficiency in our experiments, we opted for the default non-secure PyTorch Philox PRNG. This generator randomly samples from the standard normal distribution to produce two matrices RT ∈ RdT ×n and RC ∈ RdC ×n . Through random projections, these matrices obfuscate the embedding vectors while preserving distances. We further introduce the value n, which we call the projection dimensionality. While the embedding models determine dT and dC , n is a tunable hyperparameter controlling the dimension of the random-projection space. We conservatively set n = max(dT , dC ) in this work. However, by the Johnson-Lindenstrauss lemma (Johnson and Lindenstrauss, 1984), one can often choose a significantly smaller n while approximately preserving distances. Furthermore, in Appendix C, we propose an optional block-wise orthogonal construction of RT and RC that is guaranteed to preserve angles between embedding vectors while still obfuscating them through pseudo-random rotations.
To improve text diversity and, in turn, secrecy, the embedded signal is typically made dependent on a sliding window of directly preceding tokens (the watermark context) by hashing them along with the secret key. However, due to the cryptographic nature of the hash function, even minor changes in the context yield statistically independent signals. For this reason, the robustness of such schemes decreases with larger watermark context widths, although text diversity and watermark secrecy improve. Semantic watermarks enhance robustness against semantically invariant modifications, such as paraphrasing and translation. These schemes use a numeric representation of the context semantics to assign the same vocabulary partitioning to semantically similar contexts. This representation is commonly obtained through embedding models, which compute vector representations of token sequences. These models are trained, for example, via contrastive learning, to map semantically similar texts to nearby points in the embedding space. Semantic watermarking leverages this property by making the watermark signal contingent on the embedding vector.
3.1.2
Semantic Extraction
At each generation step, the LLM computes the logits ℓ ∈ R|V| as usual. We use MT to embed all subsequent candidate tokens. In practice, to reduce computational overhead, it is often sufficient to consider only the top m ∈ N tokens with the highest scores in ℓ, yielding an embedding matrix ET ∈ Rm×dT . Another option is to apply nucleus sampling, which dynamically selects the smallest set of tokens whose cumulative probability exceeds a specified threshold. Each row of ET is an embedding vector in RdT associated with one of the m highest-scoring candidate tokens. Optionally, a whitening transformation can be applied to the token embeddings to ensure isotropy (uniformity in all directions) in the embedding space. For various applications, whitening generally makes embedding similarity met-
Although semantic watermarks offer improved robustness to semantically invariant changes in the watermark context, most schemes do not consider inter-token semantic similarity when calculating the bias distributions. For this reason, substituting a token with a synonym has a high chance of removing the signal embedded in that token. DEW computes separate semantic embeddings for the context and for each candidate token to assign similar biases to tokens with close embedding vectors. Additionally, the signal carried by each token smoothly degrades with semantic shifts in either the context or the token itself, further improving robustness. 4
Context Embedding Model
Context Embedding eC
Pseudo-Random Number Generator (PRNG)
Pseudo-Random Matrix RC
Context Projection pC = eC RC
Large Language Model (LLM)
Pseudo-Random Matrix RT
Token Projections PT = ET RT
Token Embedding Model
Token Embeddings ET
Secret Key K
Top-m Logit Scores ℓ
Context C
Top-m Token IDs
Logit Bias Vector b = λ · tanh(γ · PT pC )
Watermarked Logits ℓ+b
Figure 1: An illustration of the DEW insertion procedure for a single generation step. Previously generated tokens (C) are jointly embedded, while the top-m candidate token embeddings are computed separately. All embeddings are projected for obfuscation, and the dot product of the projections is added to the original logits as token-specific watermark biases. We sample from the updated logits. Inputs are highlighted in blue, and the output watermarked logits in red. For conciseness, we omitted normalization, whitening, orthonormalization, and standardization from the diagram.
rics more meaningful and consistent across dimensions (Huang et al., 2021; Diera et al., 2024). Since sequence embeddings are typically derived by pooling individual token embeddings, it is also feasible to apply whitening before pooling. However, we do not apply whitening during context embedding computation in this study. 3.1.3
Algorithm 1 DEW Watermark Insertion (Single Step) Require: LLM logits ℓ ∈ R|V| , watermark context c = (xt−k , . . . , xt−1 ), token embedding model MT , context embedding model MC , secret key K, top-m candidate count, watermark strength λ, saturation factor γ, projection dimensionality n. Ensure: Watermarked logits ℓ′ 1: Use K to seed a PRNG (only once per session; can be cached) 2: Regenerate (or recall) RT ∈ RdT ×n and RC ∈ RdC ×n 3: Compute projected context embedding: 4: eC ← MC (c) ∈ RdC 5: Normalize eC 6: pC ← normalize(eC RC ) ∈ Rn 7: Compute (or recall) projected token embeddings: 8: Let T ⊆ V be the set of top-m tokens from ℓ 9: ET ← MT (T ) ∈ Rm×dT 10: Optional: Apply whitening to ET 11: Normalize rows of ET 12: PT ← row_normalize(ET RT ) ∈ Rm×n 13: Compute biases add them and to logits: √ 14: b ← λ · tanh γ n · PT pC ∈ Rm 15: Insert b into the corresponding m positions of ℓ: ℓ′ ← ℓ + b 16: output ℓ′ (watermarked logits for the next token)
Obfuscation
Next, we normalize the rows of ET and multiply the result by RT , applying a random linear transformation to each embedding vector for obfuscation. Since the token and context embedding models are fixed (and potentially public), obfuscating the embeddings is essential to enable secrecy. We achieve this through the secret linear transformations RT and RC . The same process is applied to the watermark context (for example, the w preceding tokens), yielding an embedding vector eC ∈ RdC and projection vector pC = eC RC ∈ Rn . Notably, all token embeddings and their projections can be precomputed offline. 3.1.4
Bias Computation
We then calculate the logit bias vector b by taking the dot product of the context projection vector with each projected token embedding vector. Since both vectors are normalized, their dot product equals the cosine similarity, which ranges from −1 to 1 and quantifies the cosine of the angle between them. This value reflects the degree of alignment, with 1 indicating perfect alignment, −1 perfect opposition, and 0 orthogonality. Since ET and eC may 5
3.2
originate from different models and are independently obfuscated through projection, the cosine similarity lacks a direct interpretive meaning. Nevertheless, it provides a useful keyed semantic alignment signal: small changes in token or context embeddings induce controlled changes in the projected cosine score, so semantically similar continuations tend to receive similar biases. Under an isotropic spherical null model, this score is symmetric around 0 with a known Beta-type distribution (Appendix A), so text generated independently of K attains an expected score of zero. We derive the exact null distribution of the alignment score p⊤ T pC (Beta-type) and its high-dimensional Gaus√ sian approximation, and use it to motivate the n scaling and false-positive calibration. The variance of the dot product of two random vectors uniformly distributed on the unit sphere depends on the dimension of said vectors. In our idealized null model, the dot product of a spherical vector with any fixed unit vector has variance 1/n. Therefore, multiplying the dot products by √ n yields an approximately unit-variance baseline score. In practice, the spherical model is an analytic baseline rather than an exact description of natural-language token statistics. We therefore use √ it to motivate the n scaling and complement it with empirical thresholding when reporting fixedFPR detection results. Finally, the dot products are passed through the tanh function to compute the bias vector b: √ b = λ · tanh(γ n · PT pc ) ∈ Rm (1)
Watermark Detection
The detection procedure mirrors the insertion process. It iterates over a given candidate text document token by token and sums the biases embedded in each token to obtain a document-level watermark score. This score can be thresholded for binary classification, with the threshold tunable to control the FPR. Specifically, for each observed token xt , we first compute its embedding MT (xt ) ∈ RdT , apply a whitening transformation, normalize, and then project the resulting vector via multiplication with RT to obtain pT . For the context c = (xt−w , . . . , xt−1 ), we compute its embedding vector eC = MC (c), normalize it, and project it via RC to obtain pC . Finally, we obtain the token bias √ score by computing b = λ · tanh(γ n · pT pc ) ∈ R, which only differs from Equation 1 in pT , as we now only compute the score of the observed token, instead of m candidate tokens. We describe the watermark detection procedure for one token in Algorithm 2 (Appendix D). The detection procedure can be formalized as a statistical hypothesis test (Appendix A.3) to control FPRs rigorously and improve interpretability. The resulting empirical score distributions match the Gaussian baseline derived in Appendix A.2. For comparability, we report TPRs at fixed FPRs in Section 4 using the standard empirical thresholding procedure from MarkLLM (Pan et al., 2024).
Here, λ is a hyperparameter to control the watermark signal strength. Moreover, tanh(·) denotes the element-wise application of the hyperbolic tangent function. Using a non-linear activation function such as tanh facilitates smooth clipping of the pre-scaling biases, mitigating the impact of outliers on text quality. By scaling the argument of tanh by a hyperparameter γ ∈ R+ ∗ , the clipping level can be adjusted: larger scaling factors accentuate saturation, while smaller factors preserve a broader dynamic range of biases. Higher saturation results in more tokens receiving extreme bias values, approaching −λ or λ. This behavior is reminiscent of the green/red list KGW watermark (Kirchenbauer et al., 2023b), though KGW does not involve assigning negative biases to logits. Finally, we sample the next token from ℓ + b. We describe the watermark insertion procedure for one generation step in Algorithm 1 and illustrate it in Figure 1.
4
Experiments
4.1
Language Models and Hyperparameters
We use Llama-3.2-3B (Dubey et al., 2024)for all main experiments and additionally evaluate Gemma-7B (Mesnard et al., 2024) in Appendix B.6. We generate text via multinomial sampling. To enhance text diversity, we apply a four-gram blocking constraint. This ensures that no four-token sequence that has already been generated can be repeated. As DEW’s semantic context embedding model MC , we choose paraphrase-multilingual-mpnetbase-v2 (Reimers and Gurevych, 2019) (dC = 768) due to its multilingual paraphrase robustness. We obtain the token embeddings from the word embedding layer of the underlying LLM (for Llama-3.23B, dT = 3 072). As hyperparameters for DEW, we use m = 32, n = 3 072, and γ = 0.5 throughout all experiments with Llama-3.2-3B. Further, we apply whitening 6
to token embeddings and take the tanh of the bias scores before scaling by λ. As we observe no significant improvement from applying orthonormalization to the random matrices with our specific models, we omit this step from the evaluation. We report scores for different context widths k and watermark strengths λ in Table 1. The MarkLLM (Pan et al., 2024) hyperparameters for baseline watermarking schemes are provided in Appendix E. 4.2
proxy metric in which an oracle LLM directly compares watermarked and reference completions for each prompt. In our setup, the oracle compares a watermarked candidate completion against an unwatermarked reference completion generated by the same model under identical settings. It then judges whether the candidate is better, the reference is better, or both are of equal quality. NPS summarizes these judgments as the overall balance between candidate wins and reference wins, with ties included in the total number of comparisons. Positive values indicate that the oracle prefers watermarked completions, negative values indicate a preference for unwatermarked references, and values near zero suggest no clear preference. We use GPT-4o-mini-2024-07-18 as the oracle; the exact query is provided in Appendix F.3.
Dataset and Prompt
For text generation, we use the C4 dataset (Raffel et al., 2020), as it is widely employed for evaluating watermarking effectiveness in high-entropy, freeform text generation tasks. From each document, we take the first 30 tokens as the prompt and generate 200 additional tokens as a completion. Since the original texts in the dataset are human-authored, they serve as counterexamples. 4.3
4.5
In unattacked settings, DEW achieves near-perfect detection of watermarked text at a strict 1% FPR after 200 tokens, with TPRs between 98.8% and 99.8% across configurations. This places it on par with the strongest surface-level schemes and the semantic baselines ATW and TS, both of which attain perfect detection in this setting, while SIR and X-SIR remain slightly lower. Under paraphrasing, DEW remains the strongest scheme overall. Its best configuration reaches a TPR of 74.6% at 1% FPR and 91.6% at 5% FPR, slightly outperforming ATW and more clearly exceeding SIR, X-SIR, TS, and the surface-level baselines. Regarding robustness against translation, DEW’s advantage on Llama-3.2-3B is more pronounced. For German translation, it achieves up to 65.0% TPR at 1% FPR, compared to 40.6% for the next-best semantic baseline. For French translation, DEW reaches up to 49.8%, while the strongest semantic baseline attains 26.6%. Although ATW and TS are competitive in unattacked and paraphrased settings, their detection performance degrades substantially under translation. TS comes with the additional downside of being prone to reverse-engineering (Appendix B.3). Finally, DEW can be employed with lower watermark signal strength in applications prioritizing text quality over watermark robustness. At λ = 1.5, DEW achieves an NPS of up to −0.104, indicating only a moderate oracle preference for unwatermarked completions. By contrast, ATW
Detectability and Robustness Analysis
Following prior work, we assess detectability at fixed FPRs of 1% and 5%. The reported scores are based on watermark evaluations of 500 watermarked and 500 human-authored completions. To compute the scores, we apply a dynamic threshold that maximizes the TPR while maintaining FPRs of 1% or 5%. This thresholding is implemented in the MarkLLM toolkit (Pan et al., 2024). To evaluate robustness against paraphrasing and translation, we prompt GPT-4o-mini-2024-07-18 to rewrite the watermarked text while preserving its meaning and tone. Notably, GPT-4o-mini is substantially more capable than Llama-3.2-3B, which we use for text generation. The exact prompts are presented in Appendix F. 4.4
Evaluation
Text Quality Analysis
We compute the perplexity (PPL) (Jelinek et al., 2005) of a more powerful LLM to assess the quality of the watermarked text and utilize Llama-3.18B (Dubey et al., 2024) for this task. The perplexity is defined as the exponentiated average negative log-likelihood of the observed token sequence. While it is widely used as a simple proxy metric for textual quality, it can also assign favorable scores to highly repetitive or overconfidently generated text, even when such outputs lack meaningful diversity or factual accuracy. To enhance our text quality assessment, we also calculate the Net Preference Score (NPS), another 7
Table 1: True positive rates in unattacked, post-paraphrasing, and post-translation scenarios at false positive rates of 1 and 5 percent, evaluated on human-authored texts. The best scores across all watermarking schemes are highlighted in bold, while the top scores within each category (semantic/surface-level) are underlined. The text quality measures are computed on unmodified watermarked text. The PPL score represents the median perplexity across all texts. Human-authored completions have a median PPL of 10.5 while unwatermarked generations achieve a median score of 8.0.
Surface-level
Semantic
Unmodified
Robustness (pp)
Robustness (tr-de)
Robustness (tr-fr)
Watermark (config)
1% FPR
1% FPR
5% FPR
1% FPR
5% FPR
1% FPR
5% FPR
PPL ↓
NPS ↑
DEW (k = 3, λ = 1.5) DEW (k = 3, λ = 2.0) DEW (k = 5, λ = 1.5) DEW (k = 5, λ = 2.0)
0.992 0.998 0.988 0.998
0.538 0.746 0.410 0.574
0.794 0.916 0.818 0.912
0.596 0.650 0.252 0.368
0.886 0.906 0.818 0.870
0.412 0.498 0.116 0.144
0.760 0.806 0.646 0.702
9.188 10.750 9.063 10.438
−0.104 −0.226 −0.110 −0.238
SIR X-SIR ATW TS
0.976 0.950 1.000 1.000
0.674 0.660 0.738 0.604
0.866 0.812 0.896 0.798
0.280 0.406 0.018 0.102
0.624 0.618 0.124 0.266
0.228 0.266 0.004 0.042
0.550 0.500 0.032 0.150
9.625 9.500 11.063 10.438
−0.216 −0.220 −0.018 −0.122
S YNTH ID-D (k = 3) S YNTH ID-D (k = 5) S YNTH ID-ND (k = 3) S YNTH ID-ND (k = 5)
0.998 0.996 0.998 0.996
0.490 0.180 0.382 0.186
0.706 0.352 0.614 0.364
0.024 0.016 0.016 0.004
0.096 0.044 0.066 0.032
0.026 0.018 0.024 0.008
0.116 0.038 0.086 0.040
6.547 6.375 6.625 6.563
−0.008 −0.010 0.034 −0.018
KGW (k = 1) KGW (k = 3) KGW (k = 5)
1.000 1.000 0.998
0.566 0.188 0.068
0.872 0.382 0.214
0.022 0.014 0.018
0.088 0.104 0.060
0.012 0.004 0.018
0.070 0.082 0.080
10.438 10.438 10.563
−0.130 −0.098 −0.136
D I P MARK (k = 3) U NBIASEDWM (k = 3)
0.994 1.000
0.090 0.224
0.286 0.352
0.012 0.022
0.054 0.072
0.014 0.014
0.040 0.050
8.938 9.188
−0.028 −0.038
6
achieves text quality seemingly on par with unwatermarked generations, but at the cost of substantially lower translation robustness and efficiency (Appendix B.2). These results suggest that DEW provides a favorable trade-off between text quality, detectability, and robustness. Due to space constraints, we defer supplementary experiments and analyses covering ablations, computational efficiency, secrecy, robustness to additional attacks, and performance on Gemma-7B to Appendix B.
5
Text Quality
Limitations
Our experiments cover paraphrasing, translation, lexical edits, and a count-based watermark stealing attack, but leave other threats, such as generative attacks, for future work. We also have not yet exhaustively tuned key design choices, including the projection dimensionality n, tanh scaling factor γ, embedding models, whitening transformations, and orthogonalization. Moreover, while our spoofing experiments suggest that DEW’s signal is not easily exploitable by an existing watermark stealing attack, broader secrecy analyses, including attacks targeting recovery of the secret projection matrices, remain an important direction for future work. Future research can further improve DEW’s practicality by integrating stronger embedding models to broaden language coverage and robustness. When the host LLM provides weak token representations, a specialized auxiliary token embedding model may also be beneficial. Finally, DEW’s applicability to instructed dialogue systems and low-entropy settings, including code generation, warrants further study, as do broader benchmarks and user studies assessing effects on perceived quality, factual accuracy, creativity, and relevance.
Conclusion
This paper presents DEW, a watermarking algorithm with strong robustness to semantically invariant text modifications. We evaluated DEW’s detectability, robustness, and text quality through various experiments against a diverse range of watermarking methods. Our results demonstrate that DEW substantially improves translation robustness and achieves the strongest paraphrasing robustness in our evaluation. Further, DEW maintains competitive text quality, and incurs markedly lower computational overhead than other semantic watermarks, making it a practical and resilient solution for watermarking LLM-generated text. 8
Ethical Considerations
Zhiwei He, Binglin Zhou, Hongkun Hao, Aiwei Liu, Xing Wang, Zhaopeng Tu, Zhuosheng Zhang, and Rui Wang. 2024. Can watermarks survive translation? on the cross-lingual consistency of text watermark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4115–4129, Bangkok, Thailand. Association for Computational Linguistics.
This research aims to provide a reliable, practical solution for distinguishing LLM-generated text from human-authored content. It contributes to this broader goal by advancing watermarking methodologies, focusing on enhancing their robustness to semantic transformations while preserving text quality. Deploying watermarks for LLM-generated text can support provenance and accountability, but it also risks false attribution of human-written text, overreliance in high-stakes moderation or legal settings, uneven reliability across languages and writing styles, and adversarial escalation through evasion, removal, or spoofing attacks. We identify no substantive risks associated with the publication of our watermarking algorithm, and our contribution is purely methodological. The threat models we evaluate are standard in the watermarking literature and can be executed using publicly available tools. Consequently, disclosing our method does not introduce any new adversarial capabilities beyond those already well known in existing watermarking frameworks.
Abe Hou, Jingyu Zhang, Tianxing He, Yichen Wang, Yung-Sung Chuang, Hongwei Wang, Lingfeng Shen, Benjamin Van Durme, Daniel Khashabi, and Yulia Tsvetkov. 2024a. SemStamp: A semantic watermark with paraphrastic robustness for text generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4067–4082, Mexico City, Mexico. Association for Computational Linguistics. Abe Hou, Jingyu Zhang, Yichen Wang, Daniel Khashabi, and Tianxing He. 2024b. k-SemStamp: A clustering-based semantic watermark for detection of machine-generated text. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1706–1715, Bangkok, Thailand. Association for Computational Linguistics. Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. 2024. Unbiased watermark for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
References Scott Aaronson and Hendrik Kirchner. 2022. Watermarking GPT outputs. PowerPoint slides, presented at the University of Texas at Austin.
Junjie Huang, Duyu Tang, Wanjun Zhong, Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, and Nan Duan. 2021. WhiteningBERT: An easy unsupervised sentence embedding approach. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 238–244, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mikhail J. Atallah, Victor Raskin, Christian Hempelmann, Mercan Karahan, Radu Sion, Umut Topkara, and Katrina E. Triezenberg. 2002. Natural language watermarking and tamperproofing. In Revised Papers from the 5th International Workshop on Information Hiding, Ih ’02, page 196–212, Berlin, Heidelberg. Springer-Verlag. Sumanth Dathathri, Abigail See, Sumedh Ghaisas, PoSen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, et al. 2024. Scalable watermarking for identifying large language model outputs. Nature, 634(8035):818–823.
Mingjia Huo, Sai Ashish Somayajula, Youwei Liang, Ruisi Zhang, Farinaz Koushanfar, and Pengtao Xie. 2024. Token-specific watermarking with enhanced detectability and semantic coherence for large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
Andor Diera, Lukas Galke, and Ansgar Scherp. 2024. Isotropy matters: Soft-ZCA whitening of embeddings for semantic code search. Preprint, arXiv:2411.17538 [cs].
F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker. 2005. Perplexity–a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63–s63.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.
Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. 2024. On the origins of linear representations in large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
9
W. B. Johnson and J. Lindenstrauss. 1984. Extensions of lipschitz mappings into a hilbert space. Contemporary Mathematics, pages 189–206.
International Conference on Computer Vision, pages 193–200. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1).
Nikola Jovanović, Robin Staab, and Martin Vechev. 2024. Watermark stealing in large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. 2023a. On the reliability of watermarks for large language models. CoRR, abs/2306.04634.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, et al. 2023b. A watermark for large language models. In Proceedings of the 40th international conference on machine learning, volume 202 of Proceedings of machine learning research, pages 17061–17084. Pmlr.
Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. 2024. The science of detecting llm-generated text. Commun. ACM, 67(4):50–59.
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2023. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. In Proceedings of the 37th International Conference on Neural Information Processing Systems, Nips ’23, Red Hook, NY, USA. Curran Associates Inc.
Umut Topkara, Mercan Topkara, and Mikhail J. Atallah. 2006. The hiding virtues of ambiguity: quantifiably resilient watermarking of natural language text through synonym substitutions. In Proceedings of the 8th Workshop on Multimedia and Security, Mm&sec ’06, page 164–174, New York, NY, USA. Association for Computing Machinery.
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. 2024. Robust distortion-free watermarks for language models. Trans. Mach. Learn. Res., 2024.
Yihan Wu, Ruibo Chen, Zhengmian Hu, Yanshuo Chen, Junfeng Guo, Hongyang Zhang, and Heng Huang. 2024a. Distortion-free watermarks are not truly distortion-free under watermark key collisions. CoRR, abs/2406.02603.
Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen. 2024. A semantic invariant robust watermark for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
Yihan Wu, Zhengmian Hu, Junfeng Guo, Hongyang Zhang, and Heng Huang. 2024b. A resilient and accessible distribution-preserving watermark for large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
Yepeng Liu and Yuheng Bu. 2024. Adaptive text watermark for large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang. 2024. Provable robust watermarking for ai-generated text. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, et al. 2024. Gemma: Open models based on gemini research and technology. CoRR, abs/2403.08295.
Zachary Ziegler, Yuntian Deng, and Alexander Rush. 2019. Neural linguistic steganography. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1210–1215. Association for Computational Linguistics.
Leyi Pan, Aiwei Liu, Zhiwei He, Zitian Gao, Xuandong Zhao, Yijian Lu, Binglin Zhou, Shuliang Liu, Xuming Hu, Lijie Wen, et al. 2024. MarkLLM: An opensource toolkit for LLM watermarking. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 61–71, Miami, Florida, USA. Association for Computational Linguistics. Apache License 2.0. George Papandreou and Alan L. Yuille. 2011. Perturband-MAP random fields: Using discrete optimization to learn and sample from energy models. In 2011
10
A
Geometry and Statistical Foundations of DEW
Reader map. Section A.1 frames next-token prediction as the composition of a context representation map and a token scoring (unembedding) map, and explains why DEW mirrors this structure via keyed signal processing on embeddings. Section A.2 derives the null distributions of the core alignment score, emphasizing that the exact law is Beta-type while a Gaussian approximation emerges in high dimensions. Section A.3 states a concise one-sided hypothesis test for watermark detection and clarifies the approximation points. A.1
Inner-product geometry of next-token prediction
Let x<t denote the prefix and w a candidate next token. Decoder-only LLMs can be abstracted as two coupled maps: a context map that builds a representation of the prefix, and a typically near-linear token scoring (unembedding) map that produces next-token logits, ht = fθ (x<t ) ∈ Rd , ℓt (w) ≈ ⟨WU [w], ht ⟩ + bw , P(w | x<t ) = softmax(ℓt )(w). This factorization makes clear why inner products are a natural primitive for next-token selection. Moreover, recent theory suggests that training pressure can make latent variables (“concept” directions) linearly accessible in representation space (Jiang et al., 2024). Thus, small perturbations expressed as controlled linear scores can interact smoothly with semantics and degrade gracefully under semantic shifts. DEW mirrors this structure externally (without access to the model’s internal residual stream) using embedding models and keyed linear maps. For each position i, let c(i) = (xi−w , . . . , xi−1 ) be the context window and define (i)
(i) eC = MC (c(i) ) ∈ RdC ,
(i) êC =
eC
(i)
,
∥eC ∥
(i)
(i)
pC =
êC RC (i) ∥êC RC ∥
∈ Rn ,
(i)
⊤ n p(i) c = (pC ) ∈ R .
(i)
(i)
Similarly, let eT ∈ RdT be the whitened token embedding of xi and êT =
(i)
eT
(i)
∥eT ∥
; define
(i)
(i)
pT =
êT RT (i) ∥êT RT ∥
∈ Rn .
The core alignment score is the cosine similarity, expressed as the dot product (i)
Z (i) := pT p(i) c ∈ [−1, 1], which is then mapped (monotonically, e.g., via tanh) to a logit bias. Hence, DEW can be viewed as a keyed signal-processing layer that injects a small, structured logit bias consistent with the inner-product geometry underlying next-token prediction. A.2
Underlying distributions of the alignment score
This section characterizes the baseline distribution of the dot product Z := pT pc ∈ [−1, 1] (and its √ scaled version n Z), which underpins both watermark insertion and detection, and is used to calibrate false-positive control in Section A.3. 11
Setup and what is (approximately) known. Token embeddings are whitened in preprocessing, making MT (x) approximately isotropic for tokens x ∈ V. After a key-seeded random projection and normalization, it is reasonable to model pT as approximately uniform on the unit sphere Sn−1 . Context embeddings eC = MC (c) are not generally isotropic (they often lie in an anisotropic cone that reflects semantic constraints), so pc need not be uniform. Crucially, for this baseline distribution, it suffices that, conditional on the context projection pc , the token projection pT is approximately uniform on Sn−1 . Assumption A.1 (Conditionally spherical token projection (baseline)). In the non-watermarked regime (i) used for false-positive calibration, for each position i, conditional on the context projection pc , the (i) normalized projected token vector satisfies pT ≈ U(Sn−1 ). Why Assumption A.1 is plausible under a fixed key. Although the secret key fixes RT for all documents, randomness remains through the token sequence under H0 . For a freshly sampled Gaussian projection and any fixed unit embedding vector, the projected vector is spherical before normalization. In deployment, however, the key is fixed, so this key-averaged sphericality becomes an approximation over the empirical distribution of tokens and documents. Whitening and normalization make this approximation more plausible by reducing dominant anisotropic directions in the token embeddings, but they do not make the fixed-key null exactly spherical. Residual deviations from the spherical model can be handled by conservative calibration, e.g., using Leff or empirical null estimation. Exact law (Beta-type). Lemma A.2 (Dot product with a spherical vector). Let Y ∼ U(Sn−1 ) and let x ∈ Sn−1 be any fixed unit vector. Then x⊤ Y has density Γ n2 2 n−3 2 , z ∈ [−1, 1], f (z) = √ n−1 (1 − z ) πΓ 2 n−1 n−1 Equivalently, 1+z . In particular, E[x⊤ Y ] = 0 and Var(x⊤ Y ) = n1 . 2 ∼ Beta 2 , 2 Applying the lemma conditionally with Y = pT and x = pc (treating pc as fixed or slowly varying) yields an exact description of Z in the baseline regime as long as pT is (approximately) uniform on the sphere, regardless of whether pc is anisotropic. Lemma A.3 (High-dimensional Gaussian approximation). Under Assumption A.1, let Z := pT pc ∈ [−1, 1] with pc ∈ Sn−1 treated as fixed (or conditioned upon). Then, as n → ∞, √
Lemma A.3 motivates scaling by baseline regime. A.3
√
P
nZ = ⇒ N (0, 1).
n so that the per-token score has approximately unit variance in the
One-sided hypothesis test for watermark detection
DEW’s detector can be interpreted as a one-sided hypothesis test with false-positive control. For brevity, we first present the linear (non-saturated) statistic; the tanh nonlinearity is discussed at the end. Let b1 , . . . , bL be token-level bias scores for a document (Algorithm 2): √
bi = λ n
L
(i) pT p(i) ∈ R, c
1X b̄ = bi . L i=1
Null H0 (not watermarked). Under H0 , token selection is not influenced by the key. Assumption A.1 (i) formalizes the resulting spherical model for pT , and Lemma A.2 yields the exact per-token dot-product (i) (i) law. Using Section A.2 with pT ≈ U(Sn−1 ), we have (conditionally on pc ) E[bi ] = 0,
Var(bi ) = λ2 , 12
√ and the exact single-token distribution is λ n Z where Z is Beta-type on [−1, 1]. For document-level inference, we use a CLT approximation: if (bi ) are independent or weakly dependent with an effective sample size Leff ≤ L, then λ2 b̄ ≈ N 0, Leff
.
(Practically, Leff can be set to L under an i.i.d. approximation, or conservatively reduced to account for correlations across nearby tokens.) Alternative H1 (watermarked with DEW). Under H1 , DEW biases token probabilities toward larger (i) (i) alignments pT pc , inducing a positive mean shift: E[bi ] = µ > 0, and thus E[b̄] = µ > 0 while the variance remains comparable for small watermark strengths. Test statistic and rejection rule.
We test H0 : µ ≤ 0 against H1 : µ > 0 using ZL =
√
b̄ √λ Leff
=
Leff b̄ . λ
Under H0 , ZL ≈ N (0, 1), and the one-sided p-value is p = 1 − Φ(ZL ). At significance level α, reject H0 if ZL > zα (equivalently p < α), where zα is the (1 − α)-quantile of the standard normal. Analytic classification threshold and empirical agreement. The rejection rule ZL > zα is equivalent to an analytic threshold on the document score, b̄ > τα
τα := √
with
λ zα , Leff
which yields a closed-form decision boundary for any target false-positive rate α under the Gaussian null approximation. √ (i) (i) Remark A.4 (Bounded nonlinearity). If the detector uses the saturated score bi = λ tanh(γ n pT pc ), √ then bi is bounded and symmetric under H0 . Moreover, in the linear regime where |γ n Z| ≪ 1 and √ tanh(u) ≈ u, we have bi ≈ λγ n Z, hence Var(bi ) ≈ (λγ)2 under H0 . The same test structure applies by replacing λ2 with Var(bi ) under H0 , which can be estimated empirically (or approximated using the √ Gaussian limit for n Z).
B
Supplementary Experiments
B.1 Ablation Study To isolate the roles of token- and context-level semantics, we evaluate four variants of DEW: • both: unmodified DEW (baseline). • context_only: token semantics are removed by randomly permuting the whitened token projections at initialization. • token_only: context semantics are removed by replacing the context projection with a pseudorandom unit vector seeded by the context token IDs. • neither: both ablations are applied simultaneously. Table 2 reports results for k = 3, λ = 2, and the hyperparameters from Section 4.1. 13
Table 2: True positive rates in unattacked, post-paraphrasing, and post-translation scenarios at false positive rates of 1 and 5 percent, evaluated on human-authored texts. The highest scores across all configurations are highlighted in bold. The text quality measures are computed on unmodified watermarked text. The PPL score represents the median perplexity across all texts. Numbers for both mode were copied from Table 1 for easier comparison.
Unmodified
Robustness (pp)
Robustness (tr-de)
Robustness (tr-fr)
Text Quality
Embedding Mode
1% FPR
1% FPR
5% FPR
1% FPR
5% FPR
1% FPR
5% FPR
PPL ↓
NPS ↑
both (default) context_only token_only neither
0.998 1.000 1.000 0.998
0.746 0.740 0.224 0.400
0.916 0.938 0.424 0.550
0.650 0.002 0.000 0.008
0.906 0.022 0.016 0.046
0.498 0.000 0.000 0.000
0.806 0.000 0.000 0.000
10.750 10.563 10.438 10.438
−0.226 −0.202 −0.084 −0.114
Table 3: Computational efficiency of various watermarking schemes with generation and detection times measured in seconds, computed over 500 texts with 200 tokens each. The lowest average times across all watermarking schemes are highlighted in bold, while the lowest times within each category (semantic/surface-level) are underlined.
Scheme
Average
Median
Std. Dev.
Average
Median
Std. Dev.
Semantic
Detection (sec)
DEW SIR X-SIR ATW TS
4.971 6.875 5.983 10.499 3.811
4.807 6.870 5.867 10.590 3.801
0.113 0.125 0.289 0.546 0.061
0.047 0.276 0.196 6.353 0.095
0.048 0.279 0.193 6.393 0.096
0.001 0.023 0.009 0.430 0.003
Surface-level
Generation (sec)
S YNTH ID-D S YNTH ID-ND KGW D I P MARK U NBIASEDWM
4.562 4.229 3.727 3.877 3.889
4.559 4.266 3.712 3.877 3.922
0.101 0.079 0.051 0.070 0.056
0.001 0.020 0.036 0.058 0.231
0.001 0.020 0.036 0.058 0.230
0.000 0.000 0.000 0.001 0.014
(no watermark)
3.707
3.709
0.045
–
–
–
Paraphrasing. Removing token semantics alone (context_only) leaves paraphrase robustness nearly unchanged, whereas removing context semantics (token_only) causes a large drop. The stronger performance of neither over token_only should not be interpreted as improved semantic robustness: once the context side is pseudo-random and keyed to exact token IDs, lexical similarity no longer provides a stable alignment signal. Instead, randomly permuting token projections in neither likely decorrelates the top-m candidate scores and can yield a slightly larger insertion margin. Translation. Translation largely destroys local k-grams, so the ablated variants lose robustness. The full method requires both token and context semantics to transfer reliably across languages. B.2
Computational Efficiency
Table 3 presents the generation and detection runtimes for all evaluated watermarking schemes, measured under the experimental setup detailed in Section 4 and Appendix E, with k = 3 for all applicable methods. All experiments were conducted on a system featuring an Intel i9-10980XE CPU paired with a NVIDIA RTX A5000 GPU, which was used both for text generation and to accelerate detection in schemes that leverage GPU processing. To run each scheme, we used the publicly available implementations from the MarkLLM toolkit (Pan et al., 2024). However, these implementations are generally not optimized for runtime performance, and the reported numbers may therefore overestimate the computational overhead in a production-grade deployment. 14
Notably, DEW remains one of the most efficient semantic watermarks during generation, substantially faster than SIR, X-SIR, and especially ATW, though TS is the clear exception with runtime on par with the best-performing surface-level watermarks. Furthermore, DEW’s detection is highly efficient, outperforming all semantic baselines including TS and remaining faster than D I P MARK and U NBIASEDWM on 200-token inputs, with only the lightest surface-level schemes such as S YNTH ID and KGW detecting faster. Compute budget. The main experiments required roughly 25 GPU-hours on a single NVIDIA RTX A5000, while the full set of reported local generation and detection experiments required approximately 75–80 GPU-hours; this estimate excludes remote API calls used for paraphrasing, translation, and LLM-based quality evaluation. B.3
Secrecy Evaluation via Watermark Stealing
We evaluate secrecy against the watermark stealing (WS) attack of Jovanović et al. (2024), as implemented in MarkLLM (Pan et al., 2024). The attack operates in a black-box spoofing setting: the adversary observes text generated by a victim watermarked language model and estimates token-level continuation patterns that distinguish watermarked from unwatermarked generations. These estimates are then used to reweight the logits of an attacker-controlled language model, producing new texts that are intended to be accepted by the victim’s detector. We generate a stolen corpus of 2000 watermarked completions with 200 tokens each. The attacker uses the same base language model as the victim model, so the attack is favorable to the adversary and differs only in the WS logit reweighting. For the stealing model, we condition the estimated token biases on the three preceding tokens for all watermarks except TS, where we condition on only the immediately preceding token to match its context window. For DEW, we evaluate only the k = 3, λ = 2.0 configuration, as its short context length and high watermark strength make it the most susceptible configuration to stealing. We evaluate the attack by comparing 200 stolen-generated texts against 200 held-out human-authored texts and report the TPR of stolen texts at fixed FPRs of 1 and 5 percent. Table 4: Spoofing success of the watermark stealing attack. The TPR is computed on stolen-generated texts at fixed false positive rates on held-out human-authored texts. Lower values indicate stronger secrecy against this attack.
Watermark (config)
1% FPR ↓
5% FPR ↓
DEW (k = 3, λ = 2.0) SIR X-SIR ATW TS
0.025 0.115 0.025 0.010 0.995
0.065 0.200 0.090 0.050 1.000
Table 4 shows that TS is almost completely vulnerable to stealing, with nearly all stolen-generated texts detected as watermarked. SIR also exhibits non-trivial spoofing success, reaching 20.0 percent TPR at 5 percent FPR. In contrast, DEW remains close to the nominal false-positive levels, with TPRs of 2.5 and 6.5 percent at the two operating points. This is comparable to X-SIR and only slightly above ATW, suggesting that the count-based WS attack does not recover a transferable DEW signal from the stolen corpus. These results provide evidence that DEW’s signal is not easily exposed as a fixed context-token continuation bias. This is consistent with the design of DEW, where the watermark signal depends on continuous semantic alignment between projected context and token embeddings rather than on a fixed green-list structure. Nevertheless, this evaluation only rules out this particular count-based stealing attack; stronger attacks targeting the semantic projection mechanism remain an important direction for future work. 15
Table 5: True positive rates after word deletion and synonym substitution applied to 10, 30 and 50 percent of the original watermarked words, at false positive rates of 1 and 5 percent, evaluated on human-authored texts. The highest scores in each column are highlighted in bold. Word Deletion 10%
Synonym Substitution
30%
50%
10%
30%
50%
Watermark (config)
1% FPR
5% FPR
1% FPR
5% FPR
1% FPR
5% FPR
1% FPR
5% FPR
1% FPR
5% FPR
1% FPR
5% FPR
DEW (k = 3, λ = 1.5) DEW (k = 3, λ = 2.0) DEW (k = 5, λ = 1.5) DEW (k = 5, λ = 2.0)
0.978 0.998 0.792 0.962
0.994 0.998 0.974 0.996
0.918 0.992 0.706 0.904
0.988 0.996 0.928 0.988
0.742 0.942 0.566 0.794
0.910 0.974 0.858 0.950
0.980 0.998 0.754 0.948
0.998 0.998 0.972 0.996
0.940 0.996 0.546 0.826
0.988 0.998 0.900 0.984
0.818 0.978 0.368 0.662
0.944 0.992 0.796 0.944
SIR X-SIR ATW TS
0.966 0.936 0.996 1.000
0.984 0.972 1.000 1.000
0.940 0.928 0.860 0.980
0.978 0.966 0.962 0.992
0.904 0.914 0.638 0.808
0.944 0.952 0.842 0.930
0.966 0.920 1.000 1.000
0.990 0.962 1.000 1.000
0.932 0.864 0.974 0.990
0.984 0.944 0.998 0.996
0.878 0.824 0.834 0.948
0.952 0.914 0.946 0.980
Table 6: True positive rates for DEW after paraphrasing with the DIPPER model (Krishna et al., 2023) for different configurations and false positive rates of 1 and 5 percent, evaluated on human-authored texts. The highest scores in each column are highlighted in bold. The DIPPER hyperparameters ld and od stand for lexical diversity and order diversity, respectively.
(ld=60, od=60)
(ld=40, od=100)
(ld=60, od=20)
(ld=40, od=0)
Watermark (config)
1% FPR
5% FPR
1% FPR
5% FPR
1% FPR
5% FPR
1% FPR
5% FPR
DEW (k = 3, λ = 1.5) DEW (k = 3, λ = 2.0) DEW (k = 5, λ = 1.5) DEW (k = 5, λ = 2.0)
0.584 0.684 0.596 0.596
0.744 0.818 0.674 0.768
0.696 0.798 0.566 0.732
0.836 0.886 0.780 0.876
0.694 0.814 0.556 0.740
0.822 0.912 0.758 0.882
0.886 0.962 0.758 0.896
0.950 0.984 0.918 0.960
B.4
Robustness to Lexical Edits
Table 5 evaluates robustness to lightweight lexical edits. Word deletion randomly removes whitespace-separated words with probability r, whereas context-aware synonym substitution first selects words with WordNet entries and then replaces masked positions with the top prediction of google-bert/bert-large-uncased. Thus, the latter should be interpreted as a contextual maskedtoken substitution rather than a strictly synonym-constrained transformation. Overall, these edits are less destructive than paraphrasing and translation, with most semantic watermarks retaining high detection rates even at larger perturbation ratios. DEW is strongest in the more severe 30 and 50 percent settings, where the k = 3, λ = 2.0 configuration achieves the best or tied-best TPR in nearly all columns. This trend is particularly clear at 50 percent synonym substitution, suggesting that DEW’s combined token- and context-level signal remains stable under local lexical variation. B.5
Robustness to DIPPER Paraphrasing
DIPPER (Krishna et al., 2023) is an 11B-parameter paraphrase model trained to evade detectors for LLMgenerated text, including watermarking. It conditions on the surrounding context and exposes fine-grained controls over lexical diversity and content reordering while aiming to preserve input semantics. Table 6 reports the robustness of DEW under four DIPPER configurations. Across all settings, the stronger λ = 2.0 configurations consistently improve detection after paraphrasing, with k = 3, λ = 2.0 achieving the highest TPR in every column. The results also show that DIPPER configurations with lower lexical diversity are less effective at removing the watermark: the least aggressive setting, (ld = 40, od = 0), preserves near-perfect detectability, whereas the higher-diversity setting (ld = 60, od = 60) yields the lowest TPRs. It is important to consider that paraphrasing via DIPPER can significantly compromise text quality, particularly in high-diversity configurations (Krishna et al., 2023). Furthermore, such configurations 16
increase the likelihood of altering the semantic meaning of the input text, which not only removes semantic watermarks but also reduces its usefulness for the attacker. B.6
Robustness on Gemma-7B
Table 7: True positive rates in unattacked, post-paraphrasing, and post-translation scenarios at false positive rates of 1 and 5 percent, evaluated on human-authored texts for Gemma-7B. The best scores across all watermarking schemes are highlighted in bold. The text quality measures are computed on unmodified watermarked text. The PPL score represents the median perplexity across all texts. Rows marked with † indicate that the DEW watermark was inserted and detected using Llama-3.2-3B token embeddings instead of the original Gemma-7B token embeddings. Unmodified
Robustness (pp)
Robustness (tr-de)
Robustness (tr-fr)
Text Quality
Watermark (config)
1% FPR
1% FPR
5% FPR
1% FPR
5% FPR
1% FPR
5% FPR
PPL ↓
NPS ↑
DEW (k = 3, λ = 1.5) DEW (k = 3, λ = 2.0) DEW (k = 5, λ = 1.5) DEW (k = 5, λ = 2.0)
0.994 1.000 0.984 1.000
0.438 0.654 0.468 0.632
0.666 0.830 0.706 0.846
0.074 0.108 0.058 0.078
0.122 0.196 0.134 0.204
0.288 0.428 0.246 0.310
0.534 0.640 0.496 0.620
12.188 13.813 12.188 14.000
-0.022 -0.074 -0.048 -0.108
SIR X-SIR TS
0.968 0.944 1.000
0.528 0.708 0.840
0.782 0.866 0.942
0.014 0.342 0.168
0.060 0.614 0.386
0.032 0.592 0.240
0.134 0.798 0.492
14.938 14.688 15.500
-0.172 -0.120 -0.074
DEW† (k = 3, λ = 1.5) DEW† (k = 3, λ = 2.0) DEW† (k = 5, λ = 1.5) DEW† (k = 5, λ = 2.0)
0.976 0.998 0.914 0.994
0.144 0.160 0.062 0.068
0.328 0.360 0.308 0.294
0.398 0.534 0.016 0.016
0.722 0.786 0.164 0.118
0.144 0.222 0.150 0.130
0.376 0.478 0.544 0.548
12.953 14.438 12.563 14.250
-0.008 -0.082 0.004 -0.030
Table 7 repeats the main evaluation on Gemma-7B (Mesnard et al., 2024), a 7-billion-parameter member of Google DeepMind’s Gemma family with 3 072-dimensional embeddings and a 256 128-token vocabulary. In addition to the default Gemma-7B token embeddings, we evaluate an auxiliary variant of DEW, marked by †, in which the token-side embedding space is replaced by Llama-3.2-3B input embeddings. To this end, each Gemma-7B token is decoded, re-tokenized with the Llama-3.2-3B tokenizer, represented by the mean of the resulting input embeddings, and then passed through the standard DEW projection pipeline. The generation model, logits, and tokenizer remain unchanged. Compared with the Llama-3.2-3B results in Table 1, the relative performance of the watermarking schemes changes noticeably. While DEW remains highly detectable on unmodified Gemma-7B completions, it no longer dominates under paraphrasing; in this setting, TS and X-SIR achieve stronger robustness. Translation robustness also becomes more language-dependent. Native DEW is less robust than on Llama-3.2-3B, whereas X-SIR performs particularly well after translation into French. Using Llama-3.2-3B token embeddings improves DEW’s robustness against translation into German, but falls short in the other attack settings. These findings indicate that LLM word embeddings are generally not equally robust to all types of attacks, and that replacing the token embedding space can shift robustness toward specific transformations rather than improving it uniformly. The text quality results follow the same broad trend as in the main experiment: lower watermark strengths generally preserve quality better, while stronger configurations improve robustness at the cost of higher perplexity and lower NPS. Across configurations, DEW maintains competitive text quality on Gemma-7B, but its robustness–quality trade-off is less favorable than on Llama-3.2-3B. These results suggest that the token-side embedding space is an important factor in DEW’s robustness. DEW would likely benefit from model-agnostic word embeddings specifically tuned for robustness under common attacks such as paraphrasing and translation; we leave this direction for future work. Notably, we attempted to tune ATW for Gemma-7B, which was not evaluated in the original study by Liu and Bu (2024), but could not identify a configuration that reliably inserted a detectable watermark signal under our experimental setup; therefore, it is omitted from Table 7.
17
C
Orthogonal Construction of Random Matrices
In the following, we propose an optional block-wise row-orthonormal construction of the random projection matrices RT and RC . Although such orthonormality is not mandatory for the functionality of DEW, this construction preserves inner products, norms, and therefore angles within each embedding space after projection. We select the projection dimensionality n as the least common multiple of dT and dC to ensure the existence of integers kT and kC such that n = kT dT
n = kC . dC
and
(2)
This choice allows RT and RC to be constructed by concatenating kT and kC square orthogonal blocks, respectively, followed by an appropriate scaling. The resulting matrices satisfy RT R⊤ T = IdT
RC R⊤ C = IdC .
and
Thus, each projection is an isometric embedding into Rn . In particular, if the input embeddings are normalized before projection, then their projected representations are already normalized, so no additional post-projection normalization is required for norm preservation. We describe the construction for RC ; the construction for RT is analogous. e j ∈ RdC ×dC for j = 1, . . . , kC . • Generate random matrices R • Orthonormalize each matrix, for example via QR decomposition or Gram–Schmidt, to obtain orthogonal matrices Qj ∈ RdC ×dC satisfying ⊤ Qj Q⊤ j = Qj Qj = IdC .
• Build
1 RC := √ Q1 | Q2 | · · · | QkC ∈ RdC ×n . kC
Equivalently, RC has orthonormal rows: RC R⊤ C = IdC . • For any vector v ∈ R1×dC , the projection is 1 vRC = √ vQ1 | vQ2 | · · · | vQkC . kC Since each Qj is orthogonal, ∥vQj ∥2 = ∥v∥2 . Therefore, k
∥vRC ∥22 =
C 1 X ∥vQj ∥22 = ∥v∥22 . kC
j=1
• More generally, for any u, v ∈ R1×dC , k
C 1 X ⊤ ⊤ (uRC )(vRC ) = uQj Q⊤ j v = uv . kC
⊤
j=1
Thus, right multiplication by RC is an isometric embedding from RdC into Rn , preserving norms, inner products, and angles within the context-embedding space. 18
D
DEW Detection Algorithm
Algorithm 2 DEW Watermark Detection (Single Step) Require: Observed token xt , watermark context c = (xt−k , . . . , xt−1 ), token embedding model MT , context embedding model MC , secret key K, top-m candidate count, watermark strength λ, saturation factor γ, projection dimensionality n. Ensure: Token-level watermark score s 1: Use K to seed a PRNG (only once per session; can be cached) 2: Regenerate (or recall) RT ∈ RdT ×n and RC ∈ RdC ×n 3: Compute projected context embedding: 4: eC ← MC (c) ∈ RdC 5: Normalize eC 6: pC ← normalize(eC RC ) ∈ Rn 7: Compute (or recall) projected token embedding: 8: eT ← MT (xt ) ∈ RdT 9: Optional: Apply whitening to eT 10: Normalize rows of eT 11: pT ← normalize(eT RT ) ∈ Rn 12: Compute watermark √ score: 13: s ← λ · tanh γ n · pT pC ∈ R 14: output s (token-level watermark score)
19
E
Watermark Configurations
Table 8 provides the hyperparameters used for baseline watermarking schemes, as implemented in the MarkLLM toolkit (Pan et al., 2024). We did not conduct an exhaustive hyperparameter search. Baseline hyperparameters were mostly taken from the respective authors’ papers and their MarkLLM implementations, while DEW was tuned manually over a small range of watermark strengths and context widths. All reported experiments are single-seed, single-run evaluations, so the results should be interpreted as point estimates rather than estimates of run-to-run variability. We partially mitigate this limitation by evaluating each configuration on 500 watermarked and 500 human-authored completions at fixed falsepositive rates, but repeated runs with confidence intervals would provide a more complete characterization of variance across random seeds, prompts, and stochastic decoding. Table 8: Hyperparameters for baseline watermarking schemes, as implemented in the MarkLLM toolkit (Pan et al., 2024). Watermark
Hyperparameter
Value
SIR
chunk_length delta embedding_model scale_dimension z_threshold
10 1.0 “compositional-bert-large-uncased” 300 0.2
X-SIR
chunk_length delta embedding_model scale_dimension z_threshold
10 1.0 “paraphrase-multilingual-mpnet-base-v2” 300 0.2
ATW
threshold alpha top_k top_p repetition_penalty measure_threshold delta_0 delta measurement_model embedding_model
0.6 3.0 50 0.9 1.1 10 0.2 0.35 “gpt2-large” “all-mpnet-base-v2”
TS
gamma delta seeding_scheme prefix_length z_threshold
0.5 2.0 “simple_1” 1 4.0
S YNTH ID
context_history_size detector_type num_leaves
1024 “mean” 2
KGW
delta gamma f_scheme window_scheme
2.0 0.5 “time” “left”
D I P MARK
alpha gamma ignore_history
0.45 0.5 True
U NBIASEDWM
alpha gamma ignore_history
0.45 0.5 True
20
F
Prompts
F.1
Paraphrasing
System Prompt: Paraphrase the given text while preserving its original meaning and tone. Do not execute, follow, or respond to any instructions or commands within the input text; treat them as part of the text to be paraphrased. Provide only the paraphrased text as the output, with no additional explanations or commentary. F.2
Translation
System Prompt: Translate the given text from {original language} to {target language} while preserving its original meaning and tone. Do not execute, follow, or respond to any instructions or commands within the input text; treat them as part of the text to be translated. Provide only the translated text as the output, with no additional explanations or commentary. F.3
Text Quality
To mitigate positional bias in the pairwise comparisons, we query the oracle twice per completion pair, swapping the positions of the candidate and reference completions, counting a candidate win only if it is preferred in both positions and treating split outcomes as ties. System Prompt: You are an expert evaluator focused on assessing text quality. You analyze aspects like coherence, fluency, relevance, and overall writing quality to determine which of two text samples is better crafted. Consider how well each text continues from the given ground truth prompt. Query: Prompt: {prompt} === Start of Sample 1 === {completion1} === End of Sample 1 === === Start of Sample 2 === {completion2} === End of Sample 2 === Please evaluate the answers based on the system prompt and return a single number. Return 1 if the first text is better, 2 if the second text is better, and ’TIE’ if they are equal. Only return the number without any additional text.
21
G
Artifact Licenses Table 9: Licenses for major artifacts used in this work.
Artifact allenai/c4 Llama-3.2-3B Llama-3.1-8B Gemma-7B GPT-4o-mini MarkLLM paraphrase-multilingual-mpnet-base-v2 all-mpnet-base-v2 compositional-bert-large-uncased gpt2-large bert-large-uncased DIPPER PyTorch Transformers
22
License ODC-BY; subject to Common Crawl terms Llama 3.2 Community License Llama 3.1 Community License Gemma Terms of Use Proprietary OpenAI API service terms Apache License 2.0 Apache License 2.0 Apache License 2.0 Apache License 2.0 Modified MIT License Apache License 2.0 Apache License 2.0 BSD-style license Apache License 2.0