Learning to Evade: Adaptive Attacks on Audio Watermarking Weikang Ding∗1 , Hanqing Guo2 , Rui Duan1 , Guangjing Wang3 , Yuanda Wang4 , Mingzhe Chen5 , Qiben Yan4 1
University of Missouri-Kansas City, USA 2 University of Hawaii at Manoa, USA 3 University of South Florida, USA 4 Michigan State University, USA 5 University of Miami, USA
arXiv:2606.22310v1 [cs.SD] 21 Jun 2026
[email protected], [email protected]
Abstract Advances in generative audio have intensified copyright concerns, making audio watermarking increasingly important for asserting ownership. However, existing audio watermarking methods are vulnerable to adversarial attacks. We find that watermark decoder message probabilities follow normal distributions, a property exploited by defenses to detect manipulations. This paper introduces an adaptive audio watermark attack method (AWM) designed to bypass existing defense strategies. AWM uses a two-stage optimization: the first stage ensures attack success, while the second improves audio quality. To evade detection, it estimates normal distribution parameters from limited samples of the target audio, and then adaptively steers decoded probabilities back into the estimated range. Evaluated on two watermarking methods across three voice datasets, AWM achieves high success while bypassing state-of-the-art detectors: detection rates are below 10% for replacement and creation, and 0% for removal. Index Terms: Audio watermark, outlier detection, adaptive attack
1. Introduction In recent years, the rapid growth of social networking platforms has encouraged many users to publicly share their audio content, including original works such as audiobooks and selfproduced music. These audio contents might bring them income. However, many unauthorized users copy creative works, modify them, and re-upload them to mainstream platforms for profit, which significantly discourages original audio creators. Besides, voice cloning attacks can illegally synthesize the target’s voice for malicious purposes, potentially resulting in severe consequences such as financial losses and reputation damage [1]. To address these issues, deep-learning-based watermarking has been proposed [2]. It embeds a noise-tolerant signal into the target audio, which remains imperceptible to human hearing while being detectable by specialized AI models. Accordingly, most approaches adopt an encoder–decoder architecture to embed and recover the signal. To improve robustness, training pipelines incorporate training data from distortion attack models, e.g., re-recording [3], voice cloning [4, 5], and lossy codec compression [6], such that the watermark could survive common real-world transformations. However, the robustness of deep-learning-based audio watermarking methods against adversarial attacks remains a crit* Research was performed at Michigan State University.
a. Watermark Replacement
Watermarked Audio
Different Watermark
Attacker
b. Watermark Creation
Clean Audio
Perturbation
Deploy
Perturbation
New Watermark
Attacker
c. Watermark Removal
Defender
Watermark Attack Detection Attacked?
Perturbation
YES Watermarked Audio
Attacker
Without Watermark
No
Figure 1: Overview of the watermark attack (left) and the watermark attack detection process used to detect whether the audio has been tampered with (right). ical concern [7]. Recent studies [8, 9] show that attackers can remove or forge watermarks by embedding carefully crafted adversarial perturbations. These attacks operate by adding and optimizing a perturbation signal so that the watermark decoder is misled into producing incorrect outputs. Yet, existing attack strategies face the following challenges: C1: How to design an attack method to balance audio quality and attack effectiveness? The encoder ensures the watermark is imperceptible, while the decoder recovers the watermark under various distortions. In practice, the decoder is more publicly accessible and therefore becomes the primary source of model knowledge in most attack scenarios. However, deceiving the decoder may introduce certain noise artifacts. An overly aggressive attack can significantly degrade audio quality, while prioritizing perceptual quality may compromise attack success. As a result, it is essential to strike a balance that ensures sufficient attack effectiveness while preserving audio quality. C2: How to design an attack to bypass the detection strategy? For a given audio input, the decoder outputs both a binary message and the associated bit-wise probabilities. These probabilities tend to follow a normal distribution, and defenders can leverage this property to detect whether an audio sample has been tampered with. Consequently, after successfully altering the binary watermark message bits, attackers must further optimize the audio to ensure that the decoded message probabilities fall within the range classified by the defender as non-outliers. C3: How to select the suitable audio samples to estimate the decoded message probability distribution? To estimate the
distribution parameters required for a successful and stealthy attack, they must instead rely on querying and analyzing the decoder’s outputs. Because different input samples induce different probability patterns, the parameters estimated by the attacker can diverge significantly from the defender’s true parameters, and this mismatch can ultimately lead to attack failure. Therefore, it necessitates designing an effective estimation strategy to improve the attack success rate. In this paper, we propose AWM, an Adaptive audio WaterMark attack method, which is capable of bypassing the defender’s detection strategy. Figure 1 illustrates the application scenarios. The attacker obtains the target audio and adds an adversarial perturbation to generate the perturbed audio. The defender then receives the perturbed audio and uses the watermark decoder to extract the message probabilities. A predefined set of distribution parameters is employed to detect outliers. If any outliers are identified, the audio is classified as “attacked”; otherwise, it is considered “clean”. We propose three schemes to address the challenges above. To address C1, we design a two-step attack framework. The first step focuses on maximizing attack effectiveness while evading the defender’s detection strategy. The second step aims to improve audio quality. Inspired by [10], we introduce a threshold-based post-processing step that refines the perturbation to enhance audio quality, while constraining the decoded message probabilities to remain within the expected normal range. To address C2, we develop an adaptive optimization strategy that explicitly steers the decoded message probabilities toward the estimated normal distribution range. The key idea is to prioritize optimization on probabilities that fall outside this range: if a perturbed bit probability already lies within the estimated normal region, its optimization weight is reduced. This yields a dynamic optimization process that concentrates on the most “at-risk” binary message bits. To address C3, we leverage the observation that data with similar feature distributions tend to produce similar decoded message probability distributions [11, 12]. Accordingly, we estimate the required probability distribution parameters by collecting a limited set of audio samples whose features closely resemble those of the target audio. In this paper, we make the following contributions: • We empirically demonstrate that the decoded message probabilities output by the watermark decoder tend to follow normal distributions. This statistical property can be leveraged by defenders to design detection strategies based on outlier detection. • We propose AWM, an adaptive audio watermark attack method that supports three attack types. Through a twostep optimization framework, AWM can bypass distributionbased detection strategies and preserve audio quality. The adaptive optimization strategy focuses updates on the message bits with the most uncertain probabilities, further improving attack effectiveness. • We evaluate AWM on three speech datasets and two state-of-the-art watermarking models. Compared to baseline methods, AWM achieves superior performance in both Attack Success Rate (ASR) and Detection Success Rate (DSR). Moreover, even after applying five no-box perturbations, AWM consistently maintains a high ASR, with most scores approaching or reaching 100%. The source code and demos are available on https:// adaptiveaudiowmattack.github.io/
2. Related Work Deep Learning-Based Audio Watermarking. Unlike traditional schemes [13], which rely on predefined transformations, deep-learning-based schemes can learn complex feature representations and optimize watermarking dynamically [4, 5, 14, 15, 16]. These schemes follow the architecture of the EncoderDistortion-Decoder. The encoder embeds the watermark into the audio and generates the watermarked audio, the decoder receives the watermarked audio and extracts the corresponding watermark, and the distortion simulates a variety of potential attack scenarios. This paper implements the defense mechanism and attack strategy by utilizing this scheme. Audio Watermark Attack. Audio watermark attacks based on adversarial perturbations can be categorized as no-box, blackbox, or white-box, depending on the attacker’s knowledge of the watermarking model. In no-box perturbations, the attacker relies on generic audio processing techniques (e.g., compression [17], voice conversion [18, 19], and text-to-speech [20, 21]). In black-box perturbations, the attacker can query the watermarking detector but does not have access to its internal architecture or parameters [22, 23] . In white-box perturbations, the attacker has full access to the detector, including its architecture and parameters, and introduces a perturbation to perform the gradient-based attack [24, 25]. While some existing studies [9, 26, 8] are either easily detected or generate the perturbed audio with poor quality. In this paper, we focus on balancing the attack effectiveness and audio quality. Audio Watermark Attack Detection. Adversarial examples and watermarks both achieve the desired goal by adding imperceptible noise. Different from adversarial examples, adding watermarks to objects has little impact on the performance of the model inference (such as audio classification models [27] and speech recognition models [28]). Therefore, some outlier detection methods based on the time series [29, 30] are ineffective for identifying whether an audio sample has been tampered with. Recent studies have explored detection methods for generated images [31, 32], watermark images [12, 26], and audio deepfake [33, 34]. However, to the best of our knowledge, these approaches cannot be directly applicable to detecting audio watermark attacks.
3. Background 3.1. Preliminary Adding perturbations to the audio is a common strategy for attacks against watermarking systems. The core idea is to either destroy the original watermark or forge a new one by introducing perturbations that deceive the watermark decoder. Audio Watermark Decoder. The audio watermark decoder Dec(·) takes the encoded audio as input and outputs the extracted message. The extracted message can be represented in two forms: a binary message and message probabilities. The binary message is a direct representation of the decoded output as a sequence of 0s and 1s, denoted as m ∈ {0, 1}N . The message probabilities provide the decoder’s confidence values for each bit, indicating the likelihood of the bit being 1 or 0. A predefined threshold θ is used to convert probabilities into binary message values: if a probability exceeds the threshold, the binary message bit is decoded as 1; otherwise, it is decoded as 0. The form of message probabilities can be written as: p = Dec(s),
(1)
where s is clean or watermarked audio, Dec(·) is the watermark
δcreation = arg min∥Dec(sc + δ) − pt )∥. δ
δremoval = arg min∥δ∥ s.t. Detector(sw + δ) = None. (4) δ
3.2. Message Probability Distribution Benign Data Distribution. The watermark decoder outputs the message probabilities for each binary message bit. A bit is decoded as 1 if its probability exceeds a predefined threshold; otherwise, it is decoded as 0. For both clean and watermarked audio, we observe two distinct distributions, each following a normal distribution pattern. Clean audio exhibits a unimodal normal distribution, with the peak density centered near the predefined threshold. Specifically, in the Timbre, message probabilities below 0 are decoded as 0, while message probabilities above 0 are decoded as 1. As shown in Figure 2(a), where the threshold is set to 0, the mean µ of the distribution is also close to 0. In contrast, the watermarked audio follows a bimodal, approximately normal distribution (Figure 2(b)), with two peaks corresponding to the decoded binary message values 0 and 1. Audio Watermark Attack Distribution. For existing watermark attack methods [8], we observe that the distributions of message probabilities deviate significantly from the benign data distributions. Figure 3(a) illustrates the distribution of message probabilities under watermark removal attacks. Compared to Figure 2(a), the message distribution under attack differs in the range of message probabilities along the x-axis, even though it still exhibits a unimodal normal distribution. Figure 3(b) presents the distribution of watermark creation attacks. Here, most message probability values cluster around the threshold value of 0. In contrast to the clear bimodal normal distribution shown in Figure 2(b), the distribution resulting from the attack visually diverges from the benign data distribution. 3.3. Threat Model The attackers perform watermark attacks to either claim new copyright ownership or remove the original copyright. We as-
Density
Density
4
0.2
0.0
0.2
0.4
0
0.6
Message Probabilities
1.0
0.5
0.0
0.5
Message Probabilities
1.0
(a) Normal distribution with clean (b) Normal distribution with waaudio (Timbre). termarked audio (Timbre). 25
Distribution of Message Probabilities
Distribution of Message Probabilities 14 12 10 8 6 4 2 0
20 15 5 0
0.44 0.46 0.48 0.50 0.52 0.54 0.56 0.58
Message Probabilities
0.2
0.3
0.4
0.5
0.6
0.7
Message Probabilities
0.8
(c) Normal distribution with clean (d) Normal distribution with waaudio (AudioSeal). termarked audio (AudioSeal).
Figure 2: Distribution of benign message probabilities.
(3)
Watermark Removal. Watermark removal is an untargeted attack aimed at removing the embedded watermark. We introduce a watermark detector, Detector(·), which determines whether the audio contains a watermark. A perturbation δ is added to the watermarked audio sw to mislead the detector into classifying the perturbed audio as unwatermarked (i.e., clean). Formally, the goal of watermark removal can be expressed as:
6 2
0.4
10
Watermark Creation. Watermark creation involves adding a perturbation δ to a clean audio sc to generate a perturbed watermarked audio, which deceives the watermark decoder into recognizing it as containing a valid watermark. Formally, the goal of watermark creation is:
Distribution of Message Probabilities
8
Density
(2)
10
Distribution of Message Probabilities (Removal)
0.30 0.25 0.20 0.15 0.10 0.05 0.00
Distribution of Message Probabilities (Creation)
1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0
Density
δ
Distribution of Message Probabilities
Density
δreplacement = arg min∥Dec(sw + δ) − pt ∥.
4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0
Density
decoder. Let sc denote clean audio and sw denote watermarked audio. The watermark decoder Dec(·) extracts the clean message probabilities pc from sc and the watermarked message probabilities pw from sw , respectively. The attacker specifies a target message probability, denoted as pt . A perturbation δ is introduced to perform the attack on the audio. Watermark Replacement. In watermark replacement, a perturbation δ is added to the watermarked audio sw , deceiving the decoder into misclassifying the embedded watermark as a different one. Formally, the goal of watermark replacement is:
5
0
5
10
Message Probabilities
15
0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00
Message Probabilities
(a) Message distribution under (b) Message distribution under watermark removal attack. watermark creation attack.
Figure 3: Distribution of message probabilities under attacks (Timbre). We use the AudioMarkBench to perform the attacks.
sume that: 1) attackers have no access to the data used by defenders to fit the distribution, nor to the audio dataset used to train the watermarking model; 2) they have no access to the ground-truth watermark message embedded in the target audio; but they can get the message probabilities from the watermark decoder, allowing them to perform gradient-based attacks; 3) they do not possess the complete watermarking model, but they can embed watermarks into a small set of clean audio samples; 4) they are aware that the decoded message probabilities output by the watermark decoder follow a normal distribution, but they do not know the corresponding mean and standard deviation. The defenders are the attack detectors, who identify whether audio has been tampered with. We assume that: 1) defenders have access to a large number of ground-truth audio samples, which are used to fit a normal distribution and estimate the mean and variance via maximum likelihood estimation; 2) the watermarking model is publicly available through an online platform for commercial use, with a limit on the number of watermarked audio samples that each user can generate per day.
4. Methodology In this section, we first introduce a detection method based on outlier detection to identify if the given audio sample has been attacked. Second, we design our adaptive attack, which aims to achieve a successful attack while preserving perceptual quality. Meanwhile, we describe the adaptive optimization process for the three attack types.
a. Distribution-Based Detection Design Deploy
a. Distribution Estimation for Message Probabilities in Watermark Replacement and Creation
Watermark Decoder
GT Audio
Deploy
Estimate
Output
Message Probabilities
Deploy
Output
Detect
Attacked?
YES Clean Audio
No
Perturbed Audio
Watermark Decoder
Message Probabilities
Outlier Detection
Figure 4: Audio watermark attack detection. The defender uses ground truth (GT) audio (watermarked and clean) to estimate the reference distribution (top) and applies outlier detection with respect to this distribution to determine whether an audio sample has been attacked (bottom). 4.1. Audio Watermark Attack Detection As demonstrated in Section 3.2, the distribution of message probabilities in perturbed audio exhibits significant deviations from that of benign audio, thereby providing the foundation for our proposed detection mechanism to effectively distinguish between the two. Figure 4 illustrates the audio watermark attack detection method, which introduces a two-step approach. Distribution-Based Detection Design. The key intuition is that message probabilities from clean and watermarked audio follow different statistical patterns. Building on this, we propose that the defenders can collect ground-truth audio, extract message probabilities with the watermark decoder, and apply maximum likelihood estimation to model their distributions. As shown in Figure 2, this process yields three normal distributions: one derived from message probabilities of clean audio and two from those of watermarked audio. Specifically, given n audio samples, each producing a probability vector of size 1 × N message probabilities, the complete set of message probabilities is denoted as: p11 p12 · · · p1N p21 p22 · · · p2N n×N . p= . .. .. ∈ R .. .. . . . pn1 pn2 · · · pnN The mean µ and standard deviation σ of the normal distribution estimated via maximum likelihood are: n
µ=
N
1 XX pij , nN i=1 j=1
n
σ2 =
N
1 XX (pij − µ)2 . (5) nN i=1 j=1
Finally, the defender obtains the corresponding means µ and standard deviations σ, which are used to detect outliers in suspicious audio and determine whether it has been attacked. Hypothesis Testing for Outlier Detection. We formalize detection as an outlier detection problem under a statistical hypothesis testing framework. Given a potentially perturbed audio sample, the defender uses the watermark decoder to extract its message probability set {pi }, where each element is assumed to follow a normal distribution with mean µ and standard deviation σ. We then test the null hypothesis H0 : pi ∼ N (µ, σ 2 ) against the alternative H1 : pi ̸∼ N (µ, σ 2 ). If H0 is rejected, the audio sample is classified as attacked. To test the hypothesis, we first compute the z-score for each extracted message probability: zi =
pi − µ . σ
Watermarked Audio
Output Watermark Decoder
(6)
Output
Watermark Decoder
Estimate
Message Probabilities
Estimate
Message Probabilities
b. Distribution Estimation for Message Probabilities in Watermark Removal
b. Hypothesis Testing for Outlier Detection Deploy
Watermark Model
Clean Audio
Predicted Distribution
Deploy
Generate
Obtain
Estimated Distribution Obtain
Adaptive Attack
Estimated Distribution
Figure 5: The distribution estimation by the attacker. (a) Watermark replacement and creation: The attacker uses a small set of clean audio samples to generate watermarked audio samples, which are then used to estimate the distribution. (b) Watermark removal: The attacker directly uses the clean audio samples to estimate the distribution. Under the null hypothesis, zi follows the standard normal distribution, and the corresponding two-tailed p-value is given by Pi = 2 (1 − Φ(|zi |)), where Φ denotes the cumulative distribution function of the standard normal distribution. The decision rule is to reject H0 if Pi < α, with α as the predefined significance level. An audio sample is deemed being “attacked” if any extracted message probability pi rejects H0 . 4.2. Adaptive Attack Design We propose an adaptive attack designed to bypass messageprobability-distribution–based detection. First, the attacker prepares for the attack by estimating the defender’s distribution. Second, the attacker performs the watermark attack, which contains two stages: (1) modify the original audio to bypass the defense and achieve a successful attack, and (2) improve the quality of the perturbed audio, while maintaining that the decoded message probabilities remain within an acceptable range. 4.2.1. Attack Preparation: Estimate Defender’s Distribution The adversary’s goal is to estimate the mean µest and the standard deviation σest of the decoded message probabilities. These parameters are categorized into two groups: watermark replacement and creation, and watermark removal. The process is illustrated in Figure 5. Parameter Estimation for Watermark Replacement and Creation. The attackers first select several clean audio samples sc from a small dataset, which are used to query the watermark model Enc(·) and generate new watermarked audio samples sw . The watermark decoder Dec(·) is deployed to extract message probabilities, which are subsequently used to estimate the parameters of the normal distribution. Since the distribution of watermarked message probabilities is bimodal (as shown in Figure 2(b) and 2(d)), the attacker can obtain two distributions. The estimated mean and standard deviation are as follows: 0 µ0est , σest = T 0 (Dec(Enc(sc ))), 1 µ1est , σest = T 1 (Dec(Enc(sc ))),
(7)
0 where µ0est and σest represent the estimated mean and standard deviation of decoded message probabilities predicted as 0, 1 and µ1est and σest correspond to those predicted as 1. T (·) denotes the method used to estimate the parameters of the normal distribution, which depends on the attacker’s prior knowledge and the amount of available data. Common choices include Bayesian inference [35] and maximum likelihood estimation. We use T 0 (·) and T 1 (·) to denote the estimation procedures for the distributions corresponding to bits 0 and 1, respectively.
Watermark Attack (AWM)
Optimization (AWM +opt)
Update
Original Audio
Message Loss No Yes
Successful Perturbed Audio
AWM perturbation Update
Deploy
Perturbation Perturbed Audio
Watermark Decoder
Original Audio
Signal Loss Spec Loss
Perturbed Audio (+opt)
Yes
Attack goal achieved & Attack not detected
Message Probabilities
Watermark Decoder Output
No
Output
Estimated Detector
Deploy
Attack goal achieved & Attack not detected
Successful Perturbed Audio (+opt)
Extended Message Estimated Detector Probabilities
Figure 6: The design of AWM generator. The audio watermark attack step (left) ensures the success of the watermark attack, while the audio quality optimization step (right) focuses on improving audio quality. Parameter Estimation for Watermark Removal. The attackers estimate the distribution directly using clean audio samples sc . They use the watermark decoder to output message probabilities, which are used for distribution estimation. Since the estimated distribution is unimodal (as shown in Figure 2(a) and 2(c)), the parameters are defined as follows: c µcest , σest = T c (Dec(sc )),
(8)
c where µcest and σest are the estimated mean and standard deviation of the clean message probabilities, and T c represents the estimation approach applied to these values.
4.2.2. Audio Watermark Attack (AWM)
n
Lsignal =
1X |satt − s|. n i=1
(10)
The Mel-Spectrogram loss Lmel maintains the audio quality at the Mel-Spectrogram level: Lmel = ∥M el(satt ) − M el(s)∥22 .
(11)
The total loss in the attack step is:
After obtaining the estimated distribution, the attacker then adds a small adversarial perturbation to the original audio s, aiming to deceive the decoder into outputting incorrect binary messages while bypassing the detection strategy. Figure 6-left illustrates the watermark attack step. The original audio is clean audio in watermark creation attack, and it can be watermarked audio in watermark replacement or watermark removal attack. First, the perturbation is initialized by a fraction of the original audio signal. Next, the attacker adds the perturbation to original audio and obtains a perturbed audio, which is passed through the watermark decoder to obtain message probabilities and subsequently queried by the estimated detector. If the attack goal is achieved and the attack is not detected, the attacker obtains the successful perturbed audio. Otherwise, the attacker further optimizes the perturbation by message loss. The message loss Lmsg modifies the perturbed message probabilities to match the target message probabilities pt : Lmsg = ∥Dec(satt ) − pt )∥22 .
Besides the message loss, we also formulate the signal loss and mel loss to minimize the quality degradation from the attack. Specifically, the signal loss controls the audio quality at the signal level:
(9)
Let satt denote the perturbed audio, this loss enforces the decoded message to closely match the target message. Different from the prior attacks, our loss optimization step follows a strict bit-to-bit optimization design (detailed in Algorithm 1). This algorithm uses the estimated detector knowledge to ensure that the perturbed audio exhibits a distribution similar to that of benign audio, with high confidence. Importantly, our algorithm constrains the message probabilities within the normal value range. Through iterative updates and perturbation optimization, the perturbed message probabilities are gradually adjusted to fall within a strict range, ensuring that they are classified as non-outliers and yielding the final adversarial perturbation. Our results show that constraining the message probabilities of the perturbed audio to the interval [µest − σest , µest + σest ] improves the attack success rate.
L = λ1 Lsignal + λ2 Lmel + λ3 Lmsg + λ4 Lother ,
(12)
where the Lother depends on the specific watermarking method. For example, AudioSeal [5] includes a localization loss used for indicating the probability of the audio being watermarked. In the AWM attack process, the parameter λmsg is assigned a relatively high value. 4.2.3. Audio Quality Optimization (AWM +opt) Since audio watermark attacks prioritize message loss optimization, audio quality may be adversely impacted. This step aims to improve audio quality while maintaining a successful attack. To achieve this, we make three adaptations. Figure 6-right illustrates the three steps. First, the attacker takes the perturbation obtained in the earlier stage as the initial perturbation and uses the watermark decoder to extract the corresponding message probabilities. These probabilities are then fed into the estimated distribution to expand the allowable range, which we suggest setting to [µest − 2σest , µest + 2σest ]. This step is crucial for trading off audio quality against attack success rate. In the original AWM setting, the attacker enforces a strict constraint to guarantee attack success, but this rigidity limits the flexibility of the perturbation and makes it difficult to preserve audio quality. In this design, we relax the constraint by expanding the permissible range for message probability optimization, thereby providing more space for perturbation adjustment and achieving a more balanced attack that accounts for both perceptual quality and attack success rate. Second, in addition to extending the acceptable range, we also update the attack objective by enforcing optimization to proceed for a fixed number of epochs. This ensures that the perturbation is fully optimized rather than stopping at a boundary case.
Algorithm 1 Adaptive Attack in Watermark Replacement Input: Watermarked audio sw , target message probabilities pt , scale factor r, supremum and infimum of thresholds for de0 0 1 1 coded message 0 and 1 τsup and τinf i , τsup and τinf i , list of different message probabilities msgdif f Output: perturbed watermarked audio sˆw 1: δ = sw × r, pw = Dec(sw ), sˆw = sw + δ 2: for index = 1, 2, · · · , (len(pt ) − 1) do 3: if index not in msgdif f then 4: pt [index] = pw [index] 5: for i = 1, 2, · · · , Iter do 6: δ = Attack(sw , sˆw , Dec(sˆw ), pt , δ) 7: sˆw = sw + δ 8: pˆw = Dec(sˆw ) 9: if acc == 1 then 10: for index in msgdif f do 1 1 11: if τinf w [index] < τsup then i < pˆ 12: pt [index] = pˆw [index] 13: Remove msgdif f [index] 0 0 14: if τinf w [index] < τsup then i < pˆ 15: pt [index] = pˆw [index] 16: Remove msgdif f [index] 17: if meet the estimated detection Detection(pˆw ) then 18: return sˆw 19: return Failed
Third, we modify the optimization loss by replacing the Mel-spectrogram loss with a standard spectrogram loss and applying a softmax function to the spectrogram. Inspired by AudioSeal [5], this softmax adaptation better preserves loudness and perceptual similarity between the two audio signals. The resulting softmax-based spectrogram loss, denoted as Lspec , is defined as: n
Lspec =
1X |Sof tmax(Satt ) − Sof tmax(Ss )|, n i=1
(13)
where Satt and Ss are the spectrograms of the attacked audio and original audio. In the optimization (+opt) process, the parameters λ1 and λ2 are assigned a relatively high value. 4.2.4. Adaptive Attack for Watermark Replacement, Creation, and Removal During the watermark attack, we prioritize refining the decoded message probabilities that fall outside the estimated normal range. Algorithm 1 outlines the adaptive attack process used in watermark replacement. For example, suppose the original watermark bits are “101100”, which are modified to “111000”. We define a list, msgdif f , which contains the indices of binary message bits differing between the original watermarked audio sw and the perturbed watermarked audio sˆw 1 . In this case, msgdif f = [1, 3]. For the indices not included in msgdif f , we assign the original watermark message probabilities pw to the corresponding target watermark message probabilities pt . That is, for indices [0, 2, 4, 5], the pt is equal to the pw . This optimization has two advantages: (1) It directs the gradient to focus more on the indices where the binary message bits 1 This notation is defined for the watermark replacement scenario. The general representation of perturbed audio is denoted as satt , which refers either to perturbed clean audio (used for creation) or to perturbed watermarked audio (used for replacement or removal).
require modification. In watermark replacement, only the differing bits need to be changed, so bits that already align with the target message do not require further optimization. (2) It ensures that the message probabilities in the non-change bits remain the same, and within the acceptable distribution. In certain audio watermarking methods, message probabilities do not directly map to binary message values. In Figure 2(d), probabilities corresponding to binary 1 lie in the range of 0.7–0.8, and those for binary 0 fall between 0.2–0.3. Therefore, it is not appropriate to optimize message probabilities directly to 1 or 0. Instead, they should be adjusted to fall within the range of 0.7–0.8 for a binary message bit of 1, and 0.2–0.3 for a bit of 0. Then, we optimize the perturbation δ by Function Attack(·) (Line 6) to generate the perturbed watermarked audio sˆw . The Attack(·) calculates the loss in Equation (12) and use the gradient to update the perturbation. The initial perturbation is scaled based on the original watermarked audio sw to ensure that the decoded message probabilities of the perturbed audio sˆw closely resemble those of the original. In the attack justification step, we define the supremum and infimum thresholds for the watermark decoder outputs corresponding to binary 0 0 message bits. Specifically, τsup and τinf i represent the thresh1 1 olds for bit 0, while τsup and τinf i correspond to bit 1: 0 0 0 0 0 (µ0est − σest ) ≤ τinf i < τsup ≤ (µest + σest ), 1 1 1 1 1 (µ1est − σest ) ≤ τinf i < τsup ≤ (µest + σest ).
(14)
Once the message probability of the perturbed watermarked audio at a given index falls within the specified threshold range, it is assigned to the target message probability pt , and the index is removed from the list msgdif f . Optimization then proceeds with the remaining message probabilities in the list. To ensure that all perturbed message probabilities remain within the estimated normal range, the attacker simulates the defender’s role by performing outlier detection. For watermark creation attack, the adaptive attack process is the same as that of watermark replacement. The difference is that, in watermark creation, msgdif f includes all message indices, which is equal to the full binary message length. Watermark removal attack is an untargeted attack, it is not necessary to achieve an accuracy of exactly 0, any value below 1 is sufficient. Using a lower accuracy threshold requires more iteration steps to optimize the perturbation δ. We recommend using an accuracy threshold around 0.5.
5. Evaluation 5.1. Experimental Setup Datasets. We use three public datasets for our experiments. The first dataset is the LibriSpeech [36]. We select the small-sized subset, which has 6.3G audios, and covers 100.6 hours of audio data spoken by 251 speakers. The second dataset is obtained from AudioMarkData [8], which is built based on the Common Voice dataset [37]. It contains 20,000 audio samples, each with a duration of 5 seconds. The third dataset is GigaSpeech [38], which includes audio from audiobooks, podcasts, and YouTube. We use the XS subset, which contains a total of 10 hours of audio samples. Audio Watermark Methods. We select two state-of-the-art audio watermarking methods: Timbre [4] and AudioSeal [5]. We fix the binary message length to 16 bits for all experiments. Evaluation Metrics. First, we introduce the Detection Success Rate (DSR), which measures the ability to identify outliers
AudioSeal
Watermark Replacement
Timbre
AudioSeal
Watermark Creation
Timbre
AudioSeal
Watermark Removal
Timbre
DSR (%) 97.71 93.94 97.96 98.03 98.47 86.41 3.44 5.34 100.00 100.00 100.00 100.00 100.00 1.53 1.91 100.00 100.00 100.00 100.00 100.00 100.00 0.76 1.91 100.00 100.00 100.00 100.00 100.00 100.00 0.00 0.00 92.75 91.41 91.92 89.92 83.92 83.92 0.00 0.00 100.00 100.00 100.00 84.13 100.00 100.00 0.00 0.00
Librispeech FAR (%) 4.20 2.27 4.08 1.97 4.20 3.88 4.20 4.20 2.67 0.00 2.82 0.00 1.28 2.67 2.67 4.20 4.90 4.50 5.22 5.45 1.28 4.20 4.20 2.67 2.06 2.31 0.00 3.70 2.60 2.67 2.67 5.73 5.86 5.77 5.43 5.88 5.88 5.73 5.73 5.73 5.73 5.73 5.56 5.73 5.73 5.73 5.73
F1 (%) 96.79 95.75 96.97 98.03 97.08 90.82 6.40 9.75 98.68 100.00 98.61 100.00 99.36 2.93 3.64 97.94 97.61 97.79 97.46 97.35 99.36 1.45 3.60 98.68 99.00 98.86 100.00 98.18 98.71 0.00 0.00 93.46 93.62 93.03 92.00 88.25 88.25 0.00 0.00 97.27 97.22 97.22 81.54 97.22 97.22 0.00 0.00
DSR (%) 100.00 93.98 100.00 100.00 100.00 96.43 8.50 8.00 100.00 100.00 100.00 100.00 100.00 100.00 7.00 7.50 100.00 100.00 100.00 100.00 100.00 100.00 0.50 2.00 100.00 100.00 100.00 100.00 100.00 100.00 0.00 0.00 100.00 100.00 100.00 100.00 100.00 100.00 0.00 0.00 100.00 100.00 100.00 74.83 100.00 100.00 0.00 0.00
25
20
20
15
DSR (%) 100.00 96.05 99.61 98.11 96.93 68.75 11.00 17.67 100.00 100.00 100.00 100.00 6.67 8.33 100.00 100.00 100.00 100.00 100.00 100.00 0.33 13.00 100.00 100.00 100.00 100.00 100.00 100.00 0.00 0.00 99.67 99.64 99.63 99.63 98.87 98.17 0.00 0.00 100.00 100.00 100.00 85.04 98.67 100.00 0.00 0.00
Gigaspeech FAR (%) 6.00 8.45 5.85 7.14 3.40 4.76 6.00 6.00 8.33 0.00 7.64 4.00 8.33 8.33 6.00 9.00 6.76 14.89 11.76 4.00 6.00 6.00 8.33 6.09 8.25 0.00 4.55 6.53 8.33 8.33 5.67 4.57 5.14 4.78 4.81 5.58 5.67 5.67 8.67 8.98 8.98 9.73 8.98 8.98 8.67 8.67
F1 (%) 96.94 94.19 97.53 98.85 96.93 79.49 18.81 28.58 96.00 100.00 96.49 98.11 11.58 14.29 97.09 95.96 97.18 93.33 94.74 98.11 0.63 21.85 96.00 97.30 96.59 100.00 97.87 97.25 0.00 0.00 97.03 97.98 97.86 97.95 97.44 96.95 0.00 0.00 95.83 96.50 96.50 88.09 95.80 96.50 0.00 0.00
15
10
10
5
5
0.001 0.003 0.005 0.007 0.009 0.011 Watermark
F1 (%) 97.32 89.66 97.45 94.87 96.51 95.58 15.00 14.10 96.85 95.65 97.81 100.00 100.00 98.55 12.33 13.17 97.32 98.61 98.05 100.00 97.67 98.55 0.94 3.68 96.85 97.41 96.90 85.71 98.18 97.22 0.00 0.00 98.28 98.38 98.65 98.55 98.63 98.33 0.00 0.00 97.09 97.09 97.09 82.20 97.09 97.09 0.00 0.00
0.3
0.4
0.5
0.6
Message Probabilities
0.7
0.8
0.3
0.4
0.5
0.6
0.7
Message Probabilities
0.8
(a) Message probabilities distribution (AudioSeal) 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0
Timbre Distribution (AudioMarkBench)
8 7 6 5 4 3 2 1 0
Timbre Distribution (Ours)
0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00
Message Probabilities
1.0
0.5
0.0
0.5
Message Probabilities
1.0
(b) Message probabilities distribution (Timbre)
Figure 8: Message probabilities distribution comparisons between AudioMarkBench and AWM for the watermark creation.