One Framework for All: Cross-Modal Membership Inference for Generative Models

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

One Framework for All: Cross-Modal Membership Inference for Generative Models

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cryptography, security, privacy, cybersecurity

One Framework for All: Cross-Modal Membership Inference for Generative Models

arXiv:2607.04339v1 [cs.LG] 5 Jul 2026

Dayong Ye, Tianqing Zhu∗ , Kun Gao, Junhao Liu, Yichuan Chen, Shuai Zhou, Hengzhu Liu, Bo Liu, and Wanlei Zhou

Abstract—Large generative models across text-to-text, text-toimage, and image-to-text modalities have been shown to pose significant privacy risks. One fundamental threat is membership inference attacks (MIA), which aim to determine whether a given data point was used in a model’s training set. Although prior work has investigated MIAs against these three classes of generative models, existing approaches treat them in isolation and are not cross-applicable, thereby limiting their real-world utility. To address this limitation, we present the first comprehensive study of a unified membership inference framework that applies across text-to-text, text-to-image, and image-to-text modalities. Our approach is grounded in a key modality-agnostic observation: the output distribution of a generative model can approximate its training data distribution. Leveraging this property, we model the distributions of model-generated outputs and auxiliary non-member samples in a shared embedding space, and perform membership inference via likelihood ratio testing. We conduct extensive experiments in a strict black-box setting under both partial-knowledge and zero-knowledge threat models, and evaluate membership inference against both fine-tuning and pretraining data. Experimental results demonstrate our approach’s superior performance in comparison to existing state-of-the-art methods, which are typically optimized for a single model class. Index Terms—Membership Inference, Generative Models, Cross Modality

I. I NTRODUCTION

ARGE generative models have become a core component of modern machine learning (ML) systems, enabling applications ranging from text generation and image synthesis to multi-modal assistants [1, 2]. While these models are typically pre-trained on massive public corpora, their deployment often relies on fine-tuning with private or proprietary datasets to adapt them to downstream tasks. This practice raises significant privacy concerns, among which the risk of membership inference attacks stands out as a fundamental threat [3]. Membership inference attacks (MIAs) aim to determine whether a given sample belongs to a model’s training set. Prior work has shown that large generative models inadvertently leak membership information through their outputs [4, 5, 6, 7, 8]. However, existing studies mainly focus on a single class of generative models, for example, considering only text-to-text, text-to-image, or image-to-text models in isolation. This fragmentation raises an important question: is there a unified membership inference approach that can effectively target all three classes of generative models? Addressing this question is crucial. In practice, modern AI Tianqing Zhu is the corresponding author.

systems integrate multiple generative components across different modalities [9, 10], and privacy risks cannot be fully understood by analyzing each modality in isolation. From a research perspective, a unified approach would clarify whether membership leakage arises from modality-specific artifacts or from modality-agnostic properties inherent to generative modeling. Motivated by these considerations, our work aims to develop such a unified membership inference approach. However, achieving this aim presents two challenges. First, the three classes of generative models, text-to-text, text-to-image, and image-to-text, rely on fundamentally different architectures and training paradigms. Designing a single attack strategy that applies across all these model classes requires techniques that are generalizable across heterogeneous systems. Second, the data modalities associated with these generative models differ substantially. Text data are sequential and discrete, whereas image data are spatial and continuous. A unified attack framework must effectively accommodate these modality-specific characteristics to remain effective across different generative settings. To address the first challenge, we leverage a fundamental property shared by generative models: their outputs are meaningful data points in the same domain as the training data. These generated samples can therefore be used to construct an output distribution that serves as an approximation of the training-data (i.e., member) distribution. By comparing a target sample against this approximate training-data distribution, we can infer its membership status. To address the second challenge, we adopt a set of modality-appropriate feature extractors to map data samples into a shared numerical embedding space. The resulting embeddings are represented as numerical vectors that are independent of the underlying data modality, enabling unified and consistent downstream computations across multimodal generative models. In summary, this work has three contributions. • We initiate a unified study of membership inference attacks against generative models across multiple modalities, with the goal of uncovering modality-agnostic privacy leakage inherent to generative modeling. • Our approach is model-independent and lightweight. It exploits the fundamental property that the output distribution of a generative model approximates its training data distribution, and infers the membership status of target samples via likelihood ratio testing, without requiring the training of additional models. • We conduct extensive experiments comparing our method with state-of-the-art baselines across a wide range of gener-

ative models and datasets, demonstrating consistently superior membership inference performance. Beyond inferring membership with respect to fine-tuning data, we further extend our evaluation to pre-training data, highlighting the generality and extensibility of our approach.

reverse process is then parameterized by a neural network M 𝜃 , which predicts the noise at each step conditioned not only on the noisy image 𝑥 𝑡 , but also on the prompt embedding 𝑧 𝑥 : 𝜖 𝑡 ≈ M 𝜃 (𝑥 𝑡 , 𝑡, 𝑧 𝑥 ). The training objective becomes:

II. P RELIMINARIES AND T HREAT M ODEL

From the user’s perspective, a conditional diffusion model takes a prompt 𝑥 as input and produces an image 𝑦 as output, i.e., 𝑦 = 𝐷 𝑀 (𝑥).

L𝐶𝐷 𝑀 = E[||𝜖 𝑡 − M 𝜃 (𝑥 𝑡 , 𝑡, 𝑧 𝑥 )|| 22 ].

Generative Models. We focus on three representative categories of generative models: large language models (LLMs), diffusion models (DMs), and vision-language models (VLMs), which correspond to the core application domains of text-totext, text-to-image, and image-to-text, respectively. Large language models [11] define a likelihood distribution over token sequences. Given a text record 𝑥 = [𝑡1 , . . . , 𝑡 | 𝑥 | ] of length |𝑥|, an LLM estimates the conditional probability of each token based on its preceding context: 𝑝 𝜃 (𝑡 𝑖 |𝑡 1 , ..., 𝑡 𝑖−1 ), where 𝜃 denotes the model parameters. By applying the chain rule, the joint probability of the entire sequence is factorized into the product of these conditional probabilities. Accordingly, LLMs are trained to minimize the negative loglikelihood of the training corpus:

(2)

Vision language models [13] integrate visual and textual inputs to perform multimodal reasoning. An input image 𝑥 𝑣 is first processed by a vision encoder to extract its visual feature representation 𝑒 𝑣 . This feature is then projected into the language model’s embedding space, yielding 𝑡 𝑣 . In parallel, a text prompt 𝑥 𝑞 is tokenized into a sequence of text embeddings 𝑡 𝑞 . The combined token sequence [𝑡 𝑣 , 𝑡 𝑞 ] is subsequently fed into an LLM, which generates the final text response 𝑦 𝑎 . The training dataset of a VLM is typically represented as 𝑁 𝐷 = (𝑥 𝑣𝑖 , 𝑥 𝑞𝑖 , 𝑦 𝑖𝑎 ) 𝑖=1 , where each triplet consists of an image, a text query, and a corresponding textual answer. The learning objective is to maximize the likelihood of the model generating 𝑦 𝑎 given the paired inputs 𝑥 𝑣 and 𝑥 𝑞 , i.e.,

𝑁 | 𝑥 ( 𝑗) |

1 ∑︁ ∑︁ log𝑝 𝜃 (𝑡 𝑖 |𝑡 1 , ..., 𝑡 𝑖−1 ), L 𝐿𝐿 𝑀 = − 𝑁 𝑗=1 𝑖=1

𝑁

L𝑉 𝐿 𝑀 = −

where 𝑁 is the number of training records. During generation, the model produces tokens one at a time in an autoregressive manner, thereby constructing coherent text sequences. From the user’s perspective, an LLM takes a text record 𝑥 (e.g., a prompt) as input and returns a corresponding text response 𝑦, denoted as 𝑦 = 𝐿𝐿 𝑀 (𝑥). Diffusion models [12] generate data by gradually reversing a noise-adding process. An image is corrupted by incrementally adding Gaussian noise, and the model is trained to recover the original image by denoising step by step. Formally, given an image 𝑥0 , the forward diffusion process adds noise over 𝑇 steps. At time step 𝑡, the noisy image 𝑥 𝑡 is given by: √︁ √︁ 𝑥𝑡 = 𝛼𝑡 𝑥0 + 1 − 𝛼𝑡 𝜖𝑡 , Î𝑡 where 𝛼𝑡 = 𝑖=1 𝛼𝑖 , each 𝛼𝑖 ∈ (0, 1) is a predefined parameter controlling the noise schedule, and 𝜖 𝑡 is Gaussian noise obtained via the reparameterization trick. The reverse diffusion process aims to invert this corruption. Starting from 𝑥ˆ𝑇 = 𝑥𝑇 , the model denoises 𝑥ˆ𝑡 to recover 𝑥ˆ𝑡 −1 . A neural network M 𝜃 is trained to predict the removed noise at each step. The training objective is therefore: √︁ √︁ L 𝐷 𝑀 = E[||𝜖 𝑡 − M 𝜃 ( 𝛼𝑡 𝑥0 + 1 − 𝛼𝑡 𝜖 𝑡 , 𝑡)|| 22 ]. (1)

1 ∑︁ log𝑝 𝜃 (𝑦 𝑖𝑎 |𝑥 𝑣𝑖 , 𝑥 𝑞𝑖 ). 𝑁 𝑖=1

From the user’s perspective, a VLM takes an image 𝑥 𝑣 together with a text prompt 𝑥 𝑞 as input, and outputs a text response 𝑦, formally expressed as 𝑦 = 𝑉 𝐿 𝑀 (𝑥 𝑣 , 𝑥 𝑞 ). Membership Inference. The goal of membership inference is to determine whether a given data sample 𝑥 was included in the training set 𝐷 train of a target model G. Formally, the membership inference task can be expressed as: A 𝐷train : (𝑥, G) → {0, 1}, where A 𝐷train outputs 1 if 𝑥 ∈ 𝐷 train and 0 otherwise. Unlike conventional models, membership inference against generative models also applies to the fine-tuning set 𝐷 tune . This distinction arises because many generative models, especially open-source ones, are first trained on large public datasets and then fine-tuned on smaller, often private, datasets tailored to specific downstream tasks. In this setting, the inference problem becomes: A 𝐷tune : (𝑥, G) → {0, 1}, where A 𝐷tune outputs 1 if 𝑥 ∈ 𝐷 tune and 0 otherwise. From a privacy perspective, inferring membership in the fine-tuning set poses a greater threat than inferring membership in the pre-training data. This is because pre-training datasets are typically collected from public sources with relatively lower privacy sensitivity, while fine-tuning datasets often consist of proprietary or sensitive information. For this reason, our work primarily focuses on membership inference against fine-tuning data 𝐷 tune . Nevertheless, as we will demonstrate experimentally, our proposed method is also applicable to inferring membership in pre-training data 𝐷 train .

Conditional diffusion models extend the standard diffusion framework to generate high-quality images guided by text prompts. Given a prompt 𝑥, the diffusion model conditions the denoising process on a learned representation of 𝑥, ensuring that the generated sample aligns semantically with the input prompt. Formally, let 𝑥 0 denote the original image and 𝑧 𝑥 denote the embedding of the prompt 𝑥. The forward process remains unchanged, namely Gaussian noise is gradually added to 𝑥0 over 𝑇 steps, producing noisy intermediate states 𝑥 𝑡 . The 2

Threat Model. We consider the most stringent black-box setting, where the adversary can only query the target generative model G without accessing or modifying its internal parameters, and can observe only the model’s generated outputs, not its logits or intermediate representations. Formally, given a target sample 𝑥 ∗ , the adversary queries G and obtains the corresponding response 𝑦 ∗ . Assuming that G has been finetuned on a dataset 𝐷 tune , the adversary’s goal is to determine whether 𝑥 ∗ ∈ 𝐷 tune . The adversary knows the task domain of 𝐷 tune but has no access to the dataset itself. Note that a training sample typically consists of an input–output pair (𝑥, 𝑦), where 𝑦 denotes the ground-truth output. For simplicity of expression, we slightly abuse notation and use 𝑥 to denote the corresponding input–output pair when the context is clear. For the capability of collecting external data, we distinguish between two adversarial scenarios: • Partial knowledge. The adversary has the ability to gather data from real-world sources. In this case, the adversary constructs a dataset 𝐷 real that is drawn from the same task domain as the fine-tuning dataset 𝐷 tune of G, but does not overlap with 𝐷 tune . • Zero knowledge. The adversary has no access to any external data beyond the target generative model itself. This setting represents the most restrictive scenario, where real-world data relevant to G are either inaccessible due to privacy restrictions, or simply do not exist in a usable form. We design tailored inference strategies for each scenario.

dataset 𝐷 real drawn from the same task domain but without overlapping with 𝐷 tune . In parallel, the adversary queries the target generative model G to construct a synthetic dataset 𝐷 syn . For example, suppose G is a text-to-image diffusion model fine-tuned on a private medical dataset of chest Xrays paired with diagnostic captions. The adversary, while not having access to 𝐷 tune , can still gather publicly available chest X-ray images from open datasets such as NIH ChestX-ray [14] to form 𝐷 real . At the same time, the adversary queries G with generic medical prompts (e.g., “X-ray of lungs with mild abnormalities”) to generate 𝐷 syn . Step 2: Embedding Extraction. For each data point in the (𝑖) (𝑖) real and synthetic datasets, i.e., 𝑥real ∈ 𝐷 real and 𝑥 syn ∈ 𝐷 syn , the adversary feeds the sample into an encoder 𝐸 to obtain its (𝑖) (𝑖) embedding representation, denoted as 𝐸 (𝑥real ) and 𝐸 (𝑥syn ), respectively. The choice of encoder 𝐸 is aligned with the architecture and training objective of the target generative model G. Specifically, the encoder should operate in the same semantic space that G relies on for generation, so that the extracted embeddings faithfully capture the model-relevant features. For example, if G is a text-to-image diffusion model (e.g., Stable Diffusion), we adopt the BLIP [15] encoder as 𝐸. This choice is motivated by the fact that BLIP is trained with joint vision–language objectives and is designed to align visual content with its corresponding textual semantics. Step 3: Compute Embedding Distribution. We characterize the embedding distribution by computing the empirical mean vector and covariance matrix of the extracted embeddings. 𝑚 , the Specifically, given 𝑚 sampled embeddings {𝐸 (𝑥 (𝑖) )}𝑖=1 empirical mean and covariance are defined as:

III. M EMBERSHIP I NFERENCE WITH PARTIAL K NOWLEDGE Step 1: Dataset collection Real data Collect

𝐷%&'

Prompts

𝐷!"#$

𝐷%&'

Extractor 𝐸

Target model 𝒢 Step 3: Compute embedding distribution using empirical mean and covariance 𝜇( = Σ( =

) * ∑ 𝐸 𝑥 (,) , * ,.) *

1 / + 𝐸 𝑥 (,) − 𝜇( 𝐸 𝑥 (,) − 𝜇( . 𝑚 ,.)

𝑚

Step 2: Embedding extraction 𝐷!"#$

, 𝐸 𝑥!"#$ , 𝐸 𝑥%&'

Extractor 𝐸

1 ∑︁ 𝐸 (𝑥 (𝑖) ); 𝑚 𝑖=1

Σ𝑥 =

1 ∑︁ [𝐸 (𝑥 (𝑖) ) − 𝜇 𝑥 ] [𝐸 (𝑥 (𝑖) ) − 𝜇 𝑥 ] ⊤ . 𝑚 𝑖=1

𝑚

* ,.) * ,.)

Applying this procedure to the embeddings extracted from the real and synthetic datasets yields their respective distributional representations, namely (𝜇 𝑥real , Σ 𝑥real ) for real data and (𝜇 𝑥syn , Σ 𝑥syn ) for synthetic data. We now justify why the empirical mean and covariance are sufficient to represent the embedding distribution. From a probabilistic perspective, let 𝑧 = 𝐸 (𝑥) denote the embedding of 𝑥. The distribution of 𝑧 is characterized by its true mean vector 𝜇 = E[𝑧] and true covariance matrix Σ = E[(𝑧 − 𝜇) (𝑧 − 𝜇) ⊤ ]. By the law of large numbers, we have

Step 4: Membership inference Target sample 𝑥 ∗

𝜇𝑥 =

𝐸(𝑥 ∗ )

Compute log-likelihood score of 𝐸(𝑥 ∗ ): 𝑠!"#$ (𝐸(𝑥 ∗ )), 𝑠%&' (𝐸(𝑥 ∗ )). If 𝑠!"#$ 𝐸 𝑥 ∗ < 𝑠%&' (𝐸(𝑥 ∗ )), 𝑥 ∗ is a member.

Fig. 1. Overview of our attack strategy under the partial-knowledge setting. It consists of four steps: (1) collecting an auxiliary dataset from the real world and generating a synthetic dataset using the target model; (2) extracting embeddings from both datasets; (3) estimating the corresponding embedding distributions; and (4) inferring the membership status of a target sample by comparing its embedding against the two distributions, where greater proximity to the synthetic distribution indicates higher likelihood of membership.

𝑚

1 ∑︁ 𝑎.𝑠. 𝐸 (𝑥 (𝑖) ) −−−−−→ E[𝑧] = 𝜇, 𝑚→∞ 𝑚 𝑖=1 which guarantees that the empirical mean 𝜇 𝑥 converges almost surely to the true mean 𝜇 as the number of queries 𝑚 increases. Similarly, by the multivariate strong law of large numbers, the empirical covariance Σ 𝑥 converges almost surely to the true covariance Σ: 𝑚 1 ∑︁ 𝑎.𝑠. [𝐸 (𝑥 (𝑖) ) − 𝜇 𝑥 ] [𝐸 (𝑥 (𝑖) ) − 𝜇 𝑥 ] ⊤ −−−−−→ Σ. 𝑚→∞ 𝑚 𝑖=1

This section introduces the inference approach, which applies when the adversary has partial knowledge of the target generative model’s fine-tuning dataset 𝐷 tune and can collect an auxiliary dataset 𝐷 real from real-world sources. This approach consists of four steps summarized in Figure 1. Step 1: Dataset Collection. Given partial knowledge of the fine-tuning dataset 𝐷 tune , the adversary collects an auxiliary 3

Moreover, the central limit theorem states that the sampling distribution of 𝜇 𝑥 approaches a multivariate normal distribution centered at 𝜇 with covariance Σ/𝑚. This implies that the estimation error for 𝜇 𝑥 decreases at a rate of 1/𝑚, meaning that the uncertainty in estimating the true mean shrinks rapidly as 𝑚 grows. More specifically, a non-asymptotic bound is provided by Hoeffding’s inequality. If each coordinate 𝑧 𝑗 of the embedding vector 𝑧 is bounded within [𝑎 𝑗 , 𝑏 𝑗 ], the sample mean satisfies: Pr(||𝜇 𝑥 − 𝜇|| ∞ > 𝜖) ≤

𝑑 ∑︁ 𝑗=1

2exp[−

better aligned with the generative model’s learned distribution, and thus yield embeddings with higher likelihood under the synthetic distribution (𝜇 𝑥syn , Σ 𝑥syn ) than under the real-data distribution (𝜇 𝑥real , Σ 𝑥real ). In contrast, non-members are more likely to follow the real-data distribution and therefore achieve higher likelihood under (𝜇 𝑥real , Σ 𝑥real ). IV. M EMBERSHIP I NFERENCE WITH Z ERO K NOWLEDGE

2𝑚𝜖 2 ], (𝑏 𝑗 − 𝑎 𝑗 ) 2

Step 1: Dataset collection

𝐷,-.

(

𝐷!"#

𝐷,-.

Extractor 𝐸

Add noise to 𝐷,-.

Step 3: Compute embedding distribution using empirical mean and covariance 𝜇$ = Σ$ =

% & ∑ 𝐸 𝑥 (() , & (*% &

1 + * 𝐸 𝑥 (() − 𝜇$ 𝐸 𝑥 (() − 𝜇$ . 𝑚 (*%

𝐸 𝑥!"# (

𝐸 𝑥,-.

& (*% & (*%

Step 4: Membership inference Target sample 𝑥 ∗

Extractor 𝐸

𝐸(𝑥 ∗ )

Compute log-likelihood score of 𝐸(𝑥 ∗ ): 𝑠!"# (𝐸(𝑥 ∗ )), 𝑠,-. (𝐸(𝑥 ∗ )). If 𝑠!"# 𝐸 𝑥 ∗ < 𝑠,-. (𝐸(𝑥 ∗ )), 𝑥 ∗ is a member.

Fig. 2. Overview of our attack strategy under the zero knowledge setting: (1) generating a synthetic dataset using the target model and constructing an auxiliary dataset by perturbing synthetic samples; (2) extracting embeddings from both datasets; (3) estimating the corresponding embedding distributions; and (4) inferring the membership status of a target sample by comparing its embedding against the two distributions, where greater proximity to the synthetic distribution indicates higher likelihood of membership.

1 𝑠(𝐸 (𝑥 ∗ )) = − [(𝐸 (𝑥 ∗ ) − 𝜇) ⊤ Σ −1 (𝐸 (𝑥 ∗ ) − 𝜇) + log|Σ|]. (3) 2 Using Eq. 3, the adversary evaluates two likelihood scores: 𝑠real (𝐸 (𝑥 ∗ )), computed under the real-data distribution (𝜇 𝑥real , Σ 𝑥real ), and 𝑠syn (𝐸 (𝑥 ∗ )), computed under the syntheticdata distribution (𝜇 𝑥syn , Σ 𝑥syn ). The membership inference decision rule is then defined as:

This section introduces the membership inference approach for the zero-knowledge scenario. The adversary is aware only of the general domain of the fine-tuning dataset 𝐷 tune of the target model and cannot access real-world data from that domain. As in the partial-knowledge setting, the proposed approach consists of four steps summarized in Figure 2. The key distinction lies in the first step, while the remaining three steps are identical to those in the partial-knowledge scenario.

(4)

If the inequality in Eq. 4 holds, the sample 𝑥 ∗ is inferred to be a member of the fine-tuning dataset; otherwise, it is classified as a non-member. We now justify why this decision rule yields reliable membership inference. This decision rule can be interpreted as a likelihood-ratio test between two hypotheses: H0 : 𝑥 ∗ is a non-member, and its embedding is drawn from the real-data distribution; H1 : 𝑥 ∗ is a member, and its embedding is drawn from the synthetic-data distribution induced by the fine-tuned model. Under the Gaussian approximation, Eq. 4 is equivalent to comparing the log-likelihood ratio log

𝐷!"#

Target model 𝒢

where 𝜖 denotes the tolerance for estimation error and 𝑑 is the dimensionality of 𝑧. This inequality shows that the probability of a large deviation decays exponentially as the number of queries 𝑚 increases. This result establishes that both 𝜇 𝑥 and Σ 𝑥 are statistically consistent estimators of the true embedding distribution. In practice, this means that even with a moderate number of queries, the adversary can obtain a reliable approximation of the embedding distribution. Step 4: Membership Inference. Given a target sample 𝑥 ∗ , the adversary feeds it into the encoder 𝐸 and obtains its embedding representation 𝐸 (𝑥 ∗ ) 1 . Then, given an embedding distribution (𝜇, Σ), the adversary computes the log-likelihood score of 𝐸 (𝑥 ∗ ) under a multivariate Gaussian assumption [16]:

𝑠real (𝐸 (𝑥 ∗ )) < 𝑠syn (𝐸 (𝑥 ∗ )).

Step 2: Embedding extraction 𝐷,-.

Prompts

Step 1: Auxiliary Set Creation. In the zero-knowledge setting, the adversary cannot collect any external data from the real world. Instead, based solely on the domain knowledge of 𝐷 tune , the adversary queries the target generative model G to construct a synthetic dataset 𝐷 syn . For example, if the domain of 𝐷 tune consists of news articles, the adversary can issue generic prompts such as “Write a short news article” to induce G to generate news-related samples, thereby approximating the distribution of 𝐷 tune without requiring any detailed knowledge of the data in 𝐷 tune . Subsequently, an auxiliary dataset 𝐷 aux is generated by perturbing samples in 𝐷 syn with controlled noise. For image data, the adversary applies small transformations such as Gaussian noise; for text data, the adversary generates paraphrases via back-translation or synonym replacement, which preserve semantic meaning while altering token sequences. The goal of this process is to produce auxiliary samples that (i) remain semantically close to those in 𝐷 syn , thereby probing the same local region of the data manifold, and (ii) are extremely unlikely to be members of the target fine-tuning dataset 𝐷 tune . Figure 3 illustrates

𝑝 [𝐸 (𝑥 ∗ )|𝜇 𝑥syn , Σ 𝑥syn ] 𝑝 [𝐸 (𝑥 ∗ )|𝜇 𝑥real , Σ 𝑥real ]

against zero, which is the optimal decision rule in the Neyman–Pearson sense when both class-conditional distributions are known. The rationale behind this test is that samples in the fine-tuning dataset are expected to induce outputs that are 1 Strictly speaking, a training sample consists of an input–output pair ( 𝑥 ∗ , 𝑦 ∗ ). In practice, the adversary applies the encoder to 𝑦 ∗ and obtains 𝐸 ( 𝑦 ∗ ). However, for notational consistency, we write this as 𝐸 ( 𝑥 ∗ ).

Step 4: Membership Inference. This step follows the same procedure as in the partial-knowledge setting. Using Eq. 3, the adversary computes two log-likelihood scores for the target embedding: 𝑠aux (𝐸 (𝑥 ∗ )) under the auxiliary data distribution (𝜇 𝑥aux , Σ 𝑥aux ) and 𝑠syn (𝐸 (𝑥 ∗ )) under the synthetic data distribution (𝜇 𝑥syn , Σ 𝑥syn ). The adversary then infers that 𝑥 ∗ is a member if 𝑠aux (𝐸 (𝑥 ∗ )) < 𝑠syn (𝐸 (𝑥 ∗ )); otherwise, 𝑥 ∗ is classified as a non-member.

representative examples of auxiliary data construction across the three classes of generative models. Text-to-Text: GPT-2 (1.5B)

Image-to-Text: LLaVa 7B

Input prompt: London is the capital and

Input caption and image:

largest city of England and the United Kingdom. Output: London serves as the capital city of both England and the United Kingdom and is also its largest urban area. Output with noise: London is a large city in England, often described as a capital and an important urban area.

Text-to-Image: Stable Diffusion v1.5 Input prompt:

Output:

Output/noise:

landscape of a snowy mountain with a skier trail in the snow

What feature can be seen on the back of the bus?

Output: The back of the bus features an advertisement. Output with noise: The back

V. T HEORETICAL A NALYSIS Our analysis consists of three components: (i) why the output distribution of a generative model can approximate its training data distribution, (ii) why the log-likelihood score provides a basis for making membership inference decisions, and (iii) why the decision threshold in Eq. 4 is implicitly set to zero, i.e., 𝑠real (𝐸 (𝑥 ∗ )) − 𝑠syn (𝐸 (𝑥 ∗ )) < 0. Note that we focus the analysis on the partial-knowledge setting. The analysis for the zero-knowledge setting is similar. (i) Training Data Distribution Approximation. Training data typically consist of pairs of input prompts and corresponding ground-truth responses. Therefore, our analysis focuses on the approximation between the distribution of ground-truth responses and that of the model’s generated responses, rather than the relationship between the model’s input prompts and output responses. Moreover, this analysis naturally extends to the fine-tuning setting. This is because both pre-training and fine-tuning share the same underlying mechanism: likelihoodbased optimization encourages the model to approximate the empirical distribution of the data it is trained on, regardless of whether the data come from pre-training or fine-tuning. Given a generative model G 𝜃 , let 𝑝 train denote the distribution of its training data. Training a likelihood-based generative model typically amounts to minimizing the empirical negative log-likelihood: 𝑛 1 ∑︁ log𝑝 𝜃 (𝑥𝑖 ). 𝐿 (𝜃) = − 𝑛 𝑖=1

of the bus has a bombastic advertisement on it.

Fig. 3. Examples of auxiliary data construction under three classes of generative models.

We now analyze the probability that auxiliary samples are non-members of 𝐷 tune . Suppose 𝐷 tune contains 𝑛 examples drawn from some unknown data distribution. Let 𝑝 denote the probability that a single random draw from this distribution exactly matches an auxiliary sample 𝑥 ′ . Although 𝑝 is unknown, probability theory provides useful guidance: in continuous domains, exact equality implies 𝑝 = 0, while in discrete domains, 𝑝 is nonzero but extremely small. Under this formulation, the probability that 𝑥 ′ is not in 𝐷 tune is: Pr(𝑥 ′ ∉ 𝐷 tune ) = (1 − 𝑝) 𝑛 . When 𝑝 ≪ 1 and 𝑛𝑝 is moderate, it can be approximated as: Pr(𝑥 ′ ∉ 𝐷 tune ) ≈ 𝑒 −𝑛 𝑝 . If the adversary generates 𝑚 independent auxiliary samples ′ with corresponding match probabilities 𝑝 , . . . , 𝑝 , 𝑥1′ , . . . , 𝑥 𝑚 1 𝑚 the probability that none of the 𝑚 samples appear in 𝐷 tune is: Pr(𝑥𝑖′ ∉ 𝐷 tune |∀𝑖) =

𝑚 Ö 𝑖=1

(1 − 𝑝 𝑖 ) 𝑛 ≈ exp(−𝑛

𝑚 ∑︁

𝑝 𝑖 ).

(5)

𝑖=1

In expectation, this objective corresponds to ∫ E 𝑥∼ 𝑝train [−log𝑝 𝜃 (𝑥)] = 𝑝 train (𝑥) (−log𝑝 𝜃 (𝑥))d𝑥.

This result shows that the non-membership probability decreases as 𝑛, 𝑚, or any 𝑝 𝑖 increase. Since 𝑛 (the size of 𝐷 tune ) and 𝑝 𝑖 (the match probability) are neither known nor controllable by the adversary, the only adjustable parameter is 𝑚. To maintain a high probability of non-membership, 𝑚 should therefore be kept small. In practice, we treat 𝑚 as a tunable hyperparameter in the experiments. Step 2: Embedding Extraction. This step follows the same procedure as in the partial-knowledge scenario, except that the real-world dataset 𝐷 real is replaced with the auxiliary dataset 𝐷 aux . Specifically, the adversary extracts embedding representations for samples from both 𝐷 aux and 𝐷 syn , denoted (𝑖) (𝑖) as 𝐸 (𝑥aux ) and 𝐸 (𝑥 syn ), respectively. Step 3: Compute Embedding Distribution. This step follows the same computation procedure as in the partial(𝑖) knowledge scenario. Applying it to the embeddings 𝐸 (𝑥aux ) (𝑖) and 𝐸 (𝑥syn ) yields their respective distributional representations: (𝜇 𝑥aux , Σ 𝑥aux ) for auxiliary data and (𝜇 𝑥syn , Σ 𝑥syn ) for synthetic data.

This quantity measures how well the model’s output distribution 𝑝 𝜃 explains samples drawn from the true training data distribution 𝑝 train . By adding and subtracting log𝑝 train (𝑥) inside the expectation, we obtain E 𝑥∼ 𝑝train [−log𝑝 𝜃 (𝑥)] = E 𝑥∼ 𝑝train [−log𝑝 𝜃 (𝑥) + log𝑝 train (𝑥) − log𝑝 train (𝑥)] = E 𝑥∼ 𝑝train [−log𝑝 train (𝑥)] · E 𝑥∼ 𝑝train [log𝑝 train (𝑥) − log𝑝 𝜃 (𝑥)], where the first term is exactly the entropy of the training data distribution and is independent of 𝑝 𝜃 , while the second term corresponds to the Kullback–Leibler (KL) divergence: E 𝑥∼ 𝑝train [log𝑝 train (𝑥) − log𝑝 𝜃 (𝑥)] = 𝐷 KL ( 𝑝 train || 𝑝 𝜃 ). Since the entropy term is constant with respect to 𝑝 𝜃 , minimizing the expected negative log-likelihood is equivalent to minimizing 𝐷 KL ( 𝑝 train || 𝑝 𝜃 ). Consequently, likelihood-based 5

Since KL divergence is non-negative and equals zero if and only if the two distributions coincide, we have

training drives the model’s output distribution 𝑝 𝜃 toward the training data distribution 𝑝 train , i.e., 𝐷 KL ( 𝑝 train || 𝑝 𝜃 ) → 0. To qualitatively illustrate the data distributions, we use tSNE to visualize the relationships among them [17]. Figure 4 shows the distributions of training data (member data), generated data, test data (non-member data drawn from the same dataset as the training data), and external data (nonmember data from the same task domain but from a different dataset) across the three categories of generative models. We observe a consistent trend across all three categories: the distribution of generated data aligns much more closely with that of the member data than with either type of non-member data. Moreover, although the test data are drawn from the same dataset as the training data, their distribution remains farther from the training data distribution than the generated data does. This indicates that generative models capture distributional characteristics that are more strongly influenced by the specific samples seen during training, rather than merely reflecting the overall data domain, supporting our analysis.

E𝑧∼ 𝑝real [Λ(𝑧)] ≤ 0, E𝑧∼ 𝑝syn [Λ(𝑧)] ≥ 0, with strict inequalities whenever 𝑝 real ≠ 𝑝 syn . These results imply that the likelihood ratio Λ(𝑧) is, in expectation, positive for member samples and negative for non-members. Hence, any nonzero KL divergence between the real and synthetic embedding distributions induces a statistical bias that can be exploited for membership inference. Moreover, the magnitude of the KL divergence directly controls the separability of the two hypotheses: larger divergence implies a larger expectation gap and, thus, a lower achievable classification error. (iii) Optimality of the Decision Rule. Given the two hypotheses H0 and H1 , consider a test 𝛿(𝑧) ∈ {0, 1}, where 𝛿(𝑧) = 0 corresponds to deciding H0 and 𝛿(𝑧) = 1 corresponds to deciding H1 . Under equal misclassification costs (0-1 loss), the Bayes risk is defined as 𝑅(𝛿) = 𝜋0 P0 [𝛿(𝑧) = 1] + 𝜋1 P1 [𝛿(𝑧) = 0],

Text-to-Image

Text-to-Text

where 𝜋0 = P(H0 ) and 𝜋1 = P(H1 ) denote the prior probabilities of the two hypotheses. Here, P0 [𝛿(𝑧) = 1] represents the probability of deciding H1 when H0 is true (false positive rate), and P1 [𝛿(𝑧) = 0] represents the probability of deciding H0 when H1 is true (false negative rate). The Bayes risk can equivalently be written in integral form ∫ 𝑅(𝛿) = 𝜋0 𝛿(𝑧) 𝑝 real (𝑧) + 𝜋1 (1 − 𝛿(𝑧)) 𝑝 syn (𝑧) d𝑧 ∫ = 𝜋1 𝑝 syn (𝑧) + 𝛿(𝑧) (𝜋0 𝑝 real (𝑧) − 𝜋1 𝑝 syn (𝑧)) d𝑧.

Image-to-Text Members (Training data) Generated data Non-members (Test data) Non-members (External data)

∫ Since the first term, [𝜋1 𝑝 syn (𝑧)]d𝑧 = 𝜋1 , is independent of 𝛿, minimizing 𝑅(𝛿) implies minimizing the integrand pointwise for each 𝑧. This yields the Bayes-optimal decision rule ( 1, if 𝜋0 𝑝 real (𝑧) − 𝜋1 𝑝 syn (𝑧) < 0; ∗ 𝛿 (𝑧) = 0, otherwise.

Fig. 4. t-SNE Visualization of Data Distribution.

The above rule can be equivalently expressed as 𝑝 syn (𝑧) 𝜋0 > 𝛿∗ (𝑧) = I 𝜋1 𝑝 syn (𝑧) > 𝜋0 𝑝 real (𝑧) = I 𝑝 real (𝑧) 𝜋1 𝜋0 = I Λ(𝑧) > log . 𝜋1

(ii) Likelihood Ratio for Membership Inference. Let 𝑧∗ = 𝐸 (𝑥 ∗ ) denote the embedding of a target sample 𝑥 ∗ . We consider the following binary hypotheses: H0 : 𝑧∗ ∼ 𝑝 real , H1 : 𝑧∗ ∼ 𝑝 syn ,

Thus, under equal probabilities, 𝜋0 = 𝜋1 , the threshold prior simplifies to log 𝜋𝜋01 = 0, and the Bayes-optimal rule becomes

where 𝑝 real = (𝜇 𝑥real , Σ 𝑥real ) denotes the embedding distribution of real, non-member samples drawn from the same domain as 𝐷 tune , and 𝑝 syn = (𝜇 𝑥syn , Σ 𝑥syn ) denotes the embedding distribution of model-generated (synthetic) samples. We define the log likelihood ratio as Λ(𝑧) = log

𝑝 [𝑧|𝜇 𝑥syn , Σ 𝑥syn ] 𝑝 [𝑧|𝜇 𝑥real , Σ 𝑥real ]

= log

𝛿∗ (𝑧) = I[Λ(𝑧) > 0] . That means the optimal decision rule is obtained by thresholding the log-likelihood ratio Λ(𝑧) at zero, which is exactly the decision rule adopted in our method.

𝑝 syn (𝑧) . 𝑝 real (𝑧)

VI. E XPERIMENTAL S ETUP

The proposed decision rule infers 𝑥 ∗ as a member if Λ(𝑧∗ ) > 0. Taking expectations of Λ(𝑧) under each hypothesis yields

A. Models and Datasets The choice of models and datasets follows prior work to ensure a fair comparison. Detailed descriptions of these adopted datasets are provided in the Appendix.

E𝑧∼ 𝑝real [Λ(𝑧)] = −𝐷 KL ( 𝑝 real || 𝑝 syn ), E𝑧∼ 𝑝syn [Λ(𝑧)] = 𝐷 KL ( 𝑝 syn || 𝑝 real ). 6

TABLE I OVERALL R ESULTS OF O UR M ETHOD AND BASELINES ACROSS T HREE C ATEGORIES OF G ENERATIVE M ODELS IN THE PARTIAL K NOWLEDGE S ETTING

ASR ↑ AUC ↑ T@1%F ↑

Text-to-Text GPT2+Wiki103 Falcon+XSum Ours SPV ICP Ours SPV ICP

Text-to-Image Image-to-Text SD1.5+MSCOCO SD2.1+CelebA-D. LLaVA+COCO MiniGPT4+CC SBU Ours Score CLiD Ours Score CLiD Ours Temp. MaxRényi Ours Temp. MaxRényi

0.90 0.93 0.52

0.86 0.95 0.62

0.87 0.89 0.88 0.91 0.48 0.51

0.98 0.99 0.63

0.95 0.96 0.97 0.97 0.61 0.60

0.85 0.94 0.56

0.84 0.95 0.57

0.96 0.99 0.59

0.88 0.95 0.44

0.92 0.96 0.48

0.91 0.97 0.55

0.88 0.95 0.53

0.89 0.94 0.52

0.85 0.94 0.59

0.83 0.92 0.55

0.82 0.92 0.53

TABLE II OVERALL R ESULTS OF O UR M ETHOD AND BASELINES ACROSS T HREE C ATEGORIES OF G ENERATIVE M ODELS IN THE Z ERO K NOWLEDGE S ETTING

ASR ↑ AUC ↑ T@1%F ↑

Text-to-Text GPT2+Wiki103 Falcon+XSum Ours SPV ICP Ours SPV ICP

Text-to-Image Image-to-Text SD1.5+MSCOCO SD2.1+CelebA-D. LLaVA+COCO MiniGPT4+CC SBU Ours Score CLiD Ours Score CLiD Ours Temp. MaxRényi Ours Temp. MaxRényi

0.83 0.90 0.48

0.78 0.90 0.45

0.70 0.73 0.79 0.80 0.40 0.39

0.88 0.96 0.61

0.85 0.82 0.95 0.93 0.54 0.55

0.50 0.46 0.15

0.54 0.48 0.18

0.89 0.94 0.53

Text-to-Text Models. Following the setup in [6], we adopt two open-source LLMs: GPT-2 (1.5B) [18] and Falcon-7B [19]. The fine-tuning datasets include Wikitext-103 [20] and XSum [21]. GPT-2 was pre-trained on the WebText corpus, while Falcon-7B was pre-trained on the RefinedWeb dataset. Neither model’s pre-training documentation lists Wikitext-103 or XSum as part of their training data. Thus, it is considered that these benchmark datasets were not included in their pre-training corpora. In addition, we adopt DistilBERT [22] as the embedding extractor 𝐸 due to its strong semantic representation capability. Text-to-Image Models. Following the setup in [7], we use Stable Diffusion v1.5 and Stable Diffusion v2.1 [23] as our text-to-image generative models. These models were pretrained on large-scale LAION datasets, including LAION2B and LAION-5B [24]. To avoid any overlap between the models’ pre-training data and our fine-tuning datasets, we fine-tune them on MS-COCO [25] and CelebA-Dialog [26]. Additionally, we adopt BLIP [15] as the embedding extractor 𝐸 due to its ability to align textual prompts with generated images in a shared embedding space. Image-to-Text Models. Following the setup in [8], we employ LLaVA-7B [27] and MiniGPT-4 [28] as our image-to-text generative models, and fine-tune them on COCO 2017 [25] and CC SBU ALIGN [28]. These datasets are not listed in the pre-training data descriptions provided in the official documentation of LLaVA-7B and MiniGPT-4. Therefore, we consider that COCO 2017 and CC SBU ALIGN were not included in their pre-training corpora. Moreover, we use allMiniLM-L6-v2 [29] as the embedding extractor 𝐸 due to its strong capability in sentence-level similarity measurement.

0.52 0.52 0.21

0.56 0.64 0.20

0.88 0.96 0.53

0.60 0.58 0.23

0.58 0.55 0.19

0.85 0.91 0.57

0.58 0.63 0.31

0.60 0.61 0.25

C. Baseline Attacks For each category of generative models, we select two stateof-the-art membership inference attacks as baselines. Specifically, for text-to-text models, we adopt SPV-MIA (Selfcalibrated Probabilistic Variation–based Membership Inference Attack) [6], which exploits probabilistic variation induced by LLM memorization as a membership signal, and ICP-MIA (In-Context Probing-based MIA) [31], which utilizes the optimization gap signal where member samples exhibit minimal remaining loss-reduction potential, while non-members retain significant potential for further optimization. For text-to-image models, we adopt the score-based MIA [7], which leverages the diffusion training objective to quantify a model’s memorization of query samples through similarity scores. We also consider CLiD-MIA [32], which exploits the phenomenon of conditional overfitting, where the model tends to overfit the conditional distribution of images given their corresponding text prompts, rather than the marginal distribution of images alone. For image-to-text models, we adopt the temperaturebased MIA [8], which infers membership by analyzing the sensitivity of generated outputs to temperature variations across multiple samples. We also consider MaxRényi-K% MIA [33], which exploits the Rényi entropy of the next-token probability distribution over image or text tokens: if the model has seen a sample during training, it is typically more confident in predicting the next token, resulting in lower Rényi entropy. VII. E XPERIMENTAL R ESULTS Overall Results. The overall results of our method and the baseline approaches across the three classes of generative models under the partial-knowledge and zero-knowledge settings are reported in Tables I and II. We observe that, in the partialknowledge setting (Table I), our method achieves performance comparable to the baselines across all three metrics: ASR, AUC, and TPR@1%FPR. In contrast, in the zero-knowledge setting (Table II), our method consistently outperforms the baselines, with notable improvements in terms of AUC. This performance gap arises because existing baselines heavily

B. Evaluation Metrics Following prior membership inference studies [6, 7, 8, 30], we evaluate attack performance using three metrics: Attack Success Rate (ASR), which measures the overall accuracy of membership inference; Area Under the Receiver Operating Characteristic Curve (AUC); and True Positive Rate (TPR) evaluated at a low False Positive Rate (FPR). 7

rely on prior knowledge of the training data distribution or model-specific signals, which become unavailable in the zeroknowledge scenario. By contrast, our method leverages a modality-agnostic, distributional perspective that exploits the approximation between the model’s output distribution and its training data distribution, enabling effective membership inference even in the limited knowledge setting.

Text-to-Text (GPT2 + Wiki103) 1.0

Partial 2% Zero 2%

Partial 3% Zero 3%

0.8

1.2 1.0

Partial 2% Zero 2%

0.2

Text-to-Image (SD 1.5 + MSCOCO)

ASR

0.8

0.4

0.6

0.2

0.2 0.0

Ours

(a) T2T GPT2+Wiki103 Text-to-Image (SD 1.5 + MSCOCO) Partial 1% Zero 1%

Partial 2% Zero 2%

Partial 3% Zero 3%

0.8

1.0

Partial 1% Zero 1%

Partial 2% Zero 2%

0.2

0.2 0.0

Ours

Image-to-Text (LLaVA + COCO)

Image-to-Text (MiniGPT-4 + CC-SBU)

1.0

Partial 1% Zero 1%

Partial 2% Zero 2%

Partial 3% Zero 3%

1.0

0.6

0.4

0.2

0.0

Partial 2% Zero 2%

Ours

(e) I2T LLaVa+COCO

0.0

Partial 2% Zero 2%

Partial 3% Zero 3%

0.0

Ours

(d) T2I SD2.1+CelebA-D.

Image-to-Text (LLaVA + COCO)

1.2 Image-to-Text (MiniGPT-4 + CC-SBU)

Partial 1% Zero 1%

Partial 2% Zero 2%

Partial 3% Zero 3%

1.0

Partial 1% Zero 1%

Partial 2% Zero 2%

Partial 3% Zero 3%

0.8 0.6

0.4

0.2

0.0

Ours

(e) I2T LLaVa+COCO

0.0

Ours

(f) I2T MiniGPT-4+CC SUB

Partial 3% Zero 3%

Fig. 6. AUC Performance of Our Method Under Different Dataset Sizes

ASR

0.8

ASR

0.8

Partial 1% Zero 1%

0.6

(d) T2I SD2.1+CelebA-D.

Text-to-Image (SD 2.1 + CelebA-D)

0.8

Ours

1.0

Ours

1.0

0.2

1.2

ASR

0.4

1.2

0.4

Partial 3% Zero 3%

0.6

0.4

Partial 3% Zero 3%

0.2

Text-to-Image (SD 2.1 + CelebA-D)

Partial 3% Zero 3%

(b) T2T Falcon+XSum

0.4 0.0

0.8

0.6

0.0

Ours

(b) T2T Falcon+XSum 1.2

Partial 2% Zero 2%

AUC

1.0

Partial 1% Zero 1%

AUC

0.6

0.0

Ours

(a) T2T GPT2+Wiki103

0.4

Partial 2% Zero 2%

AUC

0.4

0.2

1.0

Partial 1% Zero 1%

0.6

0.4

Partial 3% Zero 3%

0.8

0.6

1.0

Text-to-Text (Falcon + XSum)

0.8

0.6

Text-to-Text (Falcon + XSum) Partial 1% Zero 1%

1.2

AUC

Partial 1% Zero 1%

Partial 3% Zero 3%

AUC

Text-to-Text (GPT2 + Wiki103)

Partial 2% Zero 2%

0.8

0.0

1.0

Partial 1% Zero 1%

extractor has a significant impact on the performance of our method. Specifically, for text-to-text generative models, we additionally consider two widely used embedding extractors: RoBERTa-base [34] and all-MiniLM-L6-v2 [29]. For text-toimage generative models, we evaluate two additional visual embedding extractors: ViT-base [35] and CLIP-ViT [36]. For image-to-text generative models, we further include ALBERTlarge-v2 [37] and all-MPNet-base-v2 [38] as alternatives. Tables III, IV, and V report the performance of our method across the three classes of generative models. The results show that, under the partial-knowledge setting, the choice of embedding extractor does affect the attack performance. For instance, in the image-to-text setting (LLaVA + COCO2017 in Table V), when using MPNet-base as the embedding extractor under the partial-knowledge scenario, all three evaluation metrics, ASR, AUC, and TPR@1%FPR, decrease compared to using MiniLM as the extractor. This performance degradation can be attributed to differences in semantic sensitivity across embedding extractors. In particular, MiniLM is optimized for sentence-level semantic similarity, which is more effective at capturing fine-grained distributional shifts between member and non-member samples in the embedding space. In contrast, MPNet-base prioritizes global contextual encoding, which may smooth out subtle membership-related signals and reduce distributional separability. Similar trends are observed in the zero-knowledge setting.

Ours

(f) I2T MiniGPT-4+CC SUB

Fig. 5. ASR Performance of Our Method Under Different Dataset Sizes

Impact of Different Dataset Sizes. Step 1 of our method focuses on data collection and synthesis. We investigate whether the amount of collected data has a significant impact on the performance of our method. Specifically, we vary the sizes of both 𝐷 syn and 𝐷 real (𝐷 aux ) to 1%, 2%, and 3% of the size of 𝐷 tune . Figure 5, 6, and 7, respectively, report the ASR, AUC, and TPR@1%FPR results across the three types of generative models. We observe that the amount of data has a limited impact on the performance of our method under both the partial-knowledge and zero-knowledge settings, demonstrating strong scalability and robustness to data availability. This behavior is expected, as our method relies on distributional estimation in the embedding space, which stabilizes quickly even with a relatively small number of samples. Impact of Different Embedding Extractors. In Step 2 of our method, we employ embedding extractors to map data samples into a numerical embedding space for subsequent computation. We investigate whether the choice of embedding 8

TABLE III R ESULTS OF O UR M ETHOD ON T EXT- TO -T EXT M ODELS WITH D IFFERENT E MBEDDING E XTRACTORS

Distil ASR ↑ AUC ↑ TPR@1%FPR ↑

Partial-Knowledge Setting GPT2+Wiki103 Falcon+XSum RoBERTa MiniLM Distil RoBERTa MiniLM

0.88 0.92 0.51

0.89 0.94 0.50

0.87 0.94 0.44

0.96 0.99 0.63

0.96 0.99 0.61

0.85 0.92 0.39

Distil

0.83 0.90 0.48

Zero-Knowledge Setting GPT2+Wiki103 Falcon+XSum RoBERTa MiniLM Distil RoBERTa MiniLM

0.71 0.91 0.40

0.73 0.92 0.42

0.88 0.96 0.51

0.68 0.91 0.32

0.70 0.92 0.37

TABLE IV R ESULTS OF O UR M ETHOD ON T EXT- TO -I MAGE M ODELS WITH D IFFERENT E MBEDDING E XTRACTORS

ASR ↑ AUC ↑ TPR@1%FPR ↑

Partial-Knowledge Setting SD1.5+MSCOCO SD2.1+CelebA-Dialog BLIP ViT-Base CLIP-ViT BLIP ViT-Base CLIP-ViT

Zero-Knowledge Setting SD1.5+MSCOCO SD2.1+CelebA-Dialog BLIP ViT-Base CLIP-ViT BLIP ViT-Base CLIP-ViT

0.84 0.92 0.62

0.78 0.90 0.45

0.86 0.93 0.57

0.73 0.84 0.23

0.96 0.99 0.59

0.98 0.99 0.54

0.98 0.99 0.57

0.79 0.87 0.18

0.66 0.72 0.19

0.89 0.94 0.53

0.70 0.82 0.13

0.43 0.32 0.10

TABLE V R ESULTS OF O UR M ETHOD ON I MAGE - TO -T EXT M ODELS WITH D IFFERENT E MBEDDING E XTRACTORS Partial-Knowledge Setting LLaVA+COCO MiniGPT4+CC SBU MiniLM ALBERT MPNet MiniLM ALBERT MPNet ASR ↑ AUC ↑ TPR@1%FPR ↑

0.89 0.97 0.55

0.88 0.82 0.50

0.62 0.66 0.25

Text-to-Text (GPT2 + Wiki103) Partial 2% Zero 2%

Partial 3% Zero 3%

0.8

Partial 1% Zero 1%

Partial 2% Zero 2%

Partial 3% Zero 3%

Ours

(b) T2T Falcon+XSum 0.8

Text-to-Image (SD 2.1 + CelebA-D) Partial 1% Zero 1%

Partial 2% Zero 2%

Partial 3% Zero 3%

0.6

TPR@1%FPR

0.4

0.0

Ours

Partial 2% Zero 2%

Partial 3% Zero 3%

Ours

(d) T2I SD2.1+CelebA-D. 0.8

Image-to-Text (MiniGPT-4 + CC-SBU) Partial 1% Zero 1%

Partial 2% Zero 2%

Partial 3% Zero 3%

0.6

TPR@1%FPR

0.6

0.4

0.2

Ours

(e) I2T LLaVa+COCO

0.0

0.89 0.92 0.58

0.68 0.73 0.35

0.85 0.91 0.57

0.62 0.65 0.24

0.68 0.75 0.25

Additionally, as shown in Table V, when using ALBERTlarge-v2 as the embedding extractor, the proposed method achieves strong performance under the LLaVA + COCO2017 configuration, but exhibits noticeable performance degradation under the MiniGPT4 + CC SBU ALIGN configuration, in both partial-knowledge and zero-knowledge settings. This discrepancy may be caused by the differences in the semantic characteristics of the underlying datasets as well as the representational bias of the extractor. In particular, ALBERT is optimized for textual semantic compression and sentencelevel similarity, which aligns well with the linguistic structure of COCO2017. However, CC SBU ALIGN contains more diverse web-scale captions, making it harder for ALBERT to capture fine-grained cross-modal correspondences.

0.2

0.88 0.96 0.53

Notably, a closer inspection of Table IV reveals that for text-to-image models, the ViT-Base and CLIP-ViT embedding extractors perform well under the partial-knowledge setting but exhibit noticeably degraded performance in the zeroknowledge setting. This discrepancy can be attributed to the fact that ViT-Base and CLIP-ViT rely more heavily on highlevel semantic alignment learned from large-scale paired data, which is beneficial when some prior knowledge about the training distribution is available. In the zero-knowledge setting, however, the lack of training data information makes it more difficult for these extractors to capture subtle distributional differences between member and non-member samples. As a result, the induced embedding distributions become less separable, leading to reduced membership inference performance.

Partial 3% Zero 3%

TPR@1%FPR

0.0

Ours

0.4

0.0

Partial 2% Zero 2%

0.2

0.6

0.8

Partial 1% Zero 1%

0.4

Text-to-Image (SD 1.5 + MSCOCO)

0.0

0.68 0.72 0.28

Text-to-Text (Falcon + XSum)

(a) T2T GPT2+Wiki103 0.8

0.63 0.67 0.26

0.6

TPR@1%FPR

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

Partial 1% Zero 1%

0.85 0.94 0.59

Zero-Knowledge Setting LLaVA+COCO MiniGPT4+CC SBU MiniLM ALBERT MPNet MiniLM ALBERT MPNet

Ours

(f) I2T MiniGPT-4+CC SUB

Impact of Different Distribution Estimation Methods. In Step 3 of our method, the embedding distribution is estimated by computing the average vector and the covariance matrix of

Fig. 7. TPR@1%FPR of Our Method Under Different Dataset Sizes

the extracted embeddings. We investigate whether alternative strategies for estimating the embedding distribution affect the performance of our method. Specifically, instead of using the average vector, we consider the Í geometric median of the embeddings, defined as: 𝑎𝑟𝑔𝑚𝑖𝑛 𝑧 𝑖 ∥𝐸 (𝑥 (𝑖) ) −𝑧∥ 2 , as a robust estimator of central tendency. The results of ASR and AUC are shown in Figure 8, while the corresponding TPR@1%FPR results are reported in Figure 9. We observe that, in Figure 8, using the geometric median to represent the embedding distribution yields performance comparable to using the average vector for text-to-text and text-to-image generative models. However, for image-to-text generative models, adopting the geometric median leads to a noticeable degradation in performance compared to the average-based estimator (Figure 8(f)). This performance drop is due to the fact that image-to-text embeddings tend to exhibit higher variance and more complex, multimodal structures. While the geometric median is robust to outliers, it captures only the central location of the distribution and ignores second-order statistics that are critical for distinguishing subtle membership-related distributional shifts. Moreover, in Figure 9, focusing on the image-to-text setting (Figure 9(c)), we observe that using the geometric median to estimate the embedding distribution leads to substantially worse performance than the average-based estimation adopted in our method. This degradation arises because image-totext embeddings often exhibit high variance and complex multimodal distributions. While the geometric median is robust to outliers, it captures only the central tendency of the distribution and discards important second-order information, thus providing an insufficient representation for likelihoodbased membership inference.

1.2 1.0

Text-to-Text Models Avg + Partial Avg + Zero

1.2

GM + Partial GM + Zero

1.0

0.8

AUC

ASR

0.6

0.4

0.2

0.0 GPT2 + Wiki103

Falcon + XSum

(a) Text-to-Text ASR

1.0

GM + Partial GM + Zero

0.8

0.6

1.2

Text-to-Text Models Avg + Partial Avg + Zero

(b) Text-to-Text AUC

Text-to-Image Models Avg + Partial Avg + Zero

Falcon + XSum

1.2

GM + Partial GM + Zero

1.0

0.8

Text-to-Image Models Avg + Partial Avg + Zero

GM + Partial GM + Zero

ASR

AUC

0.8

0.6

0.4

0.2

0.0 SD 1.5 + MSCOCO SD 2.1 + CelebA-D

(d) Text-to-Image AUC

Image-to-Text Models 1.0

Avg + Partial Avg + Zero

GM + Partial GM + Zero

0.8

1.2 1.0

Image-to-Text Models Avg + Partial Avg + Zero

GM + Partial GM + Zero

0.8

ASR

AUC

0.6

0.4

0.2

0.0 LLaVA + COCO

MiniGPT-4 + CC-SBU

0.0 LLaVA + COCO

(e) Image-to-Text ASR

MiniGPT-4 + CC-SBU

(f) Image-to-Text AUC

Fig. 8. ASR and AUC Performance of Our Method with Different Distribution Estimation Methods Text-to-Text Models

Impact of Replacing Likelihood Scores with DistanceBased Metrics. In Step 4 of our method, we determine the membership status of a target sample based on the loglikelihood score of its embedding (Eq. 3). We now evaluate the importance of using the log-likelihood score by considering an alternative, distance-based decision rule. Specifically, instead of computing log-likelihoods, we directly compare the embedding of the target sample, 𝐸 (𝑥 ∗ ), with the average embeddings of synthetic data, 𝜇 𝑥syn , and real (auxiliary) data, 𝜇 𝑥real (or 𝜇 𝑥aux ). If dist(𝐸 (𝑥 ∗ ), 𝜇 𝑥syn ) < dist(𝐸 (𝑥 ∗ ), 𝜇 𝑥real ) (or dist(𝐸 (𝑥 ∗ ), 𝜇 𝑥syn ) < dist(𝐸 (𝑥 ∗ ), 𝜇 𝑥aux )), that is, if the target sample 𝑥 ∗ is closer to the synthetic data than to the real (auxiliary) data in the embedding space, it is inferred to be a member. We evaluate this distance-based variant using two commonly adopted distance metrics, namely cosine similarity and Wasserstein distance. The results are reported in Tables VI, VII, and VIII. We observe that replacing the log-likelihood score with a distancebased decision rule leads to consistently degraded performance in most settings, regardless of whether cosine similarity or Wasserstein distance is used, and under both partial-knowledge and zero-knowledge threat models. This performance drop highlights the importance of the log-likelihood score, which explicitly incorporates both first-order and second-order statistics of the embedding distribution, rather than relying solely

GM + Partial GM + Zero

Text-to-Image Models Avg + Partial GM + Partial 0.7 Avg + Zero GM + Zero 0.6 0.5 0.4 0.3 0.2 0.1 0.0 SD 1.5 + MSCOCO SD 2.1 + CelebA-D

TPR@1%FPR

Avg + Partial 0.7 Avg + Zero 0.6 0.5 0.4 0.3 0.2 0.1 0.0 GPT2 + Wiki103

Falcon + XSum

(a) Text-to-Text TPR

(b) Text-to-Image TPR

Image-to-Text Models GM + Partial GM + Zero

TPR@1%FPR

0.7 Avg + Partial Avg + Zero 0.6 0.5 0.4 0.3 0.2 0.1 0.0 LLaVA + COCO

MiniGPT-4 + CC-SBU

on pointwise proximity. An interesting phenomenon arises for text-to-text generative models (Table VI), where using the Wasserstein distance yields relatively strong performance 10

in the partial-knowledge setting but performs poorly in the zero-knowledge setting. This behavior can be attributed to the fact that, under partial knowledge, the adversary has access to auxiliary samples that partially reflect the true training distribution, allowing Wasserstein distance to capture coarse distributional differences. In contrast, in the zeroknowledge setting, the lack of reliable auxiliary information makes distance-based comparisons highly sensitive to noise and distributional mismatch, causing the Wasserstein distance to lose discriminative power.

degradation. A positive threshold indicates that the target sample is closer to the real (auxiliary) data than to the synthetic data in the embedding space, which strongly suggests non-membership. Consequently, positive thresholds bias the decision rule toward predicting non-membership, leading to a significant reduction in membership inference effectiveness. 0.7 0.6

1.2

Partial 0.2 Zero 0.2

Partial 0 Zero 0

Partial -0.2 Zero -0.2

1.2

Partial -0.4 Zero -0.4

1.0

0.8

ASR

GPT2+Wiki103

0.0

Falcon+XSum

(a) Text-to-Text ASR

GPT2+Wiki103

Partial 0 Zero 0

Partial -0.2 Zero -0.2

1.2 1.0

0.8

ASR

0.6

SD 2.1 + CelebA-D

0.0

SD 1.5 + MSCOCO

Partial 0 Zero 0

Partial -0.1 Zero -0.1

1.2 0.8

Partial 0 Zero 0

Partial -0.1 Zero -0.1

ASR

0.6

0.4

0.2

LLaVA + COCO

SD 2.1 + CelebA-D

(e) Image-to-Text ASR

0.0

LLaVA + COCO

TPR@1%FPR

Partial -0.3 Zero -0.3

AUC

1.0

0.8

Partial 0.2 Zero 0.2

SD 2.1 + CelebA-D

Image-to-Text Models Partial 0.2 Zero 0.2

Partial 0 Zero 0

Partial -0.1 Zero -0.1

Partial -0.3 Zero -0.3

TPR@1%FPR

SD 2.1 + CelebA-D

Image-to-Text Models Partial -0.3 Zero -0.3

1.0

0.0

Partial -0.4 Zero -0.4

(d) Text-to-Image AUC

Image-to-Text Models Partial 0.2 Zero 0.2

SD 1.5 + MSCOCO

(b) Text-to-Image TPR

LLaVA + COCO

SD 2.1 + CelebA-D

Impact of Using Neural Network-based Binary Classifier to Decide Membership Status. In Step 4, our method determines the membership status of a target sample using a comparisonbased decision rule (Eq. 4). We investigate whether replacing this rule with a binary classifier affects the attack performance. Specifically, we train a neural network–based binary classifier consisting of three layers: an input layer with 𝑑 neurons corresponding to the dimensionality of the embedding vector, a hidden layer with 256 neurons, and an output layer with two neurons. The classifier is trained on a combined dataset composed of the synthetic dataset and the auxiliary dataset, where synthetic samples are labeled as members and auxiliary samples as non-members. We also experimented with deeper network architectures. However, increasing the number of layers does not lead to significant changes in performance. The results are reported in Tables IX and X. We observe that, for text-to-text and image-to-text generative models, the classifier-based approach achieves performance comparable to that of the comparison-based decision rule under both partialknowledge and zero-knowledge settings. This suggests that, in these modalities, the embedding distributions of synthetic and auxiliary data are sufficiently separable for a supervised classifier to learn an effective decision boundary. In contrast, for text-to-image generative models, the classifier-based approach exhibits significantly poorer performance than the comparisonbased decision rule. This may be due to the higher complexity and variability of image embeddings, which often exhibit substantial intra-class variance and weaker separability. In such cases, the likelihood-based comparison rule explicitly models the underlying embedding distributions, making it more robust to distributional noise than the classifier.

Partial -0.4 Zero -0.4

0.2

SD 1.5 + MSCOCO

0.0

Fig. 11. TPR@1%FPR of Our Method with Different Threshold Values

Falcon+XSum

Partial -0.2 Zero -0.2

Falcon+XSum

0.4

0.2

1.2

0.0

0.6

0.4 0.0

Partial 0 Zero 0

0.1

GPT2+Wiki103

0.1

AUC

1.0

Partial 0.2 Zero 0.2

0.2

Text-to-Image Models Partial -0.4 Zero -0.4

Partial -0.4 Zero -0.4

0.3

(b) Text-to-Text AUC

Text-to-Image Models Partial 0.2 Zero 0.2

Partial -0.2 Zero -0.2

0.4

0.2

Partial 0 Zero 0

0.5

0.4

1.2

0.6

Text-to-Image Models Partial 0.2 Zero 0.2

0.3

(a) Text-to-Text TPR 0.7

0.6 0.4

0.1 0.0

0.7 0.5

0.2

0.6

0.0

Partial -0.2 Zero -0.2

Partial -0.4 Zero -0.4

0.3

AUC

1.0

Partial 0 Zero 0

Partial -0.2 Zero -0.2

0.4

Text-to-Text Models Partial 0.2 Zero 0.2

Partial 0 Zero 0

0.5

Impact of Threshold Selection on Membership Inference. In Step 4 of our method, the inference threshold is implicitly set to 0, i.e., 𝑠real (𝐸 (𝑥 ∗ )) − 𝑠syn (𝐸 (𝑥 ∗ )) < 0. We investigate if varying the threshold value affects the performance of our method. The results of ASR and AUC are shown in Figure 10, while the results of TPR@1%FPR are presented in Figure 11. We observe that setting the threshold to 0 consistently yields the best performance across both partial-knowledge and zero-knowledge settings under all three evaluation metrics. Deviating from this value leads to performance degradation, which aligns well with our theoretical analysis. This behavior arises because thresholding the log-likelihood ratio at 0 corresponds to the Bayes-optimal decision rule under equal prior probabilities. Any deviation from this optimal threshold introduces a systematic bias toward either false positives or false negatives, increasing the overall classification error. Text-to-Text Models

Text-to-Text Models Partial 0.2 Zero 0.2

SD 2.1 + CelebA-D

(f) Image-to-Text AUC

Fig. 10. ASR and AUC of Our Method with Different Threshold Values

In particular, setting the threshold to a positive value (e.g., 0.2 in our experiments) results in substantial performance 11

TABLE VI R ESULTS OF O UR M ETHOD ON T EXT- TO -T EXT M ODELS WITH D ISTANCE - BASED M ETRICS Partial-Knowledge Setting GPT2+Wiki103 Falcon+XSum Likelihood Wassers. Cosine Likelihood Wassers. ASR ↑ AUC ↑ T@1%F ↑

0.93 0.97 0.51

0.94 0.97 0.49

0.49 0.46 0.10

0.98 0.99 0.58

0.98 0.99 0.56

Cosine

0.46 0.44 0.04

Zero-Knowledge Setting GPT2+Wiki103 Falcon+XSum Likelihood Wassers. Cosine Likelihood Wassers.

0.83 0.90 0.48

0.51 0.49 0.11

0.73 0.82 0.14

0.88 0.96 0.51

0.45 0.44 0.04

Cosine

0.81 0.89 0.34

TABLE VII R ESULTS OF O UR M ETHOD ON T EXT- TO -I MAGE M ODELS WITH D ISTANCE - BASED M ETRICS Partial-Knowledge Setting SD1.5+MSCOCO SD2.1+CelebA-Dialog Likelihood Wassers. Cosine Likelihood Wassers. Cosine ASR ↑ AUC ↑ T@1%F ↑

0.84 0.92 0.62

0.59 0.77 0.21

0.64 0.71 0.25

0.96 0.99 0.59

0.80 0.97 0.46

0.97 0.99 0.65

Zero-Knowledge Setting SD1.5+MSCOCO SD2.1+CelebA-Dialog Likelihood Wassers. Cosine Likelihood Wassers. Cosine

0.78 0.90 0.45

0.59 0.76 0.15

0.75 0.83 0.19

0.89 0.94 0.53

0.80 0.95 0.46

0.87 0.93 0.68

TABLE VIII R ESULTS OF O UR M ETHOD ON I MAGE - TO -T EXT M ODELS WITH D ISTANCE - BASED M ETRICS Partial-Knowledge Setting LLaVA+COCO MiniGPT4+CC SBU Likelihood Wassers. Cosine Likelihood Wassers. Cosine ASR ↑ AUC ↑ T@1%F ↑

0.89 0.97 0.55

0.70 0.68 0.19

0.69 0.80 0.35

0.85 0.94 0.59

0.56 0.53 0.10

0.68 0.65 0.21

Zero-Knowledge Setting LLaVA+COCO MiniGPT4+CC SBU Likelihood Wassers. Cosine Likelihood Wassers. Cosine

0.88 0.96 0.53

0.69 0.80 0.35

0.69 0.70 0.26

0.85 0.91 0.57

0.56 0.52 0.09

0.57 0.56 0.12

TABLE IX R ESULTS OF O UR M ETHOD U SING THE C OMPARISON RULE VS . A C LASSIFIER ACROSS T HREE C ATEGORIES OF G ENERATIVE M ODELS IN THE PARTIAL K NOWLEDGE S ETTING Text-to-Text GPT2+Wiki103 Falcon+XSum Compa. Classifier Compa. Classifier ASR ↑ AUC ↑ TPR@1%FPR ↑

0.88 0.92 0.51

0.90 0.93 0.53

0.96 0.99 0.63

0.97 0.99 0.65

Text-to-Image SD1.5+MSCOCO SD2.1+CelebA-D. Compa. Classifier Compa. Classifier

0.84 0.92 0.62

0.61 0.57 0.11

0.96 0.99 0.59

0.58 0.55 0.04

Image-to-Text LLaVA+COCO MiniGPT4+CC SBU Compa. Classifier Compa. Classifier

0.89 0.97 0.55

0.91 0.94 0.55

0.85 0.94 0.59

0.86 0.90 0.56

TABLE X R ESULTS OF O UR M ETHOD U SING THE C OMPARISON RULE VS . A C LASSIFIER ACROSS T HREE C ATEGORIES OF G ENERATIVE M ODELS IN THE Z ERO K NOWLEDGE S ETTING Text-to-Text GPT2+Wiki103 Falcon+XSum Compa. Classifier Compa. Classifier ASR ↑ AUC ↑ TPR@1%FPR ↑

0.83 0.90 0.48

0.85 0.89 0.45

0.88 0.96 0.61

0.90 0.93 0.58

Text-to-Image SD1.5+MSCOCO SD2.1+CelebA-D. Compa. Classifier Compa. Classifier

0.78 0.90 0.45

0.55 0.56 0.08

Summary. Overall, our method achieves performance comparable to existing baselines under the partial-knowledge setting, while consistently outperforming them in most scenarios under the more challenging zero-knowledge setting. We observe that the amount of collected data has a limited impact on the effectiveness of our method, demonstrating strong scalability. In contrast, the choice of embedding extractor, the strategy used for embedding distribution estimation, the selection of the decision rule and threshold values, and the use of an alternative binary classifier for decision making all have a noticeable impact on performance.

0.89 0.94 0.53

0.53 0.51 0.07

Image-to-Text LLaVA+COCO MiniGPT4+CC SBU Compa. Classifier Compa. Classifier

0.88 0.96 0.53

0.89 0.90 0.48

0.85 0.91 0.57

0.86 0.89 0.51

VIII. E XTENSION S TUDY The above experiments focus on membership inference against fine-tuning data. We extend our study to pre-training data. Although pre-training datasets are often considered less sensitive than fine-tuning datasets, data owners may still wish to revoke their data and thus need to verify whether their data were used during pre-training. Therefore, determining membership in pre-training data is also a practical problem. To conduct this evaluation, we select one representative model for each class of generative models: GPT-2 (1.5B) [18] for text-to-text, Guided-Diffusion [39] for text-to-image, and MiniGPT-4 [28] for image-to-text, all of which explicitly disclose their pre-training data sources. The results are reported 12

in Table XI. We observe that our method remains effective in inferring membership with respect to pre-training data. This is because likelihood-based pre-training, similar to finetuning, drives the model’s output distribution to approximate the underlying pre-training data distribution. Thus, samples drawn from the pre-training set induce embedding distributions that are statistically closer to those of model-generated outputs than non-member samples.

examples in Figure 12 reveals that DP-SGD also degrades the overall generative quality of the models. For instance, in the first row of Figure 12, the model trained without DP-SGD generates an image that closely aligns with the input prompt by producing an appropriate large parking lot. In contrast, when DP-SGD is applied, the generated image resembles a highway rather than a parking lot, indicating a loss of semantic fidelity.

W/O DP-SGD

TABLE XI R ESULTS OF O UR M ETHOD ON P RE - TRAINED M ODELS

ASR AUC T@1%F

Input prompt:

GPT-2 Partial Zero Knowl. Knowl.

Guided-diffu. Partial Zero Knowl. Knowl.

LLaVa Partial Zero Knowl. Knowl.

0.86 0.91 0.45

0.96 0.99 0.69

0.85 0.88 0.40

0.84 0.88 0.42

0.89 0.85 0.32

Buses parked in a large parking lot

0.83 0.85 0.36

Input prompt: Landscape of a snowy mountain with a skier trail in the snow

IX. D EFENSE To evaluate the robustness of our method, we examine its performance in the presence of a defense mechanism. We adopt DP-SGD (Differentially Private Stochastic Gradient Descent) [40], a widely used modality-agnostic defense against privacy leakage. By injecting calibrated noise into gradients during training, DP-SGD bounds the influence of any individual training record on the learned model parameters. Thus, the information that the model can reveal about a specific data point is formally limited, and the resulting privacy leakage is guaranteed not to exceed a predefined upper bound, regardless of how many outputs the adversary can query from the target model. In our evaluation, the hyperparameter settings for DPSGD are summarized in Table XII, and the corresponding experimental results are reported in Table XIII.

Fig. 12. Examples of images generated by a text-to-image model trained with and without DP-SGD.

In addition to DP-SGD, several defense mechanisms have also been proposed recently, such as [5] for LLMs and [41] for diffusion models. However, these approaches are designed to specific modalities and therefore are not directly applicable to our modality-agnostic setting. X. R ELATED W ORK Membership inference was first introduced by Shokri et al. [3, 42, 43]. A common attack strategy trains a binary classifier using behavioral features, such as the target model’s output probabilities, collected from shadow models, and then predicts whether a given sample is within the model’s training set [44]. Subsequent research extended membership inference to a more practical label-only setting, where the adversary only observes hard-label outputs [45, 46, 47, 48]. In this setting, attacks typically estimate the distance between the input and the target model’s decision boundary: samples farther from the boundary are more likely to be members. However, applying membership inference to generative models is substantially harder because these models lack explicit decision boundaries and produce high-dimensional stochastic outputs. Membership inference against generative models can be categorized based on whether the target data belong to the pre-training corpus or the fine-tuning dataset. Inferring membership in pre-training data is technically difficult because the pre-training corpus is extremely large, making leakage signals weak and noisy [49, 50]. In contrast, inferring membership in fine-tuning data is practically more important, since fine-tuning datasets are typically private or proprietary, whereas pretraining data are often publicly collected [8]. More recently, membership inference has also been extended to RetrievalAugmented Generation (RAG) datastores [51, 52], but these

TABLE XII H YPERPARAMETER S ETTINGS FOR DP-SGD

Text-to-Text Text-to-Image Image-to-Text

Clipping norm 𝐶

Sampling rate 𝐿/𝑁

Noise scale 𝜎

Privacy budget 𝜖

1.0 0.5 1.0

64/20000 4/59143 32/60000

1.0 2.0 1.0

3.0 1.0 3.0

TABLE XIII R ESULTS OF O UR M ETHOD WITH AND WITHOUT D EFENSE

ASR AUC T@1%F

GPT2+Wiki103 Partial Zero Knowl. Knowl.

SD1.5+MSCOCO Partial Zero Knowl. Knowl.

LLaVa+COCO Partial Zero Knowl. Knowl.

0.62/0.88 0.65/0.92 0.20/0.51

0.77/0.84 0.84/0.92 0.38/0.62

0.63/0.89 0.67/0.97 0.24/0.65

0.57/0.83 0.55/0.90 0.09/0.48

0.62/0.78 0.71/0.90 0.19/0.45

With DP-SGD

0.56/0.88 0.60/0.96 0.22/0.93

As shown in Table XIII, introducing DP-SGD during the fine-tuning process can effectively mitigate membership inference attacks. This is because the injected noise limits the influence of individual training samples on the learned model parameters, thereby reducing the distinguishability between member and non-member samples in the model’s output distribution. However, a closer inspection of the qualitative 13

studies mainly focus on LLMs and do not consider other classes of generative models. Existing work on fine-tuned generative models spans multiple modalities, including LLMs [6], diffusion models [7, 32, 53], and vision-language or captioning models [8, 54]. These methods share a common principle: membership is inferred by comparing a target sample (or its output distribution) with a reference set of known non-members and measuring their distance. For example, Fu et al. [6] compare variations in probabilistic representations measured on the target model and a self-prompt reference model; Pang et al. [7] measure similarity between generated and target images; and Hu et al. [8] compare similarity score distributions between target and reference samples. In contrast, membership inference against pre-trained generative models explores different signals and designs. For example, Zhang et al. [55] formulate membership detection as identifying local maxima under maximum-likelihood training. Other approaches rely on token-level semantic properties, such as lower perplexity for member sequences [56, 57] or higher next-token probabilities for member tokens [33].

[4] M. Ko, M. Jin, C. Wang, and R. Jia, “Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Study,” in Proc. of ICCV, 2023, pp. 4871–4881. [5] T. Tran, R. Liu, and L. Xiong, “Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Training,” in Proc. of ACL, 2025, p. 22872–22888. [6] W. Fu, H. Wang, C. Gao, G. Liu, Y. Li, and T. Jiang, “Practical Membership Inference Attacks against Finetuned Large Language Models via Self-prompt Calibration,” in Proc. of NeurIPS, 2024, pp. 134 981 – 135 010. [7] Y. Pang and T. Wang, “Black-box Membership Inference Attacks against Fine-tuned Diffusion Models,” in Proc. of NDSS, 2025. [8] Y. Hu, Z. Li, Z. Liu, Y. Zhang, Z. Qin, K. Ren, and C. Chen, “Membership Inference Attacks Against VisionLanguage Models,” in Proc. of USENIX Security, 2025. [9] D. Zhang, Y. Yu, J. Dong, C. Li, D. Su, C. Chu, and D. Yu, “MM-LLMs: Recent Advances in MultiModal Large Language Models,” in ACL, 2024, p. 12401–12430. [10] Q. Sun, Y. Cui, X. Zhang, F. Zhang, Q. Yu, Z. Luo, Y. Wang, Y. Rao, J. Liu, T. Huang, and X. Wang, “Generative Multimodal Models are In-Context Learners,” in Proc. of CVPR, 2024, p. 14398–14409. [11] OpenAI, “GPT-4 Technical Report,” 2024. [Online]. Available: https://arxiv.org/pdf/2303.08774 [12] F. Croitoru, V. Hondru, R. T. Ionescu, and M. Shah, “Diffusion Models in Vision: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 9, pp. 10 850 – 10 869, 2023. [13] J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-Language Models for Vision Tasks: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5625 – 5644, 2024. [14] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. Summers, “ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases,” in Proc. of CVPR, 2017, pp. 3462–3471. [15] J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping Language-Image Pre-training for Unified VisionLanguage Understanding and Generation,” in Proc. of ICML, 2022. [16] K. Lee, K. Lee, H. Lee, and J. Shin, “A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks,” in Proc. of NeurIPS, 2018. [17] L. van der Maaten and G. Hinton, “Visualizing Data using t-SNE,” Journal of Machine Learning Research, vol. 9, no. 86, pp. 2579–2605, 2008. [18] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners,” 2019. [Online]. Available: OpenAIblog [19] E. Almazrouei and et al., “The Falcon Series of Open Language Models,” 2023. [Online]. Available: https://arxiv.org/pdf/2311.16867

Summary. Although existing membership inference methods are effective within their respective model types, there is currently no unified framework that can generalize across diverse generative models. This paper bridges this gap by proposing a unified framework, based on likelihood ratio testing, that applies to various types of generative models under a common inference paradigm. XI. C ONCLUSION In this paper, we presented a unified study of membership inference attacks against generative models spanning text-totext, text-to-image, and image-to-text modalities. We showed that likelihood-based training drives the output distribution of a generative model to approximate its training data distribution, and leveraged this modality-agnostic property to design a unified membership inference framework. Extensive experiments across diverse generative models and datasets demonstrated that our method achieves performance superior to state-of-theart baselines tailored to individual modalities, while offering broader applicability and robustness. As future work, we plan to extend our study to additional modalities, such as text-tovideo and text-to-audio generative models. R EFERENCES [1] B. Xia, S. Wang, Y. Tao, Y. Wang, and J. Jia, “LLMGA: Multimodal Large Language Model based Generation Assistant,” in Proc. of ECCV, 2024. [2] A. Karamolegkou, M. Nikandrou, G. Pantazopoulos, D. S. Villegas, P. Rust, R. Dhar, D. Hershcovich, and A. Søgaard, “Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users,” in Proc. of ACL, 2025, p. 25949–25982. [3] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership Inference Attacks Against Machine Learning Models,” in Proc. of IEEE S & P, 2017, pp. 3–18. 14

[20] S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer Sentinel Mixture Models,” in Proc. of ICLR, 2017. [21] S. Narayan, S. B. Cohen, and M. Lapata, “Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization,” in Proc. of EMNLP, 2018, p. 1797–1807. [22] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a Distilled Vrsion of BERT: Smaller, Faster, Cheaper and Lighter,” in Proc. of NeurIPS, 2019. [23] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” in CVPR, 2022, pp. 10 684– 10 695. [24] C. Schuhmann and et al., “LAION-5B: An open largescale dataset for training next generation image-text models,” in Proc. of NeurIPS, 2022. [25] T. Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick, “Microsoft COCO: Common Objects in Context,” in Proc. of ECCV, 2014, p. 740–755. [26] Y. Jiang, Z. Huang, X. Pan, C. C. Loy, and Z. Liu, “Talkto-Edit: Fine-Grained Facial Editing via Dialog,” in Proc. of ICCV, 2021, pp. 13 799–13 808. [27] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual Instruction Tuning,” in Proc. of NeurIPS, 2023. [28] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models,” in Proc. of ICLR, 2024. [29] “all-MiniLM-L6-v2,” 2024. [Online]. Available: https://huggingface.co/sentence-transformers/ all-MiniLM-L6-v2 [30] Z. Wang, C. Zhang, Y. Chen, N. Baracaldo, S. R. Kadhe, and L. Yu, “Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble,” in Proc. of ACM CCS, 2025, pp. 1724–1738. [31] Z. Lu, H. Chi, N. Baracaldo, S. R. Kadhe, Y. Jeon, and L. Yu, “In-Context Probing for Membership Inference in Fine-Tuned Language Models,” in Proc. of NDSS Symposium, 2026. [32] S. Zhai, H. Chen, Y. Dong, J. Li, Q. Shen, Y. Gao, H. Su, and Y. Liu, “Membership Inference on Textto-Image Diffusion Models via Conditional Likelihood Discrepancy,” in Proc. of NeurIPS, 2024. [33] Z. Li, Y. Wu, Y. Chen, F. Tonin, E. A. Rocamora, and V. Cevher, “Membership Inference Attacks against Large Vision-Language Models,” in Proc. of NeurIPS, 2024. [34] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” 2019. [Online]. Available: http://arxiv.org/ abs/1907.11692 [35] B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda, “Visual transformers: Token-based image representation and processing for computer vision.” [Online]. Available: https://arxiv.org/abs/2006.03677 [36] “Model card: Clip.” [Online]. Available: https:

//huggingface.co/openai/clip-vit-base-patch32 [37] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” 2019. [Online]. Available: https://arxiv.org/abs/1909.11942 [38] “all-mpnet-base-v2.” [Online]. Available: https:// huggingface.co/sentence-transformers/all-mpnet-base-v2 [39] “Guided-diffusion.” [Online]. Available: https://github. com/openai/guided-diffusion [40] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep Learning with Differential Privacy,” in Proc. of CCS, 2016. [41] B. Q. Tran, V. Nguyen, A. Tran, and T. Tran, “DualModel Defense: Safeguarding Diffusion Models from Membership Inference Attacks through Disjoint Data Splitting,” 2025. [Online]. Available: https://arxiv.org/ pdf/2410.16657 [42] M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning,” in Proc. of IEEE S & P, 2019, pp. 739–753. [43] J. Ye, A. Maddi, S. K. Murakonda, V. Bindschaedler, and R. Shokri, “Enhanced Membership Inference Attacks against Machine Learning Models,” in CCS, 2022. [44] H. Hu, Z. Salcic, L. Sun, G. Dobbie, P. S. Yu, and X. Zhang, “Membership Inference Attacks on Machine Learning: A Survey,” ACM Computing Surveys (CSUR), vol. 54, no. 11s, pp. 235:1–37, 2022. [45] C. A. Choquette-Choo, F. Tramer, N. Carlini, and N. Papernot, “Label-Only Membership Inference Attacks,” in Proc. of ICML, 2021, pp. 1964–1974. [46] Z. Li and Y. Zhang, “Membership Leakage in Label-Only Exposures,” in Proc. of CCS, 2021. [47] Y. Wu, H. Qiu, S. Guo, J. Li, and T. Zhang, “You Only Query Once: An Efficient Label-Only Membership Inference Attack,” in Proc. of ICLR, 2024. [48] H. Li, Z. Li, S. Wu, Y. Ye, M. Zhang, D. Feng, and Y. Zhang, “Enhanced Label-Only Membership Inference Attacks with Fewer Queries,” in Proc. of USENIX Security, 2025, pp. 5465–5483. [49] M. Duan, A. Suri, N. Mireshghallah, S. Min, W. Shi, L. Zettlemoyer, Y. Tsvetkov, Y. Choi, D. Evans, and H. Hajishirzi, “Do Membership Inference Attacks Work on Large Language Models?” in Proc. of COLM, 2024. [50] H. Puerto, M. Gubri, S. Yun, and S. J. Oh, “Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models,” in Proc. of NAACL, 2025, p. 4165–4182. [51] X. Gao, X. Meng, Y. Dong, Z. Li, and S. Guo, “DCMI: A Differential Calibration Membership Inference Attack Against Retrieval-Augmented Generation,” in Proc. of ACM CCS, 2025, pp. 4184–4198. [52] A. Naseh, Y. Peng, A. Suri, H. Chaudhari, A. Oprea, and A. Houmansadr, “Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generation,” in Proc. of ACM CCS, 2025, pp. 1245–1259. [53] M. Zhang, N. Yu, R. Wen, M. Backes, and Y. Zhang, 15

“Generated Distributions Are All You Need for Membership Inference Attacks Against Generative Models,” in Proc. of WACV, 2024, pp. 4827–4837. [54] D. Samira, E. Habler, Y. Elovici, and A. Shabtai, “Variance-Based Membership Inference Attacks Against Large-Scale Image Captioning Models,” in Proc. of CVPR, 2025, pp. 9210–9219. [55] J. Zhang, J. Sun, E. Yeats, Y. Ouyang, M. Kuo, J. Zhang, H. F. Yang, and H. Li, “Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models,” in Proc. of ICLR, 2025. [56] Y. He, B. Li, L. Liu, Z. Ba, W. Dong, Y. Li, Z. Qin, K. Ren, and C. Chen, “Towards Label-Only Membership Inference Attack against Pre-trained Large Language Models,” in Proc. of USENIX Security, 2025. [57] H. Chang, A. S. Shamsabadi, K. Katevas, H. Haddadi, and R. Shokri, “Context-Aware Membership Inference Attacks against Pre-trained Large Language Models,” in Proc. of EMNLP, 2025, p. 7299–7321. I. D ESCRIPTIONS OF DATASETS Wikitext-103 is a large-scale corpus containing over 100 million tokens extracted from verified Good and Featured Wikipedia articles, comprising 28,475 entries. It is widely used for training text generation models and for fine-tuning models on sequence completion and language modeling tasks. XSum, short for Extreme Summarization, consists of 226,711 BBC news articles archived via Wayback from 2010 to 2017, covering a broad range of domains, including News, Politics, Sports, Weather, Business, Technology, Science, Health, Family, Education, and Entertainment. Each article is paired with a single-sentence summary, making the dataset particularly suitable for abstractive summarization tasks. MS COCO is a large-scale dataset for object detection, segmentation, and image captioning, containing over 330k images with more than five human-written captions per image. It is widely used for training and evaluating vision–language models, particularly for tasks such as image captioning and text-to-image or image-to-text generation. CelebA-Dialog is a large-scale vision–language face dataset comprising 30,000 high-resolution facial images and 202,599 original images, each paired with detailed textual annotations describing facial attributes. It is broadly used for training and evaluating vision–language models. COCO 2017 is a split of the MS COCO dataset, containing over 160,000 images that depict complex everyday scenes with multiple objects and rich contextual interactions. It is widely used for training vision–language models across tasks such as image captioning and multimodal generation. CC SBU ALIGN is a web-crawled vision–language dataset containing roughly 12 million image–text pairs formed by merging Conceptual Captions and SBU captions. It is commonly used for pretraining multimodal models for image–text alignment and image-to-text generation.

Record · ID 343386 · SHA-256 2fdf478f58eb33ed
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.