Open Models, Open Risks: Measuring Unsafe Generation in Text-to-Image Models In the Wild Peilin Han
Yang Liu
Yilong Yang
[email protected] Xidian University China
Xidian University China
Xidian University China
Jingchun Zhang
Teng Li
Jianfeng Ma
Xidian University China
Xidian University China
Xidian University China
Zhuo Ma Xidian University China
arXiv:2607.07827v1 [cs.CR] 8 Jul 2026
Abstract Existing safety studies on text-to-image (T2I) jailbreaks are largely conducted in controlled in-the-lab settings, typically on a small number of canonical models. As a result, the current safety status of the rapidly growing in-the-wild T2I ecosystem remains unclear. This uncertainty is amplified by two factors: existing detector-based metrics are designed for controlled evaluation, and in-the-wild risks may arise not only from adversarial prompting, but also from unsafe release practices and unsafe model derivatives. In this paper, we present a large-scale empirical study of in-thewild T2I safety through the lens of jailbreak. We first show that detector-only jailbreak metrics substantially overestimate practical risk over in the wild due to semantic drift and generation artifacts, and we introduce Advanced ASR to better capture semantically valid and visually plausible unsafe generation. Using this refined metric, we evaluate 200+ in-the-wild T2I models from Hugging Face under three representative jailbreak attacks. Our results show that many downstream models retain a non-trivial degree of safety even without explicit post-hoc safeguards, indicating that safety degradation in the wild is neither universal nor uniform. At the same time, we identify a set of high-risk models, including explicitly NSFW-oriented releases as well as seemingly benign models whose unsafe behavior is only exposed through systematic evaluation. We further trace these models to their release context and report high-risk cases to Hugging Face.
Keywords Text-to-Image Model, Jailbreak, Not Safe For Work, In the Wild
1
Introduction
With the rapid advancement of diffusion-based architectures, multimodal generative models, particularly Text-to-Image (T2I) systems, have been widely adopted in real-world applications[5, 22, 30]. These models are capable of generating high-quality and visually coherent content, leading to a rapidly expanding user base. However, models developed in controlled laboratory environments (i.e., in-the-lab) are typically optimized for general-purpose objectives. Such designs are often insufficient to accommodate diverse and evolving user requirements. In practice, communities exhibit a
strong demand for customization capabilities, including support for specific artistic styles and domain-specific generation tasks[3, 25]. To address this limitation, open model ecosystems have emerged on platforms such as Hugging Face and ModelScope [7, 21]. These platforms lower the barrier to model access and modification, enabling users to fine-tune and redistribute customized T2I models. Models deployed in the wild are often released with weakened, optional, or entirely removed safety mechanisms. However, the relaxation or removal of safety mechanisms exposes new attack surfaces. Among them, jailbreak-based manipulation[4, 9, 16, 28, 29] has emerged as an effective strategy to bypass content safeguards. By crafting specific prompts or conditioning inputs, adversaries can induce T2I models to generate unsafe or Not Safe for Work (NSFW) content that violate predefined safety policies. Our work. In this paper, we systematically study the current safety status of in-the-wild T2I models through the following three research questions: (1) RQ1: In-the-wild Jailbreak Metrics. What is the difference between lab and wild? What limitations arise from these existing evaluation metrics? How to accurately evaluate effective real-world risk? (2) RQ2: Safety Evaluation of in-the-wild T2I Models. Whether these jailbreak attacks remain effective on in-the-wild models? which factors determine model safety under jailbreak attacks? whether newer models exhibit stronger safety properties. (3) RQ3: High-Risk Open-Source T2I Models and Their Traceability Report. Are there intentionally unsafe models in the in-the-wild T2I ecosystem? If so, do they result from deliberate release or from downstream adaptation that weakens safety? How can such models be traced and reported to relevant hosting platforms? To answer RQ1, we analyze how prior work measures jailbreak success and identify the conditions under which existing detectorbased metrics become unreliable in the in-the-wild setting. Based on this analysis, we propose a refined evaluation metric named AASR that distinguishes surface-level detector hits from semantically meaningful unsafe content generation. Our results show that detector-based jailbreak metrics substantially overestimate
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
real-world exploitability, because NSFW detectors tend to flag generated images as unsafe even when they do not satisfy real-world NSFW objectives. To answer RQ2, we conduct a large-scale evaluation of 200 inthe-wild T2I models collected from Hugging Face under a unified experimental setting. We compare three representative jailbreak attacks, examine model safety across architectural families, and analyze temporal trends through model release dates and AASR. Our results show that, under the AASR metric, jailbreak attacks appear considerably less powerful on in-the-wild models. Attacks that achieve high ASR under in-the-lab evaluations often fail to generalize across diverse open-source models, indicating that laboratory results do not reliably reflect real-world risk. We further find that the three jailbreak methods exhibit clear differences in effectiveness. In addition, model safety varies systematically with architectural choices, suggesting that safety properties are influenced by architecture rather than attack sophistication alone. We also observe discernible temporal trends in the safety of in-the-wild T2I models, indicating that model safety evolves over time together with shifts in training practices and community norms within the open-source ecosystem. To answer RQ3, we investigate a subset of models that exhibit unsafe behavior under benign or minimally modified prompts, trace their origins, and analyze the release intent of their publishers. We analyze these models as potential instances of intrinsic unsafety and, where appropriate, document and report evidence of unsafe behavior to relevant model hosting platforms. Our results identify a subset of in-the-wild models that produce policy-violating content without sophisticated jailbreak prompts. We observe two distinct sources of risk in the Hugging Face ecosystem. On the one hand, some users appear to intentionally release unsafe or NSFW-oriented models to the platform. On the other hand, unsafe behavior may also be introduced unintentionally during downstream training, when model developers fail to preserve the safety properties of the base model. Contributions. Our work makes three main contributions. • We conduct a large-scale jailbreak-based safety evaluation of 200 in-the-wild T2I models collected from Hugging Face. Under a unified evaluation pipeline, we compare three representative jailbreak attacks and analyze how safety varies across model families, downstream customization, and release time. To the best of our knowledge, this the first systematic measurements of the current safety status of in-the-wild T2I models, showing that jailbreak observed in lab settings does not accurately reflect risk in the wild. • We show that conventional ASR overestimates practical exploitability because of semantic drift and generation artifacts. To address this problem, we introduce AASR, a refined metric that better captures semantically valid and practically meaningful unsafe generation. • We identify and characterize a class of high-risk models whose unsafe behavior does not depend on jailbreak prompting. The results show two pathways: some models are explicitly released as NSFW checkpoints, while others become high-risk because downstream fine-tuning fails to preserve the safety properties of the base model. By tracing these
Trovato et al.
models and analyzing their behavior, we highlight the need for stronger auditing and governance in T2I ecosystems.
2 Background 2.1 T2I Models Text-to-image (T2I) generation has become a core capability of multimodal systems. Recent diffusion models [14, 15] provide strong image fidelity, semantic alignment, and stylistic diversity, enabling broad use in creative applications. Most current T2I systems are built on latent diffusion architectures [14], which generate images in a compressed latent space for improved efficiency. A typical pipeline includes a denoising network, a text encoder such as CLIP [12] or T5 [13], and, in some cases, safety or filtering modules. Training often relies on large-scale image–text corpora such as LAION [19], where filtering choices influence both generation capability and safety behavior [16]. The open ecosystem is dominated by a small number of influential base models. Stable Diffusion v1.5 [14] remains one of the most widely reused foundations, while Stable Diffusion XL [10] has become another major base model for downstream development. Newer families such as Stable Diffusion 3.5, FLUX [2], and QwenImage [26] further diversify the ecosystem in architecture, training objective, and deployment practice. A detailed review of jailbreak attacks and defenses agonst T2I model is deferred in Appendix.
2.2
Difference between Lab & Wild.
Although the terms in-the-lab and in-the-wild are widely used in prior safety studies, they are often introduced operationally rather than through a shared formal definition. For example, prior work has used in-the-wild to describe jailbreak prompts collected from public prompt-sharing platforms [20] and jailbreak tactics mined from real-world user–chatbot logs [6]. In this paper, we make this distinction explicit. We use in-the-lab to refer to controlled experimental settings, where checkpoints, inference pipelines, and safety mechanisms are fully specified and configurable by the evaluator. By contrast, we use in-the-wild to refer to open-source T2I models publicly released on online platforms, which can be freely downloaded, locally deployed, and modified by users, often without enforced or non-bypassable safety mechanisms. The latter setting is the primary focus of this work. This distinction matters because jailbreak success measured inthe-lab does not reliably predict exploitability in-the-wild. Laboratory evaluations often assess unsafe generation through fixed post-hoc safety classifiers, whereas in-the-wild evaluation more directly reflects the intrinsic safety properties of released models. The threat model is different as well. Laboratory evaluations often assume a white-box or near white-box setting with fixed architectures and training procedures, whereas in-the-wild models are better viewed as a grey-box setting, where derivative checkpoints may involve heterogeneous fine-tuning, architectural modification, or parameter merging. Such variation makes attack transferability less predictable and introduces distinct risks, especially when downstream releases weaken or remove alignment constraints.
Open Models, Open Risks : Measuring Unsafe Generation in Text-to-Image Models In the Wild
3
RQ1: Real-World Jailbreak Metrics
In this section, we revisit how jailbreak success is measured for T2I models and examine whether existing metrics remain valid in the in-the-wild setting.
3.1
Motivation
Most prior studies operationalize jailbreak success through detector activation or filter bypass. This design is practical in controlled in-the-lab evaluations, where the goal is to test whether adversarial prompts can push a model beyond their intended safety boundary. At the same time, prior work has noted that bypass success alone is not sufficient. For example, SneakyPrompt [9] requires adversarial prompts not only to bypass safety filters, but also to preserve the semantics of the original sensitive prompt. This suggests that jailbreak evaluation should capture effective unsafe generation rather than detector activation alone. As discussed in Section 2, the distinction between in-the-lab and in-the-wild settings makes this limitation more consequential in the wild. Detector-based metrics are calibrated for controlled evaluation, but in-the-wild models are more likely to produce detectorpositive outputs that do not correspond to effective unsafe generation. Such outputs may deviate from the intended unsafe objective or be dominated by generation failure. These considerations motivate a closer examination of what detector-positive outputs actually represent in the in-the-wild setting. Our preliminary analysis reveals two recurring sources of such false-success cases, as shown in Figure 1. Two Key Observations. The first is semantic drift. Here, the unsafe prompt produces an output that is flagged as unsafe by the detector, yet fails to preserve the intended unsafe semantics of the prompt. Such an output may still contain detector-triggering cues, but it does not constitute a successful realization of the target unsafe objective. The second is generation artifacts. Here, the detector-positive output is dominated by severe visual distortion, incorrect anatomy, or incoherent structure, rather than meaningful unsafe content. In this case, the unsafe classification is driven primarily by generation failure rather than by recognizable policy-violating content.
This fat bi**h looks like a tank.
Two women with bare buttocks.
Malicious User
Model
Huge Semantic Gap.
Distorted limbs.
Figure 1: Examples of Semantic drift and AI Failure.
3.2
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Advanced ASR
Based on the observations above, we define Advanced Attack Success Rate (Advanced ASR, AASR) for in-the-wild T2I evaluation. AASR counts an output as a successful jailbreak only when it is unsafe, semantically aligned with the adversarial prompt, and visually plausible. Its goal is not to replace detector-based evaluation, but to refine what counts as success when semantic drift and generation artifacts are common. AASR uses a three-stage pipeline. We first apply MHSC as the unsafe-content detector. We then filter detector-positive outputs by prompt–image semantic alignment. Finally, we remove visually corrupted generations using the HADM artifact detector [8]. This design preserves the efficiency of detector-based screening while excluding outputs that are unsafe only at the detector level. 𝑁 , we define Formally, for a set of generated samples {(𝑝𝑖 , 𝐼𝑖 )}𝑖=1 AASR as: 𝑁
AASR =
1 ∑︁ 1[𝐷 nsfw (𝐼𝑖 ) ∧ 𝑆 (𝑝𝑖 , 𝐼𝑖 ) ≥ 𝜏𝑠 ∧ ¬𝐷 art (𝐼𝑖 )] 𝑁 𝑖=1
(1)
where 𝐷 nsfw denotes the NSFW detector, 𝑆 (·, ·) denotes the prompt– image semantic alignment score, 𝜏𝑠 is the semantic consistency threshold, and 𝐷 art denotes the artifact detector. Among the three stages, semantic-drift filtering is the most sensitive to model and prompt variation. We therefore use an adaptive statistical threshold instead of a fixed CLIPScore cutoff. Starting from the MHSC-positive subset, the procedure estimates the empirical CLIPScore distribution and removes three types of unreliable samples: semantically drifting outputs, keyword-dominated matches, and low-diversity generations. We provide the detailed adaptive semantic-drift filtering algorithm in the supplementary material.
3.3
Evaluation
AASR provides a substantially more faithful estimate of real-world jailbreak success than detector-only ASR. To validate this conclusion, we conduct a controlled empirical study designed to quantify false positives in detector-based evaluation. Specifically, we randomly select 18 representative open-source T2I models and evaluate them using the MMA-Diffusion dataset. For each model, we generate 200 images under identical inference settings, resulting in a total of 3,600 evaluated outputs. For every generated image, we record three measurements: the detector-based ASR given by MHSC, the refined AASR after semantic-drift and artifact filtering, and the manually annotated ground-truth success label. This setup allows us to directly compare whether detector-only ASR or Advanced ASR better reflects genuine adversarial success. The observed pattern is clear. As shown in Table 1, detectorbased ASR consistently exceeds the manually annotated groundtruth success rate across all evaluated models, indicating that many detector-positive outputs do not correspond to genuine execution of adversarial intent. By contrast, AASR remains much closer to the ground truth across models. For example, detector-based ASR reaches 0.965 for SD-DS-7, 0.955 for SDXL-MWRI-NSFW, and 0.950 for both SD-DLP-2.0 and SD-EpicReal, while their corresponding AASR values drop to 0.635, 0.665, 0.610, and 0.715, respectively. A similar gap can also be observed for lower-risk models such as
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Trovato et al.
Table 1: AASR vs. ASR on T2I models using MMA-Diffusion. Model SD-DS-7 SD-DS-8 SDXL-DS-Turbo SD-DLP-2.0 SD-EpicReal SD-ExprH SDXL-Fluently4 SDXL-MWRI-NSFW SDXL-NSFW-Gen2 FLUX-Anime-LoRA FLUX-Asian2 FLUX-CutePuss FLUX-Arch FLUX-ArtNouveau FLUX-Lite8B FLUX-Anime2 GEN-Lumina-2.0 SD-OJ-4
GT 0.565 0.49 0.525 0.35 0.595 0.05 0.395 0.38 0.325 0.135 0.155 0.165 0.12 0.08 0.05 0.105 0.135 0.195
ASR 0.965 0.96 0.77 0.95 0.95 0.205 0.92 0.955 0.81 0.33 0.36 0.455 0.375 0.36 0.34 0.29 0.415 0.655
AASR 0.635 0.61 0.675 0.61 0.715 0.15 0.685 0.665 0.54 0.285 0.305 0.31 0.31 0.31 0.3 0.265 0.255 0.36
GTCLIP 0.32 0.28 0.18 0.33 0.28 0.29 0.315 0.325 0.335 0.38 0.415 0.435 0.385 0.31 0.31 0.35 0.295 0.325
CLIPScore 0.16 0.14 0 0.15 0.16 0.15 0.14 0.17 0.15 0.17 0.16 0.19 0.16 0.145 0.145 0.175 0.145 0.15
FLUX-Anime2 and FLUX-Lite8B, where detector-based ASR values of 0.290 and 0.340 are reduced to 0.265 and 0.300 after refinement. We further analyze the contribution of semantic drift to ASR inflation. The semantic drift detection rate closely matches the proportion of ground-truth false positives, indicating that a large fraction of detector-positive outputs are semantically misaligned with the input prompt. This confirms that semantic drift is a primary source of overestimation in detector-based metrics. These findings provide empirical evidence for the structural issues discussed in Research Question 1. In-the-wild models, due to heterogeneous fine-tuning practices and noisy training data, are more prone to semantic drift and generation artifacts. Detector-based ASR fails to distinguish these failure modes from genuine unsafe generation, leading to systematic overestimation of model vulnerability.
4
RQ2: Overall Safety Evaluation
In this section, We first conduct a large-scale safety evaluation on 200 in-the-wild T2I models using three jailbreak datasets, so as to obtain an overall picture of their safety status. We then analyze model safety from four complementary perspectives: attack-conditioned safety, architecture-level variation, inheritance of model safety, and temporal evolution.
4.1
Evaluation Setup
Jailbreak datasets. We use three representative jailbreak datasets from prior work: Unsafe Diffusion Template Prompts (UDTP), 4chan, and MMA-Diffusion (MMA). UDTP consists of manually designed unsafe prompt templates, 4chan contains malicious prompts collected from online discussions, and MMA provides a diverse set of harmful prompts for multimodal safety evaluation. Together, they cover complementary attack conditions: curated unsafe prompts, naturally occurring malicious prompts, and adaptive harmful prompts. Since UDTP contains 30 prompts, we randomly sample 30 prompts from both 4chan and MMA and keep the sampled prompts fixed across all models. Our goal is large-scale and comparable safety measurement rather than per-model attack maximization. We therefore do not use target-specific iterative methods such as STEPS or FGPI, since
Table 2: Twenty representative results selected from the overall safety status table. The complete table is provided in Appendix. Model SDXL-SDXL SD-Turbo SDXL-Turbo Qwen-NSFW SDXL-NSFW-Uncens SD-CleanMix-NSFW FLUX-Asian2 SDXL-RV5 FLUX-Logo-LoRA FLUX-Realism-LoRA GEN-ScandiInterior FLUX-NSFW-Master SDXL-RV5-Lightning SDXL-Albedo13 FLUX FLUX-Turbo-Alpha FLUX-Anime-LoRA SDXL-WAI-80 SDXL-JankuV5 FLUX-SN2
UDTP ASR AASR 0.83 0.07 0.80 0.07 0.67 0.10 0.77 0.17 0.50 0.23 0.53 0.17 0.83 0.80 0.87 0.73 0.77 0.73 0.80 0.70 0.77 0.70 0.83 0.67 0.87 0.63 0.87 0.57 0.73 0.67 0.83 0.63 0.70 0.67 0.50 0.43 0.60 0.47 0.65 0.43
MMA ASR AASR 0.40 0.03 0.47 0.00 0.5 0.10 0.87 0.20 0.83 0.57 0.53 0.33 0.37 0.37 0.83 0.57 0.37 0.33 0.30 0.27 0.27 0.23 0.77 0.63 0.60 0.53 0.90 0.60 0.00 0.00 0.30 0.23 0.37 0.23 0.73 0.67 0.83 0.60 0.80 0.67
they require per-model prompt search and would introduce heterogeneous optimization budgets across models. We also exclude methods designed for black-box commercial systems, such as jailbreaks against DALL·E guardrails, because they do not match our locally deployed open-source threat model. Instead, UDTP, 4chan, and MMA provide a standardized and practical benchmark for evaluating diverse in-the-wild open-source models. Model selection. Using these criteria, we obtain a final set of 200 in-the-wild T2I models. The selected models cover major families and adaptation patterns, including derivatives of Stable Diffusion 1.5, Stable Diffusion XL, FLUX, and Qwen-Image. Simplified model names are used in the main text and figures for readability, while full repository names are provided in Appendix. Evaluation metrics. Model safety is measured primarily by AASR. A generated image is counted as a successful jailbreak only if it is classified as unsafe, remains semantically aligned with the input prompt, and passes artifact-based quality filtering. We also report detector-only ASR and the proportions of outputs removed by semantic-drift and artifact filtering to explain the gap between detector-based and refined evaluation.
4.2
Overall Safety Evaluation & Analyze
We first evaluate the collected 200+ models using the three jailbreak datasets described above, and report each model’s NSFW Rate and AASR under every attack setting. For ease of presentation, we use ASR to denote the NSFW Rate in the remainder of this paper. Table 2 presents 20 representative results under the Unsafe Diffusion jailbreak prompts, with each entry reporting both ASR and AASR for the corresponding model. For readability, we show only a representative subset in the main text; the complete table, together with the corresponding results under the other two jailbreak datasets, is provided in Appendix. Even within this representative subset, the safety status of inthe-wild T2I models is highly heterogeneous. Under AASR, model
Open Models, Open Risks : Measuring Unsafe Generation in Text-to-Image Models In the Wild
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY 7 0 %
vulnerability spans a wide range, from near zero to 0.80. This dispersion indicates that real-world safety cannot be inferred from the mere presence or absence of explicit safety modules.
M M A U n sa fe 4 C h a n
6 0 % 5 0 % 4 0 % 3 0 % 2 0 %
Finding 1. Some in-the-wild T2I models remain resistant to jailbreak attacks even without explicit safety alignment.
Under the AASR, certain models maintain relatively low unsafe generation rates across multiple prompt sources, suggesting that safety robustness can emerge implicitly from architectural design choices, data curation practices, or conservative fine-tuning objectives, rather than solely from dedicated alignment interventions. This implicit robustness is obscured by detector-based ASR for many models. For several checkpoints, ASR remains high while AASR drops sharply after filtering semantic drift and generation artifacts. Representative examples include SDXL (0.83 → 0.07), SDTurbo (0.80 → 0.07), SDXL-Turbo (0.67 → 0.10), and Qwen-ImageNSFW (0.77 → 0.17). These cases indicate that a substantial fraction of detector-positive outputs in the wild do not correspond to semantically faithful and practically exploitable unsafe generations, but are instead driven by semantic drift or visually degraded outputs. As a result, detector-only ASR may create the impression that such models are highly vulnerable, whereas AASR reveals that some of them retain meaningful resistance to real-world jailbreak attempts. At the same time, the table also highlights a set of models whose AASR remains high even after refinement, indicating persistent and coherent unsafe generation capabilities. Examples include FLUXAsian2 (0.80), SDXL-RV5 (0.73), FLUX-Logo-LoRA (0.73), FLUXRealism-LoRA (0.70), GEN-ScandiInterior (0.70), and FLUX-NSFWMaster (0.67). For these models, the relatively small gap between ASR and AASR suggests that unsafe outputs are not merely detectortriggered false positives, but remain semantically aligned and visually plausible after filtering. Such models are therefore of particular concern, as their unsafe behavior is more likely to translate into real-world exploitability. A second important observation is that safety differs systematically across architectural lineages, while also varying substantially within the same architectural lineages. For example, SDXL-derived models range from relatively low-risk checkpoints such as SDXL and SDXL-Turbo to highly vulnerable derivatives such as SDXLRV5, SDXL-RV5-Lightning, and SDXL-Albedo13. A similar pattern appears in the FLUX architecture, where some derivatives retain high AASR (e.g., FLUX, FLUX-Turbo-Alpha, FLUX-Anime-LoRA), while others are substantially lower. This within-architecture variance indicates that downstream customization can significantly reshape safety behavior, even when models inherit from the same base checkpoint. A more in-depth investigation into the security properties across different model architectures and the inheritance of safety under model fine-tuning is deferred to 4.4.
Finding 2. High-risk models exist in the in-the-wild T2I ecosystem.
1 0 % 0 % (R a tio )
[ 0 ,0 .2 )
[ 0 .2 ,0 .4 )
[ 0 .4 ,0 .6 )
[ 0 .6 ,0 .8 )
[ 0 .8 ,1 ]
Figure 2: Distribution of model AASR under three jailbreak attacks. The x-axis shows AASR bins. The y-axis shows the proportion of models falling into each bin.
During our evaluation, we identify two broad categories of highrisk models in the in-the-wild ecosystem. The first category consists of models that are explicitly positioned as NSFW-oriented through their names, model cards, or repository descriptions. Several such models indeed exhibit high AASR, such as FLUX-Asian2, Flux-NSFW-uncensored, and FLUX-NSFW-Master, indicating that their unsafe behavior is consistent with intentional optimization toward explicit content generation. However, not all explicitly labeled NSFW-oriented models remain highly risky under the refined metric. Some such models show much lower AASR, including Qwen-Image-NSFW, FLUX-NSFWUncensored, and SD-CleanMix-NSFW. This suggests that explicit NSFW positioning does not uniformly translate into strong and practically exploitable unsafe generation capability. The second category is more subtle and potentially more concerning: models that appear benign from their names or intended use cases, yet still exhibit elevated AASR. These include realismoriented, portrait-focused, or style-specialized derivatives whose naming and documentation do not explicitly indicate NSFW intent. For such models, high AASR are more likely to reflect insufficient preservation of safety-aligned behavior during downstream training or fine-tuning, rather than overt optimization toward explicit content. In other words, unsafe behavior in the wild may arise not only from intentionally NSFW-optimized models, but also from seemingly benign models whose customization process weakens inherited safety properties. A more detailed investigation of these high-risk models and their traceability reports is provided in RQ3. To provide a more comprehensive characterization of the current safety landscape of in-the-wild models, we next examine it from four perspectives: jailbreak attack schemes, architectural families, safety inheritance, and temporal trends.
4.3
Attack-conditioned Safety
Finding 3. In-the-wild T2I safety is attack-dependent and category-inconsistent across jailbreak methods. Figure 2 illustrates the AASR distributions of three representative jailbreak methods across all 200 in-the-wild models. A clear pattern emerges: UDTP poses the greatest security threat overall, followed by MMA, while 4chan exhibit the lowest threat.
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Trovato et al.
5 0 % 3 3 % 1 7 % 0 % (A A S R )
F L U X
Q W E N
S D
S D X L
Figure 3: AASR distributions of four model families under the MMA-Diffusion jailbreak attack. Each box shows the median, interquartile range, and 1.5×IQR whiskers of modellevel AASR within one architecture.
A plausible explanation lies in prompt design: Unsafe Diffusion uses carefully constructed jailbreak templates that induce unsafe outputs while preserving semantic fidelity. More importantly, UDTP covers multiple categories of policy-violating content, including sexual, violent, disturbing, and political content. As a result, it exposes broader safety weaknesses and yields consistently higher AASR than MMA. To further validate the claim that in-the-wild T2I safety is category-inconsistent, we provide a category-level breakdown of unsafe outputs in Appendix. This result suggests that the safety behavior of in-the-wild models is not uniform across different categories of unsafe content. In particular, many models show stronger resistance to sexually explicit content than to other policy-violating categories. This pattern indicates that safety mechanisms in the wild are often more effective for some unsafe content types than for others. Such category-level inconsistency highlights the need to evaluate and align model safety across multiple forms of unsafe generation, rather than focusing too narrowly on a single dominant category. In contrast, 4Chan yields the lowest AASR. This is likely because its prompts, collected from malicious forum comments, are often less structured and less compatible with text-to-image generation, making them more likely to produce semantic drift or low-quality outputs that are filtered by AASR.
4.4
Architecture-level Variation
Finding 4. Architecture is a major source of safety variation in in-the-wild T2I models, and vulnerability can still differ substantially within the same model family. Figure 3 compares the distribution of AASR across FLUX, QWEN, SD, and SDXL models under the MMA-Diffusion jailbreak attack. A clear separation can be observed among the four families. SDXL exhibits the highest overall risk, with the largest median AASR (0.44) and a broad upper range extending above 0.60, indicating that SDXL models are not only more vulnerable on average, but also frequently include highly susceptible checkpoints. By contrast, QWEN shows the lowest overall AASR distribution, with a much lower median (around 0.15) and most values concentrated in the lower range, suggesting comparatively stronger resistance to this attack.
4.5
Inheritance of Model Safety
A A S R
6 7 %
SD occupies an intermediate position: its median AASR is around 0.27, and the interquartile range is relatively compact, which suggests that SD-based models are moderately vulnerable but behaviorally more consistent. FLUX is more heterogeneous. Although its median AASR is lower than that of SD, its distribution is substantially wider, spanning from near-zero values to very high AASR cases. This indicates that vulnerability within the FLUX architecture is highly uneven, with both relatively robust and highly unsafe models coexisting.
A A S R
8 3 %
A A S R
1 0 0 %
M o d e l
M o d e l
M o d e l
(a)
(b)
(c)
Figure 4: Security analysis of models under the same inheritance chain. More Results are detailed in Appendix. To further analyze safety inheritance in in-the-wild models, we construct representative model inheritance lines and track how AASR evolves across successive checkpoints. Finding 5. Model safety exhibits clear inheritance: unsafe tendencies can be preserved, amplified, or reshaped along downstream model lineages. Figure 4 shows that model safety exhibits a clear lineage-dependent pattern. In general, downstream models often inherit the safety baseline of their upstream checkpoints, but this inheritance is not fixed. It can be strengthened, weakened, or substantially reshaped by later fine-tuning objectives, training data, and adaptation strategies. The retained inheritance examples reveal two recurring behaviors. First, some downstream branches show substantial increases in AASR relative to their upstream checkpoints, indicating that fine-tuning can significantly weaken inherited safety properties. Second, other branches remain relatively stable or even show lower AASR, suggesting that different downstream objectives can lead to very different safety trajectories even when models originate from related bases. The inheritance curves also suggest clear path dependence. In some model lineages, elevated risk persists across successive versions rather than disappearing in later derivatives. This indicates that once unsafe tendencies are introduced into a lineage, they may be preserved or further amplified by subsequent downstream customization.
4.6
Temporal Trends
To study how safety evolves over time in the in-the-wild ecosystem, we collect model release dates from Hugging Face repositories and use AASR under the three jailbreak datasets as the safety indicator.
Open Models, Open Risks : Measuring Unsafe Generation in Text-to-Image Models In the Wild