arXiv:2606.09670v1 [cs.CV] 8 Jun 2026
V ISUAL P ROMPTING M EETS F EATURE R ECONSTRUCTION -BASED A NOMALY D ETECTION WITH D UAL -T EACHER S UPERVISION
Mateo Diaz-Bone, Daniel Caraballo, Florian Scheidegger, Thomas Frick, Mattia Rigotti, Andrea Bartezzaghi, Roy Assaf, Niccolo Avogaro, Yagmur G. Cinar, Brown Ebouky, Filip M. Janicki, Piotr S. Kluska, Cezary Skura, Cristiano Malossi IBM Research Europe Zurich, Switzerland [email protected], [email protected]
June 9, 2026
A BSTRACT Recent Anomaly Detection methods achieve perfect detection and segmentation scores on wellestablished datasets, such as MVTec. However, many of these methods face challenges when foundational assumptions — such as consistent object scale, viewpoint, background, illumination, and centered placement — are violated. Those variations that occur render anomaly detection methods unusable in many real-world scenarios. To address these limitations, we introduce three key contributions: (1) a visual prompting pipeline that isolates objects using foreground–background masking; (2) a mechanism for unfreezing the teacher in student–teacher models to improve domain adaptability; and (3) a data augmentation strategy leveraging diffusion-generated synthetic images to enhance anomaly detection performance. We achieve a 3.5 percentage point improvement over the previous state-of-the-art on the challenging AeBAD dataset by using the Masked Multiscale Reconstruction (MMR) model as our backbone.
1
Introduction
Unsupervised anomaly detection involves identifying and localizing anomalies without prior knowledge of abnormal patterns. Since the release of the first benchmark datasets in the late 2010s Carrera et al. [2017], Bergmann et al. [2019], anomaly detection has received growing attention due to its broad range of applications in industrial contexts (IAD). However, as these datasets are often assembled in controlled environments, they frequently fail to capture real-world complexities, thereby limiting the practical deployment of IAD methods. A closer look at the MVTec dataset Bergmann et al. [2019] reveals that the images and their contained objects exhibit notable consistency: objects are typically aligned, centered, similarly scaled, and placed against homogeneous backgrounds. The images are also systematically captured from a fixed viewpoint under uniform illumination. While these characteristics may be enforceable in controlled industrial scenarios, numerous applications lack the ability to maintain such a sterile setup. Recent anomaly detection methods report near-perfect detection and segmentation performance on MVTec with up to 99.9% image level AUROC Chen et al. [2024] and 99.3% pixel level AUROC Li et al. [2024]. However, much of this success relies on the dataset’s consistency. Once these underlying assumptions are violated, performance drops significantly, as demonstrated in prior work Zhang et al. [2023a] and confirmed by our own experiments. This reliance on dataset-specific properties limits the real-world applicability of many IAD methods. To address these challenges, we propose three improvements for student-teacher reconstruction-based methods:
Visual Prompting Meets Feature Reconstruction-Based AD
Figure 1: Original and distorted MVTec Bergmann et al. [2019] samples for the bottle category. We used the VP pipeline to produce human-in-the-loop annotated masks. In our distorted version the background is replaced with a textured background. • Visual Prompting Masks Pipeline: To reduce sensitivity to background variation, we design a pipeline that leverages auto-generated foreground/background masks to isolate objects and eliminate unwanted background anomalies. Applied as a postprocessing step, this enhances model robustness in the resulting scoremaps. • Dual Teacher Approach: We propose a method to unfreeze the pre-trained frozen teacher in student-teacher reconstruction-based methods without suffering from model/ feature collapse. This increases the model’s adaptability to target domains and mitigates pre-training bias. • Synthetic Training Data: We augment the training dataset with synthetic images obtained from a diffusion model. Contrary to recent methods, we generate good images. This increases the training data diversity while at the same time acting as a regularization mechanism to prevent the model from overfitting, thereby increasing its generalization capabilities. The remainder is organized as follows. Section 2 reviews related work. Section 3 discusses a distorted MVTec version to assess background sensitivity. Section 4 details our proposed improvements. We present experiments, ablations, and results in Section 5 and conclude in Section 7.
2
Related Work
Anomaly detection (AD) for visual data has gained significant attention, particularly in unsupervised anomaly detection (UAD) settings where models must perform without access to defective samples during training. Traditional datasets like MVTec AD Bergmann et al. [2019] and VisA Zou et al. [2022] have been extensively used to benchmark models in this field. However, these datasets often assume strong spatial alignment and lack the variability needed to assess model robustness under domain shifts. To address this limitation, the AeBAD Zhang et al. [2023a] dataset was introduced, offering diverse domain shifts across defect categories and providing a more comprehensive evaluation of AD models under real-world conditions with varying distributions. The MMR model Zhang et al. [2023a] was proposed to handle such domain shifts. MMR utilizes multi-scale features for reconstructing normal patterns, effectively detecting anomalies by highlighting reconstruction errors. Despite its strengths, MMR still relies on a frozen pre-trained encoder, limiting its adaptability to new domains. 2
Visual Prompting Meets Feature Reconstruction-Based AD
Further, AD methodologies have predominantly relied on one-class learning, where models are trained exclusively on normal samples to generalize to unseen defective cases during inference. Memory-based models, such as PatchCore Roth et al. [2022], GraphCore Xie et al. [2023], and FAPM Kim et al. [2023], store nominal features for later comparison, achieving impressive results. While effective, these methods may not adapt well under significant domain shifts due to their reliance on stored normal features. To enhance feature space separation and improve discriminative power, contrastive learning and self-supervised techniques have been employed in AD tasks Zavrtanik et al. [2021a], Li et al. [2021], Yang et al. [2023] where methods like ReConPatch Hyun et al. [2024] encourage models to learn more robust and discriminative representations, facilitating better AD. However, they often require careful design of pretext tasks and may still struggle when anomalies exhibit significant differences from normal data. Knowledge distillation techniques, where a frozen teacher model guides a student in learning representations, are widely used Zhang et al. [2023b]. Methods like Reverse Distillation Deng and Li [2022] and EfficientAD Batzner et al. [2024] ensure robust feature extraction but limit domain adaptability due to the frozen teacher. Our approach addresses this by unfreezing the teacher within a dual-teacher framework, enhancing adaptability while preventing feature collapse. Concurrent works such as Recontrast Guo et al. [2023] attempt a related type of supervision which is specific to the Reverse Distillation framework. This makes it less flexible to incorporate into other distillation frameworks. Recently, diffusion models Ho et al. [2020], Rombach et al. [2022] have been used to generate synthetic anomalies, turning unsupervised AD into a supervised task Zhang et al. [2024], Wyatt et al. [2022]. However, creating relevant synthetic defects and integrating them without bias remains challenging. Instead, we synthesize high-quality normal images to enhance dataset diversity in the target domain. While isolating objects from the background can improve detection, it has remained largely underexplored in UAD. As far as we know, the only related work Baugh et al. [2023], tiles multi-object images into single objects through foreground extraction. We extend this field by using Visual Prompting (Section 4.3) to create segmentation masks that enhance anomaly detection.
3
A Distorted Version of MVTec
The CC BY-NC-SA 4.01 licensed MVTec Bergmann et al. [2019] remains a key AD benchmark but relies on controlled, aligned samples, limiting real-world applicability. Inspired by Fashion MNIST Xiao et al. [2017] as a drop-in replacement for MNIST Deng [2012], we introduce a distorted MVTec variant that retains the same splits, defect categories, and masks, but replaces backgrounds with randomly sampled textures from high-resolution, CC-licensed2 images stemming from the public domain. Details about the data are in Appendix E. Figure 1 shows examples from the bottle category, including human-validated segmentation masks that help create more realistic blended samples. We evaluate popular AD methods using anomalib3 Akcay et al. [2022] on both original and distorted MVTec. Performance drops by 1.7%-points up to 16%-points on the distorted dataset, with background noise especially noticeable in image corners, sometimes altering the ranking of algorithms. Table 1 quantifies these effects and confirms the necessity of reevaluating AD methods in less controlled conditions. Additional visual examples comparing the performance of different algorithms are provided in Appendix A. Our proposed two-stage pipeline, where visual prompting (VP) identifies the problem-specific region of interest, followed by a more traditional AD pipeline, mitigates this limitation.
4
Method
In this section, we present our novel framework for anomaly detection, which features synthetic data generation, a knowledge-preserving dual-teacher approach, and visual prompting. Our framework is architecture-agnostic and can be applied to most knowledge-distillation-based reconstruction methods. An overview of the proposed pipeline is illustrated in Figure 2. The pipeline is divided into three phases: Preparation. Before training begins, the original training data is used to train a diffusion model, which is then used to sample synthetic images that resemble the original data (Section 4.1). In parallel, the user selects a small number 1
https://creativecommons.org/licenses/by-nc-sa/4.0/ https://creativecommons.org/publicdomain/ 3 https://github.com/openvinotoolkit/anomalib 2
3
Visual Prompting Meets Feature Reconstruction-Based AD
Algorithm Data
Ref
Fre Dist
Padim Ref Dist
PatchCore Ref Dist
RD Ref Dist
Uflow Ref Dist
bottle cable capsule hazelnut metal_nut pill screw toothbrush transistor zipper
97.4 97.2 97.8 98.3 97.3 96.3 96.3 98.1 98.5 95.5
88.1 89.8 70.7 93.3 80.8 68.4 66.4 84.2 89.5 77.0
97.8 95.1 98.2 96.9 95.0 95.5 97.7 98.7 97.0 96.8
98.2 98.1 98.8 98.5 98.2 97.6 98.9 98.6 97.1 97.9
98.7 96.5 98.8 98.7 96.6 97.2 99.4 99.0 91.3 97.9
95.7 98.4 98.8 99.1 97.7 99.2 99.3 98.0 97.7 97.1
mean Change
97.3 80.8 -16.4
97.8 92.8 87.3 93.5 86.8 86.8 77.1 76.5 89.4 95.1
96.9 88.3 -8.6
96.1 95.3 81.2 97.2 91.3 85.2 84.3 89.8 77.9 92.5
98.2 89.1 -9.1
90.5 87.2 69.0 91.0 77.7 63.1 82.6 91.4 73.6 82.5
97.4 80.9 -16.6
97.4 97.1 98.0 98.9 96.4 98.6 93.5 97.7 86.6 98.4
98.1 96.3 -1.8
MMR Ref Dist 98.67 95.82 97.75 98.98 95.24 98.55 99.49 98.63 91.27 97.95
93.17 84.85 65.01 94.76 62.78 73.42 82.85 88.89 73.61 85.33
97.24 80.47 -16.7
Table 1: Performance gap between the reference and distorted versions on the MVTec dataset. All reference results are reproduced with default setups for a fair comparison. The background distortion leads to a challenging scenario that negatively impacts traditional AD algorithms. References: Fre Ndiour et al. [2022], Padim Defard et al. [2021], PatchCore Roth et al. [2022], Reverse Distillation (RD) Deng and Li [2022], Uflow Tailanian et al. [2024], and MMR Zhang et al. [2023a]. of random images from the training set and creates foreground/background (FG/BG) masks using Visual Prompting (Section 4.3). Training. During training, the model learns from a combination of real and synthetic training data. Inference. During inference, the test image is passed through the model to generate an intermediate score map. In parallel, the test image is also processed by the Matcher (Section 4.3), which uses the reference FG/BG masks from the preparation phase to automatically generate a FG/BG mask for the test image. This mask is then applied to the score map to produce the final output. 4.1
Synthetic Train Data via Diffusion Models
Inspired by RealNet Zhang et al. [2024], we employ a pre-trained diffusion model to generate synthetic training data. By fine-tuning the diffusion model on the target domain’s training set, we sample synthetic images that closely resemble the original good training images in both appearance and context. These synthetic samples are then added to the training set alongside the real images, enriching it with diverse, domain-relevant examples. The inherent stochasticity of the diffusion process further enhances data diversity, enabling the model to better capture the domain’s data distribution. Additionally, the synthetic images act as a form of regularization, mitigating overfitting to the original training data and improving generalization. 4.2
Knowledge-Preserving Dual Teacher Approach
Knowledge Distillation (KD) based student-teacher feature reconstruction architectures have received much attention in recent years due to their outstanding performances. Figure 3 (left) depicts a common configuration as featured in MMR Zhang et al. [2023a]. In this method, the student tries to reconstruct the features obtained from a frozen teacher network which was pre-trained on a large-scale dataset like Imagenet Deng et al. [2009]. However, AD images often differ significantly from the natural images used in large-scale pre-training datasets, making the pre-trained encoder features less suitable for the target domain and restricting performance. Since the teacher encoder is typically frozen, it cannot adapt these features to the target domain. To circumvent this structural limitation we present a method to unfreeze the teacher. If one naively unfreezes the teacher encoder, the model would inevitably suffer from feature collapse as it rapidly converges to a trivial solution to minimize the student teacher distance. To prevent this, we introduce a secondary teacher which acts as a regularization for the unfrozen teacher. Figure 3 (right) depicts our proposed modified architecture. We denote the unfrozen teacher as ’weak teacher’ and the supervisory teacher as ’strong teacher’. Both teachers are instantiated with the same pre-trained weights. To control the level of flexibility of the weak teacher we modify the loss function. Ltotal = LS_WT + λLWT_ST (1) 4
Visual Prompting Meets Feature Reconstruction-Based AD
Figure 2: Overview of our proposed framework.
Figure 3: Original and modified version of student-teacher feature reconstruction (MMR-style).
Here LS_WT denotes the KD loss between the student and the weak teacher and LWT_ST denotes the KD loss between the weak teacher and the strong teacher. The type of KD loss stays consistent with the original KD loss function which is cosine similarity inherited from the original formulation of MMR. The hyperparameter λ balances the two losses, controlling the regularization strength of the strong teacher. High λ values limit the weak teacher’s flexibility to deviate from its original parameters, while low values allow more adaptation to the student. The goal is to set λ low enough to enable adaptation without causing feature collapse. During inference, the strong teacher is disabled to reduce inference time, as it does not contribute to scoremap calculation. 4.3
Background Removal via Visual Prompting
To enhance prediction robustness in images with irregular backgrounds, we introduce a pipeline that generates FG/BG segmentation masks, isolating the object of interest by masking out the background. A diagram of the pipeline is shown in Figure 2. The process begins by selecting a few reference images and using Visual Prompting with the Segment Anything Model (SAM) Kirillov et al. [2023] to create initial FG/BG segmentation masks. During inference, the 5
Visual Prompting Meets Feature Reconstruction-Based AD
Method (Cited Results)
Same
Background
Illum.
View
Mean
PatchCore ReverseDistillation DRAEM NSA RIAD InTra MMR MiniMaxAD
89.5 ± 0.2 86.4 ± 0.4 71.4 ± 4.2 43.0 ± 1.3 71.9 ± 1.3 76.8 ± 0.2 89.6 ± 0.2 78.5 ± 0.2
89.4 ± 0.1 86.4 ± 0.7 44.3 ± 11.6 29.7 ± 2.1 33.4 ± 0.6 74.8 ± 0.3 90.1 ± 0.2 79.4 ± 0.4
88.2 ± 0.1 86.7 ± 0.5 67.6 ± 2.7 59.9 ± 1.3 65.3 ± 1.0 73.7 ± 0.3 90.2 ± 0.2 83.7 ± 0.3
84.0 ± 0.2 82.9 ± 0.7 71.1 ± 2.3 51.1 ± 0.1 62.2 ± 1.7 73.4 ± 0.2 86.3 ± 0.3 76.3 ± 0.3
87.8 85.6 63.6 45.9 58.2 74.7 89.1 79.4
MMR+++ (Ours)
91.3 ± 0.0
91.1 ± 0.0
91.3 ± 0.1
86.9 ± 0.1
90.2
Table 2: Pixel-level anomaly detection (AUPRO) on A E BAD-S. Bold denotes the best results. Method (Cited Results)
Same
Background
Illum.
View
Mean
PatchCore ReverseDistillation DRAEM NSA RIAD InTra MMR MiniMaxAD
75.2 ± 0.3 82.4 ± 0.6 64.0 ± 0.4 66.5 ± 1.4 38.6 ± 0.6 39.8 ± 0.8 85.6 ± 0.5 86.3 ± 0.5
74.1 ± 0.3 84.3 ± 0.9 62.1 ± 6.1 48.8 ± 3.5 41.6 ± 1.3 46.1 ± 0.5 84.4 ± 0.7 89.0 ± 0.4
74.6 ± 0.4 85.5 ± 0.9 61.6 ± 2.7 55.5 ± 3.2 46.8 ± 0.8 44.7 ± 0.3 88.8 ± 0.5 85.9 ± 0.7
60.1 ± 0.4 71.9 ± 0.8 62.3 ± 0.9 55.9 ± 1.1 33.0 ± 0.6 46.3 ± 1.5 79.9 ± 0.6 86.9 ± 0.5
71.0 81.0 62.5 56.7 40.0 44.2 84.7 87.0
MMR+++ (Ours)
88.5 ± 0.3
90.6 ± 0.2
91.5 ± 0.3
82.1 ± 0.4
88.2
Table 3: Image-level anomaly detection (I-AUROC) on A E BAD-S. Bold denotes the best results.
Matcher framework Liu et al. [2024], built on the scalable feature-matching approach by Frick et al. Frick et al. [2024], auto-generates masks for query images by referencing these user-generated masks, enabling efficient, high-quality mask production across datasets with minimal human input. This method further benefits from DinoV2 Oquab et al. [2024] to extract meaningful features for feature matching, enhancing the precision of the Matcher pipeline. The masks are then applied to the model’s score maps as a postprocessing step, without requiring model retraining.
5
Experiments
In this section, we evaluate our proposed framework. For our experiments, we use the MMR Zhang et al. [2023a] model as our backbone. We perform extensive experiments on the CC BY 4.0-licensed4 AeBAD-S Bergmann et al. [2019] dataset which entails many of the challenges that we are tackling with our improvements. We validate our approaches on the MVTec dataset to demonstrate that our framework operates reasonably even in highly controlled environments that are missing the fundamental challenges that we are addressing. 5.1
Experimental Settings
Datasets. We evaluate on two datasets. AeBAD-S includes 521 training images and 1639 testing images of aero-engine blades, captured under varying lighting conditions and backgrounds. MVTec encompasses 15 sub-datasets (10 objects, 5 textures) totaling 5354 images. Synthetic Augmentation. We adopt the DDPM approach from Zhang et al. [2024] to sample additional training images by setting the perturbation strength parameter to 0. For AeBAD-S, we add 3× more training images; for MVTec, we maintain a 1:1 ratio between real and synthetic images. Model Training. We primarily follow the training recipe of MMR Zhang et al. [2023a], scaling batch size according to the amount of synthetic data. For real images, we use the original resizing/cropping augmentations; for synthetic images, we apply center-cropping and horizontal flips. We set λ = 1.5 for AeBAD-S and λ = 0.5 for MVTec. 4
https://creativecommons.org/licenses/by/4.0/
6
Visual Prompting Meets Feature Reconstruction-Based AD
Visual Prompting. For SAM Kirillov et al. [2023], we use a single (AeBAD-S) or five (MVTec) reference FG/BG masks. We employ a DINOv2 ViT-L backbone in Matcher and dilate the FG/BG masks (40 pixels for AeBAD-S, 15 pixels for MVTec) to account for object borders.
6
Results
To evaluate the image level predictions we use the area under the receiver operating characteristic (I-AUROC) which captures the relation between true positives and false positives. For anomaly segmentation, we use pixel level Auroc P-AUROC and the area under the per-region-overlap (AUPRO) as the evaluation metric which normalizes anomalies by their size to treat anomalies of any size equally. MVTec The results in Table 5 underscore the effectiveness of background removal and the dual teacher on the MVTec dataset. When comparing to the baseline configuration (Config A), we note a progressive improvement in all three performance metrics as we iteratively add components. It is important to note, that masking was excluded from the Transistor case, due to a conceptional limitation that exists between background masking and logical defects. Synthetic data, however, was found to impair performance on the MVTec dataset. Inspection reveals that many synthetic images labeled as “good” appeared “defect-like,” likely causing these degraded results. Additionally, since training data directly influences the unfrozen teacher model, misaligned synthetic data negatively impacts teacher fine-tuning. This explains the consistent declines across all three metrics when synthetic data is combined with the dual teacher. AeBAD-S The results are presented in Table 2 and Table 3, which are averaged over 5 runs (3 for MiniMaxAD). We compare our method against established baselines on both pixel-level and image-level anomaly detection tasks on A E BAD-S. Baselines include PatchCore Roth et al. [2022] (CVPR’22), ReverseDistillation Deng and Li [2022] (CVPR’22), DRAEM Zavrtanik et al. [2021a] (ICCV’21), NSA Schlüter et al. [2022] (ECCV’22), RIAD Zavrtanik et al. [2021b] (PR’20), InTra Pirnay and Chai [2021] (ICIAP’22), MMR Zhang et al. [2023a] (CII’23), and MiniMaxAD Wang et al. [2024]. Our configuration achieves an I-AUROC of 88.2%, exceeding the previous SOTA of 87.0% by a remarkable 1.2%. Compared to MMR, which is the backbone used in our configuration, our proposed framework increases the I-AUROC by a staggering 3.5%. At the same time, our configuration produces an AUPRO score of 90.2% which exceeds the previously reported SOTA of 89.1% by 1.1%. 6.1
Hyperparameter Ablations
All hyperparameter ablations are performed on AeBAD-S which covers the challenges that our improvements are designed for. The base model used is MMR Zhang et al. [2023a]. Synthetic Data
Data R|S
b = 16 (1x)
b = 32 (2x)
b = 48 (3x)
b = 64 (4x)
b = 80 (5x)
0 | 521 521 | 0 521 | 521 521 | 1042 521 | 1563 521 | 2084 521 | 3647
80.3 / 87.6 84.7 / 88.6 83.5 / 87.7 81.7 / 87.1 80.1 / 85.5 78.5 / 83.0 76.7 / 81.1
79.9 / 87.3 84.6 / 88.5 85.7 / 88.6 85.5 / 88.7 84.9 / 88.7 83.5 / 87.1 79.4 / 83.6
—/— 84.0 / 88.5 85.1 / 88.6 85.6 / 88.9 86.0 / 88.9 85.4 / 88.8 82.4 / 87.0
79.5 / 87.3 83.5 / 88.1 84.6 / 88.3 85.3 / 88.5 85.7 / 88.7 85.7 / 88.9 84.5 / 88.2
—/— 83.2 / 88.3 84.0 / 88.4 85.1 / 88.6 85.5 / 88.7 85.4 / 88.8 85.2 / 88.8
Table 4: Ablation study for AeBAD-S real (R) and synthetic (S) data on MMR. Evaluation metrics are Image AUROC / AUPRO. b denotes batch size. Results are averaged over three runs. In Table 4, we ablate the effects of synthetic data volume and batch size. The optimal batch size scales roughly linearly with the total data. The best performance is achieved with 1563 synthetic images (a 1:3 real-to-synthetic ratio) and a batch size of 48 (3× the original). Notably, the best configurations for I-AUROC and AUPRO coincide. Even with purely synthetic training data, the model outperforms most baselines. 7
Visual Prompting Meets Feature Reconstruction-Based AD
Syn. Data A B C D E F G H
Dual Teach
MVTecAD
BG Rem
✓ ✓ ✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓ ✓
AeBAD-S
I-AUROC
P-AUROC
AUPRO
I-AUROC
AUPRO
98.3 ± 0.0 98.4 ± 0.0 98.4 ± 0.0 98.5 ± 0.0 98.6 ± 0.0 98.0 ± 0.2 98.7 ± 0.0 98.6 ± 0.0
97.2 ± 0.0 97.1 ± 0.0 97.6 ± 0.0 97.3 ± 0.0 97.7 ± 0.0 97.1 ± 0.1 97.3 ± 0.0 97.4 ± 0.0
92.7 ± 0.0 92.3 ± 0.0 92.9 ± 0.0 93.0 ± 0.0 93.2 ± 0.0 92.0 ± 0.2 92.8 ± 0.0 92.9 ± 0.0
84.7 ± 0.1 86.0 ± 0.1 85.7 ± 0.2 85.8 ± 0.1 86.8 ± 0.3 87.2 ± 0.2 86.6 ± 0.1 88.2 ± 0.1
88.7 ± 0.1 88.9 ± 0.0 89.3 ± 0.1 89.3 ± 0.0 89.9 ± 0.1 89.7 ± 0.1 89.4 ± 0.0 90.2 ± 0.1
Table 5: Ablation on MVTec AD and AeBAD-S. Results are reported from the last epoch. For AeBAD-S the results are averaged over 5 runs. Reference Images
Pairwise mIoU
HITL mIoU
I-AUROC
AUPRO
1 5 15
97.7 ± 0.7 98.8 ± 0.1 99.0 ± 0.2
98.3 ± 0.6 98.7 ± 0.1 98.9 ± 0.2
88.2 ± 0.1 88.2 ± 0.1 88.2 ± 0.1
90.2 ± 0.1 90.2 ± 0.1 90.1 ± 0.1
Table 6: Ablation study on the number of reference images given to Matcher for AeBAD-S dataset. Results are averaged over three different FG/BG mask sets using N random images as references. ’Pairwise mIoU’ denotes the average mIoU in between different Matcher generated FG/BG mask sets. ’HITL mIoU’ denotes the average mIoU in between the Matcher generated FG/BG mask sets and the human-in-the-loop generated FG/BG mask set. Dual Teacher In Figure 4, we ablate the λ parameter, which controls the flexibility of the weak teacher across varying amounts of synthetic training data. We observe that performance collapses as λ → 0 due to lack of regularization, while large λ values converge to the base model. Peak performance occurs at intermediate values. Notably, the optimal λ decreases as synthetic data increases, suggesting that the synthetic data provides implicit regularization, allowing for greater teacher flexibility without collapse.
AUPRO Ablation Results
AUROC Ablation Results
Figure 4: Ablation study for the λ parameter. ’+X’ indicates that the training dataset includes the original training data plus X synthetic images. b denotes batch size. Background Removal Figure 5 compares four post-processing variants: FG masks (raw Matcher output), FG masks+GT (oracle union with ground truth i.e the cheated upper bound which is unavailable at test time), dilated FG masks (FG masks dilated by 40px), and our mixing method that merges raw and dilated masks. Raw FG masks already reach the oracle in I-AUROC (Fig. 5(a)), confirming Matcher’s accuracy, but their AUPRO collapses because edge and fracture-type defects are missed (Fig. 5(b)). Dilating restores AUPRO at the cost of a slight I-AUROC drop, echoing the defect-coverage curves in Figures 6–7. Our mixing step dilates the FG scoremap and clips 8
Visual Prompting Meets Feature Reconstruction-Based AD
(a) AUROC
(b) AUPRO
Figure 5: Ablation of the λ parameter across multiple scoremap post-processing strategies. FG masks are the raw Matcher outputs; dilated FG masks are FG masks dilated by 40px; the mixing method fuses FG and dilated FG masks; and FG masks+GT take the union of FG masks with ground-truth test masks (upper bound). Results are from the final epoch and averaged over five runs.
Figure 6: Defect coverage ratios of FG masks for the different defect types present in AeBAD-S with and without FG mask dilation.
it to the peak value of the raw scoremap, retaining sharp discrimination while covering all defects, and thus yields the best overall balance.
6.2
Ablations on Key Contributions
To observe how our proposed improvements interact we perform extensive ablation studies on MVTecAD and AeBAD-S. Table 5 states the individual and combined effects of our proposed approaches. To assess the quality of the generated FG/BG segmentation masks we compare against a set of manually curated FG/BG masks using Visual Prompting on all images in the test dataset. The results are stated in Table 6. 9
Visual Prompting Meets Feature Reconstruction-Based AD
Figure 7: Interaction between defect coverage ratio and AUPRO score for defect type ’fracture’
7
Conclusion
In this study, we demonstrate that established anomaly detection methods heavily rely on controlled environments, limiting their applicability in real-world scenarios. To enhance robustness against variations in object positioning and background noise, we propose a framework that is architecture-agnostic and adaptable to most anomaly detection methods. The framework consists of three independent and modular improvements that are easily applicable: (1) leveraging a diffusion model to enrich the training dataset with synthetic images, increasing data diversity and enabling the model to capture the data distribution more reliably, thus improving generalization; (2) unfreezing the teacher encoder in student–teacher reconstruction-based methods, which increases adaptability to the target domain and mitigates pre-training bias, thereby boosting model performance; (3) employing Visual Prompting to isolate the object of interest by masking the image background, thereby improving robustness to background variation. Extensive experiments validate the effectiveness of our improvements under challenging conditions. On the AeBAD-S dataset, our framework achieves state-of-the-art performance in both detection and segmentation, improving scores by 3.5% and 1.1%, respectively, over the baseline model.
References Diego Carrera, Fabio Manganini, Giacomo Boracchi, and Ettore Lanzarone. Defect detection in sem images of nanofibrous materials. IEEE Transactions on Industrial Informatics, 13:551–561, 04 2017. doi:10.1109/TII.2016.2641472. Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad — a comprehensive real-world dataset for unsupervised anomaly detection. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9584–9592, 2019. doi:10.1109/CVPR.2019.00982. 10
Visual Prompting Meets Feature Reconstruction-Based AD
Qiyu Chen, Huiyuan Luo, Chengkan Lv, and Zhengtao Zhang. A unified anomaly synthesis strategy with gradient ascent for industrial anomaly detection and localization, 2024. URL https://arxiv.org/abs/2407.09359. Hanxi Li, Jingqi Wu, Lin Yuanbo Wu, Hao Chen, Deyin Liu, Mingwen Wang, and Peng Wang. Industrial anomaly detection and localization using weakly-supervised residual transformers, 2024. URL https://arxiv.org/abs/2306. 03492. Zilong Zhang, Zhibin Zhao, Xingwu Zhang, Chuang Sun, and Xuefeng Chen. Industrial anomaly detection with domain shift: A real-world dataset and masked multi-scale reconstruction. April 2023a. Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. SPot-the-difference self-supervised pre-training for anomaly detection and segmentation. 2022. Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14318–14328, June 2022. Guoyang Xie, Jinbao Wang, Jiaqi Liu, Feng Zheng, and Yaochu Jin. Pushing the limits of fewshot anomaly detection in industry vision: Graphcore, 2023. URL https://arxiv.org/abs/2301.12082. Donghyeong Kim, Chaewon Park, Suhwan Cho, and Sangyoun Lee. Fapm: Fast adaptive patch memory for real-time industrial anomaly detection. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2023. doi:10.1109/ICASSP49357.2023.10096400. Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Draem – a discriminatively trained reconstruction embedding for surface anomaly detection, 2021a. URL https://arxiv.org/abs/2108.07610. Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization, 2021. URL https://arxiv.org/abs/2104.04015. Yiyuan Yang, Chaoli Zhang, Tian Zhou, Qingsong Wen, and Liang Sun. Dcdetector: Dual attention contrastive representation learning for time series anomaly detection. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, page 3033–3045, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. doi:10.1145/3580305.3599295. URL https://doi.org/10.1145/ 3580305.3599295. Jeeho Hyun, Sangyun Kim, Giyoung Jeon, Seung Hwan Kim, Kyunghoon Bae, and Byung Jun Kang. Reconpatch: Contrastive patch representation learning for industrial anomaly detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2052–2061, January 2024. Xuan Zhang, Shiyu Li, Xi Li, Ping Huang, Jiulong Shan, and Ting Chen. Destseg: Segmentation guided denoising student-teacher for anomaly detection, 2023b. URL https://arxiv.org/abs/2211.11317. Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9737–9746, June 2022. Kilian Batzner, Lars Heckler, and Rebecca König. Efficientad: Accurate visual anomaly detection at millisecond-level latencies, 2024. URL https://arxiv.org/abs/2303.14535. Jia Guo, Shuai Lu, Lize Jia, Weihang Zhang, and Huiqi Li. Recontrast: Domain-specific anomaly detection via contrastive reconstruction, 2023. URL https://arxiv.org/abs/2306.02602. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. URL https://arxiv.org/abs/ 2006.11239. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022. URL https://arxiv.org/abs/2112.10752. Ximiao Zhang, Min Xu, and Xiuzhuang Zhou. Realnet: A feature selection network with realistic synthetic anomaly for anomaly detection, 2024. URL https://arxiv.org/abs/2403.05897. Julian Wyatt, Adam Leach, Sebastian M. Schmon, and Chris G. Willcocks. Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 649–655, 2022. doi:10.1109/CVPRW56347.2022.00080. Matthew Baugh, James Batten, Johanna P. Müller, and Bernhard Kainz. Zero-shot anomaly detection with pre-trained segmentation models, 2023. URL https://arxiv.org/abs/2306.09269. Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017. Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141–142, 2012. 11
Visual Prompting Meets Feature Reconstruction-Based AD
Samet Akcay, Dick Ameln, Ashwin Vaidya, Barath Lakshmanan, Nilesh Ahuja, and Utku Genc. Anomalib: A deep learning library for anomaly detection. In 2022 IEEE International Conference on Image Processing (ICIP), pages 1706–1710. IEEE, 2022. Ibrahima Ndiour, Nilesh Ahuja, Utku Genc, and Omesh Tickoo. Fre: A fast method for anomaly detection and segmentation. arXiv preprint arXiv:2211.12650, 2022. Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier. Padim: a patch distribution modeling framework for anomaly detection and localization. In International Conference on Pattern Recognition, pages 475–489. Springer, 2021. Matías Tailanian, Álvaro Pardo, and Pablo Musé. U-flow: A u-shaped normalizing flow for anomaly detection with unsupervised threshold. Journal of Mathematical Imaging and Vision, pages 1–19, 2024. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. doi:10.1109/CVPR.2009.5206848. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023. URL https://arxiv.org/abs/2304.02643. Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all-purpose feature matching, 2024. URL https://arxiv.org/abs/2305.13310. Thomas Frick, Cezary Skura, Filip M Janicki, Roy Assaf, Niccolo Avogaro, Daniel Caraballo, Yagmur G Cinar, Brown Ebouky, Ioana Giurgiu, Takayuki Katsuki, Piotr Kluska, Cristiano Malossi, Haoxiang Qiu, Tomoya Sakai, Florian Scheidegger, Andrej Simeski, Daniel Yang, Andrea Bartezzaghi, and Mattia Rigotti. Interactive image segmentation guided by visual prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Demos, 2024. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024. URL https://arxiv.org/abs/2304.07193. Hannah M. Schlüter, Jeremy Tan, Benjamin Hou, and Bernhard Kainz. Natural synthetic anomalies for self-supervised anomaly detection and localization, 2022. URL https://arxiv.org/abs/2109.15222. Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Reconstruction by inpainting for visual anomaly detection. Pattern Recognition, 112:107706, 2021b. ISSN 0031-3203. doi:https://doi.org/10.1016/j.patcog.2020.107706. URL https://www.sciencedirect.com/science/article/pii/S0031320320305094. Jonathan Pirnay and Keng Chai. Inpainting transformer for anomaly detection, 2021. URL https://arxiv.org/abs/2104. 13897. Fengjie Wang, Chengming Liu, Lei Shi, and Pang Haibo. Minimaxad: A lightweight autoencoder for feature-rich anomaly detection, 2024. URL https://arxiv.org/abs/2405.09933.
12