Semantic Robustness Probing via Inpainting An Interactive Tool for Safety-Critical Object Detection
arXiv:2605.27155v1 [cs.CV] 26 May 2026
Nico Steckhan1 , Krutarth Prajapati1 , Weija Shao1 , and Silvia Vock1 Federal Institute for Occupational Safety and Health (BAuA), Germany {steckhan.nico,prajapati.krutarth,shao.weija,vock.silvia}@baua.bund.de
Abstract. Testing object detectors in safety-critical domains requires semantically meaningful probes beyond pixel-level corruptions. We present SemProbe1 , a tool for semantic robustness probing: users upload deployment images, create masks manually or automatically, select operational design domain-derived factors (or custom prompts), and run diffusion-based controlled inpainting. The system supports batch jobs, parallel seed/workflow variations, and configurable generation parameters. After each output, model inference runs automatically and displays annotated before/after comparisons with performance deltas. All probes are logged as structured artifacts, enabling traceable robustness evidence aligned with safety evaluation workflows. We demonstrate SemProbe on hand detection for dimension saws, targeting factors from insurance-oriented test criteria. Keywords: robustness probing · inpainting · operational design domain · safety-critical object detection
1
Introduction
Camera-based safety systems increasingly rely on object detectors—for example, hand detection on dimension saws that triggers a safe state when a hand approaches the blade [2]. With the EU Machinery Regulation [3] applying from January 2027 and the EU AI Act [4] classifying such systems as high-risk, demonstrating detector robustness under realistic operational conditions is becoming a regulatory requirement. Yet systematic robustness testing remains difficult. Standard corruption benchmarks [5,6] apply domain-agnostic perturbations (noise, blur) unrelated to real operational factors such as gloves, sawdust, or specular glare. Collecting exhaustive real data near hazardous machinery is operationally constrained, and the critical factors—skin colour, soiling, tattoos, lighting, hand poses [2]—are precisely those that are rare yet safety-relevant. Recent generative approaches 1
Demo video: [VIDEO-URL]. Code: https://github.com/steckhan/semrob. Dataset (Zenodo): https://doi.org/10.5281/zenodo.18936355.
2
Steckhan et al. GPT-4o-mini factor generation
Input Layer Upload image(s)
Masking Layer Manual brush or GroundingDINO + SAM2
ODD Prompt Layer Factor selection (Actors, Activities, Environment, Sensors)
Generation Layer ComfyUI workflow FLUX.2-klein + Qwen3-4B + FLUX.2 VAE
Logging Layer Job metadata, CSV/JSON, artifact folders
Comparison Layer Before/after boxes confidence delta
Analysis Layer YOLO detection (auto post-process)
Execution Layer Batch queue + parallel seeds/workflows
Fig. 1. System architecture of SemProbe. The pipeline combines image input, manual/automatic masking, ODD-driven prompt construction, ComfyUI-based inpainting, automatic YOLO post-analysis, and structured logging.
target training data augmentation [7,8] or autonomous driving test generation [9], but none provide an interactive, ODD-structured evaluation tool for specific failure conditions. We present SemProbe, an open-source, locally deployable tool that closes this gap through semantic robustness probing: controlled, semantically meaningful image modifications via inpainting, with immediate detector feedback. SemProbe enables domain experts to: – derive semantic probes from an Operational Design Domain (ODD) description, structured along four dimensions (actors, activities, environment, sensors), optionally assisted by an LLM; – interactively apply these probes via mask-based inpainting and inspect before/after detector comparisons with confidence deltas; – log all probes as structured, traceable artifacts (CSV/JSON) aligned with industrial safety evaluation workflows. We demonstrate the tool on hand detection for dimension saws, where probing factors align with IFA/DGUV functional test criteria [2,10].
2
System Overview
SemProbe is a local web application built on top of ComfyUI [1]. Fig. 1 shows the end-to-end architecture; Fig. 2 shows the interactive interface. A typical probing session proceeds in five steps. Step 1: Upload. The user uploads one or multiple images from the deployment environment. Batch processing is supported with per-image mask overlays and live job progress tracking. Step 2: Mask. The region to modify is masked either manually (brush tool) or via text-prompted auto-masking with GroundingDINO + SAM2.
Semantic Robustness Probing via Controlled Inpainting
3
Step 3: Select factor. The user selects a factor and level from a structured factor catalog or enters a custom prompt. The catalog is organized along four ODD dimensions—following the taxonomy of ISO 34503 [11]: Actors (bare hands, gloved hands, skin tones, tattoos, soiling), Activities (pushing, reaching, gripping a push stick), Environment (sawdust, specular glare, shadows), and Sensors (lens dust, motion blur, overexposure). Each factor has discrete levels and an associated prompt template. Catalogs can be authored manually or semiautomatically: given a free-text ODD description, an LLM (GPT-4o-mini) extracts factors across the four dimensions and generates prompt templates. The user reviews and edits the result, enabling transfer to new domains without code changes. Step 4: Inpaint. FLUX.2 [klein] [1] is the default diffusion model for inpainting. It runs locally on a single consumer GPU (∼13 GB VRAM) at 1024×1024— critical for industrial settings where images must not leave the premises. The user configures generation parameters (seed, steps, CFG, denoise strength, number of samples). A background preservation check (SSIM outside the mask) flags unintended scene changes. Alternative models via cloud APIs (OpenAI, Google Imagen) are supported as drop-in replacements. Step 5: Compare. YOLO [12] runs automatically on each generated output. Annotated results are displayed alongside the original with bounding boxes, confidence scores, and confidence deltas. Every probe is logged with its metadata (factor, level, prompt, backend, confidence delta, bbox IoU, workflow, job status). Inputs, outputs, and YOLO annotations are stored per job in structured folders and can be exported as CSV/JSON records for documentation aligned with EU AI Act robustness requirements [4]. For systematic coverage, factors can be combined into failureoriented stress grids: compact subsets of factor combinations targeting specific failure types (false negatives, localization errors, false positives), rather than exhaustive Cartesian products.
3
Demonstration: Hand Detection on Dimension Saws
We demonstrate SemProbe on hand detection for a camera-based assistance system on dimension saws (Formatkreissägen). The detector under test is a YOLOv10 model [12] trained on overhead RGB frames from a fixed camera above a saw table. The factor catalog contains 6 factors with 3–5 levels each, derived from IFA functional test criteria [2]: hand covering, surface contamination, illumination, hand pose, mechanical occlusion, and sensor artifacts. Fig. 2 shows a representative session. The user uploads a frame showing a bare hand on the saw table (left panel ), masks the hand region, and selects hand covering → cut-resistant glove from the factor catalog. After inpainting, the tool displays the generated image alongside the original (center panel ), and YOLO annotations with confidence scores appear automatically (right panel ).
4
Steckhan et al.
Fig. 2. SemProbe interface: the user uploads an image, masks the hand region, selects a factor from the ODD-derived catalog, triggers inpainting, and receives a side-by-side detection comparison. Table 1. Representative probing results (YOLOv10, threshold 0.40). Factor
Level
(original) — Hand covering cut-resistant Hand modification motion blur Surface contam. heavy sawdust Illumination low light
Prec. Rec. FNR ∆P 0.94 0.79 0.93 0.68 0.82
0.91 0.64 0.90 0.52 0.71
∆R
0.09 — — 0.36 −0.15 −0.27 0.10 −0.01 −0.01 0.48 −0.26 −0.39 0.29 −0.12 −0.20
Table 1 summarizes probing results across several factors using precision, recall, false-negative rate (FNR), and deltas relative to baseline. A cut-resistant glove reduces recall from 0.91 to 0.64 (∆R = −0.27) and increases FNR from 0.09 to 0.36. Aggregated over multiple images, these results produce a factor sensitivity ranking—showing which operational factors cause the largest detection degradation—directly usable as structured robustness evidence for safety documentation.
4
Conclusion
We presented SemProbe, an interactive, open-source tool for semantic robustness probing of object detectors. By connecting ODD-derived semantic factors to controlled inpainting and immediate detector feedback, the tool lets domain experts discover detector vulnerabilities under specific, safety-relevant conditions— without collecting new real-world data. Built on FLUX.2 [klein], it runs entirely
Semantic Robustness Probing via Controlled Inpainting
5
locally on consumer hardware, preserving data sovereignty. The factor catalog is domain-agnostic: while we demonstrated on hand detection for dimension saws, the same workflow applies to any safety-critical vision task whose ODD can be decomposed into actors, activities, environment, and sensors. Limitations. Inpainting quality depends on the generative model and may produce artifacts for complex modifications; configurable parameters help mitigate but cannot eliminate this. The tool currently operates on single images; videolevel temporal probing is planned as future work. LLM-extracted factor catalogs require human review. SemProbe is open source: https://github.com/steckhan/semrob. Acknowledgments. This study was conducted at the Federal Institute for Occupational Safety and Health (BAuA) and sponsored by the German Federal Ministry of Labour and Social Affairs. It is part of the junior research group “Artificial Intelligence (AI) in a Safe and Healthy Working Environment.” Disclosure of Interests. The authors have no competing interests to declare.
References 1. Black Forest Labs: FLUX.2 [klein] — 4B-parameter rectified flow transformer for unified text-to-image generation and mask-based editing. Apache 2.0 (January 2026). https://huggingface.co/black-forest-labs/FLUX.2-klein-4B 2. Seifen, T.: AI-based assistance system for dimension saws (Focus on IFA’s work No. 0450e). DGUV/IFA, issue 08/2024 (2024) 3. European Parliament and Council: Regulation (EU) 2023/1230 on machinery products. Official Journal of the European Union, L 165 (2023) 4. European Parliament and Council: Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI Act). Official Journal of the European Union, L series (2024) 5. Michaelis, C., Mitzkus, B., Geirhos, R., et al.: Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv:1907.07484 (2019) 6. Mao, X., et al.: COCO-O: A benchmark for object detectors under natural distribution shifts. In: ICCV (2023) 7. Fang, H., et al.: Data augmentation for object detection via controllable diffusion models. In: WACV (2024) 8. Zhu, J., et al.: ODGEN: Domain-specific object detection data generation with diffusion models. arXiv:2405.15199 (2024) 9. Yang, C., et al.: DriveGEN: Generalized and robust 3D detection in driving via controllable text-to-image diffusion generation. In: CVPR (2025) 10. DGUV Test: General Principles for Assessing the Safety of Artificial Intelligence (AI). DGUV Test Information 05 (2021) 11. ISO: ISO 34503:2023 — Road vehicles — Taxonomy for operational design domain for automated driving systems (2023) 12. Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics YOLO (2023). https://github. com/ultralytics/ultralytics