Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models Nikolai Röhrich1,2 Julian Gleißner1 Ahmed H. A. Ibrahim1,3 Silvan Mertes4 Tobias Huber1,3 *
arXiv:2606.31603v1 [cs.CV] 30 Jun 2026
1
XITASO GmbH 2 Zuse School of Excellence in Reliable AI 3 Technische Hochschule Ingolstadt 4 Technische Hochschule Augsburg
1
Segmentation model trained on real data
2
Extract per-pixel uncertainty
Create mask of the
3 most uncertain areas
4
Regenerate surrounding context
5
Restrict loss to the uncertain area
Iterative Segmenter Improvement
Figure 1. Uncertainty-Guided Context Synthesis. We augment real samples by computing the current segmenter’s per-pixel predictive entropy and inpaint a fresh visual context around hard-to-classify regions. Our method transcends prior region selection heuristics by allocating synthetic budget on informative regions rather than always choosing background or foreground regions.
Abstract
tial mIoU gains on Cityscapes, UAVID, and BDD100K with the largest gains on rare and difficult classes such as buses, trains, or (from the aerial perspective) cars. Our results demonstrate that uncertainty-guided context augmentation is a highly effective lever to improve segmentation performance on complex datasets, with code provided.
Semantic segmentation models struggle with data sparsity and rare or visually diverse regions, e.g., dense regions or small objects in aerial or autonomous mobility data. While synthetic augmentation is an appealing solution, directly generating new labeled data risks misalignment of labels and generated pixels. Existing solutions to this problem often rely on external models, or employ coarse heuristics such as indiscriminately augmenting all foreground objects or entire backgrounds, which wastes capacity on uninformative pixels. To address this, we propose an uncertaintyguided synthetic context augmentation strategy that strictly preserves label validity and efficiently maximizes pixel informativeness per synthetic sample – no external guardrails required. Using a baseline segmenter’s predictive entropy, we identify uncertain semantic regions and inpaint only the complementary visual context. When fine-tuning the segmenter on this synthetic data, we compute the loss only over the original pixels, excluding inpainted regions. This focuses learning on the unmodified, uncertain regions while presenting them in novel contexts. We demonstrate substan-
1. Introduction Data sparsity remains a core challenge for semantic segmentation models, particularly for datasets with rare classes and visually diverse regions [8, 13]. Examples include dense scenes and small objects in datasets with a high level of local detail. Such data are particularly tedious to annotate: For instance, annotating a single urban driving scene takes up to 90 minutes [3]. Real-world data are also highly imbalanced, and consequently, the most critical classes are often those with the fewest labeled pixels [4, 19]. The rise of generative diffusion models [11, 27–29] has made synthetic data an attractive lever to close this gap. Two principal strategies have emerged here: The first generates images and labels jointly [24, 32, 33, 35]. While conceptually appealing, these approaches are constrained by a
* Corresponding author: [email protected]
1
label–pixel mismatch problem: the diffusion model must either invent labels for the pixels it generates, or generate pixels that faithfully match existing labels. In neither case, there is a guarantee for exact label-to-pixel correspondence. A second, more conservative line of work sidesteps this problem by partially regenerating real images while keeping the original annotations intact [12, 15, 18, 39]. Here, the literature has converged on coarse spatial heuristics: some methods regenerate the foreground objects of the scene while preserving the background [15], while others do the exact opposite, regenerating the entire background while preserving the foreground objects [18]. Strikingly, these two heuristics reach opposite empirical conclusions. Kupyn and Rupprecht [15] attribute their performance gains to the visual diversity injected by redrawing foreground objects, whereas Li et al. [18] report that background augmentation outperforms object augmentation by a wide margin. We argue that this contradiction is not a matter of who is right: it demonstrates that the foreground/background axis is the wrong axis along which to allocate the synthetic-data budget. The right axis, we argue, is informational, not spatial: The pixels that deserve to be preserved are those on which the current segmenter is most uncertain (see Figure 2). Conversely, regions suitable for regeneration are those which the segmenter is already confident in and which therefore provide limited training signal, regardless of whether they belong to certain spatial categories [30, 36]. Building on this insight, we propose an uncertaintytargeted synthetic context augmentation strategy. Given a segmenter trained on the available real data, we compute its per-pixel predictive entropy [9, 23] and aggregate it over each ground-truth semantic region. The regions with the highest mean entropy are selected greedily to form a preserve mask of uncertain pixels. Diffusion-based inpainting [21, 28] is then applied to the complement of this mask, generating a novel visual context around the preserved region. Crucially, the original pixels inside the mask remain bitwise identical, so the labels they carry are guaranteed valid. This design has three consequences: First, label noise is eliminated by construction rather than mitigated by external models. We require no ControlNet [38], no edge or depth conditioning, no mask-refinement network, and no auxiliary VQA prompts [17] to keep the inpainter on-label. Second, by computing the loss only for the preserved uncertain pixels, the training is driven by the most informative regions, while inpainted pixels serve as novel context. Finally, this design is naturally iterative, since the preserve mask is defined by the current segmenter’s predictive entropy. Improving the segmenter shifts uncertainty onto new regions and additional rounds of selection-and-inpainting target exactly those regions. Repeating the loop turns one-shot augmentation into an active-learning-inspired cycle [6, 10, 36] in which the diffusion inpainter (rather than a human anno-
Original Image
Segmenter Entropy Map
Background Aug. Preserve Area
Instance Aug. Preserve Area
Our Preserve Area
Figure 2. Uncertainty-based region selection transcends the foreground-background contradiction. We argue that the augmented regions should be based on uncertainty, rather than always selecting either the entire foreground [15] or background [18].
tator) supplies the answer to each round’s query, with gains accumulating until uncertainty saturates. We evaluate our method on semantic segmentation for three benchmarks with a high level of complexity and local diversity: Cityscapes [3] and BDD100K [37] (urban driving), and UAVID [22] (aerial urban scenes). On all benchmarks, mean IoU improves substantially, with gains concentrated on rare and safety-critical classes. Contributions: • Strictly label-preserving augmentation. Synthetic pixels are explicit ignore regions, so our synthetic data injects context diversity without injecting any label noise, and without requiring any external guardrails. • Uncertainty-targeted region selection. We allocate synthetic-data capacity along the informational axis rather than the spatial one, overcoming the foregroundvs-background contradiction in prior work (see Figure 2). • Empirical gains. We achieve +3.68/ + 2.64/ + 1.57 mIoU on UAVID / Cityscapes / BDD compared to the real-data only segmenter, and more than double the gain of the best augmentation baseline on Cityscapes, supporting that the right unit of augmentation is uncertainty. • An iterative active-learning loop. Because selection is driven by the current model’s uncertainty rather than a fixed spatial rule, multiple iterations can re-target the segmenter’s latest failure modes. This compounds gains in a way one-shot methods cannot: on Cityscapes the train class IoU, which the first iteration barely moves (+1.1), later climbs to +9.2 once uncertainty concentrates on it. 2
2. Related Work
[17] model, and a TRACER-7 segmentation network [16]. Simple Background Augmentation [18], targeting instance segmentation and object detection, takes the opposite view: it regenerates the entire background of each image while keeping the foreground objects bitwise unchanged. The authors report that background regeneration outperforms foreground regeneration by a wide margin. The two papers therefore arrive at opposite empirical conclusions, a contradiction we argue arises because the foreground/background axis is not the one that matters. While these works are the most relevant to our approach, it differs along four dimensions. First, our unit of selection is not a geometric region (foreground or background) but an informationally hard region chosen by the segmenter’s own predictive entropy. Second, where [15] relies on a full stack of auxiliary models, our method achieves full labelpreservation without auxiliary models. Third, where both prior methods train on pixels they themselves regenerated (and therefore inherit label-content mismatch) we mark every generated pixel as ignore (Eq. (8)) and train only on real labels. Fourth, both prior methods are static augmentations applied once before training. Because our selection criterion is model-dependent, our augmentation forms a closed loop that can be re-run against the improved segmenter.
Predictive Uncertainty in Segmentation. Active learning [6, 10, 36] traditionally uses model uncertainty to query the most informative unlabelled samples for human annotation. Our method draws on the same underlying signal but repurposes it to decide which pixels are worth surrounding with fresh synthetic visual context. Closest to our work is Uncertainty-Aware ControlNet [25], which uses a per-pixel entropy map as a conditioning input to a second ControlNet branch and generates full synthetic images to bridge a domain gap, and Active Learning Inspired ControlNet Guidance [14], which uses prediction entropy for classifier guidance, thus requiring gradient flow through each diffusion step while also employing a second ControlNet branch. The role of entropy in those pipelines is orthogonal to ours: they generate whole images conditioned on entropy, whereas we use entropy to choose what to preserve and leave the actual generative process unconditioned on the segmenter. At the same time, our method refrains from auxiliary models and generates using only an out-of-the-box inpainting model. Diffusion-Based Data Synthesis. A growing body of work uses diffusion models [27–29] to generate synthetic data for dense-prediction tasks. The dominant strategy is generate-then-label: the diffusion model produces an image and a paired label map is recovered either from the model’s internal cross-attention representations (DiffuMask [33], DatasetDM [32]) or by conditioning generation directly on a target mask (FreeMask [35], DatasetDiffusion [24]). Closely related lines synthesize data for classification using diffusion priors [31]. All such methods inherit a structural label–content mismatch: because labels are either inferred from generated pixels or imposed on them, there is no guarantee that the two actually agree. A second branch of generative augmentation targets object detection rather than semantic segmentation, leveraging ControlNet [38] to inject structural cues into the generation process. [7] conditions Stable Diffusion on detectionfriendly layouts to produce additional training images, and ReCon [40] extends this idea with region-controllable rectification and alignment to better preserve bounding-box semantics. These methods operate at the granularity of bounding boxes rather than dense per-pixel labels, and rely on auxiliary structural networks to keep the generation on-task.
3. Method Our method consists of five main steps as shown in Fig. 1. It starts (1) with a segmenter that is trained on real data. Then (2), it identifies the regions on which that segmenter is most uncertain, (3) preserves those pixels and their labels, and (4) inpaints a fresh visual context around them with an off-theshelf latent diffusion inpainter. Finally (5), the segmenter is fine-tuned on the resulting augmented dataset where the loss is restricted to original pixels and labels. Due to the active-learning-inspired nature of our approach, it can be applied iteratively for further improvements.
3.1. Setting and Preliminaries Let D = {(xn , yn )}N n=1 denote a semantic segmentation dataset, where each image x ∈ R3×H×W is paired with a dense label map y ∈ {0, 1, . . . , C − 1, ι}H×W over C semantic classes and a designated ignore index ι. For any segmenter fθ : R3×H×W → RC×H×W , let pc,i,j (x) denote the softmax probability assigned to class c at location (i, j) and let Ωy = {(i, j) : yi,j ̸= ι} be the set of valid pixels in y. Then, the standard pixel-wise cross-entropy loss with ignore-index masking [3] is given by
Preserve-and-Regenerate Augmentation. InstanceAugmentation [15] targets several tasks including semantic segmentation and redraws every annotated object in the scene while keeping the background intact. To prevent generated objects from drifting away from the original masks, the method composes a stack of auxiliary networks: ControlNets conditioned on depth and edge maps, a BLIP-VQA
L(θ; x, y) = −
1 |Ωy |
X
log pyi,j , i,j (x).
(1)
(i,j)∈Ωy
Let fθ0 be a segmenter trained on the real dataset D by minimizing Eq. (1). Our goal is to improve this real-data-only 3
model with the proposed synthetic training data augmentation. Thus, we refer to it as the baseline segmenter.
Algorithm 1: A single iteration of our uncertaintyinformed training data generation. Input: Real dataset D; baseline segmenter fθ0 ; inpainter G; text prompt t; area threshold τ ; ignore index ι. Output: Improved segmenter fθ1 . 1 foreach (x, y) ∈ D do 2 p ← pixel-wise softmax fθ0 (x) PC−1 3 Hi,j ← − c=0 pc,i,j log pc,i,j 4 foreach class c ∈ Cx do 5 y c ← 1[ y = c ] P c 6 H̄c ← ∥y1c ∥1 i,j yi,j Hi,j 7 end 8 Sort Cx in descending order of H̄c , yielding σ. 9 M ← 0H×W ; 10 k←0 1 ∥M ∥1 < τ do 11 while HW 12 k ← k + 1; 13 M ← M + y σ(k) 14 end 15 xgen ← G(x, 1 − M, t) 16 x̃ ← (1 − M ) ⊙ xgen + M ⊙ x 17 Construct ỹ from y and M . 18 Dsyn ← Dsyn ∪ {(x̃, ỹ)} 19 end 20 fθ1 ← fine-tune fθ0 on D ∪ Dsyn 21 return fθ1
3.2. Uncertain-Region Selection For any image x, we measure uncertainty through the pixelwise Shannon entropy of the predictions pc,i,j (x) of fθ0 (x): Hi,j (x) = −
C−1 X
pc,i,j (x) log pc,i,j (x)
(2)
c=0
Selecting the most uncertain regions of an image at the level of individual pixels may destroy semantic coherence, since preserving only scattered pixels of an object, such as a car, erases its class-level meaning. Thus, we aggregate uncertainty at the level of ground-truth class regions. For an image x with label map y, let Cx = {c ∈ {0, . . . , C − 1} : ∃(i, j) s.t. yi,j = c} be the set of valid semantic classes present in the image. For each class c ∈ Cx , let y c ∈ {0, 1}H×W denote its binary ground-truth mask, ( 1 if yi,j = c c yi,j = (3) 0 otherwise. The mean predictive entropy over class c is 1 X c y Hi,j (x), H̄c (x, y) = ∥y c ∥1 i,j i,j
(4)
P c where, ∥y c ∥1 = i,j yi,j denotes the number of pixels belonging to class c. We rank the classes c ∈ Cx by their mean entropy H̄c (x, y) and keep adding the most uncertain classes until their combined region covers a predefined fraction τ of the full image. Let σ : {1, . . . , |Cx |} → Cx be the permutation that ranks classes by region entropy (i.e., H̄σ(1) ≥ H̄σ(2) ≥ . . . ≥ H̄σ(|Cx |) ). Then we define our preserve mask as the sum of the most uncertain class masks, M (x, y; τ ) =
k X
y σ(j) ,
3.3. Context Inpainting with Pixel Preservation Given an image x and its preserve mask M , we generate a synthetic counterpart by inpainting only the complementary region M̄ = 1 − M . Let G : R3×H×W × {0, 1}H×W × T → R3×H×W
(5)
be a pre-trained latent diffusion inpainter that takes an image, a binary inpainting mask, and a text prompt t ∈ T , and returns an image in which the masked region is regenerated. We apply G to the complement M̄ of the preserve mask M , xgen = G x, M̄ , t , (6)
j=1
where k is the smallest index such that 1 τ . M (x, y; τ ) is itself binary HW M(x, y; τ ) 1 ≥ since the class masks are disjoint. The threshold τ ∈ (0, 1] controls a single, intuitive trade-off: smaller τ concentrates the synthetic-context budget on the hardest few classes per image, while larger τ preserves more of the original image at the cost of less novel context. τ is introduced as a hyperparameter and we study its effect in Sec. 4. By construction, M is image-adaptive: in an urban scene where the baseline confuses bus and train pixels, those pixels are preserved. In a scene where the baseline is uncertain along a wall–fence boundary, those pixels are preserved instead. The selector therefore allocates the augmentation budget along an informational axis rather than the foreground/background axis of prior work [15, 18].
using a single generic, dataset-level prompt t (e.g. describing the imaging domain, see Appendix 7.5). In contrast to prior preserve-and-regenerate methods [15, 18], we deliberately employ no class-specific prompting, no ControlNet [38] conditioning, and no mask-refinement. To guarantee label validity, we only utilize the preserve mask itself and two generic post-processing steps described below. 3.3.1. Paste-Back to Preserve Uncertain Pixels. Latent diffusion inpainters operate by encoding the input image into a VAE latent, modifying the noised latent 4
through the masked region, and decoding the result back to pixel space [29]. This encode–decode roundtrip is not lossless: even pixels outside M̄ , which the inpainter is not supposed to change, undergo small reconstruction errors [1, 21] that perturb their values (see Figure 3). This is especially problematic for uncertain regions which are often small and depict rare objects, leading to larger reconstruction errors. If left uncorrected, this drift would silently invalidate the labels we are trying to preserve. We therefore perform an explicit, pixel-exact paste-back step that restores the original values inside M : x̃ = M̄ ⊙ xgen + M ⊙ x,
Original Image
Preserve Area
xgen (No Paste-Back)
x̃ (With Paste-Back)
|x̃ – xgen| (VAE Drift)
(7)
where ⊙ denotes per-pixel multiplication broadcast over the channel dimension. I.e., the resulting image x̃ is bitwise identical to x inside M and identical to xgen outside it. 3.3.2. Synthetic Label Maps that Ensure Label Validity
Figure 3. Pixel-exact paste-back preserves label validity.
To use the generated images for fine-tuning of fθ0 , we construct a synthetic dataset Dsyn . Given a real sample (x, y) ∈ D, we create a synthetic image x̃ and define a corresponding label map ỹ that assigns the original groundtruth class to every pixel within the preserve mask M and the dataset ignore index to every generated pixel, ( yi,j if Mi,j = 1, ỹi,j = (8) ι otherwise.
1024 × 2048), UAVID [22] (aerial urban scenes, 8 classes, ∼200 training / 70 validation images at 4096 × 2160), and BDD100K [37] (urban driving in diverse weather and times of day, 19 classes, 7000 training / 1000 validation images at 720 × 1280). The three datasets span different viewpoints (street-level vs. aerial), different geographic distributions, and different scene complexities, providing a more stringent test of generalization than any single benchmark. Similar to our baselines [15, 18], we utilize fractions of the training data to evaluate low-data scenarios, but always use all available validation samples.
Together, Eqs. (1), (7) and (8) ensure label validity throughout training: Eq. (7) restores the exact RGB values of every supervised pixel, Eq. (8) assigns valid labels only to those restored pixels and marks all generated pixels as ignore, and Eq. (1) excludes ignored pixels from the loss. Consequently, synthetic context influences the training through patch-wise self-attention, but generated pixels never serve as direct training targets.
Implementation Details. For segmentation, we use a DINOv2 [26] pre-trained ViT [5] backbone paired with a linear semantic decoder. Both the real-data baseline checkpoint θ0 and the fine-tuned segmenters θ1−3 are trained with AdamW [20] and a cosine learning-rate schedule. For all experiments, we report mean validation mIoU with standard deviation over 5 seeds. Fine-tuning starts from θ0 on the union D ∪ Dsyn . For inpainting, we use a black-box latent diffusion inpainter with 40 denoising steps, classifier-free guidance scale 7.0, and no ControlNet or auxiliary conditioning. We choose SDXL-Inpaint1.0 [28] for a fair comparison with baselines [15, 18], but note that any off-the-shelf inpainting model can be used. Cityscapes images are randomly cropped to 1024 × 1024; UAVID images are processed by cropping to a random 2048 × 2048 region, bilinear-resizing to 1024 × 1024 for generation, and upsampling back; BDD100K is processed at 1024 × 1024 after letterbox padding. The text prompt is a single dataset-level descriptor (e.g. photorealistic, ultra-detailed,[...] in the style of the Cityscapes dataset, see Appendix 7.5). We run our approach for a maximum of three iterations since
3.4. Iterative Segmenter Refinement Finally, we propose to improve the baseline segmenter fθ0 by fine-tuning it on D ∪ Dsyn , which now specifically contains samples that are challenging for fθ0 . One iteration of our approach is shown in Algorithm 1. Our approach can be repeated with the improved segmenter fθ1 : recompute entropy, select the currently most informative regions, synthesize a new Dsyn , and fine-tune again. This yields an active-learning-style sequence fθ0 , fθ1 , . . . , fθT in which each round automatically targets the latest model’s remaining failure modes, until performance saturates.
4. Experiments 4.1. Experimental Setup Datasets. We evaluate on Cityscapes [3] (urban driving, 19 semantic classes, 2975 training / 500 validation images at 5
Table 1. Main results on Cityscapes 10% (mean ± std over five seeds), with representative rare and frequent classes by training-pixel share. Improvements concentrate on rare classes. Rare Classes Method
Table 2. Main results on BDD100K 10% (mean ± std over five seeds).
Frequent Classes
mIoU
truck
bus
train
road
sidewalk vegetation
Real data only Simple Background Augmentation [18] Instance Augmentation [15]
69.60 70.29 ± 0.43 70.71 ± 0.55
72.80 70.36 73.66
56.83 58.64 57.16
56.04 56.63 56.03
97.93 97.85 98.00
82.25 81.71 82.40
90.86 90.96 90.96
Ours, iteration 1 Ours, iteration 2 Ours, iteration 3
71.67 ± 0.45 77.80 68.04 57.12 72.09 ± 0.67 77.02 67.82 62.59 72.24 ± 0.18 77.47 68.77 65.28
97.90 97.91 97.92
82.03 82.08 82.26
90.59 90.68 90.78
Method
mIoU
Real data only Simple B. A. [18] Instance Aug. [15]
59.60 59.72 ± 0.25 60.45 ± 0.13
Ours, iteration 1 Ours, iteration 2 Ours, iteration 3
60.57 ± 0.14 61.00 ± 0.26 61.17 ± 0.16
Table 3. Main results on UAVID 100% (mean ± std over five seeds), with all individual class results. Our method achieves the best results for every class and is particularly effective for rare classes by training-pixel share. Rare Classes Method
mIoU
Frequent Classes
moving car static car human building
road
tree
low veg
Real data only Simple Background Augmentation [18] Instance Augmentation [15]
60.31 61.49 ± 0.35 62.24 ± 0.25
57.05 58.99 60.79
59.58 63.81 64.84
28.72 31.61 33.24
94.98 95.06 95.52
87.32 87.19 88.09
81.18 81.01 81.35
73.66 74.25 74.10
Ours, iteration 1 Ours, iteration 2 Ours, iteration 3
62.58 ± 0.45 63.48 ± 0.34 63.99 ± 0.22
62.47 64.57 65.80
64.90 68.19 69.28
31.38 32.72 33.32
95.42 95.56 95.54
88.09 82.47 88.47 83.06 88.61 82.89
75.94 77.20 77.08
4.3. Effect of Training-Set Size
we empirically observed saturation after more iterations. If not stated otherwise, we use preserve region fraction τ = 0.10.
Data scarcity is central to our motivation, so we study how the benefit of uncertainty-guided augmentation scales with the amount of available real data. Table 4 reports Cityscapes results for train splits of 5%, 10%, 33%, and 100%, comparing the real-only baseline, the strongest spatial baseline (Instance Augmentation [15]), and our method. The gain is consistent across all settings and remains substantially ahead of the baseline across the full range.
Baselines. We compare against: (i) the no-synthetic-data baseline trained only on real data D; (ii) Instance Augmentation [15], which regenerates all foreground object instances using a stack of auxiliary models; (iii) Simple Background Augmentation [18], which regenerates the entire background given a set of foreground objects. Here, we assign foreground/background based on classes (e.g. car as foreground and wall as background, see Appendix 8).
4.4. Ablation Studies
Code. https : / / github . com / XITASO / Preserve-the-Hard-Regenerate-the-Rest
Region Selection and Inpainting. The central claim of our method is that the informational axis, not the spatial one, is the right axis along which to allocate synthetic context. To test this, we keep the entire downstream pipeline (inpainting, paste-back, ignore-mask, fine-tuning) identical, varying only the rule that selects which region is preserved (Tab. 5): (i) a random square crop of area τ ; (ii) randomly chosen ground-truth classes summing to area τ ; (iii) the most-confident classes, i.e. the inverse of our criterion; (iv) our most-uncertain classes but without inpainting — the original context is kept and the loss is simply restricted to the preserved region; and (v) our full method, most-uncertain classes with inpainted context. We find that selecting most-confident classes yields near-zero gain, and that region selection by random cropping or random class selection significantly falls behind our method. Also, we find that inpainting accounts for a large portion of the
4.2. Main Results Tables 1 to 3 report our main results against both spatial preserve-and-regenerate augmentation baselines on the three datasets. On all datasets, we find consistent gains over the no-augmentation reference (+3.68/+2.64/+1.57 mIoU on UAVID / Cityscapes / BDD) vs. only +1.93/ + 1.11/ + 0.85 for the best baseline. Notably, we more than double the gain over the strongest baseline on Cityscapes. A per-class breakdown verifies the intuition that classes that are rare by proportion of annotated training pixels benefit most from our method: we obtain up to +11.94/ + 9.24/ + 5.00 IoU on bus / train / truck (Cityscapes, Tab. 1) and up to +8.75/ + 9.70/ + 4.60 IoU on moving car / static car / human (UAVID, Tab. 3). 6
Table 4. Effect of training-set size on Cityscapes (5 seed mean validation mIoU ± std).
Table 8. Preserve-area threshold sweep on Cityscapes 10% (5 seed mean validation mIoU ± standard deviation, single iteration).
Split
Real only
Instance Aug. [15]
Ours, iteration 1
Ours, iteration 2
Ours, iteration 3
τ
5% 10% 33% 100%
61.42 69.60 72.54 75.46
62.37 ± 0.14 70.71 ± 0.55 75.05 ± 0.18 76.78 ± 0.14
64.14 ± 0.18 71.67 ± 0.45 75.44 ± 0.21 76.85 ± 0.21
64.23 ± 0.11 72.09 ± 0.67 76.36 ± 0.06 77.38 ± 0.24
64.64 ± 0.18 72.24 ± 0.18 76.77 ± 0.16 77.69 ± 0.22
mIoU
mIoU
truck
bus
train
Real data only Confident classes (τ ) Random square region (τ ) Random classes (τ )
69.60 70.06 ± 0.35 70.53 ± 0.42 70.61 ± 0.20
72.80 74.19 74.80 75.08
56.83 53.74 54.29 59.69
56.04 59.45 62.81 59.89
Uncertain classes, no inpainting
70.13 ± 0.11
71.45
63.66
56.52
Uncertain classes + inpainting (Ours) 71.67 ± 0.45 77.80 68.04
57.12
0.10
0.15
0.20
0.25
71.67 ± 0.45
71.29 ± 0.15
71.16 ± 0.28
70.54 ± 0.32
Table 9. Synthetic-to-real sampling ratio on Cityscapes 10% (5 seed mean validation mIoU ± standard deviation, single iteration).
Table 5. Region-selection ablation on Cityscapes 10% (5 seed validation mIoU ± std, single augmentation iteration). Selection rule
0.05 71.00 ± 0.40
Syn : Real mIoU
0:1 (baseline)
1:2
1:1
1.5:1
2:1
69.60
71.40 ± 0.59
71.67 ± 0.45
71.44 ± 0.14
71.28 ± 0.31
4.4.1. Compute Analysis Table 10 reports per-sample wall-clock time on a single NVIDIA A100 40GB for the full pipeline. The cost of our method is dominated by the SDXL inpainting forward pass; the uncertainty-based region selection, the paste-back, and the label construction are essentially free. This means that the additional uncertainty-based component we add over a vanilla inpainting pipeline carries negligible overhead.
achieved gains and is necessary to place hard regions in novel visual context.
4.5. Generalization Across Backbone and Inpainter
Paste-Back & Ignore-Mask. In combination, the pasteback step (Eq. (7)) and the ignore-mask (Eq. (8)) guarantee that no label noise is introduced. To isolate their effect, we compare all four possible configurations: with and without original-pixel paste-back, and with and without the synthetic-pixel ignore mask applied to the inpainted region. Table 6 shows that paste-back and ignore-mask contribute roughly equally, but that both fall behind our full method.
To confirm that our gains are not tied to a single architecture, we vary the backbone and the inpainter independently (Tab. 7). Swapping the DINOv2 ViT for a SegFormer [34] backbone preserves the improvement, indicating that the method is segmenter-agnostic. Replacing the SDXL inpainter with FLUX [2] yields marginal gain, suggesting that any sufficiently capable off-the-shelf inpainter suffices.
Preserve-Area Threshold τ . The threshold τ controls the trade-off between supervision density (more preserved pixels per image) and contextual diversity (more pixels regenerated). Table 8 sweeps τ ∈ {0.05, 0.10, 0.15, 0.20, 0.25}. Gains occur across the full range, with the best setting at τ = 0.10. This suggests that selecting a small set of highuncertainty regions yields a greater benefit than coarsely selecting uncertain regions.
Figures 4a and 4b show uncurated synthetic samples and we provide more uncurated samples for Cityscapes and UAVID in Appendix 6. The preserved region remains identical, while the surrounding scene provides novel context, e.g., different lighting, season, and street layout. Although artifacts can occur in the inpainted regions, those pixels are excluded from the loss by definition.
Real-to-Synthetic Sampling Ratio. Table 9 sweeps the fine-tuning ratio of synthetic to real samples. 1:1 is best, i.e. our method is most useful as a substantial complement.
Table 10. Per-sample wall-clock breakdown of the augmentation pipeline on a single NVIDIA A100 40GB at 1024 × 1024 generation resolution. Region selection contributes negligible overhead relative to the SDXL inpainting forward pass.
Table 6. Effect of eliminating label noise (Cityscapes 10%). Both ignore mask and pixelexact paste-back contribute. Ignore Paste-back ✗ ✗ ✓ ✓
✗ ✓ ✗ ✓
mIoU 69.10 ± 0.62 70.74 ± 0.33 70.98 ± 0.29 71.67 ± 0.45
4.6. Qualitative Results
Operation
Table 7. Generalization across backbone and inpainter (Cityscapes 10%, mIoU ± 5 seed std, single iteration). Backbone
Inpaint Base
Ours
ViT (DINOv2) SDXL 69.6 71.7 ± 0.5 ViT (DINOv2) FLUX 69.6 71.9 ± 0.4 SegFormer
SDXL 67.3 68.6 ± 0.1
7
Time (s)
% of total
Segmenter forward (entropy) Per-class entropy aggregation Greedy class selection
0.026 0.003 0.000
0.42 0.04 <0.01
Region selection subtotal
0.029
0.46
SDXL inpainting (40 steps) Paste-back & label construction
6.230 0.049
98.77 0.77
Total per sample
6.308
100.00
Original Image
Uncertainty Map
Preserved Region
Original Image
Inpainted Image
Uncertainty Map
Preserved Region
Inpainted Image
Uncertainty Map
Preserved Region
Inpainted Image
(a) Cityscapes (street-level driving) Original Image
Uncertainty Map
Preserved Region
Original Image
Inpainted Image
(b) UAVID (aerial urban scenes)
Figure 4. Uncurated qualitative results across both domains. For each example, the columns show the original image, the baseline segmenter’s per-pixel predictive-entropy map, the preserve mask formed from the most uncertain classes, and the final augmented sample after inpainting and pixel-exact paste-back. Generation uses dataset-level prompts: “photorealistic, ultra-detailed, 4K high-resolution, sharp focus, high quality, in the style of the Cityscapes dataset” for Cityscapes and “photorealistic aerial drone image, urban scene, highresolution, realistic lighting, sharp overhead perspective” for UAVID. In both domains the preserved pixels remain bitwise identical to the source, so context diversity is injected without perturbing a single supervised pixel; samples are drawn at random without curation and a larger quantity of uncurated samples is provided in the Appendix.
Limitations. Our method requires (i) the baseline segmenter to compute entropy at every iteration and (ii) an SDXL-class diffusion inpainter for synthetic sample generation. The compute cost is dominated by the inpainter (Tab. 10). On domains where SDXL has weak priors (e.g. medical imaging) we expect consistent but smaller gains. Domain-specific inpainters could mitigate this problem.
lations demonstrate that all components of the approach are necessary and contribute to the results: Uncertainty-based region selection, context inpainting and pixel-exact pasteback combined with an ignore-mask loss each contribute to the final gain. These results suggest a clear use-case: our proposed method is most beneficial when segmentation performance is limited by scarce or imbalanced dense annotations, and especially when the challenging classes occupy few pixels. It is particularly suitable for saftey-critical settings where synthetic augmentation must not corrupt the labels and labeled synthetic data is difficult to trust.
5. Conclusion This paper revisited preserve-and-regenerate augmentation for semantic segmentation and showed that allocating the synthetic-data budget by model uncertainty is more effective than using fixed spatial rules such as foreground or background regeneration. Across three street-level and aerial benchmarks, our method substantially improves segmenter performance over real-data training and outperforms spatial preserve-and-regenerate baselines. The improvements are largest for challenging classes with underrepresented or small objects with few annotated pixels. Our ab-
Our method is deliberately simple, resource effective and architecture-agnostic, as we verify by transferring it across segmenter backbones and inpainters. Rather than relying on fully synthetic data or deciding with which spatial heuristic to inpaint, one can simply preserve what is hard and regenerate the rest. 8
Acknowledgements This work was carried out in the context of the ENGEL project, funded from the Federal Ministry of Economic Affairs and Energy (BMWE) of Germany through the Federal Aviation Research Program (LuFo) VI-3, under FKZ 20F2201D. This paper is also supported by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Research, Technology and Space.
[10] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning, pages 1183–1192. PMLR, 2017. 2, 3 [11] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1 [12] Khawar Islam, Muhammad Zaigham Zaheer, Arif Mahmood, and Karthik Nandakumar. Diffusemix: Labelpreserving data augmentation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27621–27630, 2024. 2 [13] Joel Janai, Fatma Güney, Aseem Behl, and Andreas Geiger. Computer vision for autonomous vehicles: Problems, datasets and state of the art. Foundations and Trends in Computer Graphics and Vision, 12(1-3):1–308, 2020. 1 [14] Hannah Kniesel, Pedro Hermosilla, and Timo Ropinski. Active learning inspired controlnet guidance for augmenting semantic segmentation datasets. arXiv preprint arXiv:2503.09221, 2025. 3 [15] Orest Kupyn and Christian Rupprecht. Dataset enhancement with instance-level augmentations. In European Conference on Computer Vision, pages 384–402. Springer, 2024. 2, 3, 4, 5, 6, 7, 13 [16] Min Seok Lee, WooSeok Shin, and Sung Won Han. Tracer: Extreme attention guided salient object tracing network (student abstract). In Proceedings of the AAAI conference on artificial intelligence, pages 12993–12994, 2022. 3 [17] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022. 2, 3 [18] Yuhang Li, Xin Dong, Chen Chen, Weiming Zhuang, and Lingjuan Lyu. A simple background augmentation method for object detection with diffusion model. In European Conference on Computer Vision, pages 462–479. Springer, 2024. 2, 3, 4, 5, 6, 13 [19] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 1 [20] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR, 2019. 5 [21] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022. 2, 5 [22] Ye Lyu, George Vosselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery. ISPRS journal of photogrammetry and remote sensing, 165:108–119, 2020. 2, 5 [23] Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, and Carsten Rother. CEREALS - costeffective region-based active learning for semantic segmen-
References [1] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM transactions on graphics (TOG), 42 (4):1–11, 2023. 5 [2] Black Forest Labs. Flux.1 fill [dev]. Hugging Face model card, 2024. Accessed 2026-06-08. 7 [3] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016. 1, 2, 3, 5, 13 [4] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268–9277, 2019. 1 [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 5 [6] Gregory Druck, Burr Settles, and Andrew McCallum. Active learning by labeling features. In Proceedings of the 2009 conference on Empirical methods in natural language processing, pages 81–90, 2009. 2, 3 [7] Haoyang Fang, Boran Han, Shuai Zhang, Su Zhou, Cuixiong Hu, and Wen-Ming Ye. Data augmentation for object detection via controllable diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1257–1266, 2024. 3 [8] Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions on Intelligent Transportation Systems, 22(3):1341–1360, 2020. 1 [9] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning, pages 1050–1059. PMLR, 2016. 2
9
[34] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34: 12077–12090, 2021. 7 [35] Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. Freemask: Synthetic images with dense annotations make stronger segmentation models. Advances in Neural Information Processing Systems, 36:18659–18675, 2023. 1, 3 [36] Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 93–102, 2019. 2, 3 [37] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636–2645, 2020. 2, 5 [38] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023. 2, 3, 4 [39] Hanqing Zhao, Dianmo Sheng, Jianmin Bao, Dongdong Chen, Dong Chen, Fang Wen, Lu Yuan, Ce Liu, Wenbo Zhou, Qi Chu, Weiming Zhang, and Nenghai Yu. X-paste: Revisiting scalable copy-paste for instance segmentation using CLIP and StableDiffusion. In Proceedings of the 40th International Conference on Machine Learning, pages 42098– 42109. PMLR, 2023. 2 [40] Haowei Zhu, Tianxiang Pan, Rui Qin, Jun-Hai Yong, and Bin Wang. Recon: Region-controllable data augmentation with rectification and alignment for object detection. Advances in Neural Information Processing Systems, 38:74897–74926, 2025. 3
tation. In British Machine Vision Conference 2018, BMVC, page 121. BMVA Press, 2018. 2 [24] Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation. Advances in Neural Information Processing Systems, 36:76872–76892, 2023. 1, 3 [25] Joshua Niemeijer, Jan Ehrhardt, Heinz Handels, and Hristina Uzunova. Uncertainty-aware controlnet: Bridging domain gaps with synthetic image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4184–4193, 2025. 3 [26] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. 5 [27] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023. 1, 3 [28] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, pages 1862–1874, 2024. 2, 5 [29] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 3, 5 [30] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761–769, 2016. 2 [31] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. In International Conference on Learning Representations, pages 14590–14612, 2024. 3 [32] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. Advances in Neural Information Processing Systems, 36:54683–54695, 2023. 1, 3 [33] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1206–1217, 2023. 1, 3
10
Appendix 6. Additional Uncurated Qualitative Samples
ation, and fine-tuning all draw from the same fixed training image IDs.
We show more examples that are sampled without any qualitative filtering in Figure 5 and Figure 6. The columns show the original image, the predictive-entropy map, the preserved uncertain region, and the final augmented image after inpainting and paste-back. Every preserved pixel in the augmented image is bit-wise identical to the source.
Dataset-specific preprocessing. Generation for all datasets operates on 1024 × 1024 pixel images. Cityscapes crops are taken from the native 1024 × 2048 frame; BDD100K is zero-padded to 1024 × 1024. For UAVID, each native 4096 × 2160 frame is randomly cropped to a 2048 × 2048 image and then resized to 1024 × 1024 (bilinear for RGB, nearest-neighbor for labels).
7. Additional Implementation Details This section specifies everything needed to reproduce the method and results in the main paper: the fixed data subsets, preprocessing, the training and fine-tuning schedules, generation settings, and the checkpoint-selection rule.
7.2. Segmentation Model and Baseline Training The segmenter fθ pairs the DINOv2-pretrained ViT encoder with a linear per-patch decoder. The decoder is a single fully connected linear layer, mapping each DINOv2 ViT patch token from 384 channels directly to C class logits. The encoder is trainable and both modules are optimized end to end under the ignore-index cross-entropy loss. We report training details in Table 11.
7.1. Data Splits and Reproducibility The training data splits used in our experiments are fixed independently of the training seed. For each dataset D, we sort all pairs of image and label map (x, y) ∈ D by filename, shuffle once, and keep the first ⌊ρN ⌋ pairs for split fraction ρ (e.g., ρ = 0.1 for the 10% Cityscapes split). The different training seeds that we used for mean and standard deviation results therefore control model initialization, data-loader order, augmentation randomness, optimizer behaviour etc.. They do not change which images are used: baseline training, preserve region selection, synthetic gener-
Training augmentation. In addition to our proposed synthetic data augmentation, we also utilize simple train-time augmentation. Each sample undergoes photometric jitter, a random horizontal flip and scale jitter.
7.3. Fine-Tuning on D ∪ Dsyn Setting
Value
Backbone Decoder Encoder update Training crop Optimizer Decoder peak LR Encoder peak LR Loss
DINOv2 ViT Linear semantic decoder Trainable; LR multiplier 0.1 1024 × 1024 AdamW 1 × 10−4 1 × 10−5 Ignore-index masked Crossentropy Cosine decay None 0.05 4 per GPU 4 steps (effective batch 16) 16-bit mixed precision 20,000 optimizer steps Every 1,000 optimizer steps 16, pinned memory A100 40GB
LR schedule Warmup Weight decay Batch size Gradient accumulation Precision Maximum length Validation interval Data-loader workers Hardware
Fine-tuning starts from the real-data-only checkpoint θ0 . We concatenate the real dataset D and the synthetic dataset Dsyn , then shuffle them together. We report fine-tuning details in Table 12.
Setting
Value
Initialization Training set Synthetic : real ratio Decoder peak LR Encoder peak LR LR schedule Maximum length Validation interval Encoder update Other settings
θ0 D ∪ Dsyn 1:1 2 × 10−5 2 × 10−6 Cosine decay; restarted 20,000 optimizer steps Every 1,000 optimizer steps Trainable Same as real-only training
Table 12. Fine-tuning configuration for the augmented training iterations. The optimizer and LR scheduler are reinitialized each iteration.
Table 11. Real-only baseline training configuration.
11
Original Image
Uncertainty Map
Preserved Region
Inpainted Image
Original Image
Uncertainty Map
Preserved Region
Inpainted Image
Original Image
Uncertainty Map
Preserved Region
Inpainted Image
Preserved Region
Inpainted Image
Figure 5. Additional uncurated qualitative samples on Cityscapes (street-level driving). Original Image
Uncertainty Map
Preserved Region
Inpainted Image
Original Image
Uncertainty Map
Preserved Region
Inpainted Image
Original Image
Uncertainty Map
Figure 6. Additional uncurated qualitative samples on UAVID (aerial urban scenes).
7.4. Diffusion Inpainting and Generation
ban driving scene, high-resolution, realistic lighting and weather, sharp focus.” • UAVID: “photorealistic aerial drone image, urban scene, high-resolution, realistic lighting, sharp overhead perspective.”
We use the public SDXL-Inpaint-1.0 checkpoint (diffusers/stable-diffusion-xl-1.0-inpai nting-0.1) with no ControlNet, classifier guidance, or auxiliary spatial conditioning. All cropped image inputs are square, so the SDXL aspect bucket is always 1024 × 1024. In general the implementation selects the nearest canonical SDXL bucket, resizes RGB bilinearly and labels/masks by nearest-neighbor, and restores the cached resolution after generation. Each dataset uses a single positive and single negative prompt, with no per-image or per-class text.
Negative prompts. • Cityscapes: “blurry, low quality, deformed, melted structures, floating objects, cartoon, illustration, unrealistic shadows, out of perspective, wrong scale.” • BDD100K: “blurry, low quality, deformed vehicles, melted structures, cartoon, illustration, unrealistic shadows, wrong scale, watermark, text.” • UAVID: “blurry, low quality, cartoon, illustration, distorted vehicles, warped roads, duplicated objects, unrealistic shadows, text, watermark.”
7.5. Generation Prompts Positive prompts. • Cityscapes: “photorealistic, ultra-detailed, 4K highresolution, sharp focus, high quality, in the style of the Cityscapes dataset.” • BDD100K: “photorealistic dashcam image, diverse ur12
Table 13. Canonical foreground/background split used for baseline implementations.
7.6. Region Selection and Mask Construction As described in the main paper, for each non-ignore groundtruth class present in the image we average the entropy over the class’s pixels, sort classes by decreasing mean entropy, and add them until their union exceeds τ HW pixels. The resulting preserve area can therefore exceed τ when the last added class is large. Further details:
Dataset
Foreground
Background
Cityscapes / person, rider, car, road, sidewalk, buildBDD100K truck, bus, train, ing, wall, fence, pole, traffic light, traffic motorcycle, bicycle sign, vegetation, terrain, sky, ignore
• Equal-entropy ties break by ascending label ID (stable sort). • No minimum object-area filter is applied. • In contrast to baselines [15, 18], no preserve-mask dilation or inpaint-mask erosion is applied. • Ignore-index pixels are excluded from the entropy calculation and cannot be selected for the preserve mask. • For BDD100K, padding and cropping are done before entropy scoring. The resulting image size is 1024 × 1024 pixels. • UAVID entropy is computed on each 1024 × 1024 image; there is no tiling, overlap, blending, or full-frame aggregation.
UAVID
static car, moving car
human, clutter, building, road, tree, vegetation, ignore
8. Foreground/Background Split For Baseline Methods The two synthetic augmentation baselines used in our experiment rely on spatial heuristics based on foreground objects and the background of each image. To apply this to densely labeled semantic segmentation use-cases like Cityscapes [3], we assign each semantic class to either foreground or background. The exact classes are listed in Table 13.
7.7. Validation and Reported Metrics During evaluation, no random crop augmentation is applied. Each validation image is resized isotropically such that both spatial dimensions are at least 1024 pixels; equivalently, the shorter side is brought to 1024 if needed. The resized image is then evaluated using overlapping 1024 × 1024 windows along the longer dimension. Logits from overlapping windows are averaged, and the resulting logit map is bilinearly resized back to the original image resolution before computing IoU. Per-class IoU is computed from a single global confusion matrix over the full validation set, excluding ignore pixels, and mIoU is the unweighted mean over semantic classes. We validate once before the first optimizer step, every 1,000 steps thereafter, and at termination, retaining the highest-mIoU checkpoint. Across training seeds we report the arithmetic mean and standard deviation of the per-seed best mIoU.
7.8. Iterative Refinement Protocol We run three fixed rounds of creating synthetic training data with our approach. Each round t computes entropy with segmenter fθt−1 , generates a fresh synthetic set from the fixed real image IDs, and then initializes fine-tuning from the round-(t − 1) checkpoint. Earlier synthetic sets are discarded rather than accumulated. The default fine-tuning LR is 2 × 10−5 (decoder) and 2 × 10−6 (encoder). 13