arXiv CSopen access

Weakly Supervised Segmentation as Semantic-Based Regularization

Weakly Supervised Segmentation as Semantic-Based Regularization Stefano Colamonaco∗1 , Andrei-Bogdan Florea2 and Jaron Maene1

arXiv:2605.13674v1 [cs.CV] 13 May 2026

1

KU Leuven, [email protected], [email protected] 2 Apixa, [email protected]

Abstract Weakly supervised semantic segmentation (WSSS) trains dense pixel-level segmentation models from partial or coarse annotations such as bounding boxes, scribbles, or image-level tags. While recent work leverages foundation models such as the Segment Anything Model (SAM) to generate pseudo-labels, these approaches typically depend on heuristic prompt choices and offer limited ways to incorporate prior knowledge or heterogeneous labels. We address this gap by taking a neurosymbolic perspective: integrating differentiable fuzzy logic with deep segmentation models. Weak annotations and domain-specific priors are unified as continuous logical constraints that fine-tune SAM under weak supervision. The refined foundation model then produces improved pseudo-labels, from which we train a second-stage prompt-free segmentation model. Experiments on Pascal VOC 2012 and the REFUGE2 optic disc/cup segmentation dataset show that our logic-guided fine-tuning yields higher-quality pseudo-labels, leading to state-of-the-art segmentation accuracy that often exceeds densely supervised baselines.

1. Introduction Semantic segmentation is a fundamental task in computer vision, enabling a pixel-level understanding of visual scenes. Despite remarkable progress, state-of-the-art methods still depend on large datasets with densely annotated masks, which is an expensive and time-consuming requirement that limits scalability. Producing pixel-perfect labels demands substantial manual effort, creating a major bottleneck for deploying segmentation models in new domains. To reduce this dependence on dense supervision, Weakly Supervised Semantic Segmentation (WSSS) has emerged as a practical alternative [8]. Instead of relying on full masks, WSSS methods learn from cheaper, indirect cues such as image-level tags [54], bounding boxes [26, 40], or sparse scribbles [29]. These weak labels lower annotation costs but introduce ambiguity about object boundaries and shape, making it difficult for models to produce coherent segmentations without additional structural guidance [52]. Recently, the landscape of WSSS has been reshaped by foundation models such as the Segment Anything Model (SAM) [24]. Thanks to their prompt-based design and extensive pretraining, foundation models can generate high-quality pseudo-masks from simple user inputs, providing a powerful prior that many WSSS pipelines now exploit. However, this paradigm is limited by its reliance on fixed, pre-trained models. When applied to specialized domains (e.g., medical imaging) these models often struggle with domain shifts and object appearances unseen during their original training. Moreover, their prompt-driven nature provides no principled way to ∗

Corresponding author.

1

Weak labels

Prompt

Structural priors

Stage 1: Logic-guided SAM finetuning

Stage 2: Pseudo-label supervised training

SAM

Segmentation model

Pseudo labels

Smoothness, superpixel boundaries, etc. Constraints

Fuzzy Loss (Eq. 2)

Segmentation Loss

Figure 1: Overview of the proposed neurosymbolic weakly supervised segmentation framework. The method operates in two stages: First, the Segment Anything Model (SAM) is fine-tuned using differentiable logic constraints derived from weak annotations and general structural priors to generate refined pseudo-labels. Second, these pseudo-labels are used as ground truth to train a standard segmentation network for final deployment. incorporate domain knowledge or symbolic constraints that could guide or correct the predictions. As a result, several current approaches still depend on heuristic prompt design rather than a general mechanism for reasoning about spatial and semantic structure. Neurosymbolic learning offers a promising direction for addressing this gap by combining the flexibility of deep networks with the interpretability and compositionality of symbolic reasoning [36]. Through differentiable logic or constraint-based formulations, neurosymbolic models can integrate structured knowledge directly into the learning process, enforcing consistency, relational structure, or expert priors in a way that remains trainable end-to-end. While most existing efforts apply such reasoning at the object or scene level [35, 11], extending it to the pixel level opens the door to a neurosymbolic segmentation guided by explicit relational rules, a direction that has remained largely unexplored. Building on this idea, we introduce a neurosymbolic approach for weakly supervised semantic segmentation that fine-tunes a foundation segmentation model using multiple weak annotation types simultaneously, expressed as differentiable fuzzy logic constraints. In contrast to promptbased adaptation or single-label weak supervision, our approach unifies heterogeneous supervision signals, such as bounding boxes and scribbles, within a single, end-to-end optimization scheme. By grounding logical constraints on these weak cues, the model learns not only to fit annotations but also to satisfy high-level spatial and relational priors. Practically, the method follows a two-stage pipeline. In the first stage, SAM is fine-tuned through differentiable logic losses derived from weak annotations, yielding refined and structurally consistent pseudo-masks. In the second stage, these pseudo-masks are used as training data for a prompt-free segmentation network, enabling scalable deployment without human prompts. In summary, the contributions of this work are (i) a neurosymbolic formulation of weakly supervised segmentation using differentiable fuzzy logic, (ii) a general training scheme that 2

fine-tunes SAM to integrate multiple weak annotations and priors jointly, and (iii) empirical evidence that such reasoning-guided learning can achieve state-of-the-art performance on standard segmentation benchmarks. 1

2. Related Work Weak supervision. WSSS has been studied extensively with different annotation types. Early work explored bounding boxes [26, 40, 46, 21], scribbles [29], or point-level cues [51] as alternatives to dense masks. Moreover, many methods exploited class activation maps (CAMs) for image-level labels [7] or to propagate labels from sparse scribbles [39]. For instance, TEL [27] optimizes a tree energy loss for scribble supervision, while AGMM [51] employs adaptive Gaussian mixtures. Concurrently, the field has seen a surge in methods leveraging vision-language models such as CLIP to extract dense pixel-level affinities from simple Image-Level Tags (ILT), as demonstrated by CLIP-ES [30] and CLIP-CPAL [45]. However, a primary limitation of these approaches is that they are highly tailored to a single, specific type of weak supervision (e.g., only image-level tags or only scribbles), making them inflexible when diverse annotation types are available. To integrate multiple forms of weak supervision, Xu et al. [52] proposed a unified framework that leverages multiple weak cues (tags, bounding boxes, and partial labels) via a latent variable Conditional Random Field (CRF). However, the complexity of its alternating optimization made it difficult to scale, leading later WSSS methods to revert to disjointed pipelines. We overcome this bottleneck by replacing intractable structured prediction with differentiable fuzzy logic. Foundation models. The recent shift of task-specific models to foundation models has also been adopted in image segmentation. A notable example is the Segment Anything Model (SAM) [24], which was trained on a billion-mask dataset and generates accurate segmentation masks from simple prompts (points or boxes). Jiang and Yuqi [22] showed that prompting SAM with weak labels yields high-quality pseudo-labels that can be used to train a segmentation network in a weakly supervised fashion. Although promptable foundation models often outperform traditional pipelines, reliance on a fixed pre-trained model can limit adaptability to out-of-distribution domains [37]. Because of this, several works have fine-tuned SAM using large-scale, densely annotated datasets. For instance, MedSAM [31] and SAM-Med2D [10] adapt SAM to a wide variety of medical modalities, while methods like SAMed [58] employ low-rank adaptation (LoRA) to efficiently fine-tune the image encoder for medical tasks. However, these specialized models still require pixel-perfect ground truth for fine-tuning. In contrast, our framework adapts foundation models to out-of-distribution domains using only weak supervision. Neurosymbolic Learning. Another orthogonal direction is to integrate prior knowledge and constraints directly into learning [13, 33]. The neurosymbolic literature has explored encoding logical or fuzzy constraints in neural training. For instance, semantic-based regularization [15] and the semantic loss [53] incorporate logical rules as soft losses. Probabilistic logic programming languages such as DeepProbLog [34] allow rich symbolic priors but also face tractability issues for high-dimensional problems such as segmentation [32]. In contrast, differentiable fuzzy logic frameworks remain tractable in the propositional case [18, 3, 48]. Building on these tractable frameworks, neurosymbolic methods have begun to be adapted for visual tasks. For weak supervision in classification, Shukla et al. [43] introduced a unified 1

The source code is available at https://github.com/StefanoColamonaco/Logic-Guided-Segmentation

3

differentiable loss that handles multiple types of weak labels, while FasterLTN [35] integrates a Faster R-CNN backbone with Logic Tensor Networks to enable visual learning under first-order constraints. Despite these advances, constraint-based approaches have seen limited application to semantic segmentation, largely due to the complexity of reasoning over pixel-level maps. Concurrently, Bergamin et al. [4] explored medical semantic segmentation using Logic Tensor Networks. While promising, their approach operates in a fully supervised setting and does not tackle weakly supervised learning. Furthermore, enforcing spatial coherence traditionally relies on disjointed post-processing like CRFs [25] or affinity matrices [1]. Our constraint-based approach bypasses these pipelines by natively embedding spatial and structural priors directly into the neurosymbolic optimization.

3. Preliminaries Notation. We write an integer interval as [a..b] = {a, a + 1, . . . , b}. We write a random variable X in uppercase and a value x ∈ Ω(X) in lowercase, where Ω(X) is the sample space of X. Sets of random variables X and sets of values x are denoted in bold. Image Segmentation. In the image segmentation setting, we have a set of random variables X for the pixel values and a set of random variables Y for the pixel labels, such that x ∈ Ω(X) is an image and y ∈ Ω(Y) is a segmentation map. Specifically, Xi,j ∈ X corresponds to the pixel vector at spatial coordinates (i, j) in the n × m grid with c channels (c = 3 for RGB). Consequently, the pixel sample space is Ω(Xi,j ) = [0, 1]c . The label sample space Ω(Yi,j ) is finite and depends on the dataset. For example, Ω(Yi,j ) = {background, cat, dog}. We are interested in the posterior distribution p(Y | x), which conditions the distribution over segmentation maps on the input image x. To this end, we assume a differentiable function fθ with weights θ, where given an image x ∈ Ω(X), p(Y | x) = fθ (x) is the desired distribution. Propositional Logic. A propositional formula ϕ is inductively defined as either an atom, a negation ¬ϕ, a conjunction ϕ1 ∧ ϕ2 , a disjunction ϕ1 ∨ ϕ2 , or an implication ϕ1 ⇒ ϕ2 . An atom either states that a specific pixel has a certain label Y = y, or that two pixels have the same label Y1 = Y2 . A formula ϕ is satisfied in a segmentation map y, written y |= ϕ, if the formula ϕ evaluates to true under the usual semantics. A formula ϕ1 entails another formula ϕ2 , written ϕ1 |= ϕ2 , if y |= (ϕ1 ⇒ ϕ2 ) for any segmentation map y. Example 1. Consider a segmentation map y specified over the domain Ω(Yi,j ) = {background, cat, dog}. We create a formula ϕcat to check if the entire diagonal of the image is all classified as cat. ϕcat := (Y1,1 = cat) ∧ (Y2,2 = cat) ∧ · · · ∧ (Y2,2 = cat) =

n ^

(Yi,i = cat)

i=1

On a tiny 2 × 2 image specified as y1,1 = cat, y1,2 = background, y2,1 = background, y2,2 = cat, it holds that y |= ϕcat . Semantic loss. Instead of a fixed segmentation map y, we have a distribution over segmentation maps Y. So, the probability of ϕ being satisfied becomes the expectation p(ϕ) = Ey∼Y [y |= ϕ]. The probability of a formula ϕ being satisfied conditioned on an input image x is then p(ϕ | x) =

X y∈Ω(Y)

p(ϕ, y | x) =

p(ϕ | y)p(y | x) = Ey∼fθ (x) [y |= ϕ].

X y∈Ω(Y)

4

(1)

In the second equality, we use the fact that the formula ϕ and image x are conditionally independent given the segmentation map y. During training, we simply optimize the negative log-likelihood of the formula being satisfied given the input image. This is also known as the semantic loss [53]. L(x, ϕ) := − log p(ϕ | x) (2) In our formulation above, all logical formulas are assumed correct and we hence do not weigh them in the loss objective. However, in practice, weak annotations and structural priors may be imperfect. In Appendix A.3 we provide a detailed theoretical discussion on how to handle inexact constraints, including potential loss weighting strategies based on their correlation with the ground truth. Approximation. Computing p(ϕ | x) exactly is #P-complete [47] and it is hence often necessary to approximate. For this reason, we introduce conditional independence assumptions, see Appendix A.1. These independence assumptions can also be understood as fuzzy logic semantics, specifically the product t-norm [38]. In other words, the approximation becomes more faithful as the predictions of the neural network become more confident (i.e., probabilities closer to 0 or 1). Optimizing the semantic loss using fuzzy semantic is known as semantic-based regularization [14].

4. Pixel-level Logical Constraints In this section, we discuss how to express various types of segmentation labels as logic formulas. These formulas can then be optimized with the semantic loss to perform weakly supervised learning. Full supervision is a special case of our logical weak supervision. In that case, we get the ground truth labels y∗ for all pixels. So, we get the logical formula ϕfs :=

∗ (Yi,j = yi,j ).

^

(3)

i∈[1..n] j∈[1..m]

The indices i and j range over all rows and columns of the image, respectively. The loss L(x, ϕfs ) equals the usual cross-entropy loss (cf. Appendix A.2). 4.1. Weakly Supervised Constraints Scribbles. A scribble s is a subset of pixel locations that should have a target class y ∗ . So the scribble constraint is similar to full supervision, except that the conjunction does not range over all pixels but only those in the scribble s, leaving unmarked pixels unconstrained. ϕscribbles :=

^

(Yi,j = y ∗ )

(4)

(i,j)∈s

Bounding boxes. A bounding box is a rectangle, defined by two corners (i1 , j1 ) and (i2 , j2 ), and a class y ∗ . The simplest constraint would be to assume that at least one of the pixels in the bounding box has the target class y ∗ . ϕbbox shallow :=

_

(Yi,j = y ∗ )

(5)

i∈[i1 ..i2 ] j∈[j1 ..j2 ]

The above constraint is rather lenient, and it is unlikely that only a single pixel in a full bounding box has the target label. A more stringent assumption (ϕbbox ) is that the bounding box is tight. 5

In that case, each row and each column of the bounding box contain a pixel with the target class y ∗ . It follows that ϕbbox |= ϕbbox shallow . 

ϕbbox := 

 ^

_



(Yi,j = y ∗ ) ∧ 

i∈[i1 ..i2 ] j∈[j1 ..j2 ]

 ^

_

(Yi,j = y ∗ )

(6)

j∈[j1 ..j2 ] i∈[i1 ..i2 ]

Lastly, given a set of bounding boxes, the background constraint ϕbackground states that all the area outside all bounding boxes has the background class as the target label, which in essence is the same as a large scribble constraint. 4.2. Unsupervised Constraints Beyond the supervision from weak annotations, segmentation masks typically satisfy structural properties. We incorporate these priors as unsupervised logical constraints (i.e., formulas that do not rely on any label) to regularize the model’s predictions and propagate information across pixels. Formal definitions for the following constraints are detailed in Appendix B.1. Smoothness. Segmentation maps typically satisfy some smoothness constraints. One example is the region filling constraint (ϕfill ), which enforces that if all neighbors of a pixel share the same class, the central pixel must also belong to that class, thereby closing small holes in the segmentation mask. To impose stricter consistency and prevent isolated artifacts, we introduce a stronger neighborhood constraint (ϕneighborhood ), requiring that for each pixel, at least one neighbor should have the same label. Superpixels. Superpixels [41] are a classic computer vision technique to group perceptually similar pixels. We leverage superpixels to introduce a border-preserving constraint (ϕborders ), which encourages class transitions to occur only at superpixel boundaries by penalizing adjacent pixels within the same superpixel that are assigned different class labels. Corners. Our framework’s flexibility makes it trivial to integrate domain-specific geometric priors when such prior knowledge is available. For instance, in medical datasets such as REFUGE2, the anatomical structures of interest (e.g., optic discs and cups) are known to be roughly circular or elliptical. To exploit this, we introduce a corner constraint (ϕcorners ). Because a circular object rarely occupies a bounding box’s extreme corners, we mathematically define an inscribed ellipse within the box. The constraint enforces that pixels inside the box but outside this ellipse must not belong to the target class.

5. Logic-Guided Pseudo-Label Refinement We now propose the logic-guided weakly supervised segmentation framework that integrates the differentiable constraints from Sections 4.1 and 4.2 with the Segment Anything Model (SAM). The method turns weak supervision signals (i.e., bounding boxes, scribbles) into logical constraints that are optimized jointly with SAM’s decoder. The training proceeds in two stages. (i) Logic-guided fine-tuning of the foundation model: SAM receives bounding-box prompts and is optimized to satisfy a set of fuzzy-logic constraints that encode the available weak labels and priors. (ii) Pseudo-label supervised training: the fine-tuned SAM generates pseudo-labels, which are used to train a standard segmentation network.

6

5.1. Logic-Guided SAM Fine-tuning In the first stage, SAM is adapted to the weakly supervised setting by optimizing its outputs with respect to a set of differentiable logical constraints. Given a training set D = {(xi , ϕi )}ni=1 of images and formulas, fine-tuning aims to learn a function fθ that minimizes the fuzzy constraint satisfaction objective. θ∗ = arg min θ

n X

L(xi , ϕi ; θ)

(7)

i=1

Note that the constraints of each image may differ, depending on the available weak annotations, and that each ϕi may include multiple constraints conjoined together. For example, to use both scribbles and bounding boxes, we may simply set ϕ = ϕscribbles ∧ ϕbbox . Rather than evaluating these formulas as strict True/False statements, they are relaxed into continuous functions using product t-norm semantics. Simply put, this replaces the discrete logical operations with differentiable operations (e.g., a logical "AND" becomes a multiplication). This continuous formulation allows the entire objective to act as a loss function that can be optimized end-to-end using standard gradient-based optimizers. Additional details of the differentiable implementation are provided in Appendix B.2. During fine-tuning, SAM receives prompts derived directly from the weak annotations (i.e., bounding boxes). To lower the computational overhead, only the mask decoder is fine-tuned, while the image encoder remains frozen. 5.2. Supervised Segmentation from Pseudo-labels The second stage follows a conventional fully supervised training pipeline. The fine-tuned SAM from the first stage generates pseudo-labels for the entire training set. These pseudomasks are then treated as the ground-truth labels. A standard segmentation architecture, such as Mask2Former or DeepLabV2, is trained from scratch on this dataset of images and their corresponding pseudo-labels. The model is trained using the conventional cross-entropy loss, learning to replicate the high-quality predictions from the logic-guided SAM. The resulting model is the final output of our framework: an efficient, prompt-free network that benefits from both the foundation model as well as the logic-guided fine-tuning.

6. Experiments We design our experiments for the following four primary research questions. [RQ1] Does the logic-guided fine-tuning strategy produce pseudo-labels that are closer to the ground truth segmentation maps compared to pretrained prompt-based methods without domain-specific fine-tuning? [RQ2] Can logic-guided fine-tuning with weak labels achieve competitive performance against medical SAM variants that were fine-tuned using full pixel-level annotations? [RQ3] Does training a standard segmentation network on our proposed pseudo-labels lead to performance improvements over other WSSS and fully supervised methods? [RQ4] What is the specific contribution and impact of each individual logical constraint on the pseudo-label performance?

7

Figure 2: Qualitative comparison of our two-stage weakly supervised segmentation pipeline on the Pascal VOC 2012 validation set. SAM is prompted with bounding boxes. Datasets. We evaluate our framework on two benchmarks: Pascal VOC 2012 [16] for natural images and the REFUGE2 retinal dataset [17] to test adaptability in a specialized medical domain. For Pascal VOC, following standard WSSS approaches [21, 28], we train on the widely used augmented dataset, reserving 10% of the images for validation. We utilize the available ground-truth bounding boxes alongside scribbles from prior work [29]. In contrast, REFUGE2 provides only dense masks, so we synthetically derive bounding box and point-level supervision directly from the ground truth (see Appendix C for details). At test time, to simulate realworld annotation noise, we evaluate our first-stage model’s robustness on REFUGE2 using non-tight bounding boxes (75% overlap). This aligns with recent robustness evaluations of medical foundation models [50]. Baselines and Setup. For our first-stage foundation model, we utilize the standard SAM for the Pascal VOC dataset. To better handle the domain shift of medical imaging, we adopt the pre-trained MedSAM as our base architecture for the REFUGE2 dataset. During fine-tuning, our Pascal VOC setup relies on standard structural and weak label constraints (ϕneighborhood , ϕbbox , ϕfill , ϕbackground , ϕscribbles , ϕborders ), while our REFUGE2 experiments additionally incorporate the domain-specific ϕcorners prior. We benchmark our approach against a diverse set of baselines: on Pascal VOC, we compare against prompt-based SAM pipelines [22] and established WSSS methods; on REFUGE2, we evaluate against zero-shot MedSAM, fully supervised MedSAM, and other densely supervised medical foundation models. Finally, for our prompt-free second-stage network, we train DeepLabV2 [6] and Mask2Former [9], standard architectures that ensure fair comparisons with published results. Evaluation. Following standard semantic segmentation practice, we report the mean Intersection over Union (mIoU). IoU measures the intersection-to-union ratio of predicted and ground-truth regions. Additionally, for the medical imaging benchmark, we evaluate our performance using the Dice score, which measures spatial overlap and serves as a standard evaluation 8

Figure 3: Qualitative comparison of selected samples from the Pascal VOC 2012 training set. The columns alternate between the original human-annotated ground truth and the pseudo-masks produced by our logic-guided fine-tuned SAM. Interestingly, in several instances like the ones shown here, the pseudo-masks generated by our model seem to provide a better segmentation than the original ground truth. criterion in the medical domain. To answer our final research question, we conduct an ablation study measuring the impact of different constraints. Since Pascal VOC 2012 test set ground-truth annotations are unknown, we obtained results by submitting predictions to the official evaluation server.

7. Results The quantitative results for our framework are presented in four tables. For Pascal VOC, Table 1 presents the mIoU of the pseudo-labels generated in our first stage, while Table 2 reports the final mIoU of the second-stage network trained on them. For the medical dataset, our evaluation takes a comparative approach. Table 3 compares the performance of our logic-guided fine-tuned SAM against other established medical SAM variants. Finally, Table 4 compares our second-stage network, trained entirely on our generated pseudo-labels, against well-known architectures trained with dense supervision. [RQ1] Logic-guided fine-tuning significantly improves pseudo-label quality. As shown in Table 1, our logic-guided pseudo-labels on Pascal VOC achieve 94.5% mIoU. This represents a substantial improvement over purely prompt-based methods, surpassing the 91.5% mIoU achieved by the baseline provided by Jiang et al. [22] when prompted with bounding boxes. This confirms that integrating multiple weak labels through differentiable logical constraints produces pseudo-masks that are much closer to the ground truth than relying solely on the foundation model’s zero-shot priors. Additionally, as shown in Figure 3, there are cases where the generated pseudo-masks appear to delineate object boundaries more precisely than the dataset’s ground truth annotations. [RQ2] Logic-guided fine-tuning achieves competitive performance against fully supervised medical SAM models. As detailed in Table 3, directly applying MedSAM without fine-tuning (prompted with bounding boxes) yields poor performance on the REFUGE2 dataset (47.7% mean Dice). However, fine-tuning with our weakly supervised logic constraints (relying solely on bounding boxes and scribbles) drastically improves the mean Dice to 87.5% 9

Table 1: Quantitative evaluation of first-stage pseudo-label quality on the Pascal VOC 2012 train set. We report the mIoU of the pseudo-masks generated by our logic-guided fine-tuned SAM, compared against established weakly supervised baselines. Method

Annotations

mIoUPL (%)

Full supervision

Pixel-wise

100

Box2TagBack [21] Sun et al. [44] FMA-WSSS [55] DHR [23] SemPLeS [28] CLIP-ES [30] VPL [54]

Boxes ILT ILT ILT ILT ILT, Language ILT, Language

90.2 88.3 80.4 83.9 78.4 75.0 80.1

Jiang et al. [22]

ILT Points Scribbles Boxes

61.9 71.7 89.7 91.5

This method

Boxes, Scribbles Scribbles Boxes

94.5 93.8 91.7

and mIoU to 77.9%. Remarkably, this performance closely approaches the upper bound of the exact same MedSAM architecture trained with full pixel-wise annotations (88.7% mean Dice). Furthermore, our weakly supervised model outperforms other densely supervised variants such as SAMed (87.1% mean Dice), while remaining highly competitive with other state-of-the-art models like SAM-Med2D and SAM-U. [RQ3] Models trained on our pseudo-labels establish a new state-of-the-art for WSSS. On Pascal VOC (Table 2), our framework establishes highly competitive results across both validation and test sets. With a DeepLabV2 architecture, our second-stage model reaches 79.7% mIoU on the validation set, exceeding the fully supervised version of the same architecture (77.7%). On the test set, this configuration achieves a strong 78.5% mIoU, nearly reaching the fully supervised upper bound of 79.7%. Furthermore, it demonstrates clear superiority over all other weakly supervised baselines sharing the DeepLabV2 architecture. The improvements are even more pronounced when upgrading to the modern Mask2Former architecture: our prompt-free model achieves an impressive 88.6% and 89.1% mIoU on the validation and test sets, respectively. This not only significantly outperforms recent WSSS approaches, but remarkably surpasses the densely supervised Mask2Former baseline across both splits. On the REFUGE2 dataset (Table 4), our second-stage network trained solely on our pseudolabels achieves a mean Dice of 87.1%. While a gap remains between our approach and fully supervised models like TransUNet (90.3%), it is highly competitive and slightly outperforms the fully supervised ResUNet baseline (86.5%). A structure-wise breakdown reveals a distinct pattern: our method (88.1% Dice) underperforms most baselines on the Optic Disc, but achieves 86.0% on the smaller Optic Cup, exceeding all evaluated fully supervised methods. This suggests that explicit structural priors provide a distinct advantage for segmenting smaller, contained regions.

10

Table 2: Quantitative comparison of final second-stage segmentation results on the Pascal VOC 2012 validation and test set. We report the mIoU of our prompt-free networks trained entirely on our logic-guided pseudo-labels, compared against weakly supervised and fully supervised methods. The specific network architecture used for each method is denoted to ensure fair comparison. The ∗ denotes the use of CRF, while † indicates that the method reports results using DeepLabV3+ for the validation set. Method

Annotations

TEL [27] Scribbles AGMM [51] Scribbles Box2TagBack [21] Boxes Sun et al. [44] ILT SemPLeS [28] ILT CLIP-ES [30] ILT, Language CLIP-CPAL [45] ILT, Language Jiang et al. [22] ILT Points Scribbles Boxes

Network

mIoUval (%) mIoUtest (%)

DeepLabV2†

77.1 76.4 75.7 77.2∗ 73.9 73.8 74.5 71.1 69.0 75.9 76.3

74.8 (76.0∗ ) 75.5 77.1∗ 73.8 73.9 74.7 72.2 68.7 76.6 75.8

Full supervision This method

Pixel-wise DeepLabV2 Boxes, Scribbles DeepLabV2

76.4 (77.7∗ ) 78.2 (79.7∗ )

79.7∗ 77.0 (78.5∗ )

VPL [54] CoSA-MS [56] WeakTr [59] FMA-WSSS [55] DHR [23] SemPLeS [28]

ILT, Language ILT ILT ILT ILT ILT

CLIP UperNet-Swin ViT-S Mask2Former Mask2Former Mask2Former

79.3∗ 81.4 78.4 82.6 82.3 83.4

79.0∗ 78.4 79.0 81.6 82.3 82.9

Full supervision This method

Pixel-wise Mask2Former Boxes, Scribbles Mask2Former

86.0 88.6

86.1 89.1

DeepLabV3+ DeepLabV2 DeepLabV2 DeepLabV2 DeepLabV2 DeepLabV2 DeepLabV2 DeepLabV2 DeepLabV2 DeepLabV2

[RQ4] The scribble and border constraints provide the largest impact. The ablation study in Table 5 on Pascal VOC pseudo-labels answers our final research question. The full set of constraints (F) achieves 94.50% mIoU on the train set. The most critical constraints are ϕborders and ϕscribbles , as removing them causes performance drops to 90.96% (-3.54%) and 91.73% (-2.77%), respectively. This indicates that the direct supervision from scribbles and the structural guidance from the border-preserving constraint are key drivers of quality. The bounding box constraint ϕbbox surprisingly does not provide a notable contribution (-0.66%), while the other structural priors also offer small gains. Furthermore, a supplementary ablation on the REFUGE2 dataset confirms the necessity of our domain-specific prior. Without the ϕcorners constraint, the mIoU on the train set drops from 78.5% to 32.5%, highlighting its role.

8. Conclusion This work proposes a neurosymbolic approach to weakly supervised semantic segmentation, combining the reasoning power of differentiable fuzzy logic with the representational strength of foundation models. The framework translates weak annotations and expert domain knowledge into logical constraints, unifying them into a single learning objective. Compared to baselines, this logic-guided fine-tuning produces significantly higher-quality pseudo-labels, narrowing the 11

Table 3: First-stage performance comparison on the REFUGE2 dataset. To isolate the impact of our logic-guided constraints, we compare our weakly supervised MedSAM directly against zero-shot MedSAM and fully supervised MedSAM. For a broader context, we also include other foundation models fine-tuned with dense supervision. All models are evaluated using non-tight bounding-box prompts. Model

Annotations

SAMed [58] Pixel-wise SAM-Med2D [10] Pixel-wise SAM-U [12] Pixel-wise MedSAM [31] This method MedSAM [31]

Optic-Disc

Optic-Cup

Dice

Dice

IoU

IoU

Mean Dice

IoU

91.8 82.7 94.8 87.0 95.2 88.6

82.4 72.5 83.7 76.6 83.0 74.9

87.1 77.6 89.2 81.8 89.1 81.7

No finetuning 78.1 64.0 Boxes, Points 89.4 80.8 Pixel-wise 94.6 86.7

17.3 9.5 85.7 75.0 82.8 75.9

47.7 36.7 87.5 77.9 88.7 81.3

Table 4: Final segmentation results on the REFUGE2 dataset. We compare the performance of our prompt-free second-stage network (Mask2Former), trained entirely on our generated pseudolabels, against established medical segmentation architectures trained with full supervision. Model ResUNet [57] BEAL [49] SegDiff [2] nnUNet [20] TransUNet [5]

Annotations Pixel-wise Pixel-wise Pixel-wise Pixel-wise Pixel-wise

Optic-Disc

Optic-Cup

Dice

IoU

Dice

IoU

Dice

IoU

92.9 93.7 92.6 94.7 95.0

85.5 86.1 85.2 87.3 87.7

80.1 83.5 82.5 84.9 85.6

72.3 74.1 71.9 75.1 75.9

86.5 88.6 87.5 89.8 90.3

78.9 80.1 78.5 81.2 81.1

This method Boxes, Points 88.1 78.7

86.0 74.6

Mean

87.1 76.6

gap to full supervision. Consequently, our second-stage models achieve state-of-the-art WSSS performance on the Pascal VOC dataset and significant results on the specialized REFUGE2 medical benchmark, proving the framework’s effectiveness. Limitations & future work. While effective, our framework’s best performance relies on domain-specific logical constraints, which may require domain expertise. To address cases where human experts are unavailable, recent studies have shown that large language models can successfully serve as an alternative source for this domain knowledge [19]. Future work could explore applying this framework to even weaker supervision, such as image-level tags, by designing new constraints to propagate sparse information [42]. A final limitation of our approach is that the fuzzy loss induces a computational overhead during the first-stage fine-tuning (see Appendix C), although this might be partially mitigated by improved implementations.

Acknowledgements This research received funding from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme, the KU Leuven Research Fund (GA No. STG/22/021), the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (Advanced Grant DeepLog No. 101142702), the Flemish 12

Table 5: Effect of removing each constraint on pseudo-label quality for the Pascal VOC train set. As a baseline, we include SAM-huge prompted with bounding boxes. Used constraints

mIoU(%) ∆mIoU(%)

F (all constraints) F \ {ϕneighborhood } F \ {ϕbbox } F \ {ϕfill } F \ {ϕbackground } F \ {ϕscribbles } F \ {ϕborders }

94.50 94.17 93.84 93.88 93.87 91.73 90.96

-0.33 -0.66 -0.72 -0.73 -2.77 -3.54

Baseline SAM

90.50

-4.00

research foundation (FWO) projects "Neurosymbolic AI and Constraint Learning" (Project G047124N) and "Relational Concept-Based Models" (GA No. G047124N).

References [1] Ahn, J., Kwak, S.: Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4981–4990 (2018) [2] Amit, T., Nachmani, E., Shaharabany, T., Wolf, L.: SegDiff: Image segmentation with diffusion probabilistic models. CoRR abs/2112.00390 (2021) [3] Badreddine, S., Garcez, A.d., Serafini, L., Spranger, M.: Logic tensor networks. Artificial Intelligence 303, 103649 (2022) [4] Bergamin, L., Dimitri, G.M., Aiolli, F.: Integrating background knowledge in medical semantic segmentation with logic tensor networks. In: 2025 International Joint Conference on Neural Networks (IJCNN). pp. 1–7. IEEE (2025) [5] Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., Zhou, Y.: TransUNet: Transformers make strong encoders for medical image segmentation. CoRR abs/2102.04306 (2021) [6] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(4), 834–848 (2018) [7] Chen, Z., Tian, Z., Zhu, J., Li, C., Du, S.: C-cam: Causal cam for weakly supervised semantic segmentation on medical image. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11676–11685 (2022) [8] Chen, Z., Sun, Q.: Weakly-supervised semantic segmentation with image-level labels: From traditional models to foundation models. ACM Computing Surveys 57, 1 – 29 (2023) [9] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022)

13

[10] Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T., Wang, H., Su, Y., Huang, Z., Chen, J., Jiang, L., Sun, H., He, J., Zhang, S., Zhu, M., Qiao, Y.: SAM-Med2D. CoRR abs/2308.16184 (2023) [11] Colamonaco, S., Debot, D., Marra, G.: Neurosymbolic object-centric learning with distant supervision. CoRR abs/2506.16129 (2025) [12] Deng, G., Zou, K., Ren, K., Wang, M., Yuan, X., Ying, S., Fu, H.: SAM-U: Multibox prompts triggered uncertainty estimation for reliable SAM in medical image. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 368–377. Springer (2023) [13] Derkinderen, V., Manhaeve, R., Adriaensen, R., Van Praet, L., De Smet, L., Marra, G., De Raedt, L.: The deeplog neurosymbolic machine. arXiv preprint arXiv:2508.13697 (2025) [14] Diligenti, M., Gori, M., Sacca, C.: Semantic-based regularization for learning and inference. Artificial Intelligence 244, 143–165 (2017) [15] Diligenti, M., Gori, M., Scoca, V.: Learning efficiently in semantic based regularization. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 33–46. Springer (2016) [16] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. International journal of computer vision 88(2), 303–338 (2010) [17] Fang, H., Li, F., Wu, J., Fu, H., Sun, X., Son, J., Yu, S., Zhang, M., Yuan, C., Bian, C., et al.: REFUGE2 challenge: A treasure trove for multi-dimension analysis and evaluation in glaucoma screening. arXiv preprint arXiv:2202.08994 (2022) [18] Fischer, M., Balunovic, M., Drachsler-Cohen, D., Gehr, T., Zhang, C., Vechev, M.: DL2: training and querying neural networks with logic. In: International Conference on Machine Learning. pp. 1931–1941. PMLR (2019) [19] Gupta, T., Kembhavi, A.: Visual programming: Compositional visual reasoning without training. In: CVPR. pp. 14953–14962. IEEE (2023) [20] Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: a selfconfiguring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021) [21] Ji, Z., Veksler, O.: Weakly supervised semantic segmentation: From box to tag and back. In: 32nd British Machine Vision Conference 2021, BMVC 2021, Online, November 22-25, 2021. p. 385. BMVA Press (2021) [22] Jiang, P.T., Yang, Y.: Segment anything is a good pseudo-label generator for weakly supervised semantic segmentation. arXiv preprint arXiv:2305.01275 (2023) [23] Jo, S., Pan, F., Yu, I.J., Kim, K.: Dhr: Dual features-driven hierarchical rebalancing in inter-and intra-class regions for weakly-supervised semantic segmentation. In: European Conference on Computer Vision. pp. 231–248. Springer (2024) [24] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023)

14

[25] Krähenbühl, P., Koltun, V.: Efficient inference in fully connected CRFs with gaussian edge potentials. In: Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., Weinberger, K. (eds.) Advances in Neural Information Processing Systems. vol. 24. Curran Associates, Inc. (2011) [26] Lempitsky, V., Zisserman, A.: Learning to count objects in images. In: Lafferty, J., Williams, C., Shawe-Taylor, J., Zemel, R., Culotta, A. (eds.) Advances in Neural Information Processing Systems. vol. 23. Curran Associates, Inc. (2010) [27] Liang, Z., Wang, T., Zhang, X., Sun, J., Shen, J.: Tree energy loss: Towards sparsely annotated semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16907–16916 (2022) [28] Lin, C.S., Wang, C.Y., Wang, Y.C.F., Chen, M.H.: Semantic prompt learning for weaklysupervised semantic segmentation. In: 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 8764–8774. IEEE (2025) [29] Lin, D., Dai, J., Jia, J., He, K., Sun, J.: ScribbleSup: Scribble-supervised convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3159–3167 (2016) [30] Lin, Y., Chen, M., Wang, W., Wu, B., Li, K., Lin, B., Liu, H., He, X.: Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15305–15314 (2023) [31] Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature communications 15(1), 654 (2024) [32] Maene, J., Derkinderen, V., De Raedt, L.: On the hardness of probabilistic neurosymbolic learning. In: Proceedings of the 41st International Conference on Machine Learning. pp. 34203–34218 (2024) [33] Manhaeve, R., Colamonaco, S., Derkinderen, V., Adriaensen, R., Praet, L.V., Raedt, L.D., Marra, G.: Deeplog: A software framework for modular neurosymbolic ai. arXiv preprint arXiv:2605.10279 (2026) [34] Manhaeve, R., Dumancic, S., Kimmig, A., Demeester, T., De Raedt, L.: DeepProbLog: Neural probabilistic logic programming. Advances in neural information processing systems 31 (2018) [35] Manigrasso, F., Miro, F.D., Morra, L., Lamberti, F.: Faster-ltn: a neuro-symbolic, endto-end object detection architecture. In: International Conference on Artificial Neural Networks. pp. 40–52. Springer (2021) [36] Marra, G., Dumančić, S., Manhaeve, R., De Raedt, L.: From statistical relational to neurosymbolic artificial intelligence: A survey. Artificial Intelligence 328, 104062 (2024) [37] Mazurowski, M.A., Dong, H., Gu, H., Yang, J., Konz, N., Zhang, Y.: Segment anything model for medical image analysis: an experimental study. Medical Image Analysis 89, 102918 (2023) [38] Novák, V., Perfilieva, I., Mockor, J.: Mathematical principles of fuzzy logic, vol. 517. Springer Science & Business Media (1999) [39] Ouassit, Y., Ardchir, S., Yassine El Ghoumari, M., Azouazi, M.: A brief survey on weakly supervised semantic segmentation. International Journal of Online & Biomedical Engineering 18(10) (2022) 15

[40] Rajchl, M., Lee, M.C., Oktay, O., Kamnitsas, K., Passerat-Palmbach, J., Bai, W., Damodaram, M., Rutherford, M.A., Hajnal, J.V., Kainz, B., et al.: DeepCut: Object segmentation from bounding box annotations using convolutional neural networks. IEEE transactions on medical imaging 36(2), 674–683 (2016) [41] Ren, Malik: Learning a classification model for segmentation. In: Proceedings ninth IEEE international conference on computer vision. pp. 10–17. IEEE (2003) [42] Shindo, H., Brack, M., Sudhakaran, G., Dhami, D.S., Schramowski, P., Kersting, K.: Deisam: Segment anything with deictic prompting. Advances in Neural Information Processing Systems 37, 52266–52295 (2024) [43] Shukla, V., Zeng, Z., Ahmed, K., Van den Broeck, G.: A unified approach to count-based weakly supervised learning. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems. vol. 36, pp. 38709–38722. Curran Associates, Inc. (2023) [44] Sun, W., Liu, Z., Zhang, Y., Zhong, Y., Barnes, N.: An alternative to WSSS? an empirical study of the segment anything model (SAM) on weakly-supervised semantic segmentation problems. arXiv preprint arXiv:2305.01586 (2023) [45] Tang, F., Xu, Z., Qu, Z., Feng, W., Jiang, X., Ge, Z.: Hunting attributes: Context prototype-aware learning for weakly supervised semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3324–3334 (2024) [46] Tian, Z., Shen, C., Wang, X., Chen, H.: BoxInst: High-performance instance segmentation with box annotations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5443–5452 (2021) [47] Valiant, L.G.: The complexity of enumeration and reliability problems. siam Journal on Computing 8(3), 410–421 (1979) [48] van Krieken, E., Acar, E., van Harmelen, F.: Analyzing differentiable fuzzy logic operators. Artificial Intelligence 302, 103602 (2022) [49] Wang, S., Yu, L., Li, K., Yang, X., Fu, C., Heng, P.: Boundary and entropy-driven adversarial learning for fundus image segmentation. In: MICCAI (1). Lecture Notes in Computer Science, vol. 11764, pp. 102–110. Springer (2019) [50] Wu, J., Wang, Z., Hong, M., Ji, W., Fu, H., Xu, Y., Xu, M., Jin, Y.: Medical sam adapter: Adapting segment anything model for medical image segmentation. Medical image analysis 102, 103547 (2025) [51] Wu, L., Zhong, Z., Fang, L., He, X., Liu, Q., Ma, J., Chen, H.: Sparsely annotated semantic segmentation with adaptive gaussian mixtures. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15454–15464 (2023) [52] Xu, J., Schwing, A.G., Urtasun, R.: Learning to segment under various forms of weak supervision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3781–3790 (2015) [53] Xu, J., Zhang, Z., Friedman, T., Liang, Y., Van den Broeck, G.: A semantic loss function for deep learning with symbolic knowledge. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 5502–5511. PMLR (10–15 Jul 2018) 16

[54] Xu, Z., Tang, F., Chen, Z., Su, Y., Zhao, Z., Zhang, G., Su, J., Ge, Z.: Toward modality gap: Vision prototype learning for weakly-supervised semantic segmentation with clip. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 9023–9031 (2025) [55] Yang, X., Gong, X.: Foundation model assisted weakly supervised semantic segmentation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 523–532 (2024) [56] Yang, X., Rahmani, H., Black, S., Williams, B.M.: Weakly supervised co-training with swapping assignments for semantic segmentation. In: European Conference on Computer Vision. pp. 459–478. Springer (2024) [57] Yu, S., Xiao, D., Frost, S., Kanagasingam, Y.: Robust optic disc and cup segmentation with deep learning for glaucoma detection. Comput. Medical Imaging Graph. 74, 61–71 (2019) [58] Zhang, K., Liu, D.: Customized segment anything model for medical image segmentation. arXiv preprint arXiv:2304.13785 (2023) [59] Zhu, L., Li, Y., Fang, J., Liu, Y., Xin, H., Liu, W., Wang, X.: WeakTr: Exploring plain vision transformer for weakly-supervised semantic segmentation. arXiv preprint arXiv:2304.01184 (2023)

17

A. Additional Explanations and Validations A.1. Conditional Independence Assumption We approximate p(ϕ | x) by introducing conditional independence assumptions. More specifically, for any formulas ϕ1 and ϕ2 , we assume p(ϕ1 ∧ ϕ2 | x) = p(ϕ1 | x) · p(ϕ2 | x). This approximation is also known as product t-norm semantics in fuzzy logic. Next, disjunctions can be reduced to conjunctions and negations using De Morgan’s law. So as p(¬ϕ | x) = 1 − p(ϕ | x), it follows that: p(ϕ1 ∨ ϕ2 | x) = p(¬(¬ϕ1 ∧ ¬ϕ2 ) | x) = 1 − p(¬ϕ1 ∧ ¬ϕ2 | x) = 1 − p(¬ϕ1 | x) · p(¬ϕ2 | x) = 1 − (1 − p(ϕ1 | x)) · (1 − p(ϕ2 | x)) Similarly, implications can be reduced as ϕ1 ⇒ ϕ2 ≡ ¬ϕ1 ∨ ϕ2 . A.2. Equivalence of Semantic Loss and Cross-Entropy We start by expanding the loss function for the fully supervised constraint ϕf s .   ^ Lϕ (x) = − log p  (Yi,j = y ∗ ) x i,j

fs

i,j

Applying the conditional independence assumption, we can convert the probability of the conjunction over all pixels into the product of the probabilities for each individual pixel, 

Lϕfs (x) = − log 

 Y

∗ p(Yi,j = yi,j | x)

i,j

Using the logarithmic identity log( i ai ) = i log(ai ), we transform the product inside the log into a summation. X ∗ Lϕfs (x) = − log p(Yi,j = yi,j | x) Q

P

i,j ∗ | x) represents the model’s predicted probability that the pixel at (i, j) The term p(Yi,j = yi,j ∗ . Consequently, the expression − P log p(Y ∗ belongs to the ground-truth class yi,j i,j = yi,j | x) i,j is the standard definition of pixel-wise cross-entropy loss. Thus, strictly minimizing L(x, ϕfs ) is equivalent to minimizing the cross-entropy loss.

A.3. Inexact Constraints Logical constraints can sometimes be imperfect. The real object is not to maximize the constraint probability p(ϕ | x) but the ground truth segmentation p(y∗ | x). We can view their relation as follows.

18

p(y∗ | x) =

p(ϕ = i, y∗ | x)

X i∈{⊤,⊥}

=

p(ϕ = i | x)p(y∗ | ϕ = i, x)

X

(8)

i∈{⊤,⊥}

= (α − β) · p(ϕ = ⊤ | x) + β with α = p(y∗ | ϕ = ⊤, x) and β = p(y∗ | ϕ = ⊥, x) In case the constraint is perfect, i.e., only true for the ground truth segmentation map y∗ (y |= ϕ ⇔ y = y∗ ), we get α = 1, β = 0 and hence p(y∗ | x) = p(ϕ = ⊤ | x). In case the constraint always holds for the ground truth segmentation map, but also for some other segmentation maps, β = 0 and p(y∗ | x) ∝ p(ϕ = ⊤ | x). So α does not depend on the constraint and can be ignored. The interesting case is when some constraints are inexact, meaning it is possible that y∗ ̸|= ϕ, which makes α and β depend on the constraint. This is the approach taken in the paper. However, in case there is a (small) set of ground truth segmentation maps available, it is also possible to estimate α and β on a small validation set for better results.

B. Detailed method formulation B.1. Unsupervised Constraints Smoothness. Segmentation maps typically satisfy some smoothness constraints. One example is that for each pixel, at least one of the neighbors should have the same label. This avoids isolated artifacts. ^ _ ϕneighborhood := (Yi,j = Yi′ ,j ′ ) i∈[1..n] (i′ ,j ′ )∈nb(i,j) j∈[1..m]

In our implementation, the neighborhood nb(i, j) of a pixel (i, j) is defined as the set of its 8 surrounding pixels (the standard Moore neighborhood). A weaker version of this constraint (ϕneighborhood |= ϕfill ) states that pixels surrounded by pixels with the same class should also have that same class. 

ϕfill :=

^ i∈[1..n] j∈[1..m]

   

 ^

(i1 ,j1 )∈nb(i,j) (i2 ,j2 )∈nb(i,j)

   Yi1 ,j1 = Yi2 ,j2  ⇒  

 ^

Yi,j = Yi1 ,j1 

(i1 ,j1 )∈nb(i,j)

Superpixels. Let S be the superpixel map where Si,j denotes the superpixel index of the pixel at coordinates (i, j). The constraint enforces that if a pixel (i, j) and its neighbor (u, v) belong to the same superpixel (i.e., Si,j = Su,v ), they must imply the same class prediction (Yi,j = Yu,v ): ϕborders :=

^

(Si,j = Su,v ⇒ Yi,j = Yu,v )

^

i∈[1..n] (u,v)∈nb(i,j) j∈[1..m]

In this formulation, the implication A ⇒ B ensures that the constraint is trivially satisfied (true) if the pixels cross a superpixel boundary (where Si,j ̸= Su,v ), thereby allowing label transitions to occur freely at those locations. 19

Corners. Let B be a bounding box defined by its vertical and horizontal limits [ymin ..ymax ] and [xmin ..xmax ], associated with a target class y ∗ . We define the spatial center of the box as min min (cy , cx ) and its radii as ry = ymax −y and rx = xmax −x . The inscribed ellipse within this 2 2 bounding box is given by the standard equation: (i − cy )2 (j − cx )2 + ≤1 ry2 rx2

(9)

We define the set of corner pixels Ebox as all pixels (i, j) inside B that strictly fall outside this ellipse (i.e., where the left side of the equation is strictly greater than 1). The constraint enforces that these corner pixels must not take the target class label y ∗ . ϕcorners :=

^

¬(Yi,j = y ∗ )

(10)

(i,j)∈Ebox

B.2. Differentiable Implementation The implementation computes the fuzzy loss in log-space to prevent numerical underflow. log p(ϕ1 ∧ ϕ2 ) = log p(ϕ1 ) + log p(ϕ2 ) log p(ϕ1 ∨ ϕ2 ) = log[1 − (1 − p(ϕ1 ))(1 − p(ϕ2 ))] As most constraints range over all pixels in the image, we tensorize the computations using PyTorch, allowing efficient GPU-based computation.

C. Training Details C.1. Datasets Pascal VOC 2012. The Pascal Visual Object Classes (VOC) 2012 dataset is the standard benchmark for evaluating Weakly Supervised Semantic Segmentation (WSSS) methods. It contains images of 20 foreground object categories along with a background class. The original dataset provides 1,464 images for training, 1,449 for validation, and 1,456 for testing. Following standard practice in WSSS literature, we utilize the augmented training set provided by Hariharan et al., which includes additional annotations from the Semantic Boundaries Dataset (SBD), expanding the total number of training images to 10,582. For weak supervision, we utilize the established ground-truth bounding boxes and the scribble annotations. As noted in the main text, we reserve a 10% subset of the training data for internal validation during the second-stage training. This internal split allows us to monitor convergence and perform hyperparameter selection without any data leakage from the original Pascal VOC validation set, which we treat as an unseen test set for internal reporting. Finally, to ensure a fair comparison with the state-of-the-art, all reported test set results are obtained by submitting our final model predictions to the official Pascal VOC evaluation server. REFUGE2. The REFUGE2 (REtinal FUndus Glaucoma challengE 2) dataset is a specialized medical imaging benchmark designed for glaucoma detection and optic nerve head segmentation. It consists of 1,200 color retinal fundus images, traditionally split into 400 training, 400 validation, and 400 testing images. The dense segmentation task focuses on accurately localizing two distinct anatomical structures: the optic disc and the smaller, contained optic cup. Unlike Pascal VOC, REFUGE2 provides only dense, pixel-perfect ground-truth masks. To adapt this dataset for our weakly supervised setting, we discard the dense masks during training and 20

synthetically derive weak annotations (bounding boxes and points) directly from the ground truth: • Bounding Boxes: For each image, we extract tight bounding boxes by identifying the four extreme spatial coordinates (minimum and maximum x and y values) of the ground-truth segmentation regions for both the optic disc and the optic cup. • Point-level Supervision: To simulate sparse human point clicks or scribbles, we randomly sample three distinct pixel coordinates from within each target class region present in the image, as well as three points from the background region. • Test-Time Prompt Jitter: During inference, human annotators rarely provide perfect, pixel-tight bounding boxes. To simulate a more realistic and challenging deployment scenario, we evaluate our first-stage model’s robustness using non-tight bounding boxes. Specifically, we apply a random spatial jitter to the coordinates of the tight bounding boxes such that the resulting prompt maintains only a 75% overlap with the target segmented region. C.2. General Training Details Hardware. We run all our experiments on a Nvidia L40S 48GB GPU card with AMD EPYC 9334 CPU and 256GB RAM. Total runtime. The first-stage fine-tuning of the foundation model requires approximately 6 hours for the Pascal VOC dataset and 1.5 hours for the REFUGE2 dataset. Note that these reported training times are approximate; actual runtimes may fluctuate depending on the specific hardware configuration and the concurrent computational load on the machine during execution. Reproducibility. To ensure the robustness and reproducibility of our findings, all experiments were repeated using three different random seeds (0, 1, and 2), and the reported quantitative results reflect the average performance across these runs. In Tables 8 to 10 we report the mean and std obtained over the runs. The code will be released upon acceptance. C.3. Stage 1: Logic-Guided SAM Fine-tuning Architecture. We utilize the Huge variant of the Segment Anything Model (SAM) as our base model for the Pascal dataset. While for the REFUGE2 dataset we utilize the MedSAM b variant. To maintain computational efficiency and leverage the robust pre-trained representations, we freeze the parameters of both the Image Encoder and the Prompt Encoder. Fine-tuning is restricted exclusively to the Mask Decoder, which learns to interpret the logic-driven supervision signals. Optimization. The model is trained using the Adam optimizer with a learning rate of 1 × 10−4 and no weight decay. We employ a physical batch size of 4 images per GPU. To stabilize training and ensure better convergence, we implement a gradient accumulation procedure that simulates an effective batch size of 64. Training Schedule. Training proceeds for 30 epochs on both Pascal VOC augmented training set and REFUGE2. Validation is performed at the end of every epoch to monitor constraint satisfaction and segmentation quality. 21

Logical Constraints. The loss function is defined as the summation of the negative loglikelihoods of the satisfied constraints. Based on the available weak annotations and prior knowledge, we incorporate the following set of constraints: • Weak Label Constraints: ϕbbox (bounding box tightness), ϕscribbles (scribble consistency), and ϕbackground (background exclusion). • Structural Priors: ϕneighborhood (neighbor consistency), ϕfill (region filling), ϕborders (superpixel-based boundary preservation) and, ϕcorners (shape prior, used only in REFUGE2). All constraints are weighted equally in the final loss objective. C.4. Stage 2: Fully Supervised Segmentation In the second stage, the pseudo-labels generated by the fine-tuned SAM are treated as groundtruth annotations to train standard, prompt-free segmentation networks. We employ three architectures to ensure fair comparison with existing literature: DeepLabV2. For our experiments with convolutional architectures, the DeepLabV2 model was instantiated with an output stride of 16. We optimized the network using Stochastic Gradient Descent (SGD) with a momentum of 0.9 and a weight decay of 0.0005. The model was trained for 20 epochs with a total batch size of 10. We employed a base learning rate of 0.00025, which was modulated by a polynomial decay schedule with a power of 0.9. To prevent the loss of pretrained feature representations, we applied a layer-specific learning rate strategy: the backbone received a 0.5× multiplier (yielding an effective rate of 0.000125), while the ASPP and classifier heads were updated at the base rate. The training objective was the standard Cross-Entropy Loss. Model selection was performed by monitoring the Mean Intersection over Union (mIoU) on the validation subset of the training data. At inference, we utilize a multi-scale evaluation strategy (scales: 0.5, 0.75, 1.0, 1.25, 1.5), fusing the predictions via a maximum response across scales, followed by optional DenseCRF post-processing. Mask2Former. For our transformer-based approach, we fine-tuned the Mask2Former architecture using a Swin-Large backbone pretrained on COCO instance segmentation. The training was conducted for 20 epochs using a batch size of 4. We utilized the AdamW optimizer with zero weight decay and a base learning rate of 0.0001. Similar to the DeepLabV2 setup, a polynomial decay schedule (power = 0.9) was applied over the total iterations. To avoid catastrophic forgetting of the pretrained representations, a layer-wise decay was implemented, dividing the backbone’s learning rate by a factor of 10. Input images were resized and normalized using standard ImageNet statistics. As with our other models, final model selection was based on the peak validation mIoU.

D. Additional results Additional comparisons for finetuned MedSAM: We implemented additional baselines finetuning SAM with partial cross-entropy on scribbles, which yielded significantly lower performance than our logic-guided approach (Table 6). To further contextualize this, we also evaluated a standard "Self-Training" baseline (finetuning on zero-shot pseudo labels). Constraint satisfaction without fine-tuning: We evaluated zero-shot SAM constraint satisfaction on the Pascal VOC validation set and found it frequently violates these structural priors, strongly motivating our method. Precisely because these constraints are initially unsatisfied, our 22

Table 6: REFUGE2 baseline comparison demonstrating that our logic-guided constraints outperform naive partial supervision. Method SAM Self-Training (Zero-Shot Pseudo-labels) SAM Partial CE (Prompt with boxes + scribbles) Ours (Boxes + scribbles)

mIoU

Dice

37.71±0.43 66.18±1.57 77.91±1.66

49.11±1.03 79.08±1.71 87.50±1.63

framework has the necessary signal to supervise the model; if satisfaction were already 100%, the logic-guided loss would provide no gradient for structural learning. After fine-tuning, satisfaction rates improve (Table 7). Table 7: Constraint satisfaction rates (%) on Pascal VOC Val. Constraint

Zero-Shot SAM

Ours (Fine-Tuned)

Scribble Neighbours Filling Background BBox

65.40 46.72 53.14 35.68 19.24

73.75 ± 0.14 74.57 ± 1.61 85.51 ± 0.78 45.66 ± 0.15 20.90 ± 0.54

Table 8: Quantitative evaluation of first-stage pseudo-label quality on the Pascal VOC 2012 training set. Results report the mean Intersection over Union alongside standard deviations across different logical constraint configurations. Used constraints

mIoUPL (%)

F (all constraints) F \ {ϕneighborhood } F \ {ϕbbox } F \ {ϕfill } F \ {ϕbackground } F \ {ϕscribbles } F \ {ϕborders }

94.50 ± 0.02 94.17 ± 0.04 93.84 ± 0.07 93.88 ± 0.15 93.87 ± 0.08 91.73 ± 0.28 90.96 ± 0.23

D.1. Computational Complexity and Runtime Analysis In this section, we provide a detailed breakdown of the computational overhead introduced by the semantic-based regularization constraints during the fine-tuning of the Segment Anything Model (SAM). Integrating logic constraints via semantic regularization involves an additional forward pass through the constraint logic and backpropagation through the resulting loss terms. As shown in the table below, most individual constraints (such as scribbles, background, and neighbours) maintain a runtime performance comparable to the baseline Cross-Entropy training (Table 11). The F (all constraints) configuration, which evaluates all logic constraints simultaneously, represents the upper bound of our computational cost at 0.6387 s/iter. This increase is expected given the cumulative nature of the constraints, yet it remains within a manageable range for standard deep learning hardware. It is interesting to observe that the evaluation of certain individual constraints exhibits a runtime comparable to, or even slightly lower than, the baseline 23

Table 9: Quantitative evaluation of the final second-stage segmentation networks on the Pascal VOC 2012 dataset. Performance is reported as mean Intersection over Union alongside standard deviations across multiple runs. Architecture

mIoUval (%)

DeepLabV2 DeepLabV2 + CRF Mask2former

78.2 ± 0.24 79.7 ± 0.28 88.6 ± 0.07

Table 10: Quantitative evaluation on the REFUGE2 dataset, detailing the performance of both the first-stage fine-tuned MedSAM and the final second-stage segmentation network. Results are reported alongside standard deviations across multiple runs. Architecture

IoUdisc (%) IoUcup (%)

mIoU(%)

Finetuned MedSAM 80.83 ± 1.02 75.00 ± 4.12 77.91 ± 1.66 Mask2former 78.67 ± 1.29 74.64 ± 1.63 76.65 ± 0.86 cross-entropy loss. This efficiency arises because these logical constraints often bypass the need to load, resize, and spatially align dense ground-truth segmentation masks. Instead, they rely purely on fast, memory-contiguous tensor operations applied directly to the network’s output logits.

Table 11: Average runtime per iteration (including both forward and backward pass) for different logic constraint configurations during SAM fine-tuning on the PASCAL Dataset. Runtime (ms/iter)

∆ vs. Baseline (%)

Scribbles Bboxes Background Neighbours Fill Borders Corners

346.6 ± 35.5 479.5 ± 68.7 334.1 ± 35.2 378.5 ± 54.5 378.3 ± 49.4 389.8 ± 40.2 357.8 ± 41.0

+1.4% +40.3% −2.2% +10.8% +10.7% +14.1% +4.7%

F (all constraints) Baseline SAM (cross-entropy)

638.7 ± 64.1 341.7 ± 43.1

+86.9% −

Configuration

24

D.2. More Visualizations The Figures 4 to 6 show further visualizations of the application of our pipeline on the Pascal VOC and REFUGE2 datasets.

Figure 4: Additional comparison of our two-stage weakly supervised segmentation pipeline on the Pascal VOC 2012 validation set. SAM is prompted with bounding boxes.

25

Figure 5: Additional comparison of our two-stage weakly supervised segmentation pipeline on the Pascal VOC 2012 validation set. SAM is prompted with bounding boxes.

26

Figure 6: Comparison of our two-stage weakly supervised segmentation pipeline on the REFUGE2 validation set. MedSAM is prompted with bounding boxes. 27