Prototype-Grounded Concept Models for Verifiable Concept Alignment Stefano Colamonaco1∗, David Debot1∗, Pietro Barbiero2 , Giuseppe Marra1
arXiv:2604.16076v1 [cs.LG] 17 Apr 2026
1
KU Leuven, Department of Computer Science 2 IBM Research, Zurich, Switzerland April 20, 2026
Abstract Concept Bottleneck Models (CBMs) aim to improve interpretability in Deep Learning by structuring predictions through human-understandable concepts, but they provide no way to verify whether learned concepts align with the human’s intended meaning, hurting interpretability. We introduce Prototype-Grounded Concept Models (PGCMs), which ground concepts in learned visual prototypes: image parts that serve as explicit evidence for the concepts. This grounding enables direct inspection of concept semantics and supports targeted human intervention at the prototype level to correct misalignments. Empirically, PGCMs match the predictive performance of state-of-the-art CBMs while substantially improving transparency, interpretability, and intervenability.
1
Introduction
Modern neural networks achieve remarkable predictive performance, yet their lack of semantic transparency remains a major obstacle to trustworthy deployment. While many Explainable AI methods aim to explain individual predictions post-hoc, an increasingly influential line of work instead builds interpretable models (Espinosa Zarlenga et al., 2022; Mahinpei et al., 2021; Barbiero et al., 2023; Vandenhirtz et al., 2024; Yuksekgonul et al., 2022). In this paradigm, predictions are structured through human-understandable intermediate representations, also called concepts, which enables human-AI interaction in the form of inspection, verification, and intervention. Arguably the most well-known of these are Concept Bottleneck Models (CBMs) (Koh et al., 2020). In CBMs, an input is first mapped to a set of high-level, symbolic concepts (e.g. has stripes, is metallic), and the final prediction is computed exclusively from these concept predictions, typically via a simple and transparent classifier. This design offers strong interpretability guarantees: users can inspect concept activations, intervene by modifying them, and reason about predictions entirely at the level of high-level concepts. ∗
Equal contribution
1
Figure 1: Comparison between standard neural networks, Concept Bottleneck Models (CBMs), and our Prototype-Grounded Concept Models (PGCMs). While CBMs explain predictions in terms of high-level concepts, they provide no way to verify what those concepts mean visually. PGCMs ground each concept in learned visual prototypes, making concept semantics explicit and inspectable. Despite these appealing properties, CBMs suffer from a fundamental limitation: their concepts lack low-level grounding. Even when concepts are directly supervised using human-provided labels, there is no guarantee that the learned representation aligns with the intended semantics. A concept labeled has stripes may in fact rely on spurious textures, background cues, or correlated artifacts. Crucially, users have no direct way to verify this alignment, because the visual or low-level evidence underlying a concept prediction remains hidden. As a result, CBMs are only interpretable under a strong and often unjustified assumption of concept alignment: that the learned concepts mean what their names suggest. In this work, we address this limitation by explicitly grounding concepts in concrete visual evidence. We introduce Prototype-Grounded Concept Models (PGCMs), a framework that augments Concept Bottleneck Models with learned visual prototypes. In a PGCM, each concept is not represented solely as an abstract scalar prediction, but is instead associated with a set of learned prototypes: localized visual patterns that serve as concrete exemplars of what the model considers evidence for that concept. At inference time, a PGCM explains its concept predictions in terms of similarity to these prototypes. Concept activations are thus derived from and can be inspected through the visual exemplars most relevant to the input. As a result, each concept is given a dual representation: a high-level symbolic label (e.g. grey hair ) and a collection of low-level visual instances that make this meaning explicit (e.g. specific image instances of grey hair ). This integration embeds visual grounding directly into the concept bottleneck. This grounding yields several important advantages. First, PGCMs enable verification of concept alignment: users can directly examine the prototypes associated with each concept to assess whether the learned semantics match their intended semantics. Second, the dual representation of concepts unlocks new forms of human intervention. Rather than intervening only on individual concepts, as in standard CBMs, users can intervene at the prototype level (removing, modifying, or selecting prototypes) which can simultaneously correct multiple concept predictions. Third, PGCMs expand the explanatory power of CBMs, enabling explanations that reference both abstract concepts and concrete visual evidence. 2
In summary, while Concept Bottleneck Models promise interpretability through abstraction, they ultimately require trust in unobservable semantics; PGCMs replace this assumption with verifiable, visually grounded concepts (Figure 1).
2
Background
Concept Bottleneck Models (CBMs) are deep learning models that provide interpretability by predicting the target in two steps. First, they map the low-level input features (e.g. pixels) to a set of high-level concepts (e.g. red, dog) using a neural network. Second, they map the predicted concepts on the target task using either an interpretable function, e.g. a linear layer, or a blackbox neural network. The concepts form a high-level, semantically meaningful abstraction through which the model can be interpreted or explained. To align concepts to the meaning intended by the human, they are typically explicitly supervised during training. A core advantage of CBMs is the ability to do concept interventions: human experts may at test time correct mispredicted concepts, which may influence the downstream task prediction, making it more accurate. Notation. We write random variables in uppercase (e.g. p(C)) and their assignments in lower case (e.g. p(C = c)). When clear from the context, we abbreviate assignments (e.g. p(C = c) becomes p(c)). We use curly brackets to write groups of random variables concisely (e.g. p({Ci }3i=1 ) = p(C1 , C2 , C3 )). We follow standard variational inference notation, with p(.) denoting generative distributions and q(.) denoting variational posteriors.
3
Model
We propose Prototype-Grounded Concept Models (PGCMs), a hybrid architecture that predicts high-level concepts by grounding the prediction in low-level prototypes. PGCMs retain the familiar CBM structure (input → concepts → task) while innovating in how concepts are inferred from input data. As is typical in CBMs, the concepts C are given by the human and directly supervised, i.e. the dataset not only has task labels, but also concept labels. In practice, the need for concept supervision is not a significant limitation as it can come from Vision-Language Models instead of being provided by humans (Oikarinen et al., 2023).
3.1
Overview
In this section, we provide a high-level description of the model. Additional details are given in Section 3.2. At a conceptual level, the model can be viewed as a three-stage mapping: x→ − p→ − c− → y, where x is the input image, p is a set of similarity scores to learned prototypes, c is the concept representation of x, and y is the task prediction.
3
Figure 2: Example inference of PGCMs. From the image X, we use a segmentation model to extract two important image parts X1 = and X2 = . For each image part, the prototype selector q(Si |Xi ) predicts a categorical distribution over the three learned prototypes. For instance, for , it assigns all probability to the first prototype. Our Concept Alignment Table shows the prototypes and their different representations: an image representation defined by the image decoder p(Xi |Si ) and a concept representation defined by the concept decoder p(Ci |Si ). For each image part, we pick the concepts of the selected prototype, which form our predicted concepts. The task predictor p(Y |C) then maps all predicted concepts on the target task. As a preprocessing step, we extract image parts from each input image using a segmentation network. The motivation is to isolate spatial regions that are informative for predicting high-level concepts. Concretely, each image x is mapped to n image parts {xi }ni=1 , with n a hyperparameter. The probabilistic graphical model (PGM) of our model is shown in Figure 3. Our model combines generative inference (black arrows) and discriminative inference (red and black arrows), which serve complementary purposes. The generative model is primarily used to inspect and interpret the learned prototypes. For each image part i, we introduce a latent variable Si , representing the selection of a prototype. Its prior p(Si ) is a categorical distribution with one value per learned prototype. The image decoder p(Xi |Si ) maps a prototype index Si = j to a concrete image part, which we call the image representation of prototype j. In parallel, the concept decoder p(Ci |Si ) maps the prototype index to a distribution over high-level concepts, yielding the prototype’s concept representation. By enumerating all possible values of Si , we can visualize each prototype via the image decoder, and inspect its concepts via the concept decoder. This provides the underlining formal semantics of our model.
i = 1..n Ci
Y
Si Xi
Figure 3: Probabilistic Graphical Model of PGCMs. Black arrows are used for generative and discriminative inference. Red arrows are only used for discriminative inference.
The discriminative model is used to make concept and task predictions given an input image x. In this setting, the image parts Xi are observed, as they 4
are extracted deterministically from x using our segmenter. The prototype selector q(Si |Xi ) assigns a distribution over prototypes to each image part Xi . Intuitively, this module selects the prototype that is the most visually similar to the given image part. Given a selected prototype, the concept decoder p(Ci |Si ) maps the selected prototype on high-level concepts. Finally, the task predictor p(Y |{Ci }ni=1 ) is standard, and can be e.g. a linear layer or neural network mapping the concept predictions on the final task. Example 1. Figure 2 shows an example of inference using our PGCMs. The segmentation model extracts two image parts and . For each part, the prototype selector assigns high probability to the most visually similar learned prototype, in this case assigning all probability mass to a single prototype: e.g. for , it selects the prototype with image representation . The concept predictions for each part are then inherited from the selected prototype’s concept representation. Finally, the task predictor maps the predicted concepts across parts to the final task output.
3.2
Parametrizations
In this section, we describe the parametrization of the distributions introduced previously. Prototype embeddings. We denote with m the number of prototypes we want to learn. Each prototype j ∈ {1, . . . , m} is associated with a learnable embedding ej ∈ Rd which represents the latent embedding of prototype j. We use these embeddings in the parametrizations of the components below when we condition on the prototype selection Si . Image decoder. The image decoder models the distribution of an image part given a prototype assignment, which we model using a multivariate Gaussian with a fixed diagonal covariance matrix: p(xi | Si = j) = N xi | µj , σ 2 I , with µj ∈ RD = fimage (ej ), fimage a neural network operating on prototype embeddings, D the number of pixels in an input image, and σ 2 a fixed constant variance. This decoder defines the image representation of each prototype. Concept decoder. The concept decoder maps a prototype to a distribution over high-level concepts, where each concept is modelled independently given a selected prototype: p(ci | Si = j) =
k Y
p(ci,l | Si = j)
where
p(ci,l | Si = j) = Bernoulli(πj,l )
l=1
with πj ∈ [0, 1]k = fconcept (ej ) the concepts’ probabilities, fconcept a neural network operating on prototype embeddings, and k the number of concepts. This distribution defines the concept representation of each prototype. Intuitively, it performs concept classification of prototype embeddings. Prototype selector. The variational posterior over prototype assignments is implemented by a prototype selector q(Si | Xi ) where each image part xi is first deterministically mapped to an embedding zi = fenc (xi ) ∈ Rd , with fenc a neural network. Prototype assignment probabilities are computed using a similarity measure sim(·, ·) between every prototype embedding and the image part’s embedding, combined with a softmax to obtain a categorical distribution: exp sim(zi , ej ) , q(Si = j | Xi ) = Pm k=1 exp sim(zi , ek ) 5
Intuitively, this corresponds to a prototype classification of image parts. In our experiments, we use the dot product as similarity measure. Segmentation model. We extract the image parts {xi }ni=1 using a segmentation network that isolates (potentially overlapping) spatial regions from the input x. This component is flexible, as it is possible to use ground-truth masks to train it or use a pretrained segmenter in their absence.
3.3
Inference
At inference time, as standard in CBMs, we observe the input image x and want to predict concepts and tasks. We first apply the trained segmentation model to obtain n image parts {xi }ni=1 . For each image part xi , we compute the posterior predictive distribution over concepts as h i p(ci,j |xi ) = Eq(si |xi ) p(ci,j | si ) (1) Each concept prediction is then ĉi,j = arg maxk∈{0,1} p(Ci,j = k|xi ). Given the predicted concepts, the task prediction is computed as ŷ = arg maxy∈Y p(y|{ĉi }i ). This means we employ hard concepts, avoiding the problem of concept leakage in CBMs which may occur when using soft concepts (Marconato et al., 2022; Havasi et al., 2022). Interpretability optimizations. We want to avoid that the prototype embeddings {ej }ni=1 encode information beyond what is visually present in their corresponding image representations. Therefore, we map each prototype embedding to its image representation and then map the resulting prototype image back to an embedding, i.e. e′j = fenc (fimage (ej )) (see Section 3.2 for fenc and fimage ). Instead of using ej , we use e′j both during training and inference. This ensures that the prototype embedding cannot contain any information not present in the prototype image. In addition, to further improve the quality and interpretability of learned prototypes, we perform a prototype swapping step halfway through training, where we replace each prototype with the closest image part in the training set. This ensures each prototype image is a realistic image. For more details regarding these optimizations, we refer to Appendix A.
3.4
Training
We maximize the joint log-likelihood of the observed image parts, concepts, and task label. Under our probabilistic graphical model, this likelihood admits the following evidence lower bound (ELBO): X (2) log p({xi }i , {ci }i , y) ≥ − KL q(si |xi ) ∥ p(si ) + log p(y | {ci }i ) | {z } task loss {z } | i regularization term
+
X i
h i Eq(si |xi ) log p(ci | si ) + log p(xi | si ) {z } | {z } | concept loss
reconstruction loss
For the derivation, we refer to Appendix C. This objective decomposes into four components, two of which are the standard task and concept loss of CBMs. The concept and task Qk components further factorize independently to each individual concept and task, e.g. p(ci |si ) = j=1 p(ci,j |si ). 6
Due to the reconstruction loss, (1) the prototype selector q(si |xi ) is encouraged to select prototypes that are visually similar to the image part xi , and (2) the learned prototypes are encouraged to be visually representative of the data. Ignoring additive constants, the reconstruction log-likelihood simplifies to log p(xi |si ) ∝ −||xi − µi ||2 . Due to the concept loss, the prototype selector is also encouraged to maximize a correct prediction of the concepts. This entails that the selector should not only select based on unsupervised, visual cues, but also using supervised, semantic cues. This is different from e.g. standard prototype-based networks, where only unsupervised visual cues and weakly-supervised task-based cues are used (see Section 5). We use the regularization term between the posterior and the prior to encourage higher-entropy prototype selection q(si |xi ), promoting exploration during training and avoiding local minima. We achieve this by choosing each p(si ) to be a fixed uniform categorical distribution over prototypes. The segmentation model used to extract image parts xi from the input image x can either be pretrained or learned using direct supervision on the image parts. In the latter case, the dataset must include ground-truth masks per image, dividing the image into important parts for prediction. The segmentation model is then trained using a standard pixel-wise cross-entropy loss.
4
Discussion
4.1
Semantics
Standard CBMs’ concept prediction is black-box, which makes it impossible for the human to check what the meaning, i.e. semantics, is of the learned concepts. PGCMs resolve this by grounding the concepts in low-level, visual prototypes. Concretely, we define the meaning of a concept explicitly through a concept alignment table (Figure 2). This table maps every prototype index j to a dual representation: • An image representation, generated by the image decoder p(Xi |Si = j).1 • A concept representation, generated by the concept decoder p(Ci |Si = j). This design mirrors a classical notion of semantics from logic (Tarski, 1944), where meaning is defined by an explicit correspondence between symbols and elements of a domain. In our case, concepts play the role of the abstract linguistic symbols, prototypes form the set of concrete entities they refer to, and the concept alignment table specifies this correspondence. As a result, the semantics of the learned concepts is no longer implicit or purely operational, as in CBMs, but given directly by how each concept is grounded in identifiable prototypes and their visual realizations. This stands in contrast to standard CBMs, where concepts are learned as latent variables without a transparent link to what they denote, making their meaning difficult to inspect or verify. By making this correspondence explicit, PGCMs provide a clear and auditable notion of semantics for the concept space. 1
Specifically, we consider the mode of the Gaussian the prototype’s image representation.
7
4.2
Interpretability and Alignment
Alignment. The human can inspect the concept alignment table to verify whether the learned concepts are aligned with their intended interpretation: for each row, they can compare the visual representation of a prototype with its associated concept representation and assess whether the listed concepts are indeed present in the image. This form of inspection is not possible in standard CBMs, where concept prediction remains a black-box process. Continuing the parallel with formal semantics, this human verification procedure is informally analogous to model checking, in the sense of evaluating whether a given model (i.e. the assignment of visual prototypes to concept) accords with an external (human) interpretation of the same concepts. The low-level semantics of a high-level concept is explicitly defined as the disjunction of the image representations of all prototypes for which that concept is active. Specifically, a concept is only true if there exists a part of the image that is similar enough to at least one of these prototypes. For instance, if the concept alignment table indicates that only prototypes having image representations x1 and x2 possess the concept Cred , then the model explicitly defines the semantics of “red” as: Cred is active ⇐⇒ (looks like x1 ) ∨ (looks like x2 )
(3)
Visual explanations. Typical CBMs provide explanations and interpretability in terms of the concepts. PGCMs can also provide this in terms of the low-level input features. In addition to showing the human which concepts led to a concrete task prediction, PGCMs show which prototypes led to that prediction. This is akin to typical prototype-based networks (see Section 5), which explain the output of the model in terms of prototypes, but lack high-level semantics (concepts). Guarantees. In PGCMs, for a concrete prediction, parts are extracted from the image by masking the rest of the image. This gives the human the guarantee that the masked input features were not used for that prediction.
4.3
Intervenability
Our proposed model allows for new types of human intervention and improves existing ones. Interventions in PGCMs operate at three levels: concept labels, prototypes, and prototype selection. Alignment interventions. The first novel intervention type is a specific type of model editing (Mazzia et al., 2024), which the human can do to re-align misaligned concepts. There are three types of such interventions possible. First, the human may edit learned prototypes. If the human disagrees with the concept representation of a prototype, they may relabel the concepts of the prototype to correct it. This relabeling simply corresponds to changing values in the table. Second, the human may remove learned prototypes. This is useful if the human detects spurious prototypes (using the prototypes’ image representations). Finally, the human may add entirely new prototypes by adding one or more image representations to the set of prototypes, e.g. image parts taken from the dataset or manually crafted. fenc and fconcept will map these to an embedding and concept representation. Improved concept interventions. In contrast to most CBMs, concepts are not independent of each other, as they are connected through the prototypes. Therefore, a single concept intervention 8
may correct multiple concepts at once: a single intervention can be used to set all similarity scores to zero of prototypes where that concept prediction is wrong. This may change not only the prediction for the intervened concept, but also the prediction for the other concepts, similar to some existing concept predictors with dependent concepts (Dominici et al., 2024; Vandenhirtz et al., 2024). Prototype interventions. Instead of intervening on a concept prediction at test time, the human may prefer to intervene on the low-level representations, and intervene on the prototype selection. A first option is to assign all probability to a single prototype (”this part is actually the most similar to that prototype”). A second option is to remove all probability of one or more prototypes (”this prototype is not similar at all”).
5
Related Work
Many extensions to Concept Bottleneck Models (CBMs) have been developed, improving their accuracy by alleviating the information bottleneck (Espinosa Zarlenga et al., 2022; Mahinpei et al., 2021; Sawada & Nakamura, 2022; Barbiero et al., 2023; Debot et al., 2024), intervenability by modeling inter-concept relations (Vandenhirtz et al., 2024; Dominici et al., 2024), and applicability by replacing concept supervision with pretrained foundation models (Oikarinen et al., 2023). However, a critical limitation that all CBMs still share is the lack of a way to inspect whether the learned concepts are correctly aligned to the human’s interpretation. This is a core limitation, as the entire claim of CBMs being interpretable rests on this alignment. Our prototype learning brings us close to part-prototype networks, which offer interpretability by grounding task predictions in prototypes. Prototype-based networks typically learn a set of prototypes and classify new inputs based on their similarity to these prototypes (Snell et al., 2017; Andolfi & Giunchiglia, 2025). Part-prototype networks typically identify prototypical parts within feature maps, enabling ”this looks like that” explanations (Chen et al., 2019; Donnelly et al., 2022; Ma et al., 2023). However, unlike Concept Bottleneck Models, prototype-networks lack high-level, controllable semantics. Since their prototypes are not supervised using concepts, they may capture recurrent visual patterns rather than human-aligned concepts. Moreover, part-prototype networks do not learn concrete image representations of prototypes, instead entirely relying on latent embedding spaces. Some prototype networks exist that do learn image representations of prototypes, but they learn prototypes of entire images as opposed to parts, and they do not employ concepts (Li et al., 2018). Other works attempts to ensure concept trustworthiness by grounding prototypes spatially within the input image (Huang et al., 2024; Jeon et al., 2025). Yet, similar to partprototype networks, they do not provide explicit image representations of prototypes, which makes it impossible to verify concept alignment. The motivation to decompose visual scenes into modular components shares philosophical roots with object-centric learning (Burgess et al., 2019; Locatello et al., 2020). By explicitly isolating object-like entities, these approaches avoid entangled global representations and facilitate robust compositional reasoning (Steinmann et al., 2025; Colamonaco et al., 2025). This structural separation is a natural fit for interpretability, as it ensures that concepts can be explicitly grounded in localized, verifiable visual evidence.
9
6
Experiments
Our experiments aim to answer the following research questions: • Accuracy: Do PGCMs achieve similar concept and task accuracy as state-of-the-art CBMs? • Interpretability and alignment: Can humans verify whether learned concepts align with their intended semantics by examining the learned concept alignment tables? Can we intervene on this alignment? • Number of prototypes: How sensitive are PGCMs to the number of prototypes used (i.e. their capacity)? • Intervenability: How effective are concept interventions in PGCMs compared to standard CBMs?
6.1
Experimental Setup
This section provides essential information about experiments. For additional details, we refer to Appendix B. Datasets. We used two standard datasets for evaluating CBMs: CelebA (Liu et al., 2015), where the each input is the face of a celebrity and where the concepts are face attributes (e.g. blond hair or beard ); ColorMNIST+, where each input is a pair of colored MNIST digits, and where concepts are the digits represented in the images, and the task is their sum (adapted from Manhaeve et al. (2018)’s MNIST+). Additionally, we used the CLEVR-Hans dataset (Stammer et al., 2021), where each input is an image consisting of different objects, where the concepts are the shape, color, size and material. Additionally, we considered a version of ColorMNIST+ where the concept labels are noisy. Concept prediction is hard in CelebA but easy in ColorMNIST+ and CLEVR-Hans, representing two common settings for CBMs. Baselines and evaluation. We compare with Concept Bottleneck Models (CBM) (Koh et al., 2020), Concept Residual Models (CRM) (Mahinpei et al., 2021), and Concept-based Memory Reasoner (CMR) (Debot et al., 2024). In all three models, the concept predictor is a neural network. For CBM, we take a neural network as task predictor. CRM and CMR are models with a sidechannel, i.e. the task is predicted not only using concepts, but also using additional information, which hurts interpretability (Debot & Marra, 2025). In CRM, this is an embedding predicted from the input, and the task predictor is a neural network. In CMR, this is a rule selection mechanism, and the task predictor is a memory of learned logic rules. We additionally train a deep neural network (DNN) to predict the task directly from the input as reference. We evaluate concept prediction using balanced accuracy and task prediction using regular accuracy. When we show images from CelebA, we make the black pixels transparent for visualisation purposes, and we upscale some of them.
10
Table 1: Example rows from the Concept Alignment Table of CelebA, ColorMNIST+ and CLEVR-Hans (seed 1). Image
Table 2: Example concept semantics expressed as a disjunction over prototypes for CelebA, ColorMNIST+ and CLEVR-Hans (seed 1).
Active Concepts
Concept
Semantics
Bangs, Blond Hair, Wavy Hair
Brown Hair
∨
Heavy Makeup, No Beard, Pale Skin
Pointy Nose
∨
Big Nose, Pointy Nose ∨
Digit is 3 Digit is 4 whitespace Small, Cyan, Sphere, Metal Small, Yellow, Cube, Rubber
6.2
Blue
∨
∨...
Cube
∨
∨...
Key findings
PGCMs make concept alignment and semantics directly inspectable (Tables 1 and 2). This is PGCMs’ core advantage over existing CBMs, and can be done by inspecting the learned concept alignment table. These tables instantiate the formal semantics defined in Section 4.2. Table 1 shows some examples rows of the learned table for both datasets. For both datasets, the concepts can be clearly recognized in the image representations of the prototypes. Table 2 gives the alternative view: it shows the semantics for individual concepts as explained in Section 4.2, showing that e.g. the hairstyle does not matter for predicting brown hair, and the color and position of the digit do not matter for predicting a digit 3. PGCMs allow for novel model editing interventions, increasing accuracy and im- Table 3: Model editing interventions (MEI): Reproving alignment (Table 3). We evalu- moving or editing wrong prototypes after training ate PGCMs before and after performing model on noisy labels improves concept accuracy on the editing, considering two interventions: remov- non-noisy test set on ColorMNIST+. MEI Accuracy before Accuracy after ing wrong prototypes and editing (correcting) their concepts. For these experiments, we gave Removing 92.9 ± 0.3 96.9 ± 0.9 ColorMNIST+’s training and validation sets Editing 92.8 ± 0.2 97.8 ± 1.4 noisy concept labels (see Appendix B for details). As a result, some of the learned prototypes are misaligned to the human, e.g. learning a prototype with image representation and concept representation c5 = 1, as opposed to c1 = 1. By dropping the wrong prototypes or editing their concept representation, accuracy on the non-noisy test set significantly increases. Such interventions are not possible with standard CBMs.
11
1.000
1.00
0.998 0.996
CRM CBM CMR PGCM PGCM*
Concept Accuracy
Concept Accuracy
CRM CBM CMR PGCM PGCM*
0.95 0.90
0.994
0.85
0.992
0.80
0.990 0
5
10
15
0
Number of Concept Interventions
50
100
150
Number of Concept Interventions
Figure 4: Concept accuracy after intervening on increasingly more concepts on ColorMNIST+ and CelebA. PGCM uses standard concept interventions, while PGCM* uses the interventions described in Section 4.3. Table 4: Concept and task accuracy on CelebA and ColorMNIST+ over three seeds. ColorMNIST+
CelebA
Model
Concept
Task
Concept
Task
DNN CBM CRM CMR
– 99.2 ± 0.1 99.0 ± 0.0 99.0 ± 0.0
99.2 ± 0.1 99.6 ± 0.1 99.4 ± 0.1 99.2 ± 0.2
– 81.3 ± 0.4 76.8 ± 0.3 76.3 ± 0.1
84.7 ± 0.2 84.0 ± 0.3 84.8 ± 0.2 84.7 ± 0.5
PGCM (ours)
99.2 ± 0.0
99.7 ± 0.0
78.5 ± 0.1
83.0 ± 0.0
PGCMs achieve similar levels of accuracy as standard CBMs, despite their interpretability advantages (Table 4). By predicting concepts via a finite set of learned prototypes, PGCMs constrain model capacity compared to existing concept-based models. This may result in a gap in accuracy w.r.t. deep neural networks, and is well-known for prototype-based networks (Chen et al., 2019). In spite of this, our concept and task accuracy is similar to existing concept-based models on ColorMNIST+ with only a small drop on CelebA. We attribute the drop in task accuracy for PGCM on CelebA due to its drop in concept accuracy. CMR and CRM do not suffer a drop in task accuracy (despite having lower concept accuracy than CBM) because they have a sidechannel (i.e. they predict the task not only using concepts). PGCMs are more or equally responsive to concept interventions compared to standard CBMs (Figure 4). In PGCMs, concepts are not modeled conditionally independent, which allows a single concept intervention to influence multiple concept predictions (dashed line, PGCM*). Figure 4 shows that this improves intervenability compared to CBMs and variants that model concepts independently for ColorMNIST+. For CelebA, we notice that standard interventions perform better (PGCM), as the inter-concept dependencies are insufficient in this dataset. PGCMs’ accuracy depends on model capacity, controlled by the number of learned prototypes, like all prototype-based models (Figure 5). Increasing the number of prototypes improves concept accuracy, which in turn leads to gains in task performance. However, as the number of prototypes grows, interpretability decreases due to higher cognitive load. When the number of prototypes is limited, the model may struggle to represent the full variability of the data, 12
Accuracy
1.000 0.975 0.950 0.925 0.900 0.875 0.850
20
40
60
Concept Task 80 100
Number of prototypes
Figure 5: Concept and task accuracy on CLEVR-Hans for different numbers of learned prototypes. resulting in reduced concept accuracy and, consequently, lower task accuracy. This highlights a fundamental trade-off in PGCMs between interpretability and capacity, similar to other prototypebased approaches.
7
Conclusion
We introduced Prototype-Grounded Concept Models (PGCMs), a new class of interpretable models that resolve a core and previously unaddressed limitation of Concept Bottleneck Models (CBMs): the inability to verify whether learned concepts align with their intended human semantics. By grounding each concept in a set of learned visual prototypes, PGCMs make concept meaning explicit, inspectable, and actionable. This replaces the strong and often unjustified assumption of concept alignment in CBMs with a concrete, verifiable mechanism. PGCMs retain the key advantages of CBMs (explicit concepts, transparent concept-to-task mappings, and concept interventions) while substantially expanding interpretability and intervenability. Through the concept alignment table, users can directly inspect the visual evidence underlying each concept, identify misalignments, and intervene by editing, removing, or adding prototypes. These prototype-level interventions enable powerful forms of model editing that are not possible in existing CBMs, and allow concept interventions to propagate to non-intervened concepts. Empirically, we showed that PGCMs match the concept and task accuracy of state-of-the-art CBMs, despite their additional structural constraints. At the same time, PGCMs provide richer explanations and support new low-level and high-level interventions.
Acknowledgements This research has received funding from the KU Leuven Research Fund (GA No. STG/22/021), from the Research Foundation-Flanders FWO (GA No. G047124N) and from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme. DD is a fellow of the Research Foundation-Flanders (FWO-Vlaanderen, 1185125N). PB is a fellow of the Swiss National Science Foundation (SNF grant 224226). 13
References Andolfi, L. and Giunchiglia, E. Right for the right reasons: Avoiding reasoning shortcuts via prototypical neurosymbolic ai. arXiv preprint arXiv:2510.25497, 2025. Barbiero, P., Ciravegna, G., Giannini, F., Zarlenga, M. E., Magister, L. C., Tonda, A., Lio’, P., Precioso, F., Jamnik, M., and Marra, G. Interpretable neural-symbolic concept reasoning. In ICML, 2023. Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A. MONet: Unsupervised Scene Decomposition and Representation. CoRR, 2019. Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., and Su, J. K. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019. Colamonaco, S., Debot, D., and Marra, G. Neurosymbolic object-centric learning with distant supervision. arXiv preprint arXiv:2506.16129, 2025. Debot, D. and Marra, G. Quantifying the accuracy-interpretability trade-off in concept-based sidechannel models. arXiv preprint arXiv:2510.05670, 2025. Debot, D., Barbiero, P., Giannini, F., Ciravegna, G., Diligenti, M., and Marra, G. Interpretable concept-based memory reasoning. Advances of neural information processing systems 37, NeurIPS 2024, 2024. Dominici, G., Barbiero, P., Zarlenga, M. E., Termine, A., Gjoreski, M., Marra, G., and Langheinrich, M. Causal concept graph models: Beyond causal opacity in deep learning. arXiv preprint arXiv:2405.16507, 2024. Donnelly, J., Barnett, A. J., and Chen, C. Deformable protopnet: An interpretable image classifier using deformable prototypes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10265–10275, 2022. Espinosa Zarlenga, M., Barbiero, P., Ciravegna, G., Marra, G., Giannini, F., Diligenti, M., Shams, Z., Precioso, F., Melacci, S., Weller, A., Lio, P., and Jamnik, M. Concept embedding models: Beyond the accuracy-explainability trade-off. Advances in Neural Information Processing Systems, 35, 2022. Havasi, M., Parbhoo, S., and Doshi-Velez, F. Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems, 35:23386–23397, 2022. Huang, Q., Song, J., Hu, J., Zhang, H., Wang, Y., and Song, M. On the concept trustworthiness in concept bottleneck models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 21161–21168, 2024. Jeon, S., Lee, H., Kim, E., Lee, S., Zhang, B.-T., and Hwang, I. Locality-aware concept bottleneck model. arXiv preprint arXiv:2508.14562, 2025. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015–4026, 2023.
14
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In International conference on machine learning, pp. 5338–5348. PMLR, 2020. Li, O., Liu, H., Chen, C., and Rudin, C. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In ICCV, 2015. Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-Centric Learning with Slot Attention. In NeurIPS, 2020. Ma, C., Zhao, B., Chen, C., and Rudin, C. This looks like those: Illuminating prototypical concepts using multiple visualizations. Advances in Neural Information Processing Systems, 36: 39212–39235, 2023. Mahinpei, A., Clark, J., Lage, I., Doshi-Velez, F., and Pan, W. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314, 2021. Manhaeve, R., Dumancic, S., Kimmig, A., Demeester, T., and Raedt, L. D. DeepProbLog: Neural Probabilistic Logic Programming. In NeurIPS, pp. 3753–3763, 2018. Marconato, E., Passerini, A., and Teso, S. Glancenets: Interpretable, leak-proof concept-based models. Advances in Neural Information Processing Systems, 35:21212–21227, 2022. Mazzia, V., Pedrani, A., Caciolai, A., Rottmann, K., and Bernardi, D. A survey on knowledge editing of neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2024. Oikarinen, T., Das, S., Nguyen, L. M., and Weng, T.-W. Label-free concept bottleneck models, 2023. Sawada, Y. and Nakamura, K. Concept bottleneck model with additional unsupervised concepts. IEEE Access, 10:41758–41765, 2022. Snell, J., Swersky, K., and Zemel, R. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017. Stammer, W., Schramowski, P., and Kersting, K. Right for the right concept: Revising neurosymbolic concepts by interacting with their explanations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3619–3629, 2021. Steinmann, D., Stammer, W., Wüst, A., and Kersting, K. Object centric concept bottlenecks. arXiv preprint arXiv:2505.24492, 2025. Tarski, A. The semantic conception of truth: and the foundations of semantics. Philosophy and phenomenological research, 4(3):341–376, 1944. Vandenhirtz, M., Laguna, S., Marcinkevičs, R., and Vogt, J. Stochastic concept bottleneck models. Advances in Neural Information Processing Systems, 37:51787–51810, 2024. Yuksekgonul, M., Wang, M., and Zou, J. Post-hoc concept bottleneck models. In ICLR 2022 Workshop on PAIRˆ2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data, 2022. URL https://openreview.net/forum?id=HAMeOIRD_g9. 15
A
Additional model details
A.1
Implementation
We implemented the Prototype-Grounded Concept Model (PGCM) using the PyTorch Lightning framework. The model architecture consists of four primary components: a segmentation network, an image encoder, a set of learnable prototypes, a prototype decoder, a concept predictor and a task predictor. Architecture and Forward Pass. The input image is first processed by a segmentation network. We employ a U-Net-based architecture with a ResNet backbone to predict n segmentation masks, where n corresponds to the number of image parts. Each extracted image part is mapped to a latent embedding using a shared Convolutional Neural Network (CNN) encoder. To determine the prototype selection distribution q(Si |Xi ), we compute the similarity between the image part embedding and the learnable embeddings of all m prototypes. This is implemented as the dot product between the part embedding and prototype embeddings, followed by a softmax operation to obtain a categorical distribution over prototypes. We do not directly use the learnable prototype embeddings (see Appendix A.2). Instead, during the forward pass, we first map the prototype embeddings to the image space using the image decoder. These generated prototype images are then re-encoded into latent embeddings using the shared image encoder (the same encoder used for input image parts). The concept predictor, implemented as an MLP, then takes these re-encoded embeddings as input to predict the concept probabilities for each prototype. For a specific input image part, the final concept prediction is computed as the expectation of these prototype concepts under the selection distribution q(Si |Xi ): effectively a weighted average based on the similarity between the input part and the prototypes. Finally, the task predictor p(Y |C) takes the aggregated concept probabilities (thresholded, to have hard concepts) across all image parts to predict the final task label. To address class imbalance, we calculate positive class weights based on the frequency of each concept in the training dataset; these weights are applied specifically to the positive targets during the binary cross-entropy calculation for the concept loss. We also apply this weighting strategy to all competitor models to ensure a fair comparison. Additionally, we include weights on the other terms of the ELBO (specifically, the reconstruction term and the entropy term) to balance the different objectives.
A.2
Interpretability optimizations
Mapping learned prototypes to concrete training instances. After half of the training epochs, we replace the learned prototypes with concrete training instances. This step improves interpretability by ensuring that each prototype corresponds exactly to a ground truth image region from the training data, and should thus be semantically meaningful. Concretely, for each learned prototype embedding ej , we identify the most similar image part xi in the training set by finding the highest dot product between their latent representations, fconcept (ej ) and fenc (xi ). We then substitute the learned prototype with the selected image part xj . 16
Following this replacement, we discard the learned prototype embeddings {ej }j , only using the predicted embedding fenc (xj ), as they are no longer necessary. The prototype images are frozen for the remainder of training, while all other components of the model continue to be optimized.
A.3
Computational complexity
Computing the ELBO during training is O(n · m) + O(|y| · k) + O(n · m · k) + O(n · m · D) with n the number of image parts, m the number of prototypes, |y| the number of tasks, k the number of concepts and D the number of pixels in an input image. At test time, the complexity is O(n · m · (k + |y|)).
B
Experimental and implementation details
Datasets. For our experiments, we utilized the CelebA-HQ dataset, leveraging its provided segmentation masks to define four specific objects: skin, hair, lips, and nose. For each object, we selected a subset of relevant concepts that can be visually inferred from that specific region. The final downstream task consists of predicting three high-level attributes: ”Young”, ”Attractive”, and ”Male”. For ColorMNIST+, we started from Manhaeve et al. (2018)’s MNIST+ dataset, but color each digit either red, green or blue with equal probability. Each instance has two ground truth masks: one per MNIST digit. In the noisy version of ColorMNIST+, we change the label of each instance of a 3 and a 4 to a 1 and an 8, respectively, with a probability of 30% (in the training and validation set). For the experiments in CLEVR-Hans, we followed the instructions and classes provided in the original dataset. To extract object segmentations, we used the pretrained Segment Anything Model (Kirillov et al., 2023), using bounding boxes as prompts. Reproducibility. We used seeds 1, 2 and 3 for all our experiments. We run all our experiments on multiple machines with a Nvidia L40S 48GB GPU card with AMD EPYC 9334 CPU and 256GB RAM. Small deviations in results are due to the different machine settings. Intervention policy. We use a random policy by generating a random concept ordering and following it for all instances. General training information. We used the AdamW optimizer with a learning rate schedule that features a linear warmup for the first 10 epochs, followed by a cosine annealing decay that gradually reduces the learning rate to a minimum of 10−5 . To make the models more responsive to concept interventions, we incorporate Espinosa Zarlenga et al. (2022) randint, also performing concept interventions at random during training. Each concept prediction has a 20% probability to be intervened on. General architectural details of competitors. To ensure fair comparison, we use the same number of layers and number of neurons within the layers for our model and all competitors. We use a Convolutional Neural Network for mapping the input image (or image part) to an embedding, from which we can continue with a multilayer perceptron (MLP). For CBM, the MLP predicts the concepts. For CRM, the MLP predicts the concepts and an additional residual embedding that is passed to the task predictor. For CMR, it predicts the concepts and the logits of CMR’s 17
categorical rule selector. For CBM, CRM and PGCM, the task predictor is another MLP. For CMR, it is CMR’s standard rule-based task predictor. Number of learned prototypes. For ColorMNIST+, we learn m = 30 prototypes. For CelebA, we learn m = 120 prototypes. The CLEVER-Hans experiment achieved the best result with m = 100 prototypes.
C
Loss derivation
In this section, we derive that our PGM (Figure 3) allows the following ELBO for the likelihood: X log p({xi }i , {ci }i , y) ≥ − KL q(si |xi ) ∥ p(si ) + log p(y | {ci }i ) | {z } task loss | i {z } regularization term
+
X i
h i Eq(si |xi ) log p(ci | si ) + log p(xi | si ) {z } | {z } | concept loss
(4)
reconstruction loss
We start from the likelihood and marginalize the unobserved variables Si : X log p({xi }i , {ci }i , y) = log p({si }i ) · p({xi }i , {ci }i , y|{si }i )
(5)
{si }i
with m the number of prototypes and n the number of image parts. We introduce a variational Q posterior q({si }i |{xi }i ) = i q(si |xi ) and use Jensen’s Inequality. X q({si }i |{xi }i ) (6) p({si }i ) · p({xi }i , {ci }i , y|{si }i ) · log p({xi }i , {ci }i , y) = log q({si }i |{xi }i ) {si }i X p({si }i ) ≥ q({si }i |{xi }i ) · log p({xi }i , {ci }i , y|{si }i ) · (7) q({si }i |{xi }i ) {si }i
We then split the logarithm: log p({xi }i , {ci }i , y) ≥
X
q({si }i |{xi }i ) · log
{si }i
+
X
p({si }i ) q({si }i |{xi }i )
q({si }i |{xi }i ) · log p({xi }i , {ci }i , y|{si }i )
(8) (9)
{si }i
We exploit the conditional independencies encoded by the PGM to simplify the Q first term. In particular, the variational posterior and prior factorize as q({si }i | {xi }i ) = i q(si | xi ) and
18
p({si }i ) =
Q
i p(si ). Therefore,
Q X Y p({si }i ) q(sj | xj ) log Q i p(si ) q({si }i | {xi }i ) log = q({si }i | {xi }i ) i q(si | xi ) j {si }i {si }i X Y X p(si ) q(sj | xj ) = log q(si | xi ) j i {si }i XX Y q(sj | xj ) log p(si ) = q(si | xi ) X
i
{si }i
XX
(11)
(12)
j
p(si ) q(si | xi ) si i X =− KL q(si | xi ) ∥ p(si ) .
=
(10)
q(si | xi ) log
(13) (14)
i
The fourth equality follows because each logarithmic term depends only on si , and all remaining latent variables marginalize to one under the factorized variational posterior. We now simplify the second term. Using the conditional independencies encoded by the PGM, we have: Y Y p({xi }i , {ci }i , y | {si }i ) = p(y | {ci }i ) p(ci | si ) p(xi | si ). (15) i
i
Taking the logarithm and substituting into the expectation yields: X q({si }i | {xi }i ) · log p({xi }i , {ci }i , y | {si }i )
(16)
{si }i
=
X
h i X X q({si }i | {xi }i ) log p(y | {ci }i ) + log p(ci | si ) + log p(xi | si )
{si }i
i
= log p(y | {ci }i ) +
(17) q({si }i | {xi }i ) log p(ci | si ) + log p(xi | si )
XX i
i
{si }i
(18) = log p(y | {ci }i ) +
XX i
= log p(y | {ci }i ) +
X
q(si | xi ) log p(ci | si ) + log p(xi | si )
(19)
si
h i Eq(si |xi ) log p(ci | si ) + log p(xi | si ) .
(20)
i
Combining both terms, we obtain the following evidence lower bound (ELBO): X log p({xi }i , {ci }i , y) ≥ − KL q(si | xi ) ∥ p(si ) + log p(y | {ci }i ) i
+
X
h i Eq(si |xi ) log p(ci | si ) + log p(xi | si ) .
i
D
Additional Visualizations 19
(21)
Figure 6: Visualization of model outputs on CelebA, comparing our PGCM with competing CBMbased approaches. For each input image, we show the predicted concepts and the corresponding visual evidence used by the models. While CBM-based methods only provide concept predictions, PGCM additionally grounds these predictions in selected prototypes, making the underlying visual reasoning explicit.
20
Figure 7: Examples from the CLEVR-Hans3 validation set together with the prototypes selected by our model for each extracted image part. The displayed prototypes correspond to the final model after the prototype swapping step, and therefore represent real image parts from the training data. The figure illustrates how the model assigns prototypes to different objects in the scene; in cases where no additional object is present, the model selects an empty prototype (black image).
21
Figure 8: Generated prototypes on CLEVR-Hans3 together with their predicted concept representations during training, before the prototype swapping step. At this stage, prototypes are still learned representations and may not correspond to realistic image parts. The figure shows how prototypes already capture meaningful visual patterns and associated concepts, even before being replaced by nearest training instances.
22