COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives David Steinmann1,2 , Antonia Wüst1 , Kristian Kersting1,2,3 , and Wolfgang Stammer4,5
arXiv:2606.28194v1 [cs.LG] 26 Jun 2026
1
Computer Science Department, TU Darmstadt 2 Hessian Center for AI (hessian.AI) 3 German Research Center for AI (DFKI) 4 Max Planck Institute for Informatics, SIC 5 RTG Neuroexplicit Models [email protected]
Abstract. While interpretable models such as concept bottleneck models (CBMs) and program synthesis methods enable verification of model decisions, their evaluation is typically limited to simple tasks, leaving complex reasoning on real-world images largely unexplored. We introduce COCOLogic-V2, an object-centric dataset for visual inductive reasoning on real-world images covering a broad subset of first-order logic. By categorizing samples into positive variants, near-boundary (NB), and far-from-boundary (FB) negatives, COCOLogic-V2 enables fine-grained diagnosis of model accountability. Our evaluations show that models tend to separate positive and FB samples well but fail on NB samples, while perceptual noise and large rule-induced search spaces pose additional challenges in few-shot settings. Together, these results highlight that visual inductive reasoning remains an open challenge and COCOLogic-V2 provides a concrete foundation for advancing methods in this direction.6
1
Introduction
Transparency and accountability are crucial requirements for machine learning in high-stakes domains such as healthcare or finance [23]. Interpretable models, such as concept bottleneck models (CBMs) [15,26,6,14] and neuro-symbolic programs [18,9,13,25,32], address this by enabling manual verification of model decisions. However, their evaluation is typically limited to narrow settings: CBMs are often assessed on single-label image classification [15,6,22,24], while program synthesis and neuro-symbolic approaches rely on symbolic representations for text editing [5], abstract pattern matching [3], rather than inductive reasoning [18,9,13]. It therefore remains unclear to what extent such models can tackle more complex tasks, such as visual inductive reasoning on real-world data. The recently introduced COCOLogic dataset [28] takes a step in this direction, providing a reasoning-inspired classification task on real-world im6
Code and dataset: https://github.com/ml-research/COCOLogic-V2
2
D. Steinmann et al.
ages from MSCOCO [17]. However, its single-label framing introduces shortcuts; for example, a bowl alone predicts the "Unlikely Breakfast Guests" class, and the resulting class imbalance further obscures the logical task difficulty. To address these limitations, we introduce COCOLogic-V2. Beyond reframing the task as multilabel classification to reduce shortcuts and class imbalance, COCOLogic-V2 demands reasoning over a wider range of first-order logical operations, including object counting and counting comparisons. Samples are categorized per rule into positive variants (the different ways a rule can be fulfilled), difficult near-boundary (NB) negatives close to the decision boundary, and far-fromboundary (FB) negatives that can often be classified correctly without proper rule understanding, serving together as an automatic diagnostic tool for model accountability. We additionally provide COCOLogic-V2-FS, a curated small version for few-shot and in-context learning. We evaluate several families of approaches on both dataset versions. On COCOLogic-V2, CBM approaches and a black-box baseline reliably separate positive and FB samples but consistently Fig. 1: COCOLogic-V2 restruggle on NB samples, suggesting they exploit quires reasoning over realstatistical patterns rather than learning the un- world images, where each derlying rules. On COCOLogic-V2-FS, perceptual logic rule depends on spenoise and large rule-induced search spaces pose cific objects within an image, additional challenges for in-context learning with illustrated here for two exVLMs and program synthesis approaches. Alto- ample rules: Signal and Ride gether, this work makes the following contribu- and Three of a Kind. tions: (1) we introduce COCOLogic-V2, an objectcentric dataset for visual inductive reasoning over natural images; (2) we provide COCOLogic-V2-FS, a curated few-shot version for data-constrained settings; (3) we show that current interpretable approaches fall short, highlighting open challenges in this setting. The remainder of the paper is structured as follows: In Sec. 2, we introduce COCOLogic-V2 and differentiate it from existing work. Afterward, we provide model evaluations in Sec. 3, a short discussion (Sec. 4), and a conclusion in Sec. 5.
2
The COCOLogic-V2 Dataset
Existing evaluations of interpretable deep learning models fall short of real-world complexity: CBMs are mainly assessed on single-label classification benchmarks [30,4,34], while program synthesis and neuro-symbolic approaches rely on synthetic tasks [20,29,26] or simple concept identification rather than complex reason-
COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives
3
Table 1: COCOLogic-V2 rule definitions. Rules are based on the occurrence or absence of object categories within an image. Some rules additionally require a specific number of certain objects to be present or compare the number of objects from different categories. Name
Logical Rule
Signal and Ride
traffic light ∧ one of {bicycle, bus, train}
Double Serving
Exactly two categories of {bottle, cup, pizza}
Herd Alone
At least two objects of the same category of {cow, elephant, sheep} ∧ no person
Either Dog or Car
Either dog or car
Three of a Kind
Exactly three bowl ∨ exactly three cup
Car Majority
More car than truck ∧ at least one of each
Empty Seat
(couch ∨ chair) ∧ no person
Single Mode Traffic
Exactly one category of {bicycle, motorcycle, car, bus}
Personal Transport
person ∧ (Either bicycle or car)
Surf Trip
Exactly as many person as surfboard ∧ at least one of each
ing [31,12]. We introduce COCOLogic-V2 as a step towards more complex tasks and an evaluation for inductive reasoning over real-world images. COCOLogicV2 builds on COCOLogic [28], expanding the logical scope to include object counting and counting comparisons and reformulating the task as multilabel classification to reduce class imbalance and shortcuts. Most importantly, a new sampling procedure introduces positive variants and near-boundary negatives for automatic model diagnosis. We also introduce COCOLogic-V2-FS, a curated version for few-shot and in-context rule learning on complex real-world images. Compositional Logical Rules. COCOLogic-V2 includes 10 logical rules based on object co-occurrences in MSCOCO [16] images. The rules are semantically meaningful and cover a subset of first-order logic, including propositional logic, object counting, counting comparisons, and object absence. The full rule list is shown in Tab. 1. Beyond binary labels, COCOLogic-V2 provides fine-grained annotations for model diagnosis: positive samples are split into positive variants and negatives into near-boundary (NB) types and far-from-boundary (FB) samples. The positive variants describe the different ways a rule can be satisfied, while the NB types represent samples near the decision boundary, providing specific information about how well a model has learned the rule’s details. The process to obtain positive variants and NB types from a rule is illustrated in Fig. 2. First, it is brought to its disjunctive normal form (DNF). Each positive variant corresponds to one disjunct of the DNF, so that the full positive set is the union of all variants. For "Double Serving", exactly two categories of {bottle, cup, pizza} must be present, yielding the positive variants: (bottle ∧ cup ∧ ¬pizza), (bottle ∧ ¬cup ∧ pizza), and (¬bottle ∧ cup ∧ pizza). Each NB type is derived from a DNF disjunct by flipping one literal, or by modifying object counts for counting rules. For "Double Serving," this yields four different NB types: three single-category variants (bottle ∧ ¬cup ∧ ¬pizza), (¬bottle ∧ ¬cup ∧ pizza),
4
D. Steinmann et al. Exactly two categories of
bottle, cup, pizza
must be present.
DNF bottle ∧ cup ∧ ¬ pizza
∨
bottle ∧ ¬ cup ∧ pizza
Enumerate Positive Variants
∨
¬ bottle ∧ cup ∧ pizza
Permute Near-Boundary (NB) Types
bottle ∧ cup ∧ ¬ pizza
bottle ∧ ¬ cup ∧ ¬ pizza
¬ bottle ∧ ¬ cup ∧ pizza
bottle ∧ ¬ cup ∧ pizza
¬ bottle ∧ cup ∧ ¬ pizza
bottle ∧ cup ∧ pizza
¬ bottle ∧ cup ∧ pizza
Fig. 2: Obtaining Positive Variants and Near-Boundary (NB) Types. First, the logic rule (top) is brought to its disjunctive normal form (DNF). Then each disjunct represents one positive variant. The NB types are obtained by permuting the disjuncts of the DNF, for example, by negating a literal.
(¬bottle ∧ cup ∧ ¬pizza), and the all-three variant (bottle ∧ cup ∧ pizza). We list every NB type and positive variant that can be derived in Sec. A.1. Sampling COCOLogic-V2 from MSCOCO. Since COCOLogic-V2 is based on images from MSCOCO, and most of these images are negative for all rules, we sample 25,000 images to achieve a more balanced distribution. Sampling proceeds in three steps, applied identically to training and test sets (with test images drawn from MSCOCO’s validation split). First, up to 1,000 images per positive variant are randomly selected, with cross-rule duplicates removed where this does not drop any variant below 1,000 samples. Second, images for each NB type are increased to at least 500 (if enough samples are available). Third, FB samples drawn from the remaining MSCOCO images are added to keep COCOLogic-V2’s overall distribution close to MSCOCO’s. The resulting training set contains 25,000 images and the test set 3,500 (see Sec. A.2 for details). All images are annotated with rule labels, positive variant identifiers, and NB type or FB status for negative samples. COCOLogic-V2-FS. This curated subset of COCOLogic-V2 is designed for few-shot and in-context learning. With only 24 training samples per rule, efficient rule learning is essential to succeed on this version. COCOLogic-V2-FS assumes a functioning perception module and is not designed to train such a module from scratch. All images were manually selected to ensure relevant objects are clearly visible and MSCOCO labels are correct. COCOLogic-V2-FS contains 239 training and 368 test images in total. Per rule, the training set contains 8 positive and 16 negative samples and the test set contains 20 of each. Positive images are evenly distributed across variants, and negative images include two fixed FB samples with the remainder spread evenly across NB types (cf. Sec. A.2 for detailed sample distributions).
COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives
5
Table 2: Concept-based Models fail to learn details of logical rules on COCOLogic-V2. High overall B-Acc paired with low performance on NB types shows that models learn a broad separation between positive and easy FB samples, but fail when challenged with samples that require a proper understanding of the logic rule. Best bold, runner up underlined, mean ± std over 5 seeds.
3
Encoder
Predictor
Oracle Oracle
Linear MLP
B-Acc
NB Types
Positive Variants
94.0 ± 0.1 99.6 ± 0.1
44.1 ± 0.3 94.3 ± 2.5
96.0 ± 0.1 99.6 ± 0.2
ResNet-50 MaskRCNN MaskRCNN Supervised Supervised DCR CLIP CLIP OCB OCB
–
86.8 ± 0.4
40.5 ± 1.4
86.5 ± 1.2
Linear MLP Linear MLP DCR Linear MLP Linear MLP
88.8 ± 0.1 92.1 ± 0.2 86.0 ± 0.1 86.7 ± 0.1 82.3 ± 3.6 84.7 ± 0.1 86.5 ± 0.2 85.8 ± 0.1 86.2 ± 0.3
42.0 ± 0.4 48.5 ± 0.8 30.8 ± 0.7 32.9 ± 0.5 48.5 ± 8.6 28.1 ± 0.2 27.1 ± 1.0 31.2 ± 0.4 31.3 ± 1.2
89.1 ± 0.3 92.9 ± 0.3 89.6 ± 0.3 90.9 ± 0.6 76.6 ± 9.1 87.8 ± 0.3 92.2 ± 0.7 88.9 ± 0.5 90.4 ± 0.5
Experimental Evaluations
In this section, we evaluate several interpretable approaches alongside black-box baselines to assess the current state of visual inductive reasoning on COCOLogicV2. We aim to answer two questions: (1) How well do current concept-based approaches perform on visual inductive reasoning? (2) What are the key bottlenecks for visual inductive reasoning in the few-shot setting? Models. On the full dataset, we evaluate a fine-tuned ResNet-50 [11] as a black-box baseline, and a range of CBM variants with linear or MLP predictors: using ground-truth object labels (Oracle), a pretrained Mask-RCNN [10] detector, a supervised ResNet-50 concept encoder [15], CLIP-based concept representations [33], object-centric CBMs (OCB) [28], and Deep Concept Reasoner (DCR) [2]. On COCOLogic-V2-FS, we additionally evaluate GPT-5.5 [21], Gemini-3.1-ProPreview [7], and Claude-Opus-4-7 [1] for in-context learning, and the VLP program synthesis approach [32] with Gemma-4-31B-it [8] as backbone. Further implementation details are in Sec. A.4. Metrics. All metrics are computed per rule and averaged across rules. We report balanced accuracy (B-Acc) as an overall summary, alongside separate accuracies on positive variants and NB types. The latter two are more informative than raw B-Acc, as they are not inflated by easy FB samples and show directly whether a model has learned the logical rules. Concept-Based Approaches on COCOLogic-V2 (RQ1). The results in Tab. 2 appear promising at first: all models exceed 80% B-Acc overall. However, NB accuracy reveals a fundamental shortcoming, as all models except Oracle + MLP perform at or below random guessing on NB samples. Rather than learning the logical rules, models separate positives from easy FB samples based
6
D. Steinmann et al.
Table 3: Combining Perception and Rule Induction on few, real-world images remains challenging. Training CBM Predictors on COCOLogic-V2FS per-rule gives low performance and unstable results. In contrast, in-context learning and program synthesis perform better, but identifying what objects are present and relevant to find the logical rules remains a key challenge. Model
B-Acc
NB Types
Positive Variants
Oracle + Linear Oracle + MLP
58.0 ± 2.4 62.8 ± 1.9
56.6 ± 8.1 62.9 ± 13.4
55.7 ± 8.4 59.0 ± 13.2
ResNet-50
62.3 ± 1.6
74.9 ± 4.2
47.0 ± 4.4
MaskRCNN + Linear MaskRCNN + MLP Supervised + Linear Supervised + MLP DCR CLIP + Linear CLIP + MLP OCB + Linear OCB + MLP
51.3 ± 1.9 49.5 ± 2.1 60.8 ± 2.6 61.7 ± 2.3 52.3 ± 3.1 55.8 ± 2.9 61.5 ± 1.4 52.4 ± 1.3 51.1 ± 1.7
68.0 ± 3.6 60.2 ± 27.7 50.6 ± 6.1 57.0 ± 19.3 69.1 ± 16.2 47.7 ± 12.9 53.2 ± 1.3 64.7 ± 2.5 69.8 ± 5.2
35.9 ± 5.6 38.6 ± 23.9 67.6 ± 6.7 62.8 ± 21.5 36.5 ± 19.6 58.6 ± 16.0 62.9 ± 2.4 40.6 ± 3.0 31.6 ± 5.7
In-Context Learning & Program Synthesis VLP (Oracle) 89.2 ± 0.0 90.6 ± 0.0
88.4 ± 0.0
VLP (Gemma-4-31B-it) Gemma-4-31B-it GPT-5.5 Gemini-3.1-Pro-Preview Claude-Opus-4-7
72.3 ± 1.3 65.4 ± 2.3 68.1 ± 1.4 74.8 ± 2.9 75.5 ± 2.4
74.5 ± 2.1 60.7 ± 4.7 74.3 ± 3.5 62.7 ± 8.1 80.5 ± 4.8
66.1 ± 3.5 65.6 ± 6.8 56.3 ± 3.8 78.3 ± 3.4 65.9 ± 5.2
on the coarse presence of relevant objects. NB accuracy degrades further as concept representations become noisier: the pretrained detector outperforms the supervised encoder, and CLIP-based representations perform worst. This suggests that fine-grained rule details become increasingly difficult to capture with lower-quality concepts. While DCR’s logic-based predictor is, in principle, well-suited for these tasks, it is hindered by noisy perception and a large concept space. The Oracle upper bound confirms the dataset is solvable in principle: a linear predictor alone fails on rules involving XOR or exact counting, but an MLP achieves near-perfect performance. The detailed results per rule further show that overall and NB accuracy are largely independent across rules (cf. Sec. A.3). Visual Inductive Reasoning in Few-Shot Settings (RQ2). We now examine the few-shot setting to identify where current approaches break down when training signal is scarce. We observe in Tab. 3 that all CBM approaches struggle on COCOLogic-V2-FS: even Oracle + MLP reaches only 62.8% B-Acc with high variance across seeds. The limited training signal from 24 samples per rule makes it difficult to learn reliable predictors, and differences between encoder types are largely overshadowed by high variance. Interestingly, NB accuracy tends to be higher here than on the full dataset, as models do not systematically misclassify NB samples without sufficient data to overfit to positive vs. FB
COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives
7
differences. In contrast, in-context learning with VLMs offers a clear advantage over CBMs, with all three tested models reaching around 70% B-Acc. In some cases, they recover the correct rule, as Claude produces the exact ground-truth description for Signal and Ride. However, for rules where identifying relevant objects is non-trivial, VLMs fall short: on Double Serving, Claude describes a semantically related but logically imprecise pattern rather than the exact counting constraint. Program synthesis with VLP (Gemma-4-31B-it) performs comparably to the best VLMs and benefits from an explicit structure for rule induction, despite the lower baseline performance of Gemma compared to other VLMs. Providing ground-truth concepts (VLP Oracle) boosts performance substantially to 88.5%, but even then, complex rules such as Single Mode Traffic remain difficult to identify within constrained search time. Together, these results suggest that reliable identification of which objects are present and decisions about which are relevant remain the primary bottleneck for few-shot visual inductive reasoning.
4
Discussion
Our evaluations on COCOLogic-V2 show that visual logical reasoning on realworld images remains a significant open challenge. On the full dataset, models achieve high overall accuracy by separating positives from easy FB samples, but fail on NB samples, indicating they exploit coarse statistical patterns rather than learning the underlying rules. The annotated sample categories enable automatic diagnosis of such shortcomings without detailed human inspection: a model that fails on NB samples while achieving high overall accuracy is clearly not learning the intended rule. In few-shot settings, perceptual noise and large rule-induced search spaces pose additional challenges that even specialized approaches such as program synthesis do not fully overcome. Together, these results validate the need for a benchmark like COCOLogic-V2 that goes beyond aggregate accuracy to reveal where and why current interpretable models fall short. However, NB annotations do not entirely replace human inspection. Models can still exploit other shortcuts not captured by the NB structure, such as using the presence of water to predict the Surf Trip rule. Mitigating general [27] and reasoning-specific shortcuts [19] thus remains important to ensure accountability.
5
Conclusion
We introduced COCOLogic-V2, a dataset for visual inductive reasoning on realworld images, along with a curated few-shot version COCOLogic-V2-FS. Our evaluations show that current approaches, including CBMs, VLMs, and program synthesis, struggle to capture the underlying logical rules, particularly on nearboundary samples, highlighting that visual logical reasoning on real-world data remains an open challenge. COCOLogic-V2 provides a useful foundation for advancing and benchmarking methods in this direction. Going forward, extending the dataset to include more complex logical structures, object relations, and more
8
D. Steinmann et al.
abstract scene-level information is a promising direction that will further test the limits of interpretable models. Acknowledgments. This work was supported by the ”ML2MT” project from the Volkswagen Stiftung, by the German Research Foundation (DFG) under Germany’s Excellence Strategy (EXC 3066/1 “The Adaptive Mind”, Project No. 533717223; and GRK 2853 "Neuroexplicit Models of Language, Vision, and Action", Project No. 471607914). It was further supported by the EU-funded “TANGO” project (EU Horizon 2023, GA No 57100431), and the Priority Program (SPP) 2422 in the subproject “Optimization of active surface design of high-speed progressive tools using machine and deep learning algorithms“ funded by the German Research Foundation (DFG). It has benefited from the HMWK project Hessian.AI, and from the Cluster of Excellence "Reasonable AI" funded by the DFG under Germany’s Excellence Strategy EXC-3057. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.
References 1. Anthropic: Claude opus 4.7. Model card (2026), https://www.anthropic.com/ claude/opus 2. Barbiero, P., Ciravegna, G., Giannini, F., Zarlenga, M.E., Magister, L.C., Tonda, A., Lió, P., Precioso, F., Jamnik, M., Marra, G.: Interpretable neural-symbolic concept reasoning. In: International Conference on Machine Learning. pp. 1801–1825. PMLR (2023) 3. Chollet, F.: On the measure of intelligence. arXiv preprint arXiv:1911.01547 (2019) 4. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009) 5. Ellis, K., Wong, L., Nye, M., Sable-Meyer, M., Cary, L., Anaya Pozo, L., Hewitt, L., Solar-Lezama, A., Tenenbaum, J.B.: Dreamcoder: growing generalizable, interpretable knowledge with wake–sleep bayesian program learning. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 381(2251) (2023) 6. Espinosa Zarlenga, M., Barbiero, P., Ciravegna, G., Marra, G., Giannini, F., Diligenti, M., Shams, Z., Precioso, F., Melacci, S., Weller, A., et al.: Concept embedding models: Beyond the accuracy-explainability trade-off. Advances in neural information processing systems 35, 21400–21413 (2022) 7. Google DeepMind: Gemini 3.1 pro (preview). Model card (2026), https://deepmind. google/models/gemini/pro/ 8. Google DeepMind: Gemma 4. Model card (2026), https://deepmind.google/ models/gemma/gemma-4/ 9. Gupta, T., Kembhavi, A.: Visual programming: Compositional visual reasoning without training. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 10. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961–2969 (2017) 11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives
9
12. Jiang, H., Ma, X., Nie, W., Yu, Z., Zhu, Y., Anandkumar, A.: Bongard-hoi: Benchmarking few-shot visual reasoning for human-object interactions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 19056–19065 (2022) 13. Kamali, D., Kordjamshidi, P.: NePTune: A neuro-pythonic framework for tunable compositional reasoning on vision-language. In: The Fourteenth International Conference on Learning Representations (2026) 14. Knab, P., Steinmann, D., Bartelt, C., Kersting, K., Schiele, B., Seidl, T., Schlegel, U., Stammer, W.: What’s in the bottle? a survey and roadmap of concept bottleneck models. Transactions on Machine Learning Research (2026) 15. Koh, P.W., Nguyen, T., Tang, Y.S., Mussmann, S., Pierson, E., Kim, B., Liang, P.: Concept bottleneck models. In: International Conference on Machine Learning (ICML). pp. 5338–5348 (2020) 16. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014) 17. Lin, T., Maire, M., Belongie, S.J., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: European Conference on Computer Vision (ECCV). pp. 740–755 (2014) 18. Mao, J., Gan, C., Kohli, P., Tenenbaum, J.B., Wu, J.: The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In: International Conference on Learning Representations (2019) 19. Marconato, E., Teso, S., Vergari, A., Passerini, A.: Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts. Advances in Neural Information Processing Systems 36, 72507–72539 (2023) 20. Müller, H., Holzinger, A.: Kandinsky patterns. Artificial intelligence 300, 103546 (2021) 21. OpenAI: Gpt-5.5. System card (2026), https://openai.com/index/ gpt-5-5-system-card/ 22. Panousis, K.P., Ienco, D., Marcos, D.: Coarse-to-fine concept bottleneck models. Advances in Neural Information Processing Systems 37, 105171–105199 (2024) 23. Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1(5), 206–215 (2019) 24. Shang, C., Zhou, S., Zhang, H., Ni, X., Yang, Y., Wang, Y.: Incremental residual concept bottleneck models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11030–11040 (2024) 25. Shindo, H., Pfanschilling, V., Dhami, D.S., Kersting, K.: α ilp: thinking visual scenes as differentiable logic programs. Machine Learning 112(5), 1465–1497 (2023) 26. Stammer, W., Schramowski, P., Kersting, K.: Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In: Conference on Computer Vision and Pattern Recognition CVPR. pp. 3619–3629 (2021) 27. Steinmann, D., Divo, F., Kraus, M., Wüst, A., Struppek, L., Friedrich, F., Kersting, K.: Navigating shortcuts, spurious correlations, and confounders: From origins via detection to mitigation. arXiv preprint arXiv:2412.05152 (2024) 28. Steinmann, D., Stammer, W., Wüst, A., Kersting, K.: Object-centric conceptbottlenecks. Advances in Neural Information Processing Systems 38, 68899–68924 (2026) 29. Vedantam, R., Szlam, A., Nickel, M., Morcos, A., Lake, B.M.: Curi: A benchmark for productive concept learning under uncertainty. In: International Conference on Machine Learning. pp. 10519–10529. PMLR (2021)
10
D. Steinmann et al.
30. Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-ucsd birds 200 (2010) 31. Wu, R., Ma, X., Zhang, Z., Wang, W., Li, Q., Zhu, S.C., Wang, Y.: Bongardopenworld: Few-shot reasoning for free-form visual concepts in the real world. In: International Conference on Learning Representations. vol. 2024, pp. 20688–20718 (2024) 32. Wüst, A., Stammer, W., Shindo, H., Helff, L., Dhami, D.S., Kersting, K.: Synthesizing visual concepts as vision-language programs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17346–17356 (2026) 33. Yang, Y., Panagopoulou, A., Zhou, S., Jin, D., Callison-Burch, C., Yatskar, M.: Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 19187–19197 (2023) 34. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence 40(6), 1452–1464 (2017)
COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives
11
Supplementary Materials A
Additional Details for COCOLogic-V2
A.1
Positive Variants and Near Boundary Types
COCOLogic-V2 consists of 10 rules, each with several positive variants and near-boundary (NB) types. In Fig. 3, an example image of each rule is given, where the objects that are relevant to the respective rule are highlighted. These examples highlight the wide variety and difficulty of identifying potential objects and deciding which might be relevant to the task, compared to synthetic datasets that are often clearer and contain less potentially relevant information. bottle traffic light bus trafficlight light traffic
trafficlight light traffic traffic light
cup
cup
cow
cup
cow cowcow
car bottlecup cup bottle bottle
Signal and Ride
Double Serving
Herd Alone
couch truck truck car car carcar car car car car
couch
Three of a kind
Dog or Car car car person car person
person
bicycle
car person
cartruck
Car Majority
cup
surfboard
Empty Seat
Single Mode Traffic
Personal Transport
Surf Trip
Fig. 3: Example images for all rules of COCOLogic-V2 with the relevant objects highlighted.
In the following, we provide a detailed list of all positive variants and NB types for all rules. The positive variants are enumerated with Pi and the NB types are enumerated with Ni. In cases where rules require object counting, we simplify the logical structure for object counts and assume the presence of a predicate that provides the number of times an object appears in an image, for further use in the logical formulation. Following this, the NB types for these rules modify the required object counts instead of directly flipping a literal (e.g., for rule 5, where N1 is "1 – 2 cup, rather than "¬ exactly 3 cup"). Rule 1: traffic light ∧ one of {bicycle, bus, train} Positive Variants P1 P2 P3
traffic light ∧ bicycle traffic light ∧ bus traffic light ∧ train
Near Boundary Types N1 N2 N3 N4
bicycle ∧ ¬ traffic light bus ∧ ¬ traffic light train ∧ ¬ traffic light traffic light ∧ none bicycle/bus/train
of
12
D. Steinmann et al. Rule 2: Exactly two categories of {bottle, cup, pizza} Positive Variants P1 P2 P3
bottle ∧ cup ∧ ¬ pizza bottle ∧ pizza ∧ ¬ cup cup ∧ pizza ∧ ¬ bottle
Near Boundary Types N1 N2 N3 N4
bottle ∧ ¬ cup ∧ ¬ pizza cup ∧ ¬ bottle ∧ ¬ pizza pizza ∧ ¬ bottle ∧ ¬ cup bottle ∧ cup ∧ pizza
Rule 3: At least two objects of the same category of {cow, elephant, sheep} ∧ no person Positive Variants P1 P2 P3
Two or more cow ∧ ¬ person Two or more elephant ∧ ¬ person Two or more sheep ∧ ¬ person
Near Boundary Types N1 N2 N3 N4 N5 N6
Exactly one elephant ∧ ¬ person Exactly one cow ∧ ¬ person Exactly one sheep ∧ ¬ person Two or more elephant ∧ person Two or more cow ∧ person Two or more sheep ∧ person
Rule 4: Either dog or car Positive Variants P1 P2
dog ∧ ¬ car car ∧ ¬ dog
Near Boundary Types N1 N2
dog ∧ car ¬ dog ∧ ¬ car
Rule 5: Exactly three bowl ∨ exactly three cup Positive Variants
Near Boundary Types
P1 P2
N1 N2 N3 N4
Exactly 3 bowl Exactly 3 cup
1–2 cup 4 or more cup 1–2 bowl 4 or more bowl
Rule 6: More car than truck ∧ at least one of each Positive Variants
Near Boundary Types
P1
N1 N2
More car than truck; both present
N3
car ∧ ¬ truck Equal number of car and truck; both present More truck than car; both present
COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives
13
Rule 7: (couch ∨ chair) ∧ ¬ person Positive Variants P1 P2
couch ∧ ¬ person chair ∧ ¬ person
Near Boundary Types N1 N2 N3
∧ person chair ∧ person ¬ couch ∧ ¬ chair ∧ ¬ person
Rule 8: Exactly one category of {bicycle, motorcycle, car, bus} Positive Variants
Near Boundary Types
P1
bicycle ∧ none of motorcycle/car/bus
N1
P2
motorcycle ∧ none of bicycle/car/bus
N2
P3
car ∧ none of bicycle/motorcycle/bus
N3
P4
bus ∧ none of bicycle/motorcycle/car
N4 N5 N6
bicycle ∧ motorcycle ∧ none car/bus bicycle ∧ car ∧ none motorcycle/bus bicycle ∧ bus ∧ none motorcycle/car motorcycle ∧ car ∧ none bicycle/bus motorcycle ∧ bus ∧ none bicycle/car car ∧ bus ∧ none bicycle/motorcycle
of of of of of of
Rule 9: person ∧ (Either bicycle or car) Positive Variants P1 P2
person ∧ bicycle ∧ ¬ car person ∧ car ∧ ¬ bicycle
Near Boundary Types N1 N2 N3 N4
person ∧ bicycle ∧ car person ∧ ¬ bicycle ∧ ¬ car bicycle ∧ ¬ car ∧ ¬ person car ∧ ¬ bicycle ∧ ¬ person
Rule 10: Exactly as many person as surfboard Positive Variants
Near Boundary Types
P1
N1
surfboard ∧ ¬ person
N2 N3
person ∧ ¬ surfboard More surfboard than person; both present More person than surfboard; both present
Equal number of person and surfboard; both present
N4
A.2
Number of Samples per Variant and NB Type
In the tables Tab. 4 and Tab. 5, the exact number of samples for each positive variant and NB type is presented. It is noteworthy that, due to the small size of the COCO validation set, the total number of samples in the COCOLogic-V2 test set is low for some NB types.
14
D. Steinmann et al.
Table 4: COCOLogic-V2 per-rule sample counts. Pi and Ni denote positive variants and NB types for the respective rules. All other samples that are not a positive variant or an NB type are considered FB samples. Rule
Split
Positive Variants
Near Boundary
PV1
PV2
PV3
PV4
NB1
NB2
NB3
NB4
NB5
NB6
Rule 1 (Signal and Ride)
train test
344 24
734 35
376 23
– –
1926 125
1227 154
541 134
616 117
– –
– –
Rule 2 (Double Serving)
train test
1106 112
311 20
657 39
– –
639 231
954 223
521 78
351 16
– –
– –
Rule 3 (Herd Alone)
train test
936 48
801 42
948 31
– –
476 13
370 14
167 8
368 15
336 11
275 15
Rule 4 (Either Dog or Car)
train test
1308 158
3308 516
– –
– –
481 19
113190 4307
– –
– –
– –
– –
Rule 5 (Three of a Kind)
train test
608 25
842 29
– –
– –
1632 296
594 65
957 256
500 33
– –
– –
Rule 6 (Car Majority)
train test
1213 104
– –
– –
– –
1247 260
523 37
560 109
– –
– –
– –
Rule 7 (Empty Seat)
train test
1102 81
1338 189
– –
– –
500 90
1442 367
9697 1344
– –
– –
– –
Rule 8 (Single Mode Traffic)
train test
1334 75
1071 81
1991 356
1007 76
205 14
500 33
137 6
500 36
70 3
639 78
Rule 9 (Personal Transport)
train test
1205 77
2201 314
– –
– –
515 45
9080 1488
475 20
998 169
– –
– –
Rule 10 (Surf Trip)
train test
1088 97
– –
– –
– –
38 3
7796 1175
133 3
500 42
– –
– –
COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives
15
Table 5: COCOLogic-V2-FS per-rule sample counts. Pi and Ni denote positive variants and NB types for the respective rules. All other samples that are not a positive variant or an NB type are considered FB samples. Rule
Split
Positive Variants
Near Boundary
PV1
PV2
PV3
PV4
NB1
NB2
NB3
NB4
NB5
NB6
Rule 1 (Signal and Ride)
train test
3 7
3 7
2 6
– –
3 4
3 4
3 3
3 4
– –
– –
Rule 2 (Double Serving)
train test
3 7
3 7
2 6
– –
3 4
3 4
3 4
3 3
– –
– –
Rule 3 (Herd Alone)
train test
3 7
3 7
2 6
– –
2 3
2 3
2 3
2 2
2 2
2 3
Rule 4 (Either Dog or Car)
train test
4 10
4 10
– –
– –
8 10
8 10
– –
– –
– –
– –
Rule 5 (Three of a Kind)
train test
4 10
4 10
– –
– –
3 4
3 3
3 4
3 4
– –
– –
Rule 6 (Car Majority)
train test
8 20
– –
– –
– –
4 5
4 4
4 5
– –
– –
– –
Rule 7 (Empty Seat)
train test
4 10
4 10
– –
– –
4 5
4 5
3 4
– –
– –
– –
Rule 8 (Single Mode Traffic)
train test
2 5
2 5
2 5
2 5
2 3
2 3
2 2
2 3
2 2
2 3
Rule 9 (Personal Transport)
train test
4 10
4 10
– –
– –
3 4
3 4
3 3
3 4
– –
– –
Rule 10 (Surf Trip)
train test
8 20
– –
– –
– –
3 3
3 4
3 2
3 4
– –
– –
16
D. Steinmann et al.
A.3
Detailed Results
In Fig. 4, the performance of all models on each individual rule is shown. The results show that the overall performance does not correlate much with performance on the NB types. Overall, models perform best on Surf Trip and Herd Alone, but the NB performance stays around random guessing. The rule with the best NB performance is Empty Seat, which is reasonable as the NB types are quite simple and can be separated from positive samples by identifying a person. When considering rules that require object counting (i.e., Herd Alone, Three of a Kind, Car Majority, and Surf Trip), detector-based CBMs and OCB do not perform much better despite having access to object counts directly. The noise in the concept encoder makes the object counts insufficient to gain a substantial advantage on these rules.
Overall
100 80 60 40 20 0
Positive Variants
100 80 60 40 20 0
NB Types
100 80 60 40 20 0
Signal and Ride
Double Serving
ResNet-50 Oracle Linear
Herd Alone Oracle MLP Detector Linear
Either Dog or Car
Three of a kind
Detector MLP Sup. CBM Linear
Car Majority
Empty Seat Single Mode Traffic
Sup. CBM MLP CLIP CBM Linear
Personal Transport
CLIP CBM MLP OCB Linear
Surf Trip OCB MLP Deep CoRe
Fig. 4: Per-rule performance on COCOLogic-V2. Overall balanced accuracy (top), accuracy averaged over all positive variants of a rule (middle), and accuracy averaged over all NB types of a rule (bottom) on the full dataset.
COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives
A.4
17
Experimental Details
Code. The code will be made public upon acceptance. It also includes detailed information on hyperparameters and experimental configurations. In the following, we provide an overview of the experimental setup. Training Models on COCOLogic-V2. All models were trained on the training split of COCOLogic-V2, where 10% of the samples were randomly used as a validation set. Learning rate, number of epochs trained, and modelspecific parameters were optimized using the validation set. The final models were trained for 5 seeds with the best parameter configuration and early stopping. For the CLIP-based CBM and OCB, the textual vocabulary is the list of all object-category names of MSCOCO without further modifications. For DCR, the encoder is a concept embedding model [6] (also instantiated with a pretrained ResNet-50) that is jointly trained with the logic-based predictor as in [2]. Training Models on COCOLogic-V2-FS. The training setup differs from the full version due to the small number of training instances and the different general setup. For this version, one model is trained to predict each rule only on the training samples that belong to this rule, i.e., 10 models are trained on 24 samples each, and each is evaluated only on the 40 test samples for its rule. As the training set for each of these models contains only 24 samples, using a dedicated validation set is of little use. Instead, the hyperparameters of all models were optimized using leave-one-out cross-validation. Besides these differences, the models were trained similarly to the full version, with the same hyperparameters being optimized. The concept encoders were fixed for all CBMs except DCR, which jointly trains the encoder and the predictor. The ResNet-50 model was also fine-tuned directly on COCOLogic-V2-FS. In-Context Learning on COCOLogic-V2-FS. For the In-Context Learning baseline, we provide the models with the few-shot images of a task and ask them to infer the underlying rule. All models are run with their default reasoning effort, and max_tokens is set to 32768 to allow for long reasoning traces. At test time, the model is then shown each test image individually, together with the inferred rule, and asked to classify the image according to that rule. Each model is evaluated with 5 seeds. Prompt for In-Context Learning You are given {n} images. Each image depicts a scene with specific objects, interactions , and environments. Your task is to determine the underlying concept that distinguishes the positive examples from the negative examples, based on the objects, their properties, and the actions occurring in each scene. - The first {m} images are positive examples. - The remaining {o} images are negative examples. ## Task Identify the rule that defines the positive examples:
18
D. Steinmann et al.
- The rule must apply to all positive examples. - The rule must not apply to any negative example. ## Step-by-Step Process 1. Image Analysis: - Carefully describe each image, noting objects, their attributes, and conceptual features (such as relationships, actions, or settings). 2. Rule Derivation: - From your analysis, infer the rule that uniquely characterizes the positive examples. - Confirm that the rule does not hold for the negative examples. ## Final Answer Format Provide your final answer in the following format: ‘‘‘python answer = { ’rule’: ’[RULE]’, } ‘‘‘ Ensure that the rule is clearly defined and concise.
Test Prompt Given the rule ’{response}’, determine if the image follows the rule or not. Answer with ’Yes’ or ’No’, nothing else.
Program Synthesis on COCOLogic-V2-FS. For the Vision-Language Program approach, we use the DSL shown in Tab. 6. It comprises the VLM function get_objects, logical functions that check for the existence of objects and properties, as well as counting functions, arithmetic functions, and Boolean logic operators, all of which were already used in the original paper. We extend the DSL with a new primitive, addb, which adds an integer and a Boolean, enabling us to count how many of several logical statements are true. Programs are enumerated via heap search with a budget of 300 seconds and a maximum program depth of 8.
COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives
19
Table 6: Primitives of the DSL used for VLP and their types. Arrows associate to the right, i.e. A → B → C denotes a curried function A → (B → C). [[Str]] abbreviates a list of lists of strings. Primitive
Type
Perception get_objects
Img → [[Str]]
Existence predicates exists_object exists_object_with_property exists_property
[[Str]] → Obj → Bool [[Str]] → Obj → Prop → Bool [[Str]] → Prop → Bool
Counting count_object_in_img count_all_objects
[[Str]] → Obj → Int [[Str]] → Int
Boolean logic and, or, xor not
Bool → Bool → Bool Bool → Bool
Arithmetic and comparison gt?, eq? addb 0, 1, 2, 3
Int → Int → Bool Int → Bool → Int Int