arXiv:2606.06375v1 [cs.AI] 4 Jun 2026
Rethinking Infrastructure Inspection as Image Difference Classification: A Traffic Sign Case Study Ching Yau Fergus Mok University of Cambridge Cambridge, UK
Lavindra de Silva University of Cambridge Cambridge, UK
Varun Kumar Reja IIT Bombay Mumbai, India
Ioannis Brilakis University of Cambridge Cambridge, UK [email protected]
Abstract
erence image of the same asset taken previously to produce a grounded output. Asset images over time already exist in real road asset management systems (AMSs), and this task formulation uses them as references without needing additional annotation. Due to the wide range of assets, this work focuses on traffic signs as they are both critical to road safety and contain a large number of failure modes. The contributions of this paper are: (1) Curating a high quality and versatile traffic sign imagery dataset with condition annotation. (2) Experimenting with IDC pipelines on traffic sign inspection tasks and quantifying the effects of adding a reference image in reducing data-dependency in low-resource settings.
Digital twins (DTs) allow the digitalization of road infrastructure inspection, though this is hindered by limited annotated data. This work exploits the relational nature of continuous asset condition monitoring to reformulate imagebased defect detection as image difference classification (IDC) to reduce data reliance. This was evaluated in a case study on low-resource traffic sign inspection with different IDC classifiers using a newly-curated, high quality dataset. Results indicate that the instruction-based classifier outperforms encoder-based ones and gains from comparison with reference images. This shows that IDC can be an effective task modeling for tackling data constraints in infrastructure inspection and DT asset condition updating.
2. Related Work Vision models are increasingly capable due to large-scale pre-training which uses large data sources to inject general knowledge to models. They have been used in various IDC and traffic sign maintenance related work.
1. Introduction Maintaining road infrastructure requires manual visual condition inspection. The proposal of a Road Digital Twin (DT) [1], coinciding with increasing road imagery capturing from smart vehicles, presents an opportunity to digitalize this process. Particularly, the Road DT framework tracks the evolving condition of road assets through successive updating steps each involving classifying an image into different condition labels specified by asset-specific data requirements [15]. Annotated data for road maintenance is scarce like in many Architecture, Engineering, and Construction (AEC) fields. To reduce data dependency, this work exploits the relational nature of the DT updating step by rethinking defect identification as an image difference classification (IDC) task where an inspection image is compared against a ref-
2.1. Vision models and IDC A notable pre-trained family is CLIP [12] which performs image-text and image-image comparisons with embeddings similarly to Sentence Transformers for text [14]. Stateof-the-art vision-encoders include DINOv3 [18], SigLIP2 [20], and MetaCLIP2 [4]. Extending from Large Language Models, increasingly popular generative vision language models (VLMs) allow for instruction-tuning. The most sophisticated close-source models are unsuitable for privacysensitive AEC tasks like road maintenance which use government data, so this work only considers open-sourced families like Qwen VL [2] and Gemma [19]. Vision models have been used for image difference 1
tasks. [5] enforced orthogonality to improve IDC for subtle changes. [16] improved difference reasoning with CLIP by aligning image embedding differences with texts. [8] adapted CLIP for difference captioning by designing an image-image-text fusion architecture, while [10] introduced a difference captioning VLM with learned delta tokens.
face aged, sign face vandalized, sign face dirty, and a no defect flag), extending from at most five in other work [17]. Note that significant class imbalance exists between categories, with counts from 20 (sign face vandalized) to 184 (sign face detached or insecure). However, the numbers of damaged and undamaged signs are balanced (∼4:6).
2.2. Computer vision in traffic sign maintenance
3.2. Objective and experiments
Traffic sign imagery datasets with condition annotations are scarce. [17] annotated the Mapillary sign dataset [7] with five condition labels and augmented them to form 20,000 examples while [11] labeled 4,000 sign images from dashboard cameras into four conditions. [13] synthetically added graffiti to signs from the GTSDB dataset [9]. [3] annotated signs into binary condition labels and augmented them to form 2164 damaged and 4330 undamaged signs. For task modeling, [3] created a binary classifier for defect presence, [11] classified for defect types, while [6] and [13] classified for severity modeled by occlusion amount. To the best of the authors’ knowledge, existing work misaligns with real sign inspection needs and there remain large gaps in research. Firstly for datasets, many only contain close-ups of the sign face (without post or background) [17][11] or very coarse condition labeling (binary or one defect) [13][3][6], and none allows for condition tracking over time. Secondly for task modeling, none treats defect identification as a multi-class multi-label task (different defects often occur simultaneously), none leverages multiple images for detection (simulating Road DT updating and condition tracking), with all using only single-images and most relying solely on augmentation to tackle data scarcity [17][13].
The objective of this work is to investigate whether reference images can improve traffic sign defect classification performance through image-image comparison and reduce data-dependency in low-resource settings. The authors hypothesize that asset images over time can be used as references to supplement training data. Two tasks were studied: binary defect presence detection and multi-class multi-label defect classification. The former is relevant to filtering systems for prioritizing images for manual review, while the latter can be directly applicable to Road DT asset condition updating. Both tasks were modeled as a combined multi-class multi-label IDC problem with an additional flag for defect presence. Data constraints were simulated by fine-tuning classifiers using different numbers of examples per class (shot). The performance of IDC pipelines was then compared with the best performing equivalent single-image pipelines that do not use reference images. Experiments used an RTX 4080 with 16GB VRAM, using libraries including Transformers, Pytorch, and Unsloth. All training-set reference images were used during finetuning as pseudo “no defect” examples to ensure models see the same images irrespective of whether classification uses references. Every experiment was repeated with five different data splits to mitigate few-shot learning instability (similarly to [21]), with averaged results reported. Classifier pipelines are divided into encoder-based and instruction-based. They are described in detail below.
3. Methodology To cover the gaps, this section introduces a new dataset and discusses the objectives and experiments performed.
3.2.1. Encoder Based Pipelines
3.1. Dataset
Four encoder-based pipelines were proposed. Two IDC classifiers are purely vision-based: one simply uses a linear classifier to fuse the two concatenated image embeddings while the other adds an image-image cross attention layer (modeled after the cross attention block in the original transformer) prior to the linear layer to allow explicit interaction. Two more IDC classifiers also use textual class descriptions: one adds an image-image-text cross attention layer prior to the linear classifier, while the other uses similarity between image embedding difference and class texts as classification logits (similarly to [16]). The equivalent single-image pipeline for purely vision-based IDC classifiers was chosen as a linear classifier on the single-image embedding, while those for the text-enhanced classifiers were chosen as image-text cross attention prior to the linear classifier and
Traffic sign images were collected from a real road AMS used by National Highways, the UK highway authority. This dataset fills the gap by providing: (1) Real inspection imagery showing all parts of a sign (sign face, post, and background) to capture more failure modes. (2) Two images per unique sign (undamaged reference + inspection) to track conditions. (3) Fine-grained multi-label condition annotation to match real maintenance needs. The dataset (available here1 ) contains 970 image pairs along with annotations into nine condition categories (sign post tilted or fallen, sign post deformed, sign post rotten, sign face detached or insecure, sign face deformed, sign 1 Provisional release, full dataset with additional annotations like sign attributes will be released later.
2
4.2. Defect classification performance
single-image embedding similarity with class texts. MetaCLIP2 was chosen as the backbone as it outperformed other models tested (e.g. SigLIP2).
The second task goes further than simply detecting defect presence by classifying the defect type(s), modeling this as a multi-class multi-label classification task. Tab. 2 shows the macro f 1 scores (averaged over classes) of different IDC pipelines along with the improvements over the best equivalent single-image pipelines that do not use reference images. Scores for this task are lower than the first task due to the increased complexity, with none of the encoder-based pipelines (first four rows) exceeding 0.5 and the instruction-based one achieving just over 0.6. As before, the instruction-based pipeline has a consistent lead over all encoder-based ones, with its 1-shot score exceeding even the 8-shot scores of encoder-based pipelines. The addition of reference images again does not improve the encoder-based scores, while the instruction-based pipeline benefits from it at every shot, with gains ranging from 0.008 to 0.038. The only exception is at 0-shot where providing the reference image to the untrained model degrades performance, again demonstrating that “calibration” through fine-tuning is needed to take advantage of the extra input. Statistical significance of the gains from reference images in the fine-tuned instruction-based pipeline was evaluated through a two-sided paired t-test across data splits, with statistics shown in Tab. 3. 1 and 8-shots see statistically significant improvements with very low p-values while 2 and 4-shots do not, despite all settings still showing a consistently positive trend. This is likely due to instability from training with very few examples in few-shot learning, with the standard deviation being especially big at 2-shot.
3.2.2. Instruction Based Pipeline Instruction-based VLMs have shown strong performance in comparative reasoning [16]. The inspection and reference images are inserted into the user prompt where it is instructed to use the latter as a classification guide. Textual class descriptions are provided in the system prompt. Structured classification outputs are collected from model response in json. Primary investigations focused on finetuning the model based on example interactions. Qwen3 8B was chosen as the model as it outperformed other models tested (e.g. Gemma 3).
4. Results IDC results of the two experimented tasks are explored in this section, with metric values reported being the averages over five data splits to mitigate few-shot instability.
4.1. Defect presence detection performance Defect presence detection was modeled as a binary classification task with a dedicated defect presence flag. Tab. 1 shows the IDC f 1 scores for this task along with the improvements over the best equivalent single-image pipelines that do not use reference images. Most pipelines perform this task well, with many attaining ∼0.9 f 1 even when trained with very few examples per class (shot). This is expected as detecting whether a sign contains a defect is a simple binary task with only two possible outcomes. With the exception of embedding difference similarity, all encoderbased pipelines attain comparable results with none having a significant and consistent lead. However, the instructionbased pipeline consistently outperforms all encoder-based ones, with f 1 reaching above 0.9 at just 1-shot. Encoder-based models (first four rows) do not appear to benefit from the addition of a reference image, with IDC pipelines shifting between being better and worse than the single-image pipelines evidenced by the constantly changing signs. The instruction-based IDC pipeline, however, is able to consistently improve over the single-image pipeline, with every training scenario showing improvements between 0.009 and 0.031. Interestingly though, the untrained instruction-based IDC pipeline classifies almost all images as having a defect, much more so than when not using reference images, resulting in an almost 0 score. This suggests that fine-tuning on just one example per class already effectively “calibrates” the model, showing what a defect is and how to use the reference image.
5. Discussion It is important to note that all pipelines (IDC and singleimage) have access to the same images during fine-tuning, as reference images are still used as pseudo “no defect” examples irrespective of whether they are used for classification. Therefore, any improvement is a result of the additional context of optimal asset states provided by reference images and not of an effective increase in data availability. That being said, two main observations can be made: Firstly, the instruction-based classifier outperforms all encoder-based ones and is able to leverage reference images for performance gains. A potential reason for this gap can be the different number of model parameters, which will be explored in the Ablations in Sec. 6. Secondly, models require “calibration” before being able to make use of the additional reference input. This can be done by fine-tuning with as few as one example per class, which raised f 1 for binary defect presence detection from 0.07 to 0.905. With these observations, IDC can be an effective task modeling for continuous infrastructure inspection and DT condition updating. Reference images can improve detec3
Table 1. Binary defect presence detection f 1 of IDC pipelines along with improvements over the best equivalent single-image pipelines ( f 1±improvement , + means IDC is better). Shots represent minimum number of examples per class. LC stands for Linear Classifier. First four rows are encoder-based classifiers with MetaCLIP2 (2B), final row is an instruction-based classifier with Qwen3-VL-8B.
IDC Best Equivalent Single-Image LC LC Attn. (im-im) + LC LC Attn. (im-txt&im) + LC Attn. (im-txt) + LC Emb. Diff. Sim. Attn. (im-txt) + LC Instruction Instruction
0-shot
1-shot
2-shot
4-shot
8-shot
N/A N/A N/A N/A 0.007−0.222
0.679−0.142 0.800−0.020 0.790−0.035 0.589−0.236 0.905+0.009
0.767−0.102 0.864−0.005 0.851+0.007 0.625−0.218 0.935+0.017
0.870−0.020 0.877−0.013 0.885+0.007 0.681−0.198 0.938+0.012
0.884+0.013 0.897+0.026 0.891−0.007 0.687−0.211 0.940+0.031
Table 2. Multi-class multi-label defect classification macro f 1 of IDC pipelines along with improvements over the best equivalent singleimage pipelines ( f 1±improvement , + means IDC is better). Shots represent minimum number of examples per class. LC stands for Linear Classifier. First four rows are encoder-based classifiers with MetaCLIP2 (2B), final row is an instruction-based classifier with Qwen3-VL8B.
IDC Best Equivalent Single-Image LC LC Attn. (im-im) + LC LC Attn. (im-txt&im) + LC Attn. (im-txt) + LC Emb. Diff. Sim. Attn. (im-txt) + LC Instruction Instruction
0-shot
1-shot
2-shot
4-shot
8-shot
N/A N/A N/A N/A 0.346−0.007
0.266−0.080 0.321−0.025 0.164−0.071 0.206−0.029 0.544+0.033
0.327−0.033 0.371+0.011 0.324+0.008 0.216−0.099 0.536+0.012
0.402−0.023 0.388−0.037 0.410−0.002 0.214−0.197 0.568+0.008
0.449+0.005 0.470+0.026 0.479−0.010 0.219−0.269 0.601+0.038
Table 3. Statistics of improvements in defect classification macro f 1 of instruction-based IDC over the single-image pipeline.
Statistics standard deviation p-value
1-shot 0.012 0.006
2-shot 0.035 0.543
4-shot 0.015 0.366
Table 4. Multi-class multi-label defect classification macro f 1 of IDC pipelines for MetaCLIP2 (2B) and DINOv3 (7B).
8-shot 0.014 0.006
tion results and supplement training data in low-resource settings. However, this is limited to only instruction-based classifiers and requires a small “calibrating” fine-tuning set. A drawback of this is that instability from few-shot learning can create fluctuating outputs which can affect reliability.
Pipeline
CLIP2 (2B) 1-shot 8-shot
DINOv3 (7B) 1-shot 8-shot
LC Attn. (im-im) + LC
0.266 0.321
0.125 0.168
0.449 0.470
0.360 0.327
based and instruction-based pipelines.
6.2. Pseudo defects In the main setup, reference images were used as “no defect” pseudo defect images to keep training data identical between pipelines to avoid unintended gains from an effective increase in data availability when using reference images. Experiments without pseudo defect images were performed for the instruction-based pipeline to understand their effects on performance. Results show that their use improves performance across all training scenarios (e.g. +0.016 for 8-shot IDC), and that the IDC pipeline maintains its lead over the single-image pipeline even without pseudo defects (e.g. +0.053 for 8-shot).
6. Ablations The following ablation studies were carried out.
6.1. Bigger encoder backbone To investigate whether using a bigger vision-encoder comparable to the size of the instruction-based backbone can bridge the gap in performance between the encoder-based and instruction-based pipelines, the 7B DINOv3 was tested in addition to the 2B MetaCLIP2. Tab. 4 shows macro f 1 scores of the two vision-only pipelines with the two backbones. MetaCLIP2 significantly outperforms DINOv3 in every training shot despite being smaller in size. Pure number of model parameters does not seem to be the sole determinant of the difference in performance between encoder-
7. Conclusion The Road DT aims to digitalize road asset condition monitoring. This work exploits the relational nature of continuous asset tracking to reduce data dependency by rethink4
ing defect identification as IDC. This uses already-existing asset images over time as references for grounded classification through image comparison. Using a newly curated dataset of real inspection images, this task formulation was tested in a case study on low-resource traffic sign inspection. Results show that the instruction-based classifier consistently outperforms encoder-based ones, and that with a small fine-tuning dataset of as few as one example per class, it is able to leverage the additional reference inputs for performance gains. Therefore, IDC can be an effective task modeling for reducing data reliance in continuous infrastructure inspection and DT condition updating. Future work includes testing larger datasets to eliminate few-shot fluctuations and experimenting with using more than one reference images per sign.
[10] Erdong Hu, Longteng Guo, Tongtian Yue, Zijia Zhao, Shuning Xue, and Jing Liu. Onediff: A generalist model for image difference captioning. In Computer Vision – ACCV 2024: 17th Asian Conference on Computer Vision, Hanoi, Vietnam, December 8–12, 2024, Proceedings, Part III, page 114–130, Berlin, Heidelberg, 2024. Springer-Verlag. 2 [11] Amr M. Nagy and László Czúni. Deep neural network models for the recognition of traffic signs defects. In 2021 11th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS), pages 725–729, 2021. 2 [12] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR, 2021. 1 [13] Kristian Radoš, Jack Downes, Duc-Son Pham, and Aneesh Krishna. End-to-end traffic sign damage assessment. In 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pages 1–8, 2022. 2 [14] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), pages 3982–3992, Hong Kong, China, 2019. Association for Computational Linguistics. 1 [15] V. K. Reja, M. Yin, D. Davletshina, R. Wei, J. Fauth, F. Perrotta, and I. Brilakis. Establishing & fulfilling information requirements for computer vision enabled digital-twin based control of roadside vegetation. Journal of Construction Engineering and Management, 2026. 1 [16] Dylan Sam, Devin Willmott, Joao D. Semedo, and J. Zico Kolter. Finetuning clip to reason about pairwise differences, 2025. 2, 3 [17] Hanni Sandhu, Joana Kühne, Oliver Sawade, Martin Stellmacher, Elmar Matthes, and Olaf Hellwich. A novel traffic sign dataset with condition annotations. In 2023 IEEE Symposium Series on Computational Intelligence (SSCI), pages 194–199, 2023. 2 [18] Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. Dinov3, 2025. 1 [19] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report, 2025. 1 [20] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil
References [1] Introduction to digital roads. National Highways, 2021. 1 [2] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report, 2025. 1 [3] Tengyang Chen and Jiangtao Ren. Mfl-yolo: An object detection model for damaged traffic signs, 2023. 2 [4] Yung-Sung Chuang, Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, Xinlei Chen, Zhuang Liu, Saining Xie, Wen tau Yih, Shang-Wen Li, and Hu Xu. Meta clip 2: A worldwide scaling recipe, 2025. 1 [5] Fei Dou, Jin Lu, Tan Zhu, and Jinbo Bi. Local: Latent orthonormal contrastive learning for paired image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 6467– 6476, 2025. 2 [6] Can Ersü, Karl Janson, Eduard Petlenkov, and Uljana Reinsalu. Automatic visual traffic sign damage detection and measurement of damaged area. In 2024 18th International Conference on Control, Automation, Robotics and Vision (ICARCV), pages 505–510, 2024. 2 [7] Christian Ertler, Jerneja Mislej, Tobias Ollmann, Lorenzo Porzi, Gerhard Neuhold, and Yubin Kuang. The mapillary traffic sign dataset for detection and classification on a global scale. In Computer Vision – ECCV 2020, pages 68–84, Cham, 2020. Springer International Publishing. 2 [8] Zixin Guo, Tzu-Jui Wang, and Jorma Laaksonen. CLIP4IDC: CLIP for image difference captioning. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 33–42, Online only, 2022. Association for Computational Linguistics. 2 [9] Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. In The 2013 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2013. 2
5
Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual visionlanguage encoders with improved semantic understanding, localization, and dense features, 2025. 1 [21] Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. Efficient few-shot learning without prompts, 2022. 2
6