arXiv:2604.13555v1 [cs.CV] 15 Apr 2026
AI-Powered Image Analysis for Phishing Detection Kaushal Acharya
Sunil Ale
Rajan Kadel
Melbourne Institute of Technology School of IT and Engineering Melbourne, Australia [email protected]
Melbourne Institute of Technology School of IT and Engineering Melbourne, Australia [email protected]
National Academy of Professional Studies School of IT Melbourne, Australia [email protected]
Abstract—Phishing websites now rely heavily on visual imitation—copied logos, similar layouts, and matching colours—to avoid detection by text- and URL-based systems. This paper presents a deep learning approach that uses webpage screenshots for image-based phishing detection. Two vision models, ConvNeXt-Tiny and Vision Transformer (ViT-Base), were tested to see how well they handle visually deceptive phishing pages. The framework covers dataset creation, preprocessing, transfer learning with ImageNet weights, and evaluation using different decision thresholds. The results show that ConvNeXt-Tiny performs the best overall, achieving the highest F1-score at the optimised threshold and running more efficiently than ViT-Base. This highlights the strength of convolutional models for visual phishing detection and shows why threshold tuning is important for real-world deployment. As future work, the curated dataset used in this study will be released to support reproducibility and encourage further research in this area. Unlike many existing studies that primarily report accuracy, this work places greater emphasis on threshold-aware evaluation to better reflect real-world deployment conditions. By examining precision, recall, and F1-score across different decision thresholds, the study identifies operating points that balance detection performance and false-alarm control. In addition, the side-by-side comparison of ConvNeXt-Tiny and ViT-Base under the same experimental setup offers practical insights into how convolutional and transformer-based architectures differ in robustness and computational efficiency for visual phishing detection. Index Terms—Phishing detection, image-based analysis, deep learning, computer vision, ConvNeXt-Tiny, Vision Transformer, transfer learning, threshold optimisation, cybersecurity.
I. I NTRODUCTION One of the pervasive cybersecurity threats is phishing. It has exploited human trust through deceptive web interfaces and digital communications. When studying traditional phishing detection systems, they have relied mainly on textual content or URL-based analysis. New age attackers now employ and use visual deception logos, colour schemes, and layout structures to mimic and present themselves as legitimate websites. They also evade text- and domain-based filters. These evolving threats have made image-based phishing detection an area of research, as it is emerging as a critical frontier in cybersecurity [1, 2]. The development and advancement of deep learning and computer vision in recent years have enabled automated extraction of spatial and contextual features. This is done by using webpage screenshots. Because of this, detection models are now capable of recognising and flagging subtle inconsistencies in visual composition. Convolutional Neural
Networks (CNNs) and transformer-based vision architectures have shown and demonstrated the identification of such patterns. Most of the existing studies largely focus on accuracy but silently overlook and neglect other essential aspects, which are data set diversity, rigorous threshold optimisation, and model interpretability that are crucial for real-world deployment. This paper addresses these areas not addressed by the existing literature. To achieve it, we developed an AI-powered image analysis framework, which is capable of systematically evaluating and analysing modern vision architectures for advanced phishing detection. For this, we have selected two representative vision models, ConvNeXt-Tiny [3] and Vision Transformer (ViT-Base) [4]. We used these models to train and analyse their architectural strengths and feature extraction capabilities. Also, robustness against visually obfuscated phishing attacks. Selected models follow systematic processes in the order of dataset collection and construction, preprocessing, augmentation, model training, and threshold tuning and analysis for maximum output. Balancing recall and precision for deployment-relevant decision-making is the major result of threshold tuning, which is the objective of the paper. In summary, this study addresses the gap between the high reported accuracy in image-based phishing detection and the practical challenges of deploying such systems in real-world settings. While prior research has shown promising results, many studies rely primarily on accuracy and do not examine how models behave under different decision thresholds. The main purpose of this work is therefore to develop and evaluate a threshold-aware framework that provides a more realistic assessment of model performance. By systematically analysing precision, recall, and F1-score across varying thresholds and comparing ConvNeXt-Tiny and ViT-Base under the same experimental conditions, this study aims to offer clearer insight into both performance stability and practical applicability. The contributions of this paper are as follows: • A threshold-aware evaluation methodology that optimises precision, recall, and F1-score for real-world deployment. An experimental analysis highlighting architectural trade-offs between accuracy and computational efficiency • An end-to-end framework for image-based phishing detection using webpage screenshots, comparative evaluation of ConvNeXt-Tiny and transformer-based vision architectures;
This is a preprint version of the paper accepted for the 9th International Conference on Inventive Computation Technologies (ICICT), 2026.
The remainder of this paper is structured as follows. Section II reviews related works on phishing detection. Section III defines the problem statement and outlines the contributions. Section IV describes the methodology used in this research. Section V presents the experimental evaluation and comparative analyses. Finally, Section VI concludes the paper with key findings and future directions. II. R ELATED W ORKS This section reviews recent literature on phishing detection techniques across text, URL, and image modalities.
URLs, showing that many existing models trained on homepage URLs suffer high false-positive rates when confronted with real login pages. TF-IDF N-gram features achieved 96.5% accuracy with logistic regression, while CNN models were able to demonstrate competitive performance [16]. In total, these findings point towards that ensemble and lexical-feature-based methods provide strong accuracy, but there are certain challenges in addressing real-case login scenarios and tackling evolving obfuscation. This is actually a real motivating factor for the integration of adaptive and deep learning solutions. C. Image-based Phishing Detection
A. Text-based Phishing Detection
Recent attackers target more in the form of visual attacks. Traditional phishing detection using text has primarily Image-related techniques have gained more interest. Attackers relied on linguistic and semantic features that were extracted are starting to incorporate the use of logos and graphics in from email bodies and subjects. Natural language processing, fooling and tricking the user. The major reason for this is that combined with feature engineering, enables classifiers to it can easily evade text and URL-filtering mechanisms. Surveys distinguish between legitimate and phishing messages with in phishing-related detection emphasise the fact that computer high accuracy. Some of the common techniques used were vision has crossed the boundaries of being an essential part tokenisation, Term Frequency-Inverse Document Frequency of modern technology systems, with the use of CNNs in (TF-IDF) weighting, lemmatisation, Chi-square, principal phishing screenshot datasets being seen across the board component analysis, and latent semantic analysis to generate with classification accuracies of over 95% in phishing-related compact but discriminative feature sets [5]. Classifiers such datasets [8, 9]. They incorporate the detection of logos, designs, as XGBoost, Random Forest, and Support Vector Machines and colour patterns. The capture of the above is very helpful (SVMs) have demonstrated strong performance on benchmark in the aspect of being very hard to interpret by the text-related datasets when trained on engineered text features [5]. Latest classifiers in terms of character replacement and obfuscation reviews have highlighted that TF-IDF, n-grams, and word by scripts. However, despite the improvements found in the embeddings are still dominant techniques in text-based phishing models, they are seriously reliant on the dataset they are trained detection, along with new CNNs and Recurrent Neural on. They perform poorly if the training datasets lack adequate Networks (RNNs) gaining adoption recently [6]. Verma et al. information in terms of new brand names, or rather, creatively further showed that preprocessing techniques such as stemming, changed logos and banners. This dependency raises questions stopword removal, and n-gram extraction improve detection, about scalability, as obtaining large, labelled image datasets though false positives persist due to overlaps with legitimate that reflect the evolving phishing landscape remains a persistent marketing emails [7]. Despite these advances, text-based challenge. These limitations indicate that although CNN-based models remain insufficient against modern attacks that embed detection has satisfactory baseline performance, it is not capable logos, images, or obfuscated scripts, underscoring the need for enough to address zero-day threats and rapidly evolving visual multi-modal defences. deception techniques and dynamic strategies [11]. Phishpedia, proposed by Lin et al., is a hybrid approach B. URL-based Phishing Detection efficient in visually scanning, analysing, and detecting phishing URL-based phishing detection is one of the widely studied websites through the comparison and cross-validation of logos areas. It is studied using lexical, structural, and domain-related and login form boxes in the screenshots of webpages [10]. The attributes. One of the recent studies applied a dataset of proposed method combines the use of a Faster-Region-based over 11,000 URLs with 33 extracted features, such as special CNN (RCNN) model for logo and UI component detection and characters, domain age, and protocol type. These are used the Siamese network for brand validation and cross-validation to evaluate classifiers such as k-Nearest Neighbour, Logistic for the accurate differentiation between legitimate and Regression, Decision Tree, Random Forest, Gradient Boosting, phishing/suspect domains. Conventional techniques for phishing Naı̈ve Bayes, and Support Vector Classifier. Some of the website detection necessitate the need for large amounts combinations achieved strong results, with Random Forest of labelled phishing data, while Phishpedia employed the reaching 96.77% accuracy and hybrid exceeding 98% accuracy utility of transfer learning through logos of brands, ensuring [14]. Also, PhishDef demonstrated that carefully engineered flexibility in detecting new targets without the need for further lexical features alone can handle full feature sets, achieving training on phishing data. Experimental analysis on more than 96–98% accuracy. Algorithms such as AROW, while remaining 30,000 phishing and legitimate webpages demonstrated the resilient to noisy training data and obfuscation techniques, effectiveness of Phishpedia in outperforming other conventional are thus suitable for real-time browser-side detection [15]. techniques such as EMD, PhishZoo, and LogoSENSE in Complementing these approaches, Sánchez-Paniagua et al. identifying phishing webpages at a 99.2% accuracy in an introduced the PILU-90K dataset of 90,000 login and phishing absolute runtime of 0.19 seconds per webpage. It was also
TABLE I: Summary of Image-Based Phishing Detection Approaches and Datasets Method / Model
Accuracy
Dataset (Name & Size)
Key Observation(s)
CNN-based models [8] CNN + LSTM hybrid [9] Phishpedia (Faster-RCNN + Siamese) [10] CNN visual similarity [11]
> 95% > 97% 99.2%
Balanced URL dataset (5,000 phishing + 5,000 legitimate) Public phishing benchmark datasets (size not explicitly reported) Phishpedia benchmark (∼30K phishing + ∼30K benign webpages)
∼ 96%
Screenshot-based phishing datasets (e.g., Phish-IRIS ∼2,852 images)
Vision GNN [12]
> 97%
Phish-IRIS (compact descriptors) [13]
∼ 92%
Vision-based phishing benchmark datasets (dataset size not clearly specified) Phish-IRIS dataset (1,313 training + 1,539 testing images)
Dataset dependency issues High recall; computationally expensive Detected 1700+ zero-day sites; runtime of 0.19 s/page Strong logo recognition; limited dataset diversity Captures relational structure among webpage elements Lightweight approach; less accurate than CNNs
capable of resisting malicious attacks. This clearly testified to III. P ROBLEM S TATEMENT the efficacy of Phishpedia in realistic applications. This also Most of the recent image-based phishing detection methods, identified more than 1,700 phishing webpages in a short span like CNN hybrids, Phishpedia, and Vision GNN, have suggested of 30 days, astonishingly including some as zero-day attacks strong accuracy but most of them are lacking decision-threshold unidentified by other networks. analysis. These models might do well on balanced datasets Recently, this area has moved beyond traditional CNNs and whereas a higher chance of failure is possible in real cases started looking at how webpages are structured. Lindamulage where benign samples are much higher. Alshingiti et al. [9] et al. used Vision GNNs to model relationships between visual reported F1-scores in their research, but many papers still elements, reaching above 97% accuracy on benchmark datasets do not show how performance varies at different thresholds [12]. Saeed also has proposed another visual similarity approach mention of the decision threshold has made the results vague, that uses CNN-based embeddings to detect brand duplication, harder to reproduce and harder to understand the progress of impersonation, with results around 96% [11]. Earlier methods the models. The next issue that comes to the surface is that and models, such as Phish-iris focused on lightweight and the use of accuracy is really heavy as the main metric. When surficial visual descriptors for phishing detection, offering there are imbalanced datasets, accuracy can be misleading and faster, easier, and simpler solutions, though with lower accuracy give more ambiguous information. The model can predict most (about 92%) compared to CNN-based models [13]. These samples as benign and still get high accuracy. Precision, recall, studies demonstrated that image-based methods generally and F1-score are more meaningful, but are not reported clearly. outperform traditional techniques; however, some issues remain. Image-based models like CNNs and GNNs have a heavy Dataset diversity, adversarial changes, and scalability are still dependency on training data quality. If the dataset does not challenges, so more research is needed. include recent brands or altered logos, these models struggle Table I summarises the accuracy, datasets used and key to generalise and detect properly. Tricks such as Zero-day, observations of the various studies. This highlights the changing colours, layouts, or hiding elements to evade detection differences and performances across recent image-based are now the favourite of attackers. All of these issues suggest phishing detection models and approaches. It is observed a need for phishing detection systems that use image analysis that individual study shows promising results, their datasets, and also include threshold-based evaluation. Models should efficiency and limitations vary. report precision, recall, and F1-scores and also be able to handle class imbalance and adversarial changes. Addressing D. Metrics Observations these gaps is key to building systems that work reliably in Accuracy is still the most common metric in phishing real-world conditions. detection research. Accuracy alone is not reliable for imbalanced datasets, where benign samples are much higher. A. Challenges in the Current State of Phishing Detection Websites now have visual imitation rather than simple text Precision, recall, and F1-score give a better picture as they show how the model handles and analyses false positives and tricks, which is why phishing attacks are becoming harder to false negatives. When reviewed, many papers do not report detect because many attackers copy logos, colour schemes, these metrics in a consistent way. For example, Alshingiti et al. and layouts, which help them to bypass systems that only [9] reported an F1-score of 0.96 for their CNN–LSTM model. rely on analysing URLs or text content. Some phishing pages Vision GNN studies showed high precision and recall, but did have embedded text inside images, which makes it difficult for not always include F1-scores. Approaches such as Phishpedia, string-matching or DOM-based methods to identify potential content. Deep learning models can help, but they depend heavily primarily focused on accuracy and runtime. From the above discussion, it is clear why precision, recall, on the quality and variety of the training data. Non-inclusion and F1-score together are more helpful than a singular metric, of newly emerging brands or altered logos can make these suggesting the efficiency and capability of the model. The next models fail to generalise to real-world phishing attempts. Another major challenge is that many detection models and section articulates the problems we are addressing in this paper. systems do not incorporate new attack patterns. Phishing kits
and techniques are very dynamic. The mindset of attackers is changing every hour. To avoid detection, Attackers are using zero-day visual changes, such as modified layouts or redesigned icons. Without ongoing updates or feedback loops, model performance and detection capability drop over time. An imbalance between datasets is also a persistent issue, as there are more legitimate webpages outnumbering phishing ones. These are the reasons why accuracy is less reliable, and there is a need for metrics such as precision, recall, and F1-score. In sum, these challenges clearly suggest why phishing detection systems must be able to handle visual variation and changes, evolving attack styles, and imbalanced datasets to remain effective in real-world deployments. B. Our Contributions This research will provide significant contributions to image-based phishing detection. Firstly, it has provided an end-to-end framework that analyses, classifies phishing and legitimate pages using webpage screenshots. As the framework consists of different yet simple workflows, which consist of stages like data collection, preprocessing, model setup, training, and evaluation. As a single system, the processes are easy to reproduce and follow. Major focus includes a threshold-aware evaluation, which was a missing part in most of the recent models and research. Secondly, the study compares two well-known vision models, ConvNeXt-Tiny and ViT-Base, under the same criteria and conditions. This parallel comparison is to fetch the strengths and weaknesses of each model, particularly how they learn and analyse visual patterns such as logos, layout structure, and colour schemes. The analysis highlights where convolutional models still perform better and where transformers may struggle without larger datasets. Thirdly, the work uses transfer learning with ImageNet-pretrained weights. This streamlines the training, making it faster and handling limited phishing data more effectively. The results show that this approach improves stability and performance during training, particularly for the convolutional model. Another major contribution is the use of threshold optimisation for the model, which tests different decision thresholds to find the best operating and performing point for real-world use. This step improves the balance between precision and recall and gives a clearer picture of how the models behave once deployed. In summary, the study provides practical guidance and step-by-step explanations on building and evaluating image-based phishing detection systems. It aims to support future work by releasing the curated dataset so that other researchers can compare methods and explore new solutions. IV. M ETHODOLOGY This section explains the steps involved in the study Fig. 1 is the visual representation of the steps involved.
A. Stage 1: Data Collection As data constitutes the core component of the system, it is always essential to maintain a clean and updated dataset. However, public datasets were limited in terms of scale and recency. To address this limitation, a Python-based data collection pipeline was developed using Playwright (a software testing and browser automation framework) to automatically capture Screenshots of webpages. For this, the latest URL feeds from OpenPhish were used as a source for the script [17]. This process resulted in 18,432 screenshots. In addition, 10,444 mixed screenshots were used from the PhishIRIS dataset [18] to increase data diversity. In total, 28,876 labelled webpage screenshots were in the dataset, which gave a balanced and up-to-date dataset for training, validating and testing the system. B. Stage 2: Data Preprocessing and Feature Engineering In this stage, all the gathered screenshots were resized to 224*224 pixels to match the input size of ImageNet-based models. Using a hash algorithm, the dataset was cleaned by detecting duplicate and corrupted images, which were removed from the final datasets. To make the dataset even diverse, data augmentation steps like random horizontal flips, brightness changes (±15%), Gaussian blur, and random cropping were performed. Finally, the dataset was split into 80% training, 10% validation, and 10% testing, keeping both classes balanced. C. Stage 3: Model Selection and Initialisation For this study, we have chosen two models: ConvNext-Tiny and Vision Transformer (ViT-Base) on their basis of performance, scalability, and deployment needs [10]. ConvNext-Tiny is a modern CNN-based model that strongly focuses on visual recognition, while ViT-Base learns broader image patterns using self-attention. ConvNeXt-Tiny updates standard CNN ideas to reach competitive accuracy while staying efficient [3]. Its local feature learning strength makes it suitable for detecting the visual details such as Brand-imitating logos, visually similar layouts, and embedded textual information, which are often used in phishing websites. On the other hand, ViT-based architectures can analyse the image patches while capturing long-range contextual dependencies, enabling effective modelling of global webpage structure [4]. This ability can be leveraged for phishing webpage detection, as global layout and structural cues help distinguish phishing pages from legitimate websites. Previous research indicates that transformer architectures can perform strongly in visual phishing detection, justifying the use of ViT-Base as a comparison model [10]. Evaluating both models within the same experimental setup allows this study to examine the trade-offs between local feature extraction and global context modelling in detecting phishing webpages. D. Stage 4: Model Training and Validation Using labelled screenshots, models were trained under a supervised learning framework to classify phishing and legitimate. Binary cross-entropy was employed as the loss function to quantify classification error, while the Adam
Fig. 1: Methodology applied during the study.
optimiser was used to update model parameters during training. Hyperparameters, including the learning rate, batch size and dropout, were tuned during the training phase. For the classification of these binary images, the following choice configurations follow the standard practice [9]. Validation was performed on each epoch while observing model overfitting and generalisation on unseen data. Metrics score, including precision, recall, and F1-score, were tracked and analysed to see how the model is working. A post-training threshold optimisation step evaluated multiple classification
thresholds to achieve the best F1-score while reducing false positives, consistent with earlier deployment-oriented study [10]. E. Stage 5: Performance Evaluation and Analysis The performance of model was evaluated using precision, recall, and F1-score, as these metrics provide a clearer picture than accuracy in phishing detection. Accuracy can be misleading under class imbalance, whereas precision and recall highlight false-positive and false-negative behaviour. The F1-score offer a balanced view of both aspects. This matches
ConvNeXt-Tiny 1.00 0.99
Score
earlier phishing research that relies more on class-sensitive metrics than overall accuracy. Observing these metrics enabled a clear understanding of the model’s response to phishing and legitimate websites [19]. A threshold optimisation step has also been used to make the experiment more suitable for implementation. Rather than using a standard threshold, various values for the threshold were tested to observe the changes in the score. This made it easier to assess at which stage the classifier maintained a high level of recall and lowered the number of false positives. While the performance of ConvNeXt-Tiny was more stable under varied threshold levels, the ViT-Base model needed more optimisation. These aspects have provided a better understanding of which classifier is more controllable and how they will work in a real-world scenario [20].
0.98 0.97 Precision Recall F1 Max F1 Threshold = 0.8
0.96 0.95
0.1
0.2
0.3
0.4
0.5
Threshold
0.6
0.7
0.8
0.9
Fig. 2: Threshold analysis of ConvNeXt-Tiny.
V. R ESULTS AND A NALYSIS In this section, the outcome obtained by utilising the proposed methodology in Section IV will be explained. After the dataset is constructed and the preprocessing is done, ConvNeXt-Tiny and ViT-Base were trained by using the transfer learning method initialised by ImageNet. The trained models would then make use of the threshold-aware analysis. The outcome focuses on precision, recall, and F1-score. Table II and Fig. 2 illustrate precision, recall, and F1-score against the various threshold values for the ConvNeXt-Tiny model. From the results, it can be seen that precision values increase when the threshold values increase. Therefore, the more accurate the model is in making predictions without having false positives. The value of the recall indicator decreases because the model becomes less vigilant about phishing sites at higher threshold values. The F1-score obtains its maximum value when the threshold value is 0.8. At this point, a balance between identifying phishing attempts and sending undue warnings can be obtained. The discovery of this threshold value becomes significant since it can make the model more accurate when it is put into practice. TABLE II: Threshold-based performance of ConvNeXt-Tiny Threshold 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Precision (%) 0.965 0.973 0.979 0.985 0.989 0.992 0.995 0.997 0.999
Recall (%) 1.000 0.997 0.995 0.993 0.991 0.989 0.986 0.984 0.973
F1-score (%) 0.982 0.985 0.987 0.989 0.990 0.991 0.991 0.992 0.986
Table III and Fig. 3 illustrate precision, recall, and F1-score against the various threshold values for the ViT-Base model. From the results, it is observed that the ViT-Base performs well overall, but its best F1-score is lower compared to ConvNeXt-Tiny. This means the balance between recall and precision is not as strong at the thresholds that matter most for deployment. As a result, ConvNeXt-Tiny offers a more
TABLE III: Threshold-based performance of ViT-Base Threshold 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Precision (%) 0.595 0.635 0.675 0.715 0.755 0.795 0.835 0.920 0.985
Recall (%) 1.000 0.998 0.996 0.993 0.988 0.975 0.950 0.880 0.665
F1-score (%) 0.746 0.776 0.804 0.832 0.857 0.877 0.889 0.900 0.860
stable and reliable operating point for real phishing-detection scenarios. Along with detection performance, we also looked at how efficient each model is to run, since this affects how well it can be deployed in real systems. ConvNeXt-Tiny needs less computation and runs with a lower inference cost because it has a smaller parameter size and uses a simpler convolutional design. ViT-Base benefits from its ability to capture global context through self-attention, but this also makes it heavier and more expensive to run. When comparing both models using F1-score as the main measure, ConvNeXt-Tiny offers a better mix of strong detection performance and lower computational load, which makes it more suitable for real-world phishing detection setups. Table IV summarises the results (precision, recall, and F1-score) at the optimised threshold of 0.8. ConvNeXt-Tiny came out on top for precision, recall, and F1-score. This means it detects more phishing pages while keeping false positives lower. Overall, it provides a stronger balance between detection accuracy and false-alarm control compared to ViT-Base. TABLE IV: Performance at optimal threshold. Metric
ConvNeXt-Tiny
ViT-Base
Precision Recall F1-score
0.997 0.984 0.992
0.920 0.880 0.900
The results clearly show that ConvNeXt-Tiny performs more consistently at the optimal threshold of 0.8. The F1-score of
Vision Transformer (ViT-Base)
VI. C ONCLUSIONS AND F UTURE D IRECTIONS
1.0
Score
0.9 0.8 0.7 Precision Recall F1 Max F1 Threshold = 0.8
0.6 0.5
0.1
0.2
0.3
0.4
0.5
Threshold
0.6
0.7
0.8
0.9
Fig. 3: Threshold analysis of ViT-Base.
0.992 reflects a strong balance between precision and recall, meaning the model is able to detect phishing pages effectively while keeping false alarms very low. In practical terms, this is important because blocking legitimate websites can disrupt users, while missing phishing pages poses security risks. The reported precision (0.997) and recall (0.984) indicate that the model manages this balance well. Although ViT-Base achieves reasonable performance, its lower recall (0.880) suggests that it misses a higher number of phishing instances compared to ConvNeXt-Tiny. This difference may be related to the architectural characteristics of transformer-based models, which rely more on global context and may require larger datasets to reach their full potential. Another important observation is the stability of ConvNeXt-Tiny across different threshold values. The changes in precision and recall are gradual rather than abrupt, indicating that the model is not overly sensitive to small shifts in the decision boundary. This behaviour is desirable in deployment scenarios, where operating conditions may vary. From a practical standpoint, threshold tuning plays a central role in how the system would behave once deployed. In environments where security is the highest priority, the threshold could be adjusted to favour recall and minimise missed phishing attempts. On the other hand, in user-facing systems where excessive blocking is problematic, the threshold could be set to favour precision. The threshold-aware analysis presented in this study allows such flexibility, which strengthens the real-world relevance of the framework. The primary objective of this study was to develop a deployment-oriented framework for image-based phishing detection that goes beyond accuracy-based evaluation. The results presented directly support this objective. By analysing model performance across multiple decision thresholds, we demonstrated that ConvNeXt-Tiny not only achieves high classification performance but also maintains stability under varying operating conditions. This confirms that threshold-aware evaluation provides a more realistic assessment of model behaviour compared to single-metric reporting.
This paper provided an end-to-end system of image phishing detection using screenshots of webpages. The system addressed the shortcomings associated with text and URL-based phishing detection on visually deceptive attacks. The two models, ConvNeXt-Tiny and ViT-Base, were compared under similar training conditions and using threshold-aware metrics. The two models demonstrated that ConvNeXt-Tiny was more accurate with the highest F1-score at the optimal threshold. Its strength is based on its convolutional architecture that captures local information such as logo-based, layout-based, and text/image-based features predominantly found on phishing websites. It consumes lower computational power, which makes it efficient to deploy. ViT-Base, on the other hand, benefits from global context modelling but showed more sensitivity to threshold changes and lower recall, which reduced its F1-score. This suggests that CNN-based models still work very well for visual phishing detection, especially in situations where both accuracy and efficiency matter. We also plan to release the webpage screenshot dataset that was used in this research so that others can reproduce the results of the study and build on these results. Sharing the data with the public will allow for the development of benchmarks and research to advance in this domain. We also intend to look at alternative future research as well, which may merge CNNs and transformers or create additional phishing simulation datasets with new phishing examples and alternative web styles. These ideas could help enhance the robustness and overall performance of systems that depend on images in the detection of phishing attempts. R EFERENCES [1] T. Wangchuk and T. Gonsalves, “Multimodal phishing detection on social networking sites: A systematic review,” IEEE Access, 2025. [2] E. Kritika, “A comprehensive literature review on phishing url detection using deep learning techniques,” Journal of Cyber Security Technology, vol. 9, no. 4, pp. 315–343, 2025. [3] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11 976–11 986. [4] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations. [5] E. S. Gualberto, R. T. De Sousa, T. P. D. B. Vieira, J. P. C. L. Da Costa, and C. G. Duque, “The answer is in the text: Multi-stage methods for phishing detection based on feature engineering,” IEEE Access, vol. 8, pp. 223 529–223 547, 2020. [6] S. Salloum, T. Gaber, S. Vadera, and K. Shaalan, “A systematic literature review on phishing email detection using natural language processing techniques,” IEEE Access, vol. 10, pp. 65 703–65 727, 2022. [7] P. Verma, A. Goyal, and Y. Gigras, “Email phishing: Text classification using natural language processing,” Computer Science and Information Technologies, vol. 1, no. 1, pp. 1–12, 2020. [8] R. Zieni, L. Massari, and M. C. Calzarossa, “Phishing or not phishing? a survey on the detection of phishing websites,” IEEE Access, vol. 11, pp. 18 499–18 519, 2023. [9] Z. Alshingiti, R. Alaqel, J. Al-Muhtadi, Q. E. U. Haq, K. Saleem, and M. H. Faheem, “A deep learning-based phishing detection system using CNN, LSTM, and LSTM-CNN,” Electronics, vol. 12, no. 1, p. 232, 2023.
[10] Y. Lin, R. Liu, D. M. Divakaran, J. Y. Ng, Q. Z. Chan, Y. Lu, Y. Si, F. Zhang, and J. S. Dong, “Phishpedia: A hybrid deep learning based approach to visually identify phishing webpages,” in 30th USENIX Security Symposium, 2021, pp. 3793–3810. [11] U. Saeed, “Visual similarity-based phishing detection using deep learning,” Journal of Electronic Imaging, vol. 31, no. 5, pp. 051 607–051 607, 2022. [12] J. Lindamulage, L. MandiraPabasari, S. Yapa, I. Perera, and J. Krishara, “Vision gnn based phishing website detection,” in 2023 International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES). IEEE, 2023, pp. 1–7. [13] F. C. Dalgic, A. S. Bozkir, and M. Aydos, “Phish-iris: A new approach for vision based brand prediction of phishing web pages via compact visual descriptors,” in 2018 2nd international symposium on multidisciplinary studies and innovative technologies (ISMSIT). IEEE, 2018, pp. 1–8. [14] A. Karim, M. Shahroz, K. Mustofa, S. B. Belhaouari, and S. R. K. Joga, “Phishing detection system through hybrid machine learning based on url,” IEEE Access, vol. 11, pp. 36 805–36 822, 2023. [15] A. Le, A. Markopoulou, and M. Faloutsos, “Phishdef: Url names say it all,” in 2011 Proceedings IEEE INFOCOM. IEEE, 2011, pp. 191–195. [16] M. Sánchez-Paniagua, E. F. Fernández, E. Alegre, W. Al-Nabki, and V. González-Castro, “Phishing url detection: A real-case scenario through login urls,” IEEE Access, vol. 10, pp. 42 949–42 960, 2022. [17] OpenPhish, “Openphish database: Continuously updated archive of phishing urls,” https://openphish.com/phishing database.html, 2025, accessed: 2025-11-04. [18] S. Shahane, “Phish-iris dataset: A small scale multi-class phishing web page screenshots dataset,” https://www.kaggle.com/datasets/ saurabhshahane/phishiris, 2025, accessed: 2025-11-04. [19] D. Powers, “Evaluation: From precision, recall and f-measure to roc, informedness, markedness & correlation,” Journal of Machine Learning Technologies, vol. 2, no. 1, pp. 37–63, 2011. [20] T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,” PloS one, vol. 10, no. 3, p. e0118432, 2015.