FaceScanPaliGemma multi-agent vision language models for facial attribute recognition.

Skip to main content An official website of the United States government Here's how you know Here's how you know Official websites use .gov A .gov website belongs to an official government organization in the United States. Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites. Search Log in Dashboard Publications Account settings Log out Search… Search NCBI Primary site navigation Search Logged in as: Dashboard Publications Account settings Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide PERMALINK Copy As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice Sci Rep . 2026 Feb 23;16:10246. doi: 10.1038/s41598-026-39584-3 Search in PMC Search in PubMed View in NLM Catalog Add to search FaceScanPaliGemma multi-agent vision language models for facial attribute recognition Nouar AlDahoul Nouar AlDahoul 1 Computer Science Department, New York University Abu Dhabi, Abu Dhabi, UAE Find articles by Nouar AlDahoul 1 , Myles Joshua Toledo Tan Myles Joshua Toledo Tan 2 Department of Electrical and Computer Engineering, Herbert Wertheim College of Engineering, University of Florida, Florida, USA Find articles by Myles Joshua Toledo Tan 2 , Harishwar Reddy Kasireddy Harishwar Reddy Kasireddy 2 Department of Electrical and Computer Engineering, Herbert Wertheim College of Engineering, University of Florida, Florida, USA Find articles by Harishwar Reddy Kasireddy 2 , Yasir Zaki Yasir Zaki 1 Computer Science Department, New York University Abu Dhabi, Abu Dhabi, UAE Find articles by Yasir Zaki 1, ✉ Author information Article notes Copyright and License information 1 Computer Science Department, New York University Abu Dhabi, Abu Dhabi, UAE 2 Department of Electrical and Computer Engineering, Herbert Wertheim College of Engineering, University of Florida, Florida, USA ✉ Corresponding author. Received 2025 Aug 12; Accepted 2026 Feb 5; Collection date 2026. © The Author(s) 2026 Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ . PMC Copyright notice PMCID: PMC13031296  PMID: 41730945 Abstract Technologies for recognizing facial attributes such as race, gender, age, and emotion from images of human faces have several applications, including personalized advertising, sentiment analysis, and the study of demographic trends and social behaviors. Analyzing face images and facial expressions presents several challenges due to the complexity of human facial attributes and the diversity in representation. While numerous attempts have been made to improve facial attribute classification performance, there remains a strong demand for enhanced accuracy. In this paper, we propose “FaceScanPaliGemma,” a multi-agent vision language model (VLM) system consisting of four fine-tuned Google PaliGemma models, each specialized for a specific facial attribute classification. To evaluate the proposed solution, we used the public “FairFace” and “AffectNet” datasets. The results show high accuracy, reaching up to 81.1%, 95.8%, 80.0%, and 59.4% for race, gender, age group, and emotion classification, respectively, outperforming other VLMs such as OpenAI GPT, Google Gemini, LLaVA, and Google PaliGemma under zero-shot evaluation. Keywords: FaceScanPaliGemma, Facial attribute recognition, Multi-agent, Vision language models Subject terms: Mathematics and computing, Psychology, Psychology Introduction Technologies for recognizing attributes such as race, gender, age, and emotion have various applications, including improved surveillance and personalized advertising content. Face attribute recognition is also utilized in demographic surveys conducted in marketing or social science research, with the goal of understanding human social behaviors and their connections to individuals’ demographic backgrounds 1 . Accurately identifying multiple attributes in an image presents several challenges due to the complexity of human attributes and diversity in representation. Human attributes like race, gender, age, and emotion are not always well-defined and can be subjective 2 . Attributes like age and emotion exhibit significant intra-class variability, meaning that different individuals within the same category (e.g., the same age group or emotion class) can have vastly different visual appearances 3 . In the area of image processing, artificial intelligence (AI), particularly ML, has made remarkable strides, transforming how computers understand and analyze visual data 4 . Generative models represent a revolutionary step forward in artificial intelligence, particularly in the areas of natural language processing, vision, and generation. These models are designed to understand and generate human-like content, ushering in a new era of AI capabilities and applications. While large language models (LLMs) are celebrated for their near-human performance and broad conversational skills, the growing adoption of agentic AI shifts focus toward systems that repeatedly perform narrow, specialized tasks 5 . In this context, small language models (SLMs) often prove more efficient, better aligned with task requirements, and far more cost-effective. Given their current capabilities, the design of agentic architectures, and deployment economics, SLMs are well-positioned to become the foundation of many agentic AI applications. VLMs are advanced multimodal AI systems that combine visual encoders with LLMs to jointly interpret and reason over visual data—such as images and videos—and textual information within a unified framework 6 . Traditionally, these systems have been implemented as single, monolithic models. However, emerging research highlights the benefits of multi-agent designs 7 , in which multiple specialized agents work together, offering improvements in reasoning capabilities, robustness, and interpretability for complex visual tasks. The existing solutions of VLMs in various applications indicate their potential capability to serve as alternative solutions to traditional and CNN-based methods. Direct applications of multi-agent VLMs to facial-attribute recognition remain underexplored. The aim behind this work is to develop a multi-agent VLM system that leverages the capabilities of VLMs (advanced language processing and contextual analysis) to identify multiple human facial attributes from given images. The creation of this system takes into account incorporating ethical AI practices, concentrating on promoting diversity and inclusion, and reducing biases through utilizing diverse and representative datasets. In addition, we have open-sourced the models and created transparent documentation, such as model cards, including code, prompts, training, evaluation data, training procedures, governance, and responsible use instructions. Linguistic tokens represent abstract, high-level semantic concepts (e.g., emotion, gender, ethnicity), while facial attributes are encoded through fine-grained, localized visual patterns such as skin texture, color gradients, and geometric structure. This semantic granularity mismatch hinders the direct transfer of linguistic context to visual feature extraction in vision–language models. Fine-tuning mitigates this issue by adapting joint embedding spaces and attention mechanisms, allowing abstract language tokens to attend to and supervise relevant low-level visual features during representation learning. Our contributions can be summarized as follows: We formulated a facial attribute recognition task as a visual question answering task and evaluated various general VLMs such as GPT-4o and Gemini Flash 1.5. We fine-tuned four small language models, specifically PaliGemma, using facial attribute recognition datasets such as FairFace and AffectNet to perform four tasks: race, gender, age group, and emotion classification. We evaluated the proposed “FaceScanPaliGemma” multi-agent VLM system with complex scenarios and prompts. As such, we compile multiple visual datasets to answer the following research questions: RQ1 : Can general VLMs overcome limitations available in the existing facial attribute recognition methods? RQ2 : Can small VLMs, fine-tuned specifically on facial attribute recognition datasets, outperform multi-tasking large language models? RQ3 : How can a multi-agent system such as “FaceScanPaliGemma” handle a complex user’s query efficiently? Related work Challenges in facial attribute datasets Bias means unfair preference or prejudice toward certain groups, while fairness means treating everyone equally. Bias can come from two main sources: the data used and the model itself 8 . Data bias happens when datasets have problems like wrong labels, unbalanced classes, or too few samples from certain groups. Model bias comes from how the model is designed, how it is trained, or when it learns from biased data. Bias can harm performance for underrepresented groups, make results less reliable, and reduce fairness 8 . Several bias mitigation strategies have been proposed to address these issues. First, preprocessing methods, which modify the data before training, such as mining latent soft labels 9 and landmark detection 10 . Second, in-processing methods, which reduce bias during model training, such as context debiasing 11 , semi-supervised learning 12 . Third, post-processing methods, which mitigate bias after the model has been trained, such as generating synthetic faces 13 . In previous works, they identified a significant bias in public face image datasets, which predominantly feature Caucasian faces while considerably under-representing other racial groups, such as Latinx_Hispanic 1 . To address this issue of racial imbalance, they compiled a dataset, namely FairFace, consisting of 108,501 face images that is balanced across different races 1 . The advancements in gender, race, age, and emotion classification applications require datasets of diverse facial images that can address challenges such as imbalanced samples 14 , pose variations, and varying lighting conditions. Recently, Convolutional Neural Networks (CNNs), especially the FaceNet model 15 , have shown robustness in handling unbalanced data distributions 16 . The FairFace dataset has been used to train a ResNet-34 17 model to evaluate the classification performance for gender, race, and age. Similarly, recent works 18 used the same dataset with different models to improve the accuracy of the classification. They utilized VGGFace ResNet-50 CNN (VGGFace ResNet-50 CNN) 19 to extract the embedding vector from the face images in the FairFace dataset 18 . They added a support vector machine (SVM) classifier after removing the top layers. Additionally, they explored numerous models, including FaceNet+SVM 15 , tuned EfficientNet-B7 20 , and large vision transformer 21 for race and gender classification 18 . Furthermore, CLIP’s zero-shot classifier demonstrated their performance for race and gender classification using the FairFace dataset 22 . Existing annotated databases of facial expressions in the wild are small and include clean and high-quality posed facial expressions. However, posed expressions may not accurately represent many of the unopposed facial expressions encountered in daily life. As a result, a dataset, namely AffectNet 23 , was proposed to be the largest database of facial expressions with annotations for eight expressions. Several works in literature have utilized the AffectNet dataset to train and evaluate their models for emotion or facial expression recognition 24 – 27 . Deep learning for facial attributes recognition Deep learning models have been utilized for various domains 28 – 30 . Race classification has become significant in applications like surveillance 31 and market advertising 32 . Recently, various deep learning models have been used for race identification 31 , 33 – 36 . Similarly, for gender recognition, deep learning techniques have improved the gender classification accuracy 37 – 42 . Based on these techniques, the effectiveness of CNNs and AlexNet 43 under challenging conditions has been demonstrated, showing potential for applications of tracking and identification 39 . Additionally, age estimation from facial images is crucial for applications like security and social interaction. Several deep learning techniques have been proposed in the literature to improve feature extraction and thus enhance the accuracy of age classification 37 , 40 , 44 , 45 . In this context, hybrid deep learning structures that combine CNNs and Extreme Learning Machines 46 have confirmed the efficacy of integrating multiple learning strategies 45 . Emotion recognition from facial expressions is crucial in human-computer interaction, with deep learning methods addressing challenges such as pose variations, illumination changes, and occlusions 47 , 48 . Based on this, transfer learning techniques using networks like ResNet50 17 and VGG19 49 have proven effective in combining feature extraction and classification for emotion recognition tasks 47 . Multimodal or multitask deep models have shown improvement in classification accuracy, demonstrating the utility of leveraging diverse biometric inputs 40 – 42 , 50 – 52 . The work aims to integrate multiple facial attributes extracted from facial images. This integration is crucial and has become a key area of interest in computer vision applications. CNN has been utilized to integrate gender and emotion 53 . Similarly, visual attention-driven architectures have been employed for gender and ethnicity integration 54 . Additionally, the integration of age and gender prediction has enhanced biometric security and personalized systems. Several studies have highlighted the efficiency of deep learning in enhancing real-time biometric recognition, paving the way for more compact, faster, and accurate systems. A multi-task CNN has been proposed for recognizing gender, age, ethnicity, and emotion 55 . It improved efficiency in processing and memory usage while maintaining good accuracy across multiple tasks, making it suitable for embedded systems 55 and for applications requiring real-time processing with limited computational resources 53 . Previously mentioned deep learning methods have outperformed traditional techniques in classifying race, gender, age, and emotion. However, there is still room for improvement. As such, this paper aims to improve facial attribute classification using VLMs. Emerging role of LLMs in image processing While LLMs are widely known for their expertise in text processing 56 – 59 , recent studies have begun to explore their effectiveness in image generation and recognition 60 – 63 , often through multi-modal learning approaches 64 . Multimodal models like CLIP (contrastive language-image pretraining) 22 leverage the combination of LLMs and CNNs by learning from pairs of images and their textual descriptions. CLIP has achieved good performance on various image recognition benchmarks, including race, gender, and age recognition 22 , thanks to its ability to create a shared representation space for both images and text, which enables robust performance even in zero-shot settings. The use of LLMs, such as GPT 65 and BERT 66 , has extended beyond text-based tasks, delving into areas like image recognition and processing. This broadening of application stems from the models’ capacity to comprehend and generate human-like text, offering a fresh perspective on how images can be interpreted and analyzed 22 . Recent interdisciplinary research has started to investigate the potential of LLMs in image-related tasks such as construction processes, radiology, and medical visual question answering 67 – 69 . For instance, studies have shown that GPT can generate textual descriptions from images, paving the way for innovative approaches to image understanding through natural language 22 . LLMs have also been integrated with traditional vision models in tasks like visual question answering (VQA) 70 , where models are trained to respond to questions based on image content. This highlights the synergy between LLMs and image recognition, requiring a deep understanding of both visual and textual data 70 . Table 1 compares several visual recognition methods, highlighting their limitations that have been addressed by PaliGemma 71 . We utilized PaliGemma in this study because it is better than GPT-4o, as it is open source, uses less computation, runs faster, and can be easily fine-tuned to specific tasks. Compared to CLIP, PaliGemma is better because it combines vision with a language model, which allows it to reason more effectively. In addition, PaliGemma can both understand and generate outputs, while CLIP cannot generate. Table 1. Limitations of visual recognition methods. Method Core idea Key limitations Convolutional Neural Network (CNN) 72 convolutional inductive bias for spatial locality limited global context modeling, poor cross-modal understanding, weak to distribution shifts Vision Transformer (ViT) 21 pure transformer applied to image patches data hungry, unimodal, lacks explicit semantic grounding Contrastive Language Image Pretraining (CLIP) 22 contrastive learning between image-text pairs cannot generate new text, limited reasoning Generative Pre-trained Transformer (GPT-4o) 73 unified multimodal foundation model high compute cost, limited controllability, not specialized for domain-specific structure without fine-tuning Open in a new tab In face anti-spoofing (FAS), researchers have begun to use text information to help models generalize across different domains. For example, FLIP 74 was the first method to apply CLIP to FAS. Later works 75 , 76 , further improved FAS generalization by using vision–language learning. However, these methods mainly focus on cross-domain problems and do not fully exploit textual understanding to improve model performance in unseen scenarios. To overcome the limitations of existing methods, this work introduces InstructFLIP, an instruction-tuned approach that uses the rich annotations in the CelebA-Spoof (CS) dataset 77 . Integrating LLMs such as Google GEMINI 1.5 78 , 79 , GPT-4o 65 , 80 , LLAVA-NEXT 81 – 83 , and PaliGemma 71 into facial recognition tasks can significantly enhance the performance and capabilities of deep learning models. This paper investigates the use of VLMs in addressing challenging computer vision tasks, such as emotion, gender, race, and age group classification. It leverages VLMs’ capabilities in understanding and generation to improve visual recognition. Research motivation Direct applications of multi-agent VLMs to facial-attribute recognition remain underexplored. By coordinating specialized agents, each focused on tasks such as race, gender, age, or emotion, multi-agent VLMs can leverage complementary reasoning and shared context to handle challenging conditions, including blurred or noisy images, variable illumination, and diverse facial poses. This collaborative architecture promises greater robustness and accuracy than monolithic approaches. The existing solutions of VLMs 61 , 62 , 67 – 69 , 84 in various applications indicate, with their sophisticated understanding and generative abilities, their potential capability to serve as complementary, or even alternative, solutions to traditional and CNN-based methods in facial attribute recognition systems. By harnessing the advanced language processing and contextual analysis capabilities of VLMs, researchers have the potential to achieve significant improvements in the accuracy, efficiency, and adaptability of technologies used for recognizing race, gender, age, and emotion. Materials and methods This section describes the datasets used in the experiments conducted to evaluate the ‘FaceScanPaliGemma” performance on facial attribute recognition tasks. Moreover, the section discusses the baseline methods usually used in the literature and highlights our proposed solutions. Dataset overview FairFace dataset To address the issue of racial bias in existing datasets, a face image dataset was created comprising 108,501 images that are racially balanced 1 . The images were categorized into seven racial groups: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latinx_Hispanic. These images were sourced from the YFCC-100M Flickr dataset 85 and annotated according to race, gender, and age groups. The dataset has binary gender classification: male and female, and several age groups: 0-2, 3-9, 10-19, 20-29, 30-39, 40-49, 50-59, 60-69, and 70+. The images have a resolution of 224 x 224 pixels. We combined the age groups of the FairFace dataset into five categories that reflect different social and economic roles: 0-9, 10-19, 20-39, 40-59, and 60+ and found that the dataset is imbalanced in terms of race and age groups that we selected. The numbers of samples for race, gender, and age group for the training and testing sets for each category are presented in Tables 2 to 4 . In this study, the FairFace dataset was employed to evaluate VLMs and, specifically, to assess the classification capabilities of “FaceScanPaliGemma” on race, gender, and age group (Table 3 ). Table 2. Number of samples in the training and testing set in the FairFace dataset per race category. Race Train samples Test samples Black 12,233 1556 East Asian 12,287 1550 Indian 12,319 1516 Latinx_Hispanic 13,367 1623 Middle_East 9216 1209 Southeast Asian 10,795 1415 White 16,527 2085 Total 86,744 10,954 Open in a new tab Table 4. Number of samples in the training and testing set in the FairFace dataset per age category. Age group Train samples Test samples 0-9 12,200 1555 10-19 9103 1181 20-39 44,848 5630 40-59 16,972 2149 60+ 3621 439 Total 86,744 10954 Open in a new tab Table 3. Number of samples in the training and testing set in the FairFace dataset per gender category. Gender Train samples Test samples Male 45,986 5792 Female 40,758 5162 Total 86,744 10954 Open in a new tab Emotion dataset AffectNet is a very challenging and extensive facial expression dataset containing approximately 0.4 million images that have been manually labeled to represent eight different facial expressions: neutral, happy, angry, sad, fear, surprise, disgust, and contempt 23 . The images were gathered from the Internet by conducting searches across three major search engines using 1,250 emotion-related keywords in six different languages. The images have a resolution of 224 x 224 pixels. The AffectNet dataset is again imbalanced in terms of emotion or facial expression. The numbers of samples for the training and testing sets for each emotion category are shown in Table 5 . In this study, the AffectNet dataset was employed to evaluate VLMs in general, with a specific focus on assessing “FaceScanPaliGemma” for emotion classification. Table 5. Number of samples in the training/testing set in the AffectNet dataset per emotion. Emotion Train samples Test samples neutral 74,874 500 happy 134,415 500 sad 25,459 500 surprise 14,090 500 fear 6,378 500 disgust 3,803 500 anger 24,882 500 contempt 3,750 499 Total 287,651 3999 Open in a new tab Method The proposed “FaceScanPaliGemma” for facial attribute recognition is a multi-agent system that contains three Google PaliGemma models fine-tuned on the FairFace dataset and one PaliGemma fine-tuned on the AffectNet dataset. This system is designed to recognize attributes such as race, gender, age, and emotion from images based on a given prompt. In complex scenarios with complex prompts, the system also contains face detection PaliGemma and prompt analysis LLMs such as Llama-3.2-3B 86 . Each PaliGemma agent integrates both language and visual processing, enabling enhanced understanding and generation capabilities. Figure 1 describes the block diagram of the multi-agent VLMs for facial attribute recognition. As shown in Fig. 1 , the user’s query (people image and prompt) is applied to the inputs of the analysis agent, which forwards the image to the face detection agent to extract specific faces based on the prompt. Later, the detected faces are sent back to the analysis agent to forward them to the specialized agents (race, gender, age, or emotion) for further processing. The final outcome is produced by the analysis agent and sent back to the user. Algorithm 1 presents the system algorithm. Fig. 1. Open in a new tab Block diagram of the Multi-Agent VLMs for facial attribute recognition. Each of these fine-tuned models is not simply a fine-tuned classifier functioning in isolation but rather an autonomous module with two key characteristics: (1) the ability to make decisions and (2) the ability to interact with other agents. The agents in FaceScanPaliGemma interact and coordinate to handle complex prompts. They are connected in a sequential workflow, allowing one agent’s output to guide the next and ensuring that responses are coherent and aligned with the user queries. “FaceScanPaliGemma” consists of the following agents: Analysis agent: This is the main agent that has a Llama-3.2-3b 86 as a model. This agent examines the user’s request and converts it to multiple steps. Later, it routes the images or faces to the appropriate specialized agent. If the query refers to physical attributes, actions, or posture, this analysis agent routes the image to the face detection agent to extract corresponding faces. If the query contains a mention of race, gender, age group, or emotion, it routes the face images to race, gender, age, or emotion agents. This agent gives the final outcome based on the prompt. The face detection agent, which is a generic pre-trained PaliGemma, extracts all the face images of people based on specific physical attributes, actions, or posture in the prompt. The race, gender, age, or emotion agents receive a set of face images with text that has a list of categories (races, genders, age groups, or emotions), then they classify each received image into the corresponding category. Later, if they check if the images match one of the categories in the list, they are forwarded to the analysis agent again. Consider the scenario where the user’s query specifically asks about “the emotion of a Middle Eastern individual” in the image with several persons. The analysis agent converts the query into several steps: It sends a message to the face detection agent to detect all faces, and it waits to receive the detected faces. It sends a message to the race agent with text that has ’Middle Eastern’ to classify the faces and send back only ones predicted under the ’Middle Eastern’ race label. It sends a message along with ’Middle Eastern’ faces to the emotion agent to find the emotion of them. Another example is a scenario where the prompt specifically asks about “the race of the boy holding a ball” in the image with several persons: The analysis agent converts the query into several steps: It sends a message to the face detection agent to detect faces of people holding a ball, and it waits to receive the detected faces. It sends a message to the age group agent with text that has ’0-9’ to classify the faces and send back only ones predicted under the ’0-9’ age group label. It sends a message to the gender group agent. The message includes the faces under the ’0-9’ age group and the ’Male’ text to classify these faces and send back only ones predicted under the ’Male’ gender label. It sends a message along with faces that are predicted as both ’Male’ and ’0-9’ to the race agent to find the race of them. The proposed system is designed with a modular architecture that allows additional modules to be seamlessly integrated. In this context, a face anti-spoofing module can be incorporated as a plug-in stage to verify images before they are passed to the analysis agent. The advantage of a multi-agent system over a single unified multi-task model: A recent trend favors small language models (SLMs) that are specialized for specific tasks rather than large, multi-task language models. Typically, a single VLM must contain a large number of parameters to handle multiple tasks such as race, gender, age, and emotion classification, which increases computational costs for both training and inference, as well as inference latency. In contrast, SLMs with fewer parameters can be fine-tuned for a specific task, resulting in improved performance and significantly faster inference. When race, gender, age, and emotion recognition are handled by separate models, they allow bias to be measured, mitigated, and corrected independently for each attribute, preventing cross-task bias transfer and enabling targeted, task-appropriate fairness interventions. Multi-agent system provides modular maintainability by updating or replacing models independently if there is any error without affecting other tasks. Algorithm 1. Open in a new tab MultiAgentFaceAnalysis Google PaliGemma 3b We adapted generative VLM, namely PaliGemma, to a discriminative task (classification). In this work, rather than using PaliGemma for text generation, we employ it as a feature extractor to obtain joint vision–language embeddings. Specifically, given an input image and its associated textual prompt, we extract hidden-state representations from the model. Google’s PaliGemma is an open VLM that extends the PaLI vision-language model series by integrating it with the Gemma family of language models. It was built upon the SigLIP-So400m vision encoder and the Gemma-2B language model. Designed as a versatile and broadly knowledgeable base model, PaliGemma excels in transfer learning 71 . It demonstrates strong performance across a wide range of open-world tasks. The multi-task learning was performed by using task prefixes. The prefix-LM with task prefix and supervision only on the suffix tokens is an effective VLM pre-training objective. Fine-tuning a model for specific tasks is effective when the goal is to solve a particular problem. However, it is often preferable to have a single generalist model with a conversational interface. This is usually accomplished through instruction tuning, which involves fine-tuning on a diverse dataset. PaliGemma was found to be well-suited for this type of transfer 71 . In this work, we fine-tuned four PaliGemma models for the facial attribute recognition task. Experiments and results This section presents the evaluation and comparison results of “FaceScanPaliGemma” with other VLMs on tasks related to age, gender, race, and emotion classification in terms of their accuracy, recall, precision, and F1 score. Additionally, we compared “FaceScanPaliGemma” with existing methods. The comparison was done using various datasets such as FairFace 1 , and AffectNet 23 . We carried out multiple experiments to assess the vision capabilities of various VLMs for facial attribute recognition, with a specific focus on identifying race, gender, age, and emotion. Formulating a facial recognition application as a visual question-answer task allows leveraging the pre-trained VLMs with their capabilities of understanding and processing both the image and associated text. Experimental setup For race, gender, and age group classification, the fine-tuning process was done for each attribute using PaliGemma VLM containing 3b parameters and utilizing the FairFace dataset with a total of 86,744 images. The dataset was divided into a training set (75%) and a validation set (25%). The performance of “FaceScanPaliGemma” was evaluated using the testing data of FairFace with 10,954 images. The fine-tuning process in all previous scenarios used a balanced portion of 3700 images per category selected from the AffectNet dataset, ensuring representation across different races and genders for each class. The performance of fine-tuned PaliGemma was evaluated using AffectNet’s testing data with 3999 images. We fine-tuned each Paligemma model using carefully chosen hyperparameters to ensure stable training and effective convergence. The learning rate was set to 2e-05. We used a train batch size of 2 with gradient accumulation over 8 steps, resulting in a total effective batch size of 16. Optimization was carried out using the Adam optimizer. For scheduling, we applied a linear learning rate scheduler with a warmup of 2 steps, allowing the model to gradually adjust before full training. The fine-tuning process was run for 5 epochs. We used parameter-efficient fine-tuning with LoRA (rank r = 4) under 4-bit quantization (NF4). We open-sourced “FaceScanPaliGemma” on the Hugging Face platform 87 under the following links: https://huggingface.co/NYUAD-ComNets/FaceScanPaliGemma_Race https://huggingface.co/NYUAD-ComNets/FaceScanPaliGemma_Gender https://huggingface.co/NYUAD-ComNets/FaceScanPaliGemma_Age https://huggingface.co/NYUAD-ComNets/FaceScanPaliGemma_Emotion FaceScanPaliGemma for race, gender, age, and emotion classification “FaceScanPaliGemma” aims to highlight the model’s ability to scan human face images and detect race, gender, age group, and emotion. We fine-tuned PaliGemma for each task independently, as our main objective was to develop a system that outperforms existing models. The confusion matrices for “FaceScanPaliGemma” are presented for classification of race in Fig. 2 , gender in Fig. 3 , and age group in Fig. 4 . Fig. 2. Open in a new tab Confusion matrix for race classification using “FaceScanPaliGemma” tested on the FairFace dataset. Fig. 3. Open in a new tab Confusion matrix for gender classification using “FaceScanPaliGemma” tested on the FairFace dataset. Fig. 4. Open in a new tab Confusion matrix for the age group classification using “FaceScanPaliGemma” tested on the FairFace dataset. The results can be seen in the red-colored cells of tables 7 to 9 , and demonstrate the exceptional performance of “FaceScanPaliGemma” for race classification with an accuracy of 81.1% and an F1 score of 79% across the six race classes and an accuracy of 75.7% and an F1 score of 75% for the seven race classes, outperforming not only other VLMs such as GPT-4o but also traditional methods (highlighted by the bold italic values). Table 7. Comparison between FaceScanPaliGemma and other VLMs for 6-race classification. 6-Race VLMs Accuracy % Precision % Recall % F1 Score % GPT-4o 76.4% 75% 73% 74% GPT-4o-mini 75.4% 74% 72% 72% Gemini 1.5 flash 68.9% 69% 63% 61% Llava-Next 7b 64.9% 69% 58% 54% Pre-trained PaliGemma 68.1% 68% 63% 62% FaceScanPaliGemma (proposed) 81.1% 79% 79% 79% Open in a new tab Table 9. Comparison between FaceScanPaliGemma and baseline methods for 7-race classification. 7-Race classification methods Accuracy % Precision % Recall % F1 Score % VGGFace-ResNet-50-SVM 18 72.6% 72% 72% 72% FairFace’s (ResNet34) classifier 1 72% 72% 71% 72% Google’s FaceNet + SVM 18 68.9% 69% 68% 68% CLIP’s zero-shot classifier 22 64.2% 67% 65% 65% GPT-4o 68% 69% 66% 65% FaceScanPaliGemma (proposed) 75.7% 75% 75% 75% Open in a new tab For the gender classification task, the results in the red-colored cells of Table 10 demonstrate the exceptional performance of “FaceScanPaliGemma,” achieving an accuracy of 95.8% and an F1 score of 96%. These results rival GPT-4o and surpass existing methods (highlighted by the bold italic values). Table 10. Comparison between FaceScanPaliGemma and other VLMs and baseline methods for gender classification. Gender classification methods Accuracy % Precision % Recall % F1 score % VGGFace-ResNet-50-SVM 18 94% 94% 94% 94% FairFace’s (ResNet34) classifier 1 94.4% 94% 94% 94% CLIP’s zero-shot classifier 22 94% 94% 94% 94% GPT-4o 95.9% 96% 96% 96% GPT-4o-mini 95.9% 96% 96% 96% Gemini 1.5 flash 94.9% 95% 95% 95% Llava-NeXT 7b 95.3% 95% 95% 95% Pre-trained PaliGemma 93.6% 94% 94% 94% FaceScanPaliGemma (proposed) 95.8% 96% 96% 96% Open in a new tab For the age group classification task, Table 11 , shows the remarkable performance of “FaceScanPaliGemma” with an accuracy of 80% and an F1 score of 74%, outperforming other VLMs such as GPT-4o and exercising methods such as the AWS classifier 88 and the FaiFace ResNet34 classifier 1 (highlighted by the bold italic values). Table 11. Comparison between FaceScanPaliGemma and other VLMs and baseline methods for age group classification. Age group classification methods Accuracy % Precision % Recall % F1 score % AWS classifier 88 71.7% 68% 61% 63% Tuned Vision Transformer 21 76.1% 71% 72% 71% FairFace’s (ResNet34) classifier 1 79% 74% 71% 73% GPT-4o 77.4% 71% 71% 69% GPT-4o-mini 77.2% 70% 75% 72% Gemini 1.5 flash 70.2% 66% 69% 65% LlaVA-NeXT 7b 54.3% 56% 68% 56% Pre-trained PaliGemma 49.8% 54% 55% 41% FaceScanPaliGemma (proposed) 80% 75% 74% 74% Open in a new tab Supplementary Tables 1 – 4 in the supplementary material show performance metrics by race, gender, age group, and emotion. They report standard fairness metrics: disaggregated performance gaps (difference between maximum F1-score and minimum F1-score) and equal opportunity difference (difference between maximum recall and minimum recall) between demographic groups. As shown in Supplementary Table 2 , gender-based disparities are negligible, with near-identical recall, precision, and F1 scores across groups. On the contrary, age-based performance disparities are observed in Supplementary Table 3 , and the confusion matrix in Fig. 4 reveals that most errors occur between adjacent age groups. This pattern suggests that residual disparities are largely driven by visual ambiguity at age boundaries, rather than systematic bias against a particular group, even if the age dataset is imbalanced. Similarly, for race and emotion prediction in Supplementary Table 1 and Supplementary Table 4 , we observe substantial disaggregated performance gaps and an equal opportunity difference. Lower recall is concentrated in categories characterized by high intra-group visual diversity, such as Latinx, and low-intensity expressions, such as neutral. These disparities appear to stem primarily from the inherent ambiguity of appearance-based attributes because the datasets are balanced for race and emotion. For emotion classification, we fine-tuned PaliGemma under various fine-tuning scenarios: The model was fine-tuned by freezing the parameters of the vision tower and multi-modal projector using an image resolution of 224 x 224 pixels. The model was fine-tuned by freezing the parameters of the vision tower and multi-modal projector using an image resolution of 448 x 448 pixels. The model was fine-tuned by updating parameters of the vision tower and multi-modal projector using an image resolution of 224 x 224 pixels. We selected the best scenario that gave the highest accuracy and F1-score in emotion classification to fine-tune PaliGemma for race, gender, and age group classification. Table 6 shows the performance comparison of the different fine-tuned PaliGemma scenarios and the original pre-trained version (highlighted by the bold values) for emotion classification. The results underscore the superior performance of the third fine-tuning PaliGemma scenario (highlighted by the italic values) which achieved the top-ranking accuracy (59.4%) and F1 score (59%) among all VLMs under zero-shot evaluation. as shown in Table 12 . The confusion matrix of “FaceScanPaliGemma” for emotion classification is shown in Fig. 5 . Table 6. Comparison between various scenarios of fine-tuning PaliGemma for emotion classification. PaliGemma tuning scenarios Accuracy % Precision % Recall % F1 score % Pre-trained PaliGemma 39.8% 41% 40% 36% Fine-tuned PaliGemma-224 52.6% 60% 53% 51% Fine-tuned PaliGemma-448 52.9% 58% 53% 51% Fine-tuned PaliGemma-224-all-parameters (i.e., FaceScanPaliGemma) 59.4% 59% 59% 59% Open in a new tab Table 12. Comparison between FaceScanPaliGemma and other VLMs and baseline methods for emotion classification. Emotion classification methods Accuracy % Precision % Recall % F1 score % FMAE 24 65% – – – POSTER++ 25 63.77% – – – Multi-task EfficientNet-B2 26 63.03% – – – Weighted-Loss Method 23 58% – – – VIT-Base 27 57.99% – – – GPT-4o 49% 58% 49% 47% GPT-4o-mini 46.5% 58% 46% 45% Gemini 1.5 flash 49.9% 53% 50% 48% Llava-NeXT 7b 38.8% 46% 39% 33% Pre-trained PaliGemma 39.8% 41% 40% 36% FaceScan-PaliGemma (proposed) 59.4% 59% 59% 59% Open in a new tab Fig. 5. Open in a new tab Confusion matrix for emotion classification using “FaceScanPaliGemma” tested on the AffectNet dataset. “FaceScanPaliGemma” achieved a 20% increase in the emotion classification accuracy compared to its pre-trained version. Furthermore, it achieved better accuracy than some methods 23 , 27 . However, it falls short of the accuracy achieved by some top-performing methods listed on the AffectNet dataset leaderboard 24 – 26 (highlighted by the bold italic values of the table). The AffectNet emotion dataset is very complex, and this is evident by the top accuracy of only 65%. The accuracy of fine-tuned emotion-PaliGemma in FaceScanPaliGemma is higher than that of AlexNet CNN with the Weighted-Loss Method 23 and Vision Transformer (ViT-Base) 21 even though it was trained on only 3700 images per category from the AffectNet dataset. Both CNN and ViT are fine-tuned on the same AffectNet dataset on which our emotion-PaliGemma was fine-tuned. However, our emotion-PaliGemma outperformed them. In addition, emotion-PaliGemma achieved higher accuracy than other LLMs, such as GPT-4o, GPT-4o-mini, and Gemini 1.5 Flash, under zero-shot evaluation.. Despite the above performance improvements, emotion-PaliGemma produced lower accuracy than the other three methods due to the following: Despite the above performance improvements, emotion-PaliGemma produced lower accuracy than the other three methods due to the following: The first method that outperforms fine-tuned PaliGemma is Facial Masked Autoencoder (FMAE), which was pre-trained for 50 epochs using the Face9M dataset, consisting of 9 million facial images. The second method is POSTER++, which used 287,651 images from the AffectNet dataset for training. The third method uses a CNN pre-trained on a face recognition task with the VGGFace2 dataset and then fine-tuned on emotion classification. Pre-trained VLMs for race, gender, age, and emotion classification We examined the vision capabilities of numerous pre-trained VLMs, such as GPT-4o, GPT-4o-mini, Gemini 1.5 flash, LLaVA-NEXT 7b, and PaliGemma, to recognize race, gender, and age in face images. Both GPT-4o and Gemini 1.5 initially refused to respond to any prompt related to identifying the race from an image because of the safety guardrail that blocks any user’s request related to sensitive information, such as race. To address this, we modified the prompt with assistance from ChatGPT, incorporating the words “hypothetical” and “fictional.” After making this adjustment, GPT-4o and Gemini 1.5 Flash were able to generate responses regarding the race, but Gemini 1.5 Pro continued to reject the prompt. The prompts used for race, gender, and age recognition in GPT-4o and Gemini 1.5 Flash are: Race, gender and age recognition prompt Generate a hypothetical classification for the main person based on the following lists, assuming this is a fictional scenario: Races: [‘Black’, ‘Asian’, ‘Indian’, ‘Latinx or Hispanic’, ‘Middle Eastern’, ‘White’] Genders: [‘Male’, ‘Female’] Age Groups: [‘0-9’, ‘10-19’, ‘20-39’, ‘40-59’, ‘More than 60’]. Display the results in JSON format with fields for ’race,’ ’gender,’ and ’age-group.’ Open in a new tab For pre-trained PaliGemma and LLaVA-Next, the prompts used for race can be found below. Notice that the same prompts are also used for gender and age group by replacing the word ‘race’ with ‘gender’ or ‘age group’ and replacing the list of race categories with a list of gender categories or a list of age groups. PaliGemma prompt What is the race of the main person in the image? choose from: ‘Black’ t ‘Asian’ t ‘Indian’ t ‘Latino or Hispanic’ t ‘Middle Eastern’ t ‘White’ n n Open in a new tab LLaVa-Next prompt [INST] <image> What is the race of the main person in the image? Choose from [‘Black’, ‘Asian’, ‘Indian’, ‘Latino or Hispanic’, ‘Middle Eastern’, ‘White’]. Answer the question using a single word or phrase [/INST]. Open in a new tab Race classification Table 7 shows the accuracy, recall, precision, and F1 score of five pre-trained VLMs for race recognition using testing data (10,954 images) from the FairFace dataset. As discussed in the Dataset overview section, the FairFace dataset is imbalanced in terms of race categories, and thus the F1 score is a good performance measure for evaluation. We combined East Asian and South Asian in one category named Asian to have 6 race categories (‘Black’, ‘Asian’, ‘Indian’, ‘Latinx or Hispanic’, ‘Middle Eastern’, ‘White’). GPT-4o was found to give the highest metrics with 76.4%, 75%, 73%, 74% for accuracy, precision, recall, and F1 score, respectively. Similarly, the GPT-4o mini version gave the second-ranking accuracy of 75.4% and an F1 score of 72%. Additionally, we investigated Google VLMs such as Gemini 1.5 Flash and pre-trained PaliGemma to study their capability for the race recognition task. The results indicate degradation in accuracy in both Gemini 1.5 Flash (68.9%) and pre-trained PaliGemma (68.1%). Similarly, LLaVA-NeXT has less recognition accuracy compared to the previously mentioned VLMs, producing an accuracy of 64.9% in its 7B version. In Table 8 , we compared the GPT-4o that gave the highest accuracy among the pretrained VLMs with other baseline methods in the literature in terms of accuracy. GPT-4o achieved the second-highest ranking, just behind FairFace’s ResNet34 classifier, which secured the top position. This shows the high performance of vision in GPT-4o to recognize the six races. Additionally, we compared the models by separating the Asian race into two classes: East Asian and Southeast Asian, having seven races in total (‘Black’, ‘East Asian’, ‘Indian’, ‘Latinx or Hispanic’, ‘Middle Eastern’, ‘Southeast Asian’, and ‘White’) as shown in Table 9 . The accuracy and F1 score of GPT-4o dropped to 68% and 65%, respectively, when evaluated across seven race categories. This suggests that GPT-4o struggles to differentiate between East Asian and Southeast Asian groups. Table 8. Comparison between FaceScanPaliGemma and baseline methods for 6-race classification. 6-Race classification methods Accuracy % VGGFace-ResNet-50-SVM 18 72.9% FairFace’s (ResNet34) classifier 1 77.7% CLIP’s zero-shot classifier 22 70.7% Google’s FaceNet + SVM 18 74.8% GPT-4o 76.4% FaceScanPaliGemma (proposed) 81.1% Open in a new tab Gender classification We also evaluated five pre-trained VLMs in terms of accuracy, recall, precision, and F1 score for gender classification as shown in Table 10 . The evaluation and comparison were done using testing data (10,954 images) from the FairFace dataset, which has two gender categories (‘Male’ and ‘Female’). Both GPT-4o and GPT-4o mini were found to produce the highest metrics with 95.9%, 96%, 96%, 96% for accuracy, precision, recall, and F1 score, respectively. Additionally, we investigated other pre-trained VLMs such as Gemini 1.5 Flash, pre-trained PaliGemma, and LLaVA-NeXT for gender recognition. The results show high accuracy and F1 scores in all VLMs compared to the baseline methods in the literature. However, pre-trained PaliGemma showed slightly lower accuracy of 93.6% and an F1 score of 94%. Previous results confirmed the vision capabilities of all VLMs in the gender recognition task. Age classification Moreover, five pre-trained VLMs were evaluated for age group classification as shown in Table 11 . The evaluation and comparison were done using testing data (10,954 images) from the FairFace dataset. We combined age groups of the FairFace dataset to have five age groups: 0-9, 10-19, 20-39, 40-59, and 60+. While GPT-4o was found to produce the highest accuracy at 77.4%, GPT-4o mini gave the highest F1 score of 72%. As we mentioned in the dataset overview section, the FairFace dataset is imbalanced in terms of age groups. Therefore, the F1 score is the best metric to measure the performance for the age group classification task. The results indicate a satisfactory performance from Gemini 1.5 Flash with a 65% F1 score. However, both pre-trained PaliGemma and LLaVA-NeXT had difficulty accurately identifying age groups, leading to the lowest F1 scores, especially for pre-trained PaliGemma. Conversely, when comparing VLMs to baseline methods, FairFace ResNet34 achieved the top ranking, with GPT-4o coming in second. Emotion classification In this experiment we set out to study the task of emotion classification using facial expression. We compared several pre-trained VLMs such as GPT-4o, GPT-4o-mini, Gemini 1.5 Flash, Llava-Next, and pre-trained PaliGemma using the testing data (3999 images) of the AffectNet dataset. As shown in Table 12 , the results demonstrate that all pre-trained VLMs, including GPT-4o, GPT-4o-min, Gemini 1.5 Flash, LLaVA 7b, and pre-trained PaliGemma, achieve disappointing performance (highlighted by the bold values of the table), with the results of LLaVA-NeXT 7b and PaliGemma being particularly underwhelming. The results demonstrate that pre-trained VLMs are unable to accurately determine a person’s emotions from facial images. The prompts used for the pre-trained VLMs were: GPT-4o, and Gemini 1.5 Flash prompt What is the emotion of the main person in this image? Pick one of the following: [‘neutral’, ‘happy’, ‘sad’, ‘surprise’, ‘fear’, ‘disgust’, ‘anger’, ‘contempt’]. Answer using a single word. Open in a new tab LLava-Next prompt [INST] <image> What is the emotion of the main person in the image? Choose from [‘neutral’, ‘happy’, ‘sad’, ‘surprise’, ‘fear’, ‘disgust’, ‘anger’, ‘contempt’]. Answer the question using a single word or phrase. [/INST] Open in a new tab PaliGemma prompt Answer en What is the emotion of the main person in the image? choose from: ‘neutral’, t ‘happy’, t ‘sad’ t, ‘surprise’ t ‘fear’ t, ‘disgust’, t ‘anger’, t ‘contempt’ n n Open in a new tab Finally, to study the prompt sensitivity of the pre-trained VLMs for the emotion classification task, we used a prompt with different wording, while retaining the context, and tested it using GPT-4o-mini. More specifically, we used the term ‘facial expression’ instead of ‘emotion’ in the prompt as: Sensitivity prompt What is the facial expression of the main person in this image? Pick one [’neutral’, ’happy’, ’sad’, ’surprise’, ’fear’, ’disgust’, ’anger’, ’contempt’]. Answer using a single word. Open in a new tab The results of this sensitivity analysis show that using such an alternative prompt improved the accuracy, increasing it from 46.5% (with ‘emotion’) to 48.7% (with ‘facial expression’). In summary, the four fine-tuned PaliGemma models that form the core of FaceScanPaliGemma, specializing in recognizing race, gender, age group, and emotion, have demonstrated consistently strong performance. Their results not only surpass other open-source VLMs but also proprietary models. Compared to existing methods in the literature, the models show particularly notable gains, with the emotion recognition agent remaining a more challenging task. Evaluation of FaceScanPaliGemma in complex scenarios We added a face detection VLM agent to the “FaceScanPaliGemma” system. This VLM is a pre-trained PaliGemma with the prompt “detect x.” x can be ’face’ in general to detect all faces in the image or ’face of person’ with specific physical attributes (e.g., haircut, hair color, and clothes color), action performed, postures, and facial attributes (e.g., emotion, race, gender, and age group). All of these functions can be driven by the prompt provided along with an image. The detected faces are fed to one of the race, gender, age, or emotion VLM agents in “FaceScanPaliGemma” based on the provided prompt. The images in Table 13 , Table 14 , and Table 15 are available in this link https://github.com/Nouar-NYUAD/FaceScanPaliGemma/tree/main . We evaluated “FaceScanPaliGemma” in several scenarios using various prompts. First, we evaluated the capability of “FaceScanPaliGemma” to find the race, gender, and age groups for all individuals in several images using the race-gender-age testing prompt. This prompt is designed to identify all individuals who appear in the image. Table 13 shows the capability of “FaceScanPaliGemma” in recognizing the facial attributes accurately on images with multiple individuals. Table 13. Several examples using the race-gender-age testing prompt. Image Generated text image1.jpg [ { "Black " , "F" , "40−59" } , {“White” , "F" , "60+" } , {"Black " , "F" , "20−39" } , {“Asian” , "F" , "20−39" } , {“White” , "F" , "20−39" } , {"M. Eas tern " , "F" , "20−39" } , {"Black " , "F" , "20−39" } ] image2.jpg [ { “Asian” , "M" , "0−9" } , {"Black " , "M" , "0−9" } , {“White” , "M" , "0−9" } ] image3.jpg [ { “Asian” , "F" , "60+" } , {"Black " , "M" , "60+" } , {“White” , "M" , "60+" } , {“Asian” , "M" , "60+" } , {“White” , "F" , "60+" } ] image4.jpg [ { “Asian” , "F" , "20−39" } , {“White” , "F" , "20−39" } , {"Black " , "M" , "20−39" } , {"M. Eas tern " , "F" , "20−39" } , {“White” , "M" , "20−39" } , {“Asian” , "F" , "20−39" } ] image5.jpg [ { "Latinx_Hispanic " , "F" , "60+" } , {"Latinx_Hispanic " , "F" , "40−59" } , {"Latinx_Hispanic " , "M" , "0−9" } , {"Latinx_Hispanic " , "M" , "60+" } , {"Latinx_Hispanic " , "M" , "20−39" } , {"Latinx_Hispanic " , "F" , "10−19" } ] image6.jpg [ { "Black " , "F" , "20−39" } , {"Black " , "F" , "0−9" } , {"Black " , "M" , "60+" } , {"Black " , "F" , "0−9" } , {"Black " , "F" , "60+" } , {"Black " , "M" , "20−39" } ] image7.jpg [ { “White” , "M" , "60+" } , {“White” , "F" , "60+" } , {"Latinx_Hispanic " , "F" , "20−39" } , {“White” , "M" , "40−59" } , {“White” , "F" , "20−39" } , {“White” , "F" , "0−9" } , {“White” , "M" , "0−9" } ] image8.jpg [ { “Asian” , "F" , "0−9" } , {“Asian” , "F" , "20−39" } , {“Asian” , "M" , "0−9" } , {“Asian” , "F" , "60+" } , {“Asian” , "M" , "20−39" } , {“Asian” , "M" , "60+" } ] image9.jpg [ { " Indian " , "F" , "20−39" } , {" Indian " , "M" , "40−59" } , {" Indian " , "M" , "40−59" } , {" Indian " , "M" , "20−39" } , {" Indian " , "M" , "60+" } , {" Indian " , "M" , "10−19" } , {" Indian " , "F" , "60+" } ] image10.jpg [ { "M. Eas tern " , "M" , "20−39" } , {"M. Eas tern " , "M" , "40−59" } , {"M. Eas tern " , "F" , "20−39" } , {"M. Eas tern " , "F" , "40−59" } , {"M. Eas tern " , "F" , "20−39" } , {"M. Eas tern " , "M" , "10−19" } ] Open in a new tab Table 14. Several examples using a set of testing prompts. Image Prompt Generated text image1.jpg Individual wearing a brown shirt {“Asian” , "F" , "20−39"} Individual wearing a headscarf {"M. Eas tern " , "F" , "20−39"} Individual wearing eyeglasses [ { "Black " , "F" , "60+" } , {“White” , "F" , "40−59" } ] image9.jpg Calculate the number of Indian individuals Seven image4.jpg Individual wearing a red jacket {"M. Eas tern " , "F" , "20−39"} Male Individual [ { "Black " , "M" , "20−39" } , {“White” , "M" , "20−39" } ] image3.jpg How many males are in the image? Three Calculate the number of Asians Two Calculate the number of Black individuals One image8.jpg How many individuals are under 10 in the image Two How many individuals are more than 60 in the image Two image11.jpg What is the race, gender, and age group of the person holding a ball? Black, Female, 0-9 What is the race, gender, and age group of the person singing? Asian, Female, 0-9 image12.jpg What is the race, gender, and age group of the person bending their knee? Black, Female, 0-9 What is the race, gender, and age group of the person standing? White, Male, 0-9 image13.jpg What is the race, gender, and age group of the person with the newspaper? White, Male, 20-39 What is the race, gender, and age group of the person on the carpet? White, Male, 0-9 What is the race, gender, and age group of the white-haired person? White, Female, More than 60 image14.jpg What is the race, gender, and age group of the person with the crossed fingers on both hands? Asian, Female, More than 60 What is the race, gender, and age group of the person holding a child? Asian, Female, 20-39 Open in a new tab Table 15. Several examples using emotion testing prompts. Image Prompt Generated text image15.jpg Emotion of each person Happy, Surprised Emotion of the individual wearing the grey shirt Surprise image16.jpg Emotion of each person Happy, Anger, Surprise Emotion of the Black individual Happy image17.jpg Emotion of each person Fear, Happy Emotion of the Middle Eastern individual Happy image18.jpg Emotion of each person Surprise, Happy, Fear Emotion of the Asian Eastern individual Happy image19.jpg Emotion of each person Happy, Disgust Open in a new tab Next, we evaluated capability of “FaceScanPaliGemma” in finding the race, gender, or age groups for several images using a set of the testing prompts targeting physical attributes of individuals such as: wearing a brown shirt, “wearing eyeglasses, “wearing a head scarf, “lying, standing, singing, “holding a child, “holding a ball, “holding a newspaper, and “with white hair”. Table 14 confirms the capability of “FaceScanPaliGemma” to recognize facial attributes accurately in images with multiple individuals driven by a prompt targeting specific physical attributes, actions performed, or postures. Finally, we evaluated “FaceScanPaliGemma” in terms of emotion recognition for several images using an emotion testing prompt: emotion of each person, “emotion of a specific race, “emotion of a specific gender, and “emotion of a specific age group. Table 15 shows that “FaceScanPaliGemma” is able to recognize emotions accurately in images with multiple individuals driven by a prompt targeting specific race, gender, and/or age groups. Ethical and societal implications For bias and fairness considerations, generally VLMs are subject to a number of biases embedded in training data. Demographic imbalances (e.g., underrepresentation of certain age groups, races, or genders) can lead to poor performance. In practice, this could manifest in systematically lower accuracy for marginalized populations. Such biases not only reduce model reliability but also risk reinforcing social stereotypes if deployed in sensitive contexts such as security screening. However, fairness-aware fine-tuning of FaceScanPaliGemma using the FairFace dataset that has been implemented in this work considered these issues. The confusion matrices and tables that demonstrate the metrics for each race, gender, and age group category show robustness against bias. For privacy concerns, facial attribute recognition inherently processes highly sensitive biometric data. A key ethical issue is that such images are often collected without explicit user consent. To mitigate this, FaceScanPaliGemma has been trained exclusively on FairFace, a publicly available and widely used benchmark dataset that was already released under terms permitting research use, reducing the risk of unauthorized data exploitation. However, we acknowledge that privacy risks persist at the level of application. For potential misuse, while designed for research and socially beneficial applications (e.g., demographic analysis), the model could be misappropriated for harmful uses. Examples include surveillance or automated discrimination. To address these privacy concerns and discourage misuse, we include a “Governance and Responsible Use” section in our open-source releases of models (available on Hugging Face model cards). Discussion The results show that some LLMs, like GPT-4o, can do better than most traditional methods at recognizing race, gender, and age from faces. However, GPT-4o performs poorly when it comes to recognizing emotions under zero-shot evaluation. In contrast, models like Gemini 1.5 Flash and LLaVA-Next perform worse than traditional methods for race, age, and emotion recognition. Similarly, the pre-trained version of PaliGemma also shows weak performance in recognizing age, emotion, and race. These above findings address RQ1 . However, the fine-tuned version of PaliGemma performs very well, better than other multi-task VLMs under zero-shot evaluation and traditional methods, addressing RQ2 . The disparities that appear in some demographic categories stem primarily from the inherent ambiguity of appearance-based features or attributes. This finding suggests that feature ambiguity remains a challenge even with fairness-aware fine-tuning. “FaceScanPaliGemma” shows superior performance and produces accurate outcomes in the complex scenarios and prompt targeting of specific physical attributes, actions performed, or postures. which shows the robustness against the presence of various objects and textures in the background, which answers RQ3 . The strength of each module directly affects how well the entire system works. We found that using weaker classifiers instead of our VLM-based modules leads to wrong face selections (incorrect race or gender), which then causes errors to spread to later stages of the system. For emotion classification, the PaliGemma model was fine-tuned on 3,700 images per category from AffectNet, yielding an accuracy of 59.4%. While competitive, this remains below the top reported methods (65%). Future work will explore pretraining on larger and more diverse datasets such as Face9M or VGGFace2 before fine-tuning on AffectNet. Training the model first on large face datasets can help it learn more general and robust facial features. After that, fine-tuning on emotion datasets such as AffectNet can improve its ability to recognize subtle emotions. Such a staged approach would allow the model to learn richer facial representations, particularly subtle variations in expression, thereby narrowing the accuracy gap. Larger-scale pretraining is expected to improve generalization across demographics and environmental conditions, which is critical for robust emotion recognition. We also plan to extend training to more challenging conditions, including variations in lighting and occlusion. Running multiple independent models can raise inference complexity and hinder scalability in practical systems, particularly when inference times are high. In contrast, the fine-tuned PaliGemma model developed in this work is a compact language model with just 3 billion parameters, embodying the emerging direction of Agentic AI 5 . It achieves face image inference in 0.135 seconds and processes 7 frames per second on an A100-80GB GPU. This balance of rapid inference and strong recognition accuracy makes the fine-tuned PaliGemma highly suitable for real-world applications that demand both speed and precision. Beyond ethical considerations, the societal implications of deploying “FaceScanPaliGemma” must also be recognized, particularly the potential privacy concerns in surveillance contexts, which highlight the importance of aligning technical innovation with evolving AI ethics standards. Conclusion This paper demonstrated the challenging problem of recognizing human facial attributes such as emotion, race, gender, and age group. We proposed “FaceScanPaliGemma,” a multi-agent VLM system containing four fine-tuned PaliGemma models for race, gender, age group, and emotion classification. “FaceScanPaliGemma” was found to outperform other VLMs and several methods for facial attribute recognition tasks. The results show the highest accuracy and F1 score for the race, gender, and age group classification, tested on the FairFace dataset. The results indicate that “FaceScanPaliGemma” offers excellent recognition performance combined with speed, affordability, and efficiency. The success of this design highlights the potential for leveraging such agents in future applications beyond basic facial attribute recognition, such as in computational social science and human-AI interaction studies. limitation By evaluating the performance of “FaceScanPaliGemma” on the AffectNet dataset for the emotion classification task, we observed that emotion PaliGemma was able to outperform other VLMs. However, it still falls short of the performance achieved by top state-of-the-art models that have been trained on larger datasets, such as FMAE, with several fine-tuning stages, such as Multi-task EfficientNet-B2, and several techniques, such as POSTER++. One limitation of this study is that prompt design was not explored exhaustively. While more complex prompts, providing richer descriptions of emotions and clearer distinctions between emotion categories, may improve emotion classification performance. The FaceScanPaliGemma system has primarily been evaluated on the FairFace and AffectNet datasets. However, its robustness in real-world scenarios has not yet been systematically assessed. In practical applications, facial images are often subject to a wide range of challenges, including variations in lighting conditions, partial occlusion of the face, and environmental noise. These factors can significantly degrade recognition accuracy and reliability. Therefore, a comprehensive evaluation in diverse, real-world settings is necessary to validate the generalizability and robustness of the system before it can be confidently deployed in practice. Additionally, the system lacks explainability because it does not provide interpretable reasoning behind its classifications, which limits trust and transparency. Future work To address the explainability limitation, future work could focus on integrating explainable AI techniques into FaceScanPaliGemma to highlight which facial regions most influence the model’s predictions. Incorporating these mechanisms would not only improve transparency and trust but also allow us to better detect and mitigate potential biases in the system. Furthermore, we plan to extend our system to person identification by incorporating a retrieval-augmented generation (RAG) framework. Specifically, we aim to leverage attribute-specific embeddings extracted from PaliGemma’s internal representations for emotion, race, age, and gender. In this approach, the visual encoder of PaliGemma, already fine-tuned for facial attribute recognition, will be used to generate compact, discriminative feature vectors for each attribute category. These embeddings will then serve as semantic keys in a vector database, enabling efficient similarity search against a gallery of known individuals. Supplementary Information Supplementary Information. (77.2KB, pdf) Acknowledgements The FaceScanPaliGemma model governs the processing of sensitive biometric data by prioritizing ethical use, transparency, and strict legal compliance. While it is permitted for academic research and fairness-aware development, it is strictly prohibited for unauthorized surveillance, non-consensual identification, or discriminatory practices. Due to risks like bias and privacy violations, its use in law enforcement is discouraged and must be subject to independent auditing and frameworks like the GDPR or EU AI Act to ensure accountability and the protection of individual rights. Author contributions Conceptualization by N.A.; data curation by N.A.; formal analysis by N.A., Y.Z.; funding acquisition by Y.Z.; investigation by N.A.; methodology by N.A.; project administration by Y.Z.; software by N.A.; validation by N.A.; visualization by N.A., Y.Z.; writing-original draft preparation by N.A., M.J.T.T., H.R.K.; writing, review and editing by N.A., Y.Z. Funding No Funding. Data availability Data will be available upon request. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Supplementary Information The online version contains supplementary material available at 10.1038/s41598-026-39584-3. References 1. Karkkainen, K. & Joo, J. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , 1548–1558 (2021). 10.1109/WACV48630.2021.00159. 2. Yuan, H., He, Y., Du, P. & Song, L. Multi-task learning using uncertainty to weigh losses for heterogeneous face attribute estimation. arXiv preprint arXiv:2403.00561 (2024). 3. Lee, S. H., Plataniotis, K. N. & Ro, Y. M. Intra-class variation reduction using training expression images for sparse representation based facial expression recognition. IEEE Trans Affective Comput 5 , 340–351. 10.1109/TAFFC.2014.2346515 (2014). [ Google Scholar ] 4. Singh Sengar, S., Hasan, A. B., Kumar, S. & Carroll, F. Generative artificial intelligence: A systematic review and applications. arXiv e-prints arXiv–2405 (2024). arXiv:2405.11029 . 5. Belcak, P. et al. Small language models are the future of agentic ai. arXiv preprint arXiv:2506.02153 (2025). 6. Murugappan, G. Multiagent Debate among Vision-Language Models Improves Multimodal Reasoning . Master’s thesis, Georgia Institute of Technology, Atlanta, GA (2024). 7. Ali, M. H. & Rasheed, M. A. A blockchain-based multi-agent security framework for e-commerce systems. International Journal of Theoretical & Applied Computational Intelligence (2025). URL https://orcid.org/0000-0001-7963-0918 . 8. Hosseini, M. M., Fard, A. P. & Mahoor, M. H. Faces of fairness: Examining bias in facial expression recognition datasets and models. arXiv preprint arXiv:2502.11049 (2025). 9. Jiang, J., Wang, M., Xiao, B., Hu, J. & Deng, W. Joint recognition of basic and compound facial expressions by mining latent soft labels. Pattern Recognition 148 , 110173. 10.1016/j.patcog.2023.110173 (2024). [ Google Scholar ] 10. Hosseini, M. M. & Jamzad, M. Facial mark detection and removal using graph relations and statistics. In 2017 Iranian Conference on Electrical Engineering (ICEE) , 2223–2228 (IEEE, 2017). 10.1109/IranianCEE.2017.7985432. 11. Yang, D. et al. Robust emotion recognition in context debiasing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 12447–12457 (2024). 12. Yu, J. et al. Exploring facial expression recognition through semi-supervised pre-training and temporal modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 4880–4887 (2024). 10.1109/CVPRW63382.2024.00492. 13. Liang, H., Perona, P. & Balakrishnan, G. Benchmarking algorithmic bias in face recognition: An experimental approach using synthetic faces and human evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 4977–4987 (2023). 10.1109/ICCV51070.2023.00459. 14. Hafeez, A. et al. Addressing imbalance problem for multi label classification of scholarly articles. IEEE Access 11 , 74500–74516. 10.1109/ACCESS.2023.3293852 (2023). [ Google Scholar ] 15. Schroff, F., Kalenichenko, D. & Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition , 815–823 (2015). 10.1109/CVPR.2015.7298682. 16. Mustapha, M. F., Mohamad, N. M., Osman, G. & Ab Hamid, S. H. Age group classification using convolutional neural network (cnn). In Journal of Physics: Conference Series , vol. 2084, 012028 (IOP Publishing, 2021). 10.1088/1742-6596/2084/1/012028. 17. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , 770–778 (2016). 10.1109/CVPR.2016.90. 18. AlDahoul, N., Rahwan, T. & Zaki, Y. Ai-generated faces influence gender stereotypes and racial homogenization. Scientific reports 15 , 14449. 10.1038/s41598-025-99623-3 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 19. Cao, Q., Shen, L., Xie, W., Parkhi, O. M. & Zisserman, A. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018) , 67–74 (IEEE, 2018). 10.1109/FG.2018.00020. 20. Tan, M. & Le, Q. V. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR arXiv:1905.11946 (2019). 21. Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (2021). 10.48550/arXiv.2010.11929. 22. Radford, A. et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , 8748–8763 (PMLR, 2021). 10.48550/arXiv.2103.00020. 23. Mollahosseini, A., Hasani, B. & Mahoor, M. Affectnet: a new database for facial expression, valence, and arousal computation in the wild. IEEE Trans. Affect. Comput. 99 , 1–1. 10.1109/TAFFC.2017.2740923 (2008). [ Google Scholar ] 24. Ning, M., Salah, A. A. & Ertugrul, I. O. Representation learning and identity adversarial training for facial behavior understanding. arXiv preprint arXiv:2407.11243 (2024). 25. Mao, J. et al. Poster++: A simpler and stronger facial expression recognition network. arXiv preprint arXiv:2301.12149 (2023). 26. Savchenko, A. V., Savchenko, L. V. & Makarov, I. Classifying emotions and engagement in online learning based on a single facial expression recognition neural network. IEEE Transa. Affective Comput. 13 , 2132–2143. 10.1109/TAFFC.2022.3188390 (2022). [ Google Scholar ] 27. Li, J., Nie, J., Guo, D., Hong, R. & Wang, M. Emotion separation and recognition from a facial expression by generating the poker face with vision transformers. arXiv preprint arXiv:2207.11081 (2022). 28. Hussain, N. et al. A deep neural network and classical features based scheme for objects recognition: an application for machine inspection. Multi. Tools Appl. 83 , 14935–14957. 10.1007/s11042-020-08852-3 (2024). [ Google Scholar ] 29. Ikram, A. et al. A systematic review on fundus image-based diabetic retinopathy detection and grading: Current status and future directions. IEEE Access 10.1109/ACCESS.2024.3427394 (2024). [ Google Scholar ] 30. Khan, A. Q. et al. A novel fusion of genetic grey wolf optimization and kernel extreme learning machines for precise diabetic eye disease classification. Plos one 19 , e0303094. 10.1371/journal.pone.0303094 (2024). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 31. Abdulwahid, A. A. Classification of ethnicity using efficient cnn models on morph and feret datasets based on face biometrics. Appl. Sci. 13 , 7288. 10.3390/app13127288 (2023). [ Google Scholar ] 32. Mello-Klein, C. Facebook’s ad delivery algorithm is discriminating based on race, gender and age in photos, northeastern researchers find. https://news.northeastern.edu/2022/10/25/facebook-algorithm-discrimination/ (2022). 33. Sunitha, G. et al. Intelligent deep learning based ethnicity recognition and classification using facial images. Image Vision Comput. 121 , 104404. 10.1016/j.imavis.2022.104404 (2022). [ Google Scholar ] 34. Ahmed, M. A., Choudhury, R. D. & Kashyap, K. Race estimation with deep networks. J. King Saud Univ.-Comput. Inform. Sci. 34 , 4579–4591. 10.1016/j.jksuci.2020.11.029 (2022). [ Google Scholar ] 35. Al-Humaidan, N. A. & Prince, M. A classification of arab ethnicity based on face image using deep learning approach. IEEE Access 9 , 50755–50766. 10.1109/ACCESS.2021.3069022 (2021). [ Google Scholar ] 36. AlBdairi, A. J. A. et al. Face recognition based on deep learning and fpga for ethnicity identification. Appl. Sci. 12 , 2605 (2022). [ Google Scholar ] 37. Haseena, S. et al. Prediction of the age and gender based on human face images based on deep learning algorithm. Comput. Math. Methods Med. 2022 , 1413597. 10.1155/2022/1413597 (2022). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 38. Fayyaz, M. et al. Pedestrian gender classification on imbalanced and small sample datasets using deep and traditional features. Neural Comput. Appl. 35 , 11937–11968. 10.1007/s00521-023-08331-4 (2023). [ Google Scholar ] 39. Sonthi, V. K. et al. A deep learning technique for smart gender classification system. In 2023 3rd International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE) , 983–987 (IEEE, 2023). 10.1109/ICACITE57410.2023.10182683. 40. Yaman, D., Eyiokur, F. I. & Ekenel, H. K. Multimodal soft biometrics: combining ear and face biometrics for age and gender classification. Multimedia Tools and Applications 1–19 (2022). 10.1007/s11042-021-10630-8. 41. Tunc, A., Tasdemir, S., Koklu, M. & Cinar, A. C. Age group and gender classification using convolutional neural networks with a fuzzy logic-based filter method for noise reduction. J. Intell. & Fuzzy Systems 42 , 491–501. 10.1109/CVPRW.2015.7301352 (2022). [ Google Scholar ] 42. Ciobotaru, A. et al. Comparing deep learning and genetic algorithms techniques for age and gender classification. In 2023 3rd International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME) , 1–6 (IEEE, 2023). 10.1109/ICECCME57830.2023.10253422. 43. Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inform. Processing Syst. 25 , 84–90. 10.1145/3065386 (2012). [ Google Scholar ] 44. Reddy, A. M., Krishna, V. V., Sumalatha, L. & Obulesh, A. Age classification using motif and statistical features derived on gradient facial images. Recent Adv. Comput. Sci. Commun. (Formerly: Recent Patents Comput. Sci. 13 , 965–976. 10.2174/2213275912666190417151247 (2020). [ Google Scholar ] 45. Duan, M., Li, K., Yang, C. & Li, K. A hybrid deep learning cnn-elm for age and gender classification. Neurocomputing 275 , 448–461. 10.1016/j.neucom.2017.08.062 (2018). [ Google Scholar ] 46. Huang, G.-B., Zhu, Q.-Y. & Siew, C.-K. Extreme learning machine: theory and applications. Neurocomputing 70 , 489–501. 10.1016/j.neucom.2005.12.126 (2006). [ Google Scholar ] 47. Chowdary, M. K., Nguyen, T. N. & Hemanth, D. J. Deep learning-based facial emotion recognition for human-computer interaction applications. Neural Comput. Appl. 35 , 23311–23328. 10.1007/s00521-021-06012-8 (2023). [ Google Scholar ] 48. Meena, G., Mohbey, K. K. & Lokesh, K. Fstl-sa: Few-shot transfer learning for sentiment analysis from facial expressions. Multi. Tools Appl. 84 , 24457–24485. 10.1007/s11042-024-20518-y (2025). [ Google Scholar ] 49. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014). 50. Meethongjan, K., Dzulkifli, M., Rehman, A., Altameem, A. & Saba, T. An intelligent fused approach for face recognition. J. Intell. Syst. 22 , 197–212. 10.1515/jisys-2013-0010 (2013). [ Google Scholar ] 51. Deng, S. et al. Learning to compose diversified prompts for image emotion classification. Comput. Visual Media 10 , 1169–1183. 10.1007/s41095-023-0389-6 (2024). [ Google Scholar ] 52. Wang, J. et al. Mdkat: Multimodal decoupling with knowledge aggregation and transfer for video emotion recognition. IEEE Trans. Circuits Syst. Video Technol. 10.1109/TCSVT.2025.3571534 (2025). [ Google Scholar ] 53. Pandi, C., Adi Narayana Reddy, K., Alladi, R., Chandra Sekhar Reddy, V. & Sumithabhashini, P. Emotion and gender classification using convolution neural networks. In ICT Systems and Sustainability: Proceedings of ICT4SD 2021, Volume 1 , 563–573 (Springer, 2022). 10.1007/978-981-16-5987-4_57. 54. Khellat-Kihel, S., Muhammad, J., Sun, Z. & Tistarelli, M. Gender and ethnicity recognition based on visual attention-driven deep architectures. J. Visual Commun. Image Representation 88 , 103627. 10.1016/j.jvcir.2022.103627 (2022). [ Google Scholar ] 55. Foggia, P., Greco, A., Saggese, A. & Vento, M. Multi-task learning on the edge for effective gender, age, ethnicity and emotion recognition. Eng. Appl. Artificial Intell. 118 , 105651. 10.1016/j.engappai.2022.105651 (2023). [ Google Scholar ] 56. AlDahoul, N., Tan, M. J., Kasireddy, H. R. & Zaki, Y. Guardians of digital safety: benchmarking large language models in the fight against online toxicity. J. Big Data 13 , 6. 10.1186/s40537-025-01336-x (2025). [ Google Scholar ] 57. Kuo, C. W. et al. Neutralizing the narrative: Ai-powered debiasing of online news articles. arXiv preprint arXiv:2504.03520 (2025). 58. AlDahoul, N. & Zaki, Y. Nyuad at qias shared task: Benchmarking the legal reasoning of llms in arabic islamic inheritance cases. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks , 861–866 (2025). 10.18653/v1/2025.arabicnlp-sharedtasks.118. 59. AlDahoul, N. & Zaki, Y. Nyuad at arahealthqa shared task: Benchmarking the medical understanding and reasoning of large language models in arabic healthcare tasks. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks , 119–125 (2025). 10.18653/v1/2025.arabicnlp-sharedtasks.19. 60. AlDahoul, N., Hong, J., Varvello, M. & Zaki, Y. Towards a world wide web powered by generative ai. Sci. Rep. 15 , 7251. 10.1038/s41598-024-77301-0 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 61. AlDahoul, N., Tan, M. J. T., Kasireddy, H. R. & Zaki, Y. Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos. arXiv preprint arXiv:2411.17123 (2024). 62. AlDahoul, N. et al. Multitasking vision language models for vehicle plate recognition with vehiclepaligemma. Sci. Rep. 15 , 1–15. 10.1038/s41598-025-10774-9 (2025). [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 63. AlDahoul, N. & Zaki, Y. Nyuad at mahed shared task: Detecting hope, hate, and emotion in arabic textual speech and multi-modal memes using large language models. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks , 575–584 (2025). 10.18653/v1/2025.arabicnlp-sharedtasks.76. 64. Abdelhamed, A., Afifi, M. & Go, A. What do you see? enhancing zero-shot image classification with multimodal large language models. arXiv preprint arXiv:2405.15668 (2024). 65. Hello gpt-4o. https://openai.com/index/hello-gpt-4o// (2024). 66. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). 67. Yang, B. et al. Vision transformer-based visual language understanding of the construction process. Alexandria Eng. J. 99 , 242–256. 10.1016/j.aej.2024.05.015 (2024). [ Google Scholar ] 68. Yildirim, N. et al. Multimodal healthcare ai: identifying and designing clinically relevant vision-language applications for radiology. In Proceedings of the CHI Conference on Human Factors in Computing Systems , 1–22 (2024). 10.1145/3613904.3642013. 69. Hartsock, I. & Rasool, G. Vision-language models for medical report generation and visual question answering: A review. arXiv preprint arXiv:2403.02469 (2024). [ DOI ] [ PMC free article ] [ PubMed ] 70. Antol, S. et al. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , 2425–2433 (2015). 10.1109/ICCV.2015.279. 71. Beyer, L. et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 (2024). 72. LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proceed. IEEE 86 , 2278–2324. 10.1109/5.726791 (2002). [ Google Scholar ] 73. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. et al. Improving language understanding by generative pre-training (2018). URL https://www.mikecaptain.com/resources/pdf/GPT-1.pdf . 74. Srivatsan, K., Naseer, M. & Nandakumar, K. Flip: Cross-domain face anti-spoofing with language guidance. In Proceedings of the IEEE/CVF international conference on computer vision , 19685–19696 (2023). 10.1109/ICCV51070.2023.01803. 75. Liu, A. et al. Cfpl-fas: Class free prompt learning for generalizable face anti-spoofing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 222–232 (2024). 10.1109/CVPR52733.2024.00029. 76. Liu, A. et al. Fm-clip: Flexible modal clip for face anti-spoofing. In Proceedings of the 32nd ACM International Conference on Multimedia , 8228–8237 (2024). 10.1145/3664647.3680856. 77. Lin, K.-H., Tseng, Y.-W., Huang, K.-Y., Wu, J.-C. & Cheng, W.-H. Instructflip: Exploring unified vision-language model for face anti-spoofing. In Proceedings of the 33rd ACM International Conference on Multimedia , 2987–2996 (2025). 10.1145/3746027.3754939. 78. Gemini Team, G. Gemini 1.5 technical report. https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf (2024). 79. Introducing gemini 1.5, google’s next-generation ai model. https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/#architecture (2024). 80. Gpt-4o: The comprehensive guide and explanation. https://blog.roboflow.com/gpt-4o-vision-use-cases/ (2024). 81. Llava: Large language and vision assistant explained. https://encord.com/blog/llava-large-language-vision-assistant/ (2024). 82. Liu, H. et al. Llava-next: Improved reasoning, ocr, and world knowledge (2024). URL https://llava-vl.github.io/blog/2024-01-30-llava-next/ . 83. Dang, P. Multimodal (visual and language) understanding with llava-next. https://rocm.blogs.amd.com/artificial-intelligence/llava-next/README.html (2023). 84. Zhu, H. et al. Harnessing large vision and language models in agriculture: A review. arXiv preprint arXiv:2407.19679 (2024). [ DOI ] [ PMC free article ] [ PubMed ] 85. Thomee, B. et al. Yfcc100m: The new data in multimedia research. Commun. ACM 59 , 64–73 (2016). [ Google Scholar ] 86. Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ (2024). 87. Hugging face. https://huggingface.co/ . 88. AWS. Amazon rekognition. https://docs.aws.amazon.com/rekognition/latest/APIReference/API_AgeRange.html . Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials Supplementary Information. (77.2KB, pdf) Data Availability Statement Data will be available upon request. Articles from Scientific Reports are provided here courtesy of Nature Publishing Group ACTIONS View on publisher site PDF (2.7 MB) Cite Collections Permalink PERMALINK Copy RESOURCES Similar articles Cited by other articles Links to NCBI Databases Cite Copy Download .nbib .nbib Format: AMA APA MLA NLM Add to Collections Create a new collection Add to an existing collection Name your collection * Choose a collection Unable to load your collection due to an error Please try again Add Cancel Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top