arXiv:2604.01626v1 [cs.DB] 2 Apr 2026
C OG P IC : A M ULTIMODAL DATASET FOR E ARLY C OGNITIVE I MPAIRMENT A SSESSMENT VIA P ICTURE D ESCRIPTION TASKS
Liuyu Wu∗ Nanjing Medical University Nanjing, China [email protected] Wentao Xiang Nanjing Medical University Nanjing, China [email protected] Siyang Song University of Exeter Exeter, United Kingdom [email protected]
Rui Feng∗ Nanjing Medical University Nanjing, China [email protected]
Yi Zhang Changzhou Second People’s Hospital Changzhou, China [email protected] Xiao Gu University of Oxford Oxford, United Kingdom [email protected]
Jie Li Nanjing Medical University Nanjing, China [email protected] Yin Cao Changzhou Second People’s Hospital Changzhou, China [email protected] Jianqing Li† Nanjing Medical University Nanjing, China [email protected]
Wei Wang† Nanjing Medical University Nanjing, China [email protected]
A BSTRACT The automated evaluation of cognitive status utilizing multimedia technologies presents a promising frontier in early dementia diagnosis. However, the development of robust machine learning models for cognitive impairment detection is frequently hindered by the scarcity of large-scale, strictly synchronized, and clinically validated multimodal datasets. To bridge this critical gap, we introduce the CogPic database, a comprehensive multimodal benchmark meticulously designed for fine-grained cognitive impairment detection. The dataset comprises strictly synchronized audio, visual, and linguistic data continuously collected from 574 participants during a naturalistic picture description task. To establish highly reliable diagnostic ground truth, expert clinical neuropsychologists conducted exhaustive evaluations, stratifying participants into distinct cognitive groups through a comprehensive clinical consensus. Consequently, CogPic stands as the largest, most modality-rich, and most meticulously evaluated dataset of its kind to date. By conducting extensive benchmark experiments on the CogPic dataset, we establish an exceptionally robust, unbiased, and clinically generalizable foundation to propel future multimedia research in automated cognitive health assessment. Detailed information and access application procedures for our CogPic database are available at https: //cogpic.github.io/. Keywords Cognitive Assessment Dataset · Multimodal Benchmark · Cognitive Impairment Detection · Alzheimer’s Disease
∗ †
Both authors contributed equally to this work. Corresponding author.
1
Introduction
The accurate assessment of cognitive function is crucial for the early detection of neurodegenerative diseases such as Alzheimer’s Disease (AD) and Mild Cognitive Impairment (MCI) [1]. Early identification allows for timely intervention, which can help mitigate disease progression and improve patients’ quality of life. However, current clinical diagnosis heavily relies on standard neuropsychological tests and imaging techniques, which are often time-consuming and expensive. Consequently, there is an urgent need for accessible, non-invasive, and automated screening paradigms. Recent studies have demonstrated that subtle external behavioral anomalies, including alterations in speech patterns and facial expressions, can serve as reliable indicators of prodromal AD [2, 3, 4]. Motivated by this, a growing body of research has focused on extracting digital biomarkers from such non-contact signals for cognitive assessment. Among various methodologies, the Picture Description Task has emerged as a widely adopted approach. As a non-invasive and naturalistic elicitation paradigm, it can effectively reflect a subject’s underlying language, memory, and executive functions. Despite the proven efficacy of the Picture Description Task, existing public datasets suffer from several critical limitations that hinder the advancement of comprehensive multimodal diagnostic systems for cognitive impairment detection. First, constrained by stringent privacy regulations regarding facial identifiability and the logistical challenges of high-fidelity synchronous recording, established databases, including widely used repositories like DementiaBank, predominantly provide audio recordings and their corresponding text transcripts. However, cognitive decline manifests well beyond linguistic and acoustic dimensions, heavily involving visual behaviors. Subtle visual cues, including gaze wandering during word retrieval, diminished facial expressivity, and psychomotor retardation, serve as vital clinical indicators. Without a synchronized video modality, existing computational models are forced to rely on a fundamentally incomplete representation. This modality bottleneck restricts models to capturing merely fragmented aspects of a patient’s cognitive state, severely limiting their capacity to detect early-stage impairments. Second, the inherent difficulty of recruiting vulnerable elderly cohorts, coupled with the immense clinical burden and high cost of obtaining expert neuropsychological annotations, has severely restricted the scale of prevailing datasets. This bottleneck fails to satisfy the data-hungry nature of modern deep learning architectures. With many highly cited cohorts consisting of fewer than 300 participants [5, 6], the optimization of complex, highly parameterized multimodal networks is inherently constrained. This scarcity of training samples severely exacerbates the risk of overfitting and hinders the development of robust, clinically generalizable diagnostic models. Finally, the majority of established datasets are heavily biased toward English-speaking populations. This lack of linguistic and cultural diversity creates a significant barrier to developing and validating automated screening tools for the Chinese-speaking community, thereby limiting the global applicability of these technologies. To overcome the aforementioned barriers, we introduce the CogPic database, a novel, large-scale, comprehensively annotated, and unconstrained multimodal repository. Utilizing a standardized, tablet-based acquisition framework, we successfully recruited a diverse, demographically representative cohort of 574 participants. Specifically, CogPic distinguishes itself through three fundamental advancements. First, it employs a multi-task elicitation paradigm featuring three distinct picture description tasks, explicitly facilitating cross-task comparative analysis to capture task-sensitive behavioral variations. Second, it features the synchronous collection of audio and visual modalities, achieving high-fidelity cross-modal alignment based on strict timestamps. Finally, it transcends the limitations of rigid, score-based thresholding; instead, its diagnostic ground truth relies entirely on a highly reliable, comprehensive clinical consensus formulated by expert neuropsychologists. Ultimately, by providing this exceptionally rich, clinically pure, and task-diverse repository, CogPic offers a fundamentally more robust foundation for next-generation automated cognitive screening. Our contributions are as follows: 1) Dataset Construction: We introduce CogPic, a comprehensive multimodal repository for cognitive assessment featuring a large-scale cohort of 574 participants. By encompassing a broad demographic spectrum across diverse ages, genders, educational backgrounds, and cognitive states, the database provides an exceptionally rich and representative collection of dynamic audio-visual data, establishing a robust foundation for multidimensional behavioral analysis. 2) Baseline Benchmarking: We establish a standardized benchmark for cognitive state classification by evaluating both unimodal and cross-modal architectures. This provides a rigorous baseline for assessing the diagnostic utility of different modeling approaches on the CogPic dataset. 3) Open Resource and Clinical Impact: We open-source CogPic to bridge multimedia research and clinical application. Grounded in expert consensus, it facilitates the discovery of digital biomarkers and the development of non-invasive screening tools, ultimately alleviating the diagnostic burden on healthcare systems and promoting early dementia intervention.
2
TRADITIONAL CLINICAL ASSESSMENT
Inclusion criteria
Task Introduction
Data Collection
Clinical Assessment
Stage 1
Stage 2
Stage 3
Stage 4
~ 1 min
Heavy Burden & Long Duration
AUTOMATED MULTIMODAL SCREENING
Pic. 1
Fast & Objective
< 3 min
Pic. 2 HC
<There's a cookie jar, and the lid is
come down on the floor…>
MCI
Text： off the cookie jar, the boy is about to
visual stimulus
Pic. 3
Label:
Audio：
Video：
~ 30 min
MoCA MMSE ADL …
CONSISTENCY
AD
Figure 1: The overall pipeline of CogPic dataset construction, consisting of four stages: 1). Strict inclusion screening; 2). Task introduction; 3). Synchronized multimodal data collection; and 4). Expert clinical assessment for diagnostic consensus.
2
Related Work
Existing Datasets for Cognitive Assessment The development of automated cognitive assessment tools has been significantly supported by various public datasets, as summarized in Table 1. Foundational repositories, including the widely utilized Pitt Corpus and its derived challenge datasets like ADReSS [5] and ADReSSo [6], have provided crucial audio recordings and text transcripts of the Picture Description task. Similarly, initiatives like Dem@Care [7] and TAUKADIAL [8] have played vital roles in advancing acoustic and linguistic analyses. However, these pioneering resources predominantly focus on English-speaking populations and critically omit the visual modality, restricting diagnostic models to partial behavioral representations. While certain recent efforts have attempted to incorporate visual cues, they often exhibit distinct structural limitations. For instance, the I-CONECT dataset [9] captures both video and audio data, yet it relies on unconstrained conversational tasks rather than standardized elicitation paradigms, alongside a relatively restricted sample size. Within the Chinese linguistic context, available resources remain similarly fragmented. The NCMMSC2021 challenge provided valuable Chinese audio data for picture description, whereas the ADReFV dataset [10] focuses exclusively on the visual modality during human-computer interactions with a limited cohort of 102 subjects. Consequently, a profound gap remains for a large-scale, standardized, and fully multimodal resource tailored to the Chinese community. Table 1: Summary of representative datasets related to cognitive assessment. Database
Subj. Task Modal
Pitt Corpus ADReSS [5] ADReSSo [6] TAUKADIAL [8] Dem@Care [7] I-CONECT [9] NCMMSC2021 ADReFV [10] CogPic
312 204 237 387 32 186 53 102 574
Lang.
P.D. A EN P.D. A, T EN P.D. A EN P.D. A EN / ZH P.D. A EN V.C. A, V EN P.D. A ZH HCI V ZH P.D. A, V, T ZH
P.D.: Picture Description task. HCI: Humancomputer interaction. V.C.: Video Chat, V: Video, A: Audio, T: Text. EN: English, ZH: Chinese.
3
(a)
(b)
(c)
Figure 2: Characteristics of the CogPic dataset: (a) demographic distributions across cognitive cohorts, gender, and education levels; (b) task-specific mean response durations; and (c) high-frequency linguistic tokens visualized via word cloud. 2.1
AI-Driven Cognitive Assessment.
Recent research has increasingly focused on leveraging AI to extract digital biomarkers for early cognitive impairment detection, transitioning from unimodal analysis to sophisticated multimodal frameworks. Significant efforts have been dedicated to identifying acoustic and linguistic markers in spontaneous speech. For instance, Thomas et al. [11] utilized spectral slopes and Mel-frequency cepstral coefficients (MFCCs) to predict neuropsychological scores. Similarly, Liu et al. [12] demonstrated that involuntary prosodic variations during reading tasks could distinguish MCI patients with high precision, highlighting the diagnostic potential of fine-grained acoustic features. Complementary to audio, visual cues offer vital psychomotor information. Burgio et al. [13] explored the neurocognitive correlates of facial recognition, identifying selective impairments in fear recognition and facial memory among individuals with MCI. Furthermore, Alsuhaibani et al. [14] employed convolutional autoencoders and Transformers to capture spatio-temporal facial dynamics from video recordings, achieving significant accuracy in elderly MCI screening. Despite the progress in unimodal methods, they often face limitations regarding feature reliability and susceptibility to individual variations. Consequently, the field is shifting towards multimodal paradigms that integrate visual, auditory, and linguistic streams to provide a holistic assessment [15].
3
CogPic Database
3.1
Data Acquisition
The comprehensive workflow, as illustrated in Fig. 1, began with recruiting community-dwelling elderly individuals. To ensure the clinical purity of CogPic and mitigate confounding variables, we enforced rigorous exclusion criteria: (1) alternative neurological disorders (e.g., stroke, traumatic brain injury, or Parkinson’s disease); (2) unstable psychiatric conditions (e.g., major depression or schizophrenia); (3) severe systemic diseases (e.g., hepatic / renal failure); (4) use of cognitive-impacting medications; and (5) severe communication barriers. Following institutional clearance from the Ethics Committee of Changzhou Second People’s Hospital ([2023]KY323-01), objectives were explained and written informed consent was secured. This procedure ensured all subjects were fully briefed regarding the scientific nature of the study alongside the exact procedures of the audio-visual recordings. A tablet-based system then facilitated the standardized assessment, ensuring strictly synchronized recording of auditory and facial responses during three tasks: “Cookie Theft”, “Picnic Scene”, and “Accident Scene”. Each image was presented sequentially, and subjects were guided to articulate the visual content with maximum detail. To capture genuine multimodal biomarkers, all descriptions were spontaneous and unprompted, followed by a comprehensive clinical evaluation by neurologists to establish the final diagnostic consensus. Brief rest intervals were scheduled to minimize fatigue, and all descriptions remained spontaneous. 3.2
Data Annotation
Establishing highly reliable ground truth labels is paramount for multimodal medical benchmarks. For the CogPic dataset, comprehensive clinical evaluations were meticulously conducted by experienced neurologists to annotate each participant’s cognitive status. These diagnostic procedures systematically incorporated detailed medical histories, physical examinations, and standardized cognitive assessments, specifically the Mini-Mental State Examination (MMSE) [16] and the Montreal Cognitive Assessment (MoCA) [17], alongside rigorous evaluations of Activities of Daily Living (ADL). Based on these holistic clinical profiles, participants were initially stratified to establish a normative baseline. The Healthy Control (HC) group was strictly defined as individuals presenting absolutely no subjective memory 4
complaints and possessing no prior history of major neurological, psychiatric, or metabolic disorders. Conversely, participants exhibiting verified cognitive deficits were assigned to the overarching Cognitive Impairment (CI) cohort for further pathological subtyping. Crucially, the subsequent subdivision into MCI and AD categories was not determined by rigid numerical thresholds, but rather through a comprehensive clinical consensus. While quantitative metrics offer valuable reference ranges, with MCI patients typically scoring between 19 and 25 on the MoCA and early AD patients scoring between 11 and 21, the definitive diagnostic ground truth heavily weighted the individual’s functional independence. Consequently, the definitive MCI label was assigned to individuals experiencing subjective memory decline reported either by the participants themselves or their families. These individuals exhibited objective cognitive deficits but maintained fully intact ADL without a formal dementia diagnosis. In stark contrast, the AD label was designated for individuals strictly meeting the established clinical criteria for dementia. This specific classification necessitated verifiable multi-domain cognitive impairment, a significant and measurable decline in ADL, and persistent neurodegenerative characteristics, frequently accompanied by a MoCA score falling below the clinical reference point of 18. This rigorous, consensus-driven annotation process ensures that the CogPic dataset provides an exceptionally robust foundation for training and evaluating automated diagnostic models. Table 2: Detailed clinical statistics of the CogPic dataset. Group Subj. Sex (M / F)
Age (y)
HC MCI AD Total
62.2 (7.6) 1.4 (0.8) 27.1 (1.2) 68.9 (7.0) 0.8 (0.8) 22.4 (2.0) 69.7 (4.7) 0.2 (0.6) 13.8 (3.3) 67.0 (7.5) 0.9 (0.9) 21.0 (5.7)
178 256 140 574
62 / 116 134 / 122 47 / 93 243/331
Edu.
MoCA
Edu. (Education Level): Categorized as 0: Primary school or below, 1: Junior high school, 2: High school, and 3: University or above.
Table 3: Performance comparison of different models across single modalities (%). † denotes ML-extracted features. UAR ↑ WAR ↑ AUC ↑
Method
w / Acoustic Modality †
LR SVM† XGBoost† LSTM [18] ResNet18 [19] ResNetSE [20] CRNN [21] ViT [22]
58.13 60.38 54.25 49.57 52.14 55.06 54.45 47.56
57.10 58.55 55.65 44.64 47.83 51.59 46.09 51.01
75.42 76.06 74.52 66.87 68.23 70.10 68.25 68.46
w / Visual Modality †
LR SVM† XGBoost† MC3_18 [23] C3D [24] R3D18 [25] ResNet-LSTM [26]
42.33 45.02 35.96 56.16 50.97 54.10 49.22
35.65 38.26 44.35 54.49 49.86 55.65 47.83
55.86 58.06 56.82 74.80 67.77 74.39 72.23
w / Textual Modality †
LR SVM† XGBoost† TextCNN [27] BiLSTM [28] Attention-BiLSTM [29] BERT [30]
49.75 51.17 47.81 51.28 51.69 55.13 49.13
5
46.09 47.25 50.43 52.17 50.43 51.30 49.28
65.73 66.22 65.76 72.68 69.68 71.16 70.30
3.3
Dataset Statistics
Following the rigorous clinical annotation protocol, the finalized CogPic database comprises 574 participants. The detailed demographic distribution and clinical characteristics across the three distinct cognitive states are comprehensively summarized in Table 2. To ensure a robust and transparent evaluation baseline, we meticulously recorded key demographic variables, specifically sex, age, and education level, alongside the quantitative clinical scores derived from the MoCA. Fig. 2 provides a multifaceted visual summary of the dataset characteristics. Specifically, Fig. 2(a) illustrates the hierarchical demographic distribution, explicitly detailing the complex intersections of cognitive status, gender, and educational background. Furthermore, analyzing the temporal dynamics of the clinical sessions reveals distinct behavioral biomarkers. As depicted in Fig. 2(b), the mean response durations vary notably across the three picture description tasks and the diagnostic cohorts. Interestingly, the MCI group consistently exhibits prolonged articulation times compared to both HC and AD patients, potentially reflecting the hesitation and word-retrieval difficulties characteristic of this transitional stage. Finally, Fig. 2(c) presents a comprehensive word cloud, visually capturing the high-frequency semantic vocabulary, including key task-specific nouns and action verbs, spontaneously generated by the participants during the naturalistic elicitations. 3.4
Multimodal Data Processing
To establish a high-fidelity benchmark, we implemented a rigorous processing pipeline for each behavioral stream. Regarding multimodal data processing, the raw audio signals were uniformly resampled to 16 kHz to maintain acoustic consistency. For the textual modality, to mitigate the potential impact of recognition errors on downstream cognitive assessment, we employed the FunASR [31] framework for automated speech-to-text transcription. To empirically validate the reliability of the generated transcripts, we conducted a manual calibration using stratified random sampling, selecting 30 samples from each of the three diagnostic categories. The measured Word Error Rate was 5.86%, demonstrating high phonetic fidelity across the diverse clinical groups. Furthermore, we implemented a standardized normalization protocol for disfluencies (e.g., filler words and pauses) and punctuation to ensure linguistic consistency across the dataset. For the visual modality, the facial video frames extracted utilizing the OpenFace toolkit were center-cropped and resized to a standard resolution of 224 × 224. Finally, to safeguard participant privacy for public release, we enforced a stringent de-identification protocol by redacting all Personally Identifiable Information across modalities, stripping metadata, and assigning randomized identifiers, thereby ensuring strict anonymity while preserving diagnostic fidelity.
4
Baseline Experiments
4.1
Experoimental setup
Data and Protocol. To evaluate the model’s performance at the subject level, the CogPic dataset was partitioned following a strict subject-independent protocol. Participants were randomly assigned to the training and testing sets with an 8:2 ratio. This subject-level partitioning ensures that all samples from a single individual are confined to a specific set, strictly preventing data leakage. Evaluation Metrics. To ensure a holistic and clinically-robust assessment under potential class imbalance, we primarily report Unweighted Average Recall (UAR) to treat each diagnostic cohort with equal importance. This is complemented by Weighted Average Recall (WAR) and Area Under the ROC Curve (AUC) to provide a comprehensive measure of overall accuracy and discriminative power across operating thresholds, establishing a rigorous benchmark for the CogPic dataset. Implementation Details. All baseline architectures were implemented using the PyTorch framework and trained with acceleration on NVIDIA RTX 5090 GPUs. Optimization was performed via the AdamW optimizer, with learning rates meticulously tuned through grid search to ensure optimal convergence. Early stopping was further applied to prevent overfitting. 4.2
Baseline Methods
To establish a rigorous benchmark for CogPic, we implemented a dual-track evaluation framework consisting of two distinct paradigms. Handcrafted Feature Engineering. We employed widely adopted foundational methods to extract clinically interpretable biomarkers across three modalities, providing a robust baseline for diagnostic interpretability. 1) Acoustic: Physical parameters, including fundamental frequency and pause profiles, were extracted via Parselmouth. High-order 6
Table 4: Ablation study of multi-modal combinations (%). † denotes ML-extracted features. UAR ↑ WAR ↑ AUC ↑
Method
w / Acoustic + Visual Modality CRNN + ResNet-LSTM ResNet18 + R3D18 ResNetSE + MC3_18 CRNN + R3D18
50.33 52.95 48.41 51.99
46.38 53.91 49.28 51.30
70.92 71.29 67.77 69.96
51.59 55.94 49.57
72.35 74.36 70.53
48.12 53.91 56.27
69.82 73.40 76.13
w / Acoustic + Textual Modality CRNN + Att-BiLSTM ResNet18 + BERT-base SEResNet50 + BERT-base
54.71 57.17 53.19
w / Visual + Textual Modality ResNet+LSTM + Att-BiLSTM R3D18 + BERT-base MC3_18 + TextCNN
53.09 56.11 55.42
w / Acoustic + Visual + Textual Modality †
LR SVM† XGBoost† CRNN + ResNet+LSTM + Att-BiLSTM ResNet18 + R3D18 + BERT-base ResNetSE + MC3_18 + TextCNN
56.47 57.66 56.44 52.69 54.77 62.16
56.81 55.65 57.97 49.27 54.23 59.77
74.33 77.20 75.62 72.54 72.91 76.61
temporal statistics of MFCCs were further computed using Librosa and SciPy. 2) Visual: We employed OpenFace 2.2 to capture frame-level facial action units and 3D head poses. After data cleaning with Pandas, these signals were aggregated into structured temporal statistics [32]. 3) Linguistic: We utilized Stanza for dependency and constituency parsing to measure syntactic complexity, and Jieba for quantifying lexical richness. Semantic coherence was calculated via sentence-level cosine distances using scikit-learn. These representations were evaluated using traditional classifiers, including Logistic Regression (LR), SVM, and XGBoost [33]. Deep Learning Models. We evaluated a suite of widely adopted, foundational deep learning models to establish a robust representation learning baseline. 1) Acoustic: We modeled Mel-spectrograms using ResNet-18 and CRNN, alongside raw waveform modeling via LSTM. 2) Visual: Video data were standardized into 120-frame tensors and processed using 3D-CNNs, specifically MC3, C3D, and R3D. 3) Linguistic: Textual sequences were evaluated via TextCNN, BiLSTM, and fine-tuned BERT. 4) Multimodal Fusion: Optimized unimodal backbones served as feature encoders for acoustic, visual, and linguistic representations. These embeddings were integrated via feature-level concatenation and processed by a Multi-Layer Perceptron for final diagnostic classification. Table 5: Performance of multimodal methods on different image description tasks (%). Pic. 1
Method
Pic. 2
Pic. 3
UAR ↑ WAR ↑ AUC ↑ UAR ↑ WAR ↑ AUC ↑ UAR ↑ WAR ↑ AUC ↑ CRNN + ResNet-LSTM + Att-BiLSTM 54.51 ResNet18 + R3D18 + BERT-base 52.84 ResNetSE + MC3_18 + TextCNN 63.95
4.3
51.30 53.04 61.74
72.08 71.61 75.87
54.23 55.91 62.37
51.30 53.91 60.00
74.85 74.67 78.28
49.24 55.57 60.14
45.13 55.75 57.52
70.66 72.45 75.97
Unimodal Performance Comparison
The unimodal benchmarking results, as summarized in Table 3, reveal distinct diagnostic strengths across different behavioral streams. In the acoustic modality, traditional Machine Learning (ML) trained on handcrafted features consistently outperforms end-to-end models. Specifically, SVM achieves the highest UAR of 60.38% and AUC of 76.06%, underscoring the efficacy of expert-driven prosodic and pause-related biomarkers over raw spectral 7