Expresso-AI: Explainable Video-Based Deep Learning Models for Depression Diagnosis

2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII)

arXiv:2606.25606v1 [cs.CV] 24 Jun 2026

Expresso-AI: Explainable Video-Based Deep Learning Models for Depression Diagnosis 1st Felipe Moreno

2nd Sharifa Alghowinem

3rd Hae Won Park

4th Cynthia Breazeal

Media Lab MIT Cambridge, USA [email protected]

Media Lab MIT Cambridge, USA [email protected]

Media Lab MIT Cambridge, USA [email protected]

Media Lab MIT Cambridge, USA [email protected]

Abstract—Given the widespread prevalence of depression and its consequential impact on individuals and society, it is crucial to obtain objective measures for early diagnosis and intervention. As a multidisciplinary topic, these objective measures should be interpretable and accessible to health care professionals, ensuring effective collaboration and treatment planning in the realm of mental health care. Even though current automated depression diagnosis approaches have improved over the last decade, a critical gap exists as they often lack affect-specificity and interpretability, limiting their practical application and potential impact on mental health care. In particular, interpretability from temporal activities from videos when deep models are used is not fully explored. In this study, we present a novel framework for analyzing Deep Neural Networks’ decisions when trained on facial videos, specifically focusing on automatic depression severity diagnosis. By fine-tuning Deep Convolutional Neural Networks (DCNN) pre-trained on Action Recognition datasets on depression severity facial videos from the AVEC depression dataset, our framework is able to interpret the model’s saliency maps by examining face regions and temporal expression semantics. Our approach generates both visual and quantitative explanations for the model’s decisions, providing greater insight into its reasoning. In addition to this interpretability, our video-based modeling has improved upon previous single-face benchmarks for visual depression diagnosis, resulting in enhanced predictive performance. Overall, our work demonstrates the successful development of a framework capable of generating hypotheses from a facial model’s decisions while simultaneously improving depression’s predictive capabilities. Index Terms—depression detection, video modelling, deep learning, AI explainability, AI interpretability

I. I NTRODUCTION Depression is a pervasive mental health issue affecting millions of people globally. According to the World Health Organization (WHO), over 264 million people of all ages suffer from depression, making it one of the leading causes of disability worldwide [1]. Current diagnostic methods for depression primarily rely on subjective assessments, such as selfreports and clinical interviews, which may be influenced by factors like personal bias, cultural differences, and subjective interpretations. Given the widespread prevalence of depression and its consequential impact on individuals and society, there is a pressing need for developing objective measures to accurately identify and assess depressive symptoms. Such objective measures should be interpretable and accessible to health care professionals, ensuring effective collaboration and treatment

979-8-3503-2743-4/23/$31.00 ©2023 IEEE

planning in the realm of mental health care. By addressing this need, we can pave the way for improved early diagnosis, intervention, and ultimately, better mental health outcomes for those affected by depression. In an effort to address the subjective nature of depression diagnosis, attempts have been made to employ machine learning methods to develop objective measures for diagnosing depression. These automated methods have analyzed various aspects, including video (e.g., facial expressions, head orientations, eye movements), audio, and brain signals, among others [2]. However, even when investigating the same modality, different studies implement different approaches in terms of extracted features, machine learning algorithms, and reported results, making the comparison and conclusion difficult. Among the studies focusing on automated depression diagnosis (ADD), very few have provided insights or interpretations that could explain the model’s performance to human observers. This lack of interpretability is a common issue in machine learning studies, particularly with the rise of deep learning modeling [3]. Interpretability of depression diagnosis modeling is particularly crucial for communicating its performance to clinicians who are interested not only in understanding the dynamics of depression symptoms, but also to be part of the diagnosis decision made by the model. The current state of video-based modeling for depression diagnosis lags behind image-based advances in both performance and explainability, where the majority of vision models diagnose depression based on images [2]. Video modeling, with its convolution capabilities like 3D CNNs, has the potential to extract features from the dynamics of motion, going beyond appearance-based analysis. Despite these benefits, existing explainability literature barely addresses the time dimension in their algorithms, with initial approaches mainly focusing on visualizations and heat-maps for quantitative analysis. Therefore, we developed a framework1 to better understand and explain decisions made by video-based deep learning models, specifically in the context of ADD. By training deep neural networks on facial video clips and analyzing sample attributions in relation to facial expressions and other self-interpretable lower-level face semantics, we aim 1 Code is available at https://github.com/felmoreno1726/Expresso-AI.

to identify quantifiable interpretation methods for video-based architectures by correlating these attributions with depression severity level, facial expression and facial regions. II. R ELATED W ORK Interpretability of machine learning models focuses on approaches like prediction visualization, justification, and interpretable models, with Explainable AI (XAI) techniques aiming for human-like explainability [3], [4]. Visualizing parts of an input contributing to predictions in deep learning architectures has gained attention for interpretability [5]–[7], but instance group visualization, especially for video frame sequences, has received less attention [8], [9]. Attribution-based visualization aids in model evaluation, despite current methods like instant inspection and ground truth evaluation being subjective due to their reliance on human observation [10]. ADD has gained interest among researchers for its potential to objectively identify and assess depressive symptoms, especially with the emergence of deep learning techniques and various data types (images, videos, and audio). Various studies have used complex methods like extracting local binary patterns, bag-of-words features, and eye blink indicators to model depression severity and classify patients [11]. However, interpreting these approaches is difficult due to the complexity of processing the features and the lack of further analysis to quantify them. Deep learning techniques outperform traditional methods but are complex to interpret. Various neural network architectures, such as RNN [12], LSTM [13], C-CNN [14], multi-scale temporal dilated CNN [15], DCNN [16], hybrid deep CNN [17], DepArt-Net [18], and 3D-CNN [19], have been employed for depression modeling, making interpretation challenging. Most ADD studies do not provide direct interpretation of their model performance, with only a few attempting to explain performance through statistical analysis or by linking with psychology literature [20]. For instance, [21] modeled modalities and groups of features individually, finding that speech behaviors had higher performance compared to language and visual cues. In [22], histograms were used to analyze facial landmarks and 3D head motion features, revealing that velocity and acceleration of facial movement strongly mapped onto depression severity. The DepressNet model by [23] visualized activation maps, showing the eye region to be the most indicative of depression severity, but lacked clarity on facial movement dynamics. The study by [24] used attention maps in a ResNet-50 architecture to show differences in depression levels from the face, but did not directly explain the contributing dynamics and features. Lastly, [25] used CNN to model depression severity with handcrafted features, indicating action units around the eyes and mouth as the most indicative of depression levels, but did not provide quantified conclusions or deeper insights into model performance. In our previous work, we made an effort to align our traditional depression detection models with statistical analysis of the extracted features and with cross-cultural generalizations in order to provide insight into the model performances and to

link the machine learning literature with psychology literature [26]. However, these efforts do not provide direct insight into the model performance. In this work, we aim to explain and interpret the predictions of ADD models by correlating the results with the predictions of less complex and more intuitive models as ground truths. These models include those that detect facial keypoints and facial expressions. Facial landmarks are predicted as heatmaps of the most likely region of localization, making them visually interpretable by default. The Action Unit model employs simple SVM and SVR modeling to detect expressions and calculate their intensity from facial appearance and geometric features. III. E XPERIMENTAL S ETTINGS A. Dataset We used the benchmark Depression Recognition Subchallenge of the Audio/Visual Emotion Challenge (AVEC) 2014 [27] for modeling depression for interpretation. The dataset consists of video recordings of 150 participants while performing two tasks: reading aloud and responding to questions in a freeform manner, such as discussing a favorite dish or a childhood memory. Though the AVEC (DAIC-Woz) 2019 dataset [28] provides audio-visual recordings, it has not been extensively utilized for visualization and interpretation. Consequently, we employ the AVEC 2014 dataset for comparisons with previous literature. The dataset contains 300 video recordings, provided as separate videos for each participant, and divided into three dataset splits: Training, Development, and Testing. In our experiments, we merge the Training and Development sets for use as training data, and report performance based on the evaluation of the Testing set. The videos, recorded at variable frame rates and resolutions, were later re-sampled to 30 frames per second (fps) and 640x480 pixels. The data labels correspond to the Beck Depression Inventory-II (BDI-II), a frequently used self-reported metric consisting of 21 multiple-choice questions. B. Data Preprocessing For each video, the face is detected using the Single Shot Scale-Invariant Face Detector (S3FD or SFD) [29], a CNN-based detector capable of detecting faces at multiple resolutions and scales. Frames with unreliable face detection (confidence score < 0.75) are skipped. The detected face is fed to the Face Alignment Network (FAN) [30], which extracts facial landmarks for each frame. The tightest square around the face is cropped out without alignment. The crops of each individual region are up-sampled using LANCZOS4 interpolation, a robust method for small-scale crops [31]. The dimensions are up-sampled to either 112x112 or 150x150, depending on the input model. Then, crops of each individual region are concatenated together into a video. A sample is taken from the pre-processed data. In our experiments, the video clips fed into the model have lengths of 16, 32, or 64 frames. The frames of the clip cover a window

size of 256 or 512 frames. We perform input dilation, i.e., one frame is taken from every 8 or 16 frames. The clips overlap with each other, covering 40-60% of the window. We perform regularization by applying horizontal flips to training samples with a probability of 0.5. C. Network Architectures Since we wanted to conduct the interpretation framework on the highest performing network architectures, we explored several ones. ResNet, or Residual Network, is a popular choice for various computer vision tasks due to its strong performance and ability to efficiently handle deep architectures. Therefore, in this work, we opt for ResNet-type architectures fine-tuned on Action Recognition datasets, with different variations as follows: • ResNet 3D (R3D). Deep Residual Networks (ResNet) with 3D convolution filters. Pre-trained on Kinetics-400, Kinetics-700, and/or Moments in Time (MiT) datasets. • 2-Plus-1 Dimensions ((2+1)D). Deep Residual Network which factorizes spatio-temporal (3D) convolution into a 2D spatial convolution and a 1D temporal convolution. Models were pre-trained on Kinetics-400, Kinetics-700, MiT, and/or Sports-1M. These pre-trained networks had these parameters: • Loss Function. We use a L2 loss, also known as Mean Squarred Error (MSE) loss. This loss is fitting to the regression problem. • Optimizer. We use Stochastic Gradient Descent (SGD) with momentum and weight decay. • Learning Rate Scheduling. We use Cosine Annealing. This in combination with SGD, is called SGD With Warm Restarts. • Freezer. We developed an object called Freezer, which takes in a model, freezes all but the last layer, and progressively unfreezes the layers of the model top to bottom. Research shows that when fine-tuning transfer learning models, it is often desirable to train the top layers longer than the bottom layers. • Gradient Clipping. To avoid exploding and vanishing gradient problems as well as increasing regularization, we clip gradients during backpropagation to a gradient norm of 1.0. • Regularization Noise. To reduce overfitting, we apply random Gaussian noise to the target depression level during the training routine. The noise applied has mean 0, and standard deviation 0.5, 1.0, 1.5 or 2.0. D. Evaluation To measure the model’s recognition performance, two evaluation metrics are used: Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE). These metrics were selected for comparisons with prior literature on this dataset. The MAE and RMSE are calculated for each video sample in the dataset using the following equations: These evaluation metrics help quantify the model’s performance by measuring the difference between the predicted depression scores and the actual (ground

truth) depression scores for each video sample in the dataset. The predicted depression score for a video is determined by averaging the predicted scores of all its segmented clips. IV. I NTERPRETATION OF THE ADD M ODEL Early efforts in machine learning interpretability have concentrated on visualizations. Past research has demonstrated the effectiveness of occlusion techniques, activation-based methods, and backpropagation methods when applied to images. However, when it comes to videos, the majority of visualizations have employed activation-based methods (e.g. 2DCNNs [32]), with only a few occlusion methods (e.g. SpatioTemporal Extremal Perturbation (STEP) [33]) and hardly any backpropagation methods (e.g. DeepLift [34]). More sophisticated techniques might be capable of quantifying model decisions beyond just offering qualitative visual explanations. We suggest that the fundamental properties of backpropagation methods such as DeepLift could provide satisfactory explanations that go beyond the visual domain and encompass the temporal aspect of explanations as well. We use the DeepLift algorithm with Rescale Rule to compute attribution maps, which is a computationally efficient method with a conservation property. This method may provide explanations in the temporal realm, as well as the visual realm, for depression modeling, as explained in the following subsections. A. Attribution Generation The DeepLift algorithm is a model-agnostic interpretability method used to explain the predictions of deep learning models. To identify which features contribute most to the model’s predictions, DeepLift assigns relevance scores to each input feature, which represent their contribution to the model’s output [34]. It compares the activations of each neuron to a reference activation, which is typically chosen as the activation that occurs when the input is a neutral or baseline input. The Rescale Rule is an essential part of the DeepLift algorithm, where it distributes the relevance scores across the input features while maintaining the conservation property. The Rescale Rule ensures that this property holds true while rescaling the relevance scores to maintain their relative importance. That is, the sum of the relevance scores of all input features is equal to the model’s output. In our research, we utilize the DeepLift algorithm combined with the Rescale Rule to calculate attribution maps. We selected this technique due to its low computational cost and the conservation property, mentioned above. The highest relevance in the algorithm’s back-propagation process is the model’s prediction score, which is then propagated all the way down to the model’s inputs, generating relevance (or attribution) maps that satisfy the following equation: f (x) =

d X

Rp,t

(1)

p,t

In this equation, f represents the function computed by the model, f (x) is the output generated based on input x

and Rp,t denotes the relevance of pixel p in frame t. We presume that other algorithms that fulfill the conservation property, such as Layer-wise Relevance Propagation (LRP) or Integrated Gradients, will provide interpretations similar to those obtained using DeepLift. In this context, the DeepLift algorithm is applied to a 50layer R(2+1)D model, which is a type of deep learning model used for video recognition tasks. For each sample in the testing set, attributions are computed using DeepLift, resulting in an attribution map with dimensions (T, C, W, H) = (16, 3, 112, 112) for each sample, where T is the time axis; C is channels (RGB); W is width; and H is height. By applying the DeepLift algorithm to the R(2+1)D model, we can generate attribution maps for each video sample that reveal the importance of each pixel in the video frames (across time and color channels) to the model’s predictions. This aids in comprehending and interpreting the model’s decisionmaking process. B. Relevance Pooling For model interpretability, Relevance Pooling aims to group input features into broader clusters based on their relevance scores or importance to the model’s predictions. By focusing on these coarser groups of input features rather than individual spatio-temporal pixels, it concentrates on more significant patterns in the input data. Our analysis focuses on clustering relevance based on the conservation property, as shown in the equation 1. Instead of examining every single input feature (spatio-temporal pixel), we are more interested in broader groups of input features. The conservation property over the input space can be represented by the following equation: XX X X f (x) ≈ Rt,c,w,h (2) t∈T c∈C w∈W h∈H

In this equation, Rt,c,w,h represents the relevance of each pixel in the video frames (across time and color channels) to the model’s predictions. C. Video Attribution Maps Relevance pooling is first performed along the channels axis (RGB color channels). For each attribution map, the relevance scores of the RGB channels of each spatio-temporal pixel are added together. As a result, a group of feature sets with dimensions (16, 112, 112) is obtained for each input. For the video, these aggregated feature sets are then used for heatmap visualizations. Heatmaps help to visually represent and identify the most important areas in the input data (in this case, video frames) that significantly contribute to the model’s predictions. This is performed using the following equation: XX X X f (x) ≈ Rt,c,w,h

t∈T w∈W h∈H c∈C

=

X X X t∈T w∈W h∈H

C Rt,w,h

The goal is to create a vector of temporal attributions (RT ) that simplifies the analysis of the model’s behavior by focusing on the most critical patterns in the input data over time. By pooling the relevance for each frame into a single attribution score, we focus the interpretation on the most important temporal patterns in the input data. This vector of temporal attributions RT satisfies: X X X X f (x) ≈ ( Rt,c,w,h ) t∈T c∈C w∈W h∈H

=

X

RtT

(4)

t∈T

E. Region-wise Attributions To objectively interpret the model in regard to facial parts, we calculated attributions (relevance scores) around specific regions of interest in the face, such as mouth, right eyebrow, left eyebrow, right eye, left eye, nose, jaw, besides the whole face. These regions are identified using pre-computed facial landmarks. In this approach for pooling relevance, the conservation property is not satisfied. However, in this regionwise analysis, attributions outside the regions of interest are discarded, resulting in the conservation property not being maintained. F. Video Heat Maps Visualizations Video heat maps visualizations highlight the most relevant regions in a video by normalizing across the spatio-temporal volume of the video. These video heat map visualizations could be used to identify important spatio-temporal features in videos of individuals being assessed for depression, which allow for a better understanding and interpretation of the modeling process. As discussed, this could potentially lead to improved models for detecting and diagnosing depression, as well as insights into the specific visual cues that the models are using to make their predictions. To our knowledge, no previous work has attempted to visualize back-propagation-based attribution maps for video modeling. Interestingly, we found that normalizing across the spatio-temporal volume produced very smooth maps of relevant regions. This is quite different from image visualizations, which are known to be highly pixelated in areas of interest. For each video clip attributions, we normalize across the spatio-temporal volume using a cumulative sum that maps 98% of the attributions to the desired range: positive (0, 1), negative (-1, 0), all (-1, +1), or absolute value (0, 1). We let 2% of the outliers fall outside the range. We present visualizations of the absolute value attributions normalized to the range (0, 1). These visualizations are designed to display the most important regions in brighter colors. G. Image Grids Visualizations

t∈T c∈C w∈W h∈H

X X X X = ( Rt,c,w,h )

D. Temporal Attributions

(3)

Image grids model visualizations identify and analyze the most important video frames that contribute to the modeling results. In this method, video-clip frames with the highest and lowest attributions in RT are extracted. The highest

attributions represent the frames that contributed the most to a depression diagnosis, whereas the lowest attributions represent frames that contributed the most to a non-depressed diagnosis. By visualizing the individual attributions for each frame, we can gain insights into the specific features and patterns that the depression model is using to make its predictions. These visualizations are called saliency maps, and they tend to be highly pixelated, highlighting the most relevant pixels in each frame. This method would derive the factors that the ADD model is using to diagnose depression, where visual cues are analyzed from the most influential frames. Moreover, this method can also help identify potential biases or issues in the model, ensuring that the model’s predictions are reliable and accurate. H. Action Unit Cross Correlation Analysis To provide an objective and interpretable measure of our model’s performance, we analyzed the relationship between Action Units (AUs) and the model’s Temporal Attributions (TAs) signal in the context of depression recognition. AUs are facial muscle movements that can be used to describe facial expressions. Therefore, correlation analysis of AUs and the model’s TAs would provide insight into the model’s performance. Action Units (AUs) are extracted from each of the AVEC 2014 original recordings using the OpenFace analysis toolkit [35]. AUs are extracted using dynamic normalization, which post-calibrates the predictions for each video by finding the person’s median facial expression to correct over- and under-prediction of AUs [36]. To quantify the relationship between the temporal presence of AUs and the model’s TAs, the Kendall Tau correlation coefficient is calculated. This coefficient measures the correlation between two variables, with scores ranging from -1 (negative correlation) to +1 (positive correlation). Tau is scale-invariant, making it suitable for comparing attributions across different depression classes. However, due to the short duration of the video clips (16 frames), the correlation coefficients calculated for individual clips might not be reliable. An AU may be present throughout the entire clip or not present at all, leading to no apparent correlation between the AU and the model’s decision. To address this issue, the AU cross-correlation scores are averaged across all clusters, providing a more comprehensive understanding of the relationship between AUs and the model’s decision-making process, considering the relatively short duration of the clips. V. R ESULTS A. Architecture Selection Results To select the best performing model architecture for predicting depression scores in videos, we fine-tune a few ResNet architectures trained in different datasets as backbone, with different depth levels (see Table I). The models developed in this study outperform previous single-face ADD results, with the current benchmark held by the DepressNet-Full model. Based on the results from Table I, R(2+1)D trained on both Kinetics-700 and MiT produced the highest results. Therefore,

TABLE I: Performance Comparison of single face ADD in the AVEC 2014 Dataset with our fine-tuning results. Method Baseline [27] UUIMSidorov [37] InaoeBuap [38] Brunnel [39] Zhu [40] BU-CMPE [41] Jan [42] DepressNet-Full [23] Ours R3D R(2+1)D Wide ResNet R3D R3D R3D R(2+1)D R(2+1)D

MAE 8.86 11.20 9.35 8.44 7.47 7.96 6.68 6.60

RMSE 10.86 13.87 11.91 10.50 9.55 9.97 8.04 8.88

7.430 6.820 6.984 7.228 6.999 6.700 7.027 6.626

9.012 8.640 8.529 9.278 8.903 8.694 8.687 8.543

Pre-train Dataset Kinetics-400 Kinetics-400 Kinetics-400 Kinetics-400 Kinetics-700 Kinetics-700 + MiT Kinetics-700 Kinetics-700 + MiT

Depth 18 18 50 101 101 101 50 50

interpretation analyses of the R(2+1)D model would provide more accurate and reliable predictions, paving the way for improved understanding and treatment of depression. B. Interpretation and Visualization Fig. 1 presents examples of attributions and heat maps from the top contributing frames to the model’s results when it correctly predicts depression severity levels. It can be observed that the model follows specific patterns in its diagnosis. For instance, the regions of the forehead exhibit higher attributions in severe and moderate depression levels compared to the minimal level. This suggests that tense frowns and eyebrow movements have more significant attributions in these cases. Similarly, the mouth regions display higher activations in minimal depression levels than in moderate and severe levels. However, a notable issue is the high attribution scores for headsets and microphones in the video visualizations. This is likely due to a bias in the training set, where participants with headsets and microphones are more inclined towards depressive labels. This bias adversely impacts the dataset’s generalization and affects the model’s performance. These qualitative observations call for a more objective and reliable assessment and interpretation of the model’s performance. To objectively assess these attributions, a crosscorrelation analysis is performed with specific regions and Action Units (AUs) (see Fig. 2). This analysis helps to better understand the relationship between facial expressions, as represented by AUs, and the model’s attributions, providing a more accurate and reliable interpretation of the decisionmaking process. C. Action Unit Cross Correlation Analysis Examining the Action Unit correlation clusters provides insights into the AUs that are positively correlated with each depression severity group. Focusing on the correctly predicted results, we observe that AU9 (nose wrinkling) is an indicator of depression severity and is most correlated with severe

(a) Top attribution frames of severely depressed

Fig. 2: Cluster analysis of action unit cross correlations on correctly predicted samples. (b) Top attribution frames of moderately depressed

(c) Top attribution frames of minimally depressed

Fig. 1: Visualization of the model’s top attribution frames, showing examples of correctly estimated results from different level of depression severity. Fig. 3: Region-wise analysis cluster of correctly predicted samples. depression while also being present in mild and moderate clusters. It is also worth noting that AU4 (brow furrowing) is correlated with the moderate depression cluster, suggesting a connection between frowning and depression severity. These results are aligned with depression facial expression literature, where depressed individuals express anger (AU4) and disgust (AU9) [2]. Interestingly, the AUs most negatively correlated with depression are AU6 (cheek raising) and AU12 (lip corner pulling), which, when combined, form a Duchenne Smile (or authentic smile) and represent joy or happiness. This finding is supported by the literature, and therefore, confirms that the model was able to find the association between specific facial expressions and different depression severity levels, highlighting the importance of understanding and interpreting the decision-making process of the model. By identifying these correlations between AUs and depression severity, the research validates the model’s ability to recognize patterns in facial expressions that are indicative of various depression levels. This understanding can contribute to improving the model’s performance and generalization, ultimately leading to more accurate, interpretable and reliable

detection and diagnosis of depression. D. Clustering Analysis The region-wise analysis findings are summarized in Fig. 3. This analysis reveals the importance of specific regions to the model’s prediction decisions, which would provide insights and interpretation of the model. Examining the mouth attributions, we observe that they tend to be high in correctly labeled depressed (moderate and severe) participants, but low in those misclassified (underestimated). We speculate that this lower attribution may be due to participants having high vocal expressivity, which could impact the model’s interpretation of their depression severity. On the other hand, correctly classified non-depressed (minimal or mild) participants tend to have low nose attributions, but when incorrectly labeled (overestimated) these tend to have high nose attributions. We speculate that this may be attributed to nose expressions such as AU09 (nose wrinkling), which could be associated with higher depression severity by the model.

Likewise, a positive correlation exists between attributions in the left and right eye regions and moderate to severe depression levels, while a negative correlation is observed in mild and minimal depression levels. The positive correlation of attributions in the eye regions with moderate and severe depression levels suggests that specific eye-related expressions or movements could be indicative of higher depression severity. These expressions may include reduced eye contact, increased blinking rate, or gaze aversion, which are known to be associated with depression [43]. On the other hand, the negative correlation of attributions in the eye regions with mild and minimal depression levels implies that different eye-related expressions or behaviors might be more prevalent in individuals with lower depression severity. These behaviors could include more consistent eye contact, more natural blinking rates, or more engaged gaze patterns that are associated with non-depressed or less depressed individuals. These findings further validate the main research question by demonstrating the importance of understanding and interpreting the decision-making process of the model. Identifying the significance of specific facial regions and their attributions can contribute to improving the model’s performance and generalization. By recognizing patterns in facial expressions and their corresponding attributions, the model can make more accurate and reliable predictions, ultimately leading to better detection and diagnosis of depression. VI. C ONCLUSION In this work, we aimed to modernize visual Automatic Depression Diagnosis (ADD) by employing the latest techniques and architectures in video modeling. Our findings demonstrated that video-based models outperformed singleimage-based modeling for ADD tasks. We identified the best video architectures for the task and leveraged pre-trained models on large-scale video datasets to enhance our model’s performance. As a result, we developed a more optimal model for single-face ADD. While exploring video interpretation, we recognized the potential for innovation in using a back-propagation saliency map method, allowing us to quantitatively assess both temporal and regional importance. We performed face segmentation in regions of interest, such as eyes, eyebrows, mouth, and nose, and calculated the associated relevance to better understand the decisions made by ADD models. Additionally, we expanded our understanding to the temporal frame by analyzing attributions in the context of Action Units as frame semantics. A preliminary correlation analysis found associations between severe depression and expressions of disgust and contempt, as well as a negative correlation between severe depression and expressions of joy. Among other findings, we identified talkativeness and mouth expressiveness as indicators of nonsevere depression. We were intrigued to discover significant relevance in the nose area, which may indicate the importance of nose-wrinkling, an expression of disgust, in detecting de-

pression. These findings are consistent with each other and with the clinical depression literature. Future work on the face interpretation framework should focus on a careful statistical analysis of the findings. In this work, we are pleased to have generated hypotheses on face expressions and regions, which may be validated a posteriori with scientific research. However, determining statistical significance may show that the framework’s findings are selfvalidating. E THICAL I MPACT S TATEMENT The research presented in this work aimed to advance the field of ADD by employing state-of-the-art techniques and architectures in video modeling. While the development of more accurate and reliable ADD models has the potential to improve the detection and diagnosis of depression, it also raises several ethical considerations that must be addressed. 1. Bias and Fairness: In this research, we found high attribution scores for headsets and microphones in the video visualizations, which could be an artifact of the training set. To ensure fair and unbiased predictions, future work should focus on addressing these biases and examining the model’s performance across diverse populations and settings. 2. Transparency and Interpretability: In this work, we presented several methods to analyze the model’s performance and interpretability, such as Relevance Pooling, Video Heat Maps Visualizations, and Region-wise Attributions. Ensuring transparency and interpretability allows for a better understanding of the decision-making process and can help identify potential biases or issues that may need to be addressed. 3. Clinical Validation and Application: Before deploying ADD models in real-world settings, it is crucial to validate their performance and accuracy through clinical trials and evaluations. Ensuring that the models are reliable and accurate can prevent potential misdiagnoses and unnecessary distress for patients. Additionally, it is essential to consider how these models would be integrated into existing clinical practices and to provide adequate training and support for healthcare professionals using these tools. R EFERENCES [1] WHO-Depression. (2020) Depression (who fact sheet), world health organization. [Online]. Available: https://www.who.int/news-room/ fact-sheets/detail/depression [2] A. Pampouchidou, P. Simos, K. Marias, F. Meriaudeau, F. Yang, M. Pediaditis, and M. Tsiknakis, “Automatic assessment of depression based on visual cues: A systematic review,” IEEE Transactions on Affective Computing, 2017. [3] T. Miller, “Explanation in artificial intelligence: Insights from the social sciences,” Artificial Intelligence, vol. 267, pp. 1–38, 2019. [4] O. Biran and C. Cotton, “Explanation and justification in machine learning: A survey,” in IJCAI-17 workshop on explainable AI (XAI), vol. 8, no. 1, 2017. [5] F. Hohman, M. Kahng, R. Pienta, and D. H. Chau, “Visual analytics in deep learning: An interrogative survey for the next frontiers,” IEEE transactions on visualization and computer graphics, vol. 25, no. 8, pp. 2674–2693, 2018. [6] A. Nguyen, J. Yosinski, and J. Clune, “Understanding neural networks via feature visualization: A survey,” in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer, 2019, pp. 55–76.

[7] R. Fong and A. Vedaldi, “Explanations for attributing deep neural network predictions,” in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer, 2019, pp. 149–167. [8] Z. Li, W. Wang, Z. Li, Y. Huang, and Y. Sato, “A comprehensive study on visual explanations for spatio-temporal networks,” arXiv preprint arXiv:2005.00375, 2020. [9] X. Li, W. Guo, and H. Yang, “Depression severity prediction from facial expression based on the drr depressionnet network,” in 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2020, pp. 2757–2764. [10] Z. Bylinskii, T. Judd, A. Oliva, A. Torralba, and F. Durand, “What do different evaluation metrics tell us about saliency models?” IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 3, pp. 740–757, 2018. [11] S. Al-gawwam and M. Benaissa, “Depression detection from eye blink features,” in 2018 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT). IEEE, 2018, pp. 388–392. [12] Y. Suhara, Y. Xu, and A. S. Pentland, “Deepmood: Forecasting depressed mood based on self-reported histories via recurrent neural networks,” in Proceedings of the 26th International Conference on World Wide Web, ser. WWW ’17. Republic and Canton of Geneva, Switzerland: International World Wide Web Conferences Steering Committee, 2017, pp. 715–724. [13] T. Alhanai, M. Ghassemi, and J. Glass, “Detecting depression with audio/text sequence modeling of interviews,” Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 2018-September, pp. 1716–1720, 2018. [14] A. Haque, M. Guo, A. S. Miner, and L. Fei-Fei, “Measuring depression symptom severity from spoken language and 3d facial expressions,” arXiv preprint arXiv:1811.08592, 2018. [15] S. Yin, C. Liang, H. Ding, and S. Wang, “A multi-modal hierarchical recurrent neural network for depression detection,” in Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, 2019, pp. 65–71. [16] Y. Zhu, Y. Shang, Z. Shao, and G. Guo, “Automated depression diagnosis based on deep networks to encode facial appearance and dynamics,” IEEE Transactions on Affective Computing, vol. 9, no. 4, pp. 578–584, Oct 2018. [17] L. Yang, D. Jiang, and H. Sahli, “Integrating deep and shallow models for multi-modal depression analysis—hybrid architectures,” IEEE Transactions on Affective Computing, 2018. [18] Z. Du, W. Li, D. Huang, and Y. Wang, “Encoding visual behaviors with attentive temporal convolution for depression prediction,” in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 2019, pp. 1–7. [19] W. C. de Meto, E. Granger, and M. B. Lopez, “Encoding temporal information for automatic depression recognition from facial analysis,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 1080–1084. [20] J. F. Cohn, N. Cummins, J. Epps, R. Goecke, J. Joshi, and S. Scherer, “Multimodal assessment of depression from behavioral signals,” in The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition-Volume 2, 2018, pp. 375–417. [21] E. A. Stepanov, S. Lathuiliere, S. A. Chowdhury, A. Ghosh, R.-L. Vieriu, N. Sebe, and G. Riccardi, “Depression severity estimation from multiple modalities,” in 2018 IEEE 20th International Conference on e-Health Networking, Applications and Services (Healthcom). IEEE, 2018, pp. 1–6. [22] K. Anis, H. Zakia, D. Mohamed, and C. Jeffrey, “Detecting depression severity by interpretable representations of motion dynamics,” in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018, pp. 739–745. [23] X. Zhou, K. Jin, Y. Shang, and G. Guo, “Visually interpretable representation learning for depression recognition from facial images,” IEEE Transactions on Affective Computing, 2018. [24] W. C. De Melo, E. Granger, and A. Hadid, “Depression detection based on deep distribution learning,” in 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019, pp. 4544–4548. [25] S. Song, S. Jaiswal, L. Shen, and M. Valstar, “Spectral representation of behaviour primitives for depression analysis,” IEEE Transactions on Affective Computing, 2020. [26] S. Alghowinem, R. Goecke, J. Cohn, M. Wagner, G. Parker, and M. Breakspear, “Cross-cultural detection of depression from nonverbal

behaviour,” 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, FG 2015, 2015. [27] M. Valstar, B. Schuller, K. Smith, T. Almaev, F. Eyben, J. Krajewski, R. Cowie, and M. Pantic, “Avec 2014: 3d dimensional affect and depression recognition challenge,” in Proceedings of the 4th international workshop on audio/visual emotion challenge, 2014, pp. 3–10. [28] F. Ringeval, B. Schuller, M. Valstar, N. Cummins, R. Cowie, L. Tavabi, M. Schmitt, S. Alisamir, S. Amiriparian, E.-M. Messner et al., “Avec 2019 workshop and challenge: state-of-mind, detecting depression with ai, and cross-cultural affect recognition,” in Proceedings of the 9th International on Audio/visual Emotion Challenge and Workshop, 2019, pp. 3–12. [29] S. Zhang, X. Zhu, Z. Lei, H. Shi, X. Wang, and S. Z. Li, “S3fd: Single shot scale-invariant face detector,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 192–201. [30] A. Bulat and G. Tzimiropoulos, “How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks),” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1021–1030. [31] K. Turkowski, “Turkowski filters for common resampling tasks 10 april 1990 filters for common resampling tasks.” [32] C. Gan, N. Wang, Y. Yang, D.-Y. Yeung, and A. G. Hauptmann, “Devnet: A deep event network for multimedia event detection and evidence recounting,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 2568–2577. [33] Z. Li, W. Wang, Z. Li, Y. Huang, and Y. Sato, “Towards visually explaining video understanding networks with perturbation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1120–1129. [34] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek, “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation,” PloS one, vol. 10, no. 7, p. e0130140, 2015. [35] T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L.-P. Morency, “Openface 2.0: Facial behavior analysis toolkit,” in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018, pp. 59–66. [36] T. Baltrušaitis, M. Mahmoud, and P. Robinson, “Cross-dataset learning and person-specific normalisation for automatic action unit detection,” in 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 6. IEEE, 2015, pp. 1–6. [37] M. Sidorov and W. Minker, “Emotion recognition and depression diagnosis by acoustic and visual features: A multimodal approach,” in Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, 2014, pp. 81–86. [38] H. Pérez Espinosa, H. J. Escalante, L. Villaseñor-Pineda, M. Montesy Gómez, D. Pinto-Avedaño, and V. Reyez-Meza, “Fusing affective dimensions and audio-visual features from segmented video for depression recognition: Inaoe-buap’s participation at avec’14 challenge,” in Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, 2014, pp. 49–55. [39] A. Jan, H. Meng, Y. F. A. Gaus, F. Zhang, and S. Turabzadeh, “Automatic depression scale prediction using facial expression dynamics and regression,” in Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, 2014, pp. 73–80. [40] Y. Zhu, Y. Shang, Z. Shao, and G. Guo, “Automated depression diagnosis based on deep networks to encode facial appearance and dynamics,” IEEE Transactions on Affective Computing, vol. 9, no. 4, pp. 578–584, 2017. [41] H. Kaya, F. Çilli, and A. A. Salah, “Ensemble cca for continuous emotion prediction,” in Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, 2014, pp. 19–26. [42] A. Jan, H. Meng, Y. F. B. A. Gaus, and F. Zhang, “Artificial intelligent system for automatic depression level analysis through visual and vocal expressions,” IEEE Transactions on Cognitive and Developmental Systems, vol. 10, no. 3, pp. 668–680, 2017. [43] S. Alghowinem, R. Goecke, M. Wagner, J. Epps, M. Hyett, G. Parker, and M. Breakspear, “Multimodal depression detection: Fusion analysis of paralinguistic, head pose and eye gaze behaviors,” IEEE Transactions on Affective Computing, vol. 9, no. 4, pp. 478–490, Oct 2018.