Multimodal mental health analysis in social media

Yazdavar, Amir Hossein; Mahdavinejad, Mohammad Saeid; Bajaj, Goonmeet; Romine, William L.; Sheth, Amit P.; Monadjemi, Amir Hassan; Thirunarayan, Krishnaprasad; Meddar, John M.; Myers, Annie; Pathak, Jyotishman; Hitzler, Pascal

doi:10.1371/journal.pone.0226248

Cited by 77 publications

(47 citation statements)

References 80 publications

(68 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…FER is easily accessible to researchers of all fields and is increasingly used by the scientific community. Applications can be found, for example, in psychology, where such algorithms are used to predict mental health from social media images (Yazdavar et al, 2020), to validate interventions for autism (Wu et al, 2019), or to screen for Parkinson's disease (Jin et al, 2020). A sociological example is the assessment of collective happiness in society from social media images (Abdullah et al, 2015).…”

Section: Introductionmentioning

confidence: 99%

Automatic Facial Expression Recognition in Standardized and Non-standardized Emotional Expressions

2021

View full text Add to dashboard Cite

Emotional facial expressions can inform researchers about an individual's emotional state. Recent technological advances open up new avenues to automatic Facial Expression Recognition (FER). Based on machine learning, such technology can tremendously increase the amount of processed data. FER is now easily accessible and has been validated for the classification of standardized prototypical facial expressions. However, applicability to more naturalistic facial expressions still remains uncertain. Hence, we test and compare performance of three different FER systems (Azure Face API, Microsoft; Face++, Megvii Technology; FaceReader, Noldus Information Technology) with human emotion recognition (A) for standardized posed facial expressions (from prototypical inventories) and (B) for non-standardized acted facial expressions (extracted from emotional movie scenes). For the standardized images, all three systems classify basic emotions accurately (FaceReader is most accurate) and they are mostly on par with human raters. For the non-standardized stimuli, performance drops remarkably for all three systems, but Azure still performs similarly to humans. In addition, all systems and humans alike tend to misclassify some of the non-standardized emotional facial expressions as neutral. In sum, emotion recognition by automated facial expression recognition can be an attractive alternative to human emotion recognition for standardized and non-standardized emotional facial expressions. However, we also found limitations in accuracy for specific facial expressions; clearly there is need for thorough empirical evaluation to guide future developments in computer vision of emotional facial expressions.

show abstract

Section: Introductionmentioning

confidence: 99%

Automatic Facial Expression Recognition in Standardized and Non-standardized Emotional Expressions

2021

View full text Add to dashboard Cite

show abstract

“…They also identified a tendency for the MD group to post bluer and more yellow photos. Yazdavar et al (59) reported that profile pictures of Twitter users with self-disclosed depression had a more gray, less saturated, and overall less appealing color characteristics compared to HVs. Along similar lines, Reece et al (19) showed that individuals with depression posted images on Instagram that were lower in 1 For the posting behavior throughout the day, we chose 4 time windows in accordance with De Choudhury et al, (18) color saturation and darker and had a higher hue (toward blue).…”

Section: Characteristics Of Image Posts On Instagrammentioning

confidence: 99%

Utilizing Instagram Data to Identify Usage Patterns Associated With Schizophrenia Spectrum Disorders

et al. 2021

View full text Add to dashboard Cite

Background and Objectives: Prior research has successfully identified linguistic and behavioral patterns associated with schizophrenia spectrum disorders (SSD) from user generated social media activity. Few studies, however, have explored the potential for image analysis to inform psychiatric care for individuals with SSD. Given the popularity of image-based platforms, such as Instagram, investigating user generated image data could further strengthen associations between social media activity and behavioral health.Methods: We collected 11,947 Instagram posts across 68 participants (mean age = 23.6; 59% male) with schizophrenia spectrum disorders (SSD; n = 34) and healthy volunteers (HV; n = 34). We extracted image features including color composition, aspect ratio, and number of faces depicted. Additionally, we considered social connections and behavioral features. We explored differences in usage patterns between SSD and HV participants.Results: Individuals with SSD posted images with lower saturation (p = 0.033) and lower colorfulness (p = 0.005) compared to HVs, as well as images showing fewer faces on average (SSD = 1.5, HV = 2.4, p < 0.001). Further, individuals with SSD demonstrated a lower ratio of followers to following compared to HV participants (p = 0.025).Conclusion: Differences in uploaded images and user activity on Instagram were identified in individuals with SSD. These differences highlight potential digital biomarkers of SSD from Instagram data.

show abstract

“…Computational support is still considered insufficient to provide a precise diagnostic or complete support of human behavior specialists [12], [13]. Works from the literature have focused on the extraction of emotional characteristics and identifying feelings from individual posts without considering aspects of temporal and longitudinal analysis, i.e., the user's timeline and history [6], [8], [14].…”

Section: Introductionmentioning

confidence: 99%

Tracing the Emotional Roadmap of Depressive Users on Social Media Through Sequential Pattern Mining

et al. 2021

View full text Add to dashboard Cite

Depression is one of the most growing health disorders, generating social and economic problems globally. The affective computing models focus on analyzing unique user posts, not observing temporal behavior patterns, which are essential to track changes and the evolution of emotional behavior and user context, that involves the persistent analysis of feelings and characteristics over time. This article proposes the TROAD framework for longitudinal recognition of sequential patterns from depressive users on social media. The framework identifies the best interval to analyze every user activity, extracts emotional and contextual features from user data, and models the features into time windows to recognize sequential patterns from depressive user behavior. The main characteristics of the users found in the top-10 rules are negative emotions: violence, pain, shame, depression, sadness, and silence. We obtained strong sequence patterns with a minimum of 70% of support, 81% of confidence, and 69% sequential confidence, considering periods of silence between users' posts. Without considering silent periods, the rules showed 70%, 86%, and 38% of support, confidence, and sequential confidence. TROAD computational approach is a promising tool for clinical specialists in human behavior.

show abstract

Multimodal mental health analysis in social media

Cited by 77 publications

References 80 publications

Automatic Facial Expression Recognition in Standardized and Non-standardized Emotional Expressions

Automatic Facial Expression Recognition in Standardized and Non-standardized Emotional Expressions

Utilizing Instagram Data to Identify Usage Patterns Associated With Schizophrenia Spectrum Disorders

Tracing the Emotional Roadmap of Depressive Users on Social Media Through Sequential Pattern Mining

Contact Info

Product

Resources

About