Generalization and Personalization of Mobile Sensing-Based Mood Inference Models

Abstract

Mood inference with mobile sensing data has been studied in ubicomp literature over the last decade. This inference enables context-aware and personalized user experiences in general mobile apps and valuable feedback and interventions in mobile health apps. However, even though model generalization issues have been highlighted in many studies, the focus has always been on improving the accuracies of models using different sensing modalities and machine learning techniques, with datasets collected in homogeneous populations. In contrast, less attention has been given to studying the performance of mood inference models to assess whether models generalize to new countries. In this study, we collected a mobile sensing dataset with 329K self-reports from 678 participants in eight countries (China, Denmark, India, Italy, Mexico, Mongolia, Paraguay, UK) to assess the effect of geographical diversity on mood inference models. We define and evaluate country-specific (trained and tested within a country), continent-specific (trained and tested within a continent), country-agnostic (tested on a country not seen on training data), and multi-country (trained and tested with multiple countries) approaches trained on sensor data for two mood inference tasks with population-level (non-personalized) and hybrid (partially personalized) models. We show that partially personalized country-specific models perform the best yielding area under the receiver operating characteristic curve (AUROC) scores of the range 0.78--0.98 for two-class (negative vs. positive valence) and 0.76--0.94 for three-class (negative vs. neutral vs. positive valence) inference. Further, with the country-agnostic approach, we show that models do not perform well compared to country-specific settings, even when models are partially personalized. We also show that continent-specific models outperform multi-country models in the case of Europe. Overall, we uncover generalization issues of mood inference models to new countries and how the geographical similarity of countries might impact mood inference.

Measuring algorithmic bias to analyze the reliability of AI tools that predict depression risk using smartphone sensed-behavioral data

“…Model performance was assessed by performing 5-fold cross-validation, partitioning on subjects, and predictions across folds were concatenated to calculate model performance. Similar to prior work 4 , 6 , within each cross-validation split, models were trained using data collected from 80% of the participants (520 participants), and the trained model was applied to predict CSD in the remaining 20% (130 participants). To analyze performance variability due to specific cross-validation splits, we performed 100 cross-validation trials, shuffling participants into different folds during each trial.…”

“…Initial work showed that depression risk could be predicted from sensed-behavioral data at a similar accuracy to general practitioners 10 in small populations 5 , 11 . More recent work shows that these AI tools predict depression risk at an accuracy only slightly better than a coin flip in larger, more diverse samples 4 , 6 , 12 , 13 . This prior work has not specifically explored why accuracy is reduced in larger samples, and it is unclear how to improve AI tools for clinical use.…”

Time2Stop: Adaptive and Explainable Human-AI Loop for Smartphone Overuse Intervention

“…3.1.1 Feature Design. We design a set of five passive sensing feature categories [59,87,89] to capture smartphone overuse behavior: (a) Phone and App Usage. Understanding smartphone overuse requires a thorough analysis of usage patterns.…”

Diversity and neocolonialism in Big Data research: Avoiding extractivism while struggling with paternalism

“…Second, it is elementary that those who collect the data be involved in the research, analysis, and publication. Therefore, in our case, all project partners went to great lengths to ensure that joint research was conducted with the collected data, resulting in joint publications (Schelenz et al, 2021;Meegahapola et al, 2021Meegahapola et al, , 2022Assi et al, 2023). This ensures that all stakeholders benefit from the data they help produce and that the data are used in ways that are relevant to different local contexts.…”