There are different modes of interaction with a software keyboard on a smartphone, such as typing and swyping. Patterns of such touch interactions on a keyboard may reflect emotions of a user. Since users may switch between different touch modalities while using a keyboard, therefore, automatic detection of emotion from touch patterns must consider both modalities in combination to detect the pattern. In this paper, we focus on identifying different features of touch interactions with a smartphone keyboard that lead to a personalized model for inferring user emotion. Since distinguishing typing and swyping activity is important to record the correct features, we designed a technique to correctly identify the modality. The ground truth labels for user emotion are collected directly from the user by periodically collecting self-reports. We jointly model typing and swyping features and correlate them with user provided self-reports to build a personalized machine learning model, which detects four emotion states (happy, sad, stressed, relaxed). We combine these design choices into an Android application TouchSense and evaluate the same in a 3-week in-the-wild study involving 22 participants. Our key evaluation results and post-study participant assessment demonstrate that it is possible to predict these emotion states with an average accuracy (AUCROC) of 73% (std dev. 6%, maximum 87%) combining these two touch interactions only.

In Affective Computing, different modalities, such as speech, facial expressions, physiological properties, smartphone usage patterns, and their combinations, are applied to detect the affective states of a user. Keystroke analysis i.e. study of the typing behavior in desktop computer is found to be an effective modality for emotion detection because of its reliability, non-intrusiveness and low resource overhead. As smartphones proliferate, typing behavior on smartphone presents an equally powerful modality for emotion detection. It has the added advantage to run in-situ experiments with better coverage than the experiments using desktop computer keyboards. This work explores the efficacy of smartphone typing to detect multiple affective states. We use a qualitative and experimental approach to answer the question. We conduct an online survey among 120 participants to understand the typing habits in smartphones and collect feedback on multiple measurable parameters that affect their emotion while typing. The findings lead us to design and implement an Android based emotion detection system, TapSense, which can identify four different emotion states (happy, sad, stressed, relaxed) with an average accuracy (AUCROC) of 73% (maximum of 94%) based on typing features only. The analysis also reveals that among different features, typing speed is the most discriminative one.

The rate of mental health disorders is rising across the globe. While it significantly affects the quality of life, an early detection can prevent the fatal consequences. Existing literature suggests that mobile based sensing technology can be used to determine different mental health conditions like stress, bipolar disorder. In today's smartphone based communication, a significant portion is based on instant messaging apps like WhatsApp; thus providing the opportunity to unobtrusively monitor the text input interaction pattern to track mental state. We, in this paper, leverage on the text entry pattern to track multiple emotion states. We design, develop and implement an Android based smartphone keyboard EmoKey, which monitors user's typing pattern and determines four emotion states (happy, sad, stressed, relaxed) by developing an on-device, personalized machine learning model. We evaluate EmoKey with 22 participants in a 3-week in-the-wild study, which reveals that it can detect the emotions with an average accuracy (AUCROC) of 78%.

Typing characteristics on smartphones can provide clues for emotion detection. Collecting large volumes of typing data is also easy on smartphones. This motivates the use of Deep Neural Network (DNN) to determine emotion states from smartphone typing. In this work, we developed a DNN model based on typing features to predict four emotion states (happy, sad, stressed, relaxed) and investigate its performance on a smartphone. The evaluation of the model in a 3-week study with 15 participants reveals that it can reliably detect emotions with an average accuracy of 80% with peak CPU utilization less than 15%.

Smartphones provide the capability to perform in-situ sampling of human behavior using Experience Sampling Method (ESM). Designing an ESM schedule involves probing the user repeatedly at suitable moments to collect self-reports. Timely probe generation to collect high fidelity user responses while keeping probing rate low is challenging. In mobile-based ESM, timeliness of the probe is also impacted by user's availability to respond to self-report request. Thus, a good ESM design must consider -probing frequency, timely self-report collection, and notifying at opportune moment to ensure high response quality. We propose a two-phase ESM design, where the first phase (a) balances between probing frequency and self-report timeliness, and (b) in parallel, constructs a predictive model to identify opportune probing moments. The second phase uses this model to further improve response quality by eliminating inopportune probes. We use typing-based emotion detection in smartphone as a case study to validate proposed ESM design. Our results demonstrate that it reduces probing rate by 64%, samples self-reports timely by reducing elapsed time between self-report collection, and event trigger by 9% while detecting inopportune moments with an average accuracy of 89%. These design choices improve the response quality, as manifested by 96% valid response collection and a maximum improvement of 24% in emotion classification accuracy.