Designing an Experience Sampling Method for Smartphone Based Emotion Detection

Ghosh, Surjya; Ganguly, Niloy; Mitra, Bivas; De, Pradipta

doi:10.1109/taffc.2019.2905561

Cited by 18 publications

(11 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Limitations of our study are associated with those of the contributions. With respect to the list of requirements, even though requirements were elicited via interviews, iterative design, and empirical evaluation, some requirements might be left out, in particular due to the lack of case studies that demand supporting relationships among participants (FR10) or that employ home sensors, wearable devices, and JITAIs along with machine learning-based triggers [68,69]. These limitations are reflected in the 4-dimension planning framework, in the sensor dimension which might be more detailed to support such scenarios that demand, for instance, human activity recognition [70].…”

Section: Limitationsmentioning

confidence: 99%

Experience Sampling and Programmed Intervention Method and System for Planning, Authoring, and Deploying Mobile Health Interventions: Design and Case Reports

Cunha¹,

Rodrigues²,

Zaine³

et al. 2021

J Med Internet Res

View full text Add to dashboard Cite

Background Health professionals initiating mobile health (mHealth) interventions may choose to adapt apps designed for other activities (eg, peer-to-peer communication) or to employ purpose-built apps specialized in the required intervention, or to exploit apps based on methods such as the experience sampling method (ESM). An alternative approach for professionals would be to create their own apps. While ESM-based methods offer important guidance, current systems do not expose their design at a level that promotes replicating, specializing, or extending their contributions. Thus, a twofold solution is required: a method that directs specialists in planning intervention programs themselves, and a model that guides specialists in adopting existing solutions and advises software developers on building new ones. Objective The main objectives of this study are to design the Experience Sampling and Programmed Intervention Method (ESPIM), formulated toward supporting specialists in deploying mHealth interventions, and the ESPIM model, which guides health specialists in adopting existing solutions and advises software developers on how to build new ones. Another goal is to conceive and implement a software platform allowing specialists to be users who actually plan, create, and deploy interventions (ESPIM system). Methods We conducted the design and evaluation of the ESPIM method and model alongside a software system comprising integrated web and mobile apps. A participatory design approach with stakeholders included early software prototype, predesign interviews with 12 health specialists, iterative design sustained by the software as an instance of the method’s conceptual model, support to 8 real case studies, and postdesign interviews. Results The ESPIM comprises (1) a list of requirements for mHealth experience sampling and intervention-based methods and systems, (2) a 4-dimension planning framework, (3) a 7-step-based process, and (4) an ontology-based conceptual model. The ESPIM system encompasses web and mobile apps. Eight long-term case studies, involving professionals in psychology, gerontology, computer science, speech therapy, and occupational therapy, show that the method allowed specialists to be actual users who plan, create, and deploy interventions via the associated system. Specialists’ target users were parents of children diagnosed with autism spectrum disorder, older persons, graduate and undergraduate students, children (age 8-12), and caregivers of older persons. The specialists reported being able to create and conduct their own studies without modifying their original design. A qualitative evaluation of the ontology-based conceptual model showed its compliance to the functional requirements elicited. Conclusions The ESPIM method succeeds in supporting specialists in planning, authoring, and deploying mobile-based intervention programs when employed via a software system designed and implemented according to its conceptual model. The ESPIM ontology–based conceptual model exposes the design of systems involving active or passive sampling interventions. Such exposure supports the evaluation, implementation, adaptation, or extension of new or existing systems.

show abstract

Section: Limitationsmentioning

confidence: 99%

Experience Sampling and Programmed Intervention Method and System for Planning, Authoring, and Deploying Mobile Health Interventions: Design and Case Reports

Cunha¹,

Rodrigues²,

Zaine³

et al. 2021

J Med Internet Res

View full text Add to dashboard Cite

show abstract

“…All of the events between two successive surveys are labelled with the input provided by the user. In their further study presented in [91], the authors also trained a machine learning model to detect the inopportune moments for self-reports. To get the examples of inopportune moments necessary to train the model, information from reports was used, as it allowed the users to select a NoResponse option.…”

Section: Data Labellingmentioning

confidence: 99%

A Review of Emotion Recognition Methods Based on Data Acquired via Smartphone Sensors

Kołakowska

Szwoch

2020

Sensors

View full text Add to dashboard Cite

In recent years, emotion recognition algorithms have achieved high efficiency, allowing the development of various affective and affect-aware applications. This advancement has taken place mainly in the environment of personal computers offering the appropriate hardware and sufficient power to process complex data from video, audio, and other channels. However, the increase in computing and communication capabilities of smartphones, the variety of their built-in sensors, as well as the availability of cloud computing services have made them an environment in which the task of recognising emotions can be performed at least as effectively. This is possible and particularly important due to the fact that smartphones and other mobile devices have become the main computer devices used by most people. This article provides a systematic overview of publications from the last 10 years related to emotion recognition methods using smartphone sensors. The characteristics of the most important sensors in this respect are presented, and the methods applied to extract informative features on the basis of data read from these input channels. Then, various machine learning approaches implemented to recognise emotional states are described.

show abstract

“…Once the user completes text entry in a session and changes the application, she is probed to record her perceived emotion during this session. We issue the emotion self-report collection probes using Experience Sampling Method (ESM) [21]- [23]. Finally, the user provided emotion self-reports are used as the ground truth labels for the associated sessions.…”

Section: A Data Collection Processmentioning

confidence: 99%

Representation Learning for Emotion Recognition from Smartphone Keyboard Interactions

Ghosh

Goenka

Ganguly

et al. 2019

2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)

Self Cite

View full text Add to dashboard Cite

Characteristics of typing on smartphone keyboards among different individuals can elicit emotion, similar to speech prosody or facial expressions. Existing works on typing based emotion recognition rely on feature engineering to build machine learning models, while recent speech and facial expression based techniques have shown the efficacy of learning the features automatically. Therefore, in this work, we explore the effectiveness of such learning models in keyboard interaction based emotion detection. In this paper, we propose an end-to-end framework, which first uses a sequence-based encoding method to automatically learn the representation from raw keyboard interaction pattern and subsequently uses this representation to train a multi-task learning based neural network (MTL-NN) to identify different emotions. We carry out a 3-week in-thewild study involving 24 participants using a custom keyboard capable of tracing users' interaction pattern during text entry. We collect interaction details like touch speed, error rate, pressure and self-reported emotions (happy, sad, stressed, relaxed) during the study. Our analysis on the collected dataset reveals that the representation learnt from the interaction pattern has an average correlation of 0.901 within the same emotion and 0.811 between different emotions. As a result, the representation is effective in distinguishing different emotions with an average accuracy (AUCROC) of 84%.

show abstract

Designing an Experience Sampling Method for Smartphone Based Emotion Detection

Cited by 18 publications

References 43 publications

Experience Sampling and Programmed Intervention Method and System for Planning, Authoring, and Deploying Mobile Health Interventions: Design and Case Reports

Experience Sampling and Programmed Intervention Method and System for Planning, Authoring, and Deploying Mobile Health Interventions: Design and Case Reports

A Review of Emotion Recognition Methods Based on Data Acquired via Smartphone Sensors

Representation Learning for Emotion Recognition from Smartphone Keyboard Interactions

Contact Info

Product

Resources

About