Using Natural Language Processing and Sentiment Analysis to Augment Traditional User-Centered Design: Development and Usability Study (Preprint)

Abstract

BACKGROUND Sarcopenia, defined as the age-associated loss of muscle mass and strength, can be effectively mitigated through resistance-based physical activity. With compliance at ~40% for home-based exercise prescriptions, implementing a remote-sensing system would help patients and clinicians understand treatment progress and increase compliance. Inclusion of end users in the development of mobile applications for remote sensing systems can ensure that they are both user-friendly and facilitate compliance. OBJECTIVE The objective of our study was to develop a mobile application for a novel device through a user-centered design process with both older adults and clinicians. METHODS Through a user-centered design process, we conducted semi-structured interviews during the development of a geriatric-friendly Bluetooth-connected Resistance Exercise Band App. We interviewed patients and clinicians at weeks 0, 5, and 10 of the development of the app. Each semi-structured interview consisted of heuristic evaluations, cognitive walkthroughs, and observations of a geriatric-friendly Bluetooth-connected resistance exercise-band application. We used the Bing sentiment library for sentiment analysis on each transcript, then applied natural-language processing (NLP) based Latent Dirichlet Allocation (LDA) topic modeling to identify differences and similarities in patient and clinician participant interviews. To assess utility, we used post-interview quantitative assessments – System Usability Scale (SUS) and Usefulness, Satisfaction & Ease of use (USE). Finally, we used multivariate linear models – adjusting for age, sex, subject group (clinician vs. patient), and development – to explore the association between sentiment analysis and SUS and USE outcomes. RESULTS The mean age of the 22 participants was 68 years old, 17 (77%) of which were female. The overall mean SUS score was 66.4 (SD: 13.6) and mean USE score was 41.3 (SD: 15.2). We found that both patients and clinicians provided valuable insights into the needs of older adults when designing and building an application. The mean positive-negative sentiment for patients was 16 and 6 for clinicians. We found a positive association with positive sentiment in an interview and SUS score ( = 1.38 [95%CI = 0.37-2.39], P-value = .01). There was no significant association between sentiment and USE. Latent Dirichlet Allocation (LDA) analysis found no overlap between patients and clinicians in the eight topics identified. CONCLUSIONS Involving patients and clinicians allowed us to design and build an application that is user-friendly for older adults while supporting compliance. This is the first analysis that used NLP and usability questionnaires in the quantification of the user-centered design of technology for older adults.