Context-Enhanced Human-Robot Interaction: Exploring the Role of System Interactivity and Multimodal Stimuli on the Engagement of People with Dementia

Abstract

Engaging people with dementia (PWD) in meaningful activities is the key to promote their quality of life. Design towards a higher level of user engagement has been extensively studied within the human-computer interaction community, however, few extend to PWD. It is generally considered that increased richness of experiences can lead to enhanced engagement. Therefore, this paper explores the effects of rich interaction in terms of the role of system interactivity and multimodal stimuli by engaging participants in context-enhanced human-robot interaction activities. The interaction with a social robot was considered context-enhanced due to the additional responsive sensory feedback from an augmented reality display. A field study was conducted in a Dutch nursing home with 16 residents. The study followed a two by two mixed factorial design with one within-subject variable - multimodal stimuli - and one between-subject variable - system interactivity. A mixed method of video coding analysis and observational rating scales was adopted to assess user engagement comprehensively. Results disclose that when additional auditory modality was included besides the visual-tactile stimuli, participants had significantly higher scores on attitude, more positive behavioral engagement during activity, and a higher percentage of communications displayed. The multimodal stimuli also promoted social interaction between participants and the facilitator. The findings provide sufficient evidence regarding the significant role of multimodal stimuli in promoting PWD’s engagement, which could be potentially used as a motivation strategy in future research to improve emotional aspects of activity-related engagement and social interaction with the human partner.

Impact of digital assistive technologies on the quality of life for people with dementia: a scoping review

“…The topic of engagement and social interaction was highlighted in 64 studies. They differentiated between cases where DATs increased social interaction23 31 40 54–56 58 64 66–83 and fostered higher engagement (ie, in therapy settings) 8 20 23 31 44 45 53 55 61 63–65 68 69 71–73 77 79 82 84–113…”

“…Sixty-two of the included studies demonstrated that digital therapeutics play a significant role in promoting social interaction and engagement,8 23 31 40 44 45 53–56 58 61 63–75 77–113 including other DATs from the patient monitoring40 and care support categories 30 45…”

The use of technology by seniors with neurocognitive disorders in long-term care: a scoping review

“…No differences in mood, enjoyment, apathy or engagement Dinesen et al [ 31 ] 2022 Use of a Social Robot (LOVOT) for Persons With Dementia: Exploratory Study Denmark Mixed methods Forty-two (42) residents, mean age of 83,5 years A social robot LOVOT built with artificial intelligence, to interact with residents in real time and act like a human being Robotics Well-being; Mood; Behaviour; Acceptability; Interraction with robot The intervention had no clinically significant changes in the well-being. Entertaining and calming effects on the residents and promoted communication and social interactions Feng et al [ 32 ] 2022 Context-Enhanced Human–Robot Interaction: Exploring the Role of System Interactivity and Multimodal Stimuli on the Engagement of People with Dementia Dutch Quantitative experimental; Crossover Sixteen (16) residents, mean age of 85.2 years. 12 of the subjects were women LiveNature is an interactive system design aiming to connect residents to the outdoors through an indoor interactive experience, due to their limited contact with real nature Virtual Reality Engagement The intervention has significantly impacted participants’ engagement, attitude, valence, verbal communications, visual and social engagement Ford li et al [ 46 ] 2019 Evaluating the Impact of Music & Memories Personalized Music and Tablet Engagement Program in Wisconsin Assisted Living Communities: Pilot Study United States Quasi-experimental; Pilot study Thirty-five (35) residents iPods and iPads used by residents for their app–driven activities such as games, reminiscing about life stories with music, viewing Web-based images or using Google Earth, dabbling in art, and reactivating lifelong learning interests Tablet Agitation; QOL; Medication use The intervention led to a decrease in residents’ agitation and medication use and improved their QOL Garland et al [ 19 ] 2007 A comparison of two treatments of agitated behavior in nursing hom...…”

“…Virtual reality was the second most popular technology. Five studies [ 18 , 23 , 28 , 32 , 44 ] employed software and hardware to create interactive immersive environments presented through video games or 3D videos/images that can be accompanied by music. This technology was used to stimulate residents and improve their cognitive functions, such as psychomotor abilities and memory; to promote physical activity; to reduce responsive behaviors; to increase pleasure or visual and social engagement, especially in a COVID-19 context, as showcased in one of the studies.…”

A Survey on Recent Advances in Social Robotics

“…In a social context, the intelligence that a robot display depends on the modalities it uses and each modality can have specific importance and effect on the human side of the interaction [88] which translates into the degree of trust that the robot has [89]. Moreover, the acceptance of robots in social interaction depends on their ability to express emotions and they require a proper design of emotional expressions to improve their likability and believability [90] as multimodal interaction can enhance the engagement [91][92][93].…”