Robotic Telemedicine for Mental Health: A Multimodal Approach to Improve Human-Robot Engagement

Abstract

COVID-19 has severely impacted mental health in vulnerable demographics, in particular older adults, who face unprecedented isolation. Consequences, while globally severe, are acutely pronounced in low- and middle-income countries (LMICs) confronting pronounced gaps in resources and clinician accessibility. Social robots are well-recognized for their potential to support mental health, yet user compliance (i.e., trust) demands seamless affective human-robot interactions; natural ‘human-like’ conversations are required in simple, inexpensive, deployable platforms. We present the design, development, and pilot testing of a multimodal robotic framework fusing verbal (contextual speech) and nonverbal (facial expressions) social cues, aimed to improve engagement in human-robot interaction and ultimately facilitate mental health telemedicine during and beyond the COVID-19 pandemic. We report the design optimization of a hybrid face robot, which combines digital facial expressions based on mathematical affect space mapping with static 3D facial features. We further introduce a contextual virtual assistant with integrated cloud-based AI coupled to the robot’s facial representation of emotions, such that the robot adapts its emotional response to users’ speech in real-time. Experiments with healthy participants demonstrate emotion recognition exceeding 90% for happy, tired, sad, angry, surprised and stern/disgusted robotic emotions. When separated, stern and disgusted are occasionally transposed (70%+ accuracy overall) but are easily distinguishable from other emotions. A qualitative user experience analysis indicates overall enthusiastic and engaging reception to human-robot multimodal interaction with the new framework. The robot has been modified to enable clinical telemedicine for cognitive engagement with older adults and people with dementia (PwD) in LMICs. The mechanically simple and low-cost social robot has been deployed in pilot tests to support older individuals and PwD at the Schizophrenia Research Foundation (SCARF) in Chennai, India. A procedure for deployment addressing challenges in cultural acceptance, end-user acclimatization and resource allocation is further introduced. Results indicate strong promise to stimulate human-robot psychosocial interaction through the hybrid-face robotic system. Future work is targeting deployment for telemedicine to mitigate the mental health impact of COVID-19 on older adults and PwD in both LMICs and higher income regions.

Digital Technologies and Data Science as Health Enablers: An Outline of Appealing Promises and Compelling Ethical, Legal, and Social Challenges

“…In order to achieve its highest aims while preserving trust, the uptake of digital technologies in healthcare must be transparent (40). In a world where data can be artificially created, it is increasingly important that the non-human dimensions of healthcare are disclosed, including the usage of models and algorithms, both in and outside the context of telehealth (9,14,15,54). Therefore, medical decisions supported by digital technologies should be more transparent and understandable, in order to simultaneously guarantee accountability and avoid patient disenfranchizement and exclusion.…”

Conversational Affective Social Robots for Ageing and Dementia Support

“…In relation to the design of simpler, affordable, yet expressive SAR, the implementation of an LCD screen for the robot's face has been argued to facilitate customization, adaptability to users' preferences, broader accessibility, and cultural sensibility of facial expressions [7,73,74]. Over 53% of the key SAR platforms reviewed employed LCD screens, either as part of the robot's physical appearance and expressiveness (e.g., [47,75]), or as an external source of human-robot input/output (e.g., [22,55]).…”

“…Looking at trends in human-robot interventions, there are three factors to consider: (i) sample size, which accounts for the number of end-users trialled; we consider the following: limited (1-5), small (6)(7)(8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19)(20), medium (21)(22)(23)(24)(25)(26)(27)(28)(29)(30)(31)(32)(33)(34)(35)(36)(37)(38)(39)(40), and large (40+); (ii) study duration, which can either account for the total number of HRI sessions or the length of continuous interactions over days, weeks, months or years; (iii) type of setting: controlled (e.g., laboratories, simulated home environments) or real-world (e.g., homes, clinical/care facilities, nursing homes).…”

“…This global health crisis has been exacerbated by the COVID-19 pandemic, with vulnerable populations facing further limits in care, family support, isolation, and pronounced mental health decline [4,5]; COVID-19 has caused unprecedented stress, fear and agitation among the seniors, especially those with cognitive impairment or dementia [6]. While the impact of these additional challenges will have long-lasting consequences well beyond the pandemic, they have also triggered increased use of technological tools among older populations and opened new avenues for mental health telemedicine, including robotic solutions for humanrobot cognitive engagement [7,8].…”

“…This review aims to fill said gap by surveying conversational affective social robots specifically targeted at supporting ageing and dementia, in both social and clinical scenarios. These systems are widely acknowledged as having significant future research and clinical potential to support independence in the home, telemedicine, and isolation during and beyond COVID-19 [7,44,45].…”