Learning and Personalizing Socially Assistive Robot Behaviors to Aid with Activities of Daily Living

Abstract

Socially assistive robots can autonomously provide activity assistance to vulnerable populations, including those living with cognitive impairments. To provide effective assistance, these robots should be capable of displaying appropriate behaviors and personalizing them to a user's cognitive abilities. Our research focuses on the development of a novel robot learning architecture that uniquely combines learning from demonstration (LfD) and reinforcement learning (RL) algorithms to effectively teach socially assistive robots personalized behaviors. Caregivers can demonstrate a series of assistive behaviors for an activity to the robot, which it uses to learn general behaviors via LfD. This information is used to obtain initial assistive state-behavior pairings using a decision tree. Then, the robot uses an RL algorithm to obtain a policy for selecting the appropriate behavior personalized to the user's cognition level. Experiments were conducted with the socially assistive robot Casper to investigate the effectiveness of our proposed learning architecture. Results showed that Casper was able to learn personalized behaviors for the new assistive activity of tea-making, and that combining LfD and RL algorithms significantly reduces the time required for a robot to learn a new activity.15:2 C. Moro et al. programs [5], and providing social therapy to autistic children [6]. The behaviors of socially assistive robots have traditionally been designed using one of three methods: (1) manually hand-crafting combinations of speech, gestures, and other communication modes necessary to display a behavior [7-10]; (2) teaching a robot multimodal behaviors through learning from demonstration (LfD) [11,12]; or (3) autonomously learning multimodal behaviors via reinforcement learning (RL) algorithms [13,14]. Manually preprogramming robot behaviors involves tedious annotation, without the potential for expanding the robot's skillset once the robot is deployed in an environment. LfD and RL algorithms allow robots to learn behaviors without having to preprogram them. However, they may require large numbers of interactions with demonstrators (e.g., LfD) or intended users (e.g., RL) for training purposes, which may not always be available or feasible. With respect to the latter, it is not always safe for vulnerable users to engage with a robot that has not been fully trained. In addition to learning general assistive behaviors, socially assistive robots may also have to adapt their behaviors to their specific users, as behavior personalization can positively affect robot acceptance [5, 7] and increase its use over time [7].Only a handful of work has focused on personalizing assistive robot behaviors to user profiles [5,[15][16][17]. Behaviors have been personalized to either a general user group, for example, extroverted versus introverted users [5], or to a user state during an activity, such as stress level during a memory game [15]. Personalization of assistive robot behaviors to a single user's cognitive model has yet to be inves...

Reinforcement Learning Approaches in Social Robotics

“…The evolution of the learning algorithm over time (e.g., the evolution of Q values) is another evaluation method. Several studies presented only the learning evolution of their system without mentioning how a participant would perceive the learned robot behaviors [ 13 , 34 , 61 , 63 , 64 , 65 , 66 , 108 ]. Comparison of user experiences (e.g., learning gains of children) for adaptive and non-adaptive robot is another way of evaluation [ 68 , 102 ].…”

“…Q-learning, along with its different variations, is the most commonly used RL method in social robotics. The studies using Q-learning are [ 3 , 13 , 34 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 ]. These comprise studies using standard Q-learning [ 3 , 54 , 55 , 58 , 60 , 62 ], studies modify Q-learning for dealing with delayed reward [ 52 ], tuning the parameters for Q-learning such as [ 13 , 34 , 52 ], dealing with decreasing human feedback over time [ 52 ], comparing their proposed algorithm with Q-learning [ 33 , 49 , 61 , 63 , 64 ], variation of Q-learning called Object Q-learning [ 64 , 65 , 66 ], combining Q-learning with fuzzy inference [ 67 ], SARSA [ 68 , 69 ], TD( ) [ 70 ], MAXQ [ 33 , 71 , 72 ], R-learning [ 32 ], and Deep Q-learning [ 35 , 36 , 73 , 74 ].…”

“…c represents the cost of the chosen action in cases where the costs were determined manually. Moro et al [ 61 ] is another study that considered the attention of the user. Their scenario was an assistive tea-making activity for older people with dementia.…”

“…Their scenario was an assistive tea-making activity for older people with dementia. The authors proposed an algorithm involving Learning from Demonstration (LfD) and Q-learning for personalized robot behavior according to a user’s cognitive abilities [ 61 ]. The Casper robot learned to imitate the combination of speech and gestures from a collected data set.…”

“…The robot also received reward for its actions that were suitable for the current situation. Likewise, in [ 61 ], the robot was rewarded based on actions that transitioned the user into a desirable state (e.g., completing the activity). Other papers in this category are [ 33 , 71 , 72 ].…”