A multi-modal perception based assistive robotic system for the elderly

Abstract

International audienceIn this paper, we present a multi-modal perception based framework to realize a non-intrusive domestic assistive robotic system. It is non-intrusive in that it only starts interaction with a user when it detects the user's intention to do so. All the robot's actions are based on multi-modal perceptions which include user detection based on RGB-D data, user's intention-for-interaction detection with RGB-D and audio data, and communication via user distance mediated speech recognition. The utilization of multi-modal cues in different parts of the robotic activity paves the way to successful robotic runs (94% success rate). Each presented perceptual component is systematically evaluated using appropriate dataset and evaluation metrics. Finally the complete system is fully integrated on the PR2 robotic platform and validated through system sanity check runs and user studies with the help of 17 volunteer elderly participants

Multimodal Uncertainty Reduction for Intention Recognition in Human-Robot Interaction

“…Thus, feature fusion was conducted instead of classifier fusion which would fuse the outputs of individual classifiers. Mollaret et al [8] also dealt with the recognition of an intention for interaction with an assistive robot. Using head and shoulder orientation and voice activity the corresponding intention could be inferred with a Hidden Markov Model.…”

Robot Classification of Human Interruptibility and a Study of Its Effects

“…The second category of techniques for explicitly estimating availability leverages a person's demeanor, focusing on immediate social cues of availability. Social cues, such as eye contact, are largely task-independent, and as a result, models based on social cues are more easily generalizable across a wider set of applications: in robotics, the methods have been used to estimate related measures of a person's "intent-to-engage" and awareness of the robot in applications ranging from companion robots [13,42], shopping mall assistants [9,31,57,59], receptionists [7], and bartenders [21]. Some prior work has relied on external sensors such as motion capture systems, ground-mounted LIDAR, and ceiling cameras [9,31,57,59], which can be expensive and difficult to deploy in support of mobile robots traversing a large space.…”

“…Some prior work has relied on external sensors such as motion capture systems, ground-mounted LIDAR, and ceiling cameras [9,31,57,59], which can be expensive and difficult to deploy in support of mobile robots traversing a large space. Other work has used onboard sensors to detect social cues of engagement [7,13,21,42]. Although engagement estimation is a separate problem from interruptibility estimation (because interruptibility can be high even when engagement is low), the problems are closely related, and we take inspiration from the work of Mollaret et al [42] and Chiang et al [13] in both our selection of audio-visual features for classification and in validating the use of Hidden Markov Models to estimate interruptibility.…”

“…Person state is widely used to model engagement and human awareness in robotics [13,42]. Although classifying interruptibility poses its own research problem, because interruptibility can be high even when a person shows neither intent-to-engage or awareness of the robot, we propose the cues of person state from the engagement modeling literature can be informative for interruptibility.…”

“…In our examination, we first contribute an ordinal scale of interruptibility that can be used to rate the interruptibility of a person and to influence decisions on whether or not to interrupt them (Section 3). Second, derived from factors used by humans to gauge interruptibility [53], we propose using features for person state (motivated by prior work in robotics on the closely related problem of estimating human engagement [42]) and features for interruption context (inspired by cues to interruptibility context used in prior work [47]) to classify interruptibility (Section 4). Last, we introduce the non-temporal and temporal models that we evaluated (Section 5) and the dataset of person observations that the models were evaluated upon (Section 6).…”