Hotspot Modeling of Hand-Machine Interaction Experiences from a Head-Mounted RGB-D Camera

Abstract

This paper presents an approach to analyze and model tasks of machines being operated. The executions of the tasks were captured through egocentric vision. Each task was decomposed into a sequence of physical hand-machine interactions, which are described with touch-based hotspots and interaction patterns. Modeling the tasks was achieved by integrating the experiences of multiple experts and using a hidden Markov model (HMM). Here, we present the results of more than 70 recorded egocentric experiences of the operation of a sewing machine. Our methods show good potential for the detection of hand-machine interactions and modeling of machine operation tasks.

Hotspots Integrating of Expert and Beginner Experiences of Machine Operations through Egocentric Vision

“…This method recognizes task-relevant-objects and their modes-of-interactions by user's attention fixation. Chen et al [2] built models for machine operation tasks by automatically extracting the temporal interactions using hand shape and touch.…”

“…However, these features cannot identify every step of a task step and lack semantic explanations. Attention cues are used in the above mentioned works [3] for guidance-data acquisition, and Chen conjectured that a hotspot could be a better feature for this purpose [2]. Thus, a method for automatically integrating both expert and beginner experiences based on hotspots was investigated, and hidden Markov models (HMMs) were used to model the temporal structures of the experiences.…”

“…To locate the hotspot from FPV to global location, global maps of the machine such as an IH heater and a sewing machine surfaces are constructed beforehand, and the FPV frame is matched to the global maps by estimate the camera pose and 2D registration. Details are given in [2].…”

“…For detecting hotspots and interaction patterns, the same parameters as in [2] were adopted, i.e., the depth threshold for detecting valid touches is ±7 mm. For the integration, the expert baseline model was first directly trained; then, each beginner observation was integrated by assigning the small constant probability δ = 0.01, and adding it to the original paths and stats of the baseline model.…”

Integration of Experts' and Beginners' Machine Operation Experiences to Obtain a Detailed Task Model

“…Unlike experts' efficient operation behaviors, beginners often make mistakes, or perform unnecessary operations or redundant trials; they sometimes devise an easier approach or a new order of performing operations; or they do not complete tasks that are too difficult for them. Our approach to managing such varieties is as follows: We first automatically summarize experts' experiences into the baseline model, which is a sequence of symbolized hand-machine interactions that correspond to the crucial operation locations on a machine, that is, hotspots [11]. We then integrate beginners' experiences into the baseline model and obtain a unified model.…”