Projecting Recorded Expert Hands at Real Size, at Real Speed, and onto Real Objects for Manual Work

Abstract

Expert manual workers in factories assemble more efficiently than novices because their movements are optimized for the tasks. In this paper, we present an approach to projecting the hand movements of experts at real size, and real speed and onto real objects in order to match the manual work movements of novices to those of experts. We prototyped a projector-camera system, which projects the virtual hands of experts. We conducted a user study in which users worked after watching experts work under two conditions: using a display and using our prototype system. The results show our prototype users worked more precisely and felt the tasks were easier. User ratings also show our prototype users watched videos of experts more fixedly, memorized them more clearly and distinctly tried to work in the same way shown in the videos as compared with display users.

Critiquing Physical Prototypes for a Remote Audience

“…In these tasks, an "expert" or "helper" typically provides step-by-step instructions or a description of a known end-state to a remote "worker" who assembles, fixes, or modifies physical materials at their location. These physical tasks may include building block assembly (e.g., Amores et al, 2015;Fakourfar et al, 2016;Higuch et al, 2016;Suzuki et al, 2016), construction tasks (e.g., Fussell, Setlock, & Kraut, 2003;Fussell et al, 2004), repair tasks (e.g., Gauglitz, Nuernberger, Turk, & Höllerer, 2014;Kraut, Fussell, & Siegel, 2003), or object placement (e.g., Jones et al, 2016;Tait & Billinghurst, 2015). While the helper has access to information about instructions or an end-state or goal, the worker lacks this information but instead has access to the physical media needed to complete the task.…”

“…Unlike assembly such as the Lego block assembly task shown in Figure Suzuki et al (2016). Hands of the 'helper' collaborator projected into the workspace of a 'worker' to support a worker in a Lego block assembly task.…”

Enhanced Learning of Jazz Chords with a Projector Based Piano Keyboard Augmentation

“…The idea is that the student learns by playing along by copying the movements of the master. This approach is also used in other domains to visualize the craft of an expert [21]. Nemoto and Umezu reported a similar approach where they played back a mirrored version of a video, slowing the teacher playing behind the piano keyboard [22].…”

ISAR: An Authoring System for Interactive Tabletops

“…By registering the hand of remote expert on the scene projected for the worker, it can also be possible to guide the worker using hand interactions. Similar approaches have been implemented for example in [RT07], [KSF06] and [SMF16].…”