Detecting errors in pick and place procedures

Abstract

Many human activities, such as manufacturing and assembly, are sequence-constrained procedural tasks (SPTs): they consist of a series of steps that must be executed in a specific spatial/temporal order. However, these tasks can be error prone -steps can be missed out, executed out-of-order, and repeated. The ability to automatically predict if a person is about to commit an error could greatly help in these cases. The prediction could be used, for example, to provide feedback to prevent mistakes or mitigate their effects. In this paper, we present a novel approach for real-time error prediction for multi-step sequence tasks which uses a minimum viable set of behavioural signals. We have three main contributions. The first we present an architecture for real-time error prediction based on task tracking and intent prediction. The second is to explore the effectiveness of using hand position and eye-gaze tracking for task tracking. We confirm that eye-gaze is more effective for intent prediction, hand tracking is more accurate for task tracking and that combining the two provides the best overall response. We show that using Hands and Gaze tracking data we can predict selection/placement errors with an F1 score of 97%, approximately 300ms before the error would occur. Finally, we discuss the application of this hand-gaze error detection architecture used in conjunction with head-mounted AR displays, to support industrial manual assembly.

Not All Errors Are Created Equal: Exploring Human Responses to Robot Errors with Varying Severity

“…Future work can extend to additionally include eye gaze in modeling human states, which prior works have suggested to be indicative of robot states (e.g., errors) during situated interactions. Research has shown how gaze cues may be used to understand human intent (e.g., [14]) and robot actions (e.g., [26]), as well as how they can be utilized to enhance robot learning from human demonstration [25], detect errors [2,3], and facilitate efficient human-robot collaboration [15]. All in all, multimodal human signals present opportunities to enhance human-robot interaction.…”

“…In addition, people respond differently when facing social norm violations and technical failures; in particular, technical failures generally resulted in fewer social signals and faster reaction times [20]. Prior works have demonstrated how social signals such as upper body movements (e.g., [28]), gaze (e.g., [2,3]), and gestures (e.g., [3]) can be used to detect errors effectively. It is worth noting that most of these prior works used human-like robots, which have been shown to elicit different responses to failures than non-humanoid embodiments in social error scenarios [17].…”

Effect of Changes in the Sequence of Assembly Operations on Error Rates: A Case Study From the Car Manufacturing Industry

“…Another feature of manual assembly tasks in manufacturing is that they can be considered sequenceconstrained procedural tasks (SPTs). According to Bovo et al [59], an SPT comprises a set of steps that must be performed in a specific temporal order. Following SPTs by workers can be associated with several errors: steps can be carried out in the wrong order, they can be omitted, or they can be repeated, among others.…”

The possibility of employing Intelligence Poka Yoke technology to reduce marketing risks: A study of the Iraqi oil refineries company

“…The application of the Intelligent Poka-Yuke technology achieves a number of benefits, the most important of which are (Bovo, et al, 2020, 540) [3] : 1. Information inspection: In order to support and improve production processes, it is carried out using ~ 103 ~ smart sensors and computer vision systems, to detect all errors and defects (data and information), analyze them in a timely manner, and use them to discover possibilities for error prevention and improvement.…”