A Methods-Time-Measurement based Approach to enable Action Recognition for Multi-Variant Assembly in Human-Robot Collaboration

Koch, Julian; Büsch, Lukas; Gomse, Martin; Schüppstuhl, Thorsten

doi:10.1016/j.procir.2022.02.184

Cited by 14 publications

(22 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Subsequent to the data acquisition, the collected data need to be processed to obtain a proper estimation of the assembly progress. For this, the vast majority of available methods from the state of research identify the current assembly step based on the acquired data and subsequently estimate the overall assembly progress respecting the assembly plan and the history of detected assembly steps [ 11 ].…”

Section: Related Workmentioning

confidence: 99%

“…For HAR, the poses and movements of humans in the process are captured by localization and motion-capturing systems such as camera systems or inertial measurement units. Based on this information, their current assembly action is inferred [ 11 , 15 ]. In [ 18 , 19 ], a Hidden Markov Model (HMM) is used to learn the trajectories of the human worker during the assembly process.…”

Section: Related Workmentioning

confidence: 99%

“…Various methods either observe the workpiece, the tools, or the human in the process to derive the assembly progress [ 8 , 9 , 10 ]. Thereby, as will be argued in Section 2 , the methods of Human Action Recognition (HAR) [ 11 , 12 ] are of particular relevance to this work. HAR feedback methods in manufacturing focus on the movements and actions of the human worker to obtain information about the current assembly process for inferring the overall assembly progress.…”

Section: Introductionmentioning

confidence: 99%

“…For this, the human pose and movements are tracked by localization and motion-capturing systems such as camera systems or inertial measurement units. These HAR methods have several advantages in application to flexible manufacturing systems and handling multi-variant products without introducing non-value-adding process steps for the human worker [ 11 , 13 ]. However, current implementations of HAR methods for automated progress detection in multi-variant assembly processes show deficits.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Towards Recognition of Human Actions in Collaborative Tasks with Robots: Extending Action Recognition with Tool Recognition Methods

Büsch

Koch

Schoepflin

et al. 2023

Sensors

Self Cite

View full text Add to dashboard Cite

This paper presents a novel method for online tool recognition in manual assembly processes. The goal was to develop and implement a method that can be integrated with existing Human Action Recognition (HAR) methods in collaborative tasks. We examined the state-of-the-art for progress detection in manual assembly via HAR-based methods, as well as visual tool-recognition approaches. A novel online tool-recognition pipeline for handheld tools is introduced, utilizing a two-stage approach. First, a Region Of Interest (ROI) was extracted by determining the wrist position using skeletal data. Afterward, this ROI was cropped, and the tool located within this ROI was classified. This pipeline enabled several algorithms for object recognition and demonstrated the generalizability of our approach. An extensive training dataset for tool-recognition purposes is presented, which was evaluated with two image-classification approaches. An offline pipeline evaluation was performed with twelve tool classes. Additionally, various online tests were conducted covering different aspects of this vision application, such as two assembly scenarios, unknown instances of known classes, as well as challenging backgrounds. The introduced pipeline was competitive with other approaches regarding prediction accuracy, robustness, diversity, extendability/flexibility, and online capability.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Towards Recognition of Human Actions in Collaborative Tasks with Robots: Extending Action Recognition with Tool Recognition Methods

Büsch

Koch

Schoepflin

et al. 2023

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…While these studies focus on learning actions, others focus on segmenting continuous actions into action primitives for tool use ( Ramirez-Amaro et al., 2014a ; Lioutikov et al., 2017 ). A similar line of research on general manipulation tasks is to recognize the tasks based on the classification of actions ( Ramirez-Amaro et al., 2014b ; 2015 ; Hu et al., 2014 ; Wölfel and Henrich, 2018 ; Shao et al., 2021 ; Koch et al., 2022 ). This approach attempts to ground action profiles to labels, either primitive labels or task labels, and do not relate actions to the effects on the objects being manipulated.…”

Section: Robot Tool Use Literaturementioning

confidence: 99%

Robot tool use: A survey

Qin

Brawer²,

Scassellati³

2023

Front. Robot. AI

View full text Add to dashboard Cite

Using human tools can significantly benefit robots in many application domains. Such ability would allow robots to solve problems that they were unable to without tools. However, robot tool use is a challenging task. Tool use was initially considered to be the ability that distinguishes human beings from other animals. We identify three skills required for robot tool use: perception, manipulation, and high-level cognition skills. While both general manipulation tasks and tool use tasks require the same level of perception accuracy, there are unique manipulation and cognition challenges in robot tool use. In this survey, we first define robot tool use. The definition highlighted the skills required for robot tool use. The skills coincide with an affordance model which defined a three-way relation between actions, objects, and effects. We also compile a taxonomy of robot tool use with insights from animal tool use literature. Our definition and taxonomy lay a theoretical foundation for future robot tool use studies and also serve as practical guidelines for robot tool use applications. We first categorize tool use based on the context of the task. The contexts are highly similar for the same task (e.g., cutting) in non-causal tool use, while the contexts for causal tool use are diverse. We further categorize causal tool use based on the task complexity suggested in animal tool use studies into single-manipulation tool use and multiple-manipulation tool use. Single-manipulation tool use are sub-categorized based on tool features and prior experiences of tool use. This type of tool may be considered as building blocks of causal tool use. Multiple-manipulation tool use combines these building blocks in different ways. The different combinations categorize multiple-manipulation tool use. Moreover, we identify different skills required in each sub-type in the taxonomy. We then review previous studies on robot tool use based on the taxonomy and describe how the relations are learned in these studies. We conclude with a discussion of the current applications of robot tool use and open questions to address future robot tool use.

show abstract

An Integrated Approach for Hand Motion Segmentation and Robot Skills Representation

Berger

Schilp

2023

Annals of Scientific Society for Assembly, Handling and Industrial Robotics 2022

View full text Add to dashboard Cite

In this work, an approach for robot skill learning from voice command and hand movement sequences is proposed. The motion is recorded by a 3D camera. The proposed framework consists of three elements. Firstly, a hand detector is applied on each frame to extract key points, which are represented by 21 landmarks. The trajectories of index finger tip are then taken as hand motion for further processing. Secondly, the trajectories are divided into five segments by voice command and finger moving velocities. These five segments are: reach, grasp, move, position and release, which are considered as skills in this work. The required voice commands are grasp and release, as they have short duration and can be viewed as discrete events. In the end, dynamic movement primitives are learned to represent reach, move and position. In order to show the result of the approach, a human demonstration of a pick-and-place task is recorded and evaluated.

show abstract

A Methods-Time-Measurement based Approach to enable Action Recognition for Multi-Variant Assembly in Human-Robot Collaboration

Cited by 14 publications

References 9 publications

Towards Recognition of Human Actions in Collaborative Tasks with Robots: Extending Action Recognition with Tool Recognition Methods

Towards Recognition of Human Actions in Collaborative Tasks with Robots: Extending Action Recognition with Tool Recognition Methods

Robot tool use: A survey

An Integrated Approach for Hand Motion Segmentation and Robot Skills Representation

Contact Info

Product

Resources

About