Multimodal Human Activity Recognition for Industrial Manufacturing Processes in Robotic Workcells

Roitberg, Alina; Somani, Nikhil; Perzylo, Alexander; Rickert, Markus; Knoll, Alois

doi:10.1145/2818346.2820738

Cited by 66 publications

(16 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, the applied technologies cannot be considered as state-of-theart anymore. More recently, [6] and [20] explored visionbased HAR approaches in order to advance human-robot interaction. Decent classification accuracy results could be achieved for four different human activities, such as lowgranular tasks ("general assembly") and high-granular tasks e.g.…”

Section: ) Human Activity Recognition In Industrial Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Context-Aware Cyber-Physical Assistance Systems in Industrial Systems: A Human Activity Recognition Approach

Roth

Möncks

Bohné

et al. 2020

2020 IEEE International Conference on Human-Machine Systems (ICHMS)

View full text Add to dashboard Cite

The increasing demand for product customisation is leading to higher complexities within industrial systems. This imposes new challenges for the workforce. One way to support operators' productivity may be context-aware, human-centred cyber-physical assistance systems. Human Activity Recognition (HAR) is a promising approach to enable context-awareness. However, standardised approaches to integrate HAR into existing industrial environments are rare. Particularly, there is a lack of available datasets of industrial activities. Moreover, comparative studies of inertial and visual HAR approaches are still rare. This work therefore proposes Methods-Time Measurement (MTM) as a standardised foundation for creating an industrial activity dataset. Subsequently, five different machine learning algorithms are tested for their recognition performance based on the dataset captured with an inertial sensor suit and an RGB-D sensor. A proof-of-concept is delivered for both sensor categories applied to the scope of 18 MTM-1 activities, whereas inertial data outperformed depth data. K-Nearest Neighbour and Bagged Tree algorithms revealed the best classification accuracy results in this context.

show abstract

Section: ) Human Activity Recognition In Industrial Systemsmentioning

confidence: 99%

“…hand movements. The best result was achieved by Support Vector Machines (RBF 81%, linear 77%) [20]. In contrast, another project explored wearable sensors for the recognition of a single pick-and-place task [21].…”

Section: ) Human Activity Recognition In Industrial Systemsmentioning

confidence: 99%

Context-Aware Cyber-Physical Assistance Systems in Industrial Systems: A Human Activity Recognition Approach

Roth

Möncks

Bohné

et al. 2020

2020 IEEE International Conference on Human-Machine Systems (ICHMS)

View full text Add to dashboard Cite

show abstract

“…Based on the premise that activity recognition and process mining can be combined to extract tacit knowledge of operators in industrial processes, we conducted a search of Supervised Online Fine Real [9] Supervised Online Fine Lab [40] Supervised Online Fine Lab [35,36] Supervised Online Fine State machine Real [24] Supervised Online Fine Sequence Real [28] Supervised Online Coarse Real [10] Semi Online Both Hierarchy Lab [25] Semi Post-mortem Both Sequence Real 2015-2011 [27] Supervised Online Fine Lab [38] Supervised Online Coarse Workflow Real [37] Supervised Online Both Real [14] Supervised Predictive Fine State machine Lab [7] Supervised Online Coarse Workflow [32,33] Supervised Online Both Hierarchy Lab [8] Supervised Post-mortem Fine Probabilistic [34] Supervised Predictive Fine Rules Lab 2018-2016 [26] Supervised Online Fine Sequence Lab [21] Unsupervised Online Coarse Workflow Real [12] Supervised Online Fine Real [15] Supervised Online Coarse Real [11] Unsupervised Post-mortem Coarse Sequence Lab [17] Supervised Online Fine Lab [30] Supervised Online Fine Workflow Real [5] Semi Online Both Rules Lab the existing literature on activity recognition in industrial environments. Our goal was to derive a taxonomy that helps to identify the central issues and challenges of using activity recognition and process discovery for externalizing tacit knowledge.…”

Section: Literature Overviewmentioning

confidence: 99%

A Taxonomy for Combining Activity Recognition and Process Discovery in Industrial Environments

Mannhardt

Bovo

Oliveira

et al. 2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Despite the increasing automation levels in an Industry 4.0 scenario, the tacit knowledge of highly skilled manufacturing workers remains of strategic importance. Retaining this knowledge by formally capturing it is a challenge for industrial organisations. This paper explores research on automatically capturing this knowledge by using methods from activity recognition and process mining on data obtained from sensorised workers and environments. Activity recognition lifts the abstraction level of sensor data to recognizable activities and process mining methods discover models of process executions. We classify the existing work, which largely neglects the possibility of applying process mining, and derive a taxonomy that identifies challenges and research gaps.

show abstract

“…It is an interdisciplinary technology with a multitude of applications at a commercial, social, educational and industrial level. It is applicable to many aspects of the recognition and modeling of human activity, such as medical, rehabilitation, sports, surveillance cameras, dancing, human-machine interfaces, art and entertainment, and robotics [21][22][23][24][25][26][27][28][29]. In particular, gesture and posture recognition and analysis is essential for various applications such as rehabilitation, sign language, recognition of driving fatigue, device control and others [30].…”

Section: Introductionmentioning

confidence: 99%

Toward Modeling Psychomotor Performance in Karate Combats Using Computer Vision Pose Estimation

Echeverria

Santos

2021

Sensors

View full text Add to dashboard Cite

Technological advances enable the design of systems that interact more closely with humans in a multitude of previously unsuspected fields. Martial arts are not outside the application of these techniques. From the point of view of the modeling of human movement in relation to the learning of complex motor skills, martial arts are of interest because they are articulated around a system of movements that are predefined, or at least, bounded, and governed by the laws of Physics. Their execution must be learned after continuous practice over time. Literature suggests that artificial intelligence algorithms, such as those used for computer vision, can model the movements performed. Thus, they can be compared with a good execution as well as analyze their temporal evolution during learning. We are exploring the application of this approach to model psychomotor performance in Karate combats (called kumites), which are characterized by the explosiveness of their movements. In addition, modeling psychomotor performance in a kumite requires the modeling of the joint interaction of two participants, while most current research efforts in human movement computing focus on the modeling of movements performed individually. Thus, in this work, we explore how to apply a pose estimation algorithm to extract the features of some predefined movements of Ippon Kihon kumite (a one-step conventional assault) and compare classification metrics with four data mining algorithms, obtaining high values with them.

show abstract

Multimodal Human Activity Recognition for Industrial Manufacturing Processes in Robotic Workcells

Cited by 66 publications

References 16 publications

Context-Aware Cyber-Physical Assistance Systems in Industrial Systems: A Human Activity Recognition Approach

Context-Aware Cyber-Physical Assistance Systems in Industrial Systems: A Human Activity Recognition Approach

A Taxonomy for Combining Activity Recognition and Process Discovery in Industrial Environments

Toward Modeling Psychomotor Performance in Karate Combats Using Computer Vision Pose Estimation

Contact Info

Product

Resources

About