An AI-powered Hierarchical Communication Framework for Robust Human-Robot Collaboration in Industrial Settings

Mukherjee, Debasmita; Gupta, K. C.; Najjaran, Homayoun

doi:10.1109/ro-man53752.2022.9900601

Cited by 4 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Contradictory information is handled by semantic technologies to construct a command for the robot. Directional hand gestures and corresponding voice commands were Personalization Human-Robot Communication based on User Feedback used to guide a human-robot handover task in (Mukherjee et al 2022b). The fusion architecture was designed using a fuzzy inference system and Dempster Shafer theory to handle conflicting, complementary, and ambiguous commands from multimodal communication.…”

Section: Multimodal Communication In Hrcmentioning

confidence: 99%

“…On that note, in order to ensure the safety of the human partner during an HRC task, the former's focus on the task gauged through "positive" communication signals must be continuously gathered and recognized. A particular scenario has been examined in this paper built upon the general framework presented in a previous work (Mukherjee et al 2022b) particularly on whether the human partner is "engaged" in the task. Here "engaged" refers to the lexical meaning of having one's interest or attention occupied.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Personalization of industrial human–robot communication through domain adaptation based on user feedback

Mukherjee,

Hong,

Vats

et al. 2024

User Model User-Adap Inter

Self Cite

View full text Add to dashboard Cite

Achieving safe collaboration between humans and robots in an industrial work-cell requires effective communication. This can be achieved through a robot perception system developed using data-driven machine learning. The challenge for human-robot communication is the availability of extensive, labelled datasets for training. Due to the variations in human behaviour and the impact of environmental conditions on the performance of perception models, models trained on standard, publicly available datasets fail to generalize well to domain and applicationspecific scenarios. Thus, model personalization involving the adaptation of such models to the individual humans involved in the task in the given environment would lead to better model performance. A novel framework Personalization Human-Robot Communication based on User Feedback is presented that leverages robust modes of communication and gathers feedback from the human partner to auto-label the mode with the sparse dataset. The strength of the contribution lies in using in-commensurable multimodes of inputs for personalizing models with user-specific data. The personalization through feedback-enabled human-robot communication (PF-HRCom) framework is implemented on the use of facial expression recognition as a safety feature to ensure that the human partner is engaged in the collaborative task with the robot. Additionally, PF-HRCom has been applied to a real-time human-robot handover task with a robotic manipulator. The perception module of the manipulator adapts to the user's facial expressions and personalizes the model using feedback. Having said that, the framework is applicable to other combinations of multimodal inputs in human-robot collaboration applications.

show abstract

Section: Multimodal Communication In Hrcmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Personalization of industrial human–robot communication through domain adaptation based on user feedback

Mukherjee,

Hong,

Vats

et al. 2024

User Model User-Adap Inter

Self Cite

View full text Add to dashboard Cite

show abstract

Robotic pick-and-handover maneuvers with camera-based intelligent object detection and impedance control

Chen,

Wan,

Pan

2023

Trans. Can. Soc. Mech. Eng.

View full text Add to dashboard Cite

A vision-based impedance control method is applied to a 7-degree-of-freedom (7-DOF) Franka Emika (FE) Panda robotic manipulator to complete pick-and-handover tasks with human-like grasping of fruits. The interfaces between different hardware, controller, and path planner are developed using Robot Operating System (ROS). An RGB camera is applied using YOLOv5 (You Only Look Once v5) object detection algorithm, which is trained for various fruits and human hands, and then employed for identifying the location of the targeted objects. A qb-SoftHand robotic hand is used as the end-effector for the grasping tasks. By integrating these components together with ROS, the FE Panda robot successfully achieves autonomous human-like handover tasks as shown in the experimental results.

show abstract

A Critical Analysis of Industrial Human-Robot Communication and Its Quest for Naturalness Through the Lens of Complexity Theory

2022

View full text Add to dashboard Cite

Human-robot communication is one of the actively researched fields to enable efficient and seamless collaboration between a human and an intelligent industrial robotic system. The field finds its roots in human communication with the aim to achieve the “naturalness” inherent in the latter. Industrial human-robot communication pursues communication with simplistic commands and gestures, which is not representative of an uncontrolled real-world industrial environment. In addition, naturalness in communication is a consequence of its dynamism, typically ignored as a design criterion in industrial human-robot communication. Complexity Theory-based natural communication models allow for a more accurate representation of human communication which, when adapted, could also benefit the field of human-robot communication. This paper presents a perspective by reviewing the state of human-robot communication in industrial settings and then presents a critical analysis of the same through the lens of Complexity Theory. Furthermore, the work identifies research gaps in the aforementioned field, fulfilling which, would propel the field towards a truly natural form of communication. Finally, the work briefly discusses a general framework that leverages the experiential learning of data-based techniques and naturalness of human knowledge.

show abstract

An AI-powered Hierarchical Communication Framework for Robust Human-Robot Collaboration in Industrial Settings

Cited by 4 publications

References 17 publications

Personalization of industrial human–robot communication through domain adaptation based on user feedback

Personalization of industrial human–robot communication through domain adaptation based on user feedback

Robotic pick-and-handover maneuvers with camera-based intelligent object detection and impedance control

A Critical Analysis of Industrial Human-Robot Communication and Its Quest for Naturalness Through the Lens of Complexity Theory

Contact Info

Product

Resources

About