Mastering the Working Sequence in Human-Robot Collaborative Assembly Based on Reinforcement Learning

Abstract

A long-standing goal of the Human-Robot Collaboration (HRC) in manufacturing systems is to increase the collaborative working efficiency. In line with the trend of Industry 4.0 to build up the smart manufacturing system, the collaborative robot in the HRC system deserves better designing to be more selforganized and to find the superhuman proficiency by self-learning. Inspired by the impressive machine learning algorithms developed by Google Deep Mind like Alphago Zero, in this paper, the human-robot collaborative assembly working process is formatted into a chessboard and the selection of moves in the chessboard is used to analogize the decision-making by both human and robot in the HRC assembly working process. To obtain the optimal policy of the working sequence to maximize the working efficiency, agents in the system are trained with a self-play algorithm based on reinforcement learning, without guidance or domain knowledge beyond game rules. A convolution neural network (CNN) is also trained to predict the distribution of the priority of move selections and whether a working sequence is the one resulting in the maximum of the HRC efficiency. A height-adjustable standing desk assembly is used to demonstrate the proposed HRC assembly algorithm and its efficiency in real-time task planning. INDEX TERMS Human-Robot collaboration, real-time task planning, reinforcement learning, convolution neural network.

Control Techniques for Safe, Ergonomic, and Efficient Human-Robot Collaboration in the Digital Industry: A Survey

“…For instance, Yu et al make in [121], an analogy between a chessboard and the assembly process. The selection of moves in the chessboard are compared with the HR decision-making.…”

“…Due to the intrinsic structure of this type of networks, it is proven that they are faster than typical back propagation networks when trained by a supervised learning method like in [44], [45]. Alternatively, convolutional NNs [40], [41], [57], [121] are chosen for object detection and categorization, but it is demonstrated that in the majority of cases they cannot be applied in real-time and are used with a limited number of object categories.…”

“…All the aforementioned control strategies based on learning have been developed to address uncertainties and external disturbances that might provoke the robot's performance degradation by replacing the traditional proportional-integralderivative controllers, typically characterized by a complicated tuning of control parameters. Since the above described methods suffer from several problems, ranging from a huge computation time to a limited generalizability or adaptability to unseen situations, NNs based on modern control theories -e.g., SMC [89], [90], Takagi-Sugeno fuzzy control [57], and RL [70], [109], [112], [121], [133], [141]-are introduced in the literature and modeled to overcome these complex robot's control issues. Obviously, also these innovative advanced techniques present limitations, such as chattering and sensitive problems for the SMC and possible instabilities for fuzzy approaches.…”

Deep reinforcement learning in production systems: a systematic literature review

“…A multi-component assembly sequence planning approach to increase human-robot collaboration efficiencies was proposed by Yu, Huang, and Chang (2020). Assuming an adjustable desk as an example, the scheduling process was transformed into a chessboard-shaped planning structure that was able to complete planning significantly faster than conventional methods.…”

Applications of Explainable AI for 6G: Technical Aspects, Use Cases, and Research Challenges

“…Compared with traditional featurebased approaches being more complex and time-consuming for labelling a huge amount of data, the proposed method can annotate data labels automatically by perceiving human demonstrations. In another work [135], RL with CNN architecture was leveraged to optimize the working sequence of human-robot collaborative assembly, which in turn increases the working performance of smart manufacturing systems. Some complicated learning use cases, such as robot random failure and human behaviour uncertainties, were further taken into consideration to satisfy real-world conditions.…”