Projection-based augmented reality system for assembly guidance and monitoring

Abstract

Purpose This study aims to monitor and guide the assembly process. The operators need to change the assembly process according to the products’ specifications during manual assembly of mass customized production. Traditional information inquiry and display methods, such as manual lookup of assembly drawings or electronic manuals, are inefficient and error-prone. Design/methodology/approach This paper proposes a projection-based augmented reality system (PBARS) for assembly guidance and monitoring. The system includes a projection method based on viewpoint tracking, in which the position of the operator’s head is tracked and the projection images are changed correspondingly. The assembly monitoring phase applies a method for parts recognition. First, the pixel local binary pattern (PX-LBP) operator is achieved by merging the classical LBP operator with the pixel classification process. Afterward, the PX-LBP features of the depth images are extracted and the randomized decision forests classifier is used to get the pixel classification prediction image (PCPI). Parts recognition and assembly monitoring is performed by PCPI analysis. Findings The projection image changes with the viewpoint of the human body, hence the operators always perceive the three-dimensional guiding scene from different viewpoints, improving the human-computer interaction. Part recognition and assembly monitoring were achieved by comparing the PCPIs, in which missing and erroneous assembly can be detected online. Originality/value This paper designed the PBARS to monitor and guide the assembly process simultaneously, with potential applications in mass customized production. The parts recognition and assembly monitoring based on pixels classification provides a novel method for assembly monitoring.

A posture detection method for augmented reality–aided assembly based on YOLO-6D

“…The measurement dataset is composed of part of the public dataset in LINEMOD [11] and a self-made plug-in dataset. The model for testing and validation is shown in Fig.…”

“…Lai et al [10] developed an intelligent augmented reality assisted assembly system towards assembly workers, and used the richly textured marker images as the basis for tracking registration. Chen et al [11] applied a trained classifier to measure the depth image and perform pixel classification to obtain the corresponding RGB values , which were compared them with the RGB values of the marker image to identify each part in the assembly process and it can be inferred the assembly position. Hinterstoisser [12,13] proposed the LINE2D algorithm which is only based on RGB image color gradient information and the LINEMOD algorithm which combines RGB image color gradient information and depth image normal vector information.…”

Study of Augmented Reality Based Manufacturing for Further Integration of Quality Control 4.0: A Systematic Literature Review

“…Field experiment [7,30,122,138,143,146,151,152,161,162] [ 165,167,181,188,193-196,198,199] [201,220,222,223,225] Novel stage Laboratory experiment Although the novel stage projects achieve potential results, user acceptance, humancentric issues, seamless user interaction and user interface are still challenges that need investigating for long-term industry adoption in manufacturing [2]. Because AR is a technology enhancing human perspectives by virtual and real context fusion, a universal human-centered model for AR-based solutions development can help in closing the gap between academia and industry implementations [236,237].…”

“…The projector is also popularly used in assisting the assembly process. A projection-based AR system was proposed to monitor and provide instruction for the operator during the assembly process in [161], or to provide the picking information together with assembly data for the operation as in [185]. Then, in a higher-level conceptual system, real-time instructions for assembly using SAR were comprehensively studied and demonstrated in [63].…”

“…Although the sensor-based technique is usually integrated with other tracking techniques to form hybrid tracking situations, as mentioned in the above part, it is rarely applied alone for AR-based solutions in manufacturing. This is because sensor-based tracking technically requires intensive tracking algorithms [11] and advanced deep learning methods such as CNN [161] to boost the tracking performance and reduce latency, thus achieving the real-time feature. Finally, a model-based tracking method is applied when a 3D CAD model of the tracked object's parts are available to extract, analyze and determine the pose and position for later recognition and tracking [8,73,139].…”