An AR-Assisted Deep Learning-Based Approach for Automatic Inspection of Aviation Connectors

Abstract

The mismatched-pins inspection of complex aviation connector is a critical process to ensure the correct wiring harness assembly, of which the existing manual operation is error-prone and time-consuming. Aiming to fill this gap, this work proposes an augmented reality (AR)-assisted deep learning-based approach to tackle three major challenges in aviation connector inspection, including small pins detection, multi-pins sequencing, and mismatched pins visualization. Firstly, the proposed spatial-attention pyramid network approach extracts image features in multilayers and searches for their spatial relationships among images. Secondly, based on the cluster-generation sequencing algorithm, these detected pins are clustered into annuluses of expected layers and numbered according to their polar angles. Lastly, AR glass as the inspection visualization platform, highlights mismatched pins in the augmented interface to warn operators automatically. Compared with other existing methodologies, the experimental result shows that the proposed approach can achieve better performance accuracy, and support operator's inspection process efficiently.

Study of Augmented Reality Based Manufacturing for Further Integration of Quality Control 4.0: A Systematic Literature Review

“…The algorithm is a crucial element in developing an AR system. It consists of studies relevant to Artificial Intelligent methodology, establishing the base for AR to grow into intelligent systems [191]. A conceptual framework provides a general view of what the AR systems are and how they can be implemented.…”

“…Artificial Neural Networks (ANN) and Convolutional Neural Networks (CNN) are recently applied or proposed for further improvement of the registration methods by enhancing the CV-based tracking algorithms [74,191], while Industrial IoTs and Digital Twin are frequently considered in recent studies [37,190] to utilize Big data information and the advantages of AR in visualization, fusing digital data with a real working context.…”

“…Knowledge-based AR applications are designed to incorporate the domain knowledge of experts during the authoring phase to create a knowledge-based system (KBS) built on technical documents, manuals and other relevant documentation of the authoring domain (assembly, maintenance, quality control, etc.) [30,138,191]. There has been a steady interest in Interaction design for AR-based in a manufacturing context over the years, which reached its peak in 2019.…”

“…(KBS) built on technical documents, manuals and other relevant documentation of the authoring domain (assembly, maintenance, quality control, etc.) [30,138,191]. Agent-based AR utilizes the available entities of a system and their attributes to integrate into the AR solutions, supporting autonomy decision making [102,190].…”

MARMA: A Mobile Augmented Reality Maintenance Assistant for Fast-Track Repair Procedures in the Context of Industry 4.0

“…Werrlich et al [24] developed an AR system to support the engine assembly line in an automotive manufacturing environment. In the aviation industry, the impact of AR is evaluated, by applying techniques in inspection, robot programming, maintenance and process guidance [25,26]. In specific, Ceruti et al proposed a framework [27] to identify the fault parts with AR and reserve engineering methodologies, which scans and prints the fault part, using additive manufacturing techniques.…”