Towards achieving a high degree of situational awareness and multimodal interaction with AR and semantic AI in industrial applications

Abstract

With its various available frameworks and possible devices, augmented reality is a proven useful tool in various industrial processes such as maintenance, repairing, training, reconfiguration, and even monitoring tasks of production lines in large factories. Despite its advantages, augmented reality still does not usually give meaning to the elements it complements, staying in a physical or geometric layer of its environment and without providing information that may be of great interest to industrial operators in carrying out their work. An expert’s remote human assistance is becoming an exciting complement in these environments, but this is expensive or even impossible in many cases. This paper shows how a machine learning semantic layer can complement augmented reality solutions in the industry by providing an intelligent layer, sometimes even beyond some expert’s skills. This layer, using state-of-the-art models, can provide visual validation and new inputs, natural language interaction, and automatic anomaly detection. All this new level of semantic context can be integrated into almost any current augmented reality system, improving the operator’s job with additional contextual information, new multimodal interaction and validation, increasing their work comfort, operational times, and security.

Impact of Augmented Reality on Assistance and Training in Industry 4.0: Qualitative Evaluation and Meta-Analysis

“…In the field of immersive design, AR has established itself as a transformative tool in industrial design, providing users with deeply immersive virtual experiences. Izquierdo-Domenech et al [84] advanced this field by achieving intense situational immersion and multimodal interaction in industrial applications, all enhanced by artificial intelligence. Similarly, Fuertes et al [93] presented a methodology for the creation of demonstration models that utilise AR to facilitate the comprehension of advanced technologies.…”

Knowledge Graph-Based Framework to Support Human-Centered Collaborative Manufacturing in Industry 5.0

“…A knowledge reasoning framework has been suggested, using semantic data to improve real-time data processing in a smart factory setting [27]. A machine learning semantic layer has been introduced to complement augmented reality solutions in the industry, providing an intelligent layer [28]. Operators in an Industry 5.0 environment should be able to interact effortlessly with industrial assets while dealing with more complex assets.…”

Augmented Reality in Industry 4.0 Assistance and Training Areas: A Systematic Literature Review and Bibliometric Analysis

“…Enhanced interactivity and environmental analysis: The integration of semantic layers and advanced AR and AI technologies significantly improves operator interaction with the system, providing more efficient assistance in industrial tasks and better understanding and analysis of the environment [121]. The methodologies used in [133] demonstrate the ability of the AR system to provide tailored assistance, affirming the importance of AR for facilitating a more effective understanding and analysis of the industrial environment.…”