Extended Reality Application Framework for a Digital-Twin-Based Smart Crane

Abstract

Industry 4.0 is moving forward under technology upgrades, utilizing information technology to improve the intelligence of the industry, whereas Industry 5.0 is value-driven, aiming to focus on essential societal needs, values, and responsibility. The manufacturing industry is currently moving towards the integration of productivity enhancements and sustainable human employment. Such a transformation has deeply changed the human–machine interaction (HMI), among which digital twin (DT) and extended reality (XR) are two cutting-edge technologies. A manufacturing DT offers an opportunity to simulate, monitor, and optimize the machine. In the meantime, XR empowers HMI in the industrial field. This paper presents an XR application framework for DT-based services within a manufacturing context. This work aims to develop a technological framework to improve the efficiency of the XR application development and the usability of the XR-based HMI systems. We first introduce four layers of the framework, including the perception layer with the physical machine and its ROS-based simulation model, the machine communication layer, the network layer containing three kinds of communication middleware, and the Unity-based service layer creating XR-based digital applications. Subsequently, we conduct the responsiveness test for the framework and describe several XR industrial applications for a DT-based smart crane. Finally, we highlight the research challenges and potential issues that should be further addressed by analyzing the performance of the whole framework.

A Digital Twin Platform Integrating Process Parameter Simulation Solution for Intelligent Manufacturing

“…More importantly, they have provided confidence and solutions to establish direct communication between the Unity-developed DT system and the PLC-controlled systems. For the work of developing DT systems for PLC-controlled manufacturing systems, apart from [6], Yang et al [7] developed the DT for a crane typically used on factory shop floors, while communication in between the physical crane and its DT, again developed in Unity, is established by applying the OPC UA [8] technology. Furthermore, they integrated with their DY the VR/AR technology.…”

“…Similar work was reported by Minca et al [9], which also applied OPC UA for communication in between a physical assembly line controlled by the Siemens PLC system and its DT counterpart. The work of [6][7][8][9] provided a solution for communication issues with DT platforms which involves PLC control, yet they either introduced dedicated, chargeable, plug-ins (e.g., Game4Automation) or hardware components (since OPC UA requires additional hardware communication components and authority of dedicated protocol); thus, the solutions are not widely applicable and are also expensive.…”

TwinXR: Method for using digital twin descriptions in industrial eXtended reality applications

“…"Ilmatar" is a Konecranes CXT family crane with Siemens PLCs and can lift 3.2-ton loads along three dimensions using its subsystems of the bridge, trolley, and hoist. "Ilmatar" functions as a DT-based platform for students and researchers to conduct experiments on Smart Factory-related use cases and are equipped with various external digital solutions, including XR interfaces (Tu et al, 2021;Yang et al, 2022) for user interaction, and an OPC Unified Architecture (UA) server for data access. The metadata of such cranes includes the range of their operationally safe zone, the location for their target positioning control, as well as the basic information on their equipped external applications.…”

“…Industrial solutions combining the technologies of DT and XR have been proposed in several research works: Begout et al (2022) implemented a DT in the context of a reconfigurable factory with the help of an AR authoring tool; Podder et al (2022) leveraged a DT and its models in VR to enhance the learning experience and productivity of energy efficiency construction workers; Weistroffer et al (2022) proposed a framework to evaluate a cobotic workstation, by simulating a physics-based DT and using XR to display the DT and its associated data. Besides, our previous research explored the DT-XR combination by creating XR interfaces for a DT-based overhead crane: Tu et al (2021) implemented a Mixed Reality interface for on-site crane operation, while Yang et al (2022) proposed an XR development framework with crane virtual training and remote monitoring use cases. However, these works only focused on proof-of-concept technical design and implementation of industrial XR applications for DT systems.…”

New Game Artificial Intelligence Tools for Virtual Mine on Unreal Engine

“…The analysis of literary sources showed the high contribution of scientists and practitioners in the development of computer applications, the use of virtual and augmented reality technologies, as well as artificial intelligence objects [8][9][10][11][12][13][14][15][16][17][18]. These works have covered many diverse scientific and technical fields, from the development of object detection systems [19], education [20,21], cinematography [22], equipping scientific laboratories [23], solving tourism problems [24], modeling and optimization [25], to gaming applications [26][27][28].…”