Towards Industrial IoT-AR Systems using Deep Learning-Based Object Pose Estimation

Abstract

Augmented Reality (AR) is known to enhance user experience, however, it remains under-adopted in industry. We present an AR interaction system improving human-machine coordination in Internet of Things (IoT) and Industry 4.0 applications including manufacturing and assembly, maintenance and safety, and other highly-interactive functions. A driver of slow adoption is the computational complexity and inaccuracy in localization and rendering digital content. AR systems may render digital content close to the associated physical objects, but traditional object recognition and localization modules perform poorly when tracking texture-less objects and complex shapes, presenting a need for robust and efficient digital content rendering techniques. We propose a method of improving IoT-AR by integrating Deep Learning with AR to increase accuracy and robustness of the target object localization module, taking both color and depth images as input and outputting the target's pose parameters. Quantitative and qualitative experiments prove this system's efficacy and show potential for fusing these emerging technologies in real-world applications.

Ambient Intelligence for Next-Generation AR

“…Similarly, Sun et al [190] presented MagicHand, an AR system that allows users to interact with IoT devices by detecting, localizing, and enabling augmented hand controls; the system was implemented using a 2D convolutional neural network (CNN) and achieved high gesture recognition accuracy. In [191] IoT sensor data were overlaid onto industrial machines using AR, with more accurate pose estimates (and thereby better aligned overlays) obtained by applying deep learning to RGB and depth images of the machine. Visualizing and identifying IoT objects using an AR interface has also been shown to improve shopping experiences, by increasing perceived usability and satisfaction in user interactions [85].…”

Extended Reality (XR) and Gamification in the context of the Internet of Things (IoT) and Artificial Intelligence (AI)

“…Data collection, sharing, analysis, and action at scale [35] may include systems ranging from consumer devices to critical infrastructure [39,40,41] such as smart factories and automation systems. [42,43,44] IoT has also been used to monitor equipment [45] and to embed intelligence into "dumb" systems. [46,47] The net result is that IoT combines with other technological affordances such as Big Data and AI to create smarter, more responsive environments ranging from homes to factories, often building upon the "power of 1%" at scale (e.g.…”