Intelligent Multimodal Framework for Human Assistive Robotics Based on Computer Vision Algorithms

Abstract

Assistive technologies help all persons with disabilities to improve their accessibility in all aspects of their life. The AIDE European project contributes to the improvement of current assistive technologies by developing and testing a modular and adaptive multimodal interface customizable to the individual needs of people with disabilities. This paper describes the computer vision algorithms part of the multimodal interface developed inside the AIDE European project. The main contribution of this computer vision part is the integration with the robotic system and with the other sensory systems (electrooculography (EOG) and electroencephalography (EEG)). The technical achievements solved herein are the algorithm for the selection of objects using the gaze, and especially the state-of-the-art algorithm for the efficient detection and pose estimation of textureless objects. These algorithms were tested in real conditions, and were thoroughly evaluated both qualitatively and quantitatively. The experimental results of the object selection algorithm were excellent (object selection over 90%) in less than 12 s. The detection and pose estimation algorithms evaluated using the LINEMOD database were similar to the state-of-the-art method, and were the most computationally efficient.

MANTRA: An Effective System Based on Augmented Reality and Infrared Thermography for Industrial Maintenance

“…An important contribution is our optimization of the original LINEMOD method. Although the original work has low computational cost-thanks largely to the use of SSE instructions in the search for the most similar template in the image-in this work, various optimizations were also made, such as those proposed by Ivorra et al [35], in such a way that the computational cost of the detection method was almost halved. Specifically, feature extractions were performed in separate threads for each color and depth images and numerous loops were parallelized.…”

“…Another way to estimate the pose of a 3D object is by using an RGB-D or depth camera and calculate the local 3D geometric descriptors of the 3D model, such as Point Pair Features (PPFs) [27]. The main problem with these methods is that they tend to be very computationally expensive [15,35], preventing their use in AR systems. Another alternative is using template matching techniques, such as the LINEMOD method [15].…”

“…The average time execution values are compiled in Table 5. The time efficiency of the optimized LINEMOD method [35] without using the ICP algorithm was 0.069 s (14.49 fps). On the other hand, as mentioned in Section 3.2.2, incorporation of the YOLOV4 detection algorithm into the preprocessing phase sped up the LINEMOD calculation to 0.035 s but, to this time, the YOLOV4 execution time of 0.036 should be added, such that the LINEYOLO total execution time was 0.071 s. In summary, the overall initial detector and pose estimation phase execution time was 0.122 s (8.19 fps), including the Parallel ICP using LINEYOLO (option 2), or 0.12 s without YOLOV4 (option 1).…”

Development of thermographic module for predictive maintenance system of industrial equipment

“…10.21611/qirt.2020.073 In augmented reality applications it is fundamental to have an accurate and fast pose estimation algorithm to be useful and realistic for a user. The method employed for the pose estimation engine was the algorithm implemented in [7]. This method is based in an optimized version of the template matching algorithm known as LINEMOD with a refinement step using Iterative Close Point [8].…”

Assistance Robotics and Biosensors

“…In the same field of assistance, the paper “Intelligent Multimodal Framework for Human Assistive Robotics Based on Computer Vision Algorithms” [ 2 ] shows a multimodal interface based on computer vision, which has been integrated into a robotic system together with other sensory systems (electrooculography (EOG) and electroencephalography (EEG)). The results were part of an European project, AIDE, whose purpose is to contribute to the improvement of current assistance technologies.…”