The Fusion Strategy of 2D and 3D Information Based on Deep Learning: A Review

Abstract

Recently, researchers have realized a number of achievements involving deep-learning-based neural networks for the tasks of segmentation and detection based on 2D images, 3D point clouds, etc. Using 2D and 3D information fusion for the advantages of compensation and accuracy improvement has become a hot research topic. However, there are no critical reviews focusing on the fusion strategies of 2D and 3D information integration based on various data for segmentation and detection, which are the basic tasks of computer vision. To boost the development of this research domain, the existing representative fusion strategies are collected, introduced, categorized, and summarized in this paper. In addition, the general structures of different kinds of fusion strategies were firstly abstracted and categorized, which may inspire researchers. Moreover, according to the methods included in this paper, the 2D information and 3D information of different methods come from various kinds of data. Furthermore, suitable datasets are introduced and comparatively summarized to support the relative research. Last but not least, we put forward some open challenges and promising directions for future research.

Survey on Quality of Experience Evaluation for Cloud-Based Interactive Applications

“…Other compression strategies use information from the rendering of video games to estimate movements and save compression time [29]. Graphics compression strategies generate a 3D coordinate space with information on the points of the objects that make up the scene and the 2D texts that cover them [79]. The server compresses this scene and sends it to the client, who must render that information.…”

Material augmented semantic segmentation of point clouds for building elements

“…It is worth noting that images and point clouds offer complementary information—images provide higher resolution and rich textures, while point clouds capture depth and scale details. The fusion of these two sources of information can be generally classified into three categories (Zhao et al., 2021). A rudimentary method is to directly colorize point clouds with registered images, but this approach may lead to the loss of appearance information, attributed to the inherent sparsity of 3D points.…”

Point Cloud Filtering Using 2D-3D Matching Method

“…Słowa kluczowe: chmury punktów 3D, filtracja chmur punktów, dopasowanie 2D lub algorytmy wykorzystujące dodatkowe informacje z wyspecjalizowanych kamer [39]. W pracy skupiono się na drugim podejściu: do odfiltrowania punktów odstających z chmury uzyskanej za pomocą kamery Time-of-Flight wykorzystujemy obraz z dodatkowej kamery z obiektywem telecentrycznym.…”