RGB-Depth Fusion GAN for Indoor Depth Completion

Wang, Haowen; Wang, Mingyuan; Che, Zhengping; Xu, Zhiyuan; Qiao, Xiuquan; Qi, Mingjing; Feng, Feifei; Tang, Jian

doi:10.1109/cvpr52688.2022.00611

Cited by 23 publications

(6 citation statements)

References 68 publications

(134 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To further investigate the effect of the proposed crossmodal attention module, we conducted a comparative analysis with other attention modules that have demonstrated success in various tasks. These attention modules, including Nonlocal [54] (2018), Criss-cross [55] (2019), Dual-attn [56] (2019), Point attention [57] (2020), W-AdaIN [58] (2022), and EMAF [59] (2023), were integrated into our framework. For a fair comparison, we removed our cross-modal attention TABLE 3: Object pose estimation results (AUC) with different attention modules, evaluated on the pallet RGBD dataset.…”

Section: Resultsmentioning

confidence: 99%

Occlusion-Robust Pallet Pose Estimation for Warehouse Automation

Vu,

Hoang,

Tan

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Accurate detection and estimation of pallet poses from color and depth data (RGB-D) are integral components many in advanced warehouse intelligent systems. State-of-the art object pose estimation methods follow a two-stage process, relying on off-the-shelf segmentation or object detection in the initial stage and subsequently predicting the pose of objects using cropped images. The cropped patches may include both the target object and irrelevant information, such as background or other objects, leading to challenges in handling pallets in warehouse settings with heavy occlusions from loaded objects. In this study, we propose an innovative deep learning-based approach to address the occlusion problem in pallet pose estimation from RGB-D images. Inspired by the selective attention mechanism in human perception, our developed model learns to identify and attenuate the significance of features in occluded regions, focusing on the visible and informative areas for accurate pose estimation. Instead of directly estimating pallet poses from cropped patches as in existing methods, we introduce two feature map re-weighting modules with cross-modal attention. These modules effectively filter out features from occluded regions and background, enhancing pose estimation accuracy. Furthermore, we introduce a large-scale annotated pallet dataset specifically designed to capture occlusion scenarios in warehouse environments, facilitating comprehensive training and evaluation. Experimental results on the newly collected pallet dataset show that our proposed method increases accuracy by 13.5% compared to state-of-the-art methods.

show abstract

Section: Resultsmentioning

confidence: 99%

Occlusion-Robust Pallet Pose Estimation for Warehouse Automation

Vu,

Hoang,

Tan

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Images were obtained from two popular datasets, namely the BEPM and OSU datasets. The fused results were obtained using CVT [35], DTCWT [36], LP [37], RP [38], MSVD [39], FusionGAN [40], and our proposed IFGAN. Based on the observations made in Figure 16, it becomes evident that our approach excels at preserving comprehensive details and distinctive characteristics derived from the viewable picture.…”

Section: Performance Comparison Of Ifgan With Other Image Fusion Modelsmentioning

confidence: 99%

IFGAN—A Novel Image Fusion Model to Fuse 3D Point Cloud Sensory Data

Ignatious,

El-Sayed,

Bouktif

2024

JSAN

View full text Add to dashboard Cite

To enhance the level of autonomy in driving, it is crucial to ensure optimal execution of critical maneuvers in all situations. However, numerous accidents involving autonomous vehicles (AVs) developed by major automobile manufacturers in recent years have been attributed to poor decision making caused by insufficient perception of environmental information. AVs employ diverse sensors in today’s technology-driven settings to gather this information. However, due to technical and natural factors, the data collected by these sensors may be incomplete or ambiguous, leading to misinterpretation by AVs and resulting in fatal accidents. Furthermore, environmental information obtained from multiple sources in the vehicular environment often exhibits multimodal characteristics. To address this limitation, effective preprocessing of raw sensory data becomes essential, involving two crucial tasks: data cleaning and data fusion. In this context, we propose a comprehensive data fusion engine that categorizes various sensory data formats and appropriately merges them to enhance accuracy. Specifically, we suggest a general framework to combine audio, visual, and textual data, building upon our previous research on an innovative hybrid image fusion model that fused multispectral image data. However, this previous model faced challenges when fusing 3D point cloud data and handling large volumes of sensory data. To overcome these challenges, our study introduces a novel image fusion model called Image Fusion Generative Adversarial Network (IFGAN), which incorporates a multi-scale attention mechanism into both the generator and discriminator of a Generative Adversarial Network (GAN). The primary objective of image fusion is to merge complementary data from various perspectives of the same scene to enhance the clarity and detail of the final image. The multi-scale attention mechanism serves two purposes: the first, capturing comprehensive spatial information to enable the generator to focus on foreground and background target information in the sensory data, and the second, constraining the discriminator to concentrate on attention regions rather than the entire input image. Furthermore, the proposed model integrates the color information retention concept from the previously proposed image fusion model. Furthermore, we propose simple and efficient models for extracting salient image features. We evaluate the proposed models using various standard metrics and compare them with existing popular models. The results demonstrate that our proposed image fusion model outperforms the other models in terms of performance.

show abstract

“…The authors of [23] propose a style transformation method for generating complete depth maps. The authors of [24] design a network with two branches: the first branch uses an encodingdecoding approach to convert sparse or incomplete depth maps into complete depth maps, while the other branch uses a GAN network to perform depth map style transformation on RGB images to generate depth maps for restoration. The authors of [25] use domain adaptation methods to design and train networks, generating geometric information or noise on synthetic datasets to mimic real datasets.…”

Section: Related Workmentioning

confidence: 99%

Depth Information Precise Completion-GAN: A Precisely Guided Method for Completing Ill Regions in Depth Maps

et al. 2023

View full text Add to dashboard Cite

In the depth map obtained through binocular stereo matching, there are many ill regions due to reasons such as lighting or occlusion. These ill regions cannot be accurately obtained due to the lack of information required for matching. Since the completion model based on Gan generates random results, it cannot accurately complete the depth map. Therefore, it is necessary to accurately complete the depth map according to reality. To address this issue, this paper proposes a depth information precise completion GAN (DIPC-GAN) that effectively uses the Guid layer normalization (GuidLN) module to guide the model for precise completion by utilizing depth edges. GuidLN flexibly adjusts the weights of the guiding conditions based on intermediate results, allowing modules to accurately and effectively incorporate the guiding information. The model employs multiscale discriminators to discriminate results of different resolutions at different generator stages, enhancing the generator’s grasp of overall image and detail information. Additionally, this paper proposes Attention-ResBlock, which enables all ResBlocks in each task module of the GAN-based multitask model to focus on their own task by sharing a mask. Even when the ill regions are large, the model can effectively complement the missing details in these regions. Additionally, the multiscale discriminator in the model enhances the generator’s robustness. Finally, the proposed task-specific residual module can effectively focus different subnetworks of a multitask model on their respective tasks. The model has shown good repair results on datasets, including artificial, real, and remote sensing images. The final experimental results showed that the model’s REL and RMSE decreased by 9.3% and 9.7%, respectively, compared to RDFGan.

show abstract

RGB-Depth Fusion GAN for Indoor Depth Completion

Cited by 23 publications

References 68 publications

Occlusion-Robust Pallet Pose Estimation for Warehouse Automation

Occlusion-Robust Pallet Pose Estimation for Warehouse Automation

IFGAN—A Novel Image Fusion Model to Fuse 3D Point Cloud Sensory Data

Depth Information Precise Completion-GAN: A Precisely Guided Method for Completing Ill Regions in Depth Maps

Contact Info

Product

Resources

About