Rotation adaptive grasping estimation network oriented to unknown objects based on novel RGB-D fusion strategy

Tian, Hongkun; Song, Kechen; Li, Song; Ma, Shuai; Yan, Yunhui

doi:10.1016/j.engappai.2023.105842

Cited by 9 publications

(7 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our algorithm achieved state-of-the-art accuracy of 99.4% and 97.8% in image-wise and object-wise grasp detection, respectively, as shown in Table 2 . However, the average time expenditure is 17.7 ms, which is higher compared to algorithms in [ 10 , 11 , 12 ] due to the complexity of our algorithm.…”

Section: Discussionmentioning

confidence: 81%

“…H. Tian et al [ 11 ] introduced an intermediate-fusion method for lightweight pixel-wise robot grasp detection, utilizing RGB and depth information. In 2023, H. Tian et al [ 12 ] extended their work by introducing a rotation adaptive grasp detection approach, which also utilizes intermediate data fusion. They achieved a remarkable state-of-the-art accuracy of 99.3% and 94.6% on the Cornell and Jacquard datasets, respectively.…”

Section: Related Workmentioning

confidence: 99%

“…While these methods have yielded meaningful results, the correlation between multimodal data has only been partially exploited. Recent studies have focused on exploring the mechanisms of intermediate fusion [ 11 , 12 ]. Although these methods have improved the efficiency of RGB and depth modalities in robot grasp detection, making the most of the bilateral modal information still remains a challenge.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Bilateral Cross-Modal Fusion Network for Robot Grasp Detection

Zhang

XueYing

2023

Sensors

View full text Add to dashboard Cite

In the field of vision-based robot grasping, effectively leveraging RGB and depth information to accurately determine the position and pose of a target is a critical issue. To address this challenge, we proposed a tri-stream cross-modal fusion architecture for 2-DoF visual grasp detection. This architecture facilitates the interaction of RGB and depth bilateral information and was designed to efficiently aggregate multiscale information. Our novel modal interaction module (MIM) with a spatial-wise cross-attention algorithm adaptively captures cross-modal feature information. Meanwhile, the channel interaction modules (CIM) further enhance the aggregation of different modal streams. In addition, we efficiently aggregated global multiscale information through a hierarchical structure with skipping connections. To evaluate the performance of our proposed method, we conducted validation experiments on standard public datasets and real robot grasping experiments. We achieved image-wise detection accuracy of 99.4% and 96.7% on Cornell and Jacquard datasets, respectively. The object-wise detection accuracy reached 97.8% and 94.6% on the same datasets. Furthermore, physical experiments using the 6-DoF Elite robot demonstrated a success rate of 94.5%. These experiments highlight the superior accuracy of our proposed method.

show abstract

Section: Discussionmentioning

confidence: 81%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bilateral Cross-Modal Fusion Network for Robot Grasp Detection

Zhang

XueYing

2023

Sensors

View full text Add to dashboard Cite

show abstract

“…As shown in the figure, the quality heatmaps demonstrate the robustness of our proposed method, which contributes to the superior performance of our grasp detection results. We also conducted a comparative analysis of our grasp detection algorithm with that of several other methods [5,6,[8][9][10][11][12] using the Jacquard dataset. Table 3 presents the statistical results of our experiment with the Jacquard dataset.…”

Section: Cornell Dataset Experiments Resultsmentioning

confidence: 99%

“…Tian et al [11] introduced an intermediate-fusion method for lightweight pixelwise robot grasp detection, utilizing RGB and depth information. In 2023, H. Tian et al [12] extended their work by introducing a rotation adaptive grasp detection approach, which also utilizes intermediate data fusion. They achieved a remarkable state-of-the-art accuracy of 99.3% and 94.6% on the Cornell and Jacquard datasets, respectively.…”

Section: Multiple Modality Fusion Based Grasp Detectionmentioning

confidence: 99%

Bilateral Cross-Modal Fusion Network for Robot Grasp Detection

Zhang¹,

Sun²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

In the field of vision-based robot grasping, effectively leveraging RGB and depth information to accurately determine the position and pose of a target is a critical issue. To address this challenge, we propose a tri-stream cross-modal fusion architecture for 2-DoF visual grasp detection. This architecture facilitates the interaction of RGB and depth bilateral information and is designed to efficiently aggregate multiscale information. Our novel modal interaction module (MIM) with spatial-wise cross-attention algorithm adaptively captures cross-modal feature information. Meanwhile, the channel interaction modules (CIM) further enhance the aggregation of different modal streams. In addition, we efficiently aggregate global multiscale information through a hierarchical structure with skipping connections. To evaluate the performance of our proposed method, we conduct validation experiments on standard public datasets and real robot grasping experiments. We achieve the image-wise detection accuracy of 99.4% and 96.7% on Cornell and Jacquard datasets respectively. The object-wise detection accuracy reaches 97.8% and 94.6% on the same datasets. Furthermore, physical experiments using the 6-DoF Elite robot demonstrate a success rate of 94.5%. These experiments highlight the superior accuracy of our proposed method.

show abstract