Fast Organization of Objects’ Spatial Positions in Manipulator Space from Single RGB-D Camera

Sun, Yangchang; Yang, Minghao; Li, Jialing; Qiang, Baohua; Chen, Jinlong; Jia, Qingyu

doi:10.1007/978-3-030-92238-2_15

Cited by 1 publication

(9 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to the method proposed in Sun et al (2021), the DT generation comprises three steps, including object segmentation, object reconstruction and outlier rejection. In this study, YOLACT is adopted instead of the Mask-RCNN used in Sun et al (2021) to extract the objects from the RGB image. Besides, the left two steps are the same to those of Sun et al (2021).…”

Section: Methodsmentioning

confidence: 99%

“…There are mainly two types of eye-to-hand methods to reconstruct the contour points from RGB-D image to 3D manipulator workspace: geometrical methods (Meng and Zhuang, 2001; Boby and Saha, 2016; Ðurović et al , 2017) and neural network (NN) method (Sun et al , 2021; Wu et al , 2013; Jones and Vernon, 1994; Levine et al , 2018). The camera position setting and camera inner parameters are not strictly demanded in the NN eye-to-hand method, as compared with geometry methods.…”

Section: Related Workmentioning

confidence: 99%

“…The camera position setting and camera inner parameters are not strictly demanded in the NN eye-to-hand method, as compared with geometry methods. It made the NN method more flexible and convenient than geometry methods in 3D reconstruction of real practical environments (Sun et al , 2021). In this study, we prefer the NN method to the geometrical method in this study for the 3D reconstruction.…”

Section: Related Workmentioning

confidence: 99%

“…In this study, the correlation between P RGB− D and P 3D is constructed using the method proposed in Sun et al (2021). A 3-512-128-64-16-3 full connection neuronal network transfers RGB-D points from physical twin to virtual twin.…”

Section: Frameworkmentioning

confidence: 99%

“…A 3-512-128-64-16-3 full connection neuronal network transfers RGB-D points from physical twin to virtual twin. In the original DT reconstruction method proposed in Sun et al (2021), object contours are extracted from RGB images (He et al , 2017) using Mask-RCNN, and this step cost 200 ms per frame. It is needed to accelerate the 3D reconstruction.…”

Section: Frameworkmentioning

confidence: 99%

See 4 more Smart Citations

Enhance pick-and-place performance using multimodal interaction in operation environment

Guo

Chen

2023

View full text Add to dashboard Cite

Purpose Currently, the vision and depth information obtained from the eye-to-hand RGB-D camera can apply to the reconstruction of the three-dimensional (3D) environment for a robotic operation workspace. The reconstructed 3D space contributes to a symmetrical and equal observation view for robots and humans, which can be considered a digital twin (DT) environment. The purpose of this study is to enhance the robot skill in the physical workspace, although the artificial intelligence (AI) technique has high performance of the robotic operation in the known environments. Design/methodology/approach A multimodal interaction framework is proposed in DT operation environments. Findings A fast image-based target segmentation technique is combined in the 3D reconstruction of the robotic operation environment from the eye-to-hand camera, thus expediting the 3D DT environment generation without accuracy loss. A multimodal interaction interface is integrated into the DT environment. Originality/value The users are supported to operate the virtual objects in the DT environment using speech, mouse and keyboard simultaneously. The humans’ operations in 3D DT virtual space are recorded, and cues are provided for the robot’s operations in practice.

show abstract

Section: Methodsmentioning

confidence: 99%