Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction

Zhou, Tianyu; Zhu, Qi; Du, Jing

doi:10.1016/j.aei.2020.101170

Cited by 85 publications

(12 citation statements)

References 87 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A future research direction in robotics is multirobot coordination between the base and adhesion robots [29], [30], which enables navigation and path planning of the climbing robot, similar to the proposal in Section V. Moreover, the proposed robot system can be used in industrial applications, such as the crack inspection process for social infrastructure. Considering the characteristics of the inspection process, human-robot interaction can be proposed, such as a teleoperation methodology to perform precise operations on air [31], [32], inspection methods using various deep learning technologies, or multimodal data fusion [33], [34]. Moreover, they can be applied to the proposed robot system to verify their effectiveness.…”

Section: Discussionmentioning

confidence: 99%

Development of Dual-Unit Ceiling Adhesion Robot System With Passive Hinge for Obstacle Traversal Under Kinodynamic Constraints

Song

Kang

2023

IEEE Access

View full text Add to dashboard Cite

The inspection of high ceilings or surfaces is important to ensure the quality and safety of infrastructure; however, the approach adopted by people or conventional robots is rather limited. Kinodynamic constraints (simultaneous kinematic and dynamic constraints) are presented by obstacles in real-life scenarios, such as suspended piping and wiring, which further limits the usability of robots. Therefore, this paper proposed an adhesion robot system that is attached to and maneuvers on flat and curved ceilings while traversing obstructions. To traverse obstacles, the robot comprises two units connected by a passive hinge mechanism. Traversal motion is achieved under adhesion force control with preexisting electric ducted fans without using separate hinge motors; thus, no additional weight is included. In addition to robot hardware, this study investigated the development of a control method based on dynamic analysis under the aforementioned kinodynamic constraints. Specifically, the proposed control algorithm considers the slipping and rollover conditions of the robot caused by the external force and moment applied to the unit during the obstacle traversal, respectively. The algorithm was systematically analyzed by conducting simulations to prevent the robot from experiencing adhesion failure, and the results were verified experimentally. The use of the robot in real-life scenarios was determined by performing feasibility tests in real-life applications.INDEX TERMS Ceiling inspection, climbing robot, dynamic analysis, electric ducted fan, kinodynamic constraint, mobile robot, passive hinge.

show abstract

Section: Discussionmentioning

confidence: 99%

Development of Dual-Unit Ceiling Adhesion Robot System With Passive Hinge for Obstacle Traversal Under Kinodynamic Constraints

Song

Kang

2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…They proposed a novel approach for rendering a PCVE from robot sensor data. A number of authors have extended this idea using different approaches to render PVCEs ( Schwarz et al, 2017 ; Lesniak and Tucker, 2018 ; Valenzuela-Urrutia et al, 2019 ), Voxel Virtual Environments (VVEs) ( Mossel and Kroeter, 2017 ; Zhou and Tuzel, 2018 ; Stotko et al, 2019 ; Li et al, 2022 ), and VEs composed of precomposed objects based on automated recognition of objects from the point cloud data ( Zhou et al, 2020 ).…”

Section: Literature Reviewmentioning

confidence: 99%

“…The sensors typically used to capture environmental features that can be used for digital twins, laser scanners and RGB-D cameras, generate point clouds from which digital twins can be composed. The significant size of point cloud data makes the transmission difficult and slow ( Zhou et al, 2020 ). Further, there are many factors that affect the density of the resultant point clouds, for example, scanner specifications, data gathering time, communication bandwidth, and environmental features (smoke, radiation etc .).…”

Section: Introductionmentioning

confidence: 99%

Impact of resolution, colour, and motion on object identification in digital twins from robot sensor data

Bremner

Giuliani

2022

Front. Robot. AI

View full text Add to dashboard Cite

This paper makes a contribution to research on digital twins that are generated from robot sensor data. We present the results of an online user study in which 240 participants were tasked to identify real-world objects from robot point cloud data. In the study we manipulated the render style (point clouds vs voxels), render resolution (i.e., density of point clouds and granularity of voxel grids), colour (monochrome vs coloured points/voxels), and motion (no motion vs rotational motion) of the shown objects to measure the impact of these attributes on object recognition performance. A statistical analysis of the study results suggests that there is a three-way interaction between our independent variables. Further analysis suggests: 1) objects are easier to recognise when rendered as point clouds than when rendered as voxels, particularly lower resolution voxels; 2) the effect of colour and motion is affected by how objects are rendered, e.g., utility of colour decreases with resolution for point clouds; 3) an increased resolution of point clouds only leads to an increased object recognition if points are coloured and static; 4) high resolution voxels outperform medium and low resolution voxels in all conditions, but there is little difference between medium and low resolution voxels; 5) motion is unable to improve the performance of voxels at low and medium resolutions, but is able to improve performance for medium and low resolution point clouds. Our results have implications for the design of robot sensor suites and data gathering and transmission protocols when creating digital twins from robot gathered point cloud data.

show abstract

“…Recently, 3D virtual environments have been built and used in fields that require learning or verification in various situations, such as autonomous driving or remote monitoring [1][2][3][4]. The virtual environment can be reconstructed to provide situations that can occur in the real environment using the 3D point cloud measured by light detection and range (LiDAR) [5][6][7][8].…”

Section: Introductionmentioning

confidence: 99%

DeepLabV3-Refiner-Based Semantic Segmentation Model for Dense 3D Point Clouds

Kwak

Sung

2021

Remote Sensing

View full text Add to dashboard Cite

Three-dimensional virtual environments can be configured as test environments of autonomous things, and remote sensing by 3D point clouds collected by light detection and range (LiDAR) can be used to detect virtual human objects by segmenting collected 3D point clouds in a virtual environment. The use of a traditional encoder-decoder model, such as DeepLabV3, improves the quality of the low-density 3D point clouds of human objects, where the quality is determined by the measurement gap of the LiDAR lasers. However, whenever a human object with a surrounding environment in a 3D point cloud is used by the traditional encoder-decoder model, it is difficult to increase the density fitting of the human object. This paper proposes a DeepLabV3-Refiner model, which is a model that refines the fit of human objects using human objects whose density has been increased through DeepLabV3. An RGB image that has a segmented human object is defined as a dense segmented image. DeepLabV3 is used to make predictions of dense segmented images and 3D point clouds for human objects in 3D point clouds. In the Refiner model, the results of DeepLabV3 are refined to fit human objects, and a dense segmented image fit to human objects is predicted. The dense 3D point cloud is calculated using the dense segmented image provided by the DeepLabV3-Refiner model. The 3D point clouds that were analyzed by the DeepLabV3-Refiner model had a 4-fold increase in density, which was verified experimentally. The proposed method had a 0.6% increase in density accuracy compared to that of DeepLabV3, and a 2.8-fold increase in the density corresponding to the human object. The proposed method was able to provide a 3D point cloud that increased the density to fit the human object. The proposed method can be used to provide an accurate 3D virtual environment by using the improved 3D point clouds.

show abstract

Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction

Cited by 85 publications

References 87 publications

Development of Dual-Unit Ceiling Adhesion Robot System With Passive Hinge for Obstacle Traversal Under Kinodynamic Constraints

Development of Dual-Unit Ceiling Adhesion Robot System With Passive Hinge for Obstacle Traversal Under Kinodynamic Constraints

Impact of resolution, colour, and motion on object identification in digital twins from robot sensor data

DeepLabV3-Refiner-Based Semantic Segmentation Model for Dense 3D Point Clouds

Contact Info

Product

Resources

About