An Autonomous Eye-in-Hand Robotic System for Elevator Button Operation Based on Deep Recognition Network

Zhu, Delong; Min, Zhe; Zhou, Tong; Li, Tingguang; Meng, Max Q.-H.

doi:10.1109/tim.2020.3043118

Cited by 8 publications

(10 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are two reasons: (6A1) the elevator button detection is not robust, which affected by light conditions, (6A2) the errors of the AMR and the camera are not compensated perfectly by the manipulator, (6A3) jerky motions sometimes happened because of the system communication. By contrast, the method in Zhu et al (2021) is proved to detect the elevator button robustly in various environments. Its failed execution is because of big orientation errors of the AMR and they did not have retrieving actions as well as the button light-up status checking.…”

Section: Resultsmentioning

confidence: 96%

“…Its failed execution is because of big orientation errors of the AMR and they did not have retrieving actions as well as the button light-up status checking. In the future, the elevator button recognition method in Zhu et al (2021) can be our reference to improve the performance of our robot system.…”

Section: Resultsmentioning

confidence: 99%

“…The orientation error can be measured by the motor-based odometry or by the odom sensor. In addition, these scenarios are also conducted by using the method in Zhu et al (2021), whose purpose is to give comparisons between our proposed method and previous methods. In general, the method in Zhu et al (2021) is chosen as a representative of previous methods to be compared with our method as its performance is good, and the elevator button detection is robustly in various environments.…”

Section: Experiments 61 Experimental Setupmentioning

confidence: 99%

“…In other words, the error of a sub-process is unable to be compensated; (1A3) there is no concern of the simultaneous elevator button detection and manipulation; (1A4) the safe termination mechanism for the elevator button pressing is not considered. Most recently, an autonomous eye-in-hand robotic system for the elevator button operation is presented in Zhu et al (2021), known as an alternative development of Dong et al (2017) and Zhu et al (2018). In this research, a deep neural network for the elevator button detection, a button pose estimation and a coarse-to-fine control schema are presented.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

An efficient approach for the elevator button manipulation using the visual-based self-driving mobile manipulator

Toan

Jeong²,

Jo³

2022

View full text Add to dashboard Cite

Purpose Because mobile manipulators are unable to climb stairs, the elevator operation is a crucial capacity to help those kinds of robot systems work in modern multifloor buildings. Here, the elevator button manipulation is considered as an efficient approach to fulfill that requirement. Previously, some studies presented elevator button recognition algorithms while some others designed schemes for the button manipulation work. However, the mobile robot, the manipulator and the camera in their robot systems are asynchronous. Besides, the time-consuming calibration for the camera is inevitable, especially in changeable environments. This paper aims to present an alternative method for the elevator button manipulation to overcome mentioned shortcomings. Design/methodology/approach In this paper, the elevator button manipulation is conducted by using the visual-based self-driving mobile manipulator in which the autonomous mobile robot, the manipulator and the camera cooperate more efficiently. Namely, the mobile robot does not need to be located exactly in front of the elevator panel as the manipulator has the ability to adjust the initial frame of the camera based on the system kinematic synchronization. In addition, the proposed method does not require the real world coordinates of elevator buttons, but uniquely using their pixel positions. By doing this, not only is the projection from two-dimensional pixel coordinates to three-dimensional (3D) real world coordinates unnecessary, but also the calibration of the camera is not required. Findings The proposed method is experimentally verified by using a visual-based self-driving mobile manipulator. This robotic system is the integration of an autonomous mobile robot, a manipulator and a camera mounted on the end-effector of the manipulator. Research limitations/implications Because the surface of the elevator button panel is usually mirror-like, the elevator button detection is easily affected by the glare and the brightness of the environmental light condition. Practical implications This robot system can be used for the goods delivery or the patrol in modern multifloor buildings. Originality/value This paper includes three new features: simultaneously detecting and manipulating elevator buttons without the projection from pixel coordinates to 3D real world coordinates, a kinematic synchronization to help the robot system eliminate accumulated errors and a safe human-like elevator button manipulation.

show abstract

Section: Resultsmentioning

confidence: 96%

Section: Resultsmentioning

confidence: 99%

Section: Experiments 61 Experimental Setupmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An efficient approach for the elevator button manipulation using the visual-based self-driving mobile manipulator

Toan

Jeong²,

Jo³

2022

View full text Add to dashboard Cite

show abstract

“…6D pose estimation of object is a key task for understanding the given scene, that is predicting the three-dimensional rotation and translation of the target object in the scene, and is useful in many real-word applications, such as robotic grasping and manipulation [1][2][3], augmented reality [4,5] and autonomous navigation [6,7]. However, such application scenarios are complex and changeable due to illumination changes, sensor noises, occlusion or even truncation between objects, so 6D object pose estimation in complex scenes is still a challenge.…”

Section: Introductionmentioning

confidence: 99%

Efficient 6D object pose estimation based on attentive multi‐scale contextual information

Gao

Sun

et al. 2022

IET Computer Vision

View full text Add to dashboard Cite

6D pose estimation has been pervasively applied to various robotic applications, such as service robots, collaborative robots, and unmanned warehouses. However, accurate 6D pose estimation is still a challenge problem due to the complexity of application scenarios caused by illumination changes, occlusion and even truncation between objects, and additional refinement is required for accurate 6D object pose estimation in prior work. Aiming at the efficiency and accuracy of 6D object pose estimation in these complex scenes, this paper presents a novel end-to-end network, which effectively utilises the contextual information within a neighbourhood region of each pixel to estimate the 6D object pose from RGB-D images. Specifically, our network first applies the attention mechanism to extract effective pixel-wise dense multimodal features, which are then expanded to multi-scale dense features by integrating pixel-wise features at different scales for pose estimation. The proposed method is evaluated extensively on the LineMOD and YCB-Video datasets, and the experimental results show that the proposed method is superior to several state-of-the-art baselines in terms of average point distance and average closest point distance.

show abstract

Goal and Force Switching Policy for DMP-Based Manipulation

Gorodetsky

Mironov

Pushkarev

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

An Autonomous Eye-in-Hand Robotic System for Elevator Button Operation Based on Deep Recognition Network

Cited by 8 publications

References 25 publications

An efficient approach for the elevator button manipulation using the visual-based self-driving mobile manipulator

An efficient approach for the elevator button manipulation using the visual-based self-driving mobile manipulator

Efficient 6D object pose estimation based on attentive multi‐scale contextual information

Goal and Force Switching Policy for DMP-Based Manipulation

Contact Info

Product

Resources

About