Data-Driven Microscopic Pose and Depth Estimation for Optical Microrobot Manipulation

Abstract

Optical microrobots have a wide range of applications in biomedical research for both in vitro and in vivo studies. In most microrobotic systems, the video captured by a monocular camera is the only way for capturing the movements of microrobots, and only planar motion can be captured by a monocular camera system. Accurate depth estimation is essential for the 3D reconstruction or autofocusing of micro-platforms, while the pose and depth estimation are necessary to enhance the 3D perception of the micro-robotic systems to enable dexterous micro-manipulation and other tasks. In this paper, we propose a data-driven method for pose and depth estimation in an optically manipulated microrobotic system. Focus measurement is used to obtain features for Gaussian Process Regression (GPR), which enables precise depth estimation. For mobile microrobots with varying poses, a novel method is developed based on a deep residual neural network with the incorporation of prior domain knowledge about the optical microrobots encoded via GPR. The method can simultaneously 1 track microrobots with complex shapes and estimate the pose and depth values of the optical microrobots.Cross-validation has been conducted to demonstrate the sub-micron accuracy of the proposed method and precise pose and depth perception for microrobots. We further demonstrate the generalisability of the method by adapting it to microrobots of different shapes using transfer learning with few-shot calibration. Intuitive visualization is provided to facilitate effective human-robot interaction during micro-manipulation based on pose and depth estimation results.

Online and Offline Interaction Model of International Chinese Education Based on Few-Shot Learning

“…In manipulated microbot systems, Zhang D proposed a data-driven approach to stability and depth analysis. It also demonstrates the method's generality by adapting to microrobots of various shapes using a multishot learning curve [ 4 ]. Aharchaou M gives an example of machine learning in action.…”

3D-printed microrobots from design to translation

“…DL was implemented to enable simultaneous tracking as well as pose and depth estimation of the optical microrobots 268 . Precise depth estimation of microrobots was made possible by the Gaussian process regression (GPR) algorithm and Deep Residual Network (ResNet) architecture.…”

Control and Autonomy of Microrobots: Recent Progress and Perspective

“…To date, lots of research utilizing machine learning algorithms in microrobotic applications, including kinetic model fitting, [62,120] gesture recognization, [48] and gait optimization [201,202] has been conducted. A neural network can be regarded as a black-box system.…”

“…Control theories [44,45] provide promising solutions for the automatic control of microrobots, and together with planning algorithms more practical applications like targeted navigation can be achieved. [46,47] Moreover, with intelligent algorithms like convolutional neural network (CNN) [48] and reinforcement learning, [49] microrobots possess the potential of performing higher level automatic behaviors. The overall workflow for automatic control of microrobots is shown in Figure 1.…”