Micro-object pose estimation with sim-to-real transfer learning using small dataset

Abstract

Three-dimensional (3D) pose estimation of micro/nano-objects is essential for the implementation of automatic manipulation in micro/nano-robotic systems. However, out-of-plane pose estimation of a micro/nano-object is challenging, since the images are typically obtained in 2D using a scanning electron microscope (SEM) or an optical microscope (OM). Traditional deep learning based methods require the collection of a large amount of labeled data for model training to estimate the 3D pose of an object from a monocular image. Here we present a sim-to-real learning-to-match approach for 3D pose estimation of micro/nano-objects. Instead of collecting large training datasets, simulated data is generated to enlarge the limited experimental data obtained in practice, while the domain gap between the generated and experimental data is minimized via image translation based on a generative adversarial network (GAN) model. A learning-to-match approach is used to map the generated data and the experimental data to a low-dimensional space with the same data distribution for different pose labels, which ensures effective feature embedding. Combining the labeled data obtained from experiments and simulations, a new training dataset is constructed for robust pose estimation. The proposed method is validated with images from both SEM and OM, facilitating the development of closed-loop control of micro/nano-objects with complex shapes in micro/nano-robotic systems.

Fabrication and optical manipulation of micro-robots for biomedical applications

“…Moreover, the pose estimation was implemented with a relative orientation estimation mode, which means that accumulative errors may affect the accuracy of pose estimation. Therefore, future research should focus on learning from a small database with data-efficient approaches [57]. More recently, a ResNet-GPR hybrid model has been proposed, which can enable precise 6 DoFs pose estimation of optical microrobots (see Fig.…”

AI-enhanced biomedical micro/nanorobots in microfluidics

“…The method enabled advancement of closed-loop control in micro/nanorobotic systems to handle complex shaped micro/nano-objects. 140 Khiyati et al presented optimal control strategies for thin deformable microswimmers in viscous fluids. The approach addressed complex scenarios, in which the effects of non-homogeneous flow, limited configuration information, robot motion, and decision-making were intertwined.…”

Advanced medical micro-robotics for early diagnosis and therapeutic interventions

“…However, most of the deep learning-based methods require the collection of a large database. Therefore, simulation data has been combined with experimental data to implement a sim-to-real learning-to-match approach for pose estimation of micro-robots ( Zhang et al.,2022 ).…”