Real-time object tracking for the robot-based nanohandling in a scanning electron microscope

Abstract

In this paper, current research work on an automated nanohandling station using mobile microrobots is presented. For the automated positioning of mobile microrobots a closed-loop control system is necessary, usually using data from pose sensors for the degrees of freedom (DOF) of the microrobot are needed. Mobile microrobots with piezo slip-stick actuation mostly do not have internal pose sensors to determine a global pose. This paper focuses on the continuous pose estimation (tracking) of mobile microrobots by external visual sensors. One possibility for fast pose estimation is the application of video cameras in combination with image processing algorithms as global sensors. However, for pose estimation with accuracy in the nanometer range high-resolution sensors are necessary. In consideration of resolution, image acquisition time and depth of focus a scanning electron microscope (SEM) is a powerful sensor for high-resolution pose estimation of a microrobot. On the other hand, the use of a SEM requires high demands on the image processing. High update rates of the pose data for the robot control require a short image acquisition time of the SEM images. As a result, the image noise increases as frame averaging or averaging of the detector signal is time consuming. This paper presents two approaches to tracking a micro-object in a SEM image stream. First, a cross-correlation algorithm is described, which enables pose estimation (x, y, ϕ) in extremely noised images in real-time. Afterwards object tracking with active contours is presented. This approach allows real-time tracking with more than 3 DOF by using shape spaces, instead of defining large model sets as it is necessary for correlation-based pattern matching.

Development of automated microrobot-based nanohandling stations for nanocharacterization

“…3). The raw video data provided by SEM or CCD cameras are processed by the vision program which translates this input into poses (position and orientation) using a correlation-based approach (Sievers and Fatikow 2006). Readings of the piezo-resistive AFM probe are analyzed, converted into force data by the force reader program and sent to a common sensor server program.…”

Recent advances in nanorobotic manipulation inside scanning electron microscopes

“…State-of-the-art SEM-based nanomanipulation systems are also integrated with AFMs and focused ion beam (FIB) systems, as well as various tools 38 and exchangeable toolboxes 39 . With these advances, powerful nano-laboratories have been established that are capable of simultaneous imaging, fabrication, and nanomanipulation [40][41][42][43] with high efficiency and reproducibility via the use of emerging automation techniques 30,[44][45][46] .…”

Vision-Based Haptic Feedback for Remote Micromanipulation in–SEM Environment

“…To manipulate smaller objects' specific equipments, scanning electron microscopes (SEM) must be used to visualize the scene. For detection and tracking of objects and tools in SEM images, common algorithms include rigid-body-based matching (Kratochvil, Dong and Nelson 2009), active contours (Sievers, 2008), and template matching using cross correlation (Sievers and Fatikow 2006). Though these algorithms are flexible and robust, not all of them are suitable for the simultaneous detection of a high number of simple objects, like melamine spheres.…”