Automatic Vision-Guided Micromanipulation for Versatile Deployment and Portable Setup

Abstract

In this paper, an automatic vision-guided micromanipulation approach to facilitate versatile deployment and portable setup is proposed. This work is motivated by the importance of micromanipulation and the limitations in existing automation technology in micromanipulation. Despite significant advancements in micromanipulation techniques, there remain bottlenecks in integrating and adopting automation for this application. An underlying reason for the gaps is the difficulty in deploying and setting up such systems. To address this, we identified two important design requirements, namely portability and versatility of the micromanipulation platform. A selfcontained vision-guided approach requiring no complicated preparation or setup is proposed. This is achieved through an uncalibrated self-initializing workflow algorithm also capable of assisted targeting. The feasibility of the solution is demonstrated on a low-cost portable microscope camera and compact actuated micro-stages. Results suggest subpixel accuracy in localizing the tool tip during initialization steps. The self-focus mechanism could recover intentional blurring of the tip by autonomously manipulating it 95.3 % closer to the focal plane. The average error in visual servo is less than a pixel with our depth compensation mechanism showing better maintaining of similarity score in tracking. Cell detection rate in a 1637-frame video stream is 97.7% with subpixels localization uncertainty. Our work addresses the gaps in existing automation technology in the application of robotic vision-guided micromanipulation and potentially contribute to the way cell manipulation is performed. Note to Practitioners-This paper introduces an automatic method for micromanipulation using visual information from microscopy. We design an automatic workflow, which consists of 1) self-initialization, 2) vision-guided manipulation and 3) assisted targeting, and demonstrate versatile deployment of the micromanipulator on a portable microscope camera setup. Unlike existing systems, our proposed method does not require any tedious calibration or expensive setup making it mobile and lowcost. This overcomes the constraints of traditional practices that confine automated cell manipulation to a laboratory setting. By extending the application beyond the laboratory environment, automated micromanipulation technology can be made more ubiquitous and expands readily to facilitate field study.

Microscopic Object Recognition and Localization Based on Multi-Feature Fusion for In-Situ Measurement In Vivo

“…We recorded the real-world distance from the ISME tip to the root in 25 images to test distance calculation precision of the proposed method. We calculated the root mean square error (RMS), mean error, and relative error [6,20]. The relative error of the tip location (RE T ) is defined as Equation (6), where A WP is the area of the working plane acquired by the CCD camera and RMS T is the root mean square error of the tip location of the test samples.…”

Advanced Biological Imaging for Intracellular Micromanipulation: Methods and Applications

“…The gradient of differences between the template and the recorded images was compared to determine if the micropipette was above the focal plane. A template similarity score was established to calculate template differences [149,150]. When the score is maximized, manipulation tools was aligned to the height of the focal plane.…”

Confidence-Based Hybrid Tracking to Overcome Visual Tracking Failures in Calibration-Less Vision-Guided Micromanipulation

“…To facilitate versatile deployment and portable setup of vision-guided micromanipulation, we have been developing a self-contained framework [17] that exploits the uncalibrated [18] vision-guided manipulation to achieve selfinitialization [19] and ease of deployment. It is self-contained in a sense that there is no need for an external active source of sensing other than the visual information from the microscope, which is already in place for most cell manipulation procedures.…”

“…2) Enhancing Vision-Guided Micromanipulation: The proposed hybrid tracking technique integrates seamlessly into our automated vision-guided micromanipulation system [17] including recent work on self-reinitialization and recovery [20]. The role of the hybrid tracking is illustrated in the flowchart, as shown in Fig.…”

“…Despite research efforts [14]- [16], including our own work [17], in developing a robust vision-guided system for cell manipulation, many of the methods are designed to work under a controlled environment. Unlike processes in microassembly, typical cell manipulation procedures rely highly on manual operation.…”