Vision-Based Haptic Feedback for Remote Micromanipulation in–SEM Environment

Abstract

This article presents an intuitive environment for remote micromanipulation composed of both haptic feedback and virtual reconstruction of the scene. To enable nonexpert users to perform complex teleoperated micromanipulation tasks, it is of utmost importance to provide them with information about the 3-D relative positions of the objects and the tools. Haptic feedback is an intuitive way to transmit such information. Since position sensors are not available at this scale, visual feedback is used to derive information about the scene. In this work, three different techniques are implemented, evaluated, and compared to derive the object positions from scanning electron microscope images. The modified correlation matching with generated template algorithm is accurate and provides reliable detection of objects. To track the tool, a marker-based approach is chosen since fast detection is required for stable haptic feedback. Information derived from these algorithms is used to propose an intuitive remote manipulation system that enables users situated in geographically distant sites to benefit from specific equipments, such as SEMs. Stability of the haptic feedback is ensured by the minimization of the delays, the computational efficiency of vision algorithms, and the proper tuning of the haptic coupling. Virtual guides are proposed to avoid any involuntary collisions between the tool and the objects. This approach is validated by a teleoperation involving melamine microspheres with a diameter of less than 2 l m between Paris, France and Oldenburg, Germany.

A Review of Haptic Feedback Teleoperation Systems for Micromanipulation and Microassembly

“…11. Software architecture of a teleoperation system enabling micromanipulation from France of objects situated in Germany [56]. Information about the relative positions of the objects and the tool is derived from SEM images and transmitted to the remote teleoperation system.…”

“…In [56], a 3D stereoscopic view of the scene is reconstructed based on the position of the object and the tool derived from images coming from a scanning electron microscope. Instead of transmitting full images, only two positions are sent between the two geographically distant sites to provide a real-time visual feedback (Fig.…”

Recent advances in nanorobotic manipulation inside scanning electron microscopes

“…Therefore, feedforward-and feedback-controlled nanopositioning were attempted for higher bandwidth and higher accuracy 30 . To achieve nanoscaled manipulation and assembly tasks, control approaches have been developed for tele-operated and automated nanomanipulation 49,123,124 . In tele-operated nanomanipulation, haptic and visual feedback are usually acquired and presented to a human operator.…”

A review of micro-devices assembly techniques and technology

“…Haptic-based approaches have been explored by several researchers to Fig. 1 A VR environment of a micro-assembly work cell facilitate interfacing with micro-assembly tools and environments in a range of process contexts [23][24][25][26][27].…”

“…Bolopion et al [27] presented an ecosystem for the remote micro-manipulation consisting of haptic feedback including the use of a virtual model. After underscoring the limitations associated with the use of micro-manipulation systems for inexperienced users, they developed the system with the use of a scanning electron microscope (SEM) for visual feedback and a haptic feedbackassisted cantilever type of a tool for component manipulation.…”