Endo-VMFuseNet: A Deep Visual-Magnetic Sensor Fusion Approach for Endoscopic Capsule Robots

Turan, Mehmet; Almalioglu, Yasin; Gilbert, Hunter B.; Sari, Alp Eren; Soylu, Ufuk; Sitti, Metin

doi:10.1109/icra.2018.8461129

Cited by 11 publications

(10 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the future, the robot speed will be increased to achieve faster motion and further reduce procedure duration. Other examples of the manipulation of magnetic endoscopes are available in the literature 42 , but they either lack localization 27 , an endoscope tether (required for interventional capabilities) 43 or an intelligent control system 44 , limiting the translation to clinical use. This Article shows a tethered magnetic endoscope successfully navigating the colon of a porcine model by means of a blended use of magnetic localization, closed-loop robotic control and elaboration of the endoscope camera image.…”

Section: Discussionmentioning

confidence: 99%

Enabling the future of colonoscopy with intelligent and autonomous magnetic manipulation

et al. 2020

View full text Add to dashboard Cite

Early diagnosis of colorectal cancer substantially improves survival. However, over half of cases are diagnosed late due to the demand for colonoscopy-the 'gold standard' for screening-exceeding capacity. Colonoscopy is limited by the outdated design of conventional endoscopes, which are associated with high complexity of use, cost and pain. Magnetic endoscopes are a promising alternative and overcome the drawbacks of pain and cost, but they struggle to reach the translational stage as magnetic manipulation is complex and unintuitive. In this work, we use machine vision to develop intelligent and autonomous control of a magnetic endoscope, enabling non-expert users to effectively perform magnetic colonoscopy in vivo. We combine the use of robotics, computer vision and advanced control to offer an intuitive and effective endoscopic system. Moreover, we define the characteristics required to achieve autonomy in robotic endoscopy. The paradigm described here can be adopted in a variety of applications where navigation in unstructured environments is required, such as catheters, pancreatic endoscopy, bronchoscopy and gastroscopy. This work brings alternative endoscopic technologies closer to the translational stage, increasing the availability of early-stage cancer treatments.

show abstract

Section: Discussionmentioning

confidence: 99%

Enabling the future of colonoscopy with intelligent and autonomous magnetic manipulation

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Different from supervised VO learning [2], [4], [6], where camera poses and/or depth ground truths are required to train the neural network, the core idea underlying our unsupervised pose and depth prediction method is to make use of the view synthesis constraint as the supervision metric, which forces the neural network to synthesize target image from multiple source images acquired from different camera poses. This synthesis is performed using estimated depth image, estimated target camera pose values in 6-DoF and nearby color values from source images.…”

Section: Methodsmentioning

confidence: 99%

“…We think that DL based endoscopic VO approach is more suitable for such challenge areas since the operation environment(GI tract) has similar organ tissue patterns among different patients which can be learned by a sophisticated machine learning approach easily. Even the dynamics of common artefacts such as non-rigidness, sensor occlusions, vignetting, motion blur and specularity across frame sequences could be learned and used for a better pose estimation, whereas our unsupervised odometry learning method additionally solves the common problem of missing labels on medical datasets from inner body operations [4], [6].…”

Section: Introductionmentioning

confidence: 99%

Unsupervised Odometry and Depth Learning for Endoscopic Capsule Robots

Turan

Örnek²,

Ibrahimli³

et al. 2018

2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Self Cite

View full text Add to dashboard Cite

In the last decade, many medical companies and research groups have tried to convert passive capsule endoscopes as an emerging and minimally invasive diagnostic technology into actively steerable endoscopic capsule robots which will provide more intuitive disease detection, targeted drug delivery and biopsy-like operations in the gastrointestinal(GI) tract. In this study, we introduce a fully unsupervised, realtime odometry and depth learner for monocular endoscopic capsule robots. We establish the supervision by warping view sequences and assigning the re-projection minimization to the loss function, which we adopt in multi-view pose estimation and single-view depth estimation network. Detailed quantitative and qualitative analyses of the proposed framework performed on non-rigidly deformable ex-vivo porcine stomach datasets proves the effectiveness of the method in terms of motion estimation and depth recovery.

show abstract

“…Above all, neither the absolute localization algorithm based on the sensor coordinate system, nor the relative localization algorithm based on the human coordinate system can provide the appropriate tracking results that can be effectively used for further endoscopy operations. Some researchers have utilized the simultaneous localization and mapping (SLAM) method to obtain position information [28,29,30,31,32,33]. Usually, vision information has been used for tracking by applying the learning methods.…”

Section: Introductionmentioning

confidence: 99%

A Novel Relative Position Estimation Method for Capsule Robot Moving in Gastrointestinal Tract

Wang

Shi

Song

et al. 2019

Sensors

View full text Add to dashboard Cite

Recently, a variety of positioning and tracking methods have been proposed for capsule robots moving in the gastrointestinal (GI) tract to provide real-time unobstructed spatial pose results. However, the current absolute position-based result cannot match the GI structure due to its unstructured environment. To overcome this disadvantage and provide a proper position description method to match the GI tract, we here present a relative position estimation method for tracking the capsule robot, which uses the moving distance of the robot along the GI tract to indicate the position result. The procedure of the proposed method is as follows: firstly, the absolute position results of the capsule robot are obtained with the magnetic tracking method; then, the moving status of the robot along the GI tract is determined according to the moving direction; and finally, the movement trajectory of the capsule robot is fitted with the Bézier curve, where the moving distance can then be evaluated using the integral method. Compared to state-of-the-art capsule tracking methods, the proposed method can directly help to guide medical instruments by providing physicians the insertion distance in patients’ bodies, which cannot be done based on absolute position results. Moreover, as relative distance information was used, no reference tracking objects needed to be mounted onto the human body. The experimental results prove that the proposed method achieves a good distance estimation of the capsule robot moving in the simulation platform.

show abstract

Endo-VMFuseNet: A Deep Visual-Magnetic Sensor Fusion Approach for Endoscopic Capsule Robots

Cited by 11 publications

References 33 publications

Enabling the future of colonoscopy with intelligent and autonomous magnetic manipulation

Enabling the future of colonoscopy with intelligent and autonomous magnetic manipulation

Unsupervised Odometry and Depth Learning for Endoscopic Capsule Robots

A Novel Relative Position Estimation Method for Capsule Robot Moving in Gastrointestinal Tract

Contact Info

Product

Resources

About