A deep learning based fusion of RGB camera information and magnetic localization information for endoscopic capsule robots

Turan, Mehmet; Shabbir, Jahanzaib; Araújo, Hélder; Konukoğlu, Ender; Sitti, Metin

doi:10.1007/s41315-017-0039-1

Cited by 37 publications

(20 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Monocular VO is confronted with numerous challenges such as large scale drift, the need for hand-crafted mathematical features and strict parameter tuning [3], [4]. Supervised deep learning based VO and depth recovery techniques have showed good performance in challenging environments and succesfuly alleviated issues such as scale drift, need for feature extraction and parameter finetuning [5]- [8]. VO as a regression problem in supervised deep learning exploits the capability of convolutional neural network (CNN) and recurrent neural network (RNN) to estimate camera motion, to calculate optical flow, and to extract efficient feature representations from raw RGB input [5]- [7], [9].…”

Section: Introductionmentioning

confidence: 99%

GANVO: Unsupervised Deep Monocular Visual Odometry and Depth Estimation with Generative Adversarial Networks

Almalioglu

Saputra

Gusmão

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

141

View full text Add to dashboard Cite

In the last decade, supervised deep learning approaches have been extensively employed in visual odometry (VO) applications, which is not feasible in environments where labelled data is not abundant. On the other hand, unsupervised deep learning approaches for localization and mapping in unknown environments from unlabelled data have received comparatively less attention in VO research. In this study, we propose a generative unsupervised learning framework that predicts 6-DoF pose camera motion and monocular depth map of the scene from unlabelled RGB image sequences, using deep convolutional Generative Adversarial Networks (GANs). We create a supervisory signal by warping view sequences and assigning the re-projection minimization to the objective loss function that is adopted in multi-view pose estimation and single-view depth generation network. Detailed quantitative and qualitative evaluations of the proposed framework on the KITTI [1] and Cityscapes [2] datasets show that the proposed method outperforms both existing traditional and unsupervised deep VO methods providing better results for both pose estimation and depth recovery.

show abstract

Section: Introductionmentioning

confidence: 99%

GANVO: Unsupervised Deep Monocular Visual Odometry and Depth Estimation with Generative Adversarial Networks

Almalioglu

Saputra

Gusmão

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

141

View full text Add to dashboard Cite

show abstract

“…However, current capsule endoscopes used in hospitals are passive devices controlled by peristaltic motions of the inner organs. The control over capsule's position, orientation, and functions would give the doctor a more precise reachability of targeted body parts and more intuitive and correct diagnosis opportunity [6,7,8,9,10]. Therefore, several groups have recently proposed active, remotely controllable robotic capsule endoscope prototypes equipped with additional functionalities such as local drug delivery, biopsy and other medical functions [11,2,12,13,14,15,16,17,18,19].…”

Section: Introductionmentioning

confidence: 99%

Deep EndoVO: A recurrent convolutional neural network (RCNN) based visual odometry approach for endoscopic capsule robots

et al. 2018

Self Cite

View full text Add to dashboard Cite

Ingestible wireless capsule endoscopy is an emerging minimally invasive diagnostic technology for inspection of the GI tract and diagnosis of a wide range of diseases and pathologies. Medical device companies and many research groups have recently made substantial progresses in converting passive capsule endoscopes to active capsule robots, enabling more accurate, precise, and intuitive detection of the location and size of the diseased areas. Since a reliable real time pose estimation functionality is crucial for actively controlled endoscopic capsule robots, in this study, we propose a monocular visual odometry (VO) method for endoscopic capsule robot operations. Our method lies on the application of the deep Recurrent Convolutional Neural Networks (RCNNs) for the visual odometry task, where Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) are used for the feature extraction and inference of dynamics across the frames, respectively. Detailed analyses and evaluations made on a real pig stomach dataset proves that our system achieves high translational and rotational accuracies for different types of endoscopic capsule robot trajectories.

show abstract

“…One of the highest potential scientific and social impacts of milli-scale, untethered, mobile robots is their healthcare applications. Swallowable capsule endoscopes with an onboard camera and wireless image transmission device have been commercialized and used in hospitals (FDA approved) since 2001, which has enabled access to regions of the GI tract that were impossible to access before, and has reduced the discomfort and sedation related work loss issues [1], [2], [3], [4]. However, with systems commercially available today, capsule endoscopy cannot provide precise (centimeter to millimeter accurate) localization of diseased areas, and active, wireless control remains a highly active area of research.…”

Section: Introductionmentioning

confidence: 99%

EndoSensorFusion: Particle Filtering-Based Multi-Sensory Data Fusion with Switching State-Space Model for Endoscopic Capsule Robots

Turan

Almalioglu

Gilbert

et al. 2018

2018 IEEE International Conference on Robotics and Automation (ICRA)

Self Cite

View full text Add to dashboard Cite

A reliable, real time multi-sensor fusion functionality is crucial for localization of actively controlled capsule endoscopy robots, which are an emerging, minimally invasive diagnostic and therapeutic technology for the gastrointestinal (GI) tract. In this study, we propose a novel multi-sensor fusion approach based on a particle filter that incorporates an online estimation of sensor reliability and a non-linear kinematic model learned by a recurrent neural network. Our method sequentially estimates the true robot pose from noisy pose observations delivered by multiple sensors. We experimentally test the method using 5 degree-of-freedom (5-DoF) absolute pose measurement by a magnetic localization system and a 6-DoF relative pose measurement by visual odometry. In addition, the proposed method is capable of detecting and handling sensor failures by ignoring corrupted data, providing the robustness expected of a medical device. Detailed analyses and evaluations are presented using ex-vivo experiments on a porcine stomach model prove that our system achieves high translational and rotational accuracies for different types of endoscopic capsule robot trajectories.

show abstract

A deep learning based fusion of RGB camera information and magnetic localization information for endoscopic capsule robots

Cited by 37 publications

References 43 publications

GANVO: Unsupervised Deep Monocular Visual Odometry and Depth Estimation with Generative Adversarial Networks

GANVO: Unsupervised Deep Monocular Visual Odometry and Depth Estimation with Generative Adversarial Networks

Deep EndoVO: A recurrent convolutional neural network (RCNN) based visual odometry approach for endoscopic capsule robots

EndoSensorFusion: Particle Filtering-Based Multi-Sensory Data Fusion with Switching State-Space Model for Endoscopic Capsule Robots

Contact Info

Product

Resources

About