A hybrid camera- and ultrasound-based approach for needle localization and tracking using a 3D motorized curvilinear ultrasound probe

Daoud, Mohammad I.; Alshalalfah, Abdel-Latif; Mohamed, Otmane Aït; Alazrai, Rami

doi:10.1016/j.media.2018.09.006

Cited by 14 publications

(8 citation statements)

References 45 publications

(99 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…13 Beigi et al trained a probabilistic SVM using the temporal features for pixel classification and computed the probability map of the segmented pixels, followed by using Hough transform for needle localization.15 Besides, a hybrid camera-and USbased method was designed in the work of Daoud et al to localize and track the inserted needle. 16 Recently, deep convolutional neural networks (CNNs) show the capabilities on learning hierarchical features to build the mappings from data space to objective space.17 Because of the power of nonlinear fitting, CNNs have been shown to achieve outstanding performance on various medical image-based tasks,18 such as segmentation,19-23 cancer diagnosis,24 and localization.10 For needle segmentation and localization, Pourtaherian et al trained a CNN model to identify the needle voxels from other echogenic structures and also built a fully convolutional networks (FCN) to label the needle parts, where both methods were followed by RANSAC for needle-axis estimation and visualization.25 And they also made an attempt on adopting dilated CNNs to localize the partially visible needles in US images. 26 In addition, to implement automatic needle segmentation in MRI, Mehrtash et al presented a deep anisotropic 3D FCN with skip-connections10 that is inspired by the 3D U-Net model.27 Multi-needle detection in 3D US images for US-guided prostate brachytherapy is lacking attention in current studies.…”

Section: Introductionmentioning

confidence: 99%

“…Beigi et al trained a probabilistic SVM using the temporal features for pixel classification and computed the probability map of the segmented pixels, followed by using Hough transform for needle localization 15. Besides, a hybrid camera‐ and US‐based method was designed in the work of Daoud et al to localize and track the inserted needle 16. Recently, deep convolutional neural networks (CNNs) show the capabilities on learning hierarchical features to build the mappings from data space to objective space 17.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi‐needle Localization with Attention U‐Net in US‐guided HDR Prostate Brachytherapy

et al. 2020

View full text Add to dashboard Cite

Purpose: Ultrasound (US)-guided high dose rate (HDR) prostate brachytherapy requests the clinicians to place HDR needles (catheters) into the prostate gland under transrectal US (TRUS) guidance in the operating room. The quality of the subsequent radiation treatment plan is largely dictated by the needle placements, which varies upon the experience level of the clinicians and the procedure protocols. Real-time plan dose distribution, if available, could be a vital tool to provide more subjective assessment of the needle placements, hence potentially improving the radiation plan quality and the treatment outcome. However, due to low signal-to-noise ratio (SNR) in US imaging, real-time multi-needle segmentation in 3D TRUS, which is the major obstacle for real-time dose mapping, has not been realized to date. In this study, we propose a deep learning-based method that enables accurate and real-time digitization of the multiple needles in the 3D TRUS images of HDR prostate brachytherapy. Methods: A deep learning model based on the U-Net architecture was developed to segment multiple needles in the 3D TRUS images. Attention gates were considered in our model to improve the prediction on the small needle points. Furthermore, the spatial continuity of needles was encoded into our model with total variation (TV) regularization. The combined network was trained on 3D TRUS patches with the deep supervision strategy, where the binary needle annotation images were provided as ground truth. The trained network was then used to localize and segment the HDR needles for a new patient's TRUS images. We evaluated our proposed method based on the needle shaft and tip errors against manually defined ground truth and compared our method with other state-ofart methods (U-Net and deeply supervised attention U-Net). Results: Our method detected 96% needles of 339 needles from 23 HDR prostate brachytherapy patients with 0.290 AE 0.236 mm at shaft error and 0.442 AE 0.831 mm at tip error. For shaft localization, our method resulted in 96% localizations with less than 0.8 mm error (needle diameter is 1.67 mm), while for tip localization, our method resulted in 75% needles with 0 mm error and 21% needles with 2 mm error (TRUS image slice thickness is 2 mm). No significant difference is observed (P = 0.83) on tip localization between our results with the ground truth. Compared with U-Net and deeply supervised attention U-Net, the proposed method delivers a significant improvement on both shaft error and tip error (P < 0.05). Conclusions: We proposed a new segmentation method to precisely localize the tips and shafts of multiple needles in 3D TRUS images of HDR prostate brachytherapy. The 3D rendering of the needles could help clinicians to evaluate the needle placements. It paves the way for the development of real-time plan dose assessment tools that can further elevate the quality and outcome of HDR prostate brachytherapy.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Multi‐needle Localization with Attention U‐Net in US‐guided HDR Prostate Brachytherapy

et al. 2020

View full text Add to dashboard Cite

show abstract

“…3. Although there is a recent study focusing on 3D volumetric data with temporal information [49], they considered an extra camera for giving support information, which obtained a better performance than ROI-based Kalman filtering.…”

Section: ) Parametric Space Methodsmentioning

confidence: 99%

“…Needle biopsy/anesthesia/therapy ex-vivo Kaya et al [41] 2015 2D+t Needle biopsy/drug delivery in-vitro Pourtaherian et al [37] 2016 3D Needle anesthesia/ablation ex-vivo Beigi et al [45] 2016 2D+t Needle biopsy/nerve block/anesthesias in-vitro/in-vivo Mwikirize et al [43] 2016 2D Needle biopsy/ablation/anesthesia ex-vivo Beigi et al [48] 2016 2D+t Needle biopsy/nerve block/anesthesia in-vivo Kaya et al [46] 2016 2D+t Needle biopsy/drug delivery in-vitro Daoud et al [49] 2018 3D Needle intervention ex-vivo Daoud et al [24] 2018 2D Needle intervention ex-vivo Agarwal et al [35] 2019 2D+t Anesthesia/biopsy/brachytherapy in-vitro Needle biopsy/ablation/anesthesia ex-vivo Yang et al [61] 2019 3D Cardiac catheterization in-vitro/ex-vivo/in-vivo Yang et al [70] 2019 3D Cardiac catheterization ex-vivo Yang et al [82] 2019 3D Cardiac catheterization ex-vivo Yang et al [83] 2019 3D Cardiac catheterization ex-vivo Mwikirize et al [89] 2019 2D+t Needle biopsy/anesthesia in-vitro/ex-vivo Mwikirize et al [88] 2019 2D+t Needle biopsy/anesthesia ex-vivo Arif et al [87] 2019 3D Needle biopsy in-vitro/in-vivo Yang et al [85] 2019 3D Cardiac catheterization ex-vivo/in-vivo Min et al [76] 2020 3D Cardiac catheterization ex-vivo Rodgers et al [80] 2020 2D/3D Interstitial gynecologic brachytherapy in-vitro/in-vivo Zhang et al [68], [67] 2020 3D prostate brachytherapy in-vivo Zhang et al [86] 2020 3D Prostate brachytherapy in-vivo Zhang et al [90] 2020 3D Prostate brachytherapy in-vivo Lee et al [79] 2020 2D Needle biopsy in-vivo With the above summaries, there are remaining some challenges and limitations for this area, despite the current methods obtain satisfactory results. We discuss these challenges below.…”

Section: Referencementioning

confidence: 99%

Medical Instrument Detection in Ultrasound-Guided Interventions: A Review

Yang¹,

Shan²,

Kolen³

2020

Preprint

View full text Add to dashboard Cite

Medical instrument detection is essential for computer assisted interventions, since it would facilitate the surgeons to find the instrument efficiently with a better interpretation, which leads to a better outcome. This article reviews medical instrument detection methods in ultrasound-guided intervention. First, we present a comprehensive review for instrument detection methodologies, which include traditional non-datadriven methods and data-driven methods. The non-data-driven methods were extensive studied prior to the era of machine learning, i.e. data-driven approaches. We discuss the main clinical applications of medical instrument detection in ultrasound, including anesthesia, biopsy, prostate brachytherapy and cardiac catheterization, which were validated on clinical datasets. Finally, we selected several principal publications to summarize the key issues and potential research directions for the computer assisted intervention community.

show abstract

“…We used a multi-part loss function as in basic Yolo [58] to train the nYolo network. As shown in (9), the total loss is composed of three parts, the Needle-ness loss (£ 𝑁 ), the confidence score of the existence of the needle (£ 𝐶 ) and the regressor loss (£ 𝐿 ) to predict line parameters in each grid cell. The term 𝟙 𝑖 𝑁 denotes the existence and 𝟙 𝑖 𝑁𝑜𝑁 the absence of the needle in cell i.…”

Section: Nyolo Needle Detection Networkmentioning

confidence: 99%

Spatiotemporal analysis of speckle dynamics to track invisible needle in ultrasound sequences using Convolutional Neural Networks

Zade

Aziz

Majedi

et al. 2022

Preprint

View full text Add to dashboard Cite

ObjectiveAccurate needle placement to the target point is critical for ultrasound interventions like biopsies and epidural injections. However, aligning the needle to the thin plane of the transducer is a challenging issue as it leads to the decay of visibility by the naked eye. Therefore, we have developed a CNN-based framework to track the needle using the spatiotemporal features of speckle dynamics.MethodsThere are three key techniques to optimize the network for our application. First, we proposed a motion field estimation network (RMF) to extract spatiotemporal features from the stack of consecutive frames. We also designed an efficient network based on the state-of-the-art Yolo framework (nYolo). Lastly, the Assisted Excitation (AE) module was added at the neck of the network to handle imbalance problem.ResultsTen freehand ultrasound sequences are collected by inserting an injection needle steeply into the Ultrasound Compatible Lumbar Epidural Simulator and Femoral Vascular Access Ezono test phantoms. We divided the dataset into two sub-categories. In the second category, in which the situation is more challenging and the needle is totally invisible statically, the angle and tip localization error were 2.43±1.14° and 2.3±1.76 mm using Yolov3+RMF+AE and 2.08±1.18° and 2.12±1.43 mm using nYolo+RMF+AE.Conclusion and significanceThe proposed method has the potential to track the needle in a more reliable operation compared to other state-of-the-art methods and can accurately localize it in 2D B-mode US images in real-time, allowing it to be used in in current ultrasound intervention procedures.

show abstract

A hybrid camera- and ultrasound-based approach for needle localization and tracking using a 3D motorized curvilinear ultrasound probe

Abstract: A hybrid camera-and ultrasound-based approach for needle localization and tracking using a 3D motorized curvilinear ultrasound probe,

Cited by 14 publications

References 45 publications

Multi‐needle Localization with Attention U‐Net in US‐guided HDR Prostate Brachytherapy

Multi‐needle Localization with Attention U‐Net in US‐guided HDR Prostate Brachytherapy

Medical Instrument Detection in Ultrasound-Guided Interventions: A Review

Spatiotemporal analysis of speckle dynamics to track invisible needle in ultrasound sequences using Convolutional Neural Networks

Contact Info

Product

Resources

About