Convolution neural networks for real-time needle detection and localization in 2D ultrasound

Mwikirize, Cosmas; Nosher, John L.; Hacihaliloglu, Ilker

doi:10.1007/s11548-018-1721-y

Cited by 57 publications

(43 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our precision (83.2 [70.9, 89.8]%) was much higher than that observed by Mwikirize et al 30 prior to preprocessing, with a similar recall rate; however, Mwikirize et al emphasized the importance of preprocessing US images to reduce high-intensity artifacts in the image, providing a vast improvement in the recall and precision rates of their algorithm. Although preprocessing was avoided in our study to more closely reflect the intraoperative clinical realities, this may be an area of future investigation to further improve the accuracy of our results.…”

Section: Discussioncontrasting

confidence: 45%

“…[29][30][31][32][33][34][35][36][37][38] Additionally, approaches leveraging the principles of transfer learning (applying models developed for a prior task to a new context) have been investigated in the broader context of image segmentation, including a recent study in x-ray computed tomography (CT) images by Jung et al 39 In this paper, Jung et al 39 investigated the generalizability of a CNN-based method developed on a tandem-andovoids intracavitary brachytherapy device to segment other devices used for this type of procedure, substantiating the feasibility of a generalized tool segmentation method. Some existing works have specifically investigated CNN-based approaches to tool segmentation in US images, [29][30][31]34,37,38 recognizing that the appearance of tools in this imaging modality introduces unique challenges. Recent developments by Pourtaherian et al, 31 Arif et al, 29 and Zhang et al 38 focused on needle detection in 3D US images.…”

Section: Introductionmentioning

confidence: 99%

“…37 Although this approach is susceptible to information leakage since many images were acquired per patient, this study demonstrates many of the challenges associated with localizing needles in kidney images and the extension of this method to applications other than kidney biopsy has not been explored. Mwikirize et al 30 investigated an algorithm using a fully convolutional network with a fast region-based CNN for interventional applications requiring quick localization of mid-to-steeply inserted needles (40-75°) and deep insertions (up to 9 cm) in 2D US images. This study demonstrated accurate localization with recall and precision rates > 99%, a mean tip error of 0.23 mm, and a mean angular error of 0.82°with a processing time of 0.58 s. 30 Further investigation of this approach is necessary to assess its generalizability, as the initial study was limited by only using in-plane (or slightly out-of-plane) insertions of a single tool type (17gauge epidural needle) in ex vivo bovine/porcine tissues and phantoms with recall and precision only assessed on images acquired from a single US system and probe, matching the system used for training.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Deep learning segmentation of general interventional tools in two‐dimensional ultrasound images

et al. 2020

View full text Add to dashboard Cite

Purpose: Many interventional procedures require the precise placement of needles or therapy applicators (tools) to correctly achieve planned targets for optimal diagnosis or treatment of cancer, typically leveraging the temporal resolution of ultrasound (US) to provide real-time feedback. Identifying tools in two-dimensional (2D) images can often be time-consuming with the precise position difficult to distinguish. We have developed and implemented a deep learning method to segment tools in 2D US images in near real-time for multiple anatomical sites, despite the widely varying appearances across interventional applications. Methods: A U-Net architecture with a Dice similarity coefficient (DSC) loss function was used to perform segmentation on input images resized to 256 × 256 pixels. The U-Net was modified by adding 50% dropouts and the use of transpose convolutions in the decoder section of the network. The proposed approach was trained with 917 images and manual segmentations from prostate/gynecologic brachytherapy, liver ablation, and kidney biopsy/ablation procedures, as well as phantom experiments. Real-time data augmentation was applied to improve generalizability and doubled the dataset for each epoch. Postprocessing to identify the tool tip and trajectory was performed using two different approaches, comparing the largest island with a linear fit to random sample consensus (RAN-SAC) fitting. Results: Comparing predictions from 315 unseen test images to manual segmentations, the overall median [first quartile, third quartile] tip error, angular error, and DSC were 3.5 [1.3, 13.5] mm, 0.8 [0.3, 1.7]°, and 73.3 [56.2, 82.3]%, respectively, following RANSAC postprocessing. The predictions with the lowest median tip and angular errors were observed in the gynecologic images (median tip error: 0.3 mm; median angular error: 0.4°) with the highest errors in the kidney images (median tip error: 10.1 mm; median angular error: 2.9°). The performance on the kidney images was likely due to a reduction in acoustic signal associated with oblique insertions relative to the US probe and the increased number of anatomical interfaces with similar echogenicity. Unprocessed segmentations were performed with a mean time of approximately 50 ms per image. Conclusions: We have demonstrated that our proposed approach can accurately segment tools in 2D US images from multiple anatomical locations and a variety of clinical interventional procedures in near real-time, providing the potential to improve image guidance during a broad range of diagnostic and therapeutic cancer interventions.

show abstract

Section: Discussioncontrasting

confidence: 45%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep learning segmentation of general interventional tools in two‐dimensional ultrasound images

et al. 2020

View full text Add to dashboard Cite

show abstract

“…In another study, signal transmission maps and statistical optimization were used to model attenuation information and detect the needle tip. They combined this pipeline with a deep learning model to filter out motion events [24][25]. Daoud et al [26] used the needleinduced motion and edge detection to localize needles in US images.…”

Section: Related Studies In the Literaturementioning

confidence: 99%

Gabor filter-based localization of straight and curved needles in 2D ultrasound images

Kaya¹,

Senel²,

Bebek³

2020

Turk J Elec Eng & Comp Sci

View full text Add to dashboard Cite

2D ultrasound (US) is one of the most commonly used medical imaging devices for needle localization in biopsies. However, the produced images are low-resolution and contain an excessive number of artifacts, which makes the needle localization challenging. Image processing techniques can help resolve this issue. This paper presents a novel Gabor filter-based method for needle localization in 2D US images, which enhances the needle outline in the images while suppressing other structures. The scheme works in two stages: First, the Gabor filter is applied to the image, the needle insertion angle is estimated, and the needle trajectory is found using a RANSAC line fitting; then, the Gabor filter is repeated using the estimated insertion angle and the location of the needle tip is estimated using probability mapping. The proposed scheme works with both straight and curved needles. An automatic parameter tuning method, which is used to optimize the threshold value of the Otsu's method, is also presented here. The tests on this needle localization scheme were done using two different water mixtures, three different gelatin-based phantoms, and ex vivo experiments. The accuracy was assessed using an infrared-camera-based tracking sensor and computed tomography images. The results showed that the suggested localization scheme can be effectively used in the 2D US image-guided needle procedures.

show abstract

“…Recently, a template-based tracking method with the efficient second-order minimization optimization method has been used to track the needle [16]. In recent studies, more and more novel ideas have been used to locate the needle and evaluate its tip to sagittal US images, such as the use of signal attenuation maps [17], convolution neural networks (CNN) [18], and maximum likelihood estimation sample consensus (MLESAC) method [19]. However, a demerit of using sagittal US images is that out-of-plane bending of the needle cannot be detected.…”

Section: Introductionmentioning

confidence: 99%

Tip Estimation Method in Phantoms for Curved Needle Using 2D Transverse Ultrasound Images

Song

Liu

et al. 2019

Applied Sciences

View full text Add to dashboard Cite

Flexible needles have been widely used in minimally invasive surgeries, especially in percutaneous interventions. Among the interventions, tip position of the curved needle is very important, since it directly affects the success of the surgeries. In this paper, we present a method to estimate the tip position of a long-curved needle by using 2D transverse ultrasound images from a robotic ultrasound system. Ultrasound is first used to detect the cross section of long-flexible needle. A new imaging approach is proposed based on the selection of numbers of pixels with a higher gray level, which can directly remove the lower gray level to highlight the needle. After that, the needle shape tracking method is proposed by combining the image processing with the Kalman filter by using 3D needle positions, which develop a robust needle tracking procedure from 1 mm to 8 mm scan intervals. Shape reconstruction is then achieved using the curve fitting method. Finally, the needle tip position is estimated based on the curve fitting result. Experimental results showed that the estimation error of tip position is less than 1 mm within 4 mm scan intervals. The advantage of the proposed method is that the shape and tip position can be estimated through scanning the needle’s cross sections at intervals along the direction of needle insertion without detecting the tip.

show abstract

Convolution neural networks for real-time needle detection and localization in 2D ultrasound

Cited by 57 publications

References 16 publications

Deep learning segmentation of general interventional tools in two‐dimensional ultrasound images

Deep learning segmentation of general interventional tools in two‐dimensional ultrasound images

Gabor filter-based localization of straight and curved needles in 2D ultrasound images

Tip Estimation Method in Phantoms for Curved Needle Using 2D Transverse Ultrasound Images

Contact Info

Product

Resources

About