Automatic Segmentation of Multiple Structures in Knee Arthroscopy Using Deep Learning

Abstract

Minimally invasive surgery (MIS) is among the preferred procedures for treating a number of ailments as patients benefit from fast recovery and reduced blood loss. The trade-off is that surgeons lose direct visual contact with the surgical site and have limited intra-operative imaging techniques for real-time feedback. Computer vision methods as well as segmentation and tracking of the tissues and tools in the video frames, are increasingly being adopted to MIS to alleviate such limitations. So far, most of the advances in MIS have been focused on laparoscopic applications, with scarce literature on knee arthroscopy. Here for the first time, we propose a new method for the automatic segmentation of multiple tissue structures for knee arthroscopy. The training data of 3868 images were collected from 4 cadaver experiments, 5 knees, and manually contoured by two clinicians into four classes: Femur, Anterior Cruciate Ligament (ACL), Tibia, and Meniscus. Our approach adapts the U-net and the U-net++ architectures for this segmentation task. Using the cross-validation experiment, the mean Dice similarity coefficients for Femur, Tibia, ACL, and Meniscus are 0.78, 0.50, 0.41, 0.43 using the U-net and 0.79, 0.50, 0.51, 0.48 using the U-net++. While the reported segmentation method is of great applicability in terms of contextual awareness for the surgical team, it can also be used for medical robotic applications such as SLAM and depth mapping.

Artificial Intelligence in the Management of Anterior Cruciate Ligament Injuries

“…Nevertheless, there is still a need for tools that can aid in objective intraoperative assessment. The algorithm developed by Jonmohamadi et al, 17 which was capable of identifying key anatomic features, including the ACL and menisci, is an important first step toward automated identification of complex anatomic landmarks during ACL reconstruction. For example, it has been found that the placement of tibial and femoral tunnels during ACL reconstruction is significantly different from the native anatomic footprint identified on MRI scans, suggesting that surgeons have difficulty identifying these landmarks intraoperatively.…”

“…As of the date of the most recent search conducted for this review, a single ML application has been developed for intraoperative use in knee arthroscopy. To provide additional contextual awareness for surgeons, an algorithm capable of automatic segmentation of the arthroscopic frame was developed by Jonmohamadi et al 17 The algorithm, "U-NET," is a CNN that was specifically created for the segmentation of biomedical images. This technology uses the arthroscopic video as an input parameter to produce a segmented image of the structures seen in real time by the surgeon.…”

“…The authors suggested that automated segmentation and tissue labeling could have important implications for both human surgical training and tool tracking in future robotic arthroscopic applications. 17…”

3D Semantic Mapping from Arthroscopy using Out-of-distribution Pose and Depth and In-distribution Segmentation Training

“…Hence, we advocate the use of pose annotation acquired from images from non-human environments to supervise the training of depth+pose using a self-supervised+supervised loss function. We also trained a novel supervised model for semantic segmentation with the method in [1] that extends the semantic segmentation in [11] based on the use of multi-spectral frame reconstruction [20]. By considering that the biological compositions of each tissue type namely bone, ACL, and meniscus are intrinsically different, the RGB arthroscopic frames are transformed into 36 spectral bands and then the spatial features of anatomical structures are used at wavelengths from 380-740 nm with 10 nm of intervals as a preprocessing step.…”

“…We train a system with the OOD datasets to estimate depth+pose using self-supervised view synthesis loss + supervised pose loss. We also train a method to produce semantic segmentation using the ID dataset in [11]. We then combine the pose, depth, and semantic segmentation of both systems and use the method in [29] to produce 3D semantic maps of testing images from the ID dataset.…”