Image Segmentation using Transfer Learning with DeepLabv3 to Facilitate Photogrammetric Limb Scanning

Abstract

In this paper, we explore the use of deep learning (DL) in conjunction with photogrammetry for scanning amputated limbs. Combining these two technologies can expand the scope of prosthetic telemedicine by facilitating low-cost limb scanning using cell phones. Previous research identified image segmentation as one of the main limitations of using photogrammetry for limb scanning. Based on those limitations, this work sought to answer two main research questions: (1) Can a neural network be trained to identify and segment an amputated limb automatically? (2) Will segmenting 2D limb images using neural networks impact the accuracy of 3D models generated via photogrammetry? To answer the first question, transfer learning was applied to a neural network with the DeepLabv3 architecture. After training, the model was able to successfully identify and segment limb images with an IoU of 79.9\%. To answer the second question, the fine-tuned DL model was applied to a dataset of 22 scans comprising 6312 limb images, then 3D models were rendered utilizing Agisoft Metashape. The Mean Absolute Error (MAE) of models rendered from images segmented with DL was 0.57 mm ± 0.63 mm when compared to models rendered from ground truth images. These results are important because segmentation with DL makes photogrammetry for limb scanning feasible on a large clinical scale. Future work should focus on generalizing the segmentation model for different types of amputations and imaging conditions.

A Review on 3D Scanners Studies for Producing Customized Orthoses

“…Additionally, smartphone photogrammetry has been investigated, demonstrating the optimization of methods and quantitative evaluation of suitability for prosthetics and orthotics [ 58 ]. The integration of photogrammetry with smartphone technology has been explored to facilitate low-cost limb scanning, expanding the scope of orthotic telemedicine and providing affordable limb scans to underserved areas [ 59 ]. Moreover, the combination of photogrammetry and transfer learning with DeepLabv3 for image segmentation has been proposed to facilitate low-cost limb scanning using cell phones, further emphasizing the potential for cost-effective applications in orthotics [ 59 ].…”