Acoustic signal analysis of instrument–tissue interaction for minimally invasive interventions

Abstract

Purpose Minimally invasive surgery (MIS) has become the standard for many surgical procedures as it minimizes trauma, reduces infection rates and shortens hospitalization. However, the manipulation of objects in the surgical workspace can be difficult due to the unintuitive handling of instruments and limited range of motion. Apart from the advantages of robot-assisted systems such as augmented view or improved dexterity, both robotic and MIS techniques introduce drawbacks such as limited haptic perception and their major reliance on visual perception. Methods In order to address the above-mentioned limitations, a perception study was conducted to investigate whether the transmission of intra-abdominal acoustic signals can potentially improve the perception during MIS. To investigate whether these acoustic signals can be used as a basis for further automated analysis, a large audio data set capturing the application of electrosurgery on different types of porcine tissue was acquired. A sliding window technique was applied to compute log-mel-spectrograms, which were fed to a pre-trained convolutional neural network for feature extraction. A fully connected layer was trained on the intermediate feature representation to classify instrument–tissue interaction. Results The perception study revealed that acoustic feedback has potential to improve the perception during MIS and to serve as a basis for further automated analysis. The proposed classification pipeline yielded excellent performance for four types of instrument–tissue interaction (muscle, fascia, liver and fatty tissue) and achieved top-1 accuracies of up to 89.9%. Moreover, our model is able to distinguish electrosurgical operation modes with an overall classification accuracy of 86.40%. Conclusion Our proof-of-principle indicates great application potential for guidance systems in MIS, such as controlled tissue resection. Supported by a pilot perception study with surgeons, we believe that utilizing audio signals as an additional information channel has great potential to improve the surgical performance and to partly compensate the loss of haptic feedback.

Real-time acoustic sensing and artificial intelligence for error prevention in orthopedic surgery

“…Spectrogram features are the dominant representation in deep learning for audio signal processing 38 . They have been shown to yield superior classification performances and achieve promising results in combination with convolutional neural network-based architectures for speech 42 , audio event detection 43 , and medical applications 44 . Log-mel spectrograms, a widely-used spectrogram variant, are two-dimensional matrices with time windows as columns, mel-bins (frequency) as rows, and amplitude as scalar values contained in the matrix.…”

Intraoperative tissue classification methods in orthopedic and neurological surgeries: A systematic review

“…Acceleration sensing methods performed by Dai et al ( 47 , 48 ) tackled tissue classification during robotic bone drilling to make conclusions about material density. Several studies proposed to capture acoustic signals during surgical drilling and milling using free-field microphones ( 49 , 53 , 54 ), condenser microphones ( 50 ), contact microphones ( 51 ), and sound recorders ( 55 ). A more complex solution was proposed by Dai et al ( 56 ), in which the authors combined a free-field microphone with an accelerometer.…”

“…Methods found in Ostler et al ( 50 ) and Seibold et al ( 51 ) represented the state-of-the-art in machine learning, as they suggested classification by using deep learning architectures during bone drilling. Ostler et al ( 50 ) extracted log-spectrograms acquired during tissue drilling. Seibold et al ( 51 ) used mel-spectrograms of the audio signals with a ResNet18 network ( Figure 4 ).…”

Telemedical percussion: objectifying a fundamental clinical examination technique for telemedicine

“…While this proof of concept was designed to determine the best technical approach for a robotic percussion, a major next step will be the evaluation of the ventral motorized instrument regarding detection of different pathologies. Furthermore, different areas of medicine already use automatized analysis, such as deep learning approaches, for medical audio signals [2,11,17,22].…”