Reliability of Machine and Human Examiners for Detection of Laryngeal Penetration or Aspiration in Videofluoroscopic Swallowing Studies

Abstract

Computer-assisted analysis is expected to improve the reliability of videofluoroscopic swallowing studies (VFSSs), but its usefulness is limited. Previously, we proposed a deep learning model that can detect laryngeal penetration or aspiration fully automatically in VFSS video images, but the evidence for its reliability was insufficient. This study aims to compare the intra- and inter-rater reliability of the computer model and human raters. The test dataset consisted of 173 video files from which the existence of laryngeal penetration or aspiration was judged by the computer and three physicians in two sessions separated by a one-month interval. Intra- and inter-rater reliability were calculated using Cohen’s kappa coefficient, the positive reliability ratio (PRR) and the negative reliability ratio (NRR). Intrarater reliability was almost perfect for the computer and two experienced physicians. Interrater reliability was moderate to substantial between the model and each human rater and between the human raters. The average PRR and NRR between the model and the human raters were similar to those between the human raters. The results demonstrate that the deep learning model can detect laryngeal penetration or aspiration from VFSS video as reliably as human examiners.

Deep Learning Artificial Intelligence Tool for Automated Radiographic Determination of Posterior Tibial Slope in Patients With ACL Injury

“… 9 , 12 , 15 , 32 As previously demonstrated, the computer vision algorithm will exhibit perfect intrarater reliability and substantial interrater reliability with human annotators regardless of the washout period. 17 Furthermore, the computer vision annotator developed herein has an advantage in efficiency once incorporated into routine practice. With exponential increase in digitization of imaging data given more sophisticated warehousing and management technologies, this tool can be leveraged for imaging annotations that are not practical for clinicians.…”

Artificial Intelligence in the Interpretation of Videofluoroscopic Swallow Studies: Implications and Advances for Speech–Language Pathologists

“…Lee et al [33] used a DL model to detect airway invasion from VFSS images, without clinician input, with 97.2% accuracy in classifying image frames and 93.2% in classifying video files. Kim et al [34] used the same DL model and found moderate to substantial inter-rater agreement between the machine and human. However, these studies highlight a pertinent limitation in the use of AI for aspiration detection.…”

“…However, VFSS are time-consuming to conduct and can be laborious to interpret, particularly for inexperienced clinicians. Further, VFSS interpretation is prone to human error [31,34]. DL frameworks have the potential to improve the accuracy and speed with which VFSS are interpreted, and thus, they have several clinical applications depending on the method of detection used.…”

“…Studies have shown that AI can detect the presence of laryngeal penetration or aspiration with accuracy [21,[32][33][34]. The accurate detection of aspiration has the potential to improve patient outcomes.…”

Advances in Management of Voice and Swallowing Disorders

“…A reliability study by Kim et al [2] confirmed that computer analysis using a deep learning model could detect laryngeal penetration or aspiration in recordings of videofluoroscopic swallowing studies (VFSS) as reliably as human examiners. These results provide further evidence to support the clinical application of deep learning technology in addition to the visuoperceptual evaluation of videofluoroscopic and possibly endoscopic recordings of swallowing.…”