Reliability and accuracy of ultrasound image analyses completed manually versus an automated tool

Abstract

Analysis of Brightness-mode ultrasound-captured fascicle angle (FA) and fascicle length (FL) can be completed manually with computer-based programs or by automated programs. Insufficient data exists regarding reliability and accuracy of automated tools. Therefore, the purpose of this study was to determine the test-retest reliability of automatic and manual ultrasound analyses, while determining accuracy of the automatic tool against the manual equivalent. Twenty-three participants (mean ± SD; age = 24 ± 4 years; height = 172.2 ± 10.5 cm; body mass = 73.1 ± 16.1 kg) completed one laboratory visit consisting of two trials where vastus lateralis muscle architecture was assessed with ultrasound. Images were taken at both lower (10 MHz) and higher frequency (12 MHz). Images were analyzed manually in an open-source imaging program and automatically using a separate open-source macro function. Test-retest reliability statistics were calculated for automatic and manual analyses. Accuracy was determined with validity statistics and were calculated for automatic analyses. The results show that manual ultrasound analyses for FA and FL for both lower and higher frequency displayed good reliability (ICC2,1 = 0.75–0.86). However, automatic ultrasound analyses for FA and FL revealed moderate reliability (ICC2,1 = 0.61–0.72) for the lower frequency images and poor reliability (ICC2,1 = 0.16–0.27) for higher frequency images. When assessed against manual techniques, automatic analyses presented greater total error (TE) and standard error of the estimate (SEE) for FA at lower frequency (constant error (CE) = −3.91°, TE = 5.57°, SEE = 3.45°) than higher (CE = −2.78°, TE = −4.54°, SEE = 2.45°). For FL, the higher frequency error (CE = 0.92 cm, TE = 2.12 cm, SEE = 1.15 cm) was similar to lower frequency error (CE = 1.98 cm, TE = 3.66 cm, SEE = 1.57 cm). The findings overall show that manual analyses had good reliability and low absolute error, while demonstrating the automated counterpart had poor to moderate reliability and large errors in analyses. These findings may be impactful as they highlight the good reliability and low error associated with manually analyzed ultrasound images and validate a novel automatic tool for analyzing ultrasound images. Future work should focus on improving reliability and decreasing error in automated image analysis tools. Automated tools are promising for the field as they eliminate biases between analysts and may be more time efficient than manual techniques.