Observers’ Agreement on Measurements in Fiberoptic Endoscopic Evaluation of Swallowing

Abstract

This study analyzed the effect that dysphagia etiology, different observers, and bolus consistency might have on the level of agreement for measurements in FEES images reached by independent versus consensus panel rating. Sixty patients were included and divided into two groups according to dysphagia etiology: neurological or head and neck oncological. All patients underwent standardized FEES examination using thin and thick liquid consistencies. Two observers scored the same exams, first independently and then in a consensus panel. Four ordinal FEES variables were analyzed. Statistical analysis was performed using a linear weighted kappa coefficient and Bayesian multilevel model. Intra- and interobserver agreement on FEES measurements ranged from 0.76 to 0.93 and from 0.61 to 0.88, respectively. Dysphagia etiology did not influence observers’ agreement level. However, bolus consistency resulted in decreased interobserver agreement for all measured FEES variables during thin liquid swallows. When rating on the consensus panel, the observers deviated considerably from the scores they had previously given on the independent rating task. Observer agreement on measurements in FEES exams was influenced by bolus consistency, not by dysphagia etiology. Therefore, observer agreement on FEES measurements should be analyzed by taking bolus consistency into account, as it might affect the interpretation of the outcome. Identifying factors that might influence agreement levels could lead to better understanding of the rating process and assist in developing a more precise measurement scale that would ensure higher levels of observer agreement for measurements in FEES exams.

Comparing dependent kappa coefficients obtained on multilevel data

“…The aim of the third FEES study part (Pilz et al ., 2016) is to compare the individual intraobserver agreement level of two students to the intraobserver agreement level obtained by the students in consensus. Since the FEES criteria are ordinal, the multilevel linear weighted kappa coefficient is used as agreement measure.…”

“…There was no evidence against the first assumption in the two first study parts (see Pilz et al ., 2016). To test the adequacy of the second assumption, the proportion of patients classified in the different FEES severity categories are given in Table 1 according to the liquid consistency.…”

“…This suggests that consensus ratings might offer an alternative to independent rating of FEES exams. However, changes in the scoring of the FEES criteria between the individual and the consensus ratings were observed (data not shown, Pilz et al ., 2016). Therefore, the validity of the FEES criteria for individual and consensus ratings also needs to be studied in order to better compare the two rating processes.…”

“…This paper is motivated by the third part of an exploratory study aimed at investigating the influence of different factors on inter‐ and intraobserver agreement levels on the evaluation of oropharyngeal dysphagia severity (Pilz et al ., 2016). Oropharyngeal dysphagia is characterized by difficulties in swallowing.…”

Medically Unexplained Oropharyngeal Dysphagia at the University Hospital ENT Outpatient Clinic for Dysphagia: A Cross-Sectional Cohort Study

“…To be sure that none of the selected patients had severe abnormalities during FEES examination (e.g., severe pooling, deep penetration, aspiration), suggesting a possible underlying somatic cause, five visuoperceptual ordinal variables (piecemeal deglutition, postswallow vallecular pooling, postswallow pyriform sinus pooling, laryngeal penetration, and aspiration) were scored by two independent judges [13,[21][22][23][24][25][26]. All of these variables were scored for every FEES swallow at varying speed.…”