Background RESTORE (Randomized Evaluation of Sedation Titration fOr Respiratory failurE) was a cluster randomized clinical trial evaluating a sedation strategy in children ≥ 2 weeks to < 18 years of age with acute respiratory failure supported on mechanical ventilation. A total of 31 U.S. pediatric intensive care units (PICUs) participated in the trial. Staff nurse rater agreement on measures used to assess a critical component of treatment fidelity was essential throughout the four-year data collection period. Objective The purpose of the paper is to describe the method of establishing and maintaining interrater agreement (IRA) of two core clinical assessment instruments over the course of the clinical trial. Methods IRA cycles were carried out at all control and intervention sites and included a minimum of five measurements of the State Behavioral Scale (SBS) and Withdrawal Assessment Tool-Version 1 (WAT-1). Glasgow Coma Scale scores were also obtained. PICUs demonstrating < 80% agreement repeated their IRA cycle. Fleiss’s kappa coefficient was used to assess interrater agreement. Results Repeated IRA cycles were required for 8% of 226 SBS cycles and 2% of 222 WAT-1 cycles. Fleiss’s kappa coefficients from more than 1,350 paired assessments were 0.86 for SBS and 0.92 for WAT-1, demonstrating strong agreement and similar to 0.91 for the Glasgow Coma Scale. There was no difference in Fleiss’s kappa for any of the instruments based on unit size or timing of assessment (earlier or later in the study). For SBS scores, Fleiss’s kappa was significantly different in larger and smaller PICUs (0.82 vs. 0.92, p < .003); however, Fleiss’s kappa for both groups indicated excellent agreement. Conclusion Monitoring measurement reliability is an essential step in ensuring treatment fidelity and, thus, the validity of study results. Standardization on the use of these core assessment instruments among participating sites was achieved and maintained throughout the trial.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
customersupport@researchsolutions.com
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.