Clinician’s ability to identify neck and low back interventions: an inter-rater chance-corrected agreement pilot study

Abstract

Objective: To estimate inter-rater agreement of physical therapists trained in MDT approach and participating in practice-based evidence (PBE) research to identify 72 physical therapy interventions in video demonstrations on a single model and clinical vignettes. PBE is a well designed observational study and demonstrating clinician observational consistency is an important step in conducting PBE research design. Methods: Two physical therapists volunteered to participate in pilot reliability testing and seven other physical therapists trained in McKenzie Mechanical Diagnosis and Therapy (MDT) methods volunteered for the inter-rater chance-corrected agreement study. All therapists identified interventions presented within 52 videos and 5 written clinical vignettes describing 20 more intervention techniques. Therapists independently identified all interventions. We assessed inter-rater chance-corrected agreement of therapists' ability to identify intervention techniques using Kappa coefficients with associated 95% confidence intervals and indices for bias and prevalence. Results: Of the 147 kappa coefficients estimated, 7% were (0.6, 10% were .0.6 and (0.8, and 83% were .0.8. Agreement was lowest for identifying cognitive behavioral techniques (median kappa50.79). The minimum and maximum prevalence and bias indices were 0.33 and 0.85 and 0 and 0.33, respectively suggesting kappa coefficient estimates were strong. Generalized kappa coefficients ranged from 0.73 to 1.00. Discussion: Results provide evidence that substantial to almost perfect inter-rater agreement could be expected when trained therapists identify physical therapy interventions used for patients with spinal impairments from staged videos and vignettes. This may be helpful to reassure clinicians of the quality of the reporting of intervention(s) performed when conducting multivariable analyses in future pragmatic PBE studies. Additional studies are needed to test whether these results can be validated using larger groups of therapists, trained and not trained in MDT methods, as well as examining different methods to examine inter-rater agreement for identifying diverse interventions commonly used for managing patients during routine practice.

Oral health screening: feasibility and reliability of the oral health assessment tool as used by speech pathologists

“…Moreover, the use of very small samples (only 10 videotapes) may yield misleading results from a proportion‐based Kappa‐statistic. However, videotapes offer the advantage of allowing the assessment of identical aspects of oral health status and reducing stress that influences clinical presentation. Videos also provide dynamic images and have been applied as a medium for dental health education in previous reports.…”

Cognitive behavioural interventions, and function and pain outcomes among patients with chronic neck pain managed with the McKenzie approach

“…All treatments rendered by participating physiotherapists during the study, including those involving GA/GEX, were documented following a standardized intervention taxonomy (Werneke, Hart, Deutscher, & Stratford, ). Chance‐corrected inter‐rater agreement among certified McKenzie raters using this treatment strategy was good to excellent (Werneke, Hart, Deutscher, & Stratford, ). No attempt was made to standardize treatments beyond these guidelines.…”

Provider reliability with interventions for knee impairments: a preliminary investigation to facilitate development of an MDT-based knee intervention taxonomy

“…Primary categories were adopted with permission from a previous investigation [25]. Primary categories were the least specific level and included Therapeutic Exercise, Manual Therapy Technique, Cognitive Behavior Technique, Modality Technique, Functional Training Technique, Education Technique, and Administrative Technique.…”