The effect of patient care order sets on medical resident education: a prospective before-after study

Abstract

BackgroundPatient care order sets are increasingly being used to optimize care. While studies have evaluated the impact of order sets on provider performance and patient outcomes, their impact on postgraduate medical trainee knowledge remains unknown. We sought to evaluate the impact of order sets on respirology knowledge, order-writing skills, and self-reported learning.MethodsWe conducted a prospective before-after study. Postgraduate trainees completing a Respirology rotation at a quaternary-care hospital 6 months before (no order set period) and 12 months after (order set period) order set introduction. Guideline-based admission order sets with educational prompts detailing recommended management of cystic fibrosis and chronic obstructive pulmonary disease were implemented on the respirology ward. Each resident completed a test before and after the rotation assessing knowledge and order-writing. Residents in the order set period additionally completed a questionnaire regarding the impact of order set use on their learning. Analysis: The primary outcome, the difference between pre and post rotation scores was compared between residents in the no order set period and residents in the order set period, using univariate linear regression. Test validity was assessed with a 2-sample t-test, analysis of variance and Pearson’s correlation coefficient. Self-reported impact of order set use were descriptively analyzed, and written responses were collated and coded.ResultsInvestigators consecutively recruited 11 subjects before and 28 subjects after order set implementation. Residents in the order set period had a greater improvement in post-rotation test scores than residents in the no order set period (p = 0.04); after adjustment for baseline scores, this was not significant (p = 0.3). The questionnaire demonstrated excellent convergent, discriminant and construct validity. Residents reported that order sets improved their knowledge and skills and provided a systematic approach to care.ConclusionsOrder sets do not appear to impair resident education, and may impart a benefit. This will require validation in larger studies and across diseases.

Gaps in the Inpatient Management of Chronic Obstructive Pulmonary Disease Exacerbation and Impact of An Evidence-Based Order Set

“…These improvements more likely resulted from the influence of guideline recommendations, levels of evidence, and expected outcomes provided in the order set on clinician knowledge and decision-making. This is supported by our concurrent educational study of these order sets, which suggested improvements in clinical users' knowledge of guideline-recommended care with order set use (23). Similar improvements in the use of systemic steroids and metered-dose inhalers after order set implementation were previously demonstrated in children admitted for asthma exacerbation (34).…”

“…Order set uptake was poor but, as noted, it was superior to that reported in previous studies, and this would bias against the observed improvements. There may also have been a halo effect whereby the educational effects of the order set accounted for improvements in care even in cases for which it was not directly used (23). Use of evidence-based strategies to improve uptake may result in even larger improvements than those observed in our study.…”

Default Options: Opportunities to Improve Quality and Value in Oncology

“…Although it is paternalistic to intentionally design defaults toward a particular outcome, the key feature of this "soft" or "libertarian" paternalism is that it maintains the option to choose otherwise with minimal effort. 65,66 What role will defaults ultimately play in cancer care? [17][18][19] Who would set these defaults?…”

Formative Usability Testing Reduces Severe Blood Product Ordering Errors

“…It may also be due to lack of true effect. In a study of resident physician's knowledge of cystic fibrosis and chronic obstructive pulmonary disease before and after implementation of disease-specific order sets, Yu et al 27 found a similar trend of improved test scores in those exposed to the order sets. While some argue that order sets and decision support in general prevents medical trainees from learning, 28 these findings suggest that order sets may actually improve resident education.…”

“…Similar trends have been seen with the introduction of cystic fibrosis and chronic obstructive pulmonary disease order sets. 27…”