Development and Establishment of Initial Validity Evidence for a Novel Tool for Assessing Trainee Admission Notes

Abstract

BACKGROUND: Documentation is a key component of practice, yet few curricula have been published to teach trainees proper note construction. Additionally, a gold standard for assessing note quality does not exist, and no documentation assessment tools integrate with established competency-based frameworks. OBJECTIVE: To develop and establish initial validity evidence for a novel tool that assesses key components of trainee admission notes and maps to the Accreditation Council for Graduate Medical Education (ACGME) milestone framework. DESIGN: Using an iterative, consensus building process we developed the Admission Note Assessment Tool (ANAT). Pilot testing was performed with both the supervising attending and study team raters not involved in care of the patients. The finalized tool was piloted with attendings from other institutions. PARTICIPANTS: Local experts participated in tool development and pilot testing. Additional attending physicians participated in pilot testing. MAIN MEASURES: Content, response process, and internal structure validity evidence was gathered using Messick's framework. Inter-rater reliability was assessed using percent agreement. KEY RESULTS: The final tool consists of 16 checklist items and two global assessment items. Pilot testing demonstrated rater agreement of 72% to 100% for checklist items and 63% to 70% for global assessment items. Note assessment required an average of 12.3 min (SD 3.7). The study generated validity evidence in the domains of content, response process, and internal structure for use of the tool in rating admission notes. CONCLUSIONS: The ANAT assesses individual components of a note, incorporates billing criteria, targets note "bloat," allows for narrative feedback, and provides global assessments mapped to the ACGME milestone framework. The ANAT can be used to assess admission notes by any attending and at any time after note completion with minimal rater training. The ANAT allows programs to implement routine note assessment for multiple functions with the use of a single tool.

Impact of Co-Existing Placental Pathologies in Pregnancies Complicated by Placental Abruption and Acute Neonatal Outcomes

“…A retrospective cohort study was conducted at The Ottawa Hospital (TOH) and the Children’s Hospital of Eastern Ontario (CHEO). All placentas with the pathologic or clinical diagnosis of placental abruption and/or retroplacental hematoma from 1 October 2013 to 30 April 2020 were identified from the pathology archives using the Laboratory Information Service Program (EPIC-Hyperspace) [ 39 ]. Data on maternal demographics, neonatal outcomes, and the gross and histopathological findings of the placenta were collected using EPIC-Hyperspace and recorded on RedCap [ 40 ].…”

Improving trainee clinical documentation through a novel curriculum in internal medicine

“…Little is known about how best to improve the quality of trainee admission notes. We previously described the development of and collection of validity evidence for the Admission Note Assessment Tool (ANAT) 17 . The ANAT simultaneously assesses key individual components of an admission note and provides global assessments mapped to the Accreditation Council of Graduation Medical Education (ACGME) subcompetencies.…”

Defining the gold standard: What is success in electronic health record documentation?

“…The intervention included two didactic sessions as well as regular feedback on admission notes throughout the year. Admission notes were assessed using the Admission Note Assessment Tool (ANAT), a previously validated tool developed by the study authors 5 . The authors analyzed admission notes at two points during the academic year, comparing those written by interns during the intervention year to those written in the prior academic year.…”