Development and Initial Testing of a Structured Clinical Observation Tool to Assess Pharmacotherapy Competence

Young, John Q.; Lieu, Sandra C; O’Sullivan, Patricia; Tong, Lowell

doi:10.1176/appi.ap.35.1.27

Cited by 21 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Overview Of Assessment Methods Identifiedmentioning

confidence: 99%

“…It allows faculty to provide formative feedback to the learner in real time [8][9][10][11] and tends to generate more specific feedback and constructive comments compared to global assessments. 12,13 At least 55 direct observation tools have been developed, but only a few have proven reliability, validity, or educational outcomes data measured.14 Faculty training on the use of any direct observation tool is important given the potential for variability of interpretation of a clinical encounter and the tool's language, yet few studies have demonstrated more than cursory observer training.14 There is evidence, however, that even without extensive training, certain tools have good to excellent reliability. 10,15 The correlation between direct observation and other measures of competency such as written test scores, [16][17][18][19][20][21][22][23][24][25] OSCEs, or standardized patient assessments [18][19][20][21]25,26 has been studied in a number of specialties showing modest correlation supporting the validity of certain direct observation methods.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Assessing Patient Care: Summary of the Breakout Group on Assessment of Observable Learner Performance

Takayesu

Kulstad

Wallenstein

et al. 2012

Academic Emergency Medicine

View full text Add to dashboard Cite

There is an established expectation that physicians in training demonstrate competence in all aspects of clinical care prior to entering professional practice. Multiple methods have been used to assess competence in patient care, including direct observation, simulation-based assessments, objective structured clinical examinations (OSCEs), global faculty evaluations, 360-degree evaluations, portfolios, self-reflection, clinical performance metrics, and procedure logs. A thorough assessment of competence in patient care requires a mixture of methods, taking into account each method's costs, benefits, and current level of evidence. At the 2012 Academic Emergency Medicine (AEM) consensus conference on educational research, one breakout group reviewed and discussed the evidence supporting various methods of assessing patient care and defined a research agenda for the continued development of specific assessment methods based on current best practices. In this article, the authors review each method's supporting reliability and validity evidence and make specific recommendations for future educational research.ACADEMIC EMERGENCY MEDICINE 2012; 19:1379-1389 by the Society for Academic Emergency Medicine I n 2001, the Accreditation Council for Graduate Medical Education (ACGME) introduced a timeline for the implementation of training and assessment in six core competencies that form the foundation of clinical competence. Introduced in 1996, the Canadian CanMEDS manager competency correlates to the AC-GME patient care competency, broadly defined as "the active engagement in decision-making in the operation of the healthcare system."1 The patient care competency for emergency medicine (EM) has been defined by a previous Academic Emergency Medicine (AEM) consensus conference, 2 now further elaborated on by the milestones in training, 3 as being able to efficiently gather and synthesize medical and diagnostic information, prioritize tasks, and implement management plans on multiple patients, as well as performing essential invasive procedures competently.There is an explicit expectation that physicians in training demonstrate competence in various aspects of clinical care prior to graduation and professional practice. 4 While this accountability falls squarely on the shoulders of residency training programs, it is mirrored by commensurate expectations of maintenance of competency during ongoing professional practice.The goals of the 2012 AEM consensus conference patient care working group were to describe the current state of evidence for assessment of competence in patient care and define a research agenda for the further development of specific assessment methods based on current best practices. METHODSA search was conducted using MEDLINE 1996-present using the key word search terms "assessment," "patient care," "competency," "competence," "assess*," "emergency," and "education" and limited to humans and English language [boolean search: ((assessment and patient care AND (competency or competence)) OR (assess* a...

show abstract

Section: Overview Of Assessment Methods Identifiedmentioning

confidence: 99%

mentioning

confidence: 99%

Assessing Patient Care: Summary of the Breakout Group on Assessment of Observable Learner Performance

Takayesu

Kulstad

Wallenstein

et al. 2012

Academic Emergency Medicine

View full text Add to dashboard Cite

show abstract

“…Prior to the implementation of the EPA app, faculty used a paper-based direct observation tool that included a comprehensive 27-item checklist, an overall EPA rating, and prompts for both reinforcing and corrective comments. This tool had been studied in several settings with evidence for validity and generates, on average, five highly specific comments with a 3:2 ratio of reinforcing to corrective [11,[41][42][43]. All faculty agreed to participate.…”

Section: Setting and Participantsmentioning

confidence: 99%

A mobile app to capture EPA assessment data: Utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement

et al. 2020

View full text Add to dashboard Cite

Introduction Mobile apps that utilize the framework of entrustable professional activities (EPAs) to capture and deliver feedback are being implemented. If EPA apps are to be successfully incorporated into programmatic assessment, a better understanding of how they are experienced by the end-users will be necessary. The authors conducted a qualitative study using the Consolidated Framework for Implementation Research (CFIR) to identify enablers and barriers to engagement with an EPA app. Methods Structured interviews of faculty and residents were conducted with an interview guide based on the CFIR. Transcripts were independently coded by two study authors using directed content analysis. Differences were resolved via consensus. The study team then organized codes into themes relevant to the domains of the CFIR. Results Eight faculty and 10 residents chose to participate in the study. Both faculty and residents found the app easy to use and effective in facilitating feedback immediately after the observed patient encounter. Faculty appreciated how the EPA app forced brief, distilled feedback. Both faculty and residents expressed positive attitudes and perceived the app as aligned with the department's philosophy. Barriers

show abstract

“…We used the comments generated by the Pharmacotherapy-Structured Clinical Observation (P-SCO), a direct observation tool in psychiatry, with evidence for validity. [25][26][27] This study has 3 aims: (1) Analyze the quality of the narrative comments generated by the P-SCO; (2) Characterize the themes most commonly captured by the narrative comments; and (3) Examine the relationship between the narrative comments and the checklist scores.…”

Section: Introductionmentioning

confidence: 99%

Advancing Our Understanding of Narrative Comments Generated by Direct Observation Tools: Lessons From the Psychopharmacotherapy-Structured Clinical Observation

Young

Sugarman

Holmboe

et al. 2019

Journal of Graduate Medical Education

View full text Add to dashboard Cite

Background While prior research has focused on the validity of quantitative ratings generated by direct observation tools, much less is known about the written comments. Objective This study examines the quality of written comments and their relationship with checklist scores generated by a direct observation tool, the Psychopharmacotherapy-Structured Clinical Observation (P-SCO). Methods From 2008 to 2012, faculty in a postgraduate year 3 psychiatry outpatient clinic completed 601 P-SCOs. Twenty-five percent were randomly selected from each year; the sample included 8 faculty and 57 residents. To assess quality, comments were coded for valence (reinforcing or corrective), behavioral specificity, and content. To assess the relationship between comments and scores, the authors calculated the correlation between comment and checklist score valence and examined the degree to which comments and checklist scores addressed the same content. Results Ninety-one percent of the comments were behaviorally specific. Sixty percent were reinforcing, and 40% were corrective. Eight themes were identified, including 2 constructs not adequately represented by the checklist. Comment and checklist score valence was moderately correlated (Spearman's rho = 0.57, P < .001). Sixty-seven percent of high and low checklist scores were associated with a comment of the same valence and content. Only 50% of overall comments were associated with a checklist score of the same valence and content. Conclusions A direct observation tool such as the P-SCO can generate high-quality written comments. Narrative comments both explain checklist scores and convey unique content. Thematic coding of comments can improve the content validity of a checklist.

show abstract

Development and Initial Testing of a Structured Clinical Observation Tool to Assess Pharmacotherapy Competence

Cited by 21 publications

References 27 publications

Assessing Patient Care: Summary of the Breakout Group on Assessment of Observable Learner Performance

Assessing Patient Care: Summary of the Breakout Group on Assessment of Observable Learner Performance

A mobile app to capture EPA assessment data: Utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement

Advancing Our Understanding of Narrative Comments Generated by Direct Observation Tools: Lessons From the Psychopharmacotherapy-Structured Clinical Observation

Contact Info

Product

Resources

About