Validity and Feasibility of the Minicard Direct Observation Tool in 1 Training Program

Donato, Anthony A.; Park, Yoon Soo; George, Donna L.; Schwartz, Alan; Yudkowsky, Rachel

doi:10.4300/jgme-d-14-00532.1

Cited by 10 publications

(17 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Resultsmentioning

confidence: 99%

“…Twenty studies were conducted in the United States, 14–33 13 were performed in Canada, 34–46 three took place in Australia, 47–49 and two were conducted in Taiwan 50,51 . Thirty‐three studies involved resident physicians, 15,17–23,25–32,34–47,49,50 four studies involved medical students, 14,16,24,33 and one did not report the learner level 48 . Twenty‐two were prospective studies, 14,16,17,22–28,30,31,33,37,38,40–42,46–48,51 nine were retrospective, 18–20,32,35,39,44,45,50 five were qualitative studies, 21,29,34,43,49 and two were descriptive studies 15,36 .…”

Section: Resultsmentioning

confidence: 99%

“…Thirty‐three studies involved resident physicians, 15,17–23,25–32,34–47,49,50 four studies involved medical students, 14,16,24,33 and one did not report the learner level 48 . Twenty‐two were prospective studies, 14,16,17,22–28,30,31,33,37,38,40–42,46–48,51 nine were retrospective, 18–20,32,35,39,44,45,50 five were qualitative studies, 21,29,34,43,49 and two were descriptive studies 15,36 . Twenty‐five studies were performed in the clinical environment, 14,15,18–21,24,26–31,34–37,43–45,47–51 12 in the simulation lab, 16,17,22,25,32,33,38–42,46 and one used both simulation and the clinical environment 23 …”

Section: Resultsmentioning

confidence: 99%

“…Ten studies utilized the ACGME EM Milestones, 15,16,18,19,22,23,27–29,31 seven used Observed Structured Clinical Exercises (OSCE), 16,17,23,30,32,33,42 six studies utilized the McMaster Modular Assessment Program (McMAP), 34–36,43–45 five used the Queen’s Simulation Assessment Test (QSAT), 25,38–41 and four utilized the mini‐Clinical Evaluation Exercise (mini‐CEX) 47,49–51 . Additional tools included checklists, 22,23,30 a global rating scale, 14,22,23 the Minicard, 20 a non–milestone‐based end‐of‐shift evaluation, 24,47 the Ottawa Emergency Department Shift Observation Tool, 37 the Reporter/Interpreter/Manager/Educator (RIME) framework, 14 the Standardized Direct Observation Tool (SDOT), 26 the Critical Care Direct Observation Tool (CDOT), 31 and the Resuscitation Assessment Tool (RAT) 46 . A summary of the data for each tool is provided in Table 3.…”

Section: Resultsmentioning

confidence: 99%

See 3 more Smart Citations

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

Gottlieb¹,

Jordan²,

Siegelman³

et al. 2020

AEM Education and Training

View full text Add to dashboard Cite

Objectives: Direct observation is important for assessing the competency of medical learners. Multiple tools have been described in other fields, although the degree of emergency medicine-specific literature is unclear. This review sought to summarize the current literature on direct observation tools in the emergency department (ED) setting. Methods:We searched PubMed, Scopus, CINAHL, the Cochrane Central Register of Clinical Trials, the Cochrane Database of Systematic Reviews, ERIC, PsycINFO, and Google Scholar from 2012 to 2020 for publications on direct observation tools in the ED setting. Data were dual extracted into a predefined worksheet, and quality analysis was performed using the Medical Education Research Study Quality Instrument.Results: We identified 38 publications, comprising 2,977 learners. Fifteen different tools were described. The most commonly assessed tools included the Milestones (nine studies), Observed Structured Clinical Exercises (seven studies), the McMaster Modular Assessment Program (six studies), Queen's Simulation Assessment Test (five studies), and the mini-Clinical Evaluation Exercise (four studies). Most of the studies were performed in a single institution, and there were limited validity or reliability assessments reported. Conclusions:The number of publications on direct observation tools for the ED setting has markedly increased.However, there remains a need for stronger internal and external validity data.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 2 more Smart Citations

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

Gottlieb¹,

Jordan²,

Siegelman³

et al. 2020

AEM Education and Training

View full text Add to dashboard Cite

show abstract

“…In this issue of the Journal of Graduate Medical Education, the article by Donato et al, 1 ''Validity and feasibility of the Minicard direct observation tool in 1 training program,'' illustrates how specific WBA tools may support both assessment functions. Findings from their study suggest that the Minicard direct observation tool facilitates identification of substandard performance and provides feedback to guide and stimulate residents' competence development.…”

mentioning

confidence: 99%

Workplace-Based Assessment and Assessment for Learning: Threats to Validity

Govaerts¹

2015

Journal of Graduate Medical Education

View full text Add to dashboard Cite

O ver the past decades, competency-based education has become the norm in the medical education continuum. The rise of competency-or outcomebased models of education has been paralleled by an increasing interest in workplace-based assessment (WBA) to provide direct evidence of proficiencies of interest (ie, what the trainee will ultimately do in professional practice). WBA has become a cornerstone in the summative assessment of learning and professional competence. Competency-based education, however, also calls for a greater emphasis on formative assessment, or assessment for learning. Frequent feedback on performance should effectively steer and foster the acquisition of the necessary competencies. It therefore seems beyond dispute that assessment programs in competency-based medical education need to combine summative and formative assessment purposes.In this issue of the Journal of Graduate Medical Education, the article by Donato et al, 1 ''Validity and feasibility of the Minicard direct observation tool in 1 training program,'' illustrates how specific WBA tools may support both assessment functions. Findings from their study suggest that the Minicard direct observation tool facilitates identification of substandard performance and provides feedback to guide and stimulate residents' competence development. By obtaining evidence about the quality and quantity of feedback recorded on the Minicard, Donato and colleagues 1 aimed to develop a validity argument for the proposed use of the Minicard for formative assessment (ie, as a tool for learning).Validity is essential in any assessment. In general, validity refers to the evidence that supports or refutes the interpretation and proposed use of assessment results. Results are more or less valid depending on what the intention was in using the assessment, at that particular point in time and for that particular population. Clearly, high-quality feedback is fundamental to formative assess-ment. Feedback should not only include information about observed performance (''feed-back''), but also cues to directions for performance improvement in terms of performance goals and what needs to be done in order to achieve these goals (also known as feed-up and feedforward, respectively). 2 However, if the main purpose of formative assessment is to stimulate further learning and use of feedback for performance improvement, one might argue that the key question to be addressed in the validity inquiry must be whether the assessment actually achieves these goals. Unfortunately, a wealth of research findings indicate that there is no simple answer to the questions of when, for whom, and for what feedback works. [3][4][5] Research on WBA increasingly suggests that combining formative and summative assessment purposes in WBA programs is very difficult, some would say almost impossible to achieve. A recent study by Bok et al, 6 for instance, revealed that low-stakes formative assessments were increasingly being perceived as high stakes and summative, if these assessments were used ...

show abstract

Key dimensions of innovations in workplace-based assessment for postgraduate medical education: a scoping review

Weller

Coomber

Chen

et al. 2021

British Journal of Anaesthesia

View full text Add to dashboard Cite

Background: Specialist training bodies continue to devise innovative methods of gathering information on trainee workplace performance to meet the requirements of competency-based medical education. We reviewed recent innovations in workplace-based assessment (WBA) tools to identify strengths, weaknesses, and trade-offs inherent in their design and use. Methods: In this scoping review, using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we systematically searched databases between 2009 and 2019 for WBA tools with novel characteristics not typically seen in traditional WBAs. These included innovations in rating scales, ways of collecting information, technological innovations, ways of triggering WBAs, and approaches to compiling and using information. Results: We identified 30 innovative WBA tools whose characteristics could be categorised into seven dimensions: frequency of assessment, granularity (unit of performance assessed), coverage of the curriculum, rating method, initiation of the WBA, information use, and incentives. These dimensions had multiple interdependencies and trade-offs, often balancing generating assessment data with available resources. Philosophical stance on assessment also influenced WBA choice, for example prioritising trainee-centred learning (i.e. initiation of WBA and transparency of assessment data), perceptions of assessment and feedback as burdensome or beneficial, and holistic vs reductionist views on assessment of performance. Conclusions: Our synthesis of the literature on innovative WBAs provides a framework for categorising tool characteristics across seven dimensions, systematically teasing apart the considerations in design and use of workplace assessments. It also draws attention to the trade-offs inherent in tool design and selection, and enables a more deliberate consideration of the tool characteristics most appropriate to the local context.

show abstract

Validity and Feasibility of the Minicard Direct Observation Tool in 1 Training Program

Cited by 10 publications

References 18 publications

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

Workplace-Based Assessment and Assessment for Learning: Threats to Validity

Key dimensions of innovations in workplace-based assessment for postgraduate medical education: a scoping review

Contact Info

Product

Resources

About