The Quality of Assessment of Learning (Qual) Score: Validity Evidence for a Scoring System Aimed at Rating Short, Workplace-Based Comments on Trainee Performance

Chan, Teresa M.; Sebok‐Syer, Stefanie S; Sampson, Christopher; Monteiro, Sandra

doi:10.1080/10401334.2019.1708365

Cited by 30 publications

(44 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Not much is known about how general factors determining entrustment can be derived from feedback and how feedback content varies with associated entrustment ratings. Despite this intriguing question, it would require extensive resource and human effort to analyse narrative feedback data qualitatively to extract factors that might guide learners in understanding what it takes to enhance trust in their task execution among supervisors 19 …”

Section: Introductionmentioning

confidence: 99%

“…Despite this intriguing question, it would require extensive resource and human effort to analyse narrative feedback data qualitatively to extract factors that might guide learners in understanding what it takes to enhance trust in their task execution among supervisors. 19 In this study, we sought to investigate factors associated with ad hoc entrustment decisions as documented by in-the-moment assessments completed together by clerkship-year medical students and their clinical supervisors. Generated from entrustment-granting encounters during the core clerkships, this assessment yielded a large (3300 observation) dataset of feedback narratives paired with retrospective entrustment-supervision (ES) levels-a numeric rating indicating how much supervision a student required to complete a clinic task.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Exploring how feedback reflects entrustment decisions using artificial intelligence

et al. 2021

View full text Add to dashboard Cite

Context Clinical supervisors make judgements about how much to trust learners with critical activities in patient care. Such decisions mediate trainees' opportunities for learning and competency development and thus are a critical component of education. As educators apply entrustment frameworks to assessment, it is important to determine how narrative feedback reflecting entrustment may also address learners' educational needs. Methods In this study, we used artificial intelligence (AI) and natural language processing (NLP) to identify characteristics of feedback tied to supervisors' entrustment decisions during direct observation encounters of clerkship medical students (3328 unique observations). Supervisors conducted observations of students and collaborated with them to complete an entrustment‐based assessment in which they documented narrative feedback and assigned an entrustment rating. We trained a deep neural network (DNN) to predict entrustment levels from the narrative data and developed an explainable AI protocol to uncover the latent thematic features the DNN used to make its prediction. Results We found that entrustment levels were associated with level of detail (specific steps for performing clinical tasks), feedback type (constructive versus reinforcing) and task type (procedural versus cognitive). In justifying both high and low levels of entrustment, supervisors detailed concrete steps that trainees performed (or did not yet perform) competently. Conclusions Framing our results in the factors previously identified as influencing entrustment, we find a focus on performance details related to trainees' clinical competency as opposed to nonspecific feedback on trainee qualities. The entrustment framework reflected in feedback appeared to guide specific goal‐setting, combined with details necessary to reach those goals. Our NLP methodology can also serve as a starting point for future work on entrustment and feedback as similar assessment datasets accumulate.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Exploring how feedback reflects entrustment decisions using artificial intelligence

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Notably, a tool for the assessment of narrative comments in competency-based assessments has already been developed and demonstrated some validity evidence. 44 In the future, we anticipate that natural language processing algorithms could assess narrative comments for quality using such a tool and report its results on a faculty dashboard along with guidance regarding how the faculty member's assessment feedback could be improved. More advanced still, it may be possible to have such an algorithm analyze comments as they are being entered and provide this guidance to the faculty member at the 'point of assessment.…”

Section: Discussionmentioning

confidence: 99%

“…The senior author (BT) reviewed the literature on feedback in faculty development, [43][44][45] learning analytics, 46,47 and data visualization 32,38,46,48 to generate ideas for the initial iteration of the faculty development dashboard. He then facilitated interviews with the program leaders and faculty development experts.…”

Section: Phase 1 Analysis and Explorationmentioning

confidence: 99%

Developing a dashboard for faculty development in competency-based training programs: a design-based research project

et al. 2021

View full text Add to dashboard Cite

Background: Canadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires frequent assessments of entrustable professional activities (EPAs). Faculty struggle to provide helpful feedback and assign appropriate entrustment scores. CBME faculty development initiatives rarely incorporate teaching metrics. Dashboards could be used to visualize faculty assessment data to support faculty development. Methods: Using a design-based research process, we identified faculty development needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. Data was collected within the emergency medicine residency program at the University of Saskatchewan through interviews with program leaders, faculty development experts, and faculty participating in development sessions. Two investigators thematically analyzed interview transcripts to identify faculty needs that were audited by a third investigator. The needs were described using representative quotes and the dashboard elements designed to address them. Results: Between July 1, 2019 and December 11, 2020 we conducted 15 interviews with nine participants (two program leaders, three faculty development experts, and four faculty members). Three needs emerged as themes from the analysis: analysis of assessments, contextualization of assessments, and accessible reporting. We addressed these needs by designing an accessible dashboard to present contextualized quantitative and narrative assessment data for each faculty member. Conclusions: We identified faculty development needs related to EPA assessments and designed dashboard elements to meet them. The resulting dashboard was used for faculty development sessions. This work will inform the development of CBME assessment dashboards for faculty.

show abstract

“…6 The CCERR serves well for its intended purpose but is cumbersome to use for assessments that include brief captures of formative feedback. Two recently published tools, the Quality of Assessment of Learning (QuAL) score 7 and the Quality Improvement Instrument (QII), 8 also have value, but include assumptions about how feedback will be structured, which limits their generalizability. The QuAL score emphasizes the description of the resident performance and requires that feedback include a suggestion for improvement.…”

Section: Introductionmentioning

confidence: 99%

Development of and Preliminary Validity Evidence for the EFeCT Feedback Scoring Tool

Ross

Hamza

Zulla

et al. 2022

Journal of Graduate Medical Education

View full text Add to dashboard Cite

Background Narrative feedback, like verbal feedback, is essential to learning. Regardless of form, all feedback should be of high quality. This is becoming even more important as programs incorporate narrative feedback into the constellation of evidence used for summative decision-making. Continuously improving the quality of narrative feedback requires tools for evaluating it, and time to score. A tool is needed that does not require clinical educator expertise so scoring can be delegated to others. Objective To develop an evidence-based tool to evaluate the quality of documented feedback that could be reliably used by clinical educators and non-experts. Methods Following a literature review to identify elements of high-quality feedback, an expert consensus panel developed the scoring tool. Messick's unified concept of construct validity guided the collection of validity evidence throughout development and piloting (2013–2020). Results The Evaluation of Feedback Captured Tool (EFeCT) contains 5 categories considered to be essential elements of high-quality feedback. Preliminary validity evidence supports content, substantive, and consequential validity facets. Generalizability evidence supports that EFeCT scores assigned to feedback samples show consistent interrater reliability scores between raters across 5 sessions, regardless of level of medical education or clinical expertise (Session 1: n=3, ICC=0.94; Session 2: n=6, ICC=0.90; Session 3: n=5, ICC=0.91; Session 4: n=6, ICC=0.89; Session 5: n=6, ICC=0.92). Conclusions There is preliminary validity evidence for the EFeCT as a useful tool for scoring the quality of documented feedback captured on assessment forms. Generalizability evidence indicated comparable EFeCT scores by raters regardless of level of expertise.

show abstract

The Quality of Assessment of Learning (Qual) Score: Validity Evidence for a Scoring System Aimed at Rating Short, Workplace-Based Comments on Trainee Performance

Cited by 30 publications

References 41 publications

Exploring how feedback reflects entrustment decisions using artificial intelligence

Exploring how feedback reflects entrustment decisions using artificial intelligence

Developing a dashboard for faculty development in competency-based training programs: a design-based research project

Development of and Preliminary Validity Evidence for the EFeCT Feedback Scoring Tool

Contact Info

Product

Resources

About