Consensus-Based Expert Development of Critical Items for Direct Observation of Point-of-Care Ultrasound Skills

Abstract

Background Point-of-care ultrasound (POCUS) is increasingly used in a number of medical specialties. To support competency-based POCUS education, workplace-based assessments are essential. Objective We developed a consensus-based assessment tool for POCUS skills and determined which items are critical for competence. We then performed standards setting to set cut scores for the tool. Methods Using a modified Delphi technique, 25 experts voted on 32 items over 3 rounds between August and December 2016. Consensus was defined as agreement by at least 80% of the experts. Twelve experts then performed 3 rounds of a standards setting procedure in March 2017 to establish cut scores. Results Experts reached consensus for 31 items to include in the tool. Experts reached consensus that 16 of those items were critically important. A final cut score for the tool was established at 65.2% (SD 17.0%). Cut scores for critical items are significantly higher than those for noncritical items (76.5% ± SD 12.4% versus 53.1% ± SD 12.2%, P < .0001). Conclusions We reached consensus on a 31-item workplace-based assessment tool for identifying competence in POCUS. Of those items, 16 were considered critically important. Their importance is further supported by higher cut scores compared with noncritical items.

Use of Critical Items in Determining Point-of-Care Ultrasound Competence

“…Details on tool development and the standard setting processes have previously been described (Ma et al, 2020); in 2016, 25 experts participated in three rounds of a modified Delphi process and reached consensus on 31 items for an assessment tool intended for use on medical residents to identify competence in POCUS. Competent performances were defined as minimally acceptable performances that did not require supervision.…”

“…We recently developed a consensus-based assessment tool and performed a standard setting procedure on the full tool and a focused critical item tool (Ma et al, 2020). This is a follow-up study to evaluate the evidence (Cook et al, 2015) that scores from a critical item tool accurately identify competence in POCUS.…”

Minimal Criteria for Lung Ultrasonography in Internal Medicine

“…8 Despite it being relatively easy to learn, to support the use of LUS clinically, appropriate training in image acquisition and interpretation is required. It is generally accepted by POCUS experts that images not meeting minimal criteria, such as those that have insufficient depth or inadequate optimization of the pleural line, should not be used to support clinical decision-making, 9 as they may introduce serious diagnostic errors. Both achieving minimal criteria during image acquisition and the recognition of when these criteria have not been met are fundamental to POCUS competency.…”

“…Both achieving minimal criteria during image acquisition and the recognition of when these criteria have not been met are fundamental to POCUS competency. 9 In addition, the use of minimal criteria in POCUS training programs is considered a key indicator of high-quality education. 10 Despite the critical importance of minimal criteria, existing resources on learning LUS, while useful, focus primarily on image acquisition and interpretation, 4,11,12 without an explicit list of minimal criteria that an image must meet.…”

“…As POCUS education expands across the country, articulation of such criteria becomes increasingly important if high-quality education is to be standardized. Given that trained POCUS experts agree on the importance of minimal criteria, 9 through consensus methods, this study seeks to encapsulate this codex of expert knowledge into an explicit list of minimal criteria for LUS. These minimal criteria can then be used to guide clinical use as well as LUS education for residency training programs.…”