Construct validity of a novel assessment tool for ultrasound‐guided axillary brachial plexus block

Abstract

The purpose of this study was to examine the construct validity and reliability of a novel metrics-based assessment tool, previously developed for ultrasound-guided axillary brachial plexus block. Five expert and eight novice anaesthetists performed a total of 18 ultrasound-guided axillary brachial plexus blocks on the same number of patients. A trained investigator video-taped procedures according to a pre-defined protocol. Two trained consultant anaesthetists independently scored the videos using the assessment tool. Compared with novices, experts completed more steps (mean 41.0 vs. 33.1, p = 0.001), had fewer procedural errors (2.8 vs. 7.9, p < 0.0001), had fewer critical errors (0.8 vs. 1.3, p = 0.030), and fewer total errors (3.5 vs. 9.1, p < 0.0001). The mean inter-rater reliability for scoring of experts' performance was 0.91, for novices' performance was 0.84, and for all performance combined (n = 18) was 0.88. This assessment tool is valid, and discriminates reliably between expert and novice performance for placement of ultrasound-guided axillary brachial plexus blocks.

The effect of metrics-based feedback on acquisition of sonographic skills relevant to performance of ultrasound-guided axillary brachial plexus block

“…Our methods outlined an objective approach to providing feedback to improve skills performance of ultrasound‐guided axillary brachial plexus blockade. We used a metric‐based assessment tool that was previously generated using a detailed ‘task characterisation’, and has been proven to demonstrate construct validity and a high degree of reliability within the context of ultrasound‐guided axillary brachial plexus block . The high inter‐rater reliability (as assessed by agreement between the two video assessors) observed in this study is consistent with our previous findings, when the assessment tool was used to assess performance in clinical settings.…”

“…For the metrics‐based feedback group, feedback was provided to each participant in this group using a list of metrics (steps and errors) extracted from a previously validated assessment tool (Table ). Participants practised each item of the checklist in sequence until it was satisfactorily performed, as assessed by the supervising consultant.…”

“…Neither was involved in the provision of feedback during the study. The assessors had undergone training in the application of metrics to performance assessment as previously described by Ahmed et al , and were unaware of the identities of the participants or their group allocation.…”

Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia

“…In this study of UGRA performance, we have demonstrated that both the AC and the GRS may be used to differentiate between individuals who have performed fewer (30) and many (>100) nerve blocks in the preceding year; this finding is consistent with the findings of the recent study of Ahmed and colleagues. 12 However, neither assessment tool was sufficiently sensitive to identify the anaesthetists who had performed an intermediate number of nerve blocks (31e100) in the preceding year. This could represent a Type 2 error, rather than a lack of sensitivity in the assessment tools, although our sample size is in line with similar and recent studies.…”

Development of performance and error metrics for ultrasound-guided axillary brachial plexus block

“…This has been investigated recently using a prospective observational design 22. Future studies are required to evaluate the effectiveness of new training programs based on these metrics.…”