A statistical approach to measuring the competence of anaesthetic trainees at practical procedures

Abstract

Cusum analysis is a statistical technique to distinguish deviations from an acceptable failure rate. The progress of anaesthetic trainees learning four practical procedures (obstetric extradural anaesthesia, spinal anaesthesia, central venous cannulation and arterial cannulation) was monitored from their first attempt using cusum analysis. Suitable acceptable and unacceptable failure rates for each procedure were chosen by consultant anaesthetists. For obstetric extradural anaesthesia, four trainees eventually achieved acceptable failure rates (5%) and the number of attempts required to demonstrate this statistically ranged from 29 to 185; three trainees had an unacceptable failure rate (10%) and five trainees had inconclusive records. For spinal anaesthesia, two trainees achieved an acceptable failure rate (10%) and the number of attempts required to demonstrate this statistically ranged from 39 to 67; two trainees had an unacceptable failure rate (20%) and four trainees had inconclusive records. One trainee demonstrated statistically an acceptable failure rate in arterial cannulation (20%) after 14 attempts and four trainees had inconclusive records. Two records of central venous cannulation were inconclusive. Some records showed variable failure rates which were sometimes associated with lack of practice or a change in technique. Cusum analysis can be used to monitor training in practical procedures and as a continuous audit of quality of clinical practice.

Training in airway management

“…This provides an unambiguous record of training but does not necessarily distinguish poor performance from routine variability. Various statistical methods of sequential analysis have also been used to track performance [117][118][119]. In CUSUM analysis, which is widely used in industry, the number of attempts is plotted against a cumulative score on a graph with boundary limits [117].…”

“…Various statistical methods of sequential analysis have also been used to track performance [117][118][119]. In CUSUM analysis, which is widely used in industry, the number of attempts is plotted against a cumulative score on a graph with boundary limits [117]. However, a very large number of sequential attempts is needed for some procedures to demonstrate statistically acceptable success rates.…”

Assessment of procedural skills training and performance in anesthesia using cumulative sum analysis (cusum)

“…The definition of success and failure also varied between the studies and was another reason for the differing number of trainees attaining competency. In Naik's trial, any degree of pain relief from the epidural signified a success, 17 but both Kestin 16 and de Oliveira Filho 4 had stricter criteria. Kestin required satisfactory analgesia/anesthesia, and de Oliveira Filho required technical success at the first interspace chosen and adequate surgical anesthesia.…”

“…The null hypothesis is: the true failure rate is not different from the acceptable failure rate. The calculations produce decision limits (h0 and h1) and a value for the cusum -s. The cusum value is plotted on the y-axis, and the number of consecutive attempts is plotted on the x-axis, 16 as shown in Fig. 1.…”

“…4,16,17,19 Naik 17 had a cohort of 11 novices, ten of whom achieved cusum defined competency (i.e., crossing the lower boundary line from above) with between 1-85 attempts. In contrast, Kestin 16 found only 4/12 recruits were competent, needing between 29-128 attempts, with five trainees having an unacceptable failure rate. De Oliveira Filho 4 had similar results to those of Kestin, 16 with 4/11 trainees achieving the acceptable failure rate.…”