An accessible analytical approach for investigating what happens between the rounds of a Delphi study

Abstract

The Delphi method is used to investigate consensus amongst a panel of experts using repeated rounds of a questionnaire, often in healthcare settings. However, many Delphi studies do not report any investigation into what happens to the stability of consensus or the convergence of agreement between the rounds in the study, which may be of importance. In this paper an accessible analytical approach is outlined using graphical presentations of means and standard deviations to identify what happens between rounds. For Delphi studies where the scale upon which experts are expressing their opinions can be considered to be interval, the mean will represent the group opinion whilst the standard deviation will represent the level of agreement. An example Delphi study from a healthcare setting is used to illustrate the methodology.

What adult electrocardiogram (ECG) diagnoses and/or findings do residents in emergency medicine need to know?

“…This study used a hierarchical process described by Dajani et al, 25 that included participant consensus (the percentage of participants agreeing on a particular response), followed by stability of opinion in the absence of consensus (consistency of answers between successive rounds of the questionnaire without achieving the pre-defined criteria for consensus), which is consistent with other Delphi studies. [13][14][15]25,26 The demonstration of convergence, (which is a progressive decrease in range and standard deviation of responses as rounds progressed 27 ), suggested increased panelist agreement, and further supported the study conclusions. The anonymous nature of the process avoided the influence of strong personalities and other group dynamics.…”

The Norwegian General Practice (NORGEP) criteria for assessing potentially inappropriate prescriptions to elderly patients

“…Following the feedback from previous rounds, some panellists changed their views, which reinforced the group's opinion. This increase in agreement may thus be the result of constructive feedback during the process, but it is also possible that the panellists conformed to the view held by the majority [35]. The numerous comments (which were subsequently forwarded to the others) suggest that the panellists participated actively in the study and that the feedback process worked satisfactory.…”

“…When there was agreement and the median rating fell within the 33.3Á66.7 range, it was considered equivocal, and individual comments together with scores were used to decide the relevance of the criterion. In the assessment of the dynamic process of the Delphi study, we used the standard deviation (SD) of the mean as a measure for the development of agreement throughout the three rounds [35]. Statistical significance was set at p 5 0.05.…”

What are the most important non-academic attributes of good doctors? A Delphi survey of clinicians

“…Thus, the mean, median, range and standard deviation (SD) for the rating of each item was calculated at each round of the survey. The mean, as a measure of central tendency, is taken to represent the panel's group opinion, and the SD, as a measure of spread, the amount of disagreement within the panel (Greatorex & Dexter 2000;Holey et al 2007).…”