Do Consensus Conferences Work? A Process Evaluation of the NIH Consensus Development Program

Abstract

The purpose of this evaluation study is to identify problems and suggest modifications in the NIH Consensus Development Program. The current program consists of three-day conferences in which experts assess medical technologies for issues of efficacy, safety, conditions of use, and other related topics (e.g., costs and social impact). Eight consensus conferences held between 1980 and 1982 were studied in depth using a variety of methods; five of the conferences were investigated concurrently. In addition, archival material was examined for all but one of the 33 conferences held up to that time, and four planning meetings for future conferences were observed. The delay in publishing our findings provided an opportunity to examine the changes introduced by NIH; it also allowed us to avoid the criticism of numerous prior evaluations for finding fault with programs that are still developing. NIH adopted many of the recommendations in our evaluation report and has investigated others. Based on our evaluation and more recent evidence, however, we conclude that the major problem that was uncovered--selection bias, particularly with respect to the choice of questions and panelists--remains a significant threat to the credibility of the consensus process. More specifically, the results indicate that controversial issues cannot be properly addressed within the present conference format, although that was one of its major purposes. Recommendations for improving the consensus process are presented, as are their implications for a larger set of consensus activities that are currently being conducted.

Words Without Action? The Production, Dissemination, and Impact of Consensus Recommendations

“…Most exercises have not made the criteria for resolution explicit, although many have pointed to the responsibility of the chairperson to ensure smooth operation of this aspect of group process (37,42,99). Many panels have an implicit hierarchy of criteria that place con trolled trial research above other, less methodologically stringent research evidence, which in tum is valued more highly than clinical experience or personal opinion.…”

“…Many researchers have stressed the importance of a readily understandable and accessible report. They recommend that a professional writer should at least be involved with the final draft (39,46,99).…”

“…This pressure has resulted in one strategy that has the consensus group drafting and finalizing the report over a period of 24 to 48 hours (39). The approach has been highly criticized because it leads to "lowest common denominator" recommendations on particularly difficult and controversial issues (99), or "is bound to lead to hurried conclusions" (68, p. 1088).…”

Wildlife Health 2.0: Bridging the Knowledge-to-Action Gap

“…57 Particpants may be invited to represent an interest group officially or they may sit as individuals who, although they come with a background from a particular interest group, represent only themselves. A failing noted for some consensus conferences is the arbitrariness in the selection of panel members 59 leading to selection bias.…”

Preparing guidelines and documented clinical policies

“…These include the extensive National Institutes of Health [35,36] and RAND Corporation [37,38] work in the United States, and more limited use in Europe [39] and in the UK by the Kings Fund [40] and other health service researchers [41]. Formal consensus methods define participant selection and roles, together with the process through which consensus is to be reached, with care.…”