Use of Expert Judgement Across NICE Guidance-Making Programmes: A Review of Current Processes and Suitability of Existing Tools to Support the Use of Expert Elicitation

Peel, Alison J.; Jenks, Michelle; Choudhury, Moni; Lovett, Rosemary; Rejon‐Parrilla, Juan Carlos; Sims, Andrew; Craig, Joyce

doi:10.1007/s40258-018-0415-5

Cited by 14 publications

(12 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…10 In terms of SEE in health care, NICE uses expert judgement across all guidance-making programmes, but expert elicitation (vs. expert opinion) is used less frequently. 11 Existing timelines and consequent time constraints are reported as the common obstacles when conducting expert elicitation in health care. 11 There is an increasing interest in SEE, as HTAs are conducted progressively closer to the launch of the intervention of interest.…”

Section: Discussionmentioning

confidence: 99%

“…11 Existing timelines and consequent time constraints are reported as the common obstacles when conducting expert elicitation in health care. 11 There is an increasing interest in SEE, as HTAs are conducted progressively closer to the launch of the intervention of interest. 12 SEE is also essential for 'early modelling' of new interventions or unknown diseases for which little or no evidence is available.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

Bojke

Soares

Claxton

et al. 2021

Health Technol Assess

View full text Add to dashboard Cite

Background Many decisions in health care aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations in which evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in a quantitative manner is structured expert elicitation. There is heterogeneity in the existing methodology used in health-care decision-making. A number of guidelines are available for structured expert elicitation; however, it is not clear if any of these are appropriate for health-care decision-making. Objectives The overall aim was to establish a protocol for structured expert elicitation to inform health-care decision-making. The objectives are to (1) provide clarity on methods for collecting and using experts’ judgements, (2) consider when alternative methodology may be required in particular contexts, (3) establish preferred approaches for elicitation on a range of parameters, (4) determine which elicitation methods allow experts to express uncertainty and (5) determine the usefulness of the reference protocol developed. Methods A mixed-methods approach was used: systemic review, targeted searches, experimental work and narrative synthesis. A review of the existing guidelines for structured expert elicitation was conducted. This identified the approaches used in existing guidelines (the ‘choices’) and determined if dominant approaches exist. Targeted review searches were conducted for selection of experts, level of elicitation, fitting and aggregation, assessing accuracy of judgements and heuristics and biases. To sift through the available choices, a set of principles that underpin the use of structured expert elicitation in health-care decision-making was defined using evidence generated from the targeted searches, quantities to elicit experimental evidence and consideration of constraints in health-care decision-making. These principles, including fitness for purpose and reflecting individual expert uncertainty, were applied to the set of choices to establish a reference protocol. An applied evaluation of the developed reference protocol was also undertaken. Results For many elements of structured expert elicitation, there was a lack of consistency across the existing guidelines. In almost all choices, there was a lack of empirical evidence supporting recommendations, and in some circumstances the principles are unable to provide sufficient justification for discounting particular choices. It is possible to define reference methods for health technology assessment. These include a focus on gathering experts with substantive skills, eliciting observable quantities and individual elicitation of beliefs. Additional considerations are required for decision-makers outside health technology assessment, for example at a local level, or for early technologies. Access to experts may be limited and in some circumstances group discussion may be needed to generate a distribution. Limitations The major limitation of the work conducted here lies not in the methods employed in the current work but in the evidence available from the wider literature relating to how appropriate particular methodological choices are. Conclusions The reference protocol is flexible in many choices. This may be a useful characteristic, as it is possible to apply this reference protocol across different settings. Further applied studies, which use the choices specified in this reference protocol, are required. Funding This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 37. See the NIHR Journals Library website for further project information. This work was also funded by the Medical Research Council (reference MR/N028511/1).

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

Bojke

Soares

Claxton

et al. 2021

Health Technol Assess

View full text Add to dashboard Cite

show abstract

“…Combining the literature review with the judgment of experts during guideline development is particularly important in fields in which less data are available on interventions and outcomes. 26 According to the current RCTs and meta-analyses, strategies for preventing inpatient falls in hospitals have relatively minor effects on the actual fall rates even when complex and multidisciplinary interventions are applied. 27 High-quality evidence for the effectiveness of fall prevention interventions in acute care settings is limited.…”

Section: Discussionmentioning

confidence: 99%

Bibliometrics and Co-Citation Network Analysis of Systematic Reviews of Evidence-Based Nursing Guidelines for Preventing Inpatient Falls

Cho

Kim

Park

2021

CIN: Computers, Informatics, Nursing

View full text Add to dashboard Cite

Advances in bibliometrics and co-citation analysis provide the opportunity to analyze quantitatively the large amount of nursing research used in evidence-based nursing. Numerous nursing researchers have attempted to obtain evidence that using evidence-based nursing improves the quality of nursing practices. However, little is known about how these efforts comply with the rigorous methods required for a systematic review in a subject area. This study explored the comprehensiveness of systematic reviews in four guidelines for preventing inpatient falls using bibliometrics and a co-citation network technique. Citations (n = 659) and 9417 unique bibliographic records written by 6537 authors were collected from the guidelines in January 2020. The results showed none of the references spanned all four authoring bodies, whereas only 0.1% and 4.7% of the references spanned three and two of the authoring bodies, respectively. The co-citation analysis revealed differences in the scope and primary concerns among the development groups, even in the same setting, with differences in the highly influential articles and authors. These findings imply that although the systematic reviews of guidelines on fall prevention were of good quality, there remain areas for improvement in terms of harmonizing the selection of bibliographic citations comprehensively.

show abstract

“…Bojke et al (27) recently undertook an expert elicitation exercise to validate a new framework and this took 5 months of researcher time. Indeed, even in the relatively well-resourced environment at the National Institute for Health and Care Excellence (NICE), expert elicitation is often done informally (28). Our review identified a number of examples of the application of structured methods of stakeholder consultation, but studies have generally been undertaken in the context of a translational research body such as the Center for Translational Molecular Medicine (CTMM) (29;30).…”

Section: Methods Of Development-focused Htamentioning

confidence: 99%

A toolkit of methods of development-focused health technology assessment

Bouttell

Briggs

Hawkins

2021

Int J Technol Assess Health Care

View full text Add to dashboard Cite

Health technology assessment conducted to inform decisions during technology development (development-focused or DF-HTA) has a number of distinct features compared with HTA conducted to inform reimbursement and usage decisions. In particular, there are a broad range of decisions to be informed related to the development of a technology; multiple markets and decision makers to be considered; a limited (and developing) evidence base; and constrained resources for analysis. These features impact upon methods adopted by analysts. In this paper, we (i) set out methods of DF-HTA against a timeline of technology development; (ii) provide examples of the methods’ use; and (iii) explain how they have been adapted as a result of the features of DF-HTA. We present a toolkit of methods for analysts working with developers of medical technologies. Three categories of methods are described: literature review, stakeholder consultation, and decision analytic modeling. Literature review and stakeholder consultation are often used to fill evidence gaps. Decision analytic modeling is used to synthesize available evidence alongside plausible assumptions to inform developers about price or performance requirements. Methods increase in formality and complexity as the development and evidence base progresses and more resources are available for assessment. We hope this toolkit will be used in conjunction with the framework of features of DF-HTA presented in our earlier article in order to improve the clarity and appropriateness of methods of HTA used in DF-HTA. We also seek to contribute to a continuing dialogue about the nature of, and the best approach to, DF-HTA.

show abstract

Use of Expert Judgement Across NICE Guidance-Making Programmes: A Review of Current Processes and Suitability of Existing Tools to Support the Use of Expert Elicitation

Cited by 14 publications

References 11 publications

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study

Bibliometrics and Co-Citation Network Analysis of Systematic Reviews of Evidence-Based Nursing Guidelines for Preventing Inpatient Falls

A toolkit of methods of development-focused health technology assessment

Contact Info

Product

Resources

About