Evaluating policy and service interventions: framework to guide selection and interpretation of study end points

Lilford, Richard; Chilton, Peter J; Hemming, Karla; Girling, Alan; Taylor, Celia A.; Barach, Paul

doi:10.1136/bmj.c4413

Cited by 121 publications

(96 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Models that are likely to have no impact should not proceed to prospective comparative impact studies. Accordingly, preceding decision analytic studies help in separating the chaff from the wheat [45].…”

Section: Consider To First Do a Decision Analytic Studymentioning

confidence: 99%

Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

et al. 2018

View full text Add to dashboard Cite

An important aim of clinical prediction models is to positively impact clinical decision making and subsequent patient outcomes. The impact on clinical decision making and patient outcome can be quantified in prospective comparative-ideally cluster-randomized-studies, known as 'impact studies'. However, such impact studies often require a lot of time and resources, especially when they are (cluster-)randomized studies. Before envisioning such large-scale randomized impact study, it is important to ensure a reasonable chance that the use of the prediction model by the targeted healthcare professionals and patients will indeed have a positive effect on both decision making and subsequent outcomes. We recently performed two differently designed, prospective impact studies on a clinical prediction model to be used in surgical patients. Both studies taught us new valuable lessons on several aspects of prediction model impact studies, and which considerations may guide researchers in their decision to conduct a prospective comparative impact study. We provide considerations on how to prepare a prediction model for implementation in practice, how to present the model predictions, and how to choose the proper design for a prediction model impact study.

show abstract

Section: Consider To First Do a Decision Analytic Studymentioning

confidence: 99%

Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Moreover, EP economic evaluation studies are challenging due to the diffuse effect of EP on many clinical processes across an institution [23]. Our review shows that studies exploring the economic impact of EP in this context are scarce.…”

Section: Implications For Clinical Practicementioning

confidence: 99%

Economic impact of electronic prescribing in the hospital setting: A systematic review

Ahmed

Barber

Jani

et al. 2016

International Journal of Medical Informatics

View full text Add to dashboard Cite

Objective: To examine evidence on the economic impact of electronic prescribing (EP) systems in the hospital setting. Method: We conducted a systematic search of MEDLINE, EMBASE, PsycINFO, International Pharmaceutical Abstracts, the NHS Economic Evaluation Database, the European Network of Health Economic Evaluation Database and Web of Science from inception to October 2013. Full and partial economic evaluations of EP or computerised provider order entry were included. We excluded studies assessing prescribing packages for specific drugs, and monetary outcomes that were not related to medicines. A checklist was used to evaluate risk of bias and evidence quality. Results: The search yielded 1,160 articles of which three met the inclusion criteria. Two were full economic evaluations and one a partial economic evaluation. A meta-analysis wasn't appropriate as studies were heterogeneous in design, economic evaluation method, interventions and outcome measures. Two studies investigated the financial impact of reducing preventable adverse drug events. The third measured savings related to various aspects of the system including those related to medication. Two studies reported positive financial effects. However the overall quality of the economic evidence was low and key details often not reported. Discussion There seems to be some evidence of financial benefits of EP in the hospital setting. However, it is not clear if evidence is transferable to other settings. Research is scarce and limited in quality, and reported methods are not always transparent. Further robust, high quality research is required to establish if hospital EP is cost effective and thus inform policy makers' decisions. We are pleased to submit our manuscript entitled: "Economic impact of electronic prescribing in the hospital setting: a systematic review". This systematic review examines the evidence to establish if electronic prescribing systems use in the hospital setting is cost effective. We conducted a systematic search of seven databases from inception to October 2013 and included full and partial economic evaluations of electronic prescribing systems or computerised provider order entry. COVER LETTERWe found some evidence of financial benefits of EP use. However, it is not clear if evidence is transferable to other settings. Our review also shows that studies exploring the economic impact of electronic prescribing systems in this context are scarce and limited in quality. Therefore we endorse further robust research to establish if electronic prescribing systems use in hospitals is good value for money. This manuscript has not been previously published and is not under consideration in the same or substantially similar form in any other peer-reviewed media. All authors listed have contributed sufficiently to the project to be included as authors, and all those who are qualified to be authors are listed in the author byline. To the best of our knowledge, no conflict of interest, financial or other, exists. We have included acknowle...

show abstract

“…CONCEPTUAL FRAMEWORK: Our conceptual framework aligns closely with a modified Donabedian framework proposed by Lilford et al (2010), which, examines the relationships between structures, processes and outcomes, but which separates the processes into three further subcategories: clinical processes, (involving the actual treatment, such as the processes of the rehabilitation, medication); targeted processes (designed to improve clinical processes through training in the use of a device); and generic processes (such as skill mix or staffing models adopted by an organisation).…”

Section: Methodsmentioning

confidence: 99%

Secondary analysis and literature review of community rehabilitation and intermediate care: an information resource

Ariss

Enderby

Smith

et al. 2015

Health Services and Delivery Research

View full text Add to dashboard Cite

Background and designThis research was based on a reanalysis of a merged data set from two intermediate care (IC) projects in order to identify patient characteristics associated with outcomes [Nancarrow SA, Enderby PM, Moran AM, Dixon S, Parker SG, Bradburn MJ,et al.The Relationship Between Workforce Flexibility and the Costs and Outcomes of Older Peoples’ Services (COOP). Southampton: National Institute for Health Research (NIHR) Service Delivery and Organisation (SDO); 2010 and Nancarrow SA, Enderby PM, Ariss SM, Smith T, Booth A, Campbell MJ,et al.The Impact of Enhancing the Effectiveness of Interdisciplinary Working (EEICC). Southampton: NIHR SDO; 2012]. Additionally, the impact of different team and staffing structures on patient outcomes and service costs was examined, when possible given the data sets, to enable identification of the most cost-effective service configurations and change over time with service provision. This secondary analysis was placed within updated literature reviews focused on the separate questions.Research objectives(1) To identify those patients most likely to benefit from IC and those who would be best placed to receive care elsewhere; (2) to examine the effectiveness of different models of IC; (3) to explore the differences between IC service configurations and how they have changed over time; and (4) to use the findings above to develop accessible evidence to guide service commissioning and monitoring.SettingCommunity-based services for older people are described in many different ways, among which are IC services and community rehabilitation. For the purposes of this report we call the services IC services and include all community-based provision for supporting older people who would otherwise be admitted to hospital or who would require increased length of stay in hospital (e.g. hospital at home schemes, post-acute care, step-up and step-down services).ParticipantsThe combined data set contained data on 8070 patient admissions from 32 IC teams across England and included details of the service context, costs, staffing/skill mix (800 staff), patient health status and outcomes.InterventionsThe interventions associated with the study cover the range of services and therapies available in IC settings. These are provided by a wide range of professionals and care staff, including nursing, allied health and social care.Outcome measures(1) Service data – each team provided information relating to the size, nature, staffing and resourcing of the services. Data were collected on a service pro forma. (2) Team data – all staff members of the teams participating in both studies provided individual information using the Workforce Dynamics Questionnaire. (3) Patient data – patient data were collected on admission and discharge using a client record pack. The client record pack recorded a range of data utilising a number of validated tools, such as demographic data, level of care (LoC) data, therapy outcome measure (TOM) scale, European Quality of Life-5 Dimensions (EQ-5D) questionnaire and patient satisfaction survey.Results(1) The provision of IC across England is highly variable with different referral routes, team structures, skill mix and cost-effectiveness; (2) in more recent years, patients referred to IC have more complex needs associated with more severe impairments; (3) patients most likely to improve were those requiring rehabilitation as determined by levels 3, 4 and 5 on the LoC (> 40% for impairment, activity and participation, and > 30% for well-being as determined on the TOM scale); (4) half of all patients with outcome data improved on at least one of the domains of the TOM scale; (5) for every 10-year increase in age there was a 6% decrease in the odds of returning home. The chance of remaining or returning home was greater for females than males; (6) a high percentage of patients referred to IC do not require the service; and (7) teams including clinical support staff and domiciliary staff were associated with a small relative improvement in TOM impairment scores when compared with other teams.ConclusionsThis study provides additional evidence that interdisciplinary teamworking in IC may be associated with better outcomes for patients, but care should be taken with overinterpretation. The measures that were used within the studies were found to be reliable, valid and practical and could be used for benchmarking. This study highlights the need for funding high-quality studies that attempt to examine what specific team-level factors are associated with better outcomes for patients. It is therefore important that studies in the future attempt empirically to examine what process-level team variables are associated with these outcomes.FundingThe NIHR Health Services and Delivery Research programme.

show abstract

Evaluating policy and service interventions: framework to guide selection and interpretation of study end points

Abstract: The effect of many cost effective policy and service interventions cannot be detected at the level of the patient. This new framework could help improve the design (especially choice of primary end point) and interpretation of evaluative studies

Cited by 121 publications

References 35 publications

Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

Economic impact of electronic prescribing in the hospital setting: A systematic review

Secondary analysis and literature review of community rehabilitation and intermediate care: an information resource

Contact Info

Product

Resources

About