Theoretical and methodological considerations in evaluating large-scale health information technology change programmes

Abstract

Background: Attempts to achieve digital transformation across the health service have stimulated increasingly large-scale and more complex change programmes. These encompass a growing range of functions in multiple locations across the system and may take place over extended timeframes. This calls for new approaches to evaluate these programmes. Main body: Drawing on over a decade of conducting formative and summative evaluations of health information technologies, we here build on previous work detailing evaluation challenges and ways to tackle these. Important considerations include changing organisational, economic, political, vendor and markets necessitating tracing of evolving networks, relationships, and processes; exploring mechanisms of spread; and studying selected settings in depth to understand local tensions and priorities. Conclusions: Decision-makers need to recognise that formative evaluations, if built on solid theoretical and methodological foundations, can help to mitigate risks and help to ensure that programmes have maximum chances of success.

Driving digital health transformation in hospitals: a formative qualitative evaluation of the English Global Digital Exemplar programme

“…Key here was setting national goals and monitoring progress, while allowing a degree of local freedom over how to achieve these goals. 45 Experiences with other national initiatives reinforce the effectiveness of balancing goal-setting with local choice, a perspective that is supported by the notion of loose coupling where organisational subsystems function well if they can maintain their own identity and autonomy. [47][48][49] The US HITECH initiative reinforces the important role of centrally allocated funding and goal setting in facilitating adoption.…”

“…Attribution of outcomes in large-scale digital transformation initiatives is not straightforward, as interventions are often multifaceted, stimulating digitalisation through a combination of enhancements in technological systems and organisational processes-as a result, outcomes take a long time to materialise and may not then be directly attributable to HIT. 45 In addition, large-scale change programmes are situated within evolving wider policy and economic settings that may influence outcomes. Various local factors are also likely to have an impact.…”

Promoting inter-organisational knowledge sharing: A qualitative evaluation of England’s Global Digital Exemplar and Fast Follower Programme

“…Although the longitudinal qualitative nature of data collection has allowed us to gain insights into unfolding relationships over time, it has not been sufficient to allow us to link identified processes to implementation outcomes. This shortcoming reflects a general issue with complex transformational programmes, where outcomes emerge gradually and are often difficult to attribute [ 20 ]. This work is based on insights derived from evaluating the first 24 months of the GDE Programme.…”

The Tension between National and Local Concerns in Preparing for Large-Scale Generic Systems in Healthcare

“…Given the size and scope of generic systems, there are also often several autonomous health institutions involved, all of which must be convinced that the new system will provide benefits for them before they will participate. While a formative evaluation approach cannot avoid the issues associated with large-scale projects, it can help navigate the associated complexities by capturing the perceptions of the stakeholders involved and feeding findings back to program management (Cresswell et al 2020 , p. 2).…”