Design and evaluation of an interactive quality dashboard for national clinical audit data: a realist evaluation

Abstract

Background National audits aim to reduce variations in quality by stimulating quality improvement. However, varying provider engagement with audit data means that this is not being realised. Aim The aim of the study was to develop and evaluate a quality dashboard (i.e. QualDash) to support clinical teams’ and managers’ use of national audit data. Design The study was a realist evaluation and biography of artefacts study. Setting The study involved five NHS acute trusts. Methods and results In phase 1, we developed a theory of national audits through interviews. Data use was supported by data access, audit staff skilled to produce data visualisations, data timeliness and quality, and the importance of perceived metrics. Data were mainly used by clinical teams. Organisational-level staff questioned the legitimacy of national audits. In phase 2, QualDash was co-designed and the QualDash theory was developed. QualDash provides interactive customisable visualisations to enable the exploration of relationships between variables. Locating QualDash on site servers gave users control of data upload frequency. In phase 3, we developed an adoption strategy through focus groups. ‘Champions’, awareness-raising through e-bulletins and demonstrations, and quick reference tools were agreed. In phase 4, we tested the QualDash theory using a mixed-methods evaluation. Constraints on use were metric configurations that did not match users’ expectations, affecting champions’ willingness to promote QualDash, and limited computing resources. Easy customisability supported use. The greatest use was where data use was previously constrained. In these contexts, report preparation time was reduced and efforts to improve data quality were supported, although the interrupted time series analysis did not show improved data quality. Twenty-three questionnaires were returned, revealing positive perceptions of ease of use and usefulness. In phase 5, the feasibility of conducting a cluster randomised controlled trial of QualDash was assessed. Interviews were undertaken to understand how QualDash could be revised to support a region-wide Gold Command. Requirements included multiple real-time data sources and functionality to help to identify priorities. Conclusions Audits seeking to widen engagement may find the following strategies beneficial: involving a range of professional groups in choosing metrics; real-time reporting; presenting ‘headline’ metrics important to organisational-level staff; using routinely collected clinical data to populate data fields; and dashboards that help staff to explore and report audit data. Those designing dashboards may find it beneficial to include the following: ‘at a glance’ visualisation of key metrics; visualisations configured in line with existing visualisations that teams use, with clear labelling; functionality that supports the creation of reports and presentations; the ability to explore relationships between variables and drill down to look at subgroups; and low requirements for computing resources. Organisations introducing a dashboard may find the following strategies beneficial: clinical champion to promote use; testing with real data by audit staff; establishing routines for integrating use into work practices; involving audit staff in adoption activities; and allowing customisation. Limitations The COVID-19 pandemic stopped phase 4 data collection, limiting our ability to further test and refine the QualDash theory. Questionnaire results should be treated with caution because of the small, possibly biased, sample. Control sites for the interrupted time series analysis were not possible because of research and development delays. One intervention site did not submit data. Limited uptake meant that assessing the impact on more measures was not appropriate. Future work The extent to which national audit dashboards are used and the strategies national audits use to encourage uptake, a realist review of the impact of dashboards, and rigorous evaluations of the impact of dashboards and the effectiveness of adoption strategies should be explored. Study registration This study is registered as ISRCTN18289782. Funding This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in full in Health and Social Care Delivery Research; Vol. 10, No. 12. See the NIHR Journals Library website for further project information.

How, for whom, and in what contexts will artificial intelligence be adopted in pathology? A realist interview study

“…8 , 11 Realist methods have been used for studying numerous complex interventions, including health information technology (HIT), 12 and in both evaluation and design. 13 …”

“… 15 In realist studies of HIT, relevant contextual factors identified have included resources, such as computing resources and support staff; users’ work routines, motivation, and confidence in HIT use; and the implementation process. 13 Realist studies start with elicitation of stakeholders’ theories. This can be done through interviewing stakeholders, reviewing existing literature, identifying theories from the sociological or other literatures, or some combination of these approaches.…”

“… 15 In realist studies of HIT, relevant contextual factors identified have included resources, such as computing resources and support staff; users’ work routines, motivation, and confidence in HIT use; and the implementation process. 13 …”

Requirements and challenges of hospital dashboards: a systematic literature review

“…Besides, Ghazi Saeedi et al found alarms, drill down, and timely presentation as the central features of quality dashboards [77]. In Randell et al 's study [78], the functional requirements of quality dashboards included visualization, interaction, data quality, reporting, and notification. However, functional capabilities are used in both quality and clinical dashboards depending on the intended environment and purpose.…”

Alinhando Qualidade E Segurança: Estratégias De Excelência Na Gestão De Prontuários

“…Uma das táticas estratégicas, por sua vez, consiste na implementação da auditoria em saúde, uma iniciativa que teve origem no Reino Unido como resposta à necessidade premente de aprimorar a eficiência e eficácia dos serviços de saúde, eliminando práticas que se revelam ineficazes (Randell et al, 2022). Diante do contexto crescente da demanda por serviços médicos e da imperatividade de assegurar a qualidade da assistência, essa abordagem tem-se revelado de importância crucial para a otimização dos procedimentos assistenciais, garantindo, assim, a conformidade dos prontuários médicos com os protocolos estabelecidos (Dias, 2021).…”