Diagnostic quality model (DQM): an integrated framework for the assessment of diagnostic quality when using AI/ML

Abstract

Background Laboratory medicine has reached the era where promises of artificial intelligence and machine learning (AI/ML) seem palpable. Currently, the primary responsibility for risk-benefit assessment in clinical practice resides with the medical director. Unfortunately, there is no tool or concept that enables diagnostic quality assessment for the various potential AI/ML applications. Specifically, we noted that an operational definition of laboratory diagnostic quality – for the specific purpose of assessing AI/ML improvements – is currently missing. Methods A session at the 3rd Strategic Conference of the European Federation of Laboratory Medicine in 2022 on “AI in the Laboratory of the Future” prompted an expert roundtable discussion. Here we present a conceptual diagnostic quality framework for the specific purpose of assessing AI/ML implementations. Results The presented framework is termed diagnostic quality model (DQM) and distinguishes AI/ML improvements at the test, procedure, laboratory, or healthcare ecosystem level. The operational definition illustrates the nested relationship among these levels. The model can help to define relevant objectives for implementation and how levels come together to form coherent diagnostics. The affected levels are referred to as scope and we provide a rubric to quantify AI/ML improvements while complying with existing, mandated regulatory standards. We present 4 relevant clinical scenarios including multi-modal diagnostics and compare the model to existing quality management systems. Conclusions A diagnostic quality model is essential to navigate the complexities of clinical AI/ML implementations. The presented diagnostic quality framework can help to specify and communicate the key implications of AI/ML solutions in laboratory diagnostics.

Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

“…Overall, a total 29 of guidelines or tools related to quality assessment or control in the past decade (mainly in the last four years), with 5 for developing quality, 14 for reporting quality, and 10 for both (Additional file 2 : Table S6) [ 11 , 30 – 32 , 34 , 38 , 40 , 41 , 44 – 46 , 128 – 145 ]. In addition to the study design, statistical methods, model performance, risk of bias, AI ethics risk, replicability, as well as clinical implementation, application, and implication in both developing and reporting assessments, the complexity and standardization of data acquisition and processing, required resources (such as software platforms, hardware, or technical professionals), and cost-effectiveness are also focal points in many developing assessments.…”

Insights from semi-structured interviews on integrating artificial intelligence in clinical chemistry laboratory practices

“…Despite laboratory medicine databases being a valuable source of data, they often lack suitability for the application of data science techniques. Certain medical difficulties can be efficiently solved using the computing power of AI solutions when the data is of high quality and supplied in the suitable manner [ 27 ]. Furthermore, the participants stressed the necessity for data standardization to facilitate the widespread adoption of AI in laboratories throughout the country.…”

Translating the regulatory landscape of medical devices to create fit‐for‐purpose artificial intelligence (AI) cytometry solutions

“…Radiology for example has adopted several types or classes of AI/ML‐enabled devices into the clinical environment to improve the efficiency, accuracy, or consistency of the medical image interpretation process (Petrick et al, 2023), and AI‐based assessments of peripheral blood smears and other body fluids is now an integral component of laboratory testing, as evidenced by the fact that most of currently FDA approved AI/ML hematology devices are automated image analyzers (Center for Devices and Radiological Health, 2023h). Furthermore, the laboratory medicine community is increasingly focusing on developing strategies to successfully integrate AI into their workflows and to assess diagnostic quality (Lennerz et al, 2023). For example, the 2022 European Federation of Clinical Chemistry and Laboratory Medicine strategic conference emphasized that the modularized nature of laboratory processes makes this area particularly amenable to AI solutions, but that laboratory specialists and technicians will continue to improve the analytical portfolio, diagnostic quality and laboratory turnaround times while expertise in AI implementation and partnerships with industry will become important professional competencies (Carobene et al, 2023).…”