METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII

Abstract

Purpose To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies. Methods We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated. Result In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community. Conclusion In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers. Critical relevance statement A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning. Key points • A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html) and a repository created to collect feedback from the radiomics community (https://github.com/metricsscore/metrics). Graphical Abstract

Radiomics diagnostic performance for predicting lymph node metastasis in esophageal cancer: a systematic review and meta-analysis

“…Finally, METRICS is accompanied by a user-friendly web application and a repository for community feedback, facilitating its adoption and continuous improvement. Overall, METRICS represents a significant advancement in the field, offering a robust and adaptable tool for enhancing the methodological rigor of radiomics research [ 51 , 52 ].…”

Explanation and Elaboration with Examples for CLEAR (CLEAR-E3): an EuSoMII Radiomics Auditing Group Initiative

“…For the latter case, the use of quality assessment tools ( i.e. , METRICS) is strongly recommended [ 24 ]. Furthermore, CLEAR-E3 is intended as a complement and not a substitute for CLEAR.…”

“…With its recent extended effort on convolutional filters [ 21 ], the Image Biomarker Standardisation Initiative (IBSI) has been a significant driving force to standardize feature computation-related aspects [ 22 ]. Regarding transparent reporting and methodological quality evaluation, new consensus guidelines have been published and endorsed by prominent imaging societies [ 23 , 24 ].…”

“…However, CLEAR is intended to be a reporting guideline rather than a quality assessment tool. For the latter task, the METhodological RadiomICs Score (METRICS), also endorsed by EuSoMII, has been developed to provide a structured approach for post hoc research evaluation [ 24 ].…”

Deep-learning segmentation to select liver parenchyma for categorizing hepatic steatosis on multinational chest CT

“…Notably, its potential has been demonstrated in the realms of automated liver segmentation and steatosis assessments 16 , 30 – 34 . However, these approaches have not adequately tackled the following challenges in a comprehensive manner 35 . These challenges include (i) developing a fully automated end-to-end process, (ii) achieving accuracy that is comparable to that of an experienced radiologist, (iii) showing generalizability across multinational populations, (iv) providing interpretability to clinicians and patients, and (v) ensuring reproducibility by sharing the DL system and manual annotations.…”