STAR-ML: A Rapid Screening Tool for Assessing Reporting of Machine Learning in Research

Khan, Md Asif; Koh, Ryan G. L.; Hassan, Samah; Liu, Theodore; Tucci, Victoria; Kumbhare, Dinesh; Doyle, Thomas E.

doi:10.1109/ccece49351.2022.9918312

Cited by 1 publication

(7 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 4 shows a comparison of the average time it took to screen a study in this work and in [53]. The mean and standard deviation of the average time was 4.733 ± 2.101 minutes and 4.701 ± 0.644 minutes for the former study [53] and the current pilot, respectively. The average time of each rater was 4.932 ± 1.934, 5.500 ± 1.487, 4.085 ± 0.905, and 4.286 ± 1.235 minutes in the current pilot.…”

Section: Resultsmentioning

confidence: 99%

“…These resulted in 2 versions of STAR-ML with the main changes between the iterations being improvement of the language of the questions and the inclusion of an additional question. These first versions were then piloted in 2 rounds to assess the functionality of the screening tool [53]. Three independent raters participated in that pilot process.…”

Section: A Development Of Star-mlmentioning

confidence: 99%

“…The remaining 178 studies were divided among four raters and screened using STAR-ML. As STAR-ML achieved high inter-rater reliability (0.93, lower bound: 0.83, upper bound: 0.97), which was computed based on a mean-rating (n = 3), absolute-agreement, 2-way mixed-effects model for the total scores between 3 raters in the previous study [53], a single rater was used to score the assigned set of articles. Prior to working on the articles in the scoping review, the raters underwent training on a set of 10 articles to align the level of understanding.…”

Section: ) Inclusion and Exclusion Criteriamentioning

confidence: 99%

“…Initially, experts agreed that during full text review, a study should be included if the STAR-ML score is more than 50%, which corresponded to a ≥ 5 out 9 [53]. A score of 6 or more was chosen for the pilot review to ensure that included studies were better than the minimal score determined initially by the experts.…”

Section: ) Inclusion and Exclusion Criteriamentioning

confidence: 99%

“…Given the increasing usage of ML, quality assessment of articles is essential to maintain high methodological quality and ensure rigorous standards have been followed [53], [54]. Articles that do not report or incorrectly report their algorithm can call into question the quality and transparency of the research.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Check It Before You Wreck It: A Guide to STAR-ML for Screening Machine Learning Reporting in Research

Koh,

Khan,

Rashidiani

et al. 2023

IEEE Access

Self Cite

View full text Add to dashboard Cite

Machine learning (ML) is a technique that learns to detect patterns and trends in data. However, the quality of reporting ML in research is often suboptimal, leading to inaccurate conclusions and hindering progress in the field, especially if disseminated in literature reviews that provide researchers with an overview of a field, current knowledge gaps, and future directions. While various tools are available to assess the quality and risk-of-bias of studies, there is currently no generalized tool for assessing the reporting quality of ML in the literature. To address this, this study presents a new screening tool called STAR-ML (Screening Tool for Assessing Reporting of Machine Learning), accompanied by a guide to using it. A pilot scoping review looking at ML in chronic pain was used to investigate the tool. The time it took to screen papers and how the selection of the threshold affected the papers included were explored. The tool provides researchers with a reliable and systematic way to evaluate the quality of reporting of ML studies and to make informed decisions about the inclusion of studies in scoping or systematic reviews. In addition, this study provides recommendations for authors on how to choose the threshold for inclusion and use the tool proficiently. Lastly, the STAR-ML tool can serve as a checklist for researchers seeking to develop or implement ML techniques effectively.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: A Development Of Star-mlmentioning

confidence: 99%

Section: ) Inclusion and Exclusion Criteriamentioning

confidence: 99%

Section: ) Inclusion and Exclusion Criteriamentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations