Sample-Size Determination Methodologies for Machine Learning in Medical Imaging Research: A Systematic Review

Abstract

Purpose The required training sample size for a particular machine learning (ML) model applied to medical imaging data is often unknown. The purpose of this study was to provide a descriptive review of current sample-size determination methodologies in ML applied to medical imaging and to propose recommendations for future work in the field. Methods We conducted a systematic literature search of articles using Medline and Embase with keywords including “machine learning,” “image,” and “sample size.” The search included articles published between 1946 and 2018. Data regarding the ML task, sample size, and train-test pipeline were collected. Results A total of 167 articles were identified, of which 22 were included for qualitative analysis. There were only 4 studies that discussed sample-size determination methodologies, and 18 that tested the effect of sample size on model performance as part of an exploratory analysis. The observed methods could be categorized as pre hoc model-based approaches, which relied on features of the algorithm, or post hoc curve-fitting approaches requiring empirical testing to model and extrapolate algorithm performance as a function of sample size. Between studies, we observed great variability in performance testing procedures used for curve-fitting, model assessment methods, and reporting of confidence in sample sizes. Conclusions Our study highlights the scarcity of research in training set size determination methodologies applied to ML in medical imaging, emphasizes the need to standardize current reporting practices, and guides future work in development and streamlining of pre hoc and post hoc sample size approaches.

Detection and classification of unilateral cleft alveolus with and without cleft palate on panoramic radiographs using a deep learning system

“…In this regard, it might be effective to test the model created in the present study by using the panoramic radiographs of patients with bilateral CAs. Second, the numbers of training and test data were so small that the results cannot be generalized although it was difficult to estimate an appropriate sample size 28 . Future research should be planned with larger datasets obtained from multiple hospitals through the use of different panoramic machines.…”

A Review on Traditional Machine Learning and Deep Learning Models for WBCs Classification in Blood Smear Images

“…A planned searching procedure is required to find the available literature that fulfills the searching criteria, to utilize the available digital resources purposefully [19]. In the proposed study, we incorporated both manual and automatic searches to get the most relevant research articles by following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) model [20,21].…”

“…However, selecting an optimal feature extractor is challenging due to varying feature dynamics, such as geometric invariance and photometric invariance. Nowadays, the vast emergence of DL approaches has resulted in highperformance MIA models, especially in clinical hematology using blood smear images [18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33].…”

Correcting data imbalance for semi-supervised COVID-19 detection using X-ray chest images

“…Deep learning based approaches have been proposed to tackle medical imaging problems [2] , [3] , [4] , [5] , [6] . However, deep learning models typically need large labelled datasets [7] , [8] .…”