STREAMLINE: A Simple, Transparent, End-To-End Automated Machine Learning Pipeline Facilitating Data Analysis and Algorithm Comparison

Abstract

Objective: While machine learning (ML) includes a valuable array of tools for analyzing biomedical data with multivariate and complex underlying associations, significant time and expertise is required to assemble effective, rigorous, comparable, reproducible, and unbiased pipelines. Automated ML (AutoML) tools seek to facilitate ML application by automating a subset of analysis pipeline elements. In this study we develop and validate a Simple, Transparent, End-to-end Automated Machine Learning Pipeline (STREAMLINE) and apply it to investigate the added utility of photography-based phenotypes for predicting obstructive sleep apnea (OSA); a common and underdiagnosed condition associated with a variety of health, economic, and safety consequences. Methods: STREAMLINE is designed to tackle biomedical binary classification tasks while (1) avoiding common mistakes, (2) accommodating complex associations and common data challenges, and (3) allowing scalability, reproducibility, and model interpretation. It automates the majority of established, generalizable, and reliably automatable aspects of an ML analysis pipeline while incorporating cutting edge algorithms and providing opportunities for human-in-the-loop customization. We present a broadly refactored and extended release of STREAMLINE, validating and benchmarking performance across simulated and real-world datasets. Then we applied STREAMLINE to evaluate the utility of demographics (DEM), self-reported comorbidities (DX), symptoms (SYM), and photography-based craniofacial (CF) and intraoral (IO) anatomy measures in predicting 'any OSA' or 'moderate/severe OSA' using 3,111 participants from Sleep Apnea Global Interdisciplinary Consortium (SAGIC). Results: Benchmarking analyses validated the efficacy of STREAMLINE across data simulations with increasingly complex patterns of association including epistatic interactions and genetic heterogeneity. OSA analyses identified a significant increase in ROC-AUC when adding CF to DEM+DX+SYM to predict 'moderate/severe' OSA. Additionally, a consistent but non-significant increase in PRC-AUC was observed with the addition of each subsequent feature set to predict 'any OSA', with CF and IO yielding minimal improvements. Conclusion: STREAMLINE is an effective, rigorous, transparent, and easy-to-use AutoML approach to a comparative ML analysis that adheres to best practices in data science. Application of STREAM-LINE to OSA data suggests that CF features provide additional value in predicting moderate/severe OSA, but neither CF nor IO features meaningfully improved the prediction of 'any OSA' beyond established demographics, comorbidity and symptom characteristics.Keywords automated machine learning • obstructive sleep apnea • data science • predictive modeling • craniofacial traits • intraoral anatomy user-specification of feature types (which cannot always be reliably automated) and one-hot-encoding of categorical features for modeling, (3) engineering of 'missingness features' to consider missingness as a potentially informati...

A Practical Approach for Targeting Structural Variants Genome-wide in Plasma Cell-free DNA

“…Aside from the Youden’s J statistic-based threshold model for our dichotomous diagnostic test, a variety of statistical, probabilistic, and optimization techniques can be applied to genomic data using machine learning (ML) for binary classification. An automated end-to-end ML pipeline called STREAMLINE 47 was used for the binary classification of the breast tumor and paired normal DNA (as in Fig. 3 ).…”

“…Machine learning binary classification modeling and evaluation was conducted using version 0.2.5 of the Simple Transparent End-To-End Automated Machine Learning Pipeline (STREAMLINE) 44 applying mostly default pipeline run parameters (exceptions identified below). STREAMLINE is a recently developed AutoML tool that facilitates ML modeling while enforcing rigorous modeling algorithm comparisons, analysis transparency, reproducibility, and sensitivity to complex associations in data.…”

“…Machine learning binary classification modeling and evaluation was conducted using version 0.2.5 of the Simple Transparent End-To-End Automated Machine Learning Pipeline (STREAMLINE), 47 applying mostly default pipeline run parameters (exceptions identified below). Input data were the number of top 1,000 HCBs on chromosomes 1-22 and X (23 features) unless otherwise stated.…”

Implications of serum liver enzymes for brain amyloidopathy and cognition

“…To investigate the classi cation performance of liver enzymes in differentiating amyloid status, we used a machine learning approach with penalized logistic regression using the STREAMLINE tool [30]. This approach proved advantageous in reducing the effect of multicollinearity on feature selection [31].…”

“…The ROC curves and mean AUC of machine learning approach using penalized logistic regression Sensitivity is on the y-axis and 1-speci city is on the x-axis. 30 Abbreviations: ALP, alkaline phosphatase; ALT, alanine aminotransferase; APOE, apolipoprotein E; AST, aspartate aminotransferase; AUC, area under the curve; ROC, receiver operating characteristic.…”