MLcps: Machine Learning Cumulative Performance Score for classification problems

Abstract

A performance metric is a tool to measure the correctness of a trained Machine Learning (ML) model. Numerous performance metrics have been developed for classification problems making it overwhelming to select the appropriate one since each of them represents a particular aspect of the model. Furthermore, selection of a performance metric becomes harder for problems with imbalanced and/or small datasets. Therefore, in clinical studies where datasets are frequently imbalanced and, in situations when the prevalence of a disease is low or the collection of patient samples is difficult, deciding on a suitable metric for performance evaluation of an ML model becomes quite challenging. The most common approach to address this problem is measuring multiple metrics and compare them to identify the best-performing ML model. However, comparison of multiple metrics is laborious and prone to user preference bias. Furthermore, evaluation metrics are also required by ML model optimization techniques such as hyperparameter tuning, where we train many models, each with different parameters, and compare their performances to identify the best-performing parameters. In such situations, it becomes almost impossible to assess different models by comparing multiple metrics. Here, we propose a new metric called Machine Learning Cumulative Performance Score (MLcps) as a Python package for classification problems. MLcps combines multiple pre-computed performance metrics into one metric that conserves the essence of all pre-computed metrics for a particular model. We tested MLcps on 4 different publicly available biological datasets and the results reveal that it provides a comprehensive picture of overall model robustness. MLcps is available at https://pypi.org/project/MLcps/ and cases of use are available at https://mybinder.org/v2/gh/FunctionalUrology/MLcps.git/main.

SpheroScan: A User-Friendly Deep Learning Tool for Spheroid Image Analysis

“…All supporting data, which includes images used for training, validation, and testing [22], as well as the trained model weights [23], is available at zenodo.…”

MLcps: machine learning cumulative performance score for classification problems

“…An archival copy of the code and supporting data is available via the GigaScience repository, GigaDB [ 35 ]. DOME-ML (Data, Optimisation, Model, and Evaluation in Machine Learning) annotations, supporting the current study, are available via the supporting data in GigaDB.…”

STREAMLINE: A Simple, Transparent, End-To-End Automated Machine Learning Pipeline Facilitating Data Analysis and Algorithm Comparison

“…TPOT (26,27,28), FEDOT (19), Auto-Sklearn (22), GAMA (29), RECIPE (30), and ML-Plan (31)); (4) accessibility and ease of use; some AutoMLs are designed to be broadly accessible requiring little to no coding experience to implement and run (e.g. ALIRO (24), MLme (32), MLIJAR-supervised (33), H20-3 (34), STREAMLINE (35), and Auto-WEKA (36)), while others are designed primarily as a code library to facilitate building a customizable pipeline with automated elements (e.g. LAMA (37), FLAML (38), Hyperopt-sklearn (39), TransmorgrifAI (40), MLBox (41), Xcessiv (42)); (5) output focus; the aims of AutoML vary, with focus on either a single best optimized model/pipeline (e.g.…”

“…TPOT (28)), or a direct comparison of model performance across algorithms (e.g. STREAMLINE (35), MLme (32), and PYCARET (23)); (6) inclusion and automation of different possible elements of a complete end-to-end ML pipeline; with algorithm selection and hyperparameter optimization being most common; and (7) transparency in the documentation, i.e. to what degree are the available elements, options, and automations defined and validated.…”

“…ExSTraCS (51), a rule-based algorithm designed specifically to address the challenges of detecting and characterizing epistasis (52) and heterogeneous associations (53) in biomedical data), (4) conduct statistical significance comparisons (between algorithms and datasets), (5) collectively compare and contrast feature importance (FI) estimates across modelling algorithms, and (6) generate a comprehensive sharable summary report. With respect to overall AutoML design and goals, STREAMLINE currently is most closely related to MLIJAR-supervised (33) and MLme (32) AutoML tools.…”