Introducing ROC Curves as Error Measure Functions: A New Approach to Train ANN-Based Biomedical Data Classifiers

Abstract

This paper explores the usage of the area (Az) under the Receiver Operating Characteristic (ROC) curve as error measure to guide the training process to build machine learning ANN-based classifiers for biomedical data analysis. Error measures (like root mean square error, RMS) are used to guide training algorithms measuring how far solutions are from the ideal classification, whereas it is well known that optimal classification rates do not necessarily yield to optimal Az's. Our hypothesis is that Az error measures can guide existing training algorithms to obtain better Az's than other error measures. This was tested after training 280 different configurations of ANN-based classifiers, with simulated annealing, using five biomedical binary datasets from the UCI machine learning repository with different test/train data splits. Each ANN configuration was trained both using the Az and RMS based error measures. In average Az was improved in 7.98% in testing data (9.32% for training data) when using 70% of the datasets elements for training. Further analysis reveals interesting patterns (Az improvement is greater when Az are lower). These results encourage us to further explore the usage of Az based error measures in training methods for classifiers in a more generalized manner.

A Software Framework for Building Biomedical Machine Learning Classifiers through Grid Computing Resources

“…instead of sorting out the logistics of access methods to Grid infrastructures, preparing datasets differently for each third party engine, etc. In fact, this method has enabled the authors to validate the modifications made to the Encog engines for ROC optimization with satisfactory results [23].…”

“…Currently, BiomedTK integrates engines from the Encog toolkit [24] (for ANNs), the libsvm toolkit [25] (for SVMs) and the ANN Encog engines modified by the authors for ROC optimization [23]. Table 1 lists the engines currently integrated in BiomedTK along with the parameters each one accepts.…”

“…For any engine, BiomedTK allows the evaluation of binary classifiers through plotting ROC curves and computing their area under the curve (ROC Az) [26,27], offering the bi-normal distribution method as provided by JLABROC4 [28] and the Mann-Whitney statistic provided by WEKA [29]. It also includes the Encog ANN engines modified by the authors [23], allowing optimization of the ROC Az complementing the classical optimization tasks aimed at reducing some Root Mean Squared Error measure to increase accuracy (percentage of dataset elements correctly classified). Accuracy and ROC Az measures are related, but it is known that optimization of accuracy does not necessarily yield optimization of ROC Az [30], fact that is specially relevant for biomedical data analysis.…”

“…BiomedTK was developed to harness Grid computing power for biomedical data analysis. It also describes the experimental set up through which the method was validated with different biomedical datasets of the UCI machine learning repository [22,23]. Section 3 shows the results obtained by the experiments carried out, which are finally discussed in Section 4.…”

Discovering Mammography-based Machine Learning Classifiers for Breast Cancer Diagnosis

“…Feedforward ANNs as provided by Encog can be trained with backpropagation (denoted by ffbp), simulated annealing (ffsa) and genetic algorithms (ffga), see [32] for details. In addition, the authors extended Encog to support ROC optimization based backpropagation (ffbproc) and simulated annealing (ffsaroc) [36]. From libsvm we use cost based optimization SVMs (denoted by csvm).…”