Probability Machines

Abstract

Summary Background Most machine learning approaches only provide a classification for binary responses. However, probabilities are required for risk estimation using individual patient characteristics. It has been shown recently that every statistical learning machine known to be consistent for a nonparametric regression problem is a probability machine that is provably consistent for this estimation problem. Objectives The aim of this paper is to show how random forests and nearest neighbors can be used for consistent estimation of individual probabilities. Methods Two random forest algorithms and two nearest neighbor algorithms are described in detail for estimation of individual probabilities. We discuss the consistency of random forests, nearest neighbors and other learning machines in detail. We conduct a simulation study to illustrate the validity of the methods. We exemplify the algorithms by analyzing two well-known data sets on the diagnosis of appendicitis and the diagnosis of diabetes in Pima Indians. Results Simulations demonstrate the validity of the method. With the real data application, we show the accuracy and practicality of this approach. We provide sample code from R packages in which the probability estimation is already available. This means that all calculations can be performed using existing software. Conclusions Random forest algorithms as well as nearest neighbor approaches are valid machine learning methods for estimating individual probabilities for binary responses. Freely available implementations are available in R and may be used for applications.

An empirical comparison of classification algorithms for mortgage default prediction: evidence from a distressed mortgage market

“…One intuitive method to do so is through a calibration plot. These plots have been used in bioinformatics and 430 in credit risk (Malley et al, 2012;Medema et al, 2009). They plot the class probability produced by the model (x-axis) against a non-parametric regression of the empirical proportion of defaulters with the same predicted probability (y-axis).…”

“…such as bioinformatics, image recognition, as well as in financial applications such as customer 250 attrition and credit scoring Lessmann et al, 2015;Malley et al, 2012).…”

ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R

“…Node impurity is measured with the Gini index for classification trees and with the estimated response variance for regression trees. For probability estimation, trees are grown as regression trees; for a description of the concept, see Malley, Kruppa, Dasgupta, Malley, and Ziegler (2012). Variable importance can be determined with the decrease of node impurity or with permutation.…”

Prediction of Chemotherapy Response in Primary Osteosarcoma by Use of the Multifractal Analysis of Magnetic Resonance Images

“…Images were classified from their pools in poorand good-response folders. The single-tree model with the Gini splitting algorithm was used to calculate a stratification accuracy of each parameter (DTREG predictive modelling software version 10.3.0, Brentwood, TN (24). The tree size control settings were set to minimum rows in a node: 1; minimum size node to split: 10; maximum tree levels: 10.…”