Random Forests for Big Data

Abstract

International audienceBig Data is one of the major challenges of statistical science and has numerous consequences from algorithmic and theoretical viewpoints. Big Data always involve massive data but they also often include online data and data heterogeneity. Recently some statistical methods have been adapted to process Big Data, like linear regression models, clustering methods and bootstrapping schemes. Based on decision trees combined with aggregation and bootstrap ideas, random forests were introduced by Breiman in 2001. They are a powerful nonparametric statistical method allowing to consider in a single and versatile framework regression problems, as well as two-class and multi-class classification problems. Focusing on classification problems, this paper proposes a selective review of available proposals that deal with scaling random forests to Big Data problems. These proposals rely on parallel environments or on online adaptations of random forests. We also describe how related quantities -- such as out-of-bag error and variable importance -- are addressed in these methods. Then, we formulate various remarks for random forests in the Big Data context. Finally, we experiment five variants on two massive datasets (15 and 120 millions of observations), a simulated one as well as real world data. One variant relies on subsampling while three others are related to parallel implementations of random forests and involve either various adaptations of bootstrap to Big Data or to ``divide-and-conquer'' approaches. The fifth variant relates on online learning of random forests. These numerical experiments lead to highlight the relative performance of the different variants, as well as some of their limitations

Sparsity in optimal randomized classification trees

“…Addressing global sparsity is a challenge in decision trees and, to the best of our knowledge, this has not been tackled appropriately in the literature. Standard CARTs or Random Forests (RFs) [5,7,13,16] cannot manage it due to the greedy construction of the trees. Nonetheless, some attempts have been made, see [11,12].…”

“…The S-ORCT smooth formulation (9)- (16) has been implemented using Pyomo optimization modeling language [19,20] in Python 3.5 [31]. As solver, we have used IPOPT 3.11.1 [39], and have followed a multistart approach, where the process is repeated 20 times starting from different random initial solutions.…”

Big Data Analysis for M2M Networks:Research Challenges and Open Research Issues

“…Addressing the leading challenges of statistical science, it serves to broaden not only the algorithmic but also the theoretical perspective [29,30,40]. It always involves massive data, including data streams and data heterogeneity.…”

“…In addition, since its major key features are multiple sources, huge volume, and fast-changing nature, it is difficult for commonly used traditional computing methods such as machine learning, information retrieval and data mining to efficiently support the processing, analysis and computation of Big Data [40]. Therefore, in recent years, statistical methods such as clustering methods, linear regression models and bootstrapping schemes have been adapted to process Big Data [29]. Generally, Big Data can be classified based on the five main aspects of data and its content: source, store, format, staging, and processing [11,39].…”

A Survey on Incremental Learning

“…Random forest classification algorithm [10] is proposed by Leo Breiman and Adele Cutler et al in the 1980s. Its main idea is integration thought, which is based on the development of decision trees, and includes multiple decision trees.…”