DBC-Forest: Deep forest with binning confidence screening

Ma, Pengfei; Wu, Youxi; Li, Yan; Guo, Lei; Zhao, Li

doi:10.1016/j.neucom.2021.12.075

Cited by 15 publications

(6 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many classification and regression problems presume to analyze discrete processes that could be done better by discrete models. We believe future development of randomized tree ensembles should be focused on utilizing them as a part of more sophisticated models that can catch complex relationships between features and data samples, i.e., [44], [45].…”

Section: Discussionmentioning

confidence: 99%

Random Kernel Forests

Devyatkin

Grigoriev

2022

IEEE Access

View full text Add to dashboard Cite

Random forests of axis-parallel decision trees still show competitive accuracy in various tasks; however, they have drawbacks that limit their applicability. Namely, they perform poorly for multidimensional sparse data that is frequently input in many text and image processing tasks. A straightforward solution is to create forests of decision trees with oblique splits; however, most training approaches have low performance or are hard to tune. Besides, those ensembles appear unstable and easily overfit, so they should be regularized to find the trade-off between complexity and generalization. This paper proposes an algorithm to train kernel decision trees and random forests. At the top level, this algorithm follows a common greedy procedure to train binary decision trees; however, it produces quasi-optimal linear and kernel splits at the decision stump level. At each stump, the algorithm finds a quasi-optimal distribution of classes to subtrees and trains this stump via optimization of an SVM-like loss function with a margin re-scaling approach, which helps simultaneously optimize the margin between subtree data and arbitrary impurity criteria. We also try to reveal uniform stability-based generalization bounds for those forests and apply them to select the regularization technique. The obtained bounds explicitly consider primary hyperparameters of forests, trees, and decision stumps. The bounds also show a relationship between outliers in decision tree structure, stability and generalization, and how forests smooth these outliers. We performed an experimental evaluation on several tasks, such as studying the reaction of social media users, image recognition, and bank scoring. The experimental results show that the proposed algorithm trains ensembles, which outperform other oblique or kernel forests in many commonly-recognized datasets. Namely, the proposed algorithm shows 99.1% accuracy on MNIST and 58.1% on CIFAR-10. It has been confirmed that the selected regularization technique helps reduce overfitting on several datasets. Therefore, the proposed algorithm may be considered a small step toward customized and regularized ensembles of oblique or kernel trees that keep reasonable training performance on large datasets. We believe that the proposed algorithm can be utilized as a construction element of approaches to training context-aware models on complex problems with integer inputs and outputs, such as text mining and image recognition.INDEX TERMS Generalization bounds, kernel splits, kernel forests, random forests, random uniform stability, regularization, slack re-scaling.

show abstract

Section: Discussionmentioning

confidence: 99%

Random Kernel Forests

Devyatkin

Grigoriev

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Baseline methods. To validate the performance of HW-Forest, we selected four state-of-the-art algorithms as competitive models: gcForest [9], gcForestcs [12], DBC-Forest [13], and AWDF [25]. To verify the effect of our hashing screening mechanism, we introduced this mechanism into gcForest, gcForestcs, DBC-Forest, and AWDF, to give H-gcForest, H-gcForestcs, HDBC-Forest, and H-AWDF, respectively.…”

Section: Methodsmentioning

confidence: 99%

“…The most representative of the improved deep forest models is gcForestcs [12], in which confidence screening was adopted to improve the efficiency. Inspired by gc-Forestcs, many other approaches have been proposed, such as DBC-Forest [13], AWDF [25], and, MLDF [26]. In the binning confidence screening deep forest algorithm [13], a binning confidence mechanism was adopted to improve the accuracy of gcForestcs.…”

Section: Related Workmentioning

confidence: 99%

“…Inspired by gc-Forestcs, many other approaches have been proposed, such as DBC-Forest [13], AWDF [25], and, MLDF [26]. In the binning confidence screening deep forest algorithm [13], a binning confidence mechanism was adopted to improve the accuracy of gcForestcs. In an approach called adaptive weighted deep forest [25], a special weighted scheme was utilized that could adaptively weight each training instance at each cascade level.…”

Section: Related Workmentioning

confidence: 99%

“…However, this rule cannot avoid the mispartitioning of instances, in which some low-accuracy instances are classified into the high-confidence set. To tackle this issue, we previously proposed DBC-Forest [13], in which binning confidence screening is used to replace confidence screening. More concretely, DBC-Forest packs all instances into bins based on their confidence values, and adopts the instance accuracy of the bins to avoid mispartitioning of instances.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Deep Forest with Hashing Screening and Window Screening

Ma¹,

Wu²,

Li³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

As a novel deep learning model, gcForest has been widely used in various applications. However, the current multi-grained scanning of gcForest produces many redundant feature vectors, and this increases the time cost of the model. To screen out redundant feature vectors, we introduce a hashing screening mechanism for multi-grained scanning and propose a model called HW-Forest which adopts two strategies, hashing screening and window screening. HW-Forest employs perceptual hashing algorithm to calculate the similarity between feature vectors in hashing screening strategy, which is used to remove the redundant feature vectors produced by multi-grained scanning and can significantly decrease the time cost and memory consumption. Furthermore, we adopt a selfadaptive instance screening strategy to improve the performance of our approach, called window screening, which can achieve higher accuracy without hyperparameter tuning on different datasets. Our experimental results show that HW-Forest has higher accuracy than other models, and the time cost is also reduced.

show abstract

WCDForest: a weighted cascade deep forest model toward the classification tasks

Huang,

Chen,

et al. 2023

Appl Intell

View full text Add to dashboard Cite

DBC-Forest: Deep forest with binning confidence screening

Cited by 15 publications

References 26 publications

Random Kernel Forests

Random Kernel Forests

Deep Forest with Hashing Screening and Window Screening

WCDForest: a weighted cascade deep forest model toward the classification tasks

Contact Info

Product

Resources

About