SDTR: Soft Decision Tree Regressor for Tabular Data

Abstract

Deep neural networks have been proved a success in multiple fields. However, researchers still favor traditional approaches to obtain more interpretable models, such as Bayesian methods and decision trees when processing heterogeneous tabular data. Such models are hard to differentiate, thus inconvenient to be integrated into end-to-end settings. On the other hand, traditional neural networks are differentiable but perform poorly on tabular data. We propose a hierarchical differentiable neural regression model, Soft Decision Tree Regressor (SDTR). SDTR imitates a binary decision tree by a differentiable neural network and is plausible for ensemble schemes like bagging and boosting. The SDTR method was evaluated on multiple tabular-based regression tasks (YearPredictionMSD, MSLR-Web10K, Yahoo LETOR, SARCOS and Wine quality). Its performance is comparable with non-differentiable models (gradient boosting decision trees) and better than uninterpretable models (regular FCNN). On top of that, it can produce fair results with a restricted number of parameters, only using a small forest or even a single tree. We also propose an "average entropy" metric to evaluate the level of interpretability of a trained, soft decision tree neural network. This metric also helps to select proper structure and hyperparameters for such networks.

TMSCNet: A three-stage multi-branch self-correcting trait estimation network for RGB and depth images of lettuce

“…Since the input of the auxiliary models in the second and third stages of the TMSCNet was the numerical data rather than the image, the classical machine learning regression algorithm and fully connected network (FCN) were considered. There were many classical regression models in machine learning, such as Linear Regression ( Austin and Steyerberg, 2015 ), Support Vector Regressor ( Cortes and Vapnik, 1995 ), K-Nearest Neighbors Regressor ( Song et al, 2017 ), Decision Tree Regressor ( Luo et al, 2021 ), Random Forest Regressor ( Ding and Bar-Joseph, 2017 ), AdaBoost Regressor ( Chen et al, 2019 ), and Bagging Regressor ( Dal Molin Ribeiro and Coelho, 2020 ). For selecting more suitable self-correcting models, we first conducted comparative experiments on the classical machine learning regression algorithm to select the best performing method, and then implement comparative experiments on the optimized classical machine learning algorithm and fully connected network to determine the method of the auxiliary model in the second and third stages of the TMSCNet.…”

Design and Implementation of Sales Prediction Model Using Decision Tree Regressor over Linear Regression Towards Increase in Accuracy of Prediction

“…This paper by [21] (Ali et al 2021) uses Decision tree regressor with neural networks for performance optimization and has an accuracy of 90%. (Luo et al 2021)) [22] proposed a Soft Decision Tree Regressor (SDTR),a differentiable hierarchical neural regression model SDTR is a differentiable neural network that mimics a binary decision tree and is suitable for ensemble techniques such as bagging and boosting, as well as archiving the results of 95.34%.…”

Multi‐unit building address geocoding: An approach without indoor location reference data

“…CatBoost's implementation of gradient boosted decision trees has a proven track record in the context of several different and similarly sized problems (Bentéjac et al, 2021; Luo et al, 2021). Since the proposed solution is novel the explainability of the underlying decision trees can help in investigating performance and identifying what variables play the most significant role in location predictions.…”

“…This allows for more complex models to potentially learn policy and construction year patterns. Recent work by Luo et al (2021) addresses the favorability of traditional machine learning approaches over complex deep learning methods in the context of supervised regression with heterogeneous tabular input data. Furthermore, recent work conducting comprehensive comparisons of state‐of‐the‐art interpretable and uninterpretable models on several datasets reveals that CatBoost's implementation of an ensemble algorithm for gradient boosting on decision trees outperforms all other models on average (Bentéjac et al, 2021).…”