Revisiting Deep Learning Models for Tabular Data

Abstract

The necessity of deep learning for tabular data is still an unanswered question addressed by a large number of research efforts. The recent literature on tabular DL proposes several deep architectures reported to be superior to traditional "shallow" models like Gradient Boosted Decision Trees. However, since existing works often use different benchmarks and tuning protocols, it is unclear if the proposed models universally outperform GBDT. Moreover, the models are often not compared to each other, therefore, it is challenging to identify the best deep model for practitioners.In this work, we start from a thorough review of the main families of DL models recently developed for tabular data. We carefully tune and evaluate them on a wide range of datasets and reveal two significant findings. First, we show that the choice between GBDT and DL models highly depends on data and there is still no universally superior solution. Second, we demonstrate that a simple ResNet-like architecture is a surprisingly effective baseline, which outperforms most of the sophisticated models from the DL literature. Finally, we design a simple adaptation of the Transformer architecture for tabular data that becomes a new strong DL baseline and reduces the gap between GBDT and DL models on datasets where GBDT dominates. The source code is available at https: //github.com/yandex-research/rtdl.

Deep Neural Networks and Tabular Data: A Survey

“…4.2.1) and transformers-based (Sec. 4.2.2) groups exhibit superior predictive performance compared to plain deep neural networks on various data sets (Gorishniy et al, 2021;Ke et al, 2018Ke et al, , 2019Somepalli et al, 2021). This underlines the importance of special-purpose architectures for tabular data.…”

“…We also discuss the key categorical data encoding methods in Section 4.1.1. Gorishniy et al (2021) empirically evaluated a large number of state-of-the-art deep learning approaches for tabular data on a wide range of data sets. Interestingly, the authors demonstrated that a tuned deep neural network model with the ResNet-like architecture (He et al, 2016) shows comparable performance to some state-of-the-art deep learning approaches for tabular data.…”

Look behind the Censorship: Reposting-User Characterization and Muted-Topic Restoration

“…After searching reports about his activities during that period, we find that he was racing cars for Porsche, 8 and attending fashion shows as ambassador of Louis Vuitton. 9 Apart from "Wu Yifan", we also observe topics related to "live stream", which is also a new topic that has not been observed in previous analysis. ah ah ah (0.045), like (0.041), thanks (0.035), mom (0.019), new (0.015), teacher (0.014), cry (0.011), shoot (0.010), character (0.009), great (0.008), rush (0.008), baby (0.008), song (0.008), clarify (0.008), donate (0.008), live (0.007), hope (0.007), become (0.007), Beijing (0.007), dry (0.007) 9 cute (0.029), china (0.025), support (0.020), Zhang Zhehan (0.020), child (0.014), apologize (0.013), woman (0.012), nation (0.010), gong jun (0.009), stand (0.008), friend (0.008), engage in (0.008), write (0.008), society (0.007), law (0.007), girl (0.006), chance (0.006), sad (0.006), long (0.005), willing (0.005) 10 Wu Yifan (0.126), Mr (0.048), endorsement (0.028), spokesperson (0.026), road (0.023), expect (0.022), easy Vuitton (0.020), brand (0.019), force (0.016), worldwide (0.016), music (0.014), nice (0.011), cattle (0.009), silly (0.009), high (0.008), congratulations (0.008), Wu (0.007), Li Shubai (0.007), racer (0.006), wish (0.006)…”

“…To measure the influence, we construct a decision tree-based model and analyze the feature importance. In a recent study of Gorishniy et al [9], it is found that for heterogeneous data, the baseline performance of GBDT (Gradient Boost Decision Tree) is strictly superior to DNN (Deep Neural Networks). In addition, for decision tree-based models, they are intrinsically more interpretable than deep neural networks.…”

“…In addition, for decision tree-based models, they are intrinsically more interpretable than deep neural networks. For the tree model, we pick XGBoost [7] over other boosting models because it outperforms other tree models in the experiment on heterogeneous data by Gorishniy et al [9].…”