Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain

Darabi, Sajad; Fazeli, Shayan; Pazoki, Ali; Sankararaman, Sriram; Sarrafzadeh, Majid

doi:10.48550/arxiv.2108.12296

Cited by 2 publications

(3 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Self-supervised VPCL. Most SSL tabular methods work on the whole fixed set of columns [2,24,11], which take high computational costs and are prone to overfitting. Instead, we take tabular vertical partitions to build positive and negative samples for CL under the hypothesis that the powerful representation should model view-invariant factors.…”

Section: Self-supervised and Supervised Pretraining Of Transtabmentioning

confidence: 99%

“…However, it was argued that boosting algorithms and MLPs are still the competitive choices for tabular data modeling, especially when the sample size is small [32,46,39,47]. To alleviate label scarcity issue, SSL pretraining on unlabeled tabular data was introduced [2,24,10,9,11]. Nonetheless, none of them is transferable across tables then is able to extend the success of pretraining to the tabular domain.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

TransTab: Learning Transferable Tabular Transformers Across Tables

Wang¹,

Sun²

2022

Preprint

View full text Add to dashboard Cite

Tabular data (or tables) are the most widely used data format in machine learning (ML). However, ML models often assume the table structure keeps fixed in training and testing. Before ML modeling, heavy data cleaning is required to merge disparate tables with different columns. This preprocessing often incurs significant data waste (e.g., removing unmatched columns and samples). How to learn ML models from multiple tables with partially overlapping columns? How to incrementally update ML models as more columns become available over time? Can we leverage model pretraining on multiple distinct tables? How to train an ML model which can predict on an unseen table ? To answer all those questions, we propose to relax fixed table structures by introducing a Transferable Tabular Transformer (TransTab) for tables. The goal of TransTab is to convert each sample (a row in the table) to a generalizable embedding vector, and then apply stacked transformers for feature encoding. One methodology insight is combining column description and table cells as the raw input to a gated transformer model. The other insight is to introduce supervised and self-supervised pretraining to improve model performance. We compare TransTab with multiple baseline methods on diverse benchmark datasets and five oncology clinical trial datasets. Overall, TransTab ranks 1.00, 1.00, 1.78 out of 12 methods in supervised learning, feature incremental learning, and transfer learning scenarios, respectively; and the proposed pretraining leads to 2.3% AUC lift on average over the supervised learning. 1Preprint. Under review.

show abstract

Section: Self-supervised and Supervised Pretraining Of Transtabmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

TransTab: Learning Transferable Tabular Transformers Across Tables

Wang¹,

Sun²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Lack of clear feature relationships in tabular data, fully connected dense neural networks are typically used as a parametric method for training to consider the impact of all features on the target values in supervised setting [8,12]. Some methods have been proposed to enable deep feature learning in contrastive learning paradigm for tabular data, however, they all use dense layer network [2,5,14,21]. The main drawback of dense layer is that they learn global patterns using all features.…”

Section: Introductionmentioning

confidence: 99%

Local Contrastive Feature Learning for Tabular Data

Gharibshah

Zhu

2022

Proceedings of the 31st ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Contrastive self-supervised learning has been successfully used in many domains, such as images, texts, graphs, etc., to learn features without requiring label information. In this paper, we propose a new local contrastive feature learning (LoCL) framework, and our theme is to learn local patterns/features from tabular data. In order to create a niche for local learning, we use feature correlations to create a maximum-spanning tree, and break the tree into feature subsets, with strongly correlated features being assigned next to each other. Convolutional learning of the features is used to learn latent feature space, regulated by contrastive and reconstruction losses. Experiments on public tabular datasets show the effectiveness of the proposed method versus state-of-the-art baseline methods. CCS CONCEPTS• Information systems → Data mining.

show abstract

Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain

Cited by 2 publications

References 16 publications

TransTab: Learning Transferable Tabular Transformers Across Tables

TransTab: Learning Transferable Tabular Transformers Across Tables

Local Contrastive Feature Learning for Tabular Data

Contact Info

Product

Resources

About