Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning

Abstract

We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To this end, we introduce a general-purpose deep learning architecture that takes as input the entire dataset instead of processing one datapoint at a time. Our approach uses self-attention to reason about relationships between datapoints explicitly, which can be seen as realizing non-parametric models using parametric attention mechanisms. However, unlike conventional non-parametric models, we let the model learn end-to-end from the data how to make use of other datapoints for prediction. Empirically, our models solve cross-datapoint lookup and complex reasoning tasks unsolvable by traditional deep learning models. We show highly competitive results on tabular data, early results on CIFAR-10, and give insight into how the model makes use of the interactions between points.

Why do tree-based models still outperform deep learning on tabular data?

“…Deep learning for tabular data As described by Borisov et al [2021] in their review of the field, there have been various attempts to make deep learning work on tabular data: data encoding techniques to make tabular data better suited for deep learning [Hancock andKhoshgoftaar, 2020, Yoon et al, 2020], "hybrid methods" to benefit from the flexibility of NNs while keeping the inductive biases of other algorithms like tree-based models [Lay et al, 2018, Popov et al, 2020, Abutbul et al, 2020, Hehn et al, 2019, Tanno et al, 2019, Chen, 2020, Kontschieder et al, 2015, Rodriguez et al, 2019, Popov et al, 2020, Lay et al, 2018 or Factorization Machines Guo et al [2017], tabularspecific transformers architectures Somepalli et al [2021], Kossen et al [2021], Arik and Pfister [2019], Huang et al [2020], and various regularization techniques to adapt classical architectures to tabular data [Lounici et al, 2021, Shavitt and Segal, 2018, Kadra et al, 2021a, Fiedler, 2021. In this paper, we focus on architectures directly inspired by classic deep learning models, in particular Transformers and Multi-Layer-Perceptrons (MLPs).…”

“…Deep learning has enabled tremendous progress for learning on image, language, or even audio datasets. On tabular data, however, the picture is muddier and ensemble models based on decision trees like XGBoost remain the go-to tool for most practitioners [Sta] and data science competitions [Kossen et al, 2021]. Indeed deep learning architectures have been crafted to create inductive biases matching invariances and spatial dependencies of the data.…”

ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction

“…We also augment these baselines with zero-shot predictions obtained with the same model used to extract the protein sequence embeddings. Lastly, we include ProteinNPT [Notin et al, 2023], a semi-supervised pseudo-generative architecture which jointly models sequences and labels by performing axial attention [Ho et al, 2019b, Kossen et al, 2022…”

“…ProteinNPT [Notin et al, 2023] is a semi-supervised non-parametric transformer [Kossen et al, 2022] which learns a joint representation of full batches of labeled sequences. It is trained with a hybrid objective consisting of fitness prediction and masked amino acids reconstruction.…”

ProteinNPT: Improving Protein Property Prediction and Design with Non-Parametric Transformers

“…While this works reasonably well in practice, critical information may be lost in the pooling operation and, since not all residues may be relevant to a given task, we may want to be selective about which ones to consider. In this work, we introduce ProteinNPT ( § 3), a non-parametric transformer [Kossen et al, 2022] variant which is ideally suited to label-scarce settings through an additional regularizing denoising objective, straightforwardly extends to multi-task optimization settings and addresses all aforementioned issues. In order to quantify the ability of different models to extrapolate to unseen sequence positions, we devise several cross-validation schemes ( § 4.1) which we apply to all Deep Mutational Scanning (DMS) assays in the ProteinGym benchmarks [Notin et al, 2022a].…”