Reprogramming Language Models for Molecular Representation Learning

Abstract

Recent advancements in transfer learning have made it a promising approach for domain adaptation via transfer of learned representations. This is especially when relevant when alternate tasks have limited samples of well-defined and labeled data, which is common in the molecule data domain. This makes transfer learning an ideal approach to solve molecular learning tasks. While Adversarial Reprogramming has proven to be a successful method to repurpose neural networks for alternate tasks, most works consider source and alternate tasks within the same domain. In this work, we propose a new algorithm, Representation Reprogramming via Dictionary Learning (R2DL), for adversarially reprogramming pretrained language models for molecular learning tasks, motivated by leveraging learned representations in massive state of the art language models. The adversarial program learns a linear transformation between a dense source model input space (language data) and a sparse target model input space (e.g., chemical and biological molecule data) using a k-SVD solver to approximate a sparse representation of the encoded data, via dictionary learning. This method achieves the baseline established by state of the art toxicity prediction models trained on domain-specific data using only a standard well-trained text classifier, thereby establishing avenues for domain-agnostic transfer learning for tasks with molecule data.

LaViP: Language-Grounded Visual Prompting

“…We provide some notable examples below. Vinod, Chen, and Das (2020) repurposed a language model to predict biochemical sequences; Tsai, Chen, and Ho (2020) 2022) reprogrammed a vision model to classify text sentences and DNA sequences. (Oh et al 2023) extended BAR by generating input-aware visual prompts through an external encoder-decoder model for limited data recognition.…”

“…The pioneering work of (Tsai, Chen, and Ho 2020) has demonstrated that through MR, even a CNN initially trained on ImageNet can be swiftly adapted to excel in classifying medical images, interestingly, even outperforming the traditional finetuning approach. Subsequent research efforts have extended the idea of MR into various domains, achieving successful adaption without finetuing (Vinod, Chen, and Das 2020;Yen et al 2021;Yang, Tsai, and Chen 2021;Neekhara et al 2022;Chen et al 2023).…”