Improving the generative performance of chemical autoencoders through transfer learning

Abstract

Generative models are a sub-class of machine learning models that are capable of generating new samples with a target set of properties. In chemical and materials applications, these new samples might be drug targets, novel semiconductors, or catalysts constrained to exhibit an application-specific set of properties. Given their potential to yield high-value targets from otherwise intractable design spaces, generative models are currently under intense study with respect to how predictions can be improved through changes in model architecture and data representation. Here we explore the potential of multi-task transfer learning as a complementary approach to improving the validity and property specificity of molecules generated by such models. We have compared baseline generative models trained on a single property prediction task against models trained on additional ancillary prediction tasks and observe a generic positive impact on the validity and specificity of the multi-task models. In particular, we observe that the validity of generated structures is strongly affected by whether or not the models have chemical property data, as opposed to only syntactic structural data, supplied during learning. We demonstrate this effect in both interpolative and extrapolative scenarios (i.e., where the generative targets are poorly represented in training data) for models trained to generate high energy structures and models trained to generated structures with targeted bandgaps within certain ranges. In both instances, the inclusion of additional chemical property data improves the ability of models to generate valid, unique structures with increased property specificity. This approach requires only minor alterations to existing generative models, in many cases leveraging prediction frameworks already native to these models. Additionally, the transfer learning strategy is complementary to ongoing efforts to improve model architectures and data representation and can foreseeably be stacked on top of these developments.

Actively Searching: Inverse Design of Novel Molecules with Simultaneously Optimized Properties

“…In previous work, we have demonstrated that the use of chemical autoencoders for targeted structure searching is effective for properties within the GDB19 dataset 39 , namely internal energy, zero-point vibrational energy, and HOMO-LUMO gap. 40 To examine the generality of this approach, we investigated three models trained to individually predict VIP, EA, and DM, and sampled 100,000 structures in property ranges poorly represented in the training data. The targeted ranges for VIP, EA, and DM were 10.0 to 11.0 eV, -2.0 to -1.0 eV, and 0.0 to 1.0 Debye, respectively (Fig.…”

Locally Optimizable Joint Embedding Framework to Design Nitrogen‐rich Molecules that are Similar but Improved

“…It has been shown capable of learning underlying relationships from a diverse set of molecular data by letting multiple data tasks and domains interact adaptively while generating the joint embeddings. Joint training is also a type of joint embedding that has been successfully applied to improve deep learning‐based molecule generation and enable transfer learning [25–29] . Joint training incorporates a property prediction task into a variational autoencoder (VAE) [30] and has been shown to organize points in the VAE latent space, making the latent space amenable to inverse molecular design and optimization [25,29] .…”

“…jointly trained a VAE using drug likeliness and synthetic accessibility, then performed Bayesian optimization in the resulting latent space to identify novel drug‐like molecules [25] . In addition to latent space organization, joint training provides a platform for knowledge transfer between abundant and scarce data tasks [27,29] . When more than one property is used to develop the model, this constitutes a multitask transfer learning approach.…”

“…It therefore has been reasoned that joint embedding yields more expressive models [31] in part because of the likely organization they induce in the latent representation [25,28] . This, in turn, offers improved properties of molecules obtained from such generative models [29] …”

“…However previous works on joint embedding have largely focused on properties (internal energy, HOMO‐LUMO gap, specific heat, etc …) within a single class of materials where knowledge transfer occurs from one group of molecules to another of the same class but for which certain properties are unavailable [25,29] . In contrast, our aim is for multi‐class transfer where one set of molecules simply does not possess certain properties.…”