Insights into Pre-training via Simpler Synthetic Tasks

Abstract

Pre-training produces representations that are effective for a wide range of downstream tasks, but it is still unclear what properties of pre-training are necessary for effective gains. Notably, recent work shows that even pre-training on synthetic tasks can achieve significant gains in downstream tasks. In this work, we perform three experiments that iteratively simplify pre-training and show that the simplifications still retain much of its gains. First, building on prior work, we perform a systematic evaluation of three existing synthetic pre-training methods on six downstream tasks. We find the best synthetic pre-training method, LIME, attains an average of 67% of the benefits of natural pre-training. Second, to our surprise, we find that pre-training on a simple and generic synthetic task defined by the Set function achieves 65% of the benefits, almost matching LIME. Third, we find that 39% of the benefits can be attained by using merely the parameter statistics of synthetic pre-training. We release the source code at https://github.com/felixzli/synthetic_pretraining.

Synthetic data enable experiments in atomistic machine learning

“…47 We also note that Huang et al have used atomic energies to train NN-based atomistic models. 69 In a wider perspective, the pre-training of NN models is a well-documented approach in the ML literature for various applications and domains, [70][71][72][73][74] and it has very recently been described in the context of interatomic potential models, 47,75,76 property prediction with synthetic pre-training data, 77 and as a means to learn generalpurpose representations for atomistic structure. 76…”