Active Learning to Overcome Sample Selection Bias: Application to Photometric Variable Star Classification

Abstract

Despite the great promise of machine-learning algorithms to classify and predict astrophysical parameters for the vast numbers of astrophysical sources and transients observed in large-scale surveys, the peculiarities of the training data often manifest as strongly biased predictions on the data of interest. Typically, training sets are derived from historical surveys of brighter, more nearby objects than those from more extensive, deeper surveys (testing data). This sample selection bias can cause catastrophic errors in predictions on the testing data because a) standard assumptions for machine-learned model selection procedures break down and b) dense regions of testing space might be completely devoid of training data. We explore possible remedies to sample selection bias, including importance weighting (IW), co-training (CT), and active learning (AL). We argue that AL-where the data whose inclusion in the training set would most improve predictions on the testing set are queried for manual follow-up-is an effective approach and is appropriate for many astronomical applications. For a variable star classification problem on a well-studied set of stars from Hipparcos and OGLE, AL is the optimal method in terms of error rate on the testing data, beating the off-the-shelf classifier by 3.4% and the other proposed methods by at least 3.0%. To aid with manual labeling of variable stars, we developed a web interface which allows for easy light curve visualization and querying of external databases. Finally, we apply active learning to classify variable stars in the ASAS survey, finding dramatic improvement in our agreement with the ACVS catalog, from 65.5% to 79.5%, and a significant increase in the classifier's average confidence for the testing set, from 14.6% to 42.9%, after a few AL iterations.

Comparative Analysis of a Transition Region Bright Point with a Blinker and Coronal Bright Point Using Multiple EIS Emission Lines

“…Recent development of modern machine learning techniques and high performance computing architectures have made possible the efficient execution of automated probabilistic multi-class classification of very large datasets in reasonable time frames (Debosscher et al, 2007;Sarro et al, 2009aSarro et al, , 2009bDebosscher et al, 2009;Richards et al, 2011Richards et al, , 2012Blomme et al, 2011;Matijevič et al, 2012;Morgan et al, 2012;Long et al, 2012). An essential step in the development of such a framework is processing a large set of events of known class to train the classifier.…”

K2 variable catalogue – II. Machine learning classification of variable stars and eclipsing binaries in K2 fields 0–4

“…Richards et al 2011aRichards et al , 2012Masci et al 2014). Several improvements have been proposed, in areas such as parametrizing light curves with maximal information retention (Kügler, Gianniotis & Polsterer 2015), and adjusting for training set deficiencies (Richards et al 2011b). One method of unsupervised machine learning is a Kohonen Self-Organizing Map (SOM;Kohonen 1990) demonstrated by Brett, West & Wheatley (2004) in an astronomical context.…”

Mechanistic Insights on the Iodine(III)‐Mediated α‐Oxidation of Ketones

“…Cases like these include spam detection, diagnosis of patients based on images and morphological classification of galaxies [1]. Although it might be expensive to train experts to a degree one can trust their labels [2], systems such as Amazon Mechanical Turk [3] allow each sample unit to be classified by many (not necessarily perfect) experts by a reasonably small cost. These experts do not have to be people.…”