Design of the 2015 ChaLearn AutoML challenge

Guyon, Isabelle; Bennett, Kristin P.; Cawley, Gavin C.; Escalante, Hugo Jair; Escalera, Sérgio; Ho, Tin Kam; Macià, Núria; Ray, Bisakha; Saeed, Mehreen; Statnikov, Alexander; Viegas, Evelyne

doi:10.1109/ijcnn.2015.7280767

Cited by 96 publications

(69 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This requires good knowledge of the specifics of the data and also of ML methods. A workaround to this is to apply automated ML methods, which iterate intelligently through many possible model configurations and select the best-performing option (Guyon et al 2015;Elsken, Metzen, and Hutter 2018;Feurer et al 2015;Kotthoff et al 2017) . This approach is slow but often works comparatively well.…”

Section: Caveatsmentioning

confidence: 99%

The roles of supervised machine learning in systems neuroscience

Glaser

Benjamin

Farhoodi

et al. 2019

Progress in Neurobiology

108

View full text Add to dashboard Cite

Over the last several years, the use of machine learning (ML) in neuroscience has been rapidly increasing. Here, we review ML's contributions, both realized and potential, across several areas of systems neuroscience. We describe four primary roles of ML within neuroscience: 1) creating solutions to engineering problems, 2) identifying predictive variables, 3) setting benchmarks for simple models of the brain, and 4) serving itself as a model for the brain. The breadth and ease of its applicability suggests that machine learning should be in the toolbox of most systems neuroscientists. Figure 1: Growth of Machine Learning in Neuroscience.Here we plot the proportion of neuroscience papers that have used ML over the last two decades. That is, we calculate the number of papers involving both neuroscience and machine learning, normalized by the total number of neuroscience papers. Neuroscience papers were identified using a search for "neuroscience" on Semantic Scholar. Papers involving neuroscience and machine learning were identified with a search for "machine learning" and "neuroscience" on Semantic Scholar.On the highest level, ML is typically divided into the subtypes of supervised, unsupervised, and reinforcement learning. Supervised learning builds a model that predicts outputs from input data. Unsupervised learning is concerned with finding structure in data, e.g. clustering, dimensionality reduction, and compression. Reinforcement learning allows a system to learn the best actions based on the reward that occurs at an end of a sequence of actions. This review focuses on supervised learning.Why is creating progressively more accurate regression or classification methods (see Box 1) worthy of a title like 'The AI Revolution' (Appenzeller 2017) ? It is because countless questions can be framed in this manner. When classifying images, an input picture can be used to predict the object in the picture. When playing a game, the setup of the board (input) can be used to predict an optimal move (output). When texting on our smartphones, our current text is used to create suggestions of the next word. Similarly, science has many instances where we desire to make predictions from measured data. Figure 2: Examples of the four roles of supervised machine learning in neuroscience.1 -ML can solve engineering problems . For example, it can help researchers control a prosthetic limb using brain activity. 2 -ML can identify predictive variables . For example, by using MRI data, we can identify which brain regions are most predictive for diagnosing Alzheimer's disease (Lebedev et al. 2014) . 3 -ML can benchmark simple models . For example, we can compare the predictive performance of the simple "population vector" model of how neural activity relates to movement (Georgopoulos, Schwartz, and Kettner 1986) to a ML benchmark (e.g. an RNN). 4 -ML can serve as a model of the brain . For example, researchers have studied how neurons in the visual pathway correspond to units in an artificial network that is trained to classify images...

show abstract

Section: Caveatsmentioning

confidence: 99%

The roles of supervised machine learning in systems neuroscience

Glaser

Benjamin

Farhoodi

et al. 2019

Progress in Neurobiology

108

View full text Add to dashboard Cite

show abstract

“…For each cross-validation iteration, we aggregate the predictions from all folds and calculate a single predictive performance evaluation, in order to avoid any averaging problems that might arise, especially when the dataset is imbalanced (35). For the classification experiments, we used the following evaluation measures: adjusted balanced accuracy score (36,37), an adaptation of the original accuracy measure that gives higher weights to examples from smaller classes; and the F-score with macro-averaging (38), which is the average F-score among all classes. Both measures treat different subtypes equally.…”

Section: Experimental Evaluation Of ML Algorithmsmentioning

confidence: 99%

CRISPRCasIdentifier: Machine learning for accurate identification and classification of CRISPR-Cas systems

Padilha

Alkhnbashi

Shah

et al. 2019

Preprint

View full text Add to dashboard Cite

CRISPR-Cas genes are extraordinarily diverse and evolve rapidly when compared to other prokaryotic genes. With the rapid increase in newly sequenced archaeal and bacterial genomes, manual identification of CRISPR-Cas systems is no longer viable. Thus, an automated approach is required for advancing our understanding of the evolution and diversity of these systems, and for finding new candidates for genome engineering in eukaryotic models. In this paper, we introduce a holistic strategy that combines regression and classification models for improving the quality of protein cascades, predicting their subtypes, detecting signature genes and extracting potential rules that reveal functional modules for CRISPR.

show abstract

“…However, a lack of suitable benchmarks, evaluation protocols and performance metrics has limited progress. Recently, we have organized challenges on Autonomous Machine Learning that have made significant progress in the field, see e.g., [5,4]. The challenges have attracted a large number of participants (almost 1,000 in the combined challenges), providing evidence of the relevance of the problem and the interest from the community.…”

Section: Challenge Setting and Backgroundmentioning

confidence: 99%

AutoML @ NeurIPS 2018 Challenge: Design and Results

Tu¹,

Guyon

Silver

et al. 2019

The Springer Series on Challenges in Machine Learning

Self Cite

View full text Add to dashboard Cite

We organized a competition on Autonomous Lifelong Machine Learning with Drift that was part of the competition program of NeurIPS 2018. This data driven competition asked participants to develop computer programs capable of solving supervised learning problems where the i.i.d. assumption did not hold. Large data sets were arranged in a lifelong learning and evaluation scenario and CodaLab was used as the challenge platform. The challenge attracted more than 300 participants in its two month duration. This chapter describes the design of the challenge and summarizes its main results.

show abstract

Design of the 2015 ChaLearn AutoML challenge

Cited by 96 publications

References 22 publications

The roles of supervised machine learning in systems neuroscience

The roles of supervised machine learning in systems neuroscience

CRISPRCasIdentifier: Machine learning for accurate identification and classification of CRISPR-Cas systems

AutoML @ NeurIPS 2018 Challenge: Design and Results

Contact Info

Product

Resources

About