Ambitious Data Science Can Be Painless

Monajemi, Hatef; Murri, Rita; Jonas, Eric; Liang, Percy; Stodden, Victoria; Donoho, David L.

doi:10.1162/99608f92.02ffc552

Cited by 8 publications

(8 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The total number of models fully trained for this paper is tallied below:

The massive computational experiments reported here were run painlessly using ClusterJob and ElastiCluster ( 19 – 21 ) on the Stanford Sherlock HPC (high performance computing) cluster and Google Compute Engine virtual machines.…”

Section: Setting and Methodologymentioning

confidence: 99%

Prevalence of neural collapse during the terminal phase of deep learning training

Papyan

Han

Donoho

2020

Proc. Natl. Acad. Sci. U.S.A.

144

135

View full text Add to dashboard Cite

Modern practice for training classification deepnets involves a terminal phase of training (TPT), which begins at the epoch where training error first vanishes. During TPT, the training error stays effectively zero, while training loss is pushed toward zero. Direct measurements of TPT, for three prototypical deepnet architectures and across seven canonical classification datasets, expose a pervasive inductive bias we call neural collapse (NC), involving four deeply interconnected phenomena. (NC1) Cross-example within-class variability of last-layer training activations collapses to zero, as the individual activations themselves collapse to their class means. (NC2) The class means collapse to the vertices of a simplex equiangular tight frame (ETF). (NC3) Up to rescaling, the last-layer classifiers collapse to the class means or in other words, to the simplex ETF (i.e., to a self-dual configuration). (NC4) For a given activation, the classifier’s decision collapses to simply choosing whichever class has the closest train class mean (i.e., the nearest class center [NCC] decision rule). The symmetric and very simple geometry induced by the TPT confers important benefits, including better generalization performance, better robustness, and better interpretability.

show abstract

“…The total number of models fully trained for this paper is tallied below:

Section: Setting and Methodologymentioning

confidence: 99%

Prevalence of neural collapse during the terminal phase of deep learning training

Papyan

Han

Donoho

2020

Proc. Natl. Acad. Sci. U.S.A.

144

135

View full text Add to dashboard Cite

show abstract

“…Certain problems like sensitivity study and ensemble forecasting require a large number of independent model simulations that can run on individual compute nodes (“embarrassingly parallel”). With the large resource pool on public cloud platforms, independent jobs can be executed simultaneously and finish much faster (Monajemi et al, 2019). For example, the AWS cloud has provided 40,000 compute nodes for one industrial HPC use case (Amazon, 2019g).…”

Section: Benefits Of Cloud Computing For Earth Science Researchmentioning

confidence: 99%

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Zhuang

Jacob

Lin

et al. 2020

J Adv Model Earth Syst

View full text Add to dashboard Cite

Cloud computing platforms can facilitate the use of Earth science models by providing immediate access to fully configured software, massive computing power, and large input data sets. However, slow internode communication performance has previously discouraged the use of cloud platforms for massively parallel simulations. Here we show that recent advances in the network performance on the Amazon Web Services cloud enable efficient model simulations with over a thousand cores. The choices of Message Passing Interface library configuration and internode communication protocol are critical to this success. Application to the Goddard Earth Observing System (GEOS)-Chem global 3-D chemical transport model at 50-km horizontal resolution shows efficient scaling up to at least 1,152 cores, with performance and cost comparable to the National Aeronautics and Space Administration Pleiades supercomputing cluster.Plain Language Summary Earth science model simulations are computationally expensive, typically requiring the use of high-end supercomputing clusters that are managed by universities or national laboratories. Commercial cloud computing offers an alternative. However, past work found that cloud computing platforms were not efficient for large-scale simulations on over 100 CPU cores, because the network communication performance on the cloud was slow compared to local clusters. Here we show that recent advances in the cloud network performance enable efficient model simulations with over a thousand cores, and cloud platforms can now serve as a viable alternative to local clusters for simulations at large scale. Computing on the cloud has extensive advantages, such as providing immediate access to fully configured model code and large data sets for any users, allowing full reproducibility of model simulation results, offering quick access to novel hardware that might not be available on local clusters, and being able to scale to virtually unlimited amounts of compute and storage resources. Those benefits will help advance Earth science modeling research.

show abstract

“…Certain problems like sensitivity study and ensemble forecasting require a large number of independent model simulations that can run on individual compute nodes ("embarrassingly parallel"). With the large resource pool on public cloud platforms, independent jobs can be executed simultaneously and finish much faster (Monajemi et al, 2019). For example, the AWS cloud has provided 40,000 compute nodes for one industrial HPC use case (Amazon, 2019g).…”

Section: Benefits Of Cloud Computing For Earth Science Researchmentioning

confidence: 99%

Enabling high-performance cloud computing for Earth science modeling on over a thousand cores: application to the GEOS-Chem atmospheric chemistry model

Zhuang¹,

Jacob²,

Lin³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

Ambitious Data Science Can Be Painless

Cited by 8 publications

References 22 publications

Prevalence of neural collapse during the terminal phase of deep learning training

Prevalence of neural collapse during the terminal phase of deep learning training

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Enabling high-performance cloud computing for Earth science modeling on over a thousand cores: application to the GEOS-Chem atmospheric chemistry model

Contact Info

Product

Resources

About