Energy and Policy Considerations for Modern Deep Learning Research

Abstract

The field of artificial intelligence has experienced a dramatic methodological shift towards large neural networks trained on plentiful data. This shift has been fueled by recent advances in hardware and techniques enabling remarkable levels of computation, resulting in impressive advances in AI across many applications. However, the massive computation required to obtain these exciting results is costly both financially, due to the price of specialized hardware and electricity or cloud compute time, and to the environment, as a result of non-renewable energy used to fuel modern tensor processing hardware. In a paper published this year at ACL, we brought this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training and tuning neural network models for NLP (Strubell, Ganesh, and McCallum 2019). In this extended abstract, we briefly summarize our findings in NLP, incorporating updated estimates and broader information from recent related publications, and provide actionable recommendations to reduce costs and improve equity in the machine learning and artificial intelligence community.

Unity Is Intelligence: A Collective Intelligence Experiment on ECG Reading to Improve Diagnostic Performance in Cardiology

“…The first difference regards the workloads required by the two approaches. While AI can be much more efficient than CI when evaluating a new case, the development of AI systems (especially those based on deep learning) usually requires large annotated training sets and high-performance hardware, and it may have significant computational and environmental costs Strubell et al ( 2020 ). The second difference regards interpretability.…”

Comparison of pretraining models and strategies for health-related social media text classification

“…In most research settings, it is not possible to execute all the different types of pretraining we described in this paper. Also, as reported in recent research, conducting large-scale training/pretraining has associated environmental costs, 48,49 and the establishment of effective strategies can significantly lower such costs in future research. Our findings in this paper reveal some simple but effective strategies for improving social media-based health-related text classification tasks.…”

Artificial intelligence in cancer research: learning at different levels of data granularity

“…To facilitate this progress, automated strategies for end-to-end AI processes operating on big data, from data governance to deployment of AI applications, have been developed. The intensive workloads of AI operating on big data demand computational resources that must be able to achieve extreme scale and high performance while being cost-effective and environmentally sustainable [39]. High performance computing (HPC), or supercomputing, architectures are facilitating the deployment of pioneering AI applications in biomedicine [40,41].…”