Deep Learning's Diminishing Returns: The Cost of Improvement is Becoming Unsustainable

Thompson, Neil; Greenewald, Kristjan; Lee, Keeheon; Manso, Gabriel F.

doi:10.1109/mspec.2021.9563954

Cited by 93 publications

(106 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The faulty estimates in [2] are understandable given the lack of access to internal information. It is likewise understandable that those estimates were propagated in other papers, like [1,6,7,17,18,19,20].…”

Section: Related Workmentioning

confidence: 93%

“…The opening quote in Section 1 is based on a 2019 study from the University of Massachusetts (UMass) that estimated the environmental impact of training [2]. More than 1000 papers cite this paper as the source for the impact on carbon emissions of ML models, e.g., [1,6,7,17,18,19,20]. The study calculated the energy consumed and carbon footprint of the NAS by [8] that led to Evolved Transformer.…”

Section: Related Workmentioning

confidence: 99%

“…Over the past few years, a growing number of papers have highlighted the carbon emissions of machine learning (ML) workloads. While this work has been instrumental in rightfully elevating the discussion around carbon emissions in ML, some studies significantly overestimated actual emissions, which in turn led to worrisome extrapolations [1,2]:…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink

Patterson¹,

Hölzle²,

Le³

et al. 2022

Preprint

View full text Add to dashboard Cite

<div> <div> <div> <p>Machine Learning (ML) workloads have rapidly grown in importance, but raised concerns about their carbon footprint. Four best practices can reduce ML training energy by up to 100x and CO2 emissions up to 1000x. By following best practices, overall ML energy use (across research, development, and production) held steady at <15% of Google’s total energy use for the past three years. If the whole ML field were to adopt best practices, total carbon emissions from training would reduce. Hence, we recommend that ML papers include emissions explicitly to foster competition on more than just model quality. As estimates of emissions in papers that omitted them have been off 100x–100,000x, publishing emissions has the added benefit of ensuring accurate accounting. Given the importance of climate change, we must get the numbers right to make certain that we work on its biggest challenges.<br></p> </div> </div> </div>

show abstract

Section: Related Workmentioning

confidence: 93%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink

Patterson¹,

Hölzle²,

Le³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The main strength of these techniques is that their classification accuracy typically improves as they are trained on more data, scaling to datasets containing billions of images (Mahajan et al, 2018 ). However, this strength is also becoming a main point of critique, as an exponential increase in compute (and energy) resources is required for marginal gains (Thompson et al, 2021 ). Moreover, these classifiers are known to be vulnerable to ambiguous and adversarial samples (Gilmer et al, 2018 ), and are restricted to object categories known and seen during training.…”

Section: Introductionmentioning

confidence: 99%

Embodied Object Representation Learning and Recognition

et al. 2022

View full text Add to dashboard Cite

Scene understanding and decomposition is a crucial challenge for intelligent systems, whether it is for object manipulation, navigation, or any other task. Although current machine and deep learning approaches for object detection and classification obtain high accuracy, they typically do not leverage interaction with the world and are limited to a set of objects seen during training. Humans on the other hand learn to recognize and classify different objects by actively engaging with them on first encounter. Moreover, recent theories in neuroscience suggest that cortical columns in the neocortex play an important role in this process, by building predictive models about objects in their reference frame. In this article, we present an enactive embodied agent that implements such a generative model for object interaction. For each object category, our system instantiates a deep neural network, called Cortical Column Network (CCN), that represents the object in its own reference frame by learning a generative model that predicts the expected transform in pixel space, given an action. The model parameters are optimized through the active inference paradigm, i.e., the minimization of variational free energy. When provided with a visual observation, an ensemble of CCNs each vote on their belief of observing that specific object category, yielding a potential object classification. In case the likelihood on the selected category is too low, the object is detected as an unknown category, and the agent has the ability to instantiate a novel CCN for this category. We validate our system in an simulated environment, where it needs to learn to discern multiple objects from the YCB dataset. We show that classification accuracy improves as an embodied agent can gather more evidence, and that it is able to learn about novel, previously unseen objects. Finally, we show that an agent driven through active inference can choose their actions to reach a preferred observation.

show abstract

“…The power consumption of conventional, transistor-based computers is unsustainably high, particularly with the burgeoning of fields such as neuromorphic computing and machine learning (Thompson et al, 2021). For example, the power density of typical supercomputer might be in the order of 100 Wcm −2 .…”

Section: Introductionmentioning

confidence: 99%

Neuromorphic Dynamics at the Nanoscale in Silicon Suboxide RRAM

Buckwell

Mannion

et al. 2021

Front. Nanotechnol.

View full text Add to dashboard Cite

Resistive random-access memories, also known as memristors, whose resistance can be modulated by the electrically driven formation and disruption of conductive filaments within an insulator, are promising candidates for neuromorphic applications due to their scalability, low-power operation and diverse functional behaviors. However, understanding the dynamics of individual filaments, and the surrounding material, is challenging, owing to the typically very large cross-sectional areas of test devices relative to the nanometer scale of individual filaments. In the present work, conductive atomic force microscopy is used to study the evolution of conductivity at the nanoscale in a fully CMOS-compatible silicon suboxide thin film. Distinct filamentary plasticity and background conductivity enhancement are reported, suggesting that device behavior might be best described by composite core (filament) and shell (background conductivity) dynamics. Furthermore, constant current measurements demonstrate an interplay between filament formation and rupture, resulting in current-controlled voltage spiking in nanoscale regions, with an estimated optimal energy consumption of 25 attojoules per spike. This is very promising for extremely low-power neuromorphic computation and suggests that the dynamic behavior observed in larger devices should persist and improve as dimensions are scaled down.

show abstract

Deep Learning's Diminishing Returns: The Cost of Improvement is Becoming Unsustainable

Cited by 93 publications

References 0 publications

The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink

The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink

Embodied Object Representation Learning and Recognition

Neuromorphic Dynamics at the Nanoscale in Silicon Suboxide RRAM

Contact Info

Product

Resources

About