Exploring the Performance of Pruning Methods in Neural Networks: An Empirical Study of the Lottery Ticket Hypothesis

Fladmark, Eirik; Sajjad, Muhammad Hamza; Justesen, Laura Brinkholm

doi:10.48550/arxiv.2303.15479

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The core innovation lies in reevaluating the pruning process and questioning established methodologies, opening the door to enhanced neural network compression techniques. 22 2015 weights weights magnitude Fladmark et al 34 2023 weights weights magnitude Junghun et al 35 2022 filters gradient magnitude Pruning at Onset Lee et al 35 2022 weights "dynamical isometry" Wang et al 36 2020 weights second-order derivative Tanaka et al 37 2020 weights "synaptic flow" Sparse Training Peng et al 38 2022 weights weights magnitude Chen et al 39 2021 weights weights magnitude Liu et al 40 2021 weights weights magnitude Learning by Masks Choudhary et al 41 2022 filters N/A Kusupati et al 42 2020 weights N/A Liu et al 43 2021 weights N/A Penalty Based Methods Zhu et al 44 2022 filters N/A Yeom et al 45 2021 weights weight magnitude Sanh et al 46 2020 filters weight magnitude Wang et al 47 2021 weights weight magnitude…”

Section: Pruning Methodsmentioning

confidence: 99%

Optimizing Large Models For Performance And Sustainability Through Evolutionary Pruning

Islam,

Belhaouari,

Bermak

2023

Preprint

View full text Add to dashboard Cite

The need for network optimization has become become evident when considering the rising computational costs and environmental impact caused by AI models. The substantial carbon footprint and resource consumption of Large Language Models (LLMs) underscore the urgency of optimizing network architectures. These models, with varying parameters, place a significant strain on energy resources, particularly in terms of CO2 emissions and electricity consumption. Achieving a balance between model performance and ecological responsibility is becoming increasingly critical in the era of large-scale AI models.Training and using such models consume substantial amounts of water and power, highlighting the necessity for sleeker, faster, and more efficient model training strategies. We address a crucial challenge related to the compression of deep neural networks, with the aim of enhancing their suitability for embedded devices. Deep neural networks typically exhibit high space and computational requirements, which can hinder their deployment on edge devices and limit their widespread adoption. This study investigates innovative techniques for pruning neural networks, presenting two distinct strategies, namely, "evolution of weights" and "smart pruning." These novel approaches are systematically compared to conventional pruning methods using established benchmark datasets. The proposed pruning method involves the continuous assessment of parameter importance throughout the training process, employing a weighted average methodology that surpasses the traditional magnitude-based pruning in terms of accuracy preservation during the compression process. The outcomes of this approach include accelerated computations, higher compression rates with minimal accuracy degradation, enhanced resilience against adversarial attacks, and the capacity to train models using smaller datasets. The authors have thoughtfully provided a PyTorch library for broad usage, simplifying the implementation of their approach across various model architectures.

show abstract