Weight Update Skipping: Reducing Training Time for Artificial Neural Networks

Safayenikoo, Pooneh; Akturk, Ismail

doi:10.1109/jetcas.2021.3127907

Cited by 10 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Adaptive methods represented by Adam have received increasing attention these years due to its good convergence performance and robustness. AdaGrad 12 , AdaDelta 13 and Adam 18 are most widely used. To combine the benefits of both adaptive learning rate methods and momentum-based methods leads to Adam 18 .…”

Section: Related Workmentioning

confidence: 99%

“…AdaGrad 12 , AdaDelta 13 and Adam 18 are most widely used. To combine the benefits of both adaptive learning rate methods and momentum-based methods leads to Adam 18 . QHAdam extends the Adam algorithm by introducing a quasi-hyperbolic momentum term which aims to strike a balance between the benefits of adaptive and the stability achieved by non-adaptive methods 15 .…”

Section: Related Workmentioning

confidence: 99%

“…However, even equipped with the methods above, researchers still have to carefully tuning hyperparameters to struggle for the 1% accuracy improvement, because the better performance of DNN usually comes at the availability of exceptionally large computational resources on specialized hardware accelerators in model training process, which necessitate similarly substantial energy consumption 17 , 18 . Moreover, the extensive periods need for DNN model training also prolong the validation cycle of the algorithm, which often result in researchers invest time and effort in vain.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Enhancing deep neural network training efficiency and performance through linear prediction

Ying,

Song,

Tang

et al. 2024

Sci Rep

View full text Add to dashboard Cite

Deep neural networks have achieved remarkable success in various fields. However, training an effective deep neural network still poses challenges. This paper aims to propose a method to optimize the training effectiveness of deep neural networks, with the goal of improving their performance. Firstly, based on the observation that parameters (weights and bias) of deep neural network change in certain rules during training process, the potential of parameters prediction for improving training efficiency is discovered. Secondly, the potential of parameters prediction to improve the performance of deep neural network by noise injection introduced by prediction errors is revealed. And then, considering the limitations comprehensively, a deep neural network Parameters Linear Prediction method is exploit. Finally, performance and hyperparameter sensitivity validations are carried out on some representative backbones. Experimental results show that by employing proposed Parameters Linear Prediction method, as opposed to SGD, has led to an approximate 1% increase in accuracy for optimal model, along with a reduction of about 0.01 in top-1/top-5 error. Moreover, it also exhibits stable performance under various hyperparameter settings, shown the effectiveness of the proposed method and validated its capacity in enhancing network’s training efficiency and performance.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Enhancing deep neural network training efficiency and performance through linear prediction

Ying,

Song,

Tang

et al. 2024

Sci Rep

View full text Add to dashboard Cite

show abstract

“…In the ANNs, tuning the weight plays an essential role to satisfy the best outputs [42]. To tune the weights in the ANNs, the backpropagation training method with gradient descent is incorporated since it has an efficient and simple structure [43].…”

Section: Neural Network Designmentioning

confidence: 99%

Design of a novel robust adaptive neural network‐based fractional‐order proportional‐integrated‐derivative controller on DC/DC Boost converter

Abdollahzadeh

Mollaee

Ghamari

et al. 2023

The Journal of Engineering

View full text Add to dashboard Cite

DC/ DC Boost converter has a right half-plane zero structure called a non-minimum phase system, which presents several challenging constraints for designing well-behaved control techniques. The Fractional-Order concept as a beneficial scheme provides several advantages, such as lower sensitivity to noise and parametric variation. For this purpose, a Fractional-order Proportional-Integrated-Derivative (FOPID) controller is designed for the Boost converter. On the other hand, for wider ranges of disturbances, including parametric variations, load uncertainty, supply voltage variation, and noise, this approach shows an unsuitable practical application based on its fixed gain values; therefore, the control parameters need to be optimized again to provide ideal operations. An Artificial Neural Network structure (ANNs) is adopted here to optimize the gains of the FOPID in challenging conditions. This method suffers from higher complexity and slower dynamics in practical applications. A single-layer ANNs is designed to reduce the complexity; also, particle swarm optimization (PSO) algorithm is utilized in this decision-making structure to provide better results with its online mechanism. Furthermore, the Black-box technique is applied for the proposed system, which does not require an accurate mathematical model resulting in a lower computational burden, easy implementation, and lower dependency on the states of the model. The Artificial Neural Network structure can optimize the FOPID gains real-time, even in severe challenging conditions, efficiently. To better examine the superiority of the proposed method, conventional Proportional-Integrated-Derivative (PID) and the FOPID controllers are proposed to drive a comparison with this work, which are tuned by the PSO optimization algorithm. The evaluation of simulation results demonstrates that the proposed control scheme is suitable not only for preserving stability but also for compensating for the disturbances and uncertainties in the Boost converter, properly. JEL CLASSIFICATIONEnergy engineering: Power electronics, electric power applications INTRODUCTIONAccording to the basic principle, DC/DC power converters, since they can regulate the output voltage with the minimumThis is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

“…Wavelet transform has been used widely for transient analysis in the power system [9], [10], [11] and is employed for signal smoothing in this work. We combine the results from a variety of base anomaly detectors such as Hampel filter [12], Quartile technique, and DBSCAN using MB-MLE.…”

Section: B Related Workmentioning

confidence: 99%

Denoising and Bad Data Detection in Distribution Phasor Measurements using Filtering, Clustering and Koopman Mode Analysis

Vosughi

Gholami

Srivastava

2021

2021 IEEE Industry Applications Society Annual Meeting (IAS)

View full text Add to dashboard Cite

Distribution-level phasor measurement units (D-PMU) data are prone to different types of anomalies given complex data flow and processing infrastructure in an active power distribution system with enhanced digital automation. It is essential to pre-process the data before being used by critical applications for situational awareness and control. In this work, two approaches for detection of data anomalies are introduced for offline (larger data processing window) and online (shorter data processing window) applications. A margin-based maximum likelihood estimator (MB-MLE) method is developed to detect anomalies by integrating the results of different base detectors including Hampel filter, Quartile detector and DBSCAN. A smoothing wavelet denoising method is used to remove highfrequency noises. The processed data with offline analysis is used to fit a model to the underlying dynamics of synchrophasor data using Koopman Mode Analysis, which is subsequently employed for online denoising and bad data detection (BDD) using Kalman Filter (KF). The parameters of the KF are adjusted adaptively based on similarity to the training data set for model fitting purposes. Developed techniques have been validated for the modified IEEE test system with multiple D-PMUs, modeled and simulated in real-time for different case scenarios using the OPAL-RT Hardware-In-the-Loop (HIL) Simulator.

show abstract

Weight Update Skipping: Reducing Training Time for Artificial Neural Networks

Cited by 10 publications

References 16 publications

Enhancing deep neural network training efficiency and performance through linear prediction

Enhancing deep neural network training efficiency and performance through linear prediction

Design of a novel robust adaptive neural network‐based fractional‐order proportional‐integrated‐derivative controller on DC/DC Boost converter

Denoising and Bad Data Detection in Distribution Phasor Measurements using Filtering, Clustering and Koopman Mode Analysis

Contact Info

Product

Resources

About