Adaptive Weight Decay for Deep Neural Networks

Nakamura, Kensuke; Hong, Byung-Woo

doi:10.1109/access.2019.2937139

Cited by 39 publications

(19 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To further address this problem, the networks architecture could be expanded, including one or several dropout layers, which randomly drop connections between layers during training and lessen their linkage ( 28 ). Other methods might include “early stopping” which prevents further training of the network when the peak performance is reached or “weight decay” which continually decreases the weights of the network during the training phase ( 29 ).…”

Section: Discussionmentioning

confidence: 99%

Machine Learning Classification of Inflammatory Bowel Disease in Children Based on a Large Real-World Pediatric Cohort CEDATA-GPGE® Registry

et al. 2021

View full text Add to dashboard Cite

Introduction: The rising incidence of pediatric inflammatory bowel diseases (PIBD) facilitates the need for new methods of improving diagnosis latency, quality of care and documentation. Machine learning models have shown to be applicable to classifying PIBD when using histological data or extensive serology. This study aims to evaluate the performance of algorithms based on promptly available data more suited to clinical applications.Methods: Data of inflammatory locations of the bowels from initial and follow-up visitations is extracted from the CEDATA-GPGE registry and two follow-up sets are split off containing only input from 2017 and 2018. Pre-processing excludes patients in remission and encodes the categorical data numerically. For classification of PIBD diagnosis, a support vector machine (SVM), a random forest algorithm (RF), extreme gradient boosting (XGBoost), a dense neural network (DNN) and a convolutional neural network (CNN) are employed. As best performer, a convolutional neural network is further improved using grid optimization.Results: The achieved accuracy of the optimized neural network reaches up to 90.57% on data inserted into the registry in 2018. Less performant methods reach 88.78% for the DNN down to 83.94% for the XGBoost. The accuracy of prediction for the 2018 follow-up dataset is higher than those for older datasets. Neural networks yield a higher standard deviation with 3.45 for the CNN compared to 0.83–0.86 of the support vector machine and ensemble methods.Discussion: The displayed accuracy of the convolutional neural network proofs the viability of machine learning classification in PIBD diagnostics using only timely available data.

show abstract

Section: Discussionmentioning

confidence: 99%

Machine Learning Classification of Inflammatory Bowel Disease in Children Based on a Large Real-World Pediatric Cohort CEDATA-GPGE® Registry

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The SGD is important as it updates the parameters with mini-batch B = 10 examples. The momentum was set to 9 × 10 −1 and the weight decay was set to 1 × 10 −3 , as the network is considered a shallow network [31]. The weight decay marginal value is important as it helps to minimize the model training error [31].…”

Section: Network Training Phasementioning

confidence: 99%

“…The momentum was set to 9 × 10 −1 and the weight decay was set to 1 × 10 −3 , as the network is considered a shallow network [31]. The weight decay marginal value is important as it helps to minimize the model training error [31]. The training and results are performed using an Intel Core i5 machine, 2.9 GHZ occupied with 8GB of working RAM.…”

Section: Network Training Phasementioning

confidence: 99%

An Efficient Method for Covid-19 Detection Using Light Weight Convolutional Neural Network

Bekhet¹,

Alkinani²,

Tabares-Soto³

et al. 2021

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

The COVID-19 pandemic is a significant milestone in the modern history of civilization with a catastrophic effect on global wellbeing and monetary. The situation is very complex as the COVID-19 test kits are limited, therefore, more diagnostic methods must be developed urgently. A significant initial step towards the successful diagnosis of the COVID-19 is the chest X-ray or Computed Tomography (CT), where any chest anomalies (e.g., lung inflammation) can be easily identified. Most hospitals possess X-ray or CT imaging equipments that can be used for early detection of COVID-19. Motivated by this, various artificial intelligence (AI) techniques have been developed to identify COVID-19 positive patients using the chest X-ray or CT images. However, the advance of these AI-based systems and their highly tailored results are strongly bonded to high-end GPUs, which is not widely available in several countries. This paper introduces a technique for early COVID-19 diagnosis based on medical experience and light-weight Convolutional Neural Networks (CNNs), which does not require a custom hardware to run compared to currently available CNN models. The proposed deep learning model is built carefully and fine-tuned by removing all unnecessary parameters and layers to achieve the light-weight attribute that could run smoothly on a normal CPU (0.54% of AlexNet parameters). This model is highly beneficial for countries where high-end GPUs are luxuries. Experimental outcomes on some new benchmark datasets shows the robustness of the proposed technique robustness in recognizing COVID-19 with 96% accuracy.

show abstract

“…The slow convergence of PINN due to presence of noise in the data is circumvented by the approach of weight decay [18], which eventually bounds the weights of neural network, hence resulting in a faster convergence. Therefore, the loss function in equation ( 10) is further modified and expressed as…”

Section: A Pinn For Bi-crystal Nickelmentioning

confidence: 99%

A physics-informed neural network for quantifying the microstructure properties of polycrystalline Nickel using ultrasound data

Shukla,

Jagtap,

Blackshire

et al. 2021

Preprint

View full text Add to dashboard Cite

We employ physics-informed neural networks (PINNs) to quantify the microstructure of a polycrystalline Nickel by computing the spatial variation of compliance coefficients (compressibility, stiffness and rigidity) of the material. The PINN is supervised with realistic ultrasonic surface acoustic wavefield data acquired at an ultrasonic frequency of 5 MHz for the polycrystalline material. The ultrasonic wavefield data is represented as a deformation on the top surface of the material with the deformation measured using the method of laser vibrometry. The ultrasonic data is further complemented with wavefield data generated using a finite element based solver. The neural network is physically-informed by the in-plane and out-of-plane elastic wave equations and its convergence is accelerated using adaptive activation functions. The overarching goal of this work is to infer the spatial variation of compliance coefficients of materials using PINNs, which for ultrasound involves the spatially varying speed of the elastic waves.More broadly, the resulting PINN based surrogate model shows a promising approach for solving ill-posed inverse problems, often encountered in the non-destructive evaluation of materials. I. INTRODUCTIONIn recent years, the availability of large datasets combined with sophisticated algorithms along with the exponential growth in computational power have led to an unprecedented surge of interest in machine K. Shukla and A. D. Jagtap are with the

show abstract

Adaptive Weight Decay for Deep Neural Networks

Cited by 39 publications

References 25 publications

Machine Learning Classification of Inflammatory Bowel Disease in Children Based on a Large Real-World Pediatric Cohort CEDATA-GPGE® Registry

Machine Learning Classification of Inflammatory Bowel Disease in Children Based on a Large Real-World Pediatric Cohort CEDATA-GPGE® Registry

An Efficient Method for Covid-19 Detection Using Light Weight Convolutional Neural Network

A physics-informed neural network for quantifying the microstructure properties of polycrystalline Nickel using ultrasound data

Contact Info

Product

Resources

About