ReLTanh: An activation function with vanishing gradient resistance for SAE-based DNNs and its application to rotating machinery fault diagnosis

Wang, Xin; Qin, Yi; Wang, Yi; Xiang, Sheng; Chen, Haizhou

doi:10.1016/j.neucom.2019.07.017

Cited by 118 publications

(41 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Markiewicz et al [42] finds LSTM-RNN to benefit from being relative stable, notably because it does not suffer from the vanishing or exploding gradient problem [73,74]. Zhang et al [33] states that LSTM-RNN has good performance on sequential data due to the recurrent feedback.…”

Section: Annmentioning

confidence: 99%

Predictive Maintenance for Pump Systems and Thermal Power Plants: State-of-the-Art Review, Trends and Challenges

Olesen

Shaker

2020

Sensors

View full text Add to dashboard Cite

Thermal power plants are an important asset in the current energy infrastructure, delivering ancillary services, power, and heat to their respective consumers. Faults on critical components, such as large pumping systems, can lead to material damage and opportunity losses. Pumps plays an essential role in various industries and as such clever maintenance can ensure cost reductions and high availability. Prognostics and Health Management, PHM, is the study utilizing data to estimate the current and future conditions of a system. Within the field of PHM, Predictive Maintenance, PdM, has been gaining increased attention. Data-driven models can be built to estimate the remaining-useful-lifetime of complex systems that would be difficult to identify by man. With the increased attention that the Predictive Maintenance field is receiving, review papers become increasingly important to understand what research has been conducted and what challenges need to be addressed. This paper does so by initially conceptualising the PdM field. A structured overview of literature in regard to application within PdM is presented, before delving into the domain of thermal power plants and pump systems. Finally, related challenges and trends will be outlined. This paper finds that a large number of experimental data-driven models have been successfully deployed, but the PdM field would benefit from more industrial case studies. Furthermore, investigations into the scale-ability of models would benefit industries that are looking into large-scale implementations. Here, examining a method for automatic maintenance of the developed model will be of interest. This paper can be used to understand the PdM field as a broad concept but does also provide a niche understanding of the domain in focus.

show abstract

Section: Annmentioning

confidence: 99%

Predictive Maintenance for Pump Systems and Thermal Power Plants: State-of-the-Art Review, Trends and Challenges

Olesen

Shaker

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Recent work also showed that we can design effective activation functions in a piecewise manner. For examples, Qin et al [16] introduced the improved Sigmoid function -Isigmoid, which tunes the saturated region by a learnable parameters, and Wang et al [17] proposed ReLTanh, which treats the negative and positive saturation region as two straight lines each with different slopes. These piecewise treatments are to address the vanishing gradient and bais shift problems.…”

Section: ) Activation Layer (Act)mentioning

confidence: 99%

Effective Activation Functions for Homomorphic Evaluation of Deep Neural Networks

et al. 2020

View full text Add to dashboard Cite

CryptoNets and subsequent work have demonstrated the capability of homomorphic encryption (HE) in the applications of private artificial intelligence (AI). In convolutional neural networks (CNNs), many computations are linear functions such as the convolution layer which can be homomorphically evaluated. However, there are layers such as the activation layer which is comprised of non-linear functions that cannot be homomorphically evaluated. One of the most commonly used methods is approximating these non-linear functions using low-degree polynomials. However, using the approximated polynomials as activation functions introduces errors which could have a significant impact on accuracy in classification tasks. In this paper, we present a systematic method to construct HE-friendly activation functions for CNNs. We first determine what properties in a good activation function contribute to performance by analyzing commonly used functions such as Rectified Linear Units (ReLU) and Sigmoid. Then, we compare polynomial approximation methods and search for an optimal range of approximation for the polynomial activation. We also propose a novel weighted polynomial approximation method tailored to the output distribution of a batch normalization layer. Finally, we demonstrate the effectiveness of our method using several datasets such as MNIST, FMNIST, CIFAR-10.

show abstract

“…In this study, as an activation function, the rectified linear unit (ReLU) function [43][44][45][46] rather than the Sigmoid function is used as shown in Figure 7. That is because a vanishing gradient (in which a gradient converges to zero) occurs if the Sigmoid function is used [47][48][49]. In other words, the Sigmoid function has a value between 0 and 1, as shown in Equation (3).…”

Section: Detection Of Target Regionmentioning

confidence: 99%

Robust Blocking of Human Faces with Personal Information Using Artificial Deep Neural Computing

Jang

Lee

2020

Sustainability

View full text Add to dashboard Cite

High-speed wired and wireless Internet are one of the useful ways to acquire various types of media data easily. In this circumstance, people also can easily get media data including objects with exposed personal information through the Internet. Exposure of personal information emerges as a social issue. This paper proposes an effective blocking technique that makes it possible to robustly detect target objects with exposed personal information from various types of input images with the use of deep neural computing and to effectively block the detected objects' regions. The proposed technique first utilizes the neural computing-based learning algorithm to robustly detect the target object including personal information from an image. It next generates a grid-type mosaic and lets the mosaic overlap the target object region detected in the previous step so as to effectively block the object region that includes personal information. Experimental results reveal that the proposed algorithm robustly detects the target object region with exposed personal information from a variety of input images and effectively blocks the detected region through grid-type mosaic processing. The object blocking technique proposed in this paper is expected to be applied to various application fields such as image security, sustainable anticipatory computing, object tracking, and target blocking.

show abstract

ReLTanh: An activation function with vanishing gradient resistance for SAE-based DNNs and its application to rotating machinery fault diagnosis

Cited by 118 publications

References 22 publications

Predictive Maintenance for Pump Systems and Thermal Power Plants: State-of-the-Art Review, Trends and Challenges

Predictive Maintenance for Pump Systems and Thermal Power Plants: State-of-the-Art Review, Trends and Challenges

Effective Activation Functions for Homomorphic Evaluation of Deep Neural Networks

Robust Blocking of Human Faces with Personal Information Using Artificial Deep Neural Computing

Contact Info

Product

Resources

About