A study on the uncertainty of convolutional layers in deep neural networks

Shen, Haojing; Chen, Sihong; Wang, Ran

doi:10.1007/s13042-021-01278-9

Cited by 8 publications

(19 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Let W conv represent the parameters of convolutional layers. According to [38,39], the fuzziness vector can be defined as…”

Section: B Cost-sensitive Adversarial Model (Csa)mentioning

confidence: 99%

“…The Min-Max property is first discovered by Shen et al in [38]. Shen et al propose that the neural network models with Min-Max property have stronger adversarial robustness.…”

Section: Min-max Propertymentioning

confidence: 99%

“…The process of training neural networks such that the weights in convolutional layer weights going to extremes can be implemented in different ways, e.g., by minimizing the uncertainty of the convolutional layer [38] or by adding the regularization term in the loss function. Furthermore, it is found that a DNN with strong robustness against adversarial examples may not have the Min-Max property.…”

Section: Min-max Propertymentioning

confidence: 99%

“…We recall some results in [38] where two neural network models with simple architectures are considered.…”

Section: Min-max Propertymentioning

confidence: 99%

See 3 more Smart Citations

Adversarial Learning with Cost-Sensitive Classes

Shen,

Chen,

Wang

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

It is necessary to improve the performance of some special classes or to particularly protect them from attacks in adversarial learning. This paper proposes a framework combining cost-sensitive classification and adversarial learning together to train a model that can distinguish between protected and unprotected classes, such that the protected classes are less vulnerable to adversarial examples. We find in this framework an interesting phenomenon during the training of deep neural networks, called Min-Max property, that is, the absolute values of most parameters in the convolutional layer approach zero while the absolute values of a few parameters are significantly larger becoming bigger. Based on this Min-Max property which is formulated and analyzed in a view of random distribution, we further build a new defense model against adversarial examples for adversarial robustness improvement. An advantage of the built model is that it does no longer need adversarial training, and thus, has a higher computational efficiency than most existing models of needing adversarial training. It is experimentally confirmed that, regarding the average accuracy of all classes, our model is almost as same as the existing models when an attack does not occur and is better than the existing models when an attack occurs. Specifically, regarding the accuracy of protected classes, the proposed model is much better than the existing models when an attack occurs.

show abstract

“…Let W conv represent the parameters of convolutional layers. According to [38,39], the fuzziness vector can be defined as…”

Section: B Cost-sensitive Adversarial Model (Csa)mentioning

confidence: 99%

“…The Min-Max property is first discovered by Shen et al in [38]. Shen et al propose that the neural network models with Min-Max property have stronger adversarial robustness.…”

Section: Min-max Propertymentioning

confidence: 99%

Section: Min-max Propertymentioning

confidence: 99%

“…We recall some results in [38] where two neural network models with simple architectures are considered.…”

Section: Min-max Propertymentioning

confidence: 99%

See 2 more Smart Citations

Adversarial Learning with Cost-Sensitive Classes

Shen,

Chen,

Wang

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Moreover, supervised learning approaches are unable to identify in-domain from out-domain samples [22], provide reliable uncertainty approximation [23], and lack expressiveness during inference [24]; therefore, their deployment in high-risk and safety-critical applications remains limited. To alleviate these issues, it is vital to present uncertainty estimate in a way that ignores the uncertain predictions or passes them to experts [25].…”

Section: Introductionmentioning

confidence: 99%

A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications

Zhou,

Liu,

Pourpanah

et al. 2021

Preprint

View full text Add to dashboard Cite

Quantifying the uncertainty of supervised learning models plays an important role in making more reliable predictions. Epistemic uncertainty, which usually is due to insufficient knowledge about the model, can be reduced by collecting more data or refining the learning models. Over the last few years, scholars have proposed many epistemic uncertainty handling techniques which can be roughly grouped into two categories, i.e., Bayesian and ensemble. This paper provides a comprehensive review of epistemic uncertainty learning techniques in supervised learning over the last five years. As such, we, first, decompose the epistemic uncertainty into bias and variance terms. Then, a hierarchical categorization of epistemic uncertainty learning techniques along with their representative models is introduced. In addition, several applications such as computer vision (CV) and natural language processing (NLP) are presented, followed by a discussion on research gaps and possible future research directions.

show abstract

Artificial intelligence of things (AIoT) data acquisition based on graph neural networks: A systematical review

Wang

Zhang

et al. 2023

Concurrency and Computation

View full text Add to dashboard Cite

SummaryThe power of artificial intelligence of things (AIoT) stems from adapting machine learning (ML) and artificial intelligence (AI) models into abundant intelligent IoT fields, based on a large data stream with different formats, sizes, and timestamps generated by massive numbers of heterogeneous sensors. On the one hand, data acquisition is the fundamental basis for any AIoT systems, but data sensed by massive IoT devices may be noisy and even contain adversarial samples. On the other hand, ensuring the efficiency and robustness in data acquisition is vitally important for data‐driven ML and AI. Recently, besides perceiving ability, the literature has witnessed great development of empowering things with learning and reasoning ability through deep learning models, including recurrent neural networks (RNNs) and/or convolutional neural network (CNNs). However, the existing works have one significant weakness: fail to explicitly leverage the geospatial implications and latent connections among sensors for high‐quality data acquisition and quality control. Graphs are intrinsically suitable for representing the dependencies and inter‐relationships between AIoT data sensing devices. Due to the ability of capturing the complex interactive relationships between nodes and producing high‐level representations of the graph input, graph neural networks (GNNs) have exploded onto various ML and AI fields, to learn from graph‐structured data. Our review covers the latest progresses in GNN for the fundamental atomic task of data acquisition in AIoT. Instead of surveying the abundant GNN schemes in vertically various IoT sensing applications, this paper systematically reviews the horizontal infrastructure that all AIoT fields should have, that is, AIoT data acquisition, based on GNN and other related emerging AI factors. Our contributions include the following aspects: Provide the latest progresses in GNN for the horizontal task of data acquisition in AIoT, propose the unified GNN pipeline based on encoder–decoder paradigm, and systematically categorize and summarize the emerging technologies helpful to address the issues in AIoT data acquisition, especially the noisy and adversarial data, and point out some future directions about GNN‐based AIoT data acquisition.

show abstract

A study on the uncertainty of convolutional layers in deep neural networks

Cited by 8 publications

References 24 publications

Adversarial Learning with Cost-Sensitive Classes

Adversarial Learning with Cost-Sensitive Classes

A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications

Artificial intelligence of things (AIoT) data acquisition based on graph neural networks: A systematical review

Contact Info

Product

Resources

About