Deep Polynomial Neural Networks

Chrysos, Grigorios; Moschoglou, Stylianos; Bouritsas, Giorgos; Deng, Jiankang; Panagakis, Yannis; Zafeiriou, Stefanos

doi:10.1109/tpami.2021.3058891

Cited by 28 publications

(21 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then, depending on the interactions between the layers we want to forge, we can share the corresponding factor matrices. That results (see [164], [169] for detailed derivation) in a simple recursive relationship, that can be expressed as:…”

Section: F Tensor Structures In Polynomial Network and Attention Mechanismsmentioning

confidence: 99%

Tensor Methods in Computer Vision and Deep Learning

et al. 2021

Self Cite

View full text Add to dashboard Cite

Tensors, or multidimensional arrays, are data structures that can naturally represent visual data of multiple dimensions. Inherently able to efficiently capture structured, latent semantic spaces and high-order interactions, tensors have a long history of applications in a wide span of computer vision problems. With the advent of the deep learning paradigm shift in computer vision, tensors have become even more fundamental. Indeed, essential ingredients in modern deep learning architectures, such as convolutions and attention mechanisms, can readily be considered as tensor mappings. In effect, tensor methods are increasingly finding significant applications in deep learning, including the design of memory and compute efficient network architectures, improving robustness to random noise and adversarial attacks, and aiding the theoretical understanding of deep networks.This article provides an in-depth and practical review of tensors and tensor methods in the context of representation learning and deep learning, with a particular focus on visual data analysis and computer vision applications. Concretely, besides fundamental work in tensor-based visual data analysis methods, we focus on recent developments that have brought on a gradual increase of tensor methods, especially in deep learning architectures, and their implications in computer vision applications. To further enable the newcomer to grasp such concepts quickly, we provide companion Python notebooks, covering key aspects of the paper and implementing them, step-by-step with TensorLy.

show abstract

Section: F Tensor Structures In Polynomial Network and Attention Mechanismsmentioning

confidence: 99%

Tensor Methods in Computer Vision and Deep Learning

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recently, higher-order units were revisited [22]- [25]. In [22], a quadratic convolutional filter of the complexity O(n 2 ) was proposed to replace the linear filter, while in the work by Chrysos et al [23] the higher-order units as described by Eq. ( 3) were embedded into a deep network to reduce the complexity of the individual unit via tensor decomposition and factor sharing.…”

Section: Related Workmentioning

confidence: 99%

“…Such a network achieved cutting-edge performance on several tasks. Compared to [23], our group proposed a simplified quadratic neuron with O(3n) parameters and argued that more complicated neurons are not necessary based on algebraic fundamental theorem [26]. Interestingly, when only the first and second-order terms are kept, and the rank is set to two in tensor decomposition, the network of [23] becomes a special case of our quadratic model.…”

Section: Related Workmentioning

confidence: 99%

On Expressivity and Trainability of Quadratic Networks

Fan¹,

M²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Inspired by diversity of biological neurons, quadratic artificial neurons can play an important role in deep learning models. The type of quadratic neurons of our interest replaces the inner-product operation in the conventional neuron with a quadratic function. Despite promising results so far achieved by networks of quadratic neurons, there are important issues not well addressed. Theoretically, the superior expressivity of a quadratic network over either a conventional network or a conventional network via quadratic activation is not fully elucidated, which makes the use of quadratic networks not well grounded. Practically, although a quadratic network can be trained via generic backpropagation, it can be subject to a higher risk of collapse than the conventional counterpart. To address these issues, we first apply the spline theory and a measure from algebraic geometry to give two theorems that demonstrate better model expressivity of a quadratic network than the conventional counterpart with or without quadratic activation. Then, we propose an effective and efficient training strategy referred to as ReLinear to stabilize the training process of a quadratic network, thereby unleashing the full potential in its associated machine learning tasks. Comprehensive experiments on popular datasets are performed to support our findings and evaluate the performance of quadratic deep learning.

show abstract

“…In the machine learning domain, Simon revealed an interesting correspondence between the multiplicative neuron and the additive neuron, according to the identify

[ 47 ], and remarked that the multiplicative neuron network can be expressed in the form of an additive neuron network with a different nonlinearity [ 51 ]. At another venue, polynomial neural networks have been in progress, and their advantage over additive networks has been investigated [ 52 , 53 , 54 ]. Relations between polynomial regression and classical neural networks were discussed and polynomial activation functions were shown on the basis of the Taylor theorem [ 54 ].…”

Section: Introductionmentioning

confidence: 99%

An Evolutionary Field Theorem: Evolutionary Field Optimization in Training of Power-Weighted Multiplicative Neurons for Nitrogen Oxides-Sensitive Electronic Nose Applications

Alagöz

Simsek

Ari

et al. 2022

Sensors

View full text Add to dashboard Cite

Neuroevolutionary machine learning is an emerging topic in the evolutionary computation field and enables practical modeling solutions for data-driven engineering applications. Contributions of this study to the neuroevolutionary machine learning area are twofold: firstly, this study presents an evolutionary field theorem of search agents and suggests an algorithm for Evolutionary Field Optimization with Geometric Strategies (EFO-GS) on the basis of the evolutionary field theorem. The proposed EFO-GS algorithm benefits from a field-adapted differential crossover mechanism, a field-aware metamutation process to improve the evolutionary search quality. Secondly, the multiplicative neuron model is modified to develop Power-Weighted Multiplicative (PWM) neural models. The modified PWM neuron model involves the power-weighted multiplicative units similar to dendritic branches of biological neurons, and this neuron model can better represent polynomial nonlinearity and they can operate in the real-valued neuron mode, complex-valued neuron mode, and the mixed-mode. In this study, the EFO-GS algorithm is used for the training of the PWM neuron models to perform an efficient neuroevolutionary computation. Authors implement the proposed PWM neural processing with the EFO-GS in an electronic nose application to accurately estimate Nitrogen Oxides (NOx) pollutant concentrations from low-cost multi-sensor array measurements and demonstrate improvements in estimation performance.

show abstract

Deep Polynomial Neural Networks

Cited by 28 publications

References 64 publications

Tensor Methods in Computer Vision and Deep Learning

Tensor Methods in Computer Vision and Deep Learning

On Expressivity and Trainability of Quadratic Networks

An Evolutionary Field Theorem: Evolutionary Field Optimization in Training of Power-Weighted Multiplicative Neurons for Nitrogen Oxides-Sensitive Electronic Nose Applications

Contact Info

Product

Resources

About