Deep Learning of Part-Based Representation of Data Using Sparse Autoencoders With Nonnegativity Constraints

Abstract

Abstract-We demonstrate a new deep learning autoencoder network, trained by a nonnegativity constraint algorithm (NCAE), that learns features which show part-based representation of data. The learning algorithm is based on constraining negative weights. The performance of the algorithm is assessed based on decomposing data into parts and its prediction performance is tested on three standard image data sets and one text dataset. The results indicate that the nonnegativity constraint forces the autoencoder to learn features that amount to a part-based representation of data, while improving sparsity and reconstruction quality in comparison with the traditional sparse autoencoder and Nonnegative Matrix Factorization. It is also shown that this newly acquired representation improves the prediction performance of a deep neural network.

Energy Management in Microgrids: A Combination of Game Theory and Big Data‐Based Wind Power Forecasting

“…Renewable energy generation Stochastic optimization Handling date uncertainties of renewable energy [10][11][12] Robust optimization [14][15][16][17] Wind power forecasting Linear methods Increasing the accuracy of prediction model [19,20] Nonlinear methods [24][25][26][27] Microgrid management Ordinary decision theory Optimizing energy-scheduling strategies [28][29][30] Noncooperative games [33][34][35][36] Cooperative games [37][38][39][40] and robust optimization [9]. On the one hand, stochastic optimization provides an effective framework to optimize statistical objective functions while the uncertain numerical data are assumed to follow a proverbial probability distribution.…”

“…To efficiently handle the complex, unlabeled and high-dimensional time series data, deep learning has been proposed in Ref. [26]. As an essential deep learning architecture, SAE plays a fundamental role in unsupervised learning and the objective function can be solved efficiently via fast back propagation [27].…”

Building Energy Consumption Prediction: An Extreme Deep Learning Approach

“…The autoencoder will be invalid for dimension reduction and key feature extraction if the number of hidden nodes is the same or greater than the number of input nodes, that is, L ≥ n. To solve this problem, sparsity constraints are imposed on the hidden layer to obtain the representative features and to learn useful structures from the input data [36][37][38][39]. This allows for sparse representations of inputs and is useful for pre-training in many tasks.…”

“…We minimize the following loss function, which imposes a sparsity constraint on the reconstruction error to obtain the optimal parameters of the sparsity autoencoder [36][37][38][39]:…”

“…is the average activation of the jth hidden node with respect to the training set, and KL(ρ ρ j ) is the Kullback-Leibler divergence, which is also called the relative entropy and is defined as follows [36][37][38][39]:…”