Improving the Performance of Neural Networks with an Ensemble of Activation Functions

Nandi, Arijit; Jana, Nanda Dulal; Das, Swagatam

doi:10.1109/ijcnn48605.2020.9207277

Cited by 10 publications

(5 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A family

of functions

is parameterized by weight vector w . The function

minimizes the loss or error

averaged on the training samples [ 33 ].

…”

Section: Methodsmentioning

confidence: 99%

“…An objective function L(ŷ, y) is considered to measure the difference between neural network predicted class labels (considered aŝ y) and the actual class labels y. A family F of functions f w (x) is parameterized by weight vector w. The function f ∈ F minimizes the loss or error Q(D, w) = L( f w (x), y) averaged on the training samples [33].…”

Section: Stochastic Gradient Descentmentioning

confidence: 99%

See 1 more Smart Citation

Real-Time Emotion Classification Using EEG Data Stream in E-Learning Contexts

Nandi

Xhafa

Subirats

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

In face-to-face and online learning, emotions and emotional intelligence have an influence and play an essential role. Learners’ emotions are crucial for e-learning system because they promote or restrain the learning. Many researchers have investigated the impacts of emotions in enhancing and maximizing e-learning outcomes. Several machine learning and deep learning approaches have also been proposed to achieve this goal. All such approaches are suitable for an offline mode, where the data for emotion classification are stored and can be accessed infinitely. However, these offline mode approaches are inappropriate for real-time emotion classification when the data are coming in a continuous stream and data can be seen to the model at once only. We also need real-time responses according to the emotional state. For this, we propose a real-time emotion classification system (RECS)-based Logistic Regression (LR) trained in an online fashion using the Stochastic Gradient Descent (SGD) algorithm. The proposed RECS is capable of classifying emotions in real-time by training the model in an online fashion using an EEG signal stream. To validate the performance of RECS, we have used the DEAP data set, which is the most widely used benchmark data set for emotion classification. The results show that the proposed approach can effectively classify emotions in real-time from the EEG data stream, which achieved a better accuracy and F1-score than other offline and online approaches. The developed real-time emotion classification system is analyzed in an e-learning context scenario.

show abstract

“…A family

of functions

is parameterized by weight vector w . The function

minimizes the loss or error

averaged on the training samples [ 33 ].

…”

Section: Methodsmentioning

confidence: 99%

Section: Stochastic Gradient Descentmentioning

confidence: 99%

Real-Time Emotion Classification Using EEG Data Stream in E-Learning Contexts

Nandi

Xhafa

Subirats

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the classifiers performance evaluation, accuracy (Acc) and F 1 micro score are used. The metrics are calculated as follows [41]:…”

Section: Performance Metricsmentioning

confidence: 99%

A Docker-based federated learning framework design and deployment for multi-modal data stream classification

2023

Self Cite

View full text Add to dashboard Cite

In the high-performance computing (HPC) domain, federated learning has gained immense popularity. Especially in emotional and physical health analytics and experimental facilities. Federated learning is one of the most promising distributed machine learning frameworks because it supports data privacy and security by not sharing the clients' data but instead sharing their local models. In federated learning, many clients explicitly train their Machine Learning/Deep Learning models (local training) before aggregating them as a global model at the global server. However, the FL framework is difficult to build and deploy across multiple distributed clients due to its heterogeneous nature. We developed Docker-enabled Federated Learning (DFL) by utilizing client-agnostic technologies like Docker containers to simplify the deployment of FL frameworks for data stream processing on the heterogeneous client. In the DFL, the clients and global servers are written using TensorFlow and lightweight Message Queuing Telemetry Transport (MQTT) protocol to communicate between clients and global servers in the IoT environment. Furthermore, the DFL's effectiveness, efficiency, and scalability are evaluated in the test case scenario where real-time emotion state classification is done from distributed multimodal physiological data streams under various practical configurations.

show abstract

“…They showed that an ensemble of multiple CNNs that only differed in the AFs outperformed the results of the single CNNs and of naive ensembles made of ReLU networks. [17] proposed AF ensembling by majority voting, wherein a single neural network is trained with five different AFs. They showed that model accuracy could be improved for four datasets-MNIST, Fashion MNIST, Semeion, and ARDIS IV-over methods including CNN, RNN (Recurrent Neural Network), and SVM (Support Vector Machine).…”

Section: Activation Functions: Previous Workmentioning

confidence: 99%

Neural Networks with À La Carte Selection of Activation Functions

Sipper

2021

SN COMPUT. SCI.

View full text Add to dashboard Cite

Activation functions (AFs), which are pivotal to the success (or failure) of a neural network, have received increased attention in recent years, with researchers seeking to design novel AFs that improve some aspect of network performance. In this paper we take another direction, wherein we combine a slew of known AFs into successful architectures, proposing three methods to do so beneficially: (1) generate AF architectures at random, (2) use Optuna, an automatic hyper-parameter optimization software framework, with a Tree-structured Parzen Estimator (TPE) sampler, and (3) use Optuna with a Covariance Matrix Adaptation Evolution Strategy (CMA-ES) sampler. We show that all methods often produce significantly better results for 25 classification problems when compared with a standard network composed of ReLU hidden units and a softmax output unit. Optuna with the TPE sampler emerged as the best AF architecture-producing method.

show abstract

Improving the Performance of Neural Networks with an Ensemble of Activation Functions

Cited by 10 publications

References 22 publications

Real-Time Emotion Classification Using EEG Data Stream in E-Learning Contexts

Real-Time Emotion Classification Using EEG Data Stream in E-Learning Contexts

A Docker-based federated learning framework design and deployment for multi-modal data stream classification

Neural Networks with À La Carte Selection of Activation Functions

Contact Info

Product

Resources

About