Genetic Algorithm-Neural Network (GANN): a study of neural network activation functions and depth of genetic algorithm search applied to feature selection

Abstract

Hybrid genetic algorithms (GA) and artificial neural networks (ANN) are not new in the machine learning culture. Such hybrid systems have been shown to be very successful in classification and prediction problems. However, little attention has been focused on this architecture as a feature selection method and the consequent significance of the ANN activation function and the number of GA evaluations on the feature selection performance. The activation function is one of the core components of the ANN architecture and influences the learning and generalization capability of the network. Meanwhile the GA searches for an optimal ANN classifier given a set of chromosomes selected from those available. The objective of the GA is to combine the search for optimum chromosome choices with that of finding an optimum classifier for each choice. The process operates as a form of co-evolution with the eventual objective of finding an optimum chromosome selection rather than an optimum classifier. The selection of an optimum chromosome set is referred to in this paper as feature selection. Quantitative comparisons of four of the most commonly used ANN activation functions against ten GA evaluation step counts and three population sizes are presented. These studies employ four data sets with high dimension and low significant datum instances. That is to say that each datum has a high attribute count and the unusual or abnormal data are sparse within the data set. Results suggest that the hyperbolic tangent (tanh) activation function outperforms other common activation functions by extracting a smaller, but more significant feature set. Furthermore, it was found that fitness evaluation sizes ranging from 20,000 to 40,000 within populations ranging from 200 to 300, deliver optimum feature selection capability. Again, optimum in this sense meaning a smaller but more significant feature set.

Metaheuristic design of feedforward neural networks: A review of two decades of research

“…On the other hand, a matrix-based representation of weights, where a column-wise and a row-wise crossover operators were also defined [152]. The GA-based real coded weights optimization outperforms BP and its variants for solving real-world applications [153][154][155][156]. Moreover, an evolutionary inspired algorithm, called differential evolution (DE) [105,157] that imitates mutation and crossover operator to solve complex continuous optimization problems was found to be performing efficiently for real-valued weight vector optimization [158][159][160].…”

“…Primarily, node optimization can be addressed in three ways: 1) by choosing activation functions at the FNN active nodes from a set of activation functions [156,231]; 2) by optimizing the arguments of activation function [232]; and 3) by placing a complete model at the nodes of a network [233,234].…”

Colour image segmentation with histogram and homogeneity histogram difference using evolutionary algorithms

“…All these algorithms are well-known in the field of optimization and these algorithms show their applications in various fields of engineering where problems are much complex and multidimensional. Some more techniques based on neural network and genetic algorithms [52][53][54] also proposed in recent years. In same way colour images have a histogram in 3-D and storing this information in a 3-D array and selecting multilevel threshold value for a 3-D array is a complex task.…”

Eigenclassifiers for combining correlated classifiers

“…There are other measures such as the area under ROC curve (AUC) or F-measure (based on precision and recall); recently, Wang and Dong [44] applied fuzzy entropy maximization instead of training error minimization to refine fuzzy IF-THEN rules. Tong and Mitram [39] use genetic algorithms to choose between neural network activation functions and features to increase classifier performance. Wang [43] uses mutual information for feature selection in combining nearest neighbor classifiers using the fuzzy integral.…”