Greedy information acquisition algorithm: A new information theoretic approach to dynamic information acquisition in neural networks

Abstract

In this paper, we propose a new information theoretic approach to competitive learning. The new approach is called greedy information acquisition, because networks try to absorb as much information as possible in every stage of learning. In the first phase, with minimum network architecture for realizing competition, information is maximized. In the second phase, a new unit is added, and thereby information is again increased as much as possible. This process continues until no more increase in information is possible. Through greedy information maximization, different sets of important features in input patterns can be cumulatively discovered in successive stages. We applied our approach to three problems: a dipole problem; a language classification problem; and a phonological feature detection problem. Experimental results confirmed that information maximization can be repeatedly applied and that different features in input patterns are gradually discovered. We also compared our method with conventional competitive learning and multivariate analysis. The experimental results con rmed that our new method can detect salient features in input patterns more clearly than the other methods.

Structural enhanced information and its application to improved visualization of self-organizing maps

“…However, as already mentioned in the introduction section, we have had the underutilized problem, and many methods have been proposed to overcome this problem [20][21][22][23][24][25]. To overcome this problem and to realize the softer type of competitive learning, we have introduced mutual information between input patterns and competitive units in competitive learning [15][16][17][18][19]. The use of mutual information for competitive learning is due to the fact that there is a strong similarity between competitive learning and mutual information maximization.…”

“…We have also introduced information-theoretic methods in unsupervised competitive learning [15][16][17][18][19]. They are based upon discrete cases, and it is easy to define mutual information, because mutual information is directly computed by using competitive unit activations.…”

“…Despite the many proposed methods, to the best of our knowledge, the problem remains unsolved. Our mutual information maximization is directly connected to conventional competitive learning and has been developed to overcome this problem [15][16][17][18][19]. When mutual information is completely maximized, just one neuron fires, while all the other neuron cease to fire.…”

Double enhancement learning for explicit internal representations: unifying self-enhancement and information enhancement to incorporate information on input variables

“…Much attention has so far been paid to the improvement of classification performance, and little has been paid, from our point of view, to the interpretation of the configuration of the network. For this interpretation, we have introduced information-theoretic competitive learning [25,[32][33][34] in which competitive processes are supposed to be equivalent to information maximization ones. In our informationtheoretic methods, a competitive unit output has been defined by the Gaussian-like function…”

Enhancing and Relaxing Competitive Units for Feature Discovery

“…In other words, competitive learning is only one aspect of information maximization in neural networks. We have so far proposed information-theoretic competitive learning [10,16,17,12] in which competitive processes are supposed to be equivalent to information maximization processes. When mutual information between input patterns and connection weights is maximized, only one competitive unit is active, while all the other units are inactive.…”