Enhancing and Relaxing Competitive Units for Feature Discovery

Abstract

In this paper, we propose a new information-theoretic method called enhancement and relaxation to discover main features in input patterns. We have so far shown that competitive learning is a process of mutual information maximization between input patterns and connection weights. However, because mutual information is an average over all input patterns and competitive units, it is not adequate for discovering detailed information on the roles of elements in a network. To extract information on the roles of elements in a networks, we enhance or relax competitive units through the elements. Mutual information should be changed by these processes. The change in information is called enhanced information. The enhanced information can be used to discover features in input patterns, because the information includes detailed information on elements in a network. We applied the method to the symmetry data, the well-known Iris problem and the OECD countries classification. In all cases, we succeeded in extracting the main features in input patterns.

Double enhancement learning for explicit internal representations: unifying self-enhancement and information enhancement to incorporate information on input variables

“…In addition, we have found that information on input variables is effective [2], [14] in producing an explicit class structure. Thus, we have incorporated the information on input variables into the framework of self-enhancement learning.…”

“…The information enhancement procedure is so simple that it can be defined for any component in a network. In our previous studies [2], competitive units were enhanced by using input units, competitive units, connection weights and input patterns. Thus, it is possible to train a network by taking into account the importance of any component in a network.…”

“…In this paper, we propose a new information-theoretic method called "double enhancement learning," in which self-enhancement learning [1] and information enhancement [2] are unified. The self-enhancement learning has been developed for networks to create targets for themselves.…”

“…2 The data set includes votes for each of the U.S. House of Representatives Congressmen on 16 key votes. The number of input patterns was 435.…”

Information-Theoretic Approach to Interpret Internal Representations of Self-Organizing Maps

“…In unsupervised learning, explicit evaluation functions have not been established for variable selection (Guyon & Elisseeff, 2003). We have introduced variable selection in unsupervised competitive learning by introducing a method of information loss (Kamimura, 2007;2008b;a) or information enhancement (Kamimura, 2008c;2009). In the information loss method, a specific input unit or variable is temporarily deleted, and the change in mutual information between competitive units and input patterns is measured.…”