Hidden information maximization for feature detection and rule discovery

Kamimura, Ryotaro; Nakanishi, Satoshi

doi:10.1088/0954-898x/6/4/004

Cited by 50 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kamimura et al [ 19 ] examine the entropy of hidden unit activation patterns in autoencoders. Autoencoders produce efficient encodings of data that minimize information loss.…”

Section: Discussion: Balancing Entropymentioning

confidence: 99%

Dynamic Topology Reconfiguration of Boltzmann Machines on Quantum Annealers

Liu

Yao

Spedalieri

2020

Entropy

View full text Add to dashboard Cite

Boltzmann machines have useful roles in deep learning applications, such as generative data modeling, initializing weights for other types of networks, or extracting efficient representations from high-dimensional data. Most Boltzmann machines use restricted topologies that exclude looping connectivity, as such connectivity creates complex distributions that are difficult to sample. We have used an open-system quantum annealer to sample from complex distributions and implement Boltzmann machines with looping connectivity. Further, we have created policies mapping Boltzmann machine variables to the quantum bits of an annealer. These policies, based on correlation and entropy metrics, dynamically reconfigure the topology of Boltzmann machines during training and improve performance.

show abstract

“…Kamimura et al [ 19 ] examine the entropy of hidden unit activation patterns in autoencoders. Autoencoders produce efficient encodings of data that minimize information loss.…”

Section: Discussion: Balancing Entropymentioning

confidence: 99%

Dynamic Topology Reconfiguration of Boltzmann Machines on Quantum Annealers

Liu

Yao

Spedalieri

2020

Entropy

View full text Add to dashboard Cite

show abstract

“…We developed the information-theoretic methods to increase information content in hidden neurons on input patterns. We have so far succeeded in increasing the information content to a large quantity [5], [6], [7]. However, the method was limited to networks with a relatively smaller number of hidden neurons because of the computational complexity of the information method.…”

Section: Simplified Information Maximizationmentioning

confidence: 99%

“…This means that neural networks try to maximize information content in every information processing stage. Following Linsker's information principle, we developed information theoretic methods to control the quantity of information on input patterns [5], [6], [7]. We have so far succeeded in increasing information content, keeping training errors between targets and outputs relatively small.…”

Section: Introductionmentioning

confidence: 99%

Simplified Information Acquisition Method to Improve Prediction Performance: Direct Use of Hidden Neuron Outputs and Separation of Information Acquisition and Use Phase

Kamimura

2014

Proceedings of the International Workshop on Artificial Neural Networks and Intelligent Information Processing

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a new type of information-theoretic method to improve prediction performance in supervised learning with two main technical features. First, the complicated procedures to increase information content is replaced by the direct use of hidden neuron outputs. We realize higher information by directly changing the outputs from hidden neurons. In addition, we have had difficulty in increasing information content and at the same time decreasing errors between targets and outputs. To cope with this problem, we separate information acquisition and use phase learning. In the information acquisition phase, the auto-encoder tries to acquire information content on input patterns as much as possible. In the information use phase, information obtained in the phase of information acquisition is used to train supervised learning. The method is a simplified version of actual information maximization and directly deals with the outputs from neurons. We applied the method to the protein classification problem. Experimental results showed that our simplified information acquisition method was effective in increasing the real information content. In addition, by using the information content, prediction performance was greatly improved.

show abstract

“…This corresponds to a case of dead neurons in conventional competitive learning, and our previous method of entropy minimization cannot overcome this problem (Kamimura and Nakanishi 1995). Then, information is increased and reaches a final maximum information state in which only one competitive unit is turned on, while all the other units are close to off.…”

Section: Competition Processmentioning

confidence: 99%

Greedy information acquisition algorithm: A new information theoretic approach to dynamic information acquisition in neural networks

Kamimura¹,

Kamimura²,

Takeuchi³

2002

Connection Science

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a new information theoretic approach to competitive learning. The new approach is called greedy information acquisition, because networks try to absorb as much information as possible in every stage of learning. In the first phase, with minimum network architecture for realizing competition, information is maximized. In the second phase, a new unit is added, and thereby information is again increased as much as possible. This process continues until no more increase in information is possible. Through greedy information maximization, different sets of important features in input patterns can be cumulatively discovered in successive stages. We applied our approach to three problems: a dipole problem; a language classification problem; and a phonological feature detection problem. Experimental results confirmed that information maximization can be repeatedly applied and that different features in input patterns are gradually discovered. We also compared our method with conventional competitive learning and multivariate analysis. The experimental results con rmed that our new method can detect salient features in input patterns more clearly than the other methods.

show abstract

Hidden information maximization for feature detection and rule discovery

Cited by 50 publications

References 14 publications

Dynamic Topology Reconfiguration of Boltzmann Machines on Quantum Annealers

Dynamic Topology Reconfiguration of Boltzmann Machines on Quantum Annealers

Simplified Information Acquisition Method to Improve Prediction Performance: Direct Use of Hidden Neuron Outputs and Separation of Information Acquisition and Use Phase

Greedy information acquisition algorithm: A new information theoretic approach to dynamic information acquisition in neural networks

Contact Info

Product

Resources

About