Hierarchical Temporal Memory (HTM) is an unsupervised algorithm in machine learning. It models several fundamental neocortical computational principles. Spatial Pooler (SP) is one of the main components of the HTM, which continuously encodes streams of binary input from various layers and regions into sparse distributed representations. In this paper, the goal is to evaluate the sparsification in the SP algorithm from the perspective of information theory by the information bottleneck (IB), Cramer-Rao lower bound, and Fisher information matrix. This paper makes two main contributions. First, we introduce a new upper bound for the standard information bottleneck relation, which we refer to as modified-IB in this paper. This measure is used to evaluate the performance of the SP algorithm in different sparsity levels and various amounts of noise. The MNIST, Fashion-MNIST and NYC-Taxi datasets were fed to the SP algorithm separately. The SP algorithm with learning was found to be resistant to noise. Adding up to 40% noise to the input resulted in no discernible change in the output. Using the probabilistic mapping method and Hidden Markov Model, the sparse SP output representation was reconstructed in the input space. In the modified-IB relation, it is numerically calculated that a lower noise level and a higher sparsity level in the SP algorithm lead to a more effective reconstruction and SP with 2% sparsity produces the best results. Our second contribution is to prove mathematically that more sparsity leads to better performance of the SP algorithm. The data distribution was considered the Cauchy distribution, and the Cramer–Rao lower bound was analyzed to estimate SP’s output at different sparsity levels.
This paper provided an information-theoretic framework for the performance comparison of different types of HTM-Spatial Pooler (SP) algorithms for the first time. There are two primary goals. The first goal is to measure SP's performance as a standalone component, and the second goal is to compute the SP performance on the whole performance of the HTM algorithm. For this purpose, four different SPs were introduced by making changes in each of the four parts of the SP algorithm. The SP algorithm was considered a black box. Three information-theoretic measures, i.e., Renyi mutual information, Renyi divergence, and Henze-Penrose divergence, were proposed to determine the similarities and differences between the input and output of the SP algorithm. So the accuracy of each method was computed by these information-theoretic measures. This paper was able to quantify the similarities and differences between the two SDRs in the SP algorithm, unlike previous papers that do experimentally. Then, the results were compared with the Overlap Score measure that was previously introduced and found that the results of the new measures were compatible with the previous one. Four different datasets: MNIST, Fashion-MNIST, Hotgym, and NYC-Taxi, were used to perform experiments. The performance of each SP algorithm in the whole HTM system was examined and analyzed using the accuracy and error percentage measures. Finally, it concluded that the best SP algorithm's efficiency leads to the best HTM algorithm.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.