Neural networks, despite their empirically proven abilities, have been little used for the refinement of existing knowledge because this task requires a three-step process. First, knowledge must be inserted into a neural network. Second, the network must be refined. Third, the refined knowledge must be extracted from the network. We have previously described a method for the first step of this process. Standard neural learning techniques can accomplish the second step. In this article, we propose and empirically evaluate a method for the final, and possibly most difficult, step. Our method efficiently extracts symbolic rules from trained neural networks. The four major results of empirical tests of this method are that the extracted rules 1) closely reproduce the accuracy of the network from which they are extracted; 2) are superior to the rules produced by methods that directly refine symbolic rules; 3) are superior to those produced by previous techniques for extracting rules from trained neural networks; and 4) are "human comprehensible" Thus, this method demonstrates that neural networks can be used to effectively refine symbolic knowledge. Moreover, the rule-extraction technique developed herein contributes to the understanding of how symbolic and connectionist approaches to artificial intelligence can be profitably integrated.Keywords. theory refinement, integrated learning, representational shift, rule extraction from neural networks networks created using KBANN will be referred to as Knowledge-based Neural Networks--KNNs.) This step changes the representation of the rules from symbolic to neurally based, thereby making the rules refinable by standard neural learning methods.The second link of the chain is to train the KNN using a set of classified training exampies and a standard neural learning algorithm, backpropagation (Rumelhart et al., 1986) or any other method for weight optimization of feedforward neural networks. In so doing, the rules upon which the KNN are based are corrected so that they are consistent with the training examples.The final link is to extract rules from trained KNNs. This is an extremely difficult task for arbitrarily configured networks, but it is somewhat less daunting for KNNs due to properties that stem from their initial comprehensibility. Taking advantage of these properties, we developed a method to efficiently extract intelligible rules from networks. When evaluated on two real-world test problems in terms of the ability to correctly classify examples not seen during training, the method produces sets of rules that closely approximate the networks from which they came. Moreover, the extracted rules are equal, or superior, to the rules resulting from rule-refinement methods that act directly on the rules, rather than their re-representation as a neural network. We also show that our method is superior to a previous algorithm for the extraction of rules from general neural networks (e.g., Saito & Nakano, 1988;Fu, 1991).The next section contains a brief overview of our metho...
Abstract. Despite the fact that many symbolic and neural network (connectionist) learning algorithms address the same problem of learning from classified examples, very little is known regarding their comparative strengths and weaknesses. Experiments comparing the ID3 symbolic learning algorithm with the perceptron and backpropagation neural learning algorithms have been performed using five large, real-world data sets, Overall, backpropagation performs slightly better than the other two algorithms in terms of classification accuracy on new examples, but takes much longer to train. Experimental results suggest that backpropagation can work significantly better on data sets containing numerical data. Also analyzed empirically are the effects of (1) the amount of training data, (2) imperfect training examples, and (3) the encoding of the desired outputs. Backpropagation occasionally outperforms the other two systems when given relatively small amounts of training data. It is slightly more accurate than ID3 when examples are noisy or incompletely specified. Finally, backpropagation more effectively utilizes a "distributed" output encoding.
The three corpus-based statistical sense resolution methods studied here attempt to infer the correct sense of a polysemous word by using knowledge about patterns of word cooccurrences. The techniques were based on Bayesian decision theory, neural networks, and content vectors as used in information retrieval. To understand these methods better, we posed s very specific problem: given a set of contexts, each containing the noun line in a known sense, construct a classifier that selects the correct sense of line for new contexts. To see how the degree of polysemy affects performance, results from three-and slx-sense tasks are compared.The results demonstrate that each of the techniques is able to distinguish six senses of line with an accuracy greater than 70%. Furthermore, the response patterns of the classifiers are, for the most part, statistically indistinguishable from one another. Comparison of the two tasks suggests that the degree of difficulty involved in resolving individual senses is a greater performance factor than the degree of polysemy.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.