Coupled Bayesian Sets Algorithm for Semi-supervised Learning and Information Extraction

Abstract

Our inspiration comes from Nell (Never Ending Language Learning), a computer program running at Carnegie Mellon University to extract structured information from unstructured web pages. We consider the problem of semi-supervised learning approach to extract category instances (e.g. country(USA), city(New York)) from web pages, starting with a handful of labeled training examples of each category or relation, plus hundreds of millions of unlabeled web documents. Semisupervised approaches using a small number of labeled examples together with many unlabeled examples are often unreliable as they frequently produce an internally consistent, but nevertheless, incorrect set of extractions. We believe that this problem can be overcome by simultaneously learning independent classifiers in a new approach named Coupled Bayesian Sets algorithm, based on Bayesian Sets, for many different categories and relations (in the presence of an ontology defining constraints that couple the training of these classifiers). Experimental results show that simultaneously learning a coupled collection of classifiers for random 11 categories resulted in much more accurate extractions than training classifiers through original Bayesian Sets algorithm, Naive Bayes, BaS-all and Coupled Pattern Learner (the category extractor used in NELL).

Learning relational facts from the web: A tolerance rough set approach

“…CBS algorithm by Verma and Hruschka [11] also addresses the problem of extracting noun phrases to populate category instances. Likewise, it adapts a semi-supervised approach and it uses the co-occurrence statistics between nouns and contexts.…”

“…Those candidates are then ranked and the uppermost ones are promoted to be the trusted category instances, which will then be used as seeds in the following iterations. An in-depth discussion on Bayesian Sets and Coupled Bayesian Sets is available in [11].…”

“…One major challenge posed by semi-supervised learning is that using a small number of labeled examples along with many unlabeled ones are often unreliable as they frequently produce an internally consistent, but nevertheless, incorrect set of extractions [11]. Even though semi-supervised approaches are promising, they might exhibit low accuracy, owing to the fact that initial labeled examples are limited in number and tend to be insufficient to properly constrain the learning act.…”

“…Even though semi-supervised approaches are promising, they might exhibit low accuracy, owing to the fact that initial labeled examples are limited in number and tend to be insufficient to properly constrain the learning act. This phenomenon is called "semantic (concept) drift" [12] and it is addressed by Verma and Hruschka [11] through learning independent classifiers simultaneously, in a new approach named the Coupled Bayesian Sets (CBS) algorithm. CBS outperforms CPL after 10 iterations, nominating itself as a good alternative for the free text extractor of Nell.…”

Growing a list

“…Systems for learning categories and relations of entities on the web, like the Never-Ending Language Learner (NELL) system (Carlson et al 2010a,b;Verma and Hruschka 2012), or KnowItAll (Etzioni et al 2005) can be used to construct lists but require extensive preprocessing. We do not preprocess, instead we perform information extraction online, deterministically, and virtually instantaneously given access to a search engine.…”