Harvesting relational tables from lists on the web

Abstract

A large number of web pages contain data structured in the form of "lists". Many such lists can be further split into multi-column tables, which can then be used in more semantically meaningful tasks. However, harvesting relational tables from such lists can be a challenging task. The lists are manually generated and hence need not have well defined templates -they have inconsistent delimiters (if any) and often have missing information.We propose a novel technique for extracting tables from lists. The technique is domain-independent and operates in a fully unsupervised manner. We first use multiple sources of information to split individual lines into multiple fields, and then compare the splits across multiple lines to identify and fix incorrect splits and bad alignments. In particular, we exploit a corpus of HTML tables, also extracted from the Web, to identify likely fields and good alignments. For each extracted table, we compute an extraction score that reflects our confidence in the table's quality.We conducted an extensive experimental study using both real web lists and lists derived from tables on the Web. The experiments demonstrate the ability of our technique to extract tables with high accuracy. In addition, we applied our technique on a large sample of about 100,000 lists crawled from the Web. The analysis of the extracted tables have led us to believe that there are likely to be tens of millions of useful and query-able relational tables extractable from lists on the Web.

An efficient publish/subscribe index for e-commerce databases

“…In recent years, harvesting knowledge from the web [11,24,25,28] has attracted more and more attention. For example, Google's Freebase [1] has collected and published more than 39 million real world entities, with more than 140, 000 attributes.…”

Extraction and integration of partially overlapping web sources

“…One distinguishable feature of our work is the ability to gather and leverage domain knowledge at runtime to automatically tune the integration process. The massive exploitation of the structured Web has been studied for data published in HTML tables and lists [10,20]. However, these works focus on the extraction of rich relational schemas, without addressing the issue of integrating the extracted data.…”

An analysis of structured data on the web

“…Traditional IE techniques considered in the database community tend to be source-centric, i.e., they can only be deployed to extract from a specific website or data source. However, a range of domain-independent techniques have emerged recently [2,4,9,10,11,16,20] that seek to look at extraction holistically on the entire Web.…”

“…There are some domain-independent efforts, e.g. WebTables [4,10], that extract all simple tables and lists from the Web and store them as relational data. However, domain-independence makes it difficult to attach semantics to the extracted data.…”