Harvesting relational tables from lists on the web

Abstract

A large number of web pages contain data structured in the form of "lists". Many such lists can be further split into multi-column tables, which can then be used in more semantically meaningful tasks. However, harvesting relational tables from such lists can be a challenging task. The lists are manually generated and hence need not have well defined templates -they have inconsistent delimiters (if any) and often have missing information.We propose a novel technique for extracting tables from lists. The technique is domain-independent and operates in a fully unsupervised manner. We first use multiple sources of information to split individual lines into multiple fields, and then compare the splits across multiple lines to identify and fix incorrect splits and bad alignments. In particular, we exploit a corpus of HTML tables, also extracted from the Web, to identify likely fields and good alignments. For each extracted table, we compute an extraction score that reflects our confidence in the table's quality.We conducted an extensive experimental study using both real web lists and lists derived from tables on the Web. The experiments demonstrate the ability of our technique to extract tables with high accuracy. In addition, we applied our technique on a large sample of about 100,000 lists crawled from the Web. The analysis of the extracted tables have led us to believe that there are likely to be tens of millions of useful and query-able relational tables extractable from lists on the Web.

On extracting data from tables that are encoded using HTML

“…It is relatively easy to extract data from tables that are encoded using the previous tags. Unfortunately, real-world tables have a variety of intricacies that hamper the extraction process, namely: some tables are encoded using a subset of table-related tags that hardly help locate them and their cells, which does not help interpret them; other tables are encoded using listing tags (ul, ol, dl, li, dd, and dt) [9,20,36,37]; lately, it is also relatively common to find tables that are encoded using block tags (div and span) due to their ability to create responsive layouts [50]; and, generally, speaking, there are many tables that are encoded using a variety of tags that are not actually related to tables, but look like tables when they are displayed [24,26].…”

“…Unfortunately, roughly 72% of the authors did not report on the effectiveness of their proposals; the others reported on precision, recall, and/or the F 1 score. Only Elmeleegy et al [20] and Chu et al [9] reported on the efficiency of their approaches; their figures reveal that the algorithms behind the scenes might not be scalable enough. Regarding the resources required, only the proposals by Elmeleegy et al [20] and Ling et al [39] require the user to provide a few, but they do not seem to be difficult to find.…”

“…Only Elmeleegy et al [20] and Chu et al [9] reported on the efficiency of their approaches; their figures reveal that the algorithms behind the scenes might not be scalable enough. Regarding the resources required, only the proposals by Elmeleegy et al [20] and Ling et al [39] require the user to provide a few, but they do not seem to be difficult to find. Only the proposals by Lerman et al [36,37] and Gatterbauer et al [26] require to project the input tables onto some simple features.…”

“…Only the proposals by Lerman et al [36,37] and Gatterbauer et al [26] require to project the input tables onto some simple features. Regarding the parameters, only the proposals by Elmeleegy et al [20], Ling et al [39], and Chu et al [9] need the users to set a few.…”

“…In this context, data extraction consists in transforming tables into structured formats that focus on their data and abstract away from how they are displayed. Data extraction has many applications to text mining [24,64,65], data (meta-)search [3,9,18,26,44,51,[63][64][65], query expansion [16], document summarisation [40,64], question answering [1,20,44,46,65], knowledge discovery [9,22,26,32,44,46], knowledge base construction [17,72], knowledge augmentation [1,9,18,20,56,56,57,67], synonym finding [1,3,39], improving accessibility [43,47,49,64,65], textual advertising [15], data compression [2,49], or creating linked data…”