Using the structure of Web sites for automatic segmentation of tables

Abstract

Many Web sites, especially those that dynamically generate HTML pages to display the results of a user's query, present information in the form of list or tables. Current tools that allow applications to programmatically extract this information rely heavily on user input, often in the form of labeled extracted records. The sheer size and rate of growth of the Web make any solution that relies primarily on user input is infeasible in the long term. Fortunately, many Web sites contain much explicit and implicit structure, both in layout and content, that we can exploit for the purpose of information extraction. This paper describes an approach to automatic extraction and segmentation of records from Web tables. Automatic methods do not require any user input, but rely solely on the layout and content of the Web source. Our approach relies on the common structure of many Web sites, which present information as a list or a table, with a link in each entry leading to a detail page containing additional information about that item. We describe two algorithms that use redundancies in the content of table and detail pages to aid in information extraction. The first algorithm encodes additional information provided by detail pages as constraints and finds the segmentation by solving a constraint satisfaction problem. The second algorithm uses probabilistic inference to find the record segmentation. We show how each approach can exploit the web site structure in a general, domain-independent manner, and we demonstrate the effectiveness of each algorithm on a set of twelve Web sites.

Automatic Web Data Extraction Based on Genetic Algorithms and Regular Expressions

“…It is possible to use Machine Learning algorithms to learn automatically the wrappers [22,23,26,27,31]. The automatic wrapper induction [22,31] Each landmark automaton is specialized in extracting an attribute.…”

“…Maintaining wrappers is related to two different issues: on the one hand, to detect when a wrapper is not retrieving correctly the data (wrapper verification). On the other hand, to automatically recover the wrapper generating a new wrapper that takes into account the possible changes in the Web source (wrapper reinduction) [23,26,27]. …”

NET – A System for Extracting Web Data from Flat and Nested Data Records

“…For automatic extraction, [1,4,6] find patterns or grammars from multiple pages containing similar data records. Requiring an initial set of pages containing similar data records is, however, a limitation.…”

“…Requiring an initial set of pages containing similar data records is, however, a limitation. [6] proposes a method that tries to explore the detailed information pages behind the current page to segment data records. The need for such detailed pages behind is a drawback because many data records do not have such pages or such pages are hard to find.…”

Efficiently Locating Collections of Web Pages to Wrap

“…Motivated by this observation, recently several researchers have studied techniques that exploit similarities and differences among pages generated by the same script in order to automatically infer a Web wrapper, i.e. a program to extract and organize in a structured format data from HTML pages (Arasu and Garcia-Molina, 2003;Crescenzi et al, 2001;Crescenzi and Mecca, 2004;Lerman et al, 2004;Wang and Lochovsky, 2002). Based on these techniques they have developed tools that, given a set of pages sharing the same structure, infer a wrapper, which can be used in order to extract the data from all pages conforming to that structure.…”