Wrapper Maintenance: A Machine Learning Approach

Abstract

The proliferation of online information sources has led to an increased use of wrappers for extracting data from Web sources. While most of the previous research has focused on quick and efficient generation of wrappers, the development of tools for wrapper maintenance has received less attention. This is an important research problem because Web sources often change in ways that prevent the wrappers from extracting data correctly. We present an efficient algorithm that learns structural information about data from positive examples alone. We describe how this information can be used for two wrapper maintenance applications: wrapper verification and reinduction. The wrapper verification system detects when a wrapper is not extracting correct data, usually because the Web source has changed its format. The reinduction algorithm automatically recovers from changes in the Web source by identifying data on Web pages so that a new wrapper may be generated for this source. To validate our approach, we monitored 27 wrappers over a period of a year. The verification algorithm correctly discovered 35 of the 37 wrapper changes, and made 16 mistakes, resulting in precision of 0.73 and recall of 0.95. We validated the reinduction algorithm on ten Web sources. We were able to successfully reinduce the wrappers, obtaining precision and recall values of 0.90 and 0.80 on the data extraction task.

Data Driven XPath Generation

“…On the topic of wrapper breakage, which occurs when a wrapper can no longer retrieve data because the HTML document is changed structurally, there has been less research [13][14][15][16][17]. These approaches discuss ways to identify data in labeled documents, but do not offer a solution for text analysis or entity recognition.…”

Automatic Web Data Extraction Based on Genetic Algorithms and Regular Expressions

“…It is possible to use Machine Learning algorithms to learn automatically the wrappers [22,23,26,27,31]. The automatic wrapper induction [22,31] Each landmark automaton is specialized in extracting an attribute.…”

“…Maintaining wrappers is related to two different issues: on the one hand, to detect when a wrapper is not retrieving correctly the data (wrapper verification). On the other hand, to automatically recover the wrapper generating a new wrapper that takes into account the possible changes in the Web source (wrapper reinduction) [23,26,27]. …”

Mining for Information Discovery on the Web: Overview and Illustrative Research

“…Thus, efficient maintenance of Web information systems is a crucial task, which can benefit much from mining techniques. This topic has only recently received some attention, with several works on using learning techniques to detect and repair wrappers [54,59].…”