Automatic web news extraction using tree edit distance

Abstract

The Web poses itself as the largest data repository ever available in the history of humankind. Major efforts have been made in order to provide efficient access to relevant information within this huge repository of data. Although several techniques have been developed to the problem of Web data extraction, their use is still not spread, mostly because of the need for high human intervention and the low quality of the extraction results. In this paper, we present a domain-oriented approach to Web data extraction and discuss its application to automatically extracting news from Web sites. Our approach is based on a highly efficient tree structure analysis that produces very effective results. We have tested our approach with several important Brazilian on-line news sites and achieved very precise results, correctly extracting 87.71% of the news in a set of 4088 pages distributed among 35 different sites.

Data Driven XPath Generation

“…Alternatively, there are fully automated techniques [7][8][9], which require a single URL as input and aim to return a set of objects extracted from the content. Though these tools are considered to be easier to use, their output generally requires more attention than that of semi-automated solutions.…”

“…Most of these rely on the DOM tree representation, also called tag tree, of the document [4,5,[7][8][9][10][11], though some also look at the way the data is visualized [3,12] using optical recognition software. Such techniques often focus on frequency and pattern analysis of subtrees to estimate where data of interest can be found.…”

“…In sections 7 and 8, we explain how we can use these XPaths to configure a new wrapper. In section 9 we present a use case to measure the accuracy of our research. In Section 10, we indicate what might still improve our work and finally, in Section 11 we present our final conclusions.…”

XML document-grammar comparison: related problems and applications

“…It has also been investigated in the context of Web data extraction, using structural patterns with special VLDC symbols (substituting single nodes, sets of siblings, and/or sub-trees) to identify data-rich information in Web document [68]. Nonetheless, despite being comparable, VLDC symbols are fairly different from repeatability and alternativeness operators in XML grammars (cf.…”

“…Clustering can also be critical in information extraction. Current information extraction methods either implicitly or explicitly depend on the structural features of documents [17,68]. Structural clustering allows to automatically identify the sets of XML documents and/or document patterns that are useful in information extraction algorithms, in order to produce meaningful results [68].…”