Navigation objects extraction for better content structure understanding

Abstract

Existing works for extracting navigation objects from webpages focus on navigation menus, so as to reveal the information architecture of the site. However, web 2.0 sites such as social networks, e-commerce portals etc. are making the understanding of the content structure in a web site increasingly di cult. Dynamic and personalized elements such as top stories, recommended list in a webpage are vital to the understanding of the dynamic nature of web 2.0 sites. To be er understand the content structure in web 2.0 sites, in this paper we propose a new extraction method for navigation objects in a webpage. Our method will extract not only the static navigation menus, but also the dynamic and personalized page-speci c navigation lists. Since the navigation objects in a webpage naturally come in blocks, we rst cluster hyperlinks into di erent blocks by exploiting spatial locations of hyperlinks, the hierarchical structure of the DOM-tree and the hyperlink density.en we identify navigation objects from those blocks using the SVM classi er with novel features such as anchor text lengths etc. Experiments on real-world data sets with webpages from various domains and styles veri ed the e ectiveness of our method.

A Decade of Information Architecture in HCI: A Systematic Literature Review

“…Results of their prototype evaluation displayed a 78% error free navigation element detection and a 44% error free hierarchy detection. While both [31] and [32] structure extraction methods focus on static website, authors of [33] introduce a method that captures both the static and dynamic structure of a website.…”

Harvest -- An Open Source Toolkit for Extracting Posts and Post Metadata from Web Forums

“…Dragnet [12] combines several feature sets into a single library which has demonstrated good results on several early forum extraction datasets. Another recent approach has focused on navigating the hierarchy of objects extracted from web pages (e.g., hyperlink blocks extracted from DOM trees) [13].…”

“…Dragnet 10 in turn, provided the highest precision, and depending on the metric (e.g., cleaneval, euclidean or cosine distances, etc) either Dragnet or News-Please 11 yielded the best F1 score. For our experiments, we have selected several of the systems used in Barberesi's evaluation [24] including Inscriptis, BoilerPy3 12 , jusText 13 and Dragnet. Several other tools have been initially targeted but were not included since their source code was not available online at the date the experiments were performed (e.g., Sido's forum extraction tool [25]) or due to various errors (e.g., the News-Please tool).…”