Learning block importance models for web pages

Abstract

Previous work shows that a web page can be partitioned into multiple segments or blocks, and often the importance of those blocks in a page is not equivalent. Also, it has been proven that differentiating noisy or unimportant blocks from pages can facilitate web mining, search and accessibility. However, no uniform approach and model has been presented to measure the importance of different segments in web pages. Through a user study, we found that people do have a consistent view about the importance of blocks in web pages. In this paper, we investigate how to find a model to automatically assign importance values to blocks in a web page. We define the block importance estimation as a learning problem. First, we use a vision-based page segmentation algorithm to partition a web page into semantic blocks with a hierarchical structure. Then spatial features (such as position and size) and content features (such as the number of images and links) are extracted to construct a feature vector for each block. Based on these features, learning algorithms are used to train a model to assign importance to different segments in the web page. In our experiments, the best model can achieve the performance with Micro-F1 79% and Micro-Accuracy 85.9%, which is quite close to a person's view.

A Quantitative Comparison of Semantic Web Page Segmentation Approaches

“…They test their algorithm experimentally by sampling 140 pages from different categories of the Yahoo directory and running their algorithm on it and then manually assessing whether the segmentation was "Perfect", "Satisfactory" or "Failed". Later work rates web page blocks according to their importance (Song et al 2004). (Baluja 2006) focuses on the application of optimizing existing web pages for mobile phones by, first, dividing the web page into a 3x3-grid.…”

Zero-shot Entity Extraction from Web Pages

“…To disfavor these parts, One possible solution is to add visual features that capture how the web page is rendered and favor more salient parts of the page. (Liu et al, 2003;Song et al, 2004;Zhu et al, 2005;Zheng et al, 2007).…”

Sketching-Din Elimination of Web Page

“…The point here is that, for a long page, the content in the first screen is most important and we should avoid normalizing them with the height of the whole page. Width normalization does not have the same problem since few pages have widths bigger than the screen (Song et al, 2004). Fig.…”

“…1, the caption in a news web site is much more attractive to users than the navigation bar. And users only just pay attention to the advertisement or the copyright when they browse a web page (Song et al, 2004).…”