Template detection for large scale search engines

Chen, Liang; Ye, Shaozhi; Li, Xing

doi:10.1145/1141277.1141534

Cited by 33 publications

(19 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly to Bar-Yossef, Liang Chen et al [6] is based on the identification of similar blocks (pagelets). The difference is that the whole procedure is done during the index building of a search engine.…”

Section: Related Workmentioning

confidence: 99%

On Finding Templates on Web Collections

et al. 2009

View full text Add to dashboard Cite

Templates are pieces of HTML code common to a set of web pages usually adopted by content providers to enhance the uniformity of layout and navigation of theirs Web sites. They are usually generated using authoring/publishing tools or by programs that build HTML pages to publish content from a database. In spite of their usefulness, the content of templates can negatively affect the quality of results produced by systems that automatically process information available in web sites, such as search engines, clustering and automatic categorization programs. Further, the information available in templates is redundant and thus processing and storing such information just once for a set of pages may save computational resources. In this paper, we present and evaluate methods for detecting templates considering a scenario where multiple templates can be found in a collection of Web pages. Most of previous work have studied template detection algorithms in a scenario where the collection has just a single template. The scenario with multiple templates is more realistic and, as it is discussed here, it raises important questions that may require 172 World Wide Web (2009) 12:171-211 extensions and adjustments in previously proposed template detection algorithms. We show how to apply and evaluate two template detection algorithms in this scenario, creating solutions for detecting multiple templates. The methods studied partitions the input collection into clusters that contain common HTML paths and share a high number of HTML nodes and then apply a single-template detection procedure over each cluster. We also propose a new algorithm for single template detection based on a restricted form of bottom-up tree-mapping that requires only small set of pages to correctly identify a template and which has a worst-case linear complexity. Our experimental results over a representative set of Web pages show that our approach is efficient and scalable while obtaining accurate results.

show abstract

Section: Related Workmentioning

confidence: 99%

On Finding Templates on Web Collections

et al. 2009

View full text Add to dashboard Cite

show abstract

“…One approach for content extraction is the Template Detection (TD) algorithms [1,3,7,11,20] which applies collections of documents with the same templates to learn common structures. BarYossef et al presented an approach to automatically detect templates by counting the frequent pagelet item sets, where each pagelet is a self-contained logical regions in a web with a well defined topic or functionality [1].…”

Section: Related Workmentioning

confidence: 99%

“…They segmented web pages into blocks, and blocks are clustered according to their style features. The blocks with similar layout style and content were detected and identified as templates [3]. All these template detection algorithms generally try to identify the main content by removing common parts found in different web pages.…”

Section: Related Workmentioning

confidence: 99%

Automatic Web Content Extraction by Combination of Learning and Grouping

Liu

Fan

2015

Proceedings of the 24th International Conference on World Wide Web

View full text Add to dashboard Cite

Web pages consist of not only actual content, but also other elements such as branding banners, navigational elements, advertisements, copyright etc. This noisy content is typically not related to the main subjects of the webpages. Identifying the part of actual content, or clipping web pages, has many applications, such as high quality web printing, e-reading on mobile devices and data mining. Although there are many existing methods attempting to address this task, most of them can either work only on certain types of Web pages, e.g. article pages, or has to develop different models for different websites. We formulate the actual content identifying problem as a DOM tree node selection problem. We develop multiple features by utilizing the DOM tree node properties to train a machine learning model. Then candidate nodes are selected based on the learning model. Based on the observation that the actual content is usually located in a spatially continuous block, we develop a grouping technology to further filter out noisy data and pick missing data for the candidate nodes. We conduct extensive experiments on a real dataset and demonstrate our solution has high quality outputs and outperforms several baseline methods.

show abstract

“…Several procedures have been proposed for template detection [14]. First method uses the concept of page let to divide a Web page.…”

Section: 3template Detectionmentioning

confidence: 99%