Mining the Web's link structure

Chakrabarti, Soumen; Dom, Byron; Kumar, Shailesh; Raghavan, Prabhakar; Rajagopalan, Sridhar; Tomkins, Andrew; Gibson, David; Kleinberg, Jon

doi:10.1109/2.781636

Cited by 393 publications

(168 citation statements)

References 8 publications

Supporting

Mentioning

160

Contrasting

Unclassified

Order By: Relevance

“…Similarly, when a site is known to be a good hub, it seems reasonable to assign high weights to its outgoing links. This approach has been recently applied in [5]. We coin it the anchor sites approach, since it uses user-designated sites as anchors in the collection, around which the communities of hubs and authorities are grown.…”

Section: Salsa and The In/out Degrees Of Sitesmentioning

confidence: 99%

The stochastic approach for link-structure analysis (SALSA) and the TKC effect

2000

View full text Add to dashboard Cite

This article is being produced via complete electronic text/image integration, to enable full database storage. It has therefore to meet specific standards regarding structure and presentation.This proof is an automatically produced, low-resolution laser-printer output. The final product will naturally meet our high-quality standards regarding resolution and page lay-out.Please use this proof solely for checking of typesetting/editing and completeness of text and figures. Changes to the article as accepted for publication will not be considered at this stage. Elsevier Science N.L. Computer Networks 00 (2000) COMPNW2288The stochastic approach for link-structure analysis (SALSA) and the TKC effect 1 R. Lempel Ł , S. Moran Department of Computer Science, The Technion, Haifa 32000, Israel AbstractToday, when searching for information on the World Wide Web, one usually performs a query through a term-based search engine. These engines return, as the query's result, a list of Web sites whose contents match the query. For broad topic queries, such searches often result in a huge set of retrieved documents, many of which are irrelevant to the user. However, much information is contained in the link-structure of the World Wide Web. Information such as which pages are linked to others can be used to augment search algorithms. In this context, Jon Kleinberg introduced the notion of two distinct types of Web sites: hubs and authorities. Kleinberg argued that hubs and authorities exhibit a mutually reinforcing relationship: a good hub will point to many authorities, and a good authority will be pointed at by many hubs. In light of this, he devised an algorithm aimed at finding authoritative sites. We present SALSA, a new stochastic approach for link structure analysis, which examines random walks on graphs derived from the link structure. We show that both SALSA and Kleinberg's mutual reinforcement approach employ the same meta-algorithm. We then prove that SALSA is equivalent to a weighted in-degree analysis of the link-structure of World Wide Web subgraphs, making it computationally more efficient than the mutual reinforcement approach. We compare the results of applying SALSA to the results derived through Kleinberg's approach. These comparisons reveal a topological phenomenon called the TKC effect (Tightly Knit Community) which, in certain cases, prevents the mutual reinforcement approach from identifying meaningful authorities. 

show abstract

Section: Salsa and The In/out Degrees Of Sitesmentioning

confidence: 99%

The stochastic approach for link-structure analysis (SALSA) and the TKC effect

2000

View full text Add to dashboard Cite

show abstract

“…The notion of link collection is close to that of "pagelet", which was first introduced by Chackrabati et al to denote a contiguous set of links (Chakrabarti et al, 1999a). Chackrabati et al make the hypothesis that links from the same pagelet more tightly focus on a single topic than links from the entire page can do.…”

Section: Samplementioning

confidence: 99%

Efficiently Locating Collections of Web Pages to Wrap

Blanco

Crescenzi

Merialdo

2005

Proceedings of the First International Conference on Web Information Systems and Technologies

View full text Add to dashboard Cite

Abstract:Many large web sites contain highly valuable information. Their pages are dynamically generated by scripts which retrieve data from a back-end database and embed them into HTML templates. Based on this observation several techniques have been developed to automatically extract data from a set of structurally homogeneous pages. These tools represent a step towards the automatic extraction of data from large web sites, but currently their input sample pages have to be manually collected. To scale the data extraction process this task should be automated, as well. We present techniques to automatically gathering structurally similar pages from large web sites. We have developed an algorithm that takes as input one sample page, and crawls the site to find pages similar in structure to the given page. The collected pages can feed an automatic wrapper generator to extract data. Experiments conducted over real life web sites gave us encouraging results.

show abstract

“…Kleinberg's seminal paper [20] on hubs and authorities introduced a natural paradigm for classifying and ranking web pages, setting off an avalanche of subsequent work [7,8,10,15,22,9,3,12,19,2,5,27,1,13]. Kleinberg's ideas were implemented in HITS as part of the CLEVER project [7,10].…”

Section: Introductionmentioning

confidence: 99%

“…Kleinberg's ideas were implemented in HITS as part of the CLEVER project [7,10]. Around the same time, Brin and Page [6,25,18] developed a highly successful search engine, Google [17], which orders search results according to PageRank, a measure of authority of the underlying page.…”

Section: Introductionmentioning

confidence: 99%