Controlling the robots of Web search engines

Talim, J.; Liu, Z.; Nain, Ph.; Coffman, E. G.

doi:10.1145/378420.378788

Cited by 22 publications

(27 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proof of Lemma 2 is given in (Talim et al, 2001a). The next result follows from Lemma 2 and Proposition 4 in (Bertsekas, 1987,Sec.…”

Section: The Dynamic Settingmentioning

confidence: 83%

“…In both settings the cost function is a weighted sum of the starvation probability and loss rate. The results presented in this section are based on the work of Talim et al (2001b) and Talim et al (2001a). Practical issues of deploying parallel crawlers are discussed in Cho and Garcia-Molina (2002).…”

Section: Optimizing the Number Of Crawlersmentioning

confidence: 99%

“…Throughout Section 1.2.1 we will assume that indexing times are exponentially distributed. The general case where the indexing times are arbitrarily distributed is more involved, due to the lack of closed-form expressions for the M/G/1/K queue, and is discussed in Talim et al (2001b).…”

Section: The Static Settingmentioning

confidence: 99%

“…Lemma 1 shows the existence of a unique minimum for C(ρ, γ, K) considered as a function of ρ. The proof is provided in Talim et al (2001b).…”

Section: The Static Settingmentioning

confidence: 99%

“…Technical results presented in Sections 1.2 and 1.3 were initially obtained in Talim et al (2001a), Talim et al (2001b) andCoffman Jr. et al (1998). We would like to acknowledge the contributions of the co-authors of these papers, namely, Ed.…”

Section: Acknowledgmentsmentioning

confidence: 99%

See 4 more Smart Citations

Optimization Issues in Web Search Engines

Liu

Nain²

Handbook of Optimization in Telecommunications

View full text Add to dashboard Cite

Crawlers are deployed by a Web search engine for collecting information from different Web servers in order to maintain the currency of its data base of Web pages. We present studies on the optimization of Web search engines from different perspectives. We first investigate the number of crawlers to be used by a search engine so as to maximize the currency of the data base without putting an unnecessary load on the network. Both the static setting, where crawlers are always active, and the dynamic setting where, crawlers may be activated/deactivated as a function of the state of the system, are addressed. We then consider the optimal scheduling of the visits of these crawlers to the Web pages assuming these pages are modified at different rates. Finally, we briefly discuss some other optimization issues of Web search engines, including page ranking and system optimization.

show abstract

“…The proof of Lemma 2 is given in (Talim et al, 2001a). The next result follows from Lemma 2 and Proposition 4 in (Bertsekas, 1987,Sec.…”

Section: The Dynamic Settingmentioning

confidence: 83%