CINDI Robot: an Intelligent Web Crawler Based on Multi-level Inspection

Abstract

With the explosion of the Web, focused web crawlers are gaining attention. Focused web crawlers aim at finding web pages related to the pre-defined topic. CINDI Robot is a focused web crawler devoted to finding computer science and software engineering academic documents. We propose a multi-level inspection scheme to discover relevant web pages. Through this multi-level inspection scheme, the text feature of the content contributes to the classification; furthermore other web characteristics, such as URL pattern, anchor text and so on, assist the decision process. The experiment result demonstrates this multilevel inspection method outperforms other traditional methods.

An enhanced web robot for the CINDI system

“…In the CINDI Robot, a revised context graph [5] is proposed to classify all web pages into three categories: web pages that are directly related to our topic (referred as Layer 1 web pages in following), web pages which are indirectly related to our topic but may lead to relevant web pages (referred as Layer 2 web pages in following) and web pages that are totally useless. In contrast with the classic context graph [6], our strategy gets rid of the strict link distance requirements, reduces classification times, increases classification accuracy rates and significantly increases the opportunities of discovering relevant web regions via indirectly related web pages.…”

“…For the second kind of classifier, Naïve Bayes classifier is exploited as the Layer 2 web page classifier. In addition, a novel tunneling technique is implemented based on the Layer 2 web page classifier in order to reach more relevant regions by allowing the CINDI Robot to visit some low relevancy intermediates [5].…”

“…These two working modes guarantee the CINDI Robot can achieve a high precision while trying to keep recall as high as possible. URL pattern inspection is implemented to include 5 main aspects: computer science department web site speculation, stop-directory and to-be-avoided directory filtering [5], protocol and file format filtering, seven directory level exclusion and the Robot Exclusion Protocol [7]. URL pattern inspection of the CINDI Robot can speculate if a web page is from a computer science department by examining its URL.…”

Text Recognition in the Wild: A Survey

“…As one of the most influential inventions of humanity, text has played an important role in human life. Specifically, rich and precise semantic information carried by text is important in a wide range of vision-based application scenarios, such as image search [1], intelligent inspection [2], industrial automation [3], robot navigation [4], and instant translation [5]. Therefore, text recognition in natural scenes has drawn the attention of researchers and practitioners, as indicated by the emergence of recent "ICDAR Robust Reading Competitions" [6], [7], [8], [9], [10], [11], [12].…”