BioCrawler: An intelligent crawler for the semantic web

Batzios, Alexandros; Dimou, Christos; Symeonidis, Andreas L.; Mitkas, Pericles A.

doi:10.1016/j.eswa.2007.07.054

Cited by 30 publications

(24 citation statements)

References 28 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Crawlers traverse the Web by starting from a random Web page and continue by the following links to other Web pages. Sometimes crawlers need to exchange the information with other crawlers in order to notify their peers about sites with rich semantic content [8]. So it appears as the main process in any search engine.…”

Section: Develop An Embedded Web Crawlermentioning

confidence: 99%

“…Ranking algorithms represent a secret for different search engines; therefore these companies do not provide details about how they exactly rank the Web pages and consider these algorithms as their top secret that should not be know by other competitive search engine companies and Web spammer [7]. Web crawlers or robots constitute another part of a search engine; they are responsible for visiting different Web documents to be indexed [8].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A link and Content Hybrid Approach for Arabic Web Spam Detection

Wahsheh¹,

Al-Kabi

Alsmadi³

2012

IJISA

View full text Add to dashboard Cite

Abstract-Some Web sites developers act as spammers and try to mislead the search engines by using illegal Search Engine Optimizations (SEO) tips to increase the rank of their Web documents, to be more visible at the top 10 SERP. This is since gaining more visitors for marketing and commercial goals. This study is a continuation of a series of Arabic Web spam studies conducted by the authors, where this study is dedicated to build the first Arabic content/link Web spam detection system. This Novel system is capable to extract the set of content and link features of Web pages, in order to build the largest Arabic Web spam dataset. The constructed dataset contains three groups with the following three percentages of spam contents: 2%, 30%, and 40%. These three groups with varying percentages of spam contents were collected through the embedded crawler in the proposed system. The automated classification of spam Web pages used based on the features in the benchmark dataset. The proposed system used the rules of Decision Tree; which is considered as the best classifier to detect Arabic content/link Web spam. The proposed system helps to clean the SERP from all URLs referring to Arabic spam Web pages. It produces accuracy of 90.1099% for Arabic contentbased, 93.1034% for Arabic link-based, and 89.011% in detecting both Arabic content and link Web spam, based on the collected dataset and conducted analysis.

show abstract

Section: Develop An Embedded Web Crawlermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A link and Content Hybrid Approach for Arabic Web Spam Detection

Wahsheh¹,

Al-Kabi

Alsmadi³

2012

IJISA

View full text Add to dashboard Cite

show abstract

“…Thus, in order to create proper crawling and indexing methodologies for semantically annotated data, new retrieval techniques have been proposed. BioCrawler [9] was created as an intelligent crawler for evaluating semantic contents of Web pages using the BioTope framework [10] as its engine. MultiCrawler [11] was developed as a pipelined crawler and indexer for collecting semi-structured data from the classical Web and the Semantic Web.…”

Section: Semantic Web Crawling and Metacrawlingmentioning

confidence: 99%

SpEnD: Linked Data SPARQL Endpoints Discovery Using Search Engines

Yumuşak

Doğdu

Kodaz

et al. 2017

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYLinked data endpoints are online query gateways to semantically annotated linked data sources. In order to query these data sources, SPARQL query language is used as a standard. Although a linked data endpoint (i.e. SPARQL endpoint) is a basic Web service, it provides a platform for federated online querying and data linking methods. For linked data consumers, SPARQL endpoint availability and discovery are crucial for live querying and semantic information retrieval. Current studies show that availability of linked datasets is very low, while the locations of linked data endpoints change frequently. There are linked data respsitories that collect and list the available linked data endpoints or resources. It is observed that around half of the endpoints listed in existing repositories are not accessible (temporarily or permanently offline). These endpoint URLs are shared through repository websites, such as Datahub.io, however, they are weakly maintained and revised only by their publishers. In this study, a novel metacrawling method is proposed for discovering and monitoring linked data sources on the Web. We implemented the method in a prototype system, named SPARQL Endpoints Discovery (SpEnD). SpEnD starts with a "search keyword" discovery process for finding relevant keywords for the linked data domain and specifically SPARQL endpoints. Then, the collected search keywords are utilized to find linked data sources via popular search engines (Google, Bing, Yahoo, Yandex). By using this method, most of the currently listed SPARQL endpoints in existing endpoint repositories, as well as a significant number of new SPARQL endpoints, have been discovered. We analyze our findings in comparison to Datahub collection in detail.

show abstract

“…Three metrics are involved, which are combination importance metric, association metric and ordering metric. First of all, given a webpage p, the combination importance metric CI(p) can be computed as shown below: (1 ) where aI, a2, a3 and a4 are constants, IB(p) is the number of inbound links to webpage p, IL(p) is the location weight of webpage p, JF(p) is the number of outbound links from webpage p. and lR(P) is the PageRank [30,31] weight of webpage p, which can be computed by (2) shown below:…”

Section: Ontology-based Focused Crawlersmentioning

confidence: 99%

State of the Art in Semantic Focused Crawlers

Dong

Hussain

Chang

2009

Computational Science and Its Applications – ICCSA 2009

View full text Add to dashboard Cite

Abstract. Nowadays, the research of focused crawler approaches the field of semantic web, along with the appearance of increasing semantic web documents and the rapid development of ontology mark-up languages. Semantic focused crawlers are a series of focused crawlers enhanced by various semantic web technologies. In this paper, we make a survey in this research field. We discover eleven semantic focused crawlers from the existing literature, and classify them into three categories -ontology-based focused crawlers, metadata abstraction focused crawlers and other semantic focused crawlers. By means of a multi-dimensional comparison, we conclude the features of these crawlers and draw the overall state of the art of this field.

show abstract

BioCrawler: An intelligent crawler for the semantic web

Cited by 30 publications

References 28 publications

A link and Content Hybrid Approach for Arabic Web Spam Detection

A link and Content Hybrid Approach for Arabic Web Spam Detection

SpEnD: Linked Data SPARQL Endpoints Discovery Using Search Engines

State of the Art in Semantic Focused Crawlers

Contact Info

Product

Resources

About