An XML based Web Crawler with Page Revisit Policy and Updation in Local Repository of Search Engine

Abstract

In a large collection of web pages, it is difficult for search engines to keep their online repository updated. Major search engines have hundreds of web crawlers that crawl the WWW day and night and send the downloaded web pages via a network to be stored in the search engine’s database. These results in over utilization of network resources like bandwidth, CPU cycles and so on. This paper proposes an architecture that tries to reduce the utilization of shared network resources with the help of an advanced XML based approach. This focused crawling based architecture is trained to download only the high quality data from the internet leaving behind the web pages which are not relevant to the desired domain. Here, a detailed layout of the proposed system is described which is capable of reducing the load on network and reducing the problem arise in residency of mobile agent at the remote server.

Technical Job Recommendation System Using APIs and Web Crawling

“…Crawling. Nowadays, many companies do not list their job openings on the common job listing portals on the web [35][36][37][38][39][40][41][42][43][44][45][46][47][48][49]. It was decided to individually crawl the API calls [50,51] of these companies' personal job portals.…”

An Empirical Framework for Recommendation-based Location Services Using Deep Learning

“…In the future, we plan to analyze and solve additional privacy concerns in location service recommendations, as well as improve the performance, using a parallel and distributed recommendations system to manage the required massive data. This model may also improve the results of [15,[23][24][25][26][27][28][29].…”