Downloading textual hidden web content through keyword queries

Ntoulas, Alexandros; Pzerfos, P.; Cho, Junghoo

doi:10.1145/1065385.1065407

Cited by 156 publications

(116 citation statements)

References 11 publications

Supporting

Mentioning

108

Contrasting

Unclassified

Order By: Relevance

“…If queries are not selected properly, most of the documents may be redundant. Therefore, query selection is modelled as a set covering [4] or dominating vertex [5] problem, so that the queries can return less redundancies. Since set covering problem or dominating vertex problem is NP-hard, the optimal solution cannot be found, especially because the problem size is very big that involves thousands or even more of documents and terms.…”

Section: Related Workmentioning

confidence: 99%

“…If we regard all the documents in a data source as the universe, each query is a subset of the documents it can match, the query selection problem is to find the subsets (the queries) to cover all the documents with minimal cost. Since the entire set of documents is not available, the queries have to be selected from a sample of partially downloaded documents [4,5,6,7]. In particular, [7,8] demonstrates that the queries selected from a sample set of documents can also work well for the entire data set.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

TS-IDS Algorithm for Query Selection in the Deep Web Crawling

Wang

Lü

Chen

2014

Web Technologies and Applications

View full text Add to dashboard Cite

Abstract. The deep web crawling is the process of collecting data items inside a data source hidden behind searchable interfaces. Since the only method to access the data is by sending queries, one of the research challenges is the selection of a set of queries such that they can retrieve most of the data with minimal network traffic. This is a set covering problem that is NP-hard. The large size of the problem, in terms of both large number of documents and terms involved, calls for new approximation algorithms for efficient deep web data crawling. Inspired by the TF-IDF weighting measure in information retrieval, this paper proposes the TS-IDS algorithm that assigns an importance value to each document proportional to term size (TS), and inversely proportional to document size (IDS). The algorithm is extensively tested on a variety of datasets, and compared with the traditional greedy algorithm and the more recent IDS algorithm. We demonstrate that TS-IDS outperforms the greedy algorithm and IDS algorithm up to 33% and 29%, respectively. Our work also makes a contribution to the classic set covering problem by leveraging the long-tail distributions of the terms and documents in natural languages. Since long-tail distribution is ubiquitous in real world, our approach can be applied in areas other than the deep web crawling.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

TS-IDS Algorithm for Query Selection in the Deep Web Crawling

Wang

Lü

Chen

2014

Web Technologies and Applications

View full text Add to dashboard Cite

show abstract

“…The negative effects of this, like introducing terms in a server's resource description that the server does not contain or underestimating term frequencies, are offset by the overall improvement in performance. Ntoulas et al (2005) have the goal of crawling as much content as possible from a hidden web resource and present an adaptive approach for term selection: they try to identify those terms that are most likely to return the most additional documents. This outperforms other approaches in terms of the number of documents seen.…”

Section: Can We Do Better Than Random?mentioning

confidence: 99%

Peer-to-peer information retrieval

Tigelaar

2012

SIGIR Forum

View full text Add to dashboard Cite

“…As in the cases of TextRunner and WebTables our goal was to develop techniques that would apply efficiently on large numbers of forms. This is in contrast with much prior work that have either addressed the problem by constructing mediator systems one domain at a time [12,13,26], or have needed site-specific wrappers or extractors to extract documents from text databases [5,22]. As we discuss, the pages we surface contain tables from which additional data can be extracted for the Web knowledge base.…”

Section: Accessing Deep-web Databasesmentioning

confidence: 99%

“…The set of all candidate keywords can then be pruned to select a smaller subset that ensures diversity of the exposed database contents. Similar iterative probing approaches have been used in the past to extract text documents from specific databases [5,10,18,22].…”

Section: Surfacing Deep-web Databasesmentioning

confidence: 99%

Web-scale extraction of structured data

2009

View full text Add to dashboard Cite

A long-standing goal of Web research has been to construct a unified Web knowledge base. Information extraction techniques have shown good results on Web inputs, but even most domain-independent ones are not appropriate for Web-scale operation. In this paper we describe three recent extraction systems that can be operated on the entire Web (two of which come from Google Research). The TextRunner system focuses on raw natural language text, the WebTables system focuses on HTML-embedded tables, and the deep-web surfacing system focuses on "hidden" databases. The domain, expressiveness, and accuracy of extracted data can depend strongly on its source extractor; we describe differences in the characteristics of data produced by the three extractors. Finally, we discuss a series of unique data applications (some of which have already been prototyped) that are enabled by aggregating extracted Web information.

show abstract

Downloading textual hidden web content through keyword queries

Cited by 156 publications

References 11 publications

TS-IDS Algorithm for Query Selection in the Deep Web Crawling

TS-IDS Algorithm for Query Selection in the Deep Web Crawling

Peer-to-peer information retrieval

Web-scale extraction of structured data

Contact Info

Product

Resources

About