Parallel text searching in serial files using a processor farm

Cringean, Janey K.; England, Roger; Manson, Gordon A.; Willett, Peter

doi:10.1145/96749.98249

Cited by 18 publications

(14 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Beginning in the late 1980's several researchers implemented IR systems on distributed-memory multiprocessor machines [Bailey and Hawking 1996;Cringean et al 1990;Frieder and Siegelmann 1991;Pogue and Willett 1987;Stanfill and Kahle 1986;Stanfill et al 1989]. We do not investigate multiprocessor systems in our experiments.…”

Section: Related Workmentioning

confidence: 99%

Evaluating the performance of distributed architectures for information retrieval using a variety of workloads

Cahoon

McKinley

Lu³

2000

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

The information explosion across the Internet and elsewhere offers access to an increasing number of document collections. In order for users to effectively access these collections, information retrieval (IR) systems must provide coordinated, concurrent, and distributed access. In this article, we explore how to achieve scalable performance in a distributed system for collection sizes ranging from 1GB to 128GB. We implement a fully functional distributed IR system based on a multithreaded version of the Inquery unified IR system. To explore the design space more fully, we also implement and validate a flexible simulation model. We measure performance as a function of system parameters such as client command rate, number of document collections, terms per query, query term frequency, number of answers returned, and command mixture. Our results show that it is important to model both query and document commands because the heterogeneity of commands significantly impacts performance. Based on our results, we recommend simple changes to the prototype and evaluate the changes using the simulator. Because of the significant resource demands of information retrieval, it is not difficult to generate workloads that overwhelm system resources regardless of the architecture. However under some realistic workloads, we demonstrate system organizations for which response time gracefully degrades as the workload increases and performance scales with the number of processors. This scalable architecture includes a surprisingly small number of brokers through which a large number of clients and servers communicate. INTRODUCTIONThe increasing numbers of large, unstructured text collections require full-text information retrieval (IR) systems in order for users to access them effectively. Current systems typically only allow users to connect to a single database either locally or perhaps on another machine. A distributed IR system should be able to provide multiple users with concurrent, efficient access to multiple text collections located on disparate sites. Since the documents in unstructured text collections are independent, IR systems are ideal applications to distribute across a network of workstations. However, the high resource demands of IR systems limit their performance, especially as the number of users, as well as the size and number of text collections, increases. Distributed computing offers a solution to these problems.Only recently have people published work on distributed architectures for information retrieval. The Very Large Collection track in the TREC conferences promotes the development of distributed and shared memory architectures for IR [Hawking and Thistlewaite 1997;Hawking et al. 1998]. Several researchers created distributed IR systems and demonstrated the feasibility of distributed architectures for information retrieval [Harman et al. 1991;Macleod et al. 1987]. However, it is not clear from these initial implementations how the systems will perform in practice, since, unlike the case for database syst...

show abstract

Section: Related Workmentioning

confidence: 99%

Evaluating the performance of distributed architectures for information retrieval using a variety of workloads

Cahoon

McKinley

Lu³

2000

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

show abstract

“…. |C|] 4: Return the documents to their initial positions as they were before Step 1 5: for each processor j do 6: for each Cluster C i do 7: lc(i, j ) = 0 {lc(i, j ): the local centroid of C i in processor j } 8: for each local document D k ∈ C i do 9: …”

Section: Algorithmmentioning

confidence: 99%

“…To the best of our knowledge, the attempts to adopt parallel processing techniques to large-scale text retrieval have been made either on the real environment of a supercomputer (e.g., connection machine [47], GCel3/512 Parsytec machine [19], APseries multicomputers [24], high performance transputer networks [9]) or of a PC cluster (e.g. [8,29,38,44]), or by employing specific interconnection networks for connecting the processors of the parallel machine (e.g., fat-trees, hypercubes [39]).…”

Section: Introductionmentioning

confidence: 99%

Efficient parallel Text Retrieval techniques on Bulk Synchronous Parallel (BSP)/Coarse Grained Multicomputers (CGM)

et al. 2008

View full text Add to dashboard Cite

In this paper, we present efficient, scalable, and portable parallel algorithms for the off-line clustering, the on-line retrieval and the update phases of the Text Retrieval (TR) problem based on the vector space model and using clustering to organize and handle a dynamic document collection. The algorithms are running on the Coarse-Grained Multicomputer (CGM) and/or the Bulk Synchronous Parallel (BSP) model which are two models that capture within a few parameters the characteristics of the parallel machine. To the best of our knowledge, our parallel retrieval algorithms are the first ones analyzed under these specific parallel models. For all the phases of the proposed algorithms, we analytically determine the relevant communication and computation cost thereby formally proving the efficiency of the proposed solutions. In addition, we prove that our technique for the on-line retrieval phase performs very well in comparison to other possible alternatives in the typical case of a multiuser information retrieval (IR) system where a number of user queries are concurrently submitted to an IR system. Finally, we discuss external memory issues and C. Konstantopoulos ( ) Efficient parallel TR techniques on BSP/CGM 287 show how our techniques can be adapted to the case when processors have limited main memory but sufficient disk capacity for holding their local data.

show abstract

“…An example of the two-phase search can best be illustrated by looking at one particular system, the transputer network program described by Cringean et al [58][59][60][61][62].…”

Section: Two-phase Searchmentioning

confidence: 99%

“…The dominant scheme has been the vector processing model with systems such as RAP.3 [75], the DowQuest [63], Transputer Networks [58][59][60][61][62]84] and POOMA [68], all using it in various forms.…”

Section: Term Weighting Modelsmentioning

confidence: 99%

Parallel computing in information retrieval – an updated review

MacFarlane

Robertson

McCann

1997

Journal of Documentation

View full text Add to dashboard Cite

This is the accepted version of the paper.This version of the publication may differ from the final published version. The progress of parallel computing in Information Retrieval (IR) is reviewed. In particular we stress the importance of the motivation in using parallel computing for Text Retrieval. We analyse parallel IR systems using a classification due to Rasmussen [1] and describe some parallel IR systems. We give a description of the retrieval models used in parallel Information PermanentProcessing.. We describe areas of research which we believe are needed.

show abstract

Parallel text searching in serial files using a processor farm

Abstract: This paper discusses the implementation of a parallel text retrieval system using a microprocessor network.

Cited by 18 publications

References 17 publications

Evaluating the performance of distributed architectures for information retrieval using a variety of workloads

Evaluating the performance of distributed architectures for information retrieval using a variety of workloads

Efficient parallel Text Retrieval techniques on Bulk Synchronous Parallel (BSP)/Coarse Grained Multicomputers (CGM)

Parallel computing in information retrieval – an updated review

Contact Info

Product

Resources

About