Fuzzy versus probabilistic models for user relevance judgments

Abstract

A crucial aspect of information retrieval is the process of making relevance judgments. Although this highly complex decision making procedure still eludes researchers, it is evident that a number of mental models are involved: models of the information need, retrieval system, database, user's knowledge in the subject area etc. In general, It is accepted that relevance judgments are made by evaluating documents for an overall conceptual match with the informatlon need. In this study, we take the view that, given a document and a query, users flrst judge the document against the individual concepts in the query and then use some inferenclng process to derive from these "atomic" decisions a "compound" judgment for the entire query. In other words we adopt a bottom up approach to this decision making process. In this context we examine strategies that may be used to infer compound relevance judgments made from judgments against smaller units of the information need. The probabilistic and fuzzy models are used as two points of reference against which to analyze the user's decision process in making compound judgments. Each model has a different way of approaching the problem. The objective was to empirically examine the relative effectiveness of the models at predicting compound judgments made by users, from their atomic judgments. The results show that the fuzzy model Is sometimes a better predictor than the probabilistic model. The conclusions are interesting when comparing performances across AND and OR queries. No differences were observed between the two models for the OR queries. Both models tend to underestimate relevance for the AND queries. However, the probabilistic model regularly underestlmated relevance more than the fuzzy model. The conclusion made Is that the user seldom employs the AND operator as rigidly as assumed by these models and perhaps by most IR systems. Also, matching functions that better approximate the user's decision process are required to lead to more effectlve systems.

A method to evaluate service delivery process quality

“…However, a fuzzy approach may not always yield better results. Researchers in fields such as process control (Barrett and Woodall, 1997), health risk assessment (Kentel and Aral, 2005) and information retrieval (Koll and Srinivasan, 1990) have compared results using probability and possibility analysis, and found that some situations favour the former approach while other situations favour the latter. Hence, future research could first apply fuzzy theory to the uncertain parameters in the Taguchi quality loss framework, and then compare the suitability of the probability and possibility analysis.…”

Fuzzy information technology and knowledge management

“…New approaches to database structure such as multi-dimensional databases [13] and data warehousing [5] facilitate both storing vast quantities of data and using very complex queries to extract mformation. Fuzzy data management and retrieval enhance these efforts [22] [24].…”

Fuzzy information technology and knowledge management

“…New approaches to database structure such as multi-dimensional databases [13] and data warehousing [5] facilitate both storing vast quantities of data and using very complex queries to extract information. Fuzzy data management and retrieval enhance these efforts [19] [21].…”