Utility-based information distillation over temporally sequenced documents

Yang, Yiming; Lad, Abhimanyu; Lao, Ni; Harpale, Abhay; Kisiel, Bryan; Rogati, Monica

doi:10.1145/1277741.1277750

Cited by 19 publications

(25 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, thorough investigations in leveraging context are relatively sparse so far. Yang et al investigated how to improve information distillation tasks (retrieval, adaptive filtering and novelty detection) by leveraging the history of user interactions as the context [12]. In this paper, we approach the problem of IR in context from a different angle, i.e., how to improve the total utility of multiple systems in a multi-step process with user interactions, specifically, by offering a novel solution to the problem of optimally ordering a set of interrelated prediction tasks.…”

Section: Discussionmentioning

confidence: 99%

Graph Structure Learning for Task Ordering

Yang

Lad

Shu

et al. 2009

Proceedings of the 11th International Conference on Enterprise Information

Self Cite

View full text Add to dashboard Cite

Abstract:In many practical applications, multiple interrelated tasks must be accomplished sequentially through user interaction with retrieval, classification and recommendation systems. The ordering of the tasks may have a significant impact on the overall utility (or performance) of the systems; hence optimal ordering of tasks is desirable. However, manual specification of optimal ordering is often difficult when task dependencies are complex, and exhaustive search for the optimal order is computationally intractable when the number of tasks is large. We propose a novel approach to this problem by using a directed graph to represent partialorder preferences among task pairs, and using link analysis (HITS and PageRank) over the graph as a heuristic to order tasks based on how important they are in reinforcing and propagating the ordering preference. These strategies allow us to find near-optimal solutions with efficient computation, scalable to large applications. We conducted a comparative evaluation of the proposed approach on a form-filling application involving a large collection of business proposals from the Accenture Consulting & Technology Company, using SVM classifiers to recommend keywords, collaborators, customers, technical categories and other related fillers for multiple fields in each proposal. With the proposed approach we obtained nearoptimal task orders that improved the utility of the recommendation system by 27% in macro-averaged F1, and 13% in micro-averaged F1, compared to the results obtained using arbitrarily chosen orders, and that were competitive against the best order suggested by domain experts.

show abstract

Section: Discussionmentioning

confidence: 99%

Graph Structure Learning for Task Ordering

Yang

Lad

Shu

et al. 2009

Proceedings of the 11th International Conference on Enterprise Information

Self Cite

View full text Add to dashboard Cite

show abstract

“…It consists of over 90,000 articles from various news sources published between October 2000 and January 2001. This corpus was extended for distillation evaluations by identifying 12 actionable events and defining information distillation tasks on them, as described in [5,12]. Following [12], we divided the 4-month span of the corpus into 10 chunks, each comprising 12 consecutive days.…”

Section: Methodsmentioning

confidence: 99%

“…The Normalized Discounted Cumulated Utility (NDCU) scheme [12] is an extension of the popular NDCG metric [6] to model utility as the difference between the gain and cost incurred by a user in going through a ranked list presented by the system. Specifically, the utility of each passage is defined as:…”

Section: Normalized Discounted Cumulated Utilitymentioning

confidence: 99%

“…Information distillation is an emerging area of research where the focus is to effectively combine ad-hoc retrieval (IR), novelty detection (ND) and adaptive filtering (AF) over temporally ordered documents for global utility optimization [12,2,11]. An information distillation system is typically designed for use over multiple sessions by a user or analyst.…”

Section: Introductionmentioning

confidence: 99%

“…The above challenge has been partially addressed by the NDCU (Normalized Discounted Cumulated Utility) scheme proposed by Yang et al [12] for distillation evaluation. NDCU uses nugget-level relevance judgments to enable automated determination of relevance and novelty of each passage in a system-produced ranked list.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Modeling Expected Utility of Multi-session Information Distillation

Yang

Lad

2009

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. An open challenge in information distillation is the evaluation and optimization of the utility of ranked lists with respect to flexible user interactions over multiple sessions. Utility depends on both the relevance and novelty of documents, and the novelty in turn depends on the user interaction history. However, user behavior is non-deterministic. We propose a new probabilistic framework for stochastic modeling of user behavior when browsing multi-session ranked lists, and a novel approximation method for efficient computation of the expected utility over numerous user-interaction patterns. Using this framework, we present the first utility-based evaluation over multi-session search scenarios defined on the TDT4 corpus of news stories, using a state-of-the-art information distillation system. We demonstrate that the distillation system obtains a 56.6% utility enhancement by combining multi-session adaptive filtering with novelty detection and utility-based optimization of system parameters for optimal ranked list lengths.Key words: Multi-session distillation, utility evaluation based both on novelty and relevance, stochastic modeling of user browsing behavior.

show abstract

A pattern mining approach for information filtering systems

Algarni

2010

Inf Retrieval

View full text Add to dashboard Cite

It is a big challenge to clearly identify the boundary between positive and negative streams for information filtering systems. Several attempts have used negative feedback to solve this challenge; however, there are two issues for using negative relevance feedback to improve the effectiveness of information filtering. The first one is how to select constructive negative samples in order to reduce the space of negative documents. The second issue is how to decide noisy extracted features that should be updated based on the selected negative samples. This paper proposes a pattern mining based approach to select some offenders from the negative documents, where an offender can be used to reduce the side effects of noisy features. It also classifies extracted features (i.e., terms) into three categories: positive specific terms, general terms, and negative specific terms. In this way, multiple revising strategies can be used to update extracted features. An iterative learning algorithm is also proposed to implement this approach on the RCV1 data collection, and substantial experiments show that the proposed approach achieves encouraging performance and the performance is also consistent for adaptive filtering as well.

show abstract

Utility-based information distillation over temporally sequenced documents

Cited by 19 publications

References 23 publications

Graph Structure Learning for Task Ordering

Graph Structure Learning for Task Ordering

Modeling Expected Utility of Multi-session Information Distillation

A pattern mining approach for information filtering systems

Contact Info

Product

Resources

About