Toward Optimal Ordering of Prediction Tasks

Abstract

Many applications involve a set of prediction tasks that must be accomplished sequentially through user interaction. If the tasks are interdependent, the order in which they are performed may have a significant impact on the overall performance of the prediction systems. However, manual specification of an optimal order may be difficult when the interdependencies are complex, especially if the number of tasks is large, making exhaustive search intractable. This paper presents the first attempt at solving the optimal task ordering problem using an approximate formulation in terms of pairwise task order preferences, reducing the problem to the well-known Linear Ordering Problem. We propose two approaches for inducing the pairwise task order preferences -1) a classifier-agnostic approach based on conditional entropy that determines the prediction tasks whose correct labels lead to the least uncertainty for the remaining predictions, and 2) a classifier-dependent approach that empirically determines which tasks are favored before others for better predictive performance. We apply the proposed solutions to two practical applications that involve computer-assisted trouble report generation and document annotation, respectively. In both applications, the user fills up a series of fields and at each step, the system is expected to provide useful suggestions, which comprise the prediction (i.e. classification and ranking) tasks. Our experiments show encouraging improvements in predictive performance, as compared to approaches that do not take task dependencies into account.

A Perpetual Learning Algorithm That Incrementally Improves Performance With Deliberation

“…Y. Bengio et al verified that the order of learning examples from easy to hard contributes to faster convergence and higher performance for prediction by experiments [37]. In lifelong learning, curriculum learning refers to how to arrange the task order for learning [38]. Curriculum selection has the assumption that at least the training data for a few tasks can be buffered and decides the order of tasks to be learned according to some heuristic rules.…”

“…Curriculum selection has the assumption that at least the training data for a few tasks can be buffered and decides the order of tasks to be learned according to some heuristic rules. A. Lad et al stated that the optimal task ordering problem can be reduced to the Linear Ordering Problem based on pairwise task order preferences [38]. In [16], P. Ruvolo and E. Eaton examined three rules including Information Maximization, Diversity Methods, Diversity++ for active task selection.…”

Curriculum learning of multiple tasks

“…Similarly, Kumar et al [20] introduced the selfpaced learning algorithm, which automatically chooses the order in which training examples are processed for solving a non-convex learning problem. In the context of learning multiple tasks, the question in which order to learn them was introduced in [21], where Lad et al proposed an algorithm for optimizing the task order based on pairwise preferences. However, they considered only the setting in which tasks are performed in a sequence through user interaction and therefore their approach is not applicable in the standard multi-task scenario.…”

Target Curricula via Selection of Minimum Feature Sets: a Case Study in Boolean Networks

“…However, this approach does not appear applicable to inferring highly non-linear discrete models. Lad et al (2009) present the only model-agnostic approach to determine an optimal order of tasks that we are aware of. They use conditional entropy to produce a pairwise order matrix and solve the resulting NP-Hard Linear Ordering Problem (see Ceberio et al, 2015, for a definition).…”