Interactive Elicitation of Knowledge on Feature Relevance Improves Predictions in Small Data Sets

Abstract

Providing accurate predictions is challenging for machine learning algorithms when the number of features is larger than the number of samples in the data. Prior knowledge can improve machine learning models by indicating relevant variables and parameter values. Yet, this prior knowledge is often tacit and only available from domain experts. We present a novel approach that uses interactive visualization to elicit the tacit prior knowledge and uses it to improve the accuracy of prediction models. The main component of our approach is a user model that models the domain expert's knowledge of the relevance of different features for a prediction task. In particular, based on the expert's earlier input, the user model guides the selection of the features on which to elicit user's knowledge next. The results of a controlled user study show that the user model significantly improves prior knowledge elicitation and prediction accuracy, when predicting the relative citation counts of scientific documents in a specific domain.

Knowledge elicitation via sequential probabilistic inference for high-dimensional prediction

“…The concurrent works by Micallef et al (2017), Afrabandpey et al (2016), and Soare et al (2016) also use elicited expert knowledge for improving prediction. Micallef et al (2017) use a separate multi-armed bandit model to facilitate the elicitation and directly modify the regression model priors, Soare et al (2016) consider predictions for a target patient in a simulated user setting, while Afrabandpey et al (2016) consider pairwise similarity feedback on features and uses those to create a better covariance matrix for the prior of coefficients of ridge regression. Contrary to our work, these works do not formulate an encompassing probabilistic model for the expert knowledge and prediction, a crucial feature of our approach.…”

Improving genomics-based predictions for precision medicine through active elicitation of expert knowledge

“…These types of feedback are assumed to be available for some or all of the feature-trait pairs in the dataset, and they are treated as additional data when learning the parameters of the spike-and-slab regression model. The relevance feedback has been used in Daee et al (2017) for univariate prediction in textual data, which we extend by including directional feedback ( Micallef et al , 2017 ) in the multi-output scenario.…”

“…It has earlier been used in different types of tasks, for clustering ( Balcan and Blum, 2008 ; Lu and Leen, 2007 ), Bayesian network learning ( Cano et al , 2011 ) and visualization ( House et al , 2015 ). We have applied it recently also to prediction using linear regression in our preliminary work ( Daee et al , 2017 ; Micallef et al , 2017 ; Soare et al , 2016 ). However, these methods are not immediately applicable to precision medicine due to many open questions, in particular (1) how to most effectively personalize predictions for a specific patient, (2) which of the different ways of collecting feedback interactively are the most efficient, (3) what kind of feedback most efficiently improves prediction accuracy and (4) how to handle the multi-task problem arising in multi-output settings.…”

Interactive Prior Elicitation of Feature Similarities for Small Sample Size Prediction

“…Moreover, the difference between the MSE values obtained by the interactive interface and the non-interactive interface shows the amount of improvement made to the predictions using the interface. To test the statistical significance of the improvements made by our method compared to each of the other methods, we used the same procedure introduced in [13]. The distance between the average curves in the last round (round 20 in Figure 3) is used as the test statistics.…”

“…The user's knowledge assumed to be about feature relevance and/or feature weight values. Similarly, Micallef et al [13] proposed an interactive visualization to extract user's knowledge on the relevance of individual features for a prediction task.…”