Interactive Prior Elicitation of Feature Similarities for Small Sample Size Prediction

Abstract

Regression under the "small n, large p" conditions, of small sample size n and large number of features p in the learning data set, is a recurring setting in which learning from data is difficult. With prior knowledge about relationships of the features, p can effectively be reduced, but explicating such prior knowledge is difficult for experts. In this paper we introduce a new method for eliciting expert prior knowledge about the similarity of the roles of features in the prediction task. The key idea is to use an interactive multidimensionalscaling (MDS) type scatterplot display of the features to elicit the similarity relationships, and then use the elicited relationships in the prior distribution of prediction parameters. Specifically, for learning to predict a target variable with Bayesian linear regression, the feature relationships are used to construct a Gaussian prior with a full covariance matrix for the regression coefficients. Evaluation of our method in experiments with simulated and real users on text data confirm that prior elicitation of feature similarities improves prediction accuracy. Furthermore, elicitation with an interactive scatterplot display outperforms straightforward elicitation where the users choose feature pairs from a feature list.

Knowledge elicitation via sequential probabilistic inference for high-dimensional prediction

“…The concurrent works by Micallef et al (2017), Afrabandpey et al (2016), and Soare et al (2016) also use elicited expert knowledge for improving prediction. Micallef et al (2017) use a separate multi-armed bandit model to facilitate the elicitation and directly modify the regression model priors, Soare et al (2016) consider predictions for a target patient in a simulated user setting, while Afrabandpey et al (2016) consider pairwise similarity feedback on features and uses those to create a better covariance matrix for the prior of coefficients of ridge regression. Contrary to our work, these works do not formulate an encompassing probabilistic model for the expert knowledge and prediction, a crucial feature of our approach.…”

Interactive Elicitation of Knowledge on Feature Relevance Improves Predictions in Small Data Sets

“…The method in [32] tackles this problem with the simplifying assumption that the expert may give noisy input directly on the regression coefficients, and [25] performed non-interactively a direct elicitation of logistic regression coefficients. In recent works considering a similar problem, a user specified the similarity of features as input [1], or features were chosen based on information gain [10].…”

“…We define a prior for ξ as ξ ∼ Beta (1,9), shown in Fig. 4C, which corresponds to the expectation that approximately 10% of the variance of the target is explained by the prediction model.…”

“…However, a user model selects features whose relevance is indicated by the user, a domain expert, using an interactive visualization. Here, a relevant feature is a feature that is positively correlated 1 with the target value. The user model iteratively elicits this information, to build a model of the user's tacit knowledge and select other features that would benefit from the user's input.…”

Improving genomics-based predictions for precision medicine through active elicitation of expert knowledge

“…Another method, complementary to these methods, is to collect prior knowledge directly from an expert. Such prior elicitation techniques ( O’Hagan et al , 2006 ) have been used for constructing prior distributions for Bayesian data analysis that take into account expert knowledge and hence can restrict the range of parameters in predictive models ( Afrabandpey et al , 2017 ; Garthwaite et al , 2013 ; Garthwaite and Dickey, 1988 ; Kadane et al , 1980 ).…”