Knowledge elicitation via sequential probabilistic inference for high-dimensional prediction

Daee, Pedram; Peltola, Tomi; Soare, Marta; Kaski, Samuel

doi:10.1007/s10994-017-5651-7

Cited by 28 publications

(37 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The feedback improved the predictive performance in both systems (p-value= 3 × 10 −8 in baseline and p-value= 5 × 10 −9 in IE without user model, using paired-sample t-test). This shows, as analogously claimed in [3], that the participants, on average, have the necessary knowledge to improve the prediction. More prominently, the predictive performance further improved in the IE system after inferring the latent user knowledge using the user model (p-value= 7 × 10 −7 in No, I did not consider them much Yes, I considered some of them paired-sample t-test between directly using the feedbacks and after employing the user model).…”

Section: Resultssupporting

confidence: 81%

“…This creates a dependency between the feedback and training data that needs to be accounted for in the model to avoid double use of data and overfitting. the user (e.g., active learning) [3,18,19]. Some methods use validation datasets in addition to the training set to evaluate the performance.…”

Section: Updated Knowledgementioning

confidence: 99%

“…Computation of the posterior distribution is intractable. Expectation propagation is used to approximate it (see [3]).…”

Section: Feature Relevance Elicitation In Sparse Linear Regressionmentioning

confidence: 99%

“…We considered the problem of rating prediction from user reviews on Amazon kitchen products. The task and data were previously studied in [3,7]. The data consist of review texts, represented as bag-of-words with 824 distinct unigram and bigram keywords, and their corresponding 1-5 star ratings.…”

Section: Sentiment Analysis Taskmentioning

confidence: 99%

See 3 more Smart Citations

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

Daee

Peltola

Vehtari

et al. 2018

23rd International Conference on Intelligent User Interfaces

Self Cite

View full text Add to dashboard Cite

In human-in-the-loop machine learning, the user provides information beyond that in the training data. Many algorithms and user interfaces have been designed to optimize and facilitate this human-machine interaction; however, fewer studies have addressed the potential defects the designs can cause. Effective interaction often requires exposing the user to the training data or its statistics. The design of the system is then critical, as this can lead to double use of data and overfitting, if the user reinforces noisy patterns in the data. We propose a user modelling methodology, by assuming simple rational behaviour, to correct the problem. We show, in a user study with 48 participants, that the method improves predictive performance in a sparse linear regression sentiment analysis task, where graded user knowledge on feature relevance is elicited. We believe that the key idea of inferring user knowledge with probabilistic user models has general applicability in guarding against overfitting and improving interactive machine learning. * This is the pre-print version. The paper is published in the proceedings of IUI 2018 conference. Definitive

show abstract

Section: Resultssupporting

confidence: 81%

Section: Updated Knowledgementioning

confidence: 99%

“…Computation of the posterior distribution is intractable. Expectation propagation is used to approximate it (see [3]).…”

Section: Feature Relevance Elicitation In Sparse Linear Regressionmentioning

confidence: 99%

Section: Sentiment Analysis Taskmentioning

confidence: 99%

See 2 more Smart Citations

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

Daee

Peltola

Vehtari

et al. 2018

23rd International Conference on Intelligent User Interfaces

Self Cite

View full text Add to dashboard Cite

show abstract

“…Using the extracted knowledge from the interaction, the parameters of the underlying model are tuned to reflect the user's knowledge. In recent work, Daee et al [5] proposed a method of eliciting user's knowledge on single features to improve the predictions in a sparse linear regression problem. The user's knowledge assumed to be about feature relevance and/or feature weight values.…”

mentioning

confidence: 99%

Interactive Prior Elicitation of Feature Similarities for Small Sample Size Prediction

Afrabandpey

Peltola

Kaski

2017

Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization

Self Cite

View full text Add to dashboard Cite

Regression under the "small n, large p" conditions, of small sample size n and large number of features p in the learning data set, is a recurring setting in which learning from data is difficult. With prior knowledge about relationships of the features, p can effectively be reduced, but explicating such prior knowledge is difficult for experts. In this paper we introduce a new method for eliciting expert prior knowledge about the similarity of the roles of features in the prediction task. The key idea is to use an interactive multidimensionalscaling (MDS) type scatterplot display of the features to elicit the similarity relationships, and then use the elicited relationships in the prior distribution of prediction parameters. Specifically, for learning to predict a target variable with Bayesian linear regression, the feature relationships are used to construct a Gaussian prior with a full covariance matrix for the regression coefficients. Evaluation of our method in experiments with simulated and real users on text data confirm that prior elicitation of feature similarities improves prediction accuracy. Furthermore, elicitation with an interactive scatterplot display outperforms straightforward elicitation where the users choose feature pairs from a feature list.

show abstract

A Conceptual Framework for Personalization of Indoor Comfort Parameters Based on Office Workers’ Preferences

Mirzaeifar¹,

Daee²,

Singh³

2018

Product Lifecycle Management to Support Industry 4.0

Self Cite

View full text Add to dashboard Cite

Prevention of building-related illnesses and improving indoor air quality has become an emerging research area not only because of the comfort of workers in an office or the quality of the perceived air, but also because it can provide financial benefits to both employees and employers through a potential reduction in prolonged sick leaves. Therefore, building facility managers attempt to achieve the most comfortable and healthy environment conditions for the office workers. However, the parameters associated with achieved comfort vary from person to person as workers` preferences, as well physiological characteristics, are heterogeneous. In the ideal case, the indoor health parameters should be personalized based on individuals` feedback. This paper presents a computational framework for personalization of environmental parameters based on limited office workers' feedback. We propose that by using current state of the art machine learning methods it is possible to learn the preference model of individuals, by employing both the limited feedback and the relevant literature on healthrelated symptoms. The framework is explained and discussed in a potential example scenario. Evaluation based on real data is left as a future work.

show abstract

Knowledge elicitation via sequential probabilistic inference for high-dimensional prediction

Cited by 28 publications

References 34 publications

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

Interactive Prior Elicitation of Feature Similarities for Small Sample Size Prediction

A Conceptual Framework for Personalization of Indoor Comfort Parameters Based on Office Workers’ Preferences

Contact Info

Product

Resources

About