Active Learning of Classification Models with Likert-Scale Feedback

Xue, Yanbing; Hauskrecht, Miloš

doi:10.1137/1.9781611974973.4

Cited by 16 publications

(7 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Este modelo corresponde a uma escala ordinária de 5 ou 7 pontos utilizada para medir o grau de concordância ou discordância dos avaliadores quanto a um determinado produto ou processo. A escala de Likert tem sido frequentemente utilizada na educação e pesquisa médicas, como por exemplo, na avaliação de opinião sobre treinamentos e treinadores médicos, bem como para se obter uma validação de performance após alguma intervenção educacional 8,9 .…”

Section: Discussionunclassified

Validation of the Handbook “Endocrinology for Primary Care Physicians”

Carballo¹,

Botelho²,

Andrade³

2020

Rev. Med. (São Paulo)

View full text Add to dashboard Cite

Introduction: handbooks’ elaboration and validation have been important as far as the concept of Evidence-Based Medicine is consolidated, allowing the creation of protocols or guidelines. Objective: to validate the handbook “Endocrinology for Primary Care Physicians”. Method: The handbook was developed in eight chapters (pre-diabetes, diabetes, gestational diabetes, dyslipidemias, thyroid nodules, hypo and hyperthyroidism, climacteric and menopause, overweight and obesity) based on international and current guidelines. The chapters dealt with the diagnosis, treatment, follow-up and referral of each disease and had their technical content reviewed by experts in endocrinology, and with experience in medical education. The handbook was validated by the Likert scale method, regarding the scientific adequacy of the content, as well as method-pedagogical adequacy. Results: the handbook was validated with 98% of scientific adequacy and 96% of method-pedagogical adequacy. Conclusion: the handbook “Endocrinology for Primary Care Physicians” was validated with an average of 97% of scientific and methodological-pedagogical adequacies.

show abstract

Section: Discussionunclassified

Validation of the Handbook “Endocrinology for Primary Care Physicians”

Carballo¹,

Botelho²,

Andrade³

2020

Rev. Med. (São Paulo)

View full text Add to dashboard Cite

show abstract

“…Some have been widely used in the previous active learning work ( Wine, Pima ) [Rashidi and Cook, 2011; Xue and Hauskrecht, 2017]; some have high-dimensional ( Ozone, Spam, Music ) feature space; and some are unbalanced in class distribution ( Seismic, Ozone, Wine unbalance ).…”

Section: Methodsmentioning

confidence: 99%

Hierarchical Active Learning with Group Proportion Feedback

Luo

Hauskrecht

2018

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence

Self Cite

View full text Add to dashboard Cite

Learning of classification models in practice often relies on nontrivial human annotation effort in which humans assign class labels to data instances. As this process can be very time consuming and costly, finding effective ways to reduce the annotation cost becomes critical for building such models. In this work we solve this problem by exploring a new approach that actively learns classification models from groups, which are subpopulations of instances, and human feedback on the groups. Each group is labeled with a number in [0,1] interval representing a human estimate of the proportion of instances with one of the class labels in this subpopulation. To form the groups to be annotated, we develop a hierarchical active learning framework that divides the whole population into smaller subpopulations, which allows us to gradually learn more refined models from the subpopulations and their class proportion labels. Our extensive experiments on numerous datasets show that our method is competitive and outperforms existing approaches for reducing the human annotation cost.

show abstract

“…In the human factors community, it is common to utilize a Likert scale (Likert, 1932), a set of ordinal categories representing ranges of values over an interval, to provide more coarse estimates of continuous values. Indeed, Likert scales have been used both as a means to elicit (Xue & Hauskrecht, 2017), and visualize (Guo et al, 2019), confidence information.…”

Section: Distance Functionsmentioning

confidence: 99%

What is Your Metric Telling You? Evaluating Classifier Calibration under Context-Specific Definitions of Reliability

Kirchenbauer¹,

Oaks²,

Heim³

2022

Preprint

View full text Add to dashboard Cite

Classifier calibration has received recent attention from the machine learning community due both to its practical utility in facilitating decision making, as well as the observation that modern neural network classifiers are poorly calibrated. Much of this focus has been towards the goal of learning classifiers such that their output with largest magnitude (the "predicted class") is calibrated. However, this narrow interpretation of classifier outputs does not adequately capture the variety of practical use cases in which classifiers can aid in decision making. In this work, we argue that more expressive metrics must be developed that accurately measure calibration error for the specific context in which a classifier will be deployed. To this end, we derive a number of different metrics using a generalization of Expected Calibration Error (ECE) that measure calibration error under different definitions of reliability. We then provide an extensive empirical evaluation of commonly used neural network architectures and calibration techniques with respect to these metrics. We find that: 1) definitions of ECE that focus solely on the predicted class fail to accurately measure calibration error under a selection of practically useful definitions of reliability and 2) many common calibration techniques fail to improve calibration performance uniformly across ECE metrics derived from these diverse definitions of reliability.

show abstract

Active Learning of Classification Models with Likert-Scale Feedback

Cited by 16 publications

References 16 publications

Validation of the Handbook “Endocrinology for Primary Care Physicians”

Validation of the Handbook “Endocrinology for Primary Care Physicians”

Hierarchical Active Learning with Group Proportion Feedback

What is Your Metric Telling You? Evaluating Classifier Calibration under Context-Specific Definitions of Reliability

Contact Info

Product

Resources

About