Detecting uncertain input using physiological sensing and behavioral measurements

Greis, Miriam; Karolus, Jakob; Schuff, Hendrik; Woźniak, Paweł W.; Henze, Niels

doi:10.1145/3152832.3152859

Cited by 9 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 3d shows that mean completion time monotonously decreases with increasing user certainty (with the exception of "strongly disagree"). 7 This confirms the findings of Greis et al (2017a) who investigate the effect of user uncertainty on behavioral measurements.…”

Section: Resultssupporting

confidence: 82%

“…We derive multiple dependent variables from the participants' ratings, namely completion time (Lim et al, 2009;Lage et al, 2019), several performance variables indicating how well they judged the correctness of the model (fraction of correct ratings, false positive ratio (FP), false negative ratio (FN), true positive ratio (TP), true negative ratio (TN), precision (P), recall (R) and F 1 values), agreement (fraction of model predictions that the users rate as correct (Bussone et al, 2015)), and overestimation (difference between agreement and true model accuracy (Nourani et al, 2019)). Furthermore, we collect the following variables in self-reports with five-point Likert scales: certainty of the participants (Greis et al, 2017a), completeness and helpfulness of the explanations (Nourani et al, 2019), trust of the participants in the model (Bussone et al, 2015), and satisfaction (Kulesza et al, 2012;Greis et al, 2017b).…”

Section: Dependent Variablesmentioning

confidence: 99%

See 1 more Smart Citation

F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering

Schuff

Adel

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

Explainable question answering systems predict an answer together with an explanation showing why the answer has been selected. The goal is to enable users to assess the correctness of the system and understand its reasoning process. However, we show that current models and evaluation settings have shortcomings regarding the coupling of answer and explanation which might cause serious issues in user experience. As a remedy, we propose a hierarchical model and a new regularization term to strengthen the answer-explanation coupling as well as two evaluation scores to quantify the coupling. We conduct experiments on the HOTPOTQA benchmark data set and perform a user study. The user study shows that our models increase the ability of the users to judge the correctness of the system and that scores like F 1 are not enough to estimate the usefulness of a model in a practical setting with human users. Our scores are better aligned with user experience, making them promising candidates for model selection.

show abstract

Section: Resultssupporting

confidence: 82%

Section: Dependent Variablesmentioning

confidence: 99%

F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering

Schuff

Adel

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Other measurements of interest to NLP research include completion time and bio-signals, such as gaze, EEG, ECG, and electrodermal activity. Bio-signals may provide insight into, for example, emotional state (Kim and André 2008), engagement (Renshaw, Stevens, and Denton 2009), stress (McDuff et al 2016), and user uncertainty (Greis et al 2017).…”

Section: Other Useful Metrics For Nlpmentioning

confidence: 99%

How to do human evaluation: A brief introduction to user studies in NLP

Schuff

Vanderlyn²,

Adel³

et al. 2023

Nat. Lang. Eng.

Self Cite

View full text Add to dashboard Cite

Many research topics in natural language processing (NLP), such as explanation generation, dialog modeling, or machine translation, require evaluation that goes beyond standard metrics like accuracy or F1 score toward a more human-centered approach. Therefore, understanding how to design user studies becomes increasingly important. However, few comprehensive resources exist on planning, conducting, and evaluating user studies for NLP, making it hard to get started for researchers without prior experience in the field of human evaluation. In this paper, we summarize the most important aspects of user studies and their design and evaluation, providing direct links to NLP tasks and NLP-specific challenges where appropriate. We (i) outline general study design, ethical considerations, and factors to consider for crowdsourcing, (ii) discuss the particularities of user studies in NLP, and provide starting points to select questionnaires, experimental designs, and evaluation methods that are tailored to the specific NLP tasks. Additionally, we offer examples with accompanying statistical evaluation code, to bridge the gap between theoretical guidelines and practical applications.

show abstract

“…Future research also needs to account for emotions and prior experiences, and how specific hazards are weighted based on affect (e.g., a volcano eruption may take cognitive precedence because it may be perceived as more threatening than rain). Uncertainty provokes anxiety ( Greis et al, 2017 ), and some decision-makers may become uncomfortable with depictions of uncertainty ( Hullman, 2019 ). If processing uncertainty of a single hazard already provokes anxiety, being presented with multiple uncertainties and their interactions could increase anxiety considerably, leading to situations where decision-makers are biased toward more risk-averse actions ( Lauriola et al, 2007 ; Fiala, 2017 ; Bourdeau-Brien and Kryzanowski, 2020 ).…”

Section: Multiple Hazard Uncertainty Communication Challenges and Paths Forwardmentioning

confidence: 99%

Multiple Hazard Uncertainty Visualization Challenges and Paths Forward

et al. 2021

View full text Add to dashboard Cite

Making decisions with uncertainty is challenging for the general public, policymakers, and even highly trained scientists. Nevertheless, when faced with the need to respond to a potential hazard, people must make high-risk decisions with uncertainty. In some cases, people have to consider multiple hazards with various types of uncertainties. Multiple hazards can be interconnected by location, time, and/or environmental systems, and the hazards may interact, producing complex relationships among their associated uncertainties. The interaction between multiple hazards and their uncertainties can have nonlinear effects, where the resultant risk and uncertainty are greater than the sum of the risk and uncertainty associated with individual hazards. Effectively communicating the uncertainties related to such complicated systems should be a high priority because the frequency and variability of multiple hazard events due to climate change continue to increase. However, the communication of multiple hazard uncertainties and their interactions remains largely unexplored. The lack of practical guidance on conveying multiple hazard uncertainties is likely due in part to the field’s vast expanse, making it challenging to identify entry points. Here, we offer a perspective on three critical challenges related to uncertainty communication across various multiple hazard contexts to galvanize the research community. We advocate for systematic considerations of multiple hazard uncertainty communication that focus on trade-offs between complexity and factors, including mental effort, trust, and usability.

show abstract

Detecting uncertain input using physiological sensing and behavioral measurements

Cited by 9 publications

References 20 publications

F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering

F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering

How to do human evaluation: A brief introduction to user studies in NLP

Multiple Hazard Uncertainty Visualization Challenges and Paths Forward

Contact Info

Product

Resources

About