Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly

Whitehead, Spencer; Petryk, Suzanne; Shakib, Vedaad; Darrell, Trevor; Rohrbach, Anna; Rohrbach, Marcus

doi:10.1007/978-3-031-20059-5_9

Cited by 9 publications

(1 citation statement)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work follows in this spirit, but asks the user to confirm a parse rather than generating questions for the user to answer; we also operate in a different parsing domain, preventing us from using Yao et al's system as a baseline. DidYouMean also relates broadly to selective prediction, where a model is expected to abstain from making decisions at low confidence (Chow, 1957;El-Yaniv et al, 2010;Varshney et al, 2022;Xin et al, 2021;Whitehead et al, 2022). Our system extends selective prediction's setting by including a human-inthe-loop.…”

Section: Related Workmentioning

confidence: 97%

Did You Mean...? Confidence-based Trade-offs in Semantic Parsing

Stengel-Eskin,

Van Durme

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

We illustrate how a calibrated model can help balance common trade-offs in task-oriented parsing. In a simulated annotator-in-the-loop experiment, we show that well-calibrated confidence scores allow us to balance cost with annotator load, improving accuracy with a small number of interactions. We then examine how confidence scores can help optimize the tradeoff between usability and safety. We show that confidence-based thresholding can substantially reduce the number of incorrect lowconfidence programs executed; however, this comes at a cost to usability. We propose the DidYouMean system (cf. Fig. 1) which better balances usability and safety by rephrasing low-confidence inputs.

show abstract