Utility is in the Eye of the User: A Critique of NLP Leaderboards

Abstract

Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models. While this leaderboard paradigm has been remarkably successful, a historical focus on performance-based evaluation has been at the expense of other qualities that the NLP community values in models, such as compactness, fairness, and energy efficiency. In this opinion paper, we study the divergence between what is incentivized by leaderboards and what is useful in practice through the lens of microeconomic theory. We frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. With this framing, we formalize how leaderboards -in their current form -can be poor proxies for the NLP community at large. For example, a highly inefficient model would provide less utility to practitioners but not to a leaderboard, since it is a cost that only the former must bear. To allow practitioners to better estimate a model's utility to them, we advocate for more transparency on leaderboards, such as the reporting of statistics that are of practical concern (e.g., model size, energy efficiency, and inference latency).

Dynabench: Rethinking Benchmarking in NLP

“…While crowdsourcing has been a boon for largescale NLP dataset creation (Snow et al, 2008;Munro et al, 2010), we ultimately want NLP systems to handle "natural" data (Kwiatkowski et al, 2019) and be "ecologically valid" (de Vries et al, 2020). Ethayarajh and Jurafsky (2020) analyze the distinction between what leaderboards incentivize and "what is useful in practice" through the lens of microeconomics. A natural setting for exploring these ideas might be dialogue (Hancock et al, 2019;Shuster et al, 2020).…”

“…We would be able to capture not only accuracy, for example, but also usage of computational resources, inference time, fairness, and many other relevant dimensions. This will in turn enable dynamic leaderboards, for example based on utility (Ethayarajh and Jurafsky, 2020). This would also allow for backward-compatible comparisons, not having to worry about the benchmark changing, and automatically putting new state of the art models in the loop, addressing some of the main objections.…”

What Will it Take to Fix Benchmarking in Natural Language Understanding?

“…Dodge et al (2019) lay out a set of best practices for results reporting, with a focus on the impact of hyperparameter tuning on model comparison. Ethayarajh and Jurafsky (2020) advocate for the inclusion of efficiency considerations in leaderboard design. Boyd-Graber and Börschinger (2020) describe ways that trivia competitions can provide a model for carefully-considered dataset design.…”

Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access

“…Thus, we must build systems that have the ability to instantaneously flag data abnormalities -both in the research phase and when translated into real clinical use -and pass these cases on for human review. Furthermore, rather than selecting a preferred machine learning model based on metrics such as accuracy, sensitivity, or correlation as is common in AI and NLP applications, we must seek to understand the underlying mechanisms and the context in which they will be used (Ethayarajh and Jurafsky, 2020;Hand, 2006).…”