Calibration with confidence: a principled method for panel assessment

MacKay, Robert S.; Kenna, Ralph; Low, Robert J.; Parker, Sarah J.

doi:10.1098/rsos.160760

Cited by 12 publications

(13 citation statements)

References 16 publications

(43 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The risks of equality bias can be countered if group members have equal competence at the task in hand [ 172 ]. Failing this, groups could decide to explicitly weight individual contributions [ 190 ]. Both are tricky: objective measures of competence are hard to come by and markers of reliability may be misleading.…”

Section: Recommendationsmentioning

confidence: 99%

Making better decisions in groups

2017

View full text Add to dashboard Cite

We review the literature to identify common problems of decision-making in individuals and groups. We are guided by a Bayesian framework to explain the interplay between past experience and new evidence, and the problem of exploring the space of hypotheses about all the possible states that the world could be in and all the possible actions that one could take. There are strong biases, hidden from awareness, that enter into these psychological processes. While biases increase the efficiency of information processing, they often do not lead to the most appropriate action. We highlight the advantages of group decision-making in overcoming biases and searching the hypothesis space for good models of the world and good solutions to problems. Diversity of group members can facilitate these achievements, but diverse groups also face their own problems. We discuss means of managing these pitfalls and make some recommendations on how to make better group decisions.

show abstract

Section: Recommendationsmentioning

confidence: 99%

Making better decisions in groups

2017

View full text Add to dashboard Cite

show abstract

“…(4) Miscalibration: Unfairness due to miscalibrations (e.g., strictness, leniency, extremal behavior) of the evaluator [19,52], and using ML+information theory to mitigate it. Auxilliary references: [5,17,19,35,39,39,[41][42][43]45].…”

Section: Outline Of Part 2 Of the Tutorialmentioning

confidence: 99%

SIGMOD 2020 Tutorial on Fairness and Bias in Peer Review and Other Sociotechnical Intelligent Systems

Shah

2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Questions of fairness and bias abound in all socially-consequential decisions pertaining to collection and management of data. Whether designing protocols for peer review of research papers, setting hiring policies, or framing research question in genetics, any data-management decision with the potential to allocate benefits or confer harms raises concerns about who gains or loses that may fail to surface in naively-chosen performance measures. Data science interacts with these questions in two fundamentally different ways: (i) as the technology driving the very systems responsible for certain social impacts, posing new questions about what it means for such systems to accord with ethical norms and the law; and (ii) as a set of powerful tools for analyzing existing data management systems, e.g., for auditing existing systems for various biases. This tutorial will tackle both angles on the interaction between technology and society visa -vis concerns over fairness and bias, particularly focusing on the collection and management of data. Our presentation will cover a wide range of disciplinary perspectives with the first part focusing on the social impacts of technology and the formulations of fairness and bias defined via protected characteristics and the second part taking a deep into peer review and distributed human evaluations, to explore other forms of bias, such as that due to subjectivity, miscalibration, and dishonest behavior.

show abstract

“…Although Lviv is not directly involved in these two impact-generation projects, we are quite willing to calibrate pubs in Ukraine and elsewhere. (We are nearly as happy to calibrate exams but also happy to refer interested parties to our paper or website [27].) After all, without the input of Ihor's team much of this research would not have been carried out so the indirect impact is also theirs.…”

Section: Other Dimensions Of Impactmentioning

confidence: 99%

Two cultures: ``them and us'' in the academic world

Kenna¹

2020

Condens. Matter Phys.

View full text Add to dashboard Cite

Impact of academic research onto the non-academic world is of increasing importance as authorities seek return on public investment. Impact opens new opportunities for what are known as "professional services": as scientometrical tools bestow some with confidence they can quantify quality, the impact agenda brings lay measurements to evaluation of research. This paper is partly inspired by the famous "two cultures" discussion instigated by C.P. Snow over 60 years ago. He saw a chasm between different academic disciplines and I see a chasm between academics and professional services, bound into contact through competing targets. This paper draws on my personal experience and experiences recounted to me by colleagues in different universities in the UK. It is aimed at igniting discussions amongst people interested in improving the academic world and it is intended in a spirit of collaboration and constructiveness. As a professional services colleague said, what I have to say "needs to be said". It is my pleasure to submit this paper to the Festschrift devoted to the 60th birthday of a renowned physicist, my good friend and colleague Ihor Mryglod. Ihor's role as leader of the Institute for Condensed Matter Physics in Lviv has been essential to generating some of the impact described in this paper and forms a key element of the story I wish to tell.

show abstract

Calibration with confidence: a principled method for panel assessment

Cited by 12 publications

References 16 publications

Making better decisions in groups

Making better decisions in groups

SIGMOD 2020 Tutorial on Fairness and Bias in Peer Review and Other Sociotechnical Intelligent Systems

Two cultures: ``them and us'' in the academic world

Contact Info

Product

Resources

About