Four Years of FAccT: A Reflexive, Mixed-Methods Analysis of Research Contributions, Shortcomings, and Future Prospects

Abstract

Fairness, Accountability, and Transparency (FAccT) for socio-technical systems has been a thriving area of research in recent years. An ACM conference bearing the same name has been the central venue for scholars in this area to come together, provide peer feedback to one another, and publish their work. This reflexive study aims to shed light on FAccT's activities to date and identify major gaps and opportunities for translating contributions into broader positive impact. To this end, we utilize a mixed-methods research design. On the qualitative front, we develop a protocol for reviewing and coding prior FAccT papers, tracing their distribution of topics, methods, datasets, and disciplinary roots. We also design and administer a questionnaire to reflect the voices of FAccT community members and affiliates on a wide range of topics. On the quantitative front, we use the full text and citation network associated with prior FAccT publications to provide further evidence about topics and values represented in FAccT. We organize the findings from our analysis into four main dimensions: the themes present in FAccT scholarship, the values that underpin the work, the impact of the contributions both within academic circles and beyond, and the practices and informal norms of the community that has formed around FAccT. Finally, our work identifies several suggestions on directions for change, as voiced by community members. CCS CONCEPTS• Social and professional topics → Computing / technology policy; • Applied computing → Law, social and behavioral sciences.

Beyond Accuracy: A Critical Review of Fairness in Machine Learning for Mobile and Wearable Computing

“…For instance, Linxen et al [75] conducted a meta-study on CHI findings from 2016 to 2020, reporting that 73% of CHI studies are based on Western populations, representing less than 12% of the population worldwide, invariably making CHI "WEIRD", as it is based on the knowledge and ethics of people who are Western, Educated, Industrialized, Rich, and Democratic. Similarly, a recent meta-study on FAccT proceedings from 2018 to 2021 extracted research topics and identified community values, placing fairness and ML, bias in word embeddings, bias in vision, and racial disparities among the ten largest sub-communities within the conference [68]. Yet again, as highlighted in Introduction (Section 1) in line with prior work [68], "off-the-shelf" benchmark datasets are encountered in the majority of published work, while only a ∼ 10% of FAccT papers use original, empirical datasets, let alone UbiComp data.…”

“…Similarly, a recent meta-study on FAccT proceedings from 2018 to 2021 extracted research topics and identified community values, placing fairness and ML, bias in word embeddings, bias in vision, and racial disparities among the ten largest sub-communities within the conference [68]. Yet again, as highlighted in Introduction (Section 1) in line with prior work [68], "off-the-shelf" benchmark datasets are encountered in the majority of published work, while only a ∼ 10% of FAccT papers use original, empirical datasets, let alone UbiComp data.…”

An Empirical Analysis of Racial Categories in the Algorithmic Fairness Literature

“…While the FAccT community explicitly centers the ethical values of fairness, accountability, and transparency, it often overlooks other moral values such as respect and agency [22].…”

Confronting Power and Corporate Capture at the FAccT Conference

“…In 2021, a group of concerned members of the FAccT community came together to express concern about the lack of mandatory funding disclosures by conference authors, the inclusion of work involving controversial technologies without meaningful political analyses, the production of algorithmic audits sponsored and co-authored by employees at audited firms, and the over-representation in the conference of researchers from predominantly white institutions in the Global North [36]. Published in 2022, an anonymous survey of 60 FAccT authors, reviewers, and organizers also gave voice to concerns about corporate conflict of interest: Laufer et al [41] report:…”