Fine-tuning machine confidence with human relevance for video discovery

Inel, Oana; Aroyo, Lora

doi:10.1145/3539596

Cited by 1 publication

(1 citation statement)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Third, attention is drawn to formulating dataset artifacts that describe the collection purpose, method, and raters (Bender and Friedman 2018;Ramírez et al 2020;Gebru et al 2021;Díaz et al 2022). Fourth, existing datasets have been extensively judged, improved, and re-annotated based on empirical evidence suggesting that existing annotations are not representative anymore or contain ambiguous or erroneous annotations (Yun et al 2021;Inel and Aroyo 2019;Aroyo and Welty 2015).…”

Section: Introductionmentioning

confidence: 99%

Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection

Inel,

Draws,

Aroyo

2023

HCOMP

View full text Add to dashboard Cite

The rapid entry of machine learning approaches in our daily activities and high-stakes domains demands transparency and scrutiny of their fairness and reliability. To help gauge machine learning models' robustness, research typically focuses on the massive datasets used for their deployment, e.g., creating and maintaining documentation for understanding their origin, process of development, and ethical considerations. However, data collection for AI is still typically a one-off practice, and oftentimes datasets collected for a certain purpose or application are reused for a different problem. Additionally, dataset annotations may not be representative over time, contain ambiguous or erroneous annotations, or be unable to generalize across issues or domains. Recent research has shown these practices might lead to unfair, biased, or inaccurate outcomes. We argue that data collection for AI should be performed in a responsible manner where the quality of the data is thoroughly scrutinized and measured through a systematic set of appropriate metrics. In this paper, we propose a Responsible AI (RAI) methodology designed to guide the data collection with a set of metrics for an iterative in-depth analysis of the factors influencing the quality and reliability of the generated data. We propose a granular set of measurements to inform on the internal reliability of a dataset and its external stability over time. We validate our approach across nine existing datasets and annotation tasks and four content modalities. This approach impacts the assessment of data robustness used for AI applied in the real world, where diversity of users and content is eminent. Furthermore, it deals with fairness and accountability aspects in data collection by providing systematic and transparent quality analysis for data collections.

show abstract