Toward Three-Stage Automation of Annotation for Human Values

Ishita, Emi; Fukuda, Satoshi; Oga, Toru; Oard, Douglas W.; Fleischmann, Kenneth R.; Tomiura, Yoichi; Cheng, An Shou

doi:10.1007/978-3-030-15742-5_18

Cited by 4 publications

(2 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experiments evaluated four well-known supervised learning methods, namely, support vector machines, random forest, multi-layer perceptron, and logistic regression. This study used these methods due to an earlier study on the identification of human values in text documents that reported that a deep learning approach performs less well in smaller datasets and 'achieve [s] good results in data-rich settings' (Ishita et al 2019). All of these methods have been used in previous studies to classify the content of GitHub repositories (Golzadeh et al 2021;Arya et al 2019;Fan et al 2017;Eluri et al 2019;Trockman et al 2019;Munaiah et al 2017;Kikas et al 2016;Song and Chaparro 2020).…”

Section: Classification Experimentsmentioning

confidence: 99%

Integrating human values in software development using a human values dashboard

et al. 2023

View full text Add to dashboard Cite

There is a growing awareness of the importance of human values in software systems. However, limited tools are available to support the integration of human values during software development. Most of these tools are focused on concepts related to specific, well-known human values (e.g., privacy, security) in software engineering. This paper aims to (partially) address this gap by developing a human values dashboard. We conducted a multi-stage study to design, implement and evaluate a human values dashboard. First, an exploratory study was conducted by interviewing 15 software practitioners to investigate the possibility of using a human values dashboard to help address human values in software development, its potential benefits, and required features. Second, we experimented with four Machine Learning approaches to detect the presence of human values in issue discussions. We used the best approach to develop a human values dashboard for software development. The dashboard displays whether any human values are present in each issue discussion. Finally, we interviewed ten different practitioners to investigate the usefulness of the dashboard in practice. This study found that the human values dashboard could help raise awareness, focus attention, and prioritise issues based on the presence of values. This study also identified two potential challenges to the adoption of the dashboard. First, the possible incorrect issues description that can mislead the automated values identification in the dashboard. Second, the lack of willingness of a company to adopt the dashboard.

show abstract

Section: Classification Experimentsmentioning

confidence: 99%

Integrating human values in software development using a human values dashboard

et al. 2023

View full text Add to dashboard Cite

show abstract

“…In our research, we follow the three stage annotation process introduced by Ishita et al (2019) in which the first step is to determine which documents in the corpus are actually on topic 1 . That binary (yes/no) annotation task is our focus for the experiments that we report in this paper.…”

Section: Courpus and Annotaton Taskmentioning

confidence: 99%

Using text classification to improve annotation quality by improving annotator consistency

Ishita

Fukuda

Tomiura

et al. 2020

Proc Assoc Inf Sci Technol

Self Cite

View full text Add to dashboard Cite

This paper presents results of experiments in which annotators were asked to selectively reexamine their decisions when those decisions seemed inconsistent. The annotation task was binary topic classification. To operationalize the concept of annotation consistency, a text classifier was trained on all manual annotations made during a complete first pass and then used to automatically recode every document. Annotators were then asked to perform a second manual pass, limiting their attention to cases in which their first annotation disagreed with the text classifier. On average across three annotators, each working independently, 11% of first pass annotations were reconsidered, 46% of reconsidered annotations were changed in the second pass, and 71% of changed annotations agreed with decisions made independently by an experienced fourth annotator. The net result was that for an 11% average increase in annotation cost it was possible to increase overall chance corrected agreement with the annotation decisions of an experienced annotator (as measured by kappa) from 0.70 to 0.75.

show abstract