Gamification strategies to improve the motivation and performance in accessibility information collection

Miyata, Akira; Murayama, Yusaku; Furuta, Akihiro; Okugawa, Kazuki; Ochiai, Keiichi; Murayama, Yuko

doi:10.1145/3491101.3519783

Cited by 5 publications

(1 citation statement)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In considering the performance of crowdsource workers, a lot of effort has been spent to introduce gamification of the labeling task [22,31,36,39,43], but we note that this requires significant overhead efforts to build the games, which may not be feasible when data collection is time-sensitive.…”

Section: Related Workmentioning

confidence: 99%

Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

Stureborg,

Dhingra,

Yang

2023

Preprint

View full text Add to dashboard Cite

Human data labeling is an important and expensive task at the heart of supervised learning systems. Hierarchies help humans understand and organize concepts. We ask whether and how concept hierarchies can inform the design of annotation interfaces to improve labeling quality and efficiency. We study this question through annotation of vaccine misinformation, where the labeling task is difficult and highly subjective. We investigate 6 user interface designs for crowdsourcing hierarchical labels by collecting over 18,000 individual annotations. Under a fixed budget, integrating hierarchies into the design improves crowdsource workers' F1 scores. We attribute this to (1) Grouping similar concepts, improving F1 scores by +0.16 over random groupings, (2) Strong relative performance on high-difficulty examples (relative F1 score difference of +0.40), and (3) Filtering out obvious negatives, increasing precision by +0.07. Ultimately, labeling schemes integrating the hierarchy outperform those that do not -achieving mean F1 of 0.70. CCS CONCEPTS• Human-centered computing → HCI design and evaluation methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

Stureborg,

Dhingra,

Yang

2023

Preprint

View full text Add to dashboard Cite

show abstract

Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

Stureborg

Dhingra

Jun

2023

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

On the Frontline During the COVID-19 Pandemic: Gender Inequality and Experiences of Healthcare Workers in Pakistan

Haroon,

Naeem,

Sajjad

et al. 2024

ACM J. Comput. Sustain. Soc.

View full text Add to dashboard Cite

This mixed methods study investigates the experiences of healthcare workers (HCWs) along gender lines during the Covid-19 pandemic in Lahore, the second most populous city in Pakistan. In-person semi-structured interviews ( n =62) and researcher-administered surveys ( n =631) were conducted with doctors and nurses in five private and public hospitals. The findings reveal that male and female HCWs shared experiences related to increased working hours, psychological burdens, and adverse financial impacts. However, female HCWs struggled more than male HCWs, as their responsibilities at home and in the workplace increased. Additionally, more female HCWs than their male peers reported experiencing occupational stress due to transportation issues, working during pregnancy, and discriminatory attitudes of the patients toward them. Building on the results from our study, we propose several technological and policy initiatives that can be adopted by governments and organizations, especially in countries like Pakistan, where women account for most of the healthcare workforce but continue to bear a heavier burden when balancing work and family.

show abstract

Gamification strategies to improve the motivation and performance in accessibility information collection

Cited by 5 publications

References 48 publications

Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

On the Frontline During the COVID-19 Pandemic: Gender Inequality and Experiences of Healthcare Workers in Pakistan

Contact Info

Product

Resources

About