Workshops Held at the First AAAI Conference on Human Computation and Crowdsourcing: A Report

Josephy, Tatiana; Lease, Matthew; Paritosh, Praveen; Krause, Markus; Géorgescu, Mihai; Tjalve, Michael; Braga, Daniela

doi:10.1609/aimag.v35i2.2537

Cited by 15 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The sole reliance on paid-human labor for collecting street-level accessibility data can be insufficiently scalable and it remains expensive for creating a large dataset [97].…”

Section: Semi-automated Methods To Collect Accessibility Datamentioning

confidence: 99%

See 1 more Smart Citation

Scalable methods to collect and visualize sidewalk accessibility data for people with mobility impairments

Hara

2015

SIGACCESS Access. Comput.

View full text Add to dashboard Cite

Poorly maintained sidewalks pose considerable accessibility challenges for people with mobility impairments. Despite comprehensive civil rights legislation of Americans with Disabilities Act, many city streets and sidewalks in the U.S. remain inaccessible.The problem is not just that sidewalk accessibility fundamentally affects where and how people travel in cities, but also that there are few, if any, mechanisms to determine accessible areas of a city a priori.To address this problem, my Ph.D. dissertation introduces and evaluates new scalable methods for collecting data about street-level accessibility using a combination of crowdsourcing, automated methods, and Google Street View (GSV). My dissertation has four research threads. First, we conduct a formative interview study to establish a better understanding of how people with mobility impairments currently assess accessibility in the built environment and the role of emerging location-based technologies therein. The study uncovers the existing methods for assessing accessibility of physical environment and identify useful features of future assistive technologies. Second, we develop and evaluate scalable crowdsourced accessibility data collection methods. We show that paid crowd workers recruited from an online labor marketplace can find and label accessibility attributes in GSV with accuracy of 81%. This accuracy improves to 93% with quality control mechanisms such as majority vote. Third, we design a system that combines crowdsourcing and automated methods to increase data collection efficiency. Our work shows that by combining crowdsourcing and automated methods, we can increase data collection efficiency by 13% without sacrificing accuracy. Fourth, we develop and deploy a web tool that lets volunteers to help us collect the street-level accessibility data from Washington, D.C.As of writing this dissertation, we have collected the accessibility data from 20% of the streets in D.C. We conduct a preliminary evaluation on how the said web tool is used.Finally, we implement proof-of-concept accessibility-aware applications with accessibility data collected with the help of volunteers.My dissertation contributes to the accessibility, computer science, and HCI communities by: (i) extending the knowledge of how people with mobility impairments interact with technology to navigate in cities; (ii) introducing the first work that demonstrates that GSV is a viable source for learning about the accessibility of the physical world; (iii) introducing the first method that combines crowdsourcing and automated methods to remotely collect accessibility information; (iv) deploying interactive web tools that allow volunteers to help populate the largest dataset about street-level accessibility of the world; and (v) demonstrating accessibility-aware applications that empower people with mobility impairments.

show abstract

“…The sole reliance on paid-human labor for collecting street-level accessibility data can be insufficiently scalable and it remains expensive for creating a large dataset [97].…”

Section: Semi-automated Methods To Collect Accessibility Datamentioning

confidence: 99%

“…However, even paid micro task crowdsourcing can be insufficiently scalable, and it remains expensive for creating a large dataset [97].…”

Section: Introductionmentioning

confidence: 99%

Scalable methods to collect and visualize sidewalk accessibility data for people with mobility impairments

Hara

2015

SIGACCESS Access. Comput.

View full text Add to dashboard Cite

show abstract

“…In recent years, several evaluation activities have focused on crowdsourcing for groundtruth creation, as witnessed by the TREC Crowdsourcing track series 1 from 2011 to 2013 [Smucker et al, 2013[Smucker et al, , 2014, the MediaEval Crowdsourcing tracks 2 in 2013 and 2014 [Loni et al, 2013;Yadati et al, 2014], or the CrowdScale 2013 Shared Task Challenge 3 [Josephy et al, 2014]. There is also a growing interest and attention about how crowdsourcing affects the repeatability and reproducibility of IR experiments [Blanco et al, 2011;Ferro, 2017;Ferro et al, 2016a].…”

Section: Crowdsourcing For Ground-truth Creationmentioning

confidence: 99%

Exploiting User Signals and Stochastic Models to Improve Information Retrieval Systems and Evaluation

Maistro

2019

SIGIR Forum

View full text Add to dashboard Cite

The leitmotiv throughout this thesis is represented by IR evaluation. We discuss different issues related to effectiveness measures and novel solutions that we propose to address these challenges. We start by providing a formal definition of utility-oriented measurement of retrieval effectiveness, based on the representational theory of measurement. The proposed theoretical framework contributes to a better understanding of the problem complexities, separating those due to the inherent problems in comparing systems, from those due to the expected numerical properties of measures. We then propose AWARE, a probabilistic framework for dealing with the noise and inconsistencies introduced when relevance labels are gathered with multiple crowd assessors. By modeling relevance judgements and crowd assessors as sources of uncertainty, we directly combine the performance measures computed on the ground-truth generated by each crowd assessor, instead of adopting a classification technique to merge the labels at pool level. Finally, we investigate evaluation measures able to account for user signals. We propose a new user model based on Markov chains, that allows the user to scan the result list with many degrees of freedom. We exploit this Markovian model in order to inject user models into precision, defining a new family of evaluation measures, and we embed this model as objective function of an LtR algorithm to improve system performances. Table of contents List of figures ix List of tables xi Nomenclature xv Nomenclature xv MP Markov Precision MV Majority Vote nCG normalized Cumulated Gain nDCG normalized Discounted Cumulated Gain nMCG Normalized Markov Cumulated Gain RBP Rank-Biased Precision SERP Search Engine Result Page SMART System for the Mechanical Analysis and Retrieval of Text T REC Text REtrieval Conference With the development of IR systems it became necessary to design a framework to evaluate and compare different retrieval strategies. Indeed, progress and innovation are driven by experiments, but experimentation is useless without an objective evaluation measure that allow researchers to detect the improvements and identify the successful strategies. In Chapter 4 we propose our upstream approach called Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) [Ferrante et al., 2017]. AWARE is defined as an upstream approach because it directly combines the scores of the evaluation measures computed from the relevance labels of each assessor, instead of merging the labels and then computing the measures. The focus is then shifted from the documents and the labels to the evaluation measures. This allows to account for the error introduced by incorrect labels and to develop a framework which estimates performance measures in a way more robust to crowd assessors. Up to now we provide a formal definition of utility-oriented measurement of retrieval effectiveness and we developed an approach to estimate performance measures when there is some noise due to crowd assessors variability. Thus the effective...

show abstract

“…We will update the existing website for the previous workshops: http://www.worklearn.org/. The website as is provides information on the 2014 Workshop [6].…”

Section: Websitementioning

confidence: 99%

Connecting Online Work and Online Education at Scale

Prpić¹

2017

Preprint

View full text Add to dashboard Cite

Education is one of the eight Millennium Development Goals (MDG) of the United Nations. Considerable interest has been displayed in online education at scale, a new arising concept to realize this goal. Yet connecting online education to real jobs is still a challenge. This CHI workshop bridges this gap by bringing together groups and insights from related work at HCOMP, CSCW, and Learning at Scale. The workshop aims at providing opportunities for groups not yet in the focus of online education, exemplified by students who have not have equal access to higher education, compared to typical students in MOOCs.The focus is on theoretical and empirical connections between online education and job opportunities which can reduce the financial gap, by providing students with an income during their studies. The workshop explores the technological analogue of the concept of ‘apprenticeship’, long established in the European Union, and education research [2]. This allows students to do useful work as an apprentice during their studies. This workshop tackles such questions by bringing together participants from industry (e.g., platforms similar to Upwork, Amazon Mechanical Turk); education, psychology, and MOOCs (e.g., attendees of AERA, EDM, AIED, Learning at Scale); crowdsourcing and collaborative work (e.g., attendees of CHI, CSCW, NIPS, AAAI’s HCOMP).Krause, M., Hall, M., Williams, J. J., Paritosh, P., Prpić, J., & Caton, S. (2016, May). Connecting Online Work and Online Education at Scale. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (pp. 3536-3541). ACM.

show abstract

Workshops Held at the First AAAI Conference on Human Computation and Crowdsourcing: A Report

Cited by 15 publications

References 0 publications

Scalable methods to collect and visualize sidewalk accessibility data for people with mobility impairments

Scalable methods to collect and visualize sidewalk accessibility data for people with mobility impairments

Exploiting User Signals and Stochastic Models to Improve Information Retrieval Systems and Evaluation

Connecting Online Work and Online Education at Scale

Contact Info

Product

Resources

About