Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool

Lee, Sang Won; Krosnick, Rebecca; Park, Sun Young; Keelean, Brandon; Vaidya, Sach; O’Keefe, Stephanie D.; Lasecki, Walter S.

doi:10.1145/3274373

Cited by 23 publications

(7 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous work showed such disagreements in part-of-speech tagging [36], disagreements because of lack of labelers' expertise in the area [29,41], and disagreements on controversial matters [21] (e.g., use of "annexation" vs "liberation" when describing a conflict based on the political and ideological stand of the editor). Disagreements on the question of neutrality is a reflection of the pluralist voices of a society [9].…”

Section: Resultsmentioning

confidence: 99%

Automatically Labeling Low Quality Content on Wikipedia By Leveraging Patterns in Editing Behaviors

Asthana

Thommel

Halfaker

et al. 2021

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

Wikipedia articles aim to be definitive sources of encyclopedic content. Yet, only 0.6% of Wikipedia articles have high quality according to its quality scale due to insufficient number of Wikipedia editors and enormous number of articles. Supervised Machine Learning (ML) quality improvement approaches that can automatically identify and fix content issues rely on manual labels of individual Wikipedia sentence quality. However, current labeling approaches are tedious and produce noisy labels. Here, we propose an automated labeling approach that identifies the semantic category (e.g., adding citations, clarifications) of historic Wikipedia edits and uses the modified sentences prior to the edit as examples that require that semantic improvement. Highest-rated article sentences are examples that no longer need semantic improvements. We show that training existing sentence quality classification algorithms on our labels improves their performance compared to training them on existing labels. Our work shows that editing behaviors of Wikipedia editors provide better labels than labels generated by crowdworkers who lack the context to make judgments that the editors would agree with.

show abstract

Section: Resultsmentioning

confidence: 99%

Automatically Labeling Low Quality Content on Wikipedia By Leveraging Patterns in Editing Behaviors

Asthana

Thommel

Halfaker

et al. 2021

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

show abstract

“…Also, differences in expertise levels can impede crowd workers' collaborative effectiveness. Specifically, a crowd worker team's use of modifiable shared artifacts to scaffold ideas between team members and crowd workers was shown to improve collaboration (Lee et al, 2018). Research has found that peer production group members use other's history of project activity to form impressions of other's expertise (Marlow, Dabbish, & Herbsleb, 2013).…”

Section: Technology As Creation Mediummentioning

confidence: 99%

Leading teams in the digital age: Four perspectives on technology and what they mean for leading teams

Larson

DeChurch

2020

The Leadership Quarterly

184

134

View full text Add to dashboard Cite

Digital technologies are changing the nature of teamwork in ways that have important implications for leadership. Though conceptually rich and multi-disciplinary, much of the burgeoning work on technology has not been fully integrated into the leadership literature. To fill this gap, we organize existing work on leadership and technology, outlining four perspectives: (1) technology as context, (2) technology as sociomaterial, (3) technology as creation medium, and (4) technology as teammate. Each technology perspective makes assumptions about how technologies affect teams and the needs for team leadership. Within each perspective, we detail current work on leading teams. This section takes us from virtual teams to new vistas posed by leading online communities, crowds, peer production groups, flash teams, human-robot teams, and humanartificial intelligence teams. We identify 12 leadership implications arising from the ways digital technologies affect organizing. We then leverage our review to identify directions for future leadership research and practice.

show abstract

“…Project Planning [25] Requirement Management [20] Programming/Coding [14,21,37] Software Testing [7,17] Software Integration [7] Debugging [40] Software Evolution [40] Generic development [8][9][10][11][12][13]15,16,18,19,[22][23][24][26][27][28][29][30][31][32][33][34][35][36]38,39,41,42] The activities mentioned in Table 5 are the only software engineering activities where freelancers were involved as reported by the literature studies. The formulation of the category labels and classification of the studies in these categories were made on the basis of the software engineering activity that the literature studies have focused on.…”

Section: Software Development Area Research Studymentioning

confidence: 99%

Freelancers in the Software Development Process: A Systematic Mapping Study

2020

View full text Add to dashboard Cite

[Context] Freelancers could catalyze the software development process by providing their niche skills to generate high quality outputs. They could help companies (including startups) to foster innovations by suggesting creative ideas and providing their expertise in implementing them (for instance, designing solutions, coding solutions etc.). Freelancers could effectively and efficiently work as a virtual member of the software development team. The company must make informed decisions about which task to allot to the freelancer, which freelancer to select, pricing the task, and evaluating the submitted work. On the other hand, the freelancer should make an informed decision about evaluating the monetary value of the task to be charged, trusting the requester, analyzing the skills requirement of the task (finding matches between skill requirement and skills processed), selecting the best task, and maintaining the highest level of reputation. However, the literature does not provide freelancers and the companies the guidelines that support their decision making. However, if freelancers are selected carefully for the most suitable task, the companies will benefit a lot in terms of improved software development metrics. [Objectives] The objective of this paper is to provide the research community the research trends in freelancer-supported software development. This helps to understand that which software development areas have higher concentrations of research efforts, which area has the support of empirical evidence to support management decision makings, and which area requires the research attention. [Method] The systematic study is conducted by planning the mapping protocol, executing the protocol, and reporting the findings using various visualization tools like bar charts and pie charts. The search process was planned to be executed using set of inclusion and exclusion conditions on four bibliographic databases (IEEExplore, Springerlink, Sciencedirect, and ACM digital library). The relevant papers are selected by applying inclusion and exclusion conditions. The google citations of the relevant papers are subject to the inclusion and exclusion conditions again to include the more relevant papers. Finally, the systematic schema was created and populated after analyzing the studies abstracts. [Results] The results indicate the following (a) The research focus is on generic software development (78%) rather on individual life cycle activities. (b) The number of empirical studies is limited (25%). (c) A number of studies proposing solutions and evaluating on live cases in industrial settings are missing from the literature. This is in comparison to the validation approaches (72%) i.e., solutions tested in laboratory settings. (d) At present, the literature has limited ability to provide the software companies (including startups) with the guidelines (in the form of opinions and experience reports) for involving freelancers in the software development process. (e) The reported challenges include Collaboration and Coordination (33%), Developer Recommendation (or selection) (19%), Team Formulation (14%), Task Recommendation (allocation) (14%), Task Decomposition (11%), Privacy and Security (Confidentiality) (11%), Budget Estimation (8%), Recognition (8%), Trust Issues (8%), Market Dynamism (6%), Intellectual Property Issues (6%), Participation of Crowd Worker (6%), and Capacity Utilization (3%). These challenges are highly interactive, and each challenge impacts all other challenges. (e) Recent focus of the researchers (total 7 studies in 2019) is on generic software development handling the collaboration and coordination (3 studies out of 7), Developer recommendation (2 studies out of 7), and task recommendation (2 studies out of 7). [Conclusion] The freelancer-driven software engineering research area has got the attraction of the researchers, but it will take a long time to gain maturity. This puts an urgent call for more empirical studies and evaluation-based solution research that could help companies (including startups) to foster innovations. Further, the research focus should be well distributed among the various development phases to address the unique challenges associated with individual activities. The accurate management of the freelancer in the software development could help companies and startups to foster innovations and remain competitive in the marketplace.

show abstract

Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool

Cited by 23 publications

References 50 publications

Automatically Labeling Low Quality Content on Wikipedia By Leveraging Patterns in Editing Behaviors

Automatically Labeling Low Quality Content on Wikipedia By Leveraging Patterns in Editing Behaviors

Leading teams in the digital age: Four perspectives on technology and what they mean for leading teams

Freelancers in the Software Development Process: A Systematic Mapping Study

Contact Info

Product

Resources

About