Communicating Context to the Crowd for Complex Writing Tasks

Salehi, Niloufar; Teevan, Jaime; Iqbal, Shamsi T.; Kamar, Ece

doi:10.1145/2998181.2998332

Cited by 26 publications

(20 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, future Ink tutorials can be utilized to help users file their taxes, write a paper, or decide on a purchase. Goals that are unclear at first and whose needs are uncommon may be better suited for the traditional market model where workers with relevant skills bid on tasks and engage in back-and-forth communication with the user to reach mutual understanding (Salehi et al 2017).…”

Section: Discussion and Future Workmentioning

confidence: 99%

See 1 more Smart Citation

Ink

Salehi

Bernstein

2018

ACM Trans. Comput.-Hum. Interact.

Self Cite

View full text Add to dashboard Cite

The web affords connections by which end-users can receive paid, expert help-such as programming, design, and writing-to reach their goals. While a number of online marketplaces have emerged to facilitate such connections, most end-users do not approach a market to hire an expert when faced with a challenge. To reduce friction in hiring from peer-to-peer expert crowd work markets, we propose Ink, a system that crowd workers can use to showcase their services by embedding tasks inside web tutorials-a common destination for users with information needs. Workers have agency to define and manage tasks, through which users can request their help to review or execute each step of the tutorial, for example, to give feedback on a paper outline, perform a statistical analysis, or host a practice programming interview. In a public deployment, over 25,000 pageviews led 168 tutorial readers to pay crowd workers for their services, most of whom had not previously hired from crowdsourcing marketplaces. A field experiment showed that users were more likely to hire crowd experts when the task was embedded inside the tutorial rather than when they were redirected to the same worker's Upwork profile to hire them. Qualitative analysis of interviews showed that Ink framed hiring expert crowd workers within users' well-established information seeking habits and gave workers more control over their work. CCS Concepts: • Human-centered computing → Computer supported cooperative work;

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

“…Tasks may be well-defined and repeatable, have unclear goals, or be in niche domains. Tasks that do not have clear goals require negotiation and back-and-forth communication between the user and crowd worker (Salehi et al 2017). Platforms such as Upwork support this type of communication.…”

Section: Inkmentioning

confidence: 99%

Ink

Salehi

Bernstein

2018

ACM Trans. Comput.-Hum. Interact.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Since then, many other crowdsourcing platforms have been developed, and research on crowdsourcing has flourished in the areas of data management, machine learning, and human computer interaction. There is a wide range of crowdsourcing tasks from simple ones like labeling images up to complex ones like collaboratively writing that involve multiple steps [86], [87]. Another important usage of crowdsourcing is data labeling (e.g., the ImageNet project), which we discuss in Section 3.2.…”

Section: Crowdsourcingmentioning

confidence: 99%

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective

Roh

Heo

Whang

2021

IEEE Trans. Knowl. Data Eng.

612

270

View full text Add to dashboard Cite

Data collection is a major bottleneck in machine learning and an active research topic in multiple communities. There are largely two reasons data collection has recently become a critical issue. First, as machine learning is becoming more widely-used, we are seeing new applications that do not necessarily have enough labeled data. Second, unlike traditional machine learning, deep learning techniques automatically generate features, which saves feature engineering costs, but in return may require larger amounts of labeled data. Interestingly, recent research in data collection comes not only from the machine learning, natural language, and computer vision communities, but also from the data management community due to the importance of handling large amounts of data. In this survey, we perform a comprehensive study of data collection from a data management point of view. Data collection largely consists of data acquisition, data labeling, and improvement of existing data or models. We provide a research landscape of these operations, provide guidelines on which technique to use when, and identify interesting research challenges. The integration of machine learning and data management for data collection is part of a larger trend of Big data and Artificial Intelligence (AI) integration and opens many opportunities for new research. TaskApproach Techniques Data discovery Sharing Collaborative Analysis [9]-[11] Web [12]-[17]

show abstract

“…The cognitive process model also provides one potential hurdle for effective microtask writing: writing tasks are often heavily context dependent [27]; it is difficult to imagine writing an introductory sentence without knowing the preceding and following sentences, the goals of the paragraphs, and the trajectory of the document as a whole. Salehi et al [27] explored communicating this context in crowdsourced writing. They found that one effective strategy for early drafts was providing high level feedback, while for later drafts highlighting areas of the response that were good or needed work was most effective.…”

Section: Writing Via Microtasksmentioning

confidence: 99%

“…We are in particular interested to understand writers' use of mobile microtasking for writing: what tasks do writers create for themselves to complete on a mobile device, how do they complete these tasks while mobile, and how does mobile work change task completion. Another hurdle for microtask writing is that completing writing microtasks requires context about the document [8,27]. Providing the entire document as context for a microtask can make navigation for the task difficult and frustrating, especially for a mobile phone or small screen [17].…”

Section: Introductionmentioning

confidence: 99%

Characterizing the Mobile Microtask Writing Process

August

Iqbal

Gamon

et al. 2020

22nd International Conference on Human-Computer Interaction With Mobile Devices and Services

Self Cite

View full text Add to dashboard Cite

The unique limitations of mobile environments make content creation and editing difficult. Microtasking-breaking down complex tasks into subtasks-requires shorter attention spans and quick interactions, making it suitable for mobile usage scenarios. Writing is an ideal process for mobile microtasking because of its many subgoals, but little is known about how writers can use this decomposition through the evolution of a document. In this paper we present findings from a controlled, week long study to characterize how writers use mobile microtasks while authoring a document. We found that writers created microtasks for editing and inserting information that generally required minimal writing. These tasks were especially well suited for mobile devices with writers completing tasks on commutes or while waiting for meetings. Writers who microtasked found it easy to interact with their document and complete tasks, writing and editing their document more overall compared to writers who instead edited their document directly on their phone.

show abstract

Communicating Context to the Crowd for Complex Writing Tasks

Cited by 26 publications

References 25 publications

Ink

Ink

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective

Characterizing the Mobile Microtask Writing Process

Contact Info

Product

Resources

About