Learning representations for quality estimation of crowdsourced submissions

Lyu, Shanshan; Ouyang, Wentao; Shen, Huawei; Cheng, Xueqi

doi:10.1016/j.ipm.2018.10.020

Cited by 8 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, there are many limitations due to the existence of the local optimality of the solutions. Recently, another work has focused on employing crowds for quality estimation from unstructured texts instead of traditional text documents [29].…”

Section: Related Workmentioning

confidence: 99%

A TOPSIS-Inspired Ranking Method Using Constrained Crowd Opinions for Urban Planning

Chatterjee

Lim

2022

Entropy

View full text Add to dashboard Cite

Crowdsourcing has become an important tool for gathering knowledge for urban planning problems. The questions posted to the crowd for urban planning problems are quite different from the traditional crowdsourcing models. Unlike the traditional crowdsourcing models, due to the constraints among the multiple components (e.g., multiple locations of facilities) in a single question and non-availability of the defined option sets, aggregating of multiple diverse opinions that satisfy the constraints as well as finding the ranking of the crowd workers becomes challenging. Moreover, owing to the presence of the conflicting nature of features, the traditional ranking methods such as the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) cannot always be feasible as the optimal solutions in terms of multiple objectives cannot occur simultaneously for the conflicting cases (e.g., benefit and cost criteria) for urban planning problems. Therefore, in this work, a multi-objective approach is proposed to produce better compromised solutions in terms of conflicting features from the general crowd. In addition, the solutions are employed to obtain a proper ideal solution for ranking the crowd. The experimental results are validated using two constrained crowd opinion datasets for real-world urban planning problems and compared with the state-of-the-art TOPSIS models.

show abstract

Section: Related Workmentioning

confidence: 99%

A TOPSIS-Inspired Ranking Method Using Constrained Crowd Opinions for Urban Planning

Chatterjee

Lim

2022

Entropy

View full text Add to dashboard Cite

show abstract

“…This solution however would require some major modifications in the way that requesters and workers interact with each other. Shanshan et al [18] proposed an algorithm, that computed a proper ordering of workers' submissions based on the estimated quality of the solutions. Unlike our approach, this mechanism assumed availability of requester's feedback for all the solutions provided by the workers in the past.…”

Section: Related Workmentioning

confidence: 99%

Picking Peaches or Squeezing Lemons: Selecting Crowdsourcing Workers for Reducing Cost of Redundancy

Adamska

Juźwin

Wierzbicki

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Crowdsourcing (CS) platforms are constantly gaining attention from both researchers and companies, due to the offered possibility of utilizing the "wisdom of crowds" in order to solve a great variety of problems. Despite the obvious advantages of such mechanisms, there are also numerous concerns regarding the quality assurance of work results produced by a large group of anonymous workers. In this work, we use data gathered from a real CS platform in order to study the performance of various approaches to worker selection, including a novel approach that utilizes automatic real-time monitoring of the produced results. We compare the performance of these mechanisms with respect to both overall cost and the accuracy of the final results to benchmark algorithms that aggregate results from a group of workers without pre-selection, relying solely on the "wisdom of crowd". We find that our novel approach is capable of reducing the cost of obtaining high-quality aggregated results by a factor of four, without sacrificing quality.

show abstract

Idea generation performance in open innovation communities: The role of user interaction

Wang,

Qi,

Zhou

et al. 2024

Information & Management

View full text Add to dashboard Cite

Learning representations for quality estimation of crowdsourced submissions

Cited by 8 publications

References 7 publications

A TOPSIS-Inspired Ranking Method Using Constrained Crowd Opinions for Urban Planning

A TOPSIS-Inspired Ranking Method Using Constrained Crowd Opinions for Urban Planning

Picking Peaches or Squeezing Lemons: Selecting Crowdsourcing Workers for Reducing Cost of Redundancy

Idea generation performance in open innovation communities: The role of user interaction

Contact Info

Product

Resources

About