Crowdsourcing-based multimedia subjective evaluations

Redi, Judith; Hoßfeld, Tobias; Korshunov, Pavel; Mazza, Filippo; Povoa, Isabel; Keimel, Christian

doi:10.1145/2506364.2506368

Cited by 40 publications

(7 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other side, next to the well-established lab tests, crowdsourcing tests are increasing in popularity for such perception tests. For example, it was already shown that crowd sourcing can be used for audio [65,66], video [16,33,82,87,100,105] and image quality assessment [18,37,39,90].…”

Section: Testing Frameworkmentioning

confidence: 99%

Quality assessment of higher resolution images and videos with remote testing

2023

View full text Add to dashboard Cite

In many research fields, human-annotated data plays an important role as it is used to accomplish a multitude of tasks. One such example is in the field of multimedia quality assessment where subjective annotations can be used to train or evaluate quality prediction models. Lab-based tests could be one approach to get such quality annotations. They are usually performed in well-defined and controlled environments to ensure high reliability. However, this high reliability comes at a cost of higher time consumption and costs incurred. To mitigate this, crowd or online tests could be used. Usually, online tests cover a wider range of end devices, environmental conditions, or participants, which may have an impact on the ratings. To verify whether such online tests can be used for visual quality assessment, we designed three online tests. These online tests are based on previously conducted lab tests as this enables comparison of the results of both test paradigms. Our focus is on the quality assessment of high-resolution images and videos. The online tests use AVrate Voyager, which is a publicly accessible framework for online tests. To transform the lab tests into online tests, dedicated adaptations in the test methodologies are required. The considered modifications are, for example, a patch-based or centre cropping of the images and videos, or a randomly sub-sampling of the to-be-rated stimuli. Based on the analysis of the test results in terms of correlation and SOS analysis it is shown that online tests can be used as a reliable replacement for lab tests albeit with some limitations. These limitations relate to, e.g., lack of appropriate display devices, limitation of web technologies, and modern browsers considering support for different video codecs and formats.

show abstract

Section: Testing Frameworkmentioning

confidence: 99%

Quality assessment of higher resolution images and videos with remote testing

2023

View full text Add to dashboard Cite

show abstract

“…Further, prior work found that non-expert members of the crowd are able to provide useful aesthetic preference judgements [19,44], and a sub-group of workers on online crowdsourcing platforms enjoy and actively seek out creative tasks [43]. In light of this, researchers and practitioners have explored ways of expanding the feedback elicitation from traditional co-located settings to a crowd of people online.…”

Section: Design Feedback From the Non-expert Crowdmentioning

confidence: 99%

CrowdUI: Supporting Web Design with the Crowd

Oppenlaender

Tiropanis

Hosio

2020

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

Web design is a complex and challenging task. It involves making many design decisions that materialise preconceived notions of user needs that may or may not be true. In this paper, we investigate supporting the co-design of a website with visual feedback elicited from the website's community of users. Website users can express their needs by rearranging and modifying the website's layout and design. To explore and validate this idea, we present CrowdUI, a web-based tool that enables members of the community of a website to visually express their design improvement ideas, frustrations and needs, and to send this feedback to the person in charge of designing or maintaining the website. CrowdUI is validated in a study with 45 users of a popular social media and networking website. Second, our qualitative evaluation with 60 experienced web developers shows that CrowdUI is able to elicit diverse and meaningful feedback. Put together, our results suggest that CrowdUI's approach constitutes a productive setting for eliciting visual feedback from the user community as a complement to traditional ways of eliciting feedback and participatory design. Finally, based on our experiences, we discuss a design space for crowdsourced web design and provide design recommendations for similar future tools. CCS Concepts: • Human-centered computing → Human computer interaction (HCI); User studies; User interface toolkits; • Information systems → Crowdsourcing.

show abstract

“…In the post-modernist paradigm, crowdsourcing is a subjective being. For example, Redi, Hoßfeld, Korshunov, Mazza, Povoa, and Keimel (2013) found this to be to "a powerful tool for gathering subjective ground truth for large multimedia collections". The foundation here is discourse and it is a tool for describing and presenting reality.…”

Section: The Post-modernist Paradigmmentioning

confidence: 99%

Crowdsourcing Creativity in Government: State of the Field in the Four Research Paradigms

2020

View full text Add to dashboard Cite

Creativity, innovation, openness and involving citizens in decision making belong to a set of efforts undertaken by the government. This is possible thanks to crowdsourcing that is a tool to communicate with citizens and that is a source of knowledge and that provides new, creative ideas. However, despite the research intensity in the area of crowdsourcing creativity in government, the research results obtained to date are still ambiguous and fragmentary. Research on crowdsourcing government is often limited to interpretive traditions. This gives an incomplete picture of government crowdsourcing since three additional research paradigms are omitted: interpretative, post-modern, and critical. Our ambition is to raise awareness about the presence of many paradigms in crowdsourcing government research. The aim of this article is to present crowdsourcing government from the perspective of four paradigms by Gibson Burrell and Gareth Morgan. We are trying to achieve this by presenting a review of research on crowdsourcing government taking into consideration four paradigms: positivist, interpretative, critical, and postmodern. We suggest that a single paradigm is not able to provide a complete picture of crowdsourcing government, and thus we seek interactions between the paradigms and postulate multi-paradigmatic research that may lead to further development of knowledge.

show abstract

Crowdsourcing-based multimedia subjective evaluations

Cited by 40 publications

References 18 publications

Quality assessment of higher resolution images and videos with remote testing

Quality assessment of higher resolution images and videos with remote testing

CrowdUI: Supporting Web Design with the Crowd

Crowdsourcing Creativity in Government: State of the Field in the Four Research Paradigms

Contact Info

Product

Resources

About