Modeling annotator behaviors for crowd labeling

Kara, Yunus; Genc, Gaye; Aran, Oya; Akarun, Lale

doi:10.1016/j.neucom.2014.10.082

Cited by 28 publications

(24 citation statements)

References 21 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Peng, Liu, Ihler, and Berger (2013) propose a domain-specific approach to the protein folding annotation problem by maximizing the log-likelihood of an exponential family mixture model of annotation similarities. Kara, Genc, Aran, and Akarun (2015) deal with the effects of diverse annotator behaviors on consensus estimation for continuous crowd-labeling problems. They also propose a scoring mechanism to determine annotator competence.…”

Section: Active Crowd-labeling For Categorical Annotation Problemsmentioning

confidence: 99%

“…For crowd consensus estimation, we employ the Consensus Bias Sensitive Model (M-CBS) of Kara et al (2015). The model assumes that a sample i has a single true rate (x i ) and an annotator produces an annotation (y k ) as a function of x i and their internal decision parameters.…”

Section: Crowd Consensus Estimationmentioning

confidence: 99%

“…Thus, we need to rate annotators based on their competences. As the annotator competence scoring mechanism, we refer to the formulation for M-CBS (Kara et al, 2015). The score is described as "the sum of the joint probabilities of all possible annotations that can be produced by an annotator and the most probable originating label for those annotations given the annotator parameters" (Kara et al, 2015).…”

Section: Who Annotates Better?mentioning

confidence: 99%

“…The score is derived by calculating the path integral of p(x, y|θ) along the linear mapping that defines the annotator, where d j and e j are the upper and lower limits of the path integral (for derivation details, see Kara et al, 2015).…”

Section: Who Annotates Better?mentioning

confidence: 99%

See 3 more Smart Citations

Actively Estimating Crowd Annotation Consensus

Kara¹,

Genc²,

Aran³

et al. 2018

jair

Self Cite

View full text Add to dashboard Cite

The rapid growth of storage capacity and processing power has caused machine learning applications to increasingly rely on using immense amounts of labeled data. It has become more important than ever to have fast and inexpensive ways to annotate vast amounts of data. With the emergence of crowdsourcing services, the research direction has gravitated toward putting the wisdom of crowds to better use. Unfortunately, spammers and inattentive annotators pose a threat to the quality and trustworthiness of the consensus. Thus, high quality consensus estimation from crowd annotated data requires a meticulous choice of the candidate annotator and the sample in need of a new annotation. Due to time and budget limitations, it is of utmost importance that this choice is carried out while the annotation collection is in progress. We call this process active crowd-labeling. To this end, we propose an active crowd-labeling approach for actively estimating consensus from continuous-valued crowd annotations. Our method is based on annotator models with unknown parameters, and Bayesian inference is employed to reach a consensus in the form of ordinal, binary, or continuous values. We introduce ranking functions for choosing the candidate annotator and sample pair for requesting an annotation. In addition, we propose a penalizing method for preventing annotator domination, investigate the explore-exploit trade-off for incorporating new annotators into the system, and study the effects of inducing a stopping criterion based on consensus quality. We also introduce the crowd-labeled Head Pose Annotations datasets. Experimental results on the benchmark datasets used in the literature and the Head Pose Annotations datasets suggest that our method provides high-quality consensus by using as few as one fifth of the annotations (∼ 80% cost reduction), thereby providing a budget and time-sensitive solution to the crowd-labeling problem.

show abstract

Section: Active Crowd-labeling For Categorical Annotation Problemsmentioning

confidence: 99%

Section: Crowd Consensus Estimationmentioning

confidence: 99%

Section: Who Annotates Better?mentioning

confidence: 99%

Section: Who Annotates Better?mentioning

confidence: 99%

See 2 more Smart Citations

Actively Estimating Crowd Annotation Consensus

Kara¹,

Genc²,

Aran³

et al. 2018

jair

Self Cite

View full text Add to dashboard Cite

show abstract

“…There are many reasons for the aforementioned behavior including the level of expertise, low-attention / low-concentration when they perform the task and there is always the bad intent of the annotators. Examples of annotators with bad intention can be spammers, dishonest or try to manipulate the system by answering in an unrelated or nonsense way [12]. In a research about crowdsourcing annotators' consistency Theodosiou et.…”

Section: Related Workmentioning

confidence: 99%

Instagram Hashtags as Image Annotation Metadata

Giannoulakis

Tsapatsoulis

2015

IFIP Advances in Information and Communication Technology

View full text Add to dashboard Cite

Abstract. Image tagging is an essential step for developing automatic image annotation methods that are based on the learning by example paradigm. However, manual image annotation, even for creating training sets for machine learning algorithms, requires hard effort and contains human judgment errors and subjectivity. Thus, alternative ways for automatically creating training examples, i.e., pairs of images and tags, are pursued. In this work we investigate whether tags accompanying photos in social media and especially the Instagram hashtags, provide a form of image annotation. If such a claim is proved then Instagram could be a very rich source of training data, easily collectable automatically, for the development of automatic image annotation techniques. Our hypothesis is that Instagram hashtags, and especially those provided by the photo owner / creator, express more accurately the content of a photo compared to the tags assigned to a photo during explicit image annotation processes like crowdsourcing. In this context, we explore the descriptive power of hashtags by examining whether other users would use the same, with the owner, hashtags to annotate an image. For this purpose a set of 30 randomly chosen, from Instagram, images were used as a dataset for our research. Then, one to four hashtags, considered as the most descriptive ones for the image in question, were selected among the hashtags used by the image owner. Three online questionnaires with ten images each were distributed to experiment participants in order to choose the best suitable hashtag for every image according to their interpretation. Results show that an average of 55% of the participants hashtag choices coincide with those suggested by the photo owners; thus, an initial evidence towards our hypothesis confirmation can be claimed.

show abstract