A comparative assessment of answer quality on four question answering sites

Abstract

Question answering (Q&A) sites, where communities of volunteers answer questions, may provide faster, cheaper, and better services than traditional institutions. However, like other Web 2.0 platforms, user-created content raises concerns about information quality. At the same time, Q&A sites may provide answers of different quality because they have different communities and technological platforms. This paper compares answer quality on four Q&A sites: Askville, WikiAnswers, Wikipedia Reference Desk, and Yahoo! Answers. Findings indicate that: (1) similar collaborative processes on these sites result in a wide range of outcomes, and significant differences in answer accuracy, completeness, and verifiability were evident; (2) answer multiplication does not always result in better information; it yields more complete and verifiable answers but does not result in higher accuracy levels; and (3) a Q&A site’s popularity does not correlate with its answer quality, on all three measures.

Noise or Quality? Cross-Nested Hierarchical Effects of Culture on Online Ratings

“…First, prior research has theorized or assumed that ratings are indicators of quality (Fichman, 2011;Sun, 2012). The underlying logic behind this conjecture is that crowds of individuals have the ability to make quality determinations or make decisions more accurately than experts even though the individuals in the crowd may use their own criteria and non-standardized processes to make ratings (Surowiecki, 2004).…”

“…Using a three-point scale (-1 for low quality, 0 for neither high nor low quality (or not applicable), and 1 for high quality), three PhD researchers coded each post along each of Fichman's (2011) dimensions of quality (accuracy, completeness, and verifiability). After coding, I summed the values across all three dimensions, which resulted in a scale from -3 (low quality on all three dimensions) to +3 (high quality on all three dimensions).…”

“…I determined contribution quality using the dimensions of quality defined by Fichman (2011). Fichman (2011 examined contribution quality based on the accuracy (correctness of a statement), completeness (thoroughness of the statement), and verifiability (proving a reference to a source) of each answer in her study.…”

“…At least in theory, ratings should be a somewhat objective indication of the quality of the product, service, or user-generated content being rated, which prior literature often assumes is the case (Fichman, 2011;Sun, 2012). Prior research, however, has offered minimal empirical evidence supporting or contradicting this quality assumption.…”

Rating crowdsourced annotations: evaluating contributions of variable quality and completeness

“…In situations in which there are labels derived from multiple contributors, it is possible to use, for example, the majority vote system to determine a label, but this treats all annotators equally and is of little value if the vast majority of the contributions are of low quality (Raykar and Yu 2012). If the identity of a contributor is known, it may be possible to build up levels of trust in their work over time and rate by reputation (Fichman 2011). There are still, however, concerns such as how a rated contributor copes with cases of new, previously unseen classes.…”