How Do Users Revise Answers on Technical Q&A Websites? A Case Study on Stack Overflow

Abstract

To ensure the quality of its shared knowledge, Stack Overflow encourages users to revise answers through a badge system, which is based on quantitative measures (e.g., a badge is awarded after revising more than 500 answers). Prior studies show that badges can positively steer the user behavior on Stack Overflow (e.g., increasing user participation). However, little is known whether revision-related badges have a negative impact on the quality of revisions since some studies show that certain users may game incentive systems to gain rewards. In this study, we analyze 3,871,966 revision records that are collected from 2,377,692 Stack Overflow answers. We find that: 1) Users performed a much larger than usual revisions on the badge-awarding days compared to normal days; 25% of the users did not make any more revisions once they received their first revision-related badge. 2) Performing more revisions than usual in a single day increased the likelihood of such revisions being rolled back (e.g., due to undesired or incorrect revisions). 3) Users were more likely to perform text and small revisions if they performed many revisions in a single day. Our findings are concurred by the Stack Overflow community, and they highlight the need for changes to the current badge system in order to provide a better balance between the quality and quantity of revisions.

The Impact of Surface Features on Choice of (in)Secure Answers by Stackoverflow Readers

“…Such findings are consistent with studies of the dynamics of Stack Overflow and other Q&A websites, where influence gaming is a real phenomenon [27], and experiments have shown effective ways to promote post or answer visibility and manipulate poster engagement [28]. Moreover, Wang et al [29] studied answer revision on Stack Overflow, finding that gamification stimulated posters to revise answers, although very active posters mostly made minor textual revisions, likely related to the high likelihood of revisions being reverted if a poster is very active.…”

“…(P4. [15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33] We found affinity bias to explain this decision making, as some participants showed clear preference for solutions provided by others perceived to be like them, sharing some kind of shared identity (e.g., 'programmers'). Participants contextualized their rationale for wanting brief solutions with little explanation by their status "as coders", thereby preferring answers conforming to what they would expect from others in their social group.…”

An Empirical Study of Obsolete Answers on Stack Overflow

“…Answerers are encouraged to include relevant information about the valid version or the time of their knowledge 31. https://en.wikipedia.org/wiki/Template:Dead link 32. https://stackoverflow.com/posts/comments/61093395/ when creating answers.…”

“…The majority of prior studies define the quality of content on Stack Overflow from the presentation aspect (e.g., code and text) [ [30]. Wang et al analyzed how the badge system impacts answer revision on Stack Overflow, and found that the current system fails to consider the quality of revisions [31].…”

An empirical study of COVID-19 related posts on Stack Overflow: Topics and technologies

“…( Ortega et al, 2014 , Neshati, 2017 )). To ensure that competent answering is rewarded, SO is employing gamification mechanisms, distributing “badges” to esteemed responders and encouraging revisions and edits to posted questions and answers as well as the rapid answering of posts ( Papoutsoglou et al, 2020 , Wang et al, 2018a ).…”