One size does not fit all: A study of badge behavior in stack overflow

Abstract

Badges are endemic to online interaction sites, from question and answer (Q&A) websites to ride sharing, as systems for rewarding participants for their contributions. This article studies how badge design affects people's contributions and behavior over time. Past work has shown that badges “steer” people's behavior toward substantially increasing the amount of contributions before obtaining the badge, and immediately decreasing their contributions thereafter, returning to their baseline contribution levels. In contrast, we find that the steering effect depends on the type of user, as modeled by the rate and intensity of the user's contributions. We use these measures to distinguish between different groups of user activity, including users who are not affected by the badge system despite being significant contributors to the site. We provide a predictive model of how users change their activity group over the course of their lifetime in the system. We demonstrate our approach empirically in three different Q&A sites on Stack Exchange with hundreds of thousands of users, for two types of activities (editing and voting on posts).

The phantom steering effect in Q&A websites

“…Our results revealed the following insights: First, more than 60% of the users are not steered, in that they exhibit a consistent activity rate in SO that is not effected by the badge. We prove that a "bump" in activity that is conveyed by prior work arises as an artifact of centering the data on the day of badge achievement [1,28]. We call this phenomenon the Phantom Steering Effect.…”

“…Other studies have independently confirmed that the presence of online badges increases the probability that a user will act in a manner to achieve the badge, as well as the rate at which the user will perform those actions [6,14,21,28]. Kusmierczyk and Gomez-Rodriguez [21] highlight the importance of modelling the "utility heterogeneity" among the users but they study badges which have a threshold of 1 action and do not characterise how one might change one's behaviour in the presence of the badge incentive.…”

“…Second, about 5-30% of users are steered, in that they dramatically increase their rate of activity prior to achieving the badge. It is the effect that this small population of steered users has on aggregate measures that has led to the previous and broader claims of steering [1,24,28]. Third, a large portion of these steered users decrease their activity rate beyond what is claimed in prior work [1], reaching close to 0 after the badge has been achieved.…”

“…Kusmierczyk and Gomez-Rodriguez [21] highlight the importance of modelling the "utility heterogeneity" among the users but they study badges which have a threshold of 1 action and do not characterise how one might change one's behaviour in the presence of the badge incentive. Yanovsky et al [28] study the presence of different populations within the SO database by employing a clustering routine. They discovered notably different responses to the badge based on the cluster that a user belongs to.…”

Gamification effects on users' motivation to contribute knowledge in a Portuguese Q&A community

“…Once a user receives a badge by achieving a certain goal, a more challenging badge appears as his/her next goal to achieve. Badges also track users' performance such as points, allowing them to compare their performance with others (Rigby and Ryan, 2011; Yanovsky et al , 2020). Thus, users can use badges to set goals, take up challenges and make social comparisons.…”