Three cobblers worth the mastermind? The potential of ensemble in crowdsourced classification problems

Abstract

Classification problems, where the objective is to identify the class labels of given data points, are most often the subject of open contests, in which solvers compete for awards offered by solution seekers. Extant literature in open contests has studied both the winner‐takes‐all and top‐K award schemes, in which the award is granted to the best one and the best K solutions, respectively. However, in comparing these two schemes, researchers have never considered that under a top‐K award scheme, seekers may often benefit from aggregating the solutions as an ensemble, which could achieve a performance that is superior to any individual solution. The practice of ensemble is very common especially in classification and other data science problems, and in this work, we are the first to model it with the scope of deriving the optimal award scheme in open contests. Our results formally show that, given the option of ensemble, a top‐K award scheme may have the advantage to grant a higher profit to the seeker than the winner‐takes‐all award scheme. Further, the optimal number of awards is positively affected by contest parameters including the solvers' expertise, the number of solvers, and the return on effort, whereas it is negatively affected by the technical uncertainty when such uncertainty is sufficiently large. These findings are consistent with the increasing adoption of the top‐K award scheme in contests held on Kaggle and other similar platforms.

Online community's recognition and continued participation in idea competitions

“…Limited research has examined online communities in settings where individuals compete against one another with a personal objective of winning. On the other hand, prior studies on idea competitions have examined how contest design parameters, such as reward schemes (Stouras et al., 2022; Yan et al., 2022), contest duration (Chen et al., 2021), task attributes (Sun et al., 2012; Zheng et al., 2011), and the size of participant population (Boudreau et al., 2011; Terwiesch & Xu, 2008), affect participation behavior. While these studies have examined the interaction between contest sponsors and participants, the interaction between contest participants and the broader online community has received scant scholarly attention.…”