To investigate the effect of competitive incentives under peer review, we designed a novel experimental setup called the Art Exhibition Game. We present experimental evidence of how competition introduces both positive and negative effects when creative artifacts are evaluated and selected by peer review. Competition proved to be a double-edged sword: on the one hand, it fosters innovation and product diversity, but on the other hand, it also leads to more unfair reviews and to a lower level of agreement between reviewers. Moreover, an external validation of the quality of peer reviews during the laboratory experiment, based on 23,627 online evaluations on Amazon Mechanical Turk, shows that competition does not significantly increase the level of creativity. Furthermore, the higher rejection rate under competitive conditions does not improve the average quality of published contributions, because more high-quality work is also rejected. Overall, our results could explain why many ground-breaking studies in science end up in lower-tier journals. Differences and similarities between the Art Exhibition Game and scholarly peer review are discussed and the implications for the design of new incentive systems for scientists are explained.peer review | competition | creativity | innovation | fairness C ompetitive incentives are an essential tool to manipulate effort and performance of human groups in many real-life situations (1, 2). Sport tournaments with huge prizes, goal-contingent rewards for employees, and lavish end-of-career bonuses for corporate CEOs are a few examples. However, the literature on incentives and rewards offers mixed evidence of how effective competitive incentives are in improving individual performance (3). In particular, external (monetary) incentives might crowd out intrinsic motivation, which results in no effect, or even a negative effect on individual effort (4-6). Moreover, intrinsic motivation might not only mediate effort, but might actually be necessary to achieve creative performance (7). Similarly, competitive pressure can reduce the performance of professional athletes, causing them to "choke under pressure" (8, 9). Finally, when higher interests are at stake, competition can also directly lead to negative consequences, such as uncooperative behavior and even sabotage (10,11).In this paper, we test the effect of competition in a peer review system. Peer review is a self-regulating system where individuals with similar competence (peers) assess the quality of each other's work. Peer review is widely used by governmental agencies and health care professionals, and it is one of the cornerstones of science. Scholarly peer review is a truly complex system: it involves many actors engaged in multiple roles encompassing various feedback loops (12). Thus far, its inherent complexity and the restricted access to data have made it difficult to investigate peer review. Empirical studies have documented that the review process has low levels of inter-referee agreement (13, 14), lacks reliabilit...
The purpose of this White Paper of the EU Support Action "Visioneer"(see www.visioneer.ethz.ch) is to address the following goals: 1. Develop strategies to quickly increase the objective knowledge about social and economic systems 2. Describe requirements for efficient large-scale scientific data mining of anonymized social and economic data 3. Formulate strategies how to collect stylized facts extracted from large data set 4. Sketch ways how to successfully build up centers for computational social science 5. Propose plans how to create centers for risk analysis and crisis forecasting 6. Elaborate ethical standards regarding the storage, processing, evaluation, and publication of social and economic data arXiv:1012.0178v5 [cs.CY] 26 Jul 2011• Real-time sensing and data collection ("reality mining" of weather data, environmental data, cooperativeness, compliance, trust, ...) could reduce mistakes and delays in decision-making, which often cause an inaccurate or unstable system management.In the past, data about the spreading of diseases and the gross national product, for example, became available with significant delays (often weeks, months or years). Recently, however, it has been discovered that the gross domestic product can be estimated in real-time by measuring the light intensity at night, which can be determined with satellite sensors [113]. Similarly, Google flu trends has been able to estimate epidemic spreading from search requests [111]. Furthermore, the "wisdom of crowds" effect [16] is more and more used by crowd sourcing approaches [26][27][28] such as prediction markets to estimate future economic developments, outcomes of elections, fashions, and socio-economic trends [29,30]. These areas are now becoming an own business branch, complementing classical consultancy, offering services like: real-time measurement of actual user activity, identification of trendsetters, opinion leaders, and innovators in social networks, trend prediction, trend tracking, etc. [196, 197]. They fit well with other Internet-based applications such as eBusiness [31,32] or eGovernance [33,34], for example. So far, it seems that the underlying technological innovations are mainly driven by businesses in the US. This, however, is partly a matter of legal regulations, which give American companies a large degree of freedom regarding what can be done with data of customers.With the right institutional settings and the knowledge accelerator proposed in another Visioneer white paper [3], Europe should be able to catch up with the breath-taking developments in this area, which will largely determine scientific and economic leadership in the future.The potential of the newly available data has been also articulated by leading scientific journals. Nature, for example, had a special issue on Big Data, while Science hat an issue on Complex Systems and Networks. Moreover, the National Science Foundation (NSF) has launched a large "Digging into data" initiative, providing not only large budgets, but also data (e.g. of science funding)...
The purpose of this White Paper of the EU Support Action "Visioneer" (see www.visioneer.ethz.ch) is to address the following goals:1. Identify new ways of publishing, evaluating, and reporting scientific progress.2. Promote ICT solutions to increase the awareness of new emerging trends.3. Invent tools to enhance Europe's innovation potential. 4.Develop new strategies to support a sustainable technological development.5. Lay the foundations for new ways to reach societal benefits and respond to industrial needs using ICT.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.