Post‐publication filtering and evaluation: Faculty of 1000

Abstract

Faculty of 1000 (http://www.facultyof1000.com) is a new on‐line literature awareness and assessment service of research papers, on the basis of selections by 1400 of the world's top biologists, that combines metrics with judgement. The service offers a systematic and comprehensive form of post‐publication peer review that focuses on the best papers regardless of the journal in which they are published. It is now possible to draw some conclusions about how this new form of post‐publication peer review meets the needs of scientists, and the organizations that fund them, in practice. In addition, inferences about the relative importance of journals are set out, which should also interest publishers and librarians.

Is collaboration among scientists related to the citation impact of papers because their quality increases with collaboration? An analysis based on data from F1000Prime and normalized citation scores

“…Because the so‐called faculty members can select any paper of interest (i.e., the papers are not systematically selected and rated), only a restricted set of papers from the medical and biological journals is included in F1000Prime (Kreiman & Maunsell, ; Wouters & Costas, ). However, “the great majority [of Faculty members] pick papers published within the past month, including advance online papers, meaning that users can be made aware of important papers rapidly” (Wets, Weedon, & Velterop, , p. 254). The papers included in F1000Prime are rated by the faculty members as “good,” “very good,” or “exceptional,” which is equivalent to scores of 1, 2, or 3, respectively.…”

Interrater reliability and convergent validity of F1000Prime peer review

“…On average, 1,500 new recommendations are contributed by the faculty each month (F1000, ). Faculty members can choose and evaluate any paper that interests them; however, “the great majority pick papers published within the past month, including advance online papers, meaning that users can be made aware of important papers rapidly” (Wets et al., , p. 254). Although many papers published in popular and high‐profile journals (e.g., Nature , New England Journal of Medicine , Science ) are evaluated, 85% of the papers selected come from specialized or less well‐known journals (Wouters & Costas, ).…”

“…Although many papers published in popular and high‐profile journals (e.g., Nature , New England Journal of Medicine , Science ) are evaluated, 85% of the papers selected come from specialized or less well‐known journals (Wouters & Costas, ). “Less than 18 months since Faculty of 1000 was launched, the reaction from scientists has been such that two thirds of top institutions worldwide already subscribe, and it was the recipient of the Association of Learned and Professional Society Publishers (ALPSP) award for Publishing Innovation in 2002 (http://www.alpsp.org/about.htm)” (Wets et al., , p. 249). The F1000Prime database is regarded as a meaningful aid to scientists seeking indications of the most relevant papers in their subject: “The aim of Faculty of 1000 is not to provide an evaluation for all papers, as this would simply exacerbate the ‘noise,’ but to take advantage of electronic developments to create the optimal human filter for effectively reducing the noise” (Wets et al., , p. 253).…”

“…In January 2002, a new type of peer review system was launched (in the field of the medical and biological sciences), whereby about 5,000 faculty members were asked “to identify, evaluate and comment on the most interesting papers they read for themselves each month—regardless of the journal in which they appear” (Wets, Weedon, & Velterop, , p. 251). The so‐called F1000Prime peer review system is considered not as an ex ante appraisal of manuscripts offered to a journal for publication but as an ex post appraisal of papers already published in journals .…”

Exploring the Peer Review Process: What is it, Does it Work, and Can it Be Improved?

“…In this regard, it is somewhat similar to the approach now used in AFS journals, save for being double-blinded and limiting revisions to one. Other approaches take advantage of technological advantages of the Internet, including an open moderated approach (where manuscripts are posted for a moderated comment period prior to submission for regular peer review; Gura 2002); an interactive peer review process followed by public discussion (Poschl et al 2004); posting of all intermediate versions of a manuscript, associated reviews, and author responses to the reviewer comments (Carmi and Koch 2007); an Internet collaboration approach (in Scientific American; Waldrop 2008); and a post-publication filtering approach (the Faculty of 1000; Wets et al 2003). None of these approaches is used widely, and no conclusions can currently be drawn as to their effectiveness.…”