What feedback do reviewers give when reviewing qualitative manuscripts? A focused mapping review and synthesis

Abstract

Background: Peer review is at the heart of the scientific process. With the advent of digitisation, journals started to offer electronic articles or publishing online only. A new philosophy regarding the peer review process found its way into academia: the open peer review. Open peer review as practiced by BioMed Central (BMC) is a type of peer review where the names of authors and reviewers are disclosed and reviewer comments are published alongside the article. A number of articles have been published to assess peer reviews using quantitative research. However, no studies exist that used qualitative methods to analyse the content of reviewers' comments. Methods: A focused mapping review and synthesis (FMRS) was undertaken of manuscripts reporting qualitative research submitted to BMC open access journals from 1 January-31 March 2018. Free-text reviewer comments were extracted from peer review reports using a 77-item classification system organised according to three key dimensions that represented common themes and sub-themes. A two stage analysis process was employed. First, frequency counts were undertaken that allowed revealing patterns across themes/sub-themes. Second, thematic analysis was conducted on selected themes of the narrative portion of reviewer reports. Results: A total of 107 manuscripts submitted to nine open-access journals were included in the FMRS. The frequency analysis revealed that among the 30 most frequently employed themes "writing criteria" (dimension II) is the top ranking theme, followed by comments in relation to the "methods" (dimension I). Besides that, some results suggest an underlying quantitative mindset of reviewers. Results are compared and contrasted in relation to established reporting guidelines for qualitative research to inform reviewers and authors of frequent feedback offered to enhance the quality of manuscripts.

No time for that now! Qualitative changes in manuscript peer review during the Covid-19 pandemic

“…First and foremost however, we note that the general structure of review reports is largely stable across the three sample categories (Covid, non-Covid, and pre-Covid). As was noticed before, amongst others by Herber et al. (2020) , reviewers—at least on accepted manuscripts—almost always start off their review indicating general praise for the manuscript, followed by explicit praise for aspects or sections that they found particularly valuable, before they put forward any negative feedback.…”

“…Two notable exceptions to this are the recent studies by Herber et al. (2020) and Siler and Strang (2017) .…”

“…Upon collecting the material, all material was read and manually coded using an inductive coding scheme loosely inspired by the classification of Herber et al. (2020) .…”

“…Taking advantage of these open review formats, some studies have aimed to classify reviewer comments, using quantitative and qualitative methods to distinguish between various types of content in review reports ( Herber et al. 2020 ).…”

Lessons from an International Initiative to Set and Share Good Practice on Human Health in Environmental Impact Assessment

“…This identified themes such as target audience of the reference paper; health determinants; significance/likely health effect; equity/vulnerable groups; methods/tools; role of public health professionals/competences needed; intersectoral cooperation/stakeholders/responsible authorities. The responses given to the survey and the peer-review comments were based upon a reading of the second and third drafts, respectively, of the reference paper and so these comments were also analyzed with the framework identified by Herber et al [ 39 ] which relates to peer review of a qualitative manuscript. Details about the consultation and the responses are provided in the supplementary information (Supplementary Materials File S1) .…”