Peer-review forms the backbone of scientific publication. Intended as a quality control mechanism, the hope is that peer-review will permit only the most rigorously conducted scientific studies to put forth their findings into the public sphere. Sadly, this noble intent is often lost in contemporary academia’s publish-or-perish landscape. In advance of this year’s annual Peer Review Week (September 10-15, 2018), I wanted to highlight some of the challenges and opportunities inherent in the process.
At a basic level, the peer-review process involves appraising a manuscript (submitted by an individual or group of researchers) as to its suitability for publication in an academic journal. Once a manuscript is received by a journal the formatting and style of the manuscript is assessed for conformity to the journal’s guidelines and then forwarded to an editor. At this point, the editor will make an initial judgement as to whether the content of the abstract for the manuscript falls within the remit of the journal. If the manuscript is not a good fit or there are glaring and/or irremediable flaws, the manuscript is rejected and the authors are given the bad news. This initial review usually only takes a few days: a process I refer to (in jest) as the “instant refund”. Although it is disappointing to have the result of hard work sent back almost immediately, it is often better to know that the manuscript was destined to come up short, than to receive the same verdict many months later. If a manuscript avoids the instant refund phase, an associate or section editor will select reviewers based on their expertise in the area. These selected reviewers will be invited to conduct a review and upon acceptance will be sent the full manuscript in order to provide a thorough critique: intending to provide conceptual and methodological clarity to the reader and to tighten the narrative. Or, at least this is the ideal version of what happens as captured in this flow diagram.
One of the main pitfalls of the peer-review system is that the labour force consists almost exclusively of volunteers. And, not just any volunteers. These volunteers are the same researchers whose jobs are on the line trying to publish papers, find faculty positions, secure tenure, and win grant money. Although the phrase “if you have something that needs to get done, ask a busy person” may hold true in certain circumstances, generally speaking academics aren’t seeking out unpaid work. Academics need to publish peer-reviewed papers and are therefore dependent on the good-will of reviewers, academics like themselves who may well feel that they don’t have the time to be a conscientious peer-reviewer. As a result, the peer-review process may suffer crippling delays. I am an associate editor at several journals and can attest to the immense frustration of matching appropriate reviewers to a manuscript only to receive a negative response (or simply to be ignored!). The screen capture below is from an editors’ intranet page, indicating that I had invited 52 people to review the manuscript: 27 didn’t even bother to respond to the invite, 22 indicated they could not conduct the review, 3 people agreed to conduct a review, but only one of those reviews had been conducted on time and the other two had not been submitted before the deadline (of 4 weeks after accepting the invitation. Needless to say, the authors did not receive an initial decision on this manuscript for months. This causes undue delays in the publication of scholarly works and impedes the dissemination of knowledge. It is not uncommon for articles to sit for many months in peer review without any communication between the journal and the authors.
There are, however, systems being put in place that aim to acknowledge peer-reviewers and to highlight the incredible amounts of work that go into the process, notably through interfaces such as Publons. This social media platform allows peer-reviewers to receive credit for conducting reviews, particularly ones of high quality. Once a user sets up an account, they are able to forward the token “Thank you for reviewing X manuscript for Y journal” email to Publons where they will automatically add that review to your profile. Depending on the journal the profile page owner can then provide open access to the review and/or pull statistics from the review itself, e.g. word count for the review. In my opinion, I think this is a great stride towards rewarding those who are actively contributing to the scientific process, not just in quantity, but also in quality. Further, one of the statistics available on reviewers’ Publons profile pages is the ratio of reviews conducted to the number of their own publications. It’s not unreasonable to think that this ratio provides a rough gauge of how much you are giving vs receiving from the peer review process. As part of Peer Review Week, Publons also hosts a series of Peer Review Awards acknowledging contributions to peer review in a variety of categories.
Another approach to peer-review that has emerged only in the last several years is open reviewing. This can take many forms, but the most common form does away with double-blinded reviewing – where the author is blinded to the reviewers and vice versa - and actively seeks to have the process be as transparent as possible. Depending on the journal, many post the history of the reviews and authors’ responses to those review online, for example BMJ Open: the British Medical Journal’s open-access sister journal. This allows anyone with an internet connection to see what the reviewers thought about the paper and to see how well (or poorly) the authors responded to these comments. This openness allows a broader critique of the peer review process, which increases the accountability of reviewers and also avoids shadier review practices (exemplified by BMC’s “fake review scandal” from 2015). In theory, this transparency makes the scientific process much more open, but in practice I feel as though this also introduces some social desirability bias. Working in smaller fields, it is highly likely that you will review manuscripts written by authors who will also review your work. So if you submit a scathing review of an article posted for all to see, this may bias the author in the reviewing of the your own future work. I would be interested to see a content analysis of the language used in reviews where the process is blinded vs open. Based on my personal experience, I would suspect that fewer negative editorialized comments appear in the former vs the latter.
Whether we like it or not, the peer-review process is the current gold standard for policing scientific rigour. Powered by an overburdened volunteer workforce, there can be lapses in timeliness and quality of reviews submitted. However, innovations such as Publons and open-peer reviewing have made strides towards acknowledging this work and making the whole process more transparent. In order for scientific literature to progress, the peer-review process will need to adapt to the changing nature of academic publishing and to engage the researchers that not only contribute to this process, but whose careers rely on it.
About the Author
Theodore D Cosco joined the Oxford Institute of Population Ageing in 2016 as a Research Fellow. He holds a Canadian Institutes of Health Research Postdoctoral Fellowship to conduct a project entitled “Resilience and healthy ageing across the life course” in conjunction with the MRC Unit for Lifelong Health & Ageing.
Comments Welcome:
We welcome your comments on this or any of the Institute's blog posts. Please feel free to email comments to be posted on your behalf to administrator@ageing.ox.ac.uk or use the Disqus facility linked below.
Opinions of the blogger is their own and not endorsed by the Institute
Comments Welcome: We welcome your comments on this or any of the Institute's blog posts. Please feel free to email comments to be posted on your behalf to administrator@ageing.ox.ac.uk or use the Disqus facility linked below.
