Is Peer-Review Everything It Implies?

What is Peer-Review?

It is probably safe to assume that most people, regardless of their profession, have a positive connotation of peer-review. One may even consider it a core component of the scientific method; papers that have been peer-reviewed are deemed credible and acknowledged by the experts. However, the validity of these beliefs is debatable.

Peer review is the “process of subjecting an author’s scholarly work, research or ideas to the scrutiny of others who are experts in the same field,” this includes maintaining high scientific standards, and ensuring only justified interpretations are disseminated (Kelly et al, 2014).

Figure 1: Diagram illustrating the Peer-Review process for the British Medical Journal

However, if it is true that “a scientific hypothesis or statement is generally not accepted by the academic community unless it has been published in a peer-reviewed journal,” we should consider the influence of peer-review (Kelly et al., 2014). Moreover, reviewers can prevent publications based on their assessment of the significance of the research including its potential scientific contributions based on its results (Kelly et al., 2014). This raises two questions: (1) what are the results of peer-review, and (2) how does one peer-review a paper that does not align with the scientific consensus, which is sometimes wrong?

Evaluation of Peer Review (Question 1)

An expected outcome of peer-review is suppression of papers that the reviewers disagree with. This would be more common within smaller sub-fields of science where researchers have contentious views. (An analogy might be if Freud were to peer-review Jung because Freud is also a psychoanalysis expert.) As well, the process of peer-review has expanded to even more disputed fields, such as the borders of mathematical and physics explorations. Mathematical theorist Eric Weinstein believes that “peer-review is a cancer,” and is actually “the ability of your peers to keep the world from learning about your work” whereas “real peer-review is what happens after [publications]" (2020). However, I noticed that those that work in the laboratory setting (i.e. those working primarily in the medical and biochemical field) are more fond of peer-review, and thus, emphasize that the studies that they cite are peer-reviewed. In short, the view of peer-review varies across scientific fields.

Figure 2: The Results of a Survey on Satisfaction of Peer-Review across various Disciplines (Ross-Hellauer et al., 2017)

Peer-review may hinder the exploration process occasionally, but overall, does it still deliver the values we assume it contributes to science? Richard Smith, former editor of the British Medical Journal (a well-respected source of peer-reviewed papers), summarizes its defects: “In addition to being poor at detecting gross defects and almost useless for detecting fraud[,] it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused” (Smith, 2006). A study cited in Smith's review reveals these biases. Researchers Douglas Peters and Stephen Ceci resubmitted 12 papers (retyped with insignificant tweaks) from prestigious institutions but changed the names of the authors and their institutions for peer-review: out of the 12 papers, three papers were found to be already published and eight were rejected for poor-quality. And more significantly, there is lack of accuracy. Deliberately inserted errors were also missed. Specifically, a study inserted 8 mistakes, and “the average number of errors spotted by reviewers was two, no reviewer spotted more than five errors, and 35 reviewers (16%) did not spot any'' (Kelly et al., 2014). Furthermore, its slow and costly process also limits valuable scientific findings from entering the realm of public health. Often, it seems that any claim from a new scientific paper can be dismissed with only 4 words: “it is not peer-reviewed.” Peer-review should not, in my opinion, become an authoritative stamp to legitimize science papers. The dilemma here is that papers are often faulty in some ways and thus, we cannot automatically equate science papers as scientifically sound – each should be examined carefully. In fact, there is a possibility that most published findings are wrong (Ioannidis, 2005 as cited by Muller, 2016). Nevertheless, the overemphasis on the status of the paper (reviewed or not), undermines each scientist's ability to discern the content for themselves, or even worse, undermines the discernment by various diverse individuals trained in the scientific method. Errors will occur, which is why science involves a constant scepticism and a cyclic method of consolidating knowledge, i.e., via the adjustment of hypotheses, further data collection, and meta-analyses. If these flaws are not corrected by peer-review, is it still that important? However, despite all the flaws, the solution is probably not to get rid of it, but to fix it. For example, suggested improvements include double-blind reviews, better training, and added incentives to review properly. There is a well-known description of peer-review, “a system full of problems but the least worst we have” (Smith, 2006). Therefore, I hope the brief summary above could evoke some thoughts which may one day become part of the scientific community.

Peer-Review of the Unorthodox (Question 2)

We have to consider the power that we place in these reviewers because it would be naive to assume that the pursuit of scientific truth is of the utmost value for everyone. Because scientists may favour their own interpretations of new data, conflicts of interest may prevent ingenious ideas or contradictory conclusions from reaching the scientific community. This might have played out in the 20th century: the physicians who became affiliated with the tobacco industries probably did not want to recognize or peer-review the papers that were suggesting cigarettes contain carcinogens. I am not implying that all or most researchers are profit-driven, but such a conflict seems plausible. Similarly, evolutionary biologist Bret Weinstein points out in his recent article that “science is plagued by a system of perverse incentives in which scientists are condemned to constantly compete for jobs and grant money just to stay in the game” and consequently, “scientists exaggerate, distort and mislead in order to get their own work (or their field’s work) funded" (2021). Moreover, in the borders of our web of knowledge, it is no longer reasonable to have authorities assessing the validity of new hypotheses or interpretations and therefore risk suppressing findings that may result in paradigm shifts. I would agree with Eric Weinstein in that the peer-review should come after, so that the potentially significant findings are at least available for public scrutiny. Nonetheless, another question arises, how do we prevent false knowledge from spreading and causing chaos? Hopefully, in the future, the process can be amended to balance the necessity of high scientific standards in paper with the freedom to publish that which is unorthodox. Kelly et al. shares new possibilities for the future of peer-review: one among many was PeerJ which "selects articles to publish based only on scientific and methodological soundness” (2014). Overall, the incentives in the scientific field and the (potentially oppressive) power of peer-review may be hindering the emergence of truths, or at least partial-truths.

Concluding Remarks

As I venture into this field I wonder if I will fall victim to my own biases or personal incentives. Ensuring poor science from clouding the rigorous field is essential, but the judgement of what to dismiss and what to accept should not depend solely on whether it is peer-reviewed or not. Lastly, in this digital generation, there are tendencies to use clickbait titles with false science findings – advertisement revenue at the cost of trust in science – which shifts the public in favour of peer-reviewed findings. How much of the public’s trust in peer-review is engendered in response to the media, as opposed to its merits?

Note: I am not qualified to speak about what is the proper conduct in science due to my lack of experience and knowledge but I hope that my opinions, as wrong as they may be, can still be of value.

References

Kelly, J., Sadeghieh, T., Adeli, K., & Biochemistry, C. (2014). Peer review in scientific publications: benefits, critiques, & a survival guide. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4975196/pdf/ejifcc-25-227.pdf

Muller, D. (2016). Is most published research wrong?. On YouTube. https://www.youtube.com/watch?v=42QuXLucH3Q

Channel Name: Veritasium

Ross-Hellauer, T., Deppe, A., & Schmidt, B. (2017). Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. PLOS ONE, 12(12), e0189311. https://doi.org/10.1371/journal.pone.0189311‌

Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182. https://doi.org/10.1258/jrsm.99.4.178

Weinstein, B. (2021, June 4). Why we should welcome the lab leak hypothesis. UnHerd. https://unherd.com/2021/06/why-we-should-welcome-the-lab-leak-theory/

Weinstein, E. (2020). The problem with peer review - eric weinstein | the portal podcast clips. On YouTube. https://www.youtube.com/watch?v=U5sRYsMjiAQ

Clip from Podcast #19