Figure 2: The Results of a Survey on Satisfaction of Peer-Review across various Disciplines (Ross-Hellauer et al., 2017)
Peer-review may hinder the exploration process occasionally, but overall, does it still deliver the values we assume it contributes to science? Richard Smith, former editor of the British Medical Journal (a well-respected source of peer-reviewed papers), summarizes its defects: “In addition to being poor at detecting gross defects and almost useless for detecting fraud[,] it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused” (Smith, 2006). A study cited in Smith's review reveals these biases. Researchers Douglas Peters and Stephen Ceci resubmitted 12 papers (retyped with insignificant tweaks) from prestigious institutions but changed the names of the authors and their institutions for peer-review: out of the 12 papers, three papers were found to be already published and eight were rejected for poor-quality. And more significantly, there is lack of accuracy. Deliberately inserted errors were also missed. Specifically, a study inserted 8 mistakes, and “the average number of errors spotted by reviewers was two, no reviewer spotted more than five errors, and 35 reviewers (16%) did not spot any'' (Kelly et al., 2014). Furthermore, its slow and costly process also limits valuable scientific findings from entering the realm of public health. Often, it seems that any claim from a new scientific paper can be dismissed with only 4 words: “it is not peer-reviewed.” Peer-review should not, in my opinion, become an authoritative stamp to legitimize science papers. The dilemma here is that papers are often faulty in some ways and thus, we cannot automatically equate science papers as scientifically sound – each should be examined carefully. In fact, there is a possibility that most published findings are wrong (Ioannidis, 2005 as cited by Muller, 2016). Nevertheless, the overemphasis on the status of the paper (reviewed or not), undermines each scientist's ability to discern the content for themselves, or even worse, undermines the discernment by various diverse individuals trained in the scientific method. Errors will occur, which is why science involves a constant scepticism and a cyclic method of consolidating knowledge, i.e., via the adjustment of hypotheses, further data collection, and meta-analyses. If these flaws are not corrected by peer-review, is it still that important? However, despite all the flaws, the solution is probably not to get rid of it, but to fix it. For example, suggested improvements include double-blind reviews, better training, and added incentives to review properly. There is a well-known description of peer-review, “a system full of problems but the least worst we have” (Smith, 2006). Therefore, I hope the brief summary above could evoke some thoughts which may one day become part of the scientific community.
Peer-Review of the Unorthodox (Question 2)
We have to consider the power that we place in these reviewers because it would be naive to assume that the pursuit of scientific truth is of the utmost value for everyone. Because scientists may favour their own interpretations of new data, conflicts of interest may prevent ingenious ideas or contradictory conclusions from reaching the scientific community. This might have played out in the 20th century: the physicians who became affiliated with the tobacco industries probably did not want to recognize or peer-review the papers that were suggesting cigarettes contain carcinogens. I am not implying that all or most researchers are profit-driven, but such a conflict seems plausible. Similarly, evolutionary biologist Bret Weinstein points out in his recent article that “science is plagued by a system of perverse incentives in which scientists are condemned to constantly compete for jobs and grant money just to stay in the game” and consequently, “scientists exaggerate, distort and mislead in order to get their own work (or their field’s work) funded" (2021). Moreover, in the borders of our web of knowledge, it is no longer reasonable to have authorities assessing the validity of new hypotheses or interpretations and therefore risk suppressing findings that may result in paradigm shifts. I would agree with Eric Weinstein in that the peer-review should come after, so that the potentially significant findings are at least available for public scrutiny. Nonetheless, another question arises, how do we prevent false knowledge from spreading and causing chaos? Hopefully, in the future, the process can be amended to balance the necessity of high scientific standards in paper with the freedom to publish that which is unorthodox. Kelly et al. shares new possibilities for the future of peer-review: one among many was PeerJ which "selects articles to publish based only on scientific and methodological soundness” (2014). Overall, the incentives in the scientific field and the (potentially oppressive) power of peer-review may be hindering the emergence of truths, or at least partial-truths.
As I venture into this field I wonder if I will fall victim to my own biases or personal incentives. Ensuring poor science from clouding the rigorous field is essential, but the judgement of what to dismiss and what to accept should not depend solely on whether it is peer-reviewed or not. Lastly, in this digital generation, there are tendencies to use clickbait titles with false science findings – advertisement revenue at the cost of trust in science – which shifts the public in favour of peer-reviewed findings. How much of the public’s trust in peer-review is engendered in response to the media, as opposed to its merits?
Note: I am not qualified to speak about what is the proper conduct in science due to my lack of experience and knowledge but I hope that my opinions, as wrong as they may be, can still be of value.