Comment by jostmey 6 months ago
This only addresses a small part of the problem with peer-review. The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers. If the paper doesn’t fit what is expected, it will often be rejected. This can also lead to self-reinforcing views that ignore contrarian data. Also, the data can be made up, and if it makes sense to the reviewers, it is generally not questioned
There are no silver bullets. Replication also isn't worth much:
https://blog.plan99.net/replication-studies-cant-fix-science...
Merely replicating a study doesn't tell you anything about whether the claims are correct, although failure to replicate can be a signal that the claims probably aren't correct.
Have you personally reviewed a non-zero number of papers? What is this statement based on? For a thread ostensibly about science, the comments are disappointingly lacking in evidence and heavy on vibes.
Maybe people could learn about what peer review is before posting their strong feelings about it? The purpose certainly isn't to replicate people's experiments, that happens after publication and not by referees. One of a reviewer's duties is to look at whether the study could be replicated given the included information. That is a very different thing.
Also, just because something has made it past peer review also doesn't mean it isn't controversial in the field.
I’ve only peer reviewed a few papers. I’ve submitted my share of papers for review
This is my academic profile https://scholar.google.com/citations?user=bxn78bkAAAAJ&hl=en
He isn't saying the purpose of peer review is to replicate work. And yes I've personally reviewed quite a lot of papers (from outside of academia), also occasionally reviewed peer reviews.
What he is saying is that peer review is treated by academia as a close to gold standard, when it's really more like a bronze standard. It's a bit like finding a software company that exclusively uses volunteer post-commit code review with no unit tests or static typing, and in which the only testing process is to push to production and see if anyone inside the company complains.
It's not useless as a concept, and it's generally better to have a paper that's reviewed than one that isn't unless the field has been captured by ideologues. But there are so many common problems it can't fix, not even in principle.
Bad quality work is a much bigger problem than dishonest work. Systematically well-done research with fake results or data is much rarer than just... lazy bad science.
There is a massive incentive to publish. Inflate the value, inflate the results, and stretch out projects to multiple smaller papers, fake results to make it seem important. This is lazy and fast, and can be caught by a stricter review and scrutiny.
Papers that are properly done all the way through, but with faked data meant to push an agenda, can be disproven by counter research.
> Systematically well-done research with fake results or data is much rarer than just... lazy bad science.
Genuine mistakes, logical errors, and other oversights are even more common than that. For all the issues it has, peer review is quite good at catching the things that it's intended to catch.
Reading your comment makes me think that you believe that the point of peer-review is to ensure that a paper is correct, or at least that specific aspects of it are correct. Is that the case? What do you think the point of peer-review is?
I'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be?
Research gets cited constantly in public debates and is used for policy decisions, so the public should be able to quickly separate the good from the bad, the "maybe this is true" from the "this is empirically proven."
The public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus.
Nothing is ever “proven”. There is simply more or less support for a theory or proposition.
Replication and meta-analysis are an important part of this.
Most scientists are in fact very conservative with how they claim their results - less so university PR departments and “study shows” clickbaiters.
I wish this comment was more representative of my personal experience in science.
Instead I got PIs happy to say that weak evidence "proved" their theory and to try suppress evidence that negatively impacted "fundablity". The most successful scientists I worked with were the ones who always talked like a PR puff piece.
> public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus
The public has lost trust in science because 10 to 30% of it is scientifically illiterate [1]. (Tens of millions of American adults are literally illiterate [2].)
That's what lets activists and politicians cherrypick bad science that supports their position or cast a scientific consensus as unquestionable.
[1] https://carsey.unh.edu/publication/conspiracy-vs-science-sur...
That's certainly true but I think there's also a very real issue as described by GP. Research gets cited in a political setting as a rhetorical tool. That cycle does a lot to erode trust in the establishment because it incentivizes non-scientists (who are otherwise uninvolved) to behave as though the process is a partisan effort to be interfered with for the benefit of one's "team" rather than an objective pursuit to be funded at arms length for the betterment of society at large.
Obviously reproducing results as part of peer review is not a workable (or even coherent) solution. I don't pretend to have any idea what a solution might be. The obvious issue is that academic publications were never intended as political tools and should not be made into that.
On several occasions I've had interactions with laymen where I found myself thinking "if only you hadn't had access to pubmed and way too much motivation we'd both be better off right now" yet I firmly believe that free and open access to information is a huge net benefit to society on the whole.
> I'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be?
It is not, and it cannot be. It is unrealistic to expect a referee working in their free time to confirm studies that often cost millions of dollars. This is a fundamental misunderstanding of what peer review is and why it is useful in popular or heavy vulgarised science.
Politicians, journalists, and university press offices are guilty of this, and they are those abusing peer review to give some studies more weight than they deserve.
> The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers.
This is a misunderstanding of the role of peer review. The point is not to prove that a paper is correct, the point is to ensure a minimum level of quality. You are entirely right that most reviewers cannot hope to reproduce the results presented, and very often for very good reasons. If I write 3 proposals over the course of a year to get some beam line on a neutron source, it is completely unrealistic to expect a referee to have the same level of commitment.
I think this hints at a more profound problem, which is that a lot of studies are not replicated. This is where the robustness of a scientific result comes from: anybody can make the same observation and reach the same conclusions under the same conditions. This is the real test, not whether you convinced 3 or 5 referees.
The real value of an article is not in whether it was peer reviewed (though the absence of peer review is a red flag). Instead, it is in whether different people confirmed its main results over the years that follow publication.