Comment by N_A_T_E a day ago
Is there any path forward to fixing the current reproducibility crisis in science? Individuals can do better, but that won't solve a problem at this scale. Could we make systemic changes to how papers are validated and approved for publication in major journals?
A huge benefit of that is that it would force the publication of null results, although there would still be no incentive for others to cite null result publications, and citations are unfortunately the main metric that determines the "value" of a scientist. Is there a way to make null result publications valuable?
Also, forcing pre-registration on everyone would be problematic because some types of research are not well-suited to strict planning and committee approval -- how would you quickly make adjustments to an experiment? how would you do exploratory data analysis? serendipitous discoveries would be suppressed? etc.
Wow, I didn't think it's possible, but it sounds like a great way to make research boring :).
What is boring about this? You get the guarantee of publishing your work, even if you get a negative result.
I take it you don’t do research. Cause boring is nothing compared to wasting month of time and money only to get a negative result that nobody will publish.
I'm not in the academy, but I do R&D, I published several times, and that's not how I work at all.
I have a broad and open-ended focus, I work as usual on the things I find interesting, then sometimes I see a thing that looks interesting and decide to investigate, then sometimes my initial tests give good results, but more often then they don't, but they give me an idea to do something completely different, and some iterations later I have a result.
I imagine that depends on a field of research. IT is cheap, but I imagine a physicist who wants to do an experiment must secure a funding first, because otherwise it's impossible to do anything. And it requires one to commit to a single topic of research.
You would need the funding anyway before you could start the research
Disclosure: I'm a scientist, specializing in scientific measurement equipment, so of course reproducibility is my livelihood.
But at the same time, I doubt that fields like physics and chemistry had better practices in, say, the 19th century. It would be interesting to conduct a reproducibility project on the empirical studies supporting electromagnetism or thermodynamics. There were probably a lot of crap papers!
Those fields had a backup, which was that studies and theories were interconnected, so that they tended to cross-validate one another. This also meant that individual studies were hot-pluggable. One of them could fail replication and the whole edifice wouldn't suddenly collapse.
My graduate thesis project was never replicated. For one thing, the equipment that I used had been discontinued before I finished, and cost about a million bucks in today's dollars. On the other hand, two labs built similar experiments that were considerably better, made my results obsolete, and enabled further progress. That was a much better use of resources.
I think fixing replication will have to involve fixing more than replication, but thinking about how science progresses as a whole.
Reproducibility studies are costly in time, reagents, and possibly irreplaceable primary samples. I usually would prefer a different study looking at similar mechanisms using different methods than a reproduction of the original methods, although there’s an important place for direct replication studies like this as well. We can also benefit from data sleuths uncovering fraud, better whistleblower systems, and more ability for graduate students to transfer out of toxic labs and into better ones with their funding, reputation and research progress intact.
Scientists have informal trust networks that I’d like to see made explicit. For example, I’d like to see a social media network for scientists where they can PRIVATELY specify trust levels in each other and in specific papers, and subscribe to each others’ trust networks, to get an aggregated private view of how their personal trusted community views specific labs and papers.
> Scientists have informal trust networks that I’d like to see made explicit. For example, I’d like to see a social media network for scientists where they can PRIVATELY specify trust levels in each other and in specific papers, and subscribe to each others’ trust networks, to get an aggregated private view of how their personal trusted community views specific labs and papers.
That sounds fascinating, but I'd have a darned high bar to participate to make sure I wasn't inadertently disclosing my very personal trust settings. Past experiences with intentional or unintentional data deanonymization (or just insufficient anonymization) makes me very wary of such claims.
A dream of mine was that in order to get a PhD, you would not have to publish original research, but instead you would have to _reproduce existing research_. This would bring the PhD student to the state of the art in a different way, and it would create a natural replication process for current research. Your thesis would be about your replication efforts, what was reproducible and what was not, etc.
And then, once you got your PhD, only then you would be expected to publish new, original research.
Yes, but nobody wants to acknowledge the elephant in the room. Once again, this is why defunding research has gained merit. If more than half of new research is fake, don't protest when plugs are being pulled; You're protesting empirical results.
> more than half of new research is fake
You committed the same sin you are attempting to condemn, while sophomorically claiming it is obvious this sin deserves an intellectual death penalty.
It made me smile. :) Being human is hard!
Now I'm curious, will you acknowledge the elephant in this room? It's hard to, I know, but I have a strong feeling you have a commitment to honesty even if it's hard to always enact all the time. (i.e. being a human is hard :) )
I had always envisioned an institute for reproducibility & Peer review. It would be a federally funded institute that would require Phd candidate participation as an additional requirement to receive your degree. Really it wouldn't be a single place but office or team at each university where proper equipment was available and perhaps similar conditions for reproducing specific research. Of course the feasibility of this is pretty low.
IMO, stopping the race toward better h index.
There is an huge amount of pressure to publish publish publish.
So, many researchers prefeer to write very simple things that are probably true or applicative work, which is kind of useful, or publish false/fake results.
May be try to define a "reproducible" h-index, ie your publication doesn't count or count less until a different team has reproduced your results, the team doing the reproducing work gets some points to.
(And may be add more points if in order to reproduce you didn't have to ask plenty of questions to the original team, ie the original paper didn't omit essential information)
The thing is, that would encourage two teams to cheat together, it would displace the problem, I'm not sure it will limit the effect that much (?)
I'm curious, I don't get why the down votes? Having to race for publishing pushes people to cheat, It didn't occur to me that it was a bad point, but if you have a different opinion I would gladly hear!
> I don't get why the down votes?
Because a great many who comment on this site are infantile but self-congratulating idiots who just can't help themselves on downvoting anything that doesn't fit their pet dislikes. That button should be removed or at least made not to grey-out text.
Yeah "individuals do better" is never the answer -- you've got to structure incentives, of course.
I don't think you want to slow down publication (and probably peer review and prestiage journals are useless/obsolete in era of internet); it's already crazy slow.
So let's see: you want people to incentivize two things (1) no false claims in original research (2) to have people try to reproduce claims.
So here's a humble proposal for a funding source (say...the govt): set aside a pot of money specifically for people to try to reproduce research; let this be a valid career path. Your goal should try to be getting research validated by repro before OTHER research starts to build on those premises (avoiding having the whole field go off on wild goose chases like happened w/ Alzheheimer's). And then, when results DON'T repro, blackball the original researchers from funding. (With whatever sort of due process is needed to make this reasonable.)
I think it'd sort things out.
Punishing researchers who make mistakes or get unlucky due to noise in the data is a recipe for disaster, just like in other fields. The ideal amount of fraud and false claims in research is not zero, because the policing effort it would take to accomplish this goal would destroy all other forms of value. I can't emphasize enough how bad an idea blackballing researchers for publishing irreproducible results would be.
We have money to fund direct reproducibility studies (this one is an example), and indirect replication by applying othogonal methods to similar research topics can be more powerful than direct replication.
Completely agree.
Given the way that science and statistics work, completely honest researchers that do everything correct and don't make any mistakes at all will have some research that fails to reproduce. And the flip side of that is that some completely correct work that got the right answer, some proportion of the time, the reproduction attempt will incorrectly fail to reproduce. Type 1 and Type 2 errors are both real and occur without any need for misconduct or mistakes.
Well, don't forgot I also said this!
> With whatever sort of due process is needed to make this reasonable
Is it not reasonable to not continue to fund scientists whose results consistently do not reproduce? And should we not spend the funds to verify that they do (or don't) reproduce (rather than e.g. going down an incredibly expensive goose-chase like recently happened w/ Alzheimer's research)?
Currently there is more or less no reason not to fudge results; your chances of getting caught are slim, and consequences are minimal. And if you don't fudge your results, you'll be at a huge disadvantage when competing against everyone that does!
Hence the replication crises.
So clearly something must be done. If not penalyzing failures to reproduce and funding reproduction efforst, then what?
Your way of thinking sounds alien to me. You seem to assume that people mostly just follow the incentives, rather than acting according to their internal values.
Science is a field with low wages, uncertain careers, and relatively little status. If you respond strongly to incentives, why would you choose science in the first place? People tend to choose science for other reasons. And, as a result, incentives are not a particularly effective tool for managing scientists.
> The ideal amount of fraud and false claims in research is not zero, because the policing effort it would take to accomplish this goal would destroy all other forms of value.
Surely that just means that we shouldn't spend too much effort achieving small marginal progress towards that ideal, rather than that's not the ideal? I am a scientist (well, a mathematician), and I can maintain my idealism about my discipline in the face of the idea that we can't and shouldn't try to catch and stop all fraud, but I can't maintain it in the face of the idea that we should aim for a small but positive amount of fraud.
It's not actually "Ideal" is the point.
You CANNOT create a system that has zero fraud without rejecting a HUGE amount of legitimate work/requests.
This is as true for credit card processing as it is for scientific publishing.
There's no such thing as "Reject 100% of fraud, accept 100% of non-fraud". It wouldn't be "ideal" to make our spaceships with anti-gravity drives, it would be "science fiction".
The relationship between how hard you prevent fraud and how much legitimate traffic you let through is absurdly non-linear, and super dependent on context. Is there still low hanging fruit on the fraud prevention pipeline for scientific publishing?
That depends. Scientists claim that having to treat each other as hostile entities would basically destroy scientific progress. I wholeheartedly agree.
This should be obvious to anyone who has approved a PR from a coworker. Part of our job in code review is to prevent someone from writing code to do hostile things. I'm sure most of us put some effort towards preventing obvious problems, but if you've ever seen https://en.wikipedia.org/wiki/International_Obfuscated_C_Cod... or some of the famous bits of code used to hack nation states then you should recognize that the amount of effort it would take to be VERY SURE that this PR doesn't introduce an attack is insane, and no company could afford it. Instead, we assume that job interviews, coworker vibes, and reputation are enough to dissuade that attack vector, and it works for almost everyone except the juiciest targets.
Science is a high trust industry. It also has "juicy targets" like "high temp superconductor" or "magic pill to cure cancer", but scientists approach everything with "extreme claims require extreme results" and that seems to do alright. They mostly treated LK-99 with "eh, let's not get hasty" even as most of the internet was convinced it was a new era of materials. I think scientists have a better handle on this than the rest of us.
> It's not actually "Ideal" is the point.
> You CANNOT create a system that has zero fraud without rejecting a HUGE amount of legitimate work/requests.
I think that we are using different definitions of "ideal." It sounds like your definition is something like "practically achievable," or even just "can exist in the real world," in which case, sure, zero fraud is not ideal in that sense. To check whether I am using the word completely idiosyncratically, I just looked it up in Apple Dictionary, and most of the senses seem to match my conception, but I meant especially "2b. representing an abstract or hypothetical optimum." It seems very clear to me that you would agree with zero fraud being ideal in sense "2a. existing only in the imagination; desirable or perfect but not likely to become a reality," but possibly we can even agree that it also fits sense 2b above.
On the data analysis side, I think making version control both mandatory and automatic would go a long way.
One issue is that internal science within a company/lab can move incredibly fast -- assays, protocols, datasets and algorithms change often. People tend to lose track of what data, what parameters, and what code they used to arrive at a particular figure or conclusion. Inevitably, some of those end up being published.
Journals requiring data and code for publication helps, but it's usually just one step at the end of a LONG research process. And as far as I'm aware, no one actually verifies that the code you submitted produces the figures in your paper.
It's a big reason why we started https://GoFigr.io. I think making reproducibility both real-time and automatic is key to make this situation better.
There's usually indirect reproduction. For instance I can take some principle from a study and integrate it into something else. The real issue is that if the result is negative - at least from my understanding - the likelihood of publication is minimal, so it isn't communicated. And if the principle I've taken was at fault there's a lot of space for misattribution, I could blame a litany of different confounders for failures until, after some long while I might decide to place blame on the principle itself. That itself may require a complete rework of any potential paper, redoing all the experiments (depending on how anal one is in data collection).
Just open up a comment section for institutional affiliates.
Yes. Accepting the uncertainty and publishing more than few.
Often famous/more cited studies are not replicable. But if you want to work on similar research problem and publish null/non exciting results, you're up for a fight. Journals want new, fun, exciting results but unfortunately the world doesn't work that way
Yes, but it costs money. There's no solution that wouldn't.
IMO, the best way forward would be simply doubling every study with independent researchers (ideally they shouldn't have contact with each other beyond the protocol). That certainly doubles the costs, but it's really just about the only way to catch bad actors early.
> Yes, but it costs money. There's no solution that wouldn't.
True, although, as you doubtless know, as with most things that cost money, the alternative also costs money (for example, in funding experiments chasing after worthless science). It's just that we tend to set aside the costs that we have already priced in. So I tend to think in such settings that a useful approach might be to see how we can make such costs more visible, to increase the will to address them.
This is a flaw of capitalism.
The flaw being that cost is everything. And, in particular, the initial cost matters a lot more than the true cost. This is why people don't install solar panels or energy efficient appliances.
When it comes to scientific research, proposing you do a higher cost study to avoid false results/data manipulation will be seen as a bug. Bad data/results that make a flashy journal paper (room temp superconductivity, for example) bring in more eyeballs and prestige to the institute vs a well-done study which shows negative results.
It's the same reason the public/private cooperation is often a broken model for government spending. A government agency will happily pick a road builder that puts out the lowest bid and will later eat the cost when that builder ultimately needs more money because the initial bid was a fantasy.
Making costs more visible is a good goal, I just don't know how you accomplish that when surfacing those costs will be seen as a negative for anyone in charge of the budget.
> for example, in funding experiments chasing after worthless science
This is tricky. It's basically impossible to know when an experiment will be worthless. Further, a large portion of experiments will be worthless (like 90% of them).
An example of this is superglue. It was originally supposed to be a replacement glass for jet fighters. While running refractory experiments on it and other compounds, the glue destroyed the machine. Funnily, it was known to be highly adhesive even before the experiment but putting the "maybe we can sell this as a glue" thought to it didn't happen until after the machine was destroyed.
A failed experiment that led to a useful product.
How does someone budget for that? How would you start to surface that sort of cost?
That's where I think the current US grant system isn't a terrible way to do things, provided more guidelines are put in place to enforce reproducibility.
> > for example, in funding experiments chasing after worthless science
> This is tricky. It's basically impossible to know when an experiment will be worthless. Further, a large portion of experiments will be worthless (like 90% of them).
I don't mean "worthless science" in the sense "doesn't lead to a desired or exciting outcome." Such science can still be very worthwhile. I mean "worthless science" in the sense of "based on fraudulent methods." This might accidentally arrive at the right answer, but the answer, whether wrong or accidentally right, has no scientific value.
Pre-registration is a pretty big one: essential you outline your research plan (what you’re looking for, how you will analyze the data, what bars you are setting for significance, etc.) before you do any research. You plan is reviewed and accepted (or denied), often by both funding agency and journal you want to submit to, before they know the results.
Then you perform the experiment exactly* how you said you would based on the pre-registration, and you get to publish your results whether they are positive or negative.
* Changes are allowed, but must be explicitly called out and a valid reason given.
https://en.wikipedia.org/wiki/Preregistration_(science)