Comment by kadushka
the larger the trial size, the smaller the outcome
I find this a bit surprising. Could there be something else affecting the accuracy of larger trials? Perhaps they are not as careful, or cutting corners somewhere?
the larger the trial size, the smaller the outcome
I find this a bit surprising. Could there be something else affecting the accuracy of larger trials? Perhaps they are not as careful, or cutting corners somewhere?
It does not require more explanation: publication bias means null results aren't in the literature; do enough small low quality trials and you'll find a big effect sooner or later.
Then the supposed big effect attracts attention and ultimately properly designed studies which show no effect.
No, the other way around. It's the combination of two well known effects. Well, three if you're uncharitable.
1. Small studies are more likely to give anomalous results by chance. If I pick three people at random, it's not that surprising if I happened to get three women. It would be a lot different if I sampled 1,000 people.
2. Studies that show any positive result tend to get published, and ones that don't tend to get binned.
Put those together, and you see a lot of tiny studies with small positive results. When you do a proper study, the effect goes away. Exactly as you would expect.
The less charitable effect is "they made it up". It happens.
Just my hypothesis, but I wonder if larger sample sizes provide a more diverse population.
A study with 1000 individuals is likely a poor representation of a species of 8.2 billion. I understand that studies try to their best to use a diverse population, but I often question how successful many studies are at this endeavor.
> If that's the case, we should question whether different homogeneous population groups respond differently to the substance under test.
In terms of psychological treatments, I am honestly in support of this. Many mental illnesses can have a cultural component to them.
> After all, we don't want to know the "average temperature of patients in a hospital", do we?
No, I don't think we do. Am I understanding you correctly?
Maybe. Those could be the case. But ignoring all confounding factors, this phenomenon is possible with numerical experiments alone. One of the meanings of "the Law of Small Numbers".
Basically, the possibility that the small study was underpowered, and just lucky...then the large studies with more power are closer to the truth. https://en.wikipedia.org/wiki/Faulty_generalization