Since we're talking about extraordinary claims, let's examine a claim that is definitely not true, but very interesting. If we run a study with threshold p=0.05, there is a 1 in 20 chance that we will erroneously report the claim to be true.
Now, let's say ten different scientists are interested in this claim, and they're all going to run their own experiments. The chance that all ten will run an experiment with each reporting "false" is under 60%.[0] Over 40% of the time, at least one scientist will falsely conclude the existence of the phenomenon that definitely does not exist. This is an effect of running multiple independently-considered experiments without aggregating the results.
That's the Bayesian problem that people mention. Another problem entirely comes from which results will tend to get published.
Now let's consider the effect of publishing bias. Let's assume that only 20% of the scientists will attempt to publish their results regardless of the outcome, but they will always try to publish if the (false) phenomenon is shown to exist. This effect alone results in 21% of submissions being incorrect,[1] even though an incorrect result only has 5% likelihood.
Let's additionally assume that a journal will publish a false-but-interesting result 50% of the time, and the true-but-ho-hum result only 10% of the time. The final effect is that 50% of published results for this extraordinary-but-false phenomenon incorrectly report the phenomenon to be true.
Tweak the numbers all you want, but the effects of running multiple independently-considered trials, along with biased publishing, means that we are surprisingly likely to publish false conclusions.
Now, let's say ten different scientists are interested in this claim, and they're all going to run their own experiments. The chance that all ten will run an experiment with each reporting "false" is under 60%.[0] Over 40% of the time, at least one scientist will falsely conclude the existence of the phenomenon that definitely does not exist. This is an effect of running multiple independently-considered experiments without aggregating the results.
That's the Bayesian problem that people mention. Another problem entirely comes from which results will tend to get published.
Now let's consider the effect of publishing bias. Let's assume that only 20% of the scientists will attempt to publish their results regardless of the outcome, but they will always try to publish if the (false) phenomenon is shown to exist. This effect alone results in 21% of submissions being incorrect,[1] even though an incorrect result only has 5% likelihood.
Let's additionally assume that a journal will publish a false-but-interesting result 50% of the time, and the true-but-ho-hum result only 10% of the time. The final effect is that 50% of published results for this extraordinary-but-false phenomenon incorrectly report the phenomenon to be true.
Tweak the numbers all you want, but the effects of running multiple independently-considered trials, along with biased publishing, means that we are surprisingly likely to publish false conclusions.
Notes:
[0] 0.95^10 = 0.5987
[1] 0.95 * 0.2 = 0.19; 0.05 / (0.19 + 0.05) = 0.21
[2] (0.21 * 0.5) / (0.21 * 0.5 + 0.79 * 0.1) = 0.50