Comment by npunt
Not quite apples to apples tho because you have to take into consideration what was known at the time each theory was developed (the input), not just the output.
Theory A: fits 7 known predictions but also makes a not-yet-verified prediction
Theory B: fits 8 known predictions and offers no new ones
In this example wouldn't Theory A be better, because all else equal it is less likely the product of overfitting and required more insight and effort to discover? In other words, Theory A used a different process that we know has a higher likelihood of novel discovery.
(Maybe this is a restatement of the simplicity argument, in that Theory A requires fewer predictions to discover it, ergo it is simpler)
> In this example wouldn't Theory A be better, because all else equal it is less likely the product of overfitting and required more insight and effort to discover?
No, Theory A might simply be a dead end with no new insights to offer. And alas: the universe does not care about insights, efforts, or simplicity.
All else equal if Theory B is easier to teach - easier for more people to understand - it might have value for that reason. It might also be valuable to teach multiple ways to understand the same underlying phenomenon.
> In other words, Theory A used a different process that we know has a higher likelihood of novel discovery.
How would we measure "likelihood of novel discovery"?
Now to call myself out here: the best way to answer any of these questions is to probe both theories at their limits to find differences in predictions that we can test. It may be that we don't have the right equipment or haven't designed experiments sufficient to do that currently.
Remember that Einstein's GR was validated by its prediction and the Eddington experiment, though his initial 1911 prediction was wrong and he later refined it in 1915. The 1919 Eddington measurements validated the theory.
We should remember though: That only worked out because the 1912 attempt to make the observations (which would have invalidated Einstein) got rained out. Who knows how Einstein's career would have turned out if the 1912 observations had succeeded. Perhaps people would have said he simply over-fit his theory to fit observation.