Comment by wiml
If the effect size x site traffic is so small it's statistically insignificant, why are you doing all this work in the first place? Just choose the option that makes the PHB happy and move on.
(But, it's more likely that you don't know if there's a significant effect size)
The PHB wanted A/B testing! True story. I've spent two months convincing them that it made no sense with the volume of conversion events we had.