Comment by LPisGood
This objection is mentioned specifically in the post.
You can add a forgetting factor for older results.
This objection is mentioned specifically in the post.
You can add a forgetting factor for older results.
That's a different problem. In jbentley1's scenario, A could be better, but this algorithm will choose B.
This seems like a fudge factor though. Some things are changed bc you act on them! (e.g. recommendation systems that are biased towards more popular content). So having dynamic groups makes the data harder to analyze