Comment by epolanski
So you don't doubt their conclusion that most sucked by using AI, but you doubt that they found that some learned more?
So you don't doubt their conclusion that most sucked by using AI, but you doubt that they found that some learned more?
The conclusion of the paper doesn't say that "most sucked using AI". It's says the mean quiz score was both significantly and sizably lower in the intervention group vs the control. No significant difference detected on speed.
The qualitative breakdown says how you use AI matters for understanding. It doesn't say some learned more than the control group and even if it did, it's not powered to show a statistical difference which is one of the only things keeping a study from not being another anecdote on the internet.
For the sake of argument let's say there is an individual in the treatment arm who scored higher than the highest control participant. What some want that to mean is, "Some engineers perform better using AI". It does not say that. That could be an objective fact(!), it doesn't matter. This study will not support it; it's an RCT. What if that programmer is just naturally gifted or lucky(!). This is the point of statistics.
The best you can do with outliers is say "AI usage didn't hinder some from attaining a high score" (again maybe it would have been higher w/o you just can't reason about individuals in a study like this).
I hope this helps.