Comment by siliconc0w

I mean, they're literally pointing out the negative effects of AI-assisted coding?

> We found that using AI assistance led to a statistically significant decrease in mastery. On a quiz that covered concepts they’d used just a few minutes before, participants in the AI group scored 17% lower than those who coded by hand, or the equivalent of nearly two letter grades. Using AI sped up the task slightly, but this didn’t reach the threshold of statistical significance.

This also echoes other research from a few years ago that had similar findings: https://news.ycombinator.com/item?id=46822158

  > this is a massive conflict of interests

But people like that they aren't shying away from negative results and that builds some trust. Though let's not ignore that they're still suggesting AI + manual coding.

But honestly, this sample size is so small that we need larger studies. The results around what is effective and ineffective AI usage is a complete wash with n<8.

Also anyone else feel the paper is a bit sloppy?

I mean there's a bunch of minor things but Figure 17 (first fig in the appendix) is just kinda wild. I mean there's trivial ways to solve the glaring error. The more carefully you look at even just the figures in the paper the more you say "who the fuck wrote this?" I mean like how the fuck do you even generate Figure 12? The numbers align with the grids but boxes are shifted. And Figure 16 has experience levels shuffled for some reason. And then there are a hell of a lot more confusing stuff you'll see if you do more than a glance...

> I wish they had attempted to measure product management skill.

We're definitely getting better at writing specs. The issue is the labor bottleneck is competent senior engineers, not juniors, not PMs, not box-and-arrow staff engineers.

> I think AI is shifting entry level programmers to focus on expressing requirements clearly

This is what the TDD advocates were saying years ago.

What AI development has done for my team is the following:

Dramatically improved Jira usage -- better, more descriptive tickets with actionable user stories and clearly expressed requirements. Dramatically improved github PRs. Dramatically improved test coverage. Dramatically improved documentation, not just in code but in comments.

Basically all _for free_, while at the same time probably doubling or tripling our pace at closing issues, including some issues in our backlog that had lingered for months because they were annoying and nobody felt like working on them, but were easy for claude to knock out.

I think we're going to see a small minority of juniors who managed to ignore the hype/peer pressure/easy path and actually learned to code have a huge advantage over the others.

Relevant quote from their conclusion:

> Among participants who use AI, we find a stark divide in skill formation outcomes between high-scoring interaction patterns (65%-86% quiz score) vs low-scoring interaction patterns (24%-39% quiz score). The high scorers only asked AI conceptual questions instead of code generation or asked for explanations to accompany generated code; these usage patterns demonstrate a high level of cognitive engagement.

This is very much my experience. AI is incredibly useful as a personal tutor

> there is a smaller % that benefits and learns more and faster

That's not what the study says nor it is capable of credibly making that claim. You are reasoning about individuals in an RCT where subjects did not serve as their own control. The high performers in the treatment group may have done even better had they been in the control and AI is in fact is slowing them down.

You don't know which is true because you can't know because of the study design. This is why we have statistics.

> If you sucked before using AI > you are going to suck with AI

This.