Comment by gravypod

I've heard this mentioned a few times. Here is a summarized version of the abstract:

    > ... We conduct a randomized controlled trial (RCT)
    > ... AI tools ... affect the productivity of experienced
    > open-source developers. 16 developers with moderate AI
    > experience complete 246 tasks in mature projects on which they
    > have an average of 5 years of prior experience. Each task is
    > randomly assigned to allow or disallow usage of early-2025 AI
    > tools. ... developers primarily use Cursor Pro ... and
    > Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing
    > AI will reduce completion time by 24%. After completing the
    > study, developers estimate that allowing AI reduced completion time by 20%.
    > Surprisingly, we find that allowing AI actually increases
    > completion time by 19%—AI tooling slowed developers down. This
    > slowdown also contradicts predictions from experts in economics
    > (39% shorter) and ML (38% shorter). To understand this result,
    > we collect and evaluate evidence for 21 properties of our setting
    > that a priori could contribute to the observed slowdown effect—for
    > example, the size and quality standards of projects, or prior
    > developer experience with AI tooling. Although the influence of
    > experimental artifacts cannot be entirely ruled out, the robustness
    > of the slowdown effect across our analyses suggests it is unlikely
    > to primarily be a function of our experimental design.

So what we can gather:

1. 16 people were randomly given tasks to do

2. They knew the codebase they worked on pretty well

3. They said AI would help them work 24% faster (before starting tasks)

4. They said AI made them ~20% faster (after completion of tasks)

5. ML Experts claim that they think programmers will be ~38% faster

6. Economists say ~39% faster.

7. We measured that people were actually 19% slower

This seems to be done on Cursor, with big models, on codebases people know. There are definitely problems with industry-wide statements like this but I feel like the biggest area AI tools help me is if I'm working on something I know nothing about. For example: I am really bad at web development so CSS / HTML is easier to edit through prompts. I don't have trouble believing that I would be slower trying to make an edit to code that I already know how to make.

Maybe they would see the speedups by allowing the engineer to select when to use the AI assistance and when not to.