Comment by disgruntledphd2
Comment by disgruntledphd2 4 hours ago
> My naive approach would be to just implement it twice, once together with an LLM and once without, but that has obvious flaws, most obvious that the order which you do it with impacts the results too much.
You'd get a set of 10-15 projects, and a set of 10-15 developers. Then each developer would implement the solution with LLM assistance and without such assistance. You'd ensure that half the developers did LLM first, and the others traditional first.
You'd only be able to detect large statistical effects, but that would be a good start.
If it's just you then generate a list of potential projects and then flip a coin as to whether or not to use the LLM and record how long it takes along with a bunch of other metrics that make sense to you.
The initial question was:
> wonder if they have measured their results?
Which seems to indicate that there would be a suitable way for a single individual to be able to measure this by themselves, which is why I asked.
What you're talking about is a study and beyond the scope of a single person, and also doesn't give me the information I'd need about myself.
> If it's just you then generate a list of potential projects and then flip a coin as to whether or not to use the LLM and record how long it takes along with a bunch of other metrics that make sense to you.
That sounds like I can just go by "yeah, feels like I'm faster", which I thought exactly was parent wanted to avoid...