Comment by embedding-shape

Comment by embedding-shape 8 hours ago

Almost off-topic, but got me curious: How can I measure this myself? Say I want to put concrete numbers to this, and actually measure, how should I approach it?

My naive approach would be to just implement it twice, once together with an LLM and once without, but that has obvious flaws, most obvious that the order which you do it with impacts the results too much.

So how would I actually go about and be able to provide data for this?

disgruntledphd2 4 hours ago

> My naive approach would be to just implement it twice, once together with an LLM and once without, but that has obvious flaws, most obvious that the order which you do it with impacts the results too much.

You'd get a set of 10-15 projects, and a set of 10-15 developers. Then each developer would implement the solution with LLM assistance and without such assistance. You'd ensure that half the developers did LLM first, and the others traditional first.

You'd only be able to detect large statistical effects, but that would be a good start.

If it's just you then generate a list of potential projects and then flip a coin as to whether or not to use the LLM and record how long it takes along with a bunch of other metrics that make sense to you.

Reply View 3 replies

embedding-shape 4 hours ago

The initial question was:
> wonder if they have measured their results?
Which seems to indicate that there would be a suitable way for a single individual to be able to measure this by themselves, which is why I asked.
What you're talking about is a study and beyond the scope of a single person, and also doesn't give me the information I'd need about myself.
> If it's just you then generate a list of potential projects and then flip a coin as to whether or not to use the LLM and record how long it takes along with a bunch of other metrics that make sense to you.
That sounds like I can just go by "yeah, feels like I'm faster", which I thought exactly was parent wanted to avoid...

Reply View | 2 replies
- disgruntledphd2 2 hours ago
  
  > That sounds like I can just go by "yeah, feels like I'm faster", which I thought exactly was parent wanted to avoid...
  No it doesn't, but perhaps I assumed too much context. Like, you probably want to look up the Quantified Self movement, as they do lots of social science like research on themselves.
  > Which seems to indicate that there would be a suitable way for a single individual to be able to measure this by themselves, which is why I asked.
  I honestly think pick a metric you care about and then flip a coin to use an LLM or not is the best you're gonna get within the constraints.
  
  Reply View | 1 reply
  
  embedding-shape 2 hours ago
  
  > Like, you probably want to look up the Quantified Self movement, as they do lots of social science like research on themselves.
  I guess I was looking for something bit more concrete, that one could apply themselves, which would answer the "if they have measured their results? [...] Can you provide data that objects this view" part of parents comment.
  > then flip a coin to use an LLM or not is the best you're gonna get within the constraints.
  Do you think trashb who made the initial question above would take the results of such evaluation and say "Yeah, that's good enough and answers my question"?
  
  Reply View | 0 replies