Comment by epolanski

Comment by epolanski 2 days ago

> The loss of competency seems pretty obvious but it's good to have data

That's not what the study says. It says that most users reflect your statement while there is a smaller % that benefits and learns more and faster.

Generalizations are extremely dangerous.

What the article says simply reflect that most people don't care that much and default to the path of least resistance, which is common every day knowledge, but we very well know this does not apply to everyone.

AstroBen 2 days ago

Relevant quote from their conclusion:

> Among participants who use AI, we find a stark divide in skill formation outcomes between high-scoring interaction patterns (65%-86% quiz score) vs low-scoring interaction patterns (24%-39% quiz score). The high scorers only asked AI conceptual questions instead of code generation or asked for explanations to accompany generated code; these usage patterns demonstrate a high level of cognitive engagement.

This is very much my experience. AI is incredibly useful as a personal tutor

Reply View 10 replies

rienbdj 2 days ago

Yes. I love using AI for the “where do I even start” type questions. The once I’ve had a discussion about various approaches I know what docs to actually look at and I can start thinking about implementation details. I don’t find AI very useful for generating code (weird position I know).

Reply View | 2 replies
- nottorp 2 days ago
  
  Why weird? I share this position.
  The LLMs have been trained on countless introductory tutorials for most popular topics, so they will provide you with a reasonable one.
  Ad and friction free for now.
  Enjoy it while it lasts.
  
  Reply View | 0 replies
- pxc a day ago
  
  This is also how I use LLMs at work. I have some vague worries because I'm told this is outdated, I'm falling behind, etc. I'm doing it this way in part hecause my employer is a big, old, slow company and experienting with other kinds of "AI" tools is virtually impossible. But I think it's really more my style.
  
  Reply View | 0 replies
ambicapter 2 days ago

A personal tutor who you remain skeptical of, and constantly try to disprove in order to perfect your understanding.

Reply View | 6 replies
- marcosdumay 2 days ago
  
  A tutor that can guide you through jargon and give you references. If "skepticism" is even something you have to think about, you are already outside of the optimum path.
  
  Reply View | 4 replies
  
  nativeit 2 days ago
  
  “Jargon” is shorthand for people who know what they’re doing. If you’re avoiding jargon, you’re avoiding learning.
  
  Reply View | 3 replies
- epolanski 2 days ago
  
  I see it more of a replacement for Google and digging GitHub issues. It can also replace chats for 80% of questions.
  Not much as a tutor.
  
  Reply View | 0 replies

SJMG 2 days ago

> there is a smaller % that benefits and learns more and faster

That's not what the study says nor it is capable of credibly making that claim. You are reasoning about individuals in an RCT where subjects did not serve as their own control. The high performers in the treatment group may have done even better had they been in the control and AI is in fact is slowing them down.

You don't know which is true because you can't know because of the study design. This is why we have statistics.

Reply View 3 replies

epolanski 2 days ago

So you don't doubt their conclusion that most sucked by using AI, but you doubt that they found that some learned more?

Reply View | 2 replies
- SJMG 2 days ago
  
  The conclusion of the paper doesn't say that "most sucked using AI". It's says the mean quiz score was both significantly and sizably lower in the intervention group vs the control. No significant difference detected on speed.
  The qualitative breakdown says how you use AI matters for understanding. It doesn't say some learned more than the control group and even if it did, it's not powered to show a statistical difference which is one of the only things keeping a study from not being another anecdote on the internet.
  For the sake of argument let's say there is an individual in the treatment arm who scored higher than the highest control participant. What some want that to mean is, "Some engineers perform better using AI". It does not say that. That could be an objective fact(!), it doesn't matter. This study will not support it; it's an RCT. What if that programmer is just naturally gifted or lucky(!). This is the point of statistics.
  The best you can do with outliers is say "AI usage didn't hinder some from attaining a high score" (again maybe it would have been higher w/o you just can't reason about individuals in a study like this).
  I hope this helps.
  
  Reply View | 1 reply
  
  golongboy 2 hours ago
  
  Thank you for this.
  But despite your best efforts to teach epolanski, they’ll never learn. Their comment history shows that they’re one of the MANY confidently incorrect tools on HN.
  
  Reply View | 0 replies