Comment by smaudet

vidarh 6 months ago

> The second portion of your statement is either confusing (something unsaid) or untrue (you are still ultimately on the hook).

You're missing the point.

tptacek is saying he isn't the one who needs to fix the issue because he can just reject the PR and either have the AI agent refine it or start over. Or ultimately resort to writing the code himself.

He doesn't need to make the AI written code work, and so he doesn't need to spend a lot of time reading the AI written code - he can skim it for any sign it looks even faintly off and just kill it if that's the case instead of spending more time on it.

> My previous point stands - if it was that cut and dry, then a (free) script/library could generate the same code.

There's a vast chasm between simple enough that a non-AI code generator can generate it using templates and simple enough that a fast read-through is enough to show that it's okay to run.

As an example, the other day I had my own agent generate a 1kloc API client for an API. The worst case scenario other than failing to work would be that it would do something really stupid, like deleting all my files. Since it passes its tests, skimming it was enough for me to have confidence that nowhere does it do any file manipulation other than reading the files passed in. For that use, that's sufficient since it otherwise passes the tests and I'll be the only user for some time during development of the server it's a client for.

But no template based generator could write that code, even though it's fairly trivial - it involved reading the backend API implementation and rote-implementation of a client that matched the server.

Reply View 4 replies

smaudet 6 months ago

> But no template based generator could write that code, even though it's fairly trivial
Not true at all, in fact this sort of thing used to happen all the time 10 years ago, code reading APIs and generating clients...
> He doesn't need to make the AI written code work, and so he doesn't need to spend a lot of time reading the AI written code - he can skim it for any sign it looks even faintly off and just kill it if that's the case instead of spending more time on it.
I think you are missing the point as well, that's still review, that's still being on the hook.
Words like "skim" and "kill" are the problem here, not a solution. They point to a broken process that looks like its working...until it doesn't.
But I hear you say "all software works like that", well, yes, to some degree. The difference being, one you hopefully actually wrote and have some idea what's going wrong, the other one?
Well, you just have to sort of hope it works and when it doesn't, well you said it yourself. Your code was garbage anyways, time to "kill" it and generate some new slop...

Reply View | 3 replies
- vidarh 6 months ago
  
  > Not true at all, in fact this sort of thing used to happen all the time 10 years ago, code reading APIs and generating clients...
  Where is this template based code generator that can read my code, understand it, and generate a full client including a CLI, that include knowing how to format the data, and implement the required protocols?
  I'm 30 years of development, I've seen nothing like it.
  > I think you are missing the point as well, that's still review, that's still being on the hook.
  I don't know if you're being intentionally obtuse, or what, but while, yes, you're on the hook for the final deliverable, you're not on the hook for fixing a specific instance of code, because you can just throw it away and have the AI do it all over.
  The point you seem intent on missing is that the cost of throwing out the work of another developer is high, while the cost of throwing out the work of an AI assistant is next to nothing, and so where you need to carefully review a co-workers code because throwing it away and starting over from scratch is rarely an option, with AI generated code you can do that at the slightest whiff of an issue.
  > Words like "skim" and "kill" are the problem here, not a solution. They point to a broken process that looks like its working...until it doesn't.
  No, they are not a problem at all. They point to a difference in opportunity cost. If the rate at which you kill code is too high, it's a problem irrespective of source. But the point is that this rate can be much higher for AI code than for co-workers before it becomes a problem, because the cost of starting over is orders of magnitude different, and this allows for a very different way of treating code.
  > Well, you just have to sort of hope it works and when it doesn't
  No, I don't "hope it works" - I have tests.
  
  Reply View | 2 replies
  
  smaudet 6 months ago
  
  > Where is this template based code generator that can read my code, understand it, and generate a full client including a CLI, that include knowing how to format the data, and implement the required protocols?
  I'd argue you are quite a bit beyond "rote" code at that point (with the understanding and protocol bits). But, generating client code is not hard, there are numerous generators around e.g. swagger:
  https://swagger.io/ https://swagger.io/tools/swagger-codegen/
  In ten years I expect other generators/platforms exist too, that's merely one I'm familiar with.
  > you're not on the hook for fixing a specific instance of code, because you can just throw it away and have the AI do it all over. > ... > No, I don't "hope it works" - I have tests.
  These are contradictory statements. Every instance of that code you are responsible for, or you wouldn't test it and you wouldn't deign to "need" to throw it away.
  > They point to a difference in opportunity cost.
  Yes, we are all ultimately concerned with this. However this is not an easy metric to quantify, clearly you feel your OC (Opportunity Cost) because maybe you don't work well with other humans, ok whatever, however you are likely overestimating the supposed savings, and underestimating the lost OC of working with other developers, or simply writing code that doesn't need to be thrown out at all...
  
  Reply View | 1 reply
  
  tptacek 6 months ago
  
  I know you two are off on your own thing right now, which is cool, but I just want to say that the point of my comment is solely that the kind of code review involved in LLM output is different and easier than human code review (because of the lack of obligation to salvage the code if it's suspect), and we all seem to have reached a consensus on that point.
  I explicitly wasn't trying to persuade anyone that the cost/benefit tradeoff for LLM coding was positive. I obviously believe it is, but reasonable people can disagree.
  
  Reply View | 0 replies