Comment by dcminter 4 days ago
This could not possibly go wrong...
You're braver than me if you're willing to trust the LLM here - fine if you're ready to properly review all the relevant docs once you have code in hand, but there are some very expensive risks otherwise.
There’s nothing brave in this. It generally works the way it should and even if it doesn’t - you just go back to see what went wrong.
I take code from stack overflow all the time and there’s like a 90% chance it can work. What’s the difference here?
For those high stakes cases maybe you can be more careful. You can still use an LLM to search and get references to the appropriate place and do your own verification.
But for low stakes LLM works just fine - not everything is going to blow up to a 30,000 bill.
In fact I'll take the complete opposite stance - verifying your design with an LLM will help you _save_ money more often than not. It knows things you don't and has awareness of concepts that you might have not even read about.
Well, the "accidentally making the S3 bucket public" scenario would be a good one. If you review carefully with full understanding of what e.g. all your policies are doing then great, no problem.
If you don't do that will you necessarily notice that you accidentally leaked customer data to the world?
The problem isn't the LLM it's assuming its output is correct just the same as assuming Stack Overflow answers are correct without verifying/understanding them.
I agree but its about the extent. I'm willing to accept the risk of ocassionally making S3 public but getting things done much faster, much like I don't meticulously read documentation when I can get the answer from stackoverflow.
If you are comparing with stackoverflow then I guess we are on the same page - most people are fine with taking stuff from stackoverflow and it doesn't count as "brave".
I think anyone who just copies and pastes from SO is indeed "brave" for pretty much exactly the same reason.
> I'm willing to accept the risk of ocassionally making S3 public
This is definitely where we diverge. I'm generally working with stuff that legally cannot be exposed - with hefty compliance fines on the horizon if we fuck up.
That's fair - I would definitely use stackoverflow liberally and dive into documentation when situation demands it.
The thing is that you can now ask the LLM for links and you can ask it to break down why it thinks a piece of code, for example, protects the bucket from being public. Things that are easy to verify against the actual docs.
I feel like this workflow is still less time, easier and less error prone than digging out the exact right syntax from the AWS docs.
This is LLM as semantic search- so it's way way easier to start from the basic example code and google to confirm that it's correct than it is to read the docs from scratch and piece together the basic example code. Especially for things like configurations and permissions.