Comment by tomblomfield 3 days ago
Since I wrote the blog post, I actually added automated tests which Windsurf / Claude Code runs before committing any change. It saved me a lot of headaches when the LLM decided to make random logic changes for no reason.
Absolutely, very often. It typically happens because it's trying to do more than you asked for.
Example scenario: you have a codebase that you iterated on with LLM, and it contains let's say 15 features with various implementation details. You continue tomorrow and want to make a small change to handle an edge-case. While making changes to one of the 3 required files, suddenly it will decide to also rewrite / "improve" other parts of the code that have nothing to do with your request, where pieces of previous logic will no longer exist - since it made 18 changes to the file, and there are 3 such files, good luck spotting it without thorough and detailed change review.
Also, if you made manual changes to generated code and than you ask it to add something to the code (within the same "conversation" / context, it will often replace your changes with how it originally wanted code to look like.
I experienced this, too. It is quite annoying. You have to explicitly tell the LLM to leave everything else as-is.
"Let's take a different approach. Instead of trying to fix the existing file incrementally, let's rewrite it completely with a simpler approach that uses the --print-config flag to test the configuration:"
The LLM swaps out business logic without you knowing it? As someone who worries about determinism and doesn’t do much vibe coding, I am curious what’s going on here, thanks!