Comment by benburton
I love writing documentation, and I love teaching people how to solve problems. At my day job I've written a lot of the organization's most trafficked explanations and how-to guides for understanding our codebase and engineering principles.
The other week an engineer in another group fed all of my documents, and all of our codebase, into an LLM. They were able to ask it questions, and get immediate answers that were by and large better than the guidance I would have been able to provide in between my other responsibilities.
As much as I love writing and explaining, I think we're sadly past the point that it needs to be done by humans. I have always considered documentation to be an imperfect, point in time, reflection of a codebase. When an LLM can read and synthesize all of code and immediately respond with up to date information... what's the point in writing documentation anymore?
I've read at least 8 articles this week about LLMs having massive hallucinations/brain-farts when writing testbeds for code. Unfortunately, the author didn't see the problems until he tried adding a test; then he had a huge WTF moment.
The fact that the LLM you mention gave good answers is probably more a reflection of YOUR documentation than any particular "brilliance" on the LLM's part.