Comment by andrewmcwatters

This is such an annoying issue in assisted programming as well.

Say you’re referencing a specification, and you allude to two or three specific values from that specification, you mention needing a comprehensive list and the LLM has been trained on it.

I’ll often find that all popular models will only use the examples I’ve mentioned and will fail to elaborate even a few more.

You might as well read specifications yourself.

It’s a critical feature of these models that could be an easy win. It’s autocomplete! It’s simple. And they fail to do it every single time I’ve tried a similar abstract.

I laugh any time people talk about these models actually replacing people.

They fail at reading prompts at a grade school reading level.