Comment by plasticeagle

This is wrong. I write toy languages and frameworks for fun. These are APIs that simply don't exist outside of my code base, and LLMs are consistently able to:

* Read the signatures of the functions.

* Use the code correctly.

* Answer questions about the behavior of the underlying API by consulting the code.

Of course they're just guessing if they go beyond what's in their context window, but don't underestimate context window!

This is such a pointless, tired take.

You want to say this guy's experience isn't reproducible? That's one thing, but that's probably not the case unless you're assuming they're pretty stupid themselves.

You want to say that it Is reproducible, but that "that doesn't mean AI can think"? Okay, but that's not what the thread was about.

This doesn't seem like a useful nor accurate way of describing LLMs.

When I built my own programming language and used it to build a unique toy reactivity system and then asked the LLM "what can I improve in this file", you're essentially saying it "only" could help me because it learned how it could improve arbitrary code before in other languages and then it generalized those patterns to help me with novel code and my novel reactivity system.

"It just saw that before on Stack Overflow" is a bad trivialization of that.

It saw what on Stack Overflow? Concrete code examples that it generalized into abstract concepts it could apply to novel applications? Because that's the whole damn point.

>It is absolutely true, and AI cannot think, reason, comprehend anything it has not seen before. If you're getting answers, it has seen it elsewhere, or it is literally dumb, statistical luck.

How would you reconcile this with the fact that SOTA models are only a few TB in size? Trained on exabytes of data, yet only a few TB in the end.

Correct answers couldn't be dumb luck either, because otherwise the models would pretty much only hallucinate (the space of wrong answers is many orders of magnitude larger than the space of correct answers), similar to the early proto GPT models.

What would convince you otherwise? The reason I ask is because you sound like you have made up your mind phylosophically, not based on practical experience.

It's just Pattern matching. Most APIs, and hell, most code is not unique or special. Its all been done a thousands of times before. Thats why an LLM can be helpful on some tool you've written just for yourself and never released anywhere.

As to 'knows the answer', I'm don't even know what that means with these tools. All I know is if it is helpful or not.

> It is absolutely true, and AI cannot think, reason, comprehend anything it has not seen before.

The amazing thing about LLMs is that we still don’t know how (or why) they work!

Yes, they’re magic mirrors that regurgitate the corpus of human knowledge.

But as it turns out, most human knowledge is already regurgitation (see: the patent system).

Novelty is rare, and LLMs have an incredible ability to pattern match and see issues in “novel” code, because they’ve seen those same patterns elsewhere.

Do they hallucinate? Absolutely.

Does that mean they’re useless? Or does that mean some bespoke code doesn’t provide the most obvious interface?

Having dealt with humans, the confidence problem isn’t unique to LLMs…

I'm sorry but this is a gross oversimplification. You can also apply this to the human brain.

"<the human brain> cannot think, reason, comprehend anything it has not seen before. If you're getting answers, it has seen it elsewhere, or it is literally dumb, statistical luck."

This isn't about being AGI or not, and it's not "out of thin air".

Modern implementations of LLMs can "do research" by performing searches (whose results are fed into the context), or in many code editors/plugins, the editor will index the project codebase/docs and feed relevant parts into the context.

My guess is they either were using the LLM from a code editor, or one of the many LLMs that do web searches automatically (ie. all of the popular ones).

They are answering non-stackoverflow questions every day, already.

I think the time has come to not mean LLMs when talking about AI. An agent with web access can do so much more and hallucinates way less than "just" the model. We should start seeing the model as a building block of an AI system.

> LLM has come up with out of thin air

People don't think that. Especially not the commentor you replied to. You're human-hallucinating.

People think LLM are trained on raw documents and code besides StackOverflow. Which is very likely true.