Comment by lsy

The problem with "smart" is that they fail at things that dumb people succeed at. They have ludicrous levels of knowledge and a jaw dropping ability to connect pieces while missing what's right in front of them.

The gap makes me uncomfortable with the implications of the word "smart". It is orthogonal to that.

"There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior" not really. The whole point they are trying to make is that the capability of these models IS ALREADY muddying the definition of intelligence. We can't really test it because the distribution its learned is so vast. Hence why he have things like ARC now.

Even if its just gradient descent based distribution learning and there is no "internal system" (whatever you think that should look like) to support learning the distribution, the question is if that is more than what we are doing or if we are starting to replicate our own mechanisms of learning.

> There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior.

Go on. We are listening.

I think the confusion is because you're referring to a common understanding of what AI is but I think the definition of AI is different for different people.

Can you give your definition of AI? Also what is the "generally accepted baseline definition for what crosses the threshold of intelligent behavior"?

You are doubling down on a muddled vague non-technical intuition about these terms.

Please tell us what that "baseline definition" is.

> Generally its accepted that a core trait of intelligence is an agent’s ability to achieve goals in a wide range of environments.

Be that as it may, a core trait is very different from a generally accepted threshold. What exactly is the threshold? Which environments are you referring to? How is it being measured? What goals are they?

You may have quantitative and unambiguous answers to these questions, but I don't think they would be commonly agreed upon.

What is that baseline threshold for intelligence? Could you provide concrete and objective results, that if demonstrated by a computer system would satisfy your criteria for intelligence?

LLM’s are statistically great at inferring things? Pray tell me how often Google’s AI search paragraph, at the top, is correct or useful. Is that statistically great?

> Generally its accepted that a core trait of intelligence is an agent’s ability to achieve goals in a wide range of environments.

This is the embodiment argument - that intelligence requires the ability to interact with its environment. Far from being generally accepted, it's a controversial take.

Could Stephen Hawking achieve goals in a wide range of environments without help?

And yet it's still generally accepted that Stephen Hawking was intelligent.

> intelligence is an agent’s ability to achieve goals in a wide range of environments. This means you must be able to generalize, which in turn allows intelligent beings to react to new environments and contexts without previous experience or input.

I applaud the bravery of trying to one shot a definition of intelligence, but no intelligent being acts without previous experience or input. If you're talking about in-sample vs out of sample, LLMs do that all the time. At some point in the conversation, they encounter something completely new and react to it in a way that emulates an intelligent agent.

What really makes them tick is language being a huge part of the intelligence puzzle, and language is something LLMs can generate at will. When we discover and learn to emulate the rest, we will get closer and closer to super intelligence.

> The fact that you can reason about intelligence is a counter argument to this

The fact that we can provide a chain of reasoning, and we can think that it is about intelligence, doesn't mean that we were actually reasoning about intelligence. This is immediately obvious when we encounter people whose conclusions are being thrown off by well-known cognitive biases, like cognitive dissonance. They have no trouble producing volumes of text about how they came to their conclusions and why they are right. But are consistently unable to notice the actual biases that are at play.

The ol' "I know it when I see that it thinks like me" argument.

It seems like LLMs can also reason about intelligence. Does that make them intelligent?

We don't know what intelligence is, or isn't.

No offense to johnecheck, but I'd expect an LLM to be able to raise the same counterargument.

Indeed, a good point. My comment assumes that our current model of the human brain is (sufficiently) complete.

Your comment reveals an interesting corollary - those that believe in something beyond our understanding, like the Christian soul, may never be convinced that an AI is truly sapient.

There's sentient humans who can't beat the first generation pokemon games.

Bicycles and cars are too close. The analogy I like is human leg versus tire. That is a starker depiction of how silly it is to compare the two in terms of structure rather than result.

One problem is that we have been basing too much on [human brain] for so long that we ended up with some ethical problems as we decided other brains didn't count as intelligent. As such, science has taken an approach of not assuming humans are uniquely intelligence. We seem to be the best around at doing different tasks with tools, but other animals are not completely incapable of doing the same. So [human brain] should really be [brain]. But is that good enough? Is a fruit fly brain intelligent? Is it a goal to aim for?

There is a second problem that we aren't looking for [human brain] or [brain], but [intelligence] or [sapient] or something similar. We aren't even sure what we want as many people have different ideas, and, as you pointed out, we have different people with different interest pushing for different underlying definitions of what these ideas even are.

There is also a great deal of impreciseness in most any definitions we use, and AI encroaches on this in a way that reality rarely attacks our definitions. Philosophically, we aren't well prepared to defend against such attacks. If we had every ancestor of the cat before us, could we point out the first cat from the last non-cat in that lineup? In a precise way that we would all agree upon that isn't arbitrary? I doubt we could.

If you don't know anything except how words are used, you can definitely disambiguate "bicycle" and "car" solely based on the fact that the contexts they appear in are incongruent the vast majority of the time, and when they appear in the same context, they are explicitly contrasted against each other.

This is just the "fancy statistics" argument again, and it serves to describe any similar example you can come up with better than "intelligence exists inside this black box because I'm vibing with the output".

A more reasonable application of Occam's razor is that humans also don't meet the definition of "intelligence". Reasoning and perception are separate faculties and need not align. Just because we feel like we're making decisions, doesn't mean we are.

> it all rests on (relatively) simple mathematics. We know this is true. We also know that means it has limitations and can't actually reason information.

What do you imagine is happening inside biological minds that enables reasoning that is something different to, a lot of, "simple mathematics"?

You state that because it is built up of simple mathematics it cannot be reasoning, but this does not follow at all, unless you can posit some other mechanism that gives rise to intelligence and reasoning that is not able to be modelled mathematically.

Again, if you look at the early papers on AI, you'll see things that are even farther from human intelligence than the LLMs of today. There is no "perversion" of the term, it has always been a vague hypey concept. And it was introduced in this way by academia, not business.

While it could possibly be to point out so abruptly, you seem to be the walking talking definition of the AI Effect.

>The "AI effect" refers to the phenomenon where achievements in AI, once considered significant, are re-evaluated or redefined as commonplace once they become integrated into everyday technology, no longer seen as "true AI".

Completely agree. But if Markov chains are AI (and they always were categorized as such), then fancy Markov chains are still AI.

An LLM is no more a fancy Markov Chain than you are. The math is well documented, go have a read.

The results make the method interesting, not the other way around.

Markov chains in meatspace running on 20W of power do quite a good job of actual intelligence

But that's exactly what these deep neural networks have shown, countless times. LLM's generalize on new data outside of its training set. It's called "zero shot learning" where they can solve problems that are not in their training set.

AlphaGo Zero is another example. AlphaGo Zero mastered Go from scratch, beating professional players with moves it was never trained on

> Another is the fundamental inability to self update

That's an engineering decision, not a fundamental limitation. They could engineer a solution for the model to initiate its own training sequence, if they decide to enable that.

I just made up this scenario and these words, so I'm sure it wasn't in the training data.

Kwomps can zark but they can't plimf. Ghirns are a lot like Kwomps, but better zarkers. Plyzers have the skills the Ghirns lack.

Quoning, a type of plimfing, was developed in 3985. Zhuning was developed 100 years earlier.

I have an erork that needs to be plimfed. Choose one group and one method to do it.

> Use Plyzers and do a Quoning procedure on your erork.

If that doesn't count as reasoning or generalization, I don't know what does.

https://chatgpt.com/share/67eee8dd-43a4-8004-afde-2fe3fc9630...

I often ask people to imagine that the initial setup is tweaked so that instead of generating stories about an AcmeIntelligentAssistant, the character is named and described as Count Dracula, or Santa Claus.

Would we reach the same kinds of excited guesses about what's going on behind the screen... or would we realize we've fallen for an illusion, confusing a fictional robot character with the real-world LLM algorithm?

The fictional character named "ChatGPT" is "helpful" or "chatty" or "thinking" in exactly the same sense that a character named "Count Dracula" is "brooding" or "malevolent" or "immortal".

A lot of people don't have internal monologues. But chain of thought is about expanding capacity by externalising what you're understood so far so you can work on ideas that exceeds what you're capable of getting in one go.

That people seem to think it reflects internal state is a problem, because we have no reason to think that even with internal monologue that the internal monologue accurately reflects our internal thought processes fuly.

There are some famous experiments with patients whose brainstem have been severed. Because the brain halves control different parts of the body, you can use this to "trick" on half of the brain into thinking that "the brain" has made a decision about something, such as choosing an object - while the researchers change the object. The "tricked" half of the brain will happily explain why "it" chose the object in question, expanding on thought processes that never happened.

In other words, our own verbalisation of our thought processes is woefully unreliable. It represents an idea of our thought processes that may or may not have any relation to the real ones at all, but that we have no basis for assuming is correct.

Just grabbing definitions from Google:

Rationalize: "An attempt to explain or justify (one's own or another's behavior or attitude) with logical, plausible reasons, even if these are not true or appropriate"

Rational: "based on or in accordance with reason or logic"

They sure seem like related concepts to me. Maybe you have a different understanding of what "rationalizing" is, and I'd be interested in hearing it

But if all you're going to do is drive by comment saying "You're wrong" without elaborating at all, maybe just keep it to yourself next time

The context is whether this capability is required to qualify as AGI. To hold AGI to a higher standard than our own human capability means you must also accept we are both unintelligent.

Of course I’m anthropomorphizing. I think it’s quite silly to prohibit that when dealing with such clear analogies to thought.

Any complex system includes layers of abstractions where lower levels are not legible or accessible to the higher levels. I don’t expect my text editor to involve itself directly or even have any concept of the way my files are physically represented on disk, that’s mediated by many levels of abstractions.

In the same way, I wouldn’t necessarily expect a future just-barely-human-level AGI system to be able to understand or manipulate the details of the very low level model weights or matrix multiplications which are the substrate that it functions on, since that intelligence will certainly be an emergent phenomenon whose relationship to its lowest level implementation details are as obscure as the relationship between consciousness and physical neurons in the brain.

Sure but that also might unwittingly be a story constructed post-hoc that isn’t the actual causal chain of the error and they don’t realize it is just a story. Many cases. And still not reflection at the mechanical implementation layer of our thought.

Humans have short term memory. LLMs have context windows. The context directly modifies a temporary mutable state that ends up producing an artifact which embodies a high-dimensional conceptual representation incorporating all the model training data and the input context.

Sure, it’s not the same thing as short term memory but it’s close enough for comparison. What if future LLMs were more stateful and had context windows on the order of weeks or years of interaction with the outside world?

No, what was described was specifically reporting to an external party the neural connections involved in the mistake and the source in past training data that caused them, as well as learning from new data.

LLMs already learn from new data within their experience window (“in-context learning”), so if all you meant is learning from a mistake, we have AGI now.

Yes thank you, that's what I was getting at. Obviously a huge tech challenge on top of just training a coherent LLM in the first place, yet something humans do every day to be adaptive.

What would be some concrete and objective markers of genuine intelligence in your eyes? Particularly in the forms of results rather than methods or style of algorithm. Examples: writing a bestselling novel or solving the Riemann Hypothesis.

[dead]