Comment by willvarfar

I don't think the probabilistic prediction is a problem. The problem with current LLM is that they are limited to doing "System 1" thinking, only giving you a fast instinctive response to a question. While that works great for a lot of small problems, it completely falls apart on any larger task that requires multiple steps or backtracking. "System 2" thinking is completely missing as is the ability to just self-iterate on their own output.

Reasoning models are trying to address that now, but monologueing in token-space still feels more like a hack than a real solution, but it does improve their performance a good bit nonetheless.

In practical terms all this means is that current LLMs still need a hell of a lot of hand holding and fail at anything more complex, even if their "System 1" thinking is good enough for the task (e.g. they can write Tetris in 30sec no problem, but they can't write SuperMarioBros at all, since that has numerous levels that would blow the context window size).

> But how are animals with nerve-centres or brains different?

In current LLM neural networks, the signal proceeds in one direction, from input, through the layers, to output. To the extend that LLM's have memory and feedback loops, it's that they write the output of the process to text, and then read that text and process it again though their unidirectional calculations.

Animal brains have circular signals and feedback loops.

There are Recurrent Neural Network (RNN) architectures, but current LLM's are not these.

> But how are animals with nerve-centres or brains different? What do we think us humans do differently so we are not just very big probabilistic prediction systems?

I see this statement thrown around a lot and I don't understand why. We don't process information like computers do. We don't learn like they do, either. We have huge portions of our brains dedicated to communication and problem solving. Clearly we're not stochastic parrots.

> if we develop the technology to engineer animal-style nerves and form them into big lumps called 'brains'

I think y'all vastly underestimate how complex and difficult a task this is.

It's not even "draw a circle, draw the rest of the owl", it's "draw a circle, build the rest of the Dyson sphere".

It's easy to _say_ it, it's easy to picture it, but actually doing it? We're basically at zero.

Human (and other animal) brains probably are probabilistic, but we don't understand their structure or mechanism in fine enough detail to replicate them, or simulate them.

People think LLMs are intelligent because intelligence is latent within the text they digest, process and regurgitate. Their performance reflects this trick.

> But how are animals with nerve-centres or brains different? What do we think us humans do differently so we are not just very big probabilistic prediction systems?

If you believe in free will, then we are not.