Comment by wizzwizz4

Comment by wizzwizz4 4 days ago

> Isn’t this a good thing since compute and be scaled so that the LLM can do generations of human thinking in a much shorter amount of time?

But it can't. There isn't enough planet.

> The major assumption here is that transformers can indeed solve every problem humans can.

No, the major assumptions are (a) that ChatGPT can, and (b) that we can reduce the resource requirements by many orders of magnitude. The former assumption is highly-dubious, and the latter is plainly false.

Transformers are capable of representing any algorithm, if they're allowed to be large enough and run large enough. That doesn't give them any special algorithm-finding ability, and finding the correct algorithms is the hard part of the problem!

aurareturn 3 days ago

> But it can't. There isn't enough planet.

How much resource are you assuming an AGI would consume?

Reply View 3 replies

wizzwizz4 3 days ago

Are we talking about "an AGI", or are we talking about overfitting large transformer models with human-written corpora and scaling up the result?
"An AGI"? I have no idea what that algorithm might look like. I do know that we can cover the majority of cases with not too much effort, so it all depends on the characteristics of that long tail.
ChatGPT-like transformer models? We know what that looks like, despite the AI companies creatively misrepresenting the resource use (ref: https://www.bnnbloomberg.ca/business/technology/2024/08/21/h...). Look at https://arxiv.org/pdf/2404.06405:
> Combining Wu’s method with the classic synthetic methods of deductive databases and angle, ratio, and distance chasing solves 21 out of 30 methods by just using a CPU-only laptop with a time limit of 5 minutes per problem.
AlphaGeometry had an entire supercomputer cluster, and dozens of hours. GOFAI approaches have a laptop and five minutes. Scale that inconceivable inefficiency up to AGI, and the total power output of the sun may not be enough.

Reply View | 2 replies
- aurareturn 3 days ago
  
  When computers first became useful, you needed computers the size of rooms to compute. In 2024, my earphones have more compute.
  
  Reply View | 1 reply
  
  krige 3 days ago
  
  It's always a hindsight declaration though. Currently we can only say that Intel has reused the same architecture several times already and cranking up the voltage until it breaks because they seem to be yet to find the next design leap, while AMD has been toying around with 3D placement but their latest design is woefully unimpressive. We do not know when the next compute leap will happen until it happens.
  
  Reply View | 0 replies