Comment by johnsmith1840

Totally possible!

I just like talking about it. I will say that learning outside distribution content while keeping previous knowledge in a "useful" state is a capability that would absolutely supercharge ever AI method we currently have.

It's atleast an honest atempt at a research direction other than "scale infinitely for everything" that we currently do.

Just think about how natural brains do something incredible.

1. They have fixed computation budgets per time step. 2. They continously learn entirely new tasks while still maintaining previous in a useful state.

That's a capability I would very much like in my AI.

Scaling laws are correct but they are also the reason we are nowhere near replacing humans.

Take a simple job maybe admin work. Every timestep depends on the previous timestep. While not a complex job and an AI could do it for awhile but over time the compuation required to "look back" its memory and connect it for the next step grows near exponentially.

RAG is another perfect example of this problem.

I do deeply belive AGI will be solved by a kid with a whiteboard not a supercluster. CL is my best guess at what that means.

Maybe it's a super RL or energy type method but I've never seen it.