Comment by Centigonal

I don't think forgetting correctly is something humans are really good at. I'm not convinced human brains are "exceptionally good" at much of what we do tbh. I think human brain memory capacity is so large that most of forgetting is nowhere near "clearing space for new info" but because the brain correctly knows that some past bad information interferes with learning new things.

As far as I know we have made very little progress on identifying which weights to what degree in an ANN are responsible for what output and as such we cannot discard information, that a user would mark as wrong or inaccurate or undesirable. The human mind however, can do this easily. We remember (though not perfectly) that something is wrong, classified as not useful, irrelevant, and we don't do that any longer and over time might even forget about that now less traveled path. An ANN has no obvious mechanism for that at least.

Learning is strongly related to spaced repetition.

This is often associated with learning tools like anki and stuff, but the real world is all about encountering things at certain frequencies (day night cycles, seasons, places you visit, people you see.... everything, really)

I'm wondering if there maybe some sort of inverse to SR, maybe?

Did an interesting study that actually LLMs "hide" internal data.

They don't just "forget" that information can come back at a later time if you continue to train.

So basically any time a model is trained you need to check it's entire memory not just a small part.

Is it some form of least-recently-used approach? I'm running tests on my own mind trying to figure it out now :D part of what I love about this area of computer science.