Comment by semiquaver
Comment by semiquaver 2 days ago
Humans have short term memory. LLMs have context windows. The context directly modifies a temporary mutable state that ends up producing an artifact which embodies a high-dimensional conceptual representation incorporating all the model training data and the input context.
Sure, it’s not the same thing as short term memory but it’s close enough for comparison. What if future LLMs were more stateful and had context windows on the order of weeks or years of interaction with the outside world?
Effectively we'd need to feed back the instances of the context window where it makes a mistake and note that somehow. Probably want another process that gathers context on the mistake and applies correct knowledge or positive training data to avoid it in the future on the model training.
Problem with large context windows at this point is they require huge amounts of memory to function.