Comment by pllbnk
A human, not a statistical model. I can insert any random words out of my own volition if I wanted to, not because I have been pre-programmed (pre-trained) to output tokens based on a limited 200k (tiny) context for one particular conversation and forget about it by the time a new session starts.
That’s why AI models, as they currently are, won’t ever be able to come up with anything even remotely novel.
Well, if you believe you’re powered by physical neurons and not spooky magic, that doesn’t seem very different from being a neural net.
I see no evidence for you magical ability to behave outside of being a function of context and memory.
You don’t think diffusion models are capable of novelty?