Comment by deafpolygon
Comment by deafpolygon a day ago
It will generate a correct next token 42% of the time when prompted with a 50 token quote.
Not 42% of the book.
It's a pretty big distinction.
Comment by deafpolygon a day ago
It will generate a correct next token 42% of the time when prompted with a 50 token quote.
Not 42% of the book.
It's a pretty big distinction.
This means that if we start with 50% of the book then there is 42% chance that we can recreate the remaining 50%.
What is the distinction between understanding and memorization? What is the chance that understanding results in memorization (may be in case of humans)?
next _50_ tokens 42% of the time
not just next token.
This is like: tell it a random sentence in the book, it will give you the next sentence 42% of time.