Comment by otabdeveloper4
Comment by otabdeveloper4 11 hours ago
LLMs aren't probabilistic. The randomness is bolted on top by the cloud providers as a trick to give them a more humanistic feel.
Under the hood they are 100% deterministic, modulo quantization and rounding errors.
So yes, it is very much possible to use LLMs as a lossy compressed archive for texts.
Has nothing to do with "cloud providers". The randomness is inherent to the sampler, using a sampler that picks top probability for next token would result in lower quality output as I have definitely seen it get stuck in certain endless sequences when doing that.
Ie you get something like "Complete this poem 'over yonder hills I saw' output: a fair maiden with hair of gold like the sun gold like the sun gold like the sun gold like the sun..." etc.