Comment by fennecfoxy
Comment by fennecfoxy a day ago
Has nothing to do with "cloud providers". The randomness is inherent to the sampler, using a sampler that picks top probability for next token would result in lower quality output as I have definitely seen it get stuck in certain endless sequences when doing that.
Ie you get something like "Complete this poem 'over yonder hills I saw' output: a fair maiden with hair of gold like the sun gold like the sun gold like the sun gold like the sun..." etc.
> would result in lower quality output
No it wouldn't.
> seen it get stuck in certain endless sequences when doing that
Yes, and infinite loops is just an inherent property of LLMs, like hallucinations.