Comment by jmalicki

Comment by jmalicki 3 days ago

For all practical purposes any code reliant on the output of a PRNG is non-deterministic in all but the most pedantic senses... And if the LLM temperature isn't set to 0 LLMs are sampling from a distribution.

If you're going to call a PRNG deterministic then the outcome of a complicated concurrent system with no guaranteed ordering is going to be deterministic too!

gmueckl 3 days ago

No, this isn't right. There are totally legitimate use cases for PRNGs as sources of random number sequences following a certain probability distribution where freezing the seed and getting reproducibility is actually required.

Reply View 2 replies

jmalicki 3 days ago

And for a complicated concurrent system you can also replay the exact timings and orderings as well!

Reply View | 1 reply
- gmueckl 2 days ago
  
  That's completely different from PRNGs. I don't understand why you think those things belong together.
  
  Reply View | 0 replies

bonoboTP 3 days ago

How is this related to overloading? The nondeterminism should not be a function of overloading. It should just time out or reply slower. It will only be dumber if it gets rerouted to a dumber, faster model eg quantized.

Reply View 0 replies

joquarky 3 days ago

Temperature can't be literally zero, or it creates a divide by zero error.

When people say zero, it is shorthand for “as deterministic as this system allows”, but it's still not completely deterministic.

Reply View 1 reply

forgotTheLast 2 days ago

Zero temp just uses argmax, which is what softmax approaches if you take the limit of T to zero anyway. So it could very well be deterministic.

Reply View | 0 replies