Comment by troupo
Before you get into the expensive part, how do you get past "non-deterministic blackbox with unknown layers in between imposed by vendors"
Before you get into the expensive part, how do you get past "non-deterministic blackbox with unknown layers in between imposed by vendors"
You can measure probabilistic systems that you can't examine! I don't want to throw the baby out with the bathwater here - before LLMs became the all-encompassing elephant in the room we did this routinely.
You absolutely can quantify the results of a chaotic black box, in the same way you can quantify the bias of a loaded die without examining its molecular structure.