datadrivenangel 4 days ago

With an open model, you could probably reverse engineer the token probabilities and get that probability estimate.

Something like: "Is {sentence_a} a plausible answer to {sentence_b}? Respond only with a single yes/no token" and then look at the probabilities of those.

Reply View 1 reply
  • wongarsu 4 days ago

    If the model is not open turn up the temperature a bit (if the API allows that) and ask the above question multiple times. The less sure the model is the more the answer will vary.

    Reply View 0 replies
danielmarkbruce 4 days ago

Absolutely you can. Rip off the last layer, add a regression layer in it's place, fine tune.

Reply View 0 replies
OutOfHere 3 days ago

Of course one can just ask the LLM for the output probability. It will give a reasonably calibrated output, typically a multiple of 0.05. I would ask it for an integer percentage though.

Reply View 0 replies