Comment by IIsi50MHz

Comment by IIsi50MHz 2 months ago

At work, we had an appliance which went into failsafe on average 8 times per day. The failsafe is meant to remove power from a device-under-test in case of something like fire in the DUT. The few actual critical failures were not detected by the appliance.

Instead, the failsafe has the effect of merely invalidating the current test, and making the appliance unable to run a test correctly until either power cycled or the appliance's developer executes a secret series of commands that are not shared with us.

So of course an operator of the appliance found a way to feed in a false "I'm here!" with a loop, to trick the appliance into never going into failsafe…

That's for ~6.8% of all tests being false-positive, ~93.2% being true-negative, and ~3 tests that should have triggered failsafe did not.

PoignardAzur 2 months ago

Sooooo... You're saying that the chance of a true positive given an alert was much less than 33%?

I don't if you meant it as a counterpoint for what I said, but it really isn't.

Reply View 2 replies

IIsi50MHz 2 months ago

Sorry, I meant to say that with only 6.8% of all tests triggering a false alarm (and 0% true alarm), a test operator still found a way to prevent the alarm from occurring rather than being kept on their toes.

Reply View | 1 reply
- PoignardAzur 2 months ago
  
  Yeah, but again, the problem isn't the high false positive rate.
  The problem is that given any positive at all, the chance it points to a problem is still virtually zero.
  If it was 6.8% of all tests as false positives and 2% true positives, probably people wouldn't have silenced the alarm.
  If it goes off 8 times a day and 2 of them are true positives, then people have recent memories of having to fix problems pointed by the alarm.
  
  Reply View | 0 replies