Comment by AlotOfReading
Comment by AlotOfReading 3 days ago
Long ago I worked on the firmware for a game controller. We started getting reports back of ghost inputs like stuck buttons and false presses after we sent some early hardware to media reviewers. Given the power of game media at the time, this was an immediate code red. We took shifts playtesting various video games for nearly a week straight just to try and replicate the issues. No luck, only the reviewers could manifest it. We were about to put reviewers on a plane to demonstrate the issue in person when I decided to clean my desk. In doing so I tossed a bare PCB running debug to the other side of the desk and my console went wild.
Turns out the PCBs were shock/pressure sensitive, and the debouncing was just a bit off. Reviewers were getting really into their games and mechanically stressing the controllers. Stressed hard enough, the PCB would bend slightly, causing line level fluctuations and eventually ghost inputs. Back in the office we were just doing a job and not getting too emotionally involved in our playtesting.
Some new molds and review units later we shipped the working system. Percussive debugging has solved a number of otherwise intractable bugs over my career.
Had a bug about two years ago that I just could not reproduce. In fact, only the engineer who reported it could reproduce it. Finally, I did "go to gemba" and went into the test lab and watched him use the machine.
He was doing a complex operation and was young and fast enough to overcome the keypress reporting interval. Literally it was just someone doing something we didn't expect and doing it fast enough that the sequence got messed up.