Comment by mmsc

With all due respect, That is the price you pay for your users doing _free_ software testing for you! We are on the “listen to your users” mecca and you’re complaining that listening to your users is hard and you wish a machine could help you with it.

I think a lot of people here seem to totally fail to understand the user's perspective. Reporting on bugs is hard, because adding actual, helpful context to a bug is actual (free) labour. Yes, filtering out useless reports is hard for you, but that's the price you are gonna have to pay for having people do free labour (you get some unhelpful reports). You want to increase signal-to-noise ratio by focusing on decreasing the noise, whereas you should actually focus on increasing the signal.

Making simple, useless bug reports is easy and it will always be the easiest. Also the "my neighbour spies for the government" types will anyway always be the most motivated ones. There is no way to make it hard for "bad" reports without making it harder also for useful reports (barring some obvious cases of bots, ip filters etc, which are not what is discussed here and are a general problem not just for bug feedback). By trying to reduce the noise, you also reduce the signal thus get a worse SNR.

The specific tool is smart in trying to increase the signal. If you make it easier for users to add some useful context, MAYBE you get more users actually giving you sth useful, maybe even users who otherwise would not bother to add anything more useful than "it does not work".

I use software that recently made much simpler to make bug reports and add context, and they say they actually receive much better bug reports after. And most importantly, the users actually see that the bugs get fixed, which motivate them to make more, and more detailed, bug reports. Imo getting bugs fixed (and maybe even recognise the users' contribution in reporting them) is the best way to get good bug reports. Honestly, from my user's perspective having my feedback taken seriously is the best motivation for me to continue submitting reports. Because, honestly, sometimes bugs come up in complex situations that may be tricky to understand/reproduce, and it is hard to understand what context is relevant. I am not usually motivated as a user to spend like 20 minutes figuring out exactly how to reproduce a bug, but if I see that the company/engineers actually care and try to make it easy to me to report to them, I may actually do it.

Yes you are gonna have bad interactions also (and remember people have their own jobs/lives/not enough time to always engage with you the way you may want them to in providing feedback), but the point is to increase the good/useful interactions (compared to them), not decrease interactions in general. Unless you do not care much about bug reports anyway, that's also fine.

> The difficulty in reporting a bug comes from the friction required to filter the "page doesn't work" with no further explanation reports

This so much.

I can't tell you how often I've seen someone trying to get tech support on something say "When I load the program, I get an error" but don't even say what the error says. I understand that most people have never worked a QA job and so don't know how to write a good bug report, but certainly I would expect someone to copy/paste the error message.

On the LLM idea, if you could group reports by issue (by parsing the user provided input and whatever context you save from the page screenshot into some embedding) and then only escalate things when several different IPs have reported a similar thing within X amount of time, I think you could handle two birds with one stone. Limits how annoying spammers can be, and also makes the good reports easier to understand since a few bug reports combined should make a better whole.

I however wouldn't shorten/transform reports with an LLM, or make spammy reports inaccessible. Just doing the semantic grouping for escalation. It's true you're getting free work from your users, and the human factor is pretty important here, even if an LLM might sometimes misinterpret it.

> "page doesn't work" with no further explanation reports

It cuts both ways. Guess what's one of the most popular format for apps and webpages to report failures to the user?

"Oops. Something went wrong."

Not exactly overflowing with useful information, either.

Sure, the system is probably logging the fault internally, and is always collecting metrics that help with contextualization later. But the system and its owner aren't usually the ones most affected by any given bug - it's the user who is. The user who's now worrying whether it means they're about to lose the time and work they put in the current session, or whether the app just ate their money (failures half-way through payment processes are the cutest, aren't they?). They don't know - maybe the "Oops!" was just benign, or irrelevant. Then again, maybe they've already lost it all 10 minutes ago - back when the previous "Oops!" briefly flashed to gently inform them that the service's back-end tripped over itself and died - but they won't discover that until later, at which point they'll be neither able nor willing to make a proper bug report.

Point being, if one sees their users as being 5 years old (but with parents' credit card in hands), one shouldn't be surprised to only ever get a kindergarten-level error reports like the ones you mentioned[0].

This is not just me complaining on a tangential issue - I believe showing specific and accurate error messages improves the ratio of useful error reports. It's not a full solution, but it's a step in the right direction. Treating them as partners, instead of a bunch of brats you have to put up with until they complete the payment, makes them more willing to reciprocate; giving users means to contextualize their experience allows some of them[1] to understand what's going on, and gives them something useful to put in the report too.

That, or I guess nowadays you can also keep the "Oops."-es, double-down on telemetry, and feed the metrics to a SOTA LLM to identify and interpret failures for your engineering/operations team, which we all know has neither time nor patience to do it.

--

[0] - "Page doesn't work" is the adult version of a kid suddenly starting to cry for unclear and possibly non-specific reason.

[1] - Obviously, not all, or even most. Software is complex, most users still behave as if half-drunk and unable to read, etc. Still, even 5 year olds can comprehend basic words and identify patterns. Figuring out that "could not connect to payment gateway" is serious, that "failed to write [blah blah tech terms]" that happens at random is probably not, etc. is within the cognitive reach of most users.