Comment by simonw

Comment by simonw 21 hours ago

Here are 55 closed PRs in the curl repo which credit "sarif data" - I think those are the ones Daniel is talking about here https://github.com/curl/curl/pulls?q=is%3Apr+sarif+is%3Aclos...

This is notable given Daniel Stenberg's reports of being bombarded by total slop AI-generated false security issues in the past: https://www.linkedin.com/posts/danielstenberg_hackerone-curl...

Concerning HackerOne: "We now ban every reporter INSTANTLY who submits reports we deem AI slop. A threshold has been reached. We are effectively being DDoSed. If we could, we would charge them for this waste of our time"

Also this from January 2024: https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...

tomjakubowski 20 hours ago

Some of those bugs, like using the wrong printf-specifier for a size_t, would be flagged by the compiler with the right warning flags set. An AI oracle which tells me, "your project is missing these important bug-catching compiler warning flags," would be quite useful.

A few of these PRs are dependabot PRs which match on "sarif", I am guessing because the string shows up somewhere in the project's dependency list. "Joshua sarif data" returns a more specific set of closed PRs. https://github.com/curl/curl/pulls?q=is%3Apr+Joshua+sarif+da...

Reply View 0 replies

octocop 21 hours ago

The models used have improved quite well since then, I guess his change of opinion shows that.

Reply View 4 replies

Twirrim 20 hours ago

No, he's still dealing with a flood of crap, even in the last few weeks, off more modern models.
It's primarily from people just throwing source code at an LLM, asking it to find a vulnerability, and reporting it as-read, without having any actual understanding of if it is or isn't a vulnerability.
The difference in this particular case is it's someone who is: 1) Using tools specifically designed for security audits and investigations. 2) Takes the time to read and understand the vulnerability reported, and verifies that it is actually a vulnerability before reporting.
Point 2 is the most significant bar that people are woefully failing to meet and wasting a terrific amount of his time. The one that got shared from a couple of weeks ago https://hackerone.com/reports/3340109 didn't even call curl. It was straight up hallucination.

Reply View | 0 replies
simonw 21 hours ago

I think it's more about how people are using it. An amateur who spams him with GPT-5-Codex produced bug reports is still a waste of his time. Here a professional ran the tools and then applied their own judgement before sending the results to the curl maintainers.

Reply View | 1 reply
- tptacek 20 hours ago
  
  I keep irritating people with this observation but this was the status quo ante before AI, and at least an AI slop report shows clear intent; you can ban those submitters without even a glance at anything else they send.
  
  Reply View | 0 replies
whizzter 21 hours ago

It's probably also the difference of idiots hoping to cash out/get credit for vulnerabilities by just throwing ChatGPT at the wall compared to this where it seems a somewhat seasoned researcher is trialing more customized tools.

Reply View | 0 replies