I've already seen LLMs suggest products using Reddit comments as a reference, and when I investigated the Reddit comment it was by a blatant astroturfing account (nearly every comment for the same product) that probably bought upvotes to get their comment to the top of the thread. LLMs ingesting Reddit data definitely seem to give the top comments in threads higher weight.
I used to be able to kind of deep dive music with the AI models. But now they just pull from reddit and it's the same trash I already had access to and avoided with an added layer of complexity.
Similar to this story from the other day: https://news.ycombinator.com/item?id=45521920
i've seen this in my niche, too. they posed as a customer of their product on reddit (i have the receipts) and now they brag on linkedin about being the google AI answer for their hyper-specific google search lol
The ability for LLMs to search the web made a big splash. Yet little emphasis was made on the fact that the web is a poisoned well. Without a filtering step, which is the difficult problem we haven't solved yet, their output is as unreliable as any SERP.