Comment by nikolayasdf123 3 days ago
> and often lives behind advanced bot-detection systems.
for a purpose. they own that data. now you are scraping and re-selling it. how you think they will like that? expect court orders and cease and desist letters coming in.
OpenAI and Perplexity is notoriously getting sued by NYT, newspapers in Japan.
all do the same, which may be the reason why web is going to "close up", to prevent leaking data like this.
might be selective bias. try talking to folks who blocked scraping and you scraped them anyways and resold their data. see how happy they are. or try to talk to amazon, ebay, facebook for scraping their data, see how happy they are too.
We've spoken with many merchants who are super excited about this use-case, actually! We think the web is opening up – e.g. Exa, Parallel, OpenAI, Anthropic, Perplexity, etc. all do the same, and it seems like this agentic future demands it.