Comment by hartator
If API doesn’t have the data you want, this point is moot.
If API doesn’t have the data you want, this point is moot.
This is probably just a parallel discussion. I written plenty of successful web scrapers without LLM's, but in the last couple years, I've written a lot more where I didn't need to look at the web markup for more than a few seconds first, if at all. Often you can just copy-paste an example page into the LLM and have it generate accurate, consistent selectors. It's not much different when integrating with a formal API, except that the API usually has more explicit usage rules, and APIs will also often restrict data that can very obviously be used competitively.
Not GP, but I disagree. I've written successful, robust web scrapers without LLMs for decades.
What do you think the E in perl stands for?