Comment by orliesaurus
Comment by orliesaurus 2 days ago
I'm a dev who's built both APIs and scrapers...
The API-first dream is nice in theory, BUT in practice most "public" APIs are behind paywalls or rate limits, and sometimes the API quietly omits the very data you're after. When that happens, you're flying blind if you refuse to look at the HTML...
Scraping isn't some moral failing... it's often the only way to see what real users see. ALSO, making your HTML semantic and accessible benefits humans and machines alike. It's weird to shame people for using the only reliable interface you provide.
I think the future is some kind of permission economy where trusted agents can fetch data without breaking TOS... Until that exists, complaining about scrapers while having no stable API seems like yelling at the weather.
You’re making up your position here though. They have a stable API. It’s the built in wordpress API. It’s on almost every Wordpress site on the planet. It’s about the most stable API you could use. You just want to use html and not be bothered with an API so don’t pretend like you would do something else if it was “stable”.