Comment by hyperpape

I’ve implemented a search crawler before, and detecting and switching to the WordPress API was one of the first things I implemented because it’s such an easy win. Practically every WordPress website had it open and there are a vast number of WordPress sites. The content that you can pull from the API is far easier to deal with because you can just pull all the articles and have the raw content plus metadata like tags, without having to try to separate the page content from all the junk that whatever theme they are using adds.

> The reality is that the ratio of "total websites" to "websites with an API" is likely on the order of 1M:1 (a guess).

This is entirely wrong. Aside from the vast number of WordPress sites, the other APIs the article mentions are things like ActivityPub, oEmbed, and sitemaps. Add on things like Atom, RSS, JSON Feed, etc. and the majority of sites have some kind of alternative to HTML that is easier for crawlers to deal with. It’s nothing like 1M:1.

> Investing the effort to 1) recognize, without programmer intervention, that some random website has an API and then 2) automatically, without further programmer intervention, retrieve the website data from that API and make intelligent use of it, is just not worth it to them when retrieving the HTML just works every time.

You are treating this like it’s some kind of open-ended exercise where you have to write code to figure out APIs on the fly. This is not the case. This is just “Hey, is there a <link rel=https://api.w.org/> in the page? Pull from the WordPress API instead”. That gets you better quality content, more efficiently, for >40% of all sites just by implementing one API.

> Investing the effort to 1) recognize, without programmer intervention, that some random website has an API

Hrm…

>> Like most WordPress blogs, my site has an API.

I think WordPress is big enough to warrant the effort. The fact that AI companies are destroying the web isn't news. But they could certainly do it a with a little less jackass. I support this take.

Right - the scraper operators already have an implementation which can use the HTML; why would they waste programmers time writing an API client when the existing system already does what they need?

If only there were some convenient technology that could help us sort out these many small cases automatically...

1M:1 by the way, but I agree.

That's a good point. Anyone who used the API properly were left with egg on their face and anyone who misused the site and just scraped HTML ended up unharmed

I think it's Jon Postel who was the original source of the principle (it's often called Postel's Law). https://www.rfc-editor.org/rfc/rfc761#section-2.10 is an example dating back to 1980.

This is Postel's Law, aka the Principle of Robustness:

    "be conservative in what you send, be liberal in what you accept"

> LLMs a lot of the ambiguity of HTML disappears as far as a scraper is concerned

The more effective way to think about it is that "the ambiguity" silently gets blended into the data. It might disappear from superficial inspection, but it's not gone.

The LLM is essentially just doing educated guesswork without leaving a consistent or thorough audit trail. This is a fairly novel capability and there are times where this can be sufficient, so I don't mean to understate it.

But it's a different thing than making ambiguity "disappear" when it comes to systems that actually need true accuracy, specificity, and non-ambiguity.

Where it matters, there's no substitute for "very explicit structured data" and never really can be.

"computers used to require"

please do not write code. ever. Thinking like this is why people now think that 16GB RAM is to little and 4 cores is the minimum.

API -> ~200,000 cycles to get data, RAM O(size of data), precise result

HTML -> LLM -> ~30,000,000,000 cycles to get data, RAM O(size of LLM weights), results partially random and unpredictable

I like to go a level beyond this and say: "Passing tests are fine and all, but the moment your tests mock or record-replay even the smallest bit of external data, the only accurate docs are your production error logs, or lack thereof."

As I wrote:

> Like most WordPress blogs, my site has an API.

WordPress, for all its faults, powers a fair number of websites. The schema is identical across all of them.

Shouldn't the llm that all this scraping is powering be able to help figure out which websites have an API and how to use it?