Comment by jt2190

> The “client-side problems” Siebenmann is talking about are the various anti-bot measures (CAPTCHAs, rate limiters, etc.)

Directly from the article:

> it's not new, and it goes well beyond anti-crawler and anti-robot defenses. As covered by people like Alex Russell, it's routine for websites to ignore most real world client side concerns (also, and including on desktops). Just recently (as of August 2025), Github put out a major update that many people are finding immensely slow even on developer desktops.

The things he links to are about things that are unrelated to anti-bot measures.

The fact is, the web is an increasingly unpleasant place to visit. Users are subject to terrible UX – dark patterns, tracking, consent popups, ads everywhere, etc.

Then along come chatbots and when somebody asks about something, they are given the response on the spot without having to battle their way through all that crap to get what they want.

Of course users are going to flock to chatbots. If a site owner is worried they are losing traffic to chatbots, perhaps they should take a long, hard look at what kind of user experience they are serving up to people.

This is like streaming media all over again. Would you rather buy a legit DVD and wait for it to arrive in the post, then wait through an unskippable lecture about piracy, then wait through unstoppable trailers, then find your way through a weird, horrible DVD menu… or would you rather download it and avoid all that? The thing that alleviated piracy was not locking things down even more, it was making the legitimate route more convenient.

We need to make websites pleasant experiences again, and we can’t do that when we care about everything else more than the user experience.

Another implicit social contract is that you can tell whether a request is coming from a commercial or non-commercial source based on the originating ISP. This was always a heuristic but it was more reliable in the past.

If 1000 AWS boxes start hammering your API you might raise an eyebrow, but 1000 requests coming from residential ISPs around the world could be an organic surge in demand for your service.

Residential proxy services break this - which has been happening on some level for a long time, but the AI-training-set arms race has driven up demand and thus also supply.

It's quite easy to block all of AWS, for example, but it's less easy to figure out which residential IPs are part of a commercially-operated botnet.

The Internet and Web were both designed with the assumption of cooperation. I wonder what they would have built if they'd taken hostility into account from day one.

As we've seen security is really hard to build in after the fact. It has to be part of your design concept from the very first, and pervades every other decision you make. If you try to layer security on top you will lose.

Of course you may discover that a genuinely secure system is also unusably inconvenient and you lose to someone willing to take risks, and it's all moot.

> > One thing I've wound up feeling from all this is that the current web is surprisingly fragile. A significant amount of the web seems to have been held up by implicit understandings and bargains, not by technology.

This is something I've been pondering, and honestly I feel like the author doesn't go far enough. I would go as far as to say a lot of our modern society has been held up by these implicit social contracts. But nowadays we see things like gerrymandering in the US, or overusing the 49-3 in France to pass laws despite the parliament voting against them. Just an overall trend of only feeling constrained by the exact letter of the law and ignoring the spirit of it.

Except it turns out these implicit understandings that you shouldn't do that existed because breaking them makes life shittier for everyone, and that's what we're experiencing now.

The only possible contract moving forward will be something along the lines of agreeing to paying for access. I can't imagine how this will work, but with the collapse of the ad-supported Internet, that's the only way forward. It's not a straightforward problem to solve because people are not willing to pay for something before they see it, but after seeing the page, the incentive to pay is gone.