It's been going on for decades now too. It's a cat and mouse game that will be with us for as long as people try to exploit online resources with bots. Which will be until the internet is divided into nation nets, suffocated by commercial interests, and we all decide to go play outside instead.
That doesn't really challenge what I said, there's not much "different this time" except the scale is commensurate to the era. Search engine crawlers used to take down websites as well.
I understand and agree with what you are saying though, the cat and mouse is not necessarily technical. Part of solving the searchbot issue was also social, with things like robots.txt being a social contract between companies and websites, not a technical one.
No. This went into overdrive in the "AI" (crawlers for massive LLM for ML chatbot) era.
Frankly it's something I'm sad we don't yet see a lawsuit for similar to the times v OpenAI. A lot of "new crawlers" claim to innocently forget about established standards like robots.txt
I just wish people would name and shame the massive companies at the top stomping on the rest of the internet in an edge to "get a step up over the competition".