Comment by nektro
it's sad we've gotten to the point where mitigations against this have to be such a consideration when hosting a site
it's sad we've gotten to the point where mitigations against this have to be such a consideration when hosting a site
I used to say that, but last year it stopped being true for me.
They don't really have to be. I don't have many mitigations and the AI bots crawl my site and it's fine. The robots.txt is pretty simple too and is really just set up to help the robot not get stuck in loops (I use Mediawiki as the CMS and it has a lot of GET paths that a normal person wouldn't choose). In my case, a machine near my desk hosts everything and it's fine.