Comment by zenmac
Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers?
Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers?
Yep. I have my own small gitweb server, and I was getting literally thousands requests per second from bots who try to crawl through all the links.