Comment by cyberax Comment by cyberax a day ago 2 replies Copy Link View on Hacker News You can thank AI bots for that.
Copy Link zenmac 19 hours ago Collapse Comment - Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers? Reply View | 1 reply Copy Link cyberax 14 hours ago Parent Collapse Comment - Yep. I have my own small gitweb server, and I was getting literally thousands requests per second from bots who try to crawl through all the links. Reply View | 0 replies
Copy Link cyberax 14 hours ago Parent Collapse Comment - Yep. I have my own small gitweb server, and I was getting literally thousands requests per second from bots who try to crawl through all the links. Reply View | 0 replies
Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers?