Comment by idlewords Comment by idlewords 11 hours ago 0 replies Copy Link View on Hacker News This doesn't work with the kind of highly distributed crawling that is the problem now.