Comment by Reason077 Comment by Reason077 4 hours ago 1 reply Copy Link View on Hacker News Do we need a "robots must respect robots.txt" law?
Copy Link WhyNotHugo 3 hours ago Collapse Comment - If we did, bot authors would comply by just changing their User-Agent to something different that’s not expressly forbidden.(Disallowing * isn’t usually an option since it makes you disappear from search engines). Reply View | 0 replies
If we did, bot authors would comply by just changing their User-Agent to something different that’s not expressly forbidden.
(Disallowing * isn’t usually an option since it makes you disappear from search engines).