Comment by ehhthing Comment by ehhthing a day ago 2 replies Copy Link View on Hacker News You say this like robots.txt doesn't exist.
Copy Link toofy 21 hours ago Next Collapse Comment - it almost sounds like they’re saying the contents of robots.txt shouldn’t matter… because google exists? or something?implying “robots.txt explicitly says i can’t scrape their site, well i want that data, so im directing my bot to take it anyway.” Reply View | 0 replies
Copy Link sitzkrieg 21 hours ago Prev Collapse Comment - so many things flat out ignore it in 2026 let's be real Reply View | 0 replies
it almost sounds like they’re saying the contents of robots.txt shouldn’t matter… because google exists? or something?
implying “robots.txt explicitly says i can’t scrape their site, well i want that data, so im directing my bot to take it anyway.”