Comment by tgv Comment by tgv 6 months ago 1 reply Copy Link View on Hacker News You can't make money out of studying robots.txt, but you can avoid costs skipping bad web sites.
Copy Link xeromal 6 months ago Collapse Comment - Sounds like a benefit for the site owner. lol. It accomplished what they wanted. Reply View | 0 replies
Sounds like a benefit for the site owner. lol. It accomplished what they wanted.