jsheard 2 days ago

> No, put up a loginwall or paywall, authenticate users, and go private.

We know for a fact that AI companies don't respect that, if they want data that's behind a paywall then they'll jump through hoops to take it anyway.

https://www.theguardian.com/technology/2025/jan/10/mark-zuck...

If they don't have to abide by "norms" then we don't have to for their sake. Fuck 'em.

Reply View 3 replies
  • observationist 2 days ago

    [flagged]

    Reply View 2 replies
    • tir 2 days ago

      >the law explicitly allows scraping and crawling.

      Nepenthes also allows scraping and crawling, for as long as you like.

      Reply View 0 replies
    • blibble 2 days ago

      this is a very US-ian view of the world

      my site is not in the US, I am not a US citizen. US law does not apply to me.

      under UK law: robots.txt is an access control mechanism (weak or otherwise)

      knowingly bypassing it is likely a criminal offence under the Computer Misuse Act

      good luck suing me because you got stuck when you smashed my window and climbed through it

      Reply View 0 replies