Comment by stanfordkid
Comment by stanfordkid 13 hours ago
In my opinion the whole agents.txt thing is bullshit and/or politeness, and should have zero legal significance.
If something is openly available on the internet you should be able to crawl it, and it is the server's responsibility to identify, authenticate and/or ban clients that do not adhere to it's requirements. If Amazon states "You must not use agents" in it's website terms of service then it would be the individual perplexity user not perplexity itself that is breaking it's terms of service, since they are the one operating the agent.
All in all lame move by Amazon.
Does everything you say apply equally to DDoS attackers? If not, you need to come up with some way to differentiate ‘malicious users’ from benign ones. Amazon’s ‘bright line’ seems reasonable, though I’d personally love to see an even better one.