Comment by Retr0id
> This seems like a simple test.
Not when the singular bot has a pool of millions of IPs to originate each request from.
If you think there's an easy solution here, productize it and make billions.
> This seems like a simple test.
Not when the singular bot has a pool of millions of IPs to originate each request from.
If you think there's an easy solution here, productize it and make billions.
IPs that you've never seen before are hitting a single random page deep within your site are bots, or first-time followers of a search engine link. Grey list them and respond slowly. If they are seen again at normal human rates, unthrottle them.