Comment by chii
> many anti-scraping services explicitly allow access to Google and maybe couple of other search engines.
because google (and the couple of other search engines) provide enough value that offset the crawler's resource consumption.
> many anti-scraping services explicitly allow access to Google and maybe couple of other search engines.
because google (and the couple of other search engines) provide enough value that offset the crawler's resource consumption.
That's cool, but it's impossible for anyone to ever build a competitor that'd replace google without bypassing such services.