Comment by miki123211

Comment by miki123211 a day ago

2 replies

Google does not use residential proxies.

This does nothing against your ability to scrape the web the Google way, AKA from your own assigned IP range, obeying robots.txt, and with an user agent that explicitly says what you're doing and gives website owners a way to opt out.

What Google doesn't want (and I don't think that's a bad thing) is competitors scraping the web in bad faith, without disclosing what they're doing to site owners and without giving them the ability to opt out.

If Google doesn't stop these proxies, unscrupulous parties will have a competitive advantage over Google, it's that simple. Then Google will have to decide between just giving up (unlikely) or becoming unscrupulous themselves.

ryanjshaw a day ago

> This does nothing against your ability to scrape the web the Google way

I thought that Google has access to significant portions of the internet that non-Google bots won’t have access to?

  • morkalork a day ago

    Their crawler has known IPs that get a white-glove treatment by every site with a paywall for example