Comment by dlenski
Can't/won't these scrapers just switch to using VPNs or sshuttle or basically anything else that doesn't leak timing info about termination of TCP vs HTTP?
Can't/won't these scrapers just switch to using VPNs or sshuttle or basically anything else that doesn't leak timing info about termination of TCP vs HTTP?
Not really. You can have 100,000 IPs from proxies or use VPNs and have only 5 egress IPs.
Anybody who wants to stop the scraper could get browser fingerprints, cross reference similar ones with those IPs and quite safely ban them as its highly likely theyre not a legitimate customer.
Its a lot harder to do it for the 100k IPs because those IPs will also have legitimate customer traffic on them and its a lot more likely the browser fingerprint could just be legitimate.
The risk of false postives (blocking real people) is usually higher than just allowing the scrapers and the incetives of a lot of sites arent aligned with stopping scrapers anyway. Think eccommerce, do they _really_ care if the product is being sold to scalpers or real customers? If anything, that behaviour can raise perception of their brand, increase demand, increase prices.
This tool should have less false positives than most, so maybe it will see more adoption than others (TCP fingerprinting for example) but I dont think this is going to affect anyone doing scraping seriously/at scale.