Comment by davidsojevic 9 days ago
There's a fork of this that has some great improvements over to the top of the original and it is also actively maintained: https://github.com/lexiforest/curl-impersonate
There's also Python bindings for the fork for anyone who uses Python: https://github.com/lexiforest/curl_cffi
Easy. Just make a small fragment shader to produce a token in your client. No bot is going to waste GPU resources to compile your shader.
Why do people even think this? Bots almost always just use headful instrumented browsers now. if a human sitting at a keyboard can load the content, so can a bot.
Security measures never prevent all abuse. They raise the cost of abuse above an acceptable threshold. Many things work like this. Cleaning doesn't eliminate dirt, it dilutes the dirt below an acceptable threshold. Same for "repairing" and "defects", and some other pairs of things that escape me atm.
We are talking about Curl bots here. How is what you are saying relevant?
Maybe you can call a WebGL extension that isn't supported. Or better yet have a couple of overdraws of quads. Their bot will handle it, but it will throttle their CPU like gangbusters.
You are just guessing, please stop. Also, you’re wrong. All serious scraping is using browsers today.
How do you automate that? Just generate a new token for each day.
There's also a module for fully integrating this with the Python requests library: https://github.com/el1s7/curl-adapter
All these "advanced" technologies that change faster than I can turn my neck, to make a simple request that looks like it was one of the "certified" big 3 web browsers, which will ironically tax the server less than a certified browser. Is this the nightmare dystopia I was warned about in the 90's? I wonder if anyone here can name the one company that is responsible for this despite positioning themselves as a good guy open source / hacker community contributor.
I suppose it does make sense that a "make curl look like a browser" program would get sponsored by "bypass bot detection" services...