Comment by ghxst

Comment by ghxst 3 days ago

8 replies

Cool service but how will you deal / how do you plan to deal with anti scraping and anti bot services like Akamai, Arkose, Cloudflare, DataDome etc.? Automation of the web isn't solved by another playwright or puppeteer abstraction, you need to solve more fundemental problems in order to mitigate the issues you run into at scale.

jasonwcfan 3 days ago

I mentioned this in another comment, but I know from experience that it's impossible to reliably differentiate bots from humans over a network. And since the right to automate browsers has survived repeated legal challenges, all vendors can do is make it incrementally harder to weed out the low sophistication actors.

This actually creates an evergreen problem that companies need to overcome, and our paid version will probably involve helping companies overcome these barriers.

Also I should clarify that we're explicitly not trying to build a playwright abstraction - we're trying to remain as unopinionated as possible about how developers code the bot, and just help with the network-level infrastructure they'll need to make it reliable and make it scale.

It's good feedback for us, we'll make that point more clear!

  • ghxst 2 days ago

    > but I know from experience that it's impossible to reliably differentiate bots from humans over a network

    While this might be true in theory, it doesn't stop them from trying! And believe me, it's getting to a point where the WAF settings on some websites are even annoying the majority of the real users! Some of the issues I am hinting at however are fundemental issues you run into when automating the web using any mainstream browser that hasn't had some source code patches, I'm curious to see if a solution to that will be part of your service if you decide to tackle it.

  • candiddevmike 3 days ago

    Don't take this the wrong way, but this is the kind of unethical behavior that our industry should frown upon IMO. I view this kind of thing on the same level as DDoS-as-a-Service companies.

    I wish your company the kind of success it deserves.

    • jasonwcfan 3 days ago

      Why is it unethical when courts have repeatedly affirmed browser automation to be legal and permitted?

      If anything, it's unethical for companies to dictate how their customers can access services they've already paid for. If I'm paying hundreds of thousands per year for software, shouldn't I be allowed to build automations over it? Instead, many enterprise products go to great lengths to restrict this kind of usage.

      I led the team that dealt with DDoS and other network level attacks at Robinhood so I know how harmful they are. But I also got to see many developers using our services in creative ways that could have been a whole new product (example: https://github.com/sanko/Robinhood).

      Instead we had to go after these people and shut them down because it wasn't aligned with the company's long term risk profile. It sucked.

      That's why we're focused on authenticated agents for B2B use cases, not the kind of malicious bots you might be thinking of.

      • tempest_ 3 days ago

        > they've already paid for.

        That is the crux, rarely is it a service being scraped that they paid for

      • [removed] 3 days ago
        [deleted]