Comment by hedora

Comment by hedora 3 days ago

16 replies

View on Hacker News

This was obviously dumb when it launched:

1) scrapers just run a full browser and wait for the page to stabilize. They did this before this thing launched, so it probably never worked.

2) The AI reading the page needs something like 5 seconds * 1600W to process it. Assuming my phone can even perform that much compute as efficiently as a server class machine, it’d take a large multiple of five seconds to do it, and get stupid hot in the process.

Note that (2) holds even if the AI is doing something smart like batch processing 10-ish articles at once.

pilif 3 days ago

> This was obviously dumb when it launched:

Yes. Obviously dumb but also nearly 100% successful at the current point in time.

And likely going to stay successful as the non-protected internet still provides enough information to dumb crawlers that it’s not financially worth it to even vibe-code a workaround.

Or in other words: Anubis may be dumb, but the average crawler that completely exhausting some sites resources is even dumber.

And so it all works out.

And so the question remains: how dumb was it exactly, when it works so well and continues to work so well?

Reply View 9 replies
  • account42 3 days ago

    > Yes. Obviously dumb but also nearly 100% successful at the current point in time.

    Only if you don't care about negatively affecting real users.

    Reply View 1 reply
    • pilif 3 days ago

      I understand this as an argument that it’s better to be down for everyone than have a minority of users switch browsers.

      I’m not convinced by that makes sense.

      Now ideally you would have the resources to serve all users and all the AI bots without performance degradation, but for some projects that’s not feasible.

      In the end it’s all a compromise.

      Reply View 0 replies
  • kldg 3 days ago

    does it work well? I run chromium controlled by playwright for scraping and typically make Gemini implement the script for it because it's not worth my time otherwise. -but I'm not crawling the Internet generally (which I think there is very little financial incentive to do; it's a very expensive process even ignoring Anubis et al); it's always that I want something specific and am sufficiently annoyed by lack of API.

    regarding authentication mentioned elsewhere, passing cookies is no big deal.

    Reply View 1 reply
    • eaglefield 3 days ago

      Anubis is not meant to stop single endpoints from scraping. It's meant to make it harder for massive AI scrapers. The problematic ones evade rate limiting by using many different ip addresses, and make scraping cheaper on themselves by running headless. Anubis is specifically built to make that kind of scraping harder as i understand it.

      Reply View 0 replies
  • bananalychee 3 days ago

    Does it actually? I don't think I've seen a case study with hard numbers.

    Reply View 3 replies
  • snickerdoodle12 3 days ago

    the workaround is literally just running a headless browser, and that's pretty much the default nowadays.

    if you want to save some $$$ you can spend like 30 minutes making a cracker like in the article. just make it multi threaded, add a queue and boom, your scraper nodes can go back to their cheap configuration. or since these are AI orgs we're talking about, write a gpu cracker and laugh as it solves challenges far faster than any user could.

    custom solutions aren't worth it for individual sites, but with how widespread anubis is it's become worth it.

    Reply View 0 replies
pama 3 days ago

I agree. Your estimate for (2), about 0.0022 kWh, corresponds to about a sixth of the charge of an iPhone 15 pro and would take longer than ten minutes on the phone, even at max power draw. It feels about right for the amount of energy/compute of a large modern MoE loading large pages of several 10k tokens. For example this tech (couple month old) could input 52.3k tokens per second to a 672B parameter model, per H100 node instance, which probably burns about 6–8kW while doing it. The new B200s should be about 2x to 3x more energy efficient, but your point still holds within an order of magnitude.

https://lmsys.org/blog/2025-05-05-large-scale-ep/

Reply View 0 replies
rob_c 3 days ago

The argument doesn't quite hold. The mass scraping (for training) is almost never doing by a GPU system it's almost always done by a dedicated system running a full chrome fork in some automated way (not just the signatures but some bugs give that away).

And frankly processing a single page of text is run within a single token window so likely is run for a blink (ms) before moving onto the next data entry. The kicker is it's run over potentially thousands of times depending on your training strategy.

At inference there's now a dedicated tool that may perform a "live" request to scrape the site contents. But then this is just pushed into a massive context window to give the next token anyway.

Reply View 4 replies
  • account42 3 days ago

    The point is that scraping is already inherently cost-intensive so a small additional cost from having to solve a challenge is not going to make a dent in the equation. It doesn't matter what server is doing what for that.

    Reply View 3 replies
    • mistercheph 3 days ago

      100 billion web pages * 0.02 USD of PoW/page = 2 billion dollars, the point is not to stop every scraper/crawler, the point is to raise the costs enough to avoid being bombarded by all of them

      Reply View 2 replies
      • jsnell 3 days ago

        Yes, but it's not going to be 0.02 USD of PoW per page! That is an absurd number. It'd mean a two-hour proof of work for a server CPU, a ten hour proof of work for a phone.

        In reality you can do maybe a 1/10000th of that before the latency hit to real users becomes unacceptable.

        And then, the cost is not per page. The cost is per cookie. Even if the cookie is rate-limited, you could easily use it for 1000 downloads.

        Those two errors are multiplicative, so your numbers are probably off by about 7 orders of magnitudes. The cost of the PoW is not going to be $2B, but about $200.

        Reply View 0 replies
      • skeptrune 3 days ago

        I'm going to phrase the explanation like this in the future. Couldn't have said it better myself.

        Reply View 0 replies