Comment by diggan

Comment by diggan 2 days ago

8 replies

> There are already “infinite” websites like these on the Internet.

Cool. And how much of the software driving these websites is FOSS and I can download and run it for my own (popular enough to be crawled more than daily by multiple scrapers) website?

gruez 2 days ago
  • johnisgood a day ago

    How is that infinite if the last one is always the same? Am I misunderstanding this? I assumed it is almost like an infinite scroll or something.

  • diggan 2 days ago

    Aren't those finite lists? How is a scraper (normal or LLM) supposed to "get stuck" on those?

    • gruez 2 days ago

      even though 2^128 uuids is technically "finite", for all intents and purposes is infinite to a scraper.

hartator 2 days ago

Every not found pages that don’t return a 404 http header is basically an infinite trap.

It’s useless to do this though as all crawlers have a way to handle this. It’s very crawler 101.