account42 3 days ago

It is because you are dealing with crawlers that already have a nontrivial cost per page, adding something relatively trivial that is still within the bounds regular users accept won't change the motivations of bad actors at all.

  • TZubiri 3 days ago

    What is the existing cost per page? as far as I know an http request and some string parsing is somewhat trivial, say 14kb of bandwidth per page?