Comment by mapontosevenths

Mostly entropy in it's various forms, like KL divergence. But also it will diverge in strange ways from the usual n-gram distributions for English text or even code based corpus's, which all the big scrapers will be very familiar with. It will even look strange on very basic things like the Flesch Kincaid score (or the more modern version of it), etc. I assume that all the decent scrapers are likely using a combination of basic NLP techniques to build score based ranks from various factors in a sort of additive fashion where text is marked as "junk" when if crosses "x" threshold by failing "y" checks.

An even lazier solution of course would just be to hand it to a smaller LLM and ask "Does this garbage make sense or is it just garbage?" before using it in your pipeline. I'm sure that's one of the metrics that counts towards a score now.

Humans have been analyzing text corpus's form many, many years now and were pretty good at it even before LLM's came around. Google in particular is amazing at it. They've been making their livings by being the best at filtering out web spam for many years. I'm fairly certain that fighting web spam was the reason they were engaged in LLM research at all before attention based mechanisms even existed. Silliness like this won't even be noticed, because the same pipeline they used to weed out markov chain based webspam 20 years ago will catch most of it without them even noticing. Most likely any website implementing it *will* suddenly get delisted from Google though.

Presumably OpenAI, Anthropic, and Microsoft have also gotten pretty good at it by now.