Comment by SkiFire13

> For someone who does not follow the web's history, how would one produce direct evidence that the bias exists

Take a bunch of websites, fetch their robots.txt file and check how many allow GoogleBot but not others?