Comment by matt-p
I do kind of yern for the simpler days when if a website didn't mind bots it allowed it and if they did they blocked your user agent.
I do kind of yern for the simpler days when if a website didn't mind bots it allowed it and if they did they blocked your user agent.
Back then websites weren’t so resource intensive. The negative backlash towards bots is kind of a side effect of how demanding expectations of web experiences has become.