Comment by p0w3n3d
Is robots.txt still a thing?
Is robots.txt still a thing?
Maybe less so soon with content-signals
Some European laws give robots.txt files some legal weight. That's why you often see this in robots.txt that have content-signals
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
It is. A typically ignored thing.