Comment by WhyNotHugo

If we did, bot authors would comply by just changing their User-Agent to something different that’s not expressly forbidden.

(Disallowing * isn’t usually an option since it makes you disappear from search engines).