Comment by Aloisius

Using status code 418 (I'm a teapot), while cute, actually works against you since even well behaved bots don't know how to handle it and thus might not treat it as a permanent status causing them to try to recrawl again later.

Plus you'll want to allow access to /robots.txt.

Of course, if they're hammering new connections, then automatically adding temporary firewall rules if the user agent requests anything but /robots.txt might be the easiest solution. Well or just stick Cloudflare in front of everything.