Comment by toast0
> /etc/hosts works until you need to change an IP across 10,000 servers in under a minute.
I wouldn't use /etc/hosts for something with those needs, but pushing a smallish file to 10,000 hosts in a minute is readily achievable if you need it. I don't have an estimate kf the number of ssh handshakes per second a typical client machine could do, but if you run 512 pushes in parallel, I bet you can get it pushed in way under a minute; assuming you have sufficient bandwidth I would avoid trying to do a differential update because the interactivity is too many round trips and you'll blow your time budget, but you could keep several known hashes and differential update from those and full push if no match). Up to you if you want to also run a command to reload the file, or if you'd rather kqueue/inotify/just a periodic poll for change.
If you have frequent changes, you probably want to do a phone tree approach where you push to a few servers per location and those push to the local servers. And probably a way to do single name announcements. Or maybe something else.
But, if your system communicates by server hostnames and your server hostnames don't change often and ips of active servers don't change often, using /etc/hosts for that can be pretty useful. Everytime you get new servers, you have to repush the hostfile, but it's really not a big deal; and it changes a dynamic dependency to a static one. You can even run monitors to confirm your servers aren't resolving each other and that they all have acceptable host files.
(Late edit, composed while I lost connectivity) Also, if your scale is hundreds of thousands of hosts, it might get pretty nasty, but chances are you wouldn't resolve every name on every host, so something smarter is needed; which could be DNS or could be selective pushing or could be some other dynamic system that's not entirely DNS.